2024-04-21T08:32:28.739 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-21T08:32:28.739 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527 --owner scheduled_leonidus@teuthology --name leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi --description fs/workload/{0-ubuntu_22.04 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/ffsb}} -- /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527/orig.config.yaml 2024-04-21T08:32:28.799 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-21T08:32:28.891 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527 branch: wip-lusov-quiescer description: fs/workload/{0-ubuntu_22.04 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/ffsb}} email: null first_in_suite: false job_id: '7666527' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: 4f6bc5273643aaf5660909e85400647a8951c10b owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7599 sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e sleep_before_teardown: 0 subset: 111/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa suite_relpath: qa suite_repo: https://github.com/leonid-s-usov/ceph-ceph suite_sha1: 4f6bc5273643aaf5660909e85400647a8951c10b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEgJqGb177jMCgam1FMGMC5J+mtGAMJm/YUa1LD1jz7gae3F+oiFOAJR4r0Tm+WlDiDiyqk4pya/RDJXg7xtlD8= smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIRTBmb2VQKdjWibsEHJj6weWUTphcKqwGFANtcH+FdN5XkWCTPV8GZ5qUzt0DpD8HLPnGzeKbEcKrjHYq+sZv8= smithi171.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+AIZOqUmPiig+5fMqf6G5AeCEwBXiFVSsm5N/qUeE0fKTzPlxCjMIQgSV4fDBnQgTVwJYbI0TlE2r2p3MO+GM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-21_08:12:55 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-21T08:32:28.891 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa; will attempt to use it 2024-04-21T08:32:28.892 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks 2024-04-21T08:32:28.892 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-21T08:32:28.893 INFO:teuthology.task.internal:Checking packages... 2024-04-21T08:32:28.917 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a630765c1f93c6e61d9b40c4b76437611d5ef29e' 2024-04-21T08:32:28.917 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-21T08:32:28.917 INFO:teuthology.packaging:ref: None 2024-04-21T08:32:28.917 INFO:teuthology.packaging:tag: None 2024-04-21T08:32:28.917 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-21T08:32:28.917 INFO:teuthology.packaging:sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:32:28.917 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-21T08:32:29.086 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3154.ga630765c 2024-04-21T08:32:29.087 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-21T08:32:29.097 INFO:teuthology.task.internal:no buildpackages task found 2024-04-21T08:32:29.097 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-21T08:32:29.114 INFO:teuthology.task.internal:Saving configuration 2024-04-21T08:32:29.136 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-21T08:32:29.142 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T08:32:29.166 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:27:11.241841', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEgJqGb177jMCgam1FMGMC5J+mtGAMJm/YUa1LD1jz7gae3F+oiFOAJR4r0Tm+WlDiDiyqk4pya/RDJXg7xtlD8='} 2024-04-21T08:32:29.188 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:27:11.243361', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIRTBmb2VQKdjWibsEHJj6weWUTphcKqwGFANtcH+FdN5XkWCTPV8GZ5qUzt0DpD8HLPnGzeKbEcKrjHYq+sZv8='} 2024-04-21T08:32:29.211 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:27:11.244753', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+AIZOqUmPiig+5fMqf6G5AeCEwBXiFVSsm5N/qUeE0fKTzPlxCjMIQgSV4fDBnQgTVwJYbI0TlE2r2p3MO+GM='} 2024-04-21T08:32:29.211 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-21T08:32:29.217 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-21T08:32:29.217 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-21T08:32:29.217 INFO:teuthology.task.internal:roles: ubuntu@smithi171.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-21T08:32:29.217 INFO:teuthology.run_tasks:Running task console_log... 2024-04-21T08:32:29.315 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fce6dafe0d0>, signals=[15]) 2024-04-21T08:32:29.316 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-21T08:32:29.323 INFO:teuthology.task.internal:Opening connections... 2024-04-21T08:32:29.323 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-04-21T08:32:29.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:32:29.403 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2024-04-21T08:32:29.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:32:29.477 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi171.front.sepia.ceph.com 2024-04-21T08:32:29.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:32:29.552 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-21T08:32:29.560 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-04-21T08:32:29.578 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-04-21T08:32:29.579 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-04-21T08:32:29.636 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-04-21T08:32:29.636 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-04-21T08:32:29.636 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-04-21T08:32:29.636 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-04-21T08:32:29.637 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-04-21T08:32:29.637 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-04-21T08:32:29.637 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T08:32:29.637 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-04-21T08:32:29.637 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-04-21T08:32:29.637 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T08:32:29.637 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-04-21T08:32:29.637 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T08:32:29.637 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T08:32:29.637 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T08:32:29.638 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-04-21T08:32:29.657 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2024-04-21T08:32:29.675 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2024-04-21T08:32:29.675 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2024-04-21T08:32:29.732 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2024-04-21T08:32:29.732 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2024-04-21T08:32:29.732 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2024-04-21T08:32:29.732 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2024-04-21T08:32:29.732 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2024-04-21T08:32:29.732 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2024-04-21T08:32:29.732 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T08:32:29.732 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2024-04-21T08:32:29.733 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2024-04-21T08:32:29.733 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T08:32:29.733 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2024-04-21T08:32:29.733 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T08:32:29.733 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T08:32:29.733 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T08:32:29.733 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2024-04-21T08:32:29.757 DEBUG:teuthology.orchestra.run.smithi171:> uname -m 2024-04-21T08:32:29.776 INFO:teuthology.orchestra.run.smithi171.stdout:x86_64 2024-04-21T08:32:29.776 DEBUG:teuthology.orchestra.run.smithi171:> cat /etc/os-release 2024-04-21T08:32:29.833 INFO:teuthology.orchestra.run.smithi171.stdout:NAME="CentOS Stream" 2024-04-21T08:32:29.833 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION="9" 2024-04-21T08:32:29.833 INFO:teuthology.orchestra.run.smithi171.stdout:ID="centos" 2024-04-21T08:32:29.834 INFO:teuthology.orchestra.run.smithi171.stdout:ID_LIKE="rhel fedora" 2024-04-21T08:32:29.834 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_ID="9" 2024-04-21T08:32:29.834 INFO:teuthology.orchestra.run.smithi171.stdout:PLATFORM_ID="platform:el9" 2024-04-21T08:32:29.834 INFO:teuthology.orchestra.run.smithi171.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T08:32:29.834 INFO:teuthology.orchestra.run.smithi171.stdout:ANSI_COLOR="0;31" 2024-04-21T08:32:29.834 INFO:teuthology.orchestra.run.smithi171.stdout:LOGO="fedora-logo-icon" 2024-04-21T08:32:29.834 INFO:teuthology.orchestra.run.smithi171.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T08:32:29.834 INFO:teuthology.orchestra.run.smithi171.stdout:HOME_URL="https://centos.org/" 2024-04-21T08:32:29.834 INFO:teuthology.orchestra.run.smithi171.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T08:32:29.834 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T08:32:29.834 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T08:32:29.834 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2024-04-21T08:32:29.857 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-21T08:32:29.867 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-21T08:32:29.873 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-21T08:32:29.873 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-04-21T08:32:29.876 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2024-04-21T08:32:29.878 DEBUG:teuthology.orchestra.run.smithi171:> test '!' -e /home/ubuntu/cephtest 2024-04-21T08:32:29.894 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-21T08:32:29.900 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-21T08:32:29.900 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-04-21T08:32:29.933 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2024-04-21T08:32:29.936 DEBUG:teuthology.orchestra.run.smithi171:> test -z $(ls -A /var/lib/ceph) 2024-04-21T08:32:30.035 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-21T08:32:30.113 INFO:teuthology.run_tasks:Running task kernel... 2024-04-21T08:32:30.127 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-21T08:32:30.128 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-21T08:32:30.128 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-21T08:32:30.128 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T08:32:30.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:32:30.145 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-04-21T08:32:30.204 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-437.el9.x86_64 2024-04-21T08:32:30.204 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.14.0-437.el9.x86_64 2024-04-21T08:32:30.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-04-21T08:32:40.817 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 911 kB/s | 8.0 MB 00:09 2024-04-21T08:32:42.801 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-04-21T08:32:54.268 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 817 kB/s | 6.2 MB 00:07 2024-04-21T08:32:55.727 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 16 kB 00:00 2024-04-21T08:32:57.582 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-21T08:33:02.812 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-21T08:33:04.785 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-21T08:33:04.785 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-21T08:33:04.868 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-21T08:33:04.869 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-21T08:33:04.869 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-21T08:33:04.989 DEBUG:teuthology.orchestra.run.smithi028:> echo no | sudo yum reinstall kernel || true 2024-04-21T08:33:05.551 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:03 ago on Sun 21 Apr 2024 08:33:02 AM UTC. 2024-04-21T08:33:05.683 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-21T08:33:05.692 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-21T08:33:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-04-21T08:33:05.692 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-21T08:33:05.692 INFO:teuthology.orchestra.run.smithi028.stdout:Reinstalling: 2024-04-21T08:33:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-21T08:33:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:33:05.692 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-21T08:33:05.692 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-21T08:33:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:33:05.693 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 255 k 2024-04-21T08:33:05.693 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 0 2024-04-21T08:33:05.693 INFO:teuthology.orchestra.run.smithi028.stderr:Operation aborted. 2024-04-21T08:33:05.752 INFO:teuthology.orchestra.run.smithi028.stdout:Is this ok [y/N]: 2024-04-21T08:33:05.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum reinstall -y kernel || true 2024-04-21T08:33:06.271 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:04 ago on Sun 21 Apr 2024 08:33:02 AM UTC. 2024-04-21T08:33:06.391 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-21T08:33:06.392 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-21T08:33:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-04-21T08:33:06.392 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-21T08:33:06.392 INFO:teuthology.orchestra.run.smithi028.stdout:Reinstalling: 2024-04-21T08:33:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-21T08:33:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:33:06.392 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-21T08:33:06.392 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-21T08:33:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:33:06.393 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 255 k 2024-04-21T08:33:06.393 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 0 2024-04-21T08:33:06.393 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-21T08:33:07.097 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-437.el9.x86_64.rpm 402 kB/s | 255 kB 00:00 2024-04-21T08:33:07.097 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:33:07.097 INFO:teuthology.orchestra.run.smithi028.stdout:Total 362 kB/s | 255 kB 00:00 2024-04-21T08:33:07.113 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-21T08:33:07.301 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-21T08:33:07.301 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-21T08:33:07.306 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-21T08:33:07.306 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-21T08:33:07.454 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-21T08:33:07.523 INFO:teuthology.orchestra.run.smithi028.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-21T08:33:07.523 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T08:33:08.199 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T08:33:08.199 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-21T08:33:08.938 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T08:33:08.938 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:33:08.938 INFO:teuthology.orchestra.run.smithi028.stdout:Reinstalled: 2024-04-21T08:33:08.938 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-21T08:33:08.938 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:33:08.938 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-21T08:33:09.109 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-04-21T08:33:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-21T08:33:09.198 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-21T08:33:09.198 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-21T08:33:09.198 DEBUG:teuthology.parallel:result is None 2024-04-21T08:33:09.198 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-21T08:33:09.210 INFO:teuthology.task.internal:Creating test directory... 2024-04-21T08:33:09.210 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T08:33:09.213 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T08:33:09.215 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T08:33:09.233 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-21T08:33:09.240 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-21T08:33:09.249 INFO:teuthology.task.internal:Creating archive directory... 2024-04-21T08:33:09.250 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T08:33:09.268 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T08:33:09.272 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T08:33:09.370 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-21T08:33:09.377 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-21T08:33:09.377 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T08:33:09.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:33:09.394 DEBUG:teuthology.orchestra.run.smithi171:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T08:33:09.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:33:09.412 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T08:33:09.414 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T08:33:09.436 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T08:33:09.451 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:33:09.466 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:33:09.477 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:33:09.488 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:33:09.516 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:33:09.522 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:33:09.523 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-21T08:33:09.531 INFO:teuthology.task.internal:Configuring sudo... 2024-04-21T08:33:09.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T08:33:09.533 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T08:33:09.566 DEBUG:teuthology.orchestra.run.smithi171:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T08:33:09.593 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-21T08:33:09.602 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-21T08:33:09.602 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T08:33:09.605 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T08:33:09.634 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T08:33:09.650 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:33:09.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:33:09.744 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:33:09.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:33:09.866 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:33:09.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T08:33:09.930 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:33:09.972 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:33:10.023 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:33:10.105 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:33:10.146 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-21T08:33:10.146 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T08:33:10.210 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:33:10.253 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:33:10.313 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:33:10.398 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:33:10.439 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-21T08:33:10.439 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T08:33:10.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-04-21T08:33:10.508 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2024-04-21T08:33:10.510 DEBUG:teuthology.orchestra.run.smithi171:> sudo service rsyslog restart 2024-04-21T08:33:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T08:33:10.554 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T08:33:10.583 INFO:teuthology.orchestra.run.smithi171.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T08:33:10.901 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-21T08:33:10.909 INFO:teuthology.task.internal:Starting timer... 2024-04-21T08:33:10.909 INFO:teuthology.run_tasks:Running task pcp... 2024-04-21T08:33:10.919 INFO:teuthology.run_tasks:Running task selinux... 2024-04-21T08:33:10.926 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-21T08:33:10.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-04-21T08:33:10.988 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-04-21T08:33:10.991 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2024-04-21T08:33:11.059 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2024-04-21T08:33:11.061 DEBUG:teuthology.orchestra.run.smithi171:> sudo service auditd rotate 2024-04-21T08:33:11.120 INFO:teuthology.orchestra.run.smithi171.stdout:Rotating logs: 2024-04-21T08:33:11.122 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-21T08:33:11.123 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-04-21T08:33:11.148 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-04-21T08:33:11.148 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2024-04-21T08:33:11.171 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2024-04-21T08:33:11.172 DEBUG:teuthology.orchestra.run.smithi171:> /usr/sbin/getenforce 2024-04-21T08:33:11.191 INFO:teuthology.orchestra.run.smithi171.stdout:Permissive 2024-04-21T08:33:11.191 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi152.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi171.front.sepia.ceph.com': 'permissive'} 2024-04-21T08:33:11.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T08:33:11.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:33:11.219 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T08:33:11.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:33:11.244 DEBUG:teuthology.orchestra.run.smithi171:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T08:33:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:33:11.272 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-21T08:33:11.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-04-21T08:33:11.303 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2024-04-21T08:33:11.332 DEBUG:teuthology.orchestra.run.smithi171:> sudo /usr/sbin/setenforce permissive 2024-04-21T08:33:11.357 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-21T08:33:11.366 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-21T08:33:11.372 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-21T08:33:11.519 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-21T08:33:11.535 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-21T08:33:11.536 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi152.front.sepia.ceph.com,smithi171.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-21T08:37:11.997 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi152.front.sepia.ceph.com'), Remote(name='ubuntu@smithi171.front.sepia.ceph.com')] 2024-04-21T08:37:11.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-21T08:37:11.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:37:12.076 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-04-21T08:37:12.157 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-21T08:37:12.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-04-21T08:37:12.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:37:12.233 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-04-21T08:37:12.316 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-04-21T08:37:12.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-04-21T08:37:12.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:37:12.393 DEBUG:teuthology.orchestra.run.smithi171:> true 2024-04-21T08:37:12.477 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-04-21T08:37:12.477 INFO:teuthology.run_tasks:Running task clock... 2024-04-21T08:37:12.491 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-21T08:37:12.491 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T08:37:12.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T08:37:12.494 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T08:37:12.494 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T08:37:12.496 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T08:37:12.496 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T08:37:12.522 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T08:37:12.523 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T08:37:12.537 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T08:37:12.537 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T08:37:12.545 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T08:37:12.559 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T08:37:12.559 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2024-04-21T08:37:12.561 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-04-21T08:37:12.570 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2024-04-21T08:37:12.572 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-04-21T08:37:12.582 INFO:teuthology.orchestra.run.smithi171.stderr:sudo: ntpd: command not found 2024-04-21T08:37:12.583 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T08:37:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T08:37:12.593 INFO:teuthology.orchestra.run.smithi171.stdout:506 Cannot talk to daemon 2024-04-21T08:37:12.596 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T08:37:12.600 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T08:37:12.607 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T08:37:12.620 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T08:37:12.646 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-04-21T08:37:12.650 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T08:37:12.650 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-04-21T08:37:12.650 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:37:12.650 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:37:12.650 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:37:12.650 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:37:12.655 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-04-21T08:37:12.659 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T08:37:12.659 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-04-21T08:37:12.659 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:37:12.659 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:37:12.659 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:37:12.659 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:37:12.676 INFO:teuthology.orchestra.run.smithi171.stderr:bash: line 1: ntpq: command not found 2024-04-21T08:37:12.679 INFO:teuthology.orchestra.run.smithi171.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T08:37:12.679 INFO:teuthology.orchestra.run.smithi171.stdout:=============================================================================== 2024-04-21T08:37:12.679 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:37:12.680 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:37:12.680 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:37:12.680 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:37:12.680 INFO:teuthology.run_tasks:Running task install... 2024-04-21T08:37:12.688 DEBUG:teuthology.task.install:project ceph 2024-04-21T08:37:12.688 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e'}} 2024-04-21T08:37:12.688 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e'} 2024-04-21T08:37:12.688 INFO:teuthology.task.install:Using flavor: default 2024-04-21T08:37:12.693 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-21T08:37:12.693 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-21T08:37:12.694 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'tag': None, 'wait_for_package': False} 2024-04-21T08:37:12.694 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:37:12.696 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'tag': None, 'wait_for_package': False} 2024-04-21T08:37:12.696 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:37:12.697 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'tag': None, 'wait_for_package': False} 2024-04-21T08:37:12.697 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:37:12.839 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/ 2024-04-21T08:37:12.839 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3154.ga630765c 2024-04-21T08:37:12.848 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/ 2024-04-21T08:37:12.848 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3154.ga630765c 2024-04-21T08:37:12.907 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/ 2024-04-21T08:37:12.908 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3154.ga630765c 2024-04-21T08:37:12.959 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T08:37:12.959 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-21T08:37:12.960 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T08:37:12.985 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T08:37:12.985 DEBUG:teuthology.orchestra.run.smithi152:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a630765c1f93c6e61d9b40c4b76437611d5ef29e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T08:37:13.009 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T08:37:13.009 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:37:13.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T08:37:13.016 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T08:37:13.016 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-21T08:37:13.016 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T08:37:13.035 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T08:37:13.036 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a630765c1f93c6e61d9b40c4b76437611d5ef29e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T08:37:13.040 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T08:37:13.040 DEBUG:teuthology.orchestra.run.smithi171:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a630765c1f93c6e61d9b40c4b76437611d5ef29e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T08:37:13.049 DEBUG:teuthology.orchestra.run.smithi152:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T08:37:13.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T08:37:13.104 DEBUG:teuthology.orchestra.run.smithi171:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T08:37:13.121 DEBUG:teuthology.orchestra.run.smithi152:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T08:37:13.176 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T08:37:13.178 DEBUG:teuthology.orchestra.run.smithi171:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T08:37:13.184 INFO:teuthology.orchestra.run.smithi152.stdout:check_obsoletes = 1 2024-04-21T08:37:13.186 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum clean all 2024-04-21T08:37:13.240 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-04-21T08:37:13.240 INFO:teuthology.orchestra.run.smithi171.stdout:check_obsoletes = 1 2024-04-21T08:37:13.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-04-21T08:37:13.242 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum clean all 2024-04-21T08:37:13.410 INFO:teuthology.orchestra.run.smithi152.stdout:48 files removed 2024-04-21T08:37:13.437 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T08:37:13.465 INFO:teuthology.orchestra.run.smithi171.stdout:48 files removed 2024-04-21T08:37:13.475 INFO:teuthology.orchestra.run.smithi028.stdout:48 files removed 2024-04-21T08:37:13.492 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T08:37:13.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T08:37:14.113 INFO:teuthology.orchestra.run.smithi152.stdout:ceph packages for x86_64 201 kB/s | 83 kB 00:00 2024-04-21T08:37:14.142 INFO:teuthology.orchestra.run.smithi171.stdout:ceph packages for x86_64 214 kB/s | 83 kB 00:00 2024-04-21T08:37:14.243 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 178 kB/s | 83 kB 00:00 2024-04-21T08:37:14.447 INFO:teuthology.orchestra.run.smithi152.stdout:ceph noarch packages 40 kB/s | 12 kB 00:00 2024-04-21T08:37:14.500 INFO:teuthology.orchestra.run.smithi171.stdout:ceph noarch packages 37 kB/s | 12 kB 00:00 2024-04-21T08:37:14.694 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-04-21T08:37:14.881 INFO:teuthology.orchestra.run.smithi152.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-04-21T08:37:14.934 INFO:teuthology.orchestra.run.smithi171.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-04-21T08:37:15.070 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-04-21T08:37:15.582 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-04-21T08:37:15.695 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-21T08:37:16.515 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.0 MB 00:01 2024-04-21T08:37:17.346 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-04-21T08:37:18.269 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-04-21T08:37:21.739 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.2 MB 00:00 2024-04-21T08:37:23.598 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-04-21T08:37:24.693 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-04-21T08:37:29.644 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - CRB 808 kB/s | 6.2 MB 00:07 2024-04-21T08:37:29.991 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-21T08:37:31.245 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-21T08:37:32.210 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:37:32.210 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:37:32.222 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T08:37:32.223 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T08:37:32.224 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T08:37:32.225 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T08:37:32.226 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T08:37:32.229 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T08:37:32.229 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T08:37:32.294 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-21T08:37:32.303 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-21T08:37:32.303 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-21T08:37:32.304 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-21T08:37:32.304 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-21T08:37:32.304 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T08:37:32.304 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.0.0-3154.ga630765c.el9 ceph 6.6 k 2024-04-21T08:37:32.304 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:37:32.304 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 890 k 2024-04-21T08:37:32.304 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3154.ga630765c.el9 ceph 145 k 2024-04-21T08:37:32.304 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.0.0-3154.ga630765c.el9 ceph 1.5 M 2024-04-21T08:37:32.304 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 142 k 2024-04-21T08:37:32.304 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 3.5 M 2024-04-21T08:37:32.304 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 7.4 M 2024-04-21T08:37:32.304 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 49 k 2024-04-21T08:37:32.304 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 11 M 2024-04-21T08:37:32.305 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.0.0-3154.ga630765c.el9 ceph 49 M 2024-04-21T08:37:32.305 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 273 k 2024-04-21T08:37:32.305 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 762 k 2024-04-21T08:37:32.305 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 225 k 2024-04-21T08:37:32.305 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell x86_64 2:19.0.0-3154.ga630765c.el9 ceph 27 k 2024-04-21T08:37:32.305 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 18 k 2024-04-21T08:37:32.305 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T08:37:32.305 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T08:37:32.305 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T08:37:32.305 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 32 k 2024-04-21T08:37:32.305 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 746 k 2024-04-21T08:37:32.305 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 127 k 2024-04-21T08:37:32.305 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T08:37:32.305 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T08:37:32.306 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T08:37:32.306 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T08:37:32.306 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T08:37:32.306 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T08:37:32.306 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.0.0-3154.ga630765c.el9 ceph 162 k 2024-04-21T08:37:32.306 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.0.0-3154.ga630765c.el9 ceph 321 k 2024-04-21T08:37:32.306 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 298 k 2024-04-21T08:37:32.306 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 100 k 2024-04-21T08:37:32.306 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 85 k 2024-04-21T08:37:32.306 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:37:32.306 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 171 k 2024-04-21T08:37:32.306 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T08:37:32.306 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-04-21T08:37:32.307 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.4 M 2024-04-21T08:37:32.307 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:37:32.307 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-21T08:37:32.307 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T08:37:32.307 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T08:37:32.307 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T08:37:32.307 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 21 M 2024-04-21T08:37:32.307 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 34 k 2024-04-21T08:37:32.307 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.0.0-3154.ga630765c.el9 ceph 2.4 M 2024-04-21T08:37:32.307 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 249 k 2024-04-21T08:37:32.307 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.0.0-3154.ga630765c.el9 ceph 4.7 M 2024-04-21T08:37:32.307 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 17 M 2024-04-21T08:37:32.307 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 16 k 2024-04-21T08:37:32.307 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.0.0-3154.ga630765c.el9 ceph 25 k 2024-04-21T08:37:32.308 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T08:37:32.308 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T08:37:32.308 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T08:37:32.308 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T08:37:32.308 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T08:37:32.308 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T08:37:32.308 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T08:37:32.308 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T08:37:32.308 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.0.0-3154.ga630765c.el9 ceph 163 k 2024-04-21T08:37:32.308 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T08:37:32.308 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T08:37:32.308 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T08:37:32.308 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T08:37:32.309 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T08:37:32.309 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 501 k 2024-04-21T08:37:32.309 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T08:37:32.309 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:37:32.309 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T08:37:32.309 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T08:37:32.309 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T08:37:32.309 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T08:37:32.309 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T08:37:32.309 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T08:37:32.309 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T08:37:32.309 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T08:37:32.309 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T08:37:32.310 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T08:37:32.310 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T08:37:32.310 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T08:37:32.310 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T08:37:32.310 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T08:37:32.310 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T08:37:32.310 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T08:37:32.310 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T08:37:32.310 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 45 k 2024-04-21T08:37:32.310 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 135 k 2024-04-21T08:37:32.310 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T08:37:32.310 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T08:37:32.310 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T08:37:32.311 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T08:37:32.311 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T08:37:32.311 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T08:37:32.311 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T08:37:32.311 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T08:37:32.311 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T08:37:32.311 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T08:37:32.311 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T08:37:32.311 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T08:37:32.311 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T08:37:32.311 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T08:37:32.311 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T08:37:32.311 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T08:37:32.311 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T08:37:32.312 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T08:37:32.312 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T08:37:32.312 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T08:37:32.312 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T08:37:32.312 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T08:37:32.312 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T08:37:32.312 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T08:37:32.312 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T08:37:32.312 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T08:37:32.312 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T08:37:32.312 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T08:37:32.312 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T08:37:32.312 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T08:37:32.312 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T08:37:32.313 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T08:37:32.313 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T08:37:32.313 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T08:37:32.313 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T08:37:32.313 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T08:37:32.313 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T08:37:32.313 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T08:37:32.313 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T08:37:32.313 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T08:37:32.313 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T08:37:32.313 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T08:37:32.313 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T08:37:32.313 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T08:37:32.313 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T08:37:32.314 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T08:37:32.314 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T08:37:32.314 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T08:37:32.314 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T08:37:32.314 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T08:37:32.314 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T08:37:32.314 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T08:37:32.314 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T08:37:32.314 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T08:37:32.314 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T08:37:32.314 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T08:37:32.314 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T08:37:32.314 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T08:37:32.314 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T08:37:32.315 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T08:37:32.315 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T08:37:32.315 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T08:37:32.315 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T08:37:32.315 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-04-21T08:37:32.315 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T08:37:32.315 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:37:32.315 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-21T08:37:32.315 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-21T08:37:32.315 INFO:teuthology.orchestra.run.smithi028.stdout:Install 148 Packages 2024-04-21T08:37:32.315 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 2 Packages 2024-04-21T08:37:32.315 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:37:32.316 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 218 M 2024-04-21T08:37:32.316 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-21T08:37:32.538 INFO:teuthology.orchestra.run.smithi171.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-21T08:37:33.393 INFO:teuthology.orchestra.run.smithi028.stdout:(1/150): ceph-19.0.0-3154.ga630765c.el9.x86_64. 27 kB/s | 6.6 kB 00:00 2024-04-21T08:37:33.693 INFO:teuthology.orchestra.run.smithi028.stdout:(2/150): ceph-base-19.0.0-3154.ga630765c.el9.x8 9.8 MB/s | 5.4 MB 00:00 2024-04-21T08:37:33.743 INFO:teuthology.orchestra.run.smithi028.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 2.8 MB/s | 145 kB 00:00 2024-04-21T08:37:33.902 INFO:teuthology.orchestra.run.smithi028.stdout:(4/150): ceph-mds-19.0.0-3154.ga630765c.el9.x86 15 MB/s | 2.4 MB 00:00 2024-04-21T08:37:34.027 INFO:teuthology.orchestra.run.smithi028.stdout:(5/150): ceph-mgr-19.0.0-3154.ga630765c.el9.x86 12 MB/s | 1.5 MB 00:00 2024-04-21T08:37:34.052 INFO:teuthology.orchestra.run.smithi028.stdout:(6/150): ceph-fuse-19.0.0-3154.ga630765c.el9.x8 1.3 MB/s | 890 kB 00:00 2024-04-21T08:37:34.335 INFO:teuthology.orchestra.run.smithi028.stdout:(7/150): ceph-mon-19.0.0-3154.ga630765c.el9.x86 15 MB/s | 4.7 MB 00:00 2024-04-21T08:37:35.028 INFO:teuthology.orchestra.run.smithi028.stdout:(8/150): ceph-radosgw-19.0.0-3154.ga630765c.el9 15 MB/s | 11 MB 00:00 2024-04-21T08:37:35.086 INFO:teuthology.orchestra.run.smithi028.stdout:(9/150): ceph-selinux-19.0.0-3154.ga630765c.el9 434 kB/s | 25 kB 00:00 2024-04-21T08:37:35.945 INFO:teuthology.orchestra.run.smithi028.stdout:(10/150): ceph-common-19.0.0-3154.ga630765c.el9 7.6 MB/s | 21 MB 00:02 2024-04-21T08:37:36.037 INFO:teuthology.orchestra.run.smithi028.stdout:(11/150): cephfs-mirror-19.0.0-3154.ga630765c.e 2.4 MB/s | 225 kB 00:00 2024-04-21T08:37:36.103 INFO:teuthology.orchestra.run.smithi028.stdout:(12/150): cephfs-shell-19.0.0-3154.ga630765c.el 408 kB/s | 27 kB 00:00 2024-04-21T08:37:36.178 INFO:teuthology.orchestra.run.smithi028.stdout:(13/150): libcephfs-devel-19.0.0-3154.ga630765c 423 kB/s | 32 kB 00:00 2024-04-21T08:37:36.287 INFO:teuthology.orchestra.run.smithi028.stdout:(14/150): libcephfs2-19.0.0-3154.ga630765c.el9. 6.7 MB/s | 746 kB 00:00 2024-04-21T08:37:36.379 INFO:teuthology.orchestra.run.smithi028.stdout:(15/150): libcephsqlite-19.0.0-3154.ga630765c.e 1.7 MB/s | 163 kB 00:00 2024-04-21T08:37:36.471 INFO:teuthology.orchestra.run.smithi028.stdout:(16/150): librados-devel-19.0.0-3154.ga630765c. 1.4 MB/s | 127 kB 00:00 2024-04-21T08:37:36.580 INFO:teuthology.orchestra.run.smithi028.stdout:(17/150): libradosstriper1-19.0.0-3154.ga630765 4.5 MB/s | 501 kB 00:00 2024-04-21T08:37:37.347 INFO:teuthology.orchestra.run.smithi028.stdout:(18/150): librgw2-19.0.0-3154.ga630765c.el9.x86 7.0 MB/s | 5.4 MB 00:00 2024-04-21T08:37:37.505 INFO:teuthology.orchestra.run.smithi028.stdout:(19/150): ceph-osd-19.0.0-3154.ga630765c.el9.x8 4.9 MB/s | 17 MB 00:03 2024-04-21T08:37:37.591 INFO:teuthology.orchestra.run.smithi028.stdout:(20/150): python3-ceph-argparse-19.0.0-3154.ga6 185 kB/s | 45 kB 00:00 2024-04-21T08:37:37.631 INFO:teuthology.orchestra.run.smithi028.stdout:(21/150): python3-ceph-common-19.0.0-3154.ga630 1.1 MB/s | 135 kB 00:00 2024-04-21T08:37:37.656 INFO:teuthology.orchestra.run.smithi028.stdout:(22/150): python3-cephfs-19.0.0-3154.ga630765c. 2.4 MB/s | 162 kB 00:00 2024-04-21T08:37:37.671 INFO:teuthology.orchestra.run.smithi171.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-21T08:37:37.698 INFO:teuthology.orchestra.run.smithi028.stdout:(23/150): python3-rados-19.0.0-3154.ga630765c.e 4.7 MB/s | 321 kB 00:00 2024-04-21T08:37:37.732 INFO:teuthology.orchestra.run.smithi028.stdout:(24/150): python3-rbd-19.0.0-3154.ga630765c.el9 3.9 MB/s | 298 kB 00:00 2024-04-21T08:37:37.782 INFO:teuthology.orchestra.run.smithi028.stdout:(25/150): python3-rgw-19.0.0-3154.ga630765c.el9 1.2 MB/s | 100 kB 00:00 2024-04-21T08:37:37.808 INFO:teuthology.orchestra.run.smithi028.stdout:(26/150): rbd-fuse-19.0.0-3154.ga630765c.el9.x8 1.1 MB/s | 85 kB 00:00 2024-04-21T08:37:37.908 INFO:teuthology.orchestra.run.smithi028.stdout:(27/150): rbd-nbd-19.0.0-3154.ga630765c.el9.x86 1.7 MB/s | 171 kB 00:00 2024-04-21T08:37:38.000 INFO:teuthology.orchestra.run.smithi028.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3154.g 374 kB/s | 34 kB 00:00 2024-04-21T08:37:38.075 INFO:teuthology.orchestra.run.smithi028.stdout:(29/150): rbd-mirror-19.0.0-3154.ga630765c.el9. 11 MB/s | 3.1 MB 00:00 2024-04-21T08:37:38.092 INFO:teuthology.orchestra.run.smithi028.stdout:(30/150): ceph-mgr-cephadm-19.0.0-3154.ga630765 1.5 MB/s | 142 kB 00:00 2024-04-21T08:37:38.426 INFO:teuthology.orchestra.run.smithi028.stdout:(31/150): ceph-mgr-dashboard-19.0.0-3154.ga6307 10 MB/s | 3.5 MB 00:00 2024-04-21T08:37:38.509 INFO:teuthology.orchestra.run.smithi028.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3154.ga6 2.9 MB/s | 249 kB 00:00 2024-04-21T08:37:38.944 INFO:teuthology.orchestra.run.smithi028.stdout:(33/150): ceph-test-19.0.0-3154.ga630765c.el9.x 13 MB/s | 49 MB 00:03 2024-04-21T08:37:38.969 INFO:teuthology.orchestra.run.smithi028.stdout:(34/150): ceph-mgr-rook-19.0.0-3154.ga630765c.e 107 kB/s | 49 kB 00:00 2024-04-21T08:37:38.994 INFO:teuthology.orchestra.run.smithi028.stdout:(35/150): ceph-prometheus-alerts-19.0.0-3154.ga 330 kB/s | 16 kB 00:00 2024-04-21T08:37:39.036 INFO:teuthology.orchestra.run.smithi028.stdout:(36/150): ceph-volume-19.0.0-3154.ga630765c.el9 4.0 MB/s | 273 kB 00:00 2024-04-21T08:37:39.136 INFO:teuthology.orchestra.run.smithi028.stdout:(37/150): ceph-mgr-diskprediction-local-19.0.0- 7.1 MB/s | 7.4 MB 00:01 2024-04-21T08:37:39.162 INFO:teuthology.orchestra.run.smithi028.stdout:(38/150): cephfs-top-19.0.0-3154.ga630765c.el9. 145 kB/s | 18 kB 00:00 2024-04-21T08:37:39.203 INFO:teuthology.orchestra.run.smithi028.stdout:(39/150): cephadm-19.0.0-3154.ga630765c.el9.noa 3.6 MB/s | 762 kB 00:00 2024-04-21T08:37:39.313 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 896 kB/s | 19 MB 00:21 2024-04-21T08:37:39.832 INFO:teuthology.orchestra.run.smithi171.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:37:39.832 INFO:teuthology.orchestra.run.smithi171.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:37:39.843 INFO:teuthology.orchestra.run.smithi171.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T08:37:39.844 INFO:teuthology.orchestra.run.smithi171.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T08:37:39.846 INFO:teuthology.orchestra.run.smithi171.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T08:37:39.846 INFO:teuthology.orchestra.run.smithi171.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T08:37:39.848 INFO:teuthology.orchestra.run.smithi171.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T08:37:39.850 INFO:teuthology.orchestra.run.smithi171.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T08:37:39.851 INFO:teuthology.orchestra.run.smithi171.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T08:37:39.895 INFO:teuthology.orchestra.run.smithi028.stdout:(40/150): fuse-2.9.9-15.el9.x86_64.rpm 106 kB/s | 80 kB 00:00 2024-04-21T08:37:39.911 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-04-21T08:37:39.937 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================= 2024-04-21T08:37:39.937 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-04-21T08:37:39.937 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================= 2024-04-21T08:37:39.937 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-04-21T08:37:39.937 INFO:teuthology.orchestra.run.smithi171.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T08:37:39.937 INFO:teuthology.orchestra.run.smithi171.stdout: ceph x86_64 2:19.0.0-3154.ga630765c.el9 ceph 6.6 k 2024-04-21T08:37:39.937 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:37:39.937 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 890 k 2024-04-21T08:37:39.937 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3154.ga630765c.el9 ceph 145 k 2024-04-21T08:37:39.937 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr x86_64 2:19.0.0-3154.ga630765c.el9 ceph 1.5 M 2024-04-21T08:37:39.937 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 142 k 2024-04-21T08:37:39.937 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 3.5 M 2024-04-21T08:37:39.938 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 7.4 M 2024-04-21T08:37:39.938 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 49 k 2024-04-21T08:37:39.938 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 11 M 2024-04-21T08:37:39.938 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test x86_64 2:19.0.0-3154.ga630765c.el9 ceph 49 M 2024-04-21T08:37:39.938 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-volume noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 273 k 2024-04-21T08:37:39.938 INFO:teuthology.orchestra.run.smithi171.stdout: cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 762 k 2024-04-21T08:37:39.938 INFO:teuthology.orchestra.run.smithi171.stdout: cephfs-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 225 k 2024-04-21T08:37:39.938 INFO:teuthology.orchestra.run.smithi171.stdout: cephfs-shell x86_64 2:19.0.0-3154.ga630765c.el9 ceph 27 k 2024-04-21T08:37:39.938 INFO:teuthology.orchestra.run.smithi171.stdout: cephfs-top noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 18 k 2024-04-21T08:37:39.938 INFO:teuthology.orchestra.run.smithi171.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T08:37:39.938 INFO:teuthology.orchestra.run.smithi171.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T08:37:39.938 INFO:teuthology.orchestra.run.smithi171.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T08:37:39.938 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 32 k 2024-04-21T08:37:39.939 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 746 k 2024-04-21T08:37:39.939 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 127 k 2024-04-21T08:37:39.939 INFO:teuthology.orchestra.run.smithi171.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T08:37:39.939 INFO:teuthology.orchestra.run.smithi171.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T08:37:39.939 INFO:teuthology.orchestra.run.smithi171.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T08:37:39.939 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T08:37:39.939 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T08:37:39.939 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T08:37:39.939 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs x86_64 2:19.0.0-3154.ga630765c.el9 ceph 162 k 2024-04-21T08:37:39.939 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados x86_64 2:19.0.0-3154.ga630765c.el9 ceph 321 k 2024-04-21T08:37:39.939 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 298 k 2024-04-21T08:37:39.939 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 100 k 2024-04-21T08:37:39.940 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 85 k 2024-04-21T08:37:39.940 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:37:39.940 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 171 k 2024-04-21T08:37:39.940 INFO:teuthology.orchestra.run.smithi171.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T08:37:39.940 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrading: 2024-04-21T08:37:39.940 INFO:teuthology.orchestra.run.smithi171.stdout: librados2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.4 M 2024-04-21T08:37:39.940 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:37:39.940 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-04-21T08:37:39.940 INFO:teuthology.orchestra.run.smithi171.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T08:37:39.940 INFO:teuthology.orchestra.run.smithi171.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T08:37:39.940 INFO:teuthology.orchestra.run.smithi171.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T08:37:39.940 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 21 M 2024-04-21T08:37:39.940 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 34 k 2024-04-21T08:37:39.941 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds x86_64 2:19.0.0-3154.ga630765c.el9 ceph 2.4 M 2024-04-21T08:37:39.941 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 249 k 2024-04-21T08:37:39.941 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon x86_64 2:19.0.0-3154.ga630765c.el9 ceph 4.7 M 2024-04-21T08:37:39.941 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 17 M 2024-04-21T08:37:39.941 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 16 k 2024-04-21T08:37:39.941 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux x86_64 2:19.0.0-3154.ga630765c.el9 ceph 25 k 2024-04-21T08:37:39.941 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T08:37:39.941 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T08:37:39.941 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T08:37:39.941 INFO:teuthology.orchestra.run.smithi171.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T08:37:39.941 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T08:37:39.941 INFO:teuthology.orchestra.run.smithi171.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T08:37:39.941 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T08:37:39.941 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T08:37:39.942 INFO:teuthology.orchestra.run.smithi171.stdout: libcephsqlite x86_64 2:19.0.0-3154.ga630765c.el9 ceph 163 k 2024-04-21T08:37:39.942 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T08:37:39.942 INFO:teuthology.orchestra.run.smithi171.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T08:37:39.942 INFO:teuthology.orchestra.run.smithi171.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T08:37:39.942 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T08:37:39.942 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T08:37:39.942 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 501 k 2024-04-21T08:37:39.942 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T08:37:39.942 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:37:39.942 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T08:37:39.942 INFO:teuthology.orchestra.run.smithi171.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T08:37:39.942 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T08:37:39.942 INFO:teuthology.orchestra.run.smithi171.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T08:37:39.942 INFO:teuthology.orchestra.run.smithi171.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T08:37:39.943 INFO:teuthology.orchestra.run.smithi171.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T08:37:39.943 INFO:teuthology.orchestra.run.smithi171.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T08:37:39.943 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T08:37:39.943 INFO:teuthology.orchestra.run.smithi171.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T08:37:39.943 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T08:37:39.943 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T08:37:39.943 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T08:37:39.943 INFO:teuthology.orchestra.run.smithi171.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T08:37:39.943 INFO:teuthology.orchestra.run.smithi171.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T08:37:39.943 INFO:teuthology.orchestra.run.smithi171.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T08:37:39.943 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T08:37:39.943 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T08:37:39.943 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 45 k 2024-04-21T08:37:39.944 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 135 k 2024-04-21T08:37:39.944 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T08:37:39.944 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T08:37:39.944 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T08:37:39.944 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T08:37:39.944 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T08:37:39.944 INFO:teuthology.orchestra.run.smithi171.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T08:37:39.944 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T08:37:39.944 INFO:teuthology.orchestra.run.smithi171.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T08:37:39.945 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T08:37:39.945 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T08:37:39.945 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T08:37:39.945 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T08:37:39.945 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T08:37:39.945 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T08:37:39.945 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T08:37:39.945 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T08:37:39.945 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T08:37:39.945 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T08:37:39.945 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T08:37:39.945 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T08:37:39.945 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T08:37:39.945 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T08:37:39.946 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T08:37:39.946 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T08:37:39.946 INFO:teuthology.orchestra.run.smithi171.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T08:37:39.946 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T08:37:39.946 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T08:37:39.946 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T08:37:39.946 INFO:teuthology.orchestra.run.smithi171.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T08:37:39.946 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T08:37:39.946 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T08:37:39.946 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T08:37:39.946 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T08:37:39.946 INFO:teuthology.orchestra.run.smithi171.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T08:37:39.946 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T08:37:39.947 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T08:37:39.947 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T08:37:39.947 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T08:37:39.947 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T08:37:39.947 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T08:37:39.947 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T08:37:39.947 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T08:37:39.947 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T08:37:39.947 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T08:37:39.947 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T08:37:39.947 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T08:37:39.947 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T08:37:39.947 INFO:teuthology.orchestra.run.smithi171.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T08:37:39.947 INFO:teuthology.orchestra.run.smithi171.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T08:37:39.948 INFO:teuthology.orchestra.run.smithi171.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T08:37:39.948 INFO:teuthology.orchestra.run.smithi171.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T08:37:39.948 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T08:37:39.948 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T08:37:39.948 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T08:37:39.948 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T08:37:39.948 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T08:37:39.948 INFO:teuthology.orchestra.run.smithi171.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T08:37:39.948 INFO:teuthology.orchestra.run.smithi171.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T08:37:39.948 INFO:teuthology.orchestra.run.smithi171.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T08:37:39.948 INFO:teuthology.orchestra.run.smithi171.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T08:37:39.948 INFO:teuthology.orchestra.run.smithi171.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T08:37:39.948 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T08:37:39.949 INFO:teuthology.orchestra.run.smithi171.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T08:37:39.949 INFO:teuthology.orchestra.run.smithi171.stdout:Installing weak dependencies: 2024-04-21T08:37:39.949 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T08:37:39.949 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:37:39.949 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-04-21T08:37:39.949 INFO:teuthology.orchestra.run.smithi171.stdout:======================================================================================= 2024-04-21T08:37:39.949 INFO:teuthology.orchestra.run.smithi171.stdout:Install 148 Packages 2024-04-21T08:37:39.949 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrade 2 Packages 2024-04-21T08:37:39.949 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:37:39.950 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 218 M 2024-04-21T08:37:39.950 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-04-21T08:37:40.062 INFO:teuthology.orchestra.run.smithi028.stdout:(41/150): libquadmath-11.4.1-3.el9.x86_64.rpm 227 kB/s | 195 kB 00:00 2024-04-21T08:37:40.779 INFO:teuthology.orchestra.run.smithi028.stdout:(42/150): libgfortran-11.4.1-3.el9.x86_64.rpm 497 kB/s | 803 kB 00:01 2024-04-21T08:37:40.854 INFO:teuthology.orchestra.run.smithi028.stdout:(43/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 264 kB/s | 253 kB 00:00 2024-04-21T08:37:41.298 INFO:teuthology.orchestra.run.smithi171.stdout:(1/150): ceph-19.0.0-3154.ga630765c.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-04-21T08:37:41.506 INFO:teuthology.orchestra.run.smithi171.stdout:(2/150): ceph-fuse-19.0.0-3154.ga630765c.el9.x8 4.2 MB/s | 890 kB 00:00 2024-04-21T08:37:41.546 INFO:teuthology.orchestra.run.smithi028.stdout:(44/150): python3-ply-3.11-14.el9.noarch.rpm 139 kB/s | 106 kB 00:00 2024-04-21T08:37:41.556 INFO:teuthology.orchestra.run.smithi171.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 2.8 MB/s | 145 kB 00:00 2024-04-21T08:37:41.655 INFO:teuthology.orchestra.run.smithi028.stdout:(45/150): python3-pycparser-2.20-6.el9.noarch.r 169 kB/s | 135 kB 00:00 2024-04-21T08:37:41.931 INFO:teuthology.orchestra.run.smithi171.stdout:(4/150): ceph-mds-19.0.0-3154.ga630765c.el9.x86 6.3 MB/s | 2.4 MB 00:00 2024-04-21T08:37:42.065 INFO:teuthology.orchestra.run.smithi171.stdout:(5/150): ceph-base-19.0.0-3154.ga630765c.el9.x8 4.7 MB/s | 5.4 MB 00:01 2024-04-21T08:37:42.222 INFO:teuthology.orchestra.run.smithi028.stdout:(46/150): python3-cryptography-36.0.1-4.el9.x86 581 kB/s | 1.2 MB 00:02 2024-04-21T08:37:42.305 INFO:teuthology.orchestra.run.smithi028.stdout:(47/150): python3-requests-2.25.1-8.el9.noarch. 165 kB/s | 125 kB 00:00 2024-04-21T08:37:42.389 INFO:teuthology.orchestra.run.smithi028.stdout:(48/150): automake-1.16.2-8.el9.noarch.rpm 4.0 MB/s | 689 kB 00:00 2024-04-21T08:37:42.456 INFO:teuthology.orchestra.run.smithi028.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-21T08:37:42.498 INFO:teuthology.orchestra.run.smithi028.stdout:(50/150): bison-3.7.4-5.el9.x86_64.rpm 4.8 MB/s | 943 kB 00:00 2024-04-21T08:37:42.523 INFO:teuthology.orchestra.run.smithi028.stdout:(51/150): elfutils-libelf-devel-0.190-2.el9.x86 321 kB/s | 21 kB 00:00 2024-04-21T08:37:42.548 INFO:teuthology.orchestra.run.smithi028.stdout:(52/150): python3-urllib3-1.26.5-5.el9.noarch.r 240 kB/s | 215 kB 00:00 2024-04-21T08:37:42.599 INFO:teuthology.orchestra.run.smithi028.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 434 kB/s | 33 kB 00:00 2024-04-21T08:37:42.624 INFO:teuthology.orchestra.run.smithi028.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 2.4 MB/s | 313 kB 00:00 2024-04-21T08:37:42.699 INFO:teuthology.orchestra.run.smithi028.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 149 kB/s | 11 kB 00:00 2024-04-21T08:37:42.715 INFO:teuthology.orchestra.run.smithi171.stdout:(6/150): ceph-mon-19.0.0-3154.ga630765c.el9.x86 7.2 MB/s | 4.7 MB 00:00 2024-04-21T08:37:42.724 INFO:teuthology.orchestra.run.smithi028.stdout:(56/150): flexiblas-openblas-openmp-3.0.4-8.el9 145 kB/s | 18 kB 00:00 2024-04-21T08:37:42.757 INFO:teuthology.orchestra.run.smithi171.stdout:(7/150): ceph-mgr-19.0.0-3154.ga630765c.el9.x86 1.8 MB/s | 1.5 MB 00:00 2024-04-21T08:37:42.800 INFO:teuthology.orchestra.run.smithi028.stdout:(57/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 602 kB/s | 45 kB 00:00 2024-04-21T08:37:42.908 INFO:teuthology.orchestra.run.smithi028.stdout:(58/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.3 MB/s | 3.0 MB 00:00 2024-04-21T08:37:42.934 INFO:teuthology.orchestra.run.smithi028.stdout:(59/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 684 kB/s | 160 kB 00:00 2024-04-21T08:37:42.991 INFO:teuthology.orchestra.run.smithi171.stdout:(8/150): ceph-common-19.0.0-3154.ga630765c.el9. 10 MB/s | 21 MB 00:02 2024-04-21T08:37:43.000 INFO:teuthology.orchestra.run.smithi028.stdout:(60/150): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 662 kB 00:00 2024-04-21T08:37:43.026 INFO:teuthology.orchestra.run.smithi028.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 277 kB/s | 25 kB 00:00 2024-04-21T08:37:43.074 INFO:teuthology.orchestra.run.smithi171.stdout:(9/150): ceph-selinux-19.0.0-3154.ga630765c.el9 303 kB/s | 25 kB 00:00 2024-04-21T08:37:43.076 INFO:teuthology.orchestra.run.smithi028.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 631 kB/s | 47 kB 00:00 2024-04-21T08:37:43.110 INFO:teuthology.orchestra.run.smithi028.stdout:(63/150): libtool-2.4.6-45.el9.x86_64.rpm 2.8 MB/s | 582 kB 00:00 2024-04-21T08:37:43.143 INFO:teuthology.orchestra.run.smithi028.stdout:(64/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-04-21T08:37:43.169 INFO:teuthology.orchestra.run.smithi028.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 600 kB/s | 35 kB 00:00 2024-04-21T08:37:43.194 INFO:teuthology.orchestra.run.smithi028.stdout:(66/150): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-04-21T08:37:43.469 INFO:teuthology.orchestra.run.smithi028.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 5.8 MB/s | 1.6 MB 00:00 2024-04-21T08:37:43.569 INFO:teuthology.orchestra.run.smithi028.stdout:(68/150): openblas-openmp-0.3.21-2.el9.x86_64.r 11 MB/s | 4.7 MB 00:00 2024-04-21T08:37:43.636 INFO:teuthology.orchestra.run.smithi028.stdout:(69/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 8.8 MB/s | 4.1 MB 00:00 2024-04-21T08:37:43.678 INFO:teuthology.orchestra.run.smithi028.stdout:(70/150): postgresql-contrib-13.14-1.el9.x86_64 4.3 MB/s | 911 kB 00:00 2024-04-21T08:37:43.720 INFO:teuthology.orchestra.run.smithi028.stdout:(71/150): postgresql-private-libs-13.14-1.el9.x 886 kB/s | 133 kB 00:00 2024-04-21T08:37:43.779 INFO:teuthology.orchestra.run.smithi028.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-21T08:37:43.862 INFO:teuthology.orchestra.run.smithi028.stdout:(73/150): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-04-21T08:37:43.971 INFO:teuthology.orchestra.run.smithi028.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-04-21T08:37:44.063 INFO:teuthology.orchestra.run.smithi028.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-04-21T08:37:44.105 INFO:teuthology.orchestra.run.smithi028.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 617 kB/s | 26 kB 00:00 2024-04-21T08:37:44.197 INFO:teuthology.orchestra.run.smithi028.stdout:(77/150): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-04-21T08:37:44.214 INFO:teuthology.orchestra.run.smithi028.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 176 kB/s | 19 kB 00:00 2024-04-21T08:37:44.289 INFO:teuthology.orchestra.run.smithi028.stdout:(79/150): postgresql-server-13.14-1.el9.x86_64. 8.9 MB/s | 5.8 MB 00:00 2024-04-21T08:37:44.331 INFO:teuthology.orchestra.run.smithi028.stdout:(80/150): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-04-21T08:37:44.356 INFO:teuthology.orchestra.run.smithi028.stdout:(81/150): python3-markupsafe-1.1.1-12.el9.x86_6 244 kB/s | 35 kB 00:00 2024-04-21T08:37:44.448 INFO:teuthology.orchestra.run.smithi028.stdout:(82/150): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-04-21T08:37:44.473 INFO:teuthology.orchestra.run.smithi028.stdout:(83/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-04-21T08:37:44.582 INFO:teuthology.orchestra.run.smithi028.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 577 kB/s | 77 kB 00:00 2024-04-21T08:37:44.607 INFO:teuthology.orchestra.run.smithi028.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 316 kB/s | 42 kB 00:00 2024-04-21T08:37:44.699 INFO:teuthology.orchestra.run.smithi028.stdout:(86/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-04-21T08:37:44.725 INFO:teuthology.orchestra.run.smithi028.stdout:(87/150): python3-protobuf-3.14.0-13.el9.noarch 1.8 MB/s | 269 kB 00:00 2024-04-21T08:37:44.792 INFO:teuthology.orchestra.run.smithi028.stdout:(88/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 5.1 MB 00:00 2024-04-21T08:37:44.842 INFO:teuthology.orchestra.run.smithi028.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 436 kB/s | 51 kB 00:00 2024-04-21T08:37:44.867 INFO:teuthology.orchestra.run.smithi028.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 715 kB/s | 54 kB 00:00 2024-04-21T08:37:44.892 INFO:teuthology.orchestra.run.smithi028.stdout:(91/150): python3-pyasn1-modules-0.4.8-6.el9.no 1.4 MB/s | 279 kB 00:00 2024-04-21T08:37:44.943 INFO:teuthology.orchestra.run.smithi028.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-04-21T08:37:44.966 INFO:teuthology.orchestra.run.smithi171.stdout:(10/150): ceph-radosgw-19.0.0-3154.ga630765c.el 4.8 MB/s | 11 MB 00:02 2024-04-21T08:37:44.968 INFO:teuthology.orchestra.run.smithi028.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 588 kB/s | 44 kB 00:00 2024-04-21T08:37:45.018 INFO:teuthology.orchestra.run.smithi028.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 720 kB/s | 36 kB 00:00 2024-04-21T08:37:45.060 INFO:teuthology.orchestra.run.smithi028.stdout:(95/150): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-04-21T08:37:45.085 INFO:teuthology.orchestra.run.smithi028.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 708 kB/s | 47 kB 00:00 2024-04-21T08:37:45.152 INFO:teuthology.orchestra.run.smithi028.stdout:(97/150): uuid-1.6.2-55.el9.x86_64.rpm 866 kB/s | 58 kB 00:00 2024-04-21T08:37:45.184 INFO:teuthology.orchestra.run.smithi171.stdout:(11/150): ceph-osd-19.0.0-3154.ga630765c.el9.x8 6.8 MB/s | 17 MB 00:02 2024-04-21T08:37:45.194 INFO:teuthology.orchestra.run.smithi028.stdout:(98/150): socat-1.7.4.1-5.el9.x86_64.rpm 2.2 MB/s | 305 kB 00:00 2024-04-21T08:37:45.209 INFO:teuthology.orchestra.run.smithi171.stdout:(12/150): cephfs-mirror-19.0.0-3154.ga630765c.e 930 kB/s | 225 kB 00:00 2024-04-21T08:37:45.236 INFO:teuthology.orchestra.run.smithi028.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 94 kB 00:00 2024-04-21T08:37:45.251 INFO:teuthology.orchestra.run.smithi171.stdout:(13/150): cephfs-shell-19.0.0-3154.ga630765c.el 406 kB/s | 27 kB 00:00 2024-04-21T08:37:45.277 INFO:teuthology.orchestra.run.smithi171.stdout:(14/150): libcephfs-devel-19.0.0-3154.ga630765c 473 kB/s | 32 kB 00:00 2024-04-21T08:37:45.303 INFO:teuthology.orchestra.run.smithi028.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 587 kB/s | 64 kB 00:00 2024-04-21T08:37:45.329 INFO:teuthology.orchestra.run.smithi028.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 487 kB/s | 45 kB 00:00 2024-04-21T08:37:45.368 INFO:teuthology.orchestra.run.smithi171.stdout:(15/150): libcephsqlite-19.0.0-3154.ga630765c.e 1.7 MB/s | 163 kB 00:00 2024-04-21T08:37:45.418 INFO:teuthology.orchestra.run.smithi171.stdout:(16/150): libcephfs2-19.0.0-3154.ga630765c.el9. 4.4 MB/s | 746 kB 00:00 2024-04-21T08:37:45.444 INFO:teuthology.orchestra.run.smithi171.stdout:(17/150): librados-devel-19.0.0-3154.ga630765c. 1.6 MB/s | 127 kB 00:00 2024-04-21T08:37:45.504 INFO:teuthology.orchestra.run.smithi028.stdout:(102/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-04-21T08:37:45.519 INFO:teuthology.orchestra.run.smithi171.stdout:(18/150): libradosstriper1-19.0.0-3154.ga630765 4.9 MB/s | 501 kB 00:00 2024-04-21T08:37:45.571 INFO:teuthology.orchestra.run.smithi028.stdout:(103/150): lua-devel-5.4.4-4.el9.x86_64.rpm 83 kB/s | 22 kB 00:00 2024-04-21T08:37:45.627 INFO:teuthology.orchestra.run.smithi171.stdout:(19/150): python3-ceph-argparse-19.0.0-3154.ga6 417 kB/s | 45 kB 00:00 2024-04-21T08:37:45.639 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.2 MB 00:02 2024-04-21T08:37:45.713 INFO:teuthology.orchestra.run.smithi028.stdout:(104/150): protobuf-compiler-3.14.0-13.el9.x86_ 2.2 MB/s | 863 kB 00:00 2024-04-21T08:37:45.736 INFO:teuthology.orchestra.run.smithi171.stdout:(20/150): python3-ceph-common-19.0.0-3154.ga630 1.2 MB/s | 135 kB 00:00 2024-04-21T08:37:45.971 INFO:teuthology.orchestra.run.smithi028.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-04-21T08:37:46.163 INFO:teuthology.orchestra.run.smithi028.stdout:(106/150): abseil-cpp-20211102.0-3.el9.x86_64.r 832 kB/s | 548 kB 00:00 2024-04-21T08:37:46.188 INFO:teuthology.orchestra.run.smithi028.stdout:(107/150): gperftools-libs-2.9.1-2.el9.x86_64.r 500 kB/s | 309 kB 00:00 2024-04-21T08:37:46.211 INFO:teuthology.orchestra.run.smithi171.stdout:(21/150): ceph-test-19.0.0-3154.ga630765c.el9.x 16 MB/s | 49 MB 00:03 2024-04-21T08:37:46.237 INFO:teuthology.orchestra.run.smithi171.stdout:(22/150): python3-cephfs-19.0.0-3154.ga630765c. 323 kB/s | 162 kB 00:00 2024-04-21T08:37:46.272 INFO:teuthology.orchestra.run.smithi028.stdout:(108/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-04-21T08:37:46.297 INFO:teuthology.orchestra.run.smithi028.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 447 kB/s | 49 kB 00:00 2024-04-21T08:37:46.337 INFO:teuthology.orchestra.run.smithi171.stdout:(23/150): librgw2-19.0.0-3154.ga630765c.el9.x86 6.0 MB/s | 5.4 MB 00:00 2024-04-21T08:37:46.362 INFO:teuthology.orchestra.run.smithi171.stdout:(24/150): python3-rados-19.0.0-3154.ga630765c.e 2.1 MB/s | 321 kB 00:00 2024-04-21T08:37:46.381 INFO:teuthology.orchestra.run.smithi028.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2024-04-21T08:37:46.387 INFO:teuthology.orchestra.run.smithi171.stdout:(25/150): python3-rgw-19.0.0-3154.ga630765c.el9 1.9 MB/s | 100 kB 00:00 2024-04-21T08:37:46.412 INFO:teuthology.orchestra.run.smithi171.stdout:(26/150): python3-rbd-19.0.0-3154.ga630765c.el9 1.7 MB/s | 298 kB 00:00 2024-04-21T08:37:46.431 INFO:teuthology.orchestra.run.smithi028.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-04-21T08:37:46.438 INFO:teuthology.orchestra.run.smithi171.stdout:(27/150): rbd-fuse-19.0.0-3154.ga630765c.el9.x8 1.1 MB/s | 85 kB 00:00 2024-04-21T08:37:46.480 INFO:teuthology.orchestra.run.smithi171.stdout:(28/150): rbd-nbd-19.0.0-3154.ga630765c.el9.x86 2.5 MB/s | 171 kB 00:00 2024-04-21T08:37:46.497 INFO:teuthology.orchestra.run.smithi171.stdout:(29/150): ceph-grafana-dashboards-19.0.0-3154.g 583 kB/s | 34 kB 00:00 2024-04-21T08:37:46.547 INFO:teuthology.orchestra.run.smithi171.stdout:(30/150): ceph-mgr-cephadm-19.0.0-3154.ga630765 2.1 MB/s | 142 kB 00:00 2024-04-21T08:37:46.581 INFO:teuthology.orchestra.run.smithi028.stdout:(112/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-04-21T08:37:46.640 INFO:teuthology.orchestra.run.smithi028.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 2.6 MB/s | 548 kB 00:00 2024-04-21T08:37:46.682 INFO:teuthology.orchestra.run.smithi028.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 432 kB/s | 43 kB 00:00 2024-04-21T08:37:46.749 INFO:teuthology.orchestra.run.smithi028.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 296 kB/s | 32 kB 00:00 2024-04-21T08:37:46.756 INFO:teuthology.orchestra.run.smithi171.stdout:(31/150): rbd-mirror-19.0.0-3154.ga630765c.el9. 8.4 MB/s | 3.1 MB 00:00 2024-04-21T08:37:46.816 INFO:teuthology.orchestra.run.smithi028.stdout:(116/150): libarrow-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-04-21T08:37:46.831 INFO:teuthology.orchestra.run.smithi171.stdout:(32/150): ceph-mgr-dashboard-19.0.0-3154.ga6307 11 MB/s | 3.5 MB 00:00 2024-04-21T08:37:46.833 INFO:teuthology.orchestra.run.smithi028.stdout:(117/150): python3-certifi-2021.10.8-2.el9.noar 99 kB/s | 15 kB 00:00 2024-04-21T08:37:46.856 INFO:teuthology.orchestra.run.smithi171.stdout:(33/150): ceph-mgr-modules-core-19.0.0-3154.ga6 2.4 MB/s | 249 kB 00:00 2024-04-21T08:37:46.867 INFO:teuthology.orchestra.run.smithi028.stdout:(118/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-04-21T08:37:46.881 INFO:teuthology.orchestra.run.smithi171.stdout:(34/150): ceph-mgr-rook-19.0.0-3154.ga630765c.e 983 kB/s | 49 kB 00:00 2024-04-21T08:37:46.898 INFO:teuthology.orchestra.run.smithi171.stdout:(35/150): ceph-prometheus-alerts-19.0.0-3154.ga 395 kB/s | 16 kB 00:00 2024-04-21T08:37:46.934 INFO:teuthology.orchestra.run.smithi028.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-04-21T08:37:46.949 INFO:teuthology.orchestra.run.smithi171.stdout:(36/150): ceph-volume-19.0.0-3154.ga630765c.el9 4.0 MB/s | 273 kB 00:00 2024-04-21T08:37:46.959 INFO:teuthology.orchestra.run.smithi028.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-04-21T08:37:46.984 INFO:teuthology.orchestra.run.smithi028.stdout:(121/150): python3-colorama-0.4.4-12.el9.noarch 301 kB/s | 35 kB 00:00 2024-04-21T08:37:47.032 INFO:teuthology.orchestra.run.smithi171.stdout:(37/150): cephadm-19.0.0-3154.ga630765c.el9.noa 5.6 MB/s | 762 kB 00:00 2024-04-21T08:37:47.043 INFO:teuthology.orchestra.run.smithi028.stdout:(122/150): python3-google-auth-2.28.2-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-04-21T08:37:47.048 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 16 kB 00:00 2024-04-21T08:37:47.058 INFO:teuthology.orchestra.run.smithi171.stdout:(38/150): cephfs-top-19.0.0-3154.ga630765c.el9. 168 kB/s | 18 kB 00:00 2024-04-21T08:37:47.110 INFO:teuthology.orchestra.run.smithi028.stdout:(123/150): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-04-21T08:37:47.135 INFO:teuthology.orchestra.run.smithi028.stdout:(124/150): python3-grpcio-tools-1.46.7-10.el9.x 961 kB/s | 144 kB 00:00 2024-04-21T08:37:47.160 INFO:teuthology.orchestra.run.smithi028.stdout:(125/150): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-04-21T08:37:47.211 INFO:teuthology.orchestra.run.smithi028.stdout:(126/150): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-04-21T08:37:47.236 INFO:teuthology.orchestra.run.smithi028.stdout:(127/150): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-04-21T08:37:47.261 INFO:teuthology.orchestra.run.smithi028.stdout:(128/150): python3-jaraco-functools-3.5.0-2.el9 193 kB/s | 19 kB 00:00 2024-04-21T08:37:47.266 INFO:teuthology.orchestra.run.smithi171.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 344 kB/s | 80 kB 00:00 2024-04-21T08:37:47.312 INFO:teuthology.orchestra.run.smithi028.stdout:(129/150): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-21T08:37:47.367 INFO:teuthology.orchestra.run.smithi171.stdout:(40/150): ceph-mgr-diskprediction-local-19.0.0- 9.0 MB/s | 7.4 MB 00:00 2024-04-21T08:37:47.370 INFO:teuthology.orchestra.run.smithi028.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 426 kB/s | 46 kB 00:00 2024-04-21T08:37:47.403 INFO:teuthology.orchestra.run.smithi028.stdout:(131/150): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-04-21T08:37:47.425 INFO:teuthology.orchestra.run.smithi171.stdout:(41/150): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-04-21T08:37:47.445 INFO:teuthology.orchestra.run.smithi028.stdout:(132/150): python3-more-itertools-8.12.0-2.el9. 590 kB/s | 79 kB 00:00 2024-04-21T08:37:47.475 INFO:teuthology.orchestra.run.smithi171.stdout:(42/150): libquadmath-11.4.1-3.el9.x86_64.rpm 936 kB/s | 195 kB 00:00 2024-04-21T08:37:47.480 INFO:teuthology.orchestra.run.smithi028.stdout:(133/150): python3-natsort-7.1.1-5.el9.noarch.r 530 kB/s | 58 kB 00:00 2024-04-21T08:37:47.521 INFO:teuthology.orchestra.run.smithi028.stdout:(134/150): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-21T08:37:47.542 INFO:teuthology.orchestra.run.smithi171.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-21T08:37:47.555 INFO:teuthology.orchestra.run.smithi028.stdout:(135/150): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-04-21T08:37:47.584 INFO:teuthology.orchestra.run.smithi171.stdout:(44/150): python3-cryptography-36.0.1-4.el9.x86 9.2 MB/s | 1.2 MB 00:00 2024-04-21T08:37:47.588 INFO:teuthology.orchestra.run.smithi028.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 827 kB/s | 90 kB 00:00 2024-04-21T08:37:47.622 INFO:teuthology.orchestra.run.smithi028.stdout:(137/150): python3-pyperclip-1.8.2-2.el9.noarch 280 kB/s | 28 kB 00:00 2024-04-21T08:37:47.626 INFO:teuthology.orchestra.run.smithi171.stdout:(45/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 977 kB/s | 253 kB 00:00 2024-04-21T08:37:47.651 INFO:teuthology.orchestra.run.smithi171.stdout:(46/150): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-04-21T08:37:47.655 INFO:teuthology.orchestra.run.smithi028.stdout:(138/150): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-04-21T08:37:47.677 INFO:teuthology.orchestra.run.smithi171.stdout:(47/150): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-04-21T08:37:47.702 INFO:teuthology.orchestra.run.smithi171.stdout:(48/150): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-04-21T08:37:47.706 INFO:teuthology.orchestra.run.smithi028.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-04-21T08:37:47.731 INFO:teuthology.orchestra.run.smithi028.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 544 kB/s | 59 kB 00:00 2024-04-21T08:37:47.756 INFO:teuthology.orchestra.run.smithi028.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 355 kB/s | 36 kB 00:00 2024-04-21T08:37:47.848 INFO:teuthology.orchestra.run.smithi028.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 359 kB/s | 51 kB 00:00 2024-04-21T08:37:47.865 INFO:teuthology.orchestra.run.smithi028.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 825 kB/s | 90 kB 00:00 2024-04-21T08:37:47.885 INFO:teuthology.orchestra.run.smithi171.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 583 kB/s | 107 kB 00:00 2024-04-21T08:37:47.891 INFO:teuthology.orchestra.run.smithi028.stdout:(144/150): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-04-21T08:37:47.944 INFO:teuthology.orchestra.run.smithi171.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 368 kB/s | 21 kB 00:00 2024-04-21T08:37:47.978 INFO:teuthology.orchestra.run.smithi171.stdout:(51/150): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-04-21T08:37:47.982 INFO:teuthology.orchestra.run.smithi028.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 171 kB/s | 20 kB 00:00 2024-04-21T08:37:48.008 INFO:teuthology.orchestra.run.smithi028.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-04-21T08:37:48.020 INFO:teuthology.orchestra.run.smithi171.stdout:(52/150): bison-3.7.4-5.el9.x86_64.rpm 2.7 MB/s | 943 kB 00:00 2024-04-21T08:37:48.041 INFO:teuthology.orchestra.run.smithi028.stdout:(147/150): python3-werkzeug-2.0.3-3.el9.1.noarc 2.2 MB/s | 427 kB 00:00 2024-04-21T08:37:48.045 INFO:teuthology.orchestra.run.smithi171.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 489 kB/s | 33 kB 00:00 2024-04-21T08:37:48.070 INFO:teuthology.orchestra.run.smithi171.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 2.4 MB/s | 313 kB 00:00 2024-04-21T08:37:48.083 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-04-21T08:37:48.104 INFO:teuthology.orchestra.run.smithi171.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 312 kB/s | 18 kB 00:00 2024-04-21T08:37:48.125 INFO:teuthology.orchestra.run.smithi028.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-21T08:37:48.129 INFO:teuthology.orchestra.run.smithi171.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 190 kB/s | 11 kB 00:00 2024-04-21T08:37:48.163 INFO:teuthology.orchestra.run.smithi171.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-21T08:37:48.188 INFO:teuthology.orchestra.run.smithi171.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 772 kB/s | 45 kB 00:00 2024-04-21T08:37:48.255 INFO:teuthology.orchestra.run.smithi171.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 662 kB 00:00 2024-04-21T08:37:48.305 INFO:teuthology.orchestra.run.smithi171.stdout:(60/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 10 MB/s | 3.0 MB 00:00 2024-04-21T08:37:48.322 INFO:teuthology.orchestra.run.smithi171.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 380 kB/s | 25 kB 00:00 2024-04-21T08:37:48.348 INFO:teuthology.orchestra.run.smithi171.stdout:(62/150): libtool-2.4.6-45.el9.x86_64.rpm 3.6 MB/s | 582 kB 00:00 2024-04-21T08:37:48.367 INFO:teuthology.orchestra.run.smithi028.stdout:(149/150): librbd1-19.0.0-3154.ga630765c.el9.x8 9.7 MB/s | 3.1 MB 00:00 2024-04-21T08:37:48.373 INFO:teuthology.orchestra.run.smithi171.stdout:(63/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 706 kB/s | 47 kB 00:00 2024-04-21T08:37:48.398 INFO:teuthology.orchestra.run.smithi171.stdout:(64/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-21T08:37:48.423 INFO:teuthology.orchestra.run.smithi171.stdout:(65/150): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-04-21T08:37:48.440 INFO:teuthology.orchestra.run.smithi171.stdout:(66/150): openblas-0.3.21-2.el9.x86_64.rpm 525 kB/s | 35 kB 00:00 2024-04-21T08:37:48.450 INFO:teuthology.orchestra.run.smithi028.stdout:(150/150): librados2-19.0.0-3154.ga630765c.el9. 7.7 MB/s | 3.4 MB 00:00 2024-04-21T08:37:48.454 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:37:48.454 INFO:teuthology.orchestra.run.smithi028.stdout:Total 13 MB/s | 218 MB 00:16 2024-04-21T08:37:48.599 INFO:teuthology.orchestra.run.smithi171.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-21T08:37:48.691 INFO:teuthology.orchestra.run.smithi171.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 9.7 MB/s | 911 kB 00:00 2024-04-21T08:37:48.749 INFO:teuthology.orchestra.run.smithi171.stdout:(69/150): openblas-openmp-0.3.21-2.el9.x86_64.r 14 MB/s | 4.7 MB 00:00 2024-04-21T08:37:48.783 INFO:teuthology.orchestra.run.smithi171.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-04-21T08:37:48.842 INFO:teuthology.orchestra.run.smithi171.stdout:(71/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 9.9 MB/s | 4.1 MB 00:00 2024-04-21T08:37:48.917 INFO:teuthology.orchestra.run.smithi171.stdout:(72/150): protobuf-3.14.0-13.el9.x86_64.rpm 7.6 MB/s | 1.0 MB 00:00 2024-04-21T08:37:48.942 INFO:teuthology.orchestra.run.smithi171.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 875 kB/s | 87 kB 00:00 2024-04-21T08:37:49.001 INFO:teuthology.orchestra.run.smithi171.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-21T08:37:49.159 INFO:teuthology.orchestra.run.smithi171.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 1.5 MB/s | 248 kB 00:00 2024-04-21T08:37:49.243 INFO:teuthology.orchestra.run.smithi171.stdout:(76/150): postgresql-server-13.14-1.el9.x86_64. 12 MB/s | 5.8 MB 00:00 2024-04-21T08:37:49.269 INFO:teuthology.orchestra.run.smithi171.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 236 kB/s | 26 kB 00:00 2024-04-21T08:37:49.302 INFO:teuthology.orchestra.run.smithi171.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 328 kB/s | 19 kB 00:00 2024-04-21T08:37:49.328 INFO:teuthology.orchestra.run.smithi171.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-21T08:37:49.361 INFO:teuthology.orchestra.run.smithi171.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 594 kB/s | 35 kB 00:00 2024-04-21T08:37:49.453 INFO:teuthology.orchestra.run.smithi171.stdout:(81/150): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-04-21T08:37:49.478 INFO:teuthology.orchestra.run.smithi171.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-04-21T08:37:49.520 INFO:teuthology.orchestra.run.smithi171.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 3.3 MB/s | 222 kB 00:00 2024-04-21T08:37:49.545 INFO:teuthology.orchestra.run.smithi171.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-04-21T08:37:49.574 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-21T08:37:49.579 INFO:teuthology.orchestra.run.smithi171.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 722 kB/s | 42 kB 00:00 2024-04-21T08:37:49.604 INFO:teuthology.orchestra.run.smithi171.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-04-21T08:37:49.638 INFO:teuthology.orchestra.run.smithi171.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-04-21T08:37:49.671 INFO:teuthology.orchestra.run.smithi171.stdout:(88/150): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-04-21T08:37:49.711 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-21T08:37:49.711 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-21T08:37:49.738 INFO:teuthology.orchestra.run.smithi171.stdout:(89/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-04-21T08:37:49.763 INFO:teuthology.orchestra.run.smithi171.stdout:(90/150): python3-pytz-2021.1-5.el9.noarch.rpm 408 kB/s | 51 kB 00:00 2024-04-21T08:37:49.789 INFO:teuthology.orchestra.run.smithi171.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 458 kB/s | 54 kB 00:00 2024-04-21T08:37:49.822 INFO:teuthology.orchestra.run.smithi171.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 714 kB/s | 42 kB 00:00 2024-04-21T08:37:49.848 INFO:teuthology.orchestra.run.smithi171.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 751 kB/s | 44 kB 00:00 2024-04-21T08:37:49.881 INFO:teuthology.orchestra.run.smithi171.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-21T08:37:49.907 INFO:teuthology.orchestra.run.smithi171.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 614 kB/s | 36 kB 00:00 2024-04-21T08:37:49.940 INFO:teuthology.orchestra.run.smithi171.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-04-21T08:37:49.966 INFO:teuthology.orchestra.run.smithi171.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-21T08:37:49.999 INFO:teuthology.orchestra.run.smithi171.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 990 kB/s | 58 kB 00:00 2024-04-21T08:37:50.025 INFO:teuthology.orchestra.run.smithi171.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-21T08:37:50.058 INFO:teuthology.orchestra.run.smithi171.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-21T08:37:50.084 INFO:teuthology.orchestra.run.smithi171.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 765 kB/s | 45 kB 00:00 2024-04-21T08:37:50.442 INFO:teuthology.orchestra.run.smithi171.stdout:(102/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-04-21T08:37:50.467 INFO:teuthology.orchestra.run.smithi171.stdout:(103/150): lua-devel-5.4.4-4.el9.x86_64.rpm 54 kB/s | 22 kB 00:00 2024-04-21T08:37:50.709 INFO:teuthology.orchestra.run.smithi171.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 2.0 MB/s | 548 kB 00:00 2024-04-21T08:37:50.734 INFO:teuthology.orchestra.run.smithi171.stdout:(105/150): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-04-21T08:37:50.752 INFO:teuthology.orchestra.run.smithi171.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 469 kB/s | 19 kB 00:00 2024-04-21T08:37:50.818 INFO:teuthology.orchestra.run.smithi171.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 391 kB/s | 26 kB 00:00 2024-04-21T08:37:50.927 INFO:teuthology.orchestra.run.smithi171.stdout:(108/150): libarrow-9.0.0-10.el9.x86_64.rpm 23 MB/s | 4.4 MB 00:00 2024-04-21T08:37:50.944 INFO:teuthology.orchestra.run.smithi171.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 394 kB/s | 49 kB 00:00 2024-04-21T08:37:50.978 INFO:teuthology.orchestra.run.smithi171.stdout:(110/150): protobuf-compiler-3.14.0-13.el9.x86_ 966 kB/s | 863 kB 00:00 2024-04-21T08:37:51.003 INFO:teuthology.orchestra.run.smithi171.stdout:(111/150): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-21T08:37:51.028 INFO:teuthology.orchestra.run.smithi171.stdout:(112/150): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-04-21T08:37:51.070 INFO:teuthology.orchestra.run.smithi171.stdout:(113/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.0 MB/s | 838 kB 00:00 2024-04-21T08:37:51.095 INFO:teuthology.orchestra.run.smithi171.stdout:(114/150): python3-asyncssh-2.13.2-5.el9.noarch 8.2 MB/s | 548 kB 00:00 2024-04-21T08:37:51.120 INFO:teuthology.orchestra.run.smithi171.stdout:(115/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 481 kB/s | 43 kB 00:00 2024-04-21T08:37:51.145 INFO:teuthology.orchestra.run.smithi171.stdout:(116/150): python3-cachetools-4.2.4-1.el9.noarc 433 kB/s | 32 kB 00:00 2024-04-21T08:37:51.171 INFO:teuthology.orchestra.run.smithi171.stdout:(117/150): python3-certifi-2021.10.8-2.el9.noar 302 kB/s | 15 kB 00:00 2024-04-21T08:37:51.196 INFO:teuthology.orchestra.run.smithi171.stdout:(118/150): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-21T08:37:51.221 INFO:teuthology.orchestra.run.smithi171.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-21T08:37:51.246 INFO:teuthology.orchestra.run.smithi171.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.0 MB/s | 303 kB 00:00 2024-04-21T08:37:51.272 INFO:teuthology.orchestra.run.smithi171.stdout:(121/150): python3-colorama-0.4.4-12.el9.noarch 474 kB/s | 35 kB 00:00 2024-04-21T08:37:51.297 INFO:teuthology.orchestra.run.smithi171.stdout:(122/150): python3-google-auth-2.28.2-1.el9.noa 4.0 MB/s | 203 kB 00:00 2024-04-21T08:37:51.347 INFO:teuthology.orchestra.run.smithi171.stdout:(123/150): python3-grpcio-1.46.7-10.el9.x86_64. 20 MB/s | 2.0 MB 00:00 2024-04-21T08:37:51.372 INFO:teuthology.orchestra.run.smithi171.stdout:(124/150): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-04-21T08:37:51.398 INFO:teuthology.orchestra.run.smithi171.stdout:(125/150): python3-jaraco-8.2.1-3.el9.noarch.rp 109 kB/s | 11 kB 00:00 2024-04-21T08:37:51.423 INFO:teuthology.orchestra.run.smithi171.stdout:(126/150): python3-jaraco-classes-3.2.1-5.el9.n 236 kB/s | 18 kB 00:00 2024-04-21T08:37:51.448 INFO:teuthology.orchestra.run.smithi171.stdout:(127/150): python3-jaraco-collections-3.0.0-8.e 464 kB/s | 23 kB 00:00 2024-04-21T08:37:51.473 INFO:teuthology.orchestra.run.smithi171.stdout:(128/150): python3-jaraco-functools-3.5.0-2.el9 259 kB/s | 19 kB 00:00 2024-04-21T08:37:51.498 INFO:teuthology.orchestra.run.smithi171.stdout:(129/150): python3-jaraco-text-3.2.0-6.el9.noar 262 kB/s | 20 kB 00:00 2024-04-21T08:37:51.533 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-21T08:37:51.533 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-21T08:37:51.540 INFO:teuthology.orchestra.run.smithi171.stdout:(130/150): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-04-21T08:37:51.566 INFO:teuthology.orchestra.run.smithi171.stdout:(131/150): python3-logutils-0.3.5-21.el9.noarch 504 kB/s | 46 kB 00:00 2024-04-21T08:37:51.591 INFO:teuthology.orchestra.run.smithi171.stdout:(132/150): python3-more-itertools-8.12.0-2.el9. 861 kB/s | 79 kB 00:00 2024-04-21T08:37:51.625 INFO:teuthology.orchestra.run.smithi171.stdout:(133/150): python3-natsort-7.1.1-5.el9.noarch.r 690 kB/s | 58 kB 00:00 2024-04-21T08:37:51.659 INFO:teuthology.orchestra.run.smithi171.stdout:(134/150): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-04-21T08:37:51.683 INFO:teuthology.orchestra.run.smithi171.stdout:(135/150): python3-portend-3.1.0-2.el9.noarch.r 179 kB/s | 16 kB 00:00 2024-04-21T08:37:51.709 INFO:teuthology.orchestra.run.smithi171.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-04-21T08:37:51.734 INFO:teuthology.orchestra.run.smithi171.stdout:(137/150): python3-pyperclip-1.8.2-2.el9.noarch 566 kB/s | 28 kB 00:00 2024-04-21T08:37:51.759 INFO:teuthology.orchestra.run.smithi171.stdout:(138/150): python3-repoze-lru-0.7-16.el9.noarch 410 kB/s | 31 kB 00:00 2024-04-21T08:37:51.784 INFO:teuthology.orchestra.run.smithi171.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 2.5 MB/s | 188 kB 00:00 2024-04-21T08:37:51.810 INFO:teuthology.orchestra.run.smithi171.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-21T08:37:51.835 INFO:teuthology.orchestra.run.smithi171.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 478 kB/s | 36 kB 00:00 2024-04-21T08:37:51.860 INFO:teuthology.orchestra.run.smithi171.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 678 kB/s | 51 kB 00:00 2024-04-21T08:37:51.885 INFO:teuthology.orchestra.run.smithi171.stdout:(143/150): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-21T08:37:51.911 INFO:teuthology.orchestra.run.smithi171.stdout:(144/150): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-21T08:37:51.936 INFO:teuthology.orchestra.run.smithi171.stdout:(145/150): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-04-21T08:37:51.961 INFO:teuthology.orchestra.run.smithi171.stdout:(146/150): python3-zc-lockfile-2.0-10.el9.noarc 267 kB/s | 20 kB 00:00 2024-04-21T08:37:51.987 INFO:teuthology.orchestra.run.smithi171.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-21T08:37:52.037 INFO:teuthology.orchestra.run.smithi171.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-04-21T08:37:52.445 INFO:teuthology.orchestra.run.smithi171.stdout:(149/150): librados2-19.0.0-3154.ga630765c.el9. 7.0 MB/s | 3.4 MB 00:00 2024-04-21T08:37:52.970 INFO:teuthology.orchestra.run.smithi171.stdout:(150/150): librbd1-19.0.0-3154.ga630765c.el9.x8 3.2 MB/s | 3.1 MB 00:00 2024-04-21T08:37:52.977 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:37:52.977 INFO:teuthology.orchestra.run.smithi171.stdout:Total 17 MB/s | 218 MB 00:13 2024-04-21T08:37:53.166 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-21T08:37:54.038 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-04-21T08:37:54.164 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-04-21T08:37:54.164 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-04-21T08:37:55.409 INFO:teuthology.orchestra.run.smithi152.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:37:55.410 INFO:teuthology.orchestra.run.smithi152.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:37:55.421 INFO:teuthology.orchestra.run.smithi152.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T08:37:55.422 INFO:teuthology.orchestra.run.smithi152.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T08:37:55.423 INFO:teuthology.orchestra.run.smithi152.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T08:37:55.424 INFO:teuthology.orchestra.run.smithi152.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T08:37:55.425 INFO:teuthology.orchestra.run.smithi152.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T08:37:55.428 INFO:teuthology.orchestra.run.smithi152.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T08:37:55.429 INFO:teuthology.orchestra.run.smithi152.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T08:37:55.489 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-21T08:37:55.501 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-04-21T08:37:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-04-21T08:37:55.502 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-04-21T08:37:55.502 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-21T08:37:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T08:37:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: ceph x86_64 2:19.0.0-3154.ga630765c.el9 ceph 6.6 k 2024-04-21T08:37:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:37:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 890 k 2024-04-21T08:37:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3154.ga630765c.el9 ceph 145 k 2024-04-21T08:37:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr x86_64 2:19.0.0-3154.ga630765c.el9 ceph 1.5 M 2024-04-21T08:37:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 142 k 2024-04-21T08:37:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 3.5 M 2024-04-21T08:37:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 7.4 M 2024-04-21T08:37:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 49 k 2024-04-21T08:37:55.502 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 11 M 2024-04-21T08:37:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test x86_64 2:19.0.0-3154.ga630765c.el9 ceph 49 M 2024-04-21T08:37:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 273 k 2024-04-21T08:37:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 762 k 2024-04-21T08:37:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 225 k 2024-04-21T08:37:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-shell x86_64 2:19.0.0-3154.ga630765c.el9 ceph 27 k 2024-04-21T08:37:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-top noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 18 k 2024-04-21T08:37:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T08:37:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T08:37:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T08:37:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 32 k 2024-04-21T08:37:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 746 k 2024-04-21T08:37:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 127 k 2024-04-21T08:37:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T08:37:55.503 INFO:teuthology.orchestra.run.smithi152.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T08:37:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T08:37:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T08:37:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T08:37:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T08:37:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs x86_64 2:19.0.0-3154.ga630765c.el9 ceph 162 k 2024-04-21T08:37:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados x86_64 2:19.0.0-3154.ga630765c.el9 ceph 321 k 2024-04-21T08:37:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 298 k 2024-04-21T08:37:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 100 k 2024-04-21T08:37:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 85 k 2024-04-21T08:37:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:37:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 171 k 2024-04-21T08:37:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T08:37:55.504 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrading: 2024-04-21T08:37:55.504 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.4 M 2024-04-21T08:37:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:37:55.505 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-04-21T08:37:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T08:37:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T08:37:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T08:37:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 21 M 2024-04-21T08:37:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 34 k 2024-04-21T08:37:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds x86_64 2:19.0.0-3154.ga630765c.el9 ceph 2.4 M 2024-04-21T08:37:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 249 k 2024-04-21T08:37:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon x86_64 2:19.0.0-3154.ga630765c.el9 ceph 4.7 M 2024-04-21T08:37:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 17 M 2024-04-21T08:37:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 16 k 2024-04-21T08:37:55.505 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux x86_64 2:19.0.0-3154.ga630765c.el9 ceph 25 k 2024-04-21T08:37:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T08:37:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T08:37:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T08:37:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T08:37:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T08:37:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T08:37:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T08:37:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T08:37:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite x86_64 2:19.0.0-3154.ga630765c.el9 ceph 163 k 2024-04-21T08:37:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T08:37:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T08:37:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T08:37:55.506 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T08:37:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T08:37:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 501 k 2024-04-21T08:37:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T08:37:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:37:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T08:37:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T08:37:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T08:37:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T08:37:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T08:37:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T08:37:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T08:37:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T08:37:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T08:37:55.507 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T08:37:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T08:37:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T08:37:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T08:37:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T08:37:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T08:37:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T08:37:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T08:37:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 45 k 2024-04-21T08:37:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 135 k 2024-04-21T08:37:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T08:37:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T08:37:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T08:37:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T08:37:55.508 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T08:37:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T08:37:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T08:37:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T08:37:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T08:37:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T08:37:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T08:37:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T08:37:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T08:37:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T08:37:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T08:37:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T08:37:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T08:37:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T08:37:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T08:37:55.509 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T08:37:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T08:37:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T08:37:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T08:37:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T08:37:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T08:37:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T08:37:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T08:37:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T08:37:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T08:37:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T08:37:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T08:37:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T08:37:55.510 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T08:37:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T08:37:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T08:37:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T08:37:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T08:37:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T08:37:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T08:37:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T08:37:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T08:37:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T08:37:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T08:37:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T08:37:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T08:37:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T08:37:55.511 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T08:37:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T08:37:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T08:37:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T08:37:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T08:37:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T08:37:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T08:37:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T08:37:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T08:37:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T08:37:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T08:37:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T08:37:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T08:37:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T08:37:55.512 INFO:teuthology.orchestra.run.smithi152.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T08:37:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T08:37:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T08:37:55.513 INFO:teuthology.orchestra.run.smithi152.stdout:Installing weak dependencies: 2024-04-21T08:37:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T08:37:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-21T08:37:55.513 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-21T08:37:55.513 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-04-21T08:37:55.513 INFO:teuthology.orchestra.run.smithi152.stdout:Install 148 Packages 2024-04-21T08:37:55.513 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrade 2 Packages 2024-04-21T08:37:55.513 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-21T08:37:55.514 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 218 M 2024-04-21T08:37:55.514 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-21T08:37:55.882 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-04-21T08:37:55.927 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-04-21T08:37:56.518 INFO:teuthology.orchestra.run.smithi152.stdout:(1/150): ceph-19.0.0-3154.ga630765c.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-04-21T08:37:56.735 INFO:teuthology.orchestra.run.smithi152.stdout:(2/150): ceph-fuse-19.0.0-3154.ga630765c.el9.x8 4.0 MB/s | 890 kB 00:00 2024-04-21T08:37:56.802 INFO:teuthology.orchestra.run.smithi152.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 2.1 MB/s | 145 kB 00:00 2024-04-21T08:37:56.944 INFO:teuthology.orchestra.run.smithi152.stdout:(4/150): ceph-base-19.0.0-3154.ga630765c.el9.x8 9.0 MB/s | 5.4 MB 00:00 2024-04-21T08:37:57.010 INFO:teuthology.orchestra.run.smithi152.stdout:(5/150): ceph-mds-19.0.0-3154.ga630765c.el9.x86 11 MB/s | 2.4 MB 00:00 2024-04-21T08:37:57.069 INFO:teuthology.orchestra.run.smithi152.stdout:(6/150): ceph-mgr-19.0.0-3154.ga630765c.el9.x86 12 MB/s | 1.5 MB 00:00 2024-04-21T08:37:57.433 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-21T08:37:57.436 INFO:teuthology.orchestra.run.smithi152.stdout:(7/150): ceph-mon-19.0.0-3154.ga630765c.el9.x86 11 MB/s | 4.7 MB 00:00 2024-04-21T08:37:57.572 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T08:37:57.796 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T08:37:57.843 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T08:37:58.185 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T08:37:58.288 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:37:58.345 INFO:teuthology.orchestra.run.smithi152.stdout:(8/150): ceph-osd-19.0.0-3154.ga630765c.el9.x86 13 MB/s | 17 MB 00:01 2024-04-21T08:37:58.404 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:37:58.438 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:37:58.470 INFO:teuthology.orchestra.run.smithi152.stdout:(9/150): ceph-radosgw-19.0.0-3154.ga630765c.el9 10 MB/s | 11 MB 00:01 2024-04-21T08:37:58.496 INFO:teuthology.orchestra.run.smithi152.stdout:(10/150): ceph-selinux-19.0.0-3154.ga630765c.el 167 kB/s | 25 kB 00:00 2024-04-21T08:37:58.531 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:37:58.554 INFO:teuthology.orchestra.run.smithi152.stdout:(11/150): cephfs-mirror-19.0.0-3154.ga630765c.e 3.8 MB/s | 225 kB 00:00 2024-04-21T08:37:58.575 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:37:58.605 INFO:teuthology.orchestra.run.smithi152.stdout:(12/150): cephfs-shell-19.0.0-3154.ga630765c.el 542 kB/s | 27 kB 00:00 2024-04-21T08:37:58.611 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T08:37:58.637 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T08:37:58.672 INFO:teuthology.orchestra.run.smithi152.stdout:(13/150): libcephfs-devel-19.0.0-3154.ga630765c 475 kB/s | 32 kB 00:00 2024-04-21T08:37:58.780 INFO:teuthology.orchestra.run.smithi152.stdout:(14/150): libcephfs2-19.0.0-3154.ga630765c.el9. 6.7 MB/s | 746 kB 00:00 2024-04-21T08:37:58.806 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T08:37:58.847 INFO:teuthology.orchestra.run.smithi152.stdout:(15/150): libcephsqlite-19.0.0-3154.ga630765c.e 2.4 MB/s | 163 kB 00:00 2024-04-21T08:37:58.852 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T08:37:58.879 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:37:58.923 INFO:teuthology.orchestra.run.smithi152.stdout:(16/150): librados-devel-19.0.0-3154.ga630765c. 1.7 MB/s | 127 kB 00:00 2024-04-21T08:37:58.984 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:37:58.998 INFO:teuthology.orchestra.run.smithi152.stdout:(17/150): libradosstriper1-19.0.0-3154.ga630765 6.5 MB/s | 501 kB 00:00 2024-04-21T08:37:59.013 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:37:59.059 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:37:59.081 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:37:59.107 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:37:59.307 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:37:59.339 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:37:59.449 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T08:37:59.465 INFO:teuthology.orchestra.run.smithi152.stdout:(18/150): librgw2-19.0.0-3154.ga630765c.el9.x86 12 MB/s | 5.4 MB 00:00 2024-04-21T08:37:59.515 INFO:teuthology.orchestra.run.smithi152.stdout:(19/150): python3-ceph-argparse-19.0.0-3154.ga6 903 kB/s | 45 kB 00:00 2024-04-21T08:37:59.550 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T08:37:59.582 INFO:teuthology.orchestra.run.smithi152.stdout:(20/150): python3-ceph-common-19.0.0-3154.ga630 2.0 MB/s | 135 kB 00:00 2024-04-21T08:37:59.657 INFO:teuthology.orchestra.run.smithi152.stdout:(21/150): python3-cephfs-19.0.0-3154.ga630765c. 2.1 MB/s | 162 kB 00:00 2024-04-21T08:37:59.705 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T08:37:59.741 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T08:37:59.749 INFO:teuthology.orchestra.run.smithi152.stdout:(22/150): python3-rados-19.0.0-3154.ga630765c.e 3.4 MB/s | 321 kB 00:00 2024-04-21T08:37:59.818 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T08:37:59.824 INFO:teuthology.orchestra.run.smithi152.stdout:(23/150): python3-rbd-19.0.0-3154.ga630765c.el9 3.9 MB/s | 298 kB 00:00 2024-04-21T08:37:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T08:37:59.900 INFO:teuthology.orchestra.run.smithi152.stdout:(24/150): python3-rgw-19.0.0-3154.ga630765c.el9 1.3 MB/s | 100 kB 00:00 2024-04-21T08:37:59.908 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T08:37:59.940 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T08:37:59.958 INFO:teuthology.orchestra.run.smithi152.stdout:(25/150): rbd-fuse-19.0.0-3154.ga630765c.el9.x8 1.4 MB/s | 85 kB 00:00 2024-04-21T08:37:59.968 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T08:38:00.042 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T08:38:00.093 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T08:38:00.120 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T08:38:00.164 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T08:38:00.245 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T08:38:00.267 INFO:teuthology.orchestra.run.smithi152.stdout:(26/150): rbd-mirror-19.0.0-3154.ga630765c.el9. 10 MB/s | 3.1 MB 00:00 2024-04-21T08:38:00.275 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 30/152 2024-04-21T08:38:00.319 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 31/152 2024-04-21T08:38:00.334 INFO:teuthology.orchestra.run.smithi152.stdout:(27/150): rbd-nbd-19.0.0-3154.ga630765c.el9.x86 2.5 MB/s | 171 kB 00:00 2024-04-21T08:38:00.347 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 32/152 2024-04-21T08:38:00.393 INFO:teuthology.orchestra.run.smithi152.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3154.g 588 kB/s | 34 kB 00:00 2024-04-21T08:38:00.424 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T08:38:00.468 INFO:teuthology.orchestra.run.smithi152.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3154.ga630765 1.8 MB/s | 142 kB 00:00 2024-04-21T08:38:00.470 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T08:38:00.551 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T08:38:00.710 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T08:38:00.784 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T08:38:00.802 INFO:teuthology.orchestra.run.smithi152.stdout:(30/150): ceph-mgr-dashboard-19.0.0-3154.ga6307 11 MB/s | 3.5 MB 00:00 2024-04-21T08:38:01.364 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T08:38:01.449 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T08:38:01.480 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T08:38:01.523 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T08:38:01.562 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 42/152 2024-04-21T08:38:01.612 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T08:38:01.636 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-04-21T08:38:01.669 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T08:38:01.686 INFO:teuthology.orchestra.run.smithi152.stdout:(31/150): ceph-test-19.0.0-3154.ga630765c.el9.x 15 MB/s | 49 MB 00:03 2024-04-21T08:38:01.714 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T08:38:01.744 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T08:38:01.783 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T08:38:01.786 INFO:teuthology.orchestra.run.smithi152.stdout:(32/150): ceph-mgr-diskprediction-local-19.0.0- 7.5 MB/s | 7.4 MB 00:00 2024-04-21T08:38:01.786 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T08:38:01.811 INFO:teuthology.orchestra.run.smithi152.stdout:(33/150): ceph-mgr-modules-core-19.0.0-3154.ga6 1.9 MB/s | 249 kB 00:00 2024-04-21T08:38:01.823 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T08:38:01.836 INFO:teuthology.orchestra.run.smithi152.stdout:(34/150): ceph-mgr-rook-19.0.0-3154.ga630765c.e 989 kB/s | 49 kB 00:00 2024-04-21T08:38:01.858 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T08:38:01.862 INFO:teuthology.orchestra.run.smithi152.stdout:(35/150): ceph-prometheus-alerts-19.0.0-3154.ga 325 kB/s | 16 kB 00:00 2024-04-21T08:38:01.872 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T08:38:01.887 INFO:teuthology.orchestra.run.smithi152.stdout:(36/150): ceph-volume-19.0.0-3154.ga630765c.el9 5.3 MB/s | 273 kB 00:00 2024-04-21T08:38:01.914 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T08:38:01.929 INFO:teuthology.orchestra.run.smithi152.stdout:(37/150): cephfs-top-19.0.0-3154.ga630765c.el9. 437 kB/s | 18 kB 00:00 2024-04-21T08:38:01.955 INFO:teuthology.orchestra.run.smithi152.stdout:(38/150): cephadm-19.0.0-3154.ga630765c.el9.noa 8.1 MB/s | 762 kB 00:00 2024-04-21T08:38:01.959 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T08:38:02.001 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T08:38:02.071 INFO:teuthology.orchestra.run.smithi152.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 567 kB/s | 80 kB 00:00 2024-04-21T08:38:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T08:38:02.188 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T08:38:02.197 INFO:teuthology.orchestra.run.smithi152.stdout:(40/150): libgfortran-11.4.1-3.el9.x86_64.rpm 3.2 MB/s | 803 kB 00:00 2024-04-21T08:38:02.197 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T08:38:02.222 INFO:teuthology.orchestra.run.smithi152.stdout:(41/150): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-04-21T08:38:02.235 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T08:38:02.247 INFO:teuthology.orchestra.run.smithi152.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 253 kB 00:00 2024-04-21T08:38:02.278 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T08:38:02.298 INFO:teuthology.orchestra.run.smithi152.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-21T08:38:02.307 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:38:02.348 INFO:teuthology.orchestra.run.smithi152.stdout:(44/150): python3-pycparser-2.20-6.el9.noarch.r 2.7 MB/s | 135 kB 00:00 2024-04-21T08:38:02.379 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T08:38:02.398 INFO:teuthology.orchestra.run.smithi152.stdout:(45/150): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-21T08:38:02.409 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T08:38:02.429 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:38:02.441 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T08:38:02.449 INFO:teuthology.orchestra.run.smithi152.stdout:(46/150): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-21T08:38:02.458 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:38:02.475 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T08:38:02.500 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T08:38:02.550 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:38:02.581 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T08:38:02.592 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:38:02.621 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T08:38:02.624 INFO:teuthology.orchestra.run.smithi152.stdout:(47/150): python3-cryptography-36.0.1-4.el9.x86 3.1 MB/s | 1.2 MB 00:00 2024-04-21T08:38:02.646 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T08:38:02.718 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T08:38:02.757 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T08:38:02.785 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T08:38:02.803 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T08:38:02.819 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T08:38:02.852 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T08:38:02.865 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T08:38:02.880 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:38:02.902 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T08:38:02.941 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T08:38:02.967 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:38:02.970 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T08:38:02.989 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:38:02.996 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T08:38:03.030 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T08:38:03.052 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:38:03.072 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T08:38:03.074 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:38:03.099 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:38:03.106 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T08:38:03.199 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T08:38:03.246 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T08:38:03.256 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:38:03.276 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T08:38:03.290 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:38:03.325 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T08:38:03.407 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T08:38:03.439 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T08:38:03.441 INFO:teuthology.orchestra.run.smithi152.stdout:(48/150): ceph-common-19.0.0-3154.ga630765c.el9 3.0 MB/s | 21 MB 00:07 2024-04-21T08:38:03.496 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T08:38:03.505 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T08:38:03.548 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T08:38:03.592 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T08:38:03.667 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T08:38:03.707 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T08:38:03.751 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T08:38:03.783 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T08:38:03.811 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T08:38:03.882 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T08:38:03.935 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T08:38:03.964 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T08:38:03.991 INFO:teuthology.orchestra.run.smithi152.stdout:(49/150): automake-1.16.2-8.el9.noarch.rpm 447 kB/s | 689 kB 00:01 2024-04-21T08:38:04.007 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T08:38:04.086 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T08:38:04.118 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 30/152 2024-04-21T08:38:04.161 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 31/152 2024-04-21T08:38:04.190 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 32/152 2024-04-21T08:38:04.191 INFO:teuthology.orchestra.run.smithi152.stdout:(50/150): boost-program-options-1.75.0-8.el9.x8 142 kB/s | 107 kB 00:00 2024-04-21T08:38:04.263 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T08:38:04.305 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T08:38:04.307 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T08:38:04.349 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T08:38:04.382 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T08:38:04.383 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T08:38:04.409 INFO:teuthology.orchestra.run.smithi152.stdout:(51/150): bison-3.7.4-5.el9.x86_64.rpm 529 kB/s | 943 kB 00:01 2024-04-21T08:38:04.494 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T08:38:04.527 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T08:38:04.543 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T08:38:04.583 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T08:38:04.584 INFO:teuthology.orchestra.run.smithi152.stdout:(52/150): elfutils-libelf-devel-0.190-2.el9.x86 36 kB/s | 21 kB 00:00 2024-04-21T08:38:04.637 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T08:38:04.685 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T08:38:04.867 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T08:38:04.899 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T08:38:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T08:38:05.009 INFO:teuthology.orchestra.run.smithi152.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 54 kB/s | 33 kB 00:00 2024-04-21T08:38:05.129 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T08:38:05.201 INFO:teuthology.orchestra.run.smithi152.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 311 kB/s | 313 kB 00:01 2024-04-21T08:38:05.213 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T08:38:05.248 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T08:38:05.292 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T08:38:05.329 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 42/152 2024-04-21T08:38:05.362 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T08:38:05.418 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T08:38:05.491 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T08:38:05.530 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T08:38:05.543 INFO:teuthology.orchestra.run.smithi152.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 34 kB/s | 18 kB 00:00 2024-04-21T08:38:05.567 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T08:38:05.614 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T08:38:05.649 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T08:38:05.692 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T08:38:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T08:38:05.726 INFO:teuthology.orchestra.run.smithi152.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 21 kB/s | 11 kB 00:00 2024-04-21T08:38:05.735 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T08:38:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T08:38:05.874 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T08:38:05.915 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T08:38:05.945 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T08:38:05.986 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T08:38:06.082 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T08:38:06.119 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T08:38:06.159 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T08:38:06.185 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T08:38:06.210 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T08:38:06.286 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T08:38:06.360 INFO:teuthology.orchestra.run.smithi152.stdout:(57/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 71 kB/s | 45 kB 00:00 2024-04-21T08:38:06.385 INFO:teuthology.orchestra.run.smithi152.stdout:(58/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 190 kB/s | 160 kB 00:00 2024-04-21T08:38:06.414 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T08:38:06.450 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T08:38:06.478 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T08:38:06.514 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T08:38:06.557 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T08:38:06.594 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T08:38:06.634 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T08:38:06.664 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T08:38:06.698 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T08:38:06.732 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T08:38:06.774 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T08:38:06.808 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T08:38:06.897 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T08:38:06.938 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T08:38:06.969 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T08:38:07.109 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T08:38:07.177 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T08:38:07.253 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T08:38:07.281 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T08:38:07.287 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T08:38:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T08:38:07.338 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T08:38:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T08:38:07.676 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:38:07.769 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:38:07.798 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 98/152 2024-04-21T08:38:07.810 INFO:teuthology.orchestra.run.smithi152.stdout:(59/150): libtool-2.4.6-45.el9.x86_64.rpm 409 kB/s | 582 kB 00:01 2024-04-21T08:38:07.834 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T08:38:07.835 INFO:teuthology.orchestra.run.smithi152.stdout:(60/150): librdkafka-1.6.1-102.el9.x86_64.rpm 449 kB/s | 662 kB 00:01 2024-04-21T08:38:08.090 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T08:38:08.146 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T08:38:08.176 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T08:38:08.278 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T08:38:08.335 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T08:38:08.340 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T08:38:08.363 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:38:08.394 INFO:teuthology.orchestra.run.smithi152.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 44 kB/s | 25 kB 00:00 2024-04-21T08:38:08.417 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:38:08.424 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T08:38:08.439 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 102/152 2024-04-21T08:38:08.461 INFO:teuthology.orchestra.run.smithi152.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 76 kB/s | 47 kB 00:00 2024-04-21T08:38:08.462 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T08:38:08.638 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T08:38:08.667 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T08:38:08.738 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T08:38:09.103 INFO:teuthology.orchestra.run.smithi152.stdout:(63/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 677 kB/s | 3.0 MB 00:04 2024-04-21T08:38:09.320 INFO:teuthology.orchestra.run.smithi152.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 219 kB/s | 188 kB 00:00 2024-04-21T08:38:09.345 INFO:teuthology.orchestra.run.smithi152.stdout:(65/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 308 kB/s | 292 kB 00:00 2024-04-21T08:38:09.405 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T08:38:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T08:38:09.746 INFO:teuthology.orchestra.run.smithi152.stdout:(66/150): openblas-0.3.21-2.el9.x86_64.rpm 55 kB/s | 35 kB 00:00 2024-04-21T08:38:10.791 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T08:38:10.805 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:38:10.844 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:38:10.880 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T08:38:10.908 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T08:38:10.941 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T08:38:11.255 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T08:38:11.304 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:38:11.396 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:38:11.415 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:38:11.425 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 98/152 2024-04-21T08:38:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:38:11.462 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T08:38:11.539 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:38:11.539 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T08:38:11.539 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T08:38:11.539 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:38:11.573 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:38:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T08:38:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:38:12.054 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:38:12.084 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 102/152 2024-04-21T08:38:12.329 INFO:teuthology.orchestra.run.smithi152.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 630 kB/s | 1.6 MB 00:02 2024-04-21T08:38:14.088 INFO:teuthology.orchestra.run.smithi152.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 518 kB/s | 911 kB 00:01 2024-04-21T08:38:14.345 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:38:14.381 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:38:14.871 INFO:teuthology.orchestra.run.smithi152.stdout:(69/150): postgresql-private-libs-13.14-1.el9.x 170 kB/s | 133 kB 00:00 2024-04-21T08:38:14.929 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:38:14.967 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:38:15.059 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:38:15.059 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T08:38:15.059 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T08:38:15.059 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:38:15.085 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:38:15.263 INFO:teuthology.orchestra.run.smithi152.stdout:(70/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 718 kB/s | 4.1 MB 00:05 2024-04-21T08:38:15.930 INFO:teuthology.orchestra.run.smithi152.stdout:(71/150): openblas-openmp-0.3.21-2.el9.x86_64.r 734 kB/s | 4.7 MB 00:06 2024-04-21T08:38:16.630 INFO:teuthology.orchestra.run.smithi152.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 125 kB/s | 87 kB 00:00 2024-04-21T08:38:17.106 INFO:teuthology.orchestra.run.smithi152.stdout:(73/150): protobuf-3.14.0-13.el9.x86_64.rpm 561 kB/s | 1.0 MB 00:01 2024-04-21T08:38:18.031 INFO:teuthology.orchestra.run.smithi152.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 264 kB/s | 244 kB 00:00 2024-04-21T08:38:18.964 INFO:teuthology.orchestra.run.smithi152.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 266 kB/s | 248 kB 00:00 2024-04-21T08:38:19.540 INFO:teuthology.orchestra.run.smithi152.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 45 kB/s | 26 kB 00:00 2024-04-21T08:38:20.082 INFO:teuthology.orchestra.run.smithi152.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 35 kB/s | 19 kB 00:00 2024-04-21T08:38:20.923 INFO:teuthology.orchestra.run.smithi152.stdout:(78/150): python3-mako-1.1.4-6.el9.noarch.rpm 205 kB/s | 172 kB 00:00 2024-04-21T08:38:21.532 INFO:teuthology.orchestra.run.smithi152.stdout:(79/150): python3-markupsafe-1.1.1-12.el9.x86_6 57 kB/s | 35 kB 00:00 2024-04-21T08:38:22.582 INFO:teuthology.orchestra.run.smithi152.stdout:(80/150): postgresql-server-13.14-1.el9.x86_64. 767 kB/s | 5.8 MB 00:07 2024-04-21T08:38:23.341 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:38:23.341 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-04-21T08:38:23.341 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-04-21T08:38:23.341 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-04-21T08:38:23.341 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-04-21T08:38:23.342 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-04-21T08:38:23.342 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-04-21T08:38:23.342 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-04-21T08:38:23.342 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:38:23.528 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:38:23.541 INFO:teuthology.orchestra.run.smithi152.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 262 kB/s | 250 kB 00:00 2024-04-21T08:38:23.545 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:38:23.545 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:23.545 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T08:38:23.545 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:38:24.449 INFO:teuthology.orchestra.run.smithi152.stdout:(82/150): python3-oauthlib-3.1.1-5.el9.noarch.r 245 kB/s | 222 kB 00:00 2024-04-21T08:38:25.075 INFO:teuthology.orchestra.run.smithi152.stdout:(83/150): python3-babel-2.9.1-2.el9.noarch.rpm 723 kB/s | 6.0 MB 00:08 2024-04-21T08:38:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:38:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:38:25.109 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:25.109 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T08:38:25.109 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:38:25.109 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:38:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:38:25.150 INFO:teuthology.orchestra.run.smithi152.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 110 kB/s | 77 kB 00:00 2024-04-21T08:38:25.340 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:38:25.364 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:38:25.364 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:25.364 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T08:38:25.365 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:38:25.365 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:38:25.365 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:38:25.701 INFO:teuthology.orchestra.run.smithi152.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 68 kB/s | 42 kB 00:00 2024-04-21T08:38:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:38:25.857 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:38:25.891 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:25.891 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T08:38:25.891 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:38:25.891 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:38:25.891 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:38:25.964 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T08:38:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T08:38:26.077 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T08:38:26.092 INFO:teuthology.orchestra.run.smithi152.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 286 kB/s | 269 kB 00:00 2024-04-21T08:38:26.276 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T08:38:26.325 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T08:38:26.407 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T08:38:26.497 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T08:38:26.559 INFO:teuthology.orchestra.run.smithi152.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 185 kB/s | 159 kB 00:00 2024-04-21T08:38:26.640 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T08:38:26.684 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T08:38:26.715 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T08:38:26.719 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:38:26.720 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /sys 2024-04-21T08:38:26.720 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /proc 2024-04-21T08:38:26.720 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /mnt 2024-04-21T08:38:26.720 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /var/tmp 2024-04-21T08:38:26.720 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /home 2024-04-21T08:38:26.720 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /root 2024-04-21T08:38:26.720 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /tmp 2024-04-21T08:38:26.720 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:38:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T08:38:26.897 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:38:26.912 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:38:26.912 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:26.912 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T08:38:26.912 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:38:27.035 INFO:teuthology.orchestra.run.smithi152.stdout:(88/150): python3-pyasn1-modules-0.4.8-6.el9.no 296 kB/s | 279 kB 00:00 2024-04-21T08:38:27.113 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:38:27.141 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:38:27.174 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 122/152 2024-04-21T08:38:27.193 INFO:teuthology.orchestra.run.smithi152.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 80 kB/s | 51 kB 00:00 2024-04-21T08:38:27.238 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 123/152 2024-04-21T08:38:27.264 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:38:27.693 INFO:teuthology.orchestra.run.smithi152.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 81 kB/s | 54 kB 00:00 2024-04-21T08:38:28.310 INFO:teuthology.orchestra.run.smithi152.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 68 kB/s | 42 kB 00:00 2024-04-21T08:38:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:38:28.376 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:38:28.401 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:38:28.402 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:28.402 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T08:38:28.402 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:38:28.402 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:38:28.402 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:38:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:38:28.544 INFO:teuthology.orchestra.run.smithi152.stdout:(92/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 744 kB/s | 5.1 MB 00:07 2024-04-21T08:38:28.626 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:38:28.649 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:38:28.649 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:28.649 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T08:38:28.649 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:38:28.649 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:38:28.649 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:38:28.944 INFO:teuthology.orchestra.run.smithi152.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 70 kB/s | 44 kB 00:00 2024-04-21T08:38:29.094 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:38:29.117 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:38:29.117 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:29.117 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T08:38:29.117 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:38:29.117 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:38:29.117 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:38:29.224 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T08:38:29.281 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T08:38:29.336 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T08:38:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:38:29.428 INFO:teuthology.orchestra.run.smithi152.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 258 kB/s | 221 kB 00:00 2024-04-21T08:38:29.450 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:38:29.523 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T08:38:29.560 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:38:29.561 INFO:teuthology.orchestra.run.smithi152.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 58 kB/s | 36 kB 00:00 2024-04-21T08:38:29.569 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T08:38:29.648 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T08:38:29.729 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 127/152 2024-04-21T08:38:29.746 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T08:38:29.769 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:38:29.793 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:38:29.793 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:29.793 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T08:38:29.794 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:38:29.794 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:38:29.794 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:38:29.845 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:38:29.856 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:38:29.887 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T08:38:29.928 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T08:38:29.957 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 130/152 2024-04-21T08:38:29.960 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T08:38:29.996 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T08:38:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 131/152 2024-04-21T08:38:30.087 INFO:teuthology.orchestra.run.smithi152.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 72 kB/s | 47 kB 00:00 2024-04-21T08:38:30.458 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:38:30.486 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:38:30.518 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 122/152 2024-04-21T08:38:30.553 INFO:teuthology.orchestra.run.smithi152.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 307 kB/s | 305 kB 00:00 2024-04-21T08:38:30.637 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 123/152 2024-04-21T08:38:30.667 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:38:30.737 INFO:teuthology.orchestra.run.smithi152.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 89 kB/s | 58 kB 00:00 2024-04-21T08:38:30.978 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T08:38:31.005 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:38:31.030 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:38:31.031 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:31.031 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T08:38:31.031 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:38:31.031 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:38:31.031 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:38:31.073 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:38:31.097 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:38:31.097 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:31.097 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T08:38:31.097 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:38:31.148 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:38:31.173 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:38:31.173 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:31.173 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T08:38:31.173 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:38:31.173 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:38:31.173 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:38:31.271 INFO:teuthology.orchestra.run.smithi152.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 131 kB/s | 94 kB 00:00 2024-04-21T08:38:31.396 INFO:teuthology.orchestra.run.smithi152.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 97 kB/s | 64 kB 00:00 2024-04-21T08:38:31.499 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:38:31.524 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:38:31.524 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:31.524 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T08:38:31.524 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:38:31.524 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:38:31.524 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:38:31.588 INFO:teuthology.orchestra.run.smithi152.stdout:(101/150): lua-devel-5.4.4-4.el9.x86_64.rpm 117 kB/s | 22 kB 00:00 2024-04-21T08:38:31.714 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:38:31.738 INFO:teuthology.orchestra.run.smithi152.stdout:(102/150): protobuf-compiler-3.14.0-13.el9.x86_ 5.6 MB/s | 863 kB 00:00 2024-04-21T08:38:31.753 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:38:31.905 INFO:teuthology.orchestra.run.smithi152.stdout:(103/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 71 kB/s | 45 kB 00:00 2024-04-21T08:38:32.230 INFO:teuthology.orchestra.run.smithi152.stdout:(104/150): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-21T08:38:32.256 INFO:teuthology.orchestra.run.smithi152.stdout:(105/150): gperftools-libs-2.9.1-2.el9.x86_64.r 882 kB/s | 309 kB 00:00 2024-04-21T08:38:32.297 INFO:teuthology.orchestra.run.smithi152.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-04-21T08:38:32.448 INFO:teuthology.orchestra.run.smithi152.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 173 kB/s | 26 kB 00:00 2024-04-21T08:38:32.532 INFO:teuthology.orchestra.run.smithi152.stdout:(108/150): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-21T08:38:32.557 INFO:teuthology.orchestra.run.smithi152.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 451 kB/s | 49 kB 00:00 2024-04-21T08:38:32.607 INFO:teuthology.orchestra.run.smithi152.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 895 kB/s | 67 kB 00:00 2024-04-21T08:38:32.633 INFO:teuthology.orchestra.run.smithi152.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-21T08:38:32.692 INFO:teuthology.orchestra.run.smithi152.stdout:(112/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.9 MB/s | 838 kB 00:00 2024-04-21T08:38:32.717 INFO:teuthology.orchestra.run.smithi152.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-04-21T08:38:32.718 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:38:32.753 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:38:32.758 INFO:teuthology.orchestra.run.smithi152.stdout:(114/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 657 kB/s | 43 kB 00:00 2024-04-21T08:38:32.784 INFO:teuthology.orchestra.run.smithi152.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 480 kB/s | 32 kB 00:00 2024-04-21T08:38:32.826 INFO:teuthology.orchestra.run.smithi152.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 223 kB/s | 15 kB 00:00 2024-04-21T08:38:32.857 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:38:32.860 INFO:teuthology.orchestra.run.smithi152.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-21T08:38:32.902 INFO:teuthology.orchestra.run.smithi152.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-21T08:38:32.935 INFO:teuthology.orchestra.run.smithi152.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-04-21T08:38:32.969 INFO:teuthology.orchestra.run.smithi152.stdout:(120/150): python3-colorama-0.4.4-12.el9.noarch 527 kB/s | 35 kB 00:00 2024-04-21T08:38:33.011 INFO:teuthology.orchestra.run.smithi152.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-04-21T08:38:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 127/152 2024-04-21T08:38:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:38:33.070 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:38:33.071 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:33.071 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T08:38:33.071 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:38:33.071 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:38:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:38:33.086 INFO:teuthology.orchestra.run.smithi152.stdout:(122/150): python3-grpcio-1.46.7-10.el9.x86_64. 17 MB/s | 2.0 MB 00:00 2024-04-21T08:38:33.111 INFO:teuthology.orchestra.run.smithi152.stdout:(123/150): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-04-21T08:38:33.123 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:38:33.134 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:38:33.145 INFO:teuthology.orchestra.run.smithi152.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-04-21T08:38:33.170 INFO:teuthology.orchestra.run.smithi152.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 301 kB/s | 18 kB 00:00 2024-04-21T08:38:33.204 INFO:teuthology.orchestra.run.smithi152.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 395 kB/s | 23 kB 00:00 2024-04-21T08:38:33.230 INFO:teuthology.orchestra.run.smithi152.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 332 kB/s | 19 kB 00:00 2024-04-21T08:38:33.231 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 130/152 2024-04-21T08:38:33.263 INFO:teuthology.orchestra.run.smithi152.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 335 kB/s | 20 kB 00:00 2024-04-21T08:38:33.279 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 131/152 2024-04-21T08:38:33.321 INFO:teuthology.orchestra.run.smithi152.stdout:(129/150): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-04-21T08:38:33.347 INFO:teuthology.orchestra.run.smithi152.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 554 kB/s | 46 kB 00:00 2024-04-21T08:38:33.389 INFO:teuthology.orchestra.run.smithi152.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-04-21T08:38:33.414 INFO:teuthology.orchestra.run.smithi152.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 863 kB/s | 58 kB 00:00 2024-04-21T08:38:33.464 INFO:teuthology.orchestra.run.smithi152.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-21T08:38:33.490 INFO:teuthology.orchestra.run.smithi152.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-04-21T08:38:33.532 INFO:teuthology.orchestra.run.smithi152.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-04-21T08:38:33.557 INFO:teuthology.orchestra.run.smithi152.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 422 kB/s | 28 kB 00:00 2024-04-21T08:38:33.599 INFO:teuthology.orchestra.run.smithi152.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 458 kB/s | 31 kB 00:00 2024-04-21T08:38:33.633 INFO:teuthology.orchestra.run.smithi152.stdout:(138/150): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-21T08:38:33.666 INFO:teuthology.orchestra.run.smithi152.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 882 kB/s | 59 kB 00:00 2024-04-21T08:38:33.700 INFO:teuthology.orchestra.run.smithi152.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 533 kB/s | 36 kB 00:00 2024-04-21T08:38:33.734 INFO:teuthology.orchestra.run.smithi152.stdout:(141/150): python3-typing-extensions-4.4.0-2.el 760 kB/s | 51 kB 00:00 2024-04-21T08:38:33.776 INFO:teuthology.orchestra.run.smithi152.stdout:(142/150): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-21T08:38:33.793 INFO:teuthology.orchestra.run.smithi152.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-04-21T08:38:33.852 INFO:teuthology.orchestra.run.smithi152.stdout:(144/150): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-04-21T08:38:33.877 INFO:teuthology.orchestra.run.smithi152.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 239 kB/s | 20 kB 00:00 2024-04-21T08:38:33.927 INFO:teuthology.orchestra.run.smithi152.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-21T08:38:33.978 INFO:teuthology.orchestra.run.smithi152.stdout:(147/150): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-21T08:38:34.203 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T08:38:34.233 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:38:34.256 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:38:34.256 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:34.256 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T08:38:34.256 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:38:34.256 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:38:34.256 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:38:34.294 INFO:teuthology.orchestra.run.smithi152.stdout:(148/150): librados2-19.0.0-3154.ga630765c.el9. 9.3 MB/s | 3.4 MB 00:00 2024-04-21T08:38:34.301 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:38:34.322 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:38:34.323 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:34.323 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T08:38:34.323 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:38:34.353 INFO:teuthology.orchestra.run.smithi152.stdout:(149/150): librbd1-19.0.0-3154.ga630765c.el9.x8 8.4 MB/s | 3.1 MB 00:00 2024-04-21T08:38:34.368 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:38:34.390 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:38:34.390 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:34.390 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T08:38:34.390 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:38:34.391 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:38:34.391 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:38:34.694 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:38:34.716 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:38:34.716 INFO:teuthology.orchestra.run.smithi171.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:38:34.716 INFO:teuthology.orchestra.run.smithi171.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T08:38:34.716 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:38:34.716 INFO:teuthology.orchestra.run.smithi171.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:38:34.717 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:38:36.370 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 137/152 2024-04-21T08:38:36.412 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 138/152 2024-04-21T08:38:36.453 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 139/152 2024-04-21T08:38:36.476 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T08:38:36.503 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T08:38:36.707 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 142/152 2024-04-21T08:38:36.749 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T08:38:36.779 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 144/152 2024-04-21T08:38:37.373 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:38:37.418 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:38:37.440 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:38:38.297 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 146/152 2024-04-21T08:38:38.373 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T08:38:38.415 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T08:38:38.502 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T08:38:38.535 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T08:38:38.535 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:38:38.627 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:38:38.628 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:38:39.306 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 137/152 2024-04-21T08:38:39.347 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 138/152 2024-04-21T08:38:39.388 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 139/152 2024-04-21T08:38:39.428 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T08:38:39.456 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T08:38:39.641 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 142/152 2024-04-21T08:38:39.693 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T08:38:39.723 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 144/152 2024-04-21T08:38:40.348 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:38:40.396 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:38:40.415 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:38:41.224 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 146/152 2024-04-21T08:38:41.309 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T08:38:41.350 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T08:38:41.433 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T08:38:41.463 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T08:38:41.463 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:38:41.546 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:38:41.547 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:38:44.382 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:38:44.382 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 1/152 2024-04-21T08:38:44.382 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2/152 2024-04-21T08:38:44.382 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 3/152 2024-04-21T08:38:44.382 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 4/152 2024-04-21T08:38:44.382 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 5/152 2024-04-21T08:38:44.382 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:38:44.383 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:38:44.383 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 8/152 2024-04-21T08:38:44.383 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 9/152 2024-04-21T08:38:44.383 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 10/152 2024-04-21T08:38:44.383 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 11/152 2024-04-21T08:38:44.383 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:38:44.383 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 13/152 2024-04-21T08:38:44.383 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 14/152 2024-04-21T08:38:44.384 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 15/152 2024-04-21T08:38:44.384 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 16/152 2024-04-21T08:38:44.384 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 17/152 2024-04-21T08:38:44.384 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 18/152 2024-04-21T08:38:44.384 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 19/152 2024-04-21T08:38:44.384 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 20/152 2024-04-21T08:38:44.384 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 21/152 2024-04-21T08:38:44.384 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 22/152 2024-04-21T08:38:44.384 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 23/152 2024-04-21T08:38:44.385 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 24/152 2024-04-21T08:38:44.385 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 25/152 2024-04-21T08:38:44.385 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 26/152 2024-04-21T08:38:44.385 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 27/152 2024-04-21T08:38:44.385 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 28/152 2024-04-21T08:38:44.385 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 29/152 2024-04-21T08:38:44.386 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 30/152 2024-04-21T08:38:44.386 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 31/152 2024-04-21T08:38:44.386 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 32/152 2024-04-21T08:38:44.386 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 33/152 2024-04-21T08:38:44.386 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 34/152 2024-04-21T08:38:44.386 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 35/152 2024-04-21T08:38:44.386 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 36/152 2024-04-21T08:38:44.386 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 37/152 2024-04-21T08:38:44.386 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 38/152 2024-04-21T08:38:44.386 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 39/152 2024-04-21T08:38:44.387 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T08:38:44.387 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T08:38:44.387 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T08:38:44.387 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T08:38:44.387 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T08:38:44.387 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T08:38:44.387 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T08:38:44.387 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T08:38:44.387 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T08:38:44.387 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T08:38:44.387 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T08:38:44.387 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T08:38:44.387 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T08:38:44.387 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T08:38:44.387 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T08:38:44.388 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T08:38:44.388 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T08:38:44.388 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T08:38:44.388 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T08:38:44.388 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T08:38:44.388 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T08:38:44.388 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T08:38:44.388 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T08:38:44.388 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T08:38:44.388 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T08:38:44.389 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T08:38:44.389 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T08:38:44.389 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T08:38:44.389 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T08:38:44.389 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T08:38:44.389 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T08:38:44.389 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T08:38:44.389 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T08:38:44.389 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T08:38:44.389 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T08:38:44.389 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T08:38:44.389 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T08:38:44.389 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T08:38:44.389 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T08:38:44.390 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T08:38:44.390 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T08:38:44.390 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T08:38:44.390 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T08:38:44.390 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T08:38:44.390 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T08:38:44.390 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T08:38:44.390 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T08:38:44.390 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T08:38:44.390 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T08:38:44.390 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T08:38:44.390 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T08:38:44.390 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T08:38:44.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T08:38:44.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T08:38:44.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T08:38:44.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T08:38:44.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T08:38:44.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T08:38:44.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T08:38:44.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T08:38:44.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T08:38:44.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T08:38:44.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T08:38:44.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T08:38:44.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T08:38:44.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T08:38:44.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T08:38:44.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T08:38:44.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T08:38:44.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T08:38:44.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T08:38:44.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T08:38:44.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T08:38:44.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T08:38:44.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T08:38:44.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T08:38:44.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T08:38:44.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T08:38:44.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T08:38:44.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T08:38:44.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T08:38:44.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T08:38:44.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T08:38:44.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T08:38:44.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T08:38:44.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T08:38:44.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T08:38:44.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T08:38:44.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T08:38:44.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T08:38:44.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T08:38:44.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T08:38:44.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T08:38:44.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T08:38:44.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T08:38:44.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T08:38:44.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T08:38:44.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T08:38:44.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T08:38:44.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T08:38:44.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T08:38:44.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T08:38:44.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T08:38:44.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T08:38:44.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T08:38:44.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T08:38:44.395 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T08:38:44.395 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T08:38:44.395 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T08:38:44.395 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 149/152 2024-04-21T08:38:44.395 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T08:38:44.395 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 151/152 2024-04-21T08:38:46.521 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:38:46.521 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:38:46.521 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-04-21T08:38:46.521 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.522 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.522 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-21T08:38:46.522 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T08:38:46.522 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T08:38:46.522 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T08:38:46.522 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T08:38:46.522 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.522 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.522 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.522 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.522 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:46.522 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.522 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.522 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.523 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:46.523 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:46.523 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:46.523 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:46.523 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:46.523 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.523 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.523 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:46.523 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.523 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.523 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.523 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:46.523 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:46.523 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T08:38:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T08:38:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T08:38:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T08:38:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T08:38:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T08:38:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T08:38:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T08:38:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T08:38:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T08:38:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T08:38:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.525 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.525 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.525 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T08:38:46.525 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T08:38:46.525 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T08:38:46.525 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T08:38:46.525 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T08:38:46.525 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.525 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.525 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T08:38:46.525 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.525 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T08:38:46.525 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T08:38:46.525 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T08:38:46.525 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T08:38:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T08:38:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T08:38:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T08:38:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T08:38:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T08:38:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T08:38:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T08:38:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T08:38:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T08:38:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T08:38:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T08:38:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T08:38:46.526 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T08:38:46.527 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T08:38:46.527 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T08:38:46.527 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T08:38:46.527 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T08:38:46.527 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T08:38:46.527 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T08:38:46.527 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.527 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.527 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.527 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T08:38:46.527 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T08:38:46.528 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T08:38:46.528 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T08:38:46.528 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T08:38:46.528 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T08:38:46.528 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T08:38:46.528 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T08:38:46.528 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T08:38:46.528 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T08:38:46.528 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T08:38:46.528 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T08:38:46.528 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T08:38:46.528 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T08:38:46.528 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T08:38:46.528 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T08:38:46.528 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T08:38:46.529 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T08:38:46.529 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T08:38:46.529 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T08:38:46.529 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T08:38:46.529 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T08:38:46.529 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T08:38:46.529 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T08:38:46.529 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T08:38:46.529 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T08:38:46.529 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T08:38:46.529 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T08:38:46.529 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T08:38:46.529 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T08:38:46.530 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T08:38:46.530 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T08:38:46.530 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T08:38:46.530 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T08:38:46.530 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T08:38:46.530 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T08:38:46.530 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T08:38:46.530 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T08:38:46.530 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T08:38:46.530 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T08:38:46.530 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.530 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.530 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T08:38:46.530 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T08:38:46.531 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T08:38:46.531 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.531 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T08:38:46.531 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T08:38:46.531 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T08:38:46.531 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T08:38:46.531 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T08:38:46.531 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T08:38:46.531 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T08:38:46.531 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T08:38:46.531 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T08:38:46.531 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T08:38:46.531 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T08:38:46.531 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T08:38:46.531 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T08:38:46.532 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T08:38:46.532 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T08:38:46.532 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.532 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.532 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:46.532 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T08:38:46.532 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T08:38:46.532 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T08:38:46.532 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T08:38:46.532 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T08:38:46.532 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T08:38:46.533 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T08:38:46.533 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:38:46.533 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-21T08:38:46.902 DEBUG:teuthology.parallel:result is None 2024-04-21T08:38:47.651 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:38:47.651 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 1/152 2024-04-21T08:38:47.651 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2/152 2024-04-21T08:38:47.651 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 3/152 2024-04-21T08:38:47.651 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 4/152 2024-04-21T08:38:47.651 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 5/152 2024-04-21T08:38:47.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:38:47.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:38:47.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 8/152 2024-04-21T08:38:47.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 9/152 2024-04-21T08:38:47.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 10/152 2024-04-21T08:38:47.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 11/152 2024-04-21T08:38:47.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:38:47.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 13/152 2024-04-21T08:38:47.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 14/152 2024-04-21T08:38:47.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 15/152 2024-04-21T08:38:47.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 16/152 2024-04-21T08:38:47.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 17/152 2024-04-21T08:38:47.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 18/152 2024-04-21T08:38:47.652 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 19/152 2024-04-21T08:38:47.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 20/152 2024-04-21T08:38:47.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 21/152 2024-04-21T08:38:47.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 22/152 2024-04-21T08:38:47.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 23/152 2024-04-21T08:38:47.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 24/152 2024-04-21T08:38:47.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 25/152 2024-04-21T08:38:47.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 26/152 2024-04-21T08:38:47.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 27/152 2024-04-21T08:38:47.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 28/152 2024-04-21T08:38:47.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 29/152 2024-04-21T08:38:47.653 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 30/152 2024-04-21T08:38:47.664 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 31/152 2024-04-21T08:38:47.665 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 32/152 2024-04-21T08:38:47.665 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 33/152 2024-04-21T08:38:47.665 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 34/152 2024-04-21T08:38:47.665 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 35/152 2024-04-21T08:38:47.665 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 36/152 2024-04-21T08:38:47.665 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 37/152 2024-04-21T08:38:47.665 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 38/152 2024-04-21T08:38:47.665 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 39/152 2024-04-21T08:38:47.665 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T08:38:47.665 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T08:38:47.665 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T08:38:47.665 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T08:38:47.666 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T08:38:47.666 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T08:38:47.666 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T08:38:47.666 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T08:38:47.666 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T08:38:47.666 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T08:38:47.666 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T08:38:47.666 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T08:38:47.666 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T08:38:47.666 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T08:38:47.666 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T08:38:47.666 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T08:38:47.666 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T08:38:47.666 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T08:38:47.666 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T08:38:47.667 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T08:38:47.667 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T08:38:47.667 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T08:38:47.667 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T08:38:47.667 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T08:38:47.667 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T08:38:47.667 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T08:38:47.667 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T08:38:47.667 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T08:38:47.667 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T08:38:47.667 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T08:38:47.667 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T08:38:47.667 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T08:38:47.667 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T08:38:47.668 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T08:38:47.668 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T08:38:47.668 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T08:38:47.668 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T08:38:47.668 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T08:38:47.668 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T08:38:47.668 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T08:38:47.668 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T08:38:47.668 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T08:38:47.668 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T08:38:47.668 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T08:38:47.668 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T08:38:47.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T08:38:47.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T08:38:47.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T08:38:47.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T08:38:47.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T08:38:47.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T08:38:47.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T08:38:47.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T08:38:47.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T08:38:47.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T08:38:47.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T08:38:47.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T08:38:47.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T08:38:47.669 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T08:38:47.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T08:38:47.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T08:38:47.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T08:38:47.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T08:38:47.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T08:38:47.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T08:38:47.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T08:38:47.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T08:38:47.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T08:38:47.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T08:38:47.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T08:38:47.670 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T08:38:47.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T08:38:47.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T08:38:47.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T08:38:47.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T08:38:47.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T08:38:47.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T08:38:47.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T08:38:47.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T08:38:47.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T08:38:47.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T08:38:47.671 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T08:38:47.672 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T08:38:47.672 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T08:38:47.672 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T08:38:47.672 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T08:38:47.672 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T08:38:47.672 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T08:38:47.672 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T08:38:47.672 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T08:38:47.672 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T08:38:47.672 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T08:38:47.672 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T08:38:47.672 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T08:38:47.672 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T08:38:47.672 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T08:38:47.673 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T08:38:47.673 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T08:38:47.673 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T08:38:47.673 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T08:38:47.673 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T08:38:47.673 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T08:38:47.673 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T08:38:47.673 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T08:38:47.673 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T08:38:47.673 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T08:38:47.673 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T08:38:47.673 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T08:38:47.673 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T08:38:47.673 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 149/152 2024-04-21T08:38:47.674 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T08:38:47.674 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 151/152 2024-04-21T08:38:48.010 INFO:teuthology.orchestra.run.smithi152.stdout:(150/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 799 kB/s | 16 MB 00:20 2024-04-21T08:38:48.016 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:38:48.016 INFO:teuthology.orchestra.run.smithi152.stdout:Total 4.1 MB/s | 218 MB 00:52 2024-04-21T08:38:49.022 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-21T08:38:49.147 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-21T08:38:49.148 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-21T08:38:49.770 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:38:49.770 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:38:49.770 INFO:teuthology.orchestra.run.smithi171.stdout:Upgraded: 2024-04-21T08:38:49.770 INFO:teuthology.orchestra.run.smithi171.stdout: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.770 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.771 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-04-21T08:38:49.771 INFO:teuthology.orchestra.run.smithi171.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T08:38:49.771 INFO:teuthology.orchestra.run.smithi171.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T08:38:49.771 INFO:teuthology.orchestra.run.smithi171.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T08:38:49.771 INFO:teuthology.orchestra.run.smithi171.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T08:38:49.771 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.771 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.771 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.771 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.771 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:49.771 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.771 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.771 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.772 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:49.772 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:49.772 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:49.772 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:49.772 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:49.772 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.772 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.772 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:49.772 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.772 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.772 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.773 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:49.773 INFO:teuthology.orchestra.run.smithi171.stdout: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:49.773 INFO:teuthology.orchestra.run.smithi171.stdout: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.773 INFO:teuthology.orchestra.run.smithi171.stdout: cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.773 INFO:teuthology.orchestra.run.smithi171.stdout: cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:38:49.773 INFO:teuthology.orchestra.run.smithi171.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T08:38:49.773 INFO:teuthology.orchestra.run.smithi171.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T08:38:49.773 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T08:38:49.773 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T08:38:49.773 INFO:teuthology.orchestra.run.smithi171.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T08:38:49.773 INFO:teuthology.orchestra.run.smithi171.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T08:38:49.773 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T08:38:49.774 INFO:teuthology.orchestra.run.smithi171.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T08:38:49.774 INFO:teuthology.orchestra.run.smithi171.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T08:38:49.774 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T08:38:49.774 INFO:teuthology.orchestra.run.smithi171.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T08:38:49.774 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.774 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.774 INFO:teuthology.orchestra.run.smithi171.stdout: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.774 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T08:38:49.774 INFO:teuthology.orchestra.run.smithi171.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T08:38:49.774 INFO:teuthology.orchestra.run.smithi171.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T08:38:49.774 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T08:38:49.774 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T08:38:49.775 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.775 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.775 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T08:38:49.775 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.775 INFO:teuthology.orchestra.run.smithi171.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T08:38:49.775 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T08:38:49.775 INFO:teuthology.orchestra.run.smithi171.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T08:38:49.775 INFO:teuthology.orchestra.run.smithi171.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T08:38:49.775 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T08:38:49.775 INFO:teuthology.orchestra.run.smithi171.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T08:38:49.775 INFO:teuthology.orchestra.run.smithi171.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T08:38:49.776 INFO:teuthology.orchestra.run.smithi171.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T08:38:49.776 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T08:38:49.776 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T08:38:49.776 INFO:teuthology.orchestra.run.smithi171.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T08:38:49.776 INFO:teuthology.orchestra.run.smithi171.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T08:38:49.776 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T08:38:49.776 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T08:38:49.776 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T08:38:49.776 INFO:teuthology.orchestra.run.smithi171.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T08:38:49.776 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T08:38:49.776 INFO:teuthology.orchestra.run.smithi171.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T08:38:49.776 INFO:teuthology.orchestra.run.smithi171.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T08:38:49.777 INFO:teuthology.orchestra.run.smithi171.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T08:38:49.777 INFO:teuthology.orchestra.run.smithi171.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T08:38:49.777 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T08:38:49.777 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T08:38:49.777 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.777 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.777 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.777 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T08:38:49.777 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T08:38:49.777 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T08:38:49.777 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T08:38:49.777 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T08:38:49.777 INFO:teuthology.orchestra.run.smithi171.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T08:38:49.778 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T08:38:49.778 INFO:teuthology.orchestra.run.smithi171.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T08:38:49.778 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T08:38:49.778 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T08:38:49.778 INFO:teuthology.orchestra.run.smithi171.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T08:38:49.778 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T08:38:49.778 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T08:38:49.778 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T08:38:49.778 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T08:38:49.778 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T08:38:49.778 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T08:38:49.778 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T08:38:49.779 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T08:38:49.779 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T08:38:49.779 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T08:38:49.779 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T08:38:49.779 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T08:38:49.779 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T08:38:49.779 INFO:teuthology.orchestra.run.smithi171.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T08:38:49.779 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T08:38:49.779 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T08:38:49.779 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T08:38:49.779 INFO:teuthology.orchestra.run.smithi171.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T08:38:49.779 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T08:38:49.780 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T08:38:49.780 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T08:38:49.780 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T08:38:49.780 INFO:teuthology.orchestra.run.smithi171.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T08:38:49.780 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T08:38:49.780 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T08:38:49.780 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T08:38:49.780 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T08:38:49.780 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T08:38:49.780 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T08:38:49.780 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.780 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.781 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T08:38:49.781 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T08:38:49.781 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T08:38:49.781 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.781 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T08:38:49.781 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T08:38:49.781 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T08:38:49.781 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T08:38:49.781 INFO:teuthology.orchestra.run.smithi171.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T08:38:49.781 INFO:teuthology.orchestra.run.smithi171.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T08:38:49.781 INFO:teuthology.orchestra.run.smithi171.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T08:38:49.781 INFO:teuthology.orchestra.run.smithi171.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T08:38:49.781 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T08:38:49.781 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T08:38:49.782 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T08:38:49.782 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T08:38:49.782 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T08:38:49.782 INFO:teuthology.orchestra.run.smithi171.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T08:38:49.782 INFO:teuthology.orchestra.run.smithi171.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T08:38:49.782 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.782 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.782 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:38:49.782 INFO:teuthology.orchestra.run.smithi171.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T08:38:49.782 INFO:teuthology.orchestra.run.smithi171.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T08:38:49.782 INFO:teuthology.orchestra.run.smithi171.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T08:38:49.782 INFO:teuthology.orchestra.run.smithi171.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T08:38:49.783 INFO:teuthology.orchestra.run.smithi171.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T08:38:49.783 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T08:38:49.783 INFO:teuthology.orchestra.run.smithi171.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T08:38:49.783 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:38:49.783 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-04-21T08:38:50.165 DEBUG:teuthology.parallel:result is None 2024-04-21T08:38:50.860 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-21T08:38:50.861 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-21T08:38:56.610 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-21T08:38:56.696 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T08:38:56.721 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T08:38:56.763 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T08:38:57.092 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T08:38:57.239 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:38:57.359 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:38:57.406 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:38:57.497 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:38:57.539 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:38:57.569 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T08:38:57.595 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T08:38:57.745 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T08:38:57.792 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T08:38:57.820 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:38:57.923 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:38:57.947 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:38:58.009 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:38:58.031 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:38:58.052 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:38:58.248 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:38:58.282 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:38:58.388 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T08:38:58.484 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T08:38:58.522 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T08:38:58.557 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T08:38:58.626 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T08:38:58.665 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T08:38:58.717 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T08:38:58.749 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T08:38:58.778 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T08:38:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T08:38:58.909 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T08:38:58.939 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T08:38:58.983 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T08:38:59.061 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T08:38:59.094 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 30/152 2024-04-21T08:38:59.138 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 31/152 2024-04-21T08:38:59.166 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 32/152 2024-04-21T08:38:59.239 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T08:38:59.290 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T08:38:59.365 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T08:38:59.501 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T08:38:59.559 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T08:39:00.090 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T08:39:00.188 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T08:39:00.225 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T08:39:00.268 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T08:39:00.306 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 42/152 2024-04-21T08:39:00.339 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T08:39:00.417 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T08:39:00.493 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T08:39:00.524 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T08:39:00.561 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T08:39:00.607 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T08:39:00.643 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T08:39:00.687 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T08:39:00.728 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T08:39:00.866 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T08:39:00.910 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T08:39:00.940 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T08:39:00.981 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T08:39:01.077 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T08:39:01.123 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T08:39:01.163 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T08:39:01.189 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T08:39:01.214 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T08:39:01.289 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T08:39:01.417 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T08:39:01.455 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T08:39:01.483 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T08:39:01.517 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T08:39:01.561 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T08:39:01.599 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T08:39:01.631 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T08:39:01.677 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T08:39:01.712 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T08:39:01.745 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T08:39:01.787 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T08:39:01.822 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T08:39:01.909 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T08:39:01.969 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T08:39:02.001 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T08:39:02.047 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T08:39:02.147 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T08:39:02.194 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T08:39:02.937 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T08:39:02.983 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T08:39:03.017 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T08:39:03.120 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T08:39:03.176 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T08:39:03.263 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T08:39:03.294 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T08:39:03.472 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T08:39:03.509 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T08:39:03.586 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T08:39:04.243 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T08:39:04.350 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T08:39:05.604 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T08:39:05.699 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T08:39:05.727 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T08:39:05.759 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T08:39:06.072 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T08:39:06.114 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:39:06.196 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:39:06.228 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 98/152 2024-04-21T08:39:06.264 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T08:39:06.741 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T08:39:06.768 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:39:06.839 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:39:06.869 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 102/152 2024-04-21T08:39:09.201 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:39:09.241 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:39:09.784 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:39:09.837 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:39:09.935 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:39:09.935 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T08:39:09.935 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T08:39:09.935 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-21T08:39:09.963 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:39:21.544 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:39:21.544 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /sys 2024-04-21T08:39:21.544 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /proc 2024-04-21T08:39:21.544 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /mnt 2024-04-21T08:39:21.544 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /var/tmp 2024-04-21T08:39:21.544 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /home 2024-04-21T08:39:21.544 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /root 2024-04-21T08:39:21.544 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /tmp 2024-04-21T08:39:21.544 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-21T08:39:21.730 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:39:21.745 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:39:21.745 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:39:21.745 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T08:39:21.745 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-21T08:39:23.246 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:39:23.272 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:39:23.272 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:39:23.272 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T08:39:23.272 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:39:23.272 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:39:23.272 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-21T08:39:23.492 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:39:23.514 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:39:23.514 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:39:23.514 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T08:39:23.515 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:39:23.515 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:39:23.515 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-21T08:39:23.962 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:39:23.982 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:39:23.982 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:39:23.982 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T08:39:23.982 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:39:23.982 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:39:23.982 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-21T08:39:24.090 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T08:39:24.139 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T08:39:24.194 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T08:39:24.387 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T08:39:24.436 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T08:39:24.514 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T08:39:24.602 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T08:39:24.750 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T08:39:24.794 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T08:39:24.827 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T08:39:24.854 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T08:39:25.207 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:39:25.237 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:39:25.269 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 122/152 2024-04-21T08:39:25.339 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 123/152 2024-04-21T08:39:25.368 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:39:26.407 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:39:26.437 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:39:27.406 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:39:27.438 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:39:27.539 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:39:27.710 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 127/152 2024-04-21T08:39:27.740 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:39:27.762 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:39:27.762 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:39:27.763 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T08:39:27.763 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:39:27.763 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:39:27.763 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-21T08:39:27.807 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:39:27.818 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:39:27.916 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 130/152 2024-04-21T08:39:27.964 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 131/152 2024-04-21T08:39:28.890 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T08:39:28.918 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:39:28.940 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:39:28.940 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:39:28.940 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T08:39:28.940 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:39:28.940 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:39:28.940 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-21T08:39:28.986 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:39:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:39:29.007 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:39:29.007 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T08:39:29.007 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-21T08:39:29.070 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:39:29.091 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:39:29.091 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:39:29.092 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T08:39:29.092 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:39:29.092 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:39:29.092 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-21T08:39:29.404 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:39:29.426 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:39:29.426 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:39:29.426 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T08:39:29.426 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:39:29.426 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:39:29.426 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-21T08:39:34.033 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 137/152 2024-04-21T08:39:34.074 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 138/152 2024-04-21T08:39:34.116 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 139/152 2024-04-21T08:39:34.147 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T08:39:34.175 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T08:39:34.358 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 142/152 2024-04-21T08:39:34.405 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T08:39:34.435 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 144/152 2024-04-21T08:39:35.040 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:39:35.085 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:39:35.105 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:39:35.882 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 146/152 2024-04-21T08:39:35.974 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T08:39:36.014 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T08:39:36.097 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T08:39:36.137 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T08:39:36.137 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:39:36.221 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:39:36.221 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:39:42.073 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:39:42.073 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 1/152 2024-04-21T08:39:42.073 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2/152 2024-04-21T08:39:42.073 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 3/152 2024-04-21T08:39:42.073 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 4/152 2024-04-21T08:39:42.073 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 5/152 2024-04-21T08:39:42.073 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:39:42.073 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:39:42.073 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 8/152 2024-04-21T08:39:42.073 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 9/152 2024-04-21T08:39:42.073 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 10/152 2024-04-21T08:39:42.074 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 11/152 2024-04-21T08:39:42.074 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:39:42.074 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 13/152 2024-04-21T08:39:42.074 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 14/152 2024-04-21T08:39:42.074 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 15/152 2024-04-21T08:39:42.074 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 16/152 2024-04-21T08:39:42.074 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 17/152 2024-04-21T08:39:42.074 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 18/152 2024-04-21T08:39:42.074 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 19/152 2024-04-21T08:39:42.074 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 20/152 2024-04-21T08:39:42.074 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 21/152 2024-04-21T08:39:42.074 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 22/152 2024-04-21T08:39:42.074 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 23/152 2024-04-21T08:39:42.074 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 24/152 2024-04-21T08:39:42.075 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 25/152 2024-04-21T08:39:42.075 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 26/152 2024-04-21T08:39:42.075 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 27/152 2024-04-21T08:39:42.075 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 28/152 2024-04-21T08:39:42.075 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 29/152 2024-04-21T08:39:42.075 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 30/152 2024-04-21T08:39:42.086 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 31/152 2024-04-21T08:39:42.086 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 32/152 2024-04-21T08:39:42.086 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 33/152 2024-04-21T08:39:42.086 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 34/152 2024-04-21T08:39:42.086 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 35/152 2024-04-21T08:39:42.086 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 36/152 2024-04-21T08:39:42.086 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 37/152 2024-04-21T08:39:42.086 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 38/152 2024-04-21T08:39:42.086 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 39/152 2024-04-21T08:39:42.086 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T08:39:42.086 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T08:39:42.087 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T08:39:42.087 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T08:39:42.087 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T08:39:42.087 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T08:39:42.087 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T08:39:42.087 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T08:39:42.087 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T08:39:42.087 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T08:39:42.087 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T08:39:42.087 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T08:39:42.087 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T08:39:42.087 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T08:39:42.087 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T08:39:42.087 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T08:39:42.088 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T08:39:42.088 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T08:39:42.088 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T08:39:42.088 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T08:39:42.088 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T08:39:42.088 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T08:39:42.088 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T08:39:42.088 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T08:39:42.088 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T08:39:42.088 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T08:39:42.088 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T08:39:42.088 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T08:39:42.088 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T08:39:42.089 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T08:39:42.089 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T08:39:42.089 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T08:39:42.089 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T08:39:42.089 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T08:39:42.089 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T08:39:42.089 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T08:39:42.089 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T08:39:42.089 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T08:39:42.089 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T08:39:42.089 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T08:39:42.089 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T08:39:42.089 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T08:39:42.089 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T08:39:42.090 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T08:39:42.090 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T08:39:42.090 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T08:39:42.090 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T08:39:42.090 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T08:39:42.090 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T08:39:42.090 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T08:39:42.090 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T08:39:42.090 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T08:39:42.090 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T08:39:42.090 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T08:39:42.090 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T08:39:42.090 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T08:39:42.090 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T08:39:42.091 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T08:39:42.091 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T08:39:42.091 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T08:39:42.091 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T08:39:42.091 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T08:39:42.091 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T08:39:42.091 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T08:39:42.091 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T08:39:42.091 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T08:39:42.091 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T08:39:42.091 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T08:39:42.091 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T08:39:42.091 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T08:39:42.091 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T08:39:42.092 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T08:39:42.092 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T08:39:42.092 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T08:39:42.092 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T08:39:42.092 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T08:39:42.092 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T08:39:42.092 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T08:39:42.092 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T08:39:42.092 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T08:39:42.092 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T08:39:42.092 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T08:39:42.092 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T08:39:42.092 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T08:39:42.092 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T08:39:42.092 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T08:39:42.093 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T08:39:42.093 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T08:39:42.093 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T08:39:42.093 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T08:39:42.093 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T08:39:42.093 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T08:39:42.093 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T08:39:42.093 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T08:39:42.093 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T08:39:42.093 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T08:39:42.093 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T08:39:42.093 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T08:39:42.093 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T08:39:42.094 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T08:39:42.094 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T08:39:42.094 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T08:39:42.094 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T08:39:42.094 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T08:39:42.094 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T08:39:42.094 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T08:39:42.094 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T08:39:42.094 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T08:39:42.094 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T08:39:42.094 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 149/152 2024-04-21T08:39:42.094 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T08:39:42.094 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 151/152 2024-04-21T08:39:44.190 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:39:44.191 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-21T08:39:44.191 INFO:teuthology.orchestra.run.smithi152.stdout:Upgraded: 2024-04-21T08:39:44.191 INFO:teuthology.orchestra.run.smithi152.stdout: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.191 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.191 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-21T08:39:44.192 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T08:39:44.192 INFO:teuthology.orchestra.run.smithi152.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T08:39:44.192 INFO:teuthology.orchestra.run.smithi152.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T08:39:44.192 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T08:39:44.192 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.192 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.193 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.193 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.193 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:39:44.193 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.193 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.194 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.194 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:39:44.194 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:39:44.194 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:39:44.194 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:39:44.194 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:39:44.194 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.194 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.194 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:39:44.195 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.195 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.195 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.195 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:39:44.195 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:39:44.195 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.195 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.196 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:39:44.196 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T08:39:44.196 INFO:teuthology.orchestra.run.smithi152.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T08:39:44.196 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T08:39:44.196 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T08:39:44.196 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T08:39:44.197 INFO:teuthology.orchestra.run.smithi152.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T08:39:44.197 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T08:39:44.197 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T08:39:44.197 INFO:teuthology.orchestra.run.smithi152.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T08:39:44.197 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T08:39:44.198 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T08:39:44.198 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.198 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.198 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.198 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T08:39:44.198 INFO:teuthology.orchestra.run.smithi152.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T08:39:44.198 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T08:39:44.198 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T08:39:44.198 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T08:39:44.198 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.198 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.198 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T08:39:44.199 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.199 INFO:teuthology.orchestra.run.smithi152.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T08:39:44.199 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T08:39:44.199 INFO:teuthology.orchestra.run.smithi152.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T08:39:44.199 INFO:teuthology.orchestra.run.smithi152.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T08:39:44.199 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T08:39:44.199 INFO:teuthology.orchestra.run.smithi152.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T08:39:44.199 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T08:39:44.200 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T08:39:44.200 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T08:39:44.200 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T08:39:44.200 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T08:39:44.200 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T08:39:44.200 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T08:39:44.200 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T08:39:44.200 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T08:39:44.200 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T08:39:44.201 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T08:39:44.201 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T08:39:44.201 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T08:39:44.201 INFO:teuthology.orchestra.run.smithi152.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T08:39:44.201 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T08:39:44.201 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T08:39:44.201 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T08:39:44.201 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.201 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.201 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.201 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T08:39:44.201 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T08:39:44.201 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T08:39:44.201 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T08:39:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T08:39:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T08:39:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T08:39:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T08:39:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T08:39:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T08:39:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T08:39:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T08:39:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T08:39:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T08:39:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T08:39:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T08:39:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T08:39:44.202 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T08:39:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T08:39:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T08:39:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T08:39:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T08:39:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T08:39:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T08:39:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T08:39:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T08:39:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T08:39:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T08:39:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T08:39:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T08:39:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T08:39:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T08:39:44.203 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T08:39:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T08:39:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T08:39:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T08:39:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T08:39:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T08:39:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T08:39:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T08:39:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T08:39:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T08:39:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T08:39:44.204 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T08:39:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T08:39:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T08:39:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T08:39:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T08:39:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T08:39:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T08:39:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T08:39:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T08:39:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T08:39:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T08:39:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T08:39:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T08:39:44.205 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T08:39:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T08:39:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:39:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T08:39:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T08:39:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T08:39:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T08:39:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T08:39:44.206 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T08:39:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T08:39:44.207 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-21T08:39:44.207 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-21T08:39:44.562 DEBUG:teuthology.parallel:result is None 2024-04-21T08:39:44.563 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:39:44.712 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T08:39:44.740 INFO:teuthology.orchestra.run.smithi028.stdout:19.0.0-3154.ga630765c.el9 2024-04-21T08:39:44.740 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154.ga630765c.el9 2024-04-21T08:39:44.740 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154.ga630765c is installed. 2024-04-21T08:39:44.743 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:39:44.961 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T08:39:44.983 INFO:teuthology.orchestra.run.smithi152.stdout:19.0.0-3154.ga630765c.el9 2024-04-21T08:39:44.983 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154.ga630765c.el9 2024-04-21T08:39:44.984 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154.ga630765c is installed. 2024-04-21T08:39:44.986 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:39:45.203 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T08:39:45.229 INFO:teuthology.orchestra.run.smithi171.stdout:19.0.0-3154.ga630765c.el9 2024-04-21T08:39:45.229 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154.ga630765c.el9 2024-04-21T08:39:45.229 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154.ga630765c is installed. 2024-04-21T08:39:45.231 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-21T08:39:45.231 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:39:45.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T08:39:45.263 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-21T08:39:45.263 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T08:39:45.289 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-21T08:39:45.289 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T08:39:45.319 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-21T08:39:45.319 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:39:45.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T08:39:45.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T08:39:45.417 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-21T08:39:45.417 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T08:39:45.441 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T08:39:45.504 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-21T08:39:45.505 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T08:39:45.533 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T08:39:45.601 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-21T08:39:45.601 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:39:45.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T08:39:45.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T08:39:45.700 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-21T08:39:45.700 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T08:39:45.725 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T08:39:45.789 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-21T08:39:45.789 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T08:39:45.814 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T08:39:45.878 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-21T08:39:45.879 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:39:45.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T08:39:45.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T08:39:45.977 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-21T08:39:45.977 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T08:39:46.001 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T08:39:46.064 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-21T08:39:46.065 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T08:39:46.088 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T08:39:46.154 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-21T08:39:46.246 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-21T08:39:46.247 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:39:46.247 INFO:tasks.cephadm:Cluster fsid is b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:39:46.247 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-21T08:39:46.247 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.28', 'mon.b': '172.21.15.152', 'mon.c': '172.21.15.171'} 2024-04-21T08:39:46.247 INFO:tasks.cephadm:First mon is mon.a on smithi028 2024-04-21T08:39:46.248 INFO:tasks.cephadm:First mgr is x 2024-04-21T08:39:46.248 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-21T08:39:46.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-04-21T08:39:46.277 DEBUG:teuthology.orchestra.run.smithi152:> sudo hostname $(hostname -s) 2024-04-21T08:39:46.301 DEBUG:teuthology.orchestra.run.smithi171:> sudo hostname $(hostname -s) 2024-04-21T08:39:46.325 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-21T08:39:46.325 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:39:46.468 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78740/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3154-ga630765c', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3154.ga630765c'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-19 11:13:53.331234', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-21T08:39:46.616 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 a630765c1f93c6e61d9b40c4b76437611d5ef29e from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:39:46.617 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm 2024-04-21T08:39:46.618 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T08:39:47.063 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 781532 Apr 21 08:39 /home/ubuntu/cephtest/cephadm 2024-04-21T08:39:47.063 DEBUG:teuthology.orchestra.run.smithi152:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T08:39:47.504 INFO:teuthology.orchestra.run.smithi152.stdout:-rw-r--r--. 1 ubuntu ubuntu 781532 Apr 21 08:39 /home/ubuntu/cephtest/cephadm 2024-04-21T08:39:47.504 DEBUG:teuthology.orchestra.run.smithi171:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T08:39:47.954 INFO:teuthology.orchestra.run.smithi171.stdout:-rw-r--r--. 1 ubuntu ubuntu 781532 Apr 21 08:39 /home/ubuntu/cephtest/cephadm 2024-04-21T08:39:47.954 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T08:39:47.974 DEBUG:teuthology.orchestra.run.smithi152:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T08:39:47.993 DEBUG:teuthology.orchestra.run.smithi171:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T08:39:48.028 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e on all hosts... 2024-04-21T08:39:48.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e pull 2024-04-21T08:39:48.031 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e pull 2024-04-21T08:39:48.036 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e pull 2024-04-21T08:39:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:39:49.589 INFO:teuthology.orchestra.run.smithi171.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:39:49.735 INFO:teuthology.orchestra.run.smithi152.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:40:21.510 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-04-21T08:40:21.510 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:40:21.510 INFO:teuthology.orchestra.run.smithi152.stdout: "image_id": "b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192", 2024-04-21T08:40:21.510 INFO:teuthology.orchestra.run.smithi152.stdout: "repo_digests": [ 2024-04-21T08:40:21.510 INFO:teuthology.orchestra.run.smithi152.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda" 2024-04-21T08:40:21.510 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-04-21T08:40:21.510 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-04-21T08:40:22.899 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-21T08:40:22.899 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:40:22.899 INFO:teuthology.orchestra.run.smithi171.stdout: "image_id": "b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192", 2024-04-21T08:40:22.899 INFO:teuthology.orchestra.run.smithi171.stdout: "repo_digests": [ 2024-04-21T08:40:22.899 INFO:teuthology.orchestra.run.smithi171.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda" 2024-04-21T08:40:22.899 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-04-21T08:40:22.899 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-21T08:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192", 2024-04-21T08:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-04-21T08:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda" 2024-04-21T08:40:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-21T08:40:24.298 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:40:24.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-04-21T08:40:24.361 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph 2024-04-21T08:40:24.392 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /etc/ceph 2024-04-21T08:40:24.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-04-21T08:40:24.450 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /etc/ceph 2024-04-21T08:40:24.477 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 777 /etc/ceph 2024-04-21T08:40:24.503 INFO:tasks.cephadm:Writing seed config... 2024-04-21T08:40:24.504 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-21T08:40:24.504 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-21T08:40:24.505 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-21T08:40:24.505 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-21T08:40:24.505 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-21T08:40:24.505 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-21T08:40:24.505 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-21T08:40:24.505 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-21T08:40:24.505 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-21T08:40:24.505 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-21T08:40:24.505 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-21T08:40:24.505 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-21T08:40:24.505 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-21T08:40:24.505 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-21T08:40:24.505 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-21T08:40:24.505 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-21T08:40:24.506 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-21T08:40:24.506 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-21T08:40:24.506 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-21T08:40:24.506 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-21T08:40:24.506 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-21T08:40:24.506 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-21T08:40:24.506 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-21T08:40:24.506 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-21T08:40:24.506 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-21T08:40:24.506 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-21T08:40:24.506 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-21T08:40:24.506 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-21T08:40:24.506 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-21T08:40:24.506 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-21T08:40:24.507 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-21T08:40:24.507 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-21T08:40:24.507 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-21T08:40:24.507 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-21T08:40:24.507 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-21T08:40:24.507 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-21T08:40:24.507 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-21T08:40:24.507 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-21T08:40:24.507 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-21T08:40:24.507 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-04-21T08:40:24.507 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-21T08:40:24.507 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-21T08:40:24.507 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-21T08:40:24.507 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-21T08:40:24.507 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-04-21T08:40:24.508 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-21T08:40:24.508 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-21T08:40:24.508 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-21T08:40:24.508 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-21T08:40:24.508 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:40:24.509 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-21T08:40:24.524 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = b51a76c8-ffba-11ee-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-21T08:40:24.525 DEBUG:teuthology.orchestra.run.smithi028:mon.a> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mon.a.service 2024-04-21T08:40:24.567 DEBUG:teuthology.orchestra.run.smithi028:mgr.x> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mgr.x.service 2024-04-21T08:40:24.610 INFO:tasks.cephadm:Bootstrapping... 2024-04-21T08:40:24.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e -v bootstrap --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.28 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:40:24.808 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:40:24.808 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e', '-v', 'bootstrap', '--fsid', 'b51a76c8-ffba-11ee-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.28', '--skip-admin-label'] 2024-04-21T08:40:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-21T08:40:24.808 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying podman|docker is present... 2024-04-21T08:40:24.842 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-21T08:40:24.843 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying lvm2 is present... 2024-04-21T08:40:24.843 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying time synchronization is in place... 2024-04-21T08:40:24.850 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-21T08:40:24.850 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-21T08:40:24.856 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-21T08:40:24.857 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-21T08:40:24.863 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-04-21T08:40:24.869 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-04-21T08:40:24.869 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-04-21T08:40:24.870 INFO:teuthology.orchestra.run.smithi028.stdout:Repeating the final host check... 2024-04-21T08:40:24.894 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-21T08:40:24.894 INFO:teuthology.orchestra.run.smithi028.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-21T08:40:24.895 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl is present 2024-04-21T08:40:24.895 INFO:teuthology.orchestra.run.smithi028.stdout:lvcreate is present 2024-04-21T08:40:24.901 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-21T08:40:24.901 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-21T08:40:24.907 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-21T08:40:24.907 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-21T08:40:24.913 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-04-21T08:40:24.920 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-04-21T08:40:24.920 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-04-21T08:40:24.920 INFO:teuthology.orchestra.run.smithi028.stdout:Host looks OK 2024-04-21T08:40:24.920 INFO:teuthology.orchestra.run.smithi028.stdout:Cluster fsid: b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:40:24.920 INFO:teuthology.orchestra.run.smithi028.stdout:Acquiring lock 140077182848160 on /run/cephadm/b51a76c8-ffba-11ee-bc93-c7b262605968.lock 2024-04-21T08:40:24.920 INFO:teuthology.orchestra.run.smithi028.stdout:Lock 140077182848160 acquired on /run/cephadm/b51a76c8-ffba-11ee-bc93-c7b262605968.lock 2024-04-21T08:40:24.920 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 3300 ... 2024-04-21T08:40:24.921 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 6789 ... 2024-04-21T08:40:24.921 INFO:teuthology.orchestra.run.smithi028.stdout:Base mon IP(s) is [172.21.15.28:3300, 172.21.15.28:6789], mon addrv is [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-04-21T08:40:24.924 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.28 metric 100 2024-04-21T08:40:24.924 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.28 metric 100 2024-04-21T08:40:24.927 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-21T08:40:24.927 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-21T08:40:24.927 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-21T08:40:24.929 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-21T08:40:24.929 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-21T08:40:24.929 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-21T08:40:24.929 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-21T08:40:24.929 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-21T08:40:24.929 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-21T08:40:24.930 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-04-21T08:40:24.930 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-04-21T08:40:24.930 INFO:teuthology.orchestra.run.smithi028.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-21T08:40:24.931 INFO:teuthology.orchestra.run.smithi028.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-21T08:40:24.931 INFO:teuthology.orchestra.run.smithi028.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:40:25.400 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 2024-04-21T08:40:25.400 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:40:25.400 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Getting image source signatures 2024-04-21T08:40:25.400 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:7eab8a2da10d06ac88aea1fd8998a2559b4de9d07c585e3b1e6ee6c4d35fc119 2024-04-21T08:40:25.400 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-21T08:40:25.400 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying config sha256:b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 2024-04-21T08:40:25.400 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-21T08:40:26.262 INFO:teuthology.orchestra.run.smithi028.stdout:ceph: stdout ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev) 2024-04-21T08:40:26.262 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph version: ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev) 2024-04-21T08:40:26.262 INFO:teuthology.orchestra.run.smithi028.stdout:Extracting ceph user uid/gid from container image... 2024-04-21T08:40:27.243 INFO:teuthology.orchestra.run.smithi028.stdout:stat: stdout 167 167 2024-04-21T08:40:27.243 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial keys... 2024-04-21T08:40:28.256 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQD70CRm3I32IBAARc8kF4HT7A1KYnuXLtXBeA== 2024-04-21T08:40:29.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQD80CRmo4k+JhAAi9nwngyAfibozj0DnsRMgA== 2024-04-21T08:40:30.061 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQD90CRmTyGMHhAAXIqL0pKIwxmi632we2B4tQ== 2024-04-21T08:40:30.062 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial monmap... 2024-04-21T08:40:30.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-21T08:40:30.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-21T08:40:30.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:40:30.933 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-21T08:40:30.933 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool for a [v2:172.21.15.28:3300,v1:172.21.15.28:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-21T08:40:30.934 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-04-21T08:40:30.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: set fsid to b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:40:30.934 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-21T08:40:30.934 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:40:30.934 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mon... 2024-04-21T08:40:31.848 INFO:teuthology.orchestra.run.smithi028.stdout:create mon.a on 2024-04-21T08:40:32.073 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-21T08:40:32.245 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-21T08:40:32.437 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-b51a76c8-ffba-11ee-bc93-c7b262605968.target → /etc/systemd/system/ceph-b51a76c8-ffba-11ee-bc93-c7b262605968.target. 2024-04-21T08:40:32.437 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-b51a76c8-ffba-11ee-bc93-c7b262605968.target → /etc/systemd/system/ceph-b51a76c8-ffba-11ee-bc93-c7b262605968.target. 2024-04-21T08:40:32.695 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mon.a 2024-04-21T08:40:32.695 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mon.a.service: Unit ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mon.a.service not loaded. 2024-04-21T08:40:32.867 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b51a76c8-ffba-11ee-bc93-c7b262605968.target.wants/ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@.service. 2024-04-21T08:40:33.446 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:40:33.447 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-21T08:40:33.454 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:40:33.454 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-21T08:40:33.454 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-21T08:40:33.454 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-04-21T08:40:33.454 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon to start... 2024-04-21T08:40:33.454 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon... 2024-04-21T08:40:34.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:34 smithi028 ceph-mon[27632]: from='client.? 172.21.15.28:0/1159907137' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-21T08:40:34.540 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout cluster: 2024-04-21T08:40:34.540 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout id: b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:40:34.540 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-21T08:40:34.540 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-21T08:40:34.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout services: 2024-04-21T08:40:34.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.564541s) 2024-04-21T08:40:34.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-21T08:40:34.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-21T08:40:34.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-21T08:40:34.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout data: 2024-04-21T08:40:34.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-21T08:40:34.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-21T08:40:34.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-21T08:40:34.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pgs: 2024-04-21T08:40:34.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-21T08:40:34.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.887+0000 7ffbcd898700 1 Processor -- start 2024-04-21T08:40:34.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.887+0000 7ffbcd898700 1 -- start start 2024-04-21T08:40:34.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.887+0000 7ffbcd898700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc8108590 0x7ffbc81089b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:34.541 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.887+0000 7ffbcd898700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc8108f80 con 0x7ffbc8108590 2024-04-21T08:40:34.542 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.887+0000 7ffbc6ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc8108590 0x7ffbc81089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:34.542 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.887+0000 7ffbc6ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc8108590 0x7ffbc81089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55748/0 (socket says 172.21.15.28:55748) 2024-04-21T08:40:34.542 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.887+0000 7ffbc6ffd700 1 -- 172.21.15.28:0/2873467215 learned_addr learned my addr 172.21.15.28:0/2873467215 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:34.542 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.887+0000 7ffbc6ffd700 1 -- 172.21.15.28:0/2873467215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc8109740 con 0x7ffbc8108590 2024-04-21T08:40:34.542 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.888+0000 7ffbc6ffd700 1 --2- 172.21.15.28:0/2873467215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc8108590 0x7ffbc81089b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80098a0 tx=0x7ffbb8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343286c87194484c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:34.542 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.888+0000 7ffbc5ffb700 1 -- 172.21.15.28:0/2873467215 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbb8004160 con 0x7ffbc8108590 2024-04-21T08:40:34.542 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.888+0000 7ffbc5ffb700 1 -- 172.21.15.28:0/2873467215 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbb8004300 con 0x7ffbc8108590 2024-04-21T08:40:34.542 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.888+0000 7ffbc5ffb700 1 -- 172.21.15.28:0/2873467215 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbb80036a0 con 0x7ffbc8108590 2024-04-21T08:40:34.542 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.888+0000 7ffbcd898700 1 -- 172.21.15.28:0/2873467215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc8108590 msgr2=0x7ffbc81089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:34.542 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.888+0000 7ffbcd898700 1 --2- 172.21.15.28:0/2873467215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc8108590 0x7ffbc81089b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80098a0 tx=0x7ffbb8009bb0 comp rx=0 tx=0).stop 2024-04-21T08:40:34.542 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.888+0000 7ffbcd898700 1 -- 172.21.15.28:0/2873467215 shutdown_connections 2024-04-21T08:40:34.542 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.888+0000 7ffbcd898700 1 --2- 172.21.15.28:0/2873467215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc8108590 0x7ffbc81089b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:34.542 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.888+0000 7ffbcd898700 1 -- 172.21.15.28:0/2873467215 >> 172.21.15.28:0/2873467215 conn(0x7ffbc8103b70 msgr2=0x7ffbc8105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:34.543 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.888+0000 7ffbcd898700 1 -- 172.21.15.28:0/2873467215 shutdown_connections 2024-04-21T08:40:34.543 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.889+0000 7ffbcd898700 1 -- 172.21.15.28:0/2873467215 wait complete. 2024-04-21T08:40:34.543 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.889+0000 7ffbcd898700 1 Processor -- start 2024-04-21T08:40:34.543 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.889+0000 7ffbcd898700 1 -- start start 2024-04-21T08:40:34.543 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.889+0000 7ffbcd898700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc8108590 0x7ffbc807db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:34.543 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.889+0000 7ffbcd898700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffbc8109c70 con 0x7ffbc8108590 2024-04-21T08:40:34.543 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.890+0000 7ffbc6ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc8108590 0x7ffbc807db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:34.543 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.890+0000 7ffbc6ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc8108590 0x7ffbc807db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55754/0 (socket says 172.21.15.28:55754) 2024-04-21T08:40:34.543 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.890+0000 7ffbc6ffd700 1 -- 172.21.15.28:0/1159907137 learned_addr learned my addr 172.21.15.28:0/1159907137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:34.543 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.890+0000 7ffbc6ffd700 1 -- 172.21.15.28:0/1159907137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbc807e0c0 con 0x7ffbc8108590 2024-04-21T08:40:34.543 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.890+0000 7ffbc6ffd700 1 --2- 172.21.15.28:0/1159907137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc8108590 0x7ffbc807db80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8000c00 tx=0x7ffbb8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:34.543 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.890+0000 7ffbcc896700 1 -- 172.21.15.28:0/1159907137 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbb8003db0 con 0x7ffbc8108590 2024-04-21T08:40:34.543 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.890+0000 7ffbcc896700 1 -- 172.21.15.28:0/1159907137 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbb8011630 con 0x7ffbc8108590 2024-04-21T08:40:34.543 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.890+0000 7ffbcd898700 1 -- 172.21.15.28:0/1159907137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbc807c2b0 con 0x7ffbc8108590 2024-04-21T08:40:34.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.890+0000 7ffbcc896700 1 -- 172.21.15.28:0/1159907137 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffbb8011950 con 0x7ffbc8108590 2024-04-21T08:40:34.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.890+0000 7ffbcd898700 1 -- 172.21.15.28:0/1159907137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbc807c7f0 con 0x7ffbc8108590 2024-04-21T08:40:34.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.890+0000 7ffbcc896700 1 -- 172.21.15.28:0/1159907137 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ffbb801a460 con 0x7ffbc8108590 2024-04-21T08:40:34.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.891+0000 7ffbcc896700 1 -- 172.21.15.28:0/1159907137 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ffbb801baf0 con 0x7ffbc8108590 2024-04-21T08:40:34.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.891+0000 7ffbcd898700 1 -- 172.21.15.28:0/1159907137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbc80640a0 con 0x7ffbc8108590 2024-04-21T08:40:34.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.894+0000 7ffbcc896700 1 -- 172.21.15.28:0/1159907137 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ffbb8004260 con 0x7ffbc8108590 2024-04-21T08:40:34.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.968+0000 7ffbcd898700 1 -- 172.21.15.28:0/1159907137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7ffbc8195600 con 0x7ffbc8108590 2024-04-21T08:40:34.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.968+0000 7ffbcc896700 1 -- 172.21.15.28:0/1159907137 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7ffbb801bcf0 con 0x7ffbc8108590 2024-04-21T08:40:34.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.969+0000 7ffbcd898700 1 -- 172.21.15.28:0/1159907137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc8108590 msgr2=0x7ffbc807db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:34.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.969+0000 7ffbcd898700 1 --2- 172.21.15.28:0/1159907137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc8108590 0x7ffbc807db80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8000c00 tx=0x7ffbb8004750 comp rx=0 tx=0).stop 2024-04-21T08:40:34.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.969+0000 7ffbcd898700 1 -- 172.21.15.28:0/1159907137 shutdown_connections 2024-04-21T08:40:34.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.969+0000 7ffbcd898700 1 --2- 172.21.15.28:0/1159907137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc8108590 0x7ffbc807db80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:34.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.969+0000 7ffbcd898700 1 -- 172.21.15.28:0/1159907137 >> 172.21.15.28:0/1159907137 conn(0x7ffbc8103b70 msgr2=0x7ffbc8192940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:34.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.969+0000 7ffbcd898700 1 -- 172.21.15.28:0/1159907137 shutdown_connections 2024-04-21T08:40:34.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:33.969+0000 7ffbcd898700 1 -- 172.21.15.28:0/1159907137 wait complete. 2024-04-21T08:40:34.545 INFO:teuthology.orchestra.run.smithi028.stdout:mon is available 2024-04-21T08:40:34.545 INFO:teuthology.orchestra.run.smithi028.stdout:Assimilating anything we can from ceph.conf... 2024-04-21T08:40:35.621 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-21T08:40:35.621 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-04-21T08:40:35.621 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:40:35.621 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-21T08:40:35.621 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-04-21T08:40:35.621 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-21T08:40:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-21T08:40:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-21T08:40:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-21T08:40:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-21T08:40:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-21T08:40:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-21T08:40:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-21T08:40:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-04-21T08:40:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-04-21T08:40:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-21T08:40:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-21T08:40:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-21T08:40:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-21T08:40:35.622 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-21T08:40:35.623 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.000+0000 7f2656dee700 1 Processor -- start 2024-04-21T08:40:35.623 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.000+0000 7f2656dee700 1 -- start start 2024-04-21T08:40:35.623 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.000+0000 7f2656dee700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2650107450 0x7f2650107870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:35.623 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.000+0000 7f2656dee700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2650107e40 con 0x7f2650107450 2024-04-21T08:40:35.623 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.000+0000 7f2654b8a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2650107450 0x7f2650107870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:35.623 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.000+0000 7f2654b8a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2650107450 0x7f2650107870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55770/0 (socket says 172.21.15.28:55770) 2024-04-21T08:40:35.623 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.000+0000 7f2654b8a700 1 -- 172.21.15.28:0/385707451 learned_addr learned my addr 172.21.15.28:0/385707451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:35.623 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.001+0000 7f2654b8a700 1 -- 172.21.15.28:0/385707451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2650108650 con 0x7f2650107450 2024-04-21T08:40:35.623 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.001+0000 7f2654b8a700 1 --2- 172.21.15.28:0/385707451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2650107450 0x7f2650107870 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f264400bb40 tx=0x7f264400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1bfabd616598516 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:35.623 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.001+0000 7f264f7fe700 1 -- 172.21.15.28:0/385707451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f264400dbe0 con 0x7f2650107450 2024-04-21T08:40:35.623 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.001+0000 7f264f7fe700 1 -- 172.21.15.28:0/385707451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f264400dd80 con 0x7f2650107450 2024-04-21T08:40:35.623 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.001+0000 7f264f7fe700 1 -- 172.21.15.28:0/385707451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2644004180 con 0x7f2650107450 2024-04-21T08:40:35.623 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.002+0000 7f2656dee700 1 -- 172.21.15.28:0/385707451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2650107450 msgr2=0x7f2650107870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:35.623 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.002+0000 7f2656dee700 1 --2- 172.21.15.28:0/385707451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2650107450 0x7f2650107870 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f264400bb40 tx=0x7f264400be50 comp rx=0 tx=0).stop 2024-04-21T08:40:35.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.002+0000 7f2656dee700 1 -- 172.21.15.28:0/385707451 shutdown_connections 2024-04-21T08:40:35.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.002+0000 7f2656dee700 1 --2- 172.21.15.28:0/385707451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2650107450 0x7f2650107870 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:35.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.002+0000 7f2656dee700 1 -- 172.21.15.28:0/385707451 >> 172.21.15.28:0/385707451 conn(0x7f26501029f0 msgr2=0x7f2650104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:35.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.002+0000 7f2656dee700 1 -- 172.21.15.28:0/385707451 shutdown_connections 2024-04-21T08:40:35.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.002+0000 7f2656dee700 1 -- 172.21.15.28:0/385707451 wait complete. 2024-04-21T08:40:35.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.002+0000 7f2656dee700 1 Processor -- start 2024-04-21T08:40:35.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.002+0000 7f2656dee700 1 -- start start 2024-04-21T08:40:35.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.003+0000 7f2656dee700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2650107450 0x7f2650199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:35.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.003+0000 7f2656dee700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2650108ac0 con 0x7f2650107450 2024-04-21T08:40:35.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.003+0000 7f2654b8a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2650107450 0x7f2650199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:35.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.003+0000 7f2654b8a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2650107450 0x7f2650199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55780/0 (socket says 172.21.15.28:55780) 2024-04-21T08:40:35.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.003+0000 7f2654b8a700 1 -- 172.21.15.28:0/3165866465 learned_addr learned my addr 172.21.15.28:0/3165866465 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:35.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.003+0000 7f2654b8a700 1 -- 172.21.15.28:0/3165866465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2650199f90 con 0x7f2650107450 2024-04-21T08:40:35.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.003+0000 7f2654b8a700 1 --2- 172.21.15.28:0/3165866465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2650107450 0x7f2650199a50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f264400b340 tx=0x7f2644004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:35.624 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.003+0000 7f264dffb700 1 -- 172.21.15.28:0/3165866465 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2644029070 con 0x7f2650107450 2024-04-21T08:40:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.003+0000 7f264dffb700 1 -- 172.21.15.28:0/3165866465 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26440044e0 con 0x7f2650107450 2024-04-21T08:40:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.003+0000 7f2656dee700 1 -- 172.21.15.28:0/3165866465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f265019a220 con 0x7f2650107450 2024-04-21T08:40:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.003+0000 7f264dffb700 1 -- 172.21.15.28:0/3165866465 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26440105a0 con 0x7f2650107450 2024-04-21T08:40:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.004+0000 7f2656dee700 1 -- 172.21.15.28:0/3165866465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f265019cf10 con 0x7f2650107450 2024-04-21T08:40:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.004+0000 7f264dffb700 1 -- 172.21.15.28:0/3165866465 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f264402c070 con 0x7f2650107450 2024-04-21T08:40:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.004+0000 7f264dffb700 1 -- 172.21.15.28:0/3165866465 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2644021070 con 0x7f2650107450 2024-04-21T08:40:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.005+0000 7f2656dee700 1 -- 172.21.15.28:0/3165866465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26500640a0 con 0x7f2650107450 2024-04-21T08:40:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.007+0000 7f264dffb700 1 -- 172.21.15.28:0/3165866465 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f2644015070 con 0x7f2650107450 2024-04-21T08:40:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.078+0000 7f2656dee700 1 -- 172.21.15.28:0/3165866465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f265002e150 con 0x7f2650107450 2024-04-21T08:40:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.082+0000 7f264dffb700 1 -- 172.21.15.28:0/3165866465 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+490 (secure 0 0 0) 0x7f264400dce0 con 0x7f2650107450 2024-04-21T08:40:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.082+0000 7f264dffb700 1 -- 172.21.15.28:0/3165866465 <== mon.0 v2:172.21.15.28:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2644010bd0 con 0x7f2650107450 2024-04-21T08:40:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.083+0000 7f2656dee700 1 -- 172.21.15.28:0/3165866465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2650107450 msgr2=0x7f2650199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.083+0000 7f2656dee700 1 --2- 172.21.15.28:0/3165866465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2650107450 0x7f2650199a50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f264400b340 tx=0x7f2644004750 comp rx=0 tx=0).stop 2024-04-21T08:40:35.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.084+0000 7f2656dee700 1 -- 172.21.15.28:0/3165866465 shutdown_connections 2024-04-21T08:40:35.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.084+0000 7f2656dee700 1 --2- 172.21.15.28:0/3165866465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2650107450 0x7f2650199a50 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:35.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.084+0000 7f2656dee700 1 -- 172.21.15.28:0/3165866465 >> 172.21.15.28:0/3165866465 conn(0x7f26501029f0 msgr2=0x7f2650190750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:35.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.084+0000 7f2656dee700 1 -- 172.21.15.28:0/3165866465 shutdown_connections 2024-04-21T08:40:35.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:35.084+0000 7f2656dee700 1 -- 172.21.15.28:0/3165866465 wait complete. 2024-04-21T08:40:35.626 INFO:teuthology.orchestra.run.smithi028.stdout:Generating new minimal ceph.conf... 2024-04-21T08:40:35.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:35 smithi028 ceph-mon[27632]: from='client.? 172.21.15.28:0/3165866465' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-21T08:40:35.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:35 smithi028 ceph-mon[27632]: from='client.? 172.21.15.28:0/3165866465' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-21T08:40:36.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:36 smithi028 ceph-mon[27632]: from='client.? 172.21.15.28:0/2998115014' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:36.693 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.036+0000 7f070fcb8700 1 Processor -- start 2024-04-21T08:40:36.693 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.036+0000 7f070fcb8700 1 -- start start 2024-04-21T08:40:36.693 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.036+0000 7f070fcb8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0708108590 0x7f07081089b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.693 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.036+0000 7f070fcb8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0708108f80 con 0x7f0708108590 2024-04-21T08:40:36.694 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.036+0000 7f070da54700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0708108590 0x7f07081089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.694 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.037+0000 7f070da54700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0708108590 0x7f07081089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55786/0 (socket says 172.21.15.28:55786) 2024-04-21T08:40:36.694 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.037+0000 7f070da54700 1 -- 172.21.15.28:0/2822299308 learned_addr learned my addr 172.21.15.28:0/2822299308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:36.694 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.037+0000 7f070da54700 1 -- 172.21.15.28:0/2822299308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0708109740 con 0x7f0708108590 2024-04-21T08:40:36.694 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.037+0000 7f070da54700 1 --2- 172.21.15.28:0/2822299308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0708108590 0x7f07081089b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0700009e80 tx=0x7f070000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197bd31dbad3bc77 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.694 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.037+0000 7f070ca52700 1 -- 172.21.15.28:0/2822299308 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0700017070 con 0x7f0708108590 2024-04-21T08:40:36.694 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.037+0000 7f070ca52700 1 -- 172.21.15.28:0/2822299308 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f070000b8c0 con 0x7f0708108590 2024-04-21T08:40:36.694 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.037+0000 7f070ca52700 1 -- 172.21.15.28:0/2822299308 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0700004490 con 0x7f0708108590 2024-04-21T08:40:36.694 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.038+0000 7f070fcb8700 1 -- 172.21.15.28:0/2822299308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0708108590 msgr2=0x7f07081089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.694 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.038+0000 7f070fcb8700 1 --2- 172.21.15.28:0/2822299308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0708108590 0x7f07081089b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0700009e80 tx=0x7f070000b330 comp rx=0 tx=0).stop 2024-04-21T08:40:36.694 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.038+0000 7f070fcb8700 1 -- 172.21.15.28:0/2822299308 shutdown_connections 2024-04-21T08:40:36.694 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.038+0000 7f070fcb8700 1 --2- 172.21.15.28:0/2822299308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0708108590 0x7f07081089b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.694 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.038+0000 7f070fcb8700 1 -- 172.21.15.28:0/2822299308 >> 172.21.15.28:0/2822299308 conn(0x7f0708103b70 msgr2=0x7f0708105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.038+0000 7f070fcb8700 1 -- 172.21.15.28:0/2822299308 shutdown_connections 2024-04-21T08:40:36.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.038+0000 7f070fcb8700 1 -- 172.21.15.28:0/2822299308 wait complete. 2024-04-21T08:40:36.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.038+0000 7f070fcb8700 1 Processor -- start 2024-04-21T08:40:36.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.038+0000 7f070fcb8700 1 -- start start 2024-04-21T08:40:36.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.039+0000 7f070fcb8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0708108590 0x7f070819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:36.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.039+0000 7f070fcb8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f070810caf0 con 0x7f0708108590 2024-04-21T08:40:36.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.039+0000 7f070da54700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0708108590 0x7f070819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:36.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.039+0000 7f070da54700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0708108590 0x7f070819e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55790/0 (socket says 172.21.15.28:55790) 2024-04-21T08:40:36.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.039+0000 7f070da54700 1 -- 172.21.15.28:0/2998115014 learned_addr learned my addr 172.21.15.28:0/2998115014 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:36.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.039+0000 7f070da54700 1 -- 172.21.15.28:0/2998115014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f070819ec10 con 0x7f0708108590 2024-04-21T08:40:36.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.039+0000 7f070da54700 1 --2- 172.21.15.28:0/2998115014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0708108590 0x7f070819e6d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f070000b7b0 tx=0x7f070000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:36.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.039+0000 7f0706ffd700 1 -- 172.21.15.28:0/2998115014 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0700020020 con 0x7f0708108590 2024-04-21T08:40:36.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.039+0000 7f0706ffd700 1 -- 172.21.15.28:0/2998115014 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0700012430 con 0x7f0708108590 2024-04-21T08:40:36.695 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.039+0000 7f070fcb8700 1 -- 172.21.15.28:0/2998115014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f070819eea0 con 0x7f0708108590 2024-04-21T08:40:36.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.039+0000 7f0706ffd700 1 -- 172.21.15.28:0/2998115014 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0700017040 con 0x7f0708108590 2024-04-21T08:40:36.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.039+0000 7f070fcb8700 1 -- 172.21.15.28:0/2998115014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07081a1b90 con 0x7f0708108590 2024-04-21T08:40:36.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.039+0000 7f0706ffd700 1 -- 172.21.15.28:0/2998115014 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0700023020 con 0x7f0708108590 2024-04-21T08:40:36.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.040+0000 7f0706ffd700 1 -- 172.21.15.28:0/2998115014 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f070000f050 con 0x7f0708108590 2024-04-21T08:40:36.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.040+0000 7f070fcb8700 1 -- 172.21.15.28:0/2998115014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07080640a0 con 0x7f0708108590 2024-04-21T08:40:36.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.043+0000 7f0706ffd700 1 -- 172.21.15.28:0/2998115014 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f070002a020 con 0x7f0708108590 2024-04-21T08:40:36.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.113+0000 7f070fcb8700 1 -- 172.21.15.28:0/2998115014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f07081a1d60 con 0x7f0708108590 2024-04-21T08:40:36.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.114+0000 7f0706ffd700 1 -- 172.21.15.28:0/2998115014 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f070000b860 con 0x7f0708108590 2024-04-21T08:40:36.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.115+0000 7f070fcb8700 1 -- 172.21.15.28:0/2998115014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0708108590 msgr2=0x7f070819e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:36.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.115+0000 7f070fcb8700 1 --2- 172.21.15.28:0/2998115014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0708108590 0x7f070819e6d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f070000b7b0 tx=0x7f070000be20 comp rx=0 tx=0).stop 2024-04-21T08:40:36.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.115+0000 7f070fcb8700 1 -- 172.21.15.28:0/2998115014 shutdown_connections 2024-04-21T08:40:36.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.115+0000 7f070fcb8700 1 --2- 172.21.15.28:0/2998115014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0708108590 0x7f070819e6d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:36.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.115+0000 7f070fcb8700 1 -- 172.21.15.28:0/2998115014 >> 172.21.15.28:0/2998115014 conn(0x7f0708103b70 msgr2=0x7f0708104630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:36.696 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.115+0000 7f070fcb8700 1 -- 172.21.15.28:0/2998115014 shutdown_connections 2024-04-21T08:40:36.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:36.115+0000 7f070fcb8700 1 -- 172.21.15.28:0/2998115014 wait complete. 2024-04-21T08:40:36.697 INFO:teuthology.orchestra.run.smithi028.stdout:Restarting the monitor... 2024-04-21T08:40:36.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:36 smithi028 systemd[1]: Stopping Ceph mon.a for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T08:40:37.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:36 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a[27628]: 2024-04-21T08:40:36.950+0000 7fd0f28de700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:40:37.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:36 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a[27628]: 2024-04-21T08:40:36.950+0000 7fd0f28de700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-21T08:40:37.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:37 smithi028 podman[27878]: 2024-04-21 08:40:37.103937129 +0000 UTC m=+0.246261900 container died fd90df14cebc2229115b01ab17fa9bac79a3472bd79be1663c6df9dfbf5e0536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:40:37.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:37 smithi028 podman[27878]: 2024-04-21 08:40:37.451050436 +0000 UTC m=+0.593375206 container cleanup fd90df14cebc2229115b01ab17fa9bac79a3472bd79be1663c6df9dfbf5e0536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:40:37.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:37 smithi028 bash[27878]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a 2024-04-21T08:40:38.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:37 smithi028 podman[27890]: 2024-04-21 08:40:37.726722324 +0000 UTC m=+0.623024173 container remove fd90df14cebc2229115b01ab17fa9bac79a3472bd79be1663c6df9dfbf5e0536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-21T08:40:38.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:37 smithi028 systemd[1]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-21T08:40:38.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:37 smithi028 systemd[1]: Stopped Ceph mon.a for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:40:38.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:37 smithi028 systemd[1]: Starting Ceph mon.a for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T08:40:38.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 podman[27984]: 2024-04-21 08:40:38.002313431 +0000 UTC m=+0.018597704 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:40:38.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 podman[27984]: 2024-04-21 08:40:38.102334908 +0000 UTC m=+0.118619160 container create 4043afed007a0acf9d66e917e9be68c5f83a0d818afe8a0bc85c7831d16a515c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-21T08:40:38.435 INFO:teuthology.orchestra.run.smithi028.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-21T08:40:38.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 podman[27984]: 2024-04-21 08:40:38.357290901 +0000 UTC m=+0.373575168 container init 4043afed007a0acf9d66e917e9be68c5f83a0d818afe8a0bc85c7831d16a515c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-21T08:40:38.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 podman[27984]: 2024-04-21 08:40:38.360625866 +0000 UTC m=+0.376910122 container start 4043afed007a0acf9d66e917e9be68c5f83a0d818afe8a0bc85c7831d16a515c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:40:38.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: set uid:gid to 167:167 (ceph:ceph) 2024-04-21T08:40:38.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev), process ceph-mon, pid 2 2024-04-21T08:40:38.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: pidfile_write: ignore empty --pid-file 2024-04-21T08:40:38.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: load: jerasure load: lrc 2024-04-21T08:40:38.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: RocksDB version: 7.9.2 2024-04-21T08:40:38.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Git sha 0 2024-04-21T08:40:38.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Compile date 2024-04-19 14:19:22 2024-04-21T08:40:38.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: DB SUMMARY 2024-04-21T08:40:38.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: DB Session ID: AQ87WJDUPO1US8O659JS 2024-04-21T08:40:38.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: CURRENT file: CURRENT 2024-04-21T08:40:38.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: IDENTITY file: IDENTITY 2024-04-21T08:40:38.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-21T08:40:38.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-21T08:40:38.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105570 ; 2024-04-21T08:40:38.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.error_if_exists: 0 2024-04-21T08:40:38.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.create_if_missing: 0 2024-04-21T08:40:38.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.paranoid_checks: 1 2024-04-21T08:40:38.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-21T08:40:38.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-21T08:40:38.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-21T08:40:38.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.env: 0x55fcd9a87cc0 2024-04-21T08:40:38.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.fs: PosixFileSystem 2024-04-21T08:40:38.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.info_log: 0x55fcdb8d5be0 2024-04-21T08:40:38.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_file_opening_threads: 16 2024-04-21T08:40:38.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.statistics: (nil) 2024-04-21T08:40:38.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.use_fsync: 0 2024-04-21T08:40:38.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_log_file_size: 0 2024-04-21T08:40:38.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-21T08:40:38.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-21T08:40:38.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.keep_log_file_num: 1000 2024-04-21T08:40:38.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.recycle_log_file_num: 0 2024-04-21T08:40:38.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.allow_fallocate: 1 2024-04-21T08:40:38.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.allow_mmap_reads: 0 2024-04-21T08:40:38.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.allow_mmap_writes: 0 2024-04-21T08:40:38.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.use_direct_reads: 0 2024-04-21T08:40:38.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-21T08:40:38.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.create_missing_column_families: 0 2024-04-21T08:40:38.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.db_log_dir: 2024-04-21T08:40:38.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.wal_dir: 2024-04-21T08:40:38.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-21T08:40:38.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-21T08:40:38.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-21T08:40:38.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-21T08:40:38.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-21T08:40:38.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-21T08:40:38.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.advise_random_on_open: 1 2024-04-21T08:40:38.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.db_write_buffer_size: 0 2024-04-21T08:40:38.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.write_buffer_manager: 0x55fcdb8690e0 2024-04-21T08:40:38.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-21T08:40:38.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-21T08:40:38.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-21T08:40:38.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.rate_limiter: (nil) 2024-04-21T08:40:38.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-21T08:40:38.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.wal_recovery_mode: 2 2024-04-21T08:40:38.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.enable_thread_tracking: 0 2024-04-21T08:40:38.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.enable_pipelined_write: 0 2024-04-21T08:40:38.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.unordered_write: 0 2024-04-21T08:40:38.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-21T08:40:38.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-21T08:40:38.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-21T08:40:38.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-21T08:40:38.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.row_cache: None 2024-04-21T08:40:38.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.wal_filter: None 2024-04-21T08:40:38.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-21T08:40:38.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.allow_ingest_behind: 0 2024-04-21T08:40:38.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.two_write_queues: 0 2024-04-21T08:40:38.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.manual_wal_flush: 0 2024-04-21T08:40:38.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.wal_compression: 0 2024-04-21T08:40:38.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.atomic_flush: 0 2024-04-21T08:40:38.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-21T08:40:38.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-21T08:40:38.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-21T08:40:38.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.log_readahead_size: 0 2024-04-21T08:40:38.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-21T08:40:38.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.best_efforts_recovery: 0 2024-04-21T08:40:38.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-21T08:40:38.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-21T08:40:38.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.allow_data_in_errors: 0 2024-04-21T08:40:38.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.db_host_id: __hostname__ 2024-04-21T08:40:38.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-21T08:40:38.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_background_jobs: 2 2024-04-21T08:40:38.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_background_compactions: -1 2024-04-21T08:40:38.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_subcompactions: 1 2024-04-21T08:40:38.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-21T08:40:38.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_total_wal_size: 0 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_open_files: -1 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.bytes_per_sync: 0 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compaction_readahead_size: 0 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_background_flushes: -1 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Compression algorithms supported: 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: kZSTD supported: 0 2024-04-21T08:40:38.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: kXpressCompression supported: 0 2024-04-21T08:40:38.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: kLZ4HCCompression supported: 1 2024-04-21T08:40:38.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: kZlibCompression supported: 1 2024-04-21T08:40:38.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: kSnappyCompression supported: 1 2024-04-21T08:40:38.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: kLZ4Compression supported: 1 2024-04-21T08:40:38.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: kBZip2Compression supported: 0 2024-04-21T08:40:38.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-21T08:40:38.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-21T08:40:38.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-21T08:40:38.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-21T08:40:38.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-21T08:40:38.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.merge_operator: 2024-04-21T08:40:38.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compaction_filter: None 2024-04-21T08:40:38.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compaction_filter_factory: None 2024-04-21T08:40:38.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.sst_partitioner_factory: None 2024-04-21T08:40:38.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-21T08:40:38.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-21T08:40:38.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fcdb8d5dc0) 2024-04-21T08:40:38.634 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-04-21T08:40:38.634 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-21T08:40:38.634 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-21T08:40:38.634 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-04-21T08:40:38.634 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_type: 0 2024-04-21T08:40:38.634 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_index_type: 0 2024-04-21T08:40:38.634 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_shortening: 1 2024-04-21T08:40:38.634 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-21T08:40:38.634 INFO:journalctl@ceph.mon.a.smithi028.stdout: checksum: 4 2024-04-21T08:40:38.634 INFO:journalctl@ceph.mon.a.smithi028.stdout: no_block_cache: 0 2024-04-21T08:40:38.634 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache: 0x55fcdb89b1f0 2024-04-21T08:40:38.634 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-04-21T08:40:38.635 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_options: 2024-04-21T08:40:38.635 INFO:journalctl@ceph.mon.a.smithi028.stdout: capacity : 536870912 2024-04-21T08:40:38.635 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_shard_bits : 4 2024-04-21T08:40:38.635 INFO:journalctl@ceph.mon.a.smithi028.stdout: strict_capacity_limit : 0 2024-04-21T08:40:38.635 INFO:journalctl@ceph.mon.a.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-04-21T08:40:38.635 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_compressed: (nil) 2024-04-21T08:40:38.635 INFO:journalctl@ceph.mon.a.smithi028.stdout: persistent_cache: (nil) 2024-04-21T08:40:38.635 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size: 4096 2024-04-21T08:40:38.635 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size_deviation: 10 2024-04-21T08:40:38.635 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_restart_interval: 16 2024-04-21T08:40:38.635 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_block_restart_interval: 1 2024-04-21T08:40:38.635 INFO:journalctl@ceph.mon.a.smithi028.stdout: metadata_block_size: 4096 2024-04-21T08:40:38.635 INFO:journalctl@ceph.mon.a.smithi028.stdout: partition_filters: 0 2024-04-21T08:40:38.635 INFO:journalctl@ceph.mon.a.smithi028.stdout: use_delta_encoding: 1 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout: filter_policy: bloomfilter 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout: whole_key_filtering: 1 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout: verify_compression: 0 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout: format_version: 5 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_index_compression: 1 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_align: 0 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout: max_auto_readahead_size: 262144 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout: prepopulate_block_cache: 0 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout: initial_auto_readahead_size: 8192 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.write_buffer_size: 33554432 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_write_buffer_number: 2 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compression: NoCompression 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.bottommost_compression: Disabled 2024-04-21T08:40:38.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.prefix_extractor: nullptr 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.num_levels: 7 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-21T08:40:38.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compression_opts.level: 32767 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compression_opts.strategy: 0 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compression_opts.enabled: false 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.target_file_size_base: 67108864 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-21T08:40:38.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-21T08:40:38.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-21T08:40:38.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-21T08:40:38.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-21T08:40:38.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-21T08:40:38.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-21T08:40:38.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-21T08:40:38.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-21T08:40:38.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-21T08:40:38.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-21T08:40:38.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.arena_block_size: 1048576 2024-04-21T08:40:38.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-21T08:40:38.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-21T08:40:38.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.disable_auto_compactions: 0 2024-04-21T08:40:38.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-21T08:40:38.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-21T08:40:38.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-21T08:40:38.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-21T08:40:38.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-21T08:40:38.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-21T08:40:38.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-21T08:40:38.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-21T08:40:38.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-21T08:40:38.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-21T08:40:38.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-21T08:40:38.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.inplace_update_support: 0 2024-04-21T08:40:38.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-21T08:40:38.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-21T08:40:38.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-21T08:40:38.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-21T08:40:38.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.bloom_locality: 0 2024-04-21T08:40:38.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.max_successive_merges: 0 2024-04-21T08:40:38.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-21T08:40:38.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.paranoid_file_checks: 0 2024-04-21T08:40:38.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.force_consistency_checks: 1 2024-04-21T08:40:38.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.report_bg_io_stats: 0 2024-04-21T08:40:38.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.ttl: 2592000 2024-04-21T08:40:38.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-21T08:40:38.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-21T08:40:38.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-21T08:40:38.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.enable_blob_files: false 2024-04-21T08:40:38.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.min_blob_size: 0 2024-04-21T08:40:38.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.blob_file_size: 268435456 2024-04-21T08:40:38.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-21T08:40:38.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-21T08:40:38.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-21T08:40:38.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-21T08:40:38.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-21T08:40:38.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.blob_file_starting_level: 0 2024-04-21T08:40:38.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-21T08:40:38.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-21T08:40:38.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-21T08:40:38.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9bc33544-1eb9-4aae-88f8-cc006f9958e5 2024-04-21T08:40:38.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688838390193, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-21T08:40:38.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-21T08:40:38.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688838391221, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100677, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98561, "index_size": 308, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14286, "raw_average_key_size": 49, "raw_value_size": 90799, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713688838, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9bc33544-1eb9-4aae-88f8-cc006f9958e5", "db_session_id": "AQ87WJDUPO1US8O659JS", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-21T08:40:38.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688838391305, "job": 1, "event": "recovery_finished"} 2024-04-21T08:40:38.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-21T08:40:38.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-21T08:40:38.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fcdb9b0000 2024-04-21T08:40:38.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: DB pointer 0x55fcdb99a000 2024-04-21T08:40:38.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-21T08:40:38.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-21T08:40:38.644 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** DB Stats ** 2024-04-21T08:40:38.644 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:40:38.644 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-21T08:40:38.644 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:40:38.644 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:40:38.644 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-21T08:40:38.644 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:40:38.645 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:40:38.645 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-21T08:40:38.645 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-04-21T08:40:38.645 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:40:38.645 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-21T08:40:38.645 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 100.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:40:38.645 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 100.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:40:38.645 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:40:38.645 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-21T08:40:38.645 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-04-21T08:40:38.645 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:40:38.645 INFO:journalctl@ceph.mon.a.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-21T08:40:38.645 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 156.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:40:38.646 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-21T08:40:38.646 INFO:journalctl@ceph.mon.a.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-21T08:40:38.646 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-21T08:40:38.646 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:40:38.646 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-21T08:40:38.646 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-21T08:40:38.646 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-21T08:40:38.646 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-21T08:40:38.646 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-21T08:40:38.646 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 29.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:40:38.646 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 29.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:40:38.646 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-21T08:40:38.646 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache BinnedLRUCache@0x55fcdb89b1f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-21T08:40:38.646 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-21T08:40:38.646 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-21T08:40:38.647 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-21T08:40:38.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: starting mon.a rank 0 at public addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] at bind addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:40:38.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: mon.a@-1(???) e1 preinit fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:40:38.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: mon.a@-1(???).mds e1 new map 2024-04-21T08:40:38.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: mon.a@-1(???).mds e1 print_map 2024-04-21T08:40:38.647 INFO:journalctl@ceph.mon.a.smithi028.stdout: e1 2024-04-21T08:40:38.647 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T08:40:38.647 INFO:journalctl@ceph.mon.a.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:40:38.647 INFO:journalctl@ceph.mon.a.smithi028.stdout: legacy client fscid: -1 2024-04-21T08:40:38.647 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-21T08:40:38.647 INFO:journalctl@ceph.mon.a.smithi028.stdout: No filesystems configured 2024-04-21T08:40:38.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-21T08:40:38.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:40:38.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:40:38.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:40:38.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 ceph-mon[28003]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-21T08:40:38.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 bash[27984]: 4043afed007a0acf9d66e917e9be68c5f83a0d818afe8a0bc85c7831d16a515c 2024-04-21T08:40:38.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:38 smithi028 systemd[1]: Started Ceph mon.a for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:40:39.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.830+0000 7f284b605700 1 Processor -- start 2024-04-21T08:40:39.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.830+0000 7f284b605700 1 -- start start 2024-04-21T08:40:39.494 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.830+0000 7f284b605700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2844106e00 0x7f2844107220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:39.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.830+0000 7f284b605700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f284407be20 con 0x7f2844106e00 2024-04-21T08:40:39.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.831+0000 7f28493a1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2844106e00 0x7f2844107220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:39.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.831+0000 7f28493a1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2844106e00 0x7f2844107220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55792/0 (socket says 172.21.15.28:55792) 2024-04-21T08:40:39.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.831+0000 7f28493a1700 1 -- 172.21.15.28:0/1525941482 learned_addr learned my addr 172.21.15.28:0/1525941482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:39.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.831+0000 7f28493a1700 1 -- 172.21.15.28:0/1525941482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2844107760 con 0x7f2844106e00 2024-04-21T08:40:39.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.831+0000 7f28493a1700 1 --2- 172.21.15.28:0/1525941482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2844106e00 0x7f2844107220 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f28400098a0 tx=0x7f2840009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda6cdbeed83aa0f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:39.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.831+0000 7f283bfff700 1 -- 172.21.15.28:0/1525941482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2840004160 con 0x7f2844106e00 2024-04-21T08:40:39.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.832+0000 7f283bfff700 1 -- 172.21.15.28:0/1525941482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2840004300 con 0x7f2844106e00 2024-04-21T08:40:39.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.832+0000 7f283bfff700 1 -- 172.21.15.28:0/1525941482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2840011af0 con 0x7f2844106e00 2024-04-21T08:40:39.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.832+0000 7f284b605700 1 -- 172.21.15.28:0/1525941482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2844106e00 msgr2=0x7f2844107220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:39.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.832+0000 7f284b605700 1 --2- 172.21.15.28:0/1525941482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2844106e00 0x7f2844107220 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f28400098a0 tx=0x7f2840009bb0 comp rx=0 tx=0).stop 2024-04-21T08:40:39.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.832+0000 7f284b605700 1 -- 172.21.15.28:0/1525941482 shutdown_connections 2024-04-21T08:40:39.495 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.832+0000 7f284b605700 1 --2- 172.21.15.28:0/1525941482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2844106e00 0x7f2844107220 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.832+0000 7f284b605700 1 -- 172.21.15.28:0/1525941482 >> 172.21.15.28:0/1525941482 conn(0x7f28441029a0 msgr2=0x7f2844104de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.832+0000 7f284b605700 1 -- 172.21.15.28:0/1525941482 shutdown_connections 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.832+0000 7f284b605700 1 -- 172.21.15.28:0/1525941482 wait complete. 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.833+0000 7f284b605700 1 Processor -- start 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.833+0000 7f284b605700 1 -- start start 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.833+0000 7f284b605700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2844106e00 0x7f284419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.833+0000 7f284b605700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2844108790 con 0x7f2844106e00 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.833+0000 7f28493a1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2844106e00 0x7f284419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.833+0000 7f28493a1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2844106e00 0x7f284419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55806/0 (socket says 172.21.15.28:55806) 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.833+0000 7f28493a1700 1 -- 172.21.15.28:0/3486964902 learned_addr learned my addr 172.21.15.28:0/3486964902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.834+0000 7f28493a1700 1 -- 172.21.15.28:0/3486964902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f284419ebe0 con 0x7f2844106e00 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.834+0000 7f28493a1700 1 --2- 172.21.15.28:0/3486964902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2844106e00 0x7f284419e6a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2840003f40 tx=0x7f2840011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.834+0000 7f283a7fc700 1 -- 172.21.15.28:0/3486964902 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28400038b0 con 0x7f2844106e00 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.834+0000 7f283a7fc700 1 -- 172.21.15.28:0/3486964902 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2840003a50 con 0x7f2844106e00 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.834+0000 7f284b605700 1 -- 172.21.15.28:0/3486964902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f284419ee70 con 0x7f2844106e00 2024-04-21T08:40:39.496 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.834+0000 7f283a7fc700 1 -- 172.21.15.28:0/3486964902 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f284001a5c0 con 0x7f2844106e00 2024-04-21T08:40:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.834+0000 7f284b605700 1 -- 172.21.15.28:0/3486964902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28441a1b60 con 0x7f2844106e00 2024-04-21T08:40:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.834+0000 7f283a7fc700 1 -- 172.21.15.28:0/3486964902 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f284001ac00 con 0x7f2844106e00 2024-04-21T08:40:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.834+0000 7f283a7fc700 1 -- 172.21.15.28:0/3486964902 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f28400199b0 con 0x7f2844106e00 2024-04-21T08:40:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.835+0000 7f284b605700 1 -- 172.21.15.28:0/3486964902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28440640a0 con 0x7f2844106e00 2024-04-21T08:40:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.837+0000 7f283a7fc700 1 -- 172.21.15.28:0/3486964902 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f2840019bb0 con 0x7f2844106e00 2024-04-21T08:40:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.909+0000 7f284b605700 1 -- 172.21.15.28:0/3486964902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f284402df80 con 0x7f2844106e00 2024-04-21T08:40:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.911+0000 7f283a7fc700 1 -- 172.21.15.28:0/3486964902 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f2840019d90 con 0x7f2844106e00 2024-04-21T08:40:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.911+0000 7f284b605700 1 -- 172.21.15.28:0/3486964902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2844106e00 msgr2=0x7f284419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.911+0000 7f284b605700 1 --2- 172.21.15.28:0/3486964902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2844106e00 0x7f284419e6a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2840003f40 tx=0x7f2840011890 comp rx=0 tx=0).stop 2024-04-21T08:40:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.912+0000 7f284b605700 1 -- 172.21.15.28:0/3486964902 shutdown_connections 2024-04-21T08:40:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.912+0000 7f284b605700 1 --2- 172.21.15.28:0/3486964902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2844106e00 0x7f284419e6a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.912+0000 7f284b605700 1 -- 172.21.15.28:0/3486964902 >> 172.21.15.28:0/3486964902 conn(0x7f28441029a0 msgr2=0x7f2844195310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.912+0000 7f284b605700 1 -- 172.21.15.28:0/3486964902 shutdown_connections 2024-04-21T08:40:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:38.912+0000 7f284b605700 1 -- 172.21.15.28:0/3486964902 wait complete. 2024-04-21T08:40:39.497 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-21T08:40:39.511 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:40:39.511 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mgr... 2024-04-21T08:40:39.512 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-21T08:40:39.512 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-21T08:40:39.746 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mgr.x 2024-04-21T08:40:39.746 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mgr.x.service: Unit ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-21T08:40:39.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:39 smithi028 ceph-mon[28003]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-21T08:40:39.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:39 smithi028 ceph-mon[28003]: monmap epoch 1 2024-04-21T08:40:39.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:39 smithi028 ceph-mon[28003]: fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:40:39.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:39 smithi028 ceph-mon[28003]: last_changed 2024-04-21T08:40:30.404569+0000 2024-04-21T08:40:39.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:39 smithi028 ceph-mon[28003]: created 2024-04-21T08:40:30.404569+0000 2024-04-21T08:40:39.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:39 smithi028 ceph-mon[28003]: min_mon_release 19 (squid) 2024-04-21T08:40:39.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:39 smithi028 ceph-mon[28003]: election_strategy: 1 2024-04-21T08:40:39.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:39 smithi028 ceph-mon[28003]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-21T08:40:39.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:39 smithi028 ceph-mon[28003]: fsmap 2024-04-21T08:40:39.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:39 smithi028 ceph-mon[28003]: osdmap e1: 0 total, 0 up, 0 in 2024-04-21T08:40:39.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:39 smithi028 ceph-mon[28003]: mgrmap e1: no daemons active 2024-04-21T08:40:39.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:39 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3486964902' entity='client.admin' 2024-04-21T08:40:39.900 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b51a76c8-ffba-11ee-bc93-c7b262605968.target.wants/ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@.service. 2024-04-21T08:40:40.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:39 smithi028 systemd[1]: Starting Ceph mgr.x for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T08:40:40.424 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:40 smithi028 podman[28221]: 2024-04-21 08:40:40.048398229 +0000 UTC m=+0.018379759 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:40:40.424 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:40 smithi028 podman[28221]: 2024-04-21 08:40:40.148427919 +0000 UTC m=+0.118409434 container create fa359b820dc10aba01f4fe9af11850e7b550260a7501798dd55eeb42bb6c8a54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-21T08:40:40.425 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:40 smithi028 podman[28221]: 2024-04-21 08:40:40.342376811 +0000 UTC m=+0.312358329 container init fa359b820dc10aba01f4fe9af11850e7b550260a7501798dd55eeb42bb6c8a54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-21T08:40:40.425 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:40 smithi028 podman[28221]: 2024-04-21 08:40:40.345003825 +0000 UTC m=+0.314985341 container start fa359b820dc10aba01f4fe9af11850e7b550260a7501798dd55eeb42bb6c8a54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:40:40.425 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:40 smithi028 bash[28221]: fa359b820dc10aba01f4fe9af11850e7b550260a7501798dd55eeb42bb6c8a54 2024-04-21T08:40:40.438 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:40:40.438 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-21T08:40:40.446 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:40:40.446 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-21T08:40:40.446 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-21T08:40:40.446 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-04-21T08:40:40.454 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:40:40.454 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-21T08:40:40.462 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:40:40.462 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-21T08:40:40.462 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-21T08:40:40.462 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-21T08:40:40.462 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr to start... 2024-04-21T08:40:40.463 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr... 2024-04-21T08:40:40.702 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:40 smithi028 systemd[1]: Started Ceph mgr.x for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:40:40.702 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:40 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:40.506+0000 7f6eeabc5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:40:40.702 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:40 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:40.701+0000 7f6eeabc5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:40:40.952 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:40 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:40.810+0000 7f6eeabc5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:40:41.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:41 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2349977546' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:40:41.247 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:41 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:41.157+0000 7f6eeabc5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:40:41.247 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:41 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:41.246+0000 7f6eeabc5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:40:41.616 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-21T08:40:41.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:40:41.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "b51a76c8-ffba-11ee-bc93-c7b262605968", 2024-04-21T08:40:41.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:40:41.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:40:41.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:40:41.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:40:41.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:41.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:40:41.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:40:41.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:40:41.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:40:41.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:40:41.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:40:41.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:40:41.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-21T08:40:41.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:40:41.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:41.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:40:41.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:40:41.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:41.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:40:41.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:41.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:40:41.621 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:40:41.621 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:40:41.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:40:41.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:40:41.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:40:41.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:41.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:40:41.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:40:41.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:40:41.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:40:41.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:40:41.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:40:41.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:40:41.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:40:41.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:40:41.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:41.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:40:41.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:41.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:40:41.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:40:41.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:41.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:40:41.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T08:40:41.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:40:41.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:40:41.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:40:41.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:40:41.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:40:33.405908+0000", 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.946+0000 7f07162e3700 1 Processor -- start 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.946+0000 7f07162e3700 1 -- start start 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.947+0000 7f07162e3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0710107920 0x7f0710107d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.947+0000 7f07162e3700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0710108310 con 0x7f0710107920 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.947+0000 7f070ffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0710107920 0x7f0710107d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:41.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.947+0000 7f070ffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0710107920 0x7f0710107d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55826/0 (socket says 172.21.15.28:55826) 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.947+0000 7f070ffff700 1 -- 172.21.15.28:0/2632063616 learned_addr learned my addr 172.21.15.28:0/2632063616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.947+0000 7f070ffff700 1 -- 172.21.15.28:0/2632063616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0710108640 con 0x7f0710107920 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.947+0000 7f070ffff700 1 --2- 172.21.15.28:0/2632063616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0710107920 0x7f0710107d40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f07000098a0 tx=0x7f0700009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b85d94c130ae1d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.948+0000 7f070effd700 1 -- 172.21.15.28:0/2632063616 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0700004160 con 0x7f0710107920 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.948+0000 7f070effd700 1 -- 172.21.15.28:0/2632063616 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0700004300 con 0x7f0710107920 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.948+0000 7f070effd700 1 -- 172.21.15.28:0/2632063616 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0700011af0 con 0x7f0710107920 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.949+0000 7f07162e3700 1 -- 172.21.15.28:0/2632063616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0710107920 msgr2=0x7f0710107d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.949+0000 7f07162e3700 1 --2- 172.21.15.28:0/2632063616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0710107920 0x7f0710107d40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f07000098a0 tx=0x7f0700009bb0 comp rx=0 tx=0).stop 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.949+0000 7f07162e3700 1 -- 172.21.15.28:0/2632063616 shutdown_connections 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.949+0000 7f07162e3700 1 --2- 172.21.15.28:0/2632063616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0710107920 0x7f0710107d40 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.949+0000 7f07162e3700 1 -- 172.21.15.28:0/2632063616 >> 172.21.15.28:0/2632063616 conn(0x7f0710102140 msgr2=0x7f0710104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.949+0000 7f07162e3700 1 -- 172.21.15.28:0/2632063616 shutdown_connections 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.949+0000 7f07162e3700 1 -- 172.21.15.28:0/2632063616 wait complete. 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.950+0000 7f07162e3700 1 Processor -- start 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.950+0000 7f07162e3700 1 -- start start 2024-04-21T08:40:41.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.950+0000 7f07162e3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0710107920 0x7f071019b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:41.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.950+0000 7f07162e3700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f071010b9f0 con 0x7f0710107920 2024-04-21T08:40:41.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.950+0000 7f070ffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0710107920 0x7f071019b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:41.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.950+0000 7f070ffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0710107920 0x7f071019b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55840/0 (socket says 172.21.15.28:55840) 2024-04-21T08:40:41.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.950+0000 7f070ffff700 1 -- 172.21.15.28:0/2349977546 learned_addr learned my addr 172.21.15.28:0/2349977546 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:41.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.951+0000 7f070ffff700 1 -- 172.21.15.28:0/2349977546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071019b920 con 0x7f0710107920 2024-04-21T08:40:41.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.951+0000 7f070ffff700 1 --2- 172.21.15.28:0/2349977546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0710107920 0x7f071019b3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0700009fb0 tx=0x7f0700011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:41.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.951+0000 7f070d7fa700 1 -- 172.21.15.28:0/2349977546 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07000036a0 con 0x7f0710107920 2024-04-21T08:40:41.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.951+0000 7f070d7fa700 1 -- 172.21.15.28:0/2349977546 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f070001a430 con 0x7f0710107920 2024-04-21T08:40:41.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.951+0000 7f07162e3700 1 -- 172.21.15.28:0/2349977546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f071019bbb0 con 0x7f0710107920 2024-04-21T08:40:41.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.951+0000 7f070d7fa700 1 -- 172.21.15.28:0/2349977546 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0700019550 con 0x7f0710107920 2024-04-21T08:40:41.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.951+0000 7f07162e3700 1 -- 172.21.15.28:0/2349977546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f071019e8a0 con 0x7f0710107920 2024-04-21T08:40:41.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.951+0000 7f070d7fa700 1 -- 172.21.15.28:0/2349977546 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0700019b90 con 0x7f0710107920 2024-04-21T08:40:41.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.951+0000 7f070d7fa700 1 -- 172.21.15.28:0/2349977546 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f07000248a0 con 0x7f0710107920 2024-04-21T08:40:41.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.952+0000 7f07162e3700 1 -- 172.21.15.28:0/2349977546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07100640a0 con 0x7f0710107920 2024-04-21T08:40:41.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:40.956+0000 7f070d7fa700 1 -- 172.21.15.28:0/2349977546 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f0700043e50 con 0x7f0710107920 2024-04-21T08:40:41.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:41.039+0000 7f07162e3700 1 -- 172.21.15.28:0/2349977546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f0710106850 con 0x7f0710107920 2024-04-21T08:40:41.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:41.039+0000 7f070d7fa700 1 -- 172.21.15.28:0/2349977546 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f0700032020 con 0x7f0710107920 2024-04-21T08:40:41.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:41.040+0000 7f07162e3700 1 -- 172.21.15.28:0/2349977546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0710107920 msgr2=0x7f071019b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:41.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:41.040+0000 7f07162e3700 1 --2- 172.21.15.28:0/2349977546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0710107920 0x7f071019b3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0700009fb0 tx=0x7f0700011940 comp rx=0 tx=0).stop 2024-04-21T08:40:41.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:41.041+0000 7f07162e3700 1 -- 172.21.15.28:0/2349977546 shutdown_connections 2024-04-21T08:40:41.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:41.041+0000 7f07162e3700 1 --2- 172.21.15.28:0/2349977546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0710107920 0x7f071019b3e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:41.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:41.041+0000 7f07162e3700 1 -- 172.21.15.28:0/2349977546 >> 172.21.15.28:0/2349977546 conn(0x7f0710102140 msgr2=0x7f0710102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:41.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:41.041+0000 7f07162e3700 1 -- 172.21.15.28:0/2349977546 shutdown_connections 2024-04-21T08:40:41.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:41.041+0000 7f07162e3700 1 -- 172.21.15.28:0/2349977546 wait complete. 2024-04-21T08:40:41.634 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (1/15)... 2024-04-21T08:40:42.854 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:42 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:42.526+0000 7f6eeabc5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:40:42.854 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:42 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:42.702+0000 7f6eeabc5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:40:43.183 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:42 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:42.879+0000 7f6eeabc5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:40:43.183 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:43 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:43.182+0000 7f6eeabc5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:40:43.489 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:43 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:43.277+0000 7f6eeabc5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:40:43.489 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:43 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:43.488+0000 7f6eeabc5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:40:43.747 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:43 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:43.702+0000 7f6eeabc5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:40:44.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:44 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2926927138' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:40:44.604 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:44 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:44.348+0000 7f6eeabc5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:40:44.707 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-21T08:40:44.707 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:40:44.707 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "b51a76c8-ffba-11ee-bc93-c7b262605968", 2024-04-21T08:40:44.707 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:40:44.707 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:40:44.707 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:40:44.707 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:40:44.707 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:44.707 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-21T08:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:44.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:40:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:44.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:40:44.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:44.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:40:44.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:40:44.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:44.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:40:44.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T08:40:44.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:40:44.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:40:44.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:40:44.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:40:44.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:40:44.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:40:44.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:40:44.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:44.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:40:44.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:44.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:40:33.405908+0000", 2024-04-21T08:40:44.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:40:44.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:44.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:40:44.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:40:44.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.044+0000 7f66217bb700 1 Processor -- start 2024-04-21T08:40:44.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.044+0000 7f66217bb700 1 -- start start 2024-04-21T08:40:44.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.045+0000 7f66217bb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f661c108590 0x7f661c1089b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:44.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.045+0000 7f66217bb700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f661c108f80 con 0x7f661c108590 2024-04-21T08:40:44.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.045+0000 7f661affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f661c108590 0x7f661c1089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:44.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.045+0000 7f661affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f661c108590 0x7f661c1089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49690/0 (socket says 172.21.15.28:49690) 2024-04-21T08:40:44.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.045+0000 7f661affd700 1 -- 172.21.15.28:0/4231847564 learned_addr learned my addr 172.21.15.28:0/4231847564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:44.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.045+0000 7f661affd700 1 -- 172.21.15.28:0/4231847564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f661c109740 con 0x7f661c108590 2024-04-21T08:40:44.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.045+0000 7f661affd700 1 --2- 172.21.15.28:0/4231847564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f661c108590 0x7f661c1089b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f660c0098a0 tx=0x7f660c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e1b8eccce11fea server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:44.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.045+0000 7f6619ffb700 1 -- 172.21.15.28:0/4231847564 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f660c004160 con 0x7f661c108590 2024-04-21T08:40:44.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.045+0000 7f6619ffb700 1 -- 172.21.15.28:0/4231847564 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f660c004300 con 0x7f661c108590 2024-04-21T08:40:44.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.045+0000 7f6619ffb700 1 -- 172.21.15.28:0/4231847564 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f660c011af0 con 0x7f661c108590 2024-04-21T08:40:44.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.046+0000 7f66217bb700 1 -- 172.21.15.28:0/4231847564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f661c108590 msgr2=0x7f661c1089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:44.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.046+0000 7f66217bb700 1 --2- 172.21.15.28:0/4231847564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f661c108590 0x7f661c1089b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f660c0098a0 tx=0x7f660c009bb0 comp rx=0 tx=0).stop 2024-04-21T08:40:44.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.046+0000 7f66217bb700 1 -- 172.21.15.28:0/4231847564 shutdown_connections 2024-04-21T08:40:44.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.046+0000 7f66217bb700 1 --2- 172.21.15.28:0/4231847564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f661c108590 0x7f661c1089b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:44.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.046+0000 7f66217bb700 1 -- 172.21.15.28:0/4231847564 >> 172.21.15.28:0/4231847564 conn(0x7f661c103b70 msgr2=0x7f661c105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:44.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.046+0000 7f66217bb700 1 -- 172.21.15.28:0/4231847564 shutdown_connections 2024-04-21T08:40:44.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.046+0000 7f66217bb700 1 -- 172.21.15.28:0/4231847564 wait complete. 2024-04-21T08:40:44.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.047+0000 7f66217bb700 1 Processor -- start 2024-04-21T08:40:44.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.047+0000 7f66217bb700 1 -- start start 2024-04-21T08:40:44.712 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.047+0000 7f66217bb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f661c108590 0x7f661c19c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:44.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.047+0000 7f66217bb700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f661c10caf0 con 0x7f661c108590 2024-04-21T08:40:44.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.047+0000 7f661affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f661c108590 0x7f661c19c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:44.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.047+0000 7f661affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f661c108590 0x7f661c19c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49706/0 (socket says 172.21.15.28:49706) 2024-04-21T08:40:44.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.047+0000 7f661affd700 1 -- 172.21.15.28:0/2926927138 learned_addr learned my addr 172.21.15.28:0/2926927138 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:44.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.048+0000 7f661affd700 1 -- 172.21.15.28:0/2926927138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f661c19ca80 con 0x7f661c108590 2024-04-21T08:40:44.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.048+0000 7f661affd700 1 --2- 172.21.15.28:0/2926927138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f661c108590 0x7f661c19c540 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f660c009fb0 tx=0x7f660c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:44.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.048+0000 7f6603fff700 1 -- 172.21.15.28:0/2926927138 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f660c0036a0 con 0x7f661c108590 2024-04-21T08:40:44.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.048+0000 7f6603fff700 1 -- 172.21.15.28:0/2926927138 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f660c01a430 con 0x7f661c108590 2024-04-21T08:40:44.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.048+0000 7f66217bb700 1 -- 172.21.15.28:0/2926927138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f661c19cd10 con 0x7f661c108590 2024-04-21T08:40:44.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.048+0000 7f6603fff700 1 -- 172.21.15.28:0/2926927138 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f660c019550 con 0x7f661c108590 2024-04-21T08:40:44.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.048+0000 7f66217bb700 1 -- 172.21.15.28:0/2926927138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f661c19fa00 con 0x7f661c108590 2024-04-21T08:40:44.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.048+0000 7f6603fff700 1 -- 172.21.15.28:0/2926927138 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f660c019be0 con 0x7f661c108590 2024-04-21T08:40:44.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.048+0000 7f6603fff700 1 -- 172.21.15.28:0/2926927138 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f660c024730 con 0x7f661c108590 2024-04-21T08:40:44.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.049+0000 7f66217bb700 1 -- 172.21.15.28:0/2926927138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f661c0640a0 con 0x7f661c108590 2024-04-21T08:40:44.713 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.051+0000 7f6603fff700 1 -- 172.21.15.28:0/2926927138 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f660c043e50 con 0x7f661c108590 2024-04-21T08:40:44.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.125+0000 7f66217bb700 1 -- 172.21.15.28:0/2926927138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f661c19ff40 con 0x7f661c108590 2024-04-21T08:40:44.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.125+0000 7f6603fff700 1 -- 172.21.15.28:0/2926927138 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f660c032020 con 0x7f661c108590 2024-04-21T08:40:44.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.126+0000 7f66217bb700 1 -- 172.21.15.28:0/2926927138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f661c108590 msgr2=0x7f661c19c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:44.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.126+0000 7f66217bb700 1 --2- 172.21.15.28:0/2926927138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f661c108590 0x7f661c19c540 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f660c009fb0 tx=0x7f660c011940 comp rx=0 tx=0).stop 2024-04-21T08:40:44.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.126+0000 7f66217bb700 1 -- 172.21.15.28:0/2926927138 shutdown_connections 2024-04-21T08:40:44.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.126+0000 7f66217bb700 1 --2- 172.21.15.28:0/2926927138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f661c108590 0x7f661c19c540 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:44.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.126+0000 7f66217bb700 1 -- 172.21.15.28:0/2926927138 >> 172.21.15.28:0/2926927138 conn(0x7f661c103b70 msgr2=0x7f661c104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:44.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.126+0000 7f66217bb700 1 -- 172.21.15.28:0/2926927138 shutdown_connections 2024-04-21T08:40:44.714 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:44.126+0000 7f66217bb700 1 -- 172.21.15.28:0/2926927138 wait complete. 2024-04-21T08:40:44.714 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (2/15)... 2024-04-21T08:40:45.099 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:44 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:44.790+0000 7f6eeabc5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:40:45.099 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:44 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:44.890+0000 7f6eeabc5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:40:45.099 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:45 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:45.099+0000 7f6eeabc5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:40:45.997 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:45 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:45.648+0000 7f6eeabc5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:40:45.997 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:45 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:45.744+0000 7f6eeabc5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:40:46.354 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:45 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:45.997+0000 7f6eeabc5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:40:46.354 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:46 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:46.137+0000 7f6eeabc5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:40:46.355 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:46 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:46.236+0000 7f6eeabc5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:40:46.671 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:46 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:46.358+0000 7f6eeabc5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:40:46.671 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:46 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:46.671+0000 7f6eeabc5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:40:47.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:47 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/4203426388' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:40:47.604 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:47 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:47.320+0000 7f6eeabc5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:40:47.605 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:40:47 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:40:47.549+0000 7f6eeabc5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:40:48.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:48 smithi028 ceph-mon[28003]: Activating manager daemon x 2024-04-21T08:40:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:48 smithi028 ceph-mon[28003]: mgrmap e2: x(active, starting, since 0.00327883s) 2024-04-21T08:40:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:48 smithi028 ceph-mon[28003]: from='mgr.14100 172.21.15.28:0/595789386' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:40:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:48 smithi028 ceph-mon[28003]: from='mgr.14100 172.21.15.28:0/595789386' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:40:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:48 smithi028 ceph-mon[28003]: from='mgr.14100 172.21.15.28:0/595789386' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:40:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:48 smithi028 ceph-mon[28003]: from='mgr.14100 172.21.15.28:0/595789386' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:40:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:48 smithi028 ceph-mon[28003]: from='mgr.14100 172.21.15.28:0/595789386' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:40:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:48 smithi028 ceph-mon[28003]: Manager daemon x is now available 2024-04-21T08:40:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:48 smithi028 ceph-mon[28003]: from='mgr.14100 172.21.15.28:0/595789386' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:40:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:48 smithi028 ceph-mon[28003]: from='mgr.14100 172.21.15.28:0/595789386' entity='mgr.x' 2024-04-21T08:40:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:48 smithi028 ceph-mon[28003]: from='mgr.14100 172.21.15.28:0/595789386' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:40:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:48 smithi028 ceph-mon[28003]: from='mgr.14100 172.21.15.28:0/595789386' entity='mgr.x' 2024-04-21T08:40:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:48 smithi028 ceph-mon[28003]: from='mgr.14100 172.21.15.28:0/595789386' entity='mgr.x' 2024-04-21T08:40:49.742 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-21T08:40:49.742 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:40:49.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "b51a76c8-ffba-11ee-bc93-c7b262605968", 2024-04-21T08:40:49.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:40:49.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:40:49.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:40:49.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:40:49.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:49.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:40:49.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:40:49.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:40:49.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:40:49.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:40:49.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:40:49.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:40:49.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-21T08:40:49.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:40:49.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:49.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:40:49.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:40:49.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:49.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:40:49.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:49.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:40:49.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:40:49.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:40:49.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:40:49.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:40:49.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:40:49.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:49.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:40:49.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:40:49.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:40:49.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:40:49.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:40:49.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:40:49.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:40:49.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:40:49.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:40:49.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:49.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:40:49.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:49.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:40:49.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:40:49.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:49.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:40:49.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T08:40:49.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:40:49.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:40:49.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:40:49.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:40:49.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:40:49.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:40:49.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:40:49.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:49.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:40:49.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:49.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:40:33.405908+0000", 2024-04-21T08:40:49.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:40:49.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:49.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:40:49.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:40:49.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.072+0000 7f4f41e1f700 1 Processor -- start 2024-04-21T08:40:49.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.072+0000 7f4f41e1f700 1 -- start start 2024-04-21T08:40:49.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.073+0000 7f4f41e1f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f3c07cda0 0x7f4f3c07b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.073+0000 7f4f41e1f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c07b7d0 con 0x7f4f3c07cda0 2024-04-21T08:40:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.073+0000 7f4f3b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f3c07cda0 0x7f4f3c07b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.073+0000 7f4f3b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f3c07cda0 0x7f4f3c07b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49720/0 (socket says 172.21.15.28:49720) 2024-04-21T08:40:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.073+0000 7f4f3b7fe700 1 -- 172.21.15.28:0/3667767625 learned_addr learned my addr 172.21.15.28:0/3667767625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.073+0000 7f4f3b7fe700 1 -- 172.21.15.28:0/3667767625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3c07b950 con 0x7f4f3c07cda0 2024-04-21T08:40:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.073+0000 7f4f3b7fe700 1 --2- 172.21.15.28:0/3667767625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f3c07cda0 0x7f4f3c07b200 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4f240098a0 tx=0x7f4f24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a5324054794a17 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.074+0000 7f4f3a7fc700 1 -- 172.21.15.28:0/3667767625 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f24004160 con 0x7f4f3c07cda0 2024-04-21T08:40:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.074+0000 7f4f3a7fc700 1 -- 172.21.15.28:0/3667767625 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4f24004300 con 0x7f4f3c07cda0 2024-04-21T08:40:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.074+0000 7f4f3a7fc700 1 -- 172.21.15.28:0/3667767625 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f24011af0 con 0x7f4f3c07cda0 2024-04-21T08:40:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.074+0000 7f4f41e1f700 1 -- 172.21.15.28:0/3667767625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f3c07cda0 msgr2=0x7f4f3c07b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.074+0000 7f4f41e1f700 1 --2- 172.21.15.28:0/3667767625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f3c07cda0 0x7f4f3c07b200 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4f240098a0 tx=0x7f4f24009bb0 comp rx=0 tx=0).stop 2024-04-21T08:40:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.074+0000 7f4f41e1f700 1 -- 172.21.15.28:0/3667767625 shutdown_connections 2024-04-21T08:40:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.074+0000 7f4f41e1f700 1 --2- 172.21.15.28:0/3667767625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f3c07cda0 0x7f4f3c07b200 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.074+0000 7f4f41e1f700 1 -- 172.21.15.28:0/3667767625 >> 172.21.15.28:0/3667767625 conn(0x7f4f3c103b70 msgr2=0x7f4f3c105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:49.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.074+0000 7f4f41e1f700 1 -- 172.21.15.28:0/3667767625 shutdown_connections 2024-04-21T08:40:49.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.074+0000 7f4f41e1f700 1 -- 172.21.15.28:0/3667767625 wait complete. 2024-04-21T08:40:49.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.075+0000 7f4f41e1f700 1 Processor -- start 2024-04-21T08:40:49.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.075+0000 7f4f41e1f700 1 -- start start 2024-04-21T08:40:49.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.076+0000 7f4f41e1f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f3c07cda0 0x7f4f3c19e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:49.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.076+0000 7f4f41e1f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c10eae0 con 0x7f4f3c07cda0 2024-04-21T08:40:49.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.076+0000 7f4f3b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f3c07cda0 0x7f4f3c19e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:49.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.076+0000 7f4f3b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f3c07cda0 0x7f4f3c19e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49732/0 (socket says 172.21.15.28:49732) 2024-04-21T08:40:49.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.076+0000 7f4f3b7fe700 1 -- 172.21.15.28:0/4203426388 learned_addr learned my addr 172.21.15.28:0/4203426388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:49.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.076+0000 7f4f3b7fe700 1 -- 172.21.15.28:0/4203426388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3c19ec00 con 0x7f4f3c07cda0 2024-04-21T08:40:49.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.076+0000 7f4f3b7fe700 1 --2- 172.21.15.28:0/4203426388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f3c07cda0 0x7f4f3c19e6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4f24000c00 tx=0x7f4f24011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:49.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.076+0000 7f4f38ff9700 1 -- 172.21.15.28:0/4203426388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f24003b20 con 0x7f4f3c07cda0 2024-04-21T08:40:49.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.076+0000 7f4f38ff9700 1 -- 172.21.15.28:0/4203426388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4f2401a8c0 con 0x7f4f3c07cda0 2024-04-21T08:40:49.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.076+0000 7f4f41e1f700 1 -- 172.21.15.28:0/4203426388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3c19ee90 con 0x7f4f3c07cda0 2024-04-21T08:40:49.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.076+0000 7f4f41e1f700 1 -- 172.21.15.28:0/4203426388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f3c1a1b80 con 0x7f4f3c07cda0 2024-04-21T08:40:49.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.076+0000 7f4f38ff9700 1 -- 172.21.15.28:0/4203426388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f24024940 con 0x7f4f3c07cda0 2024-04-21T08:40:49.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.077+0000 7f4f38ff9700 1 -- 172.21.15.28:0/4203426388 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4f24022020 con 0x7f4f3c07cda0 2024-04-21T08:40:49.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.077+0000 7f4f38ff9700 1 -- 172.21.15.28:0/4203426388 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4f240193c0 con 0x7f4f3c07cda0 2024-04-21T08:40:49.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.077+0000 7f4f41e1f700 1 -- 172.21.15.28:0/4203426388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f1c005360 con 0x7f4f3c07cda0 2024-04-21T08:40:49.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.080+0000 7f4f38ff9700 1 -- 172.21.15.28:0/4203426388 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4f2401aa70 con 0x7f4f3c07cda0 2024-04-21T08:40:49.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.153+0000 7f4f41e1f700 1 -- 172.21.15.28:0/4203426388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f4f1c005920 con 0x7f4f3c07cda0 2024-04-21T08:40:49.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.154+0000 7f4f38ff9700 1 -- 172.21.15.28:0/4203426388 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f4f2401a430 con 0x7f4f3c07cda0 2024-04-21T08:40:49.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.155+0000 7f4f41e1f700 1 -- 172.21.15.28:0/4203426388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f3c07cda0 msgr2=0x7f4f3c19e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:49.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.155+0000 7f4f41e1f700 1 --2- 172.21.15.28:0/4203426388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f3c07cda0 0x7f4f3c19e6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4f24000c00 tx=0x7f4f24011b70 comp rx=0 tx=0).stop 2024-04-21T08:40:49.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.155+0000 7f4f41e1f700 1 -- 172.21.15.28:0/4203426388 shutdown_connections 2024-04-21T08:40:49.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.155+0000 7f4f41e1f700 1 --2- 172.21.15.28:0/4203426388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f3c07cda0 0x7f4f3c19e6c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:49.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.155+0000 7f4f41e1f700 1 -- 172.21.15.28:0/4203426388 >> 172.21.15.28:0/4203426388 conn(0x7f4f3c103b70 msgr2=0x7f4f3c104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:49.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.155+0000 7f4f41e1f700 1 -- 172.21.15.28:0/4203426388 shutdown_connections 2024-04-21T08:40:49.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:47.155+0000 7f4f41e1f700 1 -- 172.21.15.28:0/4203426388 wait complete. 2024-04-21T08:40:49.759 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (3/15)... 2024-04-21T08:40:49.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:49 smithi028 ceph-mon[28003]: mgrmap e3: x(active, since 1.00779s) 2024-04-21T08:40:50.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:50 smithi028 ceph-mon[28003]: mgrmap e4: x(active, since 2s) 2024-04-21T08:40:56.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:55 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/4258565153' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:40:58.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-21T08:40:58.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:40:58.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "b51a76c8-ffba-11ee-bc93-c7b262605968", 2024-04-21T08:40:58.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:40:58.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:40:58.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:40:58.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:40:58.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:58.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:40:58.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:40:58.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:40:58.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:40:58.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:40:58.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:40:58.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:40:58.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2024-04-21T08:40:58.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:40:58.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:58.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:40:58.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:40:58.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:58.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:40:58.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:58.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:40:58.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:40:58.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:40:58.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:40:58.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:40:58.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:40:58.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:58.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:40:58.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:40:58.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:40:58.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:40:58.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:40:58.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:40:58.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:40:58.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:40:58.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:40:58.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:58.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:40:58.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:58.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:40:58.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:40:58.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:58.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:40:58.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T08:40:58.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:40:58.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:40:58.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:40:58.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:40:58.498 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:40:58.499 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:40:58.499 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:40:58.499 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:58.499 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:40:58.499 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:40:58.499 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:40:33.405908+0000", 2024-04-21T08:40:58.499 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:40:58.499 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:40:58.499 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:40:58.499 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:40:58.499 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.313+0000 7f4ef19b9700 1 Processor -- start 2024-04-21T08:40:58.499 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.313+0000 7f4ef19b9700 1 -- start start 2024-04-21T08:40:58.499 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.313+0000 7f4ef19b9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eec109000 0x7f4eec10b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:58.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.313+0000 7f4ef19b9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec063d20 con 0x7f4eec109000 2024-04-21T08:40:58.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.314+0000 7f4eeaffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eec109000 0x7f4eec10b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:58.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.314+0000 7f4eeaffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eec109000 0x7f4eec10b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36596/0 (socket says 172.21.15.28:36596) 2024-04-21T08:40:58.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.314+0000 7f4eeaffd700 1 -- 172.21.15.28:0/3866754188 learned_addr learned my addr 172.21.15.28:0/3866754188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:58.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.314+0000 7f4eeaffd700 1 -- 172.21.15.28:0/3866754188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eec10b930 con 0x7f4eec109000 2024-04-21T08:40:58.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.314+0000 7f4eeaffd700 1 --2- 172.21.15.28:0/3866754188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eec109000 0x7f4eec10b3f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4edc0098a0 tx=0x7f4edc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1574c88c766235c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:58.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.314+0000 7f4ee9ffb700 1 -- 172.21.15.28:0/3866754188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4edc004160 con 0x7f4eec109000 2024-04-21T08:40:58.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.314+0000 7f4ee9ffb700 1 -- 172.21.15.28:0/3866754188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4edc004300 con 0x7f4eec109000 2024-04-21T08:40:58.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.314+0000 7f4ee9ffb700 1 -- 172.21.15.28:0/3866754188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4edc011af0 con 0x7f4eec109000 2024-04-21T08:40:58.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.315+0000 7f4ef19b9700 1 -- 172.21.15.28:0/3866754188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eec109000 msgr2=0x7f4eec10b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:58.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.315+0000 7f4ef19b9700 1 --2- 172.21.15.28:0/3866754188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eec109000 0x7f4eec10b3f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4edc0098a0 tx=0x7f4edc009bb0 comp rx=0 tx=0).stop 2024-04-21T08:40:58.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.315+0000 7f4ef19b9700 1 -- 172.21.15.28:0/3866754188 shutdown_connections 2024-04-21T08:40:58.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.315+0000 7f4ef19b9700 1 --2- 172.21.15.28:0/3866754188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eec109000 0x7f4eec10b3f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:58.500 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.315+0000 7f4ef19b9700 1 -- 172.21.15.28:0/3866754188 >> 172.21.15.28:0/3866754188 conn(0x7f4eec1029f0 msgr2=0x7f4eec104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.315+0000 7f4ef19b9700 1 -- 172.21.15.28:0/3866754188 shutdown_connections 2024-04-21T08:40:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.315+0000 7f4ef19b9700 1 -- 172.21.15.28:0/3866754188 wait complete. 2024-04-21T08:40:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.316+0000 7f4ef19b9700 1 Processor -- start 2024-04-21T08:40:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.316+0000 7f4ef19b9700 1 -- start start 2024-04-21T08:40:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.316+0000 7f4ef19b9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eec109000 0x7f4eec108880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.316+0000 7f4ef19b9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec10c8f0 con 0x7f4eec109000 2024-04-21T08:40:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.316+0000 7f4eeaffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eec109000 0x7f4eec108880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.316+0000 7f4eeaffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eec109000 0x7f4eec108880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36608/0 (socket says 172.21.15.28:36608) 2024-04-21T08:40:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.316+0000 7f4eeaffd700 1 -- 172.21.15.28:0/4258565153 learned_addr learned my addr 172.21.15.28:0/4258565153 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.316+0000 7f4eeaffd700 1 -- 172.21.15.28:0/4258565153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eec108dc0 con 0x7f4eec109000 2024-04-21T08:40:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.316+0000 7f4eeaffd700 1 --2- 172.21.15.28:0/4258565153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eec109000 0x7f4eec108880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4edc003f40 tx=0x7f4edc011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.317+0000 7f4ef09b7700 1 -- 172.21.15.28:0/4258565153 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4edc0038b0 con 0x7f4eec109000 2024-04-21T08:40:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.317+0000 7f4ef09b7700 1 -- 172.21.15.28:0/4258565153 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4edc003a50 con 0x7f4eec109000 2024-04-21T08:40:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.317+0000 7f4ef19b9700 1 -- 172.21.15.28:0/4258565153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eec106fb0 con 0x7f4eec109000 2024-04-21T08:40:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.317+0000 7f4ef09b7700 1 -- 172.21.15.28:0/4258565153 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4edc01a5c0 con 0x7f4eec109000 2024-04-21T08:40:58.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.317+0000 7f4ef19b9700 1 -- 172.21.15.28:0/4258565153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eec1074f0 con 0x7f4eec109000 2024-04-21T08:40:58.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.317+0000 7f4ef09b7700 1 -- 172.21.15.28:0/4258565153 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f4edc01a870 con 0x7f4eec109000 2024-04-21T08:40:58.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.318+0000 7f4ef19b9700 1 -- 172.21.15.28:0/4258565153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eec0640a0 con 0x7f4eec109000 2024-04-21T08:40:58.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.318+0000 7f4ef09b7700 1 --2- 172.21.15.28:0/4258565153 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f4ed4039560 0x7f4ed403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:58.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.318+0000 7f4ef09b7700 1 -- 172.21.15.28:0/4258565153 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4edc04da60 con 0x7f4eec109000 2024-04-21T08:40:58.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.318+0000 7f4eea7fc700 1 --2- 172.21.15.28:0/4258565153 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f4ed4039560 0x7f4ed403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:58.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.318+0000 7f4eea7fc700 1 --2- 172.21.15.28:0/4258565153 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f4ed4039560 0x7f4ed403ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0006fd0 tx=0x7f4ee0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:58.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.323+0000 7f4ef09b7700 1 -- 172.21.15.28:0/4258565153 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4edc019c70 con 0x7f4eec109000 2024-04-21T08:40:58.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.621+0000 7f4ef19b9700 1 -- 172.21.15.28:0/4258565153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f4eec107770 con 0x7f4eec109000 2024-04-21T08:40:58.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.622+0000 7f4ef09b7700 1 -- 172.21.15.28:0/4258565153 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f4edc04d480 con 0x7f4eec109000 2024-04-21T08:40:58.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.624+0000 7f4ef19b9700 1 -- 172.21.15.28:0/4258565153 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f4ed4039560 msgr2=0x7f4ed403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:58.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.624+0000 7f4ef19b9700 1 --2- 172.21.15.28:0/4258565153 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f4ed4039560 0x7f4ed403ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0006fd0 tx=0x7f4ee0006e40 comp rx=0 tx=0).stop 2024-04-21T08:40:58.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.624+0000 7f4ef19b9700 1 -- 172.21.15.28:0/4258565153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eec109000 msgr2=0x7f4eec108880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:58.502 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.624+0000 7f4ef19b9700 1 --2- 172.21.15.28:0/4258565153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eec109000 0x7f4eec108880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4edc003f40 tx=0x7f4edc011890 comp rx=0 tx=0).stop 2024-04-21T08:40:58.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.624+0000 7f4ef19b9700 1 -- 172.21.15.28:0/4258565153 shutdown_connections 2024-04-21T08:40:58.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.624+0000 7f4ef19b9700 1 --2- 172.21.15.28:0/4258565153 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f4ed4039560 0x7f4ed403ba20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:58.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.624+0000 7f4ef19b9700 1 --2- 172.21.15.28:0/4258565153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eec109000 0x7f4eec108880 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:58.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.624+0000 7f4ef19b9700 1 -- 172.21.15.28:0/4258565153 >> 172.21.15.28:0/4258565153 conn(0x7f4eec1029f0 msgr2=0x7f4eec103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:58.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.624+0000 7f4ef19b9700 1 -- 172.21.15.28:0/4258565153 shutdown_connections 2024-04-21T08:40:58.503 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:55.624+0000 7f4ef19b9700 1 -- 172.21.15.28:0/4258565153 wait complete. 2024-04-21T08:40:58.503 INFO:teuthology.orchestra.run.smithi028.stdout:mgr is available 2024-04-21T08:40:59.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:40:59 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1454044737' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-21T08:40:59.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-21T08:40:59.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-04-21T08:40:59.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:40:59.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-21T08:40:59.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-21T08:40:59.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-21T08:40:59.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-21T08:40:59.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-21T08:40:59.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-21T08:40:59.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-21T08:40:59.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-21T08:40:59.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-21T08:40:59.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-04-21T08:40:59.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-04-21T08:40:59.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-21T08:40:59.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-21T08:40:59.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-21T08:40:59.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-21T08:40:59.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-21T08:40:59.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.878+0000 7f72ed948700 1 Processor -- start 2024-04-21T08:40:59.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.878+0000 7f72ed948700 1 -- start start 2024-04-21T08:40:59.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.878+0000 7f72ed948700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8108590 0x7f72e81089b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:59.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.878+0000 7f72ed948700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72e8108f80 con 0x7f72e8108590 2024-04-21T08:40:59.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.878+0000 7f72e6ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8108590 0x7f72e81089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:59.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.878+0000 7f72e6ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8108590 0x7f72e81089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36614/0 (socket says 172.21.15.28:36614) 2024-04-21T08:40:59.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.878+0000 7f72e6ffd700 1 -- 172.21.15.28:0/3823921481 learned_addr learned my addr 172.21.15.28:0/3823921481 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:59.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.878+0000 7f72e6ffd700 1 -- 172.21.15.28:0/3823921481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e8109740 con 0x7f72e8108590 2024-04-21T08:40:59.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.879+0000 7f72e6ffd700 1 --2- 172.21.15.28:0/3823921481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8108590 0x7f72e81089b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f72d00098a0 tx=0x7f72d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de562bbddd4e5379 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:59.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.879+0000 7f72e5ffb700 1 -- 172.21.15.28:0/3823921481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72d0004160 con 0x7f72e8108590 2024-04-21T08:40:59.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.879+0000 7f72e5ffb700 1 -- 172.21.15.28:0/3823921481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f72d0004300 con 0x7f72e8108590 2024-04-21T08:40:59.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.879+0000 7f72e5ffb700 1 -- 172.21.15.28:0/3823921481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72d0011af0 con 0x7f72e8108590 2024-04-21T08:40:59.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.879+0000 7f72ed948700 1 -- 172.21.15.28:0/3823921481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8108590 msgr2=0x7f72e81089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:59.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.879+0000 7f72ed948700 1 --2- 172.21.15.28:0/3823921481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8108590 0x7f72e81089b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f72d00098a0 tx=0x7f72d0009bb0 comp rx=0 tx=0).stop 2024-04-21T08:40:59.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.879+0000 7f72ed948700 1 -- 172.21.15.28:0/3823921481 shutdown_connections 2024-04-21T08:40:59.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.879+0000 7f72ed948700 1 --2- 172.21.15.28:0/3823921481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8108590 0x7f72e81089b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:59.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.879+0000 7f72ed948700 1 -- 172.21.15.28:0/3823921481 >> 172.21.15.28:0/3823921481 conn(0x7f72e8103b70 msgr2=0x7f72e8105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:59.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.879+0000 7f72ed948700 1 -- 172.21.15.28:0/3823921481 shutdown_connections 2024-04-21T08:40:59.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.879+0000 7f72ed948700 1 -- 172.21.15.28:0/3823921481 wait complete. 2024-04-21T08:40:59.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.880+0000 7f72ed948700 1 Processor -- start 2024-04-21T08:40:59.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.880+0000 7f72ed948700 1 -- start start 2024-04-21T08:40:59.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.880+0000 7f72ed948700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8108590 0x7f72e819c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:59.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.880+0000 7f72ed948700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72e810caf0 con 0x7f72e8108590 2024-04-21T08:40:59.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.880+0000 7f72e6ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8108590 0x7f72e819c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:59.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.880+0000 7f72e6ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8108590 0x7f72e819c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36618/0 (socket says 172.21.15.28:36618) 2024-04-21T08:40:59.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.881+0000 7f72e6ffd700 1 -- 172.21.15.28:0/1454044737 learned_addr learned my addr 172.21.15.28:0/1454044737 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:40:59.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.881+0000 7f72e6ffd700 1 -- 172.21.15.28:0/1454044737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e819ca80 con 0x7f72e8108590 2024-04-21T08:40:59.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.881+0000 7f72e6ffd700 1 --2- 172.21.15.28:0/1454044737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8108590 0x7f72e819c540 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009fb0 tx=0x7f72d0011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:59.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.881+0000 7f72ec946700 1 -- 172.21.15.28:0/1454044737 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72d00036a0 con 0x7f72e8108590 2024-04-21T08:40:59.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.881+0000 7f72ec946700 1 -- 172.21.15.28:0/1454044737 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f72d001a430 con 0x7f72e8108590 2024-04-21T08:40:59.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.881+0000 7f72ed948700 1 -- 172.21.15.28:0/1454044737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e819cd10 con 0x7f72e8108590 2024-04-21T08:40:59.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.881+0000 7f72ec946700 1 -- 172.21.15.28:0/1454044737 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72d0019550 con 0x7f72e8108590 2024-04-21T08:40:59.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.881+0000 7f72ed948700 1 -- 172.21.15.28:0/1454044737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e819fa00 con 0x7f72e8108590 2024-04-21T08:40:59.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.882+0000 7f72ec946700 1 -- 172.21.15.28:0/1454044737 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f72d0019800 con 0x7f72e8108590 2024-04-21T08:40:59.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.882+0000 7f72ec946700 1 --2- 172.21.15.28:0/1454044737 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f72d40395b0 0x7f72d403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:59.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.882+0000 7f72ed948700 1 -- 172.21.15.28:0/1454044737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72e80640a0 con 0x7f72e8108590 2024-04-21T08:40:59.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.882+0000 7f72ec946700 1 -- 172.21.15.28:0/1454044737 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f72d004db10 con 0x7f72e8108590 2024-04-21T08:40:59.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.882+0000 7f72e67fc700 1 --2- 172.21.15.28:0/1454044737 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f72d40395b0 0x7f72d403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:59.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.883+0000 7f72e67fc700 1 --2- 172.21.15.28:0/1454044737 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f72d40395b0 0x7f72d403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f72d8006fd0 tx=0x7f72d8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:59.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:58.888+0000 7f72ec946700 1 -- 172.21.15.28:0/1454044737 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f72d004d530 con 0x7f72e8108590 2024-04-21T08:40:59.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:59.094+0000 7f72ed948700 1 -- 172.21.15.28:0/1454044737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f72e819fbd0 con 0x7f72e8108590 2024-04-21T08:40:59.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:59.095+0000 7f72ec946700 1 -- 172.21.15.28:0/1454044737 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+434 (secure 0 0 0) 0x7f72d0032020 con 0x7f72e8108590 2024-04-21T08:40:59.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:59.097+0000 7f72ed948700 1 -- 172.21.15.28:0/1454044737 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f72d40395b0 msgr2=0x7f72d403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:59.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:59.097+0000 7f72ed948700 1 --2- 172.21.15.28:0/1454044737 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f72d40395b0 0x7f72d403ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f72d8006fd0 tx=0x7f72d8006e40 comp rx=0 tx=0).stop 2024-04-21T08:40:59.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:59.097+0000 7f72ed948700 1 -- 172.21.15.28:0/1454044737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8108590 msgr2=0x7f72e819c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:59.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:59.097+0000 7f72ed948700 1 --2- 172.21.15.28:0/1454044737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8108590 0x7f72e819c540 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009fb0 tx=0x7f72d0011940 comp rx=0 tx=0).stop 2024-04-21T08:40:59.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:59.097+0000 7f72ed948700 1 -- 172.21.15.28:0/1454044737 shutdown_connections 2024-04-21T08:40:59.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:59.097+0000 7f72ed948700 1 --2- 172.21.15.28:0/1454044737 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f72d40395b0 0x7f72d403ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:59.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:59.097+0000 7f72ed948700 1 --2- 172.21.15.28:0/1454044737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8108590 0x7f72e819c540 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:59.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:59.097+0000 7f72ed948700 1 -- 172.21.15.28:0/1454044737 >> 172.21.15.28:0/1454044737 conn(0x7f72e8103b70 msgr2=0x7f72e8104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:59.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:59.097+0000 7f72ed948700 1 -- 172.21.15.28:0/1454044737 shutdown_connections 2024-04-21T08:40:59.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:40:59.097+0000 7f72ed948700 1 -- 172.21.15.28:0/1454044737 wait complete. 2024-04-21T08:40:59.669 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling cephadm module... 2024-04-21T08:41:00.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:00 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1139184154' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-21T08:41:00.725 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:00 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: ignoring --setuser ceph since I am not root 2024-04-21T08:41:00.725 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:00 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: ignoring --setgroup ceph since I am not root 2024-04-21T08:41:00.725 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:00 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:00.592+0000 7f4fd0754200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:41:01.017 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:00 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:00.789+0000 7f4fd0754200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:41:01.018 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:00 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:00.896+0000 7f4fd0754200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:41:01.031 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.078+0000 7f735ca54700 1 Processor -- start 2024-04-21T08:41:01.032 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.078+0000 7f735ca54700 1 -- start start 2024-04-21T08:41:01.032 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.079+0000 7f735ca54700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7358108fc0 0x7f735810b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:01.032 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.079+0000 7f735ca54700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7358063d60 con 0x7f7358108fc0 2024-04-21T08:41:01.032 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.079+0000 7f735659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7358108fc0 0x7f735810b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:01.032 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.079+0000 7f735659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7358108fc0 0x7f735810b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36630/0 (socket says 172.21.15.28:36630) 2024-04-21T08:41:01.032 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.079+0000 7f735659c700 1 -- 172.21.15.28:0/1385480518 learned_addr learned my addr 172.21.15.28:0/1385480518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:01.033 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.079+0000 7f735659c700 1 -- 172.21.15.28:0/1385480518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f735810b8f0 con 0x7f7358108fc0 2024-04-21T08:41:01.033 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.079+0000 7f735659c700 1 --2- 172.21.15.28:0/1385480518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7358108fc0 0x7f735810b3b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f73480098a0 tx=0x7f7348009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd1d2bfe9809048 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:01.033 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.079+0000 7f735559a700 1 -- 172.21.15.28:0/1385480518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7348004160 con 0x7f7358108fc0 2024-04-21T08:41:01.033 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.080+0000 7f735559a700 1 -- 172.21.15.28:0/1385480518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7348004300 con 0x7f7358108fc0 2024-04-21T08:41:01.033 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.080+0000 7f735559a700 1 -- 172.21.15.28:0/1385480518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7348011af0 con 0x7f7358108fc0 2024-04-21T08:41:01.033 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.080+0000 7f735ca54700 1 -- 172.21.15.28:0/1385480518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7358108fc0 msgr2=0x7f735810b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:01.034 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.080+0000 7f735ca54700 1 --2- 172.21.15.28:0/1385480518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7358108fc0 0x7f735810b3b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f73480098a0 tx=0x7f7348009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:01.034 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.080+0000 7f735ca54700 1 -- 172.21.15.28:0/1385480518 shutdown_connections 2024-04-21T08:41:01.034 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.080+0000 7f735ca54700 1 --2- 172.21.15.28:0/1385480518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7358108fc0 0x7f735810b3b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:01.034 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.080+0000 7f735ca54700 1 -- 172.21.15.28:0/1385480518 >> 172.21.15.28:0/1385480518 conn(0x7f73581029f0 msgr2=0x7f7358104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:01.034 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.080+0000 7f735ca54700 1 -- 172.21.15.28:0/1385480518 shutdown_connections 2024-04-21T08:41:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.080+0000 7f735ca54700 1 -- 172.21.15.28:0/1385480518 wait complete. 2024-04-21T08:41:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.081+0000 7f735ca54700 1 Processor -- start 2024-04-21T08:41:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.081+0000 7f735ca54700 1 -- start start 2024-04-21T08:41:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.081+0000 7f735ca54700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7358108fc0 0x7f735819c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.081+0000 7f735ca54700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f735810c8c0 con 0x7f7358108fc0 2024-04-21T08:41:01.035 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.081+0000 7f735659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7358108fc0 0x7f735819c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:01.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.081+0000 7f735659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7358108fc0 0x7f735819c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36646/0 (socket says 172.21.15.28:36646) 2024-04-21T08:41:01.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.081+0000 7f735659c700 1 -- 172.21.15.28:0/1139184154 learned_addr learned my addr 172.21.15.28:0/1139184154 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:01.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.082+0000 7f735659c700 1 -- 172.21.15.28:0/1139184154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f735819c9b0 con 0x7f7358108fc0 2024-04-21T08:41:01.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.082+0000 7f735659c700 1 --2- 172.21.15.28:0/1139184154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7358108fc0 0x7f735819c470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7348003f40 tx=0x7f7348011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:01.036 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.082+0000 7f73477fe700 1 -- 172.21.15.28:0/1139184154 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73480038b0 con 0x7f7358108fc0 2024-04-21T08:41:01.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.082+0000 7f73477fe700 1 -- 172.21.15.28:0/1139184154 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7348003a50 con 0x7f7358108fc0 2024-04-21T08:41:01.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.082+0000 7f735ca54700 1 -- 172.21.15.28:0/1139184154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f735819cc40 con 0x7f7358108fc0 2024-04-21T08:41:01.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.082+0000 7f73477fe700 1 -- 172.21.15.28:0/1139184154 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f734801a5c0 con 0x7f7358108fc0 2024-04-21T08:41:01.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.082+0000 7f735ca54700 1 -- 172.21.15.28:0/1139184154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f735819f930 con 0x7f7358108fc0 2024-04-21T08:41:01.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.083+0000 7f73477fe700 1 -- 172.21.15.28:0/1139184154 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f734801a8c0 con 0x7f7358108fc0 2024-04-21T08:41:01.037 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.083+0000 7f735ca54700 1 -- 172.21.15.28:0/1139184154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73580640a0 con 0x7f7358108fc0 2024-04-21T08:41:01.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.083+0000 7f73477fe700 1 --2- 172.21.15.28:0/1139184154 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f7340039510 0x7f734003b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:01.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.083+0000 7f73477fe700 1 -- 172.21.15.28:0/1139184154 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f734804dc10 con 0x7f7358108fc0 2024-04-21T08:41:01.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.083+0000 7f7355d9b700 1 --2- 172.21.15.28:0/1139184154 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f7340039510 0x7f734003b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:01.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.084+0000 7f7355d9b700 1 --2- 172.21.15.28:0/1139184154 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f7340039510 0x7f734003b9d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f734c006fd0 tx=0x7f734c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:01.038 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.088+0000 7f73477fe700 1 -- 172.21.15.28:0/1139184154 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7348019c70 con 0x7f7358108fc0 2024-04-21T08:41:01.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.360+0000 7f735ca54700 1 -- 172.21.15.28:0/1139184154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f735819fb00 con 0x7f7358108fc0 2024-04-21T08:41:01.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.412+0000 7f73477fe700 1 -- 172.21.15.28:0/1139184154 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f735819fb00 con 0x7f7358108fc0 2024-04-21T08:41:01.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.413+0000 7f73477fe700 1 -- 172.21.15.28:0/1139184154 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f734802d910 con 0x7f7358108fc0 2024-04-21T08:41:01.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.415+0000 7f735ca54700 1 -- 172.21.15.28:0/1139184154 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f7340039510 msgr2=0x7f734003b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:01.039 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.415+0000 7f735ca54700 1 --2- 172.21.15.28:0/1139184154 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f7340039510 0x7f734003b9d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f734c006fd0 tx=0x7f734c006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:01.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.415+0000 7f735ca54700 1 -- 172.21.15.28:0/1139184154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7358108fc0 msgr2=0x7f735819c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:01.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.415+0000 7f735ca54700 1 --2- 172.21.15.28:0/1139184154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7358108fc0 0x7f735819c470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7348003f40 tx=0x7f7348011890 comp rx=0 tx=0).stop 2024-04-21T08:41:01.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.415+0000 7f735ca54700 1 -- 172.21.15.28:0/1139184154 shutdown_connections 2024-04-21T08:41:01.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.415+0000 7f735ca54700 1 --2- 172.21.15.28:0/1139184154 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7f7340039510 0x7f734003b9d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:01.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.415+0000 7f735ca54700 1 --2- 172.21.15.28:0/1139184154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7358108fc0 0x7f735819c470 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:01.040 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.415+0000 7f735ca54700 1 -- 172.21.15.28:0/1139184154 >> 172.21.15.28:0/1139184154 conn(0x7f73581029f0 msgr2=0x7f7358103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:01.041 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.415+0000 7f735ca54700 1 -- 172.21.15.28:0/1139184154 shutdown_connections 2024-04-21T08:41:01.041 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:00.415+0000 7f735ca54700 1 -- 172.21.15.28:0/1139184154 wait complete. 2024-04-21T08:41:01.314 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:01 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:01.223+0000 7f4fd0754200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:41:01.314 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:01 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:01.315+0000 7f4fd0754200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:41:01.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:01 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1139184154' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-21T08:41:01.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:01 smithi028 ceph-mon[28003]: mgrmap e5: x(active, since 12s) 2024-04-21T08:41:02.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:02 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1369869457' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-21T08:41:02.854 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:02 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:02.578+0000 7f4fd0754200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:41:02.854 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:02 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:02.756+0000 7f4fd0754200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:41:02.875 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:41:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-21T08:41:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T08:41:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-21T08:41:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-21T08:41:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:41:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.023+0000 7fdceece5700 1 Processor -- start 2024-04-21T08:41:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.023+0000 7fdceece5700 1 -- start start 2024-04-21T08:41:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.023+0000 7fdceece5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce8108550 0x7fdce8108970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.023+0000 7fdceece5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdce8108f40 con 0x7fdce8108550 2024-04-21T08:41:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.023+0000 7fdceca81700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce8108550 0x7fdce8108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.023+0000 7fdceca81700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce8108550 0x7fdce8108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36668/0 (socket says 172.21.15.28:36668) 2024-04-21T08:41:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.023+0000 7fdceca81700 1 -- 172.21.15.28:0/3399963779 learned_addr learned my addr 172.21.15.28:0/3399963779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:02.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.023+0000 7fdceca81700 1 -- 172.21.15.28:0/3399963779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce8109700 con 0x7fdce8108550 2024-04-21T08:41:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.023+0000 7fdceca81700 1 --2- 172.21.15.28:0/3399963779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce8108550 0x7fdce8108970 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc0098a0 tx=0x7fdcdc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b89b8c9af5ecc05b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.024+0000 7fdce77fe700 1 -- 172.21.15.28:0/3399963779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdcdc004160 con 0x7fdce8108550 2024-04-21T08:41:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.024+0000 7fdce77fe700 1 -- 172.21.15.28:0/3399963779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdcdc004300 con 0x7fdce8108550 2024-04-21T08:41:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.024+0000 7fdce77fe700 1 -- 172.21.15.28:0/3399963779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdcdc011af0 con 0x7fdce8108550 2024-04-21T08:41:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.024+0000 7fdceece5700 1 -- 172.21.15.28:0/3399963779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce8108550 msgr2=0x7fdce8108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.024+0000 7fdceece5700 1 --2- 172.21.15.28:0/3399963779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce8108550 0x7fdce8108970 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc0098a0 tx=0x7fdcdc009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.024+0000 7fdceece5700 1 -- 172.21.15.28:0/3399963779 shutdown_connections 2024-04-21T08:41:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.024+0000 7fdceece5700 1 --2- 172.21.15.28:0/3399963779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce8108550 0x7fdce8108970 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.024+0000 7fdceece5700 1 -- 172.21.15.28:0/3399963779 >> 172.21.15.28:0/3399963779 conn(0x7fdce8103b10 msgr2=0x7fdce8105f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.024+0000 7fdceece5700 1 -- 172.21.15.28:0/3399963779 shutdown_connections 2024-04-21T08:41:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.024+0000 7fdceece5700 1 -- 172.21.15.28:0/3399963779 wait complete. 2024-04-21T08:41:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.025+0000 7fdceece5700 1 Processor -- start 2024-04-21T08:41:02.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.025+0000 7fdceece5700 1 -- start start 2024-04-21T08:41:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.025+0000 7fdceece5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce8108550 0x7fdce819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.025+0000 7fdceece5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdce810cab0 con 0x7fdce8108550 2024-04-21T08:41:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.025+0000 7fdceca81700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce8108550 0x7fdce819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.025+0000 7fdceca81700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce8108550 0x7fdce819c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36672/0 (socket says 172.21.15.28:36672) 2024-04-21T08:41:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.025+0000 7fdceca81700 1 -- 172.21.15.28:0/1369869457 learned_addr learned my addr 172.21.15.28:0/1369869457 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.025+0000 7fdceca81700 1 -- 172.21.15.28:0/1369869457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce819ca40 con 0x7fdce8108550 2024-04-21T08:41:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.026+0000 7fdceca81700 1 --2- 172.21.15.28:0/1369869457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce8108550 0x7fdce819c500 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc009fb0 tx=0x7fdcdc011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.026+0000 7fdce5ffb700 1 -- 172.21.15.28:0/1369869457 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdcdc0036a0 con 0x7fdce8108550 2024-04-21T08:41:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.026+0000 7fdce5ffb700 1 -- 172.21.15.28:0/1369869457 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdcdc01a430 con 0x7fdce8108550 2024-04-21T08:41:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.026+0000 7fdceece5700 1 -- 172.21.15.28:0/1369869457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdce819ccd0 con 0x7fdce8108550 2024-04-21T08:41:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.026+0000 7fdce5ffb700 1 -- 172.21.15.28:0/1369869457 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdcdc019550 con 0x7fdce8108550 2024-04-21T08:41:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.026+0000 7fdceece5700 1 -- 172.21.15.28:0/1369869457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdce819f9c0 con 0x7fdce8108550 2024-04-21T08:41:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.027+0000 7fdce5ffb700 1 -- 172.21.15.28:0/1369869457 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fdcdc019850 con 0x7fdce8108550 2024-04-21T08:41:02.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.027+0000 7fdceece5700 1 -- 172.21.15.28:0/1369869457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdce80640a0 con 0x7fdce8108550 2024-04-21T08:41:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.027+0000 7fdce5ffb700 1 --2- 172.21.15.28:0/1369869457 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7fdcd8039560 0x7fdcd803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.027+0000 7fdce5ffb700 1 -- 172.21.15.28:0/1369869457 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdcdc04d910 con 0x7fdce8108550 2024-04-21T08:41:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.027+0000 7fdce7fff700 1 -- 172.21.15.28:0/1369869457 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7fdcd8039560 msgr2=0x7fdcd803ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2874636306 2024-04-21T08:41:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.027+0000 7fdce7fff700 1 --2- 172.21.15.28:0/1369869457 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7fdcd8039560 0x7fdcd803ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:41:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.032+0000 7fdce5ffb700 1 -- 172.21.15.28:0/1369869457 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdcdc04d330 con 0x7fdce8108550 2024-04-21T08:41:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.228+0000 7fdce7fff700 1 -- 172.21.15.28:0/1369869457 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7fdcd8039560 msgr2=0x7fdcd803ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2874636306 2024-04-21T08:41:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.228+0000 7fdce7fff700 1 --2- 172.21.15.28:0/1369869457 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7fdcd8039560 0x7fdcd803ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:41:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.300+0000 7fdceece5700 1 -- 172.21.15.28:0/1369869457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fdce819fb90 con 0x7fdce8108550 2024-04-21T08:41:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.301+0000 7fdce5ffb700 1 -- 172.21.15.28:0/1369869457 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fdce819fb90 con 0x7fdce8108550 2024-04-21T08:41:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.303+0000 7fdceece5700 1 -- 172.21.15.28:0/1369869457 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7fdcd8039560 msgr2=0x7fdcd803ba20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:41:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.303+0000 7fdceece5700 1 --2- 172.21.15.28:0/1369869457 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7fdcd8039560 0x7fdcd803ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.303+0000 7fdceece5700 1 -- 172.21.15.28:0/1369869457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce8108550 msgr2=0x7fdce819c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.303+0000 7fdceece5700 1 --2- 172.21.15.28:0/1369869457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce8108550 0x7fdce819c500 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc009fb0 tx=0x7fdcdc011940 comp rx=0 tx=0).stop 2024-04-21T08:41:02.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.303+0000 7fdceece5700 1 -- 172.21.15.28:0/1369869457 shutdown_connections 2024-04-21T08:41:02.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.303+0000 7fdceece5700 1 --2- 172.21.15.28:0/1369869457 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7fdcd8039560 0x7fdcd803ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:02.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.303+0000 7fdceece5700 1 --2- 172.21.15.28:0/1369869457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce8108550 0x7fdce819c500 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:02.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.303+0000 7fdceece5700 1 -- 172.21.15.28:0/1369869457 >> 172.21.15.28:0/1369869457 conn(0x7fdce8103b10 msgr2=0x7fdce8104710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:02.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.303+0000 7fdceece5700 1 -- 172.21.15.28:0/1369869457 shutdown_connections 2024-04-21T08:41:02.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:02.303+0000 7fdceece5700 1 -- 172.21.15.28:0/1369869457 wait complete. 2024-04-21T08:41:02.880 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-04-21T08:41:02.880 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 5... 2024-04-21T08:41:03.135 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:02 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:02.937+0000 7f4fd0754200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:41:03.553 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:03 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:03.242+0000 7f4fd0754200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:41:03.553 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:03 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:03.341+0000 7f4fd0754200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:41:03.553 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:03 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:03.553+0000 7f4fd0754200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:41:03.854 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:03 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:03.757+0000 7f4fd0754200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:41:04.640 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:04 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:04.386+0000 7f4fd0754200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:41:04.943 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:04 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:04.843+0000 7f4fd0754200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:41:04.943 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:04 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:04.943+0000 7f4fd0754200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:41:05.214 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:05 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:05.153+0000 7f4fd0754200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:41:06.053 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:05 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:05.704+0000 7f4fd0754200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:41:06.054 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:05 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:05.800+0000 7f4fd0754200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:41:06.354 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:06 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:06.054+0000 7f4fd0754200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:41:06.354 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:06 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:06.194+0000 7f4fd0754200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:41:06.355 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:06 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:06.293+0000 7f4fd0754200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:41:06.724 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:06 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:06.414+0000 7f4fd0754200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:41:06.725 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:06 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:06.725+0000 7f4fd0754200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:41:07.633 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:07.356+0000 7f4fd0754200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:41:07.633 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:07.581+0000 7f4fd0754200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:41:08.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-mon[28003]: Active manager daemon x restarted 2024-04-21T08:41:08.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-mon[28003]: Activating manager daemon x 2024-04-21T08:41:08.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-mon[28003]: osdmap e2: 0 total, 0 up, 0 in 2024-04-21T08:41:08.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-mon[28003]: mgrmap e6: x(active, starting, since 0.00334133s) 2024-04-21T08:41:08.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:41:08.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:41:08.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:41:08.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:41:08.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:41:08.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-mon[28003]: Manager daemon x is now available 2024-04-21T08:41:08.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' 2024-04-21T08:41:08.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' 2024-04-21T08:41:08.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:08.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:08.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:07 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:41:08.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:08 smithi028 ceph-mon[28003]: Found migration_current of "None". Setting to last migration. 2024-04-21T08:41:08.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:08 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:41:08.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:08 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' 2024-04-21T08:41:08.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:08 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' 2024-04-21T08:41:08.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:08 smithi028 ceph-mon[28003]: mgrmap e7: x(active, since 1.00566s) 2024-04-21T08:41:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:41:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-21T08:41:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-21T08:41:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:41:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.363+0000 7ff5da10f700 1 Processor -- start 2024-04-21T08:41:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.363+0000 7ff5da10f700 1 -- start start 2024-04-21T08:41:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.363+0000 7ff5da10f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d4107450 0x7ff5d4107870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.363+0000 7ff5da10f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d4107e40 con 0x7ff5d4107450 2024-04-21T08:41:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.363+0000 7ff5d37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d4107450 0x7ff5d4107870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.363+0000 7ff5d37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d4107450 0x7ff5d4107870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36686/0 (socket says 172.21.15.28:36686) 2024-04-21T08:41:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.363+0000 7ff5d37fe700 1 -- 172.21.15.28:0/148454939 learned_addr learned my addr 172.21.15.28:0/148454939 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.364+0000 7ff5d37fe700 1 -- 172.21.15.28:0/148454939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d4108650 con 0x7ff5d4107450 2024-04-21T08:41:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.364+0000 7ff5d37fe700 1 --2- 172.21.15.28:0/148454939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d4107450 0x7ff5d4107870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff5c80098a0 tx=0x7ff5c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30d86546da97ae7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.364+0000 7ff5d27fc700 1 -- 172.21.15.28:0/148454939 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5c8004160 con 0x7ff5d4107450 2024-04-21T08:41:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.364+0000 7ff5d27fc700 1 -- 172.21.15.28:0/148454939 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff5c8004300 con 0x7ff5d4107450 2024-04-21T08:41:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.364+0000 7ff5d27fc700 1 -- 172.21.15.28:0/148454939 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5c8011af0 con 0x7ff5d4107450 2024-04-21T08:41:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.365+0000 7ff5da10f700 1 -- 172.21.15.28:0/148454939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d4107450 msgr2=0x7ff5d4107870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.365+0000 7ff5da10f700 1 --2- 172.21.15.28:0/148454939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d4107450 0x7ff5d4107870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff5c80098a0 tx=0x7ff5c8009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.365+0000 7ff5da10f700 1 -- 172.21.15.28:0/148454939 shutdown_connections 2024-04-21T08:41:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.365+0000 7ff5da10f700 1 --2- 172.21.15.28:0/148454939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d4107450 0x7ff5d4107870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.365+0000 7ff5da10f700 1 -- 172.21.15.28:0/148454939 >> 172.21.15.28:0/148454939 conn(0x7ff5d41029f0 msgr2=0x7ff5d4104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.365+0000 7ff5da10f700 1 -- 172.21.15.28:0/148454939 shutdown_connections 2024-04-21T08:41:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.365+0000 7ff5da10f700 1 -- 172.21.15.28:0/148454939 wait complete. 2024-04-21T08:41:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.366+0000 7ff5da10f700 1 Processor -- start 2024-04-21T08:41:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.366+0000 7ff5da10f700 1 -- start start 2024-04-21T08:41:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.366+0000 7ff5da10f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d4107450 0x7ff5d419b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.366+0000 7ff5da10f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d410ba00 con 0x7ff5d4107450 2024-04-21T08:41:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.366+0000 7ff5d37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d4107450 0x7ff5d419b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.366+0000 7ff5d37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d4107450 0x7ff5d419b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36694/0 (socket says 172.21.15.28:36694) 2024-04-21T08:41:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.366+0000 7ff5d37fe700 1 -- 172.21.15.28:0/3180223728 learned_addr learned my addr 172.21.15.28:0/3180223728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.366+0000 7ff5d37fe700 1 -- 172.21.15.28:0/3180223728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d419b940 con 0x7ff5d4107450 2024-04-21T08:41:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.367+0000 7ff5d37fe700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d4107450 0x7ff5d419b400 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8003f40 tx=0x7ff5c8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.367+0000 7ff5d0ff9700 1 -- 172.21.15.28:0/3180223728 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5c80038b0 con 0x7ff5d4107450 2024-04-21T08:41:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.367+0000 7ff5da10f700 1 -- 172.21.15.28:0/3180223728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d419bbd0 con 0x7ff5d4107450 2024-04-21T08:41:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.367+0000 7ff5d0ff9700 1 -- 172.21.15.28:0/3180223728 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff5c8003a50 con 0x7ff5d4107450 2024-04-21T08:41:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.367+0000 7ff5da10f700 1 -- 172.21.15.28:0/3180223728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d419e8c0 con 0x7ff5d4107450 2024-04-21T08:41:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.367+0000 7ff5d0ff9700 1 -- 172.21.15.28:0/3180223728 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5c801a5c0 con 0x7ff5d4107450 2024-04-21T08:41:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.368+0000 7ff5d0ff9700 1 -- 172.21.15.28:0/3180223728 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7ff5c801a870 con 0x7ff5d4107450 2024-04-21T08:41:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.368+0000 7ff5d0ff9700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7ff5bc0395b0 0x7ff5bc03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.368+0000 7ff5d0ff9700 1 -- 172.21.15.28:0/3180223728 --> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff5bc03c180 con 0x7ff5bc0395b0 2024-04-21T08:41:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.368+0000 7ff5d0ff9700 1 -- 172.21.15.28:0/3180223728 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff5c804da80 con 0x7ff5d4107450 2024-04-21T08:41:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.368+0000 7ff5d2ffd700 1 -- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7ff5bc0395b0 msgr2=0x7ff5bc03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2874636306 2024-04-21T08:41:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.368+0000 7ff5d2ffd700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7ff5bc0395b0 0x7ff5bc03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:41:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.569+0000 7ff5d2ffd700 1 -- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7ff5bc0395b0 msgr2=0x7ff5bc03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2874636306 2024-04-21T08:41:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.569+0000 7ff5d2ffd700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7ff5bc0395b0 0x7ff5bc03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:41:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.970+0000 7ff5d2ffd700 1 -- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7ff5bc0395b0 msgr2=0x7ff5bc03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2874636306 2024-04-21T08:41:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:03.970+0000 7ff5d2ffd700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7ff5bc0395b0 0x7ff5bc03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T08:41:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:04.771+0000 7ff5d2ffd700 1 -- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7ff5bc0395b0 msgr2=0x7ff5bc03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2874636306 2024-04-21T08:41:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:04.771+0000 7ff5d2ffd700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7ff5bc0395b0 0x7ff5bc03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-21T08:41:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:06.372+0000 7ff5d2ffd700 1 -- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7ff5bc0395b0 msgr2=0x7ff5bc03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2874636306 2024-04-21T08:41:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:06.372+0000 7ff5d2ffd700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7ff5bc0395b0 0x7ff5bc03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-21T08:41:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:07.584+0000 7ff5d0ff9700 1 -- 172.21.15.28:0/3180223728 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7ff5c801abb0 con 0x7ff5d4107450 2024-04-21T08:41:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:07.584+0000 7ff5d0ff9700 1 -- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7ff5bc0395b0 msgr2=0x7ff5bc03ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:41:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:07.584+0000 7ff5d0ff9700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7ff5bc0395b0 0x7ff5bc03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.586+0000 7ff5d0ff9700 1 -- 172.21.15.28:0/3180223728 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7ff5c804f3d0 con 0x7ff5d4107450 2024-04-21T08:41:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.586+0000 7ff5d0ff9700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7ff5bc03cea0 0x7ff5bc03f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.586+0000 7ff5d0ff9700 1 -- 172.21.15.28:0/3180223728 --> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff5c800ce90 con 0x7ff5bc03cea0 2024-04-21T08:41:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.586+0000 7ff5d2ffd700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7ff5bc03cea0 0x7ff5bc03f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.587+0000 7ff5d2ffd700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7ff5bc03cea0 0x7ff5bc03f290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4003a10 tx=0x7ff5c40092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.587+0000 7ff5d0ff9700 1 -- 172.21.15.28:0/3180223728 <== mgr.14120 v2:172.21.15.28:6800/334321812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7ff5c800ce90 con 0x7ff5bc03cea0 2024-04-21T08:41:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.594+0000 7ff5da10f700 1 -- 172.21.15.28:0/3180223728 --> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7ff5c0002630 con 0x7ff5bc03cea0 2024-04-21T08:41:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.595+0000 7ff5d0ff9700 1 -- 172.21.15.28:0/3180223728 <== mgr.14120 v2:172.21.15.28:6800/334321812 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7ff5c0002630 con 0x7ff5bc03cea0 2024-04-21T08:41:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.595+0000 7ff5da10f700 1 -- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7ff5bc03cea0 msgr2=0x7ff5bc03f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.595+0000 7ff5da10f700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7ff5bc03cea0 0x7ff5bc03f290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4003a10 tx=0x7ff5c40092b0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.595+0000 7ff5da10f700 1 -- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d4107450 msgr2=0x7ff5d419b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.595+0000 7ff5da10f700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d4107450 0x7ff5d419b400 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8003f40 tx=0x7ff5c8011890 comp rx=0 tx=0).stop 2024-04-21T08:41:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.595+0000 7ff5da10f700 1 -- 172.21.15.28:0/3180223728 shutdown_connections 2024-04-21T08:41:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.595+0000 7ff5da10f700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d4107450 0x7ff5d419b400 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.595+0000 7ff5da10f700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7ff5bc03cea0 0x7ff5bc03f290 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.595+0000 7ff5da10f700 1 --2- 172.21.15.28:0/3180223728 >> [v2:172.21.15.28:6800/2874636306,v1:172.21.15.28:6801/2874636306] conn(0x7ff5bc0395b0 0x7ff5bc03ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.595+0000 7ff5da10f700 1 -- 172.21.15.28:0/3180223728 >> 172.21.15.28:0/3180223728 conn(0x7ff5d41029f0 msgr2=0x7ff5d4103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.595+0000 7ff5da10f700 1 -- 172.21.15.28:0/3180223728 shutdown_connections 2024-04-21T08:41:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:08.595+0000 7ff5da10f700 1 -- 172.21.15.28:0/3180223728 wait complete. 2024-04-21T08:41:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 5 is available 2024-04-21T08:41:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:Setting orchestrator backend to cephadm... 2024-04-21T08:41:09.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:09 smithi028 ceph-mon[28003]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-21T08:41:09.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:09 smithi028 ceph-mon[28003]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-21T08:41:10.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.527+0000 7f30ecddf700 1 Processor -- start 2024-04-21T08:41:10.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.527+0000 7f30ecddf700 1 -- start start 2024-04-21T08:41:10.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.527+0000 7f30ecddf700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e8108fc0 0x7f30e810b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.527+0000 7f30ecddf700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f30e8063d60 con 0x7f30e8108fc0 2024-04-21T08:41:10.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.527+0000 7f30e659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e8108fc0 0x7f30e810b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.527+0000 7f30e659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e8108fc0 0x7f30e810b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52596/0 (socket says 172.21.15.28:52596) 2024-04-21T08:41:10.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.527+0000 7f30e659c700 1 -- 172.21.15.28:0/878656383 learned_addr learned my addr 172.21.15.28:0/878656383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:10.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.527+0000 7f30e659c700 1 -- 172.21.15.28:0/878656383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e810b8f0 con 0x7f30e8108fc0 2024-04-21T08:41:10.352 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.528+0000 7f30e659c700 1 --2- 172.21.15.28:0/878656383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e8108fc0 0x7f30e810b3b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f30d80098a0 tx=0x7f30d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d421ebf0adec50 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:10.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.528+0000 7f30e559a700 1 -- 172.21.15.28:0/878656383 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30d8004160 con 0x7f30e8108fc0 2024-04-21T08:41:10.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.528+0000 7f30e559a700 1 -- 172.21.15.28:0/878656383 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f30d8004300 con 0x7f30e8108fc0 2024-04-21T08:41:10.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.528+0000 7f30e559a700 1 -- 172.21.15.28:0/878656383 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30d8011af0 con 0x7f30e8108fc0 2024-04-21T08:41:10.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.528+0000 7f30ecddf700 1 -- 172.21.15.28:0/878656383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e8108fc0 msgr2=0x7f30e810b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.528+0000 7f30ecddf700 1 --2- 172.21.15.28:0/878656383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e8108fc0 0x7f30e810b3b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f30d80098a0 tx=0x7f30d8009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.528+0000 7f30ecddf700 1 -- 172.21.15.28:0/878656383 shutdown_connections 2024-04-21T08:41:10.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.529+0000 7f30ecddf700 1 --2- 172.21.15.28:0/878656383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e8108fc0 0x7f30e810b3b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.529+0000 7f30ecddf700 1 -- 172.21.15.28:0/878656383 >> 172.21.15.28:0/878656383 conn(0x7f30e81029f0 msgr2=0x7f30e8104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:10.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.529+0000 7f30ecddf700 1 -- 172.21.15.28:0/878656383 shutdown_connections 2024-04-21T08:41:10.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.529+0000 7f30ecddf700 1 -- 172.21.15.28:0/878656383 wait complete. 2024-04-21T08:41:10.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.529+0000 7f30ecddf700 1 Processor -- start 2024-04-21T08:41:10.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.529+0000 7f30ecddf700 1 -- start start 2024-04-21T08:41:10.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.529+0000 7f30ecddf700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e8108fc0 0x7f30e819c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.353 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.529+0000 7f30ecddf700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f30e810c8c0 con 0x7f30e8108fc0 2024-04-21T08:41:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.530+0000 7f30e659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e8108fc0 0x7f30e819c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.530+0000 7f30e659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e8108fc0 0x7f30e819c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52600/0 (socket says 172.21.15.28:52600) 2024-04-21T08:41:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.530+0000 7f30e659c700 1 -- 172.21.15.28:0/3573453402 learned_addr learned my addr 172.21.15.28:0/3573453402 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.530+0000 7f30e659c700 1 -- 172.21.15.28:0/3573453402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e819c9b0 con 0x7f30e8108fc0 2024-04-21T08:41:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.530+0000 7f30e659c700 1 --2- 172.21.15.28:0/3573453402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e8108fc0 0x7f30e819c470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f30d8003f40 tx=0x7f30d8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.530+0000 7f30d77fe700 1 -- 172.21.15.28:0/3573453402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30d80038b0 con 0x7f30e8108fc0 2024-04-21T08:41:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.530+0000 7f30ecddf700 1 -- 172.21.15.28:0/3573453402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e819cc40 con 0x7f30e8108fc0 2024-04-21T08:41:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.530+0000 7f30d77fe700 1 -- 172.21.15.28:0/3573453402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f30d8003a50 con 0x7f30e8108fc0 2024-04-21T08:41:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.530+0000 7f30d77fe700 1 -- 172.21.15.28:0/3573453402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30d801a5c0 con 0x7f30e8108fc0 2024-04-21T08:41:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.530+0000 7f30ecddf700 1 -- 172.21.15.28:0/3573453402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e819f930 con 0x7f30e8108fc0 2024-04-21T08:41:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.531+0000 7f30d77fe700 1 -- 172.21.15.28:0/3573453402 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f30d801a8c0 con 0x7f30e8108fc0 2024-04-21T08:41:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.531+0000 7f30d77fe700 1 --2- 172.21.15.28:0/3573453402 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f30d0039450 0x7f30d003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.531+0000 7f30ecddf700 1 -- 172.21.15.28:0/3573453402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30e80640a0 con 0x7f30e8108fc0 2024-04-21T08:41:10.354 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.531+0000 7f30d77fe700 1 -- 172.21.15.28:0/3573453402 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f30d804d9b0 con 0x7f30e8108fc0 2024-04-21T08:41:10.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.532+0000 7f30e5d9b700 1 --2- 172.21.15.28:0/3573453402 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f30d0039450 0x7f30d003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:10.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.532+0000 7f30e5d9b700 1 --2- 172.21.15.28:0/3573453402 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f30d0039450 0x7f30d003b910 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f30dc006fd0 tx=0x7f30dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:10.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.537+0000 7f30d77fe700 1 -- 172.21.15.28:0/3573453402 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f30d8019c70 con 0x7f30e8108fc0 2024-04-21T08:41:10.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.639+0000 7f30d77fe700 1 -- 172.21.15.28:0/3573453402 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f30d802d910 con 0x7f30e8108fc0 2024-04-21T08:41:10.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.762+0000 7f30ecddf700 1 -- 172.21.15.28:0/3573453402 --> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f30e8103c40 con 0x7f30d0039450 2024-04-21T08:41:10.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.768+0000 7f30d77fe700 1 -- 172.21.15.28:0/3573453402 <== mgr.14120 v2:172.21.15.28:6800/334321812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f30e8103c40 con 0x7f30d0039450 2024-04-21T08:41:10.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.770+0000 7f30ecddf700 1 -- 172.21.15.28:0/3573453402 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f30d0039450 msgr2=0x7f30d003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.770+0000 7f30ecddf700 1 --2- 172.21.15.28:0/3573453402 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f30d0039450 0x7f30d003b910 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f30dc006fd0 tx=0x7f30dc006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:10.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.770+0000 7f30ecddf700 1 -- 172.21.15.28:0/3573453402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e8108fc0 msgr2=0x7f30e819c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:10.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.770+0000 7f30ecddf700 1 --2- 172.21.15.28:0/3573453402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e8108fc0 0x7f30e819c470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f30d8003f40 tx=0x7f30d8011890 comp rx=0 tx=0).stop 2024-04-21T08:41:10.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.770+0000 7f30ecddf700 1 -- 172.21.15.28:0/3573453402 shutdown_connections 2024-04-21T08:41:10.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.770+0000 7f30ecddf700 1 --2- 172.21.15.28:0/3573453402 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f30d0039450 0x7f30d003b910 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.770+0000 7f30ecddf700 1 --2- 172.21.15.28:0/3573453402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e8108fc0 0x7f30e819c470 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:10.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.770+0000 7f30ecddf700 1 -- 172.21.15.28:0/3573453402 >> 172.21.15.28:0/3573453402 conn(0x7f30e81029f0 msgr2=0x7f30e8103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:10.355 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.770+0000 7f30ecddf700 1 -- 172.21.15.28:0/3573453402 shutdown_connections 2024-04-21T08:41:10.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:09.770+0000 7f30ecddf700 1 -- 172.21.15.28:0/3573453402 wait complete. 2024-04-21T08:41:10.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:10 smithi028 ceph-mon[28003]: mgrmap e8: x(active, since 2s) 2024-04-21T08:41:10.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:10 smithi028 ceph-mon[28003]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:10.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:10 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' 2024-04-21T08:41:10.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:10 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:10.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:10 smithi028 ceph-mon[28003]: [21/Apr/2024:08:41:09] ENGINE Bus STARTING 2024-04-21T08:41:10.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:10 smithi028 ceph-mon[28003]: [21/Apr/2024:08:41:09] ENGINE Serving on https://172.21.15.28:7150 2024-04-21T08:41:10.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:10 smithi028 ceph-mon[28003]: [21/Apr/2024:08:41:10] ENGINE Serving on http://172.21.15.28:8765 2024-04-21T08:41:10.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:10 smithi028 ceph-mon[28003]: [21/Apr/2024:08:41:10] ENGINE Bus STARTED 2024-04-21T08:41:10.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:10 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:11.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-21T08:41:11.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.801+0000 7fb00d65f700 1 Processor -- start 2024-04-21T08:41:11.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.801+0000 7fb00d65f700 1 -- start start 2024-04-21T08:41:11.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.801+0000 7fb00d65f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb008108590 0x7fb0081089b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:11.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.801+0000 7fb00d65f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb008108f80 con 0x7fb008108590 2024-04-21T08:41:11.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.801+0000 7fb006ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb008108590 0x7fb0081089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:11.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.801+0000 7fb006ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb008108590 0x7fb0081089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52606/0 (socket says 172.21.15.28:52606) 2024-04-21T08:41:11.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.801+0000 7fb006ffd700 1 -- 172.21.15.28:0/268089640 learned_addr learned my addr 172.21.15.28:0/268089640 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:11.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.802+0000 7fb006ffd700 1 -- 172.21.15.28:0/268089640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb008109740 con 0x7fb008108590 2024-04-21T08:41:11.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.802+0000 7fb006ffd700 1 --2- 172.21.15.28:0/268089640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb008108590 0x7fb0081089b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb000009e80 tx=0x7fb00000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d271cc3410670b62 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:11.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.802+0000 7fb005ffb700 1 -- 172.21.15.28:0/268089640 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb000017070 con 0x7fb008108590 2024-04-21T08:41:11.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.802+0000 7fb005ffb700 1 -- 172.21.15.28:0/268089640 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb00000b8c0 con 0x7fb008108590 2024-04-21T08:41:11.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.802+0000 7fb005ffb700 1 -- 172.21.15.28:0/268089640 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb000004490 con 0x7fb008108590 2024-04-21T08:41:11.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.803+0000 7fb00d65f700 1 -- 172.21.15.28:0/268089640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb008108590 msgr2=0x7fb0081089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:11.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.803+0000 7fb00d65f700 1 --2- 172.21.15.28:0/268089640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb008108590 0x7fb0081089b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb000009e80 tx=0x7fb00000b330 comp rx=0 tx=0).stop 2024-04-21T08:41:11.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.803+0000 7fb00d65f700 1 -- 172.21.15.28:0/268089640 shutdown_connections 2024-04-21T08:41:11.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.803+0000 7fb00d65f700 1 --2- 172.21.15.28:0/268089640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb008108590 0x7fb0081089b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:11.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.803+0000 7fb00d65f700 1 -- 172.21.15.28:0/268089640 >> 172.21.15.28:0/268089640 conn(0x7fb008103b70 msgr2=0x7fb008105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:11.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.803+0000 7fb00d65f700 1 -- 172.21.15.28:0/268089640 shutdown_connections 2024-04-21T08:41:11.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.803+0000 7fb00d65f700 1 -- 172.21.15.28:0/268089640 wait complete. 2024-04-21T08:41:11.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.803+0000 7fb00d65f700 1 Processor -- start 2024-04-21T08:41:11.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.804+0000 7fb00d65f700 1 -- start start 2024-04-21T08:41:11.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.804+0000 7fb00d65f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb008108590 0x7fb00819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:11.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.804+0000 7fb00d65f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb00810caf0 con 0x7fb008108590 2024-04-21T08:41:11.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.804+0000 7fb006ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb008108590 0x7fb00819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:11.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.804+0000 7fb006ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb008108590 0x7fb00819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52614/0 (socket says 172.21.15.28:52614) 2024-04-21T08:41:11.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.804+0000 7fb006ffd700 1 -- 172.21.15.28:0/2118452421 learned_addr learned my addr 172.21.15.28:0/2118452421 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:11.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.804+0000 7fb006ffd700 1 -- 172.21.15.28:0/2118452421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb00819ca50 con 0x7fb008108590 2024-04-21T08:41:11.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.804+0000 7fb006ffd700 1 --2- 172.21.15.28:0/2118452421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb008108590 0x7fb00819c510 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb00000b7b0 tx=0x7fb00000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:11.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.804+0000 7faffffff700 1 -- 172.21.15.28:0/2118452421 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb000020020 con 0x7fb008108590 2024-04-21T08:41:11.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.805+0000 7faffffff700 1 -- 172.21.15.28:0/2118452421 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb000012430 con 0x7fb008108590 2024-04-21T08:41:11.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.805+0000 7faffffff700 1 -- 172.21.15.28:0/2118452421 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb000017040 con 0x7fb008108590 2024-04-21T08:41:11.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.805+0000 7fb00d65f700 1 -- 172.21.15.28:0/2118452421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb00819cce0 con 0x7fb008108590 2024-04-21T08:41:11.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.805+0000 7fb00d65f700 1 -- 172.21.15.28:0/2118452421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb00819f9d0 con 0x7fb008108590 2024-04-21T08:41:11.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.805+0000 7faffffff700 1 -- 172.21.15.28:0/2118452421 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fb000023020 con 0x7fb008108590 2024-04-21T08:41:11.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.806+0000 7fb00d65f700 1 -- 172.21.15.28:0/2118452421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb008069ba0 con 0x7fb008108590 2024-04-21T08:41:11.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.806+0000 7faffffff700 1 --2- 172.21.15.28:0/2118452421 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7faff4039560 0x7faff403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:11.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.806+0000 7faffffff700 1 -- 172.21.15.28:0/2118452421 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb00000f050 con 0x7fb008108590 2024-04-21T08:41:11.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.806+0000 7fb0067fc700 1 --2- 172.21.15.28:0/2118452421 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7faff4039560 0x7faff403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:11.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.806+0000 7fb0067fc700 1 --2- 172.21.15.28:0/2118452421 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7faff4039560 0x7faff403ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faff0006fd0 tx=0x7faff0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:11.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:10.811+0000 7faffffff700 1 -- 172.21.15.28:0/2118452421 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb00002a020 con 0x7fb008108590 2024-04-21T08:41:11.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:11.022+0000 7fb00d65f700 1 -- 172.21.15.28:0/2118452421 --> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fb008104e90 con 0x7faff4039560 2024-04-21T08:41:11.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:11.023+0000 7faffffff700 1 -- 172.21.15.28:0/2118452421 <== mgr.14120 v2:172.21.15.28:6800/334321812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fb008104e90 con 0x7faff4039560 2024-04-21T08:41:11.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:11.025+0000 7fb00d65f700 1 -- 172.21.15.28:0/2118452421 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7faff4039560 msgr2=0x7faff403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:11.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:11.025+0000 7fb00d65f700 1 --2- 172.21.15.28:0/2118452421 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7faff4039560 0x7faff403ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faff0006fd0 tx=0x7faff0006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:11.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:11.025+0000 7fb00d65f700 1 -- 172.21.15.28:0/2118452421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb008108590 msgr2=0x7fb00819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:11.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:11.025+0000 7fb00d65f700 1 --2- 172.21.15.28:0/2118452421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb008108590 0x7fb00819c510 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb00000b7b0 tx=0x7fb00000be20 comp rx=0 tx=0).stop 2024-04-21T08:41:11.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:11.025+0000 7fb00d65f700 1 -- 172.21.15.28:0/2118452421 shutdown_connections 2024-04-21T08:41:11.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:11.025+0000 7fb00d65f700 1 --2- 172.21.15.28:0/2118452421 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7faff4039560 0x7faff403ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:11.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:11.025+0000 7fb00d65f700 1 --2- 172.21.15.28:0/2118452421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb008108590 0x7fb00819c510 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:11.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:11.025+0000 7fb00d65f700 1 -- 172.21.15.28:0/2118452421 >> 172.21.15.28:0/2118452421 conn(0x7fb008103b70 msgr2=0x7fb008104780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:11.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:11.025+0000 7fb00d65f700 1 -- 172.21.15.28:0/2118452421 shutdown_connections 2024-04-21T08:41:11.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:11.025+0000 7fb00d65f700 1 -- 172.21.15.28:0/2118452421 wait complete. 2024-04-21T08:41:11.639 INFO:teuthology.orchestra.run.smithi028.stdout:Generating ssh key... 2024-04-21T08:41:11.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:11 smithi028 ceph-mon[28003]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:13.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-mon[28003]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:13.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-mon[28003]: Generating ssh key... 2024-04-21T08:41:13.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: Generating public/private rsa key pair. 2024-04-21T08:41:13.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: Your identification has been saved in /tmp/tmprkfn5vya/key. 2024-04-21T08:41:13.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: Your public key has been saved in /tmp/tmprkfn5vya/key.pub. 2024-04-21T08:41:13.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: The key fingerprint is: 2024-04-21T08:41:13.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: SHA256:O7u4UZ5a2uKzpcRWBMQiXxKNvEyjSEIRN/m1JNOgFYo ceph-b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:41:13.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: The key's randomart image is: 2024-04-21T08:41:13.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: +---[RSA 3072]----+ 2024-04-21T08:41:13.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: |.+o+oX* | 2024-04-21T08:41:13.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: |..+.%o+= | 2024-04-21T08:41:13.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: |oE O.*= o | 2024-04-21T08:41:13.106 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: |. . +. o | 2024-04-21T08:41:13.106 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: | S | 2024-04-21T08:41:13.106 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: | . + o | 2024-04-21T08:41:13.106 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: | = B | 2024-04-21T08:41:13.106 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: | ooX o | 2024-04-21T08:41:13.106 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: | .O=+. | 2024-04-21T08:41:13.106 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: +----[SHA256]-----+ 2024-04-21T08:41:13.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.045+0000 7f72d7166700 1 Processor -- start 2024-04-21T08:41:13.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.045+0000 7f72d7166700 1 -- start start 2024-04-21T08:41:13.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.045+0000 7f72d7166700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d0108b20 0x7f72d0108f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.045+0000 7f72d7166700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72d0107060 con 0x7f72d0108b20 2024-04-21T08:41:13.395 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.046+0000 7f72d4f02700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d0108b20 0x7f72d0108f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.046+0000 7f72d4f02700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d0108b20 0x7f72d0108f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52624/0 (socket says 172.21.15.28:52624) 2024-04-21T08:41:13.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.046+0000 7f72d4f02700 1 -- 172.21.15.28:0/3427766591 learned_addr learned my addr 172.21.15.28:0/3427766591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:13.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.046+0000 7f72d4f02700 1 -- 172.21.15.28:0/3427766591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d01071e0 con 0x7f72d0108b20 2024-04-21T08:41:13.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.046+0000 7f72d4f02700 1 --2- 172.21.15.28:0/3427766591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d0108b20 0x7f72d0108f40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f72c40098a0 tx=0x7f72c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e6d1b96a0e0454 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:13.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.046+0000 7f72cf7fe700 1 -- 172.21.15.28:0/3427766591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72c4004160 con 0x7f72d0108b20 2024-04-21T08:41:13.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.046+0000 7f72cf7fe700 1 -- 172.21.15.28:0/3427766591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f72c4004300 con 0x7f72d0108b20 2024-04-21T08:41:13.397 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.046+0000 7f72cf7fe700 1 -- 172.21.15.28:0/3427766591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72c4011af0 con 0x7f72d0108b20 2024-04-21T08:41:13.397 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.047+0000 7f72d7166700 1 -- 172.21.15.28:0/3427766591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d0108b20 msgr2=0x7f72d0108f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:13.397 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.047+0000 7f72d7166700 1 --2- 172.21.15.28:0/3427766591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d0108b20 0x7f72d0108f40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f72c40098a0 tx=0x7f72c4009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.397 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.047+0000 7f72d7166700 1 -- 172.21.15.28:0/3427766591 shutdown_connections 2024-04-21T08:41:13.397 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.047+0000 7f72d7166700 1 --2- 172.21.15.28:0/3427766591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d0108b20 0x7f72d0108f40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.397 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.047+0000 7f72d7166700 1 -- 172.21.15.28:0/3427766591 >> 172.21.15.28:0/3427766591 conn(0x7f72d0102a60 msgr2=0x7f72d0104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:13.398 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.047+0000 7f72d7166700 1 -- 172.21.15.28:0/3427766591 shutdown_connections 2024-04-21T08:41:13.398 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.047+0000 7f72d7166700 1 -- 172.21.15.28:0/3427766591 wait complete. 2024-04-21T08:41:13.398 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.047+0000 7f72d7166700 1 Processor -- start 2024-04-21T08:41:13.398 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.048+0000 7f72d7166700 1 -- start start 2024-04-21T08:41:13.398 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.048+0000 7f72d7166700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d0108b20 0x7f72d019c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.048+0000 7f72d7166700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72d010b9f0 con 0x7f72d0108b20 2024-04-21T08:41:13.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.048+0000 7f72d4f02700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d0108b20 0x7f72d019c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.048+0000 7f72d4f02700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d0108b20 0x7f72d019c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52626/0 (socket says 172.21.15.28:52626) 2024-04-21T08:41:13.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.048+0000 7f72d4f02700 1 -- 172.21.15.28:0/2818015087 learned_addr learned my addr 172.21.15.28:0/2818015087 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:13.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.048+0000 7f72d4f02700 1 -- 172.21.15.28:0/2818015087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d019cb10 con 0x7f72d0108b20 2024-04-21T08:41:13.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.048+0000 7f72d4f02700 1 --2- 172.21.15.28:0/2818015087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d0108b20 0x7f72d019c5d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f72c4000c00 tx=0x7f72c4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:13.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.048+0000 7f72cdffb700 1 -- 172.21.15.28:0/2818015087 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72c4003b20 con 0x7f72d0108b20 2024-04-21T08:41:13.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.048+0000 7f72cdffb700 1 -- 172.21.15.28:0/2818015087 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f72c401a8c0 con 0x7f72d0108b20 2024-04-21T08:41:13.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.048+0000 7f72cdffb700 1 -- 172.21.15.28:0/2818015087 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72c4024940 con 0x7f72d0108b20 2024-04-21T08:41:13.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.048+0000 7f72d7166700 1 -- 172.21.15.28:0/2818015087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d019cda0 con 0x7f72d0108b20 2024-04-21T08:41:13.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.049+0000 7f72d7166700 1 -- 172.21.15.28:0/2818015087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d019fa90 con 0x7f72d0108b20 2024-04-21T08:41:13.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.049+0000 7f72cdffb700 1 -- 172.21.15.28:0/2818015087 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f72c4022020 con 0x7f72d0108b20 2024-04-21T08:41:13.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.050+0000 7f72d7166700 1 -- 172.21.15.28:0/2818015087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d010bb70 con 0x7f72d0108b20 2024-04-21T08:41:13.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.050+0000 7f72cdffb700 1 --2- 172.21.15.28:0/2818015087 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f72c0039600 0x7f72c003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:13.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.050+0000 7f72cdffb700 1 -- 172.21.15.28:0/2818015087 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f72c404d7e0 con 0x7f72d0108b20 2024-04-21T08:41:13.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.050+0000 7f72cffff700 1 --2- 172.21.15.28:0/2818015087 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f72c0039600 0x7f72c003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:13.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.050+0000 7f72cffff700 1 --2- 172.21.15.28:0/2818015087 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f72c0039600 0x7f72c003bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f72bc006fd0 tx=0x7f72bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:13.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.055+0000 7f72cdffb700 1 -- 172.21.15.28:0/2818015087 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f72c400be50 con 0x7f72d0108b20 2024-04-21T08:41:13.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.266+0000 7f72d7166700 1 -- 172.21.15.28:0/2818015087 --> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f72d002e1b0 con 0x7f72c0039600 2024-04-21T08:41:13.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.855+0000 7f72cdffb700 1 -- 172.21.15.28:0/2818015087 <== mgr.14120 v2:172.21.15.28:6800/334321812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f72d002e1b0 con 0x7f72c0039600 2024-04-21T08:41:13.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.857+0000 7f72d7166700 1 -- 172.21.15.28:0/2818015087 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f72c0039600 msgr2=0x7f72c003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:13.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.858+0000 7f72d7166700 1 --2- 172.21.15.28:0/2818015087 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f72c0039600 0x7f72c003bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f72bc006fd0 tx=0x7f72bc006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:13.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.858+0000 7f72d7166700 1 -- 172.21.15.28:0/2818015087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d0108b20 msgr2=0x7f72d019c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:13.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.858+0000 7f72d7166700 1 --2- 172.21.15.28:0/2818015087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d0108b20 0x7f72d019c5d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f72c4000c00 tx=0x7f72c4011b70 comp rx=0 tx=0).stop 2024-04-21T08:41:13.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.858+0000 7f72d7166700 1 -- 172.21.15.28:0/2818015087 shutdown_connections 2024-04-21T08:41:13.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.858+0000 7f72d7166700 1 --2- 172.21.15.28:0/2818015087 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f72c0039600 0x7f72c003bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.858+0000 7f72d7166700 1 --2- 172.21.15.28:0/2818015087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d0108b20 0x7f72d019c5d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:13.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.858+0000 7f72d7166700 1 -- 172.21.15.28:0/2818015087 >> 172.21.15.28:0/2818015087 conn(0x7f72d0102a60 msgr2=0x7f72d0103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:13.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.858+0000 7f72d7166700 1 -- 172.21.15.28:0/2818015087 shutdown_connections 2024-04-21T08:41:13.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:12.858+0000 7f72d7166700 1 -- 172.21.15.28:0/2818015087 wait complete. 2024-04-21T08:41:14.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:13 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' 2024-04-21T08:41:14.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:13 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' 2024-04-21T08:41:14.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCit2a8zvWLEusP1o7015j8V7/b1pTPBcLqRY3GjJRV1jImJcxBZKxpsWwnuNKfCCAitL6AysyTdF99aRfKoZ8DoWqUNVIEkNVlTi422dacXat3lLnkCocP6sJiO1gMgG/tslraPQVsz1KaAQDGMCwjOWUgm7UG/lM4rhlYMpPBLylLe1QsLJ+hciJD2oYuJHxjhYys5vxE80F8mqgEBk4dIO+nvk8ViOBNEzqP3i/cYKAfr9Mx0L7vpkr+gDy4zb2tNYK7Me8rMzVTUkoaBNbys0b2EjlsKd1wJtSNexQbwXzYlzhVnQO/ITPJsAScFpH8T9q2APp3rM7iAIcRybjVafLiYHr+dn8tAprFRAj3G/k0+mEhD2VK9kS5/82Qudf5YiQlpduSrMTIuRD1XtskfxIUCMWeqt2MCssz2WBU0VnrMqkjkpzEcTl6qmz6GCGi4h+c6ZsCu12wtdM1JKBweUv4Q3rQg+f3jVFJEkY5ABY1iEw+r3jeUElPpSun4Lk= ceph-b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:41:14.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.820+0000 7f34433a5700 1 Processor -- start 2024-04-21T08:41:14.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.820+0000 7f34433a5700 1 -- start start 2024-04-21T08:41:14.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.820+0000 7f34433a5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343c108580 0x7f343c1089a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:14.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.820+0000 7f34433a5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f343c108f70 con 0x7f343c108580 2024-04-21T08:41:14.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.821+0000 7f3441141700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343c108580 0x7f343c1089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:14.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.821+0000 7f3441141700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343c108580 0x7f343c1089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36570/0 (socket says 172.21.15.28:36570) 2024-04-21T08:41:14.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.821+0000 7f3441141700 1 -- 172.21.15.28:0/192820678 learned_addr learned my addr 172.21.15.28:0/192820678 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:14.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.821+0000 7f3441141700 1 -- 172.21.15.28:0/192820678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343c109730 con 0x7f343c108580 2024-04-21T08:41:14.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.821+0000 7f3441141700 1 --2- 172.21.15.28:0/192820678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343c108580 0x7f343c1089a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3434009e80 tx=0x7f343400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c0758113c6c360 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:14.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.821+0000 7f343bfff700 1 -- 172.21.15.28:0/192820678 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3434017070 con 0x7f343c108580 2024-04-21T08:41:14.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.821+0000 7f343bfff700 1 -- 172.21.15.28:0/192820678 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f343400b8c0 con 0x7f343c108580 2024-04-21T08:41:14.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.821+0000 7f343bfff700 1 -- 172.21.15.28:0/192820678 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3434004490 con 0x7f343c108580 2024-04-21T08:41:14.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.822+0000 7f34433a5700 1 -- 172.21.15.28:0/192820678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343c108580 msgr2=0x7f343c1089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:14.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.822+0000 7f34433a5700 1 --2- 172.21.15.28:0/192820678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343c108580 0x7f343c1089a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3434009e80 tx=0x7f343400b330 comp rx=0 tx=0).stop 2024-04-21T08:41:14.604 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.822+0000 7f34433a5700 1 -- 172.21.15.28:0/192820678 shutdown_connections 2024-04-21T08:41:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.822+0000 7f34433a5700 1 --2- 172.21.15.28:0/192820678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343c108580 0x7f343c1089a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.822+0000 7f34433a5700 1 -- 172.21.15.28:0/192820678 >> 172.21.15.28:0/192820678 conn(0x7f343c103b40 msgr2=0x7f343c105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.822+0000 7f34433a5700 1 -- 172.21.15.28:0/192820678 shutdown_connections 2024-04-21T08:41:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.822+0000 7f34433a5700 1 -- 172.21.15.28:0/192820678 wait complete. 2024-04-21T08:41:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.822+0000 7f34433a5700 1 Processor -- start 2024-04-21T08:41:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.822+0000 7f34433a5700 1 -- start start 2024-04-21T08:41:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.823+0000 7f34433a5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343c108580 0x7f343c07db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.823+0000 7f34433a5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f343c10cae0 con 0x7f343c108580 2024-04-21T08:41:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.823+0000 7f3441141700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343c108580 0x7f343c07db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.823+0000 7f3441141700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343c108580 0x7f343c07db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36584/0 (socket says 172.21.15.28:36584) 2024-04-21T08:41:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.823+0000 7f3441141700 1 -- 172.21.15.28:0/3657563163 learned_addr learned my addr 172.21.15.28:0/3657563163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.823+0000 7f3441141700 1 -- 172.21.15.28:0/3657563163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343c07e0c0 con 0x7f343c108580 2024-04-21T08:41:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.823+0000 7f3441141700 1 --2- 172.21.15.28:0/3657563163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343c108580 0x7f343c07db80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f343400b7b0 tx=0x7f343400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.823+0000 7f343a7fc700 1 -- 172.21.15.28:0/3657563163 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3434020020 con 0x7f343c108580 2024-04-21T08:41:14.605 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.823+0000 7f343a7fc700 1 -- 172.21.15.28:0/3657563163 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3434012430 con 0x7f343c108580 2024-04-21T08:41:14.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.823+0000 7f34433a5700 1 -- 172.21.15.28:0/3657563163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343c07c2b0 con 0x7f343c108580 2024-04-21T08:41:14.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.823+0000 7f343a7fc700 1 -- 172.21.15.28:0/3657563163 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3434017040 con 0x7f343c108580 2024-04-21T08:41:14.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.823+0000 7f34433a5700 1 -- 172.21.15.28:0/3657563163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f343c07c7f0 con 0x7f343c108580 2024-04-21T08:41:14.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.824+0000 7f343a7fc700 1 -- 172.21.15.28:0/3657563163 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f3434023020 con 0x7f343c108580 2024-04-21T08:41:14.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.824+0000 7f34433a5700 1 -- 172.21.15.28:0/3657563163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f343c0640a0 con 0x7f343c108580 2024-04-21T08:41:14.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.824+0000 7f343a7fc700 1 --2- 172.21.15.28:0/3657563163 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f34240395b0 0x7f342403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:14.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.825+0000 7f343a7fc700 1 -- 172.21.15.28:0/3657563163 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f343400f050 con 0x7f343c108580 2024-04-21T08:41:14.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.825+0000 7f3440940700 1 --2- 172.21.15.28:0/3657563163 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f34240395b0 0x7f342403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:14.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.825+0000 7f3440940700 1 --2- 172.21.15.28:0/3657563163 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f34240395b0 0x7f342403ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3430006fd0 tx=0x7f3430006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:14.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:13.830+0000 7f343a7fc700 1 -- 172.21.15.28:0/3657563163 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f343402a020 con 0x7f343c108580 2024-04-21T08:41:14.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:14.038+0000 7f34433a5700 1 -- 172.21.15.28:0/3657563163 --> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f343c07d050 con 0x7f34240395b0 2024-04-21T08:41:14.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:14.038+0000 7f343a7fc700 1 -- 172.21.15.28:0/3657563163 <== mgr.14120 v2:172.21.15.28:6800/334321812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f343c07d050 con 0x7f34240395b0 2024-04-21T08:41:14.606 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:14.040+0000 7f34433a5700 1 -- 172.21.15.28:0/3657563163 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f34240395b0 msgr2=0x7f342403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:14.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:14.040+0000 7f34433a5700 1 --2- 172.21.15.28:0/3657563163 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f34240395b0 0x7f342403ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3430006fd0 tx=0x7f3430006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:14.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:14.040+0000 7f34433a5700 1 -- 172.21.15.28:0/3657563163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343c108580 msgr2=0x7f343c07db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:14.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:14.040+0000 7f34433a5700 1 --2- 172.21.15.28:0/3657563163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343c108580 0x7f343c07db80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f343400b7b0 tx=0x7f343400be20 comp rx=0 tx=0).stop 2024-04-21T08:41:14.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:14.040+0000 7f34433a5700 1 -- 172.21.15.28:0/3657563163 shutdown_connections 2024-04-21T08:41:14.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:14.040+0000 7f34433a5700 1 --2- 172.21.15.28:0/3657563163 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f34240395b0 0x7f342403ba70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:14.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:14.040+0000 7f34433a5700 1 --2- 172.21.15.28:0/3657563163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343c108580 0x7f343c07db80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:14.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:14.040+0000 7f34433a5700 1 -- 172.21.15.28:0/3657563163 >> 172.21.15.28:0/3657563163 conn(0x7f343c103b40 msgr2=0x7f343c1046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:14.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:14.040+0000 7f34433a5700 1 -- 172.21.15.28:0/3657563163 shutdown_connections 2024-04-21T08:41:14.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:14.041+0000 7f34433a5700 1 -- 172.21.15.28:0/3657563163 wait complete. 2024-04-21T08:41:14.607 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-21T08:41:14.607 INFO:teuthology.orchestra.run.smithi028.stdout:Adding key to root@localhost authorized_keys... 2024-04-21T08:41:14.607 INFO:teuthology.orchestra.run.smithi028.stdout:Adding host smithi028... 2024-04-21T08:41:15.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:14 smithi028 ceph-mon[28003]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:16.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:15 smithi028 ceph-mon[28003]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:17.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:16 smithi028 ceph-mon[28003]: Deploying cephadm binary to smithi028 2024-04-21T08:41:20.101 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Added host 'smithi028' with addr '172.21.15.28' 2024-04-21T08:41:20.101 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.016+0000 7f5dc7dda700 1 Processor -- start 2024-04-21T08:41:20.101 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.016+0000 7f5dc7dda700 1 -- start start 2024-04-21T08:41:20.101 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.016+0000 7f5dc7dda700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc0108590 0x7f5dc01089b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.016+0000 7f5dc7dda700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc0108f80 con 0x7f5dc0108590 2024-04-21T08:41:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.016+0000 7f5dc5b76700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc0108590 0x7f5dc01089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.016+0000 7f5dc5b76700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc0108590 0x7f5dc01089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36600/0 (socket says 172.21.15.28:36600) 2024-04-21T08:41:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.016+0000 7f5dc5b76700 1 -- 172.21.15.28:0/1690835080 learned_addr learned my addr 172.21.15.28:0/1690835080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.017+0000 7f5dc5b76700 1 -- 172.21.15.28:0/1690835080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc0109740 con 0x7f5dc0108590 2024-04-21T08:41:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.017+0000 7f5dc5b76700 1 --2- 172.21.15.28:0/1690835080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc0108590 0x7f5dc01089b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5db00098a0 tx=0x7f5db0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38594199765f3ff8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.017+0000 7f5dc4b74700 1 -- 172.21.15.28:0/1690835080 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5db0004160 con 0x7f5dc0108590 2024-04-21T08:41:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.017+0000 7f5dc4b74700 1 -- 172.21.15.28:0/1690835080 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5db0004300 con 0x7f5dc0108590 2024-04-21T08:41:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.017+0000 7f5dc4b74700 1 -- 172.21.15.28:0/1690835080 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5db0011af0 con 0x7f5dc0108590 2024-04-21T08:41:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.017+0000 7f5dc7dda700 1 -- 172.21.15.28:0/1690835080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc0108590 msgr2=0x7f5dc01089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.017+0000 7f5dc7dda700 1 --2- 172.21.15.28:0/1690835080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc0108590 0x7f5dc01089b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5db00098a0 tx=0x7f5db0009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.018+0000 7f5dc7dda700 1 -- 172.21.15.28:0/1690835080 shutdown_connections 2024-04-21T08:41:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.018+0000 7f5dc7dda700 1 --2- 172.21.15.28:0/1690835080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc0108590 0x7f5dc01089b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.018+0000 7f5dc7dda700 1 -- 172.21.15.28:0/1690835080 >> 172.21.15.28:0/1690835080 conn(0x7f5dc0103b70 msgr2=0x7f5dc0105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.018+0000 7f5dc7dda700 1 -- 172.21.15.28:0/1690835080 shutdown_connections 2024-04-21T08:41:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.018+0000 7f5dc7dda700 1 -- 172.21.15.28:0/1690835080 wait complete. 2024-04-21T08:41:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.018+0000 7f5dc7dda700 1 Processor -- start 2024-04-21T08:41:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.018+0000 7f5dc7dda700 1 -- start start 2024-04-21T08:41:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.019+0000 7f5dc7dda700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc0108590 0x7f5dc019e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.019+0000 7f5dc7dda700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc010caf0 con 0x7f5dc0108590 2024-04-21T08:41:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.019+0000 7f5dc5b76700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc0108590 0x7f5dc019e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.019+0000 7f5dc5b76700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc0108590 0x7f5dc019e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36610/0 (socket says 172.21.15.28:36610) 2024-04-21T08:41:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.019+0000 7f5dc5b76700 1 -- 172.21.15.28:0/1816796001 learned_addr learned my addr 172.21.15.28:0/1816796001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.019+0000 7f5dc5b76700 1 -- 172.21.15.28:0/1816796001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc019ec90 con 0x7f5dc0108590 2024-04-21T08:41:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.019+0000 7f5dc5b76700 1 --2- 172.21.15.28:0/1816796001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc0108590 0x7f5dc019e750 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5db00117b0 tx=0x7f5db0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.019+0000 7f5db6ffd700 1 -- 172.21.15.28:0/1816796001 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5db0003af0 con 0x7f5dc0108590 2024-04-21T08:41:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.019+0000 7f5dc7dda700 1 -- 172.21.15.28:0/1816796001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc019ef20 con 0x7f5dc0108590 2024-04-21T08:41:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.019+0000 7f5dc7dda700 1 -- 172.21.15.28:0/1816796001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dc01a1c10 con 0x7f5dc0108590 2024-04-21T08:41:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.019+0000 7f5db6ffd700 1 -- 172.21.15.28:0/1816796001 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5db0020070 con 0x7f5dc0108590 2024-04-21T08:41:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.019+0000 7f5db6ffd700 1 -- 172.21.15.28:0/1816796001 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5db002a060 con 0x7f5dc0108590 2024-04-21T08:41:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.020+0000 7f5db6ffd700 1 -- 172.21.15.28:0/1816796001 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f5db00195a0 con 0x7f5dc0108590 2024-04-21T08:41:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.020+0000 7f5dc7dda700 1 -- 172.21.15.28:0/1816796001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5da4005360 con 0x7f5dc0108590 2024-04-21T08:41:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.020+0000 7f5db6ffd700 1 --2- 172.21.15.28:0/1816796001 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f5dac0395b0 0x7f5dac03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.020+0000 7f5db6ffd700 1 -- 172.21.15.28:0/1816796001 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5db004dc60 con 0x7f5dc0108590 2024-04-21T08:41:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.021+0000 7f5dc5375700 1 --2- 172.21.15.28:0/1816796001 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f5dac0395b0 0x7f5dac03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.021+0000 7f5dc5375700 1 --2- 172.21.15.28:0/1816796001 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f5dac0395b0 0x7f5dac03ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc006fd0 tx=0x7f5dbc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.026+0000 7f5db6ffd700 1 -- 172.21.15.28:0/1816796001 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5db001a430 con 0x7f5dc0108590 2024-04-21T08:41:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:15.239+0000 7f5dc7dda700 1 -- 172.21.15.28:0/1816796001 --> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}) -- 0x7f5da4000bf0 con 0x7f5dac0395b0 2024-04-21T08:41:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:19.508+0000 7f5db6ffd700 1 -- 172.21.15.28:0/1816796001 <== mgr.14120 v2:172.21.15.28:6800/334321812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f5da4000bf0 con 0x7f5dac0395b0 2024-04-21T08:41:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:19.510+0000 7f5dc7dda700 1 -- 172.21.15.28:0/1816796001 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f5dac0395b0 msgr2=0x7f5dac03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:19.510+0000 7f5dc7dda700 1 --2- 172.21.15.28:0/1816796001 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f5dac0395b0 0x7f5dac03ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc006fd0 tx=0x7f5dbc006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:19.510+0000 7f5dc7dda700 1 -- 172.21.15.28:0/1816796001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc0108590 msgr2=0x7f5dc019e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:19.510+0000 7f5dc7dda700 1 --2- 172.21.15.28:0/1816796001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc0108590 0x7f5dc019e750 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5db00117b0 tx=0x7f5db0011b70 comp rx=0 tx=0).stop 2024-04-21T08:41:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:19.511+0000 7f5dc7dda700 1 -- 172.21.15.28:0/1816796001 shutdown_connections 2024-04-21T08:41:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:19.511+0000 7f5dc7dda700 1 --2- 172.21.15.28:0/1816796001 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f5dac0395b0 0x7f5dac03ba70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:19.511+0000 7f5dc7dda700 1 --2- 172.21.15.28:0/1816796001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dc0108590 0x7f5dc019e750 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:19.511+0000 7f5dc7dda700 1 -- 172.21.15.28:0/1816796001 >> 172.21.15.28:0/1816796001 conn(0x7f5dc0103b70 msgr2=0x7f5dc0104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:19.511+0000 7f5dc7dda700 1 -- 172.21.15.28:0/1816796001 shutdown_connections 2024-04-21T08:41:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:19.511+0000 7f5dc7dda700 1 -- 172.21.15.28:0/1816796001 wait complete. 2024-04-21T08:41:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mon service... 2024-04-21T08:41:20.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:20 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' 2024-04-21T08:41:20.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:20 smithi028 ceph-mon[28003]: Added host smithi028 2024-04-21T08:41:20.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:20 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:21.990 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:21 smithi028 ceph-mon[28003]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:21.990 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:21 smithi028 ceph-mon[28003]: Saving service mon spec with placement count:5 2024-04-21T08:41:21.990 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:21 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' 2024-04-21T08:41:21.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:21 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' 2024-04-21T08:41:22.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-21T08:41:22.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.561+0000 7fd4a9e5a700 1 Processor -- start 2024-04-21T08:41:22.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.561+0000 7fd4a9e5a700 1 -- start start 2024-04-21T08:41:22.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.561+0000 7fd4a9e5a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4a4109000 0x7fd4a410b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:22.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.561+0000 7fd4a9e5a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a407be80 con 0x7fd4a4109000 2024-04-21T08:41:22.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.561+0000 7fd4a37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4a4109000 0x7fd4a410b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:22.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.561+0000 7fd4a37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4a4109000 0x7fd4a410b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36624/0 (socket says 172.21.15.28:36624) 2024-04-21T08:41:22.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.561+0000 7fd4a37fe700 1 -- 172.21.15.28:0/4189800818 learned_addr learned my addr 172.21.15.28:0/4189800818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:22.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.562+0000 7fd4a37fe700 1 -- 172.21.15.28:0/4189800818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a410b930 con 0x7fd4a4109000 2024-04-21T08:41:22.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.562+0000 7fd4a37fe700 1 --2- 172.21.15.28:0/4189800818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4a4109000 0x7fd4a410b3f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd4940098a0 tx=0x7fd494009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11dfa6a09d26a1a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:22.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.562+0000 7fd4a27fc700 1 -- 172.21.15.28:0/4189800818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd494004160 con 0x7fd4a4109000 2024-04-21T08:41:22.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.562+0000 7fd4a27fc700 1 -- 172.21.15.28:0/4189800818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd494004300 con 0x7fd4a4109000 2024-04-21T08:41:22.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.562+0000 7fd4a27fc700 1 -- 172.21.15.28:0/4189800818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd494011af0 con 0x7fd4a4109000 2024-04-21T08:41:22.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.562+0000 7fd4a9e5a700 1 -- 172.21.15.28:0/4189800818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4a4109000 msgr2=0x7fd4a410b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:22.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.562+0000 7fd4a9e5a700 1 --2- 172.21.15.28:0/4189800818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4a4109000 0x7fd4a410b3f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd4940098a0 tx=0x7fd494009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:22.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.563+0000 7fd4a9e5a700 1 -- 172.21.15.28:0/4189800818 shutdown_connections 2024-04-21T08:41:22.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.563+0000 7fd4a9e5a700 1 --2- 172.21.15.28:0/4189800818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4a4109000 0x7fd4a410b3f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:22.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.563+0000 7fd4a9e5a700 1 -- 172.21.15.28:0/4189800818 >> 172.21.15.28:0/4189800818 conn(0x7fd4a4102a90 msgr2=0x7fd4a4104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:22.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.563+0000 7fd4a9e5a700 1 -- 172.21.15.28:0/4189800818 shutdown_connections 2024-04-21T08:41:22.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.563+0000 7fd4a9e5a700 1 -- 172.21.15.28:0/4189800818 wait complete. 2024-04-21T08:41:22.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.563+0000 7fd4a9e5a700 1 Processor -- start 2024-04-21T08:41:22.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.563+0000 7fd4a9e5a700 1 -- start start 2024-04-21T08:41:22.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.563+0000 7fd4a9e5a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4a4109000 0x7fd4a4108880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:22.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.563+0000 7fd4a9e5a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a410c8f0 con 0x7fd4a4109000 2024-04-21T08:41:22.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.564+0000 7fd4a37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4a4109000 0x7fd4a4108880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.564+0000 7fd4a37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4a4109000 0x7fd4a4108880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36638/0 (socket says 172.21.15.28:36638) 2024-04-21T08:41:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.564+0000 7fd4a37fe700 1 -- 172.21.15.28:0/1612948788 learned_addr learned my addr 172.21.15.28:0/1612948788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.564+0000 7fd4a37fe700 1 -- 172.21.15.28:0/1612948788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a4108dc0 con 0x7fd4a4109000 2024-04-21T08:41:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.564+0000 7fd4a37fe700 1 --2- 172.21.15.28:0/1612948788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4a4109000 0x7fd4a4108880 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd494011ac0 tx=0x7fd494011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.564+0000 7fd4a0ff9700 1 -- 172.21.15.28:0/1612948788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd494003970 con 0x7fd4a4109000 2024-04-21T08:41:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.564+0000 7fd4a9e5a700 1 -- 172.21.15.28:0/1612948788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4a4106fb0 con 0x7fd4a4109000 2024-04-21T08:41:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.564+0000 7fd4a0ff9700 1 -- 172.21.15.28:0/1612948788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd494003b10 con 0x7fd4a4109000 2024-04-21T08:41:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.564+0000 7fd4a0ff9700 1 -- 172.21.15.28:0/1612948788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd494024610 con 0x7fd4a4109000 2024-04-21T08:41:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.564+0000 7fd4a9e5a700 1 -- 172.21.15.28:0/1612948788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4a41074f0 con 0x7fd4a4109000 2024-04-21T08:41:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.565+0000 7fd4a0ff9700 1 -- 172.21.15.28:0/1612948788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fd494024890 con 0x7fd4a4109000 2024-04-21T08:41:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.565+0000 7fd4a0ff9700 1 --2- 172.21.15.28:0/1612948788 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fd48c0395b0 0x7fd48c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.565+0000 7fd4a9e5a700 1 -- 172.21.15.28:0/1612948788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4a40640a0 con 0x7fd4a4109000 2024-04-21T08:41:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.566+0000 7fd4a0ff9700 1 -- 172.21.15.28:0/1612948788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd49404db00 con 0x7fd4a4109000 2024-04-21T08:41:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.566+0000 7fd4a2ffd700 1 --2- 172.21.15.28:0/1612948788 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fd48c0395b0 0x7fd48c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:22.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.566+0000 7fd4a2ffd700 1 --2- 172.21.15.28:0/1612948788 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fd48c0395b0 0x7fd48c03ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd498006fd0 tx=0x7fd498006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.571+0000 7fd4a0ff9700 1 -- 172.21.15.28:0/1612948788 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd494019c70 con 0x7fd4a4109000 2024-04-21T08:41:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.800+0000 7fd4a9e5a700 1 -- 172.21.15.28:0/1612948788 --> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fd4a4103cd0 con 0x7fd48c0395b0 2024-04-21T08:41:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.805+0000 7fd4a0ff9700 1 -- 172.21.15.28:0/1612948788 <== mgr.14120 v2:172.21.15.28:6800/334321812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd4a4103cd0 con 0x7fd48c0395b0 2024-04-21T08:41:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.807+0000 7fd4a9e5a700 1 -- 172.21.15.28:0/1612948788 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fd48c0395b0 msgr2=0x7fd48c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.807+0000 7fd4a9e5a700 1 --2- 172.21.15.28:0/1612948788 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fd48c0395b0 0x7fd48c03ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd498006fd0 tx=0x7fd498006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.807+0000 7fd4a9e5a700 1 -- 172.21.15.28:0/1612948788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4a4109000 msgr2=0x7fd4a4108880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.807+0000 7fd4a9e5a700 1 --2- 172.21.15.28:0/1612948788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4a4109000 0x7fd4a4108880 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd494011ac0 tx=0x7fd494011ba0 comp rx=0 tx=0).stop 2024-04-21T08:41:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.807+0000 7fd4a9e5a700 1 -- 172.21.15.28:0/1612948788 shutdown_connections 2024-04-21T08:41:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.807+0000 7fd4a9e5a700 1 --2- 172.21.15.28:0/1612948788 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fd48c0395b0 0x7fd48c03ba70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.807+0000 7fd4a9e5a700 1 --2- 172.21.15.28:0/1612948788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4a4109000 0x7fd4a4108880 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.807+0000 7fd4a9e5a700 1 -- 172.21.15.28:0/1612948788 >> 172.21.15.28:0/1612948788 conn(0x7fd4a4102a90 msgr2=0x7fd4a41035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.807+0000 7fd4a9e5a700 1 -- 172.21.15.28:0/1612948788 shutdown_connections 2024-04-21T08:41:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:20.807+0000 7fd4a9e5a700 1 -- 172.21.15.28:0/1612948788 wait complete. 2024-04-21T08:41:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mgr service... 2024-04-21T08:41:23.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:23 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' 2024-04-21T08:41:23.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:23 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' 2024-04-21T08:41:23.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-21T08:41:23.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.723+0000 7f4bb190f700 1 Processor -- start 2024-04-21T08:41:23.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.723+0000 7f4bb190f700 1 -- start start 2024-04-21T08:41:23.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.723+0000 7f4bb190f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac109000 0x7f4bac10b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:23.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.723+0000 7f4bb190f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac063d20 con 0x7f4bac109000 2024-04-21T08:41:23.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.724+0000 7f4baaffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac109000 0x7f4bac10b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:23.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.724+0000 7f4baaffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac109000 0x7f4bac10b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36650/0 (socket says 172.21.15.28:36650) 2024-04-21T08:41:23.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.724+0000 7f4baaffd700 1 -- 172.21.15.28:0/740208435 learned_addr learned my addr 172.21.15.28:0/740208435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:23.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.724+0000 7f4baaffd700 1 -- 172.21.15.28:0/740208435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bac10b930 con 0x7f4bac109000 2024-04-21T08:41:23.697 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.724+0000 7f4baaffd700 1 --2- 172.21.15.28:0/740208435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac109000 0x7f4bac10b3f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c0098a0 tx=0x7f4b9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c562942a8075037 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:23.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.724+0000 7f4ba9ffb700 1 -- 172.21.15.28:0/740208435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b9c004160 con 0x7f4bac109000 2024-04-21T08:41:23.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.724+0000 7f4ba9ffb700 1 -- 172.21.15.28:0/740208435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b9c004300 con 0x7f4bac109000 2024-04-21T08:41:23.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.724+0000 7f4ba9ffb700 1 -- 172.21.15.28:0/740208435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b9c011a20 con 0x7f4bac109000 2024-04-21T08:41:23.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.725+0000 7f4bb190f700 1 -- 172.21.15.28:0/740208435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac109000 msgr2=0x7f4bac10b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:23.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.725+0000 7f4bb190f700 1 --2- 172.21.15.28:0/740208435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac109000 0x7f4bac10b3f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c0098a0 tx=0x7f4b9c009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:23.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.725+0000 7f4bb190f700 1 -- 172.21.15.28:0/740208435 shutdown_connections 2024-04-21T08:41:23.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.725+0000 7f4bb190f700 1 --2- 172.21.15.28:0/740208435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac109000 0x7f4bac10b3f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:23.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.725+0000 7f4bb190f700 1 -- 172.21.15.28:0/740208435 >> 172.21.15.28:0/740208435 conn(0x7f4bac1029f0 msgr2=0x7f4bac104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:23.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.725+0000 7f4bb190f700 1 -- 172.21.15.28:0/740208435 shutdown_connections 2024-04-21T08:41:23.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.725+0000 7f4bb190f700 1 -- 172.21.15.28:0/740208435 wait complete. 2024-04-21T08:41:23.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.726+0000 7f4bb190f700 1 Processor -- start 2024-04-21T08:41:23.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.726+0000 7f4bb190f700 1 -- start start 2024-04-21T08:41:23.698 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.726+0000 7f4bb190f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac109000 0x7f4bac108880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.726+0000 7f4bb190f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac10c930 con 0x7f4bac109000 2024-04-21T08:41:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.726+0000 7f4baaffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac109000 0x7f4bac108880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.726+0000 7f4baaffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac109000 0x7f4bac108880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36652/0 (socket says 172.21.15.28:36652) 2024-04-21T08:41:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.726+0000 7f4baaffd700 1 -- 172.21.15.28:0/567045180 learned_addr learned my addr 172.21.15.28:0/567045180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.727+0000 7f4baaffd700 1 -- 172.21.15.28:0/567045180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bac108dc0 con 0x7f4bac109000 2024-04-21T08:41:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.727+0000 7f4baaffd700 1 --2- 172.21.15.28:0/567045180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac109000 0x7f4bac108880 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c000c00 tx=0x7f4b9c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.727+0000 7f4bb090d700 1 -- 172.21.15.28:0/567045180 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b9c003aa0 con 0x7f4bac109000 2024-04-21T08:41:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.727+0000 7f4bb190f700 1 -- 172.21.15.28:0/567045180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bac106fb0 con 0x7f4bac109000 2024-04-21T08:41:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.727+0000 7f4bb090d700 1 -- 172.21.15.28:0/567045180 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b9c01b8f0 con 0x7f4bac109000 2024-04-21T08:41:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.727+0000 7f4bb190f700 1 -- 172.21.15.28:0/567045180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bac1074f0 con 0x7f4bac109000 2024-04-21T08:41:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.727+0000 7f4bb090d700 1 -- 172.21.15.28:0/567045180 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b9c023870 con 0x7f4bac109000 2024-04-21T08:41:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.728+0000 7f4bb090d700 1 -- 172.21.15.28:0/567045180 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4b9c023b20 con 0x7f4bac109000 2024-04-21T08:41:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.728+0000 7f4bb090d700 1 --2- 172.21.15.28:0/567045180 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f4b98039600 0x7f4b9803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.728+0000 7f4bb190f700 1 -- 172.21.15.28:0/567045180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b8c005360 con 0x7f4bac109000 2024-04-21T08:41:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.728+0000 7f4bb090d700 1 -- 172.21.15.28:0/567045180 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4b9c04e150 con 0x7f4bac109000 2024-04-21T08:41:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.728+0000 7f4baa7fc700 1 --2- 172.21.15.28:0/567045180 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f4b98039600 0x7f4b9803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.729+0000 7f4baa7fc700 1 --2- 172.21.15.28:0/567045180 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f4b98039600 0x7f4b9803bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4b94006fd0 tx=0x7f4b94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.733+0000 7f4bb090d700 1 -- 172.21.15.28:0/567045180 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b9c01f030 con 0x7f4bac109000 2024-04-21T08:41:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.975+0000 7f4bb190f700 1 -- 172.21.15.28:0/567045180 --> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f4b8c000bf0 con 0x7f4b98039600 2024-04-21T08:41:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.979+0000 7f4bb090d700 1 -- 172.21.15.28:0/567045180 <== mgr.14120 v2:172.21.15.28:6800/334321812 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4b8c000bf0 con 0x7f4b98039600 2024-04-21T08:41:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.982+0000 7f4bb190f700 1 -- 172.21.15.28:0/567045180 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f4b98039600 msgr2=0x7f4b9803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.982+0000 7f4bb190f700 1 --2- 172.21.15.28:0/567045180 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f4b98039600 0x7f4b9803bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4b94006fd0 tx=0x7f4b94006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.982+0000 7f4bb190f700 1 -- 172.21.15.28:0/567045180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac109000 msgr2=0x7f4bac108880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.982+0000 7f4bb190f700 1 --2- 172.21.15.28:0/567045180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac109000 0x7f4bac108880 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c000c00 tx=0x7f4b9c011aa0 comp rx=0 tx=0).stop 2024-04-21T08:41:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.982+0000 7f4bb190f700 1 -- 172.21.15.28:0/567045180 shutdown_connections 2024-04-21T08:41:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.982+0000 7f4bb190f700 1 --2- 172.21.15.28:0/567045180 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f4b98039600 0x7f4b9803bac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.982+0000 7f4bb190f700 1 --2- 172.21.15.28:0/567045180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac109000 0x7f4bac108880 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.982+0000 7f4bb190f700 1 -- 172.21.15.28:0/567045180 >> 172.21.15.28:0/567045180 conn(0x7f4bac1029f0 msgr2=0x7f4bac1034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.982+0000 7f4bb190f700 1 -- 172.21.15.28:0/567045180 shutdown_connections 2024-04-21T08:41:23.701 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:22.982+0000 7f4bb190f700 1 -- 172.21.15.28:0/567045180 wait complete. 2024-04-21T08:41:24.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:24 smithi028 ceph-mon[28003]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:24.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:24 smithi028 ceph-mon[28003]: Saving service mgr spec with placement count:2 2024-04-21T08:41:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.520+0000 7fe2d9e80700 1 Processor -- start 2024-04-21T08:41:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.520+0000 7fe2d9e80700 1 -- start start 2024-04-21T08:41:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.521+0000 7fe2d9e80700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d4108fc0 0x7fe2d410b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.521+0000 7fe2d9e80700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d4063d60 con 0x7fe2d4108fc0 2024-04-21T08:41:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.521+0000 7fe2d37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d4108fc0 0x7fe2d410b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.521+0000 7fe2d37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d4108fc0 0x7fe2d410b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53424/0 (socket says 172.21.15.28:53424) 2024-04-21T08:41:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.521+0000 7fe2d37fe700 1 -- 172.21.15.28:0/553308670 learned_addr learned my addr 172.21.15.28:0/553308670 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.521+0000 7fe2d37fe700 1 -- 172.21.15.28:0/553308670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d410b8f0 con 0x7fe2d4108fc0 2024-04-21T08:41:25.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.521+0000 7fe2d37fe700 1 --2- 172.21.15.28:0/553308670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d4108fc0 0x7fe2d410b3b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe2c40098a0 tx=0x7fe2c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e9f5f64280b489 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.522+0000 7fe2d27fc700 1 -- 172.21.15.28:0/553308670 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2c4004160 con 0x7fe2d4108fc0 2024-04-21T08:41:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.522+0000 7fe2d27fc700 1 -- 172.21.15.28:0/553308670 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2c4004300 con 0x7fe2d4108fc0 2024-04-21T08:41:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.522+0000 7fe2d27fc700 1 -- 172.21.15.28:0/553308670 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2c4011a20 con 0x7fe2d4108fc0 2024-04-21T08:41:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.522+0000 7fe2d9e80700 1 -- 172.21.15.28:0/553308670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d4108fc0 msgr2=0x7fe2d410b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.522+0000 7fe2d9e80700 1 --2- 172.21.15.28:0/553308670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d4108fc0 0x7fe2d410b3b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe2c40098a0 tx=0x7fe2c4009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.522+0000 7fe2d9e80700 1 -- 172.21.15.28:0/553308670 shutdown_connections 2024-04-21T08:41:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.522+0000 7fe2d9e80700 1 --2- 172.21.15.28:0/553308670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d4108fc0 0x7fe2d410b3b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.522+0000 7fe2d9e80700 1 -- 172.21.15.28:0/553308670 >> 172.21.15.28:0/553308670 conn(0x7fe2d41029f0 msgr2=0x7fe2d4104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.522+0000 7fe2d9e80700 1 -- 172.21.15.28:0/553308670 shutdown_connections 2024-04-21T08:41:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.522+0000 7fe2d9e80700 1 -- 172.21.15.28:0/553308670 wait complete. 2024-04-21T08:41:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.523+0000 7fe2d9e80700 1 Processor -- start 2024-04-21T08:41:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.523+0000 7fe2d9e80700 1 -- start start 2024-04-21T08:41:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.523+0000 7fe2d9e80700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d4108fc0 0x7fe2d419c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:25.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.523+0000 7fe2d9e80700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d410c900 con 0x7fe2d4108fc0 2024-04-21T08:41:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.523+0000 7fe2d37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d4108fc0 0x7fe2d419c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.524+0000 7fe2d37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d4108fc0 0x7fe2d419c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53438/0 (socket says 172.21.15.28:53438) 2024-04-21T08:41:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.524+0000 7fe2d37fe700 1 -- 172.21.15.28:0/146474312 learned_addr learned my addr 172.21.15.28:0/146474312 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.524+0000 7fe2d37fe700 1 -- 172.21.15.28:0/146474312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d419c940 con 0x7fe2d4108fc0 2024-04-21T08:41:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.524+0000 7fe2d37fe700 1 --2- 172.21.15.28:0/146474312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d4108fc0 0x7fe2d419c400 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4000c00 tx=0x7fe2c4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.524+0000 7fe2d0ff9700 1 -- 172.21.15.28:0/146474312 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2c4003aa0 con 0x7fe2d4108fc0 2024-04-21T08:41:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.524+0000 7fe2d0ff9700 1 -- 172.21.15.28:0/146474312 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2c401b8f0 con 0x7fe2d4108fc0 2024-04-21T08:41:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.524+0000 7fe2d0ff9700 1 -- 172.21.15.28:0/146474312 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2c4023870 con 0x7fe2d4108fc0 2024-04-21T08:41:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.524+0000 7fe2d9e80700 1 -- 172.21.15.28:0/146474312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2d419cbd0 con 0x7fe2d4108fc0 2024-04-21T08:41:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.524+0000 7fe2d9e80700 1 -- 172.21.15.28:0/146474312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2d419f8c0 con 0x7fe2d4108fc0 2024-04-21T08:41:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.525+0000 7fe2d0ff9700 1 -- 172.21.15.28:0/146474312 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fe2c4023b70 con 0x7fe2d4108fc0 2024-04-21T08:41:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.525+0000 7fe2d9e80700 1 -- 172.21.15.28:0/146474312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2d40640a0 con 0x7fe2d4108fc0 2024-04-21T08:41:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.525+0000 7fe2d0ff9700 1 --2- 172.21.15.28:0/146474312 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fe2bc0395b0 0x7fe2bc03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.525+0000 7fe2d0ff9700 1 -- 172.21.15.28:0/146474312 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe2c404de40 con 0x7fe2d4108fc0 2024-04-21T08:41:25.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.526+0000 7fe2d2ffd700 1 --2- 172.21.15.28:0/146474312 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fe2bc0395b0 0x7fe2bc03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.526+0000 7fe2d2ffd700 1 --2- 172.21.15.28:0/146474312 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fe2bc0395b0 0x7fe2bc03ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8006fd0 tx=0x7fe2c8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.531+0000 7fe2d0ff9700 1 -- 172.21.15.28:0/146474312 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe2c401f030 con 0x7fe2d4108fc0 2024-04-21T08:41:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.739+0000 7fe2d9e80700 1 -- 172.21.15.28:0/146474312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fe2d419feb0 con 0x7fe2d4108fc0 2024-04-21T08:41:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.741+0000 7fe2d0ff9700 1 -- 172.21.15.28:0/146474312 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fe2c401a3d0 con 0x7fe2d4108fc0 2024-04-21T08:41:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.744+0000 7fe2d9e80700 1 -- 172.21.15.28:0/146474312 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fe2bc0395b0 msgr2=0x7fe2bc03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.744+0000 7fe2d9e80700 1 --2- 172.21.15.28:0/146474312 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fe2bc0395b0 0x7fe2bc03ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8006fd0 tx=0x7fe2c8006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.744+0000 7fe2d9e80700 1 -- 172.21.15.28:0/146474312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d4108fc0 msgr2=0x7fe2d419c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.744+0000 7fe2d9e80700 1 --2- 172.21.15.28:0/146474312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d4108fc0 0x7fe2d419c400 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe2c4000c00 tx=0x7fe2c4011aa0 comp rx=0 tx=0).stop 2024-04-21T08:41:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.744+0000 7fe2d9e80700 1 -- 172.21.15.28:0/146474312 shutdown_connections 2024-04-21T08:41:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.744+0000 7fe2d9e80700 1 --2- 172.21.15.28:0/146474312 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fe2bc0395b0 0x7fe2bc03ba70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.744+0000 7fe2d9e80700 1 --2- 172.21.15.28:0/146474312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d4108fc0 0x7fe2d419c400 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.744+0000 7fe2d9e80700 1 -- 172.21.15.28:0/146474312 >> 172.21.15.28:0/146474312 conn(0x7fe2d41029f0 msgr2=0x7fe2d41034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.744+0000 7fe2d9e80700 1 -- 172.21.15.28:0/146474312 shutdown_connections 2024-04-21T08:41:25.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:24.744+0000 7fe2d9e80700 1 -- 172.21.15.28:0/146474312 wait complete. 2024-04-21T08:41:26.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:25 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/146474312' entity='client.admin' 2024-04-21T08:41:26.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.714+0000 7f1054edf700 1 Processor -- start 2024-04-21T08:41:26.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.714+0000 7f1054edf700 1 -- start start 2024-04-21T08:41:26.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.715+0000 7f1054edf700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1050109020 0x7f105010b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.715+0000 7f1054edf700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1050063d00 con 0x7f1050109020 2024-04-21T08:41:26.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.715+0000 7f104e59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1050109020 0x7f105010b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.715+0000 7f104e59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1050109020 0x7f105010b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53452/0 (socket says 172.21.15.28:53452) 2024-04-21T08:41:26.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.715+0000 7f104e59c700 1 -- 172.21.15.28:0/3901453898 learned_addr learned my addr 172.21.15.28:0/3901453898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:26.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.715+0000 7f104e59c700 1 -- 172.21.15.28:0/3901453898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105010b950 con 0x7f1050109020 2024-04-21T08:41:26.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.715+0000 7f104e59c700 1 --2- 172.21.15.28:0/3901453898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1050109020 0x7f105010b410 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f10400098a0 tx=0x7f1040009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801104ba3a58daac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:26.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.715+0000 7f104d59a700 1 -- 172.21.15.28:0/3901453898 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1040004160 con 0x7f1050109020 2024-04-21T08:41:26.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.716+0000 7f104d59a700 1 -- 172.21.15.28:0/3901453898 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1040004300 con 0x7f1050109020 2024-04-21T08:41:26.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.716+0000 7f104d59a700 1 -- 172.21.15.28:0/3901453898 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1040011a20 con 0x7f1050109020 2024-04-21T08:41:26.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.716+0000 7f1054edf700 1 -- 172.21.15.28:0/3901453898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1050109020 msgr2=0x7f105010b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:26.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.716+0000 7f1054edf700 1 --2- 172.21.15.28:0/3901453898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1050109020 0x7f105010b410 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f10400098a0 tx=0x7f1040009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.717+0000 7f1054edf700 1 -- 172.21.15.28:0/3901453898 shutdown_connections 2024-04-21T08:41:26.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.717+0000 7f1054edf700 1 --2- 172.21.15.28:0/3901453898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1050109020 0x7f105010b410 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.717+0000 7f1054edf700 1 -- 172.21.15.28:0/3901453898 >> 172.21.15.28:0/3901453898 conn(0x7f10501029f0 msgr2=0x7f1050104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:26.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.717+0000 7f1054edf700 1 -- 172.21.15.28:0/3901453898 shutdown_connections 2024-04-21T08:41:26.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.717+0000 7f1054edf700 1 -- 172.21.15.28:0/3901453898 wait complete. 2024-04-21T08:41:26.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.717+0000 7f1054edf700 1 Processor -- start 2024-04-21T08:41:26.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.718+0000 7f1054edf700 1 -- start start 2024-04-21T08:41:26.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.718+0000 7f1054edf700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1050109020 0x7f105019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.718+0000 7f1054edf700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f105010c900 con 0x7f1050109020 2024-04-21T08:41:26.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.718+0000 7f104e59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1050109020 0x7f105019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.718+0000 7f104e59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1050109020 0x7f105019c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53458/0 (socket says 172.21.15.28:53458) 2024-04-21T08:41:26.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.718+0000 7f104e59c700 1 -- 172.21.15.28:0/2777294366 learned_addr learned my addr 172.21.15.28:0/2777294366 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:26.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.718+0000 7f104e59c700 1 -- 172.21.15.28:0/2777294366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105019c960 con 0x7f1050109020 2024-04-21T08:41:26.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.718+0000 7f104e59c700 1 --2- 172.21.15.28:0/2777294366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1050109020 0x7f105019c420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f10400116e0 tx=0x7f1040011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:26.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.718+0000 7f103f7fe700 1 -- 172.21.15.28:0/2777294366 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1040003a60 con 0x7f1050109020 2024-04-21T08:41:26.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.719+0000 7f1054edf700 1 -- 172.21.15.28:0/2777294366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f105019cbf0 con 0x7f1050109020 2024-04-21T08:41:26.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.719+0000 7f103f7fe700 1 -- 172.21.15.28:0/2777294366 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f104001b8f0 con 0x7f1050109020 2024-04-21T08:41:26.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.719+0000 7f103f7fe700 1 -- 172.21.15.28:0/2777294366 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1040019860 con 0x7f1050109020 2024-04-21T08:41:26.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.719+0000 7f1054edf700 1 -- 172.21.15.28:0/2777294366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f105019f8e0 con 0x7f1050109020 2024-04-21T08:41:26.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.719+0000 7f103f7fe700 1 -- 172.21.15.28:0/2777294366 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f1040019b30 con 0x7f1050109020 2024-04-21T08:41:26.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.720+0000 7f1054edf700 1 -- 172.21.15.28:0/2777294366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10500640a0 con 0x7f1050109020 2024-04-21T08:41:26.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.720+0000 7f103f7fe700 1 --2- 172.21.15.28:0/2777294366 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f103803d9c0 0x7f103803fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:26.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.720+0000 7f103f7fe700 1 -- 172.21.15.28:0/2777294366 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f104004ec60 con 0x7f1050109020 2024-04-21T08:41:26.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.720+0000 7f104dd9b700 1 --2- 172.21.15.28:0/2777294366 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f103803d9c0 0x7f103803fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:26.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.720+0000 7f104dd9b700 1 --2- 172.21.15.28:0/2777294366 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f103803d9c0 0x7f103803fe80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1044006fd0 tx=0x7f1044006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:26.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.725+0000 7f103f7fe700 1 -- 172.21.15.28:0/2777294366 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f104001f030 con 0x7f1050109020 2024-04-21T08:41:26.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.934+0000 7f1054edf700 1 -- 172.21.15.28:0/2777294366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f1050107260 con 0x7f1050109020 2024-04-21T08:41:26.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.937+0000 7f103f7fe700 1 -- 172.21.15.28:0/2777294366 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f1040029260 con 0x7f1050109020 2024-04-21T08:41:26.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.940+0000 7f1054edf700 1 -- 172.21.15.28:0/2777294366 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f103803d9c0 msgr2=0x7f103803fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:26.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.940+0000 7f1054edf700 1 --2- 172.21.15.28:0/2777294366 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f103803d9c0 0x7f103803fe80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1044006fd0 tx=0x7f1044006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:26.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.940+0000 7f1054edf700 1 -- 172.21.15.28:0/2777294366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1050109020 msgr2=0x7f105019c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:26.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.940+0000 7f1054edf700 1 --2- 172.21.15.28:0/2777294366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1050109020 0x7f105019c420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f10400116e0 tx=0x7f1040011aa0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.940+0000 7f1054edf700 1 -- 172.21.15.28:0/2777294366 shutdown_connections 2024-04-21T08:41:26.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.940+0000 7f1054edf700 1 --2- 172.21.15.28:0/2777294366 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f103803d9c0 0x7f103803fe80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.940+0000 7f1054edf700 1 --2- 172.21.15.28:0/2777294366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1050109020 0x7f105019c420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:26.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.940+0000 7f1054edf700 1 -- 172.21.15.28:0/2777294366 >> 172.21.15.28:0/2777294366 conn(0x7f10501029f0 msgr2=0x7f1050103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:26.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.940+0000 7f1054edf700 1 -- 172.21.15.28:0/2777294366 shutdown_connections 2024-04-21T08:41:26.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:25.940+0000 7f1054edf700 1 -- 172.21.15.28:0/2777294366 wait complete. 2024-04-21T08:41:26.489 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling the dashboard module... 2024-04-21T08:41:26.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:26 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' 2024-04-21T08:41:26.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:26 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2777294366' entity='client.admin' 2024-04-21T08:41:28.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:28 smithi028 ceph-mon[28003]: from='mgr.14120 172.21.15.28:0/2824909112' entity='mgr.x' 2024-04-21T08:41:28.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:28 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1640190867' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-21T08:41:28.410 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:28 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: ignoring --setuser ceph since I am not root 2024-04-21T08:41:28.410 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:28 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: ignoring --setgroup ceph since I am not root 2024-04-21T08:41:28.410 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:28 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:28.410+0000 7f33bb1dc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:41:28.709 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:28 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:28.602+0000 7f33bb1dc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:41:28.709 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:28 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:28.709+0000 7f33bb1dc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:41:28.886 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.972+0000 7fa6aac6d700 1 Processor -- start 2024-04-21T08:41:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.972+0000 7fa6aac6d700 1 -- start start 2024-04-21T08:41:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.972+0000 7fa6aac6d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6a4108fe0 0x7fa6a410b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.972+0000 7fa6aac6d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a407bea0 con 0x7fa6a4108fe0 2024-04-21T08:41:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.973+0000 7fa6a8a09700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6a4108fe0 0x7fa6a410b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.973+0000 7fa6a8a09700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6a4108fe0 0x7fa6a410b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53472/0 (socket says 172.21.15.28:53472) 2024-04-21T08:41:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.973+0000 7fa6a8a09700 1 -- 172.21.15.28:0/2658290323 learned_addr learned my addr 172.21.15.28:0/2658290323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.973+0000 7fa6a8a09700 1 -- 172.21.15.28:0/2658290323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a410b910 con 0x7fa6a4108fe0 2024-04-21T08:41:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.973+0000 7fa6a8a09700 1 --2- 172.21.15.28:0/2658290323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6a4108fe0 0x7fa6a410b3d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa6980098a0 tx=0x7fa698009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88acca6bedb15b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.973+0000 7fa6a37fe700 1 -- 172.21.15.28:0/2658290323 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa698004160 con 0x7fa6a4108fe0 2024-04-21T08:41:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.974+0000 7fa6a37fe700 1 -- 172.21.15.28:0/2658290323 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa698004300 con 0x7fa6a4108fe0 2024-04-21T08:41:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.974+0000 7fa6a37fe700 1 -- 172.21.15.28:0/2658290323 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa698011a20 con 0x7fa6a4108fe0 2024-04-21T08:41:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.974+0000 7fa6aac6d700 1 -- 172.21.15.28:0/2658290323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6a4108fe0 msgr2=0x7fa6a410b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:28.887 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.974+0000 7fa6aac6d700 1 --2- 172.21.15.28:0/2658290323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6a4108fe0 0x7fa6a410b3d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa6980098a0 tx=0x7fa698009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.975+0000 7fa6aac6d700 1 -- 172.21.15.28:0/2658290323 shutdown_connections 2024-04-21T08:41:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.975+0000 7fa6aac6d700 1 --2- 172.21.15.28:0/2658290323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6a4108fe0 0x7fa6a410b3d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.975+0000 7fa6aac6d700 1 -- 172.21.15.28:0/2658290323 >> 172.21.15.28:0/2658290323 conn(0x7fa6a41029d0 msgr2=0x7fa6a4104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.975+0000 7fa6aac6d700 1 -- 172.21.15.28:0/2658290323 shutdown_connections 2024-04-21T08:41:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.975+0000 7fa6aac6d700 1 -- 172.21.15.28:0/2658290323 wait complete. 2024-04-21T08:41:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.975+0000 7fa6aac6d700 1 Processor -- start 2024-04-21T08:41:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.975+0000 7fa6aac6d700 1 -- start start 2024-04-21T08:41:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.975+0000 7fa6aac6d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6a4108fe0 0x7fa6a419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.975+0000 7fa6aac6d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a410c8b0 con 0x7fa6a4108fe0 2024-04-21T08:41:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.976+0000 7fa6a8a09700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6a4108fe0 0x7fa6a419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.976+0000 7fa6a8a09700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6a4108fe0 0x7fa6a419e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53486/0 (socket says 172.21.15.28:53486) 2024-04-21T08:41:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.976+0000 7fa6a8a09700 1 -- 172.21.15.28:0/1640190867 learned_addr learned my addr 172.21.15.28:0/1640190867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.976+0000 7fa6a8a09700 1 -- 172.21.15.28:0/1640190867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a419eba0 con 0x7fa6a4108fe0 2024-04-21T08:41:28.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.976+0000 7fa6a8a09700 1 --2- 172.21.15.28:0/1640190867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6a4108fe0 0x7fa6a419e660 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa6980116e0 tx=0x7fa698011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.976+0000 7fa6a1ffb700 1 -- 172.21.15.28:0/1640190867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa698003b20 con 0x7fa6a4108fe0 2024-04-21T08:41:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.976+0000 7fa6a1ffb700 1 -- 172.21.15.28:0/1640190867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa69801b8f0 con 0x7fa6a4108fe0 2024-04-21T08:41:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.976+0000 7fa6aac6d700 1 -- 172.21.15.28:0/1640190867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6a419ee90 con 0x7fa6a4108fe0 2024-04-21T08:41:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.976+0000 7fa6a1ffb700 1 -- 172.21.15.28:0/1640190867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa6980238c0 con 0x7fa6a4108fe0 2024-04-21T08:41:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.976+0000 7fa6aac6d700 1 -- 172.21.15.28:0/1640190867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6a41a1b20 con 0x7fa6a4108fe0 2024-04-21T08:41:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.977+0000 7fa6a1ffb700 1 -- 172.21.15.28:0/1640190867 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fa698023b70 con 0x7fa6a4108fe0 2024-04-21T08:41:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.977+0000 7fa6a1ffb700 1 --2- 172.21.15.28:0/1640190867 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fa694039560 0x7fa69403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.977+0000 7fa6a1ffb700 1 -- 172.21.15.28:0/1640190867 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa69804eeb0 con 0x7fa6a4108fe0 2024-04-21T08:41:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.978+0000 7fa6a3fff700 1 --2- 172.21.15.28:0/1640190867 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fa694039560 0x7fa69403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.978+0000 7fa6aac6d700 1 -- 172.21.15.28:0/1640190867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6a40640a0 con 0x7fa6a4108fe0 2024-04-21T08:41:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.978+0000 7fa6a3fff700 1 --2- 172.21.15.28:0/1640190867 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fa694039560 0x7fa69403ba20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa690006fd0 tx=0x7fa690006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:26.983+0000 7fa6a1ffb700 1 -- 172.21.15.28:0/1640190867 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa69801f030 con 0x7fa6a4108fe0 2024-04-21T08:41:28.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:27.258+0000 7fa6aac6d700 1 -- 172.21.15.28:0/1640190867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fa6a41a20f0 con 0x7fa6a4108fe0 2024-04-21T08:41:28.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:28.227+0000 7fa6a1ffb700 1 -- 172.21.15.28:0/1640190867 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fa698018e30 con 0x7fa6a4108fe0 2024-04-21T08:41:28.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:28.228+0000 7fa6a1ffb700 1 -- 172.21.15.28:0/1640190867 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fa69801baa0 con 0x7fa6a4108fe0 2024-04-21T08:41:28.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:28.230+0000 7fa6aac6d700 1 -- 172.21.15.28:0/1640190867 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fa694039560 msgr2=0x7fa69403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:28.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:28.230+0000 7fa6aac6d700 1 --2- 172.21.15.28:0/1640190867 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fa694039560 0x7fa69403ba20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa690006fd0 tx=0x7fa690006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:28.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:28.230+0000 7fa6aac6d700 1 -- 172.21.15.28:0/1640190867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6a4108fe0 msgr2=0x7fa6a419e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:28.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:28.230+0000 7fa6aac6d700 1 --2- 172.21.15.28:0/1640190867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6a4108fe0 0x7fa6a419e660 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa6980116e0 tx=0x7fa698011aa0 comp rx=0 tx=0).stop 2024-04-21T08:41:28.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:28.231+0000 7fa6aac6d700 1 -- 172.21.15.28:0/1640190867 shutdown_connections 2024-04-21T08:41:28.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:28.231+0000 7fa6aac6d700 1 --2- 172.21.15.28:0/1640190867 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fa694039560 0x7fa69403ba20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:28.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:28.231+0000 7fa6aac6d700 1 --2- 172.21.15.28:0/1640190867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6a4108fe0 0x7fa6a419e660 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:28.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:28.231+0000 7fa6aac6d700 1 -- 172.21.15.28:0/1640190867 >> 172.21.15.28:0/1640190867 conn(0x7fa6a41029d0 msgr2=0x7fa6a41034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:28.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:28.231+0000 7fa6aac6d700 1 -- 172.21.15.28:0/1640190867 shutdown_connections 2024-04-21T08:41:28.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:28.231+0000 7fa6aac6d700 1 -- 172.21.15.28:0/1640190867 wait complete. 2024-04-21T08:41:29.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:29 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1640190867' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-21T08:41:29.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:29 smithi028 ceph-mon[28003]: mgrmap e9: x(active, since 20s) 2024-04-21T08:41:29.266 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:29 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:29.042+0000 7f33bb1dc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:41:29.266 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:29 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:29.134+0000 7f33bb1dc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:41:30.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:30 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3116140454' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-21T08:41:30.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:41:30.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-21T08:41:30.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T08:41:30.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-21T08:41:30.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-21T08:41:30.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:41:30.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.285+0000 7fbadf107700 1 Processor -- start 2024-04-21T08:41:30.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.285+0000 7fbadf107700 1 -- start start 2024-04-21T08:41:30.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.286+0000 7fbadf107700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad8108540 0x7fbad8108960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:30.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.286+0000 7fbadf107700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbad8108f30 con 0x7fbad8108540 2024-04-21T08:41:30.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.286+0000 7fbadcea3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad8108540 0x7fbad8108960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:30.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.286+0000 7fbadcea3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad8108540 0x7fbad8108960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53504/0 (socket says 172.21.15.28:53504) 2024-04-21T08:41:30.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.286+0000 7fbadcea3700 1 -- 172.21.15.28:0/1284658576 learned_addr learned my addr 172.21.15.28:0/1284658576 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:30.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.286+0000 7fbadcea3700 1 -- 172.21.15.28:0/1284658576 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad81096f0 con 0x7fbad8108540 2024-04-21T08:41:30.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.286+0000 7fbadcea3700 1 --2- 172.21.15.28:0/1284658576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad8108540 0x7fbad8108960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbacc0098a0 tx=0x7fbacc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff499f4077c9e224 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:30.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.286+0000 7fbad77fe700 1 -- 172.21.15.28:0/1284658576 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbacc004160 con 0x7fbad8108540 2024-04-21T08:41:30.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.287+0000 7fbad77fe700 1 -- 172.21.15.28:0/1284658576 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbacc004300 con 0x7fbad8108540 2024-04-21T08:41:30.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.287+0000 7fbad77fe700 1 -- 172.21.15.28:0/1284658576 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbacc011a20 con 0x7fbad8108540 2024-04-21T08:41:30.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.287+0000 7fbadf107700 1 -- 172.21.15.28:0/1284658576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad8108540 msgr2=0x7fbad8108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:30.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.287+0000 7fbadf107700 1 --2- 172.21.15.28:0/1284658576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad8108540 0x7fbad8108960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbacc0098a0 tx=0x7fbacc009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.287+0000 7fbadf107700 1 -- 172.21.15.28:0/1284658576 shutdown_connections 2024-04-21T08:41:30.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.287+0000 7fbadf107700 1 --2- 172.21.15.28:0/1284658576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad8108540 0x7fbad8108960 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.287+0000 7fbadf107700 1 -- 172.21.15.28:0/1284658576 >> 172.21.15.28:0/1284658576 conn(0x7fbad8103ae0 msgr2=0x7fbad8105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:30.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.287+0000 7fbadf107700 1 -- 172.21.15.28:0/1284658576 shutdown_connections 2024-04-21T08:41:30.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.287+0000 7fbadf107700 1 -- 172.21.15.28:0/1284658576 wait complete. 2024-04-21T08:41:30.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.288+0000 7fbadf107700 1 Processor -- start 2024-04-21T08:41:30.456 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.288+0000 7fbadf107700 1 -- start start 2024-04-21T08:41:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.288+0000 7fbadf107700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad8108540 0x7fbad819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.288+0000 7fbadf107700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbad810caa0 con 0x7fbad8108540 2024-04-21T08:41:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.288+0000 7fbadcea3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad8108540 0x7fbad819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.288+0000 7fbadcea3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad8108540 0x7fbad819e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53520/0 (socket says 172.21.15.28:53520) 2024-04-21T08:41:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.288+0000 7fbadcea3700 1 -- 172.21.15.28:0/3116140454 learned_addr learned my addr 172.21.15.28:0/3116140454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.289+0000 7fbadcea3700 1 -- 172.21.15.28:0/3116140454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad819ebc0 con 0x7fbad8108540 2024-04-21T08:41:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.289+0000 7fbadcea3700 1 --2- 172.21.15.28:0/3116140454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad8108540 0x7fbad819e680 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbacc000c00 tx=0x7fbacc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.289+0000 7fbad5ffb700 1 -- 172.21.15.28:0/3116140454 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbacc003aa0 con 0x7fbad8108540 2024-04-21T08:41:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.289+0000 7fbadf107700 1 -- 172.21.15.28:0/3116140454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbad819ee50 con 0x7fbad8108540 2024-04-21T08:41:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.289+0000 7fbad5ffb700 1 -- 172.21.15.28:0/3116140454 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbacc01b8f0 con 0x7fbad8108540 2024-04-21T08:41:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.289+0000 7fbad5ffb700 1 -- 172.21.15.28:0/3116140454 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbacc023890 con 0x7fbad8108540 2024-04-21T08:41:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.289+0000 7fbadf107700 1 -- 172.21.15.28:0/3116140454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbad81a1b40 con 0x7fbad8108540 2024-04-21T08:41:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.290+0000 7fbadf107700 1 -- 172.21.15.28:0/3116140454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbad80640a0 con 0x7fbad8108540 2024-04-21T08:41:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.290+0000 7fbad5ffb700 1 -- 172.21.15.28:0/3116140454 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fbacc02c3f0 con 0x7fbad8108540 2024-04-21T08:41:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.290+0000 7fbad5ffb700 1 --2- 172.21.15.28:0/3116140454 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fbac8039600 0x7fbac803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.290+0000 7fbad5ffb700 1 -- 172.21.15.28:0/3116140454 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbacc04e080 con 0x7fbad8108540 2024-04-21T08:41:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.290+0000 7fbad7fff700 1 -- 172.21.15.28:0/3116140454 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fbac8039600 msgr2=0x7fbac803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/334321812 2024-04-21T08:41:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.290+0000 7fbad7fff700 1 --2- 172.21.15.28:0/3116140454 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fbac8039600 0x7fbac803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:41:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.295+0000 7fbad5ffb700 1 -- 172.21.15.28:0/3116140454 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbacc01f030 con 0x7fbad8108540 2024-04-21T08:41:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.491+0000 7fbad7fff700 1 -- 172.21.15.28:0/3116140454 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fbac8039600 msgr2=0x7fbac803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/334321812 2024-04-21T08:41:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.491+0000 7fbad7fff700 1 --2- 172.21.15.28:0/3116140454 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fbac8039600 0x7fbac803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:41:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.609+0000 7fbadf107700 1 -- 172.21.15.28:0/3116140454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fbad81a1d10 con 0x7fbad8108540 2024-04-21T08:41:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.609+0000 7fbad5ffb700 1 -- 172.21.15.28:0/3116140454 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fbacc00be50 con 0x7fbad8108540 2024-04-21T08:41:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.612+0000 7fbadf107700 1 -- 172.21.15.28:0/3116140454 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fbac8039600 msgr2=0x7fbac803bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:41:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.612+0000 7fbadf107700 1 --2- 172.21.15.28:0/3116140454 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fbac8039600 0x7fbac803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.612+0000 7fbadf107700 1 -- 172.21.15.28:0/3116140454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad8108540 msgr2=0x7fbad819e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.612+0000 7fbadf107700 1 --2- 172.21.15.28:0/3116140454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad8108540 0x7fbad819e680 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbacc000c00 tx=0x7fbacc011aa0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.612+0000 7fbadf107700 1 -- 172.21.15.28:0/3116140454 shutdown_connections 2024-04-21T08:41:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.612+0000 7fbadf107700 1 --2- 172.21.15.28:0/3116140454 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7fbac8039600 0x7fbac803bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.458 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.612+0000 7fbadf107700 1 --2- 172.21.15.28:0/3116140454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad8108540 0x7fbad819e680 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:30.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.612+0000 7fbadf107700 1 -- 172.21.15.28:0/3116140454 >> 172.21.15.28:0/3116140454 conn(0x7fbad8103ae0 msgr2=0x7fbad8104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:30.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.612+0000 7fbadf107700 1 -- 172.21.15.28:0/3116140454 shutdown_connections 2024-04-21T08:41:30.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:29.612+0000 7fbadf107700 1 -- 172.21.15.28:0/3116140454 wait complete. 2024-04-21T08:41:30.459 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-04-21T08:41:30.459 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 9... 2024-04-21T08:41:30.682 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:30 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:30.397+0000 7f33bb1dc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:41:30.682 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:30 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:30.579+0000 7f33bb1dc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:41:31.065 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:30 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:30.760+0000 7f33bb1dc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:41:31.354 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:31 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:31.066+0000 7f33bb1dc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:41:31.355 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:31 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:31.163+0000 7f33bb1dc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:41:31.855 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:31 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:31.374+0000 7f33bb1dc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:41:31.855 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:31 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:31.575+0000 7f33bb1dc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:41:32.604 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:32 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:32.202+0000 7f33bb1dc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:41:32.952 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:32 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:32.642+0000 7f33bb1dc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:41:32.952 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:32 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:32.742+0000 7f33bb1dc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:41:32.952 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:32 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:32.953+0000 7f33bb1dc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:41:33.847 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:33 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:33.498+0000 7f33bb1dc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:41:33.847 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:33 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:33.594+0000 7f33bb1dc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:41:34.104 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:33 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:33.847+0000 7f33bb1dc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:41:34.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:33 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:33.985+0000 7f33bb1dc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:41:34.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:34 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:34.085+0000 7f33bb1dc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:41:34.517 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:34 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:34.205+0000 7f33bb1dc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:41:34.517 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:34 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:34.518+0000 7f33bb1dc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:41:35.424 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:35 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:35.147+0000 7f33bb1dc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:41:35.424 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:41:35 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:41:35.372+0000 7f33bb1dc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:41:35.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:35 smithi028 ceph-mon[28003]: Active manager daemon x restarted 2024-04-21T08:41:35.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:35 smithi028 ceph-mon[28003]: Activating manager daemon x 2024-04-21T08:41:35.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:35 smithi028 ceph-mon[28003]: osdmap e3: 0 total, 0 up, 0 in 2024-04-21T08:41:35.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:35 smithi028 ceph-mon[28003]: mgrmap e10: x(active, starting, since 0.00352158s) 2024-04-21T08:41:35.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:41:35.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:41:35.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:41:35.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:41:35.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:41:35.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:35 smithi028 ceph-mon[28003]: Manager daemon x is now available 2024-04-21T08:41:35.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:36.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:41:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:41:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:36 smithi028 ceph-mon[28003]: [21/Apr/2024:08:41:35] ENGINE Bus STARTING 2024-04-21T08:41:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:36 smithi028 ceph-mon[28003]: [21/Apr/2024:08:41:35] ENGINE Serving on https://172.21.15.28:7150 2024-04-21T08:41:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:36 smithi028 ceph-mon[28003]: [21/Apr/2024:08:41:36] ENGINE Serving on http://172.21.15.28:8765 2024-04-21T08:41:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:36 smithi028 ceph-mon[28003]: [21/Apr/2024:08:41:36] ENGINE Bus STARTED 2024-04-21T08:41:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:36 smithi028 ceph-mon[28003]: mgrmap e11: x(active, since 1.00616s) 2024-04-21T08:41:37.388 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:41:37.388 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-21T08:41:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-21T08:41:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:41:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.877+0000 7f2b29a05700 1 Processor -- start 2024-04-21T08:41:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.877+0000 7f2b29a05700 1 -- start start 2024-04-21T08:41:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.877+0000 7f2b29a05700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24108590 0x7f2b241089b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.877+0000 7f2b29a05700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b24108f80 con 0x7f2b24108590 2024-04-21T08:41:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.877+0000 7f2b22ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24108590 0x7f2b241089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.877+0000 7f2b22ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24108590 0x7f2b241089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53532/0 (socket says 172.21.15.28:53532) 2024-04-21T08:41:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.877+0000 7f2b22ffd700 1 -- 172.21.15.28:0/1521686375 learned_addr learned my addr 172.21.15.28:0/1521686375 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.877+0000 7f2b22ffd700 1 -- 172.21.15.28:0/1521686375 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b24109740 con 0x7f2b24108590 2024-04-21T08:41:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.878+0000 7f2b22ffd700 1 --2- 172.21.15.28:0/1521686375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24108590 0x7f2b241089b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2b140098a0 tx=0x7f2b14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d4b64546c611f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.878+0000 7f2b21ffb700 1 -- 172.21.15.28:0/1521686375 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b14004160 con 0x7f2b24108590 2024-04-21T08:41:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.878+0000 7f2b21ffb700 1 -- 172.21.15.28:0/1521686375 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b14004300 con 0x7f2b24108590 2024-04-21T08:41:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.878+0000 7f2b21ffb700 1 -- 172.21.15.28:0/1521686375 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b14011a20 con 0x7f2b24108590 2024-04-21T08:41:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.878+0000 7f2b29a05700 1 -- 172.21.15.28:0/1521686375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24108590 msgr2=0x7f2b241089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.878+0000 7f2b29a05700 1 --2- 172.21.15.28:0/1521686375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24108590 0x7f2b241089b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2b140098a0 tx=0x7f2b14009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.878+0000 7f2b29a05700 1 -- 172.21.15.28:0/1521686375 shutdown_connections 2024-04-21T08:41:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.878+0000 7f2b29a05700 1 --2- 172.21.15.28:0/1521686375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24108590 0x7f2b241089b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.878+0000 7f2b29a05700 1 -- 172.21.15.28:0/1521686375 >> 172.21.15.28:0/1521686375 conn(0x7f2b24103b70 msgr2=0x7f2b24105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.878+0000 7f2b29a05700 1 -- 172.21.15.28:0/1521686375 shutdown_connections 2024-04-21T08:41:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.878+0000 7f2b29a05700 1 -- 172.21.15.28:0/1521686375 wait complete. 2024-04-21T08:41:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.879+0000 7f2b29a05700 1 Processor -- start 2024-04-21T08:41:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.879+0000 7f2b29a05700 1 -- start start 2024-04-21T08:41:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.879+0000 7f2b29a05700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24108590 0x7f2b2419c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.879+0000 7f2b29a05700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2410caf0 con 0x7f2b24108590 2024-04-21T08:41:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.880+0000 7f2b22ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24108590 0x7f2b2419c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.880+0000 7f2b22ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24108590 0x7f2b2419c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53548/0 (socket says 172.21.15.28:53548) 2024-04-21T08:41:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.880+0000 7f2b22ffd700 1 -- 172.21.15.28:0/2422437206 learned_addr learned my addr 172.21.15.28:0/2422437206 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.880+0000 7f2b22ffd700 1 -- 172.21.15.28:0/2422437206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b2419c9f0 con 0x7f2b24108590 2024-04-21T08:41:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.880+0000 7f2b22ffd700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24108590 0x7f2b2419c4b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2b140119f0 tx=0x7f2b14011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.880+0000 7f2b28a03700 1 -- 172.21.15.28:0/2422437206 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b140039e0 con 0x7f2b24108590 2024-04-21T08:41:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.880+0000 7f2b29a05700 1 -- 172.21.15.28:0/2422437206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b2419cc80 con 0x7f2b24108590 2024-04-21T08:41:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.880+0000 7f2b28a03700 1 -- 172.21.15.28:0/2422437206 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b1401b8f0 con 0x7f2b24108590 2024-04-21T08:41:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.880+0000 7f2b28a03700 1 -- 172.21.15.28:0/2422437206 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b140237a0 con 0x7f2b24108590 2024-04-21T08:41:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.880+0000 7f2b29a05700 1 -- 172.21.15.28:0/2422437206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b2419f970 con 0x7f2b24108590 2024-04-21T08:41:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.881+0000 7f2b28a03700 1 -- 172.21.15.28:0/2422437206 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f2b1402c3f0 con 0x7f2b24108590 2024-04-21T08:41:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.881+0000 7f2b28a03700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f2b0c0395b0 0x7f2b0c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.881+0000 7f2b29a05700 1 -- 172.21.15.28:0/2422437206 --> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2b24069ba0 con 0x7f2b0c0395b0 2024-04-21T08:41:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.881+0000 7f2b28a03700 1 -- 172.21.15.28:0/2422437206 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2b1404dee0 con 0x7f2b24108590 2024-04-21T08:41:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.881+0000 7f2b227fc700 1 -- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f2b0c0395b0 msgr2=0x7f2b0c03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/334321812 2024-04-21T08:41:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:30.882+0000 7f2b227fc700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f2b0c0395b0 0x7f2b0c03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:41:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:31.083+0000 7f2b227fc700 1 -- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f2b0c0395b0 msgr2=0x7f2b0c03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/334321812 2024-04-21T08:41:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:31.083+0000 7f2b227fc700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f2b0c0395b0 0x7f2b0c03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:41:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:31.484+0000 7f2b227fc700 1 -- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f2b0c0395b0 msgr2=0x7f2b0c03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/334321812 2024-04-21T08:41:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:31.484+0000 7f2b227fc700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f2b0c0395b0 0x7f2b0c03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T08:41:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:32.285+0000 7f2b227fc700 1 -- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f2b0c0395b0 msgr2=0x7f2b0c03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/334321812 2024-04-21T08:41:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:32.285+0000 7f2b227fc700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f2b0c0395b0 0x7f2b0c03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-21T08:41:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:33.886+0000 7f2b227fc700 1 -- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f2b0c0395b0 msgr2=0x7f2b0c03ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/334321812 2024-04-21T08:41:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:33.886+0000 7f2b227fc700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f2b0c0395b0 0x7f2b0c03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-21T08:41:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:35.375+0000 7f2b28a03700 1 -- 172.21.15.28:0/2422437206 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f2b1402bc70 con 0x7f2b24108590 2024-04-21T08:41:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:35.375+0000 7f2b28a03700 1 -- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f2b0c0395b0 msgr2=0x7f2b0c03ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:41:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:35.375+0000 7f2b28a03700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f2b0c0395b0 0x7f2b0c03ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.377+0000 7f2b28a03700 1 -- 172.21.15.28:0/2422437206 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f2b1404fad0 con 0x7f2b24108590 2024-04-21T08:41:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.377+0000 7f2b28a03700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2b0c03ce40 0x7f2b0c03f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.378+0000 7f2b28a03700 1 -- 172.21.15.28:0/2422437206 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2b24069ba0 con 0x7f2b0c03ce40 2024-04-21T08:41:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.378+0000 7f2b227fc700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2b0c03ce40 0x7f2b0c03f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:37.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.379+0000 7f2b227fc700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2b0c03ce40 0x7f2b0c03f230 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2b18003a10 tx=0x7f2b180092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:37.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.380+0000 7f2b28a03700 1 -- 172.21.15.28:0/2422437206 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f2b24069ba0 con 0x7f2b0c03ce40 2024-04-21T08:41:37.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.386+0000 7f2b29a05700 1 -- 172.21.15.28:0/2422437206 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f2b24104dc0 con 0x7f2b0c03ce40 2024-04-21T08:41:37.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.386+0000 7f2b28a03700 1 -- 172.21.15.28:0/2422437206 <== mgr.14152 v2:172.21.15.28:6800/226897930 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f2b24104dc0 con 0x7f2b0c03ce40 2024-04-21T08:41:37.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.387+0000 7f2b29a05700 1 -- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2b0c03ce40 msgr2=0x7f2b0c03f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.387+0000 7f2b29a05700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2b0c03ce40 0x7f2b0c03f230 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2b18003a10 tx=0x7f2b180092b0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.387+0000 7f2b29a05700 1 -- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24108590 msgr2=0x7f2b2419c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:37.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.387+0000 7f2b29a05700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24108590 0x7f2b2419c4b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2b140119f0 tx=0x7f2b14011ad0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.387+0000 7f2b29a05700 1 -- 172.21.15.28:0/2422437206 shutdown_connections 2024-04-21T08:41:37.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.387+0000 7f2b29a05700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2b0c03ce40 0x7f2b0c03f230 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.387+0000 7f2b29a05700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24108590 0x7f2b2419c4b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.387+0000 7f2b29a05700 1 --2- 172.21.15.28:0/2422437206 >> [v2:172.21.15.28:6800/334321812,v1:172.21.15.28:6801/334321812] conn(0x7f2b0c0395b0 0x7f2b0c03ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:37.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.387+0000 7f2b29a05700 1 -- 172.21.15.28:0/2422437206 >> 172.21.15.28:0/2422437206 conn(0x7f2b24103b70 msgr2=0x7f2b241046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:37.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.387+0000 7f2b29a05700 1 -- 172.21.15.28:0/2422437206 shutdown_connections 2024-04-21T08:41:37.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:36.387+0000 7f2b29a05700 1 -- 172.21.15.28:0/2422437206 wait complete. 2024-04-21T08:41:37.394 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 9 is available 2024-04-21T08:41:37.394 INFO:teuthology.orchestra.run.smithi028.stdout:Generating a dashboard self-signed certificate... 2024-04-21T08:41:38.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:37 smithi028 ceph-mon[28003]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-21T08:41:38.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:37 smithi028 ceph-mon[28003]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-21T08:41:38.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:41:38.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-21T08:41:38.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.786+0000 7f28c487a700 1 Processor -- start 2024-04-21T08:41:38.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.786+0000 7f28c487a700 1 -- start start 2024-04-21T08:41:38.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.787+0000 7f28c487a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0106820 0x7f28c0106c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:38.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.787+0000 7f28c487a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28c0107210 con 0x7f28c0106820 2024-04-21T08:41:38.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.787+0000 7f28be59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0106820 0x7f28c0106c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:38.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.787+0000 7f28be59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0106820 0x7f28c0106c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50998/0 (socket says 172.21.15.28:50998) 2024-04-21T08:41:38.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.787+0000 7f28be59c700 1 -- 172.21.15.28:0/2956960958 learned_addr learned my addr 172.21.15.28:0/2956960958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:38.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.787+0000 7f28be59c700 1 -- 172.21.15.28:0/2956960958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c01097b0 con 0x7f28c0106820 2024-04-21T08:41:38.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.787+0000 7f28be59c700 1 --2- 172.21.15.28:0/2956960958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0106820 0x7f28c0106c40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f28b00098a0 tx=0x7f28b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1f439cb4a342ef server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:38.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.787+0000 7f28bd59a700 1 -- 172.21.15.28:0/2956960958 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28b0004160 con 0x7f28c0106820 2024-04-21T08:41:38.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.787+0000 7f28bd59a700 1 -- 172.21.15.28:0/2956960958 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28b0004300 con 0x7f28c0106820 2024-04-21T08:41:38.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.787+0000 7f28bd59a700 1 -- 172.21.15.28:0/2956960958 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28b0011a20 con 0x7f28c0106820 2024-04-21T08:41:38.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.788+0000 7f28c487a700 1 -- 172.21.15.28:0/2956960958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0106820 msgr2=0x7f28c0106c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:38.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.788+0000 7f28c487a700 1 --2- 172.21.15.28:0/2956960958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0106820 0x7f28c0106c40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f28b00098a0 tx=0x7f28b0009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:38.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.788+0000 7f28c487a700 1 -- 172.21.15.28:0/2956960958 shutdown_connections 2024-04-21T08:41:38.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.788+0000 7f28c487a700 1 --2- 172.21.15.28:0/2956960958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0106820 0x7f28c0106c40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:38.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.788+0000 7f28c487a700 1 -- 172.21.15.28:0/2956960958 >> 172.21.15.28:0/2956960958 conn(0x7f28c0102130 msgr2=0x7f28c0104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:38.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.788+0000 7f28c487a700 1 -- 172.21.15.28:0/2956960958 shutdown_connections 2024-04-21T08:41:38.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.788+0000 7f28c487a700 1 -- 172.21.15.28:0/2956960958 wait complete. 2024-04-21T08:41:38.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.789+0000 7f28c487a700 1 Processor -- start 2024-04-21T08:41:38.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.789+0000 7f28c487a700 1 -- start start 2024-04-21T08:41:38.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.789+0000 7f28c487a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0106820 0x7f28c019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:38.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.789+0000 7f28c487a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28c010cb60 con 0x7f28c0106820 2024-04-21T08:41:38.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.789+0000 7f28be59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0106820 0x7f28c019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:38.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.789+0000 7f28be59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0106820 0x7f28c019c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51012/0 (socket says 172.21.15.28:51012) 2024-04-21T08:41:38.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.789+0000 7f28be59c700 1 -- 172.21.15.28:0/211498932 learned_addr learned my addr 172.21.15.28:0/211498932 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:38.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.789+0000 7f28be59c700 1 -- 172.21.15.28:0/211498932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c019c980 con 0x7f28c0106820 2024-04-21T08:41:38.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.790+0000 7f28be59c700 1 --2- 172.21.15.28:0/211498932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0106820 0x7f28c019c440 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f28b0009520 tx=0x7f28b0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:38.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.790+0000 7f28af7fe700 1 -- 172.21.15.28:0/211498932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28b00039e0 con 0x7f28c0106820 2024-04-21T08:41:38.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.790+0000 7f28af7fe700 1 -- 172.21.15.28:0/211498932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28b001b8f0 con 0x7f28c0106820 2024-04-21T08:41:38.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.790+0000 7f28c487a700 1 -- 172.21.15.28:0/211498932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c019cc10 con 0x7f28c0106820 2024-04-21T08:41:38.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.790+0000 7f28af7fe700 1 -- 172.21.15.28:0/211498932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28b0019770 con 0x7f28c0106820 2024-04-21T08:41:38.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.790+0000 7f28c487a700 1 -- 172.21.15.28:0/211498932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c019f900 con 0x7f28c0106820 2024-04-21T08:41:38.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.791+0000 7f28af7fe700 1 -- 172.21.15.28:0/211498932 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f28b0019a70 con 0x7f28c0106820 2024-04-21T08:41:38.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.791+0000 7f28c487a700 1 -- 172.21.15.28:0/211498932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28c00640a0 con 0x7f28c0106820 2024-04-21T08:41:38.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.791+0000 7f28af7fe700 1 --2- 172.21.15.28:0/211498932 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f28a80394e0 0x7f28a803b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:38.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.791+0000 7f28af7fe700 1 -- 172.21.15.28:0/211498932 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f28b004dfa0 con 0x7f28c0106820 2024-04-21T08:41:38.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.791+0000 7f28bdd9b700 1 --2- 172.21.15.28:0/211498932 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f28a80394e0 0x7f28a803b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:38.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.791+0000 7f28bdd9b700 1 --2- 172.21.15.28:0/211498932 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f28a80394e0 0x7f28a803b9a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f28b4006fd0 tx=0x7f28b4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:38.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.796+0000 7f28af7fe700 1 -- 172.21.15.28:0/211498932 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28b001f030 con 0x7f28c0106820 2024-04-21T08:41:38.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:37.987+0000 7f28af7fe700 1 -- 172.21.15.28:0/211498932 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f28b001a8f0 con 0x7f28c0106820 2024-04-21T08:41:38.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:38.016+0000 7f28c487a700 1 -- 172.21.15.28:0/211498932 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f28c0103290 con 0x7f28a80394e0 2024-04-21T08:41:38.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:38.206+0000 7f28af7fe700 1 -- 172.21.15.28:0/211498932 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f28c0103290 con 0x7f28a80394e0 2024-04-21T08:41:38.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:38.208+0000 7f28c487a700 1 -- 172.21.15.28:0/211498932 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f28a80394e0 msgr2=0x7f28a803b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:38.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:38.208+0000 7f28c487a700 1 --2- 172.21.15.28:0/211498932 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f28a80394e0 0x7f28a803b9a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f28b4006fd0 tx=0x7f28b4006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:38.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:38.208+0000 7f28c487a700 1 -- 172.21.15.28:0/211498932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0106820 msgr2=0x7f28c019c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:38.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:38.208+0000 7f28c487a700 1 --2- 172.21.15.28:0/211498932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0106820 0x7f28c019c440 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f28b0009520 tx=0x7f28b0011aa0 comp rx=0 tx=0).stop 2024-04-21T08:41:38.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:38.208+0000 7f28c487a700 1 -- 172.21.15.28:0/211498932 shutdown_connections 2024-04-21T08:41:38.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:38.208+0000 7f28c487a700 1 --2- 172.21.15.28:0/211498932 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f28a80394e0 0x7f28a803b9a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:38.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:38.208+0000 7f28c487a700 1 --2- 172.21.15.28:0/211498932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c0106820 0x7f28c019c440 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:38.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:38.208+0000 7f28c487a700 1 -- 172.21.15.28:0/211498932 >> 172.21.15.28:0/211498932 conn(0x7f28c0102130 msgr2=0x7f28c0102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:38.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:38.208+0000 7f28c487a700 1 -- 172.21.15.28:0/211498932 shutdown_connections 2024-04-21T08:41:38.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:38.208+0000 7f28c487a700 1 -- 172.21.15.28:0/211498932 wait complete. 2024-04-21T08:41:38.754 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial admin user... 2024-04-21T08:41:39.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:38 smithi028 ceph-mon[28003]: mgrmap e12: x(active, since 2s) 2024-04-21T08:41:39.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:38 smithi028 ceph-mon[28003]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:39.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:38 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:41:39.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:38 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:41:40.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:40 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:41:40.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:40 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:41:41.221 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:41 smithi028 ceph-mon[28003]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:41.235 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$dB/OZCnDIn0w7Q703A6Cgu04nVUk/pkH.Z8MdXJJyEiPh3HiU0WcS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713688899, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-21T08:41:41.235 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.172+0000 7f06d3830700 1 Processor -- start 2024-04-21T08:41:41.235 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.173+0000 7f06d3830700 1 -- start start 2024-04-21T08:41:41.235 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.173+0000 7f06d3830700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06cc106e20 0x7f06cc107240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.235 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.173+0000 7f06d3830700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc07be40 con 0x7f06cc106e20 2024-04-21T08:41:41.235 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.173+0000 7f06d15cc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06cc106e20 0x7f06cc107240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.235 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.173+0000 7f06d15cc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06cc106e20 0x7f06cc107240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51018/0 (socket says 172.21.15.28:51018) 2024-04-21T08:41:41.235 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.173+0000 7f06d15cc700 1 -- 172.21.15.28:0/3403945172 learned_addr learned my addr 172.21.15.28:0/3403945172 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:41.235 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.173+0000 7f06d15cc700 1 -- 172.21.15.28:0/3403945172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06cc107780 con 0x7f06cc106e20 2024-04-21T08:41:41.235 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.173+0000 7f06d15cc700 1 --2- 172.21.15.28:0/3403945172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06cc106e20 0x7f06cc107240 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f06c80098a0 tx=0x7f06c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc53d26294e6ec1c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.235 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.174+0000 7f06bffff700 1 -- 172.21.15.28:0/3403945172 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06c8004160 con 0x7f06cc106e20 2024-04-21T08:41:41.235 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.174+0000 7f06bffff700 1 -- 172.21.15.28:0/3403945172 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06c8004300 con 0x7f06cc106e20 2024-04-21T08:41:41.235 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.174+0000 7f06bffff700 1 -- 172.21.15.28:0/3403945172 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06c8011a20 con 0x7f06cc106e20 2024-04-21T08:41:41.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.174+0000 7f06d3830700 1 -- 172.21.15.28:0/3403945172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06cc106e20 msgr2=0x7f06cc107240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.174+0000 7f06d3830700 1 --2- 172.21.15.28:0/3403945172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06cc106e20 0x7f06cc107240 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f06c80098a0 tx=0x7f06c8009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.174+0000 7f06d3830700 1 -- 172.21.15.28:0/3403945172 shutdown_connections 2024-04-21T08:41:41.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.174+0000 7f06d3830700 1 --2- 172.21.15.28:0/3403945172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06cc106e20 0x7f06cc107240 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.174+0000 7f06d3830700 1 -- 172.21.15.28:0/3403945172 >> 172.21.15.28:0/3403945172 conn(0x7f06cc102a60 msgr2=0x7f06cc104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.174+0000 7f06d3830700 1 -- 172.21.15.28:0/3403945172 shutdown_connections 2024-04-21T08:41:41.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.174+0000 7f06d3830700 1 -- 172.21.15.28:0/3403945172 wait complete. 2024-04-21T08:41:41.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.175+0000 7f06d3830700 1 Processor -- start 2024-04-21T08:41:41.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.175+0000 7f06d3830700 1 -- start start 2024-04-21T08:41:41.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.175+0000 7f06d3830700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06cc106e20 0x7f06cc19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.175+0000 7f06d3830700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc1087f0 con 0x7f06cc106e20 2024-04-21T08:41:41.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.175+0000 7f06d15cc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06cc106e20 0x7f06cc19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.175+0000 7f06d15cc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06cc106e20 0x7f06cc19e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51022/0 (socket says 172.21.15.28:51022) 2024-04-21T08:41:41.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.175+0000 7f06d15cc700 1 -- 172.21.15.28:0/2387205655 learned_addr learned my addr 172.21.15.28:0/2387205655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:41.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.176+0000 7f06d15cc700 1 -- 172.21.15.28:0/2387205655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06cc19ebd0 con 0x7f06cc106e20 2024-04-21T08:41:41.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.176+0000 7f06d15cc700 1 --2- 172.21.15.28:0/2387205655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06cc106e20 0x7f06cc19e690 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f06c8004660 tx=0x7f06c8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.176+0000 7f06be7fc700 1 -- 172.21.15.28:0/2387205655 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06c8003aa0 con 0x7f06cc106e20 2024-04-21T08:41:41.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.176+0000 7f06be7fc700 1 -- 172.21.15.28:0/2387205655 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06c801b8f0 con 0x7f06cc106e20 2024-04-21T08:41:41.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.176+0000 7f06d3830700 1 -- 172.21.15.28:0/2387205655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06cc19ee60 con 0x7f06cc106e20 2024-04-21T08:41:41.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.176+0000 7f06be7fc700 1 -- 172.21.15.28:0/2387205655 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06c8023900 con 0x7f06cc106e20 2024-04-21T08:41:41.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.176+0000 7f06d3830700 1 -- 172.21.15.28:0/2387205655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06cc1a1b50 con 0x7f06cc106e20 2024-04-21T08:41:41.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.177+0000 7f06be7fc700 1 -- 172.21.15.28:0/2387205655 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f06c8023b70 con 0x7f06cc106e20 2024-04-21T08:41:41.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.177+0000 7f06be7fc700 1 --2- 172.21.15.28:0/2387205655 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f06b8039660 0x7f06b803bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:41.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.177+0000 7f06be7fc700 1 -- 172.21.15.28:0/2387205655 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f06c804df70 con 0x7f06cc106e20 2024-04-21T08:41:41.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.177+0000 7f06d3830700 1 -- 172.21.15.28:0/2387205655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06cc0640a0 con 0x7f06cc106e20 2024-04-21T08:41:41.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.177+0000 7f06d0dcb700 1 --2- 172.21.15.28:0/2387205655 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f06b8039660 0x7f06b803bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:41.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.178+0000 7f06d0dcb700 1 --2- 172.21.15.28:0/2387205655 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f06b8039660 0x7f06b803bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f06c0006fd0 tx=0x7f06c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:41.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.183+0000 7f06be7fc700 1 -- 172.21.15.28:0/2387205655 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f06c801f030 con 0x7f06cc106e20 2024-04-21T08:41:41.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.410+0000 7f06d3830700 1 -- 172.21.15.28:0/2387205655 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f06cc1981f0 con 0x7f06b8039660 2024-04-21T08:41:41.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.658+0000 7f06be7fc700 1 -- 172.21.15.28:0/2387205655 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f06cc1981f0 con 0x7f06b8039660 2024-04-21T08:41:41.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.661+0000 7f06d3830700 1 -- 172.21.15.28:0/2387205655 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f06b8039660 msgr2=0x7f06b803bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.661+0000 7f06d3830700 1 --2- 172.21.15.28:0/2387205655 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f06b8039660 0x7f06b803bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f06c0006fd0 tx=0x7f06c0006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:41.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.661+0000 7f06d3830700 1 -- 172.21.15.28:0/2387205655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06cc106e20 msgr2=0x7f06cc19e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:41.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.661+0000 7f06d3830700 1 --2- 172.21.15.28:0/2387205655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06cc106e20 0x7f06cc19e690 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f06c8004660 tx=0x7f06c8011aa0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.661+0000 7f06d3830700 1 -- 172.21.15.28:0/2387205655 shutdown_connections 2024-04-21T08:41:41.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.661+0000 7f06d3830700 1 --2- 172.21.15.28:0/2387205655 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f06b8039660 0x7f06b803bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.661+0000 7f06d3830700 1 --2- 172.21.15.28:0/2387205655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06cc106e20 0x7f06cc19e690 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:41.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.661+0000 7f06d3830700 1 -- 172.21.15.28:0/2387205655 >> 172.21.15.28:0/2387205655 conn(0x7f06cc102a60 msgr2=0x7f06cc195330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:41.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.661+0000 7f06d3830700 1 -- 172.21.15.28:0/2387205655 shutdown_connections 2024-04-21T08:41:41.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:39.661+0000 7f06d3830700 1 -- 172.21.15.28:0/2387205655 wait complete. 2024-04-21T08:41:41.239 INFO:teuthology.orchestra.run.smithi028.stdout:Fetching dashboard port number... 2024-04-21T08:41:42.168 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:42 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1004607741' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-21T08:41:42.435 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 8443 2024-04-21T08:41:42.435 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.666+0000 7f9b79b95700 1 Processor -- start 2024-04-21T08:41:42.435 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.666+0000 7f9b79b95700 1 -- start start 2024-04-21T08:41:42.435 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.666+0000 7f9b79b95700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b74108590 0x7f9b741089b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:42.435 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.666+0000 7f9b79b95700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b74108f80 con 0x7f9b74108590 2024-04-21T08:41:42.435 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.666+0000 7f9b737fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b74108590 0x7f9b741089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:42.435 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.666+0000 7f9b737fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b74108590 0x7f9b741089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51034/0 (socket says 172.21.15.28:51034) 2024-04-21T08:41:42.435 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.666+0000 7f9b737fe700 1 -- 172.21.15.28:0/1392621473 learned_addr learned my addr 172.21.15.28:0/1392621473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:42.435 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.667+0000 7f9b737fe700 1 -- 172.21.15.28:0/1392621473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b74109740 con 0x7f9b74108590 2024-04-21T08:41:42.435 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.667+0000 7f9b737fe700 1 --2- 172.21.15.28:0/1392621473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b74108590 0x7f9b741089b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9b64009e80 tx=0x7f9b6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8eb54b1e077f40 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:42.435 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.667+0000 7f9b727fc700 1 -- 172.21.15.28:0/1392621473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b64017070 con 0x7f9b74108590 2024-04-21T08:41:42.436 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.667+0000 7f9b727fc700 1 -- 172.21.15.28:0/1392621473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b6400b8c0 con 0x7f9b74108590 2024-04-21T08:41:42.436 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.667+0000 7f9b727fc700 1 -- 172.21.15.28:0/1392621473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b64004490 con 0x7f9b74108590 2024-04-21T08:41:42.436 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.668+0000 7f9b79b95700 1 -- 172.21.15.28:0/1392621473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b74108590 msgr2=0x7f9b741089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:42.436 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.668+0000 7f9b79b95700 1 --2- 172.21.15.28:0/1392621473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b74108590 0x7f9b741089b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9b64009e80 tx=0x7f9b6400b330 comp rx=0 tx=0).stop 2024-04-21T08:41:42.436 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.668+0000 7f9b79b95700 1 -- 172.21.15.28:0/1392621473 shutdown_connections 2024-04-21T08:41:42.436 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.668+0000 7f9b79b95700 1 --2- 172.21.15.28:0/1392621473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b74108590 0x7f9b741089b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:42.436 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.668+0000 7f9b79b95700 1 -- 172.21.15.28:0/1392621473 >> 172.21.15.28:0/1392621473 conn(0x7f9b74103b70 msgr2=0x7f9b74105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:42.436 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.668+0000 7f9b79b95700 1 -- 172.21.15.28:0/1392621473 shutdown_connections 2024-04-21T08:41:42.436 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.668+0000 7f9b79b95700 1 -- 172.21.15.28:0/1392621473 wait complete. 2024-04-21T08:41:42.436 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.668+0000 7f9b79b95700 1 Processor -- start 2024-04-21T08:41:42.436 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.669+0000 7f9b79b95700 1 -- start start 2024-04-21T08:41:42.436 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.669+0000 7f9b79b95700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b74108590 0x7f9b7419c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:42.436 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.669+0000 7f9b79b95700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b7410caf0 con 0x7f9b74108590 2024-04-21T08:41:42.436 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.669+0000 7f9b737fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b74108590 0x7f9b7419c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.669+0000 7f9b737fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b74108590 0x7f9b7419c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51042/0 (socket says 172.21.15.28:51042) 2024-04-21T08:41:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.669+0000 7f9b737fe700 1 -- 172.21.15.28:0/1004607741 learned_addr learned my addr 172.21.15.28:0/1004607741 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.669+0000 7f9b737fe700 1 -- 172.21.15.28:0/1004607741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b7419ca10 con 0x7f9b74108590 2024-04-21T08:41:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.669+0000 7f9b737fe700 1 --2- 172.21.15.28:0/1004607741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b74108590 0x7f9b7419c4d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9b64009e50 tx=0x7f9b6400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.669+0000 7f9b70ff9700 1 -- 172.21.15.28:0/1004607741 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b64020020 con 0x7f9b74108590 2024-04-21T08:41:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.669+0000 7f9b70ff9700 1 -- 172.21.15.28:0/1004607741 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b64003b50 con 0x7f9b74108590 2024-04-21T08:41:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.669+0000 7f9b79b95700 1 -- 172.21.15.28:0/1004607741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b7419cca0 con 0x7f9b74108590 2024-04-21T08:41:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.670+0000 7f9b70ff9700 1 -- 172.21.15.28:0/1004607741 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b64017040 con 0x7f9b74108590 2024-04-21T08:41:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.670+0000 7f9b79b95700 1 -- 172.21.15.28:0/1004607741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b7419f990 con 0x7f9b74108590 2024-04-21T08:41:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.670+0000 7f9b79b95700 1 -- 172.21.15.28:0/1004607741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b740640a0 con 0x7f9b74108590 2024-04-21T08:41:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.670+0000 7f9b70ff9700 1 -- 172.21.15.28:0/1004607741 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f9b64023020 con 0x7f9b74108590 2024-04-21T08:41:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.671+0000 7f9b70ff9700 1 --2- 172.21.15.28:0/1004607741 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f9b60039610 0x7f9b6003bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.671+0000 7f9b70ff9700 1 -- 172.21.15.28:0/1004607741 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9b6400f050 con 0x7f9b74108590 2024-04-21T08:41:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.671+0000 7f9b72ffd700 1 --2- 172.21.15.28:0/1004607741 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f9b60039610 0x7f9b6003bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.671+0000 7f9b72ffd700 1 --2- 172.21.15.28:0/1004607741 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f9b60039610 0x7f9b6003bad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c006fd0 tx=0x7f9b5c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:42.438 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.676+0000 7f9b70ff9700 1 -- 172.21.15.28:0/1004607741 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9b6402a020 con 0x7f9b74108590 2024-04-21T08:41:42.438 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.883+0000 7f9b79b95700 1 -- 172.21.15.28:0/1004607741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f9b7419ff60 con 0x7f9b74108590 2024-04-21T08:41:42.438 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.884+0000 7f9b70ff9700 1 -- 172.21.15.28:0/1004607741 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f9b64003d00 con 0x7f9b74108590 2024-04-21T08:41:42.438 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.886+0000 7f9b79b95700 1 -- 172.21.15.28:0/1004607741 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f9b60039610 msgr2=0x7f9b6003bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:42.438 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.886+0000 7f9b79b95700 1 --2- 172.21.15.28:0/1004607741 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f9b60039610 0x7f9b6003bad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c006fd0 tx=0x7f9b5c006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:42.438 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.886+0000 7f9b79b95700 1 -- 172.21.15.28:0/1004607741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b74108590 msgr2=0x7f9b7419c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:42.438 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.886+0000 7f9b79b95700 1 --2- 172.21.15.28:0/1004607741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b74108590 0x7f9b7419c4d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9b64009e50 tx=0x7f9b6400bd50 comp rx=0 tx=0).stop 2024-04-21T08:41:42.438 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.886+0000 7f9b79b95700 1 -- 172.21.15.28:0/1004607741 shutdown_connections 2024-04-21T08:41:42.438 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.886+0000 7f9b79b95700 1 --2- 172.21.15.28:0/1004607741 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f9b60039610 0x7f9b6003bad0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:42.438 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.886+0000 7f9b79b95700 1 --2- 172.21.15.28:0/1004607741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b74108590 0x7f9b7419c4d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:42.438 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.886+0000 7f9b79b95700 1 -- 172.21.15.28:0/1004607741 >> 172.21.15.28:0/1004607741 conn(0x7f9b74103b70 msgr2=0x7f9b741046d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:42.438 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.886+0000 7f9b79b95700 1 -- 172.21.15.28:0/1004607741 shutdown_connections 2024-04-21T08:41:42.439 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:41.886+0000 7f9b79b95700 1 -- 172.21.15.28:0/1004607741 wait complete. 2024-04-21T08:41:42.443 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:41:42.443 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-21T08:41:42.450 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:41:42.450 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-21T08:41:42.450 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-21T08:41:42.450 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-21T08:41:42.452 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph Dashboard is now available at: 2024-04-21T08:41:42.452 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:41:42.452 INFO:teuthology.orchestra.run.smithi028.stdout: URL: https://smithi028:8443/ 2024-04-21T08:41:42.452 INFO:teuthology.orchestra.run.smithi028.stdout: User: admin 2024-04-21T08:41:42.452 INFO:teuthology.orchestra.run.smithi028.stdout: Password: k0ez797bc1 2024-04-21T08:41:42.452 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:41:42.453 INFO:teuthology.orchestra.run.smithi028.stdout:Saving cluster configuration to /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config directory 2024-04-21T08:41:42.453 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling autotune for osd_memory_target 2024-04-21T08:41:43.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:43 smithi028 ceph-mon[28003]: mgrmap e13: x(active, since 6s) 2024-04-21T08:41:43.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.969+0000 7f324f6c5700 1 Processor -- start 2024-04-21T08:41:43.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.969+0000 7f324f6c5700 1 -- start start 2024-04-21T08:41:43.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.969+0000 7f324f6c5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3248109000 0x7f324810b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.969+0000 7f324f6c5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3248063d20 con 0x7f3248109000 2024-04-21T08:41:43.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.970+0000 7f324d461700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3248109000 0x7f324810b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.970+0000 7f324d461700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3248109000 0x7f324810b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51058/0 (socket says 172.21.15.28:51058) 2024-04-21T08:41:43.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.970+0000 7f324d461700 1 -- 172.21.15.28:0/3973566118 learned_addr learned my addr 172.21.15.28:0/3973566118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:43.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.970+0000 7f324d461700 1 -- 172.21.15.28:0/3973566118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324810b930 con 0x7f3248109000 2024-04-21T08:41:43.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.970+0000 7f324d461700 1 --2- 172.21.15.28:0/3973566118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3248109000 0x7f324810b3f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f32440098a0 tx=0x7f3244009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b91e833e1ffdb6a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:43.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.970+0000 7f323ffff700 1 -- 172.21.15.28:0/3973566118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3244004160 con 0x7f3248109000 2024-04-21T08:41:43.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.970+0000 7f323ffff700 1 -- 172.21.15.28:0/3973566118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3244004300 con 0x7f3248109000 2024-04-21T08:41:43.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.971+0000 7f323ffff700 1 -- 172.21.15.28:0/3973566118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3244011a20 con 0x7f3248109000 2024-04-21T08:41:43.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.971+0000 7f324f6c5700 1 -- 172.21.15.28:0/3973566118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3248109000 msgr2=0x7f324810b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.971+0000 7f324f6c5700 1 --2- 172.21.15.28:0/3973566118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3248109000 0x7f324810b3f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f32440098a0 tx=0x7f3244009bb0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.971+0000 7f324f6c5700 1 -- 172.21.15.28:0/3973566118 shutdown_connections 2024-04-21T08:41:43.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.971+0000 7f324f6c5700 1 --2- 172.21.15.28:0/3973566118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3248109000 0x7f324810b3f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.971+0000 7f324f6c5700 1 -- 172.21.15.28:0/3973566118 >> 172.21.15.28:0/3973566118 conn(0x7f32481029f0 msgr2=0x7f3248104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:43.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.971+0000 7f324f6c5700 1 -- 172.21.15.28:0/3973566118 shutdown_connections 2024-04-21T08:41:43.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.971+0000 7f324f6c5700 1 -- 172.21.15.28:0/3973566118 wait complete. 2024-04-21T08:41:43.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.972+0000 7f324f6c5700 1 Processor -- start 2024-04-21T08:41:43.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.972+0000 7f324f6c5700 1 -- start start 2024-04-21T08:41:43.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.972+0000 7f324f6c5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3248109000 0x7f324819c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.972+0000 7f324f6c5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f324810c930 con 0x7f3248109000 2024-04-21T08:41:43.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.972+0000 7f324d461700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3248109000 0x7f324819c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.972+0000 7f324d461700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3248109000 0x7f324819c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51062/0 (socket says 172.21.15.28:51062) 2024-04-21T08:41:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.972+0000 7f324d461700 1 -- 172.21.15.28:0/1492250265 learned_addr learned my addr 172.21.15.28:0/1492250265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.973+0000 7f324d461700 1 -- 172.21.15.28:0/1492250265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324819c940 con 0x7f3248109000 2024-04-21T08:41:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.973+0000 7f324d461700 1 --2- 172.21.15.28:0/1492250265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3248109000 0x7f324819c400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f32440116e0 tx=0x7f3244011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.973+0000 7f323e7fc700 1 -- 172.21.15.28:0/1492250265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3244003a60 con 0x7f3248109000 2024-04-21T08:41:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.973+0000 7f323e7fc700 1 -- 172.21.15.28:0/1492250265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f324401b8f0 con 0x7f3248109000 2024-04-21T08:41:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.973+0000 7f324f6c5700 1 -- 172.21.15.28:0/1492250265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f324819cbd0 con 0x7f3248109000 2024-04-21T08:41:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.973+0000 7f323e7fc700 1 -- 172.21.15.28:0/1492250265 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3244019860 con 0x7f3248109000 2024-04-21T08:41:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.973+0000 7f324f6c5700 1 -- 172.21.15.28:0/1492250265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f324819f8c0 con 0x7f3248109000 2024-04-21T08:41:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.974+0000 7f323e7fc700 1 -- 172.21.15.28:0/1492250265 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f324401b440 con 0x7f3248109000 2024-04-21T08:41:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.974+0000 7f324f6c5700 1 -- 172.21.15.28:0/1492250265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32480640a0 con 0x7f3248109000 2024-04-21T08:41:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.974+0000 7f323e7fc700 1 --2- 172.21.15.28:0/1492250265 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f32340396b0 0x7f323403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:43.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.974+0000 7f323e7fc700 1 -- 172.21.15.28:0/1492250265 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f324404de90 con 0x7f3248109000 2024-04-21T08:41:43.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.974+0000 7f324cc60700 1 --2- 172.21.15.28:0/1492250265 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f32340396b0 0x7f323403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:43.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.974+0000 7f324cc60700 1 --2- 172.21.15.28:0/1492250265 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f32340396b0 0x7f323403bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3238006fd0 tx=0x7f3238006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:43.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:42.979+0000 7f323e7fc700 1 -- 172.21.15.28:0/1492250265 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f324401f030 con 0x7f3248109000 2024-04-21T08:41:43.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:43.187+0000 7f324f6c5700 1 -- 172.21.15.28:0/1492250265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f324819fa90 con 0x7f3248109000 2024-04-21T08:41:43.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:43.187+0000 7f323e7fc700 1 -- 172.21.15.28:0/1492250265 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f32440223b0 con 0x7f3248109000 2024-04-21T08:41:43.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:43.189+0000 7f324f6c5700 1 -- 172.21.15.28:0/1492250265 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f32340396b0 msgr2=0x7f323403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:43.189+0000 7f324f6c5700 1 --2- 172.21.15.28:0/1492250265 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f32340396b0 0x7f323403bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3238006fd0 tx=0x7f3238006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:43.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:43.189+0000 7f324f6c5700 1 -- 172.21.15.28:0/1492250265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3248109000 msgr2=0x7f324819c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:43.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:43.189+0000 7f324f6c5700 1 --2- 172.21.15.28:0/1492250265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3248109000 0x7f324819c400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f32440116e0 tx=0x7f3244011aa0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:43.189+0000 7f324f6c5700 1 -- 172.21.15.28:0/1492250265 shutdown_connections 2024-04-21T08:41:43.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:43.189+0000 7f324f6c5700 1 --2- 172.21.15.28:0/1492250265 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f32340396b0 0x7f323403bb70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:43.189+0000 7f324f6c5700 1 --2- 172.21.15.28:0/1492250265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3248109000 0x7f324819c400 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:43.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:43.189+0000 7f324f6c5700 1 -- 172.21.15.28:0/1492250265 >> 172.21.15.28:0/1492250265 conn(0x7f32481029f0 msgr2=0x7f32481034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:43.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:43.189+0000 7f324f6c5700 1 -- 172.21.15.28:0/1492250265 shutdown_connections 2024-04-21T08:41:43.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:43.189+0000 7f324f6c5700 1 -- 172.21.15.28:0/1492250265 wait complete. 2024-04-21T08:41:45.080 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.173+0000 7fa18f52f700 1 Processor -- start 2024-04-21T08:41:45.080 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.173+0000 7fa18f52f700 1 -- start start 2024-04-21T08:41:45.080 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.174+0000 7fa18f52f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa188106dc0 0x7fa1881071e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:45.080 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.174+0000 7fa18f52f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa18807be80 con 0x7fa188106dc0 2024-04-21T08:41:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.174+0000 7fa18d2cb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa188106dc0 0x7fa1881071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.174+0000 7fa18d2cb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa188106dc0 0x7fa1881071e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44140/0 (socket says 172.21.15.28:44140) 2024-04-21T08:41:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.174+0000 7fa18d2cb700 1 -- 172.21.15.28:0/4163911652 learned_addr learned my addr 172.21.15.28:0/4163911652 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.174+0000 7fa18d2cb700 1 -- 172.21.15.28:0/4163911652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa188107720 con 0x7fa188106dc0 2024-04-21T08:41:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.174+0000 7fa18d2cb700 1 --2- 172.21.15.28:0/4163911652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa188106dc0 0x7fa1881071e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa180009e80 tx=0x7fa18000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41809624bc015dd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.174+0000 7fa187fff700 1 -- 172.21.15.28:0/4163911652 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa180017070 con 0x7fa188106dc0 2024-04-21T08:41:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.174+0000 7fa187fff700 1 -- 172.21.15.28:0/4163911652 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa18000b8c0 con 0x7fa188106dc0 2024-04-21T08:41:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.174+0000 7fa187fff700 1 -- 172.21.15.28:0/4163911652 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa180004490 con 0x7fa188106dc0 2024-04-21T08:41:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.175+0000 7fa18f52f700 1 -- 172.21.15.28:0/4163911652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa188106dc0 msgr2=0x7fa1881071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.175+0000 7fa18f52f700 1 --2- 172.21.15.28:0/4163911652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa188106dc0 0x7fa1881071e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa180009e80 tx=0x7fa18000b330 comp rx=0 tx=0).stop 2024-04-21T08:41:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.175+0000 7fa18f52f700 1 -- 172.21.15.28:0/4163911652 shutdown_connections 2024-04-21T08:41:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.175+0000 7fa18f52f700 1 --2- 172.21.15.28:0/4163911652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa188106dc0 0x7fa1881071e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.175+0000 7fa18f52f700 1 -- 172.21.15.28:0/4163911652 >> 172.21.15.28:0/4163911652 conn(0x7fa188102a00 msgr2=0x7fa188104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.175+0000 7fa18f52f700 1 -- 172.21.15.28:0/4163911652 shutdown_connections 2024-04-21T08:41:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.175+0000 7fa18f52f700 1 -- 172.21.15.28:0/4163911652 wait complete. 2024-04-21T08:41:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.176+0000 7fa18f52f700 1 Processor -- start 2024-04-21T08:41:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.176+0000 7fa18f52f700 1 -- start start 2024-04-21T08:41:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.176+0000 7fa18f52f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa188106dc0 0x7fa18819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.176+0000 7fa18f52f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa188108790 con 0x7fa188106dc0 2024-04-21T08:41:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.176+0000 7fa18d2cb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa188106dc0 0x7fa18819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.176+0000 7fa18d2cb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa188106dc0 0x7fa18819e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44152/0 (socket says 172.21.15.28:44152) 2024-04-21T08:41:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.176+0000 7fa18d2cb700 1 -- 172.21.15.28:0/2435884493 learned_addr learned my addr 172.21.15.28:0/2435884493 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.176+0000 7fa18d2cb700 1 -- 172.21.15.28:0/2435884493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa18819ebe0 con 0x7fa188106dc0 2024-04-21T08:41:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.176+0000 7fa18d2cb700 1 --2- 172.21.15.28:0/2435884493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa188106dc0 0x7fa18819e6a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa18000b7b0 tx=0x7fa18000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.176+0000 7fa1867fc700 1 -- 172.21.15.28:0/2435884493 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa180020020 con 0x7fa188106dc0 2024-04-21T08:41:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.177+0000 7fa1867fc700 1 -- 172.21.15.28:0/2435884493 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa180003c60 con 0x7fa188106dc0 2024-04-21T08:41:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.177+0000 7fa1867fc700 1 -- 172.21.15.28:0/2435884493 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa180017040 con 0x7fa188106dc0 2024-04-21T08:41:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.177+0000 7fa18f52f700 1 -- 172.21.15.28:0/2435884493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa18819ee70 con 0x7fa188106dc0 2024-04-21T08:41:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.177+0000 7fa18f52f700 1 -- 172.21.15.28:0/2435884493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1881a1b60 con 0x7fa188106dc0 2024-04-21T08:41:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.177+0000 7fa1867fc700 1 -- 172.21.15.28:0/2435884493 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fa180023020 con 0x7fa188106dc0 2024-04-21T08:41:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.178+0000 7fa18f52f700 1 -- 172.21.15.28:0/2435884493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1880640a0 con 0x7fa188106dc0 2024-04-21T08:41:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.178+0000 7fa1867fc700 1 --2- 172.21.15.28:0/2435884493 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa170039700 0x7fa17003bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.178+0000 7fa1867fc700 1 -- 172.21.15.28:0/2435884493 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa18000f050 con 0x7fa188106dc0 2024-04-21T08:41:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.178+0000 7fa18caca700 1 --2- 172.21.15.28:0/2435884493 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa170039700 0x7fa17003bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.178+0000 7fa18caca700 1 --2- 172.21.15.28:0/2435884493 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa170039700 0x7fa17003bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa17c006fd0 tx=0x7fa17c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.183+0000 7fa1867fc700 1 -- 172.21.15.28:0/2435884493 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa18002a020 con 0x7fa188106dc0 2024-04-21T08:41:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.511+0000 7fa18f52f700 1 -- 172.21.15.28:0/2435884493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fa18802e1d0 con 0x7fa188106dc0 2024-04-21T08:41:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.513+0000 7fa1867fc700 1 -- 172.21.15.28:0/2435884493 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7fa1800124e0 con 0x7fa188106dc0 2024-04-21T08:41:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.515+0000 7fa18f52f700 1 -- 172.21.15.28:0/2435884493 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa170039700 msgr2=0x7fa17003bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.515+0000 7fa18f52f700 1 --2- 172.21.15.28:0/2435884493 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa170039700 0x7fa17003bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa17c006fd0 tx=0x7fa17c006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.515+0000 7fa18f52f700 1 -- 172.21.15.28:0/2435884493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa188106dc0 msgr2=0x7fa18819e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.515+0000 7fa18f52f700 1 --2- 172.21.15.28:0/2435884493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa188106dc0 0x7fa18819e6a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa18000b7b0 tx=0x7fa18000bd50 comp rx=0 tx=0).stop 2024-04-21T08:41:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.515+0000 7fa18f52f700 1 -- 172.21.15.28:0/2435884493 shutdown_connections 2024-04-21T08:41:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.515+0000 7fa18f52f700 1 --2- 172.21.15.28:0/2435884493 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa170039700 0x7fa17003bbc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.515+0000 7fa18f52f700 1 --2- 172.21.15.28:0/2435884493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa188106dc0 0x7fa18819e6a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.515+0000 7fa18f52f700 1 -- 172.21.15.28:0/2435884493 >> 172.21.15.28:0/2435884493 conn(0x7fa188102a00 msgr2=0x7fa1881952c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.515+0000 7fa18f52f700 1 -- 172.21.15.28:0/2435884493 shutdown_connections 2024-04-21T08:41:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-21T08:41:44.515+0000 7fa18f52f700 1 -- 172.21.15.28:0/2435884493 wait complete. 2024-04-21T08:41:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-21T08:41:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-21T08:41:45.084 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:41:45.084 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:41:45.084 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:41:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:Or, if you are only running a single cluster on this host: 2024-04-21T08:41:45.084 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:41:45.085 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-21T08:41:45.085 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:41:45.085 INFO:teuthology.orchestra.run.smithi028.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-21T08:41:45.085 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:41:45.085 INFO:teuthology.orchestra.run.smithi028.stdout: ceph telemetry on 2024-04-21T08:41:45.085 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:41:45.085 INFO:teuthology.orchestra.run.smithi028.stdout:For more information see: 2024-04-21T08:41:45.085 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:41:45.085 INFO:teuthology.orchestra.run.smithi028.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-21T08:41:45.085 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:41:45.085 INFO:teuthology.orchestra.run.smithi028.stdout:Bootstrap complete. 2024-04-21T08:41:45.118 INFO:tasks.cephadm:Fetching config... 2024-04-21T08:41:45.118 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:41:45.118 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-21T08:41:45.134 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-21T08:41:45.135 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:41:45.135 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-21T08:41:45.191 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-21T08:41:45.191 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:41:45.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-21T08:41:45.256 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-21T08:41:45.256 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:41:45.256 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-21T08:41:45.311 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-21T08:41:45.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCit2a8zvWLEusP1o7015j8V7/b1pTPBcLqRY3GjJRV1jImJcxBZKxpsWwnuNKfCCAitL6AysyTdF99aRfKoZ8DoWqUNVIEkNVlTi422dacXat3lLnkCocP6sJiO1gMgG/tslraPQVsz1KaAQDGMCwjOWUgm7UG/lM4rhlYMpPBLylLe1QsLJ+hciJD2oYuJHxjhYys5vxE80F8mqgEBk4dIO+nvk8ViOBNEzqP3i/cYKAfr9Mx0L7vpkr+gDy4zb2tNYK7Me8rMzVTUkoaBNbys0b2EjlsKd1wJtSNexQbwXzYlzhVnQO/ITPJsAScFpH8T9q2APp3rM7iAIcRybjVafLiYHr+dn8tAprFRAj3G/k0+mEhD2VK9kS5/82Qudf5YiQlpduSrMTIuRD1XtskfxIUCMWeqt2MCssz2WBU0VnrMqkjkpzEcTl6qmz6GCGi4h+c6ZsCu12wtdM1JKBweUv4Q3rQg+f3jVFJEkY5ABY1iEw+r3jeUElPpSun4Lk= ceph-b51a76c8-ffba-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T08:41:45.388 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCit2a8zvWLEusP1o7015j8V7/b1pTPBcLqRY3GjJRV1jImJcxBZKxpsWwnuNKfCCAitL6AysyTdF99aRfKoZ8DoWqUNVIEkNVlTi422dacXat3lLnkCocP6sJiO1gMgG/tslraPQVsz1KaAQDGMCwjOWUgm7UG/lM4rhlYMpPBLylLe1QsLJ+hciJD2oYuJHxjhYys5vxE80F8mqgEBk4dIO+nvk8ViOBNEzqP3i/cYKAfr9Mx0L7vpkr+gDy4zb2tNYK7Me8rMzVTUkoaBNbys0b2EjlsKd1wJtSNexQbwXzYlzhVnQO/ITPJsAScFpH8T9q2APp3rM7iAIcRybjVafLiYHr+dn8tAprFRAj3G/k0+mEhD2VK9kS5/82Qudf5YiQlpduSrMTIuRD1XtskfxIUCMWeqt2MCssz2WBU0VnrMqkjkpzEcTl6qmz6GCGi4h+c6ZsCu12wtdM1JKBweUv4Q3rQg+f3jVFJEkY5ABY1iEw+r3jeUElPpSun4Lk= ceph-b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:41:45.399 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCit2a8zvWLEusP1o7015j8V7/b1pTPBcLqRY3GjJRV1jImJcxBZKxpsWwnuNKfCCAitL6AysyTdF99aRfKoZ8DoWqUNVIEkNVlTi422dacXat3lLnkCocP6sJiO1gMgG/tslraPQVsz1KaAQDGMCwjOWUgm7UG/lM4rhlYMpPBLylLe1QsLJ+hciJD2oYuJHxjhYys5vxE80F8mqgEBk4dIO+nvk8ViOBNEzqP3i/cYKAfr9Mx0L7vpkr+gDy4zb2tNYK7Me8rMzVTUkoaBNbys0b2EjlsKd1wJtSNexQbwXzYlzhVnQO/ITPJsAScFpH8T9q2APp3rM7iAIcRybjVafLiYHr+dn8tAprFRAj3G/k0+mEhD2VK9kS5/82Qudf5YiQlpduSrMTIuRD1XtskfxIUCMWeqt2MCssz2WBU0VnrMqkjkpzEcTl6qmz6GCGi4h+c6ZsCu12wtdM1JKBweUv4Q3rQg+f3jVFJEkY5ABY1iEw+r3jeUElPpSun4Lk= ceph-b51a76c8-ffba-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T08:41:45.438 INFO:teuthology.orchestra.run.smithi152.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCit2a8zvWLEusP1o7015j8V7/b1pTPBcLqRY3GjJRV1jImJcxBZKxpsWwnuNKfCCAitL6AysyTdF99aRfKoZ8DoWqUNVIEkNVlTi422dacXat3lLnkCocP6sJiO1gMgG/tslraPQVsz1KaAQDGMCwjOWUgm7UG/lM4rhlYMpPBLylLe1QsLJ+hciJD2oYuJHxjhYys5vxE80F8mqgEBk4dIO+nvk8ViOBNEzqP3i/cYKAfr9Mx0L7vpkr+gDy4zb2tNYK7Me8rMzVTUkoaBNbys0b2EjlsKd1wJtSNexQbwXzYlzhVnQO/ITPJsAScFpH8T9q2APp3rM7iAIcRybjVafLiYHr+dn8tAprFRAj3G/k0+mEhD2VK9kS5/82Qudf5YiQlpduSrMTIuRD1XtskfxIUCMWeqt2MCssz2WBU0VnrMqkjkpzEcTl6qmz6GCGi4h+c6ZsCu12wtdM1JKBweUv4Q3rQg+f3jVFJEkY5ABY1iEw+r3jeUElPpSun4Lk= ceph-b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:41:45.450 DEBUG:teuthology.orchestra.run.smithi171:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCit2a8zvWLEusP1o7015j8V7/b1pTPBcLqRY3GjJRV1jImJcxBZKxpsWwnuNKfCCAitL6AysyTdF99aRfKoZ8DoWqUNVIEkNVlTi422dacXat3lLnkCocP6sJiO1gMgG/tslraPQVsz1KaAQDGMCwjOWUgm7UG/lM4rhlYMpPBLylLe1QsLJ+hciJD2oYuJHxjhYys5vxE80F8mqgEBk4dIO+nvk8ViOBNEzqP3i/cYKAfr9Mx0L7vpkr+gDy4zb2tNYK7Me8rMzVTUkoaBNbys0b2EjlsKd1wJtSNexQbwXzYlzhVnQO/ITPJsAScFpH8T9q2APp3rM7iAIcRybjVafLiYHr+dn8tAprFRAj3G/k0+mEhD2VK9kS5/82Qudf5YiQlpduSrMTIuRD1XtskfxIUCMWeqt2MCssz2WBU0VnrMqkjkpzEcTl6qmz6GCGi4h+c6ZsCu12wtdM1JKBweUv4Q3rQg+f3jVFJEkY5ABY1iEw+r3jeUElPpSun4Lk= ceph-b51a76c8-ffba-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T08:41:45.489 INFO:teuthology.orchestra.run.smithi171.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCit2a8zvWLEusP1o7015j8V7/b1pTPBcLqRY3GjJRV1jImJcxBZKxpsWwnuNKfCCAitL6AysyTdF99aRfKoZ8DoWqUNVIEkNVlTi422dacXat3lLnkCocP6sJiO1gMgG/tslraPQVsz1KaAQDGMCwjOWUgm7UG/lM4rhlYMpPBLylLe1QsLJ+hciJD2oYuJHxjhYys5vxE80F8mqgEBk4dIO+nvk8ViOBNEzqP3i/cYKAfr9Mx0L7vpkr+gDy4zb2tNYK7Me8rMzVTUkoaBNbys0b2EjlsKd1wJtSNexQbwXzYlzhVnQO/ITPJsAScFpH8T9q2APp3rM7iAIcRybjVafLiYHr+dn8tAprFRAj3G/k0+mEhD2VK9kS5/82Qudf5YiQlpduSrMTIuRD1XtskfxIUCMWeqt2MCssz2WBU0VnrMqkjkpzEcTl6qmz6GCGi4h+c6ZsCu12wtdM1JKBweUv4Q3rQg+f3jVFJEkY5ABY1iEw+r3jeUElPpSun4Lk= ceph-b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:41:45.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-21T08:41:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:41:45.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:45 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2435884493' entity='client.admin' 2024-04-21T08:41:48.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:41:48.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:41:48.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:48.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:41:48.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:48.165 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:41:48.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.273+0000 7facae594700 1 -- 172.21.15.28:0/4246889558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca8102f30 msgr2=0x7faca8105320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:48.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.273+0000 7facae594700 1 --2- 172.21.15.28:0/4246889558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca8102f30 0x7faca8105320 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fac98009910 tx=0x7fac98009c20 comp rx=0 tx=0).stop 2024-04-21T08:41:48.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.273+0000 7facae594700 1 -- 172.21.15.28:0/4246889558 shutdown_connections 2024-04-21T08:41:48.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.273+0000 7facae594700 1 --2- 172.21.15.28:0/4246889558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca8102f30 0x7faca8105320 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:48.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.273+0000 7facae594700 1 -- 172.21.15.28:0/4246889558 >> 172.21.15.28:0/4246889558 conn(0x7faca80fc940 msgr2=0x7faca80fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:48.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.273+0000 7facae594700 1 -- 172.21.15.28:0/4246889558 shutdown_connections 2024-04-21T08:41:48.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.273+0000 7facae594700 1 -- 172.21.15.28:0/4246889558 wait complete. 2024-04-21T08:41:48.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.273+0000 7facae594700 1 Processor -- start 2024-04-21T08:41:48.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.273+0000 7facae594700 1 -- start start 2024-04-21T08:41:48.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.274+0000 7facae594700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca8102f30 0x7faca8197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:48.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.274+0000 7facae594700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faca8108380 con 0x7faca8102f30 2024-04-21T08:41:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.274+0000 7faca7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca8102f30 0x7faca8197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.274+0000 7faca7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca8102f30 0x7faca8197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44172/0 (socket says 172.21.15.28:44172) 2024-04-21T08:41:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.274+0000 7faca7fff700 1 -- 172.21.15.28:0/1143527078 learned_addr learned my addr 172.21.15.28:0/1143527078 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.274+0000 7faca7fff700 1 -- 172.21.15.28:0/1143527078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faca8197f70 con 0x7faca8102f30 2024-04-21T08:41:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.274+0000 7faca7fff700 1 --2- 172.21.15.28:0/1143527078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca8102f30 0x7faca8197a30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fac98005b40 tx=0x7fac98004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.275+0000 7faca57fa700 1 -- 172.21.15.28:0/1143527078 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac98027070 con 0x7faca8102f30 2024-04-21T08:41:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.275+0000 7faca57fa700 1 -- 172.21.15.28:0/1143527078 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac9800be10 con 0x7faca8102f30 2024-04-21T08:41:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.275+0000 7facae594700 1 -- 172.21.15.28:0/1143527078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faca8198260 con 0x7faca8102f30 2024-04-21T08:41:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.275+0000 7faca57fa700 1 -- 172.21.15.28:0/1143527078 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac98022de0 con 0x7faca8102f30 2024-04-21T08:41:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.275+0000 7facae594700 1 -- 172.21.15.28:0/1143527078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faca819aef0 con 0x7faca8102f30 2024-04-21T08:41:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.276+0000 7faca57fa700 1 -- 172.21.15.28:0/1143527078 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fac9802a070 con 0x7faca8102f30 2024-04-21T08:41:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.276+0000 7faca57fa700 1 --2- 172.21.15.28:0/1143527078 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fac90039650 0x7fac9003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.276+0000 7facae594700 1 -- 172.21.15.28:0/1143527078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faca8108b00 con 0x7faca8102f30 2024-04-21T08:41:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.276+0000 7faca57fa700 1 -- 172.21.15.28:0/1143527078 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fac9801e030 con 0x7faca8102f30 2024-04-21T08:41:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.276+0000 7faca77fe700 1 --2- 172.21.15.28:0/1143527078 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fac90039650 0x7fac9003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.276+0000 7faca77fe700 1 --2- 172.21.15.28:0/1143527078 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fac90039650 0x7fac9003bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fac9c006fd0 tx=0x7fac9c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.284+0000 7faca57fa700 1 -- 172.21.15.28:0/1143527078 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac98013070 con 0x7faca8102f30 2024-04-21T08:41:48.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.493+0000 7facae594700 1 -- 172.21.15.28:0/1143527078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7faca802e0d0 con 0x7faca8102f30 2024-04-21T08:41:48.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.495+0000 7faca57fa700 1 -- 172.21.15.28:0/1143527078 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fac9802a360 con 0x7faca8102f30 2024-04-21T08:41:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.499+0000 7facae594700 1 -- 172.21.15.28:0/1143527078 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fac90039650 msgr2=0x7fac9003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.499+0000 7facae594700 1 --2- 172.21.15.28:0/1143527078 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fac90039650 0x7fac9003bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fac9c006fd0 tx=0x7fac9c006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.499+0000 7facae594700 1 -- 172.21.15.28:0/1143527078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca8102f30 msgr2=0x7faca8197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.499+0000 7facae594700 1 --2- 172.21.15.28:0/1143527078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca8102f30 0x7faca8197a30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fac98005b40 tx=0x7fac98004f80 comp rx=0 tx=0).stop 2024-04-21T08:41:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.499+0000 7facae594700 1 -- 172.21.15.28:0/1143527078 shutdown_connections 2024-04-21T08:41:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.499+0000 7facae594700 1 --2- 172.21.15.28:0/1143527078 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fac90039650 0x7fac9003bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.499+0000 7facae594700 1 --2- 172.21.15.28:0/1143527078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca8102f30 0x7faca8197a30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.499+0000 7facae594700 1 -- 172.21.15.28:0/1143527078 >> 172.21.15.28:0/1143527078 conn(0x7faca80fc940 msgr2=0x7faca80fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.499+0000 7facae594700 1 -- 172.21.15.28:0/1143527078 shutdown_connections 2024-04-21T08:41:48.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:48.499+0000 7facae594700 1 -- 172.21.15.28:0/1143527078 wait complete. 2024-04-21T08:41:49.276 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-21T08:41:49.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-21T08:41:49.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:49 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1143527078' entity='client.admin' 2024-04-21T08:41:49.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:49 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:49.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:49 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:41:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:41:51.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.325+0000 7f310396e700 1 -- 172.21.15.28:0/123505089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30fc1024e0 msgr2=0x7f30fc102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:51.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.325+0000 7f310396e700 1 --2- 172.21.15.28:0/123505089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30fc1024e0 0x7f30fc102900 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f30f8009910 tx=0x7f30f8009c20 comp rx=0 tx=0).stop 2024-04-21T08:41:51.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.325+0000 7f310396e700 1 -- 172.21.15.28:0/123505089 shutdown_connections 2024-04-21T08:41:51.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.325+0000 7f310396e700 1 --2- 172.21.15.28:0/123505089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30fc1024e0 0x7f30fc102900 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.325+0000 7f310396e700 1 -- 172.21.15.28:0/123505089 >> 172.21.15.28:0/123505089 conn(0x7f30fc0fda40 msgr2=0x7f30fc0ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.325+0000 7f310396e700 1 -- 172.21.15.28:0/123505089 shutdown_connections 2024-04-21T08:41:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.325+0000 7f310396e700 1 -- 172.21.15.28:0/123505089 wait complete. 2024-04-21T08:41:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.325+0000 7f310396e700 1 Processor -- start 2024-04-21T08:41:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.325+0000 7f310396e700 1 -- start start 2024-04-21T08:41:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.325+0000 7f310396e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30fc1024e0 0x7f30fc197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.325+0000 7f310396e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc108420 con 0x7f30fc1024e0 2024-04-21T08:41:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.326+0000 7f310170a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30fc1024e0 0x7f30fc197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.326+0000 7f310170a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30fc1024e0 0x7f30fc197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44198/0 (socket says 172.21.15.28:44198) 2024-04-21T08:41:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.326+0000 7f310170a700 1 -- 172.21.15.28:0/4269405261 learned_addr learned my addr 172.21.15.28:0/4269405261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.326+0000 7f310170a700 1 -- 172.21.15.28:0/4269405261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30fc198070 con 0x7f30fc1024e0 2024-04-21T08:41:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.326+0000 7f310170a700 1 --2- 172.21.15.28:0/4269405261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30fc1024e0 0x7f30fc197b30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f30f8005800 tx=0x7f30f8009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:51.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.326+0000 7f30f27fc700 1 -- 172.21.15.28:0/4269405261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30f8027070 con 0x7f30fc1024e0 2024-04-21T08:41:51.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.326+0000 7f310396e700 1 -- 172.21.15.28:0/4269405261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30fc198300 con 0x7f30fc1024e0 2024-04-21T08:41:51.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.326+0000 7f30f27fc700 1 -- 172.21.15.28:0/4269405261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30f800be10 con 0x7f30fc1024e0 2024-04-21T08:41:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.326+0000 7f30f27fc700 1 -- 172.21.15.28:0/4269405261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30f800f460 con 0x7f30fc1024e0 2024-04-21T08:41:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.326+0000 7f310396e700 1 -- 172.21.15.28:0/4269405261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30fc19afa0 con 0x7f30fc1024e0 2024-04-21T08:41:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.327+0000 7f30f27fc700 1 -- 172.21.15.28:0/4269405261 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f30f802a070 con 0x7f30fc1024e0 2024-04-21T08:41:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.327+0000 7f30f27fc700 1 --2- 172.21.15.28:0/4269405261 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f30e8039650 0x7f30e803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.327+0000 7f310396e700 1 -- 172.21.15.28:0/4269405261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30fc108ba0 con 0x7f30fc1024e0 2024-04-21T08:41:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.327+0000 7f30f27fc700 1 -- 172.21.15.28:0/4269405261 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f30f801e030 con 0x7f30fc1024e0 2024-04-21T08:41:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.327+0000 7f3100f09700 1 --2- 172.21.15.28:0/4269405261 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f30e8039650 0x7f30e803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.328+0000 7f3100f09700 1 --2- 172.21.15.28:0/4269405261 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f30e8039650 0x7f30e803bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f30ec006fd0 tx=0x7f30ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:51.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.333+0000 7f30f27fc700 1 -- 172.21.15.28:0/4269405261 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f30f804de20 con 0x7f30fc1024e0 2024-04-21T08:41:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.545+0000 7f310396e700 1 -- 172.21.15.28:0/4269405261 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f30fc02e080 con 0x7f30e8039650 2024-04-21T08:41:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.548+0000 7f30f27fc700 1 -- 172.21.15.28:0/4269405261 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f30fc02e080 con 0x7f30e8039650 2024-04-21T08:41:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.550+0000 7f310396e700 1 -- 172.21.15.28:0/4269405261 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f30e8039650 msgr2=0x7f30e803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.550+0000 7f310396e700 1 --2- 172.21.15.28:0/4269405261 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f30e8039650 0x7f30e803bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f30ec006fd0 tx=0x7f30ec006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.550+0000 7f310396e700 1 -- 172.21.15.28:0/4269405261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30fc1024e0 msgr2=0x7f30fc197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.550+0000 7f310396e700 1 --2- 172.21.15.28:0/4269405261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30fc1024e0 0x7f30fc197b30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f30f8005800 tx=0x7f30f8009f60 comp rx=0 tx=0).stop 2024-04-21T08:41:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.550+0000 7f310396e700 1 -- 172.21.15.28:0/4269405261 shutdown_connections 2024-04-21T08:41:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.550+0000 7f310396e700 1 --2- 172.21.15.28:0/4269405261 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f30e8039650 0x7f30e803bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.550+0000 7f310396e700 1 --2- 172.21.15.28:0/4269405261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30fc1024e0 0x7f30fc197b30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.550+0000 7f310396e700 1 -- 172.21.15.28:0/4269405261 >> 172.21.15.28:0/4269405261 conn(0x7f30fc0fda40 msgr2=0x7f30fc0fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.551+0000 7f310396e700 1 -- 172.21.15.28:0/4269405261 shutdown_connections 2024-04-21T08:41:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:51.551+0000 7f310396e700 1 -- 172.21.15.28:0/4269405261 wait complete. 2024-04-21T08:41:52.297 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi152 2024-04-21T08:41:52.297 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-21T08:41:52.297 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.conf 2024-04-21T08:41:52.317 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-21T08:41:52.317 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:41:52.374 INFO:tasks.cephadm:Adding host smithi152 to orchestrator... 2024-04-21T08:41:52.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch host add smithi152 2024-04-21T08:41:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:41:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:52 smithi028 ceph-mon[28003]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:41:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:52 smithi028 ceph-mon[28003]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-21T08:41:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:52 smithi028 ceph-mon[28003]: Updating smithi028:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:41:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:52 smithi028 ceph-mon[28003]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:41:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:52 smithi028 ceph-mon[28003]: Updating smithi028:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:41:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:41:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:41:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:41:54.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.061+0000 7fefc7531700 1 -- 172.21.15.28:0/1050426165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0103650 msgr2=0x7fefc0103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:54.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.061+0000 7fefc7531700 1 --2- 172.21.15.28:0/1050426165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0103650 0x7fefc0103a30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fefbc009910 tx=0x7fefbc009c20 comp rx=0 tx=0).stop 2024-04-21T08:41:54.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.061+0000 7fefc7531700 1 -- 172.21.15.28:0/1050426165 shutdown_connections 2024-04-21T08:41:54.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.061+0000 7fefc7531700 1 --2- 172.21.15.28:0/1050426165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0103650 0x7fefc0103a30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:54.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.061+0000 7fefc7531700 1 -- 172.21.15.28:0/1050426165 >> 172.21.15.28:0/1050426165 conn(0x7fefc00fcea0 msgr2=0x7fefc00ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:54.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.062+0000 7fefc7531700 1 -- 172.21.15.28:0/1050426165 shutdown_connections 2024-04-21T08:41:54.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.062+0000 7fefc7531700 1 -- 172.21.15.28:0/1050426165 wait complete. 2024-04-21T08:41:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.062+0000 7fefc7531700 1 Processor -- start 2024-04-21T08:41:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.062+0000 7fefc7531700 1 -- start start 2024-04-21T08:41:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.062+0000 7fefc7531700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0103650 0x7fefc0197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.062+0000 7fefc7531700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fefc010c9d0 con 0x7fefc0103650 2024-04-21T08:41:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.063+0000 7fefc52cd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0103650 0x7fefc0197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.063+0000 7fefc52cd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0103650 0x7fefc0197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42134/0 (socket says 172.21.15.28:42134) 2024-04-21T08:41:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.063+0000 7fefc52cd700 1 -- 172.21.15.28:0/3296944872 learned_addr learned my addr 172.21.15.28:0/3296944872 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:41:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.063+0000 7fefc52cd700 1 -- 172.21.15.28:0/3296944872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc01984b0 con 0x7fefc0103650 2024-04-21T08:41:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.063+0000 7fefc52cd700 1 --2- 172.21.15.28:0/3296944872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0103650 0x7fefc0197f70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fefbc0056d0 tx=0x7fefbc0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.063+0000 7fefb67fc700 1 -- 172.21.15.28:0/3296944872 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fefbc026070 con 0x7fefc0103650 2024-04-21T08:41:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.063+0000 7fefb67fc700 1 -- 172.21.15.28:0/3296944872 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefbc004370 con 0x7fefc0103650 2024-04-21T08:41:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.063+0000 7fefc7531700 1 -- 172.21.15.28:0/3296944872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefc0198740 con 0x7fefc0103650 2024-04-21T08:41:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.063+0000 7fefb67fc700 1 -- 172.21.15.28:0/3296944872 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fefbc0039b0 con 0x7fefc0103650 2024-04-21T08:41:54.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.063+0000 7fefc7531700 1 -- 172.21.15.28:0/3296944872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefc0198ba0 con 0x7fefc0103650 2024-04-21T08:41:54.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.064+0000 7fefb67fc700 1 -- 172.21.15.28:0/3296944872 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fefbc029070 con 0x7fefc0103650 2024-04-21T08:41:54.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.064+0000 7fefc7531700 1 -- 172.21.15.28:0/3296944872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefc006b190 con 0x7fefc0103650 2024-04-21T08:41:54.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.064+0000 7fefb67fc700 1 --2- 172.21.15.28:0/3296944872 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fefac039650 0x7fefac03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:54.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.065+0000 7fefb67fc700 1 -- 172.21.15.28:0/3296944872 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fefbc01e070 con 0x7fefc0103650 2024-04-21T08:41:54.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.065+0000 7fefc4acc700 1 --2- 172.21.15.28:0/3296944872 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fefac039650 0x7fefac03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:54.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.065+0000 7fefc4acc700 1 --2- 172.21.15.28:0/3296944872 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fefac039650 0x7fefac03bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fefb0006fd0 tx=0x7fefb0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:54.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.070+0000 7fefb67fc700 1 -- 172.21.15.28:0/3296944872 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fefbc013070 con 0x7fefc0103650 2024-04-21T08:41:54.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:54.284+0000 7fefc7531700 1 -- 172.21.15.28:0/3296944872 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}) -- 0x7fefc0199340 con 0x7fefac039650 2024-04-21T08:41:54.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:54 smithi028 ceph-mon[28003]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:55.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:55 smithi028 ceph-mon[28003]: Deploying cephadm binary to smithi152 2024-04-21T08:41:56.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:56 smithi028 ceph-mon[28003]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:41:58.551 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi152' with addr '172.21.15.152' 2024-04-21T08:41:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:58.551+0000 7fefb67fc700 1 -- 172.21.15.28:0/3296944872 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fefc0199340 con 0x7fefac039650 2024-04-21T08:41:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:58.554+0000 7fefc7531700 1 -- 172.21.15.28:0/3296944872 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fefac039650 msgr2=0x7fefac03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:58.554+0000 7fefc7531700 1 --2- 172.21.15.28:0/3296944872 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fefac039650 0x7fefac03bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fefb0006fd0 tx=0x7fefb0006e40 comp rx=0 tx=0).stop 2024-04-21T08:41:58.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:58.554+0000 7fefc7531700 1 -- 172.21.15.28:0/3296944872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0103650 msgr2=0x7fefc0197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:58.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:58.554+0000 7fefc7531700 1 --2- 172.21.15.28:0/3296944872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0103650 0x7fefc0197f70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fefbc0056d0 tx=0x7fefbc0057b0 comp rx=0 tx=0).stop 2024-04-21T08:41:58.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:58.554+0000 7fefc7531700 1 -- 172.21.15.28:0/3296944872 shutdown_connections 2024-04-21T08:41:58.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:58.554+0000 7fefc7531700 1 --2- 172.21.15.28:0/3296944872 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fefac039650 0x7fefac03bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:58.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:58.554+0000 7fefc7531700 1 --2- 172.21.15.28:0/3296944872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0103650 0x7fefc0197f70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:58.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:58.554+0000 7fefc7531700 1 -- 172.21.15.28:0/3296944872 >> 172.21.15.28:0/3296944872 conn(0x7fefc00fcea0 msgr2=0x7fefc01043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:58.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:58.554+0000 7fefc7531700 1 -- 172.21.15.28:0/3296944872 shutdown_connections 2024-04-21T08:41:58.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:41:58.554+0000 7fefc7531700 1 -- 172.21.15.28:0/3296944872 wait complete. 2024-04-21T08:41:58.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:58 smithi028 ceph-mon[28003]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:41:59.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-21T08:41:59.437 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:41:59.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:59 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:41:59.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:59 smithi028 ceph-mon[28003]: Added host smithi152 2024-04-21T08:41:59.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:59 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:59.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:41:59 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:00.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.920+0000 7f4f63354700 1 -- 172.21.15.28:0/535355536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c103650 msgr2=0x7f4f5c103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:00.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.920+0000 7f4f63354700 1 --2- 172.21.15.28:0/535355536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c103650 0x7f4f5c103a30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c009910 tx=0x7f4f4c009c20 comp rx=0 tx=0).stop 2024-04-21T08:42:00.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.920+0000 7f4f63354700 1 -- 172.21.15.28:0/535355536 shutdown_connections 2024-04-21T08:42:00.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.920+0000 7f4f63354700 1 --2- 172.21.15.28:0/535355536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c103650 0x7f4f5c103a30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:00.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.920+0000 7f4f63354700 1 -- 172.21.15.28:0/535355536 >> 172.21.15.28:0/535355536 conn(0x7f4f5c0fce40 msgr2=0x7f4f5c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:00.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.920+0000 7f4f63354700 1 -- 172.21.15.28:0/535355536 shutdown_connections 2024-04-21T08:42:00.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.920+0000 7f4f63354700 1 -- 172.21.15.28:0/535355536 wait complete. 2024-04-21T08:42:00.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:00 smithi028 ceph-mon[28003]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:00.921 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:00 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:00.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.921+0000 7f4f63354700 1 Processor -- start 2024-04-21T08:42:00.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.921+0000 7f4f63354700 1 -- start start 2024-04-21T08:42:00.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.921+0000 7f4f63354700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c103650 0x7f4f5c1012b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:00.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.921+0000 7f4f63354700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5c10c980 con 0x7f4f5c103650 2024-04-21T08:42:00.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.921+0000 7f4f610f0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c103650 0x7f4f5c1012b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:00.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.921+0000 7f4f610f0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c103650 0x7f4f5c1012b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42148/0 (socket says 172.21.15.28:42148) 2024-04-21T08:42:00.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.921+0000 7f4f610f0700 1 -- 172.21.15.28:0/3151919330 learned_addr learned my addr 172.21.15.28:0/3151919330 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:42:00.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.922+0000 7f4f610f0700 1 -- 172.21.15.28:0/3151919330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f5c1031a0 con 0x7f4f5c103650 2024-04-21T08:42:00.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.922+0000 7f4f610f0700 1 --2- 172.21.15.28:0/3151919330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c103650 0x7f4f5c1012b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c006010 tx=0x7f4f4c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:00.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.922+0000 7f4f527fc700 1 -- 172.21.15.28:0/3151919330 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f4c026070 con 0x7f4f5c103650 2024-04-21T08:42:00.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.922+0000 7f4f527fc700 1 -- 172.21.15.28:0/3151919330 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f4c00bc00 con 0x7f4f5c103650 2024-04-21T08:42:00.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.922+0000 7f4f527fc700 1 -- 172.21.15.28:0/3151919330 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f4c003910 con 0x7f4f5c103650 2024-04-21T08:42:00.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.922+0000 7f4f63354700 1 -- 172.21.15.28:0/3151919330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f5c1017f0 con 0x7f4f5c103650 2024-04-21T08:42:00.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.922+0000 7f4f63354700 1 -- 172.21.15.28:0/3151919330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f5c101cb0 con 0x7f4f5c103650 2024-04-21T08:42:00.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.923+0000 7f4f527fc700 1 -- 172.21.15.28:0/3151919330 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4f4c029070 con 0x7f4f5c103650 2024-04-21T08:42:00.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.923+0000 7f4f63354700 1 -- 172.21.15.28:0/3151919330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f5c06af70 con 0x7f4f5c103650 2024-04-21T08:42:00.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.923+0000 7f4f527fc700 1 --2- 172.21.15.28:0/3151919330 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f48039650 0x7f4f4803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:00.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.923+0000 7f4f527fc700 1 -- 172.21.15.28:0/3151919330 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4f4c01e070 con 0x7f4f5c103650 2024-04-21T08:42:00.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.923+0000 7f4f608ef700 1 --2- 172.21.15.28:0/3151919330 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f48039650 0x7f4f4803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:00.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.924+0000 7f4f608ef700 1 --2- 172.21.15.28:0/3151919330 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f48039650 0x7f4f4803bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4f58006fd0 tx=0x7f4f58006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:00.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:00.928+0000 7f4f527fc700 1 -- 172.21.15.28:0/3151919330 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f4c013070 con 0x7f4f5c103650 2024-04-21T08:42:01.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:01.144+0000 7f4f63354700 1 -- 172.21.15.28:0/3151919330 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4f5c1027e0 con 0x7f4f48039650 2024-04-21T08:42:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:01.146+0000 7f4f527fc700 1 -- 172.21.15.28:0/3151919330 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f4f5c1027e0 con 0x7f4f48039650 2024-04-21T08:42:01.145 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:42:01.145 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.152", "hostname": "smithi152", "labels": [], "status": ""}] 2024-04-21T08:42:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:01.148+0000 7f4f63354700 1 -- 172.21.15.28:0/3151919330 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f48039650 msgr2=0x7f4f4803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:01.148+0000 7f4f63354700 1 --2- 172.21.15.28:0/3151919330 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f48039650 0x7f4f4803bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4f58006fd0 tx=0x7f4f58006e40 comp rx=0 tx=0).stop 2024-04-21T08:42:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:01.148+0000 7f4f63354700 1 -- 172.21.15.28:0/3151919330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c103650 msgr2=0x7f4f5c1012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:01.148+0000 7f4f63354700 1 --2- 172.21.15.28:0/3151919330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c103650 0x7f4f5c1012b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c006010 tx=0x7f4f4c005780 comp rx=0 tx=0).stop 2024-04-21T08:42:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:01.148+0000 7f4f63354700 1 -- 172.21.15.28:0/3151919330 shutdown_connections 2024-04-21T08:42:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:01.148+0000 7f4f63354700 1 --2- 172.21.15.28:0/3151919330 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f48039650 0x7f4f4803bb10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:01.148+0000 7f4f63354700 1 --2- 172.21.15.28:0/3151919330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5c103650 0x7f4f5c1012b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:01.148+0000 7f4f63354700 1 -- 172.21.15.28:0/3151919330 >> 172.21.15.28:0/3151919330 conn(0x7f4f5c0fce40 msgr2=0x7f4f5c104400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:01.148+0000 7f4f63354700 1 -- 172.21.15.28:0/3151919330 shutdown_connections 2024-04-21T08:42:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:01.148+0000 7f4f63354700 1 -- 172.21.15.28:0/3151919330 wait complete. 2024-04-21T08:42:01.773 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi171 2024-04-21T08:42:01.773 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-21T08:42:01.773 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/etc/ceph/ceph.conf 2024-04-21T08:42:01.792 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-21T08:42:01.793 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:42:01.849 INFO:tasks.cephadm:Adding host smithi171 to orchestrator... 2024-04-21T08:42:01.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch host add smithi171 2024-04-21T08:42:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:02.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:02 smithi028 ceph-mon[28003]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:42:02.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:02 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:03.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:03 smithi028 ceph-mon[28003]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:03.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.510+0000 7f8ba2c3b700 1 -- 172.21.15.28:0/794387909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c1015f0 msgr2=0x7f8b9c1019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:03.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.510+0000 7f8ba2c3b700 1 --2- 172.21.15.28:0/794387909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c1015f0 0x7f8b9c1019d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b90009910 tx=0x7f8b90009c20 comp rx=0 tx=0).stop 2024-04-21T08:42:03.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.510+0000 7f8ba2c3b700 1 -- 172.21.15.28:0/794387909 shutdown_connections 2024-04-21T08:42:03.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.510+0000 7f8ba2c3b700 1 --2- 172.21.15.28:0/794387909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c1015f0 0x7f8b9c1019d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:03.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.510+0000 7f8ba2c3b700 1 -- 172.21.15.28:0/794387909 >> 172.21.15.28:0/794387909 conn(0x7f8b9c076ea0 msgr2=0x7f8b9c0772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.510+0000 7f8ba2c3b700 1 -- 172.21.15.28:0/794387909 shutdown_connections 2024-04-21T08:42:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.510+0000 7f8ba2c3b700 1 -- 172.21.15.28:0/794387909 wait complete. 2024-04-21T08:42:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.510+0000 7f8ba2c3b700 1 Processor -- start 2024-04-21T08:42:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.510+0000 7f8ba2c3b700 1 -- start start 2024-04-21T08:42:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.511+0000 7f8ba2c3b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c1015f0 0x7f8b9c103040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.511+0000 7f8ba2c3b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c10b820 con 0x7f8b9c1015f0 2024-04-21T08:42:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.511+0000 7f8ba09d7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c1015f0 0x7f8b9c103040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.511+0000 7f8ba09d7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c1015f0 0x7f8b9c103040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42160/0 (socket says 172.21.15.28:42160) 2024-04-21T08:42:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.511+0000 7f8ba09d7700 1 -- 172.21.15.28:0/2945768861 learned_addr learned my addr 172.21.15.28:0/2945768861 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:42:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.511+0000 7f8ba09d7700 1 -- 172.21.15.28:0/2945768861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b9c104f30 con 0x7f8b9c1015f0 2024-04-21T08:42:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.511+0000 7f8ba09d7700 1 --2- 172.21.15.28:0/2945768861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c1015f0 0x7f8b9c103040 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8b900056d0 tx=0x7f8b900057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:03.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.511+0000 7f8b99ffb700 1 -- 172.21.15.28:0/2945768861 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b90026070 con 0x7f8b9c1015f0 2024-04-21T08:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.511+0000 7f8ba2c3b700 1 -- 172.21.15.28:0/2945768861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b9c1035e0 con 0x7f8b9c1015f0 2024-04-21T08:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.511+0000 7f8b99ffb700 1 -- 172.21.15.28:0/2945768861 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b9000bc00 con 0x7f8b9c1015f0 2024-04-21T08:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.511+0000 7f8b99ffb700 1 -- 172.21.15.28:0/2945768861 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b90003920 con 0x7f8b9c1015f0 2024-04-21T08:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.512+0000 7f8ba2c3b700 1 -- 172.21.15.28:0/2945768861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b9c103aa0 con 0x7f8b9c1015f0 2024-04-21T08:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.512+0000 7f8b99ffb700 1 -- 172.21.15.28:0/2945768861 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8b90029070 con 0x7f8b9c1015f0 2024-04-21T08:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.512+0000 7f8ba2c3b700 1 -- 172.21.15.28:0/2945768861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b9c102a80 con 0x7f8b9c1015f0 2024-04-21T08:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.513+0000 7f8b99ffb700 1 --2- 172.21.15.28:0/2945768861 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8b8c0396b0 0x7f8b8c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.513+0000 7f8b99ffb700 1 -- 172.21.15.28:0/2945768861 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8b9001e070 con 0x7f8b9c1015f0 2024-04-21T08:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.513+0000 7f8b9bfff700 1 --2- 172.21.15.28:0/2945768861 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8b8c0396b0 0x7f8b8c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.513+0000 7f8b9bfff700 1 --2- 172.21.15.28:0/2945768861 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8b8c0396b0 0x7f8b8c03bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8b88006fd0 tx=0x7f8b88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:03.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.518+0000 7f8b99ffb700 1 -- 172.21.15.28:0/2945768861 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b90013070 con 0x7f8b9c1015f0 2024-04-21T08:42:03.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:03.731+0000 7f8ba2c3b700 1 -- 172.21.15.28:0/2945768861 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi171", "target": ["mon-mgr", ""]}) -- 0x7f8b9c104570 con 0x7f8b8c0396b0 2024-04-21T08:42:05.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:05 smithi028 ceph-mon[28003]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:05.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:05 smithi028 ceph-mon[28003]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi171", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:05.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:05.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:05.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:05.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:05.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:05.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:05.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:05.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:05 smithi028 ceph-mon[28003]: Updating smithi152:/etc/ceph/ceph.conf 2024-04-21T08:42:06.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:06 smithi028 ceph-mon[28003]: Deploying cephadm binary to smithi171 2024-04-21T08:42:06.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:06 smithi028 ceph-mon[28003]: Updating smithi152:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:42:06.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:06 smithi028 ceph-mon[28003]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:42:07.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:07 smithi028 ceph-mon[28003]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:08.086+0000 7f8b99ffb700 1 -- 172.21.15.28:0/2945768861 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f8b9c104570 con 0x7f8b8c0396b0 2024-04-21T08:42:08.087 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi171' with addr '172.21.15.171' 2024-04-21T08:42:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:08.089+0000 7f8ba2c3b700 1 -- 172.21.15.28:0/2945768861 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8b8c0396b0 msgr2=0x7f8b8c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:08.089+0000 7f8ba2c3b700 1 --2- 172.21.15.28:0/2945768861 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8b8c0396b0 0x7f8b8c03bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8b88006fd0 tx=0x7f8b88006e40 comp rx=0 tx=0).stop 2024-04-21T08:42:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:08.089+0000 7f8ba2c3b700 1 -- 172.21.15.28:0/2945768861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c1015f0 msgr2=0x7f8b9c103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:08.089+0000 7f8ba2c3b700 1 --2- 172.21.15.28:0/2945768861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c1015f0 0x7f8b9c103040 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8b900056d0 tx=0x7f8b900057b0 comp rx=0 tx=0).stop 2024-04-21T08:42:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:08.089+0000 7f8ba2c3b700 1 -- 172.21.15.28:0/2945768861 shutdown_connections 2024-04-21T08:42:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:08.089+0000 7f8ba2c3b700 1 --2- 172.21.15.28:0/2945768861 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8b8c0396b0 0x7f8b8c03bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:08.089+0000 7f8ba2c3b700 1 --2- 172.21.15.28:0/2945768861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c1015f0 0x7f8b9c103040 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:08.089+0000 7f8ba2c3b700 1 -- 172.21.15.28:0/2945768861 >> 172.21.15.28:0/2945768861 conn(0x7f8b9c076ea0 msgr2=0x7f8b9c069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:08.089+0000 7f8ba2c3b700 1 -- 172.21.15.28:0/2945768861 shutdown_connections 2024-04-21T08:42:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:08.089+0000 7f8ba2c3b700 1 -- 172.21.15.28:0/2945768861 wait complete. 2024-04-21T08:42:08.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-21T08:42:08.975 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:09.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:08 smithi028 ceph-mon[28003]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:09.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:08 smithi028 ceph-mon[28003]: Updating smithi152:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:42:09.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:08 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:09.088 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:08 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:09.088 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:08 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:09.088 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:08 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:09.088 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:08 smithi028 ceph-mon[28003]: Added host smithi171 2024-04-21T08:42:09.088 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:08 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:09.088 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:08 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:10.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:10 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.637+0000 7f60f1b72700 1 -- 172.21.15.28:0/620014481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ec101590 msgr2=0x7f60ec101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.637+0000 7f60f1b72700 1 --2- 172.21.15.28:0/620014481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ec101590 0x7f60ec101970 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f60e0009910 tx=0x7f60e0009c20 comp rx=0 tx=0).stop 2024-04-21T08:42:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.637+0000 7f60f1b72700 1 -- 172.21.15.28:0/620014481 shutdown_connections 2024-04-21T08:42:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.637+0000 7f60f1b72700 1 --2- 172.21.15.28:0/620014481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ec101590 0x7f60ec101970 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.637+0000 7f60f1b72700 1 -- 172.21.15.28:0/620014481 >> 172.21.15.28:0/620014481 conn(0x7f60ec0fce40 msgr2=0x7f60ec0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.637+0000 7f60f1b72700 1 -- 172.21.15.28:0/620014481 shutdown_connections 2024-04-21T08:42:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.637+0000 7f60f1b72700 1 -- 172.21.15.28:0/620014481 wait complete. 2024-04-21T08:42:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.637+0000 7f60f1b72700 1 Processor -- start 2024-04-21T08:42:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.638+0000 7f60f1b72700 1 -- start start 2024-04-21T08:42:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.638+0000 7f60f1b72700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ec101590 0x7f60ec103030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.638+0000 7f60f1b72700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec10b810 con 0x7f60ec101590 2024-04-21T08:42:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.638+0000 7f60eb7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ec101590 0x7f60ec103030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.638+0000 7f60eb7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ec101590 0x7f60ec103030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41684/0 (socket says 172.21.15.28:41684) 2024-04-21T08:42:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.638+0000 7f60eb7fe700 1 -- 172.21.15.28:0/1844091709 learned_addr learned my addr 172.21.15.28:0/1844091709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:42:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.638+0000 7f60eb7fe700 1 -- 172.21.15.28:0/1844091709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ec104f20 con 0x7f60ec101590 2024-04-21T08:42:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.638+0000 7f60eb7fe700 1 --2- 172.21.15.28:0/1844091709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ec101590 0x7f60ec103030 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f60e00056d0 tx=0x7f60e00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.638+0000 7f60e8ff9700 1 -- 172.21.15.28:0/1844091709 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60e0026070 con 0x7f60ec101590 2024-04-21T08:42:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.639+0000 7f60e8ff9700 1 -- 172.21.15.28:0/1844091709 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60e000bc00 con 0x7f60ec101590 2024-04-21T08:42:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.639+0000 7f60f1b72700 1 -- 172.21.15.28:0/1844091709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60ec1035d0 con 0x7f60ec101590 2024-04-21T08:42:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.639+0000 7f60e8ff9700 1 -- 172.21.15.28:0/1844091709 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60e0003920 con 0x7f60ec101590 2024-04-21T08:42:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.639+0000 7f60f1b72700 1 -- 172.21.15.28:0/1844091709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60ec103a90 con 0x7f60ec101590 2024-04-21T08:42:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.639+0000 7f60e8ff9700 1 -- 172.21.15.28:0/1844091709 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f60e0029070 con 0x7f60ec101590 2024-04-21T08:42:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.640+0000 7f60e8ff9700 1 --2- 172.21.15.28:0/1844091709 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f60d40396b0 0x7f60d403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.640+0000 7f60f1b72700 1 -- 172.21.15.28:0/1844091709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60ec102a70 con 0x7f60ec101590 2024-04-21T08:42:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.640+0000 7f60e8ff9700 1 -- 172.21.15.28:0/1844091709 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f60e001e070 con 0x7f60ec101590 2024-04-21T08:42:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.640+0000 7f60eaffd700 1 --2- 172.21.15.28:0/1844091709 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f60d40396b0 0x7f60d403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.640+0000 7f60eaffd700 1 --2- 172.21.15.28:0/1844091709 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f60d40396b0 0x7f60d403bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f60dc006fd0 tx=0x7f60dc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.645+0000 7f60e8ff9700 1 -- 172.21.15.28:0/1844091709 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f60e0013070 con 0x7f60ec101590 2024-04-21T08:42:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.861+0000 7f60f1b72700 1 -- 172.21.15.28:0/1844091709 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f60ec104560 con 0x7f60d40396b0 2024-04-21T08:42:10.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.862+0000 7f60e8ff9700 1 -- 172.21.15.28:0/1844091709 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f60ec104560 con 0x7f60d40396b0 2024-04-21T08:42:10.861 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:42:10.861 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.152", "hostname": "smithi152", "labels": [], "status": ""}, {"addr": "172.21.15.171", "hostname": "smithi171", "labels": [], "status": ""}] 2024-04-21T08:42:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.864+0000 7f60f1b72700 1 -- 172.21.15.28:0/1844091709 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f60d40396b0 msgr2=0x7f60d403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.864+0000 7f60f1b72700 1 --2- 172.21.15.28:0/1844091709 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f60d40396b0 0x7f60d403bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f60dc006fd0 tx=0x7f60dc006e40 comp rx=0 tx=0).stop 2024-04-21T08:42:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.864+0000 7f60f1b72700 1 -- 172.21.15.28:0/1844091709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ec101590 msgr2=0x7f60ec103030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.864+0000 7f60f1b72700 1 --2- 172.21.15.28:0/1844091709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ec101590 0x7f60ec103030 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f60e00056d0 tx=0x7f60e00057b0 comp rx=0 tx=0).stop 2024-04-21T08:42:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.864+0000 7f60f1b72700 1 -- 172.21.15.28:0/1844091709 shutdown_connections 2024-04-21T08:42:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.864+0000 7f60f1b72700 1 --2- 172.21.15.28:0/1844091709 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f60d40396b0 0x7f60d403bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.864+0000 7f60f1b72700 1 --2- 172.21.15.28:0/1844091709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ec101590 0x7f60ec103030 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.864+0000 7f60f1b72700 1 -- 172.21.15.28:0/1844091709 >> 172.21.15.28:0/1844091709 conn(0x7f60ec0fce40 msgr2=0x7f60ec06a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.864+0000 7f60f1b72700 1 -- 172.21.15.28:0/1844091709 shutdown_connections 2024-04-21T08:42:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:10.864+0000 7f60f1b72700 1 -- 172.21.15.28:0/1844091709 wait complete. 2024-04-21T08:42:11.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:11 smithi028 ceph-mon[28003]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:11.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:11 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:11.455 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-21T08:42:11.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-21T08:42:11.713 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:12.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:12 smithi028 ceph-mon[28003]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:42:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.325+0000 7f03e3c1a700 1 -- 172.21.15.28:0/1717319272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03dc1015f0 msgr2=0x7f03dc1019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.325+0000 7f03e3c1a700 1 --2- 172.21.15.28:0/1717319272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03dc1015f0 0x7f03dc1019d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f03d8009910 tx=0x7f03d8009c20 comp rx=0 tx=0).stop 2024-04-21T08:42:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.325+0000 7f03e3c1a700 1 -- 172.21.15.28:0/1717319272 shutdown_connections 2024-04-21T08:42:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.325+0000 7f03e3c1a700 1 --2- 172.21.15.28:0/1717319272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03dc1015f0 0x7f03dc1019d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.325+0000 7f03e3c1a700 1 -- 172.21.15.28:0/1717319272 >> 172.21.15.28:0/1717319272 conn(0x7f03dc0fcea0 msgr2=0x7f03dc0ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.325+0000 7f03e3c1a700 1 -- 172.21.15.28:0/1717319272 shutdown_connections 2024-04-21T08:42:13.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.325+0000 7f03e3c1a700 1 -- 172.21.15.28:0/1717319272 wait complete. 2024-04-21T08:42:13.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.326+0000 7f03e3c1a700 1 Processor -- start 2024-04-21T08:42:13.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.326+0000 7f03e3c1a700 1 -- start start 2024-04-21T08:42:13.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.326+0000 7f03e3c1a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03dc1015f0 0x7f03dc196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.326+0000 7f03e3c1a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03dc10b870 con 0x7f03dc1015f0 2024-04-21T08:42:13.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.326+0000 7f03e19b6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03dc1015f0 0x7f03dc196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.326+0000 7f03e19b6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03dc1015f0 0x7f03dc196de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41688/0 (socket says 172.21.15.28:41688) 2024-04-21T08:42:13.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.326+0000 7f03e19b6700 1 -- 172.21.15.28:0/2690422058 learned_addr learned my addr 172.21.15.28:0/2690422058 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:42:13.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.326+0000 7f03e19b6700 1 -- 172.21.15.28:0/2690422058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03dc197320 con 0x7f03dc1015f0 2024-04-21T08:42:13.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.327+0000 7f03e19b6700 1 --2- 172.21.15.28:0/2690422058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03dc1015f0 0x7f03dc196de0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f03d80056d0 tx=0x7f03d80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:13.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.327+0000 7f03d2ffd700 1 -- 172.21.15.28:0/2690422058 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03d8028070 con 0x7f03dc1015f0 2024-04-21T08:42:13.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.327+0000 7f03d2ffd700 1 -- 172.21.15.28:0/2690422058 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03d800bba0 con 0x7f03dc1015f0 2024-04-21T08:42:13.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.327+0000 7f03e3c1a700 1 -- 172.21.15.28:0/2690422058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03dc1975b0 con 0x7f03dc1015f0 2024-04-21T08:42:13.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.327+0000 7f03d2ffd700 1 -- 172.21.15.28:0/2690422058 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03d8003920 con 0x7f03dc1015f0 2024-04-21T08:42:13.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.327+0000 7f03e3c1a700 1 -- 172.21.15.28:0/2690422058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03dc19b6b0 con 0x7f03dc1015f0 2024-04-21T08:42:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.328+0000 7f03d2ffd700 1 -- 172.21.15.28:0/2690422058 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f03d802b070 con 0x7f03dc1015f0 2024-04-21T08:42:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.328+0000 7f03d2ffd700 1 --2- 172.21.15.28:0/2690422058 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f03c80396b0 0x7f03c803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.328+0000 7f03e3c1a700 1 -- 172.21.15.28:0/2690422058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03dc102ad0 con 0x7f03dc1015f0 2024-04-21T08:42:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.328+0000 7f03d2ffd700 1 -- 172.21.15.28:0/2690422058 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f03d8020070 con 0x7f03dc1015f0 2024-04-21T08:42:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.328+0000 7f03e11b5700 1 --2- 172.21.15.28:0/2690422058 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f03c80396b0 0x7f03c803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.328+0000 7f03e11b5700 1 --2- 172.21.15.28:0/2690422058 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f03c80396b0 0x7f03c803bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f03cc006fd0 tx=0x7f03cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.333+0000 7f03d2ffd700 1 -- 172.21.15.28:0/2690422058 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03d8032020 con 0x7f03dc1015f0 2024-04-21T08:42:13.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:13 smithi028 ceph-mon[28003]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:13.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:13.548+0000 7f03e3c1a700 1 -- 172.21.15.28:0/2690422058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f03dc198140 con 0x7f03dc1015f0 2024-04-21T08:42:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:14.173+0000 7f03d2ffd700 1 -- 172.21.15.28:0/2690422058 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f03d8013070 con 0x7f03dc1015f0 2024-04-21T08:42:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:14.175+0000 7f03e3c1a700 1 -- 172.21.15.28:0/2690422058 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f03c80396b0 msgr2=0x7f03c803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:14.175+0000 7f03e3c1a700 1 --2- 172.21.15.28:0/2690422058 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f03c80396b0 0x7f03c803bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f03cc006fd0 tx=0x7f03cc006e40 comp rx=0 tx=0).stop 2024-04-21T08:42:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:14.175+0000 7f03e3c1a700 1 -- 172.21.15.28:0/2690422058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03dc1015f0 msgr2=0x7f03dc196de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:14.175+0000 7f03e3c1a700 1 --2- 172.21.15.28:0/2690422058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03dc1015f0 0x7f03dc196de0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f03d80056d0 tx=0x7f03d80057b0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:14.175+0000 7f03e3c1a700 1 -- 172.21.15.28:0/2690422058 shutdown_connections 2024-04-21T08:42:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:14.175+0000 7f03e3c1a700 1 --2- 172.21.15.28:0/2690422058 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f03c80396b0 0x7f03c803bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:14.175+0000 7f03e3c1a700 1 --2- 172.21.15.28:0/2690422058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03dc1015f0 0x7f03dc196de0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:14.175+0000 7f03e3c1a700 1 -- 172.21.15.28:0/2690422058 >> 172.21.15.28:0/2690422058 conn(0x7f03dc0fcea0 msgr2=0x7f03dc102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:14.175+0000 7f03e3c1a700 1 -- 172.21.15.28:0/2690422058 shutdown_connections 2024-04-21T08:42:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:14.175+0000 7f03e3c1a700 1 -- 172.21.15.28:0/2690422058 wait complete. 2024-04-21T08:42:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-04-21T08:42:14.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:14 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2690422058' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-21T08:42:14.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:14.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:14.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:14.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:14.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:14.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:14.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:14.852 INFO:tasks.cephadm:Adding mon.a on smithi028 2024-04-21T08:42:14.853 INFO:tasks.cephadm:Adding mon.b on smithi152 2024-04-21T08:42:14.853 INFO:tasks.cephadm:Adding mon.c on smithi171 2024-04-21T08:42:14.853 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch apply mon '3;smithi028:172.21.15.28=a;smithi152:172.21.15.152=b;smithi171:172.21.15.171=c' 2024-04-21T08:42:15.089 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:42:15.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:15 smithi028 ceph-mon[28003]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:15.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:15 smithi028 ceph-mon[28003]: Updating smithi171:/etc/ceph/ceph.conf 2024-04-21T08:42:15.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:15 smithi028 ceph-mon[28003]: Updating smithi171:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:42:15.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:15 smithi028 ceph-mon[28003]: Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:42:15.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:15 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2690422058' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-21T08:42:15.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:15 smithi028 ceph-mon[28003]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:42:15.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:15 smithi028 ceph-mon[28003]: Updating smithi171:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:42:15.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:15 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:15.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:15 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:15.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:15 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:16.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.591+0000 7f3662fbb700 1 -- 172.21.15.171:0/1775528447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f365c1015b0 msgr2=0x7f365c101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.591+0000 7f3662fbb700 1 --2- 172.21.15.171:0/1775528447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f365c1015b0 0x7f365c101990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3650009910 tx=0x7f3650009c20 comp rx=0 tx=0).stop 2024-04-21T08:42:16.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.591+0000 7f3662fbb700 1 -- 172.21.15.171:0/1775528447 shutdown_connections 2024-04-21T08:42:16.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.591+0000 7f3662fbb700 1 --2- 172.21.15.171:0/1775528447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f365c1015b0 0x7f365c101990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.591+0000 7f3662fbb700 1 -- 172.21.15.171:0/1775528447 >> 172.21.15.171:0/1775528447 conn(0x7f365c0fce40 msgr2=0x7f365c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:16.595 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.591+0000 7f3662fbb700 1 -- 172.21.15.171:0/1775528447 shutdown_connections 2024-04-21T08:42:16.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.591+0000 7f3662fbb700 1 -- 172.21.15.171:0/1775528447 wait complete. 2024-04-21T08:42:16.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.592+0000 7f3662fbb700 1 Processor -- start 2024-04-21T08:42:16.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.592+0000 7f3662fbb700 1 -- start start 2024-04-21T08:42:16.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.592+0000 7f3662fbb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f365c103050 0x7f365c103430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.592+0000 7f3662fbb700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f365c10b830 con 0x7f365c103050 2024-04-21T08:42:16.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.592+0000 7f3660d57700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f365c103050 0x7f365c103430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.592+0000 7f3660d57700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f365c103050 0x7f365c103430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.171:60312/0 (socket says 172.21.15.171:60312) 2024-04-21T08:42:16.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.592+0000 7f3660d57700 1 -- 172.21.15.171:0/4284371639 learned_addr learned my addr 172.21.15.171:0/4284371639 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-04-21T08:42:16.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.593+0000 7f3660d57700 1 -- 172.21.15.171:0/4284371639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f365c104f40 con 0x7f365c103050 2024-04-21T08:42:16.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.593+0000 7f3660d57700 1 --2- 172.21.15.171:0/4284371639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f365c103050 0x7f365c103430 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f365000b5c0 tx=0x7f3650005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:16.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.593+0000 7f3659ffb700 1 -- 172.21.15.171:0/4284371639 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3650026070 con 0x7f365c103050 2024-04-21T08:42:16.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.593+0000 7f3662fbb700 1 -- 172.21.15.171:0/4284371639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f365c1039d0 con 0x7f365c103050 2024-04-21T08:42:16.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.593+0000 7f3659ffb700 1 -- 172.21.15.171:0/4284371639 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f365000b750 con 0x7f365c103050 2024-04-21T08:42:16.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.593+0000 7f3659ffb700 1 -- 172.21.15.171:0/4284371639 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3650003810 con 0x7f365c103050 2024-04-21T08:42:16.596 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.593+0000 7f3662fbb700 1 -- 172.21.15.171:0/4284371639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f365c103e90 con 0x7f365c103050 2024-04-21T08:42:16.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.594+0000 7f3662fbb700 1 -- 172.21.15.171:0/4284371639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f365c102a90 con 0x7f365c103050 2024-04-21T08:42:16.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.594+0000 7f3659ffb700 1 -- 172.21.15.171:0/4284371639 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f3650029070 con 0x7f365c103050 2024-04-21T08:42:16.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.594+0000 7f3659ffb700 1 --2- 172.21.15.171:0/4284371639 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f364c039600 0x7f364c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:16.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.594+0000 7f3659ffb700 1 -- 172.21.15.171:0/4284371639 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f365001e070 con 0x7f365c103050 2024-04-21T08:42:16.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.595+0000 7f365bfff700 1 --2- 172.21.15.171:0/4284371639 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f364c039600 0x7f364c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:16.597 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.595+0000 7f365bfff700 1 --2- 172.21.15.171:0/4284371639 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f364c039600 0x7f364c03bac0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3648006fd0 tx=0x7f3648006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:16.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.600+0000 7f3659ffb700 1 -- 172.21.15.171:0/4284371639 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3650013070 con 0x7f365c103050 2024-04-21T08:42:16.807 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.808+0000 7f3662fbb700 1 -- 172.21.15.171:0/4284371639 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi152:172.21.15.152=b;smithi171:172.21.15.171=c", "target": ["mon-mgr", ""]}) -- 0x7f365c1045f0 con 0x7f364c039600 2024-04-21T08:42:16.814 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.814+0000 7f3659ffb700 1 -- 172.21.15.171:0/4284371639 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f365c1045f0 con 0x7f364c039600 2024-04-21T08:42:16.814 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled mon update... 2024-04-21T08:42:16.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.816+0000 7f3662fbb700 1 -- 172.21.15.171:0/4284371639 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f364c039600 msgr2=0x7f364c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.816+0000 7f3662fbb700 1 --2- 172.21.15.171:0/4284371639 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f364c039600 0x7f364c03bac0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3648006fd0 tx=0x7f3648006e40 comp rx=0 tx=0).stop 2024-04-21T08:42:16.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.816+0000 7f3662fbb700 1 -- 172.21.15.171:0/4284371639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f365c103050 msgr2=0x7f365c103430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:16.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.816+0000 7f3662fbb700 1 --2- 172.21.15.171:0/4284371639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f365c103050 0x7f365c103430 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f365000b5c0 tx=0x7f3650005780 comp rx=0 tx=0).stop 2024-04-21T08:42:16.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.816+0000 7f3662fbb700 1 -- 172.21.15.171:0/4284371639 shutdown_connections 2024-04-21T08:42:16.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.816+0000 7f3662fbb700 1 --2- 172.21.15.171:0/4284371639 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f364c039600 0x7f364c03bac0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.816+0000 7f3662fbb700 1 --2- 172.21.15.171:0/4284371639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f365c103050 0x7f365c103430 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:16.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.816+0000 7f3662fbb700 1 -- 172.21.15.171:0/4284371639 >> 172.21.15.171:0/4284371639 conn(0x7f365c0fce40 msgr2=0x7f365c109120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:16.817 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.816+0000 7f3662fbb700 1 -- 172.21.15.171:0/4284371639 shutdown_connections 2024-04-21T08:42:16.817 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:16.816+0000 7f3662fbb700 1 -- 172.21.15.171:0/4284371639 wait complete. 2024-04-21T08:42:17.463 DEBUG:teuthology.orchestra.run.smithi152:mon.b> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mon.b.service 2024-04-21T08:42:17.466 DEBUG:teuthology.orchestra.run.smithi171:mon.c> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mon.c.service 2024-04-21T08:42:17.468 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T08:42:17.469 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T08:42:17.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:17 smithi028 ceph-mon[28003]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:17.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:17 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:17.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:17 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:17.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:17 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:17.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:17 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:17.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:17 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:17.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:17 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:42:17.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:17 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:18.484 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:42:18.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:18 smithi028 ceph-mon[28003]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi152:172.21.15.152=b;smithi171:172.21.15.171=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:18.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:18 smithi028 ceph-mon[28003]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi152:172.21.15.152=b;smithi171:172.21.15.171=c;count:3 2024-04-21T08:42:18.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:18 smithi028 ceph-mon[28003]: Deploying daemon mon.c on smithi171 2024-04-21T08:42:19.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:19 smithi028 ceph-mon[28003]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:21.466 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: rocksdb: DB pointer 0x55d84243a000 2024-04-21T08:42:21.466 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-21T08:42:21.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:21 smithi028 ceph-mon[28003]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:21.645 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.440+0000 7f336b9f6700 1 -- 172.21.15.171:0/1017105218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33641084e0 msgr2=0x7f33641088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:21.645 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.440+0000 7f336b9f6700 1 --2- 172.21.15.171:0/1017105218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33641084e0 0x7f33641088c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3360009910 tx=0x7f3360009c20 comp rx=0 tx=0).stop 2024-04-21T08:42:21.645 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.440+0000 7f336b9f6700 1 -- 172.21.15.171:0/1017105218 shutdown_connections 2024-04-21T08:42:21.645 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.440+0000 7f336b9f6700 1 --2- 172.21.15.171:0/1017105218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33641084e0 0x7f33641088c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:21.645 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.440+0000 7f336b9f6700 1 -- 172.21.15.171:0/1017105218 >> 172.21.15.171:0/1017105218 conn(0x7f33640fdf60 msgr2=0x7f3364100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:21.645 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.440+0000 7f336b9f6700 1 -- 172.21.15.171:0/1017105218 shutdown_connections 2024-04-21T08:42:21.645 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.440+0000 7f336b9f6700 1 -- 172.21.15.171:0/1017105218 wait complete. 2024-04-21T08:42:21.645 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.441+0000 7f336b9f6700 1 Processor -- start 2024-04-21T08:42:21.645 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.441+0000 7f336b9f6700 1 -- start start 2024-04-21T08:42:21.645 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.441+0000 7f336b9f6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33641084e0 0x7f3364197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:21.645 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.441+0000 7f336b9f6700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f336410c940 con 0x7f33641084e0 2024-04-21T08:42:21.646 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.442+0000 7f3369792700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33641084e0 0x7f3364197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:21.646 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.442+0000 7f3369792700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33641084e0 0x7f3364197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.171:60340/0 (socket says 172.21.15.171:60340) 2024-04-21T08:42:21.646 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.442+0000 7f3369792700 1 -- 172.21.15.171:0/327902736 learned_addr learned my addr 172.21.15.171:0/327902736 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-04-21T08:42:21.646 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.442+0000 7f3369792700 1 -- 172.21.15.171:0/327902736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3364198410 con 0x7f33641084e0 2024-04-21T08:42:21.646 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.443+0000 7f3369792700 1 --2- 172.21.15.171:0/327902736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33641084e0 0x7f3364197ed0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3360005700 tx=0x7f33600057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:21.646 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.443+0000 7f335a7fc700 1 -- 172.21.15.171:0/327902736 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3360026070 con 0x7f33641084e0 2024-04-21T08:42:21.646 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.443+0000 7f336b9f6700 1 -- 172.21.15.171:0/327902736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3364198700 con 0x7f33641084e0 2024-04-21T08:42:21.646 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.444+0000 7f336b9f6700 1 -- 172.21.15.171:0/327902736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f336419c7a0 con 0x7f33641084e0 2024-04-21T08:42:21.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.444+0000 7f335a7fc700 1 -- 172.21.15.171:0/327902736 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f336000b6f0 con 0x7f33641084e0 2024-04-21T08:42:21.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.444+0000 7f335a7fc700 1 -- 172.21.15.171:0/327902736 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3360003810 con 0x7f33641084e0 2024-04-21T08:42:21.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.444+0000 7f336b9f6700 1 -- 172.21.15.171:0/327902736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3348005330 con 0x7f33641084e0 2024-04-21T08:42:21.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.445+0000 7f335a7fc700 1 -- 172.21.15.171:0/327902736 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f3360029070 con 0x7f33641084e0 2024-04-21T08:42:21.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.445+0000 7f335a7fc700 1 --2- 172.21.15.171:0/327902736 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f33500396a0 0x7f335003bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:21.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.445+0000 7f335a7fc700 1 -- 172.21.15.171:0/327902736 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f336001e070 con 0x7f33641084e0 2024-04-21T08:42:21.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.446+0000 7f3368f91700 1 --2- 172.21.15.171:0/327902736 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f33500396a0 0x7f335003bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:21.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.447+0000 7f3368f91700 1 --2- 172.21.15.171:0/327902736 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f33500396a0 0x7f335003bb60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3354006fd0 tx=0x7f3354006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:21.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.451+0000 7f335a7fc700 1 -- 172.21.15.171:0/327902736 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3360013070 con 0x7f33641084e0 2024-04-21T08:42:21.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.505+0000 7f335a7fc700 1 -- 172.21.15.171:0/327902736 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f3360021410 con 0x7f33641084e0 2024-04-21T08:42:21.745 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:21.745+0000 7f336b9f6700 1 -- 172.21.15.171:0/327902736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f33480058f0 con 0x7f33641084e0 2024-04-21T08:42:21.745 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-21T08:42:21.745 INFO:journalctl@ceph.mon.c.smithi171.stdout: ** DB Stats ** 2024-04-21T08:42:21.746 INFO:journalctl@ceph.mon.c.smithi171.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:42:21.746 INFO:journalctl@ceph.mon.c.smithi171.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-21T08:42:21.746 INFO:journalctl@ceph.mon.c.smithi171.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:42:21.746 INFO:journalctl@ceph.mon.c.smithi171.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:42:21.746 INFO:journalctl@ceph.mon.c.smithi171.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-21T08:42:21.746 INFO:journalctl@ceph.mon.c.smithi171.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:42:21.746 INFO:journalctl@ceph.mon.c.smithi171.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:42:21.746 INFO:journalctl@ceph.mon.c.smithi171.stdout: 2024-04-21T08:42:21.746 INFO:journalctl@ceph.mon.c.smithi171.stdout: ** Compaction Stats [default] ** 2024-04-21T08:42:21.746 INFO:journalctl@ceph.mon.c.smithi171.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:42:21.746 INFO:journalctl@ceph.mon.c.smithi171.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-21T08:42:21.747 INFO:journalctl@ceph.mon.c.smithi171.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:42:21.747 INFO:journalctl@ceph.mon.c.smithi171.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:42:21.747 INFO:journalctl@ceph.mon.c.smithi171.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:42:21.747 INFO:journalctl@ceph.mon.c.smithi171.stdout: 2024-04-21T08:42:21.747 INFO:journalctl@ceph.mon.c.smithi171.stdout: ** Compaction Stats [default] ** 2024-04-21T08:42:21.747 INFO:journalctl@ceph.mon.c.smithi171.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:42:21.747 INFO:journalctl@ceph.mon.c.smithi171.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-21T08:42:21.747 INFO:journalctl@ceph.mon.c.smithi171.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:42:21.747 INFO:journalctl@ceph.mon.c.smithi171.stdout: 2024-04-21T08:42:21.747 INFO:journalctl@ceph.mon.c.smithi171.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-21T08:42:21.747 INFO:journalctl@ceph.mon.c.smithi171.stdout: 2024-04-21T08:42:21.747 INFO:journalctl@ceph.mon.c.smithi171.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:42:21.747 INFO:journalctl@ceph.mon.c.smithi171.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-21T08:42:21.747 INFO:journalctl@ceph.mon.c.smithi171.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-21T08:42:21.748 INFO:journalctl@ceph.mon.c.smithi171.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-21T08:42:21.748 INFO:journalctl@ceph.mon.c.smithi171.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-21T08:42:21.748 INFO:journalctl@ceph.mon.c.smithi171.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-21T08:42:21.748 INFO:journalctl@ceph.mon.c.smithi171.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:42:21.748 INFO:journalctl@ceph.mon.c.smithi171.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:42:21.748 INFO:journalctl@ceph.mon.c.smithi171.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-21T08:42:21.748 INFO:journalctl@ceph.mon.c.smithi171.stdout: Block cache BinnedLRUCache@0x55d84233b1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-21T08:42:21.748 INFO:journalctl@ceph.mon.c.smithi171.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-21T08:42:21.748 INFO:journalctl@ceph.mon.c.smithi171.stdout: 2024-04-21T08:42:21.748 INFO:journalctl@ceph.mon.c.smithi171.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-21T08:42:21.748 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-21T08:42:21.748 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: using public_addr v2:172.21.15.171:0/0 -> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] 2024-04-21T08:42:21.748 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: starting mon.c rank -1 at public addrs [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] at bind addrs [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:42:21.749 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: mon.c@-1(???) e0 preinit fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:42:21.749 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: mon.c@-1(synchronizing).mds e1 new map 2024-04-21T08:42:21.749 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-21T08:42:21.749 INFO:journalctl@ceph.mon.c.smithi171.stdout: e1 2024-04-21T08:42:21.749 INFO:journalctl@ceph.mon.c.smithi171.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T08:42:21.749 INFO:journalctl@ceph.mon.c.smithi171.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:42:21.749 INFO:journalctl@ceph.mon.c.smithi171.stdout: legacy client fscid: -1 2024-04-21T08:42:21.749 INFO:journalctl@ceph.mon.c.smithi171.stdout: 2024-04-21T08:42:21.749 INFO:journalctl@ceph.mon.c.smithi171.stdout: No filesystems configured 2024-04-21T08:42:21.749 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-21T08:42:21.749 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-21T08:42:21.749 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-21T08:42:21.749 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-21T08:42:21.749 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-21T08:42:21.750 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-21T08:42:21.750 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-21T08:42:21.750 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:42:21.750 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:42:21.750 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:42:21.750 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:21.750 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.750 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: Added host smithi152 2024-04-21T08:42:21.750 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:21.750 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.750 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:21.750 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.750 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:42:21.751 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.751 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:21.751 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:21.751 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi171", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:21.751 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.751 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.751 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.751 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.751 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:21.751 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:21.751 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:21.751 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: Updating smithi152:/etc/ceph/ceph.conf 2024-04-21T08:42:21.751 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: Deploying cephadm binary to smithi171 2024-04-21T08:42:21.751 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: Updating smithi152:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:42:21.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:42:21.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:21.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:21.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: Updating smithi152:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:42:21.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: Added host smithi171 2024-04-21T08:42:21.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:21.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.752 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:21.753 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.753 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:42:21.753 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:21.753 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2690422058' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-21T08:42:21.753 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.753 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.753 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.753 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.753 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:21.754 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:21.754 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:21.754 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:21.754 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: Updating smithi171:/etc/ceph/ceph.conf 2024-04-21T08:42:21.754 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: Updating smithi171:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:42:21.754 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:42:21.754 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2690422058' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-21T08:42:21.754 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:42:21.754 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: Updating smithi171:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:42:21.754 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.754 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.754 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.754 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:21.755 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.755 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:21.755 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:21.755 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:21.755 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:21.755 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:42:21.755 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:21.755 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi152:172.21.15.152=b;smithi171:172.21.15.171=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:21.755 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi152:172.21.15.152=b;smithi171:172.21.15.171=c;count:3 2024-04-21T08:42:21.755 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: Deploying daemon mon.c on smithi171 2024-04-21T08:42:21.755 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:21.755 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:21.755 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 ceph-mon[31358]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-21T08:42:21.755 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 bash[31331]: a03a67da3e6e9471e68e15dd4ed8b1e1aba5c2f79fa2845266d86846f745af4a 2024-04-21T08:42:21.756 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:21 smithi171 systemd[1]: Started Ceph mon.c for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:42:26.517 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:26.517+0000 7f335a7fc700 1 -- 172.21.15.171:0/327902736 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f3360034020 con 0x7f33641084e0 2024-04-21T08:42:26.517 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:42:26.517 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":2,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","modified":"2024-04-21T08:42:21.505266Z","created":"2024-04-21T08:40:30.404569Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:3300","nonce":0},{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-21T08:42:26.520 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:26.520+0000 7f336b9f6700 1 -- 172.21.15.171:0/327902736 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f33500396a0 msgr2=0x7f335003bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:26.520 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:26.520+0000 7f336b9f6700 1 --2- 172.21.15.171:0/327902736 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f33500396a0 0x7f335003bb60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3354006fd0 tx=0x7f3354006e40 comp rx=0 tx=0).stop 2024-04-21T08:42:26.520 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:26.520+0000 7f336b9f6700 1 -- 172.21.15.171:0/327902736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33641084e0 msgr2=0x7f3364197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:26.520 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:26.520+0000 7f336b9f6700 1 --2- 172.21.15.171:0/327902736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33641084e0 0x7f3364197ed0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3360005700 tx=0x7f33600057e0 comp rx=0 tx=0).stop 2024-04-21T08:42:26.520 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:26.520+0000 7f336b9f6700 1 -- 172.21.15.171:0/327902736 shutdown_connections 2024-04-21T08:42:26.520 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:26.520+0000 7f336b9f6700 1 --2- 172.21.15.171:0/327902736 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f33500396a0 0x7f335003bb60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:26.520 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:26.520+0000 7f336b9f6700 1 --2- 172.21.15.171:0/327902736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33641084e0 0x7f3364197ed0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:26.520 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:26.520+0000 7f336b9f6700 1 -- 172.21.15.171:0/327902736 >> 172.21.15.171:0/327902736 conn(0x7f33640fdf60 msgr2=0x7f33640fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:26.521 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:26.520+0000 7f336b9f6700 1 -- 172.21.15.171:0/327902736 shutdown_connections 2024-04-21T08:42:26.521 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:26.520+0000 7f336b9f6700 1 -- 172.21.15.171:0/327902736 wait complete. 2024-04-21T08:42:26.521 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 2 2024-04-21T08:42:26.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:26.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:42:26.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:26.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: mon.a calling monitor election 2024-04-21T08:42:26.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: from='client.? 172.21.15.171:0/327902736' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:42:26.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:26.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:26.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:26.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: mon.c calling monitor election 2024-04-21T08:42:26.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:26.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:26.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:26.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:26.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:42:26.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: monmap epoch 2 2024-04-21T08:42:26.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:42:26.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: last_changed 2024-04-21T08:42:21.505266+0000 2024-04-21T08:42:26.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: created 2024-04-21T08:40:30.404569+0000 2024-04-21T08:42:26.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: min_mon_release 19 (squid) 2024-04-21T08:42:26.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: election_strategy: 1 2024-04-21T08:42:26.816 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-21T08:42:26.817 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-04-21T08:42:26.817 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: fsmap 2024-04-21T08:42:26.817 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:42:26.817 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: mgrmap e13: x(active, since 51s) 2024-04-21T08:42:26.817 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: overall HEALTH_OK 2024-04-21T08:42:26.817 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:26.817 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:26 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:26.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:26.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:42:26.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:26.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: mon.a calling monitor election 2024-04-21T08:42:26.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: from='client.? 172.21.15.171:0/327902736' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:42:26.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:26.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:26.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:26.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: mon.c calling monitor election 2024-04-21T08:42:26.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:26.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:26.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:26.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:26.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:42:26.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: monmap epoch 2 2024-04-21T08:42:26.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:42:26.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: last_changed 2024-04-21T08:42:21.505266+0000 2024-04-21T08:42:26.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: created 2024-04-21T08:40:30.404569+0000 2024-04-21T08:42:26.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: min_mon_release 19 (squid) 2024-04-21T08:42:26.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: election_strategy: 1 2024-04-21T08:42:26.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-21T08:42:26.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-04-21T08:42:26.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: fsmap 2024-04-21T08:42:26.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:42:26.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: mgrmap e13: x(active, since 51s) 2024-04-21T08:42:26.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: overall HEALTH_OK 2024-04-21T08:42:26.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:26.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:26 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:27.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:27 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:27.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:27 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:42:27.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:27 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:27.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:27 smithi028 ceph-mon[28003]: Deploying daemon mon.b on smithi152 2024-04-21T08:42:27.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:27 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:27.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:27 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:27.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:27 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:42:27.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:27 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:27.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:27 smithi171 ceph-mon[31358]: Deploying daemon mon.b on smithi152 2024-04-21T08:42:27.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:27 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:28.098 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T08:42:28.098 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T08:42:28.343 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:42:28.728 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:28 smithi171 ceph-mon[31358]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:28.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:28 smithi028 ceph-mon[28003]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:29.674 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.674+0000 7f9d14c94700 1 -- 172.21.15.171:0/574933167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d101074d0 msgr2=0x7f9d101078b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.674 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.674+0000 7f9d14c94700 1 --2- 172.21.15.171:0/574933167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d101074d0 0x7f9d101078b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009910 tx=0x7f9d00009c20 comp rx=0 tx=0).stop 2024-04-21T08:42:29.674 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.674+0000 7f9d14c94700 1 -- 172.21.15.171:0/574933167 shutdown_connections 2024-04-21T08:42:29.674 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.674+0000 7f9d14c94700 1 --2- 172.21.15.171:0/574933167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d101074d0 0x7f9d101078b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.674 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.674+0000 7f9d14c94700 1 -- 172.21.15.171:0/574933167 >> 172.21.15.171:0/574933167 conn(0x7f9d100fcf70 msgr2=0x7f9d100ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:29.674 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.674+0000 7f9d14c94700 1 -- 172.21.15.171:0/574933167 shutdown_connections 2024-04-21T08:42:29.674 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.674+0000 7f9d14c94700 1 -- 172.21.15.171:0/574933167 wait complete. 2024-04-21T08:42:29.676 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.675+0000 7f9d14c94700 1 Processor -- start 2024-04-21T08:42:29.676 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.675+0000 7f9d14c94700 1 -- start start 2024-04-21T08:42:29.676 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.675+0000 7f9d14c94700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d101074d0 0x7f9d10197000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:29.676 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.675+0000 7f9d14c94700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d10197540 0x7f9d1019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:29.676 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.675+0000 7f9d14c94700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1010b930 con 0x7f9d101074d0 2024-04-21T08:42:29.677 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.675+0000 7f9d14c94700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1010b7b0 con 0x7f9d10197540 2024-04-21T08:42:29.677 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.675+0000 7f9d0dd9b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d10197540 0x7f9d1019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:29.677 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.675+0000 7f9d0dd9b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d10197540 0x7f9d1019b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:33774/0 (socket says 172.21.15.171:33774) 2024-04-21T08:42:29.677 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.675+0000 7f9d0dd9b700 1 -- 172.21.15.171:0/2868163324 learned_addr learned my addr 172.21.15.171:0/2868163324 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-04-21T08:42:29.677 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.675+0000 7f9d0e59c700 1 --2- 172.21.15.171:0/2868163324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d101074d0 0x7f9d10197000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:29.677 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.675+0000 7f9d0dd9b700 1 -- 172.21.15.171:0/2868163324 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d10197540 msgr2=0x7f9d1019b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-21T08:42:29.677 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.675+0000 7f9d0dd9b700 1 -- 172.21.15.171:0/2868163324 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d10197540 msgr2=0x7f9d1019b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-21T08:42:29.677 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.675+0000 7f9d0dd9b700 1 --2- 172.21.15.171:0/2868163324 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d10197540 0x7f9d1019b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-21T08:42:29.677 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.675+0000 7f9d0dd9b700 1 --2- 172.21.15.171:0/2868163324 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d10197540 0x7f9d1019b8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:42:29.677 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.676+0000 7f9d0e59c700 1 -- 172.21.15.171:0/2868163324 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d10197540 msgr2=0x7f9d1019b8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:42:29.677 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.676+0000 7f9d0e59c700 1 --2- 172.21.15.171:0/2868163324 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d10197540 0x7f9d1019b8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.677 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.676+0000 7f9d0e59c700 1 -- 172.21.15.171:0/2868163324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d10197b80 con 0x7f9d101074d0 2024-04-21T08:42:29.677 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.676+0000 7f9d0e59c700 1 --2- 172.21.15.171:0/2868163324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d101074d0 0x7f9d10197000 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9d00014000 tx=0x7f9d00005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:29.678 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.676+0000 7f9cff7fe700 1 -- 172.21.15.171:0/2868163324 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9d00005dc0 con 0x7f9d101074d0 2024-04-21T08:42:29.678 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.676+0000 7f9d14c94700 1 -- 172.21.15.171:0/2868163324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1019be10 con 0x7f9d101074d0 2024-04-21T08:42:29.678 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.676+0000 7f9d14c94700 1 -- 172.21.15.171:0/2868163324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d1019c370 con 0x7f9d101074d0 2024-04-21T08:42:29.678 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.676+0000 7f9cff7fe700 1 -- 172.21.15.171:0/2868163324 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d0000b6f0 con 0x7f9d101074d0 2024-04-21T08:42:29.678 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.676+0000 7f9cff7fe700 1 -- 172.21.15.171:0/2868163324 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9d000038f0 con 0x7f9d101074d0 2024-04-21T08:42:29.678 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.677+0000 7f9d14c94700 1 -- 172.21.15.171:0/2868163324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d10102b80 con 0x7f9d101074d0 2024-04-21T08:42:29.679 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.677+0000 7f9cff7fe700 1 -- 172.21.15.171:0/2868163324 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f9d00029070 con 0x7f9d101074d0 2024-04-21T08:42:29.679 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.677+0000 7f9cff7fe700 1 --2- 172.21.15.171:0/2868163324 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f9cf803b600 0x7f9cf803dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:29.679 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.677+0000 7f9cff7fe700 1 -- 172.21.15.171:0/2868163324 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9d0001e070 con 0x7f9d101074d0 2024-04-21T08:42:29.679 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.678+0000 7f9d0dd9b700 1 --2- 172.21.15.171:0/2868163324 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f9cf803b600 0x7f9cf803dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:29.679 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.678+0000 7f9d0dd9b700 1 --2- 172.21.15.171:0/2868163324 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f9cf803b600 0x7f9cf803dac0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009180 tx=0x7f9d04008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:29.682 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.683+0000 7f9cff7fe700 1 -- 172.21.15.171:0/2868163324 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d00013070 con 0x7f9d101074d0 2024-04-21T08:42:29.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.963+0000 7f9d14c94700 1 -- 172.21.15.171:0/2868163324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9d1019c690 con 0x7f9d101074d0 2024-04-21T08:42:29.963 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.964+0000 7f9cff7fe700 1 -- 172.21.15.171:0/2868163324 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f9d00030090 con 0x7f9d101074d0 2024-04-21T08:42:29.963 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:42:29.963 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":2,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","modified":"2024-04-21T08:42:21.505266Z","created":"2024-04-21T08:40:30.404569Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:3300","nonce":0},{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-21T08:42:29.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.966+0000 7f9d14c94700 1 -- 172.21.15.171:0/2868163324 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f9cf803b600 msgr2=0x7f9cf803dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.966+0000 7f9d14c94700 1 --2- 172.21.15.171:0/2868163324 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f9cf803b600 0x7f9cf803dac0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009180 tx=0x7f9d04008040 comp rx=0 tx=0).stop 2024-04-21T08:42:29.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.966+0000 7f9d14c94700 1 -- 172.21.15.171:0/2868163324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d101074d0 msgr2=0x7f9d10197000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.966+0000 7f9d14c94700 1 --2- 172.21.15.171:0/2868163324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d101074d0 0x7f9d10197000 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9d00014000 tx=0x7f9d00005030 comp rx=0 tx=0).stop 2024-04-21T08:42:29.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.966+0000 7f9d14c94700 1 -- 172.21.15.171:0/2868163324 shutdown_connections 2024-04-21T08:42:29.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.966+0000 7f9d14c94700 1 --2- 172.21.15.171:0/2868163324 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f9cf803b600 0x7f9cf803dac0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.966+0000 7f9d14c94700 1 --2- 172.21.15.171:0/2868163324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d101074d0 0x7f9d10197000 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.966+0000 7f9d14c94700 1 --2- 172.21.15.171:0/2868163324 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d10197540 0x7f9d1019b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.966+0000 7f9d14c94700 1 -- 172.21.15.171:0/2868163324 >> 172.21.15.171:0/2868163324 conn(0x7f9d100fcf70 msgr2=0x7f9d10102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:29.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.966+0000 7f9d14c94700 1 -- 172.21.15.171:0/2868163324 shutdown_connections 2024-04-21T08:42:29.966 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:29.966+0000 7f9d14c94700 1 -- 172.21.15.171:0/2868163324 wait complete. 2024-04-21T08:42:29.967 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 2 2024-04-21T08:42:30.389 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.b@-1(???) e0 preinit fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:42:30.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.b@-1(synchronizing).mds e1 new map 2024-04-21T08:42:30.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-21T08:42:30.855 INFO:journalctl@ceph.mon.b.smithi152.stdout: e1 2024-04-21T08:42:30.855 INFO:journalctl@ceph.mon.b.smithi152.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T08:42:30.855 INFO:journalctl@ceph.mon.b.smithi152.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:42:30.855 INFO:journalctl@ceph.mon.b.smithi152.stdout: legacy client fscid: -1 2024-04-21T08:42:30.855 INFO:journalctl@ceph.mon.b.smithi152.stdout: 2024-04-21T08:42:30.855 INFO:journalctl@ceph.mon.b.smithi152.stdout: No filesystems configured 2024-04-21T08:42:30.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-21T08:42:30.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-21T08:42:30.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-21T08:42:30.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-21T08:42:30.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-21T08:42:30.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-21T08:42:30.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-21T08:42:30.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:42:30.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:42:30.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:42:30.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: Added host smithi152 2024-04-21T08:42:30.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:30.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:42:30.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi171", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:30.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:30.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:30.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:30.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: Updating smithi152:/etc/ceph/ceph.conf 2024-04-21T08:42:30.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: Deploying cephadm binary to smithi171 2024-04-21T08:42:30.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: Updating smithi152:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:42:30.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:42:30.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: Updating smithi152:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:42:30.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: Added host smithi171 2024-04-21T08:42:30.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:30.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:42:30.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2690422058' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-21T08:42:30.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.859 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:30.859 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:30.859 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:30.859 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.859 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: Updating smithi171:/etc/ceph/ceph.conf 2024-04-21T08:42:30.859 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: Updating smithi171:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:42:30.859 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:42:30.859 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2690422058' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-21T08:42:30.859 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:42:30.860 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: Updating smithi171:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:42:30.860 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.860 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.860 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.860 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.860 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.860 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:30.860 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:30.860 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:30.860 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.860 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:42:30.860 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:30.860 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi152:172.21.15.152=b;smithi171:172.21.15.171=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:30.860 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi152:172.21.15.152=b;smithi171:172.21.15.171=c;count:3 2024-04-21T08:42:30.860 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: Deploying daemon mon.c on smithi171 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.a calling monitor election 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='client.? 172.21.15.171:0/327902736' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.c calling monitor election 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:30.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:42:30.862 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: monmap epoch 2 2024-04-21T08:42:30.862 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:42:30.862 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: last_changed 2024-04-21T08:42:21.505266+0000 2024-04-21T08:42:30.862 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: created 2024-04-21T08:40:30.404569+0000 2024-04-21T08:42:30.862 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: min_mon_release 19 (squid) 2024-04-21T08:42:30.862 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: election_strategy: 1 2024-04-21T08:42:30.862 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-21T08:42:30.862 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-04-21T08:42:30.862 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: fsmap 2024-04-21T08:42:30.862 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:42:30.862 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mgrmap e13: x(active, since 51s) 2024-04-21T08:42:30.862 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: overall HEALTH_OK 2024-04-21T08:42:30.862 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.863 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.863 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:30.863 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:42:30.863 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:30.863 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: Deploying daemon mon.b on smithi152 2024-04-21T08:42:30.863 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:30.863 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:30.863 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 ceph-mon[31401]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-21T08:42:30.863 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 bash[31382]: 01fa7845fe29d64dda646b47388082716b538eb4b77e8af2158e096cbef91b17 2024-04-21T08:42:30.863 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:30 smithi152 systemd[1]: Started Ceph mon.b for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:42:31.538 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T08:42:31.538 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T08:42:31.779 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:42:35.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:42:35.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: mon.a calling monitor election 2024-04-21T08:42:35.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: mon.c calling monitor election 2024-04-21T08:42:35.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:35.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:35.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:35.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:35.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:35.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:35.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:35.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:35.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:35.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:42:35.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: monmap epoch 3 2024-04-21T08:42:35.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:42:35.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: last_changed 2024-04-21T08:42:30.409916+0000 2024-04-21T08:42:35.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: created 2024-04-21T08:40:30.404569+0000 2024-04-21T08:42:35.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: min_mon_release 19 (squid) 2024-04-21T08:42:35.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: election_strategy: 1 2024-04-21T08:42:35.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-21T08:42:35.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-04-21T08:42:35.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: 2: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2024-04-21T08:42:35.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: fsmap 2024-04-21T08:42:35.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:42:35.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: mgrmap e13: x(active, since 60s) 2024-04-21T08:42:35.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: overall HEALTH_OK 2024-04-21T08:42:35.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:35.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:35.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:35.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:42:35.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: mon.a calling monitor election 2024-04-21T08:42:35.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: mon.c calling monitor election 2024-04-21T08:42:35.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:35.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:35.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:35.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:35.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:35.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:35.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:35.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:35.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:35.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:42:35.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: monmap epoch 3 2024-04-21T08:42:35.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:42:35.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: last_changed 2024-04-21T08:42:30.409916+0000 2024-04-21T08:42:35.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: created 2024-04-21T08:40:30.404569+0000 2024-04-21T08:42:35.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: min_mon_release 19 (squid) 2024-04-21T08:42:35.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: election_strategy: 1 2024-04-21T08:42:35.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-21T08:42:35.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-04-21T08:42:35.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: 2: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2024-04-21T08:42:35.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: fsmap 2024-04-21T08:42:35.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:42:35.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: mgrmap e13: x(active, since 60s) 2024-04-21T08:42:35.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: overall HEALTH_OK 2024-04-21T08:42:35.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:35.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:35.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:36.280 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.278+0000 7f069916f700 1 -- 172.21.15.171:0/153211379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c0035c0 msgr2=0x7f067c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:36.280 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.278+0000 7f069916f700 1 --2- 172.21.15.171:0/153211379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c0035c0 0x7f067c005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0694102490 tx=0x7f068400feb0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.280 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.278+0000 7f069916f700 1 -- 172.21.15.171:0/153211379 shutdown_connections 2024-04-21T08:42:36.280 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.278+0000 7f069916f700 1 --2- 172.21.15.171:0/153211379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c0035c0 0x7f067c005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.280 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.278+0000 7f069916f700 1 -- 172.21.15.171:0/153211379 >> 172.21.15.171:0/153211379 conn(0x7f06940fe050 msgr2=0x7f0694100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:36.280 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.278+0000 7f069916f700 1 -- 172.21.15.171:0/153211379 shutdown_connections 2024-04-21T08:42:36.281 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.278+0000 7f069916f700 1 -- 172.21.15.171:0/153211379 wait complete. 2024-04-21T08:42:36.281 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.279+0000 7f069916f700 1 Processor -- start 2024-04-21T08:42:36.281 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.279+0000 7f069916f700 1 -- start start 2024-04-21T08:42:36.281 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.279+0000 7f069916f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c0035c0 0x7f0694198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:36.281 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069916f700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06941985b0 0x7f069419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:36.281 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069916f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f069419ce80 0x7f069419d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:36.281 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069916f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f069410a460 con 0x7f067c0035c0 2024-04-21T08:42:36.281 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069916f700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f069410a2e0 con 0x7f06941985b0 2024-04-21T08:42:36.281 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069916f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f069410a5e0 con 0x7f069419ce80 2024-04-21T08:42:36.281 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069359e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f069419ce80 0x7f069419d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:36.281 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069359e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f069419ce80 0x7f069419d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:37000/0 (socket says 172.21.15.171:37000) 2024-04-21T08:42:36.281 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069359e700 1 -- 172.21.15.171:0/4041628015 learned_addr learned my addr 172.21.15.171:0/4041628015 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-04-21T08:42:36.281 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069359e700 1 -- 172.21.15.171:0/4041628015 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f069419ce80 msgr2=0x7f069419d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-21T08:42:36.282 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069359e700 1 -- 172.21.15.171:0/4041628015 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f069419ce80 msgr2=0x7f069419d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-21T08:42:36.282 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069359e700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f069419ce80 0x7f069419d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-21T08:42:36.282 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069259c700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06941985b0 0x7f069419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:36.282 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069359e700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f069419ce80 0x7f069419d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:42:36.282 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f0692d9d700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c0035c0 0x7f0694198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:36.282 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069259c700 1 -- 172.21.15.171:0/4041628015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06941985b0 msgr2=0x7f069419c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-21T08:42:36.282 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069259c700 1 -- 172.21.15.171:0/4041628015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06941985b0 msgr2=0x7f069419c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-21T08:42:36.282 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069259c700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06941985b0 0x7f069419c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-21T08:42:36.282 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.280+0000 7f069259c700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06941985b0 0x7f069419c940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:42:36.284 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.281+0000 7f0692d9d700 1 -- 172.21.15.171:0/4041628015 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f069419ce80 msgr2=0x7f069419d330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:42:36.284 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.281+0000 7f0692d9d700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f069419ce80 0x7f069419d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.284 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.281+0000 7f0692d9d700 1 -- 172.21.15.171:0/4041628015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06941985b0 msgr2=0x7f069419c940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:42:36.284 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.281+0000 7f0692d9d700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06941985b0 0x7f069419c940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.284 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.281+0000 7f0692d9d700 1 -- 172.21.15.171:0/4041628015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06941a4bf0 con 0x7f067c0035c0 2024-04-21T08:42:36.284 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.281+0000 7f0692d9d700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c0035c0 0x7f0694198070 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0684005ed0 tx=0x7f0684013960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:36.284 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.282+0000 7f067bfff700 1 -- 172.21.15.171:0/4041628015 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0684029030 con 0x7f067c0035c0 2024-04-21T08:42:36.284 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.282+0000 7f069916f700 1 -- 172.21.15.171:0/4041628015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06941a4dc0 con 0x7f067c0035c0 2024-04-21T08:42:36.284 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.282+0000 7f067bfff700 1 -- 172.21.15.171:0/4041628015 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06840044e0 con 0x7f067c0035c0 2024-04-21T08:42:36.284 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.282+0000 7f067bfff700 1 -- 172.21.15.171:0/4041628015 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06840038a0 con 0x7f067c0035c0 2024-04-21T08:42:36.284 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.282+0000 7f069916f700 1 -- 172.21.15.171:0/4041628015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06941a52f0 con 0x7f067c0035c0 2024-04-21T08:42:36.284 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.283+0000 7f069916f700 1 -- 172.21.15.171:0/4041628015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06940640a0 con 0x7f067c0035c0 2024-04-21T08:42:36.285 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.283+0000 7f067bfff700 1 -- 172.21.15.171:0/4041628015 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f068402c070 con 0x7f067c0035c0 2024-04-21T08:42:36.285 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.283+0000 7f067bfff700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0680039ae0 0x7f068003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:36.285 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.283+0000 7f067bfff700 1 -- 172.21.15.171:0/4041628015 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f068404ed60 con 0x7f067c0035c0 2024-04-21T08:42:36.285 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.283+0000 7f069259c700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0680039ae0 0x7f068003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:36.285 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.284+0000 7f069259c700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0680039ae0 0x7f068003bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0688009850 tx=0x7f0688008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:36.289 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.289+0000 7f067bfff700 1 -- 172.21.15.171:0/4041628015 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0684054020 con 0x7f067c0035c0 2024-04-21T08:42:36.578 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.579+0000 7f069916f700 1 -- 172.21.15.171:0/4041628015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f069402df80 con 0x7f067c0035c0 2024-04-21T08:42:36.579 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:36 smithi171 ceph-mon[31358]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:36.579 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:36.579 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:36.579 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:36.579 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.580+0000 7f067bfff700 1 -- 172.21.15.171:0/4041628015 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f068401abd0 con 0x7f067c0035c0 2024-04-21T08:42:36.580 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-04-21T08:42:36.580 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":3,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","modified":"2024-04-21T08:42:30.409916Z","created":"2024-04-21T08:40:30.404569Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:3300","nonce":0},{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:3300","nonce":0},{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-21T08:42:36.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.582+0000 7f069916f700 1 -- 172.21.15.171:0/4041628015 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0680039ae0 msgr2=0x7f068003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:36.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.582+0000 7f069916f700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0680039ae0 0x7f068003bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0688009850 tx=0x7f0688008040 comp rx=0 tx=0).stop 2024-04-21T08:42:36.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.582+0000 7f069916f700 1 -- 172.21.15.171:0/4041628015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c0035c0 msgr2=0x7f0694198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:36.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.582+0000 7f069916f700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c0035c0 0x7f0694198070 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0684005ed0 tx=0x7f0684013960 comp rx=0 tx=0).stop 2024-04-21T08:42:36.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.582+0000 7f069916f700 1 -- 172.21.15.171:0/4041628015 shutdown_connections 2024-04-21T08:42:36.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.582+0000 7f069916f700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0680039ae0 0x7f068003bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.582+0000 7f069916f700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c0035c0 0x7f0694198070 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.582+0000 7f069916f700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f069419ce80 0x7f069419d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.582+0000 7f069916f700 1 --2- 172.21.15.171:0/4041628015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06941985b0 0x7f069419c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.582+0000 7f069916f700 1 -- 172.21.15.171:0/4041628015 >> 172.21.15.171:0/4041628015 conn(0x7f06940fe050 msgr2=0x7f06941070f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:36.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.582+0000 7f069916f700 1 -- 172.21.15.171:0/4041628015 shutdown_connections 2024-04-21T08:42:36.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:36.582+0000 7f069916f700 1 -- 172.21.15.171:0/4041628015 wait complete. 2024-04-21T08:42:36.583 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 3 2024-04-21T08:42:36.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:36 smithi028 ceph-mon[28003]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:36.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:36.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:36.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:37.452 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-21T08:42:37.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-21T08:42:37.716 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:37.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:42:37.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: mon.a calling monitor election 2024-04-21T08:42:37.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: mon.c calling monitor election 2024-04-21T08:42:37.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:37.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:42:37.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:37.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:37.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:37.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:37.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:37.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:37.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:37.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:42:37.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: monmap epoch 3 2024-04-21T08:42:37.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:42:37.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: last_changed 2024-04-21T08:42:30.409916+0000 2024-04-21T08:42:37.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: created 2024-04-21T08:40:30.404569+0000 2024-04-21T08:42:37.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: min_mon_release 19 (squid) 2024-04-21T08:42:37.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: election_strategy: 1 2024-04-21T08:42:37.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-21T08:42:37.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-04-21T08:42:37.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: 2: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2024-04-21T08:42:37.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: fsmap 2024-04-21T08:42:37.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:42:37.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: mgrmap e13: x(active, since 60s) 2024-04-21T08:42:37.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: overall HEALTH_OK 2024-04-21T08:42:37.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:37.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:37.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:37.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:37.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:37.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:37.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:38.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: mon.b calling monitor election 2024-04-21T08:42:38.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: mon.b calling monitor election 2024-04-21T08:42:38.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: mon.c calling monitor election 2024-04-21T08:42:38.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: mon.a calling monitor election 2024-04-21T08:42:38.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T08:42:38.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: monmap epoch 3 2024-04-21T08:42:38.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:42:38.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: last_changed 2024-04-21T08:42:30.409916+0000 2024-04-21T08:42:38.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: created 2024-04-21T08:40:30.404569+0000 2024-04-21T08:42:38.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: min_mon_release 19 (squid) 2024-04-21T08:42:38.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: election_strategy: 1 2024-04-21T08:42:38.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-21T08:42:38.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-04-21T08:42:38.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: 2: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2024-04-21T08:42:38.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: fsmap 2024-04-21T08:42:38.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:42:38.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: mgrmap e13: x(active, since 62s) 2024-04-21T08:42:38.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: overall HEALTH_OK 2024-04-21T08:42:38.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:42:38.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:42:38.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:38.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: mon.b calling monitor election 2024-04-21T08:42:38.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: mon.b calling monitor election 2024-04-21T08:42:38.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: mon.c calling monitor election 2024-04-21T08:42:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: mon.a calling monitor election 2024-04-21T08:42:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T08:42:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: monmap epoch 3 2024-04-21T08:42:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:42:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: last_changed 2024-04-21T08:42:30.409916+0000 2024-04-21T08:42:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: created 2024-04-21T08:40:30.404569+0000 2024-04-21T08:42:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: min_mon_release 19 (squid) 2024-04-21T08:42:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: election_strategy: 1 2024-04-21T08:42:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-21T08:42:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-04-21T08:42:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: 2: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2024-04-21T08:42:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: fsmap 2024-04-21T08:42:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:42:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: mgrmap e13: x(active, since 62s) 2024-04-21T08:42:38.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: overall HEALTH_OK 2024-04-21T08:42:38.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:42:38.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:42:38.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: mon.b calling monitor election 2024-04-21T08:42:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: mon.b calling monitor election 2024-04-21T08:42:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: mon.c calling monitor election 2024-04-21T08:42:38.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: mon.a calling monitor election 2024-04-21T08:42:38.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T08:42:38.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: monmap epoch 3 2024-04-21T08:42:38.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:42:38.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: last_changed 2024-04-21T08:42:30.409916+0000 2024-04-21T08:42:38.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: created 2024-04-21T08:40:30.404569+0000 2024-04-21T08:42:38.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: min_mon_release 19 (squid) 2024-04-21T08:42:38.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: election_strategy: 1 2024-04-21T08:42:38.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-21T08:42:38.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: 1: [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon.c 2024-04-21T08:42:38.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: 2: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.b 2024-04-21T08:42:38.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: fsmap 2024-04-21T08:42:38.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:42:38.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: mgrmap e13: x(active, since 62s) 2024-04-21T08:42:38.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: overall HEALTH_OK 2024-04-21T08:42:38.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:38.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:42:38.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:42:38.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:39.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:38 smithi028 ceph-mon[28003]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:39.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:38 smithi028 ceph-mon[28003]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T08:42:39.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:38 smithi028 ceph-mon[28003]: Reconfiguring daemon mon.a on smithi028 2024-04-21T08:42:39.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:38 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:39.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:38 smithi152 ceph-mon[31401]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:39.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:38 smithi152 ceph-mon[31401]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T08:42:39.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:38 smithi152 ceph-mon[31401]: Reconfiguring daemon mon.a on smithi028 2024-04-21T08:42:39.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:38 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:39.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:38 smithi171 ceph-mon[31358]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:39.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:38 smithi171 ceph-mon[31358]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T08:42:39.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:38 smithi171 ceph-mon[31358]: Reconfiguring daemon mon.a on smithi028 2024-04-21T08:42:39.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:38 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:42:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.331+0000 7fa1538e9700 1 -- 172.21.15.28:0/3755713800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c069e60 msgr2=0x7fa14c06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.331+0000 7fa1538e9700 1 --2- 172.21.15.28:0/3755713800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c069e60 0x7fa14c06a240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa140009910 tx=0x7fa140009c20 comp rx=0 tx=0).stop 2024-04-21T08:42:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.331+0000 7fa1538e9700 1 -- 172.21.15.28:0/3755713800 shutdown_connections 2024-04-21T08:42:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.331+0000 7fa1538e9700 1 --2- 172.21.15.28:0/3755713800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c069e60 0x7fa14c06a240 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.331+0000 7fa1538e9700 1 -- 172.21.15.28:0/3755713800 >> 172.21.15.28:0/3755713800 conn(0x7fa14c076ea0 msgr2=0x7fa14c0772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.331+0000 7fa1538e9700 1 -- 172.21.15.28:0/3755713800 shutdown_connections 2024-04-21T08:42:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.331+0000 7fa1538e9700 1 -- 172.21.15.28:0/3755713800 wait complete. 2024-04-21T08:42:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa1538e9700 1 Processor -- start 2024-04-21T08:42:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa1538e9700 1 -- start start 2024-04-21T08:42:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa1538e9700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa14c069e60 0x7fa14c19a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa1538e9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c19a8a0 0x7fa14c19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa1538e9700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa14c19f170 0x7fa14c19f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa1538e9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa14c10eb90 con 0x7fa14c19a8a0 2024-04-21T08:42:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa1538e9700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa14c10ea10 con 0x7fa14c19f170 2024-04-21T08:42:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa1538e9700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa14c10ed10 con 0x7fa14c069e60 2024-04-21T08:42:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa150e84700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c19a8a0 0x7fa14c19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa150e84700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c19a8a0 0x7fa14c19ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33552/0 (socket says 172.21.15.28:33552) 2024-04-21T08:42:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa150e84700 1 -- 172.21.15.28:0/1345999931 learned_addr learned my addr 172.21.15.28:0/1345999931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:42:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa150e84700 1 -- 172.21.15.28:0/1345999931 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa14c069e60 msgr2=0x7fa14c19a360 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:42:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa150e84700 1 --2- 172.21.15.28:0/1345999931 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa14c069e60 0x7fa14c19a360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa150e84700 1 -- 172.21.15.28:0/1345999931 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa14c19f170 msgr2=0x7fa14c19f620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:42:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa150e84700 1 --2- 172.21.15.28:0/1345999931 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa14c19f170 0x7fa14c19f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.332+0000 7fa150e84700 1 -- 172.21.15.28:0/1345999931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa14c1a6ee0 con 0x7fa14c19a8a0 2024-04-21T08:42:40.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.333+0000 7fa150e84700 1 --2- 172.21.15.28:0/1345999931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c19a8a0 0x7fa14c19ec30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa14800b780 tx=0x7fa14800bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.333+0000 7fa13e7fc700 1 -- 172.21.15.28:0/1345999931 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa148004230 con 0x7fa14c19a8a0 2024-04-21T08:42:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.333+0000 7fa13e7fc700 1 -- 172.21.15.28:0/1345999931 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1480043d0 con 0x7fa14c19a8a0 2024-04-21T08:42:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.333+0000 7fa1538e9700 1 -- 172.21.15.28:0/1345999931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa14c1a7110 con 0x7fa14c19a8a0 2024-04-21T08:42:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.333+0000 7fa1538e9700 1 -- 172.21.15.28:0/1345999931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa14c1a7650 con 0x7fa14c19a8a0 2024-04-21T08:42:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.333+0000 7fa13e7fc700 1 -- 172.21.15.28:0/1345999931 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa148011bd0 con 0x7fa14c19a8a0 2024-04-21T08:42:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.334+0000 7fa13e7fc700 1 -- 172.21.15.28:0/1345999931 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fa14801a400 con 0x7fa14c19a8a0 2024-04-21T08:42:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.334+0000 7fa1538e9700 1 -- 172.21.15.28:0/1345999931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa14c0640a0 con 0x7fa14c19a8a0 2024-04-21T08:42:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.334+0000 7fa13e7fc700 1 --2- 172.21.15.28:0/1345999931 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa138039ae0 0x7fa13803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.334+0000 7fa13e7fc700 1 -- 172.21.15.28:0/1345999931 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa14804c450 con 0x7fa14c19a8a0 2024-04-21T08:42:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.334+0000 7fa151685700 1 --2- 172.21.15.28:0/1345999931 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa138039ae0 0x7fa13803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.335+0000 7fa151685700 1 --2- 172.21.15.28:0/1345999931 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa138039ae0 0x7fa13803bfa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa140005e00 tx=0x7fa140005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.339+0000 7fa13e7fc700 1 -- 172.21.15.28:0/1345999931 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa1480202a0 con 0x7fa14c19a8a0 2024-04-21T08:42:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.556+0000 7fa1538e9700 1 -- 172.21.15.28:0/1345999931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fa14c1a79c0 con 0x7fa14c19a8a0 2024-04-21T08:42:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.556+0000 7fa13e7fc700 1 -- 172.21.15.28:0/1345999931 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7fa148018750 con 0x7fa14c19a8a0 2024-04-21T08:42:40.556 INFO:teuthology.orchestra.run.smithi028.stdout:# minimal ceph.conf for b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:42:40.556 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-04-21T08:42:40.556 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:42:40.556 INFO:teuthology.orchestra.run.smithi028.stdout: mon_host = [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] 2024-04-21T08:42:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.558+0000 7fa1538e9700 1 -- 172.21.15.28:0/1345999931 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa138039ae0 msgr2=0x7fa13803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.558+0000 7fa1538e9700 1 --2- 172.21.15.28:0/1345999931 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa138039ae0 0x7fa13803bfa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa140005e00 tx=0x7fa140005fd0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.558+0000 7fa1538e9700 1 -- 172.21.15.28:0/1345999931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c19a8a0 msgr2=0x7fa14c19ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.558+0000 7fa1538e9700 1 --2- 172.21.15.28:0/1345999931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c19a8a0 0x7fa14c19ec30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa14800b780 tx=0x7fa14800bb40 comp rx=0 tx=0).stop 2024-04-21T08:42:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.558+0000 7fa1538e9700 1 -- 172.21.15.28:0/1345999931 shutdown_connections 2024-04-21T08:42:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.558+0000 7fa1538e9700 1 --2- 172.21.15.28:0/1345999931 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa14c19f170 0x7fa14c19f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.558+0000 7fa1538e9700 1 --2- 172.21.15.28:0/1345999931 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa14c069e60 0x7fa14c19a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.558+0000 7fa1538e9700 1 --2- 172.21.15.28:0/1345999931 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa138039ae0 0x7fa13803bfa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.558+0000 7fa1538e9700 1 --2- 172.21.15.28:0/1345999931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c19a8a0 0x7fa14c19ec30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.558+0000 7fa1538e9700 1 -- 172.21.15.28:0/1345999931 >> 172.21.15.28:0/1345999931 conn(0x7fa14c076ea0 msgr2=0x7fa14c0ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.559+0000 7fa1538e9700 1 -- 172.21.15.28:0/1345999931 shutdown_connections 2024-04-21T08:42:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:40.559+0000 7fa1538e9700 1 -- 172.21.15.28:0/1345999931 wait complete. 2024-04-21T08:42:41.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:41 smithi028 ceph-mon[28003]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:41.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:41 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:41.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:41 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:41.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:41 smithi028 ceph-mon[28003]: Reconfiguring mon.b (monmap changed)... 2024-04-21T08:42:41.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:41 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:42:41.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:41 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:42:41.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:41 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:41.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:41 smithi028 ceph-mon[28003]: Reconfiguring daemon mon.b on smithi152 2024-04-21T08:42:41.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:41 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:41.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:41 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1345999931' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:41.161 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-21T08:42:41.161 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:42:41.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T08:42:41.188 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:42:41.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:42:41.252 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-21T08:42:41.253 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T08:42:41.280 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-21T08:42:41.280 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:42:41.341 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:41 smithi152 ceph-mon[31401]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:41.342 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:41 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:41.342 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:41 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:41.342 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:41 smithi152 ceph-mon[31401]: Reconfiguring mon.b (monmap changed)... 2024-04-21T08:42:41.342 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:41 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:42:41.342 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:41 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:42:41.342 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:41 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:41.342 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:41 smithi152 ceph-mon[31401]: Reconfiguring daemon mon.b on smithi152 2024-04-21T08:42:41.342 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:41 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:41.342 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:41 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1345999931' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:41.345 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-21T08:42:41.345 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T08:42:41.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:41 smithi171 ceph-mon[31358]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:41 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:41 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:41 smithi171 ceph-mon[31358]: Reconfiguring mon.b (monmap changed)... 2024-04-21T08:42:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:41 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:42:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:41 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:42:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:41 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:41 smithi171 ceph-mon[31358]: Reconfiguring daemon mon.b on smithi152 2024-04-21T08:42:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:41 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:41 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1345999931' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:41.376 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-21T08:42:41.376 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:42:41.444 INFO:tasks.cephadm:Adding mgr.x on smithi028 2024-04-21T08:42:41.444 INFO:tasks.cephadm:Adding mgr.y on smithi152 2024-04-21T08:42:41.444 INFO:tasks.cephadm:Adding mgr.z on smithi171 2024-04-21T08:42:41.444 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi028=x;smithi152=y;smithi171=z' 2024-04-21T08:42:41.717 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:42:42.955 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:42 smithi171 ceph-mon[31358]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:42.955 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:42.955 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:42.955 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:42 smithi171 ceph-mon[31358]: Reconfiguring mon.c (monmap changed)... 2024-04-21T08:42:42.955 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:42:42.956 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:42:42.956 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:42.956 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:42 smithi171 ceph-mon[31358]: Reconfiguring daemon mon.c on smithi171 2024-04-21T08:42:43.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:42 smithi152 ceph-mon[31401]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:43.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:43.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:43.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:42 smithi152 ceph-mon[31401]: Reconfiguring mon.c (monmap changed)... 2024-04-21T08:42:43.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:42:43.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:42:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:42 smithi152 ceph-mon[31401]: Reconfiguring daemon mon.c on smithi171 2024-04-21T08:42:43.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:42 smithi028 ceph-mon[28003]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:43.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:43.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:43.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:42 smithi028 ceph-mon[28003]: Reconfiguring mon.c (monmap changed)... 2024-04-21T08:42:43.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:42:43.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:42:43.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:43.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:42 smithi028 ceph-mon[28003]: Reconfiguring daemon mon.c on smithi171 2024-04-21T08:42:43.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.166+0000 7f25e4d80700 1 -- 172.21.15.171:0/1806386856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e0103030 msgr2=0x7f25e0105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.166+0000 7f25e4d80700 1 --2- 172.21.15.171:0/1806386856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e0103030 0x7f25e0105420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f25d0009910 tx=0x7f25d0009c20 comp rx=0 tx=0).stop 2024-04-21T08:42:43.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.166+0000 7f25e4d80700 1 -- 172.21.15.171:0/1806386856 shutdown_connections 2024-04-21T08:42:43.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.166+0000 7f25e4d80700 1 --2- 172.21.15.171:0/1806386856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e0103030 0x7f25e0105420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.166+0000 7f25e4d80700 1 -- 172.21.15.171:0/1806386856 >> 172.21.15.171:0/1806386856 conn(0x7f25e00fca40 msgr2=0x7f25e00fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:43.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.166+0000 7f25e4d80700 1 -- 172.21.15.171:0/1806386856 shutdown_connections 2024-04-21T08:42:43.166 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.166+0000 7f25e4d80700 1 -- 172.21.15.171:0/1806386856 wait complete. 2024-04-21T08:42:43.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.167+0000 7f25e4d80700 1 Processor -- start 2024-04-21T08:42:43.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.167+0000 7f25e4d80700 1 -- start start 2024-04-21T08:42:43.168 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.167+0000 7f25e4d80700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e0103030 0x7f25e0197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.167+0000 7f25e4d80700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25e01981c0 0x7f25e019d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.167+0000 7f25e4d80700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25e0198640 0x7f25e019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.167+0000 7f25e4d80700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25e0108480 con 0x7f25e0103030 2024-04-21T08:42:43.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.167+0000 7f25e4d80700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f25e0108300 con 0x7f25e0198640 2024-04-21T08:42:43.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.167+0000 7f25e4d80700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f25e0108600 con 0x7f25e01981c0 2024-04-21T08:42:43.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.167+0000 7f25ddd9b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25e01981c0 0x7f25e019d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.167+0000 7f25ddd9b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25e01981c0 0x7f25e019d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:37002/0 (socket says 172.21.15.171:37002) 2024-04-21T08:42:43.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.167+0000 7f25ddd9b700 1 -- 172.21.15.171:0/1628390855 learned_addr learned my addr 172.21.15.171:0/1628390855 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-04-21T08:42:43.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.168+0000 7f25de59c700 1 --2- 172.21.15.171:0/1628390855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e0103030 0x7f25e0197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.168+0000 7f25ded9d700 1 --2- 172.21.15.171:0/1628390855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25e0198640 0x7f25e019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.168+0000 7f25ddd9b700 1 -- 172.21.15.171:0/1628390855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25e0198640 msgr2=0x7f25e019f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.168+0000 7f25ddd9b700 1 --2- 172.21.15.171:0/1628390855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25e0198640 0x7f25e019f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.168+0000 7f25ddd9b700 1 -- 172.21.15.171:0/1628390855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e0103030 msgr2=0x7f25e0197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.169 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.168+0000 7f25ddd9b700 1 --2- 172.21.15.171:0/1628390855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e0103030 0x7f25e0197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.170 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.168+0000 7f25ddd9b700 1 -- 172.21.15.171:0/1628390855 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e01a00d0 con 0x7f25e01981c0 2024-04-21T08:42:43.170 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.168+0000 7f25de59c700 1 --2- 172.21.15.171:0/1628390855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e0103030 0x7f25e0197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:43.170 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.168+0000 7f25ddd9b700 1 --2- 172.21.15.171:0/1628390855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25e01981c0 0x7f25e019d220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f25d400b780 tx=0x7f25d400bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:43.170 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.168+0000 7f25cf7fe700 1 -- 172.21.15.171:0/1628390855 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4004230 con 0x7f25e01981c0 2024-04-21T08:42:43.170 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.168+0000 7f25cf7fe700 1 -- 172.21.15.171:0/1628390855 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25d40043d0 con 0x7f25e01981c0 2024-04-21T08:42:43.170 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.168+0000 7f25e4d80700 1 -- 172.21.15.171:0/1628390855 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e01a03c0 con 0x7f25e01981c0 2024-04-21T08:42:43.170 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.168+0000 7f25cf7fe700 1 -- 172.21.15.171:0/1628390855 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4011bd0 con 0x7f25e01981c0 2024-04-21T08:42:43.170 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.168+0000 7f25e4d80700 1 -- 172.21.15.171:0/1628390855 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e01a0980 con 0x7f25e01981c0 2024-04-21T08:42:43.170 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.169+0000 7f25cf7fe700 1 -- 172.21.15.171:0/1628390855 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f25d401a400 con 0x7f25e01981c0 2024-04-21T08:42:43.171 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.169+0000 7f25cf7fe700 1 --2- 172.21.15.171:0/1628390855 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f25c8039ae0 0x7f25c803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.171 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.169+0000 7f25e4d80700 1 -- 172.21.15.171:0/1628390855 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25e00640a0 con 0x7f25e01981c0 2024-04-21T08:42:43.171 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.169+0000 7f25cf7fe700 1 -- 172.21.15.171:0/1628390855 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f25d404c430 con 0x7f25e01981c0 2024-04-21T08:42:43.171 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.169+0000 7f25de59c700 1 --2- 172.21.15.171:0/1628390855 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f25c8039ae0 0x7f25c803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.171 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.170+0000 7f25de59c700 1 --2- 172.21.15.171:0/1628390855 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f25c8039ae0 0x7f25c803bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f25d000b5c0 tx=0x7f25d001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:43.174 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.174+0000 7f25cf7fe700 1 -- 172.21.15.171:0/1628390855 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f25d4014160 con 0x7f25e01981c0 2024-04-21T08:42:43.391 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.391+0000 7f25e4d80700 1 -- 172.21.15.171:0/1628390855 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi152=y;smithi171=z", "target": ["mon-mgr", ""]}) -- 0x7f25e0062e80 con 0x7f25c8039ae0 2024-04-21T08:42:43.397 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.398+0000 7f25cf7fe700 1 -- 172.21.15.171:0/1628390855 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f25e0062e80 con 0x7f25c8039ae0 2024-04-21T08:42:43.398 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled mgr update... 2024-04-21T08:42:43.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.400+0000 7f25e4d80700 1 -- 172.21.15.171:0/1628390855 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f25c8039ae0 msgr2=0x7f25c803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.400+0000 7f25e4d80700 1 --2- 172.21.15.171:0/1628390855 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f25c8039ae0 0x7f25c803bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f25d000b5c0 tx=0x7f25d001a040 comp rx=0 tx=0).stop 2024-04-21T08:42:43.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.400+0000 7f25e4d80700 1 -- 172.21.15.171:0/1628390855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25e01981c0 msgr2=0x7f25e019d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.400 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.400+0000 7f25e4d80700 1 --2- 172.21.15.171:0/1628390855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25e01981c0 0x7f25e019d220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f25d400b780 tx=0x7f25d400bb40 comp rx=0 tx=0).stop 2024-04-21T08:42:43.401 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.400+0000 7f25e4d80700 1 -- 172.21.15.171:0/1628390855 shutdown_connections 2024-04-21T08:42:43.401 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.400+0000 7f25e4d80700 1 --2- 172.21.15.171:0/1628390855 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f25c8039ae0 0x7f25c803bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.401 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.400+0000 7f25e4d80700 1 --2- 172.21.15.171:0/1628390855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e0103030 0x7f25e0197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.401 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.400+0000 7f25e4d80700 1 --2- 172.21.15.171:0/1628390855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25e0198640 0x7f25e019f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.401 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.400+0000 7f25e4d80700 1 --2- 172.21.15.171:0/1628390855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f25e01981c0 0x7f25e019d220 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.401 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.400+0000 7f25e4d80700 1 -- 172.21.15.171:0/1628390855 >> 172.21.15.171:0/1628390855 conn(0x7f25e00fca40 msgr2=0x7f25e00fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:43.401 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.401+0000 7f25e4d80700 1 -- 172.21.15.171:0/1628390855 shutdown_connections 2024-04-21T08:42:43.401 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:42:43.401+0000 7f25e4d80700 1 -- 172.21.15.171:0/1628390855 wait complete. 2024-04-21T08:42:44.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:44 smithi171 ceph-mon[31358]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:44.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:44 smithi171 ceph-mon[31358]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi152=y;smithi171=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:44.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:44 smithi171 ceph-mon[31358]: Saving service mgr spec with placement smithi028=x;smithi152=y;smithi171=z;count:3 2024-04-21T08:42:44.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:44 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:44.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:44 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:44.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:44 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:44.761 DEBUG:teuthology.orchestra.run.smithi152:mgr.y> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mgr.y.service 2024-04-21T08:42:44.764 DEBUG:teuthology.orchestra.run.smithi171:mgr.z> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mgr.z.service 2024-04-21T08:42:44.766 INFO:tasks.cephadm:Deploying OSDs... 2024-04-21T08:42:44.766 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:42:44.766 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T08:42:44.782 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T08:42:44.782 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-04-21T08:42:44.786 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:44 smithi152 ceph-mon[31401]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:44.787 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:44 smithi152 ceph-mon[31401]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi152=y;smithi171=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:44.787 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:44 smithi152 ceph-mon[31401]: Saving service mgr spec with placement smithi028=x;smithi152=y;smithi171=z;count:3 2024-04-21T08:42:44.787 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:44 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:44.787 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:44 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:44.787 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:44 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:44.837 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T08:42:44.837 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:42:44.837 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-21T08:42:44.837 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:42:44.837 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:42:44.837 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-21 08:41:48.329947279 +0000 2024-04-21T08:42:44.837 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-21 08:36:24.700847791 +0000 2024-04-21T08:42:44.837 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-21 08:36:24.700847791 +0000 2024-04-21T08:42:44.837 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-21 08:36:24.700847791 +0000 2024-04-21T08:42:44.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T08:42:44.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:44 smithi028 ceph-mon[28003]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:44.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:44 smithi028 ceph-mon[28003]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi152=y;smithi171=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:44.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:44 smithi028 ceph-mon[28003]: Saving service mgr spec with placement smithi028=x;smithi152=y;smithi171=z;count:3 2024-04-21T08:42:44.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:44 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:44.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:44 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:44.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:44 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:44.877 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-21T08:42:44.878 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-21T08:42:44.878 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 9.7779e-05 s, 5.2 MB/s 2024-04-21T08:42:44.879 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T08:42:44.936 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-04-21T08:42:44.991 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T08:42:44.991 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:42:44.991 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-21T08:42:44.991 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:42:44.991 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:42:44.991 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-21 08:41:48.362946317 +0000 2024-04-21T08:42:44.991 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-21 08:36:25.041837856 +0000 2024-04-21T08:42:44.991 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-21 08:36:25.041837856 +0000 2024-04-21T08:42:44.991 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-21 08:36:25.041837856 +0000 2024-04-21T08:42:44.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T08:42:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-21T08:42:45.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-21T08:42:45.056 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 9.5276e-05 s, 5.4 MB/s 2024-04-21T08:42:45.057 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T08:42:45.113 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-04-21T08:42:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T08:42:45.169 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:42:45.169 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-21T08:42:45.169 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:42:45.169 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:42:45.169 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-21 08:41:48.393945414 +0000 2024-04-21T08:42:45.170 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-21 08:36:25.357828652 +0000 2024-04-21T08:42:45.170 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-21 08:36:25.357828652 +0000 2024-04-21T08:42:45.170 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-21 08:36:25.357828652 +0000 2024-04-21T08:42:45.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T08:42:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-21T08:42:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-21T08:42:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000102316 s, 5.0 MB/s 2024-04-21T08:42:45.233 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T08:42:45.289 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-04-21T08:42:45.345 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T08:42:45.345 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:42:45.345 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-21T08:42:45.345 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:42:45.345 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:42:45.346 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-21 08:41:48.424944510 +0000 2024-04-21T08:42:45.346 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-21 08:36:25.677819330 +0000 2024-04-21T08:42:45.346 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-21 08:36:25.677819330 +0000 2024-04-21T08:42:45.346 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-21 08:36:25.677819330 +0000 2024-04-21T08:42:45.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T08:42:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-21T08:42:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-21T08:42:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000110937 s, 4.6 MB/s 2024-04-21T08:42:45.409 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T08:42:45.466 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-21T08:42:45.466 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T08:42:45.485 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T08:42:45.485 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2024-04-21T08:42:45.543 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T08:42:45.543 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:42:45.543 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-21T08:42:45.543 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:42:45.543 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:42:45.543 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-04-21 08:42:29.826016605 +0000 2024-04-21T08:42:45.543 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-04-21 08:36:38.296679779 +0000 2024-04-21T08:42:45.543 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-04-21 08:36:38.296679779 +0000 2024-04-21T08:42:45.543 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-04-21 08:36:38.296679779 +0000 2024-04-21T08:42:45.544 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T08:42:45.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:45.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:45.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:45.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:45.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:42:45.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:42:45.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:42:45.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:45.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:45 smithi028 ceph-mon[28003]: Deploying daemon mgr.z on smithi171 2024-04-21T08:42:45.608 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-04-21T08:42:45.609 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-04-21T08:42:45.609 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000284976 s, 1.8 MB/s 2024-04-21T08:42:45.610 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T08:42:45.668 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2024-04-21T08:42:45.702 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:45.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:45.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:45.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:45.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:42:45.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:42:45.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:42:45.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:45.703 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:45 smithi171 ceph-mon[31358]: Deploying daemon mgr.z on smithi171 2024-04-21T08:42:45.724 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T08:42:45.724 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:42:45.724 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-21T08:42:45.725 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:42:45.725 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:42:45.725 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-04-21 08:42:29.827016593 +0000 2024-04-21T08:42:45.725 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-04-21 08:36:38.631675965 +0000 2024-04-21T08:42:45.725 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-04-21 08:36:38.631675965 +0000 2024-04-21T08:42:45.725 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-04-21 08:36:38.631675965 +0000 2024-04-21T08:42:45.725 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T08:42:45.786 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:45.786 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:45.786 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:45.787 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:45.787 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:42:45.787 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:42:45.787 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:42:45.787 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:45.787 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:45 smithi152 ceph-mon[31401]: Deploying daemon mgr.z on smithi171 2024-04-21T08:42:45.789 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-04-21T08:42:45.789 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-04-21T08:42:45.789 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000254948 s, 2.0 MB/s 2024-04-21T08:42:45.790 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T08:42:45.847 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2024-04-21T08:42:45.903 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T08:42:45.903 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:42:45.903 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-21T08:42:45.903 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:42:45.903 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:42:45.903 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-04-21 08:42:29.827016593 +0000 2024-04-21T08:42:45.904 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-04-21 08:36:38.967672139 +0000 2024-04-21T08:42:45.904 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-04-21 08:36:38.967672139 +0000 2024-04-21T08:42:45.904 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-04-21 08:36:38.967672139 +0000 2024-04-21T08:42:45.904 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T08:42:45.967 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-04-21T08:42:45.967 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-04-21T08:42:45.967 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.00026858 s, 1.9 MB/s 2024-04-21T08:42:45.969 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T08:42:46.025 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2024-04-21T08:42:46.081 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T08:42:46.081 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:42:46.081 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-21T08:42:46.081 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:42:46.081 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:42:46.082 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-04-21 08:42:29.827016593 +0000 2024-04-21T08:42:46.082 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-04-21 08:36:39.305668291 +0000 2024-04-21T08:42:46.082 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-04-21 08:36:39.305668291 +0000 2024-04-21T08:42:46.082 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-04-21 08:36:39.305668291 +0000 2024-04-21T08:42:46.082 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T08:42:46.144 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-04-21T08:42:46.144 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-04-21T08:42:46.144 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000189141 s, 2.7 MB/s 2024-04-21T08:42:46.145 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T08:42:46.201 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-04-21T08:42:46.201 DEBUG:teuthology.orchestra.run.smithi171:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T08:42:46.217 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T08:42:46.217 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_1 2024-04-21T08:42:46.271 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T08:42:46.272 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:42:46.272 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-21T08:42:46.272 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:42:46.272 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:42:46.272 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-04-21 08:42:20.478821545 +0000 2024-04-21T08:42:46.272 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-04-21 08:36:53.128800420 +0000 2024-04-21T08:42:46.272 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-04-21 08:36:53.128800420 +0000 2024-04-21T08:42:46.272 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-04-21 08:36:53.128800420 +0000 2024-04-21T08:42:46.272 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T08:42:46.335 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-04-21T08:42:46.336 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-04-21T08:42:46.336 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000268808 s, 1.9 MB/s 2024-04-21T08:42:46.337 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T08:42:46.393 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_2 2024-04-21T08:42:46.450 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T08:42:46.450 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:42:46.450 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-21T08:42:46.450 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:42:46.450 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:42:46.450 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-04-21 08:42:20.479821524 +0000 2024-04-21T08:42:46.450 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-04-21 08:36:53.488792915 +0000 2024-04-21T08:42:46.450 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-04-21 08:36:53.488792915 +0000 2024-04-21T08:42:46.450 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-04-21 08:36:53.488792915 +0000 2024-04-21T08:42:46.451 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T08:42:46.513 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-04-21T08:42:46.514 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-04-21T08:42:46.514 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000258203 s, 2.0 MB/s 2024-04-21T08:42:46.515 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T08:42:46.572 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_3 2024-04-21T08:42:46.627 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T08:42:46.628 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:42:46.628 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-21T08:42:46.628 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:42:46.628 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:42:46.628 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-04-21 08:42:20.479821524 +0000 2024-04-21T08:42:46.628 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-04-21 08:36:53.823785929 +0000 2024-04-21T08:42:46.628 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-04-21 08:36:53.823785929 +0000 2024-04-21T08:42:46.628 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-04-21 08:36:53.823785929 +0000 2024-04-21T08:42:46.628 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T08:42:46.825 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-04-21T08:42:46.825 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-04-21T08:42:46.825 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000230259 s, 2.2 MB/s 2024-04-21T08:42:46.826 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T08:42:46.841 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_4 2024-04-21T08:42:46.899 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T08:42:46.899 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:42:46.900 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-21T08:42:46.900 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:42:46.900 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:42:46.900 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-04-21 08:42:46.779286653 +0000 2024-04-21T08:42:46.900 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-04-21 08:36:54.169778713 +0000 2024-04-21T08:42:46.900 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-04-21 08:36:54.169778713 +0000 2024-04-21T08:42:46.900 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: 2024-04-21 08:36:54.169778713 +0000 2024-04-21T08:42:46.900 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T08:42:46.999 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-04-21T08:42:46.999 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-04-21T08:42:47.000 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000216624 s, 2.4 MB/s 2024-04-21T08:42:47.001 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T08:42:47.017 INFO:tasks.cephadm:Deploying osd.0 on smithi028 with /dev/vg_nvme/lv_4... 2024-04-21T08:42:47.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T08:42:47.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:47 smithi171 ceph-mon[31358]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:47.272 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:47.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:47 smithi152 ceph-mon[31401]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:47.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:47 smithi028 ceph-mon[28003]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:47.541 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:47 smithi171 ceph-mgr[33253]: -- 172.21.15.171:0/828975245 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555cc03b85a0 con 0x555cc038c800 2024-04-21T08:42:47.798 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:47 smithi171 bash[33233]: 0b190d088b4173e6c5e84279d6e799e6faeba1b87bfecedc3dc868ec2bbc392b 2024-04-21T08:42:47.798 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:47 smithi171 systemd[1]: Started Ceph mgr.z for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:42:47.798 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:47 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:47.613+0000 7f01483ad200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:42:47.798 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:47 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:47.797+0000 7f01483ad200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:42:48.120 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:47 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:47.900+0000 7f01483ad200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:42:48.614 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:48 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:48.210+0000 7f01483ad200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:42:48.614 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:48 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:48.294+0000 7f01483ad200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:42:48.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:48 smithi152 ceph-mon[31401]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:48.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:48 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:48.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:48 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:48.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:48 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:48.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:48 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:42:48.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:48 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:42:48.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:48 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:42:48.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:48 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:48.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:48 smithi152 ceph-mon[31401]: Deploying daemon mgr.y on smithi152 2024-04-21T08:42:48.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:48 smithi028 ceph-mon[28003]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:48.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:48.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:48.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:48.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:42:48.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:42:48.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:42:48.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:48.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:48 smithi028 ceph-mon[28003]: Deploying daemon mgr.y on smithi152 2024-04-21T08:42:48.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:48 smithi171 ceph-mon[31358]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:48.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:48 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:48.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:48 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:48.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:48 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:48.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:48 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:42:48.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:48 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:42:48.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:48 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:42:48.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:48 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:48.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:48 smithi171 ceph-mon[31358]: Deploying daemon mgr.y on smithi152 2024-04-21T08:42:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T08:42:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T08:42:49.113 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-21T08:42:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-21T08:42:49.113 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00845847 s, 1.2 GB/s 2024-04-21T08:42:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-21T08:42:49.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-04-21T08:42:49.839 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:49 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:49.500+0000 7f01483ad200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:42:49.840 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:49 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:49.668+0000 7f01483ad200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:42:49.840 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:49 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:49.839+0000 7f01483ad200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:42:49.854 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:49 smithi152 podman[32773]: 2024-04-21 08:42:49.70497486 +0000 UTC m=+0.017952314 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:42:49.995 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:50.331 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:49 smithi152 podman[32773]: 2024-04-21 08:42:49.986811464 +0000 UTC m=+0.299788926 container init aceaaf35e2108ba1c7e1625724e782c6713066a7f78d91fbc673ecbb2f403d31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-21T08:42:50.332 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:49 smithi152 podman[32773]: 2024-04-21 08:42:49.989203936 +0000 UTC m=+0.302181390 container start aceaaf35e2108ba1c7e1625724e782c6713066a7f78d91fbc673ecbb2f403d31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-21T08:42:50.332 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:50 smithi152 ceph-mgr[32793]: -- 172.21.15.152:0/3612275536 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a998ad45a0 con 0x55a998aa8c00 2024-04-21T08:42:50.332 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:50 smithi152 bash[32773]: aceaaf35e2108ba1c7e1625724e782c6713066a7f78d91fbc673ecbb2f403d31 2024-04-21T08:42:50.332 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:50 smithi152 systemd[1]: Started Ceph mgr.y for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:42:50.332 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:50 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:50.143+0000 7fec74e67200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:42:50.332 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:50 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:50.331+0000 7fec74e67200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:42:50.433 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:50 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:50.126+0000 7f01483ad200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:42:50.433 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:50 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:50.218+0000 7f01483ad200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:42:50.433 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:50 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:50.433+0000 7f01483ad200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:42:50.604 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:50 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:50.441+0000 7fec74e67200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:42:50.869 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:50 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:50.629+0000 7f01483ad200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:42:51.056 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:50 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:50.770+0000 7fec74e67200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:42:51.057 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:50 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:50.868+0000 7fec74e67200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:42:51.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:51 smithi028 ceph-mon[28003]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:51.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:51.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:51.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:51.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:51.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:51.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:51 smithi152 ceph-mon[31401]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:51.370 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:51 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:51.271+0000 7f01483ad200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:42:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:51 smithi171 ceph-mon[31358]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:51.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.540+0000 7fcd6e320700 1 -- 172.21.15.28:0/183227559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd68108830 msgr2=0x7fcd6810ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.540+0000 7fcd6e320700 1 --2- 172.21.15.28:0/183227559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd68108830 0x7fcd6810ac20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000b0a0 tx=0x7fcd6000b3b0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.540+0000 7fcd6e320700 1 -- 172.21.15.28:0/183227559 shutdown_connections 2024-04-21T08:42:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.540+0000 7fcd6e320700 1 --2- 172.21.15.28:0/183227559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd68108830 0x7fcd6810ac20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.540+0000 7fcd6e320700 1 --2- 172.21.15.28:0/183227559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd681035d0 0x7fcd681059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.540+0000 7fcd6e320700 1 --2- 172.21.15.28:0/183227559 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd68105f00 0x7fcd681082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.540+0000 7fcd6e320700 1 -- 172.21.15.28:0/183227559 >> 172.21.15.28:0/183227559 conn(0x7fcd680fcf40 msgr2=0x7fcd680ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.540+0000 7fcd6e320700 1 -- 172.21.15.28:0/183227559 shutdown_connections 2024-04-21T08:42:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.540+0000 7fcd6e320700 1 -- 172.21.15.28:0/183227559 wait complete. 2024-04-21T08:42:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.541+0000 7fcd6e320700 1 Processor -- start 2024-04-21T08:42:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.541+0000 7fcd6e320700 1 -- start start 2024-04-21T08:42:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.541+0000 7fcd6e320700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd681035d0 0x7fcd68102f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.541+0000 7fcd6e320700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd68105f00 0x7fcd68101580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.541+0000 7fcd6e320700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd68108830 0x7fcd68101ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.541+0000 7fcd6e320700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcd6810e550 con 0x7fcd681035d0 2024-04-21T08:42:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.541+0000 7fcd6e320700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fcd6810e3d0 con 0x7fcd68105f00 2024-04-21T08:42:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.541+0000 7fcd6e320700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcd6810e6d0 con 0x7fcd68108830 2024-04-21T08:42:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.541+0000 7fcd67fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd681035d0 0x7fcd68102f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.541+0000 7fcd67fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd681035d0 0x7fcd68102f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41132/0 (socket says 172.21.15.28:41132) 2024-04-21T08:42:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.541+0000 7fcd67fff700 1 -- 172.21.15.28:0/2619012070 learned_addr learned my addr 172.21.15.28:0/2619012070 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:42:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.541+0000 7fcd5ffff700 1 --2- 172.21.15.28:0/2619012070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd68105f00 0x7fcd68101580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.541+0000 7fcd6c8bd700 1 --2- 172.21.15.28:0/2619012070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd68108830 0x7fcd68101ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.542+0000 7fcd67fff700 1 -- 172.21.15.28:0/2619012070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd68108830 msgr2=0x7fcd68101ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.542+0000 7fcd67fff700 1 --2- 172.21.15.28:0/2619012070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd68108830 0x7fcd68101ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.542+0000 7fcd67fff700 1 -- 172.21.15.28:0/2619012070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd68105f00 msgr2=0x7fcd68101580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.542+0000 7fcd67fff700 1 --2- 172.21.15.28:0/2619012070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd68105f00 0x7fcd68101580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.542+0000 7fcd67fff700 1 -- 172.21.15.28:0/2619012070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd68102370 con 0x7fcd681035d0 2024-04-21T08:42:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.542+0000 7fcd67fff700 1 --2- 172.21.15.28:0/2619012070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd681035d0 0x7fcd68102f30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcd5800d980 tx=0x7fcd5800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.542+0000 7fcd65ffb700 1 -- 172.21.15.28:0/2619012070 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd58016070 con 0x7fcd681035d0 2024-04-21T08:42:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.542+0000 7fcd65ffb700 1 -- 172.21.15.28:0/2619012070 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd58004830 con 0x7fcd681035d0 2024-04-21T08:42:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.542+0000 7fcd6e320700 1 -- 172.21.15.28:0/2619012070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd681a2db0 con 0x7fcd681035d0 2024-04-21T08:42:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.542+0000 7fcd65ffb700 1 -- 172.21.15.28:0/2619012070 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd58005540 con 0x7fcd681035d0 2024-04-21T08:42:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.542+0000 7fcd6e320700 1 -- 172.21.15.28:0/2619012070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd681a3200 con 0x7fcd681035d0 2024-04-21T08:42:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.543+0000 7fcd65ffb700 1 -- 172.21.15.28:0/2619012070 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fcd58004220 con 0x7fcd681035d0 2024-04-21T08:42:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.543+0000 7fcd6e320700 1 -- 172.21.15.28:0/2619012070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd68069730 con 0x7fcd681035d0 2024-04-21T08:42:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.544+0000 7fcd65ffb700 1 --2- 172.21.15.28:0/2619012070 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fcd54039890 0x7fcd5403bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.544+0000 7fcd65ffb700 1 -- 172.21.15.28:0/2619012070 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fcd5804c770 con 0x7fcd681035d0 2024-04-21T08:42:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.544+0000 7fcd5ffff700 1 --2- 172.21.15.28:0/2619012070 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fcd54039890 0x7fcd5403bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.544+0000 7fcd5ffff700 1 --2- 172.21.15.28:0/2619012070 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fcd54039890 0x7fcd5403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcd50005950 tx=0x7fcd5000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.549+0000 7fcd65ffb700 1 -- 172.21.15.28:0/2619012070 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcd5801f020 con 0x7fcd681035d0 2024-04-21T08:42:51.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:51.773+0000 7fcd6e320700 1 -- 172.21.15.28:0/2619012070 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fcd68062e80 con 0x7fcd54039890 2024-04-21T08:42:52.008 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:51 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:51.708+0000 7f01483ad200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:42:52.008 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:51 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:51.806+0000 7f01483ad200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:42:52.008 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:52.008+0000 7f01483ad200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:42:52.354 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:52.104+0000 7fec74e67200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:42:52.354 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:52.274+0000 7fec74e67200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:42:52.708 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:52.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:52.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:52.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:52.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:52.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:52.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:52.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:52.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T08:42:52.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:52.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:42:52.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:42:52.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:42:52.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:52.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:42:52.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:52.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:52 smithi028 ceph-mon[28003]: Reconfiguring daemon mgr.x on smithi028 2024-04-21T08:42:52.731 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:52.446+0000 7fec74e67200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:42:52.731 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:52.731+0000 7fec74e67200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:42:52.731 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:52.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:52.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:52.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:52.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:52.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:52.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:52.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:52.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T08:42:52.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:52.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:42:52.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:42:52.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:42:52.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:52.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:42:52.732 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:52.733 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-mon[31401]: Reconfiguring daemon mgr.x on smithi028 2024-04-21T08:42:52.870 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:52.546+0000 7f01483ad200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:42:52.870 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:52.638+0000 7f01483ad200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:42:52.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:52.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:52.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:52.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:52.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:52.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:52.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:52.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:52.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T08:42:52.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:52.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:42:52.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:42:52.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:42:52.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:52.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:42:52.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:52.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-mon[31358]: Reconfiguring daemon mgr.x on smithi028 2024-04-21T08:42:53.027 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:52 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:52.824+0000 7fec74e67200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:42:53.027 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:53 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:53.028+0000 7fec74e67200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:42:53.227 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:52 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:52.883+0000 7f01483ad200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:42:53.227 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:53 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:53.014+0000 7f01483ad200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:42:53.227 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:53 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:53.110+0000 7f01483ad200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:42:53.227 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:53 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:53.227+0000 7f01483ad200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:42:53.354 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:53 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:53.218+0000 7fec74e67200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:42:53.869 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:53 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:53.556+0000 7f01483ad200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:42:54.104 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:53 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:53.816+0000 7fec74e67200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:42:54.535 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:54 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:54.241+0000 7fec74e67200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:42:54.535 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:54 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:54.336+0000 7fec74e67200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:42:54.535 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:54 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:54.535+0000 7fec74e67200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:42:54.620 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:54 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:54.151+0000 7f01483ad200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:42:54.620 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:42:54 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:42:54.368+0000 7f01483ad200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:42:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:55.108+0000 7fcd65ffb700 1 -- 172.21.15.28:0/2619012070 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7fcd58003a00 con 0x7fcd681035d0 2024-04-21T08:42:55.354 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:55.081+0000 7fec74e67200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:42:55.354 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:55.173+0000 7fec74e67200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:42:55.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-mon[31401]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:55.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:55.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:55.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:55.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:55.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:55.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:55.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-mon[31401]: Standby manager daemon z started 2024-04-21T08:42:55.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.171:0/1146495111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:42:55.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.171:0/1146495111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:42:55.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.171:0/1146495111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:42:55.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.171:0/1146495111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:42:55.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:55 smithi028 ceph-mon[28003]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:55.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:55.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:55.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:55.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:55.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:55.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:55.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:55 smithi028 ceph-mon[28003]: Standby manager daemon z started 2024-04-21T08:42:55.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:55 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.171:0/1146495111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:42:55.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:55 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.171:0/1146495111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:42:55.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:55 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.171:0/1146495111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:42:55.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:55 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.171:0/1146495111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:42:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:55 smithi171 ceph-mon[31358]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:55.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:55.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:55.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:42:55.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:55 smithi171 ceph-mon[31358]: Standby manager daemon z started 2024-04-21T08:42:55.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:55 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.171:0/1146495111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:42:55.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:55 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.171:0/1146495111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:42:55.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:55 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.171:0/1146495111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:42:55.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:55 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.171:0/1146495111' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:42:55.764 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:55.417+0000 7fec74e67200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:42:55.764 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:55.550+0000 7fec74e67200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:42:55.764 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:55.648+0000 7fec74e67200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:42:55.764 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:55 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:55.764+0000 7fec74e67200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:42:56.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:56 smithi028 ceph-mon[28003]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-21T08:42:56.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:56 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:42:56.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:56 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/20057862' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b82ad926-994c-4e96-90d8-585391ba5e7f"}]: dispatch 2024-04-21T08:42:56.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:56 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/20057862' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b82ad926-994c-4e96-90d8-585391ba5e7f"}]': finished 2024-04-21T08:42:56.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:56 smithi028 ceph-mon[28003]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T08:42:56.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:56 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:42:56.354 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:56 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:56.098+0000 7fec74e67200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:42:56.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:56 smithi152 ceph-mon[31401]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-21T08:42:56.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:56 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:42:56.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:56 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/20057862' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b82ad926-994c-4e96-90d8-585391ba5e7f"}]: dispatch 2024-04-21T08:42:56.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:56 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/20057862' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b82ad926-994c-4e96-90d8-585391ba5e7f"}]': finished 2024-04-21T08:42:56.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:56 smithi152 ceph-mon[31401]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T08:42:56.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:56 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:42:56.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:56 smithi171 ceph-mon[31358]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-21T08:42:56.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:56 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:42:56.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:56 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/20057862' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b82ad926-994c-4e96-90d8-585391ba5e7f"}]: dispatch 2024-04-21T08:42:56.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:56 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/20057862' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b82ad926-994c-4e96-90d8-585391ba5e7f"}]': finished 2024-04-21T08:42:56.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:56 smithi171 ceph-mon[31358]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T08:42:56.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:56 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:42:57.104 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:56 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:56.695+0000 7fec74e67200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:42:57.104 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:42:56 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:42:56.907+0000 7fec74e67200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:42:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:42:57.122+0000 7fcd65ffb700 1 -- 172.21.15.28:0/2619012070 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fcd58008230 con 0x7fcd681035d0 2024-04-21T08:42:57.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:57 smithi171 ceph-mon[31358]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:57.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:57 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2492841681' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:42:57.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:57 smithi171 ceph-mon[31358]: Standby manager daemon y started 2024-04-21T08:42:57.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:57 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.152:0/4133600667' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:42:57.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:57 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.152:0/4133600667' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:42:57.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:57 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.152:0/4133600667' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:42:57.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:57 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.152:0/4133600667' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:42:57.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:57 smithi028 ceph-mon[28003]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:57.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:57 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2492841681' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:42:57.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:57 smithi028 ceph-mon[28003]: Standby manager daemon y started 2024-04-21T08:42:57.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:57 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.152:0/4133600667' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:42:57.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:57 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.152:0/4133600667' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:42:57.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:57 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.152:0/4133600667' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:42:57.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:57 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.152:0/4133600667' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:42:57.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:57 smithi152 ceph-mon[31401]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:57.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:57 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2492841681' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:42:57.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:57 smithi152 ceph-mon[31401]: Standby manager daemon y started 2024-04-21T08:42:57.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:57 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.152:0/4133600667' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:42:57.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:57 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.152:0/4133600667' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:42:57.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:57 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.152:0/4133600667' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:42:57.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:57 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.152:0/4133600667' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:42:58.375 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:58 smithi152 ceph-mon[31401]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-04-21T08:42:58.375 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:58 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:42:58.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:58 smithi028 ceph-mon[28003]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-04-21T08:42:58.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:58 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:42:58.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:58 smithi171 ceph-mon[31358]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-04-21T08:42:58.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:58 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:42:59.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:42:59 smithi152 ceph-mon[31401]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:59.549 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:42:59 smithi028 ceph-mon[28003]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:42:59.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:42:59 smithi171 ceph-mon[31358]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:01.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:01 smithi028 ceph-mon[28003]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:01.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:01 smithi152 ceph-mon[31401]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:01.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:01 smithi171 ceph-mon[31358]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:03.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:03 smithi028 ceph-mon[28003]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:03.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:03 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T08:43:03.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:03 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:03.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:03 smithi152 ceph-mon[31401]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:03.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:03 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T08:43:03.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:03 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:03.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:03 smithi171 ceph-mon[31358]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:03.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:03 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T08:43:03.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:03 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:04.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:04 smithi028 ceph-mon[28003]: Deploying daemon osd.0 on smithi028 2024-04-21T08:43:04.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:04 smithi152 ceph-mon[31401]: Deploying daemon osd.0 on smithi028 2024-04-21T08:43:04.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:04 smithi171 ceph-mon[31358]: Deploying daemon osd.0 on smithi028 2024-04-21T08:43:05.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:05 smithi028 ceph-mon[28003]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:05.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:05 smithi152 ceph-mon[31401]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:05.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:05 smithi171 ceph-mon[31358]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:07.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:07 smithi028 ceph-mon[28003]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:07.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:07 smithi152 ceph-mon[31401]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:07.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:07 smithi171 ceph-mon[31358]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:09.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:09 smithi028 ceph-mon[28003]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:09.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:09 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:09.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:09 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:09.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:09 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:09.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:09 smithi152 ceph-mon[31401]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:09.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:09 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:09.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:09 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:09.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:09 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:09.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:09 smithi171 ceph-mon[31358]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:09.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:09 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:09.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:09 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:09.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:09 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:11.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:11 smithi028 ceph-mon[28003]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:11.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:11 smithi152 ceph-mon[31401]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:11.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:11 smithi171 ceph-mon[31358]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:12.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:12 smithi028 ceph-mon[28003]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:12.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:12.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:12.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:12.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:12.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:12.535 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 0 on host 'smithi028' 2024-04-21T08:43:12.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:43:12.535+0000 7fcd65ffb700 1 -- 172.21.15.28:0/2619012070 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcd68062e80 con 0x7fcd54039890 2024-04-21T08:43:12.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:43:12.538+0000 7fcd6e320700 1 -- 172.21.15.28:0/2619012070 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fcd54039890 msgr2=0x7fcd5403bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:43:12.538+0000 7fcd6e320700 1 --2- 172.21.15.28:0/2619012070 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fcd54039890 0x7fcd5403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcd50005950 tx=0x7fcd5000b410 comp rx=0 tx=0).stop 2024-04-21T08:43:12.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:43:12.538+0000 7fcd6e320700 1 -- 172.21.15.28:0/2619012070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd681035d0 msgr2=0x7fcd68102f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:43:12.538+0000 7fcd6e320700 1 --2- 172.21.15.28:0/2619012070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd681035d0 0x7fcd68102f30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcd5800d980 tx=0x7fcd5800dc90 comp rx=0 tx=0).stop 2024-04-21T08:43:12.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:43:12.538+0000 7fcd6e320700 1 -- 172.21.15.28:0/2619012070 shutdown_connections 2024-04-21T08:43:12.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:43:12.538+0000 7fcd6e320700 1 --2- 172.21.15.28:0/2619012070 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fcd54039890 0x7fcd5403bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:43:12.538+0000 7fcd6e320700 1 --2- 172.21.15.28:0/2619012070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd681035d0 0x7fcd68102f30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:43:12.538+0000 7fcd6e320700 1 --2- 172.21.15.28:0/2619012070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd68108830 0x7fcd68101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:43:12.538+0000 7fcd6e320700 1 --2- 172.21.15.28:0/2619012070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd68105f00 0x7fcd68101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:43:12.538+0000 7fcd6e320700 1 -- 172.21.15.28:0/2619012070 >> 172.21.15.28:0/2619012070 conn(0x7fcd680fcf40 msgr2=0x7fcd68103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:12.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:43:12.538+0000 7fcd6e320700 1 -- 172.21.15.28:0/2619012070 shutdown_connections 2024-04-21T08:43:12.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:43:12.538+0000 7fcd6e320700 1 -- 172.21.15.28:0/2619012070 wait complete. 2024-04-21T08:43:12.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:12 smithi152 ceph-mon[31401]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:12.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:12.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:12.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:12.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:12.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:12.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:12 smithi171 ceph-mon[31358]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:12.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:12.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:12.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:12.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:12.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:13.303 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.0.service 2024-04-21T08:43:13.306 INFO:tasks.cephadm:Deploying osd.1 on smithi152 with /dev/vg_nvme/lv_4... 2024-04-21T08:43:13.307 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T08:43:13.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:13 smithi028 ceph-mon[28003]: from='osd.0 [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T08:43:13.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:13 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:13.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:13 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:13.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:13 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:13.553 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:43:13.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:13 smithi152 ceph-mon[31401]: from='osd.0 [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T08:43:13.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:13 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:13.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:13 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:13.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:13 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:13.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:13 smithi171 ceph-mon[31358]: from='osd.0 [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T08:43:13.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:13 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:13.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:13 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:13.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:13 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:14.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:14 smithi028 ceph-mon[28003]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:14.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:14 smithi028 ceph-mon[28003]: from='osd.0 [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T08:43:14.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:14 smithi028 ceph-mon[28003]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T08:43:14.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:14 smithi028 ceph-mon[28003]: from='osd.0 [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-21T08:43:14.549 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:43:14.549 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 08:43:14 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-0[43046]: 2024-04-21T08:43:14.418+0000 7fa90e381700 -1 osd.0 0 waiting for initial osdmap 2024-04-21T08:43:14.549 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 08:43:14 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-0[43046]: 2024-04-21T08:43:14.421+0000 7fa90a174700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:43:14.595 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:14 smithi152 ceph-mon[31401]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:14.595 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:14 smithi152 ceph-mon[31401]: from='osd.0 [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T08:43:14.596 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:14 smithi152 ceph-mon[31401]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T08:43:14.596 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:14 smithi152 ceph-mon[31401]: from='osd.0 [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-21T08:43:14.596 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:14 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:43:14.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:14 smithi171 ceph-mon[31358]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:14.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:14 smithi171 ceph-mon[31358]: from='osd.0 [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T08:43:14.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:14 smithi171 ceph-mon[31358]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T08:43:14.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:14 smithi171 ceph-mon[31358]: from='osd.0 [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-21T08:43:14.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:14 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:43:15.389 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T08:43:15.389 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T08:43:15.389 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-04-21T08:43:15.389 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-04-21T08:43:15.389 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115292 s, 909 MB/s 2024-04-21T08:43:15.389 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-04-21T08:43:15.689 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:15 smithi152 ceph-mon[31401]: purged_snaps scrub starts 2024-04-21T08:43:15.690 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:15 smithi152 ceph-mon[31401]: purged_snaps scrub ok 2024-04-21T08:43:15.690 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:15 smithi152 ceph-mon[31401]: from='osd.0 [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-21T08:43:15.690 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:15 smithi152 ceph-mon[31401]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T08:43:15.690 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:15 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:43:15.690 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:15 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:43:15.771 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:15 smithi171 ceph-mon[31358]: purged_snaps scrub starts 2024-04-21T08:43:15.771 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:15 smithi171 ceph-mon[31358]: purged_snaps scrub ok 2024-04-21T08:43:15.772 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:15 smithi171 ceph-mon[31358]: from='osd.0 [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-21T08:43:15.772 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:15 smithi171 ceph-mon[31358]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T08:43:15.772 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:15 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:43:15.772 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:15 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:43:15.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:15 smithi028 ceph-mon[28003]: purged_snaps scrub starts 2024-04-21T08:43:15.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:15 smithi028 ceph-mon[28003]: purged_snaps scrub ok 2024-04-21T08:43:15.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:15 smithi028 ceph-mon[28003]: from='osd.0 [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-21T08:43:15.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:15 smithi028 ceph-mon[28003]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T08:43:15.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:15 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:43:15.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:15 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:43:15.933 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi152:vg_nvme/lv_4 2024-04-21T08:43:16.182 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:43:16.591 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:16 smithi152 ceph-mon[31401]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:16.591 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:43:16.591 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:16 smithi152 ceph-mon[31401]: osd.0 [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648] boot 2024-04-21T08:43:16.591 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:16 smithi152 ceph-mon[31401]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T08:43:16.591 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:43:16.591 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:16.591 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:16.591 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:43:16.591 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:16.591 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:16.592 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:16.592 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:16.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:16 smithi028 ceph-mon[28003]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:16.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:43:16.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:16 smithi028 ceph-mon[28003]: osd.0 [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648] boot 2024-04-21T08:43:16.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:16 smithi028 ceph-mon[28003]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T08:43:16.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:43:16.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:16.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:16.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:43:16.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:16.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:16.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:16.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:16 smithi171 ceph-mon[31358]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:43:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:43:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:16 smithi171 ceph-mon[31358]: osd.0 [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648] boot 2024-04-21T08:43:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:16 smithi171 ceph-mon[31358]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T08:43:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:43:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:43:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:16.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:17.456 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:17 smithi152 ceph-mon[31401]: Detected new or changed devices on smithi028 2024-04-21T08:43:17.456 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:17 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi028 to 17093M 2024-04-21T08:43:17.456 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:17 smithi152 ceph-mon[31401]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T08:43:17.651 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.650+0000 7fe28fc56700 1 -- 172.21.15.152:0/2686628527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe288103ca0 msgr2=0x7fe288104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:17.651 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.650+0000 7fe28fc56700 1 --2- 172.21.15.152:0/2686628527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe288103ca0 0x7fe288104100 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe27c0099c0 tx=0x7fe27c009cd0 comp rx=0 tx=0).stop 2024-04-21T08:43:17.651 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.650+0000 7fe28fc56700 1 -- 172.21.15.152:0/2686628527 shutdown_connections 2024-04-21T08:43:17.651 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.650+0000 7fe28fc56700 1 --2- 172.21.15.152:0/2686628527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe288102aa0 0x7fe288102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:17.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.650+0000 7fe28fc56700 1 --2- 172.21.15.152:0/2686628527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe288104640 0x7fe28810aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:17.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.650+0000 7fe28fc56700 1 --2- 172.21.15.152:0/2686628527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe288103ca0 0x7fe288104100 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:17.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.650+0000 7fe28fc56700 1 -- 172.21.15.152:0/2686628527 >> 172.21.15.152:0/2686628527 conn(0x7fe2880fe040 msgr2=0x7fe288100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:17.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.650+0000 7fe28fc56700 1 -- 172.21.15.152:0/2686628527 shutdown_connections 2024-04-21T08:43:17.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.650+0000 7fe28fc56700 1 -- 172.21.15.152:0/2686628527 wait complete. 2024-04-21T08:43:17.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.650+0000 7fe28fc56700 1 Processor -- start 2024-04-21T08:43:17.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.650+0000 7fe28fc56700 1 -- start start 2024-04-21T08:43:17.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28fc56700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe288102aa0 0x7fe2880767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:17.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28fc56700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe288103ca0 0x7fe288074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:17.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28fc56700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe288104640 0x7fe288075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:17.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28fc56700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe28810e510 con 0x7fe288103ca0 2024-04-21T08:43:17.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28fc56700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe28810e390 con 0x7fe288104640 2024-04-21T08:43:17.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28fc56700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe28810e690 con 0x7fe288102aa0 2024-04-21T08:43:17.652 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28e1f3700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe288104640 0x7fe288075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:17.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28e1f3700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe288104640 0x7fe288075370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:45930/0 (socket says 172.21.15.152:45930) 2024-04-21T08:43:17.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28e1f3700 1 -- 172.21.15.152:0/967535848 learned_addr learned my addr 172.21.15.152:0/967535848 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-04-21T08:43:17.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28d9f2700 1 --2- 172.21.15.152:0/967535848 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe288102aa0 0x7fe2880767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:17.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28d1f1700 1 --2- 172.21.15.152:0/967535848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe288103ca0 0x7fe288074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:17.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28e1f3700 1 -- 172.21.15.152:0/967535848 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe288102aa0 msgr2=0x7fe2880767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:17.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28e1f3700 1 --2- 172.21.15.152:0/967535848 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe288102aa0 0x7fe2880767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:17.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28e1f3700 1 -- 172.21.15.152:0/967535848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe288103ca0 msgr2=0x7fe288074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:17.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28e1f3700 1 --2- 172.21.15.152:0/967535848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe288103ca0 0x7fe288074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:17.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28e1f3700 1 -- 172.21.15.152:0/967535848 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe288075c20 con 0x7fe288104640 2024-04-21T08:43:17.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28d1f1700 1 --2- 172.21.15.152:0/967535848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe288103ca0 0x7fe288074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:17.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28e1f3700 1 --2- 172.21.15.152:0/967535848 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe288104640 0x7fe288075370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe27800c900 tx=0x7fe27800ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:17.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe286ffd700 1 -- 172.21.15.152:0/967535848 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278007b00 con 0x7fe288104640 2024-04-21T08:43:17.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe286ffd700 1 -- 172.21.15.152:0/967535848 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe278007ca0 con 0x7fe288104640 2024-04-21T08:43:17.653 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28fc56700 1 -- 172.21.15.152:0/967535848 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2881a2cf0 con 0x7fe288104640 2024-04-21T08:43:17.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe286ffd700 1 -- 172.21.15.152:0/967535848 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe278018660 con 0x7fe288104640 2024-04-21T08:43:17.654 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.651+0000 7fe28fc56700 1 -- 172.21.15.152:0/967535848 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2881a31b0 con 0x7fe288104640 2024-04-21T08:43:17.655 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.652+0000 7fe28fc56700 1 -- 172.21.15.152:0/967535848 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe288069730 con 0x7fe288104640 2024-04-21T08:43:17.655 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.653+0000 7fe286ffd700 1 -- 172.21.15.152:0/967535848 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fe27800f450 con 0x7fe288104640 2024-04-21T08:43:17.655 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.654+0000 7fe286ffd700 1 --2- 172.21.15.152:0/967535848 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe2700a3bb0 0x7fe2700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:17.655 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.654+0000 7fe286ffd700 1 -- 172.21.15.152:0/967535848 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fe2780ce940 con 0x7fe288104640 2024-04-21T08:43:17.655 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.654+0000 7fe28d9f2700 1 --2- 172.21.15.152:0/967535848 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe2700a3bb0 0x7fe2700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:17.655 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.655+0000 7fe28d9f2700 1 --2- 172.21.15.152:0/967535848 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe2700a3bb0 0x7fe2700a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe280007720 tx=0x7fe2800076b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:17.658 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.659+0000 7fe286ffd700 1 -- 172.21.15.152:0/967535848 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe278014030 con 0x7fe288104640 2024-04-21T08:43:17.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:17 smithi028 ceph-mon[28003]: Detected new or changed devices on smithi028 2024-04-21T08:43:17.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:17 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi028 to 17093M 2024-04-21T08:43:17.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:17 smithi028 ceph-mon[28003]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T08:43:17.866 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:17.867+0000 7fe28fc56700 1 -- 172.21.15.152:0/967535848 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe288062e80 con 0x7fe2700a3bb0 2024-04-21T08:43:17.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:17 smithi171 ceph-mon[31358]: Detected new or changed devices on smithi028 2024-04-21T08:43:17.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:17 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi028 to 17093M 2024-04-21T08:43:17.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:17 smithi171 ceph-mon[31358]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T08:43:18.757 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:18 smithi152 ceph-mon[31401]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:18.757 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:18 smithi152 ceph-mon[31401]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:18.757 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:18 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:43:18.757 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:18 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:43:18.757 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:18 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:18.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:18 smithi028 ceph-mon[28003]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:18.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:18 smithi028 ceph-mon[28003]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:18.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:18 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:43:18.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:18 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:43:18.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:18 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:18.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:18 smithi171 ceph-mon[31358]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:18.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:18 smithi171 ceph-mon[31358]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:18.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:18 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:43:18.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:18 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:43:18.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:18 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:20.833 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:20 smithi152 ceph-mon[31401]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:20.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:20 smithi028 ceph-mon[28003]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:20.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:20 smithi171 ceph-mon[31358]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:21.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:21 smithi152 ceph-mon[31401]: from='client.? 172.21.15.152:0/3449569811' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a75fcaea-75f3-44fe-8b41-5ace42be38d2"}]: dispatch 2024-04-21T08:43:21.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:21 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a75fcaea-75f3-44fe-8b41-5ace42be38d2"}]: dispatch 2024-04-21T08:43:21.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:21 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a75fcaea-75f3-44fe-8b41-5ace42be38d2"}]': finished 2024-04-21T08:43:21.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:21 smithi152 ceph-mon[31401]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T08:43:21.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:21 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:43:21.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:21 smithi152 ceph-mon[31401]: from='client.? 172.21.15.152:0/1130394381' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:43:21.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:21 smithi028 ceph-mon[28003]: from='client.? 172.21.15.152:0/3449569811' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a75fcaea-75f3-44fe-8b41-5ace42be38d2"}]: dispatch 2024-04-21T08:43:21.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:21 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a75fcaea-75f3-44fe-8b41-5ace42be38d2"}]: dispatch 2024-04-21T08:43:21.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:21 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a75fcaea-75f3-44fe-8b41-5ace42be38d2"}]': finished 2024-04-21T08:43:21.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:21 smithi028 ceph-mon[28003]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T08:43:21.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:21 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:43:21.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:21 smithi028 ceph-mon[28003]: from='client.? 172.21.15.152:0/1130394381' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:43:21.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:21 smithi171 ceph-mon[31358]: from='client.? 172.21.15.152:0/3449569811' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a75fcaea-75f3-44fe-8b41-5ace42be38d2"}]: dispatch 2024-04-21T08:43:21.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:21 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a75fcaea-75f3-44fe-8b41-5ace42be38d2"}]: dispatch 2024-04-21T08:43:21.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:21 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a75fcaea-75f3-44fe-8b41-5ace42be38d2"}]': finished 2024-04-21T08:43:21.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:21 smithi171 ceph-mon[31358]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T08:43:21.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:43:21.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:21 smithi171 ceph-mon[31358]: from='client.? 172.21.15.152:0/1130394381' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:43:22.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:22 smithi152 ceph-mon[31401]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:22.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:22 smithi028 ceph-mon[28003]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:22.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:22 smithi171 ceph-mon[31358]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:24.734 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:24 smithi152 ceph-mon[31401]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:24.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:24 smithi028 ceph-mon[28003]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:24.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:24 smithi171 ceph-mon[31358]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:26.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:26 smithi152 ceph-mon[31401]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:26.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:26 smithi028 ceph-mon[28003]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:26.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:26 smithi171 ceph-mon[31358]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:28.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:28 smithi152 ceph-mon[31401]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:28.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T08:43:28.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:28.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:28 smithi152 ceph-mon[31401]: Deploying daemon osd.1 on smithi152 2024-04-21T08:43:28.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:28 smithi028 ceph-mon[28003]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:28.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T08:43:28.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:28.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:28 smithi028 ceph-mon[28003]: Deploying daemon osd.1 on smithi152 2024-04-21T08:43:28.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:28 smithi171 ceph-mon[31358]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:28.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T08:43:28.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:28.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:28 smithi171 ceph-mon[31358]: Deploying daemon osd.1 on smithi152 2024-04-21T08:43:30.843 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:30 smithi152 ceph-mon[31401]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:30.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:30 smithi028 ceph-mon[28003]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:30.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:30 smithi171 ceph-mon[31358]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:32.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:32 smithi152 ceph-mon[31401]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:32.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:32 smithi028 ceph-mon[28003]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:32.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:32 smithi171 ceph-mon[31358]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:34.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:34.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:34.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:34.189 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:34.189 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:34.189 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:34.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:34.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:34.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:35.017 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:34 smithi152 ceph-mon[31401]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:35.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:34 smithi171 ceph-mon[31358]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:35.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:34 smithi028 ceph-mon[28003]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:36.456 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:36.457 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:36.457 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:36.457 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:36.457 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:36.534 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:36.534+0000 7fe286ffd700 1 -- 172.21.15.152:0/967535848 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe288062e80 con 0x7fe2700a3bb0 2024-04-21T08:43:36.534 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 1 on host 'smithi152' 2024-04-21T08:43:36.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:36.536+0000 7fe28fc56700 1 -- 172.21.15.152:0/967535848 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe2700a3bb0 msgr2=0x7fe2700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:36.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:36.536+0000 7fe28fc56700 1 --2- 172.21.15.152:0/967535848 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe2700a3bb0 0x7fe2700a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe280007720 tx=0x7fe2800076b0 comp rx=0 tx=0).stop 2024-04-21T08:43:36.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:36.536+0000 7fe28fc56700 1 -- 172.21.15.152:0/967535848 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe288104640 msgr2=0x7fe288075370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:36.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:36.536+0000 7fe28fc56700 1 --2- 172.21.15.152:0/967535848 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe288104640 0x7fe288075370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe27800c900 tx=0x7fe27800ccc0 comp rx=0 tx=0).stop 2024-04-21T08:43:36.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:36.537+0000 7fe28fc56700 1 -- 172.21.15.152:0/967535848 shutdown_connections 2024-04-21T08:43:36.536 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:36.537+0000 7fe28fc56700 1 --2- 172.21.15.152:0/967535848 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe288104640 0x7fe288075370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:36.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:36.537+0000 7fe28fc56700 1 --2- 172.21.15.152:0/967535848 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe288102aa0 0x7fe2880767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:36.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:36.537+0000 7fe28fc56700 1 --2- 172.21.15.152:0/967535848 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe2700a3bb0 0x7fe2700a6070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:36.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:36.537+0000 7fe28fc56700 1 --2- 172.21.15.152:0/967535848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe288103ca0 0x7fe288074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:36.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:36.537+0000 7fe28fc56700 1 -- 172.21.15.152:0/967535848 >> 172.21.15.152:0/967535848 conn(0x7fe2880fe040 msgr2=0x7fe288107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:36.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:36.537+0000 7fe28fc56700 1 -- 172.21.15.152:0/967535848 shutdown_connections 2024-04-21T08:43:36.537 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:43:36.537+0000 7fe28fc56700 1 -- 172.21.15.152:0/967535848 wait complete. 2024-04-21T08:43:36.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:37.144 DEBUG:teuthology.orchestra.run.smithi152:osd.1> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.1.service 2024-04-21T08:43:37.146 INFO:tasks.cephadm:Deploying osd.2 on smithi171 with /dev/vg_nvme/lv_4... 2024-04-21T08:43:37.146 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T08:43:37.389 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:43:37.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:37 smithi028 ceph-mon[28003]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:37.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:37.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:37.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:37.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:37 smithi028 ceph-mon[28003]: from='osd.1 [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:43:37.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:37 smithi028 ceph-mon[28003]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:43:37.420 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:37 smithi152 ceph-mon[31401]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:37.420 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:37.420 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:37.420 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:37.420 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:37 smithi152 ceph-mon[31401]: from='osd.1 [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:43:37.420 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:37 smithi152 ceph-mon[31401]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:43:37.512 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:37 smithi171 ceph-mon[31358]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:37.512 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:37.513 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:37.513 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:37.513 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:37 smithi171 ceph-mon[31358]: from='osd.1 [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:43:37.513 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:37 smithi171 ceph-mon[31358]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:43:38.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:38 smithi171 ceph-mon[31358]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:38.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:38 smithi171 ceph-mon[31358]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T08:43:38.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:38 smithi171 ceph-mon[31358]: from='osd.1 [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:43:38.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:38 smithi171 ceph-mon[31358]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T08:43:38.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:38 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:43:38.738 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:38 smithi171 ceph-mon[31358]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:43:38.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:38 smithi152 ceph-mon[31401]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:38.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:38 smithi152 ceph-mon[31401]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T08:43:38.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:38 smithi152 ceph-mon[31401]: from='osd.1 [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:43:38.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:38 smithi152 ceph-mon[31401]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T08:43:38.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:38 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:43:38.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:38 smithi152 ceph-mon[31401]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:43:38.855 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 08:43:38 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-1[35984]: 2024-04-21T08:43:38.507+0000 7f615d3d6700 -1 osd.1 0 waiting for initial osdmap 2024-04-21T08:43:38.855 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 08:43:38 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-1[35984]: 2024-04-21T08:43:38.511+0000 7f61591c9700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:43:38.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:38 smithi028 ceph-mon[28003]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:38.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:38 smithi028 ceph-mon[28003]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T08:43:38.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:38 smithi028 ceph-mon[28003]: from='osd.1 [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:43:38.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:38 smithi028 ceph-mon[28003]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T08:43:38.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:38 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:43:38.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:38 smithi028 ceph-mon[28003]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:43:39.336 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T08:43:39.336 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T08:43:39.336 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-04-21T08:43:39.336 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-04-21T08:43:39.336 INFO:teuthology.orchestra.run.smithi171.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114488 s, 916 MB/s 2024-04-21T08:43:39.336 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-04-21T08:43:39.650 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:39 smithi171 ceph-mon[31358]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-04-21T08:43:39.650 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:39 smithi171 ceph-mon[31358]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T08:43:39.650 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:39 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:43:39.650 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:39 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:43:39.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:39 smithi152 ceph-mon[31401]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-04-21T08:43:39.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:39 smithi152 ceph-mon[31401]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T08:43:39.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:39 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:43:39.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:39 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:43:39.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:39 smithi028 ceph-mon[28003]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-04-21T08:43:39.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:39 smithi028 ceph-mon[28003]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T08:43:39.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:39 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:43:39.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:39 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:43:39.920 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi171:vg_nvme/lv_4 2024-04-21T08:43:40.170 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:43:40.550 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:40 smithi171 ceph-mon[31358]: purged_snaps scrub starts 2024-04-21T08:43:40.550 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:40 smithi171 ceph-mon[31358]: purged_snaps scrub ok 2024-04-21T08:43:40.550 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:40 smithi171 ceph-mon[31358]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:40.550 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:40 smithi171 ceph-mon[31358]: osd.1 [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438] boot 2024-04-21T08:43:40.550 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:40 smithi171 ceph-mon[31358]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T08:43:40.550 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:40 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:43:40.550 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:40 smithi171 ceph-mon[31358]: Detected new or changed devices on smithi152 2024-04-21T08:43:40.550 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:40 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:40.550 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:40 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:40.550 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:40 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:43:40.550 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:40 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi152 to 17088M 2024-04-21T08:43:40.551 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:40 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:40.551 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:40 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:40.551 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:40 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:40.551 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:40 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:40.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:40 smithi152 ceph-mon[31401]: purged_snaps scrub starts 2024-04-21T08:43:40.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:40 smithi152 ceph-mon[31401]: purged_snaps scrub ok 2024-04-21T08:43:40.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:40 smithi152 ceph-mon[31401]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:40.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:40 smithi152 ceph-mon[31401]: osd.1 [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438] boot 2024-04-21T08:43:40.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:40 smithi152 ceph-mon[31401]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T08:43:40.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:40 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:43:40.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:40 smithi152 ceph-mon[31401]: Detected new or changed devices on smithi152 2024-04-21T08:43:40.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:40 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:40.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:40 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:40.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:40 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:43:40.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:40 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi152 to 17088M 2024-04-21T08:43:40.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:40 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:40.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:40 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:40.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:40 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:40.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:40 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:40.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:40 smithi028 ceph-mon[28003]: purged_snaps scrub starts 2024-04-21T08:43:40.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:40 smithi028 ceph-mon[28003]: purged_snaps scrub ok 2024-04-21T08:43:40.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:40 smithi028 ceph-mon[28003]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:43:40.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:40 smithi028 ceph-mon[28003]: osd.1 [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438] boot 2024-04-21T08:43:40.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:40 smithi028 ceph-mon[28003]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T08:43:40.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:40 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:43:40.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:40 smithi028 ceph-mon[28003]: Detected new or changed devices on smithi152 2024-04-21T08:43:40.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:40 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:40.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:40 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:40.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:40 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:43:40.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:40 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi152 to 17088M 2024-04-21T08:43:40.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:40 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:40.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:40 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:40.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:40 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:43:40.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:40 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:41.704 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:41 smithi171 ceph-mon[31358]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T08:43:41.724 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.725+0000 7f7f4fc5b700 1 -- 172.21.15.171:0/4081690469 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f48104640 msgr2=0x7f7f4810aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:41.725 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.725+0000 7f7f4fc5b700 1 --2- 172.21.15.171:0/4081690469 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f48104640 0x7f7f4810aef0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000b0a0 tx=0x7f7f4000b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:41.725 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.725+0000 7f7f4fc5b700 1 -- 172.21.15.171:0/4081690469 shutdown_connections 2024-04-21T08:43:41.725 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.725+0000 7f7f4fc5b700 1 --2- 172.21.15.171:0/4081690469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f48102aa0 0x7f7f48102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:41.725 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.725+0000 7f7f4fc5b700 1 --2- 172.21.15.171:0/4081690469 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f48104640 0x7f7f4810aef0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:41.725 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.725+0000 7f7f4fc5b700 1 --2- 172.21.15.171:0/4081690469 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f48103ca0 0x7f7f48104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:41.725 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.725+0000 7f7f4fc5b700 1 -- 172.21.15.171:0/4081690469 >> 172.21.15.171:0/4081690469 conn(0x7f7f480fe040 msgr2=0x7f7f48100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:41.725 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.725+0000 7f7f4fc5b700 1 -- 172.21.15.171:0/4081690469 shutdown_connections 2024-04-21T08:43:41.725 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.725+0000 7f7f4fc5b700 1 -- 172.21.15.171:0/4081690469 wait complete. 2024-04-21T08:43:41.727 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.725+0000 7f7f4fc5b700 1 Processor -- start 2024-04-21T08:43:41.727 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.725+0000 7f7f4fc5b700 1 -- start start 2024-04-21T08:43:41.727 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4fc5b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f48102aa0 0x7f7f48198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:41.727 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4fc5b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f48103ca0 0x7f7f48199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:41.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4fc5b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f48104640 0x7f7f481a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:41.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4fc5b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4810e630 con 0x7f7f48104640 2024-04-21T08:43:41.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4fc5b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4810e4b0 con 0x7f7f48102aa0 2024-04-21T08:43:41.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4fc5b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4810e7b0 con 0x7f7f48103ca0 2024-04-21T08:43:41.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4d1f6700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f48103ca0 0x7f7f48199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:41.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4d1f6700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f48103ca0 0x7f7f48199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:53764/0 (socket says 172.21.15.171:53764) 2024-04-21T08:43:41.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4d1f6700 1 -- 172.21.15.171:0/1326034094 learned_addr learned my addr 172.21.15.171:0/1326034094 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-04-21T08:43:41.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4d9f7700 1 --2- 172.21.15.171:0/1326034094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f48102aa0 0x7f7f48198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:41.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4e1f8700 1 --2- 172.21.15.171:0/1326034094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f48104640 0x7f7f481a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:41.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4d1f6700 1 -- 172.21.15.171:0/1326034094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f48102aa0 msgr2=0x7f7f48198d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:41.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4d1f6700 1 --2- 172.21.15.171:0/1326034094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f48102aa0 0x7f7f48198d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:41.728 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4d1f6700 1 -- 172.21.15.171:0/1326034094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f48104640 msgr2=0x7f7f481a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:41.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4d1f6700 1 --2- 172.21.15.171:0/1326034094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f48104640 0x7f7f481a02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:41.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4d1f6700 1 -- 172.21.15.171:0/1326034094 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f481a0970 con 0x7f7f48103ca0 2024-04-21T08:43:41.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4d1f6700 1 --2- 172.21.15.171:0/1326034094 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f48103ca0 0x7f7f48199260 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7f3800b530 tx=0x7f7f3800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:41.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f3effd700 1 -- 172.21.15.171:0/1326034094 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f38015020 con 0x7f7f48103ca0 2024-04-21T08:43:41.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f3effd700 1 -- 172.21.15.171:0/1326034094 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f38011ce0 con 0x7f7f48103ca0 2024-04-21T08:43:41.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4fc5b700 1 -- 172.21.15.171:0/1326034094 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f481a0c60 con 0x7f7f48103ca0 2024-04-21T08:43:41.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f3effd700 1 -- 172.21.15.171:0/1326034094 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f38010d20 con 0x7f7f48103ca0 2024-04-21T08:43:41.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.726+0000 7f7f4fc5b700 1 -- 172.21.15.171:0/1326034094 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f481a1150 con 0x7f7f48103ca0 2024-04-21T08:43:41.729 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.727+0000 7f7f4fc5b700 1 -- 172.21.15.171:0/1326034094 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f48069730 con 0x7f7f48103ca0 2024-04-21T08:43:41.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.728+0000 7f7f3effd700 1 -- 172.21.15.171:0/1326034094 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f7f38018020 con 0x7f7f48103ca0 2024-04-21T08:43:41.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.729+0000 7f7f3effd700 1 --2- 172.21.15.171:0/1326034094 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7f340a3bb0 0x7f7f340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:41.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.729+0000 7f7f3effd700 1 -- 172.21.15.171:0/1326034094 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f7f380ce970 con 0x7f7f48103ca0 2024-04-21T08:43:41.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.729+0000 7f7f4d9f7700 1 --2- 172.21.15.171:0/1326034094 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7f340a3bb0 0x7f7f340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:41.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.730+0000 7f7f4d9f7700 1 --2- 172.21.15.171:0/1326034094 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7f340a3bb0 0x7f7f340a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7f44006fd0 tx=0x7f7f44008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:41.732 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.733+0000 7f7f3effd700 1 -- 172.21.15.171:0/1326034094 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7f3801f020 con 0x7f7f48103ca0 2024-04-21T08:43:41.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:41 smithi152 ceph-mon[31401]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T08:43:41.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:41 smithi028 ceph-mon[28003]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T08:43:41.939 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:43:41.939+0000 7f7f4fc5b700 1 -- 172.21.15.171:0/1326034094 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7f48062e80 con 0x7f7f340a3bb0 2024-04-21T08:43:42.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:42 smithi171 ceph-mon[31358]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:42.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:42 smithi171 ceph-mon[31358]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:42.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:43:42.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:43:42.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:42.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:42 smithi152 ceph-mon[31401]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:42.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:42 smithi152 ceph-mon[31401]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:42.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:43:42.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:43:42.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:42.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:42 smithi028 ceph-mon[28003]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:42.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:42 smithi028 ceph-mon[28003]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:42.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:43:42.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:43:42.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:44.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:44 smithi152 ceph-mon[31401]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:44.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:44 smithi152 ceph-mon[31401]: from='client.? 172.21.15.171:0/2046769327' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e72b05a3-79ac-4a80-a9dd-6192b5e36ae0"}]: dispatch 2024-04-21T08:43:44.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:44 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e72b05a3-79ac-4a80-a9dd-6192b5e36ae0"}]: dispatch 2024-04-21T08:43:44.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:44 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e72b05a3-79ac-4a80-a9dd-6192b5e36ae0"}]': finished 2024-04-21T08:43:44.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:44 smithi152 ceph-mon[31401]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T08:43:44.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:44 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:43:44.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:44 smithi028 ceph-mon[28003]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:44.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:44 smithi028 ceph-mon[28003]: from='client.? 172.21.15.171:0/2046769327' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e72b05a3-79ac-4a80-a9dd-6192b5e36ae0"}]: dispatch 2024-04-21T08:43:44.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:44 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e72b05a3-79ac-4a80-a9dd-6192b5e36ae0"}]: dispatch 2024-04-21T08:43:44.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:44 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e72b05a3-79ac-4a80-a9dd-6192b5e36ae0"}]': finished 2024-04-21T08:43:44.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:44 smithi028 ceph-mon[28003]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T08:43:44.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:44 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:43:44.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:44 smithi171 ceph-mon[31358]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:44.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:44 smithi171 ceph-mon[31358]: from='client.? 172.21.15.171:0/2046769327' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e72b05a3-79ac-4a80-a9dd-6192b5e36ae0"}]: dispatch 2024-04-21T08:43:44.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:44 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e72b05a3-79ac-4a80-a9dd-6192b5e36ae0"}]: dispatch 2024-04-21T08:43:44.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:44 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e72b05a3-79ac-4a80-a9dd-6192b5e36ae0"}]': finished 2024-04-21T08:43:44.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:44 smithi171 ceph-mon[31358]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T08:43:44.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:44 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:43:45.792 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:45 smithi171 ceph-mon[31358]: from='client.? 172.21.15.171:0/1273909502' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:43:45.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:45 smithi152 ceph-mon[31401]: from='client.? 172.21.15.171:0/1273909502' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:43:45.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:45 smithi028 ceph-mon[28003]: from='client.? 172.21.15.171:0/1273909502' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:43:46.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:46 smithi152 ceph-mon[31401]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:46.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:46 smithi028 ceph-mon[28003]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:46.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:46 smithi171 ceph-mon[31358]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:48.723 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:48 smithi171 ceph-mon[31358]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:48.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:48 smithi152 ceph-mon[31401]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:48.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:48 smithi028 ceph-mon[28003]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:50.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:50 smithi152 ceph-mon[31401]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:50.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:50 smithi028 ceph-mon[28003]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:50.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:50 smithi171 ceph-mon[31358]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:51.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T08:43:51.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:51.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T08:43:51.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:51.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T08:43:51.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:43:52.780 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:52 smithi171 ceph-mon[31358]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:52.780 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:52 smithi171 ceph-mon[31358]: Deploying daemon osd.2 on smithi171 2024-04-21T08:43:52.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:52 smithi152 ceph-mon[31401]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:52.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:52 smithi152 ceph-mon[31401]: Deploying daemon osd.2 on smithi171 2024-04-21T08:43:52.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:52 smithi028 ceph-mon[28003]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:52.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:52 smithi028 ceph-mon[28003]: Deploying daemon osd.2 on smithi171 2024-04-21T08:43:54.782 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:54 smithi171 ceph-mon[31358]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:54.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:54 smithi152 ceph-mon[31401]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:54.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:54 smithi028 ceph-mon[28003]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:56.828 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:56 smithi171 ceph-mon[31358]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:56.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:56 smithi152 ceph-mon[31401]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:56.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:56 smithi028 ceph-mon[28003]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:58.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:57 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:58.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:57 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:58.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:57 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:58.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:57 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:58.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:57 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:58.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:57 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:58.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:57 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:58.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:57 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:43:58.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:57 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:59.089 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:43:58 smithi171 ceph-mon[31358]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:59.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:43:58 smithi152 ceph-mon[31401]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:43:59.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:43:58 smithi028 ceph-mon[28003]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:44:00.309 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:00.309+0000 7f7f3effd700 1 -- 172.21.15.171:0/1326034094 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7f48062e80 con 0x7f7f340a3bb0 2024-04-21T08:44:00.309 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 2 on host 'smithi171' 2024-04-21T08:44:00.311 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:00.311+0000 7f7f4fc5b700 1 -- 172.21.15.171:0/1326034094 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7f340a3bb0 msgr2=0x7f7f340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:00.311 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:00.311+0000 7f7f4fc5b700 1 --2- 172.21.15.171:0/1326034094 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7f340a3bb0 0x7f7f340a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7f44006fd0 tx=0x7f7f44008040 comp rx=0 tx=0).stop 2024-04-21T08:44:00.311 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:00.311+0000 7f7f4fc5b700 1 -- 172.21.15.171:0/1326034094 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f48103ca0 msgr2=0x7f7f48199260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:00.311 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:00.311+0000 7f7f4fc5b700 1 --2- 172.21.15.171:0/1326034094 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f48103ca0 0x7f7f48199260 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7f3800b530 tx=0x7f7f3800b8f0 comp rx=0 tx=0).stop 2024-04-21T08:44:00.311 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:00.311+0000 7f7f4fc5b700 1 -- 172.21.15.171:0/1326034094 shutdown_connections 2024-04-21T08:44:00.312 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:00.311+0000 7f7f4fc5b700 1 --2- 172.21.15.171:0/1326034094 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7f340a3bb0 0x7f7f340a6070 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:00.312 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:00.311+0000 7f7f4fc5b700 1 --2- 172.21.15.171:0/1326034094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f48104640 0x7f7f481a02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:00.312 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:00.311+0000 7f7f4fc5b700 1 --2- 172.21.15.171:0/1326034094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f48102aa0 0x7f7f48198d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:00.312 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:00.311+0000 7f7f4fc5b700 1 --2- 172.21.15.171:0/1326034094 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7f48103ca0 0x7f7f48199260 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:00.312 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:00.312+0000 7f7f4fc5b700 1 -- 172.21.15.171:0/1326034094 >> 172.21.15.171:0/1326034094 conn(0x7f7f480fe040 msgr2=0x7f7f481070e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:00.312 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:00.312+0000 7f7f4fc5b700 1 -- 172.21.15.171:0/1326034094 shutdown_connections 2024-04-21T08:44:00.312 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:00.312+0000 7f7f4fc5b700 1 -- 172.21.15.171:0/1326034094 wait complete. 2024-04-21T08:44:00.557 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:00 smithi171 ceph-mon[31358]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:44:00.557 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:00 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:00.557 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:00 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:00.557 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:00 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:00.557 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:00 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:00.557 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:00 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:00.557 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:00 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:00.557 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:00 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:00.558 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:00 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:00.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:00 smithi152 ceph-mon[31401]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:44:00.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:00 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:00.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:00 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:00.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:00 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:00.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:00 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:00.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:00 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:00.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:00 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:00.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:00 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:00.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:00 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:00.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:00 smithi028 ceph-mon[28003]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:44:00.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:00 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:00.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:00 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:00.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:00 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:00.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:00 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:00.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:00 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:00.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:00 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:00.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:00 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:00.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:00 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:00.947 DEBUG:teuthology.orchestra.run.smithi171:osd.2> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.2.service 2024-04-21T08:44:00.949 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_3... 2024-04-21T08:44:00.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T08:44:01.205 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:01.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:01 smithi028 ceph-mon[28003]: from='osd.2 [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:44:01.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:01 smithi028 ceph-mon[28003]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:44:01.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:01 smithi171 ceph-mon[31358]: from='osd.2 [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:44:01.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:01 smithi171 ceph-mon[31358]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:44:01.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:01 smithi152 ceph-mon[31401]: from='osd.2 [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:44:01.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:01 smithi152 ceph-mon[31401]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:44:02.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:02 smithi028 ceph-mon[28003]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:44:02.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:02 smithi028 ceph-mon[28003]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T08:44:02.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:02 smithi028 ceph-mon[28003]: from='osd.2 [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:44:02.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:02 smithi028 ceph-mon[28003]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T08:44:02.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:02 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:02.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:02 smithi028 ceph-mon[28003]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:44:02.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:02 smithi171 ceph-mon[31358]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:44:02.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:02 smithi171 ceph-mon[31358]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T08:44:02.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:02 smithi171 ceph-mon[31358]: from='osd.2 [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:44:02.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:02 smithi171 ceph-mon[31358]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T08:44:02.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:02 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:02.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:02 smithi171 ceph-mon[31358]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:44:02.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:02 smithi152 ceph-mon[31401]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:44:02.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:02 smithi152 ceph-mon[31401]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T08:44:02.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:02 smithi152 ceph-mon[31401]: from='osd.2 [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:44:02.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:02 smithi152 ceph-mon[31401]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T08:44:02.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:02 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:02.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:02 smithi152 ceph-mon[31401]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:44:03.255 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 08:44:02 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-2[36540]: 2024-04-21T08:44:02.923+0000 7f7bd3c4e700 -1 osd.2 0 waiting for initial osdmap 2024-04-21T08:44:03.255 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 08:44:02 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-2[36540]: 2024-04-21T08:44:02.928+0000 7f7bcba39700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:44:03.314 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T08:44:03.315 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T08:44:03.315 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-21T08:44:03.315 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-21T08:44:03.315 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00839296 s, 1.2 GB/s 2024-04-21T08:44:03.315 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-21T08:44:03.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:03 smithi028 ceph-mon[28003]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-21T08:44:03.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:03 smithi028 ceph-mon[28003]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T08:44:03.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:03 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:03.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:03 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:03.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:03 smithi028 ceph-mon[28003]: from='osd.2 ' entity='osd.2' 2024-04-21T08:44:03.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:03 smithi171 ceph-mon[31358]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-21T08:44:03.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:03 smithi171 ceph-mon[31358]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T08:44:03.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:03 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:03.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:03 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:03.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:03 smithi171 ceph-mon[31358]: from='osd.2 ' entity='osd.2' 2024-04-21T08:44:03.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:03 smithi152 ceph-mon[31401]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-21T08:44:03.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:03 smithi152 ceph-mon[31401]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T08:44:03.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:03 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:03.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:03 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:03.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:03 smithi152 ceph-mon[31401]: from='osd.2 ' entity='osd.2' 2024-04-21T08:44:03.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-04-21T08:44:04.225 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: purged_snaps scrub starts 2024-04-21T08:44:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: purged_snaps scrub ok 2024-04-21T08:44:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:44:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: osd.2 [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582] boot 2024-04-21T08:44:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T08:44:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: Detected new or changed devices on smithi171 2024-04-21T08:44:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:04.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:04.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:04.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi171 to 17088M 2024-04-21T08:44:04.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:04.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:04.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:04.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:04 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:04.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: purged_snaps scrub starts 2024-04-21T08:44:04.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: purged_snaps scrub ok 2024-04-21T08:44:04.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:44:04.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:04.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: osd.2 [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582] boot 2024-04-21T08:44:04.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T08:44:04.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:04.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: Detected new or changed devices on smithi171 2024-04-21T08:44:04.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:04.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:04.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:04.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi171 to 17088M 2024-04-21T08:44:04.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:04.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:04.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:04.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:04 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:04.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: purged_snaps scrub starts 2024-04-21T08:44:04.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: purged_snaps scrub ok 2024-04-21T08:44:04.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:44:04.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:04.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: osd.2 [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582] boot 2024-04-21T08:44:04.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T08:44:04.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:44:04.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: Detected new or changed devices on smithi171 2024-04-21T08:44:04.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:04.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:04.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:04.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi171 to 17088M 2024-04-21T08:44:04.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:04.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:04.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:04.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:04 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:05.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:44:05.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:05 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:44:05.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:05 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:44:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.093+0000 7f0e4fc30700 1 -- 172.21.15.28:0/3369322443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e48103540 msgr2=0x7f0e48109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.093+0000 7f0e4fc30700 1 --2- 172.21.15.28:0/3369322443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e48103540 0x7f0e48109df0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0e4000b0d0 tx=0x7f0e4000b3e0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.093+0000 7f0e4fc30700 1 -- 172.21.15.28:0/3369322443 shutdown_connections 2024-04-21T08:44:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.093+0000 7f0e4fc30700 1 --2- 172.21.15.28:0/3369322443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e48103540 0x7f0e48109df0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.093+0000 7f0e4fc30700 1 --2- 172.21.15.28:0/3369322443 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0e481019a0 0x7f0e48101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.093+0000 7f0e4fc30700 1 --2- 172.21.15.28:0/3369322443 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e48102ba0 0x7f0e48103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.093+0000 7f0e4fc30700 1 -- 172.21.15.28:0/3369322443 >> 172.21.15.28:0/3369322443 conn(0x7f0e480fcf40 msgr2=0x7f0e480ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.093+0000 7f0e4fc30700 1 -- 172.21.15.28:0/3369322443 shutdown_connections 2024-04-21T08:44:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.093+0000 7f0e4fc30700 1 -- 172.21.15.28:0/3369322443 wait complete. 2024-04-21T08:44:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.094+0000 7f0e4fc30700 1 Processor -- start 2024-04-21T08:44:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.094+0000 7f0e4fc30700 1 -- start start 2024-04-21T08:44:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.094+0000 7f0e4fc30700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e481019a0 0x7f0e4806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.094+0000 7f0e4fc30700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e48102ba0 0x7f0e4819d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.094+0000 7f0e4fc30700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0e48103540 0x7f0e4819ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.094+0000 7f0e4fc30700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4810d420 con 0x7f0e481019a0 2024-04-21T08:44:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.094+0000 7f0e4fc30700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4810d2a0 con 0x7f0e48103540 2024-04-21T08:44:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.094+0000 7f0e4fc30700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4810d5a0 con 0x7f0e48102ba0 2024-04-21T08:44:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.094+0000 7f0e4d9cc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e481019a0 0x7f0e4806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.094+0000 7f0e4d9cc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e481019a0 0x7f0e4806b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57302/0 (socket says 172.21.15.28:57302) 2024-04-21T08:44:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.094+0000 7f0e4d9cc700 1 -- 172.21.15.28:0/1836432481 learned_addr learned my addr 172.21.15.28:0/1836432481 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:44:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.094+0000 7f0e4d1cb700 1 --2- 172.21.15.28:0/1836432481 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e48102ba0 0x7f0e4819d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.094+0000 7f0e4e1cd700 1 --2- 172.21.15.28:0/1836432481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0e48103540 0x7f0e4819ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.095+0000 7f0e4d9cc700 1 -- 172.21.15.28:0/1836432481 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e48102ba0 msgr2=0x7f0e4819d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.095+0000 7f0e4d9cc700 1 --2- 172.21.15.28:0/1836432481 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e48102ba0 0x7f0e4819d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.095+0000 7f0e4d9cc700 1 -- 172.21.15.28:0/1836432481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0e48103540 msgr2=0x7f0e4819ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.095+0000 7f0e4d9cc700 1 --2- 172.21.15.28:0/1836432481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0e48103540 0x7f0e4819ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.095+0000 7f0e4d9cc700 1 -- 172.21.15.28:0/1836432481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e481a0820 con 0x7f0e481019a0 2024-04-21T08:44:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.095+0000 7f0e4d1cb700 1 --2- 172.21.15.28:0/1836432481 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e48102ba0 0x7f0e4819d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.095+0000 7f0e4d9cc700 1 --2- 172.21.15.28:0/1836432481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e481019a0 0x7f0e4806b2e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0e3800d980 tx=0x7f0e3800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.095+0000 7f0e3effd700 1 -- 172.21.15.28:0/1836432481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e38017030 con 0x7f0e481019a0 2024-04-21T08:44:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.095+0000 7f0e4fc30700 1 -- 172.21.15.28:0/1836432481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e481976f0 con 0x7f0e481019a0 2024-04-21T08:44:06.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.095+0000 7f0e3effd700 1 -- 172.21.15.28:0/1836432481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e380044d0 con 0x7f0e481019a0 2024-04-21T08:44:06.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.095+0000 7f0e3effd700 1 -- 172.21.15.28:0/1836432481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e380039e0 con 0x7f0e481019a0 2024-04-21T08:44:06.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.095+0000 7f0e4fc30700 1 -- 172.21.15.28:0/1836432481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e48197c80 con 0x7f0e481019a0 2024-04-21T08:44:06.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.096+0000 7f0e4fc30700 1 -- 172.21.15.28:0/1836432481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e48069730 con 0x7f0e481019a0 2024-04-21T08:44:06.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.097+0000 7f0e3effd700 1 -- 172.21.15.28:0/1836432481 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f0e3801a070 con 0x7f0e481019a0 2024-04-21T08:44:06.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.098+0000 7f0e3effd700 1 --2- 172.21.15.28:0/1836432481 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0e340a3c30 0x7f0e340a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:06.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.098+0000 7f0e3effd700 1 -- 172.21.15.28:0/1836432481 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7f0e380ceb80 con 0x7f0e481019a0 2024-04-21T08:44:06.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.098+0000 7f0e4d1cb700 1 --2- 172.21.15.28:0/1836432481 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0e340a3c30 0x7f0e340a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:06.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.098+0000 7f0e4d1cb700 1 --2- 172.21.15.28:0/1836432481 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0e340a3c30 0x7f0e340a60f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0e44005950 tx=0x7f0e440058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.102+0000 7f0e3effd700 1 -- 172.21.15.28:0/1836432481 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0e3800e350 con 0x7f0e481019a0 2024-04-21T08:44:06.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:06.321+0000 7f0e4fc30700 1 -- 172.21.15.28:0/1836432481 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0e48199130 con 0x7f0e340a3c30 2024-04-21T08:44:06.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:06 smithi028 ceph-mon[28003]: pgmap v83: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:06.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:06 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T08:44:06.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:06 smithi028 ceph-mon[28003]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T08:44:06.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:06 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:44:06.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:06 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:44:06.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:06 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:44:06.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:06 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:06.757 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40602]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:44:06.757 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40602]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:44:06.757 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40602]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:44:06.757 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40602]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:44:06.757 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:06 smithi171 ceph-mon[31358]: pgmap v83: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:06.757 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:06 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T08:44:06.758 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:06 smithi171 ceph-mon[31358]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T08:44:06.758 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:06 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:44:06.758 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:06 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:44:06.758 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:06 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:44:06.758 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:06 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:06.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:06 smithi152 ceph-mon[31401]: pgmap v83: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:06.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:06 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T08:44:06.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:06 smithi152 ceph-mon[31401]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T08:44:06.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:06 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:44:06.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:06 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:44:06.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:06 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:44:06.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:06 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:06.855 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40115]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:44:06.855 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40115]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:44:06.855 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40115]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:44:06.855 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40115]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:44:06.855 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40118]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:44:06.855 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40118]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:44:06.855 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40118]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:44:06.855 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40118]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:44:06.990 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47522]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:44:06.990 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47522]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:44:06.990 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47522]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:44:06.990 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47522]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:44:06.990 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47525]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:44:06.991 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47525]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:44:06.991 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47525]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:44:06.991 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47525]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:44:06.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47539]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:44:06.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47539]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:44:06.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47539]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:44:06.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47539]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:44:06.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47557]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:44:06.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47557]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:44:06.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47557]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:44:06.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:06 smithi028 sudo[47557]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:44:07.120 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40605]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:44:07.120 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40605]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:44:07.120 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40605]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:44:07.120 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40605]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:44:07.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40608]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:44:07.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40608]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:44:07.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40608]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:44:07.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40608]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:44:07.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40611]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:44:07.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40611]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:44:07.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40611]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:44:07.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:06 smithi171 sudo[40611]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:44:07.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40121]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:44:07.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40121]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:44:07.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40121]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:44:07.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40121]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:44:07.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40124]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:44:07.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40124]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:44:07.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40124]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:44:07.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:06 smithi152 sudo[40124]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:44:07.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:07.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:44:07.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T08:44:07.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:07.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:07.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:07.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:44:07.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:44:07.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:44:07.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:44:07.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:44:07.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:44:07.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:44:07.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:44:07.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:44:07.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:44:07.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:44:07.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:44:07.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:44:07.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:44:07.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:44:07.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:07 smithi028 ceph-mon[28003]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T08:44:07.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:07.805+0000 7f0e3effd700 1 -- 172.21.15.28:0/1836432481 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0e38098a10 con 0x7f0e481019a0 2024-04-21T08:44:07.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:07.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:44:07.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T08:44:07.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:07.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:07.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:07.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:44:07.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:44:07.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:44:07.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:44:07.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:44:07.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:44:07.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:44:07.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:44:07.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:44:07.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:44:07.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:44:07.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:44:07.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:44:07.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:44:07.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:44:07.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:07 smithi152 ceph-mon[31401]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T08:44:07.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:07.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:44:07.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T08:44:07.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:07.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:07.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:07.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:44:07.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:44:07.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:44:07.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:44:07.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:44:07.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:44:07.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:44:07.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:44:07.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:44:07.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:44:07.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:44:07.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:44:07.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:44:07.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:44:07.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:44:07.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:07 smithi171 ceph-mon[31358]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T08:44:09.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:08 smithi152 ceph-mon[31401]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:09.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:08 smithi152 ceph-mon[31401]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T08:44:09.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:08 smithi028 ceph-mon[28003]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:09.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:08 smithi028 ceph-mon[28003]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T08:44:09.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:08 smithi171 ceph-mon[31358]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:09.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:08 smithi171 ceph-mon[31358]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T08:44:10.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:09 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3383077865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8265dd0b-19d9-45f2-887d-a0467bbca91a"}]: dispatch 2024-04-21T08:44:10.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:09 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3383077865' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8265dd0b-19d9-45f2-887d-a0467bbca91a"}]': finished 2024-04-21T08:44:10.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:09 smithi152 ceph-mon[31401]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T08:44:10.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:09 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:10.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:09 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3383077865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8265dd0b-19d9-45f2-887d-a0467bbca91a"}]: dispatch 2024-04-21T08:44:10.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:09 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3383077865' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8265dd0b-19d9-45f2-887d-a0467bbca91a"}]': finished 2024-04-21T08:44:10.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:09 smithi028 ceph-mon[28003]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T08:44:10.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:09 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:10.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:09 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3383077865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8265dd0b-19d9-45f2-887d-a0467bbca91a"}]: dispatch 2024-04-21T08:44:10.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:09 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3383077865' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8265dd0b-19d9-45f2-887d-a0467bbca91a"}]': finished 2024-04-21T08:44:10.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:09 smithi171 ceph-mon[31358]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T08:44:10.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:09 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:11.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:10 smithi152 ceph-mon[31401]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:11.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:10 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/213387783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:44:11.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:10 smithi028 ceph-mon[28003]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:10 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/213387783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:44:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:10 smithi171 ceph-mon[31358]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:10 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/213387783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:44:13.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:12 smithi152 ceph-mon[31401]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:13.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:12 smithi028 ceph-mon[28003]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:12 smithi171 ceph-mon[31358]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:15.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:14 smithi152 ceph-mon[31401]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:15.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:14 smithi028 ceph-mon[28003]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:15.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:14 smithi171 ceph-mon[31358]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:17.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:16 smithi152 ceph-mon[31401]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:17.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T08:44:17.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:17.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:16 smithi028 ceph-mon[28003]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:17.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T08:44:17.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:17.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:16 smithi171 ceph-mon[31358]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T08:44:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:18.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:17 smithi152 ceph-mon[31401]: Deploying daemon osd.3 on smithi028 2024-04-21T08:44:18.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:17 smithi028 ceph-mon[28003]: Deploying daemon osd.3 on smithi028 2024-04-21T08:44:18.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:17 smithi171 ceph-mon[31358]: Deploying daemon osd.3 on smithi028 2024-04-21T08:44:19.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:18 smithi028 ceph-mon[28003]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:19.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:18 smithi152 ceph-mon[31401]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:19.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:18 smithi171 ceph-mon[31358]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:21.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:20 smithi152 ceph-mon[31401]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:21.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:20 smithi028 ceph-mon[28003]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:21.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:20 smithi171 ceph-mon[31358]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:23.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:22 smithi028 ceph-mon[28003]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:23.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:22 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:23.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:22 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:23.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:22 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:23.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:22 smithi152 ceph-mon[31401]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:23.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:22 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:23.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:22 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:23.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:22 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:23.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:22 smithi171 ceph-mon[31358]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:23.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:22 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:23.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:22 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:23.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:22 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:25.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:24 smithi028 ceph-mon[28003]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:25.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:24 smithi152 ceph-mon[31401]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:25.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:24 smithi171 ceph-mon[31358]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:25.431+0000 7f0e3effd700 1 -- 172.21.15.28:0/1836432481 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0e48199130 con 0x7f0e340a3c30 2024-04-21T08:44:25.431 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-04-21T08:44:25.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:25.433+0000 7f0e4fc30700 1 -- 172.21.15.28:0/1836432481 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0e340a3c30 msgr2=0x7f0e340a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:25.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:25.433+0000 7f0e4fc30700 1 --2- 172.21.15.28:0/1836432481 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0e340a3c30 0x7f0e340a60f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0e44005950 tx=0x7f0e440058e0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:25.433+0000 7f0e4fc30700 1 -- 172.21.15.28:0/1836432481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e481019a0 msgr2=0x7f0e4806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:25.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:25.433+0000 7f0e4fc30700 1 --2- 172.21.15.28:0/1836432481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e481019a0 0x7f0e4806b2e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0e3800d980 tx=0x7f0e3800dc90 comp rx=0 tx=0).stop 2024-04-21T08:44:25.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:25.433+0000 7f0e4fc30700 1 -- 172.21.15.28:0/1836432481 shutdown_connections 2024-04-21T08:44:25.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:25.433+0000 7f0e4fc30700 1 --2- 172.21.15.28:0/1836432481 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0e340a3c30 0x7f0e340a60f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:25.433+0000 7f0e4fc30700 1 --2- 172.21.15.28:0/1836432481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e481019a0 0x7f0e4806b2e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:25.433+0000 7f0e4fc30700 1 --2- 172.21.15.28:0/1836432481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0e48103540 0x7f0e4819ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:25.433+0000 7f0e4fc30700 1 --2- 172.21.15.28:0/1836432481 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e48102ba0 0x7f0e4819d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:25.433+0000 7f0e4fc30700 1 -- 172.21.15.28:0/1836432481 >> 172.21.15.28:0/1836432481 conn(0x7f0e480fcf40 msgr2=0x7f0e48105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:25.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:25.433+0000 7f0e4fc30700 1 -- 172.21.15.28:0/1836432481 shutdown_connections 2024-04-21T08:44:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:44:25.433+0000 7f0e4fc30700 1 -- 172.21.15.28:0/1836432481 wait complete. 2024-04-21T08:44:26.475 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.3.service 2024-04-21T08:44:26.478 INFO:tasks.cephadm:Deploying osd.4 on smithi152 with /dev/vg_nvme/lv_3... 2024-04-21T08:44:26.478 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T08:44:26.817 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:44:26.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:26 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:26.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:26 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:26.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:26 smithi152 ceph-mon[31401]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:26.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:26 smithi152 ceph-mon[31401]: from='osd.3 [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T08:44:26.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:26 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:26.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:26 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:26.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:26 smithi028 ceph-mon[28003]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:26.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:26 smithi028 ceph-mon[28003]: from='osd.3 [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T08:44:26.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:26 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:26.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:26 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:26.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:26 smithi171 ceph-mon[31358]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:26.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:26 smithi171 ceph-mon[31358]: from='osd.3 [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T08:44:27.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:27 smithi028 ceph-mon[28003]: from='osd.3 [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T08:44:27.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:27 smithi028 ceph-mon[28003]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T08:44:27.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:27 smithi028 ceph-mon[28003]: from='osd.3 [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-21T08:44:27.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:27 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:27.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:27 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:27.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:27 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:27.485 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 08:44:27 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-3[49730]: 2024-04-21T08:44:27.409+0000 7ffa31c42700 -1 osd.3 0 waiting for initial osdmap 2024-04-21T08:44:27.485 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 08:44:27 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-3[49730]: 2024-04-21T08:44:27.414+0000 7ffa2da35700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:44:27.592 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:27 smithi152 ceph-mon[31401]: from='osd.3 [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T08:44:27.592 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:27 smithi152 ceph-mon[31401]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T08:44:27.592 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:27 smithi152 ceph-mon[31401]: from='osd.3 [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-21T08:44:27.592 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:27 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:27.592 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:27 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:27.592 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:27 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:27.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:27 smithi171 ceph-mon[31358]: from='osd.3 [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T08:44:27.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:27 smithi171 ceph-mon[31358]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T08:44:27.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:27 smithi171 ceph-mon[31358]: from='osd.3 [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-21T08:44:27.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:27 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:27.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:27 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:27.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:27 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:28.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:28 smithi028 ceph-mon[28003]: from='osd.3 [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-21T08:44:28.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:28 smithi028 ceph-mon[28003]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T08:44:28.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:28.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:28 smithi028 ceph-mon[28003]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:28.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:28.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:28 smithi152 ceph-mon[31401]: from='osd.3 [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-21T08:44:28.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:28 smithi152 ceph-mon[31401]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T08:44:28.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:28.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:28 smithi152 ceph-mon[31401]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:28.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:28 smithi171 ceph-mon[31358]: from='osd.3 [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-21T08:44:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:28 smithi171 ceph-mon[31358]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T08:44:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:28 smithi171 ceph-mon[31358]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:28.783 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T08:44:28.783 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T08:44:28.783 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-04-21T08:44:28.783 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-04-21T08:44:28.783 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114787 s, 913 MB/s 2024-04-21T08:44:28.783 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-04-21T08:44:29.423 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi152:vg_nvme/lv_3 2024-04-21T08:44:29.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:29 smithi028 ceph-mon[28003]: purged_snaps scrub starts 2024-04-21T08:44:29.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:29 smithi028 ceph-mon[28003]: purged_snaps scrub ok 2024-04-21T08:44:29.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:29 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:29.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:29 smithi028 ceph-mon[28003]: osd.3 [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959] boot 2024-04-21T08:44:29.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:29 smithi028 ceph-mon[28003]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T08:44:29.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:29 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:29.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:29 smithi028 ceph-mon[28003]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T08:44:29.672 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:44:29.712 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:29 smithi152 ceph-mon[31401]: purged_snaps scrub starts 2024-04-21T08:44:29.712 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:29 smithi152 ceph-mon[31401]: purged_snaps scrub ok 2024-04-21T08:44:29.713 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:29 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:29.713 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:29 smithi152 ceph-mon[31401]: osd.3 [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959] boot 2024-04-21T08:44:29.713 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:29 smithi152 ceph-mon[31401]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T08:44:29.713 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:29 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:29.713 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:29 smithi152 ceph-mon[31401]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T08:44:29.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:29 smithi171 ceph-mon[31358]: purged_snaps scrub starts 2024-04-21T08:44:29.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:29 smithi171 ceph-mon[31358]: purged_snaps scrub ok 2024-04-21T08:44:29.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:29 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:29.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:29 smithi171 ceph-mon[31358]: osd.3 [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959] boot 2024-04-21T08:44:29.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:29 smithi171 ceph-mon[31358]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T08:44:29.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:29 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:44:29.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:29 smithi171 ceph-mon[31358]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T08:44:30.664 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:30.665 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T08:44:30.665 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: Detected new or changed devices on smithi028 2024-04-21T08:44:30.665 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:30.665 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:30.665 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:30.665 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:30.665 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi028 to 8546M 2024-04-21T08:44:30.665 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:30.665 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:30.665 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:30.665 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:30.665 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:30.665 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:30.666 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:30.666 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:30.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:30.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T08:44:30.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: Detected new or changed devices on smithi028 2024-04-21T08:44:30.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:30.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:30.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:30.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:30.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi028 to 8546M 2024-04-21T08:44:30.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:30.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:30.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:30.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:30.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:30.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:30.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:30.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:30.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:44:30.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T08:44:30.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: Detected new or changed devices on smithi028 2024-04-21T08:44:30.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:30.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:30.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:30.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:30.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi028 to 8546M 2024-04-21T08:44:30.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:30.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:30.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:30.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:30.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:30.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:30.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:30.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:31.212 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.213+0000 7f11d4982700 1 -- 172.21.15.152:0/3595465259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11d0101990 msgr2=0x7f11d0101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.213 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.213+0000 7f11d4982700 1 --2- 172.21.15.152:0/3595465259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11d0101990 0x7f11d0101db0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f11c00099c0 tx=0x7f11c0009cd0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.213 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.213+0000 7f11d4982700 1 -- 172.21.15.152:0/3595465259 shutdown_connections 2024-04-21T08:44:31.213 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.213+0000 7f11d4982700 1 --2- 172.21.15.152:0/3595465259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d0103530 0x7f11d0109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.213 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.213+0000 7f11d4982700 1 --2- 172.21.15.152:0/3595465259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11d0101990 0x7f11d0101db0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.213 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.213+0000 7f11d4982700 1 --2- 172.21.15.152:0/3595465259 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11d0102b90 0x7f11d0102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.213 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.213+0000 7f11d4982700 1 -- 172.21.15.152:0/3595465259 >> 172.21.15.152:0/3595465259 conn(0x7f11d00fcf30 msgr2=0x7f11d00ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:31.213 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.213+0000 7f11d4982700 1 -- 172.21.15.152:0/3595465259 shutdown_connections 2024-04-21T08:44:31.213 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.213+0000 7f11d4982700 1 -- 172.21.15.152:0/3595465259 wait complete. 2024-04-21T08:44:31.215 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11d4982700 1 Processor -- start 2024-04-21T08:44:31.215 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11d4982700 1 -- start start 2024-04-21T08:44:31.215 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11d4982700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11d0101990 0x7f11d006bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.215 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11d4982700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d0102b90 0x7f11d019d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.216 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11d4982700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11d0103530 0x7f11d019ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.216 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11d4982700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11d010d420 con 0x7f11d0102b90 2024-04-21T08:44:31.216 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11d4982700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f11d010d2a0 con 0x7f11d0101990 2024-04-21T08:44:31.216 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11d4982700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f11d010d5a0 con 0x7f11d0103530 2024-04-21T08:44:31.216 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11ce59c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11d0101990 0x7f11d006bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.216 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11ce59c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11d0101990 0x7f11d006bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:60940/0 (socket says 172.21.15.152:60940) 2024-04-21T08:44:31.216 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11ce59c700 1 -- 172.21.15.152:0/2054138865 learned_addr learned my addr 172.21.15.152:0/2054138865 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-04-21T08:44:31.216 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11cdd9b700 1 --2- 172.21.15.152:0/2054138865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d0102b90 0x7f11d019d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.216 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11ced9d700 1 --2- 172.21.15.152:0/2054138865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11d0103530 0x7f11d019ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.216 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11ce59c700 1 -- 172.21.15.152:0/2054138865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11d0103530 msgr2=0x7f11d019ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.216 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11ce59c700 1 --2- 172.21.15.152:0/2054138865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11d0103530 0x7f11d019ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.216 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11ce59c700 1 -- 172.21.15.152:0/2054138865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d0102b90 msgr2=0x7f11d019d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.216 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11ce59c700 1 --2- 172.21.15.152:0/2054138865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d0102b90 0x7f11d019d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.217 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11ce59c700 1 -- 172.21.15.152:0/2054138865 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d01a0860 con 0x7f11d0101990 2024-04-21T08:44:31.217 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.214+0000 7f11ce59c700 1 --2- 172.21.15.152:0/2054138865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11d0101990 0x7f11d006bce0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f11c00038c0 tx=0x7f11c00047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:31.217 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.215+0000 7f11bf7fe700 1 -- 172.21.15.152:0/2054138865 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0027070 con 0x7f11d0101990 2024-04-21T08:44:31.217 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.215+0000 7f11bf7fe700 1 -- 172.21.15.152:0/2054138865 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11c00041a0 con 0x7f11d0101990 2024-04-21T08:44:31.217 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.215+0000 7f11bf7fe700 1 -- 172.21.15.152:0/2054138865 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c000f460 con 0x7f11d0101990 2024-04-21T08:44:31.217 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.215+0000 7f11d4982700 1 -- 172.21.15.152:0/2054138865 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d01a0af0 con 0x7f11d0101990 2024-04-21T08:44:31.217 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.215+0000 7f11d4982700 1 -- 172.21.15.152:0/2054138865 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d01a0fd0 con 0x7f11d0101990 2024-04-21T08:44:31.217 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.216+0000 7f11d4982700 1 -- 172.21.15.152:0/2054138865 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d0069730 con 0x7f11d0101990 2024-04-21T08:44:31.218 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.217+0000 7f11bf7fe700 1 -- 172.21.15.152:0/2054138865 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f11c002a070 con 0x7f11d0101990 2024-04-21T08:44:31.218 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.217+0000 7f11bf7fe700 1 --2- 172.21.15.152:0/2054138865 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f11b80a3c40 0x7f11b80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.218 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.218+0000 7f11bf7fe700 1 -- 172.21.15.152:0/2054138865 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f11c00d05e0 con 0x7f11d0101990 2024-04-21T08:44:31.218 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.218+0000 7f11cdd9b700 1 --2- 172.21.15.152:0/2054138865 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f11b80a3c40 0x7f11b80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.218 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.218+0000 7f11cdd9b700 1 --2- 172.21.15.152:0/2054138865 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f11b80a3c40 0x7f11b80a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f11c4005950 tx=0x7f11c40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:31.220 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.221+0000 7f11bf7fe700 1 -- 172.21.15.152:0/2054138865 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f11c0031070 con 0x7f11d0101990 2024-04-21T08:44:31.425 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:31.426+0000 7f11d4982700 1 -- 172.21.15.152:0/2054138865 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f11d0062e80 con 0x7f11b80a3c40 2024-04-21T08:44:31.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:31 smithi152 ceph-mon[31401]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T08:44:31.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:31 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:44:31.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:31 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:44:31.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:31 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:31.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:31 smithi028 ceph-mon[28003]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T08:44:31.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:31 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:44:31.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:31 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:44:31.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:31 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:31.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:31 smithi171 ceph-mon[31358]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T08:44:31.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:31 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:44:31.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:31 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:44:31.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:31 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:32.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:32 smithi152 ceph-mon[31401]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:44:32.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:32 smithi152 ceph-mon[31401]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:32.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:32 smithi152 ceph-mon[31401]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T08:44:32.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:32 smithi028 ceph-mon[28003]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:44:32.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:32 smithi028 ceph-mon[28003]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:32.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:32 smithi028 ceph-mon[28003]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T08:44:32.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:32 smithi171 ceph-mon[31358]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:44:32.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:32 smithi171 ceph-mon[31358]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:32.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:32 smithi171 ceph-mon[31358]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T08:44:34.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:34 smithi152 ceph-mon[31401]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-21T08:44:34.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:34 smithi152 ceph-mon[31401]: from='client.? 172.21.15.152:0/3063615956' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7316601a-d423-480c-9039-27c15d5f4de4"}]: dispatch 2024-04-21T08:44:34.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:34 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7316601a-d423-480c-9039-27c15d5f4de4"}]: dispatch 2024-04-21T08:44:34.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:34 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7316601a-d423-480c-9039-27c15d5f4de4"}]': finished 2024-04-21T08:44:34.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:34 smithi152 ceph-mon[31401]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T08:44:34.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:34 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:34.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:34 smithi028 ceph-mon[28003]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-21T08:44:34.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:34 smithi028 ceph-mon[28003]: from='client.? 172.21.15.152:0/3063615956' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7316601a-d423-480c-9039-27c15d5f4de4"}]: dispatch 2024-04-21T08:44:34.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:34 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7316601a-d423-480c-9039-27c15d5f4de4"}]: dispatch 2024-04-21T08:44:34.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:34 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7316601a-d423-480c-9039-27c15d5f4de4"}]': finished 2024-04-21T08:44:34.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:34 smithi028 ceph-mon[28003]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T08:44:34.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:34 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:34.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:34 smithi171 ceph-mon[31358]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-21T08:44:34.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:34 smithi171 ceph-mon[31358]: from='client.? 172.21.15.152:0/3063615956' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7316601a-d423-480c-9039-27c15d5f4de4"}]: dispatch 2024-04-21T08:44:34.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:34 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7316601a-d423-480c-9039-27c15d5f4de4"}]: dispatch 2024-04-21T08:44:34.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:34 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7316601a-d423-480c-9039-27c15d5f4de4"}]': finished 2024-04-21T08:44:34.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:34 smithi171 ceph-mon[31358]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T08:44:34.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:34 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:35.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:35 smithi152 ceph-mon[31401]: from='client.? 172.21.15.152:0/3973150598' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:44:35.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:35 smithi028 ceph-mon[28003]: from='client.? 172.21.15.152:0/3973150598' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:44:35.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:35 smithi171 ceph-mon[31358]: from='client.? 172.21.15.152:0/3973150598' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:44:36.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:36 smithi152 ceph-mon[31401]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-21T08:44:36.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:36 smithi028 ceph-mon[28003]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-21T08:44:36.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:36 smithi171 ceph-mon[31358]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-21T08:44:38.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:38 smithi028 ceph-mon[28003]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-21T08:44:38.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:38 smithi152 ceph-mon[31401]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-21T08:44:38.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:38 smithi171 ceph-mon[31358]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-21T08:44:40.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:40 smithi152 ceph-mon[31401]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:40.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:40 smithi028 ceph-mon[28003]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:40.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:40 smithi171 ceph-mon[31358]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:41.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:41 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T08:44:41.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:41 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:41.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:41 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T08:44:41.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:41 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:41.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:41 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T08:44:41.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:41 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:42.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:42 smithi152 ceph-mon[31401]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:42.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:42 smithi152 ceph-mon[31401]: Deploying daemon osd.4 on smithi152 2024-04-21T08:44:42.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:42 smithi028 ceph-mon[28003]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:42.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:42 smithi028 ceph-mon[28003]: Deploying daemon osd.4 on smithi152 2024-04-21T08:44:42.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:42 smithi171 ceph-mon[31358]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:42.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:42 smithi171 ceph-mon[31358]: Deploying daemon osd.4 on smithi152 2024-04-21T08:44:44.709 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:44 smithi152 ceph-mon[31401]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:44.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:44 smithi028 ceph-mon[28003]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:44.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:44 smithi171 ceph-mon[31358]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:46.718 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:46 smithi152 ceph-mon[31401]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:46.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:46 smithi028 ceph-mon[28003]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:46.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:46 smithi171 ceph-mon[31358]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:47.789 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:47 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:47.789 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:47 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:47.789 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:47 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:47.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:47 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:47.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:47 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:47.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:47 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:47.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:47 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:47.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:47 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:47.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:47 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:49.249 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:48 smithi152 ceph-mon[31401]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:49.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:48 smithi028 ceph-mon[28003]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:49.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:48 smithi171 ceph-mon[31358]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:50.995 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 4 on host 'smithi152' 2024-04-21T08:44:50.995 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:50.995+0000 7f11bf7fe700 1 -- 172.21.15.152:0/2054138865 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f11d0062e80 con 0x7f11b80a3c40 2024-04-21T08:44:50.997 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:50.998+0000 7f11d4982700 1 -- 172.21.15.152:0/2054138865 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f11b80a3c40 msgr2=0x7f11b80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:50.998+0000 7f11d4982700 1 --2- 172.21.15.152:0/2054138865 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f11b80a3c40 0x7f11b80a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f11c4005950 tx=0x7f11c40058e0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:50.998+0000 7f11d4982700 1 -- 172.21.15.152:0/2054138865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11d0101990 msgr2=0x7f11d006bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:50.998+0000 7f11d4982700 1 --2- 172.21.15.152:0/2054138865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11d0101990 0x7f11d006bce0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f11c00038c0 tx=0x7f11c00047b0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:50.998+0000 7f11d4982700 1 -- 172.21.15.152:0/2054138865 shutdown_connections 2024-04-21T08:44:50.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:50.998+0000 7f11d4982700 1 --2- 172.21.15.152:0/2054138865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11d0103530 0x7f11d019ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:50.998+0000 7f11d4982700 1 --2- 172.21.15.152:0/2054138865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f11d0101990 0x7f11d006bce0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:50.998+0000 7f11d4982700 1 --2- 172.21.15.152:0/2054138865 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f11b80a3c40 0x7f11b80a6100 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:50.998+0000 7f11d4982700 1 --2- 172.21.15.152:0/2054138865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11d0102b90 0x7f11d019d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:50.998+0000 7f11d4982700 1 -- 172.21.15.152:0/2054138865 >> 172.21.15.152:0/2054138865 conn(0x7f11d00fcf30 msgr2=0x7f11d0105fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:50.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:50.998+0000 7f11d4982700 1 -- 172.21.15.152:0/2054138865 shutdown_connections 2024-04-21T08:44:50.998 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:44:50.998+0000 7f11d4982700 1 -- 172.21.15.152:0/2054138865 wait complete. 2024-04-21T08:44:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:51 smithi152 ceph-mon[31401]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:51.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:51.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:51.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:51 smithi152 ceph-mon[31401]: from='osd.4 [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:44:51.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:51 smithi152 ceph-mon[31401]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:44:51.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:51.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:51.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:51.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:51 smithi028 ceph-mon[28003]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:51.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:51.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:51.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:51.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:51.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:51.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:51 smithi028 ceph-mon[28003]: from='osd.4 [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:44:51.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:51 smithi028 ceph-mon[28003]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:44:51.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:51.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:51.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:51 smithi171 ceph-mon[31358]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:51 smithi171 ceph-mon[31358]: from='osd.4 [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:44:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:51 smithi171 ceph-mon[31358]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:44:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:44:51.729 DEBUG:teuthology.orchestra.run.smithi152:osd.4> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.4.service 2024-04-21T08:44:51.731 INFO:tasks.cephadm:Deploying osd.5 on smithi171 with /dev/vg_nvme/lv_3... 2024-04-21T08:44:51.731 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T08:44:51.978 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:44:52.265 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:52 smithi152 ceph-mon[31401]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T08:44:52.265 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:52 smithi152 ceph-mon[31401]: from='osd.4 [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:44:52.265 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:52 smithi152 ceph-mon[31401]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T08:44:52.266 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:52.266 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:52 smithi152 ceph-mon[31401]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:44:52.266 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 08:44:52 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-4[42919]: 2024-04-21T08:44:52.034+0000 7ff4b8535700 -1 osd.4 0 waiting for initial osdmap 2024-04-21T08:44:52.266 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 08:44:52 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-4[42919]: 2024-04-21T08:44:52.042+0000 7ff4b4328700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:44:52.310 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:52 smithi171 ceph-mon[31358]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T08:44:52.310 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:52 smithi171 ceph-mon[31358]: from='osd.4 [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:44:52.311 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:52 smithi171 ceph-mon[31358]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T08:44:52.311 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:52.311 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:52 smithi171 ceph-mon[31358]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:44:52.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:52 smithi028 ceph-mon[28003]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T08:44:52.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:52 smithi028 ceph-mon[28003]: from='osd.4 [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:44:52.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:52 smithi028 ceph-mon[28003]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T08:44:52.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:52.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:52 smithi028 ceph-mon[28003]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:44:53.235 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:53 smithi152 ceph-mon[31401]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:53.235 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:53 smithi152 ceph-mon[31401]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-04-21T08:44:53.235 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:53 smithi152 ceph-mon[31401]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T08:44:53.235 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:53 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:53.235 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:53 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:53.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:53 smithi171 ceph-mon[31358]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:53.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:53 smithi171 ceph-mon[31358]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-04-21T08:44:53.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:53 smithi171 ceph-mon[31358]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T08:44:53.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:53 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:53.301 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:53 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:53.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:53 smithi028 ceph-mon[28003]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:53.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:53 smithi028 ceph-mon[28003]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-04-21T08:44:53.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:53 smithi028 ceph-mon[28003]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T08:44:53.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:53 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:53.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:53 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:53.851 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T08:44:53.852 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T08:44:53.852 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-04-21T08:44:53.852 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-04-21T08:44:53.852 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130092 s, 806 MB/s 2024-04-21T08:44:53.852 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-04-21T08:44:54.162 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:54 smithi171 ceph-mon[31358]: purged_snaps scrub starts 2024-04-21T08:44:54.162 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:54 smithi171 ceph-mon[31358]: purged_snaps scrub ok 2024-04-21T08:44:54.162 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:54 smithi171 ceph-mon[31358]: osd.4 [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376] boot 2024-04-21T08:44:54.162 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:54 smithi171 ceph-mon[31358]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T08:44:54.162 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:54 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:54.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:54 smithi152 ceph-mon[31401]: purged_snaps scrub starts 2024-04-21T08:44:54.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:54 smithi152 ceph-mon[31401]: purged_snaps scrub ok 2024-04-21T08:44:54.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:54 smithi152 ceph-mon[31401]: osd.4 [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376] boot 2024-04-21T08:44:54.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:54 smithi152 ceph-mon[31401]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T08:44:54.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:54 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:54.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:54 smithi028 ceph-mon[28003]: purged_snaps scrub starts 2024-04-21T08:44:54.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:54 smithi028 ceph-mon[28003]: purged_snaps scrub ok 2024-04-21T08:44:54.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:54 smithi028 ceph-mon[28003]: osd.4 [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376] boot 2024-04-21T08:44:54.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:54 smithi028 ceph-mon[28003]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T08:44:54.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:54 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:44:54.441 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi171:vg_nvme/lv_3 2024-04-21T08:44:54.688 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:44:55.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:55 smithi152 ceph-mon[31401]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:55.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:55 smithi152 ceph-mon[31401]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T08:44:55.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:55.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:55.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:55.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:55.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:55.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:55.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:55.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:55.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:55 smithi028 ceph-mon[28003]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:55.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:55 smithi028 ceph-mon[28003]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T08:44:55.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:55.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:55.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:55.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:55.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:55.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:55.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:55.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:55 smithi171 ceph-mon[31358]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-21T08:44:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:55 smithi171 ceph-mon[31358]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T08:44:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:44:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:55.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:55.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:55.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:44:56.269 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.267+0000 7fc47844c700 1 -- 172.21.15.171:0/1546933228 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc470102b90 msgr2=0x7fc470102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:56.269 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.267+0000 7fc47844c700 1 --2- 172.21.15.171:0/1546933228 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc470102b90 0x7fc470102ff0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc464009990 tx=0x7fc464009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:56.269 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.267+0000 7fc47844c700 1 -- 172.21.15.171:0/1546933228 shutdown_connections 2024-04-21T08:44:56.270 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.267+0000 7fc47844c700 1 --2- 172.21.15.171:0/1546933228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc470101990 0x7fc470101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:56.270 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.267+0000 7fc47844c700 1 --2- 172.21.15.171:0/1546933228 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc470103530 0x7fc470109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:56.270 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.267+0000 7fc47844c700 1 --2- 172.21.15.171:0/1546933228 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc470102b90 0x7fc470102ff0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:56.270 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.267+0000 7fc47844c700 1 -- 172.21.15.171:0/1546933228 >> 172.21.15.171:0/1546933228 conn(0x7fc4700fcf30 msgr2=0x7fc4700ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:56.270 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.267+0000 7fc47844c700 1 -- 172.21.15.171:0/1546933228 shutdown_connections 2024-04-21T08:44:56.270 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.267+0000 7fc47844c700 1 -- 172.21.15.171:0/1546933228 wait complete. 2024-04-21T08:44:56.270 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc47844c700 1 Processor -- start 2024-04-21T08:44:56.270 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc47844c700 1 -- start start 2024-04-21T08:44:56.270 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc47844c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc470101990 0x7fc470196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:56.270 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc47844c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc470102b90 0x7fc470196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:56.270 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc47844c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc470103530 0x7fc47019dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:56.270 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc47844c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc47010c3b0 con 0x7fc470103530 2024-04-21T08:44:56.271 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc47844c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc47010c230 con 0x7fc470102b90 2024-04-21T08:44:56.271 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc47844c700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc47010c530 con 0x7fc470101990 2024-04-21T08:44:56.271 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc4761e8700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc470101990 0x7fc470196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:56.271 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc4761e8700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc470101990 0x7fc470196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:33014/0 (socket says 172.21.15.171:33014) 2024-04-21T08:44:56.271 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc4761e8700 1 -- 172.21.15.171:0/4059445965 learned_addr learned my addr 172.21.15.171:0/4059445965 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-04-21T08:44:56.271 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc4769e9700 1 --2- 172.21.15.171:0/4059445965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc470103530 0x7fc47019dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:56.271 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc4759e7700 1 --2- 172.21.15.171:0/4059445965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc470102b90 0x7fc470196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:56.271 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc4761e8700 1 -- 172.21.15.171:0/4059445965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc470102b90 msgr2=0x7fc470196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:56.271 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc4761e8700 1 --2- 172.21.15.171:0/4059445965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc470102b90 0x7fc470196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:56.271 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc4761e8700 1 -- 172.21.15.171:0/4059445965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc470103530 msgr2=0x7fc47019dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:56.271 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc4761e8700 1 --2- 172.21.15.171:0/4059445965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc470103530 0x7fc47019dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:56.271 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc4761e8700 1 -- 172.21.15.171:0/4059445965 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc47019e660 con 0x7fc470101990 2024-04-21T08:44:56.271 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.268+0000 7fc4759e7700 1 --2- 172.21.15.171:0/4059445965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc470102b90 0x7fc470196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:56.271 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.269+0000 7fc4761e8700 1 --2- 172.21.15.171:0/4059445965 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc470101990 0x7fc470196a00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc460012350 tx=0x7fc460012710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:56.271 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.269+0000 7fc46f7fe700 1 -- 172.21.15.171:0/4059445965 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc460012e40 con 0x7fc470101990 2024-04-21T08:44:56.272 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.269+0000 7fc46f7fe700 1 -- 172.21.15.171:0/4059445965 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4600092e0 con 0x7fc470101990 2024-04-21T08:44:56.272 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.269+0000 7fc47844c700 1 -- 172.21.15.171:0/4059445965 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc47019e8f0 con 0x7fc470101990 2024-04-21T08:44:56.272 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.269+0000 7fc46f7fe700 1 -- 172.21.15.171:0/4059445965 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc460007500 con 0x7fc470101990 2024-04-21T08:44:56.272 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.269+0000 7fc47844c700 1 -- 172.21.15.171:0/4059445965 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc47019ee80 con 0x7fc470101990 2024-04-21T08:44:56.272 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.270+0000 7fc47844c700 1 -- 172.21.15.171:0/4059445965 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc470069730 con 0x7fc470101990 2024-04-21T08:44:56.273 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.271+0000 7fc46f7fe700 1 -- 172.21.15.171:0/4059445965 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc46001f030 con 0x7fc470101990 2024-04-21T08:44:56.273 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.271+0000 7fc46f7fe700 1 --2- 172.21.15.171:0/4059445965 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc4580a3cc0 0x7fc4580a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:56.273 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.272+0000 7fc46f7fe700 1 -- 172.21.15.171:0/4059445965 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fc4600d3300 con 0x7fc470101990 2024-04-21T08:44:56.273 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.272+0000 7fc4759e7700 1 --2- 172.21.15.171:0/4059445965 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc4580a3cc0 0x7fc4580a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:56.273 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.272+0000 7fc4759e7700 1 --2- 172.21.15.171:0/4059445965 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc4580a3cc0 0x7fc4580a6180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc464006010 tx=0x7fc46400de40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:56.276 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.276+0000 7fc46f7fe700 1 -- 172.21.15.171:0/4059445965 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc460026020 con 0x7fc470101990 2024-04-21T08:44:56.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:56 smithi152 ceph-mon[31401]: Detected new or changed devices on smithi152 2024-04-21T08:44:56.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:56 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi152 to 8544M 2024-04-21T08:44:56.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:56 smithi028 ceph-mon[28003]: Detected new or changed devices on smithi152 2024-04-21T08:44:56.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:56 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi152 to 8544M 2024-04-21T08:44:56.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:56 smithi171 ceph-mon[31358]: Detected new or changed devices on smithi152 2024-04-21T08:44:56.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:56 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi152 to 8544M 2024-04-21T08:44:56.486 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:44:56.486+0000 7fc47844c700 1 -- 172.21.15.171:0/4059445965 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fc470062f40 con 0x7fc4580a3cc0 2024-04-21T08:44:57.316 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:57 smithi171 ceph-mon[31358]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:44:57.317 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:57 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:44:57.317 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:57 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:44:57.317 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:57 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:57.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:57 smithi152 ceph-mon[31401]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:44:57.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:57 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:44:57.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:57 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:44:57.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:57 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:57.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:57 smithi028 ceph-mon[28003]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:44:57.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:57 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:44:57.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:57 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:44:57.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:57 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:58.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:58 smithi152 ceph-mon[31401]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:58.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:58 smithi028 ceph-mon[28003]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:58.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:58 smithi171 ceph-mon[31358]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:59.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:59 smithi152 ceph-mon[31401]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:44:59.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:59 smithi152 ceph-mon[31401]: from='client.? 172.21.15.171:0/3837942346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee83cf31-51ec-4b8f-a5f9-8457b98c44c3"}]: dispatch 2024-04-21T08:44:59.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:59 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee83cf31-51ec-4b8f-a5f9-8457b98c44c3"}]: dispatch 2024-04-21T08:44:59.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:59 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee83cf31-51ec-4b8f-a5f9-8457b98c44c3"}]': finished 2024-04-21T08:44:59.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:59 smithi152 ceph-mon[31401]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T08:44:59.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:44:59 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:44:59.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:59 smithi028 ceph-mon[28003]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:44:59.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:59 smithi028 ceph-mon[28003]: from='client.? 172.21.15.171:0/3837942346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee83cf31-51ec-4b8f-a5f9-8457b98c44c3"}]: dispatch 2024-04-21T08:44:59.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:59 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee83cf31-51ec-4b8f-a5f9-8457b98c44c3"}]: dispatch 2024-04-21T08:44:59.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:59 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee83cf31-51ec-4b8f-a5f9-8457b98c44c3"}]': finished 2024-04-21T08:44:59.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:59 smithi028 ceph-mon[28003]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T08:44:59.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:44:59 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:44:59.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:59 smithi171 ceph-mon[31358]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:44:59.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:59 smithi171 ceph-mon[31358]: from='client.? 172.21.15.171:0/3837942346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee83cf31-51ec-4b8f-a5f9-8457b98c44c3"}]: dispatch 2024-04-21T08:44:59.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:59 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee83cf31-51ec-4b8f-a5f9-8457b98c44c3"}]: dispatch 2024-04-21T08:44:59.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:59 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee83cf31-51ec-4b8f-a5f9-8457b98c44c3"}]': finished 2024-04-21T08:44:59.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:59 smithi171 ceph-mon[31358]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T08:44:59.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:44:59 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:00.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:00 smithi152 ceph-mon[31401]: from='client.? 172.21.15.171:0/2081101922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:45:00.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:00 smithi028 ceph-mon[28003]: from='client.? 172.21.15.171:0/2081101922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:45:00.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:00 smithi171 ceph-mon[31358]: from='client.? 172.21.15.171:0/2081101922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:45:01.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:01 smithi152 ceph-mon[31401]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:01.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:01 smithi028 ceph-mon[28003]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:01.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:01 smithi171 ceph-mon[31358]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:03.236 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:03 smithi171 ceph-mon[31358]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:03.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:03 smithi152 ceph-mon[31401]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:03.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:03 smithi028 ceph-mon[28003]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:05.247 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:05 smithi171 ceph-mon[31358]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:05.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:05 smithi152 ceph-mon[31401]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:05.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:05 smithi028 ceph-mon[28003]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:07.228 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:07 smithi171 ceph-mon[31358]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:07.228 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T08:45:07.228 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:07.228 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:07 smithi171 ceph-mon[31358]: Deploying daemon osd.5 on smithi171 2024-04-21T08:45:07.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:07 smithi152 ceph-mon[31401]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:07.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T08:45:07.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:07.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:07 smithi152 ceph-mon[31401]: Deploying daemon osd.5 on smithi171 2024-04-21T08:45:07.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:07 smithi028 ceph-mon[28003]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:07.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T08:45:07.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:07.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:07 smithi028 ceph-mon[28003]: Deploying daemon osd.5 on smithi171 2024-04-21T08:45:09.271 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:09 smithi171 ceph-mon[31358]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:09.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:09 smithi152 ceph-mon[31401]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:09.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:09 smithi028 ceph-mon[28003]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:11.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:11 smithi152 ceph-mon[31401]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:11.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:11 smithi028 ceph-mon[28003]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:11.355 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:11 smithi171 ceph-mon[31358]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:13.347 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:12 smithi171 ceph-mon[31358]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:13.347 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:13.348 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:13.348 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:13.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:12 smithi152 ceph-mon[31401]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:13.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:13.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:13.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:13.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:12 smithi028 ceph-mon[28003]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:13.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:13.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:13.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:15.208 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:14 smithi171 ceph-mon[31358]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:15.208 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:14 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:15.208 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:14 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:15.208 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:14 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:15.208 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:14 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:15.209 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:14 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:15.254 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 5 on host 'smithi171' 2024-04-21T08:45:15.254 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:45:15.253+0000 7fc46f7fe700 1 -- 172.21.15.171:0/4059445965 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc470062f40 con 0x7fc4580a3cc0 2024-04-21T08:45:15.256 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:45:15.256+0000 7fc47844c700 1 -- 172.21.15.171:0/4059445965 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc4580a3cc0 msgr2=0x7fc4580a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.256 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:45:15.256+0000 7fc47844c700 1 --2- 172.21.15.171:0/4059445965 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc4580a3cc0 0x7fc4580a6180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc464006010 tx=0x7fc46400de40 comp rx=0 tx=0).stop 2024-04-21T08:45:15.256 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:45:15.256+0000 7fc47844c700 1 -- 172.21.15.171:0/4059445965 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc470101990 msgr2=0x7fc470196a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.256 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:45:15.256+0000 7fc47844c700 1 --2- 172.21.15.171:0/4059445965 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc470101990 0x7fc470196a00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc460012350 tx=0x7fc460012710 comp rx=0 tx=0).stop 2024-04-21T08:45:15.256 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:45:15.256+0000 7fc47844c700 1 -- 172.21.15.171:0/4059445965 shutdown_connections 2024-04-21T08:45:15.256 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:45:15.256+0000 7fc47844c700 1 --2- 172.21.15.171:0/4059445965 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc4580a3cc0 0x7fc4580a6180 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.257 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:45:15.256+0000 7fc47844c700 1 --2- 172.21.15.171:0/4059445965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc470103530 0x7fc47019dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.257 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:45:15.256+0000 7fc47844c700 1 --2- 172.21.15.171:0/4059445965 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc470101990 0x7fc470196a00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.257 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:45:15.256+0000 7fc47844c700 1 --2- 172.21.15.171:0/4059445965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc470102b90 0x7fc470196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.257 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:45:15.256+0000 7fc47844c700 1 -- 172.21.15.171:0/4059445965 >> 172.21.15.171:0/4059445965 conn(0x7fc4700fcf30 msgr2=0x7fc470105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:15.257 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:45:15.256+0000 7fc47844c700 1 -- 172.21.15.171:0/4059445965 shutdown_connections 2024-04-21T08:45:15.257 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:45:15.256+0000 7fc47844c700 1 -- 172.21.15.171:0/4059445965 wait complete. 2024-04-21T08:45:15.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:14 smithi152 ceph-mon[31401]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:15.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:14 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:15.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:14 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:15.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:14 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:15.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:14 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:15.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:14 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:15.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:14 smithi028 ceph-mon[28003]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:15.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:15.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:15.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:15.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:15.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:15.907 DEBUG:teuthology.orchestra.run.smithi171:osd.5> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.5.service 2024-04-21T08:45:15.909 INFO:tasks.cephadm:Deploying osd.6 on smithi028 with /dev/vg_nvme/lv_2... 2024-04-21T08:45:15.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T08:45:16.167 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:45:16.373 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:16.374 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:16.374 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:16.374 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:16 smithi171 ceph-mon[31358]: from='osd.5 [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:45:16.374 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:16 smithi171 ceph-mon[31358]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:45:16.547 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:16.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:16.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:16.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:16 smithi028 ceph-mon[28003]: from='osd.5 [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:45:16.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:16 smithi028 ceph-mon[28003]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:45:16.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:16.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:16.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:16.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:16 smithi152 ceph-mon[31401]: from='osd.5 [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:45:16.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:16 smithi152 ceph-mon[31401]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:45:17.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:17 smithi171 ceph-mon[31358]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:17.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:17 smithi171 ceph-mon[31358]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T08:45:17.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:17 smithi171 ceph-mon[31358]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T08:45:17.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:17 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:17.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:17 smithi171 ceph-mon[31358]: from='osd.5 [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:45:17.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:17 smithi171 ceph-mon[31358]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:45:17.593 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:17 smithi028 ceph-mon[28003]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:17.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:17 smithi028 ceph-mon[28003]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T08:45:17.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:17 smithi028 ceph-mon[28003]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T08:45:17.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:17 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:17.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:17 smithi028 ceph-mon[28003]: from='osd.5 [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:45:17.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:17 smithi028 ceph-mon[28003]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:45:17.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:17 smithi152 ceph-mon[31401]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:17.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:17 smithi152 ceph-mon[31401]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T08:45:17.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:17 smithi152 ceph-mon[31401]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T08:45:17.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:17 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:17.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:17 smithi152 ceph-mon[31401]: from='osd.5 [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:45:17.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:17 smithi152 ceph-mon[31401]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:45:18.120 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 08:45:17 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-5[43487]: 2024-04-21T08:45:17.696+0000 7f5abe761700 -1 osd.5 0 waiting for initial osdmap 2024-04-21T08:45:18.121 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 08:45:17 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-5[43487]: 2024-04-21T08:45:17.703+0000 7f5ab5d4b700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:45:18.185 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T08:45:18.185 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T08:45:18.185 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-21T08:45:18.185 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-21T08:45:18.185 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00859265 s, 1.2 GB/s 2024-04-21T08:45:18.185 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-21T08:45:18.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:18 smithi028 ceph-mon[28003]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-21T08:45:18.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:18 smithi028 ceph-mon[28003]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T08:45:18.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:18 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:18.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:18 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:18.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:18 smithi028 ceph-mon[28003]: from='osd.5 ' entity='osd.5' 2024-04-21T08:45:18.521 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:18 smithi152 ceph-mon[31401]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-21T08:45:18.521 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:18 smithi152 ceph-mon[31401]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T08:45:18.521 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:18 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:18.521 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:18 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:18.521 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:18 smithi152 ceph-mon[31401]: from='osd.5 ' entity='osd.5' 2024-04-21T08:45:18.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:18 smithi171 ceph-mon[31358]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-21T08:45:18.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:18 smithi171 ceph-mon[31358]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T08:45:18.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:18 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:18.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:18 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:18.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:18 smithi171 ceph-mon[31358]: from='osd.5 ' entity='osd.5' 2024-04-21T08:45:18.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-04-21T08:45:19.125 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:45:19.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:19 smithi028 ceph-mon[28003]: purged_snaps scrub starts 2024-04-21T08:45:19.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:19 smithi028 ceph-mon[28003]: purged_snaps scrub ok 2024-04-21T08:45:19.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:19 smithi028 ceph-mon[28003]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:19.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:19 smithi028 ceph-mon[28003]: osd.5 [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239] boot 2024-04-21T08:45:19.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:19 smithi028 ceph-mon[28003]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T08:45:19.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:19 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:19.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:19 smithi028 ceph-mon[28003]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T08:45:19.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:19 smithi152 ceph-mon[31401]: purged_snaps scrub starts 2024-04-21T08:45:19.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:19 smithi152 ceph-mon[31401]: purged_snaps scrub ok 2024-04-21T08:45:19.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:19 smithi152 ceph-mon[31401]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:19.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:19 smithi152 ceph-mon[31401]: osd.5 [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239] boot 2024-04-21T08:45:19.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:19 smithi152 ceph-mon[31401]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T08:45:19.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:19 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:19.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:19 smithi152 ceph-mon[31401]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T08:45:19.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:19 smithi171 ceph-mon[31358]: purged_snaps scrub starts 2024-04-21T08:45:19.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:19 smithi171 ceph-mon[31358]: purged_snaps scrub ok 2024-04-21T08:45:19.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:19 smithi171 ceph-mon[31358]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:45:19.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:19 smithi171 ceph-mon[31358]: osd.5 [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239] boot 2024-04-21T08:45:19.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:19 smithi171 ceph-mon[31358]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T08:45:19.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:19 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:19.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:19 smithi171 ceph-mon[31358]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T08:45:20.556 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:20 smithi028 ceph-mon[28003]: Detected new or changed devices on smithi171 2024-04-21T08:45:20.557 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:20 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:20.557 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:20 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:20.557 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:20 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:20.557 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:20 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:20.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:20 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi171 to 8544M 2024-04-21T08:45:20.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:20 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:20.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:20 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:20.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:20 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:20.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:20 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:20.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:20 smithi028 ceph-mon[28003]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:20.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:20 smithi028 ceph-mon[28003]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T08:45:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:20 smithi171 ceph-mon[31358]: Detected new or changed devices on smithi171 2024-04-21T08:45:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:20 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:20 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:20 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:20 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:20 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi171 to 8544M 2024-04-21T08:45:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:20 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:20.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:20 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:20.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:20 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:20.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:20 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:20.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:20 smithi171 ceph-mon[31358]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:20.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:20 smithi171 ceph-mon[31358]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T08:45:20.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.779+0000 7f505fc0d700 1 -- 172.21.15.28:0/848692901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50581026c0 msgr2=0x7f5058102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:20.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.779+0000 7f505fc0d700 1 --2- 172.21.15.28:0/848692901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50581026c0 0x7f5058102b40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f505000b0a0 tx=0x7f505000b3b0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.779+0000 7f505fc0d700 1 -- 172.21.15.28:0/848692901 shutdown_connections 2024-04-21T08:45:20.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.779+0000 7f505fc0d700 1 --2- 172.21.15.28:0/848692901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50581026c0 0x7f5058102b40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.779+0000 7f505fc0d700 1 --2- 172.21.15.28:0/848692901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50581013a0 0x7f50581017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.779+0000 7f505fc0d700 1 --2- 172.21.15.28:0/848692901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5058101d00 0x7f5058102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.779+0000 7f505fc0d700 1 -- 172.21.15.28:0/848692901 >> 172.21.15.28:0/848692901 conn(0x7f50580fcf40 msgr2=0x7f50580ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:20.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.779+0000 7f505fc0d700 1 -- 172.21.15.28:0/848692901 shutdown_connections 2024-04-21T08:45:20.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.779+0000 7f505fc0d700 1 -- 172.21.15.28:0/848692901 wait complete. 2024-04-21T08:45:20.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.779+0000 7f505fc0d700 1 Processor -- start 2024-04-21T08:45:20.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.779+0000 7f505fc0d700 1 -- start start 2024-04-21T08:45:20.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505fc0d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50581013a0 0x7f505819ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:20.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505fc0d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5058101d00 0x7f505819b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:20.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505fc0d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50581026c0 0x7f50581a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:20.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505fc0d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5058110700 con 0x7f50581026c0 2024-04-21T08:45:20.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505fc0d700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5058110580 con 0x7f50581013a0 2024-04-21T08:45:20.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505fc0d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5058110880 con 0x7f5058101d00 2024-04-21T08:45:20.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505e1aa700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50581026c0 0x7f50581a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:20.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505e1aa700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50581026c0 0x7f50581a2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36622/0 (socket says 172.21.15.28:36622) 2024-04-21T08:45:20.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505e1aa700 1 -- 172.21.15.28:0/845103100 learned_addr learned my addr 172.21.15.28:0/845103100 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:45:20.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505d9a9700 1 --2- 172.21.15.28:0/845103100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50581013a0 0x7f505819ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:20.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505d1a8700 1 --2- 172.21.15.28:0/845103100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5058101d00 0x7f505819b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:20.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505e1aa700 1 -- 172.21.15.28:0/845103100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5058101d00 msgr2=0x7f505819b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:20.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505e1aa700 1 --2- 172.21.15.28:0/845103100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5058101d00 0x7f505819b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505e1aa700 1 -- 172.21.15.28:0/845103100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50581013a0 msgr2=0x7f505819ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:20.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505e1aa700 1 --2- 172.21.15.28:0/845103100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50581013a0 0x7f505819ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505e1aa700 1 -- 172.21.15.28:0/845103100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50581a29e0 con 0x7f50581026c0 2024-04-21T08:45:20.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f505e1aa700 1 --2- 172.21.15.28:0/845103100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50581026c0 0x7f50581a2310 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f505000b830 tx=0x7f5050004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:20.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.780+0000 7f504effd700 1 -- 172.21.15.28:0/845103100 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5050021070 con 0x7f50581026c0 2024-04-21T08:45:20.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.781+0000 7f504effd700 1 -- 172.21.15.28:0/845103100 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5050003c00 con 0x7f50581026c0 2024-04-21T08:45:20.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.781+0000 7f505fc0d700 1 -- 172.21.15.28:0/845103100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50581a2c70 con 0x7f50581026c0 2024-04-21T08:45:20.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.781+0000 7f504effd700 1 -- 172.21.15.28:0/845103100 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5050004810 con 0x7f50581026c0 2024-04-21T08:45:20.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.781+0000 7f505fc0d700 1 -- 172.21.15.28:0/845103100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50581a3120 con 0x7f50581026c0 2024-04-21T08:45:20.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.782+0000 7f505fc0d700 1 -- 172.21.15.28:0/845103100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5058069730 con 0x7f50581026c0 2024-04-21T08:45:20.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.783+0000 7f504effd700 1 -- 172.21.15.28:0/845103100 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5050007a00 con 0x7f50581026c0 2024-04-21T08:45:20.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.783+0000 7f504effd700 1 --2- 172.21.15.28:0/845103100 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50440a3c40 0x7f50440a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:20.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.784+0000 7f505d9a9700 1 --2- 172.21.15.28:0/845103100 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50440a3c40 0x7f50440a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:20.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.784+0000 7f504effd700 1 -- 172.21.15.28:0/845103100 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4533+0+0 (secure 0 0 0) 0x7f505000f020 con 0x7f50581026c0 2024-04-21T08:45:20.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.784+0000 7f505d9a9700 1 --2- 172.21.15.28:0/845103100 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50440a3c40 0x7f50440a6100 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f505819c270 tx=0x7f505400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:20.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:20.787+0000 7f504effd700 1 -- 172.21.15.28:0/845103100 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f505002a020 con 0x7f50581026c0 2024-04-21T08:45:20.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:20 smithi152 ceph-mon[31401]: Detected new or changed devices on smithi171 2024-04-21T08:45:20.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:20 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:20.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:20 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:20.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:20 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:20.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:20 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:20.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:20 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi171 to 8544M 2024-04-21T08:45:20.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:20 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:20.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:20 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:20.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:20 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:20.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:20 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:20.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:20 smithi152 ceph-mon[31401]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:20.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:20 smithi152 ceph-mon[31401]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T08:45:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:21.005+0000 7f505fc0d700 1 -- 172.21.15.28:0/845103100 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f5058062e80 con 0x7f50440a3c40 2024-04-21T08:45:21.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:21 smithi028 ceph-mon[28003]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T08:45:21.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:21 smithi028 ceph-mon[28003]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:21.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:21 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:45:21.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:21 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:45:21.711 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:21 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:21.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:21 smithi152 ceph-mon[31401]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T08:45:21.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:21 smithi152 ceph-mon[31401]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:21.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:21 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:45:21.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:21 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:45:21.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:21 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:21.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:21 smithi171 ceph-mon[31358]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T08:45:21.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:21 smithi171 ceph-mon[31358]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:21.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:45:21.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:45:21.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:22.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:22 smithi028 ceph-mon[28003]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:45:22.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:22 smithi028 ceph-mon[28003]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T08:45:22.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:22 smithi152 ceph-mon[31401]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:45:22.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:22 smithi152 ceph-mon[31401]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T08:45:22.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:22 smithi171 ceph-mon[31358]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:45:22.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:22 smithi171 ceph-mon[31358]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T08:45:24.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:24 smithi152 ceph-mon[31401]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-21T08:45:24.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:24 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2131586010' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab718a5d-64ee-43c9-b8e8-212ce636329c"}]: dispatch 2024-04-21T08:45:24.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:24 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2131586010' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab718a5d-64ee-43c9-b8e8-212ce636329c"}]': finished 2024-04-21T08:45:24.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:24 smithi152 ceph-mon[31401]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T08:45:24.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:24 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:24.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:24 smithi028 ceph-mon[28003]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-21T08:45:24.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:24 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2131586010' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab718a5d-64ee-43c9-b8e8-212ce636329c"}]: dispatch 2024-04-21T08:45:24.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:24 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2131586010' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab718a5d-64ee-43c9-b8e8-212ce636329c"}]': finished 2024-04-21T08:45:24.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:24 smithi028 ceph-mon[28003]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T08:45:24.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:24 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:24.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:24 smithi171 ceph-mon[31358]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-21T08:45:24.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:24 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2131586010' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab718a5d-64ee-43c9-b8e8-212ce636329c"}]: dispatch 2024-04-21T08:45:24.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:24 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2131586010' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab718a5d-64ee-43c9-b8e8-212ce636329c"}]': finished 2024-04-21T08:45:24.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:24 smithi171 ceph-mon[31358]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T08:45:24.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:24 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:25.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:25 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2658451129' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:45:25.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:25 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2658451129' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:45:25.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:25 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2658451129' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:45:26.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:26 smithi028 ceph-mon[28003]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T08:45:26.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:26 smithi152 ceph-mon[31401]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T08:45:26.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:26 smithi171 ceph-mon[31358]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T08:45:28.766 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:28 smithi171 ceph-mon[31358]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-21T08:45:28.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:28 smithi152 ceph-mon[31401]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-21T08:45:28.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:28 smithi028 ceph-mon[28003]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-21T08:45:30.663 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:30 smithi028 ceph-mon[28003]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:30.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:30 smithi152 ceph-mon[31401]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:30.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:30 smithi171 ceph-mon[31358]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:32.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:32 smithi028 ceph-mon[28003]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:32.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:32 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T08:45:32.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:32 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:32.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:32 smithi028 ceph-mon[28003]: Deploying daemon osd.6 on smithi028 2024-04-21T08:45:32.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:32 smithi152 ceph-mon[31401]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:32.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:32 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T08:45:32.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:32 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:32.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:32 smithi152 ceph-mon[31401]: Deploying daemon osd.6 on smithi028 2024-04-21T08:45:32.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:32 smithi171 ceph-mon[31358]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:32.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:32 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T08:45:32.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:32 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:32.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:32 smithi171 ceph-mon[31358]: Deploying daemon osd.6 on smithi028 2024-04-21T08:45:34.549 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:34 smithi028 ceph-mon[28003]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:34.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:34 smithi152 ceph-mon[31401]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:34.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:34 smithi171 ceph-mon[31358]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:36 smithi028 ceph-mon[28003]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:36.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:36 smithi152 ceph-mon[31401]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:36.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:36 smithi171 ceph-mon[31358]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:39.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:38 smithi152 ceph-mon[31401]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:39.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:38 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:39.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:38 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:39.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:38 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:39.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:38 smithi028 ceph-mon[28003]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:39.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:38 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:39.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:38 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:39.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:38 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:39.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:38 smithi171 ceph-mon[31358]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:39.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:38 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:39.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:38 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:39.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:38 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:40.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:40 smithi028 ceph-mon[28003]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:41.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:40 smithi152 ceph-mon[31401]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:41.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:40 smithi171 ceph-mon[31358]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:41.820+0000 7f504effd700 1 -- 172.21.15.28:0/845103100 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5058062e80 con 0x7f50440a3c40 2024-04-21T08:45:41.820 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 6 on host 'smithi028' 2024-04-21T08:45:41.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:41.823+0000 7f505fc0d700 1 -- 172.21.15.28:0/845103100 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50440a3c40 msgr2=0x7f50440a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:41.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:41.823+0000 7f505fc0d700 1 --2- 172.21.15.28:0/845103100 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50440a3c40 0x7f50440a6100 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f505819c270 tx=0x7f505400b410 comp rx=0 tx=0).stop 2024-04-21T08:45:41.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:41.823+0000 7f505fc0d700 1 -- 172.21.15.28:0/845103100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50581026c0 msgr2=0x7f50581a2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:41.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:41.823+0000 7f505fc0d700 1 --2- 172.21.15.28:0/845103100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50581026c0 0x7f50581a2310 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f505000b830 tx=0x7f5050004e40 comp rx=0 tx=0).stop 2024-04-21T08:45:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:41.823+0000 7f505fc0d700 1 -- 172.21.15.28:0/845103100 shutdown_connections 2024-04-21T08:45:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:41.823+0000 7f505fc0d700 1 --2- 172.21.15.28:0/845103100 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50440a3c40 0x7f50440a6100 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:41.823+0000 7f505fc0d700 1 --2- 172.21.15.28:0/845103100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50581026c0 0x7f50581a2310 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:41.823+0000 7f505fc0d700 1 --2- 172.21.15.28:0/845103100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50581013a0 0x7f505819ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:41.823+0000 7f505fc0d700 1 --2- 172.21.15.28:0/845103100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5058101d00 0x7f505819b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:41.823+0000 7f505fc0d700 1 -- 172.21.15.28:0/845103100 >> 172.21.15.28:0/845103100 conn(0x7f50580fcf40 msgr2=0x7f50581002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:41.823+0000 7f505fc0d700 1 -- 172.21.15.28:0/845103100 shutdown_connections 2024-04-21T08:45:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:45:41.823+0000 7f505fc0d700 1 -- 172.21.15.28:0/845103100 wait complete. 2024-04-21T08:45:42.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:41 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:42.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:41 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:42.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:41 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:42.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:41 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:42.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:41 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:42.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:41 smithi152 ceph-mon[31401]: from='osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T08:45:42.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:41 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:42.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:41 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:42.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:41 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:42.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:41 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:42.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:41 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:42.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:41 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:42.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:41 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:42.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:41 smithi028 ceph-mon[28003]: from='osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T08:45:42.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:41 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:42.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:41 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:42.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:41 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:42.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:41 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:42.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:41 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:42.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:41 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:42.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:41 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:42.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:41 smithi171 ceph-mon[31358]: from='osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T08:45:42.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:41 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:42.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:41 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:42.502 DEBUG:teuthology.orchestra.run.smithi028:osd.6> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.6.service 2024-04-21T08:45:42.505 INFO:tasks.cephadm:Deploying osd.7 on smithi152 with /dev/vg_nvme/lv_2... 2024-04-21T08:45:42.505 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T08:45:42.752 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:45:43.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:42 smithi152 ceph-mon[31401]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:42 smithi152 ceph-mon[31401]: from='osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T08:45:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:42 smithi152 ceph-mon[31401]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T08:45:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:42 smithi152 ceph-mon[31401]: from='osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-21T08:45:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:43.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:42 smithi028 ceph-mon[28003]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:43.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:42 smithi028 ceph-mon[28003]: from='osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T08:45:43.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:42 smithi028 ceph-mon[28003]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T08:45:43.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:42 smithi028 ceph-mon[28003]: from='osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-21T08:45:43.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:43.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:43.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:42 smithi171 ceph-mon[31358]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:43.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:42 smithi171 ceph-mon[31358]: from='osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T08:45:43.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:42 smithi171 ceph-mon[31358]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T08:45:43.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:42 smithi171 ceph-mon[31358]: from='osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-21T08:45:43.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:43.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:43.515 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 08:45:43 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-6[56290]: 2024-04-21T08:45:43.277+0000 7f7f09e1b700 -1 osd.6 0 waiting for initial osdmap 2024-04-21T08:45:43.515 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 08:45:43 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-6[56290]: 2024-04-21T08:45:43.285+0000 7f7f01c06700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:45:44.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:43 smithi152 ceph-mon[31401]: from='osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-21T08:45:44.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:43 smithi152 ceph-mon[31401]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T08:45:44.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:43 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:44.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:43 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:44.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:43 smithi152 ceph-mon[31401]: from='osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122]' entity='osd.6' 2024-04-21T08:45:44.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:43 smithi028 ceph-mon[28003]: from='osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-21T08:45:44.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:43 smithi028 ceph-mon[28003]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T08:45:44.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:43 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:44.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:43 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:44.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:43 smithi028 ceph-mon[28003]: from='osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122]' entity='osd.6' 2024-04-21T08:45:44.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:43 smithi171 ceph-mon[31358]: from='osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-21T08:45:44.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:43 smithi171 ceph-mon[31358]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T08:45:44.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:43 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:44.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:43 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:44.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:43 smithi171 ceph-mon[31358]: from='osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122]' entity='osd.6' 2024-04-21T08:45:44.993 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T08:45:44.993 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T08:45:44.993 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-04-21T08:45:44.993 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-04-21T08:45:44.993 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114715 s, 914 MB/s 2024-04-21T08:45:44.993 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-04-21T08:45:45.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:44 smithi152 ceph-mon[31401]: purged_snaps scrub starts 2024-04-21T08:45:45.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:44 smithi152 ceph-mon[31401]: purged_snaps scrub ok 2024-04-21T08:45:45.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:44 smithi152 ceph-mon[31401]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:45.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:44 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:45.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:44 smithi152 ceph-mon[31401]: osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122] boot 2024-04-21T08:45:45.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:44 smithi152 ceph-mon[31401]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T08:45:45.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:44 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:45.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:44 smithi028 ceph-mon[28003]: purged_snaps scrub starts 2024-04-21T08:45:45.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:44 smithi028 ceph-mon[28003]: purged_snaps scrub ok 2024-04-21T08:45:45.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:44 smithi028 ceph-mon[28003]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:45.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:44 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:45.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:44 smithi028 ceph-mon[28003]: osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122] boot 2024-04-21T08:45:45.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:44 smithi028 ceph-mon[28003]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T08:45:45.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:44 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:44 smithi171 ceph-mon[31358]: purged_snaps scrub starts 2024-04-21T08:45:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:44 smithi171 ceph-mon[31358]: purged_snaps scrub ok 2024-04-21T08:45:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:44 smithi171 ceph-mon[31358]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:45:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:44 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:44 smithi171 ceph-mon[31358]: osd.6 [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122] boot 2024-04-21T08:45:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:44 smithi171 ceph-mon[31358]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T08:45:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:44 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:45.662 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi152:vg_nvme/lv_2 2024-04-21T08:45:45.904 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:45:46.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:46 smithi152 ceph-mon[31401]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T08:45:46.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:46 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:46.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:46 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:46.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:46 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:46.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:46 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:46.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:46 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:46.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:46 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:46.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:46 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:46.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:46 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:46.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:46 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:46.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:46 smithi028 ceph-mon[28003]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T08:45:46.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:46 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:46.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:46 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:46.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:46 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:46.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:46 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:46.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:46 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:46.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:46 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:46.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:46 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:46.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:46 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:46.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:46 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:46.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:46 smithi171 ceph-mon[31358]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T08:45:46.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:46 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:46.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:46 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:46.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:46 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:46.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:46 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:46.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:46 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:46.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:46 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:46.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:46 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:46.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:46 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:46.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:46 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:45:47.295 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:47 smithi171 ceph-mon[31358]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:47.295 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:47 smithi171 ceph-mon[31358]: Detected new or changed devices on smithi028 2024-04-21T08:45:47.295 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:47 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi028 to 5697M 2024-04-21T08:45:47.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:47 smithi152 ceph-mon[31401]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:47.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:47 smithi152 ceph-mon[31401]: Detected new or changed devices on smithi028 2024-04-21T08:45:47.561 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:47 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi028 to 5697M 2024-04-21T08:45:47.564 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.565+0000 7f7c21ac5700 1 -- 172.21.15.152:0/368268250 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c1c101be0 msgr2=0x7f7c1c102060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:47.564 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.565+0000 7f7c21ac5700 1 --2- 172.21.15.152:0/368268250 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c1c101be0 0x7f7c1c102060 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009990 tx=0x7f7c04009ca0 comp rx=0 tx=0).stop 2024-04-21T08:45:47.564 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.565+0000 7f7c21ac5700 1 -- 172.21.15.152:0/368268250 shutdown_connections 2024-04-21T08:45:47.565 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.565+0000 7f7c21ac5700 1 --2- 172.21.15.152:0/368268250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c1c1025a0 0x7f7c1c109e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:47.565 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.565+0000 7f7c21ac5700 1 --2- 172.21.15.152:0/368268250 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7c1c103010 0x7f7c1c103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:47.565 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.565+0000 7f7c21ac5700 1 --2- 172.21.15.152:0/368268250 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c1c101be0 0x7f7c1c102060 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:47.565 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.565+0000 7f7c21ac5700 1 -- 172.21.15.152:0/368268250 >> 172.21.15.152:0/368268250 conn(0x7f7c1c0fc6d0 msgr2=0x7f7c1c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:47.565 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.565+0000 7f7c21ac5700 1 -- 172.21.15.152:0/368268250 shutdown_connections 2024-04-21T08:45:47.565 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.565+0000 7f7c21ac5700 1 -- 172.21.15.152:0/368268250 wait complete. 2024-04-21T08:45:47.567 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c21ac5700 1 Processor -- start 2024-04-21T08:45:47.567 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c21ac5700 1 -- start start 2024-04-21T08:45:47.567 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c21ac5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c1c101be0 0x7f7c1c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:47.567 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c21ac5700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7c1c1025a0 0x7f7c1c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:47.567 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c21ac5700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c1c103010 0x7f7c1c19f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:47.567 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c21ac5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c10d4c0 con 0x7f7c1c101be0 2024-04-21T08:45:47.567 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c21ac5700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c10d340 con 0x7f7c1c103010 2024-04-21T08:45:47.568 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c21ac5700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c10d640 con 0x7f7c1c1025a0 2024-04-21T08:45:47.568 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c1bfff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c1c103010 0x7f7c1c19f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:47.568 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c1bfff700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c1c103010 0x7f7c1c19f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:60958/0 (socket says 172.21.15.152:60958) 2024-04-21T08:45:47.568 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c1bfff700 1 -- 172.21.15.152:0/825082449 learned_addr learned my addr 172.21.15.152:0/825082449 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-04-21T08:45:47.568 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c13fff700 1 --2- 172.21.15.152:0/825082449 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7c1c1025a0 0x7f7c1c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:47.568 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c1bfff700 1 -- 172.21.15.152:0/825082449 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7c1c1025a0 msgr2=0x7f7c1c198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:47.568 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c1bfff700 1 --2- 172.21.15.152:0/825082449 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7c1c1025a0 0x7f7c1c198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:47.568 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c1bfff700 1 -- 172.21.15.152:0/825082449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c1c101be0 msgr2=0x7f7c1c197b00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:45:47.568 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c1bfff700 1 --2- 172.21.15.152:0/825082449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c1c101be0 0x7f7c1c197b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:47.568 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.566+0000 7f7c1bfff700 1 -- 172.21.15.152:0/825082449 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1c19f760 con 0x7f7c1c103010 2024-04-21T08:45:47.568 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.567+0000 7f7c1bfff700 1 --2- 172.21.15.152:0/825082449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c1c103010 0x7f7c1c19f0c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7c1400b780 tx=0x7f7c1400ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:47.568 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.567+0000 7f7c197fa700 1 -- 172.21.15.152:0/825082449 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c14014070 con 0x7f7c1c103010 2024-04-21T08:45:47.568 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.567+0000 7f7c197fa700 1 -- 172.21.15.152:0/825082449 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c14010d10 con 0x7f7c1c103010 2024-04-21T08:45:47.568 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.567+0000 7f7c197fa700 1 -- 172.21.15.152:0/825082449 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1400fe30 con 0x7f7c1c103010 2024-04-21T08:45:47.569 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.567+0000 7f7c21ac5700 1 -- 172.21.15.152:0/825082449 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c1c19fa50 con 0x7f7c1c103010 2024-04-21T08:45:47.569 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.567+0000 7f7c21ac5700 1 -- 172.21.15.152:0/825082449 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c1c19ff90 con 0x7f7c1c103010 2024-04-21T08:45:47.569 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.568+0000 7f7c21ac5700 1 -- 172.21.15.152:0/825082449 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c1c069730 con 0x7f7c1c103010 2024-04-21T08:45:47.569 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.569+0000 7f7c197fa700 1 -- 172.21.15.152:0/825082449 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7c14017070 con 0x7f7c1c103010 2024-04-21T08:45:47.570 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.569+0000 7f7c197fa700 1 --2- 172.21.15.152:0/825082449 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7c080a3cc0 0x7f7c080a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:47.570 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.569+0000 7f7c197fa700 1 -- 172.21.15.152:0/825082449 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f7c140cf540 con 0x7f7c1c103010 2024-04-21T08:45:47.570 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.570+0000 7f7c1b7fe700 1 --2- 172.21.15.152:0/825082449 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7c080a3cc0 0x7f7c080a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:47.570 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.570+0000 7f7c1b7fe700 1 --2- 172.21.15.152:0/825082449 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7c080a3cc0 0x7f7c080a6180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009af0 tx=0x7f7c0c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:47.573 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.573+0000 7f7c197fa700 1 -- 172.21.15.152:0/825082449 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c1409cc30 con 0x7f7c1c103010 2024-04-21T08:45:47.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:47 smithi028 ceph-mon[28003]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:47.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:47 smithi028 ceph-mon[28003]: Detected new or changed devices on smithi028 2024-04-21T08:45:47.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:47 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi028 to 5697M 2024-04-21T08:45:47.779 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:45:47.780+0000 7f7c21ac5700 1 -- 172.21.15.152:0/825082449 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7c1c062e80 con 0x7f7c080a3cc0 2024-04-21T08:45:48.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:48 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:45:48.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:48 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:45:48.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:48 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:45:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:45:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:48.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:48 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:45:48.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:48 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:45:48.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:48 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:49.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:49 smithi152 ceph-mon[31401]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:49.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:49 smithi152 ceph-mon[31401]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:49.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:49 smithi028 ceph-mon[28003]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:49.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:49 smithi028 ceph-mon[28003]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:49.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:49 smithi171 ceph-mon[31358]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:49.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:49 smithi171 ceph-mon[31358]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:51 smithi152 ceph-mon[31401]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:51 smithi152 ceph-mon[31401]: from='client.? 172.21.15.152:0/2726712521' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4175e074-9115-40ef-8582-26478d134260"}]: dispatch 2024-04-21T08:45:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:51 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4175e074-9115-40ef-8582-26478d134260"}]: dispatch 2024-04-21T08:45:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:51 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4175e074-9115-40ef-8582-26478d134260"}]': finished 2024-04-21T08:45:51.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:51 smithi152 ceph-mon[31401]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T08:45:51.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:45:51.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:51 smithi028 ceph-mon[28003]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:51.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:51 smithi028 ceph-mon[28003]: from='client.? 172.21.15.152:0/2726712521' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4175e074-9115-40ef-8582-26478d134260"}]: dispatch 2024-04-21T08:45:51.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:51 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4175e074-9115-40ef-8582-26478d134260"}]: dispatch 2024-04-21T08:45:51.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:51 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4175e074-9115-40ef-8582-26478d134260"}]': finished 2024-04-21T08:45:51.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:51 smithi028 ceph-mon[28003]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T08:45:51.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:45:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:51 smithi171 ceph-mon[31358]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:51 smithi171 ceph-mon[31358]: from='client.? 172.21.15.152:0/2726712521' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4175e074-9115-40ef-8582-26478d134260"}]: dispatch 2024-04-21T08:45:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:51 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4175e074-9115-40ef-8582-26478d134260"}]: dispatch 2024-04-21T08:45:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:51 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4175e074-9115-40ef-8582-26478d134260"}]': finished 2024-04-21T08:45:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:51 smithi171 ceph-mon[31358]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T08:45:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:45:52.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:52 smithi152 ceph-mon[31401]: from='client.? 172.21.15.152:0/2885230352' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:45:52.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:52 smithi028 ceph-mon[28003]: from='client.? 172.21.15.152:0/2885230352' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:45:52.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:52 smithi171 ceph-mon[31358]: from='client.? 172.21.15.152:0/2885230352' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:45:53.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:53 smithi152 ceph-mon[31401]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:53.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:53 smithi028 ceph-mon[28003]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:53.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:53 smithi171 ceph-mon[31358]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:55.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:55 smithi152 ceph-mon[31401]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:55.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:55 smithi028 ceph-mon[28003]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:55.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:55 smithi171 ceph-mon[31358]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:57.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:57 smithi152 ceph-mon[31401]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:57.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:57 smithi028 ceph-mon[28003]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:57.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:57 smithi171 ceph-mon[31358]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:58.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:58 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T08:45:58.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:58 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:58.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:58 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T08:45:58.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:58 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:58.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:58 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T08:45:58.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:58 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:59.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:59 smithi152 ceph-mon[31401]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:59.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:45:59 smithi152 ceph-mon[31401]: Deploying daemon osd.7 on smithi152 2024-04-21T08:45:59.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:59 smithi028 ceph-mon[28003]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:59.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:45:59 smithi028 ceph-mon[28003]: Deploying daemon osd.7 on smithi152 2024-04-21T08:45:59.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:59 smithi171 ceph-mon[31358]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:45:59.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:45:59 smithi171 ceph-mon[31358]: Deploying daemon osd.7 on smithi152 2024-04-21T08:46:01.466 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:01 smithi152 ceph-mon[31401]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:01.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:01 smithi028 ceph-mon[28003]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:01.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:01 smithi171 ceph-mon[31358]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:03.579 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:03 smithi152 ceph-mon[31401]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:03.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:03 smithi028 ceph-mon[28003]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:03.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:03 smithi171 ceph-mon[31358]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:05.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:04 smithi152 ceph-mon[31401]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:05.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:04 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:05.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:04 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:05.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:04 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:05.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:04 smithi171 ceph-mon[31358]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:05.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:04 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:05.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:04 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:05.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:04 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:05.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:04 smithi028 ceph-mon[28003]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:05.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:04 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:05.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:04 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:05.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:04 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:07.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:06 smithi152 ceph-mon[31401]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:07.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:06 smithi171 ceph-mon[31358]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:07.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:06 smithi028 ceph-mon[28003]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:07.949 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:46:07.949+0000 7f7c197fa700 1 -- 172.21.15.152:0/825082449 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7c1c062e80 con 0x7f7c080a3cc0 2024-04-21T08:46:07.949 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 7 on host 'smithi152' 2024-04-21T08:46:07.951 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:46:07.952+0000 7f7c21ac5700 1 -- 172.21.15.152:0/825082449 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7c080a3cc0 msgr2=0x7f7c080a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:07.952 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:46:07.952+0000 7f7c21ac5700 1 --2- 172.21.15.152:0/825082449 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7c080a3cc0 0x7f7c080a6180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009af0 tx=0x7f7c0c009260 comp rx=0 tx=0).stop 2024-04-21T08:46:07.952 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:46:07.952+0000 7f7c21ac5700 1 -- 172.21.15.152:0/825082449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c1c103010 msgr2=0x7f7c1c19f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:07.952 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:46:07.952+0000 7f7c21ac5700 1 --2- 172.21.15.152:0/825082449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c1c103010 0x7f7c1c19f0c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7c1400b780 tx=0x7f7c1400ba90 comp rx=0 tx=0).stop 2024-04-21T08:46:07.952 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:46:07.952+0000 7f7c21ac5700 1 -- 172.21.15.152:0/825082449 shutdown_connections 2024-04-21T08:46:07.952 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:46:07.952+0000 7f7c21ac5700 1 --2- 172.21.15.152:0/825082449 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7c080a3cc0 0x7f7c080a6180 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:07.952 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:46:07.952+0000 7f7c21ac5700 1 --2- 172.21.15.152:0/825082449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c1c101be0 0x7f7c1c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:07.952 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:46:07.952+0000 7f7c21ac5700 1 --2- 172.21.15.152:0/825082449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7c1c103010 0x7f7c1c19f0c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:07.952 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:46:07.952+0000 7f7c21ac5700 1 --2- 172.21.15.152:0/825082449 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7c1c1025a0 0x7f7c1c198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:07.952 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:46:07.952+0000 7f7c21ac5700 1 -- 172.21.15.152:0/825082449 >> 172.21.15.152:0/825082449 conn(0x7f7c1c0fc6d0 msgr2=0x7f7c1c105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:07.952 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:46:07.952+0000 7f7c21ac5700 1 -- 172.21.15.152:0/825082449 shutdown_connections 2024-04-21T08:46:07.952 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:46:07.952+0000 7f7c21ac5700 1 -- 172.21.15.152:0/825082449 wait complete. 2024-04-21T08:46:08.333 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:08.333 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:08.333 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:08.333 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:08.333 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:08.334 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:07 smithi152 ceph-mon[31401]: from='osd.7 [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:46:08.334 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:07 smithi152 ceph-mon[31401]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:46:08.334 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:08.334 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:08.334 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:08.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:08.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:08.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:08.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:08.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:08.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:07 smithi028 ceph-mon[28003]: from='osd.7 [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:46:08.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:07 smithi028 ceph-mon[28003]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:46:08.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:08.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:08.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:08.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:08.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:08.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:08.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:08.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:08.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:07 smithi171 ceph-mon[31358]: from='osd.7 [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:46:08.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:07 smithi171 ceph-mon[31358]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:46:08.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:08.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:08.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:08.668 DEBUG:teuthology.orchestra.run.smithi152:osd.7> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.7.service 2024-04-21T08:46:08.670 INFO:tasks.cephadm:Deploying osd.8 on smithi171 with /dev/vg_nvme/lv_2... 2024-04-21T08:46:08.671 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T08:46:08.919 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:46:09.162 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:08 smithi152 ceph-mon[31401]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:09.162 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:08 smithi152 ceph-mon[31401]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T08:46:09.162 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:08 smithi152 ceph-mon[31401]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T08:46:09.162 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:08 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:46:09.162 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:08 smithi152 ceph-mon[31401]: from='osd.7 [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:46:09.162 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:08 smithi152 ceph-mon[31401]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:46:09.295 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:08 smithi171 ceph-mon[31358]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:09.295 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:08 smithi171 ceph-mon[31358]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T08:46:09.295 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:08 smithi171 ceph-mon[31358]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T08:46:09.295 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:08 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:46:09.295 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:08 smithi171 ceph-mon[31358]: from='osd.7 [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:46:09.295 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:08 smithi171 ceph-mon[31358]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:46:09.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:08 smithi028 ceph-mon[28003]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:09.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:08 smithi028 ceph-mon[28003]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T08:46:09.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:08 smithi028 ceph-mon[28003]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T08:46:09.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:08 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:46:09.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:08 smithi028 ceph-mon[28003]: from='osd.7 [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:46:09.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:08 smithi028 ceph-mon[28003]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:46:09.854 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 08:46:09 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-7[49471]: 2024-04-21T08:46:09.424+0000 7f4d3a8d1700 -1 osd.7 0 waiting for initial osdmap 2024-04-21T08:46:09.855 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 08:46:09 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-7[49471]: 2024-04-21T08:46:09.435+0000 7f4d326bc700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:46:10.232 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:10 smithi152 ceph-mon[31401]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-04-21T08:46:10.232 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:10 smithi152 ceph-mon[31401]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T08:46:10.232 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:10 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:46:10.232 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:10 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:46:10.232 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:10 smithi152 ceph-mon[31401]: from='osd.7 ' entity='osd.7' 2024-04-21T08:46:10.286 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:10 smithi171 ceph-mon[31358]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-04-21T08:46:10.286 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:10 smithi171 ceph-mon[31358]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T08:46:10.286 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:10 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:46:10.286 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:10 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:46:10.286 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:10 smithi171 ceph-mon[31358]: from='osd.7 ' entity='osd.7' 2024-04-21T08:46:10.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:10 smithi028 ceph-mon[28003]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-04-21T08:46:10.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:10 smithi028 ceph-mon[28003]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T08:46:10.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:10 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:46:10.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:10 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:46:10.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:10 smithi028 ceph-mon[28003]: from='osd.7 ' entity='osd.7' 2024-04-21T08:46:10.979 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T08:46:10.979 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T08:46:10.979 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-04-21T08:46:10.979 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-04-21T08:46:10.979 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011609 s, 903 MB/s 2024-04-21T08:46:10.979 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-04-21T08:46:11.284 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:11 smithi171 ceph-mon[31358]: purged_snaps scrub starts 2024-04-21T08:46:11.284 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:11 smithi171 ceph-mon[31358]: purged_snaps scrub ok 2024-04-21T08:46:11.284 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:11 smithi171 ceph-mon[31358]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:11.284 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:11 smithi171 ceph-mon[31358]: osd.7 [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516] boot 2024-04-21T08:46:11.284 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:11 smithi171 ceph-mon[31358]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T08:46:11.284 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:11 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:46:11.284 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:11 smithi171 ceph-mon[31358]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T08:46:11.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:11 smithi152 ceph-mon[31401]: purged_snaps scrub starts 2024-04-21T08:46:11.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:11 smithi152 ceph-mon[31401]: purged_snaps scrub ok 2024-04-21T08:46:11.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:11 smithi152 ceph-mon[31401]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:11.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:11 smithi152 ceph-mon[31401]: osd.7 [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516] boot 2024-04-21T08:46:11.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:11 smithi152 ceph-mon[31401]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T08:46:11.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:11 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:46:11.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:11 smithi152 ceph-mon[31401]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T08:46:11.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:11 smithi028 ceph-mon[28003]: purged_snaps scrub starts 2024-04-21T08:46:11.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:11 smithi028 ceph-mon[28003]: purged_snaps scrub ok 2024-04-21T08:46:11.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:11 smithi028 ceph-mon[28003]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:46:11.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:11 smithi028 ceph-mon[28003]: osd.7 [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516] boot 2024-04-21T08:46:11.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:11 smithi028 ceph-mon[28003]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T08:46:11.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:11 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:46:11.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:11 smithi028 ceph-mon[28003]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T08:46:11.700 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi171:vg_nvme/lv_2 2024-04-21T08:46:11.951 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:46:13.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:12 smithi152 ceph-mon[31401]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:13.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:12 smithi152 ceph-mon[31401]: Detected new or changed devices on smithi152 2024-04-21T08:46:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:12 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi152 to 5696M 2024-04-21T08:46:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:12 smithi152 ceph-mon[31401]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T08:46:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:12 smithi028 ceph-mon[28003]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:12 smithi028 ceph-mon[28003]: Detected new or changed devices on smithi152 2024-04-21T08:46:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:12 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi152 to 5696M 2024-04-21T08:46:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:12 smithi028 ceph-mon[28003]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T08:46:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:12 smithi171 ceph-mon[31358]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:12 smithi171 ceph-mon[31358]: Detected new or changed devices on smithi152 2024-04-21T08:46:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:12 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi152 to 5696M 2024-04-21T08:46:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:12 smithi171 ceph-mon[31358]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T08:46:13.640 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.638+0000 7f886e2c8700 1 -- 172.21.15.171:0/536583061 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8868103c90 msgr2=0x7f88681040f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:13.640 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.638+0000 7f886e2c8700 1 --2- 172.21.15.171:0/536583061 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8868103c90 0x7f88681040f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f885c009990 tx=0x7f885c009ca0 comp rx=0 tx=0).stop 2024-04-21T08:46:13.640 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.638+0000 7f886e2c8700 1 -- 172.21.15.171:0/536583061 shutdown_connections 2024-04-21T08:46:13.640 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.638+0000 7f886e2c8700 1 --2- 172.21.15.171:0/536583061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868104630 0x7f886810aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:13.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.638+0000 7f886e2c8700 1 --2- 172.21.15.171:0/536583061 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8868102a90 0x7f8868102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:13.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.638+0000 7f886e2c8700 1 --2- 172.21.15.171:0/536583061 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8868103c90 0x7f88681040f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:13.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.638+0000 7f886e2c8700 1 -- 172.21.15.171:0/536583061 >> 172.21.15.171:0/536583061 conn(0x7f88680fe030 msgr2=0x7f8868100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:13.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.638+0000 7f886e2c8700 1 -- 172.21.15.171:0/536583061 shutdown_connections 2024-04-21T08:46:13.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.638+0000 7f886e2c8700 1 -- 172.21.15.171:0/536583061 wait complete. 2024-04-21T08:46:13.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.639+0000 7f886e2c8700 1 Processor -- start 2024-04-21T08:46:13.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.639+0000 7f886e2c8700 1 -- start start 2024-04-21T08:46:13.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.639+0000 7f886e2c8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868102a90 0x7f88680767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:13.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.639+0000 7f886e2c8700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8868103c90 0x7f8868074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:13.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.639+0000 7f886e2c8700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8868104630 0x7f8868075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:13.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.639+0000 7f886e2c8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f886810e520 con 0x7f8868102a90 2024-04-21T08:46:13.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.639+0000 7f886e2c8700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f886810e3a0 con 0x7f8868103c90 2024-04-21T08:46:13.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.639+0000 7f886e2c8700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f886810e6a0 con 0x7f8868104630 2024-04-21T08:46:13.641 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f886c865700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8868104630 0x7f8868075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:13.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f886c865700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8868104630 0x7f8868075370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:46112/0 (socket says 172.21.15.171:46112) 2024-04-21T08:46:13.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f886c865700 1 -- 172.21.15.171:0/911577191 learned_addr learned my addr 172.21.15.171:0/911577191 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-04-21T08:46:13.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f8867fff700 1 --2- 172.21.15.171:0/911577191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868102a90 0x7f88680767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:13.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f88677fe700 1 --2- 172.21.15.171:0/911577191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8868103c90 0x7f8868074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:13.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f886c865700 1 -- 172.21.15.171:0/911577191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8868103c90 msgr2=0x7f8868074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:13.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f886c865700 1 --2- 172.21.15.171:0/911577191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8868103c90 0x7f8868074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:13.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f886c865700 1 -- 172.21.15.171:0/911577191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868102a90 msgr2=0x7f88680767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:13.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f886c865700 1 --2- 172.21.15.171:0/911577191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868102a90 0x7f88680767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:13.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f886c865700 1 -- 172.21.15.171:0/911577191 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8868075c20 con 0x7f8868104630 2024-04-21T08:46:13.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f886c865700 1 --2- 172.21.15.171:0/911577191 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8868104630 0x7f8868075370 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8860007ee0 tx=0x7f886000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:13.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f88657fa700 1 -- 172.21.15.171:0/911577191 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8860004bb0 con 0x7f8868104630 2024-04-21T08:46:13.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f88657fa700 1 -- 172.21.15.171:0/911577191 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8860004d50 con 0x7f8868104630 2024-04-21T08:46:13.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f886e2c8700 1 -- 172.21.15.171:0/911577191 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88681a2d60 con 0x7f8868104630 2024-04-21T08:46:13.642 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f88657fa700 1 -- 172.21.15.171:0/911577191 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8860005920 con 0x7f8868104630 2024-04-21T08:46:13.643 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.640+0000 7f886e2c8700 1 -- 172.21.15.171:0/911577191 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88681a31c0 con 0x7f8868104630 2024-04-21T08:46:13.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.641+0000 7f886e2c8700 1 -- 172.21.15.171:0/911577191 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8868069730 con 0x7f8868104630 2024-04-21T08:46:13.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.642+0000 7f88657fa700 1 -- 172.21.15.171:0/911577191 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f886001a020 con 0x7f8868104630 2024-04-21T08:46:13.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.643+0000 7f88657fa700 1 --2- 172.21.15.171:0/911577191 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f88500a3c40 0x7f88500a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:13.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.643+0000 7f88657fa700 1 -- 172.21.15.171:0/911577191 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5397+0+0 (secure 0 0 0) 0x7f88600cf5b0 con 0x7f8868104630 2024-04-21T08:46:13.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.643+0000 7f8867fff700 1 --2- 172.21.15.171:0/911577191 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f88500a3c40 0x7f88500a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:13.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.643+0000 7f8867fff700 1 --2- 172.21.15.171:0/911577191 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f88500a3c40 0x7f88500a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8858005950 tx=0x7f885800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:13.647 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.646+0000 7f88657fa700 1 -- 172.21.15.171:0/911577191 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8860010070 con 0x7f8868104630 2024-04-21T08:46:13.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:13.853+0000 7f886e2c8700 1 -- 172.21.15.171:0/911577191 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f8868062e80 con 0x7f88500a3c40 2024-04-21T08:46:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:14 smithi171 ceph-mon[31358]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T08:46:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:14 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:46:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:14 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:46:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:14 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:14.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:14 smithi152 ceph-mon[31401]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T08:46:14.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:14 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:46:14.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:14 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:46:14.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:14 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:14.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:14 smithi028 ceph-mon[28003]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T08:46:14.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:46:14.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:46:14.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:14 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:15.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:15 smithi152 ceph-mon[31401]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:15.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:15 smithi152 ceph-mon[31401]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:46:15.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:15 smithi152 ceph-mon[31401]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T08:46:15.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:15 smithi028 ceph-mon[28003]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:15.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:15 smithi028 ceph-mon[28003]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:46:15.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:15 smithi028 ceph-mon[28003]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T08:46:15.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:15 smithi171 ceph-mon[31358]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:15.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:15 smithi171 ceph-mon[31358]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:46:15.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:15 smithi171 ceph-mon[31358]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T08:46:17.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:17 smithi152 ceph-mon[31401]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:17.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:17 smithi152 ceph-mon[31401]: from='client.? 172.21.15.171:0/3370771700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32c9b173-a70e-4ebf-8a9f-81a0ecf6f727"}]: dispatch 2024-04-21T08:46:17.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:17 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32c9b173-a70e-4ebf-8a9f-81a0ecf6f727"}]: dispatch 2024-04-21T08:46:17.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:17 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32c9b173-a70e-4ebf-8a9f-81a0ecf6f727"}]': finished 2024-04-21T08:46:17.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:17 smithi152 ceph-mon[31401]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T08:46:17.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:17 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:17.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:17 smithi028 ceph-mon[28003]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:17.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:17 smithi028 ceph-mon[28003]: from='client.? 172.21.15.171:0/3370771700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32c9b173-a70e-4ebf-8a9f-81a0ecf6f727"}]: dispatch 2024-04-21T08:46:17.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:17 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32c9b173-a70e-4ebf-8a9f-81a0ecf6f727"}]: dispatch 2024-04-21T08:46:17.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:17 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32c9b173-a70e-4ebf-8a9f-81a0ecf6f727"}]': finished 2024-04-21T08:46:17.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:17 smithi028 ceph-mon[28003]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T08:46:17.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:17 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:17.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:17 smithi171 ceph-mon[31358]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:17.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:17 smithi171 ceph-mon[31358]: from='client.? 172.21.15.171:0/3370771700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32c9b173-a70e-4ebf-8a9f-81a0ecf6f727"}]: dispatch 2024-04-21T08:46:17.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:17 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32c9b173-a70e-4ebf-8a9f-81a0ecf6f727"}]: dispatch 2024-04-21T08:46:17.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:17 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32c9b173-a70e-4ebf-8a9f-81a0ecf6f727"}]': finished 2024-04-21T08:46:17.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:17 smithi171 ceph-mon[31358]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T08:46:17.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:17 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:18.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:18 smithi152 ceph-mon[31401]: from='client.? 172.21.15.171:0/496447107' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:46:18.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:18 smithi028 ceph-mon[28003]: from='client.? 172.21.15.171:0/496447107' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:46:18.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:18 smithi171 ceph-mon[31358]: from='client.? 172.21.15.171:0/496447107' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:46:19.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:19 smithi152 ceph-mon[31401]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:19.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:19 smithi028 ceph-mon[28003]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:19.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:19 smithi171 ceph-mon[31358]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:21.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:21 smithi152 ceph-mon[31401]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:21.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:21 smithi028 ceph-mon[28003]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:21.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:21 smithi171 ceph-mon[31358]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:23.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:23 smithi152 ceph-mon[31401]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:23.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:23 smithi028 ceph-mon[28003]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:23.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:23 smithi171 ceph-mon[31358]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:25.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:25 smithi152 ceph-mon[31401]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:25.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:25 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T08:46:25.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:25 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:25.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:25 smithi152 ceph-mon[31401]: Deploying daemon osd.8 on smithi171 2024-04-21T08:46:25.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:25 smithi028 ceph-mon[28003]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:25.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:25 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T08:46:25.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:25 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:25.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:25 smithi028 ceph-mon[28003]: Deploying daemon osd.8 on smithi171 2024-04-21T08:46:25.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:25 smithi171 ceph-mon[31358]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:25.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:25 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T08:46:25.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:25 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:25.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:25 smithi171 ceph-mon[31358]: Deploying daemon osd.8 on smithi171 2024-04-21T08:46:27.261 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:27 smithi171 ceph-mon[31358]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:27.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:27 smithi152 ceph-mon[31401]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:27.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:27 smithi028 ceph-mon[28003]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:29.229 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:29 smithi171 ceph-mon[31358]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:29.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:29 smithi152 ceph-mon[31401]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:29.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:29 smithi028 ceph-mon[28003]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:31.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:30 smithi171 ceph-mon[31358]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:31.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:31.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:31.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:31.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:30 smithi152 ceph-mon[31401]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:31.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:31.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:31.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:31.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:30 smithi028 ceph-mon[28003]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:31.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:31.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:31.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:33.149 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:32 smithi171 ceph-mon[31358]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:33.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:32 smithi152 ceph-mon[31401]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:33.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:32 smithi028 ceph-mon[28003]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:33.925 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:33.925+0000 7f88657fa700 1 -- 172.21.15.171:0/911577191 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8868062e80 con 0x7f88500a3c40 2024-04-21T08:46:33.925 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 8 on host 'smithi171' 2024-04-21T08:46:33.927 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:33.927+0000 7f886e2c8700 1 -- 172.21.15.171:0/911577191 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f88500a3c40 msgr2=0x7f88500a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.927 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:33.927+0000 7f886e2c8700 1 --2- 172.21.15.171:0/911577191 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f88500a3c40 0x7f88500a6100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8858005950 tx=0x7f885800a300 comp rx=0 tx=0).stop 2024-04-21T08:46:33.927 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:33.927+0000 7f886e2c8700 1 -- 172.21.15.171:0/911577191 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8868104630 msgr2=0x7f8868075370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.927 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:33.927+0000 7f886e2c8700 1 --2- 172.21.15.171:0/911577191 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8868104630 0x7f8868075370 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8860007ee0 tx=0x7f886000d3b0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.927 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:33.927+0000 7f886e2c8700 1 -- 172.21.15.171:0/911577191 shutdown_connections 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:33.927+0000 7f886e2c8700 1 --2- 172.21.15.171:0/911577191 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f88500a3c40 0x7f88500a6100 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:33.927+0000 7f886e2c8700 1 --2- 172.21.15.171:0/911577191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8868102a90 0x7f88680767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:33.927+0000 7f886e2c8700 1 --2- 172.21.15.171:0/911577191 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8868104630 0x7f8868075370 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:33.927+0000 7f886e2c8700 1 --2- 172.21.15.171:0/911577191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8868103c90 0x7f8868074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:33.927+0000 7f886e2c8700 1 -- 172.21.15.171:0/911577191 >> 172.21.15.171:0/911577191 conn(0x7f88680fe030 msgr2=0x7f88681070b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:33.927+0000 7f886e2c8700 1 -- 172.21.15.171:0/911577191 shutdown_connections 2024-04-21T08:46:33.928 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:46:33.927+0000 7f886e2c8700 1 -- 172.21.15.171:0/911577191 wait complete. 2024-04-21T08:46:34.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:34.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:34.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:34.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:34.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:34.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:33 smithi171 ceph-mon[31358]: from='osd.8 [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:46:34.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:33 smithi171 ceph-mon[31358]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:46:34.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:34.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:34.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:34.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:34.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:34.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:34.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:34.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:34.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:33 smithi152 ceph-mon[31401]: from='osd.8 [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:46:34.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:33 smithi152 ceph-mon[31401]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:46:34.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:34.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:34.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:34.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:34.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:34.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:34.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:34.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:34.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:33 smithi028 ceph-mon[28003]: from='osd.8 [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:46:34.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:33 smithi028 ceph-mon[28003]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:46:34.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:34.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:34.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:34.624 DEBUG:teuthology.orchestra.run.smithi171:osd.8> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.8.service 2024-04-21T08:46:34.627 INFO:tasks.cephadm:Deploying osd.9 on smithi028 with /dev/vg_nvme/lv_1... 2024-04-21T08:46:34.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T08:46:34.886 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:46:35.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:34 smithi171 ceph-mon[31358]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:35.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:34 smithi171 ceph-mon[31358]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T08:46:35.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:34 smithi171 ceph-mon[31358]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T08:46:35.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:34 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:35.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:34 smithi171 ceph-mon[31358]: from='osd.8 [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:46:35.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:34 smithi171 ceph-mon[31358]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:46:35.121 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 08:46:34 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-8[50056]: 2024-04-21T08:46:34.960+0000 7f5954727700 -1 osd.8 0 waiting for initial osdmap 2024-04-21T08:46:35.121 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 08:46:34 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-8[50056]: 2024-04-21T08:46:34.969+0000 7f595051a700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:46:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:34 smithi028 ceph-mon[28003]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:34 smithi028 ceph-mon[28003]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T08:46:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:34 smithi028 ceph-mon[28003]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T08:46:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:34 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:34 smithi028 ceph-mon[28003]: from='osd.8 [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:46:35.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:34 smithi028 ceph-mon[28003]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:46:35.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:34 smithi152 ceph-mon[31401]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:35.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:34 smithi152 ceph-mon[31401]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T08:46:35.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:34 smithi152 ceph-mon[31401]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T08:46:35.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:34 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:35.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:34 smithi152 ceph-mon[31401]: from='osd.8 [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:46:35.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:34 smithi152 ceph-mon[31401]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:46:36.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:35 smithi171 ceph-mon[31358]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-21T08:46:36.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:35 smithi171 ceph-mon[31358]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T08:46:36.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:36.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:36.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:35 smithi028 ceph-mon[28003]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-21T08:46:36.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:35 smithi028 ceph-mon[28003]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T08:46:36.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:36.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:36.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:35 smithi152 ceph-mon[31401]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-21T08:46:36.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:35 smithi152 ceph-mon[31401]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T08:46:36.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:35 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:36.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:35 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:37.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:36 smithi028 ceph-mon[28003]: purged_snaps scrub starts 2024-04-21T08:46:37.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:36 smithi028 ceph-mon[28003]: purged_snaps scrub ok 2024-04-21T08:46:37.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:36 smithi028 ceph-mon[28003]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:37.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:37.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:36 smithi028 ceph-mon[28003]: osd.8 [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587] boot 2024-04-21T08:46:37.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:36 smithi028 ceph-mon[28003]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T08:46:37.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:37.153 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T08:46:37.153 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T08:46:37.153 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-21T08:46:37.154 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-21T08:46:37.154 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00848911 s, 1.2 GB/s 2024-04-21T08:46:37.154 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-21T08:46:37.287 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:36 smithi171 ceph-mon[31358]: purged_snaps scrub starts 2024-04-21T08:46:37.287 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:36 smithi171 ceph-mon[31358]: purged_snaps scrub ok 2024-04-21T08:46:37.287 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:36 smithi171 ceph-mon[31358]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:37.287 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:37.287 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:36 smithi171 ceph-mon[31358]: osd.8 [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587] boot 2024-04-21T08:46:37.287 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:36 smithi171 ceph-mon[31358]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T08:46:37.288 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:37.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:36 smithi152 ceph-mon[31401]: purged_snaps scrub starts 2024-04-21T08:46:37.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:36 smithi152 ceph-mon[31401]: purged_snaps scrub ok 2024-04-21T08:46:37.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:36 smithi152 ceph-mon[31401]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:46:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:36 smithi152 ceph-mon[31401]: osd.8 [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587] boot 2024-04-21T08:46:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:36 smithi152 ceph-mon[31401]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T08:46:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:46:37.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-04-21T08:46:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:46:38.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:37 smithi028 ceph-mon[28003]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T08:46:38.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:38.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:38.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:38.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:38.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:38.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:38.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:38.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:38.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:38.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:37 smithi152 ceph-mon[31401]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T08:46:38.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:38.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:38.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:38.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:38.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:37 smithi171 ceph-mon[31358]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T08:46:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:46:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:38.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:39.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:38 smithi152 ceph-mon[31401]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:39.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:38 smithi152 ceph-mon[31401]: Detected new or changed devices on smithi171 2024-04-21T08:46:39.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:38 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi171 to 5696M 2024-04-21T08:46:39.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:38 smithi152 ceph-mon[31401]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T08:46:39.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:38 smithi028 ceph-mon[28003]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:39.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:38 smithi028 ceph-mon[28003]: Detected new or changed devices on smithi171 2024-04-21T08:46:39.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:38 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi171 to 5696M 2024-04-21T08:46:39.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:38 smithi028 ceph-mon[28003]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T08:46:39.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:38 smithi171 ceph-mon[31358]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:39.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:38 smithi171 ceph-mon[31358]: Detected new or changed devices on smithi171 2024-04-21T08:46:39.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:38 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi171 to 5696M 2024-04-21T08:46:39.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:38 smithi171 ceph-mon[31358]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T08:46:40.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:39 smithi028 ceph-mon[28003]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T08:46:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.122+0000 7f8040eab700 1 -- 172.21.15.28:0/3106524253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f803c103560 msgr2=0x7f803c109e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.122+0000 7f8040eab700 1 --2- 172.21.15.28:0/3106524253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f803c103560 0x7f803c109e10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f803400b0d0 tx=0x7f803400b3e0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.122+0000 7f8040eab700 1 -- 172.21.15.28:0/3106524253 shutdown_connections 2024-04-21T08:46:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.122+0000 7f8040eab700 1 --2- 172.21.15.28:0/3106524253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f803c103560 0x7f803c109e10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.122+0000 7f8040eab700 1 --2- 172.21.15.28:0/3106524253 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f803c1019c0 0x7f803c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.122+0000 7f8040eab700 1 --2- 172.21.15.28:0/3106524253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f803c102bc0 0x7f803c103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.122+0000 7f8040eab700 1 -- 172.21.15.28:0/3106524253 >> 172.21.15.28:0/3106524253 conn(0x7f803c0fcf60 msgr2=0x7f803c0ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.122+0000 7f8040eab700 1 -- 172.21.15.28:0/3106524253 shutdown_connections 2024-04-21T08:46:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.122+0000 7f8040eab700 1 -- 172.21.15.28:0/3106524253 wait complete. 2024-04-21T08:46:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.123+0000 7f8040eab700 1 Processor -- start 2024-04-21T08:46:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.123+0000 7f8040eab700 1 -- start start 2024-04-21T08:46:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.123+0000 7f8040eab700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f803c1019c0 0x7f803c198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.123+0000 7f8040eab700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f803c102bc0 0x7f803c1990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.123+0000 7f8040eab700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f803c103560 0x7f803c1a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.123+0000 7f8040eab700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f803c10d460 con 0x7f803c1019c0 2024-04-21T08:46:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.123+0000 7f8040eab700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f803c10d2e0 con 0x7f803c103560 2024-04-21T08:46:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.123+0000 7f8040eab700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f803c10d5e0 con 0x7f803c102bc0 2024-04-21T08:46:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f803a59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f803c1019c0 0x7f803c198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f803a59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f803c1019c0 0x7f803c198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33788/0 (socket says 172.21.15.28:33788) 2024-04-21T08:46:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f803a59c700 1 -- 172.21.15.28:0/107175195 learned_addr learned my addr 172.21.15.28:0/107175195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:46:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f8039d9b700 1 --2- 172.21.15.28:0/107175195 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f803c102bc0 0x7f803c1990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f803ad9d700 1 --2- 172.21.15.28:0/107175195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f803c103560 0x7f803c1a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f803a59c700 1 -- 172.21.15.28:0/107175195 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f803c102bc0 msgr2=0x7f803c1990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f803a59c700 1 --2- 172.21.15.28:0/107175195 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f803c102bc0 0x7f803c1990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f803a59c700 1 -- 172.21.15.28:0/107175195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f803c103560 msgr2=0x7f803c1a0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f803a59c700 1 --2- 172.21.15.28:0/107175195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f803c103560 0x7f803c1a0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f803a59c700 1 -- 172.21.15.28:0/107175195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f803c1a07b0 con 0x7f803c1019c0 2024-04-21T08:46:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f803a59c700 1 --2- 172.21.15.28:0/107175195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f803c1019c0 0x7f803c198ba0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f802c00e980 tx=0x7f802c00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f802b7fe700 1 -- 172.21.15.28:0/107175195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c00cbd0 con 0x7f803c1019c0 2024-04-21T08:46:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f8040eab700 1 -- 172.21.15.28:0/107175195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f803c1a0aa0 con 0x7f803c1019c0 2024-04-21T08:46:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f802b7fe700 1 -- 172.21.15.28:0/107175195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f802c0044e0 con 0x7f803c1019c0 2024-04-21T08:46:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f802b7fe700 1 -- 172.21.15.28:0/107175195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c010430 con 0x7f803c1019c0 2024-04-21T08:46:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.124+0000 7f8040eab700 1 -- 172.21.15.28:0/107175195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f803c1a0f90 con 0x7f803c1019c0 2024-04-21T08:46:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.125+0000 7f8040eab700 1 -- 172.21.15.28:0/107175195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f803c069730 con 0x7f803c1019c0 2024-04-21T08:46:40.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.126+0000 7f802b7fe700 1 -- 172.21.15.28:0/107175195 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f802c003f10 con 0x7f803c1019c0 2024-04-21T08:46:40.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.127+0000 7f802b7fe700 1 --2- 172.21.15.28:0/107175195 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80240a3c40 0x7f80240a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:40.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.127+0000 7f8039d9b700 1 --2- 172.21.15.28:0/107175195 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80240a3c40 0x7f80240a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:40.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.127+0000 7f802b7fe700 1 -- 172.21.15.28:0/107175195 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f802c0cfed0 con 0x7f803c1019c0 2024-04-21T08:46:40.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.128+0000 7f8039d9b700 1 --2- 172.21.15.28:0/107175195 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80240a3c40 0x7f80240a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8030005950 tx=0x7f80300058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:40.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.131+0000 7f802b7fe700 1 -- 172.21.15.28:0/107175195 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f802c09d260 con 0x7f803c1019c0 2024-04-21T08:46:40.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:46:40.351+0000 7f8040eab700 1 -- 172.21.15.28:0/107175195 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f803c1a1220 con 0x7f80240a3c40 2024-04-21T08:46:40.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:40 smithi152 ceph-mon[31401]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T08:46:40.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:39 smithi171 ceph-mon[31358]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T08:46:41.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:40 smithi028 ceph-mon[28003]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:41.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:40 smithi028 ceph-mon[28003]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T08:46:41.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:40 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:46:41.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:40 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:46:41.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:40 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:41.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:40 smithi152 ceph-mon[31401]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:41.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:40 smithi152 ceph-mon[31401]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T08:46:41.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:40 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:46:41.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:40 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:46:41.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:40 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:40 smithi171 ceph-mon[31358]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:40 smithi171 ceph-mon[31358]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T08:46:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:40 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:46:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:40 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:46:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:40 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:42.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:42 smithi152 ceph-mon[31401]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:46:42.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:42 smithi028 ceph-mon[28003]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:46:42.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:42 smithi171 ceph-mon[31358]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:46:43.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:43 smithi152 ceph-mon[31401]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-21T08:46:43.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:43 smithi028 ceph-mon[28003]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-21T08:46:43.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:43 smithi171 ceph-mon[31358]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-21T08:46:44.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:44 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2193913835' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "181899a9-f577-4b15-b710-4681b4169ea1"}]: dispatch 2024-04-21T08:46:44.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:44 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2193913835' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "181899a9-f577-4b15-b710-4681b4169ea1"}]': finished 2024-04-21T08:46:44.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:44 smithi028 ceph-mon[28003]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T08:46:44.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:44 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:46:44.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:44 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2193913835' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "181899a9-f577-4b15-b710-4681b4169ea1"}]: dispatch 2024-04-21T08:46:44.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:44 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2193913835' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "181899a9-f577-4b15-b710-4681b4169ea1"}]': finished 2024-04-21T08:46:44.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:44 smithi152 ceph-mon[31401]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T08:46:44.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:44 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:46:44.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:44 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2193913835' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "181899a9-f577-4b15-b710-4681b4169ea1"}]: dispatch 2024-04-21T08:46:44.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:44 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2193913835' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "181899a9-f577-4b15-b710-4681b4169ea1"}]': finished 2024-04-21T08:46:44.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:44 smithi171 ceph-mon[31358]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T08:46:44.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:44 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:46:45.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:45 smithi152 ceph-mon[31401]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T08:46:45.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:45 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2407177096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:46:45.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:45 smithi028 ceph-mon[28003]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T08:46:45.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:45 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2407177096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:46:45.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:45 smithi171 ceph-mon[31358]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T08:46:45.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:45 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2407177096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:46:47.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:47 smithi152 ceph-mon[31401]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-21T08:46:47.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:47 smithi028 ceph-mon[28003]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-21T08:46:47.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:47 smithi171 ceph-mon[31358]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-21T08:46:49.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:49 smithi028 ceph-mon[28003]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:46:49.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:49 smithi152 ceph-mon[31401]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:46:49.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:49 smithi171 ceph-mon[31358]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:46:51.233 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:51 smithi028 ceph-mon[28003]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-21T08:46:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:51 smithi152 ceph-mon[31401]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-21T08:46:51.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:51 smithi171 ceph-mon[31358]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-21T08:46:52.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T08:46:52.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:52 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:52.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T08:46:52.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:52 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:52.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T08:46:52.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:52 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:53.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:53 smithi152 ceph-mon[31401]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:53.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:53 smithi152 ceph-mon[31401]: Deploying daemon osd.9 on smithi028 2024-04-21T08:46:53.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:53 smithi028 ceph-mon[28003]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:53.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:53 smithi028 ceph-mon[28003]: Deploying daemon osd.9 on smithi028 2024-04-21T08:46:53.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:53 smithi171 ceph-mon[31358]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:53.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:53 smithi171 ceph-mon[31358]: Deploying daemon osd.9 on smithi028 2024-04-21T08:46:55.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:55 smithi028 ceph-mon[28003]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:55.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:55 smithi152 ceph-mon[31401]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:55.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:55 smithi171 ceph-mon[31358]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:57.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:57 smithi028 ceph-mon[28003]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:57.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:57 smithi152 ceph-mon[31401]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:57.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:57 smithi171 ceph-mon[31358]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:59.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:58 smithi028 ceph-mon[28003]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:59.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:58 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:59.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:58 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:59.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:46:58 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:59.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:58 smithi152 ceph-mon[31401]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:59.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:58 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:59.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:58 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:59.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:46:58 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:59.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:58 smithi171 ceph-mon[31358]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:46:59.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:58 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:59.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:58 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:46:59.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:46:58 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:01.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:00 smithi028 ceph-mon[28003]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:47:01.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:00 smithi152 ceph-mon[31401]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:47:01.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:00 smithi171 ceph-mon[31358]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:47:01.542 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 9 on host 'smithi028' 2024-04-21T08:47:01.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:01.541+0000 7f802b7fe700 1 -- 172.21.15.28:0/107175195 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f803c1a1220 con 0x7f80240a3c40 2024-04-21T08:47:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:01.543+0000 7f8040eab700 1 -- 172.21.15.28:0/107175195 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80240a3c40 msgr2=0x7f80240a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:01.543+0000 7f8040eab700 1 --2- 172.21.15.28:0/107175195 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80240a3c40 0x7f80240a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8030005950 tx=0x7f80300058e0 comp rx=0 tx=0).stop 2024-04-21T08:47:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:01.543+0000 7f8040eab700 1 -- 172.21.15.28:0/107175195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f803c1019c0 msgr2=0x7f803c198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:01.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:01.543+0000 7f8040eab700 1 --2- 172.21.15.28:0/107175195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f803c1019c0 0x7f803c198ba0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f802c00e980 tx=0x7f802c00ed40 comp rx=0 tx=0).stop 2024-04-21T08:47:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:01.543+0000 7f8040eab700 1 -- 172.21.15.28:0/107175195 shutdown_connections 2024-04-21T08:47:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:01.543+0000 7f8040eab700 1 --2- 172.21.15.28:0/107175195 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80240a3c40 0x7f80240a6100 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:01.543+0000 7f8040eab700 1 --2- 172.21.15.28:0/107175195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f803c1019c0 0x7f803c198ba0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:01.543+0000 7f8040eab700 1 --2- 172.21.15.28:0/107175195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f803c103560 0x7f803c1a0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:01.543+0000 7f8040eab700 1 --2- 172.21.15.28:0/107175195 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f803c102bc0 0x7f803c1990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:01.543+0000 7f8040eab700 1 -- 172.21.15.28:0/107175195 >> 172.21.15.28:0/107175195 conn(0x7f803c0fcf60 msgr2=0x7f803c0fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:01.544+0000 7f8040eab700 1 -- 172.21.15.28:0/107175195 shutdown_connections 2024-04-21T08:47:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:01.544+0000 7f8040eab700 1 -- 172.21.15.28:0/107175195 wait complete. 2024-04-21T08:47:02.366 DEBUG:teuthology.orchestra.run.smithi028:osd.9> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.9.service 2024-04-21T08:47:02.369 INFO:tasks.cephadm:Deploying osd.10 on smithi152 with /dev/vg_nvme/lv_1... 2024-04-21T08:47:02.369 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T08:47:02.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:02 smithi028 ceph-mon[28003]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:47:02.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:02 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:02.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:02 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:02.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:02 smithi028 ceph-mon[28003]: from='osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T08:47:02.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:02 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:02.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:02 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:02.618 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:47:02.740 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:02 smithi152 ceph-mon[31401]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:47:02.740 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:02 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:02.740 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:02 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:02.740 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:02 smithi152 ceph-mon[31401]: from='osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T08:47:02.740 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:02 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:02.740 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:02 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:02.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:02 smithi171 ceph-mon[31358]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:47:02.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:02 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:02.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:02 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:02.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:02 smithi171 ceph-mon[31358]: from='osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T08:47:02.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:02 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:02.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:02 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:03.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:03 smithi028 ceph-mon[28003]: from='osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T08:47:03.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:03 smithi028 ceph-mon[28003]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T08:47:03.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:03 smithi028 ceph-mon[28003]: from='osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-21T08:47:03.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:03 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:47:03.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:03 smithi152 ceph-mon[31401]: from='osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T08:47:03.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:03 smithi152 ceph-mon[31401]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T08:47:03.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:03 smithi152 ceph-mon[31401]: from='osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-21T08:47:03.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:03 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:47:03.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:03 smithi171 ceph-mon[31358]: from='osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T08:47:03.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:03 smithi171 ceph-mon[31358]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T08:47:03.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:03 smithi171 ceph-mon[31358]: from='osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-21T08:47:03.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:03 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:47:04.060 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 08:47:04 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T08:47:04.017+0000 7feb81f0b700 -1 osd.9 0 waiting for initial osdmap 2024-04-21T08:47:04.061 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 08:47:04 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T08:47:04.028+0000 7feb79cf6700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:47:04.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:04 smithi028 ceph-mon[28003]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:47:04.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:04 smithi028 ceph-mon[28003]: from='osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-21T08:47:04.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:04 smithi028 ceph-mon[28003]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T08:47:04.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:04 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:47:04.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:04 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:47:04.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:04 smithi028 ceph-mon[28003]: from='osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197]' entity='osd.9' 2024-04-21T08:47:04.799 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:04 smithi152 ceph-mon[31401]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:47:04.799 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:04 smithi152 ceph-mon[31401]: from='osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-21T08:47:04.800 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:04 smithi152 ceph-mon[31401]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T08:47:04.800 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:04 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:47:04.800 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:04 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:47:04.800 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:04 smithi152 ceph-mon[31401]: from='osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197]' entity='osd.9' 2024-04-21T08:47:04.800 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T08:47:04.801 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T08:47:04.801 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-04-21T08:47:04.801 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-04-21T08:47:04.801 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131985 s, 794 MB/s 2024-04-21T08:47:04.801 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-04-21T08:47:04.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:04 smithi171 ceph-mon[31358]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:47:04.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:04 smithi171 ceph-mon[31358]: from='osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-21T08:47:04.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:04 smithi171 ceph-mon[31358]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T08:47:04.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:04 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:47:04.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:04 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:47:04.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:04 smithi171 ceph-mon[31358]: from='osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197]' entity='osd.9' 2024-04-21T08:47:05.534 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi152:vg_nvme/lv_1 2024-04-21T08:47:05.778 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:47:05.817 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: purged_snaps scrub starts 2024-04-21T08:47:05.817 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: purged_snaps scrub ok 2024-04-21T08:47:05.817 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:47:05.817 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197] boot 2024-04-21T08:47:05.817 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T08:47:05.817 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:47:05.817 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:05.817 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:05.818 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:05.818 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:05.818 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:05.818 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:05.818 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:05.818 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:05.818 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:05.818 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:05 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:05.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: purged_snaps scrub starts 2024-04-21T08:47:05.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: purged_snaps scrub ok 2024-04-21T08:47:05.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:47:05.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197] boot 2024-04-21T08:47:05.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T08:47:05.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:47:05.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:05.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:05.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:05.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:05.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:05.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:05.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:05.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:05.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:05.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:05 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:05.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: purged_snaps scrub starts 2024-04-21T08:47:05.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: purged_snaps scrub ok 2024-04-21T08:47:05.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:47:05.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: osd.9 [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197] boot 2024-04-21T08:47:05.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T08:47:05.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:47:05.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:05.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:05.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:05.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:05.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:05.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:05.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:05.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:05.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:05.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:05 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:06.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:06 smithi152 ceph-mon[31401]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:47:06.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:06 smithi152 ceph-mon[31401]: Detected new or changed devices on smithi028 2024-04-21T08:47:06.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:06 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi028 to 4273M 2024-04-21T08:47:06.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:06 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:06.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:06 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:06.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:06 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:06.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:06 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:06.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:06 smithi028 ceph-mon[28003]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:47:06.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:06 smithi028 ceph-mon[28003]: Detected new or changed devices on smithi028 2024-04-21T08:47:06.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:06 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi028 to 4273M 2024-04-21T08:47:06.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:06 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:06.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:06 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:06.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:06 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:06.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:06 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:06.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:06 smithi171 ceph-mon[31358]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:47:06.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:06 smithi171 ceph-mon[31358]: Detected new or changed devices on smithi028 2024-04-21T08:47:06.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:06 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi028 to 4273M 2024-04-21T08:47:06.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:06 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:06.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:06 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:06.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:06 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:06.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:06 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:07.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.523+0000 7fc775037700 1 -- 172.21.15.152:0/3065375887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc770102ba0 msgr2=0x7fc770103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:07.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.523+0000 7fc775037700 1 --2- 172.21.15.152:0/3065375887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc770102ba0 0x7fc770103000 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc7580099c0 tx=0x7fc758009cd0 comp rx=0 tx=0).stop 2024-04-21T08:47:07.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.523+0000 7fc775037700 1 -- 172.21.15.152:0/3065375887 shutdown_connections 2024-04-21T08:47:07.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.523+0000 7fc775037700 1 --2- 172.21.15.152:0/3065375887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc770103540 0x7fc770109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:07.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.523+0000 7fc775037700 1 --2- 172.21.15.152:0/3065375887 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc7701019a0 0x7fc770101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:07.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.523+0000 7fc775037700 1 --2- 172.21.15.152:0/3065375887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc770102ba0 0x7fc770103000 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:07.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.523+0000 7fc775037700 1 -- 172.21.15.152:0/3065375887 >> 172.21.15.152:0/3065375887 conn(0x7fc7700fcf40 msgr2=0x7fc7700ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:07.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.523+0000 7fc775037700 1 -- 172.21.15.152:0/3065375887 shutdown_connections 2024-04-21T08:47:07.523 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.523+0000 7fc775037700 1 -- 172.21.15.152:0/3065375887 wait complete. 2024-04-21T08:47:07.525 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.523+0000 7fc775037700 1 Processor -- start 2024-04-21T08:47:07.525 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc775037700 1 -- start start 2024-04-21T08:47:07.525 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc775037700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc7701019a0 0x7fc770198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:07.526 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc775037700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc770102ba0 0x7fc770199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:07.526 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc775037700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc770103540 0x7fc7701a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:07.526 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc775037700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc77010e5b0 con 0x7fc770102ba0 2024-04-21T08:47:07.526 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc775037700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc77010e430 con 0x7fc770103540 2024-04-21T08:47:07.526 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc775037700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc77010e730 con 0x7fc7701019a0 2024-04-21T08:47:07.526 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc76f59e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc770103540 0x7fc7701a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:07.526 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc76f59e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc770103540 0x7fc7701a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:44866/0 (socket says 172.21.15.152:44866) 2024-04-21T08:47:07.526 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc76f59e700 1 -- 172.21.15.152:0/979861391 learned_addr learned my addr 172.21.15.152:0/979861391 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-04-21T08:47:07.526 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc76e59c700 1 --2- 172.21.15.152:0/979861391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc770102ba0 0x7fc770199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:07.526 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc76ed9d700 1 --2- 172.21.15.152:0/979861391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc7701019a0 0x7fc770198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:07.526 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc76f59e700 1 -- 172.21.15.152:0/979861391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc7701019a0 msgr2=0x7fc770198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:07.526 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc76f59e700 1 --2- 172.21.15.152:0/979861391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc7701019a0 0x7fc770198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:07.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc76f59e700 1 -- 172.21.15.152:0/979861391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc770102ba0 msgr2=0x7fc770199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:07.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc76f59e700 1 --2- 172.21.15.152:0/979861391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc770102ba0 0x7fc770199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:07.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.524+0000 7fc76f59e700 1 -- 172.21.15.152:0/979861391 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7701a0830 con 0x7fc770103540 2024-04-21T08:47:07.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.525+0000 7fc76f59e700 1 --2- 172.21.15.152:0/979861391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc770103540 0x7fc7701a0190 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc76000c900 tx=0x7fc76000cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:07.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.525+0000 7fc767fff700 1 -- 172.21.15.152:0/979861391 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7600078d0 con 0x7fc770103540 2024-04-21T08:47:07.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.525+0000 7fc775037700 1 -- 172.21.15.152:0/979861391 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7701a0b20 con 0x7fc770103540 2024-04-21T08:47:07.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.525+0000 7fc767fff700 1 -- 172.21.15.152:0/979861391 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc76000f450 con 0x7fc770103540 2024-04-21T08:47:07.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.525+0000 7fc767fff700 1 -- 172.21.15.152:0/979861391 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc760018640 con 0x7fc770103540 2024-04-21T08:47:07.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.525+0000 7fc775037700 1 -- 172.21.15.152:0/979861391 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7701a1060 con 0x7fc770103540 2024-04-21T08:47:07.527 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.526+0000 7fc775037700 1 -- 172.21.15.152:0/979861391 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc770069730 con 0x7fc770103540 2024-04-21T08:47:07.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.528+0000 7fc767fff700 1 -- 172.21.15.152:0/979861391 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc760007a70 con 0x7fc770103540 2024-04-21T08:47:07.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.528+0000 7fc767fff700 1 --2- 172.21.15.152:0/979861391 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc75c0a3c40 0x7fc75c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:07.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.528+0000 7fc767fff700 1 -- 172.21.15.152:0/979861391 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7fc7600cfbf0 con 0x7fc770103540 2024-04-21T08:47:07.528 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.529+0000 7fc76ed9d700 1 --2- 172.21.15.152:0/979861391 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc75c0a3c40 0x7fc75c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:07.529 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.529+0000 7fc76ed9d700 1 --2- 172.21.15.152:0/979861391 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc75c0a3c40 0x7fc75c0a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc770102a00 tx=0x7fc768009180 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:07.531 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.532+0000 7fc767fff700 1 -- 172.21.15.152:0/979861391 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc760014030 con 0x7fc770103540 2024-04-21T08:47:07.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:07.739+0000 7fc775037700 1 -- 172.21.15.152:0/979861391 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc770062e80 con 0x7fc75c0a3c40 2024-04-21T08:47:07.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:07 smithi152 ceph-mon[31401]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T08:47:08.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:07 smithi028 ceph-mon[28003]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T08:47:08.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:07 smithi171 ceph-mon[31358]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T08:47:09.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:08 smithi152 ceph-mon[31401]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:09.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:08 smithi152 ceph-mon[31401]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:47:09.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:08 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:47:09.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:08 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:47:09.107 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:08 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:09.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:08 smithi028 ceph-mon[28003]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:09.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:08 smithi028 ceph-mon[28003]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:47:09.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:08 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:47:09.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:08 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:47:09.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:08 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:09.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:08 smithi171 ceph-mon[31358]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:09.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:08 smithi171 ceph-mon[31358]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:47:09.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:08 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:47:09.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:08 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:47:09.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:08 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:10.941 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:10 smithi152 ceph-mon[31401]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:10.941 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:10 smithi152 ceph-mon[31401]: from='client.? 172.21.15.152:0/2939283104' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03c59ec8-844b-4120-87bd-d4421e5027c2"}]: dispatch 2024-04-21T08:47:10.941 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:10 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03c59ec8-844b-4120-87bd-d4421e5027c2"}]: dispatch 2024-04-21T08:47:10.942 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:10 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03c59ec8-844b-4120-87bd-d4421e5027c2"}]': finished 2024-04-21T08:47:10.942 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:10 smithi152 ceph-mon[31401]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T08:47:10.942 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:10 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:10 smithi028 ceph-mon[28003]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:10 smithi028 ceph-mon[28003]: from='client.? 172.21.15.152:0/2939283104' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03c59ec8-844b-4120-87bd-d4421e5027c2"}]: dispatch 2024-04-21T08:47:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:10 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03c59ec8-844b-4120-87bd-d4421e5027c2"}]: dispatch 2024-04-21T08:47:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:10 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03c59ec8-844b-4120-87bd-d4421e5027c2"}]': finished 2024-04-21T08:47:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:10 smithi028 ceph-mon[28003]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T08:47:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:10 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:10 smithi171 ceph-mon[31358]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:10 smithi171 ceph-mon[31358]: from='client.? 172.21.15.152:0/2939283104' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03c59ec8-844b-4120-87bd-d4421e5027c2"}]: dispatch 2024-04-21T08:47:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:10 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03c59ec8-844b-4120-87bd-d4421e5027c2"}]: dispatch 2024-04-21T08:47:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:10 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03c59ec8-844b-4120-87bd-d4421e5027c2"}]': finished 2024-04-21T08:47:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:10 smithi171 ceph-mon[31358]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T08:47:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:10 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:12.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:11 smithi152 ceph-mon[31401]: from='client.? 172.21.15.152:0/1214234364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:47:12.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:11 smithi028 ceph-mon[28003]: from='client.? 172.21.15.152:0/1214234364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:47:12.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:11 smithi171 ceph-mon[31358]: from='client.? 172.21.15.152:0/1214234364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:47:13.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:12 smithi152 ceph-mon[31401]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:13.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:12 smithi028 ceph-mon[28003]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:12 smithi171 ceph-mon[31358]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:14.972 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:14 smithi152 ceph-mon[31401]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:15.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:14 smithi028 ceph-mon[28003]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:15.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:14 smithi171 ceph-mon[31358]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:16.940 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:16 smithi152 ceph-mon[31401]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:17.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:16 smithi028 ceph-mon[28003]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:17.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:16 smithi171 ceph-mon[31358]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:18.937 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:18 smithi171 ceph-mon[31358]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:18.937 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:18 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T08:47:18.937 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:18 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:18.937 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:18 smithi171 ceph-mon[31358]: Deploying daemon osd.10 on smithi152 2024-04-21T08:47:18.993 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:18 smithi152 ceph-mon[31401]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:18.993 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:18 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T08:47:18.993 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:18 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:18.993 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:18 smithi152 ceph-mon[31401]: Deploying daemon osd.10 on smithi152 2024-04-21T08:47:19.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:18 smithi028 ceph-mon[28003]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:19.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:18 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T08:47:19.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:18 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:19.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:18 smithi028 ceph-mon[28003]: Deploying daemon osd.10 on smithi152 2024-04-21T08:47:21.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:20 smithi152 ceph-mon[31401]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:21.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:20 smithi028 ceph-mon[28003]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:21.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:20 smithi171 ceph-mon[31358]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:23.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:22 smithi152 ceph-mon[31401]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:23.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:22 smithi028 ceph-mon[28003]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:23.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:22 smithi171 ceph-mon[31358]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:25.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:25 smithi152 ceph-mon[31401]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:25.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:25 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:25.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:25 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:25.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:25 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:25.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:25 smithi028 ceph-mon[28003]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:25.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:25 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:25.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:25 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:25.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:25 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:25.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:25 smithi171 ceph-mon[31358]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:25.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:25 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:25.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:25 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:25.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:25 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:27.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:27 smithi152 ceph-mon[31401]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:27.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:27 smithi028 ceph-mon[28003]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:27.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:27 smithi171 ceph-mon[31358]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:28.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:28.112+0000 7fc767fff700 1 -- 172.21.15.152:0/979861391 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fc770062e80 con 0x7fc75c0a3c40 2024-04-21T08:47:28.113 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 10 on host 'smithi152' 2024-04-21T08:47:28.114 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:28.115+0000 7fc775037700 1 -- 172.21.15.152:0/979861391 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc75c0a3c40 msgr2=0x7fc75c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:28.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:28.115+0000 7fc775037700 1 --2- 172.21.15.152:0/979861391 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc75c0a3c40 0x7fc75c0a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc770102a00 tx=0x7fc768009180 comp rx=0 tx=0).stop 2024-04-21T08:47:28.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:28.115+0000 7fc775037700 1 -- 172.21.15.152:0/979861391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc770103540 msgr2=0x7fc7701a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:28.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:28.115+0000 7fc775037700 1 --2- 172.21.15.152:0/979861391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc770103540 0x7fc7701a0190 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc76000c900 tx=0x7fc76000cc10 comp rx=0 tx=0).stop 2024-04-21T08:47:28.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:28.115+0000 7fc775037700 1 -- 172.21.15.152:0/979861391 shutdown_connections 2024-04-21T08:47:28.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:28.115+0000 7fc775037700 1 --2- 172.21.15.152:0/979861391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc770103540 0x7fc7701a0190 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:28.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:28.115+0000 7fc775037700 1 --2- 172.21.15.152:0/979861391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc7701019a0 0x7fc770198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:28.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:28.115+0000 7fc775037700 1 --2- 172.21.15.152:0/979861391 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc75c0a3c40 0x7fc75c0a6100 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:28.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:28.115+0000 7fc775037700 1 --2- 172.21.15.152:0/979861391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc770102ba0 0x7fc770199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:28.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:28.115+0000 7fc775037700 1 -- 172.21.15.152:0/979861391 >> 172.21.15.152:0/979861391 conn(0x7fc7700fcf40 msgr2=0x7fc770106060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:28.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:28.115+0000 7fc775037700 1 -- 172.21.15.152:0/979861391 shutdown_connections 2024-04-21T08:47:28.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-21T08:47:28.115+0000 7fc775037700 1 -- 172.21.15.152:0/979861391 wait complete. 2024-04-21T08:47:28.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:28.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:28.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:28.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:28.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:28.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:28 smithi152 ceph-mon[31401]: from='osd.10 [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:47:28.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:28 smithi152 ceph-mon[31401]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:47:28.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:28.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:28.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:28.411 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:28.411 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:28.411 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:28.411 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:28.411 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:28.411 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:28 smithi028 ceph-mon[28003]: from='osd.10 [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:47:28.411 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:28 smithi028 ceph-mon[28003]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:47:28.411 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:28.411 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:28.411 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:28.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:28 smithi171 ceph-mon[31358]: from='osd.10 [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:47:28.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:28 smithi171 ceph-mon[31358]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:47:28.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:28.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:28.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:28.941 DEBUG:teuthology.orchestra.run.smithi152:osd.10> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.10.service 2024-04-21T08:47:28.944 INFO:tasks.cephadm:Deploying osd.11 on smithi171 with /dev/vg_nvme/lv_1... 2024-04-21T08:47:28.944 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T08:47:29.180 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:29 smithi152 ceph-mon[31401]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:29.180 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:29 smithi152 ceph-mon[31401]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T08:47:29.181 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:29 smithi152 ceph-mon[31401]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T08:47:29.181 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:29 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:29.181 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:29 smithi152 ceph-mon[31401]: from='osd.10 [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:47:29.181 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:29 smithi152 ceph-mon[31401]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:47:29.191 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:47:29.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:29 smithi171 ceph-mon[31358]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:29.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:29 smithi171 ceph-mon[31358]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T08:47:29.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:29 smithi171 ceph-mon[31358]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T08:47:29.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:29 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:29.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:29 smithi171 ceph-mon[31358]: from='osd.10 [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:47:29.313 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:29 smithi171 ceph-mon[31358]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:47:29.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:29 smithi028 ceph-mon[28003]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:29.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:29 smithi028 ceph-mon[28003]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T08:47:29.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:29 smithi028 ceph-mon[28003]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T08:47:29.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:29 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:29.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:29 smithi028 ceph-mon[28003]: from='osd.10 [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:47:29.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:29 smithi028 ceph-mon[28003]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-04-21T08:47:29.854 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 08:47:29 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T08:47:29.579+0000 7f4d877d3700 -1 osd.10 0 waiting for initial osdmap 2024-04-21T08:47:29.855 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 08:47:29 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T08:47:29.590+0000 7f4d7f5be700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:47:30.532 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:30 smithi152 ceph-mon[31401]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-04-21T08:47:30.532 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:30 smithi152 ceph-mon[31401]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T08:47:30.532 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:30.532 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:30 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:30.532 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:30 smithi152 ceph-mon[31401]: from='osd.10 ' entity='osd.10' 2024-04-21T08:47:30.565 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:30 smithi171 ceph-mon[31358]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-04-21T08:47:30.565 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:30 smithi171 ceph-mon[31358]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T08:47:30.566 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:30.566 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:30 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:30.566 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:30 smithi171 ceph-mon[31358]: from='osd.10 ' entity='osd.10' 2024-04-21T08:47:30.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:30 smithi028 ceph-mon[28003]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-04-21T08:47:30.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:30 smithi028 ceph-mon[28003]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T08:47:30.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:30.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:30 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:30.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:30 smithi028 ceph-mon[28003]: from='osd.10 ' entity='osd.10' 2024-04-21T08:47:31.345 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T08:47:31.345 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T08:47:31.345 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-04-21T08:47:31.345 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-04-21T08:47:31.346 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011464 s, 915 MB/s 2024-04-21T08:47:31.346 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-04-21T08:47:31.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:31 smithi152 ceph-mon[31401]: purged_snaps scrub starts 2024-04-21T08:47:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:31 smithi152 ceph-mon[31401]: purged_snaps scrub ok 2024-04-21T08:47:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:31 smithi152 ceph-mon[31401]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:31 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:31 smithi152 ceph-mon[31401]: osd.10 [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203] boot 2024-04-21T08:47:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:31 smithi152 ceph-mon[31401]: osdmap e72: 11 total, 11 up, 11 in 2024-04-21T08:47:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:31 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:31.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:31 smithi028 ceph-mon[28003]: purged_snaps scrub starts 2024-04-21T08:47:31.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:31 smithi028 ceph-mon[28003]: purged_snaps scrub ok 2024-04-21T08:47:31.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:31 smithi028 ceph-mon[28003]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:31 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:31 smithi028 ceph-mon[28003]: osd.10 [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203] boot 2024-04-21T08:47:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:31 smithi028 ceph-mon[28003]: osdmap e72: 11 total, 11 up, 11 in 2024-04-21T08:47:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:31 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:31 smithi171 ceph-mon[31358]: purged_snaps scrub starts 2024-04-21T08:47:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:31 smithi171 ceph-mon[31358]: purged_snaps scrub ok 2024-04-21T08:47:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:31 smithi171 ceph-mon[31358]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:47:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:31 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:31 smithi171 ceph-mon[31358]: osd.10 [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203] boot 2024-04-21T08:47:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:31 smithi171 ceph-mon[31358]: osdmap e72: 11 total, 11 up, 11 in 2024-04-21T08:47:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:31 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:47:32.016 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi171:vg_nvme/lv_1 2024-04-21T08:47:32.258 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:47:32.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:32 smithi152 ceph-mon[31401]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T08:47:32.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:32 smithi171 ceph-mon[31358]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T08:47:32.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:32 smithi028 ceph-mon[28003]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T08:47:33.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:33 smithi171 ceph-mon[31358]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:33.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:33 smithi171 ceph-mon[31358]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T08:47:33.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:33.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:33.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:33.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:33.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:33.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:33.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:33.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:33.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:33.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:33.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:33 smithi152 ceph-mon[31401]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:33.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:33 smithi152 ceph-mon[31401]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T08:47:33.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:33.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:33.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:33.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:33.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:33.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:33.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:33.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:33.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:33.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:33.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:33 smithi028 ceph-mon[28003]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:33.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:33 smithi028 ceph-mon[28003]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T08:47:33.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:33.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:33.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:33.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:33.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:33.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:33.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:33.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:33.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:33.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:34.047 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.045+0000 7f2e382c7700 1 -- 172.21.15.171:0/2988365722 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e30103ca0 msgr2=0x7f2e30104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:34.047 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.045+0000 7f2e382c7700 1 --2- 172.21.15.171:0/2988365722 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e30103ca0 0x7f2e30104100 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2e20009990 tx=0x7f2e20009ca0 comp rx=0 tx=0).stop 2024-04-21T08:47:34.047 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.045+0000 7f2e382c7700 1 -- 172.21.15.171:0/2988365722 shutdown_connections 2024-04-21T08:47:34.047 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.045+0000 7f2e382c7700 1 --2- 172.21.15.171:0/2988365722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e30102aa0 0x7f2e30102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:34.047 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.045+0000 7f2e382c7700 1 --2- 172.21.15.171:0/2988365722 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e30104640 0x7f2e3010aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:34.047 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.045+0000 7f2e382c7700 1 --2- 172.21.15.171:0/2988365722 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e30103ca0 0x7f2e30104100 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:34.047 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.045+0000 7f2e382c7700 1 -- 172.21.15.171:0/2988365722 >> 172.21.15.171:0/2988365722 conn(0x7f2e300fe040 msgr2=0x7f2e30100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:34.047 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.045+0000 7f2e382c7700 1 -- 172.21.15.171:0/2988365722 shutdown_connections 2024-04-21T08:47:34.047 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.045+0000 7f2e382c7700 1 -- 172.21.15.171:0/2988365722 wait complete. 2024-04-21T08:47:34.047 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.045+0000 7f2e382c7700 1 Processor -- start 2024-04-21T08:47:34.048 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e382c7700 1 -- start start 2024-04-21T08:47:34.048 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e382c7700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e30102aa0 0x7f2e30198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:34.048 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e382c7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e30103ca0 0x7f2e30199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:34.048 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e382c7700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e30104640 0x7f2e301a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:34.048 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e382c7700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3010e510 con 0x7f2e30103ca0 2024-04-21T08:47:34.048 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e382c7700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3010e390 con 0x7f2e30102aa0 2024-04-21T08:47:34.048 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e382c7700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3010e690 con 0x7f2e30104640 2024-04-21T08:47:34.048 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e36864700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e30104640 0x7f2e301a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:34.048 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e36864700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e30104640 0x7f2e301a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:40512/0 (socket says 172.21.15.171:40512) 2024-04-21T08:47:34.048 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e36864700 1 -- 172.21.15.171:0/1973893619 learned_addr learned my addr 172.21.15.171:0/1973893619 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-04-21T08:47:34.048 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e36063700 1 --2- 172.21.15.171:0/1973893619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e30102aa0 0x7f2e30198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:34.048 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e35862700 1 --2- 172.21.15.171:0/1973893619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e30103ca0 0x7f2e30199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:34.048 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e36864700 1 -- 172.21.15.171:0/1973893619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e30102aa0 msgr2=0x7f2e30198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:34.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e36864700 1 --2- 172.21.15.171:0/1973893619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e30102aa0 0x7f2e30198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:34.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e36864700 1 -- 172.21.15.171:0/1973893619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e30103ca0 msgr2=0x7f2e30199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:34.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e36864700 1 --2- 172.21.15.171:0/1973893619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e30103ca0 0x7f2e30199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:34.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e36864700 1 -- 172.21.15.171:0/1973893619 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e301a0870 con 0x7f2e30104640 2024-04-21T08:47:34.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e36864700 1 --2- 172.21.15.171:0/1973893619 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e30104640 0x7f2e301a01a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2e28007ee0 tx=0x7f2e2800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:34.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.046+0000 7f2e277fe700 1 -- 172.21.15.171:0/1973893619 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e2800dbd0 con 0x7f2e30104640 2024-04-21T08:47:34.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.047+0000 7f2e382c7700 1 -- 172.21.15.171:0/1973893619 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e301a0b60 con 0x7f2e30104640 2024-04-21T08:47:34.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.047+0000 7f2e277fe700 1 -- 172.21.15.171:0/1973893619 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e2800dd70 con 0x7f2e30104640 2024-04-21T08:47:34.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.047+0000 7f2e277fe700 1 -- 172.21.15.171:0/1973893619 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e28005990 con 0x7f2e30104640 2024-04-21T08:47:34.049 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.047+0000 7f2e382c7700 1 -- 172.21.15.171:0/1973893619 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e301a10a0 con 0x7f2e30104640 2024-04-21T08:47:34.050 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.048+0000 7f2e382c7700 1 -- 172.21.15.171:0/1973893619 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e30069730 con 0x7f2e30104640 2024-04-21T08:47:34.050 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.049+0000 7f2e277fe700 1 -- 172.21.15.171:0/1973893619 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2e2801a020 con 0x7f2e30104640 2024-04-21T08:47:34.050 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.049+0000 7f2e277fe700 1 --2- 172.21.15.171:0/1973893619 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2e1c0a3c40 0x7f2e1c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:34.050 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.049+0000 7f2e277fe700 1 -- 172.21.15.171:0/1973893619 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7f2e280cf810 con 0x7f2e30104640 2024-04-21T08:47:34.050 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.049+0000 7f2e36063700 1 --2- 172.21.15.171:0/1973893619 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2e1c0a3c40 0x7f2e1c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:34.051 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.050+0000 7f2e36063700 1 --2- 172.21.15.171:0/1973893619 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2e1c0a3c40 0x7f2e1c0a6100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c005fd0 tx=0x7f2e2c005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:34.052 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.053+0000 7f2e277fe700 1 -- 172.21.15.171:0/1973893619 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2e28010070 con 0x7f2e30104640 2024-04-21T08:47:34.262 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:34.263+0000 7f2e382c7700 1 -- 172.21.15.171:0/1973893619 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2e30062e80 con 0x7f2e1c0a3c40 2024-04-21T08:47:34.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:34 smithi152 ceph-mon[31401]: Detected new or changed devices on smithi152 2024-04-21T08:47:34.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:34 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi152 to 4272M 2024-04-21T08:47:34.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:34 smithi152 ceph-mon[31401]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T08:47:34.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:34 smithi028 ceph-mon[28003]: Detected new or changed devices on smithi152 2024-04-21T08:47:34.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:34 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi152 to 4272M 2024-04-21T08:47:34.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:34 smithi028 ceph-mon[28003]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T08:47:34.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:34 smithi171 ceph-mon[31358]: Detected new or changed devices on smithi152 2024-04-21T08:47:34.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:34 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi152 to 4272M 2024-04-21T08:47:34.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:34 smithi171 ceph-mon[31358]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T08:47:35.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:35 smithi152 ceph-mon[31401]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:35.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:35 smithi152 ceph-mon[31401]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T08:47:35.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:35 smithi152 ceph-mon[31401]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:47:35.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:35 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:47:35.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:35 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:47:35.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:35 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:35.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:35 smithi028 ceph-mon[28003]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:35.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:35 smithi028 ceph-mon[28003]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T08:47:35.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:35 smithi028 ceph-mon[28003]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:47:35.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:47:35.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:47:35.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:35.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:35 smithi171 ceph-mon[31358]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:35.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:35 smithi171 ceph-mon[31358]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T08:47:35.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:35 smithi171 ceph-mon[31358]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:47:35.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:47:35.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:47:35.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:37.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:37 smithi152 ceph-mon[31401]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-04-21T08:47:37.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:37 smithi152 ceph-mon[31401]: from='client.? 172.21.15.171:0/2339216050' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d57b2d5-a408-49a2-baf3-a0ebce5097f7"}]: dispatch 2024-04-21T08:47:37.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:37 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d57b2d5-a408-49a2-baf3-a0ebce5097f7"}]: dispatch 2024-04-21T08:47:37.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:37 smithi152 ceph-mon[31401]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d57b2d5-a408-49a2-baf3-a0ebce5097f7"}]': finished 2024-04-21T08:47:37.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:37 smithi152 ceph-mon[31401]: osdmap e77: 12 total, 11 up, 12 in 2024-04-21T08:47:37.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:47:37.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:37 smithi028 ceph-mon[28003]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-04-21T08:47:37.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:37 smithi028 ceph-mon[28003]: from='client.? 172.21.15.171:0/2339216050' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d57b2d5-a408-49a2-baf3-a0ebce5097f7"}]: dispatch 2024-04-21T08:47:37.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:37 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d57b2d5-a408-49a2-baf3-a0ebce5097f7"}]: dispatch 2024-04-21T08:47:37.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:37 smithi028 ceph-mon[28003]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d57b2d5-a408-49a2-baf3-a0ebce5097f7"}]': finished 2024-04-21T08:47:37.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:37 smithi028 ceph-mon[28003]: osdmap e77: 12 total, 11 up, 12 in 2024-04-21T08:47:37.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:47:37.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:37 smithi171 ceph-mon[31358]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-04-21T08:47:37.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:37 smithi171 ceph-mon[31358]: from='client.? 172.21.15.171:0/2339216050' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d57b2d5-a408-49a2-baf3-a0ebce5097f7"}]: dispatch 2024-04-21T08:47:37.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:37 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d57b2d5-a408-49a2-baf3-a0ebce5097f7"}]: dispatch 2024-04-21T08:47:37.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:37 smithi171 ceph-mon[31358]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d57b2d5-a408-49a2-baf3-a0ebce5097f7"}]': finished 2024-04-21T08:47:37.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:37 smithi171 ceph-mon[31358]: osdmap e77: 12 total, 11 up, 12 in 2024-04-21T08:47:37.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:47:38.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:38 smithi028 ceph-mon[28003]: from='client.? 172.21.15.171:0/2958656893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:47:38.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:38 smithi152 ceph-mon[31401]: from='client.? 172.21.15.171:0/2958656893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:47:38.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:38 smithi171 ceph-mon[31358]: from='client.? 172.21.15.171:0/2958656893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:47:39.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:39 smithi152 ceph-mon[31401]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-04-21T08:47:39.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:39 smithi028 ceph-mon[28003]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-04-21T08:47:39.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:39 smithi171 ceph-mon[31358]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-04-21T08:47:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:41 smithi171 ceph-mon[31358]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-21T08:47:41.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:41 smithi152 ceph-mon[31401]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-21T08:47:41.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:41 smithi028 ceph-mon[28003]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-21T08:47:43.512 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:43 smithi171 ceph-mon[31358]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:47:43.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:43 smithi152 ceph-mon[31401]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:47:43.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:43 smithi028 ceph-mon[28003]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:47:45.462 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:45 smithi171 ceph-mon[31358]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-21T08:47:45.462 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T08:47:45.463 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:45.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:45 smithi152 ceph-mon[31401]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-21T08:47:45.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T08:47:45.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:45.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:45 smithi028 ceph-mon[28003]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-21T08:47:45.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T08:47:45.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:46.574 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:46 smithi171 ceph-mon[31358]: Deploying daemon osd.11 on smithi171 2024-04-21T08:47:46.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:46 smithi152 ceph-mon[31401]: Deploying daemon osd.11 on smithi171 2024-04-21T08:47:46.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:46 smithi028 ceph-mon[28003]: Deploying daemon osd.11 on smithi171 2024-04-21T08:47:47.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:47 smithi152 ceph-mon[31401]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:47.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:47 smithi028 ceph-mon[28003]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:47.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:47 smithi171 ceph-mon[31358]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:49.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:49 smithi171 ceph-mon[31358]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:49.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:49 smithi152 ceph-mon[31401]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:49.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:49 smithi028 ceph-mon[28003]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:51.435 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:51 smithi171 ceph-mon[31358]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:51.435 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:51.435 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:51.435 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:51 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:51.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:51 smithi152 ceph-mon[31401]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:51.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:51.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:51.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:51 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:51.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:51 smithi028 ceph-mon[28003]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:51.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:51.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:51.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:51 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:53.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:53 smithi171 ceph-mon[31358]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:53.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:53 smithi152 ceph-mon[31401]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:53.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:53 smithi028 ceph-mon[28003]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:54.585 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 11 on host 'smithi171' 2024-04-21T08:47:54.585 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:54.586+0000 7f2e277fe700 1 -- 172.21.15.171:0/1973893619 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f2e30062e80 con 0x7f2e1c0a3c40 2024-04-21T08:47:54.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:54.589+0000 7f2e382c7700 1 -- 172.21.15.171:0/1973893619 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2e1c0a3c40 msgr2=0x7f2e1c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:54.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:54.589+0000 7f2e382c7700 1 --2- 172.21.15.171:0/1973893619 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2e1c0a3c40 0x7f2e1c0a6100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c005fd0 tx=0x7f2e2c005dd0 comp rx=0 tx=0).stop 2024-04-21T08:47:54.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:54.589+0000 7f2e382c7700 1 -- 172.21.15.171:0/1973893619 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e30104640 msgr2=0x7f2e301a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:54.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:54.589+0000 7f2e382c7700 1 --2- 172.21.15.171:0/1973893619 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e30104640 0x7f2e301a01a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2e28007ee0 tx=0x7f2e2800d3b0 comp rx=0 tx=0).stop 2024-04-21T08:47:54.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:54.589+0000 7f2e382c7700 1 -- 172.21.15.171:0/1973893619 shutdown_connections 2024-04-21T08:47:54.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:54.589+0000 7f2e382c7700 1 --2- 172.21.15.171:0/1973893619 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e30104640 0x7f2e301a01a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:54.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:54.589+0000 7f2e382c7700 1 --2- 172.21.15.171:0/1973893619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e30102aa0 0x7f2e30198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:54.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:54.589+0000 7f2e382c7700 1 --2- 172.21.15.171:0/1973893619 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2e1c0a3c40 0x7f2e1c0a6100 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:54.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:54.589+0000 7f2e382c7700 1 --2- 172.21.15.171:0/1973893619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e30103ca0 0x7f2e30199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:54.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:54.589+0000 7f2e382c7700 1 -- 172.21.15.171:0/1973893619 >> 172.21.15.171:0/1973893619 conn(0x7f2e300fe040 msgr2=0x7f2e30107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:54.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:54.589+0000 7f2e382c7700 1 -- 172.21.15.171:0/1973893619 shutdown_connections 2024-04-21T08:47:54.588 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:47:54.589+0000 7f2e382c7700 1 -- 172.21.15.171:0/1973893619 wait complete. 2024-04-21T08:47:55.217 DEBUG:teuthology.orchestra.run.smithi171:osd.11> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.11.service 2024-04-21T08:47:55.219 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-21T08:47:55.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-21T08:47:55.482 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:47:55.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:55 smithi152 ceph-mon[31401]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:55.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:55.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:55 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:55.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:55 smithi152 ceph-mon[31401]: from='osd.11 [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:47:55.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:55 smithi152 ceph-mon[31401]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:47:55.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:55 smithi028 ceph-mon[28003]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:55.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:55.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:55 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:55.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:55 smithi028 ceph-mon[28003]: from='osd.11 [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:47:55.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:55 smithi028 ceph-mon[28003]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:47:55.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:55 smithi171 ceph-mon[31358]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:55.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:55.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:55 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:55.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:55 smithi171 ceph-mon[31358]: from='osd.11 [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:47:55.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:55 smithi171 ceph-mon[31358]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:47:56.567 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:56 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:56.567 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:56 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:56.567 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:56 smithi171 ceph-mon[31358]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T08:47:56.567 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:56 smithi171 ceph-mon[31358]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T08:47:56.567 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:56 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:47:56.567 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:56 smithi171 ceph-mon[31358]: from='osd.11 [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:47:56.567 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:56 smithi171 ceph-mon[31358]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:47:56.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:56 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:56.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:56 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:56.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:56 smithi028 ceph-mon[28003]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T08:47:56.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:56 smithi028 ceph-mon[28003]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T08:47:56.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:56 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:47:56.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:56 smithi028 ceph-mon[28003]: from='osd.11 [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:47:56.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:56 smithi028 ceph-mon[28003]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:47:56.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:56 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:56.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:56 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:56.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:56 smithi152 ceph-mon[31401]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T08:47:56.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:56 smithi152 ceph-mon[31401]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T08:47:56.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:56 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:47:56.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:56 smithi152 ceph-mon[31401]: from='osd.11 [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:47:56.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:56 smithi152 ceph-mon[31401]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-04-21T08:47:56.870 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 08:47:56 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T08:47:56.566+0000 7f11ba7fc700 -1 osd.11 0 waiting for initial osdmap 2024-04-21T08:47:56.870 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 08:47:56 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T08:47:56.581+0000 7f11b65ef700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:47:57.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.346+0000 7fd9be7c7700 1 -- 172.21.15.28:0/2625404676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8104660 msgr2=0x7fd9b810af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:57.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.346+0000 7fd9be7c7700 1 --2- 172.21.15.28:0/2625404676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8104660 0x7fd9b810af10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000b0d0 tx=0x7fd9b000b3e0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.346+0000 7fd9be7c7700 1 -- 172.21.15.28:0/2625404676 shutdown_connections 2024-04-21T08:47:57.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.346+0000 7fd9be7c7700 1 --2- 172.21.15.28:0/2625404676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8104660 0x7fd9b810af10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.346+0000 7fd9be7c7700 1 --2- 172.21.15.28:0/2625404676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd9b8102ac0 0x7fd9b8102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.346+0000 7fd9be7c7700 1 --2- 172.21.15.28:0/2625404676 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9b8103cc0 0x7fd9b8104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.346+0000 7fd9be7c7700 1 -- 172.21.15.28:0/2625404676 >> 172.21.15.28:0/2625404676 conn(0x7fd9b80fe060 msgr2=0x7fd9b81004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:57.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.346+0000 7fd9be7c7700 1 -- 172.21.15.28:0/2625404676 shutdown_connections 2024-04-21T08:47:57.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.346+0000 7fd9be7c7700 1 -- 172.21.15.28:0/2625404676 wait complete. 2024-04-21T08:47:57.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.346+0000 7fd9be7c7700 1 Processor -- start 2024-04-21T08:47:57.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.346+0000 7fd9be7c7700 1 -- start start 2024-04-21T08:47:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.347+0000 7fd9be7c7700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9b8102ac0 0x7fd9b8198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.347+0000 7fd9be7c7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8103cc0 0x7fd9b8199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.347+0000 7fd9be7c7700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd9b8104660 0x7fd9b81a00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.347+0000 7fd9be7c7700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b810e540 con 0x7fd9b8103cc0 2024-04-21T08:47:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.347+0000 7fd9be7c7700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b810e3c0 con 0x7fd9b8104660 2024-04-21T08:47:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.347+0000 7fd9be7c7700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b810e6c0 con 0x7fd9b8102ac0 2024-04-21T08:47:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.347+0000 7fd9b77fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8103cc0 0x7fd9b8199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.347+0000 7fd9b77fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8103cc0 0x7fd9b8199070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52484/0 (socket says 172.21.15.28:52484) 2024-04-21T08:47:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.347+0000 7fd9b77fe700 1 -- 172.21.15.28:0/1219115493 learned_addr learned my addr 172.21.15.28:0/1219115493 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:47:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.347+0000 7fd9b7fff700 1 --2- 172.21.15.28:0/1219115493 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9b8102ac0 0x7fd9b8198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.347+0000 7fd9bcd64700 1 --2- 172.21.15.28:0/1219115493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd9b8104660 0x7fd9b81a00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.348+0000 7fd9b77fe700 1 -- 172.21.15.28:0/1219115493 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9b8102ac0 msgr2=0x7fd9b8198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:57.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.348+0000 7fd9b77fe700 1 --2- 172.21.15.28:0/1219115493 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9b8102ac0 0x7fd9b8198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.348+0000 7fd9b77fe700 1 -- 172.21.15.28:0/1219115493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd9b8104660 msgr2=0x7fd9b81a00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:57.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.348+0000 7fd9b77fe700 1 --2- 172.21.15.28:0/1219115493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd9b8104660 0x7fd9b81a00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.348+0000 7fd9b77fe700 1 -- 172.21.15.28:0/1219115493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b8074c70 con 0x7fd9b8103cc0 2024-04-21T08:47:57.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.348+0000 7fd9bcd64700 1 --2- 172.21.15.28:0/1219115493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd9b8104660 0x7fd9b81a00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:57.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.348+0000 7fd9b77fe700 1 --2- 172.21.15.28:0/1219115493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8103cc0 0x7fd9b8199070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00d980 tx=0x7fd9ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:57.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.348+0000 7fd9b57fa700 1 -- 172.21.15.28:0/1219115493 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac016070 con 0x7fd9b8103cc0 2024-04-21T08:47:57.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.348+0000 7fd9be7c7700 1 -- 172.21.15.28:0/1219115493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b8074f60 con 0x7fd9b8103cc0 2024-04-21T08:47:57.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.348+0000 7fd9b57fa700 1 -- 172.21.15.28:0/1219115493 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9ac00de70 con 0x7fd9b8103cc0 2024-04-21T08:47:57.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.349+0000 7fd9b57fa700 1 -- 172.21.15.28:0/1219115493 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac00f3c0 con 0x7fd9b8103cc0 2024-04-21T08:47:57.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.349+0000 7fd9be7c7700 1 -- 172.21.15.28:0/1219115493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b80754f0 con 0x7fd9b8103cc0 2024-04-21T08:47:57.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.350+0000 7fd9be7c7700 1 -- 172.21.15.28:0/1219115493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9b8069730 con 0x7fd9b8103cc0 2024-04-21T08:47:57.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.350+0000 7fd9b57fa700 1 -- 172.21.15.28:0/1219115493 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd9ac019070 con 0x7fd9b8103cc0 2024-04-21T08:47:57.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.351+0000 7fd9b57fa700 1 --2- 172.21.15.28:0/1219115493 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd9a00a3c40 0x7fd9a00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.352+0000 7fd9b7fff700 1 --2- 172.21.15.28:0/1219115493 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd9a00a3c40 0x7fd9a00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.352+0000 7fd9b57fa700 1 -- 172.21.15.28:0/1219115493 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7fd9ac0cfd90 con 0x7fd9b8103cc0 2024-04-21T08:47:57.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.352+0000 7fd9b7fff700 1 --2- 172.21.15.28:0/1219115493 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd9a00a3c40 0x7fd9a00a6100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8103b20 tx=0x7fd9a800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:57.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.356+0000 7fd9b57fa700 1 -- 172.21.15.28:0/1219115493 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd9ac09cc90 con 0x7fd9b8103cc0 2024-04-21T08:47:57.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.568+0000 7fd9be7c7700 1 -- 172.21.15.28:0/1219115493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fd9b8109960 con 0x7fd9b8103cc0 2024-04-21T08:47:57.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.569+0000 7fd9b57fa700 1 -- 172.21.15.28:0/1219115493 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7fd9ac09c590 con 0x7fd9b8103cc0 2024-04-21T08:47:57.568 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:47:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.571+0000 7fd9be7c7700 1 -- 172.21.15.28:0/1219115493 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd9a00a3c40 msgr2=0x7fd9a00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.571+0000 7fd9be7c7700 1 --2- 172.21.15.28:0/1219115493 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd9a00a3c40 0x7fd9a00a6100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8103b20 tx=0x7fd9a800a380 comp rx=0 tx=0).stop 2024-04-21T08:47:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.571+0000 7fd9be7c7700 1 -- 172.21.15.28:0/1219115493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8103cc0 msgr2=0x7fd9b8199070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.571+0000 7fd9be7c7700 1 --2- 172.21.15.28:0/1219115493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8103cc0 0x7fd9b8199070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00d980 tx=0x7fd9ac00dc90 comp rx=0 tx=0).stop 2024-04-21T08:47:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.571+0000 7fd9be7c7700 1 -- 172.21.15.28:0/1219115493 shutdown_connections 2024-04-21T08:47:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.571+0000 7fd9be7c7700 1 --2- 172.21.15.28:0/1219115493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd9b8104660 0x7fd9b81a00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.571+0000 7fd9be7c7700 1 --2- 172.21.15.28:0/1219115493 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9b8102ac0 0x7fd9b8198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.571+0000 7fd9be7c7700 1 --2- 172.21.15.28:0/1219115493 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd9a00a3c40 0x7fd9a00a6100 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.571+0000 7fd9be7c7700 1 --2- 172.21.15.28:0/1219115493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8103cc0 0x7fd9b8199070 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.571+0000 7fd9be7c7700 1 -- 172.21.15.28:0/1219115493 >> 172.21.15.28:0/1219115493 conn(0x7fd9b80fe060 msgr2=0x7fd9b8106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.571+0000 7fd9be7c7700 1 -- 172.21.15.28:0/1219115493 shutdown_connections 2024-04-21T08:47:57.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:47:57.571+0000 7fd9be7c7700 1 -- 172.21.15.28:0/1219115493 wait complete. 2024-04-21T08:47:57.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:57 smithi028 ceph-mon[28003]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:57.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:57 smithi028 ceph-mon[28003]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-21T08:47:57.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:57 smithi028 ceph-mon[28003]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T08:47:57.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:57 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:47:57.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:57 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:47:57.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:57 smithi171 ceph-mon[31358]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:57.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:57 smithi171 ceph-mon[31358]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-21T08:47:57.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:57 smithi171 ceph-mon[31358]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T08:47:57.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:57 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:47:57.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:57 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:47:57.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:57 smithi152 ceph-mon[31401]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:57.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:57 smithi152 ceph-mon[31401]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-04-21T08:47:57.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:57 smithi152 ceph-mon[31401]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T08:47:57.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:57 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:47:57.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:57 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:47:58.255 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1713689277,"num_in_osds":12,"osd_in_since":1713689257,"num_remapped_pgs":0} 2024-04-21T08:47:58.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T08:47:58.504 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:47:58.734 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:58 smithi171 ceph-mon[31358]: purged_snaps scrub starts 2024-04-21T08:47:58.734 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:58 smithi171 ceph-mon[31358]: purged_snaps scrub ok 2024-04-21T08:47:58.734 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:58 smithi171 ceph-mon[31358]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:58.734 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:58 smithi171 ceph-mon[31358]: osd.11 [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417] boot 2024-04-21T08:47:58.734 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:58 smithi171 ceph-mon[31358]: osdmap e80: 12 total, 12 up, 12 in 2024-04-21T08:47:58.734 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:58 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:47:58.734 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:58 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1219115493' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:47:58.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:58 smithi152 ceph-mon[31401]: purged_snaps scrub starts 2024-04-21T08:47:58.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:58 smithi152 ceph-mon[31401]: purged_snaps scrub ok 2024-04-21T08:47:58.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:58 smithi152 ceph-mon[31401]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:58.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:58 smithi152 ceph-mon[31401]: osd.11 [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417] boot 2024-04-21T08:47:58.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:58 smithi152 ceph-mon[31401]: osdmap e80: 12 total, 12 up, 12 in 2024-04-21T08:47:58.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:58 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:47:58.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:58 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1219115493' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:47:58.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:58 smithi028 ceph-mon[28003]: purged_snaps scrub starts 2024-04-21T08:47:58.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:58 smithi028 ceph-mon[28003]: purged_snaps scrub ok 2024-04-21T08:47:58.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:58 smithi028 ceph-mon[28003]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:47:58.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:58 smithi028 ceph-mon[28003]: osd.11 [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417] boot 2024-04-21T08:47:58.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:58 smithi028 ceph-mon[28003]: osdmap e80: 12 total, 12 up, 12 in 2024-04-21T08:47:58.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:58 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:47:58.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:58 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1219115493' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:47:59.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:59 smithi152 ceph-mon[31401]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T08:47:59.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:59 smithi152 ceph-mon[31401]: Detected new or changed devices on smithi171 2024-04-21T08:47:59.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:59 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:59.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:59 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:59.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:59 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:59.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:59 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:59.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:59 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:59.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:59 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:59.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:59 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi171 to 4272M 2024-04-21T08:47:59.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:59 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:59.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:59 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:59.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:59 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:59.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:59 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:59.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:47:59 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:59.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:59 smithi028 ceph-mon[28003]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T08:47:59.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:59 smithi028 ceph-mon[28003]: Detected new or changed devices on smithi171 2024-04-21T08:47:59.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:59 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:59.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:59 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:59.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:59 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:59.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:59 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:59.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:59 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:59.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:59 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:59.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:59 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi171 to 4272M 2024-04-21T08:47:59.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:59 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:59.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:59 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:59.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:59 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:59.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:59 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:59.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:47:59 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:59.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:59 smithi171 ceph-mon[31358]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T08:47:59.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:59 smithi171 ceph-mon[31358]: Detected new or changed devices on smithi171 2024-04-21T08:47:59.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:59 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:59.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:59 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:59.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:59 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:59.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:59 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:59.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:59 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:59.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:59 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:47:59.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:59 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi171 to 4272M 2024-04-21T08:47:59.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:59 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:59.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:59 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:59.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:59 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:59.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:59 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:47:59.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:47:59 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:00.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.444+0000 7f41e4d76700 1 -- 172.21.15.28:0/2060021623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e0102ac0 msgr2=0x7f41e0102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:00.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.444+0000 7f41e4d76700 1 --2- 172.21.15.28:0/2060021623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e0102ac0 0x7f41e0102f40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f41d4009990 tx=0x7f41d4009ca0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.444+0000 7f41e4d76700 1 -- 172.21.15.28:0/2060021623 shutdown_connections 2024-04-21T08:48:00.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.444+0000 7f41e4d76700 1 --2- 172.21.15.28:0/2060021623 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f41e0103480 0x7f41e010f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.444+0000 7f41e4d76700 1 --2- 172.21.15.28:0/2060021623 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41e0108a70 0x7f41e0108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.444+0000 7f41e4d76700 1 --2- 172.21.15.28:0/2060021623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e0102ac0 0x7f41e0102f40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.444+0000 7f41e4d76700 1 -- 172.21.15.28:0/2060021623 >> 172.21.15.28:0/2060021623 conn(0x7f41e0079bb0 msgr2=0x7f41e0079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:00.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.444+0000 7f41e4d76700 1 -- 172.21.15.28:0/2060021623 shutdown_connections 2024-04-21T08:48:00.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.444+0000 7f41e4d76700 1 -- 172.21.15.28:0/2060021623 wait complete. 2024-04-21T08:48:00.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41e4d76700 1 Processor -- start 2024-04-21T08:48:00.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41e4d76700 1 -- start start 2024-04-21T08:48:00.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41e4d76700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41e0102ac0 0x7f41e0074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:00.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41e4d76700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e0103480 0x7f41e0075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:00.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41e4d76700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f41e0108a70 0x7f41e00757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:00.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41e4d76700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41e0112c80 con 0x7f41e0103480 2024-04-21T08:48:00.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41e4d76700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f41e0112b00 con 0x7f41e0102ac0 2024-04-21T08:48:00.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41e4d76700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f41e0112e00 con 0x7f41e0108a70 2024-04-21T08:48:00.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41ddd9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e0103480 0x7f41e0075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:00.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41ddd9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e0103480 0x7f41e0075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52496/0 (socket says 172.21.15.28:52496) 2024-04-21T08:48:00.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41ddd9b700 1 -- 172.21.15.28:0/4099190470 learned_addr learned my addr 172.21.15.28:0/4099190470 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:00.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41de59c700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41e0102ac0 0x7f41e0074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:00.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41ded9d700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f41e0108a70 0x7f41e00757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:00.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41ddd9b700 1 -- 172.21.15.28:0/4099190470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f41e0108a70 msgr2=0x7f41e00757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41ddd9b700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f41e0108a70 0x7f41e00757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41ddd9b700 1 -- 172.21.15.28:0/4099190470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41e0102ac0 msgr2=0x7f41e0074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41ddd9b700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41e0102ac0 0x7f41e0074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41ddd9b700 1 -- 172.21.15.28:0/4099190470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e01a7620 con 0x7f41e0103480 2024-04-21T08:48:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41ded9d700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f41e0108a70 0x7f41e00757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.445+0000 7f41de59c700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41e0102ac0 0x7f41e0074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.446+0000 7f41ddd9b700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e0103480 0x7f41e0075280 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f41d40057f0 tx=0x7f41d4003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.446+0000 7f41cf7fe700 1 -- 172.21.15.28:0/4099190470 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4027070 con 0x7f41e0103480 2024-04-21T08:48:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.446+0000 7f41e4d76700 1 -- 172.21.15.28:0/4099190470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41e01a78b0 con 0x7f41e0103480 2024-04-21T08:48:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.446+0000 7f41cf7fe700 1 -- 172.21.15.28:0/4099190470 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41d4004020 con 0x7f41e0103480 2024-04-21T08:48:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.446+0000 7f41cf7fe700 1 -- 172.21.15.28:0/4099190470 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4022d60 con 0x7f41e0103480 2024-04-21T08:48:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.446+0000 7f41e4d76700 1 -- 172.21.15.28:0/4099190470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41e01a7d90 con 0x7f41e0103480 2024-04-21T08:48:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.447+0000 7f41e4d76700 1 -- 172.21.15.28:0/4099190470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41e0069730 con 0x7f41e0103480 2024-04-21T08:48:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.448+0000 7f41cf7fe700 1 -- 172.21.15.28:0/4099190470 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f41d402a070 con 0x7f41e0103480 2024-04-21T08:48:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.448+0000 7f41cf7fe700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f41c80a3c40 0x7f41c80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.449+0000 7f41de59c700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f41c80a3c40 0x7f41c80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.449+0000 7f41cf7fe700 1 -- 172.21.15.28:0/4099190470 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f41d40d0e30 con 0x7f41e0103480 2024-04-21T08:48:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.449+0000 7f41de59c700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f41c80a3c40 0x7f41c80a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009de0 tx=0x7f41d0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.453+0000 7f41cf7fe700 1 -- 172.21.15.28:0/4099190470 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f41d405eff0 con 0x7f41e0103480 2024-04-21T08:48:00.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.664+0000 7f41e4d76700 1 -- 172.21.15.28:0/4099190470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41e002df80 con 0x7f41e0103480 2024-04-21T08:48:00.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.665+0000 7f41cf7fe700 1 -- 172.21.15.28:0/4099190470 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18429 (secure 0 0 0) 0x7f41d4013070 con 0x7f41e0103480 2024-04-21T08:48:00.665 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:00.666 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:47:58.562865+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:48:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.668+0000 7f41e4d76700 1 -- 172.21.15.28:0/4099190470 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f41c80a3c40 msgr2=0x7f41c80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.668+0000 7f41e4d76700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f41c80a3c40 0x7f41c80a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009de0 tx=0x7f41d0009190 comp rx=0 tx=0).stop 2024-04-21T08:48:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.668+0000 7f41e4d76700 1 -- 172.21.15.28:0/4099190470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e0103480 msgr2=0x7f41e0075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.668+0000 7f41e4d76700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e0103480 0x7f41e0075280 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f41d40057f0 tx=0x7f41d4003680 comp rx=0 tx=0).stop 2024-04-21T08:48:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.668+0000 7f41e4d76700 1 -- 172.21.15.28:0/4099190470 shutdown_connections 2024-04-21T08:48:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.668+0000 7f41e4d76700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f41e0108a70 0x7f41e00757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.668+0000 7f41e4d76700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41e0102ac0 0x7f41e0074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.668+0000 7f41e4d76700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f41c80a3c40 0x7f41c80a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.668+0000 7f41e4d76700 1 --2- 172.21.15.28:0/4099190470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41e0103480 0x7f41e0075280 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.668+0000 7f41e4d76700 1 -- 172.21.15.28:0/4099190470 >> 172.21.15.28:0/4099190470 conn(0x7f41e0079bb0 msgr2=0x7f41e0101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.668+0000 7f41e4d76700 1 -- 172.21.15.28:0/4099190470 shutdown_connections 2024-04-21T08:48:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:00.668+0000 7f41e4d76700 1 -- 172.21.15.28:0/4099190470 wait complete. 2024-04-21T08:48:00.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:00 smithi152 ceph-mon[31401]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:00.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:00 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:00.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:00 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:00.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:00 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:00.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:00 smithi028 ceph-mon[28003]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:00.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:00 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:00.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:00 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:00.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:00 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:00.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:00 smithi171 ceph-mon[31358]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:00.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:00 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:00.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:00 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:00.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:00 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:01.496 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T08:48:01.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-21T08:48:01.753 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:48:01.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:01 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/4099190470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:01.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:01 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/4099190470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:01.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:01 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/4099190470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:02.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:02 smithi152 ceph-mon[31401]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:02.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:02 smithi028 ceph-mon[28003]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:02.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:02 smithi171 ceph-mon[31358]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.588+0000 7fd240d90700 1 -- 172.21.15.28:0/2828331848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd23c108a50 msgr2=0x7fd23c108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.588+0000 7fd240d90700 1 --2- 172.21.15.28:0/2828331848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd23c108a50 0x7fd23c108e30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd22c009990 tx=0x7fd22c009ca0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.588+0000 7fd240d90700 1 -- 172.21.15.28:0/2828331848 shutdown_connections 2024-04-21T08:48:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.588+0000 7fd240d90700 1 --2- 172.21.15.28:0/2828331848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd23c108a50 0x7fd23c108e30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.588+0000 7fd240d90700 1 --2- 172.21.15.28:0/2828331848 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd23c103450 0x7fd23c10f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.588+0000 7fd240d90700 1 --2- 172.21.15.28:0/2828331848 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd23c102a90 0x7fd23c102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.588+0000 7fd240d90700 1 -- 172.21.15.28:0/2828331848 >> 172.21.15.28:0/2828331848 conn(0x7fd23c079ba0 msgr2=0x7fd23c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.588+0000 7fd240d90700 1 -- 172.21.15.28:0/2828331848 shutdown_connections 2024-04-21T08:48:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.588+0000 7fd240d90700 1 -- 172.21.15.28:0/2828331848 wait complete. 2024-04-21T08:48:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.589+0000 7fd240d90700 1 Processor -- start 2024-04-21T08:48:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.589+0000 7fd240d90700 1 -- start start 2024-04-21T08:48:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.589+0000 7fd240d90700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd23c102a90 0x7fd23c19b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.589+0000 7fd240d90700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd23c103450 0x7fd23c19b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.589+0000 7fd240d90700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd23c108a50 0x7fd23c19fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.589+0000 7fd240d90700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c112ce0 con 0x7fd23c102a90 2024-04-21T08:48:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.589+0000 7fd240d90700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c112b60 con 0x7fd23c103450 2024-04-21T08:48:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.589+0000 7fd240d90700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd23c112e60 con 0x7fd23c108a50 2024-04-21T08:48:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.589+0000 7fd23a59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd23c102a90 0x7fd23c19b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.589+0000 7fd23a59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd23c102a90 0x7fd23c19b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52520/0 (socket says 172.21.15.28:52520) 2024-04-21T08:48:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.589+0000 7fd23a59c700 1 -- 172.21.15.28:0/4040871483 learned_addr learned my addr 172.21.15.28:0/4040871483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.589+0000 7fd23a59c700 1 -- 172.21.15.28:0/4040871483 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd23c108a50 msgr2=0x7fd23c19fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.589+0000 7fd23ad9d700 1 --2- 172.21.15.28:0/4040871483 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd23c108a50 0x7fd23c19fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.590+0000 7fd239d9b700 1 --2- 172.21.15.28:0/4040871483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd23c103450 0x7fd23c19b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.590+0000 7fd23a59c700 1 --2- 172.21.15.28:0/4040871483 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd23c108a50 0x7fd23c19fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.590+0000 7fd23a59c700 1 -- 172.21.15.28:0/4040871483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd23c103450 msgr2=0x7fd23c19b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.590+0000 7fd23a59c700 1 --2- 172.21.15.28:0/4040871483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd23c103450 0x7fd23c19b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.590+0000 7fd23a59c700 1 -- 172.21.15.28:0/4040871483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd23c1a04a0 con 0x7fd23c102a90 2024-04-21T08:48:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.590+0000 7fd239d9b700 1 --2- 172.21.15.28:0/4040871483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd23c103450 0x7fd23c19b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.590+0000 7fd23a59c700 1 --2- 172.21.15.28:0/4040871483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd23c102a90 0x7fd23c19b490 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd22c000c00 tx=0x7fd22c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.590+0000 7fd22b7fe700 1 -- 172.21.15.28:0/4040871483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22c027070 con 0x7fd23c102a90 2024-04-21T08:48:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.590+0000 7fd22b7fe700 1 -- 172.21.15.28:0/4040871483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd22c0041d0 con 0x7fd23c102a90 2024-04-21T08:48:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.590+0000 7fd240d90700 1 -- 172.21.15.28:0/4040871483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd23c1a0730 con 0x7fd23c102a90 2024-04-21T08:48:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.590+0000 7fd22b7fe700 1 -- 172.21.15.28:0/4040871483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22c00f460 con 0x7fd23c102a90 2024-04-21T08:48:03.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.590+0000 7fd240d90700 1 -- 172.21.15.28:0/4040871483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd23c1a8010 con 0x7fd23c102a90 2024-04-21T08:48:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.591+0000 7fd240d90700 1 -- 172.21.15.28:0/4040871483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd23c069730 con 0x7fd23c102a90 2024-04-21T08:48:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.592+0000 7fd22b7fe700 1 -- 172.21.15.28:0/4040871483 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd22c02a070 con 0x7fd23c102a90 2024-04-21T08:48:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.593+0000 7fd22b7fe700 1 --2- 172.21.15.28:0/4040871483 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd2240a3c40 0x7fd2240a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.593+0000 7fd239d9b700 1 --2- 172.21.15.28:0/4040871483 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd2240a3c40 0x7fd2240a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.593+0000 7fd22b7fe700 1 -- 172.21.15.28:0/4040871483 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd22c0d13e0 con 0x7fd23c102a90 2024-04-21T08:48:03.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.594+0000 7fd239d9b700 1 --2- 172.21.15.28:0/4040871483 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd2240a3c40 0x7fd2240a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd23c07a430 tx=0x7fd230008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.597+0000 7fd22b7fe700 1 -- 172.21.15.28:0/4040871483 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd22c031070 con 0x7fd23c102a90 2024-04-21T08:48:03.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.805+0000 7fd240d90700 1 -- 172.21.15.28:0/4040871483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd23c02df80 con 0x7fd23c102a90 2024-04-21T08:48:03.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.806+0000 7fd22b7fe700 1 -- 172.21.15.28:0/4040871483 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fd22c0a4030 con 0x7fd23c102a90 2024-04-21T08:48:03.806 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T08:48:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.808+0000 7fd240d90700 1 -- 172.21.15.28:0/4040871483 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd2240a3c40 msgr2=0x7fd2240a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.808+0000 7fd240d90700 1 --2- 172.21.15.28:0/4040871483 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd2240a3c40 0x7fd2240a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd23c07a430 tx=0x7fd230008040 comp rx=0 tx=0).stop 2024-04-21T08:48:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.808+0000 7fd240d90700 1 -- 172.21.15.28:0/4040871483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd23c102a90 msgr2=0x7fd23c19b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.808+0000 7fd240d90700 1 --2- 172.21.15.28:0/4040871483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd23c102a90 0x7fd23c19b490 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd22c000c00 tx=0x7fd22c0047b0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.808+0000 7fd240d90700 1 -- 172.21.15.28:0/4040871483 shutdown_connections 2024-04-21T08:48:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.808+0000 7fd240d90700 1 --2- 172.21.15.28:0/4040871483 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd2240a3c40 0x7fd2240a6100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.808+0000 7fd240d90700 1 --2- 172.21.15.28:0/4040871483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd23c102a90 0x7fd23c19b490 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.808+0000 7fd240d90700 1 --2- 172.21.15.28:0/4040871483 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd23c108a50 0x7fd23c19fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.808+0000 7fd240d90700 1 --2- 172.21.15.28:0/4040871483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd23c103450 0x7fd23c19b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.808+0000 7fd240d90700 1 -- 172.21.15.28:0/4040871483 >> 172.21.15.28:0/4040871483 conn(0x7fd23c079ba0 msgr2=0x7fd23c10e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.809+0000 7fd240d90700 1 -- 172.21.15.28:0/4040871483 shutdown_connections 2024-04-21T08:48:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:03.809+0000 7fd240d90700 1 -- 172.21.15.28:0/4040871483 wait complete. 2024-04-21T08:48:04.725 INFO:tasks.cephadm:Adding mds.a on smithi028 2024-04-21T08:48:04.725 INFO:tasks.cephadm:Adding mds.d on smithi028 2024-04-21T08:48:04.725 INFO:tasks.cephadm:Adding mds.g on smithi028 2024-04-21T08:48:04.726 INFO:tasks.cephadm:Adding mds.j on smithi028 2024-04-21T08:48:04.726 INFO:tasks.cephadm:Adding mds.b on smithi152 2024-04-21T08:48:04.726 INFO:tasks.cephadm:Adding mds.e on smithi152 2024-04-21T08:48:04.726 INFO:tasks.cephadm:Adding mds.h on smithi152 2024-04-21T08:48:04.726 INFO:tasks.cephadm:Adding mds.k on smithi152 2024-04-21T08:48:04.726 INFO:tasks.cephadm:Adding mds.c on smithi171 2024-04-21T08:48:04.726 INFO:tasks.cephadm:Adding mds.f on smithi171 2024-04-21T08:48:04.726 INFO:tasks.cephadm:Adding mds.i on smithi171 2024-04-21T08:48:04.726 INFO:tasks.cephadm:Adding mds.l on smithi171 2024-04-21T08:48:04.726 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l' 2024-04-21T08:48:04.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:04 smithi152 ceph-mon[31401]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:04.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:04 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/4040871483' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:48:04.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:04 smithi028 ceph-mon[28003]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:04.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:04 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/4040871483' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:48:04.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:04 smithi171 ceph-mon[31358]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:04.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:04 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/4040871483' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:48:04.974 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:48:06.654 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.625+0000 7f2982868700 1 -- 172.21.15.171:0/684885680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f297c1035e0 msgr2=0x7f297c1059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:06.654 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.625+0000 7f2982868700 1 --2- 172.21.15.171:0/684885680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f297c1035e0 0x7f297c1059d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f296c009990 tx=0x7f296c009ca0 comp rx=0 tx=0).stop 2024-04-21T08:48:06.654 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.625+0000 7f2982868700 1 -- 172.21.15.171:0/684885680 shutdown_connections 2024-04-21T08:48:06.654 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.625+0000 7f2982868700 1 --2- 172.21.15.171:0/684885680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f297c108840 0x7f297c10ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:06.654 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.625+0000 7f2982868700 1 --2- 172.21.15.171:0/684885680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f297c1035e0 0x7f297c1059d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:06.654 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.625+0000 7f2982868700 1 --2- 172.21.15.171:0/684885680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f297c105f10 0x7f297c108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:06.655 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.625+0000 7f2982868700 1 -- 172.21.15.171:0/684885680 >> 172.21.15.171:0/684885680 conn(0x7f297c0fcf70 msgr2=0x7f297c0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:06.655 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.625+0000 7f2982868700 1 -- 172.21.15.171:0/684885680 shutdown_connections 2024-04-21T08:48:06.655 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.625+0000 7f2982868700 1 -- 172.21.15.171:0/684885680 wait complete. 2024-04-21T08:48:06.655 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.625+0000 7f2982868700 1 Processor -- start 2024-04-21T08:48:06.655 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.625+0000 7f2982868700 1 -- start start 2024-04-21T08:48:06.655 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f2982868700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f297c1035e0 0x7f297c102f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:06.655 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f2982868700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f297c105f10 0x7f297c101590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:06.655 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f2982868700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f297c108840 0x7f297c101ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:06.655 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f2982868700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f297c10e530 con 0x7f297c108840 2024-04-21T08:48:06.655 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f2982868700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f297c10e3b0 con 0x7f297c1035e0 2024-04-21T08:48:06.655 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f2982868700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f297c10e6b0 con 0x7f297c105f10 2024-04-21T08:48:06.655 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f297b7fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f297c105f10 0x7f297c101590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:06.655 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f297b7fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f297c105f10 0x7f297c101590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:41090/0 (socket says 172.21.15.171:41090) 2024-04-21T08:48:06.656 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f297b7fe700 1 -- 172.21.15.171:0/1318398563 learned_addr learned my addr 172.21.15.171:0/1318398563 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-04-21T08:48:06.656 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f2980e05700 1 --2- 172.21.15.171:0/1318398563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f297c108840 0x7f297c101ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:06.656 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f297bfff700 1 --2- 172.21.15.171:0/1318398563 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f297c1035e0 0x7f297c102f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:06.656 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f297b7fe700 1 -- 172.21.15.171:0/1318398563 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f297c1035e0 msgr2=0x7f297c102f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:06.656 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f297b7fe700 1 --2- 172.21.15.171:0/1318398563 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f297c1035e0 0x7f297c102f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:06.656 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f297b7fe700 1 -- 172.21.15.171:0/1318398563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f297c108840 msgr2=0x7f297c101ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:06.656 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f297b7fe700 1 --2- 172.21.15.171:0/1318398563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f297c108840 0x7f297c101ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:06.656 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f297b7fe700 1 -- 172.21.15.171:0/1318398563 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f297c102380 con 0x7f297c105f10 2024-04-21T08:48:06.656 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f297b7fe700 1 --2- 172.21.15.171:0/1318398563 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f297c105f10 0x7f297c101590 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f297000d950 tx=0x7f297000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:06.656 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f29797fa700 1 -- 172.21.15.171:0/1318398563 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2970016070 con 0x7f297c105f10 2024-04-21T08:48:06.656 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f29797fa700 1 -- 172.21.15.171:0/1318398563 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f297000fd40 con 0x7f297c105f10 2024-04-21T08:48:06.656 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f2982868700 1 -- 172.21.15.171:0/1318398563 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f297c1a2dc0 con 0x7f297c105f10 2024-04-21T08:48:06.656 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f29797fa700 1 -- 172.21.15.171:0/1318398563 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297000e3b0 con 0x7f297c105f10 2024-04-21T08:48:06.656 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.626+0000 7f2982868700 1 -- 172.21.15.171:0/1318398563 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f297c1a3270 con 0x7f297c105f10 2024-04-21T08:48:06.657 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.627+0000 7f2982868700 1 -- 172.21.15.171:0/1318398563 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f297c02df80 con 0x7f297c105f10 2024-04-21T08:48:06.657 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.629+0000 7f29797fa700 1 -- 172.21.15.171:0/1318398563 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2970019070 con 0x7f297c105f10 2024-04-21T08:48:06.657 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.629+0000 7f29797fa700 1 --2- 172.21.15.171:0/1318398563 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f29640a3cc0 0x7f29640a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:06.657 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.629+0000 7f29797fa700 1 -- 172.21.15.171:0/1318398563 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f29700cf900 con 0x7f297c105f10 2024-04-21T08:48:06.657 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.629+0000 7f297bfff700 1 --2- 172.21.15.171:0/1318398563 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f29640a3cc0 0x7f29640a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:06.657 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.630+0000 7f297bfff700 1 --2- 172.21.15.171:0/1318398563 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f29640a3cc0 0x7f29640a6180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f296c00b5c0 tx=0x7f296c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:06.657 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.633+0000 7f29797fa700 1 -- 172.21.15.171:0/1318398563 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f297009c780 con 0x7f297c105f10 2024-04-21T08:48:06.841 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.842+0000 7f2982868700 1 -- 172.21.15.171:0/1318398563 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l", "target": ["mon-mgr", ""]}) -- 0x7f297c062f80 con 0x7f29640a3cc0 2024-04-21T08:48:06.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:06 smithi171 ceph-mon[31358]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:06.849 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.850+0000 7f29797fa700 1 -- 172.21.15.171:0/1318398563 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f297c062f80 con 0x7f29640a3cc0 2024-04-21T08:48:06.849 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled mds.all update... 2024-04-21T08:48:06.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.852+0000 7f2982868700 1 -- 172.21.15.171:0/1318398563 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f29640a3cc0 msgr2=0x7f29640a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:06.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.852+0000 7f2982868700 1 --2- 172.21.15.171:0/1318398563 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f29640a3cc0 0x7f29640a6180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f296c00b5c0 tx=0x7f296c005fb0 comp rx=0 tx=0).stop 2024-04-21T08:48:06.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.852+0000 7f2982868700 1 -- 172.21.15.171:0/1318398563 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f297c105f10 msgr2=0x7f297c101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:06.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.852+0000 7f2982868700 1 --2- 172.21.15.171:0/1318398563 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f297c105f10 0x7f297c101590 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f297000d950 tx=0x7f297000dc60 comp rx=0 tx=0).stop 2024-04-21T08:48:06.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.852+0000 7f2982868700 1 -- 172.21.15.171:0/1318398563 shutdown_connections 2024-04-21T08:48:06.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.852+0000 7f2982868700 1 --2- 172.21.15.171:0/1318398563 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f29640a3cc0 0x7f29640a6180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:06.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.852+0000 7f2982868700 1 --2- 172.21.15.171:0/1318398563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f297c108840 0x7f297c101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:06.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.852+0000 7f2982868700 1 --2- 172.21.15.171:0/1318398563 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f297c1035e0 0x7f297c102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:06.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.852+0000 7f2982868700 1 --2- 172.21.15.171:0/1318398563 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f297c105f10 0x7f297c101590 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:06.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.852+0000 7f2982868700 1 -- 172.21.15.171:0/1318398563 >> 172.21.15.171:0/1318398563 conn(0x7f297c0fcf70 msgr2=0x7f297c104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:06.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.852+0000 7f2982868700 1 -- 172.21.15.171:0/1318398563 shutdown_connections 2024-04-21T08:48:06.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-04-21T08:48:06.852+0000 7f2982868700 1 -- 172.21.15.171:0/1318398563 wait complete. 2024-04-21T08:48:06.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:06 smithi152 ceph-mon[31401]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:06.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:06 smithi028 ceph-mon[28003]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:07.510 DEBUG:teuthology.orchestra.run.smithi028:mds.a> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.a.service 2024-04-21T08:48:07.513 DEBUG:teuthology.orchestra.run.smithi028:mds.d> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.d.service 2024-04-21T08:48:07.556 DEBUG:teuthology.orchestra.run.smithi028:mds.g> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.g.service 2024-04-21T08:48:07.599 DEBUG:teuthology.orchestra.run.smithi028:mds.j> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.j.service 2024-04-21T08:48:07.642 DEBUG:teuthology.orchestra.run.smithi152:mds.b> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.b.service 2024-04-21T08:48:07.645 DEBUG:teuthology.orchestra.run.smithi152:mds.e> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.e.service 2024-04-21T08:48:07.688 DEBUG:teuthology.orchestra.run.smithi152:mds.h> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.h.service 2024-04-21T08:48:07.731 DEBUG:teuthology.orchestra.run.smithi152:mds.k> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.k.service 2024-04-21T08:48:07.775 DEBUG:teuthology.orchestra.run.smithi171:mds.c> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.c.service 2024-04-21T08:48:07.778 DEBUG:teuthology.orchestra.run.smithi171:mds.f> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.f.service 2024-04-21T08:48:07.821 DEBUG:teuthology.orchestra.run.smithi171:mds.i> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.i.service 2024-04-21T08:48:07.865 DEBUG:teuthology.orchestra.run.smithi171:mds.l> sudo journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.l.service 2024-04-21T08:48:07.907 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-21T08:48:07.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:48:08.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.018+0000 7fabda648640 1 -- 172.21.15.28:0/565238095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabd4100e70 msgr2=0x7fabd4101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.018+0000 7fabda648640 1 --2- 172.21.15.28:0/565238095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabd4100e70 0x7fabd4101290 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fabbc009900 tx=0x7fabbc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.018+0000 7fabda648640 1 -- 172.21.15.28:0/565238095 shutdown_connections 2024-04-21T08:48:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.018+0000 7fabda648640 1 --2- 172.21.15.28:0/565238095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fabd4102a10 0x7fabd4109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.018+0000 7fabda648640 1 --2- 172.21.15.28:0/565238095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fabd4102050 0x7fabd41024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.018+0000 7fabda648640 1 --2- 172.21.15.28:0/565238095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabd4100e70 0x7fabd4101290 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.018+0000 7fabda648640 1 -- 172.21.15.28:0/565238095 >> 172.21.15.28:0/565238095 conn(0x7fabd40fc470 msgr2=0x7fabd40fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.018+0000 7fabda648640 1 -- 172.21.15.28:0/565238095 shutdown_connections 2024-04-21T08:48:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.018+0000 7fabda648640 1 -- 172.21.15.28:0/565238095 wait complete. 2024-04-21T08:48:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.018+0000 7fabda648640 1 Processor -- start 2024-04-21T08:48:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.018+0000 7fabda648640 1 -- start start 2024-04-21T08:48:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabda648640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fabd4100e70 0x7fabd419a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabda648640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fabd4102050 0x7fabd419a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabda648640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabd4102a10 0x7fabd41a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabda648640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fabd410fd30 con 0x7fabd4102a10 2024-04-21T08:48:08.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabda648640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fabd410fbb0 con 0x7fabd4102050 2024-04-21T08:48:08.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabda648640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fabd410feb0 con 0x7fabd4100e70 2024-04-21T08:48:08.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabd8be5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabd4102a10 0x7fabd41a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabd8be5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabd4102a10 0x7fabd41a18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59216/0 (socket says 172.21.15.28:59216) 2024-04-21T08:48:08.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabd8be5640 1 -- 172.21.15.28:0/763967022 learned_addr learned my addr 172.21.15.28:0/763967022 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:08.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabd37fe640 1 --2- 172.21.15.28:0/763967022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fabd4102050 0x7fabd419a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabd3fff640 1 --2- 172.21.15.28:0/763967022 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fabd4100e70 0x7fabd419a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabd8be5640 1 -- 172.21.15.28:0/763967022 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fabd4100e70 msgr2=0x7fabd419a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabd8be5640 1 --2- 172.21.15.28:0/763967022 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fabd4100e70 0x7fabd419a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabd8be5640 1 -- 172.21.15.28:0/763967022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fabd4102050 msgr2=0x7fabd419a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabd8be5640 1 --2- 172.21.15.28:0/763967022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fabd4102050 0x7fabd419a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabd8be5640 1 -- 172.21.15.28:0/763967022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabd41a1f60 con 0x7fabd4102a10 2024-04-21T08:48:08.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabd8be5640 1 --2- 172.21.15.28:0/763967022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabd4102a10 0x7fabd41a18f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00cce0 tx=0x7fabcc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:08.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabd17fa640 1 -- 172.21.15.28:0/763967022 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc00f450 con 0x7fabd4102a10 2024-04-21T08:48:08.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabda648640 1 -- 172.21.15.28:0/763967022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabd41a2250 con 0x7fabd4102a10 2024-04-21T08:48:08.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabd17fa640 1 -- 172.21.15.28:0/763967022 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabcc00faa0 con 0x7fabd4102a10 2024-04-21T08:48:08.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabd17fa640 1 -- 172.21.15.28:0/763967022 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc018b70 con 0x7fabd4102a10 2024-04-21T08:48:08.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.019+0000 7fabda648640 1 -- 172.21.15.28:0/763967022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabd41a27a0 con 0x7fabd4102a10 2024-04-21T08:48:08.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.020+0000 7fabda648640 1 -- 172.21.15.28:0/763967022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabd41081b0 con 0x7fabd4102a10 2024-04-21T08:48:08.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.022+0000 7fabd17fa640 1 -- 172.21.15.28:0/763967022 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fabcc018d10 con 0x7fabd4102a10 2024-04-21T08:48:08.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.022+0000 7fabd17fa640 1 --2- 172.21.15.28:0/763967022 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fabc00a3b90 0x7fabc00a6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.022+0000 7fabd17fa640 1 -- 172.21.15.28:0/763967022 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fabcc0d03a0 con 0x7fabd4102a10 2024-04-21T08:48:08.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.022+0000 7fabd3fff640 1 --2- 172.21.15.28:0/763967022 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fabc00a3b90 0x7fabc00a6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.023+0000 7fabd3fff640 1 --2- 172.21.15.28:0/763967022 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fabc00a3b90 0x7fabc00a6050 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fabbc02c240 tx=0x7fabbc02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:08.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.026+0000 7fabd17fa640 1 -- 172.21.15.28:0/763967022 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fabcc014030 con 0x7fabd4102a10 2024-04-21T08:48:08.054 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:07 smithi171 ceph-mon[31358]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:48:08.054 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:07 smithi171 ceph-mon[31358]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l;count:12 2024-04-21T08:48:08.054 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:08.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:08.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:08.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:08.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:08.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:08.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:08.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:07 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:08.055 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:07 smithi171 ceph-mon[31358]: Deploying daemon mds.l on smithi171 2024-04-21T08:48:08.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:07 smithi152 ceph-mon[31401]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:48:08.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:07 smithi152 ceph-mon[31401]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l;count:12 2024-04-21T08:48:08.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:08.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:08.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:08.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:08.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:08.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:08.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:08.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:07 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:08.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:07 smithi152 ceph-mon[31401]: Deploying daemon mds.l on smithi171 2024-04-21T08:48:08.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:07 smithi028 ceph-mon[28003]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:48:08.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:07 smithi028 ceph-mon[28003]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l;count:12 2024-04-21T08:48:08.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:08.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:08.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:08.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:08.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:08.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:08.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:08.108 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:07 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:08.108 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:07 smithi028 ceph-mon[28003]: Deploying daemon mds.l on smithi171 2024-04-21T08:48:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.214+0000 7fabda648640 1 -- 172.21.15.28:0/763967022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fabd4062b50 con 0x7fabd4102a10 2024-04-21T08:48:08.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.215+0000 7fabd17fa640 1 -- 172.21.15.28:0/763967022 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18429 (secure 0 0 0) 0x7fabcc09d2b0 con 0x7fabd4102a10 2024-04-21T08:48:08.216 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:08.216 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:47:58.562865+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:48:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.217+0000 7fabda648640 1 -- 172.21.15.28:0/763967022 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fabc00a3b90 msgr2=0x7fabc00a6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.217+0000 7fabda648640 1 --2- 172.21.15.28:0/763967022 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fabc00a3b90 0x7fabc00a6050 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fabbc02c240 tx=0x7fabbc02c7e0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.217+0000 7fabda648640 1 -- 172.21.15.28:0/763967022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabd4102a10 msgr2=0x7fabd41a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.217+0000 7fabda648640 1 --2- 172.21.15.28:0/763967022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabd4102a10 0x7fabd41a18f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00cce0 tx=0x7fabcc00a620 comp rx=0 tx=0).stop 2024-04-21T08:48:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.217+0000 7fabda648640 1 -- 172.21.15.28:0/763967022 shutdown_connections 2024-04-21T08:48:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.217+0000 7fabda648640 1 --2- 172.21.15.28:0/763967022 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fabc00a3b90 0x7fabc00a6050 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.217+0000 7fabda648640 1 --2- 172.21.15.28:0/763967022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabd4102a10 0x7fabd41a18f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.217+0000 7fabda648640 1 --2- 172.21.15.28:0/763967022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fabd4102050 0x7fabd419a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.217+0000 7fabda648640 1 --2- 172.21.15.28:0/763967022 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fabd4100e70 0x7fabd419a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.217+0000 7fabda648640 1 -- 172.21.15.28:0/763967022 >> 172.21.15.28:0/763967022 conn(0x7fabd40fc470 msgr2=0x7fabd4105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.217+0000 7fabda648640 1 -- 172.21.15.28:0/763967022 shutdown_connections 2024-04-21T08:48:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.217+0000 7fabda648640 1 -- 172.21.15.28:0/763967022 wait complete. 2024-04-21T08:48:08.229 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T08:48:08.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:48:08.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.333+0000 7f5b783f4640 1 -- 172.21.15.28:0/3543536783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b70100e60 msgr2=0x7f5b70101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.333+0000 7f5b783f4640 1 --2- 172.21.15.28:0/3543536783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b70100e60 0x7f5b70101280 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5b68009900 tx=0x7f5b6802bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:08.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.333+0000 7f5b783f4640 1 -- 172.21.15.28:0/3543536783 shutdown_connections 2024-04-21T08:48:08.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.333+0000 7f5b783f4640 1 --2- 172.21.15.28:0/3543536783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b70102a00 0x7f5b70109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.333+0000 7f5b783f4640 1 --2- 172.21.15.28:0/3543536783 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5b70102040 0x7f5b701024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.333+0000 7f5b783f4640 1 --2- 172.21.15.28:0/3543536783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b70100e60 0x7f5b70101280 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.333+0000 7f5b783f4640 1 -- 172.21.15.28:0/3543536783 >> 172.21.15.28:0/3543536783 conn(0x7f5b700fc440 msgr2=0x7f5b700fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:08.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.334+0000 7f5b783f4640 1 -- 172.21.15.28:0/3543536783 shutdown_connections 2024-04-21T08:48:08.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.334+0000 7f5b783f4640 1 -- 172.21.15.28:0/3543536783 wait complete. 2024-04-21T08:48:08.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.334+0000 7f5b783f4640 1 Processor -- start 2024-04-21T08:48:08.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.334+0000 7f5b783f4640 1 -- start start 2024-04-21T08:48:08.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.334+0000 7f5b783f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b70100e60 0x7f5b70198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.334+0000 7f5b783f4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5b70102040 0x7f5b70198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.334+0000 7f5b783f4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b70102a00 0x7f5b7019f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.334+0000 7f5b783f4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7010daf0 con 0x7f5b70100e60 2024-04-21T08:48:08.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.334+0000 7f5b783f4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7010d970 con 0x7f5b70102a00 2024-04-21T08:48:08.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.334+0000 7f5b783f4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7010dc70 con 0x7f5b70102040 2024-04-21T08:48:08.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.334+0000 7f5b76190640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b70100e60 0x7f5b70198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.334+0000 7f5b76190640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b70100e60 0x7f5b70198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59232/0 (socket says 172.21.15.28:59232) 2024-04-21T08:48:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.334+0000 7f5b76190640 1 -- 172.21.15.28:0/1205603068 learned_addr learned my addr 172.21.15.28:0/1205603068 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.334+0000 7f5b7598f640 1 --2- 172.21.15.28:0/1205603068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5b70102040 0x7f5b70198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.334+0000 7f5b76991640 1 --2- 172.21.15.28:0/1205603068 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b70102a00 0x7f5b7019f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.335+0000 7f5b76190640 1 -- 172.21.15.28:0/1205603068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5b70102040 msgr2=0x7f5b70198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.335+0000 7f5b76190640 1 --2- 172.21.15.28:0/1205603068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5b70102040 0x7f5b70198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.335+0000 7f5b76190640 1 -- 172.21.15.28:0/1205603068 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b70102a00 msgr2=0x7f5b7019f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.335+0000 7f5b76190640 1 --2- 172.21.15.28:0/1205603068 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b70102a00 0x7f5b7019f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.335+0000 7f5b76190640 1 -- 172.21.15.28:0/1205603068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b7019fe20 con 0x7f5b70100e60 2024-04-21T08:48:08.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.335+0000 7f5b76991640 1 --2- 172.21.15.28:0/1205603068 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b70102a00 0x7f5b7019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:08.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.335+0000 7f5b76190640 1 --2- 172.21.15.28:0/1205603068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b70100e60 0x7f5b70198200 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5b68000c00 tx=0x7f5b68005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:08.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.335+0000 7f5b6f7fe640 1 -- 172.21.15.28:0/1205603068 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b68043020 con 0x7f5b70100e60 2024-04-21T08:48:08.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.335+0000 7f5b6f7fe640 1 -- 172.21.15.28:0/1205603068 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b68004350 con 0x7f5b70100e60 2024-04-21T08:48:08.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.335+0000 7f5b783f4640 1 -- 172.21.15.28:0/1205603068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b701a00b0 con 0x7f5b70100e60 2024-04-21T08:48:08.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.335+0000 7f5b6f7fe640 1 -- 172.21.15.28:0/1205603068 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b68003b80 con 0x7f5b70100e60 2024-04-21T08:48:08.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.335+0000 7f5b783f4640 1 -- 172.21.15.28:0/1205603068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b701a0600 con 0x7f5b70100e60 2024-04-21T08:48:08.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.336+0000 7f5b783f4640 1 -- 172.21.15.28:0/1205603068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b701082f0 con 0x7f5b70100e60 2024-04-21T08:48:08.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.337+0000 7f5b6f7fe640 1 -- 172.21.15.28:0/1205603068 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5b68046020 con 0x7f5b70100e60 2024-04-21T08:48:08.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.338+0000 7f5b6f7fe640 1 --2- 172.21.15.28:0/1205603068 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f5b580a3bb0 0x7f5b580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.338+0000 7f5b7598f640 1 --2- 172.21.15.28:0/1205603068 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f5b580a3bb0 0x7f5b580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.338+0000 7f5b6f7fe640 1 -- 172.21.15.28:0/1205603068 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5b680f2130 con 0x7f5b70100e60 2024-04-21T08:48:08.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.338+0000 7f5b7598f640 1 --2- 172.21.15.28:0/1205603068 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f5b580a3bb0 0x7f5b580a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5b70199710 tx=0x7f5b64005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:08.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.342+0000 7f5b6f7fe640 1 -- 172.21.15.28:0/1205603068 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b6804d020 con 0x7f5b70100e60 2024-04-21T08:48:08.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.529+0000 7f5b783f4640 1 -- 172.21.15.28:0/1205603068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5b70062b50 con 0x7f5b70100e60 2024-04-21T08:48:08.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.530+0000 7f5b6f7fe640 1 -- 172.21.15.28:0/1205603068 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f5b680befb0 con 0x7f5b70100e60 2024-04-21T08:48:08.530 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T08:48:08.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.531+0000 7f5b783f4640 1 -- 172.21.15.28:0/1205603068 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f5b580a3bb0 msgr2=0x7f5b580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.531+0000 7f5b783f4640 1 --2- 172.21.15.28:0/1205603068 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f5b580a3bb0 0x7f5b580a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5b70199710 tx=0x7f5b64005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:08.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.531+0000 7f5b783f4640 1 -- 172.21.15.28:0/1205603068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b70100e60 msgr2=0x7f5b70198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.531+0000 7f5b783f4640 1 --2- 172.21.15.28:0/1205603068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b70100e60 0x7f5b70198200 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5b68000c00 tx=0x7f5b68005dc0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.532+0000 7f5b783f4640 1 -- 172.21.15.28:0/1205603068 shutdown_connections 2024-04-21T08:48:08.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.532+0000 7f5b783f4640 1 --2- 172.21.15.28:0/1205603068 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b70102a00 0x7f5b7019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.532+0000 7f5b783f4640 1 --2- 172.21.15.28:0/1205603068 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5b70102040 0x7f5b70198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.532+0000 7f5b783f4640 1 --2- 172.21.15.28:0/1205603068 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f5b580a3bb0 0x7f5b580a6070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.532+0000 7f5b783f4640 1 --2- 172.21.15.28:0/1205603068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b70100e60 0x7f5b70198200 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.532+0000 7f5b783f4640 1 -- 172.21.15.28:0/1205603068 >> 172.21.15.28:0/1205603068 conn(0x7f5b700fc440 msgr2=0x7f5b701054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:08.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.532+0000 7f5b783f4640 1 -- 172.21.15.28:0/1205603068 shutdown_connections 2024-04-21T08:48:08.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.532+0000 7f5b783f4640 1 -- 172.21.15.28:0/1205603068 wait complete. 2024-04-21T08:48:08.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.644+0000 7f3e516c1640 1 -- 172.21.15.28:0/1689371796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c102a10 msgr2=0x7f3e4c109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.644+0000 7f3e516c1640 1 --2- 172.21.15.28:0/1689371796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c102a10 0x7f3e4c109330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009ca0 tx=0x7f3e4402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.644+0000 7f3e516c1640 1 -- 172.21.15.28:0/1689371796 shutdown_connections 2024-04-21T08:48:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.644+0000 7f3e516c1640 1 --2- 172.21.15.28:0/1689371796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c102a10 0x7f3e4c109330 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.644+0000 7f3e516c1640 1 --2- 172.21.15.28:0/1689371796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e4c102050 0x7f3e4c1024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.644+0000 7f3e516c1640 1 --2- 172.21.15.28:0/1689371796 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e4c100e70 0x7f3e4c101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.644+0000 7f3e516c1640 1 -- 172.21.15.28:0/1689371796 >> 172.21.15.28:0/1689371796 conn(0x7f3e4c0fc470 msgr2=0x7f3e4c0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.644+0000 7f3e516c1640 1 -- 172.21.15.28:0/1689371796 shutdown_connections 2024-04-21T08:48:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.644+0000 7f3e516c1640 1 -- 172.21.15.28:0/1689371796 wait complete. 2024-04-21T08:48:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.645+0000 7f3e516c1640 1 Processor -- start 2024-04-21T08:48:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.645+0000 7f3e516c1640 1 -- start start 2024-04-21T08:48:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.645+0000 7f3e516c1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e4c100e70 0x7f3e4c198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.645+0000 7f3e516c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c102050 0x7f3e4c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.645+0000 7f3e516c1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e4c102a10 0x7f3e4c19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.645+0000 7f3e516c1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c10db40 con 0x7f3e4c102050 2024-04-21T08:48:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.645+0000 7f3e516c1640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c10d9c0 con 0x7f3e4c102a10 2024-04-21T08:48:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.645+0000 7f3e516c1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4c10dcc0 con 0x7f3e4c100e70 2024-04-21T08:48:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.645+0000 7f3e4a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c102050 0x7f3e4c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.645+0000 7f3e4a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c102050 0x7f3e4c198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59250/0 (socket says 172.21.15.28:59250) 2024-04-21T08:48:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.645+0000 7f3e4a7fc640 1 -- 172.21.15.28:0/4058104392 learned_addr learned my addr 172.21.15.28:0/4058104392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.645+0000 7f3e4affd640 1 --2- 172.21.15.28:0/4058104392 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e4c100e70 0x7f3e4c198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.645+0000 7f3e4b7fe640 1 --2- 172.21.15.28:0/4058104392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e4c102a10 0x7f3e4c19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.646+0000 7f3e4a7fc640 1 -- 172.21.15.28:0/4058104392 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e4c100e70 msgr2=0x7f3e4c198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.646+0000 7f3e4a7fc640 1 --2- 172.21.15.28:0/4058104392 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e4c100e70 0x7f3e4c198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.646+0000 7f3e4a7fc640 1 -- 172.21.15.28:0/4058104392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e4c102a10 msgr2=0x7f3e4c19f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.646+0000 7f3e4a7fc640 1 --2- 172.21.15.28:0/4058104392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e4c102a10 0x7f3e4c19f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.646+0000 7f3e4a7fc640 1 -- 172.21.15.28:0/4058104392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4c19fd50 con 0x7f3e4c102050 2024-04-21T08:48:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.646+0000 7f3e4affd640 1 --2- 172.21.15.28:0/4058104392 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e4c100e70 0x7f3e4c198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.646+0000 7f3e4a7fc640 1 --2- 172.21.15.28:0/4058104392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c102050 0x7f3e4c198670 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000d950 tx=0x7f3e4000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.646+0000 7f3e33fff640 1 -- 172.21.15.28:0/4058104392 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40017070 con 0x7f3e4c102050 2024-04-21T08:48:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.646+0000 7f3e33fff640 1 -- 172.21.15.28:0/4058104392 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e40009c30 con 0x7f3e4c102050 2024-04-21T08:48:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.646+0000 7f3e516c1640 1 -- 172.21.15.28:0/4058104392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e4c19ffe0 con 0x7f3e4c102050 2024-04-21T08:48:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.646+0000 7f3e33fff640 1 -- 172.21.15.28:0/4058104392 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4000f620 con 0x7f3e4c102050 2024-04-21T08:48:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.646+0000 7f3e516c1640 1 -- 172.21.15.28:0/4058104392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e4c1a0500 con 0x7f3e4c102050 2024-04-21T08:48:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.647+0000 7f3e516c1640 1 -- 172.21.15.28:0/4058104392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e4c069230 con 0x7f3e4c102050 2024-04-21T08:48:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.648+0000 7f3e33fff640 1 -- 172.21.15.28:0/4058104392 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3e40005030 con 0x7f3e4c102050 2024-04-21T08:48:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.649+0000 7f3e33fff640 1 --2- 172.21.15.28:0/4058104392 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3e340a3bb0 0x7f3e340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.649+0000 7f3e4affd640 1 --2- 172.21.15.28:0/4058104392 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3e340a3bb0 0x7f3e340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.649+0000 7f3e33fff640 1 -- 172.21.15.28:0/4058104392 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3e400cfab0 con 0x7f3e4c102050 2024-04-21T08:48:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.650+0000 7f3e4affd640 1 --2- 172.21.15.28:0/4058104392 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3e340a3bb0 0x7f3e340a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c101bc0 tx=0x7f3e3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:08.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.653+0000 7f3e33fff640 1 -- 172.21.15.28:0/4058104392 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e40009de0 con 0x7f3e4c102050 2024-04-21T08:48:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.883+0000 7f3e516c1640 1 -- 172.21.15.28:0/4058104392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e4c108260 con 0x7f3e4c102050 2024-04-21T08:48:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.884+0000 7f3e33fff640 1 -- 172.21.15.28:0/4058104392 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f3e4c108260 con 0x7f3e4c102050 2024-04-21T08:48:08.884 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:08.885 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-21T08:48:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 1 2024-04-21T08:48:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.886+0000 7f3e516c1640 1 -- 172.21.15.28:0/4058104392 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3e340a3bb0 msgr2=0x7f3e340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.886+0000 7f3e516c1640 1 --2- 172.21.15.28:0/4058104392 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3e340a3bb0 0x7f3e340a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c101bc0 tx=0x7f3e3c005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.886+0000 7f3e516c1640 1 -- 172.21.15.28:0/4058104392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c102050 msgr2=0x7f3e4c198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.886+0000 7f3e516c1640 1 --2- 172.21.15.28:0/4058104392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c102050 0x7f3e4c198670 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000d950 tx=0x7f3e4000de10 comp rx=0 tx=0).stop 2024-04-21T08:48:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.886+0000 7f3e516c1640 1 -- 172.21.15.28:0/4058104392 shutdown_connections 2024-04-21T08:48:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.886+0000 7f3e516c1640 1 --2- 172.21.15.28:0/4058104392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e4c102a10 0x7f3e4c19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.886+0000 7f3e516c1640 1 --2- 172.21.15.28:0/4058104392 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3e340a3bb0 0x7f3e340a6070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.886+0000 7f3e516c1640 1 --2- 172.21.15.28:0/4058104392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c102050 0x7f3e4c198670 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.886+0000 7f3e516c1640 1 --2- 172.21.15.28:0/4058104392 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e4c100e70 0x7f3e4c198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.886+0000 7f3e516c1640 1 -- 172.21.15.28:0/4058104392 >> 172.21.15.28:0/4058104392 conn(0x7f3e4c0fc470 msgr2=0x7f3e4c1053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.886+0000 7f3e516c1640 1 -- 172.21.15.28:0/4058104392 shutdown_connections 2024-04-21T08:48:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:08.886+0000 7f3e516c1640 1 -- 172.21.15.28:0/4058104392 wait complete. 2024-04-21T08:48:09.033 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:08 smithi171 ceph-mon[31358]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:09.033 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:08 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/763967022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:09.033 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:08 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1205603068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:48:09.033 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:08 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/4058104392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:09.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:08 smithi152 ceph-mon[31401]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:09.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:08 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/763967022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:09.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:08 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1205603068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:48:09.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:08 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/4058104392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:09.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:08 smithi028 ceph-mon[28003]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:09.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:08 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/763967022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:09.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:08 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1205603068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:48:09.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:08 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/4058104392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:09.370 INFO:journalctl@ceph.mds.l.smithi171.stdout:Apr 21 08:48:09 smithi171 systemd[1]: Starting Ceph mds.l for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T08:48:09.690 INFO:journalctl@ceph.mds.l.smithi171.stdout:Apr 21 08:48:09 smithi171 podman[61179]: 2024-04-21 08:48:09.3290829 +0000 UTC m=+0.017956882 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:48:09.691 INFO:journalctl@ceph.mds.l.smithi171.stdout:Apr 21 08:48:09 smithi171 podman[61179]: 2024-04-21 08:48:09.446156378 +0000 UTC m=+0.135030341 container create 813b88aa70676a576cd77097e052f183ec1a3be2f74cc62ea36bc15c9e4ec94f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:48:09.691 INFO:journalctl@ceph.mds.l.smithi171.stdout:Apr 21 08:48:09 smithi171 podman[61179]: 2024-04-21 08:48:09.665554007 +0000 UTC m=+0.354427971 container init 813b88aa70676a576cd77097e052f183ec1a3be2f74cc62ea36bc15c9e4ec94f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-l, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-04-21T08:48:09.691 INFO:journalctl@ceph.mds.l.smithi171.stdout:Apr 21 08:48:09 smithi171 podman[61179]: 2024-04-21 08:48:09.668224932 +0000 UTC m=+0.357098895 container start 813b88aa70676a576cd77097e052f183ec1a3be2f74cc62ea36bc15c9e4ec94f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:48:09.691 INFO:journalctl@ceph.mds.l.smithi171.stdout:Apr 21 08:48:09 smithi171 ceph-mds[61198]: -- 172.21.15.171:0/3817127883 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559c34479a40 con 0x559c3444d000 2024-04-21T08:48:10.120 INFO:journalctl@ceph.mds.l.smithi171.stdout:Apr 21 08:48:09 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-l[61194]: starting mds.l at 2024-04-21T08:48:10.120 INFO:journalctl@ceph.mds.l.smithi171.stdout:Apr 21 08:48:09 smithi171 bash[61179]: 813b88aa70676a576cd77097e052f183ec1a3be2f74cc62ea36bc15c9e4ec94f 2024-04-21T08:48:10.120 INFO:journalctl@ceph.mds.l.smithi171.stdout:Apr 21 08:48:09 smithi171 systemd[1]: Started Ceph mds.l for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:48:10.843 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:10 smithi152 ceph-mon[31401]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:10.843 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:10 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.171:6832/3170910324,v1:172.21.15.171:6833/3170910324] up:boot 2024-04-21T08:48:10.843 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:10 smithi152 ceph-mon[31401]: fsmap 1 up:standby 2024-04-21T08:48:10.843 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:10 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:48:10.843 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:10 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:10.844 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:10 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:10.844 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:10 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:10.844 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:10 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:10.844 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:10 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:10.844 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:10 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:10.844 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:10 smithi152 ceph-mon[31401]: Deploying daemon mds.k on smithi152 2024-04-21T08:48:10.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.014+0000 7fb907fa4640 1 -- 172.21.15.28:0/2482129290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9001029f0 msgr2=0x7fb900104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.014+0000 7fb907fa4640 1 --2- 172.21.15.28:0/2482129290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9001029f0 0x7fb900104de0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009ca0 tx=0x7fb8fc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.014+0000 7fb907fa4640 1 -- 172.21.15.28:0/2482129290 shutdown_connections 2024-04-21T08:48:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.014+0000 7fb907fa4640 1 --2- 172.21.15.28:0/2482129290 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb900107c50 0x7fb90010a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.014+0000 7fb907fa4640 1 --2- 172.21.15.28:0/2482129290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb900105320 0x7fb900107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.014+0000 7fb907fa4640 1 --2- 172.21.15.28:0/2482129290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9001029f0 0x7fb900104de0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.014+0000 7fb907fa4640 1 -- 172.21.15.28:0/2482129290 >> 172.21.15.28:0/2482129290 conn(0x7fb9000fc3e0 msgr2=0x7fb9000fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.014+0000 7fb907fa4640 1 -- 172.21.15.28:0/2482129290 shutdown_connections 2024-04-21T08:48:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.014+0000 7fb907fa4640 1 -- 172.21.15.28:0/2482129290 wait complete. 2024-04-21T08:48:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.014+0000 7fb907fa4640 1 Processor -- start 2024-04-21T08:48:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb907fa4640 1 -- start start 2024-04-21T08:48:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb907fa4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9001029f0 0x7fb900198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb907fa4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb900105320 0x7fb9001986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb907fa4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900107c50 0x7fb90019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb907fa4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb90010daa0 con 0x7fb900107c50 2024-04-21T08:48:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb907fa4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb90010d920 con 0x7fb9001029f0 2024-04-21T08:48:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb907fa4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb90010dc20 con 0x7fb900105320 2024-04-21T08:48:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb906541640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900107c50 0x7fb90019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb906541640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900107c50 0x7fb90019f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59272/0 (socket says 172.21.15.28:59272) 2024-04-21T08:48:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb906541640 1 -- 172.21.15.28:0/1263838198 learned_addr learned my addr 172.21.15.28:0/1263838198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb905d40640 1 --2- 172.21.15.28:0/1263838198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9001029f0 0x7fb900198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb90553f640 1 --2- 172.21.15.28:0/1263838198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb900105320 0x7fb9001986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb906541640 1 -- 172.21.15.28:0/1263838198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb900105320 msgr2=0x7fb9001986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb906541640 1 --2- 172.21.15.28:0/1263838198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb900105320 0x7fb9001986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb906541640 1 -- 172.21.15.28:0/1263838198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9001029f0 msgr2=0x7fb900198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb906541640 1 --2- 172.21.15.28:0/1263838198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9001029f0 0x7fb900198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb906541640 1 -- 172.21.15.28:0/1263838198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90019fd90 con 0x7fb900107c50 2024-04-21T08:48:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb906541640 1 --2- 172.21.15.28:0/1263838198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900107c50 0x7fb90019f720 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb8f800d950 tx=0x7fb8f800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb8f6ffd640 1 -- 172.21.15.28:0/1263838198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f8017030 con 0x7fb900107c50 2024-04-21T08:48:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.015+0000 7fb8f6ffd640 1 -- 172.21.15.28:0/1263838198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8f8004020 con 0x7fb900107c50 2024-04-21T08:48:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.016+0000 7fb907fa4640 1 -- 172.21.15.28:0/1263838198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9001a0080 con 0x7fb900107c50 2024-04-21T08:48:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.016+0000 7fb8f6ffd640 1 -- 172.21.15.28:0/1263838198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f8003af0 con 0x7fb900107c50 2024-04-21T08:48:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.016+0000 7fb907fa4640 1 -- 172.21.15.28:0/1263838198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9001a05d0 con 0x7fb900107c50 2024-04-21T08:48:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.017+0000 7fb907fa4640 1 -- 172.21.15.28:0/1263838198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9000fded0 con 0x7fb900107c50 2024-04-21T08:48:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.018+0000 7fb8f6ffd640 1 -- 172.21.15.28:0/1263838198 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb8f801a070 con 0x7fb900107c50 2024-04-21T08:48:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.018+0000 7fb8f6ffd640 1 --2- 172.21.15.28:0/1263838198 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb8ec0a3c30 0x7fb8ec0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.018+0000 7fb905d40640 1 --2- 172.21.15.28:0/1263838198 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb8ec0a3c30 0x7fb8ec0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.018+0000 7fb8f6ffd640 1 -- 172.21.15.28:0/1263838198 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb8f80cf980 con 0x7fb900107c50 2024-04-21T08:48:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.019+0000 7fb905d40640 1 --2- 172.21.15.28:0/1263838198 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb8ec0a3c30 0x7fb8ec0a60f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc005e30 tx=0x7fb8fc000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:11.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.023+0000 7fb8f6ffd640 1 -- 172.21.15.28:0/1263838198 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb8f809c800 con 0x7fb900107c50 2024-04-21T08:48:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:10 smithi028 ceph-mon[28003]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:10 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.171:6832/3170910324,v1:172.21.15.171:6833/3170910324] up:boot 2024-04-21T08:48:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:10 smithi028 ceph-mon[28003]: fsmap 1 up:standby 2024-04-21T08:48:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:10 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:48:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:10 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:10 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:10 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:10 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:10 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:11.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:10 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:11.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:10 smithi028 ceph-mon[28003]: Deploying daemon mds.k on smithi152 2024-04-21T08:48:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:10 smithi171 ceph-mon[31358]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:10 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.171:6832/3170910324,v1:172.21.15.171:6833/3170910324] up:boot 2024-04-21T08:48:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:10 smithi171 ceph-mon[31358]: fsmap 1 up:standby 2024-04-21T08:48:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:10 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:48:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:10 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:10 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:10 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:10 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:10 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:10 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:10 smithi171 ceph-mon[31358]: Deploying daemon mds.k on smithi152 2024-04-21T08:48:11.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.251+0000 7fb907fa4640 1 -- 172.21.15.28:0/1263838198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb900062b50 con 0x7fb900107c50 2024-04-21T08:48:11.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.252+0000 7fb8f6ffd640 1 -- 172.21.15.28:0/1263838198 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fb8f809c100 con 0x7fb900107c50 2024-04-21T08:48:11.251 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:11.252 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:48:11.252 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2024-04-21T08:48:11.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.253+0000 7fb907fa4640 1 -- 172.21.15.28:0/1263838198 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb8ec0a3c30 msgr2=0x7fb8ec0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:11.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.253+0000 7fb907fa4640 1 --2- 172.21.15.28:0/1263838198 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb8ec0a3c30 0x7fb8ec0a60f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc005e30 tx=0x7fb8fc000f40 comp rx=0 tx=0).stop 2024-04-21T08:48:11.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.253+0000 7fb907fa4640 1 -- 172.21.15.28:0/1263838198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900107c50 msgr2=0x7fb90019f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.253+0000 7fb907fa4640 1 --2- 172.21.15.28:0/1263838198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900107c50 0x7fb90019f720 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb8f800d950 tx=0x7fb8f800dd50 comp rx=0 tx=0).stop 2024-04-21T08:48:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.254+0000 7fb907fa4640 1 -- 172.21.15.28:0/1263838198 shutdown_connections 2024-04-21T08:48:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.254+0000 7fb907fa4640 1 --2- 172.21.15.28:0/1263838198 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb8ec0a3c30 0x7fb8ec0a60f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.254+0000 7fb907fa4640 1 --2- 172.21.15.28:0/1263838198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb900107c50 0x7fb90019f720 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.254+0000 7fb907fa4640 1 --2- 172.21.15.28:0/1263838198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb900105320 0x7fb9001986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.254+0000 7fb907fa4640 1 --2- 172.21.15.28:0/1263838198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9001029f0 0x7fb900198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.254+0000 7fb907fa4640 1 -- 172.21.15.28:0/1263838198 >> 172.21.15.28:0/1263838198 conn(0x7fb9000fc3e0 msgr2=0x7fb900103290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.254+0000 7fb907fa4640 1 -- 172.21.15.28:0/1263838198 shutdown_connections 2024-04-21T08:48:11.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:11.254+0000 7fb907fa4640 1 -- 172.21.15.28:0/1263838198 wait complete. 2024-04-21T08:48:11.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:11 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1263838198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:12.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:11 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1263838198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:12.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:11 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1263838198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:12.301 INFO:journalctl@ceph.mds.k.smithi152.stdout:Apr 21 08:48:12 smithi152 systemd[1]: Starting Ceph mds.k for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T08:48:12.571 INFO:journalctl@ceph.mds.k.smithi152.stdout:Apr 21 08:48:12 smithi152 podman[61268]: 2024-04-21 08:48:12.205405938 +0000 UTC m=+0.017667128 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:48:12.572 INFO:journalctl@ceph.mds.k.smithi152.stdout:Apr 21 08:48:12 smithi152 podman[61268]: 2024-04-21 08:48:12.310496054 +0000 UTC m=+0.122757235 container create f2ea8b06fbbdeabbc04ce2dca311597b1bbc4c3d8ebcc56f58448df3e551b47f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-k, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-21T08:48:12.572 INFO:journalctl@ceph.mds.k.smithi152.stdout:Apr 21 08:48:12 smithi152 podman[61268]: 2024-04-21 08:48:12.546771588 +0000 UTC m=+0.359032769 container init f2ea8b06fbbdeabbc04ce2dca311597b1bbc4c3d8ebcc56f58448df3e551b47f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-k, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-21T08:48:12.572 INFO:journalctl@ceph.mds.k.smithi152.stdout:Apr 21 08:48:12 smithi152 podman[61268]: 2024-04-21 08:48:12.549517471 +0000 UTC m=+0.361778653 container start f2ea8b06fbbdeabbc04ce2dca311597b1bbc4c3d8ebcc56f58448df3e551b47f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:48:12.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:12 smithi152 ceph-mon[31401]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:12.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:12 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:12.855 INFO:journalctl@ceph.mds.k.smithi152.stdout:Apr 21 08:48:12 smithi152 ceph-mds[61286]: -- 172.21.15.152:0/3731655118 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eb37905a40 con 0x55eb378d8c00 2024-04-21T08:48:12.855 INFO:journalctl@ceph.mds.k.smithi152.stdout:Apr 21 08:48:12 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-k[61282]: starting mds.k at 2024-04-21T08:48:12.855 INFO:journalctl@ceph.mds.k.smithi152.stdout:Apr 21 08:48:12 smithi152 bash[61268]: f2ea8b06fbbdeabbc04ce2dca311597b1bbc4c3d8ebcc56f58448df3e551b47f 2024-04-21T08:48:12.855 INFO:journalctl@ceph.mds.k.smithi152.stdout:Apr 21 08:48:12 smithi152 systemd[1]: Started Ceph mds.k for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:48:13.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:12 smithi028 ceph-mon[28003]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:13.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:12 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:12 smithi171 ceph-mon[31358]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:12 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:13.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:13.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.377+0000 7f2a92406640 1 -- 172.21.15.28:0/3358332595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a8c105410 msgr2=0x7f2a8c107800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.377+0000 7f2a92406640 1 --2- 172.21.15.28:0/3358332595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a8c105410 0x7f2a8c107800 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2a80009900 tx=0x7f2a8002bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.377+0000 7f2a92406640 1 -- 172.21.15.28:0/3358332595 shutdown_connections 2024-04-21T08:48:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.377+0000 7f2a92406640 1 --2- 172.21.15.28:0/3358332595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a8c107d40 0x7f2a8c10a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.377+0000 7f2a92406640 1 --2- 172.21.15.28:0/3358332595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a8c105410 0x7f2a8c107800 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.377+0000 7f2a92406640 1 --2- 172.21.15.28:0/3358332595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2a8c102ae0 0x7f2a8c104ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.377+0000 7f2a92406640 1 -- 172.21.15.28:0/3358332595 >> 172.21.15.28:0/3358332595 conn(0x7f2a8c0fc470 msgr2=0x7f2a8c0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.377+0000 7f2a92406640 1 -- 172.21.15.28:0/3358332595 shutdown_connections 2024-04-21T08:48:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.377+0000 7f2a92406640 1 -- 172.21.15.28:0/3358332595 wait complete. 2024-04-21T08:48:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.378+0000 7f2a92406640 1 Processor -- start 2024-04-21T08:48:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.378+0000 7f2a92406640 1 -- start start 2024-04-21T08:48:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.378+0000 7f2a92406640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a8c102ae0 0x7f2a8c198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.378+0000 7f2a92406640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a8c105410 0x7f2a8c198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.378+0000 7f2a92406640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2a8c107d40 0x7f2a8c19f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.378+0000 7f2a92406640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8c10db40 con 0x7f2a8c102ae0 2024-04-21T08:48:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.378+0000 7f2a92406640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8c10d9c0 con 0x7f2a8c107d40 2024-04-21T08:48:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.378+0000 7f2a92406640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8c10dcc0 con 0x7f2a8c105410 2024-04-21T08:48:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.378+0000 7f2a8bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a8c102ae0 0x7f2a8c198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.378+0000 7f2a8bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a8c102ae0 0x7f2a8c198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59294/0 (socket says 172.21.15.28:59294) 2024-04-21T08:48:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.378+0000 7f2a8bfff640 1 -- 172.21.15.28:0/2467409756 learned_addr learned my addr 172.21.15.28:0/2467409756 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.379+0000 7f2a909a3640 1 --2- 172.21.15.28:0/2467409756 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2a8c107d40 0x7f2a8c19f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.379+0000 7f2a8bfff640 1 -- 172.21.15.28:0/2467409756 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a8c105410 msgr2=0x7f2a8c198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.379+0000 7f2a8bfff640 1 --2- 172.21.15.28:0/2467409756 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a8c105410 0x7f2a8c198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.379+0000 7f2a8bfff640 1 -- 172.21.15.28:0/2467409756 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2a8c107d40 msgr2=0x7f2a8c19f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.379+0000 7f2a8bfff640 1 --2- 172.21.15.28:0/2467409756 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2a8c107d40 0x7f2a8c19f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.379+0000 7f2a8bfff640 1 -- 172.21.15.28:0/2467409756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8c19fe60 con 0x7f2a8c102ae0 2024-04-21T08:48:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.379+0000 7f2a909a3640 1 --2- 172.21.15.28:0/2467409756 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2a8c107d40 0x7f2a8c19f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.379+0000 7f2a8bfff640 1 --2- 172.21.15.28:0/2467409756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a8c102ae0 0x7f2a8c198240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c00c890 tx=0x7f2a7c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.379+0000 7f2a897fa640 1 -- 172.21.15.28:0/2467409756 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c004220 con 0x7f2a8c102ae0 2024-04-21T08:48:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.379+0000 7f2a897fa640 1 -- 172.21.15.28:0/2467409756 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a7c0043c0 con 0x7f2a8c102ae0 2024-04-21T08:48:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.379+0000 7f2a92406640 1 -- 172.21.15.28:0/2467409756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a8c1a0150 con 0x7f2a8c102ae0 2024-04-21T08:48:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.379+0000 7f2a897fa640 1 -- 172.21.15.28:0/2467409756 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c004a00 con 0x7f2a8c102ae0 2024-04-21T08:48:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.379+0000 7f2a92406640 1 -- 172.21.15.28:0/2467409756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a8c1a0720 con 0x7f2a8c102ae0 2024-04-21T08:48:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.380+0000 7f2a92406640 1 -- 172.21.15.28:0/2467409756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a8c0fdf60 con 0x7f2a8c102ae0 2024-04-21T08:48:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.381+0000 7f2a897fa640 1 -- 172.21.15.28:0/2467409756 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2a7c0079b0 con 0x7f2a8c102ae0 2024-04-21T08:48:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.382+0000 7f2a897fa640 1 --2- 172.21.15.28:0/2467409756 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2a740a3bb0 0x7f2a740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.382+0000 7f2a8b7fe640 1 --2- 172.21.15.28:0/2467409756 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2a740a3bb0 0x7f2a740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.382+0000 7f2a897fa640 1 -- 172.21.15.28:0/2467409756 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2a7c0cfdf0 con 0x7f2a8c102ae0 2024-04-21T08:48:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.382+0000 7f2a8b7fe640 1 --2- 172.21.15.28:0/2467409756 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2a740a3bb0 0x7f2a740a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2a80036040 tx=0x7f2a80000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:13.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.386+0000 7f2a897fa640 1 -- 172.21.15.28:0/2467409756 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2a7c014030 con 0x7f2a8c102ae0 2024-04-21T08:48:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.615+0000 7f2a92406640 1 -- 172.21.15.28:0/2467409756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a8c062b50 con 0x7f2a8c102ae0 2024-04-21T08:48:13.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.616+0000 7f2a897fa640 1 -- 172.21.15.28:0/2467409756 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f2a7c09cbd0 con 0x7f2a8c102ae0 2024-04-21T08:48:13.616 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:13.616 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:48:13.616 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-04-21T08:48:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.618+0000 7f2a92406640 1 -- 172.21.15.28:0/2467409756 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2a740a3bb0 msgr2=0x7f2a740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.618+0000 7f2a92406640 1 --2- 172.21.15.28:0/2467409756 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2a740a3bb0 0x7f2a740a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2a80036040 tx=0x7f2a80000bc0 comp rx=0 tx=0).stop 2024-04-21T08:48:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.618+0000 7f2a92406640 1 -- 172.21.15.28:0/2467409756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a8c102ae0 msgr2=0x7f2a8c198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.618+0000 7f2a92406640 1 --2- 172.21.15.28:0/2467409756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a8c102ae0 0x7f2a8c198240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c00c890 tx=0x7f2a7c00cd50 comp rx=0 tx=0).stop 2024-04-21T08:48:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.618+0000 7f2a92406640 1 -- 172.21.15.28:0/2467409756 shutdown_connections 2024-04-21T08:48:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.618+0000 7f2a92406640 1 --2- 172.21.15.28:0/2467409756 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2a8c107d40 0x7f2a8c19f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.618+0000 7f2a92406640 1 --2- 172.21.15.28:0/2467409756 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2a8c105410 0x7f2a8c198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.618+0000 7f2a92406640 1 --2- 172.21.15.28:0/2467409756 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2a740a3bb0 0x7f2a740a6070 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.618+0000 7f2a92406640 1 --2- 172.21.15.28:0/2467409756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a8c102ae0 0x7f2a8c198240 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.618+0000 7f2a92406640 1 -- 172.21.15.28:0/2467409756 >> 172.21.15.28:0/2467409756 conn(0x7f2a8c0fc470 msgr2=0x7f2a8c103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.618+0000 7f2a92406640 1 -- 172.21.15.28:0/2467409756 shutdown_connections 2024-04-21T08:48:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:13.618+0000 7f2a92406640 1 -- 172.21.15.28:0/2467409756 wait complete. 2024-04-21T08:48:13.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:13 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:13.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:13 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.152:6832/1529635773,v1:172.21.15.152:6833/1529635773] up:boot 2024-04-21T08:48:13.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:13 smithi028 ceph-mon[28003]: fsmap 2 up:standby 2024-04-21T08:48:13.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:13 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:48:13.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:13 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:13.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:13 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:13.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:13 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:13.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:13 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:13.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:13 smithi028 ceph-mon[28003]: Deploying daemon mds.j on smithi028 2024-04-21T08:48:13.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:13 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2467409756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:14.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:13 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:14.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:13 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.152:6832/1529635773,v1:172.21.15.152:6833/1529635773] up:boot 2024-04-21T08:48:14.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:13 smithi152 ceph-mon[31401]: fsmap 2 up:standby 2024-04-21T08:48:14.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:13 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:48:14.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:13 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:14.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:13 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:14.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:13 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:14.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:13 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:14.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:13 smithi152 ceph-mon[31401]: Deploying daemon mds.j on smithi028 2024-04-21T08:48:14.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:13 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2467409756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:13 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:13 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.152:6832/1529635773,v1:172.21.15.152:6833/1529635773] up:boot 2024-04-21T08:48:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:13 smithi171 ceph-mon[31358]: fsmap 2 up:standby 2024-04-21T08:48:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:13 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:48:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:13 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:13 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:14.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:13 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:14.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:13 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:14.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:13 smithi171 ceph-mon[31358]: Deploying daemon mds.j on smithi028 2024-04-21T08:48:14.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:13 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2467409756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:14.967 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:14 smithi152 ceph-mon[31401]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:15.095 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:14 smithi028 ceph-mon[28003]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:15.095 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 08:48:15 smithi028 systemd[1]: Starting Ceph mds.j for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T08:48:15.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:14 smithi171 ceph-mon[31358]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:15.355 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 08:48:15 smithi028 podman[67938]: 2024-04-21 08:48:15.22070061 +0000 UTC m=+0.018671084 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:48:15.355 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 08:48:15 smithi028 podman[67938]: 2024-04-21 08:48:15.326123635 +0000 UTC m=+0.124094134 container create 167ccde53324610a7848153b0a16609282551242bacadb48b01139ab12de71f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-21T08:48:15.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:15.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.758+0000 7fbca9ba4640 1 -- 172.21.15.28:0/481467418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca41029f0 msgr2=0x7fbca4104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:15.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.758+0000 7fbca9ba4640 1 --2- 172.21.15.28:0/481467418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca41029f0 0x7fbca4104de0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c009900 tx=0x7fbc8c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:15.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.758+0000 7fbca9ba4640 1 -- 172.21.15.28:0/481467418 shutdown_connections 2024-04-21T08:48:15.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.758+0000 7fbca9ba4640 1 --2- 172.21.15.28:0/481467418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbca4107c50 0x7fbca410a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.758+0000 7fbca9ba4640 1 --2- 172.21.15.28:0/481467418 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbca4105320 0x7fbca4107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.758+0000 7fbca9ba4640 1 --2- 172.21.15.28:0/481467418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca41029f0 0x7fbca4104de0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.758+0000 7fbca9ba4640 1 -- 172.21.15.28:0/481467418 >> 172.21.15.28:0/481467418 conn(0x7fbca40fc3e0 msgr2=0x7fbca40fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:15.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.758+0000 7fbca9ba4640 1 -- 172.21.15.28:0/481467418 shutdown_connections 2024-04-21T08:48:15.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.758+0000 7fbca9ba4640 1 -- 172.21.15.28:0/481467418 wait complete. 2024-04-21T08:48:15.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.758+0000 7fbca9ba4640 1 Processor -- start 2024-04-21T08:48:15.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.758+0000 7fbca9ba4640 1 -- start start 2024-04-21T08:48:15.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca9ba4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca41029f0 0x7fbca4102360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca9ba4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbca4105320 0x7fbca41009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca9ba4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbca4107c50 0x7fbca4100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca9ba4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbca410dab0 con 0x7fbca41029f0 2024-04-21T08:48:15.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca9ba4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbca410d930 con 0x7fbca4105320 2024-04-21T08:48:15.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca9ba4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbca410dc30 con 0x7fbca4107c50 2024-04-21T08:48:15.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca41029f0 0x7fbca4102360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca41029f0 0x7fbca4102360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45060/0 (socket says 172.21.15.28:45060) 2024-04-21T08:48:15.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca37fe640 1 -- 172.21.15.28:0/4183829698 learned_addr learned my addr 172.21.15.28:0/4183829698 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:15.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca2ffd640 1 --2- 172.21.15.28:0/4183829698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbca4105320 0x7fbca41009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca3fff640 1 --2- 172.21.15.28:0/4183829698 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbca4107c50 0x7fbca4100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca37fe640 1 -- 172.21.15.28:0/4183829698 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbca4107c50 msgr2=0x7fbca4100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:15.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca37fe640 1 --2- 172.21.15.28:0/4183829698 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbca4107c50 0x7fbca4100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca37fe640 1 -- 172.21.15.28:0/4183829698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbca4105320 msgr2=0x7fbca41009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:15.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca37fe640 1 --2- 172.21.15.28:0/4183829698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbca4105320 0x7fbca41009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:15.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca37fe640 1 -- 172.21.15.28:0/4183829698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca4101770 con 0x7fbca41029f0 2024-04-21T08:48:15.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca2ffd640 1 --2- 172.21.15.28:0/4183829698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbca4105320 0x7fbca41009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:15.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca37fe640 1 --2- 172.21.15.28:0/4183829698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca41029f0 0x7fbca4102360 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c006010 tx=0x7fbc8c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:15.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.759+0000 7fbca0ff9640 1 -- 172.21.15.28:0/4183829698 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c043020 con 0x7fbca41029f0 2024-04-21T08:48:15.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.760+0000 7fbca9ba4640 1 -- 172.21.15.28:0/4183829698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbca41a22e0 con 0x7fbca41029f0 2024-04-21T08:48:15.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.760+0000 7fbca0ff9640 1 -- 172.21.15.28:0/4183829698 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc8c004440 con 0x7fbca41029f0 2024-04-21T08:48:15.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.760+0000 7fbca0ff9640 1 -- 172.21.15.28:0/4183829698 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c003bf0 con 0x7fbca41029f0 2024-04-21T08:48:15.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.760+0000 7fbca9ba4640 1 -- 172.21.15.28:0/4183829698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbca41a2830 con 0x7fbca41029f0 2024-04-21T08:48:15.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.761+0000 7fbca9ba4640 1 -- 172.21.15.28:0/4183829698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbca40fded0 con 0x7fbca41029f0 2024-04-21T08:48:15.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.762+0000 7fbca0ff9640 1 -- 172.21.15.28:0/4183829698 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbc8c046020 con 0x7fbca41029f0 2024-04-21T08:48:15.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.762+0000 7fbca0ff9640 1 --2- 172.21.15.28:0/4183829698 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbc900a3bb0 0x7fbc900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:15.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.763+0000 7fbca2ffd640 1 --2- 172.21.15.28:0/4183829698 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbc900a3bb0 0x7fbc900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:15.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.763+0000 7fbca0ff9640 1 -- 172.21.15.28:0/4183829698 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbc8c0f28d0 con 0x7fbca41029f0 2024-04-21T08:48:15.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.763+0000 7fbca2ffd640 1 --2- 172.21.15.28:0/4183829698 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbc900a3bb0 0x7fbc900a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbca41020d0 tx=0x7fbc9400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:15.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:15.766+0000 7fbca0ff9640 1 -- 172.21.15.28:0/4183829698 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbc8c04d020 con 0x7fbca41029f0 2024-04-21T08:48:15.855 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 08:48:15 smithi028 podman[67938]: 2024-04-21 08:48:15.703926602 +0000 UTC m=+0.501897072 container init 167ccde53324610a7848153b0a16609282551242bacadb48b01139ab12de71f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-21T08:48:15.855 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 08:48:15 smithi028 podman[67938]: 2024-04-21 08:48:15.707110941 +0000 UTC m=+0.505081401 container start 167ccde53324610a7848153b0a16609282551242bacadb48b01139ab12de71f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-21T08:48:15.855 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 08:48:15 smithi028 ceph-mds[67977]: -- 172.21.15.28:0/498639123 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5625ae00ba40 con 0x5625adfde800 2024-04-21T08:48:15.855 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 08:48:15 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-j[67953]: starting mds.j at 2024-04-21T08:48:15.855 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 08:48:15 smithi028 bash[67938]: 167ccde53324610a7848153b0a16609282551242bacadb48b01139ab12de71f5 2024-04-21T08:48:15.855 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 08:48:15 smithi028 systemd[1]: Started Ceph mds.j for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:48:16.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:16.030+0000 7fbca9ba4640 1 -- 172.21.15.28:0/4183829698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbca402d5c0 con 0x7fbca41029f0 2024-04-21T08:48:16.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:16.030+0000 7fbca0ff9640 1 -- 172.21.15.28:0/4183829698 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7fbc8c0bf750 con 0x7fbca41029f0 2024-04-21T08:48:16.030 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:16.030 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:48:16.030 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 4 2024-04-21T08:48:16.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:16.032+0000 7fbca9ba4640 1 -- 172.21.15.28:0/4183829698 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbc900a3bb0 msgr2=0x7fbc900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:16.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:16.032+0000 7fbca9ba4640 1 --2- 172.21.15.28:0/4183829698 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbc900a3bb0 0x7fbc900a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbca41020d0 tx=0x7fbc9400b390 comp rx=0 tx=0).stop 2024-04-21T08:48:16.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:16.032+0000 7fbca9ba4640 1 -- 172.21.15.28:0/4183829698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca41029f0 msgr2=0x7fbca4102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:16.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:16.032+0000 7fbca9ba4640 1 --2- 172.21.15.28:0/4183829698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca41029f0 0x7fbca4102360 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c006010 tx=0x7fbc8c004040 comp rx=0 tx=0).stop 2024-04-21T08:48:16.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:16.032+0000 7fbca9ba4640 1 -- 172.21.15.28:0/4183829698 shutdown_connections 2024-04-21T08:48:16.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:16.032+0000 7fbca9ba4640 1 --2- 172.21.15.28:0/4183829698 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbca4107c50 0x7fbca4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:16.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:16.032+0000 7fbca9ba4640 1 --2- 172.21.15.28:0/4183829698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbca4105320 0x7fbca41009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:16.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:16.032+0000 7fbca9ba4640 1 --2- 172.21.15.28:0/4183829698 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbc900a3bb0 0x7fbc900a6070 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:16.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:16.032+0000 7fbca9ba4640 1 --2- 172.21.15.28:0/4183829698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca41029f0 0x7fbca4102360 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:16.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:16.032+0000 7fbca9ba4640 1 -- 172.21.15.28:0/4183829698 >> 172.21.15.28:0/4183829698 conn(0x7fbca40fc3e0 msgr2=0x7fbca41034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:16.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:16.032+0000 7fbca9ba4640 1 -- 172.21.15.28:0/4183829698 shutdown_connections 2024-04-21T08:48:16.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:16.032+0000 7fbca9ba4640 1 -- 172.21.15.28:0/4183829698 wait complete. 2024-04-21T08:48:16.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:16 smithi028 ceph-mon[28003]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:16.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:16 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.28:6834/3468163100,v1:172.21.15.28:6835/3468163100] up:boot 2024-04-21T08:48:16.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:16 smithi028 ceph-mon[28003]: fsmap 3 up:standby 2024-04-21T08:48:16.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:48:16.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:16.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:16.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:16.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:16.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:16.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:16 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:16.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:16 smithi028 ceph-mon[28003]: Deploying daemon mds.d on smithi028 2024-04-21T08:48:16.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:16 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/4183829698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:17.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:16 smithi152 ceph-mon[31401]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:17.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:16 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.28:6834/3468163100,v1:172.21.15.28:6835/3468163100] up:boot 2024-04-21T08:48:17.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:16 smithi152 ceph-mon[31401]: fsmap 3 up:standby 2024-04-21T08:48:17.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:48:17.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:17.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:17.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:17.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:17.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:17.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:16 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:17.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:16 smithi152 ceph-mon[31401]: Deploying daemon mds.d on smithi028 2024-04-21T08:48:17.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:16 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/4183829698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:16 smithi171 ceph-mon[31358]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:16 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.28:6834/3468163100,v1:172.21.15.28:6835/3468163100] up:boot 2024-04-21T08:48:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:16 smithi171 ceph-mon[31358]: fsmap 3 up:standby 2024-04-21T08:48:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:48:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:17.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:16 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:17.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:16 smithi171 ceph-mon[31358]: Deploying daemon mds.d on smithi028 2024-04-21T08:48:17.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:16 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/4183829698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:18.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:18.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.219+0000 7f4460426640 1 -- 172.21.15.28:0/847221325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4458105d60 msgr2=0x7f445810c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:18.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.219+0000 7f4460426640 1 --2- 172.21.15.28:0/847221325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4458105d60 0x7f445810c5f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4450009ca0 tx=0x7f445002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:18.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.219+0000 7f4460426640 1 -- 172.21.15.28:0/847221325 shutdown_connections 2024-04-21T08:48:18.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.219+0000 7f4460426640 1 --2- 172.21.15.28:0/847221325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4458105d60 0x7f445810c5f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:18.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.219+0000 7f4460426640 1 --2- 172.21.15.28:0/847221325 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44581053a0 0x7f4458105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:18.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.219+0000 7f4460426640 1 --2- 172.21.15.28:0/847221325 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44581041b0 0x7f44581045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:18.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.219+0000 7f4460426640 1 -- 172.21.15.28:0/847221325 >> 172.21.15.28:0/847221325 conn(0x7f44580ff740 msgr2=0x7f4458101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:18.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.219+0000 7f4460426640 1 -- 172.21.15.28:0/847221325 shutdown_connections 2024-04-21T08:48:18.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.219+0000 7f4460426640 1 -- 172.21.15.28:0/847221325 wait complete. 2024-04-21T08:48:18.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f4460426640 1 Processor -- start 2024-04-21T08:48:18.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f4460426640 1 -- start start 2024-04-21T08:48:18.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f4460426640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44581041b0 0x7f445819a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:18.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f4460426640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44581053a0 0x7f445819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:18.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f4460426640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4458105d60 0x7f44581a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:18.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f4460426640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f445810fcc0 con 0x7f44581041b0 2024-04-21T08:48:18.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f4460426640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f445810fb40 con 0x7f44581053a0 2024-04-21T08:48:18.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f4460426640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f445810fe40 con 0x7f4458105d60 2024-04-21T08:48:18.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f445e1c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44581041b0 0x7f445819a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:18.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f445e1c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44581041b0 0x7f445819a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45080/0 (socket says 172.21.15.28:45080) 2024-04-21T08:48:18.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f445e1c2640 1 -- 172.21.15.28:0/3058572435 learned_addr learned my addr 172.21.15.28:0/3058572435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:18.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f445d9c1640 1 --2- 172.21.15.28:0/3058572435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44581053a0 0x7f445819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:18.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f445e9c3640 1 --2- 172.21.15.28:0/3058572435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4458105d60 0x7f44581a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:18.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f445e1c2640 1 -- 172.21.15.28:0/3058572435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4458105d60 msgr2=0x7f44581a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:18.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f445e1c2640 1 --2- 172.21.15.28:0/3058572435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4458105d60 0x7f44581a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:18.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f445e1c2640 1 -- 172.21.15.28:0/3058572435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44581053a0 msgr2=0x7f445819a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f445e1c2640 1 --2- 172.21.15.28:0/3058572435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44581053a0 0x7f445819a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f445e1c2640 1 -- 172.21.15.28:0/3058572435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44581a1f20 con 0x7f44581041b0 2024-04-21T08:48:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f445e9c3640 1 --2- 172.21.15.28:0/3058572435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4458105d60 0x7f44581a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.220+0000 7f445e1c2640 1 --2- 172.21.15.28:0/3058572435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44581041b0 0x7f445819a300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f445400d770 tx=0x7f445400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.221+0000 7f444f7fe640 1 -- 172.21.15.28:0/3058572435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4454017070 con 0x7f44581041b0 2024-04-21T08:48:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.221+0000 7f4460426640 1 -- 172.21.15.28:0/3058572435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44581a2210 con 0x7f44581041b0 2024-04-21T08:48:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.221+0000 7f444f7fe640 1 -- 172.21.15.28:0/3058572435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4454009c30 con 0x7f44581041b0 2024-04-21T08:48:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.221+0000 7f444f7fe640 1 -- 172.21.15.28:0/3058572435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4454003cc0 con 0x7f44581041b0 2024-04-21T08:48:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.221+0000 7f4460426640 1 -- 172.21.15.28:0/3058572435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44581a27b0 con 0x7f44581041b0 2024-04-21T08:48:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.222+0000 7f4460426640 1 -- 172.21.15.28:0/3058572435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f445810b550 con 0x7f44581041b0 2024-04-21T08:48:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.223+0000 7f444f7fe640 1 -- 172.21.15.28:0/3058572435 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4454003e60 con 0x7f44581041b0 2024-04-21T08:48:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.224+0000 7f444f7fe640 1 --2- 172.21.15.28:0/3058572435 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f44440a3bb0 0x7f44440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.224+0000 7f445d9c1640 1 --2- 172.21.15.28:0/3058572435 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f44440a3bb0 0x7f44440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.224+0000 7f444f7fe640 1 -- 172.21.15.28:0/3058572435 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f44540cf8f0 con 0x7f44581041b0 2024-04-21T08:48:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.224+0000 7f445d9c1640 1 --2- 172.21.15.28:0/3058572435 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f44440a3bb0 0x7f44440a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f445819b810 tx=0x7f444800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:18.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.229+0000 7f444f7fe640 1 -- 172.21.15.28:0/3058572435 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f445409c770 con 0x7f44581041b0 2024-04-21T08:48:18.355 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 08:48:18 smithi028 systemd[1]: Starting Ceph mds.d for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T08:48:18.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.467+0000 7f4460426640 1 -- 172.21.15.28:0/3058572435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f445802d5c0 con 0x7f44581041b0 2024-04-21T08:48:18.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.468+0000 7f444f7fe640 1 -- 172.21.15.28:0/3058572435 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7f445802d5c0 con 0x7f44581041b0 2024-04-21T08:48:18.468 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:18.468 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:48:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 4 2024-04-21T08:48:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.470+0000 7f4460426640 1 -- 172.21.15.28:0/3058572435 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f44440a3bb0 msgr2=0x7f44440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.470+0000 7f4460426640 1 --2- 172.21.15.28:0/3058572435 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f44440a3bb0 0x7f44440a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f445819b810 tx=0x7f444800a2d0 comp rx=0 tx=0).stop 2024-04-21T08:48:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.470+0000 7f4460426640 1 -- 172.21.15.28:0/3058572435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44581041b0 msgr2=0x7f445819a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.470+0000 7f4460426640 1 --2- 172.21.15.28:0/3058572435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44581041b0 0x7f445819a300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f445400d770 tx=0x7f445400dc30 comp rx=0 tx=0).stop 2024-04-21T08:48:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.470+0000 7f4460426640 1 -- 172.21.15.28:0/3058572435 shutdown_connections 2024-04-21T08:48:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.470+0000 7f4460426640 1 --2- 172.21.15.28:0/3058572435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4458105d60 0x7f44581a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.470+0000 7f4460426640 1 --2- 172.21.15.28:0/3058572435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44581053a0 0x7f445819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.470+0000 7f4460426640 1 --2- 172.21.15.28:0/3058572435 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f44440a3bb0 0x7f44440a6070 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.470+0000 7f4460426640 1 --2- 172.21.15.28:0/3058572435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44581041b0 0x7f445819a300 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.470+0000 7f4460426640 1 -- 172.21.15.28:0/3058572435 >> 172.21.15.28:0/3058572435 conn(0x7f44580ff740 msgr2=0x7f4458108720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.470+0000 7f4460426640 1 -- 172.21.15.28:0/3058572435 shutdown_connections 2024-04-21T08:48:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:18.470+0000 7f4460426640 1 -- 172.21.15.28:0/3058572435 wait complete. 2024-04-21T08:48:18.648 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 08:48:18 smithi028 podman[68321]: 2024-04-21 08:48:18.289066459 +0000 UTC m=+0.020509468 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:48:18.649 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 08:48:18 smithi028 podman[68321]: 2024-04-21 08:48:18.396957459 +0000 UTC m=+0.128400456 container create 763b96d0c50c833f001fdb4143bf6cb2bc4e74efc6a12c1d24f4044ac33af645 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-21T08:48:19.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:18 smithi152 ceph-mon[31401]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:19.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:18 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3058572435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:19.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:18 smithi028 ceph-mon[28003]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:19.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:18 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3058572435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:19.105 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 08:48:18 smithi028 podman[68321]: 2024-04-21 08:48:18.649885925 +0000 UTC m=+0.381328924 container init 763b96d0c50c833f001fdb4143bf6cb2bc4e74efc6a12c1d24f4044ac33af645 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-21T08:48:19.105 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 08:48:18 smithi028 podman[68321]: 2024-04-21 08:48:18.652941587 +0000 UTC m=+0.384384584 container start 763b96d0c50c833f001fdb4143bf6cb2bc4e74efc6a12c1d24f4044ac33af645 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:48:19.105 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 08:48:18 smithi028 ceph-mds[68342]: -- 172.21.15.28:0/3741637741 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5605e1307a40 con 0x5605e12da800 2024-04-21T08:48:19.105 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 08:48:18 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-d[68338]: starting mds.d at 2024-04-21T08:48:19.106 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 08:48:18 smithi028 bash[68321]: 763b96d0c50c833f001fdb4143bf6cb2bc4e74efc6a12c1d24f4044ac33af645 2024-04-21T08:48:19.106 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 08:48:18 smithi028 systemd[1]: Started Ceph mds.d for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:48:19.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:18 smithi171 ceph-mon[31358]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:19.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:18 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3058572435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:20.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:19 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.28:6836/3823720712,v1:172.21.15.28:6837/3823720712] up:boot 2024-04-21T08:48:20.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:19 smithi152 ceph-mon[31401]: fsmap 4 up:standby 2024-04-21T08:48:20.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:19 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:48:20.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:19 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:20.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:19 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:20.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:19 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:20.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:19 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:20.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:19 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:20.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:19 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:20.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:19 smithi152 ceph-mon[31401]: Deploying daemon mds.e on smithi152 2024-04-21T08:48:20.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:19 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.28:6836/3823720712,v1:172.21.15.28:6837/3823720712] up:boot 2024-04-21T08:48:20.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:19 smithi028 ceph-mon[28003]: fsmap 4 up:standby 2024-04-21T08:48:20.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:19 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:48:20.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:19 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:20.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:19 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:20.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:19 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:20.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:19 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:20.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:19 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:20.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:19 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:20.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:19 smithi028 ceph-mon[28003]: Deploying daemon mds.e on smithi152 2024-04-21T08:48:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:19 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.28:6836/3823720712,v1:172.21.15.28:6837/3823720712] up:boot 2024-04-21T08:48:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:19 smithi171 ceph-mon[31358]: fsmap 4 up:standby 2024-04-21T08:48:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:19 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:48:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:19 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:19 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:19 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:19 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:19 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:19 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:19 smithi171 ceph-mon[31358]: Deploying daemon mds.e on smithi152 2024-04-21T08:48:20.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.602+0000 7f6a511a4640 1 -- 172.21.15.28:0/1803561711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c107c50 msgr2=0x7f6a4c10a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.602+0000 7f6a511a4640 1 --2- 172.21.15.28:0/1803561711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c107c50 0x7f6a4c10a0a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6a44009ca0 tx=0x7f6a4402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.602+0000 7f6a511a4640 1 -- 172.21.15.28:0/1803561711 shutdown_connections 2024-04-21T08:48:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.602+0000 7f6a511a4640 1 --2- 172.21.15.28:0/1803561711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c107c50 0x7f6a4c10a0a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.602+0000 7f6a511a4640 1 --2- 172.21.15.28:0/1803561711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a4c105320 0x7f6a4c107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.602+0000 7f6a511a4640 1 --2- 172.21.15.28:0/1803561711 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a4c1029f0 0x7f6a4c104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.602+0000 7f6a511a4640 1 -- 172.21.15.28:0/1803561711 >> 172.21.15.28:0/1803561711 conn(0x7f6a4c0fc3e0 msgr2=0x7f6a4c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.602+0000 7f6a511a4640 1 -- 172.21.15.28:0/1803561711 shutdown_connections 2024-04-21T08:48:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.602+0000 7f6a511a4640 1 -- 172.21.15.28:0/1803561711 wait complete. 2024-04-21T08:48:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.602+0000 7f6a511a4640 1 Processor -- start 2024-04-21T08:48:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a511a4640 1 -- start start 2024-04-21T08:48:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a511a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1029f0 0x7f6a4c198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a511a4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a4c105320 0x7f6a4c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a511a4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a4c107c50 0x7f6a4c19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a511a4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4c10daa0 con 0x7f6a4c1029f0 2024-04-21T08:48:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a511a4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4c10d920 con 0x7f6a4c105320 2024-04-21T08:48:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a511a4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4c10dc20 con 0x7f6a4c107c50 2024-04-21T08:48:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a4ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1029f0 0x7f6a4c198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a4ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1029f0 0x7f6a4c198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45108/0 (socket says 172.21.15.28:45108) 2024-04-21T08:48:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a4ad9d640 1 -- 172.21.15.28:0/2636185992 learned_addr learned my addr 172.21.15.28:0/2636185992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a4b59e640 1 --2- 172.21.15.28:0/2636185992 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a4c107c50 0x7f6a4c19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a4a59c640 1 --2- 172.21.15.28:0/2636185992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a4c105320 0x7f6a4c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a4ad9d640 1 -- 172.21.15.28:0/2636185992 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a4c107c50 msgr2=0x7f6a4c19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a4ad9d640 1 --2- 172.21.15.28:0/2636185992 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a4c107c50 0x7f6a4c19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a4ad9d640 1 -- 172.21.15.28:0/2636185992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a4c105320 msgr2=0x7f6a4c198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a4ad9d640 1 --2- 172.21.15.28:0/2636185992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a4c105320 0x7f6a4c198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.603+0000 7f6a4ad9d640 1 -- 172.21.15.28:0/2636185992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4c19fd60 con 0x7f6a4c1029f0 2024-04-21T08:48:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.604+0000 7f6a4ad9d640 1 --2- 172.21.15.28:0/2636185992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1029f0 0x7f6a4c198140 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c00d950 tx=0x7f6a3c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.604+0000 7f6a33fff640 1 -- 172.21.15.28:0/2636185992 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c017070 con 0x7f6a4c1029f0 2024-04-21T08:48:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.604+0000 7f6a33fff640 1 -- 172.21.15.28:0/2636185992 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a3c009c30 con 0x7f6a4c1029f0 2024-04-21T08:48:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.604+0000 7f6a511a4640 1 -- 172.21.15.28:0/2636185992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a4c19fff0 con 0x7f6a4c1029f0 2024-04-21T08:48:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.604+0000 7f6a33fff640 1 -- 172.21.15.28:0/2636185992 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c00f620 con 0x7f6a4c1029f0 2024-04-21T08:48:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.604+0000 7f6a511a4640 1 -- 172.21.15.28:0/2636185992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a4c1a04b0 con 0x7f6a4c1029f0 2024-04-21T08:48:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.605+0000 7f6a511a4640 1 -- 172.21.15.28:0/2636185992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a4c0fded0 con 0x7f6a4c1029f0 2024-04-21T08:48:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.606+0000 7f6a33fff640 1 -- 172.21.15.28:0/2636185992 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6a3c010460 con 0x7f6a4c1029f0 2024-04-21T08:48:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.607+0000 7f6a33fff640 1 --2- 172.21.15.28:0/2636185992 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6a340a3bb0 0x7f6a340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.607+0000 7f6a4a59c640 1 --2- 172.21.15.28:0/2636185992 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6a340a3bb0 0x7f6a340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.607+0000 7f6a33fff640 1 -- 172.21.15.28:0/2636185992 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6a3c0cfc50 con 0x7f6a4c1029f0 2024-04-21T08:48:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.607+0000 7f6a4a59c640 1 --2- 172.21.15.28:0/2636185992 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6a340a3bb0 0x7f6a340a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c199650 tx=0x7f6a4000a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:20.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.612+0000 7f6a33fff640 1 -- 172.21.15.28:0/2636185992 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a3c09ca30 con 0x7f6a4c1029f0 2024-04-21T08:48:20.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.844+0000 7f6a511a4640 1 -- 172.21.15.28:0/2636185992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a4c062b50 con 0x7f6a4c1029f0 2024-04-21T08:48:20.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.845+0000 7f6a33fff640 1 -- 172.21.15.28:0/2636185992 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7f6a3c009de0 con 0x7f6a4c1029f0 2024-04-21T08:48:20.845 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:20.845 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:48:20.845 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 5 2024-04-21T08:48:20.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.847+0000 7f6a511a4640 1 -- 172.21.15.28:0/2636185992 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6a340a3bb0 msgr2=0x7f6a340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:20.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.847+0000 7f6a511a4640 1 --2- 172.21.15.28:0/2636185992 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6a340a3bb0 0x7f6a340a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c199650 tx=0x7f6a4000a4c0 comp rx=0 tx=0).stop 2024-04-21T08:48:20.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.847+0000 7f6a511a4640 1 -- 172.21.15.28:0/2636185992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1029f0 msgr2=0x7f6a4c198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:20.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.847+0000 7f6a511a4640 1 --2- 172.21.15.28:0/2636185992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1029f0 0x7f6a4c198140 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c00d950 tx=0x7f6a3c00de10 comp rx=0 tx=0).stop 2024-04-21T08:48:20.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.847+0000 7f6a511a4640 1 -- 172.21.15.28:0/2636185992 shutdown_connections 2024-04-21T08:48:20.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.847+0000 7f6a511a4640 1 --2- 172.21.15.28:0/2636185992 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a4c107c50 0x7f6a4c19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:20.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.847+0000 7f6a511a4640 1 --2- 172.21.15.28:0/2636185992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6a4c105320 0x7f6a4c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:20.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.847+0000 7f6a511a4640 1 --2- 172.21.15.28:0/2636185992 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6a340a3bb0 0x7f6a340a6070 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:20.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.847+0000 7f6a511a4640 1 --2- 172.21.15.28:0/2636185992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a4c1029f0 0x7f6a4c198140 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:20.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.847+0000 7f6a511a4640 1 -- 172.21.15.28:0/2636185992 >> 172.21.15.28:0/2636185992 conn(0x7f6a4c0fc3e0 msgr2=0x7f6a4c1034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:20.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.847+0000 7f6a511a4640 1 -- 172.21.15.28:0/2636185992 shutdown_connections 2024-04-21T08:48:20.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:20.847+0000 7f6a511a4640 1 -- 172.21.15.28:0/2636185992 wait complete. 2024-04-21T08:48:21.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:20 smithi152 ceph-mon[31401]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:21.105 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 08:48:21 smithi152 systemd[1]: Starting Ceph mds.e for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T08:48:21.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:20 smithi028 ceph-mon[28003]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:21.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:20 smithi171 ceph-mon[31358]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:21.566 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 08:48:21 smithi152 podman[61589]: 2024-04-21 08:48:21.200859954 +0000 UTC m=+0.017850992 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:48:21.566 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 08:48:21 smithi152 podman[61589]: 2024-04-21 08:48:21.339366772 +0000 UTC m=+0.156357828 container create de6b52f61a578cd77fae01191559b8783c1ec8cf6f0bfeca8a9cdfc37f6a3208 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-21T08:48:21.566 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 08:48:21 smithi152 podman[61589]: 2024-04-21 08:48:21.567101035 +0000 UTC m=+0.384092052 container init de6b52f61a578cd77fae01191559b8783c1ec8cf6f0bfeca8a9cdfc37f6a3208 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-21T08:48:21.827 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 08:48:21 smithi152 podman[61589]: 2024-04-21 08:48:21.569847757 +0000 UTC m=+0.386838780 container start de6b52f61a578cd77fae01191559b8783c1ec8cf6f0bfeca8a9cdfc37f6a3208 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-e, RELEASE=wip-lusov-quiescer-a630765, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-21T08:48:21.827 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 08:48:21 smithi152 ceph-mds[61608]: -- 172.21.15.152:0/2349712146 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a575c6ba40 con 0x55a575c3ec00 2024-04-21T08:48:21.828 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 08:48:21 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-e[61604]: starting mds.e at 2024-04-21T08:48:21.828 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 08:48:21 smithi152 bash[61589]: de6b52f61a578cd77fae01191559b8783c1ec8cf6f0bfeca8a9cdfc37f6a3208 2024-04-21T08:48:21.828 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 08:48:21 smithi152 systemd[1]: Started Ceph mds.e for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:48:22.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:21 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2636185992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:22.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:21 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:22.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:21 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:22.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:21 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:22.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:21 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:22.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:21 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:22.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:21 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:22.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:21 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2636185992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:22.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:21 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:22.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:21 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:22.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:21 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:22.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:21 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:22.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:21 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:22.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:21 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:22.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:21 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2636185992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:22.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:22.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:22.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:22.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:22.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:22.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:21 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:22.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.974+0000 7f8a6fb2b640 1 -- 172.21.15.28:0/3145415334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a68103230 msgr2=0x7f8a681036b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.974+0000 7f8a6fb2b640 1 --2- 172.21.15.28:0/3145415334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a68103230 0x7f8a681036b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8a58009930 tx=0x7f8a5802bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.974+0000 7f8a6fb2b640 1 -- 172.21.15.28:0/3145415334 shutdown_connections 2024-04-21T08:48:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.974+0000 7f8a6fb2b640 1 --2- 172.21.15.28:0/3145415334 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a68103bf0 0x7f8a6810a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.974+0000 7f8a6fb2b640 1 --2- 172.21.15.28:0/3145415334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a68103230 0x7f8a681036b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.974+0000 7f8a6fb2b640 1 --2- 172.21.15.28:0/3145415334 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a68100260 0x7f8a68100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.974+0000 7f8a6fb2b640 1 -- 172.21.15.28:0/3145415334 >> 172.21.15.28:0/3145415334 conn(0x7f8a680fbb90 msgr2=0x7f8a680fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.974+0000 7f8a6fb2b640 1 -- 172.21.15.28:0/3145415334 shutdown_connections 2024-04-21T08:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.974+0000 7f8a6fb2b640 1 -- 172.21.15.28:0/3145415334 wait complete. 2024-04-21T08:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.975+0000 7f8a6fb2b640 1 Processor -- start 2024-04-21T08:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.975+0000 7f8a6fb2b640 1 -- start start 2024-04-21T08:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.975+0000 7f8a6fb2b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a68100260 0x7f8a681981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.975+0000 7f8a6fb2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a68103230 0x7f8a68198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.975+0000 7f8a6fb2b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a68103bf0 0x7f8a6819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.975+0000 7f8a6fb2b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6810db00 con 0x7f8a68103230 2024-04-21T08:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.975+0000 7f8a6fb2b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6810d980 con 0x7f8a68100260 2024-04-21T08:48:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.975+0000 7f8a6fb2b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6810dc80 con 0x7f8a68103bf0 2024-04-21T08:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.975+0000 7f8a6d0c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a68103230 0x7f8a68198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.975+0000 7f8a6d0c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a68103230 0x7f8a68198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45120/0 (socket says 172.21.15.28:45120) 2024-04-21T08:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.975+0000 7f8a6d0c6640 1 -- 172.21.15.28:0/3561932986 learned_addr learned my addr 172.21.15.28:0/3561932986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.975+0000 7f8a6d8c7640 1 --2- 172.21.15.28:0/3561932986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a68100260 0x7f8a681981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.975+0000 7f8a6e0c8640 1 --2- 172.21.15.28:0/3561932986 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a68103bf0 0x7f8a6819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.976+0000 7f8a6d0c6640 1 -- 172.21.15.28:0/3561932986 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a68103bf0 msgr2=0x7f8a6819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.976+0000 7f8a6d0c6640 1 --2- 172.21.15.28:0/3561932986 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a68103bf0 0x7f8a6819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.976+0000 7f8a6d0c6640 1 -- 172.21.15.28:0/3561932986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a68100260 msgr2=0x7f8a681981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.976+0000 7f8a6d0c6640 1 --2- 172.21.15.28:0/3561932986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a68100260 0x7f8a681981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.976+0000 7f8a6d0c6640 1 -- 172.21.15.28:0/3561932986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6819fe00 con 0x7f8a68103230 2024-04-21T08:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.976+0000 7f8a6d0c6640 1 --2- 172.21.15.28:0/3561932986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a68103230 0x7f8a68198720 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8a58009230 tx=0x7f8a58004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.976+0000 7f8a5effd640 1 -- 172.21.15.28:0/3561932986 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a58043020 con 0x7f8a68103230 2024-04-21T08:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.976+0000 7f8a5effd640 1 -- 172.21.15.28:0/3561932986 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a580052b0 con 0x7f8a68103230 2024-04-21T08:48:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.976+0000 7f8a6fb2b640 1 -- 172.21.15.28:0/3561932986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a681a0090 con 0x7f8a68103230 2024-04-21T08:48:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.976+0000 7f8a5effd640 1 -- 172.21.15.28:0/3561932986 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a58003c60 con 0x7f8a68103230 2024-04-21T08:48:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.976+0000 7f8a6fb2b640 1 -- 172.21.15.28:0/3561932986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a681a05e0 con 0x7f8a68103230 2024-04-21T08:48:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.977+0000 7f8a6fb2b640 1 -- 172.21.15.28:0/3561932986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a68109430 con 0x7f8a68103230 2024-04-21T08:48:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.978+0000 7f8a5effd640 1 -- 172.21.15.28:0/3561932986 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8a58046020 con 0x7f8a68103230 2024-04-21T08:48:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.979+0000 7f8a5effd640 1 --2- 172.21.15.28:0/3561932986 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8a540a3bb0 0x7f8a540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.979+0000 7f8a6d8c7640 1 --2- 172.21.15.28:0/3561932986 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8a540a3bb0 0x7f8a540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.979+0000 7f8a5effd640 1 -- 172.21.15.28:0/3561932986 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8a580f2520 con 0x7f8a68103230 2024-04-21T08:48:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.980+0000 7f8a6d8c7640 1 --2- 172.21.15.28:0/3561932986 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8a540a3bb0 0x7f8a540a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8a6806a480 tx=0x7f8a6400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:22.984+0000 7f8a5effd640 1 -- 172.21.15.28:0/3561932986 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8a5804d020 con 0x7f8a68103230 2024-04-21T08:48:23.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:22 smithi152 ceph-mon[31401]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:23.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:22 smithi152 ceph-mon[31401]: Deploying daemon mds.f on smithi171 2024-04-21T08:48:23.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:22 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.152:6834/3982885739,v1:172.21.15.152:6835/3982885739] up:boot 2024-04-21T08:48:23.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:22 smithi152 ceph-mon[31401]: fsmap 5 up:standby 2024-04-21T08:48:23.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:22 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:48:23.105 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:22 smithi171 ceph-mon[31358]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:23.105 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:22 smithi171 ceph-mon[31358]: Deploying daemon mds.f on smithi171 2024-04-21T08:48:23.105 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:22 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.152:6834/3982885739,v1:172.21.15.152:6835/3982885739] up:boot 2024-04-21T08:48:23.106 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:22 smithi171 ceph-mon[31358]: fsmap 5 up:standby 2024-04-21T08:48:23.106 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:22 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:48:23.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:22 smithi028 ceph-mon[28003]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:23.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:22 smithi028 ceph-mon[28003]: Deploying daemon mds.f on smithi171 2024-04-21T08:48:23.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:22 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.152:6834/3982885739,v1:172.21.15.152:6835/3982885739] up:boot 2024-04-21T08:48:23.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:22 smithi028 ceph-mon[28003]: fsmap 5 up:standby 2024-04-21T08:48:23.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:22 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:48:23.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:23.210+0000 7f8a6fb2b640 1 -- 172.21.15.28:0/3561932986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a68062b50 con 0x7f8a68103230 2024-04-21T08:48:23.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:23.211+0000 7f8a5effd640 1 -- 172.21.15.28:0/3561932986 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7f8a580bf300 con 0x7f8a68103230 2024-04-21T08:48:23.211 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:23.211 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-21T08:48:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 6 2024-04-21T08:48:23.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:23.213+0000 7f8a6fb2b640 1 -- 172.21.15.28:0/3561932986 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8a540a3bb0 msgr2=0x7f8a540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:23.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:23.213+0000 7f8a6fb2b640 1 --2- 172.21.15.28:0/3561932986 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8a540a3bb0 0x7f8a540a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8a6806a480 tx=0x7f8a6400b390 comp rx=0 tx=0).stop 2024-04-21T08:48:23.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:23.213+0000 7f8a6fb2b640 1 -- 172.21.15.28:0/3561932986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a68103230 msgr2=0x7f8a68198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:23.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:23.213+0000 7f8a6fb2b640 1 --2- 172.21.15.28:0/3561932986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a68103230 0x7f8a68198720 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8a58009230 tx=0x7f8a58004270 comp rx=0 tx=0).stop 2024-04-21T08:48:23.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:23.213+0000 7f8a6fb2b640 1 -- 172.21.15.28:0/3561932986 shutdown_connections 2024-04-21T08:48:23.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:23.213+0000 7f8a6fb2b640 1 --2- 172.21.15.28:0/3561932986 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a68103bf0 0x7f8a6819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:23.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:23.213+0000 7f8a6fb2b640 1 --2- 172.21.15.28:0/3561932986 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8a540a3bb0 0x7f8a540a6070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:23.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:23.213+0000 7f8a6fb2b640 1 --2- 172.21.15.28:0/3561932986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a68103230 0x7f8a68198720 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:23.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:23.213+0000 7f8a6fb2b640 1 --2- 172.21.15.28:0/3561932986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a68100260 0x7f8a681981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:23.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:23.213+0000 7f8a6fb2b640 1 -- 172.21.15.28:0/3561932986 >> 172.21.15.28:0/3561932986 conn(0x7f8a680fbb90 msgr2=0x7f8a68106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:23.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:23.213+0000 7f8a6fb2b640 1 -- 172.21.15.28:0/3561932986 shutdown_connections 2024-04-21T08:48:23.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:23.213+0000 7f8a6fb2b640 1 -- 172.21.15.28:0/3561932986 wait complete. 2024-04-21T08:48:24.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:23 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3561932986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:24.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:23 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3561932986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:24.110 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:23 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3561932986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:24.110 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 08:48:23 smithi171 systemd[1]: Starting Ceph mds.f for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T08:48:24.370 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 08:48:24 smithi171 podman[61493]: 2024-04-21 08:48:24.013685769 +0000 UTC m=+0.018531516 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:48:24.370 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 08:48:24 smithi171 podman[61493]: 2024-04-21 08:48:24.118003322 +0000 UTC m=+0.122849047 container create 29ece509f8b476830d0a6d40980f6648d209dffa5670a37217d2b880d755edf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:48:24.370 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 08:48:24 smithi171 podman[61493]: 2024-04-21 08:48:24.354121442 +0000 UTC m=+0.358967171 container init 29ece509f8b476830d0a6d40980f6648d209dffa5670a37217d2b880d755edf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux ) 2024-04-21T08:48:24.370 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 08:48:24 smithi171 podman[61493]: 2024-04-21 08:48:24.356715581 +0000 UTC m=+0.361561306 container start 29ece509f8b476830d0a6d40980f6648d209dffa5670a37217d2b880d755edf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-f, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:48:24.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 08:48:24 smithi171 ceph-mds[61512]: -- 172.21.15.171:0/573216629 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56409792da40 con 0x564097900c00 2024-04-21T08:48:24.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 08:48:24 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-f[61508]: starting mds.f at 2024-04-21T08:48:24.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 08:48:24 smithi171 bash[61493]: 29ece509f8b476830d0a6d40980f6648d209dffa5670a37217d2b880d755edf3 2024-04-21T08:48:24.841 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 08:48:24 smithi171 systemd[1]: Started Ceph mds.f for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:48:24.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:24 smithi171 ceph-mon[31358]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:24.842 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:24 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:25.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:24 smithi152 ceph-mon[31401]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:25.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:24 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:25.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:24 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:25.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:24 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:25.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:24 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:25.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:24 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:25.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:24 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:25.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:24 smithi028 ceph-mon[28003]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:25.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:24 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:25.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:24 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:25.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:24 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:25.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:24 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:25.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:24 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:25.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:24 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:25.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:24 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:25.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:24 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:25.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:24 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:25.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:24 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:25.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:24 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:25.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.339+0000 7fb05b94b640 1 -- 172.21.15.28:0/1061926336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb054107d00 msgr2=0x7fb05410a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.339+0000 7fb05b94b640 1 --2- 172.21.15.28:0/1061926336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb054107d00 0x7fb05410a0f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb04c009900 tx=0x7fb04c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.339+0000 7fb05b94b640 1 -- 172.21.15.28:0/1061926336 shutdown_connections 2024-04-21T08:48:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.339+0000 7fb05b94b640 1 --2- 172.21.15.28:0/1061926336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb054107d00 0x7fb05410a0f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.339+0000 7fb05b94b640 1 --2- 172.21.15.28:0/1061926336 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb0541053d0 0x7fb0541077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.339+0000 7fb05b94b640 1 --2- 172.21.15.28:0/1061926336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb054102aa0 0x7fb054104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.339+0000 7fb05b94b640 1 -- 172.21.15.28:0/1061926336 >> 172.21.15.28:0/1061926336 conn(0x7fb0540fc470 msgr2=0x7fb0540fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.339+0000 7fb05b94b640 1 -- 172.21.15.28:0/1061926336 shutdown_connections 2024-04-21T08:48:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.339+0000 7fb05b94b640 1 -- 172.21.15.28:0/1061926336 wait complete. 2024-04-21T08:48:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.340+0000 7fb05b94b640 1 Processor -- start 2024-04-21T08:48:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.340+0000 7fb05b94b640 1 -- start start 2024-04-21T08:48:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.340+0000 7fb05b94b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb054102aa0 0x7fb0541980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.340+0000 7fb05b94b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0541053d0 0x7fb0541985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.340+0000 7fb05b94b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb054107d00 0x7fb05419f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.340+0000 7fb05b94b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb05410db50 con 0x7fb054107d00 2024-04-21T08:48:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.340+0000 7fb05b94b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb05410d9d0 con 0x7fb0541053d0 2024-04-21T08:48:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.340+0000 7fb05b94b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb05410dcd0 con 0x7fb054102aa0 2024-04-21T08:48:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.340+0000 7fb059ee8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb054107d00 0x7fb05419f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.340+0000 7fb059ee8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb054107d00 0x7fb05419f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59154/0 (socket says 172.21.15.28:59154) 2024-04-21T08:48:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.340+0000 7fb059ee8640 1 -- 172.21.15.28:0/3514520892 learned_addr learned my addr 172.21.15.28:0/3514520892 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.340+0000 7fb0596e7640 1 --2- 172.21.15.28:0/3514520892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb054102aa0 0x7fb0541980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.341+0000 7fb058ee6640 1 --2- 172.21.15.28:0/3514520892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0541053d0 0x7fb0541985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.341+0000 7fb059ee8640 1 -- 172.21.15.28:0/3514520892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb054102aa0 msgr2=0x7fb0541980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.341+0000 7fb059ee8640 1 --2- 172.21.15.28:0/3514520892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb054102aa0 0x7fb0541980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.341+0000 7fb059ee8640 1 -- 172.21.15.28:0/3514520892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0541053d0 msgr2=0x7fb0541985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.341+0000 7fb059ee8640 1 --2- 172.21.15.28:0/3514520892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0541053d0 0x7fb0541985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.341+0000 7fb059ee8640 1 -- 172.21.15.28:0/3514520892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05419fcd0 con 0x7fb054107d00 2024-04-21T08:48:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.341+0000 7fb058ee6640 1 --2- 172.21.15.28:0/3514520892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0541053d0 0x7fb0541985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.341+0000 7fb059ee8640 1 --2- 172.21.15.28:0/3514520892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb054107d00 0x7fb05419f660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb04c036040 tx=0x7fb04c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.341+0000 7fb0527fc640 1 -- 172.21.15.28:0/3514520892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c043020 con 0x7fb054107d00 2024-04-21T08:48:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.341+0000 7fb0527fc640 1 -- 172.21.15.28:0/3514520892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb04c004490 con 0x7fb054107d00 2024-04-21T08:48:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.341+0000 7fb05b94b640 1 -- 172.21.15.28:0/3514520892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb05419ff60 con 0x7fb054107d00 2024-04-21T08:48:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.341+0000 7fb0527fc640 1 -- 172.21.15.28:0/3514520892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c003d10 con 0x7fb054107d00 2024-04-21T08:48:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.341+0000 7fb05b94b640 1 -- 172.21.15.28:0/3514520892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0541a0420 con 0x7fb054107d00 2024-04-21T08:48:25.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.342+0000 7fb05b94b640 1 -- 172.21.15.28:0/3514520892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0540fdf60 con 0x7fb054107d00 2024-04-21T08:48:25.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.343+0000 7fb0527fc640 1 -- 172.21.15.28:0/3514520892 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb04c046020 con 0x7fb054107d00 2024-04-21T08:48:25.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.344+0000 7fb0527fc640 1 --2- 172.21.15.28:0/3514520892 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb03c0a3bb0 0x7fb03c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:25.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.344+0000 7fb0596e7640 1 --2- 172.21.15.28:0/3514520892 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb03c0a3bb0 0x7fb03c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:25.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.344+0000 7fb0527fc640 1 -- 172.21.15.28:0/3514520892 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb04c0f22d0 con 0x7fb054107d00 2024-04-21T08:48:25.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.344+0000 7fb0596e7640 1 --2- 172.21.15.28:0/3514520892 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb03c0a3bb0 0x7fb03c0a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb044005fd0 tx=0x7fb044009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:25.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.349+0000 7fb0527fc640 1 -- 172.21.15.28:0/3514520892 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb04c04d020 con 0x7fb054107d00 2024-04-21T08:48:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.576+0000 7fb05b94b640 1 -- 172.21.15.28:0/3514520892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb05402d810 con 0x7fb054107d00 2024-04-21T08:48:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.576+0000 7fb0527fc640 1 -- 172.21.15.28:0/3514520892 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7fb04c0bf150 con 0x7fb054107d00 2024-04-21T08:48:25.576 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:25.576 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-21T08:48:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-04-21T08:48:25.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.578+0000 7fb05b94b640 1 -- 172.21.15.28:0/3514520892 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb03c0a3bb0 msgr2=0x7fb03c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.578+0000 7fb05b94b640 1 --2- 172.21.15.28:0/3514520892 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb03c0a3bb0 0x7fb03c0a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb044005fd0 tx=0x7fb044009680 comp rx=0 tx=0).stop 2024-04-21T08:48:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.578+0000 7fb05b94b640 1 -- 172.21.15.28:0/3514520892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb054107d00 msgr2=0x7fb05419f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.578+0000 7fb05b94b640 1 --2- 172.21.15.28:0/3514520892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb054107d00 0x7fb05419f660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb04c036040 tx=0x7fb04c004270 comp rx=0 tx=0).stop 2024-04-21T08:48:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.579+0000 7fb05b94b640 1 -- 172.21.15.28:0/3514520892 shutdown_connections 2024-04-21T08:48:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.579+0000 7fb05b94b640 1 --2- 172.21.15.28:0/3514520892 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb03c0a3bb0 0x7fb03c0a6070 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.579+0000 7fb05b94b640 1 --2- 172.21.15.28:0/3514520892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb054107d00 0x7fb05419f660 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.579+0000 7fb05b94b640 1 --2- 172.21.15.28:0/3514520892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0541053d0 0x7fb0541985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.579+0000 7fb05b94b640 1 --2- 172.21.15.28:0/3514520892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb054102aa0 0x7fb0541980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.579+0000 7fb05b94b640 1 -- 172.21.15.28:0/3514520892 >> 172.21.15.28:0/3514520892 conn(0x7fb0540fc470 msgr2=0x7fb0541033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.579+0000 7fb05b94b640 1 -- 172.21.15.28:0/3514520892 shutdown_connections 2024-04-21T08:48:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:25.579+0000 7fb05b94b640 1 -- 172.21.15.28:0/3514520892 wait complete. 2024-04-21T08:48:26.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:25 smithi152 ceph-mon[31401]: Deploying daemon mds.c on smithi171 2024-04-21T08:48:26.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:25 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.171:6834/663606808,v1:172.21.15.171:6835/663606808] up:boot 2024-04-21T08:48:26.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:25 smithi152 ceph-mon[31401]: fsmap 6 up:standby 2024-04-21T08:48:26.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:25 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:48:26.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:25 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3514520892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:26.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:25 smithi028 ceph-mon[28003]: Deploying daemon mds.c on smithi171 2024-04-21T08:48:26.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:25 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.171:6834/663606808,v1:172.21.15.171:6835/663606808] up:boot 2024-04-21T08:48:26.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:25 smithi028 ceph-mon[28003]: fsmap 6 up:standby 2024-04-21T08:48:26.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:25 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:48:26.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:25 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3514520892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:26.111 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:25 smithi171 ceph-mon[31358]: Deploying daemon mds.c on smithi171 2024-04-21T08:48:26.111 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:25 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.171:6834/663606808,v1:172.21.15.171:6835/663606808] up:boot 2024-04-21T08:48:26.111 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:25 smithi171 ceph-mon[31358]: fsmap 6 up:standby 2024-04-21T08:48:26.111 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:25 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:48:26.111 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:25 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3514520892' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:26.789 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 08:48:26 smithi171 systemd[1]: Starting Ceph mds.c for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T08:48:27.083 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:26 smithi171 ceph-mon[31358]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:27.084 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 08:48:26 smithi171 podman[61801]: 2024-04-21 08:48:26.693341164 +0000 UTC m=+0.018418805 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:48:27.084 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 08:48:26 smithi171 podman[61801]: 2024-04-21 08:48:26.823725686 +0000 UTC m=+0.148803327 container create 2430e65ae0c66903fddc2872f0721b43fa406ba559f74b358e819a5ebecbd72c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-c, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:48:27.084 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 08:48:27 smithi171 podman[61801]: 2024-04-21 08:48:27.084783942 +0000 UTC m=+0.409861583 container init 2430e65ae0c66903fddc2872f0721b43fa406ba559f74b358e819a5ebecbd72c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-21T08:48:27.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:26 smithi152 ceph-mon[31401]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:27.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:26 smithi028 ceph-mon[28003]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:27.370 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 08:48:27 smithi171 podman[61801]: 2024-04-21 08:48:27.087589598 +0000 UTC m=+0.412667248 container start 2430e65ae0c66903fddc2872f0721b43fa406ba559f74b358e819a5ebecbd72c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-21T08:48:27.370 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 08:48:27 smithi171 ceph-mds[61820]: -- 172.21.15.171:0/4187748558 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5581bb68ba40 con 0x5581bb65f000 2024-04-21T08:48:27.370 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 08:48:27 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-c[61816]: starting mds.c at 2024-04-21T08:48:27.370 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 08:48:27 smithi171 bash[61801]: 2430e65ae0c66903fddc2872f0721b43fa406ba559f74b358e819a5ebecbd72c 2024-04-21T08:48:27.370 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 08:48:27 smithi171 systemd[1]: Started Ceph mds.c for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:48:27.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.694+0000 7f8ee5ab3640 1 -- 172.21.15.28:0/949425922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0102ab0 msgr2=0x7f8ee0104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.694+0000 7f8ee5ab3640 1 --2- 172.21.15.28:0/949425922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0102ab0 0x7f8ee0104ea0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009860 tx=0x7f8ed002bb70 comp rx=0 tx=0).stop 2024-04-21T08:48:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.695+0000 7f8ee5ab3640 1 -- 172.21.15.28:0/949425922 shutdown_connections 2024-04-21T08:48:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.695+0000 7f8ee5ab3640 1 --2- 172.21.15.28:0/949425922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ee0107d10 0x7f8ee010a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.695+0000 7f8ee5ab3640 1 --2- 172.21.15.28:0/949425922 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ee01053e0 0x7f8ee01077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.695+0000 7f8ee5ab3640 1 --2- 172.21.15.28:0/949425922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0102ab0 0x7f8ee0104ea0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.695+0000 7f8ee5ab3640 1 -- 172.21.15.28:0/949425922 >> 172.21.15.28:0/949425922 conn(0x7f8ee00fc3e0 msgr2=0x7f8ee00fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.695+0000 7f8ee5ab3640 1 -- 172.21.15.28:0/949425922 shutdown_connections 2024-04-21T08:48:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.695+0000 7f8ee5ab3640 1 -- 172.21.15.28:0/949425922 wait complete. 2024-04-21T08:48:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.695+0000 7f8ee5ab3640 1 Processor -- start 2024-04-21T08:48:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.695+0000 7f8ee5ab3640 1 -- start start 2024-04-21T08:48:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8ee5ab3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0102ab0 0x7f8ee0198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8ee5ab3640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ee01053e0 0x7f8ee0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:27.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8ee5ab3640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ee0107d10 0x7f8ee019f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:27.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8ee5ab3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee010da60 con 0x7f8ee0102ab0 2024-04-21T08:48:27.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8ee5ab3640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee010d8e0 con 0x7f8ee01053e0 2024-04-21T08:48:27.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8ee5ab3640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee010dbe0 con 0x7f8ee0107d10 2024-04-21T08:48:27.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8edf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0102ab0 0x7f8ee0198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:27.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8edf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0102ab0 0x7f8ee0198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59176/0 (socket says 172.21.15.28:59176) 2024-04-21T08:48:27.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8edf7fe640 1 -- 172.21.15.28:0/4160795021 learned_addr learned my addr 172.21.15.28:0/4160795021 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:27.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8edeffd640 1 --2- 172.21.15.28:0/4160795021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ee01053e0 0x7f8ee0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:27.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8edffff640 1 --2- 172.21.15.28:0/4160795021 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ee0107d10 0x7f8ee019f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:27.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8edf7fe640 1 -- 172.21.15.28:0/4160795021 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ee0107d10 msgr2=0x7f8ee019f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:27.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8edf7fe640 1 --2- 172.21.15.28:0/4160795021 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ee0107d10 0x7f8ee019f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:27.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8edf7fe640 1 -- 172.21.15.28:0/4160795021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ee01053e0 msgr2=0x7f8ee0198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:27.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8edf7fe640 1 --2- 172.21.15.28:0/4160795021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ee01053e0 0x7f8ee0198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:27.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8edf7fe640 1 -- 172.21.15.28:0/4160795021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee019fe20 con 0x7f8ee0102ab0 2024-04-21T08:48:27.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8edf7fe640 1 --2- 172.21.15.28:0/4160795021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0102ab0 0x7f8ee0198200 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0005940 tx=0x7f8ed00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8edcff9640 1 -- 172.21.15.28:0/4160795021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed0041020 con 0x7f8ee0102ab0 2024-04-21T08:48:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8edcff9640 1 -- 172.21.15.28:0/4160795021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ed00035f0 con 0x7f8ee0102ab0 2024-04-21T08:48:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8ee5ab3640 1 -- 172.21.15.28:0/4160795021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee01a00b0 con 0x7f8ee0102ab0 2024-04-21T08:48:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.696+0000 7f8edcff9640 1 -- 172.21.15.28:0/4160795021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed0039040 con 0x7f8ee0102ab0 2024-04-21T08:48:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.697+0000 7f8ee5ab3640 1 -- 172.21.15.28:0/4160795021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee01a05a0 con 0x7f8ee0102ab0 2024-04-21T08:48:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.698+0000 7f8ee5ab3640 1 -- 172.21.15.28:0/4160795021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ee00fded0 con 0x7f8ee0102ab0 2024-04-21T08:48:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.699+0000 7f8edcff9640 1 -- 172.21.15.28:0/4160795021 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8ed0044020 con 0x7f8ee0102ab0 2024-04-21T08:48:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.699+0000 7f8edcff9640 1 --2- 172.21.15.28:0/4160795021 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8ec80a3bb0 0x7f8ec80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.699+0000 7f8edeffd640 1 --2- 172.21.15.28:0/4160795021 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8ec80a3bb0 0x7f8ec80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.699+0000 7f8edcff9640 1 -- 172.21.15.28:0/4160795021 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8ed00f19c0 con 0x7f8ee0102ab0 2024-04-21T08:48:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.700+0000 7f8edeffd640 1 --2- 172.21.15.28:0/4160795021 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8ec80a3bb0 0x7f8ec80a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0199710 tx=0x7f8ed4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.704+0000 7f8edcff9640 1 -- 172.21.15.28:0/4160795021 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ed00f3e20 con 0x7f8ee0102ab0 2024-04-21T08:48:27.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.968+0000 7f8ee5ab3640 1 -- 172.21.15.28:0/4160795021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ee0062b50 con 0x7f8ee0102ab0 2024-04-21T08:48:27.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.968+0000 7f8edcff9640 1 -- 172.21.15.28:0/4160795021 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7f8ed004b020 con 0x7f8ee0102ab0 2024-04-21T08:48:27.968 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:27.968 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-21T08:48:27.968 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-04-21T08:48:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.970+0000 7f8ee5ab3640 1 -- 172.21.15.28:0/4160795021 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8ec80a3bb0 msgr2=0x7f8ec80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.970+0000 7f8ee5ab3640 1 --2- 172.21.15.28:0/4160795021 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8ec80a3bb0 0x7f8ec80a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0199710 tx=0x7f8ed4005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.970+0000 7f8ee5ab3640 1 -- 172.21.15.28:0/4160795021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0102ab0 msgr2=0x7f8ee0198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.970+0000 7f8ee5ab3640 1 --2- 172.21.15.28:0/4160795021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0102ab0 0x7f8ee0198200 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0005940 tx=0x7f8ed00042f0 comp rx=0 tx=0).stop 2024-04-21T08:48:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.970+0000 7f8ee5ab3640 1 -- 172.21.15.28:0/4160795021 shutdown_connections 2024-04-21T08:48:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.970+0000 7f8ee5ab3640 1 --2- 172.21.15.28:0/4160795021 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ee0107d10 0x7f8ee019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.970+0000 7f8ee5ab3640 1 --2- 172.21.15.28:0/4160795021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ee01053e0 0x7f8ee0198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.970+0000 7f8ee5ab3640 1 --2- 172.21.15.28:0/4160795021 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8ec80a3bb0 0x7f8ec80a6070 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.970+0000 7f8ee5ab3640 1 --2- 172.21.15.28:0/4160795021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ee0102ab0 0x7f8ee0198200 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.970+0000 7f8ee5ab3640 1 -- 172.21.15.28:0/4160795021 >> 172.21.15.28:0/4160795021 conn(0x7f8ee00fc3e0 msgr2=0x7f8ee0104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.971+0000 7f8ee5ab3640 1 -- 172.21.15.28:0/4160795021 shutdown_connections 2024-04-21T08:48:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:27.971+0000 7f8ee5ab3640 1 -- 172.21.15.28:0/4160795021 wait complete. 2024-04-21T08:48:28.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:28.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:28.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:28.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:28.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:28.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:28.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:28 smithi028 ceph-mon[28003]: Deploying daemon mds.h on smithi152 2024-04-21T08:48:28.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:28 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] up:boot 2024-04-21T08:48:28.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:28 smithi028 ceph-mon[28003]: fsmap 7 up:standby 2024-04-21T08:48:28.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:28 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:48:28.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:28 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/4160795021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:28.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:28.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:28.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:28.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:28.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:28.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:28.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:28 smithi152 ceph-mon[31401]: Deploying daemon mds.h on smithi152 2024-04-21T08:48:28.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:28 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] up:boot 2024-04-21T08:48:28.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:28 smithi152 ceph-mon[31401]: fsmap 7 up:standby 2024-04-21T08:48:28.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:28 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:48:28.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:28 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/4160795021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:28 smithi171 ceph-mon[31358]: Deploying daemon mds.h on smithi152 2024-04-21T08:48:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:28 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] up:boot 2024-04-21T08:48:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:28 smithi171 ceph-mon[31358]: fsmap 7 up:standby 2024-04-21T08:48:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:28 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:48:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:28 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/4160795021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:29.539 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:29 smithi152 ceph-mon[31401]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:29.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:29 smithi028 ceph-mon[28003]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:29.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:29 smithi171 ceph-mon[31358]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:29.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:30.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.098+0000 7f2b85e4e640 1 -- 172.21.15.28:0/298608351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b80102a10 msgr2=0x7f2b80109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:30.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.098+0000 7f2b85e4e640 1 --2- 172.21.15.28:0/298608351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b80102a10 0x7f2b80109330 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009ca0 tx=0x7f2b7802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:30.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.098+0000 7f2b85e4e640 1 -- 172.21.15.28:0/298608351 shutdown_connections 2024-04-21T08:48:30.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.098+0000 7f2b85e4e640 1 --2- 172.21.15.28:0/298608351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b80102a10 0x7f2b80109330 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:30.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.098+0000 7f2b85e4e640 1 --2- 172.21.15.28:0/298608351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b80102050 0x7f2b801024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.098+0000 7f2b85e4e640 1 --2- 172.21.15.28:0/298608351 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b80100e70 0x7f2b80101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.098+0000 7f2b85e4e640 1 -- 172.21.15.28:0/298608351 >> 172.21.15.28:0/298608351 conn(0x7f2b800fc470 msgr2=0x7f2b800fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.098+0000 7f2b85e4e640 1 -- 172.21.15.28:0/298608351 shutdown_connections 2024-04-21T08:48:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.098+0000 7f2b85e4e640 1 -- 172.21.15.28:0/298608351 wait complete. 2024-04-21T08:48:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.099+0000 7f2b85e4e640 1 Processor -- start 2024-04-21T08:48:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.099+0000 7f2b85e4e640 1 -- start start 2024-04-21T08:48:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.099+0000 7f2b85e4e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b80100e70 0x7f2b80198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.099+0000 7f2b85e4e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b80102050 0x7f2b801986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.099+0000 7f2b85e4e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b80102a10 0x7f2b8019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.099+0000 7f2b85e4e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8010ca00 con 0x7f2b80102050 2024-04-21T08:48:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.099+0000 7f2b85e4e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8010c880 con 0x7f2b80102a10 2024-04-21T08:48:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.099+0000 7f2b85e4e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8010cb80 con 0x7f2b80100e70 2024-04-21T08:48:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.099+0000 7f2b7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b80102050 0x7f2b801986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.099+0000 7f2b7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b80102050 0x7f2b801986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59206/0 (socket says 172.21.15.28:59206) 2024-04-21T08:48:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.099+0000 7f2b7effd640 1 -- 172.21.15.28:0/1479633555 learned_addr learned my addr 172.21.15.28:0/1479633555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:30.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.099+0000 7f2b7ffff640 1 --2- 172.21.15.28:0/1479633555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b80102a10 0x7f2b8019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:30.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.100+0000 7f2b7f7fe640 1 --2- 172.21.15.28:0/1479633555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b80100e70 0x7f2b80198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:30.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.100+0000 7f2b7effd640 1 -- 172.21.15.28:0/1479633555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b80100e70 msgr2=0x7f2b80198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:30.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.100+0000 7f2b7effd640 1 --2- 172.21.15.28:0/1479633555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b80100e70 0x7f2b80198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:30.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.100+0000 7f2b7effd640 1 -- 172.21.15.28:0/1479633555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b80102a10 msgr2=0x7f2b8019f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:30.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.100+0000 7f2b7effd640 1 --2- 172.21.15.28:0/1479633555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b80102a10 0x7f2b8019f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:30.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.100+0000 7f2b7effd640 1 -- 172.21.15.28:0/1479633555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8019fd80 con 0x7f2b80102050 2024-04-21T08:48:30.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.100+0000 7f2b7f7fe640 1 --2- 172.21.15.28:0/1479633555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b80100e70 0x7f2b80198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:30.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.100+0000 7f2b7effd640 1 --2- 172.21.15.28:0/1479633555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b80102050 0x7f2b801986a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000e9f0 tx=0x7f2b7000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:30.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.100+0000 7f2b7cff9640 1 -- 172.21.15.28:0/1479633555 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7000cd80 con 0x7f2b80102050 2024-04-21T08:48:30.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.100+0000 7f2b7cff9640 1 -- 172.21.15.28:0/1479633555 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b700108e0 con 0x7f2b80102050 2024-04-21T08:48:30.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.100+0000 7f2b85e4e640 1 -- 172.21.15.28:0/1479633555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b801a0070 con 0x7f2b80102050 2024-04-21T08:48:30.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.100+0000 7f2b7cff9640 1 -- 172.21.15.28:0/1479633555 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b70018980 con 0x7f2b80102050 2024-04-21T08:48:30.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.100+0000 7f2b85e4e640 1 -- 172.21.15.28:0/1479633555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b801a05c0 con 0x7f2b80102050 2024-04-21T08:48:30.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.101+0000 7f2b85e4e640 1 -- 172.21.15.28:0/1479633555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b801a0820 con 0x7f2b80102050 2024-04-21T08:48:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.102+0000 7f2b7cff9640 1 -- 172.21.15.28:0/1479633555 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2b70010430 con 0x7f2b80102050 2024-04-21T08:48:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.103+0000 7f2b7cff9640 1 --2- 172.21.15.28:0/1479633555 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2b680a3bb0 0x7f2b680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.103+0000 7f2b7f7fe640 1 --2- 172.21.15.28:0/1479633555 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2b680a3bb0 0x7f2b680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.103+0000 7f2b7cff9640 1 -- 172.21.15.28:0/1479633555 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2b700d0880 con 0x7f2b80102050 2024-04-21T08:48:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.103+0000 7f2b7f7fe640 1 --2- 172.21.15.28:0/1479633555 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2b680a3bb0 0x7f2b680a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2b740058c0 tx=0x7f2b74005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.107+0000 7f2b7cff9640 1 -- 172.21.15.28:0/1479633555 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b7009d660 con 0x7f2b80102050 2024-04-21T08:48:30.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.334+0000 7f2b85e4e640 1 -- 172.21.15.28:0/1479633555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b801a09f0 con 0x7f2b80102050 2024-04-21T08:48:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.334+0000 7f2b7cff9640 1 -- 172.21.15.28:0/1479633555 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7180 (secure 0 0 0) 0x7f2b7009cf60 con 0x7f2b80102050 2024-04-21T08:48:30.334 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:30.334 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-21T08:48:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-04-21T08:48:30.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.336+0000 7f2b85e4e640 1 -- 172.21.15.28:0/1479633555 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2b680a3bb0 msgr2=0x7f2b680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:30.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.336+0000 7f2b85e4e640 1 --2- 172.21.15.28:0/1479633555 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2b680a3bb0 0x7f2b680a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2b740058c0 tx=0x7f2b74005850 comp rx=0 tx=0).stop 2024-04-21T08:48:30.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.336+0000 7f2b85e4e640 1 -- 172.21.15.28:0/1479633555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b80102050 msgr2=0x7f2b801986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:30.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.336+0000 7f2b85e4e640 1 --2- 172.21.15.28:0/1479633555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b80102050 0x7f2b801986a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000e9f0 tx=0x7f2b7000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:48:30.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.336+0000 7f2b85e4e640 1 -- 172.21.15.28:0/1479633555 shutdown_connections 2024-04-21T08:48:30.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.336+0000 7f2b85e4e640 1 --2- 172.21.15.28:0/1479633555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b80102a10 0x7f2b8019f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:30.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.336+0000 7f2b85e4e640 1 --2- 172.21.15.28:0/1479633555 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2b680a3bb0 0x7f2b680a6070 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:30.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.336+0000 7f2b85e4e640 1 --2- 172.21.15.28:0/1479633555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b80102050 0x7f2b801986a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:30.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.336+0000 7f2b85e4e640 1 --2- 172.21.15.28:0/1479633555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b80100e70 0x7f2b80198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:30.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.336+0000 7f2b85e4e640 1 -- 172.21.15.28:0/1479633555 >> 172.21.15.28:0/1479633555 conn(0x7f2b800fc470 msgr2=0x7f2b800fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:30.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.337+0000 7f2b85e4e640 1 -- 172.21.15.28:0/1479633555 shutdown_connections 2024-04-21T08:48:30.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:30.337+0000 7f2b85e4e640 1 -- 172.21.15.28:0/1479633555 wait complete. 2024-04-21T08:48:30.354 INFO:journalctl@ceph.mds.h.smithi152.stdout:Apr 21 08:48:30 smithi152 systemd[1]: Started Ceph mds.h for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:48:31.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:31 smithi152 ceph-mon[31401]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:31 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:31 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:31 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.152:6836/1261226890,v1:172.21.15.152:6837/1261226890] up:boot 2024-04-21T08:48:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:31 smithi152 ceph-mon[31401]: fsmap 8 up:standby 2024-04-21T08:48:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:31 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:48:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:31 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:31 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:31 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:31 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:31 smithi152 ceph-mon[31401]: Deploying daemon mds.i on smithi171 2024-04-21T08:48:31.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:31 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1479633555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:31 smithi028 ceph-mon[28003]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:31 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:31 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:31 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.152:6836/1261226890,v1:172.21.15.152:6837/1261226890] up:boot 2024-04-21T08:48:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:31 smithi028 ceph-mon[28003]: fsmap 8 up:standby 2024-04-21T08:48:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:31 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:48:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:31 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:31 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:31 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:31 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:31 smithi028 ceph-mon[28003]: Deploying daemon mds.i on smithi171 2024-04-21T08:48:31.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:31 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1479633555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:31 smithi171 ceph-mon[31358]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:31 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:31 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:31 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.152:6836/1261226890,v1:172.21.15.152:6837/1261226890] up:boot 2024-04-21T08:48:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:31 smithi171 ceph-mon[31358]: fsmap 8 up:standby 2024-04-21T08:48:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:31 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:48:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:31 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:31 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:31 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:31 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:31 smithi171 ceph-mon[31358]: Deploying daemon mds.i on smithi171 2024-04-21T08:48:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:31 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1479633555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:32.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:32.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.463+0000 7efde3ec1640 1 -- 172.21.15.28:0/2571393157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efddc100830 msgr2=0x7efddc100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.463+0000 7efde3ec1640 1 --2- 172.21.15.28:0/2571393157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efddc100830 0x7efddc100c50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efdd8009900 tx=0x7efdd802bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.464+0000 7efde3ec1640 1 -- 172.21.15.28:0/2571393157 shutdown_connections 2024-04-21T08:48:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.464+0000 7efde3ec1640 1 --2- 172.21.15.28:0/2571393157 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efddc101b30 0x7efddc101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.464+0000 7efde3ec1640 1 --2- 172.21.15.28:0/2571393157 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efddc101190 0x7efddc1015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.464+0000 7efde3ec1640 1 --2- 172.21.15.28:0/2571393157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efddc100830 0x7efddc100c50 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.464+0000 7efde3ec1640 1 -- 172.21.15.28:0/2571393157 >> 172.21.15.28:0/2571393157 conn(0x7efddc0fc470 msgr2=0x7efddc0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.464+0000 7efde3ec1640 1 -- 172.21.15.28:0/2571393157 shutdown_connections 2024-04-21T08:48:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.464+0000 7efde3ec1640 1 -- 172.21.15.28:0/2571393157 wait complete. 2024-04-21T08:48:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.464+0000 7efde3ec1640 1 Processor -- start 2024-04-21T08:48:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.464+0000 7efde3ec1640 1 -- start start 2024-04-21T08:48:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.464+0000 7efde3ec1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efddc100830 0x7efddc19a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.464+0000 7efde3ec1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efddc101190 0x7efddc19a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.464+0000 7efde3ec1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efddc101b30 0x7efddc1a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.464+0000 7efde3ec1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efddc10fd10 con 0x7efddc100830 2024-04-21T08:48:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.464+0000 7efde3ec1640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efddc10fb90 con 0x7efddc101b30 2024-04-21T08:48:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.464+0000 7efde3ec1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7efddc10fe90 con 0x7efddc101190 2024-04-21T08:48:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efde1c5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efddc100830 0x7efddc19a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efde1c5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efddc100830 0x7efddc19a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59230/0 (socket says 172.21.15.28:59230) 2024-04-21T08:48:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efde1c5d640 1 -- 172.21.15.28:0/3639999209 learned_addr learned my addr 172.21.15.28:0/3639999209 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efde145c640 1 --2- 172.21.15.28:0/3639999209 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efddc101190 0x7efddc19a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efde245e640 1 --2- 172.21.15.28:0/3639999209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efddc101b30 0x7efddc1a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efde1c5d640 1 -- 172.21.15.28:0/3639999209 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efddc101190 msgr2=0x7efddc19a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efde1c5d640 1 --2- 172.21.15.28:0/3639999209 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efddc101190 0x7efddc19a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efde1c5d640 1 -- 172.21.15.28:0/3639999209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efddc101b30 msgr2=0x7efddc1a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efde1c5d640 1 --2- 172.21.15.28:0/3639999209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efddc101b30 0x7efddc1a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efde1c5d640 1 -- 172.21.15.28:0/3639999209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efddc1a1f60 con 0x7efddc100830 2024-04-21T08:48:32.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efde1c5d640 1 --2- 172.21.15.28:0/3639999209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efddc100830 0x7efddc19a340 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efdd8006010 tx=0x7efdd80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:32.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efdd2ffd640 1 -- 172.21.15.28:0/3639999209 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd8043020 con 0x7efddc100830 2024-04-21T08:48:32.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efdd2ffd640 1 -- 172.21.15.28:0/3639999209 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdd8004aa0 con 0x7efddc100830 2024-04-21T08:48:32.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efde3ec1640 1 -- 172.21.15.28:0/3639999209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efddc1a21f0 con 0x7efddc100830 2024-04-21T08:48:32.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efdd2ffd640 1 -- 172.21.15.28:0/3639999209 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd8003c70 con 0x7efddc100830 2024-04-21T08:48:32.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.465+0000 7efde3ec1640 1 -- 172.21.15.28:0/3639999209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efddc1a2740 con 0x7efddc100830 2024-04-21T08:48:32.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.467+0000 7efde3ec1640 1 -- 172.21.15.28:0/3639999209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efddc10b710 con 0x7efddc100830 2024-04-21T08:48:32.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.468+0000 7efdd2ffd640 1 -- 172.21.15.28:0/3639999209 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efdd8046020 con 0x7efddc100830 2024-04-21T08:48:32.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.469+0000 7efdd2ffd640 1 --2- 172.21.15.28:0/3639999209 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efdc80a3bb0 0x7efdc80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:32.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.469+0000 7efde145c640 1 --2- 172.21.15.28:0/3639999209 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efdc80a3bb0 0x7efdc80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:32.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.469+0000 7efdd2ffd640 1 -- 172.21.15.28:0/3639999209 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7efdd80f2240 con 0x7efddc100830 2024-04-21T08:48:32.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.469+0000 7efde145c640 1 --2- 172.21.15.28:0/3639999209 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efdc80a3bb0 0x7efdc80a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efddc19b850 tx=0x7efdcc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:32.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.473+0000 7efdd2ffd640 1 -- 172.21.15.28:0/3639999209 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efdd804d020 con 0x7efddc100830 2024-04-21T08:48:32.620 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 08:48:32 smithi171 systemd[1]: Starting Ceph mds.i for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T08:48:32.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.698+0000 7efde3ec1640 1 -- 172.21.15.28:0/3639999209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efddc02d5c0 con 0x7efddc100830 2024-04-21T08:48:32.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.699+0000 7efdd2ffd640 1 -- 172.21.15.28:0/3639999209 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7180 (secure 0 0 0) 0x7efdd80bf0c0 con 0x7efddc100830 2024-04-21T08:48:32.698 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:32.699 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-21T08:48:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-04-21T08:48:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.701+0000 7efde3ec1640 1 -- 172.21.15.28:0/3639999209 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efdc80a3bb0 msgr2=0x7efdc80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.701+0000 7efde3ec1640 1 --2- 172.21.15.28:0/3639999209 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efdc80a3bb0 0x7efdc80a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efddc19b850 tx=0x7efdcc00b410 comp rx=0 tx=0).stop 2024-04-21T08:48:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.701+0000 7efde3ec1640 1 -- 172.21.15.28:0/3639999209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efddc100830 msgr2=0x7efddc19a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.701+0000 7efde3ec1640 1 --2- 172.21.15.28:0/3639999209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efddc100830 0x7efddc19a340 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efdd8006010 tx=0x7efdd80042f0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.701+0000 7efde3ec1640 1 -- 172.21.15.28:0/3639999209 shutdown_connections 2024-04-21T08:48:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.701+0000 7efde3ec1640 1 --2- 172.21.15.28:0/3639999209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efddc101b30 0x7efddc1a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.701+0000 7efde3ec1640 1 --2- 172.21.15.28:0/3639999209 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efddc101190 0x7efddc19a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.701+0000 7efde3ec1640 1 --2- 172.21.15.28:0/3639999209 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efdc80a3bb0 0x7efdc80a6070 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.701+0000 7efde3ec1640 1 --2- 172.21.15.28:0/3639999209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efddc100830 0x7efddc19a340 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.701+0000 7efde3ec1640 1 -- 172.21.15.28:0/3639999209 >> 172.21.15.28:0/3639999209 conn(0x7efddc0fc470 msgr2=0x7efddc0fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.701+0000 7efde3ec1640 1 -- 172.21.15.28:0/3639999209 shutdown_connections 2024-04-21T08:48:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:32.701+0000 7efde3ec1640 1 -- 172.21.15.28:0/3639999209 wait complete. 2024-04-21T08:48:32.906 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 08:48:32 smithi171 podman[62113]: 2024-04-21 08:48:32.524048656 +0000 UTC m=+0.017682075 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:48:32.906 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 08:48:32 smithi171 podman[62113]: 2024-04-21 08:48:32.645960068 +0000 UTC m=+0.139593475 container create 6273c894fe9f37e740b8f4aeb77df55307265238dfc791bb713096fe424a0fdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-i, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-21T08:48:32.906 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 08:48:32 smithi171 podman[62113]: 2024-04-21 08:48:32.907445038 +0000 UTC m=+0.401078436 container init 6273c894fe9f37e740b8f4aeb77df55307265238dfc791bb713096fe424a0fdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-i, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:48:33.246 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 08:48:32 smithi171 podman[62113]: 2024-04-21 08:48:32.91022208 +0000 UTC m=+0.403855480 container start 6273c894fe9f37e740b8f4aeb77df55307265238dfc791bb713096fe424a0fdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0) 2024-04-21T08:48:33.246 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 08:48:32 smithi171 ceph-mds[62132]: -- 172.21.15.171:0/4029503769 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564b81175a40 con 0x564b81149000 2024-04-21T08:48:33.246 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 08:48:32 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-i[62128]: starting mds.i at 2024-04-21T08:48:33.246 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 08:48:32 smithi171 bash[62113]: 6273c894fe9f37e740b8f4aeb77df55307265238dfc791bb713096fe424a0fdb 2024-04-21T08:48:33.247 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 08:48:33 smithi171 systemd[1]: Started Ceph mds.i for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:48:33.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:33 smithi152 ceph-mon[31401]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:33.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:33 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3639999209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:33.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:33.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:33.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:33.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:33.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:33.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:33 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:33.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:33 smithi028 ceph-mon[28003]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:33.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:33 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3639999209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:33.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:33.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:33.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:33.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:33.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:33.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:33 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:33.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:33 smithi171 ceph-mon[31358]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:33.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:33 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3639999209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:33.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:33.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:33.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:33.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:33.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:33.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:33 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:34.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:34 smithi152 ceph-mon[31401]: Deploying daemon mds.g on smithi028 2024-04-21T08:48:34.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:34 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] up:boot 2024-04-21T08:48:34.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:34 smithi152 ceph-mon[31401]: fsmap 9 up:standby 2024-04-21T08:48:34.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:34 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:48:34.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:34 smithi028 ceph-mon[28003]: Deploying daemon mds.g on smithi028 2024-04-21T08:48:34.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:34 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] up:boot 2024-04-21T08:48:34.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:34 smithi028 ceph-mon[28003]: fsmap 9 up:standby 2024-04-21T08:48:34.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:34 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:48:34.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:34 smithi171 ceph-mon[31358]: Deploying daemon mds.g on smithi028 2024-04-21T08:48:34.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:34 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] up:boot 2024-04-21T08:48:34.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:34 smithi171 ceph-mon[31358]: fsmap 9 up:standby 2024-04-21T08:48:34.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:34 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:48:34.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.824+0000 7f92356c0640 1 -- 172.21.15.28:0/701075330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9230101fc0 msgr2=0x7f9230102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.824+0000 7f92356c0640 1 --2- 172.21.15.28:0/701075330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9230101fc0 0x7f9230102440 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9224009900 tx=0x7f922402bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.824+0000 7f92356c0640 1 -- 172.21.15.28:0/701075330 shutdown_connections 2024-04-21T08:48:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.824+0000 7f92356c0640 1 --2- 172.21.15.28:0/701075330 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9230102980 0x7f92301092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.824+0000 7f92356c0640 1 --2- 172.21.15.28:0/701075330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9230101fc0 0x7f9230102440 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.824+0000 7f92356c0640 1 --2- 172.21.15.28:0/701075330 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9230100e70 0x7f9230101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.824+0000 7f92356c0640 1 -- 172.21.15.28:0/701075330 >> 172.21.15.28:0/701075330 conn(0x7f92300fc3e0 msgr2=0x7f92300fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.824+0000 7f92356c0640 1 -- 172.21.15.28:0/701075330 shutdown_connections 2024-04-21T08:48:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.824+0000 7f92356c0640 1 -- 172.21.15.28:0/701075330 wait complete. 2024-04-21T08:48:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.824+0000 7f92356c0640 1 Processor -- start 2024-04-21T08:48:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f92356c0640 1 -- start start 2024-04-21T08:48:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f92356c0640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9230100e70 0x7f9230104df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f92356c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9230101fc0 0x7f9230103430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f92356c0640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9230102980 0x7f9230103970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f92356c0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f923010c9f0 con 0x7f9230101fc0 2024-04-21T08:48:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f92356c0640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f923010c870 con 0x7f9230100e70 2024-04-21T08:48:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f92356c0640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f923010cb70 con 0x7f9230102980 2024-04-21T08:48:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f922e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9230101fc0 0x7f9230103430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f922effd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9230100e70 0x7f9230104df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f922e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9230101fc0 0x7f9230103430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45538/0 (socket says 172.21.15.28:45538) 2024-04-21T08:48:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f922e7fc640 1 -- 172.21.15.28:0/62630797 learned_addr learned my addr 172.21.15.28:0/62630797 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f922f7fe640 1 --2- 172.21.15.28:0/62630797 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9230102980 0x7f9230103970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f922e7fc640 1 -- 172.21.15.28:0/62630797 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9230102980 msgr2=0x7f9230103970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f922e7fc640 1 --2- 172.21.15.28:0/62630797 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9230102980 0x7f9230103970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f922e7fc640 1 -- 172.21.15.28:0/62630797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9230100e70 msgr2=0x7f9230104df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f922e7fc640 1 --2- 172.21.15.28:0/62630797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9230100e70 0x7f9230104df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:34.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.825+0000 7f922e7fc640 1 -- 172.21.15.28:0/62630797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9230104200 con 0x7f9230101fc0 2024-04-21T08:48:34.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.826+0000 7f922e7fc640 1 --2- 172.21.15.28:0/62630797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9230101fc0 0x7f9230103430 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9224005ab0 tx=0x7f92240042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:34.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.826+0000 7f9217fff640 1 -- 172.21.15.28:0/62630797 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9224043020 con 0x7f9230101fc0 2024-04-21T08:48:34.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.826+0000 7f9217fff640 1 -- 172.21.15.28:0/62630797 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92240054c0 con 0x7f9230101fc0 2024-04-21T08:48:34.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.826+0000 7f92356c0640 1 -- 172.21.15.28:0/62630797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92301a22c0 con 0x7f9230101fc0 2024-04-21T08:48:34.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.826+0000 7f9217fff640 1 -- 172.21.15.28:0/62630797 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9224003d70 con 0x7f9230101fc0 2024-04-21T08:48:34.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.826+0000 7f92356c0640 1 -- 172.21.15.28:0/62630797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92301a27e0 con 0x7f9230101fc0 2024-04-21T08:48:34.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.827+0000 7f92356c0640 1 -- 172.21.15.28:0/62630797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92301a29e0 con 0x7f9230101fc0 2024-04-21T08:48:34.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.829+0000 7f9217fff640 1 -- 172.21.15.28:0/62630797 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9224046020 con 0x7f9230101fc0 2024-04-21T08:48:34.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.829+0000 7f9217fff640 1 --2- 172.21.15.28:0/62630797 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f92180a3bb0 0x7f92180a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:34.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.829+0000 7f922effd640 1 --2- 172.21.15.28:0/62630797 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f92180a3bb0 0x7f92180a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:34.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.829+0000 7f9217fff640 1 -- 172.21.15.28:0/62630797 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f92240f23c0 con 0x7f9230101fc0 2024-04-21T08:48:34.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.830+0000 7f922effd640 1 --2- 172.21.15.28:0/62630797 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f92180a3bb0 0x7f92180a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f922000abe0 tx=0x7f922000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:34.834+0000 7f9217fff640 1 -- 172.21.15.28:0/62630797 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f922404d020 con 0x7f9230101fc0 2024-04-21T08:48:35.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:35.080+0000 7f92356c0640 1 -- 172.21.15.28:0/62630797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92301a2bb0 con 0x7f9230101fc0 2024-04-21T08:48:35.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:35.082+0000 7f9217fff640 1 -- 172.21.15.28:0/62630797 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8010 (secure 0 0 0) 0x7f92240bf240 con 0x7f9230101fc0 2024-04-21T08:48:35.081 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:35.081 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T08:48:35.082 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 10 2024-04-21T08:48:35.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:35.084+0000 7f92356c0640 1 -- 172.21.15.28:0/62630797 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f92180a3bb0 msgr2=0x7f92180a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:35.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:35.084+0000 7f92356c0640 1 --2- 172.21.15.28:0/62630797 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f92180a3bb0 0x7f92180a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f922000abe0 tx=0x7f922000a2d0 comp rx=0 tx=0).stop 2024-04-21T08:48:35.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:35.084+0000 7f92356c0640 1 -- 172.21.15.28:0/62630797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9230101fc0 msgr2=0x7f9230103430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:35.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:35.084+0000 7f92356c0640 1 --2- 172.21.15.28:0/62630797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9230101fc0 0x7f9230103430 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9224005ab0 tx=0x7f92240042f0 comp rx=0 tx=0).stop 2024-04-21T08:48:35.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:35.084+0000 7f92356c0640 1 -- 172.21.15.28:0/62630797 shutdown_connections 2024-04-21T08:48:35.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:35.084+0000 7f92356c0640 1 --2- 172.21.15.28:0/62630797 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9230102980 0x7f9230103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:35.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:35.084+0000 7f92356c0640 1 --2- 172.21.15.28:0/62630797 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f92180a3bb0 0x7f92180a6070 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:35.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:35.084+0000 7f92356c0640 1 --2- 172.21.15.28:0/62630797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9230101fc0 0x7f9230103430 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:35.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:35.084+0000 7f92356c0640 1 --2- 172.21.15.28:0/62630797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9230100e70 0x7f9230104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:35.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:35.084+0000 7f92356c0640 1 -- 172.21.15.28:0/62630797 >> 172.21.15.28:0/62630797 conn(0x7f92300fc3e0 msgr2=0x7f92300fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:35.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:35.084+0000 7f92356c0640 1 -- 172.21.15.28:0/62630797 shutdown_connections 2024-04-21T08:48:35.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:35.084+0000 7f92356c0640 1 -- 172.21.15.28:0/62630797 wait complete. 2024-04-21T08:48:35.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:35 smithi028 ceph-mon[28003]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:35.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:35 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/62630797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:35.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:35 smithi152 ceph-mon[31401]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:35.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:35 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/62630797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:35.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:35 smithi171 ceph-mon[31358]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:35.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:35 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/62630797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:36.105 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 21 08:48:35 smithi028 systemd[1]: Started Ceph mds.g for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:48:37.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:37.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:36 smithi028 ceph-mon[28003]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:37.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:37.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:37.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:37.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:37.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:37.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:37.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:36 smithi028 ceph-mon[28003]: Deploying daemon mds.a on smithi028 2024-04-21T08:48:37.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:36 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.28:6838/3409769959,v1:172.21.15.28:6839/3409769959] up:boot 2024-04-21T08:48:37.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:36 smithi028 ceph-mon[28003]: fsmap 10 up:standby 2024-04-21T08:48:37.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:48:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.207+0000 7ffac414b640 1 -- 172.21.15.28:0/2883738042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc102a10 msgr2=0x7ffabc109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.207+0000 7ffac414b640 1 --2- 172.21.15.28:0/2883738042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc102a10 0x7ffabc109330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ffab4009ca0 tx=0x7ffab402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.207+0000 7ffac414b640 1 -- 172.21.15.28:0/2883738042 shutdown_connections 2024-04-21T08:48:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.207+0000 7ffac414b640 1 --2- 172.21.15.28:0/2883738042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc102a10 0x7ffabc109330 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.207+0000 7ffac414b640 1 --2- 172.21.15.28:0/2883738042 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffabc102050 0x7ffabc1024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.207+0000 7ffac414b640 1 --2- 172.21.15.28:0/2883738042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffabc100e70 0x7ffabc101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.207+0000 7ffac414b640 1 -- 172.21.15.28:0/2883738042 >> 172.21.15.28:0/2883738042 conn(0x7ffabc0fc470 msgr2=0x7ffabc0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.208+0000 7ffac414b640 1 -- 172.21.15.28:0/2883738042 shutdown_connections 2024-04-21T08:48:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.208+0000 7ffac414b640 1 -- 172.21.15.28:0/2883738042 wait complete. 2024-04-21T08:48:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.208+0000 7ffac414b640 1 Processor -- start 2024-04-21T08:48:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.208+0000 7ffac414b640 1 -- start start 2024-04-21T08:48:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.208+0000 7ffac414b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffabc100e70 0x7ffabc104e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.208+0000 7ffac414b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffabc102050 0x7ffabc1034c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.208+0000 7ffac414b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc102a10 0x7ffabc103a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.208+0000 7ffac414b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffabc10ca10 con 0x7ffabc102a10 2024-04-21T08:48:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.208+0000 7ffac414b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ffabc10c890 con 0x7ffabc100e70 2024-04-21T08:48:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.208+0000 7ffac414b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffabc10cb90 con 0x7ffabc102050 2024-04-21T08:48:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.208+0000 7ffac26e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc102a10 0x7ffabc103a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.208+0000 7ffac26e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc102a10 0x7ffabc103a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45566/0 (socket says 172.21.15.28:45566) 2024-04-21T08:48:37.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.208+0000 7ffac26e8640 1 -- 172.21.15.28:0/2233008732 learned_addr learned my addr 172.21.15.28:0/2233008732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:37.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.209+0000 7ffac1ee7640 1 --2- 172.21.15.28:0/2233008732 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffabc100e70 0x7ffabc104e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:37.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.209+0000 7ffac16e6640 1 --2- 172.21.15.28:0/2233008732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffabc102050 0x7ffabc1034c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:37.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.209+0000 7ffac26e8640 1 -- 172.21.15.28:0/2233008732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffabc102050 msgr2=0x7ffabc1034c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:37.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.209+0000 7ffac26e8640 1 --2- 172.21.15.28:0/2233008732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffabc102050 0x7ffabc1034c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.209+0000 7ffac26e8640 1 -- 172.21.15.28:0/2233008732 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffabc100e70 msgr2=0x7ffabc104e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:37.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.209+0000 7ffac26e8640 1 --2- 172.21.15.28:0/2233008732 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffabc100e70 0x7ffabc104e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.209+0000 7ffac26e8640 1 -- 172.21.15.28:0/2233008732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffabc104290 con 0x7ffabc102a10 2024-04-21T08:48:37.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.209+0000 7ffac26e8640 1 --2- 172.21.15.28:0/2233008732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc102a10 0x7ffabc103a00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ffab4032520 tx=0x7ffab40347b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:37.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.209+0000 7ffab2ffd640 1 -- 172.21.15.28:0/2233008732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab4039040 con 0x7ffabc102a10 2024-04-21T08:48:37.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.209+0000 7ffab2ffd640 1 -- 172.21.15.28:0/2233008732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffab4044070 con 0x7ffabc102a10 2024-04-21T08:48:37.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.209+0000 7ffac414b640 1 -- 172.21.15.28:0/2233008732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffabc1a2210 con 0x7ffabc102a10 2024-04-21T08:48:37.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.209+0000 7ffab2ffd640 1 -- 172.21.15.28:0/2233008732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab40042a0 con 0x7ffabc102a10 2024-04-21T08:48:37.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.209+0000 7ffac414b640 1 -- 172.21.15.28:0/2233008732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffabc1a2760 con 0x7ffabc102a10 2024-04-21T08:48:37.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.210+0000 7ffac414b640 1 -- 172.21.15.28:0/2233008732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffabc1a29c0 con 0x7ffabc102a10 2024-04-21T08:48:37.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.211+0000 7ffab2ffd640 1 -- 172.21.15.28:0/2233008732 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffab4003970 con 0x7ffabc102a10 2024-04-21T08:48:37.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.212+0000 7ffab2ffd640 1 --2- 172.21.15.28:0/2233008732 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffaa80a3bb0 0x7ffaa80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:37.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.212+0000 7ffac1ee7640 1 --2- 172.21.15.28:0/2233008732 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffaa80a3bb0 0x7ffaa80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:37.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.212+0000 7ffab2ffd640 1 -- 172.21.15.28:0/2233008732 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ffab40f2360 con 0x7ffabc102a10 2024-04-21T08:48:37.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.212+0000 7ffac1ee7640 1 --2- 172.21.15.28:0/2233008732 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffaa80a3bb0 0x7ffaa80a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ffabc101bc0 tx=0x7ffab8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:37.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.216+0000 7ffab2ffd640 1 -- 172.21.15.28:0/2233008732 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffab40333a0 con 0x7ffabc102a10 2024-04-21T08:48:37.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:36 smithi152 ceph-mon[31401]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:36 smithi152 ceph-mon[31401]: Deploying daemon mds.a on smithi028 2024-04-21T08:48:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:36 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.28:6838/3409769959,v1:172.21.15.28:6839/3409769959] up:boot 2024-04-21T08:48:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:36 smithi152 ceph-mon[31401]: fsmap 10 up:standby 2024-04-21T08:48:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:48:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:36 smithi171 ceph-mon[31358]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:36 smithi171 ceph-mon[31358]: Deploying daemon mds.a on smithi028 2024-04-21T08:48:37.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:36 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.28:6838/3409769959,v1:172.21.15.28:6839/3409769959] up:boot 2024-04-21T08:48:37.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:36 smithi171 ceph-mon[31358]: fsmap 10 up:standby 2024-04-21T08:48:37.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:48:37.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.444+0000 7ffac414b640 1 -- 172.21.15.28:0/2233008732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffabc1a2cb0 con 0x7ffabc102a10 2024-04-21T08:48:37.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.444+0000 7ffab2ffd640 1 -- 172.21.15.28:0/2233008732 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8836 (secure 0 0 0) 0x7ffab40bf140 con 0x7ffabc102a10 2024-04-21T08:48:37.444 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:37.444 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T08:48:37.444 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 11 2024-04-21T08:48:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.446+0000 7ffac414b640 1 -- 172.21.15.28:0/2233008732 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffaa80a3bb0 msgr2=0x7ffaa80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.446+0000 7ffac414b640 1 --2- 172.21.15.28:0/2233008732 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffaa80a3bb0 0x7ffaa80a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ffabc101bc0 tx=0x7ffab8005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.446+0000 7ffac414b640 1 -- 172.21.15.28:0/2233008732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc102a10 msgr2=0x7ffabc103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.446+0000 7ffac414b640 1 --2- 172.21.15.28:0/2233008732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc102a10 0x7ffabc103a00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ffab4032520 tx=0x7ffab40347b0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.446+0000 7ffac414b640 1 -- 172.21.15.28:0/2233008732 shutdown_connections 2024-04-21T08:48:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.446+0000 7ffac414b640 1 --2- 172.21.15.28:0/2233008732 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffaa80a3bb0 0x7ffaa80a6070 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.446+0000 7ffac414b640 1 --2- 172.21.15.28:0/2233008732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffabc102a10 0x7ffabc103a00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.446+0000 7ffac414b640 1 --2- 172.21.15.28:0/2233008732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffabc102050 0x7ffabc1034c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.446+0000 7ffac414b640 1 --2- 172.21.15.28:0/2233008732 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffabc100e70 0x7ffabc104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.446+0000 7ffac414b640 1 -- 172.21.15.28:0/2233008732 >> 172.21.15.28:0/2233008732 conn(0x7ffabc0fc470 msgr2=0x7ffabc0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.446+0000 7ffac414b640 1 -- 172.21.15.28:0/2233008732 shutdown_connections 2024-04-21T08:48:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:37.446+0000 7ffac414b640 1 -- 172.21.15.28:0/2233008732 wait complete. 2024-04-21T08:48:38.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:37 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2233008732' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:38.105 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 21 08:48:37 smithi028 systemd[1]: Starting Ceph mds.a for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T08:48:38.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:37 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2233008732' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:37 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2233008732' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:38.496 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 21 08:48:38 smithi028 podman[69273]: 2024-04-21 08:48:38.126262818 +0000 UTC m=+0.018697139 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:48:38.496 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 21 08:48:38 smithi028 podman[69273]: 2024-04-21 08:48:38.225263202 +0000 UTC m=+0.117697509 container create c433953f81215dbd218a974aa1120cc84df45c57a057e9f2bfbc46975aed88d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-21T08:48:38.496 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 21 08:48:38 smithi028 podman[69273]: 2024-04-21 08:48:38.4702843 +0000 UTC m=+0.362718609 container init c433953f81215dbd218a974aa1120cc84df45c57a057e9f2bfbc46975aed88d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:48:38.496 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 21 08:48:38 smithi028 podman[69273]: 2024-04-21 08:48:38.472879201 +0000 UTC m=+0.365313510 container start c433953f81215dbd218a974aa1120cc84df45c57a057e9f2bfbc46975aed88d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:48:38.855 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 21 08:48:38 smithi028 ceph-mds[69292]: -- 172.21.15.28:0/3096646415 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b7b9a19a40 con 0x55b7b99ec800 2024-04-21T08:48:38.855 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 21 08:48:38 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-a[69288]: starting mds.a at 2024-04-21T08:48:38.855 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 21 08:48:38 smithi028 bash[69273]: c433953f81215dbd218a974aa1120cc84df45c57a057e9f2bfbc46975aed88d3 2024-04-21T08:48:38.855 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 21 08:48:38 smithi028 systemd[1]: Started Ceph mds.a for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:48:39.299 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:38 smithi152 ceph-mon[31401]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:39.299 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:38 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:39.299 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:38 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:39.299 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:38 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:39.299 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:38 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:39.300 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:38 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:39.300 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:38 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:39.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:38 smithi028 ceph-mon[28003]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:39.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:38 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:39.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:38 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:39.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:38 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:39.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:38 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:39.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:38 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:39.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:38 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:39.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:38 smithi171 ceph-mon[31358]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:39.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:38 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:39.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:38 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:39.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:38 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:39.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:38 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:48:39.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:38 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:48:39.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:38 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:39.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.576+0000 7fec56f7e640 1 -- 172.21.15.28:0/1235403098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec501029f0 msgr2=0x7fec50104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.576+0000 7fec56f7e640 1 --2- 172.21.15.28:0/1235403098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec501029f0 0x7fec50104de0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fec44009900 tx=0x7fec4402bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.576+0000 7fec56f7e640 1 -- 172.21.15.28:0/1235403098 shutdown_connections 2024-04-21T08:48:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.576+0000 7fec56f7e640 1 --2- 172.21.15.28:0/1235403098 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec50107c50 0x7fec5010a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.576+0000 7fec56f7e640 1 --2- 172.21.15.28:0/1235403098 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec50105320 0x7fec50107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.576+0000 7fec56f7e640 1 --2- 172.21.15.28:0/1235403098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec501029f0 0x7fec50104de0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.576+0000 7fec56f7e640 1 -- 172.21.15.28:0/1235403098 >> 172.21.15.28:0/1235403098 conn(0x7fec500fc3e0 msgr2=0x7fec500fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.576+0000 7fec56f7e640 1 -- 172.21.15.28:0/1235403098 shutdown_connections 2024-04-21T08:48:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.576+0000 7fec56f7e640 1 -- 172.21.15.28:0/1235403098 wait complete. 2024-04-21T08:48:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec56f7e640 1 Processor -- start 2024-04-21T08:48:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec56f7e640 1 -- start start 2024-04-21T08:48:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec56f7e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec501029f0 0x7fec501981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec56f7e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec50105320 0x7fec50198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec56f7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec50107c50 0x7fec5019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec56f7e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec5010dac0 con 0x7fec50107c50 2024-04-21T08:48:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec56f7e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fec5010d940 con 0x7fec501029f0 2024-04-21T08:48:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec56f7e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fec5010dc40 con 0x7fec50105320 2024-04-21T08:48:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec5551b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec50107c50 0x7fec5019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec54d1a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec501029f0 0x7fec501981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec54d1a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec501029f0 0x7fec501981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:48148/0 (socket says 172.21.15.28:48148) 2024-04-21T08:48:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec54d1a640 1 -- 172.21.15.28:0/272742459 learned_addr learned my addr 172.21.15.28:0/272742459 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec4ffff640 1 --2- 172.21.15.28:0/272742459 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec50105320 0x7fec50198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec54d1a640 1 -- 172.21.15.28:0/272742459 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec50105320 msgr2=0x7fec50198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec54d1a640 1 --2- 172.21.15.28:0/272742459 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec50105320 0x7fec50198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec54d1a640 1 -- 172.21.15.28:0/272742459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec50107c50 msgr2=0x7fec5019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.577+0000 7fec54d1a640 1 --2- 172.21.15.28:0/272742459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec50107c50 0x7fec5019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.578+0000 7fec54d1a640 1 -- 172.21.15.28:0/272742459 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec5019fde0 con 0x7fec501029f0 2024-04-21T08:48:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.578+0000 7fec5551b640 1 --2- 172.21.15.28:0/272742459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec50107c50 0x7fec5019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.578+0000 7fec4ffff640 1 --2- 172.21.15.28:0/272742459 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec50105320 0x7fec50198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:39.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.578+0000 7fec54d1a640 1 --2- 172.21.15.28:0/272742459 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec501029f0 0x7fec501981c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fec44006010 tx=0x7fec44004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:39.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.578+0000 7fec4dffb640 1 -- 172.21.15.28:0/272742459 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec44043020 con 0x7fec501029f0 2024-04-21T08:48:39.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.578+0000 7fec56f7e640 1 -- 172.21.15.28:0/272742459 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec501a0070 con 0x7fec501029f0 2024-04-21T08:48:39.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.578+0000 7fec4dffb640 1 -- 172.21.15.28:0/272742459 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec44004440 con 0x7fec501029f0 2024-04-21T08:48:39.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.578+0000 7fec4dffb640 1 -- 172.21.15.28:0/272742459 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec44003bf0 con 0x7fec501029f0 2024-04-21T08:48:39.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.578+0000 7fec56f7e640 1 -- 172.21.15.28:0/272742459 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec501a05c0 con 0x7fec501029f0 2024-04-21T08:48:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.579+0000 7fec56f7e640 1 -- 172.21.15.28:0/272742459 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec500fded0 con 0x7fec501029f0 2024-04-21T08:48:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.580+0000 7fec4dffb640 1 -- 172.21.15.28:0/272742459 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fec44046020 con 0x7fec501029f0 2024-04-21T08:48:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.581+0000 7fec4dffb640 1 --2- 172.21.15.28:0/272742459 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fec400a3bb0 0x7fec400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.581+0000 7fec4ffff640 1 --2- 172.21.15.28:0/272742459 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fec400a3bb0 0x7fec400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.581+0000 7fec4dffb640 1 -- 172.21.15.28:0/272742459 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fec440f29d0 con 0x7fec501029f0 2024-04-21T08:48:39.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.582+0000 7fec4ffff640 1 --2- 172.21.15.28:0/272742459 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fec400a3bb0 0x7fec400a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fec501996d0 tx=0x7fec3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.586+0000 7fec4dffb640 1 -- 172.21.15.28:0/272742459 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fec4404d020 con 0x7fec501029f0 2024-04-21T08:48:39.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.812+0000 7fec56f7e640 1 -- 172.21.15.28:0/272742459 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec5002d5c0 con 0x7fec501029f0 2024-04-21T08:48:39.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.813+0000 7fec4dffb640 1 -- 172.21.15.28:0/272742459 <== mon.2 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9662 (secure 0 0 0) 0x7fec440bf7b0 con 0x7fec501029f0 2024-04-21T08:48:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:39.815 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T08:48:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 12 2024-04-21T08:48:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.815+0000 7fec56f7e640 1 -- 172.21.15.28:0/272742459 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fec400a3bb0 msgr2=0x7fec400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.816+0000 7fec56f7e640 1 --2- 172.21.15.28:0/272742459 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fec400a3bb0 0x7fec400a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fec501996d0 tx=0x7fec3c005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.816+0000 7fec56f7e640 1 -- 172.21.15.28:0/272742459 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec501029f0 msgr2=0x7fec501981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.816+0000 7fec56f7e640 1 --2- 172.21.15.28:0/272742459 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec501029f0 0x7fec501981c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fec44006010 tx=0x7fec44004040 comp rx=0 tx=0).stop 2024-04-21T08:48:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.816+0000 7fec56f7e640 1 -- 172.21.15.28:0/272742459 shutdown_connections 2024-04-21T08:48:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.816+0000 7fec56f7e640 1 --2- 172.21.15.28:0/272742459 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fec400a3bb0 0x7fec400a6070 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.816+0000 7fec56f7e640 1 --2- 172.21.15.28:0/272742459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec50107c50 0x7fec5019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.816+0000 7fec56f7e640 1 --2- 172.21.15.28:0/272742459 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fec50105320 0x7fec50198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.816+0000 7fec56f7e640 1 --2- 172.21.15.28:0/272742459 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec501029f0 0x7fec501981c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.816+0000 7fec56f7e640 1 -- 172.21.15.28:0/272742459 >> 172.21.15.28:0/272742459 conn(0x7fec500fc3e0 msgr2=0x7fec501034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.816+0000 7fec56f7e640 1 -- 172.21.15.28:0/272742459 shutdown_connections 2024-04-21T08:48:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:39.816+0000 7fec56f7e640 1 -- 172.21.15.28:0/272742459 wait complete. 2024-04-21T08:48:40.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:39 smithi028 ceph-mon[28003]: Deploying daemon mds.b on smithi152 2024-04-21T08:48:40.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:39 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.28:6840/1709589702,v1:172.21.15.28:6841/1709589702] up:boot 2024-04-21T08:48:40.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:39 smithi028 ceph-mon[28003]: fsmap 11 up:standby 2024-04-21T08:48:40.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:39 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:48:40.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:39 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/272742459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:40.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:39 smithi152 ceph-mon[31401]: Deploying daemon mds.b on smithi152 2024-04-21T08:48:40.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:39 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.28:6840/1709589702,v1:172.21.15.28:6841/1709589702] up:boot 2024-04-21T08:48:40.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:39 smithi152 ceph-mon[31401]: fsmap 11 up:standby 2024-04-21T08:48:40.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:39 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:48:40.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:39 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/272742459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:40.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:39 smithi171 ceph-mon[31358]: Deploying daemon mds.b on smithi152 2024-04-21T08:48:40.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:39 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.28:6840/1709589702,v1:172.21.15.28:6841/1709589702] up:boot 2024-04-21T08:48:40.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:39 smithi171 ceph-mon[31358]: fsmap 11 up:standby 2024-04-21T08:48:40.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:39 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:48:40.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:39 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/272742459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:41.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:40 smithi152 ceph-mon[31401]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:41.104 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 08:48:41 smithi152 systemd[1]: Starting Ceph mds.b for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T08:48:41.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:40 smithi028 ceph-mon[28003]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:40 smithi171 ceph-mon[31358]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:41.531 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 08:48:41 smithi152 podman[62208]: 2024-04-21 08:48:41.150868411 +0000 UTC m=+0.018238750 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:48:41.531 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 08:48:41 smithi152 podman[62208]: 2024-04-21 08:48:41.271119216 +0000 UTC m=+0.138489556 container create 90080740055cf1fa361e88f6f6bb41444ad12b4ac58c4e382516759848cf7bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-04-21T08:48:41.532 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 08:48:41 smithi152 podman[62208]: 2024-04-21 08:48:41.532469949 +0000 UTC m=+0.399840289 container init 90080740055cf1fa361e88f6f6bb41444ad12b4ac58c4e382516759848cf7bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:48:41.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:41.854 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 08:48:41 smithi152 podman[62208]: 2024-04-21 08:48:41.535211755 +0000 UTC m=+0.402582105 container start 90080740055cf1fa361e88f6f6bb41444ad12b4ac58c4e382516759848cf7bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:48:41.854 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 08:48:41 smithi152 ceph-mds[62227]: -- 172.21.15.152:0/1438102399 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dc46823a40 con 0x55dc467f6c00 2024-04-21T08:48:41.854 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 08:48:41 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-b[62223]: starting mds.b at 2024-04-21T08:48:41.854 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 08:48:41 smithi152 bash[62208]: 90080740055cf1fa361e88f6f6bb41444ad12b4ac58c4e382516759848cf7bb8 2024-04-21T08:48:41.854 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 08:48:41 smithi152 systemd[1]: Started Ceph mds.b for b51a76c8-ffba-11ee-bc93-c7b262605968. 2024-04-21T08:48:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.942+0000 7fa668d23640 1 -- 172.21.15.28:0/4149392782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa664101b60 msgr2=0x7fa664101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.942+0000 7fa668d23640 1 --2- 172.21.15.28:0/4149392782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa664101b60 0x7fa664101fe0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009ca0 tx=0x7fa65c02c180 comp rx=0 tx=0).stop 2024-04-21T08:48:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.942+0000 7fa668d23640 1 -- 172.21.15.28:0/4149392782 shutdown_connections 2024-04-21T08:48:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.942+0000 7fa668d23640 1 --2- 172.21.15.28:0/4149392782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa664101b60 0x7fa664101fe0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.942+0000 7fa668d23640 1 --2- 172.21.15.28:0/4149392782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6641011c0 0x7fa664101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.942+0000 7fa668d23640 1 --2- 172.21.15.28:0/4149392782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa664100860 0x7fa664100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.942+0000 7fa668d23640 1 -- 172.21.15.28:0/4149392782 >> 172.21.15.28:0/4149392782 conn(0x7fa6640fc3e0 msgr2=0x7fa6640fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.942+0000 7fa668d23640 1 -- 172.21.15.28:0/4149392782 shutdown_connections 2024-04-21T08:48:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.942+0000 7fa668d23640 1 -- 172.21.15.28:0/4149392782 wait complete. 2024-04-21T08:48:41.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.943+0000 7fa668d23640 1 Processor -- start 2024-04-21T08:48:41.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.943+0000 7fa668d23640 1 -- start start 2024-04-21T08:48:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.943+0000 7fa668d23640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa664100860 0x7fa66419a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.943+0000 7fa668d23640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6641011c0 0x7fa66419a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.943+0000 7fa668d23640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa664101b60 0x7fa6641a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.943+0000 7fa668d23640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa66410fcb0 con 0x7fa664101b60 2024-04-21T08:48:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.943+0000 7fa668d23640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa66410fb30 con 0x7fa6641011c0 2024-04-21T08:48:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.943+0000 7fa668d23640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa66410fe30 con 0x7fa664100860 2024-04-21T08:48:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.943+0000 7fa662d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa664101b60 0x7fa6641a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.943+0000 7fa662d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa664101b60 0x7fa6641a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45622/0 (socket says 172.21.15.28:45622) 2024-04-21T08:48:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.943+0000 7fa662d9d640 1 -- 172.21.15.28:0/3533033071 learned_addr learned my addr 172.21.15.28:0/3533033071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.944+0000 7fa66259c640 1 --2- 172.21.15.28:0/3533033071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa664100860 0x7fa66419a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.944+0000 7fa661d9b640 1 --2- 172.21.15.28:0/3533033071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6641011c0 0x7fa66419a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.944+0000 7fa662d9d640 1 -- 172.21.15.28:0/3533033071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa664100860 msgr2=0x7fa66419a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.944+0000 7fa662d9d640 1 --2- 172.21.15.28:0/3533033071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa664100860 0x7fa66419a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.944+0000 7fa662d9d640 1 -- 172.21.15.28:0/3533033071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6641011c0 msgr2=0x7fa66419a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:41.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.944+0000 7fa662d9d640 1 --2- 172.21.15.28:0/3533033071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6641011c0 0x7fa66419a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:41.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.944+0000 7fa662d9d640 1 -- 172.21.15.28:0/3533033071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6641a1f80 con 0x7fa664101b60 2024-04-21T08:48:41.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.944+0000 7fa662d9d640 1 --2- 172.21.15.28:0/3533033071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa664101b60 0x7fa6641a1910 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0325a0 tx=0x7fa65c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.944+0000 7fa6537fe640 1 -- 172.21.15.28:0/3533033071 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c039040 con 0x7fa664101b60 2024-04-21T08:48:41.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.944+0000 7fa668d23640 1 -- 172.21.15.28:0/3533033071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6641a2210 con 0x7fa664101b60 2024-04-21T08:48:41.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.944+0000 7fa6537fe640 1 -- 172.21.15.28:0/3533033071 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa65c044070 con 0x7fa664101b60 2024-04-21T08:48:41.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.944+0000 7fa6537fe640 1 -- 172.21.15.28:0/3533033071 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0329c0 con 0x7fa664101b60 2024-04-21T08:48:41.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.944+0000 7fa668d23640 1 -- 172.21.15.28:0/3533033071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6641a2700 con 0x7fa664101b60 2024-04-21T08:48:41.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.945+0000 7fa668d23640 1 -- 172.21.15.28:0/3533033071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa66410b740 con 0x7fa664101b60 2024-04-21T08:48:41.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.947+0000 7fa6537fe640 1 -- 172.21.15.28:0/3533033071 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa65c032b60 con 0x7fa664101b60 2024-04-21T08:48:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.948+0000 7fa6537fe640 1 --2- 172.21.15.28:0/3533033071 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa64c0a3bb0 0x7fa64c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.948+0000 7fa66259c640 1 --2- 172.21.15.28:0/3533033071 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa64c0a3bb0 0x7fa64c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.948+0000 7fa6537fe640 1 -- 172.21.15.28:0/3533033071 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa65c0f2c40 con 0x7fa664101b60 2024-04-21T08:48:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.949+0000 7fa66259c640 1 --2- 172.21.15.28:0/3533033071 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa64c0a3bb0 0x7fa64c0a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa654009bd0 tx=0x7fa654005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:41.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:41.951+0000 7fa6537fe640 1 -- 172.21.15.28:0/3533033071 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa65c032e50 con 0x7fa664101b60 2024-04-21T08:48:42.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.183+0000 7fa668d23640 1 -- 172.21.15.28:0/3533033071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa66402d5c0 con 0x7fa664101b60 2024-04-21T08:48:42.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.184+0000 7fa6537fe640 1 -- 172.21.15.28:0/3533033071 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10491 (secure 0 0 0) 0x7fa65c0bfac0 con 0x7fa664101b60 2024-04-21T08:48:42.185 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:42.185 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24488,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-21T08:48:42.185 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 13 2024-04-21T08:48:42.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.186+0000 7fa668d23640 1 -- 172.21.15.28:0/3533033071 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa64c0a3bb0 msgr2=0x7fa64c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.186+0000 7fa668d23640 1 --2- 172.21.15.28:0/3533033071 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa64c0a3bb0 0x7fa64c0a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa654009bd0 tx=0x7fa654005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:42.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.186+0000 7fa668d23640 1 -- 172.21.15.28:0/3533033071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa664101b60 msgr2=0x7fa6641a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.186+0000 7fa668d23640 1 --2- 172.21.15.28:0/3533033071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa664101b60 0x7fa6641a1910 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0325a0 tx=0x7fa65c003970 comp rx=0 tx=0).stop 2024-04-21T08:48:42.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.186+0000 7fa668d23640 1 -- 172.21.15.28:0/3533033071 shutdown_connections 2024-04-21T08:48:42.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.186+0000 7fa668d23640 1 --2- 172.21.15.28:0/3533033071 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa64c0a3bb0 0x7fa64c0a6070 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.186+0000 7fa668d23640 1 --2- 172.21.15.28:0/3533033071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa664101b60 0x7fa6641a1910 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.186+0000 7fa668d23640 1 --2- 172.21.15.28:0/3533033071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6641011c0 0x7fa66419a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.186+0000 7fa668d23640 1 --2- 172.21.15.28:0/3533033071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa664100860 0x7fa66419a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.187+0000 7fa668d23640 1 -- 172.21.15.28:0/3533033071 >> 172.21.15.28:0/3533033071 conn(0x7fa6640fc3e0 msgr2=0x7fa6640fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.187+0000 7fa668d23640 1 -- 172.21.15.28:0/3533033071 shutdown_connections 2024-04-21T08:48:42.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.187+0000 7fa668d23640 1 -- 172.21.15.28:0/3533033071 wait complete. 2024-04-21T08:48:42.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:48:42.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.301+0000 7f712c23f640 1 -- 172.21.15.28:0/3939253090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7124101fc0 msgr2=0x7f7124102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.301+0000 7f712c23f640 1 --2- 172.21.15.28:0/3939253090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7124101fc0 0x7f7124102440 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7120009900 tx=0x7f712002bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:42.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.301+0000 7f712c23f640 1 -- 172.21.15.28:0/3939253090 shutdown_connections 2024-04-21T08:48:42.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.301+0000 7f712c23f640 1 --2- 172.21.15.28:0/3939253090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7124102980 0x7f71241092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.301+0000 7f712c23f640 1 --2- 172.21.15.28:0/3939253090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7124101fc0 0x7f7124102440 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.301+0000 7f712c23f640 1 --2- 172.21.15.28:0/3939253090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7124100e70 0x7f7124101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.301+0000 7f712c23f640 1 -- 172.21.15.28:0/3939253090 >> 172.21.15.28:0/3939253090 conn(0x7f71240fc3e0 msgr2=0x7f71240fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.301+0000 7f712c23f640 1 -- 172.21.15.28:0/3939253090 shutdown_connections 2024-04-21T08:48:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.301+0000 7f712c23f640 1 -- 172.21.15.28:0/3939253090 wait complete. 2024-04-21T08:48:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.302+0000 7f712c23f640 1 Processor -- start 2024-04-21T08:48:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.302+0000 7f712c23f640 1 -- start start 2024-04-21T08:48:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.302+0000 7f712c23f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7124100e70 0x7f7124198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.302+0000 7f712c23f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7124101fc0 0x7f71241986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.302+0000 7f712c23f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7124102980 0x7f712419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.302+0000 7f712c23f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7124069b30 con 0x7f7124100e70 2024-04-21T08:48:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.302+0000 7f712c23f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f71240699b0 con 0x7f7124102980 2024-04-21T08:48:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.302+0000 7f712c23f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7124069cb0 con 0x7f7124101fc0 2024-04-21T08:48:42.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.302+0000 7f7129fdb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7124100e70 0x7f7124198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.302+0000 7f7129fdb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7124100e70 0x7f7124198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45640/0 (socket says 172.21.15.28:45640) 2024-04-21T08:48:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.302+0000 7f7129fdb640 1 -- 172.21.15.28:0/2550233624 learned_addr learned my addr 172.21.15.28:0/2550233624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.302+0000 7f71297da640 1 --2- 172.21.15.28:0/2550233624 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7124101fc0 0x7f71241986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.302+0000 7f712a7dc640 1 --2- 172.21.15.28:0/2550233624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7124102980 0x7f712419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.303+0000 7f7129fdb640 1 -- 172.21.15.28:0/2550233624 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7124101fc0 msgr2=0x7f71241986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.303+0000 7f7129fdb640 1 --2- 172.21.15.28:0/2550233624 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7124101fc0 0x7f71241986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.303+0000 7f7129fdb640 1 -- 172.21.15.28:0/2550233624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7124102980 msgr2=0x7f712419f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.303+0000 7f7129fdb640 1 --2- 172.21.15.28:0/2550233624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7124102980 0x7f712419f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.303+0000 7f7129fdb640 1 -- 172.21.15.28:0/2550233624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f712419fdb0 con 0x7f7124100e70 2024-04-21T08:48:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.303+0000 7f7129fdb640 1 --2- 172.21.15.28:0/2550233624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7124100e70 0x7f7124198190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f711400e9f0 tx=0x7f711400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.303+0000 7f711affd640 1 -- 172.21.15.28:0/2550233624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711400cd80 con 0x7f7124100e70 2024-04-21T08:48:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.303+0000 7f711affd640 1 -- 172.21.15.28:0/2550233624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71140108e0 con 0x7f7124100e70 2024-04-21T08:48:42.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.303+0000 7f712c23f640 1 -- 172.21.15.28:0/2550233624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71241a00a0 con 0x7f7124100e70 2024-04-21T08:48:42.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.303+0000 7f711affd640 1 -- 172.21.15.28:0/2550233624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7114018980 con 0x7f7124100e70 2024-04-21T08:48:42.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.303+0000 7f712c23f640 1 -- 172.21.15.28:0/2550233624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71241a05c0 con 0x7f7124100e70 2024-04-21T08:48:42.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.304+0000 7f712c23f640 1 -- 172.21.15.28:0/2550233624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7124108180 con 0x7f7124100e70 2024-04-21T08:48:42.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.305+0000 7f711affd640 1 -- 172.21.15.28:0/2550233624 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7114010430 con 0x7f7124100e70 2024-04-21T08:48:42.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.306+0000 7f711affd640 1 --2- 172.21.15.28:0/2550233624 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71100a3bb0 0x7f71100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.306+0000 7f71297da640 1 --2- 172.21.15.28:0/2550233624 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71100a3bb0 0x7f71100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.307+0000 7f711affd640 1 -- 172.21.15.28:0/2550233624 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f71140d0790 con 0x7f7124100e70 2024-04-21T08:48:42.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.307+0000 7f71297da640 1 --2- 172.21.15.28:0/2550233624 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71100a3bb0 0x7f71100a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f712002c240 tx=0x7f712002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.310+0000 7f711affd640 1 -- 172.21.15.28:0/2550233624 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f711409cf60 con 0x7f7124100e70 2024-04-21T08:48:42.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.510+0000 7f712c23f640 1 -- 172.21.15.28:0/2550233624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7124062b50 con 0x7f7124100e70 2024-04-21T08:48:42.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.511+0000 7f711affd640 1 -- 172.21.15.28:0/2550233624 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18429 (secure 0 0 0) 0x7f71140a1260 con 0x7f7124100e70 2024-04-21T08:48:42.512 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:42.512 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:47:58.562865+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:48:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.513+0000 7f712c23f640 1 -- 172.21.15.28:0/2550233624 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71100a3bb0 msgr2=0x7f71100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.513+0000 7f712c23f640 1 --2- 172.21.15.28:0/2550233624 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71100a3bb0 0x7f71100a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f712002c240 tx=0x7f712002c7e0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.513+0000 7f712c23f640 1 -- 172.21.15.28:0/2550233624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7124100e70 msgr2=0x7f7124198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.513+0000 7f712c23f640 1 --2- 172.21.15.28:0/2550233624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7124100e70 0x7f7124198190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f711400e9f0 tx=0x7f711400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.513+0000 7f712c23f640 1 -- 172.21.15.28:0/2550233624 shutdown_connections 2024-04-21T08:48:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.513+0000 7f712c23f640 1 --2- 172.21.15.28:0/2550233624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7124102980 0x7f712419f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.513+0000 7f712c23f640 1 --2- 172.21.15.28:0/2550233624 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7124101fc0 0x7f71241986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.513+0000 7f712c23f640 1 --2- 172.21.15.28:0/2550233624 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71100a3bb0 0x7f71100a6070 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.513+0000 7f712c23f640 1 --2- 172.21.15.28:0/2550233624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7124100e70 0x7f7124198190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.513+0000 7f712c23f640 1 -- 172.21.15.28:0/2550233624 >> 172.21.15.28:0/2550233624 conn(0x7f71240fc3e0 msgr2=0x7f7124105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.513+0000 7f712c23f640 1 -- 172.21.15.28:0/2550233624 shutdown_connections 2024-04-21T08:48:42.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.513+0000 7f712c23f640 1 -- 172.21.15.28:0/2550233624 wait complete. 2024-04-21T08:48:42.526 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T08:48:42.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:48:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.633+0000 7fa0587c7640 1 -- 172.21.15.28:0/1647853733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa050102a30 msgr2=0x7fa050104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.633+0000 7fa0587c7640 1 --2- 172.21.15.28:0/1647853733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa050102a30 0x7fa050104e20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009900 tx=0x7fa04c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa0587c7640 1 -- 172.21.15.28:0/1647853733 shutdown_connections 2024-04-21T08:48:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa0587c7640 1 --2- 172.21.15.28:0/1647853733 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa050107c90 0x7fa05010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa0587c7640 1 --2- 172.21.15.28:0/1647853733 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa050105360 0x7fa050107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa0587c7640 1 --2- 172.21.15.28:0/1647853733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa050102a30 0x7fa050104e20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa0587c7640 1 -- 172.21.15.28:0/1647853733 >> 172.21.15.28:0/1647853733 conn(0x7fa0500fc440 msgr2=0x7fa0500fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa0587c7640 1 -- 172.21.15.28:0/1647853733 shutdown_connections 2024-04-21T08:48:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa0587c7640 1 -- 172.21.15.28:0/1647853733 wait complete. 2024-04-21T08:48:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa0587c7640 1 Processor -- start 2024-04-21T08:48:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa0587c7640 1 -- start start 2024-04-21T08:48:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa0587c7640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa050102a30 0x7fa050198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa0587c7640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa050105360 0x7fa050198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa0587c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa050107c90 0x7fa05019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa0587c7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa05010daf0 con 0x7fa050107c90 2024-04-21T08:48:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa0587c7640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa05010d970 con 0x7fa050105360 2024-04-21T08:48:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa0587c7640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa05010dc70 con 0x7fa050102a30 2024-04-21T08:48:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa056d64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa050107c90 0x7fa05019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa056d64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa050107c90 0x7fa05019f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45660/0 (socket says 172.21.15.28:45660) 2024-04-21T08:48:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.634+0000 7fa056d64640 1 -- 172.21.15.28:0/1703248501 learned_addr learned my addr 172.21.15.28:0/1703248501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.635+0000 7fa056563640 1 --2- 172.21.15.28:0/1703248501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa050102a30 0x7fa050198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.635+0000 7fa055d62640 1 --2- 172.21.15.28:0/1703248501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa050105360 0x7fa050198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.635+0000 7fa056d64640 1 -- 172.21.15.28:0/1703248501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa050102a30 msgr2=0x7fa050198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.635+0000 7fa056d64640 1 --2- 172.21.15.28:0/1703248501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa050102a30 0x7fa050198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.635+0000 7fa056d64640 1 -- 172.21.15.28:0/1703248501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa050105360 msgr2=0x7fa050198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.635+0000 7fa056d64640 1 --2- 172.21.15.28:0/1703248501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa050105360 0x7fa050198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.635+0000 7fa056d64640 1 -- 172.21.15.28:0/1703248501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa05019fe30 con 0x7fa050107c90 2024-04-21T08:48:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.635+0000 7fa056d64640 1 --2- 172.21.15.28:0/1703248501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa050107c90 0x7fa05019f7c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa04800eaa0 tx=0x7fa04800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.635+0000 7fa0477fe640 1 -- 172.21.15.28:0/1703248501 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04800cdb0 con 0x7fa050107c90 2024-04-21T08:48:42.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.635+0000 7fa0477fe640 1 -- 172.21.15.28:0/1703248501 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa048005020 con 0x7fa050107c90 2024-04-21T08:48:42.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.635+0000 7fa0587c7640 1 -- 172.21.15.28:0/1703248501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0501a0120 con 0x7fa050107c90 2024-04-21T08:48:42.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.635+0000 7fa0477fe640 1 -- 172.21.15.28:0/1703248501 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0480116d0 con 0x7fa050107c90 2024-04-21T08:48:42.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.635+0000 7fa0587c7640 1 -- 172.21.15.28:0/1703248501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0501a0620 con 0x7fa050107c90 2024-04-21T08:48:42.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.636+0000 7fa0587c7640 1 -- 172.21.15.28:0/1703248501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0500fdf30 con 0x7fa050107c90 2024-04-21T08:48:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.637+0000 7fa0477fe640 1 -- 172.21.15.28:0/1703248501 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa0480051d0 con 0x7fa050107c90 2024-04-21T08:48:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.638+0000 7fa0477fe640 1 --2- 172.21.15.28:0/1703248501 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa03c0a3bb0 0x7fa03c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.638+0000 7fa056563640 1 --2- 172.21.15.28:0/1703248501 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa03c0a3bb0 0x7fa03c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.638+0000 7fa0477fe640 1 -- 172.21.15.28:0/1703248501 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa0480d04a0 con 0x7fa050107c90 2024-04-21T08:48:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.638+0000 7fa056563640 1 --2- 172.21.15.28:0/1703248501 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa03c0a3bb0 0x7fa03c0a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009230 tx=0x7fa04c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.641+0000 7fa0477fe640 1 -- 172.21.15.28:0/1703248501 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa048016030 con 0x7fa050107c90 2024-04-21T08:48:42.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.829+0000 7fa0587c7640 1 -- 172.21.15.28:0/1703248501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa05002d5c0 con 0x7fa050107c90 2024-04-21T08:48:42.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.830+0000 7fa0477fe640 1 -- 172.21.15.28:0/1703248501 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fa04809ccd0 con 0x7fa050107c90 2024-04-21T08:48:42.831 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.832+0000 7fa0587c7640 1 -- 172.21.15.28:0/1703248501 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa03c0a3bb0 msgr2=0x7fa03c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.832+0000 7fa0587c7640 1 --2- 172.21.15.28:0/1703248501 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa03c0a3bb0 0x7fa03c0a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009230 tx=0x7fa04c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.832+0000 7fa0587c7640 1 -- 172.21.15.28:0/1703248501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa050107c90 msgr2=0x7fa05019f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.832+0000 7fa0587c7640 1 --2- 172.21.15.28:0/1703248501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa050107c90 0x7fa05019f7c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa04800eaa0 tx=0x7fa04800ef60 comp rx=0 tx=0).stop 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.832+0000 7fa0587c7640 1 -- 172.21.15.28:0/1703248501 shutdown_connections 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.832+0000 7fa0587c7640 1 --2- 172.21.15.28:0/1703248501 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa03c0a3bb0 0x7fa03c0a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.832+0000 7fa0587c7640 1 --2- 172.21.15.28:0/1703248501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa050107c90 0x7fa05019f7c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.832+0000 7fa0587c7640 1 --2- 172.21.15.28:0/1703248501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa050105360 0x7fa050198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.832+0000 7fa0587c7640 1 --2- 172.21.15.28:0/1703248501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa050102a30 0x7fa050198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.832+0000 7fa0587c7640 1 -- 172.21.15.28:0/1703248501 >> 172.21.15.28:0/1703248501 conn(0x7fa0500fc440 msgr2=0x7fa050103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.832+0000 7fa0587c7640 1 -- 172.21.15.28:0/1703248501 shutdown_connections 2024-04-21T08:48:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.832+0000 7fa0587c7640 1 -- 172.21.15.28:0/1703248501 wait complete. 2024-04-21T08:48:42.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:42 smithi028 ceph-mon[28003]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:42.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:42.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:42.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:42.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:42.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:42.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:42.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:42 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] up:boot 2024-04-21T08:48:42.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:42 smithi028 ceph-mon[28003]: fsmap 12 up:standby 2024-04-21T08:48:42.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:42 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:48:42.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:42 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3533033071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:42.843 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:42 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2550233624' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:42.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-21T08:48:42.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.949+0000 7f108965c640 1 -- 172.21.15.28:0/1452512802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1084103190 msgr2=0x7f1084103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.949+0000 7f108965c640 1 --2- 172.21.15.28:0/1452512802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1084103190 0x7f1084103610 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1078009900 tx=0x7f107802bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:42.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.949+0000 7f108965c640 1 -- 172.21.15.28:0/1452512802 shutdown_connections 2024-04-21T08:48:42.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.949+0000 7f108965c640 1 --2- 172.21.15.28:0/1452512802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1084103b50 0x7f108410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.949+0000 7f108965c640 1 --2- 172.21.15.28:0/1452512802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1084103190 0x7f1084103610 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.949+0000 7f108965c640 1 --2- 172.21.15.28:0/1452512802 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1084101fa0 0x7f10841023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.949+0000 7f108965c640 1 -- 172.21.15.28:0/1452512802 >> 172.21.15.28:0/1452512802 conn(0x7f10840fd530 msgr2=0x7f10840ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:42.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.949+0000 7f108965c640 1 -- 172.21.15.28:0/1452512802 shutdown_connections 2024-04-21T08:48:42.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.949+0000 7f108965c640 1 -- 172.21.15.28:0/1452512802 wait complete. 2024-04-21T08:48:42.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.949+0000 7f108965c640 1 Processor -- start 2024-04-21T08:48:42.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f108965c640 1 -- start start 2024-04-21T08:48:42.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f108965c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1084101fa0 0x7f1084198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f108965c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1084103190 0x7f10841986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f108965c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1084103b50 0x7f108419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f108965c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f108410db30 con 0x7f1084103190 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f108965c640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f108410d9b0 con 0x7f1084101fa0 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f108965c640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f108410dcb0 con 0x7f1084103b50 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f10827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1084103190 0x7f10841986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f1082ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1084101fa0 0x7f1084198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f10827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1084103190 0x7f10841986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45686/0 (socket says 172.21.15.28:45686) 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f10827fc640 1 -- 172.21.15.28:0/2607020617 learned_addr learned my addr 172.21.15.28:0/2607020617 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f1082ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1084101fa0 0x7f1084198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:48210/0 (socket says 172.21.15.28:48210) 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f10837fe640 1 --2- 172.21.15.28:0/2607020617 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1084103b50 0x7f108419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f10827fc640 1 -- 172.21.15.28:0/2607020617 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1084103b50 msgr2=0x7f108419f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f10827fc640 1 --2- 172.21.15.28:0/2607020617 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1084103b50 0x7f108419f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f10827fc640 1 -- 172.21.15.28:0/2607020617 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1084101fa0 msgr2=0x7f1084198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f10827fc640 1 --2- 172.21.15.28:0/2607020617 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1084101fa0 0x7f1084198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f10827fc640 1 -- 172.21.15.28:0/2607020617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f108419fdb0 con 0x7f1084103190 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f1082ffd640 1 --2- 172.21.15.28:0/2607020617 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1084101fa0 0x7f1084198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f10827fc640 1 --2- 172.21.15.28:0/2607020617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1084103190 0x7f10841986d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1078000c00 tx=0x7f10780040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.950+0000 7f106bfff640 1 -- 172.21.15.28:0/2607020617 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1078043020 con 0x7f1084103190 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.951+0000 7f106bfff640 1 -- 172.21.15.28:0/2607020617 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10780044c0 con 0x7f1084103190 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.951+0000 7f108965c640 1 -- 172.21.15.28:0/2607020617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10841a0040 con 0x7f1084103190 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.951+0000 7f106bfff640 1 -- 172.21.15.28:0/2607020617 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1078003c60 con 0x7f1084103190 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.951+0000 7f108965c640 1 -- 172.21.15.28:0/2607020617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10841a05e0 con 0x7f1084103190 2024-04-21T08:48:42.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.952+0000 7f108965c640 1 -- 172.21.15.28:0/2607020617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1084109340 con 0x7f1084103190 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.953+0000 7f106bfff640 1 -- 172.21.15.28:0/2607020617 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1078046020 con 0x7f1084103190 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.953+0000 7f106bfff640 1 --2- 172.21.15.28:0/2607020617 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f106c0a3c30 0x7f106c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.954+0000 7f1082ffd640 1 --2- 172.21.15.28:0/2607020617 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f106c0a3c30 0x7f106c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.954+0000 7f106bfff640 1 -- 172.21.15.28:0/2607020617 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f10780f25c0 con 0x7f1084103190 2024-04-21T08:48:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.954+0000 7f1082ffd640 1 --2- 172.21.15.28:0/2607020617 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f106c0a3c30 0x7f106c0a60f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1084102ff0 tx=0x7f107c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:42.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:42.957+0000 7f106bfff640 1 -- 172.21.15.28:0/2607020617 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f107804d020 con 0x7f1084103190 2024-04-21T08:48:42.968 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:42 smithi171 ceph-mon[31358]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:42.968 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:42.968 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:42.968 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:42.968 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:42.968 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:42.968 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:42.968 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:42 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] up:boot 2024-04-21T08:48:42.968 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:42 smithi171 ceph-mon[31358]: fsmap 12 up:standby 2024-04-21T08:48:42.968 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:42 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:48:42.968 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:42 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3533033071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:42.968 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:42 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2550233624' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:43.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:42 smithi152 ceph-mon[31401]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:43.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:42 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] up:boot 2024-04-21T08:48:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:42 smithi152 ceph-mon[31401]: fsmap 12 up:standby 2024-04-21T08:48:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:42 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:48:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:42 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3533033071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:43.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:42 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2550233624' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:43.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.147+0000 7f108965c640 1 -- 172.21.15.28:0/2607020617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f108402d810 con 0x7f1084103190 2024-04-21T08:48:43.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.147+0000 7f106bfff640 1 -- 172.21.15.28:0/2607020617 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f10780bf4c0 con 0x7f1084103190 2024-04-21T08:48:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:43.148 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-04-21T08:48:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T08:48:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-04-21T08:48:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-04-21T08:48:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:48:43.149 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-04-21T08:48:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.149+0000 7f108965c640 1 -- 172.21.15.28:0/2607020617 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f106c0a3c30 msgr2=0x7f106c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.149+0000 7f108965c640 1 --2- 172.21.15.28:0/2607020617 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f106c0a3c30 0x7f106c0a60f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1084102ff0 tx=0x7f107c005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.149+0000 7f108965c640 1 -- 172.21.15.28:0/2607020617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1084103190 msgr2=0x7f10841986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.149+0000 7f108965c640 1 --2- 172.21.15.28:0/2607020617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1084103190 0x7f10841986d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1078000c00 tx=0x7f10780040c0 comp rx=0 tx=0).stop 2024-04-21T08:48:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.149+0000 7f108965c640 1 -- 172.21.15.28:0/2607020617 shutdown_connections 2024-04-21T08:48:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.149+0000 7f108965c640 1 --2- 172.21.15.28:0/2607020617 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1084103b50 0x7f108419f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.149+0000 7f108965c640 1 --2- 172.21.15.28:0/2607020617 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f106c0a3c30 0x7f106c0a60f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.149+0000 7f108965c640 1 --2- 172.21.15.28:0/2607020617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1084103190 0x7f10841986d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.149+0000 7f108965c640 1 --2- 172.21.15.28:0/2607020617 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1084101fa0 0x7f1084198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.149+0000 7f108965c640 1 -- 172.21.15.28:0/2607020617 >> 172.21.15.28:0/2607020617 conn(0x7f10840fd530 msgr2=0x7f1084106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.150+0000 7f108965c640 1 -- 172.21.15.28:0/2607020617 shutdown_connections 2024-04-21T08:48:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.150+0000 7f108965c640 1 -- 172.21.15.28:0/2607020617 wait complete. 2024-04-21T08:48:43.161 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-21T08:48:43.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-21T08:48:43.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.274+0000 7ff94efb6640 1 -- 172.21.15.28:0/2643382479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff948101fa0 msgr2=0x7ff9481023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:43.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.274+0000 7ff94efb6640 1 --2- 172.21.15.28:0/2643382479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff948101fa0 0x7ff9481023c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff93c009900 tx=0x7ff93c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:43.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.274+0000 7ff94efb6640 1 -- 172.21.15.28:0/2643382479 shutdown_connections 2024-04-21T08:48:43.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.274+0000 7ff94efb6640 1 --2- 172.21.15.28:0/2643382479 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff948103b50 0x7ff94810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:43.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.274+0000 7ff94efb6640 1 --2- 172.21.15.28:0/2643382479 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff948103190 0x7ff948103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:43.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.274+0000 7ff94efb6640 1 --2- 172.21.15.28:0/2643382479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff948101fa0 0x7ff9481023c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:43.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.274+0000 7ff94efb6640 1 -- 172.21.15.28:0/2643382479 >> 172.21.15.28:0/2643382479 conn(0x7ff9480fd530 msgr2=0x7ff9480ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:43.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.274+0000 7ff94efb6640 1 -- 172.21.15.28:0/2643382479 shutdown_connections 2024-04-21T08:48:43.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.274+0000 7ff94efb6640 1 -- 172.21.15.28:0/2643382479 wait complete. 2024-04-21T08:48:43.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.274+0000 7ff94efb6640 1 Processor -- start 2024-04-21T08:48:43.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94efb6640 1 -- start start 2024-04-21T08:48:43.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94efb6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff948101fa0 0x7ff9481981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:43.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94efb6640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff948103190 0x7ff948198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:43.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94efb6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff948103b50 0x7ff94819f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:43.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94efb6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff94810dae0 con 0x7ff948101fa0 2024-04-21T08:48:43.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94efb6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff94810d960 con 0x7ff948103b50 2024-04-21T08:48:43.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94efb6640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff94810dc60 con 0x7ff948103190 2024-04-21T08:48:43.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94cd52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff948101fa0 0x7ff9481981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:43.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94cd52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff948101fa0 0x7ff9481981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45704/0 (socket says 172.21.15.28:45704) 2024-04-21T08:48:43.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94cd52640 1 -- 172.21.15.28:0/372475529 learned_addr learned my addr 172.21.15.28:0/372475529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:43.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff947fff640 1 --2- 172.21.15.28:0/372475529 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff948103190 0x7ff948198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:43.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94d553640 1 --2- 172.21.15.28:0/372475529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff948103b50 0x7ff94819f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:43.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94cd52640 1 -- 172.21.15.28:0/372475529 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff948103190 msgr2=0x7ff948198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:43.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94cd52640 1 --2- 172.21.15.28:0/372475529 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff948103190 0x7ff948198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:43.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94cd52640 1 -- 172.21.15.28:0/372475529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff948103b50 msgr2=0x7ff94819f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:43.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94cd52640 1 --2- 172.21.15.28:0/372475529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff948103b50 0x7ff94819f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:43.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94cd52640 1 -- 172.21.15.28:0/372475529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff94819fe10 con 0x7ff948101fa0 2024-04-21T08:48:43.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff94cd52640 1 --2- 172.21.15.28:0/372475529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff948101fa0 0x7ff9481981f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff93c000c00 tx=0x7ff93c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:43.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.275+0000 7ff945ffb640 1 -- 172.21.15.28:0/372475529 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93c043020 con 0x7ff948101fa0 2024-04-21T08:48:43.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.276+0000 7ff94efb6640 1 -- 172.21.15.28:0/372475529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9481a00a0 con 0x7ff948101fa0 2024-04-21T08:48:43.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.276+0000 7ff945ffb640 1 -- 172.21.15.28:0/372475529 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff93c004350 con 0x7ff948101fa0 2024-04-21T08:48:43.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.276+0000 7ff945ffb640 1 -- 172.21.15.28:0/372475529 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93c003b80 con 0x7ff948101fa0 2024-04-21T08:48:43.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.276+0000 7ff94efb6640 1 -- 172.21.15.28:0/372475529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff948074360 con 0x7ff948101fa0 2024-04-21T08:48:43.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.277+0000 7ff94efb6640 1 -- 172.21.15.28:0/372475529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9481093e0 con 0x7ff948101fa0 2024-04-21T08:48:43.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.278+0000 7ff945ffb640 1 -- 172.21.15.28:0/372475529 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff93c046020 con 0x7ff948101fa0 2024-04-21T08:48:43.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.279+0000 7ff945ffb640 1 --2- 172.21.15.28:0/372475529 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9380a7fc0 0x7ff9380aa480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:43.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.279+0000 7ff947fff640 1 --2- 172.21.15.28:0/372475529 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9380a7fc0 0x7ff9380aa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:43.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.279+0000 7ff945ffb640 1 -- 172.21.15.28:0/372475529 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff93c0f2060 con 0x7ff948101fa0 2024-04-21T08:48:43.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.279+0000 7ff947fff640 1 --2- 172.21.15.28:0/372475529 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9380a7fc0 0x7ff9380aa480 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff948199700 tx=0x7ff934005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:43.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.283+0000 7ff945ffb640 1 -- 172.21.15.28:0/372475529 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff93c04d020 con 0x7ff948101fa0 2024-04-21T08:48:43.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:43.538+0000 7ff94efb6640 1 -- 172.21.15.28:0/372475529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ff94802d7c0 con 0x7ff948101fa0 2024-04-21T08:48:44.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:43 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1703248501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:48:44.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:43 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2607020617' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T08:48:44.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:43 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:44.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:43 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:44.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:43 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/372475529' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T08:48:44.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:43 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:44.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:43 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:44.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:43 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1703248501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:48:44.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:43 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2607020617' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T08:48:44.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:43 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:44.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:43 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:44.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:43 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/372475529' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T08:48:44.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:43 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:44.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:43 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:44.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:43 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1703248501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:48:44.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:43 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2607020617' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T08:48:44.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:43 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:44.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:43 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:44.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:43 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/372475529' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T08:48:44.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:43 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:44.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:43 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:44.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.529+0000 7ff945ffb640 1 -- 172.21.15.28:0/372475529 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7ff93c0beee0 con 0x7ff948101fa0 2024-04-21T08:48:44.529 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-04-21T08:48:44.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.531+0000 7ff94efb6640 1 -- 172.21.15.28:0/372475529 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9380a7fc0 msgr2=0x7ff9380aa480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:44.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.531+0000 7ff94efb6640 1 --2- 172.21.15.28:0/372475529 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9380a7fc0 0x7ff9380aa480 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff948199700 tx=0x7ff934005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:44.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.531+0000 7ff94efb6640 1 -- 172.21.15.28:0/372475529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff948101fa0 msgr2=0x7ff9481981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:44.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.531+0000 7ff94efb6640 1 --2- 172.21.15.28:0/372475529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff948101fa0 0x7ff9481981f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff93c000c00 tx=0x7ff93c005dc0 comp rx=0 tx=0).stop 2024-04-21T08:48:44.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.532+0000 7ff94efb6640 1 -- 172.21.15.28:0/372475529 shutdown_connections 2024-04-21T08:48:44.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.532+0000 7ff94efb6640 1 --2- 172.21.15.28:0/372475529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff948103b50 0x7ff94819f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:44.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.532+0000 7ff94efb6640 1 --2- 172.21.15.28:0/372475529 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff948103190 0x7ff948198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:44.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.532+0000 7ff94efb6640 1 --2- 172.21.15.28:0/372475529 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9380a7fc0 0x7ff9380aa480 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:44.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.532+0000 7ff94efb6640 1 --2- 172.21.15.28:0/372475529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff948101fa0 0x7ff9481981f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:44.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.532+0000 7ff94efb6640 1 -- 172.21.15.28:0/372475529 >> 172.21.15.28:0/372475529 conn(0x7ff9480fd530 msgr2=0x7ff9481065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:44.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.532+0000 7ff94efb6640 1 -- 172.21.15.28:0/372475529 shutdown_connections 2024-04-21T08:48:44.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.532+0000 7ff94efb6640 1 -- 172.21.15.28:0/372475529 wait complete. 2024-04-21T08:48:44.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-21T08:48:44.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.655+0000 7f269cefd640 1 -- 172.21.15.28:0/2306601220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26981031a0 msgr2=0x7f2698103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:44.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.655+0000 7f269cefd640 1 --2- 172.21.15.28:0/2306601220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26981031a0 0x7f2698103620 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f268c009900 tx=0x7f268c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:44.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.655+0000 7f269cefd640 1 -- 172.21.15.28:0/2306601220 shutdown_connections 2024-04-21T08:48:44.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.655+0000 7f269cefd640 1 --2- 172.21.15.28:0/2306601220 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2698103b60 0x7f269810a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:44.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.655+0000 7f269cefd640 1 --2- 172.21.15.28:0/2306601220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26981031a0 0x7f2698103620 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:44.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.655+0000 7f269cefd640 1 --2- 172.21.15.28:0/2306601220 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2698101fb0 0x7f26981023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.655+0000 7f269cefd640 1 -- 172.21.15.28:0/2306601220 >> 172.21.15.28:0/2306601220 conn(0x7f26980fd520 msgr2=0x7f26980ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.655+0000 7f269cefd640 1 -- 172.21.15.28:0/2306601220 shutdown_connections 2024-04-21T08:48:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.655+0000 7f269cefd640 1 -- 172.21.15.28:0/2306601220 wait complete. 2024-04-21T08:48:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.656+0000 7f269cefd640 1 Processor -- start 2024-04-21T08:48:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.656+0000 7f269cefd640 1 -- start start 2024-04-21T08:48:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.656+0000 7f269cefd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2698101fb0 0x7f26981981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.656+0000 7f269cefd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26981031a0 0x7f2698198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.656+0000 7f269cefd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2698103b60 0x7f269819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.656+0000 7f269cefd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f269810daf0 con 0x7f26981031a0 2024-04-21T08:48:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.656+0000 7f269cefd640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f269810d970 con 0x7f2698103b60 2024-04-21T08:48:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.656+0000 7f269cefd640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f269810dc70 con 0x7f2698101fb0 2024-04-21T08:48:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.656+0000 7f2695d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26981031a0 0x7f2698198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.656+0000 7f2695d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26981031a0 0x7f2698198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38582/0 (socket says 172.21.15.28:38582) 2024-04-21T08:48:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.656+0000 7f2695d9b640 1 -- 172.21.15.28:0/1998232593 learned_addr learned my addr 172.21.15.28:0/1998232593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.656+0000 7f269659c640 1 --2- 172.21.15.28:0/1998232593 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2698101fb0 0x7f26981981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.656+0000 7f2696d9d640 1 --2- 172.21.15.28:0/1998232593 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2698103b60 0x7f269819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.656+0000 7f2695d9b640 1 -- 172.21.15.28:0/1998232593 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2698101fb0 msgr2=0x7f26981981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.657+0000 7f2695d9b640 1 --2- 172.21.15.28:0/1998232593 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2698101fb0 0x7f26981981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.657+0000 7f2695d9b640 1 -- 172.21.15.28:0/1998232593 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2698103b60 msgr2=0x7f269819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.657+0000 7f2695d9b640 1 --2- 172.21.15.28:0/1998232593 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2698103b60 0x7f269819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.657+0000 7f2695d9b640 1 -- 172.21.15.28:0/1998232593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f269819fe00 con 0x7f26981031a0 2024-04-21T08:48:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.657+0000 7f2696d9d640 1 --2- 172.21.15.28:0/1998232593 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2698103b60 0x7f269819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.657+0000 7f2695d9b640 1 --2- 172.21.15.28:0/1998232593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26981031a0 0x7f2698198720 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f268c006010 tx=0x7f268c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.657+0000 7f26877fe640 1 -- 172.21.15.28:0/1998232593 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c043020 con 0x7f26981031a0 2024-04-21T08:48:44.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.657+0000 7f26877fe640 1 -- 172.21.15.28:0/1998232593 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f268c004aa0 con 0x7f26981031a0 2024-04-21T08:48:44.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.657+0000 7f269cefd640 1 -- 172.21.15.28:0/1998232593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26981a0090 con 0x7f26981031a0 2024-04-21T08:48:44.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.657+0000 7f26877fe640 1 -- 172.21.15.28:0/1998232593 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c003c70 con 0x7f26981031a0 2024-04-21T08:48:44.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.657+0000 7f269cefd640 1 -- 172.21.15.28:0/1998232593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26981a0630 con 0x7f26981031a0 2024-04-21T08:48:44.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.658+0000 7f269cefd640 1 -- 172.21.15.28:0/1998232593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26981092f0 con 0x7f26981031a0 2024-04-21T08:48:44.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.659+0000 7f26877fe640 1 -- 172.21.15.28:0/1998232593 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f268c046020 con 0x7f26981031a0 2024-04-21T08:48:44.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.660+0000 7f26877fe640 1 --2- 172.21.15.28:0/1998232593 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f26800a3c30 0x7f26800a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:44.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.660+0000 7f269659c640 1 --2- 172.21.15.28:0/1998232593 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f26800a3c30 0x7f26800a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:44.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.660+0000 7f26877fe640 1 -- 172.21.15.28:0/1998232593 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f268c0f2490 con 0x7f26981031a0 2024-04-21T08:48:44.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.660+0000 7f269659c640 1 --2- 172.21.15.28:0/1998232593 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f26800a3c30 0x7f26800a60f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2698103000 tx=0x7f268800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:44.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.665+0000 7f26877fe640 1 -- 172.21.15.28:0/1998232593 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f268c04d020 con 0x7f26981031a0 2024-04-21T08:48:44.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:44 smithi028 ceph-mon[28003]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:44.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:44 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:44.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:44 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:44.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:44 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:44.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:44 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:44.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:44 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:44.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:44 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:44.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:44 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/372475529' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T08:48:44.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:44 smithi028 ceph-mon[28003]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T08:48:44.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:44.859+0000 7f269cefd640 1 -- 172.21.15.28:0/1998232593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f269802d5c0 con 0x7f26981031a0 2024-04-21T08:48:45.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:44 smithi152 ceph-mon[31401]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:45.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:44 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:45.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:44 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:45.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:44 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:45.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:44 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:45.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:44 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:45.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:44 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:45.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:44 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/372475529' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T08:48:45.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:44 smithi152 ceph-mon[31401]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T08:48:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:44 smithi171 ceph-mon[31358]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:44 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:44 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:44 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:44 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:44 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:44 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:44 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/372475529' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T08:48:45.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:44 smithi171 ceph-mon[31358]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T08:48:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.532+0000 7f26877fe640 1 -- 172.21.15.28:0/1998232593 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7f268c0bf100 con 0x7f26981031a0 2024-04-21T08:48:45.532 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-04-21T08:48:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.535+0000 7f269cefd640 1 -- 172.21.15.28:0/1998232593 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f26800a3c30 msgr2=0x7f26800a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.535+0000 7f269cefd640 1 --2- 172.21.15.28:0/1998232593 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f26800a3c30 0x7f26800a60f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2698103000 tx=0x7f268800b390 comp rx=0 tx=0).stop 2024-04-21T08:48:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.535+0000 7f269cefd640 1 -- 172.21.15.28:0/1998232593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26981031a0 msgr2=0x7f2698198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.535+0000 7f269cefd640 1 --2- 172.21.15.28:0/1998232593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26981031a0 0x7f2698198720 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f268c006010 tx=0x7f268c0042f0 comp rx=0 tx=0).stop 2024-04-21T08:48:45.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.535+0000 7f269cefd640 1 -- 172.21.15.28:0/1998232593 shutdown_connections 2024-04-21T08:48:45.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.535+0000 7f269cefd640 1 --2- 172.21.15.28:0/1998232593 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2698103b60 0x7f269819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:45.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.535+0000 7f269cefd640 1 --2- 172.21.15.28:0/1998232593 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f26800a3c30 0x7f26800a60f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:45.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.535+0000 7f269cefd640 1 --2- 172.21.15.28:0/1998232593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26981031a0 0x7f2698198720 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:45.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.535+0000 7f269cefd640 1 --2- 172.21.15.28:0/1998232593 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2698101fb0 0x7f26981981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:45.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.535+0000 7f269cefd640 1 -- 172.21.15.28:0/1998232593 >> 172.21.15.28:0/1998232593 conn(0x7f26980fd520 msgr2=0x7f26981064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:45.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.535+0000 7f269cefd640 1 -- 172.21.15.28:0/1998232593 shutdown_connections 2024-04-21T08:48:45.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.535+0000 7f269cefd640 1 -- 172.21.15.28:0/1998232593 wait complete. 2024-04-21T08:48:45.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-21T08:48:45.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.673+0000 7ff7493b2640 1 -- 172.21.15.28:0/2047772431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff744100e60 msgr2=0x7ff744101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:45.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.673+0000 7ff7493b2640 1 --2- 172.21.15.28:0/2047772431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff744100e60 0x7ff744101280 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff734009900 tx=0x7ff73402bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:45.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.673+0000 7ff7493b2640 1 -- 172.21.15.28:0/2047772431 shutdown_connections 2024-04-21T08:48:45.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.673+0000 7ff7493b2640 1 --2- 172.21.15.28:0/2047772431 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff744102a00 0x7ff744109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:45.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.673+0000 7ff7493b2640 1 --2- 172.21.15.28:0/2047772431 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff744102040 0x7ff7441024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:45.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.673+0000 7ff7493b2640 1 --2- 172.21.15.28:0/2047772431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff744100e60 0x7ff744101280 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:45.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.673+0000 7ff7493b2640 1 -- 172.21.15.28:0/2047772431 >> 172.21.15.28:0/2047772431 conn(0x7ff7440fc440 msgr2=0x7ff7440fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:45.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.673+0000 7ff7493b2640 1 -- 172.21.15.28:0/2047772431 shutdown_connections 2024-04-21T08:48:45.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.673+0000 7ff7493b2640 1 -- 172.21.15.28:0/2047772431 wait complete. 2024-04-21T08:48:45.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.673+0000 7ff7493b2640 1 Processor -- start 2024-04-21T08:48:45.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff7493b2640 1 -- start start 2024-04-21T08:48:45.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff7493b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff744100e60 0x7ff7441970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:45.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff7493b2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff744102040 0x7ff744197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:45.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff7493b2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff744102a00 0x7ff74419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:45.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff7493b2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff74410c9f0 con 0x7ff744100e60 2024-04-21T08:48:45.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff7493b2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff74410c870 con 0x7ff744102a00 2024-04-21T08:48:45.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff7493b2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff74410cb70 con 0x7ff744102040 2024-04-21T08:48:45.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff742ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff744100e60 0x7ff7441970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:45.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff742ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff744100e60 0x7ff7441970f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38602/0 (socket says 172.21.15.28:38602) 2024-04-21T08:48:45.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff742ffd640 1 -- 172.21.15.28:0/469491879 learned_addr learned my addr 172.21.15.28:0/469491879 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:45.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff73bfff640 1 --2- 172.21.15.28:0/469491879 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff744102040 0x7ff744197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:45.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff7437fe640 1 --2- 172.21.15.28:0/469491879 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff744102a00 0x7ff74419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:45.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff742ffd640 1 -- 172.21.15.28:0/469491879 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff744102040 msgr2=0x7ff744197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:45.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff742ffd640 1 --2- 172.21.15.28:0/469491879 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff744102040 0x7ff744197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:45.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff742ffd640 1 -- 172.21.15.28:0/469491879 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff744102a00 msgr2=0x7ff74419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:45.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff742ffd640 1 --2- 172.21.15.28:0/469491879 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff744102a00 0x7ff74419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:45.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff742ffd640 1 -- 172.21.15.28:0/469491879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff74419ed10 con 0x7ff744100e60 2024-04-21T08:48:45.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff73bfff640 1 --2- 172.21.15.28:0/469491879 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff744102040 0x7ff744197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:48:45.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.674+0000 7ff742ffd640 1 --2- 172.21.15.28:0/469491879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff744100e60 0x7ff7441970f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff734000c00 tx=0x7ff734004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:45.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.675+0000 7ff740ff9640 1 -- 172.21.15.28:0/469491879 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff734005000 con 0x7ff744100e60 2024-04-21T08:48:45.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.675+0000 7ff740ff9640 1 -- 172.21.15.28:0/469491879 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff734004aa0 con 0x7ff744100e60 2024-04-21T08:48:45.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.675+0000 7ff740ff9640 1 -- 172.21.15.28:0/469491879 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff734047480 con 0x7ff744100e60 2024-04-21T08:48:45.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.675+0000 7ff7493b2640 1 -- 172.21.15.28:0/469491879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff74406a550 con 0x7ff744100e60 2024-04-21T08:48:45.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.675+0000 7ff7493b2640 1 -- 172.21.15.28:0/469491879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff74406aa50 con 0x7ff744100e60 2024-04-21T08:48:45.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.676+0000 7ff7493b2640 1 -- 172.21.15.28:0/469491879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff74406b470 con 0x7ff744100e60 2024-04-21T08:48:45.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.678+0000 7ff740ff9640 1 -- 172.21.15.28:0/469491879 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff734003780 con 0x7ff744100e60 2024-04-21T08:48:45.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.678+0000 7ff740ff9640 1 --2- 172.21.15.28:0/469491879 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff7300a3bb0 0x7ff7300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:45.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.679+0000 7ff73bfff640 1 --2- 172.21.15.28:0/469491879 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff7300a3bb0 0x7ff7300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:45.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.679+0000 7ff740ff9640 1 -- 172.21.15.28:0/469491879 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7ff7340f4a20 con 0x7ff744100e60 2024-04-21T08:48:45.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.679+0000 7ff73bfff640 1 --2- 172.21.15.28:0/469491879 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff7300a3bb0 0x7ff7300a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff744198600 tx=0x7ff72c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:45.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.682+0000 7ff740ff9640 1 -- 172.21.15.28:0/469491879 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff7340c15b0 con 0x7ff744100e60 2024-04-21T08:48:45.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:45 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1998232593' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T08:48:45.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:45.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:45.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:45.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:45 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1998232593' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T08:48:45.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:45 smithi028 ceph-mon[28003]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T08:48:45.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:45.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:45.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:45.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:45.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:45.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:45.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:45.910+0000 7ff7493b2640 1 -- 172.21.15.28:0/469491879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff744107d40 con 0x7ff744100e60 2024-04-21T08:48:46.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:45 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1998232593' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T08:48:46.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:45 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1998232593' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T08:48:46.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:45 smithi152 ceph-mon[31401]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T08:48:46.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:46.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:46.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:45 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1998232593' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T08:48:46.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:45 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1998232593' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T08:48:46.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:45 smithi171 ceph-mon[31358]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T08:48:46.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:46.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:46.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:48:46.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.731+0000 7ff740ff9640 1 -- 172.21.15.28:0/469491879 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7ff7340c0eb0 con 0x7ff744100e60 2024-04-21T08:48:46.731 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-04-21T08:48:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.734+0000 7ff7493b2640 1 -- 172.21.15.28:0/469491879 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff7300a3bb0 msgr2=0x7ff7300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.734+0000 7ff7493b2640 1 --2- 172.21.15.28:0/469491879 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff7300a3bb0 0x7ff7300a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff744198600 tx=0x7ff72c00b410 comp rx=0 tx=0).stop 2024-04-21T08:48:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.734+0000 7ff7493b2640 1 -- 172.21.15.28:0/469491879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff744100e60 msgr2=0x7ff7441970f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.734+0000 7ff7493b2640 1 --2- 172.21.15.28:0/469491879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff744100e60 0x7ff7441970f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff734000c00 tx=0x7ff734004270 comp rx=0 tx=0).stop 2024-04-21T08:48:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.734+0000 7ff7493b2640 1 -- 172.21.15.28:0/469491879 shutdown_connections 2024-04-21T08:48:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.734+0000 7ff7493b2640 1 --2- 172.21.15.28:0/469491879 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff744102a00 0x7ff74419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.734+0000 7ff7493b2640 1 --2- 172.21.15.28:0/469491879 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff744102040 0x7ff744197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.734+0000 7ff7493b2640 1 --2- 172.21.15.28:0/469491879 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff7300a3bb0 0x7ff7300a6070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.734+0000 7ff7493b2640 1 --2- 172.21.15.28:0/469491879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff744100e60 0x7ff7441970f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.734+0000 7ff7493b2640 1 -- 172.21.15.28:0/469491879 >> 172.21.15.28:0/469491879 conn(0x7ff7440fc440 msgr2=0x7ff7441054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:46.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.734+0000 7ff7493b2640 1 -- 172.21.15.28:0/469491879 shutdown_connections 2024-04-21T08:48:46.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.734+0000 7ff7493b2640 1 -- 172.21.15.28:0/469491879 wait complete. 2024-04-21T08:48:46.750 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-21T08:48:46.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-21T08:48:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.863+0000 7efef5959640 1 -- 172.21.15.28:0/1240293728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef0101fe0 msgr2=0x7efef0102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.864+0000 7efef5959640 1 --2- 172.21.15.28:0/1240293728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef0101fe0 0x7efef0102460 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efee4009900 tx=0x7efee402bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.864+0000 7efef5959640 1 -- 172.21.15.28:0/1240293728 shutdown_connections 2024-04-21T08:48:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.864+0000 7efef5959640 1 --2- 172.21.15.28:0/1240293728 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efef01029a0 0x7efef01092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.864+0000 7efef5959640 1 --2- 172.21.15.28:0/1240293728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef0101fe0 0x7efef0102460 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.864+0000 7efef5959640 1 --2- 172.21.15.28:0/1240293728 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efef0100e90 0x7efef01012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.864+0000 7efef5959640 1 -- 172.21.15.28:0/1240293728 >> 172.21.15.28:0/1240293728 conn(0x7efef00fc440 msgr2=0x7efef00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.864+0000 7efef5959640 1 -- 172.21.15.28:0/1240293728 shutdown_connections 2024-04-21T08:48:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.864+0000 7efef5959640 1 -- 172.21.15.28:0/1240293728 wait complete. 2024-04-21T08:48:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.864+0000 7efef5959640 1 Processor -- start 2024-04-21T08:48:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.864+0000 7efef5959640 1 -- start start 2024-04-21T08:48:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.864+0000 7efef5959640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efef0100e90 0x7efef0198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efef5959640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef0101fe0 0x7efef0198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efef5959640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efef01029a0 0x7efef019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efef5959640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efef010c9d0 con 0x7efef0101fe0 2024-04-21T08:48:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efef5959640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efef010c850 con 0x7efef0100e90 2024-04-21T08:48:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efef5959640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7efef010cb50 con 0x7efef01029a0 2024-04-21T08:48:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efeee7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef0101fe0 0x7efef0198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efeee7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef0101fe0 0x7efef0198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38622/0 (socket says 172.21.15.28:38622) 2024-04-21T08:48:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efeee7fc640 1 -- 172.21.15.28:0/3115372268 learned_addr learned my addr 172.21.15.28:0/3115372268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efeeeffd640 1 --2- 172.21.15.28:0/3115372268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efef0100e90 0x7efef0198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efeef7fe640 1 --2- 172.21.15.28:0/3115372268 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efef01029a0 0x7efef019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efeee7fc640 1 -- 172.21.15.28:0/3115372268 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efef01029a0 msgr2=0x7efef019f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efeee7fc640 1 --2- 172.21.15.28:0/3115372268 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efef01029a0 0x7efef019f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efeee7fc640 1 -- 172.21.15.28:0/3115372268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efef0100e90 msgr2=0x7efef0198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efeee7fc640 1 --2- 172.21.15.28:0/3115372268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efef0100e90 0x7efef0198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efeee7fc640 1 -- 172.21.15.28:0/3115372268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef019fe70 con 0x7efef0101fe0 2024-04-21T08:48:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efeeeffd640 1 --2- 172.21.15.28:0/3115372268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efef0100e90 0x7efef0198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efeee7fc640 1 --2- 172.21.15.28:0/3115372268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef0101fe0 0x7efef0198790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7efee4000c00 tx=0x7efee4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efef4957640 1 -- 172.21.15.28:0/3115372268 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4043020 con 0x7efef0101fe0 2024-04-21T08:48:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efef5959640 1 -- 172.21.15.28:0/3115372268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef01a0100 con 0x7efef0101fe0 2024-04-21T08:48:46.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efef4957640 1 -- 172.21.15.28:0/3115372268 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee4004440 con 0x7efef0101fe0 2024-04-21T08:48:46.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efef4957640 1 -- 172.21.15.28:0/3115372268 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4003c90 con 0x7efef0101fe0 2024-04-21T08:48:46.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.865+0000 7efef5959640 1 -- 172.21.15.28:0/3115372268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef01a0650 con 0x7efef0101fe0 2024-04-21T08:48:46.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.866+0000 7efef5959640 1 -- 172.21.15.28:0/3115372268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef01a0850 con 0x7efef0101fe0 2024-04-21T08:48:46.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.867+0000 7efef4957640 1 -- 172.21.15.28:0/3115372268 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efee4046020 con 0x7efef0101fe0 2024-04-21T08:48:46.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.868+0000 7efef4957640 1 --2- 172.21.15.28:0/3115372268 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efed80a3bb0 0x7efed80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:46.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.868+0000 7efef4957640 1 -- 172.21.15.28:0/3115372268 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7efee40f2a40 con 0x7efef0101fe0 2024-04-21T08:48:46.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.868+0000 7efeeeffd640 1 --2- 172.21.15.28:0/3115372268 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efed80a3bb0 0x7efed80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:46.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.868+0000 7efeeeffd640 1 --2- 172.21.15.28:0/3115372268 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efed80a3bb0 0x7efed80a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efee0005d00 tx=0x7efee0005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:46.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:46.872+0000 7efef4957640 1 -- 172.21.15.28:0/3115372268 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efee404d020 con 0x7efef0101fe0 2024-04-21T08:48:47.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.068+0000 7efef5959640 1 -- 172.21.15.28:0/3115372268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7efef01a0a20 con 0x7efef0101fe0 2024-04-21T08:48:47.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:46 smithi152 ceph-mon[31401]: pgmap v287: 33 pgs: 6 creating+peering, 26 unknown, 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:47.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:46 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/469491879' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T08:48:47.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:46 smithi152 ceph-mon[31401]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T08:48:47.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:46 smithi152 ceph-mon[31401]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:48:47.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:46 smithi152 ceph-mon[31401]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T08:48:47.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:46 smithi028 ceph-mon[28003]: pgmap v287: 33 pgs: 6 creating+peering, 26 unknown, 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:47.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:46 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/469491879' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T08:48:47.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:46 smithi028 ceph-mon[28003]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T08:48:47.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:46 smithi028 ceph-mon[28003]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:48:47.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:46 smithi028 ceph-mon[28003]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T08:48:47.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:46 smithi171 ceph-mon[31358]: pgmap v287: 33 pgs: 6 creating+peering, 26 unknown, 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:48:47.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:46 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/469491879' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T08:48:47.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:46 smithi171 ceph-mon[31358]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T08:48:47.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:46 smithi171 ceph-mon[31358]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:48:47.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:46 smithi171 ceph-mon[31358]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T08:48:47.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.449+0000 7efef4957640 1 -- 172.21.15.28:0/3115372268 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7efee40bf550 con 0x7efef0101fe0 2024-04-21T08:48:47.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.451+0000 7efef5959640 1 -- 172.21.15.28:0/3115372268 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efed80a3bb0 msgr2=0x7efed80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:47.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.451+0000 7efef5959640 1 --2- 172.21.15.28:0/3115372268 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efed80a3bb0 0x7efed80a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efee0005d00 tx=0x7efee0005c70 comp rx=0 tx=0).stop 2024-04-21T08:48:47.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.451+0000 7efef5959640 1 -- 172.21.15.28:0/3115372268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef0101fe0 msgr2=0x7efef0198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:47.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.451+0000 7efef5959640 1 --2- 172.21.15.28:0/3115372268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef0101fe0 0x7efef0198790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7efee4000c00 tx=0x7efee4004040 comp rx=0 tx=0).stop 2024-04-21T08:48:47.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.452+0000 7efef5959640 1 -- 172.21.15.28:0/3115372268 shutdown_connections 2024-04-21T08:48:47.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.452+0000 7efef5959640 1 --2- 172.21.15.28:0/3115372268 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efef01029a0 0x7efef019f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:47.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.452+0000 7efef5959640 1 --2- 172.21.15.28:0/3115372268 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efed80a3bb0 0x7efed80a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:47.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.452+0000 7efef5959640 1 --2- 172.21.15.28:0/3115372268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef0101fe0 0x7efef0198790 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:47.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.452+0000 7efef5959640 1 --2- 172.21.15.28:0/3115372268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efef0100e90 0x7efef0198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:47.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.452+0000 7efef5959640 1 -- 172.21.15.28:0/3115372268 >> 172.21.15.28:0/3115372268 conn(0x7efef00fc440 msgr2=0x7efef006a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:47.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.452+0000 7efef5959640 1 -- 172.21.15.28:0/3115372268 shutdown_connections 2024-04-21T08:48:47.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.452+0000 7efef5959640 1 -- 172.21.15.28:0/3115372268 wait complete. 2024-04-21T08:48:47.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-21T08:48:47.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.577+0000 7fada9e26640 1 -- 172.21.15.28:0/389614503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada41029a0 msgr2=0x7fada41092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:47.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.577+0000 7fada9e26640 1 --2- 172.21.15.28:0/389614503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada41029a0 0x7fada41092c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fad9c009ca0 tx=0x7fad9c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:47.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.577+0000 7fada9e26640 1 -- 172.21.15.28:0/389614503 shutdown_connections 2024-04-21T08:48:47.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.577+0000 7fada9e26640 1 --2- 172.21.15.28:0/389614503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada41029a0 0x7fada41092c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:47.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.577+0000 7fada9e26640 1 --2- 172.21.15.28:0/389614503 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fada4101fe0 0x7fada4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:47.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.577+0000 7fada9e26640 1 --2- 172.21.15.28:0/389614503 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fada4100e90 0x7fada41012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:47.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.577+0000 7fada9e26640 1 -- 172.21.15.28:0/389614503 >> 172.21.15.28:0/389614503 conn(0x7fada40fc400 msgr2=0x7fada40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:47.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.577+0000 7fada9e26640 1 -- 172.21.15.28:0/389614503 shutdown_connections 2024-04-21T08:48:47.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.577+0000 7fada9e26640 1 -- 172.21.15.28:0/389614503 wait complete. 2024-04-21T08:48:47.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.577+0000 7fada9e26640 1 Processor -- start 2024-04-21T08:48:47.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.577+0000 7fada9e26640 1 -- start start 2024-04-21T08:48:47.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada9e26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada4100e90 0x7fada410b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:47.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada9e26640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fada4101fe0 0x7fada4109e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:47.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada9e26640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fada41029a0 0x7fada410a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:47.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada9e26640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fada410dbf0 con 0x7fada4100e90 2024-04-21T08:48:47.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada9e26640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fada410da70 con 0x7fada41029a0 2024-04-21T08:48:47.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada9e26640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fada410dd70 con 0x7fada4101fe0 2024-04-21T08:48:47.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada4100e90 0x7fada410b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:47.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada4100e90 0x7fada410b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38644/0 (socket says 172.21.15.28:38644) 2024-04-21T08:48:47.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada37fe640 1 -- 172.21.15.28:0/2825159531 learned_addr learned my addr 172.21.15.28:0/2825159531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:47.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada3fff640 1 --2- 172.21.15.28:0/2825159531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fada41029a0 0x7fada410a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:47.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada2ffd640 1 --2- 172.21.15.28:0/2825159531 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fada4101fe0 0x7fada4109e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:47.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada37fe640 1 -- 172.21.15.28:0/2825159531 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fada4101fe0 msgr2=0x7fada4109e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:47.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada37fe640 1 --2- 172.21.15.28:0/2825159531 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fada4101fe0 0x7fada4109e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:47.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada37fe640 1 -- 172.21.15.28:0/2825159531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fada41029a0 msgr2=0x7fada410a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:47.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada37fe640 1 --2- 172.21.15.28:0/2825159531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fada41029a0 0x7fada410a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:47.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada37fe640 1 -- 172.21.15.28:0/2825159531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fada410ac40 con 0x7fada4100e90 2024-04-21T08:48:47.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada2ffd640 1 --2- 172.21.15.28:0/2825159531 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fada4101fe0 0x7fada4109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:47.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada37fe640 1 --2- 172.21.15.28:0/2825159531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada4100e90 0x7fada410b830 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fad9400b4a0 tx=0x7fad9400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:47.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada0ff9640 1 -- 172.21.15.28:0/2825159531 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad94014070 con 0x7fada4100e90 2024-04-21T08:48:47.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada0ff9640 1 -- 172.21.15.28:0/2825159531 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad94010ac0 con 0x7fada4100e90 2024-04-21T08:48:47.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada9e26640 1 -- 172.21.15.28:0/2825159531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fada41a2390 con 0x7fada4100e90 2024-04-21T08:48:47.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada0ff9640 1 -- 172.21.15.28:0/2825159531 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad94017c30 con 0x7fada4100e90 2024-04-21T08:48:47.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.578+0000 7fada9e26640 1 -- 172.21.15.28:0/2825159531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fada41a2960 con 0x7fada4100e90 2024-04-21T08:48:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.580+0000 7fada9e26640 1 -- 172.21.15.28:0/2825159531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fada41081a0 con 0x7fada4100e90 2024-04-21T08:48:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.580+0000 7fada0ff9640 1 -- 172.21.15.28:0/2825159531 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fad94017070 con 0x7fada4100e90 2024-04-21T08:48:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.581+0000 7fada0ff9640 1 --2- 172.21.15.28:0/2825159531 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fad8c0a3bb0 0x7fad8c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.581+0000 7fada2ffd640 1 --2- 172.21.15.28:0/2825159531 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fad8c0a3bb0 0x7fad8c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.581+0000 7fada0ff9640 1 -- 172.21.15.28:0/2825159531 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7fad940cfbd0 con 0x7fada4100e90 2024-04-21T08:48:47.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.582+0000 7fada2ffd640 1 --2- 172.21.15.28:0/2825159531 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fad8c0a3bb0 0x7fad8c0a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fad98006fd0 tx=0x7fad98008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:47.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.586+0000 7fada0ff9640 1 -- 172.21.15.28:0/2825159531 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad9409c520 con 0x7fada4100e90 2024-04-21T08:48:47.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:47 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/469491879' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T08:48:47.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:47 smithi028 ceph-mon[28003]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T08:48:47.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:47 smithi028 ceph-mon[28003]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T08:48:47.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:47 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3115372268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T08:48:47.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:47 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3115372268' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T08:48:47.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:47 smithi028 ceph-mon[28003]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T08:48:47.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:47.773+0000 7fada9e26640 1 -- 172.21.15.28:0/2825159531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fada402d5c0 con 0x7fada4100e90 2024-04-21T08:48:48.021 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:47 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/469491879' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T08:48:48.021 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:47 smithi171 ceph-mon[31358]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T08:48:48.022 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:47 smithi171 ceph-mon[31358]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T08:48:48.022 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:47 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3115372268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T08:48:48.022 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:47 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3115372268' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T08:48:48.022 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:47 smithi171 ceph-mon[31358]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T08:48:48.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:47 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/469491879' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T08:48:48.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:47 smithi152 ceph-mon[31401]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T08:48:48.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:47 smithi152 ceph-mon[31401]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T08:48:48.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:47 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3115372268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T08:48:48.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:47 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3115372268' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T08:48:48.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:47 smithi152 ceph-mon[31401]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T08:48:49.006 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:48 smithi152 ceph-mon[31401]: pgmap v291: 97 pgs: 3 creating+activating, 21 creating+peering, 59 unknown, 14 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-04-21T08:48:49.006 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:48 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] up:active 2024-04-21T08:48:49.006 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:48 smithi152 ceph-mon[31401]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:48:49.006 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:48 smithi152 ceph-mon[31401]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:48:49.006 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:48 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2825159531' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T08:48:49.006 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:48 smithi152 ceph-mon[31401]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:48:49.006 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:48 smithi152 ceph-mon[31401]: Cluster is now healthy 2024-04-21T08:48:49.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:48 smithi028 ceph-mon[28003]: pgmap v291: 97 pgs: 3 creating+activating, 21 creating+peering, 59 unknown, 14 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-04-21T08:48:49.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:48 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] up:active 2024-04-21T08:48:49.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:48 smithi028 ceph-mon[28003]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:48:49.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:48 smithi028 ceph-mon[28003]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:48:49.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:48 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2825159531' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T08:48:49.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:48 smithi028 ceph-mon[28003]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:48:49.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:48 smithi028 ceph-mon[28003]: Cluster is now healthy 2024-04-21T08:48:49.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:48 smithi171 ceph-mon[31358]: pgmap v291: 97 pgs: 3 creating+activating, 21 creating+peering, 59 unknown, 14 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-04-21T08:48:49.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:48 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] up:active 2024-04-21T08:48:49.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:48 smithi171 ceph-mon[31358]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:48:49.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:48 smithi171 ceph-mon[31358]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:48:49.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:48 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2825159531' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T08:48:49.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:48 smithi171 ceph-mon[31358]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:48:49.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:48 smithi171 ceph-mon[31358]: Cluster is now healthy 2024-04-21T08:48:49.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.740+0000 7fada0ff9640 1 -- 172.21.15.28:0/2825159531 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7fad9409be20 con 0x7fada4100e90 2024-04-21T08:48:49.740 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' created 2024-04-21T08:48:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.744+0000 7fada9e26640 1 -- 172.21.15.28:0/2825159531 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fad8c0a3bb0 msgr2=0x7fad8c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.744+0000 7fada9e26640 1 --2- 172.21.15.28:0/2825159531 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fad8c0a3bb0 0x7fad8c0a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fad98006fd0 tx=0x7fad98008040 comp rx=0 tx=0).stop 2024-04-21T08:48:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.744+0000 7fada9e26640 1 -- 172.21.15.28:0/2825159531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada4100e90 msgr2=0x7fada410b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.744+0000 7fada9e26640 1 --2- 172.21.15.28:0/2825159531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada4100e90 0x7fada410b830 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fad9400b4a0 tx=0x7fad9400b960 comp rx=0 tx=0).stop 2024-04-21T08:48:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.744+0000 7fada9e26640 1 -- 172.21.15.28:0/2825159531 shutdown_connections 2024-04-21T08:48:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.744+0000 7fada9e26640 1 --2- 172.21.15.28:0/2825159531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fada41029a0 0x7fada410a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.744+0000 7fada9e26640 1 --2- 172.21.15.28:0/2825159531 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fada4101fe0 0x7fada4109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.744+0000 7fada9e26640 1 --2- 172.21.15.28:0/2825159531 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fad8c0a3bb0 0x7fad8c0a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.744+0000 7fada9e26640 1 --2- 172.21.15.28:0/2825159531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada4100e90 0x7fada410b830 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.744+0000 7fada9e26640 1 -- 172.21.15.28:0/2825159531 >> 172.21.15.28:0/2825159531 conn(0x7fada40fc400 msgr2=0x7fada4105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.744+0000 7fada9e26640 1 -- 172.21.15.28:0/2825159531 shutdown_connections 2024-04-21T08:48:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.744+0000 7fada9e26640 1 -- 172.21.15.28:0/2825159531 wait complete. 2024-04-21T08:48:49.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-21T08:48:49.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.877+0000 7f7d9df98640 1 -- 172.21.15.28:0/1339301641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d98102a90 msgr2=0x7f7d98104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.877+0000 7f7d9df98640 1 --2- 172.21.15.28:0/1339301641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d98102a90 0x7f7d98104e80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7d88009900 tx=0x7f7d8802bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.877+0000 7f7d9df98640 1 -- 172.21.15.28:0/1339301641 shutdown_connections 2024-04-21T08:48:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.877+0000 7f7d9df98640 1 --2- 172.21.15.28:0/1339301641 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7d98107cf0 0x7f7d9810a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.877+0000 7f7d9df98640 1 --2- 172.21.15.28:0/1339301641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d981053c0 0x7f7d981077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.877+0000 7f7d9df98640 1 --2- 172.21.15.28:0/1339301641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d98102a90 0x7f7d98104e80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.877+0000 7f7d9df98640 1 -- 172.21.15.28:0/1339301641 >> 172.21.15.28:0/1339301641 conn(0x7f7d980fc440 msgr2=0x7f7d980fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.877+0000 7f7d9df98640 1 -- 172.21.15.28:0/1339301641 shutdown_connections 2024-04-21T08:48:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.877+0000 7f7d9df98640 1 -- 172.21.15.28:0/1339301641 wait complete. 2024-04-21T08:48:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.877+0000 7f7d9df98640 1 Processor -- start 2024-04-21T08:48:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d9df98640 1 -- start start 2024-04-21T08:48:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d9df98640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7d98102a90 0x7f7d98198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d9df98640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d981053c0 0x7f7d981987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d9df98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d98107cf0 0x7f7d9819f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d9df98640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9810daf0 con 0x7f7d98107cf0 2024-04-21T08:48:49.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d9df98640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9810d970 con 0x7f7d981053c0 2024-04-21T08:48:49.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d9df98640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9810dc70 con 0x7f7d98102a90 2024-04-21T08:48:49.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d97fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d98107cf0 0x7f7d9819f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d97fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d98107cf0 0x7f7d9819f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38658/0 (socket says 172.21.15.28:38658) 2024-04-21T08:48:49.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d97fff640 1 -- 172.21.15.28:0/3500531256 learned_addr learned my addr 172.21.15.28:0/3500531256 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:49.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d977fe640 1 --2- 172.21.15.28:0/3500531256 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7d98102a90 0x7f7d98198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d96ffd640 1 --2- 172.21.15.28:0/3500531256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d981053c0 0x7f7d981987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d97fff640 1 -- 172.21.15.28:0/3500531256 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7d98102a90 msgr2=0x7f7d98198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:49.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d97fff640 1 --2- 172.21.15.28:0/3500531256 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7d98102a90 0x7f7d98198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d97fff640 1 -- 172.21.15.28:0/3500531256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d981053c0 msgr2=0x7f7d981987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:49.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d97fff640 1 --2- 172.21.15.28:0/3500531256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d981053c0 0x7f7d981987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:49.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d97fff640 1 -- 172.21.15.28:0/3500531256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d9819fe60 con 0x7f7d98107cf0 2024-04-21T08:48:49.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d96ffd640 1 --2- 172.21.15.28:0/3500531256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d981053c0 0x7f7d981987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:49.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.878+0000 7f7d97fff640 1 --2- 172.21.15.28:0/3500531256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d98107cf0 0x7f7d9819f7f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000eaa0 tx=0x7f7d9000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:49.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.879+0000 7f7d94ff9640 1 -- 172.21.15.28:0/3500531256 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9000cdb0 con 0x7f7d98107cf0 2024-04-21T08:48:49.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.879+0000 7f7d9df98640 1 -- 172.21.15.28:0/3500531256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d981a0150 con 0x7f7d98107cf0 2024-04-21T08:48:49.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.879+0000 7f7d94ff9640 1 -- 172.21.15.28:0/3500531256 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d90005020 con 0x7f7d98107cf0 2024-04-21T08:48:49.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.879+0000 7f7d94ff9640 1 -- 172.21.15.28:0/3500531256 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d900116d0 con 0x7f7d98107cf0 2024-04-21T08:48:49.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.879+0000 7f7d9df98640 1 -- 172.21.15.28:0/3500531256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d981a06a0 con 0x7f7d98107cf0 2024-04-21T08:48:49.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.880+0000 7f7d9df98640 1 -- 172.21.15.28:0/3500531256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d980fdf30 con 0x7f7d98107cf0 2024-04-21T08:48:49.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.881+0000 7f7d94ff9640 1 -- 172.21.15.28:0/3500531256 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7d900054a0 con 0x7f7d98107cf0 2024-04-21T08:48:49.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.881+0000 7f7d94ff9640 1 --2- 172.21.15.28:0/3500531256 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7d800a3bb0 0x7f7d800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:49.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.882+0000 7f7d977fe640 1 --2- 172.21.15.28:0/3500531256 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7d800a3bb0 0x7f7d800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:49.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.882+0000 7f7d94ff9640 1 -- 172.21.15.28:0/3500531256 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7f7d900d0c50 con 0x7f7d98107cf0 2024-04-21T08:48:49.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.882+0000 7f7d977fe640 1 --2- 172.21.15.28:0/3500531256 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7d800a3bb0 0x7f7d800a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7d88000c00 tx=0x7f7d88005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:49.886+0000 7f7d94ff9640 1 -- 172.21.15.28:0/3500531256 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7d90016030 con 0x7f7d98107cf0 2024-04-21T08:48:50.019 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:49 smithi152 ceph-mon[31401]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T08:48:50.019 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:49 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2825159531' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T08:48:50.019 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:49 smithi152 ceph-mon[31401]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T08:48:50.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.073+0000 7f7d9df98640 1 -- 172.21.15.28:0/3500531256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f7d9802d5c0 con 0x7f7d98107cf0 2024-04-21T08:48:50.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:49 smithi028 ceph-mon[28003]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T08:48:50.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:49 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2825159531' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T08:48:50.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:49 smithi028 ceph-mon[28003]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T08:48:50.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:49 smithi171 ceph-mon[31358]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T08:48:50.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:49 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2825159531' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T08:48:50.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:49 smithi171 ceph-mon[31358]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T08:48:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.742+0000 7f7d94ff9640 1 -- 172.21.15.28:0/3500531256 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7f7d9009d3b0 con 0x7f7d98107cf0 2024-04-21T08:48:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 4 allow_ec_overwrites to true 2024-04-21T08:48:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.746+0000 7f7d9df98640 1 -- 172.21.15.28:0/3500531256 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7d800a3bb0 msgr2=0x7f7d800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.746+0000 7f7d9df98640 1 --2- 172.21.15.28:0/3500531256 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7d800a3bb0 0x7f7d800a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7d88000c00 tx=0x7f7d88005be0 comp rx=0 tx=0).stop 2024-04-21T08:48:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.746+0000 7f7d9df98640 1 -- 172.21.15.28:0/3500531256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d98107cf0 msgr2=0x7f7d9819f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.746+0000 7f7d9df98640 1 --2- 172.21.15.28:0/3500531256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d98107cf0 0x7f7d9819f7f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000eaa0 tx=0x7f7d9000ef60 comp rx=0 tx=0).stop 2024-04-21T08:48:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.746+0000 7f7d9df98640 1 -- 172.21.15.28:0/3500531256 shutdown_connections 2024-04-21T08:48:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.746+0000 7f7d9df98640 1 --2- 172.21.15.28:0/3500531256 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7d800a3bb0 0x7f7d800a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.746+0000 7f7d9df98640 1 --2- 172.21.15.28:0/3500531256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d98107cf0 0x7f7d9819f7f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.746+0000 7f7d9df98640 1 --2- 172.21.15.28:0/3500531256 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d981053c0 0x7f7d981987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.746+0000 7f7d9df98640 1 --2- 172.21.15.28:0/3500531256 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7d98102a90 0x7f7d98198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:50.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.746+0000 7f7d9df98640 1 -- 172.21.15.28:0/3500531256 >> 172.21.15.28:0/3500531256 conn(0x7f7d980fc440 msgr2=0x7f7d981035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:50.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.746+0000 7f7d9df98640 1 -- 172.21.15.28:0/3500531256 shutdown_connections 2024-04-21T08:48:50.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.746+0000 7f7d9df98640 1 -- 172.21.15.28:0/3500531256 wait complete. 2024-04-21T08:48:50.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-21T08:48:50.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.881+0000 7f149c5bb640 1 -- 172.21.15.28:0/2591737428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14941029a0 msgr2=0x7f14941092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:50.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.881+0000 7f149c5bb640 1 --2- 172.21.15.28:0/2591737428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14941029a0 0x7f14941092c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f148c009ca0 tx=0x7f148c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.882+0000 7f149c5bb640 1 -- 172.21.15.28:0/2591737428 shutdown_connections 2024-04-21T08:48:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.882+0000 7f149c5bb640 1 --2- 172.21.15.28:0/2591737428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14941029a0 0x7f14941092c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.882+0000 7f149c5bb640 1 --2- 172.21.15.28:0/2591737428 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1494101fe0 0x7f1494102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.882+0000 7f149c5bb640 1 --2- 172.21.15.28:0/2591737428 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1494100e90 0x7f14941012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.882+0000 7f149c5bb640 1 -- 172.21.15.28:0/2591737428 >> 172.21.15.28:0/2591737428 conn(0x7f14940fc400 msgr2=0x7f14940fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.882+0000 7f149c5bb640 1 -- 172.21.15.28:0/2591737428 shutdown_connections 2024-04-21T08:48:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.882+0000 7f149c5bb640 1 -- 172.21.15.28:0/2591737428 wait complete. 2024-04-21T08:48:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.882+0000 7f149c5bb640 1 Processor -- start 2024-04-21T08:48:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.882+0000 7f149c5bb640 1 -- start start 2024-04-21T08:48:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.882+0000 7f149c5bb640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1494100e90 0x7f14941970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.882+0000 7f149c5bb640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1494101fe0 0x7f1494197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.882+0000 7f149c5bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14941029a0 0x7f149419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.882+0000 7f149c5bb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f149410c9e0 con 0x7f14941029a0 2024-04-21T08:48:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.882+0000 7f149c5bb640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f149410c860 con 0x7f1494101fe0 2024-04-21T08:48:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.882+0000 7f149c5bb640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f149410cb60 con 0x7f1494100e90 2024-04-21T08:48:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f149ab58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14941029a0 0x7f149419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f149ab58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14941029a0 0x7f149419e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38676/0 (socket says 172.21.15.28:38676) 2024-04-21T08:48:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f149ab58640 1 -- 172.21.15.28:0/1755903089 learned_addr learned my addr 172.21.15.28:0/1755903089 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f1499b56640 1 --2- 172.21.15.28:0/1755903089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1494101fe0 0x7f1494197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f149a357640 1 --2- 172.21.15.28:0/1755903089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1494100e90 0x7f14941970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f149ab58640 1 -- 172.21.15.28:0/1755903089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1494100e90 msgr2=0x7f14941970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f149ab58640 1 --2- 172.21.15.28:0/1755903089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1494100e90 0x7f14941970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f149ab58640 1 -- 172.21.15.28:0/1755903089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1494101fe0 msgr2=0x7f1494197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f149ab58640 1 --2- 172.21.15.28:0/1755903089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1494101fe0 0x7f1494197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f149ab58640 1 -- 172.21.15.28:0/1755903089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149419ed00 con 0x7f14941029a0 2024-04-21T08:48:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f149ab58640 1 --2- 172.21.15.28:0/1755903089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14941029a0 0x7f149419e690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f148c009dc0 tx=0x7f148c032fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f148b7fe640 1 -- 172.21.15.28:0/1755903089 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148c034aa0 con 0x7f14941029a0 2024-04-21T08:48:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f148b7fe640 1 -- 172.21.15.28:0/1755903089 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f148c034c40 con 0x7f14941029a0 2024-04-21T08:48:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f149c5bb640 1 -- 172.21.15.28:0/1755903089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14940690e0 con 0x7f14941029a0 2024-04-21T08:48:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f148b7fe640 1 -- 172.21.15.28:0/1755903089 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148c0042a0 con 0x7f14941029a0 2024-04-21T08:48:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.883+0000 7f149c5bb640 1 -- 172.21.15.28:0/1755903089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1494069620 con 0x7f14941029a0 2024-04-21T08:48:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.884+0000 7f149c5bb640 1 -- 172.21.15.28:0/1755903089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1494068d70 con 0x7f14941029a0 2024-04-21T08:48:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.885+0000 7f148b7fe640 1 -- 172.21.15.28:0/1755903089 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f148c0325c0 con 0x7f14941029a0 2024-04-21T08:48:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.886+0000 7f148b7fe640 1 --2- 172.21.15.28:0/1755903089 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f14800a3c30 0x7f14800a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.886+0000 7f149a357640 1 --2- 172.21.15.28:0/1755903089 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f14800a3c30 0x7f14800a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.886+0000 7f148b7fe640 1 -- 172.21.15.28:0/1755903089 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f148c0f44e0 con 0x7f14941029a0 2024-04-21T08:48:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.887+0000 7f149a357640 1 --2- 172.21.15.28:0/1755903089 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f14800a3c30 0x7f14800a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1490009bd0 tx=0x7f1490005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:50.890+0000 7f148b7fe640 1 -- 172.21.15.28:0/1755903089 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f148c0c0c40 con 0x7f14941029a0 2024-04-21T08:48:51.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:50 smithi152 ceph-mon[31401]: pgmap v293: 97 pgs: 3 creating+activating, 21 creating+peering, 27 unknown, 46 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 6 op/s 2024-04-21T08:48:51.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:50 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3500531256' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T08:48:51.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:50 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3500531256' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T08:48:51.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:50 smithi152 ceph-mon[31401]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T08:48:51.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:50 smithi028 ceph-mon[28003]: pgmap v293: 97 pgs: 3 creating+activating, 21 creating+peering, 27 unknown, 46 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 6 op/s 2024-04-21T08:48:51.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:50 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3500531256' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T08:48:51.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:50 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3500531256' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T08:48:51.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:50 smithi028 ceph-mon[28003]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T08:48:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.119+0000 7f149c5bb640 1 -- 172.21.15.28:0/1755903089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f1494107e40 con 0x7f14941029a0 2024-04-21T08:48:51.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:50 smithi171 ceph-mon[31358]: pgmap v293: 97 pgs: 3 creating+activating, 21 creating+peering, 27 unknown, 46 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 6 op/s 2024-04-21T08:48:51.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:50 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3500531256' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T08:48:51.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:50 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3500531256' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T08:48:51.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:50 smithi171 ceph-mon[31358]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T08:48:51.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.768+0000 7f148b7fe640 1 -- 172.21.15.28:0/1755903089 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f148c0c0540 con 0x7f14941029a0 2024-04-21T08:48:51.768 INFO:teuthology.orchestra.run.smithi028.stderr:added data pool 4 to fsmap 2024-04-21T08:48:51.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.771+0000 7f149c5bb640 1 -- 172.21.15.28:0/1755903089 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f14800a3c30 msgr2=0x7f14800a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:51.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.771+0000 7f149c5bb640 1 --2- 172.21.15.28:0/1755903089 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f14800a3c30 0x7f14800a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1490009bd0 tx=0x7f1490005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:51.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.771+0000 7f149c5bb640 1 -- 172.21.15.28:0/1755903089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14941029a0 msgr2=0x7f149419e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:51.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.771+0000 7f149c5bb640 1 --2- 172.21.15.28:0/1755903089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14941029a0 0x7f149419e690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f148c009dc0 tx=0x7f148c032fb0 comp rx=0 tx=0).stop 2024-04-21T08:48:51.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.771+0000 7f149c5bb640 1 -- 172.21.15.28:0/1755903089 shutdown_connections 2024-04-21T08:48:51.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.771+0000 7f149c5bb640 1 --2- 172.21.15.28:0/1755903089 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f14800a3c30 0x7f14800a60f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:51.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.771+0000 7f149c5bb640 1 --2- 172.21.15.28:0/1755903089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14941029a0 0x7f149419e690 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:51.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.771+0000 7f149c5bb640 1 --2- 172.21.15.28:0/1755903089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1494101fe0 0x7f1494197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:51.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.771+0000 7f149c5bb640 1 --2- 172.21.15.28:0/1755903089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1494100e90 0x7f14941970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:51.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.771+0000 7f149c5bb640 1 -- 172.21.15.28:0/1755903089 >> 172.21.15.28:0/1755903089 conn(0x7f14940fc400 msgr2=0x7f14941053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:51.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.771+0000 7f149c5bb640 1 -- 172.21.15.28:0/1755903089 shutdown_connections 2024-04-21T08:48:51.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.771+0000 7f149c5bb640 1 -- 172.21.15.28:0/1755903089 wait complete. 2024-04-21T08:48:51.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:51.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.893+0000 7fe399746640 1 -- 172.21.15.28:0/488600652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe394101fa0 msgr2=0x7fe3941023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:51.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.893+0000 7fe399746640 1 --2- 172.21.15.28:0/488600652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe394101fa0 0x7fe3941023c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe384009900 tx=0x7fe38402bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:51.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.893+0000 7fe399746640 1 -- 172.21.15.28:0/488600652 shutdown_connections 2024-04-21T08:48:51.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.893+0000 7fe399746640 1 --2- 172.21.15.28:0/488600652 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe394103b50 0x7fe39410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:51.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.893+0000 7fe399746640 1 --2- 172.21.15.28:0/488600652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe394103190 0x7fe394103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:51.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.893+0000 7fe399746640 1 --2- 172.21.15.28:0/488600652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe394101fa0 0x7fe3941023c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:51.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.893+0000 7fe399746640 1 -- 172.21.15.28:0/488600652 >> 172.21.15.28:0/488600652 conn(0x7fe3940fd530 msgr2=0x7fe3940ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:51.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.893+0000 7fe399746640 1 -- 172.21.15.28:0/488600652 shutdown_connections 2024-04-21T08:48:51.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.893+0000 7fe399746640 1 -- 172.21.15.28:0/488600652 wait complete. 2024-04-21T08:48:51.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.893+0000 7fe399746640 1 Processor -- start 2024-04-21T08:48:51.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe399746640 1 -- start start 2024-04-21T08:48:51.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe399746640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe394101fa0 0x7fe394075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:51.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe399746640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe394103190 0x7fe394074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:51.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe399746640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe394103b50 0x7fe394074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:51.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe399746640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe39410dae0 con 0x7fe394101fa0 2024-04-21T08:48:51.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe399746640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe39410d960 con 0x7fe394103b50 2024-04-21T08:48:51.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe399746640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe39410dc60 con 0x7fe394103190 2024-04-21T08:48:51.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe392ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe394101fa0 0x7fe394075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:51.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe392ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe394101fa0 0x7fe394075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38700/0 (socket says 172.21.15.28:38700) 2024-04-21T08:48:51.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe392ffd640 1 -- 172.21.15.28:0/1219669058 learned_addr learned my addr 172.21.15.28:0/1219669058 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:51.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe3937fe640 1 --2- 172.21.15.28:0/1219669058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe394103b50 0x7fe394074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:51.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe3927fc640 1 --2- 172.21.15.28:0/1219669058 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe394103190 0x7fe394074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:51.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe392ffd640 1 -- 172.21.15.28:0/1219669058 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe394103190 msgr2=0x7fe394074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:51.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe392ffd640 1 --2- 172.21.15.28:0/1219669058 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe394103190 0x7fe394074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe392ffd640 1 -- 172.21.15.28:0/1219669058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe394103b50 msgr2=0x7fe394074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe392ffd640 1 --2- 172.21.15.28:0/1219669058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe394103b50 0x7fe394074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe392ffd640 1 -- 172.21.15.28:0/1219669058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe394075110 con 0x7fe394101fa0 2024-04-21T08:48:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe3927fc640 1 --2- 172.21.15.28:0/1219669058 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe394103190 0x7fe394074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe392ffd640 1 --2- 172.21.15.28:0/1219669058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe394101fa0 0x7fe394075d00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe384000c00 tx=0x7fe384004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe37bfff640 1 -- 172.21.15.28:0/1219669058 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe384043020 con 0x7fe394101fa0 2024-04-21T08:48:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe37bfff640 1 -- 172.21.15.28:0/1219669058 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe384004aa0 con 0x7fe394101fa0 2024-04-21T08:48:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe399746640 1 -- 172.21.15.28:0/1219669058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3941a21f0 con 0x7fe394101fa0 2024-04-21T08:48:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.894+0000 7fe37bfff640 1 -- 172.21.15.28:0/1219669058 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe384003d00 con 0x7fe394101fa0 2024-04-21T08:48:51.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.895+0000 7fe399746640 1 -- 172.21.15.28:0/1219669058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3941a2740 con 0x7fe394101fa0 2024-04-21T08:48:51.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.896+0000 7fe399746640 1 -- 172.21.15.28:0/1219669058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3941092c0 con 0x7fe394101fa0 2024-04-21T08:48:51.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.897+0000 7fe37bfff640 1 -- 172.21.15.28:0/1219669058 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe384046020 con 0x7fe394101fa0 2024-04-21T08:48:51.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.897+0000 7fe37bfff640 1 --2- 172.21.15.28:0/1219669058 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe37c0a3bb0 0x7fe37c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:51.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.897+0000 7fe3927fc640 1 --2- 172.21.15.28:0/1219669058 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe37c0a3bb0 0x7fe37c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:51.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.897+0000 7fe37bfff640 1 -- 172.21.15.28:0/1219669058 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe3840f2be0 con 0x7fe394101fa0 2024-04-21T08:48:51.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.898+0000 7fe3927fc640 1 --2- 172.21.15.28:0/1219669058 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe37c0a3bb0 0x7fe37c0a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe394075a70 tx=0x7fe38800a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:51.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:51.902+0000 7fe37bfff640 1 -- 172.21.15.28:0/1219669058 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe38404d020 con 0x7fe394101fa0 2024-04-21T08:48:52.104 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 08:48:51 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-b[62223]: 2024-04-21T08:48:51.908+0000 7f5feb533700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-21T08:48:52.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:51 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1755903089' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T08:48:52.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:51 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1755903089' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T08:48:52.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:51 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1755903089' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T08:48:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.127+0000 7fe399746640 1 -- 172.21.15.28:0/1219669058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe39402d5c0 con 0x7fe394101fa0 2024-04-21T08:48:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.128+0000 7fe37bfff640 1 -- 172.21.15.28:0/1219669058 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11809 (secure 0 0 0) 0x7fe3840bf380 con 0x7fe394101fa0 2024-04-21T08:48:52.127 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:52.128 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:48:51.121352+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24488,"qdb_cluster":[24488]},"id":1}]} 2024-04-21T08:48:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-04-21T08:48:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.130+0000 7fe399746640 1 -- 172.21.15.28:0/1219669058 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe37c0a3bb0 msgr2=0x7fe37c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.130+0000 7fe399746640 1 --2- 172.21.15.28:0/1219669058 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe37c0a3bb0 0x7fe37c0a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe394075a70 tx=0x7fe38800a5c0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.130+0000 7fe399746640 1 -- 172.21.15.28:0/1219669058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe394101fa0 msgr2=0x7fe394075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.130+0000 7fe399746640 1 --2- 172.21.15.28:0/1219669058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe394101fa0 0x7fe394075d00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe384000c00 tx=0x7fe384004270 comp rx=0 tx=0).stop 2024-04-21T08:48:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.130+0000 7fe399746640 1 -- 172.21.15.28:0/1219669058 shutdown_connections 2024-04-21T08:48:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.130+0000 7fe399746640 1 --2- 172.21.15.28:0/1219669058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe394103b50 0x7fe394074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.130+0000 7fe399746640 1 --2- 172.21.15.28:0/1219669058 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe394103190 0x7fe394074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.130+0000 7fe399746640 1 --2- 172.21.15.28:0/1219669058 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe37c0a3bb0 0x7fe37c0a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.130+0000 7fe399746640 1 --2- 172.21.15.28:0/1219669058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe394101fa0 0x7fe394075d00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.130+0000 7fe399746640 1 -- 172.21.15.28:0/1219669058 >> 172.21.15.28:0/1219669058 conn(0x7fe3940fd530 msgr2=0x7fe394106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.130+0000 7fe399746640 1 -- 172.21.15.28:0/1219669058 shutdown_connections 2024-04-21T08:48:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.130+0000 7fe399746640 1 -- 172.21.15.28:0/1219669058 wait complete. 2024-04-21T08:48:52.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:48:52.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.243+0000 7f0dde3c7640 1 -- 172.21.15.28:0/2156369233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd8102a10 msgr2=0x7f0dd8109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.243+0000 7f0dde3c7640 1 --2- 172.21.15.28:0/2156369233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd8102a10 0x7f0dd8109330 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0009ca0 tx=0x7f0dd002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.243+0000 7f0dde3c7640 1 -- 172.21.15.28:0/2156369233 shutdown_connections 2024-04-21T08:48:52.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.243+0000 7f0dde3c7640 1 --2- 172.21.15.28:0/2156369233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd8102a10 0x7f0dd8109330 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.243+0000 7f0dde3c7640 1 --2- 172.21.15.28:0/2156369233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0dd8102050 0x7f0dd81024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.243+0000 7f0dde3c7640 1 --2- 172.21.15.28:0/2156369233 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0dd8100e70 0x7f0dd8101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.243+0000 7f0dde3c7640 1 -- 172.21.15.28:0/2156369233 >> 172.21.15.28:0/2156369233 conn(0x7f0dd80fc470 msgr2=0x7f0dd80fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:52.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.243+0000 7f0dde3c7640 1 -- 172.21.15.28:0/2156369233 shutdown_connections 2024-04-21T08:48:52.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.243+0000 7f0dde3c7640 1 -- 172.21.15.28:0/2156369233 wait complete. 2024-04-21T08:48:52.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.244+0000 7f0dde3c7640 1 Processor -- start 2024-04-21T08:48:52.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.244+0000 7f0dde3c7640 1 -- start start 2024-04-21T08:48:52.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.244+0000 7f0dde3c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd8100e70 0x7f0dd81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:52.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.244+0000 7f0dde3c7640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0dd8102050 0x7f0dd81985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:52.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.244+0000 7f0dde3c7640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0dd8102a10 0x7f0dd819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.244+0000 7f0dde3c7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd810ca00 con 0x7f0dd8100e70 2024-04-21T08:48:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.244+0000 7f0dde3c7640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd810c880 con 0x7f0dd8102a10 2024-04-21T08:48:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.244+0000 7f0dde3c7640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd810cb80 con 0x7f0dd8102050 2024-04-21T08:48:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.244+0000 7f0dd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd8100e70 0x7f0dd81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.244+0000 7f0dd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd8100e70 0x7f0dd81980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38708/0 (socket says 172.21.15.28:38708) 2024-04-21T08:48:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.244+0000 7f0dd7fff640 1 -- 172.21.15.28:0/1157042173 learned_addr learned my addr 172.21.15.28:0/1157042173 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.245+0000 7f0dd77fe640 1 --2- 172.21.15.28:0/1157042173 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0dd8102050 0x7f0dd81985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.245+0000 7f0ddc964640 1 --2- 172.21.15.28:0/1157042173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0dd8102a10 0x7f0dd819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.245+0000 7f0dd7fff640 1 -- 172.21.15.28:0/1157042173 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0dd8102050 msgr2=0x7f0dd81985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.245+0000 7f0dd7fff640 1 --2- 172.21.15.28:0/1157042173 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0dd8102050 0x7f0dd81985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.245+0000 7f0dd7fff640 1 -- 172.21.15.28:0/1157042173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0dd8102a10 msgr2=0x7f0dd819f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.245+0000 7f0dd7fff640 1 --2- 172.21.15.28:0/1157042173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0dd8102a10 0x7f0dd819f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.245+0000 7f0dd7fff640 1 -- 172.21.15.28:0/1157042173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd819fcc0 con 0x7f0dd8100e70 2024-04-21T08:48:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.245+0000 7f0ddc964640 1 --2- 172.21.15.28:0/1157042173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0dd8102a10 0x7f0dd819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.245+0000 7f0dd7fff640 1 --2- 172.21.15.28:0/1157042173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd8100e70 0x7f0dd81980a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0dc800c970 tx=0x7f0dc800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:52.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.245+0000 7f0dd57fa640 1 -- 172.21.15.28:0/1157042173 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8007c10 con 0x7f0dd8100e70 2024-04-21T08:48:52.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.245+0000 7f0dd57fa640 1 -- 172.21.15.28:0/1157042173 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dc8007db0 con 0x7f0dd8100e70 2024-04-21T08:48:52.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.245+0000 7f0dde3c7640 1 -- 172.21.15.28:0/1157042173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dd819ffb0 con 0x7f0dd8100e70 2024-04-21T08:48:52.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.245+0000 7f0dd57fa640 1 -- 172.21.15.28:0/1157042173 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc800f7d0 con 0x7f0dd8100e70 2024-04-21T08:48:52.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.245+0000 7f0dde3c7640 1 -- 172.21.15.28:0/1157042173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dd81a0410 con 0x7f0dd8100e70 2024-04-21T08:48:52.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.246+0000 7f0dde3c7640 1 -- 172.21.15.28:0/1157042173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dd81a06d0 con 0x7f0dd8100e70 2024-04-21T08:48:52.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.247+0000 7f0dd57fa640 1 -- 172.21.15.28:0/1157042173 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0dc8018480 con 0x7f0dd8100e70 2024-04-21T08:48:52.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.248+0000 7f0dd57fa640 1 --2- 172.21.15.28:0/1157042173 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0dc00a3bb0 0x7f0dc00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:52.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.248+0000 7f0dd77fe640 1 --2- 172.21.15.28:0/1157042173 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0dc00a3bb0 0x7f0dc00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:52.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.248+0000 7f0dd57fa640 1 -- 172.21.15.28:0/1157042173 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0dc80d0a10 con 0x7f0dd8100e70 2024-04-21T08:48:52.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.249+0000 7f0dd77fe640 1 --2- 172.21.15.28:0/1157042173 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0dc00a3bb0 0x7f0dc00a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0dd81995b0 tx=0x7f0dcc00b590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:52.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.253+0000 7f0dd57fa640 1 -- 172.21.15.28:0/1157042173 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0dc809d240 con 0x7f0dd8100e70 2024-04-21T08:48:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.439+0000 7f0dde3c7640 1 -- 172.21.15.28:0/1157042173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0dd81a08a0 con 0x7f0dd8100e70 2024-04-21T08:48:52.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.440+0000 7f0dd57fa640 1 -- 172.21.15.28:0/1157042173 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24101 (secure 0 0 0) 0x7f0dc809cb40 con 0x7f0dd8100e70 2024-04-21T08:48:52.440 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:52.440 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:48:51.121128+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:48:52.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.442+0000 7f0dde3c7640 1 -- 172.21.15.28:0/1157042173 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0dc00a3bb0 msgr2=0x7f0dc00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.442+0000 7f0dde3c7640 1 --2- 172.21.15.28:0/1157042173 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0dc00a3bb0 0x7f0dc00a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0dd81995b0 tx=0x7f0dcc00b590 comp rx=0 tx=0).stop 2024-04-21T08:48:52.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.442+0000 7f0dde3c7640 1 -- 172.21.15.28:0/1157042173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd8100e70 msgr2=0x7f0dd81980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.442+0000 7f0dde3c7640 1 --2- 172.21.15.28:0/1157042173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd8100e70 0x7f0dd81980a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0dc800c970 tx=0x7f0dc800ce30 comp rx=0 tx=0).stop 2024-04-21T08:48:52.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.442+0000 7f0dde3c7640 1 -- 172.21.15.28:0/1157042173 shutdown_connections 2024-04-21T08:48:52.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.442+0000 7f0dde3c7640 1 --2- 172.21.15.28:0/1157042173 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0dd8102a10 0x7f0dd819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.442+0000 7f0dde3c7640 1 --2- 172.21.15.28:0/1157042173 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0dd8102050 0x7f0dd81985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.442+0000 7f0dde3c7640 1 --2- 172.21.15.28:0/1157042173 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0dc00a3bb0 0x7f0dc00a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.442+0000 7f0dde3c7640 1 --2- 172.21.15.28:0/1157042173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd8100e70 0x7f0dd81980a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.442+0000 7f0dde3c7640 1 -- 172.21.15.28:0/1157042173 >> 172.21.15.28:0/1157042173 conn(0x7f0dd80fc470 msgr2=0x7f0dd806a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:52.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.442+0000 7f0dde3c7640 1 -- 172.21.15.28:0/1157042173 shutdown_connections 2024-04-21T08:48:52.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.442+0000 7f0dde3c7640 1 -- 172.21.15.28:0/1157042173 wait complete. 2024-04-21T08:48:52.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:48:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.556+0000 7f7618f85640 1 -- 172.21.15.28:0/4292476092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614101f90 msgr2=0x7f76141023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.556+0000 7f7618f85640 1 --2- 172.21.15.28:0/4292476092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614101f90 0x7f76141023b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7604009900 tx=0x7f760402bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.556+0000 7f7618f85640 1 -- 172.21.15.28:0/4292476092 shutdown_connections 2024-04-21T08:48:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.556+0000 7f7618f85640 1 --2- 172.21.15.28:0/4292476092 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7614103b40 0x7f761410a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.556+0000 7f7618f85640 1 --2- 172.21.15.28:0/4292476092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7614103180 0x7f7614103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.556+0000 7f7618f85640 1 --2- 172.21.15.28:0/4292476092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614101f90 0x7f76141023b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.556+0000 7f7618f85640 1 -- 172.21.15.28:0/4292476092 >> 172.21.15.28:0/4292476092 conn(0x7f76140fd500 msgr2=0x7f76140ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.556+0000 7f7618f85640 1 -- 172.21.15.28:0/4292476092 shutdown_connections 2024-04-21T08:48:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.556+0000 7f7618f85640 1 -- 172.21.15.28:0/4292476092 wait complete. 2024-04-21T08:48:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.557+0000 7f7618f85640 1 Processor -- start 2024-04-21T08:48:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.557+0000 7f7618f85640 1 -- start start 2024-04-21T08:48:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.557+0000 7f7618f85640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7614101f90 0x7f7614075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.557+0000 7f7618f85640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7614103180 0x7f7614074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.557+0000 7f7618f85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614103b40 0x7f7614074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.557+0000 7f7618f85640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f761410da90 con 0x7f7614103b40 2024-04-21T08:48:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.557+0000 7f7618f85640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f761410d910 con 0x7f7614101f90 2024-04-21T08:48:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.557+0000 7f7618f85640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f761410dc10 con 0x7f7614103180 2024-04-21T08:48:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.557+0000 7f7612d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614103b40 0x7f7614074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.558+0000 7f7612d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614103b40 0x7f7614074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38716/0 (socket says 172.21.15.28:38716) 2024-04-21T08:48:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.558+0000 7f7612d9d640 1 -- 172.21.15.28:0/474378785 learned_addr learned my addr 172.21.15.28:0/474378785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.558+0000 7f761259c640 1 --2- 172.21.15.28:0/474378785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7614101f90 0x7f7614075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.558+0000 7f760bfff640 1 --2- 172.21.15.28:0/474378785 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7614103180 0x7f7614074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.558+0000 7f7612d9d640 1 -- 172.21.15.28:0/474378785 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7614103180 msgr2=0x7f7614074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.558+0000 7f7612d9d640 1 --2- 172.21.15.28:0/474378785 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7614103180 0x7f7614074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.558+0000 7f7612d9d640 1 -- 172.21.15.28:0/474378785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7614101f90 msgr2=0x7f7614075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.558+0000 7f7612d9d640 1 --2- 172.21.15.28:0/474378785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7614101f90 0x7f7614075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.558+0000 7f7612d9d640 1 -- 172.21.15.28:0/474378785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7614075110 con 0x7f7614103b40 2024-04-21T08:48:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.558+0000 7f7612d9d640 1 --2- 172.21.15.28:0/474378785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614103b40 0x7f7614074880 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f760c00cad0 tx=0x7f760c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.558+0000 7f760b7fe640 1 -- 172.21.15.28:0/474378785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c00ae20 con 0x7f7614103b40 2024-04-21T08:48:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.558+0000 7f760b7fe640 1 -- 172.21.15.28:0/474378785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f760c005020 con 0x7f7614103b40 2024-04-21T08:48:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.558+0000 7f7618f85640 1 -- 172.21.15.28:0/474378785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76141a22f0 con 0x7f7614103b40 2024-04-21T08:48:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.558+0000 7f760b7fe640 1 -- 172.21.15.28:0/474378785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c00f6b0 con 0x7f7614103b40 2024-04-21T08:48:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.558+0000 7f7618f85640 1 -- 172.21.15.28:0/474378785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76141a27f0 con 0x7f7614103b40 2024-04-21T08:48:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.559+0000 7f7618f85640 1 -- 172.21.15.28:0/474378785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76141093d0 con 0x7f7614103b40 2024-04-21T08:48:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.560+0000 7f760b7fe640 1 -- 172.21.15.28:0/474378785 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f760c0051d0 con 0x7f7614103b40 2024-04-21T08:48:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.561+0000 7f760b7fe640 1 --2- 172.21.15.28:0/474378785 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f76000a3bb0 0x7f76000a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.561+0000 7f761259c640 1 --2- 172.21.15.28:0/474378785 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f76000a3bb0 0x7f76000a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.561+0000 7f760b7fe640 1 -- 172.21.15.28:0/474378785 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f760c0d0650 con 0x7f7614103b40 2024-04-21T08:48:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.561+0000 7f761259c640 1 --2- 172.21.15.28:0/474378785 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f76000a3bb0 0x7f76000a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7604005ab0 tx=0x7f7604005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:52.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.565+0000 7f760b7fe640 1 -- 172.21.15.28:0/474378785 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f760c014030 con 0x7f7614103b40 2024-04-21T08:48:52.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.753+0000 7f7618f85640 1 -- 172.21.15.28:0/474378785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7614062b50 con 0x7f7614103b40 2024-04-21T08:48:52.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.755+0000 7f760b7fe640 1 -- 172.21.15.28:0/474378785 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24101 (secure 0 0 0) 0x7f760c09c7a0 con 0x7f7614103b40 2024-04-21T08:48:52.755 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:52.755 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:48:51.121128+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.757+0000 7f7618f85640 1 -- 172.21.15.28:0/474378785 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f76000a3bb0 msgr2=0x7f76000a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.757+0000 7f7618f85640 1 --2- 172.21.15.28:0/474378785 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f76000a3bb0 0x7f76000a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7604005ab0 tx=0x7f7604005fd0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.757+0000 7f7618f85640 1 -- 172.21.15.28:0/474378785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614103b40 msgr2=0x7f7614074880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.757+0000 7f7618f85640 1 --2- 172.21.15.28:0/474378785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614103b40 0x7f7614074880 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f760c00cad0 tx=0x7f760c00cf90 comp rx=0 tx=0).stop 2024-04-21T08:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.757+0000 7f7618f85640 1 -- 172.21.15.28:0/474378785 shutdown_connections 2024-04-21T08:48:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.757+0000 7f7618f85640 1 --2- 172.21.15.28:0/474378785 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f76000a3bb0 0x7f76000a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.757+0000 7f7618f85640 1 --2- 172.21.15.28:0/474378785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614103b40 0x7f7614074880 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.757+0000 7f7618f85640 1 --2- 172.21.15.28:0/474378785 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7614103180 0x7f7614074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.757+0000 7f7618f85640 1 --2- 172.21.15.28:0/474378785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7614101f90 0x7f7614075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.757+0000 7f7618f85640 1 -- 172.21.15.28:0/474378785 >> 172.21.15.28:0/474378785 conn(0x7f76140fd500 msgr2=0x7f76141065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:52.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.757+0000 7f7618f85640 1 -- 172.21.15.28:0/474378785 shutdown_connections 2024-04-21T08:48:52.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.758+0000 7f7618f85640 1 -- 172.21.15.28:0/474378785 wait complete. 2024-04-21T08:48:52.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:52.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.888+0000 7ff5be35f640 1 -- 172.21.15.28:0/4136688911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5b8102a10 msgr2=0x7ff5b8109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.888+0000 7ff5be35f640 1 --2- 172.21.15.28:0/4136688911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5b8102a10 0x7ff5b8109330 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0009ca0 tx=0x7ff5b002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.888+0000 7ff5be35f640 1 -- 172.21.15.28:0/4136688911 shutdown_connections 2024-04-21T08:48:52.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.888+0000 7ff5be35f640 1 --2- 172.21.15.28:0/4136688911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5b8102a10 0x7ff5b8109330 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.888+0000 7ff5be35f640 1 --2- 172.21.15.28:0/4136688911 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff5b8102050 0x7ff5b81024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.888+0000 7ff5be35f640 1 --2- 172.21.15.28:0/4136688911 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff5b8100e70 0x7ff5b8101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.888+0000 7ff5be35f640 1 -- 172.21.15.28:0/4136688911 >> 172.21.15.28:0/4136688911 conn(0x7ff5b80fc470 msgr2=0x7ff5b80fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:52.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.888+0000 7ff5be35f640 1 -- 172.21.15.28:0/4136688911 shutdown_connections 2024-04-21T08:48:52.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.888+0000 7ff5be35f640 1 -- 172.21.15.28:0/4136688911 wait complete. 2024-04-21T08:48:52.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.888+0000 7ff5be35f640 1 Processor -- start 2024-04-21T08:48:52.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.888+0000 7ff5be35f640 1 -- start start 2024-04-21T08:48:52.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5be35f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff5b8100e70 0x7ff5b8197020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:52.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5be35f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff5b8102050 0x7ff5b8197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:52.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5be35f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5b8102a10 0x7ff5b819e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:52.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5be35f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b810ca10 con 0x7ff5b8102a10 2024-04-21T08:48:52.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5be35f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b810c890 con 0x7ff5b8102050 2024-04-21T08:48:52.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5be35f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b810cb90 con 0x7ff5b8100e70 2024-04-21T08:48:52.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5bc8fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5b8102a10 0x7ff5b819e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:52.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5bc8fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5b8102a10 0x7ff5b819e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38736/0 (socket says 172.21.15.28:38736) 2024-04-21T08:48:52.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5bc8fc640 1 -- 172.21.15.28:0/798721232 learned_addr learned my addr 172.21.15.28:0/798721232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:52.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5b77fe640 1 --2- 172.21.15.28:0/798721232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff5b8102050 0x7ff5b8197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:52.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5b7fff640 1 --2- 172.21.15.28:0/798721232 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff5b8100e70 0x7ff5b8197020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:52.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5bc8fc640 1 -- 172.21.15.28:0/798721232 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff5b8100e70 msgr2=0x7ff5b8197020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5bc8fc640 1 --2- 172.21.15.28:0/798721232 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff5b8100e70 0x7ff5b8197020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5bc8fc640 1 -- 172.21.15.28:0/798721232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff5b8102050 msgr2=0x7ff5b8197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:52.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5bc8fc640 1 --2- 172.21.15.28:0/798721232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff5b8102050 0x7ff5b8197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:52.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5bc8fc640 1 -- 172.21.15.28:0/798721232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b819ec40 con 0x7ff5b8102a10 2024-04-21T08:48:52.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5b7fff640 1 --2- 172.21.15.28:0/798721232 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff5b8100e70 0x7ff5b8197020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:52.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5bc8fc640 1 --2- 172.21.15.28:0/798721232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5b8102a10 0x7ff5b819e5d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0009dc0 tx=0x7ff5b0032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:52.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5b57fa640 1 -- 172.21.15.28:0/798721232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b0034aa0 con 0x7ff5b8102a10 2024-04-21T08:48:52.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5b57fa640 1 -- 172.21.15.28:0/798721232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5b0034c40 con 0x7ff5b8102a10 2024-04-21T08:48:52.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5b57fa640 1 -- 172.21.15.28:0/798721232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b0003e60 con 0x7ff5b8102a10 2024-04-21T08:48:52.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.889+0000 7ff5be35f640 1 -- 172.21.15.28:0/798721232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5b806a640 con 0x7ff5b8102a10 2024-04-21T08:48:52.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.890+0000 7ff5be35f640 1 -- 172.21.15.28:0/798721232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5b806ab30 con 0x7ff5b8102a10 2024-04-21T08:48:52.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.890+0000 7ff5be35f640 1 -- 172.21.15.28:0/798721232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5b8069230 con 0x7ff5b8102a10 2024-04-21T08:48:52.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.892+0000 7ff5b57fa640 1 -- 172.21.15.28:0/798721232 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff5b00325c0 con 0x7ff5b8102a10 2024-04-21T08:48:52.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.892+0000 7ff5b57fa640 1 --2- 172.21.15.28:0/798721232 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff5a40a3bb0 0x7ff5a40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:52.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.892+0000 7ff5b7fff640 1 --2- 172.21.15.28:0/798721232 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff5a40a3bb0 0x7ff5a40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:52.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.892+0000 7ff5b57fa640 1 -- 172.21.15.28:0/798721232 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff5b00f43e0 con 0x7ff5b8102a10 2024-04-21T08:48:52.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.893+0000 7ff5b7fff640 1 --2- 172.21.15.28:0/798721232 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff5a40a3bb0 0x7ff5a40a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00058c0 tx=0x7ff5a0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:52.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:52.897+0000 7ff5b57fa640 1 -- 172.21.15.28:0/798721232 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff5b00c0b80 con 0x7ff5b8102a10 2024-04-21T08:48:53.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:52 smithi152 ceph-mon[31401]: pgmap v296: 129 pgs: 9 creating+peering, 21 unknown, 99 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 6 op/s 2024-04-21T08:48:53.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:52 smithi152 ceph-mon[31401]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:48:53.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:52 smithi152 ceph-mon[31401]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T08:48:53.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:52 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1755903089' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T08:48:53.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:52 smithi152 ceph-mon[31401]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:48:53.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:52 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1219669058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:53.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:52 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1157042173' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:53.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:52 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/474378785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:53.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:52 smithi028 ceph-mon[28003]: pgmap v296: 129 pgs: 9 creating+peering, 21 unknown, 99 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 6 op/s 2024-04-21T08:48:53.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:52 smithi028 ceph-mon[28003]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:48:53.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:52 smithi028 ceph-mon[28003]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T08:48:53.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:52 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1755903089' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T08:48:53.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:52 smithi028 ceph-mon[28003]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:48:53.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:52 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1219669058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:53.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:52 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1157042173' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:53.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:52 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/474378785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:53.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:52 smithi171 ceph-mon[31358]: pgmap v296: 129 pgs: 9 creating+peering, 21 unknown, 99 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 6 op/s 2024-04-21T08:48:53.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:52 smithi171 ceph-mon[31358]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:48:53.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:52 smithi171 ceph-mon[31358]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T08:48:53.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:52 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1755903089' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T08:48:53.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:52 smithi171 ceph-mon[31358]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:48:53.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:52 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1219669058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:53.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:52 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1157042173' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:53.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:52 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/474378785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:53.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.124+0000 7ff5be35f640 1 -- 172.21.15.28:0/798721232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5b8108210 con 0x7ff5b8102a10 2024-04-21T08:48:53.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.125+0000 7ff5b57fa640 1 -- 172.21.15.28:0/798721232 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11809 (secure 0 0 0) 0x7ff5b00c0480 con 0x7ff5b8102a10 2024-04-21T08:48:53.125 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:53.125 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24419,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24452,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:48:51.121352+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24488,"qdb_cluster":[24488]},"id":1}]} 2024-04-21T08:48:53.125 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-04-21T08:48:53.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.127+0000 7ff5be35f640 1 -- 172.21.15.28:0/798721232 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff5a40a3bb0 msgr2=0x7ff5a40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.127+0000 7ff5be35f640 1 --2- 172.21.15.28:0/798721232 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff5a40a3bb0 0x7ff5a40a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00058c0 tx=0x7ff5a0005850 comp rx=0 tx=0).stop 2024-04-21T08:48:53.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.127+0000 7ff5be35f640 1 -- 172.21.15.28:0/798721232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5b8102a10 msgr2=0x7ff5b819e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.127+0000 7ff5be35f640 1 --2- 172.21.15.28:0/798721232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5b8102a10 0x7ff5b819e5d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0009dc0 tx=0x7ff5b0032f90 comp rx=0 tx=0).stop 2024-04-21T08:48:53.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.127+0000 7ff5be35f640 1 -- 172.21.15.28:0/798721232 shutdown_connections 2024-04-21T08:48:53.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.127+0000 7ff5be35f640 1 --2- 172.21.15.28:0/798721232 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff5a40a3bb0 0x7ff5a40a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.127+0000 7ff5be35f640 1 --2- 172.21.15.28:0/798721232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5b8102a10 0x7ff5b819e5d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.127+0000 7ff5be35f640 1 --2- 172.21.15.28:0/798721232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff5b8102050 0x7ff5b8197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.127+0000 7ff5be35f640 1 --2- 172.21.15.28:0/798721232 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff5b8100e70 0x7ff5b8197020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.127+0000 7ff5be35f640 1 -- 172.21.15.28:0/798721232 >> 172.21.15.28:0/798721232 conn(0x7ff5b80fc470 msgr2=0x7ff5b8105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:53.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.127+0000 7ff5be35f640 1 -- 172.21.15.28:0/798721232 shutdown_connections 2024-04-21T08:48:53.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.127+0000 7ff5be35f640 1 -- 172.21.15.28:0/798721232 wait complete. 2024-04-21T08:48:53.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:48:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.240+0000 7f76190c1640 1 -- 172.21.15.28:0/770411272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614102980 msgr2=0x7f76141092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.240+0000 7f76190c1640 1 --2- 172.21.15.28:0/770411272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614102980 0x7f76141092a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f760c009ca0 tx=0x7f760c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.240+0000 7f76190c1640 1 -- 172.21.15.28:0/770411272 shutdown_connections 2024-04-21T08:48:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.240+0000 7f76190c1640 1 --2- 172.21.15.28:0/770411272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614102980 0x7f76141092a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.240+0000 7f76190c1640 1 --2- 172.21.15.28:0/770411272 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7614101fc0 0x7f7614102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.240+0000 7f76190c1640 1 --2- 172.21.15.28:0/770411272 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7614100e70 0x7f7614101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.240+0000 7f76190c1640 1 -- 172.21.15.28:0/770411272 >> 172.21.15.28:0/770411272 conn(0x7f76140fc3e0 msgr2=0x7f76140fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.240+0000 7f76190c1640 1 -- 172.21.15.28:0/770411272 shutdown_connections 2024-04-21T08:48:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.240+0000 7f76190c1640 1 -- 172.21.15.28:0/770411272 wait complete. 2024-04-21T08:48:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f76190c1640 1 Processor -- start 2024-04-21T08:48:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f76190c1640 1 -- start start 2024-04-21T08:48:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f76190c1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7614100e70 0x7f7614198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f76190c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614101fc0 0x7f76141986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f76190c1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7614102980 0x7f761419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f76190c1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f761410c9c0 con 0x7f7614101fc0 2024-04-21T08:48:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f76190c1640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f761410c840 con 0x7f7614100e70 2024-04-21T08:48:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f76190c1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f761410cb40 con 0x7f7614102980 2024-04-21T08:48:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f761259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614101fc0 0x7f76141986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f761259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614101fc0 0x7f76141986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38742/0 (socket says 172.21.15.28:38742) 2024-04-21T08:48:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f761259c640 1 -- 172.21.15.28:0/242683103 learned_addr learned my addr 172.21.15.28:0/242683103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f7612d9d640 1 --2- 172.21.15.28:0/242683103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7614100e70 0x7f7614198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f761359e640 1 --2- 172.21.15.28:0/242683103 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7614102980 0x7f761419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f761259c640 1 -- 172.21.15.28:0/242683103 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7614102980 msgr2=0x7f761419f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f761259c640 1 --2- 172.21.15.28:0/242683103 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7614102980 0x7f761419f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f761259c640 1 -- 172.21.15.28:0/242683103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7614100e70 msgr2=0x7f7614198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f761259c640 1 --2- 172.21.15.28:0/242683103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7614100e70 0x7f7614198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.241+0000 7f761259c640 1 -- 172.21.15.28:0/242683103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f761419fd80 con 0x7f7614101fc0 2024-04-21T08:48:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.242+0000 7f761259c640 1 --2- 172.21.15.28:0/242683103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614101fc0 0x7f76141986a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f760800d770 tx=0x7f760800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.242+0000 7f75fbfff640 1 -- 172.21.15.28:0/242683103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7608017070 con 0x7f7614101fc0 2024-04-21T08:48:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.242+0000 7f75fbfff640 1 -- 172.21.15.28:0/242683103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7608009c30 con 0x7f7614101fc0 2024-04-21T08:48:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.242+0000 7f76190c1640 1 -- 172.21.15.28:0/242683103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76141a0070 con 0x7f7614101fc0 2024-04-21T08:48:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.242+0000 7f75fbfff640 1 -- 172.21.15.28:0/242683103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7608003cc0 con 0x7f7614101fc0 2024-04-21T08:48:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.242+0000 7f76190c1640 1 -- 172.21.15.28:0/242683103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76141a05c0 con 0x7f7614101fc0 2024-04-21T08:48:53.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.243+0000 7f76190c1640 1 -- 172.21.15.28:0/242683103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76141a0820 con 0x7f7614101fc0 2024-04-21T08:48:53.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.244+0000 7f75fbfff640 1 -- 172.21.15.28:0/242683103 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7608003e60 con 0x7f7614101fc0 2024-04-21T08:48:53.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.244+0000 7f75fbfff640 1 --2- 172.21.15.28:0/242683103 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f75fc0a3bb0 0x7f75fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.245+0000 7f7612d9d640 1 --2- 172.21.15.28:0/242683103 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f75fc0a3bb0 0x7f75fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.245+0000 7f75fbfff640 1 -- 172.21.15.28:0/242683103 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f76080d0030 con 0x7f7614101fc0 2024-04-21T08:48:53.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.245+0000 7f7612d9d640 1 --2- 172.21.15.28:0/242683103 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f75fc0a3bb0 0x7f75fc0a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f76040058c0 tx=0x7f760400a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:53.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.249+0000 7f75fbfff640 1 -- 172.21.15.28:0/242683103 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f760809c770 con 0x7f7614101fc0 2024-04-21T08:48:53.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.437+0000 7f76190c1640 1 -- 172.21.15.28:0/242683103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f76141a0ef0 con 0x7f7614101fc0 2024-04-21T08:48:53.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.439+0000 7f75fbfff640 1 -- 172.21.15.28:0/242683103 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24101 (secure 0 0 0) 0x7f7608009de0 con 0x7f7614101fc0 2024-04-21T08:48:53.439 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:53.439 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:48:51.121128+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:48:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.441+0000 7f76190c1640 1 -- 172.21.15.28:0/242683103 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f75fc0a3bb0 msgr2=0x7f75fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.441+0000 7f76190c1640 1 --2- 172.21.15.28:0/242683103 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f75fc0a3bb0 0x7f75fc0a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f76040058c0 tx=0x7f760400a250 comp rx=0 tx=0).stop 2024-04-21T08:48:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.441+0000 7f76190c1640 1 -- 172.21.15.28:0/242683103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614101fc0 msgr2=0x7f76141986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.441+0000 7f76190c1640 1 --2- 172.21.15.28:0/242683103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614101fc0 0x7f76141986a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f760800d770 tx=0x7f760800dc30 comp rx=0 tx=0).stop 2024-04-21T08:48:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.441+0000 7f76190c1640 1 -- 172.21.15.28:0/242683103 shutdown_connections 2024-04-21T08:48:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.441+0000 7f76190c1640 1 --2- 172.21.15.28:0/242683103 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7614102980 0x7f761419f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.441+0000 7f76190c1640 1 --2- 172.21.15.28:0/242683103 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f75fc0a3bb0 0x7f75fc0a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.441+0000 7f76190c1640 1 --2- 172.21.15.28:0/242683103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7614101fc0 0x7f76141986a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.441+0000 7f76190c1640 1 --2- 172.21.15.28:0/242683103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7614100e70 0x7f7614198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.441+0000 7f76190c1640 1 -- 172.21.15.28:0/242683103 >> 172.21.15.28:0/242683103 conn(0x7f76140fc3e0 msgr2=0x7f76140fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.441+0000 7f76190c1640 1 -- 172.21.15.28:0/242683103 shutdown_connections 2024-04-21T08:48:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.441+0000 7f76190c1640 1 -- 172.21.15.28:0/242683103 wait complete. 2024-04-21T08:48:53.453 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-21T08:48:45.911185+0000', 'modified': '2024-04-21T08:48:51.121352+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24488}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24488': {'gid': 24488, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.152:6839/3821365920', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6838', 'nonce': 3821365920}, {'type': 'v1', 'addr': '172.21.15.152:6839', 'nonce': 3821365920}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24488, 'qdb_cluster': [24488]} 2024-04-21T08:48:53.453 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-21T08:48:53.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:48:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.556+0000 7fc1e3b8f640 1 -- 172.21.15.28:0/2543179011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1dc107c90 msgr2=0x7fc1dc10a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.556+0000 7fc1e3b8f640 1 --2- 172.21.15.28:0/2543179011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1dc107c90 0x7fc1dc10a0e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4009ca0 tx=0x7fc1d402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.556+0000 7fc1e3b8f640 1 -- 172.21.15.28:0/2543179011 shutdown_connections 2024-04-21T08:48:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.556+0000 7fc1e3b8f640 1 --2- 172.21.15.28:0/2543179011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1dc107c90 0x7fc1dc10a0e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.556+0000 7fc1e3b8f640 1 --2- 172.21.15.28:0/2543179011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1dc105360 0x7fc1dc107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.556+0000 7fc1e3b8f640 1 --2- 172.21.15.28:0/2543179011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1dc102a30 0x7fc1dc104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.556+0000 7fc1e3b8f640 1 -- 172.21.15.28:0/2543179011 >> 172.21.15.28:0/2543179011 conn(0x7fc1dc0fc440 msgr2=0x7fc1dc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.557+0000 7fc1e3b8f640 1 -- 172.21.15.28:0/2543179011 shutdown_connections 2024-04-21T08:48:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.557+0000 7fc1e3b8f640 1 -- 172.21.15.28:0/2543179011 wait complete. 2024-04-21T08:48:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.557+0000 7fc1e3b8f640 1 Processor -- start 2024-04-21T08:48:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.557+0000 7fc1e3b8f640 1 -- start start 2024-04-21T08:48:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.557+0000 7fc1e3b8f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1dc102a30 0x7fc1dc1023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.557+0000 7fc1e3b8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1dc105360 0x7fc1dc1009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.557+0000 7fc1e3b8f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1dc107c90 0x7fc1dc100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.557+0000 7fc1e3b8f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc10daf0 con 0x7fc1dc105360 2024-04-21T08:48:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.557+0000 7fc1e3b8f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc10d970 con 0x7fc1dc102a30 2024-04-21T08:48:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.557+0000 7fc1e3b8f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc10dc70 con 0x7fc1dc107c90 2024-04-21T08:48:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.557+0000 7fc1e112a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1dc105360 0x7fc1dc1009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.557+0000 7fc1e112a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1dc105360 0x7fc1dc1009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38764/0 (socket says 172.21.15.28:38764) 2024-04-21T08:48:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.557+0000 7fc1e112a640 1 -- 172.21.15.28:0/2016566788 learned_addr learned my addr 172.21.15.28:0/2016566788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.558+0000 7fc1e212c640 1 --2- 172.21.15.28:0/2016566788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1dc107c90 0x7fc1dc100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.558+0000 7fc1e192b640 1 --2- 172.21.15.28:0/2016566788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1dc102a30 0x7fc1dc1023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.558+0000 7fc1e112a640 1 -- 172.21.15.28:0/2016566788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1dc107c90 msgr2=0x7fc1dc100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.558+0000 7fc1e112a640 1 --2- 172.21.15.28:0/2016566788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1dc107c90 0x7fc1dc100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.558+0000 7fc1e112a640 1 -- 172.21.15.28:0/2016566788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1dc102a30 msgr2=0x7fc1dc1023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.558+0000 7fc1e112a640 1 --2- 172.21.15.28:0/2016566788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1dc102a30 0x7fc1dc1023a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.558+0000 7fc1e112a640 1 -- 172.21.15.28:0/2016566788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1dc1017b0 con 0x7fc1dc105360 2024-04-21T08:48:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.558+0000 7fc1e112a640 1 --2- 172.21.15.28:0/2016566788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1dc105360 0x7fc1dc1009e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc00c970 tx=0x7fc1cc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.558+0000 7fc1d2ffd640 1 -- 172.21.15.28:0/2016566788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc007bb0 con 0x7fc1dc105360 2024-04-21T08:48:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.558+0000 7fc1d2ffd640 1 -- 172.21.15.28:0/2016566788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1cc007d50 con 0x7fc1dc105360 2024-04-21T08:48:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.558+0000 7fc1e3b8f640 1 -- 172.21.15.28:0/2016566788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1dc1a22c0 con 0x7fc1dc105360 2024-04-21T08:48:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.558+0000 7fc1d2ffd640 1 -- 172.21.15.28:0/2016566788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc00f450 con 0x7fc1dc105360 2024-04-21T08:48:53.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.558+0000 7fc1e3b8f640 1 -- 172.21.15.28:0/2016566788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1dc1a2810 con 0x7fc1dc105360 2024-04-21T08:48:53.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.560+0000 7fc1d2ffd640 1 -- 172.21.15.28:0/2016566788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc1cc004f20 con 0x7fc1dc105360 2024-04-21T08:48:53.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.561+0000 7fc1e677c740 1 Processor -- start 2024-04-21T08:48:53.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.561+0000 7fc1e677c740 1 -- start start 2024-04-21T08:48:53.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.561+0000 7fc1e677c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef46260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.561+0000 7fc1e677c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef46830 0x56056ef46cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.561+0000 7fc1e677c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef472d0 0x56056ef4deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.561+0000 7fc1e677c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56056eeaab10 con 0x56056ef45e40 2024-04-21T08:48:53.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.561+0000 7fc1e677c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56056ee48240 con 0x56056ef46830 2024-04-21T08:48:53.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.561+0000 7fc1e677c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56056eea6ad0 con 0x56056ef472d0 2024-04-21T08:48:53.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.561+0000 7fc1e212c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef46260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.561+0000 7fc1e212c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef46260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38780/0 (socket says 172.21.15.28:38780) 2024-04-21T08:48:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1e212c640 1 -- 172.21.15.28:0/3718123736 learned_addr learned my addr 172.21.15.28:0/3718123736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1e192b640 1 --2- 172.21.15.28:0/3718123736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef46830 0x56056ef46cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1d2ffd640 1 --2- 172.21.15.28:0/2016566788 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc1c80a3bb0 0x7fc1c80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1e112a640 1 --2- 172.21.15.28:0/3718123736 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef472d0 0x56056ef4deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1e212c640 1 -- 172.21.15.28:0/3718123736 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef472d0 msgr2=0x56056ef4deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1e212c640 1 --2- 172.21.15.28:0/3718123736 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef472d0 0x56056ef4deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1e212c640 1 -- 172.21.15.28:0/3718123736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef46830 msgr2=0x56056ef46cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1e212c640 1 --2- 172.21.15.28:0/3718123736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef46830 0x56056ef46cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1e212c640 1 -- 172.21.15.28:0/3718123736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56056ebbd2d0 con 0x56056ef45e40 2024-04-21T08:48:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1d2ffd640 1 -- 172.21.15.28:0/2016566788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc1cc0d0790 con 0x7fc1dc105360 2024-04-21T08:48:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1e212c640 1 --2- 172.21.15.28:0/2016566788 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc1c80a3bb0 0x7fc1c80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1e212c640 1 --2- 172.21.15.28:0/3718123736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef46260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4032380 tx=0x7fc1d40099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936a9165253718f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1c7fff640 1 -- 172.21.15.28:0/3718123736 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4039040 con 0x56056ef45e40 2024-04-21T08:48:53.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1c7fff640 1 -- 172.21.15.28:0/3718123736 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1d4003b70 con 0x56056ef45e40 2024-04-21T08:48:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1e212c640 1 --2- 172.21.15.28:0/2016566788 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc1c80a3bb0 0x7fc1c80a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4047000 tx=0x7fc1d4044000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.562+0000 7fc1c7fff640 1 -- 172.21.15.28:0/3718123736 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4034640 con 0x56056ef45e40 2024-04-21T08:48:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.563+0000 7fc1e677c740 1 -- 172.21.15.28:0/3718123736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 msgr2=0x56056ef46260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.563+0000 7fc1e677c740 1 --2- 172.21.15.28:0/3718123736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef46260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4032380 tx=0x7fc1d40099f0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.563+0000 7fc1e677c740 1 -- 172.21.15.28:0/3718123736 shutdown_connections 2024-04-21T08:48:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.563+0000 7fc1e677c740 1 --2- 172.21.15.28:0/3718123736 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef472d0 0x56056ef4deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.563+0000 7fc1e677c740 1 --2- 172.21.15.28:0/3718123736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef46830 0x56056ef46cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.563+0000 7fc1e677c740 1 --2- 172.21.15.28:0/3718123736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef46260 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.563+0000 7fc1e677c740 1 -- 172.21.15.28:0/3718123736 >> 172.21.15.28:0/3718123736 conn(0x56056ef41f40 msgr2=0x56056ef40490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.563+0000 7fc1e677c740 1 -- 172.21.15.28:0/3718123736 shutdown_connections 2024-04-21T08:48:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.563+0000 7fc1e677c740 1 -- 172.21.15.28:0/3718123736 wait complete. 2024-04-21T08:48:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.563+0000 7fc1e677c740 1 -- start start 2024-04-21T08:48:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e677c740 1 Processor -- start 2024-04-21T08:48:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e677c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef45e40 0x56056ef4feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e677c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef46830 0x56056ef503f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e677c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef472d0 0x56056ef57430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e677c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56056ef4e4b0 con 0x56056ef46830 2024-04-21T08:48:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e677c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56056ee33190 con 0x56056ef472d0 2024-04-21T08:48:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e677c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56056ef4e630 con 0x56056ef45e40 2024-04-21T08:48:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e677c740 10 client.0 _ll_register_callbacks cb 0x56056ee2e8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e112a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef46830 0x56056ef503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e112a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef46830 0x56056ef503f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38792/0 (socket says 172.21.15.28:38792) 2024-04-21T08:48:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e112a640 1 -- 172.21.15.28:0/503509107 learned_addr learned my addr 172.21.15.28:0/503509107 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e192b640 1 --2- 172.21.15.28:0/503509107 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef45e40 0x56056ef4feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e212c640 1 --2- 172.21.15.28:0/503509107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef472d0 0x56056ef57430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e112a640 1 -- 172.21.15.28:0/503509107 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef45e40 msgr2=0x56056ef4feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e112a640 1 --2- 172.21.15.28:0/503509107 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef45e40 0x56056ef4feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e112a640 1 -- 172.21.15.28:0/503509107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef472d0 msgr2=0x56056ef57430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:53.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e112a640 1 --2- 172.21.15.28:0/503509107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef472d0 0x56056ef57430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e112a640 1 -- 172.21.15.28:0/503509107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56056ec876b0 con 0x56056ef46830 2024-04-21T08:48:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e212c640 1 --2- 172.21.15.28:0/503509107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef472d0 0x56056ef57430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1e112a640 1 --2- 172.21.15.28:0/503509107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef46830 0x56056ef503f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0ce050 tx=0x7fc1cc09cff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb5d11560904718 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.564+0000 7fc1c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:48:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.565+0000 7fc1c57fa640 1 -- 172.21.15.28:0/503509107 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc01d040 con 0x56056ef46830 2024-04-21T08:48:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.565+0000 7fc1c57fa640 1 -- 172.21.15.28:0/503509107 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc1cc0055d0 con 0x56056ef46830 2024-04-21T08:48:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.565+0000 7fc1e677c740 1 -- 172.21.15.28:0/503509107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1d4003b70 con 0x56056ef46830 2024-04-21T08:48:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.565+0000 7fc1c57fa640 1 -- 172.21.15.28:0/503509107 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fc1cc0a54a0 con 0x56056ef46830 2024-04-21T08:48:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.565+0000 7fc1e677c740 10 client.14754 fetch_fsmap learned FSMap version 17 2024-04-21T08:48:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.565+0000 7fc1e677c740 1 -- 172.21.15.28:0/503509107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x56056ef5a810 con 0x56056ef46830 2024-04-21T08:48:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.565+0000 7fc1c57fa640 1 -- 172.21.15.28:0/503509107 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1d4003b70 con 0x56056ef46830 2024-04-21T08:48:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.565+0000 7fc1c57fa640 1 -- 172.21.15.28:0/503509107 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fc1cc087070 con 0x56056ef46830 2024-04-21T08:48:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.565+0000 7fc1e677c740 10 client.14754 fetch_fsmap finished waiting for FSMap version 17 2024-04-21T08:48:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.565+0000 7fc1e677c740 10 client.14754 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:48:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.565+0000 7fc1e677c740 20 client.14754 populate_metadata read hostname 'smithi028' 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.565+0000 7fc1e677c740 1 --2- 172.21.15.28:0/503509107 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56056ef5ce70 0x56056ef5e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.565+0000 7fc1e677c740 4 client.14754 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.565+0000 7fc1e677c740 1 -- 172.21.15.28:0/503509107 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56056eeb8530 con 0x56056ef5ce70 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.565+0000 7fc1e192b640 1 --2- 172.21.15.28:0/503509107 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56056ef5ce70 0x56056ef5e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.566+0000 7fc1e192b640 1 --2- 172.21.15.28:0/503509107 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56056ef5ce70 0x56056ef5e2e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a80f203451f335b server_cookie=b76a77277fc2713f in_seq=0 out_seq=0 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.566+0000 7fc1c57fa640 10 client.14754 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.567+0000 7fc1c57fa640 1 -- 172.21.15.28:0/503509107 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56056eeb8530 con 0x56056ef5ce70 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.567+0000 7fc1c57fa640 10 client.14754 handle_command_reply: tid=0 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.567+0000 7fc1e677c740 1 client.14754 shutdown 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.567+0000 7fc1e677c740 1 -- 172.21.15.28:0/503509107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef46830 msgr2=0x56056ef503f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.567+0000 7fc1e677c740 1 --2- 172.21.15.28:0/503509107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef46830 0x56056ef503f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc0ce050 tx=0x7fc1cc09cff0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.567+0000 7fc1e677c740 1 -- 172.21.15.28:0/503509107 shutdown_connections 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.567+0000 7fc1e677c740 1 --2- 172.21.15.28:0/503509107 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56056ef5ce70 0x56056ef5e2e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.567+0000 7fc1e677c740 1 --2- 172.21.15.28:0/503509107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef472d0 0x56056ef57430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.567+0000 7fc1e677c740 1 --2- 172.21.15.28:0/503509107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef46830 0x56056ef503f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.567+0000 7fc1e677c740 1 --2- 172.21.15.28:0/503509107 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef45e40 0x56056ef4feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.567+0000 7fc1e677c740 1 -- 172.21.15.28:0/503509107 >> 172.21.15.28:0/503509107 conn(0x56056ef41f40 msgr2=0x56056ef3fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:53.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.567+0000 7fc1c57fa640 0 client.14754 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:48:53.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.567+0000 7fc1e677c740 1 -- 172.21.15.28:0/503509107 shutdown_connections 2024-04-21T08:48:53.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.567+0000 7fc1e677c740 1 -- 172.21.15.28:0/503509107 wait complete. 2024-04-21T08:48:53.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.567+0000 7fc1e677c740 20 client.14754 trim_cache size 0 max 16384 2024-04-21T08:48:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.585+0000 7fc1e677c740 1 Processor -- start 2024-04-21T08:48:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.585+0000 7fc1e677c740 1 -- start start 2024-04-21T08:48:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.585+0000 7fc1e677c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef4a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.585+0000 7fc1e677c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef46830 0x56056ef4ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.585+0000 7fc1e677c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef472d0 0x56056ef573f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.585+0000 7fc1e677c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56056ef5c620 con 0x56056ef45e40 2024-04-21T08:48:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.585+0000 7fc1e677c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56056ef5ccf0 con 0x56056ef472d0 2024-04-21T08:48:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.585+0000 7fc1e677c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56056ef5d3c0 con 0x56056ef46830 2024-04-21T08:48:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.586+0000 7fc1e192b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef4a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.586+0000 7fc1e192b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef4a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38794/0 (socket says 172.21.15.28:38794) 2024-04-21T08:48:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.586+0000 7fc1e192b640 1 -- 172.21.15.28:0/2673277520 learned_addr learned my addr 172.21.15.28:0/2673277520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.586+0000 7fc1e212c640 1 --2- 172.21.15.28:0/2673277520 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef472d0 0x56056ef573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.586+0000 7fc1e192b640 1 -- 172.21.15.28:0/2673277520 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef46830 msgr2=0x56056ef4ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.586+0000 7fc1e192b640 1 --2- 172.21.15.28:0/2673277520 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef46830 0x56056ef4ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.586+0000 7fc1e192b640 1 -- 172.21.15.28:0/2673277520 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef472d0 msgr2=0x56056ef573f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.586+0000 7fc1e192b640 1 --2- 172.21.15.28:0/2673277520 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef472d0 0x56056ef573f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.586+0000 7fc1e192b640 1 -- 172.21.15.28:0/2673277520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56056ef3be70 con 0x56056ef45e40 2024-04-21T08:48:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.586+0000 7fc1e212c640 1 --2- 172.21.15.28:0/2673277520 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef472d0 0x56056ef573f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.586+0000 7fc1e192b640 1 --2- 172.21.15.28:0/2673277520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef4a840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009a80 tx=0x7fc1d80093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2048ac5185302879 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.586+0000 7fc1c77fe640 1 -- 172.21.15.28:0/2673277520 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8014a90 con 0x56056ef45e40 2024-04-21T08:48:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.586+0000 7fc1c77fe640 1 -- 172.21.15.28:0/2673277520 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1d801b7e0 con 0x56056ef45e40 2024-04-21T08:48:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.586+0000 7fc1c77fe640 1 -- 172.21.15.28:0/2673277520 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d80128b0 con 0x56056ef45e40 2024-04-21T08:48:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.587+0000 7fc1e677c740 1 -- 172.21.15.28:0/2673277520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 msgr2=0x56056ef4a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.587+0000 7fc1e677c740 1 --2- 172.21.15.28:0/2673277520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef4a840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009a80 tx=0x7fc1d80093d0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.587+0000 7fc1e677c740 1 -- 172.21.15.28:0/2673277520 shutdown_connections 2024-04-21T08:48:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.587+0000 7fc1e677c740 1 --2- 172.21.15.28:0/2673277520 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef472d0 0x56056ef573f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.587+0000 7fc1e677c740 1 --2- 172.21.15.28:0/2673277520 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef46830 0x56056ef4ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.587+0000 7fc1e677c740 1 --2- 172.21.15.28:0/2673277520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef4a840 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.587+0000 7fc1e677c740 1 -- 172.21.15.28:0/2673277520 >> 172.21.15.28:0/2673277520 conn(0x56056ef41f40 msgr2=0x56056ef4ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.587+0000 7fc1e677c740 1 -- 172.21.15.28:0/2673277520 shutdown_connections 2024-04-21T08:48:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.587+0000 7fc1e677c740 1 -- 172.21.15.28:0/2673277520 wait complete. 2024-04-21T08:48:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.587+0000 7fc1e677c740 1 -- start start 2024-04-21T08:48:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.587+0000 7fc1e677c740 1 Processor -- start 2024-04-21T08:48:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e677c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef533d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e677c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef46830 0x56056ef55920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e677c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef472d0 0x56056ef55f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e677c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56056ef5e160 con 0x56056ef45e40 2024-04-21T08:48:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e677c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56056ef5da90 con 0x56056ef46830 2024-04-21T08:48:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e677c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56056ef5e830 con 0x56056ef472d0 2024-04-21T08:48:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e677c740 10 client.0 _ll_register_callbacks cb 0x7fc1cc098410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e192b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e192b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38810/0 (socket says 172.21.15.28:38810) 2024-04-21T08:48:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e192b640 1 -- 172.21.15.28:0/2754593277 learned_addr learned my addr 172.21.15.28:0/2754593277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e112a640 1 --2- 172.21.15.28:0/2754593277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef46830 0x56056ef55920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e212c640 1 --2- 172.21.15.28:0/2754593277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef472d0 0x56056ef55f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e192b640 1 -- 172.21.15.28:0/2754593277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef472d0 msgr2=0x56056ef55f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e192b640 1 --2- 172.21.15.28:0/2754593277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef472d0 0x56056ef55f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e192b640 1 -- 172.21.15.28:0/2754593277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef46830 msgr2=0x56056ef55920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e192b640 1 --2- 172.21.15.28:0/2754593277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef46830 0x56056ef55920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e192b640 1 -- 172.21.15.28:0/2754593277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56056ef56780 con 0x56056ef45e40 2024-04-21T08:48:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e192b640 1 --2- 172.21.15.28:0/2754593277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef533d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8016010 tx=0x7fc1d8009d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728581b8ebfae831 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:48:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1c57fa640 1 -- 172.21.15.28:0/2754593277 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8031060 con 0x56056ef45e40 2024-04-21T08:48:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1c57fa640 1 -- 172.21.15.28:0/2754593277 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc1d8014190 con 0x56056ef45e40 2024-04-21T08:48:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e677c740 1 -- 172.21.15.28:0/2754593277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1d801b7e0 con 0x56056ef45e40 2024-04-21T08:48:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1c57fa640 1 -- 172.21.15.28:0/2754593277 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fc1d8038050 con 0x56056ef45e40 2024-04-21T08:48:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e677c740 10 client.14760 fetch_fsmap learned FSMap version 17 2024-04-21T08:48:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.588+0000 7fc1e677c740 1 -- 172.21.15.28:0/2754593277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x56056efc2ee0 con 0x56056ef45e40 2024-04-21T08:48:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.589+0000 7fc1c57fa640 1 -- 172.21.15.28:0/2754593277 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1d801b7e0 con 0x56056ef45e40 2024-04-21T08:48:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.589+0000 7fc1c57fa640 1 -- 172.21.15.28:0/2754593277 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fc1d802b030 con 0x56056ef45e40 2024-04-21T08:48:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.589+0000 7fc1e677c740 10 client.14760 fetch_fsmap finished waiting for FSMap version 17 2024-04-21T08:48:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.589+0000 7fc1e677c740 10 client.14760 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:48:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.589+0000 7fc1e677c740 20 client.14760 populate_metadata read hostname 'smithi028' 2024-04-21T08:48:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.589+0000 7fc1e677c740 1 --2- 172.21.15.28:0/2754593277 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56056efc78f0 0x56056efc6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.589+0000 7fc1e677c740 4 client.14760 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:48:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.589+0000 7fc1e677c740 1 -- 172.21.15.28:0/2754593277 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "status"}) -- 0x56056eeb23a0 con 0x56056efc78f0 2024-04-21T08:48:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.589+0000 7fc1e112a640 1 --2- 172.21.15.28:0/2754593277 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56056efc78f0 0x56056efc6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1c57fa640 10 client.14760 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:48:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1e112a640 1 --2- 172.21.15.28:0/2754593277 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56056efc78f0 0x56056efc6600 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c4bb7016c89099c server_cookie=3e4be99b5a7cdca6 in_seq=0 out_seq=0 2024-04-21T08:48:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1c57fa640 1 -- 172.21.15.28:0/2754593277 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x56056eeb23a0 con 0x56056efc78f0 2024-04-21T08:48:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1c57fa640 10 client.14760 handle_command_reply: tid=0 2024-04-21T08:48:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1e677c740 1 client.14760 shutdown 2024-04-21T08:48:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1e677c740 1 -- 172.21.15.28:0/2754593277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 msgr2=0x56056ef533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1e677c740 1 --2- 172.21.15.28:0/2754593277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef533d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8016010 tx=0x7fc1d8009d50 comp rx=0 tx=0).stop 2024-04-21T08:48:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1e677c740 1 -- 172.21.15.28:0/2754593277 shutdown_connections 2024-04-21T08:48:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1e677c740 1 --2- 172.21.15.28:0/2754593277 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56056efc78f0 0x56056efc6600 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1e677c740 1 --2- 172.21.15.28:0/2754593277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56056ef472d0 0x56056ef55f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1e677c740 1 --2- 172.21.15.28:0/2754593277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56056ef46830 0x56056ef55920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1e677c740 1 --2- 172.21.15.28:0/2754593277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56056ef45e40 0x56056ef533d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1c57fa640 0 client.14760 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:48:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1e677c740 1 -- 172.21.15.28:0/2754593277 >> 172.21.15.28:0/2754593277 conn(0x56056ef41f40 msgr2=0x56056ef44c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1e677c740 1 -- 172.21.15.28:0/2754593277 shutdown_connections 2024-04-21T08:48:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1e677c740 1 -- 172.21.15.28:0/2754593277 wait complete. 2024-04-21T08:48:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.590+0000 7fc1e677c740 20 client.14760 trim_cache size 0 max 16384 2024-04-21T08:48:53.596 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:48:53.596 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "b51a76c8-ffba-11ee-bc93-c7b262605968", 2024-04-21T08:48:53.596 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-21T08:48:53.596 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24488, 2024-04-21T08:48:53.596 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-21T08:48:53.596 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-21T08:48:53.596 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-21T08:48:53.596 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.8578419119999996, 2024-04-21T08:48:53.597 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 17, 2024-04-21T08:48:53.597 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 89, 2024-04-21T08:48:53.597 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:48:53.597 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 12.030045196 2024-04-21T08:48:53.597 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:48:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.591+0000 7fc1e3b8f640 1 -- 172.21.15.28:0/2016566788 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc1c80a3bb0 msgr2=0x7fc1c80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.591+0000 7fc1e3b8f640 1 --2- 172.21.15.28:0/2016566788 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc1c80a3bb0 0x7fc1c80a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4047000 tx=0x7fc1d4044000 comp rx=0 tx=0).stop 2024-04-21T08:48:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.591+0000 7fc1e3b8f640 1 -- 172.21.15.28:0/2016566788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1dc105360 msgr2=0x7fc1dc1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.591+0000 7fc1e3b8f640 1 --2- 172.21.15.28:0/2016566788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1dc105360 0x7fc1dc1009e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc00c970 tx=0x7fc1cc00ce30 comp rx=0 tx=0).stop 2024-04-21T08:48:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.591+0000 7fc1e3b8f640 1 -- 172.21.15.28:0/2016566788 shutdown_connections 2024-04-21T08:48:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.591+0000 7fc1e3b8f640 1 --2- 172.21.15.28:0/2016566788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1dc107c90 0x7fc1dc100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.591+0000 7fc1e3b8f640 1 --2- 172.21.15.28:0/2016566788 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc1c80a3bb0 0x7fc1c80a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.591+0000 7fc1e3b8f640 1 --2- 172.21.15.28:0/2016566788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1dc105360 0x7fc1dc1009e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.591+0000 7fc1e3b8f640 1 --2- 172.21.15.28:0/2016566788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1dc102a30 0x7fc1dc1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.591+0000 7fc1e3b8f640 1 -- 172.21.15.28:0/2016566788 >> 172.21.15.28:0/2016566788 conn(0x7fc1dc0fc440 msgr2=0x7fc1dc103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.591+0000 7fc1e3b8f640 1 -- 172.21.15.28:0/2016566788 shutdown_connections 2024-04-21T08:48:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:53.591+0000 7fc1e3b8f640 1 -- 172.21.15.28:0/2016566788 wait complete. 2024-04-21T08:48:53.601 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-21T08:48:53.601 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:48:53.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph fs ls 2024-04-21T08:48:53.859 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:48:54.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:53 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/798721232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:54.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:53 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/242683103' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:54.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:53 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/798721232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:54.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:53 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/242683103' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:54.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:53 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/798721232' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:54.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:53 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/242683103' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:48:55.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:54 smithi028 ceph-mon[28003]: pgmap v298: 129 pgs: 9 creating+peering, 13 unknown, 107 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 4 op/s 2024-04-21T08:48:55.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:54 smithi028 ceph-mon[28003]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:48:55.026 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:54 smithi028 ceph-mon[28003]: Cluster is now healthy 2024-04-21T08:48:55.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:54 smithi152 ceph-mon[31401]: pgmap v298: 129 pgs: 9 creating+peering, 13 unknown, 107 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 4 op/s 2024-04-21T08:48:55.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:54 smithi152 ceph-mon[31401]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:48:55.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:54 smithi152 ceph-mon[31401]: Cluster is now healthy 2024-04-21T08:48:55.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:54 smithi171 ceph-mon[31358]: pgmap v298: 129 pgs: 9 creating+peering, 13 unknown, 107 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 4 op/s 2024-04-21T08:48:55.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:54 smithi171 ceph-mon[31358]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:48:55.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:54 smithi171 ceph-mon[31358]: Cluster is now healthy 2024-04-21T08:48:55.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.604+0000 7f278f26c700 1 -- 172.21.15.28:0/2211166585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2788108a70 msgr2=0x7f2788108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.604+0000 7f278f26c700 1 --2- 172.21.15.28:0/2211166585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2788108a70 0x7f2788108e50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2784009990 tx=0x7f2784009ca0 comp rx=0 tx=0).stop 2024-04-21T08:48:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.604+0000 7f278f26c700 1 -- 172.21.15.28:0/2211166585 shutdown_connections 2024-04-21T08:48:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.604+0000 7f278f26c700 1 --2- 172.21.15.28:0/2211166585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2788108a70 0x7f2788108e50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.604+0000 7f278f26c700 1 --2- 172.21.15.28:0/2211166585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2788103480 0x7f278810f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.604+0000 7f278f26c700 1 --2- 172.21.15.28:0/2211166585 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2788102ac0 0x7f2788102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.604+0000 7f278f26c700 1 -- 172.21.15.28:0/2211166585 >> 172.21.15.28:0/2211166585 conn(0x7f2788079bb0 msgr2=0x7f2788079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.604+0000 7f278f26c700 1 -- 172.21.15.28:0/2211166585 shutdown_connections 2024-04-21T08:48:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.605+0000 7f278f26c700 1 -- 172.21.15.28:0/2211166585 wait complete. 2024-04-21T08:48:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.605+0000 7f278f26c700 1 Processor -- start 2024-04-21T08:48:55.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.605+0000 7f278f26c700 1 -- start start 2024-04-21T08:48:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.605+0000 7f278f26c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2788102ac0 0x7f2788199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.605+0000 7f278f26c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2788103480 0x7f27881996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.605+0000 7f278f26c700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2788108a70 0x7f278819da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.605+0000 7f278f26c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2788112ca0 con 0x7f2788102ac0 2024-04-21T08:48:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.605+0000 7f278f26c700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2788112b20 con 0x7f2788108a70 2024-04-21T08:48:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.605+0000 7f278f26c700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2788112e20 con 0x7f2788103480 2024-04-21T08:48:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f278d008700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2788102ac0 0x7f2788199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f278d008700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2788102ac0 0x7f2788199190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44006/0 (socket says 172.21.15.28:44006) 2024-04-21T08:48:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f278d008700 1 -- 172.21.15.28:0/756310228 learned_addr learned my addr 172.21.15.28:0/756310228 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f278d809700 1 --2- 172.21.15.28:0/756310228 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2788108a70 0x7f278819da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f278c807700 1 --2- 172.21.15.28:0/756310228 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2788103480 0x7f27881996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f278d008700 1 -- 172.21.15.28:0/756310228 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2788103480 msgr2=0x7f27881996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f278d008700 1 --2- 172.21.15.28:0/756310228 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2788103480 0x7f27881996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f278d008700 1 -- 172.21.15.28:0/756310228 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2788108a70 msgr2=0x7f278819da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f278d008700 1 --2- 172.21.15.28:0/756310228 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2788108a70 0x7f278819da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f278d008700 1 -- 172.21.15.28:0/756310228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278819e1a0 con 0x7f2788102ac0 2024-04-21T08:48:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f278d809700 1 --2- 172.21.15.28:0/756310228 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2788108a70 0x7f278819da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f278d008700 1 --2- 172.21.15.28:0/756310228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2788102ac0 0x7f2788199190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f27840057f0 tx=0x7f2784003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f277e7fc700 1 -- 172.21.15.28:0/756310228 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784027070 con 0x7f2788102ac0 2024-04-21T08:48:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f278f26c700 1 -- 172.21.15.28:0/756310228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f278819e430 con 0x7f2788102ac0 2024-04-21T08:48:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f277e7fc700 1 -- 172.21.15.28:0/756310228 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27840041c0 con 0x7f2788102ac0 2024-04-21T08:48:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f277e7fc700 1 -- 172.21.15.28:0/756310228 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278400f460 con 0x7f2788102ac0 2024-04-21T08:48:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.606+0000 7f278f26c700 1 -- 172.21.15.28:0/756310228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27881a5d10 con 0x7f2788102ac0 2024-04-21T08:48:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.607+0000 7f278f26c700 1 -- 172.21.15.28:0/756310228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2788069730 con 0x7f2788102ac0 2024-04-21T08:48:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.608+0000 7f277e7fc700 1 -- 172.21.15.28:0/756310228 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f278402a070 con 0x7f2788102ac0 2024-04-21T08:48:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.609+0000 7f277e7fc700 1 --2- 172.21.15.28:0/756310228 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f27740a3c40 0x7f27740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.609+0000 7f278c807700 1 --2- 172.21.15.28:0/756310228 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f27740a3c40 0x7f27740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.609+0000 7f277e7fc700 1 -- 172.21.15.28:0/756310228 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f27840d1db0 con 0x7f2788102ac0 2024-04-21T08:48:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.610+0000 7f278c807700 1 --2- 172.21.15.28:0/756310228 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f27740a3c40 0x7f27740a6100 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f278807a290 tx=0x7f2778005eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:55.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.613+0000 7f277e7fc700 1 -- 172.21.15.28:0/756310228 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2784031020 con 0x7f2788102ac0 2024-04-21T08:48:55.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.868+0000 7f278f26c700 1 -- 172.21.15.28:0/756310228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f278802df80 con 0x7f2788102ac0 2024-04-21T08:48:55.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.869+0000 7f277e7fc700 1 -- 172.21.15.28:0/756310228 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f2784013070 con 0x7f2788102ac0 2024-04-21T08:48:55.868 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-21T08:48:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.871+0000 7f278f26c700 1 -- 172.21.15.28:0/756310228 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f27740a3c40 msgr2=0x7f27740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.871+0000 7f278f26c700 1 --2- 172.21.15.28:0/756310228 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f27740a3c40 0x7f27740a6100 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f278807a290 tx=0x7f2778005eb0 comp rx=0 tx=0).stop 2024-04-21T08:48:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.871+0000 7f278f26c700 1 -- 172.21.15.28:0/756310228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2788102ac0 msgr2=0x7f2788199190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.871+0000 7f278f26c700 1 --2- 172.21.15.28:0/756310228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2788102ac0 0x7f2788199190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f27840057f0 tx=0x7f2784003730 comp rx=0 tx=0).stop 2024-04-21T08:48:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.871+0000 7f278f26c700 1 -- 172.21.15.28:0/756310228 shutdown_connections 2024-04-21T08:48:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.871+0000 7f278f26c700 1 --2- 172.21.15.28:0/756310228 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f27740a3c40 0x7f27740a6100 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.871+0000 7f278f26c700 1 --2- 172.21.15.28:0/756310228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2788102ac0 0x7f2788199190 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.871+0000 7f278f26c700 1 --2- 172.21.15.28:0/756310228 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2788108a70 0x7f278819da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.871+0000 7f278f26c700 1 --2- 172.21.15.28:0/756310228 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2788103480 0x7f27881996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.871+0000 7f278f26c700 1 -- 172.21.15.28:0/756310228 >> 172.21.15.28:0/756310228 conn(0x7f2788079bb0 msgr2=0x7f2788101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.871+0000 7f278f26c700 1 -- 172.21.15.28:0/756310228 shutdown_connections 2024-04-21T08:48:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:55.871+0000 7f278f26c700 1 -- 172.21.15.28:0/756310228 wait complete. 2024-04-21T08:48:56.556 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-21T08:48:56.556 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-21T08:48:56.556 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-04-21T08:48:56.556 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-21T08:48:56.556 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:48:56.556 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-21T08:48:56.556 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-21T08:48:56.556 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-21T08:48:56.556 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-21T08:48:56.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:48:56.556 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-04-21T08:48:56.574 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-21T08:48:56.574 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-21T08:48:56.574 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-04-21T08:48:56.574 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:48:56.574 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-04-21T08:48:56.574 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:48:56.574 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:48:56.574 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:48:56.575 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-04-21T08:48:56.575 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:48:56.575 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:48:56.575 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-04-21T08:48:56.575 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-21T08:48:56.575 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-04-21T08:48:56.575 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-04-21T08:48:56.575 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-21T08:48:56.575 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42133sec preferred_lft 42133sec 2024-04-21T08:48:56.575 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-21T08:48:56.575 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:48:56.575 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:48:56.575 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-04-21T08:48:56.575 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-04-21T08:48:56.576 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-21T08:48:56.576 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:48:56.576 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-21T08:48:56.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-04-21T08:48:56.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-04-21T08:48:56.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-04-21T08:48:56.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-21T08:48:56.576 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-21T08:48:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:48:56.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:48:56.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:48:56.767 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-21T08:48:56.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:56 smithi028 ceph-mon[28003]: pgmap v299: 129 pgs: 5 creating+peering, 124 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 2 op/s 2024-04-21T08:48:56.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:56 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/756310228' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:48:56.793 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-04-21T08:48:56.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:48:56.795 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-04-21T08:48:56.869 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-04-21T08:48:56.869 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-21T08:48:56.869 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-21T08:48:56.869 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-21T08:48:56.869 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-21T08:48:56.870 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:48:56.870 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-21T08:48:56.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-21T08:48:56.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-21T08:48:56.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-21T08:48:56.870 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-21T08:48:56.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:48:57.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:56 smithi152 ceph-mon[31401]: pgmap v299: 129 pgs: 5 creating+peering, 124 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 2 op/s 2024-04-21T08:48:57.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:56 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/756310228' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:48:57.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:56 smithi171 ceph-mon[31358]: pgmap v299: 129 pgs: 5 creating+peering, 124 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 2 op/s 2024-04-21T08:48:57.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:56 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/756310228' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:48:57.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:48:57.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:48:57.164 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-21T08:48:57.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:48:57.180 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-04-21T08:48:57.236 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:48:57.236 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-21T08:48:57.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:48:57.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-21T08:48:57.236 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-21T08:48:57.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:48:57.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:48:57.357 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-21T08:48:57.357 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:48:57.357 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-21T08:48:57.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-21T08:48:57.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-21T08:48:57.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-21T08:48:57.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-21T08:48:57.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-21T08:48:57.358 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-21T08:48:57.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:48:57.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:48:57.489 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:48:57.489 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-21T08:48:57.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-04-21T08:48:57.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-04-21T08:48:57.489 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-21T08:48:57.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:48:57.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:48:57.569 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-21T08:48:57.569 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:48:57.569 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:48:57.584 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-21T08:48:57.584 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:48:57.585 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:48:57.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-04-21T08:48:57.704 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-04-21T08:48:57.760 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-04-21T08:48:57.760 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-04-21T08:48:57.761 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-04-21T08:48:57.761 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-04-21T08:48:57.761 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-04-21T08:48:57.761 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-04-21T08:48:57.761 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-04-21T08:48:57.761 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-04-21T08:48:57.761 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-04-21T08:48:57.761 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-04-21T08:48:57.761 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-04-21T08:48:57.761 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-04-21T08:48:57.761 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:48:57.761 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-04-21T08:48:57.761 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-04-21T08:48:57.762 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-04-21T08:48:57.762 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-04-21T08:48:57.762 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-04-21T08:48:57.762 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-04-21T08:48:57.762 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-04-21T08:48:57.762 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-04-21T08:48:57.762 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-21T08:48:57.762 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-21T08:48:57.762 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-04-21T08:48:57.762 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-04-21T08:48:57.762 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-04-21T08:48:57.762 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay 2024-04-21T08:48:57.762 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/fc9e8145942c3a1e33a9faa82d0e7dd0a6da51db9675f943cc6a8ffc36a7a63f/merged 2024-04-21T08:48:57.762 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/6bc0fa31b48937f182be085e4231b9165ee341f7e559319ce6aba9a63dd04555/merged 2024-04-21T08:48:57.762 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/0 2024-04-21T08:48:57.763 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:48:57.763 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/0c93af090e13963d2a68bbfe939a63cbad717f022d8ea9a85538592ea9442e9a/merged 2024-04-21T08:48:57.763 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/e4bf82040817311da57e60dbcf75b66ecfb7a1251c2f16e78fcdb2f4e30648a7/merged 2024-04-21T08:48:57.763 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/4561a8a65d4b9a03925bbbb71250277e0eaac84f727b5632fe5fac6227d860d0/merged 2024-04-21T08:48:57.763 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/d7f09f0a4e431fea850e8b347ffa41405c80e349e86e79c8492bc69aae9c60b2/merged 2024-04-21T08:48:57.763 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/70db4784ef8e1d061852822306212a89d716fe61bceb45f756e9758c4d43dbde/merged 2024-04-21T08:48:57.763 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/4aa33676866ac72bc1d92525a8837d774b6697312475d7ac1650d7cacc1fd18c/merged 2024-04-21T08:48:57.763 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/42f91d25b33941790d0497c4ac0acdb73a170db622231a6de19d042e04883380/merged 2024-04-21T08:48:57.763 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/6255e4d2cb3ec34ef7022b80735146031f3a667e0091afe4912ae575b768eef7/merged 2024-04-21T08:48:57.763 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-04-21T08:48:57.763 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:48:57.763 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:48:57.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:48:57.764 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-04-21T08:48:57.816 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-21T08:48:57.816 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-21T08:48:57.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-04-21T08:48:57.883 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-04-21T08:48:57.939 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-04-21T08:48:57.940 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-04-21T08:48:57.940 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-04-21T08:48:57.940 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-04-21T08:48:57.940 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-04-21T08:48:57.940 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-04-21T08:48:57.940 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-04-21T08:48:57.940 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-04-21T08:48:57.940 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-04-21T08:48:57.940 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-04-21T08:48:57.940 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-04-21T08:48:57.941 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-04-21T08:48:57.941 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:48:57.941 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-04-21T08:48:57.941 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-04-21T08:48:57.941 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-04-21T08:48:57.941 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-04-21T08:48:57.941 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-04-21T08:48:57.941 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-04-21T08:48:57.941 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-04-21T08:48:57.941 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-04-21T08:48:57.941 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-21T08:48:57.941 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-21T08:48:57.941 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-04-21T08:48:57.942 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-04-21T08:48:57.942 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-04-21T08:48:57.942 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay 2024-04-21T08:48:57.942 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/fc9e8145942c3a1e33a9faa82d0e7dd0a6da51db9675f943cc6a8ffc36a7a63f/merged 2024-04-21T08:48:57.942 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/6bc0fa31b48937f182be085e4231b9165ee341f7e559319ce6aba9a63dd04555/merged 2024-04-21T08:48:57.942 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/0 2024-04-21T08:48:57.942 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:48:57.942 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/0c93af090e13963d2a68bbfe939a63cbad717f022d8ea9a85538592ea9442e9a/merged 2024-04-21T08:48:57.942 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/e4bf82040817311da57e60dbcf75b66ecfb7a1251c2f16e78fcdb2f4e30648a7/merged 2024-04-21T08:48:57.942 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/4561a8a65d4b9a03925bbbb71250277e0eaac84f727b5632fe5fac6227d860d0/merged 2024-04-21T08:48:57.942 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/d7f09f0a4e431fea850e8b347ffa41405c80e349e86e79c8492bc69aae9c60b2/merged 2024-04-21T08:48:57.942 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/70db4784ef8e1d061852822306212a89d716fe61bceb45f756e9758c4d43dbde/merged 2024-04-21T08:48:57.943 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/4aa33676866ac72bc1d92525a8837d774b6697312475d7ac1650d7cacc1fd18c/merged 2024-04-21T08:48:57.943 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/42f91d25b33941790d0497c4ac0acdb73a170db622231a6de19d042e04883380/merged 2024-04-21T08:48:57.943 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/6255e4d2cb3ec34ef7022b80735146031f3a667e0091afe4912ae575b768eef7/merged 2024-04-21T08:48:57.943 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-04-21T08:48:57.943 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:48:57.943 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:48:57.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:48:57.943 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-04-21T08:48:57.951 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:2024-04-21T08:48:57.951+0000 7f44cdc40480 -1 init, newargv = 0x55a649d38f90 newargc=15 2024-04-21T08:48:57.951 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[71416]: starting ceph client 2024-04-21T08:48:57.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[71416]: starting fuse 2024-04-21T08:48:57.967 INFO:teuthology.orchestra.run.smithi028.stdout:115 2024-04-21T08:48:57.967 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-21T08:48:57.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-04-21T08:48:57.968 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-04-21T08:48:57.968 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-04-21T08:48:57.968 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-04-21T08:48:57.968 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-04-21T08:48:57.968 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-21T08:48:57.968 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-04-21T08:48:57.968 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-21T08:48:57.968 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-04-21T08:48:57.968 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-21T08:48:57.968 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-21T08:48:57.968 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-04-21T08:48:57.969 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-04-21T08:48:57.969 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-04-21T08:48:57.969 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-04-21T08:48:57.969 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-21T08:48:57.969 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-04-21T08:48:57.969 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-21T08:48:57.969 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-21T08:48:57.969 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-21T08:48:57.969 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-04-21T08:48:57.969 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-04-21T08:48:57.969 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-04-21T08:48:57.969 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-21T08:48:57.970 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-21T08:48:57.970 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.admin")) 2024-04-21T08:48:57.970 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-04-21T08:48:58.086 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.admin.71416.asok 2024-04-21T08:48:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:48:58.096 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.71416.asok 2024-04-21T08:48:58.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:48:58.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.71416.asok status 2024-04-21T08:48:58.233 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:48:58.233 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-04-21T08:48:58.233 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "a630765c1f93c6e61d9b40c4b76437611d5ef29e", 2024-04-21T08:48:58.233 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:48:58.233 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "admin", 2024-04-21T08:48:58.233 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-04-21T08:48:58.233 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-21T08:48:58.233 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "71416", 2024-04-21T08:48:58.234 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-04-21T08:48:58.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T08:48:58.234 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-04-21T08:48:58.234 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-04-21T08:48:58.234 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 14772, 2024-04-21T08:48:58.234 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-04-21T08:48:58.234 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-04-21T08:48:58.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-04-21T08:48:58.234 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 14772 2024-04-21T08:48:58.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T08:48:58.234 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-04-21T08:48:58.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-04-21T08:48:58.234 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-04-21T08:48:58.234 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4068048467 2024-04-21T08:48:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:48:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T08:48:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-04-21T08:48:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-04-21T08:48:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-04-21T08:48:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4068048467 2024-04-21T08:48:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T08:48:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.14772 192.168.0.1:0/4068048467", 2024-04-21T08:48:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.1:0/4068048467", 2024-04-21T08:48:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-04-21T08:48:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 17, 2024-04-21T08:48:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 89, 2024-04-21T08:48:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-04-21T08:48:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-04-21T08:48:58.235 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-04-21T08:48:58.236 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:48:58.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:48:58.242 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-04-21T08:48:58.242 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:48:58.257 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-04-21T08:48:58.257 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:48:58.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:48:58.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:48:58.323 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-21T08:48:58.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:48:58.435 INFO:teuthology.orchestra.run.smithi028.stdout:# file: . 2024-04-21T08:48:58.435 INFO:teuthology.orchestra.run.smithi028.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-21T08:48:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:58.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:48:58.441 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:48:58.441 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:48:58.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:48:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.601+0000 7f0be2ff4640 1 -- 172.21.15.28:0/385266140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bdc102a00 msgr2=0x7f0bdc104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.601+0000 7f0be2ff4640 1 --2- 172.21.15.28:0/385266140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bdc102a00 0x7f0bdc104df0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009ca0 tx=0x7f0bcc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.601+0000 7f0be2ff4640 1 -- 172.21.15.28:0/385266140 shutdown_connections 2024-04-21T08:48:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.601+0000 7f0be2ff4640 1 --2- 172.21.15.28:0/385266140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0bdc107c60 0x7f0bdc10a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.601+0000 7f0be2ff4640 1 --2- 172.21.15.28:0/385266140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bdc105330 0x7f0bdc107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.601+0000 7f0be2ff4640 1 --2- 172.21.15.28:0/385266140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bdc102a00 0x7f0bdc104df0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.601+0000 7f0be2ff4640 1 -- 172.21.15.28:0/385266140 >> 172.21.15.28:0/385266140 conn(0x7f0bdc0fc410 msgr2=0x7f0bdc0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.601+0000 7f0be2ff4640 1 -- 172.21.15.28:0/385266140 shutdown_connections 2024-04-21T08:48:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.601+0000 7f0be2ff4640 1 -- 172.21.15.28:0/385266140 wait complete. 2024-04-21T08:48:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.602+0000 7f0be2ff4640 1 Processor -- start 2024-04-21T08:48:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.602+0000 7f0be2ff4640 1 -- start start 2024-04-21T08:48:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.602+0000 7f0be2ff4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0bdc102a00 0x7f0bdc1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.602+0000 7f0be2ff4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bdc105330 0x7f0bdc1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.602+0000 7f0be2ff4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bdc107c60 0x7f0bdc19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.602+0000 7f0be2ff4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc10dab0 con 0x7f0bdc105330 2024-04-21T08:48:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.602+0000 7f0be2ff4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc10d930 con 0x7f0bdc107c60 2024-04-21T08:48:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.602+0000 7f0be2ff4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc10dc30 con 0x7f0bdc102a00 2024-04-21T08:48:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.602+0000 7f0bdbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bdc105330 0x7f0bdc1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.602+0000 7f0bdbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bdc105330 0x7f0bdc1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44024/0 (socket says 172.21.15.28:44024) 2024-04-21T08:48:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.602+0000 7f0bdbfff640 1 -- 172.21.15.28:0/2769144855 learned_addr learned my addr 172.21.15.28:0/2769144855 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.602+0000 7f0be0d90640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0bdc102a00 0x7f0bdc1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.602+0000 7f0be1591640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bdc107c60 0x7f0bdc19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.603+0000 7f0bdbfff640 1 -- 172.21.15.28:0/2769144855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0bdc102a00 msgr2=0x7f0bdc1981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.603+0000 7f0bdbfff640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0bdc102a00 0x7f0bdc1981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.603+0000 7f0bdbfff640 1 -- 172.21.15.28:0/2769144855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bdc107c60 msgr2=0x7f0bdc19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.603+0000 7f0bdbfff640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bdc107c60 0x7f0bdc19f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.603+0000 7f0bdbfff640 1 -- 172.21.15.28:0/2769144855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bdc19fdd0 con 0x7f0bdc105330 2024-04-21T08:48:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.603+0000 7f0be0d90640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0bdc102a00 0x7f0bdc1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.603+0000 7f0be1591640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bdc107c60 0x7f0bdc19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.603+0000 7f0bdbfff640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bdc105330 0x7f0bdc1986f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400c970 tx=0x7f0bc400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.603+0000 7f0bd9ffb640 1 -- 172.21.15.28:0/2769144855 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4007bb0 con 0x7f0bdc105330 2024-04-21T08:48:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.603+0000 7f0bd9ffb640 1 -- 172.21.15.28:0/2769144855 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bc4007d50 con 0x7f0bdc105330 2024-04-21T08:48:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.603+0000 7f0be2ff4640 1 -- 172.21.15.28:0/2769144855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bdc1a00c0 con 0x7f0bdc105330 2024-04-21T08:48:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.603+0000 7f0bd9ffb640 1 -- 172.21.15.28:0/2769144855 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc400f450 con 0x7f0bdc105330 2024-04-21T08:48:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.603+0000 7f0be2ff4640 1 -- 172.21.15.28:0/2769144855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bdc1a0660 con 0x7f0bdc105330 2024-04-21T08:48:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.604+0000 7f0be2ff4640 1 -- 172.21.15.28:0/2769144855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bdc0fdf00 con 0x7f0bdc105330 2024-04-21T08:48:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.605+0000 7f0bd9ffb640 1 -- 172.21.15.28:0/2769144855 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0bc4004f20 con 0x7f0bdc105330 2024-04-21T08:48:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.606+0000 7f0bd9ffb640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0bc80a3c30 0x7f0bc80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.606+0000 7f0be0d90640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0bc80a3c30 0x7f0bc80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.606+0000 7f0bd9ffb640 1 -- 172.21.15.28:0/2769144855 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0bc40d0690 con 0x7f0bdc105330 2024-04-21T08:48:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.606+0000 7f0be0d90640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0bc80a3c30 0x7f0bc80a60f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc000f80 tx=0x7f0bcc0324c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:58.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.611+0000 7f0bd9ffb640 1 -- 172.21.15.28:0/2769144855 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0bc4014030 con 0x7f0bdc105330 2024-04-21T08:48:58.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.797+0000 7f0be2ff4640 1 -- 172.21.15.28:0/2769144855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0bdc062b50 con 0x7f0bdc105330 2024-04-21T08:48:58.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.799+0000 7f0bd9ffb640 1 -- 172.21.15.28:0/2769144855 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24101 (secure 0 0 0) 0x7f0bc409ce30 con 0x7f0bdc105330 2024-04-21T08:48:58.798 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:48:58.799 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:48:51.121128+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:48:58.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.801+0000 7f0be2ff4640 1 -- 172.21.15.28:0/2769144855 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0bc80a3c30 msgr2=0x7f0bc80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:58.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.801+0000 7f0be2ff4640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0bc80a3c30 0x7f0bc80a60f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc000f80 tx=0x7f0bcc0324c0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.801+0000 7f0be2ff4640 1 -- 172.21.15.28:0/2769144855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bdc105330 msgr2=0x7f0bdc1986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.801+0000 7f0be2ff4640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bdc105330 0x7f0bdc1986f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400c970 tx=0x7f0bc400ce30 comp rx=0 tx=0).stop 2024-04-21T08:48:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.801+0000 7f0be2ff4640 1 -- 172.21.15.28:0/2769144855 shutdown_connections 2024-04-21T08:48:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.801+0000 7f0be2ff4640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bdc107c60 0x7f0bdc19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.801+0000 7f0be2ff4640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0bc80a3c30 0x7f0bc80a60f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.801+0000 7f0be2ff4640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bdc105330 0x7f0bdc1986f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.801+0000 7f0be2ff4640 1 --2- 172.21.15.28:0/2769144855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0bdc102a00 0x7f0bdc1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.801+0000 7f0be2ff4640 1 -- 172.21.15.28:0/2769144855 >> 172.21.15.28:0/2769144855 conn(0x7f0bdc0fc410 msgr2=0x7f0bdc1032c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.801+0000 7f0be2ff4640 1 -- 172.21.15.28:0/2769144855 shutdown_connections 2024-04-21T08:48:58.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.801+0000 7f0be2ff4640 1 -- 172.21.15.28:0/2769144855 wait complete. 2024-04-21T08:48:58.811 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:48:43.543799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:48:44.864564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:48:48.747842+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:48:58.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:48:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.917+0000 7f6e5b081640 1 -- 172.21.15.28:0/912155103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e54105360 msgr2=0x7f6e54107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.917+0000 7f6e5b081640 1 --2- 172.21.15.28:0/912155103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e54105360 0x7f6e54107750 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6e40009900 tx=0x7f6e4002bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.917+0000 7f6e5b081640 1 -- 172.21.15.28:0/912155103 shutdown_connections 2024-04-21T08:48:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.917+0000 7f6e5b081640 1 --2- 172.21.15.28:0/912155103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e54107c90 0x7f6e5410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.917+0000 7f6e5b081640 1 --2- 172.21.15.28:0/912155103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e54105360 0x7f6e54107750 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.917+0000 7f6e5b081640 1 --2- 172.21.15.28:0/912155103 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e54102a30 0x7f6e54104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.917+0000 7f6e5b081640 1 -- 172.21.15.28:0/912155103 >> 172.21.15.28:0/912155103 conn(0x7f6e540fc440 msgr2=0x7f6e540fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.917+0000 7f6e5b081640 1 -- 172.21.15.28:0/912155103 shutdown_connections 2024-04-21T08:48:58.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.917+0000 7f6e5b081640 1 -- 172.21.15.28:0/912155103 wait complete. 2024-04-21T08:48:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.918+0000 7f6e5b081640 1 Processor -- start 2024-04-21T08:48:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.918+0000 7f6e5b081640 1 -- start start 2024-04-21T08:48:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.918+0000 7f6e5b081640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e54102a30 0x7f6e54198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.918+0000 7f6e5b081640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e54105360 0x7f6e54198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.918+0000 7f6e5b081640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e54107c90 0x7f6e5419f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.918+0000 7f6e5b081640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5410daf0 con 0x7f6e54107c90 2024-04-21T08:48:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.918+0000 7f6e5b081640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5410d970 con 0x7f6e54105360 2024-04-21T08:48:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.918+0000 7f6e5b081640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5410dc70 con 0x7f6e54102a30 2024-04-21T08:48:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.918+0000 7f6e5961e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e54107c90 0x7f6e5419f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.918+0000 7f6e5961e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e54107c90 0x7f6e5419f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44038/0 (socket says 172.21.15.28:44038) 2024-04-21T08:48:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.918+0000 7f6e5961e640 1 -- 172.21.15.28:0/2062715580 learned_addr learned my addr 172.21.15.28:0/2062715580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.918+0000 7f6e58e1d640 1 --2- 172.21.15.28:0/2062715580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e54102a30 0x7f6e54198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.918+0000 7f6e53fff640 1 --2- 172.21.15.28:0/2062715580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e54105360 0x7f6e54198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.919+0000 7f6e5961e640 1 -- 172.21.15.28:0/2062715580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e54102a30 msgr2=0x7f6e54198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.919+0000 7f6e5961e640 1 --2- 172.21.15.28:0/2062715580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e54102a30 0x7f6e54198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.919+0000 7f6e5961e640 1 -- 172.21.15.28:0/2062715580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e54105360 msgr2=0x7f6e54198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.919+0000 7f6e5961e640 1 --2- 172.21.15.28:0/2062715580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e54105360 0x7f6e54198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.919+0000 7f6e5961e640 1 -- 172.21.15.28:0/2062715580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5419fe50 con 0x7f6e54107c90 2024-04-21T08:48:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.919+0000 7f6e5961e640 1 --2- 172.21.15.28:0/2062715580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e54107c90 0x7f6e5419f7e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009fd0 tx=0x7f6e4c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.919+0000 7f6e51ffb640 1 -- 172.21.15.28:0/2062715580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c017070 con 0x7f6e54107c90 2024-04-21T08:48:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.919+0000 7f6e51ffb640 1 -- 172.21.15.28:0/2062715580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e4c003e20 con 0x7f6e54107c90 2024-04-21T08:48:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.919+0000 7f6e51ffb640 1 -- 172.21.15.28:0/2062715580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c010460 con 0x7f6e54107c90 2024-04-21T08:48:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.919+0000 7f6e5b081640 1 -- 172.21.15.28:0/2062715580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e541a0140 con 0x7f6e54107c90 2024-04-21T08:48:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.919+0000 7f6e5b081640 1 -- 172.21.15.28:0/2062715580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e541a0610 con 0x7f6e54107c90 2024-04-21T08:48:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.920+0000 7f6e5b081640 1 -- 172.21.15.28:0/2062715580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e540fdf30 con 0x7f6e54107c90 2024-04-21T08:48:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.921+0000 7f6e51ffb640 1 -- 172.21.15.28:0/2062715580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6e4c003970 con 0x7f6e54107c90 2024-04-21T08:48:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.922+0000 7f6e51ffb640 1 --2- 172.21.15.28:0/2062715580 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6e440a3bb0 0x7f6e440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.922+0000 7f6e51ffb640 1 -- 172.21.15.28:0/2062715580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6e4c0d03b0 con 0x7f6e54107c90 2024-04-21T08:48:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.922+0000 7f6e58e1d640 1 --2- 172.21.15.28:0/2062715580 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6e440a3bb0 0x7f6e440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.922+0000 7f6e58e1d640 1 --2- 172.21.15.28:0/2062715580 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6e440a3bb0 0x7f6e440a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6e48009bd0 tx=0x7f6e48005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:58.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:58.926+0000 7f6e51ffb640 1 -- 172.21.15.28:0/2062715580 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6e4c09c4a0 con 0x7f6e54107c90 2024-04-21T08:48:59.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:58 smithi152 ceph-mon[31401]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 530 B/s wr, 1 op/s 2024-04-21T08:48:59.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:58 smithi028 ceph-mon[28003]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 530 B/s wr, 1 op/s 2024-04-21T08:48:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.112+0000 7f6e5b081640 1 -- 172.21.15.28:0/2062715580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6e54062b50 con 0x7f6e54107c90 2024-04-21T08:48:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.113+0000 7f6e51ffb640 1 -- 172.21.15.28:0/2062715580 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f6e4c0a07a0 con 0x7f6e54107c90 2024-04-21T08:48:59.112 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T08:48:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.115+0000 7f6e5b081640 1 -- 172.21.15.28:0/2062715580 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6e440a3bb0 msgr2=0x7f6e440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.115+0000 7f6e5b081640 1 --2- 172.21.15.28:0/2062715580 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6e440a3bb0 0x7f6e440a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6e48009bd0 tx=0x7f6e48005f90 comp rx=0 tx=0).stop 2024-04-21T08:48:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.115+0000 7f6e5b081640 1 -- 172.21.15.28:0/2062715580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e54107c90 msgr2=0x7f6e5419f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.115+0000 7f6e5b081640 1 --2- 172.21.15.28:0/2062715580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e54107c90 0x7f6e5419f7e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009fd0 tx=0x7f6e4c00b260 comp rx=0 tx=0).stop 2024-04-21T08:48:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.115+0000 7f6e5b081640 1 -- 172.21.15.28:0/2062715580 shutdown_connections 2024-04-21T08:48:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.115+0000 7f6e5b081640 1 --2- 172.21.15.28:0/2062715580 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6e440a3bb0 0x7f6e440a6070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.115+0000 7f6e5b081640 1 --2- 172.21.15.28:0/2062715580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e54107c90 0x7f6e5419f7e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.115+0000 7f6e5b081640 1 --2- 172.21.15.28:0/2062715580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e54105360 0x7f6e54198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.115+0000 7f6e5b081640 1 --2- 172.21.15.28:0/2062715580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e54102a30 0x7f6e54198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.115+0000 7f6e5b081640 1 -- 172.21.15.28:0/2062715580 >> 172.21.15.28:0/2062715580 conn(0x7f6e540fc440 msgr2=0x7f6e541032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.115+0000 7f6e5b081640 1 -- 172.21.15.28:0/2062715580 shutdown_connections 2024-04-21T08:48:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.115+0000 7f6e5b081640 1 -- 172.21.15.28:0/2062715580 wait complete. 2024-04-21T08:48:59.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:58 smithi171 ceph-mon[31358]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 530 B/s wr, 1 op/s 2024-04-21T08:48:59.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:48:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.231+0000 7feeb7b00640 1 -- 172.21.15.28:0/463615345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeb0101fe0 msgr2=0x7feeb0102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.231+0000 7feeb7b00640 1 --2- 172.21.15.28:0/463615345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeb0101fe0 0x7feeb0102460 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7feea0009900 tx=0x7feea002bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.231+0000 7feeb7b00640 1 -- 172.21.15.28:0/463615345 shutdown_connections 2024-04-21T08:48:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.231+0000 7feeb7b00640 1 --2- 172.21.15.28:0/463615345 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feeb01029a0 0x7feeb01092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.231+0000 7feeb7b00640 1 --2- 172.21.15.28:0/463615345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeb0101fe0 0x7feeb0102460 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.231+0000 7feeb7b00640 1 --2- 172.21.15.28:0/463615345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feeb0100e90 0x7feeb01012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.231+0000 7feeb7b00640 1 -- 172.21.15.28:0/463615345 >> 172.21.15.28:0/463615345 conn(0x7feeb00fc400 msgr2=0x7feeb00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.231+0000 7feeb7b00640 1 -- 172.21.15.28:0/463615345 shutdown_connections 2024-04-21T08:48:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.231+0000 7feeb7b00640 1 -- 172.21.15.28:0/463615345 wait complete. 2024-04-21T08:48:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.231+0000 7feeb7b00640 1 Processor -- start 2024-04-21T08:48:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.231+0000 7feeb7b00640 1 -- start start 2024-04-21T08:48:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb7b00640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feeb0100e90 0x7feeb0198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb7b00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeb0101fe0 0x7feeb0198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb7b00640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feeb01029a0 0x7feeb019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb7b00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feeb010ca30 con 0x7feeb0101fe0 2024-04-21T08:48:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb7b00640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7feeb010c8b0 con 0x7feeb0100e90 2024-04-21T08:48:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb7b00640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7feeb010cbb0 con 0x7feeb01029a0 2024-04-21T08:48:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb509b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeb0101fe0 0x7feeb0198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb509b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeb0101fe0 0x7feeb0198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44066/0 (socket says 172.21.15.28:44066) 2024-04-21T08:48:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb509b640 1 -- 172.21.15.28:0/762914776 learned_addr learned my addr 172.21.15.28:0/762914776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb589c640 1 --2- 172.21.15.28:0/762914776 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feeb0100e90 0x7feeb0198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb609d640 1 --2- 172.21.15.28:0/762914776 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feeb01029a0 0x7feeb019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb509b640 1 -- 172.21.15.28:0/762914776 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feeb01029a0 msgr2=0x7feeb019f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb509b640 1 --2- 172.21.15.28:0/762914776 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feeb01029a0 0x7feeb019f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb509b640 1 -- 172.21.15.28:0/762914776 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feeb0100e90 msgr2=0x7feeb0198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb509b640 1 --2- 172.21.15.28:0/762914776 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feeb0100e90 0x7feeb0198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb509b640 1 -- 172.21.15.28:0/762914776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb019fe30 con 0x7feeb0101fe0 2024-04-21T08:48:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb609d640 1 --2- 172.21.15.28:0/762914776 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feeb01029a0 0x7feeb019f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb509b640 1 --2- 172.21.15.28:0/762914776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeb0101fe0 0x7feeb0198750 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7feea0005ab0 tx=0x7feea00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feea6ffd640 1 -- 172.21.15.28:0/762914776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0043020 con 0x7feeb0101fe0 2024-04-21T08:48:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feea6ffd640 1 -- 172.21.15.28:0/762914776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feea00054c0 con 0x7feeb0101fe0 2024-04-21T08:48:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb7b00640 1 -- 172.21.15.28:0/762914776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeb01a00c0 con 0x7feeb0101fe0 2024-04-21T08:48:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feea6ffd640 1 -- 172.21.15.28:0/762914776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0003d10 con 0x7feeb0101fe0 2024-04-21T08:48:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.232+0000 7feeb7b00640 1 -- 172.21.15.28:0/762914776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeb01a0660 con 0x7feeb0101fe0 2024-04-21T08:48:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.233+0000 7feeb7b00640 1 -- 172.21.15.28:0/762914776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeb01a0980 con 0x7feeb0101fe0 2024-04-21T08:48:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.234+0000 7feea6ffd640 1 -- 172.21.15.28:0/762914776 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7feea0046020 con 0x7feeb0101fe0 2024-04-21T08:48:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.235+0000 7feea6ffd640 1 --2- 172.21.15.28:0/762914776 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fee9c0a3c30 0x7fee9c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.235+0000 7feeb589c640 1 --2- 172.21.15.28:0/762914776 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fee9c0a3c30 0x7fee9c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.235+0000 7feea6ffd640 1 -- 172.21.15.28:0/762914776 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7feea00f2920 con 0x7feeb0101fe0 2024-04-21T08:48:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.236+0000 7feeb589c640 1 --2- 172.21.15.28:0/762914776 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fee9c0a3c30 0x7fee9c0a60f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7feeac00bbe0 tx=0x7feeac00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.240+0000 7feea6ffd640 1 -- 172.21.15.28:0/762914776 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feea004d020 con 0x7feeb0101fe0 2024-04-21T08:48:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.425+0000 7feeb7b00640 1 -- 172.21.15.28:0/762914776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7feeb01a0b50 con 0x7feeb0101fe0 2024-04-21T08:48:59.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.425+0000 7feea6ffd640 1 -- 172.21.15.28:0/762914776 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7feea00bf0c0 con 0x7feeb0101fe0 2024-04-21T08:48:59.425 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:48:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.427+0000 7feeb7b00640 1 -- 172.21.15.28:0/762914776 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fee9c0a3c30 msgr2=0x7fee9c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.427+0000 7feeb7b00640 1 --2- 172.21.15.28:0/762914776 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fee9c0a3c30 0x7fee9c0a60f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7feeac00bbe0 tx=0x7feeac00b390 comp rx=0 tx=0).stop 2024-04-21T08:48:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.427+0000 7feeb7b00640 1 -- 172.21.15.28:0/762914776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeb0101fe0 msgr2=0x7feeb0198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.427+0000 7feeb7b00640 1 --2- 172.21.15.28:0/762914776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeb0101fe0 0x7feeb0198750 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7feea0005ab0 tx=0x7feea00042f0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.427+0000 7feeb7b00640 1 -- 172.21.15.28:0/762914776 shutdown_connections 2024-04-21T08:48:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.427+0000 7feeb7b00640 1 --2- 172.21.15.28:0/762914776 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feeb01029a0 0x7feeb019f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.427+0000 7feeb7b00640 1 --2- 172.21.15.28:0/762914776 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fee9c0a3c30 0x7fee9c0a60f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.427+0000 7feeb7b00640 1 --2- 172.21.15.28:0/762914776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeb0101fe0 0x7feeb0198750 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.427+0000 7feeb7b00640 1 --2- 172.21.15.28:0/762914776 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feeb0100e90 0x7feeb0198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.427+0000 7feeb7b00640 1 -- 172.21.15.28:0/762914776 >> 172.21.15.28:0/762914776 conn(0x7feeb00fc400 msgr2=0x7feeb00690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.427+0000 7feeb7b00640 1 -- 172.21.15.28:0/762914776 shutdown_connections 2024-04-21T08:48:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.428+0000 7feeb7b00640 1 -- 172.21.15.28:0/762914776 wait complete. 2024-04-21T08:48:59.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:48:59.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.544+0000 7f46cc1f4640 1 -- 172.21.15.28:0/1329163989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46c4101ba0 msgr2=0x7f46c4102020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:59.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.544+0000 7f46cc1f4640 1 --2- 172.21.15.28:0/1329163989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46c4101ba0 0x7f46c4102020 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009ca0 tx=0x7f46bc02c180 comp rx=0 tx=0).stop 2024-04-21T08:48:59.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.544+0000 7f46cc1f4640 1 -- 172.21.15.28:0/1329163989 shutdown_connections 2024-04-21T08:48:59.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.544+0000 7f46cc1f4640 1 --2- 172.21.15.28:0/1329163989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46c4101ba0 0x7f46c4102020 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.544+0000 7f46cc1f4640 1 --2- 172.21.15.28:0/1329163989 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f46c4101200 0x7f46c4101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.544+0000 7f46cc1f4640 1 --2- 172.21.15.28:0/1329163989 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46c41008a0 0x7f46c4100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.544+0000 7f46cc1f4640 1 -- 172.21.15.28:0/1329163989 >> 172.21.15.28:0/1329163989 conn(0x7f46c40fc420 msgr2=0x7f46c40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.544+0000 7f46cc1f4640 1 -- 172.21.15.28:0/1329163989 shutdown_connections 2024-04-21T08:48:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.544+0000 7f46cc1f4640 1 -- 172.21.15.28:0/1329163989 wait complete. 2024-04-21T08:48:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.545+0000 7f46cc1f4640 1 Processor -- start 2024-04-21T08:48:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.545+0000 7f46cc1f4640 1 -- start start 2024-04-21T08:48:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.545+0000 7f46cc1f4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46c41008a0 0x7f46c419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.545+0000 7f46cc1f4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f46c4101200 0x7f46c419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.545+0000 7f46cc1f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46c4101ba0 0x7f46c41a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.545+0000 7f46cc1f4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46c410fcf0 con 0x7f46c4101ba0 2024-04-21T08:48:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.545+0000 7f46cc1f4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f46c410fb70 con 0x7f46c4101200 2024-04-21T08:48:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.545+0000 7f46cc1f4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f46c410fe70 con 0x7f46c41008a0 2024-04-21T08:48:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.545+0000 7f46ca791640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46c4101ba0 0x7f46c41a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.545+0000 7f46ca791640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46c4101ba0 0x7f46c41a18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44078/0 (socket says 172.21.15.28:44078) 2024-04-21T08:48:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.545+0000 7f46ca791640 1 -- 172.21.15.28:0/3352855037 learned_addr learned my addr 172.21.15.28:0/3352855037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.545+0000 7f46c9f90640 1 --2- 172.21.15.28:0/3352855037 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46c41008a0 0x7f46c419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:59.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.545+0000 7f46c978f640 1 --2- 172.21.15.28:0/3352855037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f46c4101200 0x7f46c419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:59.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.546+0000 7f46ca791640 1 -- 172.21.15.28:0/3352855037 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46c41008a0 msgr2=0x7f46c419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:59.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.546+0000 7f46ca791640 1 --2- 172.21.15.28:0/3352855037 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46c41008a0 0x7f46c419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.546+0000 7f46ca791640 1 -- 172.21.15.28:0/3352855037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f46c4101200 msgr2=0x7f46c419a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:59.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.546+0000 7f46ca791640 1 --2- 172.21.15.28:0/3352855037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f46c4101200 0x7f46c419a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.546+0000 7f46ca791640 1 -- 172.21.15.28:0/3352855037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c41a1f10 con 0x7f46c4101ba0 2024-04-21T08:48:59.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.546+0000 7f46ca791640 1 --2- 172.21.15.28:0/3352855037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46c4101ba0 0x7f46c41a18a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f46bc005cc0 tx=0x7f46bc003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:59.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.546+0000 7f46baffd640 1 -- 172.21.15.28:0/3352855037 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc039040 con 0x7f46c4101ba0 2024-04-21T08:48:59.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.546+0000 7f46baffd640 1 -- 172.21.15.28:0/3352855037 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46bc044070 con 0x7f46c4101ba0 2024-04-21T08:48:59.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.546+0000 7f46cc1f4640 1 -- 172.21.15.28:0/3352855037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46c41a21a0 con 0x7f46c4101ba0 2024-04-21T08:48:59.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.546+0000 7f46baffd640 1 -- 172.21.15.28:0/3352855037 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc032880 con 0x7f46c4101ba0 2024-04-21T08:48:59.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.546+0000 7f46cc1f4640 1 -- 172.21.15.28:0/3352855037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46c41a2690 con 0x7f46c4101ba0 2024-04-21T08:48:59.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.547+0000 7f46cc1f4640 1 -- 172.21.15.28:0/3352855037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46c410b790 con 0x7f46c4101ba0 2024-04-21T08:48:59.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.548+0000 7f46baffd640 1 -- 172.21.15.28:0/3352855037 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f46bc004240 con 0x7f46c4101ba0 2024-04-21T08:48:59.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.549+0000 7f46baffd640 1 --2- 172.21.15.28:0/3352855037 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f46b00a3bb0 0x7f46b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:59.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.549+0000 7f46c9f90640 1 --2- 172.21.15.28:0/3352855037 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f46b00a3bb0 0x7f46b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:59.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.549+0000 7f46baffd640 1 -- 172.21.15.28:0/3352855037 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f46bc0f3360 con 0x7f46c4101ba0 2024-04-21T08:48:59.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.549+0000 7f46c9f90640 1 --2- 172.21.15.28:0/3352855037 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f46b00a3bb0 0x7f46b00a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f46c000abe0 tx=0x7f46c000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:59.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.552+0000 7f46baffd640 1 -- 172.21.15.28:0/3352855037 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46bc0bfaa0 con 0x7f46c4101ba0 2024-04-21T08:48:59.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.739+0000 7f46cc1f4640 1 -- 172.21.15.28:0/3352855037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f46c402d5c0 con 0x7f46c4101ba0 2024-04-21T08:48:59.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.739+0000 7f46baffd640 1 -- 172.21.15.28:0/3352855037 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f46bc0bf3a0 con 0x7f46c4101ba0 2024-04-21T08:48:59.739 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-21T08:48:59.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.741+0000 7f46cc1f4640 1 -- 172.21.15.28:0/3352855037 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f46b00a3bb0 msgr2=0x7f46b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:59.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.741+0000 7f46cc1f4640 1 --2- 172.21.15.28:0/3352855037 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f46b00a3bb0 0x7f46b00a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f46c000abe0 tx=0x7f46c000a2d0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.741+0000 7f46cc1f4640 1 -- 172.21.15.28:0/3352855037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46c4101ba0 msgr2=0x7f46c41a18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:59.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.741+0000 7f46cc1f4640 1 --2- 172.21.15.28:0/3352855037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46c4101ba0 0x7f46c41a18a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f46bc005cc0 tx=0x7f46bc003c20 comp rx=0 tx=0).stop 2024-04-21T08:48:59.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.741+0000 7f46cc1f4640 1 -- 172.21.15.28:0/3352855037 shutdown_connections 2024-04-21T08:48:59.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.741+0000 7f46cc1f4640 1 --2- 172.21.15.28:0/3352855037 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f46b00a3bb0 0x7f46b00a6070 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.741+0000 7f46cc1f4640 1 --2- 172.21.15.28:0/3352855037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46c4101ba0 0x7f46c41a18a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.741+0000 7f46cc1f4640 1 --2- 172.21.15.28:0/3352855037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f46c4101200 0x7f46c419a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.741+0000 7f46cc1f4640 1 --2- 172.21.15.28:0/3352855037 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46c41008a0 0x7f46c419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.741+0000 7f46cc1f4640 1 -- 172.21.15.28:0/3352855037 >> 172.21.15.28:0/3352855037 conn(0x7f46c40fc420 msgr2=0x7f46c40fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:59.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.741+0000 7f46cc1f4640 1 -- 172.21.15.28:0/3352855037 shutdown_connections 2024-04-21T08:48:59.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.742+0000 7f46cc1f4640 1 -- 172.21.15.28:0/3352855037 wait complete. 2024-04-21T08:48:59.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:48:59.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.858+0000 7f11720a5640 1 -- 172.21.15.28:0/3785587535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116c1053c0 msgr2=0x7f116c1077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:59.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.858+0000 7f11720a5640 1 --2- 172.21.15.28:0/3785587535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116c1053c0 0x7f116c1077b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1160009900 tx=0x7f116002bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:59.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.858+0000 7f11720a5640 1 -- 172.21.15.28:0/3785587535 shutdown_connections 2024-04-21T08:48:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.858+0000 7f11720a5640 1 --2- 172.21.15.28:0/3785587535 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f116c107cf0 0x7f116c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.858+0000 7f11720a5640 1 --2- 172.21.15.28:0/3785587535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116c1053c0 0x7f116c1077b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.858+0000 7f11720a5640 1 --2- 172.21.15.28:0/3785587535 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f116c102a90 0x7f116c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.858+0000 7f11720a5640 1 -- 172.21.15.28:0/3785587535 >> 172.21.15.28:0/3785587535 conn(0x7f116c0fc440 msgr2=0x7f116c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.858+0000 7f11720a5640 1 -- 172.21.15.28:0/3785587535 shutdown_connections 2024-04-21T08:48:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.859+0000 7f11720a5640 1 -- 172.21.15.28:0/3785587535 wait complete. 2024-04-21T08:48:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.859+0000 7f11720a5640 1 Processor -- start 2024-04-21T08:48:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.859+0000 7f11720a5640 1 -- start start 2024-04-21T08:48:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.859+0000 7f11720a5640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f116c102a90 0x7f116c198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.859+0000 7f11720a5640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f116c1053c0 0x7f116c198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.859+0000 7f11720a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116c107cf0 0x7f116c19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.859+0000 7f11720a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f116c10db40 con 0x7f116c107cf0 2024-04-21T08:48:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.859+0000 7f11720a5640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f116c10d9c0 con 0x7f116c1053c0 2024-04-21T08:48:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.859+0000 7f11720a5640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f116c10dcc0 con 0x7f116c102a90 2024-04-21T08:48:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.859+0000 7f116bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116c107cf0 0x7f116c19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.860+0000 7f116bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116c107cf0 0x7f116c19f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44102/0 (socket says 172.21.15.28:44102) 2024-04-21T08:48:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.860+0000 7f116bfff640 1 -- 172.21.15.28:0/1700676845 learned_addr learned my addr 172.21.15.28:0/1700676845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:48:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.860+0000 7f116b7fe640 1 --2- 172.21.15.28:0/1700676845 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f116c102a90 0x7f116c198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.860+0000 7f116affd640 1 --2- 172.21.15.28:0/1700676845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f116c1053c0 0x7f116c198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.860+0000 7f116bfff640 1 -- 172.21.15.28:0/1700676845 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f116c102a90 msgr2=0x7f116c198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.860+0000 7f116bfff640 1 --2- 172.21.15.28:0/1700676845 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f116c102a90 0x7f116c198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.860+0000 7f116bfff640 1 -- 172.21.15.28:0/1700676845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f116c1053c0 msgr2=0x7f116c198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.860+0000 7f116bfff640 1 --2- 172.21.15.28:0/1700676845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f116c1053c0 0x7f116c198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.860+0000 7f116bfff640 1 -- 172.21.15.28:0/1700676845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f116c19fe70 con 0x7f116c107cf0 2024-04-21T08:48:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.860+0000 7f116bfff640 1 --2- 172.21.15.28:0/1700676845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116c107cf0 0x7f116c19f800 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f116400eaa0 tx=0x7f116400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.860+0000 7f1168ff9640 1 -- 172.21.15.28:0/1700676845 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116400cdb0 con 0x7f116c107cf0 2024-04-21T08:48:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.860+0000 7f1168ff9640 1 -- 172.21.15.28:0/1700676845 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1164005020 con 0x7f116c107cf0 2024-04-21T08:48:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.860+0000 7f11720a5640 1 -- 172.21.15.28:0/1700676845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f116c1a0160 con 0x7f116c107cf0 2024-04-21T08:48:59.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.860+0000 7f1168ff9640 1 -- 172.21.15.28:0/1700676845 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11640116d0 con 0x7f116c107cf0 2024-04-21T08:48:59.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.860+0000 7f11720a5640 1 -- 172.21.15.28:0/1700676845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f116c1a0630 con 0x7f116c107cf0 2024-04-21T08:48:59.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.861+0000 7f11720a5640 1 -- 172.21.15.28:0/1700676845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f116c0fdf30 con 0x7f116c107cf0 2024-04-21T08:48:59.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.862+0000 7f1168ff9640 1 -- 172.21.15.28:0/1700676845 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f11640051d0 con 0x7f116c107cf0 2024-04-21T08:48:59.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.863+0000 7f1168ff9640 1 --2- 172.21.15.28:0/1700676845 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f11540a3bb0 0x7f11540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:59.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.863+0000 7f1168ff9640 1 -- 172.21.15.28:0/1700676845 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f11640d0790 con 0x7f116c107cf0 2024-04-21T08:48:59.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.863+0000 7f116b7fe640 1 --2- 172.21.15.28:0/1700676845 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f11540a3bb0 0x7f11540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:59.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.863+0000 7f116b7fe640 1 --2- 172.21.15.28:0/1700676845 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f11540a3bb0 0x7f11540a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f115c005fd0 tx=0x7f115c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:59.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:48:59.868+0000 7f1168ff9640 1 -- 172.21.15.28:0/1700676845 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1164016030 con 0x7f116c107cf0 2024-04-21T08:49:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.053+0000 7f11720a5640 1 -- 172.21.15.28:0/1700676845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f116c062b50 con 0x7f116c107cf0 2024-04-21T08:49:00.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.053+0000 7f1168ff9640 1 -- 172.21.15.28:0/1700676845 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f116409c8e0 con 0x7f116c107cf0 2024-04-21T08:49:00.053 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:49:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.055+0000 7f11720a5640 1 -- 172.21.15.28:0/1700676845 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f11540a3bb0 msgr2=0x7f11540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.055+0000 7f11720a5640 1 --2- 172.21.15.28:0/1700676845 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f11540a3bb0 0x7f11540a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f115c005fd0 tx=0x7f115c00b390 comp rx=0 tx=0).stop 2024-04-21T08:49:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.055+0000 7f11720a5640 1 -- 172.21.15.28:0/1700676845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116c107cf0 msgr2=0x7f116c19f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.055+0000 7f11720a5640 1 --2- 172.21.15.28:0/1700676845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116c107cf0 0x7f116c19f800 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f116400eaa0 tx=0x7f116400ef60 comp rx=0 tx=0).stop 2024-04-21T08:49:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.055+0000 7f11720a5640 1 -- 172.21.15.28:0/1700676845 shutdown_connections 2024-04-21T08:49:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.055+0000 7f11720a5640 1 --2- 172.21.15.28:0/1700676845 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f11540a3bb0 0x7f11540a6070 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.055+0000 7f11720a5640 1 --2- 172.21.15.28:0/1700676845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f116c107cf0 0x7f116c19f800 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.055+0000 7f11720a5640 1 --2- 172.21.15.28:0/1700676845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f116c1053c0 0x7f116c198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.055+0000 7f11720a5640 1 --2- 172.21.15.28:0/1700676845 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f116c102a90 0x7f116c198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.055+0000 7f11720a5640 1 -- 172.21.15.28:0/1700676845 >> 172.21.15.28:0/1700676845 conn(0x7f116c0fc440 msgr2=0x7f116c103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.055+0000 7f11720a5640 1 -- 172.21.15.28:0/1700676845 shutdown_connections 2024-04-21T08:49:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.055+0000 7f11720a5640 1 -- 172.21.15.28:0/1700676845 wait complete. 2024-04-21T08:49:00.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:59 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2769144855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:00.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:59 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2062715580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:49:00.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:59 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/762914776' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:49:00.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:48:59 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3352855037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:49:00.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T08:49:00.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:59 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2769144855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:00.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:59 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2062715580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:49:00.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:59 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/762914776' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:49:00.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:48:59 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3352855037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:49:00.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:59 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2769144855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:00.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:59 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2062715580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:49:00.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:59 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/762914776' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:49:00.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:48:59 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3352855037' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:49:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.169+0000 7efc52881640 1 -- 172.21.15.28:0/1523833369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc4c107d20 msgr2=0x7efc4c10a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.169+0000 7efc52881640 1 --2- 172.21.15.28:0/1523833369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc4c107d20 0x7efc4c10a110 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efc44009ca0 tx=0x7efc4402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.169+0000 7efc52881640 1 -- 172.21.15.28:0/1523833369 shutdown_connections 2024-04-21T08:49:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.169+0000 7efc52881640 1 --2- 172.21.15.28:0/1523833369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc4c107d20 0x7efc4c10a110 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.169+0000 7efc52881640 1 --2- 172.21.15.28:0/1523833369 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc4c1053f0 0x7efc4c1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.169+0000 7efc52881640 1 --2- 172.21.15.28:0/1523833369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc4c102ac0 0x7efc4c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.169+0000 7efc52881640 1 -- 172.21.15.28:0/1523833369 >> 172.21.15.28:0/1523833369 conn(0x7efc4c0fc3f0 msgr2=0x7efc4c0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.169+0000 7efc52881640 1 -- 172.21.15.28:0/1523833369 shutdown_connections 2024-04-21T08:49:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.169+0000 7efc52881640 1 -- 172.21.15.28:0/1523833369 wait complete. 2024-04-21T08:49:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.170+0000 7efc52881640 1 Processor -- start 2024-04-21T08:49:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.170+0000 7efc52881640 1 -- start start 2024-04-21T08:49:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.170+0000 7efc52881640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc4c102ac0 0x7efc4c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.170+0000 7efc52881640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc4c1053f0 0x7efc4c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.170+0000 7efc52881640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc4c107d20 0x7efc4c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.170+0000 7efc52881640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c10dad0 con 0x7efc4c1053f0 2024-04-21T08:49:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.170+0000 7efc52881640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c10d950 con 0x7efc4c102ac0 2024-04-21T08:49:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.170+0000 7efc52881640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7efc4c10dc50 con 0x7efc4c107d20 2024-04-21T08:49:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc4b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc4c1053f0 0x7efc4c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc4b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc4c1053f0 0x7efc4c198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44118/0 (socket says 172.21.15.28:44118) 2024-04-21T08:49:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc4b7fe640 1 -- 172.21.15.28:0/3050947479 learned_addr learned my addr 172.21.15.28:0/3050947479 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc4bfff640 1 --2- 172.21.15.28:0/3050947479 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc4c102ac0 0x7efc4c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc50e1e640 1 --2- 172.21.15.28:0/3050947479 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc4c107d20 0x7efc4c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc4b7fe640 1 -- 172.21.15.28:0/3050947479 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc4c107d20 msgr2=0x7efc4c19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc4b7fe640 1 --2- 172.21.15.28:0/3050947479 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc4c107d20 0x7efc4c19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc4b7fe640 1 -- 172.21.15.28:0/3050947479 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc4c102ac0 msgr2=0x7efc4c198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc4b7fe640 1 --2- 172.21.15.28:0/3050947479 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc4c102ac0 0x7efc4c198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc4b7fe640 1 -- 172.21.15.28:0/3050947479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc4c19fe20 con 0x7efc4c1053f0 2024-04-21T08:49:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc4b7fe640 1 --2- 172.21.15.28:0/3050947479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc4c1053f0 0x7efc4c198740 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efc4000e9f0 tx=0x7efc4000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc497fa640 1 -- 172.21.15.28:0/3050947479 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4000ccc0 con 0x7efc4c1053f0 2024-04-21T08:49:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc52881640 1 -- 172.21.15.28:0/3050947479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc4c1a0110 con 0x7efc4c1053f0 2024-04-21T08:49:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc497fa640 1 -- 172.21.15.28:0/3050947479 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc400054d0 con 0x7efc4c1053f0 2024-04-21T08:49:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc497fa640 1 -- 172.21.15.28:0/3050947479 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc40010430 con 0x7efc4c1053f0 2024-04-21T08:49:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.171+0000 7efc52881640 1 -- 172.21.15.28:0/3050947479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc4c1a0660 con 0x7efc4c1053f0 2024-04-21T08:49:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.172+0000 7efc52881640 1 -- 172.21.15.28:0/3050947479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc4c0fdee0 con 0x7efc4c1053f0 2024-04-21T08:49:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.173+0000 7efc497fa640 1 -- 172.21.15.28:0/3050947479 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efc400046c0 con 0x7efc4c1053f0 2024-04-21T08:49:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.174+0000 7efc497fa640 1 --2- 172.21.15.28:0/3050947479 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efc340a3bb0 0x7efc340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.174+0000 7efc4bfff640 1 --2- 172.21.15.28:0/3050947479 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efc340a3bb0 0x7efc340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.174+0000 7efc497fa640 1 -- 172.21.15.28:0/3050947479 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7efc400d0a90 con 0x7efc4c1053f0 2024-04-21T08:49:00.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.174+0000 7efc4bfff640 1 --2- 172.21.15.28:0/3050947479 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efc340a3bb0 0x7efc340a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efc3c009bd0 tx=0x7efc3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.179+0000 7efc497fa640 1 -- 172.21.15.28:0/3050947479 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efc4009d1d0 con 0x7efc4c1053f0 2024-04-21T08:49:00.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.377+0000 7efc52881640 1 -- 172.21.15.28:0/3050947479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efc4c062b50 con 0x7efc4c1053f0 2024-04-21T08:49:00.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.378+0000 7efc497fa640 1 -- 172.21.15.28:0/3050947479 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+587 (secure 0 0 0) 0x7efc4009cad0 con 0x7efc4c1053f0 2024-04-21T08:49:00.377 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3741297321 2024-04-22T08:41:35.374122+0000 2024-04-21T08:49:00.378 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/334321812 2024-04-22T08:41:35.374122+0000 2024-04-21T08:49:00.378 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2874636306 2024-04-22T08:41:07.583432+0000 2024-04-21T08:49:00.378 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/334321812 2024-04-22T08:41:35.374122+0000 2024-04-21T08:49:00.378 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1716427535 2024-04-22T08:41:07.583432+0000 2024-04-21T08:49:00.378 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/731326464 2024-04-22T08:41:35.374122+0000 2024-04-21T08:49:00.378 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2874636306 2024-04-22T08:41:07.583432+0000 2024-04-21T08:49:00.378 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/326837648 2024-04-22T08:41:35.374122+0000 2024-04-21T08:49:00.378 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1980013253 2024-04-22T08:41:07.583432+0000 2024-04-21T08:49:00.378 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/917824119 2024-04-22T08:41:07.583432+0000 2024-04-21T08:49:00.378 INFO:teuthology.orchestra.run.smithi028.stderr:listed 10 entries 2024-04-21T08:49:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.380+0000 7efc52881640 1 -- 172.21.15.28:0/3050947479 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efc340a3bb0 msgr2=0x7efc340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.380+0000 7efc52881640 1 --2- 172.21.15.28:0/3050947479 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efc340a3bb0 0x7efc340a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efc3c009bd0 tx=0x7efc3c005f90 comp rx=0 tx=0).stop 2024-04-21T08:49:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.380+0000 7efc52881640 1 -- 172.21.15.28:0/3050947479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc4c1053f0 msgr2=0x7efc4c198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.380+0000 7efc52881640 1 --2- 172.21.15.28:0/3050947479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc4c1053f0 0x7efc4c198740 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efc4000e9f0 tx=0x7efc4000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.380+0000 7efc52881640 1 -- 172.21.15.28:0/3050947479 shutdown_connections 2024-04-21T08:49:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.380+0000 7efc52881640 1 --2- 172.21.15.28:0/3050947479 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc4c107d20 0x7efc4c19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.380+0000 7efc52881640 1 --2- 172.21.15.28:0/3050947479 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efc340a3bb0 0x7efc340a6070 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.380+0000 7efc52881640 1 --2- 172.21.15.28:0/3050947479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc4c1053f0 0x7efc4c198740 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.380+0000 7efc52881640 1 --2- 172.21.15.28:0/3050947479 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efc4c102ac0 0x7efc4c198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.380+0000 7efc52881640 1 -- 172.21.15.28:0/3050947479 >> 172.21.15.28:0/3050947479 conn(0x7efc4c0fc3f0 msgr2=0x7efc4c1035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.380+0000 7efc52881640 1 -- 172.21.15.28:0/3050947479 shutdown_connections 2024-04-21T08:49:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.380+0000 7efc52881640 1 -- 172.21.15.28:0/3050947479 wait complete. 2024-04-21T08:49:00.390 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:49:00.390 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:49:00.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:49:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.549+0000 7f7d07b8e640 1 -- 172.21.15.28:0/1606212737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d001029f0 msgr2=0x7f7d00104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.549+0000 7f7d07b8e640 1 --2- 172.21.15.28:0/1606212737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d001029f0 0x7f7d00104de0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009ca0 tx=0x7f7cf802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d07b8e640 1 -- 172.21.15.28:0/1606212737 shutdown_connections 2024-04-21T08:49:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d07b8e640 1 --2- 172.21.15.28:0/1606212737 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d00107c50 0x7f7d0010a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d07b8e640 1 --2- 172.21.15.28:0/1606212737 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7d00105320 0x7f7d00107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d07b8e640 1 --2- 172.21.15.28:0/1606212737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d001029f0 0x7f7d00104de0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d07b8e640 1 -- 172.21.15.28:0/1606212737 >> 172.21.15.28:0/1606212737 conn(0x7f7d000fc3e0 msgr2=0x7f7d000fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d07b8e640 1 -- 172.21.15.28:0/1606212737 shutdown_connections 2024-04-21T08:49:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d07b8e640 1 -- 172.21.15.28:0/1606212737 wait complete. 2024-04-21T08:49:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d07b8e640 1 Processor -- start 2024-04-21T08:49:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d07b8e640 1 -- start start 2024-04-21T08:49:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d07b8e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d001029f0 0x7f7d00102360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d07b8e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7d00105320 0x7f7d001009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d07b8e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d00107c50 0x7f7d00100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d07b8e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0010dab0 con 0x7f7d001029f0 2024-04-21T08:49:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d07b8e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0010d930 con 0x7f7d00107c50 2024-04-21T08:49:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d07b8e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0010dc30 con 0x7f7d00105320 2024-04-21T08:49:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.550+0000 7f7d0592a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d001029f0 0x7f7d00102360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.551+0000 7f7d0592a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d001029f0 0x7f7d00102360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44142/0 (socket says 172.21.15.28:44142) 2024-04-21T08:49:00.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.551+0000 7f7d0592a640 1 -- 172.21.15.28:0/2932204050 learned_addr learned my addr 172.21.15.28:0/2932204050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.551+0000 7f7d0592a640 1 -- 172.21.15.28:0/2932204050 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7d00105320 msgr2=0x7f7d001009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.551+0000 7f7d0592a640 1 --2- 172.21.15.28:0/2932204050 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7d00105320 0x7f7d001009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.551+0000 7f7d0592a640 1 -- 172.21.15.28:0/2932204050 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d00107c50 msgr2=0x7f7d00100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.551+0000 7f7d0612b640 1 --2- 172.21.15.28:0/2932204050 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d00107c50 0x7f7d00100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.551+0000 7f7d0592a640 1 --2- 172.21.15.28:0/2932204050 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d00107c50 0x7f7d00100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.551+0000 7f7d0592a640 1 -- 172.21.15.28:0/2932204050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d00101770 con 0x7f7d001029f0 2024-04-21T08:49:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.551+0000 7f7d0592a640 1 --2- 172.21.15.28:0/2932204050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d001029f0 0x7f7d00102360 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8005e30 tx=0x7f7cf8034830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.551+0000 7f7cfeffd640 1 -- 172.21.15.28:0/2932204050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8039040 con 0x7f7d001029f0 2024-04-21T08:49:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.551+0000 7f7cfeffd640 1 -- 172.21.15.28:0/2932204050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7cf8035c30 con 0x7f7d001029f0 2024-04-21T08:49:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.551+0000 7f7d07b8e640 1 -- 172.21.15.28:0/2932204050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d001a2280 con 0x7f7d001029f0 2024-04-21T08:49:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.551+0000 7f7cfeffd640 1 -- 172.21.15.28:0/2932204050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf80042a0 con 0x7f7d001029f0 2024-04-21T08:49:00.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.551+0000 7f7d07b8e640 1 -- 172.21.15.28:0/2932204050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d001a2820 con 0x7f7d001029f0 2024-04-21T08:49:00.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.552+0000 7f7d07b8e640 1 -- 172.21.15.28:0/2932204050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d000fded0 con 0x7f7d001029f0 2024-04-21T08:49:00.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.553+0000 7f7cfeffd640 1 -- 172.21.15.28:0/2932204050 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7cf8046030 con 0x7f7d001029f0 2024-04-21T08:49:00.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.554+0000 7f7cfeffd640 1 --2- 172.21.15.28:0/2932204050 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7ce80a3c30 0x7f7ce80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:00.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.554+0000 7f7d05129640 1 --2- 172.21.15.28:0/2932204050 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7ce80a3c30 0x7f7ce80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:00.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.554+0000 7f7cfeffd640 1 -- 172.21.15.28:0/2932204050 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7cf80f2620 con 0x7f7d001029f0 2024-04-21T08:49:00.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.555+0000 7f7d05129640 1 --2- 172.21.15.28:0/2932204050 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7ce80a3c30 0x7f7ce80a60f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4005fd0 tx=0x7f7cf4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:00.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.558+0000 7f7cfeffd640 1 -- 172.21.15.28:0/2932204050 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7cf804d060 con 0x7f7d001029f0 2024-04-21T08:49:00.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.746+0000 7f7d07b8e640 1 -- 172.21.15.28:0/2932204050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d0002d790 con 0x7f7d001029f0 2024-04-21T08:49:00.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.747+0000 7f7cfeffd640 1 -- 172.21.15.28:0/2932204050 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24101 (secure 0 0 0) 0x7f7cf80bedf0 con 0x7f7d001029f0 2024-04-21T08:49:00.747 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:49:00.747 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:48:51.121128+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:49:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.749+0000 7f7d07b8e640 1 -- 172.21.15.28:0/2932204050 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7ce80a3c30 msgr2=0x7f7ce80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.749+0000 7f7d07b8e640 1 --2- 172.21.15.28:0/2932204050 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7ce80a3c30 0x7f7ce80a60f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4005fd0 tx=0x7f7cf4005850 comp rx=0 tx=0).stop 2024-04-21T08:49:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.749+0000 7f7d07b8e640 1 -- 172.21.15.28:0/2932204050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d001029f0 msgr2=0x7f7d00102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.749+0000 7f7d07b8e640 1 --2- 172.21.15.28:0/2932204050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d001029f0 0x7f7d00102360 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8005e30 tx=0x7f7cf8034830 comp rx=0 tx=0).stop 2024-04-21T08:49:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.749+0000 7f7d07b8e640 1 -- 172.21.15.28:0/2932204050 shutdown_connections 2024-04-21T08:49:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.749+0000 7f7d07b8e640 1 --2- 172.21.15.28:0/2932204050 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7d00107c50 0x7f7d00100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.749+0000 7f7d07b8e640 1 --2- 172.21.15.28:0/2932204050 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7d00105320 0x7f7d001009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.749+0000 7f7d07b8e640 1 --2- 172.21.15.28:0/2932204050 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7ce80a3c30 0x7f7ce80a60f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.749+0000 7f7d07b8e640 1 --2- 172.21.15.28:0/2932204050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d001029f0 0x7f7d00102360 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.749+0000 7f7d07b8e640 1 -- 172.21.15.28:0/2932204050 >> 172.21.15.28:0/2932204050 conn(0x7f7d000fc3e0 msgr2=0x7f7d001034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.749+0000 7f7d07b8e640 1 -- 172.21.15.28:0/2932204050 shutdown_connections 2024-04-21T08:49:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.749+0000 7f7d07b8e640 1 -- 172.21.15.28:0/2932204050 wait complete. 2024-04-21T08:49:00.760 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:48:43.543799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:48:44.864564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:48:48.747842+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:49:00.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:49:00.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.866+0000 7f923a63f640 1 -- 172.21.15.28:0/2542039305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9234101fa0 msgr2=0x7f92341023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.866+0000 7f923a63f640 1 --2- 172.21.15.28:0/2542039305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9234101fa0 0x7f92341023c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9224009900 tx=0x7f922402bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.866+0000 7f923a63f640 1 -- 172.21.15.28:0/2542039305 shutdown_connections 2024-04-21T08:49:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.866+0000 7f923a63f640 1 --2- 172.21.15.28:0/2542039305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9234103b50 0x7f923410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.866+0000 7f923a63f640 1 --2- 172.21.15.28:0/2542039305 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9234103190 0x7f9234103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.866+0000 7f923a63f640 1 --2- 172.21.15.28:0/2542039305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9234101fa0 0x7f92341023c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.866+0000 7f923a63f640 1 -- 172.21.15.28:0/2542039305 >> 172.21.15.28:0/2542039305 conn(0x7f92340fd530 msgr2=0x7f92340ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.866+0000 7f923a63f640 1 -- 172.21.15.28:0/2542039305 shutdown_connections 2024-04-21T08:49:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.866+0000 7f923a63f640 1 -- 172.21.15.28:0/2542039305 wait complete. 2024-04-21T08:49:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f923a63f640 1 Processor -- start 2024-04-21T08:49:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f923a63f640 1 -- start start 2024-04-21T08:49:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f923a63f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9234101fa0 0x7f92341981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f923a63f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9234103190 0x7f9234198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f923a63f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9234103b50 0x7f923419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f923a63f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f923410daa0 con 0x7f9234101fa0 2024-04-21T08:49:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f923a63f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f923410d920 con 0x7f9234103190 2024-04-21T08:49:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f923a63f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f923410dc20 con 0x7f9234103b50 2024-04-21T08:49:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f9233fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9234101fa0 0x7f92341981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f9233fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9234101fa0 0x7f92341981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44148/0 (socket says 172.21.15.28:44148) 2024-04-21T08:49:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f9233fff640 1 -- 172.21.15.28:0/3812438024 learned_addr learned my addr 172.21.15.28:0/3812438024 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f92337fe640 1 --2- 172.21.15.28:0/3812438024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9234103190 0x7f9234198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f9238bdc640 1 --2- 172.21.15.28:0/3812438024 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9234103b50 0x7f923419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f9233fff640 1 -- 172.21.15.28:0/3812438024 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9234103b50 msgr2=0x7f923419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f9233fff640 1 --2- 172.21.15.28:0/3812438024 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9234103b50 0x7f923419f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f9233fff640 1 -- 172.21.15.28:0/3812438024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9234103190 msgr2=0x7f9234198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f9233fff640 1 --2- 172.21.15.28:0/3812438024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9234103190 0x7f9234198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.867+0000 7f9233fff640 1 -- 172.21.15.28:0/3812438024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f923419fe10 con 0x7f9234101fa0 2024-04-21T08:49:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.868+0000 7f9233fff640 1 --2- 172.21.15.28:0/3812438024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9234101fa0 0x7f92341981f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9224000c00 tx=0x7f9224004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.868+0000 7f92317fa640 1 -- 172.21.15.28:0/3812438024 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9224043020 con 0x7f9234101fa0 2024-04-21T08:49:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.868+0000 7f92317fa640 1 -- 172.21.15.28:0/3812438024 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9224004aa0 con 0x7f9234101fa0 2024-04-21T08:49:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.868+0000 7f92317fa640 1 -- 172.21.15.28:0/3812438024 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9224003bf0 con 0x7f9234101fa0 2024-04-21T08:49:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.868+0000 7f923a63f640 1 -- 172.21.15.28:0/3812438024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92341a00a0 con 0x7f9234101fa0 2024-04-21T08:49:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.868+0000 7f923a63f640 1 -- 172.21.15.28:0/3812438024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92341a05f0 con 0x7f9234101fa0 2024-04-21T08:49:00.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.869+0000 7f923a63f640 1 -- 172.21.15.28:0/3812438024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9234109360 con 0x7f9234101fa0 2024-04-21T08:49:00.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.870+0000 7f92317fa640 1 -- 172.21.15.28:0/3812438024 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9224046020 con 0x7f9234101fa0 2024-04-21T08:49:00.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.870+0000 7f92317fa640 1 --2- 172.21.15.28:0/3812438024 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f921c0a3bb0 0x7f921c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:00.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.871+0000 7f92317fa640 1 -- 172.21.15.28:0/3812438024 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f92240f2bd0 con 0x7f9234101fa0 2024-04-21T08:49:00.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.871+0000 7f92337fe640 1 --2- 172.21.15.28:0/3812438024 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f921c0a3bb0 0x7f921c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:00.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.871+0000 7f92337fe640 1 --2- 172.21.15.28:0/3812438024 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f921c0a3bb0 0x7f921c0a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9234199700 tx=0x7f922800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:00.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:00.875+0000 7f92317fa640 1 -- 172.21.15.28:0/3812438024 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f922404d020 con 0x7f9234101fa0 2024-04-21T08:49:01.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.061+0000 7f923a63f640 1 -- 172.21.15.28:0/3812438024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9234062b50 con 0x7f9234101fa0 2024-04-21T08:49:01.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.061+0000 7f92317fa640 1 -- 172.21.15.28:0/3812438024 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f92240bf310 con 0x7f9234101fa0 2024-04-21T08:49:01.061 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T08:49:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.063+0000 7f923a63f640 1 -- 172.21.15.28:0/3812438024 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f921c0a3bb0 msgr2=0x7f921c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.063+0000 7f923a63f640 1 --2- 172.21.15.28:0/3812438024 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f921c0a3bb0 0x7f921c0a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9234199700 tx=0x7f922800a2d0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.063+0000 7f923a63f640 1 -- 172.21.15.28:0/3812438024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9234101fa0 msgr2=0x7f92341981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.063+0000 7f923a63f640 1 --2- 172.21.15.28:0/3812438024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9234101fa0 0x7f92341981f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9224000c00 tx=0x7f9224004270 comp rx=0 tx=0).stop 2024-04-21T08:49:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.063+0000 7f923a63f640 1 -- 172.21.15.28:0/3812438024 shutdown_connections 2024-04-21T08:49:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.063+0000 7f923a63f640 1 --2- 172.21.15.28:0/3812438024 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9234103b50 0x7f923419f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.063+0000 7f923a63f640 1 --2- 172.21.15.28:0/3812438024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9234103190 0x7f9234198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.063+0000 7f923a63f640 1 --2- 172.21.15.28:0/3812438024 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f921c0a3bb0 0x7f921c0a6070 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.063+0000 7f923a63f640 1 --2- 172.21.15.28:0/3812438024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9234101fa0 0x7f92341981f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.063+0000 7f923a63f640 1 -- 172.21.15.28:0/3812438024 >> 172.21.15.28:0/3812438024 conn(0x7f92340fd530 msgr2=0x7f9234106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.063+0000 7f923a63f640 1 -- 172.21.15.28:0/3812438024 shutdown_connections 2024-04-21T08:49:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.063+0000 7f923a63f640 1 -- 172.21.15.28:0/3812438024 wait complete. 2024-04-21T08:49:01.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:00 smithi028 ceph-mon[28003]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 352 B/s wr, 0 op/s 2024-04-21T08:49:01.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:00 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1700676845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:49:01.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:00 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3050947479' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:49:01.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:00 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2932204050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:01.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:49:01.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:00 smithi152 ceph-mon[31401]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 352 B/s wr, 0 op/s 2024-04-21T08:49:01.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:00 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1700676845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:49:01.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:00 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3050947479' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:49:01.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:00 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2932204050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:01.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:00 smithi171 ceph-mon[31358]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 352 B/s wr, 0 op/s 2024-04-21T08:49:01.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:00 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1700676845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:49:01.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:00 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3050947479' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:49:01.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:00 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2932204050' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:01.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.179+0000 7f6af98d9640 1 -- 172.21.15.28:0/1913527614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af4107c90 msgr2=0x7f6af410a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:01.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.179+0000 7f6af98d9640 1 --2- 172.21.15.28:0/1913527614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af4107c90 0x7f6af410a0e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6aec009ca0 tx=0x7f6aec02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.179+0000 7f6af98d9640 1 -- 172.21.15.28:0/1913527614 shutdown_connections 2024-04-21T08:49:01.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.179+0000 7f6af98d9640 1 --2- 172.21.15.28:0/1913527614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af4107c90 0x7f6af410a0e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.179+0000 7f6af98d9640 1 --2- 172.21.15.28:0/1913527614 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6af4105360 0x7f6af4107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.179+0000 7f6af98d9640 1 --2- 172.21.15.28:0/1913527614 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6af4102a30 0x7f6af4104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.179+0000 7f6af98d9640 1 -- 172.21.15.28:0/1913527614 >> 172.21.15.28:0/1913527614 conn(0x7f6af40fc440 msgr2=0x7f6af40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:01.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.179+0000 7f6af98d9640 1 -- 172.21.15.28:0/1913527614 shutdown_connections 2024-04-21T08:49:01.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.179+0000 7f6af98d9640 1 -- 172.21.15.28:0/1913527614 wait complete. 2024-04-21T08:49:01.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af98d9640 1 Processor -- start 2024-04-21T08:49:01.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af98d9640 1 -- start start 2024-04-21T08:49:01.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af98d9640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6af4102a30 0x7f6af41981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:01.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af98d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af4105360 0x7f6af4198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:01.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af98d9640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6af4107c90 0x7f6af419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:01.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af98d9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6af410daf0 con 0x7f6af4105360 2024-04-21T08:49:01.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af98d9640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6af410d970 con 0x7f6af4107c90 2024-04-21T08:49:01.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af98d9640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6af410dc70 con 0x7f6af4102a30 2024-04-21T08:49:01.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af4105360 0x7f6af4198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:01.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af4105360 0x7f6af4198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44168/0 (socket says 172.21.15.28:44168) 2024-04-21T08:49:01.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af27fc640 1 -- 172.21.15.28:0/1561411016 learned_addr learned my addr 172.21.15.28:0/1561411016 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:01.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af37fe640 1 --2- 172.21.15.28:0/1561411016 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6af4107c90 0x7f6af419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:01.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af2ffd640 1 --2- 172.21.15.28:0/1561411016 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6af4102a30 0x7f6af41981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:01.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af27fc640 1 -- 172.21.15.28:0/1561411016 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6af4102a30 msgr2=0x7f6af41981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:01.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af27fc640 1 --2- 172.21.15.28:0/1561411016 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6af4102a30 0x7f6af41981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af27fc640 1 -- 172.21.15.28:0/1561411016 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6af4107c90 msgr2=0x7f6af419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:01.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af27fc640 1 --2- 172.21.15.28:0/1561411016 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6af4107c90 0x7f6af419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.180+0000 7f6af27fc640 1 -- 172.21.15.28:0/1561411016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af419fdf0 con 0x7f6af4105360 2024-04-21T08:49:01.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.181+0000 7f6af27fc640 1 --2- 172.21.15.28:0/1561411016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af4105360 0x7f6af4198710 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6ae400d770 tx=0x7f6ae400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:01.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.181+0000 7f6af88d7640 1 -- 172.21.15.28:0/1561411016 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae4017070 con 0x7f6af4105360 2024-04-21T08:49:01.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.181+0000 7f6af88d7640 1 -- 172.21.15.28:0/1561411016 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ae4009c30 con 0x7f6af4105360 2024-04-21T08:49:01.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.181+0000 7f6af88d7640 1 -- 172.21.15.28:0/1561411016 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae400f620 con 0x7f6af4105360 2024-04-21T08:49:01.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.181+0000 7f6af98d9640 1 -- 172.21.15.28:0/1561411016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6af41a00e0 con 0x7f6af4105360 2024-04-21T08:49:01.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.181+0000 7f6af98d9640 1 -- 172.21.15.28:0/1561411016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6af41a05e0 con 0x7f6af4105360 2024-04-21T08:49:01.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.182+0000 7f6af98d9640 1 -- 172.21.15.28:0/1561411016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6af40fdf30 con 0x7f6af4105360 2024-04-21T08:49:01.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.183+0000 7f6af88d7640 1 -- 172.21.15.28:0/1561411016 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6ae401a070 con 0x7f6af4105360 2024-04-21T08:49:01.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.184+0000 7f6af88d7640 1 --2- 172.21.15.28:0/1561411016 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6adc0a3bb0 0x7f6adc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:01.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.184+0000 7f6af2ffd640 1 --2- 172.21.15.28:0/1561411016 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6adc0a3bb0 0x7f6adc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:01.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.184+0000 7f6af88d7640 1 -- 172.21.15.28:0/1561411016 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6ae40d01a0 con 0x7f6af4105360 2024-04-21T08:49:01.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.184+0000 7f6af2ffd640 1 --2- 172.21.15.28:0/1561411016 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6adc0a3bb0 0x7f6adc0a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8005fd0 tx=0x7f6ae8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:01.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.187+0000 7f6af88d7640 1 -- 172.21.15.28:0/1561411016 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ae4009de0 con 0x7f6af4105360 2024-04-21T08:49:01.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.375+0000 7f6af98d9640 1 -- 172.21.15.28:0/1561411016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6af402d810 con 0x7f6af4105360 2024-04-21T08:49:01.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.375+0000 7f6af88d7640 1 -- 172.21.15.28:0/1561411016 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f6ae409c8e0 con 0x7f6af4105360 2024-04-21T08:49:01.375 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:49:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.377+0000 7f6af98d9640 1 -- 172.21.15.28:0/1561411016 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6adc0a3bb0 msgr2=0x7f6adc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.377+0000 7f6af98d9640 1 --2- 172.21.15.28:0/1561411016 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6adc0a3bb0 0x7f6adc0a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8005fd0 tx=0x7f6ae8005850 comp rx=0 tx=0).stop 2024-04-21T08:49:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.377+0000 7f6af98d9640 1 -- 172.21.15.28:0/1561411016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af4105360 msgr2=0x7f6af4198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.377+0000 7f6af98d9640 1 --2- 172.21.15.28:0/1561411016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af4105360 0x7f6af4198710 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6ae400d770 tx=0x7f6ae400dc30 comp rx=0 tx=0).stop 2024-04-21T08:49:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.377+0000 7f6af98d9640 1 -- 172.21.15.28:0/1561411016 shutdown_connections 2024-04-21T08:49:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.377+0000 7f6af98d9640 1 --2- 172.21.15.28:0/1561411016 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6af4107c90 0x7f6af419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.377+0000 7f6af98d9640 1 --2- 172.21.15.28:0/1561411016 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6adc0a3bb0 0x7f6adc0a6070 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.377+0000 7f6af98d9640 1 --2- 172.21.15.28:0/1561411016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af4105360 0x7f6af4198710 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.377+0000 7f6af98d9640 1 --2- 172.21.15.28:0/1561411016 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6af4102a30 0x7f6af41981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.377+0000 7f6af98d9640 1 -- 172.21.15.28:0/1561411016 >> 172.21.15.28:0/1561411016 conn(0x7f6af40fc440 msgr2=0x7f6af4103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.377+0000 7f6af98d9640 1 -- 172.21.15.28:0/1561411016 shutdown_connections 2024-04-21T08:49:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.377+0000 7f6af98d9640 1 -- 172.21.15.28:0/1561411016 wait complete. 2024-04-21T08:49:01.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:49:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.493+0000 7f5650e9a640 1 -- 172.21.15.28:0/302571895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f564c101fa0 msgr2=0x7f564c1023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.493+0000 7f5650e9a640 1 --2- 172.21.15.28:0/302571895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f564c101fa0 0x7f564c1023c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f563c009900 tx=0x7f563c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.493+0000 7f5650e9a640 1 -- 172.21.15.28:0/302571895 shutdown_connections 2024-04-21T08:49:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.493+0000 7f5650e9a640 1 --2- 172.21.15.28:0/302571895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f564c103b50 0x7f564c10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.493+0000 7f5650e9a640 1 --2- 172.21.15.28:0/302571895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f564c103190 0x7f564c103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.493+0000 7f5650e9a640 1 --2- 172.21.15.28:0/302571895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f564c101fa0 0x7f564c1023c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.493+0000 7f5650e9a640 1 -- 172.21.15.28:0/302571895 >> 172.21.15.28:0/302571895 conn(0x7f564c0fd530 msgr2=0x7f564c0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.493+0000 7f5650e9a640 1 -- 172.21.15.28:0/302571895 shutdown_connections 2024-04-21T08:49:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.493+0000 7f5650e9a640 1 -- 172.21.15.28:0/302571895 wait complete. 2024-04-21T08:49:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.494+0000 7f5650e9a640 1 Processor -- start 2024-04-21T08:49:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.494+0000 7f5650e9a640 1 -- start start 2024-04-21T08:49:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.494+0000 7f5650e9a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f564c101fa0 0x7f564c198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.494+0000 7f5650e9a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f564c103190 0x7f564c198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.494+0000 7f5650e9a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f564c103b50 0x7f564c19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.494+0000 7f5650e9a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f564c10dae0 con 0x7f564c103b50 2024-04-21T08:49:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.494+0000 7f5650e9a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f564c10d960 con 0x7f564c103190 2024-04-21T08:49:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.494+0000 7f5650e9a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f564c10dc60 con 0x7f564c101fa0 2024-04-21T08:49:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.494+0000 7f564ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f564c103b50 0x7f564c19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.494+0000 7f564ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f564c103b50 0x7f564c19f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44178/0 (socket says 172.21.15.28:44178) 2024-04-21T08:49:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.494+0000 7f564ad9d640 1 -- 172.21.15.28:0/924728274 learned_addr learned my addr 172.21.15.28:0/924728274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.494+0000 7f564a59c640 1 --2- 172.21.15.28:0/924728274 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f564c101fa0 0x7f564c198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.495+0000 7f5641d9b640 1 --2- 172.21.15.28:0/924728274 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f564c103190 0x7f564c198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.495+0000 7f564ad9d640 1 -- 172.21.15.28:0/924728274 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f564c101fa0 msgr2=0x7f564c198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.495+0000 7f564ad9d640 1 --2- 172.21.15.28:0/924728274 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f564c101fa0 0x7f564c198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.495+0000 7f564ad9d640 1 -- 172.21.15.28:0/924728274 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f564c103190 msgr2=0x7f564c198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.495+0000 7f564ad9d640 1 --2- 172.21.15.28:0/924728274 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f564c103190 0x7f564c198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.495+0000 7f564ad9d640 1 -- 172.21.15.28:0/924728274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f564c19fe50 con 0x7f564c103b50 2024-04-21T08:49:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.495+0000 7f564ad9d640 1 --2- 172.21.15.28:0/924728274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f564c103b50 0x7f564c19f7e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f564400cad0 tx=0x7f564400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.495+0000 7f5643fff640 1 -- 172.21.15.28:0/924728274 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564400ae20 con 0x7f564c103b50 2024-04-21T08:49:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.495+0000 7f5650e9a640 1 -- 172.21.15.28:0/924728274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f564c1a0140 con 0x7f564c103b50 2024-04-21T08:49:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.495+0000 7f5643fff640 1 -- 172.21.15.28:0/924728274 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5644005020 con 0x7f564c103b50 2024-04-21T08:49:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.495+0000 7f5643fff640 1 -- 172.21.15.28:0/924728274 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564400f6b0 con 0x7f564c103b50 2024-04-21T08:49:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.495+0000 7f5650e9a640 1 -- 172.21.15.28:0/924728274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f564c1a0640 con 0x7f564c103b50 2024-04-21T08:49:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.496+0000 7f5650e9a640 1 -- 172.21.15.28:0/924728274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f564c109480 con 0x7f564c103b50 2024-04-21T08:49:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.497+0000 7f5643fff640 1 -- 172.21.15.28:0/924728274 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f56440051d0 con 0x7f564c103b50 2024-04-21T08:49:01.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.498+0000 7f5643fff640 1 --2- 172.21.15.28:0/924728274 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f56380a3bb0 0x7f56380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:01.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.498+0000 7f564a59c640 1 --2- 172.21.15.28:0/924728274 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f56380a3bb0 0x7f56380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:01.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.498+0000 7f5643fff640 1 -- 172.21.15.28:0/924728274 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f56440d0700 con 0x7f564c103b50 2024-04-21T08:49:01.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.498+0000 7f564a59c640 1 --2- 172.21.15.28:0/924728274 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f56380a3bb0 0x7f56380a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f563c005ab0 tx=0x7f563c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:01.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.503+0000 7f5643fff640 1 -- 172.21.15.28:0/924728274 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5644014030 con 0x7f564c103b50 2024-04-21T08:49:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.692+0000 7f5650e9a640 1 -- 172.21.15.28:0/924728274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f564c062b50 con 0x7f564c103b50 2024-04-21T08:49:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.692+0000 7f5643fff640 1 -- 172.21.15.28:0/924728274 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f564409cea0 con 0x7f564c103b50 2024-04-21T08:49:01.692 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-21T08:49:01.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.694+0000 7f5650e9a640 1 -- 172.21.15.28:0/924728274 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f56380a3bb0 msgr2=0x7f56380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:01.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.694+0000 7f5650e9a640 1 --2- 172.21.15.28:0/924728274 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f56380a3bb0 0x7f56380a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f563c005ab0 tx=0x7f563c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.694+0000 7f5650e9a640 1 -- 172.21.15.28:0/924728274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f564c103b50 msgr2=0x7f564c19f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.694+0000 7f5650e9a640 1 --2- 172.21.15.28:0/924728274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f564c103b50 0x7f564c19f7e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f564400cad0 tx=0x7f564400cf90 comp rx=0 tx=0).stop 2024-04-21T08:49:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.695+0000 7f5650e9a640 1 -- 172.21.15.28:0/924728274 shutdown_connections 2024-04-21T08:49:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.695+0000 7f5650e9a640 1 --2- 172.21.15.28:0/924728274 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f56380a3bb0 0x7f56380a6070 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.695+0000 7f5650e9a640 1 --2- 172.21.15.28:0/924728274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f564c103b50 0x7f564c19f7e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.695+0000 7f5650e9a640 1 --2- 172.21.15.28:0/924728274 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f564c103190 0x7f564c198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.695+0000 7f5650e9a640 1 --2- 172.21.15.28:0/924728274 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f564c101fa0 0x7f564c198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.695+0000 7f5650e9a640 1 -- 172.21.15.28:0/924728274 >> 172.21.15.28:0/924728274 conn(0x7f564c0fd530 msgr2=0x7f564c106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.695+0000 7f5650e9a640 1 -- 172.21.15.28:0/924728274 shutdown_connections 2024-04-21T08:49:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.695+0000 7f5650e9a640 1 -- 172.21.15.28:0/924728274 wait complete. 2024-04-21T08:49:01.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:49:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.813+0000 7f6908f9d640 1 -- 172.21.15.28:0/3922499764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904103b50 msgr2=0x7f690410a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.813+0000 7f6908f9d640 1 --2- 172.21.15.28:0/3922499764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904103b50 0x7f690410a3e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009ca0 tx=0x7f68fc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.813+0000 7f6908f9d640 1 -- 172.21.15.28:0/3922499764 shutdown_connections 2024-04-21T08:49:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.813+0000 7f6908f9d640 1 --2- 172.21.15.28:0/3922499764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904103b50 0x7f690410a3e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.813+0000 7f6908f9d640 1 --2- 172.21.15.28:0/3922499764 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6904103190 0x7f6904103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.813+0000 7f6908f9d640 1 --2- 172.21.15.28:0/3922499764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6904101fa0 0x7f69041023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.813+0000 7f6908f9d640 1 -- 172.21.15.28:0/3922499764 >> 172.21.15.28:0/3922499764 conn(0x7f69040fd530 msgr2=0x7f69040ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:01.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.813+0000 7f6908f9d640 1 -- 172.21.15.28:0/3922499764 shutdown_connections 2024-04-21T08:49:01.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.813+0000 7f6908f9d640 1 -- 172.21.15.28:0/3922499764 wait complete. 2024-04-21T08:49:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.814+0000 7f6908f9d640 1 Processor -- start 2024-04-21T08:49:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.814+0000 7f6908f9d640 1 -- start start 2024-04-21T08:49:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.814+0000 7f6908f9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101fa0 0x7f6904198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.814+0000 7f6908f9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6904103190 0x7f69041986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.814+0000 7f6908f9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6904103b50 0x7f690419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.814+0000 7f6908f9d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f690410daa0 con 0x7f6904101fa0 2024-04-21T08:49:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.814+0000 7f6908f9d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f690410d920 con 0x7f6904103190 2024-04-21T08:49:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.814+0000 7f6908f9d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f690410dc20 con 0x7f6904103b50 2024-04-21T08:49:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.814+0000 7f690259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101fa0 0x7f6904198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.814+0000 7f690259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101fa0 0x7f6904198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44196/0 (socket says 172.21.15.28:44196) 2024-04-21T08:49:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.814+0000 7f690259c640 1 -- 172.21.15.28:0/4219248116 learned_addr learned my addr 172.21.15.28:0/4219248116 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.815+0000 7f6902d9d640 1 --2- 172.21.15.28:0/4219248116 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6904103b50 0x7f690419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.815+0000 7f6901d9b640 1 --2- 172.21.15.28:0/4219248116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6904103190 0x7f69041986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.815+0000 7f690259c640 1 -- 172.21.15.28:0/4219248116 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6904103b50 msgr2=0x7f690419f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.815+0000 7f690259c640 1 --2- 172.21.15.28:0/4219248116 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6904103b50 0x7f690419f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.815+0000 7f690259c640 1 -- 172.21.15.28:0/4219248116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6904103190 msgr2=0x7f69041986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.815+0000 7f690259c640 1 --2- 172.21.15.28:0/4219248116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6904103190 0x7f69041986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.815+0000 7f690259c640 1 -- 172.21.15.28:0/4219248116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f690419fd90 con 0x7f6904101fa0 2024-04-21T08:49:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.815+0000 7f6901d9b640 1 --2- 172.21.15.28:0/4219248116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6904103190 0x7f69041986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.815+0000 7f690259c640 1 --2- 172.21.15.28:0/4219248116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101fa0 0x7f6904198170 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f68f400c970 tx=0x7f68f400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.815+0000 7f68f37fe640 1 -- 172.21.15.28:0/4219248116 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f4004220 con 0x7f6904101fa0 2024-04-21T08:49:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.815+0000 7f6908f9d640 1 -- 172.21.15.28:0/4219248116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69041a0080 con 0x7f6904101fa0 2024-04-21T08:49:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.815+0000 7f68f37fe640 1 -- 172.21.15.28:0/4219248116 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68f40043c0 con 0x7f6904101fa0 2024-04-21T08:49:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.815+0000 7f68f37fe640 1 -- 172.21.15.28:0/4219248116 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f400f450 con 0x7f6904101fa0 2024-04-21T08:49:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.815+0000 7f6908f9d640 1 -- 172.21.15.28:0/4219248116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69041a05d0 con 0x7f6904101fa0 2024-04-21T08:49:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.816+0000 7f6908f9d640 1 -- 172.21.15.28:0/4219248116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6904109340 con 0x7f6904101fa0 2024-04-21T08:49:01.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.817+0000 7f68f37fe640 1 -- 172.21.15.28:0/4219248116 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f68f4003610 con 0x7f6904101fa0 2024-04-21T08:49:01.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.818+0000 7f68f37fe640 1 --2- 172.21.15.28:0/4219248116 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f68ec0a3bb0 0x7f68ec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:01.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.818+0000 7f6901d9b640 1 --2- 172.21.15.28:0/4219248116 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f68ec0a3bb0 0x7f68ec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:01.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.818+0000 7f68f37fe640 1 -- 172.21.15.28:0/4219248116 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f68f40d0870 con 0x7f6904101fa0 2024-04-21T08:49:01.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.819+0000 7f6901d9b640 1 --2- 172.21.15.28:0/4219248116 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f68ec0a3bb0 0x7f68ec0a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6904199680 tx=0x7f68f8005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:01.822+0000 7f68f37fe640 1 -- 172.21.15.28:0/4219248116 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f68f4014030 con 0x7f6904101fa0 2024-04-21T08:49:02.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.008+0000 7f6908f9d640 1 -- 172.21.15.28:0/4219248116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f690402d870 con 0x7f6904101fa0 2024-04-21T08:49:02.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.009+0000 7f68f37fe640 1 -- 172.21.15.28:0/4219248116 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f68f409cfb0 con 0x7f6904101fa0 2024-04-21T08:49:02.009 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:49:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.011+0000 7f6908f9d640 1 -- 172.21.15.28:0/4219248116 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f68ec0a3bb0 msgr2=0x7f68ec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.011+0000 7f6908f9d640 1 --2- 172.21.15.28:0/4219248116 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f68ec0a3bb0 0x7f68ec0a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6904199680 tx=0x7f68f8005e50 comp rx=0 tx=0).stop 2024-04-21T08:49:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.011+0000 7f6908f9d640 1 -- 172.21.15.28:0/4219248116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101fa0 msgr2=0x7f6904198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.011+0000 7f6908f9d640 1 --2- 172.21.15.28:0/4219248116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101fa0 0x7f6904198170 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f68f400c970 tx=0x7f68f400ce30 comp rx=0 tx=0).stop 2024-04-21T08:49:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.011+0000 7f6908f9d640 1 -- 172.21.15.28:0/4219248116 shutdown_connections 2024-04-21T08:49:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.011+0000 7f6908f9d640 1 --2- 172.21.15.28:0/4219248116 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6904103b50 0x7f690419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.011+0000 7f6908f9d640 1 --2- 172.21.15.28:0/4219248116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6904103190 0x7f69041986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.011+0000 7f6908f9d640 1 --2- 172.21.15.28:0/4219248116 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f68ec0a3bb0 0x7f68ec0a6070 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.011+0000 7f6908f9d640 1 --2- 172.21.15.28:0/4219248116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101fa0 0x7f6904198170 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.011+0000 7f6908f9d640 1 -- 172.21.15.28:0/4219248116 >> 172.21.15.28:0/4219248116 conn(0x7f69040fd530 msgr2=0x7f6904106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.011+0000 7f6908f9d640 1 -- 172.21.15.28:0/4219248116 shutdown_connections 2024-04-21T08:49:02.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.011+0000 7f6908f9d640 1 -- 172.21.15.28:0/4219248116 wait complete. 2024-04-21T08:49:02.020 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:01 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3812438024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:49:02.020 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:01 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1561411016' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:49:02.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:01 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/924728274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:49:02.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T08:49:02.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:01 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3812438024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:49:02.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:01 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1561411016' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:49:02.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:01 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/924728274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:49:02.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:01 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3812438024' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:49:02.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:01 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1561411016' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:49:02.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:01 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/924728274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:49:02.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.125+0000 7f50f0a25640 1 -- 172.21.15.28:0/4185452497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec102af0 msgr2=0x7f50ec104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:02.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.125+0000 7f50f0a25640 1 --2- 172.21.15.28:0/4185452497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec102af0 0x7f50ec104ee0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f50dc009900 tx=0x7f50dc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.125+0000 7f50f0a25640 1 -- 172.21.15.28:0/4185452497 shutdown_connections 2024-04-21T08:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.125+0000 7f50f0a25640 1 --2- 172.21.15.28:0/4185452497 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50ec107d50 0x7f50ec10a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.125+0000 7f50f0a25640 1 --2- 172.21.15.28:0/4185452497 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50ec105420 0x7f50ec107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.125+0000 7f50f0a25640 1 --2- 172.21.15.28:0/4185452497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec102af0 0x7f50ec104ee0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.125+0000 7f50f0a25640 1 -- 172.21.15.28:0/4185452497 >> 172.21.15.28:0/4185452497 conn(0x7f50ec0fc420 msgr2=0x7f50ec0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.125+0000 7f50f0a25640 1 -- 172.21.15.28:0/4185452497 shutdown_connections 2024-04-21T08:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.125+0000 7f50f0a25640 1 -- 172.21.15.28:0/4185452497 wait complete. 2024-04-21T08:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.126+0000 7f50f0a25640 1 Processor -- start 2024-04-21T08:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.126+0000 7f50f0a25640 1 -- start start 2024-04-21T08:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.126+0000 7f50f0a25640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50ec102af0 0x7f50ec102460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.126+0000 7f50f0a25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec105420 0x7f50ec100aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.126+0000 7f50f0a25640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50ec107d50 0x7f50ec100fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.126+0000 7f50f0a25640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec10dbe0 con 0x7f50ec105420 2024-04-21T08:49:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.126+0000 7f50f0a25640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec10da60 con 0x7f50ec107d50 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.126+0000 7f50f0a25640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec10dd60 con 0x7f50ec102af0 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.126+0000 7f50e9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec105420 0x7f50ec100aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.126+0000 7f50e9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec105420 0x7f50ec100aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44206/0 (socket says 172.21.15.28:44206) 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.126+0000 7f50e9d9b640 1 -- 172.21.15.28:0/415777555 learned_addr learned my addr 172.21.15.28:0/415777555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.127+0000 7f50ead9d640 1 --2- 172.21.15.28:0/415777555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50ec107d50 0x7f50ec100fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.127+0000 7f50ea59c640 1 --2- 172.21.15.28:0/415777555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50ec102af0 0x7f50ec102460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.127+0000 7f50e9d9b640 1 -- 172.21.15.28:0/415777555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50ec102af0 msgr2=0x7f50ec102460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.127+0000 7f50e9d9b640 1 --2- 172.21.15.28:0/415777555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50ec102af0 0x7f50ec102460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.127+0000 7f50e9d9b640 1 -- 172.21.15.28:0/415777555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50ec107d50 msgr2=0x7f50ec100fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.127+0000 7f50e9d9b640 1 --2- 172.21.15.28:0/415777555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50ec107d50 0x7f50ec100fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.127+0000 7f50e9d9b640 1 -- 172.21.15.28:0/415777555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50ec101870 con 0x7f50ec105420 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.127+0000 7f50ea59c640 1 --2- 172.21.15.28:0/415777555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50ec102af0 0x7f50ec102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.127+0000 7f50e9d9b640 1 --2- 172.21.15.28:0/415777555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec105420 0x7f50ec100aa0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f50e000b450 tx=0x7f50e000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.127+0000 7f50db7fe640 1 -- 172.21.15.28:0/415777555 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e0015020 con 0x7f50ec105420 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.127+0000 7f50db7fe640 1 -- 172.21.15.28:0/415777555 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50e0011d40 con 0x7f50ec105420 2024-04-21T08:49:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.127+0000 7f50f0a25640 1 -- 172.21.15.28:0/415777555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50ec1a24f0 con 0x7f50ec105420 2024-04-21T08:49:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.127+0000 7f50db7fe640 1 -- 172.21.15.28:0/415777555 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e000f340 con 0x7f50ec105420 2024-04-21T08:49:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.127+0000 7f50f0a25640 1 -- 172.21.15.28:0/415777555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50ec1a2b10 con 0x7f50ec105420 2024-04-21T08:49:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.128+0000 7f50f0a25640 1 -- 172.21.15.28:0/415777555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50ec0fdf10 con 0x7f50ec105420 2024-04-21T08:49:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.130+0000 7f50db7fe640 1 -- 172.21.15.28:0/415777555 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f50e0018020 con 0x7f50ec105420 2024-04-21T08:49:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.131+0000 7f50db7fe640 1 --2- 172.21.15.28:0/415777555 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50d40a3c30 0x7f50d40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.131+0000 7f50db7fe640 1 -- 172.21.15.28:0/415777555 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f50e00d0770 con 0x7f50ec105420 2024-04-21T08:49:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.131+0000 7f50ea59c640 1 --2- 172.21.15.28:0/415777555 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50d40a3c30 0x7f50d40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.131+0000 7f50ea59c640 1 --2- 172.21.15.28:0/415777555 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50d40a3c30 0x7f50d40a60f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0096c0 tx=0x7f50dc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:02.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.134+0000 7f50db7fe640 1 -- 172.21.15.28:0/415777555 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f50e001f020 con 0x7f50ec105420 2024-04-21T08:49:02.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.333+0000 7f50f0a25640 1 -- 172.21.15.28:0/415777555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f50ec02d810 con 0x7f50ec105420 2024-04-21T08:49:02.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.334+0000 7f50db7fe640 1 -- 172.21.15.28:0/415777555 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+587 (secure 0 0 0) 0x7f50e000d040 con 0x7f50ec105420 2024-04-21T08:49:02.334 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3741297321 2024-04-22T08:41:35.374122+0000 2024-04-21T08:49:02.334 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/334321812 2024-04-22T08:41:35.374122+0000 2024-04-21T08:49:02.334 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2874636306 2024-04-22T08:41:07.583432+0000 2024-04-21T08:49:02.334 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/334321812 2024-04-22T08:41:35.374122+0000 2024-04-21T08:49:02.334 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1716427535 2024-04-22T08:41:07.583432+0000 2024-04-21T08:49:02.334 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/731326464 2024-04-22T08:41:35.374122+0000 2024-04-21T08:49:02.334 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2874636306 2024-04-22T08:41:07.583432+0000 2024-04-21T08:49:02.334 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/326837648 2024-04-22T08:41:35.374122+0000 2024-04-21T08:49:02.334 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1980013253 2024-04-22T08:41:07.583432+0000 2024-04-21T08:49:02.334 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/917824119 2024-04-22T08:41:07.583432+0000 2024-04-21T08:49:02.334 INFO:teuthology.orchestra.run.smithi028.stderr:listed 10 entries 2024-04-21T08:49:02.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.336+0000 7f50f0a25640 1 -- 172.21.15.28:0/415777555 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50d40a3c30 msgr2=0x7f50d40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.336+0000 7f50f0a25640 1 --2- 172.21.15.28:0/415777555 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50d40a3c30 0x7f50d40a60f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0096c0 tx=0x7f50dc000bc0 comp rx=0 tx=0).stop 2024-04-21T08:49:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.336+0000 7f50f0a25640 1 -- 172.21.15.28:0/415777555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec105420 msgr2=0x7f50ec100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.336+0000 7f50f0a25640 1 --2- 172.21.15.28:0/415777555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec105420 0x7f50ec100aa0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f50e000b450 tx=0x7f50e000b910 comp rx=0 tx=0).stop 2024-04-21T08:49:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.336+0000 7f50f0a25640 1 -- 172.21.15.28:0/415777555 shutdown_connections 2024-04-21T08:49:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.336+0000 7f50f0a25640 1 --2- 172.21.15.28:0/415777555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f50ec107d50 0x7f50ec100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.336+0000 7f50f0a25640 1 --2- 172.21.15.28:0/415777555 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50d40a3c30 0x7f50d40a60f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.336+0000 7f50f0a25640 1 --2- 172.21.15.28:0/415777555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50ec105420 0x7f50ec100aa0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:02.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.336+0000 7f50f0a25640 1 --2- 172.21.15.28:0/415777555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50ec102af0 0x7f50ec102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:02.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.336+0000 7f50f0a25640 1 -- 172.21.15.28:0/415777555 >> 172.21.15.28:0/415777555 conn(0x7f50ec0fc420 msgr2=0x7f50ec1049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:02.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.336+0000 7f50f0a25640 1 -- 172.21.15.28:0/415777555 shutdown_connections 2024-04-21T08:49:02.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:02.336+0000 7f50f0a25640 1 -- 172.21.15.28:0/415777555 wait complete. 2024-04-21T08:49:02.346 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-04-21T08:49:02.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:49:02.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:49:02.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[71416]: fuse finished with error 0 and tester_r 0 2024-04-21T08:49:02.371 INFO:teuthology.orchestra.run:waiting for 300 2024-04-21T08:49:03.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:02 smithi152 ceph-mon[31401]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-04-21T08:49:03.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:02 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/4219248116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:49:03.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:02 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/415777555' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:49:03.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:02 smithi028 ceph-mon[28003]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-04-21T08:49:03.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:02 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/4219248116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:49:03.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:02 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/415777555' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:49:03.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:02 smithi171 ceph-mon[31358]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-04-21T08:49:03.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:02 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/4219248116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:49:03.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:02 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/415777555' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:49:05.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:04 smithi152 ceph-mon[31401]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 262 B/s wr, 0 op/s 2024-04-21T08:49:05.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:04 smithi028 ceph-mon[28003]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 262 B/s wr, 0 op/s 2024-04-21T08:49:05.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:04 smithi171 ceph-mon[31358]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 262 B/s wr, 0 op/s 2024-04-21T08:49:07.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:06 smithi152 ceph-mon[31401]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-21T08:49:07.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:06 smithi028 ceph-mon[28003]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-21T08:49:07.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:06 smithi171 ceph-mon[31358]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-21T08:49:08.473 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-04-21T08:49:08.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:49:08.473 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-21T08:49:08.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:49:08.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.641+0000 7f64ab796640 1 -- 172.21.15.28:0/3495645025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64a4100ea0 msgr2=0x7f64a41012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:08.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.641+0000 7f64ab796640 1 --2- 172.21.15.28:0/3495645025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64a4100ea0 0x7f64a41012c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f64a0009900 tx=0x7f64a002bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.641+0000 7f64ab796640 1 -- 172.21.15.28:0/3495645025 shutdown_connections 2024-04-21T08:49:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.641+0000 7f64ab796640 1 --2- 172.21.15.28:0/3495645025 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64a41029b0 0x7f64a41092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.641+0000 7f64ab796640 1 --2- 172.21.15.28:0/3495645025 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64a4101ff0 0x7f64a4102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.641+0000 7f64ab796640 1 --2- 172.21.15.28:0/3495645025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64a4100ea0 0x7f64a41012c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.641+0000 7f64ab796640 1 -- 172.21.15.28:0/3495645025 >> 172.21.15.28:0/3495645025 conn(0x7f64a40fc410 msgr2=0x7f64a40fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.641+0000 7f64ab796640 1 -- 172.21.15.28:0/3495645025 shutdown_connections 2024-04-21T08:49:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.641+0000 7f64ab796640 1 -- 172.21.15.28:0/3495645025 wait complete. 2024-04-21T08:49:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.642+0000 7f64ab796640 1 Processor -- start 2024-04-21T08:49:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.642+0000 7f64ab796640 1 -- start start 2024-04-21T08:49:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.642+0000 7f64ab796640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64a4100ea0 0x7f64a4197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.642+0000 7f64ab796640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64a4101ff0 0x7f64a4197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.642+0000 7f64ab796640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64a41029b0 0x7f64a419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.642+0000 7f64ab796640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64a410caa0 con 0x7f64a4100ea0 2024-04-21T08:49:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.642+0000 7f64ab796640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f64a410c920 con 0x7f64a41029b0 2024-04-21T08:49:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.642+0000 7f64ab796640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f64a410cc20 con 0x7f64a4101ff0 2024-04-21T08:49:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.642+0000 7f64a9532640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64a4100ea0 0x7f64a4197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.642+0000 7f64a9532640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64a4100ea0 0x7f64a4197150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53602/0 (socket says 172.21.15.28:53602) 2024-04-21T08:49:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.642+0000 7f64a9532640 1 -- 172.21.15.28:0/702927479 learned_addr learned my addr 172.21.15.28:0/702927479 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.642+0000 7f64a8d31640 1 --2- 172.21.15.28:0/702927479 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64a4101ff0 0x7f64a4197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.643+0000 7f64a9d33640 1 --2- 172.21.15.28:0/702927479 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64a41029b0 0x7f64a419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.643+0000 7f64a9532640 1 -- 172.21.15.28:0/702927479 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64a4101ff0 msgr2=0x7f64a4197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.643+0000 7f64a9532640 1 --2- 172.21.15.28:0/702927479 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64a4101ff0 0x7f64a4197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.643+0000 7f64a9532640 1 -- 172.21.15.28:0/702927479 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64a41029b0 msgr2=0x7f64a419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.643+0000 7f64a9532640 1 --2- 172.21.15.28:0/702927479 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64a41029b0 0x7f64a419e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.643+0000 7f64a9532640 1 -- 172.21.15.28:0/702927479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a419ed70 con 0x7f64a4100ea0 2024-04-21T08:49:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.643+0000 7f64a9d33640 1 --2- 172.21.15.28:0/702927479 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64a41029b0 0x7f64a419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.643+0000 7f64a9532640 1 --2- 172.21.15.28:0/702927479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64a4100ea0 0x7f64a4197150 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f64a002c840 tx=0x7f64a00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.643+0000 7f649a7fc640 1 -- 172.21.15.28:0/702927479 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0043020 con 0x7f64a4100ea0 2024-04-21T08:49:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.643+0000 7f649a7fc640 1 -- 172.21.15.28:0/702927479 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64a0004aa0 con 0x7f64a4100ea0 2024-04-21T08:49:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.643+0000 7f64ab796640 1 -- 172.21.15.28:0/702927479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64a419f000 con 0x7f64a4100ea0 2024-04-21T08:49:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.643+0000 7f649a7fc640 1 -- 172.21.15.28:0/702927479 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0003cd0 con 0x7f64a4100ea0 2024-04-21T08:49:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.643+0000 7f64ab796640 1 -- 172.21.15.28:0/702927479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64a419f5d0 con 0x7f64a4100ea0 2024-04-21T08:49:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.644+0000 7f64ab796640 1 -- 172.21.15.28:0/702927479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64a4108150 con 0x7f64a4100ea0 2024-04-21T08:49:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.645+0000 7f649a7fc640 1 -- 172.21.15.28:0/702927479 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f64a0046020 con 0x7f64a4100ea0 2024-04-21T08:49:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.646+0000 7f649a7fc640 1 --2- 172.21.15.28:0/702927479 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f64900a3bb0 0x7f64900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.646+0000 7f64a8d31640 1 --2- 172.21.15.28:0/702927479 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f64900a3bb0 0x7f64900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.646+0000 7f649a7fc640 1 -- 172.21.15.28:0/702927479 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f64a00f2de0 con 0x7f64a4100ea0 2024-04-21T08:49:08.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.646+0000 7f64a8d31640 1 --2- 172.21.15.28:0/702927479 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f64900a3bb0 0x7f64900a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6494005fd0 tx=0x7f6494005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.649+0000 7f649a7fc640 1 -- 172.21.15.28:0/702927479 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f64a004d020 con 0x7f64a4100ea0 2024-04-21T08:49:08.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.836+0000 7f64ab796640 1 -- 172.21.15.28:0/702927479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64a4062b50 con 0x7f64a4100ea0 2024-04-21T08:49:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.837+0000 7f649a7fc640 1 -- 172.21.15.28:0/702927479 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24101 (secure 0 0 0) 0x7f64a00bf580 con 0x7f64a4100ea0 2024-04-21T08:49:08.837 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:49:08.837 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:48:51.121128+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:49:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.839+0000 7f64ab796640 1 -- 172.21.15.28:0/702927479 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f64900a3bb0 msgr2=0x7f64900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.839+0000 7f64ab796640 1 --2- 172.21.15.28:0/702927479 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f64900a3bb0 0x7f64900a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6494005fd0 tx=0x7f6494005850 comp rx=0 tx=0).stop 2024-04-21T08:49:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.839+0000 7f64ab796640 1 -- 172.21.15.28:0/702927479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64a4100ea0 msgr2=0x7f64a4197150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.839+0000 7f64ab796640 1 --2- 172.21.15.28:0/702927479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64a4100ea0 0x7f64a4197150 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f64a002c840 tx=0x7f64a00042f0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.839+0000 7f64ab796640 1 -- 172.21.15.28:0/702927479 shutdown_connections 2024-04-21T08:49:08.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.839+0000 7f64ab796640 1 --2- 172.21.15.28:0/702927479 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64a41029b0 0x7f64a419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.839+0000 7f64ab796640 1 --2- 172.21.15.28:0/702927479 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64a4101ff0 0x7f64a4197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.839+0000 7f64ab796640 1 --2- 172.21.15.28:0/702927479 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f64900a3bb0 0x7f64900a6070 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.839+0000 7f64ab796640 1 --2- 172.21.15.28:0/702927479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64a4100ea0 0x7f64a4197150 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.839+0000 7f64ab796640 1 -- 172.21.15.28:0/702927479 >> 172.21.15.28:0/702927479 conn(0x7f64a40fc410 msgr2=0x7f64a4105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:08.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.840+0000 7f64ab796640 1 -- 172.21.15.28:0/702927479 shutdown_connections 2024-04-21T08:49:08.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.840+0000 7f64ab796640 1 -- 172.21.15.28:0/702927479 wait complete. 2024-04-21T08:49:08.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:49:08.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.952+0000 7fb6b66b2640 1 -- 172.21.15.28:0/1339835508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100e70 msgr2=0x7fb6b0101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.952+0000 7fb6b66b2640 1 --2- 172.21.15.28:0/1339835508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100e70 0x7fb6b0101290 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb698009900 tx=0x7fb69802bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.953+0000 7fb6b66b2640 1 -- 172.21.15.28:0/1339835508 shutdown_connections 2024-04-21T08:49:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.953+0000 7fb6b66b2640 1 --2- 172.21.15.28:0/1339835508 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6b0102980 0x7fb6b01092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.953+0000 7fb6b66b2640 1 --2- 172.21.15.28:0/1339835508 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6b0101fc0 0x7fb6b0102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.953+0000 7fb6b66b2640 1 --2- 172.21.15.28:0/1339835508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100e70 0x7fb6b0101290 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.953+0000 7fb6b66b2640 1 -- 172.21.15.28:0/1339835508 >> 172.21.15.28:0/1339835508 conn(0x7fb6b00fc3e0 msgr2=0x7fb6b00fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.953+0000 7fb6b66b2640 1 -- 172.21.15.28:0/1339835508 shutdown_connections 2024-04-21T08:49:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.953+0000 7fb6b66b2640 1 -- 172.21.15.28:0/1339835508 wait complete. 2024-04-21T08:49:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.953+0000 7fb6b66b2640 1 Processor -- start 2024-04-21T08:49:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.953+0000 7fb6b66b2640 1 -- start start 2024-04-21T08:49:08.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.953+0000 7fb6b66b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100e70 0x7fb6b01981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.953+0000 7fb6b66b2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6b0101fc0 0x7fb6b0198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6b66b2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6b0102980 0x7fb6b019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6b66b2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b010c9c0 con 0x7fb6b0100e70 2024-04-21T08:49:08.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6b66b2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b010c840 con 0x7fb6b0101fc0 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6b66b2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b010cb40 con 0x7fb6b0102980 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6affff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100e70 0x7fb6b01981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6affff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100e70 0x7fb6b01981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53626/0 (socket says 172.21.15.28:53626) 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6affff640 1 -- 172.21.15.28:0/2431405702 learned_addr learned my addr 172.21.15.28:0/2431405702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6af7fe640 1 --2- 172.21.15.28:0/2431405702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6b0101fc0 0x7fb6b0198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6b4c4f640 1 --2- 172.21.15.28:0/2431405702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6b0102980 0x7fb6b019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6affff640 1 -- 172.21.15.28:0/2431405702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6b0102980 msgr2=0x7fb6b019f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6affff640 1 --2- 172.21.15.28:0/2431405702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6b0102980 0x7fb6b019f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6affff640 1 -- 172.21.15.28:0/2431405702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6b0101fc0 msgr2=0x7fb6b0198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6affff640 1 --2- 172.21.15.28:0/2431405702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6b0101fc0 0x7fb6b0198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6affff640 1 -- 172.21.15.28:0/2431405702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6b019fe00 con 0x7fb6b0100e70 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6b4c4f640 1 --2- 172.21.15.28:0/2431405702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6b0102980 0x7fb6b019f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6affff640 1 --2- 172.21.15.28:0/2431405702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100e70 0x7fb6b01981e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb698006010 tx=0x7fb6980042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6ad7fa640 1 -- 172.21.15.28:0/2431405702 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb698043020 con 0x7fb6b0100e70 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6ad7fa640 1 -- 172.21.15.28:0/2431405702 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb698004aa0 con 0x7fb6b0100e70 2024-04-21T08:49:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6b66b2640 1 -- 172.21.15.28:0/2431405702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6b01a0090 con 0x7fb6b0100e70 2024-04-21T08:49:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6ad7fa640 1 -- 172.21.15.28:0/2431405702 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb698003c70 con 0x7fb6b0100e70 2024-04-21T08:49:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.954+0000 7fb6b66b2640 1 -- 172.21.15.28:0/2431405702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6b01a05e0 con 0x7fb6b0100e70 2024-04-21T08:49:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.955+0000 7fb6b66b2640 1 -- 172.21.15.28:0/2431405702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6b01a07e0 con 0x7fb6b0100e70 2024-04-21T08:49:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.957+0000 7fb6ad7fa640 1 -- 172.21.15.28:0/2431405702 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb698046020 con 0x7fb6b0100e70 2024-04-21T08:49:08.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.957+0000 7fb6ad7fa640 1 --2- 172.21.15.28:0/2431405702 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb69c0a3bb0 0x7fb69c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:08.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.957+0000 7fb6af7fe640 1 --2- 172.21.15.28:0/2431405702 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb69c0a3bb0 0x7fb69c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:08.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.957+0000 7fb6ad7fa640 1 -- 172.21.15.28:0/2431405702 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb6980f3090 con 0x7fb6b0100e70 2024-04-21T08:49:08.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.958+0000 7fb6af7fe640 1 --2- 172.21.15.28:0/2431405702 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb69c0a3bb0 0x7fb69c0a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb6b01996f0 tx=0x7fb6a000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:08.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:08.961+0000 7fb6ad7fa640 1 -- 172.21.15.28:0/2431405702 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb69804d020 con 0x7fb6b0100e70 2024-04-21T08:49:09.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:08 smithi152 ceph-mon[31401]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:49:09.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:08 smithi028 ceph-mon[28003]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:49:09.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:08 smithi171 ceph-mon[31358]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:49:09.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.149+0000 7fb6b66b2640 1 -- 172.21.15.28:0/2431405702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6b01a09b0 con 0x7fb6b0100e70 2024-04-21T08:49:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.151+0000 7fb6ad7fa640 1 -- 172.21.15.28:0/2431405702 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24101 (secure 0 0 0) 0x7fb6980bf830 con 0x7fb6b0100e70 2024-04-21T08:49:09.150 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:49:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:48:51.121128+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:49:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.153+0000 7fb6b66b2640 1 -- 172.21.15.28:0/2431405702 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb69c0a3bb0 msgr2=0x7fb69c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.153+0000 7fb6b66b2640 1 --2- 172.21.15.28:0/2431405702 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb69c0a3bb0 0x7fb69c0a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb6b01996f0 tx=0x7fb6a000b390 comp rx=0 tx=0).stop 2024-04-21T08:49:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.153+0000 7fb6b66b2640 1 -- 172.21.15.28:0/2431405702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100e70 msgr2=0x7fb6b01981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.153+0000 7fb6b66b2640 1 --2- 172.21.15.28:0/2431405702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100e70 0x7fb6b01981e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb698006010 tx=0x7fb6980042f0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.153+0000 7fb6b66b2640 1 -- 172.21.15.28:0/2431405702 shutdown_connections 2024-04-21T08:49:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.153+0000 7fb6b66b2640 1 --2- 172.21.15.28:0/2431405702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6b0102980 0x7fb6b019f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.153+0000 7fb6b66b2640 1 --2- 172.21.15.28:0/2431405702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6b0101fc0 0x7fb6b0198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.153+0000 7fb6b66b2640 1 --2- 172.21.15.28:0/2431405702 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb69c0a3bb0 0x7fb69c0a6070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.153+0000 7fb6b66b2640 1 --2- 172.21.15.28:0/2431405702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100e70 0x7fb6b01981e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.153+0000 7fb6b66b2640 1 -- 172.21.15.28:0/2431405702 >> 172.21.15.28:0/2431405702 conn(0x7fb6b00fc3e0 msgr2=0x7fb6b00fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.153+0000 7fb6b66b2640 1 -- 172.21.15.28:0/2431405702 shutdown_connections 2024-04-21T08:49:09.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.153+0000 7fb6b66b2640 1 -- 172.21.15.28:0/2431405702 wait complete. 2024-04-21T08:49:09.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-21T08:49:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.269+0000 7f173f1d5640 1 -- 172.21.15.28:0/3460914910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1738105360 msgr2=0x7f1738107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.269+0000 7f173f1d5640 1 --2- 172.21.15.28:0/3460914910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1738105360 0x7f1738107750 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1724009900 tx=0x7f172402bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.269+0000 7f173f1d5640 1 -- 172.21.15.28:0/3460914910 shutdown_connections 2024-04-21T08:49:09.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.269+0000 7f173f1d5640 1 --2- 172.21.15.28:0/3460914910 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1738107c90 0x7f173810a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.269+0000 7f173f1d5640 1 --2- 172.21.15.28:0/3460914910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1738105360 0x7f1738107750 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.269+0000 7f173f1d5640 1 --2- 172.21.15.28:0/3460914910 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1738102a30 0x7f1738104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.269+0000 7f173f1d5640 1 -- 172.21.15.28:0/3460914910 >> 172.21.15.28:0/3460914910 conn(0x7f17380fc440 msgr2=0x7f17380fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:09.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.269+0000 7f173f1d5640 1 -- 172.21.15.28:0/3460914910 shutdown_connections 2024-04-21T08:49:09.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.269+0000 7f173f1d5640 1 -- 172.21.15.28:0/3460914910 wait complete. 2024-04-21T08:49:09.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173f1d5640 1 Processor -- start 2024-04-21T08:49:09.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173f1d5640 1 -- start start 2024-04-21T08:49:09.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173f1d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1738102a30 0x7f1738198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:09.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173f1d5640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1738105360 0x7f1738198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:09.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173f1d5640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1738107c90 0x7f173819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:09.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173f1d5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f173810daf0 con 0x7f1738102a30 2024-04-21T08:49:09.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173f1d5640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f173810d970 con 0x7f1738107c90 2024-04-21T08:49:09.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173f1d5640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f173810dc70 con 0x7f1738105360 2024-04-21T08:49:09.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173cf71640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1738102a30 0x7f1738198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:09.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173cf71640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1738102a30 0x7f1738198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53644/0 (socket says 172.21.15.28:53644) 2024-04-21T08:49:09.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173cf71640 1 -- 172.21.15.28:0/3420684960 learned_addr learned my addr 172.21.15.28:0/3420684960 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:09.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173d772640 1 --2- 172.21.15.28:0/3420684960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1738107c90 0x7f173819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:09.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173cf71640 1 -- 172.21.15.28:0/3420684960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1738105360 msgr2=0x7f1738198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:09.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173cf71640 1 --2- 172.21.15.28:0/3420684960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1738105360 0x7f1738198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173cf71640 1 -- 172.21.15.28:0/3420684960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1738107c90 msgr2=0x7f173819f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:09.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173cf71640 1 --2- 172.21.15.28:0/3420684960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1738107c90 0x7f173819f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173cf71640 1 -- 172.21.15.28:0/3420684960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f173819fe40 con 0x7f1738102a30 2024-04-21T08:49:09.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.270+0000 7f173d772640 1 --2- 172.21.15.28:0/3420684960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1738107c90 0x7f173819f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:09.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.271+0000 7f173cf71640 1 --2- 172.21.15.28:0/3420684960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1738102a30 0x7f1738198220 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f172c00d950 tx=0x7f172c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:09.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.271+0000 7f1735ffb640 1 -- 172.21.15.28:0/3420684960 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c017070 con 0x7f1738102a30 2024-04-21T08:49:09.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.271+0000 7f1735ffb640 1 -- 172.21.15.28:0/3420684960 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f172c009c30 con 0x7f1738102a30 2024-04-21T08:49:09.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.271+0000 7f173f1d5640 1 -- 172.21.15.28:0/3420684960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17381a0130 con 0x7f1738102a30 2024-04-21T08:49:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.271+0000 7f1735ffb640 1 -- 172.21.15.28:0/3420684960 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c00f620 con 0x7f1738102a30 2024-04-21T08:49:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.271+0000 7f173f1d5640 1 -- 172.21.15.28:0/3420684960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17381a06d0 con 0x7f1738102a30 2024-04-21T08:49:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.272+0000 7f173f1d5640 1 -- 172.21.15.28:0/3420684960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17380fdf30 con 0x7f1738102a30 2024-04-21T08:49:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.273+0000 7f1735ffb640 1 -- 172.21.15.28:0/3420684960 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f172c010460 con 0x7f1738102a30 2024-04-21T08:49:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.274+0000 7f1735ffb640 1 --2- 172.21.15.28:0/3420684960 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f17280a3c30 0x7f17280a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:09.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.274+0000 7f1735ffb640 1 -- 172.21.15.28:0/3420684960 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f172c0d02e0 con 0x7f1738102a30 2024-04-21T08:49:09.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.274+0000 7f1737fff640 1 --2- 172.21.15.28:0/3420684960 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f17280a3c30 0x7f17280a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:09.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.274+0000 7f1737fff640 1 --2- 172.21.15.28:0/3420684960 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f17280a3c30 0x7f17280a60f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1724036040 tx=0x7f1724000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:09.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.277+0000 7f1735ffb640 1 -- 172.21.15.28:0/3420684960 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f172c09caa0 con 0x7f1738102a30 2024-04-21T08:49:09.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.507+0000 7f173f1d5640 1 -- 172.21.15.28:0/3420684960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f173802d850 con 0x7f1738102a30 2024-04-21T08:49:09.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.807+0000 7f1735ffb640 1 -- 172.21.15.28:0/3420684960 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f172c009de0 con 0x7f1738102a30 2024-04-21T08:49:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.810+0000 7f173f1d5640 1 -- 172.21.15.28:0/3420684960 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f17280a3c30 msgr2=0x7f17280a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.810+0000 7f173f1d5640 1 --2- 172.21.15.28:0/3420684960 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f17280a3c30 0x7f17280a60f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1724036040 tx=0x7f1724000bc0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.810+0000 7f173f1d5640 1 -- 172.21.15.28:0/3420684960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1738102a30 msgr2=0x7f1738198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.810+0000 7f173f1d5640 1 --2- 172.21.15.28:0/3420684960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1738102a30 0x7f1738198220 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f172c00d950 tx=0x7f172c00de10 comp rx=0 tx=0).stop 2024-04-21T08:49:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.810+0000 7f173f1d5640 1 -- 172.21.15.28:0/3420684960 shutdown_connections 2024-04-21T08:49:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.810+0000 7f173f1d5640 1 --2- 172.21.15.28:0/3420684960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1738107c90 0x7f173819f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.810+0000 7f173f1d5640 1 --2- 172.21.15.28:0/3420684960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1738105360 0x7f1738198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.810+0000 7f173f1d5640 1 --2- 172.21.15.28:0/3420684960 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f17280a3c30 0x7f17280a60f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.810+0000 7f173f1d5640 1 --2- 172.21.15.28:0/3420684960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1738102a30 0x7f1738198220 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.810+0000 7f173f1d5640 1 -- 172.21.15.28:0/3420684960 >> 172.21.15.28:0/3420684960 conn(0x7f17380fc440 msgr2=0x7f1738103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.810+0000 7f173f1d5640 1 -- 172.21.15.28:0/3420684960 shutdown_connections 2024-04-21T08:49:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.810+0000 7f173f1d5640 1 -- 172.21.15.28:0/3420684960 wait complete. 2024-04-21T08:49:09.822 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-21T08:49:09.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-21T08:49:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.929+0000 7f233434a640 1 -- 172.21.15.28:0/1689428238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f232c102040 msgr2=0x7f232c1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.929+0000 7f233434a640 1 --2- 172.21.15.28:0/1689428238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f232c102040 0x7f232c1024c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f231c009900 tx=0x7f231c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.929+0000 7f233434a640 1 -- 172.21.15.28:0/1689428238 shutdown_connections 2024-04-21T08:49:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.929+0000 7f233434a640 1 --2- 172.21.15.28:0/1689428238 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f232c102a00 0x7f232c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.929+0000 7f233434a640 1 --2- 172.21.15.28:0/1689428238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f232c102040 0x7f232c1024c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.929+0000 7f233434a640 1 --2- 172.21.15.28:0/1689428238 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f232c100e60 0x7f232c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.929+0000 7f233434a640 1 -- 172.21.15.28:0/1689428238 >> 172.21.15.28:0/1689428238 conn(0x7f232c0fc440 msgr2=0x7f232c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.929+0000 7f233434a640 1 -- 172.21.15.28:0/1689428238 shutdown_connections 2024-04-21T08:49:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.929+0000 7f233434a640 1 -- 172.21.15.28:0/1689428238 wait complete. 2024-04-21T08:49:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f233434a640 1 Processor -- start 2024-04-21T08:49:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f233434a640 1 -- start start 2024-04-21T08:49:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f233434a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f232c100e60 0x7f232c198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f233434a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f232c102040 0x7f232c1987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f233434a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f232c102a00 0x7f232c19f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f233434a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f232c10cac0 con 0x7f232c100e60 2024-04-21T08:49:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f233434a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f232c10c940 con 0x7f232c102040 2024-04-21T08:49:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f233434a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f232c10cc40 con 0x7f232c102a00 2024-04-21T08:49:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f23320e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f232c100e60 0x7f232c198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f23320e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f232c100e60 0x7f232c198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53660/0 (socket says 172.21.15.28:53660) 2024-04-21T08:49:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f23320e6640 1 -- 172.21.15.28:0/1118630186 learned_addr learned my addr 172.21.15.28:0/1118630186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f23318e5640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f232c102040 0x7f232c1987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f23328e7640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f232c102a00 0x7f232c19f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f23320e6640 1 -- 172.21.15.28:0/1118630186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f232c102a00 msgr2=0x7f232c19f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f23320e6640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f232c102a00 0x7f232c19f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f23320e6640 1 -- 172.21.15.28:0/1118630186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f232c102040 msgr2=0x7f232c1987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f23320e6640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f232c102040 0x7f232c1987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f23320e6640 1 -- 172.21.15.28:0/1118630186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f232c19feb0 con 0x7f232c100e60 2024-04-21T08:49:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f23328e7640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f232c102a00 0x7f232c19f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.930+0000 7f23318e5640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f232c102040 0x7f232c1987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.931+0000 7f23320e6640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f232c100e60 0x7f232c198290 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f232800b450 tx=0x7f232800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.931+0000 7f23237fe640 1 -- 172.21.15.28:0/1118630186 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2328015020 con 0x7f232c100e60 2024-04-21T08:49:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.931+0000 7f23237fe640 1 -- 172.21.15.28:0/1118630186 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2328011d40 con 0x7f232c100e60 2024-04-21T08:49:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.931+0000 7f233434a640 1 -- 172.21.15.28:0/1118630186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f232c1a01a0 con 0x7f232c100e60 2024-04-21T08:49:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.931+0000 7f23237fe640 1 -- 172.21.15.28:0/1118630186 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232800f340 con 0x7f232c100e60 2024-04-21T08:49:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.931+0000 7f233434a640 1 -- 172.21.15.28:0/1118630186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f232c1a0770 con 0x7f232c100e60 2024-04-21T08:49:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.932+0000 7f233434a640 1 -- 172.21.15.28:0/1118630186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f232c1a0a30 con 0x7f232c100e60 2024-04-21T08:49:09.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.933+0000 7f23237fe640 1 -- 172.21.15.28:0/1118630186 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2328018020 con 0x7f232c100e60 2024-04-21T08:49:09.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.933+0000 7f23237fe640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f23180a3bb0 0x7f23180a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:09.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.934+0000 7f23237fe640 1 -- 172.21.15.28:0/1118630186 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f23280d0760 con 0x7f232c100e60 2024-04-21T08:49:09.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.934+0000 7f23318e5640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f23180a3bb0 0x7f23180a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:09.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.934+0000 7f23318e5640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f23180a3bb0 0x7f23180a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f232c1997a0 tx=0x7f231c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:09.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:09.937+0000 7f23237fe640 1 -- 172.21.15.28:0/1118630186 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f232801f020 con 0x7f232c100e60 2024-04-21T08:49:10.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:09 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/702927479' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:10.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:09 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2431405702' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:10.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:09 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3420684960' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T08:49:10.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:09 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/702927479' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:10.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:09 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2431405702' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:10.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:09 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3420684960' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T08:49:10.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:09 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/702927479' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:10.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:09 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2431405702' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:10.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:09 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3420684960' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T08:49:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.167+0000 7f233434a640 1 -- 172.21.15.28:0/1118630186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f232c1a0c00 con 0x7f232c100e60 2024-04-21T08:49:10.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.816+0000 7f23237fe640 1 -- 172.21.15.28:0/1118630186 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f232800d040 con 0x7f232c100e60 2024-04-21T08:49:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.819+0000 7f233434a640 1 -- 172.21.15.28:0/1118630186 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f23180a3bb0 msgr2=0x7f23180a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.819+0000 7f233434a640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f23180a3bb0 0x7f23180a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f232c1997a0 tx=0x7f231c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:49:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.819+0000 7f233434a640 1 -- 172.21.15.28:0/1118630186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f232c100e60 msgr2=0x7f232c198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.819+0000 7f233434a640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f232c100e60 0x7f232c198290 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f232800b450 tx=0x7f232800b910 comp rx=0 tx=0).stop 2024-04-21T08:49:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.819+0000 7f233434a640 1 -- 172.21.15.28:0/1118630186 shutdown_connections 2024-04-21T08:49:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.819+0000 7f233434a640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f232c102a00 0x7f232c19f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.819+0000 7f233434a640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f232c102040 0x7f232c1987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.819+0000 7f233434a640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f23180a3bb0 0x7f23180a6070 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.819+0000 7f233434a640 1 --2- 172.21.15.28:0/1118630186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f232c100e60 0x7f232c198290 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.819+0000 7f233434a640 1 -- 172.21.15.28:0/1118630186 >> 172.21.15.28:0/1118630186 conn(0x7f232c0fc440 msgr2=0x7f232c0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.819+0000 7f233434a640 1 -- 172.21.15.28:0/1118630186 shutdown_connections 2024-04-21T08:49:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.819+0000 7f233434a640 1 -- 172.21.15.28:0/1118630186 wait complete. 2024-04-21T08:49:10.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-21T08:49:10.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.947+0000 7ffa711ef640 1 -- 172.21.15.28:0/2593491542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa6c107c70 msgr2=0x7ffa6c10a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:10.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.947+0000 7ffa711ef640 1 --2- 172.21.15.28:0/2593491542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa6c107c70 0x7ffa6c10a0c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009ca0 tx=0x7ffa6402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:49:10.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.947+0000 7ffa711ef640 1 -- 172.21.15.28:0/2593491542 shutdown_connections 2024-04-21T08:49:10.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.947+0000 7ffa711ef640 1 --2- 172.21.15.28:0/2593491542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa6c107c70 0x7ffa6c10a0c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.947+0000 7ffa711ef640 1 --2- 172.21.15.28:0/2593491542 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa6c105340 0x7ffa6c107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.947+0000 7ffa711ef640 1 --2- 172.21.15.28:0/2593491542 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa6c102a10 0x7ffa6c104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.947+0000 7ffa711ef640 1 -- 172.21.15.28:0/2593491542 >> 172.21.15.28:0/2593491542 conn(0x7ffa6c0fc400 msgr2=0x7ffa6c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.947+0000 7ffa711ef640 1 -- 172.21.15.28:0/2593491542 shutdown_connections 2024-04-21T08:49:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.947+0000 7ffa711ef640 1 -- 172.21.15.28:0/2593491542 wait complete. 2024-04-21T08:49:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.948+0000 7ffa711ef640 1 Processor -- start 2024-04-21T08:49:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.948+0000 7ffa711ef640 1 -- start start 2024-04-21T08:49:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa711ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa6c102a10 0x7ffa6c1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa711ef640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa6c105340 0x7ffa6c198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa711ef640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa6c107c70 0x7ffa6c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa711ef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c10dac0 con 0x7ffa6c102a10 2024-04-21T08:49:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa711ef640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c10d940 con 0x7ffa6c105340 2024-04-21T08:49:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa711ef640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c10dc40 con 0x7ffa6c107c70 2024-04-21T08:49:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa6ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa6c102a10 0x7ffa6c1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa6ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa6c102a10 0x7ffa6c1981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53684/0 (socket says 172.21.15.28:53684) 2024-04-21T08:49:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa6ad9d640 1 -- 172.21.15.28:0/1865999480 learned_addr learned my addr 172.21.15.28:0/1865999480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa6b59e640 1 --2- 172.21.15.28:0/1865999480 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa6c107c70 0x7ffa6c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa6a59c640 1 --2- 172.21.15.28:0/1865999480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa6c105340 0x7ffa6c198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa6ad9d640 1 -- 172.21.15.28:0/1865999480 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa6c107c70 msgr2=0x7ffa6c19f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa6ad9d640 1 --2- 172.21.15.28:0/1865999480 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa6c107c70 0x7ffa6c19f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa6ad9d640 1 -- 172.21.15.28:0/1865999480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa6c105340 msgr2=0x7ffa6c198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa6ad9d640 1 --2- 172.21.15.28:0/1865999480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa6c105340 0x7ffa6c198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa6ad9d640 1 -- 172.21.15.28:0/1865999480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6c19fde0 con 0x7ffa6c102a10 2024-04-21T08:49:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa6ad9d640 1 --2- 172.21.15.28:0/1865999480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa6c102a10 0x7ffa6c1981c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00d950 tx=0x7ffa5c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa53fff640 1 -- 172.21.15.28:0/1865999480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c017070 con 0x7ffa6c102a10 2024-04-21T08:49:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa711ef640 1 -- 172.21.15.28:0/1865999480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa6c1a00d0 con 0x7ffa6c102a10 2024-04-21T08:49:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa53fff640 1 -- 172.21.15.28:0/1865999480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa5c009c30 con 0x7ffa6c102a10 2024-04-21T08:49:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa53fff640 1 -- 172.21.15.28:0/1865999480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c00f620 con 0x7ffa6c102a10 2024-04-21T08:49:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.949+0000 7ffa711ef640 1 -- 172.21.15.28:0/1865999480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa6c1a0620 con 0x7ffa6c102a10 2024-04-21T08:49:10.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.950+0000 7ffa711ef640 1 -- 172.21.15.28:0/1865999480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa6c0fdef0 con 0x7ffa6c102a10 2024-04-21T08:49:10.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.952+0000 7ffa53fff640 1 -- 172.21.15.28:0/1865999480 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffa5c010460 con 0x7ffa6c102a10 2024-04-21T08:49:10.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.952+0000 7ffa53fff640 1 --2- 172.21.15.28:0/1865999480 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffa540a3bb0 0x7ffa540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:10.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.952+0000 7ffa6a59c640 1 --2- 172.21.15.28:0/1865999480 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffa540a3bb0 0x7ffa540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.952+0000 7ffa53fff640 1 -- 172.21.15.28:0/1865999480 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ffa5c0d02d0 con 0x7ffa6c102a10 2024-04-21T08:49:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.953+0000 7ffa6a59c640 1 --2- 172.21.15.28:0/1865999480 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffa540a3bb0 0x7ffa540a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffa60005fd0 tx=0x7ffa60005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:10.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:10.957+0000 7ffa53fff640 1 -- 172.21.15.28:0/1865999480 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa5c09ca10 con 0x7ffa6c102a10 2024-04-21T08:49:11.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:10 smithi152 ceph-mon[31401]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:49:11.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:10 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3420684960' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T08:49:11.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:10 smithi152 ceph-mon[31401]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:49:11.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:10 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1118630186' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-21T08:49:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:10 smithi028 ceph-mon[28003]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:49:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:10 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3420684960' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T08:49:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:10 smithi028 ceph-mon[28003]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:49:11.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:10 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1118630186' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-21T08:49:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:10 smithi171 ceph-mon[31358]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:49:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:10 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3420684960' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T08:49:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:10 smithi171 ceph-mon[31358]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:49:11.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:10 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1118630186' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-21T08:49:11.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.183+0000 7ffa711ef640 1 -- 172.21.15.28:0/1865999480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7ffa6c02d5c0 con 0x7ffa6c102a10 2024-04-21T08:49:11.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.828+0000 7ffa53fff640 1 -- 172.21.15.28:0/1865999480 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7ffa5c009de0 con 0x7ffa6c102a10 2024-04-21T08:49:11.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.831+0000 7ffa711ef640 1 -- 172.21.15.28:0/1865999480 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffa540a3bb0 msgr2=0x7ffa540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:11.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.831+0000 7ffa711ef640 1 --2- 172.21.15.28:0/1865999480 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffa540a3bb0 0x7ffa540a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffa60005fd0 tx=0x7ffa60005850 comp rx=0 tx=0).stop 2024-04-21T08:49:11.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.831+0000 7ffa711ef640 1 -- 172.21.15.28:0/1865999480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa6c102a10 msgr2=0x7ffa6c1981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:11.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.831+0000 7ffa711ef640 1 --2- 172.21.15.28:0/1865999480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa6c102a10 0x7ffa6c1981c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00d950 tx=0x7ffa5c00de10 comp rx=0 tx=0).stop 2024-04-21T08:49:11.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.831+0000 7ffa711ef640 1 -- 172.21.15.28:0/1865999480 shutdown_connections 2024-04-21T08:49:11.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.831+0000 7ffa711ef640 1 --2- 172.21.15.28:0/1865999480 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa6c107c70 0x7ffa6c19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:11.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.831+0000 7ffa711ef640 1 --2- 172.21.15.28:0/1865999480 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa6c105340 0x7ffa6c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:11.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.831+0000 7ffa711ef640 1 --2- 172.21.15.28:0/1865999480 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffa540a3bb0 0x7ffa540a6070 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:11.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.831+0000 7ffa711ef640 1 --2- 172.21.15.28:0/1865999480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa6c102a10 0x7ffa6c1981c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:11.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.831+0000 7ffa711ef640 1 -- 172.21.15.28:0/1865999480 >> 172.21.15.28:0/1865999480 conn(0x7ffa6c0fc400 msgr2=0x7ffa6c1033d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:11.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.831+0000 7ffa711ef640 1 -- 172.21.15.28:0/1865999480 shutdown_connections 2024-04-21T08:49:11.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.831+0000 7ffa711ef640 1 -- 172.21.15.28:0/1865999480 wait complete. 2024-04-21T08:49:11.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-21T08:49:11.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.960+0000 7fbd2f29a640 1 -- 172.21.15.28:0/1651559976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd281031a0 msgr2=0x7fbd28103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.960+0000 7fbd2f29a640 1 --2- 172.21.15.28:0/1651559976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd281031a0 0x7fbd28103620 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbd18009900 tx=0x7fbd1802bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.960+0000 7fbd2f29a640 1 -- 172.21.15.28:0/1651559976 shutdown_connections 2024-04-21T08:49:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.960+0000 7fbd2f29a640 1 --2- 172.21.15.28:0/1651559976 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd28103b60 0x7fbd2810a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.960+0000 7fbd2f29a640 1 --2- 172.21.15.28:0/1651559976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd281031a0 0x7fbd28103620 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.960+0000 7fbd2f29a640 1 --2- 172.21.15.28:0/1651559976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd28101fb0 0x7fbd281023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.960+0000 7fbd2f29a640 1 -- 172.21.15.28:0/1651559976 >> 172.21.15.28:0/1651559976 conn(0x7fbd280fd520 msgr2=0x7fbd280ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.960+0000 7fbd2f29a640 1 -- 172.21.15.28:0/1651559976 shutdown_connections 2024-04-21T08:49:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.960+0000 7fbd2f29a640 1 -- 172.21.15.28:0/1651559976 wait complete. 2024-04-21T08:49:11.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2f29a640 1 Processor -- start 2024-04-21T08:49:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2f29a640 1 -- start start 2024-04-21T08:49:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2f29a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd28101fb0 0x7fbd2819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2f29a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd281031a0 0x7fbd2819a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2f29a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd28103b60 0x7fbd281a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2f29a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2810daf0 con 0x7fbd28103b60 2024-04-21T08:49:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2f29a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2810d970 con 0x7fbd281031a0 2024-04-21T08:49:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2f29a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2810dc70 con 0x7fbd28101fb0 2024-04-21T08:49:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2d837640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd28103b60 0x7fbd281a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2d837640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd28103b60 0x7fbd281a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53702/0 (socket says 172.21.15.28:53702) 2024-04-21T08:49:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2d837640 1 -- 172.21.15.28:0/3126352895 learned_addr learned my addr 172.21.15.28:0/3126352895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:11.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2c835640 1 --2- 172.21.15.28:0/3126352895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd281031a0 0x7fbd2819a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2d837640 1 -- 172.21.15.28:0/3126352895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd28101fb0 msgr2=0x7fbd2819a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2d036640 1 --2- 172.21.15.28:0/3126352895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd28101fb0 0x7fbd2819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2d837640 1 --2- 172.21.15.28:0/3126352895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd28101fb0 0x7fbd2819a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2d837640 1 -- 172.21.15.28:0/3126352895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd281031a0 msgr2=0x7fbd2819a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2d837640 1 --2- 172.21.15.28:0/3126352895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd281031a0 0x7fbd2819a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.961+0000 7fbd2d837640 1 -- 172.21.15.28:0/3126352895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd281a1fa0 con 0x7fbd28103b60 2024-04-21T08:49:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.962+0000 7fbd2d837640 1 --2- 172.21.15.28:0/3126352895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd28103b60 0x7fbd281a1930 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbd2000cce0 tx=0x7fbd2000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.962+0000 7fbd1e7fc640 1 -- 172.21.15.28:0/3126352895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2000f450 con 0x7fbd28103b60 2024-04-21T08:49:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.962+0000 7fbd2f29a640 1 -- 172.21.15.28:0/3126352895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd281a2290 con 0x7fbd28103b60 2024-04-21T08:49:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.962+0000 7fbd1e7fc640 1 -- 172.21.15.28:0/3126352895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd2000faa0 con 0x7fbd28103b60 2024-04-21T08:49:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.962+0000 7fbd1e7fc640 1 -- 172.21.15.28:0/3126352895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20018b70 con 0x7fbd28103b60 2024-04-21T08:49:11.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.962+0000 7fbd2f29a640 1 -- 172.21.15.28:0/3126352895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd281a2760 con 0x7fbd28103b60 2024-04-21T08:49:11.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.963+0000 7fbd2f29a640 1 -- 172.21.15.28:0/3126352895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd281a2a20 con 0x7fbd28103b60 2024-04-21T08:49:11.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.964+0000 7fbd1e7fc640 1 -- 172.21.15.28:0/3126352895 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbd20018d10 con 0x7fbd28103b60 2024-04-21T08:49:11.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.965+0000 7fbd1e7fc640 1 --2- 172.21.15.28:0/3126352895 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbd140a3bb0 0x7fbd140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:11.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.965+0000 7fbd1e7fc640 1 -- 172.21.15.28:0/3126352895 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbd200d0ab0 con 0x7fbd28103b60 2024-04-21T08:49:11.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.965+0000 7fbd2d036640 1 --2- 172.21.15.28:0/3126352895 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbd140a3bb0 0x7fbd140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:11.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.965+0000 7fbd2d036640 1 --2- 172.21.15.28:0/3126352895 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbd140a3bb0 0x7fbd140a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbd28103000 tx=0x7fbd24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:11.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:11.968+0000 7fbd1e7fc640 1 -- 172.21.15.28:0/3126352895 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbd20014030 con 0x7fbd28103b60 2024-04-21T08:49:12.104 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 08:49:11 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-b[62223]: 2024-04-21T08:49:11.910+0000 7f5feb533700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-21T08:49:12.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:11 smithi152 ceph-mon[31401]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-21T08:49:12.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:11 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1118630186' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-21T08:49:12.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:11 smithi152 ceph-mon[31401]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-21T08:49:12.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:11 smithi152 ceph-mon[31401]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:49:12.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:11 smithi152 ceph-mon[31401]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-21T08:49:12.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:11 smithi152 ceph-mon[31401]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-21T08:49:12.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:11 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1865999480' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T08:49:12.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:11 smithi028 ceph-mon[28003]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-21T08:49:12.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:11 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1118630186' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-21T08:49:12.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:11 smithi028 ceph-mon[28003]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-21T08:49:12.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:11 smithi028 ceph-mon[28003]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:49:12.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:11 smithi028 ceph-mon[28003]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-21T08:49:12.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:11 smithi028 ceph-mon[28003]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-21T08:49:12.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:11 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1865999480' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T08:49:12.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:11 smithi171 ceph-mon[31358]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-21T08:49:12.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:11 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1118630186' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-21T08:49:12.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:11 smithi171 ceph-mon[31358]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-21T08:49:12.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:11 smithi171 ceph-mon[31358]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:49:12.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:11 smithi171 ceph-mon[31358]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-21T08:49:12.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:11 smithi171 ceph-mon[31358]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-21T08:49:12.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:11 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1865999480' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T08:49:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.196+0000 7fbd2f29a640 1 -- 172.21.15.28:0/3126352895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fbd281a2d70 con 0x7fbd28103b60 2024-04-21T08:49:12.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.835+0000 7fbd1e7fc640 1 -- 172.21.15.28:0/3126352895 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7fbd2009d250 con 0x7fbd28103b60 2024-04-21T08:49:12.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.838+0000 7fbd2f29a640 1 -- 172.21.15.28:0/3126352895 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbd140a3bb0 msgr2=0x7fbd140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:12.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.838+0000 7fbd2f29a640 1 --2- 172.21.15.28:0/3126352895 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbd140a3bb0 0x7fbd140a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbd28103000 tx=0x7fbd24005f90 comp rx=0 tx=0).stop 2024-04-21T08:49:12.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.838+0000 7fbd2f29a640 1 -- 172.21.15.28:0/3126352895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd28103b60 msgr2=0x7fbd281a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:12.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.838+0000 7fbd2f29a640 1 --2- 172.21.15.28:0/3126352895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd28103b60 0x7fbd281a1930 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbd2000cce0 tx=0x7fbd2000a620 comp rx=0 tx=0).stop 2024-04-21T08:49:12.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.838+0000 7fbd2f29a640 1 -- 172.21.15.28:0/3126352895 shutdown_connections 2024-04-21T08:49:12.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.838+0000 7fbd2f29a640 1 --2- 172.21.15.28:0/3126352895 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbd140a3bb0 0x7fbd140a6070 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:12.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.838+0000 7fbd2f29a640 1 --2- 172.21.15.28:0/3126352895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd28103b60 0x7fbd281a1930 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:12.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.838+0000 7fbd2f29a640 1 --2- 172.21.15.28:0/3126352895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd281031a0 0x7fbd2819a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:12.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.838+0000 7fbd2f29a640 1 --2- 172.21.15.28:0/3126352895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd28101fb0 0x7fbd2819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:12.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.838+0000 7fbd2f29a640 1 -- 172.21.15.28:0/3126352895 >> 172.21.15.28:0/3126352895 conn(0x7fbd280fd520 msgr2=0x7fbd28074180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:12.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.838+0000 7fbd2f29a640 1 -- 172.21.15.28:0/3126352895 shutdown_connections 2024-04-21T08:49:12.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.838+0000 7fbd2f29a640 1 -- 172.21.15.28:0/3126352895 wait complete. 2024-04-21T08:49:12.853 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-21T08:49:12.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-21T08:49:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.965+0000 7fa7ca5e6640 1 -- 172.21.15.28:0/2198524847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7c41031a0 msgr2=0x7fa7c4103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.965+0000 7fa7ca5e6640 1 --2- 172.21.15.28:0/2198524847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7c41031a0 0x7fa7c4103620 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4009900 tx=0x7fa7b402bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.965+0000 7fa7ca5e6640 1 -- 172.21.15.28:0/2198524847 shutdown_connections 2024-04-21T08:49:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.965+0000 7fa7ca5e6640 1 --2- 172.21.15.28:0/2198524847 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa7c4103b60 0x7fa7c410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.965+0000 7fa7ca5e6640 1 --2- 172.21.15.28:0/2198524847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7c41031a0 0x7fa7c4103620 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.965+0000 7fa7ca5e6640 1 --2- 172.21.15.28:0/2198524847 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7c4101fb0 0x7fa7c41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.965+0000 7fa7ca5e6640 1 -- 172.21.15.28:0/2198524847 >> 172.21.15.28:0/2198524847 conn(0x7fa7c40fd520 msgr2=0x7fa7c40ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:12.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.965+0000 7fa7ca5e6640 1 -- 172.21.15.28:0/2198524847 shutdown_connections 2024-04-21T08:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.965+0000 7fa7ca5e6640 1 -- 172.21.15.28:0/2198524847 wait complete. 2024-04-21T08:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.965+0000 7fa7ca5e6640 1 Processor -- start 2024-04-21T08:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.965+0000 7fa7ca5e6640 1 -- start start 2024-04-21T08:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7ca5e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7c4101fb0 0x7fa7c41981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7ca5e6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7c41031a0 0x7fa7c41986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7ca5e6640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa7c4103b60 0x7fa7c419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7ca5e6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c410daf0 con 0x7fa7c4101fb0 2024-04-21T08:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7ca5e6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c410d970 con 0x7fa7c41031a0 2024-04-21T08:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7ca5e6640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c410dc70 con 0x7fa7c4103b60 2024-04-21T08:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7c4101fb0 0x7fa7c41981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7c4101fb0 0x7fa7c41981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53718/0 (socket says 172.21.15.28:53718) 2024-04-21T08:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7c3fff640 1 -- 172.21.15.28:0/3898043415 learned_addr learned my addr 172.21.15.28:0/3898043415 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7c8b83640 1 --2- 172.21.15.28:0/3898043415 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa7c4103b60 0x7fa7c419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7c37fe640 1 --2- 172.21.15.28:0/3898043415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7c41031a0 0x7fa7c41986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7c3fff640 1 -- 172.21.15.28:0/3898043415 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa7c4103b60 msgr2=0x7fa7c419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7c3fff640 1 --2- 172.21.15.28:0/3898043415 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa7c4103b60 0x7fa7c419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7c3fff640 1 -- 172.21.15.28:0/3898043415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7c41031a0 msgr2=0x7fa7c41986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7c3fff640 1 --2- 172.21.15.28:0/3898043415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7c41031a0 0x7fa7c41986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7c3fff640 1 -- 172.21.15.28:0/3898043415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c419fdc0 con 0x7fa7c4101fb0 2024-04-21T08:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.966+0000 7fa7c3fff640 1 --2- 172.21.15.28:0/3898043415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7c4101fb0 0x7fa7c41981a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00e9c0 tx=0x7fa7ac00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.967+0000 7fa7c17fa640 1 -- 172.21.15.28:0/3898043415 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac00cd80 con 0x7fa7c4101fb0 2024-04-21T08:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.967+0000 7fa7c17fa640 1 -- 172.21.15.28:0/3898043415 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7ac0035f0 con 0x7fa7c4101fb0 2024-04-21T08:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.967+0000 7fa7ca5e6640 1 -- 172.21.15.28:0/3898043415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7c41a00b0 con 0x7fa7c4101fb0 2024-04-21T08:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.967+0000 7fa7c17fa640 1 -- 172.21.15.28:0/3898043415 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac010690 con 0x7fa7c4101fb0 2024-04-21T08:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.967+0000 7fa7ca5e6640 1 -- 172.21.15.28:0/3898043415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7c41a05d0 con 0x7fa7c4101fb0 2024-04-21T08:49:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.968+0000 7fa7ca5e6640 1 -- 172.21.15.28:0/3898043415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7c4109350 con 0x7fa7c4101fb0 2024-04-21T08:49:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.969+0000 7fa7c17fa640 1 -- 172.21.15.28:0/3898043415 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa7ac0037a0 con 0x7fa7c4101fb0 2024-04-21T08:49:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.969+0000 7fa7c17fa640 1 --2- 172.21.15.28:0/3898043415 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa7b00a3bb0 0x7fa7b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.970+0000 7fa7c17fa640 1 -- 172.21.15.28:0/3898043415 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa7ac0d09a0 con 0x7fa7c4101fb0 2024-04-21T08:49:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.970+0000 7fa7c37fe640 1 --2- 172.21.15.28:0/3898043415 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa7b00a3bb0 0x7fa7b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.970+0000 7fa7c37fe640 1 --2- 172.21.15.28:0/3898043415 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa7b00a3bb0 0x7fa7b00a6070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4009230 tx=0x7fa7b4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:12.974+0000 7fa7c17fa640 1 -- 172.21.15.28:0/3898043415 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa7ac09d140 con 0x7fa7c4101fb0 2024-04-21T08:49:13.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:12 smithi152 ceph-mon[31401]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-04-21T08:49:13.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:12 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1865999480' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T08:49:13.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:12 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] up:active 2024-04-21T08:49:13.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:12 smithi152 ceph-mon[31401]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-21T08:49:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:12 smithi152 ceph-mon[31401]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-21T08:49:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:12 smithi152 ceph-mon[31401]: Cluster is now healthy 2024-04-21T08:49:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:12 smithi152 ceph-mon[31401]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-21T08:49:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:12 smithi152 ceph-mon[31401]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-21T08:49:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:12 smithi152 ceph-mon[31401]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-21T08:49:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:12 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3126352895' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T08:49:13.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:12 smithi028 ceph-mon[28003]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-04-21T08:49:13.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:12 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1865999480' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T08:49:13.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:12 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] up:active 2024-04-21T08:49:13.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:12 smithi028 ceph-mon[28003]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-21T08:49:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:12 smithi028 ceph-mon[28003]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-21T08:49:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:12 smithi028 ceph-mon[28003]: Cluster is now healthy 2024-04-21T08:49:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:12 smithi028 ceph-mon[28003]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-21T08:49:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:12 smithi028 ceph-mon[28003]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-21T08:49:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:12 smithi028 ceph-mon[28003]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-21T08:49:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:12 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3126352895' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T08:49:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:12 smithi171 ceph-mon[31358]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-04-21T08:49:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:12 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1865999480' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T08:49:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:12 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] up:active 2024-04-21T08:49:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:12 smithi171 ceph-mon[31358]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-21T08:49:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:12 smithi171 ceph-mon[31358]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-21T08:49:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:12 smithi171 ceph-mon[31358]: Cluster is now healthy 2024-04-21T08:49:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:12 smithi171 ceph-mon[31358]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-21T08:49:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:12 smithi171 ceph-mon[31358]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-21T08:49:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:12 smithi171 ceph-mon[31358]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-21T08:49:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:12 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3126352895' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T08:49:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.187+0000 7fa7ca5e6640 1 -- 172.21.15.28:0/3898043415 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fa7c4062390 con 0x7fa7b00a3bb0 2024-04-21T08:49:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.211+0000 7fa7c17fa640 1 -- 172.21.15.28:0/3898043415 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa7c4062390 con 0x7fa7b00a3bb0 2024-04-21T08:49:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.213+0000 7fa7ca5e6640 1 -- 172.21.15.28:0/3898043415 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa7b00a3bb0 msgr2=0x7fa7b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.213+0000 7fa7ca5e6640 1 --2- 172.21.15.28:0/3898043415 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa7b00a3bb0 0x7fa7b00a6070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4009230 tx=0x7fa7b4005fd0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.213+0000 7fa7ca5e6640 1 -- 172.21.15.28:0/3898043415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7c4101fb0 msgr2=0x7fa7c41981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.213+0000 7fa7ca5e6640 1 --2- 172.21.15.28:0/3898043415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7c4101fb0 0x7fa7c41981a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00e9c0 tx=0x7fa7ac00ee80 comp rx=0 tx=0).stop 2024-04-21T08:49:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.213+0000 7fa7ca5e6640 1 -- 172.21.15.28:0/3898043415 shutdown_connections 2024-04-21T08:49:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.213+0000 7fa7ca5e6640 1 --2- 172.21.15.28:0/3898043415 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa7c4103b60 0x7fa7c419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.213+0000 7fa7ca5e6640 1 --2- 172.21.15.28:0/3898043415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7c41031a0 0x7fa7c41986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.213+0000 7fa7ca5e6640 1 --2- 172.21.15.28:0/3898043415 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fa7b00a3bb0 0x7fa7b00a6070 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.213+0000 7fa7ca5e6640 1 --2- 172.21.15.28:0/3898043415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7c4101fb0 0x7fa7c41981a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.213+0000 7fa7ca5e6640 1 -- 172.21.15.28:0/3898043415 >> 172.21.15.28:0/3898043415 conn(0x7fa7c40fd520 msgr2=0x7fa7c4106520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.213+0000 7fa7ca5e6640 1 -- 172.21.15.28:0/3898043415 shutdown_connections 2024-04-21T08:49:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.213+0000 7fa7ca5e6640 1 -- 172.21.15.28:0/3898043415 wait complete. 2024-04-21T08:49:13.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-21T08:49:13.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.332+0000 7f0d85ad2640 1 -- 172.21.15.28:0/2550591901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80105360 msgr2=0x7f0d80107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.332+0000 7f0d85ad2640 1 --2- 172.21.15.28:0/2550591901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80105360 0x7f0d80107750 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0d74009900 tx=0x7f0d7402bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:13.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.332+0000 7f0d85ad2640 1 -- 172.21.15.28:0/2550591901 shutdown_connections 2024-04-21T08:49:13.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.332+0000 7f0d85ad2640 1 --2- 172.21.15.28:0/2550591901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d80107c90 0x7f0d8010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.332+0000 7f0d85ad2640 1 --2- 172.21.15.28:0/2550591901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80105360 0x7f0d80107750 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.332+0000 7f0d85ad2640 1 --2- 172.21.15.28:0/2550591901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d80102a30 0x7f0d80104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.332+0000 7f0d85ad2640 1 -- 172.21.15.28:0/2550591901 >> 172.21.15.28:0/2550591901 conn(0x7f0d800fc440 msgr2=0x7f0d800fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.332+0000 7f0d85ad2640 1 -- 172.21.15.28:0/2550591901 shutdown_connections 2024-04-21T08:49:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.332+0000 7f0d85ad2640 1 -- 172.21.15.28:0/2550591901 wait complete. 2024-04-21T08:49:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d85ad2640 1 Processor -- start 2024-04-21T08:49:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d85ad2640 1 -- start start 2024-04-21T08:49:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d85ad2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80102a30 0x7f0d801981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d85ad2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d80105360 0x7f0d801986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d85ad2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d80107c90 0x7f0d8019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d85ad2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8010daf0 con 0x7f0d80102a30 2024-04-21T08:49:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d85ad2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8010d970 con 0x7f0d80105360 2024-04-21T08:49:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d85ad2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8010dc70 con 0x7f0d80107c90 2024-04-21T08:49:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d7f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80102a30 0x7f0d801981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d7f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80102a30 0x7f0d801981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53756/0 (socket says 172.21.15.28:53756) 2024-04-21T08:49:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d7f7fe640 1 -- 172.21.15.28:0/3884942787 learned_addr learned my addr 172.21.15.28:0/3884942787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d7effd640 1 --2- 172.21.15.28:0/3884942787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d80105360 0x7f0d801986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d7ffff640 1 --2- 172.21.15.28:0/3884942787 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d80107c90 0x7f0d8019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d7f7fe640 1 -- 172.21.15.28:0/3884942787 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d80107c90 msgr2=0x7f0d8019f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d7f7fe640 1 --2- 172.21.15.28:0/3884942787 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d80107c90 0x7f0d8019f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d7f7fe640 1 -- 172.21.15.28:0/3884942787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d80105360 msgr2=0x7f0d801986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.333+0000 7f0d7f7fe640 1 --2- 172.21.15.28:0/3884942787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d80105360 0x7f0d801986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.334+0000 7f0d7f7fe640 1 -- 172.21.15.28:0/3884942787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d8019fdd0 con 0x7f0d80102a30 2024-04-21T08:49:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.334+0000 7f0d7ffff640 1 --2- 172.21.15.28:0/3884942787 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d80107c90 0x7f0d8019f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.334+0000 7f0d7f7fe640 1 --2- 172.21.15.28:0/3884942787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80102a30 0x7f0d801981b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000e9f0 tx=0x7f0d7000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.334+0000 7f0d7cff9640 1 -- 172.21.15.28:0/3884942787 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7000cd80 con 0x7f0d80102a30 2024-04-21T08:49:13.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.334+0000 7f0d7cff9640 1 -- 172.21.15.28:0/3884942787 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d700108e0 con 0x7f0d80102a30 2024-04-21T08:49:13.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.334+0000 7f0d85ad2640 1 -- 172.21.15.28:0/3884942787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d801a00c0 con 0x7f0d80102a30 2024-04-21T08:49:13.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.334+0000 7f0d7cff9640 1 -- 172.21.15.28:0/3884942787 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d70018980 con 0x7f0d80102a30 2024-04-21T08:49:13.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.334+0000 7f0d85ad2640 1 -- 172.21.15.28:0/3884942787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d801a05e0 con 0x7f0d80102a30 2024-04-21T08:49:13.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.335+0000 7f0d85ad2640 1 -- 172.21.15.28:0/3884942787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d800fdf30 con 0x7f0d80102a30 2024-04-21T08:49:13.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.336+0000 7f0d7cff9640 1 -- 172.21.15.28:0/3884942787 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0d70010430 con 0x7f0d80102a30 2024-04-21T08:49:13.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.337+0000 7f0d7cff9640 1 --2- 172.21.15.28:0/3884942787 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0d680a3bb0 0x7f0d680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.337+0000 7f0d7effd640 1 --2- 172.21.15.28:0/3884942787 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0d680a3bb0 0x7f0d680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.337+0000 7f0d7cff9640 1 -- 172.21.15.28:0/3884942787 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0d700d0a20 con 0x7f0d80102a30 2024-04-21T08:49:13.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.337+0000 7f0d7effd640 1 --2- 172.21.15.28:0/3884942787 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0d680a3bb0 0x7f0d680a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0d74005ab0 tx=0x7f0d74005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.341+0000 7f0d7cff9640 1 -- 172.21.15.28:0/3884942787 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d7009d160 con 0x7f0d80102a30 2024-04-21T08:49:13.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.559+0000 7f0d85ad2640 1 -- 172.21.15.28:0/3884942787 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f0d80062420 con 0x7f0d680a3bb0 2024-04-21T08:49:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.582+0000 7f0d7cff9640 1 -- 172.21.15.28:0/3884942787 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0d80062420 con 0x7f0d680a3bb0 2024-04-21T08:49:13.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.584+0000 7f0d85ad2640 1 -- 172.21.15.28:0/3884942787 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0d680a3bb0 msgr2=0x7f0d680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.584+0000 7f0d85ad2640 1 --2- 172.21.15.28:0/3884942787 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0d680a3bb0 0x7f0d680a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0d74005ab0 tx=0x7f0d74005fd0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.584+0000 7f0d85ad2640 1 -- 172.21.15.28:0/3884942787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80102a30 msgr2=0x7f0d801981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.584+0000 7f0d85ad2640 1 --2- 172.21.15.28:0/3884942787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80102a30 0x7f0d801981b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000e9f0 tx=0x7f0d7000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.584+0000 7f0d85ad2640 1 -- 172.21.15.28:0/3884942787 shutdown_connections 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.584+0000 7f0d85ad2640 1 --2- 172.21.15.28:0/3884942787 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d80107c90 0x7f0d8019f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.585+0000 7f0d85ad2640 1 --2- 172.21.15.28:0/3884942787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d80105360 0x7f0d801986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.585+0000 7f0d85ad2640 1 --2- 172.21.15.28:0/3884942787 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0d680a3bb0 0x7f0d680a6070 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.585+0000 7f0d85ad2640 1 --2- 172.21.15.28:0/3884942787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80102a30 0x7f0d801981b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.585+0000 7f0d85ad2640 1 -- 172.21.15.28:0/3884942787 >> 172.21.15.28:0/3884942787 conn(0x7f0d800fc440 msgr2=0x7f0d80103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.585+0000 7f0d85ad2640 1 -- 172.21.15.28:0/3884942787 shutdown_connections 2024-04-21T08:49:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.585+0000 7f0d85ad2640 1 -- 172.21.15.28:0/3884942787 wait complete. 2024-04-21T08:49:13.595 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:49:13 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:49:13.192+0000 7f331d5d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:49:13.595 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:49:13 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:49:13.192+0000 7f331d5d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:49:13.595 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:49:13 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:49:13.192+0000 7f331d5d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:49:13.596 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:49:13 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:49:13.192+0000 7f331d5d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:49:13.596 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:49:13 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:49:13.192+0000 7f331d5d5700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:49:13.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-21T08:49:13.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.703+0000 7f311faeb640 1 -- 172.21.15.28:0/1011497471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3118103b70 msgr2=0x7f311810a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.703+0000 7f311faeb640 1 --2- 172.21.15.28:0/1011497471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3118103b70 0x7f311810a400 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3110009ca0 tx=0x7f311002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.703+0000 7f311faeb640 1 -- 172.21.15.28:0/1011497471 shutdown_connections 2024-04-21T08:49:13.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.703+0000 7f311faeb640 1 --2- 172.21.15.28:0/1011497471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3118103b70 0x7f311810a400 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.703+0000 7f311faeb640 1 --2- 172.21.15.28:0/1011497471 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31181031b0 0x7f3118103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.703+0000 7f311faeb640 1 --2- 172.21.15.28:0/1011497471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3118101fc0 0x7f31181023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.703+0000 7f311faeb640 1 -- 172.21.15.28:0/1011497471 >> 172.21.15.28:0/1011497471 conn(0x7f31180fd510 msgr2=0x7f31180ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.703+0000 7f311faeb640 1 -- 172.21.15.28:0/1011497471 shutdown_connections 2024-04-21T08:49:13.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.703+0000 7f311faeb640 1 -- 172.21.15.28:0/1011497471 wait complete. 2024-04-21T08:49:13.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.704+0000 7f311faeb640 1 Processor -- start 2024-04-21T08:49:13.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.704+0000 7f311faeb640 1 -- start start 2024-04-21T08:49:13.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.704+0000 7f311faeb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3118101fc0 0x7f3118198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.704+0000 7f311faeb640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31181031b0 0x7f3118198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.704+0000 7f311faeb640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3118103b70 0x7f311819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.704+0000 7f311faeb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f311810dac0 con 0x7f3118101fc0 2024-04-21T08:49:13.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.704+0000 7f311faeb640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f311810d940 con 0x7f31181031b0 2024-04-21T08:49:13.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.704+0000 7f311faeb640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f311810dc40 con 0x7f3118103b70 2024-04-21T08:49:13.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.704+0000 7f311d887640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3118101fc0 0x7f3118198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.704+0000 7f311d887640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3118101fc0 0x7f3118198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53774/0 (socket says 172.21.15.28:53774) 2024-04-21T08:49:13.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.704+0000 7f311d887640 1 -- 172.21.15.28:0/349519574 learned_addr learned my addr 172.21.15.28:0/349519574 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:13.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.704+0000 7f311d086640 1 --2- 172.21.15.28:0/349519574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31181031b0 0x7f3118198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.704+0000 7f311e088640 1 --2- 172.21.15.28:0/349519574 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3118103b70 0x7f311819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.705+0000 7f311d887640 1 -- 172.21.15.28:0/349519574 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3118103b70 msgr2=0x7f311819f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.705+0000 7f311d887640 1 --2- 172.21.15.28:0/349519574 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3118103b70 0x7f311819f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.705+0000 7f311d887640 1 -- 172.21.15.28:0/349519574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31181031b0 msgr2=0x7f3118198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.705+0000 7f311d887640 1 --2- 172.21.15.28:0/349519574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31181031b0 0x7f3118198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.705+0000 7f311d887640 1 -- 172.21.15.28:0/349519574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311819fd50 con 0x7f3118101fc0 2024-04-21T08:49:13.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.705+0000 7f311d887640 1 --2- 172.21.15.28:0/349519574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3118101fc0 0x7f3118198130 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f310c00e9c0 tx=0x7f310c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.705+0000 7f310affd640 1 -- 172.21.15.28:0/349519574 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c00cd80 con 0x7f3118101fc0 2024-04-21T08:49:13.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.705+0000 7f310affd640 1 -- 172.21.15.28:0/349519574 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f310c0035f0 con 0x7f3118101fc0 2024-04-21T08:49:13.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.705+0000 7f311faeb640 1 -- 172.21.15.28:0/349519574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f311819ffe0 con 0x7f3118101fc0 2024-04-21T08:49:13.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.705+0000 7f310affd640 1 -- 172.21.15.28:0/349519574 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310c010690 con 0x7f3118101fc0 2024-04-21T08:49:13.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.705+0000 7f311faeb640 1 -- 172.21.15.28:0/349519574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31181a04f0 con 0x7f3118101fc0 2024-04-21T08:49:13.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.706+0000 7f311faeb640 1 -- 172.21.15.28:0/349519574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3118109360 con 0x7f3118101fc0 2024-04-21T08:49:13.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.707+0000 7f310affd640 1 -- 172.21.15.28:0/349519574 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f310c0037a0 con 0x7f3118101fc0 2024-04-21T08:49:13.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.708+0000 7f310affd640 1 --2- 172.21.15.28:0/349519574 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f31040a3c30 0x7f31040a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:13.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.708+0000 7f311d086640 1 --2- 172.21.15.28:0/349519574 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f31040a3c30 0x7f31040a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:13.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.708+0000 7f310affd640 1 -- 172.21.15.28:0/349519574 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f310c0d0ae0 con 0x7f3118101fc0 2024-04-21T08:49:13.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.708+0000 7f311d086640 1 --2- 172.21.15.28:0/349519574 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f31040a3c30 0x7f31040a60f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3118199640 tx=0x7f311400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.712+0000 7f310affd640 1 -- 172.21.15.28:0/349519574 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f310c09d220 con 0x7f3118101fc0 2024-04-21T08:49:13.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:13 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3126352895' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T08:49:13.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:13 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] up:active 2024-04-21T08:49:13.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:13 smithi028 ceph-mon[28003]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-21T08:49:13.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:13 smithi028 ceph-mon[28003]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:49:13.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:13 smithi028 ceph-mon[28003]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:49:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.859+0000 7f310affd640 1 -- 172.21.15.28:0/349519574 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f310c099190 con 0x7f3118101fc0 2024-04-21T08:49:13.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.934+0000 7f311faeb640 1 -- 172.21.15.28:0/349519574 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f3118062390 con 0x7f31040a3c30 2024-04-21T08:49:13.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.939+0000 7f310affd640 1 -- 172.21.15.28:0/349519574 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f3118062390 con 0x7f31040a3c30 2024-04-21T08:49:13.940 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_0/cf0bbff0-e4bc-4863-8162-661fbacedc10 2024-04-21T08:49:13.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.941+0000 7f311faeb640 1 -- 172.21.15.28:0/349519574 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f31040a3c30 msgr2=0x7f31040a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.941+0000 7f311faeb640 1 --2- 172.21.15.28:0/349519574 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f31040a3c30 0x7f31040a60f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3118199640 tx=0x7f311400b390 comp rx=0 tx=0).stop 2024-04-21T08:49:13.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.941+0000 7f311faeb640 1 -- 172.21.15.28:0/349519574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3118101fc0 msgr2=0x7f3118198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.941+0000 7f311faeb640 1 --2- 172.21.15.28:0/349519574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3118101fc0 0x7f3118198130 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f310c00e9c0 tx=0x7f310c00ee80 comp rx=0 tx=0).stop 2024-04-21T08:49:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.941+0000 7f311faeb640 1 -- 172.21.15.28:0/349519574 shutdown_connections 2024-04-21T08:49:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.941+0000 7f311faeb640 1 --2- 172.21.15.28:0/349519574 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3118103b70 0x7f311819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.941+0000 7f311faeb640 1 --2- 172.21.15.28:0/349519574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31181031b0 0x7f3118198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.941+0000 7f311faeb640 1 --2- 172.21.15.28:0/349519574 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f31040a3c30 0x7f31040a60f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.941+0000 7f311faeb640 1 --2- 172.21.15.28:0/349519574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3118101fc0 0x7f3118198130 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.941+0000 7f311faeb640 1 -- 172.21.15.28:0/349519574 >> 172.21.15.28:0/349519574 conn(0x7f31180fd510 msgr2=0x7f3118106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.941+0000 7f311faeb640 1 -- 172.21.15.28:0/349519574 shutdown_connections 2024-04-21T08:49:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:13.941+0000 7f311faeb640 1 -- 172.21.15.28:0/349519574 wait complete. 2024-04-21T08:49:13.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-21T08:49:14.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.059+0000 7f62116a8640 1 -- 172.21.15.28:0/3920152883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f620c100e90 msgr2=0x7f620c1012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.059+0000 7f62116a8640 1 --2- 172.21.15.28:0/3920152883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f620c100e90 0x7f620c1012b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6204009900 tx=0x7f620402bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.059+0000 7f62116a8640 1 -- 172.21.15.28:0/3920152883 shutdown_connections 2024-04-21T08:49:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.059+0000 7f62116a8640 1 --2- 172.21.15.28:0/3920152883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f620c1029a0 0x7f620c1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.059+0000 7f62116a8640 1 --2- 172.21.15.28:0/3920152883 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f620c101fe0 0x7f620c102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.059+0000 7f62116a8640 1 --2- 172.21.15.28:0/3920152883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f620c100e90 0x7f620c1012b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.059+0000 7f62116a8640 1 -- 172.21.15.28:0/3920152883 >> 172.21.15.28:0/3920152883 conn(0x7f620c0fc400 msgr2=0x7f620c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.060+0000 7f62116a8640 1 -- 172.21.15.28:0/3920152883 shutdown_connections 2024-04-21T08:49:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.060+0000 7f62116a8640 1 -- 172.21.15.28:0/3920152883 wait complete. 2024-04-21T08:49:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.060+0000 7f62116a8640 1 Processor -- start 2024-04-21T08:49:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.060+0000 7f62116a8640 1 -- start start 2024-04-21T08:49:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.060+0000 7f62116a8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f620c100e90 0x7f620c198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.060+0000 7f62116a8640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f620c101fe0 0x7f620c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.060+0000 7f62116a8640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f620c1029a0 0x7f620c19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.060+0000 7f62116a8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f620c10c9c0 con 0x7f620c100e90 2024-04-21T08:49:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.060+0000 7f62116a8640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f620c10c840 con 0x7f620c1029a0 2024-04-21T08:49:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.060+0000 7f62116a8640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f620c10cb40 con 0x7f620c101fe0 2024-04-21T08:49:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f620affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f620c100e90 0x7f620c198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f620affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f620c100e90 0x7f620c198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41784/0 (socket says 172.21.15.28:41784) 2024-04-21T08:49:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f620affd640 1 -- 172.21.15.28:0/2031922034 learned_addr learned my addr 172.21.15.28:0/2031922034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f620b7fe640 1 --2- 172.21.15.28:0/2031922034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f620c1029a0 0x7f620c19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f620affd640 1 -- 172.21.15.28:0/2031922034 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f620c101fe0 msgr2=0x7f620c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f620affd640 1 --2- 172.21.15.28:0/2031922034 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f620c101fe0 0x7f620c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f620affd640 1 -- 172.21.15.28:0/2031922034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f620c1029a0 msgr2=0x7f620c19f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f620affd640 1 --2- 172.21.15.28:0/2031922034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f620c1029a0 0x7f620c19f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f620affd640 1 -- 172.21.15.28:0/2031922034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f620c19fda0 con 0x7f620c100e90 2024-04-21T08:49:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f620b7fe640 1 --2- 172.21.15.28:0/2031922034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f620c1029a0 0x7f620c19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f620affd640 1 --2- 172.21.15.28:0/2031922034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f620c100e90 0x7f620c198180 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6204006010 tx=0x7f6204004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f6203fff640 1 -- 172.21.15.28:0/2031922034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6204043020 con 0x7f620c100e90 2024-04-21T08:49:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f6203fff640 1 -- 172.21.15.28:0/2031922034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6204004440 con 0x7f620c100e90 2024-04-21T08:49:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f62116a8640 1 -- 172.21.15.28:0/2031922034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f620c1a0030 con 0x7f620c100e90 2024-04-21T08:49:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f6203fff640 1 -- 172.21.15.28:0/2031922034 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6204003bf0 con 0x7f620c100e90 2024-04-21T08:49:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.061+0000 7f62116a8640 1 -- 172.21.15.28:0/2031922034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f620c1a0580 con 0x7f620c100e90 2024-04-21T08:49:14.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.062+0000 7f62116a8640 1 -- 172.21.15.28:0/2031922034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f620c1a07e0 con 0x7f620c100e90 2024-04-21T08:49:14.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.063+0000 7f6203fff640 1 -- 172.21.15.28:0/2031922034 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6204046020 con 0x7f620c100e90 2024-04-21T08:49:14.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.064+0000 7f6203fff640 1 --2- 172.21.15.28:0/2031922034 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f61f80a3c40 0x7f61f80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.064+0000 7f620a7fc640 1 --2- 172.21.15.28:0/2031922034 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f61f80a3c40 0x7f61f80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.064+0000 7f6203fff640 1 -- 172.21.15.28:0/2031922034 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f62040f2b70 con 0x7f620c100e90 2024-04-21T08:49:14.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.064+0000 7f620a7fc640 1 --2- 172.21.15.28:0/2031922034 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f61f80a3c40 0x7f61f80a6100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f620c199690 tx=0x7f61f4005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:14.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.068+0000 7f6203fff640 1 -- 172.21.15.28:0/2031922034 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f620404d020 con 0x7f620c100e90 2024-04-21T08:49:14.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:13 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3126352895' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T08:49:14.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:13 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] up:active 2024-04-21T08:49:14.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:13 smithi152 ceph-mon[31401]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-21T08:49:14.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:13 smithi152 ceph-mon[31401]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:49:14.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:13 smithi152 ceph-mon[31401]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:49:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:13 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3126352895' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T08:49:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:13 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] up:active 2024-04-21T08:49:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:13 smithi171 ceph-mon[31358]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-21T08:49:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:13 smithi171 ceph-mon[31358]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:49:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:13 smithi171 ceph-mon[31358]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:49:14.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.286+0000 7f62116a8640 1 -- 172.21.15.28:0/2031922034 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f620c0fe610 con 0x7f61f80a3c40 2024-04-21T08:49:14.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.311+0000 7f6203fff640 1 -- 172.21.15.28:0/2031922034 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f620c0fe610 con 0x7f61f80a3c40 2024-04-21T08:49:14.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.314+0000 7f62116a8640 1 -- 172.21.15.28:0/2031922034 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f61f80a3c40 msgr2=0x7f61f80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.314+0000 7f62116a8640 1 --2- 172.21.15.28:0/2031922034 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f61f80a3c40 0x7f61f80a6100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f620c199690 tx=0x7f61f4005c90 comp rx=0 tx=0).stop 2024-04-21T08:49:14.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.314+0000 7f62116a8640 1 -- 172.21.15.28:0/2031922034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f620c100e90 msgr2=0x7f620c198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.314+0000 7f62116a8640 1 --2- 172.21.15.28:0/2031922034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f620c100e90 0x7f620c198180 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6204006010 tx=0x7f6204004040 comp rx=0 tx=0).stop 2024-04-21T08:49:14.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.314+0000 7f62116a8640 1 -- 172.21.15.28:0/2031922034 shutdown_connections 2024-04-21T08:49:14.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.314+0000 7f62116a8640 1 --2- 172.21.15.28:0/2031922034 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f620c1029a0 0x7f620c19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.314+0000 7f62116a8640 1 --2- 172.21.15.28:0/2031922034 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f620c101fe0 0x7f620c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.314+0000 7f62116a8640 1 --2- 172.21.15.28:0/2031922034 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f61f80a3c40 0x7f61f80a6100 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.314+0000 7f62116a8640 1 --2- 172.21.15.28:0/2031922034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f620c100e90 0x7f620c198180 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.314+0000 7f62116a8640 1 -- 172.21.15.28:0/2031922034 >> 172.21.15.28:0/2031922034 conn(0x7f620c0fc400 msgr2=0x7f620c0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.314+0000 7f62116a8640 1 -- 172.21.15.28:0/2031922034 shutdown_connections 2024-04-21T08:49:14.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.314+0000 7f62116a8640 1 -- 172.21.15.28:0/2031922034 wait complete. 2024-04-21T08:49:14.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-21T08:49:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.432+0000 7fe7d90f2640 1 -- 172.21.15.28:0/2116910562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4102a90 msgr2=0x7fe7d4104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.432+0000 7fe7d90f2640 1 --2- 172.21.15.28:0/2116910562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4102a90 0x7fe7d4104e80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4009900 tx=0x7fe7c402bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.432+0000 7fe7d90f2640 1 -- 172.21.15.28:0/2116910562 shutdown_connections 2024-04-21T08:49:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.432+0000 7fe7d90f2640 1 --2- 172.21.15.28:0/2116910562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7d4107cf0 0x7fe7d410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.432+0000 7fe7d90f2640 1 --2- 172.21.15.28:0/2116910562 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7d41053c0 0x7fe7d41077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.432+0000 7fe7d90f2640 1 --2- 172.21.15.28:0/2116910562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d4102a90 0x7fe7d4104e80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.432+0000 7fe7d90f2640 1 -- 172.21.15.28:0/2116910562 >> 172.21.15.28:0/2116910562 conn(0x7fe7d40fc440 msgr2=0x7fe7d40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.432+0000 7fe7d90f2640 1 -- 172.21.15.28:0/2116910562 shutdown_connections 2024-04-21T08:49:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.432+0000 7fe7d90f2640 1 -- 172.21.15.28:0/2116910562 wait complete. 2024-04-21T08:49:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.432+0000 7fe7d90f2640 1 Processor -- start 2024-04-21T08:49:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d90f2640 1 -- start start 2024-04-21T08:49:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d90f2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7d4102a90 0x7fe7d4102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d90f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d41053c0 0x7fe7d4100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d90f2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7d4107cf0 0x7fe7d4100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d90f2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d410daf0 con 0x7fe7d41053c0 2024-04-21T08:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d90f2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d410d970 con 0x7fe7d4107cf0 2024-04-21T08:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d90f2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d410dc70 con 0x7fe7d4102a90 2024-04-21T08:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d41053c0 0x7fe7d4100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d41053c0 0x7fe7d4100a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41812/0 (socket says 172.21.15.28:41812) 2024-04-21T08:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d259c640 1 -- 172.21.15.28:0/3606922886 learned_addr learned my addr 172.21.15.28:0/3606922886 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d2d9d640 1 --2- 172.21.15.28:0/3606922886 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7d4102a90 0x7fe7d4102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d359e640 1 --2- 172.21.15.28:0/3606922886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7d4107cf0 0x7fe7d4100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d259c640 1 -- 172.21.15.28:0/3606922886 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7d4102a90 msgr2=0x7fe7d4102400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d259c640 1 --2- 172.21.15.28:0/3606922886 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7d4102a90 0x7fe7d4102400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d259c640 1 -- 172.21.15.28:0/3606922886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7d4107cf0 msgr2=0x7fe7d4100f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d259c640 1 --2- 172.21.15.28:0/3606922886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7d4107cf0 0x7fe7d4100f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d259c640 1 -- 172.21.15.28:0/3606922886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d4101810 con 0x7fe7d41053c0 2024-04-21T08:49:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d359e640 1 --2- 172.21.15.28:0/3606922886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7d4107cf0 0x7fe7d4100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.433+0000 7fe7d259c640 1 --2- 172.21.15.28:0/3606922886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d41053c0 0x7fe7d4100a40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800e9f0 tx=0x7fe7c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.434+0000 7fe7bbfff640 1 -- 172.21.15.28:0/3606922886 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c800ccc0 con 0x7fe7d41053c0 2024-04-21T08:49:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.434+0000 7fe7d90f2640 1 -- 172.21.15.28:0/3606922886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d41a2310 con 0x7fe7d41053c0 2024-04-21T08:49:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.434+0000 7fe7bbfff640 1 -- 172.21.15.28:0/3606922886 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7c80054d0 con 0x7fe7d41053c0 2024-04-21T08:49:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.434+0000 7fe7bbfff640 1 -- 172.21.15.28:0/3606922886 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8010430 con 0x7fe7d41053c0 2024-04-21T08:49:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.434+0000 7fe7d90f2640 1 -- 172.21.15.28:0/3606922886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d41a2810 con 0x7fe7d41053c0 2024-04-21T08:49:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.435+0000 7fe7d90f2640 1 -- 172.21.15.28:0/3606922886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7d40fdf30 con 0x7fe7d41053c0 2024-04-21T08:49:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.436+0000 7fe7bbfff640 1 -- 172.21.15.28:0/3606922886 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe7c80046c0 con 0x7fe7d41053c0 2024-04-21T08:49:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.436+0000 7fe7bbfff640 1 --2- 172.21.15.28:0/3606922886 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe7bc0a3c40 0x7fe7bc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.436+0000 7fe7bbfff640 1 -- 172.21.15.28:0/3606922886 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe7c80d0c50 con 0x7fe7d41053c0 2024-04-21T08:49:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.436+0000 7fe7d2d9d640 1 --2- 172.21.15.28:0/3606922886 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe7bc0a3c40 0x7fe7bc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.437+0000 7fe7d2d9d640 1 --2- 172.21.15.28:0/3606922886 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe7bc0a3c40 0x7fe7bc0a6100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4009230 tx=0x7fe7c4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:14.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.441+0000 7fe7bbfff640 1 -- 172.21.15.28:0/3606922886 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe7c809d290 con 0x7fe7d41053c0 2024-04-21T08:49:14.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.657+0000 7fe7d90f2640 1 -- 172.21.15.28:0/3606922886 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fe7d4062420 con 0x7fe7bc0a3c40 2024-04-21T08:49:14.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.662+0000 7fe7bbfff640 1 -- 172.21.15.28:0/3606922886 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fe7d4062420 con 0x7fe7bc0a3c40 2024-04-21T08:49:14.663 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_1/e0780924-49b5-41e0-9dbf-b1363aadd776 2024-04-21T08:49:14.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.664+0000 7fe7d90f2640 1 -- 172.21.15.28:0/3606922886 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe7bc0a3c40 msgr2=0x7fe7bc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.664+0000 7fe7d90f2640 1 --2- 172.21.15.28:0/3606922886 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe7bc0a3c40 0x7fe7bc0a6100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4009230 tx=0x7fe7c4005fd0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.664+0000 7fe7d90f2640 1 -- 172.21.15.28:0/3606922886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d41053c0 msgr2=0x7fe7d4100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.664+0000 7fe7d90f2640 1 --2- 172.21.15.28:0/3606922886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d41053c0 0x7fe7d4100a40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe7c800e9f0 tx=0x7fe7c800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.665+0000 7fe7d90f2640 1 -- 172.21.15.28:0/3606922886 shutdown_connections 2024-04-21T08:49:14.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.665+0000 7fe7d90f2640 1 --2- 172.21.15.28:0/3606922886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe7d4107cf0 0x7fe7d4100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.665+0000 7fe7d90f2640 1 --2- 172.21.15.28:0/3606922886 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe7bc0a3c40 0x7fe7bc0a6100 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.665+0000 7fe7d90f2640 1 --2- 172.21.15.28:0/3606922886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d41053c0 0x7fe7d4100a40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.665+0000 7fe7d90f2640 1 --2- 172.21.15.28:0/3606922886 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe7d4102a90 0x7fe7d4102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.665+0000 7fe7d90f2640 1 -- 172.21.15.28:0/3606922886 >> 172.21.15.28:0/3606922886 conn(0x7fe7d40fc440 msgr2=0x7fe7d4103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.665+0000 7fe7d90f2640 1 -- 172.21.15.28:0/3606922886 shutdown_connections 2024-04-21T08:49:14.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.665+0000 7fe7d90f2640 1 -- 172.21.15.28:0/3606922886 wait complete. 2024-04-21T08:49:14.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:49:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.780+0000 7f51abe74640 1 -- 172.21.15.28:0/4258181067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a4105350 msgr2=0x7f51a4107740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.780+0000 7f51abe74640 1 --2- 172.21.15.28:0/4258181067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a4105350 0x7f51a4107740 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5194009900 tx=0x7f519402bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.780+0000 7f51abe74640 1 -- 172.21.15.28:0/4258181067 shutdown_connections 2024-04-21T08:49:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.780+0000 7f51abe74640 1 --2- 172.21.15.28:0/4258181067 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51a4107c80 0x7f51a410a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.780+0000 7f51abe74640 1 --2- 172.21.15.28:0/4258181067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a4105350 0x7f51a4107740 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.780+0000 7f51abe74640 1 --2- 172.21.15.28:0/4258181067 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51a4102a20 0x7f51a4104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.780+0000 7f51abe74640 1 -- 172.21.15.28:0/4258181067 >> 172.21.15.28:0/4258181067 conn(0x7f51a40fc410 msgr2=0x7f51a40fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.780+0000 7f51abe74640 1 -- 172.21.15.28:0/4258181067 shutdown_connections 2024-04-21T08:49:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.780+0000 7f51abe74640 1 -- 172.21.15.28:0/4258181067 wait complete. 2024-04-21T08:49:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51abe74640 1 Processor -- start 2024-04-21T08:49:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51abe74640 1 -- start start 2024-04-21T08:49:14.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51abe74640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a4102a20 0x7f51a4198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51abe74640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51a4105350 0x7f51a41986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51abe74640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51a4107c80 0x7f51a419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51abe74640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51a410dae0 con 0x7f51a4102a20 2024-04-21T08:49:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51abe74640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f51a410d960 con 0x7f51a4107c80 2024-04-21T08:49:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51abe74640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f51a410dc60 con 0x7f51a4105350 2024-04-21T08:49:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51a9c10640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a4102a20 0x7f51a4198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51a9c10640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a4102a20 0x7f51a4198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41828/0 (socket says 172.21.15.28:41828) 2024-04-21T08:49:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51a9c10640 1 -- 172.21.15.28:0/3926694048 learned_addr learned my addr 172.21.15.28:0/3926694048 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51aa411640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51a4107c80 0x7f51a419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51a9c10640 1 -- 172.21.15.28:0/3926694048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51a4105350 msgr2=0x7f51a41986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51a940f640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51a4105350 0x7f51a41986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51a9c10640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51a4105350 0x7f51a41986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51a9c10640 1 -- 172.21.15.28:0/3926694048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51a4107c80 msgr2=0x7f51a419f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51a9c10640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51a4107c80 0x7f51a419f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51a9c10640 1 -- 172.21.15.28:0/3926694048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a419fd90 con 0x7f51a4102a20 2024-04-21T08:49:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51aa411640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51a4107c80 0x7f51a419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51a940f640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51a4105350 0x7f51a41986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.781+0000 7f51a9c10640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a4102a20 0x7f51a4198170 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f51a000c920 tx=0x7f51a000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.782+0000 7f519affd640 1 -- 172.21.15.28:0/3926694048 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a0007c70 con 0x7f51a4102a20 2024-04-21T08:49:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.782+0000 7f51abe74640 1 -- 172.21.15.28:0/3926694048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51a41a0080 con 0x7f51a4102a20 2024-04-21T08:49:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.782+0000 7f519affd640 1 -- 172.21.15.28:0/3926694048 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51a0007e10 con 0x7f51a4102a20 2024-04-21T08:49:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.782+0000 7f519affd640 1 -- 172.21.15.28:0/3926694048 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a0018820 con 0x7f51a4102a20 2024-04-21T08:49:14.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.782+0000 7f51abe74640 1 -- 172.21.15.28:0/3926694048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51a41a0620 con 0x7f51a4102a20 2024-04-21T08:49:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.783+0000 7f51abe74640 1 -- 172.21.15.28:0/3926694048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51a40fdf00 con 0x7f51a4102a20 2024-04-21T08:49:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.784+0000 7f519affd640 1 -- 172.21.15.28:0/3926694048 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f51a000f450 con 0x7f51a4102a20 2024-04-21T08:49:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.784+0000 7f519affd640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f51900a3c40 0x7f51900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.785+0000 7f51a940f640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f51900a3c40 0x7f51900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.785+0000 7f519affd640 1 -- 172.21.15.28:0/3926694048 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f51a00d0990 con 0x7f51a4102a20 2024-04-21T08:49:14.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.785+0000 7f51a940f640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f51900a3c40 0x7f51900a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5194036040 tx=0x7f5194000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:14.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:14.789+0000 7f519affd640 1 -- 172.21.15.28:0/3926694048 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f51a0014030 con 0x7f51a4102a20 2024-04-21T08:49:15.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.021+0000 7f51abe74640 1 -- 172.21.15.28:0/3926694048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51a402d5c0 con 0x7f51a4102a20 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.021+0000 7f519affd640 1 -- 172.21.15.28:0/3926694048 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f51a009cfd0 con 0x7f51a4102a20 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:49:12.840022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24463,24487,24488]},"id":1}]} 2024-04-21T08:49:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.023+0000 7f51abe74640 1 -- 172.21.15.28:0/3926694048 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f51900a3c40 msgr2=0x7f51900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.023+0000 7f51abe74640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f51900a3c40 0x7f51900a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5194036040 tx=0x7f5194000bc0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.023+0000 7f51abe74640 1 -- 172.21.15.28:0/3926694048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a4102a20 msgr2=0x7f51a4198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.023+0000 7f51abe74640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a4102a20 0x7f51a4198170 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f51a000c920 tx=0x7f51a000cde0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.024+0000 7f51abe74640 1 -- 172.21.15.28:0/3926694048 shutdown_connections 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.024+0000 7f51abe74640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51a4107c80 0x7f51a419f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.024+0000 7f51abe74640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f51a4105350 0x7f51a41986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.024+0000 7f51abe74640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f51900a3c40 0x7f51900a6100 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.024+0000 7f51abe74640 1 --2- 172.21.15.28:0/3926694048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51a4102a20 0x7f51a4198170 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.024+0000 7f51abe74640 1 -- 172.21.15.28:0/3926694048 >> 172.21.15.28:0/3926694048 conn(0x7f51a40fc410 msgr2=0x7f51a4103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.024+0000 7f51abe74640 1 -- 172.21.15.28:0/3926694048 shutdown_connections 2024-04-21T08:49:15.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.024+0000 7f51abe74640 1 -- 172.21.15.28:0/3926694048 wait complete. 2024-04-21T08:49:15.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:14 smithi028 ceph-mon[28003]: pgmap v308: 129 pgs: 129 active+clean; 582 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-21T08:49:15.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:14 smithi028 ceph-mon[28003]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:49:15.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:14 smithi028 ceph-mon[28003]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T08:49:15.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:14 smithi028 ceph-mon[28003]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:49:15.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:14 smithi028 ceph-mon[28003]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:49:15.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:49:15.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:14 smithi152 ceph-mon[31401]: pgmap v308: 129 pgs: 129 active+clean; 582 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-21T08:49:15.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:14 smithi152 ceph-mon[31401]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:49:15.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:14 smithi152 ceph-mon[31401]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T08:49:15.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:14 smithi152 ceph-mon[31401]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:49:15.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:14 smithi152 ceph-mon[31401]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:49:15.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:14 smithi171 ceph-mon[31358]: pgmap v308: 129 pgs: 129 active+clean; 582 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-21T08:49:15.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:14 smithi171 ceph-mon[31358]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:49:15.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:14 smithi171 ceph-mon[31358]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T08:49:15.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:14 smithi171 ceph-mon[31358]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:49:15.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:14 smithi171 ceph-mon[31358]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:49:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.137+0000 7f822338f640 1 -- 172.21.15.28:0/1048678991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c102aa0 msgr2=0x7f821c104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.137+0000 7f822338f640 1 --2- 172.21.15.28:0/1048678991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c102aa0 0x7f821c104e90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8218009900 tx=0x7f821802bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.137+0000 7f822338f640 1 -- 172.21.15.28:0/1048678991 shutdown_connections 2024-04-21T08:49:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.137+0000 7f822338f640 1 --2- 172.21.15.28:0/1048678991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f821c107d00 0x7f821c10a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.137+0000 7f822338f640 1 --2- 172.21.15.28:0/1048678991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f821c1053d0 0x7f821c1077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.137+0000 7f822338f640 1 --2- 172.21.15.28:0/1048678991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c102aa0 0x7f821c104e90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.137+0000 7f822338f640 1 -- 172.21.15.28:0/1048678991 >> 172.21.15.28:0/1048678991 conn(0x7f821c0fc470 msgr2=0x7f821c0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.138+0000 7f822338f640 1 -- 172.21.15.28:0/1048678991 shutdown_connections 2024-04-21T08:49:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.138+0000 7f822338f640 1 -- 172.21.15.28:0/1048678991 wait complete. 2024-04-21T08:49:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.138+0000 7f822338f640 1 Processor -- start 2024-04-21T08:49:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.138+0000 7f822338f640 1 -- start start 2024-04-21T08:49:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.138+0000 7f822338f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c102aa0 0x7f821c19a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.138+0000 7f822338f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f821c1053d0 0x7f821c19a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.138+0000 7f822338f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f821c107d00 0x7f821c1a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.138+0000 7f822338f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f821c10db00 con 0x7f821c102aa0 2024-04-21T08:49:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.138+0000 7f822338f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f821c10d980 con 0x7f821c1053d0 2024-04-21T08:49:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.138+0000 7f822338f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f821c10dc80 con 0x7f821c107d00 2024-04-21T08:49:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.138+0000 7f822112b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c102aa0 0x7f821c19a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.138+0000 7f822112b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c102aa0 0x7f821c19a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41846/0 (socket says 172.21.15.28:41846) 2024-04-21T08:49:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.138+0000 7f822112b640 1 -- 172.21.15.28:0/1158862757 learned_addr learned my addr 172.21.15.28:0/1158862757 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.139+0000 7f822092a640 1 --2- 172.21.15.28:0/1158862757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f821c1053d0 0x7f821c19a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.139+0000 7f822192c640 1 --2- 172.21.15.28:0/1158862757 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f821c107d00 0x7f821c1a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.139+0000 7f822112b640 1 -- 172.21.15.28:0/1158862757 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f821c107d00 msgr2=0x7f821c1a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.139+0000 7f822112b640 1 --2- 172.21.15.28:0/1158862757 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f821c107d00 0x7f821c1a19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.139+0000 7f822112b640 1 -- 172.21.15.28:0/1158862757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f821c1053d0 msgr2=0x7f821c19a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.139+0000 7f822112b640 1 --2- 172.21.15.28:0/1158862757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f821c1053d0 0x7f821c19a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.139+0000 7f822112b640 1 -- 172.21.15.28:0/1158862757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f821c1a2050 con 0x7f821c102aa0 2024-04-21T08:49:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.139+0000 7f822092a640 1 --2- 172.21.15.28:0/1158862757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f821c1053d0 0x7f821c19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.139+0000 7f822112b640 1 --2- 172.21.15.28:0/1158862757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c102aa0 0x7f821c19a430 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8218006010 tx=0x7f82180042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.139+0000 7f82127fc640 1 -- 172.21.15.28:0/1158862757 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218043020 con 0x7f821c102aa0 2024-04-21T08:49:15.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.139+0000 7f822338f640 1 -- 172.21.15.28:0/1158862757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f821c1a22e0 con 0x7f821c102aa0 2024-04-21T08:49:15.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.139+0000 7f82127fc640 1 -- 172.21.15.28:0/1158862757 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8218004aa0 con 0x7f821c102aa0 2024-04-21T08:49:15.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.139+0000 7f82127fc640 1 -- 172.21.15.28:0/1158862757 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218003c70 con 0x7f821c102aa0 2024-04-21T08:49:15.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.139+0000 7f822338f640 1 -- 172.21.15.28:0/1158862757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f821c1a2880 con 0x7f821c102aa0 2024-04-21T08:49:15.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.140+0000 7f822338f640 1 -- 172.21.15.28:0/1158862757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f821c1097f0 con 0x7f821c102aa0 2024-04-21T08:49:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.141+0000 7f82127fc640 1 -- 172.21.15.28:0/1158862757 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8218046020 con 0x7f821c102aa0 2024-04-21T08:49:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.142+0000 7f82127fc640 1 --2- 172.21.15.28:0/1158862757 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f82080a3c40 0x7f82080a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.142+0000 7f822092a640 1 --2- 172.21.15.28:0/1158862757 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f82080a3c40 0x7f82080a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.142+0000 7f82127fc640 1 -- 172.21.15.28:0/1158862757 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f82180f2ba0 con 0x7f821c102aa0 2024-04-21T08:49:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.142+0000 7f822092a640 1 --2- 172.21.15.28:0/1158862757 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f82080a3c40 0x7f82080a6100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f820c005d00 tx=0x7f820c005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.147+0000 7f82127fc640 1 -- 172.21.15.28:0/1158862757 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f821804d020 con 0x7f821c102aa0 2024-04-21T08:49:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.335+0000 7f822338f640 1 -- 172.21.15.28:0/1158862757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f821c062b50 con 0x7f821c102aa0 2024-04-21T08:49:15.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.336+0000 7f82127fc640 1 -- 172.21.15.28:0/1158862757 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24101 (secure 0 0 0) 0x7f82180bf290 con 0x7f821c102aa0 2024-04-21T08:49:15.338 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:49:15.338 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:48:51.121128+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:49:15.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.338+0000 7f822338f640 1 -- 172.21.15.28:0/1158862757 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f82080a3c40 msgr2=0x7f82080a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.338+0000 7f822338f640 1 --2- 172.21.15.28:0/1158862757 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f82080a3c40 0x7f82080a6100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f820c005d00 tx=0x7f820c005c70 comp rx=0 tx=0).stop 2024-04-21T08:49:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.338+0000 7f822338f640 1 -- 172.21.15.28:0/1158862757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c102aa0 msgr2=0x7f821c19a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.338+0000 7f822338f640 1 --2- 172.21.15.28:0/1158862757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c102aa0 0x7f821c19a430 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8218006010 tx=0x7f82180042f0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.338+0000 7f822338f640 1 -- 172.21.15.28:0/1158862757 shutdown_connections 2024-04-21T08:49:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.338+0000 7f822338f640 1 --2- 172.21.15.28:0/1158862757 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f821c107d00 0x7f821c1a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.338+0000 7f822338f640 1 --2- 172.21.15.28:0/1158862757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f821c1053d0 0x7f821c19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.338+0000 7f822338f640 1 --2- 172.21.15.28:0/1158862757 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f82080a3c40 0x7f82080a6100 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.338+0000 7f822338f640 1 --2- 172.21.15.28:0/1158862757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c102aa0 0x7f821c19a430 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.338+0000 7f822338f640 1 -- 172.21.15.28:0/1158862757 >> 172.21.15.28:0/1158862757 conn(0x7f821c0fc470 msgr2=0x7f821c102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.339+0000 7f822338f640 1 -- 172.21.15.28:0/1158862757 shutdown_connections 2024-04-21T08:49:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.339+0000 7f822338f640 1 -- 172.21.15.28:0/1158862757 wait complete. 2024-04-21T08:49:15.351 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-21T08:49:15.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:49:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.452+0000 7f818a99f640 1 -- 172.21.15.28:0/4239666838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8184101fa0 msgr2=0x7f81841023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.452+0000 7f818a99f640 1 --2- 172.21.15.28:0/4239666838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8184101fa0 0x7f81841023c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8174009900 tx=0x7f817402bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.452+0000 7f818a99f640 1 -- 172.21.15.28:0/4239666838 shutdown_connections 2024-04-21T08:49:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.452+0000 7f818a99f640 1 --2- 172.21.15.28:0/4239666838 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8184103b50 0x7f818410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.452+0000 7f818a99f640 1 --2- 172.21.15.28:0/4239666838 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8184103190 0x7f8184103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.452+0000 7f818a99f640 1 --2- 172.21.15.28:0/4239666838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8184101fa0 0x7f81841023c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.452+0000 7f818a99f640 1 -- 172.21.15.28:0/4239666838 >> 172.21.15.28:0/4239666838 conn(0x7f81840fd530 msgr2=0x7f81840ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.452+0000 7f818a99f640 1 -- 172.21.15.28:0/4239666838 shutdown_connections 2024-04-21T08:49:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.452+0000 7f818a99f640 1 -- 172.21.15.28:0/4239666838 wait complete. 2024-04-21T08:49:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.453+0000 7f818a99f640 1 Processor -- start 2024-04-21T08:49:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.453+0000 7f818a99f640 1 -- start start 2024-04-21T08:49:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.453+0000 7f818a99f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8184101fa0 0x7f81841981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.453+0000 7f818a99f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8184103190 0x7f81841986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.453+0000 7f818a99f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8184103b50 0x7f818419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.453+0000 7f818a99f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f818410dae0 con 0x7f8184103b50 2024-04-21T08:49:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.453+0000 7f818a99f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f818410d960 con 0x7f8184101fa0 2024-04-21T08:49:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.453+0000 7f818a99f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f818410dc60 con 0x7f8184103190 2024-04-21T08:49:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.453+0000 7f8188f3c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8184103b50 0x7f818419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.453+0000 7f8188f3c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8184103b50 0x7f818419f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41860/0 (socket says 172.21.15.28:41860) 2024-04-21T08:49:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.453+0000 7f8188f3c640 1 -- 172.21.15.28:0/1798196491 learned_addr learned my addr 172.21.15.28:0/1798196491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.453+0000 7f8183fff640 1 --2- 172.21.15.28:0/1798196491 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8184101fa0 0x7f81841981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.453+0000 7f81837fe640 1 --2- 172.21.15.28:0/1798196491 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8184103190 0x7f81841986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.454+0000 7f8188f3c640 1 -- 172.21.15.28:0/1798196491 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8184103190 msgr2=0x7f81841986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.454+0000 7f8188f3c640 1 --2- 172.21.15.28:0/1798196491 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8184103190 0x7f81841986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.454+0000 7f8188f3c640 1 -- 172.21.15.28:0/1798196491 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8184101fa0 msgr2=0x7f81841981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.454+0000 7f8188f3c640 1 --2- 172.21.15.28:0/1798196491 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8184101fa0 0x7f81841981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.454+0000 7f8188f3c640 1 -- 172.21.15.28:0/1798196491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818419fdc0 con 0x7f8184103b50 2024-04-21T08:49:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.454+0000 7f8188f3c640 1 --2- 172.21.15.28:0/1798196491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8184103b50 0x7f818419f750 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f817c00eb10 tx=0x7f817c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.454+0000 7f81817fa640 1 -- 172.21.15.28:0/1798196491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c004b20 con 0x7f8184103b50 2024-04-21T08:49:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.454+0000 7f81817fa640 1 -- 172.21.15.28:0/1798196491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f817c004cc0 con 0x7f8184103b50 2024-04-21T08:49:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.454+0000 7f818a99f640 1 -- 172.21.15.28:0/1798196491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81841a00b0 con 0x7f8184103b50 2024-04-21T08:49:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.454+0000 7f81817fa640 1 -- 172.21.15.28:0/1798196491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c010670 con 0x7f8184103b50 2024-04-21T08:49:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.454+0000 7f818a99f640 1 -- 172.21.15.28:0/1798196491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81841a05b0 con 0x7f8184103b50 2024-04-21T08:49:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.455+0000 7f818a99f640 1 -- 172.21.15.28:0/1798196491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8184109340 con 0x7f8184103b50 2024-04-21T08:49:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.456+0000 7f81817fa640 1 -- 172.21.15.28:0/1798196491 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f817c005020 con 0x7f8184103b50 2024-04-21T08:49:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.457+0000 7f81817fa640 1 --2- 172.21.15.28:0/1798196491 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f816c0a3c40 0x7f816c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.457+0000 7f8183fff640 1 --2- 172.21.15.28:0/1798196491 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f816c0a3c40 0x7f816c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.457+0000 7f81817fa640 1 -- 172.21.15.28:0/1798196491 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f817c0d0e90 con 0x7f8184103b50 2024-04-21T08:49:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.457+0000 7f8183fff640 1 --2- 172.21.15.28:0/1798196491 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f816c0a3c40 0x7f816c0a6100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8174005ab0 tx=0x7f8174005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.461+0000 7f81817fa640 1 -- 172.21.15.28:0/1798196491 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f817c09d470 con 0x7f8184103b50 2024-04-21T08:49:15.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.645+0000 7f818a99f640 1 -- 172.21.15.28:0/1798196491 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8184062390 con 0x7f816c0a3c40 2024-04-21T08:49:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.655+0000 7f81817fa640 1 -- 172.21.15.28:0/1798196491 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340557 (secure 0 0 0) 0x7f8184062390 con 0x7f816c0a3c40 2024-04-21T08:49:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:49:15.662 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-21T08:49:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.659+0000 7f818a99f640 1 -- 172.21.15.28:0/1798196491 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f816c0a3c40 msgr2=0x7f816c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.659+0000 7f818a99f640 1 --2- 172.21.15.28:0/1798196491 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f816c0a3c40 0x7f816c0a6100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8174005ab0 tx=0x7f8174005fd0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.659+0000 7f818a99f640 1 -- 172.21.15.28:0/1798196491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8184103b50 msgr2=0x7f818419f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.659+0000 7f818a99f640 1 --2- 172.21.15.28:0/1798196491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8184103b50 0x7f818419f750 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f817c00eb10 tx=0x7f817c00c2d0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.659+0000 7f818a99f640 1 -- 172.21.15.28:0/1798196491 shutdown_connections 2024-04-21T08:49:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.659+0000 7f818a99f640 1 --2- 172.21.15.28:0/1798196491 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f816c0a3c40 0x7f816c0a6100 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.659+0000 7f818a99f640 1 --2- 172.21.15.28:0/1798196491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8184103b50 0x7f818419f750 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.659+0000 7f818a99f640 1 --2- 172.21.15.28:0/1798196491 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8184103190 0x7f81841986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.659+0000 7f818a99f640 1 --2- 172.21.15.28:0/1798196491 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8184101fa0 0x7f81841981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.659+0000 7f818a99f640 1 -- 172.21.15.28:0/1798196491 >> 172.21.15.28:0/1798196491 conn(0x7f81840fd530 msgr2=0x7f8184106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.659+0000 7f818a99f640 1 -- 172.21.15.28:0/1798196491 shutdown_connections 2024-04-21T08:49:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.659+0000 7f818a99f640 1 -- 172.21.15.28:0/1798196491 wait complete. 2024-04-21T08:49:15.671 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-04-21T08:49:15.467077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598328,"num_objects":57,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":154,"num_read_kb":134,"num_write":296,"num_write_kb":2983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369108,"kb_used_data":11616,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572716,"statfs":{"total":1151940427776,"available":1151562461184,"internally_reserved":0,"allocated":11894784,"data_stored":7540515,"data_compressed":283947,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3461,"num_objects":33,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":68,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003443"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268559+0000","last_change":"2024-04-21T08:48:46.471626+0000","last_active":"2024-04-21T08:48:53.268559+0000","last_peered":"2024-04-21T08:48:53.268559+0000","last_clean":"2024-04-21T08:48:53.268559+0000","last_became_active":"2024-04-21T08:48:46.470498+0000","last_became_peered":"2024-04-21T08:48:46.470498+0000","last_unstale":"2024-04-21T08:48:53.268559+0000","last_undegraded":"2024-04-21T08:48:53.268559+0000","last_fullsized":"2024-04-21T08:48:53.268559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:49.452181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269742+0000","last_change":"2024-04-21T08:48:46.469111+0000","last_active":"2024-04-21T08:48:53.269742+0000","last_peered":"2024-04-21T08:48:53.269742+0000","last_clean":"2024-04-21T08:48:53.269742+0000","last_became_active":"2024-04-21T08:48:46.467926+0000","last_became_peered":"2024-04-21T08:48:46.467926+0000","last_unstale":"2024-04-21T08:48:53.269742+0000","last_undegraded":"2024-04-21T08:48:53.269742+0000","last_fullsized":"2024-04-21T08:48:53.269742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:35:38.230212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269486+0000","last_change":"2024-04-21T08:48:46.475673+0000","last_active":"2024-04-21T08:48:53.269486+0000","last_peered":"2024-04-21T08:48:53.269486+0000","last_clean":"2024-04-21T08:48:53.269486+0000","last_became_active":"2024-04-21T08:48:46.474917+0000","last_became_peered":"2024-04-21T08:48:46.474917+0000","last_unstale":"2024-04-21T08:48:53.269486+0000","last_undegraded":"2024-04-21T08:48:53.269486+0000","last_fullsized":"2024-04-21T08:48:53.269486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:01.661704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929510+0000","last_change":"2024-04-21T08:48:46.468147+0000","last_active":"2024-04-21T08:48:52.929510+0000","last_peered":"2024-04-21T08:48:52.929510+0000","last_clean":"2024-04-21T08:48:52.929510+0000","last_became_active":"2024-04-21T08:48:46.467811+0000","last_became_peered":"2024-04-21T08:48:46.467811+0000","last_unstale":"2024-04-21T08:48:52.929510+0000","last_undegraded":"2024-04-21T08:48:52.929510+0000","last_fullsized":"2024-04-21T08:48:52.929510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:21.658605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536393+0000","last_change":"2024-04-21T08:48:46.471322+0000","last_active":"2024-04-21T08:48:52.536393+0000","last_peered":"2024-04-21T08:48:52.536393+0000","last_clean":"2024-04-21T08:48:52.536393+0000","last_became_active":"2024-04-21T08:48:46.470991+0000","last_became_peered":"2024-04-21T08:48:46.470991+0000","last_unstale":"2024-04-21T08:48:52.536393+0000","last_undegraded":"2024-04-21T08:48:52.536393+0000","last_fullsized":"2024-04-21T08:48:52.536393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:22.839870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.167091+0000","last_change":"2024-04-21T08:48:46.467039+0000","last_active":"2024-04-21T08:48:52.167091+0000","last_peered":"2024-04-21T08:48:52.167091+0000","last_clean":"2024-04-21T08:48:52.167091+0000","last_became_active":"2024-04-21T08:48:46.466614+0000","last_became_peered":"2024-04-21T08:48:46.466614+0000","last_unstale":"2024-04-21T08:48:52.167091+0000","last_undegraded":"2024-04-21T08:48:52.167091+0000","last_fullsized":"2024-04-21T08:48:52.167091+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:38.029215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766096+0000","last_change":"2024-04-21T08:48:46.473268+0000","last_active":"2024-04-21T08:48:51.766096+0000","last_peered":"2024-04-21T08:48:51.766096+0000","last_clean":"2024-04-21T08:48:51.766096+0000","last_became_active":"2024-04-21T08:48:46.472261+0000","last_became_peered":"2024-04-21T08:48:46.472261+0000","last_unstale":"2024-04-21T08:48:51.766096+0000","last_undegraded":"2024-04-21T08:48:51.766096+0000","last_fullsized":"2024-04-21T08:48:51.766096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:58:52.990687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268092+0000","last_change":"2024-04-21T08:48:46.472788+0000","last_active":"2024-04-21T08:48:53.268092+0000","last_peered":"2024-04-21T08:48:53.268092+0000","last_clean":"2024-04-21T08:48:53.268092+0000","last_became_active":"2024-04-21T08:48:46.472482+0000","last_became_peered":"2024-04-21T08:48:46.472482+0000","last_unstale":"2024-04-21T08:48:53.268092+0000","last_undegraded":"2024-04-21T08:48:53.268092+0000","last_fullsized":"2024-04-21T08:48:53.268092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:48.483846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268379+0000","last_change":"2024-04-21T08:48:46.460771+0000","last_active":"2024-04-21T08:48:53.268379+0000","last_peered":"2024-04-21T08:48:53.268379+0000","last_clean":"2024-04-21T08:48:53.268379+0000","last_became_active":"2024-04-21T08:48:46.460472+0000","last_became_peered":"2024-04-21T08:48:46.460472+0000","last_unstale":"2024-04-21T08:48:53.268379+0000","last_undegraded":"2024-04-21T08:48:53.268379+0000","last_fullsized":"2024-04-21T08:48:53.268379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:46:40.766245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.167473+0000","last_change":"2024-04-21T08:48:46.463475+0000","last_active":"2024-04-21T08:48:52.167473+0000","last_peered":"2024-04-21T08:48:52.167473+0000","last_clean":"2024-04-21T08:48:52.167473+0000","last_became_active":"2024-04-21T08:48:46.462559+0000","last_became_peered":"2024-04-21T08:48:46.462559+0000","last_unstale":"2024-04-21T08:48:52.167473+0000","last_undegraded":"2024-04-21T08:48:52.167473+0000","last_fullsized":"2024-04-21T08:48:52.167473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:22:06.372410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766709+0000","last_change":"2024-04-21T08:48:46.476997+0000","last_active":"2024-04-21T08:48:51.766709+0000","last_peered":"2024-04-21T08:48:51.766709+0000","last_clean":"2024-04-21T08:48:51.766709+0000","last_became_active":"2024-04-21T08:48:46.476620+0000","last_became_peered":"2024-04-21T08:48:46.476620+0000","last_unstale":"2024-04-21T08:48:51.766709+0000","last_undegraded":"2024-04-21T08:48:51.766709+0000","last_fullsized":"2024-04-21T08:48:51.766709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:37.569629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929437+0000","last_change":"2024-04-21T08:48:46.469355+0000","last_active":"2024-04-21T08:48:52.929437+0000","last_peered":"2024-04-21T08:48:52.929437+0000","last_clean":"2024-04-21T08:48:52.929437+0000","last_became_active":"2024-04-21T08:48:46.468604+0000","last_became_peered":"2024-04-21T08:48:46.468604+0000","last_unstale":"2024-04-21T08:48:52.929437+0000","last_undegraded":"2024-04-21T08:48:52.929437+0000","last_fullsized":"2024-04-21T08:48:52.929437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:23:54.500982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268515+0000","last_change":"2024-04-21T08:48:46.453116+0000","last_active":"2024-04-21T08:48:53.268515+0000","last_peered":"2024-04-21T08:48:53.268515+0000","last_clean":"2024-04-21T08:48:53.268515+0000","last_became_active":"2024-04-21T08:48:46.452607+0000","last_became_peered":"2024-04-21T08:48:46.452607+0000","last_unstale":"2024-04-21T08:48:53.268515+0000","last_undegraded":"2024-04-21T08:48:53.268515+0000","last_fullsized":"2024-04-21T08:48:53.268515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:57:04.371812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269848+0000","last_change":"2024-04-21T08:48:46.474516+0000","last_active":"2024-04-21T08:48:53.269848+0000","last_peered":"2024-04-21T08:48:53.269848+0000","last_clean":"2024-04-21T08:48:53.269848+0000","last_became_active":"2024-04-21T08:48:46.474129+0000","last_became_peered":"2024-04-21T08:48:46.474129+0000","last_unstale":"2024-04-21T08:48:53.269848+0000","last_undegraded":"2024-04-21T08:48:53.269848+0000","last_fullsized":"2024-04-21T08:48:53.269848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:40:47.784976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268217+0000","last_change":"2024-04-21T08:48:46.465205+0000","last_active":"2024-04-21T08:48:53.268217+0000","last_peered":"2024-04-21T08:48:53.268217+0000","last_clean":"2024-04-21T08:48:53.268217+0000","last_became_active":"2024-04-21T08:48:46.464797+0000","last_became_peered":"2024-04-21T08:48:46.464797+0000","last_unstale":"2024-04-21T08:48:53.268217+0000","last_undegraded":"2024-04-21T08:48:53.268217+0000","last_fullsized":"2024-04-21T08:48:53.268217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:54:31.725612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267704+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:48:53.267704+0000","last_peered":"2024-04-21T08:48:53.267704+0000","last_clean":"2024-04-21T08:48:53.267704+0000","last_became_active":"2024-04-21T08:48:46.469389+0000","last_became_peered":"2024-04-21T08:48:46.469389+0000","last_unstale":"2024-04-21T08:48:53.267704+0000","last_undegraded":"2024-04-21T08:48:53.267704+0000","last_fullsized":"2024-04-21T08:48:53.267704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:11:45.824402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268443+0000","last_change":"2024-04-21T08:48:46.470059+0000","last_active":"2024-04-21T08:48:53.268443+0000","last_peered":"2024-04-21T08:48:53.268443+0000","last_clean":"2024-04-21T08:48:53.268443+0000","last_became_active":"2024-04-21T08:48:46.469751+0000","last_became_peered":"2024-04-21T08:48:46.469751+0000","last_unstale":"2024-04-21T08:48:53.268443+0000","last_undegraded":"2024-04-21T08:48:53.268443+0000","last_fullsized":"2024-04-21T08:48:53.268443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:16.317879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765737+0000","last_change":"2024-04-21T08:48:46.470129+0000","last_active":"2024-04-21T08:48:51.765737+0000","last_peered":"2024-04-21T08:48:51.765737+0000","last_clean":"2024-04-21T08:48:51.765737+0000","last_became_active":"2024-04-21T08:48:46.469763+0000","last_became_peered":"2024-04-21T08:48:46.469763+0000","last_unstale":"2024-04-21T08:48:51.765737+0000","last_undegraded":"2024-04-21T08:48:51.765737+0000","last_fullsized":"2024-04-21T08:48:51.765737+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:47.731201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766299+0000","last_change":"2024-04-21T08:48:46.465356+0000","last_active":"2024-04-21T08:48:51.766299+0000","last_peered":"2024-04-21T08:48:51.766299+0000","last_clean":"2024-04-21T08:48:51.766299+0000","last_became_active":"2024-04-21T08:48:46.464183+0000","last_became_peered":"2024-04-21T08:48:46.464183+0000","last_unstale":"2024-04-21T08:48:51.766299+0000","last_undegraded":"2024-04-21T08:48:51.766299+0000","last_fullsized":"2024-04-21T08:48:51.766299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:37.533733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766124+0000","last_change":"2024-04-21T08:48:50.767378+0000","last_active":"2024-04-21T08:48:51.766124+0000","last_peered":"2024-04-21T08:48:51.766124+0000","last_clean":"2024-04-21T08:48:51.766124+0000","last_became_active":"2024-04-21T08:48:50.766856+0000","last_became_peered":"2024-04-21T08:48:50.766856+0000","last_unstale":"2024-04-21T08:48:51.766124+0000","last_undegraded":"2024-04-21T08:48:51.766124+0000","last_fullsized":"2024-04-21T08:48:51.766124+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:00.434540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765688+0000","last_change":"2024-04-21T08:48:45.550284+0000","last_active":"2024-04-21T08:48:51.765688+0000","last_peered":"2024-04-21T08:48:51.765688+0000","last_clean":"2024-04-21T08:48:51.765688+0000","last_became_active":"2024-04-21T08:48:45.549940+0000","last_became_peered":"2024-04-21T08:48:45.549940+0000","last_unstale":"2024-04-21T08:48:51.765688+0000","last_undegraded":"2024-04-21T08:48:51.765688+0000","last_fullsized":"2024-04-21T08:48:51.765688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:12.343299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.270040+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:48:53.270040+0000","last_peered":"2024-04-21T08:48:53.270040+0000","last_clean":"2024-04-21T08:48:53.270040+0000","last_became_active":"2024-04-21T08:48:46.469647+0000","last_became_peered":"2024-04-21T08:48:46.469647+0000","last_unstale":"2024-04-21T08:48:53.270040+0000","last_undegraded":"2024-04-21T08:48:53.270040+0000","last_fullsized":"2024-04-21T08:48:53.270040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:10:28.204554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929110+0000","last_change":"2024-04-21T08:48:50.771871+0000","last_active":"2024-04-21T08:48:52.929110+0000","last_peered":"2024-04-21T08:48:52.929110+0000","last_clean":"2024-04-21T08:48:52.929110+0000","last_became_active":"2024-04-21T08:48:50.770442+0000","last_became_peered":"2024-04-21T08:48:50.770442+0000","last_unstale":"2024-04-21T08:48:52.929110+0000","last_undegraded":"2024-04-21T08:48:52.929110+0000","last_fullsized":"2024-04-21T08:48:52.929110+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:18:57.813971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.270180+0000","last_change":"2024-04-21T08:48:45.566492+0000","last_active":"2024-04-21T08:48:53.270180+0000","last_peered":"2024-04-21T08:48:53.270180+0000","last_clean":"2024-04-21T08:48:53.270180+0000","last_became_active":"2024-04-21T08:48:45.566169+0000","last_became_peered":"2024-04-21T08:48:45.566169+0000","last_unstale":"2024-04-21T08:48:53.270180+0000","last_undegraded":"2024-04-21T08:48:53.270180+0000","last_fullsized":"2024-04-21T08:48:53.270180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:49.985664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269660+0000","last_change":"2024-04-21T08:48:46.470866+0000","last_active":"2024-04-21T08:48:53.269660+0000","last_peered":"2024-04-21T08:48:53.269660+0000","last_clean":"2024-04-21T08:48:53.269660+0000","last_became_active":"2024-04-21T08:48:46.470401+0000","last_became_peered":"2024-04-21T08:48:46.470401+0000","last_unstale":"2024-04-21T08:48:53.269660+0000","last_undegraded":"2024-04-21T08:48:53.269660+0000","last_fullsized":"2024-04-21T08:48:53.269660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:07.506212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929245+0000","last_change":"2024-04-21T08:48:50.772339+0000","last_active":"2024-04-21T08:48:52.929245+0000","last_peered":"2024-04-21T08:48:52.929245+0000","last_clean":"2024-04-21T08:48:52.929245+0000","last_became_active":"2024-04-21T08:48:50.771946+0000","last_became_peered":"2024-04-21T08:48:50.771946+0000","last_unstale":"2024-04-21T08:48:52.929245+0000","last_undegraded":"2024-04-21T08:48:52.929245+0000","last_fullsized":"2024-04-21T08:48:52.929245+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:03:56.994581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:11.843658+0000","last_change":"2024-04-21T08:48:45.553946+0000","last_active":"2024-04-21T08:49:11.843658+0000","last_peered":"2024-04-21T08:49:11.843658+0000","last_clean":"2024-04-21T08:49:11.843658+0000","last_became_active":"2024-04-21T08:48:45.553410+0000","last_became_peered":"2024-04-21T08:48:45.553410+0000","last_unstale":"2024-04-21T08:49:11.843658+0000","last_undegraded":"2024-04-21T08:49:11.843658+0000","last_fullsized":"2024-04-21T08:49:11.843658+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:24:02.387250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268108+0000","last_change":"2024-04-21T08:48:46.471815+0000","last_active":"2024-04-21T08:48:53.268108+0000","last_peered":"2024-04-21T08:48:53.268108+0000","last_clean":"2024-04-21T08:48:53.268108+0000","last_became_active":"2024-04-21T08:48:46.471492+0000","last_became_peered":"2024-04-21T08:48:46.471492+0000","last_unstale":"2024-04-21T08:48:53.268108+0000","last_undegraded":"2024-04-21T08:48:53.268108+0000","last_fullsized":"2024-04-21T08:48:53.268108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:26.051166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929308+0000","last_change":"2024-04-21T08:48:50.767916+0000","last_active":"2024-04-21T08:48:52.929308+0000","last_peered":"2024-04-21T08:48:52.929308+0000","last_clean":"2024-04-21T08:48:52.929308+0000","last_became_active":"2024-04-21T08:48:50.767326+0000","last_became_peered":"2024-04-21T08:48:50.767326+0000","last_unstale":"2024-04-21T08:48:52.929308+0000","last_undegraded":"2024-04-21T08:48:52.929308+0000","last_fullsized":"2024-04-21T08:48:52.929308+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:05:02.185369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536460+0000","last_change":"2024-04-21T08:48:45.549982+0000","last_active":"2024-04-21T08:48:52.536460+0000","last_peered":"2024-04-21T08:48:52.536460+0000","last_clean":"2024-04-21T08:48:52.536460+0000","last_became_active":"2024-04-21T08:48:45.549611+0000","last_became_peered":"2024-04-21T08:48:45.549611+0000","last_unstale":"2024-04-21T08:48:52.536460+0000","last_undegraded":"2024-04-21T08:48:52.536460+0000","last_fullsized":"2024-04-21T08:48:52.536460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:46.035388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269184+0000","last_change":"2024-04-21T08:48:46.459068+0000","last_active":"2024-04-21T08:48:53.269184+0000","last_peered":"2024-04-21T08:48:53.269184+0000","last_clean":"2024-04-21T08:48:53.269184+0000","last_became_active":"2024-04-21T08:48:46.458686+0000","last_became_peered":"2024-04-21T08:48:46.458686+0000","last_unstale":"2024-04-21T08:48:53.269184+0000","last_undegraded":"2024-04-21T08:48:53.269184+0000","last_fullsized":"2024-04-21T08:48:53.269184+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:31.373451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766215+0000","last_change":"2024-04-21T08:48:50.761392+0000","last_active":"2024-04-21T08:48:51.766215+0000","last_peered":"2024-04-21T08:48:51.766215+0000","last_clean":"2024-04-21T08:48:51.766215+0000","last_became_active":"2024-04-21T08:48:50.760917+0000","last_became_peered":"2024-04-21T08:48:50.760917+0000","last_unstale":"2024-04-21T08:48:51.766215+0000","last_undegraded":"2024-04-21T08:48:51.766215+0000","last_fullsized":"2024-04-21T08:48:51.766215+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:19.070502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:12.848261+0000","last_change":"2024-04-21T08:48:45.553686+0000","last_active":"2024-04-21T08:49:12.848261+0000","last_peered":"2024-04-21T08:49:12.848261+0000","last_clean":"2024-04-21T08:49:12.848261+0000","last_became_active":"2024-04-21T08:48:45.553322+0000","last_became_peered":"2024-04-21T08:48:45.553322+0000","last_unstale":"2024-04-21T08:49:12.848261+0000","last_undegraded":"2024-04-21T08:49:12.848261+0000","last_fullsized":"2024-04-21T08:49:12.848261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:15:24.320704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536398+0000","last_change":"2024-04-21T08:48:46.474555+0000","last_active":"2024-04-21T08:48:52.536398+0000","last_peered":"2024-04-21T08:48:52.536398+0000","last_clean":"2024-04-21T08:48:52.536398+0000","last_became_active":"2024-04-21T08:48:46.474185+0000","last_became_peered":"2024-04-21T08:48:46.474185+0000","last_unstale":"2024-04-21T08:48:52.536398+0000","last_undegraded":"2024-04-21T08:48:52.536398+0000","last_fullsized":"2024-04-21T08:48:52.536398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:52.884994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765910+0000","last_change":"2024-04-21T08:48:50.763181+0000","last_active":"2024-04-21T08:48:51.765910+0000","last_peered":"2024-04-21T08:48:51.765910+0000","last_clean":"2024-04-21T08:48:51.765910+0000","last_became_active":"2024-04-21T08:48:50.762718+0000","last_became_peered":"2024-04-21T08:48:50.762718+0000","last_unstale":"2024-04-21T08:48:51.765910+0000","last_undegraded":"2024-04-21T08:48:51.765910+0000","last_fullsized":"2024-04-21T08:48:51.765910+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:16:23.204605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.167065+0000","last_change":"2024-04-21T08:48:45.547768+0000","last_active":"2024-04-21T08:48:52.167065+0000","last_peered":"2024-04-21T08:48:52.167065+0000","last_clean":"2024-04-21T08:48:52.167065+0000","last_became_active":"2024-04-21T08:48:45.547456+0000","last_became_peered":"2024-04-21T08:48:45.547456+0000","last_unstale":"2024-04-21T08:48:52.167065+0000","last_undegraded":"2024-04-21T08:48:52.167065+0000","last_fullsized":"2024-04-21T08:48:52.167065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:23.502799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269131+0000","last_change":"2024-04-21T08:48:46.467988+0000","last_active":"2024-04-21T08:48:53.269131+0000","last_peered":"2024-04-21T08:48:53.269131+0000","last_clean":"2024-04-21T08:48:53.269131+0000","last_became_active":"2024-04-21T08:48:46.465212+0000","last_became_peered":"2024-04-21T08:48:46.465212+0000","last_unstale":"2024-04-21T08:48:53.269131+0000","last_undegraded":"2024-04-21T08:48:53.269131+0000","last_fullsized":"2024-04-21T08:48:53.269131+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:41.743506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268316+0000","last_change":"2024-04-21T08:48:50.758289+0000","last_active":"2024-04-21T08:48:53.268316+0000","last_peered":"2024-04-21T08:48:53.268316+0000","last_clean":"2024-04-21T08:48:53.268316+0000","last_became_active":"2024-04-21T08:48:50.757775+0000","last_became_peered":"2024-04-21T08:48:50.757775+0000","last_unstale":"2024-04-21T08:48:53.268316+0000","last_undegraded":"2024-04-21T08:48:53.268316+0000","last_fullsized":"2024-04-21T08:48:53.268316+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:56:27.246515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:11.843914+0000","last_change":"2024-04-21T08:48:45.545615+0000","last_active":"2024-04-21T08:49:11.843914+0000","last_peered":"2024-04-21T08:49:11.843914+0000","last_clean":"2024-04-21T08:49:11.843914+0000","last_became_active":"2024-04-21T08:48:45.545280+0000","last_became_peered":"2024-04-21T08:48:45.545280+0000","last_unstale":"2024-04-21T08:49:11.843914+0000","last_undegraded":"2024-04-21T08:49:11.843914+0000","last_fullsized":"2024-04-21T08:49:11.843914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:50:00.470903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267990+0000","last_change":"2024-04-21T08:48:46.477326+0000","last_active":"2024-04-21T08:48:53.267990+0000","last_peered":"2024-04-21T08:48:53.267990+0000","last_clean":"2024-04-21T08:48:53.267990+0000","last_became_active":"2024-04-21T08:48:46.476914+0000","last_became_peered":"2024-04-21T08:48:46.476914+0000","last_unstale":"2024-04-21T08:48:53.267990+0000","last_undegraded":"2024-04-21T08:48:53.267990+0000","last_fullsized":"2024-04-21T08:48:53.267990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:04.312061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269670+0000","last_change":"2024-04-21T08:48:50.767429+0000","last_active":"2024-04-21T08:48:53.269670+0000","last_peered":"2024-04-21T08:48:53.269670+0000","last_clean":"2024-04-21T08:48:53.269670+0000","last_became_active":"2024-04-21T08:48:50.767008+0000","last_became_peered":"2024-04-21T08:48:50.767008+0000","last_unstale":"2024-04-21T08:48:53.269670+0000","last_undegraded":"2024-04-21T08:48:53.269670+0000","last_fullsized":"2024-04-21T08:48:53.269670+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:26.865571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268116+0000","last_change":"2024-04-21T08:48:45.553170+0000","last_active":"2024-04-21T08:48:53.268116+0000","last_peered":"2024-04-21T08:48:53.268116+0000","last_clean":"2024-04-21T08:48:53.268116+0000","last_became_active":"2024-04-21T08:48:45.552645+0000","last_became_peered":"2024-04-21T08:48:45.552645+0000","last_unstale":"2024-04-21T08:48:53.268116+0000","last_undegraded":"2024-04-21T08:48:53.268116+0000","last_fullsized":"2024-04-21T08:48:53.268116+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:32:07.217212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765081+0000","last_change":"2024-04-21T08:48:46.478101+0000","last_active":"2024-04-21T08:48:51.765081+0000","last_peered":"2024-04-21T08:48:51.765081+0000","last_clean":"2024-04-21T08:48:51.765081+0000","last_became_active":"2024-04-21T08:48:46.477725+0000","last_became_peered":"2024-04-21T08:48:46.477725+0000","last_unstale":"2024-04-21T08:48:51.765081+0000","last_undegraded":"2024-04-21T08:48:51.765081+0000","last_fullsized":"2024-04-21T08:48:51.765081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:37:55.429569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269076+0000","last_change":"2024-04-21T08:48:50.762899+0000","last_active":"2024-04-21T08:48:53.269076+0000","last_peered":"2024-04-21T08:48:53.269076+0000","last_clean":"2024-04-21T08:48:53.269076+0000","last_became_active":"2024-04-21T08:48:50.762475+0000","last_became_peered":"2024-04-21T08:48:50.762475+0000","last_unstale":"2024-04-21T08:48:53.269076+0000","last_undegraded":"2024-04-21T08:48:53.269076+0000","last_fullsized":"2024-04-21T08:48:53.269076+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:27.540414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.832563+0000","last_change":"2024-04-21T08:48:45.550151+0000","last_active":"2024-04-21T08:49:10.832563+0000","last_peered":"2024-04-21T08:49:10.832563+0000","last_clean":"2024-04-21T08:49:10.832563+0000","last_became_active":"2024-04-21T08:48:45.549829+0000","last_became_peered":"2024-04-21T08:48:45.549829+0000","last_unstale":"2024-04-21T08:49:10.832563+0000","last_undegraded":"2024-04-21T08:49:10.832563+0000","last_fullsized":"2024-04-21T08:49:10.832563+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:14.061189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268354+0000","last_change":"2024-04-21T08:48:46.472072+0000","last_active":"2024-04-21T08:48:53.268354+0000","last_peered":"2024-04-21T08:48:53.268354+0000","last_clean":"2024-04-21T08:48:53.268354+0000","last_became_active":"2024-04-21T08:48:46.471660+0000","last_became_peered":"2024-04-21T08:48:46.471660+0000","last_unstale":"2024-04-21T08:48:53.268354+0000","last_undegraded":"2024-04-21T08:48:53.268354+0000","last_fullsized":"2024-04-21T08:48:53.268354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:30:11.843870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765072+0000","last_change":"2024-04-21T08:48:50.773021+0000","last_active":"2024-04-21T08:48:51.765072+0000","last_peered":"2024-04-21T08:48:51.765072+0000","last_clean":"2024-04-21T08:48:51.765072+0000","last_became_active":"2024-04-21T08:48:50.771746+0000","last_became_peered":"2024-04-21T08:48:50.771746+0000","last_unstale":"2024-04-21T08:48:51.765072+0000","last_undegraded":"2024-04-21T08:48:51.765072+0000","last_fullsized":"2024-04-21T08:48:51.765072+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:33:21.415967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.833388+0000","last_change":"2024-04-21T08:48:45.550376+0000","last_active":"2024-04-21T08:49:10.833388+0000","last_peered":"2024-04-21T08:49:10.833388+0000","last_clean":"2024-04-21T08:49:10.833388+0000","last_became_active":"2024-04-21T08:48:45.549594+0000","last_became_peered":"2024-04-21T08:48:45.549594+0000","last_unstale":"2024-04-21T08:49:10.833388+0000","last_undegraded":"2024-04-21T08:49:10.833388+0000","last_fullsized":"2024-04-21T08:49:10.833388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:26:27.046963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269903+0000","last_change":"2024-04-21T08:48:46.469499+0000","last_active":"2024-04-21T08:48:53.269903+0000","last_peered":"2024-04-21T08:48:53.269903+0000","last_clean":"2024-04-21T08:48:53.269903+0000","last_became_active":"2024-04-21T08:48:46.469124+0000","last_became_peered":"2024-04-21T08:48:46.469124+0000","last_unstale":"2024-04-21T08:48:53.269903+0000","last_undegraded":"2024-04-21T08:48:53.269903+0000","last_fullsized":"2024-04-21T08:48:53.269903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:58.708916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929597+0000","last_change":"2024-04-21T08:48:50.771331+0000","last_active":"2024-04-21T08:48:52.929597+0000","last_peered":"2024-04-21T08:48:52.929597+0000","last_clean":"2024-04-21T08:48:52.929597+0000","last_became_active":"2024-04-21T08:48:50.770018+0000","last_became_peered":"2024-04-21T08:48:50.770018+0000","last_unstale":"2024-04-21T08:48:52.929597+0000","last_undegraded":"2024-04-21T08:48:52.929597+0000","last_fullsized":"2024-04-21T08:48:52.929597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:19:03.416423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.832489+0000","last_change":"2024-04-21T08:48:45.559319+0000","last_active":"2024-04-21T08:49:10.832489+0000","last_peered":"2024-04-21T08:49:10.832489+0000","last_clean":"2024-04-21T08:49:10.832489+0000","last_became_active":"2024-04-21T08:48:45.558895+0000","last_became_peered":"2024-04-21T08:48:45.558895+0000","last_unstale":"2024-04-21T08:49:10.832489+0000","last_undegraded":"2024-04-21T08:49:10.832489+0000","last_fullsized":"2024-04-21T08:49:10.832489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:01.878169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268248+0000","last_change":"2024-04-21T08:48:46.466110+0000","last_active":"2024-04-21T08:48:53.268248+0000","last_peered":"2024-04-21T08:48:53.268248+0000","last_clean":"2024-04-21T08:48:53.268248+0000","last_became_active":"2024-04-21T08:48:46.464374+0000","last_became_peered":"2024-04-21T08:48:46.464374+0000","last_unstale":"2024-04-21T08:48:53.268248+0000","last_undegraded":"2024-04-21T08:48:53.268248+0000","last_fullsized":"2024-04-21T08:48:53.268248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:26.307051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765508+0000","last_change":"2024-04-21T08:48:50.759976+0000","last_active":"2024-04-21T08:48:51.765508+0000","last_peered":"2024-04-21T08:48:51.765508+0000","last_clean":"2024-04-21T08:48:51.765508+0000","last_became_active":"2024-04-21T08:48:50.759596+0000","last_became_peered":"2024-04-21T08:48:50.759596+0000","last_unstale":"2024-04-21T08:48:51.765508+0000","last_undegraded":"2024-04-21T08:48:51.765508+0000","last_fullsized":"2024-04-21T08:48:51.765508+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:09.678388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.832350+0000","last_change":"2024-04-21T08:48:45.549228+0000","last_active":"2024-04-21T08:49:10.832350+0000","last_peered":"2024-04-21T08:49:10.832350+0000","last_clean":"2024-04-21T08:49:10.832350+0000","last_became_active":"2024-04-21T08:48:45.548865+0000","last_became_peered":"2024-04-21T08:48:45.548865+0000","last_unstale":"2024-04-21T08:49:10.832350+0000","last_undegraded":"2024-04-21T08:49:10.832350+0000","last_fullsized":"2024-04-21T08:49:10.832350+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:28.620968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536627+0000","last_change":"2024-04-21T08:48:46.470322+0000","last_active":"2024-04-21T08:48:52.536627+0000","last_peered":"2024-04-21T08:48:52.536627+0000","last_clean":"2024-04-21T08:48:52.536627+0000","last_became_active":"2024-04-21T08:48:46.467225+0000","last_became_peered":"2024-04-21T08:48:46.467225+0000","last_unstale":"2024-04-21T08:48:52.536627+0000","last_undegraded":"2024-04-21T08:48:52.536627+0000","last_fullsized":"2024-04-21T08:48:52.536627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:31:17.390926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269634+0000","last_change":"2024-04-21T08:48:50.770982+0000","last_active":"2024-04-21T08:48:53.269634+0000","last_peered":"2024-04-21T08:48:53.269634+0000","last_clean":"2024-04-21T08:48:53.269634+0000","last_became_active":"2024-04-21T08:48:50.770312+0000","last_became_peered":"2024-04-21T08:48:50.770312+0000","last_unstale":"2024-04-21T08:48:53.269634+0000","last_undegraded":"2024-04-21T08:48:53.269634+0000","last_fullsized":"2024-04-21T08:48:53.269634+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:41:33.027351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:11.844242+0000","last_change":"2024-04-21T08:48:45.556134+0000","last_active":"2024-04-21T08:49:11.844242+0000","last_peered":"2024-04-21T08:49:11.844242+0000","last_clean":"2024-04-21T08:49:11.844242+0000","last_became_active":"2024-04-21T08:48:45.555818+0000","last_became_peered":"2024-04-21T08:48:45.555818+0000","last_unstale":"2024-04-21T08:49:11.844242+0000","last_undegraded":"2024-04-21T08:49:11.844242+0000","last_fullsized":"2024-04-21T08:49:11.844242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:00:08.083783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268443+0000","last_change":"2024-04-21T08:48:46.472855+0000","last_active":"2024-04-21T08:48:53.268443+0000","last_peered":"2024-04-21T08:48:53.268443+0000","last_clean":"2024-04-21T08:48:53.268443+0000","last_became_active":"2024-04-21T08:48:46.472448+0000","last_became_peered":"2024-04-21T08:48:46.472448+0000","last_unstale":"2024-04-21T08:48:53.268443+0000","last_undegraded":"2024-04-21T08:48:53.268443+0000","last_fullsized":"2024-04-21T08:48:53.268443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:08:32.202762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268217+0000","last_change":"2024-04-21T08:48:50.765967+0000","last_active":"2024-04-21T08:48:53.268217+0000","last_peered":"2024-04-21T08:48:53.268217+0000","last_clean":"2024-04-21T08:48:53.268217+0000","last_became_active":"2024-04-21T08:48:50.765557+0000","last_became_peered":"2024-04-21T08:48:50.765557+0000","last_unstale":"2024-04-21T08:48:53.268217+0000","last_undegraded":"2024-04-21T08:48:53.268217+0000","last_fullsized":"2024-04-21T08:48:53.268217+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:52.044058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.834014+0000","last_change":"2024-04-21T08:48:45.559939+0000","last_active":"2024-04-21T08:49:10.834014+0000","last_peered":"2024-04-21T08:49:10.834014+0000","last_clean":"2024-04-21T08:49:10.834014+0000","last_became_active":"2024-04-21T08:48:45.559635+0000","last_became_peered":"2024-04-21T08:48:45.559635+0000","last_unstale":"2024-04-21T08:49:10.834014+0000","last_undegraded":"2024-04-21T08:49:10.834014+0000","last_fullsized":"2024-04-21T08:49:10.834014+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:17.872806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766399+0000","last_change":"2024-04-21T08:48:46.464878+0000","last_active":"2024-04-21T08:48:51.766399+0000","last_peered":"2024-04-21T08:48:51.766399+0000","last_clean":"2024-04-21T08:48:51.766399+0000","last_became_active":"2024-04-21T08:48:46.464552+0000","last_became_peered":"2024-04-21T08:48:46.464552+0000","last_unstale":"2024-04-21T08:48:51.766399+0000","last_undegraded":"2024-04-21T08:48:51.766399+0000","last_fullsized":"2024-04-21T08:48:51.766399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:46:10.558693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765430+0000","last_change":"2024-04-21T08:48:50.765551+0000","last_active":"2024-04-21T08:48:51.765430+0000","last_peered":"2024-04-21T08:48:51.765430+0000","last_clean":"2024-04-21T08:48:51.765430+0000","last_became_active":"2024-04-21T08:48:50.765016+0000","last_became_peered":"2024-04-21T08:48:50.765016+0000","last_unstale":"2024-04-21T08:48:51.765430+0000","last_undegraded":"2024-04-21T08:48:51.765430+0000","last_fullsized":"2024-04-21T08:48:51.765430+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:03:05.317720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:13.852546+0000","last_change":"2024-04-21T08:48:45.548685+0000","last_active":"2024-04-21T08:49:13.852546+0000","last_peered":"2024-04-21T08:49:13.852546+0000","last_clean":"2024-04-21T08:49:13.852546+0000","last_became_active":"2024-04-21T08:48:45.548372+0000","last_became_peered":"2024-04-21T08:48:45.548372+0000","last_unstale":"2024-04-21T08:49:13.852546+0000","last_undegraded":"2024-04-21T08:49:13.852546+0000","last_fullsized":"2024-04-21T08:49:13.852546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:44.780297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766113+0000","last_change":"2024-04-21T08:48:46.471541+0000","last_active":"2024-04-21T08:48:51.766113+0000","last_peered":"2024-04-21T08:48:51.766113+0000","last_clean":"2024-04-21T08:48:51.766113+0000","last_became_active":"2024-04-21T08:48:46.471223+0000","last_became_peered":"2024-04-21T08:48:46.471223+0000","last_unstale":"2024-04-21T08:48:51.766113+0000","last_undegraded":"2024-04-21T08:48:51.766113+0000","last_fullsized":"2024-04-21T08:48:51.766113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:24:50.867905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269088+0000","last_change":"2024-04-21T08:48:50.773941+0000","last_active":"2024-04-21T08:48:53.269088+0000","last_peered":"2024-04-21T08:48:53.269088+0000","last_clean":"2024-04-21T08:48:53.269088+0000","last_became_active":"2024-04-21T08:48:50.772081+0000","last_became_peered":"2024-04-21T08:48:50.772081+0000","last_unstale":"2024-04-21T08:48:53.269088+0000","last_undegraded":"2024-04-21T08:48:53.269088+0000","last_fullsized":"2024-04-21T08:48:53.269088+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:04.438944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:11.846261+0000","last_change":"2024-04-21T08:48:45.558444+0000","last_active":"2024-04-21T08:49:11.846261+0000","last_peered":"2024-04-21T08:49:11.846261+0000","last_clean":"2024-04-21T08:49:11.846261+0000","last_became_active":"2024-04-21T08:48:45.558157+0000","last_became_peered":"2024-04-21T08:48:45.558157+0000","last_unstale":"2024-04-21T08:49:11.846261+0000","last_undegraded":"2024-04-21T08:49:11.846261+0000","last_fullsized":"2024-04-21T08:49:11.846261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:13:37.861436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268465+0000","last_change":"2024-04-21T08:48:46.465559+0000","last_active":"2024-04-21T08:48:53.268465+0000","last_peered":"2024-04-21T08:48:53.268465+0000","last_clean":"2024-04-21T08:48:53.268465+0000","last_became_active":"2024-04-21T08:48:46.465125+0000","last_became_peered":"2024-04-21T08:48:46.465125+0000","last_unstale":"2024-04-21T08:48:53.268465+0000","last_undegraded":"2024-04-21T08:48:53.268465+0000","last_fullsized":"2024-04-21T08:48:53.268465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:52:38.423925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268403+0000","last_change":"2024-04-21T08:48:50.763609+0000","last_active":"2024-04-21T08:48:53.268403+0000","last_peered":"2024-04-21T08:48:53.268403+0000","last_clean":"2024-04-21T08:48:53.268403+0000","last_became_active":"2024-04-21T08:48:50.763144+0000","last_became_peered":"2024-04-21T08:48:50.763144+0000","last_unstale":"2024-04-21T08:48:53.268403+0000","last_undegraded":"2024-04-21T08:48:53.268403+0000","last_fullsized":"2024-04-21T08:48:53.268403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:26:18.611235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536743+0000","last_change":"2024-04-21T08:48:45.549912+0000","last_active":"2024-04-21T08:48:52.536743+0000","last_peered":"2024-04-21T08:48:52.536743+0000","last_clean":"2024-04-21T08:48:52.536743+0000","last_became_active":"2024-04-21T08:48:45.549585+0000","last_became_peered":"2024-04-21T08:48:45.549585+0000","last_unstale":"2024-04-21T08:48:52.536743+0000","last_undegraded":"2024-04-21T08:48:52.536743+0000","last_fullsized":"2024-04-21T08:48:52.536743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:58.287402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268259+0000","last_change":"2024-04-21T08:48:46.462746+0000","last_active":"2024-04-21T08:48:53.268259+0000","last_peered":"2024-04-21T08:48:53.268259+0000","last_clean":"2024-04-21T08:48:53.268259+0000","last_became_active":"2024-04-21T08:48:46.461602+0000","last_became_peered":"2024-04-21T08:48:46.461602+0000","last_unstale":"2024-04-21T08:48:53.268259+0000","last_undegraded":"2024-04-21T08:48:53.268259+0000","last_fullsized":"2024-04-21T08:48:53.268259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:52.198634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536493+0000","last_change":"2024-04-21T08:48:50.770770+0000","last_active":"2024-04-21T08:48:52.536493+0000","last_peered":"2024-04-21T08:48:52.536493+0000","last_clean":"2024-04-21T08:48:52.536493+0000","last_became_active":"2024-04-21T08:48:50.770326+0000","last_became_peered":"2024-04-21T08:48:50.770326+0000","last_unstale":"2024-04-21T08:48:52.536493+0000","last_undegraded":"2024-04-21T08:48:52.536493+0000","last_fullsized":"2024-04-21T08:48:52.536493+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:38.325155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765372+0000","last_change":"2024-04-21T08:48:45.558867+0000","last_active":"2024-04-21T08:48:51.765372+0000","last_peered":"2024-04-21T08:48:51.765372+0000","last_clean":"2024-04-21T08:48:51.765372+0000","last_became_active":"2024-04-21T08:48:45.558492+0000","last_became_peered":"2024-04-21T08:48:45.558492+0000","last_unstale":"2024-04-21T08:48:51.765372+0000","last_undegraded":"2024-04-21T08:48:51.765372+0000","last_fullsized":"2024-04-21T08:48:51.765372+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:44:55.156404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.167388+0000","last_change":"2024-04-21T08:48:46.466461+0000","last_active":"2024-04-21T08:48:52.167388+0000","last_peered":"2024-04-21T08:48:52.167388+0000","last_clean":"2024-04-21T08:48:52.167388+0000","last_became_active":"2024-04-21T08:48:46.466035+0000","last_became_peered":"2024-04-21T08:48:46.466035+0000","last_unstale":"2024-04-21T08:48:52.167388+0000","last_undegraded":"2024-04-21T08:48:52.167388+0000","last_fullsized":"2024-04-21T08:48:52.167388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:47:32.307753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268799+0000","last_change":"2024-04-21T08:48:50.767655+0000","last_active":"2024-04-21T08:48:53.268799+0000","last_peered":"2024-04-21T08:48:53.268799+0000","last_clean":"2024-04-21T08:48:53.268799+0000","last_became_active":"2024-04-21T08:48:50.767305+0000","last_became_peered":"2024-04-21T08:48:50.767305+0000","last_unstale":"2024-04-21T08:48:53.268799+0000","last_undegraded":"2024-04-21T08:48:53.268799+0000","last_fullsized":"2024-04-21T08:48:53.268799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:31.210126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766269+0000","last_change":"2024-04-21T08:48:45.552952+0000","last_active":"2024-04-21T08:48:51.766269+0000","last_peered":"2024-04-21T08:48:51.766269+0000","last_clean":"2024-04-21T08:48:51.766269+0000","last_became_active":"2024-04-21T08:48:45.552556+0000","last_became_peered":"2024-04-21T08:48:45.552556+0000","last_unstale":"2024-04-21T08:48:51.766269+0000","last_undegraded":"2024-04-21T08:48:51.766269+0000","last_fullsized":"2024-04-21T08:48:51.766269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:03.428801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.167129+0000","last_change":"2024-04-21T08:48:46.469020+0000","last_active":"2024-04-21T08:48:52.167129+0000","last_peered":"2024-04-21T08:48:52.167129+0000","last_clean":"2024-04-21T08:48:52.167129+0000","last_became_active":"2024-04-21T08:48:46.468631+0000","last_became_peered":"2024-04-21T08:48:46.468631+0000","last_unstale":"2024-04-21T08:48:52.167129+0000","last_undegraded":"2024-04-21T08:48:52.167129+0000","last_fullsized":"2024-04-21T08:48:52.167129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:08.559516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269282+0000","last_change":"2024-04-21T08:48:50.773942+0000","last_active":"2024-04-21T08:48:53.269282+0000","last_peered":"2024-04-21T08:48:53.269282+0000","last_clean":"2024-04-21T08:48:53.269282+0000","last_became_active":"2024-04-21T08:48:50.771876+0000","last_became_peered":"2024-04-21T08:48:50.771876+0000","last_unstale":"2024-04-21T08:48:53.269282+0000","last_undegraded":"2024-04-21T08:48:53.269282+0000","last_fullsized":"2024-04-21T08:48:53.269282+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:19.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268285+0000","last_change":"2024-04-21T08:48:46.462727+0000","last_active":"2024-04-21T08:48:53.268285+0000","last_peered":"2024-04-21T08:48:53.268285+0000","last_clean":"2024-04-21T08:48:53.268285+0000","last_became_active":"2024-04-21T08:48:46.461669+0000","last_became_peered":"2024-04-21T08:48:46.461669+0000","last_unstale":"2024-04-21T08:48:53.268285+0000","last_undegraded":"2024-04-21T08:48:53.268285+0000","last_fullsized":"2024-04-21T08:48:53.268285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:27.290790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:12.848619+0000","last_change":"2024-04-21T08:48:45.558541+0000","last_active":"2024-04-21T08:49:12.848619+0000","last_peered":"2024-04-21T08:49:12.848619+0000","last_clean":"2024-04-21T08:49:12.848619+0000","last_became_active":"2024-04-21T08:48:45.558186+0000","last_became_peered":"2024-04-21T08:48:45.558186+0000","last_unstale":"2024-04-21T08:49:12.848619+0000","last_undegraded":"2024-04-21T08:49:12.848619+0000","last_fullsized":"2024-04-21T08:49:12.848619+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:02.919144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766532+0000","last_change":"2024-04-21T08:48:50.770838+0000","last_active":"2024-04-21T08:48:51.766532+0000","last_peered":"2024-04-21T08:48:51.766532+0000","last_clean":"2024-04-21T08:48:51.766532+0000","last_became_active":"2024-04-21T08:48:50.770447+0000","last_became_peered":"2024-04-21T08:48:50.770447+0000","last_unstale":"2024-04-21T08:48:51.766532+0000","last_undegraded":"2024-04-21T08:48:51.766532+0000","last_fullsized":"2024-04-21T08:48:51.766532+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:48:21.010284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268552+0000","last_change":"2024-04-21T08:48:46.478145+0000","last_active":"2024-04-21T08:48:53.268552+0000","last_peered":"2024-04-21T08:48:53.268552+0000","last_clean":"2024-04-21T08:48:53.268552+0000","last_became_active":"2024-04-21T08:48:46.477813+0000","last_became_peered":"2024-04-21T08:48:46.477813+0000","last_unstale":"2024-04-21T08:48:53.268552+0000","last_undegraded":"2024-04-21T08:48:53.268552+0000","last_fullsized":"2024-04-21T08:48:53.268552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:15.917619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:12.846082+0000","last_change":"2024-04-21T08:48:45.545269+0000","last_active":"2024-04-21T08:49:12.846082+0000","last_peered":"2024-04-21T08:49:12.846082+0000","last_clean":"2024-04-21T08:49:12.846082+0000","last_became_active":"2024-04-21T08:48:45.544887+0000","last_became_peered":"2024-04-21T08:48:45.544887+0000","last_unstale":"2024-04-21T08:49:12.846082+0000","last_undegraded":"2024-04-21T08:49:12.846082+0000","last_fullsized":"2024-04-21T08:49:12.846082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:27.222570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765977+0000","last_change":"2024-04-21T08:47:34.198161+0000","last_active":"2024-04-21T08:48:51.765977+0000","last_peered":"2024-04-21T08:48:51.765977+0000","last_clean":"2024-04-21T08:48:51.765977+0000","last_became_active":"2024-04-21T08:47:34.197654+0000","last_became_peered":"2024-04-21T08:47:34.197654+0000","last_unstale":"2024-04-21T08:48:51.765977+0000","last_undegraded":"2024-04-21T08:48:51.765977+0000","last_fullsized":"2024-04-21T08:48:51.765977+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_clean_scrub_stamp":"2024-04-21T08:44:05.493464+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:58:58.295783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269380+0000","last_change":"2024-04-21T08:48:50.770826+0000","last_active":"2024-04-21T08:48:53.269380+0000","last_peered":"2024-04-21T08:48:53.269380+0000","last_clean":"2024-04-21T08:48:53.269380+0000","last_became_active":"2024-04-21T08:48:50.770372+0000","last_became_peered":"2024-04-21T08:48:50.770372+0000","last_unstale":"2024-04-21T08:48:53.269380+0000","last_undegraded":"2024-04-21T08:48:53.269380+0000","last_fullsized":"2024-04-21T08:48:53.269380+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:59.637874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268011+0000","last_change":"2024-04-21T08:48:46.459330+0000","last_active":"2024-04-21T08:48:53.268011+0000","last_peered":"2024-04-21T08:48:53.268011+0000","last_clean":"2024-04-21T08:48:53.268011+0000","last_became_active":"2024-04-21T08:48:46.458963+0000","last_became_peered":"2024-04-21T08:48:46.458963+0000","last_unstale":"2024-04-21T08:48:53.268011+0000","last_undegraded":"2024-04-21T08:48:53.268011+0000","last_fullsized":"2024-04-21T08:48:53.268011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:10.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:11.843467+0000","last_change":"2024-04-21T08:48:45.552850+0000","last_active":"2024-04-21T08:49:11.843467+0000","last_peered":"2024-04-21T08:49:11.843467+0000","last_clean":"2024-04-21T08:49:11.843467+0000","last_became_active":"2024-04-21T08:48:45.552551+0000","last_became_peered":"2024-04-21T08:48:45.552551+0000","last_unstale":"2024-04-21T08:49:11.843467+0000","last_undegraded":"2024-04-21T08:49:11.843467+0000","last_fullsized":"2024-04-21T08:49:11.843467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:54.719557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268014+0000","last_change":"2024-04-21T08:48:50.765339+0000","last_active":"2024-04-21T08:48:53.268014+0000","last_peered":"2024-04-21T08:48:53.268014+0000","last_clean":"2024-04-21T08:48:53.268014+0000","last_became_active":"2024-04-21T08:48:50.764858+0000","last_became_peered":"2024-04-21T08:48:50.764858+0000","last_unstale":"2024-04-21T08:48:53.268014+0000","last_undegraded":"2024-04-21T08:48:53.268014+0000","last_fullsized":"2024-04-21T08:48:53.268014+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:26.075041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536836+0000","last_change":"2024-04-21T08:48:45.557768+0000","last_active":"2024-04-21T08:48:52.536836+0000","last_peered":"2024-04-21T08:48:52.536836+0000","last_clean":"2024-04-21T08:48:52.536836+0000","last_became_active":"2024-04-21T08:48:45.557448+0000","last_became_peered":"2024-04-21T08:48:45.557448+0000","last_unstale":"2024-04-21T08:48:52.536836+0000","last_undegraded":"2024-04-21T08:48:52.536836+0000","last_fullsized":"2024-04-21T08:48:52.536836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:09.063913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269094+0000","last_change":"2024-04-21T08:48:46.459425+0000","last_active":"2024-04-21T08:48:53.269094+0000","last_peered":"2024-04-21T08:48:53.269094+0000","last_clean":"2024-04-21T08:48:53.269094+0000","last_became_active":"2024-04-21T08:48:46.459020+0000","last_became_peered":"2024-04-21T08:48:46.459020+0000","last_unstale":"2024-04-21T08:48:53.269094+0000","last_undegraded":"2024-04-21T08:48:53.269094+0000","last_fullsized":"2024-04-21T08:48:53.269094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:08:39.042825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267933+0000","last_change":"2024-04-21T08:48:50.765740+0000","last_active":"2024-04-21T08:48:53.267933+0000","last_peered":"2024-04-21T08:48:53.267933+0000","last_clean":"2024-04-21T08:48:53.267933+0000","last_became_active":"2024-04-21T08:48:50.765359+0000","last_became_peered":"2024-04-21T08:48:50.765359+0000","last_unstale":"2024-04-21T08:48:53.267933+0000","last_undegraded":"2024-04-21T08:48:53.267933+0000","last_fullsized":"2024-04-21T08:48:53.267933+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:34:42.716165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765771+0000","last_change":"2024-04-21T08:48:45.564532+0000","last_active":"2024-04-21T08:48:51.765771+0000","last_peered":"2024-04-21T08:48:51.765771+0000","last_clean":"2024-04-21T08:48:51.765771+0000","last_became_active":"2024-04-21T08:48:45.564225+0000","last_became_peered":"2024-04-21T08:48:45.564225+0000","last_unstale":"2024-04-21T08:48:51.765771+0000","last_undegraded":"2024-04-21T08:48:51.765771+0000","last_fullsized":"2024-04-21T08:48:51.765771+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:42.850371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268283+0000","last_change":"2024-04-21T08:48:46.466622+0000","last_active":"2024-04-21T08:48:53.268283+0000","last_peered":"2024-04-21T08:48:53.268283+0000","last_clean":"2024-04-21T08:48:53.268283+0000","last_became_active":"2024-04-21T08:48:46.466129+0000","last_became_peered":"2024-04-21T08:48:46.466129+0000","last_unstale":"2024-04-21T08:48:53.268283+0000","last_undegraded":"2024-04-21T08:48:53.268283+0000","last_fullsized":"2024-04-21T08:48:53.268283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:20:26.064193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765765+0000","last_change":"2024-04-21T08:48:50.759534+0000","last_active":"2024-04-21T08:48:51.765765+0000","last_peered":"2024-04-21T08:48:51.765765+0000","last_clean":"2024-04-21T08:48:51.765765+0000","last_became_active":"2024-04-21T08:48:50.759116+0000","last_became_peered":"2024-04-21T08:48:50.759116+0000","last_unstale":"2024-04-21T08:48:51.765765+0000","last_undegraded":"2024-04-21T08:48:51.765765+0000","last_fullsized":"2024-04-21T08:48:51.765765+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:02:16.200901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.832050+0000","last_change":"2024-04-21T08:48:45.545773+0000","last_active":"2024-04-21T08:49:10.832050+0000","last_peered":"2024-04-21T08:49:10.832050+0000","last_clean":"2024-04-21T08:49:10.832050+0000","last_became_active":"2024-04-21T08:48:45.545447+0000","last_became_peered":"2024-04-21T08:48:45.545447+0000","last_unstale":"2024-04-21T08:49:10.832050+0000","last_undegraded":"2024-04-21T08:49:10.832050+0000","last_fullsized":"2024-04-21T08:49:10.832050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:59:42.171602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269787+0000","last_change":"2024-04-21T08:48:46.466590+0000","last_active":"2024-04-21T08:48:53.269787+0000","last_peered":"2024-04-21T08:48:53.269787+0000","last_clean":"2024-04-21T08:48:53.269787+0000","last_became_active":"2024-04-21T08:48:46.466199+0000","last_became_peered":"2024-04-21T08:48:46.466199+0000","last_unstale":"2024-04-21T08:48:53.269787+0000","last_undegraded":"2024-04-21T08:48:53.269787+0000","last_fullsized":"2024-04-21T08:48:53.269787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:16:34.624035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929616+0000","last_change":"2024-04-21T08:48:50.771085+0000","last_active":"2024-04-21T08:48:52.929616+0000","last_peered":"2024-04-21T08:48:52.929616+0000","last_clean":"2024-04-21T08:48:52.929616+0000","last_became_active":"2024-04-21T08:48:50.770650+0000","last_became_peered":"2024-04-21T08:48:50.770650+0000","last_unstale":"2024-04-21T08:48:52.929616+0000","last_undegraded":"2024-04-21T08:48:52.929616+0000","last_fullsized":"2024-04-21T08:48:52.929616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:43:25.933231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267784+0000","last_change":"2024-04-21T08:48:45.557541+0000","last_active":"2024-04-21T08:48:53.267784+0000","last_peered":"2024-04-21T08:48:53.267784+0000","last_clean":"2024-04-21T08:48:53.267784+0000","last_became_active":"2024-04-21T08:48:45.557245+0000","last_became_peered":"2024-04-21T08:48:45.557245+0000","last_unstale":"2024-04-21T08:48:53.267784+0000","last_undegraded":"2024-04-21T08:48:53.267784+0000","last_fullsized":"2024-04-21T08:48:53.267784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:54.019706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536271+0000","last_change":"2024-04-21T08:48:46.465475+0000","last_active":"2024-04-21T08:48:52.536271+0000","last_peered":"2024-04-21T08:48:52.536271+0000","last_clean":"2024-04-21T08:48:52.536271+0000","last_became_active":"2024-04-21T08:48:46.465138+0000","last_became_peered":"2024-04-21T08:48:46.465138+0000","last_unstale":"2024-04-21T08:48:52.536271+0000","last_undegraded":"2024-04-21T08:48:52.536271+0000","last_fullsized":"2024-04-21T08:48:52.536271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:12:53.521547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765697+0000","last_change":"2024-04-21T08:48:50.765765+0000","last_active":"2024-04-21T08:48:51.765697+0000","last_peered":"2024-04-21T08:48:51.765697+0000","last_clean":"2024-04-21T08:48:51.765697+0000","last_became_active":"2024-04-21T08:48:50.765324+0000","last_became_peered":"2024-04-21T08:48:50.765324+0000","last_unstale":"2024-04-21T08:48:51.765697+0000","last_undegraded":"2024-04-21T08:48:51.765697+0000","last_fullsized":"2024-04-21T08:48:51.765697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:41:37.846768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.835090+0000","last_change":"2024-04-21T08:48:45.557400+0000","last_active":"2024-04-21T08:49:10.835090+0000","last_peered":"2024-04-21T08:49:10.835090+0000","last_clean":"2024-04-21T08:49:10.835090+0000","last_became_active":"2024-04-21T08:48:45.555012+0000","last_became_peered":"2024-04-21T08:48:45.555012+0000","last_unstale":"2024-04-21T08:49:10.835090+0000","last_undegraded":"2024-04-21T08:49:10.835090+0000","last_fullsized":"2024-04-21T08:49:10.835090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:00:31.254353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267830+0000","last_change":"2024-04-21T08:48:46.470042+0000","last_active":"2024-04-21T08:48:53.267830+0000","last_peered":"2024-04-21T08:48:53.267830+0000","last_clean":"2024-04-21T08:48:53.267830+0000","last_became_active":"2024-04-21T08:48:46.469716+0000","last_became_peered":"2024-04-21T08:48:46.469716+0000","last_unstale":"2024-04-21T08:48:53.267830+0000","last_undegraded":"2024-04-21T08:48:53.267830+0000","last_fullsized":"2024-04-21T08:48:53.267830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:09.934751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929386+0000","last_change":"2024-04-21T08:48:50.770889+0000","last_active":"2024-04-21T08:48:52.929386+0000","last_peered":"2024-04-21T08:48:52.929386+0000","last_clean":"2024-04-21T08:48:52.929386+0000","last_became_active":"2024-04-21T08:48:50.769980+0000","last_became_peered":"2024-04-21T08:48:50.769980+0000","last_unstale":"2024-04-21T08:48:52.929386+0000","last_undegraded":"2024-04-21T08:48:52.929386+0000","last_fullsized":"2024-04-21T08:48:52.929386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:44.631633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765752+0000","last_change":"2024-04-21T08:48:45.555479+0000","last_active":"2024-04-21T08:48:51.765752+0000","last_peered":"2024-04-21T08:48:51.765752+0000","last_clean":"2024-04-21T08:48:51.765752+0000","last_became_active":"2024-04-21T08:48:45.555162+0000","last_became_peered":"2024-04-21T08:48:45.555162+0000","last_unstale":"2024-04-21T08:48:51.765752+0000","last_undegraded":"2024-04-21T08:48:51.765752+0000","last_fullsized":"2024-04-21T08:48:51.765752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:15:20.262798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268350+0000","last_change":"2024-04-21T08:48:46.467962+0000","last_active":"2024-04-21T08:48:53.268350+0000","last_peered":"2024-04-21T08:48:53.268350+0000","last_clean":"2024-04-21T08:48:53.268350+0000","last_became_active":"2024-04-21T08:48:46.467561+0000","last_became_peered":"2024-04-21T08:48:46.467561+0000","last_unstale":"2024-04-21T08:48:53.268350+0000","last_undegraded":"2024-04-21T08:48:53.268350+0000","last_fullsized":"2024-04-21T08:48:53.268350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:18.262009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929425+0000","last_change":"2024-04-21T08:48:50.767678+0000","last_active":"2024-04-21T08:48:52.929425+0000","last_peered":"2024-04-21T08:48:52.929425+0000","last_clean":"2024-04-21T08:48:52.929425+0000","last_became_active":"2024-04-21T08:48:50.767309+0000","last_became_peered":"2024-04-21T08:48:50.767309+0000","last_unstale":"2024-04-21T08:48:52.929425+0000","last_undegraded":"2024-04-21T08:48:52.929425+0000","last_fullsized":"2024-04-21T08:48:52.929425+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:16:40.065043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.833734+0000","last_change":"2024-04-21T08:48:45.553321+0000","last_active":"2024-04-21T08:49:10.833734+0000","last_peered":"2024-04-21T08:49:10.833734+0000","last_clean":"2024-04-21T08:49:10.833734+0000","last_became_active":"2024-04-21T08:48:45.552871+0000","last_became_peered":"2024-04-21T08:48:45.552871+0000","last_unstale":"2024-04-21T08:49:10.833734+0000","last_undegraded":"2024-04-21T08:49:10.833734+0000","last_fullsized":"2024-04-21T08:49:10.833734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:16:27.339801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766310+0000","last_change":"2024-04-21T08:48:46.473453+0000","last_active":"2024-04-21T08:48:51.766310+0000","last_peered":"2024-04-21T08:48:51.766310+0000","last_clean":"2024-04-21T08:48:51.766310+0000","last_became_active":"2024-04-21T08:48:46.472613+0000","last_became_peered":"2024-04-21T08:48:46.472613+0000","last_unstale":"2024-04-21T08:48:51.766310+0000","last_undegraded":"2024-04-21T08:48:51.766310+0000","last_fullsized":"2024-04-21T08:48:51.766310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:43:58.995997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267848+0000","last_change":"2024-04-21T08:48:50.765282+0000","last_active":"2024-04-21T08:48:53.267848+0000","last_peered":"2024-04-21T08:48:53.267848+0000","last_clean":"2024-04-21T08:48:53.267848+0000","last_became_active":"2024-04-21T08:48:50.764891+0000","last_became_peered":"2024-04-21T08:48:50.764891+0000","last_unstale":"2024-04-21T08:48:53.267848+0000","last_undegraded":"2024-04-21T08:48:53.267848+0000","last_fullsized":"2024-04-21T08:48:53.267848+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:18:40.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:11.846684+0000","last_change":"2024-04-21T08:48:45.557378+0000","last_active":"2024-04-21T08:49:11.846684+0000","last_peered":"2024-04-21T08:49:11.846684+0000","last_clean":"2024-04-21T08:49:11.846684+0000","last_became_active":"2024-04-21T08:48:45.556997+0000","last_became_peered":"2024-04-21T08:48:45.556997+0000","last_unstale":"2024-04-21T08:49:11.846684+0000","last_undegraded":"2024-04-21T08:49:11.846684+0000","last_fullsized":"2024-04-21T08:49:11.846684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:21:16.316889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766739+0000","last_change":"2024-04-21T08:48:46.470058+0000","last_active":"2024-04-21T08:48:51.766739+0000","last_peered":"2024-04-21T08:48:51.766739+0000","last_clean":"2024-04-21T08:48:51.766739+0000","last_became_active":"2024-04-21T08:48:46.469485+0000","last_became_peered":"2024-04-21T08:48:46.469485+0000","last_unstale":"2024-04-21T08:48:51.766739+0000","last_undegraded":"2024-04-21T08:48:51.766739+0000","last_fullsized":"2024-04-21T08:48:51.766739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:33:13.212087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269309+0000","last_change":"2024-04-21T08:48:50.762944+0000","last_active":"2024-04-21T08:48:53.269309+0000","last_peered":"2024-04-21T08:48:53.269309+0000","last_clean":"2024-04-21T08:48:53.269309+0000","last_became_active":"2024-04-21T08:48:50.762508+0000","last_became_peered":"2024-04-21T08:48:50.762508+0000","last_unstale":"2024-04-21T08:48:53.269309+0000","last_undegraded":"2024-04-21T08:48:53.269309+0000","last_fullsized":"2024-04-21T08:48:53.269309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:35.393339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269324+0000","last_change":"2024-04-21T08:48:45.560361+0000","last_active":"2024-04-21T08:48:53.269324+0000","last_peered":"2024-04-21T08:48:53.269324+0000","last_clean":"2024-04-21T08:48:53.269324+0000","last_became_active":"2024-04-21T08:48:45.559946+0000","last_became_peered":"2024-04-21T08:48:45.559946+0000","last_unstale":"2024-04-21T08:48:53.269324+0000","last_undegraded":"2024-04-21T08:48:53.269324+0000","last_fullsized":"2024-04-21T08:48:53.269324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:55.599714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766253+0000","last_change":"2024-04-21T08:48:46.472706+0000","last_active":"2024-04-21T08:48:51.766253+0000","last_peered":"2024-04-21T08:48:51.766253+0000","last_clean":"2024-04-21T08:48:51.766253+0000","last_became_active":"2024-04-21T08:48:46.472378+0000","last_became_peered":"2024-04-21T08:48:46.472378+0000","last_unstale":"2024-04-21T08:48:51.766253+0000","last_undegraded":"2024-04-21T08:48:51.766253+0000","last_fullsized":"2024-04-21T08:48:51.766253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:58.228899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267980+0000","last_change":"2024-04-21T08:48:50.760610+0000","last_active":"2024-04-21T08:48:53.267980+0000","last_peered":"2024-04-21T08:48:53.267980+0000","last_clean":"2024-04-21T08:48:53.267980+0000","last_became_active":"2024-04-21T08:48:50.759991+0000","last_became_peered":"2024-04-21T08:48:50.759991+0000","last_unstale":"2024-04-21T08:48:53.267980+0000","last_undegraded":"2024-04-21T08:48:53.267980+0000","last_fullsized":"2024-04-21T08:48:53.267980+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:43.649701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269158+0000","last_change":"2024-04-21T08:48:45.560773+0000","last_active":"2024-04-21T08:48:53.269158+0000","last_peered":"2024-04-21T08:48:53.269158+0000","last_clean":"2024-04-21T08:48:53.269158+0000","last_became_active":"2024-04-21T08:48:45.560431+0000","last_became_peered":"2024-04-21T08:48:45.560431+0000","last_unstale":"2024-04-21T08:48:53.269158+0000","last_undegraded":"2024-04-21T08:48:53.269158+0000","last_fullsized":"2024-04-21T08:48:53.269158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:07.858838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766852+0000","last_change":"2024-04-21T08:48:46.475983+0000","last_active":"2024-04-21T08:48:51.766852+0000","last_peered":"2024-04-21T08:48:51.766852+0000","last_clean":"2024-04-21T08:48:51.766852+0000","last_became_active":"2024-04-21T08:48:46.475667+0000","last_became_peered":"2024-04-21T08:48:46.475667+0000","last_unstale":"2024-04-21T08:48:51.766852+0000","last_undegraded":"2024-04-21T08:48:51.766852+0000","last_fullsized":"2024-04-21T08:48:51.766852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:28:41.013111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267920+0000","last_change":"2024-04-21T08:48:46.464675+0000","last_active":"2024-04-21T08:48:53.267920+0000","last_peered":"2024-04-21T08:48:53.267920+0000","last_clean":"2024-04-21T08:48:53.267920+0000","last_became_active":"2024-04-21T08:48:46.464314+0000","last_became_peered":"2024-04-21T08:48:46.464314+0000","last_unstale":"2024-04-21T08:48:53.267920+0000","last_undegraded":"2024-04-21T08:48:53.267920+0000","last_fullsized":"2024-04-21T08:48:53.267920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:53.048400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765875+0000","last_change":"2024-04-21T08:48:46.471012+0000","last_active":"2024-04-21T08:48:51.765875+0000","last_peered":"2024-04-21T08:48:51.765875+0000","last_clean":"2024-04-21T08:48:51.765875+0000","last_became_active":"2024-04-21T08:48:46.470689+0000","last_became_peered":"2024-04-21T08:48:46.470689+0000","last_unstale":"2024-04-21T08:48:51.765875+0000","last_undegraded":"2024-04-21T08:48:51.765875+0000","last_fullsized":"2024-04-21T08:48:51.765875+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:19:04.216593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765642+0000","last_change":"2024-04-21T08:48:46.467867+0000","last_active":"2024-04-21T08:48:51.765642+0000","last_peered":"2024-04-21T08:48:51.765642+0000","last_clean":"2024-04-21T08:48:51.765642+0000","last_became_active":"2024-04-21T08:48:46.465917+0000","last_became_peered":"2024-04-21T08:48:46.465917+0000","last_unstale":"2024-04-21T08:48:51.765642+0000","last_undegraded":"2024-04-21T08:48:51.765642+0000","last_fullsized":"2024-04-21T08:48:51.765642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:19:56.824998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269547+0000","last_change":"2024-04-21T08:48:46.467159+0000","last_active":"2024-04-21T08:48:53.269547+0000","last_peered":"2024-04-21T08:48:53.269547+0000","last_clean":"2024-04-21T08:48:53.269547+0000","last_became_active":"2024-04-21T08:48:46.466639+0000","last_became_peered":"2024-04-21T08:48:46.466639+0000","last_unstale":"2024-04-21T08:48:53.269547+0000","last_undegraded":"2024-04-21T08:48:53.269547+0000","last_fullsized":"2024-04-21T08:48:53.269547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:54:44.772535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268999+0000","last_change":"2024-04-21T08:48:46.466403+0000","last_active":"2024-04-21T08:48:53.268999+0000","last_peered":"2024-04-21T08:48:53.268999+0000","last_clean":"2024-04-21T08:48:53.268999+0000","last_became_active":"2024-04-21T08:48:46.466060+0000","last_became_peered":"2024-04-21T08:48:46.466060+0000","last_unstale":"2024-04-21T08:48:53.268999+0000","last_undegraded":"2024-04-21T08:48:53.268999+0000","last_fullsized":"2024-04-21T08:48:53.268999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:14.374193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268640+0000","last_change":"2024-04-21T08:48:46.466750+0000","last_active":"2024-04-21T08:48:53.268640+0000","last_peered":"2024-04-21T08:48:53.268640+0000","last_clean":"2024-04-21T08:48:53.268640+0000","last_became_active":"2024-04-21T08:48:46.466414+0000","last_became_peered":"2024-04-21T08:48:46.466414+0000","last_unstale":"2024-04-21T08:48:53.268640+0000","last_undegraded":"2024-04-21T08:48:53.268640+0000","last_fullsized":"2024-04-21T08:48:53.268640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:56:54.553602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766364+0000","last_change":"2024-04-21T08:48:46.464098+0000","last_active":"2024-04-21T08:48:51.766364+0000","last_peered":"2024-04-21T08:48:51.766364+0000","last_clean":"2024-04-21T08:48:51.766364+0000","last_became_active":"2024-04-21T08:48:46.463756+0000","last_became_peered":"2024-04-21T08:48:46.463756+0000","last_unstale":"2024-04-21T08:48:51.766364+0000","last_undegraded":"2024-04-21T08:48:51.766364+0000","last_fullsized":"2024-04-21T08:48:51.766364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:31.725155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268456+0000","last_change":"2024-04-21T08:48:46.467372+0000","last_active":"2024-04-21T08:48:53.268456+0000","last_peered":"2024-04-21T08:48:53.268456+0000","last_clean":"2024-04-21T08:48:53.268456+0000","last_became_active":"2024-04-21T08:48:46.467043+0000","last_became_peered":"2024-04-21T08:48:46.467043+0000","last_unstale":"2024-04-21T08:48:53.268456+0000","last_undegraded":"2024-04-21T08:48:53.268456+0000","last_fullsized":"2024-04-21T08:48:53.268456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:41.889994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268179+0000","last_change":"2024-04-21T08:48:46.460604+0000","last_active":"2024-04-21T08:48:53.268179+0000","last_peered":"2024-04-21T08:48:53.268179+0000","last_clean":"2024-04-21T08:48:53.268179+0000","last_became_active":"2024-04-21T08:48:46.460276+0000","last_became_peered":"2024-04-21T08:48:46.460276+0000","last_unstale":"2024-04-21T08:48:53.268179+0000","last_undegraded":"2024-04-21T08:48:53.268179+0000","last_fullsized":"2024-04-21T08:48:53.268179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:30.194808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268750+0000","last_change":"2024-04-21T08:48:46.463665+0000","last_active":"2024-04-21T08:48:53.268750+0000","last_peered":"2024-04-21T08:48:53.268750+0000","last_clean":"2024-04-21T08:48:53.268750+0000","last_became_active":"2024-04-21T08:48:46.463309+0000","last_became_peered":"2024-04-21T08:48:46.463309+0000","last_unstale":"2024-04-21T08:48:53.268750+0000","last_undegraded":"2024-04-21T08:48:53.268750+0000","last_fullsized":"2024-04-21T08:48:53.268750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:12:36.436565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268069+0000","last_change":"2024-04-21T08:48:46.476845+0000","last_active":"2024-04-21T08:48:53.268069+0000","last_peered":"2024-04-21T08:48:53.268069+0000","last_clean":"2024-04-21T08:48:53.268069+0000","last_became_active":"2024-04-21T08:48:46.476507+0000","last_became_peered":"2024-04-21T08:48:46.476507+0000","last_unstale":"2024-04-21T08:48:53.268069+0000","last_undegraded":"2024-04-21T08:48:53.268069+0000","last_fullsized":"2024-04-21T08:48:53.268069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:47.424576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268397+0000","last_change":"2024-04-21T08:48:46.469316+0000","last_active":"2024-04-21T08:48:53.268397+0000","last_peered":"2024-04-21T08:48:53.268397+0000","last_clean":"2024-04-21T08:48:53.268397+0000","last_became_active":"2024-04-21T08:48:46.468852+0000","last_became_peered":"2024-04-21T08:48:46.468852+0000","last_unstale":"2024-04-21T08:48:53.268397+0000","last_undegraded":"2024-04-21T08:48:53.268397+0000","last_fullsized":"2024-04-21T08:48:53.268397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:05:03.664279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268355+0000","last_change":"2024-04-21T08:48:46.466703+0000","last_active":"2024-04-21T08:48:53.268355+0000","last_peered":"2024-04-21T08:48:53.268355+0000","last_clean":"2024-04-21T08:48:53.268355+0000","last_became_active":"2024-04-21T08:48:46.466395+0000","last_became_peered":"2024-04-21T08:48:46.466395+0000","last_unstale":"2024-04-21T08:48:53.268355+0000","last_undegraded":"2024-04-21T08:48:53.268355+0000","last_fullsized":"2024-04-21T08:48:53.268355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:08:56.956283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":248791,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7843,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":114,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":58125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383696,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":1071619,"data_compressed":50669,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808858,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":477841,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":1073862,"data_compressed":50669,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":501706,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182460,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":477499,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920822,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":478250,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":1069059,"data_compressed":50669,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34373,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34373,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":41821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:49:15.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:49:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.813+0000 7f15292c1640 1 -- 172.21.15.28:0/1215411006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524102050 msgr2=0x7f15241024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.813+0000 7f15292c1640 1 --2- 172.21.15.28:0/1215411006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524102050 0x7f15241024d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1518009900 tx=0x7f151802bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.813+0000 7f15292c1640 1 -- 172.21.15.28:0/1215411006 shutdown_connections 2024-04-21T08:49:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.813+0000 7f15292c1640 1 --2- 172.21.15.28:0/1215411006 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1524102a10 0x7f1524109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.813+0000 7f15292c1640 1 --2- 172.21.15.28:0/1215411006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524102050 0x7f15241024d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.813+0000 7f15292c1640 1 --2- 172.21.15.28:0/1215411006 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1524100e70 0x7f1524101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.813+0000 7f15292c1640 1 -- 172.21.15.28:0/1215411006 >> 172.21.15.28:0/1215411006 conn(0x7f15240fc470 msgr2=0x7f15240fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.813+0000 7f15292c1640 1 -- 172.21.15.28:0/1215411006 shutdown_connections 2024-04-21T08:49:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.813+0000 7f15292c1640 1 -- 172.21.15.28:0/1215411006 wait complete. 2024-04-21T08:49:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.813+0000 7f15292c1640 1 Processor -- start 2024-04-21T08:49:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15292c1640 1 -- start start 2024-04-21T08:49:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15292c1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1524100e70 0x7f1524104e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15292c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524102050 0x7f15241034c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15292c1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1524102a10 0x7f1524103a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15292c1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f152410ca10 con 0x7f1524102050 2024-04-21T08:49:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15292c1640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f152410c890 con 0x7f1524100e70 2024-04-21T08:49:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15292c1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f152410cb90 con 0x7f1524102a10 2024-04-21T08:49:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15227fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524102050 0x7f15241034c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15227fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524102050 0x7f15241034c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41884/0 (socket says 172.21.15.28:41884) 2024-04-21T08:49:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15227fc640 1 -- 172.21.15.28:0/449994368 learned_addr learned my addr 172.21.15.28:0/449994368 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f1522ffd640 1 --2- 172.21.15.28:0/449994368 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1524100e70 0x7f1524104e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15237fe640 1 --2- 172.21.15.28:0/449994368 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1524102a10 0x7f1524103a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15227fc640 1 -- 172.21.15.28:0/449994368 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1524102a10 msgr2=0x7f1524103a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15227fc640 1 --2- 172.21.15.28:0/449994368 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1524102a10 0x7f1524103a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15227fc640 1 -- 172.21.15.28:0/449994368 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1524100e70 msgr2=0x7f1524104e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15227fc640 1 --2- 172.21.15.28:0/449994368 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1524100e70 0x7f1524104e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15227fc640 1 -- 172.21.15.28:0/449994368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1524104290 con 0x7f1524102050 2024-04-21T08:49:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.814+0000 7f15237fe640 1 --2- 172.21.15.28:0/449994368 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1524102a10 0x7f1524103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.815+0000 7f15227fc640 1 --2- 172.21.15.28:0/449994368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524102050 0x7f15241034c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1518006010 tx=0x7f1518004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.815+0000 7f150bfff640 1 -- 172.21.15.28:0/449994368 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1518043020 con 0x7f1524102050 2024-04-21T08:49:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.815+0000 7f15292c1640 1 -- 172.21.15.28:0/449994368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15241a22f0 con 0x7f1524102050 2024-04-21T08:49:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.815+0000 7f150bfff640 1 -- 172.21.15.28:0/449994368 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1518004440 con 0x7f1524102050 2024-04-21T08:49:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.815+0000 7f150bfff640 1 -- 172.21.15.28:0/449994368 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1518003bf0 con 0x7f1524102050 2024-04-21T08:49:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.815+0000 7f15292c1640 1 -- 172.21.15.28:0/449994368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15241a2890 con 0x7f1524102050 2024-04-21T08:49:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.816+0000 7f15292c1640 1 -- 172.21.15.28:0/449994368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15241a2bb0 con 0x7f1524102050 2024-04-21T08:49:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.817+0000 7f150bfff640 1 -- 172.21.15.28:0/449994368 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1518046020 con 0x7f1524102050 2024-04-21T08:49:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.818+0000 7f150bfff640 1 --2- 172.21.15.28:0/449994368 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f150c0a3c40 0x7f150c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.818+0000 7f150bfff640 1 -- 172.21.15.28:0/449994368 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f15180f30e0 con 0x7f1524102050 2024-04-21T08:49:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.818+0000 7f1522ffd640 1 --2- 172.21.15.28:0/449994368 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f150c0a3c40 0x7f150c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.818+0000 7f1522ffd640 1 --2- 172.21.15.28:0/449994368 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f150c0a3c40 0x7f150c0a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1524101bc0 tx=0x7f1514005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:15.822+0000 7f150bfff640 1 -- 172.21.15.28:0/449994368 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f151804d020 con 0x7f1524102050 2024-04-21T08:49:16.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:16.008+0000 7f15292c1640 1 -- 172.21.15.28:0/449994368 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f15240fe610 con 0x7f150c0a3c40 2024-04-21T08:49:16.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:16.018+0000 7f150bfff640 1 -- 172.21.15.28:0/449994368 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340557 (secure 0 0 0) 0x7f15240fe610 con 0x7f150c0a3c40 2024-04-21T08:49:16.022 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:49:16.024 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-21T08:49:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:16.021+0000 7f15292c1640 1 -- 172.21.15.28:0/449994368 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f150c0a3c40 msgr2=0x7f150c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:16.021+0000 7f15292c1640 1 --2- 172.21.15.28:0/449994368 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f150c0a3c40 0x7f150c0a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1524101bc0 tx=0x7f1514005f90 comp rx=0 tx=0).stop 2024-04-21T08:49:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:16.021+0000 7f15292c1640 1 -- 172.21.15.28:0/449994368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524102050 msgr2=0x7f15241034c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:16.021+0000 7f15292c1640 1 --2- 172.21.15.28:0/449994368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524102050 0x7f15241034c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1518006010 tx=0x7f1518004040 comp rx=0 tx=0).stop 2024-04-21T08:49:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:16.021+0000 7f15292c1640 1 -- 172.21.15.28:0/449994368 shutdown_connections 2024-04-21T08:49:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:16.021+0000 7f15292c1640 1 --2- 172.21.15.28:0/449994368 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1524102a10 0x7f1524103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:16.021+0000 7f15292c1640 1 --2- 172.21.15.28:0/449994368 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f150c0a3c40 0x7f150c0a6100 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:16.021+0000 7f15292c1640 1 --2- 172.21.15.28:0/449994368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1524102050 0x7f15241034c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:16.021+0000 7f15292c1640 1 --2- 172.21.15.28:0/449994368 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1524100e70 0x7f1524104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:16.021+0000 7f15292c1640 1 -- 172.21.15.28:0/449994368 >> 172.21.15.28:0/449994368 conn(0x7f15240fc470 msgr2=0x7f15240fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:16.021+0000 7f15292c1640 1 -- 172.21.15.28:0/449994368 shutdown_connections 2024-04-21T08:49:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:16.021+0000 7f15292c1640 1 -- 172.21.15.28:0/449994368 wait complete. 2024-04-21T08:49:16.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:15 smithi028 ceph-mon[28003]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:49:16.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:15 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3926694048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:49:16.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:15 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1158862757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:16.033 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-04-21T08:49:15.467077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598328,"num_objects":57,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":154,"num_read_kb":134,"num_write":296,"num_write_kb":2983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":240,"ondisk_log_size":240,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369108,"kb_used_data":11616,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572716,"statfs":{"total":1151940427776,"available":1151562461184,"internally_reserved":0,"allocated":11894784,"data_stored":7540515,"data_compressed":283947,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3461,"num_objects":33,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":68,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003443"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268559+0000","last_change":"2024-04-21T08:48:46.471626+0000","last_active":"2024-04-21T08:48:53.268559+0000","last_peered":"2024-04-21T08:48:53.268559+0000","last_clean":"2024-04-21T08:48:53.268559+0000","last_became_active":"2024-04-21T08:48:46.470498+0000","last_became_peered":"2024-04-21T08:48:46.470498+0000","last_unstale":"2024-04-21T08:48:53.268559+0000","last_undegraded":"2024-04-21T08:48:53.268559+0000","last_fullsized":"2024-04-21T08:48:53.268559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:49.452181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269742+0000","last_change":"2024-04-21T08:48:46.469111+0000","last_active":"2024-04-21T08:48:53.269742+0000","last_peered":"2024-04-21T08:48:53.269742+0000","last_clean":"2024-04-21T08:48:53.269742+0000","last_became_active":"2024-04-21T08:48:46.467926+0000","last_became_peered":"2024-04-21T08:48:46.467926+0000","last_unstale":"2024-04-21T08:48:53.269742+0000","last_undegraded":"2024-04-21T08:48:53.269742+0000","last_fullsized":"2024-04-21T08:48:53.269742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:35:38.230212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269486+0000","last_change":"2024-04-21T08:48:46.475673+0000","last_active":"2024-04-21T08:48:53.269486+0000","last_peered":"2024-04-21T08:48:53.269486+0000","last_clean":"2024-04-21T08:48:53.269486+0000","last_became_active":"2024-04-21T08:48:46.474917+0000","last_became_peered":"2024-04-21T08:48:46.474917+0000","last_unstale":"2024-04-21T08:48:53.269486+0000","last_undegraded":"2024-04-21T08:48:53.269486+0000","last_fullsized":"2024-04-21T08:48:53.269486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:01.661704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929510+0000","last_change":"2024-04-21T08:48:46.468147+0000","last_active":"2024-04-21T08:48:52.929510+0000","last_peered":"2024-04-21T08:48:52.929510+0000","last_clean":"2024-04-21T08:48:52.929510+0000","last_became_active":"2024-04-21T08:48:46.467811+0000","last_became_peered":"2024-04-21T08:48:46.467811+0000","last_unstale":"2024-04-21T08:48:52.929510+0000","last_undegraded":"2024-04-21T08:48:52.929510+0000","last_fullsized":"2024-04-21T08:48:52.929510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:21.658605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536393+0000","last_change":"2024-04-21T08:48:46.471322+0000","last_active":"2024-04-21T08:48:52.536393+0000","last_peered":"2024-04-21T08:48:52.536393+0000","last_clean":"2024-04-21T08:48:52.536393+0000","last_became_active":"2024-04-21T08:48:46.470991+0000","last_became_peered":"2024-04-21T08:48:46.470991+0000","last_unstale":"2024-04-21T08:48:52.536393+0000","last_undegraded":"2024-04-21T08:48:52.536393+0000","last_fullsized":"2024-04-21T08:48:52.536393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:22.839870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.167091+0000","last_change":"2024-04-21T08:48:46.467039+0000","last_active":"2024-04-21T08:48:52.167091+0000","last_peered":"2024-04-21T08:48:52.167091+0000","last_clean":"2024-04-21T08:48:52.167091+0000","last_became_active":"2024-04-21T08:48:46.466614+0000","last_became_peered":"2024-04-21T08:48:46.466614+0000","last_unstale":"2024-04-21T08:48:52.167091+0000","last_undegraded":"2024-04-21T08:48:52.167091+0000","last_fullsized":"2024-04-21T08:48:52.167091+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:38.029215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766096+0000","last_change":"2024-04-21T08:48:46.473268+0000","last_active":"2024-04-21T08:48:51.766096+0000","last_peered":"2024-04-21T08:48:51.766096+0000","last_clean":"2024-04-21T08:48:51.766096+0000","last_became_active":"2024-04-21T08:48:46.472261+0000","last_became_peered":"2024-04-21T08:48:46.472261+0000","last_unstale":"2024-04-21T08:48:51.766096+0000","last_undegraded":"2024-04-21T08:48:51.766096+0000","last_fullsized":"2024-04-21T08:48:51.766096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:58:52.990687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268092+0000","last_change":"2024-04-21T08:48:46.472788+0000","last_active":"2024-04-21T08:48:53.268092+0000","last_peered":"2024-04-21T08:48:53.268092+0000","last_clean":"2024-04-21T08:48:53.268092+0000","last_became_active":"2024-04-21T08:48:46.472482+0000","last_became_peered":"2024-04-21T08:48:46.472482+0000","last_unstale":"2024-04-21T08:48:53.268092+0000","last_undegraded":"2024-04-21T08:48:53.268092+0000","last_fullsized":"2024-04-21T08:48:53.268092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:48.483846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268379+0000","last_change":"2024-04-21T08:48:46.460771+0000","last_active":"2024-04-21T08:48:53.268379+0000","last_peered":"2024-04-21T08:48:53.268379+0000","last_clean":"2024-04-21T08:48:53.268379+0000","last_became_active":"2024-04-21T08:48:46.460472+0000","last_became_peered":"2024-04-21T08:48:46.460472+0000","last_unstale":"2024-04-21T08:48:53.268379+0000","last_undegraded":"2024-04-21T08:48:53.268379+0000","last_fullsized":"2024-04-21T08:48:53.268379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:46:40.766245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.167473+0000","last_change":"2024-04-21T08:48:46.463475+0000","last_active":"2024-04-21T08:48:52.167473+0000","last_peered":"2024-04-21T08:48:52.167473+0000","last_clean":"2024-04-21T08:48:52.167473+0000","last_became_active":"2024-04-21T08:48:46.462559+0000","last_became_peered":"2024-04-21T08:48:46.462559+0000","last_unstale":"2024-04-21T08:48:52.167473+0000","last_undegraded":"2024-04-21T08:48:52.167473+0000","last_fullsized":"2024-04-21T08:48:52.167473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:22:06.372410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766709+0000","last_change":"2024-04-21T08:48:46.476997+0000","last_active":"2024-04-21T08:48:51.766709+0000","last_peered":"2024-04-21T08:48:51.766709+0000","last_clean":"2024-04-21T08:48:51.766709+0000","last_became_active":"2024-04-21T08:48:46.476620+0000","last_became_peered":"2024-04-21T08:48:46.476620+0000","last_unstale":"2024-04-21T08:48:51.766709+0000","last_undegraded":"2024-04-21T08:48:51.766709+0000","last_fullsized":"2024-04-21T08:48:51.766709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:37.569629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929437+0000","last_change":"2024-04-21T08:48:46.469355+0000","last_active":"2024-04-21T08:48:52.929437+0000","last_peered":"2024-04-21T08:48:52.929437+0000","last_clean":"2024-04-21T08:48:52.929437+0000","last_became_active":"2024-04-21T08:48:46.468604+0000","last_became_peered":"2024-04-21T08:48:46.468604+0000","last_unstale":"2024-04-21T08:48:52.929437+0000","last_undegraded":"2024-04-21T08:48:52.929437+0000","last_fullsized":"2024-04-21T08:48:52.929437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:23:54.500982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268515+0000","last_change":"2024-04-21T08:48:46.453116+0000","last_active":"2024-04-21T08:48:53.268515+0000","last_peered":"2024-04-21T08:48:53.268515+0000","last_clean":"2024-04-21T08:48:53.268515+0000","last_became_active":"2024-04-21T08:48:46.452607+0000","last_became_peered":"2024-04-21T08:48:46.452607+0000","last_unstale":"2024-04-21T08:48:53.268515+0000","last_undegraded":"2024-04-21T08:48:53.268515+0000","last_fullsized":"2024-04-21T08:48:53.268515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:57:04.371812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269848+0000","last_change":"2024-04-21T08:48:46.474516+0000","last_active":"2024-04-21T08:48:53.269848+0000","last_peered":"2024-04-21T08:48:53.269848+0000","last_clean":"2024-04-21T08:48:53.269848+0000","last_became_active":"2024-04-21T08:48:46.474129+0000","last_became_peered":"2024-04-21T08:48:46.474129+0000","last_unstale":"2024-04-21T08:48:53.269848+0000","last_undegraded":"2024-04-21T08:48:53.269848+0000","last_fullsized":"2024-04-21T08:48:53.269848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:40:47.784976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268217+0000","last_change":"2024-04-21T08:48:46.465205+0000","last_active":"2024-04-21T08:48:53.268217+0000","last_peered":"2024-04-21T08:48:53.268217+0000","last_clean":"2024-04-21T08:48:53.268217+0000","last_became_active":"2024-04-21T08:48:46.464797+0000","last_became_peered":"2024-04-21T08:48:46.464797+0000","last_unstale":"2024-04-21T08:48:53.268217+0000","last_undegraded":"2024-04-21T08:48:53.268217+0000","last_fullsized":"2024-04-21T08:48:53.268217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:54:31.725612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267704+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:48:53.267704+0000","last_peered":"2024-04-21T08:48:53.267704+0000","last_clean":"2024-04-21T08:48:53.267704+0000","last_became_active":"2024-04-21T08:48:46.469389+0000","last_became_peered":"2024-04-21T08:48:46.469389+0000","last_unstale":"2024-04-21T08:48:53.267704+0000","last_undegraded":"2024-04-21T08:48:53.267704+0000","last_fullsized":"2024-04-21T08:48:53.267704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:11:45.824402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268443+0000","last_change":"2024-04-21T08:48:46.470059+0000","last_active":"2024-04-21T08:48:53.268443+0000","last_peered":"2024-04-21T08:48:53.268443+0000","last_clean":"2024-04-21T08:48:53.268443+0000","last_became_active":"2024-04-21T08:48:46.469751+0000","last_became_peered":"2024-04-21T08:48:46.469751+0000","last_unstale":"2024-04-21T08:48:53.268443+0000","last_undegraded":"2024-04-21T08:48:53.268443+0000","last_fullsized":"2024-04-21T08:48:53.268443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:16.317879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765737+0000","last_change":"2024-04-21T08:48:46.470129+0000","last_active":"2024-04-21T08:48:51.765737+0000","last_peered":"2024-04-21T08:48:51.765737+0000","last_clean":"2024-04-21T08:48:51.765737+0000","last_became_active":"2024-04-21T08:48:46.469763+0000","last_became_peered":"2024-04-21T08:48:46.469763+0000","last_unstale":"2024-04-21T08:48:51.765737+0000","last_undegraded":"2024-04-21T08:48:51.765737+0000","last_fullsized":"2024-04-21T08:48:51.765737+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:47.731201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766299+0000","last_change":"2024-04-21T08:48:46.465356+0000","last_active":"2024-04-21T08:48:51.766299+0000","last_peered":"2024-04-21T08:48:51.766299+0000","last_clean":"2024-04-21T08:48:51.766299+0000","last_became_active":"2024-04-21T08:48:46.464183+0000","last_became_peered":"2024-04-21T08:48:46.464183+0000","last_unstale":"2024-04-21T08:48:51.766299+0000","last_undegraded":"2024-04-21T08:48:51.766299+0000","last_fullsized":"2024-04-21T08:48:51.766299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:37.533733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766124+0000","last_change":"2024-04-21T08:48:50.767378+0000","last_active":"2024-04-21T08:48:51.766124+0000","last_peered":"2024-04-21T08:48:51.766124+0000","last_clean":"2024-04-21T08:48:51.766124+0000","last_became_active":"2024-04-21T08:48:50.766856+0000","last_became_peered":"2024-04-21T08:48:50.766856+0000","last_unstale":"2024-04-21T08:48:51.766124+0000","last_undegraded":"2024-04-21T08:48:51.766124+0000","last_fullsized":"2024-04-21T08:48:51.766124+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:00.434540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765688+0000","last_change":"2024-04-21T08:48:45.550284+0000","last_active":"2024-04-21T08:48:51.765688+0000","last_peered":"2024-04-21T08:48:51.765688+0000","last_clean":"2024-04-21T08:48:51.765688+0000","last_became_active":"2024-04-21T08:48:45.549940+0000","last_became_peered":"2024-04-21T08:48:45.549940+0000","last_unstale":"2024-04-21T08:48:51.765688+0000","last_undegraded":"2024-04-21T08:48:51.765688+0000","last_fullsized":"2024-04-21T08:48:51.765688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:12.343299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.270040+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:48:53.270040+0000","last_peered":"2024-04-21T08:48:53.270040+0000","last_clean":"2024-04-21T08:48:53.270040+0000","last_became_active":"2024-04-21T08:48:46.469647+0000","last_became_peered":"2024-04-21T08:48:46.469647+0000","last_unstale":"2024-04-21T08:48:53.270040+0000","last_undegraded":"2024-04-21T08:48:53.270040+0000","last_fullsized":"2024-04-21T08:48:53.270040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:10:28.204554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929110+0000","last_change":"2024-04-21T08:48:50.771871+0000","last_active":"2024-04-21T08:48:52.929110+0000","last_peered":"2024-04-21T08:48:52.929110+0000","last_clean":"2024-04-21T08:48:52.929110+0000","last_became_active":"2024-04-21T08:48:50.770442+0000","last_became_peered":"2024-04-21T08:48:50.770442+0000","last_unstale":"2024-04-21T08:48:52.929110+0000","last_undegraded":"2024-04-21T08:48:52.929110+0000","last_fullsized":"2024-04-21T08:48:52.929110+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:18:57.813971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.270180+0000","last_change":"2024-04-21T08:48:45.566492+0000","last_active":"2024-04-21T08:48:53.270180+0000","last_peered":"2024-04-21T08:48:53.270180+0000","last_clean":"2024-04-21T08:48:53.270180+0000","last_became_active":"2024-04-21T08:48:45.566169+0000","last_became_peered":"2024-04-21T08:48:45.566169+0000","last_unstale":"2024-04-21T08:48:53.270180+0000","last_undegraded":"2024-04-21T08:48:53.270180+0000","last_fullsized":"2024-04-21T08:48:53.270180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:49.985664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269660+0000","last_change":"2024-04-21T08:48:46.470866+0000","last_active":"2024-04-21T08:48:53.269660+0000","last_peered":"2024-04-21T08:48:53.269660+0000","last_clean":"2024-04-21T08:48:53.269660+0000","last_became_active":"2024-04-21T08:48:46.470401+0000","last_became_peered":"2024-04-21T08:48:46.470401+0000","last_unstale":"2024-04-21T08:48:53.269660+0000","last_undegraded":"2024-04-21T08:48:53.269660+0000","last_fullsized":"2024-04-21T08:48:53.269660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:07.506212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929245+0000","last_change":"2024-04-21T08:48:50.772339+0000","last_active":"2024-04-21T08:48:52.929245+0000","last_peered":"2024-04-21T08:48:52.929245+0000","last_clean":"2024-04-21T08:48:52.929245+0000","last_became_active":"2024-04-21T08:48:50.771946+0000","last_became_peered":"2024-04-21T08:48:50.771946+0000","last_unstale":"2024-04-21T08:48:52.929245+0000","last_undegraded":"2024-04-21T08:48:52.929245+0000","last_fullsized":"2024-04-21T08:48:52.929245+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:03:56.994581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:11.843658+0000","last_change":"2024-04-21T08:48:45.553946+0000","last_active":"2024-04-21T08:49:11.843658+0000","last_peered":"2024-04-21T08:49:11.843658+0000","last_clean":"2024-04-21T08:49:11.843658+0000","last_became_active":"2024-04-21T08:48:45.553410+0000","last_became_peered":"2024-04-21T08:48:45.553410+0000","last_unstale":"2024-04-21T08:49:11.843658+0000","last_undegraded":"2024-04-21T08:49:11.843658+0000","last_fullsized":"2024-04-21T08:49:11.843658+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:24:02.387250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268108+0000","last_change":"2024-04-21T08:48:46.471815+0000","last_active":"2024-04-21T08:48:53.268108+0000","last_peered":"2024-04-21T08:48:53.268108+0000","last_clean":"2024-04-21T08:48:53.268108+0000","last_became_active":"2024-04-21T08:48:46.471492+0000","last_became_peered":"2024-04-21T08:48:46.471492+0000","last_unstale":"2024-04-21T08:48:53.268108+0000","last_undegraded":"2024-04-21T08:48:53.268108+0000","last_fullsized":"2024-04-21T08:48:53.268108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:26.051166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929308+0000","last_change":"2024-04-21T08:48:50.767916+0000","last_active":"2024-04-21T08:48:52.929308+0000","last_peered":"2024-04-21T08:48:52.929308+0000","last_clean":"2024-04-21T08:48:52.929308+0000","last_became_active":"2024-04-21T08:48:50.767326+0000","last_became_peered":"2024-04-21T08:48:50.767326+0000","last_unstale":"2024-04-21T08:48:52.929308+0000","last_undegraded":"2024-04-21T08:48:52.929308+0000","last_fullsized":"2024-04-21T08:48:52.929308+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:05:02.185369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536460+0000","last_change":"2024-04-21T08:48:45.549982+0000","last_active":"2024-04-21T08:48:52.536460+0000","last_peered":"2024-04-21T08:48:52.536460+0000","last_clean":"2024-04-21T08:48:52.536460+0000","last_became_active":"2024-04-21T08:48:45.549611+0000","last_became_peered":"2024-04-21T08:48:45.549611+0000","last_unstale":"2024-04-21T08:48:52.536460+0000","last_undegraded":"2024-04-21T08:48:52.536460+0000","last_fullsized":"2024-04-21T08:48:52.536460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:46.035388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269184+0000","last_change":"2024-04-21T08:48:46.459068+0000","last_active":"2024-04-21T08:48:53.269184+0000","last_peered":"2024-04-21T08:48:53.269184+0000","last_clean":"2024-04-21T08:48:53.269184+0000","last_became_active":"2024-04-21T08:48:46.458686+0000","last_became_peered":"2024-04-21T08:48:46.458686+0000","last_unstale":"2024-04-21T08:48:53.269184+0000","last_undegraded":"2024-04-21T08:48:53.269184+0000","last_fullsized":"2024-04-21T08:48:53.269184+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:31.373451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766215+0000","last_change":"2024-04-21T08:48:50.761392+0000","last_active":"2024-04-21T08:48:51.766215+0000","last_peered":"2024-04-21T08:48:51.766215+0000","last_clean":"2024-04-21T08:48:51.766215+0000","last_became_active":"2024-04-21T08:48:50.760917+0000","last_became_peered":"2024-04-21T08:48:50.760917+0000","last_unstale":"2024-04-21T08:48:51.766215+0000","last_undegraded":"2024-04-21T08:48:51.766215+0000","last_fullsized":"2024-04-21T08:48:51.766215+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:19.070502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:12.848261+0000","last_change":"2024-04-21T08:48:45.553686+0000","last_active":"2024-04-21T08:49:12.848261+0000","last_peered":"2024-04-21T08:49:12.848261+0000","last_clean":"2024-04-21T08:49:12.848261+0000","last_became_active":"2024-04-21T08:48:45.553322+0000","last_became_peered":"2024-04-21T08:48:45.553322+0000","last_unstale":"2024-04-21T08:49:12.848261+0000","last_undegraded":"2024-04-21T08:49:12.848261+0000","last_fullsized":"2024-04-21T08:49:12.848261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:15:24.320704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536398+0000","last_change":"2024-04-21T08:48:46.474555+0000","last_active":"2024-04-21T08:48:52.536398+0000","last_peered":"2024-04-21T08:48:52.536398+0000","last_clean":"2024-04-21T08:48:52.536398+0000","last_became_active":"2024-04-21T08:48:46.474185+0000","last_became_peered":"2024-04-21T08:48:46.474185+0000","last_unstale":"2024-04-21T08:48:52.536398+0000","last_undegraded":"2024-04-21T08:48:52.536398+0000","last_fullsized":"2024-04-21T08:48:52.536398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:52.884994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765910+0000","last_change":"2024-04-21T08:48:50.763181+0000","last_active":"2024-04-21T08:48:51.765910+0000","last_peered":"2024-04-21T08:48:51.765910+0000","last_clean":"2024-04-21T08:48:51.765910+0000","last_became_active":"2024-04-21T08:48:50.762718+0000","last_became_peered":"2024-04-21T08:48:50.762718+0000","last_unstale":"2024-04-21T08:48:51.765910+0000","last_undegraded":"2024-04-21T08:48:51.765910+0000","last_fullsized":"2024-04-21T08:48:51.765910+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:16:23.204605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.167065+0000","last_change":"2024-04-21T08:48:45.547768+0000","last_active":"2024-04-21T08:48:52.167065+0000","last_peered":"2024-04-21T08:48:52.167065+0000","last_clean":"2024-04-21T08:48:52.167065+0000","last_became_active":"2024-04-21T08:48:45.547456+0000","last_became_peered":"2024-04-21T08:48:45.547456+0000","last_unstale":"2024-04-21T08:48:52.167065+0000","last_undegraded":"2024-04-21T08:48:52.167065+0000","last_fullsized":"2024-04-21T08:48:52.167065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:23.502799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269131+0000","last_change":"2024-04-21T08:48:46.467988+0000","last_active":"2024-04-21T08:48:53.269131+0000","last_peered":"2024-04-21T08:48:53.269131+0000","last_clean":"2024-04-21T08:48:53.269131+0000","last_became_active":"2024-04-21T08:48:46.465212+0000","last_became_peered":"2024-04-21T08:48:46.465212+0000","last_unstale":"2024-04-21T08:48:53.269131+0000","last_undegraded":"2024-04-21T08:48:53.269131+0000","last_fullsized":"2024-04-21T08:48:53.269131+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:41.743506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268316+0000","last_change":"2024-04-21T08:48:50.758289+0000","last_active":"2024-04-21T08:48:53.268316+0000","last_peered":"2024-04-21T08:48:53.268316+0000","last_clean":"2024-04-21T08:48:53.268316+0000","last_became_active":"2024-04-21T08:48:50.757775+0000","last_became_peered":"2024-04-21T08:48:50.757775+0000","last_unstale":"2024-04-21T08:48:53.268316+0000","last_undegraded":"2024-04-21T08:48:53.268316+0000","last_fullsized":"2024-04-21T08:48:53.268316+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:56:27.246515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:11.843914+0000","last_change":"2024-04-21T08:48:45.545615+0000","last_active":"2024-04-21T08:49:11.843914+0000","last_peered":"2024-04-21T08:49:11.843914+0000","last_clean":"2024-04-21T08:49:11.843914+0000","last_became_active":"2024-04-21T08:48:45.545280+0000","last_became_peered":"2024-04-21T08:48:45.545280+0000","last_unstale":"2024-04-21T08:49:11.843914+0000","last_undegraded":"2024-04-21T08:49:11.843914+0000","last_fullsized":"2024-04-21T08:49:11.843914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:50:00.470903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267990+0000","last_change":"2024-04-21T08:48:46.477326+0000","last_active":"2024-04-21T08:48:53.267990+0000","last_peered":"2024-04-21T08:48:53.267990+0000","last_clean":"2024-04-21T08:48:53.267990+0000","last_became_active":"2024-04-21T08:48:46.476914+0000","last_became_peered":"2024-04-21T08:48:46.476914+0000","last_unstale":"2024-04-21T08:48:53.267990+0000","last_undegraded":"2024-04-21T08:48:53.267990+0000","last_fullsized":"2024-04-21T08:48:53.267990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:04.312061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269670+0000","last_change":"2024-04-21T08:48:50.767429+0000","last_active":"2024-04-21T08:48:53.269670+0000","last_peered":"2024-04-21T08:48:53.269670+0000","last_clean":"2024-04-21T08:48:53.269670+0000","last_became_active":"2024-04-21T08:48:50.767008+0000","last_became_peered":"2024-04-21T08:48:50.767008+0000","last_unstale":"2024-04-21T08:48:53.269670+0000","last_undegraded":"2024-04-21T08:48:53.269670+0000","last_fullsized":"2024-04-21T08:48:53.269670+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:26.865571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268116+0000","last_change":"2024-04-21T08:48:45.553170+0000","last_active":"2024-04-21T08:48:53.268116+0000","last_peered":"2024-04-21T08:48:53.268116+0000","last_clean":"2024-04-21T08:48:53.268116+0000","last_became_active":"2024-04-21T08:48:45.552645+0000","last_became_peered":"2024-04-21T08:48:45.552645+0000","last_unstale":"2024-04-21T08:48:53.268116+0000","last_undegraded":"2024-04-21T08:48:53.268116+0000","last_fullsized":"2024-04-21T08:48:53.268116+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:32:07.217212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765081+0000","last_change":"2024-04-21T08:48:46.478101+0000","last_active":"2024-04-21T08:48:51.765081+0000","last_peered":"2024-04-21T08:48:51.765081+0000","last_clean":"2024-04-21T08:48:51.765081+0000","last_became_active":"2024-04-21T08:48:46.477725+0000","last_became_peered":"2024-04-21T08:48:46.477725+0000","last_unstale":"2024-04-21T08:48:51.765081+0000","last_undegraded":"2024-04-21T08:48:51.765081+0000","last_fullsized":"2024-04-21T08:48:51.765081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:37:55.429569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269076+0000","last_change":"2024-04-21T08:48:50.762899+0000","last_active":"2024-04-21T08:48:53.269076+0000","last_peered":"2024-04-21T08:48:53.269076+0000","last_clean":"2024-04-21T08:48:53.269076+0000","last_became_active":"2024-04-21T08:48:50.762475+0000","last_became_peered":"2024-04-21T08:48:50.762475+0000","last_unstale":"2024-04-21T08:48:53.269076+0000","last_undegraded":"2024-04-21T08:48:53.269076+0000","last_fullsized":"2024-04-21T08:48:53.269076+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:27.540414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.832563+0000","last_change":"2024-04-21T08:48:45.550151+0000","last_active":"2024-04-21T08:49:10.832563+0000","last_peered":"2024-04-21T08:49:10.832563+0000","last_clean":"2024-04-21T08:49:10.832563+0000","last_became_active":"2024-04-21T08:48:45.549829+0000","last_became_peered":"2024-04-21T08:48:45.549829+0000","last_unstale":"2024-04-21T08:49:10.832563+0000","last_undegraded":"2024-04-21T08:49:10.832563+0000","last_fullsized":"2024-04-21T08:49:10.832563+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:14.061189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268354+0000","last_change":"2024-04-21T08:48:46.472072+0000","last_active":"2024-04-21T08:48:53.268354+0000","last_peered":"2024-04-21T08:48:53.268354+0000","last_clean":"2024-04-21T08:48:53.268354+0000","last_became_active":"2024-04-21T08:48:46.471660+0000","last_became_peered":"2024-04-21T08:48:46.471660+0000","last_unstale":"2024-04-21T08:48:53.268354+0000","last_undegraded":"2024-04-21T08:48:53.268354+0000","last_fullsized":"2024-04-21T08:48:53.268354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:30:11.843870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765072+0000","last_change":"2024-04-21T08:48:50.773021+0000","last_active":"2024-04-21T08:48:51.765072+0000","last_peered":"2024-04-21T08:48:51.765072+0000","last_clean":"2024-04-21T08:48:51.765072+0000","last_became_active":"2024-04-21T08:48:50.771746+0000","last_became_peered":"2024-04-21T08:48:50.771746+0000","last_unstale":"2024-04-21T08:48:51.765072+0000","last_undegraded":"2024-04-21T08:48:51.765072+0000","last_fullsized":"2024-04-21T08:48:51.765072+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:33:21.415967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.833388+0000","last_change":"2024-04-21T08:48:45.550376+0000","last_active":"2024-04-21T08:49:10.833388+0000","last_peered":"2024-04-21T08:49:10.833388+0000","last_clean":"2024-04-21T08:49:10.833388+0000","last_became_active":"2024-04-21T08:48:45.549594+0000","last_became_peered":"2024-04-21T08:48:45.549594+0000","last_unstale":"2024-04-21T08:49:10.833388+0000","last_undegraded":"2024-04-21T08:49:10.833388+0000","last_fullsized":"2024-04-21T08:49:10.833388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:26:27.046963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269903+0000","last_change":"2024-04-21T08:48:46.469499+0000","last_active":"2024-04-21T08:48:53.269903+0000","last_peered":"2024-04-21T08:48:53.269903+0000","last_clean":"2024-04-21T08:48:53.269903+0000","last_became_active":"2024-04-21T08:48:46.469124+0000","last_became_peered":"2024-04-21T08:48:46.469124+0000","last_unstale":"2024-04-21T08:48:53.269903+0000","last_undegraded":"2024-04-21T08:48:53.269903+0000","last_fullsized":"2024-04-21T08:48:53.269903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:58.708916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929597+0000","last_change":"2024-04-21T08:48:50.771331+0000","last_active":"2024-04-21T08:48:52.929597+0000","last_peered":"2024-04-21T08:48:52.929597+0000","last_clean":"2024-04-21T08:48:52.929597+0000","last_became_active":"2024-04-21T08:48:50.770018+0000","last_became_peered":"2024-04-21T08:48:50.770018+0000","last_unstale":"2024-04-21T08:48:52.929597+0000","last_undegraded":"2024-04-21T08:48:52.929597+0000","last_fullsized":"2024-04-21T08:48:52.929597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:19:03.416423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.832489+0000","last_change":"2024-04-21T08:48:45.559319+0000","last_active":"2024-04-21T08:49:10.832489+0000","last_peered":"2024-04-21T08:49:10.832489+0000","last_clean":"2024-04-21T08:49:10.832489+0000","last_became_active":"2024-04-21T08:48:45.558895+0000","last_became_peered":"2024-04-21T08:48:45.558895+0000","last_unstale":"2024-04-21T08:49:10.832489+0000","last_undegraded":"2024-04-21T08:49:10.832489+0000","last_fullsized":"2024-04-21T08:49:10.832489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:01.878169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268248+0000","last_change":"2024-04-21T08:48:46.466110+0000","last_active":"2024-04-21T08:48:53.268248+0000","last_peered":"2024-04-21T08:48:53.268248+0000","last_clean":"2024-04-21T08:48:53.268248+0000","last_became_active":"2024-04-21T08:48:46.464374+0000","last_became_peered":"2024-04-21T08:48:46.464374+0000","last_unstale":"2024-04-21T08:48:53.268248+0000","last_undegraded":"2024-04-21T08:48:53.268248+0000","last_fullsized":"2024-04-21T08:48:53.268248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:26.307051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765508+0000","last_change":"2024-04-21T08:48:50.759976+0000","last_active":"2024-04-21T08:48:51.765508+0000","last_peered":"2024-04-21T08:48:51.765508+0000","last_clean":"2024-04-21T08:48:51.765508+0000","last_became_active":"2024-04-21T08:48:50.759596+0000","last_became_peered":"2024-04-21T08:48:50.759596+0000","last_unstale":"2024-04-21T08:48:51.765508+0000","last_undegraded":"2024-04-21T08:48:51.765508+0000","last_fullsized":"2024-04-21T08:48:51.765508+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:09.678388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.832350+0000","last_change":"2024-04-21T08:48:45.549228+0000","last_active":"2024-04-21T08:49:10.832350+0000","last_peered":"2024-04-21T08:49:10.832350+0000","last_clean":"2024-04-21T08:49:10.832350+0000","last_became_active":"2024-04-21T08:48:45.548865+0000","last_became_peered":"2024-04-21T08:48:45.548865+0000","last_unstale":"2024-04-21T08:49:10.832350+0000","last_undegraded":"2024-04-21T08:49:10.832350+0000","last_fullsized":"2024-04-21T08:49:10.832350+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:28.620968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536627+0000","last_change":"2024-04-21T08:48:46.470322+0000","last_active":"2024-04-21T08:48:52.536627+0000","last_peered":"2024-04-21T08:48:52.536627+0000","last_clean":"2024-04-21T08:48:52.536627+0000","last_became_active":"2024-04-21T08:48:46.467225+0000","last_became_peered":"2024-04-21T08:48:46.467225+0000","last_unstale":"2024-04-21T08:48:52.536627+0000","last_undegraded":"2024-04-21T08:48:52.536627+0000","last_fullsized":"2024-04-21T08:48:52.536627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:31:17.390926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269634+0000","last_change":"2024-04-21T08:48:50.770982+0000","last_active":"2024-04-21T08:48:53.269634+0000","last_peered":"2024-04-21T08:48:53.269634+0000","last_clean":"2024-04-21T08:48:53.269634+0000","last_became_active":"2024-04-21T08:48:50.770312+0000","last_became_peered":"2024-04-21T08:48:50.770312+0000","last_unstale":"2024-04-21T08:48:53.269634+0000","last_undegraded":"2024-04-21T08:48:53.269634+0000","last_fullsized":"2024-04-21T08:48:53.269634+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:41:33.027351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:11.844242+0000","last_change":"2024-04-21T08:48:45.556134+0000","last_active":"2024-04-21T08:49:11.844242+0000","last_peered":"2024-04-21T08:49:11.844242+0000","last_clean":"2024-04-21T08:49:11.844242+0000","last_became_active":"2024-04-21T08:48:45.555818+0000","last_became_peered":"2024-04-21T08:48:45.555818+0000","last_unstale":"2024-04-21T08:49:11.844242+0000","last_undegraded":"2024-04-21T08:49:11.844242+0000","last_fullsized":"2024-04-21T08:49:11.844242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:00:08.083783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268443+0000","last_change":"2024-04-21T08:48:46.472855+0000","last_active":"2024-04-21T08:48:53.268443+0000","last_peered":"2024-04-21T08:48:53.268443+0000","last_clean":"2024-04-21T08:48:53.268443+0000","last_became_active":"2024-04-21T08:48:46.472448+0000","last_became_peered":"2024-04-21T08:48:46.472448+0000","last_unstale":"2024-04-21T08:48:53.268443+0000","last_undegraded":"2024-04-21T08:48:53.268443+0000","last_fullsized":"2024-04-21T08:48:53.268443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:08:32.202762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268217+0000","last_change":"2024-04-21T08:48:50.765967+0000","last_active":"2024-04-21T08:48:53.268217+0000","last_peered":"2024-04-21T08:48:53.268217+0000","last_clean":"2024-04-21T08:48:53.268217+0000","last_became_active":"2024-04-21T08:48:50.765557+0000","last_became_peered":"2024-04-21T08:48:50.765557+0000","last_unstale":"2024-04-21T08:48:53.268217+0000","last_undegraded":"2024-04-21T08:48:53.268217+0000","last_fullsized":"2024-04-21T08:48:53.268217+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:52.044058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.834014+0000","last_change":"2024-04-21T08:48:45.559939+0000","last_active":"2024-04-21T08:49:10.834014+0000","last_peered":"2024-04-21T08:49:10.834014+0000","last_clean":"2024-04-21T08:49:10.834014+0000","last_became_active":"2024-04-21T08:48:45.559635+0000","last_became_peered":"2024-04-21T08:48:45.559635+0000","last_unstale":"2024-04-21T08:49:10.834014+0000","last_undegraded":"2024-04-21T08:49:10.834014+0000","last_fullsized":"2024-04-21T08:49:10.834014+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:17.872806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766399+0000","last_change":"2024-04-21T08:48:46.464878+0000","last_active":"2024-04-21T08:48:51.766399+0000","last_peered":"2024-04-21T08:48:51.766399+0000","last_clean":"2024-04-21T08:48:51.766399+0000","last_became_active":"2024-04-21T08:48:46.464552+0000","last_became_peered":"2024-04-21T08:48:46.464552+0000","last_unstale":"2024-04-21T08:48:51.766399+0000","last_undegraded":"2024-04-21T08:48:51.766399+0000","last_fullsized":"2024-04-21T08:48:51.766399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:46:10.558693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765430+0000","last_change":"2024-04-21T08:48:50.765551+0000","last_active":"2024-04-21T08:48:51.765430+0000","last_peered":"2024-04-21T08:48:51.765430+0000","last_clean":"2024-04-21T08:48:51.765430+0000","last_became_active":"2024-04-21T08:48:50.765016+0000","last_became_peered":"2024-04-21T08:48:50.765016+0000","last_unstale":"2024-04-21T08:48:51.765430+0000","last_undegraded":"2024-04-21T08:48:51.765430+0000","last_fullsized":"2024-04-21T08:48:51.765430+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:03:05.317720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:13.852546+0000","last_change":"2024-04-21T08:48:45.548685+0000","last_active":"2024-04-21T08:49:13.852546+0000","last_peered":"2024-04-21T08:49:13.852546+0000","last_clean":"2024-04-21T08:49:13.852546+0000","last_became_active":"2024-04-21T08:48:45.548372+0000","last_became_peered":"2024-04-21T08:48:45.548372+0000","last_unstale":"2024-04-21T08:49:13.852546+0000","last_undegraded":"2024-04-21T08:49:13.852546+0000","last_fullsized":"2024-04-21T08:49:13.852546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:44.780297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766113+0000","last_change":"2024-04-21T08:48:46.471541+0000","last_active":"2024-04-21T08:48:51.766113+0000","last_peered":"2024-04-21T08:48:51.766113+0000","last_clean":"2024-04-21T08:48:51.766113+0000","last_became_active":"2024-04-21T08:48:46.471223+0000","last_became_peered":"2024-04-21T08:48:46.471223+0000","last_unstale":"2024-04-21T08:48:51.766113+0000","last_undegraded":"2024-04-21T08:48:51.766113+0000","last_fullsized":"2024-04-21T08:48:51.766113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:24:50.867905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269088+0000","last_change":"2024-04-21T08:48:50.773941+0000","last_active":"2024-04-21T08:48:53.269088+0000","last_peered":"2024-04-21T08:48:53.269088+0000","last_clean":"2024-04-21T08:48:53.269088+0000","last_became_active":"2024-04-21T08:48:50.772081+0000","last_became_peered":"2024-04-21T08:48:50.772081+0000","last_unstale":"2024-04-21T08:48:53.269088+0000","last_undegraded":"2024-04-21T08:48:53.269088+0000","last_fullsized":"2024-04-21T08:48:53.269088+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:04.438944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:11.846261+0000","last_change":"2024-04-21T08:48:45.558444+0000","last_active":"2024-04-21T08:49:11.846261+0000","last_peered":"2024-04-21T08:49:11.846261+0000","last_clean":"2024-04-21T08:49:11.846261+0000","last_became_active":"2024-04-21T08:48:45.558157+0000","last_became_peered":"2024-04-21T08:48:45.558157+0000","last_unstale":"2024-04-21T08:49:11.846261+0000","last_undegraded":"2024-04-21T08:49:11.846261+0000","last_fullsized":"2024-04-21T08:49:11.846261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:13:37.861436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268465+0000","last_change":"2024-04-21T08:48:46.465559+0000","last_active":"2024-04-21T08:48:53.268465+0000","last_peered":"2024-04-21T08:48:53.268465+0000","last_clean":"2024-04-21T08:48:53.268465+0000","last_became_active":"2024-04-21T08:48:46.465125+0000","last_became_peered":"2024-04-21T08:48:46.465125+0000","last_unstale":"2024-04-21T08:48:53.268465+0000","last_undegraded":"2024-04-21T08:48:53.268465+0000","last_fullsized":"2024-04-21T08:48:53.268465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:52:38.423925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268403+0000","last_change":"2024-04-21T08:48:50.763609+0000","last_active":"2024-04-21T08:48:53.268403+0000","last_peered":"2024-04-21T08:48:53.268403+0000","last_clean":"2024-04-21T08:48:53.268403+0000","last_became_active":"2024-04-21T08:48:50.763144+0000","last_became_peered":"2024-04-21T08:48:50.763144+0000","last_unstale":"2024-04-21T08:48:53.268403+0000","last_undegraded":"2024-04-21T08:48:53.268403+0000","last_fullsized":"2024-04-21T08:48:53.268403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:26:18.611235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536743+0000","last_change":"2024-04-21T08:48:45.549912+0000","last_active":"2024-04-21T08:48:52.536743+0000","last_peered":"2024-04-21T08:48:52.536743+0000","last_clean":"2024-04-21T08:48:52.536743+0000","last_became_active":"2024-04-21T08:48:45.549585+0000","last_became_peered":"2024-04-21T08:48:45.549585+0000","last_unstale":"2024-04-21T08:48:52.536743+0000","last_undegraded":"2024-04-21T08:48:52.536743+0000","last_fullsized":"2024-04-21T08:48:52.536743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:58.287402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268259+0000","last_change":"2024-04-21T08:48:46.462746+0000","last_active":"2024-04-21T08:48:53.268259+0000","last_peered":"2024-04-21T08:48:53.268259+0000","last_clean":"2024-04-21T08:48:53.268259+0000","last_became_active":"2024-04-21T08:48:46.461602+0000","last_became_peered":"2024-04-21T08:48:46.461602+0000","last_unstale":"2024-04-21T08:48:53.268259+0000","last_undegraded":"2024-04-21T08:48:53.268259+0000","last_fullsized":"2024-04-21T08:48:53.268259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:52.198634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536493+0000","last_change":"2024-04-21T08:48:50.770770+0000","last_active":"2024-04-21T08:48:52.536493+0000","last_peered":"2024-04-21T08:48:52.536493+0000","last_clean":"2024-04-21T08:48:52.536493+0000","last_became_active":"2024-04-21T08:48:50.770326+0000","last_became_peered":"2024-04-21T08:48:50.770326+0000","last_unstale":"2024-04-21T08:48:52.536493+0000","last_undegraded":"2024-04-21T08:48:52.536493+0000","last_fullsized":"2024-04-21T08:48:52.536493+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:38.325155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765372+0000","last_change":"2024-04-21T08:48:45.558867+0000","last_active":"2024-04-21T08:48:51.765372+0000","last_peered":"2024-04-21T08:48:51.765372+0000","last_clean":"2024-04-21T08:48:51.765372+0000","last_became_active":"2024-04-21T08:48:45.558492+0000","last_became_peered":"2024-04-21T08:48:45.558492+0000","last_unstale":"2024-04-21T08:48:51.765372+0000","last_undegraded":"2024-04-21T08:48:51.765372+0000","last_fullsized":"2024-04-21T08:48:51.765372+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:44:55.156404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.167388+0000","last_change":"2024-04-21T08:48:46.466461+0000","last_active":"2024-04-21T08:48:52.167388+0000","last_peered":"2024-04-21T08:48:52.167388+0000","last_clean":"2024-04-21T08:48:52.167388+0000","last_became_active":"2024-04-21T08:48:46.466035+0000","last_became_peered":"2024-04-21T08:48:46.466035+0000","last_unstale":"2024-04-21T08:48:52.167388+0000","last_undegraded":"2024-04-21T08:48:52.167388+0000","last_fullsized":"2024-04-21T08:48:52.167388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:47:32.307753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268799+0000","last_change":"2024-04-21T08:48:50.767655+0000","last_active":"2024-04-21T08:48:53.268799+0000","last_peered":"2024-04-21T08:48:53.268799+0000","last_clean":"2024-04-21T08:48:53.268799+0000","last_became_active":"2024-04-21T08:48:50.767305+0000","last_became_peered":"2024-04-21T08:48:50.767305+0000","last_unstale":"2024-04-21T08:48:53.268799+0000","last_undegraded":"2024-04-21T08:48:53.268799+0000","last_fullsized":"2024-04-21T08:48:53.268799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:31.210126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766269+0000","last_change":"2024-04-21T08:48:45.552952+0000","last_active":"2024-04-21T08:48:51.766269+0000","last_peered":"2024-04-21T08:48:51.766269+0000","last_clean":"2024-04-21T08:48:51.766269+0000","last_became_active":"2024-04-21T08:48:45.552556+0000","last_became_peered":"2024-04-21T08:48:45.552556+0000","last_unstale":"2024-04-21T08:48:51.766269+0000","last_undegraded":"2024-04-21T08:48:51.766269+0000","last_fullsized":"2024-04-21T08:48:51.766269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:03.428801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.167129+0000","last_change":"2024-04-21T08:48:46.469020+0000","last_active":"2024-04-21T08:48:52.167129+0000","last_peered":"2024-04-21T08:48:52.167129+0000","last_clean":"2024-04-21T08:48:52.167129+0000","last_became_active":"2024-04-21T08:48:46.468631+0000","last_became_peered":"2024-04-21T08:48:46.468631+0000","last_unstale":"2024-04-21T08:48:52.167129+0000","last_undegraded":"2024-04-21T08:48:52.167129+0000","last_fullsized":"2024-04-21T08:48:52.167129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:08.559516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269282+0000","last_change":"2024-04-21T08:48:50.773942+0000","last_active":"2024-04-21T08:48:53.269282+0000","last_peered":"2024-04-21T08:48:53.269282+0000","last_clean":"2024-04-21T08:48:53.269282+0000","last_became_active":"2024-04-21T08:48:50.771876+0000","last_became_peered":"2024-04-21T08:48:50.771876+0000","last_unstale":"2024-04-21T08:48:53.269282+0000","last_undegraded":"2024-04-21T08:48:53.269282+0000","last_fullsized":"2024-04-21T08:48:53.269282+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:19.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268285+0000","last_change":"2024-04-21T08:48:46.462727+0000","last_active":"2024-04-21T08:48:53.268285+0000","last_peered":"2024-04-21T08:48:53.268285+0000","last_clean":"2024-04-21T08:48:53.268285+0000","last_became_active":"2024-04-21T08:48:46.461669+0000","last_became_peered":"2024-04-21T08:48:46.461669+0000","last_unstale":"2024-04-21T08:48:53.268285+0000","last_undegraded":"2024-04-21T08:48:53.268285+0000","last_fullsized":"2024-04-21T08:48:53.268285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:27.290790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:12.848619+0000","last_change":"2024-04-21T08:48:45.558541+0000","last_active":"2024-04-21T08:49:12.848619+0000","last_peered":"2024-04-21T08:49:12.848619+0000","last_clean":"2024-04-21T08:49:12.848619+0000","last_became_active":"2024-04-21T08:48:45.558186+0000","last_became_peered":"2024-04-21T08:48:45.558186+0000","last_unstale":"2024-04-21T08:49:12.848619+0000","last_undegraded":"2024-04-21T08:49:12.848619+0000","last_fullsized":"2024-04-21T08:49:12.848619+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:02.919144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766532+0000","last_change":"2024-04-21T08:48:50.770838+0000","last_active":"2024-04-21T08:48:51.766532+0000","last_peered":"2024-04-21T08:48:51.766532+0000","last_clean":"2024-04-21T08:48:51.766532+0000","last_became_active":"2024-04-21T08:48:50.770447+0000","last_became_peered":"2024-04-21T08:48:50.770447+0000","last_unstale":"2024-04-21T08:48:51.766532+0000","last_undegraded":"2024-04-21T08:48:51.766532+0000","last_fullsized":"2024-04-21T08:48:51.766532+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:48:21.010284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268552+0000","last_change":"2024-04-21T08:48:46.478145+0000","last_active":"2024-04-21T08:48:53.268552+0000","last_peered":"2024-04-21T08:48:53.268552+0000","last_clean":"2024-04-21T08:48:53.268552+0000","last_became_active":"2024-04-21T08:48:46.477813+0000","last_became_peered":"2024-04-21T08:48:46.477813+0000","last_unstale":"2024-04-21T08:48:53.268552+0000","last_undegraded":"2024-04-21T08:48:53.268552+0000","last_fullsized":"2024-04-21T08:48:53.268552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:15.917619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:12.846082+0000","last_change":"2024-04-21T08:48:45.545269+0000","last_active":"2024-04-21T08:49:12.846082+0000","last_peered":"2024-04-21T08:49:12.846082+0000","last_clean":"2024-04-21T08:49:12.846082+0000","last_became_active":"2024-04-21T08:48:45.544887+0000","last_became_peered":"2024-04-21T08:48:45.544887+0000","last_unstale":"2024-04-21T08:49:12.846082+0000","last_undegraded":"2024-04-21T08:49:12.846082+0000","last_fullsized":"2024-04-21T08:49:12.846082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:27.222570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765977+0000","last_change":"2024-04-21T08:47:34.198161+0000","last_active":"2024-04-21T08:48:51.765977+0000","last_peered":"2024-04-21T08:48:51.765977+0000","last_clean":"2024-04-21T08:48:51.765977+0000","last_became_active":"2024-04-21T08:47:34.197654+0000","last_became_peered":"2024-04-21T08:47:34.197654+0000","last_unstale":"2024-04-21T08:48:51.765977+0000","last_undegraded":"2024-04-21T08:48:51.765977+0000","last_fullsized":"2024-04-21T08:48:51.765977+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_clean_scrub_stamp":"2024-04-21T08:44:05.493464+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:58:58.295783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269380+0000","last_change":"2024-04-21T08:48:50.770826+0000","last_active":"2024-04-21T08:48:53.269380+0000","last_peered":"2024-04-21T08:48:53.269380+0000","last_clean":"2024-04-21T08:48:53.269380+0000","last_became_active":"2024-04-21T08:48:50.770372+0000","last_became_peered":"2024-04-21T08:48:50.770372+0000","last_unstale":"2024-04-21T08:48:53.269380+0000","last_undegraded":"2024-04-21T08:48:53.269380+0000","last_fullsized":"2024-04-21T08:48:53.269380+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:59.637874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268011+0000","last_change":"2024-04-21T08:48:46.459330+0000","last_active":"2024-04-21T08:48:53.268011+0000","last_peered":"2024-04-21T08:48:53.268011+0000","last_clean":"2024-04-21T08:48:53.268011+0000","last_became_active":"2024-04-21T08:48:46.458963+0000","last_became_peered":"2024-04-21T08:48:46.458963+0000","last_unstale":"2024-04-21T08:48:53.268011+0000","last_undegraded":"2024-04-21T08:48:53.268011+0000","last_fullsized":"2024-04-21T08:48:53.268011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:10.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:11.843467+0000","last_change":"2024-04-21T08:48:45.552850+0000","last_active":"2024-04-21T08:49:11.843467+0000","last_peered":"2024-04-21T08:49:11.843467+0000","last_clean":"2024-04-21T08:49:11.843467+0000","last_became_active":"2024-04-21T08:48:45.552551+0000","last_became_peered":"2024-04-21T08:48:45.552551+0000","last_unstale":"2024-04-21T08:49:11.843467+0000","last_undegraded":"2024-04-21T08:49:11.843467+0000","last_fullsized":"2024-04-21T08:49:11.843467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:54.719557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268014+0000","last_change":"2024-04-21T08:48:50.765339+0000","last_active":"2024-04-21T08:48:53.268014+0000","last_peered":"2024-04-21T08:48:53.268014+0000","last_clean":"2024-04-21T08:48:53.268014+0000","last_became_active":"2024-04-21T08:48:50.764858+0000","last_became_peered":"2024-04-21T08:48:50.764858+0000","last_unstale":"2024-04-21T08:48:53.268014+0000","last_undegraded":"2024-04-21T08:48:53.268014+0000","last_fullsized":"2024-04-21T08:48:53.268014+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:26.075041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536836+0000","last_change":"2024-04-21T08:48:45.557768+0000","last_active":"2024-04-21T08:48:52.536836+0000","last_peered":"2024-04-21T08:48:52.536836+0000","last_clean":"2024-04-21T08:48:52.536836+0000","last_became_active":"2024-04-21T08:48:45.557448+0000","last_became_peered":"2024-04-21T08:48:45.557448+0000","last_unstale":"2024-04-21T08:48:52.536836+0000","last_undegraded":"2024-04-21T08:48:52.536836+0000","last_fullsized":"2024-04-21T08:48:52.536836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:09.063913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269094+0000","last_change":"2024-04-21T08:48:46.459425+0000","last_active":"2024-04-21T08:48:53.269094+0000","last_peered":"2024-04-21T08:48:53.269094+0000","last_clean":"2024-04-21T08:48:53.269094+0000","last_became_active":"2024-04-21T08:48:46.459020+0000","last_became_peered":"2024-04-21T08:48:46.459020+0000","last_unstale":"2024-04-21T08:48:53.269094+0000","last_undegraded":"2024-04-21T08:48:53.269094+0000","last_fullsized":"2024-04-21T08:48:53.269094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:08:39.042825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267933+0000","last_change":"2024-04-21T08:48:50.765740+0000","last_active":"2024-04-21T08:48:53.267933+0000","last_peered":"2024-04-21T08:48:53.267933+0000","last_clean":"2024-04-21T08:48:53.267933+0000","last_became_active":"2024-04-21T08:48:50.765359+0000","last_became_peered":"2024-04-21T08:48:50.765359+0000","last_unstale":"2024-04-21T08:48:53.267933+0000","last_undegraded":"2024-04-21T08:48:53.267933+0000","last_fullsized":"2024-04-21T08:48:53.267933+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:34:42.716165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765771+0000","last_change":"2024-04-21T08:48:45.564532+0000","last_active":"2024-04-21T08:48:51.765771+0000","last_peered":"2024-04-21T08:48:51.765771+0000","last_clean":"2024-04-21T08:48:51.765771+0000","last_became_active":"2024-04-21T08:48:45.564225+0000","last_became_peered":"2024-04-21T08:48:45.564225+0000","last_unstale":"2024-04-21T08:48:51.765771+0000","last_undegraded":"2024-04-21T08:48:51.765771+0000","last_fullsized":"2024-04-21T08:48:51.765771+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:42.850371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268283+0000","last_change":"2024-04-21T08:48:46.466622+0000","last_active":"2024-04-21T08:48:53.268283+0000","last_peered":"2024-04-21T08:48:53.268283+0000","last_clean":"2024-04-21T08:48:53.268283+0000","last_became_active":"2024-04-21T08:48:46.466129+0000","last_became_peered":"2024-04-21T08:48:46.466129+0000","last_unstale":"2024-04-21T08:48:53.268283+0000","last_undegraded":"2024-04-21T08:48:53.268283+0000","last_fullsized":"2024-04-21T08:48:53.268283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:20:26.064193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765765+0000","last_change":"2024-04-21T08:48:50.759534+0000","last_active":"2024-04-21T08:48:51.765765+0000","last_peered":"2024-04-21T08:48:51.765765+0000","last_clean":"2024-04-21T08:48:51.765765+0000","last_became_active":"2024-04-21T08:48:50.759116+0000","last_became_peered":"2024-04-21T08:48:50.759116+0000","last_unstale":"2024-04-21T08:48:51.765765+0000","last_undegraded":"2024-04-21T08:48:51.765765+0000","last_fullsized":"2024-04-21T08:48:51.765765+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:02:16.200901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.832050+0000","last_change":"2024-04-21T08:48:45.545773+0000","last_active":"2024-04-21T08:49:10.832050+0000","last_peered":"2024-04-21T08:49:10.832050+0000","last_clean":"2024-04-21T08:49:10.832050+0000","last_became_active":"2024-04-21T08:48:45.545447+0000","last_became_peered":"2024-04-21T08:48:45.545447+0000","last_unstale":"2024-04-21T08:49:10.832050+0000","last_undegraded":"2024-04-21T08:49:10.832050+0000","last_fullsized":"2024-04-21T08:49:10.832050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:59:42.171602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269787+0000","last_change":"2024-04-21T08:48:46.466590+0000","last_active":"2024-04-21T08:48:53.269787+0000","last_peered":"2024-04-21T08:48:53.269787+0000","last_clean":"2024-04-21T08:48:53.269787+0000","last_became_active":"2024-04-21T08:48:46.466199+0000","last_became_peered":"2024-04-21T08:48:46.466199+0000","last_unstale":"2024-04-21T08:48:53.269787+0000","last_undegraded":"2024-04-21T08:48:53.269787+0000","last_fullsized":"2024-04-21T08:48:53.269787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:16:34.624035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929616+0000","last_change":"2024-04-21T08:48:50.771085+0000","last_active":"2024-04-21T08:48:52.929616+0000","last_peered":"2024-04-21T08:48:52.929616+0000","last_clean":"2024-04-21T08:48:52.929616+0000","last_became_active":"2024-04-21T08:48:50.770650+0000","last_became_peered":"2024-04-21T08:48:50.770650+0000","last_unstale":"2024-04-21T08:48:52.929616+0000","last_undegraded":"2024-04-21T08:48:52.929616+0000","last_fullsized":"2024-04-21T08:48:52.929616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:43:25.933231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267784+0000","last_change":"2024-04-21T08:48:45.557541+0000","last_active":"2024-04-21T08:48:53.267784+0000","last_peered":"2024-04-21T08:48:53.267784+0000","last_clean":"2024-04-21T08:48:53.267784+0000","last_became_active":"2024-04-21T08:48:45.557245+0000","last_became_peered":"2024-04-21T08:48:45.557245+0000","last_unstale":"2024-04-21T08:48:53.267784+0000","last_undegraded":"2024-04-21T08:48:53.267784+0000","last_fullsized":"2024-04-21T08:48:53.267784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:54.019706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.536271+0000","last_change":"2024-04-21T08:48:46.465475+0000","last_active":"2024-04-21T08:48:52.536271+0000","last_peered":"2024-04-21T08:48:52.536271+0000","last_clean":"2024-04-21T08:48:52.536271+0000","last_became_active":"2024-04-21T08:48:46.465138+0000","last_became_peered":"2024-04-21T08:48:46.465138+0000","last_unstale":"2024-04-21T08:48:52.536271+0000","last_undegraded":"2024-04-21T08:48:52.536271+0000","last_fullsized":"2024-04-21T08:48:52.536271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:12:53.521547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765697+0000","last_change":"2024-04-21T08:48:50.765765+0000","last_active":"2024-04-21T08:48:51.765697+0000","last_peered":"2024-04-21T08:48:51.765697+0000","last_clean":"2024-04-21T08:48:51.765697+0000","last_became_active":"2024-04-21T08:48:50.765324+0000","last_became_peered":"2024-04-21T08:48:50.765324+0000","last_unstale":"2024-04-21T08:48:51.765697+0000","last_undegraded":"2024-04-21T08:48:51.765697+0000","last_fullsized":"2024-04-21T08:48:51.765697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:41:37.846768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.835090+0000","last_change":"2024-04-21T08:48:45.557400+0000","last_active":"2024-04-21T08:49:10.835090+0000","last_peered":"2024-04-21T08:49:10.835090+0000","last_clean":"2024-04-21T08:49:10.835090+0000","last_became_active":"2024-04-21T08:48:45.555012+0000","last_became_peered":"2024-04-21T08:48:45.555012+0000","last_unstale":"2024-04-21T08:49:10.835090+0000","last_undegraded":"2024-04-21T08:49:10.835090+0000","last_fullsized":"2024-04-21T08:49:10.835090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:00:31.254353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267830+0000","last_change":"2024-04-21T08:48:46.470042+0000","last_active":"2024-04-21T08:48:53.267830+0000","last_peered":"2024-04-21T08:48:53.267830+0000","last_clean":"2024-04-21T08:48:53.267830+0000","last_became_active":"2024-04-21T08:48:46.469716+0000","last_became_peered":"2024-04-21T08:48:46.469716+0000","last_unstale":"2024-04-21T08:48:53.267830+0000","last_undegraded":"2024-04-21T08:48:53.267830+0000","last_fullsized":"2024-04-21T08:48:53.267830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:09.934751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929386+0000","last_change":"2024-04-21T08:48:50.770889+0000","last_active":"2024-04-21T08:48:52.929386+0000","last_peered":"2024-04-21T08:48:52.929386+0000","last_clean":"2024-04-21T08:48:52.929386+0000","last_became_active":"2024-04-21T08:48:50.769980+0000","last_became_peered":"2024-04-21T08:48:50.769980+0000","last_unstale":"2024-04-21T08:48:52.929386+0000","last_undegraded":"2024-04-21T08:48:52.929386+0000","last_fullsized":"2024-04-21T08:48:52.929386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:44.631633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765752+0000","last_change":"2024-04-21T08:48:45.555479+0000","last_active":"2024-04-21T08:48:51.765752+0000","last_peered":"2024-04-21T08:48:51.765752+0000","last_clean":"2024-04-21T08:48:51.765752+0000","last_became_active":"2024-04-21T08:48:45.555162+0000","last_became_peered":"2024-04-21T08:48:45.555162+0000","last_unstale":"2024-04-21T08:48:51.765752+0000","last_undegraded":"2024-04-21T08:48:51.765752+0000","last_fullsized":"2024-04-21T08:48:51.765752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:15:20.262798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268350+0000","last_change":"2024-04-21T08:48:46.467962+0000","last_active":"2024-04-21T08:48:53.268350+0000","last_peered":"2024-04-21T08:48:53.268350+0000","last_clean":"2024-04-21T08:48:53.268350+0000","last_became_active":"2024-04-21T08:48:46.467561+0000","last_became_peered":"2024-04-21T08:48:46.467561+0000","last_unstale":"2024-04-21T08:48:53.268350+0000","last_undegraded":"2024-04-21T08:48:53.268350+0000","last_fullsized":"2024-04-21T08:48:53.268350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:18.262009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:52.929425+0000","last_change":"2024-04-21T08:48:50.767678+0000","last_active":"2024-04-21T08:48:52.929425+0000","last_peered":"2024-04-21T08:48:52.929425+0000","last_clean":"2024-04-21T08:48:52.929425+0000","last_became_active":"2024-04-21T08:48:50.767309+0000","last_became_peered":"2024-04-21T08:48:50.767309+0000","last_unstale":"2024-04-21T08:48:52.929425+0000","last_undegraded":"2024-04-21T08:48:52.929425+0000","last_fullsized":"2024-04-21T08:48:52.929425+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:16:40.065043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:10.833734+0000","last_change":"2024-04-21T08:48:45.553321+0000","last_active":"2024-04-21T08:49:10.833734+0000","last_peered":"2024-04-21T08:49:10.833734+0000","last_clean":"2024-04-21T08:49:10.833734+0000","last_became_active":"2024-04-21T08:48:45.552871+0000","last_became_peered":"2024-04-21T08:48:45.552871+0000","last_unstale":"2024-04-21T08:49:10.833734+0000","last_undegraded":"2024-04-21T08:49:10.833734+0000","last_fullsized":"2024-04-21T08:49:10.833734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:16:27.339801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766310+0000","last_change":"2024-04-21T08:48:46.473453+0000","last_active":"2024-04-21T08:48:51.766310+0000","last_peered":"2024-04-21T08:48:51.766310+0000","last_clean":"2024-04-21T08:48:51.766310+0000","last_became_active":"2024-04-21T08:48:46.472613+0000","last_became_peered":"2024-04-21T08:48:46.472613+0000","last_unstale":"2024-04-21T08:48:51.766310+0000","last_undegraded":"2024-04-21T08:48:51.766310+0000","last_fullsized":"2024-04-21T08:48:51.766310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:43:58.995997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267848+0000","last_change":"2024-04-21T08:48:50.765282+0000","last_active":"2024-04-21T08:48:53.267848+0000","last_peered":"2024-04-21T08:48:53.267848+0000","last_clean":"2024-04-21T08:48:53.267848+0000","last_became_active":"2024-04-21T08:48:50.764891+0000","last_became_peered":"2024-04-21T08:48:50.764891+0000","last_unstale":"2024-04-21T08:48:53.267848+0000","last_undegraded":"2024-04-21T08:48:53.267848+0000","last_fullsized":"2024-04-21T08:48:53.267848+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:18:40.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:49:11.846684+0000","last_change":"2024-04-21T08:48:45.557378+0000","last_active":"2024-04-21T08:49:11.846684+0000","last_peered":"2024-04-21T08:49:11.846684+0000","last_clean":"2024-04-21T08:49:11.846684+0000","last_became_active":"2024-04-21T08:48:45.556997+0000","last_became_peered":"2024-04-21T08:48:45.556997+0000","last_unstale":"2024-04-21T08:49:11.846684+0000","last_undegraded":"2024-04-21T08:49:11.846684+0000","last_fullsized":"2024-04-21T08:49:11.846684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:21:16.316889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766739+0000","last_change":"2024-04-21T08:48:46.470058+0000","last_active":"2024-04-21T08:48:51.766739+0000","last_peered":"2024-04-21T08:48:51.766739+0000","last_clean":"2024-04-21T08:48:51.766739+0000","last_became_active":"2024-04-21T08:48:46.469485+0000","last_became_peered":"2024-04-21T08:48:46.469485+0000","last_unstale":"2024-04-21T08:48:51.766739+0000","last_undegraded":"2024-04-21T08:48:51.766739+0000","last_fullsized":"2024-04-21T08:48:51.766739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:33:13.212087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269309+0000","last_change":"2024-04-21T08:48:50.762944+0000","last_active":"2024-04-21T08:48:53.269309+0000","last_peered":"2024-04-21T08:48:53.269309+0000","last_clean":"2024-04-21T08:48:53.269309+0000","last_became_active":"2024-04-21T08:48:50.762508+0000","last_became_peered":"2024-04-21T08:48:50.762508+0000","last_unstale":"2024-04-21T08:48:53.269309+0000","last_undegraded":"2024-04-21T08:48:53.269309+0000","last_fullsized":"2024-04-21T08:48:53.269309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:35.393339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269324+0000","last_change":"2024-04-21T08:48:45.560361+0000","last_active":"2024-04-21T08:48:53.269324+0000","last_peered":"2024-04-21T08:48:53.269324+0000","last_clean":"2024-04-21T08:48:53.269324+0000","last_became_active":"2024-04-21T08:48:45.559946+0000","last_became_peered":"2024-04-21T08:48:45.559946+0000","last_unstale":"2024-04-21T08:48:53.269324+0000","last_undegraded":"2024-04-21T08:48:53.269324+0000","last_fullsized":"2024-04-21T08:48:53.269324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:55.599714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766253+0000","last_change":"2024-04-21T08:48:46.472706+0000","last_active":"2024-04-21T08:48:51.766253+0000","last_peered":"2024-04-21T08:48:51.766253+0000","last_clean":"2024-04-21T08:48:51.766253+0000","last_became_active":"2024-04-21T08:48:46.472378+0000","last_became_peered":"2024-04-21T08:48:46.472378+0000","last_unstale":"2024-04-21T08:48:51.766253+0000","last_undegraded":"2024-04-21T08:48:51.766253+0000","last_fullsized":"2024-04-21T08:48:51.766253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:58.228899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267980+0000","last_change":"2024-04-21T08:48:50.760610+0000","last_active":"2024-04-21T08:48:53.267980+0000","last_peered":"2024-04-21T08:48:53.267980+0000","last_clean":"2024-04-21T08:48:53.267980+0000","last_became_active":"2024-04-21T08:48:50.759991+0000","last_became_peered":"2024-04-21T08:48:50.759991+0000","last_unstale":"2024-04-21T08:48:53.267980+0000","last_undegraded":"2024-04-21T08:48:53.267980+0000","last_fullsized":"2024-04-21T08:48:53.267980+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:43.649701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269158+0000","last_change":"2024-04-21T08:48:45.560773+0000","last_active":"2024-04-21T08:48:53.269158+0000","last_peered":"2024-04-21T08:48:53.269158+0000","last_clean":"2024-04-21T08:48:53.269158+0000","last_became_active":"2024-04-21T08:48:45.560431+0000","last_became_peered":"2024-04-21T08:48:45.560431+0000","last_unstale":"2024-04-21T08:48:53.269158+0000","last_undegraded":"2024-04-21T08:48:53.269158+0000","last_fullsized":"2024-04-21T08:48:53.269158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:07.858838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766852+0000","last_change":"2024-04-21T08:48:46.475983+0000","last_active":"2024-04-21T08:48:51.766852+0000","last_peered":"2024-04-21T08:48:51.766852+0000","last_clean":"2024-04-21T08:48:51.766852+0000","last_became_active":"2024-04-21T08:48:46.475667+0000","last_became_peered":"2024-04-21T08:48:46.475667+0000","last_unstale":"2024-04-21T08:48:51.766852+0000","last_undegraded":"2024-04-21T08:48:51.766852+0000","last_fullsized":"2024-04-21T08:48:51.766852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:28:41.013111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.267920+0000","last_change":"2024-04-21T08:48:46.464675+0000","last_active":"2024-04-21T08:48:53.267920+0000","last_peered":"2024-04-21T08:48:53.267920+0000","last_clean":"2024-04-21T08:48:53.267920+0000","last_became_active":"2024-04-21T08:48:46.464314+0000","last_became_peered":"2024-04-21T08:48:46.464314+0000","last_unstale":"2024-04-21T08:48:53.267920+0000","last_undegraded":"2024-04-21T08:48:53.267920+0000","last_fullsized":"2024-04-21T08:48:53.267920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:53.048400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765875+0000","last_change":"2024-04-21T08:48:46.471012+0000","last_active":"2024-04-21T08:48:51.765875+0000","last_peered":"2024-04-21T08:48:51.765875+0000","last_clean":"2024-04-21T08:48:51.765875+0000","last_became_active":"2024-04-21T08:48:46.470689+0000","last_became_peered":"2024-04-21T08:48:46.470689+0000","last_unstale":"2024-04-21T08:48:51.765875+0000","last_undegraded":"2024-04-21T08:48:51.765875+0000","last_fullsized":"2024-04-21T08:48:51.765875+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:19:04.216593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.765642+0000","last_change":"2024-04-21T08:48:46.467867+0000","last_active":"2024-04-21T08:48:51.765642+0000","last_peered":"2024-04-21T08:48:51.765642+0000","last_clean":"2024-04-21T08:48:51.765642+0000","last_became_active":"2024-04-21T08:48:46.465917+0000","last_became_peered":"2024-04-21T08:48:46.465917+0000","last_unstale":"2024-04-21T08:48:51.765642+0000","last_undegraded":"2024-04-21T08:48:51.765642+0000","last_fullsized":"2024-04-21T08:48:51.765642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:19:56.824998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.269547+0000","last_change":"2024-04-21T08:48:46.467159+0000","last_active":"2024-04-21T08:48:53.269547+0000","last_peered":"2024-04-21T08:48:53.269547+0000","last_clean":"2024-04-21T08:48:53.269547+0000","last_became_active":"2024-04-21T08:48:46.466639+0000","last_became_peered":"2024-04-21T08:48:46.466639+0000","last_unstale":"2024-04-21T08:48:53.269547+0000","last_undegraded":"2024-04-21T08:48:53.269547+0000","last_fullsized":"2024-04-21T08:48:53.269547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:54:44.772535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268999+0000","last_change":"2024-04-21T08:48:46.466403+0000","last_active":"2024-04-21T08:48:53.268999+0000","last_peered":"2024-04-21T08:48:53.268999+0000","last_clean":"2024-04-21T08:48:53.268999+0000","last_became_active":"2024-04-21T08:48:46.466060+0000","last_became_peered":"2024-04-21T08:48:46.466060+0000","last_unstale":"2024-04-21T08:48:53.268999+0000","last_undegraded":"2024-04-21T08:48:53.268999+0000","last_fullsized":"2024-04-21T08:48:53.268999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:14.374193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268640+0000","last_change":"2024-04-21T08:48:46.466750+0000","last_active":"2024-04-21T08:48:53.268640+0000","last_peered":"2024-04-21T08:48:53.268640+0000","last_clean":"2024-04-21T08:48:53.268640+0000","last_became_active":"2024-04-21T08:48:46.466414+0000","last_became_peered":"2024-04-21T08:48:46.466414+0000","last_unstale":"2024-04-21T08:48:53.268640+0000","last_undegraded":"2024-04-21T08:48:53.268640+0000","last_fullsized":"2024-04-21T08:48:53.268640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:56:54.553602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:51.766364+0000","last_change":"2024-04-21T08:48:46.464098+0000","last_active":"2024-04-21T08:48:51.766364+0000","last_peered":"2024-04-21T08:48:51.766364+0000","last_clean":"2024-04-21T08:48:51.766364+0000","last_became_active":"2024-04-21T08:48:46.463756+0000","last_became_peered":"2024-04-21T08:48:46.463756+0000","last_unstale":"2024-04-21T08:48:51.766364+0000","last_undegraded":"2024-04-21T08:48:51.766364+0000","last_fullsized":"2024-04-21T08:48:51.766364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:31.725155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268456+0000","last_change":"2024-04-21T08:48:46.467372+0000","last_active":"2024-04-21T08:48:53.268456+0000","last_peered":"2024-04-21T08:48:53.268456+0000","last_clean":"2024-04-21T08:48:53.268456+0000","last_became_active":"2024-04-21T08:48:46.467043+0000","last_became_peered":"2024-04-21T08:48:46.467043+0000","last_unstale":"2024-04-21T08:48:53.268456+0000","last_undegraded":"2024-04-21T08:48:53.268456+0000","last_fullsized":"2024-04-21T08:48:53.268456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:41.889994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268179+0000","last_change":"2024-04-21T08:48:46.460604+0000","last_active":"2024-04-21T08:48:53.268179+0000","last_peered":"2024-04-21T08:48:53.268179+0000","last_clean":"2024-04-21T08:48:53.268179+0000","last_became_active":"2024-04-21T08:48:46.460276+0000","last_became_peered":"2024-04-21T08:48:46.460276+0000","last_unstale":"2024-04-21T08:48:53.268179+0000","last_undegraded":"2024-04-21T08:48:53.268179+0000","last_fullsized":"2024-04-21T08:48:53.268179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:30.194808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268750+0000","last_change":"2024-04-21T08:48:46.463665+0000","last_active":"2024-04-21T08:48:53.268750+0000","last_peered":"2024-04-21T08:48:53.268750+0000","last_clean":"2024-04-21T08:48:53.268750+0000","last_became_active":"2024-04-21T08:48:46.463309+0000","last_became_peered":"2024-04-21T08:48:46.463309+0000","last_unstale":"2024-04-21T08:48:53.268750+0000","last_undegraded":"2024-04-21T08:48:53.268750+0000","last_fullsized":"2024-04-21T08:48:53.268750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:12:36.436565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268069+0000","last_change":"2024-04-21T08:48:46.476845+0000","last_active":"2024-04-21T08:48:53.268069+0000","last_peered":"2024-04-21T08:48:53.268069+0000","last_clean":"2024-04-21T08:48:53.268069+0000","last_became_active":"2024-04-21T08:48:46.476507+0000","last_became_peered":"2024-04-21T08:48:46.476507+0000","last_unstale":"2024-04-21T08:48:53.268069+0000","last_undegraded":"2024-04-21T08:48:53.268069+0000","last_fullsized":"2024-04-21T08:48:53.268069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:47.424576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268397+0000","last_change":"2024-04-21T08:48:46.469316+0000","last_active":"2024-04-21T08:48:53.268397+0000","last_peered":"2024-04-21T08:48:53.268397+0000","last_clean":"2024-04-21T08:48:53.268397+0000","last_became_active":"2024-04-21T08:48:46.468852+0000","last_became_peered":"2024-04-21T08:48:46.468852+0000","last_unstale":"2024-04-21T08:48:53.268397+0000","last_undegraded":"2024-04-21T08:48:53.268397+0000","last_fullsized":"2024-04-21T08:48:53.268397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:05:03.664279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:48:53.268355+0000","last_change":"2024-04-21T08:48:46.466703+0000","last_active":"2024-04-21T08:48:53.268355+0000","last_peered":"2024-04-21T08:48:53.268355+0000","last_clean":"2024-04-21T08:48:53.268355+0000","last_became_active":"2024-04-21T08:48:46.466395+0000","last_became_peered":"2024-04-21T08:48:46.466395+0000","last_unstale":"2024-04-21T08:48:53.268355+0000","last_undegraded":"2024-04-21T08:48:53.268355+0000","last_fullsized":"2024-04-21T08:48:53.268355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:08:56.956283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":248791,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7843,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":114,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":58125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383696,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":1071619,"data_compressed":50669,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808858,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":477841,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":1073862,"data_compressed":50669,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":501706,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182460,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":477499,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920822,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":478250,"data_compressed":14660,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":1069059,"data_compressed":50669,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34373,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34373,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":41821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:49:16.038 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-21T08:49:16.038 INFO:tasks.cephadm:Setting up client nodes... 2024-04-21T08:49:16.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-21T08:49:16.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:15 smithi171 ceph-mon[31358]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:49:16.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:15 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3926694048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:49:16.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:15 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1158862757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:16.325 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:16.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:15 smithi152 ceph-mon[31401]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:49:16.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:15 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3926694048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:49:16.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:15 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1158862757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:16 smithi028 ceph-mon[28003]: pgmap v309: 129 pgs: 129 active+clean; 584 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-04-21T08:49:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:16 smithi028 ceph-mon[28003]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:49:16.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:16 smithi028 ceph-mon[28003]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:49:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:16 smithi171 ceph-mon[31358]: pgmap v309: 129 pgs: 129 active+clean; 584 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-04-21T08:49:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:16 smithi171 ceph-mon[31358]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:49:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:16 smithi171 ceph-mon[31358]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:49:17.354 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 08:49:16 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-b[62223]: 2024-04-21T08:49:16.910+0000 7f5feb533700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-21T08:49:17.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:16 smithi152 ceph-mon[31401]: pgmap v309: 129 pgs: 129 active+clean; 584 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-04-21T08:49:17.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:16 smithi152 ceph-mon[31401]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:49:17.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:16 smithi152 ceph-mon[31401]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:49:18.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.536+0000 7fac436e4700 1 -- 172.21.15.28:0/646722337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac3c103550 msgr2=0x7fac3c105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:18.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.536+0000 7fac436e4700 1 --2- 172.21.15.28:0/646722337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac3c103550 0x7fac3c105940 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fac38009990 tx=0x7fac38009ca0 comp rx=0 tx=0).stop 2024-04-21T08:49:18.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.536+0000 7fac436e4700 1 -- 172.21.15.28:0/646722337 shutdown_connections 2024-04-21T08:49:18.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.536+0000 7fac436e4700 1 --2- 172.21.15.28:0/646722337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac3c103550 0x7fac3c105940 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:18.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.536+0000 7fac436e4700 1 --2- 172.21.15.28:0/646722337 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac3c1087b0 0x7fac3c10ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:18.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.536+0000 7fac436e4700 1 --2- 172.21.15.28:0/646722337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac3c105e80 0x7fac3c108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:18.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.536+0000 7fac436e4700 1 -- 172.21.15.28:0/646722337 >> 172.21.15.28:0/646722337 conn(0x7fac3c0fcf60 msgr2=0x7fac3c0ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:18.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.536+0000 7fac436e4700 1 -- 172.21.15.28:0/646722337 shutdown_connections 2024-04-21T08:49:18.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.536+0000 7fac436e4700 1 -- 172.21.15.28:0/646722337 wait complete. 2024-04-21T08:49:18.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.537+0000 7fac436e4700 1 Processor -- start 2024-04-21T08:49:18.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.537+0000 7fac436e4700 1 -- start start 2024-04-21T08:49:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.537+0000 7fac436e4700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac3c103550 0x7fac3c198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.537+0000 7fac436e4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac3c105e80 0x7fac3c199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.537+0000 7fac436e4700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac3c1087b0 0x7fac3c1a0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.537+0000 7fac436e4700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c10e540 con 0x7fac3c105e80 2024-04-21T08:49:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.537+0000 7fac436e4700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c10e3c0 con 0x7fac3c1087b0 2024-04-21T08:49:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.537+0000 7fac436e4700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c10e6c0 con 0x7fac3c103550 2024-04-21T08:49:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.537+0000 7fac40c7f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac3c105e80 0x7fac3c199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:18.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.538+0000 7fac40c7f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac3c105e80 0x7fac3c199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41896/0 (socket says 172.21.15.28:41896) 2024-04-21T08:49:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.538+0000 7fac40c7f700 1 -- 172.21.15.28:0/565983580 learned_addr learned my addr 172.21.15.28:0/565983580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.538+0000 7fac41c81700 1 --2- 172.21.15.28:0/565983580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac3c1087b0 0x7fac3c1a0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.538+0000 7fac41480700 1 --2- 172.21.15.28:0/565983580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac3c103550 0x7fac3c198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.538+0000 7fac40c7f700 1 -- 172.21.15.28:0/565983580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac3c103550 msgr2=0x7fac3c198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.538+0000 7fac40c7f700 1 --2- 172.21.15.28:0/565983580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac3c103550 0x7fac3c198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.538+0000 7fac40c7f700 1 -- 172.21.15.28:0/565983580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac3c1087b0 msgr2=0x7fac3c1a0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.538+0000 7fac40c7f700 1 --2- 172.21.15.28:0/565983580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac3c1087b0 0x7fac3c1a0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.538+0000 7fac40c7f700 1 -- 172.21.15.28:0/565983580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3c1a06e0 con 0x7fac3c105e80 2024-04-21T08:49:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.538+0000 7fac40c7f700 1 --2- 172.21.15.28:0/565983580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac3c105e80 0x7fac3c199030 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00c950 tx=0x7fac2c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.538+0000 7fac327fc700 1 -- 172.21.15.28:0/565983580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0078d0 con 0x7fac3c105e80 2024-04-21T08:49:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.538+0000 7fac327fc700 1 -- 172.21.15.28:0/565983580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac2c00f450 con 0x7fac3c105e80 2024-04-21T08:49:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.538+0000 7fac436e4700 1 -- 172.21.15.28:0/565983580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac3c1a09d0 con 0x7fac3c105e80 2024-04-21T08:49:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.538+0000 7fac327fc700 1 -- 172.21.15.28:0/565983580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c018660 con 0x7fac3c105e80 2024-04-21T08:49:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.538+0000 7fac436e4700 1 -- 172.21.15.28:0/565983580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac3c1a0f10 con 0x7fac3c105e80 2024-04-21T08:49:18.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.540+0000 7fac436e4700 1 -- 172.21.15.28:0/565983580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac3c050f80 con 0x7fac3c105e80 2024-04-21T08:49:18.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.541+0000 7fac327fc700 1 -- 172.21.15.28:0/565983580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fac2c00f8e0 con 0x7fac3c105e80 2024-04-21T08:49:18.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.541+0000 7fac327fc700 1 --2- 172.21.15.28:0/565983580 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fac280a3d80 0x7fac280a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:18.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.541+0000 7fac41480700 1 --2- 172.21.15.28:0/565983580 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fac280a3d80 0x7fac280a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:18.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.541+0000 7fac327fc700 1 -- 172.21.15.28:0/565983580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fac2c0d0580 con 0x7fac3c105e80 2024-04-21T08:49:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.542+0000 7fac41480700 1 --2- 172.21.15.28:0/565983580 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fac280a3d80 0x7fac280a6240 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fac3800b5c0 tx=0x7fac38005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.546+0000 7fac327fc700 1 -- 172.21.15.28:0/565983580 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac2c014030 con 0x7fac3c105e80 2024-04-21T08:49:18.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.840+0000 7fac436e4700 1 -- 172.21.15.28:0/565983580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fac3c02e050 con 0x7fac3c105e80 2024-04-21T08:49:18.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.842+0000 7fac327fc700 1 -- 172.21.15.28:0/565983580 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fac2c09cbc0 con 0x7fac3c105e80 2024-04-21T08:49:18.843 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-04-21T08:49:18.843 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQAO0yRm93w+MhAAA9lEVtpgVmm9Ifq40g4taQ== 2024-04-21T08:49:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.845+0000 7fac436e4700 1 -- 172.21.15.28:0/565983580 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fac280a3d80 msgr2=0x7fac280a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.845+0000 7fac436e4700 1 --2- 172.21.15.28:0/565983580 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fac280a3d80 0x7fac280a6240 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fac3800b5c0 tx=0x7fac38005fb0 comp rx=0 tx=0).stop 2024-04-21T08:49:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.845+0000 7fac436e4700 1 -- 172.21.15.28:0/565983580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac3c105e80 msgr2=0x7fac3c199030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.845+0000 7fac436e4700 1 --2- 172.21.15.28:0/565983580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac3c105e80 0x7fac3c199030 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00c950 tx=0x7fac2c00cc60 comp rx=0 tx=0).stop 2024-04-21T08:49:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.845+0000 7fac436e4700 1 -- 172.21.15.28:0/565983580 shutdown_connections 2024-04-21T08:49:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.845+0000 7fac436e4700 1 --2- 172.21.15.28:0/565983580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac3c1087b0 0x7fac3c1a0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.845+0000 7fac436e4700 1 --2- 172.21.15.28:0/565983580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac3c103550 0x7fac3c198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.845+0000 7fac436e4700 1 --2- 172.21.15.28:0/565983580 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fac280a3d80 0x7fac280a6240 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.845+0000 7fac436e4700 1 --2- 172.21.15.28:0/565983580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac3c105e80 0x7fac3c199030 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.845+0000 7fac436e4700 1 -- 172.21.15.28:0/565983580 >> 172.21.15.28:0/565983580 conn(0x7fac3c0fcf60 msgr2=0x7fac3c103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.845+0000 7fac436e4700 1 -- 172.21.15.28:0/565983580 shutdown_connections 2024-04-21T08:49:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:18.845+0000 7fac436e4700 1 -- 172.21.15.28:0/565983580 wait complete. 2024-04-21T08:49:19.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:18 smithi028 ceph-mon[28003]: pgmap v310: 129 pgs: 129 active+clean; 620 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2024-04-21T08:49:19.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:18 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/565983580' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T08:49:19.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:18 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/565983580' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T08:49:19.119 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:18 smithi171 ceph-mon[31358]: pgmap v310: 129 pgs: 129 active+clean; 620 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2024-04-21T08:49:19.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:18 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/565983580' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T08:49:19.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:18 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/565983580' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T08:49:19.156 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:18 smithi152 ceph-mon[31401]: pgmap v310: 129 pgs: 129 active+clean; 620 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 5.8 KiB/s wr, 11 op/s 2024-04-21T08:49:19.156 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:18 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/565983580' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T08:49:19.156 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:18 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/565983580' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T08:49:19.646 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:49:19.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-21T08:49:19.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-21T08:49:19.683 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-21T08:49:19.683 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-21T08:49:19.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-21T08:49:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:21.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:20 smithi171 ceph-mon[31358]: pgmap v311: 129 pgs: 129 active+clean; 620 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-04-21T08:49:21.130 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:20 smithi028 ceph-mon[28003]: pgmap v311: 129 pgs: 129 active+clean; 620 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-04-21T08:49:21.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:20 smithi152 ceph-mon[31401]: pgmap v311: 129 pgs: 129 active+clean; 620 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-04-21T08:49:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.574+0000 7fc2cd7e0700 1 -- 172.21.15.28:0/3334340787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c8106930 msgr2=0x7fc2c8106d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.574+0000 7fc2cd7e0700 1 --2- 172.21.15.28:0/3334340787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c8106930 0x7fc2c8106d10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009990 tx=0x7fc2b8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:49:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.574+0000 7fc2cd7e0700 1 -- 172.21.15.28:0/3334340787 shutdown_connections 2024-04-21T08:49:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.574+0000 7fc2cd7e0700 1 --2- 172.21.15.28:0/3334340787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c8106930 0x7fc2c8106d10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.574+0000 7fc2cd7e0700 1 --2- 172.21.15.28:0/3334340787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2c8102650 0x7fc2c810e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.574+0000 7fc2cd7e0700 1 --2- 172.21.15.28:0/3334340787 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2c8101c90 0x7fc2c8102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.574+0000 7fc2cd7e0700 1 -- 172.21.15.28:0/3334340787 >> 172.21.15.28:0/3334340787 conn(0x7fc2c80777b0 msgr2=0x7fc2c8077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.574+0000 7fc2cd7e0700 1 -- 172.21.15.28:0/3334340787 shutdown_connections 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.574+0000 7fc2cd7e0700 1 -- 172.21.15.28:0/3334340787 wait complete. 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.574+0000 7fc2cd7e0700 1 Processor -- start 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2cd7e0700 1 -- start start 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2cd7e0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c8101c90 0x7fc2c8199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2cd7e0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2c8102650 0x7fc2c8199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2cd7e0700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2c8106930 0x7fc2c819d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2cd7e0700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c8111b80 con 0x7fc2c8101c90 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2cd7e0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c8111a00 con 0x7fc2c8102650 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2cd7e0700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c8111d00 con 0x7fc2c8106930 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2c6ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c8101c90 0x7fc2c8199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2c6ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c8101c90 0x7fc2c8199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41914/0 (socket says 172.21.15.28:41914) 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2c6ffd700 1 -- 172.21.15.28:0/3488228730 learned_addr learned my addr 172.21.15.28:0/3488228730 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2c77fe700 1 --2- 172.21.15.28:0/3488228730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2c8106930 0x7fc2c819d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2c67fc700 1 --2- 172.21.15.28:0/3488228730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2c8102650 0x7fc2c8199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2c6ffd700 1 -- 172.21.15.28:0/3488228730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2c8106930 msgr2=0x7fc2c819d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2c6ffd700 1 --2- 172.21.15.28:0/3488228730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2c8106930 0x7fc2c819d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2c6ffd700 1 -- 172.21.15.28:0/3488228730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2c8102650 msgr2=0x7fc2c8199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2c6ffd700 1 --2- 172.21.15.28:0/3488228730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2c8102650 0x7fc2c8199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2c6ffd700 1 -- 172.21.15.28:0/3488228730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2c819e0f0 con 0x7fc2c8101c90 2024-04-21T08:49:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.575+0000 7fc2c77fe700 1 --2- 172.21.15.28:0/3488228730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2c8106930 0x7fc2c819d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.576+0000 7fc2c6ffd700 1 --2- 172.21.15.28:0/3488228730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c8101c90 0x7fc2c8199130 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8000c00 tx=0x7fc2b80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.576+0000 7fc2affff700 1 -- 172.21.15.28:0/3488228730 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8027070 con 0x7fc2c8101c90 2024-04-21T08:49:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.576+0000 7fc2affff700 1 -- 172.21.15.28:0/3488228730 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2b80041d0 con 0x7fc2c8101c90 2024-04-21T08:49:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.576+0000 7fc2cd7e0700 1 -- 172.21.15.28:0/3488228730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2c819e380 con 0x7fc2c8101c90 2024-04-21T08:49:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.576+0000 7fc2affff700 1 -- 172.21.15.28:0/3488228730 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b800f460 con 0x7fc2c8101c90 2024-04-21T08:49:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.576+0000 7fc2cd7e0700 1 -- 172.21.15.28:0/3488228730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2c81a5c60 con 0x7fc2c8101c90 2024-04-21T08:49:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.577+0000 7fc2cd7e0700 1 -- 172.21.15.28:0/3488228730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2c81030b0 con 0x7fc2c8101c90 2024-04-21T08:49:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.578+0000 7fc2affff700 1 -- 172.21.15.28:0/3488228730 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc2b802a070 con 0x7fc2c8101c90 2024-04-21T08:49:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.579+0000 7fc2affff700 1 --2- 172.21.15.28:0/3488228730 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc2b00a3cd0 0x7fc2b00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.579+0000 7fc2c67fc700 1 --2- 172.21.15.28:0/3488228730 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc2b00a3cd0 0x7fc2b00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.579+0000 7fc2affff700 1 -- 172.21.15.28:0/3488228730 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc2b80d1c90 con 0x7fc2c8101c90 2024-04-21T08:49:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.579+0000 7fc2c67fc700 1 --2- 172.21.15.28:0/3488228730 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc2b00a3cd0 0x7fc2b00a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc006fd0 tx=0x7fc2bc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.582+0000 7fc2affff700 1 -- 172.21.15.28:0/3488228730 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc2b8031070 con 0x7fc2c8101c90 2024-04-21T08:49:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.853+0000 7fc2cd7e0700 1 -- 172.21.15.28:0/3488228730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fc2c802df80 con 0x7fc2c8101c90 2024-04-21T08:49:21.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.858+0000 7fc2affff700 1 -- 172.21.15.28:0/3488228730 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264993 (secure 0 0 0) 0x7fc2b8013070 con 0x7fc2c8101c90 2024-04-21T08:49:21.861 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:49:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.861+0000 7fc2cd7e0700 1 -- 172.21.15.28:0/3488228730 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc2b00a3cd0 msgr2=0x7fc2b00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.861+0000 7fc2cd7e0700 1 --2- 172.21.15.28:0/3488228730 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc2b00a3cd0 0x7fc2b00a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc006fd0 tx=0x7fc2bc008040 comp rx=0 tx=0).stop 2024-04-21T08:49:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.861+0000 7fc2cd7e0700 1 -- 172.21.15.28:0/3488228730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c8101c90 msgr2=0x7fc2c8199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.861+0000 7fc2cd7e0700 1 --2- 172.21.15.28:0/3488228730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c8101c90 0x7fc2c8199130 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8000c00 tx=0x7fc2b80047b0 comp rx=0 tx=0).stop 2024-04-21T08:49:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.862+0000 7fc2cd7e0700 1 -- 172.21.15.28:0/3488228730 shutdown_connections 2024-04-21T08:49:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.862+0000 7fc2cd7e0700 1 --2- 172.21.15.28:0/3488228730 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc2b00a3cd0 0x7fc2b00a6190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.862+0000 7fc2cd7e0700 1 --2- 172.21.15.28:0/3488228730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2c8101c90 0x7fc2c8199130 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.862+0000 7fc2cd7e0700 1 --2- 172.21.15.28:0/3488228730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2c8106930 0x7fc2c819d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.862+0000 7fc2cd7e0700 1 --2- 172.21.15.28:0/3488228730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc2c8102650 0x7fc2c8199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.862+0000 7fc2cd7e0700 1 -- 172.21.15.28:0/3488228730 >> 172.21.15.28:0/3488228730 conn(0x7fc2c80777b0 msgr2=0x7fc2c806b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.862+0000 7fc2cd7e0700 1 -- 172.21.15.28:0/3488228730 shutdown_connections 2024-04-21T08:49:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:21.862+0000 7fc2cd7e0700 1 -- 172.21.15.28:0/3488228730 wait complete. 2024-04-21T08:49:22.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:21 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3488228730' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T08:49:22.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:21 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3488228730' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T08:49:22.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:21 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3488228730' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T08:49:22.620 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":226897930},{"type":"v1","addr":"172.21.15.28:6801","nonce":226897930}]},"active_addr":"172.21.15.28:6801/226897930","active_change":"2024-04-21T08:41:35.374237+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.28:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1076282159}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":2132574390}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":926389767}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":892059008}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1811324071}]}]} 2024-04-21T08:49:22.624 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-21T08:49:22.624 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-21T08:49:22.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T08:49:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:23.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:22 smithi171 ceph-mon[31358]: pgmap v312: 129 pgs: 129 active+clean; 625 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 6.3 KiB/s wr, 14 op/s 2024-04-21T08:49:23.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:22 smithi028 ceph-mon[28003]: pgmap v312: 129 pgs: 129 active+clean; 625 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 6.3 KiB/s wr, 14 op/s 2024-04-21T08:49:23.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:22 smithi152 ceph-mon[31401]: pgmap v312: 129 pgs: 129 active+clean; 625 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 6.3 KiB/s wr, 14 op/s 2024-04-21T08:49:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.626+0000 7f18d9ebb700 1 -- 172.21.15.28:0/1107154600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d4108a50 msgr2=0x7f18d4108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.626+0000 7f18d9ebb700 1 --2- 172.21.15.28:0/1107154600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d4108a50 0x7f18d4108e30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f18c4009990 tx=0x7f18c4009ca0 comp rx=0 tx=0).stop 2024-04-21T08:49:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.626+0000 7f18d9ebb700 1 -- 172.21.15.28:0/1107154600 shutdown_connections 2024-04-21T08:49:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.626+0000 7f18d9ebb700 1 --2- 172.21.15.28:0/1107154600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d4108a50 0x7f18d4108e30 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.626+0000 7f18d9ebb700 1 --2- 172.21.15.28:0/1107154600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18d406bab0 0x7f18d410f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.626+0000 7f18d9ebb700 1 --2- 172.21.15.28:0/1107154600 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18d406b110 0x7f18d406b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.626+0000 7f18d9ebb700 1 -- 172.21.15.28:0/1107154600 >> 172.21.15.28:0/1107154600 conn(0x7f18d40788f0 msgr2=0x7f18d4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.626+0000 7f18d9ebb700 1 -- 172.21.15.28:0/1107154600 shutdown_connections 2024-04-21T08:49:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.626+0000 7f18d9ebb700 1 -- 172.21.15.28:0/1107154600 wait complete. 2024-04-21T08:49:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.627+0000 7f18d9ebb700 1 Processor -- start 2024-04-21T08:49:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.627+0000 7f18d9ebb700 1 -- start start 2024-04-21T08:49:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.627+0000 7f18d9ebb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d406b110 0x7f18d4101df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d9ebb700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18d406bab0 0x7f18d4102330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d9ebb700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18d4108a50 0x7f18d4102870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d9ebb700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18d4112cd0 con 0x7f18d406b110 2024-04-21T08:49:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d9ebb700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f18d4112b50 con 0x7f18d4108a50 2024-04-21T08:49:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d9ebb700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f18d4112e50 con 0x7f18d406bab0 2024-04-21T08:49:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d406b110 0x7f18d4101df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d406b110 0x7f18d4101df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60524/0 (socket says 172.21.15.28:60524) 2024-04-21T08:49:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d37fe700 1 -- 172.21.15.28:0/1592364483 learned_addr learned my addr 172.21.15.28:0/1592364483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d3fff700 1 --2- 172.21.15.28:0/1592364483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18d4108a50 0x7f18d4102870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18caffd700 1 --2- 172.21.15.28:0/1592364483 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18d406bab0 0x7f18d4102330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d37fe700 1 -- 172.21.15.28:0/1592364483 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18d406bab0 msgr2=0x7f18d4102330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d37fe700 1 --2- 172.21.15.28:0/1592364483 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18d406bab0 0x7f18d4102330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d37fe700 1 -- 172.21.15.28:0/1592364483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18d4108a50 msgr2=0x7f18d4102870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d37fe700 1 --2- 172.21.15.28:0/1592364483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18d4108a50 0x7f18d4102870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d37fe700 1 -- 172.21.15.28:0/1592364483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d41a77d0 con 0x7f18d406b110 2024-04-21T08:49:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d37fe700 1 --2- 172.21.15.28:0/1592364483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d406b110 0x7f18d4101df0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f18c40057f0 tx=0x7f18c4003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d17fa700 1 -- 172.21.15.28:0/1592364483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4027070 con 0x7f18d406b110 2024-04-21T08:49:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d9ebb700 1 -- 172.21.15.28:0/1592364483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18d41a7a60 con 0x7f18d406b110 2024-04-21T08:49:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d17fa700 1 -- 172.21.15.28:0/1592364483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18c40040a0 con 0x7f18d406b110 2024-04-21T08:49:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.628+0000 7f18d17fa700 1 -- 172.21.15.28:0/1592364483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4022dc0 con 0x7f18d406b110 2024-04-21T08:49:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.629+0000 7f18d9ebb700 1 -- 172.21.15.28:0/1592364483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18d41a7f40 con 0x7f18d406b110 2024-04-21T08:49:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.629+0000 7f18d9ebb700 1 -- 172.21.15.28:0/1592364483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18d4104190 con 0x7f18d406b110 2024-04-21T08:49:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.631+0000 7f18d17fa700 1 -- 172.21.15.28:0/1592364483 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f18c402a070 con 0x7f18d406b110 2024-04-21T08:49:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.632+0000 7f18d17fa700 1 --2- 172.21.15.28:0/1592364483 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f18c00a3cd0 0x7f18c00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.632+0000 7f18caffd700 1 --2- 172.21.15.28:0/1592364483 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f18c00a3cd0 0x7f18c00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.632+0000 7f18d17fa700 1 -- 172.21.15.28:0/1592364483 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f18c40d1cb0 con 0x7f18d406b110 2024-04-21T08:49:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.632+0000 7f18caffd700 1 --2- 172.21.15.28:0/1592364483 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f18c00a3cd0 0x7f18c00a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f18bc006fd0 tx=0x7f18bc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.636+0000 7f18d17fa700 1 -- 172.21.15.28:0/1592364483 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f18c4031020 con 0x7f18d406b110 2024-04-21T08:49:24.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.844+0000 7f18d9ebb700 1 -- 172.21.15.28:0/1592364483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f18d402df80 con 0x7f18d406b110 2024-04-21T08:49:24.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.845+0000 7f18d17fa700 1 -- 172.21.15.28:0/1592364483 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24101 (secure 0 0 0) 0x7f18c4088090 con 0x7f18d406b110 2024-04-21T08:49:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:49:24.846 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:48:51.121128+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:49:24.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.848+0000 7f18d9ebb700 1 -- 172.21.15.28:0/1592364483 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f18c00a3cd0 msgr2=0x7f18c00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:24.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.848+0000 7f18d9ebb700 1 --2- 172.21.15.28:0/1592364483 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f18c00a3cd0 0x7f18c00a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f18bc006fd0 tx=0x7f18bc008040 comp rx=0 tx=0).stop 2024-04-21T08:49:24.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.848+0000 7f18d9ebb700 1 -- 172.21.15.28:0/1592364483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d406b110 msgr2=0x7f18d4101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:24.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.848+0000 7f18d9ebb700 1 --2- 172.21.15.28:0/1592364483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d406b110 0x7f18d4101df0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f18c40057f0 tx=0x7f18c4003680 comp rx=0 tx=0).stop 2024-04-21T08:49:24.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.848+0000 7f18d9ebb700 1 -- 172.21.15.28:0/1592364483 shutdown_connections 2024-04-21T08:49:24.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.848+0000 7f18d9ebb700 1 --2- 172.21.15.28:0/1592364483 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f18c00a3cd0 0x7f18c00a6190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:24.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.848+0000 7f18d9ebb700 1 --2- 172.21.15.28:0/1592364483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d406b110 0x7f18d4101df0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:24.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.848+0000 7f18d9ebb700 1 --2- 172.21.15.28:0/1592364483 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f18d4108a50 0x7f18d4102870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:24.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.848+0000 7f18d9ebb700 1 --2- 172.21.15.28:0/1592364483 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18d406bab0 0x7f18d4102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:24.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.848+0000 7f18d9ebb700 1 -- 172.21.15.28:0/1592364483 >> 172.21.15.28:0/1592364483 conn(0x7f18d40788f0 msgr2=0x7f18d40fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:24.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.848+0000 7f18d9ebb700 1 -- 172.21.15.28:0/1592364483 shutdown_connections 2024-04-21T08:49:24.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:24.848+0000 7f18d9ebb700 1 -- 172.21.15.28:0/1592364483 wait complete. 2024-04-21T08:49:25.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:24 smithi028 ceph-mon[28003]: pgmap v313: 129 pgs: 129 active+clean; 625 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 6.2 KiB/s wr, 14 op/s 2024-04-21T08:49:25.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:24 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1592364483' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:25.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:24 smithi171 ceph-mon[31358]: pgmap v313: 129 pgs: 129 active+clean; 625 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 6.2 KiB/s wr, 14 op/s 2024-04-21T08:49:25.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:24 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1592364483' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:25.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:24 smithi152 ceph-mon[31401]: pgmap v313: 129 pgs: 129 active+clean; 625 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 6.2 KiB/s wr, 14 op/s 2024-04-21T08:49:25.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:24 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1592364483' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:25.632 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-21T08:49:25.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T08:49:25.889 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:27.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:26 smithi028 ceph-mon[28003]: pgmap v314: 129 pgs: 129 active+clean; 625 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-04-21T08:49:27.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:26 smithi152 ceph-mon[31401]: pgmap v314: 129 pgs: 129 active+clean; 625 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-04-21T08:49:27.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:26 smithi171 ceph-mon[31358]: pgmap v314: 129 pgs: 129 active+clean; 625 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-04-21T08:49:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.620+0000 7fe3e607e700 1 -- 172.21.15.28:0/595691815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e01079c0 msgr2=0x7fe3e0107da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.620+0000 7fe3e607e700 1 --2- 172.21.15.28:0/595691815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e01079c0 0x7fe3e0107da0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009990 tx=0x7fe3d0009ca0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.621+0000 7fe3e607e700 1 -- 172.21.15.28:0/595691815 shutdown_connections 2024-04-21T08:49:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.621+0000 7fe3e607e700 1 --2- 172.21.15.28:0/595691815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e01079c0 0x7fe3e0107da0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.621+0000 7fe3e607e700 1 --2- 172.21.15.28:0/595691815 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3e0102600 0x7fe3e010e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.621+0000 7fe3e607e700 1 --2- 172.21.15.28:0/595691815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3e0101c40 0x7fe3e01020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.621+0000 7fe3e607e700 1 -- 172.21.15.28:0/595691815 >> 172.21.15.28:0/595691815 conn(0x7fe3e0078740 msgr2=0x7fe3e0078b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.621+0000 7fe3e607e700 1 -- 172.21.15.28:0/595691815 shutdown_connections 2024-04-21T08:49:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.621+0000 7fe3e607e700 1 -- 172.21.15.28:0/595691815 wait complete. 2024-04-21T08:49:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.621+0000 7fe3e607e700 1 Processor -- start 2024-04-21T08:49:27.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.621+0000 7fe3e607e700 1 -- start start 2024-04-21T08:49:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3e607e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3e0101c40 0x7fe3e0103590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3e607e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e0102600 0x7fe3e0103ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3e607e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3e01079c0 0x7fe3e0104010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3e607e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e0111c00 con 0x7fe3e0102600 2024-04-21T08:49:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3e607e700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e0111a80 con 0x7fe3e0101c40 2024-04-21T08:49:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3e607e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e0111d80 con 0x7fe3e01079c0 2024-04-21T08:49:27.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3deffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e0102600 0x7fe3e0103ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3deffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e0102600 0x7fe3e0103ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60548/0 (socket says 172.21.15.28:60548) 2024-04-21T08:49:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3deffd700 1 -- 172.21.15.28:0/2627542069 learned_addr learned my addr 172.21.15.28:0/2627542069 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3df7fe700 1 --2- 172.21.15.28:0/2627542069 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3e0101c40 0x7fe3e0103590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3dffff700 1 --2- 172.21.15.28:0/2627542069 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3e01079c0 0x7fe3e0104010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3deffd700 1 -- 172.21.15.28:0/2627542069 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3e01079c0 msgr2=0x7fe3e0104010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3deffd700 1 --2- 172.21.15.28:0/2627542069 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3e01079c0 0x7fe3e0104010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3deffd700 1 -- 172.21.15.28:0/2627542069 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3e0101c40 msgr2=0x7fe3e0103590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3deffd700 1 --2- 172.21.15.28:0/2627542069 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3e0101c40 0x7fe3e0103590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3deffd700 1 -- 172.21.15.28:0/2627542069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e01a7670 con 0x7fe3e0102600 2024-04-21T08:49:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.622+0000 7fe3deffd700 1 --2- 172.21.15.28:0/2627542069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e0102600 0x7fe3e0103ad0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400c9b0 tx=0x7fe3d400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.623+0000 7fe3dcff9700 1 -- 172.21.15.28:0/2627542069 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4007b00 con 0x7fe3e0102600 2024-04-21T08:49:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.623+0000 7fe3dcff9700 1 -- 172.21.15.28:0/2627542069 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3d400ce50 con 0x7fe3e0102600 2024-04-21T08:49:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.623+0000 7fe3e607e700 1 -- 172.21.15.28:0/2627542069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e01a7960 con 0x7fe3e0102600 2024-04-21T08:49:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.623+0000 7fe3dcff9700 1 -- 172.21.15.28:0/2627542069 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4018660 con 0x7fe3e0102600 2024-04-21T08:49:27.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.623+0000 7fe3e607e700 1 -- 172.21.15.28:0/2627542069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e01a7ea0 con 0x7fe3e0102600 2024-04-21T08:49:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.624+0000 7fe3e607e700 1 -- 172.21.15.28:0/2627542069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3e0069730 con 0x7fe3e0102600 2024-04-21T08:49:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.625+0000 7fe3dcff9700 1 -- 172.21.15.28:0/2627542069 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe3d400f450 con 0x7fe3e0102600 2024-04-21T08:49:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.625+0000 7fe3dcff9700 1 --2- 172.21.15.28:0/2627542069 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe3c80a3cd0 0x7fe3c80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:27.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.626+0000 7fe3df7fe700 1 --2- 172.21.15.28:0/2627542069 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe3c80a3cd0 0x7fe3c80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:27.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.626+0000 7fe3dcff9700 1 -- 172.21.15.28:0/2627542069 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe3d40d0700 con 0x7fe3e0102600 2024-04-21T08:49:27.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.626+0000 7fe3df7fe700 1 --2- 172.21.15.28:0/2627542069 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe3c80a3cd0 0x7fe3c80a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe3d000b5c0 tx=0x7fe3d0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:27.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.630+0000 7fe3dcff9700 1 -- 172.21.15.28:0/2627542069 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe3d4014030 con 0x7fe3e0102600 2024-04-21T08:49:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.839+0000 7fe3e607e700 1 -- 172.21.15.28:0/2627542069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe3e002e140 con 0x7fe3e0102600 2024-04-21T08:49:27.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.841+0000 7fe3dcff9700 1 -- 172.21.15.28:0/2627542069 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24101 (secure 0 0 0) 0x7fe3d4098030 con 0x7fe3e0102600 2024-04-21T08:49:27.841 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:49:27.841 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:48:51.121128+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:49:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.844+0000 7fe3e607e700 1 -- 172.21.15.28:0/2627542069 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe3c80a3cd0 msgr2=0x7fe3c80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.844+0000 7fe3e607e700 1 --2- 172.21.15.28:0/2627542069 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe3c80a3cd0 0x7fe3c80a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe3d000b5c0 tx=0x7fe3d0005960 comp rx=0 tx=0).stop 2024-04-21T08:49:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.844+0000 7fe3e607e700 1 -- 172.21.15.28:0/2627542069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e0102600 msgr2=0x7fe3e0103ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.844+0000 7fe3e607e700 1 --2- 172.21.15.28:0/2627542069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e0102600 0x7fe3e0103ad0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe3d400c9b0 tx=0x7fe3d400ccc0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.844+0000 7fe3e607e700 1 -- 172.21.15.28:0/2627542069 shutdown_connections 2024-04-21T08:49:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.844+0000 7fe3e607e700 1 --2- 172.21.15.28:0/2627542069 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3e01079c0 0x7fe3e0104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.844+0000 7fe3e607e700 1 --2- 172.21.15.28:0/2627542069 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3e0101c40 0x7fe3e0103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.844+0000 7fe3e607e700 1 --2- 172.21.15.28:0/2627542069 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe3c80a3cd0 0x7fe3c80a6190 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.844+0000 7fe3e607e700 1 --2- 172.21.15.28:0/2627542069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e0102600 0x7fe3e0103ad0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.844+0000 7fe3e607e700 1 -- 172.21.15.28:0/2627542069 >> 172.21.15.28:0/2627542069 conn(0x7fe3e0078740 msgr2=0x7fe3e006b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.844+0000 7fe3e607e700 1 -- 172.21.15.28:0/2627542069 shutdown_connections 2024-04-21T08:49:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:27.844+0000 7fe3e607e700 1 -- 172.21.15.28:0/2627542069 wait complete. 2024-04-21T08:49:28.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:27 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2627542069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:28.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:27 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2627542069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:28.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:27 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2627542069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:49:28.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-21T08:49:28.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-21T08:49:28.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-21T08:49:28.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-21T08:49:28.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-21T08:49:28.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-21T08:49:28.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-21T08:49:28.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-21T08:49:28.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-21T08:49:28.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-21T08:49:28.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-21T08:49:28.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-21T08:49:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:29.328 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:29.332 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:29.333 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:29.333 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:29.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:28 smithi152 ceph-mon[31401]: pgmap v315: 129 pgs: 129 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.5 KiB/s wr, 10 op/s 2024-04-21T08:49:29.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:28 smithi028 ceph-mon[28003]: pgmap v315: 129 pgs: 129 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.5 KiB/s wr, 10 op/s 2024-04-21T08:49:29.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:28 smithi171 ceph-mon[31358]: pgmap v315: 129 pgs: 129 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.5 KiB/s wr, 10 op/s 2024-04-21T08:49:29.426 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:29.449 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:29.449 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:29.449 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:30.956 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:30 smithi028 ceph-mon[28003]: pgmap v316: 129 pgs: 129 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-21T08:49:31.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:30 smithi152 ceph-mon[31401]: pgmap v316: 129 pgs: 129 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-21T08:49:31.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:30 smithi171 ceph-mon[31358]: pgmap v316: 129 pgs: 129 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-21T08:49:33.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:32 smithi028 ceph-mon[28003]: pgmap v317: 129 pgs: 129 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-21T08:49:33.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:32 smithi152 ceph-mon[31401]: pgmap v317: 129 pgs: 129 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-21T08:49:33.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:32 smithi171 ceph-mon[31358]: pgmap v317: 129 pgs: 129 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-21T08:49:33.886 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:34.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:34 smithi028 ceph-mon[28003]: pgmap v318: 129 pgs: 129 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-04-21T08:49:35.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:34 smithi152 ceph-mon[31401]: pgmap v318: 129 pgs: 129 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-04-21T08:49:35.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:34 smithi171 ceph-mon[31358]: pgmap v318: 129 pgs: 129 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-04-21T08:49:36.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]: dispatch 2024-04-21T08:49:36.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-04-21T08:49:36.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-04-21T08:49:36.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-21T08:49:36.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2024-04-21T08:49:36.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-04-21T08:49:36.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]: dispatch 2024-04-21T08:49:36.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-04-21T08:49:36.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:35 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-04-21T08:49:36.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:35 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]: dispatch 2024-04-21T08:49:36.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:35 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-04-21T08:49:36.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:35 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-04-21T08:49:36.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:35 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-21T08:49:36.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:35 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2024-04-21T08:49:36.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:35 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-04-21T08:49:36.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:35 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]: dispatch 2024-04-21T08:49:36.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:35 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-04-21T08:49:36.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:35 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-04-21T08:49:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]: dispatch 2024-04-21T08:49:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-04-21T08:49:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-04-21T08:49:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-21T08:49:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]: dispatch 2024-04-21T08:49:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-04-21T08:49:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]: dispatch 2024-04-21T08:49:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-04-21T08:49:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:35 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-04-21T08:49:37.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:36 smithi028 ceph-mon[28003]: pgmap v319: 129 pgs: 129 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-04-21T08:49:37.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]': finished 2024-04-21T08:49:37.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-04-21T08:49:37.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-04-21T08:49:37.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-21T08:49:37.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2024-04-21T08:49:37.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-04-21T08:49:37.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]': finished 2024-04-21T08:49:37.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-04-21T08:49:37.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-04-21T08:49:37.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:36 smithi028 ceph-mon[28003]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T08:49:37.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:36 smithi152 ceph-mon[31401]: pgmap v319: 129 pgs: 129 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-04-21T08:49:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]': finished 2024-04-21T08:49:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-04-21T08:49:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-04-21T08:49:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-21T08:49:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2024-04-21T08:49:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-04-21T08:49:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]': finished 2024-04-21T08:49:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-04-21T08:49:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-04-21T08:49:37.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:36 smithi152 ceph-mon[31401]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T08:49:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:36 smithi171 ceph-mon[31358]: pgmap v319: 129 pgs: 129 active+clean; 637 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-04-21T08:49:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 1]}]': finished 2024-04-21T08:49:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-04-21T08:49:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-04-21T08:49:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-21T08:49:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 7]}]': finished 2024-04-21T08:49:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-04-21T08:49:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]': finished 2024-04-21T08:49:37.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-04-21T08:49:37.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-04-21T08:49:37.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:36 smithi171 ceph-mon[31358]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T08:49:38.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:37 smithi028 ceph-mon[28003]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T08:49:38.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:37 smithi152 ceph-mon[31401]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T08:49:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:37 smithi171 ceph-mon[31358]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T08:49:39.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:38 smithi028 ceph-mon[28003]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-21T08:49:39.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:38 smithi152 ceph-mon[31401]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-21T08:49:39.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:38 smithi171 ceph-mon[31358]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-21T08:49:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.798+0000 7f675a6cb700 1 -- 172.21.15.28:0/2512815073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6754102300 msgr2=0x7f675410e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.798+0000 7f675a6cb700 1 --2- 172.21.15.28:0/2512815073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6754102300 0x7f675410e7f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f674c00b0a0 tx=0x7f674c00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:49:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.798+0000 7f675a6cb700 1 -- 172.21.15.28:0/2512815073 shutdown_connections 2024-04-21T08:49:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.798+0000 7f675a6cb700 1 --2- 172.21.15.28:0/2512815073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6754102300 0x7f675410e7f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.798+0000 7f675a6cb700 1 --2- 172.21.15.28:0/2512815073 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6754107990 0x7f6754107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.798+0000 7f675a6cb700 1 --2- 172.21.15.28:0/2512815073 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6754101940 0x7f6754101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.798+0000 7f675a6cb700 1 -- 172.21.15.28:0/2512815073 >> 172.21.15.28:0/2512815073 conn(0x7f67540788f0 msgr2=0x7f6754078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.798+0000 7f675a6cb700 1 -- 172.21.15.28:0/2512815073 shutdown_connections 2024-04-21T08:49:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.798+0000 7f675a6cb700 1 -- 172.21.15.28:0/2512815073 wait complete. 2024-04-21T08:49:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.799+0000 7f675a6cb700 1 Processor -- start 2024-04-21T08:49:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.799+0000 7f675a6cb700 1 -- start start 2024-04-21T08:49:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.799+0000 7f675a6cb700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6754101940 0x7f6754199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.799+0000 7f675a6cb700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6754102300 0x7f67541995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.799+0000 7f675a6cb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6754107990 0x7f675419d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.799+0000 7f675a6cb700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6754111ae0 con 0x7f6754107990 2024-04-21T08:49:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.799+0000 7f675a6cb700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6754111960 con 0x7f6754102300 2024-04-21T08:49:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.799+0000 7f675a6cb700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6754111c60 con 0x7f6754101940 2024-04-21T08:49:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.799+0000 7f6758c68700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6754107990 0x7f675419d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.799+0000 7f6758c68700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6754107990 0x7f675419d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54750/0 (socket says 172.21.15.28:54750) 2024-04-21T08:49:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.799+0000 7f6758c68700 1 -- 172.21.15.28:0/1157584735 learned_addr learned my addr 172.21.15.28:0/1157584735 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.799+0000 7f6753fff700 1 --2- 172.21.15.28:0/1157584735 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6754101940 0x7f6754199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.800+0000 7f67537fe700 1 --2- 172.21.15.28:0/1157584735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6754102300 0x7f67541995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.800+0000 7f6758c68700 1 -- 172.21.15.28:0/1157584735 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6754101940 msgr2=0x7f6754199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.800+0000 7f6758c68700 1 --2- 172.21.15.28:0/1157584735 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6754101940 0x7f6754199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.800+0000 7f6758c68700 1 -- 172.21.15.28:0/1157584735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6754102300 msgr2=0x7f67541995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.800+0000 7f6758c68700 1 --2- 172.21.15.28:0/1157584735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6754102300 0x7f67541995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.800+0000 7f6758c68700 1 -- 172.21.15.28:0/1157584735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675419e070 con 0x7f6754107990 2024-04-21T08:49:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.800+0000 7f6758c68700 1 --2- 172.21.15.28:0/1157584735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6754107990 0x7f675419d930 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f674c0062a0 tx=0x7f674c004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.800+0000 7f67517fa700 1 -- 172.21.15.28:0/1157584735 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c021070 con 0x7f6754107990 2024-04-21T08:49:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.800+0000 7f675a6cb700 1 -- 172.21.15.28:0/1157584735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f675419e300 con 0x7f6754107990 2024-04-21T08:49:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.800+0000 7f67517fa700 1 -- 172.21.15.28:0/1157584735 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f674c007c80 con 0x7f6754107990 2024-04-21T08:49:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.800+0000 7f67517fa700 1 -- 172.21.15.28:0/1157584735 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c01d450 con 0x7f6754107990 2024-04-21T08:49:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.800+0000 7f675a6cb700 1 -- 172.21.15.28:0/1157584735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67541a5be0 con 0x7f6754107990 2024-04-21T08:49:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.801+0000 7f675a6cb700 1 -- 172.21.15.28:0/1157584735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6754063150 con 0x7f6754107990 2024-04-21T08:49:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.803+0000 7f67517fa700 1 -- 172.21.15.28:0/1157584735 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f674c004020 con 0x7f6754107990 2024-04-21T08:49:39.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.803+0000 7f67517fa700 1 --2- 172.21.15.28:0/1157584735 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f673c0a3cd0 0x7f673c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:39.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.804+0000 7f67517fa700 1 -- 172.21.15.28:0/1157584735 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f674c00f020 con 0x7f6754107990 2024-04-21T08:49:39.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.804+0000 7f6753fff700 1 --2- 172.21.15.28:0/1157584735 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f673c0a3cd0 0x7f673c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:39.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.804+0000 7f67517fa700 1 --2- 172.21.15.28:0/1157584735 >> [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376] conn(0x7f673c0aedb0 0x7f673c0b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:39.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.804+0000 7f67517fa700 1 -- 172.21.15.28:0/1157584735 --> [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f674c008f20 con 0x7f673c0aedb0 2024-04-21T08:49:39.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.804+0000 7f67517fa700 1 -- 172.21.15.28:0/1157584735 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f674c0d25a0 con 0x7f6754107990 2024-04-21T08:49:39.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.804+0000 7f6753fff700 1 --2- 172.21.15.28:0/1157584735 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f673c0a3cd0 0x7f673c0a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6744005950 tx=0x7f674400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:39.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.804+0000 7f67537fe700 1 --2- 172.21.15.28:0/1157584735 >> [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376] conn(0x7f673c0aedb0 0x7f673c0b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:39.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.805+0000 7f67537fe700 1 --2- 172.21.15.28:0/1157584735 >> [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376] conn(0x7f673c0aedb0 0x7f673c0b1250 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.806+0000 7f67517fa700 1 -- 172.21.15.28:0/1157584735 <== osd.4 v2:172.21.15.152:6808/3935841376 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f674c008f20 con 0x7f673c0aedb0 2024-04-21T08:49:39.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.822+0000 7f675a6cb700 1 -- 172.21.15.28:0/1157584735 --> [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6754063550 con 0x7f673c0aedb0 2024-04-21T08:49:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.823+0000 7f67517fa700 1 -- 172.21.15.28:0/1157584735 <== osd.4 v2:172.21.15.152:6808/3935841376 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6754063550 con 0x7f673c0aedb0 2024-04-21T08:49:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.823+0000 7f675a6cb700 1 -- 172.21.15.28:0/1157584735 >> [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376] conn(0x7f673c0aedb0 msgr2=0x7f673c0b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.823+0000 7f675a6cb700 1 --2- 172.21.15.28:0/1157584735 >> [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376] conn(0x7f673c0aedb0 0x7f673c0b1250 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.824+0000 7f675a6cb700 1 -- 172.21.15.28:0/1157584735 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f673c0a3cd0 msgr2=0x7f673c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.824+0000 7f675a6cb700 1 --2- 172.21.15.28:0/1157584735 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f673c0a3cd0 0x7f673c0a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6744005950 tx=0x7f674400a300 comp rx=0 tx=0).stop 2024-04-21T08:49:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.824+0000 7f675a6cb700 1 -- 172.21.15.28:0/1157584735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6754107990 msgr2=0x7f675419d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.824+0000 7f675a6cb700 1 --2- 172.21.15.28:0/1157584735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6754107990 0x7f675419d930 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f674c0062a0 tx=0x7f674c004e20 comp rx=0 tx=0).stop 2024-04-21T08:49:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.824+0000 7f6758c68700 1 -- 172.21.15.28:0/1157584735 reap_dead start 2024-04-21T08:49:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.824+0000 7f675a6cb700 1 -- 172.21.15.28:0/1157584735 shutdown_connections 2024-04-21T08:49:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.824+0000 7f675a6cb700 1 -- 172.21.15.28:0/1157584735 >> 172.21.15.28:0/1157584735 conn(0x7f67540788f0 msgr2=0x7f675406aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.824+0000 7f675a6cb700 1 -- 172.21.15.28:0/1157584735 shutdown_connections 2024-04-21T08:49:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:39.824+0000 7f675a6cb700 1 -- 172.21.15.28:0/1157584735 wait complete. 2024-04-21T08:49:40.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.659+0000 7f66eb635700 1 -- 172.21.15.28:0/2342948126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e4103950 msgr2=0x7f66e4103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:40.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.659+0000 7f66eb635700 1 --2- 172.21.15.28:0/2342948126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e4103950 0x7f66e4103d30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009990 tx=0x7f66d4009ca0 comp rx=0 tx=0).stop 2024-04-21T08:49:40.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.659+0000 7f66eb635700 1 -- 172.21.15.28:0/2342948126 shutdown_connections 2024-04-21T08:49:40.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.659+0000 7f66eb635700 1 --2- 172.21.15.28:0/2342948126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e4103950 0x7f66e4103d30 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:40.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.659+0000 7f66eb635700 1 --2- 172.21.15.28:0/2342948126 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f66e4101ad0 0x7f66e41117c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:40.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.659+0000 7f66eb635700 1 --2- 172.21.15.28:0/2342948126 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66e4101030 0x7f66e41014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:40.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.659+0000 7f66eb635700 1 -- 172.21.15.28:0/2342948126 >> 172.21.15.28:0/2342948126 conn(0x7f66e4077fb0 msgr2=0x7f66e40ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:40.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.659+0000 7f66eb635700 1 -- 172.21.15.28:0/2342948126 shutdown_connections 2024-04-21T08:49:40.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.659+0000 7f66eb635700 1 -- 172.21.15.28:0/2342948126 wait complete. 2024-04-21T08:49:40.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.659+0000 7f66eb635700 1 Processor -- start 2024-04-21T08:49:40.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66eb635700 1 -- start start 2024-04-21T08:49:40.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66eb635700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f66e4101030 0x7f66e419b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:40.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66eb635700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e4101ad0 0x7f66e419b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:40.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66eb635700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66e4103950 0x7f66e419fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:40.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66eb635700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66e4114e40 con 0x7f66e4101ad0 2024-04-21T08:49:40.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66eb635700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f66e4114cc0 con 0x7f66e4101030 2024-04-21T08:49:40.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66eb635700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f66e4114fc0 con 0x7f66e4103950 2024-04-21T08:49:40.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66e8bd0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e4101ad0 0x7f66e419b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:40.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66e8bd0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e4101ad0 0x7f66e419b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54776/0 (socket says 172.21.15.28:54776) 2024-04-21T08:49:40.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66e8bd0700 1 -- 172.21.15.28:0/2884590346 learned_addr learned my addr 172.21.15.28:0/2884590346 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:40.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66e93d1700 1 --2- 172.21.15.28:0/2884590346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f66e4101030 0x7f66e419b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:40.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66e9bd2700 1 --2- 172.21.15.28:0/2884590346 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66e4103950 0x7f66e419fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:40.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66e8bd0700 1 -- 172.21.15.28:0/2884590346 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66e4103950 msgr2=0x7f66e419fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:40.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66e8bd0700 1 --2- 172.21.15.28:0/2884590346 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66e4103950 0x7f66e419fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:40.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66e8bd0700 1 -- 172.21.15.28:0/2884590346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f66e4101030 msgr2=0x7f66e419b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:40.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66e8bd0700 1 --2- 172.21.15.28:0/2884590346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f66e4101030 0x7f66e419b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:40.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66e8bd0700 1 -- 172.21.15.28:0/2884590346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66e41a0380 con 0x7f66e4101ad0 2024-04-21T08:49:40.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66e93d1700 1 --2- 172.21.15.28:0/2884590346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f66e4101030 0x7f66e419b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:40.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.660+0000 7f66e8bd0700 1 --2- 172.21.15.28:0/2884590346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e4101ad0 0x7f66e419b8b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f66e000d980 tx=0x7f66e000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:40.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.661+0000 7f66da7fc700 1 -- 172.21.15.28:0/2884590346 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e0017030 con 0x7f66e4101ad0 2024-04-21T08:49:40.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.661+0000 7f66da7fc700 1 -- 172.21.15.28:0/2884590346 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66e0009df0 con 0x7f66e4101ad0 2024-04-21T08:49:40.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.661+0000 7f66eb635700 1 -- 172.21.15.28:0/2884590346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66e41a0670 con 0x7f66e4101ad0 2024-04-21T08:49:40.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.661+0000 7f66da7fc700 1 -- 172.21.15.28:0/2884590346 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e0005350 con 0x7f66e4101ad0 2024-04-21T08:49:40.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.661+0000 7f66eb635700 1 -- 172.21.15.28:0/2884590346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66e41a7ef0 con 0x7f66e4101ad0 2024-04-21T08:49:40.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.661+0000 7f66eb635700 1 -- 172.21.15.28:0/2884590346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f66e40631d0 con 0x7f66e4101ad0 2024-04-21T08:49:40.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.663+0000 7f66da7fc700 1 -- 172.21.15.28:0/2884590346 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f66e001a070 con 0x7f66e4101ad0 2024-04-21T08:49:40.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.663+0000 7f66da7fc700 1 --2- 172.21.15.28:0/2884590346 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f66d00a3cd0 0x7f66d00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:40.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.664+0000 7f66da7fc700 1 -- 172.21.15.28:0/2884590346 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f66e00d0760 con 0x7f66e4101ad0 2024-04-21T08:49:40.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.664+0000 7f66e93d1700 1 --2- 172.21.15.28:0/2884590346 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f66d00a3cd0 0x7f66d00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:40.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.664+0000 7f66da7fc700 1 --2- 172.21.15.28:0/2884590346 >> [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582] conn(0x7f66d00aedb0 0x7f66d00b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:40.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.664+0000 7f66da7fc700 1 -- 172.21.15.28:0/2884590346 --> [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f66e0005730 con 0x7f66d00aedb0 2024-04-21T08:49:40.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.664+0000 7f66da7fc700 1 -- 172.21.15.28:0/2884590346 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f66e00d0b10 con 0x7f66e4101ad0 2024-04-21T08:49:40.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.664+0000 7f66e93d1700 1 --2- 172.21.15.28:0/2884590346 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f66d00a3cd0 0x7f66d00a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f66d400b5c0 tx=0x7f66d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:40.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.664+0000 7f66e9bd2700 1 --2- 172.21.15.28:0/2884590346 >> [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582] conn(0x7f66d00aedb0 0x7f66d00b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:40.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.665+0000 7f66e9bd2700 1 --2- 172.21.15.28:0/2884590346 >> [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582] conn(0x7f66d00aedb0 0x7f66d00b1250 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:40.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.665+0000 7f66da7fc700 1 -- 172.21.15.28:0/2884590346 <== osd.2 v2:172.21.15.171:6800/4097967582 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f66e0005730 con 0x7f66d00aedb0 2024-04-21T08:49:40.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.682+0000 7f66eb635700 1 -- 172.21.15.28:0/2884590346 --> [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f66e4063650 con 0x7f66d00aedb0 2024-04-21T08:49:40.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.682+0000 7f66da7fc700 1 -- 172.21.15.28:0/2884590346 <== osd.2 v2:172.21.15.171:6800/4097967582 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f66e4063650 con 0x7f66d00aedb0 2024-04-21T08:49:40.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.683+0000 7f66eb635700 1 -- 172.21.15.28:0/2884590346 >> [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582] conn(0x7f66d00aedb0 msgr2=0x7f66d00b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:40.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.683+0000 7f66eb635700 1 --2- 172.21.15.28:0/2884590346 >> [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582] conn(0x7f66d00aedb0 0x7f66d00b1250 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:40.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.683+0000 7f66eb635700 1 -- 172.21.15.28:0/2884590346 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f66d00a3cd0 msgr2=0x7f66d00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:40.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.683+0000 7f66eb635700 1 --2- 172.21.15.28:0/2884590346 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f66d00a3cd0 0x7f66d00a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f66d400b5c0 tx=0x7f66d4005fd0 comp rx=0 tx=0).stop 2024-04-21T08:49:40.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.683+0000 7f66eb635700 1 -- 172.21.15.28:0/2884590346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e4101ad0 msgr2=0x7f66e419b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:40.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.683+0000 7f66eb635700 1 --2- 172.21.15.28:0/2884590346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66e4101ad0 0x7f66e419b8b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f66e000d980 tx=0x7f66e000dd40 comp rx=0 tx=0).stop 2024-04-21T08:49:40.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.683+0000 7f66e9bd2700 1 -- 172.21.15.28:0/2884590346 reap_dead start 2024-04-21T08:49:40.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.683+0000 7f66eb635700 1 -- 172.21.15.28:0/2884590346 shutdown_connections 2024-04-21T08:49:40.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.683+0000 7f66eb635700 1 -- 172.21.15.28:0/2884590346 >> 172.21.15.28:0/2884590346 conn(0x7f66e4077fb0 msgr2=0x7f66e40fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:40.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.683+0000 7f66eb635700 1 -- 172.21.15.28:0/2884590346 shutdown_connections 2024-04-21T08:49:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:40.683+0000 7f66eb635700 1 -- 172.21.15.28:0/2884590346 wait complete. 2024-04-21T08:49:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.633+0000 7fccedd05700 1 -- 172.21.15.28:0/1610495536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce8108ab0 msgr2=0x7fcce8108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.633+0000 7fccedd05700 1 --2- 172.21.15.28:0/1610495536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce8108ab0 0x7fcce8108e90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fccd8009990 tx=0x7fccd8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:49:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.634+0000 7fccedd05700 1 -- 172.21.15.28:0/1610495536 shutdown_connections 2024-04-21T08:49:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.634+0000 7fccedd05700 1 --2- 172.21.15.28:0/1610495536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce8108ab0 0x7fcce8108e90 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.634+0000 7fccedd05700 1 --2- 172.21.15.28:0/1610495536 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcce8103480 0x7fcce810f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.634+0000 7fccedd05700 1 --2- 172.21.15.28:0/1610495536 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcce8102ac0 0x7fcce8102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.634+0000 7fccedd05700 1 -- 172.21.15.28:0/1610495536 >> 172.21.15.28:0/1610495536 conn(0x7fcce8079bf0 msgr2=0x7fcce807a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.634+0000 7fccedd05700 1 -- 172.21.15.28:0/1610495536 shutdown_connections 2024-04-21T08:49:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.634+0000 7fccedd05700 1 -- 172.21.15.28:0/1610495536 wait complete. 2024-04-21T08:49:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.634+0000 7fccedd05700 1 Processor -- start 2024-04-21T08:49:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.634+0000 7fccedd05700 1 -- start start 2024-04-21T08:49:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fccedd05700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce8102ac0 0x7fcce8074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fccedd05700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcce8103480 0x7fcce8075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fccedd05700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcce8108ab0 0x7fcce80757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fccedd05700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcce8112ca0 con 0x7fcce8102ac0 2024-04-21T08:49:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fccedd05700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fcce8112b20 con 0x7fcce8103480 2024-04-21T08:49:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fccedd05700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcce8112e20 con 0x7fcce8108ab0 2024-04-21T08:49:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fcce77fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce8102ac0 0x7fcce8074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fcce77fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce8102ac0 0x7fcce8074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54738/0 (socket says 172.21.15.28:54738) 2024-04-21T08:49:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fcce77fe700 1 -- 172.21.15.28:0/1543849854 learned_addr learned my addr 172.21.15.28:0/1543849854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fcce6ffd700 1 --2- 172.21.15.28:0/1543849854 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcce8103480 0x7fcce8075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fcce7fff700 1 --2- 172.21.15.28:0/1543849854 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcce8108ab0 0x7fcce80757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fcce77fe700 1 -- 172.21.15.28:0/1543849854 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcce8108ab0 msgr2=0x7fcce80757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fcce77fe700 1 --2- 172.21.15.28:0/1543849854 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcce8108ab0 0x7fcce80757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fcce77fe700 1 -- 172.21.15.28:0/1543849854 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcce8103480 msgr2=0x7fcce8075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fcce77fe700 1 --2- 172.21.15.28:0/1543849854 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcce8103480 0x7fcce8075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fcce77fe700 1 -- 172.21.15.28:0/1543849854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce81a76c0 con 0x7fcce8102ac0 2024-04-21T08:49:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.635+0000 7fcce77fe700 1 --2- 172.21.15.28:0/1543849854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce8102ac0 0x7fcce8074d40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fccd8000c00 tx=0x7fccd8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.636+0000 7fcce4ff9700 1 -- 172.21.15.28:0/1543849854 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd8027070 con 0x7fcce8102ac0 2024-04-21T08:49:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.636+0000 7fccedd05700 1 -- 172.21.15.28:0/1543849854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcce81a7950 con 0x7fcce8102ac0 2024-04-21T08:49:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.636+0000 7fcce4ff9700 1 -- 172.21.15.28:0/1543849854 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fccd8022470 con 0x7fcce8102ac0 2024-04-21T08:49:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.636+0000 7fcce4ff9700 1 -- 172.21.15.28:0/1543849854 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd8017650 con 0x7fcce8102ac0 2024-04-21T08:49:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.636+0000 7fccedd05700 1 -- 172.21.15.28:0/1543849854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcce81a7db0 con 0x7fcce8102ac0 2024-04-21T08:49:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.636+0000 7fccedd05700 1 -- 172.21.15.28:0/1543849854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcce80630d0 con 0x7fcce8102ac0 2024-04-21T08:49:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.638+0000 7fcce4ff9700 1 -- 172.21.15.28:0/1543849854 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fccd802a070 con 0x7fcce8102ac0 2024-04-21T08:49:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.638+0000 7fcce4ff9700 1 --2- 172.21.15.28:0/1543849854 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fccd00a3cd0 0x7fccd00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.639+0000 7fcce6ffd700 1 --2- 172.21.15.28:0/1543849854 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fccd00a3cd0 0x7fccd00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.639+0000 7fcce4ff9700 1 -- 172.21.15.28:0/1543849854 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fccd80d21f0 con 0x7fcce8102ac0 2024-04-21T08:49:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.639+0000 7fcce4ff9700 1 --2- 172.21.15.28:0/1543849854 >> [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959] conn(0x7fccd00aedb0 0x7fccd00b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.639+0000 7fcce4ff9700 1 -- 172.21.15.28:0/1543849854 --> [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fccd00b1920 con 0x7fccd00aedb0 2024-04-21T08:49:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.639+0000 7fcce4ff9700 1 -- 172.21.15.28:0/1543849854 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fccd80d2640 con 0x7fcce8102ac0 2024-04-21T08:49:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.639+0000 7fcce7fff700 1 --2- 172.21.15.28:0/1543849854 >> [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959] conn(0x7fccd00aedb0 0x7fccd00b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.639+0000 7fcce6ffd700 1 --2- 172.21.15.28:0/1543849854 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fccd00a3cd0 0x7fccd00a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcce807a2d0 tx=0x7fccdc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.640+0000 7fcce7fff700 1 --2- 172.21.15.28:0/1543849854 >> [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959] conn(0x7fccd00aedb0 0x7fccd00b1250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:40.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.640+0000 7fcce4ff9700 1 -- 172.21.15.28:0/1543849854 <== osd.3 v2:172.21.15.28:6810/1806506959 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fccd00b1920 con 0x7fccd00aedb0 2024-04-21T08:49:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.673+0000 7fccedd05700 1 -- 172.21.15.28:0/1543849854 --> [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcce80634d0 con 0x7fccd00aedb0 2024-04-21T08:49:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.674+0000 7fcce4ff9700 1 -- 172.21.15.28:0/1543849854 <== osd.3 v2:172.21.15.28:6810/1806506959 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcce80634d0 con 0x7fccd00aedb0 2024-04-21T08:49:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.674+0000 7fccedd05700 1 -- 172.21.15.28:0/1543849854 >> [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959] conn(0x7fccd00aedb0 msgr2=0x7fccd00b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.674+0000 7fccedd05700 1 --2- 172.21.15.28:0/1543849854 >> [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959] conn(0x7fccd00aedb0 0x7fccd00b1250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.674+0000 7fccedd05700 1 -- 172.21.15.28:0/1543849854 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fccd00a3cd0 msgr2=0x7fccd00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.674+0000 7fccedd05700 1 --2- 172.21.15.28:0/1543849854 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fccd00a3cd0 0x7fccd00a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcce807a2d0 tx=0x7fccdc009260 comp rx=0 tx=0).stop 2024-04-21T08:49:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.674+0000 7fccedd05700 1 -- 172.21.15.28:0/1543849854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce8102ac0 msgr2=0x7fcce8074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.674+0000 7fccedd05700 1 --2- 172.21.15.28:0/1543849854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcce8102ac0 0x7fcce8074d40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fccd8000c00 tx=0x7fccd8009580 comp rx=0 tx=0).stop 2024-04-21T08:49:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.674+0000 7fcce7fff700 1 -- 172.21.15.28:0/1543849854 reap_dead start 2024-04-21T08:49:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.674+0000 7fccedd05700 1 -- 172.21.15.28:0/1543849854 shutdown_connections 2024-04-21T08:49:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.674+0000 7fccedd05700 1 -- 172.21.15.28:0/1543849854 >> 172.21.15.28:0/1543849854 conn(0x7fcce8079bf0 msgr2=0x7fcce8100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.674+0000 7fccedd05700 1 -- 172.21.15.28:0/1543849854 shutdown_connections 2024-04-21T08:49:40.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:38.674+0000 7fccedd05700 1 -- 172.21.15.28:0/1543849854 wait complete. 2024-04-21T08:49:40.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:40 smithi028 ceph-mon[28003]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 511 B/s wr, 5 op/s; 86 B/s, 1 objects/s recovering 2024-04-21T08:49:41.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:40 smithi152 ceph-mon[31401]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 511 B/s wr, 5 op/s; 86 B/s, 1 objects/s recovering 2024-04-21T08:49:41.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:40 smithi171 ceph-mon[31358]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 511 B/s wr, 5 op/s; 86 B/s, 1 objects/s recovering 2024-04-21T08:49:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.389+0000 7fe93a56e700 1 -- 172.21.15.28:0/4207015131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe934101940 msgr2=0x7fe934101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.389+0000 7fe93a56e700 1 --2- 172.21.15.28:0/4207015131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe934101940 0x7fe934101dc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe928009990 tx=0x7fe928009ca0 comp rx=0 tx=0).stop 2024-04-21T08:49:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.389+0000 7fe93a56e700 1 -- 172.21.15.28:0/4207015131 shutdown_connections 2024-04-21T08:49:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.389+0000 7fe93a56e700 1 --2- 172.21.15.28:0/4207015131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe934102300 0x7fe93410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.389+0000 7fe93a56e700 1 --2- 172.21.15.28:0/4207015131 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe934107990 0x7fe934107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.389+0000 7fe93a56e700 1 --2- 172.21.15.28:0/4207015131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe934101940 0x7fe934101dc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.389+0000 7fe93a56e700 1 -- 172.21.15.28:0/4207015131 >> 172.21.15.28:0/4207015131 conn(0x7fe9340788f0 msgr2=0x7fe934078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:41.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.389+0000 7fe93a56e700 1 -- 172.21.15.28:0/4207015131 shutdown_connections 2024-04-21T08:49:41.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.389+0000 7fe93a56e700 1 -- 172.21.15.28:0/4207015131 wait complete. 2024-04-21T08:49:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.390+0000 7fe93a56e700 1 Processor -- start 2024-04-21T08:49:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.390+0000 7fe93a56e700 1 -- start start 2024-04-21T08:49:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.390+0000 7fe93a56e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe934101940 0x7fe9341992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.390+0000 7fe93a56e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe934102300 0x7fe934199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.390+0000 7fe93a56e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe934107990 0x7fe93419db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.390+0000 7fe93a56e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe934111c20 con 0x7fe934107990 2024-04-21T08:49:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.390+0000 7fe93a56e700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe934111aa0 con 0x7fe934101940 2024-04-21T08:49:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.390+0000 7fe93a56e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe934111da0 con 0x7fe934102300 2024-04-21T08:49:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe938b0b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe934107990 0x7fe93419db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe938b0b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe934107990 0x7fe93419db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54808/0 (socket says 172.21.15.28:54808) 2024-04-21T08:49:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe938b0b700 1 -- 172.21.15.28:0/2731210415 learned_addr learned my addr 172.21.15.28:0/2731210415 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe933fff700 1 --2- 172.21.15.28:0/2731210415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe934101940 0x7fe9341992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe9337fe700 1 --2- 172.21.15.28:0/2731210415 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe934102300 0x7fe934199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe938b0b700 1 -- 172.21.15.28:0/2731210415 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe934102300 msgr2=0x7fe934199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe938b0b700 1 --2- 172.21.15.28:0/2731210415 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe934102300 0x7fe934199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe938b0b700 1 -- 172.21.15.28:0/2731210415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe934101940 msgr2=0x7fe9341992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe938b0b700 1 --2- 172.21.15.28:0/2731210415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe934101940 0x7fe9341992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe938b0b700 1 -- 172.21.15.28:0/2731210415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93419e280 con 0x7fe934107990 2024-04-21T08:49:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe9337fe700 1 --2- 172.21.15.28:0/2731210415 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe934102300 0x7fe934199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe938b0b700 1 --2- 172.21.15.28:0/2731210415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe934107990 0x7fe93419db40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe92c00b970 tx=0x7fe92c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe9317fa700 1 -- 172.21.15.28:0/2731210415 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe92c019070 con 0x7fe934107990 2024-04-21T08:49:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe9317fa700 1 -- 172.21.15.28:0/2731210415 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe92c00cde0 con 0x7fe934107990 2024-04-21T08:49:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe93a56e700 1 -- 172.21.15.28:0/2731210415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe93419e570 con 0x7fe934107990 2024-04-21T08:49:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe9317fa700 1 -- 172.21.15.28:0/2731210415 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe92c007370 con 0x7fe934107990 2024-04-21T08:49:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.391+0000 7fe93a56e700 1 -- 172.21.15.28:0/2731210415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9341a5df0 con 0x7fe934107990 2024-04-21T08:49:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.392+0000 7fe93a56e700 1 -- 172.21.15.28:0/2731210415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe934063150 con 0x7fe934107990 2024-04-21T08:49:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.394+0000 7fe9317fa700 1 -- 172.21.15.28:0/2731210415 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe92c01c070 con 0x7fe934107990 2024-04-21T08:49:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.394+0000 7fe9317fa700 1 --2- 172.21.15.28:0/2731210415 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe91c0a3cd0 0x7fe91c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.394+0000 7fe9317fa700 1 -- 172.21.15.28:0/2731210415 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fe92c0d06d0 con 0x7fe934107990 2024-04-21T08:49:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.394+0000 7fe933fff700 1 --2- 172.21.15.28:0/2731210415 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe91c0a3cd0 0x7fe91c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.394+0000 7fe9317fa700 1 --2- 172.21.15.28:0/2731210415 >> [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438] conn(0x7fe91c0aed30 0x7fe91c0b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.394+0000 7fe9317fa700 1 -- 172.21.15.28:0/2731210415 --> [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe92c004860 con 0x7fe91c0aed30 2024-04-21T08:49:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.394+0000 7fe9317fa700 1 -- 172.21.15.28:0/2731210415 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fe92c0d0b20 con 0x7fe934107990 2024-04-21T08:49:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.395+0000 7fe9337fe700 1 --2- 172.21.15.28:0/2731210415 >> [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438] conn(0x7fe91c0aed30 0x7fe91c0b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.395+0000 7fe933fff700 1 --2- 172.21.15.28:0/2731210415 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe91c0a3cd0 0x7fe91c0a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe924009a10 tx=0x7fe924009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.395+0000 7fe9337fe700 1 --2- 172.21.15.28:0/2731210415 >> [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438] conn(0x7fe91c0aed30 0x7fe91c0b11d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:41.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.396+0000 7fe9317fa700 1 -- 172.21.15.28:0/2731210415 <== osd.1 v2:172.21.15.152:6800/728547438 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe92c004860 con 0x7fe91c0aed30 2024-04-21T08:49:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.412+0000 7fe93a56e700 1 -- 172.21.15.28:0/2731210415 --> [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe9340635d0 con 0x7fe91c0aed30 2024-04-21T08:49:41.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.413+0000 7fe9317fa700 1 -- 172.21.15.28:0/2731210415 <== osd.1 v2:172.21.15.152:6800/728547438 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe9340635d0 con 0x7fe91c0aed30 2024-04-21T08:49:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.413+0000 7fe93a56e700 1 -- 172.21.15.28:0/2731210415 >> [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438] conn(0x7fe91c0aed30 msgr2=0x7fe91c0b11d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.413+0000 7fe93a56e700 1 --2- 172.21.15.28:0/2731210415 >> [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438] conn(0x7fe91c0aed30 0x7fe91c0b11d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.414+0000 7fe93a56e700 1 -- 172.21.15.28:0/2731210415 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe91c0a3cd0 msgr2=0x7fe91c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.414+0000 7fe93a56e700 1 --2- 172.21.15.28:0/2731210415 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe91c0a3cd0 0x7fe91c0a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe924009a10 tx=0x7fe924009190 comp rx=0 tx=0).stop 2024-04-21T08:49:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.414+0000 7fe93a56e700 1 -- 172.21.15.28:0/2731210415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe934107990 msgr2=0x7fe93419db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.414+0000 7fe93a56e700 1 --2- 172.21.15.28:0/2731210415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe934107990 0x7fe93419db40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe92c00b970 tx=0x7fe92c00bc80 comp rx=0 tx=0).stop 2024-04-21T08:49:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.414+0000 7fe938b0b700 1 -- 172.21.15.28:0/2731210415 reap_dead start 2024-04-21T08:49:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.414+0000 7fe93a56e700 1 -- 172.21.15.28:0/2731210415 shutdown_connections 2024-04-21T08:49:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.414+0000 7fe93a56e700 1 -- 172.21.15.28:0/2731210415 >> 172.21.15.28:0/2731210415 conn(0x7fe9340788f0 msgr2=0x7fe93406ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.414+0000 7fe93a56e700 1 -- 172.21.15.28:0/2731210415 shutdown_connections 2024-04-21T08:49:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:41.414+0000 7fe93a56e700 1 -- 172.21.15.28:0/2731210415 wait complete. 2024-04-21T08:49:42.032 INFO:teuthology.orchestra.run.smithi028.stdout:107374182465 2024-04-21T08:49:42.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-21T08:49:42.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.545+0000 7f643eb02700 1 -- 172.21.15.28:0/1482058242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438108a50 msgr2=0x7f6438108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:42.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.545+0000 7f643eb02700 1 --2- 172.21.15.28:0/1482058242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438108a50 0x7f6438108e30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6428009990 tx=0x7f6428009ca0 comp rx=0 tx=0).stop 2024-04-21T08:49:42.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.545+0000 7f643eb02700 1 -- 172.21.15.28:0/1482058242 shutdown_connections 2024-04-21T08:49:42.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.545+0000 7f643eb02700 1 --2- 172.21.15.28:0/1482058242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438108a50 0x7f6438108e30 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:42.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.545+0000 7f643eb02700 1 --2- 172.21.15.28:0/1482058242 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6438103450 0x7f643810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:42.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.545+0000 7f643eb02700 1 --2- 172.21.15.28:0/1482058242 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6438102a90 0x7f6438102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:42.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.545+0000 7f643eb02700 1 -- 172.21.15.28:0/1482058242 >> 172.21.15.28:0/1482058242 conn(0x7f6438079ba0 msgr2=0x7f6438079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:42.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.545+0000 7f643eb02700 1 -- 172.21.15.28:0/1482058242 shutdown_connections 2024-04-21T08:49:42.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.545+0000 7f643eb02700 1 -- 172.21.15.28:0/1482058242 wait complete. 2024-04-21T08:49:42.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.546+0000 7f643eb02700 1 Processor -- start 2024-04-21T08:49:42.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.546+0000 7f643eb02700 1 -- start start 2024-04-21T08:49:42.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.546+0000 7f643eb02700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6438102a90 0x7f6438074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:42.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.546+0000 7f643eb02700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438103450 0x7f6438075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f643eb02700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6438108a50 0x7f64380757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f643eb02700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6438112be0 con 0x7f6438103450 2024-04-21T08:49:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f643eb02700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6438112a60 con 0x7f6438108a50 2024-04-21T08:49:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f643eb02700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6438112d60 con 0x7f6438102a90 2024-04-21T08:49:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f6437fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438103450 0x7f6438075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f6437fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438103450 0x7f6438075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54836/0 (socket says 172.21.15.28:54836) 2024-04-21T08:49:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f6437fff700 1 -- 172.21.15.28:0/3458393475 learned_addr learned my addr 172.21.15.28:0/3458393475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f643c89e700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6438102a90 0x7f6438074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f643d09f700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6438108a50 0x7f64380757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f6437fff700 1 -- 172.21.15.28:0/3458393475 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6438102a90 msgr2=0x7f6438074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f6437fff700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6438102a90 0x7f6438074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f6437fff700 1 -- 172.21.15.28:0/3458393475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6438108a50 msgr2=0x7f64380757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f6437fff700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6438108a50 0x7f64380757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f6437fff700 1 -- 172.21.15.28:0/3458393475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64381a75c0 con 0x7f6438103450 2024-04-21T08:49:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f643c89e700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6438102a90 0x7f6438074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f643d09f700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6438108a50 0x7f64380757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f6437fff700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438103450 0x7f6438075280 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f642c00d950 tx=0x7f642c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f6435ffb700 1 -- 172.21.15.28:0/3458393475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c016070 con 0x7f6438103450 2024-04-21T08:49:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f6435ffb700 1 -- 172.21.15.28:0/3458393475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f642c010460 con 0x7f6438103450 2024-04-21T08:49:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f643eb02700 1 -- 172.21.15.28:0/3458393475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64381a78b0 con 0x7f6438103450 2024-04-21T08:49:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.547+0000 7f6435ffb700 1 -- 172.21.15.28:0/3458393475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642c00f620 con 0x7f6438103450 2024-04-21T08:49:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.548+0000 7f643eb02700 1 -- 172.21.15.28:0/3458393475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64381a7df0 con 0x7f6438103450 2024-04-21T08:49:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.548+0000 7f643eb02700 1 -- 172.21.15.28:0/3458393475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f64380630d0 con 0x7f6438103450 2024-04-21T08:49:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.550+0000 7f6435ffb700 1 -- 172.21.15.28:0/3458393475 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f642c019070 con 0x7f6438103450 2024-04-21T08:49:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.550+0000 7f6435ffb700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f64200a3cd0 0x7f64200a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.550+0000 7f6435ffb700 1 -- 172.21.15.28:0/3458393475 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f642c0d0a50 con 0x7f6438103450 2024-04-21T08:49:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.550+0000 7f643c89e700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f64200a3cd0 0x7f64200a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:42.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.550+0000 7f6435ffb700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239] conn(0x7f64200aedb0 0x7f64200b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:42.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.550+0000 7f6435ffb700 1 -- 172.21.15.28:0/3458393475 --> [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f642c013ec0 con 0x7f64200aedb0 2024-04-21T08:49:42.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.550+0000 7f6435ffb700 1 -- 172.21.15.28:0/3458393475 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f642c0d0ea0 con 0x7f6438103450 2024-04-21T08:49:42.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.551+0000 7f643c89e700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f64200a3cd0 0x7f64200a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f642800b5c0 tx=0x7f6428005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:42.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.551+0000 7f643d09f700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239] conn(0x7f64200aedb0 0x7f64200b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:42.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.551+0000 7f643d09f700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239] conn(0x7f64200aedb0 0x7f64200b1250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:42.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.552+0000 7f6435ffb700 1 -- 172.21.15.28:0/3458393475 <== osd.5 v2:172.21.15.171:6808/3821828239 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f642c013ec0 con 0x7f64200aedb0 2024-04-21T08:49:42.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.569+0000 7f643eb02700 1 -- 172.21.15.28:0/3458393475 --> [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6438063550 con 0x7f64200aedb0 2024-04-21T08:49:42.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.569+0000 7f6435ffb700 1 -- 172.21.15.28:0/3458393475 <== osd.5 v2:172.21.15.171:6808/3821828239 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6438063550 con 0x7f64200aedb0 2024-04-21T08:49:42.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.569+0000 7f643eb02700 1 -- 172.21.15.28:0/3458393475 >> [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239] conn(0x7f64200aedb0 msgr2=0x7f64200b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:42.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.569+0000 7f643eb02700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239] conn(0x7f64200aedb0 0x7f64200b1250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:42.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.570+0000 7f643eb02700 1 -- 172.21.15.28:0/3458393475 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f64200a3cd0 msgr2=0x7f64200a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:42.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.570+0000 7f643eb02700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f64200a3cd0 0x7f64200a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f642800b5c0 tx=0x7f6428005fb0 comp rx=0 tx=0).stop 2024-04-21T08:49:42.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.570+0000 7f643eb02700 1 -- 172.21.15.28:0/3458393475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438103450 msgr2=0x7f6438075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:42.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.570+0000 7f643eb02700 1 --2- 172.21.15.28:0/3458393475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6438103450 0x7f6438075280 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f642c00d950 tx=0x7f642c00dd10 comp rx=0 tx=0).stop 2024-04-21T08:49:42.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.570+0000 7f643d09f700 1 -- 172.21.15.28:0/3458393475 reap_dead start 2024-04-21T08:49:42.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.570+0000 7f643eb02700 1 -- 172.21.15.28:0/3458393475 shutdown_connections 2024-04-21T08:49:42.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.570+0000 7f643eb02700 1 -- 172.21.15.28:0/3458393475 >> 172.21.15.28:0/3458393475 conn(0x7f6438079ba0 msgr2=0x7f6438101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:42.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.570+0000 7f643eb02700 1 -- 172.21.15.28:0/3458393475 shutdown_connections 2024-04-21T08:49:42.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:42.570+0000 7f643eb02700 1 -- 172.21.15.28:0/3458393475 wait complete. 2024-04-21T08:49:43.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:42 smithi152 ceph-mon[31401]: pgmap v324: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 6 op/s; 86 B/s, 1 objects/s recovering 2024-04-21T08:49:43.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:42 smithi028 ceph-mon[28003]: pgmap v324: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 6 op/s; 86 B/s, 1 objects/s recovering 2024-04-21T08:49:43.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:42 smithi171 ceph-mon[31358]: pgmap v324: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 6 op/s; 86 B/s, 1 objects/s recovering 2024-04-21T08:49:43.384 INFO:teuthology.orchestra.run.smithi028.stdout:77309411399 2024-04-21T08:49:43.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-21T08:49:44.002 INFO:teuthology.orchestra.run.smithi028.stdout:55834574923 2024-04-21T08:49:44.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-21T08:49:44.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:44 smithi028 ceph-mon[28003]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s; 86 B/s, 1 objects/s recovering 2024-04-21T08:49:45.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:44 smithi152 ceph-mon[31401]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s; 86 B/s, 1 objects/s recovering 2024-04-21T08:49:45.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:44 smithi171 ceph-mon[31358]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s; 86 B/s, 1 objects/s recovering 2024-04-21T08:49:45.851 INFO:teuthology.orchestra.run.smithi028.stdout:141733920828 2024-04-21T08:49:45.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-21T08:49:46.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:45 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:49:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.299+0000 7fb805b17700 1 -- 172.21.15.28:0/1948705468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb80006b980 msgr2=0x7fb80010f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.299+0000 7fb805b17700 1 --2- 172.21.15.28:0/1948705468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb80006b980 0x7fb80010f8f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800b0a0 tx=0x7fb7f800b3b0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.299+0000 7fb805b17700 1 -- 172.21.15.28:0/1948705468 shutdown_connections 2024-04-21T08:49:46.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.299+0000 7fb805b17700 1 --2- 172.21.15.28:0/1948705468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb80006b980 0x7fb80010f8f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.299+0000 7fb805b17700 1 --2- 172.21.15.28:0/1948705468 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb800108a20 0x7fb800108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.299+0000 7fb805b17700 1 --2- 172.21.15.28:0/1948705468 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb80006afe0 0x7fb80006b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.299+0000 7fb805b17700 1 -- 172.21.15.28:0/1948705468 >> 172.21.15.28:0/1948705468 conn(0x7fb8000788b0 msgr2=0x7fb800078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:46.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.299+0000 7fb805b17700 1 -- 172.21.15.28:0/1948705468 shutdown_connections 2024-04-21T08:49:46.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.299+0000 7fb805b17700 1 -- 172.21.15.28:0/1948705468 wait complete. 2024-04-21T08:49:46.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.300+0000 7fb805b17700 1 Processor -- start 2024-04-21T08:49:46.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.300+0000 7fb805b17700 1 -- start start 2024-04-21T08:49:46.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.300+0000 7fb805b17700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb80006afe0 0x7fb800101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.300+0000 7fb805b17700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb80006b980 0x7fb800101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.300+0000 7fb805b17700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb800108a20 0x7fb800102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.300+0000 7fb805b17700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb800112bc0 con 0x7fb80006b980 2024-04-21T08:49:46.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.300+0000 7fb805b17700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb800112a40 con 0x7fb80006afe0 2024-04-21T08:49:46.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.300+0000 7fb805b17700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb800112d40 con 0x7fb800108a20 2024-04-21T08:49:46.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.300+0000 7fb7feffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb80006b980 0x7fb800101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.300+0000 7fb7feffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb80006b980 0x7fb800101f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36642/0 (socket says 172.21.15.28:36642) 2024-04-21T08:49:46.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.300+0000 7fb7feffd700 1 -- 172.21.15.28:0/2977882105 learned_addr learned my addr 172.21.15.28:0/2977882105 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:46.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.300+0000 7fb7ff7fe700 1 --2- 172.21.15.28:0/2977882105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb80006afe0 0x7fb800101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.300+0000 7fb7fffff700 1 --2- 172.21.15.28:0/2977882105 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb800108a20 0x7fb800102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.301+0000 7fb7fffff700 1 -- 172.21.15.28:0/2977882105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb80006afe0 msgr2=0x7fb800101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.301+0000 7fb7fffff700 1 --2- 172.21.15.28:0/2977882105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb80006afe0 0x7fb800101a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.301+0000 7fb7fffff700 1 -- 172.21.15.28:0/2977882105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb80006b980 msgr2=0x7fb800101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.301+0000 7fb7fffff700 1 --2- 172.21.15.28:0/2977882105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb80006b980 0x7fb800101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.301+0000 7fb7fffff700 1 -- 172.21.15.28:0/2977882105 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8001a75f0 con 0x7fb800108a20 2024-04-21T08:49:46.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.301+0000 7fb7ff7fe700 1 --2- 172.21.15.28:0/2977882105 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb80006afe0 0x7fb800101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:46.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.301+0000 7fb7fffff700 1 --2- 172.21.15.28:0/2977882105 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb800108a20 0x7fb800102480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800d010 tx=0x7fb7f8004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:46.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.301+0000 7fb7fcff9700 1 -- 172.21.15.28:0/2977882105 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8021070 con 0x7fb800108a20 2024-04-21T08:49:46.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.301+0000 7fb7fcff9700 1 -- 172.21.15.28:0/2977882105 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7f8007e30 con 0x7fb800108a20 2024-04-21T08:49:46.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.301+0000 7fb805b17700 1 -- 172.21.15.28:0/2977882105 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8001a7880 con 0x7fb800108a20 2024-04-21T08:49:46.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.301+0000 7fb7fcff9700 1 -- 172.21.15.28:0/2977882105 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f80045a0 con 0x7fb800108a20 2024-04-21T08:49:46.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.301+0000 7fb805b17700 1 -- 172.21.15.28:0/2977882105 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8001a7d80 con 0x7fb800108a20 2024-04-21T08:49:46.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.302+0000 7fb805b17700 1 -- 172.21.15.28:0/2977882105 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb8000630d0 con 0x7fb800108a20 2024-04-21T08:49:46.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.304+0000 7fb7fcff9700 1 -- 172.21.15.28:0/2977882105 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb7f8004740 con 0x7fb800108a20 2024-04-21T08:49:46.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.304+0000 7fb7fcff9700 1 --2- 172.21.15.28:0/2977882105 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb7e80a3d50 0x7fb7e80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.305+0000 7fb7ff7fe700 1 --2- 172.21.15.28:0/2977882105 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb7e80a3d50 0x7fb7e80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.305+0000 7fb7fcff9700 1 -- 172.21.15.28:0/2977882105 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fb7f800f020 con 0x7fb800108a20 2024-04-21T08:49:46.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.305+0000 7fb7fcff9700 1 --2- 172.21.15.28:0/2977882105 >> [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516] conn(0x7fb7e80aed00 0x7fb7e80b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:46.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.305+0000 7fb7fcff9700 1 -- 172.21.15.28:0/2977882105 --> [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7e80b1870 con 0x7fb7e80aed00 2024-04-21T08:49:46.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.305+0000 7fb7fcff9700 1 -- 172.21.15.28:0/2977882105 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fb7f80d2130 con 0x7fb800108a20 2024-04-21T08:49:46.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.305+0000 7fb7feffd700 1 --2- 172.21.15.28:0/2977882105 >> [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516] conn(0x7fb7e80aed00 0x7fb7e80b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:46.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.305+0000 7fb7ff7fe700 1 --2- 172.21.15.28:0/2977882105 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb7e80a3d50 0x7fb7e80a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009fd0 tx=0x7fb7f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:46.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.306+0000 7fb7feffd700 1 --2- 172.21.15.28:0/2977882105 >> [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516] conn(0x7fb7e80aed00 0x7fb7e80b11a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:46.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.306+0000 7fb7fcff9700 1 -- 172.21.15.28:0/2977882105 <== osd.7 v2:172.21.15.152:6816/3749693516 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb7e80b1870 con 0x7fb7e80aed00 2024-04-21T08:49:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.323+0000 7fb805b17700 1 -- 172.21.15.28:0/2977882105 --> [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb800063550 con 0x7fb7e80aed00 2024-04-21T08:49:46.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.323+0000 7fb7fcff9700 1 -- 172.21.15.28:0/2977882105 <== osd.7 v2:172.21.15.152:6816/3749693516 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb800063550 con 0x7fb7e80aed00 2024-04-21T08:49:46.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.324+0000 7fb805b17700 1 -- 172.21.15.28:0/2977882105 >> [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516] conn(0x7fb7e80aed00 msgr2=0x7fb7e80b11a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.324+0000 7fb805b17700 1 --2- 172.21.15.28:0/2977882105 >> [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516] conn(0x7fb7e80aed00 0x7fb7e80b11a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:46.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.324+0000 7fb805b17700 1 -- 172.21.15.28:0/2977882105 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb7e80a3d50 msgr2=0x7fb7e80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.324+0000 7fb805b17700 1 --2- 172.21.15.28:0/2977882105 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb7e80a3d50 0x7fb7e80a6210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009fd0 tx=0x7fb7f0009260 comp rx=0 tx=0).stop 2024-04-21T08:49:46.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.324+0000 7fb805b17700 1 -- 172.21.15.28:0/2977882105 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb800108a20 msgr2=0x7fb800102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:46.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.324+0000 7fb805b17700 1 --2- 172.21.15.28:0/2977882105 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb800108a20 0x7fb800102480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800d010 tx=0x7fb7f8004d00 comp rx=0 tx=0).stop 2024-04-21T08:49:46.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.324+0000 7fb7fffff700 1 -- 172.21.15.28:0/2977882105 reap_dead start 2024-04-21T08:49:46.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.324+0000 7fb805b17700 1 -- 172.21.15.28:0/2977882105 shutdown_connections 2024-04-21T08:49:46.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.324+0000 7fb805b17700 1 -- 172.21.15.28:0/2977882105 >> 172.21.15.28:0/2977882105 conn(0x7fb8000788b0 msgr2=0x7fb8000ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:46.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.324+0000 7fb805b17700 1 -- 172.21.15.28:0/2977882105 shutdown_connections 2024-04-21T08:49:46.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:46.324+0000 7fb805b17700 1 -- 172.21.15.28:0/2977882105 wait complete. 2024-04-21T08:49:46.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:45 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:49:46.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:45 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:49:47.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:46 smithi028 ceph-mon[28003]: pgmap v326: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 427 B/s wr, 6 op/s; 72 B/s, 1 objects/s recovering 2024-04-21T08:49:47.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:46 smithi152 ceph-mon[31401]: pgmap v326: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 427 B/s wr, 6 op/s; 72 B/s, 1 objects/s recovering 2024-04-21T08:49:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.362+0000 7f8ac9705700 1 -- 172.21.15.28:0/535337003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ac4106990 msgr2=0x7f8ac4106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.362+0000 7f8ac9705700 1 --2- 172.21.15.28:0/535337003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ac4106990 0x7f8ac4106d70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4009990 tx=0x7f8ab4009ca0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.362+0000 7f8ac9705700 1 -- 172.21.15.28:0/535337003 shutdown_connections 2024-04-21T08:49:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.362+0000 7f8ac9705700 1 --2- 172.21.15.28:0/535337003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ac4106990 0x7f8ac4106d70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.362+0000 7f8ac9705700 1 --2- 172.21.15.28:0/535337003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ac41026e0 0x7f8ac410e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.362+0000 7f8ac9705700 1 --2- 172.21.15.28:0/535337003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ac4101d20 0x7f8ac41021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.362+0000 7f8ac9705700 1 -- 172.21.15.28:0/535337003 >> 172.21.15.28:0/535337003 conn(0x7f8ac40788f0 msgr2=0x7f8ac4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.363+0000 7f8ac9705700 1 -- 172.21.15.28:0/535337003 shutdown_connections 2024-04-21T08:49:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.363+0000 7f8ac9705700 1 -- 172.21.15.28:0/535337003 wait complete. 2024-04-21T08:49:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.363+0000 7f8ac9705700 1 Processor -- start 2024-04-21T08:49:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.363+0000 7f8ac9705700 1 -- start start 2024-04-21T08:49:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.363+0000 7f8ac9705700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ac4101d20 0x7f8ac41990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.363+0000 7f8ac9705700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ac41026e0 0x7f8ac41995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.363+0000 7f8ac9705700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ac4106990 0x7f8ac419d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac9705700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac4111ad0 con 0x7f8ac4106990 2024-04-21T08:49:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac9705700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac4111950 con 0x7f8ac41026e0 2024-04-21T08:49:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac9705700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac4111c50 con 0x7f8ac4101d20 2024-04-21T08:49:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ac4106990 0x7f8ac419d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac2ffd700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ac4101d20 0x7f8ac41990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac27fc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ac41026e0 0x7f8ac41995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac37fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ac4106990 0x7f8ac419d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36650/0 (socket says 172.21.15.28:36650) 2024-04-21T08:49:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac37fe700 1 -- 172.21.15.28:0/2364608290 learned_addr learned my addr 172.21.15.28:0/2364608290 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac2ffd700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ac4101d20 0x7f8ac41990b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:40080/0 (socket says 172.21.15.28:40080) 2024-04-21T08:49:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac2ffd700 1 -- 172.21.15.28:0/2364608290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ac41026e0 msgr2=0x7f8ac41995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac2ffd700 1 --2- 172.21.15.28:0/2364608290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8ac41026e0 0x7f8ac41995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac2ffd700 1 -- 172.21.15.28:0/2364608290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ac4106990 msgr2=0x7f8ac419d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac2ffd700 1 --2- 172.21.15.28:0/2364608290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ac4106990 0x7f8ac419d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac2ffd700 1 -- 172.21.15.28:0/2364608290 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac419e0c0 con 0x7f8ac4101d20 2024-04-21T08:49:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac2ffd700 1 --2- 172.21.15.28:0/2364608290 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ac4101d20 0x7f8ac41990b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40059d0 tx=0x7f8ab40047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8aabfff700 1 -- 172.21.15.28:0/2364608290 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4027070 con 0x7f8ac4101d20 2024-04-21T08:49:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac9705700 1 -- 172.21.15.28:0/2364608290 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ac419e350 con 0x7f8ac4101d20 2024-04-21T08:49:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8aabfff700 1 -- 172.21.15.28:0/2364608290 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ab4022470 con 0x7f8ac4101d20 2024-04-21T08:49:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8aabfff700 1 -- 172.21.15.28:0/2364608290 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4017650 con 0x7f8ac4101d20 2024-04-21T08:49:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.364+0000 7f8ac9705700 1 -- 172.21.15.28:0/2364608290 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ac41a5c30 con 0x7f8ac4101d20 2024-04-21T08:49:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.365+0000 7f8ac9705700 1 -- 172.21.15.28:0/2364608290 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8ac40631d0 con 0x7f8ac4101d20 2024-04-21T08:49:47.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.367+0000 7f8aabfff700 1 -- 172.21.15.28:0/2364608290 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8ab402a070 con 0x7f8ac4101d20 2024-04-21T08:49:47.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.367+0000 7f8aabfff700 1 --2- 172.21.15.28:0/2364608290 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8aac0a3cd0 0x7f8aac0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.368+0000 7f8ac27fc700 1 --2- 172.21.15.28:0/2364608290 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8aac0a3cd0 0x7f8aac0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.368+0000 7f8aabfff700 1 -- 172.21.15.28:0/2364608290 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f8ab40d1cf0 con 0x7f8ac4101d20 2024-04-21T08:49:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.368+0000 7f8aabfff700 1 --2- 172.21.15.28:0/2364608290 >> [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122] conn(0x7f8aac0aedb0 0x7f8aac0b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.368+0000 7f8aabfff700 1 -- 172.21.15.28:0/2364608290 --> [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8aac0b1920 con 0x7f8aac0aedb0 2024-04-21T08:49:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.368+0000 7f8aabfff700 1 -- 172.21.15.28:0/2364608290 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f8ab40d20a0 con 0x7f8ac4101d20 2024-04-21T08:49:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.368+0000 7f8ac37fe700 1 --2- 172.21.15.28:0/2364608290 >> [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122] conn(0x7f8aac0aedb0 0x7f8aac0b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.368+0000 7f8ac27fc700 1 --2- 172.21.15.28:0/2364608290 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8aac0a3cd0 0x7f8aac0a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4079040 tx=0x7f8ab8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.368+0000 7f8ac37fe700 1 --2- 172.21.15.28:0/2364608290 >> [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122] conn(0x7f8aac0aedb0 0x7f8aac0b1250 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.369+0000 7f8aabfff700 1 -- 172.21.15.28:0/2364608290 <== osd.6 v2:172.21.15.28:6818/38541122 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8aac0b1920 con 0x7f8aac0aedb0 2024-04-21T08:49:47.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:46 smithi171 ceph-mon[31358]: pgmap v326: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 427 B/s wr, 6 op/s; 72 B/s, 1 objects/s recovering 2024-04-21T08:49:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.385+0000 7f8ac9705700 1 -- 172.21.15.28:0/2364608290 --> [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8ac40635d0 con 0x7f8aac0aedb0 2024-04-21T08:49:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.386+0000 7f8aabfff700 1 -- 172.21.15.28:0/2364608290 <== osd.6 v2:172.21.15.28:6818/38541122 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8ac40635d0 con 0x7f8aac0aedb0 2024-04-21T08:49:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.386+0000 7f8ac9705700 1 -- 172.21.15.28:0/2364608290 >> [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122] conn(0x7f8aac0aedb0 msgr2=0x7f8aac0b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.386+0000 7f8ac9705700 1 --2- 172.21.15.28:0/2364608290 >> [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122] conn(0x7f8aac0aedb0 0x7f8aac0b1250 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.386+0000 7f8ac9705700 1 -- 172.21.15.28:0/2364608290 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8aac0a3cd0 msgr2=0x7f8aac0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.386+0000 7f8ac9705700 1 --2- 172.21.15.28:0/2364608290 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f8aac0a3cd0 0x7f8aac0a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4079040 tx=0x7f8ab8009260 comp rx=0 tx=0).stop 2024-04-21T08:49:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.386+0000 7f8ac9705700 1 -- 172.21.15.28:0/2364608290 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ac4101d20 msgr2=0x7f8ac41990b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.386+0000 7f8ac9705700 1 --2- 172.21.15.28:0/2364608290 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ac4101d20 0x7f8ac41990b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40059d0 tx=0x7f8ab40047b0 comp rx=0 tx=0).stop 2024-04-21T08:49:47.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.386+0000 7f8ac37fe700 1 -- 172.21.15.28:0/2364608290 reap_dead start 2024-04-21T08:49:47.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.386+0000 7f8ac9705700 1 -- 172.21.15.28:0/2364608290 shutdown_connections 2024-04-21T08:49:47.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.386+0000 7f8ac9705700 1 -- 172.21.15.28:0/2364608290 >> 172.21.15.28:0/2364608290 conn(0x7f8ac40788f0 msgr2=0x7f8ac406b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:47.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.386+0000 7f8ac9705700 1 -- 172.21.15.28:0/2364608290 shutdown_connections 2024-04-21T08:49:47.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:47.386+0000 7f8ac9705700 1 -- 172.21.15.28:0/2364608290 wait complete. 2024-04-21T08:49:48.426 INFO:teuthology.orchestra.run.smithi028.stdout:163208757303 2024-04-21T08:49:48.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-21T08:49:48.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:48 smithi028 ceph-mon[28003]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 290 B/s wr, 6 op/s; 65 B/s, 1 objects/s recovering 2024-04-21T08:49:48.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:49:48.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:49:48.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:49:48.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:48 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:49:48.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:48 smithi152 ceph-mon[31401]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 290 B/s wr, 6 op/s; 65 B/s, 1 objects/s recovering 2024-04-21T08:49:48.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:48 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:49:48.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:48 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:49:48.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:48 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:49:48.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:48 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:49:48.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:48 smithi171 ceph-mon[31358]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 290 B/s wr, 6 op/s; 65 B/s, 1 objects/s recovering 2024-04-21T08:49:48.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:48 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:49:48.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:48 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:49:48.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:48 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:49:48.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:48 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:49:49.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.076+0000 7fdc276d9700 1 -- 172.21.15.28:0/3494364443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc20101ff0 msgr2=0x7fdc20105db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.076+0000 7fdc276d9700 1 --2- 172.21.15.28:0/3494364443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc20101ff0 0x7fdc20105db0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000aaa0 tx=0x7fdc1000adb0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.077+0000 7fdc276d9700 1 -- 172.21.15.28:0/3494364443 shutdown_connections 2024-04-21T08:49:49.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.077+0000 7fdc276d9700 1 --2- 172.21.15.28:0/3494364443 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc201064e0 0x7fdc2010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.077+0000 7fdc276d9700 1 --2- 172.21.15.28:0/3494364443 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc20101640 0x7fdc20101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.077+0000 7fdc276d9700 1 --2- 172.21.15.28:0/3494364443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc20101ff0 0x7fdc20105db0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.077+0000 7fdc276d9700 1 -- 172.21.15.28:0/3494364443 >> 172.21.15.28:0/3494364443 conn(0x7fdc20076c60 msgr2=0x7fdc20077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.077+0000 7fdc276d9700 1 -- 172.21.15.28:0/3494364443 shutdown_connections 2024-04-21T08:49:49.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.077+0000 7fdc276d9700 1 -- 172.21.15.28:0/3494364443 wait complete. 2024-04-21T08:49:49.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.077+0000 7fdc276d9700 1 Processor -- start 2024-04-21T08:49:49.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.077+0000 7fdc276d9700 1 -- start start 2024-04-21T08:49:49.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.077+0000 7fdc276d9700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc20101640 0x7fdc20199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.077+0000 7fdc276d9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc20101ff0 0x7fdc20199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.077+0000 7fdc276d9700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc201064e0 0x7fdc2019d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc276d9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc20110a90 con 0x7fdc20101ff0 2024-04-21T08:49:49.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc276d9700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fdc20110910 con 0x7fdc201064e0 2024-04-21T08:49:49.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc276d9700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdc20110c10 con 0x7fdc20101640 2024-04-21T08:49:49.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc24c74700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc20101ff0 0x7fdc20199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc24c74700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc20101ff0 0x7fdc20199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36660/0 (socket says 172.21.15.28:36660) 2024-04-21T08:49:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc24c74700 1 -- 172.21.15.28:0/3335185687 learned_addr learned my addr 172.21.15.28:0/3335185687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc25475700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc20101640 0x7fdc20199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc25c76700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc201064e0 0x7fdc2019d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc24c74700 1 -- 172.21.15.28:0/3335185687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc20101640 msgr2=0x7fdc20199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc24c74700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc20101640 0x7fdc20199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc24c74700 1 -- 172.21.15.28:0/3335185687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc201064e0 msgr2=0x7fdc2019d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc24c74700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc201064e0 0x7fdc2019d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc24c74700 1 -- 172.21.15.28:0/3335185687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2019e130 con 0x7fdc20101ff0 2024-04-21T08:49:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc25475700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc20101640 0x7fdc20199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc25c76700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdc201064e0 0x7fdc2019d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc24c74700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc20101ff0 0x7fdc20199660 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdc100059d0 tx=0x7fdc100039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc167fc700 1 -- 172.21.15.28:0/3335185687 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10023020 con 0x7fdc20101ff0 2024-04-21T08:49:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc276d9700 1 -- 172.21.15.28:0/3335185687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc2019e3c0 con 0x7fdc20101ff0 2024-04-21T08:49:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc167fc700 1 -- 172.21.15.28:0/3335185687 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc1000e450 con 0x7fdc20101ff0 2024-04-21T08:49:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc167fc700 1 -- 172.21.15.28:0/3335185687 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10016660 con 0x7fdc20101ff0 2024-04-21T08:49:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.078+0000 7fdc276d9700 1 -- 172.21.15.28:0/3335185687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc201a5ca0 con 0x7fdc20101ff0 2024-04-21T08:49:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.079+0000 7fdc276d9700 1 -- 172.21.15.28:0/3335185687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdc200631d0 con 0x7fdc20101ff0 2024-04-21T08:49:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.081+0000 7fdc167fc700 1 -- 172.21.15.28:0/3335185687 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdc10026020 con 0x7fdc20101ff0 2024-04-21T08:49:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.081+0000 7fdc167fc700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fdc0c0a3cd0 0x7fdc0c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.081+0000 7fdc167fc700 1 -- 172.21.15.28:0/3335185687 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fdc100d2250 con 0x7fdc20101ff0 2024-04-21T08:49:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.081+0000 7fdc25475700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fdc0c0a3cd0 0x7fdc0c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.081+0000 7fdc167fc700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648] conn(0x7fdc0c0aedb0 0x7fdc0c0b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.081+0000 7fdc167fc700 1 -- 172.21.15.28:0/3335185687 --> [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc1000c1c0 con 0x7fdc0c0aedb0 2024-04-21T08:49:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.081+0000 7fdc167fc700 1 -- 172.21.15.28:0/3335185687 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fdc100d26a0 con 0x7fdc20101ff0 2024-04-21T08:49:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.082+0000 7fdc25c76700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648] conn(0x7fdc0c0aedb0 0x7fdc0c0b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.082+0000 7fdc25475700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fdc0c0a3cd0 0x7fdc0c0a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c0093a0 tx=0x7fdc1c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.082+0000 7fdc25c76700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648] conn(0x7fdc0c0aedb0 0x7fdc0c0b1250 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.083+0000 7fdc167fc700 1 -- 172.21.15.28:0/3335185687 <== osd.0 v2:172.21.15.28:6802/855251648 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc1000c1c0 con 0x7fdc0c0aedb0 2024-04-21T08:49:49.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.099+0000 7fdc276d9700 1 -- 172.21.15.28:0/3335185687 --> [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdc200635d0 con 0x7fdc0c0aedb0 2024-04-21T08:49:49.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.100+0000 7fdc167fc700 1 -- 172.21.15.28:0/3335185687 <== osd.0 v2:172.21.15.28:6802/855251648 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdc200635d0 con 0x7fdc0c0aedb0 2024-04-21T08:49:49.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.100+0000 7fdc276d9700 1 -- 172.21.15.28:0/3335185687 >> [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648] conn(0x7fdc0c0aedb0 msgr2=0x7fdc0c0b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.100+0000 7fdc276d9700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648] conn(0x7fdc0c0aedb0 0x7fdc0c0b1250 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.100+0000 7fdc276d9700 1 -- 172.21.15.28:0/3335185687 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fdc0c0a3cd0 msgr2=0x7fdc0c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.100+0000 7fdc276d9700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fdc0c0a3cd0 0x7fdc0c0a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c0093a0 tx=0x7fdc1c00b040 comp rx=0 tx=0).stop 2024-04-21T08:49:49.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.100+0000 7fdc276d9700 1 -- 172.21.15.28:0/3335185687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc20101ff0 msgr2=0x7fdc20199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.100+0000 7fdc276d9700 1 --2- 172.21.15.28:0/3335185687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc20101ff0 0x7fdc20199660 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdc100059d0 tx=0x7fdc100039e0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.100+0000 7fdc25c76700 1 -- 172.21.15.28:0/3335185687 reap_dead start 2024-04-21T08:49:49.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.100+0000 7fdc276d9700 1 -- 172.21.15.28:0/3335185687 shutdown_connections 2024-04-21T08:49:49.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.100+0000 7fdc276d9700 1 -- 172.21.15.28:0/3335185687 >> 172.21.15.28:0/3335185687 conn(0x7fdc20076c60 msgr2=0x7fdc20105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.100+0000 7fdc276d9700 1 -- 172.21.15.28:0/3335185687 shutdown_connections 2024-04-21T08:49:49.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.101+0000 7fdc276d9700 1 -- 172.21.15.28:0/3335185687 wait complete. 2024-04-21T08:49:49.742 INFO:teuthology.orchestra.run.smithi028.stdout:219043332142 2024-04-21T08:49:49.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-21T08:49:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.918+0000 7f65bf0ed700 1 -- 172.21.15.28:0/3922424195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b8102330 msgr2=0x7f65b810e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.918+0000 7f65bf0ed700 1 --2- 172.21.15.28:0/3922424195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b8102330 0x7f65b810e7f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f65b000b0a0 tx=0x7f65b000b3b0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.918+0000 7f65bf0ed700 1 -- 172.21.15.28:0/3922424195 shutdown_connections 2024-04-21T08:49:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.918+0000 7f65bf0ed700 1 --2- 172.21.15.28:0/3922424195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b8102330 0x7f65b810e7f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.918+0000 7f65bf0ed700 1 --2- 172.21.15.28:0/3922424195 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65b8107920 0x7f65b8107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.918+0000 7f65bf0ed700 1 --2- 172.21.15.28:0/3922424195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f65b8101970 0x7f65b8101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.918+0000 7f65bf0ed700 1 -- 172.21.15.28:0/3922424195 >> 172.21.15.28:0/3922424195 conn(0x7f65b8078910 msgr2=0x7f65b8078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.918+0000 7f65bf0ed700 1 -- 172.21.15.28:0/3922424195 shutdown_connections 2024-04-21T08:49:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.918+0000 7f65bf0ed700 1 -- 172.21.15.28:0/3922424195 wait complete. 2024-04-21T08:49:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.919+0000 7f65bf0ed700 1 Processor -- start 2024-04-21T08:49:49.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.919+0000 7f65bf0ed700 1 -- start start 2024-04-21T08:49:49.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.919+0000 7f65bf0ed700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b8101970 0x7f65b8199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.919+0000 7f65bce89700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b8101970 0x7f65b8199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.919+0000 7f65bf0ed700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65b8102330 0x7f65b819a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.919+0000 7f65bce89700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b8101970 0x7f65b8199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36680/0 (socket says 172.21.15.28:36680) 2024-04-21T08:49:49.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.919+0000 7f65bf0ed700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f65b8107920 0x7f65b81a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.919+0000 7f65bf0ed700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65b8111bb0 con 0x7f65b8101970 2024-04-21T08:49:49.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65bf0ed700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f65b8111a30 con 0x7f65b8107920 2024-04-21T08:49:49.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65bf0ed700 1 -- 172.21.15.28:0/2290643162 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f65b8111d30 con 0x7f65b8102330 2024-04-21T08:49:49.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65bce89700 1 -- 172.21.15.28:0/2290643162 learned_addr learned my addr 172.21.15.28:0/2290643162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:49.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65bd68a700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f65b8107920 0x7f65b81a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65bce89700 1 -- 172.21.15.28:0/2290643162 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65b8102330 msgr2=0x7f65b819a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65b7fff700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65b8102330 0x7f65b819a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65bce89700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65b8102330 0x7f65b819a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65bce89700 1 -- 172.21.15.28:0/2290643162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f65b8107920 msgr2=0x7f65b81a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65bce89700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f65b8107920 0x7f65b81a4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65bce89700 1 -- 172.21.15.28:0/2290643162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b81a5510 con 0x7f65b8101970 2024-04-21T08:49:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65bd68a700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f65b8107920 0x7f65b81a4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65b7fff700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65b8102330 0x7f65b819a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65bce89700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b8101970 0x7f65b8199f50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00d950 tx=0x7f65ac00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65b5ffb700 1 -- 172.21.15.28:0/2290643162 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac016070 con 0x7f65b8101970 2024-04-21T08:49:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65bf0ed700 1 -- 172.21.15.28:0/2290643162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b81a5740 con 0x7f65b8101970 2024-04-21T08:49:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.920+0000 7f65b5ffb700 1 -- 172.21.15.28:0/2290643162 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65ac010460 con 0x7f65b8101970 2024-04-21T08:49:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.921+0000 7f65b5ffb700 1 -- 172.21.15.28:0/2290643162 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac00f620 con 0x7f65b8101970 2024-04-21T08:49:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.921+0000 7f65bf0ed700 1 -- 172.21.15.28:0/2290643162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b81a5ca0 con 0x7f65b8101970 2024-04-21T08:49:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.922+0000 7f65bf0ed700 1 -- 172.21.15.28:0/2290643162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f65b8063150 con 0x7f65b8101970 2024-04-21T08:49:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.923+0000 7f65b5ffb700 1 -- 172.21.15.28:0/2290643162 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f65ac019070 con 0x7f65b8101970 2024-04-21T08:49:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.923+0000 7f65b5ffb700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f65a80a3d50 0x7f65a80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.924+0000 7f65b7fff700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f65a80a3d50 0x7f65a80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.924+0000 7f65b5ffb700 1 -- 172.21.15.28:0/2290643162 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f65ac0d0810 con 0x7f65b8101970 2024-04-21T08:49:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.924+0000 7f65b5ffb700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417] conn(0x7f65a80aee30 0x7f65a80b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.924+0000 7f65b5ffb700 1 -- 172.21.15.28:0/2290643162 --> [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f65a80b19a0 con 0x7f65a80aee30 2024-04-21T08:49:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.924+0000 7f65b5ffb700 1 -- 172.21.15.28:0/2290643162 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f65ac0d7050 con 0x7f65b8101970 2024-04-21T08:49:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.924+0000 7f65bd68a700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417] conn(0x7f65a80aee30 0x7f65a80b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.925+0000 7f65b7fff700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f65a80a3d50 0x7f65a80a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f65a4005d90 tx=0x7f65a4005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.925+0000 7f65bd68a700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417] conn(0x7f65a80aee30 0x7f65a80b12d0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.926+0000 7f65b5ffb700 1 -- 172.21.15.28:0/2290643162 <== osd.11 v2:172.21.15.171:6824/2148030417 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f65a80b19a0 con 0x7f65a80aee30 2024-04-21T08:49:49.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.943+0000 7f65bf0ed700 1 -- 172.21.15.28:0/2290643162 --> [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f65b8063600 con 0x7f65a80aee30 2024-04-21T08:49:49.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.943+0000 7f65b5ffb700 1 -- 172.21.15.28:0/2290643162 <== osd.11 v2:172.21.15.171:6824/2148030417 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f65b8063600 con 0x7f65a80aee30 2024-04-21T08:49:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.944+0000 7f65bf0ed700 1 -- 172.21.15.28:0/2290643162 >> [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417] conn(0x7f65a80aee30 msgr2=0x7f65a80b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.944+0000 7f65bf0ed700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417] conn(0x7f65a80aee30 0x7f65a80b12d0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.944+0000 7f65bf0ed700 1 -- 172.21.15.28:0/2290643162 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f65a80a3d50 msgr2=0x7f65a80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.944+0000 7f65bf0ed700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f65a80a3d50 0x7f65a80a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f65a4005d90 tx=0x7f65a4005d00 comp rx=0 tx=0).stop 2024-04-21T08:49:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.944+0000 7f65bf0ed700 1 -- 172.21.15.28:0/2290643162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b8101970 msgr2=0x7f65b8199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.944+0000 7f65bf0ed700 1 --2- 172.21.15.28:0/2290643162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65b8101970 0x7f65b8199f50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f65ac00d950 tx=0x7f65ac00dd10 comp rx=0 tx=0).stop 2024-04-21T08:49:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.944+0000 7f65bd68a700 1 -- 172.21.15.28:0/2290643162 reap_dead start 2024-04-21T08:49:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.944+0000 7f65bf0ed700 1 -- 172.21.15.28:0/2290643162 shutdown_connections 2024-04-21T08:49:49.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.944+0000 7f65bf0ed700 1 -- 172.21.15.28:0/2290643162 >> 172.21.15.28:0/2290643162 conn(0x7f65b8078910 msgr2=0x7f65b80ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.944+0000 7f65bf0ed700 1 -- 172.21.15.28:0/2290643162 shutdown_connections 2024-04-21T08:49:49.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:49.944+0000 7f65bf0ed700 1 -- 172.21.15.28:0/2290643162 wait complete. 2024-04-21T08:49:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.123+0000 7f3b07f02700 1 -- 172.21.15.28:0/4052854786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b00102ac0 msgr2=0x7f3b00102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.123+0000 7f3b07f02700 1 --2- 172.21.15.28:0/4052854786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b00102ac0 0x7f3b00102f40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009990 tx=0x7f3afc009ca0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.123+0000 7f3b07f02700 1 -- 172.21.15.28:0/4052854786 shutdown_connections 2024-04-21T08:49:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.123+0000 7f3b07f02700 1 --2- 172.21.15.28:0/4052854786 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3b00103480 0x7f3b0010f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.123+0000 7f3b07f02700 1 --2- 172.21.15.28:0/4052854786 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b00108ab0 0x7f3b00108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.123+0000 7f3b07f02700 1 --2- 172.21.15.28:0/4052854786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b00102ac0 0x7f3b00102f40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.123+0000 7f3b07f02700 1 -- 172.21.15.28:0/4052854786 >> 172.21.15.28:0/4052854786 conn(0x7f3b00079bf0 msgr2=0x7f3b0007a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.123+0000 7f3b07f02700 1 -- 172.21.15.28:0/4052854786 shutdown_connections 2024-04-21T08:49:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.123+0000 7f3b07f02700 1 -- 172.21.15.28:0/4052854786 wait complete. 2024-04-21T08:49:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.124+0000 7f3b07f02700 1 Processor -- start 2024-04-21T08:49:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.124+0000 7f3b07f02700 1 -- start start 2024-04-21T08:49:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.124+0000 7f3b07f02700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b00102ac0 0x7f3b00199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.124+0000 7f3b07f02700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3b00103480 0x7f3b001996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.124+0000 7f3b07f02700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b00108ab0 0x7f3b0019da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.124+0000 7f3b07f02700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b00112cf0 con 0x7f3b00102ac0 2024-04-21T08:49:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.124+0000 7f3b07f02700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3b00112b70 con 0x7f3b00108ab0 2024-04-21T08:49:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.124+0000 7f3b07f02700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3b00112e70 con 0x7f3b00103480 2024-04-21T08:49:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.124+0000 7f3b05c9e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b00102ac0 0x7f3b00199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.124+0000 7f3b05c9e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b00102ac0 0x7f3b00199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36694/0 (socket says 172.21.15.28:36694) 2024-04-21T08:49:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.124+0000 7f3b05c9e700 1 -- 172.21.15.28:0/2830273017 learned_addr learned my addr 172.21.15.28:0/2830273017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.124+0000 7f3b0549d700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3b00103480 0x7f3b001996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.125+0000 7f3b0649f700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b00108ab0 0x7f3b0019da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.125+0000 7f3b05c9e700 1 -- 172.21.15.28:0/2830273017 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3b00103480 msgr2=0x7f3b001996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.125+0000 7f3b05c9e700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3b00103480 0x7f3b001996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.125+0000 7f3b05c9e700 1 -- 172.21.15.28:0/2830273017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b00108ab0 msgr2=0x7f3b0019da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.125+0000 7f3b05c9e700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b00108ab0 0x7f3b0019da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.125+0000 7f3b05c9e700 1 -- 172.21.15.28:0/2830273017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b0019e190 con 0x7f3b00102ac0 2024-04-21T08:49:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.125+0000 7f3b0549d700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3b00103480 0x7f3b001996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.125+0000 7f3b0649f700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b00108ab0 0x7f3b0019da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.125+0000 7f3b05c9e700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b00102ac0 0x7f3b00199180 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3af000d950 tx=0x7f3af000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.125+0000 7f3af6ffd700 1 -- 172.21.15.28:0/2830273017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af0016070 con 0x7f3b00102ac0 2024-04-21T08:49:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.125+0000 7f3af6ffd700 1 -- 172.21.15.28:0/2830273017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3af0010460 con 0x7f3b00102ac0 2024-04-21T08:49:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.125+0000 7f3af6ffd700 1 -- 172.21.15.28:0/2830273017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af000f620 con 0x7f3b00102ac0 2024-04-21T08:49:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.125+0000 7f3b07f02700 1 -- 172.21.15.28:0/2830273017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b0019e480 con 0x7f3b00102ac0 2024-04-21T08:49:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.125+0000 7f3b07f02700 1 -- 172.21.15.28:0/2830273017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b001a5d00 con 0x7f3b00102ac0 2024-04-21T08:49:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.126+0000 7f3b07f02700 1 -- 172.21.15.28:0/2830273017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3b00063150 con 0x7f3b00102ac0 2024-04-21T08:49:50.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.128+0000 7f3af6ffd700 1 -- 172.21.15.28:0/2830273017 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3af0019070 con 0x7f3b00102ac0 2024-04-21T08:49:50.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.128+0000 7f3af6ffd700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3aec0a3cd0 0x7f3aec0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.128+0000 7f3af6ffd700 1 -- 172.21.15.28:0/2830273017 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f3af00d0a50 con 0x7f3b00102ac0 2024-04-21T08:49:50.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.128+0000 7f3b0549d700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3aec0a3cd0 0x7f3aec0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.128+0000 7f3af6ffd700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203] conn(0x7f3aec0aedb0 0x7f3aec0b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.128+0000 7f3af6ffd700 1 -- 172.21.15.28:0/2830273017 --> [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3af0013ec0 con 0x7f3aec0aedb0 2024-04-21T08:49:50.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.128+0000 7f3af6ffd700 1 -- 172.21.15.28:0/2830273017 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f3af00d0ea0 con 0x7f3b00102ac0 2024-04-21T08:49:50.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.129+0000 7f3b0549d700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3aec0a3cd0 0x7f3aec0a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3afc006010 tx=0x7f3afc00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:50.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.129+0000 7f3b0649f700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203] conn(0x7f3aec0aedb0 0x7f3aec0b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.129+0000 7f3b0649f700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203] conn(0x7f3aec0aedb0 0x7f3aec0b1250 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:50.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.130+0000 7f3af6ffd700 1 -- 172.21.15.28:0/2830273017 <== osd.10 v2:172.21.15.152:6824/534177203 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3af0013ec0 con 0x7f3aec0aedb0 2024-04-21T08:49:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.147+0000 7f3b07f02700 1 -- 172.21.15.28:0/2830273017 --> [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3b000635d0 con 0x7f3aec0aedb0 2024-04-21T08:49:50.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.147+0000 7f3af6ffd700 1 -- 172.21.15.28:0/2830273017 <== osd.10 v2:172.21.15.152:6824/534177203 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3b000635d0 con 0x7f3aec0aedb0 2024-04-21T08:49:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.148+0000 7f3b07f02700 1 -- 172.21.15.28:0/2830273017 >> [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203] conn(0x7f3aec0aedb0 msgr2=0x7f3aec0b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.148+0000 7f3b07f02700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203] conn(0x7f3aec0aedb0 0x7f3aec0b1250 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.148+0000 7f3b07f02700 1 -- 172.21.15.28:0/2830273017 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3aec0a3cd0 msgr2=0x7f3aec0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.148+0000 7f3b07f02700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3aec0a3cd0 0x7f3aec0a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3afc006010 tx=0x7f3afc00b540 comp rx=0 tx=0).stop 2024-04-21T08:49:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.148+0000 7f3b07f02700 1 -- 172.21.15.28:0/2830273017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b00102ac0 msgr2=0x7f3b00199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.148+0000 7f3b07f02700 1 --2- 172.21.15.28:0/2830273017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b00102ac0 0x7f3b00199180 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3af000d950 tx=0x7f3af000dd10 comp rx=0 tx=0).stop 2024-04-21T08:49:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.148+0000 7f3b0649f700 1 -- 172.21.15.28:0/2830273017 reap_dead start 2024-04-21T08:49:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.148+0000 7f3b07f02700 1 -- 172.21.15.28:0/2830273017 shutdown_connections 2024-04-21T08:49:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.148+0000 7f3b07f02700 1 -- 172.21.15.28:0/2830273017 >> 172.21.15.28:0/2830273017 conn(0x7f3b00079bf0 msgr2=0x7f3b00100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.148+0000 7f3b07f02700 1 -- 172.21.15.28:0/2830273017 shutdown_connections 2024-04-21T08:49:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:50.148+0000 7f3b07f02700 1 -- 172.21.15.28:0/2830273017 wait complete. 2024-04-21T08:49:50.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:50 smithi028 ceph-mon[28003]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s; 57 B/s, 0 objects/s recovering 2024-04-21T08:49:50.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:50 smithi152 ceph-mon[31401]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s; 57 B/s, 0 objects/s recovering 2024-04-21T08:49:50.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:50 smithi171 ceph-mon[31358]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s; 57 B/s, 0 objects/s recovering 2024-04-21T08:49:51.055 INFO:teuthology.orchestra.run.smithi028.stdout:343597383704 2024-04-21T08:49:51.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-21T08:49:51.691 INFO:teuthology.orchestra.run.smithi028.stdout:34359738450 2024-04-21T08:49:51.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-21T08:49:52.429 INFO:teuthology.orchestra.run.smithi028.stdout:309237645343 2024-04-21T08:49:52.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-21T08:49:52.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:52 smithi152 ceph-mon[31401]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-21T08:49:52.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:52 smithi028 ceph-mon[28003]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-21T08:49:52.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:52 smithi171 ceph-mon[31358]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s; 0 B/s, 0 objects/s recovering 2024-04-21T08:49:52.940 INFO:teuthology.orchestra.run.smithi028.stdout:197568495668 2024-04-21T08:49:52.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-21T08:49:53.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.906+0000 7f1f2a351700 1 -- 172.21.15.28:0/2241979133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f24101930 msgr2=0x7f1f24101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:53.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.906+0000 7f1f2a351700 1 --2- 172.21.15.28:0/2241979133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f24101930 0x7f1f24101db0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1f18009990 tx=0x7f1f18009ca0 comp rx=0 tx=0).stop 2024-04-21T08:49:53.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.906+0000 7f1f2a351700 1 -- 172.21.15.28:0/2241979133 shutdown_connections 2024-04-21T08:49:53.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.906+0000 7f1f2a351700 1 --2- 172.21.15.28:0/2241979133 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f241022f0 0x7f1f2410e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:53.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.907+0000 7f1f2a351700 1 --2- 172.21.15.28:0/2241979133 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f24107980 0x7f1f24107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:53.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.907+0000 7f1f2a351700 1 --2- 172.21.15.28:0/2241979133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f24101930 0x7f1f24101db0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:53.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.907+0000 7f1f2a351700 1 -- 172.21.15.28:0/2241979133 >> 172.21.15.28:0/2241979133 conn(0x7f1f240788b0 msgr2=0x7f1f24078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:53.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.907+0000 7f1f2a351700 1 -- 172.21.15.28:0/2241979133 shutdown_connections 2024-04-21T08:49:53.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.907+0000 7f1f2a351700 1 -- 172.21.15.28:0/2241979133 wait complete. 2024-04-21T08:49:53.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.907+0000 7f1f2a351700 1 Processor -- start 2024-04-21T08:49:53.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.907+0000 7f1f2a351700 1 -- start start 2024-04-21T08:49:53.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.907+0000 7f1f2a351700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f24101930 0x7f1f241991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f2a351700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f241022f0 0x7f1f241996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f2a351700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f24107980 0x7f1f2419da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f2a351700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f24111ae0 con 0x7f1f24101930 2024-04-21T08:49:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f2a351700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1f24111960 con 0x7f1f241022f0 2024-04-21T08:49:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f2a351700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1f24111c60 con 0x7f1f24107980 2024-04-21T08:49:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f23fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f24101930 0x7f1f241991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f23fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f24101930 0x7f1f241991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36716/0 (socket says 172.21.15.28:36716) 2024-04-21T08:49:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f23fff700 1 -- 172.21.15.28:0/1789600588 learned_addr learned my addr 172.21.15.28:0/1789600588 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f288ee700 1 --2- 172.21.15.28:0/1789600588 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f24107980 0x7f1f2419da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f237fe700 1 --2- 172.21.15.28:0/1789600588 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f241022f0 0x7f1f241996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f23fff700 1 -- 172.21.15.28:0/1789600588 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f24107980 msgr2=0x7f1f2419da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f23fff700 1 --2- 172.21.15.28:0/1789600588 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f24107980 0x7f1f2419da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f23fff700 1 -- 172.21.15.28:0/1789600588 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f241022f0 msgr2=0x7f1f241996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f23fff700 1 --2- 172.21.15.28:0/1789600588 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f241022f0 0x7f1f241996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:53.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f23fff700 1 -- 172.21.15.28:0/1789600588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f2419e1b0 con 0x7f1f24101930 2024-04-21T08:49:53.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f237fe700 1 --2- 172.21.15.28:0/1789600588 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f241022f0 0x7f1f241996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:53.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f23fff700 1 --2- 172.21.15.28:0/1789600588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f24101930 0x7f1f241991a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1f1400d950 tx=0x7f1f1400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:53.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f217fa700 1 -- 172.21.15.28:0/1789600588 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f14016070 con 0x7f1f24101930 2024-04-21T08:49:53.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.908+0000 7f1f2a351700 1 -- 172.21.15.28:0/1789600588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f2419e4a0 con 0x7f1f24101930 2024-04-21T08:49:53.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.909+0000 7f1f217fa700 1 -- 172.21.15.28:0/1789600588 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f14009df0 con 0x7f1f24101930 2024-04-21T08:49:53.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.909+0000 7f1f217fa700 1 -- 172.21.15.28:0/1789600588 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1400f3c0 con 0x7f1f24101930 2024-04-21T08:49:53.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.909+0000 7f1f2a351700 1 -- 172.21.15.28:0/1789600588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f241a5d20 con 0x7f1f24101930 2024-04-21T08:49:53.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.909+0000 7f1f2a351700 1 -- 172.21.15.28:0/1789600588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1f24063150 con 0x7f1f24101930 2024-04-21T08:49:53.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.911+0000 7f1f217fa700 1 -- 172.21.15.28:0/1789600588 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1f14019070 con 0x7f1f24101930 2024-04-21T08:49:53.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.911+0000 7f1f217fa700 1 --2- 172.21.15.28:0/1789600588 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1f0c0a3cd0 0x7f1f0c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:53.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.912+0000 7f1f237fe700 1 --2- 172.21.15.28:0/1789600588 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1f0c0a3cd0 0x7f1f0c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:53.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.912+0000 7f1f217fa700 1 -- 172.21.15.28:0/1789600588 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f1f140d0960 con 0x7f1f24101930 2024-04-21T08:49:53.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.912+0000 7f1f217fa700 1 --2- 172.21.15.28:0/1789600588 >> [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197] conn(0x7f1f0c0aedb0 0x7f1f0c0b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.912+0000 7f1f217fa700 1 -- 172.21.15.28:0/1789600588 --> [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1f14013ec0 con 0x7f1f0c0aedb0 2024-04-21T08:49:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.912+0000 7f1f217fa700 1 -- 172.21.15.28:0/1789600588 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f1f140d0db0 con 0x7f1f24101930 2024-04-21T08:49:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.912+0000 7f1f288ee700 1 --2- 172.21.15.28:0/1789600588 >> [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197] conn(0x7f1f0c0aedb0 0x7f1f0c0b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.912+0000 7f1f237fe700 1 --2- 172.21.15.28:0/1789600588 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1f0c0a3cd0 0x7f1f0c0a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1f18006010 tx=0x7f1f1800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.912+0000 7f1f288ee700 1 --2- 172.21.15.28:0/1789600588 >> [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197] conn(0x7f1f0c0aedb0 0x7f1f0c0b1250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.913+0000 7f1f217fa700 1 -- 172.21.15.28:0/1789600588 <== osd.9 v2:172.21.15.28:6826/2255450197 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1f14013ec0 con 0x7f1f0c0aedb0 2024-04-21T08:49:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.929+0000 7f1f2a351700 1 -- 172.21.15.28:0/1789600588 --> [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1f24063550 con 0x7f1f0c0aedb0 2024-04-21T08:49:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.930+0000 7f1f217fa700 1 -- 172.21.15.28:0/1789600588 <== osd.9 v2:172.21.15.28:6826/2255450197 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1f24063550 con 0x7f1f0c0aedb0 2024-04-21T08:49:53.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.930+0000 7f1f2a351700 1 -- 172.21.15.28:0/1789600588 >> [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197] conn(0x7f1f0c0aedb0 msgr2=0x7f1f0c0b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:53.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.930+0000 7f1f2a351700 1 --2- 172.21.15.28:0/1789600588 >> [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197] conn(0x7f1f0c0aedb0 0x7f1f0c0b1250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:53.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.930+0000 7f1f2a351700 1 -- 172.21.15.28:0/1789600588 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1f0c0a3cd0 msgr2=0x7f1f0c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:53.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.930+0000 7f1f2a351700 1 --2- 172.21.15.28:0/1789600588 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1f0c0a3cd0 0x7f1f0c0a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1f18006010 tx=0x7f1f1800b540 comp rx=0 tx=0).stop 2024-04-21T08:49:53.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.930+0000 7f1f2a351700 1 -- 172.21.15.28:0/1789600588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f24101930 msgr2=0x7f1f241991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:53.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.930+0000 7f1f2a351700 1 --2- 172.21.15.28:0/1789600588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f24101930 0x7f1f241991a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1f1400d950 tx=0x7f1f1400dc60 comp rx=0 tx=0).stop 2024-04-21T08:49:53.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.930+0000 7f1f288ee700 1 -- 172.21.15.28:0/1789600588 reap_dead start 2024-04-21T08:49:53.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.930+0000 7f1f2a351700 1 -- 172.21.15.28:0/1789600588 shutdown_connections 2024-04-21T08:49:53.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.930+0000 7f1f2a351700 1 -- 172.21.15.28:0/1789600588 >> 172.21.15.28:0/1789600588 conn(0x7f1f240788b0 msgr2=0x7f1f2406b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:53.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.930+0000 7f1f2a351700 1 -- 172.21.15.28:0/1789600588 shutdown_connections 2024-04-21T08:49:53.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:51.930+0000 7f1f2a351700 1 -- 172.21.15.28:0/1789600588 wait complete. 2024-04-21T08:49:54.272 INFO:teuthology.orchestra.run.smithi028.stdout:287762808867 2024-04-21T08:49:54.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-21T08:49:54.602 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:54.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:54 smithi028 ceph-mon[28003]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T08:49:54.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:54 smithi152 ceph-mon[31401]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T08:49:54.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.835+0000 7fe1232cf700 1 -- 172.21.15.28:0/3952214235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe11c107940 msgr2=0x7fe11c107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:54.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.835+0000 7fe1232cf700 1 --2- 172.21.15.28:0/3952214235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe11c107940 0x7fe11c107d20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe1180099c0 tx=0x7fe118009cd0 comp rx=0 tx=0).stop 2024-04-21T08:49:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.835+0000 7fe1232cf700 1 -- 172.21.15.28:0/3952214235 shutdown_connections 2024-04-21T08:49:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.835+0000 7fe1232cf700 1 --2- 172.21.15.28:0/3952214235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe11c107940 0x7fe11c107d20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.835+0000 7fe1232cf700 1 --2- 172.21.15.28:0/3952214235 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe11c1034b0 0x7fe11c10f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.835+0000 7fe1232cf700 1 --2- 172.21.15.28:0/3952214235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe11c102af0 0x7fe11c102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.835+0000 7fe1232cf700 1 -- 172.21.15.28:0/3952214235 >> 172.21.15.28:0/3952214235 conn(0x7fe11c079ba0 msgr2=0x7fe11c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.835+0000 7fe1232cf700 1 -- 172.21.15.28:0/3952214235 shutdown_connections 2024-04-21T08:49:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.835+0000 7fe1232cf700 1 -- 172.21.15.28:0/3952214235 wait complete. 2024-04-21T08:49:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.835+0000 7fe1232cf700 1 Processor -- start 2024-04-21T08:49:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.835+0000 7fe1232cf700 1 -- start start 2024-04-21T08:49:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe1232cf700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe11c102af0 0x7fe11c19b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe1232cf700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe11c1034b0 0x7fe11c19b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe1232cf700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe11c107940 0x7fe11c19fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe1232cf700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c112cb0 con 0x7fe11c1034b0 2024-04-21T08:49:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe1232cf700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c112b30 con 0x7fe11c107940 2024-04-21T08:49:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe1232cf700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c112e30 con 0x7fe11c102af0 2024-04-21T08:49:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe12086a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe11c1034b0 0x7fe11c19b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe12086a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe11c1034b0 0x7fe11c19b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44802/0 (socket says 172.21.15.28:44802) 2024-04-21T08:49:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe12086a700 1 -- 172.21.15.28:0/1905547789 learned_addr learned my addr 172.21.15.28:0/1905547789 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe12186c700 1 --2- 172.21.15.28:0/1905547789 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe11c107940 0x7fe11c19fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe12106b700 1 --2- 172.21.15.28:0/1905547789 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe11c102af0 0x7fe11c19b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe12086a700 1 -- 172.21.15.28:0/1905547789 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe11c102af0 msgr2=0x7fe11c19b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe12086a700 1 --2- 172.21.15.28:0/1905547789 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe11c102af0 0x7fe11c19b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe12086a700 1 -- 172.21.15.28:0/1905547789 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe11c107940 msgr2=0x7fe11c19fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe12086a700 1 --2- 172.21.15.28:0/1905547789 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe11c107940 0x7fe11c19fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe12086a700 1 -- 172.21.15.28:0/1905547789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11c1a0380 con 0x7fe11c1034b0 2024-04-21T08:49:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe12186c700 1 --2- 172.21.15.28:0/1905547789 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe11c107940 0x7fe11c19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe12086a700 1 --2- 172.21.15.28:0/1905547789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe11c1034b0 0x7fe11c19b8b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00b530 tx=0x7fe10c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe1127fc700 1 -- 172.21.15.28:0/1905547789 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c015020 con 0x7fe11c1034b0 2024-04-21T08:49:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe1232cf700 1 -- 172.21.15.28:0/1905547789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11c1a0670 con 0x7fe11c1034b0 2024-04-21T08:49:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.836+0000 7fe1127fc700 1 -- 172.21.15.28:0/1905547789 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe10c011ce0 con 0x7fe11c1034b0 2024-04-21T08:49:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.837+0000 7fe1127fc700 1 -- 172.21.15.28:0/1905547789 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c010d20 con 0x7fe11c1034b0 2024-04-21T08:49:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.837+0000 7fe1232cf700 1 -- 172.21.15.28:0/1905547789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe11c1a7ef0 con 0x7fe11c1034b0 2024-04-21T08:49:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.837+0000 7fe1232cf700 1 -- 172.21.15.28:0/1905547789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe11c101380 con 0x7fe11c1034b0 2024-04-21T08:49:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.839+0000 7fe1127fc700 1 -- 172.21.15.28:0/1905547789 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe10c018020 con 0x7fe11c1034b0 2024-04-21T08:49:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.839+0000 7fe1127fc700 1 --2- 172.21.15.28:0/1905547789 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe1080a3cd0 0x7fe1080a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.840+0000 7fe12106b700 1 --2- 172.21.15.28:0/1905547789 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe1080a3cd0 0x7fe1080a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.840+0000 7fe1127fc700 1 -- 172.21.15.28:0/1905547789 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fe10c0d06d0 con 0x7fe11c1034b0 2024-04-21T08:49:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.840+0000 7fe1127fc700 1 --2- 172.21.15.28:0/1905547789 >> [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587] conn(0x7fe1080aedb0 0x7fe1080b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.840+0000 7fe1127fc700 1 -- 172.21.15.28:0/1905547789 --> [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe10c01deb0 con 0x7fe1080aedb0 2024-04-21T08:49:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.840+0000 7fe1127fc700 1 -- 172.21.15.28:0/1905547789 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fe10c0d0b20 con 0x7fe11c1034b0 2024-04-21T08:49:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.840+0000 7fe12106b700 1 --2- 172.21.15.28:0/1905547789 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe1080a3cd0 0x7fe1080a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe118009750 tx=0x7fe11801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.840+0000 7fe12186c700 1 --2- 172.21.15.28:0/1905547789 >> [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587] conn(0x7fe1080aedb0 0x7fe1080b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.840+0000 7fe12186c700 1 --2- 172.21.15.28:0/1905547789 >> [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587] conn(0x7fe1080aedb0 0x7fe1080b1250 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:54.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.841+0000 7fe1127fc700 1 -- 172.21.15.28:0/1905547789 <== osd.8 v2:172.21.15.171:6816/1592405587 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe10c01deb0 con 0x7fe1080aedb0 2024-04-21T08:49:54.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.859+0000 7fe1232cf700 1 -- 172.21.15.28:0/1905547789 --> [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe11c1041b0 con 0x7fe1080aedb0 2024-04-21T08:49:54.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.859+0000 7fe1127fc700 1 -- 172.21.15.28:0/1905547789 <== osd.8 v2:172.21.15.171:6816/1592405587 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe11c1041b0 con 0x7fe1080aedb0 2024-04-21T08:49:54.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.860+0000 7fe1232cf700 1 -- 172.21.15.28:0/1905547789 >> [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587] conn(0x7fe1080aedb0 msgr2=0x7fe1080b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:54.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.860+0000 7fe1232cf700 1 --2- 172.21.15.28:0/1905547789 >> [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587] conn(0x7fe1080aedb0 0x7fe1080b1250 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:54.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.860+0000 7fe1232cf700 1 -- 172.21.15.28:0/1905547789 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe1080a3cd0 msgr2=0x7fe1080a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:54.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.860+0000 7fe1232cf700 1 --2- 172.21.15.28:0/1905547789 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe1080a3cd0 0x7fe1080a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe118009750 tx=0x7fe11801a040 comp rx=0 tx=0).stop 2024-04-21T08:49:54.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.860+0000 7fe1232cf700 1 -- 172.21.15.28:0/1905547789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe11c1034b0 msgr2=0x7fe11c19b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:54.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.860+0000 7fe1232cf700 1 --2- 172.21.15.28:0/1905547789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe11c1034b0 0x7fe11c19b8b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00b530 tx=0x7fe10c00b8f0 comp rx=0 tx=0).stop 2024-04-21T08:49:54.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.860+0000 7fe12186c700 1 -- 172.21.15.28:0/1905547789 reap_dead start 2024-04-21T08:49:54.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.860+0000 7fe1232cf700 1 -- 172.21.15.28:0/1905547789 shutdown_connections 2024-04-21T08:49:54.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.860+0000 7fe1232cf700 1 -- 172.21.15.28:0/1905547789 >> 172.21.15.28:0/1905547789 conn(0x7fe11c079ba0 msgr2=0x7fe11c100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:54.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.860+0000 7fe1232cf700 1 -- 172.21.15.28:0/1905547789 shutdown_connections 2024-04-21T08:49:54.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:54.860+0000 7fe1232cf700 1 -- 172.21.15.28:0/1905547789 wait complete. 2024-04-21T08:49:54.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:54 smithi171 ceph-mon[31358]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-21T08:49:55.521 INFO:teuthology.orchestra.run.smithi028.stdout:253403070507 2024-04-21T08:49:55.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-21T08:49:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:56.213 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:49:56.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:56 smithi028 ceph-mon[28003]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T08:49:56.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:56 smithi152 ceph-mon[31401]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T08:49:56.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:56 smithi171 ceph-mon[31358]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T08:49:58.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:49:58 smithi028 ceph-mon[28003]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T08:49:58.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:49:58 smithi152 ceph-mon[31401]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T08:49:58.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:49:58 smithi171 ceph-mon[31358]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T08:49:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.316+0000 7f435f9c1700 1 -- 172.21.15.28:0/3119717335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4358108aa0 msgr2=0x7f4358108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.316+0000 7f435f9c1700 1 --2- 172.21.15.28:0/3119717335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4358108aa0 0x7f4358108e80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f435400b0a0 tx=0x7f435400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:49:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.316+0000 7f435f9c1700 1 -- 172.21.15.28:0/3119717335 shutdown_connections 2024-04-21T08:49:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.316+0000 7f435f9c1700 1 --2- 172.21.15.28:0/3119717335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4358108aa0 0x7f4358108e80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.316+0000 7f435f9c1700 1 --2- 172.21.15.28:0/3119717335 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4358103490 0x7f435810f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.316+0000 7f435f9c1700 1 --2- 172.21.15.28:0/3119717335 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4358102ad0 0x7f4358102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.316+0000 7f435f9c1700 1 -- 172.21.15.28:0/3119717335 >> 172.21.15.28:0/3119717335 conn(0x7f4358079c30 msgr2=0x7f435807a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.316+0000 7f435f9c1700 1 -- 172.21.15.28:0/3119717335 shutdown_connections 2024-04-21T08:49:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.316+0000 7f435f9c1700 1 -- 172.21.15.28:0/3119717335 wait complete. 2024-04-21T08:49:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.317+0000 7f435f9c1700 1 Processor -- start 2024-04-21T08:49:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.317+0000 7f435f9c1700 1 -- start start 2024-04-21T08:49:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.317+0000 7f435f9c1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4358102ad0 0x7f4358199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.317+0000 7f435f9c1700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4358103490 0x7f4358199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.317+0000 7f435f9c1700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4358108aa0 0x7f435819da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.317+0000 7f435f9c1700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4358112ca0 con 0x7f4358102ad0 2024-04-21T08:49:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.317+0000 7f435f9c1700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4358112b20 con 0x7f4358103490 2024-04-21T08:49:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.317+0000 7f435f9c1700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4358112e20 con 0x7f4358108aa0 2024-04-21T08:49:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.317+0000 7f435d75d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4358102ad0 0x7f4358199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.317+0000 7f435d75d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4358102ad0 0x7f4358199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44814/0 (socket says 172.21.15.28:44814) 2024-04-21T08:49:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.317+0000 7f435d75d700 1 -- 172.21.15.28:0/54251532 learned_addr learned my addr 172.21.15.28:0/54251532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:49:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.317+0000 7f435df5e700 1 --2- 172.21.15.28:0/54251532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4358108aa0 0x7f435819da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.317+0000 7f435cf5c700 1 --2- 172.21.15.28:0/54251532 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4358103490 0x7f4358199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.318+0000 7f435d75d700 1 -- 172.21.15.28:0/54251532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4358108aa0 msgr2=0x7f435819da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:59.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.318+0000 7f435d75d700 1 --2- 172.21.15.28:0/54251532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4358108aa0 0x7f435819da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.318+0000 7f435d75d700 1 -- 172.21.15.28:0/54251532 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4358103490 msgr2=0x7f4358199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.318+0000 7f435d75d700 1 --2- 172.21.15.28:0/54251532 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4358103490 0x7f4358199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.318+0000 7f435d75d700 1 -- 172.21.15.28:0/54251532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435819e140 con 0x7f4358102ad0 2024-04-21T08:49:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.318+0000 7f435d75d700 1 --2- 172.21.15.28:0/54251532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4358102ad0 0x7f4358199130 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f435400b070 tx=0x7f4354004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.318+0000 7f434e7fc700 1 -- 172.21.15.28:0/54251532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354021070 con 0x7f4358102ad0 2024-04-21T08:49:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.318+0000 7f435f9c1700 1 -- 172.21.15.28:0/54251532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f435819e3d0 con 0x7f4358102ad0 2024-04-21T08:49:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.318+0000 7f434e7fc700 1 -- 172.21.15.28:0/54251532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4354007e30 con 0x7f4358102ad0 2024-04-21T08:49:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.318+0000 7f434e7fc700 1 -- 172.21.15.28:0/54251532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354004760 con 0x7f4358102ad0 2024-04-21T08:49:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.318+0000 7f435f9c1700 1 -- 172.21.15.28:0/54251532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43581a5cb0 con 0x7f4358102ad0 2024-04-21T08:49:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.319+0000 7f435f9c1700 1 -- 172.21.15.28:0/54251532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f435810b0a0 con 0x7f4358102ad0 2024-04-21T08:49:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.320+0000 7f434e7fc700 1 -- 172.21.15.28:0/54251532 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4354003de0 con 0x7f4358102ad0 2024-04-21T08:49:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.321+0000 7f434e7fc700 1 --2- 172.21.15.28:0/54251532 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f43440a3cd0 0x7f43440a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.321+0000 7f435cf5c700 1 --2- 172.21.15.28:0/54251532 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f43440a3cd0 0x7f43440a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.321+0000 7f434e7fc700 1 -- 172.21.15.28:0/54251532 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f435400f020 con 0x7f4358102ad0 2024-04-21T08:49:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.321+0000 7f435cf5c700 1 --2- 172.21.15.28:0/54251532 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f43440a3cd0 0x7f43440a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4358102a30 tx=0x7f434800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:59.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.326+0000 7f434e7fc700 1 -- 172.21.15.28:0/54251532 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f435402a020 con 0x7f4358102ad0 2024-04-21T08:49:59.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.538+0000 7f435f9c1700 1 -- 172.21.15.28:0/54251532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f435819a490 con 0x7f4358102ad0 2024-04-21T08:49:59.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.538+0000 7f434e7fc700 1 -- 172.21.15.28:0/54251532 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f435409e3d0 con 0x7f4358102ad0 2024-04-21T08:49:59.538 INFO:teuthology.orchestra.run.smithi028.stdout:141733920832 2024-04-21T08:49:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.540+0000 7f435f9c1700 1 -- 172.21.15.28:0/54251532 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f43440a3cd0 msgr2=0x7f43440a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.540+0000 7f435f9c1700 1 --2- 172.21.15.28:0/54251532 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f43440a3cd0 0x7f43440a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4358102a30 tx=0x7f434800a380 comp rx=0 tx=0).stop 2024-04-21T08:49:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.540+0000 7f435f9c1700 1 -- 172.21.15.28:0/54251532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4358102ad0 msgr2=0x7f4358199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.540+0000 7f435f9c1700 1 --2- 172.21.15.28:0/54251532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4358102ad0 0x7f4358199130 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f435400b070 tx=0x7f4354004d00 comp rx=0 tx=0).stop 2024-04-21T08:49:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.540+0000 7f435f9c1700 1 -- 172.21.15.28:0/54251532 shutdown_connections 2024-04-21T08:49:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.540+0000 7f435f9c1700 1 --2- 172.21.15.28:0/54251532 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f43440a3cd0 0x7f43440a6190 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.540+0000 7f435f9c1700 1 --2- 172.21.15.28:0/54251532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4358102ad0 0x7f4358199130 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.540+0000 7f435f9c1700 1 --2- 172.21.15.28:0/54251532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4358108aa0 0x7f435819da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.540+0000 7f435f9c1700 1 --2- 172.21.15.28:0/54251532 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4358103490 0x7f4358199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.540+0000 7f435f9c1700 1 -- 172.21.15.28:0/54251532 >> 172.21.15.28:0/54251532 conn(0x7f4358079c30 msgr2=0x7f4358100aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.540+0000 7f435f9c1700 1 -- 172.21.15.28:0/54251532 shutdown_connections 2024-04-21T08:49:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:49:59.540+0000 7f435f9c1700 1 -- 172.21.15.28:0/54251532 wait complete. 2024-04-21T08:50:00.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:00 smithi152 ceph-mon[31401]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-04-21T08:50:00.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:00 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/54251532' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T08:50:00.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:00 smithi152 ceph-mon[31401]: overall HEALTH_OK 2024-04-21T08:50:00.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:00 smithi028 ceph-mon[28003]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-04-21T08:50:00.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:00 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/54251532' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T08:50:00.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:00 smithi028 ceph-mon[28003]: overall HEALTH_OK 2024-04-21T08:50:00.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:00 smithi171 ceph-mon[31358]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-04-21T08:50:00.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:00 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/54251532' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T08:50:00.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:00 smithi171 ceph-mon[31358]: overall HEALTH_OK 2024-04-21T08:50:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.992+0000 7fd0c63af700 1 -- 172.21.15.28:0/96535601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c0102a90 msgr2=0x7fd0c0102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.992+0000 7fd0c63af700 1 --2- 172.21.15.28:0/96535601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c0102a90 0x7fd0c0102f10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4009990 tx=0x7fd0b4009ca0 comp rx=0 tx=0).stop 2024-04-21T08:50:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.992+0000 7fd0c63af700 1 -- 172.21.15.28:0/96535601 shutdown_connections 2024-04-21T08:50:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.992+0000 7fd0c63af700 1 --2- 172.21.15.28:0/96535601 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0c0103450 0x7fd0c010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.992+0000 7fd0c63af700 1 --2- 172.21.15.28:0/96535601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0c0108a50 0x7fd0c0108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.992+0000 7fd0c63af700 1 --2- 172.21.15.28:0/96535601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c0102a90 0x7fd0c0102f10 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.992+0000 7fd0c63af700 1 -- 172.21.15.28:0/96535601 >> 172.21.15.28:0/96535601 conn(0x7fd0c0079ba0 msgr2=0x7fd0c0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.992+0000 7fd0c63af700 1 -- 172.21.15.28:0/96535601 shutdown_connections 2024-04-21T08:50:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.992+0000 7fd0c63af700 1 -- 172.21.15.28:0/96535601 wait complete. 2024-04-21T08:50:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.992+0000 7fd0c63af700 1 Processor -- start 2024-04-21T08:50:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.992+0000 7fd0c63af700 1 -- start start 2024-04-21T08:50:01.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0c63af700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0c0102a90 0x7fd0c0074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0c63af700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c0103450 0x7fd0c0075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0c63af700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0c0108a50 0x7fd0c00757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0c63af700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c0112c20 con 0x7fd0c0103450 2024-04-21T08:50:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0c63af700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c0112aa0 con 0x7fd0c0108a50 2024-04-21T08:50:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0c63af700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c0112da0 con 0x7fd0c0102a90 2024-04-21T08:50:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0bf7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c0103450 0x7fd0c0075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0bf7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c0103450 0x7fd0c0075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44830/0 (socket says 172.21.15.28:44830) 2024-04-21T08:50:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0bf7fe700 1 -- 172.21.15.28:0/158971746 learned_addr learned my addr 172.21.15.28:0/158971746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0bffff700 1 --2- 172.21.15.28:0/158971746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0c0102a90 0x7fd0c0074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0c494c700 1 --2- 172.21.15.28:0/158971746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0c0108a50 0x7fd0c00757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0bf7fe700 1 -- 172.21.15.28:0/158971746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0c0102a90 msgr2=0x7fd0c0074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0bf7fe700 1 --2- 172.21.15.28:0/158971746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0c0102a90 0x7fd0c0074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0bf7fe700 1 -- 172.21.15.28:0/158971746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0c0108a50 msgr2=0x7fd0c00757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0bf7fe700 1 --2- 172.21.15.28:0/158971746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0c0108a50 0x7fd0c00757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:01.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0bf7fe700 1 -- 172.21.15.28:0/158971746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0c01a7660 con 0x7fd0c0103450 2024-04-21T08:50:01.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.993+0000 7fd0c494c700 1 --2- 172.21.15.28:0/158971746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0c0108a50 0x7fd0c00757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:01.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.994+0000 7fd0bf7fe700 1 --2- 172.21.15.28:0/158971746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c0103450 0x7fd0c0075280 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4000c00 tx=0x7fd0b40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:01.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.994+0000 7fd0bd7fa700 1 -- 172.21.15.28:0/158971746 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4027070 con 0x7fd0c0103450 2024-04-21T08:50:01.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.994+0000 7fd0bd7fa700 1 -- 172.21.15.28:0/158971746 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0b4003f20 con 0x7fd0c0103450 2024-04-21T08:50:01.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.994+0000 7fd0c63af700 1 -- 172.21.15.28:0/158971746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0c01a78f0 con 0x7fd0c0103450 2024-04-21T08:50:01.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.994+0000 7fd0bd7fa700 1 -- 172.21.15.28:0/158971746 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4022cf0 con 0x7fd0c0103450 2024-04-21T08:50:01.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.994+0000 7fd0c63af700 1 -- 172.21.15.28:0/158971746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0c01a7dd0 con 0x7fd0c0103450 2024-04-21T08:50:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.995+0000 7fd0c63af700 1 -- 172.21.15.28:0/158971746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0c0069730 con 0x7fd0c0103450 2024-04-21T08:50:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.996+0000 7fd0bd7fa700 1 -- 172.21.15.28:0/158971746 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd0b402a070 con 0x7fd0c0103450 2024-04-21T08:50:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.997+0000 7fd0bd7fa700 1 --2- 172.21.15.28:0/158971746 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd0a80a3cd0 0x7fd0a80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.997+0000 7fd0bffff700 1 --2- 172.21.15.28:0/158971746 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd0a80a3cd0 0x7fd0a80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.997+0000 7fd0bd7fa700 1 -- 172.21.15.28:0/158971746 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fd0b40d16d0 con 0x7fd0c0103450 2024-04-21T08:50:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:01.997+0000 7fd0bffff700 1 --2- 172.21.15.28:0/158971746 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd0a80a3cd0 0x7fd0a80a6190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0007b60 tx=0x7fd0b00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:02.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:02.001+0000 7fd0bd7fa700 1 -- 172.21.15.28:0/158971746 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0b4013070 con 0x7fd0c0103450 2024-04-21T08:50:02.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:02.211+0000 7fd0c63af700 1 -- 172.21.15.28:0/158971746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fd0c002e140 con 0x7fd0c0103450 2024-04-21T08:50:02.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:02.211+0000 7fd0bd7fa700 1 -- 172.21.15.28:0/158971746 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fd0b409dc40 con 0x7fd0c0103450 2024-04-21T08:50:02.211 INFO:teuthology.orchestra.run.smithi028.stdout:343597383706 2024-04-21T08:50:02.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:02.213+0000 7fd0c63af700 1 -- 172.21.15.28:0/158971746 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd0a80a3cd0 msgr2=0x7fd0a80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:02.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:02.213+0000 7fd0c63af700 1 --2- 172.21.15.28:0/158971746 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd0a80a3cd0 0x7fd0a80a6190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0007b60 tx=0x7fd0b00073d0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:02.213+0000 7fd0c63af700 1 -- 172.21.15.28:0/158971746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c0103450 msgr2=0x7fd0c0075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:02.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:02.213+0000 7fd0c63af700 1 --2- 172.21.15.28:0/158971746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c0103450 0x7fd0c0075280 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4000c00 tx=0x7fd0b40047b0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:02.213+0000 7fd0c63af700 1 -- 172.21.15.28:0/158971746 shutdown_connections 2024-04-21T08:50:02.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:02.213+0000 7fd0c63af700 1 --2- 172.21.15.28:0/158971746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0c0108a50 0x7fd0c00757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:02.213+0000 7fd0c63af700 1 --2- 172.21.15.28:0/158971746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0c0102a90 0x7fd0c0074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:02.213+0000 7fd0c63af700 1 --2- 172.21.15.28:0/158971746 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fd0a80a3cd0 0x7fd0a80a6190 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:02.213+0000 7fd0c63af700 1 --2- 172.21.15.28:0/158971746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0c0103450 0x7fd0c0075280 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:02.213+0000 7fd0c63af700 1 -- 172.21.15.28:0/158971746 >> 172.21.15.28:0/158971746 conn(0x7fd0c0079ba0 msgr2=0x7fd0c0101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:02.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:02.213+0000 7fd0c63af700 1 -- 172.21.15.28:0/158971746 shutdown_connections 2024-04-21T08:50:02.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:02.213+0000 7fd0c63af700 1 -- 172.21.15.28:0/158971746 wait complete. 2024-04-21T08:50:02.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:02 smithi028 ceph-mon[28003]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-21T08:50:02.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:02 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/158971746' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T08:50:02.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:02 smithi152 ceph-mon[31401]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-21T08:50:02.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:02 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/158971746' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T08:50:02.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:02 smithi171 ceph-mon[31358]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-21T08:50:02.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:02 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/158971746' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T08:50:03.232 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920828 got 141733920832 for osd.4 2024-04-21T08:50:03.232 DEBUG:teuthology.parallel:result is None 2024-04-21T08:50:04.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:04 smithi152 ceph-mon[31401]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:04.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:04 smithi028 ceph-mon[28003]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:04.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:04 smithi171 ceph-mon[31358]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:05.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.551+0000 7f2cdfdba700 1 -- 172.21.15.28:0/1329928243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd8102b00 msgr2=0x7f2cd8102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:05.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.551+0000 7f2cdfdba700 1 --2- 172.21.15.28:0/1329928243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd8102b00 0x7f2cd8102f80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8009990 tx=0x7f2cc8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:50:05.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.551+0000 7f2cdfdba700 1 -- 172.21.15.28:0/1329928243 shutdown_connections 2024-04-21T08:50:05.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.551+0000 7f2cdfdba700 1 --2- 172.21.15.28:0/1329928243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cd81034c0 0x7f2cd810f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:05.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.551+0000 7f2cdfdba700 1 --2- 172.21.15.28:0/1329928243 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2cd8108a10 0x7f2cd8108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:05.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.551+0000 7f2cdfdba700 1 --2- 172.21.15.28:0/1329928243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd8102b00 0x7f2cd8102f80 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:05.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.551+0000 7f2cdfdba700 1 -- 172.21.15.28:0/1329928243 >> 172.21.15.28:0/1329928243 conn(0x7f2cd8079c00 msgr2=0x7f2cd807a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:05.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.551+0000 7f2cdfdba700 1 -- 172.21.15.28:0/1329928243 shutdown_connections 2024-04-21T08:50:05.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.551+0000 7f2cdfdba700 1 -- 172.21.15.28:0/1329928243 wait complete. 2024-04-21T08:50:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.552+0000 7f2cdfdba700 1 Processor -- start 2024-04-21T08:50:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.552+0000 7f2cdfdba700 1 -- start start 2024-04-21T08:50:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.552+0000 7f2cdfdba700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2cd8102b00 0x7f2cd81990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.552+0000 7f2cdfdba700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd81034c0 0x7f2cd8199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.552+0000 7f2cdfdba700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cd8108a10 0x7f2cd819d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.552+0000 7f2cdfdba700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd8112c50 con 0x7f2cd81034c0 2024-04-21T08:50:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.552+0000 7f2cdfdba700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd8112ad0 con 0x7f2cd8108a10 2024-04-21T08:50:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.552+0000 7f2cdfdba700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd8112dd0 con 0x7f2cd8102b00 2024-04-21T08:50:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.552+0000 7f2cdd355700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd81034c0 0x7f2cd8199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.552+0000 7f2cdd355700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd81034c0 0x7f2cd8199620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49350/0 (socket says 172.21.15.28:49350) 2024-04-21T08:50:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.552+0000 7f2cdd355700 1 -- 172.21.15.28:0/79626123 learned_addr learned my addr 172.21.15.28:0/79626123 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.552+0000 7f2cddb56700 1 --2- 172.21.15.28:0/79626123 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2cd8102b00 0x7f2cd81990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.553+0000 7f2cdd355700 1 -- 172.21.15.28:0/79626123 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2cd8102b00 msgr2=0x7f2cd81990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.553+0000 7f2cdd355700 1 --2- 172.21.15.28:0/79626123 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2cd8102b00 0x7f2cd81990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.553+0000 7f2cdd355700 1 -- 172.21.15.28:0/79626123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cd8108a10 msgr2=0x7f2cd819d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.553+0000 7f2cdd355700 1 --2- 172.21.15.28:0/79626123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cd8108a10 0x7f2cd819d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.553+0000 7f2cdd355700 1 -- 172.21.15.28:0/79626123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cd819e0f0 con 0x7f2cd81034c0 2024-04-21T08:50:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.553+0000 7f2cdd355700 1 --2- 172.21.15.28:0/79626123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd81034c0 0x7f2cd8199620 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8000c00 tx=0x7f2cc80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.553+0000 7f2cceffd700 1 -- 172.21.15.28:0/79626123 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8027070 con 0x7f2cd81034c0 2024-04-21T08:50:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.553+0000 7f2cdfdba700 1 -- 172.21.15.28:0/79626123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cd819e380 con 0x7f2cd81034c0 2024-04-21T08:50:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.553+0000 7f2cceffd700 1 -- 172.21.15.28:0/79626123 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cc8003f20 con 0x7f2cd81034c0 2024-04-21T08:50:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.553+0000 7f2cdfdba700 1 -- 172.21.15.28:0/79626123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cd81a5c60 con 0x7f2cd81034c0 2024-04-21T08:50:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.553+0000 7f2cceffd700 1 -- 172.21.15.28:0/79626123 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8022cf0 con 0x7f2cd81034c0 2024-04-21T08:50:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.554+0000 7f2cdfdba700 1 -- 172.21.15.28:0/79626123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cd8069730 con 0x7f2cd81034c0 2024-04-21T08:50:05.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.556+0000 7f2cceffd700 1 -- 172.21.15.28:0/79626123 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2cc802a070 con 0x7f2cd81034c0 2024-04-21T08:50:05.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.557+0000 7f2cceffd700 1 --2- 172.21.15.28:0/79626123 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2cc40a3cd0 0x7f2cc40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:05.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.557+0000 7f2cceffd700 1 -- 172.21.15.28:0/79626123 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f2cc80d18e0 con 0x7f2cd81034c0 2024-04-21T08:50:05.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.557+0000 7f2cddb56700 1 --2- 172.21.15.28:0/79626123 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2cc40a3cd0 0x7f2cc40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:05.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.558+0000 7f2cddb56700 1 --2- 172.21.15.28:0/79626123 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2cc40a3cd0 0x7f2cc40a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4005950 tx=0x7f2cd400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.561+0000 7f2cceffd700 1 -- 172.21.15.28:0/79626123 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2cc8013070 con 0x7f2cd81034c0 2024-04-21T08:50:05.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.773+0000 7f2cdfdba700 1 -- 172.21.15.28:0/79626123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f2cd802df80 con 0x7f2cd81034c0 2024-04-21T08:50:05.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.774+0000 7f2cceffd700 1 -- 172.21.15.28:0/79626123 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2cc809de50 con 0x7f2cd81034c0 2024-04-21T08:50:05.774 INFO:teuthology.orchestra.run.smithi028.stdout:107374182471 2024-04-21T08:50:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.776+0000 7f2cdfdba700 1 -- 172.21.15.28:0/79626123 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2cc40a3cd0 msgr2=0x7f2cc40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.776+0000 7f2cdfdba700 1 --2- 172.21.15.28:0/79626123 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2cc40a3cd0 0x7f2cc40a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4005950 tx=0x7f2cd400b410 comp rx=0 tx=0).stop 2024-04-21T08:50:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.776+0000 7f2cdfdba700 1 -- 172.21.15.28:0/79626123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd81034c0 msgr2=0x7f2cd8199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.776+0000 7f2cdfdba700 1 --2- 172.21.15.28:0/79626123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd81034c0 0x7f2cd8199620 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8000c00 tx=0x7f2cc80047b0 comp rx=0 tx=0).stop 2024-04-21T08:50:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.776+0000 7f2cdfdba700 1 -- 172.21.15.28:0/79626123 shutdown_connections 2024-04-21T08:50:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.776+0000 7f2cdfdba700 1 --2- 172.21.15.28:0/79626123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cd8108a10 0x7f2cd819d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.776+0000 7f2cdfdba700 1 --2- 172.21.15.28:0/79626123 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2cd8102b00 0x7f2cd81990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.776+0000 7f2cdfdba700 1 --2- 172.21.15.28:0/79626123 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2cc40a3cd0 0x7f2cc40a6190 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.776+0000 7f2cdfdba700 1 --2- 172.21.15.28:0/79626123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd81034c0 0x7f2cd8199620 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.776+0000 7f2cdfdba700 1 -- 172.21.15.28:0/79626123 >> 172.21.15.28:0/79626123 conn(0x7f2cd8079c00 msgr2=0x7f2cd8100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.776+0000 7f2cdfdba700 1 -- 172.21.15.28:0/79626123 shutdown_connections 2024-04-21T08:50:05.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:05.776+0000 7f2cdfdba700 1 -- 172.21.15.28:0/79626123 wait complete. 2024-04-21T08:50:06.612 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:06.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:06 smithi152 ceph-mon[31401]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:06.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:06 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/79626123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T08:50:06.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:06 smithi028 ceph-mon[28003]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:06.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:06 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/79626123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T08:50:06.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:06 smithi171 ceph-mon[31358]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:06.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:06 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/79626123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T08:50:08.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.520+0000 7f37638bc700 1 -- 172.21.15.28:0/3862663052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f375c103450 msgr2=0x7f375c10f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.520+0000 7f37638bc700 1 --2- 172.21.15.28:0/3862663052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f375c103450 0x7f375c10f8b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f375400b0a0 tx=0x7f375400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.520+0000 7f37638bc700 1 -- 172.21.15.28:0/3862663052 shutdown_connections 2024-04-21T08:50:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.520+0000 7f37638bc700 1 --2- 172.21.15.28:0/3862663052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f375c103450 0x7f375c10f8b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.520+0000 7f37638bc700 1 --2- 172.21.15.28:0/3862663052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f375c108a50 0x7f375c108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.520+0000 7f37638bc700 1 --2- 172.21.15.28:0/3862663052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f375c102a90 0x7f375c102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.520+0000 7f37638bc700 1 -- 172.21.15.28:0/3862663052 >> 172.21.15.28:0/3862663052 conn(0x7f375c079ba0 msgr2=0x7f375c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.521+0000 7f37638bc700 1 -- 172.21.15.28:0/3862663052 shutdown_connections 2024-04-21T08:50:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.521+0000 7f37638bc700 1 -- 172.21.15.28:0/3862663052 wait complete. 2024-04-21T08:50:08.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.521+0000 7f37638bc700 1 Processor -- start 2024-04-21T08:50:08.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.521+0000 7f37638bc700 1 -- start start 2024-04-21T08:50:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f37638bc700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f375c102a90 0x7f375c1991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f37638bc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f375c103450 0x7f375c1996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f37638bc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f375c108a50 0x7f375c19da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f37638bc700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f375c112c80 con 0x7f375c103450 2024-04-21T08:50:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f37638bc700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f375c112b00 con 0x7f375c108a50 2024-04-21T08:50:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f37638bc700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f375c112e00 con 0x7f375c102a90 2024-04-21T08:50:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f3760e57700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f375c103450 0x7f375c1996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f3761e59700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f375c108a50 0x7f375c19da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f3760e57700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f375c103450 0x7f375c1996f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49376/0 (socket says 172.21.15.28:49376) 2024-04-21T08:50:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f3761658700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f375c102a90 0x7f375c1991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f3760e57700 1 -- 172.21.15.28:0/2966227183 learned_addr learned my addr 172.21.15.28:0/2966227183 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f3760e57700 1 -- 172.21.15.28:0/2966227183 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f375c102a90 msgr2=0x7f375c1991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f3760e57700 1 --2- 172.21.15.28:0/2966227183 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f375c102a90 0x7f375c1991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f3760e57700 1 -- 172.21.15.28:0/2966227183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f375c108a50 msgr2=0x7f375c19da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f3760e57700 1 --2- 172.21.15.28:0/2966227183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f375c108a50 0x7f375c19da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f3760e57700 1 -- 172.21.15.28:0/2966227183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f375c19e1c0 con 0x7f375c103450 2024-04-21T08:50:08.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.522+0000 7f3760e57700 1 --2- 172.21.15.28:0/2966227183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f375c103450 0x7f375c1996f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f374c00b530 tx=0x7f374c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:08.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.525+0000 7f37527fc700 1 -- 172.21.15.28:0/2966227183 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c015020 con 0x7f375c103450 2024-04-21T08:50:08.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.525+0000 7f37527fc700 1 -- 172.21.15.28:0/2966227183 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f374c011ce0 con 0x7f375c103450 2024-04-21T08:50:08.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.525+0000 7f37638bc700 1 -- 172.21.15.28:0/2966227183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f375c19e4b0 con 0x7f375c103450 2024-04-21T08:50:08.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.525+0000 7f37527fc700 1 -- 172.21.15.28:0/2966227183 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c010dd0 con 0x7f375c103450 2024-04-21T08:50:08.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.525+0000 7f37638bc700 1 -- 172.21.15.28:0/2966227183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f375c1a5d30 con 0x7f375c103450 2024-04-21T08:50:08.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.526+0000 7f37638bc700 1 -- 172.21.15.28:0/2966227183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f375c069730 con 0x7f375c103450 2024-04-21T08:50:08.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.528+0000 7f37527fc700 1 -- 172.21.15.28:0/2966227183 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f374c018020 con 0x7f375c103450 2024-04-21T08:50:08.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.528+0000 7f37527fc700 1 --2- 172.21.15.28:0/2966227183 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f37480a3cd0 0x7f37480a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.529+0000 7f3761658700 1 --2- 172.21.15.28:0/2966227183 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f37480a3cd0 0x7f37480a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.529+0000 7f37527fc700 1 -- 172.21.15.28:0/2966227183 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f374c0d0490 con 0x7f375c103450 2024-04-21T08:50:08.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.529+0000 7f3761658700 1 --2- 172.21.15.28:0/2966227183 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f37480a3cd0 0x7f37480a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3758005950 tx=0x7f375800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:08.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.532+0000 7f37527fc700 1 -- 172.21.15.28:0/2966227183 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f374c01f020 con 0x7f375c103450 2024-04-21T08:50:08.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.721+0000 7f5ded0c0700 1 -- 172.21.15.28:0/2962206731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de8101ba0 msgr2=0x7f5de81076e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.721+0000 7f5ded0c0700 1 --2- 172.21.15.28:0/2962206731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de8101ba0 0x7f5de81076e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5dd00099c0 tx=0x7f5dd0009cd0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.721+0000 7f5ded0c0700 1 -- 172.21.15.28:0/2962206731 shutdown_connections 2024-04-21T08:50:08.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.721+0000 7f5ded0c0700 1 --2- 172.21.15.28:0/2962206731 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5de8107e10 0x7f5de810f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.721+0000 7f5ded0c0700 1 --2- 172.21.15.28:0/2962206731 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5de81011f0 0x7f5de81015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.721+0000 7f5ded0c0700 1 --2- 172.21.15.28:0/2962206731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de8101ba0 0x7f5de81076e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.721+0000 7f5ded0c0700 1 -- 172.21.15.28:0/2962206731 >> 172.21.15.28:0/2962206731 conn(0x7f5de8077f40 msgr2=0x7f5de80ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:08.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.721+0000 7f5ded0c0700 1 -- 172.21.15.28:0/2962206731 shutdown_connections 2024-04-21T08:50:08.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.721+0000 7f5ded0c0700 1 -- 172.21.15.28:0/2962206731 wait complete. 2024-04-21T08:50:08.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.722+0000 7f5ded0c0700 1 Processor -- start 2024-04-21T08:50:08.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.722+0000 7f5ded0c0700 1 -- start start 2024-04-21T08:50:08.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5ded0c0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5de81011f0 0x7f5de819b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5ded0c0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de8101ba0 0x7f5de819b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5ded0c0700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5de8107e10 0x7f5de819fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5ded0c0700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5de8112c10 con 0x7f5de8101ba0 2024-04-21T08:50:08.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5ded0c0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5de8112a90 con 0x7f5de81011f0 2024-04-21T08:50:08.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5ded0c0700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5de8112d90 con 0x7f5de8107e10 2024-04-21T08:50:08.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5de659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de8101ba0 0x7f5de819b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5de659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de8101ba0 0x7f5de819b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49392/0 (socket says 172.21.15.28:49392) 2024-04-21T08:50:08.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5de659c700 1 -- 172.21.15.28:0/1877631876 learned_addr learned my addr 172.21.15.28:0/1877631876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:08.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5de6d9d700 1 --2- 172.21.15.28:0/1877631876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5de81011f0 0x7f5de819b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5de759e700 1 --2- 172.21.15.28:0/1877631876 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5de8107e10 0x7f5de819fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5de659c700 1 -- 172.21.15.28:0/1877631876 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5de8107e10 msgr2=0x7f5de819fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5de659c700 1 --2- 172.21.15.28:0/1877631876 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5de8107e10 0x7f5de819fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5de659c700 1 -- 172.21.15.28:0/1877631876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5de81011f0 msgr2=0x7f5de819b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5de659c700 1 --2- 172.21.15.28:0/1877631876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5de81011f0 0x7f5de819b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5de659c700 1 -- 172.21.15.28:0/1877631876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de81a02e0 con 0x7f5de8101ba0 2024-04-21T08:50:08.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5de659c700 1 --2- 172.21.15.28:0/1877631876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de8101ba0 0x7f5de819b810 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009f50 tx=0x7f5dd00037c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:08.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.723+0000 7f5ddffff700 1 -- 172.21.15.28:0/1877631876 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0027070 con 0x7f5de8101ba0 2024-04-21T08:50:08.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.724+0000 7f5ded0c0700 1 -- 172.21.15.28:0/1877631876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de81a0570 con 0x7f5de8101ba0 2024-04-21T08:50:08.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.724+0000 7f5ddffff700 1 -- 172.21.15.28:0/1877631876 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5dd0004090 con 0x7f5de8101ba0 2024-04-21T08:50:08.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.724+0000 7f5ddffff700 1 -- 172.21.15.28:0/1877631876 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0022e60 con 0x7f5de8101ba0 2024-04-21T08:50:08.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.724+0000 7f5ded0c0700 1 -- 172.21.15.28:0/1877631876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de81a7e50 con 0x7f5de8101ba0 2024-04-21T08:50:08.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.725+0000 7f5ded0c0700 1 -- 172.21.15.28:0/1877631876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5de8069730 con 0x7f5de8101ba0 2024-04-21T08:50:08.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.726+0000 7f5ddffff700 1 -- 172.21.15.28:0/1877631876 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5dd002a070 con 0x7f5de8101ba0 2024-04-21T08:50:08.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.726+0000 7f5ddffff700 1 --2- 172.21.15.28:0/1877631876 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f5dd40a3cd0 0x7f5dd40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:08.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.726+0000 7f5de6d9d700 1 --2- 172.21.15.28:0/1877631876 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f5dd40a3cd0 0x7f5dd40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:08.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.727+0000 7f5ddffff700 1 -- 172.21.15.28:0/1877631876 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f5dd00d1700 con 0x7f5de8101ba0 2024-04-21T08:50:08.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.727+0000 7f5de6d9d700 1 --2- 172.21.15.28:0/1877631876 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f5dd40a3cd0 0x7f5dd40a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800ac50 tx=0x7f5dd800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:08.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.731+0000 7f5ddffff700 1 -- 172.21.15.28:0/1877631876 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5dd005eff0 con 0x7f5de8101ba0 2024-04-21T08:50:08.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.742+0000 7f37638bc700 1 -- 172.21.15.28:0/2966227183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f375c02df80 con 0x7f375c103450 2024-04-21T08:50:08.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.742+0000 7f37527fc700 1 -- 172.21.15.28:0/2966227183 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f374c09ca80 con 0x7f375c103450 2024-04-21T08:50:08.742 INFO:teuthology.orchestra.run.smithi028.stdout:55834574928 2024-04-21T08:50:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.744+0000 7f37638bc700 1 -- 172.21.15.28:0/2966227183 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f37480a3cd0 msgr2=0x7f37480a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.744+0000 7f37638bc700 1 --2- 172.21.15.28:0/2966227183 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f37480a3cd0 0x7f37480a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3758005950 tx=0x7f375800b410 comp rx=0 tx=0).stop 2024-04-21T08:50:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.744+0000 7f37638bc700 1 -- 172.21.15.28:0/2966227183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f375c103450 msgr2=0x7f375c1996f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.744+0000 7f37638bc700 1 --2- 172.21.15.28:0/2966227183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f375c103450 0x7f375c1996f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f374c00b530 tx=0x7f374c00b840 comp rx=0 tx=0).stop 2024-04-21T08:50:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.745+0000 7f37638bc700 1 -- 172.21.15.28:0/2966227183 shutdown_connections 2024-04-21T08:50:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.745+0000 7f37638bc700 1 --2- 172.21.15.28:0/2966227183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f375c108a50 0x7f375c19da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.745+0000 7f37638bc700 1 --2- 172.21.15.28:0/2966227183 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f375c102a90 0x7f375c1991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.745+0000 7f37638bc700 1 --2- 172.21.15.28:0/2966227183 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f37480a3cd0 0x7f37480a6190 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.745+0000 7f37638bc700 1 --2- 172.21.15.28:0/2966227183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f375c103450 0x7f375c1996f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.745+0000 7f37638bc700 1 -- 172.21.15.28:0/2966227183 >> 172.21.15.28:0/2966227183 conn(0x7f375c079ba0 msgr2=0x7f375c1018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.745+0000 7f37638bc700 1 -- 172.21.15.28:0/2966227183 shutdown_connections 2024-04-21T08:50:08.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.745+0000 7f37638bc700 1 -- 172.21.15.28:0/2966227183 wait complete. 2024-04-21T08:50:08.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:08 smithi028 ceph-mon[28003]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:50:08.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:08 smithi152 ceph-mon[31401]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:50:08.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:08 smithi171 ceph-mon[31358]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:50:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.946+0000 7f5ded0c0700 1 -- 172.21.15.28:0/1877631876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f5de81a8020 con 0x7f5de8101ba0 2024-04-21T08:50:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.947+0000 7f5ddffff700 1 -- 172.21.15.28:0/1877631876 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5dd0013070 con 0x7f5de8101ba0 2024-04-21T08:50:08.947 INFO:teuthology.orchestra.run.smithi028.stdout:77309411404 2024-04-21T08:50:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.949+0000 7f5ded0c0700 1 -- 172.21.15.28:0/1877631876 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f5dd40a3cd0 msgr2=0x7f5dd40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.949+0000 7f5ded0c0700 1 --2- 172.21.15.28:0/1877631876 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f5dd40a3cd0 0x7f5dd40a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800ac50 tx=0x7f5dd800a380 comp rx=0 tx=0).stop 2024-04-21T08:50:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.949+0000 7f5ded0c0700 1 -- 172.21.15.28:0/1877631876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de8101ba0 msgr2=0x7f5de819b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.949+0000 7f5ded0c0700 1 --2- 172.21.15.28:0/1877631876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de8101ba0 0x7f5de819b810 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009f50 tx=0x7f5dd00037c0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.949+0000 7f5ded0c0700 1 -- 172.21.15.28:0/1877631876 shutdown_connections 2024-04-21T08:50:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.949+0000 7f5ded0c0700 1 --2- 172.21.15.28:0/1877631876 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5de8107e10 0x7f5de819fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.949+0000 7f5ded0c0700 1 --2- 172.21.15.28:0/1877631876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5de81011f0 0x7f5de819b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.949+0000 7f5ded0c0700 1 --2- 172.21.15.28:0/1877631876 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f5dd40a3cd0 0x7f5dd40a6190 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.949+0000 7f5ded0c0700 1 --2- 172.21.15.28:0/1877631876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5de8101ba0 0x7f5de819b810 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.949+0000 7f5ded0c0700 1 -- 172.21.15.28:0/1877631876 >> 172.21.15.28:0/1877631876 conn(0x7f5de8077f40 msgr2=0x7f5de81065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.949+0000 7f5ded0c0700 1 -- 172.21.15.28:0/1877631876 shutdown_connections 2024-04-21T08:50:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:08.949+0000 7f5ded0c0700 1 -- 172.21.15.28:0/1877631876 wait complete. 2024-04-21T08:50:09.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:09 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2966227183' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T08:50:09.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:09 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1877631876' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T08:50:09.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:09 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2966227183' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T08:50:09.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:09 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1877631876' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T08:50:09.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:09 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2966227183' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T08:50:09.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:09 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1877631876' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T08:50:09.999 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182465 got 107374182471 for osd.3 2024-04-21T08:50:09.999 DEBUG:teuthology.parallel:result is None 2024-04-21T08:50:10.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:10 smithi028 ceph-mon[28003]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:10.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:10 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2643682872' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T08:50:10.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:10 smithi152 ceph-mon[31401]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:10.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:10 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2643682872' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T08:50:10.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:10 smithi171 ceph-mon[31358]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:10.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:10 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2643682872' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T08:50:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.296+0000 7f788487b700 1 -- 172.21.15.28:0/3322417266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7880108a20 msgr2=0x7f7880108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.296+0000 7f788487b700 1 --2- 172.21.15.28:0/3322417266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7880108a20 0x7f7880108e00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7870009990 tx=0x7f7870009ca0 comp rx=0 tx=0).stop 2024-04-21T08:50:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.297+0000 7f788487b700 1 -- 172.21.15.28:0/3322417266 shutdown_connections 2024-04-21T08:50:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.297+0000 7f788487b700 1 --2- 172.21.15.28:0/3322417266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7880108a20 0x7f7880108e00 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.297+0000 7f788487b700 1 --2- 172.21.15.28:0/3322417266 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f788006b980 0x7f788010f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.297+0000 7f788487b700 1 --2- 172.21.15.28:0/3322417266 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f788006afe0 0x7f788006b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.297+0000 7f788487b700 1 -- 172.21.15.28:0/3322417266 >> 172.21.15.28:0/3322417266 conn(0x7f78800788b0 msgr2=0x7f7880078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.297+0000 7f788487b700 1 -- 172.21.15.28:0/3322417266 shutdown_connections 2024-04-21T08:50:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.297+0000 7f788487b700 1 -- 172.21.15.28:0/3322417266 wait complete. 2024-04-21T08:50:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.298+0000 7f788487b700 1 Processor -- start 2024-04-21T08:50:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.298+0000 7f788487b700 1 -- start start 2024-04-21T08:50:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.298+0000 7f788487b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f788006afe0 0x7f7880199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.298+0000 7f788487b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f788006b980 0x7f7880199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.298+0000 7f788487b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7880108a20 0x7f788019d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.298+0000 7f788487b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7880112bd0 con 0x7f788006b980 2024-04-21T08:50:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.298+0000 7f788487b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7880112a50 con 0x7f788006afe0 2024-04-21T08:50:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.298+0000 7f788487b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7880112d50 con 0x7f7880108a20 2024-04-21T08:50:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.298+0000 7f787dd9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f788006b980 0x7f7880199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.298+0000 7f787dd9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f788006b980 0x7f7880199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49404/0 (socket says 172.21.15.28:49404) 2024-04-21T08:50:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.298+0000 7f787dd9b700 1 -- 172.21.15.28:0/2643682872 learned_addr learned my addr 172.21.15.28:0/2643682872 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.299+0000 7f787ed9d700 1 --2- 172.21.15.28:0/2643682872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7880108a20 0x7f788019d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.299+0000 7f787e59c700 1 --2- 172.21.15.28:0/2643682872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f788006afe0 0x7f7880199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:11.115 INFO:teuthology.orchestra.run.smithi028.stdout:219043332146 2024-04-21T08:50:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.299+0000 7f787dd9b700 1 -- 172.21.15.28:0/2643682872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7880108a20 msgr2=0x7f788019d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.299+0000 7f787dd9b700 1 --2- 172.21.15.28:0/2643682872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7880108a20 0x7f788019d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.299+0000 7f787dd9b700 1 -- 172.21.15.28:0/2643682872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f788006afe0 msgr2=0x7f7880199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.299+0000 7f787dd9b700 1 --2- 172.21.15.28:0/2643682872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f788006afe0 0x7f7880199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.299+0000 7f787dd9b700 1 -- 172.21.15.28:0/2643682872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788019e130 con 0x7f788006b980 2024-04-21T08:50:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.299+0000 7f787e59c700 1 --2- 172.21.15.28:0/2643682872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f788006afe0 0x7f7880199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.299+0000 7f787dd9b700 1 --2- 172.21.15.28:0/2643682872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f788006b980 0x7f7880199660 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f787400d980 tx=0x7f787400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.299+0000 7f786f7fe700 1 -- 172.21.15.28:0/2643682872 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874017070 con 0x7f788006b980 2024-04-21T08:50:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.299+0000 7f786f7fe700 1 -- 172.21.15.28:0/2643682872 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7874004830 con 0x7f788006b980 2024-04-21T08:50:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.299+0000 7f788487b700 1 -- 172.21.15.28:0/2643682872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f788019e420 con 0x7f788006b980 2024-04-21T08:50:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.299+0000 7f786f7fe700 1 -- 172.21.15.28:0/2643682872 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874005440 con 0x7f788006b980 2024-04-21T08:50:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.299+0000 7f788487b700 1 -- 172.21.15.28:0/2643682872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78801a5ca0 con 0x7f788006b980 2024-04-21T08:50:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.300+0000 7f788487b700 1 -- 172.21.15.28:0/2643682872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7880069730 con 0x7f788006b980 2024-04-21T08:50:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.301+0000 7f786f7fe700 1 -- 172.21.15.28:0/2643682872 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f78740055e0 con 0x7f788006b980 2024-04-21T08:50:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.302+0000 7f786f7fe700 1 --2- 172.21.15.28:0/2643682872 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f78680a3cd0 0x7f78680a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.302+0000 7f787e59c700 1 --2- 172.21.15.28:0/2643682872 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f78680a3cd0 0x7f78680a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.302+0000 7f786f7fe700 1 -- 172.21.15.28:0/2643682872 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f78740d0c90 con 0x7f788006b980 2024-04-21T08:50:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.303+0000 7f787e59c700 1 --2- 172.21.15.28:0/2643682872 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f78680a3cd0 0x7f78680a6190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7870006010 tx=0x7f787000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.306+0000 7f786f7fe700 1 -- 172.21.15.28:0/2643682872 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7874020020 con 0x7f788006b980 2024-04-21T08:50:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.517+0000 7f788487b700 1 -- 172.21.15.28:0/2643682872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f788019a480 con 0x7f788006b980 2024-04-21T08:50:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.518+0000 7f786f7fe700 1 -- 172.21.15.28:0/2643682872 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f787409d170 con 0x7f788006b980 2024-04-21T08:50:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.520+0000 7f788487b700 1 -- 172.21.15.28:0/2643682872 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f78680a3cd0 msgr2=0x7f78680a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.520+0000 7f788487b700 1 --2- 172.21.15.28:0/2643682872 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f78680a3cd0 0x7f78680a6190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7870006010 tx=0x7f787000b540 comp rx=0 tx=0).stop 2024-04-21T08:50:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.520+0000 7f788487b700 1 -- 172.21.15.28:0/2643682872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f788006b980 msgr2=0x7f7880199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.520+0000 7f788487b700 1 --2- 172.21.15.28:0/2643682872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f788006b980 0x7f7880199660 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f787400d980 tx=0x7f787400dd40 comp rx=0 tx=0).stop 2024-04-21T08:50:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.520+0000 7f788487b700 1 -- 172.21.15.28:0/2643682872 shutdown_connections 2024-04-21T08:50:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.520+0000 7f788487b700 1 --2- 172.21.15.28:0/2643682872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7880108a20 0x7f788019d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.520+0000 7f788487b700 1 --2- 172.21.15.28:0/2643682872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f788006afe0 0x7f7880199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.520+0000 7f788487b700 1 --2- 172.21.15.28:0/2643682872 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f78680a3cd0 0x7f78680a6190 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.520+0000 7f788487b700 1 --2- 172.21.15.28:0/2643682872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f788006b980 0x7f7880199660 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.520+0000 7f788487b700 1 -- 172.21.15.28:0/2643682872 >> 172.21.15.28:0/2643682872 conn(0x7f78800788b0 msgr2=0x7f78800ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.520+0000 7f788487b700 1 -- 172.21.15.28:0/2643682872 shutdown_connections 2024-04-21T08:50:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:10.520+0000 7f788487b700 1 -- 172.21.15.28:0/2643682872 wait complete. 2024-04-21T08:50:11.728 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332142 got 219043332146 for osd.7 2024-04-21T08:50:11.728 DEBUG:teuthology.parallel:result is None 2024-04-21T08:50:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:12.174 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383704 got 343597383706 for osd.11 2024-04-21T08:50:12.174 DEBUG:teuthology.parallel:result is None 2024-04-21T08:50:12.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:12 smithi028 ceph-mon[28003]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-21T08:50:12.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:12 smithi152 ceph-mon[31401]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-21T08:50:12.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:12 smithi171 ceph-mon[31358]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-21T08:50:13.071 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574923 got 55834574928 for osd.1 2024-04-21T08:50:13.071 DEBUG:teuthology.parallel:result is None 2024-04-21T08:50:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.249+0000 7ffa7bfd7700 1 -- 172.21.15.28:0/2197267601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa74108a60 msgr2=0x7ffa74108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.249+0000 7ffa7bfd7700 1 --2- 172.21.15.28:0/2197267601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa74108a60 0x7ffa74108e40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ffa70009990 tx=0x7ffa70009ca0 comp rx=0 tx=0).stop 2024-04-21T08:50:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.249+0000 7ffa7bfd7700 1 -- 172.21.15.28:0/2197267601 shutdown_connections 2024-04-21T08:50:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.249+0000 7ffa7bfd7700 1 --2- 172.21.15.28:0/2197267601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa74108a60 0x7ffa74108e40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.249+0000 7ffa7bfd7700 1 --2- 172.21.15.28:0/2197267601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa7406bbb0 0x7ffa7410f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.249+0000 7ffa7bfd7700 1 --2- 172.21.15.28:0/2197267601 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa7406b1f0 0x7ffa7406b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.249+0000 7ffa7bfd7700 1 -- 172.21.15.28:0/2197267601 >> 172.21.15.28:0/2197267601 conn(0x7ffa740789e0 msgr2=0x7ffa74078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.249+0000 7ffa7bfd7700 1 -- 172.21.15.28:0/2197267601 shutdown_connections 2024-04-21T08:50:13.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.249+0000 7ffa7bfd7700 1 -- 172.21.15.28:0/2197267601 wait complete. 2024-04-21T08:50:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.250+0000 7ffa7bfd7700 1 Processor -- start 2024-04-21T08:50:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.250+0000 7ffa7bfd7700 1 -- start start 2024-04-21T08:50:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.250+0000 7ffa7bfd7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7406b1f0 0x7ffa741019b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.250+0000 7ffa7bfd7700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa7406bbb0 0x7ffa74101ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.250+0000 7ffa7bfd7700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa74108a60 0x7ffa74102430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.250+0000 7ffa7bfd7700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa74112c60 con 0x7ffa7406b1f0 2024-04-21T08:50:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.250+0000 7ffa7bfd7700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ffa74112ae0 con 0x7ffa74108a60 2024-04-21T08:50:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.250+0000 7ffa7bfd7700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffa74112de0 con 0x7ffa7406bbb0 2024-04-21T08:50:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.250+0000 7ffa79d73700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7406b1f0 0x7ffa741019b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.250+0000 7ffa79d73700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7406b1f0 0x7ffa741019b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49420/0 (socket says 172.21.15.28:49420) 2024-04-21T08:50:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.250+0000 7ffa79d73700 1 -- 172.21.15.28:0/672324440 learned_addr learned my addr 172.21.15.28:0/672324440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.250+0000 7ffa79572700 1 --2- 172.21.15.28:0/672324440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa7406bbb0 0x7ffa74101ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.251+0000 7ffa79d73700 1 -- 172.21.15.28:0/672324440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa7406bbb0 msgr2=0x7ffa74101ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.251+0000 7ffa79d73700 1 --2- 172.21.15.28:0/672324440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa7406bbb0 0x7ffa74101ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.251+0000 7ffa79d73700 1 -- 172.21.15.28:0/672324440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa74108a60 msgr2=0x7ffa74102430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.251+0000 7ffa7a574700 1 --2- 172.21.15.28:0/672324440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa74108a60 0x7ffa74102430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.251+0000 7ffa79d73700 1 --2- 172.21.15.28:0/672324440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa74108a60 0x7ffa74102430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.251+0000 7ffa79d73700 1 -- 172.21.15.28:0/672324440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa741a7670 con 0x7ffa7406b1f0 2024-04-21T08:50:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.251+0000 7ffa7a574700 1 --2- 172.21.15.28:0/672324440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa74108a60 0x7ffa74102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.251+0000 7ffa79d73700 1 --2- 172.21.15.28:0/672324440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7406b1f0 0x7ffa741019b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ffa70000c00 tx=0x7ffa700047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.251+0000 7ffa6affd700 1 -- 172.21.15.28:0/672324440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa70027070 con 0x7ffa7406b1f0 2024-04-21T08:50:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.251+0000 7ffa6affd700 1 -- 172.21.15.28:0/672324440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa70003f20 con 0x7ffa7406b1f0 2024-04-21T08:50:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.251+0000 7ffa7bfd7700 1 -- 172.21.15.28:0/672324440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa741a7900 con 0x7ffa7406b1f0 2024-04-21T08:50:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.251+0000 7ffa6affd700 1 -- 172.21.15.28:0/672324440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa70022cf0 con 0x7ffa7406b1f0 2024-04-21T08:50:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.251+0000 7ffa7bfd7700 1 -- 172.21.15.28:0/672324440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa741a7db0 con 0x7ffa7406b1f0 2024-04-21T08:50:13.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.252+0000 7ffa7bfd7700 1 -- 172.21.15.28:0/672324440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa7410b060 con 0x7ffa7406b1f0 2024-04-21T08:50:13.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.253+0000 7ffa6affd700 1 -- 172.21.15.28:0/672324440 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ffa7002a070 con 0x7ffa7406b1f0 2024-04-21T08:50:13.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.254+0000 7ffa6affd700 1 --2- 172.21.15.28:0/672324440 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffa600a3d50 0x7ffa600a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:13.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.254+0000 7ffa79572700 1 --2- 172.21.15.28:0/672324440 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffa600a3d50 0x7ffa600a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:13.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.254+0000 7ffa6affd700 1 -- 172.21.15.28:0/672324440 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7ffa700d19a0 con 0x7ffa7406b1f0 2024-04-21T08:50:13.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.254+0000 7ffa79572700 1 --2- 172.21.15.28:0/672324440 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffa600a3d50 0x7ffa600a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ffa74079050 tx=0x7ffa6400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:13.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.257+0000 7ffa6affd700 1 -- 172.21.15.28:0/672324440 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa70013070 con 0x7ffa7406b1f0 2024-04-21T08:50:13.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.469+0000 7ffa7bfd7700 1 -- 172.21.15.28:0/672324440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7ffa7410aa40 con 0x7ffa7406b1f0 2024-04-21T08:50:13.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.469+0000 7ffa6affd700 1 -- 172.21.15.28:0/672324440 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ffa7009df70 con 0x7ffa7406b1f0 2024-04-21T08:50:13.469 INFO:teuthology.orchestra.run.smithi028.stdout:163208757309 2024-04-21T08:50:13.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.471+0000 7ffa7bfd7700 1 -- 172.21.15.28:0/672324440 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffa600a3d50 msgr2=0x7ffa600a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:13.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.471+0000 7ffa7bfd7700 1 --2- 172.21.15.28:0/672324440 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffa600a3d50 0x7ffa600a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ffa74079050 tx=0x7ffa6400b3c0 comp rx=0 tx=0).stop 2024-04-21T08:50:13.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.472+0000 7ffa7bfd7700 1 -- 172.21.15.28:0/672324440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7406b1f0 msgr2=0x7ffa741019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:13.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.472+0000 7ffa7bfd7700 1 --2- 172.21.15.28:0/672324440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7406b1f0 0x7ffa741019b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ffa70000c00 tx=0x7ffa700047b0 comp rx=0 tx=0).stop 2024-04-21T08:50:13.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.472+0000 7ffa7bfd7700 1 -- 172.21.15.28:0/672324440 shutdown_connections 2024-04-21T08:50:13.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.472+0000 7ffa7bfd7700 1 --2- 172.21.15.28:0/672324440 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffa600a3d50 0x7ffa600a6210 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:13.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.472+0000 7ffa7bfd7700 1 --2- 172.21.15.28:0/672324440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7406b1f0 0x7ffa741019b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:13.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.472+0000 7ffa7bfd7700 1 --2- 172.21.15.28:0/672324440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa74108a60 0x7ffa74102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:13.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.472+0000 7ffa7bfd7700 1 --2- 172.21.15.28:0/672324440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa7406bbb0 0x7ffa74101ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:13.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.472+0000 7ffa7bfd7700 1 -- 172.21.15.28:0/672324440 >> 172.21.15.28:0/672324440 conn(0x7ffa740789e0 msgr2=0x7ffa740ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:13.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.472+0000 7ffa7bfd7700 1 -- 172.21.15.28:0/672324440 shutdown_connections 2024-04-21T08:50:13.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:13.472+0000 7ffa7bfd7700 1 -- 172.21.15.28:0/672324440 wait complete. 2024-04-21T08:50:13.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:13 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/672324440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T08:50:13.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:13 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/672324440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T08:50:13.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:13 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/672324440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T08:50:14.728 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:14 smithi028 ceph-mon[28003]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:14.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:14 smithi152 ceph-mon[31401]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:14.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:14 smithi171 ceph-mon[31358]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:14.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.946+0000 7f1f70e0b700 1 -- 172.21.15.28:0/4054451986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f6c06b980 msgr2=0x7f1f6c10f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:14.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.946+0000 7f1f70e0b700 1 --2- 172.21.15.28:0/4054451986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f6c06b980 0x7f1f6c10f8f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400b0a0 tx=0x7f1f6400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:50:14.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.946+0000 7f1f70e0b700 1 -- 172.21.15.28:0/4054451986 shutdown_connections 2024-04-21T08:50:14.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.946+0000 7f1f70e0b700 1 --2- 172.21.15.28:0/4054451986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f6c06b980 0x7f1f6c10f8f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:14.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.946+0000 7f1f70e0b700 1 --2- 172.21.15.28:0/4054451986 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f6c108a20 0x7f1f6c108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:14.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.946+0000 7f1f70e0b700 1 --2- 172.21.15.28:0/4054451986 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f6c06afe0 0x7f1f6c06b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:14.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.946+0000 7f1f70e0b700 1 -- 172.21.15.28:0/4054451986 >> 172.21.15.28:0/4054451986 conn(0x7f1f6c0788b0 msgr2=0x7f1f6c078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:14.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.946+0000 7f1f70e0b700 1 -- 172.21.15.28:0/4054451986 shutdown_connections 2024-04-21T08:50:14.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.946+0000 7f1f70e0b700 1 -- 172.21.15.28:0/4054451986 wait complete. 2024-04-21T08:50:14.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.947+0000 7f1f70e0b700 1 Processor -- start 2024-04-21T08:50:14.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.947+0000 7f1f70e0b700 1 -- start start 2024-04-21T08:50:14.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.947+0000 7f1f70e0b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f6c06afe0 0x7f1f6c101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:14.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.947+0000 7f1f70e0b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f6c06b980 0x7f1f6c101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:14.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.947+0000 7f1f70e0b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f6c108a20 0x7f1f6c102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:14.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.947+0000 7f1f70e0b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6c112be0 con 0x7f1f6c108a20 2024-04-21T08:50:14.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.947+0000 7f1f70e0b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6c112a60 con 0x7f1f6c06afe0 2024-04-21T08:50:14.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.947+0000 7f1f70e0b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6c112d60 con 0x7f1f6c06b980 2024-04-21T08:50:14.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.947+0000 7f1f6ad9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f6c108a20 0x7f1f6c102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:14.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.947+0000 7f1f6ad9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f6c108a20 0x7f1f6c102480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59736/0 (socket says 172.21.15.28:59736) 2024-04-21T08:50:14.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.947+0000 7f1f6ad9d700 1 -- 172.21.15.28:0/393850787 learned_addr learned my addr 172.21.15.28:0/393850787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:14.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.947+0000 7f1f6a59c700 1 --2- 172.21.15.28:0/393850787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f6c06afe0 0x7f1f6c101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:14.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.947+0000 7f1f69d9b700 1 --2- 172.21.15.28:0/393850787 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f6c06b980 0x7f1f6c101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:14.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.948+0000 7f1f6ad9d700 1 -- 172.21.15.28:0/393850787 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f6c06b980 msgr2=0x7f1f6c101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:14.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.948+0000 7f1f6ad9d700 1 --2- 172.21.15.28:0/393850787 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f6c06b980 0x7f1f6c101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:14.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.948+0000 7f1f6ad9d700 1 -- 172.21.15.28:0/393850787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f6c06afe0 msgr2=0x7f1f6c101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:14.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.948+0000 7f1f6ad9d700 1 --2- 172.21.15.28:0/393850787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f6c06afe0 0x7f1f6c101a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:14.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.948+0000 7f1f6ad9d700 1 -- 172.21.15.28:0/393850787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f6c1a7620 con 0x7f1f6c108a20 2024-04-21T08:50:14.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.948+0000 7f1f6a59c700 1 --2- 172.21.15.28:0/393850787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f6c06afe0 0x7f1f6c101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:14.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.948+0000 7f1f6ad9d700 1 --2- 172.21.15.28:0/393850787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f6c108a20 0x7f1f6c102480 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400b830 tx=0x7f1f64004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:14.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.948+0000 7f1f637fe700 1 -- 172.21.15.28:0/393850787 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64021070 con 0x7f1f6c108a20 2024-04-21T08:50:14.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.949+0000 7f1f637fe700 1 -- 172.21.15.28:0/393850787 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f64007e30 con 0x7f1f6c108a20 2024-04-21T08:50:14.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.949+0000 7f1f637fe700 1 -- 172.21.15.28:0/393850787 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f640047d0 con 0x7f1f6c108a20 2024-04-21T08:50:14.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.949+0000 7f1f70e0b700 1 -- 172.21.15.28:0/393850787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f6c1a78b0 con 0x7f1f6c108a20 2024-04-21T08:50:14.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.949+0000 7f1f70e0b700 1 -- 172.21.15.28:0/393850787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f6c1a7d10 con 0x7f1f6c108a20 2024-04-21T08:50:14.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.950+0000 7f1f70e0b700 1 -- 172.21.15.28:0/393850787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f6c069730 con 0x7f1f6c108a20 2024-04-21T08:50:14.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.951+0000 7f1f637fe700 1 -- 172.21.15.28:0/393850787 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1f64007980 con 0x7f1f6c108a20 2024-04-21T08:50:14.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.952+0000 7f1f637fe700 1 --2- 172.21.15.28:0/393850787 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1f580a3cd0 0x7f1f580a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:14.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.952+0000 7f1f637fe700 1 -- 172.21.15.28:0/393850787 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f1f640d19d0 con 0x7f1f6c108a20 2024-04-21T08:50:14.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.952+0000 7f1f6a59c700 1 --2- 172.21.15.28:0/393850787 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1f580a3cd0 0x7f1f580a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:14.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.952+0000 7f1f6a59c700 1 --2- 172.21.15.28:0/393850787 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1f580a3cd0 0x7f1f580a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c005e10 tx=0x7f1f5c005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:14.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:14.956+0000 7f1f637fe700 1 -- 172.21.15.28:0/393850787 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f6409dfa0 con 0x7f1f6c108a20 2024-04-21T08:50:15.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:15.168+0000 7f1f70e0b700 1 -- 172.21.15.28:0/393850787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f1f6c02df80 con 0x7f1f6c108a20 2024-04-21T08:50:15.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:15.168+0000 7f1f637fe700 1 -- 172.21.15.28:0/393850787 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1f6409d8a0 con 0x7f1f6c108a20 2024-04-21T08:50:15.167 INFO:teuthology.orchestra.run.smithi028.stdout:34359738455 2024-04-21T08:50:15.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:15.169+0000 7f1f70e0b700 1 -- 172.21.15.28:0/393850787 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1f580a3cd0 msgr2=0x7f1f580a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:15.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:15.169+0000 7f1f70e0b700 1 --2- 172.21.15.28:0/393850787 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1f580a3cd0 0x7f1f580a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c005e10 tx=0x7f1f5c005d60 comp rx=0 tx=0).stop 2024-04-21T08:50:15.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:15.169+0000 7f1f70e0b700 1 -- 172.21.15.28:0/393850787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f6c108a20 msgr2=0x7f1f6c102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:15.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:15.169+0000 7f1f70e0b700 1 --2- 172.21.15.28:0/393850787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f6c108a20 0x7f1f6c102480 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400b830 tx=0x7f1f64004ce0 comp rx=0 tx=0).stop 2024-04-21T08:50:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:15.169+0000 7f1f70e0b700 1 -- 172.21.15.28:0/393850787 shutdown_connections 2024-04-21T08:50:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:15.170+0000 7f1f70e0b700 1 --2- 172.21.15.28:0/393850787 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1f580a3cd0 0x7f1f580a6190 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:15.170+0000 7f1f70e0b700 1 --2- 172.21.15.28:0/393850787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f6c108a20 0x7f1f6c102480 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:15.170+0000 7f1f70e0b700 1 --2- 172.21.15.28:0/393850787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f6c06afe0 0x7f1f6c101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:15.170+0000 7f1f70e0b700 1 --2- 172.21.15.28:0/393850787 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f6c06b980 0x7f1f6c101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:15.170+0000 7f1f70e0b700 1 -- 172.21.15.28:0/393850787 >> 172.21.15.28:0/393850787 conn(0x7f1f6c0788b0 msgr2=0x7f1f6c0ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:15.170+0000 7f1f70e0b700 1 -- 172.21.15.28:0/393850787 shutdown_connections 2024-04-21T08:50:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:15.170+0000 7f1f70e0b700 1 -- 172.21.15.28:0/393850787 wait complete. 2024-04-21T08:50:15.419 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757303 got 163208757309 for osd.5 2024-04-21T08:50:15.419 DEBUG:teuthology.parallel:result is None 2024-04-21T08:50:15.736 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:15 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/393850787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T08:50:15.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:15 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/393850787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T08:50:15.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:15 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/393850787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T08:50:16.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:16 smithi028 ceph-mon[28003]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:16.662 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411399 got 77309411404 for osd.2 2024-04-21T08:50:16.662 DEBUG:teuthology.parallel:result is None 2024-04-21T08:50:16.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:16 smithi152 ceph-mon[31401]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:16 smithi171 ceph-mon[31358]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:17.998 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738450 got 34359738455 for osd.0 2024-04-21T08:50:17.998 DEBUG:teuthology.parallel:result is None 2024-04-21T08:50:18.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.334+0000 7f1713b1b700 1 -- 172.21.15.28:0/2722554771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f170c108a50 msgr2=0x7f170c108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.334+0000 7f1713b1b700 1 --2- 172.21.15.28:0/2722554771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f170c108a50 0x7f170c108e30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1708009990 tx=0x7f1708009ca0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.334+0000 7f1713b1b700 1 -- 172.21.15.28:0/2722554771 shutdown_connections 2024-04-21T08:50:18.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.334+0000 7f1713b1b700 1 --2- 172.21.15.28:0/2722554771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f170c108a50 0x7f170c108e30 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.334+0000 7f1713b1b700 1 --2- 172.21.15.28:0/2722554771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f170c103450 0x7f170c10f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.334+0000 7f1713b1b700 1 --2- 172.21.15.28:0/2722554771 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f170c102a90 0x7f170c102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.334+0000 7f1713b1b700 1 -- 172.21.15.28:0/2722554771 >> 172.21.15.28:0/2722554771 conn(0x7f170c079ba0 msgr2=0x7f170c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.334+0000 7f1713b1b700 1 -- 172.21.15.28:0/2722554771 shutdown_connections 2024-04-21T08:50:18.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.334+0000 7f1713b1b700 1 -- 172.21.15.28:0/2722554771 wait complete. 2024-04-21T08:50:18.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.335+0000 7f1713b1b700 1 Processor -- start 2024-04-21T08:50:18.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.335+0000 7f1713b1b700 1 -- start start 2024-04-21T08:50:18.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.335+0000 7f1713b1b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f170c102a90 0x7f170c074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.335+0000 7f1713b1b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f170c103450 0x7f170c075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.335+0000 7f1713b1b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f170c108a50 0x7f170c0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.335+0000 7f1713b1b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f170c112c00 con 0x7f170c108a50 2024-04-21T08:50:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.335+0000 7f1713b1b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f170c112a80 con 0x7f170c103450 2024-04-21T08:50:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.335+0000 7f1713b1b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f170c112d80 con 0x7f170c102a90 2024-04-21T08:50:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.335+0000 7f17120b8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f170c108a50 0x7f170c0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.335+0000 7f17120b8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f170c108a50 0x7f170c0757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59764/0 (socket says 172.21.15.28:59764) 2024-04-21T08:50:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.335+0000 7f17120b8700 1 -- 172.21.15.28:0/686379470 learned_addr learned my addr 172.21.15.28:0/686379470 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.335+0000 7f17118b7700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f170c102a90 0x7f170c074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.336+0000 7f17110b6700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f170c103450 0x7f170c075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.336+0000 7f17120b8700 1 -- 172.21.15.28:0/686379470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f170c102a90 msgr2=0x7f170c074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.336+0000 7f17120b8700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f170c102a90 0x7f170c074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.336+0000 7f17120b8700 1 -- 172.21.15.28:0/686379470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f170c103450 msgr2=0x7f170c075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.336+0000 7f17120b8700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f170c103450 0x7f170c075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.336+0000 7f17120b8700 1 -- 172.21.15.28:0/686379470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170c1a7660 con 0x7f170c108a50 2024-04-21T08:50:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.336+0000 7f17110b6700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f170c103450 0x7f170c075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.336+0000 7f17118b7700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f170c102a90 0x7f170c074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:18.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.336+0000 7f17120b8700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f170c108a50 0x7f170c0757c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f170400ef30 tx=0x7f170400c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.336+0000 7f1702ffd700 1 -- 172.21.15.28:0/686379470 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17040092e0 con 0x7f170c108a50 2024-04-21T08:50:18.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.336+0000 7f1702ffd700 1 -- 172.21.15.28:0/686379470 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1704004bb0 con 0x7f170c108a50 2024-04-21T08:50:18.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.336+0000 7f1713b1b700 1 -- 172.21.15.28:0/686379470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f170c1a78f0 con 0x7f170c108a50 2024-04-21T08:50:18.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.336+0000 7f1713b1b700 1 -- 172.21.15.28:0/686379470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f170c1a7e30 con 0x7f170c108a50 2024-04-21T08:50:18.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.336+0000 7f1702ffd700 1 -- 172.21.15.28:0/686379470 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704004280 con 0x7f170c108a50 2024-04-21T08:50:18.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.337+0000 7f1713b1b700 1 -- 172.21.15.28:0/686379470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16f0005330 con 0x7f170c108a50 2024-04-21T08:50:18.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.338+0000 7f1702ffd700 1 -- 172.21.15.28:0/686379470 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f170401b030 con 0x7f170c108a50 2024-04-21T08:50:18.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.339+0000 7f1702ffd700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f16f80a3cd0 0x7f16f80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:18.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.339+0000 7f1702ffd700 1 -- 172.21.15.28:0/686379470 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f17040d0340 con 0x7f170c108a50 2024-04-21T08:50:18.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.339+0000 7f17118b7700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f16f80a3cd0 0x7f16f80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:18.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.340+0000 7f17118b7700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f16f80a3cd0 0x7f16f80a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f170800b2e0 tx=0x7f170801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:18.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.344+0000 7f1702ffd700 1 -- 172.21.15.28:0/686379470 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1704022020 con 0x7f170c108a50 2024-04-21T08:50:18.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.556+0000 7f1713b1b700 1 -- 172.21.15.28:0/686379470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f16f00058f0 con 0x7f170c108a50 2024-04-21T08:50:18.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.557+0000 7f1702ffd700 1 -- 172.21.15.28:0/686379470 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1704010040 con 0x7f170c108a50 2024-04-21T08:50:18.557 INFO:teuthology.orchestra.run.smithi028.stdout:253403070512 2024-04-21T08:50:18.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.559+0000 7f1713b1b700 1 -- 172.21.15.28:0/686379470 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f16f80a3cd0 msgr2=0x7f16f80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.559+0000 7f1713b1b700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f16f80a3cd0 0x7f16f80a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f170800b2e0 tx=0x7f170801a040 comp rx=0 tx=0).stop 2024-04-21T08:50:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.559+0000 7f1713b1b700 1 -- 172.21.15.28:0/686379470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f170c108a50 msgr2=0x7f170c0757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.559+0000 7f1713b1b700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f170c108a50 0x7f170c0757c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f170400ef30 tx=0x7f170400c450 comp rx=0 tx=0).stop 2024-04-21T08:50:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.559+0000 7f1713b1b700 1 -- 172.21.15.28:0/686379470 shutdown_connections 2024-04-21T08:50:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.559+0000 7f1713b1b700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f16f80a3cd0 0x7f16f80a6190 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.559+0000 7f1713b1b700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f170c108a50 0x7f170c0757c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.559+0000 7f1713b1b700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f170c102a90 0x7f170c074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.559+0000 7f1713b1b700 1 --2- 172.21.15.28:0/686379470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f170c103450 0x7f170c075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.559+0000 7f1713b1b700 1 -- 172.21.15.28:0/686379470 >> 172.21.15.28:0/686379470 conn(0x7f170c079ba0 msgr2=0x7f170c1018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.559+0000 7f1713b1b700 1 -- 172.21.15.28:0/686379470 shutdown_connections 2024-04-21T08:50:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:18.559+0000 7f1713b1b700 1 -- 172.21.15.28:0/686379470 wait complete. 2024-04-21T08:50:18.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:18 smithi028 ceph-mon[28003]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:50:18.861 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:18 smithi152 ceph-mon[31401]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:50:18.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:18 smithi171 ceph-mon[31358]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:50:19.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.010+0000 7f718feb0700 1 -- 172.21.15.28:0/3285248838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7188108a50 msgr2=0x7f7188108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.010+0000 7f718feb0700 1 --2- 172.21.15.28:0/3285248838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7188108a50 0x7f7188108e30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7178009990 tx=0x7f7178009ca0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.010+0000 7f718feb0700 1 -- 172.21.15.28:0/3285248838 shutdown_connections 2024-04-21T08:50:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.010+0000 7f718feb0700 1 --2- 172.21.15.28:0/3285248838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7188108a50 0x7f7188108e30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.010+0000 7f718feb0700 1 --2- 172.21.15.28:0/3285248838 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7188103450 0x7f718810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.010+0000 7f718feb0700 1 --2- 172.21.15.28:0/3285248838 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7188102a90 0x7f7188102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.010+0000 7f718feb0700 1 -- 172.21.15.28:0/3285248838 >> 172.21.15.28:0/3285248838 conn(0x7f7188079ba0 msgr2=0x7f7188079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.011+0000 7f718feb0700 1 -- 172.21.15.28:0/3285248838 shutdown_connections 2024-04-21T08:50:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.011+0000 7f718feb0700 1 -- 172.21.15.28:0/3285248838 wait complete. 2024-04-21T08:50:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.011+0000 7f718feb0700 1 Processor -- start 2024-04-21T08:50:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.011+0000 7f718feb0700 1 -- start start 2024-04-21T08:50:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.011+0000 7f718feb0700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7188102a90 0x7f7188074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.011+0000 7f718feb0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7188103450 0x7f7188075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.011+0000 7f718feb0700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7188108a50 0x7f71880757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.011+0000 7f718feb0700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7188112be0 con 0x7f7188103450 2024-04-21T08:50:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.011+0000 7f718feb0700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7188112a60 con 0x7f7188102a90 2024-04-21T08:50:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.011+0000 7f718feb0700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7188112d60 con 0x7f7188108a50 2024-04-21T08:50:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f718d44b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7188103450 0x7f7188075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f718d44b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7188103450 0x7f7188075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59788/0 (socket says 172.21.15.28:59788) 2024-04-21T08:50:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f718d44b700 1 -- 172.21.15.28:0/2612709943 learned_addr learned my addr 172.21.15.28:0/2612709943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f718dc4c700 1 --2- 172.21.15.28:0/2612709943 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7188102a90 0x7f7188074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f718e44d700 1 --2- 172.21.15.28:0/2612709943 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7188108a50 0x7f71880757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f718d44b700 1 -- 172.21.15.28:0/2612709943 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7188108a50 msgr2=0x7f71880757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f718d44b700 1 --2- 172.21.15.28:0/2612709943 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7188108a50 0x7f71880757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f718d44b700 1 -- 172.21.15.28:0/2612709943 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7188102a90 msgr2=0x7f7188074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f718d44b700 1 --2- 172.21.15.28:0/2612709943 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7188102a90 0x7f7188074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f718d44b700 1 -- 172.21.15.28:0/2612709943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71881a7660 con 0x7f7188103450 2024-04-21T08:50:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f718dc4c700 1 --2- 172.21.15.28:0/2612709943 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7188102a90 0x7f7188074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f718d44b700 1 --2- 172.21.15.28:0/2612709943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7188103450 0x7f7188075280 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f718400c920 tx=0x7f718400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f717effd700 1 -- 172.21.15.28:0/2612709943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7184007b00 con 0x7f7188103450 2024-04-21T08:50:19.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f718feb0700 1 -- 172.21.15.28:0/2612709943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71881a7950 con 0x7f7188103450 2024-04-21T08:50:19.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f717effd700 1 -- 172.21.15.28:0/2612709943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7184004ce0 con 0x7f7188103450 2024-04-21T08:50:19.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f717effd700 1 -- 172.21.15.28:0/2612709943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7184005650 con 0x7f7188103450 2024-04-21T08:50:19.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.012+0000 7f718feb0700 1 -- 172.21.15.28:0/2612709943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71881a7e90 con 0x7f7188103450 2024-04-21T08:50:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.013+0000 7f718feb0700 1 -- 172.21.15.28:0/2612709943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7188069730 con 0x7f7188103450 2024-04-21T08:50:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.014+0000 7f717effd700 1 -- 172.21.15.28:0/2612709943 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7184007ca0 con 0x7f7188103450 2024-04-21T08:50:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.015+0000 7f717effd700 1 --2- 172.21.15.28:0/2612709943 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71740a3cd0 0x7f71740a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.015+0000 7f718dc4c700 1 --2- 172.21.15.28:0/2612709943 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71740a3cd0 0x7f71740a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.015+0000 7f717effd700 1 -- 172.21.15.28:0/2612709943 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f71840d0860 con 0x7f7188103450 2024-04-21T08:50:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.015+0000 7f718dc4c700 1 --2- 172.21.15.28:0/2612709943 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71740a3cd0 0x7f71740a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7178006010 tx=0x7f717801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.020+0000 7f717effd700 1 -- 172.21.15.28:0/2612709943 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7184014030 con 0x7f7188103450 2024-04-21T08:50:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.231+0000 7f718feb0700 1 -- 172.21.15.28:0/2612709943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f718802df80 con 0x7f7188103450 2024-04-21T08:50:19.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.231+0000 7f717effd700 1 -- 172.21.15.28:0/2612709943 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f718409cda0 con 0x7f7188103450 2024-04-21T08:50:19.231 INFO:teuthology.orchestra.run.smithi028.stdout:309237645349 2024-04-21T08:50:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.233+0000 7f718feb0700 1 -- 172.21.15.28:0/2612709943 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71740a3cd0 msgr2=0x7f71740a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.233+0000 7f718feb0700 1 --2- 172.21.15.28:0/2612709943 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71740a3cd0 0x7f71740a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7178006010 tx=0x7f717801a040 comp rx=0 tx=0).stop 2024-04-21T08:50:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.233+0000 7f718feb0700 1 -- 172.21.15.28:0/2612709943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7188103450 msgr2=0x7f7188075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.233+0000 7f718feb0700 1 --2- 172.21.15.28:0/2612709943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7188103450 0x7f7188075280 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f718400c920 tx=0x7f718400cce0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.233+0000 7f718feb0700 1 -- 172.21.15.28:0/2612709943 shutdown_connections 2024-04-21T08:50:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.233+0000 7f718feb0700 1 --2- 172.21.15.28:0/2612709943 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7188108a50 0x7f71880757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.233+0000 7f718feb0700 1 --2- 172.21.15.28:0/2612709943 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7188102a90 0x7f7188074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.233+0000 7f718feb0700 1 --2- 172.21.15.28:0/2612709943 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71740a3cd0 0x7f71740a6190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.233+0000 7f718feb0700 1 --2- 172.21.15.28:0/2612709943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7188103450 0x7f7188075280 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.233+0000 7f718feb0700 1 -- 172.21.15.28:0/2612709943 >> 172.21.15.28:0/2612709943 conn(0x7f7188079ba0 msgr2=0x7f71881018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.234+0000 7f718feb0700 1 -- 172.21.15.28:0/2612709943 shutdown_connections 2024-04-21T08:50:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.234+0000 7f718feb0700 1 -- 172.21.15.28:0/2612709943 wait complete. 2024-04-21T08:50:19.602 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070507 got 253403070512 for osd.8 2024-04-21T08:50:19.603 DEBUG:teuthology.parallel:result is None 2024-04-21T08:50:19.833 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:19 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/686379470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T08:50:19.833 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:19 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2612709943' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T08:50:19.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.838+0000 7fc6b35f9700 1 -- 172.21.15.28:0/146810138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ac108a50 msgr2=0x7fc6ac108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.838+0000 7fc6b35f9700 1 --2- 172.21.15.28:0/146810138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ac108a50 0x7fc6ac108e30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8009990 tx=0x7fc6a8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.838+0000 7fc6b35f9700 1 -- 172.21.15.28:0/146810138 shutdown_connections 2024-04-21T08:50:19.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.838+0000 7fc6b35f9700 1 --2- 172.21.15.28:0/146810138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ac108a50 0x7fc6ac108e30 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.838+0000 7fc6b35f9700 1 --2- 172.21.15.28:0/146810138 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6ac103450 0x7fc6ac10f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.838+0000 7fc6b35f9700 1 --2- 172.21.15.28:0/146810138 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6ac102a90 0x7fc6ac102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.838+0000 7fc6b35f9700 1 -- 172.21.15.28:0/146810138 >> 172.21.15.28:0/146810138 conn(0x7fc6ac079ba0 msgr2=0x7fc6ac079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:19.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.838+0000 7fc6b35f9700 1 -- 172.21.15.28:0/146810138 shutdown_connections 2024-04-21T08:50:19.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.838+0000 7fc6b35f9700 1 -- 172.21.15.28:0/146810138 wait complete. 2024-04-21T08:50:19.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.839+0000 7fc6b35f9700 1 Processor -- start 2024-04-21T08:50:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.839+0000 7fc6b35f9700 1 -- start start 2024-04-21T08:50:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.839+0000 7fc6b35f9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ac102a90 0x7fc6ac1990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.839+0000 7fc6b35f9700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6ac103450 0x7fc6ac199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.839+0000 7fc6b35f9700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6ac108a50 0x7fc6ac19d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.839+0000 7fc6b35f9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac112c00 con 0x7fc6ac102a90 2024-04-21T08:50:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.839+0000 7fc6b35f9700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac112a80 con 0x7fc6ac108a50 2024-04-21T08:50:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.839+0000 7fc6b35f9700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac112d80 con 0x7fc6ac103450 2024-04-21T08:50:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6b1395700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ac102a90 0x7fc6ac1990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6b1395700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ac102a90 0x7fc6ac1990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59814/0 (socket says 172.21.15.28:59814) 2024-04-21T08:50:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6b1395700 1 -- 172.21.15.28:0/2295518436 learned_addr learned my addr 172.21.15.28:0/2295518436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6b0b94700 1 --2- 172.21.15.28:0/2295518436 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6ac103450 0x7fc6ac199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6b1b96700 1 --2- 172.21.15.28:0/2295518436 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6ac108a50 0x7fc6ac19d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6b1395700 1 -- 172.21.15.28:0/2295518436 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6ac103450 msgr2=0x7fc6ac199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6b1395700 1 --2- 172.21.15.28:0/2295518436 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6ac103450 0x7fc6ac199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6b1395700 1 -- 172.21.15.28:0/2295518436 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6ac108a50 msgr2=0x7fc6ac19d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6b1395700 1 --2- 172.21.15.28:0/2295518436 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6ac108a50 0x7fc6ac19d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6b1395700 1 -- 172.21.15.28:0/2295518436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ac19e100 con 0x7fc6ac102a90 2024-04-21T08:50:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6b1b96700 1 --2- 172.21.15.28:0/2295518436 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6ac108a50 0x7fc6ac19d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6b1395700 1 --2- 172.21.15.28:0/2295518436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ac102a90 0x7fc6ac1990f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8000c00 tx=0x7fc6a80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6a27fc700 1 -- 172.21.15.28:0/2295518436 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8027070 con 0x7fc6ac102a90 2024-04-21T08:50:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6a27fc700 1 -- 172.21.15.28:0/2295518436 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6a8003f20 con 0x7fc6ac102a90 2024-04-21T08:50:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6b35f9700 1 -- 172.21.15.28:0/2295518436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6ac19e390 con 0x7fc6ac102a90 2024-04-21T08:50:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6a27fc700 1 -- 172.21.15.28:0/2295518436 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8022cf0 con 0x7fc6ac102a90 2024-04-21T08:50:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.840+0000 7fc6b35f9700 1 -- 172.21.15.28:0/2295518436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ac1a5c70 con 0x7fc6ac102a90 2024-04-21T08:50:19.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.841+0000 7fc6b35f9700 1 -- 172.21.15.28:0/2295518436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6ac069730 con 0x7fc6ac102a90 2024-04-21T08:50:19.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.843+0000 7fc6a27fc700 1 -- 172.21.15.28:0/2295518436 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc6a802a070 con 0x7fc6ac102a90 2024-04-21T08:50:19.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.843+0000 7fc6a27fc700 1 --2- 172.21.15.28:0/2295518436 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc6980a3cd0 0x7fc6980a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:19.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.843+0000 7fc6b0b94700 1 --2- 172.21.15.28:0/2295518436 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc6980a3cd0 0x7fc6980a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:19.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.843+0000 7fc6a27fc700 1 -- 172.21.15.28:0/2295518436 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fc6a80d1670 con 0x7fc6ac102a90 2024-04-21T08:50:19.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.844+0000 7fc6b0b94700 1 --2- 172.21.15.28:0/2295518436 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc6980a3cd0 0x7fc6980a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac07a210 tx=0x7fc69c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:19.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:19.848+0000 7fc6a27fc700 1 -- 172.21.15.28:0/2295518436 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc6a8013070 con 0x7fc6ac102a90 2024-04-21T08:50:19.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:19 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/686379470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T08:50:19.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:19 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2612709943' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T08:50:19.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:19 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/686379470' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T08:50:19.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:19 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2612709943' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T08:50:20.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:20.063+0000 7fc6b35f9700 1 -- 172.21.15.28:0/2295518436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fc6ac02df80 con 0x7fc6ac102a90 2024-04-21T08:50:20.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:20.063+0000 7fc6a27fc700 1 -- 172.21.15.28:0/2295518436 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc6a809dc40 con 0x7fc6ac102a90 2024-04-21T08:50:20.064 INFO:teuthology.orchestra.run.smithi028.stdout:197568495674 2024-04-21T08:50:20.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:20.065+0000 7fc6b35f9700 1 -- 172.21.15.28:0/2295518436 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc6980a3cd0 msgr2=0x7fc6980a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:20.065+0000 7fc6b35f9700 1 --2- 172.21.15.28:0/2295518436 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc6980a3cd0 0x7fc6980a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac07a210 tx=0x7fc69c00b410 comp rx=0 tx=0).stop 2024-04-21T08:50:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:20.065+0000 7fc6b35f9700 1 -- 172.21.15.28:0/2295518436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ac102a90 msgr2=0x7fc6ac1990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:20.065+0000 7fc6b35f9700 1 --2- 172.21.15.28:0/2295518436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ac102a90 0x7fc6ac1990f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc6a8000c00 tx=0x7fc6a80047b0 comp rx=0 tx=0).stop 2024-04-21T08:50:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:20.065+0000 7fc6b35f9700 1 -- 172.21.15.28:0/2295518436 shutdown_connections 2024-04-21T08:50:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:20.065+0000 7fc6b35f9700 1 --2- 172.21.15.28:0/2295518436 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc6980a3cd0 0x7fc6980a6190 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:20.065+0000 7fc6b35f9700 1 --2- 172.21.15.28:0/2295518436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ac102a90 0x7fc6ac1990f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:20.065+0000 7fc6b35f9700 1 --2- 172.21.15.28:0/2295518436 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6ac108a50 0x7fc6ac19d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:20.065+0000 7fc6b35f9700 1 --2- 172.21.15.28:0/2295518436 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6ac103450 0x7fc6ac199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:20.066+0000 7fc6b35f9700 1 -- 172.21.15.28:0/2295518436 >> 172.21.15.28:0/2295518436 conn(0x7fc6ac079ba0 msgr2=0x7fc6ac1017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:20.066+0000 7fc6b35f9700 1 -- 172.21.15.28:0/2295518436 shutdown_connections 2024-04-21T08:50:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:20.066+0000 7fc6b35f9700 1 -- 172.21.15.28:0/2295518436 wait complete. 2024-04-21T08:50:20.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:20 smithi152 ceph-mon[31401]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:20.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:20 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2295518436' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T08:50:20.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:20 smithi028 ceph-mon[28003]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:20.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:20 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2295518436' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T08:50:20.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:20 smithi171 ceph-mon[31358]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:20.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:20 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2295518436' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T08:50:21.195 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495668 got 197568495674 for osd.6 2024-04-21T08:50:21.195 DEBUG:teuthology.parallel:result is None 2024-04-21T08:50:21.880 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645343 got 309237645349 for osd.10 2024-04-21T08:50:21.880 DEBUG:teuthology.parallel:result is None 2024-04-21T08:50:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.107+0000 7fde9b4ac700 1 -- 172.21.15.28:0/2356503777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9406b170 msgr2=0x7fde9406b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.107+0000 7fde9b4ac700 1 --2- 172.21.15.28:0/2356503777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9406b170 0x7fde9406b5f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fde84009990 tx=0x7fde84009ca0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.107+0000 7fde9b4ac700 1 -- 172.21.15.28:0/2356503777 shutdown_connections 2024-04-21T08:50:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.107+0000 7fde9b4ac700 1 --2- 172.21.15.28:0/2356503777 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde9406bb30 0x7fde9410f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.107+0000 7fde9b4ac700 1 --2- 172.21.15.28:0/2356503777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde94108a70 0x7fde94108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.107+0000 7fde9b4ac700 1 --2- 172.21.15.28:0/2356503777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9406b170 0x7fde9406b5f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.107+0000 7fde9b4ac700 1 -- 172.21.15.28:0/2356503777 >> 172.21.15.28:0/2356503777 conn(0x7fde940777b0 msgr2=0x7fde94077bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.107+0000 7fde9b4ac700 1 -- 172.21.15.28:0/2356503777 shutdown_connections 2024-04-21T08:50:22.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.107+0000 7fde9b4ac700 1 -- 172.21.15.28:0/2356503777 wait complete. 2024-04-21T08:50:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.108+0000 7fde9b4ac700 1 Processor -- start 2024-04-21T08:50:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.108+0000 7fde9b4ac700 1 -- start start 2024-04-21T08:50:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.108+0000 7fde9b4ac700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde9406b170 0x7fde94199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.108+0000 7fde9b4ac700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9406bb30 0x7fde941995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.108+0000 7fde9b4ac700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde94108a70 0x7fde9419d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.108+0000 7fde9b4ac700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde94112c50 con 0x7fde9406bb30 2024-04-21T08:50:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.108+0000 7fde9b4ac700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fde94112ad0 con 0x7fde94108a70 2024-04-21T08:50:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.108+0000 7fde9b4ac700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fde94112dd0 con 0x7fde9406b170 2024-04-21T08:50:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.108+0000 7fde98a47700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9406bb30 0x7fde941995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.108+0000 7fde98a47700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9406bb30 0x7fde941995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59820/0 (socket says 172.21.15.28:59820) 2024-04-21T08:50:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.108+0000 7fde98a47700 1 -- 172.21.15.28:0/1501857035 learned_addr learned my addr 172.21.15.28:0/1501857035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.108+0000 7fde99a49700 1 --2- 172.21.15.28:0/1501857035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde94108a70 0x7fde9419d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.108+0000 7fde99248700 1 --2- 172.21.15.28:0/1501857035 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde9406b170 0x7fde94199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.109+0000 7fde98a47700 1 -- 172.21.15.28:0/1501857035 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde9406b170 msgr2=0x7fde94199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.109+0000 7fde98a47700 1 --2- 172.21.15.28:0/1501857035 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde9406b170 0x7fde94199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.109+0000 7fde98a47700 1 -- 172.21.15.28:0/1501857035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde94108a70 msgr2=0x7fde9419d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.109+0000 7fde98a47700 1 --2- 172.21.15.28:0/1501857035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde94108a70 0x7fde9419d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.109+0000 7fde98a47700 1 -- 172.21.15.28:0/1501857035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9419e090 con 0x7fde9406bb30 2024-04-21T08:50:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.109+0000 7fde99248700 1 --2- 172.21.15.28:0/1501857035 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde9406b170 0x7fde94199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.109+0000 7fde98a47700 1 --2- 172.21.15.28:0/1501857035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9406bb30 0x7fde941995c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fde840047b0 tx=0x7fde84005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.109+0000 7fde8a7fc700 1 -- 172.21.15.28:0/1501857035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84026070 con 0x7fde9406bb30 2024-04-21T08:50:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.109+0000 7fde8a7fc700 1 -- 172.21.15.28:0/1501857035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde84004c90 con 0x7fde9406bb30 2024-04-21T08:50:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.109+0000 7fde9b4ac700 1 -- 172.21.15.28:0/1501857035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde9419e320 con 0x7fde9406bb30 2024-04-21T08:50:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.109+0000 7fde8a7fc700 1 -- 172.21.15.28:0/1501857035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8400f9d0 con 0x7fde9406bb30 2024-04-21T08:50:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.109+0000 7fde9b4ac700 1 -- 172.21.15.28:0/1501857035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde941a5c00 con 0x7fde9406bb30 2024-04-21T08:50:22.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.110+0000 7fde9b4ac700 1 -- 172.21.15.28:0/1501857035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde941041a0 con 0x7fde9406bb30 2024-04-21T08:50:22.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.112+0000 7fde8a7fc700 1 -- 172.21.15.28:0/1501857035 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fde84029070 con 0x7fde9406bb30 2024-04-21T08:50:22.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.112+0000 7fde8a7fc700 1 --2- 172.21.15.28:0/1501857035 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fde800a3d50 0x7fde800a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.112+0000 7fde99248700 1 --2- 172.21.15.28:0/1501857035 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fde800a3d50 0x7fde800a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.112+0000 7fde8a7fc700 1 -- 172.21.15.28:0/1501857035 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fde840d1500 con 0x7fde9406bb30 2024-04-21T08:50:22.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.113+0000 7fde99248700 1 --2- 172.21.15.28:0/1501857035 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fde800a3d50 0x7fde800a6210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fde9000baa0 tx=0x7fde90005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:22.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.116+0000 7fde8a7fc700 1 -- 172.21.15.28:0/1501857035 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde84030020 con 0x7fde9406bb30 2024-04-21T08:50:22.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.340+0000 7fde9b4ac700 1 -- 172.21.15.28:0/1501857035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fde9402df80 con 0x7fde9406bb30 2024-04-21T08:50:22.342 INFO:teuthology.orchestra.run.smithi028.stdout:287762808873 2024-04-21T08:50:22.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.341+0000 7fde8a7fc700 1 -- 172.21.15.28:0/1501857035 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fde84013070 con 0x7fde9406bb30 2024-04-21T08:50:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.344+0000 7fde9b4ac700 1 -- 172.21.15.28:0/1501857035 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fde800a3d50 msgr2=0x7fde800a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.344+0000 7fde9b4ac700 1 --2- 172.21.15.28:0/1501857035 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fde800a3d50 0x7fde800a6210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fde9000baa0 tx=0x7fde90005db0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.344+0000 7fde9b4ac700 1 -- 172.21.15.28:0/1501857035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9406bb30 msgr2=0x7fde941995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.344+0000 7fde9b4ac700 1 --2- 172.21.15.28:0/1501857035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9406bb30 0x7fde941995c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fde840047b0 tx=0x7fde84005e80 comp rx=0 tx=0).stop 2024-04-21T08:50:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.344+0000 7fde9b4ac700 1 -- 172.21.15.28:0/1501857035 shutdown_connections 2024-04-21T08:50:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.344+0000 7fde9b4ac700 1 --2- 172.21.15.28:0/1501857035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde94108a70 0x7fde9419d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.344+0000 7fde9b4ac700 1 --2- 172.21.15.28:0/1501857035 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde9406b170 0x7fde94199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.344+0000 7fde9b4ac700 1 --2- 172.21.15.28:0/1501857035 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fde800a3d50 0x7fde800a6210 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.344+0000 7fde9b4ac700 1 --2- 172.21.15.28:0/1501857035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9406bb30 0x7fde941995c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.344+0000 7fde9b4ac700 1 -- 172.21.15.28:0/1501857035 >> 172.21.15.28:0/1501857035 conn(0x7fde940777b0 msgr2=0x7fde941008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.344+0000 7fde9b4ac700 1 -- 172.21.15.28:0/1501857035 shutdown_connections 2024-04-21T08:50:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:22.344+0000 7fde9b4ac700 1 -- 172.21.15.28:0/1501857035 wait complete. 2024-04-21T08:50:23.005 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808867 got 287762808873 for osd.9 2024-04-21T08:50:23.005 DEBUG:teuthology.parallel:result is None 2024-04-21T08:50:23.005 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-21T08:50:23.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-21T08:50:23.267 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:23.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:23 smithi028 ceph-mon[28003]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-21T08:50:23.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:23 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:50:23.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:23 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:50:23.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:23 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1501857035' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T08:50:23.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:23 smithi152 ceph-mon[31401]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-21T08:50:23.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:23 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:50:23.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:23 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:50:23.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:23 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1501857035' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T08:50:23.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:23 smithi171 ceph-mon[31358]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-21T08:50:23.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:23 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:50:23.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:23 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:50:23.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:23 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1501857035' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T08:50:24.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:24 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:50:24.599 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:24 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:50:24.599 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:24 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:50:24.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:24 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:50:24.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:24 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:50:24.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:24 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:50:24.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:24 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:50:24.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:24 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:50:24.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:24 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' 2024-04-21T08:50:25.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.026+0000 7f4f95c20700 1 -- 172.21.15.28:0/1719615462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f90102300 msgr2=0x7f4f9010e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:25.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.026+0000 7f4f95c20700 1 --2- 172.21.15.28:0/1719615462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f90102300 0x7f4f9010e7f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4f8800b0a0 tx=0x7f4f8800b3b0 comp rx=0 tx=0).stop 2024-04-21T08:50:25.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.026+0000 7f4f95c20700 1 -- 172.21.15.28:0/1719615462 shutdown_connections 2024-04-21T08:50:25.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.026+0000 7f4f95c20700 1 --2- 172.21.15.28:0/1719615462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f90102300 0x7f4f9010e7f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:25.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.026+0000 7f4f95c20700 1 --2- 172.21.15.28:0/1719615462 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f90107990 0x7f4f90107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:25.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.026+0000 7f4f95c20700 1 --2- 172.21.15.28:0/1719615462 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f90101940 0x7f4f90101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:25.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.026+0000 7f4f95c20700 1 -- 172.21.15.28:0/1719615462 >> 172.21.15.28:0/1719615462 conn(0x7f4f900788f0 msgr2=0x7f4f90078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:25.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.026+0000 7f4f95c20700 1 -- 172.21.15.28:0/1719615462 shutdown_connections 2024-04-21T08:50:25.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.026+0000 7f4f95c20700 1 -- 172.21.15.28:0/1719615462 wait complete. 2024-04-21T08:50:25.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.027+0000 7f4f95c20700 1 Processor -- start 2024-04-21T08:50:25.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.027+0000 7f4f95c20700 1 -- start start 2024-04-21T08:50:25.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.027+0000 7f4f95c20700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f90101940 0x7f4f90197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:25.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.027+0000 7f4f95c20700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f90102300 0x7f4f90198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:25.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.027+0000 7f4f95c20700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f90107990 0x7f4f9019c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.027+0000 7f4f95c20700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f90111ae0 con 0x7f4f90107990 2024-04-21T08:50:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.027+0000 7f4f95c20700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4f90111960 con 0x7f4f90102300 2024-04-21T08:50:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.027+0000 7f4f95c20700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4f90111c60 con 0x7f4f90101940 2024-04-21T08:50:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f8ffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f90107990 0x7f4f9019c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f8ffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f90107990 0x7f4f9019c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36688/0 (socket says 172.21.15.28:36688) 2024-04-21T08:50:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f8ffff700 1 -- 172.21.15.28:0/1417377218 learned_addr learned my addr 172.21.15.28:0/1417377218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f8effd700 1 --2- 172.21.15.28:0/1417377218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f90102300 0x7f4f90198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f8f7fe700 1 --2- 172.21.15.28:0/1417377218 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f90101940 0x7f4f90197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f8ffff700 1 -- 172.21.15.28:0/1417377218 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f90101940 msgr2=0x7f4f90197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f8ffff700 1 --2- 172.21.15.28:0/1417377218 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f90101940 0x7f4f90197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f8ffff700 1 -- 172.21.15.28:0/1417377218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f90102300 msgr2=0x7f4f90198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f8ffff700 1 --2- 172.21.15.28:0/1417377218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f90102300 0x7f4f90198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f8ffff700 1 -- 172.21.15.28:0/1417377218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9019d000 con 0x7f4f90107990 2024-04-21T08:50:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f8f7fe700 1 --2- 172.21.15.28:0/1417377218 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f90101940 0x7f4f90197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:25.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f8ffff700 1 --2- 172.21.15.28:0/1417377218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f90107990 0x7f4f9019c8c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4f8800b830 tx=0x7f4f88004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:25.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f8cff9700 1 -- 172.21.15.28:0/1417377218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f88021070 con 0x7f4f90107990 2024-04-21T08:50:25.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f95c20700 1 -- 172.21.15.28:0/1417377218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f9019d290 con 0x7f4f90107990 2024-04-21T08:50:25.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f8cff9700 1 -- 172.21.15.28:0/1417377218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f88007e30 con 0x7f4f90107990 2024-04-21T08:50:25.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f8cff9700 1 -- 172.21.15.28:0/1417377218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f880047d0 con 0x7f4f90107990 2024-04-21T08:50:25.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.028+0000 7f4f95c20700 1 -- 172.21.15.28:0/1417377218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f901a4b70 con 0x7f4f90107990 2024-04-21T08:50:25.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.029+0000 7f4f95c20700 1 -- 172.21.15.28:0/1417377218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f90069730 con 0x7f4f90107990 2024-04-21T08:50:25.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.031+0000 7f4f8cff9700 1 -- 172.21.15.28:0/1417377218 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4f88007980 con 0x7f4f90107990 2024-04-21T08:50:25.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.032+0000 7f4f8cff9700 1 --2- 172.21.15.28:0/1417377218 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f780a3cd0 0x7f4f780a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:25.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.032+0000 7f4f8cff9700 1 -- 172.21.15.28:0/1417377218 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f4f8800f020 con 0x7f4f90107990 2024-04-21T08:50:25.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.032+0000 7f4f8f7fe700 1 --2- 172.21.15.28:0/1417377218 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f780a3cd0 0x7f4f780a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:25.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.032+0000 7f4f8f7fe700 1 --2- 172.21.15.28:0/1417377218 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f780a3cd0 0x7f4f780a6190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000bb90 tx=0x7f4f80005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:25.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.035+0000 7f4f8cff9700 1 -- 172.21.15.28:0/1417377218 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f8802a020 con 0x7f4f90107990 2024-04-21T08:50:25.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.240+0000 7f4f95c20700 1 -- 172.21.15.28:0/1417377218 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4f90199350 con 0x7f4f780a3cd0 2024-04-21T08:50:25.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.252+0000 7f4f8cff9700 1 -- 172.21.15.28:0/1417377218 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340649 (secure 0 0 0) 0x7f4f90199350 con 0x7f4f780a3cd0 2024-04-21T08:50:25.258 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:50:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.257+0000 7f4f95c20700 1 -- 172.21.15.28:0/1417377218 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f780a3cd0 msgr2=0x7f4f780a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.257+0000 7f4f95c20700 1 --2- 172.21.15.28:0/1417377218 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f780a3cd0 0x7f4f780a6190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000bb90 tx=0x7f4f80005d80 comp rx=0 tx=0).stop 2024-04-21T08:50:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.257+0000 7f4f95c20700 1 -- 172.21.15.28:0/1417377218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f90107990 msgr2=0x7f4f9019c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.257+0000 7f4f95c20700 1 --2- 172.21.15.28:0/1417377218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f90107990 0x7f4f9019c8c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4f8800b830 tx=0x7f4f88004ce0 comp rx=0 tx=0).stop 2024-04-21T08:50:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.257+0000 7f4f95c20700 1 -- 172.21.15.28:0/1417377218 shutdown_connections 2024-04-21T08:50:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.257+0000 7f4f95c20700 1 --2- 172.21.15.28:0/1417377218 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f780a3cd0 0x7f4f780a6190 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.257+0000 7f4f95c20700 1 --2- 172.21.15.28:0/1417377218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f90107990 0x7f4f9019c8c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.257+0000 7f4f95c20700 1 --2- 172.21.15.28:0/1417377218 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f90101940 0x7f4f90197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.257+0000 7f4f95c20700 1 --2- 172.21.15.28:0/1417377218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f90102300 0x7f4f90198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.257+0000 7f4f95c20700 1 -- 172.21.15.28:0/1417377218 >> 172.21.15.28:0/1417377218 conn(0x7f4f900788f0 msgr2=0x7f4f900ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.257+0000 7f4f95c20700 1 -- 172.21.15.28:0/1417377218 shutdown_connections 2024-04-21T08:50:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:25.257+0000 7f4f95c20700 1 -- 172.21.15.28:0/1417377218 wait complete. 2024-04-21T08:50:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-21T08:50:25.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:25 smithi028 ceph-mon[28003]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:25.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:25 smithi152 ceph-mon[31401]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:25.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:25 smithi171 ceph-mon[31358]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:25.949 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-04-21T08:50:23.486537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657959,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":573,"num_read_kb":407,"num_write":344,"num_write_kb":3056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181430,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369676,"kb_used_data":12152,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572148,"statfs":{"total":1151940427776,"available":1151561879552,"internally_reserved":0,"allocated":12443648,"data_stored":7958927,"data_compressed":360168,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003414"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905590+0000","last_change":"2024-04-21T08:48:46.471626+0000","last_active":"2024-04-21T08:49:36.905590+0000","last_peered":"2024-04-21T08:49:36.905590+0000","last_clean":"2024-04-21T08:49:36.905590+0000","last_became_active":"2024-04-21T08:48:46.470498+0000","last_became_peered":"2024-04-21T08:48:46.470498+0000","last_unstale":"2024-04-21T08:49:36.905590+0000","last_undegraded":"2024-04-21T08:49:36.905590+0000","last_fullsized":"2024-04-21T08:49:36.905590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:49.452181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906122+0000","last_change":"2024-04-21T08:48:46.469111+0000","last_active":"2024-04-21T08:49:36.906122+0000","last_peered":"2024-04-21T08:49:36.906122+0000","last_clean":"2024-04-21T08:49:36.906122+0000","last_became_active":"2024-04-21T08:48:46.467926+0000","last_became_peered":"2024-04-21T08:48:46.467926+0000","last_unstale":"2024-04-21T08:49:36.906122+0000","last_undegraded":"2024-04-21T08:49:36.906122+0000","last_fullsized":"2024-04-21T08:49:36.906122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:35:38.230212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905812+0000","last_change":"2024-04-21T08:48:46.475673+0000","last_active":"2024-04-21T08:49:36.905812+0000","last_peered":"2024-04-21T08:49:36.905812+0000","last_clean":"2024-04-21T08:49:36.905812+0000","last_became_active":"2024-04-21T08:48:46.474917+0000","last_became_peered":"2024-04-21T08:48:46.474917+0000","last_unstale":"2024-04-21T08:49:36.905812+0000","last_undegraded":"2024-04-21T08:49:36.905812+0000","last_fullsized":"2024-04-21T08:49:36.905812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:01.661704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908100+0000","last_change":"2024-04-21T08:48:46.468147+0000","last_active":"2024-04-21T08:49:36.908100+0000","last_peered":"2024-04-21T08:49:36.908100+0000","last_clean":"2024-04-21T08:49:36.908100+0000","last_became_active":"2024-04-21T08:48:46.467811+0000","last_became_peered":"2024-04-21T08:48:46.467811+0000","last_unstale":"2024-04-21T08:49:36.908100+0000","last_undegraded":"2024-04-21T08:49:36.908100+0000","last_fullsized":"2024-04-21T08:49:36.908100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:21.658605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908334+0000","last_change":"2024-04-21T08:48:46.471322+0000","last_active":"2024-04-21T08:49:36.908334+0000","last_peered":"2024-04-21T08:49:36.908334+0000","last_clean":"2024-04-21T08:49:36.908334+0000","last_became_active":"2024-04-21T08:48:46.470991+0000","last_became_peered":"2024-04-21T08:48:46.470991+0000","last_unstale":"2024-04-21T08:49:36.908334+0000","last_undegraded":"2024-04-21T08:49:36.908334+0000","last_fullsized":"2024-04-21T08:49:36.908334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:22.839870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963381+0000","last_change":"2024-04-21T08:48:46.467039+0000","last_active":"2024-04-21T08:49:36.963381+0000","last_peered":"2024-04-21T08:49:36.963381+0000","last_clean":"2024-04-21T08:49:36.963381+0000","last_became_active":"2024-04-21T08:48:46.466614+0000","last_became_peered":"2024-04-21T08:48:46.466614+0000","last_unstale":"2024-04-21T08:49:36.963381+0000","last_undegraded":"2024-04-21T08:49:36.963381+0000","last_fullsized":"2024-04-21T08:49:36.963381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:38.029215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.912475+0000","last_change":"2024-04-21T08:48:46.473268+0000","last_active":"2024-04-21T08:49:36.912475+0000","last_peered":"2024-04-21T08:49:36.912475+0000","last_clean":"2024-04-21T08:49:36.912475+0000","last_became_active":"2024-04-21T08:48:46.472261+0000","last_became_peered":"2024-04-21T08:48:46.472261+0000","last_unstale":"2024-04-21T08:49:36.912475+0000","last_undegraded":"2024-04-21T08:49:36.912475+0000","last_fullsized":"2024-04-21T08:49:36.912475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:58:52.990687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963050+0000","last_change":"2024-04-21T08:48:46.472788+0000","last_active":"2024-04-21T08:49:36.963050+0000","last_peered":"2024-04-21T08:49:36.963050+0000","last_clean":"2024-04-21T08:49:36.963050+0000","last_became_active":"2024-04-21T08:48:46.472482+0000","last_became_peered":"2024-04-21T08:48:46.472482+0000","last_unstale":"2024-04-21T08:49:36.963050+0000","last_undegraded":"2024-04-21T08:49:36.963050+0000","last_fullsized":"2024-04-21T08:49:36.963050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:48.483846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907852+0000","last_change":"2024-04-21T08:48:46.460771+0000","last_active":"2024-04-21T08:49:36.907852+0000","last_peered":"2024-04-21T08:49:36.907852+0000","last_clean":"2024-04-21T08:49:36.907852+0000","last_became_active":"2024-04-21T08:48:46.460472+0000","last_became_peered":"2024-04-21T08:48:46.460472+0000","last_unstale":"2024-04-21T08:49:36.907852+0000","last_undegraded":"2024-04-21T08:49:36.907852+0000","last_fullsized":"2024-04-21T08:49:36.907852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:46:40.766245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963692+0000","last_change":"2024-04-21T08:48:46.463475+0000","last_active":"2024-04-21T08:49:36.963692+0000","last_peered":"2024-04-21T08:49:36.963692+0000","last_clean":"2024-04-21T08:49:36.963692+0000","last_became_active":"2024-04-21T08:48:46.462559+0000","last_became_peered":"2024-04-21T08:48:46.462559+0000","last_unstale":"2024-04-21T08:49:36.963692+0000","last_undegraded":"2024-04-21T08:49:36.963692+0000","last_fullsized":"2024-04-21T08:49:36.963692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:22:06.372410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913327+0000","last_change":"2024-04-21T08:48:46.476997+0000","last_active":"2024-04-21T08:49:36.913327+0000","last_peered":"2024-04-21T08:49:36.913327+0000","last_clean":"2024-04-21T08:49:36.913327+0000","last_became_active":"2024-04-21T08:48:46.476620+0000","last_became_peered":"2024-04-21T08:48:46.476620+0000","last_unstale":"2024-04-21T08:49:36.913327+0000","last_undegraded":"2024-04-21T08:49:36.913327+0000","last_fullsized":"2024-04-21T08:49:36.913327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:37.569629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908164+0000","last_change":"2024-04-21T08:48:46.469355+0000","last_active":"2024-04-21T08:49:36.908164+0000","last_peered":"2024-04-21T08:49:36.908164+0000","last_clean":"2024-04-21T08:49:36.908164+0000","last_became_active":"2024-04-21T08:48:46.468604+0000","last_became_peered":"2024-04-21T08:48:46.468604+0000","last_unstale":"2024-04-21T08:49:36.908164+0000","last_undegraded":"2024-04-21T08:49:36.908164+0000","last_fullsized":"2024-04-21T08:49:36.908164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:23:54.500982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905526+0000","last_change":"2024-04-21T08:48:46.453116+0000","last_active":"2024-04-21T08:49:36.905526+0000","last_peered":"2024-04-21T08:49:36.905526+0000","last_clean":"2024-04-21T08:49:36.905526+0000","last_became_active":"2024-04-21T08:48:46.452607+0000","last_became_peered":"2024-04-21T08:48:46.452607+0000","last_unstale":"2024-04-21T08:49:36.905526+0000","last_undegraded":"2024-04-21T08:49:36.905526+0000","last_fullsized":"2024-04-21T08:49:36.905526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:57:04.371812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906749+0000","last_change":"2024-04-21T08:48:46.474516+0000","last_active":"2024-04-21T08:49:36.906749+0000","last_peered":"2024-04-21T08:49:36.906749+0000","last_clean":"2024-04-21T08:49:36.906749+0000","last_became_active":"2024-04-21T08:48:46.474129+0000","last_became_peered":"2024-04-21T08:48:46.474129+0000","last_unstale":"2024-04-21T08:49:36.906749+0000","last_undegraded":"2024-04-21T08:49:36.906749+0000","last_fullsized":"2024-04-21T08:49:36.906749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:40:47.784976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907964+0000","last_change":"2024-04-21T08:49:36.907964+0000","last_active":"2024-04-21T08:49:36.907964+0000","last_peered":"2024-04-21T08:49:36.907964+0000","last_clean":"2024-04-21T08:49:36.907964+0000","last_became_active":"2024-04-21T08:49:36.907595+0000","last_became_peered":"2024-04-21T08:49:36.907595+0000","last_unstale":"2024-04-21T08:49:36.907964+0000","last_undegraded":"2024-04-21T08:49:36.907964+0000","last_fullsized":"2024-04-21T08:49:36.907964+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:14:04.907328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908095+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:49:36.908095+0000","last_peered":"2024-04-21T08:49:36.908095+0000","last_clean":"2024-04-21T08:49:36.908095+0000","last_became_active":"2024-04-21T08:48:46.469389+0000","last_became_peered":"2024-04-21T08:48:46.469389+0000","last_unstale":"2024-04-21T08:49:36.908095+0000","last_undegraded":"2024-04-21T08:49:36.908095+0000","last_fullsized":"2024-04-21T08:49:36.908095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:11:45.824402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905463+0000","last_change":"2024-04-21T08:48:46.470059+0000","last_active":"2024-04-21T08:49:36.905463+0000","last_peered":"2024-04-21T08:49:36.905463+0000","last_clean":"2024-04-21T08:49:36.905463+0000","last_became_active":"2024-04-21T08:48:46.469751+0000","last_became_peered":"2024-04-21T08:48:46.469751+0000","last_unstale":"2024-04-21T08:49:36.905463+0000","last_undegraded":"2024-04-21T08:49:36.905463+0000","last_fullsized":"2024-04-21T08:49:36.905463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:16.317879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902639+0000","last_change":"2024-04-21T08:48:46.470129+0000","last_active":"2024-04-21T08:49:36.902639+0000","last_peered":"2024-04-21T08:49:36.902639+0000","last_clean":"2024-04-21T08:49:36.902639+0000","last_became_active":"2024-04-21T08:48:46.469763+0000","last_became_peered":"2024-04-21T08:48:46.469763+0000","last_unstale":"2024-04-21T08:49:36.902639+0000","last_undegraded":"2024-04-21T08:49:36.902639+0000","last_fullsized":"2024-04-21T08:49:36.902639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:47.731201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.903865+0000","last_change":"2024-04-21T08:48:46.465356+0000","last_active":"2024-04-21T08:49:36.903865+0000","last_peered":"2024-04-21T08:49:36.903865+0000","last_clean":"2024-04-21T08:49:36.903865+0000","last_became_active":"2024-04-21T08:48:46.464183+0000","last_became_peered":"2024-04-21T08:48:46.464183+0000","last_unstale":"2024-04-21T08:49:36.903865+0000","last_undegraded":"2024-04-21T08:49:36.903865+0000","last_fullsized":"2024-04-21T08:49:36.903865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:37.533733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.903496+0000","last_change":"2024-04-21T08:48:50.767378+0000","last_active":"2024-04-21T08:49:36.903496+0000","last_peered":"2024-04-21T08:49:36.903496+0000","last_clean":"2024-04-21T08:49:36.903496+0000","last_became_active":"2024-04-21T08:48:50.766856+0000","last_became_peered":"2024-04-21T08:48:50.766856+0000","last_unstale":"2024-04-21T08:49:36.903496+0000","last_undegraded":"2024-04-21T08:49:36.903496+0000","last_fullsized":"2024-04-21T08:49:36.903496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:00.434540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902634+0000","last_change":"2024-04-21T08:48:45.550284+0000","last_active":"2024-04-21T08:49:36.902634+0000","last_peered":"2024-04-21T08:49:36.902634+0000","last_clean":"2024-04-21T08:49:36.902634+0000","last_became_active":"2024-04-21T08:48:45.549940+0000","last_became_peered":"2024-04-21T08:48:45.549940+0000","last_unstale":"2024-04-21T08:49:36.902634+0000","last_undegraded":"2024-04-21T08:49:36.902634+0000","last_fullsized":"2024-04-21T08:49:36.902634+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:12.343299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963199+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:49:36.963199+0000","last_peered":"2024-04-21T08:49:36.963199+0000","last_clean":"2024-04-21T08:49:36.963199+0000","last_became_active":"2024-04-21T08:48:46.469647+0000","last_became_peered":"2024-04-21T08:48:46.469647+0000","last_unstale":"2024-04-21T08:49:36.963199+0000","last_undegraded":"2024-04-21T08:49:36.963199+0000","last_fullsized":"2024-04-21T08:49:36.963199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:10:28.204554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906837+0000","last_change":"2024-04-21T08:48:50.771871+0000","last_active":"2024-04-21T08:49:36.906837+0000","last_peered":"2024-04-21T08:49:36.906837+0000","last_clean":"2024-04-21T08:49:36.906837+0000","last_became_active":"2024-04-21T08:48:50.770442+0000","last_became_peered":"2024-04-21T08:48:50.770442+0000","last_unstale":"2024-04-21T08:49:36.906837+0000","last_undegraded":"2024-04-21T08:49:36.906837+0000","last_fullsized":"2024-04-21T08:49:36.906837+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:18:57.813971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963310+0000","last_change":"2024-04-21T08:48:45.566492+0000","last_active":"2024-04-21T08:49:36.963310+0000","last_peered":"2024-04-21T08:49:36.963310+0000","last_clean":"2024-04-21T08:49:36.963310+0000","last_became_active":"2024-04-21T08:48:45.566169+0000","last_became_peered":"2024-04-21T08:48:45.566169+0000","last_unstale":"2024-04-21T08:49:36.963310+0000","last_undegraded":"2024-04-21T08:49:36.963310+0000","last_fullsized":"2024-04-21T08:49:36.963310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:49.985664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906531+0000","last_change":"2024-04-21T08:48:46.470866+0000","last_active":"2024-04-21T08:49:36.906531+0000","last_peered":"2024-04-21T08:49:36.906531+0000","last_clean":"2024-04-21T08:49:36.906531+0000","last_became_active":"2024-04-21T08:48:46.470401+0000","last_became_peered":"2024-04-21T08:48:46.470401+0000","last_unstale":"2024-04-21T08:49:36.906531+0000","last_undegraded":"2024-04-21T08:49:36.906531+0000","last_fullsized":"2024-04-21T08:49:36.906531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:07.506212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907429+0000","last_change":"2024-04-21T08:48:50.772339+0000","last_active":"2024-04-21T08:49:36.907429+0000","last_peered":"2024-04-21T08:49:36.907429+0000","last_clean":"2024-04-21T08:49:36.907429+0000","last_became_active":"2024-04-21T08:48:50.771946+0000","last_became_peered":"2024-04-21T08:48:50.771946+0000","last_unstale":"2024-04-21T08:49:36.907429+0000","last_undegraded":"2024-04-21T08:49:36.907429+0000","last_fullsized":"2024-04-21T08:49:36.907429+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:03:56.994581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.909040+0000","last_change":"2024-04-21T08:48:45.553946+0000","last_active":"2024-04-21T08:49:36.909040+0000","last_peered":"2024-04-21T08:49:36.909040+0000","last_clean":"2024-04-21T08:49:36.909040+0000","last_became_active":"2024-04-21T08:48:45.553410+0000","last_became_peered":"2024-04-21T08:48:45.553410+0000","last_unstale":"2024-04-21T08:49:36.909040+0000","last_undegraded":"2024-04-21T08:49:36.909040+0000","last_fullsized":"2024-04-21T08:49:36.909040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:24:02.387250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907005+0000","last_change":"2024-04-21T08:48:46.471815+0000","last_active":"2024-04-21T08:49:36.907005+0000","last_peered":"2024-04-21T08:49:36.907005+0000","last_clean":"2024-04-21T08:49:36.907005+0000","last_became_active":"2024-04-21T08:48:46.471492+0000","last_became_peered":"2024-04-21T08:48:46.471492+0000","last_unstale":"2024-04-21T08:49:36.907005+0000","last_undegraded":"2024-04-21T08:49:36.907005+0000","last_fullsized":"2024-04-21T08:49:36.907005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:26.051166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906784+0000","last_change":"2024-04-21T08:48:50.767916+0000","last_active":"2024-04-21T08:49:36.906784+0000","last_peered":"2024-04-21T08:49:36.906784+0000","last_clean":"2024-04-21T08:49:36.906784+0000","last_became_active":"2024-04-21T08:48:50.767326+0000","last_became_peered":"2024-04-21T08:48:50.767326+0000","last_unstale":"2024-04-21T08:49:36.906784+0000","last_undegraded":"2024-04-21T08:49:36.906784+0000","last_fullsized":"2024-04-21T08:49:36.906784+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:05:02.185369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908720+0000","last_change":"2024-04-21T08:48:45.549982+0000","last_active":"2024-04-21T08:49:36.908720+0000","last_peered":"2024-04-21T08:49:36.908720+0000","last_clean":"2024-04-21T08:49:36.908720+0000","last_became_active":"2024-04-21T08:48:45.549611+0000","last_became_peered":"2024-04-21T08:48:45.549611+0000","last_unstale":"2024-04-21T08:49:36.908720+0000","last_undegraded":"2024-04-21T08:49:36.908720+0000","last_fullsized":"2024-04-21T08:49:36.908720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:46.035388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.914297+0000","last_change":"2024-04-21T08:48:46.459068+0000","last_active":"2024-04-21T08:49:36.914297+0000","last_peered":"2024-04-21T08:49:36.914297+0000","last_clean":"2024-04-21T08:49:36.914297+0000","last_became_active":"2024-04-21T08:48:46.458686+0000","last_became_peered":"2024-04-21T08:48:46.458686+0000","last_unstale":"2024-04-21T08:49:36.914297+0000","last_undegraded":"2024-04-21T08:49:36.914297+0000","last_fullsized":"2024-04-21T08:49:36.914297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:31.373451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.903738+0000","last_change":"2024-04-21T08:48:50.761392+0000","last_active":"2024-04-21T08:49:36.903738+0000","last_peered":"2024-04-21T08:49:36.903738+0000","last_clean":"2024-04-21T08:49:36.903738+0000","last_became_active":"2024-04-21T08:48:50.760917+0000","last_became_peered":"2024-04-21T08:48:50.760917+0000","last_unstale":"2024-04-21T08:49:36.903738+0000","last_undegraded":"2024-04-21T08:49:36.903738+0000","last_fullsized":"2024-04-21T08:49:36.903738+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:19.070502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:50:21.947070+0000","last_change":"2024-04-21T08:48:45.553686+0000","last_active":"2024-04-21T08:50:21.947070+0000","last_peered":"2024-04-21T08:50:21.947070+0000","last_clean":"2024-04-21T08:50:21.947070+0000","last_became_active":"2024-04-21T08:48:45.553322+0000","last_became_peered":"2024-04-21T08:48:45.553322+0000","last_unstale":"2024-04-21T08:50:21.947070+0000","last_undegraded":"2024-04-21T08:50:21.947070+0000","last_fullsized":"2024-04-21T08:50:21.947070+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:15:24.320704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908427+0000","last_change":"2024-04-21T08:48:46.474555+0000","last_active":"2024-04-21T08:49:36.908427+0000","last_peered":"2024-04-21T08:49:36.908427+0000","last_clean":"2024-04-21T08:49:36.908427+0000","last_became_active":"2024-04-21T08:48:46.474185+0000","last_became_peered":"2024-04-21T08:48:46.474185+0000","last_unstale":"2024-04-21T08:49:36.908427+0000","last_undegraded":"2024-04-21T08:49:36.908427+0000","last_fullsized":"2024-04-21T08:49:36.908427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:52.884994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.903229+0000","last_change":"2024-04-21T08:48:50.763181+0000","last_active":"2024-04-21T08:49:36.903229+0000","last_peered":"2024-04-21T08:49:36.903229+0000","last_clean":"2024-04-21T08:49:36.903229+0000","last_became_active":"2024-04-21T08:48:50.762718+0000","last_became_peered":"2024-04-21T08:48:50.762718+0000","last_unstale":"2024-04-21T08:49:36.903229+0000","last_undegraded":"2024-04-21T08:49:36.903229+0000","last_fullsized":"2024-04-21T08:49:36.903229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:16:23.204605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963370+0000","last_change":"2024-04-21T08:48:45.547768+0000","last_active":"2024-04-21T08:49:36.963370+0000","last_peered":"2024-04-21T08:49:36.963370+0000","last_clean":"2024-04-21T08:49:36.963370+0000","last_became_active":"2024-04-21T08:48:45.547456+0000","last_became_peered":"2024-04-21T08:48:45.547456+0000","last_unstale":"2024-04-21T08:49:36.963370+0000","last_undegraded":"2024-04-21T08:49:36.963370+0000","last_fullsized":"2024-04-21T08:49:36.963370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:23.502799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.914274+0000","last_change":"2024-04-21T08:48:46.467988+0000","last_active":"2024-04-21T08:49:36.914274+0000","last_peered":"2024-04-21T08:49:36.914274+0000","last_clean":"2024-04-21T08:49:36.914274+0000","last_became_active":"2024-04-21T08:48:46.465212+0000","last_became_peered":"2024-04-21T08:48:46.465212+0000","last_unstale":"2024-04-21T08:49:36.914274+0000","last_undegraded":"2024-04-21T08:49:36.914274+0000","last_fullsized":"2024-04-21T08:49:36.914274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:41.743506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905456+0000","last_change":"2024-04-21T08:48:50.758289+0000","last_active":"2024-04-21T08:49:36.905456+0000","last_peered":"2024-04-21T08:49:36.905456+0000","last_clean":"2024-04-21T08:49:36.905456+0000","last_became_active":"2024-04-21T08:48:50.757775+0000","last_became_peered":"2024-04-21T08:48:50.757775+0000","last_unstale":"2024-04-21T08:49:36.905456+0000","last_undegraded":"2024-04-21T08:49:36.905456+0000","last_fullsized":"2024-04-21T08:49:36.905456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:56:27.246515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905286+0000","last_change":"2024-04-21T08:48:45.545615+0000","last_active":"2024-04-21T08:49:36.905286+0000","last_peered":"2024-04-21T08:49:36.905286+0000","last_clean":"2024-04-21T08:49:36.905286+0000","last_became_active":"2024-04-21T08:48:45.545280+0000","last_became_peered":"2024-04-21T08:48:45.545280+0000","last_unstale":"2024-04-21T08:49:36.905286+0000","last_undegraded":"2024-04-21T08:49:36.905286+0000","last_fullsized":"2024-04-21T08:49:36.905286+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:50:00.470903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.962853+0000","last_change":"2024-04-21T08:48:46.477326+0000","last_active":"2024-04-21T08:49:36.962853+0000","last_peered":"2024-04-21T08:49:36.962853+0000","last_clean":"2024-04-21T08:49:36.962853+0000","last_became_active":"2024-04-21T08:48:46.476914+0000","last_became_peered":"2024-04-21T08:48:46.476914+0000","last_unstale":"2024-04-21T08:49:36.962853+0000","last_undegraded":"2024-04-21T08:49:36.962853+0000","last_fullsized":"2024-04-21T08:49:36.962853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:04.312061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906018+0000","last_change":"2024-04-21T08:48:50.767429+0000","last_active":"2024-04-21T08:49:36.906018+0000","last_peered":"2024-04-21T08:49:36.906018+0000","last_clean":"2024-04-21T08:49:36.906018+0000","last_became_active":"2024-04-21T08:48:50.767008+0000","last_became_peered":"2024-04-21T08:48:50.767008+0000","last_unstale":"2024-04-21T08:49:36.906018+0000","last_undegraded":"2024-04-21T08:49:36.906018+0000","last_fullsized":"2024-04-21T08:49:36.906018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:26.865571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913761+0000","last_change":"2024-04-21T08:48:45.553170+0000","last_active":"2024-04-21T08:49:36.913761+0000","last_peered":"2024-04-21T08:49:36.913761+0000","last_clean":"2024-04-21T08:49:36.913761+0000","last_became_active":"2024-04-21T08:48:45.552645+0000","last_became_peered":"2024-04-21T08:48:45.552645+0000","last_unstale":"2024-04-21T08:49:36.913761+0000","last_undegraded":"2024-04-21T08:49:36.913761+0000","last_fullsized":"2024-04-21T08:49:36.913761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:32:07.217212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902122+0000","last_change":"2024-04-21T08:48:46.478101+0000","last_active":"2024-04-21T08:49:36.902122+0000","last_peered":"2024-04-21T08:49:36.902122+0000","last_clean":"2024-04-21T08:49:36.902122+0000","last_became_active":"2024-04-21T08:48:46.477725+0000","last_became_peered":"2024-04-21T08:48:46.477725+0000","last_unstale":"2024-04-21T08:49:36.902122+0000","last_undegraded":"2024-04-21T08:49:36.902122+0000","last_fullsized":"2024-04-21T08:49:36.902122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:37:55.429569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963146+0000","last_change":"2024-04-21T08:48:50.762899+0000","last_active":"2024-04-21T08:49:36.963146+0000","last_peered":"2024-04-21T08:49:36.963146+0000","last_clean":"2024-04-21T08:49:36.963146+0000","last_became_active":"2024-04-21T08:48:50.762475+0000","last_became_peered":"2024-04-21T08:48:50.762475+0000","last_unstale":"2024-04-21T08:49:36.963146+0000","last_undegraded":"2024-04-21T08:49:36.963146+0000","last_fullsized":"2024-04-21T08:49:36.963146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:27.540414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":99,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:50:19.943630+0000","last_change":"2024-04-21T08:48:45.550151+0000","last_active":"2024-04-21T08:50:19.943630+0000","last_peered":"2024-04-21T08:50:19.943630+0000","last_clean":"2024-04-21T08:50:19.943630+0000","last_became_active":"2024-04-21T08:48:45.549829+0000","last_became_peered":"2024-04-21T08:48:45.549829+0000","last_unstale":"2024-04-21T08:50:19.943630+0000","last_undegraded":"2024-04-21T08:50:19.943630+0000","last_fullsized":"2024-04-21T08:50:19.943630+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:14.061189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907733+0000","last_change":"2024-04-21T08:48:46.472072+0000","last_active":"2024-04-21T08:49:36.907733+0000","last_peered":"2024-04-21T08:49:36.907733+0000","last_clean":"2024-04-21T08:49:36.907733+0000","last_became_active":"2024-04-21T08:48:46.471660+0000","last_became_peered":"2024-04-21T08:48:46.471660+0000","last_unstale":"2024-04-21T08:49:36.907733+0000","last_undegraded":"2024-04-21T08:49:36.907733+0000","last_fullsized":"2024-04-21T08:49:36.907733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:30:11.843870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902107+0000","last_change":"2024-04-21T08:48:50.773021+0000","last_active":"2024-04-21T08:49:36.902107+0000","last_peered":"2024-04-21T08:49:36.902107+0000","last_clean":"2024-04-21T08:49:36.902107+0000","last_became_active":"2024-04-21T08:48:50.771746+0000","last_became_peered":"2024-04-21T08:48:50.771746+0000","last_unstale":"2024-04-21T08:49:36.902107+0000","last_undegraded":"2024-04-21T08:49:36.902107+0000","last_fullsized":"2024-04-21T08:49:36.902107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:33:21.415967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913020+0000","last_change":"2024-04-21T08:48:45.550376+0000","last_active":"2024-04-21T08:49:36.913020+0000","last_peered":"2024-04-21T08:49:36.913020+0000","last_clean":"2024-04-21T08:49:36.913020+0000","last_became_active":"2024-04-21T08:48:45.549594+0000","last_became_peered":"2024-04-21T08:48:45.549594+0000","last_unstale":"2024-04-21T08:49:36.913020+0000","last_undegraded":"2024-04-21T08:49:36.913020+0000","last_fullsized":"2024-04-21T08:49:36.913020+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:26:27.046963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907060+0000","last_change":"2024-04-21T08:48:46.469499+0000","last_active":"2024-04-21T08:49:36.907060+0000","last_peered":"2024-04-21T08:49:36.907060+0000","last_clean":"2024-04-21T08:49:36.907060+0000","last_became_active":"2024-04-21T08:48:46.469124+0000","last_became_peered":"2024-04-21T08:48:46.469124+0000","last_unstale":"2024-04-21T08:49:36.907060+0000","last_undegraded":"2024-04-21T08:49:36.907060+0000","last_fullsized":"2024-04-21T08:49:36.907060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:58.708916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907631+0000","last_change":"2024-04-21T08:48:50.771331+0000","last_active":"2024-04-21T08:49:36.907631+0000","last_peered":"2024-04-21T08:49:36.907631+0000","last_clean":"2024-04-21T08:49:36.907631+0000","last_became_active":"2024-04-21T08:48:50.770018+0000","last_became_peered":"2024-04-21T08:48:50.770018+0000","last_unstale":"2024-04-21T08:49:36.907631+0000","last_undegraded":"2024-04-21T08:49:36.907631+0000","last_fullsized":"2024-04-21T08:49:36.907631+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:19:03.416423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.914163+0000","last_change":"2024-04-21T08:48:45.559319+0000","last_active":"2024-04-21T08:49:36.914163+0000","last_peered":"2024-04-21T08:49:36.914163+0000","last_clean":"2024-04-21T08:49:36.914163+0000","last_became_active":"2024-04-21T08:48:45.558895+0000","last_became_peered":"2024-04-21T08:48:45.558895+0000","last_unstale":"2024-04-21T08:49:36.914163+0000","last_undegraded":"2024-04-21T08:49:36.914163+0000","last_fullsized":"2024-04-21T08:49:36.914163+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:01.878169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907503+0000","last_change":"2024-04-21T08:48:46.466110+0000","last_active":"2024-04-21T08:49:36.907503+0000","last_peered":"2024-04-21T08:49:36.907503+0000","last_clean":"2024-04-21T08:49:36.907503+0000","last_became_active":"2024-04-21T08:48:46.464374+0000","last_became_peered":"2024-04-21T08:48:46.464374+0000","last_unstale":"2024-04-21T08:49:36.907503+0000","last_undegraded":"2024-04-21T08:49:36.907503+0000","last_fullsized":"2024-04-21T08:49:36.907503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:26.307051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902203+0000","last_change":"2024-04-21T08:48:50.759976+0000","last_active":"2024-04-21T08:49:36.902203+0000","last_peered":"2024-04-21T08:49:36.902203+0000","last_clean":"2024-04-21T08:49:36.902203+0000","last_became_active":"2024-04-21T08:48:50.759596+0000","last_became_peered":"2024-04-21T08:48:50.759596+0000","last_unstale":"2024-04-21T08:49:36.902203+0000","last_undegraded":"2024-04-21T08:49:36.902203+0000","last_fullsized":"2024-04-21T08:49:36.902203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:09.678388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'33","reported_seq":206,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:50:20.945362+0000","last_change":"2024-04-21T08:48:45.549228+0000","last_active":"2024-04-21T08:50:20.945362+0000","last_peered":"2024-04-21T08:50:20.945362+0000","last_clean":"2024-04-21T08:50:20.945362+0000","last_became_active":"2024-04-21T08:48:45.548865+0000","last_became_peered":"2024-04-21T08:48:45.548865+0000","last_unstale":"2024-04-21T08:50:20.945362+0000","last_undegraded":"2024-04-21T08:50:20.945362+0000","last_fullsized":"2024-04-21T08:50:20.945362+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:28.620968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":135,"num_write":34,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908611+0000","last_change":"2024-04-21T08:48:46.470322+0000","last_active":"2024-04-21T08:49:36.908611+0000","last_peered":"2024-04-21T08:49:36.908611+0000","last_clean":"2024-04-21T08:49:36.908611+0000","last_became_active":"2024-04-21T08:48:46.467225+0000","last_became_peered":"2024-04-21T08:48:46.467225+0000","last_unstale":"2024-04-21T08:49:36.908611+0000","last_undegraded":"2024-04-21T08:49:36.908611+0000","last_fullsized":"2024-04-21T08:49:36.908611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:31:17.390926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.914518+0000","last_change":"2024-04-21T08:48:50.770982+0000","last_active":"2024-04-21T08:49:36.914518+0000","last_peered":"2024-04-21T08:49:36.914518+0000","last_clean":"2024-04-21T08:49:36.914518+0000","last_became_active":"2024-04-21T08:48:50.770312+0000","last_became_peered":"2024-04-21T08:48:50.770312+0000","last_unstale":"2024-04-21T08:49:36.914518+0000","last_undegraded":"2024-04-21T08:49:36.914518+0000","last_fullsized":"2024-04-21T08:49:36.914518+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:41:33.027351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.903242+0000","last_change":"2024-04-21T08:48:45.556134+0000","last_active":"2024-04-21T08:49:36.903242+0000","last_peered":"2024-04-21T08:49:36.903242+0000","last_clean":"2024-04-21T08:49:36.903242+0000","last_became_active":"2024-04-21T08:48:45.555818+0000","last_became_peered":"2024-04-21T08:48:45.555818+0000","last_unstale":"2024-04-21T08:49:36.903242+0000","last_undegraded":"2024-04-21T08:49:36.903242+0000","last_fullsized":"2024-04-21T08:49:36.903242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:00:08.083783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913151+0000","last_change":"2024-04-21T08:49:36.913151+0000","last_active":"2024-04-21T08:49:36.913151+0000","last_peered":"2024-04-21T08:49:36.913151+0000","last_clean":"2024-04-21T08:49:36.913151+0000","last_became_active":"2024-04-21T08:49:36.912810+0000","last_became_peered":"2024-04-21T08:49:36.912810+0000","last_unstale":"2024-04-21T08:49:36.913151+0000","last_undegraded":"2024-04-21T08:49:36.913151+0000","last_fullsized":"2024-04-21T08:49:36.913151+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:17:32.887163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905143+0000","last_change":"2024-04-21T08:48:50.765967+0000","last_active":"2024-04-21T08:49:36.905143+0000","last_peered":"2024-04-21T08:49:36.905143+0000","last_clean":"2024-04-21T08:49:36.905143+0000","last_became_active":"2024-04-21T08:48:50.765557+0000","last_became_peered":"2024-04-21T08:48:50.765557+0000","last_unstale":"2024-04-21T08:49:36.905143+0000","last_undegraded":"2024-04-21T08:49:36.905143+0000","last_fullsized":"2024-04-21T08:49:36.905143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:52.044058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913962+0000","last_change":"2024-04-21T08:48:45.559939+0000","last_active":"2024-04-21T08:49:36.913962+0000","last_peered":"2024-04-21T08:49:36.913962+0000","last_clean":"2024-04-21T08:49:36.913962+0000","last_became_active":"2024-04-21T08:48:45.559635+0000","last_became_peered":"2024-04-21T08:48:45.559635+0000","last_unstale":"2024-04-21T08:49:36.913962+0000","last_undegraded":"2024-04-21T08:49:36.913962+0000","last_fullsized":"2024-04-21T08:49:36.913962+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:17.872806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913272+0000","last_change":"2024-04-21T08:48:46.464878+0000","last_active":"2024-04-21T08:49:36.913272+0000","last_peered":"2024-04-21T08:49:36.913272+0000","last_clean":"2024-04-21T08:49:36.913272+0000","last_became_active":"2024-04-21T08:48:46.464552+0000","last_became_peered":"2024-04-21T08:48:46.464552+0000","last_unstale":"2024-04-21T08:49:36.913272+0000","last_undegraded":"2024-04-21T08:49:36.913272+0000","last_fullsized":"2024-04-21T08:49:36.913272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:46:10.558693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902489+0000","last_change":"2024-04-21T08:48:50.765551+0000","last_active":"2024-04-21T08:49:36.902489+0000","last_peered":"2024-04-21T08:49:36.902489+0000","last_clean":"2024-04-21T08:49:36.902489+0000","last_became_active":"2024-04-21T08:48:50.765016+0000","last_became_peered":"2024-04-21T08:48:50.765016+0000","last_unstale":"2024-04-21T08:49:36.902489+0000","last_undegraded":"2024-04-21T08:49:36.902489+0000","last_fullsized":"2024-04-21T08:49:36.902489+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:03:05.317720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:50:17.955319+0000","last_change":"2024-04-21T08:48:45.548685+0000","last_active":"2024-04-21T08:50:17.955319+0000","last_peered":"2024-04-21T08:50:17.955319+0000","last_clean":"2024-04-21T08:50:17.955319+0000","last_became_active":"2024-04-21T08:48:45.548372+0000","last_became_peered":"2024-04-21T08:48:45.548372+0000","last_unstale":"2024-04-21T08:50:17.955319+0000","last_undegraded":"2024-04-21T08:50:17.955319+0000","last_fullsized":"2024-04-21T08:50:17.955319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:44.780297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.912275+0000","last_change":"2024-04-21T08:48:46.471541+0000","last_active":"2024-04-21T08:49:36.912275+0000","last_peered":"2024-04-21T08:49:36.912275+0000","last_clean":"2024-04-21T08:49:36.912275+0000","last_became_active":"2024-04-21T08:48:46.471223+0000","last_became_peered":"2024-04-21T08:48:46.471223+0000","last_unstale":"2024-04-21T08:49:36.912275+0000","last_undegraded":"2024-04-21T08:49:36.912275+0000","last_fullsized":"2024-04-21T08:49:36.912275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:24:50.867905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906371+0000","last_change":"2024-04-21T08:48:50.773941+0000","last_active":"2024-04-21T08:49:36.906371+0000","last_peered":"2024-04-21T08:49:36.906371+0000","last_clean":"2024-04-21T08:49:36.906371+0000","last_became_active":"2024-04-21T08:48:50.772081+0000","last_became_peered":"2024-04-21T08:48:50.772081+0000","last_unstale":"2024-04-21T08:49:36.906371+0000","last_undegraded":"2024-04-21T08:49:36.906371+0000","last_fullsized":"2024-04-21T08:49:36.906371+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:04.438944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.923567+0000","last_change":"2024-04-21T08:49:36.923090+0000","last_active":"2024-04-21T08:49:36.923567+0000","last_peered":"2024-04-21T08:49:36.923567+0000","last_clean":"2024-04-21T08:49:36.923567+0000","last_became_active":"2024-04-21T08:49:36.908075+0000","last_became_peered":"2024-04-21T08:49:36.908075+0000","last_unstale":"2024-04-21T08:49:36.923567+0000","last_undegraded":"2024-04-21T08:49:36.923567+0000","last_fullsized":"2024-04-21T08:49:36.923567+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:41.683372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913956+0000","last_change":"2024-04-21T08:48:46.465559+0000","last_active":"2024-04-21T08:49:36.913956+0000","last_peered":"2024-04-21T08:49:36.913956+0000","last_clean":"2024-04-21T08:49:36.913956+0000","last_became_active":"2024-04-21T08:48:46.465125+0000","last_became_peered":"2024-04-21T08:48:46.465125+0000","last_unstale":"2024-04-21T08:49:36.913956+0000","last_undegraded":"2024-04-21T08:49:36.913956+0000","last_fullsized":"2024-04-21T08:49:36.913956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:52:38.423925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913816+0000","last_change":"2024-04-21T08:48:50.763609+0000","last_active":"2024-04-21T08:49:36.913816+0000","last_peered":"2024-04-21T08:49:36.913816+0000","last_clean":"2024-04-21T08:49:36.913816+0000","last_became_active":"2024-04-21T08:48:50.763144+0000","last_became_peered":"2024-04-21T08:48:50.763144+0000","last_unstale":"2024-04-21T08:49:36.913816+0000","last_undegraded":"2024-04-21T08:49:36.913816+0000","last_fullsized":"2024-04-21T08:49:36.913816+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:26:18.611235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.909318+0000","last_change":"2024-04-21T08:48:45.549912+0000","last_active":"2024-04-21T08:49:36.909318+0000","last_peered":"2024-04-21T08:49:36.909318+0000","last_clean":"2024-04-21T08:49:36.909318+0000","last_became_active":"2024-04-21T08:48:45.549585+0000","last_became_peered":"2024-04-21T08:48:45.549585+0000","last_unstale":"2024-04-21T08:49:36.909318+0000","last_undegraded":"2024-04-21T08:49:36.909318+0000","last_fullsized":"2024-04-21T08:49:36.909318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:58.287402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905374+0000","last_change":"2024-04-21T08:48:46.462746+0000","last_active":"2024-04-21T08:49:36.905374+0000","last_peered":"2024-04-21T08:49:36.905374+0000","last_clean":"2024-04-21T08:49:36.905374+0000","last_became_active":"2024-04-21T08:48:46.461602+0000","last_became_peered":"2024-04-21T08:48:46.461602+0000","last_unstale":"2024-04-21T08:49:36.905374+0000","last_undegraded":"2024-04-21T08:49:36.905374+0000","last_fullsized":"2024-04-21T08:49:36.905374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:52.198634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908295+0000","last_change":"2024-04-21T08:48:50.770770+0000","last_active":"2024-04-21T08:49:36.908295+0000","last_peered":"2024-04-21T08:49:36.908295+0000","last_clean":"2024-04-21T08:49:36.908295+0000","last_became_active":"2024-04-21T08:48:50.770326+0000","last_became_peered":"2024-04-21T08:48:50.770326+0000","last_unstale":"2024-04-21T08:49:36.908295+0000","last_undegraded":"2024-04-21T08:49:36.908295+0000","last_fullsized":"2024-04-21T08:49:36.908295+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:38.325155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902433+0000","last_change":"2024-04-21T08:48:45.558867+0000","last_active":"2024-04-21T08:49:36.902433+0000","last_peered":"2024-04-21T08:49:36.902433+0000","last_clean":"2024-04-21T08:49:36.902433+0000","last_became_active":"2024-04-21T08:48:45.558492+0000","last_became_peered":"2024-04-21T08:48:45.558492+0000","last_unstale":"2024-04-21T08:49:36.902433+0000","last_undegraded":"2024-04-21T08:49:36.902433+0000","last_fullsized":"2024-04-21T08:49:36.902433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:44:55.156404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963608+0000","last_change":"2024-04-21T08:48:46.466461+0000","last_active":"2024-04-21T08:49:36.963608+0000","last_peered":"2024-04-21T08:49:36.963608+0000","last_clean":"2024-04-21T08:49:36.963608+0000","last_became_active":"2024-04-21T08:48:46.466035+0000","last_became_peered":"2024-04-21T08:48:46.466035+0000","last_unstale":"2024-04-21T08:49:36.963608+0000","last_undegraded":"2024-04-21T08:49:36.963608+0000","last_fullsized":"2024-04-21T08:49:36.963608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:47:32.307753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905716+0000","last_change":"2024-04-21T08:48:50.767655+0000","last_active":"2024-04-21T08:49:36.905716+0000","last_peered":"2024-04-21T08:49:36.905716+0000","last_clean":"2024-04-21T08:49:36.905716+0000","last_became_active":"2024-04-21T08:48:50.767305+0000","last_became_peered":"2024-04-21T08:48:50.767305+0000","last_unstale":"2024-04-21T08:49:36.905716+0000","last_undegraded":"2024-04-21T08:49:36.905716+0000","last_fullsized":"2024-04-21T08:49:36.905716+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:31.210126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.912771+0000","last_change":"2024-04-21T08:48:45.552952+0000","last_active":"2024-04-21T08:49:36.912771+0000","last_peered":"2024-04-21T08:49:36.912771+0000","last_clean":"2024-04-21T08:49:36.912771+0000","last_became_active":"2024-04-21T08:48:45.552556+0000","last_became_peered":"2024-04-21T08:48:45.552556+0000","last_unstale":"2024-04-21T08:49:36.912771+0000","last_undegraded":"2024-04-21T08:49:36.912771+0000","last_fullsized":"2024-04-21T08:49:36.912771+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:03.428801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963460+0000","last_change":"2024-04-21T08:48:46.469020+0000","last_active":"2024-04-21T08:49:36.963460+0000","last_peered":"2024-04-21T08:49:36.963460+0000","last_clean":"2024-04-21T08:49:36.963460+0000","last_became_active":"2024-04-21T08:48:46.468631+0000","last_became_peered":"2024-04-21T08:48:46.468631+0000","last_unstale":"2024-04-21T08:49:36.963460+0000","last_undegraded":"2024-04-21T08:49:36.963460+0000","last_fullsized":"2024-04-21T08:49:36.963460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:08.559516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906317+0000","last_change":"2024-04-21T08:48:50.773942+0000","last_active":"2024-04-21T08:49:36.906317+0000","last_peered":"2024-04-21T08:49:36.906317+0000","last_clean":"2024-04-21T08:49:36.906317+0000","last_became_active":"2024-04-21T08:48:50.771876+0000","last_became_peered":"2024-04-21T08:48:50.771876+0000","last_unstale":"2024-04-21T08:49:36.906317+0000","last_undegraded":"2024-04-21T08:49:36.906317+0000","last_fullsized":"2024-04-21T08:49:36.906317+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:19.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.914757+0000","last_change":"2024-04-21T08:48:46.462727+0000","last_active":"2024-04-21T08:49:36.914757+0000","last_peered":"2024-04-21T08:49:36.914757+0000","last_clean":"2024-04-21T08:49:36.914757+0000","last_became_active":"2024-04-21T08:48:46.461669+0000","last_became_peered":"2024-04-21T08:48:46.461669+0000","last_unstale":"2024-04-21T08:49:36.914757+0000","last_undegraded":"2024-04-21T08:49:36.914757+0000","last_fullsized":"2024-04-21T08:49:36.914757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:27.290790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":98,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:50:17.954548+0000","last_change":"2024-04-21T08:48:45.558541+0000","last_active":"2024-04-21T08:50:17.954548+0000","last_peered":"2024-04-21T08:50:17.954548+0000","last_clean":"2024-04-21T08:50:17.954548+0000","last_became_active":"2024-04-21T08:48:45.558186+0000","last_became_peered":"2024-04-21T08:48:45.558186+0000","last_unstale":"2024-04-21T08:50:17.954548+0000","last_undegraded":"2024-04-21T08:50:17.954548+0000","last_fullsized":"2024-04-21T08:50:17.954548+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:02.919144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":67,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913013+0000","last_change":"2024-04-21T08:48:50.770838+0000","last_active":"2024-04-21T08:49:36.913013+0000","last_peered":"2024-04-21T08:49:36.913013+0000","last_clean":"2024-04-21T08:49:36.913013+0000","last_became_active":"2024-04-21T08:48:50.770447+0000","last_became_peered":"2024-04-21T08:48:50.770447+0000","last_unstale":"2024-04-21T08:49:36.913013+0000","last_undegraded":"2024-04-21T08:49:36.913013+0000","last_fullsized":"2024-04-21T08:49:36.913013+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:48:21.010284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.962936+0000","last_change":"2024-04-21T08:48:46.478145+0000","last_active":"2024-04-21T08:49:36.962936+0000","last_peered":"2024-04-21T08:49:36.962936+0000","last_clean":"2024-04-21T08:49:36.962936+0000","last_became_active":"2024-04-21T08:48:46.477813+0000","last_became_peered":"2024-04-21T08:48:46.477813+0000","last_unstale":"2024-04-21T08:49:36.962936+0000","last_undegraded":"2024-04-21T08:49:36.962936+0000","last_fullsized":"2024-04-21T08:49:36.962936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:15.917619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907595+0000","last_change":"2024-04-21T08:48:45.545269+0000","last_active":"2024-04-21T08:49:36.907595+0000","last_peered":"2024-04-21T08:49:36.907595+0000","last_clean":"2024-04-21T08:49:36.907595+0000","last_became_active":"2024-04-21T08:48:45.544887+0000","last_became_peered":"2024-04-21T08:48:45.544887+0000","last_unstale":"2024-04-21T08:49:36.907595+0000","last_undegraded":"2024-04-21T08:49:36.907595+0000","last_fullsized":"2024-04-21T08:49:36.907595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:27.222570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.912256+0000","last_change":"2024-04-21T08:47:34.198161+0000","last_active":"2024-04-21T08:49:36.912256+0000","last_peered":"2024-04-21T08:49:36.912256+0000","last_clean":"2024-04-21T08:49:36.912256+0000","last_became_active":"2024-04-21T08:47:34.197654+0000","last_became_peered":"2024-04-21T08:47:34.197654+0000","last_unstale":"2024-04-21T08:49:36.912256+0000","last_undegraded":"2024-04-21T08:49:36.912256+0000","last_fullsized":"2024-04-21T08:49:36.912256+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_clean_scrub_stamp":"2024-04-21T08:44:05.493464+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:58:58.295783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.914175+0000","last_change":"2024-04-21T08:48:50.770826+0000","last_active":"2024-04-21T08:49:36.914175+0000","last_peered":"2024-04-21T08:49:36.914175+0000","last_clean":"2024-04-21T08:49:36.914175+0000","last_became_active":"2024-04-21T08:48:50.770372+0000","last_became_peered":"2024-04-21T08:48:50.770372+0000","last_unstale":"2024-04-21T08:49:36.914175+0000","last_undegraded":"2024-04-21T08:49:36.914175+0000","last_fullsized":"2024-04-21T08:49:36.914175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:59.637874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905914+0000","last_change":"2024-04-21T08:48:46.459330+0000","last_active":"2024-04-21T08:49:36.905914+0000","last_peered":"2024-04-21T08:49:36.905914+0000","last_clean":"2024-04-21T08:49:36.905914+0000","last_became_active":"2024-04-21T08:48:46.458963+0000","last_became_peered":"2024-04-21T08:48:46.458963+0000","last_unstale":"2024-04-21T08:49:36.905914+0000","last_undegraded":"2024-04-21T08:49:36.905914+0000","last_fullsized":"2024-04-21T08:49:36.905914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:10.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.917854+0000","last_change":"2024-04-21T08:49:36.917854+0000","last_active":"2024-04-21T08:49:36.917854+0000","last_peered":"2024-04-21T08:49:36.917854+0000","last_clean":"2024-04-21T08:49:36.917854+0000","last_became_active":"2024-04-21T08:49:36.915390+0000","last_became_peered":"2024-04-21T08:49:36.915390+0000","last_unstale":"2024-04-21T08:49:36.917854+0000","last_undegraded":"2024-04-21T08:49:36.917854+0000","last_fullsized":"2024-04-21T08:49:36.917854+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:45.920752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905081+0000","last_change":"2024-04-21T08:48:50.765339+0000","last_active":"2024-04-21T08:49:36.905081+0000","last_peered":"2024-04-21T08:49:36.905081+0000","last_clean":"2024-04-21T08:49:36.905081+0000","last_became_active":"2024-04-21T08:48:50.764858+0000","last_became_peered":"2024-04-21T08:48:50.764858+0000","last_unstale":"2024-04-21T08:49:36.905081+0000","last_undegraded":"2024-04-21T08:49:36.905081+0000","last_fullsized":"2024-04-21T08:49:36.905081+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:26.075041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908721+0000","last_change":"2024-04-21T08:48:45.557768+0000","last_active":"2024-04-21T08:49:36.908721+0000","last_peered":"2024-04-21T08:49:36.908721+0000","last_clean":"2024-04-21T08:49:36.908721+0000","last_became_active":"2024-04-21T08:48:45.557448+0000","last_became_peered":"2024-04-21T08:48:45.557448+0000","last_unstale":"2024-04-21T08:49:36.908721+0000","last_undegraded":"2024-04-21T08:49:36.908721+0000","last_fullsized":"2024-04-21T08:49:36.908721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:09.063913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906011+0000","last_change":"2024-04-21T08:48:46.459425+0000","last_active":"2024-04-21T08:49:36.906011+0000","last_peered":"2024-04-21T08:49:36.906011+0000","last_clean":"2024-04-21T08:49:36.906011+0000","last_became_active":"2024-04-21T08:48:46.459020+0000","last_became_peered":"2024-04-21T08:48:46.459020+0000","last_unstale":"2024-04-21T08:49:36.906011+0000","last_undegraded":"2024-04-21T08:49:36.906011+0000","last_fullsized":"2024-04-21T08:49:36.906011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:08:39.042825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.962847+0000","last_change":"2024-04-21T08:48:50.765740+0000","last_active":"2024-04-21T08:49:36.962847+0000","last_peered":"2024-04-21T08:49:36.962847+0000","last_clean":"2024-04-21T08:49:36.962847+0000","last_became_active":"2024-04-21T08:48:50.765359+0000","last_became_peered":"2024-04-21T08:48:50.765359+0000","last_unstale":"2024-04-21T08:49:36.962847+0000","last_undegraded":"2024-04-21T08:49:36.962847+0000","last_fullsized":"2024-04-21T08:49:36.962847+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:34:42.716165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902656+0000","last_change":"2024-04-21T08:48:45.564532+0000","last_active":"2024-04-21T08:49:36.902656+0000","last_peered":"2024-04-21T08:49:36.902656+0000","last_clean":"2024-04-21T08:49:36.902656+0000","last_became_active":"2024-04-21T08:48:45.564225+0000","last_became_peered":"2024-04-21T08:48:45.564225+0000","last_unstale":"2024-04-21T08:49:36.902656+0000","last_undegraded":"2024-04-21T08:49:36.902656+0000","last_fullsized":"2024-04-21T08:49:36.902656+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:42.850371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963423+0000","last_change":"2024-04-21T08:48:46.466622+0000","last_active":"2024-04-21T08:49:36.963423+0000","last_peered":"2024-04-21T08:49:36.963423+0000","last_clean":"2024-04-21T08:49:36.963423+0000","last_became_active":"2024-04-21T08:48:46.466129+0000","last_became_peered":"2024-04-21T08:48:46.466129+0000","last_unstale":"2024-04-21T08:49:36.963423+0000","last_undegraded":"2024-04-21T08:49:36.963423+0000","last_fullsized":"2024-04-21T08:49:36.963423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:20:26.064193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902852+0000","last_change":"2024-04-21T08:48:50.759534+0000","last_active":"2024-04-21T08:49:36.902852+0000","last_peered":"2024-04-21T08:49:36.902852+0000","last_clean":"2024-04-21T08:49:36.902852+0000","last_became_active":"2024-04-21T08:48:50.759116+0000","last_became_peered":"2024-04-21T08:48:50.759116+0000","last_unstale":"2024-04-21T08:49:36.902852+0000","last_undegraded":"2024-04-21T08:49:36.902852+0000","last_fullsized":"2024-04-21T08:49:36.902852+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:02:16.200901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907082+0000","last_change":"2024-04-21T08:48:45.545773+0000","last_active":"2024-04-21T08:49:36.907082+0000","last_peered":"2024-04-21T08:49:36.907082+0000","last_clean":"2024-04-21T08:49:36.907082+0000","last_became_active":"2024-04-21T08:48:45.545447+0000","last_became_peered":"2024-04-21T08:48:45.545447+0000","last_unstale":"2024-04-21T08:49:36.907082+0000","last_undegraded":"2024-04-21T08:49:36.907082+0000","last_fullsized":"2024-04-21T08:49:36.907082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:59:42.171602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.964411+0000","last_change":"2024-04-21T08:49:36.964411+0000","last_active":"2024-04-21T08:49:36.964411+0000","last_peered":"2024-04-21T08:49:36.964411+0000","last_clean":"2024-04-21T08:49:36.964411+0000","last_became_active":"2024-04-21T08:49:36.964046+0000","last_became_peered":"2024-04-21T08:49:36.964046+0000","last_unstale":"2024-04-21T08:49:36.964411+0000","last_undegraded":"2024-04-21T08:49:36.964411+0000","last_fullsized":"2024-04-21T08:49:36.964411+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:29:37.223495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906935+0000","last_change":"2024-04-21T08:48:50.771085+0000","last_active":"2024-04-21T08:49:36.906935+0000","last_peered":"2024-04-21T08:49:36.906935+0000","last_clean":"2024-04-21T08:49:36.906935+0000","last_became_active":"2024-04-21T08:48:50.770650+0000","last_became_peered":"2024-04-21T08:48:50.770650+0000","last_unstale":"2024-04-21T08:49:36.906935+0000","last_undegraded":"2024-04-21T08:49:36.906935+0000","last_fullsized":"2024-04-21T08:49:36.906935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:43:25.933231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.928993+0000","last_change":"2024-04-21T08:49:36.928993+0000","last_active":"2024-04-21T08:49:36.928993+0000","last_peered":"2024-04-21T08:49:36.928993+0000","last_clean":"2024-04-21T08:49:36.928993+0000","last_became_active":"2024-04-21T08:49:36.911247+0000","last_became_peered":"2024-04-21T08:49:36.911247+0000","last_unstale":"2024-04-21T08:49:36.928993+0000","last_undegraded":"2024-04-21T08:49:36.928993+0000","last_fullsized":"2024-04-21T08:49:36.928993+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:13.481670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908188+0000","last_change":"2024-04-21T08:48:46.465475+0000","last_active":"2024-04-21T08:49:36.908188+0000","last_peered":"2024-04-21T08:49:36.908188+0000","last_clean":"2024-04-21T08:49:36.908188+0000","last_became_active":"2024-04-21T08:48:46.465138+0000","last_became_peered":"2024-04-21T08:48:46.465138+0000","last_unstale":"2024-04-21T08:49:36.908188+0000","last_undegraded":"2024-04-21T08:49:36.908188+0000","last_fullsized":"2024-04-21T08:49:36.908188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:12:53.521547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902635+0000","last_change":"2024-04-21T08:48:50.765765+0000","last_active":"2024-04-21T08:49:36.902635+0000","last_peered":"2024-04-21T08:49:36.902635+0000","last_clean":"2024-04-21T08:49:36.902635+0000","last_became_active":"2024-04-21T08:48:50.765324+0000","last_became_peered":"2024-04-21T08:48:50.765324+0000","last_unstale":"2024-04-21T08:49:36.902635+0000","last_undegraded":"2024-04-21T08:49:36.902635+0000","last_fullsized":"2024-04-21T08:49:36.902635+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:41:37.846768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.962923+0000","last_change":"2024-04-21T08:48:45.557400+0000","last_active":"2024-04-21T08:49:36.962923+0000","last_peered":"2024-04-21T08:49:36.962923+0000","last_clean":"2024-04-21T08:49:36.962923+0000","last_became_active":"2024-04-21T08:48:45.555012+0000","last_became_peered":"2024-04-21T08:48:45.555012+0000","last_unstale":"2024-04-21T08:49:36.962923+0000","last_undegraded":"2024-04-21T08:49:36.962923+0000","last_fullsized":"2024-04-21T08:49:36.962923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:00:31.254353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905661+0000","last_change":"2024-04-21T08:48:46.470042+0000","last_active":"2024-04-21T08:49:36.905661+0000","last_peered":"2024-04-21T08:49:36.905661+0000","last_clean":"2024-04-21T08:49:36.905661+0000","last_became_active":"2024-04-21T08:48:46.469716+0000","last_became_peered":"2024-04-21T08:48:46.469716+0000","last_unstale":"2024-04-21T08:49:36.905661+0000","last_undegraded":"2024-04-21T08:49:36.905661+0000","last_fullsized":"2024-04-21T08:49:36.905661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:09.934751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908352+0000","last_change":"2024-04-21T08:48:50.770889+0000","last_active":"2024-04-21T08:49:36.908352+0000","last_peered":"2024-04-21T08:49:36.908352+0000","last_clean":"2024-04-21T08:49:36.908352+0000","last_became_active":"2024-04-21T08:48:50.769980+0000","last_became_peered":"2024-04-21T08:48:50.769980+0000","last_unstale":"2024-04-21T08:49:36.908352+0000","last_undegraded":"2024-04-21T08:49:36.908352+0000","last_fullsized":"2024-04-21T08:49:36.908352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:44.631633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902970+0000","last_change":"2024-04-21T08:48:45.555479+0000","last_active":"2024-04-21T08:49:36.902970+0000","last_peered":"2024-04-21T08:49:36.902970+0000","last_clean":"2024-04-21T08:49:36.902970+0000","last_became_active":"2024-04-21T08:48:45.555162+0000","last_became_peered":"2024-04-21T08:48:45.555162+0000","last_unstale":"2024-04-21T08:49:36.902970+0000","last_undegraded":"2024-04-21T08:49:36.902970+0000","last_fullsized":"2024-04-21T08:49:36.902970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:15:20.262798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963322+0000","last_change":"2024-04-21T08:48:46.467962+0000","last_active":"2024-04-21T08:49:36.963322+0000","last_peered":"2024-04-21T08:49:36.963322+0000","last_clean":"2024-04-21T08:49:36.963322+0000","last_became_active":"2024-04-21T08:48:46.467561+0000","last_became_peered":"2024-04-21T08:48:46.467561+0000","last_unstale":"2024-04-21T08:49:36.963322+0000","last_undegraded":"2024-04-21T08:49:36.963322+0000","last_fullsized":"2024-04-21T08:49:36.963322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:18.262009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906737+0000","last_change":"2024-04-21T08:48:50.767678+0000","last_active":"2024-04-21T08:49:36.906737+0000","last_peered":"2024-04-21T08:49:36.906737+0000","last_clean":"2024-04-21T08:49:36.906737+0000","last_became_active":"2024-04-21T08:48:50.767309+0000","last_became_peered":"2024-04-21T08:48:50.767309+0000","last_unstale":"2024-04-21T08:49:36.906737+0000","last_undegraded":"2024-04-21T08:49:36.906737+0000","last_fullsized":"2024-04-21T08:49:36.906737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:16:40.065043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908504+0000","last_change":"2024-04-21T08:48:45.553321+0000","last_active":"2024-04-21T08:49:36.908504+0000","last_peered":"2024-04-21T08:49:36.908504+0000","last_clean":"2024-04-21T08:49:36.908504+0000","last_became_active":"2024-04-21T08:48:45.552871+0000","last_became_peered":"2024-04-21T08:48:45.552871+0000","last_unstale":"2024-04-21T08:49:36.908504+0000","last_undegraded":"2024-04-21T08:49:36.908504+0000","last_fullsized":"2024-04-21T08:49:36.908504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:16:27.339801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.912942+0000","last_change":"2024-04-21T08:48:46.473453+0000","last_active":"2024-04-21T08:49:36.912942+0000","last_peered":"2024-04-21T08:49:36.912942+0000","last_clean":"2024-04-21T08:49:36.912942+0000","last_became_active":"2024-04-21T08:48:46.472613+0000","last_became_peered":"2024-04-21T08:48:46.472613+0000","last_unstale":"2024-04-21T08:49:36.912942+0000","last_undegraded":"2024-04-21T08:49:36.912942+0000","last_fullsized":"2024-04-21T08:49:36.912942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:43:58.995997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906527+0000","last_change":"2024-04-21T08:48:50.765282+0000","last_active":"2024-04-21T08:49:36.906527+0000","last_peered":"2024-04-21T08:49:36.906527+0000","last_clean":"2024-04-21T08:49:36.906527+0000","last_became_active":"2024-04-21T08:48:50.764891+0000","last_became_peered":"2024-04-21T08:48:50.764891+0000","last_unstale":"2024-04-21T08:49:36.906527+0000","last_undegraded":"2024-04-21T08:49:36.906527+0000","last_fullsized":"2024-04-21T08:49:36.906527+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:18:40.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.927020+0000","last_change":"2024-04-21T08:49:36.926642+0000","last_active":"2024-04-21T08:49:36.927020+0000","last_peered":"2024-04-21T08:49:36.927020+0000","last_clean":"2024-04-21T08:49:36.927020+0000","last_became_active":"2024-04-21T08:49:36.910453+0000","last_became_peered":"2024-04-21T08:49:36.910453+0000","last_unstale":"2024-04-21T08:49:36.927020+0000","last_undegraded":"2024-04-21T08:49:36.927020+0000","last_fullsized":"2024-04-21T08:49:36.927020+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:19.290010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913362+0000","last_change":"2024-04-21T08:48:46.470058+0000","last_active":"2024-04-21T08:49:36.913362+0000","last_peered":"2024-04-21T08:49:36.913362+0000","last_clean":"2024-04-21T08:49:36.913362+0000","last_became_active":"2024-04-21T08:48:46.469485+0000","last_became_peered":"2024-04-21T08:48:46.469485+0000","last_unstale":"2024-04-21T08:49:36.913362+0000","last_undegraded":"2024-04-21T08:49:36.913362+0000","last_fullsized":"2024-04-21T08:49:36.913362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:33:13.212087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905265+0000","last_change":"2024-04-21T08:48:50.762944+0000","last_active":"2024-04-21T08:49:36.905265+0000","last_peered":"2024-04-21T08:49:36.905265+0000","last_clean":"2024-04-21T08:49:36.905265+0000","last_became_active":"2024-04-21T08:48:50.762508+0000","last_became_peered":"2024-04-21T08:48:50.762508+0000","last_unstale":"2024-04-21T08:49:36.905265+0000","last_undegraded":"2024-04-21T08:49:36.905265+0000","last_fullsized":"2024-04-21T08:49:36.905265+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:35.393339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.914358+0000","last_change":"2024-04-21T08:48:45.560361+0000","last_active":"2024-04-21T08:49:36.914358+0000","last_peered":"2024-04-21T08:49:36.914358+0000","last_clean":"2024-04-21T08:49:36.914358+0000","last_became_active":"2024-04-21T08:48:45.559946+0000","last_became_peered":"2024-04-21T08:48:45.559946+0000","last_unstale":"2024-04-21T08:49:36.914358+0000","last_undegraded":"2024-04-21T08:49:36.914358+0000","last_fullsized":"2024-04-21T08:49:36.914358+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:55.599714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.912765+0000","last_change":"2024-04-21T08:48:46.472706+0000","last_active":"2024-04-21T08:49:36.912765+0000","last_peered":"2024-04-21T08:49:36.912765+0000","last_clean":"2024-04-21T08:49:36.912765+0000","last_became_active":"2024-04-21T08:48:46.472378+0000","last_became_peered":"2024-04-21T08:48:46.472378+0000","last_unstale":"2024-04-21T08:49:36.912765+0000","last_undegraded":"2024-04-21T08:49:36.912765+0000","last_fullsized":"2024-04-21T08:49:36.912765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:58.228899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913521+0000","last_change":"2024-04-21T08:48:50.760610+0000","last_active":"2024-04-21T08:49:36.913521+0000","last_peered":"2024-04-21T08:49:36.913521+0000","last_clean":"2024-04-21T08:49:36.913521+0000","last_became_active":"2024-04-21T08:48:50.759991+0000","last_became_peered":"2024-04-21T08:48:50.759991+0000","last_unstale":"2024-04-21T08:49:36.913521+0000","last_undegraded":"2024-04-21T08:49:36.913521+0000","last_fullsized":"2024-04-21T08:49:36.913521+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:43.649701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906406+0000","last_change":"2024-04-21T08:48:45.560773+0000","last_active":"2024-04-21T08:49:36.906406+0000","last_peered":"2024-04-21T08:49:36.906406+0000","last_clean":"2024-04-21T08:49:36.906406+0000","last_became_active":"2024-04-21T08:48:45.560431+0000","last_became_peered":"2024-04-21T08:48:45.560431+0000","last_unstale":"2024-04-21T08:49:36.906406+0000","last_undegraded":"2024-04-21T08:49:36.906406+0000","last_fullsized":"2024-04-21T08:49:36.906406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:07.858838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913224+0000","last_change":"2024-04-21T08:48:46.475983+0000","last_active":"2024-04-21T08:49:36.913224+0000","last_peered":"2024-04-21T08:49:36.913224+0000","last_clean":"2024-04-21T08:49:36.913224+0000","last_became_active":"2024-04-21T08:48:46.475667+0000","last_became_peered":"2024-04-21T08:48:46.475667+0000","last_unstale":"2024-04-21T08:49:36.913224+0000","last_undegraded":"2024-04-21T08:49:36.913224+0000","last_fullsized":"2024-04-21T08:49:36.913224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:28:41.013111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905739+0000","last_change":"2024-04-21T08:48:46.464675+0000","last_active":"2024-04-21T08:49:36.905739+0000","last_peered":"2024-04-21T08:49:36.905739+0000","last_clean":"2024-04-21T08:49:36.905739+0000","last_became_active":"2024-04-21T08:48:46.464314+0000","last_became_peered":"2024-04-21T08:48:46.464314+0000","last_unstale":"2024-04-21T08:49:36.905739+0000","last_undegraded":"2024-04-21T08:49:36.905739+0000","last_fullsized":"2024-04-21T08:49:36.905739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:53.048400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.911025+0000","last_change":"2024-04-21T08:49:36.911025+0000","last_active":"2024-04-21T08:49:36.911025+0000","last_peered":"2024-04-21T08:49:36.911025+0000","last_clean":"2024-04-21T08:49:36.911025+0000","last_became_active":"2024-04-21T08:49:36.909942+0000","last_became_peered":"2024-04-21T08:49:36.909942+0000","last_unstale":"2024-04-21T08:49:36.911025+0000","last_undegraded":"2024-04-21T08:49:36.911025+0000","last_fullsized":"2024-04-21T08:49:36.911025+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:39.864253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902957+0000","last_change":"2024-04-21T08:48:46.467867+0000","last_active":"2024-04-21T08:49:36.902957+0000","last_peered":"2024-04-21T08:49:36.902957+0000","last_clean":"2024-04-21T08:49:36.902957+0000","last_became_active":"2024-04-21T08:48:46.465917+0000","last_became_peered":"2024-04-21T08:48:46.465917+0000","last_unstale":"2024-04-21T08:49:36.902957+0000","last_undegraded":"2024-04-21T08:49:36.902957+0000","last_fullsized":"2024-04-21T08:49:36.902957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:19:56.824998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905483+0000","last_change":"2024-04-21T08:48:46.467159+0000","last_active":"2024-04-21T08:49:36.905483+0000","last_peered":"2024-04-21T08:49:36.905483+0000","last_clean":"2024-04-21T08:49:36.905483+0000","last_became_active":"2024-04-21T08:48:46.466639+0000","last_became_peered":"2024-04-21T08:48:46.466639+0000","last_unstale":"2024-04-21T08:49:36.905483+0000","last_undegraded":"2024-04-21T08:49:36.905483+0000","last_fullsized":"2024-04-21T08:49:36.905483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:54:44.772535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905904+0000","last_change":"2024-04-21T08:48:46.466403+0000","last_active":"2024-04-21T08:49:36.905904+0000","last_peered":"2024-04-21T08:49:36.905904+0000","last_clean":"2024-04-21T08:49:36.905904+0000","last_became_active":"2024-04-21T08:48:46.466060+0000","last_became_peered":"2024-04-21T08:48:46.466060+0000","last_unstale":"2024-04-21T08:49:36.905904+0000","last_undegraded":"2024-04-21T08:49:36.905904+0000","last_fullsized":"2024-04-21T08:49:36.905904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:14.374193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.909134+0000","last_change":"2024-04-21T08:49:36.909134+0000","last_active":"2024-04-21T08:49:36.909134+0000","last_peered":"2024-04-21T08:49:36.909134+0000","last_clean":"2024-04-21T08:49:36.909134+0000","last_became_active":"2024-04-21T08:49:36.908723+0000","last_became_peered":"2024-04-21T08:49:36.908723+0000","last_unstale":"2024-04-21T08:49:36.909134+0000","last_undegraded":"2024-04-21T08:49:36.909134+0000","last_fullsized":"2024-04-21T08:49:36.909134+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:28.123016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.904044+0000","last_change":"2024-04-21T08:48:46.464098+0000","last_active":"2024-04-21T08:49:36.904044+0000","last_peered":"2024-04-21T08:49:36.904044+0000","last_clean":"2024-04-21T08:49:36.904044+0000","last_became_active":"2024-04-21T08:48:46.463756+0000","last_became_peered":"2024-04-21T08:48:46.463756+0000","last_unstale":"2024-04-21T08:49:36.904044+0000","last_undegraded":"2024-04-21T08:49:36.904044+0000","last_fullsized":"2024-04-21T08:49:36.904044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:31.725155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963509+0000","last_change":"2024-04-21T08:48:46.467372+0000","last_active":"2024-04-21T08:49:36.963509+0000","last_peered":"2024-04-21T08:49:36.963509+0000","last_clean":"2024-04-21T08:49:36.963509+0000","last_became_active":"2024-04-21T08:48:46.467043+0000","last_became_peered":"2024-04-21T08:48:46.467043+0000","last_unstale":"2024-04-21T08:49:36.963509+0000","last_undegraded":"2024-04-21T08:49:36.963509+0000","last_fullsized":"2024-04-21T08:49:36.963509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:41.889994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913827+0000","last_change":"2024-04-21T08:48:46.460604+0000","last_active":"2024-04-21T08:49:36.913827+0000","last_peered":"2024-04-21T08:49:36.913827+0000","last_clean":"2024-04-21T08:49:36.913827+0000","last_became_active":"2024-04-21T08:48:46.460276+0000","last_became_peered":"2024-04-21T08:48:46.460276+0000","last_unstale":"2024-04-21T08:49:36.913827+0000","last_undegraded":"2024-04-21T08:49:36.913827+0000","last_fullsized":"2024-04-21T08:49:36.913827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:30.194808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.962941+0000","last_change":"2024-04-21T08:48:46.463665+0000","last_active":"2024-04-21T08:49:36.962941+0000","last_peered":"2024-04-21T08:49:36.962941+0000","last_clean":"2024-04-21T08:49:36.962941+0000","last_became_active":"2024-04-21T08:48:46.463309+0000","last_became_peered":"2024-04-21T08:48:46.463309+0000","last_unstale":"2024-04-21T08:49:36.962941+0000","last_undegraded":"2024-04-21T08:49:36.962941+0000","last_fullsized":"2024-04-21T08:49:36.962941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:12:36.436565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906947+0000","last_change":"2024-04-21T08:48:46.476845+0000","last_active":"2024-04-21T08:49:36.906947+0000","last_peered":"2024-04-21T08:49:36.906947+0000","last_clean":"2024-04-21T08:49:36.906947+0000","last_became_active":"2024-04-21T08:48:46.476507+0000","last_became_peered":"2024-04-21T08:48:46.476507+0000","last_unstale":"2024-04-21T08:49:36.906947+0000","last_undegraded":"2024-04-21T08:49:36.906947+0000","last_fullsized":"2024-04-21T08:49:36.906947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:47.424576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905401+0000","last_change":"2024-04-21T08:48:46.469316+0000","last_active":"2024-04-21T08:49:36.905401+0000","last_peered":"2024-04-21T08:49:36.905401+0000","last_clean":"2024-04-21T08:49:36.905401+0000","last_became_active":"2024-04-21T08:48:46.468852+0000","last_became_peered":"2024-04-21T08:48:46.468852+0000","last_unstale":"2024-04-21T08:49:36.905401+0000","last_undegraded":"2024-04-21T08:49:36.905401+0000","last_fullsized":"2024-04-21T08:49:36.905401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:05:03.664279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905201+0000","last_change":"2024-04-21T08:48:46.466703+0000","last_active":"2024-04-21T08:49:36.905201+0000","last_peered":"2024-04-21T08:49:36.905201+0000","last_clean":"2024-04-21T08:49:36.905201+0000","last_became_active":"2024-04-21T08:48:46.466395+0000","last_became_peered":"2024-04-21T08:48:46.466395+0000","last_unstale":"2024-04-21T08:49:36.905201+0000","last_undegraded":"2024-04-21T08:49:36.905201+0000","last_fullsized":"2024-04-21T08:49:36.905201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:08:56.956283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":248791,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67357,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":437,"num_read_kb":288,"num_write":161,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":202071,"data_compressed":4833,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383711,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":500872,"data_compressed":20609,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1138688,"data_stored":1151775,"data_compressed":58229,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808873,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1015808,"data_stored":505092,"data_compressed":20609,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070513,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":500804,"data_compressed":20609,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332149,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1060864,"data_stored":1096825,"data_compressed":56618,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":496172,"data_compressed":20609,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":559438,"data_compressed":22220,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":497560,"data_compressed":20609,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1077248,"data_stored":560206,"data_compressed":22220,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":496366,"data_compressed":20609,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":501795,"data_compressed":20609,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1069056,"data_stored":1092022,"data_compressed":56618,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34373,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34373,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63356,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":64124,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65325,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:50:25.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-21T08:50:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:26.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:26 smithi028 ceph-mon[28003]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:50:26.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:26 smithi152 ceph-mon[31401]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:50:26.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:26 smithi171 ceph-mon[31358]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:50:27.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:27 smithi028 ceph-mon[28003]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:27.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:27 smithi152 ceph-mon[31401]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:27.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:27 smithi171 ceph-mon[31358]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:28.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.095+0000 7f3f67677700 1 -- 172.21.15.28:0/3967814551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f6006b730 msgr2=0x7f3f6006bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:28.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.095+0000 7f3f67677700 1 --2- 172.21.15.28:0/3967814551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f6006b730 0x7f3f6006bbb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3f50009990 tx=0x7f3f50009ca0 comp rx=0 tx=0).stop 2024-04-21T08:50:28.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.095+0000 7f3f67677700 1 -- 172.21.15.28:0/3967814551 shutdown_connections 2024-04-21T08:50:28.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.095+0000 7f3f67677700 1 --2- 172.21.15.28:0/3967814551 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f601054b0 0x7f3f60105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:28.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.095+0000 7f3f67677700 1 --2- 172.21.15.28:0/3967814551 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3f601079a0 0x7f3f6006b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:28.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.095+0000 7f3f67677700 1 --2- 172.21.15.28:0/3967814551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f6006b730 0x7f3f6006bbb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:28.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.095+0000 7f3f67677700 1 -- 172.21.15.28:0/3967814551 >> 172.21.15.28:0/3967814551 conn(0x7f3f600789e0 msgr2=0x7f3f60078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:28.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.095+0000 7f3f67677700 1 -- 172.21.15.28:0/3967814551 shutdown_connections 2024-04-21T08:50:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.095+0000 7f3f67677700 1 -- 172.21.15.28:0/3967814551 wait complete. 2024-04-21T08:50:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f67677700 1 Processor -- start 2024-04-21T08:50:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f67677700 1 -- start start 2024-04-21T08:50:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f67677700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f6006b730 0x7f3f601a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f67677700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f601054b0 0x7f3f601a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f67677700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3f601079a0 0x7f3f6019f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f67677700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f601192a0 con 0x7f3f6006b730 2024-04-21T08:50:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f67677700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3f60119120 con 0x7f3f601054b0 2024-04-21T08:50:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f67677700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3f60119420 con 0x7f3f601079a0 2024-04-21T08:50:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f65413700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f6006b730 0x7f3f601a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f65413700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f6006b730 0x7f3f601a53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36704/0 (socket says 172.21.15.28:36704) 2024-04-21T08:50:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f65413700 1 -- 172.21.15.28:0/3480887899 learned_addr learned my addr 172.21.15.28:0/3480887899 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f65c14700 1 --2- 172.21.15.28:0/3480887899 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3f601079a0 0x7f3f6019f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f64c12700 1 --2- 172.21.15.28:0/3480887899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f601054b0 0x7f3f601a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f65413700 1 -- 172.21.15.28:0/3480887899 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3f601079a0 msgr2=0x7f3f6019f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f65413700 1 --2- 172.21.15.28:0/3480887899 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3f601079a0 0x7f3f6019f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f65413700 1 -- 172.21.15.28:0/3480887899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f601054b0 msgr2=0x7f3f601a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f65413700 1 --2- 172.21.15.28:0/3480887899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f601054b0 0x7f3f601a5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f65413700 1 -- 172.21.15.28:0/3480887899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f6019fd60 con 0x7f3f6006b730 2024-04-21T08:50:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.096+0000 7f3f65c14700 1 --2- 172.21.15.28:0/3480887899 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3f601079a0 0x7f3f6019f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.097+0000 7f3f65413700 1 --2- 172.21.15.28:0/3480887899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f6006b730 0x7f3f601a53f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00b530 tx=0x7f3f5c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.097+0000 7f3f567fc700 1 -- 172.21.15.28:0/3480887899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c015020 con 0x7f3f6006b730 2024-04-21T08:50:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.097+0000 7f3f67677700 1 -- 172.21.15.28:0/3480887899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f601a0050 con 0x7f3f6006b730 2024-04-21T08:50:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.097+0000 7f3f567fc700 1 -- 172.21.15.28:0/3480887899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f5c011ce0 con 0x7f3f6006b730 2024-04-21T08:50:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.097+0000 7f3f567fc700 1 -- 172.21.15.28:0/3480887899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c010d20 con 0x7f3f6006b730 2024-04-21T08:50:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.097+0000 7f3f67677700 1 -- 172.21.15.28:0/3480887899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f601ac2c0 con 0x7f3f6006b730 2024-04-21T08:50:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.099+0000 7f3f67677700 1 -- 172.21.15.28:0/3480887899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f60069730 con 0x7f3f6006b730 2024-04-21T08:50:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.099+0000 7f3f567fc700 1 -- 172.21.15.28:0/3480887899 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3f5c018020 con 0x7f3f6006b730 2024-04-21T08:50:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.100+0000 7f3f567fc700 1 --2- 172.21.15.28:0/3480887899 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3f4c0a3cd0 0x7f3f4c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.100+0000 7f3f64c12700 1 --2- 172.21.15.28:0/3480887899 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3f4c0a3cd0 0x7f3f4c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.100+0000 7f3f567fc700 1 -- 172.21.15.28:0/3480887899 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f3f5c0d0000 con 0x7f3f6006b730 2024-04-21T08:50:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.100+0000 7f3f64c12700 1 --2- 172.21.15.28:0/3480887899 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3f4c0a3cd0 0x7f3f4c0a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3f50005850 tx=0x7f3f5001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.104+0000 7f3f567fc700 1 -- 172.21.15.28:0/3480887899 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f5c01f020 con 0x7f3f6006b730 2024-04-21T08:50:28.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.309+0000 7f3f67677700 1 -- 172.21.15.28:0/3480887899 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3f60104ba0 con 0x7f3f4c0a3cd0 2024-04-21T08:50:28.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.318+0000 7f3f567fc700 1 -- 172.21.15.28:0/3480887899 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340651 (secure 0 0 0) 0x7f3f5007a2c0 con 0x7f3f4c0a3cd0 2024-04-21T08:50:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:50:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.322+0000 7f3f67677700 1 -- 172.21.15.28:0/3480887899 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3f4c0a3cd0 msgr2=0x7f3f4c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.322+0000 7f3f67677700 1 --2- 172.21.15.28:0/3480887899 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3f4c0a3cd0 0x7f3f4c0a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3f50005850 tx=0x7f3f5001a040 comp rx=0 tx=0).stop 2024-04-21T08:50:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.322+0000 7f3f67677700 1 -- 172.21.15.28:0/3480887899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f6006b730 msgr2=0x7f3f601a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.322+0000 7f3f67677700 1 --2- 172.21.15.28:0/3480887899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f6006b730 0x7f3f601a53f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00b530 tx=0x7f3f5c00b8f0 comp rx=0 tx=0).stop 2024-04-21T08:50:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.322+0000 7f3f67677700 1 -- 172.21.15.28:0/3480887899 shutdown_connections 2024-04-21T08:50:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.322+0000 7f3f67677700 1 --2- 172.21.15.28:0/3480887899 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3f4c0a3cd0 0x7f3f4c0a6190 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.322+0000 7f3f67677700 1 --2- 172.21.15.28:0/3480887899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f6006b730 0x7f3f601a53f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.322+0000 7f3f67677700 1 --2- 172.21.15.28:0/3480887899 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3f601079a0 0x7f3f6019f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.322+0000 7f3f67677700 1 --2- 172.21.15.28:0/3480887899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3f601054b0 0x7f3f601a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.322+0000 7f3f67677700 1 -- 172.21.15.28:0/3480887899 >> 172.21.15.28:0/3480887899 conn(0x7f3f600789e0 msgr2=0x7f3f60104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:28.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.322+0000 7f3f67677700 1 -- 172.21.15.28:0/3480887899 shutdown_connections 2024-04-21T08:50:28.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:28.322+0000 7f3f67677700 1 -- 172.21.15.28:0/3480887899 wait complete. 2024-04-21T08:50:28.337 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-21T08:50:29.040 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-04-21T08:50:27.487659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657959,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":598,"num_read_kb":422,"num_write":344,"num_write_kb":3056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181430,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369676,"kb_used_data":12152,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572148,"statfs":{"total":1151940427776,"available":1151561879552,"internally_reserved":0,"allocated":12443648,"data_stored":7958927,"data_compressed":360168,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003397"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905590+0000","last_change":"2024-04-21T08:48:46.471626+0000","last_active":"2024-04-21T08:49:36.905590+0000","last_peered":"2024-04-21T08:49:36.905590+0000","last_clean":"2024-04-21T08:49:36.905590+0000","last_became_active":"2024-04-21T08:48:46.470498+0000","last_became_peered":"2024-04-21T08:48:46.470498+0000","last_unstale":"2024-04-21T08:49:36.905590+0000","last_undegraded":"2024-04-21T08:49:36.905590+0000","last_fullsized":"2024-04-21T08:49:36.905590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:49.452181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906122+0000","last_change":"2024-04-21T08:48:46.469111+0000","last_active":"2024-04-21T08:49:36.906122+0000","last_peered":"2024-04-21T08:49:36.906122+0000","last_clean":"2024-04-21T08:49:36.906122+0000","last_became_active":"2024-04-21T08:48:46.467926+0000","last_became_peered":"2024-04-21T08:48:46.467926+0000","last_unstale":"2024-04-21T08:49:36.906122+0000","last_undegraded":"2024-04-21T08:49:36.906122+0000","last_fullsized":"2024-04-21T08:49:36.906122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:35:38.230212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905812+0000","last_change":"2024-04-21T08:48:46.475673+0000","last_active":"2024-04-21T08:49:36.905812+0000","last_peered":"2024-04-21T08:49:36.905812+0000","last_clean":"2024-04-21T08:49:36.905812+0000","last_became_active":"2024-04-21T08:48:46.474917+0000","last_became_peered":"2024-04-21T08:48:46.474917+0000","last_unstale":"2024-04-21T08:49:36.905812+0000","last_undegraded":"2024-04-21T08:49:36.905812+0000","last_fullsized":"2024-04-21T08:49:36.905812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:01.661704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908100+0000","last_change":"2024-04-21T08:48:46.468147+0000","last_active":"2024-04-21T08:49:36.908100+0000","last_peered":"2024-04-21T08:49:36.908100+0000","last_clean":"2024-04-21T08:49:36.908100+0000","last_became_active":"2024-04-21T08:48:46.467811+0000","last_became_peered":"2024-04-21T08:48:46.467811+0000","last_unstale":"2024-04-21T08:49:36.908100+0000","last_undegraded":"2024-04-21T08:49:36.908100+0000","last_fullsized":"2024-04-21T08:49:36.908100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:21.658605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908334+0000","last_change":"2024-04-21T08:48:46.471322+0000","last_active":"2024-04-21T08:49:36.908334+0000","last_peered":"2024-04-21T08:49:36.908334+0000","last_clean":"2024-04-21T08:49:36.908334+0000","last_became_active":"2024-04-21T08:48:46.470991+0000","last_became_peered":"2024-04-21T08:48:46.470991+0000","last_unstale":"2024-04-21T08:49:36.908334+0000","last_undegraded":"2024-04-21T08:49:36.908334+0000","last_fullsized":"2024-04-21T08:49:36.908334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:22.839870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963381+0000","last_change":"2024-04-21T08:48:46.467039+0000","last_active":"2024-04-21T08:49:36.963381+0000","last_peered":"2024-04-21T08:49:36.963381+0000","last_clean":"2024-04-21T08:49:36.963381+0000","last_became_active":"2024-04-21T08:48:46.466614+0000","last_became_peered":"2024-04-21T08:48:46.466614+0000","last_unstale":"2024-04-21T08:49:36.963381+0000","last_undegraded":"2024-04-21T08:49:36.963381+0000","last_fullsized":"2024-04-21T08:49:36.963381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:38.029215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.912475+0000","last_change":"2024-04-21T08:48:46.473268+0000","last_active":"2024-04-21T08:49:36.912475+0000","last_peered":"2024-04-21T08:49:36.912475+0000","last_clean":"2024-04-21T08:49:36.912475+0000","last_became_active":"2024-04-21T08:48:46.472261+0000","last_became_peered":"2024-04-21T08:48:46.472261+0000","last_unstale":"2024-04-21T08:49:36.912475+0000","last_undegraded":"2024-04-21T08:49:36.912475+0000","last_fullsized":"2024-04-21T08:49:36.912475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:58:52.990687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963050+0000","last_change":"2024-04-21T08:48:46.472788+0000","last_active":"2024-04-21T08:49:36.963050+0000","last_peered":"2024-04-21T08:49:36.963050+0000","last_clean":"2024-04-21T08:49:36.963050+0000","last_became_active":"2024-04-21T08:48:46.472482+0000","last_became_peered":"2024-04-21T08:48:46.472482+0000","last_unstale":"2024-04-21T08:49:36.963050+0000","last_undegraded":"2024-04-21T08:49:36.963050+0000","last_fullsized":"2024-04-21T08:49:36.963050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:48.483846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907852+0000","last_change":"2024-04-21T08:48:46.460771+0000","last_active":"2024-04-21T08:49:36.907852+0000","last_peered":"2024-04-21T08:49:36.907852+0000","last_clean":"2024-04-21T08:49:36.907852+0000","last_became_active":"2024-04-21T08:48:46.460472+0000","last_became_peered":"2024-04-21T08:48:46.460472+0000","last_unstale":"2024-04-21T08:49:36.907852+0000","last_undegraded":"2024-04-21T08:49:36.907852+0000","last_fullsized":"2024-04-21T08:49:36.907852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:46:40.766245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963692+0000","last_change":"2024-04-21T08:48:46.463475+0000","last_active":"2024-04-21T08:49:36.963692+0000","last_peered":"2024-04-21T08:49:36.963692+0000","last_clean":"2024-04-21T08:49:36.963692+0000","last_became_active":"2024-04-21T08:48:46.462559+0000","last_became_peered":"2024-04-21T08:48:46.462559+0000","last_unstale":"2024-04-21T08:49:36.963692+0000","last_undegraded":"2024-04-21T08:49:36.963692+0000","last_fullsized":"2024-04-21T08:49:36.963692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:22:06.372410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913327+0000","last_change":"2024-04-21T08:48:46.476997+0000","last_active":"2024-04-21T08:49:36.913327+0000","last_peered":"2024-04-21T08:49:36.913327+0000","last_clean":"2024-04-21T08:49:36.913327+0000","last_became_active":"2024-04-21T08:48:46.476620+0000","last_became_peered":"2024-04-21T08:48:46.476620+0000","last_unstale":"2024-04-21T08:49:36.913327+0000","last_undegraded":"2024-04-21T08:49:36.913327+0000","last_fullsized":"2024-04-21T08:49:36.913327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:37.569629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908164+0000","last_change":"2024-04-21T08:48:46.469355+0000","last_active":"2024-04-21T08:49:36.908164+0000","last_peered":"2024-04-21T08:49:36.908164+0000","last_clean":"2024-04-21T08:49:36.908164+0000","last_became_active":"2024-04-21T08:48:46.468604+0000","last_became_peered":"2024-04-21T08:48:46.468604+0000","last_unstale":"2024-04-21T08:49:36.908164+0000","last_undegraded":"2024-04-21T08:49:36.908164+0000","last_fullsized":"2024-04-21T08:49:36.908164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:23:54.500982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905526+0000","last_change":"2024-04-21T08:48:46.453116+0000","last_active":"2024-04-21T08:49:36.905526+0000","last_peered":"2024-04-21T08:49:36.905526+0000","last_clean":"2024-04-21T08:49:36.905526+0000","last_became_active":"2024-04-21T08:48:46.452607+0000","last_became_peered":"2024-04-21T08:48:46.452607+0000","last_unstale":"2024-04-21T08:49:36.905526+0000","last_undegraded":"2024-04-21T08:49:36.905526+0000","last_fullsized":"2024-04-21T08:49:36.905526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:57:04.371812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906749+0000","last_change":"2024-04-21T08:48:46.474516+0000","last_active":"2024-04-21T08:49:36.906749+0000","last_peered":"2024-04-21T08:49:36.906749+0000","last_clean":"2024-04-21T08:49:36.906749+0000","last_became_active":"2024-04-21T08:48:46.474129+0000","last_became_peered":"2024-04-21T08:48:46.474129+0000","last_unstale":"2024-04-21T08:49:36.906749+0000","last_undegraded":"2024-04-21T08:49:36.906749+0000","last_fullsized":"2024-04-21T08:49:36.906749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:40:47.784976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907964+0000","last_change":"2024-04-21T08:49:36.907964+0000","last_active":"2024-04-21T08:49:36.907964+0000","last_peered":"2024-04-21T08:49:36.907964+0000","last_clean":"2024-04-21T08:49:36.907964+0000","last_became_active":"2024-04-21T08:49:36.907595+0000","last_became_peered":"2024-04-21T08:49:36.907595+0000","last_unstale":"2024-04-21T08:49:36.907964+0000","last_undegraded":"2024-04-21T08:49:36.907964+0000","last_fullsized":"2024-04-21T08:49:36.907964+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:14:04.907328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908095+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:49:36.908095+0000","last_peered":"2024-04-21T08:49:36.908095+0000","last_clean":"2024-04-21T08:49:36.908095+0000","last_became_active":"2024-04-21T08:48:46.469389+0000","last_became_peered":"2024-04-21T08:48:46.469389+0000","last_unstale":"2024-04-21T08:49:36.908095+0000","last_undegraded":"2024-04-21T08:49:36.908095+0000","last_fullsized":"2024-04-21T08:49:36.908095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:11:45.824402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905463+0000","last_change":"2024-04-21T08:48:46.470059+0000","last_active":"2024-04-21T08:49:36.905463+0000","last_peered":"2024-04-21T08:49:36.905463+0000","last_clean":"2024-04-21T08:49:36.905463+0000","last_became_active":"2024-04-21T08:48:46.469751+0000","last_became_peered":"2024-04-21T08:48:46.469751+0000","last_unstale":"2024-04-21T08:49:36.905463+0000","last_undegraded":"2024-04-21T08:49:36.905463+0000","last_fullsized":"2024-04-21T08:49:36.905463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:16.317879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902639+0000","last_change":"2024-04-21T08:48:46.470129+0000","last_active":"2024-04-21T08:49:36.902639+0000","last_peered":"2024-04-21T08:49:36.902639+0000","last_clean":"2024-04-21T08:49:36.902639+0000","last_became_active":"2024-04-21T08:48:46.469763+0000","last_became_peered":"2024-04-21T08:48:46.469763+0000","last_unstale":"2024-04-21T08:49:36.902639+0000","last_undegraded":"2024-04-21T08:49:36.902639+0000","last_fullsized":"2024-04-21T08:49:36.902639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:47.731201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.903865+0000","last_change":"2024-04-21T08:48:46.465356+0000","last_active":"2024-04-21T08:49:36.903865+0000","last_peered":"2024-04-21T08:49:36.903865+0000","last_clean":"2024-04-21T08:49:36.903865+0000","last_became_active":"2024-04-21T08:48:46.464183+0000","last_became_peered":"2024-04-21T08:48:46.464183+0000","last_unstale":"2024-04-21T08:49:36.903865+0000","last_undegraded":"2024-04-21T08:49:36.903865+0000","last_fullsized":"2024-04-21T08:49:36.903865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:37.533733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.903496+0000","last_change":"2024-04-21T08:48:50.767378+0000","last_active":"2024-04-21T08:49:36.903496+0000","last_peered":"2024-04-21T08:49:36.903496+0000","last_clean":"2024-04-21T08:49:36.903496+0000","last_became_active":"2024-04-21T08:48:50.766856+0000","last_became_peered":"2024-04-21T08:48:50.766856+0000","last_unstale":"2024-04-21T08:49:36.903496+0000","last_undegraded":"2024-04-21T08:49:36.903496+0000","last_fullsized":"2024-04-21T08:49:36.903496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:00.434540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902634+0000","last_change":"2024-04-21T08:48:45.550284+0000","last_active":"2024-04-21T08:49:36.902634+0000","last_peered":"2024-04-21T08:49:36.902634+0000","last_clean":"2024-04-21T08:49:36.902634+0000","last_became_active":"2024-04-21T08:48:45.549940+0000","last_became_peered":"2024-04-21T08:48:45.549940+0000","last_unstale":"2024-04-21T08:49:36.902634+0000","last_undegraded":"2024-04-21T08:49:36.902634+0000","last_fullsized":"2024-04-21T08:49:36.902634+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:12.343299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963199+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:49:36.963199+0000","last_peered":"2024-04-21T08:49:36.963199+0000","last_clean":"2024-04-21T08:49:36.963199+0000","last_became_active":"2024-04-21T08:48:46.469647+0000","last_became_peered":"2024-04-21T08:48:46.469647+0000","last_unstale":"2024-04-21T08:49:36.963199+0000","last_undegraded":"2024-04-21T08:49:36.963199+0000","last_fullsized":"2024-04-21T08:49:36.963199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:10:28.204554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906837+0000","last_change":"2024-04-21T08:48:50.771871+0000","last_active":"2024-04-21T08:49:36.906837+0000","last_peered":"2024-04-21T08:49:36.906837+0000","last_clean":"2024-04-21T08:49:36.906837+0000","last_became_active":"2024-04-21T08:48:50.770442+0000","last_became_peered":"2024-04-21T08:48:50.770442+0000","last_unstale":"2024-04-21T08:49:36.906837+0000","last_undegraded":"2024-04-21T08:49:36.906837+0000","last_fullsized":"2024-04-21T08:49:36.906837+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:18:57.813971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963310+0000","last_change":"2024-04-21T08:48:45.566492+0000","last_active":"2024-04-21T08:49:36.963310+0000","last_peered":"2024-04-21T08:49:36.963310+0000","last_clean":"2024-04-21T08:49:36.963310+0000","last_became_active":"2024-04-21T08:48:45.566169+0000","last_became_peered":"2024-04-21T08:48:45.566169+0000","last_unstale":"2024-04-21T08:49:36.963310+0000","last_undegraded":"2024-04-21T08:49:36.963310+0000","last_fullsized":"2024-04-21T08:49:36.963310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:49.985664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906531+0000","last_change":"2024-04-21T08:48:46.470866+0000","last_active":"2024-04-21T08:49:36.906531+0000","last_peered":"2024-04-21T08:49:36.906531+0000","last_clean":"2024-04-21T08:49:36.906531+0000","last_became_active":"2024-04-21T08:48:46.470401+0000","last_became_peered":"2024-04-21T08:48:46.470401+0000","last_unstale":"2024-04-21T08:49:36.906531+0000","last_undegraded":"2024-04-21T08:49:36.906531+0000","last_fullsized":"2024-04-21T08:49:36.906531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:07.506212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907429+0000","last_change":"2024-04-21T08:48:50.772339+0000","last_active":"2024-04-21T08:49:36.907429+0000","last_peered":"2024-04-21T08:49:36.907429+0000","last_clean":"2024-04-21T08:49:36.907429+0000","last_became_active":"2024-04-21T08:48:50.771946+0000","last_became_peered":"2024-04-21T08:48:50.771946+0000","last_unstale":"2024-04-21T08:49:36.907429+0000","last_undegraded":"2024-04-21T08:49:36.907429+0000","last_fullsized":"2024-04-21T08:49:36.907429+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:03:56.994581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.909040+0000","last_change":"2024-04-21T08:48:45.553946+0000","last_active":"2024-04-21T08:49:36.909040+0000","last_peered":"2024-04-21T08:49:36.909040+0000","last_clean":"2024-04-21T08:49:36.909040+0000","last_became_active":"2024-04-21T08:48:45.553410+0000","last_became_peered":"2024-04-21T08:48:45.553410+0000","last_unstale":"2024-04-21T08:49:36.909040+0000","last_undegraded":"2024-04-21T08:49:36.909040+0000","last_fullsized":"2024-04-21T08:49:36.909040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:24:02.387250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907005+0000","last_change":"2024-04-21T08:48:46.471815+0000","last_active":"2024-04-21T08:49:36.907005+0000","last_peered":"2024-04-21T08:49:36.907005+0000","last_clean":"2024-04-21T08:49:36.907005+0000","last_became_active":"2024-04-21T08:48:46.471492+0000","last_became_peered":"2024-04-21T08:48:46.471492+0000","last_unstale":"2024-04-21T08:49:36.907005+0000","last_undegraded":"2024-04-21T08:49:36.907005+0000","last_fullsized":"2024-04-21T08:49:36.907005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:26.051166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906784+0000","last_change":"2024-04-21T08:48:50.767916+0000","last_active":"2024-04-21T08:49:36.906784+0000","last_peered":"2024-04-21T08:49:36.906784+0000","last_clean":"2024-04-21T08:49:36.906784+0000","last_became_active":"2024-04-21T08:48:50.767326+0000","last_became_peered":"2024-04-21T08:48:50.767326+0000","last_unstale":"2024-04-21T08:49:36.906784+0000","last_undegraded":"2024-04-21T08:49:36.906784+0000","last_fullsized":"2024-04-21T08:49:36.906784+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:05:02.185369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908720+0000","last_change":"2024-04-21T08:48:45.549982+0000","last_active":"2024-04-21T08:49:36.908720+0000","last_peered":"2024-04-21T08:49:36.908720+0000","last_clean":"2024-04-21T08:49:36.908720+0000","last_became_active":"2024-04-21T08:48:45.549611+0000","last_became_peered":"2024-04-21T08:48:45.549611+0000","last_unstale":"2024-04-21T08:49:36.908720+0000","last_undegraded":"2024-04-21T08:49:36.908720+0000","last_fullsized":"2024-04-21T08:49:36.908720+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:46.035388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.914297+0000","last_change":"2024-04-21T08:48:46.459068+0000","last_active":"2024-04-21T08:49:36.914297+0000","last_peered":"2024-04-21T08:49:36.914297+0000","last_clean":"2024-04-21T08:49:36.914297+0000","last_became_active":"2024-04-21T08:48:46.458686+0000","last_became_peered":"2024-04-21T08:48:46.458686+0000","last_unstale":"2024-04-21T08:49:36.914297+0000","last_undegraded":"2024-04-21T08:49:36.914297+0000","last_fullsized":"2024-04-21T08:49:36.914297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:31.373451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.903738+0000","last_change":"2024-04-21T08:48:50.761392+0000","last_active":"2024-04-21T08:49:36.903738+0000","last_peered":"2024-04-21T08:49:36.903738+0000","last_clean":"2024-04-21T08:49:36.903738+0000","last_became_active":"2024-04-21T08:48:50.760917+0000","last_became_peered":"2024-04-21T08:48:50.760917+0000","last_unstale":"2024-04-21T08:49:36.903738+0000","last_undegraded":"2024-04-21T08:49:36.903738+0000","last_fullsized":"2024-04-21T08:49:36.903738+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:19.070502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:50:21.947070+0000","last_change":"2024-04-21T08:48:45.553686+0000","last_active":"2024-04-21T08:50:21.947070+0000","last_peered":"2024-04-21T08:50:21.947070+0000","last_clean":"2024-04-21T08:50:21.947070+0000","last_became_active":"2024-04-21T08:48:45.553322+0000","last_became_peered":"2024-04-21T08:48:45.553322+0000","last_unstale":"2024-04-21T08:50:21.947070+0000","last_undegraded":"2024-04-21T08:50:21.947070+0000","last_fullsized":"2024-04-21T08:50:21.947070+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:15:24.320704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908427+0000","last_change":"2024-04-21T08:48:46.474555+0000","last_active":"2024-04-21T08:49:36.908427+0000","last_peered":"2024-04-21T08:49:36.908427+0000","last_clean":"2024-04-21T08:49:36.908427+0000","last_became_active":"2024-04-21T08:48:46.474185+0000","last_became_peered":"2024-04-21T08:48:46.474185+0000","last_unstale":"2024-04-21T08:49:36.908427+0000","last_undegraded":"2024-04-21T08:49:36.908427+0000","last_fullsized":"2024-04-21T08:49:36.908427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:52.884994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.903229+0000","last_change":"2024-04-21T08:48:50.763181+0000","last_active":"2024-04-21T08:49:36.903229+0000","last_peered":"2024-04-21T08:49:36.903229+0000","last_clean":"2024-04-21T08:49:36.903229+0000","last_became_active":"2024-04-21T08:48:50.762718+0000","last_became_peered":"2024-04-21T08:48:50.762718+0000","last_unstale":"2024-04-21T08:49:36.903229+0000","last_undegraded":"2024-04-21T08:49:36.903229+0000","last_fullsized":"2024-04-21T08:49:36.903229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:16:23.204605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963370+0000","last_change":"2024-04-21T08:48:45.547768+0000","last_active":"2024-04-21T08:49:36.963370+0000","last_peered":"2024-04-21T08:49:36.963370+0000","last_clean":"2024-04-21T08:49:36.963370+0000","last_became_active":"2024-04-21T08:48:45.547456+0000","last_became_peered":"2024-04-21T08:48:45.547456+0000","last_unstale":"2024-04-21T08:49:36.963370+0000","last_undegraded":"2024-04-21T08:49:36.963370+0000","last_fullsized":"2024-04-21T08:49:36.963370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:23.502799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.914274+0000","last_change":"2024-04-21T08:48:46.467988+0000","last_active":"2024-04-21T08:49:36.914274+0000","last_peered":"2024-04-21T08:49:36.914274+0000","last_clean":"2024-04-21T08:49:36.914274+0000","last_became_active":"2024-04-21T08:48:46.465212+0000","last_became_peered":"2024-04-21T08:48:46.465212+0000","last_unstale":"2024-04-21T08:49:36.914274+0000","last_undegraded":"2024-04-21T08:49:36.914274+0000","last_fullsized":"2024-04-21T08:49:36.914274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:41.743506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905456+0000","last_change":"2024-04-21T08:48:50.758289+0000","last_active":"2024-04-21T08:49:36.905456+0000","last_peered":"2024-04-21T08:49:36.905456+0000","last_clean":"2024-04-21T08:49:36.905456+0000","last_became_active":"2024-04-21T08:48:50.757775+0000","last_became_peered":"2024-04-21T08:48:50.757775+0000","last_unstale":"2024-04-21T08:49:36.905456+0000","last_undegraded":"2024-04-21T08:49:36.905456+0000","last_fullsized":"2024-04-21T08:49:36.905456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:56:27.246515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905286+0000","last_change":"2024-04-21T08:48:45.545615+0000","last_active":"2024-04-21T08:49:36.905286+0000","last_peered":"2024-04-21T08:49:36.905286+0000","last_clean":"2024-04-21T08:49:36.905286+0000","last_became_active":"2024-04-21T08:48:45.545280+0000","last_became_peered":"2024-04-21T08:48:45.545280+0000","last_unstale":"2024-04-21T08:49:36.905286+0000","last_undegraded":"2024-04-21T08:49:36.905286+0000","last_fullsized":"2024-04-21T08:49:36.905286+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:50:00.470903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.962853+0000","last_change":"2024-04-21T08:48:46.477326+0000","last_active":"2024-04-21T08:49:36.962853+0000","last_peered":"2024-04-21T08:49:36.962853+0000","last_clean":"2024-04-21T08:49:36.962853+0000","last_became_active":"2024-04-21T08:48:46.476914+0000","last_became_peered":"2024-04-21T08:48:46.476914+0000","last_unstale":"2024-04-21T08:49:36.962853+0000","last_undegraded":"2024-04-21T08:49:36.962853+0000","last_fullsized":"2024-04-21T08:49:36.962853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:04.312061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906018+0000","last_change":"2024-04-21T08:48:50.767429+0000","last_active":"2024-04-21T08:49:36.906018+0000","last_peered":"2024-04-21T08:49:36.906018+0000","last_clean":"2024-04-21T08:49:36.906018+0000","last_became_active":"2024-04-21T08:48:50.767008+0000","last_became_peered":"2024-04-21T08:48:50.767008+0000","last_unstale":"2024-04-21T08:49:36.906018+0000","last_undegraded":"2024-04-21T08:49:36.906018+0000","last_fullsized":"2024-04-21T08:49:36.906018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:26.865571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913761+0000","last_change":"2024-04-21T08:48:45.553170+0000","last_active":"2024-04-21T08:49:36.913761+0000","last_peered":"2024-04-21T08:49:36.913761+0000","last_clean":"2024-04-21T08:49:36.913761+0000","last_became_active":"2024-04-21T08:48:45.552645+0000","last_became_peered":"2024-04-21T08:48:45.552645+0000","last_unstale":"2024-04-21T08:49:36.913761+0000","last_undegraded":"2024-04-21T08:49:36.913761+0000","last_fullsized":"2024-04-21T08:49:36.913761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:32:07.217212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902122+0000","last_change":"2024-04-21T08:48:46.478101+0000","last_active":"2024-04-21T08:49:36.902122+0000","last_peered":"2024-04-21T08:49:36.902122+0000","last_clean":"2024-04-21T08:49:36.902122+0000","last_became_active":"2024-04-21T08:48:46.477725+0000","last_became_peered":"2024-04-21T08:48:46.477725+0000","last_unstale":"2024-04-21T08:49:36.902122+0000","last_undegraded":"2024-04-21T08:49:36.902122+0000","last_fullsized":"2024-04-21T08:49:36.902122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:37:55.429569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963146+0000","last_change":"2024-04-21T08:48:50.762899+0000","last_active":"2024-04-21T08:49:36.963146+0000","last_peered":"2024-04-21T08:49:36.963146+0000","last_clean":"2024-04-21T08:49:36.963146+0000","last_became_active":"2024-04-21T08:48:50.762475+0000","last_became_peered":"2024-04-21T08:48:50.762475+0000","last_unstale":"2024-04-21T08:49:36.963146+0000","last_undegraded":"2024-04-21T08:49:36.963146+0000","last_fullsized":"2024-04-21T08:49:36.963146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:27.540414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:50:24.950829+0000","last_change":"2024-04-21T08:48:45.550151+0000","last_active":"2024-04-21T08:50:24.950829+0000","last_peered":"2024-04-21T08:50:24.950829+0000","last_clean":"2024-04-21T08:50:24.950829+0000","last_became_active":"2024-04-21T08:48:45.549829+0000","last_became_peered":"2024-04-21T08:48:45.549829+0000","last_unstale":"2024-04-21T08:50:24.950829+0000","last_undegraded":"2024-04-21T08:50:24.950829+0000","last_fullsized":"2024-04-21T08:50:24.950829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:14.061189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907733+0000","last_change":"2024-04-21T08:48:46.472072+0000","last_active":"2024-04-21T08:49:36.907733+0000","last_peered":"2024-04-21T08:49:36.907733+0000","last_clean":"2024-04-21T08:49:36.907733+0000","last_became_active":"2024-04-21T08:48:46.471660+0000","last_became_peered":"2024-04-21T08:48:46.471660+0000","last_unstale":"2024-04-21T08:49:36.907733+0000","last_undegraded":"2024-04-21T08:49:36.907733+0000","last_fullsized":"2024-04-21T08:49:36.907733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:30:11.843870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902107+0000","last_change":"2024-04-21T08:48:50.773021+0000","last_active":"2024-04-21T08:49:36.902107+0000","last_peered":"2024-04-21T08:49:36.902107+0000","last_clean":"2024-04-21T08:49:36.902107+0000","last_became_active":"2024-04-21T08:48:50.771746+0000","last_became_peered":"2024-04-21T08:48:50.771746+0000","last_unstale":"2024-04-21T08:49:36.902107+0000","last_undegraded":"2024-04-21T08:49:36.902107+0000","last_fullsized":"2024-04-21T08:49:36.902107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:33:21.415967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913020+0000","last_change":"2024-04-21T08:48:45.550376+0000","last_active":"2024-04-21T08:49:36.913020+0000","last_peered":"2024-04-21T08:49:36.913020+0000","last_clean":"2024-04-21T08:49:36.913020+0000","last_became_active":"2024-04-21T08:48:45.549594+0000","last_became_peered":"2024-04-21T08:48:45.549594+0000","last_unstale":"2024-04-21T08:49:36.913020+0000","last_undegraded":"2024-04-21T08:49:36.913020+0000","last_fullsized":"2024-04-21T08:49:36.913020+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:26:27.046963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907060+0000","last_change":"2024-04-21T08:48:46.469499+0000","last_active":"2024-04-21T08:49:36.907060+0000","last_peered":"2024-04-21T08:49:36.907060+0000","last_clean":"2024-04-21T08:49:36.907060+0000","last_became_active":"2024-04-21T08:48:46.469124+0000","last_became_peered":"2024-04-21T08:48:46.469124+0000","last_unstale":"2024-04-21T08:49:36.907060+0000","last_undegraded":"2024-04-21T08:49:36.907060+0000","last_fullsized":"2024-04-21T08:49:36.907060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:58.708916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907631+0000","last_change":"2024-04-21T08:48:50.771331+0000","last_active":"2024-04-21T08:49:36.907631+0000","last_peered":"2024-04-21T08:49:36.907631+0000","last_clean":"2024-04-21T08:49:36.907631+0000","last_became_active":"2024-04-21T08:48:50.770018+0000","last_became_peered":"2024-04-21T08:48:50.770018+0000","last_unstale":"2024-04-21T08:49:36.907631+0000","last_undegraded":"2024-04-21T08:49:36.907631+0000","last_fullsized":"2024-04-21T08:49:36.907631+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:19:03.416423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.914163+0000","last_change":"2024-04-21T08:48:45.559319+0000","last_active":"2024-04-21T08:49:36.914163+0000","last_peered":"2024-04-21T08:49:36.914163+0000","last_clean":"2024-04-21T08:49:36.914163+0000","last_became_active":"2024-04-21T08:48:45.558895+0000","last_became_peered":"2024-04-21T08:48:45.558895+0000","last_unstale":"2024-04-21T08:49:36.914163+0000","last_undegraded":"2024-04-21T08:49:36.914163+0000","last_fullsized":"2024-04-21T08:49:36.914163+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:01.878169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907503+0000","last_change":"2024-04-21T08:48:46.466110+0000","last_active":"2024-04-21T08:49:36.907503+0000","last_peered":"2024-04-21T08:49:36.907503+0000","last_clean":"2024-04-21T08:49:36.907503+0000","last_became_active":"2024-04-21T08:48:46.464374+0000","last_became_peered":"2024-04-21T08:48:46.464374+0000","last_unstale":"2024-04-21T08:49:36.907503+0000","last_undegraded":"2024-04-21T08:49:36.907503+0000","last_fullsized":"2024-04-21T08:49:36.907503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:26.307051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902203+0000","last_change":"2024-04-21T08:48:50.759976+0000","last_active":"2024-04-21T08:49:36.902203+0000","last_peered":"2024-04-21T08:49:36.902203+0000","last_clean":"2024-04-21T08:49:36.902203+0000","last_became_active":"2024-04-21T08:48:50.759596+0000","last_became_peered":"2024-04-21T08:48:50.759596+0000","last_unstale":"2024-04-21T08:49:36.902203+0000","last_undegraded":"2024-04-21T08:49:36.902203+0000","last_fullsized":"2024-04-21T08:49:36.902203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:09.678388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'33","reported_seq":216,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:50:25.952272+0000","last_change":"2024-04-21T08:48:45.549228+0000","last_active":"2024-04-21T08:50:25.952272+0000","last_peered":"2024-04-21T08:50:25.952272+0000","last_clean":"2024-04-21T08:50:25.952272+0000","last_became_active":"2024-04-21T08:48:45.548865+0000","last_became_peered":"2024-04-21T08:48:45.548865+0000","last_unstale":"2024-04-21T08:50:25.952272+0000","last_undegraded":"2024-04-21T08:50:25.952272+0000","last_fullsized":"2024-04-21T08:50:25.952272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:28.620968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":156,"num_read_kb":140,"num_write":34,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908611+0000","last_change":"2024-04-21T08:48:46.470322+0000","last_active":"2024-04-21T08:49:36.908611+0000","last_peered":"2024-04-21T08:49:36.908611+0000","last_clean":"2024-04-21T08:49:36.908611+0000","last_became_active":"2024-04-21T08:48:46.467225+0000","last_became_peered":"2024-04-21T08:48:46.467225+0000","last_unstale":"2024-04-21T08:49:36.908611+0000","last_undegraded":"2024-04-21T08:49:36.908611+0000","last_fullsized":"2024-04-21T08:49:36.908611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:31:17.390926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.914518+0000","last_change":"2024-04-21T08:48:50.770982+0000","last_active":"2024-04-21T08:49:36.914518+0000","last_peered":"2024-04-21T08:49:36.914518+0000","last_clean":"2024-04-21T08:49:36.914518+0000","last_became_active":"2024-04-21T08:48:50.770312+0000","last_became_peered":"2024-04-21T08:48:50.770312+0000","last_unstale":"2024-04-21T08:49:36.914518+0000","last_undegraded":"2024-04-21T08:49:36.914518+0000","last_fullsized":"2024-04-21T08:49:36.914518+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:41:33.027351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.903242+0000","last_change":"2024-04-21T08:48:45.556134+0000","last_active":"2024-04-21T08:49:36.903242+0000","last_peered":"2024-04-21T08:49:36.903242+0000","last_clean":"2024-04-21T08:49:36.903242+0000","last_became_active":"2024-04-21T08:48:45.555818+0000","last_became_peered":"2024-04-21T08:48:45.555818+0000","last_unstale":"2024-04-21T08:49:36.903242+0000","last_undegraded":"2024-04-21T08:49:36.903242+0000","last_fullsized":"2024-04-21T08:49:36.903242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:00:08.083783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913151+0000","last_change":"2024-04-21T08:49:36.913151+0000","last_active":"2024-04-21T08:49:36.913151+0000","last_peered":"2024-04-21T08:49:36.913151+0000","last_clean":"2024-04-21T08:49:36.913151+0000","last_became_active":"2024-04-21T08:49:36.912810+0000","last_became_peered":"2024-04-21T08:49:36.912810+0000","last_unstale":"2024-04-21T08:49:36.913151+0000","last_undegraded":"2024-04-21T08:49:36.913151+0000","last_fullsized":"2024-04-21T08:49:36.913151+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:17:32.887163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905143+0000","last_change":"2024-04-21T08:48:50.765967+0000","last_active":"2024-04-21T08:49:36.905143+0000","last_peered":"2024-04-21T08:49:36.905143+0000","last_clean":"2024-04-21T08:49:36.905143+0000","last_became_active":"2024-04-21T08:48:50.765557+0000","last_became_peered":"2024-04-21T08:48:50.765557+0000","last_unstale":"2024-04-21T08:49:36.905143+0000","last_undegraded":"2024-04-21T08:49:36.905143+0000","last_fullsized":"2024-04-21T08:49:36.905143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:52.044058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913962+0000","last_change":"2024-04-21T08:48:45.559939+0000","last_active":"2024-04-21T08:49:36.913962+0000","last_peered":"2024-04-21T08:49:36.913962+0000","last_clean":"2024-04-21T08:49:36.913962+0000","last_became_active":"2024-04-21T08:48:45.559635+0000","last_became_peered":"2024-04-21T08:48:45.559635+0000","last_unstale":"2024-04-21T08:49:36.913962+0000","last_undegraded":"2024-04-21T08:49:36.913962+0000","last_fullsized":"2024-04-21T08:49:36.913962+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:17.872806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913272+0000","last_change":"2024-04-21T08:48:46.464878+0000","last_active":"2024-04-21T08:49:36.913272+0000","last_peered":"2024-04-21T08:49:36.913272+0000","last_clean":"2024-04-21T08:49:36.913272+0000","last_became_active":"2024-04-21T08:48:46.464552+0000","last_became_peered":"2024-04-21T08:48:46.464552+0000","last_unstale":"2024-04-21T08:49:36.913272+0000","last_undegraded":"2024-04-21T08:49:36.913272+0000","last_fullsized":"2024-04-21T08:49:36.913272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:46:10.558693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902489+0000","last_change":"2024-04-21T08:48:50.765551+0000","last_active":"2024-04-21T08:49:36.902489+0000","last_peered":"2024-04-21T08:49:36.902489+0000","last_clean":"2024-04-21T08:49:36.902489+0000","last_became_active":"2024-04-21T08:48:50.765016+0000","last_became_peered":"2024-04-21T08:48:50.765016+0000","last_unstale":"2024-04-21T08:49:36.902489+0000","last_undegraded":"2024-04-21T08:49:36.902489+0000","last_fullsized":"2024-04-21T08:49:36.902489+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:03:05.317720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:50:22.962284+0000","last_change":"2024-04-21T08:48:45.548685+0000","last_active":"2024-04-21T08:50:22.962284+0000","last_peered":"2024-04-21T08:50:22.962284+0000","last_clean":"2024-04-21T08:50:22.962284+0000","last_became_active":"2024-04-21T08:48:45.548372+0000","last_became_peered":"2024-04-21T08:48:45.548372+0000","last_unstale":"2024-04-21T08:50:22.962284+0000","last_undegraded":"2024-04-21T08:50:22.962284+0000","last_fullsized":"2024-04-21T08:50:22.962284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:44.780297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.912275+0000","last_change":"2024-04-21T08:48:46.471541+0000","last_active":"2024-04-21T08:49:36.912275+0000","last_peered":"2024-04-21T08:49:36.912275+0000","last_clean":"2024-04-21T08:49:36.912275+0000","last_became_active":"2024-04-21T08:48:46.471223+0000","last_became_peered":"2024-04-21T08:48:46.471223+0000","last_unstale":"2024-04-21T08:49:36.912275+0000","last_undegraded":"2024-04-21T08:49:36.912275+0000","last_fullsized":"2024-04-21T08:49:36.912275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:24:50.867905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906371+0000","last_change":"2024-04-21T08:48:50.773941+0000","last_active":"2024-04-21T08:49:36.906371+0000","last_peered":"2024-04-21T08:49:36.906371+0000","last_clean":"2024-04-21T08:49:36.906371+0000","last_became_active":"2024-04-21T08:48:50.772081+0000","last_became_peered":"2024-04-21T08:48:50.772081+0000","last_unstale":"2024-04-21T08:49:36.906371+0000","last_undegraded":"2024-04-21T08:49:36.906371+0000","last_fullsized":"2024-04-21T08:49:36.906371+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:04.438944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.923567+0000","last_change":"2024-04-21T08:49:36.923090+0000","last_active":"2024-04-21T08:49:36.923567+0000","last_peered":"2024-04-21T08:49:36.923567+0000","last_clean":"2024-04-21T08:49:36.923567+0000","last_became_active":"2024-04-21T08:49:36.908075+0000","last_became_peered":"2024-04-21T08:49:36.908075+0000","last_unstale":"2024-04-21T08:49:36.923567+0000","last_undegraded":"2024-04-21T08:49:36.923567+0000","last_fullsized":"2024-04-21T08:49:36.923567+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:41.683372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913956+0000","last_change":"2024-04-21T08:48:46.465559+0000","last_active":"2024-04-21T08:49:36.913956+0000","last_peered":"2024-04-21T08:49:36.913956+0000","last_clean":"2024-04-21T08:49:36.913956+0000","last_became_active":"2024-04-21T08:48:46.465125+0000","last_became_peered":"2024-04-21T08:48:46.465125+0000","last_unstale":"2024-04-21T08:49:36.913956+0000","last_undegraded":"2024-04-21T08:49:36.913956+0000","last_fullsized":"2024-04-21T08:49:36.913956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:52:38.423925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913816+0000","last_change":"2024-04-21T08:48:50.763609+0000","last_active":"2024-04-21T08:49:36.913816+0000","last_peered":"2024-04-21T08:49:36.913816+0000","last_clean":"2024-04-21T08:49:36.913816+0000","last_became_active":"2024-04-21T08:48:50.763144+0000","last_became_peered":"2024-04-21T08:48:50.763144+0000","last_unstale":"2024-04-21T08:49:36.913816+0000","last_undegraded":"2024-04-21T08:49:36.913816+0000","last_fullsized":"2024-04-21T08:49:36.913816+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:26:18.611235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.909318+0000","last_change":"2024-04-21T08:48:45.549912+0000","last_active":"2024-04-21T08:49:36.909318+0000","last_peered":"2024-04-21T08:49:36.909318+0000","last_clean":"2024-04-21T08:49:36.909318+0000","last_became_active":"2024-04-21T08:48:45.549585+0000","last_became_peered":"2024-04-21T08:48:45.549585+0000","last_unstale":"2024-04-21T08:49:36.909318+0000","last_undegraded":"2024-04-21T08:49:36.909318+0000","last_fullsized":"2024-04-21T08:49:36.909318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:58.287402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905374+0000","last_change":"2024-04-21T08:48:46.462746+0000","last_active":"2024-04-21T08:49:36.905374+0000","last_peered":"2024-04-21T08:49:36.905374+0000","last_clean":"2024-04-21T08:49:36.905374+0000","last_became_active":"2024-04-21T08:48:46.461602+0000","last_became_peered":"2024-04-21T08:48:46.461602+0000","last_unstale":"2024-04-21T08:49:36.905374+0000","last_undegraded":"2024-04-21T08:49:36.905374+0000","last_fullsized":"2024-04-21T08:49:36.905374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:52.198634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908295+0000","last_change":"2024-04-21T08:48:50.770770+0000","last_active":"2024-04-21T08:49:36.908295+0000","last_peered":"2024-04-21T08:49:36.908295+0000","last_clean":"2024-04-21T08:49:36.908295+0000","last_became_active":"2024-04-21T08:48:50.770326+0000","last_became_peered":"2024-04-21T08:48:50.770326+0000","last_unstale":"2024-04-21T08:49:36.908295+0000","last_undegraded":"2024-04-21T08:49:36.908295+0000","last_fullsized":"2024-04-21T08:49:36.908295+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:38.325155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902433+0000","last_change":"2024-04-21T08:48:45.558867+0000","last_active":"2024-04-21T08:49:36.902433+0000","last_peered":"2024-04-21T08:49:36.902433+0000","last_clean":"2024-04-21T08:49:36.902433+0000","last_became_active":"2024-04-21T08:48:45.558492+0000","last_became_peered":"2024-04-21T08:48:45.558492+0000","last_unstale":"2024-04-21T08:49:36.902433+0000","last_undegraded":"2024-04-21T08:49:36.902433+0000","last_fullsized":"2024-04-21T08:49:36.902433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:44:55.156404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963608+0000","last_change":"2024-04-21T08:48:46.466461+0000","last_active":"2024-04-21T08:49:36.963608+0000","last_peered":"2024-04-21T08:49:36.963608+0000","last_clean":"2024-04-21T08:49:36.963608+0000","last_became_active":"2024-04-21T08:48:46.466035+0000","last_became_peered":"2024-04-21T08:48:46.466035+0000","last_unstale":"2024-04-21T08:49:36.963608+0000","last_undegraded":"2024-04-21T08:49:36.963608+0000","last_fullsized":"2024-04-21T08:49:36.963608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:47:32.307753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905716+0000","last_change":"2024-04-21T08:48:50.767655+0000","last_active":"2024-04-21T08:49:36.905716+0000","last_peered":"2024-04-21T08:49:36.905716+0000","last_clean":"2024-04-21T08:49:36.905716+0000","last_became_active":"2024-04-21T08:48:50.767305+0000","last_became_peered":"2024-04-21T08:48:50.767305+0000","last_unstale":"2024-04-21T08:49:36.905716+0000","last_undegraded":"2024-04-21T08:49:36.905716+0000","last_fullsized":"2024-04-21T08:49:36.905716+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:31.210126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.912771+0000","last_change":"2024-04-21T08:48:45.552952+0000","last_active":"2024-04-21T08:49:36.912771+0000","last_peered":"2024-04-21T08:49:36.912771+0000","last_clean":"2024-04-21T08:49:36.912771+0000","last_became_active":"2024-04-21T08:48:45.552556+0000","last_became_peered":"2024-04-21T08:48:45.552556+0000","last_unstale":"2024-04-21T08:49:36.912771+0000","last_undegraded":"2024-04-21T08:49:36.912771+0000","last_fullsized":"2024-04-21T08:49:36.912771+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:03.428801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963460+0000","last_change":"2024-04-21T08:48:46.469020+0000","last_active":"2024-04-21T08:49:36.963460+0000","last_peered":"2024-04-21T08:49:36.963460+0000","last_clean":"2024-04-21T08:49:36.963460+0000","last_became_active":"2024-04-21T08:48:46.468631+0000","last_became_peered":"2024-04-21T08:48:46.468631+0000","last_unstale":"2024-04-21T08:49:36.963460+0000","last_undegraded":"2024-04-21T08:49:36.963460+0000","last_fullsized":"2024-04-21T08:49:36.963460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:08.559516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906317+0000","last_change":"2024-04-21T08:48:50.773942+0000","last_active":"2024-04-21T08:49:36.906317+0000","last_peered":"2024-04-21T08:49:36.906317+0000","last_clean":"2024-04-21T08:49:36.906317+0000","last_became_active":"2024-04-21T08:48:50.771876+0000","last_became_peered":"2024-04-21T08:48:50.771876+0000","last_unstale":"2024-04-21T08:49:36.906317+0000","last_undegraded":"2024-04-21T08:49:36.906317+0000","last_fullsized":"2024-04-21T08:49:36.906317+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:19.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.914757+0000","last_change":"2024-04-21T08:48:46.462727+0000","last_active":"2024-04-21T08:49:36.914757+0000","last_peered":"2024-04-21T08:49:36.914757+0000","last_clean":"2024-04-21T08:49:36.914757+0000","last_became_active":"2024-04-21T08:48:46.461669+0000","last_became_peered":"2024-04-21T08:48:46.461669+0000","last_unstale":"2024-04-21T08:49:36.914757+0000","last_undegraded":"2024-04-21T08:49:36.914757+0000","last_fullsized":"2024-04-21T08:49:36.914757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:27.290790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:50:22.961710+0000","last_change":"2024-04-21T08:48:45.558541+0000","last_active":"2024-04-21T08:50:22.961710+0000","last_peered":"2024-04-21T08:50:22.961710+0000","last_clean":"2024-04-21T08:50:22.961710+0000","last_became_active":"2024-04-21T08:48:45.558186+0000","last_became_peered":"2024-04-21T08:48:45.558186+0000","last_unstale":"2024-04-21T08:50:22.961710+0000","last_undegraded":"2024-04-21T08:50:22.961710+0000","last_fullsized":"2024-04-21T08:50:22.961710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:02.919144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913013+0000","last_change":"2024-04-21T08:48:50.770838+0000","last_active":"2024-04-21T08:49:36.913013+0000","last_peered":"2024-04-21T08:49:36.913013+0000","last_clean":"2024-04-21T08:49:36.913013+0000","last_became_active":"2024-04-21T08:48:50.770447+0000","last_became_peered":"2024-04-21T08:48:50.770447+0000","last_unstale":"2024-04-21T08:49:36.913013+0000","last_undegraded":"2024-04-21T08:49:36.913013+0000","last_fullsized":"2024-04-21T08:49:36.913013+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:48:21.010284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.962936+0000","last_change":"2024-04-21T08:48:46.478145+0000","last_active":"2024-04-21T08:49:36.962936+0000","last_peered":"2024-04-21T08:49:36.962936+0000","last_clean":"2024-04-21T08:49:36.962936+0000","last_became_active":"2024-04-21T08:48:46.477813+0000","last_became_peered":"2024-04-21T08:48:46.477813+0000","last_unstale":"2024-04-21T08:49:36.962936+0000","last_undegraded":"2024-04-21T08:49:36.962936+0000","last_fullsized":"2024-04-21T08:49:36.962936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:15.917619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907595+0000","last_change":"2024-04-21T08:48:45.545269+0000","last_active":"2024-04-21T08:49:36.907595+0000","last_peered":"2024-04-21T08:49:36.907595+0000","last_clean":"2024-04-21T08:49:36.907595+0000","last_became_active":"2024-04-21T08:48:45.544887+0000","last_became_peered":"2024-04-21T08:48:45.544887+0000","last_unstale":"2024-04-21T08:49:36.907595+0000","last_undegraded":"2024-04-21T08:49:36.907595+0000","last_fullsized":"2024-04-21T08:49:36.907595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:27.222570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.912256+0000","last_change":"2024-04-21T08:47:34.198161+0000","last_active":"2024-04-21T08:49:36.912256+0000","last_peered":"2024-04-21T08:49:36.912256+0000","last_clean":"2024-04-21T08:49:36.912256+0000","last_became_active":"2024-04-21T08:47:34.197654+0000","last_became_peered":"2024-04-21T08:47:34.197654+0000","last_unstale":"2024-04-21T08:49:36.912256+0000","last_undegraded":"2024-04-21T08:49:36.912256+0000","last_fullsized":"2024-04-21T08:49:36.912256+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_clean_scrub_stamp":"2024-04-21T08:44:05.493464+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:58:58.295783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.914175+0000","last_change":"2024-04-21T08:48:50.770826+0000","last_active":"2024-04-21T08:49:36.914175+0000","last_peered":"2024-04-21T08:49:36.914175+0000","last_clean":"2024-04-21T08:49:36.914175+0000","last_became_active":"2024-04-21T08:48:50.770372+0000","last_became_peered":"2024-04-21T08:48:50.770372+0000","last_unstale":"2024-04-21T08:49:36.914175+0000","last_undegraded":"2024-04-21T08:49:36.914175+0000","last_fullsized":"2024-04-21T08:49:36.914175+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:59.637874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905914+0000","last_change":"2024-04-21T08:48:46.459330+0000","last_active":"2024-04-21T08:49:36.905914+0000","last_peered":"2024-04-21T08:49:36.905914+0000","last_clean":"2024-04-21T08:49:36.905914+0000","last_became_active":"2024-04-21T08:48:46.458963+0000","last_became_peered":"2024-04-21T08:48:46.458963+0000","last_unstale":"2024-04-21T08:49:36.905914+0000","last_undegraded":"2024-04-21T08:49:36.905914+0000","last_fullsized":"2024-04-21T08:49:36.905914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:10.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.917854+0000","last_change":"2024-04-21T08:49:36.917854+0000","last_active":"2024-04-21T08:49:36.917854+0000","last_peered":"2024-04-21T08:49:36.917854+0000","last_clean":"2024-04-21T08:49:36.917854+0000","last_became_active":"2024-04-21T08:49:36.915390+0000","last_became_peered":"2024-04-21T08:49:36.915390+0000","last_unstale":"2024-04-21T08:49:36.917854+0000","last_undegraded":"2024-04-21T08:49:36.917854+0000","last_fullsized":"2024-04-21T08:49:36.917854+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:45.920752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905081+0000","last_change":"2024-04-21T08:48:50.765339+0000","last_active":"2024-04-21T08:49:36.905081+0000","last_peered":"2024-04-21T08:49:36.905081+0000","last_clean":"2024-04-21T08:49:36.905081+0000","last_became_active":"2024-04-21T08:48:50.764858+0000","last_became_peered":"2024-04-21T08:48:50.764858+0000","last_unstale":"2024-04-21T08:49:36.905081+0000","last_undegraded":"2024-04-21T08:49:36.905081+0000","last_fullsized":"2024-04-21T08:49:36.905081+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:26.075041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908721+0000","last_change":"2024-04-21T08:48:45.557768+0000","last_active":"2024-04-21T08:49:36.908721+0000","last_peered":"2024-04-21T08:49:36.908721+0000","last_clean":"2024-04-21T08:49:36.908721+0000","last_became_active":"2024-04-21T08:48:45.557448+0000","last_became_peered":"2024-04-21T08:48:45.557448+0000","last_unstale":"2024-04-21T08:49:36.908721+0000","last_undegraded":"2024-04-21T08:49:36.908721+0000","last_fullsized":"2024-04-21T08:49:36.908721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:09.063913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906011+0000","last_change":"2024-04-21T08:48:46.459425+0000","last_active":"2024-04-21T08:49:36.906011+0000","last_peered":"2024-04-21T08:49:36.906011+0000","last_clean":"2024-04-21T08:49:36.906011+0000","last_became_active":"2024-04-21T08:48:46.459020+0000","last_became_peered":"2024-04-21T08:48:46.459020+0000","last_unstale":"2024-04-21T08:49:36.906011+0000","last_undegraded":"2024-04-21T08:49:36.906011+0000","last_fullsized":"2024-04-21T08:49:36.906011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:08:39.042825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.962847+0000","last_change":"2024-04-21T08:48:50.765740+0000","last_active":"2024-04-21T08:49:36.962847+0000","last_peered":"2024-04-21T08:49:36.962847+0000","last_clean":"2024-04-21T08:49:36.962847+0000","last_became_active":"2024-04-21T08:48:50.765359+0000","last_became_peered":"2024-04-21T08:48:50.765359+0000","last_unstale":"2024-04-21T08:49:36.962847+0000","last_undegraded":"2024-04-21T08:49:36.962847+0000","last_fullsized":"2024-04-21T08:49:36.962847+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:34:42.716165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902656+0000","last_change":"2024-04-21T08:48:45.564532+0000","last_active":"2024-04-21T08:49:36.902656+0000","last_peered":"2024-04-21T08:49:36.902656+0000","last_clean":"2024-04-21T08:49:36.902656+0000","last_became_active":"2024-04-21T08:48:45.564225+0000","last_became_peered":"2024-04-21T08:48:45.564225+0000","last_unstale":"2024-04-21T08:49:36.902656+0000","last_undegraded":"2024-04-21T08:49:36.902656+0000","last_fullsized":"2024-04-21T08:49:36.902656+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:42.850371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963423+0000","last_change":"2024-04-21T08:48:46.466622+0000","last_active":"2024-04-21T08:49:36.963423+0000","last_peered":"2024-04-21T08:49:36.963423+0000","last_clean":"2024-04-21T08:49:36.963423+0000","last_became_active":"2024-04-21T08:48:46.466129+0000","last_became_peered":"2024-04-21T08:48:46.466129+0000","last_unstale":"2024-04-21T08:49:36.963423+0000","last_undegraded":"2024-04-21T08:49:36.963423+0000","last_fullsized":"2024-04-21T08:49:36.963423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:20:26.064193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902852+0000","last_change":"2024-04-21T08:48:50.759534+0000","last_active":"2024-04-21T08:49:36.902852+0000","last_peered":"2024-04-21T08:49:36.902852+0000","last_clean":"2024-04-21T08:49:36.902852+0000","last_became_active":"2024-04-21T08:48:50.759116+0000","last_became_peered":"2024-04-21T08:48:50.759116+0000","last_unstale":"2024-04-21T08:49:36.902852+0000","last_undegraded":"2024-04-21T08:49:36.902852+0000","last_fullsized":"2024-04-21T08:49:36.902852+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:02:16.200901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.907082+0000","last_change":"2024-04-21T08:48:45.545773+0000","last_active":"2024-04-21T08:49:36.907082+0000","last_peered":"2024-04-21T08:49:36.907082+0000","last_clean":"2024-04-21T08:49:36.907082+0000","last_became_active":"2024-04-21T08:48:45.545447+0000","last_became_peered":"2024-04-21T08:48:45.545447+0000","last_unstale":"2024-04-21T08:49:36.907082+0000","last_undegraded":"2024-04-21T08:49:36.907082+0000","last_fullsized":"2024-04-21T08:49:36.907082+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:59:42.171602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.964411+0000","last_change":"2024-04-21T08:49:36.964411+0000","last_active":"2024-04-21T08:49:36.964411+0000","last_peered":"2024-04-21T08:49:36.964411+0000","last_clean":"2024-04-21T08:49:36.964411+0000","last_became_active":"2024-04-21T08:49:36.964046+0000","last_became_peered":"2024-04-21T08:49:36.964046+0000","last_unstale":"2024-04-21T08:49:36.964411+0000","last_undegraded":"2024-04-21T08:49:36.964411+0000","last_fullsized":"2024-04-21T08:49:36.964411+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:29:37.223495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906935+0000","last_change":"2024-04-21T08:48:50.771085+0000","last_active":"2024-04-21T08:49:36.906935+0000","last_peered":"2024-04-21T08:49:36.906935+0000","last_clean":"2024-04-21T08:49:36.906935+0000","last_became_active":"2024-04-21T08:48:50.770650+0000","last_became_peered":"2024-04-21T08:48:50.770650+0000","last_unstale":"2024-04-21T08:49:36.906935+0000","last_undegraded":"2024-04-21T08:49:36.906935+0000","last_fullsized":"2024-04-21T08:49:36.906935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:43:25.933231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.928993+0000","last_change":"2024-04-21T08:49:36.928993+0000","last_active":"2024-04-21T08:49:36.928993+0000","last_peered":"2024-04-21T08:49:36.928993+0000","last_clean":"2024-04-21T08:49:36.928993+0000","last_became_active":"2024-04-21T08:49:36.911247+0000","last_became_peered":"2024-04-21T08:49:36.911247+0000","last_unstale":"2024-04-21T08:49:36.928993+0000","last_undegraded":"2024-04-21T08:49:36.928993+0000","last_fullsized":"2024-04-21T08:49:36.928993+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:13.481670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908188+0000","last_change":"2024-04-21T08:48:46.465475+0000","last_active":"2024-04-21T08:49:36.908188+0000","last_peered":"2024-04-21T08:49:36.908188+0000","last_clean":"2024-04-21T08:49:36.908188+0000","last_became_active":"2024-04-21T08:48:46.465138+0000","last_became_peered":"2024-04-21T08:48:46.465138+0000","last_unstale":"2024-04-21T08:49:36.908188+0000","last_undegraded":"2024-04-21T08:49:36.908188+0000","last_fullsized":"2024-04-21T08:49:36.908188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:12:53.521547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902635+0000","last_change":"2024-04-21T08:48:50.765765+0000","last_active":"2024-04-21T08:49:36.902635+0000","last_peered":"2024-04-21T08:49:36.902635+0000","last_clean":"2024-04-21T08:49:36.902635+0000","last_became_active":"2024-04-21T08:48:50.765324+0000","last_became_peered":"2024-04-21T08:48:50.765324+0000","last_unstale":"2024-04-21T08:49:36.902635+0000","last_undegraded":"2024-04-21T08:49:36.902635+0000","last_fullsized":"2024-04-21T08:49:36.902635+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:41:37.846768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.962923+0000","last_change":"2024-04-21T08:48:45.557400+0000","last_active":"2024-04-21T08:49:36.962923+0000","last_peered":"2024-04-21T08:49:36.962923+0000","last_clean":"2024-04-21T08:49:36.962923+0000","last_became_active":"2024-04-21T08:48:45.555012+0000","last_became_peered":"2024-04-21T08:48:45.555012+0000","last_unstale":"2024-04-21T08:49:36.962923+0000","last_undegraded":"2024-04-21T08:49:36.962923+0000","last_fullsized":"2024-04-21T08:49:36.962923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:00:31.254353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905661+0000","last_change":"2024-04-21T08:48:46.470042+0000","last_active":"2024-04-21T08:49:36.905661+0000","last_peered":"2024-04-21T08:49:36.905661+0000","last_clean":"2024-04-21T08:49:36.905661+0000","last_became_active":"2024-04-21T08:48:46.469716+0000","last_became_peered":"2024-04-21T08:48:46.469716+0000","last_unstale":"2024-04-21T08:49:36.905661+0000","last_undegraded":"2024-04-21T08:49:36.905661+0000","last_fullsized":"2024-04-21T08:49:36.905661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:09.934751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908352+0000","last_change":"2024-04-21T08:48:50.770889+0000","last_active":"2024-04-21T08:49:36.908352+0000","last_peered":"2024-04-21T08:49:36.908352+0000","last_clean":"2024-04-21T08:49:36.908352+0000","last_became_active":"2024-04-21T08:48:50.769980+0000","last_became_peered":"2024-04-21T08:48:50.769980+0000","last_unstale":"2024-04-21T08:49:36.908352+0000","last_undegraded":"2024-04-21T08:49:36.908352+0000","last_fullsized":"2024-04-21T08:49:36.908352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:44.631633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902970+0000","last_change":"2024-04-21T08:48:45.555479+0000","last_active":"2024-04-21T08:49:36.902970+0000","last_peered":"2024-04-21T08:49:36.902970+0000","last_clean":"2024-04-21T08:49:36.902970+0000","last_became_active":"2024-04-21T08:48:45.555162+0000","last_became_peered":"2024-04-21T08:48:45.555162+0000","last_unstale":"2024-04-21T08:49:36.902970+0000","last_undegraded":"2024-04-21T08:49:36.902970+0000","last_fullsized":"2024-04-21T08:49:36.902970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:15:20.262798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963322+0000","last_change":"2024-04-21T08:48:46.467962+0000","last_active":"2024-04-21T08:49:36.963322+0000","last_peered":"2024-04-21T08:49:36.963322+0000","last_clean":"2024-04-21T08:49:36.963322+0000","last_became_active":"2024-04-21T08:48:46.467561+0000","last_became_peered":"2024-04-21T08:48:46.467561+0000","last_unstale":"2024-04-21T08:49:36.963322+0000","last_undegraded":"2024-04-21T08:49:36.963322+0000","last_fullsized":"2024-04-21T08:49:36.963322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:18.262009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906737+0000","last_change":"2024-04-21T08:48:50.767678+0000","last_active":"2024-04-21T08:49:36.906737+0000","last_peered":"2024-04-21T08:49:36.906737+0000","last_clean":"2024-04-21T08:49:36.906737+0000","last_became_active":"2024-04-21T08:48:50.767309+0000","last_became_peered":"2024-04-21T08:48:50.767309+0000","last_unstale":"2024-04-21T08:49:36.906737+0000","last_undegraded":"2024-04-21T08:49:36.906737+0000","last_fullsized":"2024-04-21T08:49:36.906737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:16:40.065043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.908504+0000","last_change":"2024-04-21T08:48:45.553321+0000","last_active":"2024-04-21T08:49:36.908504+0000","last_peered":"2024-04-21T08:49:36.908504+0000","last_clean":"2024-04-21T08:49:36.908504+0000","last_became_active":"2024-04-21T08:48:45.552871+0000","last_became_peered":"2024-04-21T08:48:45.552871+0000","last_unstale":"2024-04-21T08:49:36.908504+0000","last_undegraded":"2024-04-21T08:49:36.908504+0000","last_fullsized":"2024-04-21T08:49:36.908504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:16:27.339801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.912942+0000","last_change":"2024-04-21T08:48:46.473453+0000","last_active":"2024-04-21T08:49:36.912942+0000","last_peered":"2024-04-21T08:49:36.912942+0000","last_clean":"2024-04-21T08:49:36.912942+0000","last_became_active":"2024-04-21T08:48:46.472613+0000","last_became_peered":"2024-04-21T08:48:46.472613+0000","last_unstale":"2024-04-21T08:49:36.912942+0000","last_undegraded":"2024-04-21T08:49:36.912942+0000","last_fullsized":"2024-04-21T08:49:36.912942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:43:58.995997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906527+0000","last_change":"2024-04-21T08:48:50.765282+0000","last_active":"2024-04-21T08:49:36.906527+0000","last_peered":"2024-04-21T08:49:36.906527+0000","last_clean":"2024-04-21T08:49:36.906527+0000","last_became_active":"2024-04-21T08:48:50.764891+0000","last_became_peered":"2024-04-21T08:48:50.764891+0000","last_unstale":"2024-04-21T08:49:36.906527+0000","last_undegraded":"2024-04-21T08:49:36.906527+0000","last_fullsized":"2024-04-21T08:49:36.906527+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:18:40.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.927020+0000","last_change":"2024-04-21T08:49:36.926642+0000","last_active":"2024-04-21T08:49:36.927020+0000","last_peered":"2024-04-21T08:49:36.927020+0000","last_clean":"2024-04-21T08:49:36.927020+0000","last_became_active":"2024-04-21T08:49:36.910453+0000","last_became_peered":"2024-04-21T08:49:36.910453+0000","last_unstale":"2024-04-21T08:49:36.927020+0000","last_undegraded":"2024-04-21T08:49:36.927020+0000","last_fullsized":"2024-04-21T08:49:36.927020+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:19.290010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913362+0000","last_change":"2024-04-21T08:48:46.470058+0000","last_active":"2024-04-21T08:49:36.913362+0000","last_peered":"2024-04-21T08:49:36.913362+0000","last_clean":"2024-04-21T08:49:36.913362+0000","last_became_active":"2024-04-21T08:48:46.469485+0000","last_became_peered":"2024-04-21T08:48:46.469485+0000","last_unstale":"2024-04-21T08:49:36.913362+0000","last_undegraded":"2024-04-21T08:49:36.913362+0000","last_fullsized":"2024-04-21T08:49:36.913362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:33:13.212087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905265+0000","last_change":"2024-04-21T08:48:50.762944+0000","last_active":"2024-04-21T08:49:36.905265+0000","last_peered":"2024-04-21T08:49:36.905265+0000","last_clean":"2024-04-21T08:49:36.905265+0000","last_became_active":"2024-04-21T08:48:50.762508+0000","last_became_peered":"2024-04-21T08:48:50.762508+0000","last_unstale":"2024-04-21T08:49:36.905265+0000","last_undegraded":"2024-04-21T08:49:36.905265+0000","last_fullsized":"2024-04-21T08:49:36.905265+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:35.393339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.914358+0000","last_change":"2024-04-21T08:48:45.560361+0000","last_active":"2024-04-21T08:49:36.914358+0000","last_peered":"2024-04-21T08:49:36.914358+0000","last_clean":"2024-04-21T08:49:36.914358+0000","last_became_active":"2024-04-21T08:48:45.559946+0000","last_became_peered":"2024-04-21T08:48:45.559946+0000","last_unstale":"2024-04-21T08:49:36.914358+0000","last_undegraded":"2024-04-21T08:49:36.914358+0000","last_fullsized":"2024-04-21T08:49:36.914358+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:55.599714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.912765+0000","last_change":"2024-04-21T08:48:46.472706+0000","last_active":"2024-04-21T08:49:36.912765+0000","last_peered":"2024-04-21T08:49:36.912765+0000","last_clean":"2024-04-21T08:49:36.912765+0000","last_became_active":"2024-04-21T08:48:46.472378+0000","last_became_peered":"2024-04-21T08:48:46.472378+0000","last_unstale":"2024-04-21T08:49:36.912765+0000","last_undegraded":"2024-04-21T08:49:36.912765+0000","last_fullsized":"2024-04-21T08:49:36.912765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:58.228899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913521+0000","last_change":"2024-04-21T08:48:50.760610+0000","last_active":"2024-04-21T08:49:36.913521+0000","last_peered":"2024-04-21T08:49:36.913521+0000","last_clean":"2024-04-21T08:49:36.913521+0000","last_became_active":"2024-04-21T08:48:50.759991+0000","last_became_peered":"2024-04-21T08:48:50.759991+0000","last_unstale":"2024-04-21T08:49:36.913521+0000","last_undegraded":"2024-04-21T08:49:36.913521+0000","last_fullsized":"2024-04-21T08:49:36.913521+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:43.649701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906406+0000","last_change":"2024-04-21T08:48:45.560773+0000","last_active":"2024-04-21T08:49:36.906406+0000","last_peered":"2024-04-21T08:49:36.906406+0000","last_clean":"2024-04-21T08:49:36.906406+0000","last_became_active":"2024-04-21T08:48:45.560431+0000","last_became_peered":"2024-04-21T08:48:45.560431+0000","last_unstale":"2024-04-21T08:49:36.906406+0000","last_undegraded":"2024-04-21T08:49:36.906406+0000","last_fullsized":"2024-04-21T08:49:36.906406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:07.858838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913224+0000","last_change":"2024-04-21T08:48:46.475983+0000","last_active":"2024-04-21T08:49:36.913224+0000","last_peered":"2024-04-21T08:49:36.913224+0000","last_clean":"2024-04-21T08:49:36.913224+0000","last_became_active":"2024-04-21T08:48:46.475667+0000","last_became_peered":"2024-04-21T08:48:46.475667+0000","last_unstale":"2024-04-21T08:49:36.913224+0000","last_undegraded":"2024-04-21T08:49:36.913224+0000","last_fullsized":"2024-04-21T08:49:36.913224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:28:41.013111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905739+0000","last_change":"2024-04-21T08:48:46.464675+0000","last_active":"2024-04-21T08:49:36.905739+0000","last_peered":"2024-04-21T08:49:36.905739+0000","last_clean":"2024-04-21T08:49:36.905739+0000","last_became_active":"2024-04-21T08:48:46.464314+0000","last_became_peered":"2024-04-21T08:48:46.464314+0000","last_unstale":"2024-04-21T08:49:36.905739+0000","last_undegraded":"2024-04-21T08:49:36.905739+0000","last_fullsized":"2024-04-21T08:49:36.905739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:53.048400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.911025+0000","last_change":"2024-04-21T08:49:36.911025+0000","last_active":"2024-04-21T08:49:36.911025+0000","last_peered":"2024-04-21T08:49:36.911025+0000","last_clean":"2024-04-21T08:49:36.911025+0000","last_became_active":"2024-04-21T08:49:36.909942+0000","last_became_peered":"2024-04-21T08:49:36.909942+0000","last_unstale":"2024-04-21T08:49:36.911025+0000","last_undegraded":"2024-04-21T08:49:36.911025+0000","last_fullsized":"2024-04-21T08:49:36.911025+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:39.864253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.902957+0000","last_change":"2024-04-21T08:48:46.467867+0000","last_active":"2024-04-21T08:49:36.902957+0000","last_peered":"2024-04-21T08:49:36.902957+0000","last_clean":"2024-04-21T08:49:36.902957+0000","last_became_active":"2024-04-21T08:48:46.465917+0000","last_became_peered":"2024-04-21T08:48:46.465917+0000","last_unstale":"2024-04-21T08:49:36.902957+0000","last_undegraded":"2024-04-21T08:49:36.902957+0000","last_fullsized":"2024-04-21T08:49:36.902957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:19:56.824998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905483+0000","last_change":"2024-04-21T08:48:46.467159+0000","last_active":"2024-04-21T08:49:36.905483+0000","last_peered":"2024-04-21T08:49:36.905483+0000","last_clean":"2024-04-21T08:49:36.905483+0000","last_became_active":"2024-04-21T08:48:46.466639+0000","last_became_peered":"2024-04-21T08:48:46.466639+0000","last_unstale":"2024-04-21T08:49:36.905483+0000","last_undegraded":"2024-04-21T08:49:36.905483+0000","last_fullsized":"2024-04-21T08:49:36.905483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:54:44.772535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905904+0000","last_change":"2024-04-21T08:48:46.466403+0000","last_active":"2024-04-21T08:49:36.905904+0000","last_peered":"2024-04-21T08:49:36.905904+0000","last_clean":"2024-04-21T08:49:36.905904+0000","last_became_active":"2024-04-21T08:48:46.466060+0000","last_became_peered":"2024-04-21T08:48:46.466060+0000","last_unstale":"2024-04-21T08:49:36.905904+0000","last_undegraded":"2024-04-21T08:49:36.905904+0000","last_fullsized":"2024-04-21T08:49:36.905904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:14.374193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.909134+0000","last_change":"2024-04-21T08:49:36.909134+0000","last_active":"2024-04-21T08:49:36.909134+0000","last_peered":"2024-04-21T08:49:36.909134+0000","last_clean":"2024-04-21T08:49:36.909134+0000","last_became_active":"2024-04-21T08:49:36.908723+0000","last_became_peered":"2024-04-21T08:49:36.908723+0000","last_unstale":"2024-04-21T08:49:36.909134+0000","last_undegraded":"2024-04-21T08:49:36.909134+0000","last_fullsized":"2024-04-21T08:49:36.909134+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:28.123016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.904044+0000","last_change":"2024-04-21T08:48:46.464098+0000","last_active":"2024-04-21T08:49:36.904044+0000","last_peered":"2024-04-21T08:49:36.904044+0000","last_clean":"2024-04-21T08:49:36.904044+0000","last_became_active":"2024-04-21T08:48:46.463756+0000","last_became_peered":"2024-04-21T08:48:46.463756+0000","last_unstale":"2024-04-21T08:49:36.904044+0000","last_undegraded":"2024-04-21T08:49:36.904044+0000","last_fullsized":"2024-04-21T08:49:36.904044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:31.725155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.963509+0000","last_change":"2024-04-21T08:48:46.467372+0000","last_active":"2024-04-21T08:49:36.963509+0000","last_peered":"2024-04-21T08:49:36.963509+0000","last_clean":"2024-04-21T08:49:36.963509+0000","last_became_active":"2024-04-21T08:48:46.467043+0000","last_became_peered":"2024-04-21T08:48:46.467043+0000","last_unstale":"2024-04-21T08:49:36.963509+0000","last_undegraded":"2024-04-21T08:49:36.963509+0000","last_fullsized":"2024-04-21T08:49:36.963509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:41.889994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.913827+0000","last_change":"2024-04-21T08:48:46.460604+0000","last_active":"2024-04-21T08:49:36.913827+0000","last_peered":"2024-04-21T08:49:36.913827+0000","last_clean":"2024-04-21T08:49:36.913827+0000","last_became_active":"2024-04-21T08:48:46.460276+0000","last_became_peered":"2024-04-21T08:48:46.460276+0000","last_unstale":"2024-04-21T08:49:36.913827+0000","last_undegraded":"2024-04-21T08:49:36.913827+0000","last_fullsized":"2024-04-21T08:49:36.913827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:30.194808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.962941+0000","last_change":"2024-04-21T08:48:46.463665+0000","last_active":"2024-04-21T08:49:36.962941+0000","last_peered":"2024-04-21T08:49:36.962941+0000","last_clean":"2024-04-21T08:49:36.962941+0000","last_became_active":"2024-04-21T08:48:46.463309+0000","last_became_peered":"2024-04-21T08:48:46.463309+0000","last_unstale":"2024-04-21T08:49:36.962941+0000","last_undegraded":"2024-04-21T08:49:36.962941+0000","last_fullsized":"2024-04-21T08:49:36.962941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:12:36.436565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.906947+0000","last_change":"2024-04-21T08:48:46.476845+0000","last_active":"2024-04-21T08:49:36.906947+0000","last_peered":"2024-04-21T08:49:36.906947+0000","last_clean":"2024-04-21T08:49:36.906947+0000","last_became_active":"2024-04-21T08:48:46.476507+0000","last_became_peered":"2024-04-21T08:48:46.476507+0000","last_unstale":"2024-04-21T08:49:36.906947+0000","last_undegraded":"2024-04-21T08:49:36.906947+0000","last_fullsized":"2024-04-21T08:49:36.906947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:47.424576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905401+0000","last_change":"2024-04-21T08:48:46.469316+0000","last_active":"2024-04-21T08:49:36.905401+0000","last_peered":"2024-04-21T08:49:36.905401+0000","last_clean":"2024-04-21T08:49:36.905401+0000","last_became_active":"2024-04-21T08:48:46.468852+0000","last_became_peered":"2024-04-21T08:48:46.468852+0000","last_unstale":"2024-04-21T08:49:36.905401+0000","last_undegraded":"2024-04-21T08:49:36.905401+0000","last_fullsized":"2024-04-21T08:49:36.905401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:05:03.664279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:49:36.905201+0000","last_change":"2024-04-21T08:48:46.466703+0000","last_active":"2024-04-21T08:49:36.905201+0000","last_peered":"2024-04-21T08:49:36.905201+0000","last_clean":"2024-04-21T08:49:36.905201+0000","last_became_active":"2024-04-21T08:48:46.466395+0000","last_became_peered":"2024-04-21T08:48:46.466395+0000","last_unstale":"2024-04-21T08:49:36.905201+0000","last_undegraded":"2024-04-21T08:49:36.905201+0000","last_fullsized":"2024-04-21T08:49:36.905201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:08:56.956283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":248791,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67357,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":462,"num_read_kb":303,"num_write":161,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":202071,"data_compressed":4833,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383711,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":500872,"data_compressed":20609,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645351,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1138688,"data_stored":1151775,"data_compressed":58229,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808874,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1015808,"data_stored":505092,"data_compressed":20609,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":500804,"data_compressed":20609,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1060864,"data_stored":1096825,"data_compressed":56618,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":991232,"data_stored":496172,"data_compressed":20609,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":559438,"data_compressed":22220,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":497560,"data_compressed":20609,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32228,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712924,"statfs":{"total":95995035648,"available":95962034176,"internally_reserved":0,"allocated":1077248,"data_stored":560206,"data_compressed":22220,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":496366,"data_compressed":20609,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920837,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":501795,"data_compressed":20609,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1069056,"data_stored":1092022,"data_compressed":56618,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34373,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34373,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63356,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":64124,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65325,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:50:29.046 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-21T08:50:29.046 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-21T08:50:29.046 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-21T08:50:29.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph health --format=json 2024-04-21T08:50:29.336 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:29.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:29 smithi028 ceph-mon[28003]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:50:29.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:29 smithi028 ceph-mon[28003]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:50:29.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:29 smithi152 ceph-mon[31401]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:50:29.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:29 smithi152 ceph-mon[31401]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:50:29.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:29 smithi171 ceph-mon[31358]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:50:29.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:29 smithi171 ceph-mon[31358]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:50:31.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.158+0000 7f4944bf7700 1 -- 172.21.15.28:0/1383465259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f494006bbb0 msgr2=0x7f494010f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:31.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.158+0000 7f4944bf7700 1 --2- 172.21.15.28:0/1383465259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f494006bbb0 0x7f494010f930 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4930009990 tx=0x7f4930009ca0 comp rx=0 tx=0).stop 2024-04-21T08:50:31.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.158+0000 7f4944bf7700 1 -- 172.21.15.28:0/1383465259 shutdown_connections 2024-04-21T08:50:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.158+0000 7f4944bf7700 1 --2- 172.21.15.28:0/1383465259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f494006bbb0 0x7f494010f930 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.158+0000 7f4944bf7700 1 --2- 172.21.15.28:0/1383465259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4940108a60 0x7f4940108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.158+0000 7f4944bf7700 1 --2- 172.21.15.28:0/1383465259 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f494006b1f0 0x7f494006b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.158+0000 7f4944bf7700 1 -- 172.21.15.28:0/1383465259 >> 172.21.15.28:0/1383465259 conn(0x7f49400789e0 msgr2=0x7f4940078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.158+0000 7f4944bf7700 1 -- 172.21.15.28:0/1383465259 shutdown_connections 2024-04-21T08:50:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.158+0000 7f4944bf7700 1 -- 172.21.15.28:0/1383465259 wait complete. 2024-04-21T08:50:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.158+0000 7f4944bf7700 1 Processor -- start 2024-04-21T08:50:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f4944bf7700 1 -- start start 2024-04-21T08:50:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f4944bf7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f494006b1f0 0x7f49401019b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f4944bf7700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f494006bbb0 0x7f4940101ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f4944bf7700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4940108a60 0x7f4940102430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f4944bf7700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4940112c60 con 0x7f494006b1f0 2024-04-21T08:50:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f4944bf7700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4940112ae0 con 0x7f494006bbb0 2024-04-21T08:50:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f4944bf7700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4940112de0 con 0x7f4940108a60 2024-04-21T08:50:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f493e59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f494006b1f0 0x7f49401019b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f493e59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f494006b1f0 0x7f49401019b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36726/0 (socket says 172.21.15.28:36726) 2024-04-21T08:50:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f493e59c700 1 -- 172.21.15.28:0/1643831758 learned_addr learned my addr 172.21.15.28:0/1643831758 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f493ed9d700 1 --2- 172.21.15.28:0/1643831758 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4940108a60 0x7f4940102430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f493dd9b700 1 --2- 172.21.15.28:0/1643831758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f494006bbb0 0x7f4940101ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f493e59c700 1 -- 172.21.15.28:0/1643831758 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4940108a60 msgr2=0x7f4940102430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f493e59c700 1 --2- 172.21.15.28:0/1643831758 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4940108a60 0x7f4940102430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f493e59c700 1 -- 172.21.15.28:0/1643831758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f494006bbb0 msgr2=0x7f4940101ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f493e59c700 1 --2- 172.21.15.28:0/1643831758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f494006bbb0 0x7f4940101ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f493e59c700 1 -- 172.21.15.28:0/1643831758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49401a7640 con 0x7f494006b1f0 2024-04-21T08:50:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f493dd9b700 1 --2- 172.21.15.28:0/1643831758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f494006bbb0 0x7f4940101ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.159+0000 7f493e59c700 1 --2- 172.21.15.28:0/1643831758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f494006b1f0 0x7f49401019b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f493800ada0 tx=0x7f493800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.160+0000 7f49377fe700 1 -- 172.21.15.28:0/1643831758 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4938017020 con 0x7f494006b1f0 2024-04-21T08:50:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.160+0000 7f4944bf7700 1 -- 172.21.15.28:0/1643831758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49401a7930 con 0x7f494006b1f0 2024-04-21T08:50:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.160+0000 7f49377fe700 1 -- 172.21.15.28:0/1643831758 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f493800ce30 con 0x7f494006b1f0 2024-04-21T08:50:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.160+0000 7f49377fe700 1 -- 172.21.15.28:0/1643831758 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4938005420 con 0x7f494006b1f0 2024-04-21T08:50:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.160+0000 7f4944bf7700 1 -- 172.21.15.28:0/1643831758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49401a7e70 con 0x7f494006b1f0 2024-04-21T08:50:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.161+0000 7f4944bf7700 1 -- 172.21.15.28:0/1643831758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49401041a0 con 0x7f494006b1f0 2024-04-21T08:50:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.162+0000 7f49377fe700 1 -- 172.21.15.28:0/1643831758 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4938004ad0 con 0x7f494006b1f0 2024-04-21T08:50:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.163+0000 7f49377fe700 1 --2- 172.21.15.28:0/1643831758 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f492c0a3cd0 0x7f492c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.163+0000 7f493dd9b700 1 --2- 172.21.15.28:0/1643831758 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f492c0a3cd0 0x7f492c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.163+0000 7f49377fe700 1 -- 172.21.15.28:0/1643831758 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f49380d0000 con 0x7f494006b1f0 2024-04-21T08:50:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.163+0000 7f493dd9b700 1 --2- 172.21.15.28:0/1643831758 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f492c0a3cd0 0x7f492c0a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4928005fd0 tx=0x7f4928005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.167+0000 7f49377fe700 1 -- 172.21.15.28:0/1643831758 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4938021020 con 0x7f494006b1f0 2024-04-21T08:50:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.423+0000 7f4944bf7700 1 -- 172.21.15.28:0/1643831758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f494002df80 con 0x7f494006b1f0 2024-04-21T08:50:31.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:31 smithi028 ceph-mon[28003]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.423+0000 7f49377fe700 1 -- 172.21.15.28:0/1643831758 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f493809c5d0 con 0x7f494006b1f0 2024-04-21T08:50:31.424 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:50:31.424 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-21T08:50:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.426+0000 7f4944bf7700 1 -- 172.21.15.28:0/1643831758 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f492c0a3cd0 msgr2=0x7f492c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.426+0000 7f4944bf7700 1 --2- 172.21.15.28:0/1643831758 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f492c0a3cd0 0x7f492c0a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4928005fd0 tx=0x7f4928005dc0 comp rx=0 tx=0).stop 2024-04-21T08:50:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.426+0000 7f4944bf7700 1 -- 172.21.15.28:0/1643831758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f494006b1f0 msgr2=0x7f49401019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.426+0000 7f4944bf7700 1 --2- 172.21.15.28:0/1643831758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f494006b1f0 0x7f49401019b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f493800ada0 tx=0x7f493800c330 comp rx=0 tx=0).stop 2024-04-21T08:50:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.426+0000 7f4944bf7700 1 -- 172.21.15.28:0/1643831758 shutdown_connections 2024-04-21T08:50:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.426+0000 7f4944bf7700 1 --2- 172.21.15.28:0/1643831758 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f492c0a3cd0 0x7f492c0a6190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.426+0000 7f4944bf7700 1 --2- 172.21.15.28:0/1643831758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f494006b1f0 0x7f49401019b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.426+0000 7f4944bf7700 1 --2- 172.21.15.28:0/1643831758 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4940108a60 0x7f4940102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.426+0000 7f4944bf7700 1 --2- 172.21.15.28:0/1643831758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f494006bbb0 0x7f4940101ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.426+0000 7f4944bf7700 1 -- 172.21.15.28:0/1643831758 >> 172.21.15.28:0/1643831758 conn(0x7f49400789e0 msgr2=0x7f49400ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.426+0000 7f4944bf7700 1 -- 172.21.15.28:0/1643831758 shutdown_connections 2024-04-21T08:50:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:31.426+0000 7f4944bf7700 1 -- 172.21.15.28:0/1643831758 wait complete. 2024-04-21T08:50:31.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:31 smithi152 ceph-mon[31401]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:31.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:31 smithi171 ceph-mon[31358]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:32.248 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-21T08:50:32.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:50:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 -- 172.21.15.28:0/40093978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5094105370 msgr2=0x7f5094107760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 --2- 172.21.15.28:0/40093978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5094105370 0x7f5094107760 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f507c009900 tx=0x7f507c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 -- 172.21.15.28:0/40093978 shutdown_connections 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 --2- 172.21.15.28:0/40093978 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5094107ca0 0x7f509410a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 --2- 172.21.15.28:0/40093978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5094105370 0x7f5094107760 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 --2- 172.21.15.28:0/40093978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5094102a40 0x7f5094104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 -- 172.21.15.28:0/40093978 >> 172.21.15.28:0/40093978 conn(0x7f50940fc470 msgr2=0x7f50940fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 -- 172.21.15.28:0/40093978 shutdown_connections 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 -- 172.21.15.28:0/40093978 wait complete. 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 Processor -- start 2024-04-21T08:50:32.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 -- start start 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5094102a40 0x7f509419a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5094105370 0x7f509419a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5094107ca0 0x7f50941a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f509410db00 con 0x7f5094105370 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f509410d980 con 0x7f5094102a40 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.361+0000 7f509972b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f509410dc80 con 0x7f5094107ca0 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f508bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5094105370 0x7f509419a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f508bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5094105370 0x7f509419a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36744/0 (socket says 172.21.15.28:36744) 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f508bfff640 1 -- 172.21.15.28:0/830526346 learned_addr learned my addr 172.21.15.28:0/830526346 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f50937fe640 1 --2- 172.21.15.28:0/830526346 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5094107ca0 0x7f50941a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:32.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f5092ffd640 1 --2- 172.21.15.28:0/830526346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5094102a40 0x7f509419a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f508bfff640 1 -- 172.21.15.28:0/830526346 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5094107ca0 msgr2=0x7f50941a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f508bfff640 1 --2- 172.21.15.28:0/830526346 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5094107ca0 0x7f50941a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f508bfff640 1 -- 172.21.15.28:0/830526346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5094102a40 msgr2=0x7f509419a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f508bfff640 1 --2- 172.21.15.28:0/830526346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5094102a40 0x7f509419a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f508bfff640 1 -- 172.21.15.28:0/830526346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50941a2000 con 0x7f5094105370 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f508bfff640 1 --2- 172.21.15.28:0/830526346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5094105370 0x7f509419a920 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f507c000c00 tx=0x7f507c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f5090ff9640 1 -- 172.21.15.28:0/830526346 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c043020 con 0x7f5094105370 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f5090ff9640 1 -- 172.21.15.28:0/830526346 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f507c004aa0 con 0x7f5094105370 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f509972b640 1 -- 172.21.15.28:0/830526346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50941a2290 con 0x7f5094105370 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f5090ff9640 1 -- 172.21.15.28:0/830526346 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c003d00 con 0x7f5094105370 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.362+0000 7f509972b640 1 -- 172.21.15.28:0/830526346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50941a2830 con 0x7f5094105370 2024-04-21T08:50:32.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.363+0000 7f509972b640 1 -- 172.21.15.28:0/830526346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50941097f0 con 0x7f5094105370 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.365+0000 7f5090ff9640 1 -- 172.21.15.28:0/830526346 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f507c046020 con 0x7f5094105370 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.365+0000 7f5090ff9640 1 --2- 172.21.15.28:0/830526346 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50800a3cc0 0x7f50800a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.365+0000 7f5092ffd640 1 --2- 172.21.15.28:0/830526346 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50800a3cc0 0x7f50800a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.365+0000 7f5090ff9640 1 -- 172.21.15.28:0/830526346 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f507c0f3180 con 0x7f5094105370 2024-04-21T08:50:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.366+0000 7f5092ffd640 1 --2- 172.21.15.28:0/830526346 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50800a3cc0 0x7f50800a6180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f50840058c0 tx=0x7f508400a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:32.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.369+0000 7f5090ff9640 1 -- 172.21.15.28:0/830526346 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f507c04d020 con 0x7f5094105370 2024-04-21T08:50:32.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.557+0000 7f509972b640 1 -- 172.21.15.28:0/830526346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f509402d810 con 0x7f5094105370 2024-04-21T08:50:32.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.558+0000 7f5090ff9640 1 -- 172.21.15.28:0/830526346 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24598 (secure 0 0 0) 0x7f507c0bf7d0 con 0x7f5094105370 2024-04-21T08:50:32.558 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:50:32.558 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:49:36.894878+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:50:32.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.560+0000 7f509972b640 1 -- 172.21.15.28:0/830526346 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50800a3cc0 msgr2=0x7f50800a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:32.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.560+0000 7f509972b640 1 --2- 172.21.15.28:0/830526346 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50800a3cc0 0x7f50800a6180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f50840058c0 tx=0x7f508400a4c0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.560+0000 7f509972b640 1 -- 172.21.15.28:0/830526346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5094105370 msgr2=0x7f509419a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:32.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.560+0000 7f509972b640 1 --2- 172.21.15.28:0/830526346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5094105370 0x7f509419a920 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f507c000c00 tx=0x7f507c004270 comp rx=0 tx=0).stop 2024-04-21T08:50:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.560+0000 7f509972b640 1 -- 172.21.15.28:0/830526346 shutdown_connections 2024-04-21T08:50:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.560+0000 7f509972b640 1 --2- 172.21.15.28:0/830526346 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5094107ca0 0x7f50941a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.560+0000 7f509972b640 1 --2- 172.21.15.28:0/830526346 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f50800a3cc0 0x7f50800a6180 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.560+0000 7f509972b640 1 --2- 172.21.15.28:0/830526346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5094105370 0x7f509419a920 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.560+0000 7f509972b640 1 --2- 172.21.15.28:0/830526346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5094102a40 0x7f509419a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.560+0000 7f509972b640 1 -- 172.21.15.28:0/830526346 >> 172.21.15.28:0/830526346 conn(0x7f50940fc470 msgr2=0x7f50941021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.561+0000 7f509972b640 1 -- 172.21.15.28:0/830526346 shutdown_connections 2024-04-21T08:50:32.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.561+0000 7f509972b640 1 -- 172.21.15.28:0/830526346 wait complete. 2024-04-21T08:50:32.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:32 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1643831758' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T08:50:32.572 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:48:43.543799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:48:44.864564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:48:48.747842+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:50:32.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:50:32.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:32 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1643831758' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T08:50:32.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:32 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1643831758' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T08:50:32.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.679+0000 7f3541fd5640 1 -- 172.21.15.28:0/4102864894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c102a10 msgr2=0x7f353c104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:32.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.679+0000 7f3541fd5640 1 --2- 172.21.15.28:0/4102864894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c102a10 0x7f353c104e00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f352c009900 tx=0x7f352c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:32.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.679+0000 7f3541fd5640 1 -- 172.21.15.28:0/4102864894 shutdown_connections 2024-04-21T08:50:32.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.679+0000 7f3541fd5640 1 --2- 172.21.15.28:0/4102864894 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f353c107c70 0x7f353c10a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.679+0000 7f3541fd5640 1 --2- 172.21.15.28:0/4102864894 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f353c105340 0x7f353c107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.679+0000 7f3541fd5640 1 --2- 172.21.15.28:0/4102864894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c102a10 0x7f353c104e00 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.679+0000 7f3541fd5640 1 -- 172.21.15.28:0/4102864894 >> 172.21.15.28:0/4102864894 conn(0x7f353c0fc400 msgr2=0x7f353c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:32.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.679+0000 7f3541fd5640 1 -- 172.21.15.28:0/4102864894 shutdown_connections 2024-04-21T08:50:32.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.679+0000 7f3541fd5640 1 -- 172.21.15.28:0/4102864894 wait complete. 2024-04-21T08:50:32.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.679+0000 7f3541fd5640 1 Processor -- start 2024-04-21T08:50:32.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.679+0000 7f3541fd5640 1 -- start start 2024-04-21T08:50:32.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f3541fd5640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f353c102a10 0x7f353c102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:32.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f3541fd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c105340 0x7f353c1009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:32.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f3541fd5640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f353c107c70 0x7f353c100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:32.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f3541fd5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f353c10db10 con 0x7f353c105340 2024-04-21T08:50:32.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f3541fd5640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f353c10d990 con 0x7f353c107c70 2024-04-21T08:50:32.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f3541fd5640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f353c10dc90 con 0x7f353c102a10 2024-04-21T08:50:32.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f353affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c105340 0x7f353c1009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:32.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f353affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c105340 0x7f353c1009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36758/0 (socket says 172.21.15.28:36758) 2024-04-21T08:50:32.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f353affd640 1 -- 172.21.15.28:0/1807971446 learned_addr learned my addr 172.21.15.28:0/1807971446 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:32.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f353b7fe640 1 --2- 172.21.15.28:0/1807971446 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f353c102a10 0x7f353c102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f353bfff640 1 --2- 172.21.15.28:0/1807971446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f353c107c70 0x7f353c100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f353affd640 1 -- 172.21.15.28:0/1807971446 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f353c102a10 msgr2=0x7f353c102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f353affd640 1 --2- 172.21.15.28:0/1807971446 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f353c102a10 0x7f353c102380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f353affd640 1 -- 172.21.15.28:0/1807971446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f353c107c70 msgr2=0x7f353c100f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f353affd640 1 --2- 172.21.15.28:0/1807971446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f353c107c70 0x7f353c100f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f353affd640 1 -- 172.21.15.28:0/1807971446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f353c101790 con 0x7f353c105340 2024-04-21T08:50:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f353affd640 1 --2- 172.21.15.28:0/1807971446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c105340 0x7f353c1009c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f353000d770 tx=0x7f353000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f3538ff9640 1 -- 172.21.15.28:0/1807971446 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530017070 con 0x7f353c105340 2024-04-21T08:50:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f3538ff9640 1 -- 172.21.15.28:0/1807971446 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3530009c30 con 0x7f353c105340 2024-04-21T08:50:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f3541fd5640 1 -- 172.21.15.28:0/1807971446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f353c1a23a0 con 0x7f353c105340 2024-04-21T08:50:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.680+0000 7f3538ff9640 1 -- 172.21.15.28:0/1807971446 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353000f620 con 0x7f353c105340 2024-04-21T08:50:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.681+0000 7f3541fd5640 1 -- 172.21.15.28:0/1807971446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f353c1a28f0 con 0x7f353c105340 2024-04-21T08:50:32.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.682+0000 7f3541fd5640 1 -- 172.21.15.28:0/1807971446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f353c0fdef0 con 0x7f353c105340 2024-04-21T08:50:32.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.683+0000 7f3538ff9640 1 -- 172.21.15.28:0/1807971446 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f353001a070 con 0x7f353c105340 2024-04-21T08:50:32.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.684+0000 7f3538ff9640 1 --2- 172.21.15.28:0/1807971446 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f35240a3c40 0x7f35240a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:32.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.684+0000 7f353b7fe640 1 --2- 172.21.15.28:0/1807971446 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f35240a3c40 0x7f35240a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:32.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.684+0000 7f3538ff9640 1 -- 172.21.15.28:0/1807971446 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f35300d0350 con 0x7f353c105340 2024-04-21T08:50:32.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.684+0000 7f353b7fe640 1 --2- 172.21.15.28:0/1807971446 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f35240a3c40 0x7f35240a6100 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f352c000c00 tx=0x7f352c02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:32.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.687+0000 7f3538ff9640 1 -- 172.21.15.28:0/1807971446 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3530009de0 con 0x7f353c105340 2024-04-21T08:50:32.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.872+0000 7f3541fd5640 1 -- 172.21.15.28:0/1807971446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f353c02d810 con 0x7f353c105340 2024-04-21T08:50:32.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.873+0000 7f3538ff9640 1 -- 172.21.15.28:0/1807971446 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f353009c240 con 0x7f353c105340 2024-04-21T08:50:32.873 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T08:50:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.875+0000 7f3541fd5640 1 -- 172.21.15.28:0/1807971446 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f35240a3c40 msgr2=0x7f35240a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.875+0000 7f3541fd5640 1 --2- 172.21.15.28:0/1807971446 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f35240a3c40 0x7f35240a6100 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f352c000c00 tx=0x7f352c02c7e0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.875+0000 7f3541fd5640 1 -- 172.21.15.28:0/1807971446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c105340 msgr2=0x7f353c1009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.875+0000 7f3541fd5640 1 --2- 172.21.15.28:0/1807971446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c105340 0x7f353c1009c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f353000d770 tx=0x7f353000dc30 comp rx=0 tx=0).stop 2024-04-21T08:50:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.875+0000 7f3541fd5640 1 -- 172.21.15.28:0/1807971446 shutdown_connections 2024-04-21T08:50:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.875+0000 7f3541fd5640 1 --2- 172.21.15.28:0/1807971446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f353c107c70 0x7f353c100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.875+0000 7f3541fd5640 1 --2- 172.21.15.28:0/1807971446 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f35240a3c40 0x7f35240a6100 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.875+0000 7f3541fd5640 1 --2- 172.21.15.28:0/1807971446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f353c105340 0x7f353c1009c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.875+0000 7f3541fd5640 1 --2- 172.21.15.28:0/1807971446 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f353c102a10 0x7f353c102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.875+0000 7f3541fd5640 1 -- 172.21.15.28:0/1807971446 >> 172.21.15.28:0/1807971446 conn(0x7f353c0fc400 msgr2=0x7f353c103470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.875+0000 7f3541fd5640 1 -- 172.21.15.28:0/1807971446 shutdown_connections 2024-04-21T08:50:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.875+0000 7f3541fd5640 1 -- 172.21.15.28:0/1807971446 wait complete. 2024-04-21T08:50:32.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:50:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.991+0000 7f2cbc650640 1 -- 172.21.15.28:0/3300239994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cb4101fb0 msgr2=0x7f2cb41023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.991+0000 7f2cbc650640 1 --2- 172.21.15.28:0/3300239994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cb4101fb0 0x7f2cb41023d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2cac009900 tx=0x7f2cac02bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.991+0000 7f2cbc650640 1 -- 172.21.15.28:0/3300239994 shutdown_connections 2024-04-21T08:50:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.991+0000 7f2cbc650640 1 --2- 172.21.15.28:0/3300239994 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cb4103b60 0x7f2cb410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.991+0000 7f2cbc650640 1 --2- 172.21.15.28:0/3300239994 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2cb41031a0 0x7f2cb4103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.991+0000 7f2cbc650640 1 --2- 172.21.15.28:0/3300239994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cb4101fb0 0x7f2cb41023d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.991+0000 7f2cbc650640 1 -- 172.21.15.28:0/3300239994 >> 172.21.15.28:0/3300239994 conn(0x7f2cb40fd520 msgr2=0x7f2cb40ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.991+0000 7f2cbc650640 1 -- 172.21.15.28:0/3300239994 shutdown_connections 2024-04-21T08:50:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.991+0000 7f2cbc650640 1 -- 172.21.15.28:0/3300239994 wait complete. 2024-04-21T08:50:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cbc650640 1 Processor -- start 2024-04-21T08:50:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cbc650640 1 -- start start 2024-04-21T08:50:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cbc650640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cb4101fb0 0x7f2cb4198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cbc650640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cb41031a0 0x7f2cb4198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cbc650640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2cb4103b60 0x7f2cb419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cbc650640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb410daf0 con 0x7f2cb4101fb0 2024-04-21T08:50:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cbc650640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb410d970 con 0x7f2cb41031a0 2024-04-21T08:50:32.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cbc650640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb410dc70 con 0x7f2cb4103b60 2024-04-21T08:50:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cba3ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cb4101fb0 0x7f2cb4198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cba3ec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cb4101fb0 0x7f2cb4198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36768/0 (socket says 172.21.15.28:36768) 2024-04-21T08:50:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cba3ec640 1 -- 172.21.15.28:0/3550273001 learned_addr learned my addr 172.21.15.28:0/3550273001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cb9beb640 1 --2- 172.21.15.28:0/3550273001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cb41031a0 0x7f2cb4198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cba3ec640 1 -- 172.21.15.28:0/3550273001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2cb4103b60 msgr2=0x7f2cb419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cba3ec640 1 --2- 172.21.15.28:0/3550273001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2cb4103b60 0x7f2cb419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cba3ec640 1 -- 172.21.15.28:0/3550273001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cb41031a0 msgr2=0x7f2cb4198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cba3ec640 1 --2- 172.21.15.28:0/3550273001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cb41031a0 0x7f2cb4198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.992+0000 7f2cba3ec640 1 -- 172.21.15.28:0/3550273001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb419fe20 con 0x7f2cb4101fb0 2024-04-21T08:50:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.993+0000 7f2cba3ec640 1 --2- 172.21.15.28:0/3550273001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cb4101fb0 0x7f2cb4198200 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2cac000c00 tx=0x7f2cac004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.993+0000 7f2cb37fe640 1 -- 172.21.15.28:0/3550273001 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac043020 con 0x7f2cb4101fb0 2024-04-21T08:50:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.993+0000 7f2cb37fe640 1 -- 172.21.15.28:0/3550273001 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cac004aa0 con 0x7f2cb4101fb0 2024-04-21T08:50:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.993+0000 7f2cbc650640 1 -- 172.21.15.28:0/3550273001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cb41a00b0 con 0x7f2cb4101fb0 2024-04-21T08:50:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.993+0000 7f2cb37fe640 1 -- 172.21.15.28:0/3550273001 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac003d00 con 0x7f2cb4101fb0 2024-04-21T08:50:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.993+0000 7f2cbc650640 1 -- 172.21.15.28:0/3550273001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cb41a0650 con 0x7f2cb4101fb0 2024-04-21T08:50:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.994+0000 7f2cbc650640 1 -- 172.21.15.28:0/3550273001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cb41093a0 con 0x7f2cb4101fb0 2024-04-21T08:50:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.995+0000 7f2cb37fe640 1 -- 172.21.15.28:0/3550273001 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2cac046020 con 0x7f2cb4101fb0 2024-04-21T08:50:32.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.996+0000 7f2cb37fe640 1 --2- 172.21.15.28:0/3550273001 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2c9c0a3cc0 0x7f2c9c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:32.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.996+0000 7f2cb9beb640 1 --2- 172.21.15.28:0/3550273001 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2c9c0a3cc0 0x7f2c9c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:32.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.996+0000 7f2cb37fe640 1 -- 172.21.15.28:0/3550273001 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f2cac0f3250 con 0x7f2cb4101fb0 2024-04-21T08:50:32.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.996+0000 7f2cb9beb640 1 --2- 172.21.15.28:0/3550273001 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2c9c0a3cc0 0x7f2c9c0a6180 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8005fd0 tx=0x7f2ca8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:32.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:32.999+0000 7f2cb37fe640 1 -- 172.21.15.28:0/3550273001 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2cac04d020 con 0x7f2cb4101fb0 2024-04-21T08:50:33.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.183+0000 7f2cbc650640 1 -- 172.21.15.28:0/3550273001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2cb4062b50 con 0x7f2cb4101fb0 2024-04-21T08:50:33.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.183+0000 7f2cb37fe640 1 -- 172.21.15.28:0/3550273001 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f2cac0bf8a0 con 0x7f2cb4101fb0 2024-04-21T08:50:33.183 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:50:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.185+0000 7f2cbc650640 1 -- 172.21.15.28:0/3550273001 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2c9c0a3cc0 msgr2=0x7f2c9c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.185+0000 7f2cbc650640 1 --2- 172.21.15.28:0/3550273001 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2c9c0a3cc0 0x7f2c9c0a6180 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8005fd0 tx=0x7f2ca8005850 comp rx=0 tx=0).stop 2024-04-21T08:50:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.185+0000 7f2cbc650640 1 -- 172.21.15.28:0/3550273001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cb4101fb0 msgr2=0x7f2cb4198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.185+0000 7f2cbc650640 1 --2- 172.21.15.28:0/3550273001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cb4101fb0 0x7f2cb4198200 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2cac000c00 tx=0x7f2cac004270 comp rx=0 tx=0).stop 2024-04-21T08:50:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.185+0000 7f2cbc650640 1 -- 172.21.15.28:0/3550273001 shutdown_connections 2024-04-21T08:50:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.185+0000 7f2cbc650640 1 --2- 172.21.15.28:0/3550273001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2cb4103b60 0x7f2cb419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.185+0000 7f2cbc650640 1 --2- 172.21.15.28:0/3550273001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2cb41031a0 0x7f2cb4198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.185+0000 7f2cbc650640 1 --2- 172.21.15.28:0/3550273001 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2c9c0a3cc0 0x7f2c9c0a6180 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.185+0000 7f2cbc650640 1 --2- 172.21.15.28:0/3550273001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cb4101fb0 0x7f2cb4198200 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.185+0000 7f2cbc650640 1 -- 172.21.15.28:0/3550273001 >> 172.21.15.28:0/3550273001 conn(0x7f2cb40fd520 msgr2=0x7f2cb4106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.185+0000 7f2cbc650640 1 -- 172.21.15.28:0/3550273001 shutdown_connections 2024-04-21T08:50:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.185+0000 7f2cbc650640 1 -- 172.21.15.28:0/3550273001 wait complete. 2024-04-21T08:50:33.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:50:33.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.302+0000 7f6dc17b2640 1 -- 172.21.15.28:0/934793804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc103b50 msgr2=0x7f6dbc10a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:33.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.302+0000 7f6dc17b2640 1 --2- 172.21.15.28:0/934793804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc103b50 0x7f6dbc10a3e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6db4009ca0 tx=0x7f6db402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.302+0000 7f6dc17b2640 1 -- 172.21.15.28:0/934793804 shutdown_connections 2024-04-21T08:50:33.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.302+0000 7f6dc17b2640 1 --2- 172.21.15.28:0/934793804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc103b50 0x7f6dbc10a3e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.302+0000 7f6dc17b2640 1 --2- 172.21.15.28:0/934793804 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6dbc103190 0x7f6dbc103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.302+0000 7f6dc17b2640 1 --2- 172.21.15.28:0/934793804 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6dbc101fa0 0x7f6dbc1023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.302+0000 7f6dc17b2640 1 -- 172.21.15.28:0/934793804 >> 172.21.15.28:0/934793804 conn(0x7f6dbc0fd530 msgr2=0x7f6dbc0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:33.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.302+0000 7f6dc17b2640 1 -- 172.21.15.28:0/934793804 shutdown_connections 2024-04-21T08:50:33.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.302+0000 7f6dc17b2640 1 -- 172.21.15.28:0/934793804 wait complete. 2024-04-21T08:50:33.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.303+0000 7f6dc17b2640 1 Processor -- start 2024-04-21T08:50:33.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.303+0000 7f6dc17b2640 1 -- start start 2024-04-21T08:50:33.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.303+0000 7f6dc17b2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6dbc101fa0 0x7f6dbc19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:33.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.303+0000 7f6dc17b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc103190 0x7f6dbc19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:33.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.303+0000 7f6dc17b2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6dbc103b50 0x7f6dbc1a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:33.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.303+0000 7f6dc17b2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc10daa0 con 0x7f6dbc103190 2024-04-21T08:50:33.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.303+0000 7f6dc17b2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc10d920 con 0x7f6dbc101fa0 2024-04-21T08:50:33.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.303+0000 7f6dc17b2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6dbc10dc20 con 0x7f6dbc103b50 2024-04-21T08:50:33.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.303+0000 7f6db3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc103190 0x7f6dbc19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:33.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.303+0000 7f6db3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc103190 0x7f6dbc19a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36788/0 (socket says 172.21.15.28:36788) 2024-04-21T08:50:33.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.303+0000 7f6db3fff640 1 -- 172.21.15.28:0/2532786048 learned_addr learned my addr 172.21.15.28:0/2532786048 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:33.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.304+0000 7f6dbaffd640 1 --2- 172.21.15.28:0/2532786048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6dbc101fa0 0x7f6dbc19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:33.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.304+0000 7f6db3fff640 1 -- 172.21.15.28:0/2532786048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6dbc103b50 msgr2=0x7f6dbc1a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:33.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.304+0000 7f6db3fff640 1 --2- 172.21.15.28:0/2532786048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6dbc103b50 0x7f6dbc1a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.304+0000 7f6db3fff640 1 -- 172.21.15.28:0/2532786048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6dbc101fa0 msgr2=0x7f6dbc19a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:33.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.304+0000 7f6db3fff640 1 --2- 172.21.15.28:0/2532786048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6dbc101fa0 0x7f6dbc19a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.304+0000 7f6db3fff640 1 -- 172.21.15.28:0/2532786048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dbc1a1f00 con 0x7f6dbc103190 2024-04-21T08:50:33.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.304+0000 7f6dbaffd640 1 --2- 172.21.15.28:0/2532786048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6dbc101fa0 0x7f6dbc19a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:33.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.304+0000 7f6db3fff640 1 --2- 172.21.15.28:0/2532786048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc103190 0x7f6dbc19a820 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6da400c890 tx=0x7f6da400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:33.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.304+0000 7f6db8ff9640 1 -- 172.21.15.28:0/2532786048 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4004220 con 0x7f6dbc103190 2024-04-21T08:50:33.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.304+0000 7f6db8ff9640 1 -- 172.21.15.28:0/2532786048 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6da40043c0 con 0x7f6dbc103190 2024-04-21T08:50:33.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.304+0000 7f6dc17b2640 1 -- 172.21.15.28:0/2532786048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dbc1a21f0 con 0x7f6dbc103190 2024-04-21T08:50:33.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.304+0000 7f6db8ff9640 1 -- 172.21.15.28:0/2532786048 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4004a00 con 0x7f6dbc103190 2024-04-21T08:50:33.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.304+0000 7f6dc17b2640 1 -- 172.21.15.28:0/2532786048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dbc1a2740 con 0x7f6dbc103190 2024-04-21T08:50:33.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.305+0000 7f6dc17b2640 1 -- 172.21.15.28:0/2532786048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dbc1a2a00 con 0x7f6dbc103190 2024-04-21T08:50:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.306+0000 7f6db8ff9640 1 -- 172.21.15.28:0/2532786048 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6da40079b0 con 0x7f6dbc103190 2024-04-21T08:50:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.307+0000 7f6db8ff9640 1 --2- 172.21.15.28:0/2532786048 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6da80a3c40 0x7f6da80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.307+0000 7f6dbaffd640 1 --2- 172.21.15.28:0/2532786048 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6da80a3c40 0x7f6da80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.307+0000 7f6db8ff9640 1 -- 172.21.15.28:0/2532786048 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f6da40d0720 con 0x7f6dbc103190 2024-04-21T08:50:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.307+0000 7f6dbaffd640 1 --2- 172.21.15.28:0/2532786048 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6da80a3c40 0x7f6da80a6100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc102ff0 tx=0x7f6dac005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:33.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.312+0000 7f6db8ff9640 1 -- 172.21.15.28:0/2532786048 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6da4014030 con 0x7f6dbc103190 2024-04-21T08:50:33.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.501+0000 7f6dc17b2640 1 -- 172.21.15.28:0/2532786048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6dbc1a2c70 con 0x7f6dbc103190 2024-04-21T08:50:33.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.501+0000 7f6db8ff9640 1 -- 172.21.15.28:0/2532786048 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f6da409cc00 con 0x7f6dbc103190 2024-04-21T08:50:33.501 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-21T08:50:33.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.504+0000 7f6dc17b2640 1 -- 172.21.15.28:0/2532786048 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6da80a3c40 msgr2=0x7f6da80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:33.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.504+0000 7f6dc17b2640 1 --2- 172.21.15.28:0/2532786048 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6da80a3c40 0x7f6da80a6100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc102ff0 tx=0x7f6dac005db0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.504+0000 7f6dc17b2640 1 -- 172.21.15.28:0/2532786048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc103190 msgr2=0x7f6dbc19a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:33.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.504+0000 7f6dc17b2640 1 --2- 172.21.15.28:0/2532786048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc103190 0x7f6dbc19a820 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6da400c890 tx=0x7f6da400cd50 comp rx=0 tx=0).stop 2024-04-21T08:50:33.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.504+0000 7f6dc17b2640 1 -- 172.21.15.28:0/2532786048 shutdown_connections 2024-04-21T08:50:33.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.504+0000 7f6dc17b2640 1 --2- 172.21.15.28:0/2532786048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6dbc103b50 0x7f6dbc1a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.504+0000 7f6dc17b2640 1 --2- 172.21.15.28:0/2532786048 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f6da80a3c40 0x7f6da80a6100 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.504+0000 7f6dc17b2640 1 --2- 172.21.15.28:0/2532786048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dbc103190 0x7f6dbc19a820 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.504+0000 7f6dc17b2640 1 --2- 172.21.15.28:0/2532786048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6dbc101fa0 0x7f6dbc19a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.504+0000 7f6dc17b2640 1 -- 172.21.15.28:0/2532786048 >> 172.21.15.28:0/2532786048 conn(0x7f6dbc0fd530 msgr2=0x7f6dbc074180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:33.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.504+0000 7f6dc17b2640 1 -- 172.21.15.28:0/2532786048 shutdown_connections 2024-04-21T08:50:33.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.504+0000 7f6dc17b2640 1 -- 172.21.15.28:0/2532786048 wait complete. 2024-04-21T08:50:33.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:33 smithi028 ceph-mon[28003]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-21T08:50:33.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:33 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/830526346' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:33.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:33 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1807971446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:50:33.514 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:33 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3550273001' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:50:33.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:50:33.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:33 smithi152 ceph-mon[31401]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-21T08:50:33.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:33 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/830526346' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:33.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:33 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1807971446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:50:33.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:33 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3550273001' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:50:33.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:33 smithi171 ceph-mon[31358]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-21T08:50:33.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:33 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/830526346' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:33.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:33 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1807971446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:50:33.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:33 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3550273001' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:50:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.622+0000 7f3aaa1b6640 1 -- 172.21.15.28:0/480257009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4103b60 msgr2=0x7f3aa410a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.622+0000 7f3aaa1b6640 1 --2- 172.21.15.28:0/480257009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4103b60 0x7f3aa410a3f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c009ca0 tx=0x7f3a9c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.622+0000 7f3aaa1b6640 1 -- 172.21.15.28:0/480257009 shutdown_connections 2024-04-21T08:50:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.622+0000 7f3aaa1b6640 1 --2- 172.21.15.28:0/480257009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4103b60 0x7f3aa410a3f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.622+0000 7f3aaa1b6640 1 --2- 172.21.15.28:0/480257009 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3aa41031a0 0x7f3aa4103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.622+0000 7f3aaa1b6640 1 --2- 172.21.15.28:0/480257009 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3aa4101fb0 0x7f3aa41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.622+0000 7f3aaa1b6640 1 -- 172.21.15.28:0/480257009 >> 172.21.15.28:0/480257009 conn(0x7f3aa40fd520 msgr2=0x7f3aa40ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.622+0000 7f3aaa1b6640 1 -- 172.21.15.28:0/480257009 shutdown_connections 2024-04-21T08:50:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.622+0000 7f3aaa1b6640 1 -- 172.21.15.28:0/480257009 wait complete. 2024-04-21T08:50:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.622+0000 7f3aaa1b6640 1 Processor -- start 2024-04-21T08:50:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aaa1b6640 1 -- start start 2024-04-21T08:50:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aaa1b6640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3aa4101fb0 0x7f3aa4198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aaa1b6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3aa41031a0 0x7f3aa4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aaa1b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4103b60 0x7f3aa419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aaa1b6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa410db50 con 0x7f3aa4103b60 2024-04-21T08:50:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aaa1b6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa410d9d0 con 0x7f3aa41031a0 2024-04-21T08:50:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aaa1b6640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa410dcd0 con 0x7f3aa4101fb0 2024-04-21T08:50:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aa3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4103b60 0x7f3aa419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aa3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4103b60 0x7f3aa419f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36808/0 (socket says 172.21.15.28:36808) 2024-04-21T08:50:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aa3fff640 1 -- 172.21.15.28:0/3195962567 learned_addr learned my addr 172.21.15.28:0/3195962567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aa37fe640 1 --2- 172.21.15.28:0/3195962567 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3aa4101fb0 0x7f3aa4198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aa2ffd640 1 --2- 172.21.15.28:0/3195962567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3aa41031a0 0x7f3aa4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aa3fff640 1 -- 172.21.15.28:0/3195962567 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3aa4101fb0 msgr2=0x7f3aa4198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aa3fff640 1 --2- 172.21.15.28:0/3195962567 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3aa4101fb0 0x7f3aa4198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aa3fff640 1 -- 172.21.15.28:0/3195962567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3aa41031a0 msgr2=0x7f3aa4198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aa3fff640 1 --2- 172.21.15.28:0/3195962567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3aa41031a0 0x7f3aa4198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aa3fff640 1 -- 172.21.15.28:0/3195962567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa419fe40 con 0x7f3aa4103b60 2024-04-21T08:50:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.623+0000 7f3aa3fff640 1 --2- 172.21.15.28:0/3195962567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4103b60 0x7f3aa419f7d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c02c700 tx=0x7f3a9c035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.624+0000 7f3aa0ff9640 1 -- 172.21.15.28:0/3195962567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c039040 con 0x7f3aa4103b60 2024-04-21T08:50:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.624+0000 7f3aaa1b6640 1 -- 172.21.15.28:0/3195962567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa41a00d0 con 0x7f3aa4103b60 2024-04-21T08:50:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.624+0000 7f3aa0ff9640 1 -- 172.21.15.28:0/3195962567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a9c044070 con 0x7f3aa4103b60 2024-04-21T08:50:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.624+0000 7f3aa0ff9640 1 -- 172.21.15.28:0/3195962567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c004440 con 0x7f3aa4103b60 2024-04-21T08:50:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.624+0000 7f3aaa1b6640 1 -- 172.21.15.28:0/3195962567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa41a05c0 con 0x7f3aa4103b60 2024-04-21T08:50:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.625+0000 7f3aaa1b6640 1 -- 172.21.15.28:0/3195962567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aa4109370 con 0x7f3aa4103b60 2024-04-21T08:50:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.627+0000 7f3aa0ff9640 1 -- 172.21.15.28:0/3195962567 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3a9c0045e0 con 0x7f3aa4103b60 2024-04-21T08:50:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.627+0000 7f3aa0ff9640 1 --2- 172.21.15.28:0/3195962567 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3a8c0a3c40 0x7f3a8c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.627+0000 7f3aa37fe640 1 --2- 172.21.15.28:0/3195962567 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3a8c0a3c40 0x7f3a8c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.627+0000 7f3aa0ff9640 1 -- 172.21.15.28:0/3195962567 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f3a9c0f3a10 con 0x7f3aa4103b60 2024-04-21T08:50:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.628+0000 7f3aa37fe640 1 --2- 172.21.15.28:0/3195962567 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3a8c0a3c40 0x7f3a8c0a6100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4103000 tx=0x7f3a94008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.631+0000 7f3aa0ff9640 1 -- 172.21.15.28:0/3195962567 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3a9c0c0060 con 0x7f3aa4103b60 2024-04-21T08:50:33.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.818+0000 7f3aaa1b6640 1 -- 172.21.15.28:0/3195962567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3aa402d5c0 con 0x7f3aa4103b60 2024-04-21T08:50:33.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.818+0000 7f3aa0ff9640 1 -- 172.21.15.28:0/3195962567 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f3a9c0bf960 con 0x7f3aa4103b60 2024-04-21T08:50:33.818 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:50:33.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.820+0000 7f3aaa1b6640 1 -- 172.21.15.28:0/3195962567 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3a8c0a3c40 msgr2=0x7f3a8c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:33.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.820+0000 7f3aaa1b6640 1 --2- 172.21.15.28:0/3195962567 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3a8c0a3c40 0x7f3a8c0a6100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4103000 tx=0x7f3a94008040 comp rx=0 tx=0).stop 2024-04-21T08:50:33.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.820+0000 7f3aaa1b6640 1 -- 172.21.15.28:0/3195962567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4103b60 msgr2=0x7f3aa419f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:33.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.820+0000 7f3aaa1b6640 1 --2- 172.21.15.28:0/3195962567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4103b60 0x7f3aa419f7d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c02c700 tx=0x7f3a9c035d60 comp rx=0 tx=0).stop 2024-04-21T08:50:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.820+0000 7f3aaa1b6640 1 -- 172.21.15.28:0/3195962567 shutdown_connections 2024-04-21T08:50:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.820+0000 7f3aaa1b6640 1 --2- 172.21.15.28:0/3195962567 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3a8c0a3c40 0x7f3a8c0a6100 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.820+0000 7f3aaa1b6640 1 --2- 172.21.15.28:0/3195962567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa4103b60 0x7f3aa419f7d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.820+0000 7f3aaa1b6640 1 --2- 172.21.15.28:0/3195962567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3aa41031a0 0x7f3aa4198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.820+0000 7f3aaa1b6640 1 --2- 172.21.15.28:0/3195962567 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3aa4101fb0 0x7f3aa4198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.820+0000 7f3aaa1b6640 1 -- 172.21.15.28:0/3195962567 >> 172.21.15.28:0/3195962567 conn(0x7f3aa40fd520 msgr2=0x7f3aa4106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.821+0000 7f3aaa1b6640 1 -- 172.21.15.28:0/3195962567 shutdown_connections 2024-04-21T08:50:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.821+0000 7f3aaa1b6640 1 -- 172.21.15.28:0/3195962567 wait complete. 2024-04-21T08:50:33.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:50:33.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.935+0000 7ffb11bdb640 1 -- 172.21.15.28:0/796415864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0c105370 msgr2=0x7ffb0c107760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:33.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.935+0000 7ffb11bdb640 1 --2- 172.21.15.28:0/796415864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0c105370 0x7ffb0c107760 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ffb00009900 tx=0x7ffb0002bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:33.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.935+0000 7ffb11bdb640 1 -- 172.21.15.28:0/796415864 shutdown_connections 2024-04-21T08:50:33.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.935+0000 7ffb11bdb640 1 --2- 172.21.15.28:0/796415864 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffb0c107ca0 0x7ffb0c10a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.935+0000 7ffb11bdb640 1 --2- 172.21.15.28:0/796415864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0c105370 0x7ffb0c107760 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.935+0000 7ffb11bdb640 1 --2- 172.21.15.28:0/796415864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb0c102a40 0x7ffb0c104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.935+0000 7ffb11bdb640 1 -- 172.21.15.28:0/796415864 >> 172.21.15.28:0/796415864 conn(0x7ffb0c0fc470 msgr2=0x7ffb0c0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.935+0000 7ffb11bdb640 1 -- 172.21.15.28:0/796415864 shutdown_connections 2024-04-21T08:50:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.935+0000 7ffb11bdb640 1 -- 172.21.15.28:0/796415864 wait complete. 2024-04-21T08:50:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb11bdb640 1 Processor -- start 2024-04-21T08:50:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb11bdb640 1 -- start start 2024-04-21T08:50:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb11bdb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0c102a40 0x7ffb0c1023b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb11bdb640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffb0c105370 0x7ffb0c1009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb11bdb640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb0c107ca0 0x7ffb0c100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb11bdb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c10db00 con 0x7ffb0c102a40 2024-04-21T08:50:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb11bdb640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c10d980 con 0x7ffb0c107ca0 2024-04-21T08:50:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb11bdb640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c10dc80 con 0x7ffb0c105370 2024-04-21T08:50:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb0b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0c102a40 0x7ffb0c1023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb0b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0c102a40 0x7ffb0c1023b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59088/0 (socket says 172.21.15.28:59088) 2024-04-21T08:50:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb0b7fe640 1 -- 172.21.15.28:0/2164116176 learned_addr learned my addr 172.21.15.28:0/2164116176 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:33.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb0b7fe640 1 -- 172.21.15.28:0/2164116176 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffb0c105370 msgr2=0x7ffb0c1009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:33.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb0affd640 1 --2- 172.21.15.28:0/2164116176 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffb0c105370 0x7ffb0c1009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:33.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb0b7fe640 1 --2- 172.21.15.28:0/2164116176 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffb0c105370 0x7ffb0c1009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb0b7fe640 1 -- 172.21.15.28:0/2164116176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb0c107ca0 msgr2=0x7ffb0c100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:33.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb0b7fe640 1 --2- 172.21.15.28:0/2164116176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb0c107ca0 0x7ffb0c100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:33.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb0b7fe640 1 -- 172.21.15.28:0/2164116176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb0c1017c0 con 0x7ffb0c102a40 2024-04-21T08:50:33.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.936+0000 7ffb0affd640 1 --2- 172.21.15.28:0/2164116176 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffb0c105370 0x7ffb0c1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:33.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.937+0000 7ffb0b7fe640 1 --2- 172.21.15.28:0/2164116176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0c102a40 0x7ffb0c1023b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00d950 tx=0x7ffafc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:33.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.937+0000 7ffb08ff9640 1 -- 172.21.15.28:0/2164116176 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc017030 con 0x7ffb0c102a40 2024-04-21T08:50:33.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.937+0000 7ffb08ff9640 1 -- 172.21.15.28:0/2164116176 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffafc004020 con 0x7ffb0c102a40 2024-04-21T08:50:33.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.937+0000 7ffb11bdb640 1 -- 172.21.15.28:0/2164116176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb0c1a22d0 con 0x7ffb0c102a40 2024-04-21T08:50:33.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.937+0000 7ffb08ff9640 1 -- 172.21.15.28:0/2164116176 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc0047a0 con 0x7ffb0c102a40 2024-04-21T08:50:33.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.937+0000 7ffb11bdb640 1 -- 172.21.15.28:0/2164116176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb0c1a2820 con 0x7ffb0c102a40 2024-04-21T08:50:33.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.938+0000 7ffb11bdb640 1 -- 172.21.15.28:0/2164116176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb0c0fdf60 con 0x7ffb0c102a40 2024-04-21T08:50:33.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.939+0000 7ffb08ff9640 1 -- 172.21.15.28:0/2164116176 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ffafc009a30 con 0x7ffb0c102a40 2024-04-21T08:50:33.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.940+0000 7ffb08ff9640 1 --2- 172.21.15.28:0/2164116176 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffaf40a3c40 0x7ffaf40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:33.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.940+0000 7ffb0affd640 1 --2- 172.21.15.28:0/2164116176 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffaf40a3c40 0x7ffaf40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:33.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.940+0000 7ffb08ff9640 1 -- 172.21.15.28:0/2164116176 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7ffafc0d0490 con 0x7ffb0c102a40 2024-04-21T08:50:33.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.940+0000 7ffb0affd640 1 --2- 172.21.15.28:0/2164116176 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffaf40a3c40 0x7ffaf40a6100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffb00036040 tx=0x7ffb00000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:33.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:33.945+0000 7ffb08ff9640 1 -- 172.21.15.28:0/2164116176 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffafc09c9d0 con 0x7ffb0c102a40 2024-04-21T08:50:34.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.176+0000 7ffb11bdb640 1 -- 172.21.15.28:0/2164116176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb0c02d5c0 con 0x7ffb0c102a40 2024-04-21T08:50:34.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.176+0000 7ffb08ff9640 1 -- 172.21.15.28:0/2164116176 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7ffb0c02d5c0 con 0x7ffb0c102a40 2024-04-21T08:50:34.176 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:50:34.176 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:49:12.840022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24463,24487,24488]},"id":1}]} 2024-04-21T08:50:34.177 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-21T08:50:34.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.178+0000 7ffb11bdb640 1 -- 172.21.15.28:0/2164116176 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffaf40a3c40 msgr2=0x7ffaf40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:34.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.178+0000 7ffb11bdb640 1 --2- 172.21.15.28:0/2164116176 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffaf40a3c40 0x7ffaf40a6100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffb00036040 tx=0x7ffb00000bc0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.178+0000 7ffb11bdb640 1 -- 172.21.15.28:0/2164116176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0c102a40 msgr2=0x7ffb0c1023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:34.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.178+0000 7ffb11bdb640 1 --2- 172.21.15.28:0/2164116176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0c102a40 0x7ffb0c1023b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00d950 tx=0x7ffafc00de10 comp rx=0 tx=0).stop 2024-04-21T08:50:34.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.178+0000 7ffb11bdb640 1 -- 172.21.15.28:0/2164116176 shutdown_connections 2024-04-21T08:50:34.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.178+0000 7ffb11bdb640 1 --2- 172.21.15.28:0/2164116176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb0c107ca0 0x7ffb0c100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.178+0000 7ffb11bdb640 1 --2- 172.21.15.28:0/2164116176 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffb0c105370 0x7ffb0c1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.178+0000 7ffb11bdb640 1 --2- 172.21.15.28:0/2164116176 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ffaf40a3c40 0x7ffaf40a6100 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.178+0000 7ffb11bdb640 1 --2- 172.21.15.28:0/2164116176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0c102a40 0x7ffb0c1023b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.178+0000 7ffb11bdb640 1 -- 172.21.15.28:0/2164116176 >> 172.21.15.28:0/2164116176 conn(0x7ffb0c0fc470 msgr2=0x7ffb0c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:34.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.178+0000 7ffb11bdb640 1 -- 172.21.15.28:0/2164116176 shutdown_connections 2024-04-21T08:50:34.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.178+0000 7ffb11bdb640 1 -- 172.21.15.28:0/2164116176 wait complete. 2024-04-21T08:50:34.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:50:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.295+0000 7f1cf48f2640 1 -- 172.21.15.28:0/3171024930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0102a40 msgr2=0x7f1cf0104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.295+0000 7f1cf48f2640 1 --2- 172.21.15.28:0/3171024930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0102a40 0x7f1cf0104e30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0009900 tx=0x7f1ce002bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.295+0000 7f1cf48f2640 1 -- 172.21.15.28:0/3171024930 shutdown_connections 2024-04-21T08:50:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.295+0000 7f1cf48f2640 1 --2- 172.21.15.28:0/3171024930 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1cf0107ca0 0x7f1cf010a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.295+0000 7f1cf48f2640 1 --2- 172.21.15.28:0/3171024930 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1cf0105370 0x7f1cf0107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.295+0000 7f1cf48f2640 1 --2- 172.21.15.28:0/3171024930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0102a40 0x7f1cf0104e30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.295+0000 7f1cf48f2640 1 -- 172.21.15.28:0/3171024930 >> 172.21.15.28:0/3171024930 conn(0x7f1cf00fc470 msgr2=0x7f1cf00fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.295+0000 7f1cf48f2640 1 -- 172.21.15.28:0/3171024930 shutdown_connections 2024-04-21T08:50:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.295+0000 7f1cf48f2640 1 -- 172.21.15.28:0/3171024930 wait complete. 2024-04-21T08:50:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.296+0000 7f1cf48f2640 1 Processor -- start 2024-04-21T08:50:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.296+0000 7f1cf48f2640 1 -- start start 2024-04-21T08:50:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.296+0000 7f1cf48f2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1cf0102a40 0x7f1cf0198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.296+0000 7f1cf48f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0105370 0x7f1cf0198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.296+0000 7f1cf48f2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1cf0107ca0 0x7f1cf019f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.296+0000 7f1cf48f2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf010db00 con 0x7f1cf0105370 2024-04-21T08:50:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.296+0000 7f1cf48f2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf010d980 con 0x7f1cf0102a40 2024-04-21T08:50:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.296+0000 7f1cf48f2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf010dc80 con 0x7f1cf0107ca0 2024-04-21T08:50:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.296+0000 7f1ce7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0105370 0x7f1cf0198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.296+0000 7f1ce7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0105370 0x7f1cf0198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59104/0 (socket says 172.21.15.28:59104) 2024-04-21T08:50:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.296+0000 7f1ce7fff640 1 -- 172.21.15.28:0/1267333899 learned_addr learned my addr 172.21.15.28:0/1267333899 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.297+0000 7f1cee59c640 1 --2- 172.21.15.28:0/1267333899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1cf0102a40 0x7f1cf0198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.297+0000 7f1ceed9d640 1 --2- 172.21.15.28:0/1267333899 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1cf0107ca0 0x7f1cf019f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.297+0000 7f1ce7fff640 1 -- 172.21.15.28:0/1267333899 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1cf0107ca0 msgr2=0x7f1cf019f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.297+0000 7f1ce7fff640 1 --2- 172.21.15.28:0/1267333899 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1cf0107ca0 0x7f1cf019f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.297+0000 7f1ce7fff640 1 -- 172.21.15.28:0/1267333899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1cf0102a40 msgr2=0x7f1cf0198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.297+0000 7f1ce7fff640 1 --2- 172.21.15.28:0/1267333899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1cf0102a40 0x7f1cf0198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.297+0000 7f1ce7fff640 1 -- 172.21.15.28:0/1267333899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf019fd70 con 0x7f1cf0105370 2024-04-21T08:50:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.297+0000 7f1ce7fff640 1 --2- 172.21.15.28:0/1267333899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0105370 0x7f1cf0198690 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1cd800e9f0 tx=0x7f1cd800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.297+0000 7f1ce77fe640 1 -- 172.21.15.28:0/1267333899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd800ccc0 con 0x7f1cf0105370 2024-04-21T08:50:34.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.297+0000 7f1ce77fe640 1 -- 172.21.15.28:0/1267333899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cd80054d0 con 0x7f1cf0105370 2024-04-21T08:50:34.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.297+0000 7f1cf48f2640 1 -- 172.21.15.28:0/1267333899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf01a0060 con 0x7f1cf0105370 2024-04-21T08:50:34.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.297+0000 7f1ce77fe640 1 -- 172.21.15.28:0/1267333899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8010430 con 0x7f1cf0105370 2024-04-21T08:50:34.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.297+0000 7f1cf48f2640 1 -- 172.21.15.28:0/1267333899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf01a0560 con 0x7f1cf0105370 2024-04-21T08:50:34.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.298+0000 7f1cf48f2640 1 -- 172.21.15.28:0/1267333899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cf00fdf60 con 0x7f1cf0105370 2024-04-21T08:50:34.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.299+0000 7f1ce77fe640 1 -- 172.21.15.28:0/1267333899 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1cd80046c0 con 0x7f1cf0105370 2024-04-21T08:50:34.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.300+0000 7f1ce77fe640 1 --2- 172.21.15.28:0/1267333899 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1cdc0a3c40 0x7f1cdc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.300+0000 7f1cee59c640 1 --2- 172.21.15.28:0/1267333899 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1cdc0a3c40 0x7f1cdc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.300+0000 7f1ce77fe640 1 -- 172.21.15.28:0/1267333899 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f1cd80d0c70 con 0x7f1cf0105370 2024-04-21T08:50:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.301+0000 7f1cee59c640 1 --2- 172.21.15.28:0/1267333899 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1cdc0a3c40 0x7f1cdc0a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0009230 tx=0x7f1ce0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.305+0000 7f1ce77fe640 1 -- 172.21.15.28:0/1267333899 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1cd809d150 con 0x7f1cf0105370 2024-04-21T08:50:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.494+0000 7f1cf48f2640 1 -- 172.21.15.28:0/1267333899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1cf002d810 con 0x7f1cf0105370 2024-04-21T08:50:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.495+0000 7f1ce77fe640 1 -- 172.21.15.28:0/1267333899 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24598 (secure 0 0 0) 0x7f1cd809ca50 con 0x7f1cf0105370 2024-04-21T08:50:34.495 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:50:34.495 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:49:36.894878+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:50:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.497+0000 7f1cf48f2640 1 -- 172.21.15.28:0/1267333899 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1cdc0a3c40 msgr2=0x7f1cdc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.497+0000 7f1cf48f2640 1 --2- 172.21.15.28:0/1267333899 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1cdc0a3c40 0x7f1cdc0a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0009230 tx=0x7f1ce0005fd0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.497+0000 7f1cf48f2640 1 -- 172.21.15.28:0/1267333899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0105370 msgr2=0x7f1cf0198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.497+0000 7f1cf48f2640 1 --2- 172.21.15.28:0/1267333899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0105370 0x7f1cf0198690 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1cd800e9f0 tx=0x7f1cd800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.497+0000 7f1cf48f2640 1 -- 172.21.15.28:0/1267333899 shutdown_connections 2024-04-21T08:50:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.497+0000 7f1cf48f2640 1 --2- 172.21.15.28:0/1267333899 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1cf0107ca0 0x7f1cf019f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.497+0000 7f1cf48f2640 1 --2- 172.21.15.28:0/1267333899 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f1cdc0a3c40 0x7f1cdc0a6100 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.497+0000 7f1cf48f2640 1 --2- 172.21.15.28:0/1267333899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cf0105370 0x7f1cf0198690 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.497+0000 7f1cf48f2640 1 --2- 172.21.15.28:0/1267333899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1cf0102a40 0x7f1cf0198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.497+0000 7f1cf48f2640 1 -- 172.21.15.28:0/1267333899 >> 172.21.15.28:0/1267333899 conn(0x7f1cf00fc470 msgr2=0x7f1cf0103380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:34.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.498+0000 7f1cf48f2640 1 -- 172.21.15.28:0/1267333899 shutdown_connections 2024-04-21T08:50:34.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.498+0000 7f1cf48f2640 1 -- 172.21.15.28:0/1267333899 wait complete. 2024-04-21T08:50:34.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:34 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2532786048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:50:34.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:34 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3195962567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:50:34.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:34 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2164116176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:34.509 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:48:43.543799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:48:44.864564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:48:48.747842+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:50:34.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:50:34.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:34 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2532786048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:50:34.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:34 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3195962567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:50:34.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:34 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2164116176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.615+0000 7f212e043640 1 -- 172.21.15.28:0/962556764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2128107cd0 msgr2=0x7f212810a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.615+0000 7f212e043640 1 --2- 172.21.15.28:0/962556764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2128107cd0 0x7f212810a0c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2120009ca0 tx=0x7f212002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.615+0000 7f212e043640 1 -- 172.21.15.28:0/962556764 shutdown_connections 2024-04-21T08:50:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.615+0000 7f212e043640 1 --2- 172.21.15.28:0/962556764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2128107cd0 0x7f212810a0c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.615+0000 7f212e043640 1 --2- 172.21.15.28:0/962556764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21281053a0 0x7f2128107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.615+0000 7f212e043640 1 --2- 172.21.15.28:0/962556764 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2128102a70 0x7f2128104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.615+0000 7f212e043640 1 -- 172.21.15.28:0/962556764 >> 172.21.15.28:0/962556764 conn(0x7f21280fc400 msgr2=0x7f21280fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.615+0000 7f212e043640 1 -- 172.21.15.28:0/962556764 shutdown_connections 2024-04-21T08:50:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.615+0000 7f212e043640 1 -- 172.21.15.28:0/962556764 wait complete. 2024-04-21T08:50:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.616+0000 7f212e043640 1 Processor -- start 2024-04-21T08:50:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.616+0000 7f212e043640 1 -- start start 2024-04-21T08:50:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.616+0000 7f212e043640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2128102a70 0x7f21281981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.616+0000 7f212e043640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21281053a0 0x7f2128198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.616+0000 7f212e043640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2128107cd0 0x7f212819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.616+0000 7f212e043640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f212810dac0 con 0x7f2128102a70 2024-04-21T08:50:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.616+0000 7f212e043640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f212810d940 con 0x7f21281053a0 2024-04-21T08:50:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.616+0000 7f212e043640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f212810dc40 con 0x7f2128107cd0 2024-04-21T08:50:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.616+0000 7f21277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2128102a70 0x7f21281981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.616+0000 7f21277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2128102a70 0x7f21281981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59120/0 (socket says 172.21.15.28:59120) 2024-04-21T08:50:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.616+0000 7f21277fe640 1 -- 172.21.15.28:0/2753112515 learned_addr learned my addr 172.21.15.28:0/2753112515 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.616+0000 7f2126ffd640 1 --2- 172.21.15.28:0/2753112515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21281053a0 0x7f2128198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.616+0000 7f2127fff640 1 --2- 172.21.15.28:0/2753112515 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2128107cd0 0x7f212819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.616+0000 7f21277fe640 1 -- 172.21.15.28:0/2753112515 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2128107cd0 msgr2=0x7f212819f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.617+0000 7f21277fe640 1 --2- 172.21.15.28:0/2753112515 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2128107cd0 0x7f212819f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.617+0000 7f21277fe640 1 -- 172.21.15.28:0/2753112515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21281053a0 msgr2=0x7f2128198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:34.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.617+0000 7f21277fe640 1 --2- 172.21.15.28:0/2753112515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21281053a0 0x7f2128198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.617+0000 7f21277fe640 1 -- 172.21.15.28:0/2753112515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f212819fda0 con 0x7f2128102a70 2024-04-21T08:50:34.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.617+0000 7f2127fff640 1 --2- 172.21.15.28:0/2753112515 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2128107cd0 0x7f212819f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:34.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.617+0000 7f21277fe640 1 --2- 172.21.15.28:0/2753112515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2128102a70 0x7f21281981d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f211800c920 tx=0x7f211800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:34.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.617+0000 7f2124ff9640 1 -- 172.21.15.28:0/2753112515 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2118007c70 con 0x7f2128102a70 2024-04-21T08:50:34.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.617+0000 7f2124ff9640 1 -- 172.21.15.28:0/2753112515 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2118007e10 con 0x7f2128102a70 2024-04-21T08:50:34.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.617+0000 7f212e043640 1 -- 172.21.15.28:0/2753112515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21281a0090 con 0x7f2128102a70 2024-04-21T08:50:34.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.617+0000 7f2124ff9640 1 -- 172.21.15.28:0/2753112515 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2118018820 con 0x7f2128102a70 2024-04-21T08:50:34.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.617+0000 7f212e043640 1 -- 172.21.15.28:0/2753112515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21281a05e0 con 0x7f2128102a70 2024-04-21T08:50:34.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.618+0000 7f212e043640 1 -- 172.21.15.28:0/2753112515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21280fdef0 con 0x7f2128102a70 2024-04-21T08:50:34.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:34 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2532786048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:50:34.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:34 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3195962567' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:50:34.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:34 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2164116176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:34.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.619+0000 7f2124ff9640 1 -- 172.21.15.28:0/2753112515 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f211800f450 con 0x7f2128102a70 2024-04-21T08:50:34.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.620+0000 7f2124ff9640 1 --2- 172.21.15.28:0/2753112515 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f21100a3c40 0x7f21100a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:34.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.620+0000 7f2126ffd640 1 --2- 172.21.15.28:0/2753112515 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f21100a3c40 0x7f21100a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:34.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.620+0000 7f2124ff9640 1 -- 172.21.15.28:0/2753112515 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f21180d0d90 con 0x7f2128102a70 2024-04-21T08:50:34.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.620+0000 7f2126ffd640 1 --2- 172.21.15.28:0/2753112515 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f21100a3c40 0x7f21100a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2128199690 tx=0x7f211c00a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:34.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.623+0000 7f2124ff9640 1 -- 172.21.15.28:0/2753112515 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2118014030 con 0x7f2128102a70 2024-04-21T08:50:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.807+0000 7f212e043640 1 -- 172.21.15.28:0/2753112515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2128062b80 con 0x7f2128102a70 2024-04-21T08:50:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.808+0000 7f2124ff9640 1 -- 172.21.15.28:0/2753112515 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f211809d300 con 0x7f2128102a70 2024-04-21T08:50:34.807 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T08:50:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.810+0000 7f212e043640 1 -- 172.21.15.28:0/2753112515 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f21100a3c40 msgr2=0x7f21100a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.810+0000 7f212e043640 1 --2- 172.21.15.28:0/2753112515 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f21100a3c40 0x7f21100a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2128199690 tx=0x7f211c00a4c0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.810+0000 7f212e043640 1 -- 172.21.15.28:0/2753112515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2128102a70 msgr2=0x7f21281981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.810+0000 7f212e043640 1 --2- 172.21.15.28:0/2753112515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2128102a70 0x7f21281981d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f211800c920 tx=0x7f211800cde0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.810+0000 7f212e043640 1 -- 172.21.15.28:0/2753112515 shutdown_connections 2024-04-21T08:50:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.810+0000 7f212e043640 1 --2- 172.21.15.28:0/2753112515 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2128107cd0 0x7f212819f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.810+0000 7f212e043640 1 --2- 172.21.15.28:0/2753112515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21281053a0 0x7f2128198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.810+0000 7f212e043640 1 --2- 172.21.15.28:0/2753112515 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f21100a3c40 0x7f21100a6100 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.810+0000 7f212e043640 1 --2- 172.21.15.28:0/2753112515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2128102a70 0x7f21281981d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.810+0000 7f212e043640 1 -- 172.21.15.28:0/2753112515 >> 172.21.15.28:0/2753112515 conn(0x7f21280fc400 msgr2=0x7f2128103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:34.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.810+0000 7f212e043640 1 -- 172.21.15.28:0/2753112515 shutdown_connections 2024-04-21T08:50:34.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.810+0000 7f212e043640 1 -- 172.21.15.28:0/2753112515 wait complete. 2024-04-21T08:50:34.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:50:34.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.926+0000 7f459732b640 1 -- 172.21.15.28:0/1380123762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4590107cf0 msgr2=0x7f459010a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:34.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.926+0000 7f459732b640 1 --2- 172.21.15.28:0/1380123762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4590107cf0 0x7f459010a0e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4588009900 tx=0x7f458802bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:34.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f459732b640 1 -- 172.21.15.28:0/1380123762 shutdown_connections 2024-04-21T08:50:34.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f459732b640 1 --2- 172.21.15.28:0/1380123762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4590107cf0 0x7f459010a0e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f459732b640 1 --2- 172.21.15.28:0/1380123762 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45901053c0 0x7f45901077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f459732b640 1 --2- 172.21.15.28:0/1380123762 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4590102a90 0x7f4590104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f459732b640 1 -- 172.21.15.28:0/1380123762 >> 172.21.15.28:0/1380123762 conn(0x7f45900fc400 msgr2=0x7f45900fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f459732b640 1 -- 172.21.15.28:0/1380123762 shutdown_connections 2024-04-21T08:50:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f459732b640 1 -- 172.21.15.28:0/1380123762 wait complete. 2024-04-21T08:50:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f459732b640 1 Processor -- start 2024-04-21T08:50:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f459732b640 1 -- start start 2024-04-21T08:50:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f459732b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4590102a90 0x7f4590198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f459732b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45901053c0 0x7f4590198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f459732b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4590107cf0 0x7f459019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f459732b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f459010db20 con 0x7f4590102a90 2024-04-21T08:50:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f459732b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f459010d9a0 con 0x7f45901053c0 2024-04-21T08:50:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f459732b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f459010dca0 con 0x7f4590107cf0 2024-04-21T08:50:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f45950c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4590102a90 0x7f4590198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f45950c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4590102a90 0x7f4590198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59142/0 (socket says 172.21.15.28:59142) 2024-04-21T08:50:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.927+0000 7f45950c7640 1 -- 172.21.15.28:0/1118734884 learned_addr learned my addr 172.21.15.28:0/1118734884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.928+0000 7f45958c8640 1 --2- 172.21.15.28:0/1118734884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4590107cf0 0x7f459019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.928+0000 7f45948c6640 1 --2- 172.21.15.28:0/1118734884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45901053c0 0x7f4590198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.928+0000 7f45950c7640 1 -- 172.21.15.28:0/1118734884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4590107cf0 msgr2=0x7f459019f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.928+0000 7f45950c7640 1 --2- 172.21.15.28:0/1118734884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4590107cf0 0x7f459019f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.928+0000 7f45950c7640 1 -- 172.21.15.28:0/1118734884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45901053c0 msgr2=0x7f4590198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.928+0000 7f45950c7640 1 --2- 172.21.15.28:0/1118734884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45901053c0 0x7f4590198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.928+0000 7f45950c7640 1 -- 172.21.15.28:0/1118734884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f459019fe40 con 0x7f4590102a90 2024-04-21T08:50:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.928+0000 7f45950c7640 1 --2- 172.21.15.28:0/1118734884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4590102a90 0x7f4590198220 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f458c00eab0 tx=0x7f458c00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.928+0000 7f45867fc640 1 -- 172.21.15.28:0/1118734884 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c00ce40 con 0x7f4590102a90 2024-04-21T08:50:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.928+0000 7f45867fc640 1 -- 172.21.15.28:0/1118734884 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f458c011920 con 0x7f4590102a90 2024-04-21T08:50:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.928+0000 7f45867fc640 1 -- 172.21.15.28:0/1118734884 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c010a30 con 0x7f4590102a90 2024-04-21T08:50:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.928+0000 7f459732b640 1 -- 172.21.15.28:0/1118734884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45901a0130 con 0x7f4590102a90 2024-04-21T08:50:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.928+0000 7f459732b640 1 -- 172.21.15.28:0/1118734884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45901a06d0 con 0x7f4590102a90 2024-04-21T08:50:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.929+0000 7f459732b640 1 -- 172.21.15.28:0/1118734884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45900fdef0 con 0x7f4590102a90 2024-04-21T08:50:34.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.930+0000 7f45867fc640 1 -- 172.21.15.28:0/1118734884 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f458c010bd0 con 0x7f4590102a90 2024-04-21T08:50:34.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.931+0000 7f45867fc640 1 --2- 172.21.15.28:0/1118734884 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f457c0a3cc0 0x7f457c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:34.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.931+0000 7f45948c6640 1 --2- 172.21.15.28:0/1118734884 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f457c0a3cc0 0x7f457c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:34.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.931+0000 7f45867fc640 1 -- 172.21.15.28:0/1118734884 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f458c0d0f90 con 0x7f4590102a90 2024-04-21T08:50:34.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.931+0000 7f45948c6640 1 --2- 172.21.15.28:0/1118734884 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f457c0a3cc0 0x7f457c0a6180 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4580005fd0 tx=0x7f4580005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:34.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:34.935+0000 7f45867fc640 1 -- 172.21.15.28:0/1118734884 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f458c016030 con 0x7f4590102a90 2024-04-21T08:50:35.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.121+0000 7f459732b640 1 -- 172.21.15.28:0/1118734884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4590062b50 con 0x7f4590102a90 2024-04-21T08:50:35.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.122+0000 7f45867fc640 1 -- 172.21.15.28:0/1118734884 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f458c09d4f0 con 0x7f4590102a90 2024-04-21T08:50:35.121 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:50:35.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.124+0000 7f459732b640 1 -- 172.21.15.28:0/1118734884 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f457c0a3cc0 msgr2=0x7f457c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:35.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.124+0000 7f459732b640 1 --2- 172.21.15.28:0/1118734884 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f457c0a3cc0 0x7f457c0a6180 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4580005fd0 tx=0x7f4580005850 comp rx=0 tx=0).stop 2024-04-21T08:50:35.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.124+0000 7f459732b640 1 -- 172.21.15.28:0/1118734884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4590102a90 msgr2=0x7f4590198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:35.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.124+0000 7f459732b640 1 --2- 172.21.15.28:0/1118734884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4590102a90 0x7f4590198220 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f458c00eab0 tx=0x7f458c00ef70 comp rx=0 tx=0).stop 2024-04-21T08:50:35.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.124+0000 7f459732b640 1 -- 172.21.15.28:0/1118734884 shutdown_connections 2024-04-21T08:50:35.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.124+0000 7f459732b640 1 --2- 172.21.15.28:0/1118734884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4590107cf0 0x7f459019f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.124+0000 7f459732b640 1 --2- 172.21.15.28:0/1118734884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45901053c0 0x7f4590198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.124+0000 7f459732b640 1 --2- 172.21.15.28:0/1118734884 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f457c0a3cc0 0x7f457c0a6180 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.124+0000 7f459732b640 1 --2- 172.21.15.28:0/1118734884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4590102a90 0x7f4590198220 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.124+0000 7f459732b640 1 -- 172.21.15.28:0/1118734884 >> 172.21.15.28:0/1118734884 conn(0x7f45900fc400 msgr2=0x7f45901034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:35.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.124+0000 7f459732b640 1 -- 172.21.15.28:0/1118734884 shutdown_connections 2024-04-21T08:50:35.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.124+0000 7f459732b640 1 -- 172.21.15.28:0/1118734884 wait complete. 2024-04-21T08:50:35.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:50:35.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.240+0000 7fc972f84640 1 -- 172.21.15.28:0/4224826757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc96c102a00 msgr2=0x7fc96c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:35.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.240+0000 7fc972f84640 1 --2- 172.21.15.28:0/4224826757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc96c102a00 0x7fc96c109320 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc964009ca0 tx=0x7fc96402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.240+0000 7fc972f84640 1 -- 172.21.15.28:0/4224826757 shutdown_connections 2024-04-21T08:50:35.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.240+0000 7fc972f84640 1 --2- 172.21.15.28:0/4224826757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc96c102a00 0x7fc96c109320 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.240+0000 7fc972f84640 1 --2- 172.21.15.28:0/4224826757 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc96c102040 0x7fc96c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.240+0000 7fc972f84640 1 --2- 172.21.15.28:0/4224826757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc96c100e60 0x7fc96c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.240+0000 7fc972f84640 1 -- 172.21.15.28:0/4224826757 >> 172.21.15.28:0/4224826757 conn(0x7fc96c0fc440 msgr2=0x7fc96c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:35.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.240+0000 7fc972f84640 1 -- 172.21.15.28:0/4224826757 shutdown_connections 2024-04-21T08:50:35.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.240+0000 7fc972f84640 1 -- 172.21.15.28:0/4224826757 wait complete. 2024-04-21T08:50:35.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.241+0000 7fc972f84640 1 Processor -- start 2024-04-21T08:50:35.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.241+0000 7fc972f84640 1 -- start start 2024-04-21T08:50:35.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.241+0000 7fc972f84640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc96c100e60 0x7fc96c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:35.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.241+0000 7fc972f84640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc96c102040 0x7fc96c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:35.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.241+0000 7fc972f84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc96c102a00 0x7fc96c19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:35.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.241+0000 7fc972f84640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc96c10db30 con 0x7fc96c102a00 2024-04-21T08:50:35.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.241+0000 7fc972f84640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc96c10d9b0 con 0x7fc96c100e60 2024-04-21T08:50:35.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.241+0000 7fc972f84640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc96c10dcb0 con 0x7fc96c102040 2024-04-21T08:50:35.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.241+0000 7fc971521640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc96c102a00 0x7fc96c19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:35.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.241+0000 7fc971521640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc96c102a00 0x7fc96c19f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59168/0 (socket says 172.21.15.28:59168) 2024-04-21T08:50:35.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.241+0000 7fc971521640 1 -- 172.21.15.28:0/1270697181 learned_addr learned my addr 172.21.15.28:0/1270697181 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:35.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.241+0000 7fc970d20640 1 --2- 172.21.15.28:0/1270697181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc96c100e60 0x7fc96c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:35.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.242+0000 7fc96bfff640 1 --2- 172.21.15.28:0/1270697181 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc96c102040 0x7fc96c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:35.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.242+0000 7fc971521640 1 -- 172.21.15.28:0/1270697181 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc96c102040 msgr2=0x7fc96c1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:35.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.242+0000 7fc971521640 1 --2- 172.21.15.28:0/1270697181 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc96c102040 0x7fc96c1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.242+0000 7fc971521640 1 -- 172.21.15.28:0/1270697181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc96c100e60 msgr2=0x7fc96c1981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:35.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.242+0000 7fc971521640 1 --2- 172.21.15.28:0/1270697181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc96c100e60 0x7fc96c1981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.242+0000 7fc971521640 1 -- 172.21.15.28:0/1270697181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc96c19fdd0 con 0x7fc96c102a00 2024-04-21T08:50:35.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.242+0000 7fc970d20640 1 --2- 172.21.15.28:0/1270697181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc96c100e60 0x7fc96c1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:35.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.242+0000 7fc971521640 1 --2- 172.21.15.28:0/1270697181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc96c102a00 0x7fc96c19f760 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc96402c700 tx=0x7fc9640093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:35.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.242+0000 7fc969ffb640 1 -- 172.21.15.28:0/1270697181 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc964041020 con 0x7fc96c102a00 2024-04-21T08:50:35.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.242+0000 7fc969ffb640 1 -- 172.21.15.28:0/1270697181 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc964034d60 con 0x7fc96c102a00 2024-04-21T08:50:35.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.242+0000 7fc972f84640 1 -- 172.21.15.28:0/1270697181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc96c1a0060 con 0x7fc96c102a00 2024-04-21T08:50:35.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.242+0000 7fc969ffb640 1 -- 172.21.15.28:0/1270697181 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc964039040 con 0x7fc96c102a00 2024-04-21T08:50:35.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.242+0000 7fc972f84640 1 -- 172.21.15.28:0/1270697181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc96c1a0550 con 0x7fc96c102a00 2024-04-21T08:50:35.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.243+0000 7fc972f84640 1 -- 172.21.15.28:0/1270697181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc96c108200 con 0x7fc96c102a00 2024-04-21T08:50:35.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.244+0000 7fc969ffb640 1 -- 172.21.15.28:0/1270697181 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc964044020 con 0x7fc96c102a00 2024-04-21T08:50:35.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.245+0000 7fc969ffb640 1 --2- 172.21.15.28:0/1270697181 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc9540a3c40 0x7fc9540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:35.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.245+0000 7fc970d20640 1 --2- 172.21.15.28:0/1270697181 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc9540a3c40 0x7fc9540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:35.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.245+0000 7fc969ffb640 1 -- 172.21.15.28:0/1270697181 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fc9640f2c60 con 0x7fc96c102a00 2024-04-21T08:50:35.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.245+0000 7fc970d20640 1 --2- 172.21.15.28:0/1270697181 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc9540a3c40 0x7fc9540a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc96c101bb0 tx=0x7fc95c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:35.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.248+0000 7fc969ffb640 1 -- 172.21.15.28:0/1270697181 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc96404b020 con 0x7fc96c102a00 2024-04-21T08:50:35.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.435+0000 7fc972f84640 1 -- 172.21.15.28:0/1270697181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc96c02d750 con 0x7fc96c102a00 2024-04-21T08:50:35.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.436+0000 7fc969ffb640 1 -- 172.21.15.28:0/1270697181 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fc9640bf250 con 0x7fc96c102a00 2024-04-21T08:50:35.436 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-21T08:50:35.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.438+0000 7fc972f84640 1 -- 172.21.15.28:0/1270697181 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc9540a3c40 msgr2=0x7fc9540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:35.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.438+0000 7fc972f84640 1 --2- 172.21.15.28:0/1270697181 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc9540a3c40 0x7fc9540a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc96c101bb0 tx=0x7fc95c005f90 comp rx=0 tx=0).stop 2024-04-21T08:50:35.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.438+0000 7fc972f84640 1 -- 172.21.15.28:0/1270697181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc96c102a00 msgr2=0x7fc96c19f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:35.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.438+0000 7fc972f84640 1 --2- 172.21.15.28:0/1270697181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc96c102a00 0x7fc96c19f760 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc96402c700 tx=0x7fc9640093f0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.438+0000 7fc972f84640 1 -- 172.21.15.28:0/1270697181 shutdown_connections 2024-04-21T08:50:35.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.438+0000 7fc972f84640 1 --2- 172.21.15.28:0/1270697181 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fc9540a3c40 0x7fc9540a6100 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.438+0000 7fc972f84640 1 --2- 172.21.15.28:0/1270697181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc96c102a00 0x7fc96c19f760 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.438+0000 7fc972f84640 1 --2- 172.21.15.28:0/1270697181 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc96c102040 0x7fc96c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.438+0000 7fc972f84640 1 --2- 172.21.15.28:0/1270697181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc96c100e60 0x7fc96c1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.438+0000 7fc972f84640 1 -- 172.21.15.28:0/1270697181 >> 172.21.15.28:0/1270697181 conn(0x7fc96c0fc440 msgr2=0x7fc96c1053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:35.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.438+0000 7fc972f84640 1 -- 172.21.15.28:0/1270697181 shutdown_connections 2024-04-21T08:50:35.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.438+0000 7fc972f84640 1 -- 172.21.15.28:0/1270697181 wait complete. 2024-04-21T08:50:35.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:35 smithi028 ceph-mon[28003]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:35.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:35 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1267333899' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:35.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:35 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2753112515' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:50:35.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:35 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1118734884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:50:35.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:50:35.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.578+0000 7f20337f4640 1 -- 172.21.15.28:0/3066036839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f202c102a00 msgr2=0x7f202c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.578+0000 7f20337f4640 1 --2- 172.21.15.28:0/3066036839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f202c102a00 0x7f202c109320 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2024009ca0 tx=0x7f202402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.578+0000 7f20337f4640 1 -- 172.21.15.28:0/3066036839 shutdown_connections 2024-04-21T08:50:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.578+0000 7f20337f4640 1 --2- 172.21.15.28:0/3066036839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f202c102a00 0x7f202c109320 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.578+0000 7f20337f4640 1 --2- 172.21.15.28:0/3066036839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f202c102040 0x7f202c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.578+0000 7f20337f4640 1 --2- 172.21.15.28:0/3066036839 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f202c100e60 0x7f202c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.578+0000 7f20337f4640 1 -- 172.21.15.28:0/3066036839 >> 172.21.15.28:0/3066036839 conn(0x7f202c0fc440 msgr2=0x7f202c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.578+0000 7f20337f4640 1 -- 172.21.15.28:0/3066036839 shutdown_connections 2024-04-21T08:50:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.578+0000 7f20337f4640 1 -- 172.21.15.28:0/3066036839 wait complete. 2024-04-21T08:50:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.578+0000 7f20337f4640 1 Processor -- start 2024-04-21T08:50:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.578+0000 7f20337f4640 1 -- start start 2024-04-21T08:50:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f20337f4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f202c100e60 0x7f202c1970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f20337f4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f202c102040 0x7f202c1975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:35.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f20337f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f202c102a00 0x7f202c19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f20337f4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f202c10ca10 con 0x7f202c102a00 2024-04-21T08:50:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f20337f4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f202c10c890 con 0x7f202c102040 2024-04-21T08:50:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f20337f4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f202c10cb90 con 0x7f202c100e60 2024-04-21T08:50:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f2031d91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f202c102a00 0x7f202c19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f2031d91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f202c102a00 0x7f202c19e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59196/0 (socket says 172.21.15.28:59196) 2024-04-21T08:50:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f2031d91640 1 -- 172.21.15.28:0/3279577666 learned_addr learned my addr 172.21.15.28:0/3279577666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f2030d8f640 1 --2- 172.21.15.28:0/3279577666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f202c102040 0x7f202c1975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f2031d91640 1 -- 172.21.15.28:0/3279577666 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f202c100e60 msgr2=0x7f202c1970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f2031590640 1 --2- 172.21.15.28:0/3279577666 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f202c100e60 0x7f202c1970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f2031d91640 1 --2- 172.21.15.28:0/3279577666 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f202c100e60 0x7f202c1970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f2031d91640 1 -- 172.21.15.28:0/3279577666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f202c102040 msgr2=0x7f202c1975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f2031d91640 1 --2- 172.21.15.28:0/3279577666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f202c102040 0x7f202c1975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f2031d91640 1 -- 172.21.15.28:0/3279577666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202c19ecd0 con 0x7f202c102a00 2024-04-21T08:50:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f2030d8f640 1 --2- 172.21.15.28:0/3279577666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f202c102040 0x7f202c1975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f2031d91640 1 --2- 172.21.15.28:0/3279577666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f202c102a00 0x7f202c19e660 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f202402c700 tx=0x7f2024034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:35.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f20227fc640 1 -- 172.21.15.28:0/3279577666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2024039040 con 0x7f202c102a00 2024-04-21T08:50:35.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f20227fc640 1 -- 172.21.15.28:0/3279577666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2024044070 con 0x7f202c102a00 2024-04-21T08:50:35.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f20337f4640 1 -- 172.21.15.28:0/3279577666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f202c19ef60 con 0x7f202c102a00 2024-04-21T08:50:35.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f20227fc640 1 -- 172.21.15.28:0/3279577666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20240042a0 con 0x7f202c102a00 2024-04-21T08:50:35.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.579+0000 7f20337f4640 1 -- 172.21.15.28:0/3279577666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f202c19f3c0 con 0x7f202c102a00 2024-04-21T08:50:35.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.580+0000 7f20337f4640 1 -- 172.21.15.28:0/3279577666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f202c108200 con 0x7f202c102a00 2024-04-21T08:50:35.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.582+0000 7f20227fc640 1 -- 172.21.15.28:0/3279577666 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2024003970 con 0x7f202c102a00 2024-04-21T08:50:35.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.582+0000 7f20227fc640 1 --2- 172.21.15.28:0/3279577666 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f20180a3c40 0x7f20180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:35.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.582+0000 7f20227fc640 1 -- 172.21.15.28:0/3279577666 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f20240f36e0 con 0x7f202c102a00 2024-04-21T08:50:35.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.582+0000 7f2031590640 1 --2- 172.21.15.28:0/3279577666 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f20180a3c40 0x7f20180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:35.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.583+0000 7f2031590640 1 --2- 172.21.15.28:0/3279577666 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f20180a3c40 0x7f20180a6100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2028005fd0 tx=0x7f2028005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:35.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.586+0000 7f20227fc640 1 -- 172.21.15.28:0/3279577666 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f20240bfd30 con 0x7f202c102a00 2024-04-21T08:50:35.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:35 smithi152 ceph-mon[31401]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:35.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:35 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1267333899' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:35.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:35 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2753112515' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:50:35.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:35 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1118734884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:50:35.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:35 smithi171 ceph-mon[31358]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:35.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:35 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1267333899' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:35.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:35 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2753112515' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:50:35.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:35 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1118734884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:50:35.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.772+0000 7f20337f4640 1 -- 172.21.15.28:0/3279577666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f202c02d5c0 con 0x7f202c102a00 2024-04-21T08:50:35.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.773+0000 7f20227fc640 1 -- 172.21.15.28:0/3279577666 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f20240bf630 con 0x7f202c102a00 2024-04-21T08:50:35.773 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:50:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.775+0000 7f20337f4640 1 -- 172.21.15.28:0/3279577666 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f20180a3c40 msgr2=0x7f20180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.775+0000 7f20337f4640 1 --2- 172.21.15.28:0/3279577666 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f20180a3c40 0x7f20180a6100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2028005fd0 tx=0x7f2028005850 comp rx=0 tx=0).stop 2024-04-21T08:50:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.775+0000 7f20337f4640 1 -- 172.21.15.28:0/3279577666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f202c102a00 msgr2=0x7f202c19e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.775+0000 7f20337f4640 1 --2- 172.21.15.28:0/3279577666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f202c102a00 0x7f202c19e660 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f202402c700 tx=0x7f2024034760 comp rx=0 tx=0).stop 2024-04-21T08:50:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.775+0000 7f20337f4640 1 -- 172.21.15.28:0/3279577666 shutdown_connections 2024-04-21T08:50:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.775+0000 7f20337f4640 1 --2- 172.21.15.28:0/3279577666 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f20180a3c40 0x7f20180a6100 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.775+0000 7f20337f4640 1 --2- 172.21.15.28:0/3279577666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f202c102a00 0x7f202c19e660 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.775+0000 7f20337f4640 1 --2- 172.21.15.28:0/3279577666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f202c102040 0x7f202c1975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.775+0000 7f20337f4640 1 --2- 172.21.15.28:0/3279577666 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f202c100e60 0x7f202c1970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.775+0000 7f20337f4640 1 -- 172.21.15.28:0/3279577666 >> 172.21.15.28:0/3279577666 conn(0x7f202c0fc440 msgr2=0x7f202c1053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.775+0000 7f20337f4640 1 -- 172.21.15.28:0/3279577666 shutdown_connections 2024-04-21T08:50:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.775+0000 7f20337f4640 1 -- 172.21.15.28:0/3279577666 wait complete. 2024-04-21T08:50:35.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:50:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.888+0000 7f7cf6cd2640 1 -- 172.21.15.28:0/2700787561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf01053f0 msgr2=0x7f7cf01077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.888+0000 7f7cf6cd2640 1 --2- 172.21.15.28:0/2700787561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf01053f0 0x7f7cf01077e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009900 tx=0x7f7ce402bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf6cd2640 1 -- 172.21.15.28:0/2700787561 shutdown_connections 2024-04-21T08:50:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf6cd2640 1 --2- 172.21.15.28:0/2700787561 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7cf0107d20 0x7f7cf010a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf6cd2640 1 --2- 172.21.15.28:0/2700787561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf01053f0 0x7f7cf01077e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf6cd2640 1 --2- 172.21.15.28:0/2700787561 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cf0102ac0 0x7f7cf0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf6cd2640 1 -- 172.21.15.28:0/2700787561 >> 172.21.15.28:0/2700787561 conn(0x7f7cf00fc410 msgr2=0x7f7cf00fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf6cd2640 1 -- 172.21.15.28:0/2700787561 shutdown_connections 2024-04-21T08:50:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf6cd2640 1 -- 172.21.15.28:0/2700787561 wait complete. 2024-04-21T08:50:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf6cd2640 1 Processor -- start 2024-04-21T08:50:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf6cd2640 1 -- start start 2024-04-21T08:50:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf6cd2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf0102ac0 0x7f7cf01981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf6cd2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cf01053f0 0x7f7cf0198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf6cd2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7cf0107d20 0x7f7cf019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf6cd2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf010db00 con 0x7f7cf0102ac0 2024-04-21T08:50:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf6cd2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf010d980 con 0x7f7cf01053f0 2024-04-21T08:50:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf6cd2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf010dc80 con 0x7f7cf0107d20 2024-04-21T08:50:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf4a6e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf0102ac0 0x7f7cf01981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf4a6e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf0102ac0 0x7f7cf01981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59214/0 (socket says 172.21.15.28:59214) 2024-04-21T08:50:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.889+0000 7f7cf4a6e640 1 -- 172.21.15.28:0/3692838442 learned_addr learned my addr 172.21.15.28:0/3692838442 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.890+0000 7f7cf526f640 1 --2- 172.21.15.28:0/3692838442 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7cf0107d20 0x7f7cf019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.890+0000 7f7cf4a6e640 1 -- 172.21.15.28:0/3692838442 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7cf0107d20 msgr2=0x7f7cf019f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.890+0000 7f7cf4a6e640 1 --2- 172.21.15.28:0/3692838442 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7cf0107d20 0x7f7cf019f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.890+0000 7f7cf4a6e640 1 -- 172.21.15.28:0/3692838442 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cf01053f0 msgr2=0x7f7cf0198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.890+0000 7f7ceffff640 1 --2- 172.21.15.28:0/3692838442 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cf01053f0 0x7f7cf0198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.890+0000 7f7cf4a6e640 1 --2- 172.21.15.28:0/3692838442 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cf01053f0 0x7f7cf0198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.890+0000 7f7cf4a6e640 1 -- 172.21.15.28:0/3692838442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf019fdf0 con 0x7f7cf0102ac0 2024-04-21T08:50:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.890+0000 7f7cf526f640 1 --2- 172.21.15.28:0/3692838442 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7cf0107d20 0x7f7cf019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.890+0000 7f7cf4a6e640 1 --2- 172.21.15.28:0/3692838442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf0102ac0 0x7f7cf01981d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800ebc0 tx=0x7f7ce800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.890+0000 7f7cedffb640 1 -- 172.21.15.28:0/3692838442 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8010630 con 0x7f7cf0102ac0 2024-04-21T08:50:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.890+0000 7f7cedffb640 1 -- 172.21.15.28:0/3692838442 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ce8010c80 con 0x7f7cf0102ac0 2024-04-21T08:50:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.890+0000 7f7cf6cd2640 1 -- 172.21.15.28:0/3692838442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf01a00e0 con 0x7f7cf0102ac0 2024-04-21T08:50:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.890+0000 7f7cedffb640 1 -- 172.21.15.28:0/3692838442 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8018b60 con 0x7f7cf0102ac0 2024-04-21T08:50:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.890+0000 7f7cf6cd2640 1 -- 172.21.15.28:0/3692838442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf01a0630 con 0x7f7cf0102ac0 2024-04-21T08:50:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.891+0000 7f7cf6cd2640 1 -- 172.21.15.28:0/3692838442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cf00fdf00 con 0x7f7cf0102ac0 2024-04-21T08:50:35.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.892+0000 7f7cedffb640 1 -- 172.21.15.28:0/3692838442 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7ce8018d00 con 0x7f7cf0102ac0 2024-04-21T08:50:35.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.893+0000 7f7cedffb640 1 --2- 172.21.15.28:0/3692838442 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7cd80a3c40 0x7f7cd80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:35.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.893+0000 7f7ceffff640 1 --2- 172.21.15.28:0/3692838442 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7cd80a3c40 0x7f7cd80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:35.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.893+0000 7f7cedffb640 1 -- 172.21.15.28:0/3692838442 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f7ce80d1000 con 0x7f7cf0102ac0 2024-04-21T08:50:35.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.893+0000 7f7ceffff640 1 --2- 172.21.15.28:0/3692838442 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7cd80a3c40 0x7f7cd80a6100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7ce402c840 tx=0x7f7ce4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:35.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:35.898+0000 7f7cedffb640 1 -- 172.21.15.28:0/3692838442 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7ce809d540 con 0x7f7cf0102ac0 2024-04-21T08:50:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.126+0000 7f7cf6cd2640 1 -- 172.21.15.28:0/3692838442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7cf002d5c0 con 0x7f7cf0102ac0 2024-04-21T08:50:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.126+0000 7f7cedffb640 1 -- 172.21.15.28:0/3692838442 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f7ce809ce40 con 0x7f7cf0102ac0 2024-04-21T08:50:36.127 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:50:36.127 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:49:12.840022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24463,24487,24488]},"id":1}]} 2024-04-21T08:50:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-21T08:50:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.129+0000 7f7cf6cd2640 1 -- 172.21.15.28:0/3692838442 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7cd80a3c40 msgr2=0x7f7cd80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.129+0000 7f7cf6cd2640 1 --2- 172.21.15.28:0/3692838442 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7cd80a3c40 0x7f7cd80a6100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7ce402c840 tx=0x7f7ce4000bc0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.129+0000 7f7cf6cd2640 1 -- 172.21.15.28:0/3692838442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf0102ac0 msgr2=0x7f7cf01981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.129+0000 7f7cf6cd2640 1 --2- 172.21.15.28:0/3692838442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf0102ac0 0x7f7cf01981d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800ebc0 tx=0x7f7ce800c6a0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.129+0000 7f7cf6cd2640 1 -- 172.21.15.28:0/3692838442 shutdown_connections 2024-04-21T08:50:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.129+0000 7f7cf6cd2640 1 --2- 172.21.15.28:0/3692838442 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7cf0107d20 0x7f7cf019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.129+0000 7f7cf6cd2640 1 --2- 172.21.15.28:0/3692838442 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cf01053f0 0x7f7cf0198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.129+0000 7f7cf6cd2640 1 --2- 172.21.15.28:0/3692838442 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f7cd80a3c40 0x7f7cd80a6100 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.129+0000 7f7cf6cd2640 1 --2- 172.21.15.28:0/3692838442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf0102ac0 0x7f7cf01981d0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.129+0000 7f7cf6cd2640 1 -- 172.21.15.28:0/3692838442 >> 172.21.15.28:0/3692838442 conn(0x7f7cf00fc410 msgr2=0x7f7cf01034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.129+0000 7f7cf6cd2640 1 -- 172.21.15.28:0/3692838442 shutdown_connections 2024-04-21T08:50:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.129+0000 7f7cf6cd2640 1 -- 172.21.15.28:0/3692838442 wait complete. 2024-04-21T08:50:36.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:50:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.243+0000 7fe18dd2c640 1 -- 172.21.15.28:0/492870778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188103190 msgr2=0x7fe188103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.243+0000 7fe18dd2c640 1 --2- 172.21.15.28:0/492870778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188103190 0x7fe188103610 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe17c009900 tx=0x7fe17c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.243+0000 7fe18dd2c640 1 -- 172.21.15.28:0/492870778 shutdown_connections 2024-04-21T08:50:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.243+0000 7fe18dd2c640 1 --2- 172.21.15.28:0/492870778 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe188103b50 0x7fe18810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.243+0000 7fe18dd2c640 1 --2- 172.21.15.28:0/492870778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188103190 0x7fe188103610 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.243+0000 7fe18dd2c640 1 --2- 172.21.15.28:0/492870778 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe188101fa0 0x7fe1881023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.243+0000 7fe18dd2c640 1 -- 172.21.15.28:0/492870778 >> 172.21.15.28:0/492870778 conn(0x7fe1880fd530 msgr2=0x7fe1880ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.243+0000 7fe18dd2c640 1 -- 172.21.15.28:0/492870778 shutdown_connections 2024-04-21T08:50:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.243+0000 7fe18dd2c640 1 -- 172.21.15.28:0/492870778 wait complete. 2024-04-21T08:50:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe18dd2c640 1 Processor -- start 2024-04-21T08:50:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe18dd2c640 1 -- start start 2024-04-21T08:50:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe18dd2c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188101fa0 0x7fe188198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe18dd2c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe188103190 0x7fe188198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe18dd2c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe188103b50 0x7fe18819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe18dd2c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe18810dae0 con 0x7fe188101fa0 2024-04-21T08:50:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe18dd2c640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe18810d960 con 0x7fe188103190 2024-04-21T08:50:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe18dd2c640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe18810dc60 con 0x7fe188103b50 2024-04-21T08:50:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe1877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188101fa0 0x7fe188198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe1877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188101fa0 0x7fe188198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59228/0 (socket says 172.21.15.28:59228) 2024-04-21T08:50:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe1877fe640 1 -- 172.21.15.28:0/404451728 learned_addr learned my addr 172.21.15.28:0/404451728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe187fff640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe188103b50 0x7fe18819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe186ffd640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe188103190 0x7fe188198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe1877fe640 1 -- 172.21.15.28:0/404451728 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe188103b50 msgr2=0x7fe18819f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe1877fe640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe188103b50 0x7fe18819f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe1877fe640 1 -- 172.21.15.28:0/404451728 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe188103190 msgr2=0x7fe188198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe1877fe640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe188103190 0x7fe188198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.244+0000 7fe1877fe640 1 -- 172.21.15.28:0/404451728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18819fd50 con 0x7fe188101fa0 2024-04-21T08:50:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.245+0000 7fe187fff640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe188103b50 0x7fe18819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.245+0000 7fe186ffd640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe188103190 0x7fe188198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.245+0000 7fe1877fe640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188101fa0 0x7fe188198130 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe17800d770 tx=0x7fe17800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.245+0000 7fe184ff9640 1 -- 172.21.15.28:0/404451728 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe178017070 con 0x7fe188101fa0 2024-04-21T08:50:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.245+0000 7fe184ff9640 1 -- 172.21.15.28:0/404451728 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe178009c30 con 0x7fe188101fa0 2024-04-21T08:50:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.245+0000 7fe18dd2c640 1 -- 172.21.15.28:0/404451728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe18819ffe0 con 0x7fe188101fa0 2024-04-21T08:50:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.245+0000 7fe184ff9640 1 -- 172.21.15.28:0/404451728 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe178003cc0 con 0x7fe188101fa0 2024-04-21T08:50:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.245+0000 7fe18dd2c640 1 -- 172.21.15.28:0/404451728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1881a04f0 con 0x7fe188101fa0 2024-04-21T08:50:36.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.246+0000 7fe18dd2c640 1 -- 172.21.15.28:0/404451728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1881092c0 con 0x7fe188101fa0 2024-04-21T08:50:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.247+0000 7fe184ff9640 1 -- 172.21.15.28:0/404451728 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe178003e60 con 0x7fe188101fa0 2024-04-21T08:50:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.248+0000 7fe184ff9640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe1700a3cc0 0x7fe1700a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.248+0000 7fe186ffd640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe1700a3cc0 0x7fe1700a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.248+0000 7fe184ff9640 1 -- 172.21.15.28:0/404451728 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fe1780d0690 con 0x7fe188101fa0 2024-04-21T08:50:36.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.248+0000 7fe186ffd640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe1700a3cc0 0x7fe1700a6180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe17c0096c0 tx=0x7fe17c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:36.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.251+0000 7fe184ff9640 1 -- 172.21.15.28:0/404451728 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe178009de0 con 0x7fe188101fa0 2024-04-21T08:50:36.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.290+0000 7fe184ff9640 1 -- 172.21.15.28:0/404451728 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe178098bc0 con 0x7fe188101fa0 2024-04-21T08:50:36.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.452+0000 7fe18dd2c640 1 -- 172.21.15.28:0/404451728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe18802d5c0 con 0x7fe188101fa0 2024-04-21T08:50:36.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.453+0000 7fe184ff9640 1 -- 172.21.15.28:0/404451728 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24786 (secure 0 0 0) 0x7fe17809cc50 con 0x7fe188101fa0 2024-04-21T08:50:36.453 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:50:36.453 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":92,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:50:36.268959+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:50:36.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.455+0000 7fe18dd2c640 1 -- 172.21.15.28:0/404451728 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe1700a3cc0 msgr2=0x7fe1700a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.455+0000 7fe18dd2c640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe1700a3cc0 0x7fe1700a6180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe17c0096c0 tx=0x7fe17c000bc0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.455+0000 7fe18dd2c640 1 -- 172.21.15.28:0/404451728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188101fa0 msgr2=0x7fe188198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.455+0000 7fe18dd2c640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188101fa0 0x7fe188198130 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe17800d770 tx=0x7fe17800dc30 comp rx=0 tx=0).stop 2024-04-21T08:50:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.455+0000 7fe18dd2c640 1 -- 172.21.15.28:0/404451728 shutdown_connections 2024-04-21T08:50:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.455+0000 7fe18dd2c640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe188103b50 0x7fe18819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.455+0000 7fe18dd2c640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe188103190 0x7fe188198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.455+0000 7fe18dd2c640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe1700a3cc0 0x7fe1700a6180 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.455+0000 7fe18dd2c640 1 --2- 172.21.15.28:0/404451728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188101fa0 0x7fe188198130 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.455+0000 7fe18dd2c640 1 -- 172.21.15.28:0/404451728 >> 172.21.15.28:0/404451728 conn(0x7fe1880fd530 msgr2=0x7fe188106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.456+0000 7fe18dd2c640 1 -- 172.21.15.28:0/404451728 shutdown_connections 2024-04-21T08:50:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.456+0000 7fe18dd2c640 1 -- 172.21.15.28:0/404451728 wait complete. 2024-04-21T08:50:36.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:36 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1270697181' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:50:36.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:36 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3279577666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:50:36.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-04-21T08:50:36.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 10]}]: dispatch 2024-04-21T08:50:36.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-04-21T08:50:36.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:36 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-04-21T08:50:36.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:36 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3692838442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:36.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:50:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.570+0000 7fe9d0fa4640 1 -- 172.21.15.28:0/2595345998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc101fa0 msgr2=0x7fe9cc1023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.570+0000 7fe9d0fa4640 1 --2- 172.21.15.28:0/2595345998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc101fa0 0x7fe9cc1023c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009ca0 tx=0x7fe9bc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.570+0000 7fe9d0fa4640 1 -- 172.21.15.28:0/2595345998 shutdown_connections 2024-04-21T08:50:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.570+0000 7fe9d0fa4640 1 --2- 172.21.15.28:0/2595345998 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9cc103b50 0x7fe9cc10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.570+0000 7fe9d0fa4640 1 --2- 172.21.15.28:0/2595345998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe9cc103190 0x7fe9cc103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.570+0000 7fe9d0fa4640 1 --2- 172.21.15.28:0/2595345998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc101fa0 0x7fe9cc1023c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.570+0000 7fe9d0fa4640 1 -- 172.21.15.28:0/2595345998 >> 172.21.15.28:0/2595345998 conn(0x7fe9cc0fd530 msgr2=0x7fe9cc0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.570+0000 7fe9d0fa4640 1 -- 172.21.15.28:0/2595345998 shutdown_connections 2024-04-21T08:50:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.570+0000 7fe9d0fa4640 1 -- 172.21.15.28:0/2595345998 wait complete. 2024-04-21T08:50:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9d0fa4640 1 Processor -- start 2024-04-21T08:50:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9d0fa4640 1 -- start start 2024-04-21T08:50:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9d0fa4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc101fa0 0x7fe9cc198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9d0fa4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe9cc103190 0x7fe9cc1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9d0fa4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9cc103b50 0x7fe9cc19f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9d0fa4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc10dae0 con 0x7fe9cc101fa0 2024-04-21T08:50:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9d0fa4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc10d960 con 0x7fe9cc103b50 2024-04-21T08:50:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9d0fa4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc10dc60 con 0x7fe9cc103190 2024-04-21T08:50:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9ca59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc101fa0 0x7fe9cc198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9ca59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc101fa0 0x7fe9cc198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59240/0 (socket says 172.21.15.28:59240) 2024-04-21T08:50:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9ca59c640 1 -- 172.21.15.28:0/3778635124 learned_addr learned my addr 172.21.15.28:0/3778635124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9c9d9b640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe9cc103190 0x7fe9cc1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9ca59c640 1 -- 172.21.15.28:0/3778635124 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe9cc103190 msgr2=0x7fe9cc1986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9ca59c640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe9cc103190 0x7fe9cc1986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9ca59c640 1 -- 172.21.15.28:0/3778635124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9cc103b50 msgr2=0x7fe9cc19f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9cad9d640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9cc103b50 0x7fe9cc19f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9ca59c640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9cc103b50 0x7fe9cc19f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9ca59c640 1 -- 172.21.15.28:0/3778635124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9cc19fd80 con 0x7fe9cc101fa0 2024-04-21T08:50:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9c9d9b640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe9cc103190 0x7fe9cc1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.571+0000 7fe9cad9d640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9cc103b50 0x7fe9cc19f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.572+0000 7fe9ca59c640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc101fa0 0x7fe9cc198160 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc032380 tx=0x7fe9bc0347b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.572+0000 7fe9c37fe640 1 -- 172.21.15.28:0/3778635124 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc039040 con 0x7fe9cc101fa0 2024-04-21T08:50:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.572+0000 7fe9c37fe640 1 -- 172.21.15.28:0/3778635124 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9bc044070 con 0x7fe9cc101fa0 2024-04-21T08:50:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.572+0000 7fe9d0fa4640 1 -- 172.21.15.28:0/3778635124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9cc1a0010 con 0x7fe9cc101fa0 2024-04-21T08:50:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.572+0000 7fe9c37fe640 1 -- 172.21.15.28:0/3778635124 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc0042a0 con 0x7fe9cc101fa0 2024-04-21T08:50:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.572+0000 7fe9d0fa4640 1 -- 172.21.15.28:0/3778635124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9cc1a0550 con 0x7fe9cc101fa0 2024-04-21T08:50:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.573+0000 7fe9d0fa4640 1 -- 172.21.15.28:0/3778635124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9cc1092c0 con 0x7fe9cc101fa0 2024-04-21T08:50:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.574+0000 7fe9c37fe640 1 -- 172.21.15.28:0/3778635124 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe9bc003a00 con 0x7fe9cc101fa0 2024-04-21T08:50:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.575+0000 7fe9c37fe640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe9b80a3c30 0x7fe9b80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.575+0000 7fe9c9d9b640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe9b80a3c30 0x7fe9b80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.575+0000 7fe9c37fe640 1 -- 172.21.15.28:0/3778635124 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9187+0+0 (secure 0 0 0) 0x7fe9bc0f2bb0 con 0x7fe9cc101fa0 2024-04-21T08:50:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.575+0000 7fe9c9d9b640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe9b80a3c30 0x7fe9b80a60f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc199670 tx=0x7fe9b4005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:36.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.579+0000 7fe9c37fe640 1 -- 172.21.15.28:0/3778635124 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe9bc0bf1d0 con 0x7fe9cc101fa0 2024-04-21T08:50:36.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:36 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1270697181' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:50:36.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:36 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3279577666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:50:36.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-04-21T08:50:36.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 10]}]: dispatch 2024-04-21T08:50:36.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-04-21T08:50:36.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:36 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-04-21T08:50:36.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:36 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3692838442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:36.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:36 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1270697181' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:50:36.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:36 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3279577666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:50:36.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-04-21T08:50:36.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 10]}]: dispatch 2024-04-21T08:50:36.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-04-21T08:50:36.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:36 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-04-21T08:50:36.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:36 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3692838442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:36.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.805+0000 7fe9d0fa4640 1 -- 172.21.15.28:0/3778635124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9cc02d5c0 con 0x7fe9cc101fa0 2024-04-21T08:50:36.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.806+0000 7fe9c37fe640 1 -- 172.21.15.28:0/3778635124 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7fe9bc0bead0 con 0x7fe9cc101fa0 2024-04-21T08:50:36.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:50:36.806 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:49:12.840022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24463,24487,24488]},"id":1}]} 2024-04-21T08:50:36.806 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-21T08:50:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.808+0000 7fe9d0fa4640 1 -- 172.21.15.28:0/3778635124 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe9b80a3c30 msgr2=0x7fe9b80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.808+0000 7fe9d0fa4640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe9b80a3c30 0x7fe9b80a60f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc199670 tx=0x7fe9b4005e50 comp rx=0 tx=0).stop 2024-04-21T08:50:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.808+0000 7fe9d0fa4640 1 -- 172.21.15.28:0/3778635124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc101fa0 msgr2=0x7fe9cc198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.808+0000 7fe9d0fa4640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc101fa0 0x7fe9cc198160 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc032380 tx=0x7fe9bc0347b0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.808+0000 7fe9d0fa4640 1 -- 172.21.15.28:0/3778635124 shutdown_connections 2024-04-21T08:50:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.808+0000 7fe9d0fa4640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9cc103b50 0x7fe9cc19f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.808+0000 7fe9d0fa4640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe9cc103190 0x7fe9cc1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.808+0000 7fe9d0fa4640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fe9b80a3c30 0x7fe9b80a60f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.808+0000 7fe9d0fa4640 1 --2- 172.21.15.28:0/3778635124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9cc101fa0 0x7fe9cc198160 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.808+0000 7fe9d0fa4640 1 -- 172.21.15.28:0/3778635124 >> 172.21.15.28:0/3778635124 conn(0x7fe9cc0fd530 msgr2=0x7fe9cc106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:36.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.808+0000 7fe9d0fa4640 1 -- 172.21.15.28:0/3778635124 shutdown_connections 2024-04-21T08:50:36.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.809+0000 7fe9d0fa4640 1 -- 172.21.15.28:0/3778635124 wait complete. 2024-04-21T08:50:36.819 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T08:48:45.911185+0000', 'modified': '2024-04-21T08:49:12.840022+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24488, 'mds_1': 24487, 'mds_2': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24419': {'gid': 24419, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.152:6835/3982885739', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6834', 'nonce': 3982885739}, {'type': 'v1', 'addr': '172.21.15.152:6835', 'nonce': 3982885739}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24452': {'gid': 24452, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.152:6837/1261226890', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6836', 'nonce': 1261226890}, {'type': 'v1', 'addr': '172.21.15.152:6837', 'nonce': 1261226890}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.171:6835/663606808', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6834', 'nonce': 663606808}, {'type': 'v1', 'addr': '172.21.15.171:6835', 'nonce': 663606808}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.171:6837/895402983', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6836', 'nonce': 895402983}, {'type': 'v1', 'addr': '172.21.15.171:6837', 'nonce': 895402983}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.171:6839/3423264764', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6838', 'nonce': 3423264764}, {'type': 'v1', 'addr': '172.21.15.171:6839', 'nonce': 3423264764}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.152:6839/3821365920', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6838', 'nonce': 3821365920}, {'type': 'v1', 'addr': '172.21.15.152:6839', 'nonce': 3821365920}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24488, 'qdb_cluster': [24463, 24487, 24488]} 2024-04-21T08:50:36.820 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-21T08:50:36.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-21T08:50:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.923+0000 7fbb0c025640 1 -- 172.21.15.28:0/2638881709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb041029a0 msgr2=0x7fbb041092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.923+0000 7fbb0c025640 1 --2- 172.21.15.28:0/2638881709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb041029a0 0x7fbb041092c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fbafc009ca0 tx=0x7fbafc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.923+0000 7fbb0c025640 1 -- 172.21.15.28:0/2638881709 shutdown_connections 2024-04-21T08:50:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.923+0000 7fbb0c025640 1 --2- 172.21.15.28:0/2638881709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb041029a0 0x7fbb041092c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.923+0000 7fbb0c025640 1 --2- 172.21.15.28:0/2638881709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb04101fe0 0x7fbb04102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.923+0000 7fbb0c025640 1 --2- 172.21.15.28:0/2638881709 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb04100e90 0x7fbb041012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.923+0000 7fbb0c025640 1 -- 172.21.15.28:0/2638881709 >> 172.21.15.28:0/2638881709 conn(0x7fbb040fc400 msgr2=0x7fbb040fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.923+0000 7fbb0c025640 1 -- 172.21.15.28:0/2638881709 shutdown_connections 2024-04-21T08:50:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.923+0000 7fbb0c025640 1 -- 172.21.15.28:0/2638881709 wait complete. 2024-04-21T08:50:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0c025640 1 Processor -- start 2024-04-21T08:50:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0c025640 1 -- start start 2024-04-21T08:50:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0c025640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb04100e90 0x7fbb04069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0c025640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb04101fe0 0x7fbb0419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0c025640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb041029a0 0x7fbb0419f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0c025640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0410c9d0 con 0x7fbb041029a0 2024-04-21T08:50:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0c025640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0410c850 con 0x7fbb04101fe0 2024-04-21T08:50:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0c025640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0410cb50 con 0x7fbb04100e90 2024-04-21T08:50:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0a5c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb041029a0 0x7fbb0419f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0a5c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb041029a0 0x7fbb0419f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59264/0 (socket says 172.21.15.28:59264) 2024-04-21T08:50:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0a5c2640 1 -- 172.21.15.28:0/3086273425 learned_addr learned my addr 172.21.15.28:0/3086273425 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb095c0640 1 --2- 172.21.15.28:0/3086273425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb04101fe0 0x7fbb0419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb09dc1640 1 --2- 172.21.15.28:0/3086273425 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb04100e90 0x7fbb04069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0a5c2640 1 -- 172.21.15.28:0/3086273425 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb04100e90 msgr2=0x7fbb04069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0a5c2640 1 --2- 172.21.15.28:0/3086273425 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb04100e90 0x7fbb04069c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0a5c2640 1 -- 172.21.15.28:0/3086273425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb04101fe0 msgr2=0x7fbb0419ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0a5c2640 1 --2- 172.21.15.28:0/3086273425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb04101fe0 0x7fbb0419ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0a5c2640 1 -- 172.21.15.28:0/3086273425 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb0419fdc0 con 0x7fbb041029a0 2024-04-21T08:50:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb095c0640 1 --2- 172.21.15.28:0/3086273425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb04101fe0 0x7fbb0419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbb0a5c2640 1 --2- 172.21.15.28:0/3086273425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb041029a0 0x7fbb0419f4b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbafc02c700 tx=0x7fbafc034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.924+0000 7fbafaffd640 1 -- 172.21.15.28:0/3086273425 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc039040 con 0x7fbb041029a0 2024-04-21T08:50:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.925+0000 7fbafaffd640 1 -- 172.21.15.28:0/3086273425 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbafc044070 con 0x7fbb041029a0 2024-04-21T08:50:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.925+0000 7fbb0c025640 1 -- 172.21.15.28:0/3086273425 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb0419fff0 con 0x7fbb041029a0 2024-04-21T08:50:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.925+0000 7fbafaffd640 1 -- 172.21.15.28:0/3086273425 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc0042a0 con 0x7fbb041029a0 2024-04-21T08:50:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.925+0000 7fbb0c025640 1 -- 172.21.15.28:0/3086273425 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb041a0450 con 0x7fbb041029a0 2024-04-21T08:50:36.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.927+0000 7fbafaffd640 1 -- 172.21.15.28:0/3086273425 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbafc003970 con 0x7fbb041029a0 2024-04-21T08:50:36.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb0ec13740 1 Processor -- start 2024-04-21T08:50:36.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb0ec13740 1 -- start start 2024-04-21T08:50:36.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb0ec13740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea97e40 0x56125ea98260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb0ec13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea98830 0x56125ea98cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb0ec13740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea992d0 0x56125ea9feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb0ec13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56125e9fcb10 con 0x56125ea98830 2024-04-21T08:50:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb0ec13740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56125e99a240 con 0x56125ea992d0 2024-04-21T08:50:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb0ec13740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56125e9f8ad0 con 0x56125ea97e40 2024-04-21T08:50:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb09dc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea98830 0x56125ea98cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb09dc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea98830 0x56125ea98cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59268/0 (socket says 172.21.15.28:59268) 2024-04-21T08:50:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb09dc1640 1 -- 172.21.15.28:0/1882177073 learned_addr learned my addr 172.21.15.28:0/1882177073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbafaffd640 1 --2- 172.21.15.28:0/3086273425 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbaf00a3bb0 0x7fbaf00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb095c0640 1 --2- 172.21.15.28:0/1882177073 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea992d0 0x56125ea9feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb095c0640 1 --2- 172.21.15.28:0/1882177073 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea97e40 0x56125ea98260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb09dc1640 1 -- 172.21.15.28:0/1882177073 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea97e40 msgr2=0x56125ea98260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb09dc1640 1 --2- 172.21.15.28:0/1882177073 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea97e40 0x56125ea98260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb09dc1640 1 -- 172.21.15.28:0/1882177073 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea992d0 msgr2=0x56125ea9feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb0a5c2640 1 --2- 172.21.15.28:0/3086273425 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbaf00a3bb0 0x7fbaf00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb09dc1640 1 --2- 172.21.15.28:0/1882177073 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea992d0 0x56125ea9feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb09dc1640 1 -- 172.21.15.28:0/1882177073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56125e70f2d0 con 0x56125ea98830 2024-04-21T08:50:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.928+0000 7fbb095c0640 1 --2- 172.21.15.28:0/1882177073 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea992d0 0x56125ea9feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.929+0000 7fbafaffd640 1 -- 172.21.15.28:0/3086273425 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9187+0+0 (secure 0 0 0) 0x7fbafc0f2ca0 con 0x7fbb041029a0 2024-04-21T08:50:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.929+0000 7fbb09dc1640 1 --2- 172.21.15.28:0/1882177073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea98830 0x56125ea98cb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400bcb0 tx=0x7fbaf400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9639a438ebb5091d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.929+0000 7fbaeffff640 1 -- 172.21.15.28:0/1882177073 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4015070 con 0x56125ea98830 2024-04-21T08:50:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.929+0000 7fbaeffff640 1 -- 172.21.15.28:0/1882177073 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaf401a910 con 0x56125ea98830 2024-04-21T08:50:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.929+0000 7fbaeffff640 1 -- 172.21.15.28:0/1882177073 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf40118b0 con 0x56125ea98830 2024-04-21T08:50:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.929+0000 7fbb0a5c2640 1 --2- 172.21.15.28:0/3086273425 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbaf00a3bb0 0x7fbaf00a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0be860 tx=0x7fbafc0e9c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.930+0000 7fbb0ec13740 1 -- 172.21.15.28:0/1882177073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea98830 msgr2=0x56125ea98cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.930+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/1882177073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea98830 0x56125ea98cb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbaf400bcb0 tx=0x7fbaf400f730 comp rx=0 tx=0).stop 2024-04-21T08:50:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.930+0000 7fbb0ec13740 1 -- 172.21.15.28:0/1882177073 shutdown_connections 2024-04-21T08:50:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.930+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/1882177073 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea992d0 0x56125ea9feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.930+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/1882177073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea98830 0x56125ea98cb0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.930+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/1882177073 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea97e40 0x56125ea98260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.930+0000 7fbb0ec13740 1 -- 172.21.15.28:0/1882177073 >> 172.21.15.28:0/1882177073 conn(0x56125ea93f40 msgr2=0x56125ea92490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.930+0000 7fbb0ec13740 1 -- 172.21.15.28:0/1882177073 shutdown_connections 2024-04-21T08:50:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.930+0000 7fbb0ec13740 1 -- 172.21.15.28:0/1882177073 wait complete. 2024-04-21T08:50:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.930+0000 7fbb0ec13740 1 -- start start 2024-04-21T08:50:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.930+0000 7fbb0ec13740 1 Processor -- start 2024-04-21T08:50:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb0ec13740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea97e40 0x56125eaa1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb0ec13740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea98830 0x56125eaa23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb0ec13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea992d0 0x56125eaa9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb0ec13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56125eaa04b0 con 0x56125ea992d0 2024-04-21T08:50:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb0ec13740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56125e985190 con 0x56125ea98830 2024-04-21T08:50:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb0ec13740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56125eaa0630 con 0x56125ea97e40 2024-04-21T08:50:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb0ec13740 10 client.0 _ll_register_callbacks cb 0x56125e9808b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb095c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea992d0 0x56125eaa9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb095c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea992d0 0x56125eaa9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59270/0 (socket says 172.21.15.28:59270) 2024-04-21T08:50:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb095c0640 1 -- 172.21.15.28:0/2493355072 learned_addr learned my addr 172.21.15.28:0/2493355072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb09dc1640 1 --2- 172.21.15.28:0/2493355072 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea98830 0x56125eaa23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb095c0640 1 --2- 172.21.15.28:0/2493355072 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea97e40 0x56125eaa1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb095c0640 1 -- 172.21.15.28:0/2493355072 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea97e40 msgr2=0x56125eaa1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb095c0640 1 --2- 172.21.15.28:0/2493355072 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea97e40 0x56125eaa1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb095c0640 1 -- 172.21.15.28:0/2493355072 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea98830 msgr2=0x56125eaa23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb095c0640 1 --2- 172.21.15.28:0/2493355072 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea98830 0x56125eaa23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb095c0640 1 -- 172.21.15.28:0/2493355072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56125e7d96b0 con 0x56125ea992d0 2024-04-21T08:50:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb095c0640 1 --2- 172.21.15.28:0/2493355072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea992d0 0x56125eaa9410 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbb040699c0 tx=0x7fbb00015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f6e8f3da948ed1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbaed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:50:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbaed7fa640 1 -- 172.21.15.28:0/2493355072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb00005020 con 0x56125ea992d0 2024-04-21T08:50:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbaed7fa640 1 -- 172.21.15.28:0/2493355072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9187+0+0 (secure 0 0 0) 0x7fbb000051c0 con 0x56125ea992d0 2024-04-21T08:50:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.931+0000 7fbb0ec13740 1 -- 172.21.15.28:0/2493355072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbaf401a910 con 0x56125ea992d0 2024-04-21T08:50:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.932+0000 7fbaed7fa640 1 -- 172.21.15.28:0/2493355072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbb00025050 con 0x56125ea992d0 2024-04-21T08:50:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.932+0000 7fbb0ec13740 10 client.15222 fetch_fsmap learned FSMap version 24 2024-04-21T08:50:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.932+0000 7fbb0ec13740 1 -- 172.21.15.28:0/2493355072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56125eaac7f0 con 0x56125ea992d0 2024-04-21T08:50:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.932+0000 7fbaed7fa640 1 -- 172.21.15.28:0/2493355072 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaf401a910 con 0x56125ea992d0 2024-04-21T08:50:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.932+0000 7fbaed7fa640 1 -- 172.21.15.28:0/2493355072 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fbb00011440 con 0x56125ea992d0 2024-04-21T08:50:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.932+0000 7fbb0ec13740 10 client.15222 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:50:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.932+0000 7fbb0ec13740 10 client.15222 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-21T08:50:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.932+0000 7fbb0ec13740 20 client.15222 populate_metadata read hostname 'smithi028' 2024-04-21T08:50:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.932+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/2493355072 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56125eaaeed0 0x56125eab0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.932+0000 7fbb0ec13740 4 client.15222 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.932+0000 7fbb0ec13740 1 -- 172.21.15.28:0/2493355072 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56125ea0a530 con 0x56125eaaeed0 2024-04-21T08:50:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.932+0000 7fbb09dc1640 1 --2- 172.21.15.28:0/2493355072 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56125eaaeed0 0x56125eab0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.933+0000 7fbaed7fa640 10 client.15222 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:50:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.933+0000 7fbb09dc1640 1 --2- 172.21.15.28:0/2493355072 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56125eaaeed0 0x56125eab0340 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4b337ba79529c8a7 server_cookie=146ea77807af274c in_seq=0 out_seq=0 2024-04-21T08:50:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.933+0000 7fbaed7fa640 1 -- 172.21.15.28:0/2493355072 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56125ea0a530 con 0x56125eaaeed0 2024-04-21T08:50:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.933+0000 7fbaed7fa640 10 client.15222 handle_command_reply: tid=0 2024-04-21T08:50:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.934+0000 7fbb0ec13740 1 client.15222 shutdown 2024-04-21T08:50:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.934+0000 7fbb0ec13740 1 -- 172.21.15.28:0/2493355072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea992d0 msgr2=0x56125eaa9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.934+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/2493355072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea992d0 0x56125eaa9410 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fbb040699c0 tx=0x7fbb00015040 comp rx=0 tx=0).stop 2024-04-21T08:50:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.934+0000 7fbb0ec13740 1 -- 172.21.15.28:0/2493355072 shutdown_connections 2024-04-21T08:50:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.934+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/2493355072 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56125eaaeed0 0x56125eab0340 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.934+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/2493355072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea992d0 0x56125eaa9410 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.934+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/2493355072 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea98830 0x56125eaa23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.934+0000 7fbaed7fa640 0 client.15222 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:50:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.934+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/2493355072 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea97e40 0x56125eaa1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.934+0000 7fbb0ec13740 1 -- 172.21.15.28:0/2493355072 >> 172.21.15.28:0/2493355072 conn(0x56125ea93f40 msgr2=0x56125ea91a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.934+0000 7fbb0ec13740 1 -- 172.21.15.28:0/2493355072 shutdown_connections 2024-04-21T08:50:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.934+0000 7fbb0ec13740 1 -- 172.21.15.28:0/2493355072 wait complete. 2024-04-21T08:50:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.934+0000 7fbb0ec13740 20 client.15222 trim_cache size 0 max 16384 2024-04-21T08:50:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.952+0000 7fbb0ec13740 1 Processor -- start 2024-04-21T08:50:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.952+0000 7fbb0ec13740 1 -- start start 2024-04-21T08:50:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.952+0000 7fbb0ec13740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea97e40 0x56125ea92400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.952+0000 7fbb0ec13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea98830 0x56125eaa9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.952+0000 7fbb0ec13740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea992d0 0x56125ea9fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.952+0000 7fbb0ec13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56125eaae680 con 0x56125ea98830 2024-04-21T08:50:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.952+0000 7fbb0ec13740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56125eaaed50 con 0x56125ea97e40 2024-04-21T08:50:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.952+0000 7fbb0ec13740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56125eaaf420 con 0x56125ea992d0 2024-04-21T08:50:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.952+0000 7fbb09dc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea98830 0x56125eaa9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.953+0000 7fbb09dc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea98830 0x56125eaa9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59282/0 (socket says 172.21.15.28:59282) 2024-04-21T08:50:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.953+0000 7fbb09dc1640 1 -- 172.21.15.28:0/4276817493 learned_addr learned my addr 172.21.15.28:0/4276817493 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.953+0000 7fbb095c0640 1 --2- 172.21.15.28:0/4276817493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea97e40 0x56125ea92400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.953+0000 7fbb09dc1640 1 -- 172.21.15.28:0/4276817493 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea992d0 msgr2=0x56125ea9fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.953+0000 7fbb09dc1640 1 --2- 172.21.15.28:0/4276817493 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea992d0 0x56125ea9fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.953+0000 7fbb09dc1640 1 -- 172.21.15.28:0/4276817493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea97e40 msgr2=0x56125ea92400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.953+0000 7fbb09dc1640 1 --2- 172.21.15.28:0/4276817493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea97e40 0x56125ea92400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.953+0000 7fbb09dc1640 1 -- 172.21.15.28:0/4276817493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56125eaa0f70 con 0x56125ea98830 2024-04-21T08:50:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.953+0000 7fbb09dc1640 1 --2- 172.21.15.28:0/4276817493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea98830 0x56125eaa9330 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4019000 tx=0x7fbaf4005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5cb1169769fd341 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.953+0000 7fbaef7fe640 1 -- 172.21.15.28:0/4276817493 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4015040 con 0x56125ea98830 2024-04-21T08:50:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.953+0000 7fbaef7fe640 1 -- 172.21.15.28:0/4276817493 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaf4005500 con 0x56125ea98830 2024-04-21T08:50:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.953+0000 7fbaef7fe640 1 -- 172.21.15.28:0/4276817493 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4004ae0 con 0x56125ea98830 2024-04-21T08:50:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 -- 172.21.15.28:0/4276817493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea98830 msgr2=0x56125eaa9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/4276817493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea98830 0x56125eaa9330 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4019000 tx=0x7fbaf4005c90 comp rx=0 tx=0).stop 2024-04-21T08:50:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 -- 172.21.15.28:0/4276817493 shutdown_connections 2024-04-21T08:50:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/4276817493 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea992d0 0x56125ea9fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/4276817493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea98830 0x56125eaa9330 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/4276817493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea97e40 0x56125ea92400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 -- 172.21.15.28:0/4276817493 >> 172.21.15.28:0/4276817493 conn(0x56125ea93f40 msgr2=0x56125eaadcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 -- 172.21.15.28:0/4276817493 shutdown_connections 2024-04-21T08:50:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 -- 172.21.15.28:0/4276817493 wait complete. 2024-04-21T08:50:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 -- start start 2024-04-21T08:50:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 Processor -- start 2024-04-21T08:50:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea97e40 0x56125ea9b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea98830 0x56125ea9bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea992d0 0x56125eaa79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56125eab01c0 con 0x56125ea992d0 2024-04-21T08:50:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56125eaafaf0 con 0x56125ea97e40 2024-04-21T08:50:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.954+0000 7fbb0ec13740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56125eab0890 con 0x56125ea98830 2024-04-21T08:50:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb0ec13740 10 client.0 _ll_register_callbacks cb 0x7fbb0000ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb095c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea992d0 0x56125eaa79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb095c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea992d0 0x56125eaa79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59284/0 (socket says 172.21.15.28:59284) 2024-04-21T08:50:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb095c0640 1 -- 172.21.15.28:0/3174801198 learned_addr learned my addr 172.21.15.28:0/3174801198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb095c0640 1 --2- 172.21.15.28:0/3174801198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea97e40 0x56125ea9b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb09dc1640 1 --2- 172.21.15.28:0/3174801198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea98830 0x56125ea9bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb095c0640 1 -- 172.21.15.28:0/3174801198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea98830 msgr2=0x56125ea9bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb095c0640 1 --2- 172.21.15.28:0/3174801198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea98830 0x56125ea9bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb095c0640 1 -- 172.21.15.28:0/3174801198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea97e40 msgr2=0x56125ea9b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb095c0640 1 --2- 172.21.15.28:0/3174801198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea97e40 0x56125ea9b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb095c0640 1 -- 172.21.15.28:0/3174801198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56125eaa7fe0 con 0x56125ea992d0 2024-04-21T08:50:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb09dc1640 1 --2- 172.21.15.28:0/3174801198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea98830 0x56125ea9bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb095c0640 1 --2- 172.21.15.28:0/3174801198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea992d0 0x56125eaa79a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbb00013020 tx=0x7fbb0000be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351c9454fa2b1959 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbaed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:50:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbaed7fa640 1 -- 172.21.15.28:0/3174801198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0000baf0 con 0x56125ea992d0 2024-04-21T08:50:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbaed7fa640 1 -- 172.21.15.28:0/3174801198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9187+0+0 (secure 0 0 0) 0x7fbb00019070 con 0x56125ea992d0 2024-04-21T08:50:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb0ec13740 1 -- 172.21.15.28:0/3174801198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbaf4005500 con 0x56125ea992d0 2024-04-21T08:50:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbaed7fa640 1 -- 172.21.15.28:0/3174801198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbb0003a050 con 0x56125ea992d0 2024-04-21T08:50:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb0ec13740 10 client.15228 fetch_fsmap learned FSMap version 24 2024-04-21T08:50:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.955+0000 7fbb0ec13740 1 -- 172.21.15.28:0/3174801198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56125eaa9300 con 0x56125ea992d0 2024-04-21T08:50:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.956+0000 7fbaed7fa640 1 -- 172.21.15.28:0/3174801198 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbaf4005500 con 0x56125ea992d0 2024-04-21T08:50:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.956+0000 7fbaed7fa640 1 -- 172.21.15.28:0/3174801198 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fbb00011ad0 con 0x56125ea992d0 2024-04-21T08:50:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.956+0000 7fbb0ec13740 10 client.15228 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:50:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.956+0000 7fbb0ec13740 10 client.15228 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-21T08:50:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.956+0000 7fbb0ec13740 20 client.15228 populate_metadata read hostname 'smithi028' 2024-04-21T08:50:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.956+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/3174801198 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56125eb194f0 0x56125eb18200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.956+0000 7fbb0ec13740 4 client.15228 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:50:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.956+0000 7fbb0ec13740 1 -- 172.21.15.28:0/3174801198 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "status"}) -- 0x56125ea043a0 con 0x56125eb194f0 2024-04-21T08:50:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.956+0000 7fbb09dc1640 1 --2- 172.21.15.28:0/3174801198 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56125eb194f0 0x56125eb18200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbb09dc1640 1 --2- 172.21.15.28:0/3174801198 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56125eb194f0 0x56125eb18200 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=18ea5c67210807a1 server_cookie=60e4ba868d926a03 in_seq=0 out_seq=0 2024-04-21T08:50:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbaed7fa640 10 client.15228 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:50:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbaed7fa640 1 -- 172.21.15.28:0/3174801198 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56125ea043a0 con 0x56125eb194f0 2024-04-21T08:50:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbaed7fa640 10 client.15228 handle_command_reply: tid=0 2024-04-21T08:50:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbb0ec13740 1 client.15228 shutdown 2024-04-21T08:50:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbb0ec13740 1 -- 172.21.15.28:0/3174801198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea992d0 msgr2=0x56125eaa79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/3174801198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea992d0 0x56125eaa79a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbb00013020 tx=0x7fbb0000be40 comp rx=0 tx=0).stop 2024-04-21T08:50:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbb0ec13740 1 -- 172.21.15.28:0/3174801198 shutdown_connections 2024-04-21T08:50:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/3174801198 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56125eb194f0 0x56125eb18200 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/3174801198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56125ea992d0 0x56125eaa79a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/3174801198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56125ea98830 0x56125ea9bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbb0ec13740 1 --2- 172.21.15.28:0/3174801198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56125ea97e40 0x56125ea9b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbaed7fa640 0 client.15228 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:50:36.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbb0ec13740 1 -- 172.21.15.28:0/3174801198 >> 172.21.15.28:0/3174801198 conn(0x56125ea93f40 msgr2=0x56125ea9c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:36.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbb0ec13740 1 -- 172.21.15.28:0/3174801198 shutdown_connections 2024-04-21T08:50:36.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbb0ec13740 1 -- 172.21.15.28:0/3174801198 wait complete. 2024-04-21T08:50:36.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.957+0000 7fbb0ec13740 20 client.15228 trim_cache size 0 max 16384 2024-04-21T08:50:36.963 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:50:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "b51a76c8-ffba-11ee-bc93-c7b262605968", 2024-04-21T08:50:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-04-21T08:50:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24463, 2024-04-21T08:50:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-21T08:50:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-21T08:50:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-21T08:50:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 85.122298857999994, 2024-04-21T08:50:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-04-21T08:50:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 92, 2024-04-21T08:50:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:50:36.964 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 129.84501435300001 2024-04-21T08:50:36.964 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:50:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.958+0000 7fbb0c025640 1 -- 172.21.15.28:0/3086273425 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbaf00a3bb0 msgr2=0x7fbaf00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.958+0000 7fbb0c025640 1 --2- 172.21.15.28:0/3086273425 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbaf00a3bb0 0x7fbaf00a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0be860 tx=0x7fbafc0e9c70 comp rx=0 tx=0).stop 2024-04-21T08:50:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.958+0000 7fbb0c025640 1 -- 172.21.15.28:0/3086273425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb041029a0 msgr2=0x7fbb0419f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.958+0000 7fbb0c025640 1 --2- 172.21.15.28:0/3086273425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb041029a0 0x7fbb0419f4b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbafc02c700 tx=0x7fbafc034760 comp rx=0 tx=0).stop 2024-04-21T08:50:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.958+0000 7fbb0c025640 1 -- 172.21.15.28:0/3086273425 shutdown_connections 2024-04-21T08:50:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.958+0000 7fbb0c025640 1 --2- 172.21.15.28:0/3086273425 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbaf00a3bb0 0x7fbaf00a6070 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.958+0000 7fbb0c025640 1 --2- 172.21.15.28:0/3086273425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb041029a0 0x7fbb0419f4b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.958+0000 7fbb0c025640 1 --2- 172.21.15.28:0/3086273425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb04101fe0 0x7fbb0419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.958+0000 7fbb0c025640 1 --2- 172.21.15.28:0/3086273425 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb04100e90 0x7fbb04069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.958+0000 7fbb0c025640 1 -- 172.21.15.28:0/3086273425 >> 172.21.15.28:0/3086273425 conn(0x7fbb040fc400 msgr2=0x7fbb04105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.958+0000 7fbb0c025640 1 -- 172.21.15.28:0/3086273425 shutdown_connections 2024-04-21T08:50:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:36.958+0000 7fbb0c025640 1 -- 172.21.15.28:0/3086273425 wait complete. 2024-04-21T08:50:36.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-21T08:50:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.071+0000 7efdb4c4e640 1 -- 172.21.15.28:0/975361445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb01021d0 msgr2=0x7efdb0108a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.071+0000 7efdb4c4e640 1 --2- 172.21.15.28:0/975361445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb01021d0 0x7efdb0108a60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7efda8009ca0 tx=0x7efda802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.071+0000 7efdb4c4e640 1 -- 172.21.15.28:0/975361445 shutdown_connections 2024-04-21T08:50:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.071+0000 7efdb4c4e640 1 --2- 172.21.15.28:0/975361445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb01021d0 0x7efdb0108a60 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.071+0000 7efdb4c4e640 1 --2- 172.21.15.28:0/975361445 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efdb0101810 0x7efdb0101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.071+0000 7efdb4c4e640 1 --2- 172.21.15.28:0/975361445 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efdb0100620 0x7efdb0100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.071+0000 7efdb4c4e640 1 -- 172.21.15.28:0/975361445 >> 172.21.15.28:0/975361445 conn(0x7efdb00fbb90 msgr2=0x7efdb00fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.071+0000 7efdb4c4e640 1 -- 172.21.15.28:0/975361445 shutdown_connections 2024-04-21T08:50:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.071+0000 7efdb4c4e640 1 -- 172.21.15.28:0/975361445 wait complete. 2024-04-21T08:50:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.072+0000 7efdb4c4e640 1 Processor -- start 2024-04-21T08:50:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.072+0000 7efdb4c4e640 1 -- start start 2024-04-21T08:50:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.072+0000 7efdb4c4e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efdb0100620 0x7efdb0198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.072+0000 7efdb4c4e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efdb0101810 0x7efdb0198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.072+0000 7efdb4c4e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb01021d0 0x7efdb019f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.072+0000 7efdb4c4e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efdb010db20 con 0x7efdb01021d0 2024-04-21T08:50:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.072+0000 7efdb4c4e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efdb010d9a0 con 0x7efdb0101810 2024-04-21T08:50:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.072+0000 7efdb4c4e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7efdb010dca0 con 0x7efdb0100620 2024-04-21T08:50:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.072+0000 7efdaed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb01021d0 0x7efdb019f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.072+0000 7efdaed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb01021d0 0x7efdb019f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59308/0 (socket says 172.21.15.28:59308) 2024-04-21T08:50:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.072+0000 7efdaed9d640 1 -- 172.21.15.28:0/2302639582 learned_addr learned my addr 172.21.15.28:0/2302639582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.072+0000 7efdadd9b640 1 --2- 172.21.15.28:0/2302639582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efdb0101810 0x7efdb0198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.073+0000 7efdae59c640 1 --2- 172.21.15.28:0/2302639582 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efdb0100620 0x7efdb0198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.073+0000 7efdaed9d640 1 -- 172.21.15.28:0/2302639582 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efdb0100620 msgr2=0x7efdb0198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.073+0000 7efdaed9d640 1 --2- 172.21.15.28:0/2302639582 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efdb0100620 0x7efdb0198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.073+0000 7efdaed9d640 1 -- 172.21.15.28:0/2302639582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efdb0101810 msgr2=0x7efdb0198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.073+0000 7efdaed9d640 1 --2- 172.21.15.28:0/2302639582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efdb0101810 0x7efdb0198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.073+0000 7efdaed9d640 1 -- 172.21.15.28:0/2302639582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb019fd40 con 0x7efdb01021d0 2024-04-21T08:50:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.073+0000 7efdae59c640 1 --2- 172.21.15.28:0/2302639582 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efdb0100620 0x7efdb0198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.073+0000 7efdaed9d640 1 --2- 172.21.15.28:0/2302639582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb01021d0 0x7efdb019f6d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efda802c700 tx=0x7efda8034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.073+0000 7efd9f7fe640 1 -- 172.21.15.28:0/2302639582 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda8039040 con 0x7efdb01021d0 2024-04-21T08:50:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.073+0000 7efd9f7fe640 1 -- 172.21.15.28:0/2302639582 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efda8044070 con 0x7efdb01021d0 2024-04-21T08:50:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.073+0000 7efd9f7fe640 1 -- 172.21.15.28:0/2302639582 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda80042a0 con 0x7efdb01021d0 2024-04-21T08:50:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.073+0000 7efdb4c4e640 1 -- 172.21.15.28:0/2302639582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdb019ffd0 con 0x7efdb01021d0 2024-04-21T08:50:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.073+0000 7efdb4c4e640 1 -- 172.21.15.28:0/2302639582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdb01a0430 con 0x7efdb01021d0 2024-04-21T08:50:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.075+0000 7efd9f7fe640 1 -- 172.21.15.28:0/2302639582 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7efda8003970 con 0x7efdb01021d0 2024-04-21T08:50:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.075+0000 7efdb7842740 1 Processor -- start 2024-04-21T08:50:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdb7842740 1 -- start start 2024-04-21T08:50:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdb7842740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb3219250 0x558cb3219670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdb7842740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219c40 0x558cb321a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdb7842740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb321a6e0 0x558cb32212c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdb7842740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558cb3179b10 con 0x558cb3219c40 2024-04-21T08:50:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdb7842740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558cb3117240 con 0x558cb321a6e0 2024-04-21T08:50:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdb7842740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558cb3175ad0 con 0x558cb3219250 2024-04-21T08:50:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdae59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219c40 0x558cb321a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efd9f7fe640 1 --2- 172.21.15.28:0/2302639582 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efd980a3bb0 0x7efd980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdae59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219c40 0x558cb321a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59320/0 (socket says 172.21.15.28:59320) 2024-04-21T08:50:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdae59c640 1 -- 172.21.15.28:0/1154469205 learned_addr learned my addr 172.21.15.28:0/1154469205 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdadd9b640 1 --2- 172.21.15.28:0/1154469205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb321a6e0 0x558cb32212c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdaed9d640 1 --2- 172.21.15.28:0/2302639582 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efd980a3bb0 0x7efd980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdadd9b640 1 --2- 172.21.15.28:0/1154469205 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb3219250 0x558cb3219670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efd9f7fe640 1 -- 172.21.15.28:0/2302639582 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9187+0+0 (secure 0 0 0) 0x7efda80f2c50 con 0x7efdb01021d0 2024-04-21T08:50:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdae59c640 1 -- 172.21.15.28:0/1154469205 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb3219250 msgr2=0x558cb3219670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdae59c640 1 --2- 172.21.15.28:0/1154469205 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb3219250 0x558cb3219670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdae59c640 1 -- 172.21.15.28:0/1154469205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb321a6e0 msgr2=0x558cb32212c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdae59c640 1 --2- 172.21.15.28:0/1154469205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb321a6e0 0x558cb32212c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdae59c640 1 -- 172.21.15.28:0/1154469205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cb2e8c2d0 con 0x558cb3219c40 2024-04-21T08:50:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.076+0000 7efdadd9b640 1 --2- 172.21.15.28:0/1154469205 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb3219250 0x558cb3219670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.077+0000 7efdae59c640 1 --2- 172.21.15.28:0/1154469205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219c40 0x558cb321a0c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efda000bbf0 tx=0x7efda000d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5508b22808f066 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.077+0000 7efdaed9d640 1 --2- 172.21.15.28:0/2302639582 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efd980a3bb0 0x7efd980a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efda80bb4e0 tx=0x7efda8041040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.077+0000 7efd9cff9640 1 -- 172.21.15.28:0/1154469205 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda0004220 con 0x558cb3219c40 2024-04-21T08:50:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.077+0000 7efd9cff9640 1 -- 172.21.15.28:0/1154469205 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efda00043c0 con 0x558cb3219c40 2024-04-21T08:50:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.077+0000 7efd9cff9640 1 -- 172.21.15.28:0/1154469205 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda000f790 con 0x558cb3219c40 2024-04-21T08:50:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.077+0000 7efdb7842740 1 -- 172.21.15.28:0/1154469205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219c40 msgr2=0x558cb321a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.077+0000 7efdb7842740 1 --2- 172.21.15.28:0/1154469205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219c40 0x558cb321a0c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efda000bbf0 tx=0x7efda000d710 comp rx=0 tx=0).stop 2024-04-21T08:50:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 1 -- 172.21.15.28:0/1154469205 shutdown_connections 2024-04-21T08:50:37.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 1 --2- 172.21.15.28:0/1154469205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb321a6e0 0x558cb32212c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 1 --2- 172.21.15.28:0/1154469205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219c40 0x558cb321a0c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 1 --2- 172.21.15.28:0/1154469205 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb3219250 0x558cb3219670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 1 -- 172.21.15.28:0/1154469205 >> 172.21.15.28:0/1154469205 conn(0x558cb3210f40 msgr2=0x558cb320f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 1 -- 172.21.15.28:0/1154469205 shutdown_connections 2024-04-21T08:50:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 1 -- 172.21.15.28:0/1154469205 wait complete. 2024-04-21T08:50:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 1 -- start start 2024-04-21T08:50:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 1 Processor -- start 2024-04-21T08:50:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219250 0x558cb32232c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb3219c40 0x558cb3223800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb321a6e0 0x558cb322a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558cb32218c0 con 0x558cb3219250 2024-04-21T08:50:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558cb3102190 con 0x558cb3219c40 2024-04-21T08:50:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558cb3221a40 con 0x558cb321a6e0 2024-04-21T08:50:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdb7842740 10 client.0 _ll_register_callbacks cb 0x558cb30fd8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdadd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219250 0x558cb32232c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdadd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219250 0x558cb32232c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59322/0 (socket says 172.21.15.28:59322) 2024-04-21T08:50:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.078+0000 7efdadd9b640 1 -- 172.21.15.28:0/3927038569 learned_addr learned my addr 172.21.15.28:0/3927038569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efdae59c640 1 --2- 172.21.15.28:0/3927038569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb3219c40 0x558cb3223800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efdadd9b640 1 --2- 172.21.15.28:0/3927038569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb321a6e0 0x558cb322a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efdadd9b640 1 -- 172.21.15.28:0/3927038569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb321a6e0 msgr2=0x558cb322a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efdadd9b640 1 --2- 172.21.15.28:0/3927038569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb321a6e0 0x558cb322a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efdadd9b640 1 -- 172.21.15.28:0/3927038569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb3219c40 msgr2=0x558cb3223800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efdadd9b640 1 --2- 172.21.15.28:0/3927038569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb3219c40 0x558cb3223800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efdadd9b640 1 -- 172.21.15.28:0/3927038569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cb2f566b0 con 0x558cb3219250 2024-04-21T08:50:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efdadd9b640 1 --2- 172.21.15.28:0/3927038569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219250 0x558cb32232c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efda4014f00 tx=0x7efda40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3c495a42f951ef server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efd95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:50:37.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efd95ffb640 1 -- 172.21.15.28:0/3927038569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda4011e90 con 0x558cb3219250 2024-04-21T08:50:37.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efdb7842740 1 -- 172.21.15.28:0/3927038569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efda00043c0 con 0x558cb3219250 2024-04-21T08:50:37.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efd95ffb640 1 -- 172.21.15.28:0/3927038569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9187+0+0 (secure 0 0 0) 0x7efda40047a0 con 0x558cb3219250 2024-04-21T08:50:37.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efd95ffb640 1 -- 172.21.15.28:0/3927038569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efda4028050 con 0x558cb3219250 2024-04-21T08:50:37.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efdb7842740 10 client.15240 fetch_fsmap learned FSMap version 24 2024-04-21T08:50:37.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efdb7842740 1 -- 172.21.15.28:0/3927038569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558cb322dcd0 con 0x558cb3219250 2024-04-21T08:50:37.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.079+0000 7efd95ffb640 1 -- 172.21.15.28:0/3927038569 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efda00043c0 con 0x558cb3219250 2024-04-21T08:50:37.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.080+0000 7efd95ffb640 1 -- 172.21.15.28:0/3927038569 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7efda40103d0 con 0x558cb3219250 2024-04-21T08:50:37.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.080+0000 7efdb7842740 10 client.15240 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:50:37.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.080+0000 7efdb7842740 10 client.15240 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T08:50:37.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.080+0000 7efdb7842740 20 client.15240 populate_metadata read hostname 'smithi028' 2024-04-21T08:50:37.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.080+0000 7efdb7842740 1 --2- 172.21.15.28:0/3927038569 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558cb32302e0 0x558cb3231750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.080+0000 7efdb7842740 4 client.15240 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:37.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.080+0000 7efdb7842740 1 -- 172.21.15.28:0/3927038569 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cb3187530 con 0x558cb32302e0 2024-04-21T08:50:37.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.080+0000 7efdae59c640 1 --2- 172.21.15.28:0/3927038569 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558cb32302e0 0x558cb3231750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.080+0000 7efdae59c640 1 --2- 172.21.15.28:0/3927038569 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558cb32302e0 0x558cb3231750 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ebde42a8ea5f83ee server_cookie=40e615e07fb57d90 in_seq=0 out_seq=0 2024-04-21T08:50:37.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.080+0000 7efd95ffb640 10 client.15240 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:50:37.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.081+0000 7efd95ffb640 1 -- 172.21.15.28:0/3927038569 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558cb3187530 con 0x558cb32302e0 2024-04-21T08:50:37.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.081+0000 7efd95ffb640 10 client.15240 handle_command_reply: tid=0 2024-04-21T08:50:37.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.081+0000 7efdb7842740 1 client.15240 shutdown 2024-04-21T08:50:37.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.082+0000 7efdb7842740 1 -- 172.21.15.28:0/3927038569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219250 msgr2=0x558cb32232c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.082+0000 7efdb7842740 1 --2- 172.21.15.28:0/3927038569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219250 0x558cb32232c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efda4014f00 tx=0x7efda40046c0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.082+0000 7efdb7842740 1 -- 172.21.15.28:0/3927038569 shutdown_connections 2024-04-21T08:50:37.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.082+0000 7efdb7842740 1 --2- 172.21.15.28:0/3927038569 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558cb32302e0 0x558cb3231750 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.082+0000 7efdb7842740 1 --2- 172.21.15.28:0/3927038569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb321a6e0 0x558cb322a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.082+0000 7efdb7842740 1 --2- 172.21.15.28:0/3927038569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb3219c40 0x558cb3223800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.082+0000 7efd95ffb640 0 client.15240 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:50:37.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.082+0000 7efdb7842740 1 --2- 172.21.15.28:0/3927038569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219250 0x558cb32232c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.082+0000 7efdb7842740 1 -- 172.21.15.28:0/3927038569 >> 172.21.15.28:0/3927038569 conn(0x558cb3210f40 msgr2=0x558cb320ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:37.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.082+0000 7efdb7842740 1 -- 172.21.15.28:0/3927038569 shutdown_connections 2024-04-21T08:50:37.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.082+0000 7efdb7842740 1 -- 172.21.15.28:0/3927038569 wait complete. 2024-04-21T08:50:37.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.082+0000 7efdb7842740 20 client.15240 trim_cache size 0 max 16384 2024-04-21T08:50:37.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.100+0000 7efdb7842740 1 Processor -- start 2024-04-21T08:50:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.100+0000 7efdb7842740 1 -- start start 2024-04-21T08:50:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.100+0000 7efdb7842740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219250 0x558cb321dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.100+0000 7efdb7842740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb3219c40 0x558cb322a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.100+0000 7efdb7842740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb321a6e0 0x558cb321e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.100+0000 7efdb7842740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558cb322fa90 con 0x558cb3219250 2024-04-21T08:50:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.100+0000 7efdb7842740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558cb3230160 con 0x558cb321a6e0 2024-04-21T08:50:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.100+0000 7efdb7842740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558cb3230830 con 0x558cb3219c40 2024-04-21T08:50:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.100+0000 7efdadd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219250 0x558cb321dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.100+0000 7efdadd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219250 0x558cb321dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59336/0 (socket says 172.21.15.28:59336) 2024-04-21T08:50:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.100+0000 7efdadd9b640 1 -- 172.21.15.28:0/1906530880 learned_addr learned my addr 172.21.15.28:0/1906530880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.101+0000 7efdadd9b640 1 -- 172.21.15.28:0/1906530880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb3219c40 msgr2=0x558cb322a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.101+0000 7efdae59c640 1 --2- 172.21.15.28:0/1906530880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb3219c40 0x558cb322a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.101+0000 7efdadd9b640 1 --2- 172.21.15.28:0/1906530880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb3219c40 0x558cb322a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.101+0000 7efdadd9b640 1 -- 172.21.15.28:0/1906530880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb321a6e0 msgr2=0x558cb321e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.101+0000 7efdadd9b640 1 --2- 172.21.15.28:0/1906530880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb321a6e0 0x558cb321e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.101+0000 7efdadd9b640 1 -- 172.21.15.28:0/1906530880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cb3210630 con 0x558cb3219250 2024-04-21T08:50:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.101+0000 7efdadd9b640 1 --2- 172.21.15.28:0/1906530880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219250 0x558cb321dbf0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efda4010990 tx=0x7efda4014990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b6f21666c69493 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.101+0000 7efd97fff640 1 -- 172.21.15.28:0/1906530880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda4011e90 con 0x558cb3219250 2024-04-21T08:50:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.101+0000 7efd97fff640 1 -- 172.21.15.28:0/1906530880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efda4014300 con 0x558cb3219250 2024-04-21T08:50:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.101+0000 7efd97fff640 1 -- 172.21.15.28:0/1906530880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda4005450 con 0x558cb3219250 2024-04-21T08:50:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 -- 172.21.15.28:0/1906530880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219250 msgr2=0x558cb321dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 --2- 172.21.15.28:0/1906530880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219250 0x558cb321dbf0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efda4010990 tx=0x7efda4014990 comp rx=0 tx=0).stop 2024-04-21T08:50:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 -- 172.21.15.28:0/1906530880 shutdown_connections 2024-04-21T08:50:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 --2- 172.21.15.28:0/1906530880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb321a6e0 0x558cb321e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 --2- 172.21.15.28:0/1906530880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb3219c40 0x558cb322a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 --2- 172.21.15.28:0/1906530880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb3219250 0x558cb321dbf0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 -- 172.21.15.28:0/1906530880 >> 172.21.15.28:0/1906530880 conn(0x558cb3210f40 msgr2=0x558cb3221260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:37.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 -- 172.21.15.28:0/1906530880 shutdown_connections 2024-04-21T08:50:37.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 -- 172.21.15.28:0/1906530880 wait complete. 2024-04-21T08:50:37.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 -- start start 2024-04-21T08:50:37.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 Processor -- start 2024-04-21T08:50:37.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb3219250 0x558cb32261e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb3219c40 0x558cb3226720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb321a6e0 0x558cb3229740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558cb32315d0 con 0x558cb321a6e0 2024-04-21T08:50:37.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558cb3230f00 con 0x558cb3219250 2024-04-21T08:50:37.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558cb3231ca0 con 0x558cb3219c40 2024-04-21T08:50:37.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdb7842740 10 client.0 _ll_register_callbacks cb 0x7efda400ce50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:37.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdadd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb321a6e0 0x558cb3229740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdadd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb321a6e0 0x558cb3229740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59348/0 (socket says 172.21.15.28:59348) 2024-04-21T08:50:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.102+0000 7efdadd9b640 1 -- 172.21.15.28:0/1123349558 learned_addr learned my addr 172.21.15.28:0/1123349558 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdadd9b640 1 --2- 172.21.15.28:0/1123349558 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb3219250 0x558cb32261e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdadd9b640 1 -- 172.21.15.28:0/1123349558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb3219c40 msgr2=0x558cb3226720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdadd9b640 1 --2- 172.21.15.28:0/1123349558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb3219c40 0x558cb3226720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdadd9b640 1 -- 172.21.15.28:0/1123349558 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb3219250 msgr2=0x558cb32261e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdadd9b640 1 --2- 172.21.15.28:0/1123349558 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb3219250 0x558cb32261e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdadd9b640 1 -- 172.21.15.28:0/1123349558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cb3229d30 con 0x558cb321a6e0 2024-04-21T08:50:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdadd9b640 1 --2- 172.21.15.28:0/1123349558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb321a6e0 0x558cb3229740 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efda40108c0 tx=0x7efda4011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=745c956e033c3b66 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efd95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:50:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efd95ffb640 1 -- 172.21.15.28:0/1123349558 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda403f060 con 0x558cb321a6e0 2024-04-21T08:50:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efd95ffb640 1 -- 172.21.15.28:0/1123349558 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9187+0+0 (secure 0 0 0) 0x7efda4005230 con 0x558cb321a6e0 2024-04-21T08:50:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdb7842740 1 -- 172.21.15.28:0/1123349558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efda4014300 con 0x558cb321a6e0 2024-04-21T08:50:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efd95ffb640 1 -- 172.21.15.28:0/1123349558 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efda4005470 con 0x558cb321a6e0 2024-04-21T08:50:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdb7842740 10 client.15246 fetch_fsmap learned FSMap version 24 2024-04-21T08:50:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdb7842740 1 -- 172.21.15.28:0/1123349558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x558cb3296580 con 0x558cb321a6e0 2024-04-21T08:50:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efd95ffb640 1 -- 172.21.15.28:0/1123349558 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efda4014300 con 0x558cb321a6e0 2024-04-21T08:50:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efd95ffb640 1 -- 172.21.15.28:0/1123349558 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7efda4004ba0 con 0x558cb321a6e0 2024-04-21T08:50:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdb7842740 10 client.15246 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:50:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdb7842740 10 client.15246 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T08:50:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdb7842740 20 client.15246 populate_metadata read hostname 'smithi028' 2024-04-21T08:50:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdb7842740 1 --2- 172.21.15.28:0/1123349558 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558cb329b0b0 0x558cb32974f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdb7842740 4 client.15246 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:50:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.103+0000 7efdb7842740 1 -- 172.21.15.28:0/1123349558 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "status"}) -- 0x558cb31813a0 con 0x558cb329b0b0 2024-04-21T08:50:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.104+0000 7efdae59c640 1 --2- 172.21.15.28:0/1123349558 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558cb329b0b0 0x558cb32974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.104+0000 7efd95ffb640 10 client.15246 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:50:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.104+0000 7efdae59c640 1 --2- 172.21.15.28:0/1123349558 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558cb329b0b0 0x558cb32974f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d12a6c2e2ba23656 server_cookie=6b69f741c629b3db in_seq=0 out_seq=0 2024-04-21T08:50:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.104+0000 7efd95ffb640 1 -- 172.21.15.28:0/1123349558 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x558cb31813a0 con 0x558cb329b0b0 2024-04-21T08:50:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.104+0000 7efd95ffb640 10 client.15246 handle_command_reply: tid=0 2024-04-21T08:50:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.104+0000 7efdb7842740 1 client.15246 shutdown 2024-04-21T08:50:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.104+0000 7efdb7842740 1 -- 172.21.15.28:0/1123349558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb321a6e0 msgr2=0x558cb3229740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.104+0000 7efdb7842740 1 --2- 172.21.15.28:0/1123349558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb321a6e0 0x558cb3229740 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efda40108c0 tx=0x7efda4011d60 comp rx=0 tx=0).stop 2024-04-21T08:50:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.104+0000 7efdb7842740 1 -- 172.21.15.28:0/1123349558 shutdown_connections 2024-04-21T08:50:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.104+0000 7efdb7842740 1 --2- 172.21.15.28:0/1123349558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cb321a6e0 0x558cb3229740 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.104+0000 7efdb7842740 1 --2- 172.21.15.28:0/1123349558 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558cb329b0b0 0x558cb32974f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.104+0000 7efdb7842740 1 --2- 172.21.15.28:0/1123349558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558cb3219c40 0x558cb3226720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.104+0000 7efdb7842740 1 --2- 172.21.15.28:0/1123349558 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cb3219250 0x558cb32261e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.104+0000 7efd95ffb640 0 client.15246 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:50:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.104+0000 7efdb7842740 1 -- 172.21.15.28:0/1123349558 >> 172.21.15.28:0/1123349558 conn(0x558cb3210f40 msgr2=0x558cb3222ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.105+0000 7efdb7842740 1 -- 172.21.15.28:0/1123349558 shutdown_connections 2024-04-21T08:50:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.105+0000 7efdb7842740 1 -- 172.21.15.28:0/1123349558 wait complete. 2024-04-21T08:50:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.105+0000 7efdb7842740 20 client.15246 trim_cache size 0 max 16384 2024-04-21T08:50:37.111 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:50:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "b51a76c8-ffba-11ee-bc93-c7b262605968", 2024-04-21T08:50:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-04-21T08:50:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24487, 2024-04-21T08:50:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-21T08:50:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-21T08:50:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-21T08:50:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 86.281661314000004, 2024-04-21T08:50:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-04-21T08:50:37.111 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 92, 2024-04-21T08:50:37.112 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:50:37.112 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 124.169810093 2024-04-21T08:50:37.112 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:50:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.105+0000 7efdb4c4e640 1 -- 172.21.15.28:0/2302639582 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efd980a3bb0 msgr2=0x7efd980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.105+0000 7efdb4c4e640 1 --2- 172.21.15.28:0/2302639582 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efd980a3bb0 0x7efd980a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efda80bb4e0 tx=0x7efda8041040 comp rx=0 tx=0).stop 2024-04-21T08:50:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.105+0000 7efdb4c4e640 1 -- 172.21.15.28:0/2302639582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb01021d0 msgr2=0x7efdb019f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.105+0000 7efdb4c4e640 1 --2- 172.21.15.28:0/2302639582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb01021d0 0x7efdb019f6d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efda802c700 tx=0x7efda8034760 comp rx=0 tx=0).stop 2024-04-21T08:50:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.105+0000 7efdb4c4e640 1 -- 172.21.15.28:0/2302639582 shutdown_connections 2024-04-21T08:50:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.105+0000 7efdb4c4e640 1 --2- 172.21.15.28:0/2302639582 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7efd980a3bb0 0x7efd980a6070 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.105+0000 7efdb4c4e640 1 --2- 172.21.15.28:0/2302639582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb01021d0 0x7efdb019f6d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.105+0000 7efdb4c4e640 1 --2- 172.21.15.28:0/2302639582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efdb0101810 0x7efdb0198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.105+0000 7efdb4c4e640 1 --2- 172.21.15.28:0/2302639582 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efdb0100620 0x7efdb0198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.105+0000 7efdb4c4e640 1 -- 172.21.15.28:0/2302639582 >> 172.21.15.28:0/2302639582 conn(0x7efdb00fbb90 msgr2=0x7efdb0104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.105+0000 7efdb4c4e640 1 -- 172.21.15.28:0/2302639582 shutdown_connections 2024-04-21T08:50:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.105+0000 7efdb4c4e640 1 -- 172.21.15.28:0/2302639582 wait complete. 2024-04-21T08:50:37.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:50:37.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.218+0000 7f604591e640 1 -- 172.21.15.28:0/4210421196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6040100e60 msgr2=0x7f6040101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.218+0000 7f604591e640 1 --2- 172.21.15.28:0/4210421196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6040100e60 0x7f6040101280 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6034009900 tx=0x7f603402bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.219+0000 7f604591e640 1 -- 172.21.15.28:0/4210421196 shutdown_connections 2024-04-21T08:50:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.219+0000 7f604591e640 1 --2- 172.21.15.28:0/4210421196 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6040102a00 0x7f6040109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.219+0000 7f604591e640 1 --2- 172.21.15.28:0/4210421196 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6040102040 0x7f60401024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.219+0000 7f604591e640 1 --2- 172.21.15.28:0/4210421196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6040100e60 0x7f6040101280 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.219+0000 7f604591e640 1 -- 172.21.15.28:0/4210421196 >> 172.21.15.28:0/4210421196 conn(0x7f60400fc420 msgr2=0x7f60400fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.219+0000 7f604591e640 1 -- 172.21.15.28:0/4210421196 shutdown_connections 2024-04-21T08:50:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.219+0000 7f604591e640 1 -- 172.21.15.28:0/4210421196 wait complete. 2024-04-21T08:50:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.219+0000 7f604591e640 1 Processor -- start 2024-04-21T08:50:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.219+0000 7f604591e640 1 -- start start 2024-04-21T08:50:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.219+0000 7f604591e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6040100e60 0x7f6040197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.219+0000 7f604591e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6040102040 0x7f6040197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.219+0000 7f604591e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6040102a00 0x7f604019e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.219+0000 7f604591e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f604010c9f0 con 0x7f6040102a00 2024-04-21T08:50:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f604591e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f604010c870 con 0x7f6040102040 2024-04-21T08:50:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f604591e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f604010cb70 con 0x7f6040100e60 2024-04-21T08:50:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f603f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6040102a00 0x7f604019e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f603f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6040102a00 0x7f604019e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59356/0 (socket says 172.21.15.28:59356) 2024-04-21T08:50:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f603f7fe640 1 -- 172.21.15.28:0/2916188003 learned_addr learned my addr 172.21.15.28:0/2916188003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f603e7fc640 1 --2- 172.21.15.28:0/2916188003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6040102040 0x7f6040197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f603effd640 1 --2- 172.21.15.28:0/2916188003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6040100e60 0x7f6040197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f603f7fe640 1 -- 172.21.15.28:0/2916188003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6040100e60 msgr2=0x7f6040197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f603f7fe640 1 --2- 172.21.15.28:0/2916188003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6040100e60 0x7f6040197120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f603f7fe640 1 -- 172.21.15.28:0/2916188003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6040102040 msgr2=0x7f6040197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f603f7fe640 1 --2- 172.21.15.28:0/2916188003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6040102040 0x7f6040197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f603f7fe640 1 -- 172.21.15.28:0/2916188003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604019ed40 con 0x7f6040102a00 2024-04-21T08:50:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f603effd640 1 --2- 172.21.15.28:0/2916188003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6040100e60 0x7f6040197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f603f7fe640 1 --2- 172.21.15.28:0/2916188003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6040102a00 0x7f604019e6d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f603800eaa0 tx=0x7f603800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f604491c640 1 -- 172.21.15.28:0/2916188003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603800cdb0 con 0x7f6040102a00 2024-04-21T08:50:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f604591e640 1 -- 172.21.15.28:0/2916188003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f604019efd0 con 0x7f6040102a00 2024-04-21T08:50:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f604491c640 1 -- 172.21.15.28:0/2916188003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6038005020 con 0x7f6040102a00 2024-04-21T08:50:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f604491c640 1 -- 172.21.15.28:0/2916188003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60380116d0 con 0x7f6040102a00 2024-04-21T08:50:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.220+0000 7f604591e640 1 -- 172.21.15.28:0/2916188003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f604019f4a0 con 0x7f6040102a00 2024-04-21T08:50:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.223+0000 7f604491c640 1 -- 172.21.15.28:0/2916188003 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f60380051d0 con 0x7f6040102a00 2024-04-21T08:50:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.223+0000 7f60485df740 1 Processor -- start 2024-04-21T08:50:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f60485df740 1 -- start start 2024-04-21T08:50:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f60485df740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ecff540 0x55707ecff960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f60485df740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ecfff30 0x55707ed003b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f60485df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ed009d0 0x55707ed00e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f60485df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55707ec69b10 con 0x55707ed009d0 2024-04-21T08:50:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f60485df740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55707ec07240 con 0x55707ecff540 2024-04-21T08:50:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f60485df740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55707ec65ad0 con 0x55707ecfff30 2024-04-21T08:50:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f603e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ed009d0 0x55707ed00e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f603e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ed009d0 0x55707ed00e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59370/0 (socket says 172.21.15.28:59370) 2024-04-21T08:50:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f603e7fc640 1 -- 172.21.15.28:0/2905215186 learned_addr learned my addr 172.21.15.28:0/2905215186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f603e7fc640 1 --2- 172.21.15.28:0/2905215186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ecff540 0x55707ecff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f603effd640 1 --2- 172.21.15.28:0/2905215186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ecfff30 0x55707ed003b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f604491c640 1 --2- 172.21.15.28:0/2916188003 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f60280a3bb0 0x7f60280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f603f7fe640 1 --2- 172.21.15.28:0/2916188003 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f60280a3bb0 0x7f60280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f603e7fc640 1 -- 172.21.15.28:0/2905215186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ecfff30 msgr2=0x55707ed003b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f603e7fc640 1 --2- 172.21.15.28:0/2905215186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ecfff30 0x55707ed003b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f603e7fc640 1 -- 172.21.15.28:0/2905215186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ecff540 msgr2=0x55707ecff960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f603e7fc640 1 --2- 172.21.15.28:0/2905215186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ecff540 0x55707ecff960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f603e7fc640 1 -- 172.21.15.28:0/2905215186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55707e97c2d0 con 0x55707ed009d0 2024-04-21T08:50:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.224+0000 7f604491c640 1 -- 172.21.15.28:0/2916188003 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9187+0+0 (secure 0 0 0) 0x7f60380d0670 con 0x7f6040102a00 2024-04-21T08:50:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.225+0000 7f603e7fc640 1 --2- 172.21.15.28:0/2905215186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ed009d0 0x55707ed00e50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6030012770 tx=0x7f603000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75cc4c94183f9fdb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.225+0000 7f6025ffb640 1 -- 172.21.15.28:0/2905215186 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6030012cf0 con 0x55707ed009d0 2024-04-21T08:50:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.225+0000 7f6025ffb640 1 -- 172.21.15.28:0/2905215186 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6030005020 con 0x55707ed009d0 2024-04-21T08:50:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.225+0000 7f6025ffb640 1 -- 172.21.15.28:0/2905215186 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6030004a00 con 0x55707ed009d0 2024-04-21T08:50:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.225+0000 7f603f7fe640 1 --2- 172.21.15.28:0/2916188003 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f60280a3bb0 0x7f60280a6070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f603809c290 tx=0x7f6038087040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.225+0000 7f60485df740 1 -- 172.21.15.28:0/2905215186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ed009d0 msgr2=0x55707ed00e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.225+0000 7f60485df740 1 --2- 172.21.15.28:0/2905215186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ed009d0 0x55707ed00e50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6030012770 tx=0x7f603000b7c0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.225+0000 7f60485df740 1 -- 172.21.15.28:0/2905215186 shutdown_connections 2024-04-21T08:50:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.225+0000 7f60485df740 1 --2- 172.21.15.28:0/2905215186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ed009d0 0x55707ed00e50 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.225+0000 7f60485df740 1 --2- 172.21.15.28:0/2905215186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ecfff30 0x55707ed003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.225+0000 7f60485df740 1 --2- 172.21.15.28:0/2905215186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ecff540 0x55707ecff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.225+0000 7f60485df740 1 -- 172.21.15.28:0/2905215186 >> 172.21.15.28:0/2905215186 conn(0x55707ed03150 msgr2=0x55707ed016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.226+0000 7f60485df740 1 -- 172.21.15.28:0/2905215186 shutdown_connections 2024-04-21T08:50:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.226+0000 7f60485df740 1 -- 172.21.15.28:0/2905215186 wait complete. 2024-04-21T08:50:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.226+0000 7f60485df740 1 -- start start 2024-04-21T08:50:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.226+0000 7f60485df740 1 Processor -- start 2024-04-21T08:50:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.226+0000 7f60485df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecff540 0x55707ed132c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.226+0000 7f60485df740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ecfff30 0x55707ed13800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.226+0000 7f60485df740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ed009d0 0x55707ed1a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.226+0000 7f60485df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55707ed118b0 con 0x55707ecff540 2024-04-21T08:50:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.226+0000 7f60485df740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55707ebf2190 con 0x55707ecfff30 2024-04-21T08:50:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.226+0000 7f60485df740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55707ed11a30 con 0x55707ed009d0 2024-04-21T08:50:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.226+0000 7f60485df740 10 client.0 _ll_register_callbacks cb 0x55707ebed8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.226+0000 7f603e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecff540 0x55707ed132c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.226+0000 7f603e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecff540 0x55707ed132c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59374/0 (socket says 172.21.15.28:59374) 2024-04-21T08:50:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.226+0000 7f603e7fc640 1 -- 172.21.15.28:0/3740042063 learned_addr learned my addr 172.21.15.28:0/3740042063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.227+0000 7f603effd640 1 --2- 172.21.15.28:0/3740042063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ecfff30 0x55707ed13800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.227+0000 7f603e7fc640 1 -- 172.21.15.28:0/3740042063 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ed009d0 msgr2=0x55707ed1a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.227+0000 7f603e7fc640 1 --2- 172.21.15.28:0/3740042063 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ed009d0 0x55707ed1a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.227+0000 7f603e7fc640 1 -- 172.21.15.28:0/3740042063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ecfff30 msgr2=0x55707ed13800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.227+0000 7f603e7fc640 1 --2- 172.21.15.28:0/3740042063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ecfff30 0x55707ed13800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.227+0000 7f603e7fc640 1 -- 172.21.15.28:0/3740042063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55707ea466b0 con 0x55707ecff540 2024-04-21T08:50:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.227+0000 7f603e7fc640 1 --2- 172.21.15.28:0/3740042063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecff540 0x55707ed132c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f6030015040 tx=0x7f6030005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30fdd3fa8cd92b76 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.227+0000 7f601effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:50:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.227+0000 7f601effd640 1 -- 172.21.15.28:0/3740042063 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6030010740 con 0x55707ecff540 2024-04-21T08:50:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.227+0000 7f601effd640 1 -- 172.21.15.28:0/3740042063 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9187+0+0 (secure 0 0 0) 0x7f60300108e0 con 0x55707ecff540 2024-04-21T08:50:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.227+0000 7f60485df740 1 -- 172.21.15.28:0/3740042063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6030005020 con 0x55707ecff540 2024-04-21T08:50:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.227+0000 7f601effd640 1 -- 172.21.15.28:0/3740042063 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6030030050 con 0x55707ecff540 2024-04-21T08:50:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.227+0000 7f60485df740 10 client.15258 fetch_fsmap learned FSMap version 24 2024-04-21T08:50:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.227+0000 7f60485df740 1 -- 172.21.15.28:0/3740042063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55707ed1dbb0 con 0x55707ecff540 2024-04-21T08:50:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.227+0000 7f601effd640 1 -- 172.21.15.28:0/3740042063 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6030005020 con 0x55707ecff540 2024-04-21T08:50:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.228+0000 7f601effd640 1 -- 172.21.15.28:0/3740042063 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f603001e410 con 0x55707ecff540 2024-04-21T08:50:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.228+0000 7f60485df740 10 client.15258 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:50:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.228+0000 7f60485df740 10 client.15258 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:50:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.228+0000 7f60485df740 20 client.15258 populate_metadata read hostname 'smithi028' 2024-04-21T08:50:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.228+0000 7f60485df740 1 --2- 172.21.15.28:0/3740042063 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55707ed1ff90 0x55707ed21400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.228+0000 7f60485df740 4 client.15258 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.228+0000 7f60485df740 1 -- 172.21.15.28:0/3740042063 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55707ec77530 con 0x55707ed1ff90 2024-04-21T08:50:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.228+0000 7f603effd640 1 --2- 172.21.15.28:0/3740042063 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55707ed1ff90 0x55707ed21400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.228+0000 7f601effd640 10 client.15258 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:50:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.228+0000 7f603effd640 1 --2- 172.21.15.28:0/3740042063 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55707ed1ff90 0x55707ed21400 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=576ddee5fed380c3 server_cookie=1ae4a4d2c6724a2 in_seq=0 out_seq=0 2024-04-21T08:50:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.229+0000 7f601effd640 1 -- 172.21.15.28:0/3740042063 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55707ec77530 con 0x55707ed1ff90 2024-04-21T08:50:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.229+0000 7f601effd640 10 client.15258 handle_command_reply: tid=0 2024-04-21T08:50:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.229+0000 7f60485df740 1 client.15258 shutdown 2024-04-21T08:50:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.230+0000 7f60485df740 1 -- 172.21.15.28:0/3740042063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecff540 msgr2=0x55707ed132c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.230+0000 7f60485df740 1 --2- 172.21.15.28:0/3740042063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecff540 0x55707ed132c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f6030015040 tx=0x7f6030005c50 comp rx=0 tx=0).stop 2024-04-21T08:50:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.230+0000 7f60485df740 1 -- 172.21.15.28:0/3740042063 shutdown_connections 2024-04-21T08:50:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.230+0000 7f60485df740 1 --2- 172.21.15.28:0/3740042063 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55707ed1ff90 0x55707ed21400 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.230+0000 7f60485df740 1 --2- 172.21.15.28:0/3740042063 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ed009d0 0x55707ed1a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.230+0000 7f601effd640 0 client.15258 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:50:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.230+0000 7f60485df740 1 --2- 172.21.15.28:0/3740042063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ecfff30 0x55707ed13800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.230+0000 7f60485df740 1 --2- 172.21.15.28:0/3740042063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecff540 0x55707ed132c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.230+0000 7f60485df740 1 -- 172.21.15.28:0/3740042063 >> 172.21.15.28:0/3740042063 conn(0x55707ed03150 msgr2=0x55707ecfea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:37.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.230+0000 7f60485df740 1 -- 172.21.15.28:0/3740042063 shutdown_connections 2024-04-21T08:50:37.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.230+0000 7f60485df740 1 -- 172.21.15.28:0/3740042063 wait complete. 2024-04-21T08:50:37.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.230+0000 7f60485df740 20 client.15258 trim_cache size 0 max 16384 2024-04-21T08:50:37.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.248+0000 7f60485df740 1 Processor -- start 2024-04-21T08:50:37.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.248+0000 7f60485df740 1 -- start start 2024-04-21T08:50:37.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.248+0000 7f60485df740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ecff540 0x55707ed00db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.248+0000 7f60485df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecfff30 0x55707ed1a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.248+0000 7f60485df740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ed009d0 0x55707ed015f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.248+0000 7f60485df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55707ed1f740 con 0x55707ecfff30 2024-04-21T08:50:37.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.248+0000 7f60485df740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55707ed1fe10 con 0x55707ecff540 2024-04-21T08:50:37.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.248+0000 7f60485df740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55707ed204e0 con 0x55707ed009d0 2024-04-21T08:50:37.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.248+0000 7f603effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecfff30 0x55707ed1a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.248+0000 7f603effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecfff30 0x55707ed1a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59382/0 (socket says 172.21.15.28:59382) 2024-04-21T08:50:37.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.248+0000 7f603effd640 1 -- 172.21.15.28:0/3226507918 learned_addr learned my addr 172.21.15.28:0/3226507918 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:37.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.249+0000 7f603e7fc640 1 --2- 172.21.15.28:0/3226507918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ed009d0 0x55707ed015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.249+0000 7f603effd640 1 -- 172.21.15.28:0/3226507918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ed009d0 msgr2=0x55707ed015f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.249+0000 7f603effd640 1 --2- 172.21.15.28:0/3226507918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ed009d0 0x55707ed015f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.249+0000 7f603effd640 1 -- 172.21.15.28:0/3226507918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ecff540 msgr2=0x55707ed00db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.249+0000 7f603effd640 1 --2- 172.21.15.28:0/3226507918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ecff540 0x55707ed00db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.249+0000 7f603effd640 1 -- 172.21.15.28:0/3226507918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55707ed02f50 con 0x55707ecfff30 2024-04-21T08:50:37.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.249+0000 7f603e7fc640 1 --2- 172.21.15.28:0/3226507918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ed009d0 0x55707ed015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:37.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.249+0000 7f603effd640 1 --2- 172.21.15.28:0/3226507918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecfff30 0x55707ed1a7d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6040101bb0 tx=0x7f6034009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca3c6c926109b9c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:37.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.249+0000 7f60257fa640 1 -- 172.21.15.28:0/3226507918 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603402cc80 con 0x55707ecfff30 2024-04-21T08:50:37.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.249+0000 7f60257fa640 1 -- 172.21.15.28:0/3226507918 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6034005040 con 0x55707ecfff30 2024-04-21T08:50:37.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.249+0000 7f60257fa640 1 -- 172.21.15.28:0/3226507918 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6034004ae0 con 0x55707ecfff30 2024-04-21T08:50:37.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 -- 172.21.15.28:0/3226507918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecfff30 msgr2=0x55707ed1a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 --2- 172.21.15.28:0/3226507918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecfff30 0x55707ed1a7d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6040101bb0 tx=0x7f6034009260 comp rx=0 tx=0).stop 2024-04-21T08:50:37.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 -- 172.21.15.28:0/3226507918 shutdown_connections 2024-04-21T08:50:37.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 --2- 172.21.15.28:0/3226507918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ed009d0 0x55707ed015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 --2- 172.21.15.28:0/3226507918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecfff30 0x55707ed1a7d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 --2- 172.21.15.28:0/3226507918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ecff540 0x55707ed00db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 -- 172.21.15.28:0/3226507918 >> 172.21.15.28:0/3226507918 conn(0x55707ed03150 msgr2=0x55707ed1efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 -- 172.21.15.28:0/3226507918 shutdown_connections 2024-04-21T08:50:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 -- 172.21.15.28:0/3226507918 wait complete. 2024-04-21T08:50:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 -- start start 2024-04-21T08:50:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 Processor -- start 2024-04-21T08:50:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecff540 0x55707ed0e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ecfff30 0x55707ed1a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ed009d0 0x55707ed0ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55707ed21280 con 0x55707ecff540 2024-04-21T08:50:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55707ed20bb0 con 0x55707ed009d0 2024-04-21T08:50:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55707ed21950 con 0x55707ecfff30 2024-04-21T08:50:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.250+0000 7f60485df740 10 client.0 _ll_register_callbacks cb 0x7f603000be70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f603e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecff540 0x55707ed0e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f603e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecff540 0x55707ed0e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59390/0 (socket says 172.21.15.28:59390) 2024-04-21T08:50:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f603e7fc640 1 -- 172.21.15.28:0/1360553964 learned_addr learned my addr 172.21.15.28:0/1360553964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f603effd640 1 --2- 172.21.15.28:0/1360553964 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ecfff30 0x55707ed1a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f603e7fc640 1 -- 172.21.15.28:0/1360553964 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ecfff30 msgr2=0x55707ed1a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f603e7fc640 1 --2- 172.21.15.28:0/1360553964 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ecfff30 0x55707ed1a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f603e7fc640 1 -- 172.21.15.28:0/1360553964 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ed009d0 msgr2=0x55707ed0ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f603e7fc640 1 --2- 172.21.15.28:0/1360553964 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ed009d0 0x55707ed0ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f603e7fc640 1 -- 172.21.15.28:0/1360553964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55707ed0f340 con 0x55707ecff540 2024-04-21T08:50:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f603effd640 1 --2- 172.21.15.28:0/1360553964 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ecfff30 0x55707ed1a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f603e7fc640 1 --2- 172.21.15.28:0/1360553964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecff540 0x55707ed0e8c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6030015040 tx=0x7f603001ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7170ed7a942c1184 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f601effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:50:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f601effd640 1 -- 172.21.15.28:0/1360553964 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603002d070 con 0x55707ecff540 2024-04-21T08:50:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f601effd640 1 -- 172.21.15.28:0/1360553964 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9187+0+0 (secure 0 0 0) 0x7f603001e3e0 con 0x55707ecff540 2024-04-21T08:50:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f60485df740 1 -- 172.21.15.28:0/1360553964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6034005040 con 0x55707ecff540 2024-04-21T08:50:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f601effd640 1 -- 172.21.15.28:0/1360553964 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f603000bcd0 con 0x55707ecff540 2024-04-21T08:50:37.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f60485df740 10 client.15264 fetch_fsmap learned FSMap version 24 2024-04-21T08:50:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.251+0000 7f60485df740 1 -- 172.21.15.28:0/1360553964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55707ed10680 con 0x55707ecff540 2024-04-21T08:50:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.252+0000 7f601effd640 1 -- 172.21.15.28:0/1360553964 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6034005040 con 0x55707ecff540 2024-04-21T08:50:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.252+0000 7f601effd640 1 -- 172.21.15.28:0/1360553964 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f6030010840 con 0x55707ecff540 2024-04-21T08:50:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.252+0000 7f60485df740 10 client.15264 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:50:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.252+0000 7f60485df740 10 client.15264 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:50:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.252+0000 7f60485df740 20 client.15264 populate_metadata read hostname 'smithi028' 2024-04-21T08:50:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.252+0000 7f60485df740 1 --2- 172.21.15.28:0/1360553964 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55707ed8a810 0x55707ed89520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.252+0000 7f60485df740 4 client.15264 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:50:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.252+0000 7f60485df740 1 -- 172.21.15.28:0/1360553964 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "status"}) -- 0x55707ec713a0 con 0x55707ed8a810 2024-04-21T08:50:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.252+0000 7f603effd640 1 --2- 172.21.15.28:0/1360553964 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55707ed8a810 0x55707ed89520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.252+0000 7f603effd640 1 --2- 172.21.15.28:0/1360553964 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55707ed8a810 0x55707ed89520 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf49db533698441b server_cookie=3554aedd3ff69e88 in_seq=0 out_seq=0 2024-04-21T08:50:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.252+0000 7f601effd640 10 client.15264 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:50:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.253+0000 7f601effd640 1 -- 172.21.15.28:0/1360553964 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55707ec713a0 con 0x55707ed8a810 2024-04-21T08:50:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.253+0000 7f601effd640 10 client.15264 handle_command_reply: tid=0 2024-04-21T08:50:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.253+0000 7f60485df740 1 client.15264 shutdown 2024-04-21T08:50:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.253+0000 7f60485df740 1 -- 172.21.15.28:0/1360553964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecff540 msgr2=0x55707ed0e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.253+0000 7f60485df740 1 --2- 172.21.15.28:0/1360553964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecff540 0x55707ed0e8c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6030015040 tx=0x7f603001ea90 comp rx=0 tx=0).stop 2024-04-21T08:50:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.253+0000 7f60485df740 1 -- 172.21.15.28:0/1360553964 shutdown_connections 2024-04-21T08:50:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.253+0000 7f60485df740 1 --2- 172.21.15.28:0/1360553964 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55707ed8a810 0x55707ed89520 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.253+0000 7f60485df740 1 --2- 172.21.15.28:0/1360553964 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55707ed009d0 0x55707ed0ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.253+0000 7f60485df740 1 --2- 172.21.15.28:0/1360553964 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55707ecfff30 0x55707ed1a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.253+0000 7f601effd640 0 client.15264 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:50:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.253+0000 7f60485df740 1 --2- 172.21.15.28:0/1360553964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55707ecff540 0x55707ed0e8c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.253+0000 7f60485df740 1 -- 172.21.15.28:0/1360553964 >> 172.21.15.28:0/1360553964 conn(0x55707ed03150 msgr2=0x55707ed10ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.253+0000 7f60485df740 1 -- 172.21.15.28:0/1360553964 shutdown_connections 2024-04-21T08:50:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.253+0000 7f60485df740 1 -- 172.21.15.28:0/1360553964 wait complete. 2024-04-21T08:50:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.253+0000 7f60485df740 20 client.15264 trim_cache size 0 max 16384 2024-04-21T08:50:37.260 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:50:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "b51a76c8-ffba-11ee-bc93-c7b262605968", 2024-04-21T08:50:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-21T08:50:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24488, 2024-04-21T08:50:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-21T08:50:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-21T08:50:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-21T08:50:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 110.520768975, 2024-04-21T08:50:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-04-21T08:50:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 92, 2024-04-21T08:50:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:50:37.261 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 115.692973331 2024-04-21T08:50:37.261 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:50:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.254+0000 7f604591e640 1 -- 172.21.15.28:0/2916188003 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f60280a3bb0 msgr2=0x7f60280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.254+0000 7f604591e640 1 --2- 172.21.15.28:0/2916188003 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f60280a3bb0 0x7f60280a6070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f603809c290 tx=0x7f6038087040 comp rx=0 tx=0).stop 2024-04-21T08:50:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.254+0000 7f604591e640 1 -- 172.21.15.28:0/2916188003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6040102a00 msgr2=0x7f604019e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.254+0000 7f604591e640 1 --2- 172.21.15.28:0/2916188003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6040102a00 0x7f604019e6d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f603800eaa0 tx=0x7f603800ef60 comp rx=0 tx=0).stop 2024-04-21T08:50:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.254+0000 7f604591e640 1 -- 172.21.15.28:0/2916188003 shutdown_connections 2024-04-21T08:50:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.254+0000 7f604591e640 1 --2- 172.21.15.28:0/2916188003 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f60280a3bb0 0x7f60280a6070 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.254+0000 7f604591e640 1 --2- 172.21.15.28:0/2916188003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6040102a00 0x7f604019e6d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.254+0000 7f604591e640 1 --2- 172.21.15.28:0/2916188003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6040102040 0x7f6040197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.254+0000 7f604591e640 1 --2- 172.21.15.28:0/2916188003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6040100e60 0x7f6040197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.254+0000 7f604591e640 1 -- 172.21.15.28:0/2916188003 >> 172.21.15.28:0/2916188003 conn(0x7f60400fc420 msgr2=0x7f60401054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.254+0000 7f604591e640 1 -- 172.21.15.28:0/2916188003 shutdown_connections 2024-04-21T08:50:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:37.254+0000 7f604591e640 1 -- 172.21.15.28:0/2916188003 wait complete. 2024-04-21T08:50:37.264 INFO:tasks.cephadm:Setup complete, yielding 2024-04-21T08:50:37.264 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-21T08:50:37.273 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-21T08:50:37.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-21T08:50:37.524 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:37.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:37 smithi152 ceph-mon[31401]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:37.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-04-21T08:50:37.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 10]}]': finished 2024-04-21T08:50:37.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-04-21T08:50:37.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:37 smithi152 ceph-mon[31401]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-04-21T08:50:37.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:37 smithi152 ceph-mon[31401]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T08:50:37.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:37 smithi152 ceph-mon[31401]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-21T08:50:37.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:37 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/404451728' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:37.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:37 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3778635124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:37.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:37 smithi028 ceph-mon[28003]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:37.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-04-21T08:50:37.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 10]}]': finished 2024-04-21T08:50:37.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-04-21T08:50:37.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:37 smithi028 ceph-mon[28003]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-04-21T08:50:37.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:37 smithi028 ceph-mon[28003]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T08:50:37.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:37 smithi028 ceph-mon[28003]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-21T08:50:37.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:37 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/404451728' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:37.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:37 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3778635124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:37.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:37 smithi171 ceph-mon[31358]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:37.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-04-21T08:50:37.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 10]}]': finished 2024-04-21T08:50:37.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-04-21T08:50:37.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:37 smithi171 ceph-mon[31358]: from='mgr.14152 172.21.15.28:0/289953998' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-04-21T08:50:37.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:37 smithi171 ceph-mon[31358]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T08:50:37.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:37 smithi171 ceph-mon[31358]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-21T08:50:37.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:37 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/404451728' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:37.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:37 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3778635124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:38.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:38 smithi028 ceph-mon[28003]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T08:50:38.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:38 smithi152 ceph-mon[31401]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T08:50:38.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:38 smithi171 ceph-mon[31358]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T08:50:39.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.357+0000 7f81032b2700 1 -- 172.21.15.28:0/2290883976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80fc103550 msgr2=0x7f80fc105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.357+0000 7f81032b2700 1 --2- 172.21.15.28:0/2290883976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80fc103550 0x7f80fc105940 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f80f8009990 tx=0x7f80f8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.357+0000 7f81032b2700 1 -- 172.21.15.28:0/2290883976 shutdown_connections 2024-04-21T08:50:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.357+0000 7f81032b2700 1 --2- 172.21.15.28:0/2290883976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80fc103550 0x7f80fc105940 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.357+0000 7f81032b2700 1 --2- 172.21.15.28:0/2290883976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80fc1087b0 0x7f80fc10ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.357+0000 7f81032b2700 1 --2- 172.21.15.28:0/2290883976 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80fc105e80 0x7f80fc108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.357+0000 7f81032b2700 1 -- 172.21.15.28:0/2290883976 >> 172.21.15.28:0/2290883976 conn(0x7f80fc0fcf60 msgr2=0x7f80fc0ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.357+0000 7f81032b2700 1 -- 172.21.15.28:0/2290883976 shutdown_connections 2024-04-21T08:50:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.357+0000 7f81032b2700 1 -- 172.21.15.28:0/2290883976 wait complete. 2024-04-21T08:50:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f81032b2700 1 Processor -- start 2024-04-21T08:50:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f81032b2700 1 -- start start 2024-04-21T08:50:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f81032b2700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80fc103550 0x7f80fc102eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f81032b2700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80fc105e80 0x7f80fc101500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f81032b2700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80fc1087b0 0x7f80fc101a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f81032b2700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc10e540 con 0x7f80fc103550 2024-04-21T08:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f81032b2700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc10e3c0 con 0x7f80fc1087b0 2024-04-21T08:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f81032b2700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f80fc10e6c0 con 0x7f80fc105e80 2024-04-21T08:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f810104e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80fc103550 0x7f80fc102eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f810104e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80fc103550 0x7f80fc102eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59406/0 (socket says 172.21.15.28:59406) 2024-04-21T08:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f810104e700 1 -- 172.21.15.28:0/1431480081 learned_addr learned my addr 172.21.15.28:0/1431480081 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f810084d700 1 --2- 172.21.15.28:0/1431480081 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80fc105e80 0x7f80fc101500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f810184f700 1 --2- 172.21.15.28:0/1431480081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80fc1087b0 0x7f80fc101a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f810104e700 1 -- 172.21.15.28:0/1431480081 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80fc105e80 msgr2=0x7f80fc101500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f810104e700 1 --2- 172.21.15.28:0/1431480081 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80fc105e80 0x7f80fc101500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f810104e700 1 -- 172.21.15.28:0/1431480081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80fc1087b0 msgr2=0x7f80fc101a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f810104e700 1 --2- 172.21.15.28:0/1431480081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80fc1087b0 0x7f80fc101a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f810104e700 1 -- 172.21.15.28:0/1431480081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80fc1022f0 con 0x7f80fc103550 2024-04-21T08:50:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f810084d700 1 --2- 172.21.15.28:0/1431480081 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80fc105e80 0x7f80fc101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.358+0000 7f810104e700 1 --2- 172.21.15.28:0/1431480081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80fc103550 0x7f80fc102eb0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f80f80036f0 tx=0x7f80f8009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.359+0000 7f80f27fc700 1 -- 172.21.15.28:0/1431480081 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f8027070 con 0x7f80fc103550 2024-04-21T08:50:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.359+0000 7f80f27fc700 1 -- 172.21.15.28:0/1431480081 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80f8022470 con 0x7f80fc103550 2024-04-21T08:50:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.359+0000 7f81032b2700 1 -- 172.21.15.28:0/1431480081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80fc1a2c80 con 0x7f80fc103550 2024-04-21T08:50:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.359+0000 7f80f27fc700 1 -- 172.21.15.28:0/1431480081 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f800f6c0 con 0x7f80fc103550 2024-04-21T08:50:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.359+0000 7f81032b2700 1 -- 172.21.15.28:0/1431480081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80fc1a30d0 con 0x7f80fc103550 2024-04-21T08:50:39.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.360+0000 7f81032b2700 1 -- 172.21.15.28:0/1431480081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80fc069730 con 0x7f80fc103550 2024-04-21T08:50:39.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.361+0000 7f80f27fc700 1 -- 172.21.15.28:0/1431480081 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f80f802a070 con 0x7f80fc103550 2024-04-21T08:50:39.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.362+0000 7f80f27fc700 1 --2- 172.21.15.28:0/1431480081 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80e80a3c40 0x7f80e80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.362+0000 7f810084d700 1 --2- 172.21.15.28:0/1431480081 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80e80a3c40 0x7f80e80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.362+0000 7f80f27fc700 1 -- 172.21.15.28:0/1431480081 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f80f80d1920 con 0x7f80fc103550 2024-04-21T08:50:39.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.362+0000 7f810084d700 1 --2- 172.21.15.28:0/1431480081 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80e80a3c40 0x7f80e80a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f80ec005fd0 tx=0x7f80ec005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:39.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.365+0000 7f80f27fc700 1 -- 172.21.15.28:0/1431480081 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f80f8013070 con 0x7f80fc103550 2024-04-21T08:50:39.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.591+0000 7f81032b2700 1 -- 172.21.15.28:0/1431480081 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f80fc062e80 con 0x7f80e80a3c40 2024-04-21T08:50:39.591 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:39 smithi028 ceph-mon[28003]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 6 op/s 2024-04-21T08:50:39.591 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:39 smithi028 ceph-mon[28003]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-21T08:50:39.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.592+0000 7f80f27fc700 1 -- 172.21.15.28:0/1431480081 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f80fc062e80 con 0x7f80e80a3c40 2024-04-21T08:50:39.592 INFO:teuthology.orchestra.run.smithi028.stdout:Backend: cephadm 2024-04-21T08:50:39.592 INFO:teuthology.orchestra.run.smithi028.stdout:Available: Yes 2024-04-21T08:50:39.592 INFO:teuthology.orchestra.run.smithi028.stdout:Paused: No 2024-04-21T08:50:39.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.594+0000 7f81032b2700 1 -- 172.21.15.28:0/1431480081 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80e80a3c40 msgr2=0x7f80e80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.594+0000 7f81032b2700 1 --2- 172.21.15.28:0/1431480081 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80e80a3c40 0x7f80e80a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f80ec005fd0 tx=0x7f80ec005e30 comp rx=0 tx=0).stop 2024-04-21T08:50:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.594+0000 7f81032b2700 1 -- 172.21.15.28:0/1431480081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80fc103550 msgr2=0x7f80fc102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.594+0000 7f81032b2700 1 --2- 172.21.15.28:0/1431480081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80fc103550 0x7f80fc102eb0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f80f80036f0 tx=0x7f80f8009ee0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.594+0000 7f81032b2700 1 -- 172.21.15.28:0/1431480081 shutdown_connections 2024-04-21T08:50:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.594+0000 7f81032b2700 1 --2- 172.21.15.28:0/1431480081 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80e80a3c40 0x7f80e80a6100 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.594+0000 7f81032b2700 1 --2- 172.21.15.28:0/1431480081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80fc103550 0x7f80fc102eb0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.594+0000 7f81032b2700 1 --2- 172.21.15.28:0/1431480081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80fc1087b0 0x7f80fc101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.594+0000 7f81032b2700 1 --2- 172.21.15.28:0/1431480081 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80fc105e80 0x7f80fc101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.594+0000 7f81032b2700 1 -- 172.21.15.28:0/1431480081 >> 172.21.15.28:0/1431480081 conn(0x7f80fc0fcf60 msgr2=0x7f80fc103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.594+0000 7f81032b2700 1 -- 172.21.15.28:0/1431480081 shutdown_connections 2024-04-21T08:50:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:39.594+0000 7f81032b2700 1 -- 172.21.15.28:0/1431480081 wait complete. 2024-04-21T08:50:39.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:39 smithi152 ceph-mon[31401]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 6 op/s 2024-04-21T08:50:39.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:39 smithi152 ceph-mon[31401]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-21T08:50:39.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:39 smithi171 ceph-mon[31358]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 6 op/s 2024-04-21T08:50:39.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:39 smithi171 ceph-mon[31358]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-21T08:50:40.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-21T08:50:40.539 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:41.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:41 smithi152 ceph-mon[31401]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-21T08:50:41.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:41 smithi152 ceph-mon[31401]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:41.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:41 smithi028 ceph-mon[28003]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-21T08:50:41.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:41 smithi028 ceph-mon[28003]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:41.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:41 smithi171 ceph-mon[31358]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-21T08:50:41.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:41 smithi171 ceph-mon[31358]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.368+0000 7f7449828700 1 -- 172.21.15.28:0/3103190395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7444105e60 msgr2=0x7f7444108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.368+0000 7f7449828700 1 --2- 172.21.15.28:0/3103190395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7444105e60 0x7f7444108250 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7438009990 tx=0x7f7438009ca0 comp rx=0 tx=0).stop 2024-04-21T08:50:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.368+0000 7f7449828700 1 -- 172.21.15.28:0/3103190395 shutdown_connections 2024-04-21T08:50:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.368+0000 7f7449828700 1 --2- 172.21.15.28:0/3103190395 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7444108790 0x7f744410ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.368+0000 7f7449828700 1 --2- 172.21.15.28:0/3103190395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7444103530 0x7f7444105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.368+0000 7f7449828700 1 --2- 172.21.15.28:0/3103190395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7444105e60 0x7f7444108250 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.368+0000 7f7449828700 1 -- 172.21.15.28:0/3103190395 >> 172.21.15.28:0/3103190395 conn(0x7f74440fcf40 msgr2=0x7f74440ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.368+0000 7f7449828700 1 -- 172.21.15.28:0/3103190395 shutdown_connections 2024-04-21T08:50:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.368+0000 7f7449828700 1 -- 172.21.15.28:0/3103190395 wait complete. 2024-04-21T08:50:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.368+0000 7f7449828700 1 Processor -- start 2024-04-21T08:50:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f7449828700 1 -- start start 2024-04-21T08:50:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f7449828700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7444103530 0x7f744419af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f7449828700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7444105e60 0x7f744419b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f7449828700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7444108790 0x7f74441a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f7449828700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f744410e5a0 con 0x7f7444108790 2024-04-21T08:50:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f7449828700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f744410e420 con 0x7f7444103530 2024-04-21T08:50:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f7449828700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f744410e720 con 0x7f7444105e60 2024-04-21T08:50:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f74437fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7444108790 0x7f74441a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f74437fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7444108790 0x7f74441a24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59432/0 (socket says 172.21.15.28:59432) 2024-04-21T08:50:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f74437fe700 1 -- 172.21.15.28:0/2103871393 learned_addr learned my addr 172.21.15.28:0/2103871393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f74427fc700 1 --2- 172.21.15.28:0/2103871393 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7444105e60 0x7f744419b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f7442ffd700 1 --2- 172.21.15.28:0/2103871393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7444103530 0x7f744419af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f74437fe700 1 -- 172.21.15.28:0/2103871393 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7444105e60 msgr2=0x7f744419b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f74437fe700 1 --2- 172.21.15.28:0/2103871393 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7444105e60 0x7f744419b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f74437fe700 1 -- 172.21.15.28:0/2103871393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7444103530 msgr2=0x7f744419af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f74437fe700 1 --2- 172.21.15.28:0/2103871393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7444103530 0x7f744419af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f74437fe700 1 -- 172.21.15.28:0/2103871393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74441a2b80 con 0x7f7444108790 2024-04-21T08:50:42.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.369+0000 7f74437fe700 1 --2- 172.21.15.28:0/2103871393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7444108790 0x7f74441a24e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f743c007ee0 tx=0x7f743c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.370+0000 7f7448826700 1 -- 172.21.15.28:0/2103871393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c004db0 con 0x7f7444108790 2024-04-21T08:50:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.370+0000 7f7448826700 1 -- 172.21.15.28:0/2103871393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f743c005020 con 0x7f7444108790 2024-04-21T08:50:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.370+0000 7f7449828700 1 -- 172.21.15.28:0/2103871393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74441a2e70 con 0x7f7444108790 2024-04-21T08:50:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.370+0000 7f7448826700 1 -- 172.21.15.28:0/2103871393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c015660 con 0x7f7444108790 2024-04-21T08:50:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.370+0000 7f7449828700 1 -- 172.21.15.28:0/2103871393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74441a33a0 con 0x7f7444108790 2024-04-21T08:50:42.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.371+0000 7f7449828700 1 -- 172.21.15.28:0/2103871393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7444069730 con 0x7f7444108790 2024-04-21T08:50:42.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.373+0000 7f7448826700 1 -- 172.21.15.28:0/2103871393 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f743c00db20 con 0x7f7444108790 2024-04-21T08:50:42.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.373+0000 7f7448826700 1 --2- 172.21.15.28:0/2103871393 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f742c0a3cc0 0x7f742c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:42.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.373+0000 7f7442ffd700 1 --2- 172.21.15.28:0/2103871393 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f742c0a3cc0 0x7f742c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:42.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.373+0000 7f7448826700 1 -- 172.21.15.28:0/2103871393 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f743c0d0870 con 0x7f7444108790 2024-04-21T08:50:42.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.374+0000 7f7442ffd700 1 --2- 172.21.15.28:0/2103871393 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f742c0a3cc0 0x7f742c0a6180 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7434009a10 tx=0x7f7434009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:42.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.377+0000 7f7448826700 1 -- 172.21.15.28:0/2103871393 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f743c010070 con 0x7f7444108790 2024-04-21T08:50:42.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.598+0000 7f7449828700 1 -- 172.21.15.28:0/2103871393 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f7444101cb0 con 0x7f742c0a3cc0 2024-04-21T08:50:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.612+0000 7f7448826700 1 -- 172.21.15.28:0/2103871393 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f7444101cb0 con 0x7f742c0a3cc0 2024-04-21T08:50:42.612 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-21T08:50:42.612 INFO:teuthology.orchestra.run.smithi028.stdout:mds.a smithi028 running (2m) 20s ago 2m 16.3M - 19.0.0-3154-ga630765c b45a994450e2 c433953f8121 2024-04-21T08:50:42.612 INFO:teuthology.orchestra.run.smithi028.stdout:mds.b smithi152 running (2m) 55s ago 2m 28.5M - 19.0.0-3154-ga630765c b45a994450e2 90080740055c 2024-04-21T08:50:42.613 INFO:teuthology.orchestra.run.smithi028.stdout:mds.c smithi171 running (2m) 55s ago 2m 18.6M - 19.0.0-3154-ga630765c b45a994450e2 2430e65ae0c6 2024-04-21T08:50:42.613 INFO:teuthology.orchestra.run.smithi028.stdout:mds.d smithi028 running (2m) 20s ago 2m 13.1M - 19.0.0-3154-ga630765c b45a994450e2 763b96d0c50c 2024-04-21T08:50:42.613 INFO:teuthology.orchestra.run.smithi028.stdout:mds.e smithi152 running (2m) 55s ago 2m 15.7M - 19.0.0-3154-ga630765c b45a994450e2 de6b52f61a57 2024-04-21T08:50:42.613 INFO:teuthology.orchestra.run.smithi028.stdout:mds.f smithi171 running (2m) 55s ago 2m 16.1M - 19.0.0-3154-ga630765c b45a994450e2 29ece509f8b4 2024-04-21T08:50:42.613 INFO:teuthology.orchestra.run.smithi028.stdout:mds.g smithi028 running (2m) 20s ago 2m 13.0M - 19.0.0-3154-ga630765c b45a994450e2 261f4b2c1b48 2024-04-21T08:50:42.613 INFO:teuthology.orchestra.run.smithi028.stdout:mds.h smithi152 running (2m) 55s ago 2m 16.1M - 19.0.0-3154-ga630765c b45a994450e2 ccbde41de8b5 2024-04-21T08:50:42.613 INFO:teuthology.orchestra.run.smithi028.stdout:mds.i smithi171 running (2m) 55s ago 2m 17.1M - 19.0.0-3154-ga630765c b45a994450e2 6273c894fe9f 2024-04-21T08:50:42.613 INFO:teuthology.orchestra.run.smithi028.stdout:mds.j smithi028 running (2m) 20s ago 2m 16.5M - 19.0.0-3154-ga630765c b45a994450e2 167ccde53324 2024-04-21T08:50:42.613 INFO:teuthology.orchestra.run.smithi028.stdout:mds.k smithi152 running (2m) 55s ago 2m 13.2M - 19.0.0-3154-ga630765c b45a994450e2 f2ea8b06fbbd 2024-04-21T08:50:42.613 INFO:teuthology.orchestra.run.smithi028.stdout:mds.l smithi171 running (2m) 55s ago 2m 13.7M - 19.0.0-3154-ga630765c b45a994450e2 813b88aa7067 2024-04-21T08:50:42.613 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi028 *:9283,8765 running (10m) 20s ago 10m 515M - 19.0.0-3154-ga630765c b45a994450e2 fa359b820dc1 2024-04-21T08:50:42.613 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi152 *:8443,8765 running (7m) 55s ago 7m 453M - 19.0.0-3154-ga630765c b45a994450e2 aceaaf35e210 2024-04-21T08:50:42.613 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.z smithi171 *:8443,8765 running (7m) 55s ago 7m 454M - 19.0.0-3154-ga630765c b45a994450e2 0b190d088b41 2024-04-21T08:50:42.613 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (10m) 20s ago 10m 50.6M 2048M 19.0.0-3154-ga630765c b45a994450e2 4043afed007a 2024-04-21T08:50:42.614 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi152 running (8m) 55s ago 8m 40.0M 2048M 19.0.0-3154-ga630765c b45a994450e2 01fa7845fe29 2024-04-21T08:50:42.614 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi171 running (8m) 55s ago 8m 39.7M 2048M 19.0.0-3154-ga630765c b45a994450e2 a03a67da3e6e 2024-04-21T08:50:42.614 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (7m) 20s ago 7m 44.9M 4273M 19.0.0-3154-ga630765c b45a994450e2 546645dba2f5 2024-04-21T08:50:42.614 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi152 running (7m) 55s ago 7m 47.6M 4272M 19.0.0-3154-ga630765c b45a994450e2 287af4d070e2 2024-04-21T08:50:42.614 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi171 running (6m) 55s ago 6m 58.4M 4272M 19.0.0-3154-ga630765c b45a994450e2 c63ff3178d96 2024-04-21T08:50:42.614 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (6m) 20s ago 6m 43.3M 4273M 19.0.0-3154-ga630765c b45a994450e2 3e33ba8a40ad 2024-04-21T08:50:42.614 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi152 running (5m) 55s ago 5m 43.4M 4272M 19.0.0-3154-ga630765c b45a994450e2 9f8798209e0a 2024-04-21T08:50:42.614 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi171 running (5m) 55s ago 5m 58.5M 4272M 19.0.0-3154-ga630765c b45a994450e2 68f305bfa46f 2024-04-21T08:50:42.614 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi028 running (5m) 20s ago 5m 56.8M 4273M 19.0.0-3154-ga630765c b45a994450e2 68b532d10cdc 2024-04-21T08:50:42.614 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi152 running (4m) 55s ago 4m 57.7M 4272M 19.0.0-3154-ga630765c b45a994450e2 274c121220ac 2024-04-21T08:50:42.615 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 smithi171 running (4m) 55s ago 4m 42.5M 4272M 19.0.0-3154-ga630765c b45a994450e2 b9e2e45f84cc 2024-04-21T08:50:42.615 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 smithi028 running (3m) 20s ago 3m 53.3M 4273M 19.0.0-3154-ga630765c b45a994450e2 49cee985e9b0 2024-04-21T08:50:42.615 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 smithi152 running (3m) 55s ago 3m 57.6M 4272M 19.0.0-3154-ga630765c b45a994450e2 38655e4320c7 2024-04-21T08:50:42.615 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 smithi171 running (2m) 55s ago 2m 42.6M 4272M 19.0.0-3154-ga630765c b45a994450e2 8d96563737ba 2024-04-21T08:50:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.614+0000 7f7449828700 1 -- 172.21.15.28:0/2103871393 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f742c0a3cc0 msgr2=0x7f742c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.614+0000 7f7449828700 1 --2- 172.21.15.28:0/2103871393 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f742c0a3cc0 0x7f742c0a6180 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7434009a10 tx=0x7f7434009190 comp rx=0 tx=0).stop 2024-04-21T08:50:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.614+0000 7f7449828700 1 -- 172.21.15.28:0/2103871393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7444108790 msgr2=0x7f74441a24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.614+0000 7f7449828700 1 --2- 172.21.15.28:0/2103871393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7444108790 0x7f74441a24e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f743c007ee0 tx=0x7f743c00d3b0 comp rx=0 tx=0).stop 2024-04-21T08:50:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.614+0000 7f7449828700 1 -- 172.21.15.28:0/2103871393 shutdown_connections 2024-04-21T08:50:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.614+0000 7f7449828700 1 --2- 172.21.15.28:0/2103871393 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f742c0a3cc0 0x7f742c0a6180 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.614+0000 7f7449828700 1 --2- 172.21.15.28:0/2103871393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7444108790 0x7f74441a24e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.614+0000 7f7449828700 1 --2- 172.21.15.28:0/2103871393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7444103530 0x7f744419af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.614+0000 7f7449828700 1 --2- 172.21.15.28:0/2103871393 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7444105e60 0x7f744419b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.614+0000 7f7449828700 1 -- 172.21.15.28:0/2103871393 >> 172.21.15.28:0/2103871393 conn(0x7f74440fcf40 msgr2=0x7f74441015a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:42.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.615+0000 7f7449828700 1 -- 172.21.15.28:0/2103871393 shutdown_connections 2024-04-21T08:50:42.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:42.615+0000 7f7449828700 1 -- 172.21.15.28:0/2103871393 wait complete. 2024-04-21T08:50:43.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-21T08:50:43.485 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:43.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:43 smithi152 ceph-mon[31401]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 7 op/s 2024-04-21T08:50:43.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:43 smithi028 ceph-mon[28003]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 7 op/s 2024-04-21T08:50:43.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:43 smithi171 ceph-mon[31358]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 639 B/s wr, 7 op/s 2024-04-21T08:50:44.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:44 smithi152 ceph-mon[31401]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:44.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:44 smithi028 ceph-mon[28003]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:44.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:44 smithi171 ceph-mon[31358]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.377+0000 7ff1aca43700 1 -- 172.21.15.28:0/4116138643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1a8105e60 msgr2=0x7ff1a8108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.377+0000 7ff1aca43700 1 --2- 172.21.15.28:0/4116138643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1a8105e60 0x7ff1a8108250 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff19c009990 tx=0x7ff19c009ca0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.378 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:45 smithi028 ceph-mon[28003]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 639 B/s wr, 7 op/s 2024-04-21T08:50:45.378 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:45 smithi028 ceph-mon[28003]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-21T08:50:45.378 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:45 smithi028 ceph-mon[28003]: Cluster is now healthy 2024-04-21T08:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.377+0000 7ff1aca43700 1 -- 172.21.15.28:0/4116138643 shutdown_connections 2024-04-21T08:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.377+0000 7ff1aca43700 1 --2- 172.21.15.28:0/4116138643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff1a8108790 0x7ff1a810ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.377+0000 7ff1aca43700 1 --2- 172.21.15.28:0/4116138643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1a8103530 0x7ff1a8105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.377+0000 7ff1aca43700 1 --2- 172.21.15.28:0/4116138643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1a8105e60 0x7ff1a8108250 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.377+0000 7ff1aca43700 1 -- 172.21.15.28:0/4116138643 >> 172.21.15.28:0/4116138643 conn(0x7ff1a80fcf40 msgr2=0x7ff1a80ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.378+0000 7ff1aca43700 1 -- 172.21.15.28:0/4116138643 shutdown_connections 2024-04-21T08:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.378+0000 7ff1aca43700 1 -- 172.21.15.28:0/4116138643 wait complete. 2024-04-21T08:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.378+0000 7ff1aca43700 1 Processor -- start 2024-04-21T08:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1aca43700 1 -- start start 2024-04-21T08:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1aca43700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1a8103530 0x7ff1a8198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1aca43700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1a8105e60 0x7ff1a8199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:45.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1aca43700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff1a8108790 0x7ff1a81a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1aca43700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a810e550 con 0x7ff1a8103530 2024-04-21T08:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1aca43700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a810e3d0 con 0x7ff1a8105e60 2024-04-21T08:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1aca43700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a810e6d0 con 0x7ff1a8108790 2024-04-21T08:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1a659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1a8103530 0x7ff1a8198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1a659c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1a8103530 0x7ff1a8198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42134/0 (socket says 172.21.15.28:42134) 2024-04-21T08:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1a659c700 1 -- 172.21.15.28:0/82212845 learned_addr learned my addr 172.21.15.28:0/82212845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1a5d9b700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1a8105e60 0x7ff1a8199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1a6d9d700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff1a8108790 0x7ff1a81a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1a659c700 1 -- 172.21.15.28:0/82212845 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff1a8108790 msgr2=0x7ff1a81a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1a659c700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff1a8108790 0x7ff1a81a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1a659c700 1 -- 172.21.15.28:0/82212845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1a8105e60 msgr2=0x7ff1a8199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1a659c700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1a8105e60 0x7ff1a8199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1a659c700 1 -- 172.21.15.28:0/82212845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a81a08a0 con 0x7ff1a8103530 2024-04-21T08:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1a6d9d700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff1a8108790 0x7ff1a81a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.379+0000 7ff1a5d9b700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1a8105e60 0x7ff1a8199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.380+0000 7ff1a659c700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1a8103530 0x7ff1a8198c50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff19800d950 tx=0x7ff19800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.380+0000 7ff1977fe700 1 -- 172.21.15.28:0/82212845 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff198016070 con 0x7ff1a8103530 2024-04-21T08:50:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.380+0000 7ff1977fe700 1 -- 172.21.15.28:0/82212845 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff198010460 con 0x7ff1a8103530 2024-04-21T08:50:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.380+0000 7ff1aca43700 1 -- 172.21.15.28:0/82212845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1a81a0b90 con 0x7ff1a8103530 2024-04-21T08:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.380+0000 7ff1977fe700 1 -- 172.21.15.28:0/82212845 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19800f620 con 0x7ff1a8103530 2024-04-21T08:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.380+0000 7ff1aca43700 1 -- 172.21.15.28:0/82212845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1a81a10d0 con 0x7ff1a8103530 2024-04-21T08:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.381+0000 7ff1aca43700 1 -- 172.21.15.28:0/82212845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1a8069730 con 0x7ff1a8103530 2024-04-21T08:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.382+0000 7ff1977fe700 1 -- 172.21.15.28:0/82212845 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff198019070 con 0x7ff1a8103530 2024-04-21T08:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.383+0000 7ff1977fe700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff1900a3c40 0x7ff1900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.383+0000 7ff1a5d9b700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff1900a3c40 0x7ff1900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.383+0000 7ff1977fe700 1 -- 172.21.15.28:0/82212845 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7ff1980d03b0 con 0x7ff1a8103530 2024-04-21T08:50:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.384+0000 7ff1a5d9b700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff1900a3c40 0x7ff1900a6100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00b5c0 tx=0x7ff19c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.387+0000 7ff1977fe700 1 -- 172.21.15.28:0/82212845 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff19809ca70 con 0x7ff1a8103530 2024-04-21T08:50:45.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:45 smithi152 ceph-mon[31401]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 639 B/s wr, 7 op/s 2024-04-21T08:50:45.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:45 smithi152 ceph-mon[31401]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-21T08:50:45.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:45 smithi152 ceph-mon[31401]: Cluster is now healthy 2024-04-21T08:50:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.607+0000 7ff1aca43700 1 -- 172.21.15.28:0/82212845 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7ff1a8062f40 con 0x7ff1900a3c40 2024-04-21T08:50:45.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.614+0000 7ff1977fe700 1 -- 172.21.15.28:0/82212845 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7ff1a8062f40 con 0x7ff1900a3c40 2024-04-21T08:50:45.613 INFO:teuthology.orchestra.run.smithi028.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-21T08:50:45.613 INFO:teuthology.orchestra.run.smithi028.stdout:mds.all 12/12 58s ago 2m smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi152=b;smithi152=e;smithi152=h;smithi152=k;smithi171=c;smithi171=f;smithi171=i;smithi171=l;count:12 2024-04-21T08:50:45.613 INFO:teuthology.orchestra.run.smithi028.stdout:mgr 3/3 58s ago 8m smithi028=x;smithi152=y;smithi171=z;count:3 2024-04-21T08:50:45.614 INFO:teuthology.orchestra.run.smithi028.stdout:mon 3/3 58s ago 8m smithi028:172.21.15.28=a;smithi152:172.21.15.152=b;smithi171:172.21.15.171=c;count:3 2024-04-21T08:50:45.614 INFO:teuthology.orchestra.run.smithi028.stdout:osd 12 58s ago - 2024-04-21T08:50:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.616+0000 7ff1aca43700 1 -- 172.21.15.28:0/82212845 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff1900a3c40 msgr2=0x7ff1900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.616+0000 7ff1aca43700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff1900a3c40 0x7ff1900a6100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00b5c0 tx=0x7ff19c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.616+0000 7ff1aca43700 1 -- 172.21.15.28:0/82212845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1a8103530 msgr2=0x7ff1a8198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:45.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.616+0000 7ff1aca43700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1a8103530 0x7ff1a8198c50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff19800d950 tx=0x7ff19800dd10 comp rx=0 tx=0).stop 2024-04-21T08:50:45.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.616+0000 7ff1aca43700 1 -- 172.21.15.28:0/82212845 shutdown_connections 2024-04-21T08:50:45.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.616+0000 7ff1aca43700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff1900a3c40 0x7ff1900a6100 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.616+0000 7ff1aca43700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1a8103530 0x7ff1a8198c50 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.616+0000 7ff1aca43700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff1a8108790 0x7ff1a81a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.616+0000 7ff1aca43700 1 --2- 172.21.15.28:0/82212845 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1a8105e60 0x7ff1a8199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:45.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.616+0000 7ff1aca43700 1 -- 172.21.15.28:0/82212845 >> 172.21.15.28:0/82212845 conn(0x7ff1a80fcf40 msgr2=0x7ff1a8103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:45.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.616+0000 7ff1aca43700 1 -- 172.21.15.28:0/82212845 shutdown_connections 2024-04-21T08:50:45.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:45.616+0000 7ff1aca43700 1 -- 172.21.15.28:0/82212845 wait complete. 2024-04-21T08:50:45.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:45 smithi171 ceph-mon[31358]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 639 B/s wr, 7 op/s 2024-04-21T08:50:45.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:45 smithi171 ceph-mon[31358]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-21T08:50:45.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:45 smithi171 ceph-mon[31358]: Cluster is now healthy 2024-04-21T08:50:46.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-21T08:50:46.661 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:47.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:47 smithi028 ceph-mon[28003]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 222 B/s wr, 5 op/s 2024-04-21T08:50:47.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:47 smithi028 ceph-mon[28003]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:47.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:47 smithi152 ceph-mon[31401]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 222 B/s wr, 5 op/s 2024-04-21T08:50:47.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:47 smithi152 ceph-mon[31401]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:47.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:47 smithi171 ceph-mon[31358]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 222 B/s wr, 5 op/s 2024-04-21T08:50:47.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:47 smithi171 ceph-mon[31358]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.505+0000 7f6235c80700 1 -- 172.21.15.28:0/2306354904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6230101320 msgr2=0x7f6230101740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.505+0000 7f6235c80700 1 --2- 172.21.15.28:0/2306354904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6230101320 0x7f6230101740 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6220009990 tx=0x7f6220009ca0 comp rx=0 tx=0).stop 2024-04-21T08:50:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.506+0000 7f6235c80700 1 -- 172.21.15.28:0/2306354904 shutdown_connections 2024-04-21T08:50:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.506+0000 7f6235c80700 1 --2- 172.21.15.28:0/2306354904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6230101320 0x7f6230101740 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.506+0000 7f6235c80700 1 --2- 172.21.15.28:0/2306354904 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6230102640 0x7f6230102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.506+0000 7f6235c80700 1 --2- 172.21.15.28:0/2306354904 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6230101c80 0x7f6230102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.506+0000 7f6235c80700 1 -- 172.21.15.28:0/2306354904 >> 172.21.15.28:0/2306354904 conn(0x7f62300fcf40 msgr2=0x7f62300ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.506+0000 7f6235c80700 1 -- 172.21.15.28:0/2306354904 shutdown_connections 2024-04-21T08:50:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.506+0000 7f6235c80700 1 -- 172.21.15.28:0/2306354904 wait complete. 2024-04-21T08:50:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.506+0000 7f6235c80700 1 Processor -- start 2024-04-21T08:50:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.506+0000 7f6235c80700 1 -- start start 2024-04-21T08:50:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f6235c80700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6230101320 0x7f623019aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f6235c80700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6230101c80 0x7f623019b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f6235c80700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6230102640 0x7f62301a2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f6235c80700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6230110760 con 0x7f6230101c80 2024-04-21T08:50:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f6235c80700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f62301105e0 con 0x7f6230102640 2024-04-21T08:50:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f6235c80700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f62301108e0 con 0x7f6230101320 2024-04-21T08:50:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f622effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6230101c80 0x7f623019b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f622effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6230101c80 0x7f623019b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42152/0 (socket says 172.21.15.28:42152) 2024-04-21T08:50:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f622effd700 1 -- 172.21.15.28:0/1716661918 learned_addr learned my addr 172.21.15.28:0/1716661918 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f622ffff700 1 --2- 172.21.15.28:0/1716661918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6230102640 0x7f62301a2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f622f7fe700 1 --2- 172.21.15.28:0/1716661918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6230101320 0x7f623019aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f622effd700 1 -- 172.21.15.28:0/1716661918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6230101320 msgr2=0x7f623019aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f622effd700 1 --2- 172.21.15.28:0/1716661918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6230101320 0x7f623019aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f622effd700 1 -- 172.21.15.28:0/1716661918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6230102640 msgr2=0x7f62301a2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f622effd700 1 --2- 172.21.15.28:0/1716661918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6230102640 0x7f62301a2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f622effd700 1 -- 172.21.15.28:0/1716661918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62301a2b20 con 0x7f6230101c80 2024-04-21T08:50:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.507+0000 7f622effd700 1 --2- 172.21.15.28:0/1716661918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6230101c80 0x7f623019b400 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f622400d9b0 tx=0x7f622400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.508+0000 7f622cff9700 1 -- 172.21.15.28:0/1716661918 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6224017070 con 0x7f6230101c80 2024-04-21T08:50:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.508+0000 7f622cff9700 1 -- 172.21.15.28:0/1716661918 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6224005020 con 0x7f6230101c80 2024-04-21T08:50:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.508+0000 7f622cff9700 1 -- 172.21.15.28:0/1716661918 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6224009a20 con 0x7f6230101c80 2024-04-21T08:50:48.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.508+0000 7f6235c80700 1 -- 172.21.15.28:0/1716661918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62301a2e10 con 0x7f6230101c80 2024-04-21T08:50:48.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.508+0000 7f6235c80700 1 -- 172.21.15.28:0/1716661918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62301a3270 con 0x7f6230101c80 2024-04-21T08:50:48.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.509+0000 7f6235c80700 1 -- 172.21.15.28:0/1716661918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6230069730 con 0x7f6230101c80 2024-04-21T08:50:48.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.511+0000 7f622cff9700 1 -- 172.21.15.28:0/1716661918 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f62240051d0 con 0x7f6230101c80 2024-04-21T08:50:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.511+0000 7f622cff9700 1 --2- 172.21.15.28:0/1716661918 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f62180a3c40 0x7f62180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.511+0000 7f622f7fe700 1 --2- 172.21.15.28:0/1716661918 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f62180a3c40 0x7f62180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.511+0000 7f622cff9700 1 -- 172.21.15.28:0/1716661918 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f62240d0900 con 0x7f6230101c80 2024-04-21T08:50:48.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.512+0000 7f622f7fe700 1 --2- 172.21.15.28:0/1716661918 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f62180a3c40 0x7f62180a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f622000b5c0 tx=0x7f6220005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.515+0000 7f622cff9700 1 -- 172.21.15.28:0/1716661918 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6224020020 con 0x7f6230101c80 2024-04-21T08:50:48.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.734+0000 7f6235c80700 1 -- 172.21.15.28:0/1716661918 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f6230062e80 con 0x7f62180a3c40 2024-04-21T08:50:48.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.735+0000 7f622cff9700 1 -- 172.21.15.28:0/1716661918 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f6230062e80 con 0x7f62180a3c40 2024-04-21T08:50:48.735 INFO:teuthology.orchestra.run.smithi028.stdout:HOST ADDR LABELS STATUS 2024-04-21T08:50:48.735 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 172.21.15.28 2024-04-21T08:50:48.735 INFO:teuthology.orchestra.run.smithi028.stdout:smithi152 172.21.15.152 2024-04-21T08:50:48.735 INFO:teuthology.orchestra.run.smithi028.stdout:smithi171 172.21.15.171 2024-04-21T08:50:48.735 INFO:teuthology.orchestra.run.smithi028.stdout:3 hosts in cluster 2024-04-21T08:50:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.737+0000 7f6235c80700 1 -- 172.21.15.28:0/1716661918 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f62180a3c40 msgr2=0x7f62180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.737+0000 7f6235c80700 1 --2- 172.21.15.28:0/1716661918 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f62180a3c40 0x7f62180a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f622000b5c0 tx=0x7f6220005fb0 comp rx=0 tx=0).stop 2024-04-21T08:50:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.737+0000 7f6235c80700 1 -- 172.21.15.28:0/1716661918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6230101c80 msgr2=0x7f623019b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.737+0000 7f6235c80700 1 --2- 172.21.15.28:0/1716661918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6230101c80 0x7f623019b400 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f622400d9b0 tx=0x7f622400dcc0 comp rx=0 tx=0).stop 2024-04-21T08:50:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.737+0000 7f6235c80700 1 -- 172.21.15.28:0/1716661918 shutdown_connections 2024-04-21T08:50:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.737+0000 7f6235c80700 1 --2- 172.21.15.28:0/1716661918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6230102640 0x7f62301a2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.737+0000 7f6235c80700 1 --2- 172.21.15.28:0/1716661918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6230101320 0x7f623019aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.737+0000 7f6235c80700 1 --2- 172.21.15.28:0/1716661918 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f62180a3c40 0x7f62180a6100 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.737+0000 7f6235c80700 1 --2- 172.21.15.28:0/1716661918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6230101c80 0x7f623019b400 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.737+0000 7f6235c80700 1 -- 172.21.15.28:0/1716661918 >> 172.21.15.28:0/1716661918 conn(0x7f62300fcf40 msgr2=0x7f62301002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.737+0000 7f6235c80700 1 -- 172.21.15.28:0/1716661918 shutdown_connections 2024-04-21T08:50:48.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:48.737+0000 7f6235c80700 1 -- 172.21.15.28:0/1716661918 wait complete. 2024-04-21T08:50:49.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:49 smithi028 ceph-mon[28003]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 200 B/s wr, 6 op/s 2024-04-21T08:50:49.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-21T08:50:49.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:49 smithi152 ceph-mon[31401]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 200 B/s wr, 6 op/s 2024-04-21T08:50:49.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:49 smithi171 ceph-mon[31358]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 200 B/s wr, 6 op/s 2024-04-21T08:50:49.732 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:50.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:50 smithi152 ceph-mon[31401]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:50.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:50 smithi028 ceph-mon[28003]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:50.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:50 smithi171 ceph-mon[31358]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:51.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:51 smithi028 ceph-mon[28003]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-21T08:50:51.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.520+0000 7f839bbfd700 1 -- 172.21.15.28:0/1395170111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83941019a0 msgr2=0x7f8394101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.520+0000 7f839bbfd700 1 --2- 172.21.15.28:0/1395170111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83941019a0 0x7f8394101dc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f838c00b0d0 tx=0x7f838c00b3e0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.520+0000 7f839bbfd700 1 -- 172.21.15.28:0/1395170111 shutdown_connections 2024-04-21T08:50:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.520+0000 7f839bbfd700 1 --2- 172.21.15.28:0/1395170111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83941019a0 0x7f8394101dc0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.520+0000 7f839bbfd700 1 --2- 172.21.15.28:0/1395170111 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8394103540 0x7f8394109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.520+0000 7f839bbfd700 1 --2- 172.21.15.28:0/1395170111 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8394102ba0 0x7f8394103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.520+0000 7f839bbfd700 1 -- 172.21.15.28:0/1395170111 >> 172.21.15.28:0/1395170111 conn(0x7f83940fcf40 msgr2=0x7f83940ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.520+0000 7f839bbfd700 1 -- 172.21.15.28:0/1395170111 shutdown_connections 2024-04-21T08:50:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.520+0000 7f839bbfd700 1 -- 172.21.15.28:0/1395170111 wait complete. 2024-04-21T08:50:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.521+0000 7f839bbfd700 1 Processor -- start 2024-04-21T08:50:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.521+0000 7f839bbfd700 1 -- start start 2024-04-21T08:50:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.521+0000 7f839bbfd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83941019a0 0x7f839406b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.521+0000 7f839bbfd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8394102ba0 0x7f839419d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.521+0000 7f839bbfd700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8394103540 0x7f839419ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.521+0000 7f839bbfd700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f839410d490 con 0x7f8394102ba0 2024-04-21T08:50:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.521+0000 7f839bbfd700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f839410d310 con 0x7f83941019a0 2024-04-21T08:50:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.521+0000 7f839bbfd700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f839410d610 con 0x7f8394103540 2024-04-21T08:50:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.521+0000 7f8399198700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8394102ba0 0x7f839419d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.521+0000 7f8399198700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8394102ba0 0x7f839419d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42172/0 (socket says 172.21.15.28:42172) 2024-04-21T08:50:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.521+0000 7f8399198700 1 -- 172.21.15.28:0/641578278 learned_addr learned my addr 172.21.15.28:0/641578278 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.522+0000 7f8399999700 1 --2- 172.21.15.28:0/641578278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83941019a0 0x7f839406b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.522+0000 7f839a19a700 1 --2- 172.21.15.28:0/641578278 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8394103540 0x7f839419ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.522+0000 7f8399198700 1 -- 172.21.15.28:0/641578278 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8394103540 msgr2=0x7f839419ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.522+0000 7f8399198700 1 --2- 172.21.15.28:0/641578278 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8394103540 0x7f839419ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.522+0000 7f8399198700 1 -- 172.21.15.28:0/641578278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83941019a0 msgr2=0x7f839406b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.522+0000 7f8399198700 1 --2- 172.21.15.28:0/641578278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83941019a0 0x7f839406b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.522+0000 7f8399198700 1 -- 172.21.15.28:0/641578278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83941a0890 con 0x7f8394102ba0 2024-04-21T08:50:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.522+0000 7f8399198700 1 --2- 172.21.15.28:0/641578278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8394102ba0 0x7f839419d970 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f838800c9b0 tx=0x7f838800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.522+0000 7f8392ffd700 1 -- 172.21.15.28:0/641578278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8388007b00 con 0x7f8394102ba0 2024-04-21T08:50:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.522+0000 7f8392ffd700 1 -- 172.21.15.28:0/641578278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f838800ce50 con 0x7f8394102ba0 2024-04-21T08:50:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.522+0000 7f839bbfd700 1 -- 172.21.15.28:0/641578278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83941a0b80 con 0x7f8394102ba0 2024-04-21T08:50:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.522+0000 7f8392ffd700 1 -- 172.21.15.28:0/641578278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8388018660 con 0x7f8394102ba0 2024-04-21T08:50:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.522+0000 7f839bbfd700 1 -- 172.21.15.28:0/641578278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83941a1110 con 0x7f8394102ba0 2024-04-21T08:50:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.523+0000 7f839bbfd700 1 -- 172.21.15.28:0/641578278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8394069730 con 0x7f8394102ba0 2024-04-21T08:50:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.524+0000 7f8392ffd700 1 -- 172.21.15.28:0/641578278 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f838800f450 con 0x7f8394102ba0 2024-04-21T08:50:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.525+0000 7f8392ffd700 1 --2- 172.21.15.28:0/641578278 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f837c0a3cc0 0x7f837c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.525+0000 7f8399999700 1 --2- 172.21.15.28:0/641578278 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f837c0a3cc0 0x7f837c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.525+0000 7f8392ffd700 1 -- 172.21.15.28:0/641578278 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f83880d0670 con 0x7f8394102ba0 2024-04-21T08:50:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.526+0000 7f8399999700 1 --2- 172.21.15.28:0/641578278 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f837c0a3cc0 0x7f837c0a6180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f838c009fd0 tx=0x7f838c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:51.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.530+0000 7f8392ffd700 1 -- 172.21.15.28:0/641578278 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8388014030 con 0x7f8394102ba0 2024-04-21T08:50:51.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:51 smithi152 ceph-mon[31401]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-21T08:50:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:51 smithi171 ceph-mon[31358]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-21T08:50:51.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.745+0000 7f839bbfd700 1 -- 172.21.15.28:0/641578278 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f8394062e80 con 0x7f837c0a3cc0 2024-04-21T08:50:51.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.747+0000 7f8392ffd700 1 -- 172.21.15.28:0/641578278 <== mgr.14152 v2:172.21.15.28:6800/226897930 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f8394062e80 con 0x7f837c0a3cc0 2024-04-21T08:50:51.747 INFO:teuthology.orchestra.run.smithi028.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-21T08:50:51.747 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747401F4480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-21T08:50:51.747 INFO:teuthology.orchestra.run.smithi028.stdout:smithi152 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JT400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-21T08:50:51.747 INFO:teuthology.orchestra.run.smithi028.stdout:smithi171 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233001S400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-21T08:50:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.749+0000 7f839bbfd700 1 -- 172.21.15.28:0/641578278 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f837c0a3cc0 msgr2=0x7f837c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.749+0000 7f839bbfd700 1 --2- 172.21.15.28:0/641578278 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f837c0a3cc0 0x7f837c0a6180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f838c009fd0 tx=0x7f838c00b040 comp rx=0 tx=0).stop 2024-04-21T08:50:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.749+0000 7f839bbfd700 1 -- 172.21.15.28:0/641578278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8394102ba0 msgr2=0x7f839419d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.749+0000 7f839bbfd700 1 --2- 172.21.15.28:0/641578278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8394102ba0 0x7f839419d970 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f838800c9b0 tx=0x7f838800ccc0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.749+0000 7f839bbfd700 1 -- 172.21.15.28:0/641578278 shutdown_connections 2024-04-21T08:50:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.749+0000 7f839bbfd700 1 --2- 172.21.15.28:0/641578278 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8394103540 0x7f839419ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.749+0000 7f839bbfd700 1 --2- 172.21.15.28:0/641578278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83941019a0 0x7f839406b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.749+0000 7f839bbfd700 1 --2- 172.21.15.28:0/641578278 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f837c0a3cc0 0x7f837c0a6180 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.749+0000 7f839bbfd700 1 --2- 172.21.15.28:0/641578278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8394102ba0 0x7f839419d970 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:51.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.749+0000 7f839bbfd700 1 -- 172.21.15.28:0/641578278 >> 172.21.15.28:0/641578278 conn(0x7f83940fcf40 msgr2=0x7f8394105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:51.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.749+0000 7f839bbfd700 1 -- 172.21.15.28:0/641578278 shutdown_connections 2024-04-21T08:50:51.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:51.749+0000 7f839bbfd700 1 -- 172.21.15.28:0/641578278 wait complete. 2024-04-21T08:50:52.439 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-21T08:50:52.448 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-21T08:50:52.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-21T08:50:52.702 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:53.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:53 smithi028 ceph-mon[28003]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-21T08:50:53.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:53 smithi028 ceph-mon[28003]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:53.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:53 smithi152 ceph-mon[31401]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-21T08:50:53.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:53 smithi152 ceph-mon[31401]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:53.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:53 smithi171 ceph-mon[31358]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-21T08:50:53.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:53 smithi171 ceph-mon[31358]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:50:54.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.502+0000 7ff8cae9d700 1 -- 172.21.15.28:0/1507109069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8c4104640 msgr2=0x7ff8c410aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.502+0000 7ff8cae9d700 1 --2- 172.21.15.28:0/1507109069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8c4104640 0x7ff8c410aef0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc00b0d0 tx=0x7ff8bc00b3e0 comp rx=0 tx=0).stop 2024-04-21T08:50:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.502+0000 7ff8cae9d700 1 -- 172.21.15.28:0/1507109069 shutdown_connections 2024-04-21T08:50:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.502+0000 7ff8cae9d700 1 --2- 172.21.15.28:0/1507109069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8c4104640 0x7ff8c410aef0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.502+0000 7ff8cae9d700 1 --2- 172.21.15.28:0/1507109069 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff8c4102aa0 0x7ff8c4102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.502+0000 7ff8cae9d700 1 --2- 172.21.15.28:0/1507109069 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8c4103ca0 0x7ff8c4104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.502+0000 7ff8cae9d700 1 -- 172.21.15.28:0/1507109069 >> 172.21.15.28:0/1507109069 conn(0x7ff8c40fe040 msgr2=0x7ff8c4100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.502+0000 7ff8cae9d700 1 -- 172.21.15.28:0/1507109069 shutdown_connections 2024-04-21T08:50:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.502+0000 7ff8cae9d700 1 -- 172.21.15.28:0/1507109069 wait complete. 2024-04-21T08:50:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.502+0000 7ff8cae9d700 1 Processor -- start 2024-04-21T08:50:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.502+0000 7ff8cae9d700 1 -- start start 2024-04-21T08:50:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8cae9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8c4102aa0 0x7ff8c4198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8cae9d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8c4103ca0 0x7ff8c41990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8cae9d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff8c4104640 0x7ff8c41a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8cae9d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c410e540 con 0x7ff8c4102aa0 2024-04-21T08:50:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8cae9d700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c410e3c0 con 0x7ff8c4103ca0 2024-04-21T08:50:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8cae9d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c410e6c0 con 0x7ff8c4104640 2024-04-21T08:50:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8c8c39700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8c4102aa0 0x7ff8c4198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8c8c39700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8c4102aa0 0x7ff8c4198b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37468/0 (socket says 172.21.15.28:37468) 2024-04-21T08:50:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8c8c39700 1 -- 172.21.15.28:0/2429168286 learned_addr learned my addr 172.21.15.28:0/2429168286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8c943a700 1 --2- 172.21.15.28:0/2429168286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff8c4104640 0x7ff8c41a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8c3fff700 1 --2- 172.21.15.28:0/2429168286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8c4103ca0 0x7ff8c41990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8c8c39700 1 -- 172.21.15.28:0/2429168286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff8c4104640 msgr2=0x7ff8c41a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8c8c39700 1 --2- 172.21.15.28:0/2429168286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff8c4104640 0x7ff8c41a0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8c8c39700 1 -- 172.21.15.28:0/2429168286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8c4103ca0 msgr2=0x7ff8c41990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8c8c39700 1 --2- 172.21.15.28:0/2429168286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8c4103ca0 0x7ff8c41990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8c8c39700 1 -- 172.21.15.28:0/2429168286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8c41a07e0 con 0x7ff8c4102aa0 2024-04-21T08:50:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.503+0000 7ff8c8c39700 1 --2- 172.21.15.28:0/2429168286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8c4102aa0 0x7ff8c4198b80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff8b800e980 tx=0x7ff8b800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.504+0000 7ff8c1ffb700 1 -- 172.21.15.28:0/2429168286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b800cbd0 con 0x7ff8c4102aa0 2024-04-21T08:50:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.504+0000 7ff8c1ffb700 1 -- 172.21.15.28:0/2429168286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8b80044e0 con 0x7ff8c4102aa0 2024-04-21T08:50:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.504+0000 7ff8c1ffb700 1 -- 172.21.15.28:0/2429168286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b8003e00 con 0x7ff8c4102aa0 2024-04-21T08:50:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.504+0000 7ff8cae9d700 1 -- 172.21.15.28:0/2429168286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8c41a0ad0 con 0x7ff8c4102aa0 2024-04-21T08:50:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.504+0000 7ff8cae9d700 1 -- 172.21.15.28:0/2429168286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8c41a1010 con 0x7ff8c4102aa0 2024-04-21T08:50:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.505+0000 7ff8cae9d700 1 -- 172.21.15.28:0/2429168286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8c4069730 con 0x7ff8c4102aa0 2024-04-21T08:50:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.506+0000 7ff8c1ffb700 1 -- 172.21.15.28:0/2429168286 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff8b8003fa0 con 0x7ff8c4102aa0 2024-04-21T08:50:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.507+0000 7ff8c1ffb700 1 --2- 172.21.15.28:0/2429168286 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff8b40a3cc0 0x7ff8b40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.507+0000 7ff8c3fff700 1 --2- 172.21.15.28:0/2429168286 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff8b40a3cc0 0x7ff8b40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.507+0000 7ff8c1ffb700 1 -- 172.21.15.28:0/2429168286 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7ff8b80d0af0 con 0x7ff8c4102aa0 2024-04-21T08:50:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.507+0000 7ff8c3fff700 1 --2- 172.21.15.28:0/2429168286 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff8b40a3cc0 0x7ff8b40a6180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0005950 tx=0x7ff8b00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.511+0000 7ff8c1ffb700 1 -- 172.21.15.28:0/2429168286 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff8b809d110 con 0x7ff8c4102aa0 2024-04-21T08:50:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.769+0000 7ff8cae9d700 1 -- 172.21.15.28:0/2429168286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7ff8c4063650 con 0x7ff8c4102aa0 2024-04-21T08:50:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.770+0000 7ff8c1ffb700 1 -- 172.21.15.28:0/2429168286 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3058 (secure 0 0 0) 0x7ff8b809ca10 con 0x7ff8c4102aa0 2024-04-21T08:50:54.770 INFO:teuthology.orchestra.run.smithi028.stdout:e24 2024-04-21T08:50:54.770 INFO:teuthology.orchestra.run.smithi028.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T08:50:54.770 INFO:teuthology.orchestra.run.smithi028.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:50:54.770 INFO:teuthology.orchestra.run.smithi028.stdout:legacy client fscid: 1 2024-04-21T08:50:54.770 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:50:54.770 INFO:teuthology.orchestra.run.smithi028.stdout:Filesystem 'cephfs' (1) 2024-04-21T08:50:54.770 INFO:teuthology.orchestra.run.smithi028.stdout:fs_name cephfs 2024-04-21T08:50:54.770 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 24 2024-04-21T08:50:54.771 INFO:teuthology.orchestra.run.smithi028.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-21T08:50:54.771 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-04-21T08:48:45.911185+0000 2024-04-21T08:50:54.771 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-04-21T08:49:12.840022+0000 2024-04-21T08:50:54.771 INFO:teuthology.orchestra.run.smithi028.stdout:tableserver 0 2024-04-21T08:50:54.771 INFO:teuthology.orchestra.run.smithi028.stdout:root 0 2024-04-21T08:50:54.771 INFO:teuthology.orchestra.run.smithi028.stdout:session_timeout 300 2024-04-21T08:50:54.771 INFO:teuthology.orchestra.run.smithi028.stdout:session_autoclose 300 2024-04-21T08:50:54.771 INFO:teuthology.orchestra.run.smithi028.stdout:max_file_size 1099511627776 2024-04-21T08:50:54.771 INFO:teuthology.orchestra.run.smithi028.stdout:max_xattr_size 65536 2024-04-21T08:50:54.771 INFO:teuthology.orchestra.run.smithi028.stdout:required_client_features {} 2024-04-21T08:50:54.771 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure 0 2024-04-21T08:50:54.771 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure_osd_epoch 0 2024-04-21T08:50:54.771 INFO:teuthology.orchestra.run.smithi028.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:50:54.771 INFO:teuthology.orchestra.run.smithi028.stdout:max_mds 3 2024-04-21T08:50:54.771 INFO:teuthology.orchestra.run.smithi028.stdout:in 0,1,2 2024-04-21T08:50:54.772 INFO:teuthology.orchestra.run.smithi028.stdout:up {0=24488,1=24487,2=24463} 2024-04-21T08:50:54.772 INFO:teuthology.orchestra.run.smithi028.stdout:failed 2024-04-21T08:50:54.772 INFO:teuthology.orchestra.run.smithi028.stdout:damaged 2024-04-21T08:50:54.772 INFO:teuthology.orchestra.run.smithi028.stdout:stopped 2024-04-21T08:50:54.772 INFO:teuthology.orchestra.run.smithi028.stdout:data_pools [3,4] 2024-04-21T08:50:54.772 INFO:teuthology.orchestra.run.smithi028.stdout:metadata_pool 2 2024-04-21T08:50:54.772 INFO:teuthology.orchestra.run.smithi028.stdout:inline_data disabled 2024-04-21T08:50:54.772 INFO:teuthology.orchestra.run.smithi028.stdout:balancer 2024-04-21T08:50:54.772 INFO:teuthology.orchestra.run.smithi028.stdout:bal_rank_mask -1 2024-04-21T08:50:54.772 INFO:teuthology.orchestra.run.smithi028.stdout:standby_count_wanted 0 2024-04-21T08:50:54.772 INFO:teuthology.orchestra.run.smithi028.stdout:qdb_cluster leader: 24488 members: 24463,24487,24488 2024-04-21T08:50:54.772 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b{0:24488} state up:active seq 3 addr [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:50:54.772 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.f{0:24454} state up:standby-replay seq 1 addr [v2:172.21.15.171:6834/663606808,v1:172.21.15.171:6835/663606808] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:50:54.772 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.i{1:24487} state up:active seq 11 addr [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:50:54.772 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.h{1:24452} state up:standby-replay seq 1 addr [v2:172.21.15.152:6836/1261226890,v1:172.21.15.152:6837/1261226890] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:50:54.773 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c{2:24463} state up:active seq 13 addr [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:50:54.773 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.e{2:24419} state up:standby-replay seq 1 addr [v2:172.21.15.152:6834/3982885739,v1:172.21.15.152:6835/3982885739] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:50:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:50:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:50:54.773 INFO:teuthology.orchestra.run.smithi028.stdout:Standby daemons: 2024-04-21T08:50:54.773 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:50:54.773 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.j{-1:14535} state up:standby seq 1 addr [v2:172.21.15.28:6834/3468163100,v1:172.21.15.28:6835/3468163100] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:50:54.773 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d{-1:14553} state up:standby seq 1 addr [v2:172.21.15.28:6836/3823720712,v1:172.21.15.28:6837/3823720712] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:50:54.773 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.g{-1:14622} state up:standby seq 1 addr [v2:172.21.15.28:6838/3409769959,v1:172.21.15.28:6839/3409769959] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:50:54.773 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a{-1:14634} state up:standby seq 1 addr [v2:172.21.15.28:6840/1709589702,v1:172.21.15.28:6841/1709589702] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:50:54.773 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.k{-1:24392} state up:standby seq 1 addr [v2:172.21.15.152:6832/1529635773,v1:172.21.15.152:6833/1529635773] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:50:54.773 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.l{-1:24403} state up:standby seq 1 addr [v2:172.21.15.171:6832/3170910324,v1:172.21.15.171:6833/3170910324] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:50:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.773+0000 7ff8cae9d700 1 -- 172.21.15.28:0/2429168286 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff8b40a3cc0 msgr2=0x7ff8b40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.773+0000 7ff8cae9d700 1 --2- 172.21.15.28:0/2429168286 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff8b40a3cc0 0x7ff8b40a6180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0005950 tx=0x7ff8b00058e0 comp rx=0 tx=0).stop 2024-04-21T08:50:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.773+0000 7ff8cae9d700 1 -- 172.21.15.28:0/2429168286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8c4102aa0 msgr2=0x7ff8c4198b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.773+0000 7ff8cae9d700 1 --2- 172.21.15.28:0/2429168286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8c4102aa0 0x7ff8c4198b80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff8b800e980 tx=0x7ff8b800ec90 comp rx=0 tx=0).stop 2024-04-21T08:50:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.773+0000 7ff8cae9d700 1 -- 172.21.15.28:0/2429168286 shutdown_connections 2024-04-21T08:50:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.773+0000 7ff8cae9d700 1 --2- 172.21.15.28:0/2429168286 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff8b40a3cc0 0x7ff8b40a6180 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.773+0000 7ff8cae9d700 1 --2- 172.21.15.28:0/2429168286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8c4102aa0 0x7ff8c4198b80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.773+0000 7ff8cae9d700 1 --2- 172.21.15.28:0/2429168286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff8c4104640 0x7ff8c41a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.773+0000 7ff8cae9d700 1 --2- 172.21.15.28:0/2429168286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8c4103ca0 0x7ff8c41990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.773+0000 7ff8cae9d700 1 -- 172.21.15.28:0/2429168286 >> 172.21.15.28:0/2429168286 conn(0x7ff8c40fe040 msgr2=0x7ff8c4107020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.773+0000 7ff8cae9d700 1 -- 172.21.15.28:0/2429168286 shutdown_connections 2024-04-21T08:50:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:54.773+0000 7ff8cae9d700 1 -- 172.21.15.28:0/2429168286 wait complete. 2024-04-21T08:50:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-21T08:50:55.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-21T08:50:55.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:55 smithi152 ceph-mon[31401]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:55.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:55 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2429168286' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T08:50:55.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:55 smithi028 ceph-mon[28003]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:55.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:55 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2429168286' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T08:50:55.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:55 smithi171 ceph-mon[31358]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:50:55.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:55 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2429168286' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T08:50:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:50:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.301+0000 7f0a4b922700 1 -- 172.21.15.28:0/3797900724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a4410f0a0 msgr2=0x7f0a44111490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.301+0000 7f0a4b922700 1 --2- 172.21.15.28:0/3797900724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a4410f0a0 0x7f0a44111490 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00b0a0 tx=0x7f0a3c00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.301+0000 7f0a4b922700 1 -- 172.21.15.28:0/3797900724 shutdown_connections 2024-04-21T08:50:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.301+0000 7f0a4b922700 1 --2- 172.21.15.28:0/3797900724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a4410f0a0 0x7f0a44111490 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.301+0000 7f0a4b922700 1 --2- 172.21.15.28:0/3797900724 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a44076a70 0x7f0a44074ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.301+0000 7f0a4b922700 1 --2- 172.21.15.28:0/3797900724 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a440754a0 0x7f0a44075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.301+0000 7f0a4b922700 1 -- 172.21.15.28:0/3797900724 >> 172.21.15.28:0/3797900724 conn(0x7f0a440fe060 msgr2=0x7f0a441004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.301+0000 7f0a4b922700 1 -- 172.21.15.28:0/3797900724 shutdown_connections 2024-04-21T08:50:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.301+0000 7f0a4b922700 1 -- 172.21.15.28:0/3797900724 wait complete. 2024-04-21T08:50:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.301+0000 7f0a4b922700 1 Processor -- start 2024-04-21T08:50:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a4b922700 1 -- start start 2024-04-21T08:50:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a4b922700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a440754a0 0x7f0a4419f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a4b922700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a44076a70 0x7f0a4419f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a4b922700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a4410f0a0 0x7f0a441a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a4b922700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a44114b70 con 0x7f0a440754a0 2024-04-21T08:50:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a4b922700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0a441149f0 con 0x7f0a4410f0a0 2024-04-21T08:50:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a4b922700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0a44114cf0 con 0x7f0a44076a70 2024-04-21T08:50:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a496be700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a440754a0 0x7f0a4419f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a496be700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a440754a0 0x7f0a4419f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37496/0 (socket says 172.21.15.28:37496) 2024-04-21T08:50:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a496be700 1 -- 172.21.15.28:0/2052895584 learned_addr learned my addr 172.21.15.28:0/2052895584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a48ebd700 1 --2- 172.21.15.28:0/2052895584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a44076a70 0x7f0a4419f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a49ebf700 1 --2- 172.21.15.28:0/2052895584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a4410f0a0 0x7f0a441a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a496be700 1 -- 172.21.15.28:0/2052895584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a44076a70 msgr2=0x7f0a4419f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:57.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a496be700 1 --2- 172.21.15.28:0/2052895584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a44076a70 0x7f0a4419f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a496be700 1 -- 172.21.15.28:0/2052895584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a4410f0a0 msgr2=0x7f0a441a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:57.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a496be700 1 --2- 172.21.15.28:0/2052895584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a4410f0a0 0x7f0a441a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.302+0000 7f0a496be700 1 -- 172.21.15.28:0/2052895584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a441a6e10 con 0x7f0a440754a0 2024-04-21T08:50:57.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.303+0000 7f0a496be700 1 --2- 172.21.15.28:0/2052895584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a440754a0 0x7f0a4419f1b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000d980 tx=0x7f0a4000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:57.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.303+0000 7f0a3a7fc700 1 -- 172.21.15.28:0/2052895584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a40016070 con 0x7f0a440754a0 2024-04-21T08:50:57.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.303+0000 7f0a4b922700 1 -- 172.21.15.28:0/2052895584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a441a7100 con 0x7f0a440754a0 2024-04-21T08:50:57.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.303+0000 7f0a3a7fc700 1 -- 172.21.15.28:0/2052895584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a40004830 con 0x7f0a440754a0 2024-04-21T08:50:57.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.303+0000 7f0a3a7fc700 1 -- 172.21.15.28:0/2052895584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a40005560 con 0x7f0a440754a0 2024-04-21T08:50:57.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.303+0000 7f0a4b922700 1 -- 172.21.15.28:0/2052895584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a441a7640 con 0x7f0a440754a0 2024-04-21T08:50:57.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.304+0000 7f0a4b922700 1 -- 172.21.15.28:0/2052895584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a44069730 con 0x7f0a440754a0 2024-04-21T08:50:57.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.305+0000 7f0a3a7fc700 1 -- 172.21.15.28:0/2052895584 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0a40004220 con 0x7f0a440754a0 2024-04-21T08:50:57.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.305+0000 7f0a3a7fc700 1 --2- 172.21.15.28:0/2052895584 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0a300a3c40 0x7f0a300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:57.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.306+0000 7f0a48ebd700 1 --2- 172.21.15.28:0/2052895584 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0a300a3c40 0x7f0a300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:57.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.306+0000 7f0a3a7fc700 1 -- 172.21.15.28:0/2052895584 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f0a400d0500 con 0x7f0a440754a0 2024-04-21T08:50:57.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.306+0000 7f0a48ebd700 1 --2- 172.21.15.28:0/2052895584 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0a300a3c40 0x7f0a300a6100 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0a440767a0 tx=0x7f0a3400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:57.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.310+0000 7f0a3a7fc700 1 -- 172.21.15.28:0/2052895584 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a4001f020 con 0x7f0a440754a0 2024-04-21T08:50:57.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:57 smithi028 ceph-mon[28003]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.520+0000 7f0a4b922700 1 -- 172.21.15.28:0/2052895584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f0a44063650 con 0x7f0a440754a0 2024-04-21T08:50:57.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.521+0000 7f0a3a7fc700 1 -- 172.21.15.28:0/2052895584 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5672 (secure 0 0 0) 0x7f0a40022350 con 0x7f0a440754a0 2024-04-21T08:50:57.522 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 93 2024-04-21T08:50:57.522 INFO:teuthology.orchestra.run.smithi028.stdout:fsid b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T08:50:57.522 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-04-21T08:40:33.405421+0000 2024-04-21T08:50:57.522 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-04-21T08:50:37.275823+0000 2024-04-21T08:50:57.522 INFO:teuthology.orchestra.run.smithi028.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-21T08:50:57.522 INFO:teuthology.orchestra.run.smithi028.stdout:crush_version 27 2024-04-21T08:50:57.522 INFO:teuthology.orchestra.run.smithi028.stdout:full_ratio 0.95 2024-04-21T08:50:57.522 INFO:teuthology.orchestra.run.smithi028.stdout:backfillfull_ratio 0.9 2024-04-21T08:50:57.522 INFO:teuthology.orchestra.run.smithi028.stdout:nearfull_ratio 0.85 2024-04-21T08:50:57.522 INFO:teuthology.orchestra.run.smithi028.stdout:require_min_compat_client luminous 2024-04-21T08:50:57.522 INFO:teuthology.orchestra.run.smithi028.stdout:min_compat_client luminous 2024-04-21T08:50:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:require_osd_release squid 2024-04-21T08:50:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:stretch_mode_enabled false 2024-04-21T08:50:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-21T08:50:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-21T08:50:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-21T08:50:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-21T08:50:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:max_osd 12 2024-04-21T08:50:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6802/855251648,v1:172.21.15.28:6803/855251648] [v2:172.21.15.28:6804/855251648,v1:172.21.15.28:6805/855251648] exists,up b82ad926-994c-4e96-90d8-585391ba5e7f 2024-04-21T08:50:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 up in weight 1 up_from 13 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6800/728547438,v1:172.21.15.152:6801/728547438] [v2:172.21.15.152:6802/728547438,v1:172.21.15.152:6803/728547438] exists,up a75fcaea-75f3-44fe-8b41-5ace42be38d2 2024-04-21T08:50:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.171:6800/4097967582,v1:172.21.15.171:6801/4097967582] [v2:172.21.15.171:6802/4097967582,v1:172.21.15.171:6803/4097967582] exists,up e72b05a3-79ac-4a80-a9dd-6192b5e36ae0 2024-04-21T08:50:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6810/1806506959,v1:172.21.15.28:6811/1806506959] [v2:172.21.15.28:6812/1806506959,v1:172.21.15.28:6813/1806506959] exists,up 8265dd0b-19d9-45f2-887d-a0467bbca91a 2024-04-21T08:50:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 up in weight 1 up_from 33 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6808/3935841376,v1:172.21.15.152:6809/3935841376] [v2:172.21.15.152:6810/3935841376,v1:172.21.15.152:6811/3935841376] exists,up 7316601a-d423-480c-9039-27c15d5f4de4 2024-04-21T08:50:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 up in weight 1 up_from 38 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.171:6808/3821828239,v1:172.21.15.171:6809/3821828239] [v2:172.21.15.171:6810/3821828239,v1:172.21.15.171:6811/3821828239] exists,up ee83cf31-51ec-4b8f-a5f9-8457b98c44c3 2024-04-21T08:50:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6818/38541122,v1:172.21.15.28:6819/38541122] [v2:172.21.15.28:6820/38541122,v1:172.21.15.28:6821/38541122] exists,up ab718a5d-64ee-43c9-b8e8-212ce636329c 2024-04-21T08:50:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6816/3749693516,v1:172.21.15.152:6817/3749693516] [v2:172.21.15.152:6818/3749693516,v1:172.21.15.152:6819/3749693516] exists,up 4175e074-9115-40ef-8582-26478d134260 2024-04-21T08:50:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 up in weight 1 up_from 59 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.171:6816/1592405587,v1:172.21.15.171:6817/1592405587] [v2:172.21.15.171:6818/1592405587,v1:172.21.15.171:6819/1592405587] exists,up 32c9b173-a70e-4ebf-8a9f-81a0ecf6f727 2024-04-21T08:50:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 up in weight 1 up_from 67 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6826/2255450197,v1:172.21.15.28:6827/2255450197] [v2:172.21.15.28:6828/2255450197,v1:172.21.15.28:6829/2255450197] exists,up 181899a9-f577-4b15-b710-4681b4169ea1 2024-04-21T08:50:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6824/534177203,v1:172.21.15.152:6825/534177203] [v2:172.21.15.152:6826/534177203,v1:172.21.15.152:6827/534177203] exists,up 03c59ec8-844b-4120-87bd-d4421e5027c2 2024-04-21T08:50:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 up in weight 1 up_from 80 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.171:6824/2148030417,v1:172.21.15.171:6825/2148030417] [v2:172.21.15.171:6826/2148030417,v1:172.21.15.171:6827/2148030417] exists,up 0d57b2d5-a408-49a2-baf3-a0ebce5097f7 2024-04-21T08:50:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.0 [8,1] 2024-04-21T08:50:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.8 [2,11] 2024-04-21T08:50:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.c [2,4] 2024-04-21T08:50:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.10 [8,11] 2024-04-21T08:50:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.6 [4,7] 2024-04-21T08:50:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.12 [3,9] 2024-04-21T08:50:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.21 [4,8,5,9] 2024-04-21T08:50:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.25 [5,9] 2024-04-21T08:50:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.31 [4,6] 2024-04-21T08:50:57.525 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.e [5,8] 2024-04-21T08:50:57.525 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.13 [9,10] 2024-04-21T08:50:57.525 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.17 [9,8] 2024-04-21T08:50:57.525 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.1f [9,8] 2024-04-21T08:50:57.525 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/3741297321 expires 2024-04-22T08:41:35.374122+0000 2024-04-21T08:50:57.525 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/334321812 expires 2024-04-22T08:41:35.374122+0000 2024-04-21T08:50:57.525 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/2874636306 expires 2024-04-22T08:41:07.583432+0000 2024-04-21T08:50:57.525 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/334321812 expires 2024-04-22T08:41:35.374122+0000 2024-04-21T08:50:57.525 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1716427535 expires 2024-04-22T08:41:07.583432+0000 2024-04-21T08:50:57.525 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/731326464 expires 2024-04-22T08:41:35.374122+0000 2024-04-21T08:50:57.525 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/2874636306 expires 2024-04-22T08:41:07.583432+0000 2024-04-21T08:50:57.525 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/326837648 expires 2024-04-22T08:41:35.374122+0000 2024-04-21T08:50:57.525 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1980013253 expires 2024-04-22T08:41:07.583432+0000 2024-04-21T08:50:57.525 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/917824119 expires 2024-04-22T08:41:07.583432+0000 2024-04-21T08:50:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.524+0000 7f0a4b922700 1 -- 172.21.15.28:0/2052895584 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0a300a3c40 msgr2=0x7f0a300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.524+0000 7f0a4b922700 1 --2- 172.21.15.28:0/2052895584 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0a300a3c40 0x7f0a300a6100 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0a440767a0 tx=0x7f0a3400b410 comp rx=0 tx=0).stop 2024-04-21T08:50:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.524+0000 7f0a4b922700 1 -- 172.21.15.28:0/2052895584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a440754a0 msgr2=0x7f0a4419f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.524+0000 7f0a4b922700 1 --2- 172.21.15.28:0/2052895584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a440754a0 0x7f0a4419f1b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000d980 tx=0x7f0a4000dc90 comp rx=0 tx=0).stop 2024-04-21T08:50:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.524+0000 7f0a4b922700 1 -- 172.21.15.28:0/2052895584 shutdown_connections 2024-04-21T08:50:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.524+0000 7f0a4b922700 1 --2- 172.21.15.28:0/2052895584 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0a300a3c40 0x7f0a300a6100 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.524+0000 7f0a4b922700 1 --2- 172.21.15.28:0/2052895584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a440754a0 0x7f0a4419f1b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.524+0000 7f0a4b922700 1 --2- 172.21.15.28:0/2052895584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a4410f0a0 0x7f0a441a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.524+0000 7f0a4b922700 1 --2- 172.21.15.28:0/2052895584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a44076a70 0x7f0a4419f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.524+0000 7f0a4b922700 1 -- 172.21.15.28:0/2052895584 >> 172.21.15.28:0/2052895584 conn(0x7f0a440fe060 msgr2=0x7f0a44102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.524+0000 7f0a4b922700 1 -- 172.21.15.28:0/2052895584 shutdown_connections 2024-04-21T08:50:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:57.524+0000 7f0a4b922700 1 -- 172.21.15.28:0/2052895584 wait complete. 2024-04-21T08:50:57.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:57 smithi152 ceph-mon[31401]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:57.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:57 smithi171 ceph-mon[31358]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:50:58.180 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-21T08:50:58.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:50:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.302+0000 7f8224677640 1 -- 172.21.15.28:0/14516690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c102aa0 msgr2=0x7f821c104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.302+0000 7f8224677640 1 --2- 172.21.15.28:0/14516690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c102aa0 0x7f821c104e90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8218009900 tx=0x7f821802bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.302+0000 7f8224677640 1 -- 172.21.15.28:0/14516690 shutdown_connections 2024-04-21T08:50:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.302+0000 7f8224677640 1 --2- 172.21.15.28:0/14516690 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f821c107d00 0x7f821c10a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.302+0000 7f8224677640 1 --2- 172.21.15.28:0/14516690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f821c1053d0 0x7f821c1077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.302+0000 7f8224677640 1 --2- 172.21.15.28:0/14516690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c102aa0 0x7f821c104e90 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.302+0000 7f8224677640 1 -- 172.21.15.28:0/14516690 >> 172.21.15.28:0/14516690 conn(0x7f821c0fc470 msgr2=0x7f821c0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.302+0000 7f8224677640 1 -- 172.21.15.28:0/14516690 shutdown_connections 2024-04-21T08:50:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.302+0000 7f8224677640 1 -- 172.21.15.28:0/14516690 wait complete. 2024-04-21T08:50:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.302+0000 7f8224677640 1 Processor -- start 2024-04-21T08:50:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.302+0000 7f8224677640 1 -- start start 2024-04-21T08:50:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8224677640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f821c102aa0 0x7f821c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8224677640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f821c1053d0 0x7f821c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8224677640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c107d00 0x7f821c19f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8224677640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f821c10db20 con 0x7f821c107d00 2024-04-21T08:50:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8224677640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f821c10d9a0 con 0x7f821c1053d0 2024-04-21T08:50:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8224677640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f821c10dca0 con 0x7f821c102aa0 2024-04-21T08:50:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8222c14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c107d00 0x7f821c19f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8222c14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c107d00 0x7f821c19f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37502/0 (socket says 172.21.15.28:37502) 2024-04-21T08:50:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8222c14640 1 -- 172.21.15.28:0/3535146579 learned_addr learned my addr 172.21.15.28:0/3535146579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8222413640 1 --2- 172.21.15.28:0/3535146579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f821c102aa0 0x7f821c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8221c12640 1 --2- 172.21.15.28:0/3535146579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f821c1053d0 0x7f821c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8222c14640 1 -- 172.21.15.28:0/3535146579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f821c102aa0 msgr2=0x7f821c1981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8222c14640 1 --2- 172.21.15.28:0/3535146579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f821c102aa0 0x7f821c1981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8222c14640 1 -- 172.21.15.28:0/3535146579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f821c1053d0 msgr2=0x7f821c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8222c14640 1 --2- 172.21.15.28:0/3535146579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f821c1053d0 0x7f821c198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8222c14640 1 -- 172.21.15.28:0/3535146579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f821c19fe10 con 0x7f821c107d00 2024-04-21T08:50:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f8222c14640 1 --2- 172.21.15.28:0/3535146579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c107d00 0x7f821c19f7a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f821400eaa0 tx=0x7f821400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.303+0000 7f82137fe640 1 -- 172.21.15.28:0/3535146579 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821400cdb0 con 0x7f821c107d00 2024-04-21T08:50:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.304+0000 7f82137fe640 1 -- 172.21.15.28:0/3535146579 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8214005020 con 0x7f821c107d00 2024-04-21T08:50:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.304+0000 7f8224677640 1 -- 172.21.15.28:0/3535146579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f821c1a0100 con 0x7f821c107d00 2024-04-21T08:50:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.304+0000 7f82137fe640 1 -- 172.21.15.28:0/3535146579 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82140116d0 con 0x7f821c107d00 2024-04-21T08:50:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.304+0000 7f8224677640 1 -- 172.21.15.28:0/3535146579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f821c1a06a0 con 0x7f821c107d00 2024-04-21T08:50:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.305+0000 7f8224677640 1 -- 172.21.15.28:0/3535146579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f821c0fdf60 con 0x7f821c107d00 2024-04-21T08:50:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.306+0000 7f82137fe640 1 -- 172.21.15.28:0/3535146579 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f82140051d0 con 0x7f821c107d00 2024-04-21T08:50:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.307+0000 7f82137fe640 1 --2- 172.21.15.28:0/3535146579 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f82080a3c30 0x7f82080a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.307+0000 7f8222413640 1 --2- 172.21.15.28:0/3535146579 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f82080a3c30 0x7f82080a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.307+0000 7f82137fe640 1 -- 172.21.15.28:0/3535146579 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f82140d0880 con 0x7f821c107d00 2024-04-21T08:50:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.307+0000 7f8222413640 1 --2- 172.21.15.28:0/3535146579 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f82080a3c30 0x7f82080a60f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8218009230 tx=0x7f8218005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.312+0000 7f82137fe640 1 -- 172.21.15.28:0/3535146579 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8214016030 con 0x7f821c107d00 2024-04-21T08:50:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.498+0000 7f8224677640 1 -- 172.21.15.28:0/3535146579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f821c062b50 con 0x7f821c107d00 2024-04-21T08:50:58.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.500+0000 7f82137fe640 1 -- 172.21.15.28:0/3535146579 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24786 (secure 0 0 0) 0x7f821409c7f0 con 0x7f821c107d00 2024-04-21T08:50:58.501 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:50:58.501 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:50:37.275823+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:50:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.502+0000 7f8224677640 1 -- 172.21.15.28:0/3535146579 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f82080a3c30 msgr2=0x7f82080a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.502+0000 7f8224677640 1 --2- 172.21.15.28:0/3535146579 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f82080a3c30 0x7f82080a60f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8218009230 tx=0x7f8218005fd0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.502+0000 7f8224677640 1 -- 172.21.15.28:0/3535146579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c107d00 msgr2=0x7f821c19f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.502+0000 7f8224677640 1 --2- 172.21.15.28:0/3535146579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c107d00 0x7f821c19f7a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f821400eaa0 tx=0x7f821400ef60 comp rx=0 tx=0).stop 2024-04-21T08:50:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.502+0000 7f8224677640 1 -- 172.21.15.28:0/3535146579 shutdown_connections 2024-04-21T08:50:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.502+0000 7f8224677640 1 --2- 172.21.15.28:0/3535146579 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f82080a3c30 0x7f82080a60f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.502+0000 7f8224677640 1 --2- 172.21.15.28:0/3535146579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f821c107d00 0x7f821c19f7a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.502+0000 7f8224677640 1 --2- 172.21.15.28:0/3535146579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f821c1053d0 0x7f821c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.502+0000 7f8224677640 1 --2- 172.21.15.28:0/3535146579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f821c102aa0 0x7f821c1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.502+0000 7f8224677640 1 -- 172.21.15.28:0/3535146579 >> 172.21.15.28:0/3535146579 conn(0x7f821c0fc470 msgr2=0x7f821c103400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.502+0000 7f8224677640 1 -- 172.21.15.28:0/3535146579 shutdown_connections 2024-04-21T08:50:58.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.502+0000 7f8224677640 1 -- 172.21.15.28:0/3535146579 wait complete. 2024-04-21T08:50:58.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:58 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2052895584' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T08:50:58.515 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:48:43.543799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:48:44.864564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:48:48.747842+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:50:58.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:50:58.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:58 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2052895584' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T08:50:58.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:58 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2052895584' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T08:50:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.620+0000 7f8863f11640 1 -- 172.21.15.28:0/2416623821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c102040 msgr2=0x7f885c1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.620+0000 7f8863f11640 1 --2- 172.21.15.28:0/2416623821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c102040 0x7f885c1024c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8850009900 tx=0x7f885002bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.620+0000 7f8863f11640 1 -- 172.21.15.28:0/2416623821 shutdown_connections 2024-04-21T08:50:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.620+0000 7f8863f11640 1 --2- 172.21.15.28:0/2416623821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f885c102a00 0x7f885c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.620+0000 7f8863f11640 1 --2- 172.21.15.28:0/2416623821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c102040 0x7f885c1024c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.620+0000 7f8863f11640 1 --2- 172.21.15.28:0/2416623821 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f885c100e60 0x7f885c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.620+0000 7f8863f11640 1 -- 172.21.15.28:0/2416623821 >> 172.21.15.28:0/2416623821 conn(0x7f885c0fc440 msgr2=0x7f885c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.620+0000 7f8863f11640 1 -- 172.21.15.28:0/2416623821 shutdown_connections 2024-04-21T08:50:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.620+0000 7f8863f11640 1 -- 172.21.15.28:0/2416623821 wait complete. 2024-04-21T08:50:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8863f11640 1 Processor -- start 2024-04-21T08:50:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8863f11640 1 -- start start 2024-04-21T08:50:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8863f11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c100e60 0x7f885c198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8863f11640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f885c102040 0x7f885c1987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8863f11640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f885c102a00 0x7f885c19f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8863f11640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f885c10ca70 con 0x7f885c100e60 2024-04-21T08:50:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8863f11640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f885c10c8f0 con 0x7f885c102a00 2024-04-21T08:50:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8863f11640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f885c10cbf0 con 0x7f885c102040 2024-04-21T08:50:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8861cad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c100e60 0x7f885c198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8861cad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c100e60 0x7f885c198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37514/0 (socket says 172.21.15.28:37514) 2024-04-21T08:50:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8861cad640 1 -- 172.21.15.28:0/3195586186 learned_addr learned my addr 172.21.15.28:0/3195586186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f88614ac640 1 --2- 172.21.15.28:0/3195586186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f885c102040 0x7f885c1987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8861cad640 1 -- 172.21.15.28:0/3195586186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f885c102040 msgr2=0x7f885c1987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8861cad640 1 --2- 172.21.15.28:0/3195586186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f885c102040 0x7f885c1987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8861cad640 1 -- 172.21.15.28:0/3195586186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f885c102a00 msgr2=0x7f885c19f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8861cad640 1 --2- 172.21.15.28:0/3195586186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f885c102a00 0x7f885c19f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8861cad640 1 -- 172.21.15.28:0/3195586186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f885c19feb0 con 0x7f885c100e60 2024-04-21T08:50:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.621+0000 7f8861cad640 1 --2- 172.21.15.28:0/3195586186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c100e60 0x7f885c198290 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f885800b450 tx=0x7f885800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.622+0000 7f884effd640 1 -- 172.21.15.28:0/3195586186 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8858015020 con 0x7f885c100e60 2024-04-21T08:50:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.622+0000 7f884effd640 1 -- 172.21.15.28:0/3195586186 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8858011d40 con 0x7f885c100e60 2024-04-21T08:50:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.622+0000 7f8863f11640 1 -- 172.21.15.28:0/3195586186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f885c1a01a0 con 0x7f885c100e60 2024-04-21T08:50:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.622+0000 7f884effd640 1 -- 172.21.15.28:0/3195586186 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885800f340 con 0x7f885c100e60 2024-04-21T08:50:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.622+0000 7f8863f11640 1 -- 172.21.15.28:0/3195586186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f885c1a0720 con 0x7f885c100e60 2024-04-21T08:50:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.623+0000 7f8863f11640 1 -- 172.21.15.28:0/3195586186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f885c1a09e0 con 0x7f885c100e60 2024-04-21T08:50:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.624+0000 7f884effd640 1 -- 172.21.15.28:0/3195586186 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8858018020 con 0x7f885c100e60 2024-04-21T08:50:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.624+0000 7f884effd640 1 --2- 172.21.15.28:0/3195586186 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f88480a3bb0 0x7f88480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.625+0000 7f88614ac640 1 --2- 172.21.15.28:0/3195586186 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f88480a3bb0 0x7f88480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.625+0000 7f884effd640 1 -- 172.21.15.28:0/3195586186 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f88580d0920 con 0x7f885c100e60 2024-04-21T08:50:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.625+0000 7f88614ac640 1 --2- 172.21.15.28:0/3195586186 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f88480a3bb0 0x7f88480a6070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8850036040 tx=0x7f8850000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.628+0000 7f884effd640 1 -- 172.21.15.28:0/3195586186 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f885801f020 con 0x7f885c100e60 2024-04-21T08:50:58.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.815+0000 7f8863f11640 1 -- 172.21.15.28:0/3195586186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f885c1a0bb0 con 0x7f885c100e60 2024-04-21T08:50:58.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.815+0000 7f884effd640 1 -- 172.21.15.28:0/3195586186 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f885800d040 con 0x7f885c100e60 2024-04-21T08:50:58.816 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T08:50:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.817+0000 7f8863f11640 1 -- 172.21.15.28:0/3195586186 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f88480a3bb0 msgr2=0x7f88480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.817+0000 7f8863f11640 1 --2- 172.21.15.28:0/3195586186 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f88480a3bb0 0x7f88480a6070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8850036040 tx=0x7f8850000bc0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.817+0000 7f8863f11640 1 -- 172.21.15.28:0/3195586186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c100e60 msgr2=0x7f885c198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.817+0000 7f8863f11640 1 --2- 172.21.15.28:0/3195586186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c100e60 0x7f885c198290 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f885800b450 tx=0x7f885800b910 comp rx=0 tx=0).stop 2024-04-21T08:50:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.817+0000 7f8863f11640 1 -- 172.21.15.28:0/3195586186 shutdown_connections 2024-04-21T08:50:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.817+0000 7f8863f11640 1 --2- 172.21.15.28:0/3195586186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f885c102a00 0x7f885c19f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.817+0000 7f8863f11640 1 --2- 172.21.15.28:0/3195586186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f885c102040 0x7f885c1987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.817+0000 7f8863f11640 1 --2- 172.21.15.28:0/3195586186 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f88480a3bb0 0x7f88480a6070 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.817+0000 7f8863f11640 1 --2- 172.21.15.28:0/3195586186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f885c100e60 0x7f885c198290 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.817+0000 7f8863f11640 1 -- 172.21.15.28:0/3195586186 >> 172.21.15.28:0/3195586186 conn(0x7f885c0fc440 msgr2=0x7f885c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.817+0000 7f8863f11640 1 -- 172.21.15.28:0/3195586186 shutdown_connections 2024-04-21T08:50:58.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.817+0000 7f8863f11640 1 -- 172.21.15.28:0/3195586186 wait complete. 2024-04-21T08:50:58.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:50:58.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.933+0000 7f533a06d640 1 -- 172.21.15.28:0/2218155704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5334107cd0 msgr2=0x7f533410a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.933+0000 7f533a06d640 1 --2- 172.21.15.28:0/2218155704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5334107cd0 0x7f533410a0c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f532c009ca0 tx=0x7f532c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.933+0000 7f533a06d640 1 -- 172.21.15.28:0/2218155704 shutdown_connections 2024-04-21T08:50:58.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.933+0000 7f533a06d640 1 --2- 172.21.15.28:0/2218155704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5334107cd0 0x7f533410a0c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.933+0000 7f533a06d640 1 --2- 172.21.15.28:0/2218155704 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f53341053a0 0x7f5334107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.933+0000 7f533a06d640 1 --2- 172.21.15.28:0/2218155704 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5334102a70 0x7f5334104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.933+0000 7f533a06d640 1 -- 172.21.15.28:0/2218155704 >> 172.21.15.28:0/2218155704 conn(0x7f53340fc400 msgr2=0x7f53340fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.933+0000 7f533a06d640 1 -- 172.21.15.28:0/2218155704 shutdown_connections 2024-04-21T08:50:58.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.934+0000 7f533a06d640 1 -- 172.21.15.28:0/2218155704 wait complete. 2024-04-21T08:50:58.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.934+0000 7f533a06d640 1 Processor -- start 2024-04-21T08:50:58.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.934+0000 7f533a06d640 1 -- start start 2024-04-21T08:50:58.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.934+0000 7f533a06d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5334102a70 0x7f5334198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.934+0000 7f533a06d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f53341053a0 0x7f53341987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.934+0000 7f533a06d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5334107cd0 0x7f533419f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.934+0000 7f533a06d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f533410dae0 con 0x7f5334102a70 2024-04-21T08:50:58.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.934+0000 7f533a06d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f533410d960 con 0x7f5334107cd0 2024-04-21T08:50:58.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.934+0000 7f533a06d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f533410dc60 con 0x7f53341053a0 2024-04-21T08:50:58.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.934+0000 7f53337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5334102a70 0x7f5334198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f53337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5334102a70 0x7f5334198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37538/0 (socket says 172.21.15.28:37538) 2024-04-21T08:50:58.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f53337fe640 1 -- 172.21.15.28:0/2384637785 learned_addr learned my addr 172.21.15.28:0/2384637785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:58.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f5333fff640 1 --2- 172.21.15.28:0/2384637785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5334107cd0 0x7f533419f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f5332ffd640 1 --2- 172.21.15.28:0/2384637785 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f53341053a0 0x7f53341987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f53337fe640 1 -- 172.21.15.28:0/2384637785 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f53341053a0 msgr2=0x7f53341987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f53337fe640 1 --2- 172.21.15.28:0/2384637785 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f53341053a0 0x7f53341987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f53337fe640 1 -- 172.21.15.28:0/2384637785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5334107cd0 msgr2=0x7f533419f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f53337fe640 1 --2- 172.21.15.28:0/2384637785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5334107cd0 0x7f533419f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f53337fe640 1 -- 172.21.15.28:0/2384637785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f533419fe30 con 0x7f5334102a70 2024-04-21T08:50:58.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f5332ffd640 1 --2- 172.21.15.28:0/2384637785 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f53341053a0 0x7f53341987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:58.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f53337fe640 1 --2- 172.21.15.28:0/2384637785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5334102a70 0x7f5334198260 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f532400c970 tx=0x7f532400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f5330ff9640 1 -- 172.21.15.28:0/2384637785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5324007bb0 con 0x7f5334102a70 2024-04-21T08:50:58.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f533a06d640 1 -- 172.21.15.28:0/2384637785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53341a0120 con 0x7f5334102a70 2024-04-21T08:50:58.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f5330ff9640 1 -- 172.21.15.28:0/2384637785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5324007d50 con 0x7f5334102a70 2024-04-21T08:50:58.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f5330ff9640 1 -- 172.21.15.28:0/2384637785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532400f450 con 0x7f5334102a70 2024-04-21T08:50:58.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.935+0000 7f533a06d640 1 -- 172.21.15.28:0/2384637785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53341a0620 con 0x7f5334102a70 2024-04-21T08:50:58.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.936+0000 7f533a06d640 1 -- 172.21.15.28:0/2384637785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53340fdef0 con 0x7f5334102a70 2024-04-21T08:50:58.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.937+0000 7f5330ff9640 1 -- 172.21.15.28:0/2384637785 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f5324004f20 con 0x7f5334102a70 2024-04-21T08:50:58.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.938+0000 7f5330ff9640 1 --2- 172.21.15.28:0/2384637785 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f531c0a3bb0 0x7f531c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.938+0000 7f5332ffd640 1 --2- 172.21.15.28:0/2384637785 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f531c0a3bb0 0x7f531c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.938+0000 7f5330ff9640 1 -- 172.21.15.28:0/2384637785 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f53240d05e0 con 0x7f5334102a70 2024-04-21T08:50:58.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.939+0000 7f5332ffd640 1 --2- 172.21.15.28:0/2384637785 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f531c0a3bb0 0x7f531c0a6070 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5328005d00 tx=0x7f5328009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:58.943+0000 7f5330ff9640 1 -- 172.21.15.28:0/2384637785 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5324014030 con 0x7f5334102a70 2024-04-21T08:50:59.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.127+0000 7f533a06d640 1 -- 172.21.15.28:0/2384637785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5334062b80 con 0x7f5334102a70 2024-04-21T08:50:59.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.128+0000 7f5330ff9640 1 -- 172.21.15.28:0/2384637785 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f532409cc00 con 0x7f5334102a70 2024-04-21T08:50:59.129 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:50:59.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.130+0000 7f533a06d640 1 -- 172.21.15.28:0/2384637785 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f531c0a3bb0 msgr2=0x7f531c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.130+0000 7f533a06d640 1 --2- 172.21.15.28:0/2384637785 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f531c0a3bb0 0x7f531c0a6070 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5328005d00 tx=0x7f5328009550 comp rx=0 tx=0).stop 2024-04-21T08:50:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.130+0000 7f533a06d640 1 -- 172.21.15.28:0/2384637785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5334102a70 msgr2=0x7f5334198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.130+0000 7f533a06d640 1 --2- 172.21.15.28:0/2384637785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5334102a70 0x7f5334198260 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f532400c970 tx=0x7f532400ce30 comp rx=0 tx=0).stop 2024-04-21T08:50:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.130+0000 7f533a06d640 1 -- 172.21.15.28:0/2384637785 shutdown_connections 2024-04-21T08:50:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.130+0000 7f533a06d640 1 --2- 172.21.15.28:0/2384637785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5334107cd0 0x7f533419f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.130+0000 7f533a06d640 1 --2- 172.21.15.28:0/2384637785 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f53341053a0 0x7f53341987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.130+0000 7f533a06d640 1 --2- 172.21.15.28:0/2384637785 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f531c0a3bb0 0x7f531c0a6070 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.130+0000 7f533a06d640 1 --2- 172.21.15.28:0/2384637785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5334102a70 0x7f5334198260 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.130+0000 7f533a06d640 1 -- 172.21.15.28:0/2384637785 >> 172.21.15.28:0/2384637785 conn(0x7f53340fc400 msgr2=0x7f53341034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.130+0000 7f533a06d640 1 -- 172.21.15.28:0/2384637785 shutdown_connections 2024-04-21T08:50:59.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.130+0000 7f533a06d640 1 -- 172.21.15.28:0/2384637785 wait complete. 2024-04-21T08:50:59.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:50:59.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 -- 172.21.15.28:0/2316390286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff67c101ff0 msgr2=0x7ff67c102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 --2- 172.21.15.28:0/2316390286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff67c101ff0 0x7ff67c102470 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff66c009900 tx=0x7ff66c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:59.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 -- 172.21.15.28:0/2316390286 shutdown_connections 2024-04-21T08:50:59.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 --2- 172.21.15.28:0/2316390286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff67c1029b0 0x7ff67c1092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 --2- 172.21.15.28:0/2316390286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff67c101ff0 0x7ff67c102470 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 --2- 172.21.15.28:0/2316390286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff67c100ea0 0x7ff67c1012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 -- 172.21.15.28:0/2316390286 >> 172.21.15.28:0/2316390286 conn(0x7ff67c0fc410 msgr2=0x7ff67c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:59.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 -- 172.21.15.28:0/2316390286 shutdown_connections 2024-04-21T08:50:59.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 -- 172.21.15.28:0/2316390286 wait complete. 2024-04-21T08:50:59.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 Processor -- start 2024-04-21T08:50:59.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 -- start start 2024-04-21T08:50:59.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff67c100ea0 0x7ff67c06adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff67c101ff0 0x7ff67c19cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff67c1029b0 0x7ff67c19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c10ca40 con 0x7ff67c100ea0 2024-04-21T08:50:59.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c10c8c0 con 0x7ff67c1029b0 2024-04-21T08:50:59.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.246+0000 7ff683d1e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff67c10cbc0 con 0x7ff67c101ff0 2024-04-21T08:50:59.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff681aba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff67c100ea0 0x7ff67c06adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:59.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff681aba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff67c100ea0 0x7ff67c06adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37552/0 (socket says 172.21.15.28:37552) 2024-04-21T08:50:59.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff681aba640 1 -- 172.21.15.28:0/1822876961 learned_addr learned my addr 172.21.15.28:0/1822876961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:59.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff6822bb640 1 --2- 172.21.15.28:0/1822876961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff67c1029b0 0x7ff67c19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:59.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff681aba640 1 -- 172.21.15.28:0/1822876961 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff67c101ff0 msgr2=0x7ff67c19cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff681aba640 1 --2- 172.21.15.28:0/1822876961 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff67c101ff0 0x7ff67c19cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff681aba640 1 -- 172.21.15.28:0/1822876961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff67c1029b0 msgr2=0x7ff67c19f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff681aba640 1 --2- 172.21.15.28:0/1822876961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff67c1029b0 0x7ff67c19f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff681aba640 1 -- 172.21.15.28:0/1822876961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff67c19fec0 con 0x7ff67c100ea0 2024-04-21T08:50:59.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff6822bb640 1 --2- 172.21.15.28:0/1822876961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff67c1029b0 0x7ff67c19f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:59.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff681aba640 1 --2- 172.21.15.28:0/1822876961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff67c100ea0 0x7ff67c06adb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff67800d950 tx=0x7ff67800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:59.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff672ffd640 1 -- 172.21.15.28:0/1822876961 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678017070 con 0x7ff67c100ea0 2024-04-21T08:50:59.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff683d1e640 1 -- 172.21.15.28:0/1822876961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff67c1a01b0 con 0x7ff67c100ea0 2024-04-21T08:50:59.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff672ffd640 1 -- 172.21.15.28:0/1822876961 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff678009c30 con 0x7ff67c100ea0 2024-04-21T08:50:59.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff672ffd640 1 -- 172.21.15.28:0/1822876961 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67800f620 con 0x7ff67c100ea0 2024-04-21T08:50:59.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.247+0000 7ff683d1e640 1 -- 172.21.15.28:0/1822876961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff67c1a0750 con 0x7ff67c100ea0 2024-04-21T08:50:59.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.248+0000 7ff683d1e640 1 -- 172.21.15.28:0/1822876961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff67c108280 con 0x7ff67c100ea0 2024-04-21T08:50:59.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.249+0000 7ff672ffd640 1 -- 172.21.15.28:0/1822876961 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff678010460 con 0x7ff67c100ea0 2024-04-21T08:50:59.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.250+0000 7ff672ffd640 1 --2- 172.21.15.28:0/1822876961 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff6680a3bb0 0x7ff6680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.250+0000 7ff672ffd640 1 -- 172.21.15.28:0/1822876961 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7ff6780d0480 con 0x7ff67c100ea0 2024-04-21T08:50:59.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.250+0000 7ff6812b9640 1 --2- 172.21.15.28:0/1822876961 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff6680a3bb0 0x7ff6680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:59.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.251+0000 7ff6812b9640 1 --2- 172.21.15.28:0/1822876961 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff6680a3bb0 0x7ff6680a6070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff66c036040 tx=0x7ff66c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:59.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.254+0000 7ff672ffd640 1 -- 172.21.15.28:0/1822876961 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff67809caa0 con 0x7ff67c100ea0 2024-04-21T08:50:59.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.440+0000 7ff683d1e640 1 -- 172.21.15.28:0/1822876961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff67c02d820 con 0x7ff67c100ea0 2024-04-21T08:50:59.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.441+0000 7ff672ffd640 1 -- 172.21.15.28:0/1822876961 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7ff678009de0 con 0x7ff67c100ea0 2024-04-21T08:50:59.441 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-21T08:50:59.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.442+0000 7ff683d1e640 1 -- 172.21.15.28:0/1822876961 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff6680a3bb0 msgr2=0x7ff6680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.442+0000 7ff683d1e640 1 --2- 172.21.15.28:0/1822876961 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff6680a3bb0 0x7ff6680a6070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff66c036040 tx=0x7ff66c000bc0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.442+0000 7ff683d1e640 1 -- 172.21.15.28:0/1822876961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff67c100ea0 msgr2=0x7ff67c06adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.442+0000 7ff683d1e640 1 --2- 172.21.15.28:0/1822876961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff67c100ea0 0x7ff67c06adb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff67800d950 tx=0x7ff67800de10 comp rx=0 tx=0).stop 2024-04-21T08:50:59.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.442+0000 7ff683d1e640 1 -- 172.21.15.28:0/1822876961 shutdown_connections 2024-04-21T08:50:59.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.442+0000 7ff683d1e640 1 --2- 172.21.15.28:0/1822876961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff67c1029b0 0x7ff67c19f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.442+0000 7ff683d1e640 1 --2- 172.21.15.28:0/1822876961 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff67c101ff0 0x7ff67c19cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.442+0000 7ff683d1e640 1 --2- 172.21.15.28:0/1822876961 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff6680a3bb0 0x7ff6680a6070 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.443+0000 7ff683d1e640 1 --2- 172.21.15.28:0/1822876961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff67c100ea0 0x7ff67c06adb0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.443+0000 7ff683d1e640 1 -- 172.21.15.28:0/1822876961 >> 172.21.15.28:0/1822876961 conn(0x7ff67c0fc410 msgr2=0x7ff67c105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:59.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.443+0000 7ff683d1e640 1 -- 172.21.15.28:0/1822876961 shutdown_connections 2024-04-21T08:50:59.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.443+0000 7ff683d1e640 1 -- 172.21.15.28:0/1822876961 wait complete. 2024-04-21T08:50:59.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:59 smithi028 ceph-mon[28003]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:50:59.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:59 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3535146579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:59.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:59 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3195586186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:50:59.454 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:50:59 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2384637785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:50:59.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:50:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.560+0000 7fbf5a39f640 1 -- 172.21.15.28:0/365526481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf54107d50 msgr2=0x7fbf5410a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.560+0000 7fbf5a39f640 1 --2- 172.21.15.28:0/365526481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf54107d50 0x7fbf5410a140 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c009ca0 tx=0x7fbf4c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.560+0000 7fbf5a39f640 1 -- 172.21.15.28:0/365526481 shutdown_connections 2024-04-21T08:50:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.560+0000 7fbf5a39f640 1 --2- 172.21.15.28:0/365526481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf54107d50 0x7fbf5410a140 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.560+0000 7fbf5a39f640 1 --2- 172.21.15.28:0/365526481 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf54105420 0x7fbf54107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.560+0000 7fbf5a39f640 1 --2- 172.21.15.28:0/365526481 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf54102af0 0x7fbf54104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.560+0000 7fbf5a39f640 1 -- 172.21.15.28:0/365526481 >> 172.21.15.28:0/365526481 conn(0x7fbf540fc4a0 msgr2=0x7fbf540fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.560+0000 7fbf5a39f640 1 -- 172.21.15.28:0/365526481 shutdown_connections 2024-04-21T08:50:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.560+0000 7fbf5a39f640 1 -- 172.21.15.28:0/365526481 wait complete. 2024-04-21T08:50:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf5a39f640 1 Processor -- start 2024-04-21T08:50:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf5a39f640 1 -- start start 2024-04-21T08:50:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf5a39f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf54102af0 0x7fbf54198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf5a39f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf54105420 0x7fbf54198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf5a39f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf54107d50 0x7fbf5419f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf5a39f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5410dae0 con 0x7fbf54102af0 2024-04-21T08:50:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf5a39f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5410d960 con 0x7fbf54105420 2024-04-21T08:50:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf5a39f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5410dc60 con 0x7fbf54107d50 2024-04-21T08:50:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf53fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf54102af0 0x7fbf54198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf53fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf54102af0 0x7fbf54198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37566/0 (socket says 172.21.15.28:37566) 2024-04-21T08:50:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf53fff640 1 -- 172.21.15.28:0/1339927810 learned_addr learned my addr 172.21.15.28:0/1339927810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf5893c640 1 --2- 172.21.15.28:0/1339927810 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf54107d50 0x7fbf5419f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf53fff640 1 -- 172.21.15.28:0/1339927810 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf54107d50 msgr2=0x7fbf5419f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf537fe640 1 --2- 172.21.15.28:0/1339927810 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf54105420 0x7fbf54198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf53fff640 1 --2- 172.21.15.28:0/1339927810 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf54107d50 0x7fbf5419f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf53fff640 1 -- 172.21.15.28:0/1339927810 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf54105420 msgr2=0x7fbf54198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf53fff640 1 --2- 172.21.15.28:0/1339927810 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf54105420 0x7fbf54198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf53fff640 1 -- 172.21.15.28:0/1339927810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf5419fd40 con 0x7fbf54102af0 2024-04-21T08:50:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.561+0000 7fbf537fe640 1 --2- 172.21.15.28:0/1339927810 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf54105420 0x7fbf54198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.562+0000 7fbf53fff640 1 --2- 172.21.15.28:0/1339927810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf54102af0 0x7fbf54198120 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00d770 tx=0x7fbf3c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.562+0000 7fbf517fa640 1 -- 172.21.15.28:0/1339927810 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c017070 con 0x7fbf54102af0 2024-04-21T08:50:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.562+0000 7fbf517fa640 1 -- 172.21.15.28:0/1339927810 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf3c0054d0 con 0x7fbf54102af0 2024-04-21T08:50:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.562+0000 7fbf5a39f640 1 -- 172.21.15.28:0/1339927810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf5419ffd0 con 0x7fbf54102af0 2024-04-21T08:50:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.562+0000 7fbf517fa640 1 -- 172.21.15.28:0/1339927810 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c004c30 con 0x7fbf54102af0 2024-04-21T08:50:59.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.562+0000 7fbf5a39f640 1 -- 172.21.15.28:0/1339927810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf541a0490 con 0x7fbf54102af0 2024-04-21T08:50:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.563+0000 7fbf5a39f640 1 -- 172.21.15.28:0/1339927810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf540fdf90 con 0x7fbf54102af0 2024-04-21T08:50:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.564+0000 7fbf517fa640 1 -- 172.21.15.28:0/1339927810 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbf3c004e10 con 0x7fbf54102af0 2024-04-21T08:50:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.565+0000 7fbf517fa640 1 --2- 172.21.15.28:0/1339927810 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbf400a3c30 0x7fbf400a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.565+0000 7fbf537fe640 1 --2- 172.21.15.28:0/1339927810 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbf400a3c30 0x7fbf400a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.565+0000 7fbf517fa640 1 -- 172.21.15.28:0/1339927810 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7fbf3c0d0310 con 0x7fbf54102af0 2024-04-21T08:50:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.565+0000 7fbf537fe640 1 --2- 172.21.15.28:0/1339927810 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbf400a3c30 0x7fbf400a60f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbf54199630 tx=0x7fbf44005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:59.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.570+0000 7fbf517fa640 1 -- 172.21.15.28:0/1339927810 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf3c09c8d0 con 0x7fbf54102af0 2024-04-21T08:50:59.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:59 smithi152 ceph-mon[31401]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:50:59.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:59 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3535146579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:59.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:59 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3195586186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:50:59.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:50:59 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2384637785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:50:59.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:59 smithi171 ceph-mon[31358]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:50:59.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:59 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3535146579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:50:59.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:59 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3195586186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:50:59.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:50:59 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2384637785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:50:59.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.756+0000 7fbf5a39f640 1 -- 172.21.15.28:0/1339927810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbf54062bf0 con 0x7fbf54102af0 2024-04-21T08:50:59.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.756+0000 7fbf517fa640 1 -- 172.21.15.28:0/1339927810 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fbf3c09c1d0 con 0x7fbf54102af0 2024-04-21T08:50:59.757 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:50:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.758+0000 7fbf5a39f640 1 -- 172.21.15.28:0/1339927810 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbf400a3c30 msgr2=0x7fbf400a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.758+0000 7fbf5a39f640 1 --2- 172.21.15.28:0/1339927810 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbf400a3c30 0x7fbf400a60f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbf54199630 tx=0x7fbf44005d90 comp rx=0 tx=0).stop 2024-04-21T08:50:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.758+0000 7fbf5a39f640 1 -- 172.21.15.28:0/1339927810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf54102af0 msgr2=0x7fbf54198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.758+0000 7fbf5a39f640 1 --2- 172.21.15.28:0/1339927810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf54102af0 0x7fbf54198120 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00d770 tx=0x7fbf3c00dc30 comp rx=0 tx=0).stop 2024-04-21T08:50:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.759+0000 7fbf5a39f640 1 -- 172.21.15.28:0/1339927810 shutdown_connections 2024-04-21T08:50:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.759+0000 7fbf5a39f640 1 --2- 172.21.15.28:0/1339927810 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf54107d50 0x7fbf5419f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.759+0000 7fbf5a39f640 1 --2- 172.21.15.28:0/1339927810 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf54105420 0x7fbf54198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.759+0000 7fbf5a39f640 1 --2- 172.21.15.28:0/1339927810 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fbf400a3c30 0x7fbf400a60f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.759+0000 7fbf5a39f640 1 --2- 172.21.15.28:0/1339927810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf54102af0 0x7fbf54198120 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.759+0000 7fbf5a39f640 1 -- 172.21.15.28:0/1339927810 >> 172.21.15.28:0/1339927810 conn(0x7fbf540fc4a0 msgr2=0x7fbf54103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.759+0000 7fbf5a39f640 1 -- 172.21.15.28:0/1339927810 shutdown_connections 2024-04-21T08:50:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.759+0000 7fbf5a39f640 1 -- 172.21.15.28:0/1339927810 wait complete. 2024-04-21T08:50:59.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:50:59.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.874+0000 7f3ad8ee0640 1 -- 172.21.15.28:0/3014821758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad4103b50 msgr2=0x7f3ad410a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.874+0000 7f3ad8ee0640 1 --2- 172.21.15.28:0/3014821758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad4103b50 0x7f3ad410a3e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009ca0 tx=0x7f3acc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.874+0000 7f3ad8ee0640 1 -- 172.21.15.28:0/3014821758 shutdown_connections 2024-04-21T08:50:59.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.874+0000 7f3ad8ee0640 1 --2- 172.21.15.28:0/3014821758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad4103b50 0x7f3ad410a3e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.874+0000 7f3ad8ee0640 1 --2- 172.21.15.28:0/3014821758 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ad4103190 0x7f3ad4103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.874+0000 7f3ad8ee0640 1 --2- 172.21.15.28:0/3014821758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ad4101fa0 0x7f3ad41023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.874+0000 7f3ad8ee0640 1 -- 172.21.15.28:0/3014821758 >> 172.21.15.28:0/3014821758 conn(0x7f3ad40fd530 msgr2=0x7f3ad40ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.874+0000 7f3ad8ee0640 1 -- 172.21.15.28:0/3014821758 shutdown_connections 2024-04-21T08:50:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.874+0000 7f3ad8ee0640 1 -- 172.21.15.28:0/3014821758 wait complete. 2024-04-21T08:50:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad8ee0640 1 Processor -- start 2024-04-21T08:50:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad8ee0640 1 -- start start 2024-04-21T08:50:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad8ee0640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ad4101fa0 0x7f3ad4075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad8ee0640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ad4103190 0x7f3ad4074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad8ee0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad4103b50 0x7f3ad4074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad8ee0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad410dae0 con 0x7f3ad4103b50 2024-04-21T08:50:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad8ee0640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad410d960 con 0x7f3ad4101fa0 2024-04-21T08:50:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad8ee0640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad410dc60 con 0x7f3ad4103190 2024-04-21T08:50:59.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad4103b50 0x7f3ad4074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:59.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad4103b50 0x7f3ad4074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37580/0 (socket says 172.21.15.28:37580) 2024-04-21T08:50:59.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad2d9d640 1 -- 172.21.15.28:0/3847786482 learned_addr learned my addr 172.21.15.28:0/3847786482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:50:59.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3acbfff640 1 --2- 172.21.15.28:0/3847786482 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ad4103190 0x7f3ad4074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:59.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad2d9d640 1 -- 172.21.15.28:0/3847786482 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ad4103190 msgr2=0x7f3ad4074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad2d9d640 1 --2- 172.21.15.28:0/3847786482 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ad4103190 0x7f3ad4074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad2d9d640 1 -- 172.21.15.28:0/3847786482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ad4101fa0 msgr2=0x7f3ad4075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:59.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad259c640 1 --2- 172.21.15.28:0/3847786482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ad4101fa0 0x7f3ad4075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:59.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad2d9d640 1 --2- 172.21.15.28:0/3847786482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ad4101fa0 0x7f3ad4075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:59.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3ad2d9d640 1 -- 172.21.15.28:0/3847786482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad4075110 con 0x7f3ad4103b50 2024-04-21T08:50:59.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.875+0000 7f3acbfff640 1 --2- 172.21.15.28:0/3847786482 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ad4103190 0x7f3ad4074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:59.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.876+0000 7f3ad2d9d640 1 --2- 172.21.15.28:0/3847786482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad4103b50 0x7f3ad4074880 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3acc02c700 tx=0x7f3acc0093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:59.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.876+0000 7f3acb7fe640 1 -- 172.21.15.28:0/3847786482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc041020 con 0x7f3ad4103b50 2024-04-21T08:50:59.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.876+0000 7f3acb7fe640 1 -- 172.21.15.28:0/3847786482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3acc034d60 con 0x7f3ad4103b50 2024-04-21T08:50:59.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.876+0000 7f3acb7fe640 1 -- 172.21.15.28:0/3847786482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc039040 con 0x7f3ad4103b50 2024-04-21T08:50:59.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.876+0000 7f3ad8ee0640 1 -- 172.21.15.28:0/3847786482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ad41a21c0 con 0x7f3ad4103b50 2024-04-21T08:50:59.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.876+0000 7f3ad8ee0640 1 -- 172.21.15.28:0/3847786482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ad41a2710 con 0x7f3ad4103b50 2024-04-21T08:50:59.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.877+0000 7f3ad8ee0640 1 -- 172.21.15.28:0/3847786482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ad41092c0 con 0x7f3ad4103b50 2024-04-21T08:50:59.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.878+0000 7f3acb7fe640 1 -- 172.21.15.28:0/3847786482 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3acc044020 con 0x7f3ad4103b50 2024-04-21T08:50:59.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.879+0000 7f3acb7fe640 1 --2- 172.21.15.28:0/3847786482 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3ac00a3bb0 0x7f3ac00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:59.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.879+0000 7f3ad259c640 1 --2- 172.21.15.28:0/3847786482 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3ac00a3bb0 0x7f3ac00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:59.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.879+0000 7f3acb7fe640 1 -- 172.21.15.28:0/3847786482 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f3acc0f2810 con 0x7f3ad4103b50 2024-04-21T08:50:59.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.879+0000 7f3ad259c640 1 --2- 172.21.15.28:0/3847786482 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3ac00a3bb0 0x7f3ac00a6070 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4102ff0 tx=0x7f3ac4005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:59.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:50:59.883+0000 7f3acb7fe640 1 -- 172.21.15.28:0/3847786482 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3acc04b020 con 0x7f3ad4103b50 2024-04-21T08:51:00.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.108+0000 7f3ad8ee0640 1 -- 172.21.15.28:0/3847786482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ad4062b50 con 0x7f3ad4103b50 2024-04-21T08:51:00.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.108+0000 7f3acb7fe640 1 -- 172.21.15.28:0/3847786482 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f3acc0bee30 con 0x7f3ad4103b50 2024-04-21T08:51:00.109 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:00.109 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:49:12.840022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24463,24487,24488]},"id":1}]} 2024-04-21T08:51:00.110 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-21T08:51:00.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.111+0000 7f3ad8ee0640 1 -- 172.21.15.28:0/3847786482 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3ac00a3bb0 msgr2=0x7f3ac00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.111+0000 7f3ad8ee0640 1 --2- 172.21.15.28:0/3847786482 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3ac00a3bb0 0x7f3ac00a6070 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4102ff0 tx=0x7f3ac4005c60 comp rx=0 tx=0).stop 2024-04-21T08:51:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.111+0000 7f3ad8ee0640 1 -- 172.21.15.28:0/3847786482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad4103b50 msgr2=0x7f3ad4074880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.111+0000 7f3ad8ee0640 1 --2- 172.21.15.28:0/3847786482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad4103b50 0x7f3ad4074880 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3acc02c700 tx=0x7f3acc0093f0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.111+0000 7f3ad8ee0640 1 -- 172.21.15.28:0/3847786482 shutdown_connections 2024-04-21T08:51:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.111+0000 7f3ad8ee0640 1 --2- 172.21.15.28:0/3847786482 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3ac00a3bb0 0x7f3ac00a6070 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.111+0000 7f3ad8ee0640 1 --2- 172.21.15.28:0/3847786482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad4103b50 0x7f3ad4074880 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.111+0000 7f3ad8ee0640 1 --2- 172.21.15.28:0/3847786482 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ad4103190 0x7f3ad4074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.111+0000 7f3ad8ee0640 1 --2- 172.21.15.28:0/3847786482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ad4101fa0 0x7f3ad4075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.111+0000 7f3ad8ee0640 1 -- 172.21.15.28:0/3847786482 >> 172.21.15.28:0/3847786482 conn(0x7f3ad40fd530 msgr2=0x7f3ad4106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.111+0000 7f3ad8ee0640 1 -- 172.21.15.28:0/3847786482 shutdown_connections 2024-04-21T08:51:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.111+0000 7f3ad8ee0640 1 -- 172.21.15.28:0/3847786482 wait complete. 2024-04-21T08:51:00.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:51:00.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.224+0000 7f61ffd2b640 1 -- 172.21.15.28:0/105494282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f81029f0 msgr2=0x7f61f8104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.224+0000 7f61ffd2b640 1 --2- 172.21.15.28:0/105494282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f81029f0 0x7f61f8104de0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f61f4009ca0 tx=0x7f61f402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.224+0000 7f61ffd2b640 1 -- 172.21.15.28:0/105494282 shutdown_connections 2024-04-21T08:51:00.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.224+0000 7f61ffd2b640 1 --2- 172.21.15.28:0/105494282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61f8107c50 0x7f61f810a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.224+0000 7f61ffd2b640 1 --2- 172.21.15.28:0/105494282 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61f8105320 0x7f61f8107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.224+0000 7f61ffd2b640 1 --2- 172.21.15.28:0/105494282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f81029f0 0x7f61f8104de0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.225+0000 7f61ffd2b640 1 -- 172.21.15.28:0/105494282 >> 172.21.15.28:0/105494282 conn(0x7f61f80fc3e0 msgr2=0x7f61f80fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.225+0000 7f61ffd2b640 1 -- 172.21.15.28:0/105494282 shutdown_connections 2024-04-21T08:51:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.225+0000 7f61ffd2b640 1 -- 172.21.15.28:0/105494282 wait complete. 2024-04-21T08:51:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.225+0000 7f61ffd2b640 1 Processor -- start 2024-04-21T08:51:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.225+0000 7f61ffd2b640 1 -- start start 2024-04-21T08:51:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.225+0000 7f61ffd2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f81029f0 0x7f61f81981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.225+0000 7f61ffd2b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61f8105320 0x7f61f81986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.225+0000 7f61ffd2b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61f8107c50 0x7f61f819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.225+0000 7f61ffd2b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61f810daa0 con 0x7f61f81029f0 2024-04-21T08:51:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.225+0000 7f61ffd2b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f61f810d920 con 0x7f61f8105320 2024-04-21T08:51:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.225+0000 7f61ffd2b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f61f810dc20 con 0x7f61f8107c50 2024-04-21T08:51:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.225+0000 7f61fdac7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f81029f0 0x7f61f81981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.225+0000 7f61fdac7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f81029f0 0x7f61f81981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37604/0 (socket says 172.21.15.28:37604) 2024-04-21T08:51:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.225+0000 7f61fdac7640 1 -- 172.21.15.28:0/2972741162 learned_addr learned my addr 172.21.15.28:0/2972741162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.226+0000 7f61fd2c6640 1 --2- 172.21.15.28:0/2972741162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61f8105320 0x7f61f81986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.226+0000 7f61fdac7640 1 -- 172.21.15.28:0/2972741162 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61f8107c50 msgr2=0x7f61f819f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.226+0000 7f61fdac7640 1 --2- 172.21.15.28:0/2972741162 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61f8107c50 0x7f61f819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.226+0000 7f61fdac7640 1 -- 172.21.15.28:0/2972741162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61f8105320 msgr2=0x7f61f81986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.226+0000 7f61fdac7640 1 --2- 172.21.15.28:0/2972741162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61f8105320 0x7f61f81986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.226+0000 7f61fdac7640 1 -- 172.21.15.28:0/2972741162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f819fdc0 con 0x7f61f81029f0 2024-04-21T08:51:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.226+0000 7f61fdac7640 1 --2- 172.21.15.28:0/2972741162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f81029f0 0x7f61f81981a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f61f4005e30 tx=0x7f61f4034830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.226+0000 7f61eeffd640 1 -- 172.21.15.28:0/2972741162 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f4039040 con 0x7f61f81029f0 2024-04-21T08:51:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.226+0000 7f61ffd2b640 1 -- 172.21.15.28:0/2972741162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f81a0050 con 0x7f61f81029f0 2024-04-21T08:51:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.226+0000 7f61eeffd640 1 -- 172.21.15.28:0/2972741162 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61f4035c30 con 0x7f61f81029f0 2024-04-21T08:51:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.226+0000 7f61eeffd640 1 -- 172.21.15.28:0/2972741162 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f40042a0 con 0x7f61f81029f0 2024-04-21T08:51:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.226+0000 7f61ffd2b640 1 -- 172.21.15.28:0/2972741162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f81a0540 con 0x7f61f81029f0 2024-04-21T08:51:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.227+0000 7f61ffd2b640 1 -- 172.21.15.28:0/2972741162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61f80fded0 con 0x7f61f81029f0 2024-04-21T08:51:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.228+0000 7f61eeffd640 1 -- 172.21.15.28:0/2972741162 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f61f4046030 con 0x7f61f81029f0 2024-04-21T08:51:00.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.229+0000 7f61eeffd640 1 --2- 172.21.15.28:0/2972741162 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f61e40a3bb0 0x7f61e40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.229+0000 7f61eeffd640 1 -- 172.21.15.28:0/2972741162 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f61f40f2850 con 0x7f61f81029f0 2024-04-21T08:51:00.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.229+0000 7f61fd2c6640 1 --2- 172.21.15.28:0/2972741162 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f61e40a3bb0 0x7f61e40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:00.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.230+0000 7f61fd2c6640 1 --2- 172.21.15.28:0/2972741162 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f61e40a3bb0 0x7f61e40a6070 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f61f81996b0 tx=0x7f61e8005cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:00.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.232+0000 7f61eeffd640 1 -- 172.21.15.28:0/2972741162 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f61f404d060 con 0x7f61f81029f0 2024-04-21T08:51:00.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.420+0000 7f61ffd2b640 1 -- 172.21.15.28:0/2972741162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f61f802d790 con 0x7f61f81029f0 2024-04-21T08:51:00.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.422+0000 7f61eeffd640 1 -- 172.21.15.28:0/2972741162 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24786 (secure 0 0 0) 0x7f61f40beea0 con 0x7f61f81029f0 2024-04-21T08:51:00.423 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:00.423 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:50:37.275823+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:51:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.424+0000 7f61ffd2b640 1 -- 172.21.15.28:0/2972741162 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f61e40a3bb0 msgr2=0x7f61e40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.424+0000 7f61ffd2b640 1 --2- 172.21.15.28:0/2972741162 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f61e40a3bb0 0x7f61e40a6070 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f61f81996b0 tx=0x7f61e8005cc0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.424+0000 7f61ffd2b640 1 -- 172.21.15.28:0/2972741162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f81029f0 msgr2=0x7f61f81981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.424+0000 7f61ffd2b640 1 --2- 172.21.15.28:0/2972741162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f81029f0 0x7f61f81981a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f61f4005e30 tx=0x7f61f4034830 comp rx=0 tx=0).stop 2024-04-21T08:51:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.424+0000 7f61ffd2b640 1 -- 172.21.15.28:0/2972741162 shutdown_connections 2024-04-21T08:51:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.424+0000 7f61ffd2b640 1 --2- 172.21.15.28:0/2972741162 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61f8107c50 0x7f61f819f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.424+0000 7f61ffd2b640 1 --2- 172.21.15.28:0/2972741162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f61f8105320 0x7f61f81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.424+0000 7f61ffd2b640 1 --2- 172.21.15.28:0/2972741162 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f61e40a3bb0 0x7f61e40a6070 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.424+0000 7f61ffd2b640 1 --2- 172.21.15.28:0/2972741162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f81029f0 0x7f61f81981a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.424+0000 7f61ffd2b640 1 -- 172.21.15.28:0/2972741162 >> 172.21.15.28:0/2972741162 conn(0x7f61f80fc3e0 msgr2=0x7f61f8103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.424+0000 7f61ffd2b640 1 -- 172.21.15.28:0/2972741162 shutdown_connections 2024-04-21T08:51:00.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.424+0000 7f61ffd2b640 1 -- 172.21.15.28:0/2972741162 wait complete. 2024-04-21T08:51:00.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:00 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1822876961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:00.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:00 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1339927810' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:00.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:00 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3847786482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:00.437 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:48:43.543799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:48:44.864564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:48:48.747842+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:51:00.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:51:00.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.542+0000 7f78b4681640 1 -- 172.21.15.28:0/2025776756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac103b60 msgr2=0x7f78ac10a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.542+0000 7f78b4681640 1 --2- 172.21.15.28:0/2025776756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac103b60 0x7f78ac10a3f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009ca0 tx=0x7f78a402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.542+0000 7f78b4681640 1 -- 172.21.15.28:0/2025776756 shutdown_connections 2024-04-21T08:51:00.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.542+0000 7f78b4681640 1 --2- 172.21.15.28:0/2025776756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac103b60 0x7f78ac10a3f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.542+0000 7f78b4681640 1 --2- 172.21.15.28:0/2025776756 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78ac1031a0 0x7f78ac103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.542+0000 7f78b4681640 1 --2- 172.21.15.28:0/2025776756 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78ac101fb0 0x7f78ac1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.542+0000 7f78b4681640 1 -- 172.21.15.28:0/2025776756 >> 172.21.15.28:0/2025776756 conn(0x7f78ac0fd520 msgr2=0x7f78ac0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:00.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.542+0000 7f78b4681640 1 -- 172.21.15.28:0/2025776756 shutdown_connections 2024-04-21T08:51:00.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.542+0000 7f78b4681640 1 -- 172.21.15.28:0/2025776756 wait complete. 2024-04-21T08:51:00.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b4681640 1 Processor -- start 2024-04-21T08:51:00.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b4681640 1 -- start start 2024-04-21T08:51:00.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b4681640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac101fb0 0x7f78ac19a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b4681640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78ac1031a0 0x7f78ac19a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b4681640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78ac103b60 0x7f78ac1a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b4681640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac10daf0 con 0x7f78ac101fb0 2024-04-21T08:51:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b4681640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac10d970 con 0x7f78ac1031a0 2024-04-21T08:51:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b4681640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac10dc70 con 0x7f78ac103b60 2024-04-21T08:51:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b241d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac101fb0 0x7f78ac19a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b241d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac101fb0 0x7f78ac19a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37624/0 (socket says 172.21.15.28:37624) 2024-04-21T08:51:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b241d640 1 -- 172.21.15.28:0/1246042226 learned_addr learned my addr 172.21.15.28:0/1246042226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b2c1e640 1 --2- 172.21.15.28:0/1246042226 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78ac103b60 0x7f78ac1a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b241d640 1 -- 172.21.15.28:0/1246042226 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78ac103b60 msgr2=0x7f78ac1a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b241d640 1 --2- 172.21.15.28:0/1246042226 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78ac103b60 0x7f78ac1a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b241d640 1 -- 172.21.15.28:0/1246042226 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78ac1031a0 msgr2=0x7f78ac19a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b241d640 1 --2- 172.21.15.28:0/1246042226 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78ac1031a0 0x7f78ac19a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b241d640 1 -- 172.21.15.28:0/1246042226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac1a1f70 con 0x7f78ac101fb0 2024-04-21T08:51:00.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.543+0000 7f78b241d640 1 --2- 172.21.15.28:0/1246042226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac101fb0 0x7f78ac19a3f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f78a800c970 tx=0x7f78a800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:00.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.544+0000 7f78a37fe640 1 -- 172.21.15.28:0/1246042226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8004220 con 0x7f78ac101fb0 2024-04-21T08:51:00.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.544+0000 7f78a37fe640 1 -- 172.21.15.28:0/1246042226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78a80043c0 con 0x7f78ac101fb0 2024-04-21T08:51:00.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.544+0000 7f78a37fe640 1 -- 172.21.15.28:0/1246042226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a800f450 con 0x7f78ac101fb0 2024-04-21T08:51:00.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.544+0000 7f78b4681640 1 -- 172.21.15.28:0/1246042226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78ac1a2260 con 0x7f78ac101fb0 2024-04-21T08:51:00.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.544+0000 7f78b4681640 1 -- 172.21.15.28:0/1246042226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78ac1a2760 con 0x7f78ac101fb0 2024-04-21T08:51:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.545+0000 7f78b4681640 1 -- 172.21.15.28:0/1246042226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78ac1a2a20 con 0x7f78ac101fb0 2024-04-21T08:51:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.547+0000 7f78a37fe640 1 -- 172.21.15.28:0/1246042226 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f78a8003610 con 0x7f78ac101fb0 2024-04-21T08:51:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.547+0000 7f78a37fe640 1 --2- 172.21.15.28:0/1246042226 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f78980a3c30 0x7f78980a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.547+0000 7f78b1c1c640 1 --2- 172.21.15.28:0/1246042226 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f78980a3c30 0x7f78980a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.547+0000 7f78a37fe640 1 -- 172.21.15.28:0/1246042226 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f78a80d09c0 con 0x7f78ac101fb0 2024-04-21T08:51:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.548+0000 7f78b1c1c640 1 --2- 172.21.15.28:0/1246042226 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f78980a3c30 0x7f78980a60f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f789c005fd0 tx=0x7f789c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.551+0000 7f78a37fe640 1 -- 172.21.15.28:0/1246042226 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f78a8014030 con 0x7f78ac101fb0 2024-04-21T08:51:00.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:00 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1822876961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:00.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:00 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1339927810' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:00.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:00 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3847786482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:00.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:00 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1822876961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:00.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:00 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1339927810' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:00.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:00 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3847786482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:00.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.738+0000 7f78b4681640 1 -- 172.21.15.28:0/1246042226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f78ac1a2d60 con 0x7f78ac101fb0 2024-04-21T08:51:00.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.738+0000 7f78a37fe640 1 -- 172.21.15.28:0/1246042226 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f78a809d060 con 0x7f78ac101fb0 2024-04-21T08:51:00.739 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T08:51:00.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.740+0000 7f78b4681640 1 -- 172.21.15.28:0/1246042226 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f78980a3c30 msgr2=0x7f78980a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.740+0000 7f78b4681640 1 --2- 172.21.15.28:0/1246042226 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f78980a3c30 0x7f78980a60f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f789c005fd0 tx=0x7f789c005850 comp rx=0 tx=0).stop 2024-04-21T08:51:00.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.740+0000 7f78b4681640 1 -- 172.21.15.28:0/1246042226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac101fb0 msgr2=0x7f78ac19a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.740+0000 7f78b4681640 1 --2- 172.21.15.28:0/1246042226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac101fb0 0x7f78ac19a3f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f78a800c970 tx=0x7f78a800ce30 comp rx=0 tx=0).stop 2024-04-21T08:51:00.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.740+0000 7f78b4681640 1 -- 172.21.15.28:0/1246042226 shutdown_connections 2024-04-21T08:51:00.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.740+0000 7f78b4681640 1 --2- 172.21.15.28:0/1246042226 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78ac103b60 0x7f78ac1a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.740+0000 7f78b4681640 1 --2- 172.21.15.28:0/1246042226 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78ac1031a0 0x7f78ac19a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.740+0000 7f78b4681640 1 --2- 172.21.15.28:0/1246042226 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f78980a3c30 0x7f78980a60f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.740+0000 7f78b4681640 1 --2- 172.21.15.28:0/1246042226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78ac101fb0 0x7f78ac19a3f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.740+0000 7f78b4681640 1 -- 172.21.15.28:0/1246042226 >> 172.21.15.28:0/1246042226 conn(0x7f78ac0fd520 msgr2=0x7f78ac074180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:00.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.740+0000 7f78b4681640 1 -- 172.21.15.28:0/1246042226 shutdown_connections 2024-04-21T08:51:00.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.740+0000 7f78b4681640 1 -- 172.21.15.28:0/1246042226 wait complete. 2024-04-21T08:51:00.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:51:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.856+0000 7f805c4c1640 1 -- 172.21.15.28:0/2821710522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8054102a30 msgr2=0x7f8054104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.856+0000 7f805c4c1640 1 --2- 172.21.15.28:0/2821710522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8054102a30 0x7f8054104e20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8044009900 tx=0x7f804402bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.856+0000 7f805c4c1640 1 -- 172.21.15.28:0/2821710522 shutdown_connections 2024-04-21T08:51:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.856+0000 7f805c4c1640 1 --2- 172.21.15.28:0/2821710522 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8054107c90 0x7f805410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.856+0000 7f805c4c1640 1 --2- 172.21.15.28:0/2821710522 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8054105360 0x7f8054107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.856+0000 7f805c4c1640 1 --2- 172.21.15.28:0/2821710522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8054102a30 0x7f8054104e20 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.856+0000 7f805c4c1640 1 -- 172.21.15.28:0/2821710522 >> 172.21.15.28:0/2821710522 conn(0x7f80540fc440 msgr2=0x7f80540fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.857+0000 7f805c4c1640 1 -- 172.21.15.28:0/2821710522 shutdown_connections 2024-04-21T08:51:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.857+0000 7f805c4c1640 1 -- 172.21.15.28:0/2821710522 wait complete. 2024-04-21T08:51:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.857+0000 7f805c4c1640 1 Processor -- start 2024-04-21T08:51:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.857+0000 7f805c4c1640 1 -- start start 2024-04-21T08:51:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.857+0000 7f805c4c1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8054102a30 0x7f8054198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.857+0000 7f805c4c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8054105360 0x7f8054198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.857+0000 7f805c4c1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8054107c90 0x7f805419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.857+0000 7f805c4c1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f805410daf0 con 0x7f8054105360 2024-04-21T08:51:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.857+0000 7f805c4c1640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f805410d970 con 0x7f8054102a30 2024-04-21T08:51:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.857+0000 7f805c4c1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f805410dc70 con 0x7f8054107c90 2024-04-21T08:51:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.857+0000 7f8059a5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8054105360 0x7f8054198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.857+0000 7f8059a5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8054105360 0x7f8054198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37642/0 (socket says 172.21.15.28:37642) 2024-04-21T08:51:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.857+0000 7f8059a5c640 1 -- 172.21.15.28:0/3303798122 learned_addr learned my addr 172.21.15.28:0/3303798122 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.858+0000 7f805a25d640 1 --2- 172.21.15.28:0/3303798122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8054102a30 0x7f8054198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.858+0000 7f805aa5e640 1 --2- 172.21.15.28:0/3303798122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8054107c90 0x7f805419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.858+0000 7f8059a5c640 1 -- 172.21.15.28:0/3303798122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8054107c90 msgr2=0x7f805419f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.858+0000 7f8059a5c640 1 --2- 172.21.15.28:0/3303798122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8054107c90 0x7f805419f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.858+0000 7f8059a5c640 1 -- 172.21.15.28:0/3303798122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8054102a30 msgr2=0x7f8054198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.858+0000 7f8059a5c640 1 --2- 172.21.15.28:0/3303798122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8054102a30 0x7f8054198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.858+0000 7f8059a5c640 1 -- 172.21.15.28:0/3303798122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805419fe70 con 0x7f8054105360 2024-04-21T08:51:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.858+0000 7f805a25d640 1 --2- 172.21.15.28:0/3303798122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8054102a30 0x7f8054198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.858+0000 7f8059a5c640 1 --2- 172.21.15.28:0/3303798122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8054105360 0x7f8054198790 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f805000d770 tx=0x7f805000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.858+0000 7f804b7fe640 1 -- 172.21.15.28:0/3303798122 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050017070 con 0x7f8054105360 2024-04-21T08:51:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.858+0000 7f805c4c1640 1 -- 172.21.15.28:0/3303798122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80541a0160 con 0x7f8054105360 2024-04-21T08:51:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.858+0000 7f804b7fe640 1 -- 172.21.15.28:0/3303798122 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8050009c30 con 0x7f8054105360 2024-04-21T08:51:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.858+0000 7f804b7fe640 1 -- 172.21.15.28:0/3303798122 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805000f620 con 0x7f8054105360 2024-04-21T08:51:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.858+0000 7f805c4c1640 1 -- 172.21.15.28:0/3303798122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80541a0700 con 0x7f8054105360 2024-04-21T08:51:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.859+0000 7f805c4c1640 1 -- 172.21.15.28:0/3303798122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80540fdf30 con 0x7f8054105360 2024-04-21T08:51:00.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.860+0000 7f804b7fe640 1 -- 172.21.15.28:0/3303798122 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f805001a070 con 0x7f8054105360 2024-04-21T08:51:00.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.861+0000 7f804b7fe640 1 --2- 172.21.15.28:0/3303798122 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80400a3bb0 0x7f80400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:00.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.861+0000 7f805a25d640 1 --2- 172.21.15.28:0/3303798122 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80400a3bb0 0x7f80400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:00.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.861+0000 7f804b7fe640 1 -- 172.21.15.28:0/3303798122 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f80500d0400 con 0x7f8054105360 2024-04-21T08:51:00.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.861+0000 7f805a25d640 1 --2- 172.21.15.28:0/3303798122 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80400a3bb0 0x7f80400a6070 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f80440096c0 tx=0x7f8044000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:00.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:00.864+0000 7f804b7fe640 1 -- 172.21.15.28:0/3303798122 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8050009de0 con 0x7f8054105360 2024-04-21T08:51:01.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.051+0000 7f805c4c1640 1 -- 172.21.15.28:0/3303798122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f805402d7e0 con 0x7f8054105360 2024-04-21T08:51:01.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.052+0000 7f804b7fe640 1 -- 172.21.15.28:0/3303798122 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f805009c9c0 con 0x7f8054105360 2024-04-21T08:51:01.053 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:51:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.054+0000 7f805c4c1640 1 -- 172.21.15.28:0/3303798122 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80400a3bb0 msgr2=0x7f80400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.054+0000 7f805c4c1640 1 --2- 172.21.15.28:0/3303798122 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80400a3bb0 0x7f80400a6070 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f80440096c0 tx=0x7f8044000bc0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.054+0000 7f805c4c1640 1 -- 172.21.15.28:0/3303798122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8054105360 msgr2=0x7f8054198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.054+0000 7f805c4c1640 1 --2- 172.21.15.28:0/3303798122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8054105360 0x7f8054198790 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f805000d770 tx=0x7f805000dc30 comp rx=0 tx=0).stop 2024-04-21T08:51:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.054+0000 7f805c4c1640 1 -- 172.21.15.28:0/3303798122 shutdown_connections 2024-04-21T08:51:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.054+0000 7f805c4c1640 1 --2- 172.21.15.28:0/3303798122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8054107c90 0x7f805419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.054+0000 7f805c4c1640 1 --2- 172.21.15.28:0/3303798122 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f80400a3bb0 0x7f80400a6070 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.054+0000 7f805c4c1640 1 --2- 172.21.15.28:0/3303798122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8054105360 0x7f8054198790 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.054+0000 7f805c4c1640 1 --2- 172.21.15.28:0/3303798122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8054102a30 0x7f8054198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.054+0000 7f805c4c1640 1 -- 172.21.15.28:0/3303798122 >> 172.21.15.28:0/3303798122 conn(0x7f80540fc440 msgr2=0x7f8054103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.054+0000 7f805c4c1640 1 -- 172.21.15.28:0/3303798122 shutdown_connections 2024-04-21T08:51:01.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.054+0000 7f805c4c1640 1 -- 172.21.15.28:0/3303798122 wait complete. 2024-04-21T08:51:01.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:51:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.170+0000 7f62344f4640 1 -- 172.21.15.28:0/2345610140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f622c103b60 msgr2=0x7f622c10a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.170+0000 7f62344f4640 1 --2- 172.21.15.28:0/2345610140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f622c103b60 0x7f622c10a3f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f621c009ca0 tx=0x7f621c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.170+0000 7f62344f4640 1 -- 172.21.15.28:0/2345610140 shutdown_connections 2024-04-21T08:51:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.170+0000 7f62344f4640 1 --2- 172.21.15.28:0/2345610140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f622c103b60 0x7f622c10a3f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.170+0000 7f62344f4640 1 --2- 172.21.15.28:0/2345610140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f622c1031a0 0x7f622c103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.170+0000 7f62344f4640 1 --2- 172.21.15.28:0/2345610140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f622c101fb0 0x7f622c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.170+0000 7f62344f4640 1 -- 172.21.15.28:0/2345610140 >> 172.21.15.28:0/2345610140 conn(0x7f622c0fd520 msgr2=0x7f622c0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.170+0000 7f62344f4640 1 -- 172.21.15.28:0/2345610140 shutdown_connections 2024-04-21T08:51:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.170+0000 7f62344f4640 1 -- 172.21.15.28:0/2345610140 wait complete. 2024-04-21T08:51:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.170+0000 7f62344f4640 1 Processor -- start 2024-04-21T08:51:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.170+0000 7f62344f4640 1 -- start start 2024-04-21T08:51:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f62344f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f622c101fb0 0x7f622c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f62344f4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f622c1031a0 0x7f622c1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f62344f4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f622c103b60 0x7f622c19f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f62344f4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f622c10dab0 con 0x7f622c101fb0 2024-04-21T08:51:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f62344f4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f622c10d930 con 0x7f622c103b60 2024-04-21T08:51:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f62344f4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f622c10dc30 con 0x7f622c1031a0 2024-04-21T08:51:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f6232290640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f622c101fb0 0x7f622c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f6232290640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f622c101fb0 0x7f622c198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37658/0 (socket says 172.21.15.28:37658) 2024-04-21T08:51:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f6232290640 1 -- 172.21.15.28:0/2720815512 learned_addr learned my addr 172.21.15.28:0/2720815512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f6231a8f640 1 --2- 172.21.15.28:0/2720815512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f622c1031a0 0x7f622c1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f6232a91640 1 --2- 172.21.15.28:0/2720815512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f622c103b60 0x7f622c19f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f6232290640 1 -- 172.21.15.28:0/2720815512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f622c1031a0 msgr2=0x7f622c1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f6232290640 1 --2- 172.21.15.28:0/2720815512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f622c1031a0 0x7f622c1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f6232290640 1 -- 172.21.15.28:0/2720815512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f622c103b60 msgr2=0x7f622c19f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f6232290640 1 --2- 172.21.15.28:0/2720815512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f622c103b60 0x7f622c19f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f6232290640 1 -- 172.21.15.28:0/2720815512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622c19fcb0 con 0x7f622c101fb0 2024-04-21T08:51:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f6232290640 1 --2- 172.21.15.28:0/2720815512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f622c101fb0 0x7f622c198090 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f622400c970 tx=0x7f622400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f622b7fe640 1 -- 172.21.15.28:0/2720815512 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6224007c10 con 0x7f622c101fb0 2024-04-21T08:51:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f622b7fe640 1 -- 172.21.15.28:0/2720815512 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6224007db0 con 0x7f622c101fb0 2024-04-21T08:51:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f622b7fe640 1 -- 172.21.15.28:0/2720815512 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622400f870 con 0x7f622c101fb0 2024-04-21T08:51:01.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.171+0000 7f62344f4640 1 -- 172.21.15.28:0/2720815512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f622c19ffa0 con 0x7f622c101fb0 2024-04-21T08:51:01.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.172+0000 7f62344f4640 1 -- 172.21.15.28:0/2720815512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f622c1a0460 con 0x7f622c101fb0 2024-04-21T08:51:01.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.172+0000 7f62344f4640 1 -- 172.21.15.28:0/2720815512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f622c1092d0 con 0x7f622c101fb0 2024-04-21T08:51:01.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.174+0000 7f622b7fe640 1 -- 172.21.15.28:0/2720815512 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f622400fa10 con 0x7f622c101fb0 2024-04-21T08:51:01.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.174+0000 7f622b7fe640 1 --2- 172.21.15.28:0/2720815512 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f62140a3c30 0x7f62140a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.174+0000 7f6231a8f640 1 --2- 172.21.15.28:0/2720815512 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f62140a3c30 0x7f62140a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:01.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.174+0000 7f622b7fe640 1 -- 172.21.15.28:0/2720815512 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f62240d0b60 con 0x7f622c101fb0 2024-04-21T08:51:01.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.175+0000 7f6231a8f640 1 --2- 172.21.15.28:0/2720815512 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f62140a3c30 0x7f62140a60f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6220005e10 tx=0x7f6220005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:01.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.179+0000 7f622b7fe640 1 -- 172.21.15.28:0/2720815512 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f622409d210 con 0x7f622c101fb0 2024-04-21T08:51:01.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.364+0000 7f62344f4640 1 -- 172.21.15.28:0/2720815512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f622c062c50 con 0x7f622c101fb0 2024-04-21T08:51:01.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.365+0000 7f622b7fe640 1 -- 172.21.15.28:0/2720815512 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f622409cb10 con 0x7f622c101fb0 2024-04-21T08:51:01.366 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-21T08:51:01.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.367+0000 7f62344f4640 1 -- 172.21.15.28:0/2720815512 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f62140a3c30 msgr2=0x7f62140a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.367+0000 7f62344f4640 1 --2- 172.21.15.28:0/2720815512 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f62140a3c30 0x7f62140a60f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6220005e10 tx=0x7f6220005da0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.367+0000 7f62344f4640 1 -- 172.21.15.28:0/2720815512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f622c101fb0 msgr2=0x7f622c198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.367+0000 7f62344f4640 1 --2- 172.21.15.28:0/2720815512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f622c101fb0 0x7f622c198090 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f622400c970 tx=0x7f622400ce30 comp rx=0 tx=0).stop 2024-04-21T08:51:01.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.367+0000 7f62344f4640 1 -- 172.21.15.28:0/2720815512 shutdown_connections 2024-04-21T08:51:01.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.367+0000 7f62344f4640 1 --2- 172.21.15.28:0/2720815512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f622c103b60 0x7f622c19f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.367+0000 7f62344f4640 1 --2- 172.21.15.28:0/2720815512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f622c1031a0 0x7f622c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.367+0000 7f62344f4640 1 --2- 172.21.15.28:0/2720815512 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f62140a3c30 0x7f62140a60f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.367+0000 7f62344f4640 1 --2- 172.21.15.28:0/2720815512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f622c101fb0 0x7f622c198090 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.367+0000 7f62344f4640 1 -- 172.21.15.28:0/2720815512 >> 172.21.15.28:0/2720815512 conn(0x7f622c0fd520 msgr2=0x7f622c1064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:01.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.367+0000 7f62344f4640 1 -- 172.21.15.28:0/2720815512 shutdown_connections 2024-04-21T08:51:01.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.367+0000 7f62344f4640 1 -- 172.21.15.28:0/2720815512 wait complete. 2024-04-21T08:51:01.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:51:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.483+0000 7f4f9cd75640 1 -- 172.21.15.28:0/469570302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f98102a00 msgr2=0x7f4f98109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.483+0000 7f4f9cd75640 1 --2- 172.21.15.28:0/469570302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f98102a00 0x7f4f98109320 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4f90009900 tx=0x7f4f9002bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.483+0000 7f4f9cd75640 1 -- 172.21.15.28:0/469570302 shutdown_connections 2024-04-21T08:51:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.483+0000 7f4f9cd75640 1 --2- 172.21.15.28:0/469570302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f98102a00 0x7f4f98109320 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.483+0000 7f4f9cd75640 1 --2- 172.21.15.28:0/469570302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f98102040 0x7f4f981024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.483+0000 7f4f9cd75640 1 --2- 172.21.15.28:0/469570302 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f98100e60 0x7f4f98101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.483+0000 7f4f9cd75640 1 -- 172.21.15.28:0/469570302 >> 172.21.15.28:0/469570302 conn(0x7f4f980fc420 msgr2=0x7f4f980fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.483+0000 7f4f9cd75640 1 -- 172.21.15.28:0/469570302 shutdown_connections 2024-04-21T08:51:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.483+0000 7f4f9cd75640 1 -- 172.21.15.28:0/469570302 wait complete. 2024-04-21T08:51:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f9cd75640 1 Processor -- start 2024-04-21T08:51:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f9cd75640 1 -- start start 2024-04-21T08:51:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f9cd75640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f98100e60 0x7f4f98198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f9cd75640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f98102040 0x7f4f98198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f9cd75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f98102a00 0x7f4f9819f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f9cd75640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9810db90 con 0x7f4f98102a00 2024-04-21T08:51:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f9cd75640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9810da10 con 0x7f4f98102040 2024-04-21T08:51:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f9cd75640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9810dd10 con 0x7f4f98100e60 2024-04-21T08:51:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f96d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f98102a00 0x7f4f9819f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f96d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f98102a00 0x7f4f9819f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37686/0 (socket says 172.21.15.28:37686) 2024-04-21T08:51:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f96d9d640 1 -- 172.21.15.28:0/3249271336 learned_addr learned my addr 172.21.15.28:0/3249271336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f9659c640 1 --2- 172.21.15.28:0/3249271336 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f98100e60 0x7f4f98198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f96d9d640 1 -- 172.21.15.28:0/3249271336 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f98100e60 msgr2=0x7f4f98198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f95d9b640 1 --2- 172.21.15.28:0/3249271336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f98102040 0x7f4f98198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f96d9d640 1 --2- 172.21.15.28:0/3249271336 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f98100e60 0x7f4f98198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f96d9d640 1 -- 172.21.15.28:0/3249271336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f98102040 msgr2=0x7f4f98198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f96d9d640 1 --2- 172.21.15.28:0/3249271336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f98102040 0x7f4f98198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f96d9d640 1 -- 172.21.15.28:0/3249271336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9819fd60 con 0x7f4f98102a00 2024-04-21T08:51:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.484+0000 7f4f95d9b640 1 --2- 172.21.15.28:0/3249271336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f98102040 0x7f4f98198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.485+0000 7f4f96d9d640 1 --2- 172.21.15.28:0/3249271336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f98102a00 0x7f4f9819f6f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4f90036040 tx=0x7f4f90004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.485+0000 7f4f877fe640 1 -- 172.21.15.28:0/3249271336 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90043020 con 0x7f4f98102a00 2024-04-21T08:51:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.485+0000 7f4f877fe640 1 -- 172.21.15.28:0/3249271336 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f90004490 con 0x7f4f98102a00 2024-04-21T08:51:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.485+0000 7f4f9cd75640 1 -- 172.21.15.28:0/3249271336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f9819fff0 con 0x7f4f98102a00 2024-04-21T08:51:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.485+0000 7f4f877fe640 1 -- 172.21.15.28:0/3249271336 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90003d10 con 0x7f4f98102a00 2024-04-21T08:51:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.485+0000 7f4f9cd75640 1 -- 172.21.15.28:0/3249271336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f981a04b0 con 0x7f4f98102a00 2024-04-21T08:51:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.486+0000 7f4f9cd75640 1 -- 172.21.15.28:0/3249271336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f981081a0 con 0x7f4f98102a00 2024-04-21T08:51:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.487+0000 7f4f877fe640 1 -- 172.21.15.28:0/3249271336 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4f90046020 con 0x7f4f98102a00 2024-04-21T08:51:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.488+0000 7f4f877fe640 1 --2- 172.21.15.28:0/3249271336 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f800a3bb0 0x7f4f800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.488+0000 7f4f877fe640 1 -- 172.21.15.28:0/3249271336 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f4f900f2c20 con 0x7f4f98102a00 2024-04-21T08:51:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.488+0000 7f4f9659c640 1 --2- 172.21.15.28:0/3249271336 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f800a3bb0 0x7f4f800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.488+0000 7f4f9659c640 1 --2- 172.21.15.28:0/3249271336 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f800a3bb0 0x7f4f800a6070 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4f98101bb0 tx=0x7f4f88009460 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.492+0000 7f4f877fe640 1 -- 172.21.15.28:0/3249271336 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f9004d020 con 0x7f4f98102a00 2024-04-21T08:51:01.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:01 smithi152 ceph-mon[31401]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:51:01.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:01 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2972741162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:01.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:01 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1246042226' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:01.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:01 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3303798122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:01.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:01 smithi028 ceph-mon[28003]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:51:01.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:01 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2972741162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:01.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:01 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1246042226' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:01.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:01 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3303798122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:01.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:01 smithi171 ceph-mon[31358]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-21T08:51:01.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:01 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2972741162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:01.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:01 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1246042226' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:01.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:01 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3303798122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:01.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.678+0000 7f4f9cd75640 1 -- 172.21.15.28:0/3249271336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4f98062bf0 con 0x7f4f98102a00 2024-04-21T08:51:01.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.679+0000 7f4f877fe640 1 -- 172.21.15.28:0/3249271336 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f4f900bf240 con 0x7f4f98102a00 2024-04-21T08:51:01.680 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:51:01.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.681+0000 7f4f9cd75640 1 -- 172.21.15.28:0/3249271336 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f800a3bb0 msgr2=0x7f4f800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.681+0000 7f4f9cd75640 1 --2- 172.21.15.28:0/3249271336 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f800a3bb0 0x7f4f800a6070 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4f98101bb0 tx=0x7f4f88009460 comp rx=0 tx=0).stop 2024-04-21T08:51:01.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.681+0000 7f4f9cd75640 1 -- 172.21.15.28:0/3249271336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f98102a00 msgr2=0x7f4f9819f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.681+0000 7f4f9cd75640 1 --2- 172.21.15.28:0/3249271336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f98102a00 0x7f4f9819f6f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4f90036040 tx=0x7f4f90004270 comp rx=0 tx=0).stop 2024-04-21T08:51:01.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.681+0000 7f4f9cd75640 1 -- 172.21.15.28:0/3249271336 shutdown_connections 2024-04-21T08:51:01.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.681+0000 7f4f9cd75640 1 --2- 172.21.15.28:0/3249271336 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f4f800a3bb0 0x7f4f800a6070 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.681+0000 7f4f9cd75640 1 --2- 172.21.15.28:0/3249271336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f98102a00 0x7f4f9819f6f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.681+0000 7f4f9cd75640 1 --2- 172.21.15.28:0/3249271336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f98102040 0x7f4f98198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.681+0000 7f4f9cd75640 1 --2- 172.21.15.28:0/3249271336 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f98100e60 0x7f4f98198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.681+0000 7f4f9cd75640 1 -- 172.21.15.28:0/3249271336 >> 172.21.15.28:0/3249271336 conn(0x7f4f980fc420 msgr2=0x7f4f98105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:01.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.681+0000 7f4f9cd75640 1 -- 172.21.15.28:0/3249271336 shutdown_connections 2024-04-21T08:51:01.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.681+0000 7f4f9cd75640 1 -- 172.21.15.28:0/3249271336 wait complete. 2024-04-21T08:51:01.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:51:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.795+0000 7f996c0d2640 1 -- 172.21.15.28:0/1775587873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9964100260 msgr2=0x7f9964100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.795+0000 7f996c0d2640 1 --2- 172.21.15.28:0/1775587873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9964100260 0x7f9964100680 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f995c009930 tx=0x7f995c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.795+0000 7f996c0d2640 1 -- 172.21.15.28:0/1775587873 shutdown_connections 2024-04-21T08:51:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.795+0000 7f996c0d2640 1 --2- 172.21.15.28:0/1775587873 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9964103bf0 0x7f996410a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.795+0000 7f996c0d2640 1 --2- 172.21.15.28:0/1775587873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9964103230 0x7f99641036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.795+0000 7f996c0d2640 1 --2- 172.21.15.28:0/1775587873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9964100260 0x7f9964100680 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.795+0000 7f996c0d2640 1 -- 172.21.15.28:0/1775587873 >> 172.21.15.28:0/1775587873 conn(0x7f99640fbb90 msgr2=0x7f99640fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.795+0000 7f996c0d2640 1 -- 172.21.15.28:0/1775587873 shutdown_connections 2024-04-21T08:51:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.795+0000 7f996c0d2640 1 -- 172.21.15.28:0/1775587873 wait complete. 2024-04-21T08:51:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996c0d2640 1 Processor -- start 2024-04-21T08:51:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996c0d2640 1 -- start start 2024-04-21T08:51:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996c0d2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9964100260 0x7f99641981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996c0d2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9964103230 0x7f9964198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996c0d2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9964103bf0 0x7f996419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996c0d2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f996410db00 con 0x7f9964103230 2024-04-21T08:51:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996c0d2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f996410d980 con 0x7f9964100260 2024-04-21T08:51:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996c0d2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f996410dc80 con 0x7f9964103bf0 2024-04-21T08:51:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996966d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9964103230 0x7f9964198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996966d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9964103230 0x7f9964198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37698/0 (socket says 172.21.15.28:37698) 2024-04-21T08:51:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996966d640 1 -- 172.21.15.28:0/3429408496 learned_addr learned my addr 172.21.15.28:0/3429408496 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:01.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f9969e6e640 1 --2- 172.21.15.28:0/3429408496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9964100260 0x7f99641981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996966d640 1 -- 172.21.15.28:0/3429408496 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9964103bf0 msgr2=0x7f996419f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996966d640 1 --2- 172.21.15.28:0/3429408496 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9964103bf0 0x7f996419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996966d640 1 -- 172.21.15.28:0/3429408496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9964100260 msgr2=0x7f99641981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996966d640 1 --2- 172.21.15.28:0/3429408496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9964100260 0x7f99641981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996966d640 1 -- 172.21.15.28:0/3429408496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f996419fde0 con 0x7f9964103230 2024-04-21T08:51:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f9969e6e640 1 --2- 172.21.15.28:0/3429408496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9964100260 0x7f99641981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.796+0000 7f996966d640 1 --2- 172.21.15.28:0/3429408496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9964103230 0x7f9964198700 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f995800d950 tx=0x7f995800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.797+0000 7f9962ffd640 1 -- 172.21.15.28:0/3429408496 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9958017070 con 0x7f9964103230 2024-04-21T08:51:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.797+0000 7f996c0d2640 1 -- 172.21.15.28:0/3429408496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99641a00d0 con 0x7f9964103230 2024-04-21T08:51:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.797+0000 7f9962ffd640 1 -- 172.21.15.28:0/3429408496 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9958009c30 con 0x7f9964103230 2024-04-21T08:51:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.797+0000 7f9962ffd640 1 -- 172.21.15.28:0/3429408496 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995800f620 con 0x7f9964103230 2024-04-21T08:51:01.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.797+0000 7f996c0d2640 1 -- 172.21.15.28:0/3429408496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99641a0670 con 0x7f9964103230 2024-04-21T08:51:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.798+0000 7f996c0d2640 1 -- 172.21.15.28:0/3429408496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99641093e0 con 0x7f9964103230 2024-04-21T08:51:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.799+0000 7f9962ffd640 1 -- 172.21.15.28:0/3429408496 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9958010460 con 0x7f9964103230 2024-04-21T08:51:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.799+0000 7f9962ffd640 1 --2- 172.21.15.28:0/3429408496 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f994c0a3bb0 0x7f994c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:01.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.800+0000 7f9969e6e640 1 --2- 172.21.15.28:0/3429408496 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f994c0a3bb0 0x7f994c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:01.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.800+0000 7f9962ffd640 1 -- 172.21.15.28:0/3429408496 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f99580d0480 con 0x7f9964103230 2024-04-21T08:51:01.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.800+0000 7f9969e6e640 1 --2- 172.21.15.28:0/3429408496 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f994c0a3bb0 0x7f994c0a6070 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f995c036040 tx=0x7f995c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:01.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:01.803+0000 7f9962ffd640 1 -- 172.21.15.28:0/3429408496 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f995809caa0 con 0x7f9964103230 2024-04-21T08:51:02.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.030+0000 7f996c0d2640 1 -- 172.21.15.28:0/3429408496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f996402d790 con 0x7f9964103230 2024-04-21T08:51:02.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.031+0000 7f9962ffd640 1 -- 172.21.15.28:0/3429408496 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11882 (secure 0 0 0) 0x7f9958009de0 con 0x7f9964103230 2024-04-21T08:51:02.031 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:02.031 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:49:12.840022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24463,24487,24488]},"id":1}]} 2024-04-21T08:51:02.032 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-21T08:51:02.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.033+0000 7f996c0d2640 1 -- 172.21.15.28:0/3429408496 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f994c0a3bb0 msgr2=0x7f994c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.033+0000 7f996c0d2640 1 --2- 172.21.15.28:0/3429408496 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f994c0a3bb0 0x7f994c0a6070 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f995c036040 tx=0x7f995c000bc0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.033+0000 7f996c0d2640 1 -- 172.21.15.28:0/3429408496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9964103230 msgr2=0x7f9964198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.033+0000 7f996c0d2640 1 --2- 172.21.15.28:0/3429408496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9964103230 0x7f9964198700 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f995800d950 tx=0x7f995800de10 comp rx=0 tx=0).stop 2024-04-21T08:51:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.033+0000 7f996c0d2640 1 -- 172.21.15.28:0/3429408496 shutdown_connections 2024-04-21T08:51:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.033+0000 7f996c0d2640 1 --2- 172.21.15.28:0/3429408496 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9964103bf0 0x7f996419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.033+0000 7f996c0d2640 1 --2- 172.21.15.28:0/3429408496 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f994c0a3bb0 0x7f994c0a6070 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.033+0000 7f996c0d2640 1 --2- 172.21.15.28:0/3429408496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9964103230 0x7f9964198700 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.033+0000 7f996c0d2640 1 --2- 172.21.15.28:0/3429408496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9964100260 0x7f99641981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.033+0000 7f996c0d2640 1 -- 172.21.15.28:0/3429408496 >> 172.21.15.28:0/3429408496 conn(0x7f99640fbb90 msgr2=0x7f99641065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.033+0000 7f996c0d2640 1 -- 172.21.15.28:0/3429408496 shutdown_connections 2024-04-21T08:51:02.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.033+0000 7f996c0d2640 1 -- 172.21.15.28:0/3429408496 wait complete. 2024-04-21T08:51:02.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:51:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.148+0000 7fb61e304640 1 -- 172.21.15.28:0/87104660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb618102a10 msgr2=0x7fb618109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.148+0000 7fb61e304640 1 --2- 172.21.15.28:0/87104660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb618102a10 0x7fb618109330 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb610009ca0 tx=0x7fb61002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.148+0000 7fb61e304640 1 -- 172.21.15.28:0/87104660 shutdown_connections 2024-04-21T08:51:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.148+0000 7fb61e304640 1 --2- 172.21.15.28:0/87104660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb618102a10 0x7fb618109330 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.148+0000 7fb61e304640 1 --2- 172.21.15.28:0/87104660 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb618102050 0x7fb6181024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.148+0000 7fb61e304640 1 --2- 172.21.15.28:0/87104660 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb618100e70 0x7fb618101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.148+0000 7fb61e304640 1 -- 172.21.15.28:0/87104660 >> 172.21.15.28:0/87104660 conn(0x7fb6180fc470 msgr2=0x7fb6180fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.148+0000 7fb61e304640 1 -- 172.21.15.28:0/87104660 shutdown_connections 2024-04-21T08:51:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.148+0000 7fb61e304640 1 -- 172.21.15.28:0/87104660 wait complete. 2024-04-21T08:51:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb61e304640 1 Processor -- start 2024-04-21T08:51:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb61e304640 1 -- start start 2024-04-21T08:51:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb61e304640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb618100e70 0x7fb618198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb61e304640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb618102050 0x7fb618198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb61e304640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb618102a10 0x7fb61819f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb61e304640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb61810ca10 con 0x7fb618100e70 2024-04-21T08:51:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb61e304640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb61810c890 con 0x7fb618102050 2024-04-21T08:51:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb61e304640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb61810cb90 con 0x7fb618102a10 2024-04-21T08:51:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb617fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb618100e70 0x7fb618198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb617fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb618100e70 0x7fb618198150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37722/0 (socket says 172.21.15.28:37722) 2024-04-21T08:51:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb617fff640 1 -- 172.21.15.28:0/1849240336 learned_addr learned my addr 172.21.15.28:0/1849240336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb6177fe640 1 --2- 172.21.15.28:0/1849240336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb618102050 0x7fb618198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb617fff640 1 -- 172.21.15.28:0/1849240336 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb618102a10 msgr2=0x7fb61819f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb617fff640 1 --2- 172.21.15.28:0/1849240336 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb618102a10 0x7fb61819f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb617fff640 1 -- 172.21.15.28:0/1849240336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb618102050 msgr2=0x7fb618198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb617fff640 1 --2- 172.21.15.28:0/1849240336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb618102050 0x7fb618198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.149+0000 7fb617fff640 1 -- 172.21.15.28:0/1849240336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb61819fd70 con 0x7fb618100e70 2024-04-21T08:51:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.150+0000 7fb617fff640 1 --2- 172.21.15.28:0/1849240336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb618100e70 0x7fb618198150 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb60800d770 tx=0x7fb60800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.150+0000 7fb6157fa640 1 -- 172.21.15.28:0/1849240336 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb608017070 con 0x7fb618100e70 2024-04-21T08:51:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.150+0000 7fb6157fa640 1 -- 172.21.15.28:0/1849240336 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb608009c30 con 0x7fb618100e70 2024-04-21T08:51:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.150+0000 7fb61e304640 1 -- 172.21.15.28:0/1849240336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6181a0060 con 0x7fb618100e70 2024-04-21T08:51:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.150+0000 7fb6157fa640 1 -- 172.21.15.28:0/1849240336 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb608003cc0 con 0x7fb618100e70 2024-04-21T08:51:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.150+0000 7fb61e304640 1 -- 172.21.15.28:0/1849240336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6181a05b0 con 0x7fb618100e70 2024-04-21T08:51:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.151+0000 7fb61e304640 1 -- 172.21.15.28:0/1849240336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6181a0810 con 0x7fb618100e70 2024-04-21T08:51:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.152+0000 7fb6157fa640 1 -- 172.21.15.28:0/1849240336 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb608003e60 con 0x7fb618100e70 2024-04-21T08:51:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.153+0000 7fb6157fa640 1 --2- 172.21.15.28:0/1849240336 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb6000a3bb0 0x7fb6000a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.153+0000 7fb6177fe640 1 --2- 172.21.15.28:0/1849240336 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb6000a3bb0 0x7fb6000a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.153+0000 7fb6157fa640 1 -- 172.21.15.28:0/1849240336 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7fb6080d0640 con 0x7fb618100e70 2024-04-21T08:51:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.153+0000 7fb6177fe640 1 --2- 172.21.15.28:0/1849240336 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb6000a3bb0 0x7fb6000a6070 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb60c005fd0 tx=0x7fb60c005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.158+0000 7fb6157fa640 1 -- 172.21.15.28:0/1849240336 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb608009de0 con 0x7fb618100e70 2024-04-21T08:51:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.342+0000 7fb61e304640 1 -- 172.21.15.28:0/1849240336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6181a0b00 con 0x7fb618100e70 2024-04-21T08:51:02.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.344+0000 7fb6157fa640 1 -- 172.21.15.28:0/1849240336 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24786 (secure 0 0 0) 0x7fb60809cc60 con 0x7fb618100e70 2024-04-21T08:51:02.347 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:02.347 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:50:37.275823+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:51:02.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.346+0000 7fb61e304640 1 -- 172.21.15.28:0/1849240336 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb6000a3bb0 msgr2=0x7fb6000a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.346+0000 7fb61e304640 1 --2- 172.21.15.28:0/1849240336 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb6000a3bb0 0x7fb6000a6070 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb60c005fd0 tx=0x7fb60c005e20 comp rx=0 tx=0).stop 2024-04-21T08:51:02.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.346+0000 7fb61e304640 1 -- 172.21.15.28:0/1849240336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb618100e70 msgr2=0x7fb618198150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.346+0000 7fb61e304640 1 --2- 172.21.15.28:0/1849240336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb618100e70 0x7fb618198150 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb60800d770 tx=0x7fb60800dc30 comp rx=0 tx=0).stop 2024-04-21T08:51:02.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.346+0000 7fb61e304640 1 -- 172.21.15.28:0/1849240336 shutdown_connections 2024-04-21T08:51:02.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.346+0000 7fb61e304640 1 --2- 172.21.15.28:0/1849240336 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb618102a10 0x7fb61819f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.346+0000 7fb61e304640 1 --2- 172.21.15.28:0/1849240336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb618102050 0x7fb618198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.346+0000 7fb61e304640 1 --2- 172.21.15.28:0/1849240336 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fb6000a3bb0 0x7fb6000a6070 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.346+0000 7fb61e304640 1 --2- 172.21.15.28:0/1849240336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb618100e70 0x7fb618198150 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.346+0000 7fb61e304640 1 -- 172.21.15.28:0/1849240336 >> 172.21.15.28:0/1849240336 conn(0x7fb6180fc470 msgr2=0x7fb6180fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.346+0000 7fb61e304640 1 -- 172.21.15.28:0/1849240336 shutdown_connections 2024-04-21T08:51:02.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.346+0000 7fb61e304640 1 -- 172.21.15.28:0/1849240336 wait complete. 2024-04-21T08:51:02.359 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T08:48:45.911185+0000', 'modified': '2024-04-21T08:49:12.840022+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24488, 'mds_1': 24487, 'mds_2': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24419': {'gid': 24419, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.152:6835/3982885739', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6834', 'nonce': 3982885739}, {'type': 'v1', 'addr': '172.21.15.152:6835', 'nonce': 3982885739}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24452': {'gid': 24452, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.152:6837/1261226890', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6836', 'nonce': 1261226890}, {'type': 'v1', 'addr': '172.21.15.152:6837', 'nonce': 1261226890}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.171:6835/663606808', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6834', 'nonce': 663606808}, {'type': 'v1', 'addr': '172.21.15.171:6835', 'nonce': 663606808}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.171:6837/895402983', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6836', 'nonce': 895402983}, {'type': 'v1', 'addr': '172.21.15.171:6837', 'nonce': 895402983}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.171:6839/3423264764', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6838', 'nonce': 3423264764}, {'type': 'v1', 'addr': '172.21.15.171:6839', 'nonce': 3423264764}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.152:6839/3821365920', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6838', 'nonce': 3821365920}, {'type': 'v1', 'addr': '172.21.15.152:6839', 'nonce': 3821365920}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24488, 'qdb_cluster': [24463, 24487, 24488]} 2024-04-21T08:51:02.359 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-21T08:51:02.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-21T08:51:02.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.461+0000 7f14fef23640 1 -- 172.21.15.28:0/3399896100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f8105410 msgr2=0x7f14f8107800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.461+0000 7f14fef23640 1 --2- 172.21.15.28:0/3399896100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f8105410 0x7f14f8107800 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f14e4009900 tx=0x7f14e402bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:02.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fef23640 1 -- 172.21.15.28:0/3399896100 shutdown_connections 2024-04-21T08:51:02.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fef23640 1 --2- 172.21.15.28:0/3399896100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14f8107d40 0x7f14f810a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fef23640 1 --2- 172.21.15.28:0/3399896100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f8105410 0x7f14f8107800 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fef23640 1 --2- 172.21.15.28:0/3399896100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14f8102ae0 0x7f14f8104ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fef23640 1 -- 172.21.15.28:0/3399896100 >> 172.21.15.28:0/3399896100 conn(0x7f14f80fc410 msgr2=0x7f14f80fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fef23640 1 -- 172.21.15.28:0/3399896100 shutdown_connections 2024-04-21T08:51:02.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fef23640 1 -- 172.21.15.28:0/3399896100 wait complete. 2024-04-21T08:51:02.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fef23640 1 Processor -- start 2024-04-21T08:51:02.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fef23640 1 -- start start 2024-04-21T08:51:02.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fef23640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f8102ae0 0x7f14f81982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fef23640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14f8105410 0x7f14f8198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fef23640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14f8107d40 0x7f14f819f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fef23640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14f810db30 con 0x7f14f8102ae0 2024-04-21T08:51:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fef23640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f14f810d9b0 con 0x7f14f8105410 2024-04-21T08:51:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fef23640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f14f810dcb0 con 0x7f14f8107d40 2024-04-21T08:51:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.462+0000 7f14fccbf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f8102ae0 0x7f14f81982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.463+0000 7f14fccbf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f8102ae0 0x7f14f81982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37740/0 (socket says 172.21.15.28:37740) 2024-04-21T08:51:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.463+0000 7f14fccbf640 1 -- 172.21.15.28:0/3728154192 learned_addr learned my addr 172.21.15.28:0/3728154192 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.463+0000 7f14fccbf640 1 -- 172.21.15.28:0/3728154192 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14f8107d40 msgr2=0x7f14f819f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.463+0000 7f14fccbf640 1 --2- 172.21.15.28:0/3728154192 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14f8107d40 0x7f14f819f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.463+0000 7f14f7fff640 1 --2- 172.21.15.28:0/3728154192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14f8105410 0x7f14f8198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.463+0000 7f14fccbf640 1 -- 172.21.15.28:0/3728154192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14f8105410 msgr2=0x7f14f8198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.463+0000 7f14fccbf640 1 --2- 172.21.15.28:0/3728154192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14f8105410 0x7f14f8198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.463+0000 7f14fccbf640 1 -- 172.21.15.28:0/3728154192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f819fef0 con 0x7f14f8102ae0 2024-04-21T08:51:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.463+0000 7f14fccbf640 1 --2- 172.21.15.28:0/3728154192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f8102ae0 0x7f14f81982d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00d950 tx=0x7f14ec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.463+0000 7f14f5ffb640 1 -- 172.21.15.28:0/3728154192 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec017030 con 0x7f14f8102ae0 2024-04-21T08:51:02.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.463+0000 7f14f5ffb640 1 -- 172.21.15.28:0/3728154192 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14ec004020 con 0x7f14f8102ae0 2024-04-21T08:51:02.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.463+0000 7f14fef23640 1 -- 172.21.15.28:0/3728154192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14f81a01e0 con 0x7f14f8102ae0 2024-04-21T08:51:02.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.463+0000 7f14f5ffb640 1 -- 172.21.15.28:0/3728154192 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0047a0 con 0x7f14f8102ae0 2024-04-21T08:51:02.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.463+0000 7f14fef23640 1 -- 172.21.15.28:0/3728154192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14f81a0780 con 0x7f14f8102ae0 2024-04-21T08:51:02.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.466+0000 7f14f5ffb640 1 -- 172.21.15.28:0/3728154192 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f14ec009a30 con 0x7f14f8102ae0 2024-04-21T08:51:02.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.466+0000 7f1501bec740 1 Processor -- start 2024-04-21T08:51:02.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.466+0000 7f1501bec740 1 -- start start 2024-04-21T08:51:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f1501bec740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe57f250 0x55f4fe57f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f1501bec740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57fc40 0x55f4fe5800c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f1501bec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe5806e0 0x55f4fe5872c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f1501bec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f4fe4dfb10 con 0x55f4fe5806e0 2024-04-21T08:51:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f1501bec740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f4fe47d240 con 0x55f4fe57f250 2024-04-21T08:51:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f1501bec740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f4fe4dbad0 con 0x55f4fe57fc40 2024-04-21T08:51:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f14f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe5806e0 0x55f4fe5872c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f14f5ffb640 1 --2- 172.21.15.28:0/3728154192 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f14e80a3c30 0x7f14e80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f14f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe5806e0 0x55f4fe5872c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37750/0 (socket says 172.21.15.28:37750) 2024-04-21T08:51:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f14f7fff640 1 -- 172.21.15.28:0/2867591651 learned_addr learned my addr 172.21.15.28:0/2867591651 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f14fd4c0640 1 --2- 172.21.15.28:0/3728154192 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f14e80a3c30 0x7f14e80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f14f5ffb640 1 -- 172.21.15.28:0/3728154192 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f14ec0d0300 con 0x7f14f8102ae0 2024-04-21T08:51:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f14f7fff640 1 -- 172.21.15.28:0/2867591651 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57fc40 msgr2=0x55f4fe5800c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f14f7fff640 1 --2- 172.21.15.28:0/2867591651 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57fc40 0x55f4fe5800c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f14f7fff640 1 -- 172.21.15.28:0/2867591651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe57f250 msgr2=0x55f4fe57f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f14f7fff640 1 --2- 172.21.15.28:0/2867591651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe57f250 0x55f4fe57f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f14f7fff640 1 -- 172.21.15.28:0/2867591651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4fe1f22d0 con 0x55f4fe5806e0 2024-04-21T08:51:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f14f7fff640 1 --2- 172.21.15.28:0/2867591651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe5806e0 0x55f4fe5872c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f14e4006010 tx=0x7f14e40095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5700f158a1fc823 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f14fd4c0640 1 --2- 172.21.15.28:0/3728154192 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f14e80a3c30 0x7f14e80a60f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x55f4fe576200 tx=0x7f14f0001710 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.467+0000 7f14e2ffd640 1 -- 172.21.15.28:0/2867591651 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e4043020 con 0x55f4fe5806e0 2024-04-21T08:51:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.468+0000 7f14e2ffd640 1 -- 172.21.15.28:0/2867591651 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14e4004040 con 0x55f4fe5806e0 2024-04-21T08:51:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.468+0000 7f14e2ffd640 1 -- 172.21.15.28:0/2867591651 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e40055d0 con 0x55f4fe5806e0 2024-04-21T08:51:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.468+0000 7f1501bec740 1 -- 172.21.15.28:0/2867591651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe5806e0 msgr2=0x55f4fe5872c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.468+0000 7f1501bec740 1 --2- 172.21.15.28:0/2867591651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe5806e0 0x55f4fe5872c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f14e4006010 tx=0x7f14e40095e0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.468+0000 7f1501bec740 1 -- 172.21.15.28:0/2867591651 shutdown_connections 2024-04-21T08:51:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.468+0000 7f1501bec740 1 --2- 172.21.15.28:0/2867591651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe5806e0 0x55f4fe5872c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.468+0000 7f1501bec740 1 --2- 172.21.15.28:0/2867591651 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57fc40 0x55f4fe5800c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.468+0000 7f1501bec740 1 --2- 172.21.15.28:0/2867591651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe57f250 0x55f4fe57f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.468+0000 7f1501bec740 1 -- 172.21.15.28:0/2867591651 >> 172.21.15.28:0/2867591651 conn(0x55f4fe576f40 msgr2=0x55f4fe575490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.468+0000 7f1501bec740 1 -- 172.21.15.28:0/2867591651 shutdown_connections 2024-04-21T08:51:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.468+0000 7f1501bec740 1 -- 172.21.15.28:0/2867591651 wait complete. 2024-04-21T08:51:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.468+0000 7f1501bec740 1 -- start start 2024-04-21T08:51:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f1501bec740 1 Processor -- start 2024-04-21T08:51:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f1501bec740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57f250 0x55f4fe589290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f1501bec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe57fc40 0x55f4fe5897d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f1501bec740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe5806e0 0x55f4fe590810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f1501bec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f4fe587860 con 0x55f4fe57fc40 2024-04-21T08:51:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f1501bec740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f4fe468190 con 0x55f4fe5806e0 2024-04-21T08:51:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f1501bec740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f4fe5879e0 con 0x55f4fe57f250 2024-04-21T08:51:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f1501bec740 10 client.0 _ll_register_callbacks cb 0x55f4fe4638b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f14f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe57fc40 0x55f4fe5897d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f14f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe57fc40 0x55f4fe5897d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37764/0 (socket says 172.21.15.28:37764) 2024-04-21T08:51:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f14f7fff640 1 -- 172.21.15.28:0/3959982059 learned_addr learned my addr 172.21.15.28:0/3959982059 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f14fccbf640 1 --2- 172.21.15.28:0/3959982059 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57f250 0x55f4fe589290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f14f7fff640 1 -- 172.21.15.28:0/3959982059 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57f250 msgr2=0x55f4fe589290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f14f7fff640 1 --2- 172.21.15.28:0/3959982059 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57f250 0x55f4fe589290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f14f7fff640 1 -- 172.21.15.28:0/3959982059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe5806e0 msgr2=0x55f4fe590810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f14fd4c0640 1 --2- 172.21.15.28:0/3959982059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe5806e0 0x55f4fe590810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f14f7fff640 1 --2- 172.21.15.28:0/3959982059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe5806e0 0x55f4fe590810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f14f7fff640 1 -- 172.21.15.28:0/3959982059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4fe2bc6b0 con 0x55f4fe57fc40 2024-04-21T08:51:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.469+0000 7f14fccbf640 1 --2- 172.21.15.28:0/3959982059 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57f250 0x55f4fe589290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f14f7fff640 1 --2- 172.21.15.28:0/3959982059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe57fc40 0x55f4fe5897d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f14e4009900 tx=0x7f14e402c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6806905d6e3402 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f14d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f14d7fff640 1 -- 172.21.15.28:0/3959982059 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e404c030 con 0x55f4fe57fc40 2024-04-21T08:51:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f14d7fff640 1 -- 172.21.15.28:0/3959982059 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f14e40035f0 con 0x55f4fe57fc40 2024-04-21T08:51:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f1501bec740 1 -- 172.21.15.28:0/3959982059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14e4004040 con 0x55f4fe57fc40 2024-04-21T08:51:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f14d7fff640 1 -- 172.21.15.28:0/3959982059 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f14e4058050 con 0x55f4fe57fc40 2024-04-21T08:51:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f1501bec740 10 client.15396 fetch_fsmap learned FSMap version 24 2024-04-21T08:51:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f1501bec740 1 -- 172.21.15.28:0/3959982059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f4fe593b80 con 0x55f4fe57fc40 2024-04-21T08:51:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f14d7fff640 1 -- 172.21.15.28:0/3959982059 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14e4004040 con 0x55f4fe57fc40 2024-04-21T08:51:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f14d7fff640 1 -- 172.21.15.28:0/3959982059 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f14e4004c20 con 0x55f4fe57fc40 2024-04-21T08:51:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f1501bec740 10 client.15396 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:51:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f1501bec740 10 client.15396 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-21T08:51:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f1501bec740 20 client.15396 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f1501bec740 1 --2- 172.21.15.28:0/3959982059 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f4fe595d20 0x55f4fe597190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f1501bec740 4 client.15396 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.470+0000 7f1501bec740 1 -- 172.21.15.28:0/3959982059 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4fe4ed530 con 0x55f4fe595d20 2024-04-21T08:51:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.471+0000 7f14fccbf640 1 --2- 172.21.15.28:0/3959982059 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f4fe595d20 0x55f4fe597190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.471+0000 7f14fccbf640 1 --2- 172.21.15.28:0/3959982059 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f4fe595d20 0x55f4fe597190 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f123d099881ca25b server_cookie=505bbfe02db66fc6 in_seq=0 out_seq=0 2024-04-21T08:51:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.471+0000 7f14d7fff640 10 client.15396 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.472+0000 7f14d7fff640 1 -- 172.21.15.28:0/3959982059 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f4fe4ed530 con 0x55f4fe595d20 2024-04-21T08:51:02.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.472+0000 7f14d7fff640 10 client.15396 handle_command_reply: tid=0 2024-04-21T08:51:02.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.472+0000 7f1501bec740 1 client.15396 shutdown 2024-04-21T08:51:02.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.472+0000 7f1501bec740 1 -- 172.21.15.28:0/3959982059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe57fc40 msgr2=0x55f4fe5897d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.472+0000 7f1501bec740 1 --2- 172.21.15.28:0/3959982059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe57fc40 0x55f4fe5897d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f14e4009900 tx=0x7f14e402c960 comp rx=0 tx=0).stop 2024-04-21T08:51:02.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.472+0000 7f1501bec740 1 -- 172.21.15.28:0/3959982059 shutdown_connections 2024-04-21T08:51:02.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.472+0000 7f1501bec740 1 --2- 172.21.15.28:0/3959982059 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f4fe595d20 0x55f4fe597190 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.472+0000 7f1501bec740 1 --2- 172.21.15.28:0/3959982059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe5806e0 0x55f4fe590810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.472+0000 7f1501bec740 1 --2- 172.21.15.28:0/3959982059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe57fc40 0x55f4fe5897d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.472+0000 7f1501bec740 1 --2- 172.21.15.28:0/3959982059 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57f250 0x55f4fe589290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.472+0000 7f14d7fff640 0 client.15396 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:02.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.472+0000 7f1501bec740 1 -- 172.21.15.28:0/3959982059 >> 172.21.15.28:0/3959982059 conn(0x55f4fe576f40 msgr2=0x55f4fe574a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.472+0000 7f1501bec740 1 -- 172.21.15.28:0/3959982059 shutdown_connections 2024-04-21T08:51:02.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.472+0000 7f1501bec740 1 -- 172.21.15.28:0/3959982059 wait complete. 2024-04-21T08:51:02.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.472+0000 7f1501bec740 20 client.15396 trim_cache size 0 max 16384 2024-04-21T08:51:02.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.490+0000 7f1501bec740 1 Processor -- start 2024-04-21T08:51:02.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.490+0000 7f1501bec740 1 -- start start 2024-04-21T08:51:02.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.490+0000 7f1501bec740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57f250 0x55f4fe589250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f1501bec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe57fc40 0x55f4fe5907d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f1501bec740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe5806e0 0x55f4fe589790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f1501bec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f4fe5954d0 con 0x55f4fe57fc40 2024-04-21T08:51:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f1501bec740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f4fe595ba0 con 0x55f4fe5806e0 2024-04-21T08:51:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f1501bec740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f4fe596270 con 0x55f4fe57f250 2024-04-21T08:51:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f14f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe57fc40 0x55f4fe5907d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f14f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe57fc40 0x55f4fe5907d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37774/0 (socket says 172.21.15.28:37774) 2024-04-21T08:51:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f14f7fff640 1 -- 172.21.15.28:0/2814127243 learned_addr learned my addr 172.21.15.28:0/2814127243 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f14fccbf640 1 --2- 172.21.15.28:0/2814127243 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57f250 0x55f4fe589250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f14f7fff640 1 -- 172.21.15.28:0/2814127243 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57f250 msgr2=0x55f4fe589250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f14f7fff640 1 --2- 172.21.15.28:0/2814127243 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57f250 0x55f4fe589250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f14f7fff640 1 -- 172.21.15.28:0/2814127243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe5806e0 msgr2=0x55f4fe589790 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:51:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f14f7fff640 1 --2- 172.21.15.28:0/2814127243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe5806e0 0x55f4fe589790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f14f7fff640 1 -- 172.21.15.28:0/2814127243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4fe576630 con 0x55f4fe57fc40 2024-04-21T08:51:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f14fccbf640 1 --2- 172.21.15.28:0/2814127243 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57f250 0x55f4fe589250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f14f7fff640 1 --2- 172.21.15.28:0/2814127243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe57fc40 0x55f4fe5907d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f14e402bc70 tx=0x7f14e4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=313c1727fb3c768 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f14e27fc640 1 -- 172.21.15.28:0/2814127243 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e404c070 con 0x55f4fe57fc40 2024-04-21T08:51:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f14e27fc640 1 -- 172.21.15.28:0/2814127243 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14e40051b0 con 0x55f4fe57fc40 2024-04-21T08:51:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.491+0000 7f14e27fc640 1 -- 172.21.15.28:0/2814127243 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e4043040 con 0x55f4fe57fc40 2024-04-21T08:51:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.492+0000 7f1501bec740 1 -- 172.21.15.28:0/2814127243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe57fc40 msgr2=0x55f4fe5907d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.492+0000 7f1501bec740 1 --2- 172.21.15.28:0/2814127243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe57fc40 0x55f4fe5907d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f14e402bc70 tx=0x7f14e4000bc0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.492+0000 7f1501bec740 1 -- 172.21.15.28:0/2814127243 shutdown_connections 2024-04-21T08:51:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.492+0000 7f1501bec740 1 --2- 172.21.15.28:0/2814127243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe5806e0 0x55f4fe589790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.492+0000 7f1501bec740 1 --2- 172.21.15.28:0/2814127243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe57fc40 0x55f4fe5907d0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.492+0000 7f1501bec740 1 --2- 172.21.15.28:0/2814127243 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe57f250 0x55f4fe589250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.492+0000 7f1501bec740 1 -- 172.21.15.28:0/2814127243 >> 172.21.15.28:0/2814127243 conn(0x55f4fe576f40 msgr2=0x55f4fe583110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.492+0000 7f1501bec740 1 -- 172.21.15.28:0/2814127243 shutdown_connections 2024-04-21T08:51:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.492+0000 7f1501bec740 1 -- 172.21.15.28:0/2814127243 wait complete. 2024-04-21T08:51:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.492+0000 7f1501bec740 1 -- start start 2024-04-21T08:51:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.492+0000 7f1501bec740 1 Processor -- start 2024-04-21T08:51:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f1501bec740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe57f250 0x55f4fe58c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f1501bec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe5806e0 0x55f4fe58ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f1501bec740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe58f1a0 0x55f4fe58f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f1501bec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f4fe597010 con 0x55f4fe5806e0 2024-04-21T08:51:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f1501bec740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f4fe596940 con 0x55f4fe57f250 2024-04-21T08:51:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f1501bec740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f4fe5976e0 con 0x55f4fe58f1a0 2024-04-21T08:51:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f1501bec740 10 client.0 _ll_register_callbacks cb 0x7f14e402c660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f14f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe5806e0 0x55f4fe58ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f14f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe5806e0 0x55f4fe58ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37784/0 (socket says 172.21.15.28:37784) 2024-04-21T08:51:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f14f7fff640 1 -- 172.21.15.28:0/2684491071 learned_addr learned my addr 172.21.15.28:0/2684491071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f14fd4c0640 1 --2- 172.21.15.28:0/2684491071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe58f1a0 0x55f4fe58f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f14f7fff640 1 -- 172.21.15.28:0/2684491071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe58f1a0 msgr2=0x55f4fe58f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f14f7fff640 1 --2- 172.21.15.28:0/2684491071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe58f1a0 0x55f4fe58f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f14f7fff640 1 -- 172.21.15.28:0/2684491071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe57f250 msgr2=0x55f4fe58c710 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:51:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f14f7fff640 1 --2- 172.21.15.28:0/2684491071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe57f250 0x55f4fe58c710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f14f7fff640 1 -- 172.21.15.28:0/2684491071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4fe58fea0 con 0x55f4fe5806e0 2024-04-21T08:51:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f14fd4c0640 1 --2- 172.21.15.28:0/2684491071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe58f1a0 0x55f4fe58f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f14f7fff640 1 --2- 172.21.15.28:0/2684491071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe5806e0 0x55f4fe58ec60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f14e402c710 tx=0x7f14e4048750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17898c57b093f33c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f14d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f14d7fff640 1 -- 172.21.15.28:0/2684491071 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e404c070 con 0x55f4fe5806e0 2024-04-21T08:51:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f14d7fff640 1 -- 172.21.15.28:0/2684491071 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f14e4048a30 con 0x55f4fe5806e0 2024-04-21T08:51:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.493+0000 7f1501bec740 1 -- 172.21.15.28:0/2684491071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f4fe4e73a0 con 0x55f4fe5806e0 2024-04-21T08:51:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.494+0000 7f14d7fff640 1 -- 172.21.15.28:0/2684491071 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f14e4068050 con 0x55f4fe5806e0 2024-04-21T08:51:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.494+0000 7f1501bec740 10 client.15402 fetch_fsmap learned FSMap version 24 2024-04-21T08:51:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.494+0000 7f1501bec740 1 -- 172.21.15.28:0/2684491071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f4fe5fc070 con 0x55f4fe5806e0 2024-04-21T08:51:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.494+0000 7f14d7fff640 1 -- 172.21.15.28:0/2684491071 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f4fe4e73a0 con 0x55f4fe5806e0 2024-04-21T08:51:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.494+0000 7f14d7fff640 1 -- 172.21.15.28:0/2684491071 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f14e4034660 con 0x55f4fe5806e0 2024-04-21T08:51:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.494+0000 7f1501bec740 10 client.15402 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:51:02.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.494+0000 7f1501bec740 10 client.15402 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-21T08:51:02.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.494+0000 7f1501bec740 20 client.15402 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:02.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.494+0000 7f1501bec740 1 --2- 172.21.15.28:0/2684491071 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f4fe600840 0x55f4fe5ff550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.494+0000 7f1501bec740 4 client.15402 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:51:02.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.494+0000 7f1501bec740 1 -- 172.21.15.28:0/2684491071 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "status"}) -- 0x55f4fe463960 con 0x55f4fe600840 2024-04-21T08:51:02.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.494+0000 7f14fccbf640 1 --2- 172.21.15.28:0/2684491071 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f4fe600840 0x55f4fe5ff550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.495+0000 7f14fccbf640 1 --2- 172.21.15.28:0/2684491071 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f4fe600840 0x55f4fe5ff550 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6f648e11a7c5c010 server_cookie=29f79e2ad7842621 in_seq=0 out_seq=0 2024-04-21T08:51:02.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.495+0000 7f14d7fff640 10 client.15402 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:02.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.495+0000 7f14d7fff640 1 -- 172.21.15.28:0/2684491071 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55f4fe463960 con 0x55f4fe600840 2024-04-21T08:51:02.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.495+0000 7f14d7fff640 10 client.15402 handle_command_reply: tid=0 2024-04-21T08:51:02.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.495+0000 7f1501bec740 1 client.15402 shutdown 2024-04-21T08:51:02.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.495+0000 7f1501bec740 1 -- 172.21.15.28:0/2684491071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe5806e0 msgr2=0x55f4fe58ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.495+0000 7f1501bec740 1 --2- 172.21.15.28:0/2684491071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe5806e0 0x55f4fe58ec60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f14e402c710 tx=0x7f14e4048750 comp rx=0 tx=0).stop 2024-04-21T08:51:02.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.495+0000 7f1501bec740 1 -- 172.21.15.28:0/2684491071 shutdown_connections 2024-04-21T08:51:02.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.495+0000 7f1501bec740 1 --2- 172.21.15.28:0/2684491071 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f4fe600840 0x55f4fe5ff550 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.495+0000 7f1501bec740 1 --2- 172.21.15.28:0/2684491071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f4fe58f1a0 0x55f4fe58f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.495+0000 7f1501bec740 1 --2- 172.21.15.28:0/2684491071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4fe5806e0 0x55f4fe58ec60 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.495+0000 7f1501bec740 1 --2- 172.21.15.28:0/2684491071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f4fe57f250 0x55f4fe58c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.495+0000 7f14d7fff640 0 client.15402 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.495+0000 7f1501bec740 1 -- 172.21.15.28:0/2684491071 >> 172.21.15.28:0/2684491071 conn(0x55f4fe576f40 msgr2=0x55f4fe579c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.496+0000 7f1501bec740 1 -- 172.21.15.28:0/2684491071 shutdown_connections 2024-04-21T08:51:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.496+0000 7f1501bec740 1 -- 172.21.15.28:0/2684491071 wait complete. 2024-04-21T08:51:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.496+0000 7f1501bec740 20 client.15402 trim_cache size 0 max 16384 2024-04-21T08:51:02.502 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:51:02.502 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "b51a76c8-ffba-11ee-bc93-c7b262605968", 2024-04-21T08:51:02.503 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-04-21T08:51:02.503 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24463, 2024-04-21T08:51:02.503 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-21T08:51:02.503 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-21T08:51:02.503 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-21T08:51:02.503 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 110.661697101, 2024-04-21T08:51:02.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-04-21T08:51:02.503 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-04-21T08:51:02.503 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:51:02.503 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 155.38440082700001 2024-04-21T08:51:02.503 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:51:02.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.496+0000 7f14fef23640 1 -- 172.21.15.28:0/3728154192 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f14e80a3c30 msgr2=0x7f14e80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.496+0000 7f14fef23640 1 --2- 172.21.15.28:0/3728154192 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f14e80a3c30 0x7f14e80a60f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x55f4fe576200 tx=0x7f14f0001710 comp rx=0 tx=0).stop 2024-04-21T08:51:02.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.496+0000 7f14fef23640 1 -- 172.21.15.28:0/3728154192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f8102ae0 msgr2=0x7f14f81982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.496+0000 7f14fef23640 1 --2- 172.21.15.28:0/3728154192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f8102ae0 0x7f14f81982d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00d950 tx=0x7f14ec00de10 comp rx=0 tx=0).stop 2024-04-21T08:51:02.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.496+0000 7f14fef23640 1 -- 172.21.15.28:0/3728154192 shutdown_connections 2024-04-21T08:51:02.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.496+0000 7f14fef23640 1 --2- 172.21.15.28:0/3728154192 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14f8107d40 0x7f14f819f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.496+0000 7f14fef23640 1 --2- 172.21.15.28:0/3728154192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14f8105410 0x7f14f8198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.496+0000 7f14fef23640 1 --2- 172.21.15.28:0/3728154192 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f14e80a3c30 0x7f14e80a60f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.496+0000 7f14fef23640 1 --2- 172.21.15.28:0/3728154192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f8102ae0 0x7f14f81982d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.496+0000 7f14fef23640 1 -- 172.21.15.28:0/3728154192 >> 172.21.15.28:0/3728154192 conn(0x7f14f80fc410 msgr2=0x7f14f8104950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.496+0000 7f14fef23640 1 -- 172.21.15.28:0/3728154192 shutdown_connections 2024-04-21T08:51:02.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.496+0000 7f14fef23640 1 -- 172.21.15.28:0/3728154192 wait complete. 2024-04-21T08:51:02.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-21T08:51:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.611+0000 7f2e4aa32640 1 -- 172.21.15.28:0/1786569597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e44102a30 msgr2=0x7f2e44104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.611+0000 7f2e4aa32640 1 --2- 172.21.15.28:0/1786569597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e44102a30 0x7f2e44104e20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2e34009900 tx=0x7f2e3402bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.611+0000 7f2e4aa32640 1 -- 172.21.15.28:0/1786569597 shutdown_connections 2024-04-21T08:51:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.611+0000 7f2e4aa32640 1 --2- 172.21.15.28:0/1786569597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e44107c90 0x7f2e4410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.611+0000 7f2e4aa32640 1 --2- 172.21.15.28:0/1786569597 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e44105360 0x7f2e44107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.611+0000 7f2e4aa32640 1 --2- 172.21.15.28:0/1786569597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e44102a30 0x7f2e44104e20 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.611+0000 7f2e4aa32640 1 -- 172.21.15.28:0/1786569597 >> 172.21.15.28:0/1786569597 conn(0x7f2e440fc440 msgr2=0x7f2e440fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.611+0000 7f2e4aa32640 1 -- 172.21.15.28:0/1786569597 shutdown_connections 2024-04-21T08:51:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.611+0000 7f2e4aa32640 1 -- 172.21.15.28:0/1786569597 wait complete. 2024-04-21T08:51:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e4aa32640 1 Processor -- start 2024-04-21T08:51:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e4aa32640 1 -- start start 2024-04-21T08:51:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e4aa32640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e44102a30 0x7f2e441981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e4aa32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e44105360 0x7f2e441986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e4aa32640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e44107c90 0x7f2e4419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e4aa32640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4410daf0 con 0x7f2e44105360 2024-04-21T08:51:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e4aa32640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4410d970 con 0x7f2e44107c90 2024-04-21T08:51:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e4aa32640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4410dc70 con 0x7f2e44102a30 2024-04-21T08:51:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e44105360 0x7f2e441986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e44105360 0x7f2e441986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37798/0 (socket says 172.21.15.28:37798) 2024-04-21T08:51:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e437fe640 1 -- 172.21.15.28:0/974581830 learned_addr learned my addr 172.21.15.28:0/974581830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e43fff640 1 --2- 172.21.15.28:0/974581830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e44102a30 0x7f2e441981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e48fcf640 1 --2- 172.21.15.28:0/974581830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e44107c90 0x7f2e4419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e437fe640 1 -- 172.21.15.28:0/974581830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e44102a30 msgr2=0x7f2e441981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e437fe640 1 --2- 172.21.15.28:0/974581830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e44102a30 0x7f2e441981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e437fe640 1 -- 172.21.15.28:0/974581830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e44107c90 msgr2=0x7f2e4419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e437fe640 1 --2- 172.21.15.28:0/974581830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e44107c90 0x7f2e4419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e437fe640 1 -- 172.21.15.28:0/974581830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e4419fdc0 con 0x7f2e44105360 2024-04-21T08:51:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e43fff640 1 --2- 172.21.15.28:0/974581830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e44102a30 0x7f2e441981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e437fe640 1 --2- 172.21.15.28:0/974581830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e44105360 0x7f2e441986e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800c970 tx=0x7f2e3800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e417fa640 1 -- 172.21.15.28:0/974581830 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e38014070 con 0x7f2e44105360 2024-04-21T08:51:02.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e417fa640 1 -- 172.21.15.28:0/974581830 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e38004440 con 0x7f2e44105360 2024-04-21T08:51:02.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e4aa32640 1 -- 172.21.15.28:0/974581830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e441a00b0 con 0x7f2e44105360 2024-04-21T08:51:02.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e417fa640 1 -- 172.21.15.28:0/974581830 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e38003e50 con 0x7f2e44105360 2024-04-21T08:51:02.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.612+0000 7f2e4aa32640 1 -- 172.21.15.28:0/974581830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e441a0650 con 0x7f2e44105360 2024-04-21T08:51:02.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.615+0000 7f2e417fa640 1 -- 172.21.15.28:0/974581830 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2e38017070 con 0x7f2e44105360 2024-04-21T08:51:02.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.615+0000 7f2e4d619740 1 Processor -- start 2024-04-21T08:51:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.615+0000 7f2e4d619740 1 -- start start 2024-04-21T08:51:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e4d619740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af2e40 0x555d44af3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e4d619740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af3830 0x555d44af3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e4d619740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af42d0 0x555d44afaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e4d619740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555d44a57b10 con 0x555d44af42d0 2024-04-21T08:51:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e4d619740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x555d449f5240 con 0x555d44af3830 2024-04-21T08:51:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e4d619740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x555d44a53ad0 con 0x555d44af2e40 2024-04-21T08:51:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e417fa640 1 --2- 172.21.15.28:0/974581830 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2e2c0a3dd0 0x7f2e2c0a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af42d0 0x555d44afaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af42d0 0x555d44afaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37810/0 (socket says 172.21.15.28:37810) 2024-04-21T08:51:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e437fe640 1 -- 172.21.15.28:0/46726344 learned_addr learned my addr 172.21.15.28:0/46726344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e48fcf640 1 --2- 172.21.15.28:0/974581830 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2e2c0a3dd0 0x7f2e2c0a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e48fcf640 1 --2- 172.21.15.28:0/46726344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af2e40 0x555d44af3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e417fa640 1 -- 172.21.15.28:0/974581830 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f2e380d0660 con 0x7f2e44105360 2024-04-21T08:51:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e43fff640 1 --2- 172.21.15.28:0/46726344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af3830 0x555d44af3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e437fe640 1 -- 172.21.15.28:0/46726344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af2e40 msgr2=0x555d44af3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e437fe640 1 --2- 172.21.15.28:0/46726344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af2e40 0x555d44af3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e437fe640 1 -- 172.21.15.28:0/46726344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af3830 msgr2=0x555d44af3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e437fe640 1 --2- 172.21.15.28:0/46726344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af3830 0x555d44af3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e437fe640 1 -- 172.21.15.28:0/46726344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d4476a2d0 con 0x555d44af42d0 2024-04-21T08:51:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e437fe640 1 --2- 172.21.15.28:0/46726344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af42d0 0x555d44afaeb0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2e380cdda0 tx=0x7f2e38000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=630ac708a4f88fcb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.616+0000 7f2e48fcf640 1 --2- 172.21.15.28:0/974581830 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2e2c0a3dd0 0x7f2e2c0a6290 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c005fd0 tx=0x7f2e3c009d10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.617+0000 7f2e2a7fc640 1 -- 172.21.15.28:0/46726344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3809c580 con 0x555d44af42d0 2024-04-21T08:51:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.617+0000 7f2e2a7fc640 1 -- 172.21.15.28:0/46726344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e380a0930 con 0x555d44af42d0 2024-04-21T08:51:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.617+0000 7f2e2a7fc640 1 -- 172.21.15.28:0/46726344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3800ec30 con 0x555d44af42d0 2024-04-21T08:51:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.617+0000 7f2e4d619740 1 -- 172.21.15.28:0/46726344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af42d0 msgr2=0x555d44afaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.617+0000 7f2e4d619740 1 --2- 172.21.15.28:0/46726344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af42d0 0x555d44afaeb0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2e380cdda0 tx=0x7f2e38000bc0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.617+0000 7f2e4d619740 1 -- 172.21.15.28:0/46726344 shutdown_connections 2024-04-21T08:51:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.617+0000 7f2e4d619740 1 --2- 172.21.15.28:0/46726344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af42d0 0x555d44afaeb0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.617+0000 7f2e4d619740 1 --2- 172.21.15.28:0/46726344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af3830 0x555d44af3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.617+0000 7f2e4d619740 1 --2- 172.21.15.28:0/46726344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af2e40 0x555d44af3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.617+0000 7f2e4d619740 1 -- 172.21.15.28:0/46726344 >> 172.21.15.28:0/46726344 conn(0x555d44aeef40 msgr2=0x555d44aed490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.617+0000 7f2e4d619740 1 -- 172.21.15.28:0/46726344 shutdown_connections 2024-04-21T08:51:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.617+0000 7f2e4d619740 1 -- 172.21.15.28:0/46726344 wait complete. 2024-04-21T08:51:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e4d619740 1 -- start start 2024-04-21T08:51:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e4d619740 1 Processor -- start 2024-04-21T08:51:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e4d619740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af2e40 0x555d44afce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e4d619740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af3830 0x555d44afd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e4d619740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af42d0 0x555d44b04400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e4d619740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555d44afb450 con 0x555d44af42d0 2024-04-21T08:51:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e4d619740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x555d449e0190 con 0x555d44af2e40 2024-04-21T08:51:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e4d619740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x555d44afb5d0 con 0x555d44af3830 2024-04-21T08:51:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e4d619740 10 client.0 _ll_register_callbacks cb 0x555d449db8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e48fcf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af42d0 0x555d44b04400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e48fcf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af42d0 0x555d44b04400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37814/0 (socket says 172.21.15.28:37814) 2024-04-21T08:51:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e48fcf640 1 -- 172.21.15.28:0/838048569 learned_addr learned my addr 172.21.15.28:0/838048569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e437fe640 1 --2- 172.21.15.28:0/838048569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af3830 0x555d44afd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e43fff640 1 --2- 172.21.15.28:0/838048569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af2e40 0x555d44afce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e48fcf640 1 -- 172.21.15.28:0/838048569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af3830 msgr2=0x555d44afd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e48fcf640 1 --2- 172.21.15.28:0/838048569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af3830 0x555d44afd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e48fcf640 1 -- 172.21.15.28:0/838048569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af2e40 msgr2=0x555d44afce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e48fcf640 1 --2- 172.21.15.28:0/838048569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af2e40 0x555d44afce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e48fcf640 1 -- 172.21.15.28:0/838048569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d448346b0 con 0x555d44af42d0 2024-04-21T08:51:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.618+0000 7f2e43fff640 1 --2- 172.21.15.28:0/838048569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af2e40 0x555d44afce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e48fcf640 1 --2- 172.21.15.28:0/838048569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af42d0 0x555d44b04400 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c010c20 tx=0x7f2e3c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1aec2dfaa7439c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e237fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e237fe640 1 -- 172.21.15.28:0/838048569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c026070 con 0x555d44af42d0 2024-04-21T08:51:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e237fe640 1 -- 172.21.15.28:0/838048569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f2e3c01f620 con 0x555d44af42d0 2024-04-21T08:51:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e4d619740 1 -- 172.21.15.28:0/838048569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e380a0930 con 0x555d44af42d0 2024-04-21T08:51:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e237fe640 1 -- 172.21.15.28:0/838048569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2e3c032050 con 0x555d44af42d0 2024-04-21T08:51:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e4d619740 10 client.15414 fetch_fsmap learned FSMap version 24 2024-04-21T08:51:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e4d619740 1 -- 172.21.15.28:0/838048569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555d44b077e0 con 0x555d44af42d0 2024-04-21T08:51:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e237fe640 1 -- 172.21.15.28:0/838048569 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e380a0930 con 0x555d44af42d0 2024-04-21T08:51:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e237fe640 1 -- 172.21.15.28:0/838048569 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f2e3c01e650 con 0x555d44af42d0 2024-04-21T08:51:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e4d619740 10 client.15414 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:51:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e4d619740 10 client.15414 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T08:51:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e4d619740 20 client.15414 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e4d619740 1 --2- 172.21.15.28:0/838048569 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x555d44b09eb0 0x555d44b0b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e4d619740 4 client.15414 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.619+0000 7f2e4d619740 1 -- 172.21.15.28:0/838048569 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d44a65530 con 0x555d44b09eb0 2024-04-21T08:51:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.620+0000 7f2e43fff640 1 --2- 172.21.15.28:0/838048569 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x555d44b09eb0 0x555d44b0b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.620+0000 7f2e43fff640 1 --2- 172.21.15.28:0/838048569 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x555d44b09eb0 0x555d44b0b320 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9ccfae799a5349a9 server_cookie=91b9e49ebfbbd3fc in_seq=0 out_seq=0 2024-04-21T08:51:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.620+0000 7f2e237fe640 10 client.15414 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.621+0000 7f2e237fe640 1 -- 172.21.15.28:0/838048569 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555d44a65530 con 0x555d44b09eb0 2024-04-21T08:51:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.621+0000 7f2e237fe640 10 client.15414 handle_command_reply: tid=0 2024-04-21T08:51:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.621+0000 7f2e4d619740 1 client.15414 shutdown 2024-04-21T08:51:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.621+0000 7f2e4d619740 1 -- 172.21.15.28:0/838048569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af42d0 msgr2=0x555d44b04400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.621+0000 7f2e4d619740 1 --2- 172.21.15.28:0/838048569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af42d0 0x555d44b04400 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c010c20 tx=0x7f2e3c01a000 comp rx=0 tx=0).stop 2024-04-21T08:51:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.621+0000 7f2e4d619740 1 -- 172.21.15.28:0/838048569 shutdown_connections 2024-04-21T08:51:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.621+0000 7f2e4d619740 1 --2- 172.21.15.28:0/838048569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af42d0 0x555d44b04400 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.621+0000 7f2e4d619740 1 --2- 172.21.15.28:0/838048569 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x555d44b09eb0 0x555d44b0b320 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.621+0000 7f2e4d619740 1 --2- 172.21.15.28:0/838048569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af3830 0x555d44afd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.621+0000 7f2e4d619740 1 --2- 172.21.15.28:0/838048569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af2e40 0x555d44afce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.621+0000 7f2e4d619740 1 -- 172.21.15.28:0/838048569 >> 172.21.15.28:0/838048569 conn(0x555d44aeef40 msgr2=0x555d44aeca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.621+0000 7f2e237fe640 0 client.15414 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.622+0000 7f2e4d619740 1 -- 172.21.15.28:0/838048569 shutdown_connections 2024-04-21T08:51:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.622+0000 7f2e4d619740 1 -- 172.21.15.28:0/838048569 wait complete. 2024-04-21T08:51:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.622+0000 7f2e4d619740 20 client.15414 trim_cache size 0 max 16384 2024-04-21T08:51:02.629 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:02 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2720815512' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:02.629 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:02 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3249271336' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:02.629 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:02 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3429408496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:02.629 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:02 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1849240336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:02.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.640+0000 7f2e4d619740 1 Processor -- start 2024-04-21T08:51:02.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.640+0000 7f2e4d619740 1 -- start start 2024-04-21T08:51:02.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.640+0000 7f2e4d619740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af2e40 0x555d44b08c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.640+0000 7f2e4d619740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af3830 0x555d44b04320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.640+0000 7f2e4d619740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af42d0 0x555d44afce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.640+0000 7f2e4d619740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555d44b09660 con 0x555d44af3830 2024-04-21T08:51:02.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.640+0000 7f2e4d619740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x555d44b09d30 con 0x555d44af2e40 2024-04-21T08:51:02.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.640+0000 7f2e4d619740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x555d44b0a400 con 0x555d44af42d0 2024-04-21T08:51:02.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.640+0000 7f2e437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af3830 0x555d44b04320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.640+0000 7f2e437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af3830 0x555d44b04320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37824/0 (socket says 172.21.15.28:37824) 2024-04-21T08:51:02.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.640+0000 7f2e437fe640 1 -- 172.21.15.28:0/893046161 learned_addr learned my addr 172.21.15.28:0/893046161 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.640+0000 7f2e48fcf640 1 --2- 172.21.15.28:0/893046161 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af42d0 0x555d44afce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.640+0000 7f2e43fff640 1 --2- 172.21.15.28:0/893046161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af2e40 0x555d44b08c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e437fe640 1 -- 172.21.15.28:0/893046161 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af42d0 msgr2=0x555d44afce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e437fe640 1 --2- 172.21.15.28:0/893046161 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af42d0 0x555d44afce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e437fe640 1 -- 172.21.15.28:0/893046161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af2e40 msgr2=0x555d44b08c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e437fe640 1 --2- 172.21.15.28:0/893046161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af2e40 0x555d44b08c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e437fe640 1 -- 172.21.15.28:0/893046161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d44b0b650 con 0x555d44af3830 2024-04-21T08:51:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e43fff640 1 --2- 172.21.15.28:0/893046161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af2e40 0x555d44b08c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e437fe640 1 --2- 172.21.15.28:0/893046161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af3830 0x555d44b04320 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2e380cddf0 tx=0x7f2e380d0980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f8f87d5920c356 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e29ffb640 1 -- 172.21.15.28:0/893046161 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e380a0930 con 0x555d44af3830 2024-04-21T08:51:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e29ffb640 1 -- 172.21.15.28:0/893046161 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e380a51f0 con 0x555d44af3830 2024-04-21T08:51:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e29ffb640 1 -- 172.21.15.28:0/893046161 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3800ec30 con 0x555d44af3830 2024-04-21T08:51:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e4d619740 1 -- 172.21.15.28:0/893046161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af3830 msgr2=0x555d44b04320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e4d619740 1 --2- 172.21.15.28:0/893046161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af3830 0x555d44b04320 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2e380cddf0 tx=0x7f2e380d0980 comp rx=0 tx=0).stop 2024-04-21T08:51:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e4d619740 1 -- 172.21.15.28:0/893046161 shutdown_connections 2024-04-21T08:51:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e4d619740 1 --2- 172.21.15.28:0/893046161 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af42d0 0x555d44afce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e4d619740 1 --2- 172.21.15.28:0/893046161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af3830 0x555d44b04320 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e4d619740 1 --2- 172.21.15.28:0/893046161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af2e40 0x555d44b08c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.641+0000 7f2e4d619740 1 -- 172.21.15.28:0/893046161 >> 172.21.15.28:0/893046161 conn(0x555d44aeef40 msgr2=0x555d44ae8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e4d619740 1 -- 172.21.15.28:0/893046161 shutdown_connections 2024-04-21T08:51:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e4d619740 1 -- 172.21.15.28:0/893046161 wait complete. 2024-04-21T08:51:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e4d619740 1 -- start start 2024-04-21T08:51:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e4d619740 1 Processor -- start 2024-04-21T08:51:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e4d619740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af2e40 0x555d44b00300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e4d619740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af3830 0x555d44b02850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e4d619740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af42d0 0x555d44b02e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e4d619740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555d44b0b1a0 con 0x555d44af3830 2024-04-21T08:51:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e4d619740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x555d44b0aad0 con 0x555d44af42d0 2024-04-21T08:51:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e4d619740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x555d44b0b870 con 0x555d44af2e40 2024-04-21T08:51:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e4d619740 10 client.0 _ll_register_callbacks cb 0x7f2e3c0097d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af3830 0x555d44b02850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af3830 0x555d44b02850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37832/0 (socket says 172.21.15.28:37832) 2024-04-21T08:51:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e437fe640 1 -- 172.21.15.28:0/3664105047 learned_addr learned my addr 172.21.15.28:0/3664105047 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e43fff640 1 --2- 172.21.15.28:0/3664105047 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af2e40 0x555d44b00300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.642+0000 7f2e48fcf640 1 --2- 172.21.15.28:0/3664105047 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af42d0 0x555d44b02e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e437fe640 1 -- 172.21.15.28:0/3664105047 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af2e40 msgr2=0x555d44b00300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e437fe640 1 --2- 172.21.15.28:0/3664105047 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af2e40 0x555d44b00300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e437fe640 1 -- 172.21.15.28:0/3664105047 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af42d0 msgr2=0x555d44b02e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e437fe640 1 --2- 172.21.15.28:0/3664105047 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af42d0 0x555d44b02e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e437fe640 1 -- 172.21.15.28:0/3664105047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d44b03690 con 0x555d44af3830 2024-04-21T08:51:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e48fcf640 1 --2- 172.21.15.28:0/3664105047 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af42d0 0x555d44b02e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e437fe640 1 --2- 172.21.15.28:0/3664105047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af3830 0x555d44b02850 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2e38098f50 tx=0x7f2e38087050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8205ff41275c5cb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e237fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e237fe640 1 -- 172.21.15.28:0/3664105047 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3809cc80 con 0x555d44af3830 2024-04-21T08:51:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e237fe640 1 -- 172.21.15.28:0/3664105047 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f2e3800ec30 con 0x555d44af3830 2024-04-21T08:51:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e4d619740 1 -- 172.21.15.28:0/3664105047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e380a51f0 con 0x555d44af3830 2024-04-21T08:51:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e237fe640 1 -- 172.21.15.28:0/3664105047 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2e380a0930 con 0x555d44af3830 2024-04-21T08:51:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e4d619740 10 client.15420 fetch_fsmap learned FSMap version 24 2024-04-21T08:51:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e4d619740 1 -- 172.21.15.28:0/3664105047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x555d44b6fe60 con 0x555d44af3830 2024-04-21T08:51:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e237fe640 1 -- 172.21.15.28:0/3664105047 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e380a51f0 con 0x555d44af3830 2024-04-21T08:51:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e237fe640 1 -- 172.21.15.28:0/3664105047 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f2e38085ba0 con 0x555d44af3830 2024-04-21T08:51:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e4d619740 10 client.15420 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:51:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.643+0000 7f2e4d619740 10 client.15420 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T08:51:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.644+0000 7f2e4d619740 20 client.15420 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.644+0000 7f2e4d619740 1 --2- 172.21.15.28:0/3664105047 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x555d44b73f70 0x555d44b72c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.644+0000 7f2e4d619740 4 client.15420 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:51:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.644+0000 7f2e4d619740 1 -- 172.21.15.28:0/3664105047 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "status"}) -- 0x555d44a5f3a0 con 0x555d44b73f70 2024-04-21T08:51:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.644+0000 7f2e43fff640 1 --2- 172.21.15.28:0/3664105047 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x555d44b73f70 0x555d44b72c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.644+0000 7f2e43fff640 1 --2- 172.21.15.28:0/3664105047 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x555d44b73f70 0x555d44b72c80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8ce92b098c6ea954 server_cookie=fa7603bf4a0be6e7 in_seq=0 out_seq=0 2024-04-21T08:51:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.644+0000 7f2e237fe640 10 client.15420 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.644+0000 7f2e237fe640 1 -- 172.21.15.28:0/3664105047 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x555d44a5f3a0 con 0x555d44b73f70 2024-04-21T08:51:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.644+0000 7f2e237fe640 10 client.15420 handle_command_reply: tid=0 2024-04-21T08:51:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.644+0000 7f2e4d619740 1 client.15420 shutdown 2024-04-21T08:51:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4d619740 1 -- 172.21.15.28:0/3664105047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af3830 msgr2=0x555d44b02850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4d619740 1 --2- 172.21.15.28:0/3664105047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af3830 0x555d44b02850 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2e38098f50 tx=0x7f2e38087050 comp rx=0 tx=0).stop 2024-04-21T08:51:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4d619740 1 -- 172.21.15.28:0/3664105047 shutdown_connections 2024-04-21T08:51:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4d619740 1 --2- 172.21.15.28:0/3664105047 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555d44af42d0 0x555d44b02e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4d619740 1 --2- 172.21.15.28:0/3664105047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d44af3830 0x555d44b02850 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4d619740 1 --2- 172.21.15.28:0/3664105047 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x555d44b73f70 0x555d44b72c80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4d619740 1 --2- 172.21.15.28:0/3664105047 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555d44af2e40 0x555d44b00300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4d619740 1 -- 172.21.15.28:0/3664105047 >> 172.21.15.28:0/3664105047 conn(0x555d44aeef40 msgr2=0x555d44b08c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e237fe640 0 client.15420 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4d619740 1 -- 172.21.15.28:0/3664105047 shutdown_connections 2024-04-21T08:51:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4d619740 1 -- 172.21.15.28:0/3664105047 wait complete. 2024-04-21T08:51:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4d619740 20 client.15420 trim_cache size 0 max 16384 2024-04-21T08:51:02.652 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:51:02.652 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "b51a76c8-ffba-11ee-bc93-c7b262605968", 2024-04-21T08:51:02.652 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-04-21T08:51:02.652 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24487, 2024-04-21T08:51:02.652 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-21T08:51:02.652 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-21T08:51:02.652 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-21T08:51:02.652 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 111.821806733, 2024-04-21T08:51:02.652 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-04-21T08:51:02.652 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-04-21T08:51:02.652 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:51:02.652 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 149.709957086 2024-04-21T08:51:02.653 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:51:02.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4aa32640 1 -- 172.21.15.28:0/974581830 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2e2c0a3dd0 msgr2=0x7f2e2c0a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4aa32640 1 --2- 172.21.15.28:0/974581830 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2e2c0a3dd0 0x7f2e2c0a6290 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c005fd0 tx=0x7f2e3c009d10 comp rx=0 tx=0).stop 2024-04-21T08:51:02.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4aa32640 1 -- 172.21.15.28:0/974581830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e44105360 msgr2=0x7f2e441986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4aa32640 1 --2- 172.21.15.28:0/974581830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e44105360 0x7f2e441986e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800c970 tx=0x7f2e3800ce30 comp rx=0 tx=0).stop 2024-04-21T08:51:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4aa32640 1 -- 172.21.15.28:0/974581830 shutdown_connections 2024-04-21T08:51:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4aa32640 1 --2- 172.21.15.28:0/974581830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e44107c90 0x7f2e4419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4aa32640 1 --2- 172.21.15.28:0/974581830 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f2e2c0a3dd0 0x7f2e2c0a6290 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4aa32640 1 --2- 172.21.15.28:0/974581830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e44105360 0x7f2e441986e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4aa32640 1 --2- 172.21.15.28:0/974581830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2e44102a30 0x7f2e441981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.645+0000 7f2e4aa32640 1 -- 172.21.15.28:0/974581830 >> 172.21.15.28:0/974581830 conn(0x7f2e440fc440 msgr2=0x7f2e441034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.646+0000 7f2e4aa32640 1 -- 172.21.15.28:0/974581830 shutdown_connections 2024-04-21T08:51:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.646+0000 7f2e4aa32640 1 -- 172.21.15.28:0/974581830 wait complete. 2024-04-21T08:51:02.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:02 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2720815512' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:02.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:02 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3249271336' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:02.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:02 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3429408496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:02.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:02 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1849240336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:02.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:51:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.758+0000 7f71b8de0640 1 -- 172.21.15.28:0/2128637928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71b4102a70 msgr2=0x7f71b4104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.758+0000 7f71b8de0640 1 --2- 172.21.15.28:0/2128637928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71b4102a70 0x7f71b4104e60 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009900 tx=0x7f71ac02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.759+0000 7f71b8de0640 1 -- 172.21.15.28:0/2128637928 shutdown_connections 2024-04-21T08:51:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.759+0000 7f71b8de0640 1 --2- 172.21.15.28:0/2128637928 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71b4107cd0 0x7f71b410a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.759+0000 7f71b8de0640 1 --2- 172.21.15.28:0/2128637928 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71b41053a0 0x7f71b4107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.759+0000 7f71b8de0640 1 --2- 172.21.15.28:0/2128637928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71b4102a70 0x7f71b4104e60 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.759+0000 7f71b8de0640 1 -- 172.21.15.28:0/2128637928 >> 172.21.15.28:0/2128637928 conn(0x7f71b40fc400 msgr2=0x7f71b40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.759+0000 7f71b8de0640 1 -- 172.21.15.28:0/2128637928 shutdown_connections 2024-04-21T08:51:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.759+0000 7f71b8de0640 1 -- 172.21.15.28:0/2128637928 wait complete. 2024-04-21T08:51:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.759+0000 7f71b8de0640 1 Processor -- start 2024-04-21T08:51:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.759+0000 7f71b8de0640 1 -- start start 2024-04-21T08:51:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b8de0640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71b4102a70 0x7f71b4198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b8de0640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71b41053a0 0x7f71b4198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b8de0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71b4107cd0 0x7f71b419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b8de0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71b410dac0 con 0x7f71b4107cd0 2024-04-21T08:51:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b8de0640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f71b410d940 con 0x7f71b4102a70 2024-04-21T08:51:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b8de0640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f71b410dc40 con 0x7f71b41053a0 2024-04-21T08:51:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71b4107cd0 0x7f71b419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71b4107cd0 0x7f71b419f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37850/0 (socket says 172.21.15.28:37850) 2024-04-21T08:51:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b2d9d640 1 -- 172.21.15.28:0/239121014 learned_addr learned my addr 172.21.15.28:0/239121014 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b259c640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71b4102a70 0x7f71b4198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71b41053a0 0x7f71b4198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b2d9d640 1 -- 172.21.15.28:0/239121014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71b41053a0 msgr2=0x7f71b4198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b2d9d640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71b41053a0 0x7f71b4198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b2d9d640 1 -- 172.21.15.28:0/239121014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71b4102a70 msgr2=0x7f71b4198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b2d9d640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71b4102a70 0x7f71b4198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b2d9d640 1 -- 172.21.15.28:0/239121014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b419fe10 con 0x7f71b4107cd0 2024-04-21T08:51:02.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b259c640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71b4102a70 0x7f71b4198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:02.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71b41053a0 0x7f71b4198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:02.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b2d9d640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71b4107cd0 0x7f71b419f7a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f71a4009fd0 tx=0x7f71a400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71ab7fe640 1 -- 172.21.15.28:0/239121014 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a4017070 con 0x7f71b4107cd0 2024-04-21T08:51:02.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.760+0000 7f71b8de0640 1 -- 172.21.15.28:0/239121014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71b41a0100 con 0x7f71b4107cd0 2024-04-21T08:51:02.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.761+0000 7f71ab7fe640 1 -- 172.21.15.28:0/239121014 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71a4003e20 con 0x7f71b4107cd0 2024-04-21T08:51:02.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.761+0000 7f71ab7fe640 1 -- 172.21.15.28:0/239121014 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a4010460 con 0x7f71b4107cd0 2024-04-21T08:51:02.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.761+0000 7f71b8de0640 1 -- 172.21.15.28:0/239121014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71b41a0650 con 0x7f71b4107cd0 2024-04-21T08:51:02.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.763+0000 7f71ab7fe640 1 -- 172.21.15.28:0/239121014 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f71a4003970 con 0x7f71b4107cd0 2024-04-21T08:51:02.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.763+0000 7f71bb9cd740 1 Processor -- start 2024-04-21T08:51:02.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.763+0000 7f71bb9cd740 1 -- start start 2024-04-21T08:51:02.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.763+0000 7f71bb9cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e139670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.763+0000 7f71bb9cd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e139c40 0x563e5e13a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71bb9cd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e13a6e0 0x563e5e1412c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71bb9cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563e5e099b10 con 0x563e5e139250 2024-04-21T08:51:02.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71bb9cd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563e5e037240 con 0x563e5e139c40 2024-04-21T08:51:02.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71bb9cd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x563e5e095ad0 con 0x563e5e13a6e0 2024-04-21T08:51:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71b1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e139670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71b1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e139670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37852/0 (socket says 172.21.15.28:37852) 2024-04-21T08:51:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71b1d9b640 1 -- 172.21.15.28:0/3362690732 learned_addr learned my addr 172.21.15.28:0/3362690732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71b259c640 1 --2- 172.21.15.28:0/3362690732 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e139c40 0x563e5e13a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71ab7fe640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71a00a3bb0 0x7f71a00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/3362690732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e13a6e0 0x563e5e1412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71b1d9b640 1 -- 172.21.15.28:0/3362690732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e13a6e0 msgr2=0x563e5e1412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/3362690732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e13a6e0 0x563e5e1412c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71b1d9b640 1 -- 172.21.15.28:0/3362690732 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e139c40 msgr2=0x563e5e13a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/3362690732 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e139c40 0x563e5e13a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71b1d9b640 1 -- 172.21.15.28:0/3362690732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e5ddac2d0 con 0x563e5e139250 2024-04-21T08:51:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71b2d9d640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71a00a3bb0 0x7f71a00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/3362690732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e139670 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f719c00bf90 tx=0x7f719c00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=510b92d0d97abd25 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71ab7fe640 1 -- 172.21.15.28:0/239121014 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f71a40d0630 con 0x7f71b4107cd0 2024-04-21T08:51:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71a8ff9640 1 -- 172.21.15.28:0/3362690732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c005220 con 0x563e5e139250 2024-04-21T08:51:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71a8ff9640 1 -- 172.21.15.28:0/3362690732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f719c004c50 con 0x563e5e139250 2024-04-21T08:51:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.764+0000 7f71a8ff9640 1 -- 172.21.15.28:0/3362690732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c0127a0 con 0x563e5e139250 2024-04-21T08:51:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.765+0000 7f71bb9cd740 1 -- 172.21.15.28:0/3362690732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 msgr2=0x563e5e139670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.765+0000 7f71b2d9d640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71a00a3bb0 0x7f71a00a6070 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f71a409c250 tx=0x7f71a406af50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.765+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/3362690732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e139670 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f719c00bf90 tx=0x7f719c00b5b0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.765+0000 7f71bb9cd740 1 -- 172.21.15.28:0/3362690732 shutdown_connections 2024-04-21T08:51:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.765+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/3362690732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e13a6e0 0x563e5e1412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.765+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/3362690732 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e139c40 0x563e5e13a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.765+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/3362690732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e139670 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.765+0000 7f71bb9cd740 1 -- 172.21.15.28:0/3362690732 >> 172.21.15.28:0/3362690732 conn(0x563e5e130f40 msgr2=0x563e5e12f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.765+0000 7f71bb9cd740 1 -- 172.21.15.28:0/3362690732 shutdown_connections 2024-04-21T08:51:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.765+0000 7f71bb9cd740 1 -- 172.21.15.28:0/3362690732 wait complete. 2024-04-21T08:51:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71bb9cd740 1 -- start start 2024-04-21T08:51:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71bb9cd740 1 Processor -- start 2024-04-21T08:51:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71bb9cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e143270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71bb9cd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e139c40 0x563e5e1437b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71bb9cd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e13a6e0 0x563e5e14a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71bb9cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563e5e1418c0 con 0x563e5e139250 2024-04-21T08:51:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71bb9cd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563e5e022190 con 0x563e5e13a6e0 2024-04-21T08:51:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71bb9cd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x563e5e141a40 con 0x563e5e139c40 2024-04-21T08:51:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71b1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e143270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71bb9cd740 10 client.0 _ll_register_callbacks cb 0x563e5e01d8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71b1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e143270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37864/0 (socket says 172.21.15.28:37864) 2024-04-21T08:51:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71b1d9b640 1 -- 172.21.15.28:0/942871834 learned_addr learned my addr 172.21.15.28:0/942871834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71b1d9b640 1 -- 172.21.15.28:0/942871834 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e139c40 msgr2=0x563e5e1437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/942871834 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e139c40 0x563e5e1437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71b1d9b640 1 -- 172.21.15.28:0/942871834 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e13a6e0 msgr2=0x563e5e14a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/942871834 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e13a6e0 0x563e5e14a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71b1d9b640 1 -- 172.21.15.28:0/942871834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e5de766b0 con 0x563e5e139250 2024-04-21T08:51:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/942871834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e143270 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f719c016020 tx=0x7f719c004e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8168e40415024571 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.766+0000 7f7199ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.767+0000 7f7199ffb640 1 -- 172.21.15.28:0/942871834 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c004f70 con 0x563e5e139250 2024-04-21T08:51:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.767+0000 7f7199ffb640 1 -- 172.21.15.28:0/942871834 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f719c005110 con 0x563e5e139250 2024-04-21T08:51:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.767+0000 7f71bb9cd740 1 -- 172.21.15.28:0/942871834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f719c004c50 con 0x563e5e139250 2024-04-21T08:51:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.767+0000 7f7199ffb640 1 -- 172.21.15.28:0/942871834 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f719c031050 con 0x563e5e139250 2024-04-21T08:51:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.767+0000 7f71bb9cd740 10 client.15432 fetch_fsmap learned FSMap version 24 2024-04-21T08:51:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.767+0000 7f71bb9cd740 1 -- 172.21.15.28:0/942871834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563e5e14daf0 con 0x563e5e139250 2024-04-21T08:51:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.767+0000 7f7199ffb640 1 -- 172.21.15.28:0/942871834 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f719c004c50 con 0x563e5e139250 2024-04-21T08:51:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.767+0000 7f7199ffb640 1 -- 172.21.15.28:0/942871834 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f719c026020 con 0x563e5e139250 2024-04-21T08:51:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.767+0000 7f71bb9cd740 10 client.15432 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:51:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.767+0000 7f71bb9cd740 10 client.15432 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:51:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.767+0000 7f71bb9cd740 20 client.15432 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.767+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/942871834 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563e5e14ff50 0x563e5e1513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.767+0000 7f71bb9cd740 4 client.15432 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.767+0000 7f71bb9cd740 1 -- 172.21.15.28:0/942871834 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e5e0a7530 con 0x563e5e14ff50 2024-04-21T08:51:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.767+0000 7f71b259c640 1 --2- 172.21.15.28:0/942871834 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563e5e14ff50 0x563e5e1513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.768+0000 7f7199ffb640 10 client.15432 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.768+0000 7f71b259c640 1 --2- 172.21.15.28:0/942871834 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563e5e14ff50 0x563e5e1513c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8df6eb54640b262 server_cookie=339f42d66b3dcdc1 in_seq=0 out_seq=0 2024-04-21T08:51:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.769+0000 7f7199ffb640 1 -- 172.21.15.28:0/942871834 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563e5e0a7530 con 0x563e5e14ff50 2024-04-21T08:51:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.769+0000 7f7199ffb640 10 client.15432 handle_command_reply: tid=0 2024-04-21T08:51:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.769+0000 7f71bb9cd740 1 client.15432 shutdown 2024-04-21T08:51:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.769+0000 7f71bb9cd740 1 -- 172.21.15.28:0/942871834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 msgr2=0x563e5e143270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.769+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/942871834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e143270 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f719c016020 tx=0x7f719c004e50 comp rx=0 tx=0).stop 2024-04-21T08:51:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.769+0000 7f71bb9cd740 1 -- 172.21.15.28:0/942871834 shutdown_connections 2024-04-21T08:51:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.769+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/942871834 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563e5e14ff50 0x563e5e1513c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.769+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/942871834 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e13a6e0 0x563e5e14a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.769+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/942871834 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e139c40 0x563e5e1437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.769+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/942871834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e143270 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.769+0000 7f7199ffb640 0 client.15432 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.769+0000 7f71bb9cd740 1 -- 172.21.15.28:0/942871834 >> 172.21.15.28:0/942871834 conn(0x563e5e130f40 msgr2=0x563e5e12ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.769+0000 7f71bb9cd740 1 -- 172.21.15.28:0/942871834 shutdown_connections 2024-04-21T08:51:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.769+0000 7f71bb9cd740 1 -- 172.21.15.28:0/942871834 wait complete. 2024-04-21T08:51:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.769+0000 7f71bb9cd740 20 client.15432 trim_cache size 0 max 16384 2024-04-21T08:51:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71bb9cd740 1 Processor -- start 2024-04-21T08:51:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71bb9cd740 1 -- start start 2024-04-21T08:51:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71bb9cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e14ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71bb9cd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e139c40 0x563e5e14a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71bb9cd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e13a6e0 0x563e5e13dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71bb9cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563e5e14f700 con 0x563e5e139250 2024-04-21T08:51:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71bb9cd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563e5e14fdd0 con 0x563e5e139c40 2024-04-21T08:51:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71bb9cd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x563e5e1504a0 con 0x563e5e13a6e0 2024-04-21T08:51:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71b1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e14ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71b1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e14ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37868/0 (socket says 172.21.15.28:37868) 2024-04-21T08:51:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71b1d9b640 1 -- 172.21.15.28:0/777686858 learned_addr learned my addr 172.21.15.28:0/777686858 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71b1d9b640 1 -- 172.21.15.28:0/777686858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e13a6e0 msgr2=0x563e5e13dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/777686858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e13a6e0 0x563e5e13dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71b1d9b640 1 -- 172.21.15.28:0/777686858 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e139c40 msgr2=0x563e5e14a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/777686858 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e139c40 0x563e5e14a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.787+0000 7f71b1d9b640 1 -- 172.21.15.28:0/777686858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e5e144260 con 0x563e5e139250 2024-04-21T08:51:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.788+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/777686858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e14ef50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f719c014560 tx=0x7f719c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da60be7aa17998a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.788+0000 7f719bfff640 1 -- 172.21.15.28:0/777686858 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c00b6e0 con 0x563e5e139250 2024-04-21T08:51:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.788+0000 7f719bfff640 1 -- 172.21.15.28:0/777686858 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f719c012510 con 0x563e5e139250 2024-04-21T08:51:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.788+0000 7f719bfff640 1 -- 172.21.15.28:0/777686858 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c005120 con 0x563e5e139250 2024-04-21T08:51:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.788+0000 7f71bb9cd740 1 -- 172.21.15.28:0/777686858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 msgr2=0x563e5e14ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.788+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/777686858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e14ef50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f719c014560 tx=0x7f719c005dd0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.788+0000 7f71bb9cd740 1 -- 172.21.15.28:0/777686858 shutdown_connections 2024-04-21T08:51:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.788+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/777686858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e13a6e0 0x563e5e13dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.788+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/777686858 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e139c40 0x563e5e14a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.788+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/777686858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e14ef50 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.788+0000 7f71bb9cd740 1 -- 172.21.15.28:0/777686858 >> 172.21.15.28:0/777686858 conn(0x563e5e130f40 msgr2=0x563e5e141270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.788+0000 7f71bb9cd740 1 -- 172.21.15.28:0/777686858 shutdown_connections 2024-04-21T08:51:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.788+0000 7f71bb9cd740 1 -- 172.21.15.28:0/777686858 wait complete. 2024-04-21T08:51:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71bb9cd740 1 -- start start 2024-04-21T08:51:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71bb9cd740 1 Processor -- start 2024-04-21T08:51:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71bb9cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e13cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71bb9cd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e139c40 0x563e5e13d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71bb9cd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e13a6e0 0x563e5e1b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71bb9cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563e5e151240 con 0x563e5e139250 2024-04-21T08:51:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71bb9cd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563e5e150b70 con 0x563e5e13a6e0 2024-04-21T08:51:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71bb9cd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x563e5e151910 con 0x563e5e139c40 2024-04-21T08:51:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71bb9cd740 10 client.0 _ll_register_callbacks cb 0x7f719c014470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71b1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e13cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71b1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e13cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37882/0 (socket says 172.21.15.28:37882) 2024-04-21T08:51:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71b1d9b640 1 -- 172.21.15.28:0/2995143577 learned_addr learned my addr 172.21.15.28:0/2995143577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/2995143577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e13a6e0 0x563e5e1b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71b1d9b640 1 -- 172.21.15.28:0/2995143577 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e139c40 msgr2=0x563e5e13d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71b259c640 1 --2- 172.21.15.28:0/2995143577 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e139c40 0x563e5e13d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/2995143577 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e139c40 0x563e5e13d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71b1d9b640 1 -- 172.21.15.28:0/2995143577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e13a6e0 msgr2=0x563e5e1b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/2995143577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e13a6e0 0x563e5e1b3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.789+0000 7f71b1d9b640 1 -- 172.21.15.28:0/2995143577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e5e1b4500 con 0x563e5e139250 2024-04-21T08:51:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f71b1d9b640 1 --2- 172.21.15.28:0/2995143577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e13cb10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f719c005c20 tx=0x7f719c004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f0f0ce18e521d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f7199ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f7199ffb640 1 -- 172.21.15.28:0/2995143577 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c01d900 con 0x563e5e139250 2024-04-21T08:51:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f7199ffb640 1 -- 172.21.15.28:0/2995143577 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f719c01daa0 con 0x563e5e139250 2024-04-21T08:51:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f71bb9cd740 1 -- 172.21.15.28:0/2995143577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f719c012510 con 0x563e5e139250 2024-04-21T08:51:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f7199ffb640 1 -- 172.21.15.28:0/2995143577 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f719c01dca0 con 0x563e5e139250 2024-04-21T08:51:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f71bb9cd740 10 client.15438 fetch_fsmap learned FSMap version 24 2024-04-21T08:51:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f71bb9cd740 1 -- 172.21.15.28:0/2995143577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563e5e1b5800 con 0x563e5e139250 2024-04-21T08:51:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f7199ffb640 1 -- 172.21.15.28:0/2995143577 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f719c012510 con 0x563e5e139250 2024-04-21T08:51:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f7199ffb640 1 -- 172.21.15.28:0/2995143577 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f719c026050 con 0x563e5e139250 2024-04-21T08:51:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f71bb9cd740 10 client.15438 fetch_fsmap finished waiting for FSMap version 24 2024-04-21T08:51:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f71bb9cd740 10 client.15438 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:51:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f71bb9cd740 20 client.15438 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/2995143577 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563e5e1ba100 0x563e5e1b8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f71bb9cd740 4 client.15438 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:51:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.790+0000 7f71bb9cd740 1 -- 172.21.15.28:0/2995143577 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "status"}) -- 0x563e5e0a13a0 con 0x563e5e1ba100 2024-04-21T08:51:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f71b259c640 1 --2- 172.21.15.28:0/2995143577 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563e5e1ba100 0x563e5e1b8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f7199ffb640 10 client.15438 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f71b259c640 1 --2- 172.21.15.28:0/2995143577 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563e5e1ba100 0x563e5e1b8e10 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=178433127259949f server_cookie=41f6783230c19eb6 in_seq=0 out_seq=0 2024-04-21T08:51:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f7199ffb640 1 -- 172.21.15.28:0/2995143577 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x563e5e0a13a0 con 0x563e5e1ba100 2024-04-21T08:51:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f7199ffb640 10 client.15438 handle_command_reply: tid=0 2024-04-21T08:51:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f71bb9cd740 1 client.15438 shutdown 2024-04-21T08:51:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f71bb9cd740 1 -- 172.21.15.28:0/2995143577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 msgr2=0x563e5e13cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/2995143577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e13cb10 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f719c005c20 tx=0x7f719c004680 comp rx=0 tx=0).stop 2024-04-21T08:51:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f71bb9cd740 1 -- 172.21.15.28:0/2995143577 shutdown_connections 2024-04-21T08:51:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/2995143577 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563e5e1ba100 0x563e5e1b8e10 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/2995143577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563e5e13a6e0 0x563e5e1b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/2995143577 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563e5e139c40 0x563e5e13d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f71bb9cd740 1 --2- 172.21.15.28:0/2995143577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e5e139250 0x563e5e13cb10 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f7199ffb640 0 client.15438 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f71bb9cd740 1 -- 172.21.15.28:0/2995143577 >> 172.21.15.28:0/2995143577 conn(0x563e5e130f40 msgr2=0x563e5e12f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.791+0000 7f71bb9cd740 1 -- 172.21.15.28:0/2995143577 shutdown_connections 2024-04-21T08:51:02.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.792+0000 7f71bb9cd740 1 -- 172.21.15.28:0/2995143577 wait complete. 2024-04-21T08:51:02.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.792+0000 7f71bb9cd740 20 client.15438 trim_cache size 0 max 16384 2024-04-21T08:51:02.798 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:51:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "b51a76c8-ffba-11ee-bc93-c7b262605968", 2024-04-21T08:51:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-21T08:51:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24488, 2024-04-21T08:51:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-21T08:51:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-21T08:51:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-21T08:51:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 136.05934357999999, 2024-04-21T08:51:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-04-21T08:51:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-04-21T08:51:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:51:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 141.23154784100001 2024-04-21T08:51:02.799 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:51:02.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.792+0000 7f71b8de0640 1 -- 172.21.15.28:0/239121014 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71a00a3bb0 msgr2=0x7f71a00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.792+0000 7f71b8de0640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71a00a3bb0 0x7f71a00a6070 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f71a409c250 tx=0x7f71a406af50 comp rx=0 tx=0).stop 2024-04-21T08:51:02.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.792+0000 7f71b8de0640 1 -- 172.21.15.28:0/239121014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71b4107cd0 msgr2=0x7f71b419f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.792+0000 7f71b8de0640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71b4107cd0 0x7f71b419f7a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f71a4009fd0 tx=0x7f71a400b260 comp rx=0 tx=0).stop 2024-04-21T08:51:02.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.792+0000 7f71b8de0640 1 -- 172.21.15.28:0/239121014 shutdown_connections 2024-04-21T08:51:02.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.792+0000 7f71b8de0640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f71a00a3bb0 0x7f71a00a6070 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.792+0000 7f71b8de0640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71b4107cd0 0x7f71b419f7a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.792+0000 7f71b8de0640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71b41053a0 0x7f71b4198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.792+0000 7f71b8de0640 1 --2- 172.21.15.28:0/239121014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71b4102a70 0x7f71b4198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.792+0000 7f71b8de0640 1 -- 172.21.15.28:0/239121014 >> 172.21.15.28:0/239121014 conn(0x7f71b40fc400 msgr2=0x7f71b41033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.792+0000 7f71b8de0640 1 -- 172.21.15.28:0/239121014 shutdown_connections 2024-04-21T08:51:02.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.792+0000 7f71b8de0640 1 -- 172.21.15.28:0/239121014 wait complete. 2024-04-21T08:51:02.803 INFO:teuthology.run_tasks:Running task sequential... 2024-04-21T08:51:02.814 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-21T08:51:02.814 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-21T08:51:02.815 INFO:teuthology.task.print:Enabling mgr modules 2024-04-21T08:51:02.815 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-21T08:51:02.815 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T08:51:02.816 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-21T08:51:02.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-21T08:51:02.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:02 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2720815512' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:02.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:02 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3249271336' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:02.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:02 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3429408496' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:02.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:02 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1849240336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:02.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.918+0000 7f3a86fce640 1 -- 172.21.15.28:0/94611989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a80101fb0 msgr2=0x7f3a801023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.918+0000 7f3a86fce640 1 --2- 172.21.15.28:0/94611989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a80101fb0 0x7f3a801023d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3a78009ca0 tx=0x7f3a7802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.918+0000 7f3a86fce640 1 -- 172.21.15.28:0/94611989 shutdown_connections 2024-04-21T08:51:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.918+0000 7f3a86fce640 1 --2- 172.21.15.28:0/94611989 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a80103b60 0x7f3a8010a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.918+0000 7f3a86fce640 1 --2- 172.21.15.28:0/94611989 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a801031a0 0x7f3a80103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.918+0000 7f3a86fce640 1 --2- 172.21.15.28:0/94611989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a80101fb0 0x7f3a801023d0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.918+0000 7f3a86fce640 1 -- 172.21.15.28:0/94611989 >> 172.21.15.28:0/94611989 conn(0x7f3a800fd540 msgr2=0x7f3a800ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.918+0000 7f3a86fce640 1 -- 172.21.15.28:0/94611989 shutdown_connections 2024-04-21T08:51:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.918+0000 7f3a86fce640 1 -- 172.21.15.28:0/94611989 wait complete. 2024-04-21T08:51:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a86fce640 1 Processor -- start 2024-04-21T08:51:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a86fce640 1 -- start start 2024-04-21T08:51:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a86fce640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a80101fb0 0x7f3a80075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a86fce640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a801031a0 0x7f3a80074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a86fce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a80103b60 0x7f3a80074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a86fce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8010db30 con 0x7f3a80103b60 2024-04-21T08:51:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a86fce640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8010d9b0 con 0x7f3a80101fb0 2024-04-21T08:51:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a86fce640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8010dcb0 con 0x7f3a801031a0 2024-04-21T08:51:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a8556b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a80103b60 0x7f3a80074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a8556b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a80103b60 0x7f3a80074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37900/0 (socket says 172.21.15.28:37900) 2024-04-21T08:51:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a8556b640 1 -- 172.21.15.28:0/1452241440 learned_addr learned my addr 172.21.15.28:0/1452241440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a8556b640 1 -- 172.21.15.28:0/1452241440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a801031a0 msgr2=0x7f3a80074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a8556b640 1 --2- 172.21.15.28:0/1452241440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a801031a0 0x7f3a80074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a8556b640 1 -- 172.21.15.28:0/1452241440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a80101fb0 msgr2=0x7f3a80075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a8556b640 1 --2- 172.21.15.28:0/1452241440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a80101fb0 0x7f3a80075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.919+0000 7f3a8556b640 1 -- 172.21.15.28:0/1452241440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a80075110 con 0x7f3a80103b60 2024-04-21T08:51:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.920+0000 7f3a8556b640 1 --2- 172.21.15.28:0/1452241440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a80103b60 0x7f3a80074880 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000d750 tx=0x7f3a7000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.920+0000 7f3a7dffb640 1 -- 172.21.15.28:0/1452241440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a70017070 con 0x7f3a80103b60 2024-04-21T08:51:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.920+0000 7f3a86fce640 1 -- 172.21.15.28:0/1452241440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a801a22c0 con 0x7f3a80103b60 2024-04-21T08:51:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.920+0000 7f3a7dffb640 1 -- 172.21.15.28:0/1452241440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a700054d0 con 0x7f3a80103b60 2024-04-21T08:51:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.920+0000 7f3a7dffb640 1 -- 172.21.15.28:0/1452241440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a70004c60 con 0x7f3a80103b60 2024-04-21T08:51:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.920+0000 7f3a86fce640 1 -- 172.21.15.28:0/1452241440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a801a27c0 con 0x7f3a80103b60 2024-04-21T08:51:02.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.921+0000 7f3a86fce640 1 -- 172.21.15.28:0/1452241440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a80109320 con 0x7f3a80103b60 2024-04-21T08:51:02.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.922+0000 7f3a7dffb640 1 -- 172.21.15.28:0/1452241440 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3a70009bd0 con 0x7f3a80103b60 2024-04-21T08:51:02.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.922+0000 7f3a7dffb640 1 --2- 172.21.15.28:0/1452241440 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3a680a3c50 0x7f3a680a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.923+0000 7f3a84d6a640 1 --2- 172.21.15.28:0/1452241440 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3a680a3c50 0x7f3a680a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.923+0000 7f3a7dffb640 1 -- 172.21.15.28:0/1452241440 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f3a700d0200 con 0x7f3a80103b60 2024-04-21T08:51:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.923+0000 7f3a84d6a640 1 --2- 172.21.15.28:0/1452241440 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3a680a3c50 0x7f3a680a6110 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3a78006010 tx=0x7f3a78032340 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:02.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:02.927+0000 7f3a7dffb640 1 -- 172.21.15.28:0/1452241440 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3a7009c820 con 0x7f3a80103b60 2024-04-21T08:51:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.165+0000 7f3a86fce640 1 -- 172.21.15.28:0/1452241440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f3a80062b50 con 0x7f3a80103b60 2024-04-21T08:51:03.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.358+0000 7f3a7dffb640 1 -- 172.21.15.28:0/1452241440 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f3a7009c120 con 0x7f3a80103b60 2024-04-21T08:51:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.361+0000 7f3a86fce640 1 -- 172.21.15.28:0/1452241440 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3a680a3c50 msgr2=0x7f3a680a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.361+0000 7f3a86fce640 1 --2- 172.21.15.28:0/1452241440 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3a680a3c50 0x7f3a680a6110 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3a78006010 tx=0x7f3a78032340 comp rx=0 tx=0).stop 2024-04-21T08:51:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.361+0000 7f3a86fce640 1 -- 172.21.15.28:0/1452241440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a80103b60 msgr2=0x7f3a80074880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.361+0000 7f3a86fce640 1 --2- 172.21.15.28:0/1452241440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a80103b60 0x7f3a80074880 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000d750 tx=0x7f3a7000dc10 comp rx=0 tx=0).stop 2024-04-21T08:51:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.361+0000 7f3a86fce640 1 -- 172.21.15.28:0/1452241440 shutdown_connections 2024-04-21T08:51:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.361+0000 7f3a86fce640 1 --2- 172.21.15.28:0/1452241440 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f3a680a3c50 0x7f3a680a6110 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.361+0000 7f3a86fce640 1 --2- 172.21.15.28:0/1452241440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a80103b60 0x7f3a80074880 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.361+0000 7f3a86fce640 1 --2- 172.21.15.28:0/1452241440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a801031a0 0x7f3a80074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.361+0000 7f3a86fce640 1 --2- 172.21.15.28:0/1452241440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a80101fb0 0x7f3a80075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.361+0000 7f3a86fce640 1 -- 172.21.15.28:0/1452241440 >> 172.21.15.28:0/1452241440 conn(0x7f3a800fd540 msgr2=0x7f3a801064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:03.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.364+0000 7f3a86fce640 1 -- 172.21.15.28:0/1452241440 shutdown_connections 2024-04-21T08:51:03.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.364+0000 7f3a86fce640 1 -- 172.21.15.28:0/1452241440 wait complete. 2024-04-21T08:51:03.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-21T08:51:03.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.488+0000 7fdbce89a640 1 -- 172.21.15.28:0/2800452911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc8107cf0 msgr2=0x7fdbc810a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:03.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.488+0000 7fdbce89a640 1 --2- 172.21.15.28:0/2800452911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc8107cf0 0x7fdbc810a0e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0009ca0 tx=0x7fdbc002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.488+0000 7fdbce89a640 1 -- 172.21.15.28:0/2800452911 shutdown_connections 2024-04-21T08:51:03.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.488+0000 7fdbce89a640 1 --2- 172.21.15.28:0/2800452911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc8107cf0 0x7fdbc810a0e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.488+0000 7fdbce89a640 1 --2- 172.21.15.28:0/2800452911 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbc81053c0 0x7fdbc81077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.488+0000 7fdbce89a640 1 --2- 172.21.15.28:0/2800452911 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbc8102a90 0x7fdbc8104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.488+0000 7fdbce89a640 1 -- 172.21.15.28:0/2800452911 >> 172.21.15.28:0/2800452911 conn(0x7fdbc80fc420 msgr2=0x7fdbc80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.488+0000 7fdbce89a640 1 -- 172.21.15.28:0/2800452911 shutdown_connections 2024-04-21T08:51:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.488+0000 7fdbce89a640 1 -- 172.21.15.28:0/2800452911 wait complete. 2024-04-21T08:51:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.489+0000 7fdbce89a640 1 Processor -- start 2024-04-21T08:51:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.489+0000 7fdbce89a640 1 -- start start 2024-04-21T08:51:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.489+0000 7fdbce89a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc8102a90 0x7fdbc819a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.489+0000 7fdbce89a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbc81053c0 0x7fdbc819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.489+0000 7fdbce89a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbc8107cf0 0x7fdbc81a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.489+0000 7fdbce89a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc810daf0 con 0x7fdbc8102a90 2024-04-21T08:51:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.489+0000 7fdbce89a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc810d970 con 0x7fdbc8107cf0 2024-04-21T08:51:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.489+0000 7fdbce89a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc810dc70 con 0x7fdbc81053c0 2024-04-21T08:51:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbc7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc8102a90 0x7fdbc819a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbc7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc8102a90 0x7fdbc819a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37936/0 (socket says 172.21.15.28:37936) 2024-04-21T08:51:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbc7fff640 1 -- 172.21.15.28:0/3429452078 learned_addr learned my addr 172.21.15.28:0/3429452078 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbc77fe640 1 --2- 172.21.15.28:0/3429452078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbc81053c0 0x7fdbc819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbcce37640 1 --2- 172.21.15.28:0/3429452078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbc8107cf0 0x7fdbc81a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbc7fff640 1 -- 172.21.15.28:0/3429452078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbc81053c0 msgr2=0x7fdbc819a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbc7fff640 1 --2- 172.21.15.28:0/3429452078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbc81053c0 0x7fdbc819a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbc7fff640 1 -- 172.21.15.28:0/3429452078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbc8107cf0 msgr2=0x7fdbc81a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbc7fff640 1 --2- 172.21.15.28:0/3429452078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbc8107cf0 0x7fdbc81a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbc7fff640 1 -- 172.21.15.28:0/3429452078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc81a1f20 con 0x7fdbc8102a90 2024-04-21T08:51:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbc7fff640 1 --2- 172.21.15.28:0/3429452078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc8102a90 0x7fdbc819a300 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800e9f0 tx=0x7fdbb800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbc57fa640 1 -- 172.21.15.28:0/3429452078 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb800cd80 con 0x7fdbc8102a90 2024-04-21T08:51:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbce89a640 1 -- 172.21.15.28:0/3429452078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbc81a2210 con 0x7fdbc8102a90 2024-04-21T08:51:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbc57fa640 1 -- 172.21.15.28:0/3429452078 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbb80054d0 con 0x7fdbc8102a90 2024-04-21T08:51:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbce89a640 1 -- 172.21.15.28:0/3429452078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbc81a2710 con 0x7fdbc8102a90 2024-04-21T08:51:03.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.490+0000 7fdbc57fa640 1 -- 172.21.15.28:0/3429452078 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8018630 con 0x7fdbc8102a90 2024-04-21T08:51:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.491+0000 7fdbce89a640 1 -- 172.21.15.28:0/3429452078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbc81097e0 con 0x7fdbc8102a90 2024-04-21T08:51:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.493+0000 7fdbc57fa640 1 -- 172.21.15.28:0/3429452078 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fdbb8004e70 con 0x7fdbc8102a90 2024-04-21T08:51:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.493+0000 7fdbc57fa640 1 --2- 172.21.15.28:0/3429452078 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fdbb00a3ca0 0x7fdbb00a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:03.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.493+0000 7fdbc77fe640 1 -- 172.21.15.28:0/3429452078 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fdbb00a3ca0 msgr2=0x7fdbb00a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/226897930 2024-04-21T08:51:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.493+0000 7fdbc57fa640 1 -- 172.21.15.28:0/3429452078 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7fdbb80d1320 con 0x7fdbc8102a90 2024-04-21T08:51:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.493+0000 7fdbc77fe640 1 --2- 172.21.15.28:0/3429452078 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fdbb00a3ca0 0x7fdbb00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:51:03.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.501+0000 7fdbc57fa640 1 -- 172.21.15.28:0/3429452078 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdbb809d940 con 0x7fdbc8102a90 2024-04-21T08:51:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.691+0000 7fdbce89a640 1 -- 172.21.15.28:0/3429452078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fdbc802d5c0 con 0x7fdbc8102a90 2024-04-21T08:51:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.694+0000 7fdbc77fe640 1 -- 172.21.15.28:0/3429452078 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fdbb00a3ca0 msgr2=0x7fdbb00a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/226897930 2024-04-21T08:51:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.694+0000 7fdbc77fe640 1 --2- 172.21.15.28:0/3429452078 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fdbb00a3ca0 0x7fdbb00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:51:03.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.695+0000 7fdbc57fa640 1 -- 172.21.15.28:0/3429452078 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7fdbb809d240 con 0x7fdbc8102a90 2024-04-21T08:51:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.697+0000 7fdbce89a640 1 -- 172.21.15.28:0/3429452078 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fdbb00a3ca0 msgr2=0x7fdbb00a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:51:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.697+0000 7fdbce89a640 1 --2- 172.21.15.28:0/3429452078 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fdbb00a3ca0 0x7fdbb00a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.697+0000 7fdbce89a640 1 -- 172.21.15.28:0/3429452078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc8102a90 msgr2=0x7fdbc819a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.697+0000 7fdbce89a640 1 --2- 172.21.15.28:0/3429452078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc8102a90 0x7fdbc819a300 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800e9f0 tx=0x7fdbb800edf0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.698+0000 7fdbce89a640 1 -- 172.21.15.28:0/3429452078 shutdown_connections 2024-04-21T08:51:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.698+0000 7fdbce89a640 1 --2- 172.21.15.28:0/3429452078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdbc8107cf0 0x7fdbc81a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.698+0000 7fdbce89a640 1 --2- 172.21.15.28:0/3429452078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdbc81053c0 0x7fdbc819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.698+0000 7fdbce89a640 1 --2- 172.21.15.28:0/3429452078 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7fdbb00a3ca0 0x7fdbb00a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.698+0000 7fdbce89a640 1 --2- 172.21.15.28:0/3429452078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbc8102a90 0x7fdbc819a300 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.698+0000 7fdbce89a640 1 -- 172.21.15.28:0/3429452078 >> 172.21.15.28:0/3429452078 conn(0x7fdbc80fc420 msgr2=0x7fdbc8102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.698+0000 7fdbce89a640 1 -- 172.21.15.28:0/3429452078 shutdown_connections 2024-04-21T08:51:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.698+0000 7fdbce89a640 1 -- 172.21.15.28:0/3429452078 wait complete. 2024-04-21T08:51:03.708 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:03 smithi028 ceph-mon[28003]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-21T08:51:03.708 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:03 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1452241440' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T08:51:03.708 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:03 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: ignoring --setuser ceph since I am not root 2024-04-21T08:51:03.708 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:03 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: ignoring --setgroup ceph since I am not root 2024-04-21T08:51:03.708 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:03 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:03.553+0000 7f6423e25200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:51:03.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-21T08:51:03.725 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:03 smithi152 ceph-mon[31401]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-21T08:51:03.725 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:03 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1452241440' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T08:51:03.726 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:03 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: ignoring --setuser ceph since I am not root 2024-04-21T08:51:03.726 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:03 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: ignoring --setgroup ceph since I am not root 2024-04-21T08:51:03.726 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:03 smithi152 ceph-mgr[32793]: -- 172.21.15.152:0/1067698296 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559f6a9c65a0 con 0x559f6a99ac00 2024-04-21T08:51:03.726 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:03 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:03.541+0000 7fe932a53200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:51:03.726 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:03 smithi171 ceph-mon[31358]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-21T08:51:03.727 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:03 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1452241440' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T08:51:03.727 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:03 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: ignoring --setuser ceph since I am not root 2024-04-21T08:51:03.727 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:03 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: ignoring --setgroup ceph since I am not root 2024-04-21T08:51:03.727 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:03 smithi171 ceph-mgr[33253]: -- 172.21.15.171:0/3829472466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5614c9c685a0 con 0x5614c9c3c800 2024-04-21T08:51:03.727 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:03 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:03.540+0000 7f3da8e3b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:51:03.727 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:03 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:03.726+0000 7f3da8e3b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:51:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.814+0000 7f5798c31640 1 -- 172.21.15.28:0/3101598376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794107d50 msgr2=0x7f579410a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.814+0000 7f5798c31640 1 --2- 172.21.15.28:0/3101598376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794107d50 0x7f579410a140 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f578c009ca0 tx=0x7f578c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.814+0000 7f5798c31640 1 -- 172.21.15.28:0/3101598376 shutdown_connections 2024-04-21T08:51:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.814+0000 7f5798c31640 1 --2- 172.21.15.28:0/3101598376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794107d50 0x7f579410a140 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.814+0000 7f5798c31640 1 --2- 172.21.15.28:0/3101598376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5794105420 0x7f5794107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.814+0000 7f5798c31640 1 --2- 172.21.15.28:0/3101598376 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5794102af0 0x7f5794104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.814+0000 7f5798c31640 1 -- 172.21.15.28:0/3101598376 >> 172.21.15.28:0/3101598376 conn(0x7f57940fc420 msgr2=0x7f57940fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.814+0000 7f5798c31640 1 -- 172.21.15.28:0/3101598376 shutdown_connections 2024-04-21T08:51:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.814+0000 7f5798c31640 1 -- 172.21.15.28:0/3101598376 wait complete. 2024-04-21T08:51:03.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5798c31640 1 Processor -- start 2024-04-21T08:51:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5798c31640 1 -- start start 2024-04-21T08:51:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5798c31640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5794102af0 0x7f5794198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5798c31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794105420 0x7f5794198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5798c31640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5794107d50 0x7f579419f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5798c31640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f579410db00 con 0x7f5794105420 2024-04-21T08:51:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5798c31640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f579410d980 con 0x7f5794107d50 2024-04-21T08:51:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5798c31640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f579410dc80 con 0x7f5794102af0 2024-04-21T08:51:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5791d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794105420 0x7f5794198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5791d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794105420 0x7f5794198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44892/0 (socket says 172.21.15.28:44892) 2024-04-21T08:51:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5791d9b640 1 -- 172.21.15.28:0/1419526625 learned_addr learned my addr 172.21.15.28:0/1419526625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5792d9d640 1 --2- 172.21.15.28:0/1419526625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5794107d50 0x7f579419f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f579259c640 1 --2- 172.21.15.28:0/1419526625 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5794102af0 0x7f5794198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5791d9b640 1 -- 172.21.15.28:0/1419526625 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5794102af0 msgr2=0x7f5794198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:03.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5791d9b640 1 --2- 172.21.15.28:0/1419526625 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5794102af0 0x7f5794198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5791d9b640 1 -- 172.21.15.28:0/1419526625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5794107d50 msgr2=0x7f579419f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:03.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5791d9b640 1 --2- 172.21.15.28:0/1419526625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5794107d50 0x7f579419f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:03.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f5791d9b640 1 -- 172.21.15.28:0/1419526625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f579419fd60 con 0x7f5794105420 2024-04-21T08:51:03.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.815+0000 7f579259c640 1 --2- 172.21.15.28:0/1419526625 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5794102af0 0x7f5794198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:03.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.816+0000 7f5791d9b640 1 --2- 172.21.15.28:0/1419526625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794105420 0x7f5794198680 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f578400e9c0 tx=0x7f578400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:03.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.816+0000 7f57837fe640 1 -- 172.21.15.28:0/1419526625 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578400cd80 con 0x7f5794105420 2024-04-21T08:51:03.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.816+0000 7f57837fe640 1 -- 172.21.15.28:0/1419526625 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57840054d0 con 0x7f5794105420 2024-04-21T08:51:03.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.816+0000 7f5798c31640 1 -- 172.21.15.28:0/1419526625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f579419fff0 con 0x7f5794105420 2024-04-21T08:51:03.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.816+0000 7f57837fe640 1 -- 172.21.15.28:0/1419526625 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5784010430 con 0x7f5794105420 2024-04-21T08:51:03.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.816+0000 7f5798c31640 1 -- 172.21.15.28:0/1419526625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57941a0510 con 0x7f5794105420 2024-04-21T08:51:03.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.817+0000 7f5798c31640 1 -- 172.21.15.28:0/1419526625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57940fdf10 con 0x7f5794105420 2024-04-21T08:51:03.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.819+0000 7f57837fe640 1 -- 172.21.15.28:0/1419526625 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f5784004fb0 con 0x7f5794105420 2024-04-21T08:51:03.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.819+0000 7f57837fe640 1 --2- 172.21.15.28:0/1419526625 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f577c0a3ca0 0x7f577c0a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:03.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.819+0000 7f579259c640 1 -- 172.21.15.28:0/1419526625 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f577c0a3ca0 msgr2=0x7f577c0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/226897930 2024-04-21T08:51:03.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.820+0000 7f57837fe640 1 -- 172.21.15.28:0/1419526625 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f57840d0eb0 con 0x7f5794105420 2024-04-21T08:51:03.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.820+0000 7f579259c640 1 --2- 172.21.15.28:0/1419526625 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f577c0a3ca0 0x7f577c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:51:03.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:03.822+0000 7f57837fe640 1 -- 172.21.15.28:0/1419526625 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5784053ff0 con 0x7f5794105420 2024-04-21T08:51:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:04.010+0000 7f5798c31640 1 -- 172.21.15.28:0/1419526625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f5794062c60 con 0x7f5794105420 2024-04-21T08:51:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:04.013+0000 7f57837fe640 1 -- 172.21.15.28:0/1419526625 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f578409d4d0 con 0x7f5794105420 2024-04-21T08:51:04.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:04.016+0000 7f5798c31640 1 -- 172.21.15.28:0/1419526625 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f577c0a3ca0 msgr2=0x7f577c0a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:51:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:04.016+0000 7f5798c31640 1 --2- 172.21.15.28:0/1419526625 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f577c0a3ca0 0x7f577c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:04.016+0000 7f5798c31640 1 -- 172.21.15.28:0/1419526625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794105420 msgr2=0x7f5794198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:04.016+0000 7f5798c31640 1 --2- 172.21.15.28:0/1419526625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794105420 0x7f5794198680 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f578400e9c0 tx=0x7f578400ee80 comp rx=0 tx=0).stop 2024-04-21T08:51:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:04.016+0000 7f5798c31640 1 -- 172.21.15.28:0/1419526625 shutdown_connections 2024-04-21T08:51:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:04.016+0000 7f5798c31640 1 --2- 172.21.15.28:0/1419526625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5794107d50 0x7f579419f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:04.016+0000 7f5798c31640 1 --2- 172.21.15.28:0/1419526625 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f577c0a3ca0 0x7f577c0a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:04.016+0000 7f5798c31640 1 --2- 172.21.15.28:0/1419526625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794105420 0x7f5794198680 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:04.016+0000 7f5798c31640 1 --2- 172.21.15.28:0/1419526625 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5794102af0 0x7f5794198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:04.016+0000 7f5798c31640 1 -- 172.21.15.28:0/1419526625 >> 172.21.15.28:0/1419526625 conn(0x7f57940fc420 msgr2=0x7f57941033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:04.016+0000 7f5798c31640 1 -- 172.21.15.28:0/1419526625 shutdown_connections 2024-04-21T08:51:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:04.016+0000 7f5798c31640 1 -- 172.21.15.28:0/1419526625 wait complete. 2024-04-21T08:51:04.026 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:03 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:03.751+0000 7f6423e25200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:51:04.026 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:03 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:03.865+0000 7f6423e25200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:51:04.027 INFO:teuthology.run_tasks:Running task kclient... 2024-04-21T08:51:04.040 INFO:tasks.kclient:Mounting kernel clients... 2024-04-21T08:51:04.041 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-21T08:51:04.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:51:04.041 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-21T08:51:04.056 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-21T08:51:04.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:51:04.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:51:04.104 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:03 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:03.725+0000 7fe932a53200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:51:04.104 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:03 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:03.829+0000 7fe932a53200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:51:04.120 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:03 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:03.828+0000 7f3da8e3b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:51:04.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:51:04.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-21T08:51:04.226 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-21T08:51:04.227 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-21T08:51:04.227 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-21T08:51:04.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph fs ls 2024-04-21T08:51:04.298 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:04 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:04.202+0000 7f6423e25200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:51:04.299 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:04 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:04.298+0000 7f6423e25200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:51:04.491 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:51:04.604 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:04 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:04.141+0000 7fe932a53200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:51:04.604 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:04 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:04.227+0000 7fe932a53200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:51:04.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:04 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1452241440' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T08:51:04.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:04 smithi152 ceph-mon[31401]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T08:51:04.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:04 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3429452078' entity='client.admin' 2024-04-21T08:51:04.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:04 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1419526625' entity='client.admin' 2024-04-21T08:51:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:04 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1452241440' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T08:51:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:04 smithi028 ceph-mon[28003]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T08:51:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:04 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3429452078' entity='client.admin' 2024-04-21T08:51:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:04 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1419526625' entity='client.admin' 2024-04-21T08:51:04.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:04 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1452241440' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T08:51:04.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:04 smithi171 ceph-mon[31358]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T08:51:04.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:04 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3429452078' entity='client.admin' 2024-04-21T08:51:04.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:04 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1419526625' entity='client.admin' 2024-04-21T08:51:04.620 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:04 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:04.136+0000 7f3da8e3b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:51:04.620 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:04 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:04.220+0000 7f3da8e3b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:51:05.738 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:05 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:05.399+0000 7f3da8e3b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:51:05.739 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:05 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:05.567+0000 7f3da8e3b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:51:05.739 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:05 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:05.738+0000 7f3da8e3b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:51:05.746 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:05 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:05.600+0000 7f6423e25200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:51:05.755 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:05 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:05.411+0000 7fe932a53200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:51:05.755 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:05 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:05.583+0000 7fe932a53200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:51:06.036 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:05 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:05.782+0000 7f6423e25200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:51:06.036 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:05 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:05.963+0000 7f6423e25200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:51:06.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.296+0000 7f846bdb5700 1 -- 172.21.15.28:0/1976869098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8464102ac0 msgr2=0x7f8464102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:06.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.296+0000 7f846bdb5700 1 --2- 172.21.15.28:0/1976869098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8464102ac0 0x7f8464102f40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8454009990 tx=0x7f8454009ca0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.296+0000 7f846bdb5700 1 -- 172.21.15.28:0/1976869098 shutdown_connections 2024-04-21T08:51:06.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.296+0000 7f846bdb5700 1 --2- 172.21.15.28:0/1976869098 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8464103480 0x7f846410f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.296+0000 7f846bdb5700 1 --2- 172.21.15.28:0/1976869098 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8464108ab0 0x7f8464108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.296+0000 7f846bdb5700 1 --2- 172.21.15.28:0/1976869098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8464102ac0 0x7f8464102f40 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.296+0000 7f846bdb5700 1 -- 172.21.15.28:0/1976869098 >> 172.21.15.28:0/1976869098 conn(0x7f8464079bf0 msgr2=0x7f846407a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:06.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.296+0000 7f846bdb5700 1 -- 172.21.15.28:0/1976869098 shutdown_connections 2024-04-21T08:51:06.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.296+0000 7f846bdb5700 1 -- 172.21.15.28:0/1976869098 wait complete. 2024-04-21T08:51:06.297 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:06 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:06.275+0000 7f6423e25200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:51:06.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.297+0000 7f846bdb5700 1 Processor -- start 2024-04-21T08:51:06.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.297+0000 7f846bdb5700 1 -- start start 2024-04-21T08:51:06.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.297+0000 7f846bdb5700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8464102ac0 0x7f8464199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:06.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.297+0000 7f846bdb5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8464103480 0x7f84641996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:06.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.297+0000 7f846bdb5700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8464108ab0 0x7f846419da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:06.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.297+0000 7f846bdb5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8464112cb0 con 0x7f8464103480 2024-04-21T08:51:06.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.297+0000 7f846bdb5700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8464112b30 con 0x7f8464108ab0 2024-04-21T08:51:06.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.297+0000 7f846bdb5700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8464112e30 con 0x7f8464102ac0 2024-04-21T08:51:06.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.298+0000 7f8469b51700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8464102ac0 0x7f8464199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:06.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.298+0000 7f8469350700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8464103480 0x7f84641996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:06.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.298+0000 7f8469b51700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8464102ac0 0x7f8464199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:48522/0 (socket says 172.21.15.28:48522) 2024-04-21T08:51:06.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.298+0000 7f8469350700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8464103480 0x7f84641996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44902/0 (socket says 172.21.15.28:44902) 2024-04-21T08:51:06.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.298+0000 7f8469b51700 1 -- 172.21.15.28:0/2476521543 learned_addr learned my addr 172.21.15.28:0/2476521543 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:06.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.298+0000 7f846a352700 1 --2- 172.21.15.28:0/2476521543 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8464108ab0 0x7f846419da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:06.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.298+0000 7f8469b51700 1 -- 172.21.15.28:0/2476521543 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8464108ab0 msgr2=0x7f846419da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:06.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.298+0000 7f8469b51700 1 --2- 172.21.15.28:0/2476521543 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8464108ab0 0x7f846419da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.298+0000 7f8469b51700 1 -- 172.21.15.28:0/2476521543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8464103480 msgr2=0x7f84641996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:06.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.298+0000 7f8469b51700 1 --2- 172.21.15.28:0/2476521543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8464103480 0x7f84641996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.298+0000 7f8469b51700 1 -- 172.21.15.28:0/2476521543 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f846419e190 con 0x7f8464102ac0 2024-04-21T08:51:06.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.298+0000 7f846a352700 1 --2- 172.21.15.28:0/2476521543 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8464108ab0 0x7f846419da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:06.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.298+0000 7f8469b51700 1 --2- 172.21.15.28:0/2476521543 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8464102ac0 0x7f8464199180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f846000e9c0 tx=0x7f846000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:06.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.298+0000 7f845affd700 1 -- 172.21.15.28:0/2476521543 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846000cbd0 con 0x7f8464102ac0 2024-04-21T08:51:06.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.299+0000 7f845affd700 1 -- 172.21.15.28:0/2476521543 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f846000ee60 con 0x7f8464102ac0 2024-04-21T08:51:06.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.299+0000 7f846bdb5700 1 -- 172.21.15.28:0/2476521543 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f846419e480 con 0x7f8464102ac0 2024-04-21T08:51:06.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.299+0000 7f845affd700 1 -- 172.21.15.28:0/2476521543 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84600055c0 con 0x7f8464102ac0 2024-04-21T08:51:06.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.299+0000 7f846bdb5700 1 -- 172.21.15.28:0/2476521543 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84641a5d00 con 0x7f8464102ac0 2024-04-21T08:51:06.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.300+0000 7f846bdb5700 1 -- 172.21.15.28:0/2476521543 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8464069730 con 0x7f8464102ac0 2024-04-21T08:51:06.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.301+0000 7f845affd700 1 -- 172.21.15.28:0/2476521543 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f8460004750 con 0x7f8464102ac0 2024-04-21T08:51:06.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.302+0000 7f845affd700 1 --2- 172.21.15.28:0/2476521543 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f84500a3d10 0x7f84500a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:06.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.302+0000 7f8469350700 1 -- 172.21.15.28:0/2476521543 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f84500a3d10 msgr2=0x7f84500a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/226897930 2024-04-21T08:51:06.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.302+0000 7f845affd700 1 -- 172.21.15.28:0/2476521543 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f84600d0b70 con 0x7f8464102ac0 2024-04-21T08:51:06.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.302+0000 7f8469350700 1 --2- 172.21.15.28:0/2476521543 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f84500a3d10 0x7f84500a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:51:06.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.310+0000 7f845affd700 1 -- 172.21.15.28:0/2476521543 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f846009d130 con 0x7f8464102ac0 2024-04-21T08:51:06.324 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:06 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:06.026+0000 7f3da8e3b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:51:06.324 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:06 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:06.118+0000 7f3da8e3b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:51:06.341 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:05 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:05.755+0000 7fe932a53200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:51:06.341 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:06 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:06.043+0000 7fe932a53200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:51:06.341 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:06 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:06.137+0000 7fe932a53200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:51:06.341 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:06 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:06.341+0000 7fe932a53200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:51:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.503+0000 7f8469350700 1 -- 172.21.15.28:0/2476521543 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f84500a3d10 msgr2=0x7f84500a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/226897930 2024-04-21T08:51:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.503+0000 7f8469350700 1 --2- 172.21.15.28:0/2476521543 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f84500a3d10 0x7f84500a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:51:06.604 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:06 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:06.533+0000 7fe932a53200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:51:06.605 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:06 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:06.412+0000 7f6423e25200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:51:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.609+0000 7f846bdb5700 1 -- 172.21.15.28:0/2476521543 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f846419a4e0 con 0x7f8464102ac0 2024-04-21T08:51:06.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.609+0000 7f845affd700 1 -- 172.21.15.28:0/2476521543 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+88 (secure 0 0 0) 0x7f846009ca30 con 0x7f8464102ac0 2024-04-21T08:51:06.609 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-21T08:51:06.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.612+0000 7f846bdb5700 1 -- 172.21.15.28:0/2476521543 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f84500a3d10 msgr2=0x7f84500a61d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:51:06.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.612+0000 7f846bdb5700 1 --2- 172.21.15.28:0/2476521543 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f84500a3d10 0x7f84500a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.612+0000 7f846bdb5700 1 -- 172.21.15.28:0/2476521543 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8464102ac0 msgr2=0x7f8464199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:06.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.612+0000 7f846bdb5700 1 --2- 172.21.15.28:0/2476521543 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8464102ac0 0x7f8464199180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f846000e9c0 tx=0x7f846000ecd0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.612+0000 7f846bdb5700 1 -- 172.21.15.28:0/2476521543 shutdown_connections 2024-04-21T08:51:06.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.612+0000 7f846bdb5700 1 --2- 172.21.15.28:0/2476521543 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8464108ab0 0x7f846419da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.612+0000 7f846bdb5700 1 --2- 172.21.15.28:0/2476521543 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8464102ac0 0x7f8464199180 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.612+0000 7f846bdb5700 1 --2- 172.21.15.28:0/2476521543 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f84500a3d10 0x7f84500a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.612+0000 7f846bdb5700 1 --2- 172.21.15.28:0/2476521543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8464103480 0x7f84641996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:06.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.612+0000 7f846bdb5700 1 -- 172.21.15.28:0/2476521543 >> 172.21.15.28:0/2476521543 conn(0x7f8464079bf0 msgr2=0x7f8464100b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:06.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.612+0000 7f846bdb5700 1 -- 172.21.15.28:0/2476521543 shutdown_connections 2024-04-21T08:51:06.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:06.612+0000 7f846bdb5700 1 -- 172.21.15.28:0/2476521543 wait complete. 2024-04-21T08:51:06.619 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:06 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:06.324+0000 7f3da8e3b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:51:06.620 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:06 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:06.516+0000 7f3da8e3b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:51:06.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:06 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2476521543' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:51:06.936 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:06 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:06.627+0000 7f6423e25200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:51:06.936 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:06 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:06.842+0000 7f6423e25200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:51:07.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:06 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2476521543' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:51:07.118 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:06 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2476521543' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:51:07.273 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-21T08:51:07.273 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-21T08:51:07.274 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-04-21T08:51:07.274 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-21T08:51:07.274 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-21T08:51:07.274 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-21T08:51:07.274 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-21T08:51:07.274 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-21T08:51:07.274 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-21T08:51:07.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:51:07.274 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-04-21T08:51:07.291 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-21T08:51:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-21T08:51:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-04-21T08:51:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:51:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-04-21T08:51:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:51:07.291 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:51:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:51:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-04-21T08:51:07.291 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:51:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:51:07.291 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-04-21T08:51:07.292 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-21T08:51:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-04-21T08:51:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-04-21T08:51:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-21T08:51:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42002sec preferred_lft 42002sec 2024-04-21T08:51:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-21T08:51:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:51:07.292 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:51:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-04-21T08:51:07.292 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-04-21T08:51:07.292 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-21T08:51:07.293 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:51:07.293 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-21T08:51:07.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-04-21T08:51:07.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-04-21T08:51:07.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-04-21T08:51:07.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-21T08:51:07.293 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-21T08:51:07.370 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:07 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:07.118+0000 7f3da8e3b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:51:07.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:51:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:51:07.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:51:07.442 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-21T08:51:07.470 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-04-21T08:51:07.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:51:07.472 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-04-21T08:51:07.528 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-04-21T08:51:07.529 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-21T08:51:07.529 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-21T08:51:07.529 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-21T08:51:07.529 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-21T08:51:07.529 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:51:07.529 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-21T08:51:07.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-21T08:51:07.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-21T08:51:07.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-21T08:51:07.530 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-21T08:51:07.558 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:07 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:07.137+0000 7fe932a53200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:51:07.605 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:07 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:07.491+0000 7f6423e25200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:51:07.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:51:07.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:51:07.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:51:07.658 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-21T08:51:07.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:51:07.713 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-04-21T08:51:07.769 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:51:07.769 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-21T08:51:07.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-21T08:51:07.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-21T08:51:07.770 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-21T08:51:07.832 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:07 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:07.536+0000 7f3da8e3b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:51:07.832 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:07 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:07.632+0000 7f3da8e3b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:51:07.832 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:07 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:07.832+0000 7f3da8e3b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:51:07.854 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:07 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:07.558+0000 7fe932a53200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:51:07.854 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:07 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:07.656+0000 7fe932a53200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:51:07.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:51:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:51:07.889 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-21T08:51:07.889 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:51:07.889 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-21T08:51:07.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-21T08:51:07.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-21T08:51:07.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-21T08:51:07.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-21T08:51:07.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-21T08:51:07.890 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-21T08:51:07.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:51:07.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:51:08.006 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:51:08.006 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-21T08:51:08.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-04-21T08:51:08.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-04-21T08:51:08.006 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-21T08:51:08.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:51:08.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:51:08.085 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:51:08.085 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-21T08:51:08.100 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-21T08:51:08.100 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:51:08.100 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-21T08:51:08.155 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-21T08:51:08.155 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-21T08:51:08.156 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/e0780924-49b5-41e0-9dbf-b1363aadd776 2024-04-21T08:51:08.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:51:08.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/e0780924-49b5-41e0-9dbf-b1363aadd776 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-21T08:51:08.220 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:07 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:07.954+0000 7f6423e25200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:51:08.220 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:08 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:08.064+0000 7f6423e25200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:51:08.244 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-21T08:51:08.245 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-04-21T08:51:08.245 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-04-21T08:51:08.354 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:07 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:07.857+0000 7fe932a53200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:51:08.368 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-21T08:51:08.369 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-04-21T08:51:08.369 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-04-21T08:51:08.369 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.152:3300,172.21.15.171:3300" 2024-04-21T08:51:08.369 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:3300,172.21.15.152:3300,172.21.15.171:3300:/volumes/qa/sv_1/e0780924-49b5-41e0-9dbf-b1363aadd776 2024-04-21T08:51:08.369 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync,key=0,fsid=b51a76c8-ffba-11ee-bc93-c7b262605968" will pass to kernel 2024-04-21T08:51:08.370 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-21T08:51:08.370 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-04-21T08:51:08.370 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-21T08:51:08.370 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-04-21T08:51:08.370 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-21T08:51:08.372 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-21T08:51:08.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:51:08.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-21T08:51:08.443 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T08:51:08.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:51:08.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:51:08.544 INFO:teuthology.orchestra.run.smithi028.stdout:client25061 2024-04-21T08:51:08.544 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:51:08.544 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:51:08.561 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T08:51:08.604 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:08 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:08.294+0000 7f6423e25200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:51:08.620 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:08.357+0000 7f3da8e3b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:51:08.620 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:08.451+0000 7f3da8e3b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:51:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:51:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:51:08.660 INFO:teuthology.orchestra.run.smithi028.stdout:client25061 2024-04-21T08:51:08.661 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T08:51:08.724 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:08 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:08.386+0000 7fe932a53200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:51:08.725 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:08 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:08.480+0000 7fe932a53200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:51:08.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:51:08.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:51:08.759 INFO:teuthology.orchestra.run.smithi028.stdout:client25061 2024-04-21T08:51:08.759 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968.client25061/status') 2024-04-21T08:51:08.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:51:08.820 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.25061 (3)172.21.15.28:0/3286588917 2024-04-21T08:51:08.820 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-04-21T08:51:08.820 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-04-21T08:51:08.820 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-04-21T08:51:08.820 INFO:teuthology.orchestra.run.smithi028.stderr:71 bytes copied, 2.4938e-05 s, 2.8 MB/s 2024-04-21T08:51:08.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:51:08.828 INFO:teuthology.run_tasks:Running task exec... 2024-04-21T08:51:08.836 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T08:51:08.836 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-21T08:51:08.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-21T08:51:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.942+0000 7f0ee5b85640 1 -- 172.21.15.28:0/2847905604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0105360 msgr2=0x7f0ee0107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.942+0000 7f0ee5b85640 1 --2- 172.21.15.28:0/2847905604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0105360 0x7f0ee0107750 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4009900 tx=0x7f0ed402bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.942+0000 7f0ee5b85640 1 -- 172.21.15.28:0/2847905604 shutdown_connections 2024-04-21T08:51:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.942+0000 7f0ee5b85640 1 --2- 172.21.15.28:0/2847905604 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ee0107c90 0x7f0ee010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.942+0000 7f0ee5b85640 1 --2- 172.21.15.28:0/2847905604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0105360 0x7f0ee0107750 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.942+0000 7f0ee5b85640 1 --2- 172.21.15.28:0/2847905604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ee0102a30 0x7f0ee0104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.942+0000 7f0ee5b85640 1 -- 172.21.15.28:0/2847905604 >> 172.21.15.28:0/2847905604 conn(0x7f0ee00fc420 msgr2=0x7f0ee00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.942+0000 7f0ee5b85640 1 -- 172.21.15.28:0/2847905604 shutdown_connections 2024-04-21T08:51:08.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.942+0000 7f0ee5b85640 1 -- 172.21.15.28:0/2847905604 wait complete. 2024-04-21T08:51:08.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.942+0000 7f0ee5b85640 1 Processor -- start 2024-04-21T08:51:08.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0ee5b85640 1 -- start start 2024-04-21T08:51:08.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0ee5b85640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ee0102a30 0x7f0ee0198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:08.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0ee5b85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0105360 0x7f0ee0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:08.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0ee5b85640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ee0107c90 0x7f0ee019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:08.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0ee5b85640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee010daf0 con 0x7f0ee0105360 2024-04-21T08:51:08.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0ee5b85640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee010d970 con 0x7f0ee0102a30 2024-04-21T08:51:08.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0ee5b85640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee010dc70 con 0x7f0ee0107c90 2024-04-21T08:51:08.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0105360 0x7f0ee0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:08.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0105360 0x7f0ee0198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44928/0 (socket says 172.21.15.28:44928) 2024-04-21T08:51:08.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0edeffd640 1 -- 172.21.15.28:0/2533605858 learned_addr learned my addr 172.21.15.28:0/2533605858 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:08.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0edf7fe640 1 --2- 172.21.15.28:0/2533605858 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ee0102a30 0x7f0ee0198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:08.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0edffff640 1 --2- 172.21.15.28:0/2533605858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ee0107c90 0x7f0ee019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:08.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0edeffd640 1 -- 172.21.15.28:0/2533605858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ee0107c90 msgr2=0x7f0ee019f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0edeffd640 1 --2- 172.21.15.28:0/2533605858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ee0107c90 0x7f0ee019f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0edeffd640 1 -- 172.21.15.28:0/2533605858 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ee0102a30 msgr2=0x7f0ee0198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0edeffd640 1 --2- 172.21.15.28:0/2533605858 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ee0102a30 0x7f0ee0198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.943+0000 7f0edeffd640 1 -- 172.21.15.28:0/2533605858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee019fd60 con 0x7f0ee0105360 2024-04-21T08:51:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.944+0000 7f0edeffd640 1 --2- 172.21.15.28:0/2533605858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0105360 0x7f0ee0198680 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4000c00 tx=0x7f0ed4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.944+0000 7f0edcff9640 1 -- 172.21.15.28:0/2533605858 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4043020 con 0x7f0ee0105360 2024-04-21T08:51:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.944+0000 7f0ee5b85640 1 -- 172.21.15.28:0/2533605858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee019fff0 con 0x7f0ee0105360 2024-04-21T08:51:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.944+0000 7f0edcff9640 1 -- 172.21.15.28:0/2533605858 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ed4004350 con 0x7f0ee0105360 2024-04-21T08:51:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.944+0000 7f0edcff9640 1 -- 172.21.15.28:0/2533605858 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4003b80 con 0x7f0ee0105360 2024-04-21T08:51:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.944+0000 7f0ee5b85640 1 -- 172.21.15.28:0/2533605858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee0100b10 con 0x7f0ee0105360 2024-04-21T08:51:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.945+0000 7f0ee5b85640 1 -- 172.21.15.28:0/2533605858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ee00fdf10 con 0x7f0ee0105360 2024-04-21T08:51:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.946+0000 7f0edcff9640 1 -- 172.21.15.28:0/2533605858 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f0ed4046020 con 0x7f0ee0105360 2024-04-21T08:51:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.947+0000 7f0edcff9640 1 --2- 172.21.15.28:0/2533605858 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0ec80a3ca0 0x7f0ec80a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.947+0000 7f0edf7fe640 1 -- 172.21.15.28:0/2533605858 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0ec80a3ca0 msgr2=0x7f0ec80a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/226897930 2024-04-21T08:51:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.947+0000 7f0edcff9640 1 -- 172.21.15.28:0/2533605858 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7f0ed40f2920 con 0x7f0ee0105360 2024-04-21T08:51:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.947+0000 7f0edf7fe640 1 --2- 172.21.15.28:0/2533605858 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0ec80a3ca0 0x7f0ec80a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:51:08.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:08.951+0000 7f0edcff9640 1 -- 172.21.15.28:0/2533605858 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ed404d020 con 0x7f0ee0105360 2024-04-21T08:51:09.038 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:08.693+0000 7f3da8e3b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:51:09.038 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:08.826+0000 7f3da8e3b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:51:09.038 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:08.922+0000 7f3da8e3b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:51:09.038 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:09 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:09.038+0000 7f3da8e3b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:51:09.069 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:08 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:08.725+0000 7fe932a53200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:51:09.070 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:08 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:08.859+0000 7fe932a53200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:51:09.070 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:08 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:08.954+0000 7fe932a53200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:51:09.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:08 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:08.868+0000 7f6423e25200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:51:09.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:08 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:08.970+0000 7f6423e25200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:51:09.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.148+0000 7f0edf7fe640 1 -- 172.21.15.28:0/2533605858 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0ec80a3ca0 msgr2=0x7f0ec80a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/226897930 2024-04-21T08:51:09.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.148+0000 7f0edf7fe640 1 --2- 172.21.15.28:0/2533605858 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0ec80a3ca0 0x7f0ec80a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:51:09.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.180+0000 7f0ee5b85640 1 -- 172.21.15.28:0/2533605858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f0ee0062b50 con 0x7f0ee0105360 2024-04-21T08:51:09.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.232+0000 7f0edcff9640 1 -- 172.21.15.28:0/2533605858 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v25) ==== 118+0+0 (secure 0 0 0) 0x7f0ed40bef40 con 0x7f0ee0105360 2024-04-21T08:51:09.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.234+0000 7f0ee5b85640 1 -- 172.21.15.28:0/2533605858 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0ec80a3ca0 msgr2=0x7f0ec80a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:51:09.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.234+0000 7f0ee5b85640 1 --2- 172.21.15.28:0/2533605858 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0ec80a3ca0 0x7f0ec80a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:09.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.234+0000 7f0ee5b85640 1 -- 172.21.15.28:0/2533605858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0105360 msgr2=0x7f0ee0198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.234+0000 7f0ee5b85640 1 --2- 172.21.15.28:0/2533605858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0105360 0x7f0ee0198680 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4000c00 tx=0x7f0ed4005dc0 comp rx=0 tx=0).stop 2024-04-21T08:51:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.234+0000 7f0ee5b85640 1 -- 172.21.15.28:0/2533605858 shutdown_connections 2024-04-21T08:51:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.234+0000 7f0ee5b85640 1 --2- 172.21.15.28:0/2533605858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ee0107c90 0x7f0ee019f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.234+0000 7f0ee5b85640 1 --2- 172.21.15.28:0/2533605858 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7f0ec80a3ca0 0x7f0ec80a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.234+0000 7f0ee5b85640 1 --2- 172.21.15.28:0/2533605858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee0105360 0x7f0ee0198680 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.234+0000 7f0ee5b85640 1 --2- 172.21.15.28:0/2533605858 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ee0102a30 0x7f0ee0198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.234+0000 7f0ee5b85640 1 -- 172.21.15.28:0/2533605858 >> 172.21.15.28:0/2533605858 conn(0x7f0ee00fc420 msgr2=0x7f0ee0103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.234+0000 7f0ee5b85640 1 -- 172.21.15.28:0/2533605858 shutdown_connections 2024-04-21T08:51:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.234+0000 7f0ee5b85640 1 -- 172.21.15.28:0/2533605858 wait complete. 2024-04-21T08:51:09.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-21T08:51:09.345 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:09 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2533605858' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-21T08:51:09.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.354+0000 7ff98923a640 1 -- 172.21.15.28:0/2955052643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff984102a00 msgr2=0x7ff984109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.354+0000 7ff98923a640 1 --2- 172.21.15.28:0/2955052643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff984102a00 0x7ff984109320 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff974009860 tx=0x7ff97402bb70 comp rx=0 tx=0).stop 2024-04-21T08:51:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.354+0000 7ff98923a640 1 -- 172.21.15.28:0/2955052643 shutdown_connections 2024-04-21T08:51:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.354+0000 7ff98923a640 1 --2- 172.21.15.28:0/2955052643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff984102a00 0x7ff984109320 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.354+0000 7ff98923a640 1 --2- 172.21.15.28:0/2955052643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff984102040 0x7ff9841024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.354+0000 7ff98923a640 1 --2- 172.21.15.28:0/2955052643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff984100e60 0x7ff984101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.354+0000 7ff98923a640 1 -- 172.21.15.28:0/2955052643 >> 172.21.15.28:0/2955052643 conn(0x7ff9840fc420 msgr2=0x7ff9840fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.354+0000 7ff98923a640 1 -- 172.21.15.28:0/2955052643 shutdown_connections 2024-04-21T08:51:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.354+0000 7ff98923a640 1 -- 172.21.15.28:0/2955052643 wait complete. 2024-04-21T08:51:09.355 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:09 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:09.069+0000 7fe932a53200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:51:09.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:09 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2533605858' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-21T08:51:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff98923a640 1 Processor -- start 2024-04-21T08:51:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff98923a640 1 -- start start 2024-04-21T08:51:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff98923a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff984100e60 0x7ff98419a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff98923a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff984102040 0x7ff98419a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff98923a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff984102a00 0x7ff9841a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff98923a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff98410dac0 con 0x7ff984102a00 2024-04-21T08:51:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff98923a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff98410d940 con 0x7ff984100e60 2024-04-21T08:51:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff98923a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff98410dc40 con 0x7ff984102040 2024-04-21T08:51:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff98359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff984102a00 0x7ff9841a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff982d9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff984100e60 0x7ff98419a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff982d9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff984100e60 0x7ff98419a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:45304/0 (socket says 172.21.15.28:45304) 2024-04-21T08:51:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff98259c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff984102040 0x7ff98419a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff982d9d640 1 -- 172.21.15.28:0/799845154 learned_addr learned my addr 172.21.15.28:0/799845154 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff982d9d640 1 -- 172.21.15.28:0/799845154 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff984102040 msgr2=0x7ff98419a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff982d9d640 1 --2- 172.21.15.28:0/799845154 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff984102040 0x7ff98419a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff982d9d640 1 -- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff984102a00 msgr2=0x7ff9841a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.355+0000 7ff982d9d640 1 --2- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff984102a00 0x7ff9841a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.356+0000 7ff982d9d640 1 -- 172.21.15.28:0/799845154 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9841a1f20 con 0x7ff984100e60 2024-04-21T08:51:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.356+0000 7ff98259c640 1 --2- 172.21.15.28:0/799845154 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff984102040 0x7ff98419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.356+0000 7ff982d9d640 1 --2- 172.21.15.28:0/799845154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff984100e60 0x7ff98419a300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00c3a0 tx=0x7ff97c00c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.356+0000 7ff97bfff640 1 -- 172.21.15.28:0/799845154 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c003970 con 0x7ff984100e60 2024-04-21T08:51:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.356+0000 7ff98923a640 1 -- 172.21.15.28:0/799845154 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9841a2210 con 0x7ff984100e60 2024-04-21T08:51:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.356+0000 7ff97bfff640 1 -- 172.21.15.28:0/799845154 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff97c004020 con 0x7ff984100e60 2024-04-21T08:51:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.356+0000 7ff97bfff640 1 -- 172.21.15.28:0/799845154 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c00b920 con 0x7ff984100e60 2024-04-21T08:51:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.356+0000 7ff98923a640 1 -- 172.21.15.28:0/799845154 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9841a2760 con 0x7ff984100e60 2024-04-21T08:51:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.357+0000 7ff98923a640 1 -- 172.21.15.28:0/799845154 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9841a29c0 con 0x7ff984100e60 2024-04-21T08:51:09.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.358+0000 7ff97bfff640 1 -- 172.21.15.28:0/799845154 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7ff97c01d020 con 0x7ff984100e60 2024-04-21T08:51:09.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.359+0000 7ff97bfff640 1 --2- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9700a3ca0 0x7ff9700a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:09.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.359+0000 7ff98259c640 1 -- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9700a3ca0 msgr2=0x7ff9700a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/226897930 2024-04-21T08:51:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.359+0000 7ff97bfff640 1 -- 172.21.15.28:0/799845154 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9187+0+0 (secure 0 0 0) 0x7ff97c0d0900 con 0x7ff984100e60 2024-04-21T08:51:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.359+0000 7ff98259c640 1 --2- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9700a3ca0 0x7ff9700a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:51:09.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.363+0000 7ff97bfff640 1 -- 172.21.15.28:0/799845154 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff97c09cec0 con 0x7ff984100e60 2024-04-21T08:51:09.491 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:09 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2533605858' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-21T08:51:09.491 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:09 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:09.234+0000 7f6423e25200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:51:09.491 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:09 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:09.384+0000 7f6423e25200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:51:09.491 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:09 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:09.489+0000 7f6423e25200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:51:09.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.560+0000 7ff98259c640 1 -- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9700a3ca0 msgr2=0x7ff9700a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/226897930 2024-04-21T08:51:09.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.560+0000 7ff98259c640 1 --2- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9700a3ca0 0x7ff9700a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:51:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.583+0000 7ff98923a640 1 -- 172.21.15.28:0/799845154 --> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7ff98410b400 con 0x7ff9700a3ca0 2024-04-21T08:51:09.620 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:09 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:09.346+0000 7f3da8e3b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:51:09.854 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:09 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:09.373+0000 7fe932a53200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:51:09.854 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:09 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:09.616+0000 7f6423e25200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:51:09.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.961+0000 7ff98259c640 1 -- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9700a3ca0 msgr2=0x7ff9700a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/226897930 2024-04-21T08:51:09.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:09.961+0000 7ff98259c640 1 --2- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9700a3ca0 0x7ff9700a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T08:51:10.238 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:09 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:09.932+0000 7f6423e25200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:51:10.239 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:09 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:09.970+0000 7fe932a53200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:51:10.239 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 08:51:10 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y[32789]: 2024-04-21T08:51:10.182+0000 7fe932a53200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:51:10.239 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:09 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:09.946+0000 7f3da8e3b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:51:10.239 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 08:51:10 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z[33249]: 2024-04-21T08:51:10.158+0000 7f3da8e3b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:51:10.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:10.251+0000 7ff97bfff640 1 -- 172.21.15.28:0/799845154 <== mon.2 v2:172.21.15.152:3300/0 7 ==== mgrmap(e 20) ==== 138663+0+0 (secure 0 0 0) 0x7ff97c098e30 con 0x7ff984100e60 2024-04-21T08:51:10.585 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2533605858' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-21T08:51:10.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-mon[28003]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:51:10.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-mon[28003]: Standby manager daemon z restarted 2024-04-21T08:51:10.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-mon[28003]: Standby manager daemon z started 2024-04-21T08:51:10.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.171:0/375505973' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:51:10.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.171:0/375505973' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:51:10.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.171:0/375505973' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:51:10.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.171:0/375505973' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:51:10.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-mon[28003]: Standby manager daemon y restarted 2024-04-21T08:51:10.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-mon[28003]: Standby manager daemon y started 2024-04-21T08:51:10.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.152:0/3627855776' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:51:10.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.152:0/3627855776' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:51:10.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.152:0/3627855776' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:51:10.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-mon[28003]: from='mgr.? 172.21.15.152:0/3627855776' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:51:10.814 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:10 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2533605858' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-21T08:51:10.814 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:10 smithi171 ceph-mon[31358]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:51:10.814 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:10 smithi171 ceph-mon[31358]: Standby manager daemon z restarted 2024-04-21T08:51:10.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:10 smithi171 ceph-mon[31358]: Standby manager daemon z started 2024-04-21T08:51:10.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:10 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.171:0/375505973' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:51:10.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:10 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.171:0/375505973' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:51:10.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:10 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.171:0/375505973' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:51:10.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:10 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.171:0/375505973' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:51:10.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:10 smithi171 ceph-mon[31358]: Standby manager daemon y restarted 2024-04-21T08:51:10.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:10 smithi171 ceph-mon[31358]: Standby manager daemon y started 2024-04-21T08:51:10.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:10 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.152:0/3627855776' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:51:10.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:10 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.152:0/3627855776' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:51:10.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:10 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.152:0/3627855776' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:51:10.815 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:10 smithi171 ceph-mon[31358]: from='mgr.? 172.21.15.152:0/3627855776' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:51:10.815 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:10 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2533605858' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-21T08:51:10.815 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:10 smithi152 ceph-mon[31401]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:51:10.816 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:10 smithi152 ceph-mon[31401]: Standby manager daemon z restarted 2024-04-21T08:51:10.816 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:10 smithi152 ceph-mon[31401]: Standby manager daemon z started 2024-04-21T08:51:10.816 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:10 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.171:0/375505973' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:51:10.816 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:10 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.171:0/375505973' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:51:10.816 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:10 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.171:0/375505973' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:51:10.816 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:10 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.171:0/375505973' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:51:10.816 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:10 smithi152 ceph-mon[31401]: Standby manager daemon y restarted 2024-04-21T08:51:10.816 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:10 smithi152 ceph-mon[31401]: Standby manager daemon y started 2024-04-21T08:51:10.816 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:10 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.152:0/3627855776' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:51:10.816 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:10 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.152:0/3627855776' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:51:10.816 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:10 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.152:0/3627855776' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:51:10.816 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:10 smithi152 ceph-mon[31401]: from='mgr.? 172.21.15.152:0/3627855776' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:51:10.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:10.762+0000 7ff98259c640 1 -- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9700a3ca0 msgr2=0x7ff9700a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/226897930 2024-04-21T08:51:10.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:10.762+0000 7ff98259c640 1 --2- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9700a3ca0 0x7ff9700a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-21T08:51:10.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:10.835+0000 7ff97bfff640 1 -- 172.21.15.28:0/799845154 <== mon.2 v2:172.21.15.152:3300/0 8 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7ff97c0ce060 con 0x7ff984100e60 2024-04-21T08:51:10.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:10.835+0000 7ff97bfff640 1 -- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9700a3ca0 msgr2=0x7ff9700a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:51:10.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:10.835+0000 7ff97bfff640 1 --2- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/226897930,v1:172.21.15.28:6801/226897930] conn(0x7ff9700a3ca0 0x7ff9700a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:10.855 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:10.585+0000 7f6423e25200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:51:10.855 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:10 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:10.820+0000 7f6423e25200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:51:11.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-21T08:51:11.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: Active manager daemon x restarted 2024-04-21T08:51:11.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: Activating manager daemon x 2024-04-21T08:51:11.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T08:51:11.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: mgrmap e21: x(active, starting, since 0.0117742s), standbys: y, z 2024-04-21T08:51:11.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:51:11.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:51:11.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:51:11.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:51:11.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:51:11.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:51:11.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:51:11.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:51:11.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:51:11.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:51:11.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:51:11.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:51:11.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:51:11.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:51:11.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:51:11.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:51:11.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:51:11.358 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:51:11.358 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: Manager daemon x is now available 2024-04-21T08:51:11.358 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:51:11.358 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:51:11.358 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:11 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:51:11.358 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:11.091+0000 7f6383719700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:51:11.358 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:11.091+0000 7f6383719700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:51:11.358 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:11.091+0000 7f6383719700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:51:11.359 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:11.091+0000 7f6383719700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:51:11.359 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:11.091+0000 7f6383719700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:51:11.359 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:11.091+0000 7f6385f5e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:51:11.359 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:11.091+0000 7f6385f5e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:51:11.359 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:11.091+0000 7f6385f5e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:51:11.359 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:11.091+0000 7f6385f5e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:51:11.359 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:11.091+0000 7f6385f5e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:51:11.359 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-21T08:51:11.359 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: Active manager daemon x restarted 2024-04-21T08:51:11.359 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: Activating manager daemon x 2024-04-21T08:51:11.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T08:51:11.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: mgrmap e21: x(active, starting, since 0.0117742s), standbys: y, z 2024-04-21T08:51:11.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:51:11.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:51:11.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:51:11.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:51:11.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:51:11.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:51:11.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:51:11.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:51:11.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:51:11.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:51:11.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:51:11.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:51:11.360 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:51:11.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:51:11.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:51:11.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:51:11.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:51:11.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:51:11.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: Manager daemon x is now available 2024-04-21T08:51:11.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:51:11.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:51:11.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:51:11.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-21T08:51:11.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: Active manager daemon x restarted 2024-04-21T08:51:11.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: Activating manager daemon x 2024-04-21T08:51:11.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T08:51:11.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: mgrmap e21: x(active, starting, since 0.0117742s), standbys: y, z 2024-04-21T08:51:11.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:51:11.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:51:11.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:51:11.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:51:11.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:51:11.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:51:11.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:51:11.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:51:11.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:51:11.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:51:11.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:51:11.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:51:11.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:51:11.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:51:11.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:51:11.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:51:11.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:51:11.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:51:11.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:51:11.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:51:11.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:51:11.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:51:11.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:51:11.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:51:11.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:51:11.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:51:11.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:51:11.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:51:11.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:51:11.373 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:51:11.373 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:51:11.373 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:51:11.373 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:51:11.373 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: Manager daemon x is now available 2024-04-21T08:51:11.373 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:51:11.373 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:51:11.373 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:11 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:51:11.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:11.842+0000 7ff97bfff640 1 -- 172.21.15.28:0/799845154 <== mon.2 v2:172.21.15.152:3300/0 9 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7ff97c0787e0 con 0x7ff984100e60 2024-04-21T08:51:11.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:11.843+0000 7ff97bfff640 1 --2- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff970035890 0x7ff970035c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:11.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:11.843+0000 7ff97bfff640 1 -- 172.21.15.28:0/799845154 --> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7ff97c0970e0 con 0x7ff970035890 2024-04-21T08:51:11.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:11.847+0000 7ff98259c640 1 --2- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff970035890 0x7ff970035c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:11.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:11.848+0000 7ff98259c640 1 --2- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff970035890 0x7ff970035c70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff98419b810 tx=0x7ff96c005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:11.867+0000 7ff97bfff640 1 -- 172.21.15.28:0/799845154 <== mgr.25040 v2:172.21.15.28:6800/4182301910 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7ff97c0970e0 con 0x7ff970035890 2024-04-21T08:51:11.867 INFO:teuthology.orchestra.run.smithi028.stdout:{} 2024-04-21T08:51:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:11.869+0000 7ff98923a640 1 -- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff970035890 msgr2=0x7ff970035c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:11.869+0000 7ff98923a640 1 --2- 172.21.15.28:0/799845154 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff970035890 0x7ff970035c70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff98419b810 tx=0x7ff96c005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:11.869+0000 7ff98923a640 1 -- 172.21.15.28:0/799845154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff984100e60 msgr2=0x7ff98419a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:11.869+0000 7ff98923a640 1 --2- 172.21.15.28:0/799845154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff984100e60 0x7ff98419a300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00c3a0 tx=0x7ff97c00c860 comp rx=0 tx=0).stop 2024-04-21T08:51:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:11.869+0000 7ff98359e640 1 -- 172.21.15.28:0/799845154 reap_dead start 2024-04-21T08:51:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:11.869+0000 7ff98923a640 1 -- 172.21.15.28:0/799845154 shutdown_connections 2024-04-21T08:51:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:11.869+0000 7ff98923a640 1 -- 172.21.15.28:0/799845154 >> 172.21.15.28:0/799845154 conn(0x7ff9840fc420 msgr2=0x7ff984109ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:11.869+0000 7ff98923a640 1 -- 172.21.15.28:0/799845154 shutdown_connections 2024-04-21T08:51:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:11.869+0000 7ff98923a640 1 -- 172.21.15.28:0/799845154 wait complete. 2024-04-21T08:51:11.882 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-21T08:51:11.903 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-21T08:51:11.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:51:12.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.020+0000 7f0646b8f640 1 -- 172.21.15.28:0/722621146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0640102990 msgr2=0x7f06401092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:12.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.020+0000 7f0646b8f640 1 --2- 172.21.15.28:0/722621146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0640102990 0x7f06401092b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0638009ca0 tx=0x7f063802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.020+0000 7f0646b8f640 1 -- 172.21.15.28:0/722621146 shutdown_connections 2024-04-21T08:51:12.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.020+0000 7f0646b8f640 1 --2- 172.21.15.28:0/722621146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0640102990 0x7f06401092b0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.020+0000 7f0646b8f640 1 --2- 172.21.15.28:0/722621146 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0640101fd0 0x7f0640102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.020+0000 7f0646b8f640 1 --2- 172.21.15.28:0/722621146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0640100e80 0x7f06401012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.020+0000 7f0646b8f640 1 -- 172.21.15.28:0/722621146 >> 172.21.15.28:0/722621146 conn(0x7f06400fc410 msgr2=0x7f06400fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.020+0000 7f0646b8f640 1 -- 172.21.15.28:0/722621146 shutdown_connections 2024-04-21T08:51:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.020+0000 7f0646b8f640 1 -- 172.21.15.28:0/722621146 wait complete. 2024-04-21T08:51:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f0646b8f640 1 Processor -- start 2024-04-21T08:51:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f0646b8f640 1 -- start start 2024-04-21T08:51:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f0646b8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0640100e80 0x7f0640198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f0646b8f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0640101fd0 0x7f06401986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f0646b8f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0640102990 0x7f064019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f0646b8f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f064010c9d0 con 0x7f0640100e80 2024-04-21T08:51:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f0646b8f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f064010c850 con 0x7f0640102990 2024-04-21T08:51:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f0646b8f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f064010cb50 con 0x7f0640101fd0 2024-04-21T08:51:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f064492b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0640100e80 0x7f0640198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f064492b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0640100e80 0x7f0640198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45080/0 (socket says 172.21.15.28:45080) 2024-04-21T08:51:12.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f064492b640 1 -- 172.21.15.28:0/199004797 learned_addr learned my addr 172.21.15.28:0/199004797 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f063ffff640 1 --2- 172.21.15.28:0/199004797 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0640101fd0 0x7f06401986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f064512c640 1 --2- 172.21.15.28:0/199004797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0640102990 0x7f064019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f064492b640 1 -- 172.21.15.28:0/199004797 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0640101fd0 msgr2=0x7f06401986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f064492b640 1 --2- 172.21.15.28:0/199004797 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0640101fd0 0x7f06401986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f064492b640 1 -- 172.21.15.28:0/199004797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0640102990 msgr2=0x7f064019f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f064492b640 1 --2- 172.21.15.28:0/199004797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0640102990 0x7f064019f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.021+0000 7f064492b640 1 -- 172.21.15.28:0/199004797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f064019fd80 con 0x7f0640100e80 2024-04-21T08:51:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.022+0000 7f064492b640 1 --2- 172.21.15.28:0/199004797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0640100e80 0x7f0640198160 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f063400e9f0 tx=0x7f063400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.022+0000 7f063dffb640 1 -- 172.21.15.28:0/199004797 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063400cd80 con 0x7f0640100e80 2024-04-21T08:51:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.022+0000 7f063dffb640 1 -- 172.21.15.28:0/199004797 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06340108e0 con 0x7f0640100e80 2024-04-21T08:51:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.022+0000 7f0646b8f640 1 -- 172.21.15.28:0/199004797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06401a0070 con 0x7f0640100e80 2024-04-21T08:51:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.022+0000 7f063dffb640 1 -- 172.21.15.28:0/199004797 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0634018980 con 0x7f0640100e80 2024-04-21T08:51:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.022+0000 7f0646b8f640 1 -- 172.21.15.28:0/199004797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06401a05c0 con 0x7f0640100e80 2024-04-21T08:51:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.023+0000 7f0646b8f640 1 -- 172.21.15.28:0/199004797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06401a0820 con 0x7f0640100e80 2024-04-21T08:51:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.025+0000 7f063dffb640 1 -- 172.21.15.28:0/199004797 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f0634010430 con 0x7f0640100e80 2024-04-21T08:51:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.025+0000 7f063dffb640 1 --2- 172.21.15.28:0/199004797 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f06300a39b0 0x7f06300a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.026+0000 7f063dffb640 1 -- 172.21.15.28:0/199004797 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f06340d0d80 con 0x7f0640100e80 2024-04-21T08:51:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.026+0000 7f063ffff640 1 --2- 172.21.15.28:0/199004797 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f06300a39b0 0x7f06300a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:12.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.026+0000 7f063ffff640 1 --2- 172.21.15.28:0/199004797 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f06300a39b0 0x7f06300a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0640199670 tx=0x7f062c005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:12.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.029+0000 7f063dffb640 1 -- 172.21.15.28:0/199004797 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f063409d550 con 0x7f0640100e80 2024-04-21T08:51:12.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:11.854+0000 7f6380713700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:51:12.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:11.854+0000 7f6380713700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:51:12.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:11.854+0000 7f6380713700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:51:12.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:11.854+0000 7f6380713700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:51:12.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:51:11 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:51:11.854+0000 7f6380713700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:51:12.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.218+0000 7f0646b8f640 1 -- 172.21.15.28:0/199004797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06401a0a90 con 0x7f0640100e80 2024-04-21T08:51:12.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.221+0000 7f063dffb640 1 -- 172.21.15.28:0/199004797 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25160 (secure 0 0 0) 0x7f063409ce50 con 0x7f0640100e80 2024-04-21T08:51:12.221 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:12.221 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:51:10.823421+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1076282159":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6800/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/926389767":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/892059008":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/2132574390":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:51:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.223+0000 7f0646b8f640 1 -- 172.21.15.28:0/199004797 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f06300a39b0 msgr2=0x7f06300a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.223+0000 7f0646b8f640 1 --2- 172.21.15.28:0/199004797 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f06300a39b0 0x7f06300a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0640199670 tx=0x7f062c005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.223+0000 7f0646b8f640 1 -- 172.21.15.28:0/199004797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0640100e80 msgr2=0x7f0640198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.223+0000 7f0646b8f640 1 --2- 172.21.15.28:0/199004797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0640100e80 0x7f0640198160 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f063400e9f0 tx=0x7f063400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.223+0000 7f0646b8f640 1 -- 172.21.15.28:0/199004797 shutdown_connections 2024-04-21T08:51:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.223+0000 7f0646b8f640 1 --2- 172.21.15.28:0/199004797 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f06300a39b0 0x7f06300a5e70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.223+0000 7f0646b8f640 1 --2- 172.21.15.28:0/199004797 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0640102990 0x7f064019f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.223+0000 7f0646b8f640 1 --2- 172.21.15.28:0/199004797 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0640101fd0 0x7f06401986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.223+0000 7f0646b8f640 1 --2- 172.21.15.28:0/199004797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0640100e80 0x7f0640198160 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.223+0000 7f0646b8f640 1 -- 172.21.15.28:0/199004797 >> 172.21.15.28:0/199004797 conn(0x7f06400fc410 msgr2=0x7f06400fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.223+0000 7f0646b8f640 1 -- 172.21.15.28:0/199004797 shutdown_connections 2024-04-21T08:51:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.223+0000 7f0646b8f640 1 -- 172.21.15.28:0/199004797 wait complete. 2024-04-21T08:51:12.235 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:48:43.543799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:48:44.864564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:48:48.747842+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:51:12.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:51:12.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.345+0000 7fb86d59d640 1 -- 172.21.15.28:0/3863172167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8681031a0 msgr2=0x7fb868103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.345+0000 7fb86d59d640 1 --2- 172.21.15.28:0/3863172167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8681031a0 0x7fb868103620 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb85c009900 tx=0x7fb85c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.345+0000 7fb86d59d640 1 -- 172.21.15.28:0/3863172167 shutdown_connections 2024-04-21T08:51:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.345+0000 7fb86d59d640 1 --2- 172.21.15.28:0/3863172167 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb868103b60 0x7fb86810a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.345+0000 7fb86d59d640 1 --2- 172.21.15.28:0/3863172167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8681031a0 0x7fb868103620 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.345+0000 7fb86d59d640 1 --2- 172.21.15.28:0/3863172167 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb868101fb0 0x7fb8681023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.345+0000 7fb86d59d640 1 -- 172.21.15.28:0/3863172167 >> 172.21.15.28:0/3863172167 conn(0x7fb8680fd520 msgr2=0x7fb8680ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.345+0000 7fb86d59d640 1 -- 172.21.15.28:0/3863172167 shutdown_connections 2024-04-21T08:51:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.345+0000 7fb86d59d640 1 -- 172.21.15.28:0/3863172167 wait complete. 2024-04-21T08:51:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.345+0000 7fb86d59d640 1 Processor -- start 2024-04-21T08:51:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb86d59d640 1 -- start start 2024-04-21T08:51:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb86d59d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb868101fb0 0x7fb868198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb86d59d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8681031a0 0x7fb8681986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb86d59d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb868103b60 0x7fb86819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb86d59d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb86810daf0 con 0x7fb8681031a0 2024-04-21T08:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb86d59d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb86810d970 con 0x7fb868101fb0 2024-04-21T08:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb86d59d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb86810dc70 con 0x7fb868103b60 2024-04-21T08:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb8667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8681031a0 0x7fb8681986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb8667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8681031a0 0x7fb8681986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45112/0 (socket says 172.21.15.28:45112) 2024-04-21T08:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb8667fc640 1 -- 172.21.15.28:0/1447626766 learned_addr learned my addr 172.21.15.28:0/1447626766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb8677fe640 1 --2- 172.21.15.28:0/1447626766 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb868103b60 0x7fb86819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb866ffd640 1 --2- 172.21.15.28:0/1447626766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb868101fb0 0x7fb868198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb8667fc640 1 -- 172.21.15.28:0/1447626766 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb868103b60 msgr2=0x7fb86819f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb8667fc640 1 --2- 172.21.15.28:0/1447626766 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb868103b60 0x7fb86819f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb8667fc640 1 -- 172.21.15.28:0/1447626766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb868101fb0 msgr2=0x7fb868198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb8667fc640 1 --2- 172.21.15.28:0/1447626766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb868101fb0 0x7fb868198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb8667fc640 1 -- 172.21.15.28:0/1447626766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb86819fd90 con 0x7fb8681031a0 2024-04-21T08:51:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.346+0000 7fb8667fc640 1 --2- 172.21.15.28:0/1447626766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8681031a0 0x7fb8681986b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb85c036040 tx=0x7fb85c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:12.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.347+0000 7fb84ffff640 1 -- 172.21.15.28:0/1447626766 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb85c043020 con 0x7fb8681031a0 2024-04-21T08:51:12.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.347+0000 7fb86d59d640 1 -- 172.21.15.28:0/1447626766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8681a0020 con 0x7fb8681031a0 2024-04-21T08:51:12.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.347+0000 7fb84ffff640 1 -- 172.21.15.28:0/1447626766 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb85c0054c0 con 0x7fb8681031a0 2024-04-21T08:51:12.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.347+0000 7fb84ffff640 1 -- 172.21.15.28:0/1447626766 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb85c003db0 con 0x7fb8681031a0 2024-04-21T08:51:12.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.347+0000 7fb86d59d640 1 -- 172.21.15.28:0/1447626766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8681a0570 con 0x7fb8681031a0 2024-04-21T08:51:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.348+0000 7fb86d59d640 1 -- 172.21.15.28:0/1447626766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb868109320 con 0x7fb8681031a0 2024-04-21T08:51:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.349+0000 7fb84ffff640 1 -- 172.21.15.28:0/1447626766 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fb85c046020 con 0x7fb8681031a0 2024-04-21T08:51:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.349+0000 7fb84ffff640 1 --2- 172.21.15.28:0/1447626766 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb8500a39b0 0x7fb8500a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.349+0000 7fb866ffd640 1 --2- 172.21.15.28:0/1447626766 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb8500a39b0 0x7fb8500a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.349+0000 7fb84ffff640 1 -- 172.21.15.28:0/1447626766 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fb85c0f29a0 con 0x7fb8681031a0 2024-04-21T08:51:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.350+0000 7fb866ffd640 1 --2- 172.21.15.28:0/1447626766 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb8500a39b0 0x7fb8500a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb8580058c0 tx=0x7fb858005850 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:12.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.354+0000 7fb84ffff640 1 -- 172.21.15.28:0/1447626766 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb85c04d020 con 0x7fb8681031a0 2024-04-21T08:51:12.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.541+0000 7fb86d59d640 1 -- 172.21.15.28:0/1447626766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb868062b50 con 0x7fb8681031a0 2024-04-21T08:51:12.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.541+0000 7fb84ffff640 1 -- 172.21.15.28:0/1447626766 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fb85c0bf1b0 con 0x7fb8681031a0 2024-04-21T08:51:12.541 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T08:51:12.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.543+0000 7fb86d59d640 1 -- 172.21.15.28:0/1447626766 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb8500a39b0 msgr2=0x7fb8500a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:12.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.543+0000 7fb86d59d640 1 --2- 172.21.15.28:0/1447626766 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb8500a39b0 0x7fb8500a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb8580058c0 tx=0x7fb858005850 comp rx=0 tx=0).stop 2024-04-21T08:51:12.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.543+0000 7fb86d59d640 1 -- 172.21.15.28:0/1447626766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8681031a0 msgr2=0x7fb8681986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:12.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.543+0000 7fb86d59d640 1 --2- 172.21.15.28:0/1447626766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8681031a0 0x7fb8681986b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb85c036040 tx=0x7fb85c0042f0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.543+0000 7fb86d59d640 1 -- 172.21.15.28:0/1447626766 shutdown_connections 2024-04-21T08:51:12.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.543+0000 7fb86d59d640 1 --2- 172.21.15.28:0/1447626766 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb8500a39b0 0x7fb8500a5e70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.543+0000 7fb86d59d640 1 --2- 172.21.15.28:0/1447626766 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb868103b60 0x7fb86819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.543+0000 7fb86d59d640 1 --2- 172.21.15.28:0/1447626766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8681031a0 0x7fb8681986b0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.543+0000 7fb86d59d640 1 --2- 172.21.15.28:0/1447626766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb868101fb0 0x7fb868198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.543+0000 7fb86d59d640 1 -- 172.21.15.28:0/1447626766 >> 172.21.15.28:0/1447626766 conn(0x7fb8680fd520 msgr2=0x7fb8681064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:12.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.543+0000 7fb86d59d640 1 -- 172.21.15.28:0/1447626766 shutdown_connections 2024-04-21T08:51:12.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.543+0000 7fb86d59d640 1 -- 172.21.15.28:0/1447626766 wait complete. 2024-04-21T08:51:12.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:51:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.666+0000 7f86a71f4640 1 -- 172.21.15.28:0/4065552375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0103be0 msgr2=0x7f86a010a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.666+0000 7f86a71f4640 1 --2- 172.21.15.28:0/4065552375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0103be0 0x7f86a010a470 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8698009cd0 tx=0x7f869802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.666+0000 7f86a71f4640 1 -- 172.21.15.28:0/4065552375 shutdown_connections 2024-04-21T08:51:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.666+0000 7f86a71f4640 1 --2- 172.21.15.28:0/4065552375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0103be0 0x7f86a010a470 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.666+0000 7f86a71f4640 1 --2- 172.21.15.28:0/4065552375 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f86a0103220 0x7f86a01036a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.666+0000 7f86a71f4640 1 --2- 172.21.15.28:0/4065552375 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86a0100250 0x7f86a0100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.666+0000 7f86a71f4640 1 -- 172.21.15.28:0/4065552375 >> 172.21.15.28:0/4065552375 conn(0x7f86a00fbb80 msgr2=0x7f86a00fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.666+0000 7f86a71f4640 1 -- 172.21.15.28:0/4065552375 shutdown_connections 2024-04-21T08:51:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.666+0000 7f86a71f4640 1 -- 172.21.15.28:0/4065552375 wait complete. 2024-04-21T08:51:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.666+0000 7f86a71f4640 1 Processor -- start 2024-04-21T08:51:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a71f4640 1 -- start start 2024-04-21T08:51:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a71f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0100250 0x7f86a01981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a71f4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f86a0103220 0x7f86a01986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a71f4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86a0103be0 0x7f86a019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a71f4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f86a010daf0 con 0x7f86a0100250 2024-04-21T08:51:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a71f4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f86a010d970 con 0x7f86a0103220 2024-04-21T08:51:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a71f4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f86a010dc70 con 0x7f86a0103be0 2024-04-21T08:51:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a4f90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0100250 0x7f86a01981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a4f90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0100250 0x7f86a01981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45136/0 (socket says 172.21.15.28:45136) 2024-04-21T08:51:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a4f90640 1 -- 172.21.15.28:0/4034214934 learned_addr learned my addr 172.21.15.28:0/4034214934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f869ffff640 1 --2- 172.21.15.28:0/4034214934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f86a0103220 0x7f86a01986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a5791640 1 --2- 172.21.15.28:0/4034214934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86a0103be0 0x7f86a019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a4f90640 1 -- 172.21.15.28:0/4034214934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86a0103be0 msgr2=0x7f86a019f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a4f90640 1 --2- 172.21.15.28:0/4034214934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86a0103be0 0x7f86a019f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a4f90640 1 -- 172.21.15.28:0/4034214934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f86a0103220 msgr2=0x7f86a01986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a4f90640 1 --2- 172.21.15.28:0/4034214934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f86a0103220 0x7f86a01986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a4f90640 1 -- 172.21.15.28:0/4034214934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a019fdd0 con 0x7f86a0100250 2024-04-21T08:51:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f86a4f90640 1 --2- 172.21.15.28:0/4034214934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0100250 0x7f86a01981b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f869400e9c0 tx=0x7f869400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.667+0000 7f869dffb640 1 -- 172.21.15.28:0/4034214934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869400cd80 con 0x7f86a0100250 2024-04-21T08:51:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.668+0000 7f869dffb640 1 -- 172.21.15.28:0/4034214934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86940035f0 con 0x7f86a0100250 2024-04-21T08:51:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.668+0000 7f86a71f4640 1 -- 172.21.15.28:0/4034214934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a01a00c0 con 0x7f86a0100250 2024-04-21T08:51:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.668+0000 7f869dffb640 1 -- 172.21.15.28:0/4034214934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694010690 con 0x7f86a0100250 2024-04-21T08:51:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.668+0000 7f86a71f4640 1 -- 172.21.15.28:0/4034214934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a01a05c0 con 0x7f86a0100250 2024-04-21T08:51:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.669+0000 7f86a71f4640 1 -- 172.21.15.28:0/4034214934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a01094c0 con 0x7f86a0100250 2024-04-21T08:51:12.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.670+0000 7f869dffb640 1 -- 172.21.15.28:0/4034214934 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f8694004000 con 0x7f86a0100250 2024-04-21T08:51:12.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.671+0000 7f869dffb640 1 --2- 172.21.15.28:0/4034214934 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f86900a39b0 0x7f86900a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:12.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.671+0000 7f869dffb640 1 -- 172.21.15.28:0/4034214934 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f86940d09b0 con 0x7f86a0100250 2024-04-21T08:51:12.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.671+0000 7f869ffff640 1 --2- 172.21.15.28:0/4034214934 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f86900a39b0 0x7f86900a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:12.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.671+0000 7f869ffff640 1 --2- 172.21.15.28:0/4034214934 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f86900a39b0 0x7f86900a5e70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f86a01996c0 tx=0x7f868c005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:12.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.675+0000 7f869dffb640 1 -- 172.21.15.28:0/4034214934 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f869409d1c0 con 0x7f86a0100250 2024-04-21T08:51:12.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.850+0000 7f869dffb640 1 -- 172.21.15.28:0/4034214934 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f8694099130 con 0x7f86a0100250 2024-04-21T08:51:12.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.884+0000 7f86a71f4640 1 -- 172.21.15.28:0/4034214934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f86a0062b50 con 0x7f86a0100250 2024-04-21T08:51:12.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.884+0000 7f869dffb640 1 -- 172.21.15.28:0/4034214934 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f8694057b90 con 0x7f86a0100250 2024-04-21T08:51:12.884 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:51:12.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.886+0000 7f86a71f4640 1 -- 172.21.15.28:0/4034214934 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f86900a39b0 msgr2=0x7f86900a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.886+0000 7f86a71f4640 1 --2- 172.21.15.28:0/4034214934 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f86900a39b0 0x7f86900a5e70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f86a01996c0 tx=0x7f868c005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.886+0000 7f86a71f4640 1 -- 172.21.15.28:0/4034214934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0100250 msgr2=0x7f86a01981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.886+0000 7f86a71f4640 1 --2- 172.21.15.28:0/4034214934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0100250 0x7f86a01981b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f869400e9c0 tx=0x7f869400ee80 comp rx=0 tx=0).stop 2024-04-21T08:51:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.887+0000 7f86a71f4640 1 -- 172.21.15.28:0/4034214934 shutdown_connections 2024-04-21T08:51:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.887+0000 7f86a71f4640 1 --2- 172.21.15.28:0/4034214934 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f86900a39b0 0x7f86900a5e70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.887+0000 7f86a71f4640 1 --2- 172.21.15.28:0/4034214934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86a0103be0 0x7f86a019f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.887+0000 7f86a71f4640 1 --2- 172.21.15.28:0/4034214934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f86a0103220 0x7f86a01986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.887+0000 7f86a71f4640 1 --2- 172.21.15.28:0/4034214934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a0100250 0x7f86a01981b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.887+0000 7f86a71f4640 1 -- 172.21.15.28:0/4034214934 >> 172.21.15.28:0/4034214934 conn(0x7f86a00fbb80 msgr2=0x7f86a0106690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.887+0000 7f86a71f4640 1 -- 172.21.15.28:0/4034214934 shutdown_connections 2024-04-21T08:51:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:12.887+0000 7f86a71f4640 1 -- 172.21.15.28:0/4034214934 wait complete. 2024-04-21T08:51:12.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:51:13.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.021+0000 7f6508ab3640 1 -- 172.21.15.28:0/2319014073 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65041020c0 msgr2=0x7f6504102540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:13.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.021+0000 7f6508ab3640 1 --2- 172.21.15.28:0/2319014073 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65041020c0 0x7f6504102540 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f64f8009860 tx=0x7f64f802bb70 comp rx=0 tx=0).stop 2024-04-21T08:51:13.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.021+0000 7f6508ab3640 1 -- 172.21.15.28:0/2319014073 shutdown_connections 2024-04-21T08:51:13.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.021+0000 7f6508ab3640 1 --2- 172.21.15.28:0/2319014073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6504102a80 0x7f6504109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.021+0000 7f6508ab3640 1 --2- 172.21.15.28:0/2319014073 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65041020c0 0x7f6504102540 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.021+0000 7f6508ab3640 1 --2- 172.21.15.28:0/2319014073 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f65041013b0 0x7f65041017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.021+0000 7f6508ab3640 1 -- 172.21.15.28:0/2319014073 >> 172.21.15.28:0/2319014073 conn(0x7f65040fbb80 msgr2=0x7f65040fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:13.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.021+0000 7f6508ab3640 1 -- 172.21.15.28:0/2319014073 shutdown_connections 2024-04-21T08:51:13.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.021+0000 7f6508ab3640 1 -- 172.21.15.28:0/2319014073 wait complete. 2024-04-21T08:51:13.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.022+0000 7f6508ab3640 1 Processor -- start 2024-04-21T08:51:13.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.022+0000 7f6508ab3640 1 -- start start 2024-04-21T08:51:13.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.022+0000 7f6508ab3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65041013b0 0x7f6504100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:13.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.022+0000 7f6508ab3640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65041020c0 0x7f650419cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:13.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.022+0000 7f6508ab3640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6504102a80 0x7f650419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:13.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.022+0000 7f6508ab3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f650410c970 con 0x7f65041013b0 2024-04-21T08:51:13.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.022+0000 7f6508ab3640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f650410c7f0 con 0x7f6504102a80 2024-04-21T08:51:13.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.022+0000 7f6508ab3640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f650410caf0 con 0x7f65041020c0 2024-04-21T08:51:13.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f650259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65041013b0 0x7f6504100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:13.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f6501d9b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65041020c0 0x7f650419cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:13.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f6501d9b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65041020c0 0x7f650419cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:48794/0 (socket says 172.21.15.28:48794) 2024-04-21T08:51:13.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f650259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65041013b0 0x7f6504100a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45180/0 (socket says 172.21.15.28:45180) 2024-04-21T08:51:13.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f6501d9b640 1 -- 172.21.15.28:0/3252888089 learned_addr learned my addr 172.21.15.28:0/3252888089 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:13.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f6502d9d640 1 --2- 172.21.15.28:0/3252888089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6504102a80 0x7f650419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:13.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f6501d9b640 1 -- 172.21.15.28:0/3252888089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6504102a80 msgr2=0x7f650419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:13.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f6501d9b640 1 --2- 172.21.15.28:0/3252888089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6504102a80 0x7f650419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f6501d9b640 1 -- 172.21.15.28:0/3252888089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65041013b0 msgr2=0x7f6504100a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:13.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f6501d9b640 1 --2- 172.21.15.28:0/3252888089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65041013b0 0x7f6504100a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f6501d9b640 1 -- 172.21.15.28:0/3252888089 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f650419fec0 con 0x7f65041020c0 2024-04-21T08:51:13.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f6502d9d640 1 --2- 172.21.15.28:0/3252888089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6504102a80 0x7f650419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:13.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f6501d9b640 1 --2- 172.21.15.28:0/3252888089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65041020c0 0x7f650419cf90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f64f80057c0 tx=0x7f64f80042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:13.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f64f37fe640 1 -- 172.21.15.28:0/3252888089 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f8041030 con 0x7f65041020c0 2024-04-21T08:51:13.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f64f37fe640 1 -- 172.21.15.28:0/3252888089 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64f802c9c0 con 0x7f65041020c0 2024-04-21T08:51:13.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f6508ab3640 1 -- 172.21.15.28:0/3252888089 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65041a00f0 con 0x7f65041020c0 2024-04-21T08:51:13.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.023+0000 7f64f37fe640 1 -- 172.21.15.28:0/3252888089 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f8039040 con 0x7f65041020c0 2024-04-21T08:51:13.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.024+0000 7f6508ab3640 1 -- 172.21.15.28:0/3252888089 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65041a05e0 con 0x7f65041020c0 2024-04-21T08:51:13.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.025+0000 7f6508ab3640 1 -- 172.21.15.28:0/3252888089 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6504108360 con 0x7f65041020c0 2024-04-21T08:51:13.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.025+0000 7f64f37fe640 1 -- 172.21.15.28:0/3252888089 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f64f8044070 con 0x7f65041020c0 2024-04-21T08:51:13.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.026+0000 7f64f37fe640 1 --2- 172.21.15.28:0/3252888089 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f64ec0a3db0 0x7f64ec0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:13.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.026+0000 7f64f37fe640 1 -- 172.21.15.28:0/3252888089 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f64f80f2290 con 0x7f65041020c0 2024-04-21T08:51:13.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.026+0000 7f650259c640 1 --2- 172.21.15.28:0/3252888089 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f64ec0a3db0 0x7f64ec0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:13.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.027+0000 7f650259c640 1 --2- 172.21.15.28:0/3252888089 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f64ec0a3db0 0x7f64ec0a6270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f64f40098d0 tx=0x7f64f4010040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:13.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.032+0000 7f64f37fe640 1 -- 172.21.15.28:0/3252888089 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f64f80be530 con 0x7f65041020c0 2024-04-21T08:51:13.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:12 smithi152 ceph-mon[31401]: [21/Apr/2024:08:51:11] ENGINE Bus STARTING 2024-04-21T08:51:13.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:12 smithi152 ceph-mon[31401]: [21/Apr/2024:08:51:11] ENGINE Serving on https://172.21.15.28:7150 2024-04-21T08:51:13.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:12 smithi152 ceph-mon[31401]: [21/Apr/2024:08:51:11] ENGINE Serving on http://172.21.15.28:8765 2024-04-21T08:51:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:12 smithi152 ceph-mon[31401]: [21/Apr/2024:08:51:11] ENGINE Bus STARTED 2024-04-21T08:51:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:12 smithi152 ceph-mon[31401]: mgrmap e22: x(active, since 1.02009s), standbys: y, z 2024-04-21T08:51:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:12 smithi152 ceph-mon[31401]: from='client.25073 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:51:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:12 smithi152 ceph-mon[31401]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:51:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:12 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/199004797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:13.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:12 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1447626766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:13.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:12 smithi028 ceph-mon[28003]: [21/Apr/2024:08:51:11] ENGINE Bus STARTING 2024-04-21T08:51:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:12 smithi028 ceph-mon[28003]: [21/Apr/2024:08:51:11] ENGINE Serving on https://172.21.15.28:7150 2024-04-21T08:51:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:12 smithi028 ceph-mon[28003]: [21/Apr/2024:08:51:11] ENGINE Serving on http://172.21.15.28:8765 2024-04-21T08:51:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:12 smithi028 ceph-mon[28003]: [21/Apr/2024:08:51:11] ENGINE Bus STARTED 2024-04-21T08:51:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:12 smithi028 ceph-mon[28003]: mgrmap e22: x(active, since 1.02009s), standbys: y, z 2024-04-21T08:51:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:12 smithi028 ceph-mon[28003]: from='client.25073 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:51:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:12 smithi028 ceph-mon[28003]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:51:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:12 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/199004797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:13.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:12 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1447626766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:12 smithi171 ceph-mon[31358]: [21/Apr/2024:08:51:11] ENGINE Bus STARTING 2024-04-21T08:51:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:12 smithi171 ceph-mon[31358]: [21/Apr/2024:08:51:11] ENGINE Serving on https://172.21.15.28:7150 2024-04-21T08:51:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:12 smithi171 ceph-mon[31358]: [21/Apr/2024:08:51:11] ENGINE Serving on http://172.21.15.28:8765 2024-04-21T08:51:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:12 smithi171 ceph-mon[31358]: [21/Apr/2024:08:51:11] ENGINE Bus STARTED 2024-04-21T08:51:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:12 smithi171 ceph-mon[31358]: mgrmap e22: x(active, since 1.02009s), standbys: y, z 2024-04-21T08:51:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:12 smithi171 ceph-mon[31358]: from='client.25073 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:51:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:12 smithi171 ceph-mon[31358]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:51:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:12 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/199004797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:13.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:12 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1447626766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:13.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.234+0000 7f6508ab3640 1 -- 172.21.15.28:0/3252888089 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f650402d5c0 con 0x7f65041020c0 2024-04-21T08:51:13.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.235+0000 7f64f37fe640 1 -- 172.21.15.28:0/3252888089 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f64f80bde30 con 0x7f65041020c0 2024-04-21T08:51:13.235 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-21T08:51:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.237+0000 7f6508ab3640 1 -- 172.21.15.28:0/3252888089 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f64ec0a3db0 msgr2=0x7f64ec0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.237+0000 7f6508ab3640 1 --2- 172.21.15.28:0/3252888089 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f64ec0a3db0 0x7f64ec0a6270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f64f40098d0 tx=0x7f64f4010040 comp rx=0 tx=0).stop 2024-04-21T08:51:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.237+0000 7f6508ab3640 1 -- 172.21.15.28:0/3252888089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65041020c0 msgr2=0x7f650419cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.237+0000 7f6508ab3640 1 --2- 172.21.15.28:0/3252888089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65041020c0 0x7f650419cf90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f64f80057c0 tx=0x7f64f80042f0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.237+0000 7f6508ab3640 1 -- 172.21.15.28:0/3252888089 shutdown_connections 2024-04-21T08:51:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.237+0000 7f6508ab3640 1 --2- 172.21.15.28:0/3252888089 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f64ec0a3db0 0x7f64ec0a6270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.237+0000 7f6508ab3640 1 --2- 172.21.15.28:0/3252888089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6504102a80 0x7f650419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.237+0000 7f6508ab3640 1 --2- 172.21.15.28:0/3252888089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f65041020c0 0x7f650419cf90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.237+0000 7f6508ab3640 1 --2- 172.21.15.28:0/3252888089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65041013b0 0x7f6504100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.237+0000 7f6508ab3640 1 -- 172.21.15.28:0/3252888089 >> 172.21.15.28:0/3252888089 conn(0x7f65040fbb80 msgr2=0x7f6504105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.237+0000 7f6508ab3640 1 -- 172.21.15.28:0/3252888089 shutdown_connections 2024-04-21T08:51:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.237+0000 7f6508ab3640 1 -- 172.21.15.28:0/3252888089 wait complete. 2024-04-21T08:51:13.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:51:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.355+0000 7f17cd723640 1 -- 172.21.15.28:0/2833056743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c8105360 msgr2=0x7f17c8107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.355+0000 7f17cd723640 1 --2- 172.21.15.28:0/2833056743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c8105360 0x7f17c8107750 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009900 tx=0x7f17b802bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.355+0000 7f17cd723640 1 -- 172.21.15.28:0/2833056743 shutdown_connections 2024-04-21T08:51:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.355+0000 7f17cd723640 1 --2- 172.21.15.28:0/2833056743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17c8107c90 0x7f17c810a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.355+0000 7f17cd723640 1 --2- 172.21.15.28:0/2833056743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c8105360 0x7f17c8107750 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.355+0000 7f17cd723640 1 --2- 172.21.15.28:0/2833056743 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17c8102a30 0x7f17c8104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.355+0000 7f17cd723640 1 -- 172.21.15.28:0/2833056743 >> 172.21.15.28:0/2833056743 conn(0x7f17c80fc420 msgr2=0x7f17c80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.355+0000 7f17cd723640 1 -- 172.21.15.28:0/2833056743 shutdown_connections 2024-04-21T08:51:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.355+0000 7f17cd723640 1 -- 172.21.15.28:0/2833056743 wait complete. 2024-04-21T08:51:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.356+0000 7f17cd723640 1 Processor -- start 2024-04-21T08:51:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.356+0000 7f17cd723640 1 -- start start 2024-04-21T08:51:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.356+0000 7f17cd723640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17c8102a30 0x7f17c8198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.356+0000 7f17cd723640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c8105360 0x7f17c8198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.356+0000 7f17cd723640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17c8107c90 0x7f17c819f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.356+0000 7f17cd723640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17c810dab0 con 0x7f17c8105360 2024-04-21T08:51:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.356+0000 7f17cd723640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f17c810d930 con 0x7f17c8107c90 2024-04-21T08:51:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.356+0000 7f17cd723640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f17c810dc30 con 0x7f17c8102a30 2024-04-21T08:51:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.356+0000 7f17c67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c8105360 0x7f17c8198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.356+0000 7f17c67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c8105360 0x7f17c8198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45194/0 (socket says 172.21.15.28:45194) 2024-04-21T08:51:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.356+0000 7f17c67fc640 1 -- 172.21.15.28:0/1197639851 learned_addr learned my addr 172.21.15.28:0/1197639851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.356+0000 7f17c6ffd640 1 --2- 172.21.15.28:0/1197639851 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17c8102a30 0x7f17c8198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.356+0000 7f17c77fe640 1 --2- 172.21.15.28:0/1197639851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17c8107c90 0x7f17c819f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.357+0000 7f17c67fc640 1 -- 172.21.15.28:0/1197639851 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17c8102a30 msgr2=0x7f17c8198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.357+0000 7f17c67fc640 1 --2- 172.21.15.28:0/1197639851 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17c8102a30 0x7f17c8198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.357+0000 7f17c67fc640 1 -- 172.21.15.28:0/1197639851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17c8107c90 msgr2=0x7f17c819f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.357+0000 7f17c67fc640 1 --2- 172.21.15.28:0/1197639851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17c8107c90 0x7f17c819f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.357+0000 7f17c67fc640 1 -- 172.21.15.28:0/1197639851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c819fe70 con 0x7f17c8105360 2024-04-21T08:51:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.357+0000 7f17c67fc640 1 --2- 172.21.15.28:0/1197639851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c8105360 0x7f17c8198790 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f17b8000c00 tx=0x7f17b8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.357+0000 7f17affff640 1 -- 172.21.15.28:0/1197639851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8043020 con 0x7f17c8105360 2024-04-21T08:51:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.357+0000 7f17cd723640 1 -- 172.21.15.28:0/1197639851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c81a0100 con 0x7f17c8105360 2024-04-21T08:51:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.357+0000 7f17affff640 1 -- 172.21.15.28:0/1197639851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17b8004350 con 0x7f17c8105360 2024-04-21T08:51:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.357+0000 7f17affff640 1 -- 172.21.15.28:0/1197639851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8003b80 con 0x7f17c8105360 2024-04-21T08:51:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.357+0000 7f17cd723640 1 -- 172.21.15.28:0/1197639851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c81a0650 con 0x7f17c8105360 2024-04-21T08:51:13.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.359+0000 7f17cd723640 1 -- 172.21.15.28:0/1197639851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17c80fdf10 con 0x7f17c8105360 2024-04-21T08:51:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.359+0000 7f17affff640 1 -- 172.21.15.28:0/1197639851 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f17b8046020 con 0x7f17c8105360 2024-04-21T08:51:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.360+0000 7f17affff640 1 --2- 172.21.15.28:0/1197639851 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f17b00a3db0 0x7f17b00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.360+0000 7f17c6ffd640 1 --2- 172.21.15.28:0/1197639851 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f17b00a3db0 0x7f17b00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.360+0000 7f17affff640 1 -- 172.21.15.28:0/1197639851 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f17b80f2d80 con 0x7f17c8105360 2024-04-21T08:51:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.361+0000 7f17c6ffd640 1 --2- 172.21.15.28:0/1197639851 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f17b00a3db0 0x7f17b00a6270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0095f0 tx=0x7f17bc005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:13.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.365+0000 7f17affff640 1 -- 172.21.15.28:0/1197639851 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f17b804d020 con 0x7f17c8105360 2024-04-21T08:51:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.555+0000 7f17cd723640 1 -- 172.21.15.28:0/1197639851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f17c802d770 con 0x7f17c8105360 2024-04-21T08:51:13.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.555+0000 7f17affff640 1 -- 172.21.15.28:0/1197639851 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f17b80bf020 con 0x7f17c8105360 2024-04-21T08:51:13.555 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:51:13.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.557+0000 7f17cd723640 1 -- 172.21.15.28:0/1197639851 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f17b00a3db0 msgr2=0x7f17b00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:13.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.557+0000 7f17cd723640 1 --2- 172.21.15.28:0/1197639851 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f17b00a3db0 0x7f17b00a6270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0095f0 tx=0x7f17bc005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:13.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.558+0000 7f17cd723640 1 -- 172.21.15.28:0/1197639851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c8105360 msgr2=0x7f17c8198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:13.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.558+0000 7f17cd723640 1 --2- 172.21.15.28:0/1197639851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c8105360 0x7f17c8198790 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f17b8000c00 tx=0x7f17b8005dc0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.558+0000 7f17cd723640 1 -- 172.21.15.28:0/1197639851 shutdown_connections 2024-04-21T08:51:13.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.558+0000 7f17cd723640 1 --2- 172.21.15.28:0/1197639851 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f17b00a3db0 0x7f17b00a6270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.558+0000 7f17cd723640 1 --2- 172.21.15.28:0/1197639851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f17c8107c90 0x7f17c819f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.558+0000 7f17cd723640 1 --2- 172.21.15.28:0/1197639851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c8105360 0x7f17c8198790 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.558+0000 7f17cd723640 1 --2- 172.21.15.28:0/1197639851 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f17c8102a30 0x7f17c8198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.558+0000 7f17cd723640 1 -- 172.21.15.28:0/1197639851 >> 172.21.15.28:0/1197639851 conn(0x7f17c80fc420 msgr2=0x7f17c8104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:13.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.558+0000 7f17cd723640 1 -- 172.21.15.28:0/1197639851 shutdown_connections 2024-04-21T08:51:13.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.558+0000 7f17cd723640 1 -- 172.21.15.28:0/1197639851 wait complete. 2024-04-21T08:51:13.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:51:13.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.675+0000 7ff7d2784640 1 -- 172.21.15.28:0/4284484446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7cc1029f0 msgr2=0x7ff7cc104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.675+0000 7ff7d2784640 1 --2- 172.21.15.28:0/4284484446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7cc1029f0 0x7ff7cc104de0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009900 tx=0x7ff7bc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.675+0000 7ff7d2784640 1 -- 172.21.15.28:0/4284484446 shutdown_connections 2024-04-21T08:51:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.675+0000 7ff7d2784640 1 --2- 172.21.15.28:0/4284484446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7cc107c50 0x7ff7cc10a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.675+0000 7ff7d2784640 1 --2- 172.21.15.28:0/4284484446 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff7cc105320 0x7ff7cc107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.675+0000 7ff7d2784640 1 --2- 172.21.15.28:0/4284484446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7cc1029f0 0x7ff7cc104de0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.675+0000 7ff7d2784640 1 -- 172.21.15.28:0/4284484446 >> 172.21.15.28:0/4284484446 conn(0x7ff7cc0fc3e0 msgr2=0x7ff7cc0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.675+0000 7ff7d2784640 1 -- 172.21.15.28:0/4284484446 shutdown_connections 2024-04-21T08:51:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.675+0000 7ff7d2784640 1 -- 172.21.15.28:0/4284484446 wait complete. 2024-04-21T08:51:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.676+0000 7ff7d2784640 1 Processor -- start 2024-04-21T08:51:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.676+0000 7ff7d2784640 1 -- start start 2024-04-21T08:51:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.676+0000 7ff7d2784640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7cc1029f0 0x7ff7cc102360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:13.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.676+0000 7ff7d2784640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7cc105320 0x7ff7cc1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:13.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.676+0000 7ff7d2784640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff7cc107c50 0x7ff7cc100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:13.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.676+0000 7ff7d2784640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc10db00 con 0x7ff7cc105320 2024-04-21T08:51:13.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.676+0000 7ff7d2784640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc10d980 con 0x7ff7cc1029f0 2024-04-21T08:51:13.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.676+0000 7ff7d2784640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc10dc80 con 0x7ff7cc107c50 2024-04-21T08:51:13.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.677+0000 7ff7cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7cc105320 0x7ff7cc1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:13.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.677+0000 7ff7cbfff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7cc1029f0 0x7ff7cc102360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:13.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.677+0000 7ff7cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7cc105320 0x7ff7cc1009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45216/0 (socket says 172.21.15.28:45216) 2024-04-21T08:51:13.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.677+0000 7ff7cbfff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7cc1029f0 0x7ff7cc102360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:45528/0 (socket says 172.21.15.28:45528) 2024-04-21T08:51:13.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.677+0000 7ff7cb7fe640 1 -- 172.21.15.28:0/364542418 learned_addr learned my addr 172.21.15.28:0/364542418 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:13.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.677+0000 7ff7d0d21640 1 --2- 172.21.15.28:0/364542418 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff7cc107c50 0x7ff7cc100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:13.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.677+0000 7ff7cbfff640 1 -- 172.21.15.28:0/364542418 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff7cc107c50 msgr2=0x7ff7cc100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:13.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.677+0000 7ff7cbfff640 1 --2- 172.21.15.28:0/364542418 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff7cc107c50 0x7ff7cc100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.677+0000 7ff7cbfff640 1 -- 172.21.15.28:0/364542418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7cc105320 msgr2=0x7ff7cc1009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:13.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.677+0000 7ff7cbfff640 1 --2- 172.21.15.28:0/364542418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7cc105320 0x7ff7cc1009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.677+0000 7ff7cbfff640 1 -- 172.21.15.28:0/364542418 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7cc101770 con 0x7ff7cc1029f0 2024-04-21T08:51:13.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.677+0000 7ff7cb7fe640 1 --2- 172.21.15.28:0/364542418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7cc105320 0x7ff7cc1009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:13.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.677+0000 7ff7cbfff640 1 --2- 172.21.15.28:0/364542418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7cc1029f0 0x7ff7cc102360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc02c170 tx=0x7ff7bc005e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:13.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.677+0000 7ff7c97fa640 1 -- 172.21.15.28:0/364542418 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc043020 con 0x7ff7cc1029f0 2024-04-21T08:51:13.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.678+0000 7ff7d2784640 1 -- 172.21.15.28:0/364542418 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7cc1a22f0 con 0x7ff7cc1029f0 2024-04-21T08:51:13.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.678+0000 7ff7c97fa640 1 -- 172.21.15.28:0/364542418 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7bc0043d0 con 0x7ff7cc1029f0 2024-04-21T08:51:13.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.678+0000 7ff7c97fa640 1 -- 172.21.15.28:0/364542418 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc003b50 con 0x7ff7cc1029f0 2024-04-21T08:51:13.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.678+0000 7ff7d2784640 1 -- 172.21.15.28:0/364542418 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7cc1a2860 con 0x7ff7cc1029f0 2024-04-21T08:51:13.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.679+0000 7ff7d2784640 1 -- 172.21.15.28:0/364542418 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7cc0fded0 con 0x7ff7cc1029f0 2024-04-21T08:51:13.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.681+0000 7ff7c97fa640 1 -- 172.21.15.28:0/364542418 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7ff7bc046020 con 0x7ff7cc1029f0 2024-04-21T08:51:13.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.681+0000 7ff7c97fa640 1 --2- 172.21.15.28:0/364542418 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff7b40a3db0 0x7ff7b40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:13.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.681+0000 7ff7c97fa640 1 -- 172.21.15.28:0/364542418 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ff7bc0f34a0 con 0x7ff7cc1029f0 2024-04-21T08:51:13.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.681+0000 7ff7cb7fe640 1 --2- 172.21.15.28:0/364542418 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff7b40a3db0 0x7ff7b40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:13.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.682+0000 7ff7cb7fe640 1 --2- 172.21.15.28:0/364542418 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff7b40a3db0 0x7ff7b40a6270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc1020d0 tx=0x7ff7c0005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:13.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.684+0000 7ff7c97fa640 1 -- 172.21.15.28:0/364542418 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff7bc04d020 con 0x7ff7cc1029f0 2024-04-21T08:51:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.878+0000 7ff7d2784640 1 -- 172.21.15.28:0/364542418 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff7cc062b50 con 0x7ff7cc1029f0 2024-04-21T08:51:13.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.880+0000 7ff7c97fa640 1 -- 172.21.15.28:0/364542418 <== mon.2 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25160 (secure 0 0 0) 0x7ff7bc0bf7e0 con 0x7ff7cc1029f0 2024-04-21T08:51:13.880 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:13.880 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:51:10.823421+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1076282159":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6800/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/926389767":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/892059008":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/2132574390":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:51:13.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.882+0000 7ff7d2784640 1 -- 172.21.15.28:0/364542418 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff7b40a3db0 msgr2=0x7ff7b40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:13.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.882+0000 7ff7d2784640 1 --2- 172.21.15.28:0/364542418 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff7b40a3db0 0x7ff7b40a6270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc1020d0 tx=0x7ff7c0005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:13.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.882+0000 7ff7d2784640 1 -- 172.21.15.28:0/364542418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7cc1029f0 msgr2=0x7ff7cc102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:13.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.882+0000 7ff7d2784640 1 --2- 172.21.15.28:0/364542418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7cc1029f0 0x7ff7cc102360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc02c170 tx=0x7ff7bc005e60 comp rx=0 tx=0).stop 2024-04-21T08:51:13.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.883+0000 7ff7d2784640 1 -- 172.21.15.28:0/364542418 shutdown_connections 2024-04-21T08:51:13.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.883+0000 7ff7d2784640 1 --2- 172.21.15.28:0/364542418 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff7b40a3db0 0x7ff7b40a6270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.883+0000 7ff7d2784640 1 --2- 172.21.15.28:0/364542418 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff7cc107c50 0x7ff7cc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.883+0000 7ff7d2784640 1 --2- 172.21.15.28:0/364542418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7cc105320 0x7ff7cc1009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.883+0000 7ff7d2784640 1 --2- 172.21.15.28:0/364542418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff7cc1029f0 0x7ff7cc102360 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:13.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.883+0000 7ff7d2784640 1 -- 172.21.15.28:0/364542418 >> 172.21.15.28:0/364542418 conn(0x7ff7cc0fc3e0 msgr2=0x7ff7cc103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:13.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.883+0000 7ff7d2784640 1 -- 172.21.15.28:0/364542418 shutdown_connections 2024-04-21T08:51:13.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:13.883+0000 7ff7d2784640 1 -- 172.21.15.28:0/364542418 wait complete. 2024-04-21T08:51:13.894 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:48:43.543799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:48:44.864564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:48:48.747842+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:51:13.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:51:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.003+0000 7f99e6175640 1 -- 172.21.15.28:0/2817362584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e0107c70 msgr2=0x7f99e010a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.003+0000 7f99e6175640 1 --2- 172.21.15.28:0/2817362584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e0107c70 0x7f99e010a0c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009ca0 tx=0x7f99d802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.003+0000 7f99e6175640 1 -- 172.21.15.28:0/2817362584 shutdown_connections 2024-04-21T08:51:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.003+0000 7f99e6175640 1 --2- 172.21.15.28:0/2817362584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e0107c70 0x7f99e010a0c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.003+0000 7f99e6175640 1 --2- 172.21.15.28:0/2817362584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99e0105340 0x7f99e0107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.003+0000 7f99e6175640 1 --2- 172.21.15.28:0/2817362584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f99e0102a10 0x7f99e0104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.003+0000 7f99e6175640 1 -- 172.21.15.28:0/2817362584 >> 172.21.15.28:0/2817362584 conn(0x7f99e00fc400 msgr2=0x7f99e00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.003+0000 7f99e6175640 1 -- 172.21.15.28:0/2817362584 shutdown_connections 2024-04-21T08:51:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.003+0000 7f99e6175640 1 -- 172.21.15.28:0/2817362584 wait complete. 2024-04-21T08:51:14.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.003+0000 7f99e6175640 1 Processor -- start 2024-04-21T08:51:14.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.003+0000 7f99e6175640 1 -- start start 2024-04-21T08:51:14.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99e6175640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e0102a10 0x7f99e01981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:14.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99e6175640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f99e0105340 0x7f99e01986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:14.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99e6175640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99e0107c70 0x7f99e019f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:14.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99e6175640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99e010dae0 con 0x7f99e0102a10 2024-04-21T08:51:14.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99e6175640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f99e010d960 con 0x7f99e0107c70 2024-04-21T08:51:14.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99e6175640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f99e010dc60 con 0x7f99e0105340 2024-04-21T08:51:14.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e0102a10 0x7f99e01981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:14.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e0102a10 0x7f99e01981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51862/0 (socket says 172.21.15.28:51862) 2024-04-21T08:51:14.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99df7fe640 1 -- 172.21.15.28:0/2426826 learned_addr learned my addr 172.21.15.28:0/2426826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:14.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99deffd640 1 --2- 172.21.15.28:0/2426826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f99e0105340 0x7f99e01986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:14.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99dffff640 1 --2- 172.21.15.28:0/2426826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99e0107c70 0x7f99e019f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99df7fe640 1 -- 172.21.15.28:0/2426826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f99e0105340 msgr2=0x7f99e01986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99df7fe640 1 --2- 172.21.15.28:0/2426826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f99e0105340 0x7f99e01986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99df7fe640 1 -- 172.21.15.28:0/2426826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99e0107c70 msgr2=0x7f99e019f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99df7fe640 1 --2- 172.21.15.28:0/2426826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99e0107c70 0x7f99e019f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99df7fe640 1 -- 172.21.15.28:0/2426826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99e019fdd0 con 0x7f99e0102a10 2024-04-21T08:51:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99deffd640 1 --2- 172.21.15.28:0/2426826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f99e0105340 0x7f99e01986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.004+0000 7f99df7fe640 1 --2- 172.21.15.28:0/2426826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e0102a10 0x7f99e01981b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f99d400e9f0 tx=0x7f99d400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:14.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.005+0000 7f99dcff9640 1 -- 172.21.15.28:0/2426826 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d400cd80 con 0x7f99e0102a10 2024-04-21T08:51:14.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.005+0000 7f99e6175640 1 -- 172.21.15.28:0/2426826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99e01a00c0 con 0x7f99e0102a10 2024-04-21T08:51:14.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.005+0000 7f99dcff9640 1 -- 172.21.15.28:0/2426826 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99d40054d0 con 0x7f99e0102a10 2024-04-21T08:51:14.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.005+0000 7f99dcff9640 1 -- 172.21.15.28:0/2426826 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d4010690 con 0x7f99e0102a10 2024-04-21T08:51:14.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.005+0000 7f99e6175640 1 -- 172.21.15.28:0/2426826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99e01a0610 con 0x7f99e0102a10 2024-04-21T08:51:14.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.006+0000 7f99e6175640 1 -- 172.21.15.28:0/2426826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99e00fdef0 con 0x7f99e0102a10 2024-04-21T08:51:14.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.007+0000 7f99dcff9640 1 -- 172.21.15.28:0/2426826 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f99d4004e90 con 0x7f99e0102a10 2024-04-21T08:51:14.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.008+0000 7f99dcff9640 1 --2- 172.21.15.28:0/2426826 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f99c80a3db0 0x7f99c80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:14.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.008+0000 7f99deffd640 1 --2- 172.21.15.28:0/2426826 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f99c80a3db0 0x7f99c80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:14.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.008+0000 7f99dcff9640 1 -- 172.21.15.28:0/2426826 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f99d40d1140 con 0x7f99e0102a10 2024-04-21T08:51:14.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.008+0000 7f99deffd640 1 --2- 172.21.15.28:0/2426826 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f99c80a3db0 0x7f99c80a6270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f99d00095f0 tx=0x7f99d0005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:14.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.012+0000 7f99dcff9640 1 -- 172.21.15.28:0/2426826 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f99d409d2d0 con 0x7f99e0102a10 2024-04-21T08:51:14.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:13 smithi152 ceph-mon[31401]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:51:14.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:13 smithi152 ceph-mon[31401]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-21T08:51:14.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:13 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/4034214934' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:14.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:13 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:13 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:13 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:13 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:13 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:13 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:13 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3252888089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:14.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:13 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1197639851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:14.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:13 smithi028 ceph-mon[28003]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:51:14.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:13 smithi028 ceph-mon[28003]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-21T08:51:14.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:13 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/4034214934' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:14.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:13 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:13 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:13 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:13 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:13 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:13 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:13 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3252888089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:14.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:13 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1197639851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:13 smithi171 ceph-mon[31358]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:51:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:13 smithi171 ceph-mon[31358]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-21T08:51:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:13 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/4034214934' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:13 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:13 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:13 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:13 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:13 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:13 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:13 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3252888089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:14.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:13 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1197639851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:14.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.208+0000 7f99e6175640 1 -- 172.21.15.28:0/2426826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f99e002d980 con 0x7f99e0102a10 2024-04-21T08:51:14.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.208+0000 7f99dcff9640 1 -- 172.21.15.28:0/2426826 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f99d409cbd0 con 0x7f99e0102a10 2024-04-21T08:51:14.208 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T08:51:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.210+0000 7f99e6175640 1 -- 172.21.15.28:0/2426826 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f99c80a3db0 msgr2=0x7f99c80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.210+0000 7f99e6175640 1 --2- 172.21.15.28:0/2426826 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f99c80a3db0 0x7f99c80a6270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f99d00095f0 tx=0x7f99d0005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.210+0000 7f99e6175640 1 -- 172.21.15.28:0/2426826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e0102a10 msgr2=0x7f99e01981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.210+0000 7f99e6175640 1 --2- 172.21.15.28:0/2426826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e0102a10 0x7f99e01981b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f99d400e9f0 tx=0x7f99d400edf0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.210+0000 7f99e6175640 1 -- 172.21.15.28:0/2426826 shutdown_connections 2024-04-21T08:51:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.210+0000 7f99e6175640 1 --2- 172.21.15.28:0/2426826 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f99c80a3db0 0x7f99c80a6270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.210+0000 7f99e6175640 1 --2- 172.21.15.28:0/2426826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99e0107c70 0x7f99e019f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.210+0000 7f99e6175640 1 --2- 172.21.15.28:0/2426826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f99e0105340 0x7f99e01986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.210+0000 7f99e6175640 1 --2- 172.21.15.28:0/2426826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e0102a10 0x7f99e01981b0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.210+0000 7f99e6175640 1 -- 172.21.15.28:0/2426826 >> 172.21.15.28:0/2426826 conn(0x7f99e00fc400 msgr2=0x7f99e01032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.210+0000 7f99e6175640 1 -- 172.21.15.28:0/2426826 shutdown_connections 2024-04-21T08:51:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.210+0000 7f99e6175640 1 -- 172.21.15.28:0/2426826 wait complete. 2024-04-21T08:51:14.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:51:14.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.332+0000 7f1ad66eb640 1 -- 172.21.15.28:0/1382355195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ad0103190 msgr2=0x7f1ad0103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:14.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.332+0000 7f1ad66eb640 1 --2- 172.21.15.28:0/1382355195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ad0103190 0x7f1ad0103610 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009900 tx=0x7f1ac402bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:14.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.332+0000 7f1ad66eb640 1 -- 172.21.15.28:0/1382355195 shutdown_connections 2024-04-21T08:51:14.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.332+0000 7f1ad66eb640 1 --2- 172.21.15.28:0/1382355195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ad0103b50 0x7f1ad010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.332+0000 7f1ad66eb640 1 --2- 172.21.15.28:0/1382355195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ad0103190 0x7f1ad0103610 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.332+0000 7f1ad66eb640 1 --2- 172.21.15.28:0/1382355195 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ad0101fa0 0x7f1ad01023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.332+0000 7f1ad66eb640 1 -- 172.21.15.28:0/1382355195 >> 172.21.15.28:0/1382355195 conn(0x7f1ad00fd530 msgr2=0x7f1ad00ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:14.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.332+0000 7f1ad66eb640 1 -- 172.21.15.28:0/1382355195 shutdown_connections 2024-04-21T08:51:14.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.332+0000 7f1ad66eb640 1 -- 172.21.15.28:0/1382355195 wait complete. 2024-04-21T08:51:14.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1ad66eb640 1 Processor -- start 2024-04-21T08:51:14.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1ad66eb640 1 -- start start 2024-04-21T08:51:14.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1ad66eb640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ad0101fa0 0x7f1ad0198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:14.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1ad66eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ad0103190 0x7f1ad01987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:14.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1ad66eb640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ad0103b50 0x7f1ad019f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:14.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1ad66eb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad010db40 con 0x7f1ad0103190 2024-04-21T08:51:14.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1ad66eb640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad010d9c0 con 0x7f1ad0101fa0 2024-04-21T08:51:14.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1ad66eb640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad010dcc0 con 0x7f1ad0103b50 2024-04-21T08:51:14.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1acf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ad0103190 0x7f1ad01987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:14.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1acf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ad0103190 0x7f1ad01987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51888/0 (socket says 172.21.15.28:51888) 2024-04-21T08:51:14.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1acf7fe640 1 -- 172.21.15.28:0/2564727788 learned_addr learned my addr 172.21.15.28:0/2564727788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:14.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1ad4c88640 1 --2- 172.21.15.28:0/2564727788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ad0103b50 0x7f1ad019f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:14.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1acf7fe640 1 -- 172.21.15.28:0/2564727788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ad0103b50 msgr2=0x7f1ad019f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:14.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1acffff640 1 --2- 172.21.15.28:0/2564727788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ad0101fa0 0x7f1ad0198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:14.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1acf7fe640 1 --2- 172.21.15.28:0/2564727788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ad0103b50 0x7f1ad019f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1acf7fe640 1 -- 172.21.15.28:0/2564727788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ad0101fa0 msgr2=0x7f1ad0198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:14.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.333+0000 7f1acf7fe640 1 --2- 172.21.15.28:0/2564727788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ad0101fa0 0x7f1ad0198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.334+0000 7f1acf7fe640 1 -- 172.21.15.28:0/2564727788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad019fe90 con 0x7f1ad0103190 2024-04-21T08:51:14.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.334+0000 7f1acf7fe640 1 --2- 172.21.15.28:0/2564727788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ad0103190 0x7f1ad01987b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4006010 tx=0x7f1ac40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:14.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.334+0000 7f1acd7fa640 1 -- 172.21.15.28:0/2564727788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4043020 con 0x7f1ad0103190 2024-04-21T08:51:14.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.334+0000 7f1acd7fa640 1 -- 172.21.15.28:0/2564727788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ac4004aa0 con 0x7f1ad0103190 2024-04-21T08:51:14.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.334+0000 7f1ad66eb640 1 -- 172.21.15.28:0/2564727788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad01a0120 con 0x7f1ad0103190 2024-04-21T08:51:14.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.334+0000 7f1acd7fa640 1 -- 172.21.15.28:0/2564727788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4003c70 con 0x7f1ad0103190 2024-04-21T08:51:14.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.334+0000 7f1ad66eb640 1 -- 172.21.15.28:0/2564727788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad01a06f0 con 0x7f1ad0103190 2024-04-21T08:51:14.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.335+0000 7f1ad66eb640 1 -- 172.21.15.28:0/2564727788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad0109390 con 0x7f1ad0103190 2024-04-21T08:51:14.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.336+0000 7f1acd7fa640 1 -- 172.21.15.28:0/2564727788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f1ac4046020 con 0x7f1ad0103190 2024-04-21T08:51:14.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.337+0000 7f1acd7fa640 1 --2- 172.21.15.28:0/2564727788 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ab80a3db0 0x7f1ab80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:14.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.337+0000 7f1acd7fa640 1 -- 172.21.15.28:0/2564727788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1ac40f30c0 con 0x7f1ad0103190 2024-04-21T08:51:14.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.337+0000 7f1acffff640 1 --2- 172.21.15.28:0/2564727788 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ab80a3db0 0x7f1ab80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:14.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.338+0000 7f1acffff640 1 --2- 172.21.15.28:0/2564727788 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ab80a3db0 0x7f1ab80a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0005fd0 tx=0x7f1ac0005850 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:14.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.342+0000 7f1acd7fa640 1 -- 172.21.15.28:0/2564727788 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1ac404d020 con 0x7f1ad0103190 2024-04-21T08:51:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.583+0000 7f1ad66eb640 1 -- 172.21.15.28:0/2564727788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1ad002d770 con 0x7f1ad0103190 2024-04-21T08:51:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.583+0000 7f1acd7fa640 1 -- 172.21.15.28:0/2564727788 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f1ac40dde20 con 0x7f1ad0103190 2024-04-21T08:51:14.583 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:51:14.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.585+0000 7f1ad66eb640 1 -- 172.21.15.28:0/2564727788 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ab80a3db0 msgr2=0x7f1ab80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:14.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.585+0000 7f1ad66eb640 1 --2- 172.21.15.28:0/2564727788 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ab80a3db0 0x7f1ab80a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0005fd0 tx=0x7f1ac0005850 comp rx=0 tx=0).stop 2024-04-21T08:51:14.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.585+0000 7f1ad66eb640 1 -- 172.21.15.28:0/2564727788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ad0103190 msgr2=0x7f1ad01987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:14.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.585+0000 7f1ad66eb640 1 --2- 172.21.15.28:0/2564727788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ad0103190 0x7f1ad01987b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4006010 tx=0x7f1ac40042f0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.585+0000 7f1ad66eb640 1 -- 172.21.15.28:0/2564727788 shutdown_connections 2024-04-21T08:51:14.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.585+0000 7f1ad66eb640 1 --2- 172.21.15.28:0/2564727788 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ab80a3db0 0x7f1ab80a6270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.585+0000 7f1ad66eb640 1 --2- 172.21.15.28:0/2564727788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ad0103b50 0x7f1ad019f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.585+0000 7f1ad66eb640 1 --2- 172.21.15.28:0/2564727788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ad0103190 0x7f1ad01987b0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.585+0000 7f1ad66eb640 1 --2- 172.21.15.28:0/2564727788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ad0101fa0 0x7f1ad0198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.585+0000 7f1ad66eb640 1 -- 172.21.15.28:0/2564727788 >> 172.21.15.28:0/2564727788 conn(0x7f1ad00fd530 msgr2=0x7f1ad0106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:14.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.585+0000 7f1ad66eb640 1 -- 172.21.15.28:0/2564727788 shutdown_connections 2024-04-21T08:51:14.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.585+0000 7f1ad66eb640 1 -- 172.21.15.28:0/2564727788 wait complete. 2024-04-21T08:51:14.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:51:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.708+0000 7fd639ff4640 1 -- 172.21.15.28:0/4063809435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd634101fa0 msgr2=0x7fd6341023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.708+0000 7fd639ff4640 1 --2- 172.21.15.28:0/4063809435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd634101fa0 0x7fd6341023c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd61c009900 tx=0x7fd61c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:14.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.708+0000 7fd639ff4640 1 -- 172.21.15.28:0/4063809435 shutdown_connections 2024-04-21T08:51:14.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.708+0000 7fd639ff4640 1 --2- 172.21.15.28:0/4063809435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd634103b50 0x7fd63410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.708+0000 7fd639ff4640 1 --2- 172.21.15.28:0/4063809435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd634103190 0x7fd634103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.708+0000 7fd639ff4640 1 --2- 172.21.15.28:0/4063809435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd634101fa0 0x7fd6341023c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.708+0000 7fd639ff4640 1 -- 172.21.15.28:0/4063809435 >> 172.21.15.28:0/4063809435 conn(0x7fd6340fd530 msgr2=0x7fd6340ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:14.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.708+0000 7fd639ff4640 1 -- 172.21.15.28:0/4063809435 shutdown_connections 2024-04-21T08:51:14.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.708+0000 7fd639ff4640 1 -- 172.21.15.28:0/4063809435 wait complete. 2024-04-21T08:51:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.709+0000 7fd639ff4640 1 Processor -- start 2024-04-21T08:51:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.709+0000 7fd639ff4640 1 -- start start 2024-04-21T08:51:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.709+0000 7fd639ff4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd634101fa0 0x7fd6341981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.709+0000 7fd639ff4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd634103190 0x7fd634198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.709+0000 7fd639ff4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd634103b50 0x7fd63419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.709+0000 7fd639ff4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd63410dae0 con 0x7fd634101fa0 2024-04-21T08:51:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.709+0000 7fd639ff4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd63410d960 con 0x7fd634103190 2024-04-21T08:51:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.709+0000 7fd639ff4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd63410dc60 con 0x7fd634103b50 2024-04-21T08:51:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.709+0000 7fd6337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd634101fa0 0x7fd6341981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.709+0000 7fd6337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd634101fa0 0x7fd6341981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51896/0 (socket says 172.21.15.28:51896) 2024-04-21T08:51:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.709+0000 7fd6337fe640 1 -- 172.21.15.28:0/3848978672 learned_addr learned my addr 172.21.15.28:0/3848978672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:14.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.709+0000 7fd633fff640 1 --2- 172.21.15.28:0/3848978672 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd634103b50 0x7fd63419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:14.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.709+0000 7fd632ffd640 1 --2- 172.21.15.28:0/3848978672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd634103190 0x7fd634198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:14.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.710+0000 7fd6337fe640 1 -- 172.21.15.28:0/3848978672 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd634103b50 msgr2=0x7fd63419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:14.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.710+0000 7fd6337fe640 1 --2- 172.21.15.28:0/3848978672 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd634103b50 0x7fd63419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.710+0000 7fd6337fe640 1 -- 172.21.15.28:0/3848978672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd634103190 msgr2=0x7fd634198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:14.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.710+0000 7fd6337fe640 1 --2- 172.21.15.28:0/3848978672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd634103190 0x7fd634198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.710+0000 7fd6337fe640 1 -- 172.21.15.28:0/3848978672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63419fdf0 con 0x7fd634101fa0 2024-04-21T08:51:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.710+0000 7fd6337fe640 1 --2- 172.21.15.28:0/3848978672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd634101fa0 0x7fd6341981d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd61c000c00 tx=0x7fd61c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.710+0000 7fd630ff9640 1 -- 172.21.15.28:0/3848978672 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61c043020 con 0x7fd634101fa0 2024-04-21T08:51:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.710+0000 7fd630ff9640 1 -- 172.21.15.28:0/3848978672 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd61c004350 con 0x7fd634101fa0 2024-04-21T08:51:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.710+0000 7fd639ff4640 1 -- 172.21.15.28:0/3848978672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6341a0080 con 0x7fd634101fa0 2024-04-21T08:51:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.710+0000 7fd630ff9640 1 -- 172.21.15.28:0/3848978672 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61c003b80 con 0x7fd634101fa0 2024-04-21T08:51:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.710+0000 7fd639ff4640 1 -- 172.21.15.28:0/3848978672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6341a05d0 con 0x7fd634101fa0 2024-04-21T08:51:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.711+0000 7fd639ff4640 1 -- 172.21.15.28:0/3848978672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6341093b0 con 0x7fd634101fa0 2024-04-21T08:51:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.713+0000 7fd630ff9640 1 -- 172.21.15.28:0/3848978672 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7fd61c046020 con 0x7fd634101fa0 2024-04-21T08:51:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.713+0000 7fd630ff9640 1 --2- 172.21.15.28:0/3848978672 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd6200a3db0 0x7fd6200a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.714+0000 7fd630ff9640 1 -- 172.21.15.28:0/3848978672 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fd61c0f31a0 con 0x7fd634101fa0 2024-04-21T08:51:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.714+0000 7fd632ffd640 1 --2- 172.21.15.28:0/3848978672 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd6200a3db0 0x7fd6200a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.714+0000 7fd632ffd640 1 --2- 172.21.15.28:0/3848978672 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd6200a3db0 0x7fd6200a6270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd6341996e0 tx=0x7fd624005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:14.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.717+0000 7fd630ff9640 1 -- 172.21.15.28:0/3848978672 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd61c04d020 con 0x7fd634101fa0 2024-04-21T08:51:14.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/364542418' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:14.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:14.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:14.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:14.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:14.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:14.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:14.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-04-21T08:51:14.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:14.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: Unable to set osd_memory_target on smithi171 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-21T08:51:14.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:14.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi152 to 176.2M 2024-04-21T08:51:14.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: Unable to set osd_memory_target on smithi152 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-21T08:51:14.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2426826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:14.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:14.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:14.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:14.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:14.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:14.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-04-21T08:51:14.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: Unable to set osd_memory_target on smithi028 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-04-21T08:51:14.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:51:14.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:51:14.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-21T08:51:14.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: Updating smithi152:/etc/ceph/ceph.conf 2024-04-21T08:51:14.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: Updating smithi171:/etc/ceph/ceph.conf 2024-04-21T08:51:14.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:14 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2564727788' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:14.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.913+0000 7fd639ff4640 1 -- 172.21.15.28:0/3848978672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd634062b50 con 0x7fd634101fa0 2024-04-21T08:51:14.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.914+0000 7fd630ff9640 1 -- 172.21.15.28:0/3848978672 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fd61c0bf440 con 0x7fd634101fa0 2024-04-21T08:51:14.913 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-21T08:51:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.916+0000 7fd639ff4640 1 -- 172.21.15.28:0/3848978672 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd6200a3db0 msgr2=0x7fd6200a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.916+0000 7fd639ff4640 1 --2- 172.21.15.28:0/3848978672 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd6200a3db0 0x7fd6200a6270 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd6341996e0 tx=0x7fd624005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.916+0000 7fd639ff4640 1 -- 172.21.15.28:0/3848978672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd634101fa0 msgr2=0x7fd6341981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.916+0000 7fd639ff4640 1 --2- 172.21.15.28:0/3848978672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd634101fa0 0x7fd6341981d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd61c000c00 tx=0x7fd61c005dc0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.916+0000 7fd639ff4640 1 -- 172.21.15.28:0/3848978672 shutdown_connections 2024-04-21T08:51:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.916+0000 7fd639ff4640 1 --2- 172.21.15.28:0/3848978672 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd6200a3db0 0x7fd6200a6270 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.916+0000 7fd639ff4640 1 --2- 172.21.15.28:0/3848978672 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd634103b50 0x7fd63419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.916+0000 7fd639ff4640 1 --2- 172.21.15.28:0/3848978672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd634103190 0x7fd634198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.916+0000 7fd639ff4640 1 --2- 172.21.15.28:0/3848978672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd634101fa0 0x7fd6341981d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.916+0000 7fd639ff4640 1 -- 172.21.15.28:0/3848978672 >> 172.21.15.28:0/3848978672 conn(0x7fd6340fd530 msgr2=0x7fd634106580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.916+0000 7fd639ff4640 1 -- 172.21.15.28:0/3848978672 shutdown_connections 2024-04-21T08:51:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:14.916+0000 7fd639ff4640 1 -- 172.21.15.28:0/3848978672 wait complete. 2024-04-21T08:51:14.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:51:15.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.038+0000 7f1de5862640 1 -- 172.21.15.28:0/1061637683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1de01053f0 msgr2=0x7f1de01077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.038+0000 7f1de5862640 1 --2- 172.21.15.28:0/1061637683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1de01053f0 0x7f1de01077e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009900 tx=0x7f1dd402bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.038+0000 7f1de5862640 1 -- 172.21.15.28:0/1061637683 shutdown_connections 2024-04-21T08:51:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.038+0000 7f1de5862640 1 --2- 172.21.15.28:0/1061637683 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1de0107d20 0x7f1de010a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.038+0000 7f1de5862640 1 --2- 172.21.15.28:0/1061637683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1de01053f0 0x7f1de01077e0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.038+0000 7f1de5862640 1 --2- 172.21.15.28:0/1061637683 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1de0102ac0 0x7f1de0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.038+0000 7f1de5862640 1 -- 172.21.15.28:0/1061637683 >> 172.21.15.28:0/1061637683 conn(0x7f1de00fc410 msgr2=0x7f1de00fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.038+0000 7f1de5862640 1 -- 172.21.15.28:0/1061637683 shutdown_connections 2024-04-21T08:51:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.038+0000 7f1de5862640 1 -- 172.21.15.28:0/1061637683 wait complete. 2024-04-21T08:51:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1de5862640 1 Processor -- start 2024-04-21T08:51:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1de5862640 1 -- start start 2024-04-21T08:51:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1de5862640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1de0102ac0 0x7f1de0198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1de5862640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1de01053f0 0x7f1de0198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1de5862640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1de0107d20 0x7f1de019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1de5862640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1de010db00 con 0x7f1de01053f0 2024-04-21T08:51:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1de5862640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1de010d980 con 0x7f1de0102ac0 2024-04-21T08:51:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1de5862640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1de010dc80 con 0x7f1de0107d20 2024-04-21T08:51:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1dde7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1de01053f0 0x7f1de0198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1dde7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1de01053f0 0x7f1de0198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51910/0 (socket says 172.21.15.28:51910) 2024-04-21T08:51:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1dde7fc640 1 -- 172.21.15.28:0/1588229387 learned_addr learned my addr 172.21.15.28:0/1588229387 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1ddf7fe640 1 --2- 172.21.15.28:0/1588229387 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1de0107d20 0x7f1de019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1ddeffd640 1 --2- 172.21.15.28:0/1588229387 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1de0102ac0 0x7f1de0198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1dde7fc640 1 -- 172.21.15.28:0/1588229387 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1de0107d20 msgr2=0x7f1de019f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1dde7fc640 1 --2- 172.21.15.28:0/1588229387 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1de0107d20 0x7f1de019f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1dde7fc640 1 -- 172.21.15.28:0/1588229387 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1de0102ac0 msgr2=0x7f1de0198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1dde7fc640 1 --2- 172.21.15.28:0/1588229387 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1de0102ac0 0x7f1de0198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1dde7fc640 1 -- 172.21.15.28:0/1588229387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de019fe50 con 0x7f1de01053f0 2024-04-21T08:51:15.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.039+0000 7f1ddeffd640 1 --2- 172.21.15.28:0/1588229387 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1de0102ac0 0x7f1de0198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:15.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.040+0000 7f1dde7fc640 1 --2- 172.21.15.28:0/1588229387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1de01053f0 0x7f1de0198770 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4000c00 tx=0x7f1dd4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:15.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.040+0000 7f1de4860640 1 -- 172.21.15.28:0/1588229387 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4043020 con 0x7f1de01053f0 2024-04-21T08:51:15.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.040+0000 7f1de5862640 1 -- 172.21.15.28:0/1588229387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1de01a00e0 con 0x7f1de01053f0 2024-04-21T08:51:15.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.040+0000 7f1de4860640 1 -- 172.21.15.28:0/1588229387 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dd4004aa0 con 0x7f1de01053f0 2024-04-21T08:51:15.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.040+0000 7f1de4860640 1 -- 172.21.15.28:0/1588229387 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4003d00 con 0x7f1de01053f0 2024-04-21T08:51:15.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.040+0000 7f1de5862640 1 -- 172.21.15.28:0/1588229387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1de01008b0 con 0x7f1de01053f0 2024-04-21T08:51:15.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.042+0000 7f1de5862640 1 -- 172.21.15.28:0/1588229387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1de00fdf00 con 0x7f1de01053f0 2024-04-21T08:51:15.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.043+0000 7f1de4860640 1 -- 172.21.15.28:0/1588229387 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138817+0+0 (secure 0 0 0) 0x7f1dd4046020 con 0x7f1de01053f0 2024-04-21T08:51:15.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.043+0000 7f1de4860640 1 --2- 172.21.15.28:0/1588229387 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1dc80a81c0 0x7f1dc80aa680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:15.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.043+0000 7f1ddeffd640 1 --2- 172.21.15.28:0/1588229387 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1dc80a81c0 0x7f1dc80aa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:15.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.043+0000 7f1de4860640 1 -- 172.21.15.28:0/1588229387 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1dd40f3200 con 0x7f1de01053f0 2024-04-21T08:51:15.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.044+0000 7f1ddeffd640 1 --2- 172.21.15.28:0/1588229387 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1dc80a81c0 0x7f1dc80aa680 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00058c0 tx=0x7f1dd0005850 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:15.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.048+0000 7f1de4860640 1 -- 172.21.15.28:0/1588229387 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1dd404d020 con 0x7f1de01053f0 2024-04-21T08:51:15.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/364542418' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:15.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-04-21T08:51:15.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: Unable to set osd_memory_target on smithi171 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-21T08:51:15.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi152 to 176.2M 2024-04-21T08:51:15.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: Unable to set osd_memory_target on smithi152 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-21T08:51:15.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2426826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:15.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-04-21T08:51:15.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: Unable to set osd_memory_target on smithi028 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-04-21T08:51:15.107 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:51:15.107 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:51:15.107 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-21T08:51:15.107 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: Updating smithi152:/etc/ceph/ceph.conf 2024-04-21T08:51:15.107 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: Updating smithi171:/etc/ceph/ceph.conf 2024-04-21T08:51:15.107 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:14 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2564727788' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:15.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/364542418' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:15.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-04-21T08:51:15.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: Unable to set osd_memory_target on smithi171 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-21T08:51:15.122 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.122 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi152 to 176.2M 2024-04-21T08:51:15.122 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: Unable to set osd_memory_target on smithi152 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-21T08:51:15.122 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2426826' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:15.122 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.122 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.122 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.122 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.122 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.122 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:51:15.122 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-04-21T08:51:15.122 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: Unable to set osd_memory_target on smithi028 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-04-21T08:51:15.122 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:51:15.123 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:51:15.123 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-21T08:51:15.123 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: Updating smithi152:/etc/ceph/ceph.conf 2024-04-21T08:51:15.123 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: Updating smithi171:/etc/ceph/ceph.conf 2024-04-21T08:51:15.123 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:14 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2564727788' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:15.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.251+0000 7f1de5862640 1 -- 172.21.15.28:0/1588229387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1de002d770 con 0x7f1de01053f0 2024-04-21T08:51:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.251+0000 7f1de4860640 1 -- 172.21.15.28:0/1588229387 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f1dd40bf4a0 con 0x7f1de01053f0 2024-04-21T08:51:15.251 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:51:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.253+0000 7f1de5862640 1 -- 172.21.15.28:0/1588229387 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1dc80a81c0 msgr2=0x7f1dc80aa680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.253+0000 7f1de5862640 1 --2- 172.21.15.28:0/1588229387 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1dc80a81c0 0x7f1dc80aa680 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00058c0 tx=0x7f1dd0005850 comp rx=0 tx=0).stop 2024-04-21T08:51:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.253+0000 7f1de5862640 1 -- 172.21.15.28:0/1588229387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1de01053f0 msgr2=0x7f1de0198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.253+0000 7f1de5862640 1 --2- 172.21.15.28:0/1588229387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1de01053f0 0x7f1de0198770 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4000c00 tx=0x7f1dd4004270 comp rx=0 tx=0).stop 2024-04-21T08:51:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.253+0000 7f1de5862640 1 -- 172.21.15.28:0/1588229387 shutdown_connections 2024-04-21T08:51:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.253+0000 7f1de5862640 1 --2- 172.21.15.28:0/1588229387 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1dc80a81c0 0x7f1dc80aa680 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.253+0000 7f1de5862640 1 --2- 172.21.15.28:0/1588229387 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1de0107d20 0x7f1de019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.253+0000 7f1de5862640 1 --2- 172.21.15.28:0/1588229387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1de01053f0 0x7f1de0198770 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.253+0000 7f1de5862640 1 --2- 172.21.15.28:0/1588229387 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1de0102ac0 0x7f1de0198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.253+0000 7f1de5862640 1 -- 172.21.15.28:0/1588229387 >> 172.21.15.28:0/1588229387 conn(0x7f1de00fc410 msgr2=0x7f1de0104960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.254+0000 7f1de5862640 1 -- 172.21.15.28:0/1588229387 shutdown_connections 2024-04-21T08:51:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.254+0000 7f1de5862640 1 -- 172.21.15.28:0/1588229387 wait complete. 2024-04-21T08:51:15.265 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-21T08:51:15.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:51:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.375+0000 7f78c4932640 1 -- 172.21.15.28:0/90570340 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78c0101670 msgr2=0x7f78c0101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.375+0000 7f78c4932640 1 --2- 172.21.15.28:0/90570340 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78c0101670 0x7f78c0101ad0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009930 tx=0x7f78a802bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.375+0000 7f78c4932640 1 -- 172.21.15.28:0/90570340 shutdown_connections 2024-04-21T08:51:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.375+0000 7f78c4932640 1 --2- 172.21.15.28:0/90570340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78c0102010 0x7f78c010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.375+0000 7f78c4932640 1 --2- 172.21.15.28:0/90570340 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78c0101670 0x7f78c0101ad0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.375+0000 7f78c4932640 1 --2- 172.21.15.28:0/90570340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c0106f10 0x7f78c01072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.375+0000 7f78c4932640 1 -- 172.21.15.28:0/90570340 >> 172.21.15.28:0/90570340 conn(0x7f78c00774f0 msgr2=0x7f78c00fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.375+0000 7f78c4932640 1 -- 172.21.15.28:0/90570340 shutdown_connections 2024-04-21T08:51:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.376+0000 7f78c4932640 1 -- 172.21.15.28:0/90570340 wait complete. 2024-04-21T08:51:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.376+0000 7f78c4932640 1 Processor -- start 2024-04-21T08:51:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.376+0000 7f78c4932640 1 -- start start 2024-04-21T08:51:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.376+0000 7f78c4932640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78c0101670 0x7f78c0198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.376+0000 7f78c4932640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78c0102010 0x7f78c0198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.376+0000 7f78c4932640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c0106f10 0x7f78c019cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.376+0000 7f78c4932640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78c0111090 con 0x7f78c0106f10 2024-04-21T08:51:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.376+0000 7f78c4932640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f78c0110f10 con 0x7f78c0102010 2024-04-21T08:51:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.376+0000 7f78c4932640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f78c0111210 con 0x7f78c0101670 2024-04-21T08:51:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78be59c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78c0101670 0x7f78c0198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78bed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c0106f10 0x7f78c019cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78be59c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78c0101670 0x7f78c0198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:36524/0 (socket says 172.21.15.28:36524) 2024-04-21T08:51:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78bed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c0106f10 0x7f78c019cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51928/0 (socket says 172.21.15.28:51928) 2024-04-21T08:51:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78be59c640 1 -- 172.21.15.28:0/341422188 learned_addr learned my addr 172.21.15.28:0/341422188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78b7fff640 1 --2- 172.21.15.28:0/341422188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78c0102010 0x7f78c0198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78be59c640 1 -- 172.21.15.28:0/341422188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78c0102010 msgr2=0x7f78c0198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78be59c640 1 --2- 172.21.15.28:0/341422188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78c0102010 0x7f78c0198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78be59c640 1 -- 172.21.15.28:0/341422188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c0106f10 msgr2=0x7f78c019cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78be59c640 1 --2- 172.21.15.28:0/341422188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c0106f10 0x7f78c019cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78be59c640 1 -- 172.21.15.28:0/341422188 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c019d700 con 0x7f78c0101670 2024-04-21T08:51:15.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78bed9d640 1 --2- 172.21.15.28:0/341422188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c0106f10 0x7f78c019cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:15.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78be59c640 1 --2- 172.21.15.28:0/341422188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78c0101670 0x7f78c0198710 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78b000e9f0 tx=0x7f78b000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:15.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78b77fe640 1 -- 172.21.15.28:0/341422188 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b000cd80 con 0x7f78c0101670 2024-04-21T08:51:15.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78b77fe640 1 -- 172.21.15.28:0/341422188 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78b00108e0 con 0x7f78c0101670 2024-04-21T08:51:15.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78c4932640 1 -- 172.21.15.28:0/341422188 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c019d9f0 con 0x7f78c0101670 2024-04-21T08:51:15.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78b77fe640 1 -- 172.21.15.28:0/341422188 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b0018980 con 0x7f78c0101670 2024-04-21T08:51:15.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.377+0000 7f78c4932640 1 -- 172.21.15.28:0/341422188 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c01a5280 con 0x7f78c0101670 2024-04-21T08:51:15.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.378+0000 7f78c4932640 1 -- 172.21.15.28:0/341422188 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c0068550 con 0x7f78c0101670 2024-04-21T08:51:15.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.379+0000 7f78b77fe640 1 -- 172.21.15.28:0/341422188 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f78b0010430 con 0x7f78c0101670 2024-04-21T08:51:15.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.380+0000 7f78b77fe640 1 --2- 172.21.15.28:0/341422188 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f78ac0a3ec0 0x7f78ac0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:15.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.380+0000 7f78b7fff640 1 --2- 172.21.15.28:0/341422188 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f78ac0a3ec0 0x7f78ac0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:15.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.380+0000 7f78b77fe640 1 -- 172.21.15.28:0/341422188 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f78b00d1340 con 0x7f78c0101670 2024-04-21T08:51:15.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.381+0000 7f78b7fff640 1 --2- 172.21.15.28:0/341422188 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f78ac0a3ec0 0x7f78ac0a6380 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009900 tx=0x7f78a8005a20 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.384+0000 7f78b77fe640 1 -- 172.21.15.28:0/341422188 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f78b009ce00 con 0x7f78c0101670 2024-04-21T08:51:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.622+0000 7f78c4932640 1 -- 172.21.15.28:0/341422188 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78c01a54b0 con 0x7f78c0101670 2024-04-21T08:51:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.623+0000 7f78b77fe640 1 -- 172.21.15.28:0/341422188 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7f78b00a1100 con 0x7f78c0101670 2024-04-21T08:51:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:15.623 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:51:09.181803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:51:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-04-21T08:51:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.625+0000 7f78c4932640 1 -- 172.21.15.28:0/341422188 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f78ac0a3ec0 msgr2=0x7f78ac0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.625+0000 7f78c4932640 1 --2- 172.21.15.28:0/341422188 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f78ac0a3ec0 0x7f78ac0a6380 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009900 tx=0x7f78a8005a20 comp rx=0 tx=0).stop 2024-04-21T08:51:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.625+0000 7f78c4932640 1 -- 172.21.15.28:0/341422188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78c0101670 msgr2=0x7f78c0198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.625+0000 7f78c4932640 1 --2- 172.21.15.28:0/341422188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78c0101670 0x7f78c0198710 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78b000e9f0 tx=0x7f78b000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.625+0000 7f78c4932640 1 -- 172.21.15.28:0/341422188 shutdown_connections 2024-04-21T08:51:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.625+0000 7f78c4932640 1 --2- 172.21.15.28:0/341422188 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f78ac0a3ec0 0x7f78ac0a6380 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.625+0000 7f78c4932640 1 --2- 172.21.15.28:0/341422188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c0106f10 0x7f78c019cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.625+0000 7f78c4932640 1 --2- 172.21.15.28:0/341422188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78c0102010 0x7f78c0198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.625+0000 7f78c4932640 1 --2- 172.21.15.28:0/341422188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78c0101670 0x7f78c0198710 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.625+0000 7f78c4932640 1 -- 172.21.15.28:0/341422188 >> 172.21.15.28:0/341422188 conn(0x7f78c00774f0 msgr2=0x7f78c0100740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.625+0000 7f78c4932640 1 -- 172.21.15.28:0/341422188 shutdown_connections 2024-04-21T08:51:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.625+0000 7f78c4932640 1 -- 172.21.15.28:0/341422188 wait complete. 2024-04-21T08:51:15.636 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-21T08:51:15.636 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-21T08:51:15.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:51:15.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.739+0000 7f0d78db1640 1 -- 172.21.15.28:0/2804360901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d74102b20 msgr2=0x7f0d74106b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:15.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.739+0000 7f0d78db1640 1 --2- 172.21.15.28:0/2804360901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d74102b20 0x7f0d74106b60 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c009900 tx=0x7f0d5c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:15.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.739+0000 7f0d78db1640 1 -- 172.21.15.28:0/2804360901 shutdown_connections 2024-04-21T08:51:15.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.739+0000 7f0d78db1640 1 --2- 172.21.15.28:0/2804360901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d74107240 0x7f0d7410ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.739+0000 7f0d78db1640 1 --2- 172.21.15.28:0/2804360901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d74102b20 0x7f0d74106b60 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.739+0000 7f0d78db1640 1 --2- 172.21.15.28:0/2804360901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d74102170 0x7f0d74102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.739+0000 7f0d78db1640 1 -- 172.21.15.28:0/2804360901 >> 172.21.15.28:0/2804360901 conn(0x7f0d740793a0 msgr2=0x7f0d740797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:15.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.739+0000 7f0d78db1640 1 -- 172.21.15.28:0/2804360901 shutdown_connections 2024-04-21T08:51:15.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.739+0000 7f0d78db1640 1 -- 172.21.15.28:0/2804360901 wait complete. 2024-04-21T08:51:15.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.740+0000 7f0d78db1640 1 Processor -- start 2024-04-21T08:51:15.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.740+0000 7f0d78db1640 1 -- start start 2024-04-21T08:51:15.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.740+0000 7f0d78db1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d74102170 0x7f0d74074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:15.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d78db1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d74107240 0x7f0d74074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:15.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d78db1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d74074cd0 0x7f0d741a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:15.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d78db1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d74112180 con 0x7f0d74074cd0 2024-04-21T08:51:15.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d78db1640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0d74112000 con 0x7f0d74102170 2024-04-21T08:51:15.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d78db1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0d74112300 con 0x7f0d74107240 2024-04-21T08:51:15.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d72d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d74074cd0 0x7f0d741a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:15.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d72d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d74074cd0 0x7f0d741a68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51934/0 (socket says 172.21.15.28:51934) 2024-04-21T08:51:15.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d72d9d640 1 -- 172.21.15.28:0/276676190 learned_addr learned my addr 172.21.15.28:0/276676190 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:15.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d69d9b640 1 --2- 172.21.15.28:0/276676190 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d74107240 0x7f0d74074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:15.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d7259c640 1 --2- 172.21.15.28:0/276676190 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d74102170 0x7f0d74074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:15.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d72d9d640 1 -- 172.21.15.28:0/276676190 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d74107240 msgr2=0x7f0d74074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:15.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d72d9d640 1 --2- 172.21.15.28:0/276676190 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d74107240 0x7f0d74074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d72d9d640 1 -- 172.21.15.28:0/276676190 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d74102170 msgr2=0x7f0d74074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:15.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d72d9d640 1 --2- 172.21.15.28:0/276676190 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d74102170 0x7f0d74074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d72d9d640 1 -- 172.21.15.28:0/276676190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d741a6fd0 con 0x7f0d74074cd0 2024-04-21T08:51:15.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d72d9d640 1 --2- 172.21.15.28:0/276676190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d74074cd0 0x7f0d741a68d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c00eaa0 tx=0x7f0d6c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:15.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.741+0000 7f0d6bfff640 1 -- 172.21.15.28:0/276676190 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c00cdb0 con 0x7f0d74074cd0 2024-04-21T08:51:15.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.742+0000 7f0d78db1640 1 -- 172.21.15.28:0/276676190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d741a72c0 con 0x7f0d74074cd0 2024-04-21T08:51:15.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.742+0000 7f0d6bfff640 1 -- 172.21.15.28:0/276676190 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d6c005020 con 0x7f0d74074cd0 2024-04-21T08:51:15.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.742+0000 7f0d6bfff640 1 -- 172.21.15.28:0/276676190 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c0116d0 con 0x7f0d74074cd0 2024-04-21T08:51:15.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.742+0000 7f0d78db1640 1 -- 172.21.15.28:0/276676190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d741a7790 con 0x7f0d74074cd0 2024-04-21T08:51:15.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.743+0000 7f0d78db1640 1 -- 172.21.15.28:0/276676190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d74069230 con 0x7f0d74074cd0 2024-04-21T08:51:15.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.744+0000 7f0d6bfff640 1 -- 172.21.15.28:0/276676190 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f0d6c0051d0 con 0x7f0d74074cd0 2024-04-21T08:51:15.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.744+0000 7f0d6bfff640 1 --2- 172.21.15.28:0/276676190 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0d600a3e70 0x7f0d600a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:15.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.744+0000 7f0d7259c640 1 --2- 172.21.15.28:0/276676190 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0d600a3e70 0x7f0d600a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:15.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.744+0000 7f0d6bfff640 1 -- 172.21.15.28:0/276676190 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f0d6c0d0ec0 con 0x7f0d74074cd0 2024-04-21T08:51:15.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.745+0000 7f0d7259c640 1 --2- 172.21.15.28:0/276676190 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0d600a3e70 0x7f0d600a6330 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0d64005fd0 tx=0x7f0d6400b390 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:15.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.749+0000 7f0d6bfff640 1 -- 172.21.15.28:0/276676190 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d6c016030 con 0x7f0d74074cd0 2024-04-21T08:51:15.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.933+0000 7f0d78db1640 1 -- 172.21.15.28:0/276676190 --> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0d74102b20 con 0x7f0d600a3e70 2024-04-21T08:51:15.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.944+0000 7f0d6bfff640 1 -- 172.21.15.28:0/276676190 <== mgr.25040 v2:172.21.15.28:6800/4182301910 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339688 (secure 0 0 0) 0x7f0d74102b20 con 0x7f0d600a3e70 2024-04-21T08:51:15.948 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-21T08:51:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.947+0000 7f0d78db1640 1 -- 172.21.15.28:0/276676190 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0d600a3e70 msgr2=0x7f0d600a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.947+0000 7f0d78db1640 1 --2- 172.21.15.28:0/276676190 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0d600a3e70 0x7f0d600a6330 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0d64005fd0 tx=0x7f0d6400b390 comp rx=0 tx=0).stop 2024-04-21T08:51:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.947+0000 7f0d78db1640 1 -- 172.21.15.28:0/276676190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d74074cd0 msgr2=0x7f0d741a68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.947+0000 7f0d78db1640 1 --2- 172.21.15.28:0/276676190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d74074cd0 0x7f0d741a68d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c00eaa0 tx=0x7f0d6c00ef60 comp rx=0 tx=0).stop 2024-04-21T08:51:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.947+0000 7f0d78db1640 1 -- 172.21.15.28:0/276676190 shutdown_connections 2024-04-21T08:51:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.947+0000 7f0d78db1640 1 --2- 172.21.15.28:0/276676190 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0d600a3e70 0x7f0d600a6330 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.947+0000 7f0d78db1640 1 --2- 172.21.15.28:0/276676190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d74074cd0 0x7f0d741a68d0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.947+0000 7f0d78db1640 1 --2- 172.21.15.28:0/276676190 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0d74107240 0x7f0d74074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.947+0000 7f0d78db1640 1 --2- 172.21.15.28:0/276676190 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0d74102170 0x7f0d74074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.947+0000 7f0d78db1640 1 -- 172.21.15.28:0/276676190 >> 172.21.15.28:0/276676190 conn(0x7f0d740793a0 msgr2=0x7f0d740fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.947+0000 7f0d78db1640 1 -- 172.21.15.28:0/276676190 shutdown_connections 2024-04-21T08:51:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:15.947+0000 7f0d78db1640 1 -- 172.21.15.28:0/276676190 wait complete. 2024-04-21T08:51:15.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: Updating smithi028:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:51:15.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: Updating smithi152:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:51:15.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: Updating smithi171:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:51:15.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:51:15.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:51:15.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:51:15.959 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:51:15.959 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3848978672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:15.959 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: Updating smithi152:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:51:15.959 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: Updating smithi028:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:51:15.959 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: Updating smithi171:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:51:15.959 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1588229387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:15.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-21T08:51:15.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:51:15.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:51:15.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:51:15.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:15.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:15 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/341422188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:15.963 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-21T08:51:14.842869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190050,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":971,"num_read_kb":642,"num_write":473,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181430,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370192,"kb_used_data":12680,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571632,"statfs":{"total":1151940427776,"available":1151561351168,"internally_reserved":0,"allocated":12984320,"data_stored":9426854,"data_compressed":518308,"data_compressed_allocated":704512,"data_compressed_original":3862528,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.990882"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063213+0000","last_change":"2024-04-21T08:48:46.471626+0000","last_active":"2024-04-21T08:51:11.063213+0000","last_peered":"2024-04-21T08:51:11.063213+0000","last_clean":"2024-04-21T08:51:11.063213+0000","last_became_active":"2024-04-21T08:48:46.470498+0000","last_became_peered":"2024-04-21T08:48:46.470498+0000","last_unstale":"2024-04-21T08:51:11.063213+0000","last_undegraded":"2024-04-21T08:51:11.063213+0000","last_fullsized":"2024-04-21T08:51:11.063213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:49.452181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877897+0000","last_change":"2024-04-21T08:48:46.469111+0000","last_active":"2024-04-21T08:51:10.877897+0000","last_peered":"2024-04-21T08:51:10.877897+0000","last_clean":"2024-04-21T08:51:10.877897+0000","last_became_active":"2024-04-21T08:48:46.467926+0000","last_became_peered":"2024-04-21T08:48:46.467926+0000","last_unstale":"2024-04-21T08:51:10.877897+0000","last_undegraded":"2024-04-21T08:51:10.877897+0000","last_fullsized":"2024-04-21T08:51:10.877897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:35:38.230212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877755+0000","last_change":"2024-04-21T08:48:46.475673+0000","last_active":"2024-04-21T08:51:10.877755+0000","last_peered":"2024-04-21T08:51:10.877755+0000","last_clean":"2024-04-21T08:51:10.877755+0000","last_became_active":"2024-04-21T08:48:46.474917+0000","last_became_peered":"2024-04-21T08:48:46.474917+0000","last_unstale":"2024-04-21T08:51:10.877755+0000","last_undegraded":"2024-04-21T08:51:10.877755+0000","last_fullsized":"2024-04-21T08:51:10.877755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:01.661704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038205+0000","last_change":"2024-04-21T08:48:46.468147+0000","last_active":"2024-04-21T08:51:11.038205+0000","last_peered":"2024-04-21T08:51:11.038205+0000","last_clean":"2024-04-21T08:51:11.038205+0000","last_became_active":"2024-04-21T08:48:46.467811+0000","last_became_peered":"2024-04-21T08:48:46.467811+0000","last_unstale":"2024-04-21T08:51:11.038205+0000","last_undegraded":"2024-04-21T08:51:11.038205+0000","last_fullsized":"2024-04-21T08:51:11.038205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:21.658605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939250+0000","last_change":"2024-04-21T08:48:46.471322+0000","last_active":"2024-04-21T08:51:10.939250+0000","last_peered":"2024-04-21T08:51:10.939250+0000","last_clean":"2024-04-21T08:51:10.939250+0000","last_became_active":"2024-04-21T08:48:46.470991+0000","last_became_peered":"2024-04-21T08:48:46.470991+0000","last_unstale":"2024-04-21T08:51:10.939250+0000","last_undegraded":"2024-04-21T08:51:10.939250+0000","last_fullsized":"2024-04-21T08:51:10.939250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:22.839870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829796+0000","last_change":"2024-04-21T08:48:46.467039+0000","last_active":"2024-04-21T08:51:10.829796+0000","last_peered":"2024-04-21T08:51:10.829796+0000","last_clean":"2024-04-21T08:51:10.829796+0000","last_became_active":"2024-04-21T08:48:46.466614+0000","last_became_peered":"2024-04-21T08:48:46.466614+0000","last_unstale":"2024-04-21T08:51:10.829796+0000","last_undegraded":"2024-04-21T08:51:10.829796+0000","last_fullsized":"2024-04-21T08:51:10.829796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:38.029215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062714+0000","last_change":"2024-04-21T08:48:46.473268+0000","last_active":"2024-04-21T08:51:11.062714+0000","last_peered":"2024-04-21T08:51:11.062714+0000","last_clean":"2024-04-21T08:51:11.062714+0000","last_became_active":"2024-04-21T08:48:46.472261+0000","last_became_peered":"2024-04-21T08:48:46.472261+0000","last_unstale":"2024-04-21T08:51:11.062714+0000","last_undegraded":"2024-04-21T08:51:11.062714+0000","last_fullsized":"2024-04-21T08:51:11.062714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:58:52.990687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828189+0000","last_change":"2024-04-21T08:48:46.472788+0000","last_active":"2024-04-21T08:51:10.828189+0000","last_peered":"2024-04-21T08:51:10.828189+0000","last_clean":"2024-04-21T08:51:10.828189+0000","last_became_active":"2024-04-21T08:48:46.472482+0000","last_became_peered":"2024-04-21T08:48:46.472482+0000","last_unstale":"2024-04-21T08:51:10.828189+0000","last_undegraded":"2024-04-21T08:51:10.828189+0000","last_fullsized":"2024-04-21T08:51:10.828189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:48.483846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006966+0000","last_change":"2024-04-21T08:48:46.460771+0000","last_active":"2024-04-21T08:51:11.006966+0000","last_peered":"2024-04-21T08:51:11.006966+0000","last_clean":"2024-04-21T08:51:11.006966+0000","last_became_active":"2024-04-21T08:48:46.460472+0000","last_became_peered":"2024-04-21T08:48:46.460472+0000","last_unstale":"2024-04-21T08:51:11.006966+0000","last_undegraded":"2024-04-21T08:51:11.006966+0000","last_fullsized":"2024-04-21T08:51:11.006966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:46:40.766245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.830308+0000","last_change":"2024-04-21T08:48:46.463475+0000","last_active":"2024-04-21T08:51:10.830308+0000","last_peered":"2024-04-21T08:51:10.830308+0000","last_clean":"2024-04-21T08:51:10.830308+0000","last_became_active":"2024-04-21T08:48:46.462559+0000","last_became_peered":"2024-04-21T08:48:46.462559+0000","last_unstale":"2024-04-21T08:51:10.830308+0000","last_undegraded":"2024-04-21T08:51:10.830308+0000","last_fullsized":"2024-04-21T08:51:10.830308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:22:06.372410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063119+0000","last_change":"2024-04-21T08:48:46.476997+0000","last_active":"2024-04-21T08:51:11.063119+0000","last_peered":"2024-04-21T08:51:11.063119+0000","last_clean":"2024-04-21T08:51:11.063119+0000","last_became_active":"2024-04-21T08:48:46.476620+0000","last_became_peered":"2024-04-21T08:48:46.476620+0000","last_unstale":"2024-04-21T08:51:11.063119+0000","last_undegraded":"2024-04-21T08:51:11.063119+0000","last_fullsized":"2024-04-21T08:51:11.063119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:37.569629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038421+0000","last_change":"2024-04-21T08:48:46.469355+0000","last_active":"2024-04-21T08:51:11.038421+0000","last_peered":"2024-04-21T08:51:11.038421+0000","last_clean":"2024-04-21T08:51:11.038421+0000","last_became_active":"2024-04-21T08:48:46.468604+0000","last_became_peered":"2024-04-21T08:48:46.468604+0000","last_unstale":"2024-04-21T08:51:11.038421+0000","last_undegraded":"2024-04-21T08:51:11.038421+0000","last_fullsized":"2024-04-21T08:51:11.038421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:23:54.500982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036699+0000","last_change":"2024-04-21T08:48:46.453116+0000","last_active":"2024-04-21T08:51:11.036699+0000","last_peered":"2024-04-21T08:51:11.036699+0000","last_clean":"2024-04-21T08:51:11.036699+0000","last_became_active":"2024-04-21T08:48:46.452607+0000","last_became_peered":"2024-04-21T08:48:46.452607+0000","last_unstale":"2024-04-21T08:51:11.036699+0000","last_undegraded":"2024-04-21T08:51:11.036699+0000","last_fullsized":"2024-04-21T08:51:11.036699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:57:04.371812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877416+0000","last_change":"2024-04-21T08:48:46.474516+0000","last_active":"2024-04-21T08:51:10.877416+0000","last_peered":"2024-04-21T08:51:10.877416+0000","last_clean":"2024-04-21T08:51:10.877416+0000","last_became_active":"2024-04-21T08:48:46.474129+0000","last_became_peered":"2024-04-21T08:48:46.474129+0000","last_unstale":"2024-04-21T08:51:10.877416+0000","last_undegraded":"2024-04-21T08:51:10.877416+0000","last_fullsized":"2024-04-21T08:51:10.877416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:40:47.784976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877479+0000","last_change":"2024-04-21T08:49:36.907964+0000","last_active":"2024-04-21T08:51:10.877479+0000","last_peered":"2024-04-21T08:51:10.877479+0000","last_clean":"2024-04-21T08:51:10.877479+0000","last_became_active":"2024-04-21T08:49:36.907595+0000","last_became_peered":"2024-04-21T08:49:36.907595+0000","last_unstale":"2024-04-21T08:51:10.877479+0000","last_undegraded":"2024-04-21T08:51:10.877479+0000","last_fullsized":"2024-04-21T08:51:10.877479+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:14:04.907328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006192+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:51:11.006192+0000","last_peered":"2024-04-21T08:51:11.006192+0000","last_clean":"2024-04-21T08:51:11.006192+0000","last_became_active":"2024-04-21T08:48:46.469389+0000","last_became_peered":"2024-04-21T08:48:46.469389+0000","last_unstale":"2024-04-21T08:51:11.006192+0000","last_undegraded":"2024-04-21T08:51:11.006192+0000","last_fullsized":"2024-04-21T08:51:11.006192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:11:45.824402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063161+0000","last_change":"2024-04-21T08:48:46.470059+0000","last_active":"2024-04-21T08:51:11.063161+0000","last_peered":"2024-04-21T08:51:11.063161+0000","last_clean":"2024-04-21T08:51:11.063161+0000","last_became_active":"2024-04-21T08:48:46.469751+0000","last_became_peered":"2024-04-21T08:48:46.469751+0000","last_unstale":"2024-04-21T08:51:11.063161+0000","last_undegraded":"2024-04-21T08:51:11.063161+0000","last_fullsized":"2024-04-21T08:51:11.063161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:16.317879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.937061+0000","last_change":"2024-04-21T08:48:46.470129+0000","last_active":"2024-04-21T08:51:10.937061+0000","last_peered":"2024-04-21T08:51:10.937061+0000","last_clean":"2024-04-21T08:51:10.937061+0000","last_became_active":"2024-04-21T08:48:46.469763+0000","last_became_peered":"2024-04-21T08:48:46.469763+0000","last_unstale":"2024-04-21T08:51:10.937061+0000","last_undegraded":"2024-04-21T08:51:10.937061+0000","last_fullsized":"2024-04-21T08:51:10.937061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:47.731201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832982+0000","last_change":"2024-04-21T08:48:46.465356+0000","last_active":"2024-04-21T08:51:10.832982+0000","last_peered":"2024-04-21T08:51:10.832982+0000","last_clean":"2024-04-21T08:51:10.832982+0000","last_became_active":"2024-04-21T08:48:46.464183+0000","last_became_peered":"2024-04-21T08:48:46.464183+0000","last_unstale":"2024-04-21T08:51:10.832982+0000","last_undegraded":"2024-04-21T08:51:10.832982+0000","last_fullsized":"2024-04-21T08:51:10.832982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:37.533733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832348+0000","last_change":"2024-04-21T08:48:50.767378+0000","last_active":"2024-04-21T08:51:10.832348+0000","last_peered":"2024-04-21T08:51:10.832348+0000","last_clean":"2024-04-21T08:51:10.832348+0000","last_became_active":"2024-04-21T08:48:50.766856+0000","last_became_peered":"2024-04-21T08:48:50.766856+0000","last_unstale":"2024-04-21T08:51:10.832348+0000","last_undegraded":"2024-04-21T08:51:10.832348+0000","last_fullsized":"2024-04-21T08:51:10.832348+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:00.434540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936743+0000","last_change":"2024-04-21T08:48:45.550284+0000","last_active":"2024-04-21T08:51:10.936743+0000","last_peered":"2024-04-21T08:51:10.936743+0000","last_clean":"2024-04-21T08:51:10.936743+0000","last_became_active":"2024-04-21T08:48:45.549940+0000","last_became_peered":"2024-04-21T08:48:45.549940+0000","last_unstale":"2024-04-21T08:51:10.936743+0000","last_undegraded":"2024-04-21T08:51:10.936743+0000","last_fullsized":"2024-04-21T08:51:10.936743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:12.343299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828471+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:51:10.828471+0000","last_peered":"2024-04-21T08:51:10.828471+0000","last_clean":"2024-04-21T08:51:10.828471+0000","last_became_active":"2024-04-21T08:48:46.469647+0000","last_became_peered":"2024-04-21T08:48:46.469647+0000","last_unstale":"2024-04-21T08:51:10.828471+0000","last_undegraded":"2024-04-21T08:51:10.828471+0000","last_fullsized":"2024-04-21T08:51:10.828471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:10:28.204554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.037930+0000","last_change":"2024-04-21T08:48:50.771871+0000","last_active":"2024-04-21T08:51:11.037930+0000","last_peered":"2024-04-21T08:51:11.037930+0000","last_clean":"2024-04-21T08:51:11.037930+0000","last_became_active":"2024-04-21T08:48:50.770442+0000","last_became_peered":"2024-04-21T08:48:50.770442+0000","last_unstale":"2024-04-21T08:51:11.037930+0000","last_undegraded":"2024-04-21T08:51:11.037930+0000","last_fullsized":"2024-04-21T08:51:11.037930+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:18:57.813971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828497+0000","last_change":"2024-04-21T08:48:45.566492+0000","last_active":"2024-04-21T08:51:10.828497+0000","last_peered":"2024-04-21T08:51:10.828497+0000","last_clean":"2024-04-21T08:51:10.828497+0000","last_became_active":"2024-04-21T08:48:45.566169+0000","last_became_peered":"2024-04-21T08:48:45.566169+0000","last_unstale":"2024-04-21T08:51:10.828497+0000","last_undegraded":"2024-04-21T08:51:10.828497+0000","last_fullsized":"2024-04-21T08:51:10.828497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:49.985664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877476+0000","last_change":"2024-04-21T08:48:46.470866+0000","last_active":"2024-04-21T08:51:10.877476+0000","last_peered":"2024-04-21T08:51:10.877476+0000","last_clean":"2024-04-21T08:51:10.877476+0000","last_became_active":"2024-04-21T08:48:46.470401+0000","last_became_peered":"2024-04-21T08:48:46.470401+0000","last_unstale":"2024-04-21T08:51:10.877476+0000","last_undegraded":"2024-04-21T08:51:10.877476+0000","last_fullsized":"2024-04-21T08:51:10.877476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:07.506212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038069+0000","last_change":"2024-04-21T08:48:50.772339+0000","last_active":"2024-04-21T08:51:11.038069+0000","last_peered":"2024-04-21T08:51:11.038069+0000","last_clean":"2024-04-21T08:51:11.038069+0000","last_became_active":"2024-04-21T08:48:50.771946+0000","last_became_peered":"2024-04-21T08:48:50.771946+0000","last_unstale":"2024-04-21T08:51:11.038069+0000","last_undegraded":"2024-04-21T08:51:11.038069+0000","last_fullsized":"2024-04-21T08:51:11.038069+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:03:56.994581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939792+0000","last_change":"2024-04-21T08:48:45.553946+0000","last_active":"2024-04-21T08:51:10.939792+0000","last_peered":"2024-04-21T08:51:10.939792+0000","last_clean":"2024-04-21T08:51:10.939792+0000","last_became_active":"2024-04-21T08:48:45.553410+0000","last_became_peered":"2024-04-21T08:48:45.553410+0000","last_unstale":"2024-04-21T08:51:10.939792+0000","last_undegraded":"2024-04-21T08:51:10.939792+0000","last_fullsized":"2024-04-21T08:51:10.939792+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:24:02.387250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006377+0000","last_change":"2024-04-21T08:48:46.471815+0000","last_active":"2024-04-21T08:51:11.006377+0000","last_peered":"2024-04-21T08:51:11.006377+0000","last_clean":"2024-04-21T08:51:11.006377+0000","last_became_active":"2024-04-21T08:48:46.471492+0000","last_became_peered":"2024-04-21T08:48:46.471492+0000","last_unstale":"2024-04-21T08:51:11.006377+0000","last_undegraded":"2024-04-21T08:51:11.006377+0000","last_fullsized":"2024-04-21T08:51:11.006377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:26.051166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.037968+0000","last_change":"2024-04-21T08:48:50.767916+0000","last_active":"2024-04-21T08:51:11.037968+0000","last_peered":"2024-04-21T08:51:11.037968+0000","last_clean":"2024-04-21T08:51:11.037968+0000","last_became_active":"2024-04-21T08:48:50.767326+0000","last_became_peered":"2024-04-21T08:48:50.767326+0000","last_unstale":"2024-04-21T08:51:11.037968+0000","last_undegraded":"2024-04-21T08:51:11.037968+0000","last_fullsized":"2024-04-21T08:51:11.037968+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:05:02.185369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939279+0000","last_change":"2024-04-21T08:48:45.549982+0000","last_active":"2024-04-21T08:51:10.939279+0000","last_peered":"2024-04-21T08:51:10.939279+0000","last_clean":"2024-04-21T08:51:10.939279+0000","last_became_active":"2024-04-21T08:48:45.549611+0000","last_became_peered":"2024-04-21T08:48:45.549611+0000","last_unstale":"2024-04-21T08:51:10.939279+0000","last_undegraded":"2024-04-21T08:51:10.939279+0000","last_fullsized":"2024-04-21T08:51:10.939279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:46.035388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945847+0000","last_change":"2024-04-21T08:48:46.459068+0000","last_active":"2024-04-21T08:51:10.945847+0000","last_peered":"2024-04-21T08:51:10.945847+0000","last_clean":"2024-04-21T08:51:10.945847+0000","last_became_active":"2024-04-21T08:48:46.458686+0000","last_became_peered":"2024-04-21T08:48:46.458686+0000","last_unstale":"2024-04-21T08:51:10.945847+0000","last_undegraded":"2024-04-21T08:51:10.945847+0000","last_fullsized":"2024-04-21T08:51:10.945847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:31.373451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832831+0000","last_change":"2024-04-21T08:48:50.761392+0000","last_active":"2024-04-21T08:51:10.832831+0000","last_peered":"2024-04-21T08:51:10.832831+0000","last_clean":"2024-04-21T08:51:10.832831+0000","last_became_active":"2024-04-21T08:48:50.760917+0000","last_became_peered":"2024-04-21T08:48:50.760917+0000","last_unstale":"2024-04-21T08:51:10.832831+0000","last_undegraded":"2024-04-21T08:51:10.832831+0000","last_fullsized":"2024-04-21T08:51:10.832831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:19.070502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'8","reported_seq":163,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.097987+0000","last_change":"2024-04-21T08:48:45.553686+0000","last_active":"2024-04-21T08:51:11.097987+0000","last_peered":"2024-04-21T08:51:11.097987+0000","last_clean":"2024-04-21T08:51:11.097987+0000","last_became_active":"2024-04-21T08:48:45.553322+0000","last_became_peered":"2024-04-21T08:48:45.553322+0000","last_unstale":"2024-04-21T08:51:11.097987+0000","last_undegraded":"2024-04-21T08:51:11.097987+0000","last_fullsized":"2024-04-21T08:51:11.097987+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:15:24.320704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1712,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":3,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939561+0000","last_change":"2024-04-21T08:48:46.474555+0000","last_active":"2024-04-21T08:51:10.939561+0000","last_peered":"2024-04-21T08:51:10.939561+0000","last_clean":"2024-04-21T08:51:10.939561+0000","last_became_active":"2024-04-21T08:48:46.474185+0000","last_became_peered":"2024-04-21T08:48:46.474185+0000","last_unstale":"2024-04-21T08:51:10.939561+0000","last_undegraded":"2024-04-21T08:51:10.939561+0000","last_fullsized":"2024-04-21T08:51:10.939561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:52.884994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832578+0000","last_change":"2024-04-21T08:48:50.763181+0000","last_active":"2024-04-21T08:51:10.832578+0000","last_peered":"2024-04-21T08:51:10.832578+0000","last_clean":"2024-04-21T08:51:10.832578+0000","last_became_active":"2024-04-21T08:48:50.762718+0000","last_became_peered":"2024-04-21T08:48:50.762718+0000","last_unstale":"2024-04-21T08:51:10.832578+0000","last_undegraded":"2024-04-21T08:51:10.832578+0000","last_fullsized":"2024-04-21T08:51:10.832578+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:16:23.204605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829770+0000","last_change":"2024-04-21T08:48:45.547768+0000","last_active":"2024-04-21T08:51:10.829770+0000","last_peered":"2024-04-21T08:51:10.829770+0000","last_clean":"2024-04-21T08:51:10.829770+0000","last_became_active":"2024-04-21T08:48:45.547456+0000","last_became_peered":"2024-04-21T08:48:45.547456+0000","last_unstale":"2024-04-21T08:51:10.829770+0000","last_undegraded":"2024-04-21T08:51:10.829770+0000","last_fullsized":"2024-04-21T08:51:10.829770+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:23.502799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945790+0000","last_change":"2024-04-21T08:48:46.467988+0000","last_active":"2024-04-21T08:51:10.945790+0000","last_peered":"2024-04-21T08:51:10.945790+0000","last_clean":"2024-04-21T08:51:10.945790+0000","last_became_active":"2024-04-21T08:48:46.465212+0000","last_became_peered":"2024-04-21T08:48:46.465212+0000","last_unstale":"2024-04-21T08:51:10.945790+0000","last_undegraded":"2024-04-21T08:51:10.945790+0000","last_fullsized":"2024-04-21T08:51:10.945790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:41.743506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063079+0000","last_change":"2024-04-21T08:50:37.350070+0000","last_active":"2024-04-21T08:51:11.063079+0000","last_peered":"2024-04-21T08:51:11.063079+0000","last_clean":"2024-04-21T08:51:11.063079+0000","last_became_active":"2024-04-21T08:50:37.349622+0000","last_became_peered":"2024-04-21T08:50:37.349622+0000","last_unstale":"2024-04-21T08:51:11.063079+0000","last_undegraded":"2024-04-21T08:51:11.063079+0000","last_fullsized":"2024-04-21T08:51:11.063079+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:55.050244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.035911+0000","last_change":"2024-04-21T08:48:45.545615+0000","last_active":"2024-04-21T08:51:11.035911+0000","last_peered":"2024-04-21T08:51:11.035911+0000","last_clean":"2024-04-21T08:51:11.035911+0000","last_became_active":"2024-04-21T08:48:45.545280+0000","last_became_peered":"2024-04-21T08:48:45.545280+0000","last_unstale":"2024-04-21T08:51:11.035911+0000","last_undegraded":"2024-04-21T08:51:11.035911+0000","last_fullsized":"2024-04-21T08:51:11.035911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:50:00.470903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828012+0000","last_change":"2024-04-21T08:48:46.477326+0000","last_active":"2024-04-21T08:51:10.828012+0000","last_peered":"2024-04-21T08:51:10.828012+0000","last_clean":"2024-04-21T08:51:10.828012+0000","last_became_active":"2024-04-21T08:48:46.476914+0000","last_became_peered":"2024-04-21T08:48:46.476914+0000","last_unstale":"2024-04-21T08:51:10.828012+0000","last_undegraded":"2024-04-21T08:51:10.828012+0000","last_fullsized":"2024-04-21T08:51:10.828012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:04.312061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877774+0000","last_change":"2024-04-21T08:48:50.767429+0000","last_active":"2024-04-21T08:51:10.877774+0000","last_peered":"2024-04-21T08:51:10.877774+0000","last_clean":"2024-04-21T08:51:10.877774+0000","last_became_active":"2024-04-21T08:48:50.767008+0000","last_became_peered":"2024-04-21T08:48:50.767008+0000","last_unstale":"2024-04-21T08:51:10.877774+0000","last_undegraded":"2024-04-21T08:51:10.877774+0000","last_fullsized":"2024-04-21T08:51:10.877774+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:26.865571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945279+0000","last_change":"2024-04-21T08:48:45.553170+0000","last_active":"2024-04-21T08:51:10.945279+0000","last_peered":"2024-04-21T08:51:10.945279+0000","last_clean":"2024-04-21T08:51:10.945279+0000","last_became_active":"2024-04-21T08:48:45.552645+0000","last_became_peered":"2024-04-21T08:48:45.552645+0000","last_unstale":"2024-04-21T08:51:10.945279+0000","last_undegraded":"2024-04-21T08:51:10.945279+0000","last_fullsized":"2024-04-21T08:51:10.945279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:32:07.217212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936256+0000","last_change":"2024-04-21T08:48:46.478101+0000","last_active":"2024-04-21T08:51:10.936256+0000","last_peered":"2024-04-21T08:51:10.936256+0000","last_clean":"2024-04-21T08:51:10.936256+0000","last_became_active":"2024-04-21T08:48:46.477725+0000","last_became_peered":"2024-04-21T08:48:46.477725+0000","last_unstale":"2024-04-21T08:51:10.936256+0000","last_undegraded":"2024-04-21T08:51:10.936256+0000","last_fullsized":"2024-04-21T08:51:10.936256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:37:55.429569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828279+0000","last_change":"2024-04-21T08:48:50.762899+0000","last_active":"2024-04-21T08:51:10.828279+0000","last_peered":"2024-04-21T08:51:10.828279+0000","last_clean":"2024-04-21T08:51:10.828279+0000","last_became_active":"2024-04-21T08:48:50.762475+0000","last_became_peered":"2024-04-21T08:48:50.762475+0000","last_unstale":"2024-04-21T08:51:10.828279+0000","last_undegraded":"2024-04-21T08:51:10.828279+0000","last_fullsized":"2024-04-21T08:51:10.828279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:27.540414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":157,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.014871+0000","last_change":"2024-04-21T08:48:45.550151+0000","last_active":"2024-04-21T08:51:11.014871+0000","last_peered":"2024-04-21T08:51:11.014871+0000","last_clean":"2024-04-21T08:51:11.014871+0000","last_became_active":"2024-04-21T08:48:45.549829+0000","last_became_peered":"2024-04-21T08:48:45.549829+0000","last_unstale":"2024-04-21T08:51:11.014871+0000","last_undegraded":"2024-04-21T08:51:11.014871+0000","last_fullsized":"2024-04-21T08:51:11.014871+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:14.061189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006909+0000","last_change":"2024-04-21T08:48:46.472072+0000","last_active":"2024-04-21T08:51:11.006909+0000","last_peered":"2024-04-21T08:51:11.006909+0000","last_clean":"2024-04-21T08:51:11.006909+0000","last_became_active":"2024-04-21T08:48:46.471660+0000","last_became_peered":"2024-04-21T08:48:46.471660+0000","last_unstale":"2024-04-21T08:51:11.006909+0000","last_undegraded":"2024-04-21T08:51:11.006909+0000","last_fullsized":"2024-04-21T08:51:11.006909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:30:11.843870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936126+0000","last_change":"2024-04-21T08:48:50.773021+0000","last_active":"2024-04-21T08:51:10.936126+0000","last_peered":"2024-04-21T08:51:10.936126+0000","last_clean":"2024-04-21T08:51:10.936126+0000","last_became_active":"2024-04-21T08:48:50.771746+0000","last_became_peered":"2024-04-21T08:48:50.771746+0000","last_unstale":"2024-04-21T08:51:10.936126+0000","last_undegraded":"2024-04-21T08:51:10.936126+0000","last_fullsized":"2024-04-21T08:51:10.936126+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:33:21.415967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063066+0000","last_change":"2024-04-21T08:48:45.550376+0000","last_active":"2024-04-21T08:51:11.063066+0000","last_peered":"2024-04-21T08:51:11.063066+0000","last_clean":"2024-04-21T08:51:11.063066+0000","last_became_active":"2024-04-21T08:48:45.549594+0000","last_became_peered":"2024-04-21T08:48:45.549594+0000","last_unstale":"2024-04-21T08:51:11.063066+0000","last_undegraded":"2024-04-21T08:51:11.063066+0000","last_fullsized":"2024-04-21T08:51:11.063066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:26:27.046963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877696+0000","last_change":"2024-04-21T08:48:46.469499+0000","last_active":"2024-04-21T08:51:10.877696+0000","last_peered":"2024-04-21T08:51:10.877696+0000","last_clean":"2024-04-21T08:51:10.877696+0000","last_became_active":"2024-04-21T08:48:46.469124+0000","last_became_peered":"2024-04-21T08:48:46.469124+0000","last_unstale":"2024-04-21T08:51:10.877696+0000","last_undegraded":"2024-04-21T08:51:10.877696+0000","last_fullsized":"2024-04-21T08:51:10.877696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:58.708916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038212+0000","last_change":"2024-04-21T08:50:37.352313+0000","last_active":"2024-04-21T08:51:11.038212+0000","last_peered":"2024-04-21T08:51:11.038212+0000","last_clean":"2024-04-21T08:51:11.038212+0000","last_became_active":"2024-04-21T08:50:37.351775+0000","last_became_peered":"2024-04-21T08:50:37.351775+0000","last_unstale":"2024-04-21T08:51:11.038212+0000","last_undegraded":"2024-04-21T08:51:11.038212+0000","last_fullsized":"2024-04-21T08:51:11.038212+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:12:51.239368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,10],"acting":[4,1,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945985+0000","last_change":"2024-04-21T08:48:45.559319+0000","last_active":"2024-04-21T08:51:10.945985+0000","last_peered":"2024-04-21T08:51:10.945985+0000","last_clean":"2024-04-21T08:51:10.945985+0000","last_became_active":"2024-04-21T08:48:45.558895+0000","last_became_peered":"2024-04-21T08:48:45.558895+0000","last_unstale":"2024-04-21T08:51:10.945985+0000","last_undegraded":"2024-04-21T08:51:10.945985+0000","last_fullsized":"2024-04-21T08:51:10.945985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:01.878169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006563+0000","last_change":"2024-04-21T08:48:46.466110+0000","last_active":"2024-04-21T08:51:11.006563+0000","last_peered":"2024-04-21T08:51:11.006563+0000","last_clean":"2024-04-21T08:51:11.006563+0000","last_became_active":"2024-04-21T08:48:46.464374+0000","last_became_peered":"2024-04-21T08:48:46.464374+0000","last_unstale":"2024-04-21T08:51:11.006563+0000","last_undegraded":"2024-04-21T08:51:11.006563+0000","last_fullsized":"2024-04-21T08:51:11.006563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:26.307051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936310+0000","last_change":"2024-04-21T08:48:50.759976+0000","last_active":"2024-04-21T08:51:10.936310+0000","last_peered":"2024-04-21T08:51:10.936310+0000","last_clean":"2024-04-21T08:51:10.936310+0000","last_became_active":"2024-04-21T08:48:50.759596+0000","last_became_peered":"2024-04-21T08:48:50.759596+0000","last_unstale":"2024-04-21T08:51:10.936310+0000","last_undegraded":"2024-04-21T08:51:10.936310+0000","last_fullsized":"2024-04-21T08:51:10.936310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:09.678388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":322,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.136178+0000","last_change":"2024-04-21T08:48:45.549228+0000","last_active":"2024-04-21T08:51:11.136178+0000","last_peered":"2024-04-21T08:51:11.136178+0000","last_clean":"2024-04-21T08:51:11.136178+0000","last_became_active":"2024-04-21T08:48:45.548865+0000","last_became_peered":"2024-04-21T08:48:45.548865+0000","last_unstale":"2024-04-21T08:51:11.136178+0000","last_undegraded":"2024-04-21T08:51:11.136178+0000","last_fullsized":"2024-04-21T08:51:11.136178+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:28.620968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69009,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":248,"num_read_kb":187,"num_write":41,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939637+0000","last_change":"2024-04-21T08:48:46.470322+0000","last_active":"2024-04-21T08:51:10.939637+0000","last_peered":"2024-04-21T08:51:10.939637+0000","last_clean":"2024-04-21T08:51:10.939637+0000","last_became_active":"2024-04-21T08:48:46.467225+0000","last_became_peered":"2024-04-21T08:48:46.467225+0000","last_unstale":"2024-04-21T08:51:10.939637+0000","last_undegraded":"2024-04-21T08:51:10.939637+0000","last_fullsized":"2024-04-21T08:51:10.939637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:31:17.390926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.946282+0000","last_change":"2024-04-21T08:48:50.770982+0000","last_active":"2024-04-21T08:51:10.946282+0000","last_peered":"2024-04-21T08:51:10.946282+0000","last_clean":"2024-04-21T08:51:10.946282+0000","last_became_active":"2024-04-21T08:48:50.770312+0000","last_became_peered":"2024-04-21T08:48:50.770312+0000","last_unstale":"2024-04-21T08:51:10.946282+0000","last_undegraded":"2024-04-21T08:51:10.946282+0000","last_fullsized":"2024-04-21T08:51:10.946282+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:41:33.027351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832599+0000","last_change":"2024-04-21T08:48:45.556134+0000","last_active":"2024-04-21T08:51:10.832599+0000","last_peered":"2024-04-21T08:51:10.832599+0000","last_clean":"2024-04-21T08:51:10.832599+0000","last_became_active":"2024-04-21T08:48:45.555818+0000","last_became_peered":"2024-04-21T08:48:45.555818+0000","last_unstale":"2024-04-21T08:51:10.832599+0000","last_undegraded":"2024-04-21T08:51:10.832599+0000","last_fullsized":"2024-04-21T08:51:10.832599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:00:08.083783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036365+0000","last_change":"2024-04-21T08:49:36.913151+0000","last_active":"2024-04-21T08:51:11.036365+0000","last_peered":"2024-04-21T08:51:11.036365+0000","last_clean":"2024-04-21T08:51:11.036365+0000","last_became_active":"2024-04-21T08:49:36.912810+0000","last_became_peered":"2024-04-21T08:49:36.912810+0000","last_unstale":"2024-04-21T08:51:11.036365+0000","last_undegraded":"2024-04-21T08:51:11.036365+0000","last_fullsized":"2024-04-21T08:51:11.036365+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:17:32.887163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062896+0000","last_change":"2024-04-21T08:48:50.765967+0000","last_active":"2024-04-21T08:51:11.062896+0000","last_peered":"2024-04-21T08:51:11.062896+0000","last_clean":"2024-04-21T08:51:11.062896+0000","last_became_active":"2024-04-21T08:48:50.765557+0000","last_became_peered":"2024-04-21T08:48:50.765557+0000","last_unstale":"2024-04-21T08:51:11.062896+0000","last_undegraded":"2024-04-21T08:51:11.062896+0000","last_fullsized":"2024-04-21T08:51:11.062896+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:52.044058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945823+0000","last_change":"2024-04-21T08:48:45.559939+0000","last_active":"2024-04-21T08:51:10.945823+0000","last_peered":"2024-04-21T08:51:10.945823+0000","last_clean":"2024-04-21T08:51:10.945823+0000","last_became_active":"2024-04-21T08:48:45.559635+0000","last_became_peered":"2024-04-21T08:48:45.559635+0000","last_unstale":"2024-04-21T08:51:10.945823+0000","last_undegraded":"2024-04-21T08:51:10.945823+0000","last_fullsized":"2024-04-21T08:51:10.945823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:17.872806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062944+0000","last_change":"2024-04-21T08:48:46.464878+0000","last_active":"2024-04-21T08:51:11.062944+0000","last_peered":"2024-04-21T08:51:11.062944+0000","last_clean":"2024-04-21T08:51:11.062944+0000","last_became_active":"2024-04-21T08:48:46.464552+0000","last_became_peered":"2024-04-21T08:48:46.464552+0000","last_unstale":"2024-04-21T08:51:11.062944+0000","last_undegraded":"2024-04-21T08:51:11.062944+0000","last_fullsized":"2024-04-21T08:51:11.062944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:46:10.558693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939454+0000","last_change":"2024-04-21T08:50:37.351314+0000","last_active":"2024-04-21T08:51:10.939454+0000","last_peered":"2024-04-21T08:51:10.939454+0000","last_clean":"2024-04-21T08:51:10.939454+0000","last_became_active":"2024-04-21T08:50:37.350956+0000","last_became_peered":"2024-04-21T08:50:37.350956+0000","last_unstale":"2024-04-21T08:51:10.939454+0000","last_undegraded":"2024-04-21T08:51:10.939454+0000","last_fullsized":"2024-04-21T08:51:10.939454+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:01.802427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"94'6","reported_seq":163,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.103467+0000","last_change":"2024-04-21T08:48:45.548685+0000","last_active":"2024-04-21T08:51:11.103467+0000","last_peered":"2024-04-21T08:51:11.103467+0000","last_clean":"2024-04-21T08:51:11.103467+0000","last_became_active":"2024-04-21T08:48:45.548372+0000","last_became_peered":"2024-04-21T08:48:45.548372+0000","last_unstale":"2024-04-21T08:51:11.103467+0000","last_undegraded":"2024-04-21T08:51:11.103467+0000","last_fullsized":"2024-04-21T08:51:11.103467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:44.780297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":5,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062806+0000","last_change":"2024-04-21T08:48:46.471541+0000","last_active":"2024-04-21T08:51:11.062806+0000","last_peered":"2024-04-21T08:51:11.062806+0000","last_clean":"2024-04-21T08:51:11.062806+0000","last_became_active":"2024-04-21T08:48:46.471223+0000","last_became_peered":"2024-04-21T08:48:46.471223+0000","last_unstale":"2024-04-21T08:51:11.062806+0000","last_undegraded":"2024-04-21T08:51:11.062806+0000","last_fullsized":"2024-04-21T08:51:11.062806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:24:50.867905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036934+0000","last_change":"2024-04-21T08:48:50.773941+0000","last_active":"2024-04-21T08:51:11.036934+0000","last_peered":"2024-04-21T08:51:11.036934+0000","last_clean":"2024-04-21T08:51:11.036934+0000","last_became_active":"2024-04-21T08:48:50.772081+0000","last_became_peered":"2024-04-21T08:48:50.772081+0000","last_unstale":"2024-04-21T08:51:11.036934+0000","last_undegraded":"2024-04-21T08:51:11.036934+0000","last_fullsized":"2024-04-21T08:51:11.036934+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:04.438944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062783+0000","last_change":"2024-04-21T08:49:36.923090+0000","last_active":"2024-04-21T08:51:11.062783+0000","last_peered":"2024-04-21T08:51:11.062783+0000","last_clean":"2024-04-21T08:51:11.062783+0000","last_became_active":"2024-04-21T08:49:36.908075+0000","last_became_peered":"2024-04-21T08:49:36.908075+0000","last_unstale":"2024-04-21T08:51:11.062783+0000","last_undegraded":"2024-04-21T08:51:11.062783+0000","last_fullsized":"2024-04-21T08:51:11.062783+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:41.683372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945471+0000","last_change":"2024-04-21T08:48:46.465559+0000","last_active":"2024-04-21T08:51:10.945471+0000","last_peered":"2024-04-21T08:51:10.945471+0000","last_clean":"2024-04-21T08:51:10.945471+0000","last_became_active":"2024-04-21T08:48:46.465125+0000","last_became_peered":"2024-04-21T08:48:46.465125+0000","last_unstale":"2024-04-21T08:51:10.945471+0000","last_undegraded":"2024-04-21T08:51:10.945471+0000","last_fullsized":"2024-04-21T08:51:10.945471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:52:38.423925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945406+0000","last_change":"2024-04-21T08:48:50.763609+0000","last_active":"2024-04-21T08:51:10.945406+0000","last_peered":"2024-04-21T08:51:10.945406+0000","last_clean":"2024-04-21T08:51:10.945406+0000","last_became_active":"2024-04-21T08:48:50.763144+0000","last_became_peered":"2024-04-21T08:48:50.763144+0000","last_unstale":"2024-04-21T08:51:10.945406+0000","last_undegraded":"2024-04-21T08:51:10.945406+0000","last_fullsized":"2024-04-21T08:51:10.945406+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:26:18.611235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939650+0000","last_change":"2024-04-21T08:48:45.549912+0000","last_active":"2024-04-21T08:51:10.939650+0000","last_peered":"2024-04-21T08:51:10.939650+0000","last_clean":"2024-04-21T08:51:10.939650+0000","last_became_active":"2024-04-21T08:48:45.549585+0000","last_became_peered":"2024-04-21T08:48:45.549585+0000","last_unstale":"2024-04-21T08:51:10.939650+0000","last_undegraded":"2024-04-21T08:51:10.939650+0000","last_fullsized":"2024-04-21T08:51:10.939650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:58.287402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063099+0000","last_change":"2024-04-21T08:48:46.462746+0000","last_active":"2024-04-21T08:51:11.063099+0000","last_peered":"2024-04-21T08:51:11.063099+0000","last_clean":"2024-04-21T08:51:11.063099+0000","last_became_active":"2024-04-21T08:48:46.461602+0000","last_became_peered":"2024-04-21T08:48:46.461602+0000","last_unstale":"2024-04-21T08:51:11.063099+0000","last_undegraded":"2024-04-21T08:51:11.063099+0000","last_fullsized":"2024-04-21T08:51:11.063099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:52.198634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939308+0000","last_change":"2024-04-21T08:48:50.770770+0000","last_active":"2024-04-21T08:51:10.939308+0000","last_peered":"2024-04-21T08:51:10.939308+0000","last_clean":"2024-04-21T08:51:10.939308+0000","last_became_active":"2024-04-21T08:48:50.770326+0000","last_became_peered":"2024-04-21T08:48:50.770326+0000","last_unstale":"2024-04-21T08:51:10.939308+0000","last_undegraded":"2024-04-21T08:51:10.939308+0000","last_fullsized":"2024-04-21T08:51:10.939308+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:38.325155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936408+0000","last_change":"2024-04-21T08:48:45.558867+0000","last_active":"2024-04-21T08:51:10.936408+0000","last_peered":"2024-04-21T08:51:10.936408+0000","last_clean":"2024-04-21T08:51:10.936408+0000","last_became_active":"2024-04-21T08:48:45.558492+0000","last_became_peered":"2024-04-21T08:48:45.558492+0000","last_unstale":"2024-04-21T08:51:10.936408+0000","last_undegraded":"2024-04-21T08:51:10.936408+0000","last_fullsized":"2024-04-21T08:51:10.936408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:44:55.156404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.830180+0000","last_change":"2024-04-21T08:48:46.466461+0000","last_active":"2024-04-21T08:51:10.830180+0000","last_peered":"2024-04-21T08:51:10.830180+0000","last_clean":"2024-04-21T08:51:10.830180+0000","last_became_active":"2024-04-21T08:48:46.466035+0000","last_became_peered":"2024-04-21T08:48:46.466035+0000","last_unstale":"2024-04-21T08:51:10.830180+0000","last_undegraded":"2024-04-21T08:51:10.830180+0000","last_fullsized":"2024-04-21T08:51:10.830180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:47:32.307753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036555+0000","last_change":"2024-04-21T08:48:50.767655+0000","last_active":"2024-04-21T08:51:11.036555+0000","last_peered":"2024-04-21T08:51:11.036555+0000","last_clean":"2024-04-21T08:51:11.036555+0000","last_became_active":"2024-04-21T08:48:50.767305+0000","last_became_peered":"2024-04-21T08:48:50.767305+0000","last_unstale":"2024-04-21T08:51:11.036555+0000","last_undegraded":"2024-04-21T08:51:11.036555+0000","last_fullsized":"2024-04-21T08:51:11.036555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:31.210126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062757+0000","last_change":"2024-04-21T08:48:45.552952+0000","last_active":"2024-04-21T08:51:11.062757+0000","last_peered":"2024-04-21T08:51:11.062757+0000","last_clean":"2024-04-21T08:51:11.062757+0000","last_became_active":"2024-04-21T08:48:45.552556+0000","last_became_peered":"2024-04-21T08:48:45.552556+0000","last_unstale":"2024-04-21T08:51:11.062757+0000","last_undegraded":"2024-04-21T08:51:11.062757+0000","last_fullsized":"2024-04-21T08:51:11.062757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:03.428801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829859+0000","last_change":"2024-04-21T08:48:46.469020+0000","last_active":"2024-04-21T08:51:10.829859+0000","last_peered":"2024-04-21T08:51:10.829859+0000","last_clean":"2024-04-21T08:51:10.829859+0000","last_became_active":"2024-04-21T08:48:46.468631+0000","last_became_peered":"2024-04-21T08:48:46.468631+0000","last_unstale":"2024-04-21T08:51:10.829859+0000","last_undegraded":"2024-04-21T08:51:10.829859+0000","last_fullsized":"2024-04-21T08:51:10.829859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:08.559516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036969+0000","last_change":"2024-04-21T08:48:50.773942+0000","last_active":"2024-04-21T08:51:11.036969+0000","last_peered":"2024-04-21T08:51:11.036969+0000","last_clean":"2024-04-21T08:51:11.036969+0000","last_became_active":"2024-04-21T08:48:50.771876+0000","last_became_peered":"2024-04-21T08:48:50.771876+0000","last_unstale":"2024-04-21T08:51:11.036969+0000","last_undegraded":"2024-04-21T08:51:11.036969+0000","last_fullsized":"2024-04-21T08:51:11.036969+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:19.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945411+0000","last_change":"2024-04-21T08:48:46.462727+0000","last_active":"2024-04-21T08:51:10.945411+0000","last_peered":"2024-04-21T08:51:10.945411+0000","last_clean":"2024-04-21T08:51:10.945411+0000","last_became_active":"2024-04-21T08:48:46.461669+0000","last_became_peered":"2024-04-21T08:48:46.461669+0000","last_unstale":"2024-04-21T08:51:10.945411+0000","last_undegraded":"2024-04-21T08:51:10.945411+0000","last_fullsized":"2024-04-21T08:51:10.945411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:27.290790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":159,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036108+0000","last_change":"2024-04-21T08:48:45.558541+0000","last_active":"2024-04-21T08:51:11.036108+0000","last_peered":"2024-04-21T08:51:11.036108+0000","last_clean":"2024-04-21T08:51:11.036108+0000","last_became_active":"2024-04-21T08:48:45.558186+0000","last_became_peered":"2024-04-21T08:48:45.558186+0000","last_unstale":"2024-04-21T08:51:11.036108+0000","last_undegraded":"2024-04-21T08:51:11.036108+0000","last_fullsized":"2024-04-21T08:51:11.036108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:02.919144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062978+0000","last_change":"2024-04-21T08:48:50.770838+0000","last_active":"2024-04-21T08:51:11.062978+0000","last_peered":"2024-04-21T08:51:11.062978+0000","last_clean":"2024-04-21T08:51:11.062978+0000","last_became_active":"2024-04-21T08:48:50.770447+0000","last_became_peered":"2024-04-21T08:48:50.770447+0000","last_unstale":"2024-04-21T08:51:11.062978+0000","last_undegraded":"2024-04-21T08:51:11.062978+0000","last_fullsized":"2024-04-21T08:51:11.062978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:48:21.010284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828090+0000","last_change":"2024-04-21T08:48:46.478145+0000","last_active":"2024-04-21T08:51:10.828090+0000","last_peered":"2024-04-21T08:51:10.828090+0000","last_clean":"2024-04-21T08:51:10.828090+0000","last_became_active":"2024-04-21T08:48:46.477813+0000","last_became_peered":"2024-04-21T08:48:46.477813+0000","last_unstale":"2024-04-21T08:51:10.828090+0000","last_undegraded":"2024-04-21T08:51:10.828090+0000","last_fullsized":"2024-04-21T08:51:10.828090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:15.917619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038191+0000","last_change":"2024-04-21T08:48:45.545269+0000","last_active":"2024-04-21T08:51:11.038191+0000","last_peered":"2024-04-21T08:51:11.038191+0000","last_clean":"2024-04-21T08:51:11.038191+0000","last_became_active":"2024-04-21T08:48:45.544887+0000","last_became_peered":"2024-04-21T08:48:45.544887+0000","last_unstale":"2024-04-21T08:51:11.038191+0000","last_undegraded":"2024-04-21T08:51:11.038191+0000","last_fullsized":"2024-04-21T08:51:11.038191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:27.222570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":367,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062629+0000","last_change":"2024-04-21T08:47:34.198161+0000","last_active":"2024-04-21T08:51:11.062629+0000","last_peered":"2024-04-21T08:51:11.062629+0000","last_clean":"2024-04-21T08:51:11.062629+0000","last_became_active":"2024-04-21T08:47:34.197654+0000","last_became_peered":"2024-04-21T08:47:34.197654+0000","last_unstale":"2024-04-21T08:51:11.062629+0000","last_undegraded":"2024-04-21T08:51:11.062629+0000","last_fullsized":"2024-04-21T08:51:11.062629+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_clean_scrub_stamp":"2024-04-21T08:44:05.493464+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:58:58.295783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945722+0000","last_change":"2024-04-21T08:48:50.770826+0000","last_active":"2024-04-21T08:51:10.945722+0000","last_peered":"2024-04-21T08:51:10.945722+0000","last_clean":"2024-04-21T08:51:10.945722+0000","last_became_active":"2024-04-21T08:48:50.770372+0000","last_became_peered":"2024-04-21T08:48:50.770372+0000","last_unstale":"2024-04-21T08:51:10.945722+0000","last_undegraded":"2024-04-21T08:51:10.945722+0000","last_fullsized":"2024-04-21T08:51:10.945722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:59.637874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.035696+0000","last_change":"2024-04-21T08:48:46.459330+0000","last_active":"2024-04-21T08:51:11.035696+0000","last_peered":"2024-04-21T08:51:11.035696+0000","last_clean":"2024-04-21T08:51:11.035696+0000","last_became_active":"2024-04-21T08:48:46.458963+0000","last_became_peered":"2024-04-21T08:48:46.458963+0000","last_unstale":"2024-04-21T08:51:11.035696+0000","last_undegraded":"2024-04-21T08:51:11.035696+0000","last_fullsized":"2024-04-21T08:51:11.035696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:10.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062640+0000","last_change":"2024-04-21T08:49:36.917854+0000","last_active":"2024-04-21T08:51:11.062640+0000","last_peered":"2024-04-21T08:51:11.062640+0000","last_clean":"2024-04-21T08:51:11.062640+0000","last_became_active":"2024-04-21T08:49:36.915390+0000","last_became_peered":"2024-04-21T08:49:36.915390+0000","last_unstale":"2024-04-21T08:51:11.062640+0000","last_undegraded":"2024-04-21T08:51:11.062640+0000","last_fullsized":"2024-04-21T08:51:11.062640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:45.920752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062798+0000","last_change":"2024-04-21T08:48:50.765339+0000","last_active":"2024-04-21T08:51:11.062798+0000","last_peered":"2024-04-21T08:51:11.062798+0000","last_clean":"2024-04-21T08:51:11.062798+0000","last_became_active":"2024-04-21T08:48:50.764858+0000","last_became_peered":"2024-04-21T08:48:50.764858+0000","last_unstale":"2024-04-21T08:51:11.062798+0000","last_undegraded":"2024-04-21T08:51:11.062798+0000","last_fullsized":"2024-04-21T08:51:11.062798+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:26.075041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939660+0000","last_change":"2024-04-21T08:48:45.557768+0000","last_active":"2024-04-21T08:51:10.939660+0000","last_peered":"2024-04-21T08:51:10.939660+0000","last_clean":"2024-04-21T08:51:10.939660+0000","last_became_active":"2024-04-21T08:48:45.557448+0000","last_became_peered":"2024-04-21T08:48:45.557448+0000","last_unstale":"2024-04-21T08:51:10.939660+0000","last_undegraded":"2024-04-21T08:51:10.939660+0000","last_fullsized":"2024-04-21T08:51:10.939660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:09.063913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036820+0000","last_change":"2024-04-21T08:48:46.459425+0000","last_active":"2024-04-21T08:51:11.036820+0000","last_peered":"2024-04-21T08:51:11.036820+0000","last_clean":"2024-04-21T08:51:11.036820+0000","last_became_active":"2024-04-21T08:48:46.459020+0000","last_became_peered":"2024-04-21T08:48:46.459020+0000","last_unstale":"2024-04-21T08:51:11.036820+0000","last_undegraded":"2024-04-21T08:51:11.036820+0000","last_fullsized":"2024-04-21T08:51:11.036820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:08:39.042825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828005+0000","last_change":"2024-04-21T08:48:50.765740+0000","last_active":"2024-04-21T08:51:10.828005+0000","last_peered":"2024-04-21T08:51:10.828005+0000","last_clean":"2024-04-21T08:51:10.828005+0000","last_became_active":"2024-04-21T08:48:50.765359+0000","last_became_peered":"2024-04-21T08:48:50.765359+0000","last_unstale":"2024-04-21T08:51:10.828005+0000","last_undegraded":"2024-04-21T08:51:10.828005+0000","last_fullsized":"2024-04-21T08:51:10.828005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:34:42.716165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'44","reported_seq":101,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.054709+0000","last_change":"2024-04-21T08:48:45.564532+0000","last_active":"2024-04-21T08:51:11.054709+0000","last_peered":"2024-04-21T08:51:11.054709+0000","last_clean":"2024-04-21T08:51:11.054709+0000","last_became_active":"2024-04-21T08:48:45.564225+0000","last_became_peered":"2024-04-21T08:48:45.564225+0000","last_unstale":"2024-04-21T08:51:11.054709+0000","last_undegraded":"2024-04-21T08:51:11.054709+0000","last_fullsized":"2024-04-21T08:51:11.054709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:42.850371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828519+0000","last_change":"2024-04-21T08:48:46.466622+0000","last_active":"2024-04-21T08:51:10.828519+0000","last_peered":"2024-04-21T08:51:10.828519+0000","last_clean":"2024-04-21T08:51:10.828519+0000","last_became_active":"2024-04-21T08:48:46.466129+0000","last_became_peered":"2024-04-21T08:48:46.466129+0000","last_unstale":"2024-04-21T08:51:10.828519+0000","last_undegraded":"2024-04-21T08:51:10.828519+0000","last_fullsized":"2024-04-21T08:51:10.828519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:20:26.064193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832121+0000","last_change":"2024-04-21T08:48:50.759534+0000","last_active":"2024-04-21T08:51:10.832121+0000","last_peered":"2024-04-21T08:51:10.832121+0000","last_clean":"2024-04-21T08:51:10.832121+0000","last_became_active":"2024-04-21T08:48:50.759116+0000","last_became_peered":"2024-04-21T08:48:50.759116+0000","last_unstale":"2024-04-21T08:51:10.832121+0000","last_undegraded":"2024-04-21T08:51:10.832121+0000","last_fullsized":"2024-04-21T08:51:10.832121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:02:16.200901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038642+0000","last_change":"2024-04-21T08:48:45.545773+0000","last_active":"2024-04-21T08:51:11.038642+0000","last_peered":"2024-04-21T08:51:11.038642+0000","last_clean":"2024-04-21T08:51:11.038642+0000","last_became_active":"2024-04-21T08:48:45.545447+0000","last_became_peered":"2024-04-21T08:48:45.545447+0000","last_unstale":"2024-04-21T08:51:11.038642+0000","last_undegraded":"2024-04-21T08:51:11.038642+0000","last_fullsized":"2024-04-21T08:51:11.038642+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:59:42.171602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877976+0000","last_change":"2024-04-21T08:49:36.964411+0000","last_active":"2024-04-21T08:51:10.877976+0000","last_peered":"2024-04-21T08:51:10.877976+0000","last_clean":"2024-04-21T08:51:10.877976+0000","last_became_active":"2024-04-21T08:49:36.964046+0000","last_became_peered":"2024-04-21T08:49:36.964046+0000","last_unstale":"2024-04-21T08:51:10.877976+0000","last_undegraded":"2024-04-21T08:51:10.877976+0000","last_fullsized":"2024-04-21T08:51:10.877976+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:29:37.223495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038538+0000","last_change":"2024-04-21T08:50:37.351641+0000","last_active":"2024-04-21T08:51:11.038538+0000","last_peered":"2024-04-21T08:51:11.038538+0000","last_clean":"2024-04-21T08:51:11.038538+0000","last_became_active":"2024-04-21T08:50:37.351130+0000","last_became_peered":"2024-04-21T08:50:37.351130+0000","last_unstale":"2024-04-21T08:51:11.038538+0000","last_undegraded":"2024-04-21T08:51:11.038538+0000","last_fullsized":"2024-04-21T08:51:11.038538+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:56:58.989691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006155+0000","last_change":"2024-04-21T08:49:36.928993+0000","last_active":"2024-04-21T08:51:11.006155+0000","last_peered":"2024-04-21T08:51:11.006155+0000","last_clean":"2024-04-21T08:51:11.006155+0000","last_became_active":"2024-04-21T08:49:36.911247+0000","last_became_peered":"2024-04-21T08:49:36.911247+0000","last_unstale":"2024-04-21T08:51:11.006155+0000","last_undegraded":"2024-04-21T08:51:11.006155+0000","last_fullsized":"2024-04-21T08:51:11.006155+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:13.481670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939224+0000","last_change":"2024-04-21T08:48:46.465475+0000","last_active":"2024-04-21T08:51:10.939224+0000","last_peered":"2024-04-21T08:51:10.939224+0000","last_clean":"2024-04-21T08:51:10.939224+0000","last_became_active":"2024-04-21T08:48:46.465138+0000","last_became_peered":"2024-04-21T08:48:46.465138+0000","last_unstale":"2024-04-21T08:51:10.939224+0000","last_undegraded":"2024-04-21T08:51:10.939224+0000","last_fullsized":"2024-04-21T08:51:10.939224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:12:53.521547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936707+0000","last_change":"2024-04-21T08:48:50.765765+0000","last_active":"2024-04-21T08:51:10.936707+0000","last_peered":"2024-04-21T08:51:10.936707+0000","last_clean":"2024-04-21T08:51:10.936707+0000","last_became_active":"2024-04-21T08:48:50.765324+0000","last_became_peered":"2024-04-21T08:48:50.765324+0000","last_unstale":"2024-04-21T08:51:10.936707+0000","last_undegraded":"2024-04-21T08:51:10.936707+0000","last_fullsized":"2024-04-21T08:51:10.936707+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:41:37.846768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828042+0000","last_change":"2024-04-21T08:48:45.557400+0000","last_active":"2024-04-21T08:51:10.828042+0000","last_peered":"2024-04-21T08:51:10.828042+0000","last_clean":"2024-04-21T08:51:10.828042+0000","last_became_active":"2024-04-21T08:48:45.555012+0000","last_became_peered":"2024-04-21T08:48:45.555012+0000","last_unstale":"2024-04-21T08:51:10.828042+0000","last_undegraded":"2024-04-21T08:51:10.828042+0000","last_fullsized":"2024-04-21T08:51:10.828042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:00:31.254353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062791+0000","last_change":"2024-04-21T08:48:46.470042+0000","last_active":"2024-04-21T08:51:11.062791+0000","last_peered":"2024-04-21T08:51:11.062791+0000","last_clean":"2024-04-21T08:51:11.062791+0000","last_became_active":"2024-04-21T08:48:46.469716+0000","last_became_peered":"2024-04-21T08:48:46.469716+0000","last_unstale":"2024-04-21T08:51:11.062791+0000","last_undegraded":"2024-04-21T08:51:11.062791+0000","last_fullsized":"2024-04-21T08:51:11.062791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:09.934751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038592+0000","last_change":"2024-04-21T08:48:50.770889+0000","last_active":"2024-04-21T08:51:11.038592+0000","last_peered":"2024-04-21T08:51:11.038592+0000","last_clean":"2024-04-21T08:51:11.038592+0000","last_became_active":"2024-04-21T08:48:50.769980+0000","last_became_peered":"2024-04-21T08:48:50.769980+0000","last_unstale":"2024-04-21T08:51:11.038592+0000","last_undegraded":"2024-04-21T08:51:11.038592+0000","last_fullsized":"2024-04-21T08:51:11.038592+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:44.631633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832478+0000","last_change":"2024-04-21T08:48:45.555479+0000","last_active":"2024-04-21T08:51:10.832478+0000","last_peered":"2024-04-21T08:51:10.832478+0000","last_clean":"2024-04-21T08:51:10.832478+0000","last_became_active":"2024-04-21T08:48:45.555162+0000","last_became_peered":"2024-04-21T08:48:45.555162+0000","last_unstale":"2024-04-21T08:51:10.832478+0000","last_undegraded":"2024-04-21T08:51:10.832478+0000","last_fullsized":"2024-04-21T08:51:10.832478+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:15:20.262798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828417+0000","last_change":"2024-04-21T08:48:46.467962+0000","last_active":"2024-04-21T08:51:10.828417+0000","last_peered":"2024-04-21T08:51:10.828417+0000","last_clean":"2024-04-21T08:51:10.828417+0000","last_became_active":"2024-04-21T08:48:46.467561+0000","last_became_peered":"2024-04-21T08:48:46.467561+0000","last_unstale":"2024-04-21T08:51:10.828417+0000","last_undegraded":"2024-04-21T08:51:10.828417+0000","last_fullsized":"2024-04-21T08:51:10.828417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:18.262009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038487+0000","last_change":"2024-04-21T08:48:50.767678+0000","last_active":"2024-04-21T08:51:11.038487+0000","last_peered":"2024-04-21T08:51:11.038487+0000","last_clean":"2024-04-21T08:51:11.038487+0000","last_became_active":"2024-04-21T08:48:50.767309+0000","last_became_peered":"2024-04-21T08:48:50.767309+0000","last_unstale":"2024-04-21T08:51:11.038487+0000","last_undegraded":"2024-04-21T08:51:11.038487+0000","last_fullsized":"2024-04-21T08:51:11.038487+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:16:40.065043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939597+0000","last_change":"2024-04-21T08:48:45.553321+0000","last_active":"2024-04-21T08:51:10.939597+0000","last_peered":"2024-04-21T08:51:10.939597+0000","last_clean":"2024-04-21T08:51:10.939597+0000","last_became_active":"2024-04-21T08:48:45.552871+0000","last_became_peered":"2024-04-21T08:48:45.552871+0000","last_unstale":"2024-04-21T08:51:10.939597+0000","last_undegraded":"2024-04-21T08:51:10.939597+0000","last_fullsized":"2024-04-21T08:51:10.939597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:16:27.339801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062861+0000","last_change":"2024-04-21T08:48:46.473453+0000","last_active":"2024-04-21T08:51:11.062861+0000","last_peered":"2024-04-21T08:51:11.062861+0000","last_clean":"2024-04-21T08:51:11.062861+0000","last_became_active":"2024-04-21T08:48:46.472613+0000","last_became_peered":"2024-04-21T08:48:46.472613+0000","last_unstale":"2024-04-21T08:51:11.062861+0000","last_undegraded":"2024-04-21T08:51:11.062861+0000","last_fullsized":"2024-04-21T08:51:11.062861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:43:58.995997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006138+0000","last_change":"2024-04-21T08:48:50.765282+0000","last_active":"2024-04-21T08:51:11.006138+0000","last_peered":"2024-04-21T08:51:11.006138+0000","last_clean":"2024-04-21T08:51:11.006138+0000","last_became_active":"2024-04-21T08:48:50.764891+0000","last_became_peered":"2024-04-21T08:48:50.764891+0000","last_unstale":"2024-04-21T08:51:11.006138+0000","last_undegraded":"2024-04-21T08:51:11.006138+0000","last_fullsized":"2024-04-21T08:51:11.006138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:18:40.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038155+0000","last_change":"2024-04-21T08:49:36.926642+0000","last_active":"2024-04-21T08:51:11.038155+0000","last_peered":"2024-04-21T08:51:11.038155+0000","last_clean":"2024-04-21T08:51:11.038155+0000","last_became_active":"2024-04-21T08:49:36.910453+0000","last_became_peered":"2024-04-21T08:49:36.910453+0000","last_unstale":"2024-04-21T08:51:11.038155+0000","last_undegraded":"2024-04-21T08:51:11.038155+0000","last_fullsized":"2024-04-21T08:51:11.038155+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:19.290010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063219+0000","last_change":"2024-04-21T08:48:46.470058+0000","last_active":"2024-04-21T08:51:11.063219+0000","last_peered":"2024-04-21T08:51:11.063219+0000","last_clean":"2024-04-21T08:51:11.063219+0000","last_became_active":"2024-04-21T08:48:46.469485+0000","last_became_peered":"2024-04-21T08:48:46.469485+0000","last_unstale":"2024-04-21T08:51:11.063219+0000","last_undegraded":"2024-04-21T08:51:11.063219+0000","last_fullsized":"2024-04-21T08:51:11.063219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:33:13.212087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877446+0000","last_change":"2024-04-21T08:48:50.762944+0000","last_active":"2024-04-21T08:51:10.877446+0000","last_peered":"2024-04-21T08:51:10.877446+0000","last_clean":"2024-04-21T08:51:10.877446+0000","last_became_active":"2024-04-21T08:48:50.762508+0000","last_became_peered":"2024-04-21T08:48:50.762508+0000","last_unstale":"2024-04-21T08:51:10.877446+0000","last_undegraded":"2024-04-21T08:51:10.877446+0000","last_fullsized":"2024-04-21T08:51:10.877446+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:35.393339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.052626+0000","last_change":"2024-04-21T08:48:45.560361+0000","last_active":"2024-04-21T08:51:11.052626+0000","last_peered":"2024-04-21T08:51:11.052626+0000","last_clean":"2024-04-21T08:51:11.052626+0000","last_became_active":"2024-04-21T08:48:45.559946+0000","last_became_peered":"2024-04-21T08:48:45.559946+0000","last_unstale":"2024-04-21T08:51:11.052626+0000","last_undegraded":"2024-04-21T08:51:11.052626+0000","last_fullsized":"2024-04-21T08:51:11.052626+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:55.599714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063003+0000","last_change":"2024-04-21T08:48:46.472706+0000","last_active":"2024-04-21T08:51:11.063003+0000","last_peered":"2024-04-21T08:51:11.063003+0000","last_clean":"2024-04-21T08:51:11.063003+0000","last_became_active":"2024-04-21T08:48:46.472378+0000","last_became_peered":"2024-04-21T08:48:46.472378+0000","last_unstale":"2024-04-21T08:51:11.063003+0000","last_undegraded":"2024-04-21T08:51:11.063003+0000","last_fullsized":"2024-04-21T08:51:11.063003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:58.228899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945191+0000","last_change":"2024-04-21T08:48:50.760610+0000","last_active":"2024-04-21T08:51:10.945191+0000","last_peered":"2024-04-21T08:51:10.945191+0000","last_clean":"2024-04-21T08:51:10.945191+0000","last_became_active":"2024-04-21T08:48:50.759991+0000","last_became_peered":"2024-04-21T08:48:50.759991+0000","last_unstale":"2024-04-21T08:51:10.945191+0000","last_undegraded":"2024-04-21T08:51:10.945191+0000","last_fullsized":"2024-04-21T08:51:10.945191+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:43.649701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036942+0000","last_change":"2024-04-21T08:48:45.560773+0000","last_active":"2024-04-21T08:51:11.036942+0000","last_peered":"2024-04-21T08:51:11.036942+0000","last_clean":"2024-04-21T08:51:11.036942+0000","last_became_active":"2024-04-21T08:48:45.560431+0000","last_became_peered":"2024-04-21T08:48:45.560431+0000","last_unstale":"2024-04-21T08:51:11.036942+0000","last_undegraded":"2024-04-21T08:51:11.036942+0000","last_fullsized":"2024-04-21T08:51:11.036942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:07.858838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063259+0000","last_change":"2024-04-21T08:48:46.475983+0000","last_active":"2024-04-21T08:51:11.063259+0000","last_peered":"2024-04-21T08:51:11.063259+0000","last_clean":"2024-04-21T08:51:11.063259+0000","last_became_active":"2024-04-21T08:48:46.475667+0000","last_became_peered":"2024-04-21T08:48:46.475667+0000","last_unstale":"2024-04-21T08:51:11.063259+0000","last_undegraded":"2024-04-21T08:51:11.063259+0000","last_fullsized":"2024-04-21T08:51:11.063259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:28:41.013111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062960+0000","last_change":"2024-04-21T08:48:46.464675+0000","last_active":"2024-04-21T08:51:11.062960+0000","last_peered":"2024-04-21T08:51:11.062960+0000","last_clean":"2024-04-21T08:51:11.062960+0000","last_became_active":"2024-04-21T08:48:46.464314+0000","last_became_peered":"2024-04-21T08:48:46.464314+0000","last_unstale":"2024-04-21T08:51:11.062960+0000","last_undegraded":"2024-04-21T08:51:11.062960+0000","last_fullsized":"2024-04-21T08:51:11.062960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:53.048400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832307+0000","last_change":"2024-04-21T08:49:36.911025+0000","last_active":"2024-04-21T08:51:10.832307+0000","last_peered":"2024-04-21T08:51:10.832307+0000","last_clean":"2024-04-21T08:51:10.832307+0000","last_became_active":"2024-04-21T08:49:36.909942+0000","last_became_peered":"2024-04-21T08:49:36.909942+0000","last_unstale":"2024-04-21T08:51:10.832307+0000","last_undegraded":"2024-04-21T08:51:10.832307+0000","last_fullsized":"2024-04-21T08:51:10.832307+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:39.864253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832375+0000","last_change":"2024-04-21T08:48:46.467867+0000","last_active":"2024-04-21T08:51:10.832375+0000","last_peered":"2024-04-21T08:51:10.832375+0000","last_clean":"2024-04-21T08:51:10.832375+0000","last_became_active":"2024-04-21T08:48:46.465917+0000","last_became_peered":"2024-04-21T08:48:46.465917+0000","last_unstale":"2024-04-21T08:51:10.832375+0000","last_undegraded":"2024-04-21T08:51:10.832375+0000","last_fullsized":"2024-04-21T08:51:10.832375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:19:56.824998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877556+0000","last_change":"2024-04-21T08:48:46.467159+0000","last_active":"2024-04-21T08:51:10.877556+0000","last_peered":"2024-04-21T08:51:10.877556+0000","last_clean":"2024-04-21T08:51:10.877556+0000","last_became_active":"2024-04-21T08:48:46.466639+0000","last_became_peered":"2024-04-21T08:48:46.466639+0000","last_unstale":"2024-04-21T08:51:10.877556+0000","last_undegraded":"2024-04-21T08:51:10.877556+0000","last_fullsized":"2024-04-21T08:51:10.877556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:54:44.772535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036982+0000","last_change":"2024-04-21T08:48:46.466403+0000","last_active":"2024-04-21T08:51:11.036982+0000","last_peered":"2024-04-21T08:51:11.036982+0000","last_clean":"2024-04-21T08:51:11.036982+0000","last_became_active":"2024-04-21T08:48:46.466060+0000","last_became_peered":"2024-04-21T08:48:46.466060+0000","last_unstale":"2024-04-21T08:51:11.036982+0000","last_undegraded":"2024-04-21T08:51:11.036982+0000","last_fullsized":"2024-04-21T08:51:11.036982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:14.374193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063096+0000","last_change":"2024-04-21T08:49:36.909134+0000","last_active":"2024-04-21T08:51:11.063096+0000","last_peered":"2024-04-21T08:51:11.063096+0000","last_clean":"2024-04-21T08:51:11.063096+0000","last_became_active":"2024-04-21T08:49:36.908723+0000","last_became_peered":"2024-04-21T08:49:36.908723+0000","last_unstale":"2024-04-21T08:51:11.063096+0000","last_undegraded":"2024-04-21T08:51:11.063096+0000","last_fullsized":"2024-04-21T08:51:11.063096+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:28.123016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832933+0000","last_change":"2024-04-21T08:48:46.464098+0000","last_active":"2024-04-21T08:51:10.832933+0000","last_peered":"2024-04-21T08:51:10.832933+0000","last_clean":"2024-04-21T08:51:10.832933+0000","last_became_active":"2024-04-21T08:48:46.463756+0000","last_became_peered":"2024-04-21T08:48:46.463756+0000","last_unstale":"2024-04-21T08:51:10.832933+0000","last_undegraded":"2024-04-21T08:51:10.832933+0000","last_fullsized":"2024-04-21T08:51:10.832933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:31.725155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828666+0000","last_change":"2024-04-21T08:48:46.467372+0000","last_active":"2024-04-21T08:51:10.828666+0000","last_peered":"2024-04-21T08:51:10.828666+0000","last_clean":"2024-04-21T08:51:10.828666+0000","last_became_active":"2024-04-21T08:48:46.467043+0000","last_became_peered":"2024-04-21T08:48:46.467043+0000","last_unstale":"2024-04-21T08:51:10.828666+0000","last_undegraded":"2024-04-21T08:51:10.828666+0000","last_fullsized":"2024-04-21T08:51:10.828666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:41.889994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945316+0000","last_change":"2024-04-21T08:48:46.460604+0000","last_active":"2024-04-21T08:51:10.945316+0000","last_peered":"2024-04-21T08:51:10.945316+0000","last_clean":"2024-04-21T08:51:10.945316+0000","last_became_active":"2024-04-21T08:48:46.460276+0000","last_became_peered":"2024-04-21T08:48:46.460276+0000","last_unstale":"2024-04-21T08:51:10.945316+0000","last_undegraded":"2024-04-21T08:51:10.945316+0000","last_fullsized":"2024-04-21T08:51:10.945316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:30.194808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828076+0000","last_change":"2024-04-21T08:48:46.463665+0000","last_active":"2024-04-21T08:51:10.828076+0000","last_peered":"2024-04-21T08:51:10.828076+0000","last_clean":"2024-04-21T08:51:10.828076+0000","last_became_active":"2024-04-21T08:48:46.463309+0000","last_became_peered":"2024-04-21T08:48:46.463309+0000","last_unstale":"2024-04-21T08:51:10.828076+0000","last_undegraded":"2024-04-21T08:51:10.828076+0000","last_fullsized":"2024-04-21T08:51:10.828076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:12:36.436565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006407+0000","last_change":"2024-04-21T08:48:46.476845+0000","last_active":"2024-04-21T08:51:11.006407+0000","last_peered":"2024-04-21T08:51:11.006407+0000","last_clean":"2024-04-21T08:51:11.006407+0000","last_became_active":"2024-04-21T08:48:46.476507+0000","last_became_peered":"2024-04-21T08:48:46.476507+0000","last_unstale":"2024-04-21T08:51:11.006407+0000","last_undegraded":"2024-04-21T08:51:11.006407+0000","last_fullsized":"2024-04-21T08:51:11.006407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:47.424576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036615+0000","last_change":"2024-04-21T08:48:46.469316+0000","last_active":"2024-04-21T08:51:11.036615+0000","last_peered":"2024-04-21T08:51:11.036615+0000","last_clean":"2024-04-21T08:51:11.036615+0000","last_became_active":"2024-04-21T08:48:46.468852+0000","last_became_peered":"2024-04-21T08:48:46.468852+0000","last_unstale":"2024-04-21T08:51:11.036615+0000","last_undegraded":"2024-04-21T08:51:11.036615+0000","last_fullsized":"2024-04-21T08:51:11.036615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:05:03.664279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062922+0000","last_change":"2024-04-21T08:48:46.466703+0000","last_active":"2024-04-21T08:51:11.062922+0000","last_peered":"2024-04-21T08:51:11.062922+0000","last_clean":"2024-04-21T08:51:11.062922+0000","last_became_active":"2024-04-21T08:48:46.466395+0000","last_became_peered":"2024-04-21T08:48:46.466395+0000","last_unstale":"2024-04-21T08:51:11.062922+0000","last_undegraded":"2024-04-21T08:51:11.062922+0000","last_fullsized":"2024-04-21T08:51:11.062922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:08:56.956283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108027,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599448,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":835,"num_read_kb":523,"num_write":290,"num_write_kb":1443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1798344,"data_compressed":80739,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":529689,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32340,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712812,"statfs":{"total":95995035648,"available":95961919488,"internally_reserved":0,"allocated":1187840,"data_stored":1247346,"data_compressed":70096,"data_compressed_allocated":94208,"data_compressed_original":741376,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":927125,"data_compressed":48725,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":529621,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1097728,"data_stored":1127264,"data_compressed":65588,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1015808,"data_stored":515086,"data_compressed":26578,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":593908,"data_compressed":31190,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":526377,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1118208,"data_stored":595766,"data_compressed":31190,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1049999,"data_compressed":54881,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":531144,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32276,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712876,"statfs":{"total":95995035648,"available":95961985024,"internally_reserved":0,"allocated":1126400,"data_stored":1253529,"data_compressed":71744,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":69009,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":70867,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134166,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":393842,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":203110,"data_compressed":7767,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:51:15.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:51:16.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: Updating smithi028:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:51:16.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: Updating smithi152:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:51:16.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: Updating smithi171:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:51:16.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:51:16.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:51:16.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:51:16.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:51:16.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3848978672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:16.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: Updating smithi152:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:51:16.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: Updating smithi028:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:51:16.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: Updating smithi171:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:51:16.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1588229387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:16.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-21T08:51:16.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:51:16.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:51:16.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:51:16.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.106 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:15 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/341422188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:16.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.107+0000 7fea7fb84640 1 -- 172.21.15.28:0/33158195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea780690e0 msgr2=0x7fea78069540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:16.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.107+0000 7fea7fb84640 1 --2- 172.21.15.28:0/33158195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea780690e0 0x7fea78069540 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fea68009900 tx=0x7fea6802bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:16.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.107+0000 7fea7fb84640 1 -- 172.21.15.28:0/33158195 shutdown_connections 2024-04-21T08:51:16.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.107+0000 7fea7fb84640 1 --2- 172.21.15.28:0/33158195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea78069a80 0x7fea7810eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.107+0000 7fea7fb84640 1 --2- 172.21.15.28:0/33158195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea780690e0 0x7fea78069540 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.107+0000 7fea7fb84640 1 --2- 172.21.15.28:0/33158195 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea78107fe0 0x7fea781083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.107+0000 7fea7fb84640 1 -- 172.21.15.28:0/33158195 >> 172.21.15.28:0/33158195 conn(0x7fea78077e30 msgr2=0x7fea78078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:16.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.107+0000 7fea7fb84640 1 -- 172.21.15.28:0/33158195 shutdown_connections 2024-04-21T08:51:16.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.107+0000 7fea7fb84640 1 -- 172.21.15.28:0/33158195 wait complete. 2024-04-21T08:51:16.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.107+0000 7fea7fb84640 1 Processor -- start 2024-04-21T08:51:16.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.107+0000 7fea7fb84640 1 -- start start 2024-04-21T08:51:16.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.107+0000 7fea7fb84640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea780690e0 0x7fea781986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:16.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7fb84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea78069a80 0x7fea78198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:16.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7fb84640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea78107fe0 0x7fea7819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:16.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7fb84640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fea78112160 con 0x7fea78069a80 2024-04-21T08:51:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7fb84640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fea78111fe0 con 0x7fea78107fe0 2024-04-21T08:51:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7fb84640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fea781122e0 con 0x7fea780690e0 2024-04-21T08:51:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7d11f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea78069a80 0x7fea78198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7d11f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea78069a80 0x7fea78198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51956/0 (socket says 172.21.15.28:51956) 2024-04-21T08:51:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7d11f640 1 -- 172.21.15.28:0/2373334722 learned_addr learned my addr 172.21.15.28:0/2373334722 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7e121640 1 --2- 172.21.15.28:0/2373334722 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea78107fe0 0x7fea7819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7d920640 1 --2- 172.21.15.28:0/2373334722 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea780690e0 0x7fea781986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7e121640 1 -- 172.21.15.28:0/2373334722 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea780690e0 msgr2=0x7fea781986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7e121640 1 --2- 172.21.15.28:0/2373334722 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea780690e0 0x7fea781986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7e121640 1 -- 172.21.15.28:0/2373334722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea78069a80 msgr2=0x7fea78198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7e121640 1 --2- 172.21.15.28:0/2373334722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea78069a80 0x7fea78198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7e121640 1 -- 172.21.15.28:0/2373334722 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea7819d6d0 con 0x7fea78107fe0 2024-04-21T08:51:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7d11f640 1 --2- 172.21.15.28:0/2373334722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea78069a80 0x7fea78198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7e121640 1 --2- 172.21.15.28:0/2373334722 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea78107fe0 0x7fea7819cfa0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fea70009fd0 tx=0x7fea7000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:16.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea6effd640 1 -- 172.21.15.28:0/2373334722 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea7000ce20 con 0x7fea78107fe0 2024-04-21T08:51:16.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea7fb84640 1 -- 172.21.15.28:0/2373334722 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea7819d9c0 con 0x7fea78107fe0 2024-04-21T08:51:16.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea6effd640 1 -- 172.21.15.28:0/2373334722 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea70005020 con 0x7fea78107fe0 2024-04-21T08:51:16.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.108+0000 7fea6effd640 1 -- 172.21.15.28:0/2373334722 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70010690 con 0x7fea78107fe0 2024-04-21T08:51:16.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.109+0000 7fea7fb84640 1 -- 172.21.15.28:0/2373334722 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea781a5250 con 0x7fea78107fe0 2024-04-21T08:51:16.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.109+0000 7fea7fb84640 1 -- 172.21.15.28:0/2373334722 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea78068de0 con 0x7fea78107fe0 2024-04-21T08:51:16.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.111+0000 7fea6effd640 1 -- 172.21.15.28:0/2373334722 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fea70044020 con 0x7fea78107fe0 2024-04-21T08:51:16.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.111+0000 7fea6effd640 1 --2- 172.21.15.28:0/2373334722 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fea640a3e40 0x7fea640a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:16.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.111+0000 7fea7d920640 1 --2- 172.21.15.28:0/2373334722 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fea640a3e40 0x7fea640a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:16.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.111+0000 7fea6effd640 1 -- 172.21.15.28:0/2373334722 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fea700d17e0 con 0x7fea78107fe0 2024-04-21T08:51:16.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.112+0000 7fea7d920640 1 --2- 172.21.15.28:0/2373334722 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fea640a3e40 0x7fea640a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fea74005fd0 tx=0x7fea7400b390 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:16.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.115+0000 7fea6effd640 1 -- 172.21.15.28:0/2373334722 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea7009d890 con 0x7fea78107fe0 2024-04-21T08:51:16.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: Updating smithi028:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:51:16.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: Updating smithi152:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:51:16.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: Updating smithi171:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:51:16.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:51:16.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:51:16.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:51:16.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:51:16.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3848978672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:16.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: Updating smithi152:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:51:16.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: Updating smithi028:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:51:16.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: Updating smithi171:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:51:16.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1588229387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:16.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-21T08:51:16.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:51:16.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:51:16.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:51:16.121 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:51:16.122 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:15 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/341422188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:16.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.299+0000 7fea7fb84640 1 -- 172.21.15.28:0/2373334722 --> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fea78062390 con 0x7fea640a3e40 2024-04-21T08:51:16.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.308+0000 7fea6effd640 1 -- 172.21.15.28:0/2373334722 <== mgr.25040 v2:172.21.15.28:6800/4182301910 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339688 (secure 0 0 0) 0x7fea78062390 con 0x7fea640a3e40 2024-04-21T08:51:16.311 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-21T08:51:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.311+0000 7fea7fb84640 1 -- 172.21.15.28:0/2373334722 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fea640a3e40 msgr2=0x7fea640a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.311+0000 7fea7fb84640 1 --2- 172.21.15.28:0/2373334722 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fea640a3e40 0x7fea640a6300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fea74005fd0 tx=0x7fea7400b390 comp rx=0 tx=0).stop 2024-04-21T08:51:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.311+0000 7fea7fb84640 1 -- 172.21.15.28:0/2373334722 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea78107fe0 msgr2=0x7fea7819cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.311+0000 7fea7fb84640 1 --2- 172.21.15.28:0/2373334722 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea78107fe0 0x7fea7819cfa0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fea70009fd0 tx=0x7fea7000ef10 comp rx=0 tx=0).stop 2024-04-21T08:51:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.311+0000 7fea7fb84640 1 -- 172.21.15.28:0/2373334722 shutdown_connections 2024-04-21T08:51:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.311+0000 7fea7fb84640 1 --2- 172.21.15.28:0/2373334722 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fea640a3e40 0x7fea640a6300 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.311+0000 7fea7fb84640 1 --2- 172.21.15.28:0/2373334722 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea78107fe0 0x7fea7819cfa0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.311+0000 7fea7fb84640 1 --2- 172.21.15.28:0/2373334722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea78069a80 0x7fea78198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.311+0000 7fea7fb84640 1 --2- 172.21.15.28:0/2373334722 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea780690e0 0x7fea781986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.311+0000 7fea7fb84640 1 -- 172.21.15.28:0/2373334722 >> 172.21.15.28:0/2373334722 conn(0x7fea78077e30 msgr2=0x7fea780fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.311+0000 7fea7fb84640 1 -- 172.21.15.28:0/2373334722 shutdown_connections 2024-04-21T08:51:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.311+0000 7fea7fb84640 1 -- 172.21.15.28:0/2373334722 wait complete. 2024-04-21T08:51:16.322 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-21T08:51:14.842869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190050,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":971,"num_read_kb":642,"num_write":473,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181430,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370192,"kb_used_data":12680,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571632,"statfs":{"total":1151940427776,"available":1151561351168,"internally_reserved":0,"allocated":12984320,"data_stored":9426854,"data_compressed":518308,"data_compressed_allocated":704512,"data_compressed_original":3862528,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.990882"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063213+0000","last_change":"2024-04-21T08:48:46.471626+0000","last_active":"2024-04-21T08:51:11.063213+0000","last_peered":"2024-04-21T08:51:11.063213+0000","last_clean":"2024-04-21T08:51:11.063213+0000","last_became_active":"2024-04-21T08:48:46.470498+0000","last_became_peered":"2024-04-21T08:48:46.470498+0000","last_unstale":"2024-04-21T08:51:11.063213+0000","last_undegraded":"2024-04-21T08:51:11.063213+0000","last_fullsized":"2024-04-21T08:51:11.063213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:49.452181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877897+0000","last_change":"2024-04-21T08:48:46.469111+0000","last_active":"2024-04-21T08:51:10.877897+0000","last_peered":"2024-04-21T08:51:10.877897+0000","last_clean":"2024-04-21T08:51:10.877897+0000","last_became_active":"2024-04-21T08:48:46.467926+0000","last_became_peered":"2024-04-21T08:48:46.467926+0000","last_unstale":"2024-04-21T08:51:10.877897+0000","last_undegraded":"2024-04-21T08:51:10.877897+0000","last_fullsized":"2024-04-21T08:51:10.877897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:35:38.230212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877755+0000","last_change":"2024-04-21T08:48:46.475673+0000","last_active":"2024-04-21T08:51:10.877755+0000","last_peered":"2024-04-21T08:51:10.877755+0000","last_clean":"2024-04-21T08:51:10.877755+0000","last_became_active":"2024-04-21T08:48:46.474917+0000","last_became_peered":"2024-04-21T08:48:46.474917+0000","last_unstale":"2024-04-21T08:51:10.877755+0000","last_undegraded":"2024-04-21T08:51:10.877755+0000","last_fullsized":"2024-04-21T08:51:10.877755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:01.661704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038205+0000","last_change":"2024-04-21T08:48:46.468147+0000","last_active":"2024-04-21T08:51:11.038205+0000","last_peered":"2024-04-21T08:51:11.038205+0000","last_clean":"2024-04-21T08:51:11.038205+0000","last_became_active":"2024-04-21T08:48:46.467811+0000","last_became_peered":"2024-04-21T08:48:46.467811+0000","last_unstale":"2024-04-21T08:51:11.038205+0000","last_undegraded":"2024-04-21T08:51:11.038205+0000","last_fullsized":"2024-04-21T08:51:11.038205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:21.658605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939250+0000","last_change":"2024-04-21T08:48:46.471322+0000","last_active":"2024-04-21T08:51:10.939250+0000","last_peered":"2024-04-21T08:51:10.939250+0000","last_clean":"2024-04-21T08:51:10.939250+0000","last_became_active":"2024-04-21T08:48:46.470991+0000","last_became_peered":"2024-04-21T08:48:46.470991+0000","last_unstale":"2024-04-21T08:51:10.939250+0000","last_undegraded":"2024-04-21T08:51:10.939250+0000","last_fullsized":"2024-04-21T08:51:10.939250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:22.839870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829796+0000","last_change":"2024-04-21T08:48:46.467039+0000","last_active":"2024-04-21T08:51:10.829796+0000","last_peered":"2024-04-21T08:51:10.829796+0000","last_clean":"2024-04-21T08:51:10.829796+0000","last_became_active":"2024-04-21T08:48:46.466614+0000","last_became_peered":"2024-04-21T08:48:46.466614+0000","last_unstale":"2024-04-21T08:51:10.829796+0000","last_undegraded":"2024-04-21T08:51:10.829796+0000","last_fullsized":"2024-04-21T08:51:10.829796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:38.029215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062714+0000","last_change":"2024-04-21T08:48:46.473268+0000","last_active":"2024-04-21T08:51:11.062714+0000","last_peered":"2024-04-21T08:51:11.062714+0000","last_clean":"2024-04-21T08:51:11.062714+0000","last_became_active":"2024-04-21T08:48:46.472261+0000","last_became_peered":"2024-04-21T08:48:46.472261+0000","last_unstale":"2024-04-21T08:51:11.062714+0000","last_undegraded":"2024-04-21T08:51:11.062714+0000","last_fullsized":"2024-04-21T08:51:11.062714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:58:52.990687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828189+0000","last_change":"2024-04-21T08:48:46.472788+0000","last_active":"2024-04-21T08:51:10.828189+0000","last_peered":"2024-04-21T08:51:10.828189+0000","last_clean":"2024-04-21T08:51:10.828189+0000","last_became_active":"2024-04-21T08:48:46.472482+0000","last_became_peered":"2024-04-21T08:48:46.472482+0000","last_unstale":"2024-04-21T08:51:10.828189+0000","last_undegraded":"2024-04-21T08:51:10.828189+0000","last_fullsized":"2024-04-21T08:51:10.828189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:48.483846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006966+0000","last_change":"2024-04-21T08:48:46.460771+0000","last_active":"2024-04-21T08:51:11.006966+0000","last_peered":"2024-04-21T08:51:11.006966+0000","last_clean":"2024-04-21T08:51:11.006966+0000","last_became_active":"2024-04-21T08:48:46.460472+0000","last_became_peered":"2024-04-21T08:48:46.460472+0000","last_unstale":"2024-04-21T08:51:11.006966+0000","last_undegraded":"2024-04-21T08:51:11.006966+0000","last_fullsized":"2024-04-21T08:51:11.006966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:46:40.766245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.830308+0000","last_change":"2024-04-21T08:48:46.463475+0000","last_active":"2024-04-21T08:51:10.830308+0000","last_peered":"2024-04-21T08:51:10.830308+0000","last_clean":"2024-04-21T08:51:10.830308+0000","last_became_active":"2024-04-21T08:48:46.462559+0000","last_became_peered":"2024-04-21T08:48:46.462559+0000","last_unstale":"2024-04-21T08:51:10.830308+0000","last_undegraded":"2024-04-21T08:51:10.830308+0000","last_fullsized":"2024-04-21T08:51:10.830308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:22:06.372410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063119+0000","last_change":"2024-04-21T08:48:46.476997+0000","last_active":"2024-04-21T08:51:11.063119+0000","last_peered":"2024-04-21T08:51:11.063119+0000","last_clean":"2024-04-21T08:51:11.063119+0000","last_became_active":"2024-04-21T08:48:46.476620+0000","last_became_peered":"2024-04-21T08:48:46.476620+0000","last_unstale":"2024-04-21T08:51:11.063119+0000","last_undegraded":"2024-04-21T08:51:11.063119+0000","last_fullsized":"2024-04-21T08:51:11.063119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:37.569629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038421+0000","last_change":"2024-04-21T08:48:46.469355+0000","last_active":"2024-04-21T08:51:11.038421+0000","last_peered":"2024-04-21T08:51:11.038421+0000","last_clean":"2024-04-21T08:51:11.038421+0000","last_became_active":"2024-04-21T08:48:46.468604+0000","last_became_peered":"2024-04-21T08:48:46.468604+0000","last_unstale":"2024-04-21T08:51:11.038421+0000","last_undegraded":"2024-04-21T08:51:11.038421+0000","last_fullsized":"2024-04-21T08:51:11.038421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:23:54.500982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036699+0000","last_change":"2024-04-21T08:48:46.453116+0000","last_active":"2024-04-21T08:51:11.036699+0000","last_peered":"2024-04-21T08:51:11.036699+0000","last_clean":"2024-04-21T08:51:11.036699+0000","last_became_active":"2024-04-21T08:48:46.452607+0000","last_became_peered":"2024-04-21T08:48:46.452607+0000","last_unstale":"2024-04-21T08:51:11.036699+0000","last_undegraded":"2024-04-21T08:51:11.036699+0000","last_fullsized":"2024-04-21T08:51:11.036699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:57:04.371812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877416+0000","last_change":"2024-04-21T08:48:46.474516+0000","last_active":"2024-04-21T08:51:10.877416+0000","last_peered":"2024-04-21T08:51:10.877416+0000","last_clean":"2024-04-21T08:51:10.877416+0000","last_became_active":"2024-04-21T08:48:46.474129+0000","last_became_peered":"2024-04-21T08:48:46.474129+0000","last_unstale":"2024-04-21T08:51:10.877416+0000","last_undegraded":"2024-04-21T08:51:10.877416+0000","last_fullsized":"2024-04-21T08:51:10.877416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:40:47.784976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877479+0000","last_change":"2024-04-21T08:49:36.907964+0000","last_active":"2024-04-21T08:51:10.877479+0000","last_peered":"2024-04-21T08:51:10.877479+0000","last_clean":"2024-04-21T08:51:10.877479+0000","last_became_active":"2024-04-21T08:49:36.907595+0000","last_became_peered":"2024-04-21T08:49:36.907595+0000","last_unstale":"2024-04-21T08:51:10.877479+0000","last_undegraded":"2024-04-21T08:51:10.877479+0000","last_fullsized":"2024-04-21T08:51:10.877479+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:14:04.907328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006192+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:51:11.006192+0000","last_peered":"2024-04-21T08:51:11.006192+0000","last_clean":"2024-04-21T08:51:11.006192+0000","last_became_active":"2024-04-21T08:48:46.469389+0000","last_became_peered":"2024-04-21T08:48:46.469389+0000","last_unstale":"2024-04-21T08:51:11.006192+0000","last_undegraded":"2024-04-21T08:51:11.006192+0000","last_fullsized":"2024-04-21T08:51:11.006192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:11:45.824402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063161+0000","last_change":"2024-04-21T08:48:46.470059+0000","last_active":"2024-04-21T08:51:11.063161+0000","last_peered":"2024-04-21T08:51:11.063161+0000","last_clean":"2024-04-21T08:51:11.063161+0000","last_became_active":"2024-04-21T08:48:46.469751+0000","last_became_peered":"2024-04-21T08:48:46.469751+0000","last_unstale":"2024-04-21T08:51:11.063161+0000","last_undegraded":"2024-04-21T08:51:11.063161+0000","last_fullsized":"2024-04-21T08:51:11.063161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:16.317879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.937061+0000","last_change":"2024-04-21T08:48:46.470129+0000","last_active":"2024-04-21T08:51:10.937061+0000","last_peered":"2024-04-21T08:51:10.937061+0000","last_clean":"2024-04-21T08:51:10.937061+0000","last_became_active":"2024-04-21T08:48:46.469763+0000","last_became_peered":"2024-04-21T08:48:46.469763+0000","last_unstale":"2024-04-21T08:51:10.937061+0000","last_undegraded":"2024-04-21T08:51:10.937061+0000","last_fullsized":"2024-04-21T08:51:10.937061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:47.731201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832982+0000","last_change":"2024-04-21T08:48:46.465356+0000","last_active":"2024-04-21T08:51:10.832982+0000","last_peered":"2024-04-21T08:51:10.832982+0000","last_clean":"2024-04-21T08:51:10.832982+0000","last_became_active":"2024-04-21T08:48:46.464183+0000","last_became_peered":"2024-04-21T08:48:46.464183+0000","last_unstale":"2024-04-21T08:51:10.832982+0000","last_undegraded":"2024-04-21T08:51:10.832982+0000","last_fullsized":"2024-04-21T08:51:10.832982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:37.533733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832348+0000","last_change":"2024-04-21T08:48:50.767378+0000","last_active":"2024-04-21T08:51:10.832348+0000","last_peered":"2024-04-21T08:51:10.832348+0000","last_clean":"2024-04-21T08:51:10.832348+0000","last_became_active":"2024-04-21T08:48:50.766856+0000","last_became_peered":"2024-04-21T08:48:50.766856+0000","last_unstale":"2024-04-21T08:51:10.832348+0000","last_undegraded":"2024-04-21T08:51:10.832348+0000","last_fullsized":"2024-04-21T08:51:10.832348+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:00.434540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936743+0000","last_change":"2024-04-21T08:48:45.550284+0000","last_active":"2024-04-21T08:51:10.936743+0000","last_peered":"2024-04-21T08:51:10.936743+0000","last_clean":"2024-04-21T08:51:10.936743+0000","last_became_active":"2024-04-21T08:48:45.549940+0000","last_became_peered":"2024-04-21T08:48:45.549940+0000","last_unstale":"2024-04-21T08:51:10.936743+0000","last_undegraded":"2024-04-21T08:51:10.936743+0000","last_fullsized":"2024-04-21T08:51:10.936743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:12.343299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828471+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:51:10.828471+0000","last_peered":"2024-04-21T08:51:10.828471+0000","last_clean":"2024-04-21T08:51:10.828471+0000","last_became_active":"2024-04-21T08:48:46.469647+0000","last_became_peered":"2024-04-21T08:48:46.469647+0000","last_unstale":"2024-04-21T08:51:10.828471+0000","last_undegraded":"2024-04-21T08:51:10.828471+0000","last_fullsized":"2024-04-21T08:51:10.828471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:10:28.204554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.037930+0000","last_change":"2024-04-21T08:48:50.771871+0000","last_active":"2024-04-21T08:51:11.037930+0000","last_peered":"2024-04-21T08:51:11.037930+0000","last_clean":"2024-04-21T08:51:11.037930+0000","last_became_active":"2024-04-21T08:48:50.770442+0000","last_became_peered":"2024-04-21T08:48:50.770442+0000","last_unstale":"2024-04-21T08:51:11.037930+0000","last_undegraded":"2024-04-21T08:51:11.037930+0000","last_fullsized":"2024-04-21T08:51:11.037930+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:18:57.813971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828497+0000","last_change":"2024-04-21T08:48:45.566492+0000","last_active":"2024-04-21T08:51:10.828497+0000","last_peered":"2024-04-21T08:51:10.828497+0000","last_clean":"2024-04-21T08:51:10.828497+0000","last_became_active":"2024-04-21T08:48:45.566169+0000","last_became_peered":"2024-04-21T08:48:45.566169+0000","last_unstale":"2024-04-21T08:51:10.828497+0000","last_undegraded":"2024-04-21T08:51:10.828497+0000","last_fullsized":"2024-04-21T08:51:10.828497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:49.985664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877476+0000","last_change":"2024-04-21T08:48:46.470866+0000","last_active":"2024-04-21T08:51:10.877476+0000","last_peered":"2024-04-21T08:51:10.877476+0000","last_clean":"2024-04-21T08:51:10.877476+0000","last_became_active":"2024-04-21T08:48:46.470401+0000","last_became_peered":"2024-04-21T08:48:46.470401+0000","last_unstale":"2024-04-21T08:51:10.877476+0000","last_undegraded":"2024-04-21T08:51:10.877476+0000","last_fullsized":"2024-04-21T08:51:10.877476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:07.506212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038069+0000","last_change":"2024-04-21T08:48:50.772339+0000","last_active":"2024-04-21T08:51:11.038069+0000","last_peered":"2024-04-21T08:51:11.038069+0000","last_clean":"2024-04-21T08:51:11.038069+0000","last_became_active":"2024-04-21T08:48:50.771946+0000","last_became_peered":"2024-04-21T08:48:50.771946+0000","last_unstale":"2024-04-21T08:51:11.038069+0000","last_undegraded":"2024-04-21T08:51:11.038069+0000","last_fullsized":"2024-04-21T08:51:11.038069+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:03:56.994581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939792+0000","last_change":"2024-04-21T08:48:45.553946+0000","last_active":"2024-04-21T08:51:10.939792+0000","last_peered":"2024-04-21T08:51:10.939792+0000","last_clean":"2024-04-21T08:51:10.939792+0000","last_became_active":"2024-04-21T08:48:45.553410+0000","last_became_peered":"2024-04-21T08:48:45.553410+0000","last_unstale":"2024-04-21T08:51:10.939792+0000","last_undegraded":"2024-04-21T08:51:10.939792+0000","last_fullsized":"2024-04-21T08:51:10.939792+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:24:02.387250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006377+0000","last_change":"2024-04-21T08:48:46.471815+0000","last_active":"2024-04-21T08:51:11.006377+0000","last_peered":"2024-04-21T08:51:11.006377+0000","last_clean":"2024-04-21T08:51:11.006377+0000","last_became_active":"2024-04-21T08:48:46.471492+0000","last_became_peered":"2024-04-21T08:48:46.471492+0000","last_unstale":"2024-04-21T08:51:11.006377+0000","last_undegraded":"2024-04-21T08:51:11.006377+0000","last_fullsized":"2024-04-21T08:51:11.006377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:26.051166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.037968+0000","last_change":"2024-04-21T08:48:50.767916+0000","last_active":"2024-04-21T08:51:11.037968+0000","last_peered":"2024-04-21T08:51:11.037968+0000","last_clean":"2024-04-21T08:51:11.037968+0000","last_became_active":"2024-04-21T08:48:50.767326+0000","last_became_peered":"2024-04-21T08:48:50.767326+0000","last_unstale":"2024-04-21T08:51:11.037968+0000","last_undegraded":"2024-04-21T08:51:11.037968+0000","last_fullsized":"2024-04-21T08:51:11.037968+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:05:02.185369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939279+0000","last_change":"2024-04-21T08:48:45.549982+0000","last_active":"2024-04-21T08:51:10.939279+0000","last_peered":"2024-04-21T08:51:10.939279+0000","last_clean":"2024-04-21T08:51:10.939279+0000","last_became_active":"2024-04-21T08:48:45.549611+0000","last_became_peered":"2024-04-21T08:48:45.549611+0000","last_unstale":"2024-04-21T08:51:10.939279+0000","last_undegraded":"2024-04-21T08:51:10.939279+0000","last_fullsized":"2024-04-21T08:51:10.939279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:46.035388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945847+0000","last_change":"2024-04-21T08:48:46.459068+0000","last_active":"2024-04-21T08:51:10.945847+0000","last_peered":"2024-04-21T08:51:10.945847+0000","last_clean":"2024-04-21T08:51:10.945847+0000","last_became_active":"2024-04-21T08:48:46.458686+0000","last_became_peered":"2024-04-21T08:48:46.458686+0000","last_unstale":"2024-04-21T08:51:10.945847+0000","last_undegraded":"2024-04-21T08:51:10.945847+0000","last_fullsized":"2024-04-21T08:51:10.945847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:31.373451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832831+0000","last_change":"2024-04-21T08:48:50.761392+0000","last_active":"2024-04-21T08:51:10.832831+0000","last_peered":"2024-04-21T08:51:10.832831+0000","last_clean":"2024-04-21T08:51:10.832831+0000","last_became_active":"2024-04-21T08:48:50.760917+0000","last_became_peered":"2024-04-21T08:48:50.760917+0000","last_unstale":"2024-04-21T08:51:10.832831+0000","last_undegraded":"2024-04-21T08:51:10.832831+0000","last_fullsized":"2024-04-21T08:51:10.832831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:19.070502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'8","reported_seq":163,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.097987+0000","last_change":"2024-04-21T08:48:45.553686+0000","last_active":"2024-04-21T08:51:11.097987+0000","last_peered":"2024-04-21T08:51:11.097987+0000","last_clean":"2024-04-21T08:51:11.097987+0000","last_became_active":"2024-04-21T08:48:45.553322+0000","last_became_peered":"2024-04-21T08:48:45.553322+0000","last_unstale":"2024-04-21T08:51:11.097987+0000","last_undegraded":"2024-04-21T08:51:11.097987+0000","last_fullsized":"2024-04-21T08:51:11.097987+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:15:24.320704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1712,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":3,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939561+0000","last_change":"2024-04-21T08:48:46.474555+0000","last_active":"2024-04-21T08:51:10.939561+0000","last_peered":"2024-04-21T08:51:10.939561+0000","last_clean":"2024-04-21T08:51:10.939561+0000","last_became_active":"2024-04-21T08:48:46.474185+0000","last_became_peered":"2024-04-21T08:48:46.474185+0000","last_unstale":"2024-04-21T08:51:10.939561+0000","last_undegraded":"2024-04-21T08:51:10.939561+0000","last_fullsized":"2024-04-21T08:51:10.939561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:52.884994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832578+0000","last_change":"2024-04-21T08:48:50.763181+0000","last_active":"2024-04-21T08:51:10.832578+0000","last_peered":"2024-04-21T08:51:10.832578+0000","last_clean":"2024-04-21T08:51:10.832578+0000","last_became_active":"2024-04-21T08:48:50.762718+0000","last_became_peered":"2024-04-21T08:48:50.762718+0000","last_unstale":"2024-04-21T08:51:10.832578+0000","last_undegraded":"2024-04-21T08:51:10.832578+0000","last_fullsized":"2024-04-21T08:51:10.832578+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:16:23.204605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829770+0000","last_change":"2024-04-21T08:48:45.547768+0000","last_active":"2024-04-21T08:51:10.829770+0000","last_peered":"2024-04-21T08:51:10.829770+0000","last_clean":"2024-04-21T08:51:10.829770+0000","last_became_active":"2024-04-21T08:48:45.547456+0000","last_became_peered":"2024-04-21T08:48:45.547456+0000","last_unstale":"2024-04-21T08:51:10.829770+0000","last_undegraded":"2024-04-21T08:51:10.829770+0000","last_fullsized":"2024-04-21T08:51:10.829770+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:23.502799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945790+0000","last_change":"2024-04-21T08:48:46.467988+0000","last_active":"2024-04-21T08:51:10.945790+0000","last_peered":"2024-04-21T08:51:10.945790+0000","last_clean":"2024-04-21T08:51:10.945790+0000","last_became_active":"2024-04-21T08:48:46.465212+0000","last_became_peered":"2024-04-21T08:48:46.465212+0000","last_unstale":"2024-04-21T08:51:10.945790+0000","last_undegraded":"2024-04-21T08:51:10.945790+0000","last_fullsized":"2024-04-21T08:51:10.945790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:41.743506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063079+0000","last_change":"2024-04-21T08:50:37.350070+0000","last_active":"2024-04-21T08:51:11.063079+0000","last_peered":"2024-04-21T08:51:11.063079+0000","last_clean":"2024-04-21T08:51:11.063079+0000","last_became_active":"2024-04-21T08:50:37.349622+0000","last_became_peered":"2024-04-21T08:50:37.349622+0000","last_unstale":"2024-04-21T08:51:11.063079+0000","last_undegraded":"2024-04-21T08:51:11.063079+0000","last_fullsized":"2024-04-21T08:51:11.063079+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:55.050244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.035911+0000","last_change":"2024-04-21T08:48:45.545615+0000","last_active":"2024-04-21T08:51:11.035911+0000","last_peered":"2024-04-21T08:51:11.035911+0000","last_clean":"2024-04-21T08:51:11.035911+0000","last_became_active":"2024-04-21T08:48:45.545280+0000","last_became_peered":"2024-04-21T08:48:45.545280+0000","last_unstale":"2024-04-21T08:51:11.035911+0000","last_undegraded":"2024-04-21T08:51:11.035911+0000","last_fullsized":"2024-04-21T08:51:11.035911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:50:00.470903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828012+0000","last_change":"2024-04-21T08:48:46.477326+0000","last_active":"2024-04-21T08:51:10.828012+0000","last_peered":"2024-04-21T08:51:10.828012+0000","last_clean":"2024-04-21T08:51:10.828012+0000","last_became_active":"2024-04-21T08:48:46.476914+0000","last_became_peered":"2024-04-21T08:48:46.476914+0000","last_unstale":"2024-04-21T08:51:10.828012+0000","last_undegraded":"2024-04-21T08:51:10.828012+0000","last_fullsized":"2024-04-21T08:51:10.828012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:04.312061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877774+0000","last_change":"2024-04-21T08:48:50.767429+0000","last_active":"2024-04-21T08:51:10.877774+0000","last_peered":"2024-04-21T08:51:10.877774+0000","last_clean":"2024-04-21T08:51:10.877774+0000","last_became_active":"2024-04-21T08:48:50.767008+0000","last_became_peered":"2024-04-21T08:48:50.767008+0000","last_unstale":"2024-04-21T08:51:10.877774+0000","last_undegraded":"2024-04-21T08:51:10.877774+0000","last_fullsized":"2024-04-21T08:51:10.877774+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:26.865571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945279+0000","last_change":"2024-04-21T08:48:45.553170+0000","last_active":"2024-04-21T08:51:10.945279+0000","last_peered":"2024-04-21T08:51:10.945279+0000","last_clean":"2024-04-21T08:51:10.945279+0000","last_became_active":"2024-04-21T08:48:45.552645+0000","last_became_peered":"2024-04-21T08:48:45.552645+0000","last_unstale":"2024-04-21T08:51:10.945279+0000","last_undegraded":"2024-04-21T08:51:10.945279+0000","last_fullsized":"2024-04-21T08:51:10.945279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:32:07.217212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936256+0000","last_change":"2024-04-21T08:48:46.478101+0000","last_active":"2024-04-21T08:51:10.936256+0000","last_peered":"2024-04-21T08:51:10.936256+0000","last_clean":"2024-04-21T08:51:10.936256+0000","last_became_active":"2024-04-21T08:48:46.477725+0000","last_became_peered":"2024-04-21T08:48:46.477725+0000","last_unstale":"2024-04-21T08:51:10.936256+0000","last_undegraded":"2024-04-21T08:51:10.936256+0000","last_fullsized":"2024-04-21T08:51:10.936256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:37:55.429569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828279+0000","last_change":"2024-04-21T08:48:50.762899+0000","last_active":"2024-04-21T08:51:10.828279+0000","last_peered":"2024-04-21T08:51:10.828279+0000","last_clean":"2024-04-21T08:51:10.828279+0000","last_became_active":"2024-04-21T08:48:50.762475+0000","last_became_peered":"2024-04-21T08:48:50.762475+0000","last_unstale":"2024-04-21T08:51:10.828279+0000","last_undegraded":"2024-04-21T08:51:10.828279+0000","last_fullsized":"2024-04-21T08:51:10.828279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:27.540414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":157,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.014871+0000","last_change":"2024-04-21T08:48:45.550151+0000","last_active":"2024-04-21T08:51:11.014871+0000","last_peered":"2024-04-21T08:51:11.014871+0000","last_clean":"2024-04-21T08:51:11.014871+0000","last_became_active":"2024-04-21T08:48:45.549829+0000","last_became_peered":"2024-04-21T08:48:45.549829+0000","last_unstale":"2024-04-21T08:51:11.014871+0000","last_undegraded":"2024-04-21T08:51:11.014871+0000","last_fullsized":"2024-04-21T08:51:11.014871+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:14.061189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006909+0000","last_change":"2024-04-21T08:48:46.472072+0000","last_active":"2024-04-21T08:51:11.006909+0000","last_peered":"2024-04-21T08:51:11.006909+0000","last_clean":"2024-04-21T08:51:11.006909+0000","last_became_active":"2024-04-21T08:48:46.471660+0000","last_became_peered":"2024-04-21T08:48:46.471660+0000","last_unstale":"2024-04-21T08:51:11.006909+0000","last_undegraded":"2024-04-21T08:51:11.006909+0000","last_fullsized":"2024-04-21T08:51:11.006909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:30:11.843870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936126+0000","last_change":"2024-04-21T08:48:50.773021+0000","last_active":"2024-04-21T08:51:10.936126+0000","last_peered":"2024-04-21T08:51:10.936126+0000","last_clean":"2024-04-21T08:51:10.936126+0000","last_became_active":"2024-04-21T08:48:50.771746+0000","last_became_peered":"2024-04-21T08:48:50.771746+0000","last_unstale":"2024-04-21T08:51:10.936126+0000","last_undegraded":"2024-04-21T08:51:10.936126+0000","last_fullsized":"2024-04-21T08:51:10.936126+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:33:21.415967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063066+0000","last_change":"2024-04-21T08:48:45.550376+0000","last_active":"2024-04-21T08:51:11.063066+0000","last_peered":"2024-04-21T08:51:11.063066+0000","last_clean":"2024-04-21T08:51:11.063066+0000","last_became_active":"2024-04-21T08:48:45.549594+0000","last_became_peered":"2024-04-21T08:48:45.549594+0000","last_unstale":"2024-04-21T08:51:11.063066+0000","last_undegraded":"2024-04-21T08:51:11.063066+0000","last_fullsized":"2024-04-21T08:51:11.063066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:26:27.046963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877696+0000","last_change":"2024-04-21T08:48:46.469499+0000","last_active":"2024-04-21T08:51:10.877696+0000","last_peered":"2024-04-21T08:51:10.877696+0000","last_clean":"2024-04-21T08:51:10.877696+0000","last_became_active":"2024-04-21T08:48:46.469124+0000","last_became_peered":"2024-04-21T08:48:46.469124+0000","last_unstale":"2024-04-21T08:51:10.877696+0000","last_undegraded":"2024-04-21T08:51:10.877696+0000","last_fullsized":"2024-04-21T08:51:10.877696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:58.708916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038212+0000","last_change":"2024-04-21T08:50:37.352313+0000","last_active":"2024-04-21T08:51:11.038212+0000","last_peered":"2024-04-21T08:51:11.038212+0000","last_clean":"2024-04-21T08:51:11.038212+0000","last_became_active":"2024-04-21T08:50:37.351775+0000","last_became_peered":"2024-04-21T08:50:37.351775+0000","last_unstale":"2024-04-21T08:51:11.038212+0000","last_undegraded":"2024-04-21T08:51:11.038212+0000","last_fullsized":"2024-04-21T08:51:11.038212+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:12:51.239368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,10],"acting":[4,1,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945985+0000","last_change":"2024-04-21T08:48:45.559319+0000","last_active":"2024-04-21T08:51:10.945985+0000","last_peered":"2024-04-21T08:51:10.945985+0000","last_clean":"2024-04-21T08:51:10.945985+0000","last_became_active":"2024-04-21T08:48:45.558895+0000","last_became_peered":"2024-04-21T08:48:45.558895+0000","last_unstale":"2024-04-21T08:51:10.945985+0000","last_undegraded":"2024-04-21T08:51:10.945985+0000","last_fullsized":"2024-04-21T08:51:10.945985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:01.878169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006563+0000","last_change":"2024-04-21T08:48:46.466110+0000","last_active":"2024-04-21T08:51:11.006563+0000","last_peered":"2024-04-21T08:51:11.006563+0000","last_clean":"2024-04-21T08:51:11.006563+0000","last_became_active":"2024-04-21T08:48:46.464374+0000","last_became_peered":"2024-04-21T08:48:46.464374+0000","last_unstale":"2024-04-21T08:51:11.006563+0000","last_undegraded":"2024-04-21T08:51:11.006563+0000","last_fullsized":"2024-04-21T08:51:11.006563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:26.307051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936310+0000","last_change":"2024-04-21T08:48:50.759976+0000","last_active":"2024-04-21T08:51:10.936310+0000","last_peered":"2024-04-21T08:51:10.936310+0000","last_clean":"2024-04-21T08:51:10.936310+0000","last_became_active":"2024-04-21T08:48:50.759596+0000","last_became_peered":"2024-04-21T08:48:50.759596+0000","last_unstale":"2024-04-21T08:51:10.936310+0000","last_undegraded":"2024-04-21T08:51:10.936310+0000","last_fullsized":"2024-04-21T08:51:10.936310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:09.678388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":322,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.136178+0000","last_change":"2024-04-21T08:48:45.549228+0000","last_active":"2024-04-21T08:51:11.136178+0000","last_peered":"2024-04-21T08:51:11.136178+0000","last_clean":"2024-04-21T08:51:11.136178+0000","last_became_active":"2024-04-21T08:48:45.548865+0000","last_became_peered":"2024-04-21T08:48:45.548865+0000","last_unstale":"2024-04-21T08:51:11.136178+0000","last_undegraded":"2024-04-21T08:51:11.136178+0000","last_fullsized":"2024-04-21T08:51:11.136178+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:28.620968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69009,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":248,"num_read_kb":187,"num_write":41,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939637+0000","last_change":"2024-04-21T08:48:46.470322+0000","last_active":"2024-04-21T08:51:10.939637+0000","last_peered":"2024-04-21T08:51:10.939637+0000","last_clean":"2024-04-21T08:51:10.939637+0000","last_became_active":"2024-04-21T08:48:46.467225+0000","last_became_peered":"2024-04-21T08:48:46.467225+0000","last_unstale":"2024-04-21T08:51:10.939637+0000","last_undegraded":"2024-04-21T08:51:10.939637+0000","last_fullsized":"2024-04-21T08:51:10.939637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:31:17.390926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.946282+0000","last_change":"2024-04-21T08:48:50.770982+0000","last_active":"2024-04-21T08:51:10.946282+0000","last_peered":"2024-04-21T08:51:10.946282+0000","last_clean":"2024-04-21T08:51:10.946282+0000","last_became_active":"2024-04-21T08:48:50.770312+0000","last_became_peered":"2024-04-21T08:48:50.770312+0000","last_unstale":"2024-04-21T08:51:10.946282+0000","last_undegraded":"2024-04-21T08:51:10.946282+0000","last_fullsized":"2024-04-21T08:51:10.946282+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:41:33.027351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832599+0000","last_change":"2024-04-21T08:48:45.556134+0000","last_active":"2024-04-21T08:51:10.832599+0000","last_peered":"2024-04-21T08:51:10.832599+0000","last_clean":"2024-04-21T08:51:10.832599+0000","last_became_active":"2024-04-21T08:48:45.555818+0000","last_became_peered":"2024-04-21T08:48:45.555818+0000","last_unstale":"2024-04-21T08:51:10.832599+0000","last_undegraded":"2024-04-21T08:51:10.832599+0000","last_fullsized":"2024-04-21T08:51:10.832599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:00:08.083783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036365+0000","last_change":"2024-04-21T08:49:36.913151+0000","last_active":"2024-04-21T08:51:11.036365+0000","last_peered":"2024-04-21T08:51:11.036365+0000","last_clean":"2024-04-21T08:51:11.036365+0000","last_became_active":"2024-04-21T08:49:36.912810+0000","last_became_peered":"2024-04-21T08:49:36.912810+0000","last_unstale":"2024-04-21T08:51:11.036365+0000","last_undegraded":"2024-04-21T08:51:11.036365+0000","last_fullsized":"2024-04-21T08:51:11.036365+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:17:32.887163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062896+0000","last_change":"2024-04-21T08:48:50.765967+0000","last_active":"2024-04-21T08:51:11.062896+0000","last_peered":"2024-04-21T08:51:11.062896+0000","last_clean":"2024-04-21T08:51:11.062896+0000","last_became_active":"2024-04-21T08:48:50.765557+0000","last_became_peered":"2024-04-21T08:48:50.765557+0000","last_unstale":"2024-04-21T08:51:11.062896+0000","last_undegraded":"2024-04-21T08:51:11.062896+0000","last_fullsized":"2024-04-21T08:51:11.062896+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:52.044058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945823+0000","last_change":"2024-04-21T08:48:45.559939+0000","last_active":"2024-04-21T08:51:10.945823+0000","last_peered":"2024-04-21T08:51:10.945823+0000","last_clean":"2024-04-21T08:51:10.945823+0000","last_became_active":"2024-04-21T08:48:45.559635+0000","last_became_peered":"2024-04-21T08:48:45.559635+0000","last_unstale":"2024-04-21T08:51:10.945823+0000","last_undegraded":"2024-04-21T08:51:10.945823+0000","last_fullsized":"2024-04-21T08:51:10.945823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:17.872806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062944+0000","last_change":"2024-04-21T08:48:46.464878+0000","last_active":"2024-04-21T08:51:11.062944+0000","last_peered":"2024-04-21T08:51:11.062944+0000","last_clean":"2024-04-21T08:51:11.062944+0000","last_became_active":"2024-04-21T08:48:46.464552+0000","last_became_peered":"2024-04-21T08:48:46.464552+0000","last_unstale":"2024-04-21T08:51:11.062944+0000","last_undegraded":"2024-04-21T08:51:11.062944+0000","last_fullsized":"2024-04-21T08:51:11.062944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:46:10.558693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939454+0000","last_change":"2024-04-21T08:50:37.351314+0000","last_active":"2024-04-21T08:51:10.939454+0000","last_peered":"2024-04-21T08:51:10.939454+0000","last_clean":"2024-04-21T08:51:10.939454+0000","last_became_active":"2024-04-21T08:50:37.350956+0000","last_became_peered":"2024-04-21T08:50:37.350956+0000","last_unstale":"2024-04-21T08:51:10.939454+0000","last_undegraded":"2024-04-21T08:51:10.939454+0000","last_fullsized":"2024-04-21T08:51:10.939454+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:01.802427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"94'6","reported_seq":163,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.103467+0000","last_change":"2024-04-21T08:48:45.548685+0000","last_active":"2024-04-21T08:51:11.103467+0000","last_peered":"2024-04-21T08:51:11.103467+0000","last_clean":"2024-04-21T08:51:11.103467+0000","last_became_active":"2024-04-21T08:48:45.548372+0000","last_became_peered":"2024-04-21T08:48:45.548372+0000","last_unstale":"2024-04-21T08:51:11.103467+0000","last_undegraded":"2024-04-21T08:51:11.103467+0000","last_fullsized":"2024-04-21T08:51:11.103467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:44.780297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":5,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062806+0000","last_change":"2024-04-21T08:48:46.471541+0000","last_active":"2024-04-21T08:51:11.062806+0000","last_peered":"2024-04-21T08:51:11.062806+0000","last_clean":"2024-04-21T08:51:11.062806+0000","last_became_active":"2024-04-21T08:48:46.471223+0000","last_became_peered":"2024-04-21T08:48:46.471223+0000","last_unstale":"2024-04-21T08:51:11.062806+0000","last_undegraded":"2024-04-21T08:51:11.062806+0000","last_fullsized":"2024-04-21T08:51:11.062806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:24:50.867905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036934+0000","last_change":"2024-04-21T08:48:50.773941+0000","last_active":"2024-04-21T08:51:11.036934+0000","last_peered":"2024-04-21T08:51:11.036934+0000","last_clean":"2024-04-21T08:51:11.036934+0000","last_became_active":"2024-04-21T08:48:50.772081+0000","last_became_peered":"2024-04-21T08:48:50.772081+0000","last_unstale":"2024-04-21T08:51:11.036934+0000","last_undegraded":"2024-04-21T08:51:11.036934+0000","last_fullsized":"2024-04-21T08:51:11.036934+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:04.438944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062783+0000","last_change":"2024-04-21T08:49:36.923090+0000","last_active":"2024-04-21T08:51:11.062783+0000","last_peered":"2024-04-21T08:51:11.062783+0000","last_clean":"2024-04-21T08:51:11.062783+0000","last_became_active":"2024-04-21T08:49:36.908075+0000","last_became_peered":"2024-04-21T08:49:36.908075+0000","last_unstale":"2024-04-21T08:51:11.062783+0000","last_undegraded":"2024-04-21T08:51:11.062783+0000","last_fullsized":"2024-04-21T08:51:11.062783+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:41.683372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945471+0000","last_change":"2024-04-21T08:48:46.465559+0000","last_active":"2024-04-21T08:51:10.945471+0000","last_peered":"2024-04-21T08:51:10.945471+0000","last_clean":"2024-04-21T08:51:10.945471+0000","last_became_active":"2024-04-21T08:48:46.465125+0000","last_became_peered":"2024-04-21T08:48:46.465125+0000","last_unstale":"2024-04-21T08:51:10.945471+0000","last_undegraded":"2024-04-21T08:51:10.945471+0000","last_fullsized":"2024-04-21T08:51:10.945471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:52:38.423925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945406+0000","last_change":"2024-04-21T08:48:50.763609+0000","last_active":"2024-04-21T08:51:10.945406+0000","last_peered":"2024-04-21T08:51:10.945406+0000","last_clean":"2024-04-21T08:51:10.945406+0000","last_became_active":"2024-04-21T08:48:50.763144+0000","last_became_peered":"2024-04-21T08:48:50.763144+0000","last_unstale":"2024-04-21T08:51:10.945406+0000","last_undegraded":"2024-04-21T08:51:10.945406+0000","last_fullsized":"2024-04-21T08:51:10.945406+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:26:18.611235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939650+0000","last_change":"2024-04-21T08:48:45.549912+0000","last_active":"2024-04-21T08:51:10.939650+0000","last_peered":"2024-04-21T08:51:10.939650+0000","last_clean":"2024-04-21T08:51:10.939650+0000","last_became_active":"2024-04-21T08:48:45.549585+0000","last_became_peered":"2024-04-21T08:48:45.549585+0000","last_unstale":"2024-04-21T08:51:10.939650+0000","last_undegraded":"2024-04-21T08:51:10.939650+0000","last_fullsized":"2024-04-21T08:51:10.939650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:58.287402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063099+0000","last_change":"2024-04-21T08:48:46.462746+0000","last_active":"2024-04-21T08:51:11.063099+0000","last_peered":"2024-04-21T08:51:11.063099+0000","last_clean":"2024-04-21T08:51:11.063099+0000","last_became_active":"2024-04-21T08:48:46.461602+0000","last_became_peered":"2024-04-21T08:48:46.461602+0000","last_unstale":"2024-04-21T08:51:11.063099+0000","last_undegraded":"2024-04-21T08:51:11.063099+0000","last_fullsized":"2024-04-21T08:51:11.063099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:52.198634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939308+0000","last_change":"2024-04-21T08:48:50.770770+0000","last_active":"2024-04-21T08:51:10.939308+0000","last_peered":"2024-04-21T08:51:10.939308+0000","last_clean":"2024-04-21T08:51:10.939308+0000","last_became_active":"2024-04-21T08:48:50.770326+0000","last_became_peered":"2024-04-21T08:48:50.770326+0000","last_unstale":"2024-04-21T08:51:10.939308+0000","last_undegraded":"2024-04-21T08:51:10.939308+0000","last_fullsized":"2024-04-21T08:51:10.939308+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:38.325155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936408+0000","last_change":"2024-04-21T08:48:45.558867+0000","last_active":"2024-04-21T08:51:10.936408+0000","last_peered":"2024-04-21T08:51:10.936408+0000","last_clean":"2024-04-21T08:51:10.936408+0000","last_became_active":"2024-04-21T08:48:45.558492+0000","last_became_peered":"2024-04-21T08:48:45.558492+0000","last_unstale":"2024-04-21T08:51:10.936408+0000","last_undegraded":"2024-04-21T08:51:10.936408+0000","last_fullsized":"2024-04-21T08:51:10.936408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:44:55.156404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.830180+0000","last_change":"2024-04-21T08:48:46.466461+0000","last_active":"2024-04-21T08:51:10.830180+0000","last_peered":"2024-04-21T08:51:10.830180+0000","last_clean":"2024-04-21T08:51:10.830180+0000","last_became_active":"2024-04-21T08:48:46.466035+0000","last_became_peered":"2024-04-21T08:48:46.466035+0000","last_unstale":"2024-04-21T08:51:10.830180+0000","last_undegraded":"2024-04-21T08:51:10.830180+0000","last_fullsized":"2024-04-21T08:51:10.830180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:47:32.307753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036555+0000","last_change":"2024-04-21T08:48:50.767655+0000","last_active":"2024-04-21T08:51:11.036555+0000","last_peered":"2024-04-21T08:51:11.036555+0000","last_clean":"2024-04-21T08:51:11.036555+0000","last_became_active":"2024-04-21T08:48:50.767305+0000","last_became_peered":"2024-04-21T08:48:50.767305+0000","last_unstale":"2024-04-21T08:51:11.036555+0000","last_undegraded":"2024-04-21T08:51:11.036555+0000","last_fullsized":"2024-04-21T08:51:11.036555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:31.210126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062757+0000","last_change":"2024-04-21T08:48:45.552952+0000","last_active":"2024-04-21T08:51:11.062757+0000","last_peered":"2024-04-21T08:51:11.062757+0000","last_clean":"2024-04-21T08:51:11.062757+0000","last_became_active":"2024-04-21T08:48:45.552556+0000","last_became_peered":"2024-04-21T08:48:45.552556+0000","last_unstale":"2024-04-21T08:51:11.062757+0000","last_undegraded":"2024-04-21T08:51:11.062757+0000","last_fullsized":"2024-04-21T08:51:11.062757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:03.428801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829859+0000","last_change":"2024-04-21T08:48:46.469020+0000","last_active":"2024-04-21T08:51:10.829859+0000","last_peered":"2024-04-21T08:51:10.829859+0000","last_clean":"2024-04-21T08:51:10.829859+0000","last_became_active":"2024-04-21T08:48:46.468631+0000","last_became_peered":"2024-04-21T08:48:46.468631+0000","last_unstale":"2024-04-21T08:51:10.829859+0000","last_undegraded":"2024-04-21T08:51:10.829859+0000","last_fullsized":"2024-04-21T08:51:10.829859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:08.559516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036969+0000","last_change":"2024-04-21T08:48:50.773942+0000","last_active":"2024-04-21T08:51:11.036969+0000","last_peered":"2024-04-21T08:51:11.036969+0000","last_clean":"2024-04-21T08:51:11.036969+0000","last_became_active":"2024-04-21T08:48:50.771876+0000","last_became_peered":"2024-04-21T08:48:50.771876+0000","last_unstale":"2024-04-21T08:51:11.036969+0000","last_undegraded":"2024-04-21T08:51:11.036969+0000","last_fullsized":"2024-04-21T08:51:11.036969+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:19.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945411+0000","last_change":"2024-04-21T08:48:46.462727+0000","last_active":"2024-04-21T08:51:10.945411+0000","last_peered":"2024-04-21T08:51:10.945411+0000","last_clean":"2024-04-21T08:51:10.945411+0000","last_became_active":"2024-04-21T08:48:46.461669+0000","last_became_peered":"2024-04-21T08:48:46.461669+0000","last_unstale":"2024-04-21T08:51:10.945411+0000","last_undegraded":"2024-04-21T08:51:10.945411+0000","last_fullsized":"2024-04-21T08:51:10.945411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:27.290790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":159,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036108+0000","last_change":"2024-04-21T08:48:45.558541+0000","last_active":"2024-04-21T08:51:11.036108+0000","last_peered":"2024-04-21T08:51:11.036108+0000","last_clean":"2024-04-21T08:51:11.036108+0000","last_became_active":"2024-04-21T08:48:45.558186+0000","last_became_peered":"2024-04-21T08:48:45.558186+0000","last_unstale":"2024-04-21T08:51:11.036108+0000","last_undegraded":"2024-04-21T08:51:11.036108+0000","last_fullsized":"2024-04-21T08:51:11.036108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:02.919144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062978+0000","last_change":"2024-04-21T08:48:50.770838+0000","last_active":"2024-04-21T08:51:11.062978+0000","last_peered":"2024-04-21T08:51:11.062978+0000","last_clean":"2024-04-21T08:51:11.062978+0000","last_became_active":"2024-04-21T08:48:50.770447+0000","last_became_peered":"2024-04-21T08:48:50.770447+0000","last_unstale":"2024-04-21T08:51:11.062978+0000","last_undegraded":"2024-04-21T08:51:11.062978+0000","last_fullsized":"2024-04-21T08:51:11.062978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:48:21.010284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828090+0000","last_change":"2024-04-21T08:48:46.478145+0000","last_active":"2024-04-21T08:51:10.828090+0000","last_peered":"2024-04-21T08:51:10.828090+0000","last_clean":"2024-04-21T08:51:10.828090+0000","last_became_active":"2024-04-21T08:48:46.477813+0000","last_became_peered":"2024-04-21T08:48:46.477813+0000","last_unstale":"2024-04-21T08:51:10.828090+0000","last_undegraded":"2024-04-21T08:51:10.828090+0000","last_fullsized":"2024-04-21T08:51:10.828090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:15.917619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038191+0000","last_change":"2024-04-21T08:48:45.545269+0000","last_active":"2024-04-21T08:51:11.038191+0000","last_peered":"2024-04-21T08:51:11.038191+0000","last_clean":"2024-04-21T08:51:11.038191+0000","last_became_active":"2024-04-21T08:48:45.544887+0000","last_became_peered":"2024-04-21T08:48:45.544887+0000","last_unstale":"2024-04-21T08:51:11.038191+0000","last_undegraded":"2024-04-21T08:51:11.038191+0000","last_fullsized":"2024-04-21T08:51:11.038191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:27.222570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":367,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062629+0000","last_change":"2024-04-21T08:47:34.198161+0000","last_active":"2024-04-21T08:51:11.062629+0000","last_peered":"2024-04-21T08:51:11.062629+0000","last_clean":"2024-04-21T08:51:11.062629+0000","last_became_active":"2024-04-21T08:47:34.197654+0000","last_became_peered":"2024-04-21T08:47:34.197654+0000","last_unstale":"2024-04-21T08:51:11.062629+0000","last_undegraded":"2024-04-21T08:51:11.062629+0000","last_fullsized":"2024-04-21T08:51:11.062629+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_clean_scrub_stamp":"2024-04-21T08:44:05.493464+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:58:58.295783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945722+0000","last_change":"2024-04-21T08:48:50.770826+0000","last_active":"2024-04-21T08:51:10.945722+0000","last_peered":"2024-04-21T08:51:10.945722+0000","last_clean":"2024-04-21T08:51:10.945722+0000","last_became_active":"2024-04-21T08:48:50.770372+0000","last_became_peered":"2024-04-21T08:48:50.770372+0000","last_unstale":"2024-04-21T08:51:10.945722+0000","last_undegraded":"2024-04-21T08:51:10.945722+0000","last_fullsized":"2024-04-21T08:51:10.945722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:59.637874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.035696+0000","last_change":"2024-04-21T08:48:46.459330+0000","last_active":"2024-04-21T08:51:11.035696+0000","last_peered":"2024-04-21T08:51:11.035696+0000","last_clean":"2024-04-21T08:51:11.035696+0000","last_became_active":"2024-04-21T08:48:46.458963+0000","last_became_peered":"2024-04-21T08:48:46.458963+0000","last_unstale":"2024-04-21T08:51:11.035696+0000","last_undegraded":"2024-04-21T08:51:11.035696+0000","last_fullsized":"2024-04-21T08:51:11.035696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:10.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062640+0000","last_change":"2024-04-21T08:49:36.917854+0000","last_active":"2024-04-21T08:51:11.062640+0000","last_peered":"2024-04-21T08:51:11.062640+0000","last_clean":"2024-04-21T08:51:11.062640+0000","last_became_active":"2024-04-21T08:49:36.915390+0000","last_became_peered":"2024-04-21T08:49:36.915390+0000","last_unstale":"2024-04-21T08:51:11.062640+0000","last_undegraded":"2024-04-21T08:51:11.062640+0000","last_fullsized":"2024-04-21T08:51:11.062640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:45.920752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062798+0000","last_change":"2024-04-21T08:48:50.765339+0000","last_active":"2024-04-21T08:51:11.062798+0000","last_peered":"2024-04-21T08:51:11.062798+0000","last_clean":"2024-04-21T08:51:11.062798+0000","last_became_active":"2024-04-21T08:48:50.764858+0000","last_became_peered":"2024-04-21T08:48:50.764858+0000","last_unstale":"2024-04-21T08:51:11.062798+0000","last_undegraded":"2024-04-21T08:51:11.062798+0000","last_fullsized":"2024-04-21T08:51:11.062798+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:26.075041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939660+0000","last_change":"2024-04-21T08:48:45.557768+0000","last_active":"2024-04-21T08:51:10.939660+0000","last_peered":"2024-04-21T08:51:10.939660+0000","last_clean":"2024-04-21T08:51:10.939660+0000","last_became_active":"2024-04-21T08:48:45.557448+0000","last_became_peered":"2024-04-21T08:48:45.557448+0000","last_unstale":"2024-04-21T08:51:10.939660+0000","last_undegraded":"2024-04-21T08:51:10.939660+0000","last_fullsized":"2024-04-21T08:51:10.939660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:09.063913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036820+0000","last_change":"2024-04-21T08:48:46.459425+0000","last_active":"2024-04-21T08:51:11.036820+0000","last_peered":"2024-04-21T08:51:11.036820+0000","last_clean":"2024-04-21T08:51:11.036820+0000","last_became_active":"2024-04-21T08:48:46.459020+0000","last_became_peered":"2024-04-21T08:48:46.459020+0000","last_unstale":"2024-04-21T08:51:11.036820+0000","last_undegraded":"2024-04-21T08:51:11.036820+0000","last_fullsized":"2024-04-21T08:51:11.036820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:08:39.042825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828005+0000","last_change":"2024-04-21T08:48:50.765740+0000","last_active":"2024-04-21T08:51:10.828005+0000","last_peered":"2024-04-21T08:51:10.828005+0000","last_clean":"2024-04-21T08:51:10.828005+0000","last_became_active":"2024-04-21T08:48:50.765359+0000","last_became_peered":"2024-04-21T08:48:50.765359+0000","last_unstale":"2024-04-21T08:51:10.828005+0000","last_undegraded":"2024-04-21T08:51:10.828005+0000","last_fullsized":"2024-04-21T08:51:10.828005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:34:42.716165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'44","reported_seq":101,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.054709+0000","last_change":"2024-04-21T08:48:45.564532+0000","last_active":"2024-04-21T08:51:11.054709+0000","last_peered":"2024-04-21T08:51:11.054709+0000","last_clean":"2024-04-21T08:51:11.054709+0000","last_became_active":"2024-04-21T08:48:45.564225+0000","last_became_peered":"2024-04-21T08:48:45.564225+0000","last_unstale":"2024-04-21T08:51:11.054709+0000","last_undegraded":"2024-04-21T08:51:11.054709+0000","last_fullsized":"2024-04-21T08:51:11.054709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:42.850371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828519+0000","last_change":"2024-04-21T08:48:46.466622+0000","last_active":"2024-04-21T08:51:10.828519+0000","last_peered":"2024-04-21T08:51:10.828519+0000","last_clean":"2024-04-21T08:51:10.828519+0000","last_became_active":"2024-04-21T08:48:46.466129+0000","last_became_peered":"2024-04-21T08:48:46.466129+0000","last_unstale":"2024-04-21T08:51:10.828519+0000","last_undegraded":"2024-04-21T08:51:10.828519+0000","last_fullsized":"2024-04-21T08:51:10.828519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:20:26.064193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832121+0000","last_change":"2024-04-21T08:48:50.759534+0000","last_active":"2024-04-21T08:51:10.832121+0000","last_peered":"2024-04-21T08:51:10.832121+0000","last_clean":"2024-04-21T08:51:10.832121+0000","last_became_active":"2024-04-21T08:48:50.759116+0000","last_became_peered":"2024-04-21T08:48:50.759116+0000","last_unstale":"2024-04-21T08:51:10.832121+0000","last_undegraded":"2024-04-21T08:51:10.832121+0000","last_fullsized":"2024-04-21T08:51:10.832121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:02:16.200901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038642+0000","last_change":"2024-04-21T08:48:45.545773+0000","last_active":"2024-04-21T08:51:11.038642+0000","last_peered":"2024-04-21T08:51:11.038642+0000","last_clean":"2024-04-21T08:51:11.038642+0000","last_became_active":"2024-04-21T08:48:45.545447+0000","last_became_peered":"2024-04-21T08:48:45.545447+0000","last_unstale":"2024-04-21T08:51:11.038642+0000","last_undegraded":"2024-04-21T08:51:11.038642+0000","last_fullsized":"2024-04-21T08:51:11.038642+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:59:42.171602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877976+0000","last_change":"2024-04-21T08:49:36.964411+0000","last_active":"2024-04-21T08:51:10.877976+0000","last_peered":"2024-04-21T08:51:10.877976+0000","last_clean":"2024-04-21T08:51:10.877976+0000","last_became_active":"2024-04-21T08:49:36.964046+0000","last_became_peered":"2024-04-21T08:49:36.964046+0000","last_unstale":"2024-04-21T08:51:10.877976+0000","last_undegraded":"2024-04-21T08:51:10.877976+0000","last_fullsized":"2024-04-21T08:51:10.877976+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:29:37.223495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038538+0000","last_change":"2024-04-21T08:50:37.351641+0000","last_active":"2024-04-21T08:51:11.038538+0000","last_peered":"2024-04-21T08:51:11.038538+0000","last_clean":"2024-04-21T08:51:11.038538+0000","last_became_active":"2024-04-21T08:50:37.351130+0000","last_became_peered":"2024-04-21T08:50:37.351130+0000","last_unstale":"2024-04-21T08:51:11.038538+0000","last_undegraded":"2024-04-21T08:51:11.038538+0000","last_fullsized":"2024-04-21T08:51:11.038538+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:56:58.989691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006155+0000","last_change":"2024-04-21T08:49:36.928993+0000","last_active":"2024-04-21T08:51:11.006155+0000","last_peered":"2024-04-21T08:51:11.006155+0000","last_clean":"2024-04-21T08:51:11.006155+0000","last_became_active":"2024-04-21T08:49:36.911247+0000","last_became_peered":"2024-04-21T08:49:36.911247+0000","last_unstale":"2024-04-21T08:51:11.006155+0000","last_undegraded":"2024-04-21T08:51:11.006155+0000","last_fullsized":"2024-04-21T08:51:11.006155+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:13.481670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939224+0000","last_change":"2024-04-21T08:48:46.465475+0000","last_active":"2024-04-21T08:51:10.939224+0000","last_peered":"2024-04-21T08:51:10.939224+0000","last_clean":"2024-04-21T08:51:10.939224+0000","last_became_active":"2024-04-21T08:48:46.465138+0000","last_became_peered":"2024-04-21T08:48:46.465138+0000","last_unstale":"2024-04-21T08:51:10.939224+0000","last_undegraded":"2024-04-21T08:51:10.939224+0000","last_fullsized":"2024-04-21T08:51:10.939224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:12:53.521547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936707+0000","last_change":"2024-04-21T08:48:50.765765+0000","last_active":"2024-04-21T08:51:10.936707+0000","last_peered":"2024-04-21T08:51:10.936707+0000","last_clean":"2024-04-21T08:51:10.936707+0000","last_became_active":"2024-04-21T08:48:50.765324+0000","last_became_peered":"2024-04-21T08:48:50.765324+0000","last_unstale":"2024-04-21T08:51:10.936707+0000","last_undegraded":"2024-04-21T08:51:10.936707+0000","last_fullsized":"2024-04-21T08:51:10.936707+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:41:37.846768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828042+0000","last_change":"2024-04-21T08:48:45.557400+0000","last_active":"2024-04-21T08:51:10.828042+0000","last_peered":"2024-04-21T08:51:10.828042+0000","last_clean":"2024-04-21T08:51:10.828042+0000","last_became_active":"2024-04-21T08:48:45.555012+0000","last_became_peered":"2024-04-21T08:48:45.555012+0000","last_unstale":"2024-04-21T08:51:10.828042+0000","last_undegraded":"2024-04-21T08:51:10.828042+0000","last_fullsized":"2024-04-21T08:51:10.828042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:00:31.254353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062791+0000","last_change":"2024-04-21T08:48:46.470042+0000","last_active":"2024-04-21T08:51:11.062791+0000","last_peered":"2024-04-21T08:51:11.062791+0000","last_clean":"2024-04-21T08:51:11.062791+0000","last_became_active":"2024-04-21T08:48:46.469716+0000","last_became_peered":"2024-04-21T08:48:46.469716+0000","last_unstale":"2024-04-21T08:51:11.062791+0000","last_undegraded":"2024-04-21T08:51:11.062791+0000","last_fullsized":"2024-04-21T08:51:11.062791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:09.934751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038592+0000","last_change":"2024-04-21T08:48:50.770889+0000","last_active":"2024-04-21T08:51:11.038592+0000","last_peered":"2024-04-21T08:51:11.038592+0000","last_clean":"2024-04-21T08:51:11.038592+0000","last_became_active":"2024-04-21T08:48:50.769980+0000","last_became_peered":"2024-04-21T08:48:50.769980+0000","last_unstale":"2024-04-21T08:51:11.038592+0000","last_undegraded":"2024-04-21T08:51:11.038592+0000","last_fullsized":"2024-04-21T08:51:11.038592+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:44.631633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832478+0000","last_change":"2024-04-21T08:48:45.555479+0000","last_active":"2024-04-21T08:51:10.832478+0000","last_peered":"2024-04-21T08:51:10.832478+0000","last_clean":"2024-04-21T08:51:10.832478+0000","last_became_active":"2024-04-21T08:48:45.555162+0000","last_became_peered":"2024-04-21T08:48:45.555162+0000","last_unstale":"2024-04-21T08:51:10.832478+0000","last_undegraded":"2024-04-21T08:51:10.832478+0000","last_fullsized":"2024-04-21T08:51:10.832478+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:15:20.262798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828417+0000","last_change":"2024-04-21T08:48:46.467962+0000","last_active":"2024-04-21T08:51:10.828417+0000","last_peered":"2024-04-21T08:51:10.828417+0000","last_clean":"2024-04-21T08:51:10.828417+0000","last_became_active":"2024-04-21T08:48:46.467561+0000","last_became_peered":"2024-04-21T08:48:46.467561+0000","last_unstale":"2024-04-21T08:51:10.828417+0000","last_undegraded":"2024-04-21T08:51:10.828417+0000","last_fullsized":"2024-04-21T08:51:10.828417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:18.262009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038487+0000","last_change":"2024-04-21T08:48:50.767678+0000","last_active":"2024-04-21T08:51:11.038487+0000","last_peered":"2024-04-21T08:51:11.038487+0000","last_clean":"2024-04-21T08:51:11.038487+0000","last_became_active":"2024-04-21T08:48:50.767309+0000","last_became_peered":"2024-04-21T08:48:50.767309+0000","last_unstale":"2024-04-21T08:51:11.038487+0000","last_undegraded":"2024-04-21T08:51:11.038487+0000","last_fullsized":"2024-04-21T08:51:11.038487+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:16:40.065043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939597+0000","last_change":"2024-04-21T08:48:45.553321+0000","last_active":"2024-04-21T08:51:10.939597+0000","last_peered":"2024-04-21T08:51:10.939597+0000","last_clean":"2024-04-21T08:51:10.939597+0000","last_became_active":"2024-04-21T08:48:45.552871+0000","last_became_peered":"2024-04-21T08:48:45.552871+0000","last_unstale":"2024-04-21T08:51:10.939597+0000","last_undegraded":"2024-04-21T08:51:10.939597+0000","last_fullsized":"2024-04-21T08:51:10.939597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:16:27.339801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062861+0000","last_change":"2024-04-21T08:48:46.473453+0000","last_active":"2024-04-21T08:51:11.062861+0000","last_peered":"2024-04-21T08:51:11.062861+0000","last_clean":"2024-04-21T08:51:11.062861+0000","last_became_active":"2024-04-21T08:48:46.472613+0000","last_became_peered":"2024-04-21T08:48:46.472613+0000","last_unstale":"2024-04-21T08:51:11.062861+0000","last_undegraded":"2024-04-21T08:51:11.062861+0000","last_fullsized":"2024-04-21T08:51:11.062861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:43:58.995997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006138+0000","last_change":"2024-04-21T08:48:50.765282+0000","last_active":"2024-04-21T08:51:11.006138+0000","last_peered":"2024-04-21T08:51:11.006138+0000","last_clean":"2024-04-21T08:51:11.006138+0000","last_became_active":"2024-04-21T08:48:50.764891+0000","last_became_peered":"2024-04-21T08:48:50.764891+0000","last_unstale":"2024-04-21T08:51:11.006138+0000","last_undegraded":"2024-04-21T08:51:11.006138+0000","last_fullsized":"2024-04-21T08:51:11.006138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:18:40.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038155+0000","last_change":"2024-04-21T08:49:36.926642+0000","last_active":"2024-04-21T08:51:11.038155+0000","last_peered":"2024-04-21T08:51:11.038155+0000","last_clean":"2024-04-21T08:51:11.038155+0000","last_became_active":"2024-04-21T08:49:36.910453+0000","last_became_peered":"2024-04-21T08:49:36.910453+0000","last_unstale":"2024-04-21T08:51:11.038155+0000","last_undegraded":"2024-04-21T08:51:11.038155+0000","last_fullsized":"2024-04-21T08:51:11.038155+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:19.290010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063219+0000","last_change":"2024-04-21T08:48:46.470058+0000","last_active":"2024-04-21T08:51:11.063219+0000","last_peered":"2024-04-21T08:51:11.063219+0000","last_clean":"2024-04-21T08:51:11.063219+0000","last_became_active":"2024-04-21T08:48:46.469485+0000","last_became_peered":"2024-04-21T08:48:46.469485+0000","last_unstale":"2024-04-21T08:51:11.063219+0000","last_undegraded":"2024-04-21T08:51:11.063219+0000","last_fullsized":"2024-04-21T08:51:11.063219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:33:13.212087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877446+0000","last_change":"2024-04-21T08:48:50.762944+0000","last_active":"2024-04-21T08:51:10.877446+0000","last_peered":"2024-04-21T08:51:10.877446+0000","last_clean":"2024-04-21T08:51:10.877446+0000","last_became_active":"2024-04-21T08:48:50.762508+0000","last_became_peered":"2024-04-21T08:48:50.762508+0000","last_unstale":"2024-04-21T08:51:10.877446+0000","last_undegraded":"2024-04-21T08:51:10.877446+0000","last_fullsized":"2024-04-21T08:51:10.877446+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:35.393339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.052626+0000","last_change":"2024-04-21T08:48:45.560361+0000","last_active":"2024-04-21T08:51:11.052626+0000","last_peered":"2024-04-21T08:51:11.052626+0000","last_clean":"2024-04-21T08:51:11.052626+0000","last_became_active":"2024-04-21T08:48:45.559946+0000","last_became_peered":"2024-04-21T08:48:45.559946+0000","last_unstale":"2024-04-21T08:51:11.052626+0000","last_undegraded":"2024-04-21T08:51:11.052626+0000","last_fullsized":"2024-04-21T08:51:11.052626+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:55.599714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063003+0000","last_change":"2024-04-21T08:48:46.472706+0000","last_active":"2024-04-21T08:51:11.063003+0000","last_peered":"2024-04-21T08:51:11.063003+0000","last_clean":"2024-04-21T08:51:11.063003+0000","last_became_active":"2024-04-21T08:48:46.472378+0000","last_became_peered":"2024-04-21T08:48:46.472378+0000","last_unstale":"2024-04-21T08:51:11.063003+0000","last_undegraded":"2024-04-21T08:51:11.063003+0000","last_fullsized":"2024-04-21T08:51:11.063003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:58.228899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945191+0000","last_change":"2024-04-21T08:48:50.760610+0000","last_active":"2024-04-21T08:51:10.945191+0000","last_peered":"2024-04-21T08:51:10.945191+0000","last_clean":"2024-04-21T08:51:10.945191+0000","last_became_active":"2024-04-21T08:48:50.759991+0000","last_became_peered":"2024-04-21T08:48:50.759991+0000","last_unstale":"2024-04-21T08:51:10.945191+0000","last_undegraded":"2024-04-21T08:51:10.945191+0000","last_fullsized":"2024-04-21T08:51:10.945191+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:43.649701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036942+0000","last_change":"2024-04-21T08:48:45.560773+0000","last_active":"2024-04-21T08:51:11.036942+0000","last_peered":"2024-04-21T08:51:11.036942+0000","last_clean":"2024-04-21T08:51:11.036942+0000","last_became_active":"2024-04-21T08:48:45.560431+0000","last_became_peered":"2024-04-21T08:48:45.560431+0000","last_unstale":"2024-04-21T08:51:11.036942+0000","last_undegraded":"2024-04-21T08:51:11.036942+0000","last_fullsized":"2024-04-21T08:51:11.036942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:07.858838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063259+0000","last_change":"2024-04-21T08:48:46.475983+0000","last_active":"2024-04-21T08:51:11.063259+0000","last_peered":"2024-04-21T08:51:11.063259+0000","last_clean":"2024-04-21T08:51:11.063259+0000","last_became_active":"2024-04-21T08:48:46.475667+0000","last_became_peered":"2024-04-21T08:48:46.475667+0000","last_unstale":"2024-04-21T08:51:11.063259+0000","last_undegraded":"2024-04-21T08:51:11.063259+0000","last_fullsized":"2024-04-21T08:51:11.063259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:28:41.013111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062960+0000","last_change":"2024-04-21T08:48:46.464675+0000","last_active":"2024-04-21T08:51:11.062960+0000","last_peered":"2024-04-21T08:51:11.062960+0000","last_clean":"2024-04-21T08:51:11.062960+0000","last_became_active":"2024-04-21T08:48:46.464314+0000","last_became_peered":"2024-04-21T08:48:46.464314+0000","last_unstale":"2024-04-21T08:51:11.062960+0000","last_undegraded":"2024-04-21T08:51:11.062960+0000","last_fullsized":"2024-04-21T08:51:11.062960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:53.048400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832307+0000","last_change":"2024-04-21T08:49:36.911025+0000","last_active":"2024-04-21T08:51:10.832307+0000","last_peered":"2024-04-21T08:51:10.832307+0000","last_clean":"2024-04-21T08:51:10.832307+0000","last_became_active":"2024-04-21T08:49:36.909942+0000","last_became_peered":"2024-04-21T08:49:36.909942+0000","last_unstale":"2024-04-21T08:51:10.832307+0000","last_undegraded":"2024-04-21T08:51:10.832307+0000","last_fullsized":"2024-04-21T08:51:10.832307+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:39.864253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832375+0000","last_change":"2024-04-21T08:48:46.467867+0000","last_active":"2024-04-21T08:51:10.832375+0000","last_peered":"2024-04-21T08:51:10.832375+0000","last_clean":"2024-04-21T08:51:10.832375+0000","last_became_active":"2024-04-21T08:48:46.465917+0000","last_became_peered":"2024-04-21T08:48:46.465917+0000","last_unstale":"2024-04-21T08:51:10.832375+0000","last_undegraded":"2024-04-21T08:51:10.832375+0000","last_fullsized":"2024-04-21T08:51:10.832375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:19:56.824998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877556+0000","last_change":"2024-04-21T08:48:46.467159+0000","last_active":"2024-04-21T08:51:10.877556+0000","last_peered":"2024-04-21T08:51:10.877556+0000","last_clean":"2024-04-21T08:51:10.877556+0000","last_became_active":"2024-04-21T08:48:46.466639+0000","last_became_peered":"2024-04-21T08:48:46.466639+0000","last_unstale":"2024-04-21T08:51:10.877556+0000","last_undegraded":"2024-04-21T08:51:10.877556+0000","last_fullsized":"2024-04-21T08:51:10.877556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:54:44.772535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036982+0000","last_change":"2024-04-21T08:48:46.466403+0000","last_active":"2024-04-21T08:51:11.036982+0000","last_peered":"2024-04-21T08:51:11.036982+0000","last_clean":"2024-04-21T08:51:11.036982+0000","last_became_active":"2024-04-21T08:48:46.466060+0000","last_became_peered":"2024-04-21T08:48:46.466060+0000","last_unstale":"2024-04-21T08:51:11.036982+0000","last_undegraded":"2024-04-21T08:51:11.036982+0000","last_fullsized":"2024-04-21T08:51:11.036982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:14.374193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063096+0000","last_change":"2024-04-21T08:49:36.909134+0000","last_active":"2024-04-21T08:51:11.063096+0000","last_peered":"2024-04-21T08:51:11.063096+0000","last_clean":"2024-04-21T08:51:11.063096+0000","last_became_active":"2024-04-21T08:49:36.908723+0000","last_became_peered":"2024-04-21T08:49:36.908723+0000","last_unstale":"2024-04-21T08:51:11.063096+0000","last_undegraded":"2024-04-21T08:51:11.063096+0000","last_fullsized":"2024-04-21T08:51:11.063096+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:28.123016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832933+0000","last_change":"2024-04-21T08:48:46.464098+0000","last_active":"2024-04-21T08:51:10.832933+0000","last_peered":"2024-04-21T08:51:10.832933+0000","last_clean":"2024-04-21T08:51:10.832933+0000","last_became_active":"2024-04-21T08:48:46.463756+0000","last_became_peered":"2024-04-21T08:48:46.463756+0000","last_unstale":"2024-04-21T08:51:10.832933+0000","last_undegraded":"2024-04-21T08:51:10.832933+0000","last_fullsized":"2024-04-21T08:51:10.832933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:31.725155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828666+0000","last_change":"2024-04-21T08:48:46.467372+0000","last_active":"2024-04-21T08:51:10.828666+0000","last_peered":"2024-04-21T08:51:10.828666+0000","last_clean":"2024-04-21T08:51:10.828666+0000","last_became_active":"2024-04-21T08:48:46.467043+0000","last_became_peered":"2024-04-21T08:48:46.467043+0000","last_unstale":"2024-04-21T08:51:10.828666+0000","last_undegraded":"2024-04-21T08:51:10.828666+0000","last_fullsized":"2024-04-21T08:51:10.828666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:41.889994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945316+0000","last_change":"2024-04-21T08:48:46.460604+0000","last_active":"2024-04-21T08:51:10.945316+0000","last_peered":"2024-04-21T08:51:10.945316+0000","last_clean":"2024-04-21T08:51:10.945316+0000","last_became_active":"2024-04-21T08:48:46.460276+0000","last_became_peered":"2024-04-21T08:48:46.460276+0000","last_unstale":"2024-04-21T08:51:10.945316+0000","last_undegraded":"2024-04-21T08:51:10.945316+0000","last_fullsized":"2024-04-21T08:51:10.945316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:30.194808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828076+0000","last_change":"2024-04-21T08:48:46.463665+0000","last_active":"2024-04-21T08:51:10.828076+0000","last_peered":"2024-04-21T08:51:10.828076+0000","last_clean":"2024-04-21T08:51:10.828076+0000","last_became_active":"2024-04-21T08:48:46.463309+0000","last_became_peered":"2024-04-21T08:48:46.463309+0000","last_unstale":"2024-04-21T08:51:10.828076+0000","last_undegraded":"2024-04-21T08:51:10.828076+0000","last_fullsized":"2024-04-21T08:51:10.828076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:12:36.436565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006407+0000","last_change":"2024-04-21T08:48:46.476845+0000","last_active":"2024-04-21T08:51:11.006407+0000","last_peered":"2024-04-21T08:51:11.006407+0000","last_clean":"2024-04-21T08:51:11.006407+0000","last_became_active":"2024-04-21T08:48:46.476507+0000","last_became_peered":"2024-04-21T08:48:46.476507+0000","last_unstale":"2024-04-21T08:51:11.006407+0000","last_undegraded":"2024-04-21T08:51:11.006407+0000","last_fullsized":"2024-04-21T08:51:11.006407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:47.424576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036615+0000","last_change":"2024-04-21T08:48:46.469316+0000","last_active":"2024-04-21T08:51:11.036615+0000","last_peered":"2024-04-21T08:51:11.036615+0000","last_clean":"2024-04-21T08:51:11.036615+0000","last_became_active":"2024-04-21T08:48:46.468852+0000","last_became_peered":"2024-04-21T08:48:46.468852+0000","last_unstale":"2024-04-21T08:51:11.036615+0000","last_undegraded":"2024-04-21T08:51:11.036615+0000","last_fullsized":"2024-04-21T08:51:11.036615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:05:03.664279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062922+0000","last_change":"2024-04-21T08:48:46.466703+0000","last_active":"2024-04-21T08:51:11.062922+0000","last_peered":"2024-04-21T08:51:11.062922+0000","last_clean":"2024-04-21T08:51:11.062922+0000","last_became_active":"2024-04-21T08:48:46.466395+0000","last_became_peered":"2024-04-21T08:48:46.466395+0000","last_unstale":"2024-04-21T08:51:11.062922+0000","last_undegraded":"2024-04-21T08:51:11.062922+0000","last_fullsized":"2024-04-21T08:51:11.062922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:08:56.956283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108027,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599448,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":835,"num_read_kb":523,"num_write":290,"num_write_kb":1443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1798344,"data_compressed":80739,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":529689,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32340,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712812,"statfs":{"total":95995035648,"available":95961919488,"internally_reserved":0,"allocated":1187840,"data_stored":1247346,"data_compressed":70096,"data_compressed_allocated":94208,"data_compressed_original":741376,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":927125,"data_compressed":48725,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":529621,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1097728,"data_stored":1127264,"data_compressed":65588,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1015808,"data_stored":515086,"data_compressed":26578,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":593908,"data_compressed":31190,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":526377,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1118208,"data_stored":595766,"data_compressed":31190,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1049999,"data_compressed":54881,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":531144,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32276,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712876,"statfs":{"total":95995035648,"available":95961985024,"internally_reserved":0,"allocated":1126400,"data_stored":1253529,"data_compressed":71744,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":69009,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":70867,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134166,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":393842,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":203110,"data_compressed":7767,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:51:16.327 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-21T08:51:16.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:51:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.465+0000 7f7442ab3640 1 -- 172.21.15.28:0/3832265478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f743c101a60 msgr2=0x7f743c105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.465+0000 7f7442ab3640 1 --2- 172.21.15.28:0/3832265478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f743c101a60 0x7f743c105aa0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7430009900 tx=0x7f743002bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.465+0000 7f7442ab3640 1 -- 172.21.15.28:0/3832265478 shutdown_connections 2024-04-21T08:51:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.465+0000 7f7442ab3640 1 --2- 172.21.15.28:0/3832265478 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f743c106180 0x7f743c10dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.465+0000 7f7442ab3640 1 --2- 172.21.15.28:0/3832265478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f743c101a60 0x7f743c105aa0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.465+0000 7f7442ab3640 1 --2- 172.21.15.28:0/3832265478 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f743c1010b0 0x7f743c101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.465+0000 7f7442ab3640 1 -- 172.21.15.28:0/3832265478 >> 172.21.15.28:0/3832265478 conn(0x7f743c0771d0 msgr2=0x7f743c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.465+0000 7f7442ab3640 1 -- 172.21.15.28:0/3832265478 shutdown_connections 2024-04-21T08:51:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.465+0000 7f7442ab3640 1 -- 172.21.15.28:0/3832265478 wait complete. 2024-04-21T08:51:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.465+0000 7f7442ab3640 1 Processor -- start 2024-04-21T08:51:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f7442ab3640 1 -- start start 2024-04-21T08:51:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f7442ab3640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f743c1010b0 0x7f743c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f7442ab3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f743c106180 0x7f743c199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f7442ab3640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f743c19a0b0 0x7f743c1a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f7442ab3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f743c111120 con 0x7f743c106180 2024-04-21T08:51:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f7442ab3640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f743c110fa0 con 0x7f743c1010b0 2024-04-21T08:51:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f7442ab3640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f743c1112a0 con 0x7f743c19a0b0 2024-04-21T08:51:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f743bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f743c106180 0x7f743c199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f743bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f743c106180 0x7f743c199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51970/0 (socket says 172.21.15.28:51970) 2024-04-21T08:51:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f743bfff640 1 -- 172.21.15.28:0/2044094702 learned_addr learned my addr 172.21.15.28:0/2044094702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f7441050640 1 --2- 172.21.15.28:0/2044094702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f743c19a0b0 0x7f743c1a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f744084f640 1 --2- 172.21.15.28:0/2044094702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f743c1010b0 0x7f743c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f743bfff640 1 -- 172.21.15.28:0/2044094702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f743c19a0b0 msgr2=0x7f743c1a4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f743bfff640 1 --2- 172.21.15.28:0/2044094702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f743c19a0b0 0x7f743c1a4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f743bfff640 1 -- 172.21.15.28:0/2044094702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f743c1010b0 msgr2=0x7f743c1995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f743bfff640 1 --2- 172.21.15.28:0/2044094702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f743c1010b0 0x7f743c1995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f743bfff640 1 -- 172.21.15.28:0/2044094702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f743c1a4e30 con 0x7f743c106180 2024-04-21T08:51:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.466+0000 7f743bfff640 1 --2- 172.21.15.28:0/2044094702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f743c106180 0x7f743c199b20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7430036040 tx=0x7f74300045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.467+0000 7f7439ffb640 1 -- 172.21.15.28:0/2044094702 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7430043020 con 0x7f743c106180 2024-04-21T08:51:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.467+0000 7f7439ffb640 1 -- 172.21.15.28:0/2044094702 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f743002cb70 con 0x7f743c106180 2024-04-21T08:51:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.467+0000 7f7442ab3640 1 -- 172.21.15.28:0/2044094702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f743c1a50c0 con 0x7f743c106180 2024-04-21T08:51:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.467+0000 7f7439ffb640 1 -- 172.21.15.28:0/2044094702 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7430003bf0 con 0x7f743c106180 2024-04-21T08:51:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.467+0000 7f7442ab3640 1 -- 172.21.15.28:0/2044094702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f743c1a5660 con 0x7f743c106180 2024-04-21T08:51:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.468+0000 7f7442ab3640 1 -- 172.21.15.28:0/2044094702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f743c069230 con 0x7f743c106180 2024-04-21T08:51:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.469+0000 7f7439ffb640 1 -- 172.21.15.28:0/2044094702 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f7430046020 con 0x7f743c106180 2024-04-21T08:51:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.470+0000 7f7439ffb640 1 --2- 172.21.15.28:0/2044094702 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f74240a3ec0 0x7f74240a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.470+0000 7f744084f640 1 --2- 172.21.15.28:0/2044094702 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f74240a3ec0 0x7f74240a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:16.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.470+0000 7f7439ffb640 1 -- 172.21.15.28:0/2044094702 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f74300f2e10 con 0x7f743c106180 2024-04-21T08:51:16.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.470+0000 7f744084f640 1 --2- 172.21.15.28:0/2044094702 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f74240a3ec0 0x7f74240a6380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f742c009bd0 tx=0x7f742c005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:16.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.474+0000 7f7439ffb640 1 -- 172.21.15.28:0/2044094702 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f74300f6030 con 0x7f743c106180 2024-04-21T08:51:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.657+0000 7f7442ab3640 1 -- 172.21.15.28:0/2044094702 --> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f743c109480 con 0x7f74240a3ec0 2024-04-21T08:51:16.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.666+0000 7f7439ffb640 1 -- 172.21.15.28:0/2044094702 <== mgr.25040 v2:172.21.15.28:6800/4182301910 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339688 (secure 0 0 0) 0x7f742c01b3f0 con 0x7f74240a3ec0 2024-04-21T08:51:16.670 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:16.672 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-21T08:51:16.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.669+0000 7f7442ab3640 1 -- 172.21.15.28:0/2044094702 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f74240a3ec0 msgr2=0x7f74240a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:16.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.669+0000 7f7442ab3640 1 --2- 172.21.15.28:0/2044094702 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f74240a3ec0 0x7f74240a6380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f742c009bd0 tx=0x7f742c005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:16.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.669+0000 7f7442ab3640 1 -- 172.21.15.28:0/2044094702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f743c106180 msgr2=0x7f743c199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:16.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.669+0000 7f7442ab3640 1 --2- 172.21.15.28:0/2044094702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f743c106180 0x7f743c199b20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7430036040 tx=0x7f74300045e0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.669+0000 7f7442ab3640 1 -- 172.21.15.28:0/2044094702 shutdown_connections 2024-04-21T08:51:16.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.669+0000 7f7442ab3640 1 --2- 172.21.15.28:0/2044094702 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f74240a3ec0 0x7f74240a6380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.669+0000 7f7442ab3640 1 --2- 172.21.15.28:0/2044094702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f743c19a0b0 0x7f743c1a4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.669+0000 7f7442ab3640 1 --2- 172.21.15.28:0/2044094702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f743c106180 0x7f743c199b20 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.669+0000 7f7442ab3640 1 --2- 172.21.15.28:0/2044094702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f743c1010b0 0x7f743c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.669+0000 7f7442ab3640 1 -- 172.21.15.28:0/2044094702 >> 172.21.15.28:0/2044094702 conn(0x7f743c0771d0 msgr2=0x7f743c0fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:16.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.670+0000 7f7442ab3640 1 -- 172.21.15.28:0/2044094702 shutdown_connections 2024-04-21T08:51:16.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.670+0000 7f7442ab3640 1 -- 172.21.15.28:0/2044094702 wait complete. 2024-04-21T08:51:16.680 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-21T08:51:14.842869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190050,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":971,"num_read_kb":642,"num_write":473,"num_write_kb":4383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181430,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370192,"kb_used_data":12680,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571632,"statfs":{"total":1151940427776,"available":1151561351168,"internally_reserved":0,"allocated":12984320,"data_stored":9426854,"data_compressed":518308,"data_compressed_allocated":704512,"data_compressed_original":3862528,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.990882"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063213+0000","last_change":"2024-04-21T08:48:46.471626+0000","last_active":"2024-04-21T08:51:11.063213+0000","last_peered":"2024-04-21T08:51:11.063213+0000","last_clean":"2024-04-21T08:51:11.063213+0000","last_became_active":"2024-04-21T08:48:46.470498+0000","last_became_peered":"2024-04-21T08:48:46.470498+0000","last_unstale":"2024-04-21T08:51:11.063213+0000","last_undegraded":"2024-04-21T08:51:11.063213+0000","last_fullsized":"2024-04-21T08:51:11.063213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:49.452181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877897+0000","last_change":"2024-04-21T08:48:46.469111+0000","last_active":"2024-04-21T08:51:10.877897+0000","last_peered":"2024-04-21T08:51:10.877897+0000","last_clean":"2024-04-21T08:51:10.877897+0000","last_became_active":"2024-04-21T08:48:46.467926+0000","last_became_peered":"2024-04-21T08:48:46.467926+0000","last_unstale":"2024-04-21T08:51:10.877897+0000","last_undegraded":"2024-04-21T08:51:10.877897+0000","last_fullsized":"2024-04-21T08:51:10.877897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:35:38.230212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877755+0000","last_change":"2024-04-21T08:48:46.475673+0000","last_active":"2024-04-21T08:51:10.877755+0000","last_peered":"2024-04-21T08:51:10.877755+0000","last_clean":"2024-04-21T08:51:10.877755+0000","last_became_active":"2024-04-21T08:48:46.474917+0000","last_became_peered":"2024-04-21T08:48:46.474917+0000","last_unstale":"2024-04-21T08:51:10.877755+0000","last_undegraded":"2024-04-21T08:51:10.877755+0000","last_fullsized":"2024-04-21T08:51:10.877755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:01.661704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038205+0000","last_change":"2024-04-21T08:48:46.468147+0000","last_active":"2024-04-21T08:51:11.038205+0000","last_peered":"2024-04-21T08:51:11.038205+0000","last_clean":"2024-04-21T08:51:11.038205+0000","last_became_active":"2024-04-21T08:48:46.467811+0000","last_became_peered":"2024-04-21T08:48:46.467811+0000","last_unstale":"2024-04-21T08:51:11.038205+0000","last_undegraded":"2024-04-21T08:51:11.038205+0000","last_fullsized":"2024-04-21T08:51:11.038205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:21.658605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939250+0000","last_change":"2024-04-21T08:48:46.471322+0000","last_active":"2024-04-21T08:51:10.939250+0000","last_peered":"2024-04-21T08:51:10.939250+0000","last_clean":"2024-04-21T08:51:10.939250+0000","last_became_active":"2024-04-21T08:48:46.470991+0000","last_became_peered":"2024-04-21T08:48:46.470991+0000","last_unstale":"2024-04-21T08:51:10.939250+0000","last_undegraded":"2024-04-21T08:51:10.939250+0000","last_fullsized":"2024-04-21T08:51:10.939250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:22.839870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829796+0000","last_change":"2024-04-21T08:48:46.467039+0000","last_active":"2024-04-21T08:51:10.829796+0000","last_peered":"2024-04-21T08:51:10.829796+0000","last_clean":"2024-04-21T08:51:10.829796+0000","last_became_active":"2024-04-21T08:48:46.466614+0000","last_became_peered":"2024-04-21T08:48:46.466614+0000","last_unstale":"2024-04-21T08:51:10.829796+0000","last_undegraded":"2024-04-21T08:51:10.829796+0000","last_fullsized":"2024-04-21T08:51:10.829796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:38.029215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062714+0000","last_change":"2024-04-21T08:48:46.473268+0000","last_active":"2024-04-21T08:51:11.062714+0000","last_peered":"2024-04-21T08:51:11.062714+0000","last_clean":"2024-04-21T08:51:11.062714+0000","last_became_active":"2024-04-21T08:48:46.472261+0000","last_became_peered":"2024-04-21T08:48:46.472261+0000","last_unstale":"2024-04-21T08:51:11.062714+0000","last_undegraded":"2024-04-21T08:51:11.062714+0000","last_fullsized":"2024-04-21T08:51:11.062714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:58:52.990687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828189+0000","last_change":"2024-04-21T08:48:46.472788+0000","last_active":"2024-04-21T08:51:10.828189+0000","last_peered":"2024-04-21T08:51:10.828189+0000","last_clean":"2024-04-21T08:51:10.828189+0000","last_became_active":"2024-04-21T08:48:46.472482+0000","last_became_peered":"2024-04-21T08:48:46.472482+0000","last_unstale":"2024-04-21T08:51:10.828189+0000","last_undegraded":"2024-04-21T08:51:10.828189+0000","last_fullsized":"2024-04-21T08:51:10.828189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:48.483846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006966+0000","last_change":"2024-04-21T08:48:46.460771+0000","last_active":"2024-04-21T08:51:11.006966+0000","last_peered":"2024-04-21T08:51:11.006966+0000","last_clean":"2024-04-21T08:51:11.006966+0000","last_became_active":"2024-04-21T08:48:46.460472+0000","last_became_peered":"2024-04-21T08:48:46.460472+0000","last_unstale":"2024-04-21T08:51:11.006966+0000","last_undegraded":"2024-04-21T08:51:11.006966+0000","last_fullsized":"2024-04-21T08:51:11.006966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:46:40.766245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.830308+0000","last_change":"2024-04-21T08:48:46.463475+0000","last_active":"2024-04-21T08:51:10.830308+0000","last_peered":"2024-04-21T08:51:10.830308+0000","last_clean":"2024-04-21T08:51:10.830308+0000","last_became_active":"2024-04-21T08:48:46.462559+0000","last_became_peered":"2024-04-21T08:48:46.462559+0000","last_unstale":"2024-04-21T08:51:10.830308+0000","last_undegraded":"2024-04-21T08:51:10.830308+0000","last_fullsized":"2024-04-21T08:51:10.830308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:22:06.372410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063119+0000","last_change":"2024-04-21T08:48:46.476997+0000","last_active":"2024-04-21T08:51:11.063119+0000","last_peered":"2024-04-21T08:51:11.063119+0000","last_clean":"2024-04-21T08:51:11.063119+0000","last_became_active":"2024-04-21T08:48:46.476620+0000","last_became_peered":"2024-04-21T08:48:46.476620+0000","last_unstale":"2024-04-21T08:51:11.063119+0000","last_undegraded":"2024-04-21T08:51:11.063119+0000","last_fullsized":"2024-04-21T08:51:11.063119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:37.569629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038421+0000","last_change":"2024-04-21T08:48:46.469355+0000","last_active":"2024-04-21T08:51:11.038421+0000","last_peered":"2024-04-21T08:51:11.038421+0000","last_clean":"2024-04-21T08:51:11.038421+0000","last_became_active":"2024-04-21T08:48:46.468604+0000","last_became_peered":"2024-04-21T08:48:46.468604+0000","last_unstale":"2024-04-21T08:51:11.038421+0000","last_undegraded":"2024-04-21T08:51:11.038421+0000","last_fullsized":"2024-04-21T08:51:11.038421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:23:54.500982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036699+0000","last_change":"2024-04-21T08:48:46.453116+0000","last_active":"2024-04-21T08:51:11.036699+0000","last_peered":"2024-04-21T08:51:11.036699+0000","last_clean":"2024-04-21T08:51:11.036699+0000","last_became_active":"2024-04-21T08:48:46.452607+0000","last_became_peered":"2024-04-21T08:48:46.452607+0000","last_unstale":"2024-04-21T08:51:11.036699+0000","last_undegraded":"2024-04-21T08:51:11.036699+0000","last_fullsized":"2024-04-21T08:51:11.036699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:57:04.371812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877416+0000","last_change":"2024-04-21T08:48:46.474516+0000","last_active":"2024-04-21T08:51:10.877416+0000","last_peered":"2024-04-21T08:51:10.877416+0000","last_clean":"2024-04-21T08:51:10.877416+0000","last_became_active":"2024-04-21T08:48:46.474129+0000","last_became_peered":"2024-04-21T08:48:46.474129+0000","last_unstale":"2024-04-21T08:51:10.877416+0000","last_undegraded":"2024-04-21T08:51:10.877416+0000","last_fullsized":"2024-04-21T08:51:10.877416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:40:47.784976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877479+0000","last_change":"2024-04-21T08:49:36.907964+0000","last_active":"2024-04-21T08:51:10.877479+0000","last_peered":"2024-04-21T08:51:10.877479+0000","last_clean":"2024-04-21T08:51:10.877479+0000","last_became_active":"2024-04-21T08:49:36.907595+0000","last_became_peered":"2024-04-21T08:49:36.907595+0000","last_unstale":"2024-04-21T08:51:10.877479+0000","last_undegraded":"2024-04-21T08:51:10.877479+0000","last_fullsized":"2024-04-21T08:51:10.877479+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:14:04.907328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006192+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:51:11.006192+0000","last_peered":"2024-04-21T08:51:11.006192+0000","last_clean":"2024-04-21T08:51:11.006192+0000","last_became_active":"2024-04-21T08:48:46.469389+0000","last_became_peered":"2024-04-21T08:48:46.469389+0000","last_unstale":"2024-04-21T08:51:11.006192+0000","last_undegraded":"2024-04-21T08:51:11.006192+0000","last_fullsized":"2024-04-21T08:51:11.006192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:11:45.824402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063161+0000","last_change":"2024-04-21T08:48:46.470059+0000","last_active":"2024-04-21T08:51:11.063161+0000","last_peered":"2024-04-21T08:51:11.063161+0000","last_clean":"2024-04-21T08:51:11.063161+0000","last_became_active":"2024-04-21T08:48:46.469751+0000","last_became_peered":"2024-04-21T08:48:46.469751+0000","last_unstale":"2024-04-21T08:51:11.063161+0000","last_undegraded":"2024-04-21T08:51:11.063161+0000","last_fullsized":"2024-04-21T08:51:11.063161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:16.317879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.937061+0000","last_change":"2024-04-21T08:48:46.470129+0000","last_active":"2024-04-21T08:51:10.937061+0000","last_peered":"2024-04-21T08:51:10.937061+0000","last_clean":"2024-04-21T08:51:10.937061+0000","last_became_active":"2024-04-21T08:48:46.469763+0000","last_became_peered":"2024-04-21T08:48:46.469763+0000","last_unstale":"2024-04-21T08:51:10.937061+0000","last_undegraded":"2024-04-21T08:51:10.937061+0000","last_fullsized":"2024-04-21T08:51:10.937061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:47.731201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832982+0000","last_change":"2024-04-21T08:48:46.465356+0000","last_active":"2024-04-21T08:51:10.832982+0000","last_peered":"2024-04-21T08:51:10.832982+0000","last_clean":"2024-04-21T08:51:10.832982+0000","last_became_active":"2024-04-21T08:48:46.464183+0000","last_became_peered":"2024-04-21T08:48:46.464183+0000","last_unstale":"2024-04-21T08:51:10.832982+0000","last_undegraded":"2024-04-21T08:51:10.832982+0000","last_fullsized":"2024-04-21T08:51:10.832982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:37.533733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832348+0000","last_change":"2024-04-21T08:48:50.767378+0000","last_active":"2024-04-21T08:51:10.832348+0000","last_peered":"2024-04-21T08:51:10.832348+0000","last_clean":"2024-04-21T08:51:10.832348+0000","last_became_active":"2024-04-21T08:48:50.766856+0000","last_became_peered":"2024-04-21T08:48:50.766856+0000","last_unstale":"2024-04-21T08:51:10.832348+0000","last_undegraded":"2024-04-21T08:51:10.832348+0000","last_fullsized":"2024-04-21T08:51:10.832348+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:00.434540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936743+0000","last_change":"2024-04-21T08:48:45.550284+0000","last_active":"2024-04-21T08:51:10.936743+0000","last_peered":"2024-04-21T08:51:10.936743+0000","last_clean":"2024-04-21T08:51:10.936743+0000","last_became_active":"2024-04-21T08:48:45.549940+0000","last_became_peered":"2024-04-21T08:48:45.549940+0000","last_unstale":"2024-04-21T08:51:10.936743+0000","last_undegraded":"2024-04-21T08:51:10.936743+0000","last_fullsized":"2024-04-21T08:51:10.936743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:12.343299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828471+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:51:10.828471+0000","last_peered":"2024-04-21T08:51:10.828471+0000","last_clean":"2024-04-21T08:51:10.828471+0000","last_became_active":"2024-04-21T08:48:46.469647+0000","last_became_peered":"2024-04-21T08:48:46.469647+0000","last_unstale":"2024-04-21T08:51:10.828471+0000","last_undegraded":"2024-04-21T08:51:10.828471+0000","last_fullsized":"2024-04-21T08:51:10.828471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:10:28.204554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.037930+0000","last_change":"2024-04-21T08:48:50.771871+0000","last_active":"2024-04-21T08:51:11.037930+0000","last_peered":"2024-04-21T08:51:11.037930+0000","last_clean":"2024-04-21T08:51:11.037930+0000","last_became_active":"2024-04-21T08:48:50.770442+0000","last_became_peered":"2024-04-21T08:48:50.770442+0000","last_unstale":"2024-04-21T08:51:11.037930+0000","last_undegraded":"2024-04-21T08:51:11.037930+0000","last_fullsized":"2024-04-21T08:51:11.037930+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:18:57.813971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828497+0000","last_change":"2024-04-21T08:48:45.566492+0000","last_active":"2024-04-21T08:51:10.828497+0000","last_peered":"2024-04-21T08:51:10.828497+0000","last_clean":"2024-04-21T08:51:10.828497+0000","last_became_active":"2024-04-21T08:48:45.566169+0000","last_became_peered":"2024-04-21T08:48:45.566169+0000","last_unstale":"2024-04-21T08:51:10.828497+0000","last_undegraded":"2024-04-21T08:51:10.828497+0000","last_fullsized":"2024-04-21T08:51:10.828497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:49.985664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877476+0000","last_change":"2024-04-21T08:48:46.470866+0000","last_active":"2024-04-21T08:51:10.877476+0000","last_peered":"2024-04-21T08:51:10.877476+0000","last_clean":"2024-04-21T08:51:10.877476+0000","last_became_active":"2024-04-21T08:48:46.470401+0000","last_became_peered":"2024-04-21T08:48:46.470401+0000","last_unstale":"2024-04-21T08:51:10.877476+0000","last_undegraded":"2024-04-21T08:51:10.877476+0000","last_fullsized":"2024-04-21T08:51:10.877476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:07.506212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038069+0000","last_change":"2024-04-21T08:48:50.772339+0000","last_active":"2024-04-21T08:51:11.038069+0000","last_peered":"2024-04-21T08:51:11.038069+0000","last_clean":"2024-04-21T08:51:11.038069+0000","last_became_active":"2024-04-21T08:48:50.771946+0000","last_became_peered":"2024-04-21T08:48:50.771946+0000","last_unstale":"2024-04-21T08:51:11.038069+0000","last_undegraded":"2024-04-21T08:51:11.038069+0000","last_fullsized":"2024-04-21T08:51:11.038069+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:03:56.994581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939792+0000","last_change":"2024-04-21T08:48:45.553946+0000","last_active":"2024-04-21T08:51:10.939792+0000","last_peered":"2024-04-21T08:51:10.939792+0000","last_clean":"2024-04-21T08:51:10.939792+0000","last_became_active":"2024-04-21T08:48:45.553410+0000","last_became_peered":"2024-04-21T08:48:45.553410+0000","last_unstale":"2024-04-21T08:51:10.939792+0000","last_undegraded":"2024-04-21T08:51:10.939792+0000","last_fullsized":"2024-04-21T08:51:10.939792+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:24:02.387250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006377+0000","last_change":"2024-04-21T08:48:46.471815+0000","last_active":"2024-04-21T08:51:11.006377+0000","last_peered":"2024-04-21T08:51:11.006377+0000","last_clean":"2024-04-21T08:51:11.006377+0000","last_became_active":"2024-04-21T08:48:46.471492+0000","last_became_peered":"2024-04-21T08:48:46.471492+0000","last_unstale":"2024-04-21T08:51:11.006377+0000","last_undegraded":"2024-04-21T08:51:11.006377+0000","last_fullsized":"2024-04-21T08:51:11.006377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:26.051166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.037968+0000","last_change":"2024-04-21T08:48:50.767916+0000","last_active":"2024-04-21T08:51:11.037968+0000","last_peered":"2024-04-21T08:51:11.037968+0000","last_clean":"2024-04-21T08:51:11.037968+0000","last_became_active":"2024-04-21T08:48:50.767326+0000","last_became_peered":"2024-04-21T08:48:50.767326+0000","last_unstale":"2024-04-21T08:51:11.037968+0000","last_undegraded":"2024-04-21T08:51:11.037968+0000","last_fullsized":"2024-04-21T08:51:11.037968+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:05:02.185369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939279+0000","last_change":"2024-04-21T08:48:45.549982+0000","last_active":"2024-04-21T08:51:10.939279+0000","last_peered":"2024-04-21T08:51:10.939279+0000","last_clean":"2024-04-21T08:51:10.939279+0000","last_became_active":"2024-04-21T08:48:45.549611+0000","last_became_peered":"2024-04-21T08:48:45.549611+0000","last_unstale":"2024-04-21T08:51:10.939279+0000","last_undegraded":"2024-04-21T08:51:10.939279+0000","last_fullsized":"2024-04-21T08:51:10.939279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:46.035388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945847+0000","last_change":"2024-04-21T08:48:46.459068+0000","last_active":"2024-04-21T08:51:10.945847+0000","last_peered":"2024-04-21T08:51:10.945847+0000","last_clean":"2024-04-21T08:51:10.945847+0000","last_became_active":"2024-04-21T08:48:46.458686+0000","last_became_peered":"2024-04-21T08:48:46.458686+0000","last_unstale":"2024-04-21T08:51:10.945847+0000","last_undegraded":"2024-04-21T08:51:10.945847+0000","last_fullsized":"2024-04-21T08:51:10.945847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:31.373451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832831+0000","last_change":"2024-04-21T08:48:50.761392+0000","last_active":"2024-04-21T08:51:10.832831+0000","last_peered":"2024-04-21T08:51:10.832831+0000","last_clean":"2024-04-21T08:51:10.832831+0000","last_became_active":"2024-04-21T08:48:50.760917+0000","last_became_peered":"2024-04-21T08:48:50.760917+0000","last_unstale":"2024-04-21T08:51:10.832831+0000","last_undegraded":"2024-04-21T08:51:10.832831+0000","last_fullsized":"2024-04-21T08:51:10.832831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:19.070502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'8","reported_seq":163,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.097987+0000","last_change":"2024-04-21T08:48:45.553686+0000","last_active":"2024-04-21T08:51:11.097987+0000","last_peered":"2024-04-21T08:51:11.097987+0000","last_clean":"2024-04-21T08:51:11.097987+0000","last_became_active":"2024-04-21T08:48:45.553322+0000","last_became_peered":"2024-04-21T08:48:45.553322+0000","last_unstale":"2024-04-21T08:51:11.097987+0000","last_undegraded":"2024-04-21T08:51:11.097987+0000","last_fullsized":"2024-04-21T08:51:11.097987+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:15:24.320704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1712,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":3,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939561+0000","last_change":"2024-04-21T08:48:46.474555+0000","last_active":"2024-04-21T08:51:10.939561+0000","last_peered":"2024-04-21T08:51:10.939561+0000","last_clean":"2024-04-21T08:51:10.939561+0000","last_became_active":"2024-04-21T08:48:46.474185+0000","last_became_peered":"2024-04-21T08:48:46.474185+0000","last_unstale":"2024-04-21T08:51:10.939561+0000","last_undegraded":"2024-04-21T08:51:10.939561+0000","last_fullsized":"2024-04-21T08:51:10.939561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:52.884994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832578+0000","last_change":"2024-04-21T08:48:50.763181+0000","last_active":"2024-04-21T08:51:10.832578+0000","last_peered":"2024-04-21T08:51:10.832578+0000","last_clean":"2024-04-21T08:51:10.832578+0000","last_became_active":"2024-04-21T08:48:50.762718+0000","last_became_peered":"2024-04-21T08:48:50.762718+0000","last_unstale":"2024-04-21T08:51:10.832578+0000","last_undegraded":"2024-04-21T08:51:10.832578+0000","last_fullsized":"2024-04-21T08:51:10.832578+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:16:23.204605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829770+0000","last_change":"2024-04-21T08:48:45.547768+0000","last_active":"2024-04-21T08:51:10.829770+0000","last_peered":"2024-04-21T08:51:10.829770+0000","last_clean":"2024-04-21T08:51:10.829770+0000","last_became_active":"2024-04-21T08:48:45.547456+0000","last_became_peered":"2024-04-21T08:48:45.547456+0000","last_unstale":"2024-04-21T08:51:10.829770+0000","last_undegraded":"2024-04-21T08:51:10.829770+0000","last_fullsized":"2024-04-21T08:51:10.829770+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:23.502799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945790+0000","last_change":"2024-04-21T08:48:46.467988+0000","last_active":"2024-04-21T08:51:10.945790+0000","last_peered":"2024-04-21T08:51:10.945790+0000","last_clean":"2024-04-21T08:51:10.945790+0000","last_became_active":"2024-04-21T08:48:46.465212+0000","last_became_peered":"2024-04-21T08:48:46.465212+0000","last_unstale":"2024-04-21T08:51:10.945790+0000","last_undegraded":"2024-04-21T08:51:10.945790+0000","last_fullsized":"2024-04-21T08:51:10.945790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:41.743506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063079+0000","last_change":"2024-04-21T08:50:37.350070+0000","last_active":"2024-04-21T08:51:11.063079+0000","last_peered":"2024-04-21T08:51:11.063079+0000","last_clean":"2024-04-21T08:51:11.063079+0000","last_became_active":"2024-04-21T08:50:37.349622+0000","last_became_peered":"2024-04-21T08:50:37.349622+0000","last_unstale":"2024-04-21T08:51:11.063079+0000","last_undegraded":"2024-04-21T08:51:11.063079+0000","last_fullsized":"2024-04-21T08:51:11.063079+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:55.050244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.035911+0000","last_change":"2024-04-21T08:48:45.545615+0000","last_active":"2024-04-21T08:51:11.035911+0000","last_peered":"2024-04-21T08:51:11.035911+0000","last_clean":"2024-04-21T08:51:11.035911+0000","last_became_active":"2024-04-21T08:48:45.545280+0000","last_became_peered":"2024-04-21T08:48:45.545280+0000","last_unstale":"2024-04-21T08:51:11.035911+0000","last_undegraded":"2024-04-21T08:51:11.035911+0000","last_fullsized":"2024-04-21T08:51:11.035911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:50:00.470903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828012+0000","last_change":"2024-04-21T08:48:46.477326+0000","last_active":"2024-04-21T08:51:10.828012+0000","last_peered":"2024-04-21T08:51:10.828012+0000","last_clean":"2024-04-21T08:51:10.828012+0000","last_became_active":"2024-04-21T08:48:46.476914+0000","last_became_peered":"2024-04-21T08:48:46.476914+0000","last_unstale":"2024-04-21T08:51:10.828012+0000","last_undegraded":"2024-04-21T08:51:10.828012+0000","last_fullsized":"2024-04-21T08:51:10.828012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:04.312061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877774+0000","last_change":"2024-04-21T08:48:50.767429+0000","last_active":"2024-04-21T08:51:10.877774+0000","last_peered":"2024-04-21T08:51:10.877774+0000","last_clean":"2024-04-21T08:51:10.877774+0000","last_became_active":"2024-04-21T08:48:50.767008+0000","last_became_peered":"2024-04-21T08:48:50.767008+0000","last_unstale":"2024-04-21T08:51:10.877774+0000","last_undegraded":"2024-04-21T08:51:10.877774+0000","last_fullsized":"2024-04-21T08:51:10.877774+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:26.865571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945279+0000","last_change":"2024-04-21T08:48:45.553170+0000","last_active":"2024-04-21T08:51:10.945279+0000","last_peered":"2024-04-21T08:51:10.945279+0000","last_clean":"2024-04-21T08:51:10.945279+0000","last_became_active":"2024-04-21T08:48:45.552645+0000","last_became_peered":"2024-04-21T08:48:45.552645+0000","last_unstale":"2024-04-21T08:51:10.945279+0000","last_undegraded":"2024-04-21T08:51:10.945279+0000","last_fullsized":"2024-04-21T08:51:10.945279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:32:07.217212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936256+0000","last_change":"2024-04-21T08:48:46.478101+0000","last_active":"2024-04-21T08:51:10.936256+0000","last_peered":"2024-04-21T08:51:10.936256+0000","last_clean":"2024-04-21T08:51:10.936256+0000","last_became_active":"2024-04-21T08:48:46.477725+0000","last_became_peered":"2024-04-21T08:48:46.477725+0000","last_unstale":"2024-04-21T08:51:10.936256+0000","last_undegraded":"2024-04-21T08:51:10.936256+0000","last_fullsized":"2024-04-21T08:51:10.936256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:37:55.429569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828279+0000","last_change":"2024-04-21T08:48:50.762899+0000","last_active":"2024-04-21T08:51:10.828279+0000","last_peered":"2024-04-21T08:51:10.828279+0000","last_clean":"2024-04-21T08:51:10.828279+0000","last_became_active":"2024-04-21T08:48:50.762475+0000","last_became_peered":"2024-04-21T08:48:50.762475+0000","last_unstale":"2024-04-21T08:51:10.828279+0000","last_undegraded":"2024-04-21T08:51:10.828279+0000","last_fullsized":"2024-04-21T08:51:10.828279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:27.540414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":157,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.014871+0000","last_change":"2024-04-21T08:48:45.550151+0000","last_active":"2024-04-21T08:51:11.014871+0000","last_peered":"2024-04-21T08:51:11.014871+0000","last_clean":"2024-04-21T08:51:11.014871+0000","last_became_active":"2024-04-21T08:48:45.549829+0000","last_became_peered":"2024-04-21T08:48:45.549829+0000","last_unstale":"2024-04-21T08:51:11.014871+0000","last_undegraded":"2024-04-21T08:51:11.014871+0000","last_fullsized":"2024-04-21T08:51:11.014871+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:14.061189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006909+0000","last_change":"2024-04-21T08:48:46.472072+0000","last_active":"2024-04-21T08:51:11.006909+0000","last_peered":"2024-04-21T08:51:11.006909+0000","last_clean":"2024-04-21T08:51:11.006909+0000","last_became_active":"2024-04-21T08:48:46.471660+0000","last_became_peered":"2024-04-21T08:48:46.471660+0000","last_unstale":"2024-04-21T08:51:11.006909+0000","last_undegraded":"2024-04-21T08:51:11.006909+0000","last_fullsized":"2024-04-21T08:51:11.006909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:30:11.843870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936126+0000","last_change":"2024-04-21T08:48:50.773021+0000","last_active":"2024-04-21T08:51:10.936126+0000","last_peered":"2024-04-21T08:51:10.936126+0000","last_clean":"2024-04-21T08:51:10.936126+0000","last_became_active":"2024-04-21T08:48:50.771746+0000","last_became_peered":"2024-04-21T08:48:50.771746+0000","last_unstale":"2024-04-21T08:51:10.936126+0000","last_undegraded":"2024-04-21T08:51:10.936126+0000","last_fullsized":"2024-04-21T08:51:10.936126+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:33:21.415967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063066+0000","last_change":"2024-04-21T08:48:45.550376+0000","last_active":"2024-04-21T08:51:11.063066+0000","last_peered":"2024-04-21T08:51:11.063066+0000","last_clean":"2024-04-21T08:51:11.063066+0000","last_became_active":"2024-04-21T08:48:45.549594+0000","last_became_peered":"2024-04-21T08:48:45.549594+0000","last_unstale":"2024-04-21T08:51:11.063066+0000","last_undegraded":"2024-04-21T08:51:11.063066+0000","last_fullsized":"2024-04-21T08:51:11.063066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:26:27.046963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877696+0000","last_change":"2024-04-21T08:48:46.469499+0000","last_active":"2024-04-21T08:51:10.877696+0000","last_peered":"2024-04-21T08:51:10.877696+0000","last_clean":"2024-04-21T08:51:10.877696+0000","last_became_active":"2024-04-21T08:48:46.469124+0000","last_became_peered":"2024-04-21T08:48:46.469124+0000","last_unstale":"2024-04-21T08:51:10.877696+0000","last_undegraded":"2024-04-21T08:51:10.877696+0000","last_fullsized":"2024-04-21T08:51:10.877696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:58.708916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038212+0000","last_change":"2024-04-21T08:50:37.352313+0000","last_active":"2024-04-21T08:51:11.038212+0000","last_peered":"2024-04-21T08:51:11.038212+0000","last_clean":"2024-04-21T08:51:11.038212+0000","last_became_active":"2024-04-21T08:50:37.351775+0000","last_became_peered":"2024-04-21T08:50:37.351775+0000","last_unstale":"2024-04-21T08:51:11.038212+0000","last_undegraded":"2024-04-21T08:51:11.038212+0000","last_fullsized":"2024-04-21T08:51:11.038212+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:12:51.239368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,10],"acting":[4,1,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945985+0000","last_change":"2024-04-21T08:48:45.559319+0000","last_active":"2024-04-21T08:51:10.945985+0000","last_peered":"2024-04-21T08:51:10.945985+0000","last_clean":"2024-04-21T08:51:10.945985+0000","last_became_active":"2024-04-21T08:48:45.558895+0000","last_became_peered":"2024-04-21T08:48:45.558895+0000","last_unstale":"2024-04-21T08:51:10.945985+0000","last_undegraded":"2024-04-21T08:51:10.945985+0000","last_fullsized":"2024-04-21T08:51:10.945985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:01.878169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006563+0000","last_change":"2024-04-21T08:48:46.466110+0000","last_active":"2024-04-21T08:51:11.006563+0000","last_peered":"2024-04-21T08:51:11.006563+0000","last_clean":"2024-04-21T08:51:11.006563+0000","last_became_active":"2024-04-21T08:48:46.464374+0000","last_became_peered":"2024-04-21T08:48:46.464374+0000","last_unstale":"2024-04-21T08:51:11.006563+0000","last_undegraded":"2024-04-21T08:51:11.006563+0000","last_fullsized":"2024-04-21T08:51:11.006563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:26.307051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936310+0000","last_change":"2024-04-21T08:48:50.759976+0000","last_active":"2024-04-21T08:51:10.936310+0000","last_peered":"2024-04-21T08:51:10.936310+0000","last_clean":"2024-04-21T08:51:10.936310+0000","last_became_active":"2024-04-21T08:48:50.759596+0000","last_became_peered":"2024-04-21T08:48:50.759596+0000","last_unstale":"2024-04-21T08:51:10.936310+0000","last_undegraded":"2024-04-21T08:51:10.936310+0000","last_fullsized":"2024-04-21T08:51:10.936310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:09.678388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'40","reported_seq":322,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.136178+0000","last_change":"2024-04-21T08:48:45.549228+0000","last_active":"2024-04-21T08:51:11.136178+0000","last_peered":"2024-04-21T08:51:11.136178+0000","last_clean":"2024-04-21T08:51:11.136178+0000","last_became_active":"2024-04-21T08:48:45.548865+0000","last_became_peered":"2024-04-21T08:48:45.548865+0000","last_unstale":"2024-04-21T08:51:11.136178+0000","last_undegraded":"2024-04-21T08:51:11.136178+0000","last_fullsized":"2024-04-21T08:51:11.136178+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:28.620968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69009,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":248,"num_read_kb":187,"num_write":41,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939637+0000","last_change":"2024-04-21T08:48:46.470322+0000","last_active":"2024-04-21T08:51:10.939637+0000","last_peered":"2024-04-21T08:51:10.939637+0000","last_clean":"2024-04-21T08:51:10.939637+0000","last_became_active":"2024-04-21T08:48:46.467225+0000","last_became_peered":"2024-04-21T08:48:46.467225+0000","last_unstale":"2024-04-21T08:51:10.939637+0000","last_undegraded":"2024-04-21T08:51:10.939637+0000","last_fullsized":"2024-04-21T08:51:10.939637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:31:17.390926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.946282+0000","last_change":"2024-04-21T08:48:50.770982+0000","last_active":"2024-04-21T08:51:10.946282+0000","last_peered":"2024-04-21T08:51:10.946282+0000","last_clean":"2024-04-21T08:51:10.946282+0000","last_became_active":"2024-04-21T08:48:50.770312+0000","last_became_peered":"2024-04-21T08:48:50.770312+0000","last_unstale":"2024-04-21T08:51:10.946282+0000","last_undegraded":"2024-04-21T08:51:10.946282+0000","last_fullsized":"2024-04-21T08:51:10.946282+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:41:33.027351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832599+0000","last_change":"2024-04-21T08:48:45.556134+0000","last_active":"2024-04-21T08:51:10.832599+0000","last_peered":"2024-04-21T08:51:10.832599+0000","last_clean":"2024-04-21T08:51:10.832599+0000","last_became_active":"2024-04-21T08:48:45.555818+0000","last_became_peered":"2024-04-21T08:48:45.555818+0000","last_unstale":"2024-04-21T08:51:10.832599+0000","last_undegraded":"2024-04-21T08:51:10.832599+0000","last_fullsized":"2024-04-21T08:51:10.832599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:00:08.083783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036365+0000","last_change":"2024-04-21T08:49:36.913151+0000","last_active":"2024-04-21T08:51:11.036365+0000","last_peered":"2024-04-21T08:51:11.036365+0000","last_clean":"2024-04-21T08:51:11.036365+0000","last_became_active":"2024-04-21T08:49:36.912810+0000","last_became_peered":"2024-04-21T08:49:36.912810+0000","last_unstale":"2024-04-21T08:51:11.036365+0000","last_undegraded":"2024-04-21T08:51:11.036365+0000","last_fullsized":"2024-04-21T08:51:11.036365+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:17:32.887163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062896+0000","last_change":"2024-04-21T08:48:50.765967+0000","last_active":"2024-04-21T08:51:11.062896+0000","last_peered":"2024-04-21T08:51:11.062896+0000","last_clean":"2024-04-21T08:51:11.062896+0000","last_became_active":"2024-04-21T08:48:50.765557+0000","last_became_peered":"2024-04-21T08:48:50.765557+0000","last_unstale":"2024-04-21T08:51:11.062896+0000","last_undegraded":"2024-04-21T08:51:11.062896+0000","last_fullsized":"2024-04-21T08:51:11.062896+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:52.044058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945823+0000","last_change":"2024-04-21T08:48:45.559939+0000","last_active":"2024-04-21T08:51:10.945823+0000","last_peered":"2024-04-21T08:51:10.945823+0000","last_clean":"2024-04-21T08:51:10.945823+0000","last_became_active":"2024-04-21T08:48:45.559635+0000","last_became_peered":"2024-04-21T08:48:45.559635+0000","last_unstale":"2024-04-21T08:51:10.945823+0000","last_undegraded":"2024-04-21T08:51:10.945823+0000","last_fullsized":"2024-04-21T08:51:10.945823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:17.872806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062944+0000","last_change":"2024-04-21T08:48:46.464878+0000","last_active":"2024-04-21T08:51:11.062944+0000","last_peered":"2024-04-21T08:51:11.062944+0000","last_clean":"2024-04-21T08:51:11.062944+0000","last_became_active":"2024-04-21T08:48:46.464552+0000","last_became_peered":"2024-04-21T08:48:46.464552+0000","last_unstale":"2024-04-21T08:51:11.062944+0000","last_undegraded":"2024-04-21T08:51:11.062944+0000","last_fullsized":"2024-04-21T08:51:11.062944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:46:10.558693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939454+0000","last_change":"2024-04-21T08:50:37.351314+0000","last_active":"2024-04-21T08:51:10.939454+0000","last_peered":"2024-04-21T08:51:10.939454+0000","last_clean":"2024-04-21T08:51:10.939454+0000","last_became_active":"2024-04-21T08:50:37.350956+0000","last_became_peered":"2024-04-21T08:50:37.350956+0000","last_unstale":"2024-04-21T08:51:10.939454+0000","last_undegraded":"2024-04-21T08:51:10.939454+0000","last_fullsized":"2024-04-21T08:51:10.939454+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:01.802427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"94'6","reported_seq":163,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.103467+0000","last_change":"2024-04-21T08:48:45.548685+0000","last_active":"2024-04-21T08:51:11.103467+0000","last_peered":"2024-04-21T08:51:11.103467+0000","last_clean":"2024-04-21T08:51:11.103467+0000","last_became_active":"2024-04-21T08:48:45.548372+0000","last_became_peered":"2024-04-21T08:48:45.548372+0000","last_unstale":"2024-04-21T08:51:11.103467+0000","last_undegraded":"2024-04-21T08:51:11.103467+0000","last_fullsized":"2024-04-21T08:51:11.103467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:44.780297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":5,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062806+0000","last_change":"2024-04-21T08:48:46.471541+0000","last_active":"2024-04-21T08:51:11.062806+0000","last_peered":"2024-04-21T08:51:11.062806+0000","last_clean":"2024-04-21T08:51:11.062806+0000","last_became_active":"2024-04-21T08:48:46.471223+0000","last_became_peered":"2024-04-21T08:48:46.471223+0000","last_unstale":"2024-04-21T08:51:11.062806+0000","last_undegraded":"2024-04-21T08:51:11.062806+0000","last_fullsized":"2024-04-21T08:51:11.062806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:24:50.867905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036934+0000","last_change":"2024-04-21T08:48:50.773941+0000","last_active":"2024-04-21T08:51:11.036934+0000","last_peered":"2024-04-21T08:51:11.036934+0000","last_clean":"2024-04-21T08:51:11.036934+0000","last_became_active":"2024-04-21T08:48:50.772081+0000","last_became_peered":"2024-04-21T08:48:50.772081+0000","last_unstale":"2024-04-21T08:51:11.036934+0000","last_undegraded":"2024-04-21T08:51:11.036934+0000","last_fullsized":"2024-04-21T08:51:11.036934+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:04.438944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062783+0000","last_change":"2024-04-21T08:49:36.923090+0000","last_active":"2024-04-21T08:51:11.062783+0000","last_peered":"2024-04-21T08:51:11.062783+0000","last_clean":"2024-04-21T08:51:11.062783+0000","last_became_active":"2024-04-21T08:49:36.908075+0000","last_became_peered":"2024-04-21T08:49:36.908075+0000","last_unstale":"2024-04-21T08:51:11.062783+0000","last_undegraded":"2024-04-21T08:51:11.062783+0000","last_fullsized":"2024-04-21T08:51:11.062783+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:41.683372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945471+0000","last_change":"2024-04-21T08:48:46.465559+0000","last_active":"2024-04-21T08:51:10.945471+0000","last_peered":"2024-04-21T08:51:10.945471+0000","last_clean":"2024-04-21T08:51:10.945471+0000","last_became_active":"2024-04-21T08:48:46.465125+0000","last_became_peered":"2024-04-21T08:48:46.465125+0000","last_unstale":"2024-04-21T08:51:10.945471+0000","last_undegraded":"2024-04-21T08:51:10.945471+0000","last_fullsized":"2024-04-21T08:51:10.945471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:52:38.423925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945406+0000","last_change":"2024-04-21T08:48:50.763609+0000","last_active":"2024-04-21T08:51:10.945406+0000","last_peered":"2024-04-21T08:51:10.945406+0000","last_clean":"2024-04-21T08:51:10.945406+0000","last_became_active":"2024-04-21T08:48:50.763144+0000","last_became_peered":"2024-04-21T08:48:50.763144+0000","last_unstale":"2024-04-21T08:51:10.945406+0000","last_undegraded":"2024-04-21T08:51:10.945406+0000","last_fullsized":"2024-04-21T08:51:10.945406+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:26:18.611235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939650+0000","last_change":"2024-04-21T08:48:45.549912+0000","last_active":"2024-04-21T08:51:10.939650+0000","last_peered":"2024-04-21T08:51:10.939650+0000","last_clean":"2024-04-21T08:51:10.939650+0000","last_became_active":"2024-04-21T08:48:45.549585+0000","last_became_peered":"2024-04-21T08:48:45.549585+0000","last_unstale":"2024-04-21T08:51:10.939650+0000","last_undegraded":"2024-04-21T08:51:10.939650+0000","last_fullsized":"2024-04-21T08:51:10.939650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:58.287402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063099+0000","last_change":"2024-04-21T08:48:46.462746+0000","last_active":"2024-04-21T08:51:11.063099+0000","last_peered":"2024-04-21T08:51:11.063099+0000","last_clean":"2024-04-21T08:51:11.063099+0000","last_became_active":"2024-04-21T08:48:46.461602+0000","last_became_peered":"2024-04-21T08:48:46.461602+0000","last_unstale":"2024-04-21T08:51:11.063099+0000","last_undegraded":"2024-04-21T08:51:11.063099+0000","last_fullsized":"2024-04-21T08:51:11.063099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:52.198634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939308+0000","last_change":"2024-04-21T08:48:50.770770+0000","last_active":"2024-04-21T08:51:10.939308+0000","last_peered":"2024-04-21T08:51:10.939308+0000","last_clean":"2024-04-21T08:51:10.939308+0000","last_became_active":"2024-04-21T08:48:50.770326+0000","last_became_peered":"2024-04-21T08:48:50.770326+0000","last_unstale":"2024-04-21T08:51:10.939308+0000","last_undegraded":"2024-04-21T08:51:10.939308+0000","last_fullsized":"2024-04-21T08:51:10.939308+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:38.325155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936408+0000","last_change":"2024-04-21T08:48:45.558867+0000","last_active":"2024-04-21T08:51:10.936408+0000","last_peered":"2024-04-21T08:51:10.936408+0000","last_clean":"2024-04-21T08:51:10.936408+0000","last_became_active":"2024-04-21T08:48:45.558492+0000","last_became_peered":"2024-04-21T08:48:45.558492+0000","last_unstale":"2024-04-21T08:51:10.936408+0000","last_undegraded":"2024-04-21T08:51:10.936408+0000","last_fullsized":"2024-04-21T08:51:10.936408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:44:55.156404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.830180+0000","last_change":"2024-04-21T08:48:46.466461+0000","last_active":"2024-04-21T08:51:10.830180+0000","last_peered":"2024-04-21T08:51:10.830180+0000","last_clean":"2024-04-21T08:51:10.830180+0000","last_became_active":"2024-04-21T08:48:46.466035+0000","last_became_peered":"2024-04-21T08:48:46.466035+0000","last_unstale":"2024-04-21T08:51:10.830180+0000","last_undegraded":"2024-04-21T08:51:10.830180+0000","last_fullsized":"2024-04-21T08:51:10.830180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:47:32.307753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036555+0000","last_change":"2024-04-21T08:48:50.767655+0000","last_active":"2024-04-21T08:51:11.036555+0000","last_peered":"2024-04-21T08:51:11.036555+0000","last_clean":"2024-04-21T08:51:11.036555+0000","last_became_active":"2024-04-21T08:48:50.767305+0000","last_became_peered":"2024-04-21T08:48:50.767305+0000","last_unstale":"2024-04-21T08:51:11.036555+0000","last_undegraded":"2024-04-21T08:51:11.036555+0000","last_fullsized":"2024-04-21T08:51:11.036555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:31.210126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062757+0000","last_change":"2024-04-21T08:48:45.552952+0000","last_active":"2024-04-21T08:51:11.062757+0000","last_peered":"2024-04-21T08:51:11.062757+0000","last_clean":"2024-04-21T08:51:11.062757+0000","last_became_active":"2024-04-21T08:48:45.552556+0000","last_became_peered":"2024-04-21T08:48:45.552556+0000","last_unstale":"2024-04-21T08:51:11.062757+0000","last_undegraded":"2024-04-21T08:51:11.062757+0000","last_fullsized":"2024-04-21T08:51:11.062757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:03.428801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829859+0000","last_change":"2024-04-21T08:48:46.469020+0000","last_active":"2024-04-21T08:51:10.829859+0000","last_peered":"2024-04-21T08:51:10.829859+0000","last_clean":"2024-04-21T08:51:10.829859+0000","last_became_active":"2024-04-21T08:48:46.468631+0000","last_became_peered":"2024-04-21T08:48:46.468631+0000","last_unstale":"2024-04-21T08:51:10.829859+0000","last_undegraded":"2024-04-21T08:51:10.829859+0000","last_fullsized":"2024-04-21T08:51:10.829859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:08.559516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036969+0000","last_change":"2024-04-21T08:48:50.773942+0000","last_active":"2024-04-21T08:51:11.036969+0000","last_peered":"2024-04-21T08:51:11.036969+0000","last_clean":"2024-04-21T08:51:11.036969+0000","last_became_active":"2024-04-21T08:48:50.771876+0000","last_became_peered":"2024-04-21T08:48:50.771876+0000","last_unstale":"2024-04-21T08:51:11.036969+0000","last_undegraded":"2024-04-21T08:51:11.036969+0000","last_fullsized":"2024-04-21T08:51:11.036969+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:19.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945411+0000","last_change":"2024-04-21T08:48:46.462727+0000","last_active":"2024-04-21T08:51:10.945411+0000","last_peered":"2024-04-21T08:51:10.945411+0000","last_clean":"2024-04-21T08:51:10.945411+0000","last_became_active":"2024-04-21T08:48:46.461669+0000","last_became_peered":"2024-04-21T08:48:46.461669+0000","last_unstale":"2024-04-21T08:51:10.945411+0000","last_undegraded":"2024-04-21T08:51:10.945411+0000","last_fullsized":"2024-04-21T08:51:10.945411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:27.290790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":159,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036108+0000","last_change":"2024-04-21T08:48:45.558541+0000","last_active":"2024-04-21T08:51:11.036108+0000","last_peered":"2024-04-21T08:51:11.036108+0000","last_clean":"2024-04-21T08:51:11.036108+0000","last_became_active":"2024-04-21T08:48:45.558186+0000","last_became_peered":"2024-04-21T08:48:45.558186+0000","last_unstale":"2024-04-21T08:51:11.036108+0000","last_undegraded":"2024-04-21T08:51:11.036108+0000","last_fullsized":"2024-04-21T08:51:11.036108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:02.919144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062978+0000","last_change":"2024-04-21T08:48:50.770838+0000","last_active":"2024-04-21T08:51:11.062978+0000","last_peered":"2024-04-21T08:51:11.062978+0000","last_clean":"2024-04-21T08:51:11.062978+0000","last_became_active":"2024-04-21T08:48:50.770447+0000","last_became_peered":"2024-04-21T08:48:50.770447+0000","last_unstale":"2024-04-21T08:51:11.062978+0000","last_undegraded":"2024-04-21T08:51:11.062978+0000","last_fullsized":"2024-04-21T08:51:11.062978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:48:21.010284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828090+0000","last_change":"2024-04-21T08:48:46.478145+0000","last_active":"2024-04-21T08:51:10.828090+0000","last_peered":"2024-04-21T08:51:10.828090+0000","last_clean":"2024-04-21T08:51:10.828090+0000","last_became_active":"2024-04-21T08:48:46.477813+0000","last_became_peered":"2024-04-21T08:48:46.477813+0000","last_unstale":"2024-04-21T08:51:10.828090+0000","last_undegraded":"2024-04-21T08:51:10.828090+0000","last_fullsized":"2024-04-21T08:51:10.828090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:15.917619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038191+0000","last_change":"2024-04-21T08:48:45.545269+0000","last_active":"2024-04-21T08:51:11.038191+0000","last_peered":"2024-04-21T08:51:11.038191+0000","last_clean":"2024-04-21T08:51:11.038191+0000","last_became_active":"2024-04-21T08:48:45.544887+0000","last_became_peered":"2024-04-21T08:48:45.544887+0000","last_unstale":"2024-04-21T08:51:11.038191+0000","last_undegraded":"2024-04-21T08:51:11.038191+0000","last_fullsized":"2024-04-21T08:51:11.038191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:27.222570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":367,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062629+0000","last_change":"2024-04-21T08:47:34.198161+0000","last_active":"2024-04-21T08:51:11.062629+0000","last_peered":"2024-04-21T08:51:11.062629+0000","last_clean":"2024-04-21T08:51:11.062629+0000","last_became_active":"2024-04-21T08:47:34.197654+0000","last_became_peered":"2024-04-21T08:47:34.197654+0000","last_unstale":"2024-04-21T08:51:11.062629+0000","last_undegraded":"2024-04-21T08:51:11.062629+0000","last_fullsized":"2024-04-21T08:51:11.062629+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_clean_scrub_stamp":"2024-04-21T08:44:05.493464+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:58:58.295783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945722+0000","last_change":"2024-04-21T08:48:50.770826+0000","last_active":"2024-04-21T08:51:10.945722+0000","last_peered":"2024-04-21T08:51:10.945722+0000","last_clean":"2024-04-21T08:51:10.945722+0000","last_became_active":"2024-04-21T08:48:50.770372+0000","last_became_peered":"2024-04-21T08:48:50.770372+0000","last_unstale":"2024-04-21T08:51:10.945722+0000","last_undegraded":"2024-04-21T08:51:10.945722+0000","last_fullsized":"2024-04-21T08:51:10.945722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:59.637874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.035696+0000","last_change":"2024-04-21T08:48:46.459330+0000","last_active":"2024-04-21T08:51:11.035696+0000","last_peered":"2024-04-21T08:51:11.035696+0000","last_clean":"2024-04-21T08:51:11.035696+0000","last_became_active":"2024-04-21T08:48:46.458963+0000","last_became_peered":"2024-04-21T08:48:46.458963+0000","last_unstale":"2024-04-21T08:51:11.035696+0000","last_undegraded":"2024-04-21T08:51:11.035696+0000","last_fullsized":"2024-04-21T08:51:11.035696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:10.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062640+0000","last_change":"2024-04-21T08:49:36.917854+0000","last_active":"2024-04-21T08:51:11.062640+0000","last_peered":"2024-04-21T08:51:11.062640+0000","last_clean":"2024-04-21T08:51:11.062640+0000","last_became_active":"2024-04-21T08:49:36.915390+0000","last_became_peered":"2024-04-21T08:49:36.915390+0000","last_unstale":"2024-04-21T08:51:11.062640+0000","last_undegraded":"2024-04-21T08:51:11.062640+0000","last_fullsized":"2024-04-21T08:51:11.062640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:45.920752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062798+0000","last_change":"2024-04-21T08:48:50.765339+0000","last_active":"2024-04-21T08:51:11.062798+0000","last_peered":"2024-04-21T08:51:11.062798+0000","last_clean":"2024-04-21T08:51:11.062798+0000","last_became_active":"2024-04-21T08:48:50.764858+0000","last_became_peered":"2024-04-21T08:48:50.764858+0000","last_unstale":"2024-04-21T08:51:11.062798+0000","last_undegraded":"2024-04-21T08:51:11.062798+0000","last_fullsized":"2024-04-21T08:51:11.062798+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:26.075041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939660+0000","last_change":"2024-04-21T08:48:45.557768+0000","last_active":"2024-04-21T08:51:10.939660+0000","last_peered":"2024-04-21T08:51:10.939660+0000","last_clean":"2024-04-21T08:51:10.939660+0000","last_became_active":"2024-04-21T08:48:45.557448+0000","last_became_peered":"2024-04-21T08:48:45.557448+0000","last_unstale":"2024-04-21T08:51:10.939660+0000","last_undegraded":"2024-04-21T08:51:10.939660+0000","last_fullsized":"2024-04-21T08:51:10.939660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:09.063913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036820+0000","last_change":"2024-04-21T08:48:46.459425+0000","last_active":"2024-04-21T08:51:11.036820+0000","last_peered":"2024-04-21T08:51:11.036820+0000","last_clean":"2024-04-21T08:51:11.036820+0000","last_became_active":"2024-04-21T08:48:46.459020+0000","last_became_peered":"2024-04-21T08:48:46.459020+0000","last_unstale":"2024-04-21T08:51:11.036820+0000","last_undegraded":"2024-04-21T08:51:11.036820+0000","last_fullsized":"2024-04-21T08:51:11.036820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:08:39.042825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828005+0000","last_change":"2024-04-21T08:48:50.765740+0000","last_active":"2024-04-21T08:51:10.828005+0000","last_peered":"2024-04-21T08:51:10.828005+0000","last_clean":"2024-04-21T08:51:10.828005+0000","last_became_active":"2024-04-21T08:48:50.765359+0000","last_became_peered":"2024-04-21T08:48:50.765359+0000","last_unstale":"2024-04-21T08:51:10.828005+0000","last_undegraded":"2024-04-21T08:51:10.828005+0000","last_fullsized":"2024-04-21T08:51:10.828005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:34:42.716165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'44","reported_seq":101,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.054709+0000","last_change":"2024-04-21T08:48:45.564532+0000","last_active":"2024-04-21T08:51:11.054709+0000","last_peered":"2024-04-21T08:51:11.054709+0000","last_clean":"2024-04-21T08:51:11.054709+0000","last_became_active":"2024-04-21T08:48:45.564225+0000","last_became_peered":"2024-04-21T08:48:45.564225+0000","last_unstale":"2024-04-21T08:51:11.054709+0000","last_undegraded":"2024-04-21T08:51:11.054709+0000","last_fullsized":"2024-04-21T08:51:11.054709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:42.850371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828519+0000","last_change":"2024-04-21T08:48:46.466622+0000","last_active":"2024-04-21T08:51:10.828519+0000","last_peered":"2024-04-21T08:51:10.828519+0000","last_clean":"2024-04-21T08:51:10.828519+0000","last_became_active":"2024-04-21T08:48:46.466129+0000","last_became_peered":"2024-04-21T08:48:46.466129+0000","last_unstale":"2024-04-21T08:51:10.828519+0000","last_undegraded":"2024-04-21T08:51:10.828519+0000","last_fullsized":"2024-04-21T08:51:10.828519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:20:26.064193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832121+0000","last_change":"2024-04-21T08:48:50.759534+0000","last_active":"2024-04-21T08:51:10.832121+0000","last_peered":"2024-04-21T08:51:10.832121+0000","last_clean":"2024-04-21T08:51:10.832121+0000","last_became_active":"2024-04-21T08:48:50.759116+0000","last_became_peered":"2024-04-21T08:48:50.759116+0000","last_unstale":"2024-04-21T08:51:10.832121+0000","last_undegraded":"2024-04-21T08:51:10.832121+0000","last_fullsized":"2024-04-21T08:51:10.832121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:02:16.200901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038642+0000","last_change":"2024-04-21T08:48:45.545773+0000","last_active":"2024-04-21T08:51:11.038642+0000","last_peered":"2024-04-21T08:51:11.038642+0000","last_clean":"2024-04-21T08:51:11.038642+0000","last_became_active":"2024-04-21T08:48:45.545447+0000","last_became_peered":"2024-04-21T08:48:45.545447+0000","last_unstale":"2024-04-21T08:51:11.038642+0000","last_undegraded":"2024-04-21T08:51:11.038642+0000","last_fullsized":"2024-04-21T08:51:11.038642+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:59:42.171602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877976+0000","last_change":"2024-04-21T08:49:36.964411+0000","last_active":"2024-04-21T08:51:10.877976+0000","last_peered":"2024-04-21T08:51:10.877976+0000","last_clean":"2024-04-21T08:51:10.877976+0000","last_became_active":"2024-04-21T08:49:36.964046+0000","last_became_peered":"2024-04-21T08:49:36.964046+0000","last_unstale":"2024-04-21T08:51:10.877976+0000","last_undegraded":"2024-04-21T08:51:10.877976+0000","last_fullsized":"2024-04-21T08:51:10.877976+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:29:37.223495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038538+0000","last_change":"2024-04-21T08:50:37.351641+0000","last_active":"2024-04-21T08:51:11.038538+0000","last_peered":"2024-04-21T08:51:11.038538+0000","last_clean":"2024-04-21T08:51:11.038538+0000","last_became_active":"2024-04-21T08:50:37.351130+0000","last_became_peered":"2024-04-21T08:50:37.351130+0000","last_unstale":"2024-04-21T08:51:11.038538+0000","last_undegraded":"2024-04-21T08:51:11.038538+0000","last_fullsized":"2024-04-21T08:51:11.038538+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:56:58.989691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006155+0000","last_change":"2024-04-21T08:49:36.928993+0000","last_active":"2024-04-21T08:51:11.006155+0000","last_peered":"2024-04-21T08:51:11.006155+0000","last_clean":"2024-04-21T08:51:11.006155+0000","last_became_active":"2024-04-21T08:49:36.911247+0000","last_became_peered":"2024-04-21T08:49:36.911247+0000","last_unstale":"2024-04-21T08:51:11.006155+0000","last_undegraded":"2024-04-21T08:51:11.006155+0000","last_fullsized":"2024-04-21T08:51:11.006155+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:13.481670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939224+0000","last_change":"2024-04-21T08:48:46.465475+0000","last_active":"2024-04-21T08:51:10.939224+0000","last_peered":"2024-04-21T08:51:10.939224+0000","last_clean":"2024-04-21T08:51:10.939224+0000","last_became_active":"2024-04-21T08:48:46.465138+0000","last_became_peered":"2024-04-21T08:48:46.465138+0000","last_unstale":"2024-04-21T08:51:10.939224+0000","last_undegraded":"2024-04-21T08:51:10.939224+0000","last_fullsized":"2024-04-21T08:51:10.939224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:12:53.521547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936707+0000","last_change":"2024-04-21T08:48:50.765765+0000","last_active":"2024-04-21T08:51:10.936707+0000","last_peered":"2024-04-21T08:51:10.936707+0000","last_clean":"2024-04-21T08:51:10.936707+0000","last_became_active":"2024-04-21T08:48:50.765324+0000","last_became_peered":"2024-04-21T08:48:50.765324+0000","last_unstale":"2024-04-21T08:51:10.936707+0000","last_undegraded":"2024-04-21T08:51:10.936707+0000","last_fullsized":"2024-04-21T08:51:10.936707+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:41:37.846768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828042+0000","last_change":"2024-04-21T08:48:45.557400+0000","last_active":"2024-04-21T08:51:10.828042+0000","last_peered":"2024-04-21T08:51:10.828042+0000","last_clean":"2024-04-21T08:51:10.828042+0000","last_became_active":"2024-04-21T08:48:45.555012+0000","last_became_peered":"2024-04-21T08:48:45.555012+0000","last_unstale":"2024-04-21T08:51:10.828042+0000","last_undegraded":"2024-04-21T08:51:10.828042+0000","last_fullsized":"2024-04-21T08:51:10.828042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:00:31.254353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062791+0000","last_change":"2024-04-21T08:48:46.470042+0000","last_active":"2024-04-21T08:51:11.062791+0000","last_peered":"2024-04-21T08:51:11.062791+0000","last_clean":"2024-04-21T08:51:11.062791+0000","last_became_active":"2024-04-21T08:48:46.469716+0000","last_became_peered":"2024-04-21T08:48:46.469716+0000","last_unstale":"2024-04-21T08:51:11.062791+0000","last_undegraded":"2024-04-21T08:51:11.062791+0000","last_fullsized":"2024-04-21T08:51:11.062791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:09.934751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038592+0000","last_change":"2024-04-21T08:48:50.770889+0000","last_active":"2024-04-21T08:51:11.038592+0000","last_peered":"2024-04-21T08:51:11.038592+0000","last_clean":"2024-04-21T08:51:11.038592+0000","last_became_active":"2024-04-21T08:48:50.769980+0000","last_became_peered":"2024-04-21T08:48:50.769980+0000","last_unstale":"2024-04-21T08:51:11.038592+0000","last_undegraded":"2024-04-21T08:51:11.038592+0000","last_fullsized":"2024-04-21T08:51:11.038592+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:44.631633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832478+0000","last_change":"2024-04-21T08:48:45.555479+0000","last_active":"2024-04-21T08:51:10.832478+0000","last_peered":"2024-04-21T08:51:10.832478+0000","last_clean":"2024-04-21T08:51:10.832478+0000","last_became_active":"2024-04-21T08:48:45.555162+0000","last_became_peered":"2024-04-21T08:48:45.555162+0000","last_unstale":"2024-04-21T08:51:10.832478+0000","last_undegraded":"2024-04-21T08:51:10.832478+0000","last_fullsized":"2024-04-21T08:51:10.832478+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:15:20.262798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828417+0000","last_change":"2024-04-21T08:48:46.467962+0000","last_active":"2024-04-21T08:51:10.828417+0000","last_peered":"2024-04-21T08:51:10.828417+0000","last_clean":"2024-04-21T08:51:10.828417+0000","last_became_active":"2024-04-21T08:48:46.467561+0000","last_became_peered":"2024-04-21T08:48:46.467561+0000","last_unstale":"2024-04-21T08:51:10.828417+0000","last_undegraded":"2024-04-21T08:51:10.828417+0000","last_fullsized":"2024-04-21T08:51:10.828417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:18.262009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038487+0000","last_change":"2024-04-21T08:48:50.767678+0000","last_active":"2024-04-21T08:51:11.038487+0000","last_peered":"2024-04-21T08:51:11.038487+0000","last_clean":"2024-04-21T08:51:11.038487+0000","last_became_active":"2024-04-21T08:48:50.767309+0000","last_became_peered":"2024-04-21T08:48:50.767309+0000","last_unstale":"2024-04-21T08:51:11.038487+0000","last_undegraded":"2024-04-21T08:51:11.038487+0000","last_fullsized":"2024-04-21T08:51:11.038487+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:16:40.065043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939597+0000","last_change":"2024-04-21T08:48:45.553321+0000","last_active":"2024-04-21T08:51:10.939597+0000","last_peered":"2024-04-21T08:51:10.939597+0000","last_clean":"2024-04-21T08:51:10.939597+0000","last_became_active":"2024-04-21T08:48:45.552871+0000","last_became_peered":"2024-04-21T08:48:45.552871+0000","last_unstale":"2024-04-21T08:51:10.939597+0000","last_undegraded":"2024-04-21T08:51:10.939597+0000","last_fullsized":"2024-04-21T08:51:10.939597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:16:27.339801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062861+0000","last_change":"2024-04-21T08:48:46.473453+0000","last_active":"2024-04-21T08:51:11.062861+0000","last_peered":"2024-04-21T08:51:11.062861+0000","last_clean":"2024-04-21T08:51:11.062861+0000","last_became_active":"2024-04-21T08:48:46.472613+0000","last_became_peered":"2024-04-21T08:48:46.472613+0000","last_unstale":"2024-04-21T08:51:11.062861+0000","last_undegraded":"2024-04-21T08:51:11.062861+0000","last_fullsized":"2024-04-21T08:51:11.062861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:43:58.995997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006138+0000","last_change":"2024-04-21T08:48:50.765282+0000","last_active":"2024-04-21T08:51:11.006138+0000","last_peered":"2024-04-21T08:51:11.006138+0000","last_clean":"2024-04-21T08:51:11.006138+0000","last_became_active":"2024-04-21T08:48:50.764891+0000","last_became_peered":"2024-04-21T08:48:50.764891+0000","last_unstale":"2024-04-21T08:51:11.006138+0000","last_undegraded":"2024-04-21T08:51:11.006138+0000","last_fullsized":"2024-04-21T08:51:11.006138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:18:40.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038155+0000","last_change":"2024-04-21T08:49:36.926642+0000","last_active":"2024-04-21T08:51:11.038155+0000","last_peered":"2024-04-21T08:51:11.038155+0000","last_clean":"2024-04-21T08:51:11.038155+0000","last_became_active":"2024-04-21T08:49:36.910453+0000","last_became_peered":"2024-04-21T08:49:36.910453+0000","last_unstale":"2024-04-21T08:51:11.038155+0000","last_undegraded":"2024-04-21T08:51:11.038155+0000","last_fullsized":"2024-04-21T08:51:11.038155+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:19.290010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063219+0000","last_change":"2024-04-21T08:48:46.470058+0000","last_active":"2024-04-21T08:51:11.063219+0000","last_peered":"2024-04-21T08:51:11.063219+0000","last_clean":"2024-04-21T08:51:11.063219+0000","last_became_active":"2024-04-21T08:48:46.469485+0000","last_became_peered":"2024-04-21T08:48:46.469485+0000","last_unstale":"2024-04-21T08:51:11.063219+0000","last_undegraded":"2024-04-21T08:51:11.063219+0000","last_fullsized":"2024-04-21T08:51:11.063219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:33:13.212087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877446+0000","last_change":"2024-04-21T08:48:50.762944+0000","last_active":"2024-04-21T08:51:10.877446+0000","last_peered":"2024-04-21T08:51:10.877446+0000","last_clean":"2024-04-21T08:51:10.877446+0000","last_became_active":"2024-04-21T08:48:50.762508+0000","last_became_peered":"2024-04-21T08:48:50.762508+0000","last_unstale":"2024-04-21T08:51:10.877446+0000","last_undegraded":"2024-04-21T08:51:10.877446+0000","last_fullsized":"2024-04-21T08:51:10.877446+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:35.393339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.052626+0000","last_change":"2024-04-21T08:48:45.560361+0000","last_active":"2024-04-21T08:51:11.052626+0000","last_peered":"2024-04-21T08:51:11.052626+0000","last_clean":"2024-04-21T08:51:11.052626+0000","last_became_active":"2024-04-21T08:48:45.559946+0000","last_became_peered":"2024-04-21T08:48:45.559946+0000","last_unstale":"2024-04-21T08:51:11.052626+0000","last_undegraded":"2024-04-21T08:51:11.052626+0000","last_fullsized":"2024-04-21T08:51:11.052626+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:55.599714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063003+0000","last_change":"2024-04-21T08:48:46.472706+0000","last_active":"2024-04-21T08:51:11.063003+0000","last_peered":"2024-04-21T08:51:11.063003+0000","last_clean":"2024-04-21T08:51:11.063003+0000","last_became_active":"2024-04-21T08:48:46.472378+0000","last_became_peered":"2024-04-21T08:48:46.472378+0000","last_unstale":"2024-04-21T08:51:11.063003+0000","last_undegraded":"2024-04-21T08:51:11.063003+0000","last_fullsized":"2024-04-21T08:51:11.063003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:58.228899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945191+0000","last_change":"2024-04-21T08:48:50.760610+0000","last_active":"2024-04-21T08:51:10.945191+0000","last_peered":"2024-04-21T08:51:10.945191+0000","last_clean":"2024-04-21T08:51:10.945191+0000","last_became_active":"2024-04-21T08:48:50.759991+0000","last_became_peered":"2024-04-21T08:48:50.759991+0000","last_unstale":"2024-04-21T08:51:10.945191+0000","last_undegraded":"2024-04-21T08:51:10.945191+0000","last_fullsized":"2024-04-21T08:51:10.945191+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:43.649701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036942+0000","last_change":"2024-04-21T08:48:45.560773+0000","last_active":"2024-04-21T08:51:11.036942+0000","last_peered":"2024-04-21T08:51:11.036942+0000","last_clean":"2024-04-21T08:51:11.036942+0000","last_became_active":"2024-04-21T08:48:45.560431+0000","last_became_peered":"2024-04-21T08:48:45.560431+0000","last_unstale":"2024-04-21T08:51:11.036942+0000","last_undegraded":"2024-04-21T08:51:11.036942+0000","last_fullsized":"2024-04-21T08:51:11.036942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:07.858838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063259+0000","last_change":"2024-04-21T08:48:46.475983+0000","last_active":"2024-04-21T08:51:11.063259+0000","last_peered":"2024-04-21T08:51:11.063259+0000","last_clean":"2024-04-21T08:51:11.063259+0000","last_became_active":"2024-04-21T08:48:46.475667+0000","last_became_peered":"2024-04-21T08:48:46.475667+0000","last_unstale":"2024-04-21T08:51:11.063259+0000","last_undegraded":"2024-04-21T08:51:11.063259+0000","last_fullsized":"2024-04-21T08:51:11.063259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:28:41.013111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062960+0000","last_change":"2024-04-21T08:48:46.464675+0000","last_active":"2024-04-21T08:51:11.062960+0000","last_peered":"2024-04-21T08:51:11.062960+0000","last_clean":"2024-04-21T08:51:11.062960+0000","last_became_active":"2024-04-21T08:48:46.464314+0000","last_became_peered":"2024-04-21T08:48:46.464314+0000","last_unstale":"2024-04-21T08:51:11.062960+0000","last_undegraded":"2024-04-21T08:51:11.062960+0000","last_fullsized":"2024-04-21T08:51:11.062960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:53.048400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832307+0000","last_change":"2024-04-21T08:49:36.911025+0000","last_active":"2024-04-21T08:51:10.832307+0000","last_peered":"2024-04-21T08:51:10.832307+0000","last_clean":"2024-04-21T08:51:10.832307+0000","last_became_active":"2024-04-21T08:49:36.909942+0000","last_became_peered":"2024-04-21T08:49:36.909942+0000","last_unstale":"2024-04-21T08:51:10.832307+0000","last_undegraded":"2024-04-21T08:51:10.832307+0000","last_fullsized":"2024-04-21T08:51:10.832307+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:39.864253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832375+0000","last_change":"2024-04-21T08:48:46.467867+0000","last_active":"2024-04-21T08:51:10.832375+0000","last_peered":"2024-04-21T08:51:10.832375+0000","last_clean":"2024-04-21T08:51:10.832375+0000","last_became_active":"2024-04-21T08:48:46.465917+0000","last_became_peered":"2024-04-21T08:48:46.465917+0000","last_unstale":"2024-04-21T08:51:10.832375+0000","last_undegraded":"2024-04-21T08:51:10.832375+0000","last_fullsized":"2024-04-21T08:51:10.832375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:19:56.824998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877556+0000","last_change":"2024-04-21T08:48:46.467159+0000","last_active":"2024-04-21T08:51:10.877556+0000","last_peered":"2024-04-21T08:51:10.877556+0000","last_clean":"2024-04-21T08:51:10.877556+0000","last_became_active":"2024-04-21T08:48:46.466639+0000","last_became_peered":"2024-04-21T08:48:46.466639+0000","last_unstale":"2024-04-21T08:51:10.877556+0000","last_undegraded":"2024-04-21T08:51:10.877556+0000","last_fullsized":"2024-04-21T08:51:10.877556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:54:44.772535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036982+0000","last_change":"2024-04-21T08:48:46.466403+0000","last_active":"2024-04-21T08:51:11.036982+0000","last_peered":"2024-04-21T08:51:11.036982+0000","last_clean":"2024-04-21T08:51:11.036982+0000","last_became_active":"2024-04-21T08:48:46.466060+0000","last_became_peered":"2024-04-21T08:48:46.466060+0000","last_unstale":"2024-04-21T08:51:11.036982+0000","last_undegraded":"2024-04-21T08:51:11.036982+0000","last_fullsized":"2024-04-21T08:51:11.036982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:14.374193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063096+0000","last_change":"2024-04-21T08:49:36.909134+0000","last_active":"2024-04-21T08:51:11.063096+0000","last_peered":"2024-04-21T08:51:11.063096+0000","last_clean":"2024-04-21T08:51:11.063096+0000","last_became_active":"2024-04-21T08:49:36.908723+0000","last_became_peered":"2024-04-21T08:49:36.908723+0000","last_unstale":"2024-04-21T08:51:11.063096+0000","last_undegraded":"2024-04-21T08:51:11.063096+0000","last_fullsized":"2024-04-21T08:51:11.063096+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:28.123016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832933+0000","last_change":"2024-04-21T08:48:46.464098+0000","last_active":"2024-04-21T08:51:10.832933+0000","last_peered":"2024-04-21T08:51:10.832933+0000","last_clean":"2024-04-21T08:51:10.832933+0000","last_became_active":"2024-04-21T08:48:46.463756+0000","last_became_peered":"2024-04-21T08:48:46.463756+0000","last_unstale":"2024-04-21T08:51:10.832933+0000","last_undegraded":"2024-04-21T08:51:10.832933+0000","last_fullsized":"2024-04-21T08:51:10.832933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:31.725155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828666+0000","last_change":"2024-04-21T08:48:46.467372+0000","last_active":"2024-04-21T08:51:10.828666+0000","last_peered":"2024-04-21T08:51:10.828666+0000","last_clean":"2024-04-21T08:51:10.828666+0000","last_became_active":"2024-04-21T08:48:46.467043+0000","last_became_peered":"2024-04-21T08:48:46.467043+0000","last_unstale":"2024-04-21T08:51:10.828666+0000","last_undegraded":"2024-04-21T08:51:10.828666+0000","last_fullsized":"2024-04-21T08:51:10.828666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:41.889994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945316+0000","last_change":"2024-04-21T08:48:46.460604+0000","last_active":"2024-04-21T08:51:10.945316+0000","last_peered":"2024-04-21T08:51:10.945316+0000","last_clean":"2024-04-21T08:51:10.945316+0000","last_became_active":"2024-04-21T08:48:46.460276+0000","last_became_peered":"2024-04-21T08:48:46.460276+0000","last_unstale":"2024-04-21T08:51:10.945316+0000","last_undegraded":"2024-04-21T08:51:10.945316+0000","last_fullsized":"2024-04-21T08:51:10.945316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:30.194808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828076+0000","last_change":"2024-04-21T08:48:46.463665+0000","last_active":"2024-04-21T08:51:10.828076+0000","last_peered":"2024-04-21T08:51:10.828076+0000","last_clean":"2024-04-21T08:51:10.828076+0000","last_became_active":"2024-04-21T08:48:46.463309+0000","last_became_peered":"2024-04-21T08:48:46.463309+0000","last_unstale":"2024-04-21T08:51:10.828076+0000","last_undegraded":"2024-04-21T08:51:10.828076+0000","last_fullsized":"2024-04-21T08:51:10.828076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:12:36.436565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006407+0000","last_change":"2024-04-21T08:48:46.476845+0000","last_active":"2024-04-21T08:51:11.006407+0000","last_peered":"2024-04-21T08:51:11.006407+0000","last_clean":"2024-04-21T08:51:11.006407+0000","last_became_active":"2024-04-21T08:48:46.476507+0000","last_became_peered":"2024-04-21T08:48:46.476507+0000","last_unstale":"2024-04-21T08:51:11.006407+0000","last_undegraded":"2024-04-21T08:51:11.006407+0000","last_fullsized":"2024-04-21T08:51:11.006407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:47.424576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036615+0000","last_change":"2024-04-21T08:48:46.469316+0000","last_active":"2024-04-21T08:51:11.036615+0000","last_peered":"2024-04-21T08:51:11.036615+0000","last_clean":"2024-04-21T08:51:11.036615+0000","last_became_active":"2024-04-21T08:48:46.468852+0000","last_became_peered":"2024-04-21T08:48:46.468852+0000","last_unstale":"2024-04-21T08:51:11.036615+0000","last_undegraded":"2024-04-21T08:51:11.036615+0000","last_fullsized":"2024-04-21T08:51:11.036615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:05:03.664279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062922+0000","last_change":"2024-04-21T08:48:46.466703+0000","last_active":"2024-04-21T08:51:11.062922+0000","last_peered":"2024-04-21T08:51:11.062922+0000","last_clean":"2024-04-21T08:51:11.062922+0000","last_became_active":"2024-04-21T08:48:46.466395+0000","last_became_peered":"2024-04-21T08:48:46.466395+0000","last_unstale":"2024-04-21T08:51:11.062922+0000","last_undegraded":"2024-04-21T08:51:11.062922+0000","last_fullsized":"2024-04-21T08:51:11.062922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:08:56.956283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108027,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599448,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":835,"num_read_kb":523,"num_write":290,"num_write_kb":1443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1798344,"data_compressed":80739,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":529689,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32340,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712812,"statfs":{"total":95995035648,"available":95961919488,"internally_reserved":0,"allocated":1187840,"data_stored":1247346,"data_compressed":70096,"data_compressed_allocated":94208,"data_compressed_original":741376,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":927125,"data_compressed":48725,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":529621,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1097728,"data_stored":1127264,"data_compressed":65588,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1015808,"data_stored":515086,"data_compressed":26578,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":593908,"data_compressed":31190,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":526377,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1118208,"data_stored":595766,"data_compressed":31190,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1049999,"data_compressed":54881,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":531144,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32276,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712876,"statfs":{"total":95995035648,"available":95961985024,"internally_reserved":0,"allocated":1126400,"data_stored":1253529,"data_compressed":71744,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":69009,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":70867,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134166,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":393842,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":203110,"data_compressed":7767,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:51:16.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:51:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.823+0000 7fb7477b6640 1 -- 172.21.15.28:0/95149348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740102b50 msgr2=0x7fb740106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.823+0000 7fb7477b6640 1 --2- 172.21.15.28:0/95149348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740102b50 0x7fb740106b90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb730009900 tx=0x7fb73002bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.823+0000 7fb7477b6640 1 -- 172.21.15.28:0/95149348 shutdown_connections 2024-04-21T08:51:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.823+0000 7fb7477b6640 1 --2- 172.21.15.28:0/95149348 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb740107270 0x7fb74010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.823+0000 7fb7477b6640 1 --2- 172.21.15.28:0/95149348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740102b50 0x7fb740106b90 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.823+0000 7fb7477b6640 1 --2- 172.21.15.28:0/95149348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7401021a0 0x7fb740102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.823+0000 7fb7477b6640 1 -- 172.21.15.28:0/95149348 >> 172.21.15.28:0/95149348 conn(0x7fb7400793a0 msgr2=0x7fb7400797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.824+0000 7fb7477b6640 1 -- 172.21.15.28:0/95149348 shutdown_connections 2024-04-21T08:51:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.824+0000 7fb7477b6640 1 -- 172.21.15.28:0/95149348 wait complete. 2024-04-21T08:51:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.824+0000 7fb7477b6640 1 Processor -- start 2024-04-21T08:51:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.824+0000 7fb7477b6640 1 -- start start 2024-04-21T08:51:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.824+0000 7fb7477b6640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7401021a0 0x7fb740074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.824+0000 7fb7477b6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb740107270 0x7fb740074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.824+0000 7fb7477b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740074cd0 0x7fb7401a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.824+0000 7fb7477b6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7401121b0 con 0x7fb740074cd0 2024-04-21T08:51:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.824+0000 7fb7477b6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb740112030 con 0x7fb740107270 2024-04-21T08:51:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.824+0000 7fb7477b6640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb740112330 con 0x7fb7401021a0 2024-04-21T08:51:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.825+0000 7fb745d53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740074cd0 0x7fb7401a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.825+0000 7fb745d53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740074cd0 0x7fb7401a68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51992/0 (socket says 172.21.15.28:51992) 2024-04-21T08:51:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.825+0000 7fb745d53640 1 -- 172.21.15.28:0/2520529115 learned_addr learned my addr 172.21.15.28:0/2520529115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.825+0000 7fb745552640 1 --2- 172.21.15.28:0/2520529115 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7401021a0 0x7fb740074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.825+0000 7fb745d53640 1 -- 172.21.15.28:0/2520529115 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7401021a0 msgr2=0x7fb740074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.825+0000 7fb745d53640 1 --2- 172.21.15.28:0/2520529115 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7401021a0 0x7fb740074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.825+0000 7fb745d53640 1 -- 172.21.15.28:0/2520529115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb740107270 msgr2=0x7fb740074790 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:51:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.825+0000 7fb745d53640 1 --2- 172.21.15.28:0/2520529115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb740107270 0x7fb740074790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.825+0000 7fb745d53640 1 -- 172.21.15.28:0/2520529115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7401a6fe0 con 0x7fb740074cd0 2024-04-21T08:51:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.825+0000 7fb745d53640 1 --2- 172.21.15.28:0/2520529115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740074cd0 0x7fb7401a68e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fb73800eaa0 tx=0x7fb73800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.825+0000 7fb7367fc640 1 -- 172.21.15.28:0/2520529115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73800cdb0 con 0x7fb740074cd0 2024-04-21T08:51:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.825+0000 7fb7367fc640 1 -- 172.21.15.28:0/2520529115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb738005020 con 0x7fb740074cd0 2024-04-21T08:51:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.825+0000 7fb7367fc640 1 -- 172.21.15.28:0/2520529115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7380116d0 con 0x7fb740074cd0 2024-04-21T08:51:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.825+0000 7fb7477b6640 1 -- 172.21.15.28:0/2520529115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7401a72d0 con 0x7fb740074cd0 2024-04-21T08:51:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.825+0000 7fb7477b6640 1 -- 172.21.15.28:0/2520529115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7401a7870 con 0x7fb740074cd0 2024-04-21T08:51:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.826+0000 7fb7477b6640 1 -- 172.21.15.28:0/2520529115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb740069230 con 0x7fb740074cd0 2024-04-21T08:51:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.827+0000 7fb7367fc640 1 -- 172.21.15.28:0/2520529115 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fb7380051d0 con 0x7fb740074cd0 2024-04-21T08:51:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.828+0000 7fb7367fc640 1 --2- 172.21.15.28:0/2520529115 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb72c0a3ef0 0x7fb72c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.828+0000 7fb7367fc640 1 -- 172.21.15.28:0/2520529115 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fb7380d0f70 con 0x7fb740074cd0 2024-04-21T08:51:16.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.828+0000 7fb745552640 1 --2- 172.21.15.28:0/2520529115 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb72c0a3ef0 0x7fb72c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:16.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.828+0000 7fb745552640 1 --2- 172.21.15.28:0/2520529115 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb72c0a3ef0 0x7fb72c0a63b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb73c005fd0 tx=0x7fb73c005850 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:16.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:16.832+0000 7fb7367fc640 1 -- 172.21.15.28:0/2520529115 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb738016030 con 0x7fb740074cd0 2024-04-21T08:51:17.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.021+0000 7fb7477b6640 1 -- 172.21.15.28:0/2520529115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb740102b50 con 0x7fb740074cd0 2024-04-21T08:51:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.023+0000 7fb7367fc640 1 -- 172.21.15.28:0/2520529115 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25160 (secure 0 0 0) 0x7fb73809d020 con 0x7fb740074cd0 2024-04-21T08:51:17.023 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:17.023 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:51:10.823421+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1076282159":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6800/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/926389767":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/892059008":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/2132574390":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:51:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.025+0000 7fb7477b6640 1 -- 172.21.15.28:0/2520529115 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb72c0a3ef0 msgr2=0x7fb72c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.025+0000 7fb7477b6640 1 --2- 172.21.15.28:0/2520529115 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb72c0a3ef0 0x7fb72c0a63b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb73c005fd0 tx=0x7fb73c005850 comp rx=0 tx=0).stop 2024-04-21T08:51:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.025+0000 7fb7477b6640 1 -- 172.21.15.28:0/2520529115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740074cd0 msgr2=0x7fb7401a68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.025+0000 7fb7477b6640 1 --2- 172.21.15.28:0/2520529115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740074cd0 0x7fb7401a68e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fb73800eaa0 tx=0x7fb73800ef60 comp rx=0 tx=0).stop 2024-04-21T08:51:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.025+0000 7fb7477b6640 1 -- 172.21.15.28:0/2520529115 shutdown_connections 2024-04-21T08:51:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.025+0000 7fb7477b6640 1 --2- 172.21.15.28:0/2520529115 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb72c0a3ef0 0x7fb72c0a63b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.025+0000 7fb7477b6640 1 --2- 172.21.15.28:0/2520529115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb740074cd0 0x7fb7401a68e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.025+0000 7fb7477b6640 1 --2- 172.21.15.28:0/2520529115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb740107270 0x7fb740074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.025+0000 7fb7477b6640 1 --2- 172.21.15.28:0/2520529115 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7401021a0 0x7fb740074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.025+0000 7fb7477b6640 1 -- 172.21.15.28:0/2520529115 >> 172.21.15.28:0/2520529115 conn(0x7fb7400793a0 msgr2=0x7fb7400fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.025+0000 7fb7477b6640 1 -- 172.21.15.28:0/2520529115 shutdown_connections 2024-04-21T08:51:17.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.025+0000 7fb7477b6640 1 -- 172.21.15.28:0/2520529115 wait complete. 2024-04-21T08:51:17.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:16 smithi028 ceph-mon[28003]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:17.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:16 smithi028 ceph-mon[28003]: from='client.25169 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:17.036 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:48:43.543799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:48:44.864564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:48:48.747842+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:51:17.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:51:17.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:16 smithi152 ceph-mon[31401]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:17.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:16 smithi152 ceph-mon[31401]: from='client.25169 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:16 smithi171 ceph-mon[31358]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:17.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:16 smithi171 ceph-mon[31358]: from='client.25169 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:17.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.141+0000 7fe5b1f59640 1 -- 172.21.15.28:0/3183224868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ac107260 msgr2=0x7fe5ac10ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.141+0000 7fe5b1f59640 1 --2- 172.21.15.28:0/3183224868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ac107260 0x7fe5ac10ee30 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4009900 tx=0x7fe5a402bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.141+0000 7fe5b1f59640 1 -- 172.21.15.28:0/3183224868 shutdown_connections 2024-04-21T08:51:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.141+0000 7fe5b1f59640 1 --2- 172.21.15.28:0/3183224868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ac107260 0x7fe5ac10ee30 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.141+0000 7fe5b1f59640 1 --2- 172.21.15.28:0/3183224868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5ac102b40 0x7fe5ac106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.141+0000 7fe5b1f59640 1 --2- 172.21.15.28:0/3183224868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5ac102190 0x7fe5ac102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.141+0000 7fe5b1f59640 1 -- 172.21.15.28:0/3183224868 >> 172.21.15.28:0/3183224868 conn(0x7fe5ac0793a0 msgr2=0x7fe5ac0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.142+0000 7fe5b1f59640 1 -- 172.21.15.28:0/3183224868 shutdown_connections 2024-04-21T08:51:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.142+0000 7fe5b1f59640 1 -- 172.21.15.28:0/3183224868 wait complete. 2024-04-21T08:51:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.142+0000 7fe5b1f59640 1 Processor -- start 2024-04-21T08:51:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.142+0000 7fe5b1f59640 1 -- start start 2024-04-21T08:51:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.142+0000 7fe5b1f59640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5ac102190 0x7fe5ac074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.142+0000 7fe5b1f59640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5ac107260 0x7fe5ac074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5b1f59640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ac074cd0 0x7fe5ac1a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5b1f59640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac112220 con 0x7fe5ac074cd0 2024-04-21T08:51:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5b1f59640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac1120a0 con 0x7fe5ac107260 2024-04-21T08:51:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5b1f59640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac1123a0 con 0x7fe5ac102190 2024-04-21T08:51:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ac074cd0 0x7fe5ac1a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ac074cd0 0x7fe5ac1a68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51998/0 (socket says 172.21.15.28:51998) 2024-04-21T08:51:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5abfff640 1 -- 172.21.15.28:0/2108744822 learned_addr learned my addr 172.21.15.28:0/2108744822 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5ab7fe640 1 --2- 172.21.15.28:0/2108744822 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5ac102190 0x7fe5ac074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5aaffd640 1 --2- 172.21.15.28:0/2108744822 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5ac107260 0x7fe5ac074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5abfff640 1 -- 172.21.15.28:0/2108744822 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5ac102190 msgr2=0x7fe5ac074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5abfff640 1 --2- 172.21.15.28:0/2108744822 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5ac102190 0x7fe5ac074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5abfff640 1 -- 172.21.15.28:0/2108744822 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5ac107260 msgr2=0x7fe5ac074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5abfff640 1 --2- 172.21.15.28:0/2108744822 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5ac107260 0x7fe5ac074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5abfff640 1 -- 172.21.15.28:0/2108744822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ac1a6ff0 con 0x7fe5ac074cd0 2024-04-21T08:51:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5abfff640 1 --2- 172.21.15.28:0/2108744822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ac074cd0 0x7fe5ac1a68f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe5a402c170 tx=0x7fe5a40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5a8ff9640 1 -- 172.21.15.28:0/2108744822 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4043020 con 0x7fe5ac074cd0 2024-04-21T08:51:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5a8ff9640 1 -- 172.21.15.28:0/2108744822 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5a402cbb0 con 0x7fe5ac074cd0 2024-04-21T08:51:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5b1f59640 1 -- 172.21.15.28:0/2108744822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5ac1a7280 con 0x7fe5ac074cd0 2024-04-21T08:51:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5a8ff9640 1 -- 172.21.15.28:0/2108744822 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4003ab0 con 0x7fe5ac074cd0 2024-04-21T08:51:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.143+0000 7fe5b1f59640 1 -- 172.21.15.28:0/2108744822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5ac1a77d0 con 0x7fe5ac074cd0 2024-04-21T08:51:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.144+0000 7fe5b1f59640 1 -- 172.21.15.28:0/2108744822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5ac0691c0 con 0x7fe5ac074cd0 2024-04-21T08:51:17.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.146+0000 7fe5a8ff9640 1 -- 172.21.15.28:0/2108744822 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fe5a4046020 con 0x7fe5ac074cd0 2024-04-21T08:51:17.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.147+0000 7fe5a8ff9640 1 --2- 172.21.15.28:0/2108744822 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe5940a3e70 0x7fe5940a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:17.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.147+0000 7fe5ab7fe640 1 --2- 172.21.15.28:0/2108744822 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe5940a3e70 0x7fe5940a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:17.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.147+0000 7fe5a8ff9640 1 -- 172.21.15.28:0/2108744822 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe5a40f3450 con 0x7fe5ac074cd0 2024-04-21T08:51:17.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.147+0000 7fe5ab7fe640 1 --2- 172.21.15.28:0/2108744822 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe5940a3e70 0x7fe5940a6330 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe59c005fd0 tx=0x7fe59c0098b0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:17.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.150+0000 7fe5a8ff9640 1 -- 172.21.15.28:0/2108744822 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe5a404d020 con 0x7fe5ac074cd0 2024-04-21T08:51:17.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.336+0000 7fe5b1f59640 1 -- 172.21.15.28:0/2108744822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe5ac10a490 con 0x7fe5ac074cd0 2024-04-21T08:51:17.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.337+0000 7fe5a8ff9640 1 -- 172.21.15.28:0/2108744822 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fe5a40bf610 con 0x7fe5ac074cd0 2024-04-21T08:51:17.336 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T08:51:17.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.339+0000 7fe5b1f59640 1 -- 172.21.15.28:0/2108744822 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe5940a3e70 msgr2=0x7fe5940a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.339+0000 7fe5b1f59640 1 --2- 172.21.15.28:0/2108744822 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe5940a3e70 0x7fe5940a6330 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe59c005fd0 tx=0x7fe59c0098b0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.339+0000 7fe5b1f59640 1 -- 172.21.15.28:0/2108744822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ac074cd0 msgr2=0x7fe5ac1a68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.339+0000 7fe5b1f59640 1 --2- 172.21.15.28:0/2108744822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ac074cd0 0x7fe5ac1a68f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe5a402c170 tx=0x7fe5a40045e0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.339+0000 7fe5b1f59640 1 -- 172.21.15.28:0/2108744822 shutdown_connections 2024-04-21T08:51:17.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.339+0000 7fe5b1f59640 1 --2- 172.21.15.28:0/2108744822 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe5940a3e70 0x7fe5940a6330 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.339+0000 7fe5b1f59640 1 --2- 172.21.15.28:0/2108744822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5ac074cd0 0x7fe5ac1a68f0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.339+0000 7fe5b1f59640 1 --2- 172.21.15.28:0/2108744822 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5ac107260 0x7fe5ac074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.339+0000 7fe5b1f59640 1 --2- 172.21.15.28:0/2108744822 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5ac102190 0x7fe5ac074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.339+0000 7fe5b1f59640 1 -- 172.21.15.28:0/2108744822 >> 172.21.15.28:0/2108744822 conn(0x7fe5ac0793a0 msgr2=0x7fe5ac0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:17.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.339+0000 7fe5b1f59640 1 -- 172.21.15.28:0/2108744822 shutdown_connections 2024-04-21T08:51:17.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.339+0000 7fe5b1f59640 1 -- 172.21.15.28:0/2108744822 wait complete. 2024-04-21T08:51:17.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:51:17.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.455+0000 7fc5697ce640 1 -- 172.21.15.28:0/412533904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc564102b40 msgr2=0x7fc564106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.455+0000 7fc5697ce640 1 --2- 172.21.15.28:0/412533904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc564102b40 0x7fc564106b80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc558009900 tx=0x7fc55802bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:17.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.455+0000 7fc5697ce640 1 -- 172.21.15.28:0/412533904 shutdown_connections 2024-04-21T08:51:17.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.455+0000 7fc5697ce640 1 --2- 172.21.15.28:0/412533904 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc564107260 0x7fc56410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.455+0000 7fc5697ce640 1 --2- 172.21.15.28:0/412533904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc564102b40 0x7fc564106b80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.455+0000 7fc5697ce640 1 --2- 172.21.15.28:0/412533904 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc564102190 0x7fc564102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.455+0000 7fc5697ce640 1 -- 172.21.15.28:0/412533904 >> 172.21.15.28:0/412533904 conn(0x7fc5640793a0 msgr2=0x7fc5640797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:17.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.455+0000 7fc5697ce640 1 -- 172.21.15.28:0/412533904 shutdown_connections 2024-04-21T08:51:17.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.455+0000 7fc5697ce640 1 -- 172.21.15.28:0/412533904 wait complete. 2024-04-21T08:51:17.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.456+0000 7fc5697ce640 1 Processor -- start 2024-04-21T08:51:17.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.456+0000 7fc5697ce640 1 -- start start 2024-04-21T08:51:17.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.456+0000 7fc5697ce640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc564102190 0x7fc564198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:17.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.456+0000 7fc5697ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc564107260 0x7fc564198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:17.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.456+0000 7fc5697ce640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc56419d010 0x7fc56419d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:17.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.456+0000 7fc5697ce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5641121f0 con 0x7fc564107260 2024-04-21T08:51:17.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.456+0000 7fc5697ce640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc564112070 con 0x7fc56419d010 2024-04-21T08:51:17.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.456+0000 7fc5697ce640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc564112370 con 0x7fc564102190 2024-04-21T08:51:17.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.456+0000 7fc5627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc564107260 0x7fc564198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:17.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.456+0000 7fc5627fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc564107260 0x7fc564198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52014/0 (socket says 172.21.15.28:52014) 2024-04-21T08:51:17.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.456+0000 7fc5627fc640 1 -- 172.21.15.28:0/4063558082 learned_addr learned my addr 172.21.15.28:0/4063558082 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:17.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.456+0000 7fc562ffd640 1 --2- 172.21.15.28:0/4063558082 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc564102190 0x7fc564198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.456+0000 7fc5627fc640 1 -- 172.21.15.28:0/4063558082 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc564102190 msgr2=0x7fc564198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.456+0000 7fc5627fc640 1 --2- 172.21.15.28:0/4063558082 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc564102190 0x7fc564198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.457+0000 7fc5627fc640 1 -- 172.21.15.28:0/4063558082 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc56419d010 msgr2=0x7fc56419d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.457+0000 7fc5627fc640 1 --2- 172.21.15.28:0/4063558082 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc56419d010 0x7fc56419d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.457+0000 7fc5627fc640 1 -- 172.21.15.28:0/4063558082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56419db00 con 0x7fc564107260 2024-04-21T08:51:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.457+0000 7fc562ffd640 1 --2- 172.21.15.28:0/4063558082 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc564102190 0x7fc564198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.457+0000 7fc5627fc640 1 --2- 172.21.15.28:0/4063558082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc564107260 0x7fc564198c90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc558000c00 tx=0x7fc55802c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.457+0000 7fc54bfff640 1 -- 172.21.15.28:0/4063558082 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc558043020 con 0x7fc564107260 2024-04-21T08:51:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.457+0000 7fc5697ce640 1 -- 172.21.15.28:0/4063558082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc56419dd90 con 0x7fc564107260 2024-04-21T08:51:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.457+0000 7fc54bfff640 1 -- 172.21.15.28:0/4063558082 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc55802cc60 con 0x7fc564107260 2024-04-21T08:51:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.457+0000 7fc54bfff640 1 -- 172.21.15.28:0/4063558082 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc558003c30 con 0x7fc564107260 2024-04-21T08:51:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.457+0000 7fc5697ce640 1 -- 172.21.15.28:0/4063558082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5641a55b0 con 0x7fc564107260 2024-04-21T08:51:17.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.458+0000 7fc5697ce640 1 -- 172.21.15.28:0/4063558082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5640691c0 con 0x7fc564107260 2024-04-21T08:51:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.460+0000 7fc54bfff640 1 -- 172.21.15.28:0/4063558082 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fc558046020 con 0x7fc564107260 2024-04-21T08:51:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.460+0000 7fc54bfff640 1 --2- 172.21.15.28:0/4063558082 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc54c0a3e40 0x7fc54c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.461+0000 7fc562ffd640 1 --2- 172.21.15.28:0/4063558082 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc54c0a3e40 0x7fc54c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:17.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.461+0000 7fc54bfff640 1 -- 172.21.15.28:0/4063558082 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fc5580f3440 con 0x7fc564107260 2024-04-21T08:51:17.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.461+0000 7fc562ffd640 1 --2- 172.21.15.28:0/4063558082 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc54c0a3e40 0x7fc54c0a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc5540058c0 tx=0x7fc554005850 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:17.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.464+0000 7fc54bfff640 1 -- 172.21.15.28:0/4063558082 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc55804d020 con 0x7fc564107260 2024-04-21T08:51:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.651+0000 7fc5697ce640 1 -- 172.21.15.28:0/4063558082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc564102b40 con 0x7fc564107260 2024-04-21T08:51:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.651+0000 7fc54bfff640 1 -- 172.21.15.28:0/4063558082 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fc5580473b0 con 0x7fc564107260 2024-04-21T08:51:17.651 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:51:17.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.653+0000 7fc5697ce640 1 -- 172.21.15.28:0/4063558082 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc54c0a3e40 msgr2=0x7fc54c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.653+0000 7fc5697ce640 1 --2- 172.21.15.28:0/4063558082 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc54c0a3e40 0x7fc54c0a6300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc5540058c0 tx=0x7fc554005850 comp rx=0 tx=0).stop 2024-04-21T08:51:17.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.653+0000 7fc5697ce640 1 -- 172.21.15.28:0/4063558082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc564107260 msgr2=0x7fc564198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.653+0000 7fc5697ce640 1 --2- 172.21.15.28:0/4063558082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc564107260 0x7fc564198c90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc558000c00 tx=0x7fc55802c860 comp rx=0 tx=0).stop 2024-04-21T08:51:17.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.653+0000 7fc5697ce640 1 -- 172.21.15.28:0/4063558082 shutdown_connections 2024-04-21T08:51:17.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.653+0000 7fc5697ce640 1 --2- 172.21.15.28:0/4063558082 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc54c0a3e40 0x7fc54c0a6300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.653+0000 7fc5697ce640 1 --2- 172.21.15.28:0/4063558082 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc56419d010 0x7fc56419d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.653+0000 7fc5697ce640 1 --2- 172.21.15.28:0/4063558082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc564107260 0x7fc564198c90 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.653+0000 7fc5697ce640 1 --2- 172.21.15.28:0/4063558082 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc564102190 0x7fc564198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.653+0000 7fc5697ce640 1 -- 172.21.15.28:0/4063558082 >> 172.21.15.28:0/4063558082 conn(0x7fc5640793a0 msgr2=0x7fc5640fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:17.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.653+0000 7fc5697ce640 1 -- 172.21.15.28:0/4063558082 shutdown_connections 2024-04-21T08:51:17.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.653+0000 7fc5697ce640 1 -- 172.21.15.28:0/4063558082 wait complete. 2024-04-21T08:51:17.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:51:17.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.772+0000 7fd9916e0640 1 -- 172.21.15.28:0/3020875214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd98c101a40 msgr2=0x7fd98c105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.772+0000 7fd9916e0640 1 --2- 172.21.15.28:0/3020875214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd98c101a40 0x7fd98c105a80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd974009900 tx=0x7fd97402bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:17.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.772+0000 7fd9916e0640 1 -- 172.21.15.28:0/3020875214 shutdown_connections 2024-04-21T08:51:17.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.772+0000 7fd9916e0640 1 --2- 172.21.15.28:0/3020875214 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd98c106160 0x7fd98c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.772+0000 7fd9916e0640 1 --2- 172.21.15.28:0/3020875214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd98c101a40 0x7fd98c105a80 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.772+0000 7fd9916e0640 1 --2- 172.21.15.28:0/3020875214 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd98c101090 0x7fd98c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.772+0000 7fd9916e0640 1 -- 172.21.15.28:0/3020875214 >> 172.21.15.28:0/3020875214 conn(0x7fd98c077f50 msgr2=0x7fd98c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.772+0000 7fd9916e0640 1 -- 172.21.15.28:0/3020875214 shutdown_connections 2024-04-21T08:51:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.772+0000 7fd9916e0640 1 -- 172.21.15.28:0/3020875214 wait complete. 2024-04-21T08:51:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.773+0000 7fd9916e0640 1 Processor -- start 2024-04-21T08:51:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.773+0000 7fd9916e0640 1 -- start start 2024-04-21T08:51:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.773+0000 7fd9916e0640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd98c101090 0x7fd98c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.773+0000 7fd9916e0640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd98c106160 0x7fd98c1a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.773+0000 7fd9916e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd98c19a710 0x7fd98c19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.773+0000 7fd9916e0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c1110a0 con 0x7fd98c19a710 2024-04-21T08:51:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.773+0000 7fd9916e0640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c110f20 con 0x7fd98c106160 2024-04-21T08:51:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.773+0000 7fd9916e0640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c111220 con 0x7fd98c101090 2024-04-21T08:51:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.773+0000 7fd98b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd98c19a710 0x7fd98c19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.773+0000 7fd98b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd98c19a710 0x7fd98c19ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52018/0 (socket says 172.21.15.28:52018) 2024-04-21T08:51:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.773+0000 7fd98b7fe640 1 -- 172.21.15.28:0/3880471696 learned_addr learned my addr 172.21.15.28:0/3880471696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.773+0000 7fd98affd640 1 --2- 172.21.15.28:0/3880471696 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd98c101090 0x7fd98c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.773+0000 7fd983fff640 1 --2- 172.21.15.28:0/3880471696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd98c106160 0x7fd98c1a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.774+0000 7fd98b7fe640 1 -- 172.21.15.28:0/3880471696 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd98c101090 msgr2=0x7fd98c1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.774+0000 7fd98b7fe640 1 --2- 172.21.15.28:0/3880471696 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd98c101090 0x7fd98c1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.774+0000 7fd98b7fe640 1 -- 172.21.15.28:0/3880471696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd98c106160 msgr2=0x7fd98c1a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.774+0000 7fd98b7fe640 1 --2- 172.21.15.28:0/3880471696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd98c106160 0x7fd98c1a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.774+0000 7fd98b7fe640 1 -- 172.21.15.28:0/3880471696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c19b310 con 0x7fd98c19a710 2024-04-21T08:51:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.774+0000 7fd98b7fe640 1 --2- 172.21.15.28:0/3880471696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd98c19a710 0x7fd98c19ab90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd98400cce0 tx=0x7fd98400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.774+0000 7fd988ff9640 1 -- 172.21.15.28:0/3880471696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98400f450 con 0x7fd98c19a710 2024-04-21T08:51:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.774+0000 7fd9916e0640 1 -- 172.21.15.28:0/3880471696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98c1a73f0 con 0x7fd98c19a710 2024-04-21T08:51:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.774+0000 7fd988ff9640 1 -- 172.21.15.28:0/3880471696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd98400faa0 con 0x7fd98c19a710 2024-04-21T08:51:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.774+0000 7fd988ff9640 1 -- 172.21.15.28:0/3880471696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984018b70 con 0x7fd98c19a710 2024-04-21T08:51:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.774+0000 7fd9916e0640 1 -- 172.21.15.28:0/3880471696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd98c1a77a0 con 0x7fd98c19a710 2024-04-21T08:51:17.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.775+0000 7fd9916e0640 1 -- 172.21.15.28:0/3880471696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd98c102560 con 0x7fd98c19a710 2024-04-21T08:51:17.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.776+0000 7fd988ff9640 1 -- 172.21.15.28:0/3880471696 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fd984018d10 con 0x7fd98c19a710 2024-04-21T08:51:17.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.777+0000 7fd988ff9640 1 --2- 172.21.15.28:0/3880471696 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd9780a3ec0 0x7fd9780a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:17.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.777+0000 7fd98affd640 1 --2- 172.21.15.28:0/3880471696 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd9780a3ec0 0x7fd9780a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:17.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.777+0000 7fd988ff9640 1 -- 172.21.15.28:0/3880471696 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fd9840d1220 con 0x7fd98c19a710 2024-04-21T08:51:17.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.777+0000 7fd98affd640 1 --2- 172.21.15.28:0/3880471696 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd9780a3ec0 0x7fd9780a6380 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd97c005fd0 tx=0x7fd97c00b390 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:17.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.781+0000 7fd988ff9640 1 -- 172.21.15.28:0/3880471696 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd984014030 con 0x7fd98c19a710 2024-04-21T08:51:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.967+0000 7fd9916e0640 1 -- 172.21.15.28:0/3880471696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd98c101ae0 con 0x7fd98c19a710 2024-04-21T08:51:17.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.968+0000 7fd988ff9640 1 -- 172.21.15.28:0/3880471696 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fd98409d270 con 0x7fd98c19a710 2024-04-21T08:51:17.968 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-21T08:51:17.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.969+0000 7fd9916e0640 1 -- 172.21.15.28:0/3880471696 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd9780a3ec0 msgr2=0x7fd9780a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.969+0000 7fd9916e0640 1 --2- 172.21.15.28:0/3880471696 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd9780a3ec0 0x7fd9780a6380 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd97c005fd0 tx=0x7fd97c00b390 comp rx=0 tx=0).stop 2024-04-21T08:51:17.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.970+0000 7fd9916e0640 1 -- 172.21.15.28:0/3880471696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd98c19a710 msgr2=0x7fd98c19ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:17.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.970+0000 7fd9916e0640 1 --2- 172.21.15.28:0/3880471696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd98c19a710 0x7fd98c19ab90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fd98400cce0 tx=0x7fd98400a620 comp rx=0 tx=0).stop 2024-04-21T08:51:17.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.970+0000 7fd9916e0640 1 -- 172.21.15.28:0/3880471696 shutdown_connections 2024-04-21T08:51:17.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.970+0000 7fd9916e0640 1 --2- 172.21.15.28:0/3880471696 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd9780a3ec0 0x7fd9780a6380 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.970+0000 7fd9916e0640 1 --2- 172.21.15.28:0/3880471696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd98c19a710 0x7fd98c19ab90 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.970+0000 7fd9916e0640 1 --2- 172.21.15.28:0/3880471696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd98c106160 0x7fd98c1a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.970+0000 7fd9916e0640 1 --2- 172.21.15.28:0/3880471696 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd98c101090 0x7fd98c1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:17.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.970+0000 7fd9916e0640 1 -- 172.21.15.28:0/3880471696 >> 172.21.15.28:0/3880471696 conn(0x7fd98c077f50 msgr2=0x7fd98c069140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:17.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.970+0000 7fd9916e0640 1 -- 172.21.15.28:0/3880471696 shutdown_connections 2024-04-21T08:51:17.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:17.970+0000 7fd9916e0640 1 -- 172.21.15.28:0/3880471696 wait complete. 2024-04-21T08:51:17.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:17 smithi028 ceph-mon[28003]: from='client.15621 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:17.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:17 smithi028 ceph-mon[28003]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 205 B/s wr, 1 op/s 2024-04-21T08:51:17.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:17 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2520529115' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:17.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:17 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2108744822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:17.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:17 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/4063558082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:17.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:51:18.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.086+0000 7f80faeba640 1 -- 172.21.15.28:0/2289754913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f4101a50 msgr2=0x7f80f4105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.086+0000 7f80faeba640 1 --2- 172.21.15.28:0/2289754913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f4101a50 0x7f80f4105a90 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f80dc009900 tx=0x7f80dc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:18.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.086+0000 7f80faeba640 1 -- 172.21.15.28:0/2289754913 shutdown_connections 2024-04-21T08:51:18.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.086+0000 7f80faeba640 1 --2- 172.21.15.28:0/2289754913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80f4106170 0x7f80f410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.086+0000 7f80faeba640 1 --2- 172.21.15.28:0/2289754913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f4101a50 0x7f80f4105a90 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.086+0000 7f80faeba640 1 --2- 172.21.15.28:0/2289754913 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80f41010a0 0x7f80f4101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.086+0000 7f80faeba640 1 -- 172.21.15.28:0/2289754913 >> 172.21.15.28:0/2289754913 conn(0x7f80f4077eb0 msgr2=0x7f80f40782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:18.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.086+0000 7f80faeba640 1 -- 172.21.15.28:0/2289754913 shutdown_connections 2024-04-21T08:51:18.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.086+0000 7f80faeba640 1 -- 172.21.15.28:0/2289754913 wait complete. 2024-04-21T08:51:18.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80faeba640 1 Processor -- start 2024-04-21T08:51:18.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80faeba640 1 -- start start 2024-04-21T08:51:18.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80faeba640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80f41010a0 0x7f80f41a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80faeba640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80f4106170 0x7f80f41a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80faeba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f419a6a0 0x7f80f419ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80faeba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80f41121b0 con 0x7f80f419a6a0 2024-04-21T08:51:18.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80faeba640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f80f4112030 con 0x7f80f4106170 2024-04-21T08:51:18.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80faeba640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f80f4112330 con 0x7f80f41010a0 2024-04-21T08:51:18.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80f9457640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f419a6a0 0x7f80f419ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80f9457640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f419a6a0 0x7f80f419ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52034/0 (socket says 172.21.15.28:52034) 2024-04-21T08:51:18.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80f9457640 1 -- 172.21.15.28:0/443564397 learned_addr learned my addr 172.21.15.28:0/443564397 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:18.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80f8c56640 1 --2- 172.21.15.28:0/443564397 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80f41010a0 0x7f80f41a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80f3fff640 1 --2- 172.21.15.28:0/443564397 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80f4106170 0x7f80f41a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80f9457640 1 -- 172.21.15.28:0/443564397 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80f41010a0 msgr2=0x7f80f41a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80f9457640 1 --2- 172.21.15.28:0/443564397 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80f41010a0 0x7f80f41a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80f9457640 1 -- 172.21.15.28:0/443564397 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80f4106170 msgr2=0x7f80f41a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80f9457640 1 --2- 172.21.15.28:0/443564397 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80f4106170 0x7f80f41a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.087+0000 7f80f9457640 1 -- 172.21.15.28:0/443564397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80f419b2a0 con 0x7f80f419a6a0 2024-04-21T08:51:18.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.088+0000 7f80f9457640 1 --2- 172.21.15.28:0/443564397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f419a6a0 0x7f80f419ab20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00eaa0 tx=0x7f80ec00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:18.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.088+0000 7f80f1ffb640 1 -- 172.21.15.28:0/443564397 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec00cdb0 con 0x7f80f419a6a0 2024-04-21T08:51:18.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.088+0000 7f80f1ffb640 1 -- 172.21.15.28:0/443564397 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80ec005020 con 0x7f80f419a6a0 2024-04-21T08:51:18.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.088+0000 7f80faeba640 1 -- 172.21.15.28:0/443564397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80f41a7380 con 0x7f80f419a6a0 2024-04-21T08:51:18.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.088+0000 7f80f1ffb640 1 -- 172.21.15.28:0/443564397 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ec0116d0 con 0x7f80f419a6a0 2024-04-21T08:51:18.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.088+0000 7f80faeba640 1 -- 172.21.15.28:0/443564397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80f41a7730 con 0x7f80f419a6a0 2024-04-21T08:51:18.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.089+0000 7f80faeba640 1 -- 172.21.15.28:0/443564397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80f4102570 con 0x7f80f419a6a0 2024-04-21T08:51:18.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.090+0000 7f80f1ffb640 1 -- 172.21.15.28:0/443564397 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f80ec0051d0 con 0x7f80f419a6a0 2024-04-21T08:51:18.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.091+0000 7f80f1ffb640 1 --2- 172.21.15.28:0/443564397 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f80e00a3ec0 0x7f80e00a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.091+0000 7f80f8c56640 1 --2- 172.21.15.28:0/443564397 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f80e00a3ec0 0x7f80e00a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.091+0000 7f80f1ffb640 1 -- 172.21.15.28:0/443564397 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f80ec0d1060 con 0x7f80f419a6a0 2024-04-21T08:51:18.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.091+0000 7f80f8c56640 1 --2- 172.21.15.28:0/443564397 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f80e00a3ec0 0x7f80e00a6380 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f80e4005e10 tx=0x7f80e4005da0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:18.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.095+0000 7f80f1ffb640 1 -- 172.21.15.28:0/443564397 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f80ec016030 con 0x7f80f419a6a0 2024-04-21T08:51:18.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:17 smithi152 ceph-mon[31401]: from='client.15621 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:18.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:17 smithi152 ceph-mon[31401]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 205 B/s wr, 1 op/s 2024-04-21T08:51:18.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:17 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2520529115' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:18.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:17 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2108744822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:18.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:17 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/4063558082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:18.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:17 smithi171 ceph-mon[31358]: from='client.15621 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:18.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:17 smithi171 ceph-mon[31358]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 205 B/s wr, 1 op/s 2024-04-21T08:51:18.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:17 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2520529115' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:18.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:17 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2108744822' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:18.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:17 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/4063558082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:18.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.280+0000 7f80faeba640 1 -- 172.21.15.28:0/443564397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f80f4101a50 con 0x7f80f419a6a0 2024-04-21T08:51:18.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.281+0000 7f80f1ffb640 1 -- 172.21.15.28:0/443564397 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f80ec09d0b0 con 0x7f80f419a6a0 2024-04-21T08:51:18.281 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:51:18.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.283+0000 7f80faeba640 1 -- 172.21.15.28:0/443564397 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f80e00a3ec0 msgr2=0x7f80e00a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.283+0000 7f80faeba640 1 --2- 172.21.15.28:0/443564397 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f80e00a3ec0 0x7f80e00a6380 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f80e4005e10 tx=0x7f80e4005da0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.283+0000 7f80faeba640 1 -- 172.21.15.28:0/443564397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f419a6a0 msgr2=0x7f80f419ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.283+0000 7f80faeba640 1 --2- 172.21.15.28:0/443564397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f419a6a0 0x7f80f419ab20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00eaa0 tx=0x7f80ec00ef60 comp rx=0 tx=0).stop 2024-04-21T08:51:18.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.283+0000 7f80faeba640 1 -- 172.21.15.28:0/443564397 shutdown_connections 2024-04-21T08:51:18.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.283+0000 7f80faeba640 1 --2- 172.21.15.28:0/443564397 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f80e00a3ec0 0x7f80e00a6380 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.283+0000 7f80faeba640 1 --2- 172.21.15.28:0/443564397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f419a6a0 0x7f80f419ab20 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.283+0000 7f80faeba640 1 --2- 172.21.15.28:0/443564397 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f80f4106170 0x7f80f41a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.283+0000 7f80faeba640 1 --2- 172.21.15.28:0/443564397 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80f41010a0 0x7f80f41a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.283+0000 7f80faeba640 1 -- 172.21.15.28:0/443564397 >> 172.21.15.28:0/443564397 conn(0x7f80f4077eb0 msgr2=0x7f80f4100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:18.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.283+0000 7f80faeba640 1 -- 172.21.15.28:0/443564397 shutdown_connections 2024-04-21T08:51:18.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.283+0000 7f80faeba640 1 -- 172.21.15.28:0/443564397 wait complete. 2024-04-21T08:51:18.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:51:18.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.398+0000 7f1f5ded9640 1 -- 172.21.15.28:0/644501036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f58107260 msgr2=0x7f1f5810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.398+0000 7f1f5ded9640 1 --2- 172.21.15.28:0/644501036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f58107260 0x7f1f5810ee30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f1f50009ca0 tx=0x7f1f5002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.398+0000 7f1f5ded9640 1 -- 172.21.15.28:0/644501036 shutdown_connections 2024-04-21T08:51:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.398+0000 7f1f5ded9640 1 --2- 172.21.15.28:0/644501036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f58107260 0x7f1f5810ee30 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.398+0000 7f1f5ded9640 1 --2- 172.21.15.28:0/644501036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f58102b40 0x7f1f58106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.398+0000 7f1f5ded9640 1 --2- 172.21.15.28:0/644501036 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f58102190 0x7f1f58102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.398+0000 7f1f5ded9640 1 -- 172.21.15.28:0/644501036 >> 172.21.15.28:0/644501036 conn(0x7f1f580793e0 msgr2=0x7f1f580797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.398+0000 7f1f5ded9640 1 -- 172.21.15.28:0/644501036 shutdown_connections 2024-04-21T08:51:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.398+0000 7f1f5ded9640 1 -- 172.21.15.28:0/644501036 wait complete. 2024-04-21T08:51:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f5ded9640 1 Processor -- start 2024-04-21T08:51:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f5ded9640 1 -- start start 2024-04-21T08:51:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f5ded9640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f58102190 0x7f1f581a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f5ded9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f58107260 0x7f1f581a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f5ded9640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f5819a730 0x7f1f5819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f5ded9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f581121b0 con 0x7f1f58107260 2024-04-21T08:51:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f5ded9640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1f58112030 con 0x7f1f58102190 2024-04-21T08:51:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f5ded9640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1f58112330 con 0x7f1f5819a730 2024-04-21T08:51:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f58107260 0x7f1f581a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f58107260 0x7f1f581a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52056/0 (socket says 172.21.15.28:52056) 2024-04-21T08:51:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f4ffff640 1 -- 172.21.15.28:0/346528291 learned_addr learned my addr 172.21.15.28:0/346528291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f57fff640 1 --2- 172.21.15.28:0/346528291 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f5819a730 0x7f1f5819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f577fe640 1 --2- 172.21.15.28:0/346528291 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f58102190 0x7f1f581a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f4ffff640 1 -- 172.21.15.28:0/346528291 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f5819a730 msgr2=0x7f1f5819abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f4ffff640 1 --2- 172.21.15.28:0/346528291 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f5819a730 0x7f1f5819abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f4ffff640 1 -- 172.21.15.28:0/346528291 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f58102190 msgr2=0x7f1f581a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f4ffff640 1 --2- 172.21.15.28:0/346528291 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f58102190 0x7f1f581a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f4ffff640 1 -- 172.21.15.28:0/346528291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5819b330 con 0x7f1f58107260 2024-04-21T08:51:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.399+0000 7f1f577fe640 1 --2- 172.21.15.28:0/346528291 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f58102190 0x7f1f581a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.400+0000 7f1f4ffff640 1 --2- 172.21.15.28:0/346528291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f58107260 0x7f1f581a0bb0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000c970 tx=0x7f1f4000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.400+0000 7f1f557fa640 1 -- 172.21.15.28:0/346528291 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f40014070 con 0x7f1f58107260 2024-04-21T08:51:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.400+0000 7f1f557fa640 1 -- 172.21.15.28:0/346528291 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f40004440 con 0x7f1f58107260 2024-04-21T08:51:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.400+0000 7f1f5ded9640 1 -- 172.21.15.28:0/346528291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f581a7410 con 0x7f1f58107260 2024-04-21T08:51:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.400+0000 7f1f557fa640 1 -- 172.21.15.28:0/346528291 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f40003e50 con 0x7f1f58107260 2024-04-21T08:51:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.400+0000 7f1f5ded9640 1 -- 172.21.15.28:0/346528291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f581a7810 con 0x7f1f58107260 2024-04-21T08:51:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.401+0000 7f1f5ded9640 1 -- 172.21.15.28:0/346528291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f58103660 con 0x7f1f58107260 2024-04-21T08:51:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.402+0000 7f1f557fa640 1 -- 172.21.15.28:0/346528291 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f1f40017070 con 0x7f1f58107260 2024-04-21T08:51:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.403+0000 7f1f557fa640 1 --2- 172.21.15.28:0/346528291 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1f440a3fe0 0x7f1f440a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.403+0000 7f1f577fe640 1 --2- 172.21.15.28:0/346528291 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1f440a3fe0 0x7f1f440a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.403+0000 7f1f557fa640 1 -- 172.21.15.28:0/346528291 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1f400d0950 con 0x7f1f58107260 2024-04-21T08:51:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.403+0000 7f1f577fe640 1 --2- 172.21.15.28:0/346528291 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1f440a3fe0 0x7f1f440a64a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1f48005d00 tx=0x7f1f48009550 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:18.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.408+0000 7f1f557fa640 1 -- 172.21.15.28:0/346528291 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f4001e030 con 0x7f1f58107260 2024-04-21T08:51:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.637+0000 7f1f5ded9640 1 -- 172.21.15.28:0/346528291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f58069230 con 0x7f1f58107260 2024-04-21T08:51:18.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.637+0000 7f1f557fa640 1 -- 172.21.15.28:0/346528291 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7f1f4009ca00 con 0x7f1f58107260 2024-04-21T08:51:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:18.637 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:51:09.181803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:51:18.638 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-04-21T08:51:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.640+0000 7f1f5ded9640 1 -- 172.21.15.28:0/346528291 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1f440a3fe0 msgr2=0x7f1f440a64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.640+0000 7f1f5ded9640 1 --2- 172.21.15.28:0/346528291 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1f440a3fe0 0x7f1f440a64a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1f48005d00 tx=0x7f1f48009550 comp rx=0 tx=0).stop 2024-04-21T08:51:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.640+0000 7f1f5ded9640 1 -- 172.21.15.28:0/346528291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f58107260 msgr2=0x7f1f581a0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.640+0000 7f1f5ded9640 1 --2- 172.21.15.28:0/346528291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f58107260 0x7f1f581a0bb0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000c970 tx=0x7f1f4000ce30 comp rx=0 tx=0).stop 2024-04-21T08:51:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.640+0000 7f1f5ded9640 1 -- 172.21.15.28:0/346528291 shutdown_connections 2024-04-21T08:51:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.640+0000 7f1f5ded9640 1 --2- 172.21.15.28:0/346528291 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1f440a3fe0 0x7f1f440a64a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.640+0000 7f1f5ded9640 1 --2- 172.21.15.28:0/346528291 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f5819a730 0x7f1f5819abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.640+0000 7f1f5ded9640 1 --2- 172.21.15.28:0/346528291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f58107260 0x7f1f581a0bb0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.640+0000 7f1f5ded9640 1 --2- 172.21.15.28:0/346528291 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f58102190 0x7f1f581a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.640+0000 7f1f5ded9640 1 -- 172.21.15.28:0/346528291 >> 172.21.15.28:0/346528291 conn(0x7f1f580793e0 msgr2=0x7f1f580ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.640+0000 7f1f5ded9640 1 -- 172.21.15.28:0/346528291 shutdown_connections 2024-04-21T08:51:18.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.640+0000 7f1f5ded9640 1 -- 172.21.15.28:0/346528291 wait complete. 2024-04-21T08:51:18.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:51:18.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.753+0000 7f4c1bac8640 1 -- 172.21.15.28:0/3047672644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14106150 msgr2=0x7f4c1410dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.753+0000 7f4c1bac8640 1 --2- 172.21.15.28:0/3047672644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14106150 0x7f4c1410dd20 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c009900 tx=0x7f4c0c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:18.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.753+0000 7f4c1bac8640 1 -- 172.21.15.28:0/3047672644 shutdown_connections 2024-04-21T08:51:18.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.753+0000 7f4c1bac8640 1 --2- 172.21.15.28:0/3047672644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14106150 0x7f4c1410dd20 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.753+0000 7f4c1bac8640 1 --2- 172.21.15.28:0/3047672644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c141021f0 0x7f4c14105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.753+0000 7f4c1bac8640 1 --2- 172.21.15.28:0/3047672644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c14101840 0x7f4c14101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.753+0000 7f4c1bac8640 1 -- 172.21.15.28:0/3047672644 >> 172.21.15.28:0/3047672644 conn(0x7f4c14077450 msgr2=0x7f4c140fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:18.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.753+0000 7f4c1bac8640 1 -- 172.21.15.28:0/3047672644 shutdown_connections 2024-04-21T08:51:18.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.753+0000 7f4c1bac8640 1 -- 172.21.15.28:0/3047672644 wait complete. 2024-04-21T08:51:18.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.754+0000 7f4c1bac8640 1 Processor -- start 2024-04-21T08:51:18.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.754+0000 7f4c1bac8640 1 -- start start 2024-04-21T08:51:18.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.754+0000 7f4c1bac8640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c14101840 0x7f4c14197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.754+0000 7f4c1bac8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14106150 0x7f4c14197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.754+0000 7f4c1bac8640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c1419bde0 0x7f4c1419c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.754+0000 7f4c1bac8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c141110a0 con 0x7f4c14106150 2024-04-21T08:51:18.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.754+0000 7f4c1bac8640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4c14110f20 con 0x7f4c14101840 2024-04-21T08:51:18.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.754+0000 7f4c1bac8640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4c14111220 con 0x7f4c1419bde0 2024-04-21T08:51:18.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.754+0000 7f4c19063640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14106150 0x7f4c14197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.754+0000 7f4c19063640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14106150 0x7f4c14197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52074/0 (socket says 172.21.15.28:52074) 2024-04-21T08:51:18.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.754+0000 7f4c19063640 1 -- 172.21.15.28:0/3919955544 learned_addr learned my addr 172.21.15.28:0/3919955544 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:18.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.755+0000 7f4c19864640 1 --2- 172.21.15.28:0/3919955544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c14101840 0x7f4c14197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.755+0000 7f4c1a065640 1 --2- 172.21.15.28:0/3919955544 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c1419bde0 0x7f4c1419c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.755+0000 7f4c19063640 1 -- 172.21.15.28:0/3919955544 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c1419bde0 msgr2=0x7f4c1419c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.755+0000 7f4c19063640 1 --2- 172.21.15.28:0/3919955544 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c1419bde0 0x7f4c1419c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.755+0000 7f4c19063640 1 -- 172.21.15.28:0/3919955544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c14101840 msgr2=0x7f4c14197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.755+0000 7f4c19063640 1 --2- 172.21.15.28:0/3919955544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c14101840 0x7f4c14197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.755+0000 7f4c19063640 1 -- 172.21.15.28:0/3919955544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1419c8d0 con 0x7f4c14106150 2024-04-21T08:51:18.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.755+0000 7f4c19864640 1 --2- 172.21.15.28:0/3919955544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c14101840 0x7f4c14197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:18.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.755+0000 7f4c19063640 1 --2- 172.21.15.28:0/3919955544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14106150 0x7f4c14197a60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400c970 tx=0x7f4c0400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:18.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.755+0000 7f4c0affd640 1 -- 172.21.15.28:0/3919955544 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c04014070 con 0x7f4c14106150 2024-04-21T08:51:18.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.755+0000 7f4c1bac8640 1 -- 172.21.15.28:0/3919955544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c1419cbc0 con 0x7f4c14106150 2024-04-21T08:51:18.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.755+0000 7f4c0affd640 1 -- 172.21.15.28:0/3919955544 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c04004440 con 0x7f4c14106150 2024-04-21T08:51:18.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.755+0000 7f4c0affd640 1 -- 172.21.15.28:0/3919955544 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c04003e50 con 0x7f4c14106150 2024-04-21T08:51:18.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.755+0000 7f4c1bac8640 1 -- 172.21.15.28:0/3919955544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c141a43b0 con 0x7f4c14106150 2024-04-21T08:51:18.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.756+0000 7f4c1bac8640 1 -- 172.21.15.28:0/3919955544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c14198870 con 0x7f4c14106150 2024-04-21T08:51:18.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.758+0000 7f4c0affd640 1 -- 172.21.15.28:0/3919955544 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f4c04017070 con 0x7f4c14106150 2024-04-21T08:51:18.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.758+0000 7f4c0affd640 1 --2- 172.21.15.28:0/3919955544 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4c000a3fe0 0x7f4c000a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.758+0000 7f4c19864640 1 --2- 172.21.15.28:0/3919955544 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4c000a3fe0 0x7f4c000a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.758+0000 7f4c0affd640 1 -- 172.21.15.28:0/3919955544 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f4c040d1180 con 0x7f4c14106150 2024-04-21T08:51:18.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.759+0000 7f4c19864640 1 --2- 172.21.15.28:0/3919955544 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4c000a3fe0 0x7f4c000a64a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4c10005fd0 tx=0x7f4c10009520 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.762+0000 7f4c0affd640 1 -- 172.21.15.28:0/3919955544 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c0401e030 con 0x7f4c14106150 2024-04-21T08:51:18.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.950+0000 7f4c1bac8640 1 -- 172.21.15.28:0/3919955544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c141093e0 con 0x7f4c14106150 2024-04-21T08:51:18.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.952+0000 7f4c0affd640 1 -- 172.21.15.28:0/3919955544 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25160 (secure 0 0 0) 0x7f4c141093e0 con 0x7f4c14106150 2024-04-21T08:51:18.952 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:18.952 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:51:10.823421+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1076282159":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6800/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/926389767":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/892059008":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/2132574390":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:51:18.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.954+0000 7f4c1bac8640 1 -- 172.21.15.28:0/3919955544 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4c000a3fe0 msgr2=0x7f4c000a64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.954+0000 7f4c1bac8640 1 --2- 172.21.15.28:0/3919955544 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4c000a3fe0 0x7f4c000a64a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4c10005fd0 tx=0x7f4c10009520 comp rx=0 tx=0).stop 2024-04-21T08:51:18.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.954+0000 7f4c1bac8640 1 -- 172.21.15.28:0/3919955544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14106150 msgr2=0x7f4c14197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.954+0000 7f4c1bac8640 1 --2- 172.21.15.28:0/3919955544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14106150 0x7f4c14197a60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400c970 tx=0x7f4c0400ce30 comp rx=0 tx=0).stop 2024-04-21T08:51:18.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.954+0000 7f4c1bac8640 1 -- 172.21.15.28:0/3919955544 shutdown_connections 2024-04-21T08:51:18.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.954+0000 7f4c1bac8640 1 --2- 172.21.15.28:0/3919955544 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4c000a3fe0 0x7f4c000a64a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.954+0000 7f4c1bac8640 1 --2- 172.21.15.28:0/3919955544 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c1419bde0 0x7f4c1419c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.954+0000 7f4c1bac8640 1 --2- 172.21.15.28:0/3919955544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14106150 0x7f4c14197a60 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.954+0000 7f4c1bac8640 1 --2- 172.21.15.28:0/3919955544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c14101840 0x7f4c14197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.954+0000 7f4c1bac8640 1 -- 172.21.15.28:0/3919955544 >> 172.21.15.28:0/3919955544 conn(0x7f4c14077450 msgr2=0x7f4c141042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:18.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.954+0000 7f4c1bac8640 1 -- 172.21.15.28:0/3919955544 shutdown_connections 2024-04-21T08:51:18.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:18.954+0000 7f4c1bac8640 1 -- 172.21.15.28:0/3919955544 wait complete. 2024-04-21T08:51:18.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:18 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3880471696' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:18.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:18 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/443564397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:18.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:18 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/346528291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:18.965 DEBUG:tasks.fwd_scrub:Yielding 2024-04-21T08:51:18.966 INFO:teuthology.run_tasks:Running task exec... 2024-04-21T08:51:18.975 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T08:51:18.975 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-21T08:51:18.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-21T08:51:18.976 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-21T08:51:18.976 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:51:18.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:51:18.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:51:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.081+0000 7faf7e0c8640 1 -- 172.21.15.28:0/3198158519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf781021d0 msgr2=0x7faf781025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.081+0000 7faf7e0c8640 1 --2- 172.21.15.28:0/3198158519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf781021d0 0x7faf781025b0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7faf6c009900 tx=0x7faf6c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.081+0000 7faf7e0c8640 1 -- 172.21.15.28:0/3198158519 shutdown_connections 2024-04-21T08:51:19.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.081+0000 7faf7e0c8640 1 --2- 172.21.15.28:0/3198158519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf781072a0 0x7faf7810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.081+0000 7faf7e0c8640 1 --2- 172.21.15.28:0/3198158519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faf78102b80 0x7faf78106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.081+0000 7faf7e0c8640 1 --2- 172.21.15.28:0/3198158519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf781021d0 0x7faf781025b0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.081+0000 7faf7e0c8640 1 -- 172.21.15.28:0/3198158519 >> 172.21.15.28:0/3198158519 conn(0x7faf780fda50 msgr2=0x7faf780ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.081+0000 7faf7e0c8640 1 -- 172.21.15.28:0/3198158519 shutdown_connections 2024-04-21T08:51:19.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.081+0000 7faf7e0c8640 1 -- 172.21.15.28:0/3198158519 wait complete. 2024-04-21T08:51:19.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.082+0000 7faf7e0c8640 1 Processor -- start 2024-04-21T08:51:19.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.082+0000 7faf7e0c8640 1 -- start start 2024-04-21T08:51:19.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.082+0000 7fadd6fce640 1 -- 172.21.15.28:0/509340960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd006a5d0 msgr2=0x7fadd006a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.082+0000 7fadd6fce640 1 --2- 172.21.15.28:0/509340960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd006a5d0 0x7fadd006a9b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009900 tx=0x7fadc402bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:19.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7fadd6fce640 1 -- 172.21.15.28:0/509340960 shutdown_connections 2024-04-21T08:51:19.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7fadd6fce640 1 --2- 172.21.15.28:0/509340960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fadd0107270 0x7fadd010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7fadd6fce640 1 --2- 172.21.15.28:0/509340960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fadd006af80 0x7fadd0106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7fadd6fce640 1 --2- 172.21.15.28:0/509340960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd006a5d0 0x7fadd006a9b0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf7e0c8640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faf781021d0 0x7faf78198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7fadd6fce640 1 -- 172.21.15.28:0/509340960 >> 172.21.15.28:0/509340960 conn(0x7fadd0077f80 msgr2=0x7fadd0078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf7e0c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf781072a0 0x7faf78198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf7e0c8640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf7819cfb0 0x7faf7819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf7e0c8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf78112200 con 0x7faf781072a0 2024-04-21T08:51:19.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf7e0c8640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7faf78112080 con 0x7faf7819cfb0 2024-04-21T08:51:19.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf7e0c8640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7faf78112380 con 0x7faf781021d0 2024-04-21T08:51:19.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7fadd6fce640 1 -- 172.21.15.28:0/509340960 shutdown_connections 2024-04-21T08:51:19.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7fadd6fce640 1 -- 172.21.15.28:0/509340960 wait complete. 2024-04-21T08:51:19.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf781072a0 0x7faf78198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf781072a0 0x7faf78198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52100/0 (socket says 172.21.15.28:52100) 2024-04-21T08:51:19.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf76ffd640 1 -- 172.21.15.28:0/524960648 learned_addr learned my addr 172.21.15.28:0/524960648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:19.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf777fe640 1 --2- 172.21.15.28:0/524960648 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faf781021d0 0x7faf78198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf77fff640 1 --2- 172.21.15.28:0/524960648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf7819cfb0 0x7faf7819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf76ffd640 1 -- 172.21.15.28:0/524960648 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faf781021d0 msgr2=0x7faf78198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf76ffd640 1 --2- 172.21.15.28:0/524960648 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faf781021d0 0x7faf78198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf76ffd640 1 -- 172.21.15.28:0/524960648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf7819cfb0 msgr2=0x7faf7819d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf76ffd640 1 --2- 172.21.15.28:0/524960648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf7819cfb0 0x7faf7819d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf76ffd640 1 -- 172.21.15.28:0/524960648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf7819daa0 con 0x7faf781072a0 2024-04-21T08:51:19.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf76ffd640 1 --2- 172.21.15.28:0/524960648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf781072a0 0x7faf78198c80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7faf6800e9f0 tx=0x7faf6800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7fadd6fce640 1 Processor -- start 2024-04-21T08:51:19.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf74ff9640 1 -- 172.21.15.28:0/524960648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6800ccc0 con 0x7faf781072a0 2024-04-21T08:51:19.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf74ff9640 1 -- 172.21.15.28:0/524960648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf680054d0 con 0x7faf781072a0 2024-04-21T08:51:19.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf7e0c8640 1 -- 172.21.15.28:0/524960648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf7819dd90 con 0x7faf781072a0 2024-04-21T08:51:19.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7fadd6fce640 1 -- start start 2024-04-21T08:51:19.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf74ff9640 1 -- 172.21.15.28:0/524960648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf68010430 con 0x7faf781072a0 2024-04-21T08:51:19.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.083+0000 7faf7e0c8640 1 -- 172.21.15.28:0/524960648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf781a5530 con 0x7faf781072a0 2024-04-21T08:51:19.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadd6fce640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fadd006a5d0 0x7fadd0198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadd6fce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd0107270 0x7fadd0198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadd6fce640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fadd019cfe0 0x7fadd019d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadd6fce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadd0112200 con 0x7fadd0107270 2024-04-21T08:51:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadd6fce640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fadd0112080 con 0x7fadd006a5d0 2024-04-21T08:51:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadd6fce640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fadd0112380 con 0x7fadd019cfe0 2024-04-21T08:51:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadcffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd0107270 0x7fadd0198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadd4d6a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fadd006a5d0 0x7fadd0198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadcffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd0107270 0x7fadd0198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52112/0 (socket says 172.21.15.28:52112) 2024-04-21T08:51:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadd4d6a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fadd006a5d0 0x7fadd0198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:57826/0 (socket says 172.21.15.28:57826) 2024-04-21T08:51:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadcffff640 1 -- 172.21.15.28:0/2350480378 learned_addr learned my addr 172.21.15.28:0/2350480378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadd556b640 1 --2- 172.21.15.28:0/2350480378 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fadd019cfe0 0x7fadd019d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadcffff640 1 -- 172.21.15.28:0/2350480378 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fadd019cfe0 msgr2=0x7fadd019d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadcffff640 1 --2- 172.21.15.28:0/2350480378 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fadd019cfe0 0x7fadd019d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadcffff640 1 -- 172.21.15.28:0/2350480378 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fadd006a5d0 msgr2=0x7fadd0198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadcffff640 1 --2- 172.21.15.28:0/2350480378 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fadd006a5d0 0x7fadd0198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadcffff640 1 -- 172.21.15.28:0/2350480378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd019dad0 con 0x7fadd0107270 2024-04-21T08:51:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadd4d6a640 1 --2- 172.21.15.28:0/2350480378 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fadd006a5d0 0x7fadd0198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:19.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.084+0000 7fadcffff640 1 --2- 172.21.15.28:0/2350480378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd0107270 0x7fadd0198c60 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fadbc005fd0 tx=0x7fadbc00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.085+0000 7fadcdffb640 1 -- 172.21.15.28:0/2350480378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc017070 con 0x7fadd0107270 2024-04-21T08:51:19.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.085+0000 7fadcdffb640 1 -- 172.21.15.28:0/2350480378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadbc009c30 con 0x7fadd0107270 2024-04-21T08:51:19.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.085+0000 7fadd6fce640 1 -- 172.21.15.28:0/2350480378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd019ddc0 con 0x7fadd0107270 2024-04-21T08:51:19.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.085+0000 7fadcdffb640 1 -- 172.21.15.28:0/2350480378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc003d30 con 0x7fadd0107270 2024-04-21T08:51:19.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.085+0000 7fadd6fce640 1 -- 172.21.15.28:0/2350480378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd01a55b0 con 0x7fadd0107270 2024-04-21T08:51:19.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.085+0000 7faf7e0c8640 1 -- 172.21.15.28:0/524960648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf7804fce0 con 0x7faf781072a0 2024-04-21T08:51:19.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.086+0000 7faf74ff9640 1 -- 172.21.15.28:0/524960648 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7faf680046c0 con 0x7faf781072a0 2024-04-21T08:51:19.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.087+0000 7faf74ff9640 1 --2- 172.21.15.28:0/524960648 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faf600a3ee0 0x7faf600a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.087+0000 7fadcdffb640 1 -- 172.21.15.28:0/2350480378 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fadbc003ed0 con 0x7fadd0107270 2024-04-21T08:51:19.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.087+0000 7faf777fe640 1 --2- 172.21.15.28:0/524960648 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faf600a3ee0 0x7faf600a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.087+0000 7faf74ff9640 1 -- 172.21.15.28:0/524960648 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7faf680d1330 con 0x7faf781072a0 2024-04-21T08:51:19.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7faf777fe640 1 --2- 172.21.15.28:0/524960648 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faf600a3ee0 0x7faf600a63a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faf6c000c00 tx=0x7faf6c02c6b0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadd9bc0740 1 Processor -- start 2024-04-21T08:51:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadd9bc0740 1 -- start start 2024-04-21T08:51:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadcdffb640 1 --2- 172.21.15.28:0/2350480378 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fadc00a3e40 0x7fadc00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadd556b640 1 --2- 172.21.15.28:0/2350480378 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fadc00a3e40 0x7fadc00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadd9bc0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7716098e0 0x55f771609cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadd9bc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f77160a290 0x55f771612840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadd9bc0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f771612e10 0x55f771615260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadd9bc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f77156df00 con 0x55f77160a290 2024-04-21T08:51:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadd9bc0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f77150b590 con 0x55f771612e10 2024-04-21T08:51:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadd9bc0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f771569ec0 con 0x55f7716098e0 2024-04-21T08:51:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadcdffb640 1 -- 172.21.15.28:0/2350480378 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fadbc0d0a50 con 0x7fadd0107270 2024-04-21T08:51:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadcffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f77160a290 0x55f771612840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadcffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f77160a290 0x55f771612840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52114/0 (socket says 172.21.15.28:52114) 2024-04-21T08:51:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadcffff640 1 -- 172.21.15.28:0/2429304423 learned_addr learned my addr 172.21.15.28:0/2429304423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadd4d6a640 1 --2- 172.21.15.28:0/2429304423 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7716098e0 0x55f771609cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadd556b640 1 --2- 172.21.15.28:0/2429304423 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f771612e10 0x55f771615260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.088+0000 7fadd556b640 1 --2- 172.21.15.28:0/2350480378 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fadc00a3e40 0x7fadc00a6300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fadc800ad70 tx=0x7fadc800f790 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.089+0000 7fadcffff640 1 -- 172.21.15.28:0/2429304423 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7716098e0 msgr2=0x55f771609cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.089+0000 7fadcffff640 1 --2- 172.21.15.28:0/2429304423 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7716098e0 0x55f771609cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.089+0000 7fadcffff640 1 -- 172.21.15.28:0/2429304423 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f771612e10 msgr2=0x55f771615260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.089+0000 7fadcffff640 1 --2- 172.21.15.28:0/2429304423 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f771612e10 0x55f771615260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.089+0000 7fadcffff640 1 -- 172.21.15.28:0/2429304423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7713487e0 con 0x55f77160a290 2024-04-21T08:51:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.089+0000 7fadcffff640 1 --2- 172.21.15.28:0/2429304423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f77160a290 0x55f771612840 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fadbc098860 tx=0x7fadbc09c050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae33adc0ccefb6b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.092+0000 7fadbaffd640 1 -- 172.21.15.28:0/2429304423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc0a5070 con 0x55f77160a290 2024-04-21T08:51:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.092+0000 7fadbaffd640 1 -- 172.21.15.28:0/2429304423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadbc09a700 con 0x55f77160a290 2024-04-21T08:51:19.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.092+0000 7fadbaffd640 1 -- 172.21.15.28:0/2429304423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc004420 con 0x55f77160a290 2024-04-21T08:51:19.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.092+0000 7faf74ff9640 1 -- 172.21.15.28:0/524960648 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faf6809d3e0 con 0x7faf781072a0 2024-04-21T08:51:19.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.092+0000 7fadd9bc0740 1 -- 172.21.15.28:0/2429304423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f77160a290 msgr2=0x55f771612840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.092+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/2429304423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f77160a290 0x55f771612840 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fadbc098860 tx=0x7fadbc09c050 comp rx=0 tx=0).stop 2024-04-21T08:51:19.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.093+0000 7fadd9bc0740 1 -- 172.21.15.28:0/2429304423 shutdown_connections 2024-04-21T08:51:19.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.093+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/2429304423 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f771612e10 0x55f771615260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.093+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/2429304423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f77160a290 0x55f771612840 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.093+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/2429304423 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7716098e0 0x55f771609cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.093+0000 7fadd9bc0740 1 -- 172.21.15.28:0/2429304423 >> 172.21.15.28:0/2429304423 conn(0x55f771603fb0 msgr2=0x55f771604390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.093+0000 7fadd9bc0740 1 -- 172.21.15.28:0/2429304423 shutdown_connections 2024-04-21T08:51:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.093+0000 7fadd9bc0740 1 -- 172.21.15.28:0/2429304423 wait complete. 2024-04-21T08:51:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.093+0000 7fadd9bc0740 1 -- start start 2024-04-21T08:51:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.093+0000 7fadd9bc0740 1 Processor -- start 2024-04-21T08:51:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadd9bc0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7716098e0 0x55f77160fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadd9bc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f77160a290 0x55f7716103c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadd9bc0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f771612e10 0x55f771610900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadd9bc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f771615d20 con 0x55f77160a290 2024-04-21T08:51:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadd9bc0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f771527580 con 0x55f771612e10 2024-04-21T08:51:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadd9bc0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f771615ea0 con 0x55f7716098e0 2024-04-21T08:51:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadd9bc0740 10 client.0 _ll_register_callbacks cb 0x55f77152d650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadcffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f77160a290 0x55f7716103c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadcffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f77160a290 0x55f7716103c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52126/0 (socket says 172.21.15.28:52126) 2024-04-21T08:51:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadcffff640 1 -- 172.21.15.28:0/3318942636 learned_addr learned my addr 172.21.15.28:0/3318942636 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadd4d6a640 1 --2- 172.21.15.28:0/3318942636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7716098e0 0x55f77160fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadd556b640 1 --2- 172.21.15.28:0/3318942636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f771612e10 0x55f771610900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadcffff640 1 -- 172.21.15.28:0/3318942636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7716098e0 msgr2=0x55f77160fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadcffff640 1 --2- 172.21.15.28:0/3318942636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7716098e0 0x55f77160fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadcffff640 1 -- 172.21.15.28:0/3318942636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f771612e10 msgr2=0x55f771610900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadcffff640 1 --2- 172.21.15.28:0/3318942636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f771612e10 0x55f771610900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadcffff640 1 -- 172.21.15.28:0/3318942636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7713a8f70 con 0x55f77160a290 2024-04-21T08:51:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadd556b640 1 --2- 172.21.15.28:0/3318942636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f771612e10 0x55f771610900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadcffff640 1 --2- 172.21.15.28:0/3318942636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f77160a290 0x55f7716103c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fadbc098dd0 tx=0x7fadbc0cdc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec19c4eb6f45f82 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.094+0000 7fadaffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.095+0000 7fadaffff640 1 -- 172.21.15.28:0/3318942636 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc0a5070 con 0x55f77160a290 2024-04-21T08:51:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.095+0000 7fadaffff640 1 -- 172.21.15.28:0/3318942636 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fadbc0bb2a0 con 0x55f77160a290 2024-04-21T08:51:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.095+0000 7fadd9bc0740 1 -- 172.21.15.28:0/3318942636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadbc09a700 con 0x55f77160a290 2024-04-21T08:51:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.095+0000 7fadaffff640 1 -- 172.21.15.28:0/3318942636 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fadbc082050 con 0x55f77160a290 2024-04-21T08:51:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.095+0000 7fadd9bc0740 10 client.15681 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.095+0000 7fadd9bc0740 1 -- 172.21.15.28:0/3318942636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f771240170 con 0x55f77160a290 2024-04-21T08:51:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.095+0000 7fadaffff640 1 -- 172.21.15.28:0/3318942636 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadbc09a700 con 0x55f77160a290 2024-04-21T08:51:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.095+0000 7fadaffff640 1 -- 172.21.15.28:0/3318942636 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fadbc098150 con 0x55f77160a290 2024-04-21T08:51:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.095+0000 7fadd9bc0740 10 client.15681 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.095+0000 7fadd9bc0740 10 client.15681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.095+0000 7fadd9bc0740 20 client.15681 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.095+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/3318942636 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f771623220 0x55f771625610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.095+0000 7fadd9bc0740 4 client.15681 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.095+0000 7fadd9bc0740 1 -- 172.21.15.28:0/3318942636 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f77157b920 con 0x55f771623220 2024-04-21T08:51:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.096+0000 7fadd4d6a640 1 --2- 172.21.15.28:0/3318942636 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f771623220 0x55f771625610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.096+0000 7fadd4d6a640 1 --2- 172.21.15.28:0/3318942636 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f771623220 0x55f771625610 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=782a76d27715d3da server_cookie=c6f7698a9e89be1d in_seq=0 out_seq=0 2024-04-21T08:51:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.096+0000 7fadaffff640 10 client.15681 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.097+0000 7fadaffff640 1 -- 172.21.15.28:0/3318942636 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f77157b920 con 0x55f771623220 2024-04-21T08:51:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.097+0000 7fadaffff640 10 client.15681 handle_command_reply: tid=0 2024-04-21T08:51:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.097+0000 7fadd9bc0740 1 client.15681 shutdown 2024-04-21T08:51:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.097+0000 7fadd9bc0740 1 -- 172.21.15.28:0/3318942636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f77160a290 msgr2=0x55f7716103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.097+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/3318942636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f77160a290 0x55f7716103c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fadbc098dd0 tx=0x7fadbc0cdc20 comp rx=0 tx=0).stop 2024-04-21T08:51:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.097+0000 7fadd9bc0740 1 -- 172.21.15.28:0/3318942636 shutdown_connections 2024-04-21T08:51:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.097+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/3318942636 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f771623220 0x55f771625610 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.097+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/3318942636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f771612e10 0x55f771610900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.097+0000 7fadaffff640 0 client.15681 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.097+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/3318942636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f77160a290 0x55f7716103c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.097+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/3318942636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7716098e0 0x55f77160fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.097+0000 7fadd9bc0740 1 -- 172.21.15.28:0/3318942636 >> 172.21.15.28:0/3318942636 conn(0x55f771603fb0 msgr2=0x55f771605d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.097+0000 7fadd9bc0740 1 -- 172.21.15.28:0/3318942636 shutdown_connections 2024-04-21T08:51:19.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.097+0000 7fadd9bc0740 1 -- 172.21.15.28:0/3318942636 wait complete. 2024-04-21T08:51:19.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.098+0000 7fadd9bc0740 20 client.15681 trim_cache size 0 max 16384 2024-04-21T08:51:19.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:18 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3880471696' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:19.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:18 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/443564397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:19.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:18 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/346528291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.131+0000 7fadd9bc0740 1 Processor -- start 2024-04-21T08:51:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.131+0000 7fadd9bc0740 1 -- start start 2024-04-21T08:51:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.131+0000 7fadd9bc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7716098e0 0x55f7716125b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.131+0000 7fadd9bc0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f77160a290 0x55f77161ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.131+0000 7fadd9bc0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f771612e10 0x55f77160efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.131+0000 7fadd9bc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadd01b1b50 con 0x55f7716098e0 2024-04-21T08:51:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.131+0000 7fadd9bc0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fadd01af420 con 0x55f77160a290 2024-04-21T08:51:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.131+0000 7fadd9bc0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fadd01ad270 con 0x55f771612e10 2024-04-21T08:51:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.131+0000 7fadd4d6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7716098e0 0x55f7716125b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.131+0000 7fadd4d6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7716098e0 0x55f7716125b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52142/0 (socket says 172.21.15.28:52142) 2024-04-21T08:51:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.131+0000 7fadd4d6a640 1 -- 172.21.15.28:0/1191417316 learned_addr learned my addr 172.21.15.28:0/1191417316 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.131+0000 7fadd556b640 1 --2- 172.21.15.28:0/1191417316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f771612e10 0x55f77160efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.132+0000 7fadcffff640 1 --2- 172.21.15.28:0/1191417316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f77160a290 0x55f77161ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.132+0000 7fadd4d6a640 1 -- 172.21.15.28:0/1191417316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f771612e10 msgr2=0x55f77160efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.132+0000 7fadd4d6a640 1 --2- 172.21.15.28:0/1191417316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f771612e10 0x55f77160efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.132+0000 7fadd4d6a640 1 -- 172.21.15.28:0/1191417316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f77160a290 msgr2=0x55f77161ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.132+0000 7fadd4d6a640 1 --2- 172.21.15.28:0/1191417316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f77160a290 0x55f77161ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.132+0000 7fadd4d6a640 1 -- 172.21.15.28:0/1191417316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f771603790 con 0x55f7716098e0 2024-04-21T08:51:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.132+0000 7fadd556b640 1 --2- 172.21.15.28:0/1191417316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f771612e10 0x55f77160efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.132+0000 7fadcffff640 1 --2- 172.21.15.28:0/1191417316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f77160a290 0x55f77161ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.132+0000 7fadd4d6a640 1 --2- 172.21.15.28:0/1191417316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7716098e0 0x55f7716125b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fadc4005ec0 tx=0x7fadc4005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246bf186c3cc4d09 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.132+0000 7fadba7fc640 1 -- 172.21.15.28:0/1191417316 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc402c7e0 con 0x55f7716098e0 2024-04-21T08:51:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.132+0000 7fadba7fc640 1 -- 172.21.15.28:0/1191417316 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadc4050030 con 0x55f7716098e0 2024-04-21T08:51:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.132+0000 7fadba7fc640 1 -- 172.21.15.28:0/1191417316 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc4003950 con 0x55f7716098e0 2024-04-21T08:51:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.133+0000 7fadd9bc0740 1 -- 172.21.15.28:0/1191417316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7716098e0 msgr2=0x55f7716125b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.133+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/1191417316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7716098e0 0x55f7716125b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fadc4005ec0 tx=0x7fadc4005a20 comp rx=0 tx=0).stop 2024-04-21T08:51:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.133+0000 7fadd9bc0740 1 -- 172.21.15.28:0/1191417316 shutdown_connections 2024-04-21T08:51:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.133+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/1191417316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f771612e10 0x55f77160efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.133+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/1191417316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f77160a290 0x55f77161ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.133+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/1191417316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7716098e0 0x55f7716125b0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.133+0000 7fadd9bc0740 1 -- 172.21.15.28:0/1191417316 >> 172.21.15.28:0/1191417316 conn(0x55f771603fb0 msgr2=0x55f77160fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.133+0000 7fadd9bc0740 1 -- 172.21.15.28:0/1191417316 shutdown_connections 2024-04-21T08:51:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.133+0000 7fadd9bc0740 1 -- 172.21.15.28:0/1191417316 wait complete. 2024-04-21T08:51:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.133+0000 7fadd9bc0740 1 -- start start 2024-04-21T08:51:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.133+0000 7fadd9bc0740 1 Processor -- start 2024-04-21T08:51:19.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd9bc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7716098e0 0x55f7716173d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd9bc0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f77160a290 0x55f771617910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd9bc0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f771612e10 0x55f7716868f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd9bc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadd01ad8e0 con 0x55f7716098e0 2024-04-21T08:51:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd9bc0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fadd01aed50 con 0x55f77160a290 2024-04-21T08:51:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd9bc0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fadd01adfb0 con 0x55f771612e10 2024-04-21T08:51:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd9bc0740 10 client.0 _ll_register_callbacks cb 0x7fadc4005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd4d6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7716098e0 0x55f7716173d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd4d6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7716098e0 0x55f7716173d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52158/0 (socket says 172.21.15.28:52158) 2024-04-21T08:51:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd4d6a640 1 -- 172.21.15.28:0/840288420 learned_addr learned my addr 172.21.15.28:0/840288420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadcffff640 1 --2- 172.21.15.28:0/840288420 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f77160a290 0x55f771617910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd556b640 1 --2- 172.21.15.28:0/840288420 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f771612e10 0x55f7716868f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd4d6a640 1 -- 172.21.15.28:0/840288420 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f771612e10 msgr2=0x55f7716868f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd4d6a640 1 --2- 172.21.15.28:0/840288420 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f771612e10 0x55f7716868f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd4d6a640 1 -- 172.21.15.28:0/840288420 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f77160a290 msgr2=0x55f771617910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd4d6a640 1 --2- 172.21.15.28:0/840288420 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f77160a290 0x55f771617910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd4d6a640 1 -- 172.21.15.28:0/840288420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f771686eb0 con 0x55f7716098e0 2024-04-21T08:51:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadcffff640 1 --2- 172.21.15.28:0/840288420 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f77160a290 0x55f771617910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd556b640 1 --2- 172.21.15.28:0/840288420 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f771612e10 0x55f7716868f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadd4d6a640 1 --2- 172.21.15.28:0/840288420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7716098e0 0x55f7716173d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fadc40092a0 tx=0x7fadc404c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a869a2d8bce3683 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.134+0000 7fadaffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.135+0000 7fadaffff640 1 -- 172.21.15.28:0/840288420 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc402c7e0 con 0x55f7716098e0 2024-04-21T08:51:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.135+0000 7fadaffff640 1 -- 172.21.15.28:0/840288420 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fadc402c980 con 0x55f7716098e0 2024-04-21T08:51:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.135+0000 7fadd9bc0740 1 -- 172.21.15.28:0/840288420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadc4050030 con 0x55f7716098e0 2024-04-21T08:51:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.135+0000 7fadaffff640 1 -- 172.21.15.28:0/840288420 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fadc402cb80 con 0x55f7716098e0 2024-04-21T08:51:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.135+0000 7fadd9bc0740 10 client.15687 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.135+0000 7fadd9bc0740 1 -- 172.21.15.28:0/840288420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f77168e210 con 0x55f7716098e0 2024-04-21T08:51:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.135+0000 7fadaffff640 1 -- 172.21.15.28:0/840288420 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadc4050030 con 0x55f7716098e0 2024-04-21T08:51:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.135+0000 7fadaffff640 1 -- 172.21.15.28:0/840288420 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fadc4055630 con 0x55f7716098e0 2024-04-21T08:51:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.135+0000 7fadd9bc0740 10 client.15687 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.135+0000 7fadd9bc0740 10 client.15687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.135+0000 7fadd9bc0740 20 client.15687 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.135+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/840288420 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f771623220 0x55f771690940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.135+0000 7fadd9bc0740 4 client.15687 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:51:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.135+0000 7fadd9bc0740 1 -- 172.21.15.28:0/840288420 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f77160ac10 con 0x55f771623220 2024-04-21T08:51:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.136+0000 7fadcffff640 1 --2- 172.21.15.28:0/840288420 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f771623220 0x55f771690940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.136+0000 7fadcffff640 1 --2- 172.21.15.28:0/840288420 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f771623220 0x55f771690940 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3be8292f1e5bd0e server_cookie=fe90762f82404f7c in_seq=0 out_seq=0 2024-04-21T08:51:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.136+0000 7fadaffff640 10 client.15687 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.137+0000 7fadaffff640 1 -- 172.21.15.28:0/840288420 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f77160ac10 con 0x55f771623220 2024-04-21T08:51:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.137+0000 7fadaffff640 10 client.15687 handle_command_reply: tid=0 2024-04-21T08:51:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.137+0000 7fadd9bc0740 1 client.15687 shutdown 2024-04-21T08:51:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.137+0000 7fadd9bc0740 1 -- 172.21.15.28:0/840288420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7716098e0 msgr2=0x55f7716173d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.137+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/840288420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7716098e0 0x55f7716173d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fadc40092a0 tx=0x7fadc404c4d0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.137+0000 7fadd9bc0740 1 -- 172.21.15.28:0/840288420 shutdown_connections 2024-04-21T08:51:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.137+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/840288420 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f771623220 0x55f771690940 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.137+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/840288420 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f771612e10 0x55f7716868f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.137+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/840288420 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f77160a290 0x55f771617910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.137+0000 7fadaffff640 0 client.15687 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.137+0000 7fadd9bc0740 1 --2- 172.21.15.28:0/840288420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7716098e0 0x55f7716173d0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.137+0000 7fadd9bc0740 1 -- 172.21.15.28:0/840288420 >> 172.21.15.28:0/840288420 conn(0x55f771603fb0 msgr2=0x55f77160e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.138+0000 7fadd9bc0740 1 -- 172.21.15.28:0/840288420 shutdown_connections 2024-04-21T08:51:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.138+0000 7fadd9bc0740 1 -- 172.21.15.28:0/840288420 wait complete. 2024-04-21T08:51:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.138+0000 7fadd9bc0740 20 client.15687 trim_cache size 0 max 16384 2024-04-21T08:51:19.143 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:51:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:51:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "7d56b2cb-87ee-4b4d-84f6-e1e84063a464", 2024-04-21T08:51:19.143 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:51:19.143 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:51:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.138+0000 7fadd6fce640 1 -- 172.21.15.28:0/2350480378 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fadc00a3e40 msgr2=0x7fadc00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.138+0000 7fadd6fce640 1 --2- 172.21.15.28:0/2350480378 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fadc00a3e40 0x7fadc00a6300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fadc800ad70 tx=0x7fadc800f790 comp rx=0 tx=0).stop 2024-04-21T08:51:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.138+0000 7fadd6fce640 1 -- 172.21.15.28:0/2350480378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd0107270 msgr2=0x7fadd0198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.138+0000 7fadd6fce640 1 --2- 172.21.15.28:0/2350480378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd0107270 0x7fadd0198c60 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fadbc005fd0 tx=0x7fadbc00dde0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.138+0000 7fadd6fce640 1 -- 172.21.15.28:0/2350480378 shutdown_connections 2024-04-21T08:51:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.138+0000 7fadd6fce640 1 --2- 172.21.15.28:0/2350480378 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fadc00a3e40 0x7fadc00a6300 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.138+0000 7fadd6fce640 1 --2- 172.21.15.28:0/2350480378 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fadd019cfe0 0x7fadd019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.138+0000 7fadd6fce640 1 --2- 172.21.15.28:0/2350480378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd0107270 0x7fadd0198c60 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.138+0000 7fadd6fce640 1 --2- 172.21.15.28:0/2350480378 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fadd006a5d0 0x7fadd0198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.138+0000 7fadd6fce640 1 -- 172.21.15.28:0/2350480378 >> 172.21.15.28:0/2350480378 conn(0x7fadd0077f80 msgr2=0x7fadd00ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.138+0000 7fadd6fce640 1 -- 172.21.15.28:0/2350480378 shutdown_connections 2024-04-21T08:51:19.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.138+0000 7fadd6fce640 1 -- 172.21.15.28:0/2350480378 wait complete. 2024-04-21T08:51:19.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:51:19.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:51:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.254+0000 7f0552f7e640 1 -- 172.21.15.28:0/1531512696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f054c10c500 msgr2=0x7f054c10e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.254+0000 7f0552f7e640 1 --2- 172.21.15.28:0/1531512696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f054c10c500 0x7f054c10e9a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0544009ca0 tx=0x7f054402c230 comp rx=0 tx=0).stop 2024-04-21T08:51:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.254+0000 7f0552f7e640 1 -- 172.21.15.28:0/1531512696 shutdown_connections 2024-04-21T08:51:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.254+0000 7f0552f7e640 1 --2- 172.21.15.28:0/1531512696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f054c10c500 0x7f054c10e9a0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.254+0000 7f0552f7e640 1 --2- 172.21.15.28:0/1531512696 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f054c103ae0 0x7f054c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.254+0000 7f0552f7e640 1 --2- 172.21.15.28:0/1531512696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f054c1031c0 0x7f054c1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.254+0000 7f0552f7e640 1 -- 172.21.15.28:0/1531512696 >> 172.21.15.28:0/1531512696 conn(0x7f054c078000 msgr2=0x7f054c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.254+0000 7f0552f7e640 1 -- 172.21.15.28:0/1531512696 shutdown_connections 2024-04-21T08:51:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.255+0000 7f0552f7e640 1 -- 172.21.15.28:0/1531512696 wait complete. 2024-04-21T08:51:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.255+0000 7f0552f7e640 1 Processor -- start 2024-04-21T08:51:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.255+0000 7f0552f7e640 1 -- start start 2024-04-21T08:51:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.255+0000 7f0552f7e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f054c1031c0 0x7f054c100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.255+0000 7f0552f7e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f054c103ae0 0x7f054c101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.255+0000 7f0552f7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f054c10c500 0x7f054c101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.255+0000 7f0552f7e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f054c112190 con 0x7f054c10c500 2024-04-21T08:51:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.255+0000 7f0552f7e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f054c112010 con 0x7f054c103ae0 2024-04-21T08:51:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.255+0000 7f0552f7e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f054c112310 con 0x7f054c1031c0 2024-04-21T08:51:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.255+0000 7f055151b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f054c10c500 0x7f054c101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.255+0000 7f055151b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f054c10c500 0x7f054c101890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52182/0 (socket says 172.21.15.28:52182) 2024-04-21T08:51:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.255+0000 7f055151b640 1 -- 172.21.15.28:0/1679141036 learned_addr learned my addr 172.21.15.28:0/1679141036 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.256+0000 7f054bfff640 1 --2- 172.21.15.28:0/1679141036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f054c103ae0 0x7f054c101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.256+0000 7f0550d1a640 1 --2- 172.21.15.28:0/1679141036 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f054c1031c0 0x7f054c100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.256+0000 7f055151b640 1 -- 172.21.15.28:0/1679141036 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f054c1031c0 msgr2=0x7f054c100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.256+0000 7f055151b640 1 --2- 172.21.15.28:0/1679141036 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f054c1031c0 0x7f054c100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.256+0000 7f055151b640 1 -- 172.21.15.28:0/1679141036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f054c103ae0 msgr2=0x7f054c101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.256+0000 7f055151b640 1 --2- 172.21.15.28:0/1679141036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f054c103ae0 0x7f054c101350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.256+0000 7f055151b640 1 -- 172.21.15.28:0/1679141036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054c1a6bb0 con 0x7f054c10c500 2024-04-21T08:51:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.256+0000 7f055151b640 1 --2- 172.21.15.28:0/1679141036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f054c10c500 0x7f054c101890 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0544037040 tx=0x7f0544035e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.256+0000 7f0549ffb640 1 -- 172.21.15.28:0/1679141036 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544039040 con 0x7f054c10c500 2024-04-21T08:51:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.256+0000 7f0552f7e640 1 -- 172.21.15.28:0/1679141036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f054c1a6e40 con 0x7f054c10c500 2024-04-21T08:51:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.256+0000 7f0549ffb640 1 -- 172.21.15.28:0/1679141036 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0544044070 con 0x7f054c10c500 2024-04-21T08:51:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.256+0000 7f0549ffb640 1 -- 172.21.15.28:0/1679141036 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05440328a0 con 0x7f054c10c500 2024-04-21T08:51:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.256+0000 7f0552f7e640 1 -- 172.21.15.28:0/1679141036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f054c1a7350 con 0x7f054c10c500 2024-04-21T08:51:19.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0549ffb640 1 -- 172.21.15.28:0/1679141036 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f0544032a40 con 0x7f054c10c500 2024-04-21T08:51:19.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0555b6f740 1 Processor -- start 2024-04-21T08:51:19.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0555b6f740 1 -- start start 2024-04-21T08:51:19.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0555b6f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0a580 0x56377eb0a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0555b6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb0aea0 0x56377eb13540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0555b6f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb13a80 0x56377eb15e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0555b6f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56377ea6f1e0 con 0x56377eb0aea0 2024-04-21T08:51:19.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0555b6f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56377ea0c390 con 0x56377eb0a580 2024-04-21T08:51:19.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0555b6f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56377ea6b1a0 con 0x56377eb13a80 2024-04-21T08:51:19.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0550d1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb0aea0 0x56377eb13540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0550d1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb0aea0 0x56377eb13540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52190/0 (socket says 172.21.15.28:52190) 2024-04-21T08:51:19.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0550d1a640 1 -- 172.21.15.28:0/3567627198 learned_addr learned my addr 172.21.15.28:0/3567627198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:19.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f054bfff640 1 --2- 172.21.15.28:0/3567627198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb13a80 0x56377eb15e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f054bfff640 1 --2- 172.21.15.28:0/3567627198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0a580 0x56377eb0a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0550d1a640 1 -- 172.21.15.28:0/3567627198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb13a80 msgr2=0x56377eb15e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0550d1a640 1 --2- 172.21.15.28:0/3567627198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb13a80 0x56377eb15e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0550d1a640 1 -- 172.21.15.28:0/3567627198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0a580 msgr2=0x56377eb0a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0550d1a640 1 --2- 172.21.15.28:0/3567627198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0a580 0x56377eb0a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0550d1a640 1 -- 172.21.15.28:0/3567627198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56377e84b7d0 con 0x56377eb0aea0 2024-04-21T08:51:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f054bfff640 1 --2- 172.21.15.28:0/3567627198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0a580 0x56377eb0a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.259+0000 7f0549ffb640 1 --2- 172.21.15.28:0/1679141036 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f053c0a3e40 0x7f053c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.260+0000 7f0550d1a640 1 --2- 172.21.15.28:0/3567627198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb0aea0 0x56377eb13540 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0540009c80 tx=0x7f05400117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e7e5c1eb9b0626 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.260+0000 7f054bfff640 1 --2- 172.21.15.28:0/1679141036 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f053c0a3e40 0x7f053c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.260+0000 7f0536ffd640 1 -- 172.21.15.28:0/3567627198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0540012450 con 0x56377eb0aea0 2024-04-21T08:51:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.260+0000 7f0549ffb640 1 -- 172.21.15.28:0/1679141036 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f05440f3260 con 0x7f054c10c500 2024-04-21T08:51:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.260+0000 7f0536ffd640 1 -- 172.21.15.28:0/3567627198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0540012aa0 con 0x56377eb0aea0 2024-04-21T08:51:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.260+0000 7f0536ffd640 1 -- 172.21.15.28:0/3567627198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054000fb80 con 0x56377eb0aea0 2024-04-21T08:51:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f054bfff640 1 --2- 172.21.15.28:0/1679141036 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f053c0a3e40 0x7f053c0a6300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x56377eb05e40 tx=0x7f05380046c0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f0555b6f740 1 -- 172.21.15.28:0/3567627198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb0aea0 msgr2=0x56377eb13540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f0555b6f740 1 --2- 172.21.15.28:0/3567627198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb0aea0 0x56377eb13540 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0540009c80 tx=0x7f05400117b0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f0555b6f740 1 -- 172.21.15.28:0/3567627198 shutdown_connections 2024-04-21T08:51:19.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f0555b6f740 1 --2- 172.21.15.28:0/3567627198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb13a80 0x56377eb15e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f0555b6f740 1 --2- 172.21.15.28:0/3567627198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb0aea0 0x56377eb13540 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f0555b6f740 1 --2- 172.21.15.28:0/3567627198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0a580 0x56377eb0a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f0555b6f740 1 -- 172.21.15.28:0/3567627198 >> 172.21.15.28:0/3567627198 conn(0x56377eb04cb0 msgr2=0x56377eb05090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f0555b6f740 1 -- 172.21.15.28:0/3567627198 shutdown_connections 2024-04-21T08:51:19.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f0555b6f740 1 -- 172.21.15.28:0/3567627198 wait complete. 2024-04-21T08:51:19.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f0555b6f740 1 -- start start 2024-04-21T08:51:19.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f0555b6f740 1 Processor -- start 2024-04-21T08:51:19.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f0555b6f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb0a580 0x56377eb10a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f0555b6f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0aea0 0x56377eb10f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f0555b6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb13a80 0x56377eb114b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.261+0000 7f0555b6f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56377eb16870 con 0x56377eb13a80 2024-04-21T08:51:19.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f0555b6f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56377ea282f0 con 0x56377eb0aea0 2024-04-21T08:51:19.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f0555b6f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56377eb169f0 con 0x56377eb0a580 2024-04-21T08:51:19.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f0555b6f740 10 client.0 _ll_register_callbacks cb 0x56377ea35580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f055151b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb13a80 0x56377eb114b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f055151b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb13a80 0x56377eb114b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52200/0 (socket says 172.21.15.28:52200) 2024-04-21T08:51:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f055151b640 1 -- 172.21.15.28:0/2432755696 learned_addr learned my addr 172.21.15.28:0/2432755696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f054bfff640 1 --2- 172.21.15.28:0/2432755696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0aea0 0x56377eb10f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f0550d1a640 1 --2- 172.21.15.28:0/2432755696 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb0a580 0x56377eb10a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f055151b640 1 -- 172.21.15.28:0/2432755696 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb0a580 msgr2=0x56377eb10a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f055151b640 1 --2- 172.21.15.28:0/2432755696 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb0a580 0x56377eb10a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f055151b640 1 -- 172.21.15.28:0/2432755696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0aea0 msgr2=0x56377eb10f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f055151b640 1 --2- 172.21.15.28:0/2432755696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0aea0 0x56377eb10f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f055151b640 1 -- 172.21.15.28:0/2432755696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56377e7813f0 con 0x56377eb13a80 2024-04-21T08:51:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f055151b640 1 --2- 172.21.15.28:0/2432755696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb13a80 0x56377eb114b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f05440becb0 tx=0x7f05440bebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcd327e222753dda server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f052bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f052bfff640 1 -- 172.21.15.28:0/2432755696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05440bed20 con 0x56377eb13a80 2024-04-21T08:51:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f052bfff640 1 -- 172.21.15.28:0/2432755696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f0544032d30 con 0x56377eb13a80 2024-04-21T08:51:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.262+0000 7f0555b6f740 1 -- 172.21.15.28:0/2432755696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0540012aa0 con 0x56377eb13a80 2024-04-21T08:51:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.263+0000 7f052bfff640 1 -- 172.21.15.28:0/2432755696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f05440c30d0 con 0x56377eb13a80 2024-04-21T08:51:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.263+0000 7f0555b6f740 10 client.15699 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.263+0000 7f0555b6f740 1 -- 172.21.15.28:0/2432755696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56377e7410f0 con 0x56377eb13a80 2024-04-21T08:51:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.263+0000 7f052bfff640 1 -- 172.21.15.28:0/2432755696 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0540012aa0 con 0x56377eb13a80 2024-04-21T08:51:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.263+0000 7f052bfff640 1 -- 172.21.15.28:0/2432755696 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f054403c030 con 0x56377eb13a80 2024-04-21T08:51:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.263+0000 7f0555b6f740 10 client.15699 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.263+0000 7f0555b6f740 10 client.15699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.263+0000 7f0555b6f740 20 client.15699 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.263+0000 7f0555b6f740 1 --2- 172.21.15.28:0/2432755696 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56377eb23b90 0x56377eb25ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.263+0000 7f0555b6f740 4 client.15699 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.263+0000 7f0555b6f740 1 -- 172.21.15.28:0/2432755696 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56377ea7cc00 con 0x56377eb23b90 2024-04-21T08:51:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.263+0000 7f0550d1a640 1 --2- 172.21.15.28:0/2432755696 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56377eb23b90 0x56377eb25ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.264+0000 7f052bfff640 10 client.15699 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.264+0000 7f0550d1a640 1 --2- 172.21.15.28:0/2432755696 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56377eb23b90 0x56377eb25ff0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73bdec1824b845e7 server_cookie=b81c71db467f05fe in_seq=0 out_seq=0 2024-04-21T08:51:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.264+0000 7f052bfff640 1 -- 172.21.15.28:0/2432755696 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56377ea7cc00 con 0x56377eb23b90 2024-04-21T08:51:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.264+0000 7f052bfff640 10 client.15699 handle_command_reply: tid=0 2024-04-21T08:51:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.264+0000 7f0555b6f740 1 client.15699 shutdown 2024-04-21T08:51:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.264+0000 7f0555b6f740 1 -- 172.21.15.28:0/2432755696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb13a80 msgr2=0x56377eb114b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.264+0000 7f0555b6f740 1 --2- 172.21.15.28:0/2432755696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb13a80 0x56377eb114b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f05440becb0 tx=0x7f05440bebb0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.265+0000 7f0555b6f740 1 -- 172.21.15.28:0/2432755696 shutdown_connections 2024-04-21T08:51:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.265+0000 7f0555b6f740 1 --2- 172.21.15.28:0/2432755696 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56377eb23b90 0x56377eb25ff0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.265+0000 7f0555b6f740 1 --2- 172.21.15.28:0/2432755696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb13a80 0x56377eb114b0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.265+0000 7f0555b6f740 1 --2- 172.21.15.28:0/2432755696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0aea0 0x56377eb10f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.265+0000 7f0555b6f740 1 --2- 172.21.15.28:0/2432755696 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb0a580 0x56377eb10a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.265+0000 7f0555b6f740 1 -- 172.21.15.28:0/2432755696 >> 172.21.15.28:0/2432755696 conn(0x56377eb04cb0 msgr2=0x56377eb06a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.265+0000 7f052bfff640 0 client.15699 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:19.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.265+0000 7f0555b6f740 1 -- 172.21.15.28:0/2432755696 shutdown_connections 2024-04-21T08:51:19.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.265+0000 7f0555b6f740 1 -- 172.21.15.28:0/2432755696 wait complete. 2024-04-21T08:51:19.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.265+0000 7f0555b6f740 20 client.15699 trim_cache size 0 max 16384 2024-04-21T08:51:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.282+0000 7f0555b6f740 1 Processor -- start 2024-04-21T08:51:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.282+0000 7f0555b6f740 1 -- start start 2024-04-21T08:51:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f0555b6f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0a580 0x56377eb13160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f0555b6f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb0aea0 0x56377eb1f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f0555b6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb13a80 0x56377eb067f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f0555b6f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f054c1b3d10 con 0x56377eb13a80 2024-04-21T08:51:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f0555b6f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f054c1b15e0 con 0x56377eb0a580 2024-04-21T08:51:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f0555b6f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f054c1af430 con 0x56377eb0aea0 2024-04-21T08:51:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f055151b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb13a80 0x56377eb067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f055151b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb13a80 0x56377eb067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52208/0 (socket says 172.21.15.28:52208) 2024-04-21T08:51:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f055151b640 1 -- 172.21.15.28:0/496211199 learned_addr learned my addr 172.21.15.28:0/496211199 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f0550d1a640 1 --2- 172.21.15.28:0/496211199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0a580 0x56377eb13160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f054bfff640 1 --2- 172.21.15.28:0/496211199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb0aea0 0x56377eb1f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f055151b640 1 -- 172.21.15.28:0/496211199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb0aea0 msgr2=0x56377eb1f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f055151b640 1 --2- 172.21.15.28:0/496211199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb0aea0 0x56377eb1f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f055151b640 1 -- 172.21.15.28:0/496211199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0a580 msgr2=0x56377eb13160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f055151b640 1 --2- 172.21.15.28:0/496211199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0a580 0x56377eb13160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f055151b640 1 -- 172.21.15.28:0/496211199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56377eb04ae0 con 0x56377eb13a80 2024-04-21T08:51:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f055151b640 1 --2- 172.21.15.28:0/496211199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb13a80 0x56377eb067f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0544004af0 tx=0x7f054402c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e648f07ee429fe6a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f05367fc640 1 -- 172.21.15.28:0/496211199 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05440bed20 con 0x56377eb13a80 2024-04-21T08:51:19.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f05367fc640 1 -- 172.21.15.28:0/496211199 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05440c7990 con 0x56377eb13a80 2024-04-21T08:51:19.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.283+0000 7f05367fc640 1 -- 172.21.15.28:0/496211199 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05440df110 con 0x56377eb13a80 2024-04-21T08:51:19.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.284+0000 7f0555b6f740 1 -- 172.21.15.28:0/496211199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb13a80 msgr2=0x56377eb067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.284+0000 7f0555b6f740 1 --2- 172.21.15.28:0/496211199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb13a80 0x56377eb067f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0544004af0 tx=0x7f054402c230 comp rx=0 tx=0).stop 2024-04-21T08:51:19.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.284+0000 7f0555b6f740 1 -- 172.21.15.28:0/496211199 shutdown_connections 2024-04-21T08:51:19.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.284+0000 7f0555b6f740 1 --2- 172.21.15.28:0/496211199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb13a80 0x56377eb067f0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.284+0000 7f0555b6f740 1 --2- 172.21.15.28:0/496211199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb0aea0 0x56377eb1f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.284+0000 7f0555b6f740 1 --2- 172.21.15.28:0/496211199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb0a580 0x56377eb13160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.284+0000 7f0555b6f740 1 -- 172.21.15.28:0/496211199 >> 172.21.15.28:0/496211199 conn(0x56377eb04cb0 msgr2=0x56377eb060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.284+0000 7f0555b6f740 1 -- 172.21.15.28:0/496211199 shutdown_connections 2024-04-21T08:51:19.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.284+0000 7f0555b6f740 1 -- 172.21.15.28:0/496211199 wait complete. 2024-04-21T08:51:19.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.284+0000 7f0555b6f740 1 -- start start 2024-04-21T08:51:19.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0555b6f740 1 Processor -- start 2024-04-21T08:51:19.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0555b6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb0a580 0x56377eb1c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0555b6f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb0aea0 0x56377eb0d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0555b6f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb13a80 0x56377eb0dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0555b6f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f054c1afaa0 con 0x56377eb0a580 2024-04-21T08:51:19.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0555b6f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f054c1b0f10 con 0x56377eb13a80 2024-04-21T08:51:19.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0555b6f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f054c1b0170 con 0x56377eb0aea0 2024-04-21T08:51:19.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0555b6f740 10 client.0 _ll_register_callbacks cb 0x7f05440048d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:19.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0550d1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb0a580 0x56377eb1c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0550d1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb0a580 0x56377eb1c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52218/0 (socket says 172.21.15.28:52218) 2024-04-21T08:51:19.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0550d1a640 1 -- 172.21.15.28:0/159578580 learned_addr learned my addr 172.21.15.28:0/159578580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:19.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0550d1a640 1 -- 172.21.15.28:0/159578580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb0aea0 msgr2=0x56377eb0d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0550d1a640 1 --2- 172.21.15.28:0/159578580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb0aea0 0x56377eb0d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0550d1a640 1 -- 172.21.15.28:0/159578580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb13a80 msgr2=0x56377eb0dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:19.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0550d1a640 1 --2- 172.21.15.28:0/159578580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb13a80 0x56377eb0dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.285+0000 7f0550d1a640 1 -- 172.21.15.28:0/159578580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56377eb89170 con 0x56377eb0a580 2024-04-21T08:51:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f0550d1a640 1 --2- 172.21.15.28:0/159578580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb0a580 0x56377eb1c1e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x56377eb0a050 tx=0x7f05400097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd4a8549d9ffc06 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f052bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f052bfff640 1 -- 172.21.15.28:0/159578580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05400052c0 con 0x56377eb0a580 2024-04-21T08:51:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f052bfff640 1 -- 172.21.15.28:0/159578580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f0540005460 con 0x56377eb0a580 2024-04-21T08:51:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f0555b6f740 1 -- 172.21.15.28:0/159578580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05440c7990 con 0x56377eb0a580 2024-04-21T08:51:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f052bfff640 1 -- 172.21.15.28:0/159578580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f054003a050 con 0x56377eb0a580 2024-04-21T08:51:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f0555b6f740 10 client.15705 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f0555b6f740 1 -- 172.21.15.28:0/159578580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56377eb8a320 con 0x56377eb0a580 2024-04-21T08:51:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f052bfff640 1 -- 172.21.15.28:0/159578580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05440c7990 con 0x56377eb0a580 2024-04-21T08:51:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f052bfff640 1 -- 172.21.15.28:0/159578580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f054002a330 con 0x56377eb0a580 2024-04-21T08:51:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f0555b6f740 10 client.15705 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f0555b6f740 10 client.15705 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f0555b6f740 20 client.15705 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f0555b6f740 1 --2- 172.21.15.28:0/159578580 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56377eb8a790 0x56377eb8cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f0555b6f740 4 client.15705 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:51:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.286+0000 7f0555b6f740 1 -- 172.21.15.28:0/159578580 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56377ea76a70 con 0x56377eb8a790 2024-04-21T08:51:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.287+0000 7f054bfff640 1 --2- 172.21.15.28:0/159578580 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56377eb8a790 0x56377eb8cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f054bfff640 1 --2- 172.21.15.28:0/159578580 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56377eb8a790 0x56377eb8cc10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af9021070092e24c server_cookie=2e80b723ceed0f68 in_seq=0 out_seq=0 2024-04-21T08:51:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f052bfff640 10 client.15705 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f052bfff640 1 -- 172.21.15.28:0/159578580 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56377ea76a70 con 0x56377eb8a790 2024-04-21T08:51:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f052bfff640 10 client.15705 handle_command_reply: tid=0 2024-04-21T08:51:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f0555b6f740 1 client.15705 shutdown 2024-04-21T08:51:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f0555b6f740 1 -- 172.21.15.28:0/159578580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb0a580 msgr2=0x56377eb1c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f0555b6f740 1 --2- 172.21.15.28:0/159578580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb0a580 0x56377eb1c1e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x56377eb0a050 tx=0x7f05400097b0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f0555b6f740 1 -- 172.21.15.28:0/159578580 shutdown_connections 2024-04-21T08:51:19.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f0555b6f740 1 --2- 172.21.15.28:0/159578580 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56377eb8a790 0x56377eb8cc10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f0555b6f740 1 --2- 172.21.15.28:0/159578580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56377eb13a80 0x56377eb0dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f0555b6f740 1 --2- 172.21.15.28:0/159578580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56377eb0aea0 0x56377eb0d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f052bfff640 0 client.15705 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:19.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f0555b6f740 1 --2- 172.21.15.28:0/159578580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377eb0a580 0x56377eb1c1e0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f0555b6f740 1 -- 172.21.15.28:0/159578580 >> 172.21.15.28:0/159578580 conn(0x56377eb04cb0 msgr2=0x56377eb03d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f0555b6f740 1 -- 172.21.15.28:0/159578580 shutdown_connections 2024-04-21T08:51:19.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f0555b6f740 1 -- 172.21.15.28:0/159578580 wait complete. 2024-04-21T08:51:19.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.288+0000 7f0555b6f740 20 client.15705 trim_cache size 0 max 16384 2024-04-21T08:51:19.293 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:51:19.293 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:51:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:51:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "7d56b2cb-87ee-4b4d-84f6-e1e84063a464": { 2024-04-21T08:51:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:51:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "7d56b2cb-87ee-4b4d-84f6-e1e84063a464", 2024-04-21T08:51:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:51:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:51:19.294 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:51:19.294 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:51:19.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.289+0000 7f0552f7e640 1 -- 172.21.15.28:0/1679141036 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f053c0a3e40 msgr2=0x7f053c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.289+0000 7f0552f7e640 1 --2- 172.21.15.28:0/1679141036 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f053c0a3e40 0x7f053c0a6300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x56377eb05e40 tx=0x7f05380046c0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.289+0000 7f0552f7e640 1 -- 172.21.15.28:0/1679141036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f054c10c500 msgr2=0x7f054c101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.289+0000 7f0552f7e640 1 --2- 172.21.15.28:0/1679141036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f054c10c500 0x7f054c101890 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0544037040 tx=0x7f0544035e20 comp rx=0 tx=0).stop 2024-04-21T08:51:19.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.289+0000 7f0552f7e640 1 -- 172.21.15.28:0/1679141036 shutdown_connections 2024-04-21T08:51:19.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.289+0000 7f0552f7e640 1 --2- 172.21.15.28:0/1679141036 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f053c0a3e40 0x7f053c0a6300 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.289+0000 7f0552f7e640 1 --2- 172.21.15.28:0/1679141036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f054c10c500 0x7f054c101890 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.289+0000 7f0552f7e640 1 --2- 172.21.15.28:0/1679141036 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f054c103ae0 0x7f054c101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.289+0000 7f0552f7e640 1 --2- 172.21.15.28:0/1679141036 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f054c1031c0 0x7f054c100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.289+0000 7f0552f7e640 1 -- 172.21.15.28:0/1679141036 >> 172.21.15.28:0/1679141036 conn(0x7f054c078000 msgr2=0x7f054c0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.289+0000 7f0552f7e640 1 -- 172.21.15.28:0/1679141036 shutdown_connections 2024-04-21T08:51:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.289+0000 7f0552f7e640 1 -- 172.21.15.28:0/1679141036 wait complete. 2024-04-21T08:51:19.299 INFO:tasks.cephfs.filesystem:scrub status for tag:7d56b2cb-87ee-4b4d-84f6-e1e84063a464 - {'path': '/', 'tag': '7d56b2cb-87ee-4b4d-84f6-e1e84063a464', 'options': 'recursive,force'} 2024-04-21T08:51:19.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.329+0000 7faf7e0c8640 1 -- 172.21.15.28:0/524960648 --> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7faf78102b80 con 0x7faf600a3ee0 2024-04-21T08:51:19.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:18 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3880471696' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:19.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:18 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/443564397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:19.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:18 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/346528291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:19.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.398+0000 7faf74ff9640 1 -- 172.21.15.28:0/524960648 <== mgr.25040 v2:172.21.15.28:6800/4182301910 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7faf78102b80 con 0x7faf600a3ee0 2024-04-21T08:51:19.398 INFO:teuthology.orchestra.run.smithi028.stdout:Schedule set for path / 2024-04-21T08:51:19.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.400+0000 7faf7e0c8640 1 -- 172.21.15.28:0/524960648 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faf600a3ee0 msgr2=0x7faf600a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.400+0000 7faf7e0c8640 1 --2- 172.21.15.28:0/524960648 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faf600a3ee0 0x7faf600a63a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faf6c000c00 tx=0x7faf6c02c6b0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.400+0000 7faf7e0c8640 1 -- 172.21.15.28:0/524960648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf781072a0 msgr2=0x7faf78198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.400+0000 7faf7e0c8640 1 --2- 172.21.15.28:0/524960648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf781072a0 0x7faf78198c80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7faf6800e9f0 tx=0x7faf6800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.400+0000 7faf7e0c8640 1 -- 172.21.15.28:0/524960648 shutdown_connections 2024-04-21T08:51:19.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.400+0000 7faf7e0c8640 1 --2- 172.21.15.28:0/524960648 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faf600a3ee0 0x7faf600a63a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.400+0000 7faf7e0c8640 1 --2- 172.21.15.28:0/524960648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf7819cfb0 0x7faf7819d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.400+0000 7faf7e0c8640 1 --2- 172.21.15.28:0/524960648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf781072a0 0x7faf78198c80 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.400+0000 7faf7e0c8640 1 --2- 172.21.15.28:0/524960648 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faf781021d0 0x7faf78198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.400+0000 7faf7e0c8640 1 -- 172.21.15.28:0/524960648 >> 172.21.15.28:0/524960648 conn(0x7faf780fda50 msgr2=0x7faf780ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.400+0000 7faf7e0c8640 1 -- 172.21.15.28:0/524960648 shutdown_connections 2024-04-21T08:51:19.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.400+0000 7faf7e0c8640 1 -- 172.21.15.28:0/524960648 wait complete. 2024-04-21T08:51:19.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-21T08:51:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.556+0000 7fc17cd8f640 1 -- 172.21.15.28:0/3312310101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178101b60 msgr2=0x7fc17810dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.556+0000 7fc17cd8f640 1 --2- 172.21.15.28:0/3312310101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178101b60 0x7fc17810dc70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc170009ca0 tx=0x7fc17002c140 comp rx=0 tx=0).stop 2024-04-21T08:51:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.556+0000 7fc17cd8f640 1 -- 172.21.15.28:0/3312310101 shutdown_connections 2024-04-21T08:51:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.556+0000 7fc17cd8f640 1 --2- 172.21.15.28:0/3312310101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178101b60 0x7fc17810dc70 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.556+0000 7fc17cd8f640 1 --2- 172.21.15.28:0/3312310101 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1781011a0 0x7fc178101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.556+0000 7fc17cd8f640 1 --2- 172.21.15.28:0/3312310101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc178106fb0 0x7fc178107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.556+0000 7fc17cd8f640 1 -- 172.21.15.28:0/3312310101 >> 172.21.15.28:0/3312310101 conn(0x7fc178076c50 msgr2=0x7fc178077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.556+0000 7fc17cd8f640 1 -- 172.21.15.28:0/3312310101 shutdown_connections 2024-04-21T08:51:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.556+0000 7fc17cd8f640 1 -- 172.21.15.28:0/3312310101 wait complete. 2024-04-21T08:51:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc17cd8f640 1 Processor -- start 2024-04-21T08:51:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc17cd8f640 1 -- start start 2024-04-21T08:51:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc17cd8f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1781011a0 0x7fc1781985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc17cd8f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc178101b60 0x7fc178198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc17cd8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178106fb0 0x7fc17819cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc17cd8f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1781121e0 con 0x7fc178106fb0 2024-04-21T08:51:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc17cd8f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc178112060 con 0x7fc178101b60 2024-04-21T08:51:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc17cd8f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc178112360 con 0x7fc1781011a0 2024-04-21T08:51:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc176d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178106fb0 0x7fc17819cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc176d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178106fb0 0x7fc17819cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52236/0 (socket says 172.21.15.28:52236) 2024-04-21T08:51:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc176d9d640 1 -- 172.21.15.28:0/18692106 learned_addr learned my addr 172.21.15.28:0/18692106 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc17659c640 1 --2- 172.21.15.28:0/18692106 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1781011a0 0x7fc1781985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc175d9b640 1 --2- 172.21.15.28:0/18692106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc178101b60 0x7fc178198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc176d9d640 1 -- 172.21.15.28:0/18692106 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1781011a0 msgr2=0x7fc1781985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc176d9d640 1 --2- 172.21.15.28:0/18692106 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1781011a0 0x7fc1781985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc176d9d640 1 -- 172.21.15.28:0/18692106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc178101b60 msgr2=0x7fc178198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc176d9d640 1 --2- 172.21.15.28:0/18692106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc178101b60 0x7fc178198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc176d9d640 1 -- 172.21.15.28:0/18692106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17819d5d0 con 0x7fc178106fb0 2024-04-21T08:51:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc175d9b640 1 --2- 172.21.15.28:0/18692106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc178101b60 0x7fc178198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.557+0000 7fc176d9d640 1 --2- 172.21.15.28:0/18692106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178106fb0 0x7fc17819cea0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc17002c750 tx=0x7fc170035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.558+0000 7fc1677fe640 1 -- 172.21.15.28:0/18692106 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc170039040 con 0x7fc178106fb0 2024-04-21T08:51:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.558+0000 7fc1677fe640 1 -- 172.21.15.28:0/18692106 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc170044070 con 0x7fc178106fb0 2024-04-21T08:51:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.558+0000 7fc17cd8f640 1 -- 172.21.15.28:0/18692106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc17819d860 con 0x7fc178106fb0 2024-04-21T08:51:19.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.558+0000 7fc1677fe640 1 -- 172.21.15.28:0/18692106 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1700043c0 con 0x7fc178106fb0 2024-04-21T08:51:19.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.558+0000 7fc17cd8f640 1 -- 172.21.15.28:0/18692106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1781a5150 con 0x7fc178106fb0 2024-04-21T08:51:19.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.559+0000 7fc17cd8f640 1 -- 172.21.15.28:0/18692106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc178102550 con 0x7fc178106fb0 2024-04-21T08:51:19.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.560+0000 7fc1677fe640 1 -- 172.21.15.28:0/18692106 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fc170004560 con 0x7fc178106fb0 2024-04-21T08:51:19.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.560+0000 7fc1677fe640 1 --2- 172.21.15.28:0/18692106 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc1600a3ee0 0x7fc1600a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.561+0000 7fc17659c640 1 --2- 172.21.15.28:0/18692106 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc1600a3ee0 0x7fc1600a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.561+0000 7fc1677fe640 1 -- 172.21.15.28:0/18692106 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fc1700f3490 con 0x7fc178106fb0 2024-04-21T08:51:19.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.561+0000 7fc17659c640 1 --2- 172.21.15.28:0/18692106 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc1600a3ee0 0x7fc1600a63a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc168005fd0 tx=0x7fc16800b590 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.565+0000 7fc1677fe640 1 -- 172.21.15.28:0/18692106 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc1700333a0 con 0x7fc178106fb0 2024-04-21T08:51:19.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.776+0000 7fc17cd8f640 1 -- 172.21.15.28:0/18692106 --> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fc178109330 con 0x7fc1600a3ee0 2024-04-21T08:51:19.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.839+0000 7fc1677fe640 1 -- 172.21.15.28:0/18692106 <== mgr.25040 v2:172.21.15.28:6800/4182301910 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fc16800fb40 con 0x7fc1600a3ee0 2024-04-21T08:51:19.839 INFO:teuthology.orchestra.run.smithi028.stdout:Retention added to path / 2024-04-21T08:51:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.841+0000 7fc17cd8f640 1 -- 172.21.15.28:0/18692106 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc1600a3ee0 msgr2=0x7fc1600a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.841+0000 7fc17cd8f640 1 --2- 172.21.15.28:0/18692106 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc1600a3ee0 0x7fc1600a63a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc168005fd0 tx=0x7fc16800b590 comp rx=0 tx=0).stop 2024-04-21T08:51:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.841+0000 7fc17cd8f640 1 -- 172.21.15.28:0/18692106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178106fb0 msgr2=0x7fc17819cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.841+0000 7fc17cd8f640 1 --2- 172.21.15.28:0/18692106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178106fb0 0x7fc17819cea0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc17002c750 tx=0x7fc170035d60 comp rx=0 tx=0).stop 2024-04-21T08:51:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.841+0000 7fc17cd8f640 1 -- 172.21.15.28:0/18692106 shutdown_connections 2024-04-21T08:51:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.841+0000 7fc17cd8f640 1 --2- 172.21.15.28:0/18692106 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc1600a3ee0 0x7fc1600a63a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.841+0000 7fc17cd8f640 1 --2- 172.21.15.28:0/18692106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc178106fb0 0x7fc17819cea0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.841+0000 7fc17cd8f640 1 --2- 172.21.15.28:0/18692106 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc178101b60 0x7fc178198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.841+0000 7fc17cd8f640 1 --2- 172.21.15.28:0/18692106 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1781011a0 0x7fc1781985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.841+0000 7fc17cd8f640 1 -- 172.21.15.28:0/18692106 >> 172.21.15.28:0/18692106 conn(0x7fc178076c50 msgr2=0x7fc1780ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.841+0000 7fc17cd8f640 1 -- 172.21.15.28:0/18692106 shutdown_connections 2024-04-21T08:51:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.841+0000 7fc17cd8f640 1 -- 172.21.15.28:0/18692106 wait complete. 2024-04-21T08:51:19.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-21T08:51:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.954+0000 7f337d06d640 1 -- 172.21.15.28:0/2268326638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33781031e0 msgr2=0x7f33781035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.954+0000 7f337d06d640 1 --2- 172.21.15.28:0/2268326638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33781031e0 0x7f33781035c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3370009ca0 tx=0x7f337002c230 comp rx=0 tx=0).stop 2024-04-21T08:51:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.954+0000 7f337d06d640 1 -- 172.21.15.28:0/2268326638 shutdown_connections 2024-04-21T08:51:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.954+0000 7f337d06d640 1 --2- 172.21.15.28:0/2268326638 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f337810c520 0x7f337810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.954+0000 7f337d06d640 1 --2- 172.21.15.28:0/2268326638 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3378103b00 0x7f337810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.954+0000 7f337d06d640 1 --2- 172.21.15.28:0/2268326638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33781031e0 0x7f33781035c0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.954+0000 7f337d06d640 1 -- 172.21.15.28:0/2268326638 >> 172.21.15.28:0/2268326638 conn(0x7f3378077fd0 msgr2=0x7f33780783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.954+0000 7f337d06d640 1 -- 172.21.15.28:0/2268326638 shutdown_connections 2024-04-21T08:51:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.954+0000 7f337d06d640 1 -- 172.21.15.28:0/2268326638 wait complete. 2024-04-21T08:51:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.955+0000 7f337d06d640 1 Processor -- start 2024-04-21T08:51:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.955+0000 7f337d06d640 1 -- start start 2024-04-21T08:51:19.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f337d06d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33781031e0 0x7f3378100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f337d06d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3378103b00 0x7f3378101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f337d06d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f337810c520 0x7f33781018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f337d06d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3378112230 con 0x7f33781031e0 2024-04-21T08:51:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f337d06d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f33781120b0 con 0x7f337810c520 2024-04-21T08:51:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f337d06d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f33781123b0 con 0x7f3378103b00 2024-04-21T08:51:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f3376d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33781031e0 0x7f3378100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f3376d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33781031e0 0x7f3378100e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52250/0 (socket says 172.21.15.28:52250) 2024-04-21T08:51:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f3376d9d640 1 -- 172.21.15.28:0/3518342417 learned_addr learned my addr 172.21.15.28:0/3518342417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f337759e640 1 --2- 172.21.15.28:0/3518342417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f337810c520 0x7f33781018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f337659c640 1 --2- 172.21.15.28:0/3518342417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3378103b00 0x7f3378101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f3376d9d640 1 -- 172.21.15.28:0/3518342417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3378103b00 msgr2=0x7f3378101370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f3376d9d640 1 --2- 172.21.15.28:0/3518342417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3378103b00 0x7f3378101370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f3376d9d640 1 -- 172.21.15.28:0/3518342417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f337810c520 msgr2=0x7f33781018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f3376d9d640 1 --2- 172.21.15.28:0/3518342417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f337810c520 0x7f33781018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f3376d9d640 1 -- 172.21.15.28:0/3518342417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33781a6c70 con 0x7f33781031e0 2024-04-21T08:51:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f3376d9d640 1 --2- 172.21.15.28:0/3518342417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33781031e0 0x7f3378100e30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3370005e30 tx=0x7f3370003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.956+0000 7f336ffff640 1 -- 172.21.15.28:0/3518342417 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3370039040 con 0x7f33781031e0 2024-04-21T08:51:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.957+0000 7f337d06d640 1 -- 172.21.15.28:0/3518342417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33781a6f00 con 0x7f33781031e0 2024-04-21T08:51:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.957+0000 7f336ffff640 1 -- 172.21.15.28:0/3518342417 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3370044070 con 0x7f33781031e0 2024-04-21T08:51:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.957+0000 7f336ffff640 1 -- 172.21.15.28:0/3518342417 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3370032880 con 0x7f33781031e0 2024-04-21T08:51:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.957+0000 7f337d06d640 1 -- 172.21.15.28:0/3518342417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33781a7370 con 0x7f33781031e0 2024-04-21T08:51:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.958+0000 7f337d06d640 1 -- 172.21.15.28:0/3518342417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f337804fce0 con 0x7f33781031e0 2024-04-21T08:51:19.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.959+0000 7f336ffff640 1 -- 172.21.15.28:0/3518342417 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f3370032a20 con 0x7f33781031e0 2024-04-21T08:51:19.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.959+0000 7f336ffff640 1 --2- 172.21.15.28:0/3518342417 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f33640a3f10 0x7f33640a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:19.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.959+0000 7f337659c640 1 --2- 172.21.15.28:0/3518342417 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f33640a3f10 0x7f33640a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:19.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.960+0000 7f336ffff640 1 -- 172.21.15.28:0/3518342417 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f33700f33a0 con 0x7f33781031e0 2024-04-21T08:51:19.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.960+0000 7f337659c640 1 --2- 172.21.15.28:0/3518342417 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f33640a3f10 0x7f33640a63d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3360005fd0 tx=0x7f3360005850 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:19.964+0000 7f336ffff640 1 -- 172.21.15.28:0/3518342417 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f33700bf560 con 0x7f33781031e0 2024-04-21T08:51:20.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:19 smithi028 ceph-mon[28003]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-04-21T08:51:20.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:19 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3919955544' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:20.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:19 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:51:20.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:19 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:20.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:19 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:51:20.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:19 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:20.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:19 smithi028 ceph-mon[28003]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:51:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:19 smithi171 ceph-mon[31358]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-04-21T08:51:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:19 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3919955544' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:19 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:51:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:19 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:19 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:51:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:19 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:19 smithi171 ceph-mon[31358]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:51:20.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.172+0000 7f337d06d640 1 -- 172.21.15.28:0/3518342417 --> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f3378062390 con 0x7f33640a3f10 2024-04-21T08:51:20.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.180+0000 7f336ffff640 1 -- 172.21.15.28:0/3518342417 <== mgr.25040 v2:172.21.15.28:6800/4182301910 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f3378062390 con 0x7f33640a3f10 2024-04-21T08:51:20.180 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-21T00:00:00", "created": "2024-04-21T08:51:19", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-21T08:51:20.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.182+0000 7f337d06d640 1 -- 172.21.15.28:0/3518342417 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f33640a3f10 msgr2=0x7f33640a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:20.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.182+0000 7f337d06d640 1 --2- 172.21.15.28:0/3518342417 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f33640a3f10 0x7f33640a63d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3360005fd0 tx=0x7f3360005850 comp rx=0 tx=0).stop 2024-04-21T08:51:20.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.182+0000 7f337d06d640 1 -- 172.21.15.28:0/3518342417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33781031e0 msgr2=0x7f3378100e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:20.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.182+0000 7f337d06d640 1 --2- 172.21.15.28:0/3518342417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33781031e0 0x7f3378100e30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3370005e30 tx=0x7f3370003c20 comp rx=0 tx=0).stop 2024-04-21T08:51:20.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.182+0000 7f337d06d640 1 -- 172.21.15.28:0/3518342417 shutdown_connections 2024-04-21T08:51:20.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.182+0000 7f337d06d640 1 --2- 172.21.15.28:0/3518342417 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f33640a3f10 0x7f33640a63d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.182+0000 7f337d06d640 1 --2- 172.21.15.28:0/3518342417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f337810c520 0x7f33781018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.182+0000 7f337d06d640 1 --2- 172.21.15.28:0/3518342417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3378103b00 0x7f3378101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.182+0000 7f337d06d640 1 --2- 172.21.15.28:0/3518342417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33781031e0 0x7f3378100e30 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.182+0000 7f337d06d640 1 -- 172.21.15.28:0/3518342417 >> 172.21.15.28:0/3518342417 conn(0x7f3378077fd0 msgr2=0x7f33780fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:20.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.182+0000 7f337d06d640 1 -- 172.21.15.28:0/3518342417 shutdown_connections 2024-04-21T08:51:20.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.182+0000 7f337d06d640 1 -- 172.21.15.28:0/3518342417 wait complete. 2024-04-21T08:51:20.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-21T08:51:20.256 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:19 smithi152 ceph-mon[31401]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-04-21T08:51:20.256 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:19 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3919955544' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:20.256 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:19 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:51:20.256 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:19 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:20.256 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:19 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:51:20.256 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:19 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:20.256 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:19 smithi152 ceph-mon[31401]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:51:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.297+0000 7ff0cf766640 1 -- 172.21.15.28:0/3363456153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c81021d0 msgr2=0x7ff0c81025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.297+0000 7ff0cf766640 1 --2- 172.21.15.28:0/3363456153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c81021d0 0x7ff0c81025b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009900 tx=0x7ff0c002bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.297+0000 7ff0cf766640 1 -- 172.21.15.28:0/3363456153 shutdown_connections 2024-04-21T08:51:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.297+0000 7ff0cf766640 1 --2- 172.21.15.28:0/3363456153 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0c81072a0 0x7ff0c810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.297+0000 7ff0cf766640 1 --2- 172.21.15.28:0/3363456153 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0c8102b80 0x7ff0c8106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.297+0000 7ff0cf766640 1 --2- 172.21.15.28:0/3363456153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c81021d0 0x7ff0c81025b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.297+0000 7ff0cf766640 1 -- 172.21.15.28:0/3363456153 >> 172.21.15.28:0/3363456153 conn(0x7ff0c80793e0 msgr2=0x7ff0c80797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.297+0000 7ff0cf766640 1 -- 172.21.15.28:0/3363456153 shutdown_connections 2024-04-21T08:51:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.297+0000 7ff0cf766640 1 -- 172.21.15.28:0/3363456153 wait complete. 2024-04-21T08:51:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.297+0000 7ff0cf766640 1 Processor -- start 2024-04-21T08:51:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.297+0000 7ff0cf766640 1 -- start start 2024-04-21T08:51:20.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cf766640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0c81021d0 0x7ff0c81a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:20.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cf766640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0c81072a0 0x7ff0c81a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:20.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cf766640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c819a7e0 0x7ff0c819ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:20.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cf766640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c8112260 con 0x7ff0c819a7e0 2024-04-21T08:51:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cf766640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c81120e0 con 0x7ff0c81021d0 2024-04-21T08:51:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cf766640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c81123e0 con 0x7ff0c81072a0 2024-04-21T08:51:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cdd03640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c819a7e0 0x7ff0c819ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cdd03640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c819a7e0 0x7ff0c819ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52272/0 (socket says 172.21.15.28:52272) 2024-04-21T08:51:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cdd03640 1 -- 172.21.15.28:0/2430069735 learned_addr learned my addr 172.21.15.28:0/2430069735 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cd502640 1 --2- 172.21.15.28:0/2430069735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0c81021d0 0x7ff0c81a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0ccd01640 1 --2- 172.21.15.28:0/2430069735 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0c81072a0 0x7ff0c81a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cdd03640 1 -- 172.21.15.28:0/2430069735 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0c81072a0 msgr2=0x7ff0c81a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cdd03640 1 --2- 172.21.15.28:0/2430069735 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0c81072a0 0x7ff0c81a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cdd03640 1 -- 172.21.15.28:0/2430069735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0c81021d0 msgr2=0x7ff0c81a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cdd03640 1 --2- 172.21.15.28:0/2430069735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0c81021d0 0x7ff0c81a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cdd03640 1 -- 172.21.15.28:0/2430069735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c819b370 con 0x7ff0c819a7e0 2024-04-21T08:51:20.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cdd03640 1 --2- 172.21.15.28:0/2430069735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c819a7e0 0x7ff0c819ac90 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400ea60 tx=0x7ff0c400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:20.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0be7fc640 1 -- 172.21.15.28:0/2430069735 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c400cd80 con 0x7ff0c819a7e0 2024-04-21T08:51:20.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0be7fc640 1 -- 172.21.15.28:0/2430069735 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0c4005020 con 0x7ff0c819a7e0 2024-04-21T08:51:20.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cf766640 1 -- 172.21.15.28:0/2430069735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c819b660 con 0x7ff0c819a7e0 2024-04-21T08:51:20.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0be7fc640 1 -- 172.21.15.28:0/2430069735 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4010690 con 0x7ff0c819a7e0 2024-04-21T08:51:20.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.298+0000 7ff0cf766640 1 -- 172.21.15.28:0/2430069735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c81a77f0 con 0x7ff0c819a7e0 2024-04-21T08:51:20.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.299+0000 7ff0cf766640 1 -- 172.21.15.28:0/2430069735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0c81036a0 con 0x7ff0c819a7e0 2024-04-21T08:51:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.300+0000 7ff0be7fc640 1 -- 172.21.15.28:0/2430069735 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7ff0c40054d0 con 0x7ff0c819a7e0 2024-04-21T08:51:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.301+0000 7ff0be7fc640 1 --2- 172.21.15.28:0/2430069735 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff0b40a3ee0 0x7ff0b40a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.301+0000 7ff0cd502640 1 --2- 172.21.15.28:0/2430069735 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff0b40a3ee0 0x7ff0b40a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.301+0000 7ff0be7fc640 1 -- 172.21.15.28:0/2430069735 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ff0c40d1310 con 0x7ff0c819a7e0 2024-04-21T08:51:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.302+0000 7ff0cd502640 1 --2- 172.21.15.28:0/2430069735 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff0b40a3ee0 0x7ff0b40a63a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0006010 tx=0x7ff0c0005e40 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:20.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.306+0000 7ff0be7fc640 1 -- 172.21.15.28:0/2430069735 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff0c819b660 con 0x7ff0c819a7e0 2024-04-21T08:51:20.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.515+0000 7ff0cf766640 1 -- 172.21.15.28:0/2430069735 --> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7ff0c8102c60 con 0x7ff0b40a3ee0 2024-04-21T08:51:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.521+0000 7ff0be7fc640 1 -- 172.21.15.28:0/2430069735 <== mgr.25040 v2:172.21.15.28:6800/4182301910 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7ff0c8102c60 con 0x7ff0b40a3ee0 2024-04-21T08:51:20.521 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-04-21T08:51:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.523+0000 7ff0cf766640 1 -- 172.21.15.28:0/2430069735 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff0b40a3ee0 msgr2=0x7ff0b40a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.523+0000 7ff0cf766640 1 --2- 172.21.15.28:0/2430069735 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff0b40a3ee0 0x7ff0b40a63a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0006010 tx=0x7ff0c0005e40 comp rx=0 tx=0).stop 2024-04-21T08:51:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.523+0000 7ff0cf766640 1 -- 172.21.15.28:0/2430069735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c819a7e0 msgr2=0x7ff0c819ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.523+0000 7ff0cf766640 1 --2- 172.21.15.28:0/2430069735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c819a7e0 0x7ff0c819ac90 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400ea60 tx=0x7ff0c400ee60 comp rx=0 tx=0).stop 2024-04-21T08:51:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.523+0000 7ff0cf766640 1 -- 172.21.15.28:0/2430069735 shutdown_connections 2024-04-21T08:51:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.523+0000 7ff0cf766640 1 --2- 172.21.15.28:0/2430069735 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff0b40a3ee0 0x7ff0b40a63a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.523+0000 7ff0cf766640 1 --2- 172.21.15.28:0/2430069735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0c819a7e0 0x7ff0c819ac90 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.523+0000 7ff0cf766640 1 --2- 172.21.15.28:0/2430069735 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0c81072a0 0x7ff0c81a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.523+0000 7ff0cf766640 1 --2- 172.21.15.28:0/2430069735 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0c81021d0 0x7ff0c81a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.523+0000 7ff0cf766640 1 -- 172.21.15.28:0/2430069735 >> 172.21.15.28:0/2430069735 conn(0x7ff0c80793e0 msgr2=0x7ff0c80ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.523+0000 7ff0cf766640 1 -- 172.21.15.28:0/2430069735 shutdown_connections 2024-04-21T08:51:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.523+0000 7ff0cf766640 1 -- 172.21.15.28:0/2430069735 wait complete. 2024-04-21T08:51:20.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-21T08:51:20.562 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-21T08:51:20.573 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-21T08:51:20.582 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-21T08:51:20.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-21T08:51:20.605 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-21T08:51:20.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:51:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.720+0000 7f837d706640 1 -- 172.21.15.28:0/3206128360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83781072e0 msgr2=0x7f837810ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.720+0000 7f837d706640 1 --2- 172.21.15.28:0/3206128360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83781072e0 0x7f837810ee60 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8370009ca0 tx=0x7f837002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.720+0000 7f837d706640 1 -- 172.21.15.28:0/3206128360 shutdown_connections 2024-04-21T08:51:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.720+0000 7f837d706640 1 --2- 172.21.15.28:0/3206128360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83781072e0 0x7f837810ee60 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.720+0000 7f837d706640 1 --2- 172.21.15.28:0/3206128360 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8378101080 0x7f8378106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.720+0000 7f837d706640 1 --2- 172.21.15.28:0/3206128360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83781006d0 0x7f8378100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.720+0000 7f837d706640 1 -- 172.21.15.28:0/3206128360 >> 172.21.15.28:0/3206128360 conn(0x7f8378077420 msgr2=0x7f83780fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.720+0000 7f837d706640 1 -- 172.21.15.28:0/3206128360 shutdown_connections 2024-04-21T08:51:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.720+0000 7f837d706640 1 -- 172.21.15.28:0/3206128360 wait complete. 2024-04-21T08:51:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.721+0000 7f837d706640 1 Processor -- start 2024-04-21T08:51:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.721+0000 7f837d706640 1 -- start start 2024-04-21T08:51:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.721+0000 7f837d706640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83781006d0 0x7f8378101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.721+0000 7f837d706640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83781072e0 0x7f8378101b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.721+0000 7f837d706640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83781020c0 0x7f83781a6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.721+0000 7f837d706640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8378112200 con 0x7f83781020c0 2024-04-21T08:51:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.721+0000 7f837d706640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8378112080 con 0x7f83781072e0 2024-04-21T08:51:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.721+0000 7f837d706640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8378112380 con 0x7f83781006d0 2024-04-21T08:51:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f83777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83781020c0 0x7f83781a6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f83777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83781020c0 0x7f83781a6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52288/0 (socket says 172.21.15.28:52288) 2024-04-21T08:51:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f83777fe640 1 -- 172.21.15.28:0/2458385470 learned_addr learned my addr 172.21.15.28:0/2458385470 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f8376ffd640 1 --2- 172.21.15.28:0/2458385470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83781006d0 0x7f8378101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f83767fc640 1 --2- 172.21.15.28:0/2458385470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83781072e0 0x7f8378101b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f83777fe640 1 -- 172.21.15.28:0/2458385470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83781006d0 msgr2=0x7f8378101640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f83777fe640 1 --2- 172.21.15.28:0/2458385470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83781006d0 0x7f8378101640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f83777fe640 1 -- 172.21.15.28:0/2458385470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83781072e0 msgr2=0x7f8378101b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f83777fe640 1 --2- 172.21.15.28:0/2458385470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83781072e0 0x7f8378101b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f83777fe640 1 -- 172.21.15.28:0/2458385470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83781a7070 con 0x7f83781020c0 2024-04-21T08:51:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f83777fe640 1 --2- 172.21.15.28:0/2458385470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83781020c0 0x7f83781a6970 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8370005e30 tx=0x7f8370034650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f835ffff640 1 -- 172.21.15.28:0/2458385470 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8370041020 con 0x7f83781020c0 2024-04-21T08:51:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f835ffff640 1 -- 172.21.15.28:0/2458385470 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8370032970 con 0x7f83781020c0 2024-04-21T08:51:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f837d706640 1 -- 172.21.15.28:0/2458385470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83781a7300 con 0x7f83781020c0 2024-04-21T08:51:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f835ffff640 1 -- 172.21.15.28:0/2458385470 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8370039040 con 0x7f83781020c0 2024-04-21T08:51:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.722+0000 7f837d706640 1 -- 172.21.15.28:0/2458385470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83781a77f0 con 0x7f83781020c0 2024-04-21T08:51:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.723+0000 7f837d706640 1 -- 172.21.15.28:0/2458385470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8378102b70 con 0x7f83781020c0 2024-04-21T08:51:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.724+0000 7f835ffff640 1 -- 172.21.15.28:0/2458385470 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f8370044020 con 0x7f83781020c0 2024-04-21T08:51:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.725+0000 7f835ffff640 1 --2- 172.21.15.28:0/2458385470 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f83600a3e70 0x7f83600a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.725+0000 7f835ffff640 1 -- 172.21.15.28:0/2458385470 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f83700f2a30 con 0x7f83781020c0 2024-04-21T08:51:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.725+0000 7f8376ffd640 1 --2- 172.21.15.28:0/2458385470 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f83600a3e70 0x7f83600a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.725+0000 7f8376ffd640 1 --2- 172.21.15.28:0/2458385470 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f83600a3e70 0x7f83600a6330 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8368009b60 tx=0x7f8368005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.729+0000 7f835ffff640 1 -- 172.21.15.28:0/2458385470 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f837004b020 con 0x7f83781020c0 2024-04-21T08:51:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.920+0000 7f837d706640 1 -- 172.21.15.28:0/2458385470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f837810a400 con 0x7f83781020c0 2024-04-21T08:51:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.922+0000 7f835ffff640 1 -- 172.21.15.28:0/2458385470 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25160 (secure 0 0 0) 0x7f83700bebf0 con 0x7f83781020c0 2024-04-21T08:51:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:20.922 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:51:10.823421+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1076282159":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6800/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/926389767":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/892059008":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/2132574390":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:51:20.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.924+0000 7f837d706640 1 -- 172.21.15.28:0/2458385470 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f83600a3e70 msgr2=0x7f83600a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:20.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.924+0000 7f837d706640 1 --2- 172.21.15.28:0/2458385470 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f83600a3e70 0x7f83600a6330 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8368009b60 tx=0x7f8368005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:20.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.924+0000 7f837d706640 1 -- 172.21.15.28:0/2458385470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83781020c0 msgr2=0x7f83781a6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:20.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.924+0000 7f837d706640 1 --2- 172.21.15.28:0/2458385470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83781020c0 0x7f83781a6970 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8370005e30 tx=0x7f8370034650 comp rx=0 tx=0).stop 2024-04-21T08:51:20.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.924+0000 7f837d706640 1 -- 172.21.15.28:0/2458385470 shutdown_connections 2024-04-21T08:51:20.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.924+0000 7f837d706640 1 --2- 172.21.15.28:0/2458385470 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f83600a3e70 0x7f83600a6330 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.924+0000 7f837d706640 1 --2- 172.21.15.28:0/2458385470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83781020c0 0x7f83781a6970 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.924+0000 7f837d706640 1 --2- 172.21.15.28:0/2458385470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f83781072e0 0x7f8378101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.924+0000 7f837d706640 1 --2- 172.21.15.28:0/2458385470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83781006d0 0x7f8378101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:20.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.924+0000 7f837d706640 1 -- 172.21.15.28:0/2458385470 >> 172.21.15.28:0/2458385470 conn(0x7f8378077420 msgr2=0x7f8378105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:20.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.924+0000 7f837d706640 1 -- 172.21.15.28:0/2458385470 shutdown_connections 2024-04-21T08:51:20.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:20.924+0000 7f837d706640 1 -- 172.21.15.28:0/2458385470 wait complete. 2024-04-21T08:51:20.935 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:48:43.543799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:48:44.864564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:48:48.747842+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:51:20.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:51:21.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.040+0000 7f34d4a70640 1 -- 172.21.15.28:0/2451971793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d0106f30 msgr2=0x7f34d0107310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.040+0000 7f34d4a70640 1 --2- 172.21.15.28:0/2451971793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d0106f30 0x7f34d0107310 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009930 tx=0x7f34c002bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.041+0000 7f34d4a70640 1 -- 172.21.15.28:0/2451971793 shutdown_connections 2024-04-21T08:51:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.041+0000 7f34d4a70640 1 --2- 172.21.15.28:0/2451971793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34d0100e10 0x7f34d010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.041+0000 7f34d4a70640 1 --2- 172.21.15.28:0/2451971793 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f34d0100450 0x7f34d01008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.041+0000 7f34d4a70640 1 --2- 172.21.15.28:0/2451971793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d0106f30 0x7f34d0107310 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.041+0000 7f34d4a70640 1 -- 172.21.15.28:0/2451971793 >> 172.21.15.28:0/2451971793 conn(0x7f34d0077410 msgr2=0x7f34d00fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.041+0000 7f34d4a70640 1 -- 172.21.15.28:0/2451971793 shutdown_connections 2024-04-21T08:51:21.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.041+0000 7f34d4a70640 1 -- 172.21.15.28:0/2451971793 wait complete. 2024-04-21T08:51:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.041+0000 7f34d4a70640 1 Processor -- start 2024-04-21T08:51:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.041+0000 7f34d4a70640 1 -- start start 2024-04-21T08:51:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34d4a70640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34d0100450 0x7f34d019e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34d4a70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d0100e10 0x7f34d019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34d4a70640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f34d0106f30 0x7f34d0198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34d4a70640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34d0112270 con 0x7f34d0100e10 2024-04-21T08:51:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34d4a70640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f34d01120f0 con 0x7f34d0100450 2024-04-21T08:51:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34d4a70640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f34d01123f0 con 0x7f34d0106f30 2024-04-21T08:51:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34cdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d0100e10 0x7f34d019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34cdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d0100e10 0x7f34d019eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52302/0 (socket says 172.21.15.28:52302) 2024-04-21T08:51:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34cdd9b640 1 -- 172.21.15.28:0/1230965900 learned_addr learned my addr 172.21.15.28:0/1230965900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34ce59c640 1 --2- 172.21.15.28:0/1230965900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34d0100450 0x7f34d019e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34ced9d640 1 --2- 172.21.15.28:0/1230965900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f34d0106f30 0x7f34d0198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34cdd9b640 1 -- 172.21.15.28:0/1230965900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f34d0106f30 msgr2=0x7f34d0198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34cdd9b640 1 --2- 172.21.15.28:0/1230965900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f34d0106f30 0x7f34d0198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34cdd9b640 1 -- 172.21.15.28:0/1230965900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34d0100450 msgr2=0x7f34d019e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34cdd9b640 1 --2- 172.21.15.28:0/1230965900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34d0100450 0x7f34d019e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34cdd9b640 1 -- 172.21.15.28:0/1230965900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d0198ef0 con 0x7f34d0100e10 2024-04-21T08:51:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34cdd9b640 1 --2- 172.21.15.28:0/1230965900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d0100e10 0x7f34d019eab0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f34c400e9f0 tx=0x7f34c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34bf7fe640 1 -- 172.21.15.28:0/1230965900 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c400cd80 con 0x7f34d0100e10 2024-04-21T08:51:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34bf7fe640 1 -- 172.21.15.28:0/1230965900 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34c40108e0 con 0x7f34d0100e10 2024-04-21T08:51:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34d4a70640 1 -- 172.21.15.28:0/1230965900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d01991e0 con 0x7f34d0100e10 2024-04-21T08:51:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34bf7fe640 1 -- 172.21.15.28:0/1230965900 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c4018980 con 0x7f34d0100e10 2024-04-21T08:51:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.042+0000 7f34d4a70640 1 -- 172.21.15.28:0/1230965900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d01a54f0 con 0x7f34d0100e10 2024-04-21T08:51:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.043+0000 7f34d4a70640 1 -- 172.21.15.28:0/1230965900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34d0103770 con 0x7f34d0100e10 2024-04-21T08:51:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.045+0000 7f34bf7fe640 1 -- 172.21.15.28:0/1230965900 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f34c4010430 con 0x7f34d0100e10 2024-04-21T08:51:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.045+0000 7f34bf7fe640 1 --2- 172.21.15.28:0/1230965900 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f34b80a3e40 0x7f34b80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.045+0000 7f34bf7fe640 1 -- 172.21.15.28:0/1230965900 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f34c40d15e0 con 0x7f34d0100e10 2024-04-21T08:51:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.045+0000 7f34ce59c640 1 --2- 172.21.15.28:0/1230965900 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f34b80a3e40 0x7f34b80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.046+0000 7f34ce59c640 1 --2- 172.21.15.28:0/1230965900 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f34b80a3e40 0x7f34b80a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f34c0005ab0 tx=0x7f34c0005fd0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.049+0000 7f34bf7fe640 1 -- 172.21.15.28:0/1230965900 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f34c409d040 con 0x7f34d0100e10 2024-04-21T08:51:21.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:20 smithi171 ceph-mon[31358]: from='client.15711 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:51:21.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:20 smithi171 ceph-mon[31358]: from='client.15717 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:51:21.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.236+0000 7f34d4a70640 1 -- 172.21.15.28:0/1230965900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f34d010a410 con 0x7f34d0100e10 2024-04-21T08:51:21.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.236+0000 7f34bf7fe640 1 -- 172.21.15.28:0/1230965900 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f34c40a1340 con 0x7f34d0100e10 2024-04-21T08:51:21.236 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T08:51:21.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.238+0000 7f34d4a70640 1 -- 172.21.15.28:0/1230965900 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f34b80a3e40 msgr2=0x7f34b80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.238+0000 7f34d4a70640 1 --2- 172.21.15.28:0/1230965900 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f34b80a3e40 0x7f34b80a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f34c0005ab0 tx=0x7f34c0005fd0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.238+0000 7f34d4a70640 1 -- 172.21.15.28:0/1230965900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d0100e10 msgr2=0x7f34d019eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.238+0000 7f34d4a70640 1 --2- 172.21.15.28:0/1230965900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d0100e10 0x7f34d019eab0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f34c400e9f0 tx=0x7f34c400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.239+0000 7f34d4a70640 1 -- 172.21.15.28:0/1230965900 shutdown_connections 2024-04-21T08:51:21.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.239+0000 7f34d4a70640 1 --2- 172.21.15.28:0/1230965900 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f34b80a3e40 0x7f34b80a6300 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.239+0000 7f34d4a70640 1 --2- 172.21.15.28:0/1230965900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f34d0106f30 0x7f34d0198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.239+0000 7f34d4a70640 1 --2- 172.21.15.28:0/1230965900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d0100e10 0x7f34d019eab0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.239+0000 7f34d4a70640 1 --2- 172.21.15.28:0/1230965900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f34d0100450 0x7f34d019e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.239+0000 7f34d4a70640 1 -- 172.21.15.28:0/1230965900 >> 172.21.15.28:0/1230965900 conn(0x7f34d0077410 msgr2=0x7f34d0105220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:21.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.239+0000 7f34d4a70640 1 -- 172.21.15.28:0/1230965900 shutdown_connections 2024-04-21T08:51:21.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.239+0000 7f34d4a70640 1 -- 172.21.15.28:0/1230965900 wait complete. 2024-04-21T08:51:21.248 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:20 smithi028 ceph-mon[28003]: from='client.15711 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:51:21.248 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:20 smithi028 ceph-mon[28003]: from='client.15717 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:51:21.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:51:21.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:20 smithi152 ceph-mon[31401]: from='client.15711 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:51:21.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:20 smithi152 ceph-mon[31401]: from='client.15717 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:51:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.355+0000 7f535a952640 1 -- 172.21.15.28:0/4256942500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f535406b1f0 msgr2=0x7f5354106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.355+0000 7f535a952640 1 --2- 172.21.15.28:0/4256942500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f535406b1f0 0x7f5354106b90 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f533c009900 tx=0x7f533c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.355+0000 7f535a952640 1 -- 172.21.15.28:0/4256942500 shutdown_connections 2024-04-21T08:51:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.355+0000 7f535a952640 1 --2- 172.21.15.28:0/4256942500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5354107270 0x7f535410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.355+0000 7f535a952640 1 --2- 172.21.15.28:0/4256942500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f535406b1f0 0x7f5354106b90 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.355+0000 7f535a952640 1 --2- 172.21.15.28:0/4256942500 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f535406a840 0x7f535406ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.355+0000 7f535a952640 1 -- 172.21.15.28:0/4256942500 >> 172.21.15.28:0/4256942500 conn(0x7f5354077eb0 msgr2=0x7f53540782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.355+0000 7f535a952640 1 -- 172.21.15.28:0/4256942500 shutdown_connections 2024-04-21T08:51:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.355+0000 7f535a952640 1 -- 172.21.15.28:0/4256942500 wait complete. 2024-04-21T08:51:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.356+0000 7f535a952640 1 Processor -- start 2024-04-21T08:51:21.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.356+0000 7f535a952640 1 -- start start 2024-04-21T08:51:21.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.356+0000 7f535a952640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f535406a840 0x7f5354198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.356+0000 7f535a952640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5354107270 0x7f5354198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.356+0000 7f535a952640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f535419cff0 0x7f535419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.356+0000 7f535a952640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53541121b0 con 0x7f5354107270 2024-04-21T08:51:21.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.356+0000 7f535a952640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5354112030 con 0x7f535419cff0 2024-04-21T08:51:21.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.356+0000 7f535a952640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5354112330 con 0x7f535406a840 2024-04-21T08:51:21.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.356+0000 7f53537fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5354107270 0x7f5354198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.356+0000 7f53537fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5354107270 0x7f5354198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52330/0 (socket says 172.21.15.28:52330) 2024-04-21T08:51:21.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.356+0000 7f53537fe640 1 -- 172.21.15.28:0/2075696912 learned_addr learned my addr 172.21.15.28:0/2075696912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:21.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.356+0000 7f5358eef640 1 --2- 172.21.15.28:0/2075696912 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f535419cff0 0x7f535419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.357+0000 7f5353fff640 1 --2- 172.21.15.28:0/2075696912 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f535406a840 0x7f5354198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.357+0000 7f53537fe640 1 -- 172.21.15.28:0/2075696912 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f535406a840 msgr2=0x7f5354198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.357+0000 7f53537fe640 1 --2- 172.21.15.28:0/2075696912 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f535406a840 0x7f5354198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.357+0000 7f53537fe640 1 -- 172.21.15.28:0/2075696912 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f535419cff0 msgr2=0x7f535419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.357+0000 7f53537fe640 1 --2- 172.21.15.28:0/2075696912 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f535419cff0 0x7f535419d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.357+0000 7f53537fe640 1 -- 172.21.15.28:0/2075696912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f535419dae0 con 0x7f5354107270 2024-04-21T08:51:21.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.357+0000 7f53537fe640 1 --2- 172.21.15.28:0/2075696912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5354107270 0x7f5354198c70 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f533c036040 tx=0x7f533c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:21.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.357+0000 7f53517fa640 1 -- 172.21.15.28:0/2075696912 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c043020 con 0x7f5354107270 2024-04-21T08:51:21.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.357+0000 7f535a952640 1 -- 172.21.15.28:0/2075696912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f535419dd70 con 0x7f5354107270 2024-04-21T08:51:21.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.357+0000 7f53517fa640 1 -- 172.21.15.28:0/2075696912 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f533c02cb70 con 0x7f5354107270 2024-04-21T08:51:21.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.357+0000 7f53517fa640 1 -- 172.21.15.28:0/2075696912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c003bf0 con 0x7f5354107270 2024-04-21T08:51:21.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.357+0000 7f535a952640 1 -- 172.21.15.28:0/2075696912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53541a5590 con 0x7f5354107270 2024-04-21T08:51:21.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.358+0000 7f535a952640 1 -- 172.21.15.28:0/2075696912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53540691c0 con 0x7f5354107270 2024-04-21T08:51:21.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.359+0000 7f53517fa640 1 -- 172.21.15.28:0/2075696912 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f533c046020 con 0x7f5354107270 2024-04-21T08:51:21.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.360+0000 7f53517fa640 1 --2- 172.21.15.28:0/2075696912 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f53400a3e40 0x7f53400a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.360+0000 7f5353fff640 1 --2- 172.21.15.28:0/2075696912 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f53400a3e40 0x7f53400a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.361+0000 7f53517fa640 1 -- 172.21.15.28:0/2075696912 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f533c0f33a0 con 0x7f5354107270 2024-04-21T08:51:21.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.361+0000 7f5353fff640 1 --2- 172.21.15.28:0/2075696912 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f53400a3e40 0x7f53400a6300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5344005fd0 tx=0x7f5344005850 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:21.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.364+0000 7f53517fa640 1 -- 172.21.15.28:0/2075696912 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f533c04d020 con 0x7f5354107270 2024-04-21T08:51:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.552+0000 7f535a952640 1 -- 172.21.15.28:0/2075696912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f535406b1f0 con 0x7f5354107270 2024-04-21T08:51:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.553+0000 7f53517fa640 1 -- 172.21.15.28:0/2075696912 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f533c0bf540 con 0x7f5354107270 2024-04-21T08:51:21.552 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:51:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.555+0000 7f535a952640 1 -- 172.21.15.28:0/2075696912 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f53400a3e40 msgr2=0x7f53400a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.555+0000 7f535a952640 1 --2- 172.21.15.28:0/2075696912 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f53400a3e40 0x7f53400a6300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5344005fd0 tx=0x7f5344005850 comp rx=0 tx=0).stop 2024-04-21T08:51:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.555+0000 7f535a952640 1 -- 172.21.15.28:0/2075696912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5354107270 msgr2=0x7f5354198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.555+0000 7f535a952640 1 --2- 172.21.15.28:0/2075696912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5354107270 0x7f5354198c70 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f533c036040 tx=0x7f533c0045e0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.555+0000 7f535a952640 1 -- 172.21.15.28:0/2075696912 shutdown_connections 2024-04-21T08:51:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.555+0000 7f535a952640 1 --2- 172.21.15.28:0/2075696912 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f53400a3e40 0x7f53400a6300 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.555+0000 7f535a952640 1 --2- 172.21.15.28:0/2075696912 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f535419cff0 0x7f535419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.555+0000 7f535a952640 1 --2- 172.21.15.28:0/2075696912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5354107270 0x7f5354198c70 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.555+0000 7f535a952640 1 --2- 172.21.15.28:0/2075696912 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f535406a840 0x7f5354198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.555+0000 7f535a952640 1 -- 172.21.15.28:0/2075696912 >> 172.21.15.28:0/2075696912 conn(0x7f5354077eb0 msgr2=0x7f53540ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.555+0000 7f535a952640 1 -- 172.21.15.28:0/2075696912 shutdown_connections 2024-04-21T08:51:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.555+0000 7f535a952640 1 -- 172.21.15.28:0/2075696912 wait complete. 2024-04-21T08:51:21.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:51:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.671+0000 7f29fbbb2640 1 -- 172.21.15.28:0/4016018147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f4107270 msgr2=0x7f29f410ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.671+0000 7f29fbbb2640 1 --2- 172.21.15.28:0/4016018147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f4107270 0x7f29f410ee40 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009ca0 tx=0x7f29ec02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.671+0000 7f29fbbb2640 1 -- 172.21.15.28:0/4016018147 shutdown_connections 2024-04-21T08:51:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.671+0000 7f29fbbb2640 1 --2- 172.21.15.28:0/4016018147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f4107270 0x7f29f410ee40 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.671+0000 7f29fbbb2640 1 --2- 172.21.15.28:0/4016018147 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29f406b1f0 0x7f29f4106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.671+0000 7f29fbbb2640 1 --2- 172.21.15.28:0/4016018147 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29f406a840 0x7f29f406ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.671+0000 7f29fbbb2640 1 -- 172.21.15.28:0/4016018147 >> 172.21.15.28:0/4016018147 conn(0x7f29f4077eb0 msgr2=0x7f29f40782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:21.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.671+0000 7f29fbbb2640 1 -- 172.21.15.28:0/4016018147 shutdown_connections 2024-04-21T08:51:21.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.671+0000 7f29fbbb2640 1 -- 172.21.15.28:0/4016018147 wait complete. 2024-04-21T08:51:21.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.672+0000 7f29fbbb2640 1 Processor -- start 2024-04-21T08:51:21.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.672+0000 7f29fbbb2640 1 -- start start 2024-04-21T08:51:21.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.672+0000 7f29fbbb2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29f406a840 0x7f29f41986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.672+0000 7f29fbbb2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29f4107270 0x7f29f4198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.672+0000 7f29fbbb2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f419cf70 0x7f29f419d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.672+0000 7f29fbbb2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29f41121d0 con 0x7f29f419cf70 2024-04-21T08:51:21.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.672+0000 7f29fbbb2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f29f4112050 con 0x7f29f406a840 2024-04-21T08:51:21.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.672+0000 7f29fbbb2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f29f4112350 con 0x7f29f4107270 2024-04-21T08:51:21.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.672+0000 7f29fa14f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f419cf70 0x7f29f419d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.672+0000 7f29fa14f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f419cf70 0x7f29f419d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52348/0 (socket says 172.21.15.28:52348) 2024-04-21T08:51:21.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.672+0000 7f29fa14f640 1 -- 172.21.15.28:0/2096002446 learned_addr learned my addr 172.21.15.28:0/2096002446 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:21.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.672+0000 7f29f994e640 1 --2- 172.21.15.28:0/2096002446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29f406a840 0x7f29f41986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.672+0000 7f29f914d640 1 --2- 172.21.15.28:0/2096002446 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29f4107270 0x7f29f4198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.673+0000 7f29fa14f640 1 -- 172.21.15.28:0/2096002446 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29f4107270 msgr2=0x7f29f4198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.673+0000 7f29fa14f640 1 --2- 172.21.15.28:0/2096002446 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29f4107270 0x7f29f4198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.673+0000 7f29fa14f640 1 -- 172.21.15.28:0/2096002446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29f406a840 msgr2=0x7f29f41986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.673+0000 7f29fa14f640 1 --2- 172.21.15.28:0/2096002446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29f406a840 0x7f29f41986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.673+0000 7f29fa14f640 1 -- 172.21.15.28:0/2096002446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f419da60 con 0x7f29f419cf70 2024-04-21T08:51:21.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.673+0000 7f29fa14f640 1 --2- 172.21.15.28:0/2096002446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f419cf70 0x7f29f419d420 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f29ec005e30 tx=0x7f29ec034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:21.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.673+0000 7f29eaffd640 1 -- 172.21.15.28:0/2096002446 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ec039040 con 0x7f29f419cf70 2024-04-21T08:51:21.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.673+0000 7f29eaffd640 1 -- 172.21.15.28:0/2096002446 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29ec044070 con 0x7f29f419cf70 2024-04-21T08:51:21.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.673+0000 7f29fbbb2640 1 -- 172.21.15.28:0/2096002446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29f419dcf0 con 0x7f29f419cf70 2024-04-21T08:51:21.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.673+0000 7f29eaffd640 1 -- 172.21.15.28:0/2096002446 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ec003e60 con 0x7f29f419cf70 2024-04-21T08:51:21.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.673+0000 7f29fbbb2640 1 -- 172.21.15.28:0/2096002446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29f41a5510 con 0x7f29f419cf70 2024-04-21T08:51:21.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.674+0000 7f29fbbb2640 1 -- 172.21.15.28:0/2096002446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29f40691c0 con 0x7f29f419cf70 2024-04-21T08:51:21.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.675+0000 7f29eaffd640 1 -- 172.21.15.28:0/2096002446 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f29ec0324c0 con 0x7f29f419cf70 2024-04-21T08:51:21.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.676+0000 7f29eaffd640 1 --2- 172.21.15.28:0/2096002446 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f29e00a3ec0 0x7f29e00a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.676+0000 7f29f994e640 1 --2- 172.21.15.28:0/2096002446 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f29e00a3ec0 0x7f29e00a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.676+0000 7f29eaffd640 1 -- 172.21.15.28:0/2096002446 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f29ec0f2fc0 con 0x7f29f419cf70 2024-04-21T08:51:21.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.676+0000 7f29f994e640 1 --2- 172.21.15.28:0/2096002446 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f29e00a3ec0 0x7f29e00a6380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f29f0005fd0 tx=0x7f29f000b390 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:21.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.680+0000 7f29eaffd640 1 -- 172.21.15.28:0/2096002446 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f29ec0bf120 con 0x7f29f419cf70 2024-04-21T08:51:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.868+0000 7f29fbbb2640 1 -- 172.21.15.28:0/2096002446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f29f406b1f0 con 0x7f29f419cf70 2024-04-21T08:51:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.869+0000 7f29eaffd640 1 -- 172.21.15.28:0/2096002446 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f29ec0bea20 con 0x7f29f419cf70 2024-04-21T08:51:21.869 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-21T08:51:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.871+0000 7f29fbbb2640 1 -- 172.21.15.28:0/2096002446 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f29e00a3ec0 msgr2=0x7f29e00a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.871+0000 7f29fbbb2640 1 --2- 172.21.15.28:0/2096002446 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f29e00a3ec0 0x7f29e00a6380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f29f0005fd0 tx=0x7f29f000b390 comp rx=0 tx=0).stop 2024-04-21T08:51:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.871+0000 7f29fbbb2640 1 -- 172.21.15.28:0/2096002446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f419cf70 msgr2=0x7f29f419d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.871+0000 7f29fbbb2640 1 --2- 172.21.15.28:0/2096002446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f419cf70 0x7f29f419d420 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f29ec005e30 tx=0x7f29ec034760 comp rx=0 tx=0).stop 2024-04-21T08:51:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.871+0000 7f29fbbb2640 1 -- 172.21.15.28:0/2096002446 shutdown_connections 2024-04-21T08:51:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.871+0000 7f29fbbb2640 1 --2- 172.21.15.28:0/2096002446 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f29e00a3ec0 0x7f29e00a6380 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.871+0000 7f29fbbb2640 1 --2- 172.21.15.28:0/2096002446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29f419cf70 0x7f29f419d420 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.871+0000 7f29fbbb2640 1 --2- 172.21.15.28:0/2096002446 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29f4107270 0x7f29f4198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.871+0000 7f29fbbb2640 1 --2- 172.21.15.28:0/2096002446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29f406a840 0x7f29f41986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.871+0000 7f29fbbb2640 1 -- 172.21.15.28:0/2096002446 >> 172.21.15.28:0/2096002446 conn(0x7f29f4077eb0 msgr2=0x7f29f40fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.871+0000 7f29fbbb2640 1 -- 172.21.15.28:0/2096002446 shutdown_connections 2024-04-21T08:51:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.871+0000 7f29fbbb2640 1 -- 172.21.15.28:0/2096002446 wait complete. 2024-04-21T08:51:21.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:51:21.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.987+0000 7f8557db6640 1 -- 172.21.15.28:0/4276438574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85501010a0 msgr2=0x7f8550101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.987+0000 7f8557db6640 1 --2- 172.21.15.28:0/4276438574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85501010a0 0x7f8550101480 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8548009ca0 tx=0x7f854802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.987+0000 7f8557db6640 1 -- 172.21.15.28:0/4276438574 shutdown_connections 2024-04-21T08:51:21.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.987+0000 7f8557db6640 1 --2- 172.21.15.28:0/4276438574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8550106170 0x7f855010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.987+0000 7f8557db6640 1 --2- 172.21.15.28:0/4276438574 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8550101a50 0x7f8550105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.987+0000 7f8557db6640 1 --2- 172.21.15.28:0/4276438574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85501010a0 0x7f8550101480 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.987+0000 7f8557db6640 1 -- 172.21.15.28:0/4276438574 >> 172.21.15.28:0/4276438574 conn(0x7f8550077eb0 msgr2=0x7f85500782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:21.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.987+0000 7f8557db6640 1 -- 172.21.15.28:0/4276438574 shutdown_connections 2024-04-21T08:51:21.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.987+0000 7f8557db6640 1 -- 172.21.15.28:0/4276438574 wait complete. 2024-04-21T08:51:21.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8557db6640 1 Processor -- start 2024-04-21T08:51:21.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8557db6640 1 -- start start 2024-04-21T08:51:21.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8557db6640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85501010a0 0x7f855019e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8557db6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8550106170 0x7f855019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8557db6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8550198630 0x7f8550198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8557db6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85501122b0 con 0x7f8550198630 2024-04-21T08:51:21.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8557db6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8550112130 con 0x7f8550106170 2024-04-21T08:51:21.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8557db6640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8550112430 con 0x7f85501010a0 2024-04-21T08:51:21.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8556353640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8550198630 0x7f8550198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8556353640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8550198630 0x7f8550198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52364/0 (socket says 172.21.15.28:52364) 2024-04-21T08:51:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8556353640 1 -- 172.21.15.28:0/739104186 learned_addr learned my addr 172.21.15.28:0/739104186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8555b52640 1 --2- 172.21.15.28:0/739104186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85501010a0 0x7f855019e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8555351640 1 --2- 172.21.15.28:0/739104186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8550106170 0x7f855019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8556353640 1 -- 172.21.15.28:0/739104186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85501010a0 msgr2=0x7f855019e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8556353640 1 --2- 172.21.15.28:0/739104186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85501010a0 0x7f855019e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8556353640 1 -- 172.21.15.28:0/739104186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8550106170 msgr2=0x7f855019eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8556353640 1 --2- 172.21.15.28:0/739104186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8550106170 0x7f855019eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8556353640 1 -- 172.21.15.28:0/739104186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8550199230 con 0x7f8550198630 2024-04-21T08:51:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.988+0000 7f8555351640 1 --2- 172.21.15.28:0/739104186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8550106170 0x7f855019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.989+0000 7f8556353640 1 --2- 172.21.15.28:0/739104186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8550198630 0x7f8550198ab0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8540009fd0 tx=0x7f854000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:21.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.989+0000 7f854effd640 1 -- 172.21.15.28:0/739104186 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540011830 con 0x7f8550198630 2024-04-21T08:51:21.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.989+0000 7f8557db6640 1 -- 172.21.15.28:0/739104186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85501a5310 con 0x7f8550198630 2024-04-21T08:51:21.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.989+0000 7f854effd640 1 -- 172.21.15.28:0/739104186 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8540016070 con 0x7f8550198630 2024-04-21T08:51:21.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.989+0000 7f854effd640 1 -- 172.21.15.28:0/739104186 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540010de0 con 0x7f8550198630 2024-04-21T08:51:21.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.989+0000 7f8557db6640 1 -- 172.21.15.28:0/739104186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85501a56c0 con 0x7f8550198630 2024-04-21T08:51:21.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.990+0000 7f8557db6640 1 -- 172.21.15.28:0/739104186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8550102570 con 0x7f8550198630 2024-04-21T08:51:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.991+0000 7f854effd640 1 -- 172.21.15.28:0/739104186 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f85400119d0 con 0x7f8550198630 2024-04-21T08:51:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.991+0000 7f854effd640 1 --2- 172.21.15.28:0/739104186 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f85380a3e40 0x7f85380a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.992+0000 7f854effd640 1 -- 172.21.15.28:0/739104186 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f85400d0aa0 con 0x7f8550198630 2024-04-21T08:51:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.992+0000 7f8555b52640 1 --2- 172.21.15.28:0/739104186 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f85380a3e40 0x7f85380a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.992+0000 7f8555b52640 1 --2- 172.21.15.28:0/739104186 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f85380a3e40 0x7f85380a6300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8548009430 tx=0x7f8548005c50 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:21.995+0000 7f854effd640 1 -- 172.21.15.28:0/739104186 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f854009cc60 con 0x7f8550198630 2024-04-21T08:51:22.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:21 smithi171 ceph-mon[31358]: from='client.15723 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:51:22.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:21 smithi171 ceph-mon[31358]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 911 B/s wr, 18 op/s 2024-04-21T08:51:22.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:21 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2458385470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:22.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:21 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1230965900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:22.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:21 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2075696912' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.185+0000 7f8557db6640 1 -- 172.21.15.28:0/739104186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f85501093a0 con 0x7f8550198630 2024-04-21T08:51:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.186+0000 7f854effd640 1 -- 172.21.15.28:0/739104186 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f854009c560 con 0x7f8550198630 2024-04-21T08:51:22.186 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:51:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.188+0000 7f8557db6640 1 -- 172.21.15.28:0/739104186 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f85380a3e40 msgr2=0x7f85380a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.188+0000 7f8557db6640 1 --2- 172.21.15.28:0/739104186 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f85380a3e40 0x7f85380a6300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8548009430 tx=0x7f8548005c50 comp rx=0 tx=0).stop 2024-04-21T08:51:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.188+0000 7f8557db6640 1 -- 172.21.15.28:0/739104186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8550198630 msgr2=0x7f8550198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.188+0000 7f8557db6640 1 --2- 172.21.15.28:0/739104186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8550198630 0x7f8550198ab0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8540009fd0 tx=0x7f854000b8e0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.188+0000 7f8557db6640 1 -- 172.21.15.28:0/739104186 shutdown_connections 2024-04-21T08:51:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.188+0000 7f8557db6640 1 --2- 172.21.15.28:0/739104186 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f85380a3e40 0x7f85380a6300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.188+0000 7f8557db6640 1 --2- 172.21.15.28:0/739104186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8550198630 0x7f8550198ab0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.188+0000 7f8557db6640 1 --2- 172.21.15.28:0/739104186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8550106170 0x7f855019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.188+0000 7f8557db6640 1 --2- 172.21.15.28:0/739104186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85501010a0 0x7f855019e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.188+0000 7f8557db6640 1 -- 172.21.15.28:0/739104186 >> 172.21.15.28:0/739104186 conn(0x7f8550077eb0 msgr2=0x7f8550107340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.188+0000 7f8557db6640 1 -- 172.21.15.28:0/739104186 shutdown_connections 2024-04-21T08:51:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.188+0000 7f8557db6640 1 -- 172.21.15.28:0/739104186 wait complete. 2024-04-21T08:51:22.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:21 smithi028 ceph-mon[28003]: from='client.15723 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:51:22.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:21 smithi028 ceph-mon[28003]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 911 B/s wr, 18 op/s 2024-04-21T08:51:22.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:21 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2458385470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:22.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:21 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1230965900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:22.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:21 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2075696912' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:22.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:51:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.302+0000 7fc02fc6a640 1 -- 172.21.15.28:0/526724621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0281017f0 msgr2=0x7fc02810dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.302+0000 7fc02fc6a640 1 --2- 172.21.15.28:0/526724621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0281017f0 0x7fc02810dcb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc018009ca0 tx=0x7fc01802c180 comp rx=0 tx=0).stop 2024-04-21T08:51:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.302+0000 7fc02fc6a640 1 -- 172.21.15.28:0/526724621 shutdown_connections 2024-04-21T08:51:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.302+0000 7fc02fc6a640 1 --2- 172.21.15.28:0/526724621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0281017f0 0x7fc02810dcb0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.302+0000 7fc02fc6a640 1 --2- 172.21.15.28:0/526724621 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc028100e50 0x7fc0281012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.302+0000 7fc02fc6a640 1 --2- 172.21.15.28:0/526724621 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc028106f10 0x7fc0281072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.302+0000 7fc02fc6a640 1 -- 172.21.15.28:0/526724621 >> 172.21.15.28:0/526724621 conn(0x7fc028077f80 msgr2=0x7fc028078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.302+0000 7fc02fc6a640 1 -- 172.21.15.28:0/526724621 shutdown_connections 2024-04-21T08:51:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.302+0000 7fc02fc6a640 1 -- 172.21.15.28:0/526724621 wait complete. 2024-04-21T08:51:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02fc6a640 1 Processor -- start 2024-04-21T08:51:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02fc6a640 1 -- start start 2024-04-21T08:51:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02fc6a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc028100e50 0x7fc028198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02fc6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0281017f0 0x7fc028198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02fc6a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc028106f10 0x7fc02819cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02fc6a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0281121d0 con 0x7fc0281017f0 2024-04-21T08:51:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02fc6a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc028112050 con 0x7fc028106f10 2024-04-21T08:51:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02fc6a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc028112350 con 0x7fc028100e50 2024-04-21T08:51:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02d205640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0281017f0 0x7fc028198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02d205640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0281017f0 0x7fc028198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52390/0 (socket says 172.21.15.28:52390) 2024-04-21T08:51:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02d205640 1 -- 172.21.15.28:0/436275145 learned_addr learned my addr 172.21.15.28:0/436275145 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02da06640 1 --2- 172.21.15.28:0/436275145 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc028100e50 0x7fc028198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02e207640 1 --2- 172.21.15.28:0/436275145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc028106f10 0x7fc02819cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02d205640 1 -- 172.21.15.28:0/436275145 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc028100e50 msgr2=0x7fc028198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02d205640 1 --2- 172.21.15.28:0/436275145 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc028100e50 0x7fc028198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02d205640 1 -- 172.21.15.28:0/436275145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc028106f10 msgr2=0x7fc02819cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02d205640 1 --2- 172.21.15.28:0/436275145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc028106f10 0x7fc02819cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02d205640 1 -- 172.21.15.28:0/436275145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc02819d610 con 0x7fc0281017f0 2024-04-21T08:51:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.303+0000 7fc02d205640 1 --2- 172.21.15.28:0/436275145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0281017f0 0x7fc028198bb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc01c00d950 tx=0x7fc01c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.304+0000 7fc026ffd640 1 -- 172.21.15.28:0/436275145 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c017030 con 0x7fc0281017f0 2024-04-21T08:51:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.304+0000 7fc026ffd640 1 -- 172.21.15.28:0/436275145 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc01c004020 con 0x7fc0281017f0 2024-04-21T08:51:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.304+0000 7fc02fc6a640 1 -- 172.21.15.28:0/436275145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc02819d900 con 0x7fc0281017f0 2024-04-21T08:51:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.304+0000 7fc026ffd640 1 -- 172.21.15.28:0/436275145 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c0047a0 con 0x7fc0281017f0 2024-04-21T08:51:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.304+0000 7fc02fc6a640 1 -- 172.21.15.28:0/436275145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0281a5190 con 0x7fc0281017f0 2024-04-21T08:51:22.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.305+0000 7fc02fc6a640 1 -- 172.21.15.28:0/436275145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc028068550 con 0x7fc0281017f0 2024-04-21T08:51:22.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.307+0000 7fc026ffd640 1 -- 172.21.15.28:0/436275145 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fc01c009a30 con 0x7fc0281017f0 2024-04-21T08:51:22.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.307+0000 7fc026ffd640 1 --2- 172.21.15.28:0/436275145 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc0100a3e40 0x7fc0100a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:22.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.307+0000 7fc026ffd640 1 -- 172.21.15.28:0/436275145 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fc01c0d0fb0 con 0x7fc0281017f0 2024-04-21T08:51:22.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.307+0000 7fc02da06640 1 --2- 172.21.15.28:0/436275145 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc0100a3e40 0x7fc0100a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:22.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.308+0000 7fc02da06640 1 --2- 172.21.15.28:0/436275145 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc0100a3e40 0x7fc0100a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc020009b50 tx=0x7fc020005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.311+0000 7fc026ffd640 1 -- 172.21.15.28:0/436275145 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc01c09d060 con 0x7fc0281017f0 2024-04-21T08:51:22.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:21 smithi152 ceph-mon[31401]: from='client.15723 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:51:22.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:21 smithi152 ceph-mon[31401]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 911 B/s wr, 18 op/s 2024-04-21T08:51:22.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:21 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2458385470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:22.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:21 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1230965900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:22.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:21 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2075696912' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.499+0000 7fc02fc6a640 1 -- 172.21.15.28:0/436275145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc028199970 con 0x7fc0281017f0 2024-04-21T08:51:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.500+0000 7fc026ffd640 1 -- 172.21.15.28:0/436275145 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25160 (secure 0 0 0) 0x7fc028199970 con 0x7fc0281017f0 2024-04-21T08:51:22.500 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:22.500 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:51:10.823421+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1076282159":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6800/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/926389767":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/892059008":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/2132574390":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:51:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.502+0000 7fc02fc6a640 1 -- 172.21.15.28:0/436275145 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc0100a3e40 msgr2=0x7fc0100a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.502+0000 7fc02fc6a640 1 --2- 172.21.15.28:0/436275145 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc0100a3e40 0x7fc0100a6300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc020009b50 tx=0x7fc020005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.502+0000 7fc02fc6a640 1 -- 172.21.15.28:0/436275145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0281017f0 msgr2=0x7fc028198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.502+0000 7fc02fc6a640 1 --2- 172.21.15.28:0/436275145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0281017f0 0x7fc028198bb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc01c00d950 tx=0x7fc01c00de10 comp rx=0 tx=0).stop 2024-04-21T08:51:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.502+0000 7fc02fc6a640 1 -- 172.21.15.28:0/436275145 shutdown_connections 2024-04-21T08:51:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.502+0000 7fc02fc6a640 1 --2- 172.21.15.28:0/436275145 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc0100a3e40 0x7fc0100a6300 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.502+0000 7fc02fc6a640 1 --2- 172.21.15.28:0/436275145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc028106f10 0x7fc02819cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.502+0000 7fc02fc6a640 1 --2- 172.21.15.28:0/436275145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0281017f0 0x7fc028198bb0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.502+0000 7fc02fc6a640 1 --2- 172.21.15.28:0/436275145 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc028100e50 0x7fc028198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.502+0000 7fc02fc6a640 1 -- 172.21.15.28:0/436275145 >> 172.21.15.28:0/436275145 conn(0x7fc028077f80 msgr2=0x7fc0280ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.502+0000 7fc02fc6a640 1 -- 172.21.15.28:0/436275145 shutdown_connections 2024-04-21T08:51:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.503+0000 7fc02fc6a640 1 -- 172.21.15.28:0/436275145 wait complete. 2024-04-21T08:51:22.514 INFO:tasks.quiescer.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:48:43.543799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:48:44.864564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:48:48.747842+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:51:22.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:51:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.619+0000 7f0f12a6d640 1 -- 172.21.15.28:0/2812076951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0c10e900 msgr2=0x7f0f0c110cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.619+0000 7f0f12a6d640 1 --2- 172.21.15.28:0/2812076951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0c10e900 0x7f0f0c110cf0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0f04009ca0 tx=0x7f0f0402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.619+0000 7f0f12a6d640 1 -- 172.21.15.28:0/2812076951 shutdown_connections 2024-04-21T08:51:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.619+0000 7f0f12a6d640 1 --2- 172.21.15.28:0/2812076951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0c10e900 0x7f0f0c110cf0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.619+0000 7f0f12a6d640 1 --2- 172.21.15.28:0/2812076951 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f0c101670 0x7f0f0c10e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.619+0000 7f0f12a6d640 1 --2- 172.21.15.28:0/2812076951 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f0c100d50 0x7f0f0c101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.619+0000 7f0f12a6d640 1 -- 172.21.15.28:0/2812076951 >> 172.21.15.28:0/2812076951 conn(0x7f0f0c077f50 msgr2=0x7f0f0c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:22.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.619+0000 7f0f12a6d640 1 -- 172.21.15.28:0/2812076951 shutdown_connections 2024-04-21T08:51:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.619+0000 7f0f12a6d640 1 -- 172.21.15.28:0/2812076951 wait complete. 2024-04-21T08:51:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.620+0000 7f0f12a6d640 1 Processor -- start 2024-04-21T08:51:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.620+0000 7f0f12a6d640 1 -- start start 2024-04-21T08:51:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.620+0000 7f0f12a6d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f0c100d50 0x7f0f0c1a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.620+0000 7f0f12a6d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f0c101670 0x7f0f0c1a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.620+0000 7f0f12a6d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0c10e900 0x7f0f0c19a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.620+0000 7f0f12a6d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c1143c0 con 0x7f0f0c10e900 2024-04-21T08:51:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.620+0000 7f0f12a6d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c114240 con 0x7f0f0c100d50 2024-04-21T08:51:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.620+0000 7f0f12a6d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c114540 con 0x7f0f0c101670 2024-04-21T08:51:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.620+0000 7f0f1100a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0c10e900 0x7f0f0c19a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.620+0000 7f0f1100a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0c10e900 0x7f0f0c19a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52406/0 (socket says 172.21.15.28:52406) 2024-04-21T08:51:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.620+0000 7f0f1100a640 1 -- 172.21.15.28:0/2096812145 learned_addr learned my addr 172.21.15.28:0/2096812145 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.621+0000 7f0f10809640 1 --2- 172.21.15.28:0/2096812145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f0c100d50 0x7f0f0c1a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:22.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.621+0000 7f0f0bfff640 1 --2- 172.21.15.28:0/2096812145 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f0c101670 0x7f0f0c1a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:22.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.621+0000 7f0f1100a640 1 -- 172.21.15.28:0/2096812145 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f0c101670 msgr2=0x7f0f0c1a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:22.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.621+0000 7f0f1100a640 1 --2- 172.21.15.28:0/2096812145 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f0c101670 0x7f0f0c1a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.621+0000 7f0f1100a640 1 -- 172.21.15.28:0/2096812145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f0c100d50 msgr2=0x7f0f0c1a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:22.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.621+0000 7f0f1100a640 1 --2- 172.21.15.28:0/2096812145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f0c100d50 0x7f0f0c1a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.621+0000 7f0f1100a640 1 -- 172.21.15.28:0/2096812145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0c19af40 con 0x7f0f0c10e900 2024-04-21T08:51:22.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.621+0000 7f0f1100a640 1 --2- 172.21.15.28:0/2096812145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0c10e900 0x7f0f0c19a680 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0f0402c700 tx=0x7f0f04034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:22.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.621+0000 7f0f09ffb640 1 -- 172.21.15.28:0/2096812145 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f04041020 con 0x7f0f0c10e900 2024-04-21T08:51:22.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.621+0000 7f0f09ffb640 1 -- 172.21.15.28:0/2096812145 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f04034ca0 con 0x7f0f0c10e900 2024-04-21T08:51:22.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.621+0000 7f0f12a6d640 1 -- 172.21.15.28:0/2096812145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0c19b1d0 con 0x7f0f0c10e900 2024-04-21T08:51:22.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.621+0000 7f0f09ffb640 1 -- 172.21.15.28:0/2096812145 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f04039040 con 0x7f0f0c10e900 2024-04-21T08:51:22.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.621+0000 7f0f12a6d640 1 -- 172.21.15.28:0/2096812145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f0c1a7360 con 0x7f0f0c10e900 2024-04-21T08:51:22.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.622+0000 7f0f12a6d640 1 -- 172.21.15.28:0/2096812145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f0c0698e0 con 0x7f0f0c10e900 2024-04-21T08:51:22.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.623+0000 7f0f09ffb640 1 -- 172.21.15.28:0/2096812145 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f0f04044020 con 0x7f0f0c10e900 2024-04-21T08:51:22.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.624+0000 7f0f09ffb640 1 --2- 172.21.15.28:0/2096812145 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0efc0a3e40 0x7f0efc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:22.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.624+0000 7f0f09ffb640 1 -- 172.21.15.28:0/2096812145 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f0f040f2f00 con 0x7f0f0c10e900 2024-04-21T08:51:22.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.624+0000 7f0f10809640 1 --2- 172.21.15.28:0/2096812145 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0efc0a3e40 0x7f0efc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:22.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.625+0000 7f0f10809640 1 --2- 172.21.15.28:0/2096812145 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0efc0a3e40 0x7f0efc0a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0f0000abe0 tx=0x7f0f0000a2d0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:22.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.628+0000 7f0f09ffb640 1 -- 172.21.15.28:0/2096812145 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0f0404b020 con 0x7f0f0c10e900 2024-04-21T08:51:22.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.815+0000 7f0f12a6d640 1 -- 172.21.15.28:0/2096812145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0f0c109ac0 con 0x7f0f0c10e900 2024-04-21T08:51:22.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.815+0000 7f0f09ffb640 1 -- 172.21.15.28:0/2096812145 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f0f040bf1e0 con 0x7f0f0c10e900 2024-04-21T08:51:22.815 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T08:51:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.817+0000 7f0f12a6d640 1 -- 172.21.15.28:0/2096812145 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0efc0a3e40 msgr2=0x7f0efc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.817+0000 7f0f12a6d640 1 --2- 172.21.15.28:0/2096812145 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0efc0a3e40 0x7f0efc0a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0f0000abe0 tx=0x7f0f0000a2d0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.817+0000 7f0f12a6d640 1 -- 172.21.15.28:0/2096812145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0c10e900 msgr2=0x7f0f0c19a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.817+0000 7f0f12a6d640 1 --2- 172.21.15.28:0/2096812145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0c10e900 0x7f0f0c19a680 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0f0402c700 tx=0x7f0f04034670 comp rx=0 tx=0).stop 2024-04-21T08:51:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.817+0000 7f0f12a6d640 1 -- 172.21.15.28:0/2096812145 shutdown_connections 2024-04-21T08:51:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.817+0000 7f0f12a6d640 1 --2- 172.21.15.28:0/2096812145 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0efc0a3e40 0x7f0efc0a6300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.817+0000 7f0f12a6d640 1 --2- 172.21.15.28:0/2096812145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f0c10e900 0x7f0f0c19a680 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.817+0000 7f0f12a6d640 1 --2- 172.21.15.28:0/2096812145 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f0c101670 0x7f0f0c1a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.817+0000 7f0f12a6d640 1 --2- 172.21.15.28:0/2096812145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0f0c100d50 0x7f0f0c1a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.817+0000 7f0f12a6d640 1 -- 172.21.15.28:0/2096812145 >> 172.21.15.28:0/2096812145 conn(0x7f0f0c077f50 msgr2=0x7f0f0c10fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:22.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.817+0000 7f0f12a6d640 1 -- 172.21.15.28:0/2096812145 shutdown_connections 2024-04-21T08:51:22.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.817+0000 7f0f12a6d640 1 -- 172.21.15.28:0/2096812145 wait complete. 2024-04-21T08:51:22.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:51:22.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.934+0000 7faec6ee0640 1 -- 172.21.15.28:0/2687566893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec01005c0 msgr2=0x7faec0100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:22.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.934+0000 7faec6ee0640 1 --2- 172.21.15.28:0/2687566893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec01005c0 0x7faec0100a40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009900 tx=0x7faeb402bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:22.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.934+0000 7faec6ee0640 1 -- 172.21.15.28:0/2687566893 shutdown_connections 2024-04-21T08:51:22.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.934+0000 7faec6ee0640 1 --2- 172.21.15.28:0/2687566893 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faec0100f80 0x7faec010d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.934+0000 7faec6ee0640 1 --2- 172.21.15.28:0/2687566893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec01005c0 0x7faec0100a40 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.934+0000 7faec6ee0640 1 --2- 172.21.15.28:0/2687566893 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faec0106680 0x7faec0106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.934+0000 7faec6ee0640 1 -- 172.21.15.28:0/2687566893 >> 172.21.15.28:0/2687566893 conn(0x7faec0077410 msgr2=0x7faec00fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:22.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.934+0000 7faec6ee0640 1 -- 172.21.15.28:0/2687566893 shutdown_connections 2024-04-21T08:51:22.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.934+0000 7faec6ee0640 1 -- 172.21.15.28:0/2687566893 wait complete. 2024-04-21T08:51:22.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.935+0000 7faec6ee0640 1 Processor -- start 2024-04-21T08:51:22.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.935+0000 7faec6ee0640 1 -- start start 2024-04-21T08:51:22.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.935+0000 7faec6ee0640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faec01005c0 0x7faec010ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:22.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.935+0000 7faec6ee0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec0100f80 0x7faec010f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:22.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.935+0000 7faec6ee0640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faec0106680 0x7faec010f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:22.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.935+0000 7faec6ee0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faec0112240 con 0x7faec0100f80 2024-04-21T08:51:22.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.935+0000 7faec6ee0640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7faec01120c0 con 0x7faec01005c0 2024-04-21T08:51:22.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.935+0000 7faec6ee0640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7faec01123c0 con 0x7faec0106680 2024-04-21T08:51:22.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.935+0000 7faebffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec0100f80 0x7faec010f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:22.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.935+0000 7faebffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec0100f80 0x7faec010f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52438/0 (socket says 172.21.15.28:52438) 2024-04-21T08:51:22.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.935+0000 7faebffff640 1 -- 172.21.15.28:0/1802068257 learned_addr learned my addr 172.21.15.28:0/1802068257 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:22.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.935+0000 7faec4c7c640 1 --2- 172.21.15.28:0/1802068257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faec01005c0 0x7faec010ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:22.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.936+0000 7faebffff640 1 -- 172.21.15.28:0/1802068257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faec0106680 msgr2=0x7faec010f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:22.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.936+0000 7faebffff640 1 --2- 172.21.15.28:0/1802068257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faec0106680 0x7faec010f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.936+0000 7faebffff640 1 -- 172.21.15.28:0/1802068257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faec01005c0 msgr2=0x7faec010ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:22.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.936+0000 7faebffff640 1 --2- 172.21.15.28:0/1802068257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faec01005c0 0x7faec010ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:22.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.936+0000 7faebffff640 1 -- 172.21.15.28:0/1802068257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faec01a6be0 con 0x7faec0100f80 2024-04-21T08:51:22.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.936+0000 7faebffff640 1 --2- 172.21.15.28:0/1802068257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec0100f80 0x7faec010f220 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7faeb4006010 tx=0x7faeb4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:22.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.936+0000 7faebdffb640 1 -- 172.21.15.28:0/1802068257 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb4043020 con 0x7faec0100f80 2024-04-21T08:51:22.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.936+0000 7faebdffb640 1 -- 172.21.15.28:0/1802068257 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faeb4004440 con 0x7faec0100f80 2024-04-21T08:51:22.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.936+0000 7faebdffb640 1 -- 172.21.15.28:0/1802068257 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb4003b50 con 0x7faec0100f80 2024-04-21T08:51:22.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.936+0000 7faec6ee0640 1 -- 172.21.15.28:0/1802068257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faec01a6e70 con 0x7faec0100f80 2024-04-21T08:51:22.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.936+0000 7faec6ee0640 1 -- 172.21.15.28:0/1802068257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faec01a7330 con 0x7faec0100f80 2024-04-21T08:51:22.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.937+0000 7faec6ee0640 1 -- 172.21.15.28:0/1802068257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faec00691c0 con 0x7faec0100f80 2024-04-21T08:51:22.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.938+0000 7faebdffb640 1 -- 172.21.15.28:0/1802068257 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7faeb4046020 con 0x7faec0100f80 2024-04-21T08:51:22.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.939+0000 7faebdffb640 1 --2- 172.21.15.28:0/1802068257 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faea80a3e40 0x7faea80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:22.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.939+0000 7faec4c7c640 1 --2- 172.21.15.28:0/1802068257 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faea80a3e40 0x7faea80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:22.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.939+0000 7faebdffb640 1 -- 172.21.15.28:0/1802068257 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7faeb40f2930 con 0x7faec0100f80 2024-04-21T08:51:22.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.940+0000 7faec4c7c640 1 --2- 172.21.15.28:0/1802068257 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faea80a3e40 0x7faea80a6300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faeb000bb60 tx=0x7faeb000b390 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:22.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:22.944+0000 7faebdffb640 1 -- 172.21.15.28:0/1802068257 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faeb404d070 con 0x7faec0100f80 2024-04-21T08:51:23.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:22 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2096002446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:23.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:22 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/739104186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:23.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:22 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/436275145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:23.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:22 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2096812145' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:23.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.131+0000 7faec6ee0640 1 -- 172.21.15.28:0/1802068257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7faec002d5c0 con 0x7faec0100f80 2024-04-21T08:51:23.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.131+0000 7faebdffb640 1 -- 172.21.15.28:0/1802068257 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7faeb40bea90 con 0x7faec0100f80 2024-04-21T08:51:23.131 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:51:23.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.133+0000 7faec6ee0640 1 -- 172.21.15.28:0/1802068257 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faea80a3e40 msgr2=0x7faea80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.133+0000 7faec6ee0640 1 --2- 172.21.15.28:0/1802068257 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faea80a3e40 0x7faea80a6300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faeb000bb60 tx=0x7faeb000b390 comp rx=0 tx=0).stop 2024-04-21T08:51:23.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.133+0000 7faec6ee0640 1 -- 172.21.15.28:0/1802068257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec0100f80 msgr2=0x7faec010f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.133+0000 7faec6ee0640 1 --2- 172.21.15.28:0/1802068257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec0100f80 0x7faec010f220 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7faeb4006010 tx=0x7faeb4004040 comp rx=0 tx=0).stop 2024-04-21T08:51:23.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.133+0000 7faec6ee0640 1 -- 172.21.15.28:0/1802068257 shutdown_connections 2024-04-21T08:51:23.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.133+0000 7faec6ee0640 1 --2- 172.21.15.28:0/1802068257 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faea80a3e40 0x7faea80a6300 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.133+0000 7faec6ee0640 1 --2- 172.21.15.28:0/1802068257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faec0106680 0x7faec010f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.133+0000 7faec6ee0640 1 --2- 172.21.15.28:0/1802068257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faec0100f80 0x7faec010f220 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.133+0000 7faec6ee0640 1 --2- 172.21.15.28:0/1802068257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faec01005c0 0x7faec010ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.133+0000 7faec6ee0640 1 -- 172.21.15.28:0/1802068257 >> 172.21.15.28:0/1802068257 conn(0x7faec0077410 msgr2=0x7faec00fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:23.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.134+0000 7faec6ee0640 1 -- 172.21.15.28:0/1802068257 shutdown_connections 2024-04-21T08:51:23.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.134+0000 7faec6ee0640 1 -- 172.21.15.28:0/1802068257 wait complete. 2024-04-21T08:51:23.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:51:23.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.249+0000 7f7bbcdef640 1 -- 172.21.15.28:0/1877519157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bb8102b40 msgr2=0x7f7bb8106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.249+0000 7f7bbcdef640 1 --2- 172.21.15.28:0/1877519157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bb8102b40 0x7f7bb8106b80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0009900 tx=0x7f7ba002bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:23.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.249+0000 7f7bbcdef640 1 -- 172.21.15.28:0/1877519157 shutdown_connections 2024-04-21T08:51:23.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.249+0000 7f7bbcdef640 1 --2- 172.21.15.28:0/1877519157 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bb8107260 0x7f7bb810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.249+0000 7f7bbcdef640 1 --2- 172.21.15.28:0/1877519157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bb8102b40 0x7f7bb8106b80 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.249+0000 7f7bbcdef640 1 --2- 172.21.15.28:0/1877519157 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bb8102190 0x7f7bb8102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.249+0000 7f7bbcdef640 1 -- 172.21.15.28:0/1877519157 >> 172.21.15.28:0/1877519157 conn(0x7f7bb80fd990 msgr2=0x7f7bb80ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:23.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.249+0000 7f7bbcdef640 1 -- 172.21.15.28:0/1877519157 shutdown_connections 2024-04-21T08:51:23.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.249+0000 7f7bbcdef640 1 -- 172.21.15.28:0/1877519157 wait complete. 2024-04-21T08:51:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.250+0000 7f7bbcdef640 1 Processor -- start 2024-04-21T08:51:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.250+0000 7f7bbcdef640 1 -- start start 2024-04-21T08:51:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.250+0000 7f7bbcdef640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bb8102190 0x7f7bb8074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.250+0000 7f7bbcdef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bb8107260 0x7f7bb8074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.250+0000 7f7bbcdef640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bb8074cd0 0x7f7bb81a69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.250+0000 7f7bbcdef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb8112200 con 0x7f7bb8107260 2024-04-21T08:51:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.250+0000 7f7bbcdef640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb8112080 con 0x7f7bb8074cd0 2024-04-21T08:51:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.250+0000 7f7bbcdef640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb8112380 con 0x7f7bb8102190 2024-04-21T08:51:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.250+0000 7f7baffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bb8107260 0x7f7bb8074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.250+0000 7f7baffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bb8107260 0x7f7bb8074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52462/0 (socket says 172.21.15.28:52462) 2024-04-21T08:51:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.250+0000 7f7baffff640 1 -- 172.21.15.28:0/3562503030 learned_addr learned my addr 172.21.15.28:0/3562503030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.250+0000 7f7bb659c640 1 --2- 172.21.15.28:0/3562503030 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bb8102190 0x7f7bb8074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.251+0000 7f7bb6d9d640 1 --2- 172.21.15.28:0/3562503030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bb8074cd0 0x7f7bb81a69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.251+0000 7f7baffff640 1 -- 172.21.15.28:0/3562503030 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bb8102190 msgr2=0x7f7bb8074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.251+0000 7f7baffff640 1 --2- 172.21.15.28:0/3562503030 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bb8102190 0x7f7bb8074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.251+0000 7f7baffff640 1 -- 172.21.15.28:0/3562503030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bb8074cd0 msgr2=0x7f7bb81a69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.251+0000 7f7baffff640 1 --2- 172.21.15.28:0/3562503030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bb8074cd0 0x7f7bb81a69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.251+0000 7f7baffff640 1 -- 172.21.15.28:0/3562503030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb81a70b0 con 0x7f7bb8107260 2024-04-21T08:51:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.251+0000 7f7bb6d9d640 1 --2- 172.21.15.28:0/3562503030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bb8074cd0 0x7f7bb81a69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.251+0000 7f7baffff640 1 --2- 172.21.15.28:0/3562503030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bb8107260 0x7f7bb8074790 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0036040 tx=0x7f7ba002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.251+0000 7f7baf7fe640 1 -- 172.21.15.28:0/3562503030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0043020 con 0x7f7bb8107260 2024-04-21T08:51:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.251+0000 7f7bbcdef640 1 -- 172.21.15.28:0/3562503030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bb81a7340 con 0x7f7bb8107260 2024-04-21T08:51:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.251+0000 7f7baf7fe640 1 -- 172.21.15.28:0/3562503030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ba002cbe0 con 0x7f7bb8107260 2024-04-21T08:51:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.251+0000 7f7baf7fe640 1 -- 172.21.15.28:0/3562503030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0003bf0 con 0x7f7bb8107260 2024-04-21T08:51:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.251+0000 7f7bbcdef640 1 -- 172.21.15.28:0/3562503030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bb81a7890 con 0x7f7bb8107260 2024-04-21T08:51:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.252+0000 7f7bbcdef640 1 -- 172.21.15.28:0/3562503030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bb80691c0 con 0x7f7bb8107260 2024-04-21T08:51:23.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.253+0000 7f7baf7fe640 1 -- 172.21.15.28:0/3562503030 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f7ba0046020 con 0x7f7bb8107260 2024-04-21T08:51:23.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.254+0000 7f7baf7fe640 1 --2- 172.21.15.28:0/3562503030 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f7ba40a3e70 0x7f7ba40a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.254+0000 7f7bb659c640 1 --2- 172.21.15.28:0/3562503030 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f7ba40a3e70 0x7f7ba40a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.254+0000 7f7baf7fe640 1 -- 172.21.15.28:0/3562503030 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f7ba00f2fb0 con 0x7f7bb8107260 2024-04-21T08:51:23.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.254+0000 7f7bb659c640 1 --2- 172.21.15.28:0/3562503030 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f7ba40a3e70 0x7f7ba40a6330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7ba800bbe0 tx=0x7f7ba800b390 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.258+0000 7f7baf7fe640 1 -- 172.21.15.28:0/3562503030 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7ba004d020 con 0x7f7bb8107260 2024-04-21T08:51:23.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:22 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2096002446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:23.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:22 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/739104186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:23.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:22 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/436275145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:23.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:22 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2096812145' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:23.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:22 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2096002446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:23.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:22 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/739104186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:23.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:22 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/436275145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:23.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:22 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2096812145' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:23.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.446+0000 7f7bbcdef640 1 -- 172.21.15.28:0/3562503030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7bb810a4f0 con 0x7f7bb8107260 2024-04-21T08:51:23.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.446+0000 7f7baf7fe640 1 -- 172.21.15.28:0/3562503030 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f7ba00bf110 con 0x7f7bb8107260 2024-04-21T08:51:23.446 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-21T08:51:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.448+0000 7f7bbcdef640 1 -- 172.21.15.28:0/3562503030 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f7ba40a3e70 msgr2=0x7f7ba40a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.448+0000 7f7bbcdef640 1 --2- 172.21.15.28:0/3562503030 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f7ba40a3e70 0x7f7ba40a6330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7ba800bbe0 tx=0x7f7ba800b390 comp rx=0 tx=0).stop 2024-04-21T08:51:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.448+0000 7f7bbcdef640 1 -- 172.21.15.28:0/3562503030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bb8107260 msgr2=0x7f7bb8074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.448+0000 7f7bbcdef640 1 --2- 172.21.15.28:0/3562503030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bb8107260 0x7f7bb8074790 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0036040 tx=0x7f7ba002c7e0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.448+0000 7f7bbcdef640 1 -- 172.21.15.28:0/3562503030 shutdown_connections 2024-04-21T08:51:23.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.448+0000 7f7bbcdef640 1 --2- 172.21.15.28:0/3562503030 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f7ba40a3e70 0x7f7ba40a6330 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.448+0000 7f7bbcdef640 1 --2- 172.21.15.28:0/3562503030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bb8074cd0 0x7f7bb81a69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.448+0000 7f7bbcdef640 1 --2- 172.21.15.28:0/3562503030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bb8107260 0x7f7bb8074790 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.448+0000 7f7bbcdef640 1 --2- 172.21.15.28:0/3562503030 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bb8102190 0x7f7bb8074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.448+0000 7f7bbcdef640 1 -- 172.21.15.28:0/3562503030 >> 172.21.15.28:0/3562503030 conn(0x7f7bb80fd990 msgr2=0x7f7bb80fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:23.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.448+0000 7f7bbcdef640 1 -- 172.21.15.28:0/3562503030 shutdown_connections 2024-04-21T08:51:23.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.448+0000 7f7bbcdef640 1 -- 172.21.15.28:0/3562503030 wait complete. 2024-04-21T08:51:23.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:51:23.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.564+0000 7f242837e640 1 -- 172.21.15.28:0/3084627782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2420102190 msgr2=0x7f2420102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.564+0000 7f242837e640 1 --2- 172.21.15.28:0/3084627782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2420102190 0x7f2420102570 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f241c009900 tx=0x7f241c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.564+0000 7f242837e640 1 -- 172.21.15.28:0/3084627782 shutdown_connections 2024-04-21T08:51:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.564+0000 7f242837e640 1 --2- 172.21.15.28:0/3084627782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2420107260 0x7f242010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.564+0000 7f242837e640 1 --2- 172.21.15.28:0/3084627782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2420102b40 0x7f2420106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.564+0000 7f242837e640 1 --2- 172.21.15.28:0/3084627782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2420102190 0x7f2420102570 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.564+0000 7f242837e640 1 -- 172.21.15.28:0/3084627782 >> 172.21.15.28:0/3084627782 conn(0x7f24200fd990 msgr2=0x7f24200ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.564+0000 7f242837e640 1 -- 172.21.15.28:0/3084627782 shutdown_connections 2024-04-21T08:51:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.564+0000 7f242837e640 1 -- 172.21.15.28:0/3084627782 wait complete. 2024-04-21T08:51:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.565+0000 7f242837e640 1 Processor -- start 2024-04-21T08:51:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.565+0000 7f242837e640 1 -- start start 2024-04-21T08:51:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.565+0000 7f242837e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2420102190 0x7f2420198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.565+0000 7f242837e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2420107260 0x7f2420198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.565+0000 7f242837e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242019cfe0 0x7f242019d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.565+0000 7f242837e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24201121a0 con 0x7f242019cfe0 2024-04-21T08:51:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.565+0000 7f242837e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2420112020 con 0x7f2420102190 2024-04-21T08:51:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.565+0000 7f242837e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2420112320 con 0x7f2420107260 2024-04-21T08:51:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.565+0000 7f242691b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242019cfe0 0x7f242019d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.565+0000 7f242691b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242019cfe0 0x7f242019d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52474/0 (socket says 172.21.15.28:52474) 2024-04-21T08:51:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.565+0000 7f242691b640 1 -- 172.21.15.28:0/53726043 learned_addr learned my addr 172.21.15.28:0/53726043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.565+0000 7f242611a640 1 --2- 172.21.15.28:0/53726043 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2420102190 0x7f2420198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.566+0000 7f242691b640 1 -- 172.21.15.28:0/53726043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2420107260 msgr2=0x7f2420198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.566+0000 7f242691b640 1 --2- 172.21.15.28:0/53726043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2420107260 0x7f2420198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.566+0000 7f242691b640 1 -- 172.21.15.28:0/53726043 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2420102190 msgr2=0x7f2420198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.566+0000 7f242691b640 1 --2- 172.21.15.28:0/53726043 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2420102190 0x7f2420198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.566+0000 7f242691b640 1 -- 172.21.15.28:0/53726043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242019dad0 con 0x7f242019cfe0 2024-04-21T08:51:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.566+0000 7f242691b640 1 --2- 172.21.15.28:0/53726043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242019cfe0 0x7f242019d490 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f241800cce0 tx=0x7f241800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.566+0000 7f24177fe640 1 -- 172.21.15.28:0/53726043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241800f450 con 0x7f242019cfe0 2024-04-21T08:51:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.566+0000 7f24177fe640 1 -- 172.21.15.28:0/53726043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f241800faa0 con 0x7f242019cfe0 2024-04-21T08:51:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.566+0000 7f242837e640 1 -- 172.21.15.28:0/53726043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f242019ddc0 con 0x7f242019cfe0 2024-04-21T08:51:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.566+0000 7f24177fe640 1 -- 172.21.15.28:0/53726043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418018b70 con 0x7f242019cfe0 2024-04-21T08:51:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.566+0000 7f242837e640 1 -- 172.21.15.28:0/53726043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24201a5560 con 0x7f242019cfe0 2024-04-21T08:51:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.567+0000 7f242837e640 1 -- 172.21.15.28:0/53726043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24200691c0 con 0x7f242019cfe0 2024-04-21T08:51:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.568+0000 7f24177fe640 1 -- 172.21.15.28:0/53726043 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f2418018d10 con 0x7f242019cfe0 2024-04-21T08:51:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.569+0000 7f24177fe640 1 --2- 172.21.15.28:0/53726043 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f240c0a3e40 0x7f240c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.569+0000 7f24177fe640 1 -- 172.21.15.28:0/53726043 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f24180d1300 con 0x7f242019cfe0 2024-04-21T08:51:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.569+0000 7f242611a640 1 --2- 172.21.15.28:0/53726043 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f240c0a3e40 0x7f240c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.569+0000 7f242611a640 1 --2- 172.21.15.28:0/53726043 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f240c0a3e40 0x7f240c0a6300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f241c000c00 tx=0x7f241c004120 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:23.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.573+0000 7f24177fe640 1 -- 172.21.15.28:0/53726043 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2418014030 con 0x7f242019cfe0 2024-04-21T08:51:23.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.759+0000 7f242837e640 1 -- 172.21.15.28:0/53726043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2420102b40 con 0x7f242019cfe0 2024-04-21T08:51:23.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.759+0000 7f24177fe640 1 -- 172.21.15.28:0/53726043 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f241809d3b0 con 0x7f242019cfe0 2024-04-21T08:51:23.759 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:51:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.761+0000 7f242837e640 1 -- 172.21.15.28:0/53726043 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f240c0a3e40 msgr2=0x7f240c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.761+0000 7f242837e640 1 --2- 172.21.15.28:0/53726043 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f240c0a3e40 0x7f240c0a6300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f241c000c00 tx=0x7f241c004120 comp rx=0 tx=0).stop 2024-04-21T08:51:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.761+0000 7f242837e640 1 -- 172.21.15.28:0/53726043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242019cfe0 msgr2=0x7f242019d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.761+0000 7f242837e640 1 --2- 172.21.15.28:0/53726043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242019cfe0 0x7f242019d490 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f241800cce0 tx=0x7f241800a620 comp rx=0 tx=0).stop 2024-04-21T08:51:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.761+0000 7f242837e640 1 -- 172.21.15.28:0/53726043 shutdown_connections 2024-04-21T08:51:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.761+0000 7f242837e640 1 --2- 172.21.15.28:0/53726043 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f240c0a3e40 0x7f240c0a6300 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.761+0000 7f242837e640 1 --2- 172.21.15.28:0/53726043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242019cfe0 0x7f242019d490 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.761+0000 7f242837e640 1 --2- 172.21.15.28:0/53726043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2420107260 0x7f2420198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.761+0000 7f242837e640 1 --2- 172.21.15.28:0/53726043 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2420102190 0x7f2420198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.761+0000 7f242837e640 1 -- 172.21.15.28:0/53726043 >> 172.21.15.28:0/53726043 conn(0x7f24200fd990 msgr2=0x7f24200fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.761+0000 7f242837e640 1 -- 172.21.15.28:0/53726043 shutdown_connections 2024-04-21T08:51:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.761+0000 7f242837e640 1 -- 172.21.15.28:0/53726043 wait complete. 2024-04-21T08:51:23.772 INFO:tasks.quiescer.ceph_manager:waiting for clean 2024-04-21T08:51:23.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:51:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.876+0000 7f894cfc7640 1 -- 172.21.15.28:0/785891866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8948101b60 msgr2=0x7f894810dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.876+0000 7f894cfc7640 1 --2- 172.21.15.28:0/785891866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8948101b60 0x7f894810dc90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8940009cd0 tx=0x7f894002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.876+0000 7f894cfc7640 1 -- 172.21.15.28:0/785891866 shutdown_connections 2024-04-21T08:51:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.876+0000 7f894cfc7640 1 --2- 172.21.15.28:0/785891866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8948101b60 0x7f894810dc90 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.876+0000 7f894cfc7640 1 --2- 172.21.15.28:0/785891866 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f89481011a0 0x7f8948101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.876+0000 7f894cfc7640 1 --2- 172.21.15.28:0/785891866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8948105e30 0x7f8948106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.876+0000 7f894cfc7640 1 -- 172.21.15.28:0/785891866 >> 172.21.15.28:0/785891866 conn(0x7f8948077e30 msgr2=0x7f8948078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.876+0000 7f894cfc7640 1 -- 172.21.15.28:0/785891866 shutdown_connections 2024-04-21T08:51:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.876+0000 7f894cfc7640 1 -- 172.21.15.28:0/785891866 wait complete. 2024-04-21T08:51:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.877+0000 7f894cfc7640 1 Processor -- start 2024-04-21T08:51:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.877+0000 7f894cfc7640 1 -- start start 2024-04-21T08:51:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.877+0000 7f894cfc7640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f89481011a0 0x7f89481975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.877+0000 7f894cfc7640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8948101b60 0x7f8948197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.877+0000 7f894cfc7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8948105e30 0x7f894819be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.877+0000 7f894cfc7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89481110e0 con 0x7f8948105e30 2024-04-21T08:51:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.877+0000 7f894cfc7640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8948110f60 con 0x7f8948101b60 2024-04-21T08:51:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.877+0000 7f894cfc7640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8948111260 con 0x7f89481011a0 2024-04-21T08:51:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.877+0000 7f8946d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8948105e30 0x7f894819be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.877+0000 7f8946d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8948105e30 0x7f894819be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44390/0 (socket says 172.21.15.28:44390) 2024-04-21T08:51:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.877+0000 7f8946d9d640 1 -- 172.21.15.28:0/2379788197 learned_addr learned my addr 172.21.15.28:0/2379788197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.878+0000 7f894659c640 1 --2- 172.21.15.28:0/2379788197 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f89481011a0 0x7f89481975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.878+0000 7f8945d9b640 1 --2- 172.21.15.28:0/2379788197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8948101b60 0x7f8948197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.878+0000 7f8946d9d640 1 -- 172.21.15.28:0/2379788197 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f89481011a0 msgr2=0x7f89481975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.878+0000 7f8946d9d640 1 --2- 172.21.15.28:0/2379788197 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f89481011a0 0x7f89481975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.878+0000 7f8946d9d640 1 -- 172.21.15.28:0/2379788197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8948101b60 msgr2=0x7f8948197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.878+0000 7f8946d9d640 1 --2- 172.21.15.28:0/2379788197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8948101b60 0x7f8948197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.878+0000 7f8946d9d640 1 -- 172.21.15.28:0/2379788197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f894819c5b0 con 0x7f8948105e30 2024-04-21T08:51:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.878+0000 7f8946d9d640 1 --2- 172.21.15.28:0/2379788197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8948105e30 0x7f894819be80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8940004af0 tx=0x7f8940034650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.878+0000 7f89377fe640 1 -- 172.21.15.28:0/2379788197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8940041020 con 0x7f8948105e30 2024-04-21T08:51:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.878+0000 7f89377fe640 1 -- 172.21.15.28:0/2379788197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8940032970 con 0x7f8948105e30 2024-04-21T08:51:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.878+0000 7f894cfc7640 1 -- 172.21.15.28:0/2379788197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f894819c840 con 0x7f8948105e30 2024-04-21T08:51:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.878+0000 7f89377fe640 1 -- 172.21.15.28:0/2379788197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8940039040 con 0x7f8948105e30 2024-04-21T08:51:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.878+0000 7f894cfc7640 1 -- 172.21.15.28:0/2379788197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89481a4130 con 0x7f8948105e30 2024-04-21T08:51:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.879+0000 7f894cfc7640 1 -- 172.21.15.28:0/2379788197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8948068de0 con 0x7f8948105e30 2024-04-21T08:51:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.880+0000 7f89377fe640 1 -- 172.21.15.28:0/2379788197 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f8940044020 con 0x7f8948105e30 2024-04-21T08:51:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.881+0000 7f89377fe640 1 --2- 172.21.15.28:0/2379788197 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f89300a3e40 0x7f89300a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.881+0000 7f894659c640 1 --2- 172.21.15.28:0/2379788197 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f89300a3e40 0x7f89300a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.881+0000 7f89377fe640 1 -- 172.21.15.28:0/2379788197 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f89400f2950 con 0x7f8948105e30 2024-04-21T08:51:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.882+0000 7f894659c640 1 --2- 172.21.15.28:0/2379788197 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f89300a3e40 0x7f89300a6300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8938009bd0 tx=0x7f8938005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:23.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:23.886+0000 7f89377fe640 1 -- 172.21.15.28:0/2379788197 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f894004b020 con 0x7f8948105e30 2024-04-21T08:51:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.070+0000 7f894cfc7640 1 -- 172.21.15.28:0/2379788197 --> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8948062390 con 0x7f89300a3e40 2024-04-21T08:51:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.079+0000 7f89377fe640 1 -- 172.21.15.28:0/2379788197 <== mgr.25040 v2:172.21.15.28:6800/4182301910 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339719 (secure 0 0 0) 0x7f8948062390 con 0x7f89300a3e40 2024-04-21T08:51:24.083 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-21T08:51:24.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.083+0000 7f894cfc7640 1 -- 172.21.15.28:0/2379788197 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f89300a3e40 msgr2=0x7f89300a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.083+0000 7f894cfc7640 1 --2- 172.21.15.28:0/2379788197 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f89300a3e40 0x7f89300a6300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8938009bd0 tx=0x7f8938005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:24.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.083+0000 7f894cfc7640 1 -- 172.21.15.28:0/2379788197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8948105e30 msgr2=0x7f894819be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.083+0000 7f894cfc7640 1 --2- 172.21.15.28:0/2379788197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8948105e30 0x7f894819be80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8940004af0 tx=0x7f8940034650 comp rx=0 tx=0).stop 2024-04-21T08:51:24.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.083+0000 7f894cfc7640 1 -- 172.21.15.28:0/2379788197 shutdown_connections 2024-04-21T08:51:24.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.083+0000 7f894cfc7640 1 --2- 172.21.15.28:0/2379788197 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f89300a3e40 0x7f89300a6300 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.083+0000 7f894cfc7640 1 --2- 172.21.15.28:0/2379788197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8948105e30 0x7f894819be80 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.083+0000 7f894cfc7640 1 --2- 172.21.15.28:0/2379788197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8948101b60 0x7f8948197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.083+0000 7f894cfc7640 1 --2- 172.21.15.28:0/2379788197 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f89481011a0 0x7f89481975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.083+0000 7f894cfc7640 1 -- 172.21.15.28:0/2379788197 >> 172.21.15.28:0/2379788197 conn(0x7f8948077e30 msgr2=0x7f89480fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.083+0000 7f894cfc7640 1 -- 172.21.15.28:0/2379788197 shutdown_connections 2024-04-21T08:51:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.083+0000 7f894cfc7640 1 -- 172.21.15.28:0/2379788197 wait complete. 2024-04-21T08:51:24.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:23 smithi028 ceph-mon[28003]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-04-21T08:51:24.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:23 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1802068257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:24.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:23 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3562503030' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:24.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:23 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/53726043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:24.094 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-21T08:51:22.844483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1389984,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1391,"num_read_kb":1126,"num_write":628,"num_write_kb":5305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181430,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":965,"ondisk_log_size":965,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370328,"kb_used_data":12812,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571496,"statfs":{"total":1151940427776,"available":1151561211904,"internally_reserved":0,"allocated":13119488,"data_stored":10500806,"data_compressed":571281,"data_compressed_allocated":774144,"data_compressed_original":4878336,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":199934,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":420,"num_read_kb":484,"num_write":155,"num_write_kb":922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.992496"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063213+0000","last_change":"2024-04-21T08:48:46.471626+0000","last_active":"2024-04-21T08:51:11.063213+0000","last_peered":"2024-04-21T08:51:11.063213+0000","last_clean":"2024-04-21T08:51:11.063213+0000","last_became_active":"2024-04-21T08:48:46.470498+0000","last_became_peered":"2024-04-21T08:48:46.470498+0000","last_unstale":"2024-04-21T08:51:11.063213+0000","last_undegraded":"2024-04-21T08:51:11.063213+0000","last_fullsized":"2024-04-21T08:51:11.063213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:49.452181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877897+0000","last_change":"2024-04-21T08:48:46.469111+0000","last_active":"2024-04-21T08:51:10.877897+0000","last_peered":"2024-04-21T08:51:10.877897+0000","last_clean":"2024-04-21T08:51:10.877897+0000","last_became_active":"2024-04-21T08:48:46.467926+0000","last_became_peered":"2024-04-21T08:48:46.467926+0000","last_unstale":"2024-04-21T08:51:10.877897+0000","last_undegraded":"2024-04-21T08:51:10.877897+0000","last_fullsized":"2024-04-21T08:51:10.877897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:35:38.230212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877755+0000","last_change":"2024-04-21T08:48:46.475673+0000","last_active":"2024-04-21T08:51:10.877755+0000","last_peered":"2024-04-21T08:51:10.877755+0000","last_clean":"2024-04-21T08:51:10.877755+0000","last_became_active":"2024-04-21T08:48:46.474917+0000","last_became_peered":"2024-04-21T08:48:46.474917+0000","last_unstale":"2024-04-21T08:51:10.877755+0000","last_undegraded":"2024-04-21T08:51:10.877755+0000","last_fullsized":"2024-04-21T08:51:10.877755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:01.661704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038205+0000","last_change":"2024-04-21T08:48:46.468147+0000","last_active":"2024-04-21T08:51:11.038205+0000","last_peered":"2024-04-21T08:51:11.038205+0000","last_clean":"2024-04-21T08:51:11.038205+0000","last_became_active":"2024-04-21T08:48:46.467811+0000","last_became_peered":"2024-04-21T08:48:46.467811+0000","last_unstale":"2024-04-21T08:51:11.038205+0000","last_undegraded":"2024-04-21T08:51:11.038205+0000","last_fullsized":"2024-04-21T08:51:11.038205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:21.658605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939327+0000","last_change":"2024-04-21T08:48:46.471322+0000","last_active":"2024-04-21T08:51:10.939327+0000","last_peered":"2024-04-21T08:51:10.939327+0000","last_clean":"2024-04-21T08:51:10.939327+0000","last_became_active":"2024-04-21T08:48:46.470991+0000","last_became_peered":"2024-04-21T08:48:46.470991+0000","last_unstale":"2024-04-21T08:51:10.939327+0000","last_undegraded":"2024-04-21T08:51:10.939327+0000","last_fullsized":"2024-04-21T08:51:10.939327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:22.839870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829853+0000","last_change":"2024-04-21T08:48:46.467039+0000","last_active":"2024-04-21T08:51:10.829853+0000","last_peered":"2024-04-21T08:51:10.829853+0000","last_clean":"2024-04-21T08:51:10.829853+0000","last_became_active":"2024-04-21T08:48:46.466614+0000","last_became_peered":"2024-04-21T08:48:46.466614+0000","last_unstale":"2024-04-21T08:51:10.829853+0000","last_undegraded":"2024-04-21T08:51:10.829853+0000","last_fullsized":"2024-04-21T08:51:10.829853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:38.029215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062769+0000","last_change":"2024-04-21T08:48:46.473268+0000","last_active":"2024-04-21T08:51:11.062769+0000","last_peered":"2024-04-21T08:51:11.062769+0000","last_clean":"2024-04-21T08:51:11.062769+0000","last_became_active":"2024-04-21T08:48:46.472261+0000","last_became_peered":"2024-04-21T08:48:46.472261+0000","last_unstale":"2024-04-21T08:51:11.062769+0000","last_undegraded":"2024-04-21T08:51:11.062769+0000","last_fullsized":"2024-04-21T08:51:11.062769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:58:52.990687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828243+0000","last_change":"2024-04-21T08:48:46.472788+0000","last_active":"2024-04-21T08:51:10.828243+0000","last_peered":"2024-04-21T08:51:10.828243+0000","last_clean":"2024-04-21T08:51:10.828243+0000","last_became_active":"2024-04-21T08:48:46.472482+0000","last_became_peered":"2024-04-21T08:48:46.472482+0000","last_unstale":"2024-04-21T08:51:10.828243+0000","last_undegraded":"2024-04-21T08:51:10.828243+0000","last_fullsized":"2024-04-21T08:51:10.828243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:48.483846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006997+0000","last_change":"2024-04-21T08:48:46.460771+0000","last_active":"2024-04-21T08:51:11.006997+0000","last_peered":"2024-04-21T08:51:11.006997+0000","last_clean":"2024-04-21T08:51:11.006997+0000","last_became_active":"2024-04-21T08:48:46.460472+0000","last_became_peered":"2024-04-21T08:48:46.460472+0000","last_unstale":"2024-04-21T08:51:11.006997+0000","last_undegraded":"2024-04-21T08:51:11.006997+0000","last_fullsized":"2024-04-21T08:51:11.006997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:46:40.766245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.830308+0000","last_change":"2024-04-21T08:48:46.463475+0000","last_active":"2024-04-21T08:51:10.830308+0000","last_peered":"2024-04-21T08:51:10.830308+0000","last_clean":"2024-04-21T08:51:10.830308+0000","last_became_active":"2024-04-21T08:48:46.462559+0000","last_became_peered":"2024-04-21T08:48:46.462559+0000","last_unstale":"2024-04-21T08:51:10.830308+0000","last_undegraded":"2024-04-21T08:51:10.830308+0000","last_fullsized":"2024-04-21T08:51:10.830308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:22:06.372410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063168+0000","last_change":"2024-04-21T08:48:46.476997+0000","last_active":"2024-04-21T08:51:11.063168+0000","last_peered":"2024-04-21T08:51:11.063168+0000","last_clean":"2024-04-21T08:51:11.063168+0000","last_became_active":"2024-04-21T08:48:46.476620+0000","last_became_peered":"2024-04-21T08:48:46.476620+0000","last_unstale":"2024-04-21T08:51:11.063168+0000","last_undegraded":"2024-04-21T08:51:11.063168+0000","last_fullsized":"2024-04-21T08:51:11.063168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:37.569629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038421+0000","last_change":"2024-04-21T08:48:46.469355+0000","last_active":"2024-04-21T08:51:11.038421+0000","last_peered":"2024-04-21T08:51:11.038421+0000","last_clean":"2024-04-21T08:51:11.038421+0000","last_became_active":"2024-04-21T08:48:46.468604+0000","last_became_peered":"2024-04-21T08:48:46.468604+0000","last_unstale":"2024-04-21T08:51:11.038421+0000","last_undegraded":"2024-04-21T08:51:11.038421+0000","last_fullsized":"2024-04-21T08:51:11.038421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:23:54.500982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036699+0000","last_change":"2024-04-21T08:48:46.453116+0000","last_active":"2024-04-21T08:51:11.036699+0000","last_peered":"2024-04-21T08:51:11.036699+0000","last_clean":"2024-04-21T08:51:11.036699+0000","last_became_active":"2024-04-21T08:48:46.452607+0000","last_became_peered":"2024-04-21T08:48:46.452607+0000","last_unstale":"2024-04-21T08:51:11.036699+0000","last_undegraded":"2024-04-21T08:51:11.036699+0000","last_fullsized":"2024-04-21T08:51:11.036699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:57:04.371812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877416+0000","last_change":"2024-04-21T08:48:46.474516+0000","last_active":"2024-04-21T08:51:10.877416+0000","last_peered":"2024-04-21T08:51:10.877416+0000","last_clean":"2024-04-21T08:51:10.877416+0000","last_became_active":"2024-04-21T08:48:46.474129+0000","last_became_peered":"2024-04-21T08:48:46.474129+0000","last_unstale":"2024-04-21T08:51:10.877416+0000","last_undegraded":"2024-04-21T08:51:10.877416+0000","last_fullsized":"2024-04-21T08:51:10.877416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:40:47.784976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877479+0000","last_change":"2024-04-21T08:49:36.907964+0000","last_active":"2024-04-21T08:51:10.877479+0000","last_peered":"2024-04-21T08:51:10.877479+0000","last_clean":"2024-04-21T08:51:10.877479+0000","last_became_active":"2024-04-21T08:49:36.907595+0000","last_became_peered":"2024-04-21T08:49:36.907595+0000","last_unstale":"2024-04-21T08:51:10.877479+0000","last_undegraded":"2024-04-21T08:51:10.877479+0000","last_fullsized":"2024-04-21T08:51:10.877479+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:14:04.907328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006192+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:51:11.006192+0000","last_peered":"2024-04-21T08:51:11.006192+0000","last_clean":"2024-04-21T08:51:11.006192+0000","last_became_active":"2024-04-21T08:48:46.469389+0000","last_became_peered":"2024-04-21T08:48:46.469389+0000","last_unstale":"2024-04-21T08:51:11.006192+0000","last_undegraded":"2024-04-21T08:51:11.006192+0000","last_fullsized":"2024-04-21T08:51:11.006192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:11:45.824402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063161+0000","last_change":"2024-04-21T08:48:46.470059+0000","last_active":"2024-04-21T08:51:11.063161+0000","last_peered":"2024-04-21T08:51:11.063161+0000","last_clean":"2024-04-21T08:51:11.063161+0000","last_became_active":"2024-04-21T08:48:46.469751+0000","last_became_peered":"2024-04-21T08:48:46.469751+0000","last_unstale":"2024-04-21T08:51:11.063161+0000","last_undegraded":"2024-04-21T08:51:11.063161+0000","last_fullsized":"2024-04-21T08:51:11.063161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:16.317879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.937061+0000","last_change":"2024-04-21T08:48:46.470129+0000","last_active":"2024-04-21T08:51:10.937061+0000","last_peered":"2024-04-21T08:51:10.937061+0000","last_clean":"2024-04-21T08:51:10.937061+0000","last_became_active":"2024-04-21T08:48:46.469763+0000","last_became_peered":"2024-04-21T08:48:46.469763+0000","last_unstale":"2024-04-21T08:51:10.937061+0000","last_undegraded":"2024-04-21T08:51:10.937061+0000","last_fullsized":"2024-04-21T08:51:10.937061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:47.731201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832982+0000","last_change":"2024-04-21T08:48:46.465356+0000","last_active":"2024-04-21T08:51:10.832982+0000","last_peered":"2024-04-21T08:51:10.832982+0000","last_clean":"2024-04-21T08:51:10.832982+0000","last_became_active":"2024-04-21T08:48:46.464183+0000","last_became_peered":"2024-04-21T08:48:46.464183+0000","last_unstale":"2024-04-21T08:51:10.832982+0000","last_undegraded":"2024-04-21T08:51:10.832982+0000","last_fullsized":"2024-04-21T08:51:10.832982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:37.533733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832383+0000","last_change":"2024-04-21T08:48:50.767378+0000","last_active":"2024-04-21T08:51:10.832383+0000","last_peered":"2024-04-21T08:51:10.832383+0000","last_clean":"2024-04-21T08:51:10.832383+0000","last_became_active":"2024-04-21T08:48:50.766856+0000","last_became_peered":"2024-04-21T08:48:50.766856+0000","last_unstale":"2024-04-21T08:51:10.832383+0000","last_undegraded":"2024-04-21T08:51:10.832383+0000","last_fullsized":"2024-04-21T08:51:10.832383+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:00.434540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936767+0000","last_change":"2024-04-21T08:48:45.550284+0000","last_active":"2024-04-21T08:51:10.936767+0000","last_peered":"2024-04-21T08:51:10.936767+0000","last_clean":"2024-04-21T08:51:10.936767+0000","last_became_active":"2024-04-21T08:48:45.549940+0000","last_became_peered":"2024-04-21T08:48:45.549940+0000","last_unstale":"2024-04-21T08:51:10.936767+0000","last_undegraded":"2024-04-21T08:51:10.936767+0000","last_fullsized":"2024-04-21T08:51:10.936767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:12.343299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828471+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:51:10.828471+0000","last_peered":"2024-04-21T08:51:10.828471+0000","last_clean":"2024-04-21T08:51:10.828471+0000","last_became_active":"2024-04-21T08:48:46.469647+0000","last_became_peered":"2024-04-21T08:48:46.469647+0000","last_unstale":"2024-04-21T08:51:10.828471+0000","last_undegraded":"2024-04-21T08:51:10.828471+0000","last_fullsized":"2024-04-21T08:51:10.828471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:10:28.204554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.037993+0000","last_change":"2024-04-21T08:48:50.771871+0000","last_active":"2024-04-21T08:51:11.037993+0000","last_peered":"2024-04-21T08:51:11.037993+0000","last_clean":"2024-04-21T08:51:11.037993+0000","last_became_active":"2024-04-21T08:48:50.770442+0000","last_became_peered":"2024-04-21T08:48:50.770442+0000","last_unstale":"2024-04-21T08:51:11.037993+0000","last_undegraded":"2024-04-21T08:51:11.037993+0000","last_fullsized":"2024-04-21T08:51:11.037993+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:18:57.813971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828497+0000","last_change":"2024-04-21T08:48:45.566492+0000","last_active":"2024-04-21T08:51:10.828497+0000","last_peered":"2024-04-21T08:51:10.828497+0000","last_clean":"2024-04-21T08:51:10.828497+0000","last_became_active":"2024-04-21T08:48:45.566169+0000","last_became_peered":"2024-04-21T08:48:45.566169+0000","last_unstale":"2024-04-21T08:51:10.828497+0000","last_undegraded":"2024-04-21T08:51:10.828497+0000","last_fullsized":"2024-04-21T08:51:10.828497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:49.985664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877476+0000","last_change":"2024-04-21T08:48:46.470866+0000","last_active":"2024-04-21T08:51:10.877476+0000","last_peered":"2024-04-21T08:51:10.877476+0000","last_clean":"2024-04-21T08:51:10.877476+0000","last_became_active":"2024-04-21T08:48:46.470401+0000","last_became_peered":"2024-04-21T08:48:46.470401+0000","last_unstale":"2024-04-21T08:51:10.877476+0000","last_undegraded":"2024-04-21T08:51:10.877476+0000","last_fullsized":"2024-04-21T08:51:10.877476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:07.506212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038124+0000","last_change":"2024-04-21T08:48:50.772339+0000","last_active":"2024-04-21T08:51:11.038124+0000","last_peered":"2024-04-21T08:51:11.038124+0000","last_clean":"2024-04-21T08:51:11.038124+0000","last_became_active":"2024-04-21T08:48:50.771946+0000","last_became_peered":"2024-04-21T08:48:50.771946+0000","last_unstale":"2024-04-21T08:51:11.038124+0000","last_undegraded":"2024-04-21T08:51:11.038124+0000","last_fullsized":"2024-04-21T08:51:11.038124+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:03:56.994581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939819+0000","last_change":"2024-04-21T08:48:45.553946+0000","last_active":"2024-04-21T08:51:10.939819+0000","last_peered":"2024-04-21T08:51:10.939819+0000","last_clean":"2024-04-21T08:51:10.939819+0000","last_became_active":"2024-04-21T08:48:45.553410+0000","last_became_peered":"2024-04-21T08:48:45.553410+0000","last_unstale":"2024-04-21T08:51:10.939819+0000","last_undegraded":"2024-04-21T08:51:10.939819+0000","last_fullsized":"2024-04-21T08:51:10.939819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:24:02.387250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006432+0000","last_change":"2024-04-21T08:48:46.471815+0000","last_active":"2024-04-21T08:51:11.006432+0000","last_peered":"2024-04-21T08:51:11.006432+0000","last_clean":"2024-04-21T08:51:11.006432+0000","last_became_active":"2024-04-21T08:48:46.471492+0000","last_became_peered":"2024-04-21T08:48:46.471492+0000","last_unstale":"2024-04-21T08:51:11.006432+0000","last_undegraded":"2024-04-21T08:51:11.006432+0000","last_fullsized":"2024-04-21T08:51:11.006432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:26.051166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038027+0000","last_change":"2024-04-21T08:48:50.767916+0000","last_active":"2024-04-21T08:51:11.038027+0000","last_peered":"2024-04-21T08:51:11.038027+0000","last_clean":"2024-04-21T08:51:11.038027+0000","last_became_active":"2024-04-21T08:48:50.767326+0000","last_became_peered":"2024-04-21T08:48:50.767326+0000","last_unstale":"2024-04-21T08:51:11.038027+0000","last_undegraded":"2024-04-21T08:51:11.038027+0000","last_fullsized":"2024-04-21T08:51:11.038027+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:05:02.185369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939335+0000","last_change":"2024-04-21T08:48:45.549982+0000","last_active":"2024-04-21T08:51:10.939335+0000","last_peered":"2024-04-21T08:51:10.939335+0000","last_clean":"2024-04-21T08:51:10.939335+0000","last_became_active":"2024-04-21T08:48:45.549611+0000","last_became_peered":"2024-04-21T08:48:45.549611+0000","last_unstale":"2024-04-21T08:51:10.939335+0000","last_undegraded":"2024-04-21T08:51:10.939335+0000","last_fullsized":"2024-04-21T08:51:10.939335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:46.035388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945903+0000","last_change":"2024-04-21T08:48:46.459068+0000","last_active":"2024-04-21T08:51:10.945903+0000","last_peered":"2024-04-21T08:51:10.945903+0000","last_clean":"2024-04-21T08:51:10.945903+0000","last_became_active":"2024-04-21T08:48:46.458686+0000","last_became_peered":"2024-04-21T08:48:46.458686+0000","last_unstale":"2024-04-21T08:51:10.945903+0000","last_undegraded":"2024-04-21T08:51:10.945903+0000","last_fullsized":"2024-04-21T08:51:10.945903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:31.373451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832831+0000","last_change":"2024-04-21T08:48:50.761392+0000","last_active":"2024-04-21T08:51:10.832831+0000","last_peered":"2024-04-21T08:51:10.832831+0000","last_clean":"2024-04-21T08:51:10.832831+0000","last_became_active":"2024-04-21T08:48:50.760917+0000","last_became_peered":"2024-04-21T08:48:50.760917+0000","last_unstale":"2024-04-21T08:51:10.832831+0000","last_undegraded":"2024-04-21T08:51:10.832831+0000","last_fullsized":"2024-04-21T08:51:10.832831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:19.070502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'8","reported_seq":175,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.027752+0000","last_change":"2024-04-21T08:48:45.553686+0000","last_active":"2024-04-21T08:51:20.027752+0000","last_peered":"2024-04-21T08:51:20.027752+0000","last_clean":"2024-04-21T08:51:20.027752+0000","last_became_active":"2024-04-21T08:48:45.553322+0000","last_became_peered":"2024-04-21T08:48:45.553322+0000","last_unstale":"2024-04-21T08:51:20.027752+0000","last_undegraded":"2024-04-21T08:51:20.027752+0000","last_fullsized":"2024-04-21T08:51:20.027752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:15:24.320704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1712,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":133,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939678+0000","last_change":"2024-04-21T08:48:46.474555+0000","last_active":"2024-04-21T08:51:10.939678+0000","last_peered":"2024-04-21T08:51:10.939678+0000","last_clean":"2024-04-21T08:51:10.939678+0000","last_became_active":"2024-04-21T08:48:46.474185+0000","last_became_peered":"2024-04-21T08:48:46.474185+0000","last_unstale":"2024-04-21T08:51:10.939678+0000","last_undegraded":"2024-04-21T08:51:10.939678+0000","last_fullsized":"2024-04-21T08:51:10.939678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:52.884994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832578+0000","last_change":"2024-04-21T08:48:50.763181+0000","last_active":"2024-04-21T08:51:10.832578+0000","last_peered":"2024-04-21T08:51:10.832578+0000","last_clean":"2024-04-21T08:51:10.832578+0000","last_became_active":"2024-04-21T08:48:50.762718+0000","last_became_peered":"2024-04-21T08:48:50.762718+0000","last_unstale":"2024-04-21T08:51:10.832578+0000","last_undegraded":"2024-04-21T08:51:10.832578+0000","last_fullsized":"2024-04-21T08:51:10.832578+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:16:23.204605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829837+0000","last_change":"2024-04-21T08:48:45.547768+0000","last_active":"2024-04-21T08:51:10.829837+0000","last_peered":"2024-04-21T08:51:10.829837+0000","last_clean":"2024-04-21T08:51:10.829837+0000","last_became_active":"2024-04-21T08:48:45.547456+0000","last_became_peered":"2024-04-21T08:48:45.547456+0000","last_unstale":"2024-04-21T08:51:10.829837+0000","last_undegraded":"2024-04-21T08:51:10.829837+0000","last_fullsized":"2024-04-21T08:51:10.829837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:23.502799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945877+0000","last_change":"2024-04-21T08:48:46.467988+0000","last_active":"2024-04-21T08:51:10.945877+0000","last_peered":"2024-04-21T08:51:10.945877+0000","last_clean":"2024-04-21T08:51:10.945877+0000","last_became_active":"2024-04-21T08:48:46.465212+0000","last_became_peered":"2024-04-21T08:48:46.465212+0000","last_unstale":"2024-04-21T08:51:10.945877+0000","last_undegraded":"2024-04-21T08:51:10.945877+0000","last_fullsized":"2024-04-21T08:51:10.945877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:41.743506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063079+0000","last_change":"2024-04-21T08:50:37.350070+0000","last_active":"2024-04-21T08:51:11.063079+0000","last_peered":"2024-04-21T08:51:11.063079+0000","last_clean":"2024-04-21T08:51:11.063079+0000","last_became_active":"2024-04-21T08:50:37.349622+0000","last_became_peered":"2024-04-21T08:50:37.349622+0000","last_unstale":"2024-04-21T08:51:11.063079+0000","last_undegraded":"2024-04-21T08:51:11.063079+0000","last_fullsized":"2024-04-21T08:51:11.063079+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:55.050244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.035911+0000","last_change":"2024-04-21T08:48:45.545615+0000","last_active":"2024-04-21T08:51:11.035911+0000","last_peered":"2024-04-21T08:51:11.035911+0000","last_clean":"2024-04-21T08:51:11.035911+0000","last_became_active":"2024-04-21T08:48:45.545280+0000","last_became_peered":"2024-04-21T08:48:45.545280+0000","last_unstale":"2024-04-21T08:51:11.035911+0000","last_undegraded":"2024-04-21T08:51:11.035911+0000","last_fullsized":"2024-04-21T08:51:11.035911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:50:00.470903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828078+0000","last_change":"2024-04-21T08:48:46.477326+0000","last_active":"2024-04-21T08:51:10.828078+0000","last_peered":"2024-04-21T08:51:10.828078+0000","last_clean":"2024-04-21T08:51:10.828078+0000","last_became_active":"2024-04-21T08:48:46.476914+0000","last_became_peered":"2024-04-21T08:48:46.476914+0000","last_unstale":"2024-04-21T08:51:10.828078+0000","last_undegraded":"2024-04-21T08:51:10.828078+0000","last_fullsized":"2024-04-21T08:51:10.828078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:04.312061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877774+0000","last_change":"2024-04-21T08:48:50.767429+0000","last_active":"2024-04-21T08:51:10.877774+0000","last_peered":"2024-04-21T08:51:10.877774+0000","last_clean":"2024-04-21T08:51:10.877774+0000","last_became_active":"2024-04-21T08:48:50.767008+0000","last_became_peered":"2024-04-21T08:48:50.767008+0000","last_unstale":"2024-04-21T08:51:10.877774+0000","last_undegraded":"2024-04-21T08:51:10.877774+0000","last_fullsized":"2024-04-21T08:51:10.877774+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:26.865571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945328+0000","last_change":"2024-04-21T08:48:45.553170+0000","last_active":"2024-04-21T08:51:10.945328+0000","last_peered":"2024-04-21T08:51:10.945328+0000","last_clean":"2024-04-21T08:51:10.945328+0000","last_became_active":"2024-04-21T08:48:45.552645+0000","last_became_peered":"2024-04-21T08:48:45.552645+0000","last_unstale":"2024-04-21T08:51:10.945328+0000","last_undegraded":"2024-04-21T08:51:10.945328+0000","last_fullsized":"2024-04-21T08:51:10.945328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:32:07.217212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936256+0000","last_change":"2024-04-21T08:48:46.478101+0000","last_active":"2024-04-21T08:51:10.936256+0000","last_peered":"2024-04-21T08:51:10.936256+0000","last_clean":"2024-04-21T08:51:10.936256+0000","last_became_active":"2024-04-21T08:48:46.477725+0000","last_became_peered":"2024-04-21T08:48:46.477725+0000","last_unstale":"2024-04-21T08:51:10.936256+0000","last_undegraded":"2024-04-21T08:51:10.936256+0000","last_fullsized":"2024-04-21T08:51:10.936256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:37:55.429569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828279+0000","last_change":"2024-04-21T08:48:50.762899+0000","last_active":"2024-04-21T08:51:10.828279+0000","last_peered":"2024-04-21T08:51:10.828279+0000","last_clean":"2024-04-21T08:51:10.828279+0000","last_became_active":"2024-04-21T08:48:50.762475+0000","last_became_peered":"2024-04-21T08:48:50.762475+0000","last_unstale":"2024-04-21T08:51:10.828279+0000","last_undegraded":"2024-04-21T08:51:10.828279+0000","last_fullsized":"2024-04-21T08:51:10.828279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:27.540414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":167,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.027197+0000","last_change":"2024-04-21T08:48:45.550151+0000","last_active":"2024-04-21T08:51:20.027197+0000","last_peered":"2024-04-21T08:51:20.027197+0000","last_clean":"2024-04-21T08:51:20.027197+0000","last_became_active":"2024-04-21T08:48:45.549829+0000","last_became_peered":"2024-04-21T08:48:45.549829+0000","last_unstale":"2024-04-21T08:51:20.027197+0000","last_undegraded":"2024-04-21T08:51:20.027197+0000","last_fullsized":"2024-04-21T08:51:20.027197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:14.061189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":129,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006968+0000","last_change":"2024-04-21T08:48:46.472072+0000","last_active":"2024-04-21T08:51:11.006968+0000","last_peered":"2024-04-21T08:51:11.006968+0000","last_clean":"2024-04-21T08:51:11.006968+0000","last_became_active":"2024-04-21T08:48:46.471660+0000","last_became_peered":"2024-04-21T08:48:46.471660+0000","last_unstale":"2024-04-21T08:51:11.006968+0000","last_undegraded":"2024-04-21T08:51:11.006968+0000","last_fullsized":"2024-04-21T08:51:11.006968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:30:11.843870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936126+0000","last_change":"2024-04-21T08:48:50.773021+0000","last_active":"2024-04-21T08:51:10.936126+0000","last_peered":"2024-04-21T08:51:10.936126+0000","last_clean":"2024-04-21T08:51:10.936126+0000","last_became_active":"2024-04-21T08:48:50.771746+0000","last_became_peered":"2024-04-21T08:48:50.771746+0000","last_unstale":"2024-04-21T08:51:10.936126+0000","last_undegraded":"2024-04-21T08:51:10.936126+0000","last_fullsized":"2024-04-21T08:51:10.936126+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:33:21.415967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063066+0000","last_change":"2024-04-21T08:48:45.550376+0000","last_active":"2024-04-21T08:51:11.063066+0000","last_peered":"2024-04-21T08:51:11.063066+0000","last_clean":"2024-04-21T08:51:11.063066+0000","last_became_active":"2024-04-21T08:48:45.549594+0000","last_became_peered":"2024-04-21T08:48:45.549594+0000","last_unstale":"2024-04-21T08:51:11.063066+0000","last_undegraded":"2024-04-21T08:51:11.063066+0000","last_fullsized":"2024-04-21T08:51:11.063066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:26:27.046963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877696+0000","last_change":"2024-04-21T08:48:46.469499+0000","last_active":"2024-04-21T08:51:10.877696+0000","last_peered":"2024-04-21T08:51:10.877696+0000","last_clean":"2024-04-21T08:51:10.877696+0000","last_became_active":"2024-04-21T08:48:46.469124+0000","last_became_peered":"2024-04-21T08:48:46.469124+0000","last_unstale":"2024-04-21T08:51:10.877696+0000","last_undegraded":"2024-04-21T08:51:10.877696+0000","last_fullsized":"2024-04-21T08:51:10.877696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:58.708916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038260+0000","last_change":"2024-04-21T08:50:37.352313+0000","last_active":"2024-04-21T08:51:11.038260+0000","last_peered":"2024-04-21T08:51:11.038260+0000","last_clean":"2024-04-21T08:51:11.038260+0000","last_became_active":"2024-04-21T08:50:37.351775+0000","last_became_peered":"2024-04-21T08:50:37.351775+0000","last_unstale":"2024-04-21T08:51:11.038260+0000","last_undegraded":"2024-04-21T08:51:11.038260+0000","last_fullsized":"2024-04-21T08:51:11.038260+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:12:51.239368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,10],"acting":[4,1,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945985+0000","last_change":"2024-04-21T08:48:45.559319+0000","last_active":"2024-04-21T08:51:10.945985+0000","last_peered":"2024-04-21T08:51:10.945985+0000","last_clean":"2024-04-21T08:51:10.945985+0000","last_became_active":"2024-04-21T08:48:45.558895+0000","last_became_peered":"2024-04-21T08:48:45.558895+0000","last_unstale":"2024-04-21T08:51:10.945985+0000","last_undegraded":"2024-04-21T08:51:10.945985+0000","last_fullsized":"2024-04-21T08:51:10.945985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:01.878169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006563+0000","last_change":"2024-04-21T08:48:46.466110+0000","last_active":"2024-04-21T08:51:11.006563+0000","last_peered":"2024-04-21T08:51:11.006563+0000","last_clean":"2024-04-21T08:51:11.006563+0000","last_became_active":"2024-04-21T08:48:46.464374+0000","last_became_peered":"2024-04-21T08:48:46.464374+0000","last_unstale":"2024-04-21T08:51:11.006563+0000","last_undegraded":"2024-04-21T08:51:11.006563+0000","last_fullsized":"2024-04-21T08:51:11.006563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:26.307051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936310+0000","last_change":"2024-04-21T08:48:50.759976+0000","last_active":"2024-04-21T08:51:10.936310+0000","last_peered":"2024-04-21T08:51:10.936310+0000","last_clean":"2024-04-21T08:51:10.936310+0000","last_became_active":"2024-04-21T08:48:50.759596+0000","last_became_peered":"2024-04-21T08:48:50.759596+0000","last_unstale":"2024-04-21T08:51:10.936310+0000","last_undegraded":"2024-04-21T08:51:10.936310+0000","last_fullsized":"2024-04-21T08:51:10.936310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:09.678388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'42","reported_seq":347,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:21.028496+0000","last_change":"2024-04-21T08:48:45.549228+0000","last_active":"2024-04-21T08:51:21.028496+0000","last_peered":"2024-04-21T08:51:21.028496+0000","last_clean":"2024-04-21T08:51:21.028496+0000","last_became_active":"2024-04-21T08:48:45.548865+0000","last_became_peered":"2024-04-21T08:48:45.548865+0000","last_unstale":"2024-04-21T08:51:21.028496+0000","last_undegraded":"2024-04-21T08:51:21.028496+0000","last_fullsized":"2024-04-21T08:51:21.028496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:28.620968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71920,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":271,"num_read_kb":207,"num_write":43,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939637+0000","last_change":"2024-04-21T08:48:46.470322+0000","last_active":"2024-04-21T08:51:10.939637+0000","last_peered":"2024-04-21T08:51:10.939637+0000","last_clean":"2024-04-21T08:51:10.939637+0000","last_became_active":"2024-04-21T08:48:46.467225+0000","last_became_peered":"2024-04-21T08:48:46.467225+0000","last_unstale":"2024-04-21T08:51:10.939637+0000","last_undegraded":"2024-04-21T08:51:10.939637+0000","last_fullsized":"2024-04-21T08:51:10.939637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:31:17.390926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.946326+0000","last_change":"2024-04-21T08:48:50.770982+0000","last_active":"2024-04-21T08:51:10.946326+0000","last_peered":"2024-04-21T08:51:10.946326+0000","last_clean":"2024-04-21T08:51:10.946326+0000","last_became_active":"2024-04-21T08:48:50.770312+0000","last_became_peered":"2024-04-21T08:48:50.770312+0000","last_unstale":"2024-04-21T08:51:10.946326+0000","last_undegraded":"2024-04-21T08:51:10.946326+0000","last_fullsized":"2024-04-21T08:51:10.946326+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:41:33.027351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832599+0000","last_change":"2024-04-21T08:48:45.556134+0000","last_active":"2024-04-21T08:51:10.832599+0000","last_peered":"2024-04-21T08:51:10.832599+0000","last_clean":"2024-04-21T08:51:10.832599+0000","last_became_active":"2024-04-21T08:48:45.555818+0000","last_became_peered":"2024-04-21T08:48:45.555818+0000","last_unstale":"2024-04-21T08:51:10.832599+0000","last_undegraded":"2024-04-21T08:51:10.832599+0000","last_fullsized":"2024-04-21T08:51:10.832599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:00:08.083783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036365+0000","last_change":"2024-04-21T08:49:36.913151+0000","last_active":"2024-04-21T08:51:11.036365+0000","last_peered":"2024-04-21T08:51:11.036365+0000","last_clean":"2024-04-21T08:51:11.036365+0000","last_became_active":"2024-04-21T08:49:36.912810+0000","last_became_peered":"2024-04-21T08:49:36.912810+0000","last_unstale":"2024-04-21T08:51:11.036365+0000","last_undegraded":"2024-04-21T08:51:11.036365+0000","last_fullsized":"2024-04-21T08:51:11.036365+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:17:32.887163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062954+0000","last_change":"2024-04-21T08:48:50.765967+0000","last_active":"2024-04-21T08:51:11.062954+0000","last_peered":"2024-04-21T08:51:11.062954+0000","last_clean":"2024-04-21T08:51:11.062954+0000","last_became_active":"2024-04-21T08:48:50.765557+0000","last_became_peered":"2024-04-21T08:48:50.765557+0000","last_unstale":"2024-04-21T08:51:11.062954+0000","last_undegraded":"2024-04-21T08:51:11.062954+0000","last_fullsized":"2024-04-21T08:51:11.062954+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:52.044058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945823+0000","last_change":"2024-04-21T08:48:45.559939+0000","last_active":"2024-04-21T08:51:10.945823+0000","last_peered":"2024-04-21T08:51:10.945823+0000","last_clean":"2024-04-21T08:51:10.945823+0000","last_became_active":"2024-04-21T08:48:45.559635+0000","last_became_peered":"2024-04-21T08:48:45.559635+0000","last_unstale":"2024-04-21T08:51:10.945823+0000","last_undegraded":"2024-04-21T08:51:10.945823+0000","last_fullsized":"2024-04-21T08:51:10.945823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:17.872806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062944+0000","last_change":"2024-04-21T08:48:46.464878+0000","last_active":"2024-04-21T08:51:11.062944+0000","last_peered":"2024-04-21T08:51:11.062944+0000","last_clean":"2024-04-21T08:51:11.062944+0000","last_became_active":"2024-04-21T08:48:46.464552+0000","last_became_peered":"2024-04-21T08:48:46.464552+0000","last_unstale":"2024-04-21T08:51:11.062944+0000","last_undegraded":"2024-04-21T08:51:11.062944+0000","last_fullsized":"2024-04-21T08:51:11.062944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:46:10.558693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939454+0000","last_change":"2024-04-21T08:50:37.351314+0000","last_active":"2024-04-21T08:51:10.939454+0000","last_peered":"2024-04-21T08:51:10.939454+0000","last_clean":"2024-04-21T08:51:10.939454+0000","last_became_active":"2024-04-21T08:50:37.350956+0000","last_became_peered":"2024-04-21T08:50:37.350956+0000","last_unstale":"2024-04-21T08:51:10.939454+0000","last_undegraded":"2024-04-21T08:51:10.939454+0000","last_fullsized":"2024-04-21T08:51:10.939454+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:01.802427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"94'7","reported_seq":175,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.050635+0000","last_change":"2024-04-21T08:48:45.548685+0000","last_active":"2024-04-21T08:51:20.050635+0000","last_peered":"2024-04-21T08:51:20.050635+0000","last_clean":"2024-04-21T08:51:20.050635+0000","last_became_active":"2024-04-21T08:48:45.548372+0000","last_became_peered":"2024-04-21T08:48:45.548372+0000","last_unstale":"2024-04-21T08:51:20.050635+0000","last_undegraded":"2024-04-21T08:51:20.050635+0000","last_fullsized":"2024-04-21T08:51:20.050635+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:44.780297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":135,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062806+0000","last_change":"2024-04-21T08:48:46.471541+0000","last_active":"2024-04-21T08:51:11.062806+0000","last_peered":"2024-04-21T08:51:11.062806+0000","last_clean":"2024-04-21T08:51:11.062806+0000","last_became_active":"2024-04-21T08:48:46.471223+0000","last_became_peered":"2024-04-21T08:48:46.471223+0000","last_unstale":"2024-04-21T08:51:11.062806+0000","last_undegraded":"2024-04-21T08:51:11.062806+0000","last_fullsized":"2024-04-21T08:51:11.062806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:24:50.867905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036983+0000","last_change":"2024-04-21T08:48:50.773941+0000","last_active":"2024-04-21T08:51:11.036983+0000","last_peered":"2024-04-21T08:51:11.036983+0000","last_clean":"2024-04-21T08:51:11.036983+0000","last_became_active":"2024-04-21T08:48:50.772081+0000","last_became_peered":"2024-04-21T08:48:50.772081+0000","last_unstale":"2024-04-21T08:51:11.036983+0000","last_undegraded":"2024-04-21T08:51:11.036983+0000","last_fullsized":"2024-04-21T08:51:11.036983+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:04.438944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062783+0000","last_change":"2024-04-21T08:49:36.923090+0000","last_active":"2024-04-21T08:51:11.062783+0000","last_peered":"2024-04-21T08:51:11.062783+0000","last_clean":"2024-04-21T08:51:11.062783+0000","last_became_active":"2024-04-21T08:49:36.908075+0000","last_became_peered":"2024-04-21T08:49:36.908075+0000","last_unstale":"2024-04-21T08:51:11.062783+0000","last_undegraded":"2024-04-21T08:51:11.062783+0000","last_fullsized":"2024-04-21T08:51:11.062783+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:41.683372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945471+0000","last_change":"2024-04-21T08:48:46.465559+0000","last_active":"2024-04-21T08:51:10.945471+0000","last_peered":"2024-04-21T08:51:10.945471+0000","last_clean":"2024-04-21T08:51:10.945471+0000","last_became_active":"2024-04-21T08:48:46.465125+0000","last_became_peered":"2024-04-21T08:48:46.465125+0000","last_unstale":"2024-04-21T08:51:10.945471+0000","last_undegraded":"2024-04-21T08:51:10.945471+0000","last_fullsized":"2024-04-21T08:51:10.945471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:52:38.423925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945406+0000","last_change":"2024-04-21T08:48:50.763609+0000","last_active":"2024-04-21T08:51:10.945406+0000","last_peered":"2024-04-21T08:51:10.945406+0000","last_clean":"2024-04-21T08:51:10.945406+0000","last_became_active":"2024-04-21T08:48:50.763144+0000","last_became_peered":"2024-04-21T08:48:50.763144+0000","last_unstale":"2024-04-21T08:51:10.945406+0000","last_undegraded":"2024-04-21T08:51:10.945406+0000","last_fullsized":"2024-04-21T08:51:10.945406+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:26:18.611235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939721+0000","last_change":"2024-04-21T08:48:45.549912+0000","last_active":"2024-04-21T08:51:10.939721+0000","last_peered":"2024-04-21T08:51:10.939721+0000","last_clean":"2024-04-21T08:51:10.939721+0000","last_became_active":"2024-04-21T08:48:45.549585+0000","last_became_peered":"2024-04-21T08:48:45.549585+0000","last_unstale":"2024-04-21T08:51:10.939721+0000","last_undegraded":"2024-04-21T08:51:10.939721+0000","last_fullsized":"2024-04-21T08:51:10.939721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:58.287402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063099+0000","last_change":"2024-04-21T08:48:46.462746+0000","last_active":"2024-04-21T08:51:11.063099+0000","last_peered":"2024-04-21T08:51:11.063099+0000","last_clean":"2024-04-21T08:51:11.063099+0000","last_became_active":"2024-04-21T08:48:46.461602+0000","last_became_peered":"2024-04-21T08:48:46.461602+0000","last_unstale":"2024-04-21T08:51:11.063099+0000","last_undegraded":"2024-04-21T08:51:11.063099+0000","last_fullsized":"2024-04-21T08:51:11.063099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:52.198634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939308+0000","last_change":"2024-04-21T08:48:50.770770+0000","last_active":"2024-04-21T08:51:10.939308+0000","last_peered":"2024-04-21T08:51:10.939308+0000","last_clean":"2024-04-21T08:51:10.939308+0000","last_became_active":"2024-04-21T08:48:50.770326+0000","last_became_peered":"2024-04-21T08:48:50.770326+0000","last_unstale":"2024-04-21T08:51:10.939308+0000","last_undegraded":"2024-04-21T08:51:10.939308+0000","last_fullsized":"2024-04-21T08:51:10.939308+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:38.325155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936408+0000","last_change":"2024-04-21T08:48:45.558867+0000","last_active":"2024-04-21T08:51:10.936408+0000","last_peered":"2024-04-21T08:51:10.936408+0000","last_clean":"2024-04-21T08:51:10.936408+0000","last_became_active":"2024-04-21T08:48:45.558492+0000","last_became_peered":"2024-04-21T08:48:45.558492+0000","last_unstale":"2024-04-21T08:51:10.936408+0000","last_undegraded":"2024-04-21T08:51:10.936408+0000","last_fullsized":"2024-04-21T08:51:10.936408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:44:55.156404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.830180+0000","last_change":"2024-04-21T08:48:46.466461+0000","last_active":"2024-04-21T08:51:10.830180+0000","last_peered":"2024-04-21T08:51:10.830180+0000","last_clean":"2024-04-21T08:51:10.830180+0000","last_became_active":"2024-04-21T08:48:46.466035+0000","last_became_peered":"2024-04-21T08:48:46.466035+0000","last_unstale":"2024-04-21T08:51:10.830180+0000","last_undegraded":"2024-04-21T08:51:10.830180+0000","last_fullsized":"2024-04-21T08:51:10.830180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:47:32.307753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036555+0000","last_change":"2024-04-21T08:48:50.767655+0000","last_active":"2024-04-21T08:51:11.036555+0000","last_peered":"2024-04-21T08:51:11.036555+0000","last_clean":"2024-04-21T08:51:11.036555+0000","last_became_active":"2024-04-21T08:48:50.767305+0000","last_became_peered":"2024-04-21T08:48:50.767305+0000","last_unstale":"2024-04-21T08:51:11.036555+0000","last_undegraded":"2024-04-21T08:51:11.036555+0000","last_fullsized":"2024-04-21T08:51:11.036555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:31.210126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062757+0000","last_change":"2024-04-21T08:48:45.552952+0000","last_active":"2024-04-21T08:51:11.062757+0000","last_peered":"2024-04-21T08:51:11.062757+0000","last_clean":"2024-04-21T08:51:11.062757+0000","last_became_active":"2024-04-21T08:48:45.552556+0000","last_became_peered":"2024-04-21T08:48:45.552556+0000","last_unstale":"2024-04-21T08:51:11.062757+0000","last_undegraded":"2024-04-21T08:51:11.062757+0000","last_fullsized":"2024-04-21T08:51:11.062757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:03.428801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829859+0000","last_change":"2024-04-21T08:48:46.469020+0000","last_active":"2024-04-21T08:51:10.829859+0000","last_peered":"2024-04-21T08:51:10.829859+0000","last_clean":"2024-04-21T08:51:10.829859+0000","last_became_active":"2024-04-21T08:48:46.468631+0000","last_became_peered":"2024-04-21T08:48:46.468631+0000","last_unstale":"2024-04-21T08:51:10.829859+0000","last_undegraded":"2024-04-21T08:51:10.829859+0000","last_fullsized":"2024-04-21T08:51:10.829859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:08.559516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036969+0000","last_change":"2024-04-21T08:48:50.773942+0000","last_active":"2024-04-21T08:51:11.036969+0000","last_peered":"2024-04-21T08:51:11.036969+0000","last_clean":"2024-04-21T08:51:11.036969+0000","last_became_active":"2024-04-21T08:48:50.771876+0000","last_became_peered":"2024-04-21T08:48:50.771876+0000","last_unstale":"2024-04-21T08:51:11.036969+0000","last_undegraded":"2024-04-21T08:51:11.036969+0000","last_fullsized":"2024-04-21T08:51:11.036969+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:19.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945411+0000","last_change":"2024-04-21T08:48:46.462727+0000","last_active":"2024-04-21T08:51:10.945411+0000","last_peered":"2024-04-21T08:51:10.945411+0000","last_clean":"2024-04-21T08:51:10.945411+0000","last_became_active":"2024-04-21T08:48:46.461669+0000","last_became_peered":"2024-04-21T08:48:46.461669+0000","last_unstale":"2024-04-21T08:51:10.945411+0000","last_undegraded":"2024-04-21T08:51:10.945411+0000","last_fullsized":"2024-04-21T08:51:10.945411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:27.290790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.049963+0000","last_change":"2024-04-21T08:48:45.558541+0000","last_active":"2024-04-21T08:51:20.049963+0000","last_peered":"2024-04-21T08:51:20.049963+0000","last_clean":"2024-04-21T08:51:20.049963+0000","last_became_active":"2024-04-21T08:48:45.558186+0000","last_became_peered":"2024-04-21T08:48:45.558186+0000","last_unstale":"2024-04-21T08:51:20.049963+0000","last_undegraded":"2024-04-21T08:51:20.049963+0000","last_fullsized":"2024-04-21T08:51:20.049963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:02.919144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":129,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062978+0000","last_change":"2024-04-21T08:48:50.770838+0000","last_active":"2024-04-21T08:51:11.062978+0000","last_peered":"2024-04-21T08:51:11.062978+0000","last_clean":"2024-04-21T08:51:11.062978+0000","last_became_active":"2024-04-21T08:48:50.770447+0000","last_became_peered":"2024-04-21T08:48:50.770447+0000","last_unstale":"2024-04-21T08:51:11.062978+0000","last_undegraded":"2024-04-21T08:51:11.062978+0000","last_fullsized":"2024-04-21T08:51:11.062978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:48:21.010284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828090+0000","last_change":"2024-04-21T08:48:46.478145+0000","last_active":"2024-04-21T08:51:10.828090+0000","last_peered":"2024-04-21T08:51:10.828090+0000","last_clean":"2024-04-21T08:51:10.828090+0000","last_became_active":"2024-04-21T08:48:46.477813+0000","last_became_peered":"2024-04-21T08:48:46.477813+0000","last_unstale":"2024-04-21T08:51:10.828090+0000","last_undegraded":"2024-04-21T08:51:10.828090+0000","last_fullsized":"2024-04-21T08:51:10.828090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:15.917619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038191+0000","last_change":"2024-04-21T08:48:45.545269+0000","last_active":"2024-04-21T08:51:11.038191+0000","last_peered":"2024-04-21T08:51:11.038191+0000","last_clean":"2024-04-21T08:51:11.038191+0000","last_became_active":"2024-04-21T08:48:45.544887+0000","last_became_peered":"2024-04-21T08:48:45.544887+0000","last_unstale":"2024-04-21T08:51:11.038191+0000","last_undegraded":"2024-04-21T08:51:11.038191+0000","last_fullsized":"2024-04-21T08:51:11.038191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:27.222570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"94'148","reported_seq":411,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:12.946003+0000","last_change":"2024-04-21T08:47:34.198161+0000","last_active":"2024-04-21T08:51:12.946003+0000","last_peered":"2024-04-21T08:51:12.946003+0000","last_clean":"2024-04-21T08:51:12.946003+0000","last_became_active":"2024-04-21T08:47:34.197654+0000","last_became_peered":"2024-04-21T08:47:34.197654+0000","last_unstale":"2024-04-21T08:51:12.946003+0000","last_undegraded":"2024-04-21T08:51:12.946003+0000","last_fullsized":"2024-04-21T08:51:12.946003+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_clean_scrub_stamp":"2024-04-21T08:44:05.493464+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:58:58.295783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945722+0000","last_change":"2024-04-21T08:48:50.770826+0000","last_active":"2024-04-21T08:51:10.945722+0000","last_peered":"2024-04-21T08:51:10.945722+0000","last_clean":"2024-04-21T08:51:10.945722+0000","last_became_active":"2024-04-21T08:48:50.770372+0000","last_became_peered":"2024-04-21T08:48:50.770372+0000","last_unstale":"2024-04-21T08:51:10.945722+0000","last_undegraded":"2024-04-21T08:51:10.945722+0000","last_fullsized":"2024-04-21T08:51:10.945722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:59.637874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.035696+0000","last_change":"2024-04-21T08:48:46.459330+0000","last_active":"2024-04-21T08:51:11.035696+0000","last_peered":"2024-04-21T08:51:11.035696+0000","last_clean":"2024-04-21T08:51:11.035696+0000","last_became_active":"2024-04-21T08:48:46.458963+0000","last_became_peered":"2024-04-21T08:48:46.458963+0000","last_unstale":"2024-04-21T08:51:11.035696+0000","last_undegraded":"2024-04-21T08:51:11.035696+0000","last_fullsized":"2024-04-21T08:51:11.035696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:10.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062640+0000","last_change":"2024-04-21T08:49:36.917854+0000","last_active":"2024-04-21T08:51:11.062640+0000","last_peered":"2024-04-21T08:51:11.062640+0000","last_clean":"2024-04-21T08:51:11.062640+0000","last_became_active":"2024-04-21T08:49:36.915390+0000","last_became_peered":"2024-04-21T08:49:36.915390+0000","last_unstale":"2024-04-21T08:51:11.062640+0000","last_undegraded":"2024-04-21T08:51:11.062640+0000","last_fullsized":"2024-04-21T08:51:11.062640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:45.920752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062798+0000","last_change":"2024-04-21T08:48:50.765339+0000","last_active":"2024-04-21T08:51:11.062798+0000","last_peered":"2024-04-21T08:51:11.062798+0000","last_clean":"2024-04-21T08:51:11.062798+0000","last_became_active":"2024-04-21T08:48:50.764858+0000","last_became_peered":"2024-04-21T08:48:50.764858+0000","last_unstale":"2024-04-21T08:51:11.062798+0000","last_undegraded":"2024-04-21T08:51:11.062798+0000","last_fullsized":"2024-04-21T08:51:11.062798+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:26.075041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939660+0000","last_change":"2024-04-21T08:48:45.557768+0000","last_active":"2024-04-21T08:51:10.939660+0000","last_peered":"2024-04-21T08:51:10.939660+0000","last_clean":"2024-04-21T08:51:10.939660+0000","last_became_active":"2024-04-21T08:48:45.557448+0000","last_became_peered":"2024-04-21T08:48:45.557448+0000","last_unstale":"2024-04-21T08:51:10.939660+0000","last_undegraded":"2024-04-21T08:51:10.939660+0000","last_fullsized":"2024-04-21T08:51:10.939660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:09.063913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036820+0000","last_change":"2024-04-21T08:48:46.459425+0000","last_active":"2024-04-21T08:51:11.036820+0000","last_peered":"2024-04-21T08:51:11.036820+0000","last_clean":"2024-04-21T08:51:11.036820+0000","last_became_active":"2024-04-21T08:48:46.459020+0000","last_became_peered":"2024-04-21T08:48:46.459020+0000","last_unstale":"2024-04-21T08:51:11.036820+0000","last_undegraded":"2024-04-21T08:51:11.036820+0000","last_fullsized":"2024-04-21T08:51:11.036820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:08:39.042825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828005+0000","last_change":"2024-04-21T08:48:50.765740+0000","last_active":"2024-04-21T08:51:10.828005+0000","last_peered":"2024-04-21T08:51:10.828005+0000","last_clean":"2024-04-21T08:51:10.828005+0000","last_became_active":"2024-04-21T08:48:50.765359+0000","last_became_peered":"2024-04-21T08:48:50.765359+0000","last_unstale":"2024-04-21T08:51:10.828005+0000","last_undegraded":"2024-04-21T08:51:10.828005+0000","last_fullsized":"2024-04-21T08:51:10.828005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:34:42.716165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'98","reported_seq":224,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.520533+0000","last_change":"2024-04-21T08:48:45.564532+0000","last_active":"2024-04-21T08:51:20.520533+0000","last_peered":"2024-04-21T08:51:20.520533+0000","last_clean":"2024-04-21T08:51:20.520533+0000","last_became_active":"2024-04-21T08:48:45.564225+0000","last_became_peered":"2024-04-21T08:48:45.564225+0000","last_unstale":"2024-04-21T08:51:20.520533+0000","last_undegraded":"2024-04-21T08:51:20.520533+0000","last_fullsized":"2024-04-21T08:51:20.520533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:42.850371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":179,"num_write_kb":1285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828519+0000","last_change":"2024-04-21T08:48:46.466622+0000","last_active":"2024-04-21T08:51:10.828519+0000","last_peered":"2024-04-21T08:51:10.828519+0000","last_clean":"2024-04-21T08:51:10.828519+0000","last_became_active":"2024-04-21T08:48:46.466129+0000","last_became_peered":"2024-04-21T08:48:46.466129+0000","last_unstale":"2024-04-21T08:51:10.828519+0000","last_undegraded":"2024-04-21T08:51:10.828519+0000","last_fullsized":"2024-04-21T08:51:10.828519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:20:26.064193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832121+0000","last_change":"2024-04-21T08:48:50.759534+0000","last_active":"2024-04-21T08:51:10.832121+0000","last_peered":"2024-04-21T08:51:10.832121+0000","last_clean":"2024-04-21T08:51:10.832121+0000","last_became_active":"2024-04-21T08:48:50.759116+0000","last_became_peered":"2024-04-21T08:48:50.759116+0000","last_unstale":"2024-04-21T08:51:10.832121+0000","last_undegraded":"2024-04-21T08:51:10.832121+0000","last_fullsized":"2024-04-21T08:51:10.832121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:02:16.200901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038642+0000","last_change":"2024-04-21T08:48:45.545773+0000","last_active":"2024-04-21T08:51:11.038642+0000","last_peered":"2024-04-21T08:51:11.038642+0000","last_clean":"2024-04-21T08:51:11.038642+0000","last_became_active":"2024-04-21T08:48:45.545447+0000","last_became_peered":"2024-04-21T08:48:45.545447+0000","last_unstale":"2024-04-21T08:51:11.038642+0000","last_undegraded":"2024-04-21T08:51:11.038642+0000","last_fullsized":"2024-04-21T08:51:11.038642+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:59:42.171602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877976+0000","last_change":"2024-04-21T08:49:36.964411+0000","last_active":"2024-04-21T08:51:10.877976+0000","last_peered":"2024-04-21T08:51:10.877976+0000","last_clean":"2024-04-21T08:51:10.877976+0000","last_became_active":"2024-04-21T08:49:36.964046+0000","last_became_peered":"2024-04-21T08:49:36.964046+0000","last_unstale":"2024-04-21T08:51:10.877976+0000","last_undegraded":"2024-04-21T08:51:10.877976+0000","last_fullsized":"2024-04-21T08:51:10.877976+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:29:37.223495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038538+0000","last_change":"2024-04-21T08:50:37.351641+0000","last_active":"2024-04-21T08:51:11.038538+0000","last_peered":"2024-04-21T08:51:11.038538+0000","last_clean":"2024-04-21T08:51:11.038538+0000","last_became_active":"2024-04-21T08:50:37.351130+0000","last_became_peered":"2024-04-21T08:50:37.351130+0000","last_unstale":"2024-04-21T08:51:11.038538+0000","last_undegraded":"2024-04-21T08:51:11.038538+0000","last_fullsized":"2024-04-21T08:51:11.038538+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:56:58.989691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006155+0000","last_change":"2024-04-21T08:49:36.928993+0000","last_active":"2024-04-21T08:51:11.006155+0000","last_peered":"2024-04-21T08:51:11.006155+0000","last_clean":"2024-04-21T08:51:11.006155+0000","last_became_active":"2024-04-21T08:49:36.911247+0000","last_became_peered":"2024-04-21T08:49:36.911247+0000","last_unstale":"2024-04-21T08:51:11.006155+0000","last_undegraded":"2024-04-21T08:51:11.006155+0000","last_fullsized":"2024-04-21T08:51:11.006155+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:13.481670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939224+0000","last_change":"2024-04-21T08:48:46.465475+0000","last_active":"2024-04-21T08:51:10.939224+0000","last_peered":"2024-04-21T08:51:10.939224+0000","last_clean":"2024-04-21T08:51:10.939224+0000","last_became_active":"2024-04-21T08:48:46.465138+0000","last_became_peered":"2024-04-21T08:48:46.465138+0000","last_unstale":"2024-04-21T08:51:10.939224+0000","last_undegraded":"2024-04-21T08:51:10.939224+0000","last_fullsized":"2024-04-21T08:51:10.939224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:12:53.521547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936707+0000","last_change":"2024-04-21T08:48:50.765765+0000","last_active":"2024-04-21T08:51:10.936707+0000","last_peered":"2024-04-21T08:51:10.936707+0000","last_clean":"2024-04-21T08:51:10.936707+0000","last_became_active":"2024-04-21T08:48:50.765324+0000","last_became_peered":"2024-04-21T08:48:50.765324+0000","last_unstale":"2024-04-21T08:51:10.936707+0000","last_undegraded":"2024-04-21T08:51:10.936707+0000","last_fullsized":"2024-04-21T08:51:10.936707+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:41:37.846768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828042+0000","last_change":"2024-04-21T08:48:45.557400+0000","last_active":"2024-04-21T08:51:10.828042+0000","last_peered":"2024-04-21T08:51:10.828042+0000","last_clean":"2024-04-21T08:51:10.828042+0000","last_became_active":"2024-04-21T08:48:45.555012+0000","last_became_peered":"2024-04-21T08:48:45.555012+0000","last_unstale":"2024-04-21T08:51:10.828042+0000","last_undegraded":"2024-04-21T08:51:10.828042+0000","last_fullsized":"2024-04-21T08:51:10.828042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:00:31.254353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062791+0000","last_change":"2024-04-21T08:48:46.470042+0000","last_active":"2024-04-21T08:51:11.062791+0000","last_peered":"2024-04-21T08:51:11.062791+0000","last_clean":"2024-04-21T08:51:11.062791+0000","last_became_active":"2024-04-21T08:48:46.469716+0000","last_became_peered":"2024-04-21T08:48:46.469716+0000","last_unstale":"2024-04-21T08:51:11.062791+0000","last_undegraded":"2024-04-21T08:51:11.062791+0000","last_fullsized":"2024-04-21T08:51:11.062791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:09.934751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038592+0000","last_change":"2024-04-21T08:48:50.770889+0000","last_active":"2024-04-21T08:51:11.038592+0000","last_peered":"2024-04-21T08:51:11.038592+0000","last_clean":"2024-04-21T08:51:11.038592+0000","last_became_active":"2024-04-21T08:48:50.769980+0000","last_became_peered":"2024-04-21T08:48:50.769980+0000","last_unstale":"2024-04-21T08:51:11.038592+0000","last_undegraded":"2024-04-21T08:51:11.038592+0000","last_fullsized":"2024-04-21T08:51:11.038592+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:44.631633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832532+0000","last_change":"2024-04-21T08:48:45.555479+0000","last_active":"2024-04-21T08:51:10.832532+0000","last_peered":"2024-04-21T08:51:10.832532+0000","last_clean":"2024-04-21T08:51:10.832532+0000","last_became_active":"2024-04-21T08:48:45.555162+0000","last_became_peered":"2024-04-21T08:48:45.555162+0000","last_unstale":"2024-04-21T08:51:10.832532+0000","last_undegraded":"2024-04-21T08:51:10.832532+0000","last_fullsized":"2024-04-21T08:51:10.832532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:15:20.262798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828417+0000","last_change":"2024-04-21T08:48:46.467962+0000","last_active":"2024-04-21T08:51:10.828417+0000","last_peered":"2024-04-21T08:51:10.828417+0000","last_clean":"2024-04-21T08:51:10.828417+0000","last_became_active":"2024-04-21T08:48:46.467561+0000","last_became_peered":"2024-04-21T08:48:46.467561+0000","last_unstale":"2024-04-21T08:51:10.828417+0000","last_undegraded":"2024-04-21T08:51:10.828417+0000","last_fullsized":"2024-04-21T08:51:10.828417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:18.262009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038487+0000","last_change":"2024-04-21T08:48:50.767678+0000","last_active":"2024-04-21T08:51:11.038487+0000","last_peered":"2024-04-21T08:51:11.038487+0000","last_clean":"2024-04-21T08:51:11.038487+0000","last_became_active":"2024-04-21T08:48:50.767309+0000","last_became_peered":"2024-04-21T08:48:50.767309+0000","last_unstale":"2024-04-21T08:51:11.038487+0000","last_undegraded":"2024-04-21T08:51:11.038487+0000","last_fullsized":"2024-04-21T08:51:11.038487+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:16:40.065043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939597+0000","last_change":"2024-04-21T08:48:45.553321+0000","last_active":"2024-04-21T08:51:10.939597+0000","last_peered":"2024-04-21T08:51:10.939597+0000","last_clean":"2024-04-21T08:51:10.939597+0000","last_became_active":"2024-04-21T08:48:45.552871+0000","last_became_peered":"2024-04-21T08:48:45.552871+0000","last_unstale":"2024-04-21T08:51:10.939597+0000","last_undegraded":"2024-04-21T08:51:10.939597+0000","last_fullsized":"2024-04-21T08:51:10.939597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:16:27.339801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062861+0000","last_change":"2024-04-21T08:48:46.473453+0000","last_active":"2024-04-21T08:51:11.062861+0000","last_peered":"2024-04-21T08:51:11.062861+0000","last_clean":"2024-04-21T08:51:11.062861+0000","last_became_active":"2024-04-21T08:48:46.472613+0000","last_became_peered":"2024-04-21T08:48:46.472613+0000","last_unstale":"2024-04-21T08:51:11.062861+0000","last_undegraded":"2024-04-21T08:51:11.062861+0000","last_fullsized":"2024-04-21T08:51:11.062861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:43:58.995997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006138+0000","last_change":"2024-04-21T08:48:50.765282+0000","last_active":"2024-04-21T08:51:11.006138+0000","last_peered":"2024-04-21T08:51:11.006138+0000","last_clean":"2024-04-21T08:51:11.006138+0000","last_became_active":"2024-04-21T08:48:50.764891+0000","last_became_peered":"2024-04-21T08:48:50.764891+0000","last_unstale":"2024-04-21T08:51:11.006138+0000","last_undegraded":"2024-04-21T08:51:11.006138+0000","last_fullsized":"2024-04-21T08:51:11.006138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:18:40.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038155+0000","last_change":"2024-04-21T08:49:36.926642+0000","last_active":"2024-04-21T08:51:11.038155+0000","last_peered":"2024-04-21T08:51:11.038155+0000","last_clean":"2024-04-21T08:51:11.038155+0000","last_became_active":"2024-04-21T08:49:36.910453+0000","last_became_peered":"2024-04-21T08:49:36.910453+0000","last_unstale":"2024-04-21T08:51:11.038155+0000","last_undegraded":"2024-04-21T08:51:11.038155+0000","last_fullsized":"2024-04-21T08:51:11.038155+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:19.290010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063219+0000","last_change":"2024-04-21T08:48:46.470058+0000","last_active":"2024-04-21T08:51:11.063219+0000","last_peered":"2024-04-21T08:51:11.063219+0000","last_clean":"2024-04-21T08:51:11.063219+0000","last_became_active":"2024-04-21T08:48:46.469485+0000","last_became_peered":"2024-04-21T08:48:46.469485+0000","last_unstale":"2024-04-21T08:51:11.063219+0000","last_undegraded":"2024-04-21T08:51:11.063219+0000","last_fullsized":"2024-04-21T08:51:11.063219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:33:13.212087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877446+0000","last_change":"2024-04-21T08:48:50.762944+0000","last_active":"2024-04-21T08:51:10.877446+0000","last_peered":"2024-04-21T08:51:10.877446+0000","last_clean":"2024-04-21T08:51:10.877446+0000","last_became_active":"2024-04-21T08:48:50.762508+0000","last_became_peered":"2024-04-21T08:48:50.762508+0000","last_unstale":"2024-04-21T08:51:10.877446+0000","last_undegraded":"2024-04-21T08:51:10.877446+0000","last_fullsized":"2024-04-21T08:51:10.877446+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:35.393339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'65","reported_seq":130,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.177489+0000","last_change":"2024-04-21T08:48:45.560361+0000","last_active":"2024-04-21T08:51:20.177489+0000","last_peered":"2024-04-21T08:51:20.177489+0000","last_clean":"2024-04-21T08:51:20.177489+0000","last_became_active":"2024-04-21T08:48:45.559946+0000","last_became_peered":"2024-04-21T08:48:45.559946+0000","last_unstale":"2024-04-21T08:51:20.177489+0000","last_undegraded":"2024-04-21T08:51:20.177489+0000","last_fullsized":"2024-04-21T08:51:20.177489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:55.599714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063003+0000","last_change":"2024-04-21T08:48:46.472706+0000","last_active":"2024-04-21T08:51:11.063003+0000","last_peered":"2024-04-21T08:51:11.063003+0000","last_clean":"2024-04-21T08:51:11.063003+0000","last_became_active":"2024-04-21T08:48:46.472378+0000","last_became_peered":"2024-04-21T08:48:46.472378+0000","last_unstale":"2024-04-21T08:51:11.063003+0000","last_undegraded":"2024-04-21T08:51:11.063003+0000","last_fullsized":"2024-04-21T08:51:11.063003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:58.228899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945191+0000","last_change":"2024-04-21T08:48:50.760610+0000","last_active":"2024-04-21T08:51:10.945191+0000","last_peered":"2024-04-21T08:51:10.945191+0000","last_clean":"2024-04-21T08:51:10.945191+0000","last_became_active":"2024-04-21T08:48:50.759991+0000","last_became_peered":"2024-04-21T08:48:50.759991+0000","last_unstale":"2024-04-21T08:51:10.945191+0000","last_undegraded":"2024-04-21T08:51:10.945191+0000","last_fullsized":"2024-04-21T08:51:10.945191+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:43.649701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036942+0000","last_change":"2024-04-21T08:48:45.560773+0000","last_active":"2024-04-21T08:51:11.036942+0000","last_peered":"2024-04-21T08:51:11.036942+0000","last_clean":"2024-04-21T08:51:11.036942+0000","last_became_active":"2024-04-21T08:48:45.560431+0000","last_became_peered":"2024-04-21T08:48:45.560431+0000","last_unstale":"2024-04-21T08:51:11.036942+0000","last_undegraded":"2024-04-21T08:51:11.036942+0000","last_fullsized":"2024-04-21T08:51:11.036942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:07.858838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063259+0000","last_change":"2024-04-21T08:48:46.475983+0000","last_active":"2024-04-21T08:51:11.063259+0000","last_peered":"2024-04-21T08:51:11.063259+0000","last_clean":"2024-04-21T08:51:11.063259+0000","last_became_active":"2024-04-21T08:48:46.475667+0000","last_became_peered":"2024-04-21T08:48:46.475667+0000","last_unstale":"2024-04-21T08:51:11.063259+0000","last_undegraded":"2024-04-21T08:51:11.063259+0000","last_fullsized":"2024-04-21T08:51:11.063259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:28:41.013111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062960+0000","last_change":"2024-04-21T08:48:46.464675+0000","last_active":"2024-04-21T08:51:11.062960+0000","last_peered":"2024-04-21T08:51:11.062960+0000","last_clean":"2024-04-21T08:51:11.062960+0000","last_became_active":"2024-04-21T08:48:46.464314+0000","last_became_peered":"2024-04-21T08:48:46.464314+0000","last_unstale":"2024-04-21T08:51:11.062960+0000","last_undegraded":"2024-04-21T08:51:11.062960+0000","last_fullsized":"2024-04-21T08:51:11.062960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:53.048400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832307+0000","last_change":"2024-04-21T08:49:36.911025+0000","last_active":"2024-04-21T08:51:10.832307+0000","last_peered":"2024-04-21T08:51:10.832307+0000","last_clean":"2024-04-21T08:51:10.832307+0000","last_became_active":"2024-04-21T08:49:36.909942+0000","last_became_peered":"2024-04-21T08:49:36.909942+0000","last_unstale":"2024-04-21T08:51:10.832307+0000","last_undegraded":"2024-04-21T08:51:10.832307+0000","last_fullsized":"2024-04-21T08:51:10.832307+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:39.864253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832375+0000","last_change":"2024-04-21T08:48:46.467867+0000","last_active":"2024-04-21T08:51:10.832375+0000","last_peered":"2024-04-21T08:51:10.832375+0000","last_clean":"2024-04-21T08:51:10.832375+0000","last_became_active":"2024-04-21T08:48:46.465917+0000","last_became_peered":"2024-04-21T08:48:46.465917+0000","last_unstale":"2024-04-21T08:51:10.832375+0000","last_undegraded":"2024-04-21T08:51:10.832375+0000","last_fullsized":"2024-04-21T08:51:10.832375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:19:56.824998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877556+0000","last_change":"2024-04-21T08:48:46.467159+0000","last_active":"2024-04-21T08:51:10.877556+0000","last_peered":"2024-04-21T08:51:10.877556+0000","last_clean":"2024-04-21T08:51:10.877556+0000","last_became_active":"2024-04-21T08:48:46.466639+0000","last_became_peered":"2024-04-21T08:48:46.466639+0000","last_unstale":"2024-04-21T08:51:10.877556+0000","last_undegraded":"2024-04-21T08:51:10.877556+0000","last_fullsized":"2024-04-21T08:51:10.877556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:54:44.772535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036982+0000","last_change":"2024-04-21T08:48:46.466403+0000","last_active":"2024-04-21T08:51:11.036982+0000","last_peered":"2024-04-21T08:51:11.036982+0000","last_clean":"2024-04-21T08:51:11.036982+0000","last_became_active":"2024-04-21T08:48:46.466060+0000","last_became_peered":"2024-04-21T08:48:46.466060+0000","last_unstale":"2024-04-21T08:51:11.036982+0000","last_undegraded":"2024-04-21T08:51:11.036982+0000","last_fullsized":"2024-04-21T08:51:11.036982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:14.374193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063096+0000","last_change":"2024-04-21T08:49:36.909134+0000","last_active":"2024-04-21T08:51:11.063096+0000","last_peered":"2024-04-21T08:51:11.063096+0000","last_clean":"2024-04-21T08:51:11.063096+0000","last_became_active":"2024-04-21T08:49:36.908723+0000","last_became_peered":"2024-04-21T08:49:36.908723+0000","last_unstale":"2024-04-21T08:51:11.063096+0000","last_undegraded":"2024-04-21T08:51:11.063096+0000","last_fullsized":"2024-04-21T08:51:11.063096+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:28.123016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832933+0000","last_change":"2024-04-21T08:48:46.464098+0000","last_active":"2024-04-21T08:51:10.832933+0000","last_peered":"2024-04-21T08:51:10.832933+0000","last_clean":"2024-04-21T08:51:10.832933+0000","last_became_active":"2024-04-21T08:48:46.463756+0000","last_became_peered":"2024-04-21T08:48:46.463756+0000","last_unstale":"2024-04-21T08:51:10.832933+0000","last_undegraded":"2024-04-21T08:51:10.832933+0000","last_fullsized":"2024-04-21T08:51:10.832933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:31.725155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828666+0000","last_change":"2024-04-21T08:48:46.467372+0000","last_active":"2024-04-21T08:51:10.828666+0000","last_peered":"2024-04-21T08:51:10.828666+0000","last_clean":"2024-04-21T08:51:10.828666+0000","last_became_active":"2024-04-21T08:48:46.467043+0000","last_became_peered":"2024-04-21T08:48:46.467043+0000","last_unstale":"2024-04-21T08:51:10.828666+0000","last_undegraded":"2024-04-21T08:51:10.828666+0000","last_fullsized":"2024-04-21T08:51:10.828666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:41.889994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945316+0000","last_change":"2024-04-21T08:48:46.460604+0000","last_active":"2024-04-21T08:51:10.945316+0000","last_peered":"2024-04-21T08:51:10.945316+0000","last_clean":"2024-04-21T08:51:10.945316+0000","last_became_active":"2024-04-21T08:48:46.460276+0000","last_became_peered":"2024-04-21T08:48:46.460276+0000","last_unstale":"2024-04-21T08:51:10.945316+0000","last_undegraded":"2024-04-21T08:51:10.945316+0000","last_fullsized":"2024-04-21T08:51:10.945316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:30.194808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828076+0000","last_change":"2024-04-21T08:48:46.463665+0000","last_active":"2024-04-21T08:51:10.828076+0000","last_peered":"2024-04-21T08:51:10.828076+0000","last_clean":"2024-04-21T08:51:10.828076+0000","last_became_active":"2024-04-21T08:48:46.463309+0000","last_became_peered":"2024-04-21T08:48:46.463309+0000","last_unstale":"2024-04-21T08:51:10.828076+0000","last_undegraded":"2024-04-21T08:51:10.828076+0000","last_fullsized":"2024-04-21T08:51:10.828076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:12:36.436565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006407+0000","last_change":"2024-04-21T08:48:46.476845+0000","last_active":"2024-04-21T08:51:11.006407+0000","last_peered":"2024-04-21T08:51:11.006407+0000","last_clean":"2024-04-21T08:51:11.006407+0000","last_became_active":"2024-04-21T08:48:46.476507+0000","last_became_peered":"2024-04-21T08:48:46.476507+0000","last_unstale":"2024-04-21T08:51:11.006407+0000","last_undegraded":"2024-04-21T08:51:11.006407+0000","last_fullsized":"2024-04-21T08:51:11.006407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:47.424576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036615+0000","last_change":"2024-04-21T08:48:46.469316+0000","last_active":"2024-04-21T08:51:11.036615+0000","last_peered":"2024-04-21T08:51:11.036615+0000","last_clean":"2024-04-21T08:51:11.036615+0000","last_became_active":"2024-04-21T08:48:46.468852+0000","last_became_peered":"2024-04-21T08:48:46.468852+0000","last_unstale":"2024-04-21T08:51:11.036615+0000","last_undegraded":"2024-04-21T08:51:11.036615+0000","last_fullsized":"2024-04-21T08:51:11.036615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:05:03.664279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062922+0000","last_change":"2024-04-21T08:48:46.466703+0000","last_active":"2024-04-21T08:51:11.062922+0000","last_peered":"2024-04-21T08:51:11.062922+0000","last_clean":"2024-04-21T08:51:11.062922+0000","last_became_active":"2024-04-21T08:48:46.466395+0000","last_became_peered":"2024-04-21T08:48:46.466395+0000","last_unstale":"2024-04-21T08:51:11.062922+0000","last_undegraded":"2024-04-21T08:51:11.062922+0000","last_fullsized":"2024-04-21T08:51:11.062922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:08:56.956283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108027,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":799382,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1154,"num_read_kb":796,"num_write":425,"num_write_kb":2365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":2398146,"data_compressed":108306,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":529689,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645361,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32384,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712768,"statfs":{"total":95995035648,"available":95961874432,"internally_reserved":0,"allocated":1228800,"data_stored":1518311,"data_compressed":82401,"data_compressed_allocated":110592,"data_compressed_original":987136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":927125,"data_compressed":48868,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070524,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":922837,"data_compressed":48868,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1097728,"data_stored":1127655,"data_compressed":65588,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495686,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1028096,"data_stored":524989,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":596819,"data_compressed":31190,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":526377,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1122304,"data_stored":598677,"data_compressed":31190,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":1246631,"data_compressed":64070,"data_compressed_allocated":86016,"data_compressed_original":782336,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":531535,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1150976,"data_stored":1450161,"data_compressed":80790,"data_compressed_allocated":106496,"data_compressed_original":978944,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":71920,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":73778,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330798,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":393842,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":403044,"data_compressed":16813,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:51:24.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:51:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.237+0000 7fdb163f4640 1 -- 172.21.15.28:0/2203261590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb10105e30 msgr2=0x7fdb10106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.237+0000 7fdb163f4640 1 --2- 172.21.15.28:0/2203261590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb10105e30 0x7fdb10106210 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fdb00009930 tx=0x7fdb0002bc20 comp rx=0 tx=0).stop 2024-04-21T08:51:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.237+0000 7fdb163f4640 1 -- 172.21.15.28:0/2203261590 shutdown_connections 2024-04-21T08:51:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.237+0000 7fdb163f4640 1 --2- 172.21.15.28:0/2203261590 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdb10101b60 0x7fdb1010dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.237+0000 7fdb163f4640 1 --2- 172.21.15.28:0/2203261590 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdb101011a0 0x7fdb10101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.237+0000 7fdb163f4640 1 --2- 172.21.15.28:0/2203261590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb10105e30 0x7fdb10106210 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.237+0000 7fdb163f4640 1 -- 172.21.15.28:0/2203261590 >> 172.21.15.28:0/2203261590 conn(0x7fdb10077e30 msgr2=0x7fdb10078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.237+0000 7fdb163f4640 1 -- 172.21.15.28:0/2203261590 shutdown_connections 2024-04-21T08:51:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.237+0000 7fdb163f4640 1 -- 172.21.15.28:0/2203261590 wait complete. 2024-04-21T08:51:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.237+0000 7fdb163f4640 1 Processor -- start 2024-04-21T08:51:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.237+0000 7fdb163f4640 1 -- start start 2024-04-21T08:51:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb163f4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdb101011a0 0x7fdb10198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb163f4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdb10101b60 0x7fdb10198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb163f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb10105e30 0x7fdb1019cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb163f4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb10112240 con 0x7fdb10105e30 2024-04-21T08:51:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb163f4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fdb101120c0 con 0x7fdb10101b60 2024-04-21T08:51:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb163f4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdb101123c0 con 0x7fdb101011a0 2024-04-21T08:51:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb14991640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb10105e30 0x7fdb1019cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb14991640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb10105e30 0x7fdb1019cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44410/0 (socket says 172.21.15.28:44410) 2024-04-21T08:51:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb14991640 1 -- 172.21.15.28:0/2733684567 learned_addr learned my addr 172.21.15.28:0/2733684567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb0ffff640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdb101011a0 0x7fdb10198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb0f7fe640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdb10101b60 0x7fdb10198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb14991640 1 -- 172.21.15.28:0/2733684567 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdb101011a0 msgr2=0x7fdb10198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb14991640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdb101011a0 0x7fdb10198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb14991640 1 -- 172.21.15.28:0/2733684567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdb10101b60 msgr2=0x7fdb10198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb14991640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdb10101b60 0x7fdb10198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb14991640 1 -- 172.21.15.28:0/2733684567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb1019d710 con 0x7fdb10105e30 2024-04-21T08:51:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb0f7fe640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdb10101b60 0x7fdb10198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.238+0000 7fdb0ffff640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdb101011a0 0x7fdb10198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.239+0000 7fdb14991640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb10105e30 0x7fdb1019cfe0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fdb0800eaa0 tx=0x7fdb0800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.239+0000 7fdb0d7fa640 1 -- 172.21.15.28:0/2733684567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb0800cdb0 con 0x7fdb10105e30 2024-04-21T08:51:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.239+0000 7fdb0d7fa640 1 -- 172.21.15.28:0/2733684567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb08005020 con 0x7fdb10105e30 2024-04-21T08:51:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.239+0000 7fdb163f4640 1 -- 172.21.15.28:0/2733684567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb1019da00 con 0x7fdb10105e30 2024-04-21T08:51:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.239+0000 7fdb0d7fa640 1 -- 172.21.15.28:0/2733684567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb080116d0 con 0x7fdb10105e30 2024-04-21T08:51:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.239+0000 7fdb163f4640 1 -- 172.21.15.28:0/2733684567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb101a5290 con 0x7fdb10105e30 2024-04-21T08:51:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.240+0000 7fdb163f4640 1 -- 172.21.15.28:0/2733684567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb10102570 con 0x7fdb10105e30 2024-04-21T08:51:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.241+0000 7fdb0d7fa640 1 -- 172.21.15.28:0/2733684567 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fdb080051d0 con 0x7fdb10105e30 2024-04-21T08:51:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.241+0000 7fdb0d7fa640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdaf80a3e40 0x7fdaf80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.242+0000 7fdb0ffff640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdaf80a3e40 0x7fdaf80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.242+0000 7fdb0d7fa640 1 -- 172.21.15.28:0/2733684567 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fdb080d1020 con 0x7fdb10105e30 2024-04-21T08:51:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.242+0000 7fdb0ffff640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdaf80a3e40 0x7fdaf80a6300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdb00006010 tx=0x7fdb00009610 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.246+0000 7fdb0d7fa640 1 -- 172.21.15.28:0/2733684567 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb08016030 con 0x7fdb10105e30 2024-04-21T08:51:24.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:23 smithi152 ceph-mon[31401]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-04-21T08:51:24.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:23 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1802068257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:24.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:23 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3562503030' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:24.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:23 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/53726043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:24.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:23 smithi171 ceph-mon[31358]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-04-21T08:51:24.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:23 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1802068257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:24.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:23 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3562503030' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:24.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:23 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/53726043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:24.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.434+0000 7fdb163f4640 1 -- 172.21.15.28:0/2733684567 --> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdb10062390 con 0x7fdaf80a3e40 2024-04-21T08:51:24.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.443+0000 7fdb0d7fa640 1 -- 172.21.15.28:0/2733684567 <== mgr.25040 v2:172.21.15.28:6800/4182301910 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339719 (secure 0 0 0) 0x7fdb10062390 con 0x7fdaf80a3e40 2024-04-21T08:51:24.446 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:24.448 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-21T08:51:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.447+0000 7fdb163f4640 1 -- 172.21.15.28:0/2733684567 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdaf80a3e40 msgr2=0x7fdaf80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.447+0000 7fdb163f4640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdaf80a3e40 0x7fdaf80a6300 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdb00006010 tx=0x7fdb00009610 comp rx=0 tx=0).stop 2024-04-21T08:51:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.447+0000 7fdb163f4640 1 -- 172.21.15.28:0/2733684567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb10105e30 msgr2=0x7fdb1019cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.447+0000 7fdb163f4640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb10105e30 0x7fdb1019cfe0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fdb0800eaa0 tx=0x7fdb0800ef60 comp rx=0 tx=0).stop 2024-04-21T08:51:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.447+0000 7fdb163f4640 1 -- 172.21.15.28:0/2733684567 shutdown_connections 2024-04-21T08:51:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.447+0000 7fdb163f4640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdaf80a3e40 0x7fdaf80a6300 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.447+0000 7fdb163f4640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb10105e30 0x7fdb1019cfe0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.447+0000 7fdb163f4640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdb10101b60 0x7fdb10198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.447+0000 7fdb163f4640 1 --2- 172.21.15.28:0/2733684567 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdb101011a0 0x7fdb10198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.447+0000 7fdb163f4640 1 -- 172.21.15.28:0/2733684567 >> 172.21.15.28:0/2733684567 conn(0x7fdb10077e30 msgr2=0x7fdb100fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.447+0000 7fdb163f4640 1 -- 172.21.15.28:0/2733684567 shutdown_connections 2024-04-21T08:51:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.447+0000 7fdb163f4640 1 -- 172.21.15.28:0/2733684567 wait complete. 2024-04-21T08:51:24.457 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-21T08:51:22.844483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1389984,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1391,"num_read_kb":1126,"num_write":628,"num_write_kb":5305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181430,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":965,"ondisk_log_size":965,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370328,"kb_used_data":12812,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571496,"statfs":{"total":1151940427776,"available":1151561211904,"internally_reserved":0,"allocated":13119488,"data_stored":10500806,"data_compressed":571281,"data_compressed_allocated":774144,"data_compressed_original":4878336,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":199934,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":420,"num_read_kb":484,"num_write":155,"num_write_kb":922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.992496"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063213+0000","last_change":"2024-04-21T08:48:46.471626+0000","last_active":"2024-04-21T08:51:11.063213+0000","last_peered":"2024-04-21T08:51:11.063213+0000","last_clean":"2024-04-21T08:51:11.063213+0000","last_became_active":"2024-04-21T08:48:46.470498+0000","last_became_peered":"2024-04-21T08:48:46.470498+0000","last_unstale":"2024-04-21T08:51:11.063213+0000","last_undegraded":"2024-04-21T08:51:11.063213+0000","last_fullsized":"2024-04-21T08:51:11.063213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:49.452181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877897+0000","last_change":"2024-04-21T08:48:46.469111+0000","last_active":"2024-04-21T08:51:10.877897+0000","last_peered":"2024-04-21T08:51:10.877897+0000","last_clean":"2024-04-21T08:51:10.877897+0000","last_became_active":"2024-04-21T08:48:46.467926+0000","last_became_peered":"2024-04-21T08:48:46.467926+0000","last_unstale":"2024-04-21T08:51:10.877897+0000","last_undegraded":"2024-04-21T08:51:10.877897+0000","last_fullsized":"2024-04-21T08:51:10.877897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:35:38.230212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877755+0000","last_change":"2024-04-21T08:48:46.475673+0000","last_active":"2024-04-21T08:51:10.877755+0000","last_peered":"2024-04-21T08:51:10.877755+0000","last_clean":"2024-04-21T08:51:10.877755+0000","last_became_active":"2024-04-21T08:48:46.474917+0000","last_became_peered":"2024-04-21T08:48:46.474917+0000","last_unstale":"2024-04-21T08:51:10.877755+0000","last_undegraded":"2024-04-21T08:51:10.877755+0000","last_fullsized":"2024-04-21T08:51:10.877755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:01.661704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038205+0000","last_change":"2024-04-21T08:48:46.468147+0000","last_active":"2024-04-21T08:51:11.038205+0000","last_peered":"2024-04-21T08:51:11.038205+0000","last_clean":"2024-04-21T08:51:11.038205+0000","last_became_active":"2024-04-21T08:48:46.467811+0000","last_became_peered":"2024-04-21T08:48:46.467811+0000","last_unstale":"2024-04-21T08:51:11.038205+0000","last_undegraded":"2024-04-21T08:51:11.038205+0000","last_fullsized":"2024-04-21T08:51:11.038205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:21.658605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939327+0000","last_change":"2024-04-21T08:48:46.471322+0000","last_active":"2024-04-21T08:51:10.939327+0000","last_peered":"2024-04-21T08:51:10.939327+0000","last_clean":"2024-04-21T08:51:10.939327+0000","last_became_active":"2024-04-21T08:48:46.470991+0000","last_became_peered":"2024-04-21T08:48:46.470991+0000","last_unstale":"2024-04-21T08:51:10.939327+0000","last_undegraded":"2024-04-21T08:51:10.939327+0000","last_fullsized":"2024-04-21T08:51:10.939327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:22.839870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829853+0000","last_change":"2024-04-21T08:48:46.467039+0000","last_active":"2024-04-21T08:51:10.829853+0000","last_peered":"2024-04-21T08:51:10.829853+0000","last_clean":"2024-04-21T08:51:10.829853+0000","last_became_active":"2024-04-21T08:48:46.466614+0000","last_became_peered":"2024-04-21T08:48:46.466614+0000","last_unstale":"2024-04-21T08:51:10.829853+0000","last_undegraded":"2024-04-21T08:51:10.829853+0000","last_fullsized":"2024-04-21T08:51:10.829853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:38.029215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062769+0000","last_change":"2024-04-21T08:48:46.473268+0000","last_active":"2024-04-21T08:51:11.062769+0000","last_peered":"2024-04-21T08:51:11.062769+0000","last_clean":"2024-04-21T08:51:11.062769+0000","last_became_active":"2024-04-21T08:48:46.472261+0000","last_became_peered":"2024-04-21T08:48:46.472261+0000","last_unstale":"2024-04-21T08:51:11.062769+0000","last_undegraded":"2024-04-21T08:51:11.062769+0000","last_fullsized":"2024-04-21T08:51:11.062769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:58:52.990687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828243+0000","last_change":"2024-04-21T08:48:46.472788+0000","last_active":"2024-04-21T08:51:10.828243+0000","last_peered":"2024-04-21T08:51:10.828243+0000","last_clean":"2024-04-21T08:51:10.828243+0000","last_became_active":"2024-04-21T08:48:46.472482+0000","last_became_peered":"2024-04-21T08:48:46.472482+0000","last_unstale":"2024-04-21T08:51:10.828243+0000","last_undegraded":"2024-04-21T08:51:10.828243+0000","last_fullsized":"2024-04-21T08:51:10.828243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:48.483846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006997+0000","last_change":"2024-04-21T08:48:46.460771+0000","last_active":"2024-04-21T08:51:11.006997+0000","last_peered":"2024-04-21T08:51:11.006997+0000","last_clean":"2024-04-21T08:51:11.006997+0000","last_became_active":"2024-04-21T08:48:46.460472+0000","last_became_peered":"2024-04-21T08:48:46.460472+0000","last_unstale":"2024-04-21T08:51:11.006997+0000","last_undegraded":"2024-04-21T08:51:11.006997+0000","last_fullsized":"2024-04-21T08:51:11.006997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:46:40.766245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.830308+0000","last_change":"2024-04-21T08:48:46.463475+0000","last_active":"2024-04-21T08:51:10.830308+0000","last_peered":"2024-04-21T08:51:10.830308+0000","last_clean":"2024-04-21T08:51:10.830308+0000","last_became_active":"2024-04-21T08:48:46.462559+0000","last_became_peered":"2024-04-21T08:48:46.462559+0000","last_unstale":"2024-04-21T08:51:10.830308+0000","last_undegraded":"2024-04-21T08:51:10.830308+0000","last_fullsized":"2024-04-21T08:51:10.830308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:22:06.372410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063168+0000","last_change":"2024-04-21T08:48:46.476997+0000","last_active":"2024-04-21T08:51:11.063168+0000","last_peered":"2024-04-21T08:51:11.063168+0000","last_clean":"2024-04-21T08:51:11.063168+0000","last_became_active":"2024-04-21T08:48:46.476620+0000","last_became_peered":"2024-04-21T08:48:46.476620+0000","last_unstale":"2024-04-21T08:51:11.063168+0000","last_undegraded":"2024-04-21T08:51:11.063168+0000","last_fullsized":"2024-04-21T08:51:11.063168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:37.569629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038421+0000","last_change":"2024-04-21T08:48:46.469355+0000","last_active":"2024-04-21T08:51:11.038421+0000","last_peered":"2024-04-21T08:51:11.038421+0000","last_clean":"2024-04-21T08:51:11.038421+0000","last_became_active":"2024-04-21T08:48:46.468604+0000","last_became_peered":"2024-04-21T08:48:46.468604+0000","last_unstale":"2024-04-21T08:51:11.038421+0000","last_undegraded":"2024-04-21T08:51:11.038421+0000","last_fullsized":"2024-04-21T08:51:11.038421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:23:54.500982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036699+0000","last_change":"2024-04-21T08:48:46.453116+0000","last_active":"2024-04-21T08:51:11.036699+0000","last_peered":"2024-04-21T08:51:11.036699+0000","last_clean":"2024-04-21T08:51:11.036699+0000","last_became_active":"2024-04-21T08:48:46.452607+0000","last_became_peered":"2024-04-21T08:48:46.452607+0000","last_unstale":"2024-04-21T08:51:11.036699+0000","last_undegraded":"2024-04-21T08:51:11.036699+0000","last_fullsized":"2024-04-21T08:51:11.036699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:57:04.371812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877416+0000","last_change":"2024-04-21T08:48:46.474516+0000","last_active":"2024-04-21T08:51:10.877416+0000","last_peered":"2024-04-21T08:51:10.877416+0000","last_clean":"2024-04-21T08:51:10.877416+0000","last_became_active":"2024-04-21T08:48:46.474129+0000","last_became_peered":"2024-04-21T08:48:46.474129+0000","last_unstale":"2024-04-21T08:51:10.877416+0000","last_undegraded":"2024-04-21T08:51:10.877416+0000","last_fullsized":"2024-04-21T08:51:10.877416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:40:47.784976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877479+0000","last_change":"2024-04-21T08:49:36.907964+0000","last_active":"2024-04-21T08:51:10.877479+0000","last_peered":"2024-04-21T08:51:10.877479+0000","last_clean":"2024-04-21T08:51:10.877479+0000","last_became_active":"2024-04-21T08:49:36.907595+0000","last_became_peered":"2024-04-21T08:49:36.907595+0000","last_unstale":"2024-04-21T08:51:10.877479+0000","last_undegraded":"2024-04-21T08:51:10.877479+0000","last_fullsized":"2024-04-21T08:51:10.877479+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:14:04.907328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006192+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:51:11.006192+0000","last_peered":"2024-04-21T08:51:11.006192+0000","last_clean":"2024-04-21T08:51:11.006192+0000","last_became_active":"2024-04-21T08:48:46.469389+0000","last_became_peered":"2024-04-21T08:48:46.469389+0000","last_unstale":"2024-04-21T08:51:11.006192+0000","last_undegraded":"2024-04-21T08:51:11.006192+0000","last_fullsized":"2024-04-21T08:51:11.006192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:11:45.824402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063161+0000","last_change":"2024-04-21T08:48:46.470059+0000","last_active":"2024-04-21T08:51:11.063161+0000","last_peered":"2024-04-21T08:51:11.063161+0000","last_clean":"2024-04-21T08:51:11.063161+0000","last_became_active":"2024-04-21T08:48:46.469751+0000","last_became_peered":"2024-04-21T08:48:46.469751+0000","last_unstale":"2024-04-21T08:51:11.063161+0000","last_undegraded":"2024-04-21T08:51:11.063161+0000","last_fullsized":"2024-04-21T08:51:11.063161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:16.317879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.937061+0000","last_change":"2024-04-21T08:48:46.470129+0000","last_active":"2024-04-21T08:51:10.937061+0000","last_peered":"2024-04-21T08:51:10.937061+0000","last_clean":"2024-04-21T08:51:10.937061+0000","last_became_active":"2024-04-21T08:48:46.469763+0000","last_became_peered":"2024-04-21T08:48:46.469763+0000","last_unstale":"2024-04-21T08:51:10.937061+0000","last_undegraded":"2024-04-21T08:51:10.937061+0000","last_fullsized":"2024-04-21T08:51:10.937061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:47.731201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832982+0000","last_change":"2024-04-21T08:48:46.465356+0000","last_active":"2024-04-21T08:51:10.832982+0000","last_peered":"2024-04-21T08:51:10.832982+0000","last_clean":"2024-04-21T08:51:10.832982+0000","last_became_active":"2024-04-21T08:48:46.464183+0000","last_became_peered":"2024-04-21T08:48:46.464183+0000","last_unstale":"2024-04-21T08:51:10.832982+0000","last_undegraded":"2024-04-21T08:51:10.832982+0000","last_fullsized":"2024-04-21T08:51:10.832982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:37.533733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832383+0000","last_change":"2024-04-21T08:48:50.767378+0000","last_active":"2024-04-21T08:51:10.832383+0000","last_peered":"2024-04-21T08:51:10.832383+0000","last_clean":"2024-04-21T08:51:10.832383+0000","last_became_active":"2024-04-21T08:48:50.766856+0000","last_became_peered":"2024-04-21T08:48:50.766856+0000","last_unstale":"2024-04-21T08:51:10.832383+0000","last_undegraded":"2024-04-21T08:51:10.832383+0000","last_fullsized":"2024-04-21T08:51:10.832383+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:00.434540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936767+0000","last_change":"2024-04-21T08:48:45.550284+0000","last_active":"2024-04-21T08:51:10.936767+0000","last_peered":"2024-04-21T08:51:10.936767+0000","last_clean":"2024-04-21T08:51:10.936767+0000","last_became_active":"2024-04-21T08:48:45.549940+0000","last_became_peered":"2024-04-21T08:48:45.549940+0000","last_unstale":"2024-04-21T08:51:10.936767+0000","last_undegraded":"2024-04-21T08:51:10.936767+0000","last_fullsized":"2024-04-21T08:51:10.936767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:12.343299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828471+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:51:10.828471+0000","last_peered":"2024-04-21T08:51:10.828471+0000","last_clean":"2024-04-21T08:51:10.828471+0000","last_became_active":"2024-04-21T08:48:46.469647+0000","last_became_peered":"2024-04-21T08:48:46.469647+0000","last_unstale":"2024-04-21T08:51:10.828471+0000","last_undegraded":"2024-04-21T08:51:10.828471+0000","last_fullsized":"2024-04-21T08:51:10.828471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:10:28.204554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.037993+0000","last_change":"2024-04-21T08:48:50.771871+0000","last_active":"2024-04-21T08:51:11.037993+0000","last_peered":"2024-04-21T08:51:11.037993+0000","last_clean":"2024-04-21T08:51:11.037993+0000","last_became_active":"2024-04-21T08:48:50.770442+0000","last_became_peered":"2024-04-21T08:48:50.770442+0000","last_unstale":"2024-04-21T08:51:11.037993+0000","last_undegraded":"2024-04-21T08:51:11.037993+0000","last_fullsized":"2024-04-21T08:51:11.037993+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:18:57.813971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828497+0000","last_change":"2024-04-21T08:48:45.566492+0000","last_active":"2024-04-21T08:51:10.828497+0000","last_peered":"2024-04-21T08:51:10.828497+0000","last_clean":"2024-04-21T08:51:10.828497+0000","last_became_active":"2024-04-21T08:48:45.566169+0000","last_became_peered":"2024-04-21T08:48:45.566169+0000","last_unstale":"2024-04-21T08:51:10.828497+0000","last_undegraded":"2024-04-21T08:51:10.828497+0000","last_fullsized":"2024-04-21T08:51:10.828497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:49.985664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877476+0000","last_change":"2024-04-21T08:48:46.470866+0000","last_active":"2024-04-21T08:51:10.877476+0000","last_peered":"2024-04-21T08:51:10.877476+0000","last_clean":"2024-04-21T08:51:10.877476+0000","last_became_active":"2024-04-21T08:48:46.470401+0000","last_became_peered":"2024-04-21T08:48:46.470401+0000","last_unstale":"2024-04-21T08:51:10.877476+0000","last_undegraded":"2024-04-21T08:51:10.877476+0000","last_fullsized":"2024-04-21T08:51:10.877476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:07.506212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038124+0000","last_change":"2024-04-21T08:48:50.772339+0000","last_active":"2024-04-21T08:51:11.038124+0000","last_peered":"2024-04-21T08:51:11.038124+0000","last_clean":"2024-04-21T08:51:11.038124+0000","last_became_active":"2024-04-21T08:48:50.771946+0000","last_became_peered":"2024-04-21T08:48:50.771946+0000","last_unstale":"2024-04-21T08:51:11.038124+0000","last_undegraded":"2024-04-21T08:51:11.038124+0000","last_fullsized":"2024-04-21T08:51:11.038124+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:03:56.994581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939819+0000","last_change":"2024-04-21T08:48:45.553946+0000","last_active":"2024-04-21T08:51:10.939819+0000","last_peered":"2024-04-21T08:51:10.939819+0000","last_clean":"2024-04-21T08:51:10.939819+0000","last_became_active":"2024-04-21T08:48:45.553410+0000","last_became_peered":"2024-04-21T08:48:45.553410+0000","last_unstale":"2024-04-21T08:51:10.939819+0000","last_undegraded":"2024-04-21T08:51:10.939819+0000","last_fullsized":"2024-04-21T08:51:10.939819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:24:02.387250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006432+0000","last_change":"2024-04-21T08:48:46.471815+0000","last_active":"2024-04-21T08:51:11.006432+0000","last_peered":"2024-04-21T08:51:11.006432+0000","last_clean":"2024-04-21T08:51:11.006432+0000","last_became_active":"2024-04-21T08:48:46.471492+0000","last_became_peered":"2024-04-21T08:48:46.471492+0000","last_unstale":"2024-04-21T08:51:11.006432+0000","last_undegraded":"2024-04-21T08:51:11.006432+0000","last_fullsized":"2024-04-21T08:51:11.006432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:26.051166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038027+0000","last_change":"2024-04-21T08:48:50.767916+0000","last_active":"2024-04-21T08:51:11.038027+0000","last_peered":"2024-04-21T08:51:11.038027+0000","last_clean":"2024-04-21T08:51:11.038027+0000","last_became_active":"2024-04-21T08:48:50.767326+0000","last_became_peered":"2024-04-21T08:48:50.767326+0000","last_unstale":"2024-04-21T08:51:11.038027+0000","last_undegraded":"2024-04-21T08:51:11.038027+0000","last_fullsized":"2024-04-21T08:51:11.038027+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:05:02.185369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939335+0000","last_change":"2024-04-21T08:48:45.549982+0000","last_active":"2024-04-21T08:51:10.939335+0000","last_peered":"2024-04-21T08:51:10.939335+0000","last_clean":"2024-04-21T08:51:10.939335+0000","last_became_active":"2024-04-21T08:48:45.549611+0000","last_became_peered":"2024-04-21T08:48:45.549611+0000","last_unstale":"2024-04-21T08:51:10.939335+0000","last_undegraded":"2024-04-21T08:51:10.939335+0000","last_fullsized":"2024-04-21T08:51:10.939335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:46.035388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945903+0000","last_change":"2024-04-21T08:48:46.459068+0000","last_active":"2024-04-21T08:51:10.945903+0000","last_peered":"2024-04-21T08:51:10.945903+0000","last_clean":"2024-04-21T08:51:10.945903+0000","last_became_active":"2024-04-21T08:48:46.458686+0000","last_became_peered":"2024-04-21T08:48:46.458686+0000","last_unstale":"2024-04-21T08:51:10.945903+0000","last_undegraded":"2024-04-21T08:51:10.945903+0000","last_fullsized":"2024-04-21T08:51:10.945903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:31.373451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832831+0000","last_change":"2024-04-21T08:48:50.761392+0000","last_active":"2024-04-21T08:51:10.832831+0000","last_peered":"2024-04-21T08:51:10.832831+0000","last_clean":"2024-04-21T08:51:10.832831+0000","last_became_active":"2024-04-21T08:48:50.760917+0000","last_became_peered":"2024-04-21T08:48:50.760917+0000","last_unstale":"2024-04-21T08:51:10.832831+0000","last_undegraded":"2024-04-21T08:51:10.832831+0000","last_fullsized":"2024-04-21T08:51:10.832831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:19.070502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'8","reported_seq":175,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.027752+0000","last_change":"2024-04-21T08:48:45.553686+0000","last_active":"2024-04-21T08:51:20.027752+0000","last_peered":"2024-04-21T08:51:20.027752+0000","last_clean":"2024-04-21T08:51:20.027752+0000","last_became_active":"2024-04-21T08:48:45.553322+0000","last_became_peered":"2024-04-21T08:48:45.553322+0000","last_unstale":"2024-04-21T08:51:20.027752+0000","last_undegraded":"2024-04-21T08:51:20.027752+0000","last_fullsized":"2024-04-21T08:51:20.027752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:15:24.320704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1712,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":133,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939678+0000","last_change":"2024-04-21T08:48:46.474555+0000","last_active":"2024-04-21T08:51:10.939678+0000","last_peered":"2024-04-21T08:51:10.939678+0000","last_clean":"2024-04-21T08:51:10.939678+0000","last_became_active":"2024-04-21T08:48:46.474185+0000","last_became_peered":"2024-04-21T08:48:46.474185+0000","last_unstale":"2024-04-21T08:51:10.939678+0000","last_undegraded":"2024-04-21T08:51:10.939678+0000","last_fullsized":"2024-04-21T08:51:10.939678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:52.884994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832578+0000","last_change":"2024-04-21T08:48:50.763181+0000","last_active":"2024-04-21T08:51:10.832578+0000","last_peered":"2024-04-21T08:51:10.832578+0000","last_clean":"2024-04-21T08:51:10.832578+0000","last_became_active":"2024-04-21T08:48:50.762718+0000","last_became_peered":"2024-04-21T08:48:50.762718+0000","last_unstale":"2024-04-21T08:51:10.832578+0000","last_undegraded":"2024-04-21T08:51:10.832578+0000","last_fullsized":"2024-04-21T08:51:10.832578+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:16:23.204605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829837+0000","last_change":"2024-04-21T08:48:45.547768+0000","last_active":"2024-04-21T08:51:10.829837+0000","last_peered":"2024-04-21T08:51:10.829837+0000","last_clean":"2024-04-21T08:51:10.829837+0000","last_became_active":"2024-04-21T08:48:45.547456+0000","last_became_peered":"2024-04-21T08:48:45.547456+0000","last_unstale":"2024-04-21T08:51:10.829837+0000","last_undegraded":"2024-04-21T08:51:10.829837+0000","last_fullsized":"2024-04-21T08:51:10.829837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:23.502799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945877+0000","last_change":"2024-04-21T08:48:46.467988+0000","last_active":"2024-04-21T08:51:10.945877+0000","last_peered":"2024-04-21T08:51:10.945877+0000","last_clean":"2024-04-21T08:51:10.945877+0000","last_became_active":"2024-04-21T08:48:46.465212+0000","last_became_peered":"2024-04-21T08:48:46.465212+0000","last_unstale":"2024-04-21T08:51:10.945877+0000","last_undegraded":"2024-04-21T08:51:10.945877+0000","last_fullsized":"2024-04-21T08:51:10.945877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:41.743506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063079+0000","last_change":"2024-04-21T08:50:37.350070+0000","last_active":"2024-04-21T08:51:11.063079+0000","last_peered":"2024-04-21T08:51:11.063079+0000","last_clean":"2024-04-21T08:51:11.063079+0000","last_became_active":"2024-04-21T08:50:37.349622+0000","last_became_peered":"2024-04-21T08:50:37.349622+0000","last_unstale":"2024-04-21T08:51:11.063079+0000","last_undegraded":"2024-04-21T08:51:11.063079+0000","last_fullsized":"2024-04-21T08:51:11.063079+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:55.050244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.035911+0000","last_change":"2024-04-21T08:48:45.545615+0000","last_active":"2024-04-21T08:51:11.035911+0000","last_peered":"2024-04-21T08:51:11.035911+0000","last_clean":"2024-04-21T08:51:11.035911+0000","last_became_active":"2024-04-21T08:48:45.545280+0000","last_became_peered":"2024-04-21T08:48:45.545280+0000","last_unstale":"2024-04-21T08:51:11.035911+0000","last_undegraded":"2024-04-21T08:51:11.035911+0000","last_fullsized":"2024-04-21T08:51:11.035911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:50:00.470903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828078+0000","last_change":"2024-04-21T08:48:46.477326+0000","last_active":"2024-04-21T08:51:10.828078+0000","last_peered":"2024-04-21T08:51:10.828078+0000","last_clean":"2024-04-21T08:51:10.828078+0000","last_became_active":"2024-04-21T08:48:46.476914+0000","last_became_peered":"2024-04-21T08:48:46.476914+0000","last_unstale":"2024-04-21T08:51:10.828078+0000","last_undegraded":"2024-04-21T08:51:10.828078+0000","last_fullsized":"2024-04-21T08:51:10.828078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:04.312061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877774+0000","last_change":"2024-04-21T08:48:50.767429+0000","last_active":"2024-04-21T08:51:10.877774+0000","last_peered":"2024-04-21T08:51:10.877774+0000","last_clean":"2024-04-21T08:51:10.877774+0000","last_became_active":"2024-04-21T08:48:50.767008+0000","last_became_peered":"2024-04-21T08:48:50.767008+0000","last_unstale":"2024-04-21T08:51:10.877774+0000","last_undegraded":"2024-04-21T08:51:10.877774+0000","last_fullsized":"2024-04-21T08:51:10.877774+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:26.865571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945328+0000","last_change":"2024-04-21T08:48:45.553170+0000","last_active":"2024-04-21T08:51:10.945328+0000","last_peered":"2024-04-21T08:51:10.945328+0000","last_clean":"2024-04-21T08:51:10.945328+0000","last_became_active":"2024-04-21T08:48:45.552645+0000","last_became_peered":"2024-04-21T08:48:45.552645+0000","last_unstale":"2024-04-21T08:51:10.945328+0000","last_undegraded":"2024-04-21T08:51:10.945328+0000","last_fullsized":"2024-04-21T08:51:10.945328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:32:07.217212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936256+0000","last_change":"2024-04-21T08:48:46.478101+0000","last_active":"2024-04-21T08:51:10.936256+0000","last_peered":"2024-04-21T08:51:10.936256+0000","last_clean":"2024-04-21T08:51:10.936256+0000","last_became_active":"2024-04-21T08:48:46.477725+0000","last_became_peered":"2024-04-21T08:48:46.477725+0000","last_unstale":"2024-04-21T08:51:10.936256+0000","last_undegraded":"2024-04-21T08:51:10.936256+0000","last_fullsized":"2024-04-21T08:51:10.936256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:37:55.429569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828279+0000","last_change":"2024-04-21T08:48:50.762899+0000","last_active":"2024-04-21T08:51:10.828279+0000","last_peered":"2024-04-21T08:51:10.828279+0000","last_clean":"2024-04-21T08:51:10.828279+0000","last_became_active":"2024-04-21T08:48:50.762475+0000","last_became_peered":"2024-04-21T08:48:50.762475+0000","last_unstale":"2024-04-21T08:51:10.828279+0000","last_undegraded":"2024-04-21T08:51:10.828279+0000","last_fullsized":"2024-04-21T08:51:10.828279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:27.540414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":167,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.027197+0000","last_change":"2024-04-21T08:48:45.550151+0000","last_active":"2024-04-21T08:51:20.027197+0000","last_peered":"2024-04-21T08:51:20.027197+0000","last_clean":"2024-04-21T08:51:20.027197+0000","last_became_active":"2024-04-21T08:48:45.549829+0000","last_became_peered":"2024-04-21T08:48:45.549829+0000","last_unstale":"2024-04-21T08:51:20.027197+0000","last_undegraded":"2024-04-21T08:51:20.027197+0000","last_fullsized":"2024-04-21T08:51:20.027197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:14.061189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":129,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006968+0000","last_change":"2024-04-21T08:48:46.472072+0000","last_active":"2024-04-21T08:51:11.006968+0000","last_peered":"2024-04-21T08:51:11.006968+0000","last_clean":"2024-04-21T08:51:11.006968+0000","last_became_active":"2024-04-21T08:48:46.471660+0000","last_became_peered":"2024-04-21T08:48:46.471660+0000","last_unstale":"2024-04-21T08:51:11.006968+0000","last_undegraded":"2024-04-21T08:51:11.006968+0000","last_fullsized":"2024-04-21T08:51:11.006968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:30:11.843870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936126+0000","last_change":"2024-04-21T08:48:50.773021+0000","last_active":"2024-04-21T08:51:10.936126+0000","last_peered":"2024-04-21T08:51:10.936126+0000","last_clean":"2024-04-21T08:51:10.936126+0000","last_became_active":"2024-04-21T08:48:50.771746+0000","last_became_peered":"2024-04-21T08:48:50.771746+0000","last_unstale":"2024-04-21T08:51:10.936126+0000","last_undegraded":"2024-04-21T08:51:10.936126+0000","last_fullsized":"2024-04-21T08:51:10.936126+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:33:21.415967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063066+0000","last_change":"2024-04-21T08:48:45.550376+0000","last_active":"2024-04-21T08:51:11.063066+0000","last_peered":"2024-04-21T08:51:11.063066+0000","last_clean":"2024-04-21T08:51:11.063066+0000","last_became_active":"2024-04-21T08:48:45.549594+0000","last_became_peered":"2024-04-21T08:48:45.549594+0000","last_unstale":"2024-04-21T08:51:11.063066+0000","last_undegraded":"2024-04-21T08:51:11.063066+0000","last_fullsized":"2024-04-21T08:51:11.063066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:26:27.046963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877696+0000","last_change":"2024-04-21T08:48:46.469499+0000","last_active":"2024-04-21T08:51:10.877696+0000","last_peered":"2024-04-21T08:51:10.877696+0000","last_clean":"2024-04-21T08:51:10.877696+0000","last_became_active":"2024-04-21T08:48:46.469124+0000","last_became_peered":"2024-04-21T08:48:46.469124+0000","last_unstale":"2024-04-21T08:51:10.877696+0000","last_undegraded":"2024-04-21T08:51:10.877696+0000","last_fullsized":"2024-04-21T08:51:10.877696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:58.708916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038260+0000","last_change":"2024-04-21T08:50:37.352313+0000","last_active":"2024-04-21T08:51:11.038260+0000","last_peered":"2024-04-21T08:51:11.038260+0000","last_clean":"2024-04-21T08:51:11.038260+0000","last_became_active":"2024-04-21T08:50:37.351775+0000","last_became_peered":"2024-04-21T08:50:37.351775+0000","last_unstale":"2024-04-21T08:51:11.038260+0000","last_undegraded":"2024-04-21T08:51:11.038260+0000","last_fullsized":"2024-04-21T08:51:11.038260+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:12:51.239368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,10],"acting":[4,1,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945985+0000","last_change":"2024-04-21T08:48:45.559319+0000","last_active":"2024-04-21T08:51:10.945985+0000","last_peered":"2024-04-21T08:51:10.945985+0000","last_clean":"2024-04-21T08:51:10.945985+0000","last_became_active":"2024-04-21T08:48:45.558895+0000","last_became_peered":"2024-04-21T08:48:45.558895+0000","last_unstale":"2024-04-21T08:51:10.945985+0000","last_undegraded":"2024-04-21T08:51:10.945985+0000","last_fullsized":"2024-04-21T08:51:10.945985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:01.878169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006563+0000","last_change":"2024-04-21T08:48:46.466110+0000","last_active":"2024-04-21T08:51:11.006563+0000","last_peered":"2024-04-21T08:51:11.006563+0000","last_clean":"2024-04-21T08:51:11.006563+0000","last_became_active":"2024-04-21T08:48:46.464374+0000","last_became_peered":"2024-04-21T08:48:46.464374+0000","last_unstale":"2024-04-21T08:51:11.006563+0000","last_undegraded":"2024-04-21T08:51:11.006563+0000","last_fullsized":"2024-04-21T08:51:11.006563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:26.307051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936310+0000","last_change":"2024-04-21T08:48:50.759976+0000","last_active":"2024-04-21T08:51:10.936310+0000","last_peered":"2024-04-21T08:51:10.936310+0000","last_clean":"2024-04-21T08:51:10.936310+0000","last_became_active":"2024-04-21T08:48:50.759596+0000","last_became_peered":"2024-04-21T08:48:50.759596+0000","last_unstale":"2024-04-21T08:51:10.936310+0000","last_undegraded":"2024-04-21T08:51:10.936310+0000","last_fullsized":"2024-04-21T08:51:10.936310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:09.678388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'42","reported_seq":347,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:21.028496+0000","last_change":"2024-04-21T08:48:45.549228+0000","last_active":"2024-04-21T08:51:21.028496+0000","last_peered":"2024-04-21T08:51:21.028496+0000","last_clean":"2024-04-21T08:51:21.028496+0000","last_became_active":"2024-04-21T08:48:45.548865+0000","last_became_peered":"2024-04-21T08:48:45.548865+0000","last_unstale":"2024-04-21T08:51:21.028496+0000","last_undegraded":"2024-04-21T08:51:21.028496+0000","last_fullsized":"2024-04-21T08:51:21.028496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:28.620968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71920,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":271,"num_read_kb":207,"num_write":43,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939637+0000","last_change":"2024-04-21T08:48:46.470322+0000","last_active":"2024-04-21T08:51:10.939637+0000","last_peered":"2024-04-21T08:51:10.939637+0000","last_clean":"2024-04-21T08:51:10.939637+0000","last_became_active":"2024-04-21T08:48:46.467225+0000","last_became_peered":"2024-04-21T08:48:46.467225+0000","last_unstale":"2024-04-21T08:51:10.939637+0000","last_undegraded":"2024-04-21T08:51:10.939637+0000","last_fullsized":"2024-04-21T08:51:10.939637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:31:17.390926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.946326+0000","last_change":"2024-04-21T08:48:50.770982+0000","last_active":"2024-04-21T08:51:10.946326+0000","last_peered":"2024-04-21T08:51:10.946326+0000","last_clean":"2024-04-21T08:51:10.946326+0000","last_became_active":"2024-04-21T08:48:50.770312+0000","last_became_peered":"2024-04-21T08:48:50.770312+0000","last_unstale":"2024-04-21T08:51:10.946326+0000","last_undegraded":"2024-04-21T08:51:10.946326+0000","last_fullsized":"2024-04-21T08:51:10.946326+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:41:33.027351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832599+0000","last_change":"2024-04-21T08:48:45.556134+0000","last_active":"2024-04-21T08:51:10.832599+0000","last_peered":"2024-04-21T08:51:10.832599+0000","last_clean":"2024-04-21T08:51:10.832599+0000","last_became_active":"2024-04-21T08:48:45.555818+0000","last_became_peered":"2024-04-21T08:48:45.555818+0000","last_unstale":"2024-04-21T08:51:10.832599+0000","last_undegraded":"2024-04-21T08:51:10.832599+0000","last_fullsized":"2024-04-21T08:51:10.832599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:00:08.083783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036365+0000","last_change":"2024-04-21T08:49:36.913151+0000","last_active":"2024-04-21T08:51:11.036365+0000","last_peered":"2024-04-21T08:51:11.036365+0000","last_clean":"2024-04-21T08:51:11.036365+0000","last_became_active":"2024-04-21T08:49:36.912810+0000","last_became_peered":"2024-04-21T08:49:36.912810+0000","last_unstale":"2024-04-21T08:51:11.036365+0000","last_undegraded":"2024-04-21T08:51:11.036365+0000","last_fullsized":"2024-04-21T08:51:11.036365+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:17:32.887163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062954+0000","last_change":"2024-04-21T08:48:50.765967+0000","last_active":"2024-04-21T08:51:11.062954+0000","last_peered":"2024-04-21T08:51:11.062954+0000","last_clean":"2024-04-21T08:51:11.062954+0000","last_became_active":"2024-04-21T08:48:50.765557+0000","last_became_peered":"2024-04-21T08:48:50.765557+0000","last_unstale":"2024-04-21T08:51:11.062954+0000","last_undegraded":"2024-04-21T08:51:11.062954+0000","last_fullsized":"2024-04-21T08:51:11.062954+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:52.044058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945823+0000","last_change":"2024-04-21T08:48:45.559939+0000","last_active":"2024-04-21T08:51:10.945823+0000","last_peered":"2024-04-21T08:51:10.945823+0000","last_clean":"2024-04-21T08:51:10.945823+0000","last_became_active":"2024-04-21T08:48:45.559635+0000","last_became_peered":"2024-04-21T08:48:45.559635+0000","last_unstale":"2024-04-21T08:51:10.945823+0000","last_undegraded":"2024-04-21T08:51:10.945823+0000","last_fullsized":"2024-04-21T08:51:10.945823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:17.872806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062944+0000","last_change":"2024-04-21T08:48:46.464878+0000","last_active":"2024-04-21T08:51:11.062944+0000","last_peered":"2024-04-21T08:51:11.062944+0000","last_clean":"2024-04-21T08:51:11.062944+0000","last_became_active":"2024-04-21T08:48:46.464552+0000","last_became_peered":"2024-04-21T08:48:46.464552+0000","last_unstale":"2024-04-21T08:51:11.062944+0000","last_undegraded":"2024-04-21T08:51:11.062944+0000","last_fullsized":"2024-04-21T08:51:11.062944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:46:10.558693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939454+0000","last_change":"2024-04-21T08:50:37.351314+0000","last_active":"2024-04-21T08:51:10.939454+0000","last_peered":"2024-04-21T08:51:10.939454+0000","last_clean":"2024-04-21T08:51:10.939454+0000","last_became_active":"2024-04-21T08:50:37.350956+0000","last_became_peered":"2024-04-21T08:50:37.350956+0000","last_unstale":"2024-04-21T08:51:10.939454+0000","last_undegraded":"2024-04-21T08:51:10.939454+0000","last_fullsized":"2024-04-21T08:51:10.939454+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:01.802427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"94'7","reported_seq":175,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.050635+0000","last_change":"2024-04-21T08:48:45.548685+0000","last_active":"2024-04-21T08:51:20.050635+0000","last_peered":"2024-04-21T08:51:20.050635+0000","last_clean":"2024-04-21T08:51:20.050635+0000","last_became_active":"2024-04-21T08:48:45.548372+0000","last_became_peered":"2024-04-21T08:48:45.548372+0000","last_unstale":"2024-04-21T08:51:20.050635+0000","last_undegraded":"2024-04-21T08:51:20.050635+0000","last_fullsized":"2024-04-21T08:51:20.050635+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:44.780297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":135,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062806+0000","last_change":"2024-04-21T08:48:46.471541+0000","last_active":"2024-04-21T08:51:11.062806+0000","last_peered":"2024-04-21T08:51:11.062806+0000","last_clean":"2024-04-21T08:51:11.062806+0000","last_became_active":"2024-04-21T08:48:46.471223+0000","last_became_peered":"2024-04-21T08:48:46.471223+0000","last_unstale":"2024-04-21T08:51:11.062806+0000","last_undegraded":"2024-04-21T08:51:11.062806+0000","last_fullsized":"2024-04-21T08:51:11.062806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:24:50.867905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036983+0000","last_change":"2024-04-21T08:48:50.773941+0000","last_active":"2024-04-21T08:51:11.036983+0000","last_peered":"2024-04-21T08:51:11.036983+0000","last_clean":"2024-04-21T08:51:11.036983+0000","last_became_active":"2024-04-21T08:48:50.772081+0000","last_became_peered":"2024-04-21T08:48:50.772081+0000","last_unstale":"2024-04-21T08:51:11.036983+0000","last_undegraded":"2024-04-21T08:51:11.036983+0000","last_fullsized":"2024-04-21T08:51:11.036983+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:04.438944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062783+0000","last_change":"2024-04-21T08:49:36.923090+0000","last_active":"2024-04-21T08:51:11.062783+0000","last_peered":"2024-04-21T08:51:11.062783+0000","last_clean":"2024-04-21T08:51:11.062783+0000","last_became_active":"2024-04-21T08:49:36.908075+0000","last_became_peered":"2024-04-21T08:49:36.908075+0000","last_unstale":"2024-04-21T08:51:11.062783+0000","last_undegraded":"2024-04-21T08:51:11.062783+0000","last_fullsized":"2024-04-21T08:51:11.062783+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:41.683372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945471+0000","last_change":"2024-04-21T08:48:46.465559+0000","last_active":"2024-04-21T08:51:10.945471+0000","last_peered":"2024-04-21T08:51:10.945471+0000","last_clean":"2024-04-21T08:51:10.945471+0000","last_became_active":"2024-04-21T08:48:46.465125+0000","last_became_peered":"2024-04-21T08:48:46.465125+0000","last_unstale":"2024-04-21T08:51:10.945471+0000","last_undegraded":"2024-04-21T08:51:10.945471+0000","last_fullsized":"2024-04-21T08:51:10.945471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:52:38.423925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945406+0000","last_change":"2024-04-21T08:48:50.763609+0000","last_active":"2024-04-21T08:51:10.945406+0000","last_peered":"2024-04-21T08:51:10.945406+0000","last_clean":"2024-04-21T08:51:10.945406+0000","last_became_active":"2024-04-21T08:48:50.763144+0000","last_became_peered":"2024-04-21T08:48:50.763144+0000","last_unstale":"2024-04-21T08:51:10.945406+0000","last_undegraded":"2024-04-21T08:51:10.945406+0000","last_fullsized":"2024-04-21T08:51:10.945406+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:26:18.611235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939721+0000","last_change":"2024-04-21T08:48:45.549912+0000","last_active":"2024-04-21T08:51:10.939721+0000","last_peered":"2024-04-21T08:51:10.939721+0000","last_clean":"2024-04-21T08:51:10.939721+0000","last_became_active":"2024-04-21T08:48:45.549585+0000","last_became_peered":"2024-04-21T08:48:45.549585+0000","last_unstale":"2024-04-21T08:51:10.939721+0000","last_undegraded":"2024-04-21T08:51:10.939721+0000","last_fullsized":"2024-04-21T08:51:10.939721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:58.287402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063099+0000","last_change":"2024-04-21T08:48:46.462746+0000","last_active":"2024-04-21T08:51:11.063099+0000","last_peered":"2024-04-21T08:51:11.063099+0000","last_clean":"2024-04-21T08:51:11.063099+0000","last_became_active":"2024-04-21T08:48:46.461602+0000","last_became_peered":"2024-04-21T08:48:46.461602+0000","last_unstale":"2024-04-21T08:51:11.063099+0000","last_undegraded":"2024-04-21T08:51:11.063099+0000","last_fullsized":"2024-04-21T08:51:11.063099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:52.198634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939308+0000","last_change":"2024-04-21T08:48:50.770770+0000","last_active":"2024-04-21T08:51:10.939308+0000","last_peered":"2024-04-21T08:51:10.939308+0000","last_clean":"2024-04-21T08:51:10.939308+0000","last_became_active":"2024-04-21T08:48:50.770326+0000","last_became_peered":"2024-04-21T08:48:50.770326+0000","last_unstale":"2024-04-21T08:51:10.939308+0000","last_undegraded":"2024-04-21T08:51:10.939308+0000","last_fullsized":"2024-04-21T08:51:10.939308+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:38.325155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936408+0000","last_change":"2024-04-21T08:48:45.558867+0000","last_active":"2024-04-21T08:51:10.936408+0000","last_peered":"2024-04-21T08:51:10.936408+0000","last_clean":"2024-04-21T08:51:10.936408+0000","last_became_active":"2024-04-21T08:48:45.558492+0000","last_became_peered":"2024-04-21T08:48:45.558492+0000","last_unstale":"2024-04-21T08:51:10.936408+0000","last_undegraded":"2024-04-21T08:51:10.936408+0000","last_fullsized":"2024-04-21T08:51:10.936408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:44:55.156404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.830180+0000","last_change":"2024-04-21T08:48:46.466461+0000","last_active":"2024-04-21T08:51:10.830180+0000","last_peered":"2024-04-21T08:51:10.830180+0000","last_clean":"2024-04-21T08:51:10.830180+0000","last_became_active":"2024-04-21T08:48:46.466035+0000","last_became_peered":"2024-04-21T08:48:46.466035+0000","last_unstale":"2024-04-21T08:51:10.830180+0000","last_undegraded":"2024-04-21T08:51:10.830180+0000","last_fullsized":"2024-04-21T08:51:10.830180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:47:32.307753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036555+0000","last_change":"2024-04-21T08:48:50.767655+0000","last_active":"2024-04-21T08:51:11.036555+0000","last_peered":"2024-04-21T08:51:11.036555+0000","last_clean":"2024-04-21T08:51:11.036555+0000","last_became_active":"2024-04-21T08:48:50.767305+0000","last_became_peered":"2024-04-21T08:48:50.767305+0000","last_unstale":"2024-04-21T08:51:11.036555+0000","last_undegraded":"2024-04-21T08:51:11.036555+0000","last_fullsized":"2024-04-21T08:51:11.036555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:31.210126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062757+0000","last_change":"2024-04-21T08:48:45.552952+0000","last_active":"2024-04-21T08:51:11.062757+0000","last_peered":"2024-04-21T08:51:11.062757+0000","last_clean":"2024-04-21T08:51:11.062757+0000","last_became_active":"2024-04-21T08:48:45.552556+0000","last_became_peered":"2024-04-21T08:48:45.552556+0000","last_unstale":"2024-04-21T08:51:11.062757+0000","last_undegraded":"2024-04-21T08:51:11.062757+0000","last_fullsized":"2024-04-21T08:51:11.062757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:03.428801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829859+0000","last_change":"2024-04-21T08:48:46.469020+0000","last_active":"2024-04-21T08:51:10.829859+0000","last_peered":"2024-04-21T08:51:10.829859+0000","last_clean":"2024-04-21T08:51:10.829859+0000","last_became_active":"2024-04-21T08:48:46.468631+0000","last_became_peered":"2024-04-21T08:48:46.468631+0000","last_unstale":"2024-04-21T08:51:10.829859+0000","last_undegraded":"2024-04-21T08:51:10.829859+0000","last_fullsized":"2024-04-21T08:51:10.829859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:08.559516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036969+0000","last_change":"2024-04-21T08:48:50.773942+0000","last_active":"2024-04-21T08:51:11.036969+0000","last_peered":"2024-04-21T08:51:11.036969+0000","last_clean":"2024-04-21T08:51:11.036969+0000","last_became_active":"2024-04-21T08:48:50.771876+0000","last_became_peered":"2024-04-21T08:48:50.771876+0000","last_unstale":"2024-04-21T08:51:11.036969+0000","last_undegraded":"2024-04-21T08:51:11.036969+0000","last_fullsized":"2024-04-21T08:51:11.036969+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:19.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945411+0000","last_change":"2024-04-21T08:48:46.462727+0000","last_active":"2024-04-21T08:51:10.945411+0000","last_peered":"2024-04-21T08:51:10.945411+0000","last_clean":"2024-04-21T08:51:10.945411+0000","last_became_active":"2024-04-21T08:48:46.461669+0000","last_became_peered":"2024-04-21T08:48:46.461669+0000","last_unstale":"2024-04-21T08:51:10.945411+0000","last_undegraded":"2024-04-21T08:51:10.945411+0000","last_fullsized":"2024-04-21T08:51:10.945411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:27.290790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.049963+0000","last_change":"2024-04-21T08:48:45.558541+0000","last_active":"2024-04-21T08:51:20.049963+0000","last_peered":"2024-04-21T08:51:20.049963+0000","last_clean":"2024-04-21T08:51:20.049963+0000","last_became_active":"2024-04-21T08:48:45.558186+0000","last_became_peered":"2024-04-21T08:48:45.558186+0000","last_unstale":"2024-04-21T08:51:20.049963+0000","last_undegraded":"2024-04-21T08:51:20.049963+0000","last_fullsized":"2024-04-21T08:51:20.049963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:02.919144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":129,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062978+0000","last_change":"2024-04-21T08:48:50.770838+0000","last_active":"2024-04-21T08:51:11.062978+0000","last_peered":"2024-04-21T08:51:11.062978+0000","last_clean":"2024-04-21T08:51:11.062978+0000","last_became_active":"2024-04-21T08:48:50.770447+0000","last_became_peered":"2024-04-21T08:48:50.770447+0000","last_unstale":"2024-04-21T08:51:11.062978+0000","last_undegraded":"2024-04-21T08:51:11.062978+0000","last_fullsized":"2024-04-21T08:51:11.062978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:48:21.010284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828090+0000","last_change":"2024-04-21T08:48:46.478145+0000","last_active":"2024-04-21T08:51:10.828090+0000","last_peered":"2024-04-21T08:51:10.828090+0000","last_clean":"2024-04-21T08:51:10.828090+0000","last_became_active":"2024-04-21T08:48:46.477813+0000","last_became_peered":"2024-04-21T08:48:46.477813+0000","last_unstale":"2024-04-21T08:51:10.828090+0000","last_undegraded":"2024-04-21T08:51:10.828090+0000","last_fullsized":"2024-04-21T08:51:10.828090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:15.917619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038191+0000","last_change":"2024-04-21T08:48:45.545269+0000","last_active":"2024-04-21T08:51:11.038191+0000","last_peered":"2024-04-21T08:51:11.038191+0000","last_clean":"2024-04-21T08:51:11.038191+0000","last_became_active":"2024-04-21T08:48:45.544887+0000","last_became_peered":"2024-04-21T08:48:45.544887+0000","last_unstale":"2024-04-21T08:51:11.038191+0000","last_undegraded":"2024-04-21T08:51:11.038191+0000","last_fullsized":"2024-04-21T08:51:11.038191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:27.222570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"94'148","reported_seq":411,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:12.946003+0000","last_change":"2024-04-21T08:47:34.198161+0000","last_active":"2024-04-21T08:51:12.946003+0000","last_peered":"2024-04-21T08:51:12.946003+0000","last_clean":"2024-04-21T08:51:12.946003+0000","last_became_active":"2024-04-21T08:47:34.197654+0000","last_became_peered":"2024-04-21T08:47:34.197654+0000","last_unstale":"2024-04-21T08:51:12.946003+0000","last_undegraded":"2024-04-21T08:51:12.946003+0000","last_fullsized":"2024-04-21T08:51:12.946003+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_clean_scrub_stamp":"2024-04-21T08:44:05.493464+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:58:58.295783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945722+0000","last_change":"2024-04-21T08:48:50.770826+0000","last_active":"2024-04-21T08:51:10.945722+0000","last_peered":"2024-04-21T08:51:10.945722+0000","last_clean":"2024-04-21T08:51:10.945722+0000","last_became_active":"2024-04-21T08:48:50.770372+0000","last_became_peered":"2024-04-21T08:48:50.770372+0000","last_unstale":"2024-04-21T08:51:10.945722+0000","last_undegraded":"2024-04-21T08:51:10.945722+0000","last_fullsized":"2024-04-21T08:51:10.945722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:59.637874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.035696+0000","last_change":"2024-04-21T08:48:46.459330+0000","last_active":"2024-04-21T08:51:11.035696+0000","last_peered":"2024-04-21T08:51:11.035696+0000","last_clean":"2024-04-21T08:51:11.035696+0000","last_became_active":"2024-04-21T08:48:46.458963+0000","last_became_peered":"2024-04-21T08:48:46.458963+0000","last_unstale":"2024-04-21T08:51:11.035696+0000","last_undegraded":"2024-04-21T08:51:11.035696+0000","last_fullsized":"2024-04-21T08:51:11.035696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:10.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062640+0000","last_change":"2024-04-21T08:49:36.917854+0000","last_active":"2024-04-21T08:51:11.062640+0000","last_peered":"2024-04-21T08:51:11.062640+0000","last_clean":"2024-04-21T08:51:11.062640+0000","last_became_active":"2024-04-21T08:49:36.915390+0000","last_became_peered":"2024-04-21T08:49:36.915390+0000","last_unstale":"2024-04-21T08:51:11.062640+0000","last_undegraded":"2024-04-21T08:51:11.062640+0000","last_fullsized":"2024-04-21T08:51:11.062640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:45.920752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062798+0000","last_change":"2024-04-21T08:48:50.765339+0000","last_active":"2024-04-21T08:51:11.062798+0000","last_peered":"2024-04-21T08:51:11.062798+0000","last_clean":"2024-04-21T08:51:11.062798+0000","last_became_active":"2024-04-21T08:48:50.764858+0000","last_became_peered":"2024-04-21T08:48:50.764858+0000","last_unstale":"2024-04-21T08:51:11.062798+0000","last_undegraded":"2024-04-21T08:51:11.062798+0000","last_fullsized":"2024-04-21T08:51:11.062798+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:26.075041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939660+0000","last_change":"2024-04-21T08:48:45.557768+0000","last_active":"2024-04-21T08:51:10.939660+0000","last_peered":"2024-04-21T08:51:10.939660+0000","last_clean":"2024-04-21T08:51:10.939660+0000","last_became_active":"2024-04-21T08:48:45.557448+0000","last_became_peered":"2024-04-21T08:48:45.557448+0000","last_unstale":"2024-04-21T08:51:10.939660+0000","last_undegraded":"2024-04-21T08:51:10.939660+0000","last_fullsized":"2024-04-21T08:51:10.939660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:09.063913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036820+0000","last_change":"2024-04-21T08:48:46.459425+0000","last_active":"2024-04-21T08:51:11.036820+0000","last_peered":"2024-04-21T08:51:11.036820+0000","last_clean":"2024-04-21T08:51:11.036820+0000","last_became_active":"2024-04-21T08:48:46.459020+0000","last_became_peered":"2024-04-21T08:48:46.459020+0000","last_unstale":"2024-04-21T08:51:11.036820+0000","last_undegraded":"2024-04-21T08:51:11.036820+0000","last_fullsized":"2024-04-21T08:51:11.036820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:08:39.042825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828005+0000","last_change":"2024-04-21T08:48:50.765740+0000","last_active":"2024-04-21T08:51:10.828005+0000","last_peered":"2024-04-21T08:51:10.828005+0000","last_clean":"2024-04-21T08:51:10.828005+0000","last_became_active":"2024-04-21T08:48:50.765359+0000","last_became_peered":"2024-04-21T08:48:50.765359+0000","last_unstale":"2024-04-21T08:51:10.828005+0000","last_undegraded":"2024-04-21T08:51:10.828005+0000","last_fullsized":"2024-04-21T08:51:10.828005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:34:42.716165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'98","reported_seq":224,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.520533+0000","last_change":"2024-04-21T08:48:45.564532+0000","last_active":"2024-04-21T08:51:20.520533+0000","last_peered":"2024-04-21T08:51:20.520533+0000","last_clean":"2024-04-21T08:51:20.520533+0000","last_became_active":"2024-04-21T08:48:45.564225+0000","last_became_peered":"2024-04-21T08:48:45.564225+0000","last_unstale":"2024-04-21T08:51:20.520533+0000","last_undegraded":"2024-04-21T08:51:20.520533+0000","last_fullsized":"2024-04-21T08:51:20.520533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:42.850371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":179,"num_write_kb":1285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828519+0000","last_change":"2024-04-21T08:48:46.466622+0000","last_active":"2024-04-21T08:51:10.828519+0000","last_peered":"2024-04-21T08:51:10.828519+0000","last_clean":"2024-04-21T08:51:10.828519+0000","last_became_active":"2024-04-21T08:48:46.466129+0000","last_became_peered":"2024-04-21T08:48:46.466129+0000","last_unstale":"2024-04-21T08:51:10.828519+0000","last_undegraded":"2024-04-21T08:51:10.828519+0000","last_fullsized":"2024-04-21T08:51:10.828519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:20:26.064193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832121+0000","last_change":"2024-04-21T08:48:50.759534+0000","last_active":"2024-04-21T08:51:10.832121+0000","last_peered":"2024-04-21T08:51:10.832121+0000","last_clean":"2024-04-21T08:51:10.832121+0000","last_became_active":"2024-04-21T08:48:50.759116+0000","last_became_peered":"2024-04-21T08:48:50.759116+0000","last_unstale":"2024-04-21T08:51:10.832121+0000","last_undegraded":"2024-04-21T08:51:10.832121+0000","last_fullsized":"2024-04-21T08:51:10.832121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:02:16.200901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038642+0000","last_change":"2024-04-21T08:48:45.545773+0000","last_active":"2024-04-21T08:51:11.038642+0000","last_peered":"2024-04-21T08:51:11.038642+0000","last_clean":"2024-04-21T08:51:11.038642+0000","last_became_active":"2024-04-21T08:48:45.545447+0000","last_became_peered":"2024-04-21T08:48:45.545447+0000","last_unstale":"2024-04-21T08:51:11.038642+0000","last_undegraded":"2024-04-21T08:51:11.038642+0000","last_fullsized":"2024-04-21T08:51:11.038642+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:59:42.171602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877976+0000","last_change":"2024-04-21T08:49:36.964411+0000","last_active":"2024-04-21T08:51:10.877976+0000","last_peered":"2024-04-21T08:51:10.877976+0000","last_clean":"2024-04-21T08:51:10.877976+0000","last_became_active":"2024-04-21T08:49:36.964046+0000","last_became_peered":"2024-04-21T08:49:36.964046+0000","last_unstale":"2024-04-21T08:51:10.877976+0000","last_undegraded":"2024-04-21T08:51:10.877976+0000","last_fullsized":"2024-04-21T08:51:10.877976+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:29:37.223495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038538+0000","last_change":"2024-04-21T08:50:37.351641+0000","last_active":"2024-04-21T08:51:11.038538+0000","last_peered":"2024-04-21T08:51:11.038538+0000","last_clean":"2024-04-21T08:51:11.038538+0000","last_became_active":"2024-04-21T08:50:37.351130+0000","last_became_peered":"2024-04-21T08:50:37.351130+0000","last_unstale":"2024-04-21T08:51:11.038538+0000","last_undegraded":"2024-04-21T08:51:11.038538+0000","last_fullsized":"2024-04-21T08:51:11.038538+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:56:58.989691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006155+0000","last_change":"2024-04-21T08:49:36.928993+0000","last_active":"2024-04-21T08:51:11.006155+0000","last_peered":"2024-04-21T08:51:11.006155+0000","last_clean":"2024-04-21T08:51:11.006155+0000","last_became_active":"2024-04-21T08:49:36.911247+0000","last_became_peered":"2024-04-21T08:49:36.911247+0000","last_unstale":"2024-04-21T08:51:11.006155+0000","last_undegraded":"2024-04-21T08:51:11.006155+0000","last_fullsized":"2024-04-21T08:51:11.006155+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:13.481670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939224+0000","last_change":"2024-04-21T08:48:46.465475+0000","last_active":"2024-04-21T08:51:10.939224+0000","last_peered":"2024-04-21T08:51:10.939224+0000","last_clean":"2024-04-21T08:51:10.939224+0000","last_became_active":"2024-04-21T08:48:46.465138+0000","last_became_peered":"2024-04-21T08:48:46.465138+0000","last_unstale":"2024-04-21T08:51:10.939224+0000","last_undegraded":"2024-04-21T08:51:10.939224+0000","last_fullsized":"2024-04-21T08:51:10.939224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:12:53.521547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936707+0000","last_change":"2024-04-21T08:48:50.765765+0000","last_active":"2024-04-21T08:51:10.936707+0000","last_peered":"2024-04-21T08:51:10.936707+0000","last_clean":"2024-04-21T08:51:10.936707+0000","last_became_active":"2024-04-21T08:48:50.765324+0000","last_became_peered":"2024-04-21T08:48:50.765324+0000","last_unstale":"2024-04-21T08:51:10.936707+0000","last_undegraded":"2024-04-21T08:51:10.936707+0000","last_fullsized":"2024-04-21T08:51:10.936707+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:41:37.846768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828042+0000","last_change":"2024-04-21T08:48:45.557400+0000","last_active":"2024-04-21T08:51:10.828042+0000","last_peered":"2024-04-21T08:51:10.828042+0000","last_clean":"2024-04-21T08:51:10.828042+0000","last_became_active":"2024-04-21T08:48:45.555012+0000","last_became_peered":"2024-04-21T08:48:45.555012+0000","last_unstale":"2024-04-21T08:51:10.828042+0000","last_undegraded":"2024-04-21T08:51:10.828042+0000","last_fullsized":"2024-04-21T08:51:10.828042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:00:31.254353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062791+0000","last_change":"2024-04-21T08:48:46.470042+0000","last_active":"2024-04-21T08:51:11.062791+0000","last_peered":"2024-04-21T08:51:11.062791+0000","last_clean":"2024-04-21T08:51:11.062791+0000","last_became_active":"2024-04-21T08:48:46.469716+0000","last_became_peered":"2024-04-21T08:48:46.469716+0000","last_unstale":"2024-04-21T08:51:11.062791+0000","last_undegraded":"2024-04-21T08:51:11.062791+0000","last_fullsized":"2024-04-21T08:51:11.062791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:09.934751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038592+0000","last_change":"2024-04-21T08:48:50.770889+0000","last_active":"2024-04-21T08:51:11.038592+0000","last_peered":"2024-04-21T08:51:11.038592+0000","last_clean":"2024-04-21T08:51:11.038592+0000","last_became_active":"2024-04-21T08:48:50.769980+0000","last_became_peered":"2024-04-21T08:48:50.769980+0000","last_unstale":"2024-04-21T08:51:11.038592+0000","last_undegraded":"2024-04-21T08:51:11.038592+0000","last_fullsized":"2024-04-21T08:51:11.038592+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:44.631633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832532+0000","last_change":"2024-04-21T08:48:45.555479+0000","last_active":"2024-04-21T08:51:10.832532+0000","last_peered":"2024-04-21T08:51:10.832532+0000","last_clean":"2024-04-21T08:51:10.832532+0000","last_became_active":"2024-04-21T08:48:45.555162+0000","last_became_peered":"2024-04-21T08:48:45.555162+0000","last_unstale":"2024-04-21T08:51:10.832532+0000","last_undegraded":"2024-04-21T08:51:10.832532+0000","last_fullsized":"2024-04-21T08:51:10.832532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:15:20.262798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828417+0000","last_change":"2024-04-21T08:48:46.467962+0000","last_active":"2024-04-21T08:51:10.828417+0000","last_peered":"2024-04-21T08:51:10.828417+0000","last_clean":"2024-04-21T08:51:10.828417+0000","last_became_active":"2024-04-21T08:48:46.467561+0000","last_became_peered":"2024-04-21T08:48:46.467561+0000","last_unstale":"2024-04-21T08:51:10.828417+0000","last_undegraded":"2024-04-21T08:51:10.828417+0000","last_fullsized":"2024-04-21T08:51:10.828417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:18.262009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038487+0000","last_change":"2024-04-21T08:48:50.767678+0000","last_active":"2024-04-21T08:51:11.038487+0000","last_peered":"2024-04-21T08:51:11.038487+0000","last_clean":"2024-04-21T08:51:11.038487+0000","last_became_active":"2024-04-21T08:48:50.767309+0000","last_became_peered":"2024-04-21T08:48:50.767309+0000","last_unstale":"2024-04-21T08:51:11.038487+0000","last_undegraded":"2024-04-21T08:51:11.038487+0000","last_fullsized":"2024-04-21T08:51:11.038487+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:16:40.065043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939597+0000","last_change":"2024-04-21T08:48:45.553321+0000","last_active":"2024-04-21T08:51:10.939597+0000","last_peered":"2024-04-21T08:51:10.939597+0000","last_clean":"2024-04-21T08:51:10.939597+0000","last_became_active":"2024-04-21T08:48:45.552871+0000","last_became_peered":"2024-04-21T08:48:45.552871+0000","last_unstale":"2024-04-21T08:51:10.939597+0000","last_undegraded":"2024-04-21T08:51:10.939597+0000","last_fullsized":"2024-04-21T08:51:10.939597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:16:27.339801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062861+0000","last_change":"2024-04-21T08:48:46.473453+0000","last_active":"2024-04-21T08:51:11.062861+0000","last_peered":"2024-04-21T08:51:11.062861+0000","last_clean":"2024-04-21T08:51:11.062861+0000","last_became_active":"2024-04-21T08:48:46.472613+0000","last_became_peered":"2024-04-21T08:48:46.472613+0000","last_unstale":"2024-04-21T08:51:11.062861+0000","last_undegraded":"2024-04-21T08:51:11.062861+0000","last_fullsized":"2024-04-21T08:51:11.062861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:43:58.995997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006138+0000","last_change":"2024-04-21T08:48:50.765282+0000","last_active":"2024-04-21T08:51:11.006138+0000","last_peered":"2024-04-21T08:51:11.006138+0000","last_clean":"2024-04-21T08:51:11.006138+0000","last_became_active":"2024-04-21T08:48:50.764891+0000","last_became_peered":"2024-04-21T08:48:50.764891+0000","last_unstale":"2024-04-21T08:51:11.006138+0000","last_undegraded":"2024-04-21T08:51:11.006138+0000","last_fullsized":"2024-04-21T08:51:11.006138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:18:40.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038155+0000","last_change":"2024-04-21T08:49:36.926642+0000","last_active":"2024-04-21T08:51:11.038155+0000","last_peered":"2024-04-21T08:51:11.038155+0000","last_clean":"2024-04-21T08:51:11.038155+0000","last_became_active":"2024-04-21T08:49:36.910453+0000","last_became_peered":"2024-04-21T08:49:36.910453+0000","last_unstale":"2024-04-21T08:51:11.038155+0000","last_undegraded":"2024-04-21T08:51:11.038155+0000","last_fullsized":"2024-04-21T08:51:11.038155+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:19.290010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063219+0000","last_change":"2024-04-21T08:48:46.470058+0000","last_active":"2024-04-21T08:51:11.063219+0000","last_peered":"2024-04-21T08:51:11.063219+0000","last_clean":"2024-04-21T08:51:11.063219+0000","last_became_active":"2024-04-21T08:48:46.469485+0000","last_became_peered":"2024-04-21T08:48:46.469485+0000","last_unstale":"2024-04-21T08:51:11.063219+0000","last_undegraded":"2024-04-21T08:51:11.063219+0000","last_fullsized":"2024-04-21T08:51:11.063219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:33:13.212087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877446+0000","last_change":"2024-04-21T08:48:50.762944+0000","last_active":"2024-04-21T08:51:10.877446+0000","last_peered":"2024-04-21T08:51:10.877446+0000","last_clean":"2024-04-21T08:51:10.877446+0000","last_became_active":"2024-04-21T08:48:50.762508+0000","last_became_peered":"2024-04-21T08:48:50.762508+0000","last_unstale":"2024-04-21T08:51:10.877446+0000","last_undegraded":"2024-04-21T08:51:10.877446+0000","last_fullsized":"2024-04-21T08:51:10.877446+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:35.393339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'65","reported_seq":130,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.177489+0000","last_change":"2024-04-21T08:48:45.560361+0000","last_active":"2024-04-21T08:51:20.177489+0000","last_peered":"2024-04-21T08:51:20.177489+0000","last_clean":"2024-04-21T08:51:20.177489+0000","last_became_active":"2024-04-21T08:48:45.559946+0000","last_became_peered":"2024-04-21T08:48:45.559946+0000","last_unstale":"2024-04-21T08:51:20.177489+0000","last_undegraded":"2024-04-21T08:51:20.177489+0000","last_fullsized":"2024-04-21T08:51:20.177489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:55.599714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063003+0000","last_change":"2024-04-21T08:48:46.472706+0000","last_active":"2024-04-21T08:51:11.063003+0000","last_peered":"2024-04-21T08:51:11.063003+0000","last_clean":"2024-04-21T08:51:11.063003+0000","last_became_active":"2024-04-21T08:48:46.472378+0000","last_became_peered":"2024-04-21T08:48:46.472378+0000","last_unstale":"2024-04-21T08:51:11.063003+0000","last_undegraded":"2024-04-21T08:51:11.063003+0000","last_fullsized":"2024-04-21T08:51:11.063003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:58.228899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945191+0000","last_change":"2024-04-21T08:48:50.760610+0000","last_active":"2024-04-21T08:51:10.945191+0000","last_peered":"2024-04-21T08:51:10.945191+0000","last_clean":"2024-04-21T08:51:10.945191+0000","last_became_active":"2024-04-21T08:48:50.759991+0000","last_became_peered":"2024-04-21T08:48:50.759991+0000","last_unstale":"2024-04-21T08:51:10.945191+0000","last_undegraded":"2024-04-21T08:51:10.945191+0000","last_fullsized":"2024-04-21T08:51:10.945191+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:43.649701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036942+0000","last_change":"2024-04-21T08:48:45.560773+0000","last_active":"2024-04-21T08:51:11.036942+0000","last_peered":"2024-04-21T08:51:11.036942+0000","last_clean":"2024-04-21T08:51:11.036942+0000","last_became_active":"2024-04-21T08:48:45.560431+0000","last_became_peered":"2024-04-21T08:48:45.560431+0000","last_unstale":"2024-04-21T08:51:11.036942+0000","last_undegraded":"2024-04-21T08:51:11.036942+0000","last_fullsized":"2024-04-21T08:51:11.036942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:07.858838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063259+0000","last_change":"2024-04-21T08:48:46.475983+0000","last_active":"2024-04-21T08:51:11.063259+0000","last_peered":"2024-04-21T08:51:11.063259+0000","last_clean":"2024-04-21T08:51:11.063259+0000","last_became_active":"2024-04-21T08:48:46.475667+0000","last_became_peered":"2024-04-21T08:48:46.475667+0000","last_unstale":"2024-04-21T08:51:11.063259+0000","last_undegraded":"2024-04-21T08:51:11.063259+0000","last_fullsized":"2024-04-21T08:51:11.063259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:28:41.013111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062960+0000","last_change":"2024-04-21T08:48:46.464675+0000","last_active":"2024-04-21T08:51:11.062960+0000","last_peered":"2024-04-21T08:51:11.062960+0000","last_clean":"2024-04-21T08:51:11.062960+0000","last_became_active":"2024-04-21T08:48:46.464314+0000","last_became_peered":"2024-04-21T08:48:46.464314+0000","last_unstale":"2024-04-21T08:51:11.062960+0000","last_undegraded":"2024-04-21T08:51:11.062960+0000","last_fullsized":"2024-04-21T08:51:11.062960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:53.048400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832307+0000","last_change":"2024-04-21T08:49:36.911025+0000","last_active":"2024-04-21T08:51:10.832307+0000","last_peered":"2024-04-21T08:51:10.832307+0000","last_clean":"2024-04-21T08:51:10.832307+0000","last_became_active":"2024-04-21T08:49:36.909942+0000","last_became_peered":"2024-04-21T08:49:36.909942+0000","last_unstale":"2024-04-21T08:51:10.832307+0000","last_undegraded":"2024-04-21T08:51:10.832307+0000","last_fullsized":"2024-04-21T08:51:10.832307+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:39.864253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832375+0000","last_change":"2024-04-21T08:48:46.467867+0000","last_active":"2024-04-21T08:51:10.832375+0000","last_peered":"2024-04-21T08:51:10.832375+0000","last_clean":"2024-04-21T08:51:10.832375+0000","last_became_active":"2024-04-21T08:48:46.465917+0000","last_became_peered":"2024-04-21T08:48:46.465917+0000","last_unstale":"2024-04-21T08:51:10.832375+0000","last_undegraded":"2024-04-21T08:51:10.832375+0000","last_fullsized":"2024-04-21T08:51:10.832375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:19:56.824998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877556+0000","last_change":"2024-04-21T08:48:46.467159+0000","last_active":"2024-04-21T08:51:10.877556+0000","last_peered":"2024-04-21T08:51:10.877556+0000","last_clean":"2024-04-21T08:51:10.877556+0000","last_became_active":"2024-04-21T08:48:46.466639+0000","last_became_peered":"2024-04-21T08:48:46.466639+0000","last_unstale":"2024-04-21T08:51:10.877556+0000","last_undegraded":"2024-04-21T08:51:10.877556+0000","last_fullsized":"2024-04-21T08:51:10.877556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:54:44.772535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036982+0000","last_change":"2024-04-21T08:48:46.466403+0000","last_active":"2024-04-21T08:51:11.036982+0000","last_peered":"2024-04-21T08:51:11.036982+0000","last_clean":"2024-04-21T08:51:11.036982+0000","last_became_active":"2024-04-21T08:48:46.466060+0000","last_became_peered":"2024-04-21T08:48:46.466060+0000","last_unstale":"2024-04-21T08:51:11.036982+0000","last_undegraded":"2024-04-21T08:51:11.036982+0000","last_fullsized":"2024-04-21T08:51:11.036982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:14.374193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063096+0000","last_change":"2024-04-21T08:49:36.909134+0000","last_active":"2024-04-21T08:51:11.063096+0000","last_peered":"2024-04-21T08:51:11.063096+0000","last_clean":"2024-04-21T08:51:11.063096+0000","last_became_active":"2024-04-21T08:49:36.908723+0000","last_became_peered":"2024-04-21T08:49:36.908723+0000","last_unstale":"2024-04-21T08:51:11.063096+0000","last_undegraded":"2024-04-21T08:51:11.063096+0000","last_fullsized":"2024-04-21T08:51:11.063096+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:28.123016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832933+0000","last_change":"2024-04-21T08:48:46.464098+0000","last_active":"2024-04-21T08:51:10.832933+0000","last_peered":"2024-04-21T08:51:10.832933+0000","last_clean":"2024-04-21T08:51:10.832933+0000","last_became_active":"2024-04-21T08:48:46.463756+0000","last_became_peered":"2024-04-21T08:48:46.463756+0000","last_unstale":"2024-04-21T08:51:10.832933+0000","last_undegraded":"2024-04-21T08:51:10.832933+0000","last_fullsized":"2024-04-21T08:51:10.832933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:31.725155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828666+0000","last_change":"2024-04-21T08:48:46.467372+0000","last_active":"2024-04-21T08:51:10.828666+0000","last_peered":"2024-04-21T08:51:10.828666+0000","last_clean":"2024-04-21T08:51:10.828666+0000","last_became_active":"2024-04-21T08:48:46.467043+0000","last_became_peered":"2024-04-21T08:48:46.467043+0000","last_unstale":"2024-04-21T08:51:10.828666+0000","last_undegraded":"2024-04-21T08:51:10.828666+0000","last_fullsized":"2024-04-21T08:51:10.828666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:41.889994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945316+0000","last_change":"2024-04-21T08:48:46.460604+0000","last_active":"2024-04-21T08:51:10.945316+0000","last_peered":"2024-04-21T08:51:10.945316+0000","last_clean":"2024-04-21T08:51:10.945316+0000","last_became_active":"2024-04-21T08:48:46.460276+0000","last_became_peered":"2024-04-21T08:48:46.460276+0000","last_unstale":"2024-04-21T08:51:10.945316+0000","last_undegraded":"2024-04-21T08:51:10.945316+0000","last_fullsized":"2024-04-21T08:51:10.945316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:30.194808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828076+0000","last_change":"2024-04-21T08:48:46.463665+0000","last_active":"2024-04-21T08:51:10.828076+0000","last_peered":"2024-04-21T08:51:10.828076+0000","last_clean":"2024-04-21T08:51:10.828076+0000","last_became_active":"2024-04-21T08:48:46.463309+0000","last_became_peered":"2024-04-21T08:48:46.463309+0000","last_unstale":"2024-04-21T08:51:10.828076+0000","last_undegraded":"2024-04-21T08:51:10.828076+0000","last_fullsized":"2024-04-21T08:51:10.828076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:12:36.436565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006407+0000","last_change":"2024-04-21T08:48:46.476845+0000","last_active":"2024-04-21T08:51:11.006407+0000","last_peered":"2024-04-21T08:51:11.006407+0000","last_clean":"2024-04-21T08:51:11.006407+0000","last_became_active":"2024-04-21T08:48:46.476507+0000","last_became_peered":"2024-04-21T08:48:46.476507+0000","last_unstale":"2024-04-21T08:51:11.006407+0000","last_undegraded":"2024-04-21T08:51:11.006407+0000","last_fullsized":"2024-04-21T08:51:11.006407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:47.424576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036615+0000","last_change":"2024-04-21T08:48:46.469316+0000","last_active":"2024-04-21T08:51:11.036615+0000","last_peered":"2024-04-21T08:51:11.036615+0000","last_clean":"2024-04-21T08:51:11.036615+0000","last_became_active":"2024-04-21T08:48:46.468852+0000","last_became_peered":"2024-04-21T08:48:46.468852+0000","last_unstale":"2024-04-21T08:51:11.036615+0000","last_undegraded":"2024-04-21T08:51:11.036615+0000","last_fullsized":"2024-04-21T08:51:11.036615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:05:03.664279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062922+0000","last_change":"2024-04-21T08:48:46.466703+0000","last_active":"2024-04-21T08:51:11.062922+0000","last_peered":"2024-04-21T08:51:11.062922+0000","last_clean":"2024-04-21T08:51:11.062922+0000","last_became_active":"2024-04-21T08:48:46.466395+0000","last_became_peered":"2024-04-21T08:48:46.466395+0000","last_unstale":"2024-04-21T08:51:11.062922+0000","last_undegraded":"2024-04-21T08:51:11.062922+0000","last_fullsized":"2024-04-21T08:51:11.062922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:08:56.956283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108027,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":799382,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1154,"num_read_kb":796,"num_write":425,"num_write_kb":2365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":2398146,"data_compressed":108306,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":529689,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645361,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32384,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712768,"statfs":{"total":95995035648,"available":95961874432,"internally_reserved":0,"allocated":1228800,"data_stored":1518311,"data_compressed":82401,"data_compressed_allocated":110592,"data_compressed_original":987136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":927125,"data_compressed":48868,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070524,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":922837,"data_compressed":48868,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1097728,"data_stored":1127655,"data_compressed":65588,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495686,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1028096,"data_stored":524989,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":596819,"data_compressed":31190,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":526377,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1122304,"data_stored":598677,"data_compressed":31190,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":1246631,"data_compressed":64070,"data_compressed_allocated":86016,"data_compressed_original":782336,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":531535,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1150976,"data_stored":1450161,"data_compressed":80790,"data_compressed_allocated":106496,"data_compressed_original":978944,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":71920,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":73778,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330798,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":393842,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":403044,"data_compressed":16813,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:51:24.462 INFO:tasks.quiescer.ceph_manager:clean! 2024-04-21T08:51:24.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:51:24.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.600+0000 7faefcd2c640 1 -- 172.21.15.28:0/690730647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef8106180 msgr2=0x7faef810dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.600+0000 7faefcd2c640 1 --2- 172.21.15.28:0/690730647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef8106180 0x7faef810dd50 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7faef0009ca0 tx=0x7faef002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.600+0000 7faefcd2c640 1 -- 172.21.15.28:0/690730647 shutdown_connections 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.600+0000 7faefcd2c640 1 --2- 172.21.15.28:0/690730647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef8106180 0x7faef810dd50 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.600+0000 7faefcd2c640 1 --2- 172.21.15.28:0/690730647 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faef8101a60 0x7faef8105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.600+0000 7faefcd2c640 1 --2- 172.21.15.28:0/690730647 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faef81010b0 0x7faef8101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.600+0000 7faefcd2c640 1 -- 172.21.15.28:0/690730647 >> 172.21.15.28:0/690730647 conn(0x7faef80771d0 msgr2=0x7faef80775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.600+0000 7faefcd2c640 1 -- 172.21.15.28:0/690730647 shutdown_connections 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.600+0000 7faefcd2c640 1 -- 172.21.15.28:0/690730647 wait complete. 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.601+0000 7faefcd2c640 1 Processor -- start 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.601+0000 7faefcd2c640 1 -- start start 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.601+0000 7faefcd2c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef81010b0 0x7faef81975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.601+0000 7faefcd2c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faef8106180 0x7faef8197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.601+0000 7faefcd2c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faef819be20 0x7faef819c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.601+0000 7faefcd2c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faef81110f0 con 0x7faef81010b0 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.601+0000 7faefcd2c640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7faef8110f70 con 0x7faef8106180 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.601+0000 7faefcd2c640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7faef8111270 con 0x7faef819be20 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.601+0000 7faef659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef81010b0 0x7faef81975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.601+0000 7faef659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef81010b0 0x7faef81975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44432/0 (socket says 172.21.15.28:44432) 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.601+0000 7faef659c640 1 -- 172.21.15.28:0/1213775197 learned_addr learned my addr 172.21.15.28:0/1213775197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.602+0000 7faef6d9d640 1 --2- 172.21.15.28:0/1213775197 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faef819be20 0x7faef819c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.602+0000 7faef659c640 1 -- 172.21.15.28:0/1213775197 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faef819be20 msgr2=0x7faef819c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.602+0000 7faef659c640 1 --2- 172.21.15.28:0/1213775197 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faef819be20 0x7faef819c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.602+0000 7faef5d9b640 1 --2- 172.21.15.28:0/1213775197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faef8106180 0x7faef8197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.602+0000 7faef659c640 1 -- 172.21.15.28:0/1213775197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faef8106180 msgr2=0x7faef8197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.602+0000 7faef659c640 1 --2- 172.21.15.28:0/1213775197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faef8106180 0x7faef8197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.602+0000 7faef659c640 1 -- 172.21.15.28:0/1213775197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef819c910 con 0x7faef81010b0 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.602+0000 7faef6d9d640 1 --2- 172.21.15.28:0/1213775197 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faef819be20 0x7faef819c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.602+0000 7faef659c640 1 --2- 172.21.15.28:0/1213775197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef81010b0 0x7faef81975b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7faee800c920 tx=0x7faee800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.602+0000 7faee77fe640 1 -- 172.21.15.28:0/1213775197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee8007c70 con 0x7faef81010b0 2024-04-21T08:51:24.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.602+0000 7faefcd2c640 1 -- 172.21.15.28:0/1213775197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef819cc00 con 0x7faef81010b0 2024-04-21T08:51:24.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.602+0000 7faee77fe640 1 -- 172.21.15.28:0/1213775197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faee8007e10 con 0x7faef81010b0 2024-04-21T08:51:24.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.602+0000 7faee77fe640 1 -- 172.21.15.28:0/1213775197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee8018820 con 0x7faef81010b0 2024-04-21T08:51:24.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.602+0000 7faefcd2c640 1 -- 172.21.15.28:0/1213775197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef81a4440 con 0x7faef81010b0 2024-04-21T08:51:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.603+0000 7faefcd2c640 1 -- 172.21.15.28:0/1213775197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faef8069230 con 0x7faef81010b0 2024-04-21T08:51:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.604+0000 7faee77fe640 1 -- 172.21.15.28:0/1213775197 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7faee800f450 con 0x7faef81010b0 2024-04-21T08:51:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.605+0000 7faee77fe640 1 --2- 172.21.15.28:0/1213775197 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faee00a3e40 0x7faee00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.605+0000 7faee77fe640 1 -- 172.21.15.28:0/1213775197 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7faee80d1080 con 0x7faef81010b0 2024-04-21T08:51:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.605+0000 7faef5d9b640 1 --2- 172.21.15.28:0/1213775197 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faee00a3e40 0x7faee00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.605+0000 7faef5d9b640 1 --2- 172.21.15.28:0/1213775197 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faee00a3e40 0x7faee00a6300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faef80fe430 tx=0x7faeec005be0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:24.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.609+0000 7faee77fe640 1 -- 172.21.15.28:0/1213775197 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faee8014030 con 0x7faef81010b0 2024-04-21T08:51:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.792+0000 7faefcd2c640 1 -- 172.21.15.28:0/1213775197 --> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faef8101a60 con 0x7faee00a3e40 2024-04-21T08:51:24.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.801+0000 7faee77fe640 1 -- 172.21.15.28:0/1213775197 <== mgr.25040 v2:172.21.15.28:6800/4182301910 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339719 (secure 0 0 0) 0x7faef8101a60 con 0x7faee00a3e40 2024-04-21T08:51:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-21T08:51:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.804+0000 7faefcd2c640 1 -- 172.21.15.28:0/1213775197 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faee00a3e40 msgr2=0x7faee00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.804+0000 7faefcd2c640 1 --2- 172.21.15.28:0/1213775197 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faee00a3e40 0x7faee00a6300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faef80fe430 tx=0x7faeec005be0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.804+0000 7faefcd2c640 1 -- 172.21.15.28:0/1213775197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef81010b0 msgr2=0x7faef81975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.804+0000 7faefcd2c640 1 --2- 172.21.15.28:0/1213775197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef81010b0 0x7faef81975b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7faee800c920 tx=0x7faee800cde0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.805+0000 7faefcd2c640 1 -- 172.21.15.28:0/1213775197 shutdown_connections 2024-04-21T08:51:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.805+0000 7faefcd2c640 1 --2- 172.21.15.28:0/1213775197 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faee00a3e40 0x7faee00a6300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.805+0000 7faefcd2c640 1 --2- 172.21.15.28:0/1213775197 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faef819be20 0x7faef819c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.805+0000 7faefcd2c640 1 --2- 172.21.15.28:0/1213775197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faef8106180 0x7faef8197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.805+0000 7faefcd2c640 1 --2- 172.21.15.28:0/1213775197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef81010b0 0x7faef81975b0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.805+0000 7faefcd2c640 1 -- 172.21.15.28:0/1213775197 >> 172.21.15.28:0/1213775197 conn(0x7faef80771d0 msgr2=0x7faef80febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.805+0000 7faefcd2c640 1 -- 172.21.15.28:0/1213775197 shutdown_connections 2024-04-21T08:51:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.805+0000 7faefcd2c640 1 -- 172.21.15.28:0/1213775197 wait complete. 2024-04-21T08:51:24.815 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-21T08:51:22.844483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1389984,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1391,"num_read_kb":1126,"num_write":628,"num_write_kb":5305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1181430,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":965,"ondisk_log_size":965,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370328,"kb_used_data":12812,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571496,"statfs":{"total":1151940427776,"available":1151561211904,"internally_reserved":0,"allocated":13119488,"data_stored":10500806,"data_compressed":571281,"data_compressed_allocated":774144,"data_compressed_original":4878336,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":199934,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":420,"num_read_kb":484,"num_write":155,"num_write_kb":922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.992496"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063213+0000","last_change":"2024-04-21T08:48:46.471626+0000","last_active":"2024-04-21T08:51:11.063213+0000","last_peered":"2024-04-21T08:51:11.063213+0000","last_clean":"2024-04-21T08:51:11.063213+0000","last_became_active":"2024-04-21T08:48:46.470498+0000","last_became_peered":"2024-04-21T08:48:46.470498+0000","last_unstale":"2024-04-21T08:51:11.063213+0000","last_undegraded":"2024-04-21T08:51:11.063213+0000","last_fullsized":"2024-04-21T08:51:11.063213+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:42:49.452181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877897+0000","last_change":"2024-04-21T08:48:46.469111+0000","last_active":"2024-04-21T08:51:10.877897+0000","last_peered":"2024-04-21T08:51:10.877897+0000","last_clean":"2024-04-21T08:51:10.877897+0000","last_became_active":"2024-04-21T08:48:46.467926+0000","last_became_peered":"2024-04-21T08:48:46.467926+0000","last_unstale":"2024-04-21T08:51:10.877897+0000","last_undegraded":"2024-04-21T08:51:10.877897+0000","last_fullsized":"2024-04-21T08:51:10.877897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:35:38.230212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877755+0000","last_change":"2024-04-21T08:48:46.475673+0000","last_active":"2024-04-21T08:51:10.877755+0000","last_peered":"2024-04-21T08:51:10.877755+0000","last_clean":"2024-04-21T08:51:10.877755+0000","last_became_active":"2024-04-21T08:48:46.474917+0000","last_became_peered":"2024-04-21T08:48:46.474917+0000","last_unstale":"2024-04-21T08:51:10.877755+0000","last_undegraded":"2024-04-21T08:51:10.877755+0000","last_fullsized":"2024-04-21T08:51:10.877755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:01:01.661704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038205+0000","last_change":"2024-04-21T08:48:46.468147+0000","last_active":"2024-04-21T08:51:11.038205+0000","last_peered":"2024-04-21T08:51:11.038205+0000","last_clean":"2024-04-21T08:51:11.038205+0000","last_became_active":"2024-04-21T08:48:46.467811+0000","last_became_peered":"2024-04-21T08:48:46.467811+0000","last_unstale":"2024-04-21T08:51:11.038205+0000","last_undegraded":"2024-04-21T08:51:11.038205+0000","last_fullsized":"2024-04-21T08:51:11.038205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:32:21.658605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939327+0000","last_change":"2024-04-21T08:48:46.471322+0000","last_active":"2024-04-21T08:51:10.939327+0000","last_peered":"2024-04-21T08:51:10.939327+0000","last_clean":"2024-04-21T08:51:10.939327+0000","last_became_active":"2024-04-21T08:48:46.470991+0000","last_became_peered":"2024-04-21T08:48:46.470991+0000","last_unstale":"2024-04-21T08:51:10.939327+0000","last_undegraded":"2024-04-21T08:51:10.939327+0000","last_fullsized":"2024-04-21T08:51:10.939327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:22.839870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829853+0000","last_change":"2024-04-21T08:48:46.467039+0000","last_active":"2024-04-21T08:51:10.829853+0000","last_peered":"2024-04-21T08:51:10.829853+0000","last_clean":"2024-04-21T08:51:10.829853+0000","last_became_active":"2024-04-21T08:48:46.466614+0000","last_became_peered":"2024-04-21T08:48:46.466614+0000","last_unstale":"2024-04-21T08:51:10.829853+0000","last_undegraded":"2024-04-21T08:51:10.829853+0000","last_fullsized":"2024-04-21T08:51:10.829853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:38.029215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062769+0000","last_change":"2024-04-21T08:48:46.473268+0000","last_active":"2024-04-21T08:51:11.062769+0000","last_peered":"2024-04-21T08:51:11.062769+0000","last_clean":"2024-04-21T08:51:11.062769+0000","last_became_active":"2024-04-21T08:48:46.472261+0000","last_became_peered":"2024-04-21T08:48:46.472261+0000","last_unstale":"2024-04-21T08:51:11.062769+0000","last_undegraded":"2024-04-21T08:51:11.062769+0000","last_fullsized":"2024-04-21T08:51:11.062769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:58:52.990687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828243+0000","last_change":"2024-04-21T08:48:46.472788+0000","last_active":"2024-04-21T08:51:10.828243+0000","last_peered":"2024-04-21T08:51:10.828243+0000","last_clean":"2024-04-21T08:51:10.828243+0000","last_became_active":"2024-04-21T08:48:46.472482+0000","last_became_peered":"2024-04-21T08:48:46.472482+0000","last_unstale":"2024-04-21T08:51:10.828243+0000","last_undegraded":"2024-04-21T08:51:10.828243+0000","last_fullsized":"2024-04-21T08:51:10.828243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:48.483846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006997+0000","last_change":"2024-04-21T08:48:46.460771+0000","last_active":"2024-04-21T08:51:11.006997+0000","last_peered":"2024-04-21T08:51:11.006997+0000","last_clean":"2024-04-21T08:51:11.006997+0000","last_became_active":"2024-04-21T08:48:46.460472+0000","last_became_peered":"2024-04-21T08:48:46.460472+0000","last_unstale":"2024-04-21T08:51:11.006997+0000","last_undegraded":"2024-04-21T08:51:11.006997+0000","last_fullsized":"2024-04-21T08:51:11.006997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:46:40.766245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.830308+0000","last_change":"2024-04-21T08:48:46.463475+0000","last_active":"2024-04-21T08:51:10.830308+0000","last_peered":"2024-04-21T08:51:10.830308+0000","last_clean":"2024-04-21T08:51:10.830308+0000","last_became_active":"2024-04-21T08:48:46.462559+0000","last_became_peered":"2024-04-21T08:48:46.462559+0000","last_unstale":"2024-04-21T08:51:10.830308+0000","last_undegraded":"2024-04-21T08:51:10.830308+0000","last_fullsized":"2024-04-21T08:51:10.830308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:22:06.372410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063168+0000","last_change":"2024-04-21T08:48:46.476997+0000","last_active":"2024-04-21T08:51:11.063168+0000","last_peered":"2024-04-21T08:51:11.063168+0000","last_clean":"2024-04-21T08:51:11.063168+0000","last_became_active":"2024-04-21T08:48:46.476620+0000","last_became_peered":"2024-04-21T08:48:46.476620+0000","last_unstale":"2024-04-21T08:51:11.063168+0000","last_undegraded":"2024-04-21T08:51:11.063168+0000","last_fullsized":"2024-04-21T08:51:11.063168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:56:37.569629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038421+0000","last_change":"2024-04-21T08:48:46.469355+0000","last_active":"2024-04-21T08:51:11.038421+0000","last_peered":"2024-04-21T08:51:11.038421+0000","last_clean":"2024-04-21T08:51:11.038421+0000","last_became_active":"2024-04-21T08:48:46.468604+0000","last_became_peered":"2024-04-21T08:48:46.468604+0000","last_unstale":"2024-04-21T08:51:11.038421+0000","last_undegraded":"2024-04-21T08:51:11.038421+0000","last_fullsized":"2024-04-21T08:51:11.038421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:23:54.500982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036699+0000","last_change":"2024-04-21T08:48:46.453116+0000","last_active":"2024-04-21T08:51:11.036699+0000","last_peered":"2024-04-21T08:51:11.036699+0000","last_clean":"2024-04-21T08:51:11.036699+0000","last_became_active":"2024-04-21T08:48:46.452607+0000","last_became_peered":"2024-04-21T08:48:46.452607+0000","last_unstale":"2024-04-21T08:51:11.036699+0000","last_undegraded":"2024-04-21T08:51:11.036699+0000","last_fullsized":"2024-04-21T08:51:11.036699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:57:04.371812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877416+0000","last_change":"2024-04-21T08:48:46.474516+0000","last_active":"2024-04-21T08:51:10.877416+0000","last_peered":"2024-04-21T08:51:10.877416+0000","last_clean":"2024-04-21T08:51:10.877416+0000","last_became_active":"2024-04-21T08:48:46.474129+0000","last_became_peered":"2024-04-21T08:48:46.474129+0000","last_unstale":"2024-04-21T08:51:10.877416+0000","last_undegraded":"2024-04-21T08:51:10.877416+0000","last_fullsized":"2024-04-21T08:51:10.877416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:40:47.784976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877479+0000","last_change":"2024-04-21T08:49:36.907964+0000","last_active":"2024-04-21T08:51:10.877479+0000","last_peered":"2024-04-21T08:51:10.877479+0000","last_clean":"2024-04-21T08:51:10.877479+0000","last_became_active":"2024-04-21T08:49:36.907595+0000","last_became_peered":"2024-04-21T08:49:36.907595+0000","last_unstale":"2024-04-21T08:51:10.877479+0000","last_undegraded":"2024-04-21T08:51:10.877479+0000","last_fullsized":"2024-04-21T08:51:10.877479+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:14:04.907328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006192+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:51:11.006192+0000","last_peered":"2024-04-21T08:51:11.006192+0000","last_clean":"2024-04-21T08:51:11.006192+0000","last_became_active":"2024-04-21T08:48:46.469389+0000","last_became_peered":"2024-04-21T08:48:46.469389+0000","last_unstale":"2024-04-21T08:51:11.006192+0000","last_undegraded":"2024-04-21T08:51:11.006192+0000","last_fullsized":"2024-04-21T08:51:11.006192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:11:45.824402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063161+0000","last_change":"2024-04-21T08:48:46.470059+0000","last_active":"2024-04-21T08:51:11.063161+0000","last_peered":"2024-04-21T08:51:11.063161+0000","last_clean":"2024-04-21T08:51:11.063161+0000","last_became_active":"2024-04-21T08:48:46.469751+0000","last_became_peered":"2024-04-21T08:48:46.469751+0000","last_unstale":"2024-04-21T08:51:11.063161+0000","last_undegraded":"2024-04-21T08:51:11.063161+0000","last_fullsized":"2024-04-21T08:51:11.063161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:16.317879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.937061+0000","last_change":"2024-04-21T08:48:46.470129+0000","last_active":"2024-04-21T08:51:10.937061+0000","last_peered":"2024-04-21T08:51:10.937061+0000","last_clean":"2024-04-21T08:51:10.937061+0000","last_became_active":"2024-04-21T08:48:46.469763+0000","last_became_peered":"2024-04-21T08:48:46.469763+0000","last_unstale":"2024-04-21T08:51:10.937061+0000","last_undegraded":"2024-04-21T08:51:10.937061+0000","last_fullsized":"2024-04-21T08:51:10.937061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:30:47.731201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832982+0000","last_change":"2024-04-21T08:48:46.465356+0000","last_active":"2024-04-21T08:51:10.832982+0000","last_peered":"2024-04-21T08:51:10.832982+0000","last_clean":"2024-04-21T08:51:10.832982+0000","last_became_active":"2024-04-21T08:48:46.464183+0000","last_became_peered":"2024-04-21T08:48:46.464183+0000","last_unstale":"2024-04-21T08:51:10.832982+0000","last_undegraded":"2024-04-21T08:51:10.832982+0000","last_fullsized":"2024-04-21T08:51:10.832982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:37.533733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832383+0000","last_change":"2024-04-21T08:48:50.767378+0000","last_active":"2024-04-21T08:51:10.832383+0000","last_peered":"2024-04-21T08:51:10.832383+0000","last_clean":"2024-04-21T08:51:10.832383+0000","last_became_active":"2024-04-21T08:48:50.766856+0000","last_became_peered":"2024-04-21T08:48:50.766856+0000","last_unstale":"2024-04-21T08:51:10.832383+0000","last_undegraded":"2024-04-21T08:51:10.832383+0000","last_fullsized":"2024-04-21T08:51:10.832383+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:00.434540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936767+0000","last_change":"2024-04-21T08:48:45.550284+0000","last_active":"2024-04-21T08:51:10.936767+0000","last_peered":"2024-04-21T08:51:10.936767+0000","last_clean":"2024-04-21T08:51:10.936767+0000","last_became_active":"2024-04-21T08:48:45.549940+0000","last_became_peered":"2024-04-21T08:48:45.549940+0000","last_unstale":"2024-04-21T08:51:10.936767+0000","last_undegraded":"2024-04-21T08:51:10.936767+0000","last_fullsized":"2024-04-21T08:51:10.936767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:36:12.343299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828471+0000","last_change":"2024-04-21T08:48:46.469965+0000","last_active":"2024-04-21T08:51:10.828471+0000","last_peered":"2024-04-21T08:51:10.828471+0000","last_clean":"2024-04-21T08:51:10.828471+0000","last_became_active":"2024-04-21T08:48:46.469647+0000","last_became_peered":"2024-04-21T08:48:46.469647+0000","last_unstale":"2024-04-21T08:51:10.828471+0000","last_undegraded":"2024-04-21T08:51:10.828471+0000","last_fullsized":"2024-04-21T08:51:10.828471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:10:28.204554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.037993+0000","last_change":"2024-04-21T08:48:50.771871+0000","last_active":"2024-04-21T08:51:11.037993+0000","last_peered":"2024-04-21T08:51:11.037993+0000","last_clean":"2024-04-21T08:51:11.037993+0000","last_became_active":"2024-04-21T08:48:50.770442+0000","last_became_peered":"2024-04-21T08:48:50.770442+0000","last_unstale":"2024-04-21T08:51:11.037993+0000","last_undegraded":"2024-04-21T08:51:11.037993+0000","last_fullsized":"2024-04-21T08:51:11.037993+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:18:57.813971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828497+0000","last_change":"2024-04-21T08:48:45.566492+0000","last_active":"2024-04-21T08:51:10.828497+0000","last_peered":"2024-04-21T08:51:10.828497+0000","last_clean":"2024-04-21T08:51:10.828497+0000","last_became_active":"2024-04-21T08:48:45.566169+0000","last_became_peered":"2024-04-21T08:48:45.566169+0000","last_unstale":"2024-04-21T08:51:10.828497+0000","last_undegraded":"2024-04-21T08:51:10.828497+0000","last_fullsized":"2024-04-21T08:51:10.828497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:57:49.985664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877476+0000","last_change":"2024-04-21T08:48:46.470866+0000","last_active":"2024-04-21T08:51:10.877476+0000","last_peered":"2024-04-21T08:51:10.877476+0000","last_clean":"2024-04-21T08:51:10.877476+0000","last_became_active":"2024-04-21T08:48:46.470401+0000","last_became_peered":"2024-04-21T08:48:46.470401+0000","last_unstale":"2024-04-21T08:51:10.877476+0000","last_undegraded":"2024-04-21T08:51:10.877476+0000","last_fullsized":"2024-04-21T08:51:10.877476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:53:07.506212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038124+0000","last_change":"2024-04-21T08:48:50.772339+0000","last_active":"2024-04-21T08:51:11.038124+0000","last_peered":"2024-04-21T08:51:11.038124+0000","last_clean":"2024-04-21T08:51:11.038124+0000","last_became_active":"2024-04-21T08:48:50.771946+0000","last_became_peered":"2024-04-21T08:48:50.771946+0000","last_unstale":"2024-04-21T08:51:11.038124+0000","last_undegraded":"2024-04-21T08:51:11.038124+0000","last_fullsized":"2024-04-21T08:51:11.038124+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:03:56.994581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939819+0000","last_change":"2024-04-21T08:48:45.553946+0000","last_active":"2024-04-21T08:51:10.939819+0000","last_peered":"2024-04-21T08:51:10.939819+0000","last_clean":"2024-04-21T08:51:10.939819+0000","last_became_active":"2024-04-21T08:48:45.553410+0000","last_became_peered":"2024-04-21T08:48:45.553410+0000","last_unstale":"2024-04-21T08:51:10.939819+0000","last_undegraded":"2024-04-21T08:51:10.939819+0000","last_fullsized":"2024-04-21T08:51:10.939819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:24:02.387250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006432+0000","last_change":"2024-04-21T08:48:46.471815+0000","last_active":"2024-04-21T08:51:11.006432+0000","last_peered":"2024-04-21T08:51:11.006432+0000","last_clean":"2024-04-21T08:51:11.006432+0000","last_became_active":"2024-04-21T08:48:46.471492+0000","last_became_peered":"2024-04-21T08:48:46.471492+0000","last_unstale":"2024-04-21T08:51:11.006432+0000","last_undegraded":"2024-04-21T08:51:11.006432+0000","last_fullsized":"2024-04-21T08:51:11.006432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:26.051166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038027+0000","last_change":"2024-04-21T08:48:50.767916+0000","last_active":"2024-04-21T08:51:11.038027+0000","last_peered":"2024-04-21T08:51:11.038027+0000","last_clean":"2024-04-21T08:51:11.038027+0000","last_became_active":"2024-04-21T08:48:50.767326+0000","last_became_peered":"2024-04-21T08:48:50.767326+0000","last_unstale":"2024-04-21T08:51:11.038027+0000","last_undegraded":"2024-04-21T08:51:11.038027+0000","last_fullsized":"2024-04-21T08:51:11.038027+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:05:02.185369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939335+0000","last_change":"2024-04-21T08:48:45.549982+0000","last_active":"2024-04-21T08:51:10.939335+0000","last_peered":"2024-04-21T08:51:10.939335+0000","last_clean":"2024-04-21T08:51:10.939335+0000","last_became_active":"2024-04-21T08:48:45.549611+0000","last_became_peered":"2024-04-21T08:48:45.549611+0000","last_unstale":"2024-04-21T08:51:10.939335+0000","last_undegraded":"2024-04-21T08:51:10.939335+0000","last_fullsized":"2024-04-21T08:51:10.939335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:02:46.035388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945903+0000","last_change":"2024-04-21T08:48:46.459068+0000","last_active":"2024-04-21T08:51:10.945903+0000","last_peered":"2024-04-21T08:51:10.945903+0000","last_clean":"2024-04-21T08:51:10.945903+0000","last_became_active":"2024-04-21T08:48:46.458686+0000","last_became_peered":"2024-04-21T08:48:46.458686+0000","last_unstale":"2024-04-21T08:51:10.945903+0000","last_undegraded":"2024-04-21T08:51:10.945903+0000","last_fullsized":"2024-04-21T08:51:10.945903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:31.373451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832831+0000","last_change":"2024-04-21T08:48:50.761392+0000","last_active":"2024-04-21T08:51:10.832831+0000","last_peered":"2024-04-21T08:51:10.832831+0000","last_clean":"2024-04-21T08:51:10.832831+0000","last_became_active":"2024-04-21T08:48:50.760917+0000","last_became_peered":"2024-04-21T08:48:50.760917+0000","last_unstale":"2024-04-21T08:51:10.832831+0000","last_undegraded":"2024-04-21T08:51:10.832831+0000","last_fullsized":"2024-04-21T08:51:10.832831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:36:19.070502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'8","reported_seq":175,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.027752+0000","last_change":"2024-04-21T08:48:45.553686+0000","last_active":"2024-04-21T08:51:20.027752+0000","last_peered":"2024-04-21T08:51:20.027752+0000","last_clean":"2024-04-21T08:51:20.027752+0000","last_became_active":"2024-04-21T08:48:45.553322+0000","last_became_peered":"2024-04-21T08:48:45.553322+0000","last_unstale":"2024-04-21T08:51:20.027752+0000","last_undegraded":"2024-04-21T08:51:20.027752+0000","last_fullsized":"2024-04-21T08:51:20.027752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:15:24.320704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1712,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":133,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939678+0000","last_change":"2024-04-21T08:48:46.474555+0000","last_active":"2024-04-21T08:51:10.939678+0000","last_peered":"2024-04-21T08:51:10.939678+0000","last_clean":"2024-04-21T08:51:10.939678+0000","last_became_active":"2024-04-21T08:48:46.474185+0000","last_became_peered":"2024-04-21T08:48:46.474185+0000","last_unstale":"2024-04-21T08:51:10.939678+0000","last_undegraded":"2024-04-21T08:51:10.939678+0000","last_fullsized":"2024-04-21T08:51:10.939678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:52.884994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832578+0000","last_change":"2024-04-21T08:48:50.763181+0000","last_active":"2024-04-21T08:51:10.832578+0000","last_peered":"2024-04-21T08:51:10.832578+0000","last_clean":"2024-04-21T08:51:10.832578+0000","last_became_active":"2024-04-21T08:48:50.762718+0000","last_became_peered":"2024-04-21T08:48:50.762718+0000","last_unstale":"2024-04-21T08:51:10.832578+0000","last_undegraded":"2024-04-21T08:51:10.832578+0000","last_fullsized":"2024-04-21T08:51:10.832578+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:16:23.204605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829837+0000","last_change":"2024-04-21T08:48:45.547768+0000","last_active":"2024-04-21T08:51:10.829837+0000","last_peered":"2024-04-21T08:51:10.829837+0000","last_clean":"2024-04-21T08:51:10.829837+0000","last_became_active":"2024-04-21T08:48:45.547456+0000","last_became_peered":"2024-04-21T08:48:45.547456+0000","last_unstale":"2024-04-21T08:51:10.829837+0000","last_undegraded":"2024-04-21T08:51:10.829837+0000","last_fullsized":"2024-04-21T08:51:10.829837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:23.502799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945877+0000","last_change":"2024-04-21T08:48:46.467988+0000","last_active":"2024-04-21T08:51:10.945877+0000","last_peered":"2024-04-21T08:51:10.945877+0000","last_clean":"2024-04-21T08:51:10.945877+0000","last_became_active":"2024-04-21T08:48:46.465212+0000","last_became_peered":"2024-04-21T08:48:46.465212+0000","last_unstale":"2024-04-21T08:51:10.945877+0000","last_undegraded":"2024-04-21T08:51:10.945877+0000","last_fullsized":"2024-04-21T08:51:10.945877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:14:41.743506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063079+0000","last_change":"2024-04-21T08:50:37.350070+0000","last_active":"2024-04-21T08:51:11.063079+0000","last_peered":"2024-04-21T08:51:11.063079+0000","last_clean":"2024-04-21T08:51:11.063079+0000","last_became_active":"2024-04-21T08:50:37.349622+0000","last_became_peered":"2024-04-21T08:50:37.349622+0000","last_unstale":"2024-04-21T08:51:11.063079+0000","last_undegraded":"2024-04-21T08:51:11.063079+0000","last_fullsized":"2024-04-21T08:51:11.063079+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:55.050244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.035911+0000","last_change":"2024-04-21T08:48:45.545615+0000","last_active":"2024-04-21T08:51:11.035911+0000","last_peered":"2024-04-21T08:51:11.035911+0000","last_clean":"2024-04-21T08:51:11.035911+0000","last_became_active":"2024-04-21T08:48:45.545280+0000","last_became_peered":"2024-04-21T08:48:45.545280+0000","last_unstale":"2024-04-21T08:51:11.035911+0000","last_undegraded":"2024-04-21T08:51:11.035911+0000","last_fullsized":"2024-04-21T08:51:11.035911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:50:00.470903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828078+0000","last_change":"2024-04-21T08:48:46.477326+0000","last_active":"2024-04-21T08:51:10.828078+0000","last_peered":"2024-04-21T08:51:10.828078+0000","last_clean":"2024-04-21T08:51:10.828078+0000","last_became_active":"2024-04-21T08:48:46.476914+0000","last_became_peered":"2024-04-21T08:48:46.476914+0000","last_unstale":"2024-04-21T08:51:10.828078+0000","last_undegraded":"2024-04-21T08:51:10.828078+0000","last_fullsized":"2024-04-21T08:51:10.828078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:04.312061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877774+0000","last_change":"2024-04-21T08:48:50.767429+0000","last_active":"2024-04-21T08:51:10.877774+0000","last_peered":"2024-04-21T08:51:10.877774+0000","last_clean":"2024-04-21T08:51:10.877774+0000","last_became_active":"2024-04-21T08:48:50.767008+0000","last_became_peered":"2024-04-21T08:48:50.767008+0000","last_unstale":"2024-04-21T08:51:10.877774+0000","last_undegraded":"2024-04-21T08:51:10.877774+0000","last_fullsized":"2024-04-21T08:51:10.877774+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:26.865571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945328+0000","last_change":"2024-04-21T08:48:45.553170+0000","last_active":"2024-04-21T08:51:10.945328+0000","last_peered":"2024-04-21T08:51:10.945328+0000","last_clean":"2024-04-21T08:51:10.945328+0000","last_became_active":"2024-04-21T08:48:45.552645+0000","last_became_peered":"2024-04-21T08:48:45.552645+0000","last_unstale":"2024-04-21T08:51:10.945328+0000","last_undegraded":"2024-04-21T08:51:10.945328+0000","last_fullsized":"2024-04-21T08:51:10.945328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:32:07.217212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936256+0000","last_change":"2024-04-21T08:48:46.478101+0000","last_active":"2024-04-21T08:51:10.936256+0000","last_peered":"2024-04-21T08:51:10.936256+0000","last_clean":"2024-04-21T08:51:10.936256+0000","last_became_active":"2024-04-21T08:48:46.477725+0000","last_became_peered":"2024-04-21T08:48:46.477725+0000","last_unstale":"2024-04-21T08:51:10.936256+0000","last_undegraded":"2024-04-21T08:51:10.936256+0000","last_fullsized":"2024-04-21T08:51:10.936256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:37:55.429569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828279+0000","last_change":"2024-04-21T08:48:50.762899+0000","last_active":"2024-04-21T08:51:10.828279+0000","last_peered":"2024-04-21T08:51:10.828279+0000","last_clean":"2024-04-21T08:51:10.828279+0000","last_became_active":"2024-04-21T08:48:50.762475+0000","last_became_peered":"2024-04-21T08:48:50.762475+0000","last_unstale":"2024-04-21T08:51:10.828279+0000","last_undegraded":"2024-04-21T08:51:10.828279+0000","last_fullsized":"2024-04-21T08:51:10.828279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:27.540414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":167,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.027197+0000","last_change":"2024-04-21T08:48:45.550151+0000","last_active":"2024-04-21T08:51:20.027197+0000","last_peered":"2024-04-21T08:51:20.027197+0000","last_clean":"2024-04-21T08:51:20.027197+0000","last_became_active":"2024-04-21T08:48:45.549829+0000","last_became_peered":"2024-04-21T08:48:45.549829+0000","last_unstale":"2024-04-21T08:51:20.027197+0000","last_undegraded":"2024-04-21T08:51:20.027197+0000","last_fullsized":"2024-04-21T08:51:20.027197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:57:14.061189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":129,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006968+0000","last_change":"2024-04-21T08:48:46.472072+0000","last_active":"2024-04-21T08:51:11.006968+0000","last_peered":"2024-04-21T08:51:11.006968+0000","last_clean":"2024-04-21T08:51:11.006968+0000","last_became_active":"2024-04-21T08:48:46.471660+0000","last_became_peered":"2024-04-21T08:48:46.471660+0000","last_unstale":"2024-04-21T08:51:11.006968+0000","last_undegraded":"2024-04-21T08:51:11.006968+0000","last_fullsized":"2024-04-21T08:51:11.006968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:30:11.843870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936126+0000","last_change":"2024-04-21T08:48:50.773021+0000","last_active":"2024-04-21T08:51:10.936126+0000","last_peered":"2024-04-21T08:51:10.936126+0000","last_clean":"2024-04-21T08:51:10.936126+0000","last_became_active":"2024-04-21T08:48:50.771746+0000","last_became_peered":"2024-04-21T08:48:50.771746+0000","last_unstale":"2024-04-21T08:51:10.936126+0000","last_undegraded":"2024-04-21T08:51:10.936126+0000","last_fullsized":"2024-04-21T08:51:10.936126+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:33:21.415967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063066+0000","last_change":"2024-04-21T08:48:45.550376+0000","last_active":"2024-04-21T08:51:11.063066+0000","last_peered":"2024-04-21T08:51:11.063066+0000","last_clean":"2024-04-21T08:51:11.063066+0000","last_became_active":"2024-04-21T08:48:45.549594+0000","last_became_peered":"2024-04-21T08:48:45.549594+0000","last_unstale":"2024-04-21T08:51:11.063066+0000","last_undegraded":"2024-04-21T08:51:11.063066+0000","last_fullsized":"2024-04-21T08:51:11.063066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:26:27.046963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877696+0000","last_change":"2024-04-21T08:48:46.469499+0000","last_active":"2024-04-21T08:51:10.877696+0000","last_peered":"2024-04-21T08:51:10.877696+0000","last_clean":"2024-04-21T08:51:10.877696+0000","last_became_active":"2024-04-21T08:48:46.469124+0000","last_became_peered":"2024-04-21T08:48:46.469124+0000","last_unstale":"2024-04-21T08:51:10.877696+0000","last_undegraded":"2024-04-21T08:51:10.877696+0000","last_fullsized":"2024-04-21T08:51:10.877696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:58.708916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038260+0000","last_change":"2024-04-21T08:50:37.352313+0000","last_active":"2024-04-21T08:51:11.038260+0000","last_peered":"2024-04-21T08:51:11.038260+0000","last_clean":"2024-04-21T08:51:11.038260+0000","last_became_active":"2024-04-21T08:50:37.351775+0000","last_became_peered":"2024-04-21T08:50:37.351775+0000","last_unstale":"2024-04-21T08:51:11.038260+0000","last_undegraded":"2024-04-21T08:51:11.038260+0000","last_fullsized":"2024-04-21T08:51:11.038260+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:12:51.239368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,10],"acting":[4,1,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945985+0000","last_change":"2024-04-21T08:48:45.559319+0000","last_active":"2024-04-21T08:51:10.945985+0000","last_peered":"2024-04-21T08:51:10.945985+0000","last_clean":"2024-04-21T08:51:10.945985+0000","last_became_active":"2024-04-21T08:48:45.558895+0000","last_became_peered":"2024-04-21T08:48:45.558895+0000","last_unstale":"2024-04-21T08:51:10.945985+0000","last_undegraded":"2024-04-21T08:51:10.945985+0000","last_fullsized":"2024-04-21T08:51:10.945985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:51:01.878169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006563+0000","last_change":"2024-04-21T08:48:46.466110+0000","last_active":"2024-04-21T08:51:11.006563+0000","last_peered":"2024-04-21T08:51:11.006563+0000","last_clean":"2024-04-21T08:51:11.006563+0000","last_became_active":"2024-04-21T08:48:46.464374+0000","last_became_peered":"2024-04-21T08:48:46.464374+0000","last_unstale":"2024-04-21T08:51:11.006563+0000","last_undegraded":"2024-04-21T08:51:11.006563+0000","last_fullsized":"2024-04-21T08:51:11.006563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:16:26.307051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936310+0000","last_change":"2024-04-21T08:48:50.759976+0000","last_active":"2024-04-21T08:51:10.936310+0000","last_peered":"2024-04-21T08:51:10.936310+0000","last_clean":"2024-04-21T08:51:10.936310+0000","last_became_active":"2024-04-21T08:48:50.759596+0000","last_became_peered":"2024-04-21T08:48:50.759596+0000","last_unstale":"2024-04-21T08:51:10.936310+0000","last_undegraded":"2024-04-21T08:51:10.936310+0000","last_fullsized":"2024-04-21T08:51:10.936310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:23:09.678388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'42","reported_seq":347,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:21.028496+0000","last_change":"2024-04-21T08:48:45.549228+0000","last_active":"2024-04-21T08:51:21.028496+0000","last_peered":"2024-04-21T08:51:21.028496+0000","last_clean":"2024-04-21T08:51:21.028496+0000","last_became_active":"2024-04-21T08:48:45.548865+0000","last_became_peered":"2024-04-21T08:48:45.548865+0000","last_unstale":"2024-04-21T08:51:21.028496+0000","last_undegraded":"2024-04-21T08:51:21.028496+0000","last_fullsized":"2024-04-21T08:51:21.028496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:11:28.620968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71920,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":271,"num_read_kb":207,"num_write":43,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939637+0000","last_change":"2024-04-21T08:48:46.470322+0000","last_active":"2024-04-21T08:51:10.939637+0000","last_peered":"2024-04-21T08:51:10.939637+0000","last_clean":"2024-04-21T08:51:10.939637+0000","last_became_active":"2024-04-21T08:48:46.467225+0000","last_became_peered":"2024-04-21T08:48:46.467225+0000","last_unstale":"2024-04-21T08:51:10.939637+0000","last_undegraded":"2024-04-21T08:51:10.939637+0000","last_fullsized":"2024-04-21T08:51:10.939637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:31:17.390926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.946326+0000","last_change":"2024-04-21T08:48:50.770982+0000","last_active":"2024-04-21T08:51:10.946326+0000","last_peered":"2024-04-21T08:51:10.946326+0000","last_clean":"2024-04-21T08:51:10.946326+0000","last_became_active":"2024-04-21T08:48:50.770312+0000","last_became_peered":"2024-04-21T08:48:50.770312+0000","last_unstale":"2024-04-21T08:51:10.946326+0000","last_undegraded":"2024-04-21T08:51:10.946326+0000","last_fullsized":"2024-04-21T08:51:10.946326+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:41:33.027351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832599+0000","last_change":"2024-04-21T08:48:45.556134+0000","last_active":"2024-04-21T08:51:10.832599+0000","last_peered":"2024-04-21T08:51:10.832599+0000","last_clean":"2024-04-21T08:51:10.832599+0000","last_became_active":"2024-04-21T08:48:45.555818+0000","last_became_peered":"2024-04-21T08:48:45.555818+0000","last_unstale":"2024-04-21T08:51:10.832599+0000","last_undegraded":"2024-04-21T08:51:10.832599+0000","last_fullsized":"2024-04-21T08:51:10.832599+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:00:08.083783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036365+0000","last_change":"2024-04-21T08:49:36.913151+0000","last_active":"2024-04-21T08:51:11.036365+0000","last_peered":"2024-04-21T08:51:11.036365+0000","last_clean":"2024-04-21T08:51:11.036365+0000","last_became_active":"2024-04-21T08:49:36.912810+0000","last_became_peered":"2024-04-21T08:49:36.912810+0000","last_unstale":"2024-04-21T08:51:11.036365+0000","last_undegraded":"2024-04-21T08:51:11.036365+0000","last_fullsized":"2024-04-21T08:51:11.036365+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:17:32.887163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062954+0000","last_change":"2024-04-21T08:48:50.765967+0000","last_active":"2024-04-21T08:51:11.062954+0000","last_peered":"2024-04-21T08:51:11.062954+0000","last_clean":"2024-04-21T08:51:11.062954+0000","last_became_active":"2024-04-21T08:48:50.765557+0000","last_became_peered":"2024-04-21T08:48:50.765557+0000","last_unstale":"2024-04-21T08:51:11.062954+0000","last_undegraded":"2024-04-21T08:51:11.062954+0000","last_fullsized":"2024-04-21T08:51:11.062954+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:18:52.044058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945823+0000","last_change":"2024-04-21T08:48:45.559939+0000","last_active":"2024-04-21T08:51:10.945823+0000","last_peered":"2024-04-21T08:51:10.945823+0000","last_clean":"2024-04-21T08:51:10.945823+0000","last_became_active":"2024-04-21T08:48:45.559635+0000","last_became_peered":"2024-04-21T08:48:45.559635+0000","last_unstale":"2024-04-21T08:51:10.945823+0000","last_undegraded":"2024-04-21T08:51:10.945823+0000","last_fullsized":"2024-04-21T08:51:10.945823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:17.872806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062944+0000","last_change":"2024-04-21T08:48:46.464878+0000","last_active":"2024-04-21T08:51:11.062944+0000","last_peered":"2024-04-21T08:51:11.062944+0000","last_clean":"2024-04-21T08:51:11.062944+0000","last_became_active":"2024-04-21T08:48:46.464552+0000","last_became_peered":"2024-04-21T08:48:46.464552+0000","last_unstale":"2024-04-21T08:51:11.062944+0000","last_undegraded":"2024-04-21T08:51:11.062944+0000","last_fullsized":"2024-04-21T08:51:11.062944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:46:10.558693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939454+0000","last_change":"2024-04-21T08:50:37.351314+0000","last_active":"2024-04-21T08:51:10.939454+0000","last_peered":"2024-04-21T08:51:10.939454+0000","last_clean":"2024-04-21T08:51:10.939454+0000","last_became_active":"2024-04-21T08:50:37.350956+0000","last_became_peered":"2024-04-21T08:50:37.350956+0000","last_unstale":"2024-04-21T08:51:10.939454+0000","last_undegraded":"2024-04-21T08:51:10.939454+0000","last_fullsized":"2024-04-21T08:51:10.939454+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:01.802427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"94'7","reported_seq":175,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.050635+0000","last_change":"2024-04-21T08:48:45.548685+0000","last_active":"2024-04-21T08:51:20.050635+0000","last_peered":"2024-04-21T08:51:20.050635+0000","last_clean":"2024-04-21T08:51:20.050635+0000","last_became_active":"2024-04-21T08:48:45.548372+0000","last_became_peered":"2024-04-21T08:48:45.548372+0000","last_unstale":"2024-04-21T08:51:20.050635+0000","last_undegraded":"2024-04-21T08:51:20.050635+0000","last_fullsized":"2024-04-21T08:51:20.050635+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:24:44.780297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":135,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062806+0000","last_change":"2024-04-21T08:48:46.471541+0000","last_active":"2024-04-21T08:51:11.062806+0000","last_peered":"2024-04-21T08:51:11.062806+0000","last_clean":"2024-04-21T08:51:11.062806+0000","last_became_active":"2024-04-21T08:48:46.471223+0000","last_became_peered":"2024-04-21T08:48:46.471223+0000","last_unstale":"2024-04-21T08:51:11.062806+0000","last_undegraded":"2024-04-21T08:51:11.062806+0000","last_fullsized":"2024-04-21T08:51:11.062806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:24:50.867905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036983+0000","last_change":"2024-04-21T08:48:50.773941+0000","last_active":"2024-04-21T08:51:11.036983+0000","last_peered":"2024-04-21T08:51:11.036983+0000","last_clean":"2024-04-21T08:51:11.036983+0000","last_became_active":"2024-04-21T08:48:50.772081+0000","last_became_peered":"2024-04-21T08:48:50.772081+0000","last_unstale":"2024-04-21T08:51:11.036983+0000","last_undegraded":"2024-04-21T08:51:11.036983+0000","last_fullsized":"2024-04-21T08:51:11.036983+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:00:04.438944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062783+0000","last_change":"2024-04-21T08:49:36.923090+0000","last_active":"2024-04-21T08:51:11.062783+0000","last_peered":"2024-04-21T08:51:11.062783+0000","last_clean":"2024-04-21T08:51:11.062783+0000","last_became_active":"2024-04-21T08:49:36.908075+0000","last_became_peered":"2024-04-21T08:49:36.908075+0000","last_unstale":"2024-04-21T08:51:11.062783+0000","last_undegraded":"2024-04-21T08:51:11.062783+0000","last_fullsized":"2024-04-21T08:51:11.062783+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:41.683372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945471+0000","last_change":"2024-04-21T08:48:46.465559+0000","last_active":"2024-04-21T08:51:10.945471+0000","last_peered":"2024-04-21T08:51:10.945471+0000","last_clean":"2024-04-21T08:51:10.945471+0000","last_became_active":"2024-04-21T08:48:46.465125+0000","last_became_peered":"2024-04-21T08:48:46.465125+0000","last_unstale":"2024-04-21T08:51:10.945471+0000","last_undegraded":"2024-04-21T08:51:10.945471+0000","last_fullsized":"2024-04-21T08:51:10.945471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:52:38.423925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945406+0000","last_change":"2024-04-21T08:48:50.763609+0000","last_active":"2024-04-21T08:51:10.945406+0000","last_peered":"2024-04-21T08:51:10.945406+0000","last_clean":"2024-04-21T08:51:10.945406+0000","last_became_active":"2024-04-21T08:48:50.763144+0000","last_became_peered":"2024-04-21T08:48:50.763144+0000","last_unstale":"2024-04-21T08:51:10.945406+0000","last_undegraded":"2024-04-21T08:51:10.945406+0000","last_fullsized":"2024-04-21T08:51:10.945406+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:26:18.611235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939721+0000","last_change":"2024-04-21T08:48:45.549912+0000","last_active":"2024-04-21T08:51:10.939721+0000","last_peered":"2024-04-21T08:51:10.939721+0000","last_clean":"2024-04-21T08:51:10.939721+0000","last_became_active":"2024-04-21T08:48:45.549585+0000","last_became_peered":"2024-04-21T08:48:45.549585+0000","last_unstale":"2024-04-21T08:51:10.939721+0000","last_undegraded":"2024-04-21T08:51:10.939721+0000","last_fullsized":"2024-04-21T08:51:10.939721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:58.287402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063099+0000","last_change":"2024-04-21T08:48:46.462746+0000","last_active":"2024-04-21T08:51:11.063099+0000","last_peered":"2024-04-21T08:51:11.063099+0000","last_clean":"2024-04-21T08:51:11.063099+0000","last_became_active":"2024-04-21T08:48:46.461602+0000","last_became_peered":"2024-04-21T08:48:46.461602+0000","last_unstale":"2024-04-21T08:51:11.063099+0000","last_undegraded":"2024-04-21T08:51:11.063099+0000","last_fullsized":"2024-04-21T08:51:11.063099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:06:52.198634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939308+0000","last_change":"2024-04-21T08:48:50.770770+0000","last_active":"2024-04-21T08:51:10.939308+0000","last_peered":"2024-04-21T08:51:10.939308+0000","last_clean":"2024-04-21T08:51:10.939308+0000","last_became_active":"2024-04-21T08:48:50.770326+0000","last_became_peered":"2024-04-21T08:48:50.770326+0000","last_unstale":"2024-04-21T08:51:10.939308+0000","last_undegraded":"2024-04-21T08:51:10.939308+0000","last_fullsized":"2024-04-21T08:51:10.939308+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:30:38.325155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936408+0000","last_change":"2024-04-21T08:48:45.558867+0000","last_active":"2024-04-21T08:51:10.936408+0000","last_peered":"2024-04-21T08:51:10.936408+0000","last_clean":"2024-04-21T08:51:10.936408+0000","last_became_active":"2024-04-21T08:48:45.558492+0000","last_became_peered":"2024-04-21T08:48:45.558492+0000","last_unstale":"2024-04-21T08:51:10.936408+0000","last_undegraded":"2024-04-21T08:51:10.936408+0000","last_fullsized":"2024-04-21T08:51:10.936408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:44:55.156404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.830180+0000","last_change":"2024-04-21T08:48:46.466461+0000","last_active":"2024-04-21T08:51:10.830180+0000","last_peered":"2024-04-21T08:51:10.830180+0000","last_clean":"2024-04-21T08:51:10.830180+0000","last_became_active":"2024-04-21T08:48:46.466035+0000","last_became_peered":"2024-04-21T08:48:46.466035+0000","last_unstale":"2024-04-21T08:51:10.830180+0000","last_undegraded":"2024-04-21T08:51:10.830180+0000","last_fullsized":"2024-04-21T08:51:10.830180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:47:32.307753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036555+0000","last_change":"2024-04-21T08:48:50.767655+0000","last_active":"2024-04-21T08:51:11.036555+0000","last_peered":"2024-04-21T08:51:11.036555+0000","last_clean":"2024-04-21T08:51:11.036555+0000","last_became_active":"2024-04-21T08:48:50.767305+0000","last_became_peered":"2024-04-21T08:48:50.767305+0000","last_unstale":"2024-04-21T08:51:11.036555+0000","last_undegraded":"2024-04-21T08:51:11.036555+0000","last_fullsized":"2024-04-21T08:51:11.036555+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:42:31.210126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062757+0000","last_change":"2024-04-21T08:48:45.552952+0000","last_active":"2024-04-21T08:51:11.062757+0000","last_peered":"2024-04-21T08:51:11.062757+0000","last_clean":"2024-04-21T08:51:11.062757+0000","last_became_active":"2024-04-21T08:48:45.552556+0000","last_became_peered":"2024-04-21T08:48:45.552556+0000","last_unstale":"2024-04-21T08:51:11.062757+0000","last_undegraded":"2024-04-21T08:51:11.062757+0000","last_fullsized":"2024-04-21T08:51:11.062757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:25:03.428801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.829859+0000","last_change":"2024-04-21T08:48:46.469020+0000","last_active":"2024-04-21T08:51:10.829859+0000","last_peered":"2024-04-21T08:51:10.829859+0000","last_clean":"2024-04-21T08:51:10.829859+0000","last_became_active":"2024-04-21T08:48:46.468631+0000","last_became_peered":"2024-04-21T08:48:46.468631+0000","last_unstale":"2024-04-21T08:51:10.829859+0000","last_undegraded":"2024-04-21T08:51:10.829859+0000","last_fullsized":"2024-04-21T08:51:10.829859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:56:08.559516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036969+0000","last_change":"2024-04-21T08:48:50.773942+0000","last_active":"2024-04-21T08:51:11.036969+0000","last_peered":"2024-04-21T08:51:11.036969+0000","last_clean":"2024-04-21T08:51:11.036969+0000","last_became_active":"2024-04-21T08:48:50.771876+0000","last_became_peered":"2024-04-21T08:48:50.771876+0000","last_unstale":"2024-04-21T08:51:11.036969+0000","last_undegraded":"2024-04-21T08:51:11.036969+0000","last_fullsized":"2024-04-21T08:51:11.036969+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:19.656118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945411+0000","last_change":"2024-04-21T08:48:46.462727+0000","last_active":"2024-04-21T08:51:10.945411+0000","last_peered":"2024-04-21T08:51:10.945411+0000","last_clean":"2024-04-21T08:51:10.945411+0000","last_became_active":"2024-04-21T08:48:46.461669+0000","last_became_peered":"2024-04-21T08:48:46.461669+0000","last_unstale":"2024-04-21T08:51:10.945411+0000","last_undegraded":"2024-04-21T08:51:10.945411+0000","last_fullsized":"2024-04-21T08:51:10.945411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:20:27.290790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.049963+0000","last_change":"2024-04-21T08:48:45.558541+0000","last_active":"2024-04-21T08:51:20.049963+0000","last_peered":"2024-04-21T08:51:20.049963+0000","last_clean":"2024-04-21T08:51:20.049963+0000","last_became_active":"2024-04-21T08:48:45.558186+0000","last_became_peered":"2024-04-21T08:48:45.558186+0000","last_unstale":"2024-04-21T08:51:20.049963+0000","last_undegraded":"2024-04-21T08:51:20.049963+0000","last_fullsized":"2024-04-21T08:51:20.049963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:31:02.919144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":129,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062978+0000","last_change":"2024-04-21T08:48:50.770838+0000","last_active":"2024-04-21T08:51:11.062978+0000","last_peered":"2024-04-21T08:51:11.062978+0000","last_clean":"2024-04-21T08:51:11.062978+0000","last_became_active":"2024-04-21T08:48:50.770447+0000","last_became_peered":"2024-04-21T08:48:50.770447+0000","last_unstale":"2024-04-21T08:51:11.062978+0000","last_undegraded":"2024-04-21T08:51:11.062978+0000","last_fullsized":"2024-04-21T08:51:11.062978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:48:21.010284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828090+0000","last_change":"2024-04-21T08:48:46.478145+0000","last_active":"2024-04-21T08:51:10.828090+0000","last_peered":"2024-04-21T08:51:10.828090+0000","last_clean":"2024-04-21T08:51:10.828090+0000","last_became_active":"2024-04-21T08:48:46.477813+0000","last_became_peered":"2024-04-21T08:48:46.477813+0000","last_unstale":"2024-04-21T08:51:10.828090+0000","last_undegraded":"2024-04-21T08:51:10.828090+0000","last_fullsized":"2024-04-21T08:51:10.828090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:21:15.917619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038191+0000","last_change":"2024-04-21T08:48:45.545269+0000","last_active":"2024-04-21T08:51:11.038191+0000","last_peered":"2024-04-21T08:51:11.038191+0000","last_clean":"2024-04-21T08:51:11.038191+0000","last_became_active":"2024-04-21T08:48:45.544887+0000","last_became_peered":"2024-04-21T08:48:45.544887+0000","last_unstale":"2024-04-21T08:51:11.038191+0000","last_undegraded":"2024-04-21T08:51:11.038191+0000","last_fullsized":"2024-04-21T08:51:11.038191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:37:27.222570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"94'148","reported_seq":411,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:12.946003+0000","last_change":"2024-04-21T08:47:34.198161+0000","last_active":"2024-04-21T08:51:12.946003+0000","last_peered":"2024-04-21T08:51:12.946003+0000","last_clean":"2024-04-21T08:51:12.946003+0000","last_became_active":"2024-04-21T08:47:34.197654+0000","last_became_peered":"2024-04-21T08:47:34.197654+0000","last_unstale":"2024-04-21T08:51:12.946003+0000","last_undegraded":"2024-04-21T08:51:12.946003+0000","last_fullsized":"2024-04-21T08:51:12.946003+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:44:05.493464+0000","last_clean_scrub_stamp":"2024-04-21T08:44:05.493464+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:58:58.295783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945722+0000","last_change":"2024-04-21T08:48:50.770826+0000","last_active":"2024-04-21T08:51:10.945722+0000","last_peered":"2024-04-21T08:51:10.945722+0000","last_clean":"2024-04-21T08:51:10.945722+0000","last_became_active":"2024-04-21T08:48:50.770372+0000","last_became_peered":"2024-04-21T08:48:50.770372+0000","last_unstale":"2024-04-21T08:51:10.945722+0000","last_undegraded":"2024-04-21T08:51:10.945722+0000","last_fullsized":"2024-04-21T08:51:10.945722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:59.637874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.035696+0000","last_change":"2024-04-21T08:48:46.459330+0000","last_active":"2024-04-21T08:51:11.035696+0000","last_peered":"2024-04-21T08:51:11.035696+0000","last_clean":"2024-04-21T08:51:11.035696+0000","last_became_active":"2024-04-21T08:48:46.458963+0000","last_became_peered":"2024-04-21T08:48:46.458963+0000","last_unstale":"2024-04-21T08:51:11.035696+0000","last_undegraded":"2024-04-21T08:51:11.035696+0000","last_fullsized":"2024-04-21T08:51:11.035696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:10.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062640+0000","last_change":"2024-04-21T08:49:36.917854+0000","last_active":"2024-04-21T08:51:11.062640+0000","last_peered":"2024-04-21T08:51:11.062640+0000","last_clean":"2024-04-21T08:51:11.062640+0000","last_became_active":"2024-04-21T08:49:36.915390+0000","last_became_peered":"2024-04-21T08:49:36.915390+0000","last_unstale":"2024-04-21T08:51:11.062640+0000","last_undegraded":"2024-04-21T08:51:11.062640+0000","last_fullsized":"2024-04-21T08:51:11.062640+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:45.920752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062798+0000","last_change":"2024-04-21T08:48:50.765339+0000","last_active":"2024-04-21T08:51:11.062798+0000","last_peered":"2024-04-21T08:51:11.062798+0000","last_clean":"2024-04-21T08:51:11.062798+0000","last_became_active":"2024-04-21T08:48:50.764858+0000","last_became_peered":"2024-04-21T08:48:50.764858+0000","last_unstale":"2024-04-21T08:51:11.062798+0000","last_undegraded":"2024-04-21T08:51:11.062798+0000","last_fullsized":"2024-04-21T08:51:11.062798+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:26.075041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939660+0000","last_change":"2024-04-21T08:48:45.557768+0000","last_active":"2024-04-21T08:51:10.939660+0000","last_peered":"2024-04-21T08:51:10.939660+0000","last_clean":"2024-04-21T08:51:10.939660+0000","last_became_active":"2024-04-21T08:48:45.557448+0000","last_became_peered":"2024-04-21T08:48:45.557448+0000","last_unstale":"2024-04-21T08:51:10.939660+0000","last_undegraded":"2024-04-21T08:51:10.939660+0000","last_fullsized":"2024-04-21T08:51:10.939660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:57:09.063913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036820+0000","last_change":"2024-04-21T08:48:46.459425+0000","last_active":"2024-04-21T08:51:11.036820+0000","last_peered":"2024-04-21T08:51:11.036820+0000","last_clean":"2024-04-21T08:51:11.036820+0000","last_became_active":"2024-04-21T08:48:46.459020+0000","last_became_peered":"2024-04-21T08:48:46.459020+0000","last_unstale":"2024-04-21T08:51:11.036820+0000","last_undegraded":"2024-04-21T08:51:11.036820+0000","last_fullsized":"2024-04-21T08:51:11.036820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:08:39.042825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828005+0000","last_change":"2024-04-21T08:48:50.765740+0000","last_active":"2024-04-21T08:51:10.828005+0000","last_peered":"2024-04-21T08:51:10.828005+0000","last_clean":"2024-04-21T08:51:10.828005+0000","last_became_active":"2024-04-21T08:48:50.765359+0000","last_became_peered":"2024-04-21T08:48:50.765359+0000","last_unstale":"2024-04-21T08:51:10.828005+0000","last_undegraded":"2024-04-21T08:51:10.828005+0000","last_fullsized":"2024-04-21T08:51:10.828005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:34:42.716165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'98","reported_seq":224,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.520533+0000","last_change":"2024-04-21T08:48:45.564532+0000","last_active":"2024-04-21T08:51:20.520533+0000","last_peered":"2024-04-21T08:51:20.520533+0000","last_clean":"2024-04-21T08:51:20.520533+0000","last_became_active":"2024-04-21T08:48:45.564225+0000","last_became_peered":"2024-04-21T08:48:45.564225+0000","last_unstale":"2024-04-21T08:51:20.520533+0000","last_undegraded":"2024-04-21T08:51:20.520533+0000","last_fullsized":"2024-04-21T08:51:20.520533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:24:42.850371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":179,"num_write_kb":1285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828519+0000","last_change":"2024-04-21T08:48:46.466622+0000","last_active":"2024-04-21T08:51:10.828519+0000","last_peered":"2024-04-21T08:51:10.828519+0000","last_clean":"2024-04-21T08:51:10.828519+0000","last_became_active":"2024-04-21T08:48:46.466129+0000","last_became_peered":"2024-04-21T08:48:46.466129+0000","last_unstale":"2024-04-21T08:51:10.828519+0000","last_undegraded":"2024-04-21T08:51:10.828519+0000","last_fullsized":"2024-04-21T08:51:10.828519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:20:26.064193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832121+0000","last_change":"2024-04-21T08:48:50.759534+0000","last_active":"2024-04-21T08:51:10.832121+0000","last_peered":"2024-04-21T08:51:10.832121+0000","last_clean":"2024-04-21T08:51:10.832121+0000","last_became_active":"2024-04-21T08:48:50.759116+0000","last_became_peered":"2024-04-21T08:48:50.759116+0000","last_unstale":"2024-04-21T08:51:10.832121+0000","last_undegraded":"2024-04-21T08:51:10.832121+0000","last_fullsized":"2024-04-21T08:51:10.832121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:02:16.200901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038642+0000","last_change":"2024-04-21T08:48:45.545773+0000","last_active":"2024-04-21T08:51:11.038642+0000","last_peered":"2024-04-21T08:51:11.038642+0000","last_clean":"2024-04-21T08:51:11.038642+0000","last_became_active":"2024-04-21T08:48:45.545447+0000","last_became_peered":"2024-04-21T08:48:45.545447+0000","last_unstale":"2024-04-21T08:51:11.038642+0000","last_undegraded":"2024-04-21T08:51:11.038642+0000","last_fullsized":"2024-04-21T08:51:11.038642+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T08:59:42.171602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877976+0000","last_change":"2024-04-21T08:49:36.964411+0000","last_active":"2024-04-21T08:51:10.877976+0000","last_peered":"2024-04-21T08:51:10.877976+0000","last_clean":"2024-04-21T08:51:10.877976+0000","last_became_active":"2024-04-21T08:49:36.964046+0000","last_became_peered":"2024-04-21T08:49:36.964046+0000","last_unstale":"2024-04-21T08:51:10.877976+0000","last_undegraded":"2024-04-21T08:51:10.877976+0000","last_fullsized":"2024-04-21T08:51:10.877976+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:29:37.223495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038538+0000","last_change":"2024-04-21T08:50:37.351641+0000","last_active":"2024-04-21T08:51:11.038538+0000","last_peered":"2024-04-21T08:51:11.038538+0000","last_clean":"2024-04-21T08:51:11.038538+0000","last_became_active":"2024-04-21T08:50:37.351130+0000","last_became_peered":"2024-04-21T08:50:37.351130+0000","last_unstale":"2024-04-21T08:51:11.038538+0000","last_undegraded":"2024-04-21T08:51:11.038538+0000","last_fullsized":"2024-04-21T08:51:11.038538+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:56:58.989691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006155+0000","last_change":"2024-04-21T08:49:36.928993+0000","last_active":"2024-04-21T08:51:11.006155+0000","last_peered":"2024-04-21T08:51:11.006155+0000","last_clean":"2024-04-21T08:51:11.006155+0000","last_became_active":"2024-04-21T08:49:36.911247+0000","last_became_peered":"2024-04-21T08:49:36.911247+0000","last_unstale":"2024-04-21T08:51:11.006155+0000","last_undegraded":"2024-04-21T08:51:11.006155+0000","last_fullsized":"2024-04-21T08:51:11.006155+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:28:13.481670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939224+0000","last_change":"2024-04-21T08:48:46.465475+0000","last_active":"2024-04-21T08:51:10.939224+0000","last_peered":"2024-04-21T08:51:10.939224+0000","last_clean":"2024-04-21T08:51:10.939224+0000","last_became_active":"2024-04-21T08:48:46.465138+0000","last_became_peered":"2024-04-21T08:48:46.465138+0000","last_unstale":"2024-04-21T08:51:10.939224+0000","last_undegraded":"2024-04-21T08:51:10.939224+0000","last_fullsized":"2024-04-21T08:51:10.939224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:12:53.521547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.936707+0000","last_change":"2024-04-21T08:48:50.765765+0000","last_active":"2024-04-21T08:51:10.936707+0000","last_peered":"2024-04-21T08:51:10.936707+0000","last_clean":"2024-04-21T08:51:10.936707+0000","last_became_active":"2024-04-21T08:48:50.765324+0000","last_became_peered":"2024-04-21T08:48:50.765324+0000","last_unstale":"2024-04-21T08:51:10.936707+0000","last_undegraded":"2024-04-21T08:51:10.936707+0000","last_fullsized":"2024-04-21T08:51:10.936707+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:41:37.846768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828042+0000","last_change":"2024-04-21T08:48:45.557400+0000","last_active":"2024-04-21T08:51:10.828042+0000","last_peered":"2024-04-21T08:51:10.828042+0000","last_clean":"2024-04-21T08:51:10.828042+0000","last_became_active":"2024-04-21T08:48:45.555012+0000","last_became_peered":"2024-04-21T08:48:45.555012+0000","last_unstale":"2024-04-21T08:51:10.828042+0000","last_undegraded":"2024-04-21T08:51:10.828042+0000","last_fullsized":"2024-04-21T08:51:10.828042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:00:31.254353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062791+0000","last_change":"2024-04-21T08:48:46.470042+0000","last_active":"2024-04-21T08:51:11.062791+0000","last_peered":"2024-04-21T08:51:11.062791+0000","last_clean":"2024-04-21T08:51:11.062791+0000","last_became_active":"2024-04-21T08:48:46.469716+0000","last_became_peered":"2024-04-21T08:48:46.469716+0000","last_unstale":"2024-04-21T08:51:11.062791+0000","last_undegraded":"2024-04-21T08:51:11.062791+0000","last_fullsized":"2024-04-21T08:51:11.062791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:45:09.934751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038592+0000","last_change":"2024-04-21T08:48:50.770889+0000","last_active":"2024-04-21T08:51:11.038592+0000","last_peered":"2024-04-21T08:51:11.038592+0000","last_clean":"2024-04-21T08:51:11.038592+0000","last_became_active":"2024-04-21T08:48:50.769980+0000","last_became_peered":"2024-04-21T08:48:50.769980+0000","last_unstale":"2024-04-21T08:51:11.038592+0000","last_undegraded":"2024-04-21T08:51:11.038592+0000","last_fullsized":"2024-04-21T08:51:11.038592+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:01:44.631633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832532+0000","last_change":"2024-04-21T08:48:45.555479+0000","last_active":"2024-04-21T08:51:10.832532+0000","last_peered":"2024-04-21T08:51:10.832532+0000","last_clean":"2024-04-21T08:51:10.832532+0000","last_became_active":"2024-04-21T08:48:45.555162+0000","last_became_peered":"2024-04-21T08:48:45.555162+0000","last_unstale":"2024-04-21T08:51:10.832532+0000","last_undegraded":"2024-04-21T08:51:10.832532+0000","last_fullsized":"2024-04-21T08:51:10.832532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:15:20.262798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828417+0000","last_change":"2024-04-21T08:48:46.467962+0000","last_active":"2024-04-21T08:51:10.828417+0000","last_peered":"2024-04-21T08:51:10.828417+0000","last_clean":"2024-04-21T08:51:10.828417+0000","last_became_active":"2024-04-21T08:48:46.467561+0000","last_became_peered":"2024-04-21T08:48:46.467561+0000","last_unstale":"2024-04-21T08:51:10.828417+0000","last_undegraded":"2024-04-21T08:51:10.828417+0000","last_fullsized":"2024-04-21T08:51:10.828417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:09:18.262009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038487+0000","last_change":"2024-04-21T08:48:50.767678+0000","last_active":"2024-04-21T08:51:11.038487+0000","last_peered":"2024-04-21T08:51:11.038487+0000","last_clean":"2024-04-21T08:51:11.038487+0000","last_became_active":"2024-04-21T08:48:50.767309+0000","last_became_peered":"2024-04-21T08:48:50.767309+0000","last_unstale":"2024-04-21T08:51:11.038487+0000","last_undegraded":"2024-04-21T08:51:11.038487+0000","last_fullsized":"2024-04-21T08:51:11.038487+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:16:40.065043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.939597+0000","last_change":"2024-04-21T08:48:45.553321+0000","last_active":"2024-04-21T08:51:10.939597+0000","last_peered":"2024-04-21T08:51:10.939597+0000","last_clean":"2024-04-21T08:51:10.939597+0000","last_became_active":"2024-04-21T08:48:45.552871+0000","last_became_peered":"2024-04-21T08:48:45.552871+0000","last_unstale":"2024-04-21T08:51:10.939597+0000","last_undegraded":"2024-04-21T08:51:10.939597+0000","last_fullsized":"2024-04-21T08:51:10.939597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:16:27.339801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062861+0000","last_change":"2024-04-21T08:48:46.473453+0000","last_active":"2024-04-21T08:51:11.062861+0000","last_peered":"2024-04-21T08:51:11.062861+0000","last_clean":"2024-04-21T08:51:11.062861+0000","last_became_active":"2024-04-21T08:48:46.472613+0000","last_became_peered":"2024-04-21T08:48:46.472613+0000","last_unstale":"2024-04-21T08:51:11.062861+0000","last_undegraded":"2024-04-21T08:51:11.062861+0000","last_fullsized":"2024-04-21T08:51:11.062861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:43:58.995997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006138+0000","last_change":"2024-04-21T08:48:50.765282+0000","last_active":"2024-04-21T08:51:11.006138+0000","last_peered":"2024-04-21T08:51:11.006138+0000","last_clean":"2024-04-21T08:51:11.006138+0000","last_became_active":"2024-04-21T08:48:50.764891+0000","last_became_peered":"2024-04-21T08:48:50.764891+0000","last_unstale":"2024-04-21T08:51:11.006138+0000","last_undegraded":"2024-04-21T08:51:11.006138+0000","last_fullsized":"2024-04-21T08:51:11.006138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:18:40.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.038155+0000","last_change":"2024-04-21T08:49:36.926642+0000","last_active":"2024-04-21T08:51:11.038155+0000","last_peered":"2024-04-21T08:51:11.038155+0000","last_clean":"2024-04-21T08:51:11.038155+0000","last_became_active":"2024-04-21T08:49:36.910453+0000","last_became_peered":"2024-04-21T08:49:36.910453+0000","last_unstale":"2024-04-21T08:51:11.038155+0000","last_undegraded":"2024-04-21T08:51:11.038155+0000","last_fullsized":"2024-04-21T08:51:11.038155+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:22:19.290010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063219+0000","last_change":"2024-04-21T08:48:46.470058+0000","last_active":"2024-04-21T08:51:11.063219+0000","last_peered":"2024-04-21T08:51:11.063219+0000","last_clean":"2024-04-21T08:51:11.063219+0000","last_became_active":"2024-04-21T08:48:46.469485+0000","last_became_peered":"2024-04-21T08:48:46.469485+0000","last_unstale":"2024-04-21T08:51:11.063219+0000","last_undegraded":"2024-04-21T08:51:11.063219+0000","last_fullsized":"2024-04-21T08:51:11.063219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:33:13.212087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877446+0000","last_change":"2024-04-21T08:48:50.762944+0000","last_active":"2024-04-21T08:51:10.877446+0000","last_peered":"2024-04-21T08:51:10.877446+0000","last_clean":"2024-04-21T08:51:10.877446+0000","last_became_active":"2024-04-21T08:48:50.762508+0000","last_became_peered":"2024-04-21T08:48:50.762508+0000","last_unstale":"2024-04-21T08:51:10.877446+0000","last_undegraded":"2024-04-21T08:51:10.877446+0000","last_fullsized":"2024-04-21T08:51:10.877446+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:35.393339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'65","reported_seq":130,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:20.177489+0000","last_change":"2024-04-21T08:48:45.560361+0000","last_active":"2024-04-21T08:51:20.177489+0000","last_peered":"2024-04-21T08:51:20.177489+0000","last_clean":"2024-04-21T08:51:20.177489+0000","last_became_active":"2024-04-21T08:48:45.559946+0000","last_became_peered":"2024-04-21T08:48:45.559946+0000","last_unstale":"2024-04-21T08:51:20.177489+0000","last_undegraded":"2024-04-21T08:51:20.177489+0000","last_fullsized":"2024-04-21T08:51:20.177489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:46:55.599714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063003+0000","last_change":"2024-04-21T08:48:46.472706+0000","last_active":"2024-04-21T08:51:11.063003+0000","last_peered":"2024-04-21T08:51:11.063003+0000","last_clean":"2024-04-21T08:51:11.063003+0000","last_became_active":"2024-04-21T08:48:46.472378+0000","last_became_peered":"2024-04-21T08:48:46.472378+0000","last_unstale":"2024-04-21T08:51:11.063003+0000","last_undegraded":"2024-04-21T08:51:11.063003+0000","last_fullsized":"2024-04-21T08:51:11.063003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:16:58.228899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945191+0000","last_change":"2024-04-21T08:48:50.760610+0000","last_active":"2024-04-21T08:51:10.945191+0000","last_peered":"2024-04-21T08:51:10.945191+0000","last_clean":"2024-04-21T08:51:10.945191+0000","last_became_active":"2024-04-21T08:48:50.759991+0000","last_became_peered":"2024-04-21T08:48:50.759991+0000","last_unstale":"2024-04-21T08:51:10.945191+0000","last_undegraded":"2024-04-21T08:51:10.945191+0000","last_fullsized":"2024-04-21T08:51:10.945191+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:49.739565+0000","last_clean_scrub_stamp":"2024-04-21T08:48:49.739565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:12:43.649701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036942+0000","last_change":"2024-04-21T08:48:45.560773+0000","last_active":"2024-04-21T08:51:11.036942+0000","last_peered":"2024-04-21T08:51:11.036942+0000","last_clean":"2024-04-21T08:51:11.036942+0000","last_became_active":"2024-04-21T08:48:45.560431+0000","last_became_peered":"2024-04-21T08:48:45.560431+0000","last_unstale":"2024-04-21T08:51:11.036942+0000","last_undegraded":"2024-04-21T08:51:11.036942+0000","last_fullsized":"2024-04-21T08:51:11.036942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:44.529183+0000","last_clean_scrub_stamp":"2024-04-21T08:48:44.529183+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:20:07.858838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063259+0000","last_change":"2024-04-21T08:48:46.475983+0000","last_active":"2024-04-21T08:51:11.063259+0000","last_peered":"2024-04-21T08:51:11.063259+0000","last_clean":"2024-04-21T08:51:11.063259+0000","last_became_active":"2024-04-21T08:48:46.475667+0000","last_became_peered":"2024-04-21T08:48:46.475667+0000","last_unstale":"2024-04-21T08:51:11.063259+0000","last_undegraded":"2024-04-21T08:51:11.063259+0000","last_fullsized":"2024-04-21T08:51:11.063259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:28:41.013111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062960+0000","last_change":"2024-04-21T08:48:46.464675+0000","last_active":"2024-04-21T08:51:11.062960+0000","last_peered":"2024-04-21T08:51:11.062960+0000","last_clean":"2024-04-21T08:51:11.062960+0000","last_became_active":"2024-04-21T08:48:46.464314+0000","last_became_peered":"2024-04-21T08:48:46.464314+0000","last_unstale":"2024-04-21T08:51:11.062960+0000","last_undegraded":"2024-04-21T08:51:11.062960+0000","last_fullsized":"2024-04-21T08:51:11.062960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:53.048400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832307+0000","last_change":"2024-04-21T08:49:36.911025+0000","last_active":"2024-04-21T08:51:10.832307+0000","last_peered":"2024-04-21T08:51:10.832307+0000","last_clean":"2024-04-21T08:51:10.832307+0000","last_became_active":"2024-04-21T08:49:36.909942+0000","last_became_peered":"2024-04-21T08:49:36.909942+0000","last_unstale":"2024-04-21T08:51:10.832307+0000","last_undegraded":"2024-04-21T08:51:10.832307+0000","last_fullsized":"2024-04-21T08:51:10.832307+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:51:39.864253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832375+0000","last_change":"2024-04-21T08:48:46.467867+0000","last_active":"2024-04-21T08:51:10.832375+0000","last_peered":"2024-04-21T08:51:10.832375+0000","last_clean":"2024-04-21T08:51:10.832375+0000","last_became_active":"2024-04-21T08:48:46.465917+0000","last_became_peered":"2024-04-21T08:48:46.465917+0000","last_unstale":"2024-04-21T08:51:10.832375+0000","last_undegraded":"2024-04-21T08:51:10.832375+0000","last_fullsized":"2024-04-21T08:51:10.832375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:19:56.824998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.877556+0000","last_change":"2024-04-21T08:48:46.467159+0000","last_active":"2024-04-21T08:51:10.877556+0000","last_peered":"2024-04-21T08:51:10.877556+0000","last_clean":"2024-04-21T08:51:10.877556+0000","last_became_active":"2024-04-21T08:48:46.466639+0000","last_became_peered":"2024-04-21T08:48:46.466639+0000","last_unstale":"2024-04-21T08:51:10.877556+0000","last_undegraded":"2024-04-21T08:51:10.877556+0000","last_fullsized":"2024-04-21T08:51:10.877556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:54:44.772535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036982+0000","last_change":"2024-04-21T08:48:46.466403+0000","last_active":"2024-04-21T08:51:11.036982+0000","last_peered":"2024-04-21T08:51:11.036982+0000","last_clean":"2024-04-21T08:51:11.036982+0000","last_became_active":"2024-04-21T08:48:46.466060+0000","last_became_peered":"2024-04-21T08:48:46.466060+0000","last_unstale":"2024-04-21T08:51:11.036982+0000","last_undegraded":"2024-04-21T08:51:11.036982+0000","last_fullsized":"2024-04-21T08:51:11.036982+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:14.374193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.063096+0000","last_change":"2024-04-21T08:49:36.909134+0000","last_active":"2024-04-21T08:51:11.063096+0000","last_peered":"2024-04-21T08:51:11.063096+0000","last_clean":"2024-04-21T08:51:11.063096+0000","last_became_active":"2024-04-21T08:49:36.908723+0000","last_became_peered":"2024-04-21T08:49:36.908723+0000","last_unstale":"2024-04-21T08:51:11.063096+0000","last_undegraded":"2024-04-21T08:51:11.063096+0000","last_fullsized":"2024-04-21T08:51:11.063096+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:28.123016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.832933+0000","last_change":"2024-04-21T08:48:46.464098+0000","last_active":"2024-04-21T08:51:10.832933+0000","last_peered":"2024-04-21T08:51:10.832933+0000","last_clean":"2024-04-21T08:51:10.832933+0000","last_became_active":"2024-04-21T08:48:46.463756+0000","last_became_peered":"2024-04-21T08:48:46.463756+0000","last_unstale":"2024-04-21T08:51:10.832933+0000","last_undegraded":"2024-04-21T08:51:10.832933+0000","last_fullsized":"2024-04-21T08:51:10.832933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:31.725155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828666+0000","last_change":"2024-04-21T08:48:46.467372+0000","last_active":"2024-04-21T08:51:10.828666+0000","last_peered":"2024-04-21T08:51:10.828666+0000","last_clean":"2024-04-21T08:51:10.828666+0000","last_became_active":"2024-04-21T08:48:46.467043+0000","last_became_peered":"2024-04-21T08:48:46.467043+0000","last_unstale":"2024-04-21T08:51:10.828666+0000","last_undegraded":"2024-04-21T08:51:10.828666+0000","last_fullsized":"2024-04-21T08:51:10.828666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:17:41.889994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.945316+0000","last_change":"2024-04-21T08:48:46.460604+0000","last_active":"2024-04-21T08:51:10.945316+0000","last_peered":"2024-04-21T08:51:10.945316+0000","last_clean":"2024-04-21T08:51:10.945316+0000","last_became_active":"2024-04-21T08:48:46.460276+0000","last_became_peered":"2024-04-21T08:48:46.460276+0000","last_unstale":"2024-04-21T08:51:10.945316+0000","last_undegraded":"2024-04-21T08:51:10.945316+0000","last_fullsized":"2024-04-21T08:51:10.945316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:30.194808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:10.828076+0000","last_change":"2024-04-21T08:48:46.463665+0000","last_active":"2024-04-21T08:51:10.828076+0000","last_peered":"2024-04-21T08:51:10.828076+0000","last_clean":"2024-04-21T08:51:10.828076+0000","last_became_active":"2024-04-21T08:48:46.463309+0000","last_became_peered":"2024-04-21T08:48:46.463309+0000","last_unstale":"2024-04-21T08:51:10.828076+0000","last_undegraded":"2024-04-21T08:51:10.828076+0000","last_fullsized":"2024-04-21T08:51:10.828076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:12:36.436565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.006407+0000","last_change":"2024-04-21T08:48:46.476845+0000","last_active":"2024-04-21T08:51:11.006407+0000","last_peered":"2024-04-21T08:51:11.006407+0000","last_clean":"2024-04-21T08:51:11.006407+0000","last_became_active":"2024-04-21T08:48:46.476507+0000","last_became_peered":"2024-04-21T08:48:46.476507+0000","last_unstale":"2024-04-21T08:51:11.006407+0000","last_undegraded":"2024-04-21T08:51:11.006407+0000","last_fullsized":"2024-04-21T08:51:11.006407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:47.424576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.036615+0000","last_change":"2024-04-21T08:48:46.469316+0000","last_active":"2024-04-21T08:51:11.036615+0000","last_peered":"2024-04-21T08:51:11.036615+0000","last_clean":"2024-04-21T08:51:11.036615+0000","last_became_active":"2024-04-21T08:48:46.468852+0000","last_became_peered":"2024-04-21T08:48:46.468852+0000","last_unstale":"2024-04-21T08:51:11.036615+0000","last_undegraded":"2024-04-21T08:51:11.036615+0000","last_fullsized":"2024-04-21T08:51:11.036615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:05:03.664279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:51:11.062922+0000","last_change":"2024-04-21T08:48:46.466703+0000","last_active":"2024-04-21T08:51:11.062922+0000","last_peered":"2024-04-21T08:51:11.062922+0000","last_clean":"2024-04-21T08:51:11.062922+0000","last_became_active":"2024-04-21T08:48:46.466395+0000","last_became_peered":"2024-04-21T08:48:46.466395+0000","last_unstale":"2024-04-21T08:51:11.062922+0000","last_undegraded":"2024-04-21T08:51:11.062922+0000","last_fullsized":"2024-04-21T08:51:11.062922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:48:45.530683+0000","last_clean_scrub_stamp":"2024-04-21T08:48:45.530683+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:08:56.956283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108027,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":799382,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1154,"num_read_kb":796,"num_write":425,"num_write_kb":2365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":2398146,"data_compressed":108306,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":529689,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645361,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32384,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712768,"statfs":{"total":95995035648,"available":95961874432,"internally_reserved":0,"allocated":1228800,"data_stored":1518311,"data_compressed":82401,"data_compressed_allocated":110592,"data_compressed_original":987136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":927125,"data_compressed":48868,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070524,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":922837,"data_compressed":48868,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1097728,"data_stored":1127655,"data_compressed":65588,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495686,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1028096,"data_stored":524989,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":596819,"data_compressed":31190,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":526377,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1122304,"data_stored":598677,"data_compressed":31190,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":1246631,"data_compressed":64070,"data_compressed_allocated":86016,"data_compressed_original":782336,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":531535,"data_compressed":29579,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1150976,"data_stored":1450161,"data_compressed":80790,"data_compressed_allocated":106496,"data_compressed_original":978944,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36009,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":71920,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":73778,"data_compressed":1611,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34491,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330798,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":393842,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19289,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":403044,"data_compressed":16813,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:51:24.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:51:24.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.958+0000 7fa135ac1640 1 -- 172.21.15.28:0/3816967692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130106180 msgr2=0x7fa13010dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.958+0000 7fa135ac1640 1 --2- 172.21.15.28:0/3816967692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130106180 0x7fa13010dd50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa128009ca0 tx=0x7fa12802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.958+0000 7fa135ac1640 1 -- 172.21.15.28:0/3816967692 shutdown_connections 2024-04-21T08:51:24.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.958+0000 7fa135ac1640 1 --2- 172.21.15.28:0/3816967692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130106180 0x7fa13010dd50 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.958+0000 7fa135ac1640 1 --2- 172.21.15.28:0/3816967692 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa130101a60 0x7fa130105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.958+0000 7fa135ac1640 1 --2- 172.21.15.28:0/3816967692 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1301010b0 0x7fa130101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.958+0000 7fa135ac1640 1 -- 172.21.15.28:0/3816967692 >> 172.21.15.28:0/3816967692 conn(0x7fa1300771d0 msgr2=0x7fa1300775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:24.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.958+0000 7fa135ac1640 1 -- 172.21.15.28:0/3816967692 shutdown_connections 2024-04-21T08:51:24.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.958+0000 7fa135ac1640 1 -- 172.21.15.28:0/3816967692 wait complete. 2024-04-21T08:51:24.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.959+0000 7fa135ac1640 1 Processor -- start 2024-04-21T08:51:24.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.959+0000 7fa135ac1640 1 -- start start 2024-04-21T08:51:24.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.959+0000 7fa135ac1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1301010b0 0x7fa13019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.959+0000 7fa135ac1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130106180 0x7fa13019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.959+0000 7fa135ac1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa130198590 0x7fa130198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.959+0000 7fa135ac1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa130111140 con 0x7fa130106180 2024-04-21T08:51:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.959+0000 7fa135ac1640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa130110fc0 con 0x7fa130198590 2024-04-21T08:51:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.959+0000 7fa135ac1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa1301112c0 con 0x7fa1301010b0 2024-04-21T08:51:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.959+0000 7fa12effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130106180 0x7fa13019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.959+0000 7fa12effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130106180 0x7fa13019e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44454/0 (socket says 172.21.15.28:44454) 2024-04-21T08:51:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.959+0000 7fa12effd640 1 -- 172.21.15.28:0/832391549 learned_addr learned my addr 172.21.15.28:0/832391549 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.959+0000 7fa12ffff640 1 --2- 172.21.15.28:0/832391549 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa130198590 0x7fa130198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.960+0000 7fa12f7fe640 1 --2- 172.21.15.28:0/832391549 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1301010b0 0x7fa13019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.960+0000 7fa12effd640 1 -- 172.21.15.28:0/832391549 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1301010b0 msgr2=0x7fa13019e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.960+0000 7fa12effd640 1 --2- 172.21.15.28:0/832391549 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1301010b0 0x7fa13019e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.960+0000 7fa12effd640 1 -- 172.21.15.28:0/832391549 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa130198590 msgr2=0x7fa130198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.960+0000 7fa12effd640 1 --2- 172.21.15.28:0/832391549 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa130198590 0x7fa130198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.960+0000 7fa12effd640 1 -- 172.21.15.28:0/832391549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1301990b0 con 0x7fa130106180 2024-04-21T08:51:24.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.960+0000 7fa12effd640 1 --2- 172.21.15.28:0/832391549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130106180 0x7fa13019e920 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa12400c970 tx=0x7fa12400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:24.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.960+0000 7fa12cff9640 1 -- 172.21.15.28:0/832391549 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa124007bb0 con 0x7fa130106180 2024-04-21T08:51:24.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.960+0000 7fa12cff9640 1 -- 172.21.15.28:0/832391549 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa124007d50 con 0x7fa130106180 2024-04-21T08:51:24.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.960+0000 7fa135ac1640 1 -- 172.21.15.28:0/832391549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1301993a0 con 0x7fa130106180 2024-04-21T08:51:24.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.960+0000 7fa12cff9640 1 -- 172.21.15.28:0/832391549 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa124004dd0 con 0x7fa130106180 2024-04-21T08:51:24.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.960+0000 7fa135ac1640 1 -- 172.21.15.28:0/832391549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1301a55f0 con 0x7fa130106180 2024-04-21T08:51:24.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.961+0000 7fa135ac1640 1 -- 172.21.15.28:0/832391549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa130069230 con 0x7fa130106180 2024-04-21T08:51:24.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.962+0000 7fa12cff9640 1 -- 172.21.15.28:0/832391549 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fa124004fb0 con 0x7fa130106180 2024-04-21T08:51:24.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.963+0000 7fa12cff9640 1 --2- 172.21.15.28:0/832391549 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa1180a3e40 0x7fa1180a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.963+0000 7fa12cff9640 1 -- 172.21.15.28:0/832391549 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fa1240d0b20 con 0x7fa130106180 2024-04-21T08:51:24.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.963+0000 7fa12f7fe640 1 --2- 172.21.15.28:0/832391549 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa1180a3e40 0x7fa1180a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.963+0000 7fa12f7fe640 1 --2- 172.21.15.28:0/832391549 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa1180a3e40 0x7fa1180a6300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa120009990 tx=0x7fa120009110 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:24.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:24.968+0000 7fa12cff9640 1 -- 172.21.15.28:0/832391549 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa124014030 con 0x7fa130106180 2024-04-21T08:51:25.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:24 smithi028 ceph-mon[28003]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:25.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.192+0000 7fa135ac1640 1 -- 172.21.15.28:0/832391549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa130101a60 con 0x7fa130106180 2024-04-21T08:51:25.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.193+0000 7fa12cff9640 1 -- 172.21.15.28:0/832391549 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7fa12409cb70 con 0x7fa130106180 2024-04-21T08:51:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:25.193 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:51:09.181803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:51:25.193 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-04-21T08:51:25.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.195+0000 7fa135ac1640 1 -- 172.21.15.28:0/832391549 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa1180a3e40 msgr2=0x7fa1180a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.195+0000 7fa135ac1640 1 --2- 172.21.15.28:0/832391549 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa1180a3e40 0x7fa1180a6300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa120009990 tx=0x7fa120009110 comp rx=0 tx=0).stop 2024-04-21T08:51:25.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.195+0000 7fa135ac1640 1 -- 172.21.15.28:0/832391549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130106180 msgr2=0x7fa13019e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.195+0000 7fa135ac1640 1 --2- 172.21.15.28:0/832391549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130106180 0x7fa13019e920 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa12400c970 tx=0x7fa12400cd70 comp rx=0 tx=0).stop 2024-04-21T08:51:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.196+0000 7fa135ac1640 1 -- 172.21.15.28:0/832391549 shutdown_connections 2024-04-21T08:51:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.196+0000 7fa135ac1640 1 --2- 172.21.15.28:0/832391549 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa1180a3e40 0x7fa1180a6300 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.196+0000 7fa135ac1640 1 --2- 172.21.15.28:0/832391549 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa130198590 0x7fa130198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.196+0000 7fa135ac1640 1 --2- 172.21.15.28:0/832391549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130106180 0x7fa13019e920 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.196+0000 7fa135ac1640 1 --2- 172.21.15.28:0/832391549 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1301010b0 0x7fa13019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.196+0000 7fa135ac1640 1 -- 172.21.15.28:0/832391549 >> 172.21.15.28:0/832391549 conn(0x7fa1300771d0 msgr2=0x7fa13006a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.196+0000 7fa135ac1640 1 -- 172.21.15.28:0/832391549 shutdown_connections 2024-04-21T08:51:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.196+0000 7fa135ac1640 1 -- 172.21.15.28:0/832391549 wait complete. 2024-04-21T08:51:25.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:51:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.309+0000 7f960f6ba640 1 -- 172.21.15.28:0/3559634472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9608101ad0 msgr2=0x7f960810dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.309+0000 7f960f6ba640 1 --2- 172.21.15.28:0/3559634472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9608101ad0 0x7f960810dcb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9600009ca0 tx=0x7f960002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.309+0000 7f960f6ba640 1 -- 172.21.15.28:0/3559634472 shutdown_connections 2024-04-21T08:51:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.309+0000 7f960f6ba640 1 --2- 172.21.15.28:0/3559634472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9608101ad0 0x7f960810dcb0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.309+0000 7f960f6ba640 1 --2- 172.21.15.28:0/3559634472 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9608101130 0x7f9608101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.309+0000 7f960f6ba640 1 --2- 172.21.15.28:0/3559634472 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9608106f10 0x7f96081072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.309+0000 7f960f6ba640 1 -- 172.21.15.28:0/3559634472 >> 172.21.15.28:0/3559634472 conn(0x7f9608077e30 msgr2=0x7f9608078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.309+0000 7f960f6ba640 1 -- 172.21.15.28:0/3559634472 shutdown_connections 2024-04-21T08:51:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.309+0000 7f960f6ba640 1 -- 172.21.15.28:0/3559634472 wait complete. 2024-04-21T08:51:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.310+0000 7f960f6ba640 1 Processor -- start 2024-04-21T08:51:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.310+0000 7f960f6ba640 1 -- start start 2024-04-21T08:51:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.310+0000 7f960f6ba640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9608101130 0x7f9608102b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.310+0000 7f960f6ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9608101ad0 0x7f9608103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.310+0000 7f960f6ba640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9608106f10 0x7f96081035d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.310+0000 7f960f6ba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96081110f0 con 0x7f9608101ad0 2024-04-21T08:51:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.310+0000 7f960f6ba640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9608110f70 con 0x7f9608106f10 2024-04-21T08:51:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.310+0000 7f960f6ba640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9608111270 con 0x7f9608101130 2024-04-21T08:51:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.310+0000 7f960cc55640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9608101ad0 0x7f9608103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.310+0000 7f960cc55640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9608101ad0 0x7f9608103090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44468/0 (socket says 172.21.15.28:44468) 2024-04-21T08:51:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.310+0000 7f960cc55640 1 -- 172.21.15.28:0/96299866 learned_addr learned my addr 172.21.15.28:0/96299866 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.310+0000 7f960d456640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9608101130 0x7f9608102b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.311+0000 7f960dc57640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9608106f10 0x7f96081035d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.311+0000 7f960cc55640 1 -- 172.21.15.28:0/96299866 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9608101130 msgr2=0x7f9608102b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.311+0000 7f960cc55640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9608101130 0x7f9608102b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.311+0000 7f960cc55640 1 -- 172.21.15.28:0/96299866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9608106f10 msgr2=0x7f96081035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.311+0000 7f960cc55640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9608106f10 0x7f96081035d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.311+0000 7f960cc55640 1 -- 172.21.15.28:0/96299866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96081a6ba0 con 0x7f9608101ad0 2024-04-21T08:51:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.311+0000 7f960d456640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9608101130 0x7f9608102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.311+0000 7f960dc57640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9608106f10 0x7f96081035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.311+0000 7f960cc55640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9608101ad0 0x7f9608103090 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f95f800c890 tx=0x7f95f800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.311+0000 7f95fe7fc640 1 -- 172.21.15.28:0/96299866 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f8004220 con 0x7f9608101ad0 2024-04-21T08:51:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.311+0000 7f95fe7fc640 1 -- 172.21.15.28:0/96299866 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95f80043c0 con 0x7f9608101ad0 2024-04-21T08:51:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.311+0000 7f960f6ba640 1 -- 172.21.15.28:0/96299866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96081a6e90 con 0x7f9608101ad0 2024-04-21T08:51:25.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.311+0000 7f95fe7fc640 1 -- 172.21.15.28:0/96299866 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f8004a00 con 0x7f9608101ad0 2024-04-21T08:51:25.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.311+0000 7f960f6ba640 1 -- 172.21.15.28:0/96299866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96081a7350 con 0x7f9608101ad0 2024-04-21T08:51:25.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.312+0000 7f960f6ba640 1 -- 172.21.15.28:0/96299866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9608069230 con 0x7f9608101ad0 2024-04-21T08:51:25.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.313+0000 7f95fe7fc640 1 -- 172.21.15.28:0/96299866 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f95f80079b0 con 0x7f9608101ad0 2024-04-21T08:51:25.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.314+0000 7f95fe7fc640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95f40a3e70 0x7f95f40a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.314+0000 7f960d456640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95f40a3e70 0x7f95f40a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.314+0000 7f95fe7fc640 1 -- 172.21.15.28:0/96299866 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f95f80d0c00 con 0x7f9608101ad0 2024-04-21T08:51:25.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.314+0000 7f960d456640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95f40a3e70 0x7f95f40a6330 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9604005fd0 tx=0x7f96040094d0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:25.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.318+0000 7f95fe7fc640 1 -- 172.21.15.28:0/96299866 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f95f80d7080 con 0x7f9608101ad0 2024-04-21T08:51:25.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:24 smithi152 ceph-mon[31401]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:25.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:24 smithi171 ceph-mon[31358]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:25.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.505+0000 7f960f6ba640 1 -- 172.21.15.28:0/96299866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f96081a76b0 con 0x7f9608101ad0 2024-04-21T08:51:25.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.507+0000 7f95fe7fc640 1 -- 172.21.15.28:0/96299866 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25160 (secure 0 0 0) 0x7f95f8014030 con 0x7f9608101ad0 2024-04-21T08:51:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:25.507 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:51:10.823421+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1076282159":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6800/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/926389767":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/892059008":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/2132574390":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:51:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.509+0000 7f960f6ba640 1 -- 172.21.15.28:0/96299866 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95f40a3e70 msgr2=0x7f95f40a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.509+0000 7f960f6ba640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95f40a3e70 0x7f95f40a6330 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9604005fd0 tx=0x7f96040094d0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.509+0000 7f960f6ba640 1 -- 172.21.15.28:0/96299866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9608101ad0 msgr2=0x7f9608103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.509+0000 7f960f6ba640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9608101ad0 0x7f9608103090 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f95f800c890 tx=0x7f95f800cd50 comp rx=0 tx=0).stop 2024-04-21T08:51:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.509+0000 7f960f6ba640 1 -- 172.21.15.28:0/96299866 shutdown_connections 2024-04-21T08:51:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.509+0000 7f960f6ba640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95f40a3e70 0x7f95f40a6330 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.509+0000 7f960f6ba640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9608106f10 0x7f96081035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.509+0000 7f960f6ba640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9608101ad0 0x7f9608103090 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.509+0000 7f960f6ba640 1 --2- 172.21.15.28:0/96299866 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9608101130 0x7f9608102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.509+0000 7f960f6ba640 1 -- 172.21.15.28:0/96299866 >> 172.21.15.28:0/96299866 conn(0x7f9608077e30 msgr2=0x7f960806a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.509+0000 7f960f6ba640 1 -- 172.21.15.28:0/96299866 shutdown_connections 2024-04-21T08:51:25.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.509+0000 7f960f6ba640 1 -- 172.21.15.28:0/96299866 wait complete. 2024-04-21T08:51:25.520 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:48:43.543799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:48:44.864564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:48:48.747842+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:51:25.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:51:25.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.626+0000 7f5d800f2640 1 -- 172.21.15.28:0/4113294522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d781011a0 msgr2=0x7f5d78101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.626+0000 7f5d800f2640 1 --2- 172.21.15.28:0/4113294522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d781011a0 0x7f5d78101600 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009930 tx=0x7f5d6802bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:25.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.626+0000 7f5d800f2640 1 -- 172.21.15.28:0/4113294522 shutdown_connections 2024-04-21T08:51:25.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.626+0000 7f5d800f2640 1 --2- 172.21.15.28:0/4113294522 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d78101b40 0x7f5d7810dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.626+0000 7f5d800f2640 1 --2- 172.21.15.28:0/4113294522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d781011a0 0x7f5d78101600 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.626+0000 7f5d800f2640 1 --2- 172.21.15.28:0/4113294522 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d78107030 0x7f5d78107410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.626+0000 7f5d800f2640 1 -- 172.21.15.28:0/4113294522 >> 172.21.15.28:0/4113294522 conn(0x7f5d78076c50 msgr2=0x7f5d78077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:25.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.626+0000 7f5d800f2640 1 -- 172.21.15.28:0/4113294522 shutdown_connections 2024-04-21T08:51:25.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.626+0000 7f5d800f2640 1 -- 172.21.15.28:0/4113294522 wait complete. 2024-04-21T08:51:25.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.626+0000 7f5d800f2640 1 Processor -- start 2024-04-21T08:51:25.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d800f2640 1 -- start start 2024-04-21T08:51:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d800f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d781011a0 0x7f5d7819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d800f2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d78101b40 0x7f5d7819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d800f2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d78107030 0x7f5d781985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d800f2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d78112290 con 0x7f5d781011a0 2024-04-21T08:51:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d800f2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5d78112110 con 0x7f5d78107030 2024-04-21T08:51:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d800f2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5d78112410 con 0x7f5d78101b40 2024-04-21T08:51:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d7de8e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d781011a0 0x7f5d7819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d7de8e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d781011a0 0x7f5d7819e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44498/0 (socket says 172.21.15.28:44498) 2024-04-21T08:51:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d7de8e640 1 -- 172.21.15.28:0/1146826309 learned_addr learned my addr 172.21.15.28:0/1146826309 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d7d68d640 1 --2- 172.21.15.28:0/1146826309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d78101b40 0x7f5d7819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d7e68f640 1 --2- 172.21.15.28:0/1146826309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d78107030 0x7f5d781985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d7de8e640 1 -- 172.21.15.28:0/1146826309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d78101b40 msgr2=0x7f5d7819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d7de8e640 1 --2- 172.21.15.28:0/1146826309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d78101b40 0x7f5d7819e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d7de8e640 1 -- 172.21.15.28:0/1146826309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d78107030 msgr2=0x7f5d781985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d7de8e640 1 --2- 172.21.15.28:0/1146826309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d78107030 0x7f5d781985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d7de8e640 1 -- 172.21.15.28:0/1146826309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d78198d70 con 0x7f5d781011a0 2024-04-21T08:51:25.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d7d68d640 1 --2- 172.21.15.28:0/1146826309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d78101b40 0x7f5d7819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:25.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.627+0000 7f5d7de8e640 1 --2- 172.21.15.28:0/1146826309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d781011a0 0x7f5d7819e3f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5d7400d950 tx=0x7f5d7400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:25.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.628+0000 7f5d6effd640 1 -- 172.21.15.28:0/1146826309 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d74017030 con 0x7f5d781011a0 2024-04-21T08:51:25.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.628+0000 7f5d800f2640 1 -- 172.21.15.28:0/1146826309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d78199060 con 0x7f5d781011a0 2024-04-21T08:51:25.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.628+0000 7f5d6effd640 1 -- 172.21.15.28:0/1146826309 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d74004020 con 0x7f5d781011a0 2024-04-21T08:51:25.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.628+0000 7f5d6effd640 1 -- 172.21.15.28:0/1146826309 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d740047a0 con 0x7f5d781011a0 2024-04-21T08:51:25.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.628+0000 7f5d800f2640 1 -- 172.21.15.28:0/1146826309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d781a52b0 con 0x7f5d781011a0 2024-04-21T08:51:25.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.629+0000 7f5d800f2640 1 -- 172.21.15.28:0/1146826309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d78102590 con 0x7f5d781011a0 2024-04-21T08:51:25.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.630+0000 7f5d6effd640 1 -- 172.21.15.28:0/1146826309 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f5d74009a30 con 0x7f5d781011a0 2024-04-21T08:51:25.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.630+0000 7f5d6effd640 1 --2- 172.21.15.28:0/1146826309 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5d640a3e40 0x7f5d640a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.631+0000 7f5d6effd640 1 -- 172.21.15.28:0/1146826309 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f5d740d0fb0 con 0x7f5d781011a0 2024-04-21T08:51:25.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.631+0000 7f5d7d68d640 1 --2- 172.21.15.28:0/1146826309 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5d640a3e40 0x7f5d640a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.631+0000 7f5d7d68d640 1 --2- 172.21.15.28:0/1146826309 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5d640a3e40 0x7f5d640a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009900 tx=0x7f5d68005fd0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:25.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.634+0000 7f5d6effd640 1 -- 172.21.15.28:0/1146826309 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d7409d060 con 0x7f5d781011a0 2024-04-21T08:51:25.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.821+0000 7f5d800f2640 1 -- 172.21.15.28:0/1146826309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5d78109370 con 0x7f5d781011a0 2024-04-21T08:51:25.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.822+0000 7f5d6effd640 1 -- 172.21.15.28:0/1146826309 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f5d7409c960 con 0x7f5d781011a0 2024-04-21T08:51:25.822 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T08:51:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.824+0000 7f5d800f2640 1 -- 172.21.15.28:0/1146826309 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5d640a3e40 msgr2=0x7f5d640a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.824+0000 7f5d800f2640 1 --2- 172.21.15.28:0/1146826309 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5d640a3e40 0x7f5d640a6300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009900 tx=0x7f5d68005fd0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.824+0000 7f5d800f2640 1 -- 172.21.15.28:0/1146826309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d781011a0 msgr2=0x7f5d7819e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.824+0000 7f5d800f2640 1 --2- 172.21.15.28:0/1146826309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d781011a0 0x7f5d7819e3f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5d7400d950 tx=0x7f5d7400de10 comp rx=0 tx=0).stop 2024-04-21T08:51:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.824+0000 7f5d800f2640 1 -- 172.21.15.28:0/1146826309 shutdown_connections 2024-04-21T08:51:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.824+0000 7f5d800f2640 1 --2- 172.21.15.28:0/1146826309 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5d640a3e40 0x7f5d640a6300 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.824+0000 7f5d800f2640 1 --2- 172.21.15.28:0/1146826309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d78107030 0x7f5d781985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.824+0000 7f5d800f2640 1 --2- 172.21.15.28:0/1146826309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d78101b40 0x7f5d7819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.824+0000 7f5d800f2640 1 --2- 172.21.15.28:0/1146826309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d781011a0 0x7f5d7819e3f0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.824+0000 7f5d800f2640 1 -- 172.21.15.28:0/1146826309 >> 172.21.15.28:0/1146826309 conn(0x7f5d78076c50 msgr2=0x7f5d780ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.824+0000 7f5d800f2640 1 -- 172.21.15.28:0/1146826309 shutdown_connections 2024-04-21T08:51:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.824+0000 7f5d800f2640 1 -- 172.21.15.28:0/1146826309 wait complete. 2024-04-21T08:51:25.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:51:25.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.940+0000 7f4d56a6d640 1 -- 172.21.15.28:0/3750906482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50102190 msgr2=0x7f4d50102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.940+0000 7f4d56a6d640 1 --2- 172.21.15.28:0/3750906482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50102190 0x7f4d50102570 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009900 tx=0x7f4d4002bc50 comp rx=0 tx=0).stop 2024-04-21T08:51:25.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.940+0000 7f4d56a6d640 1 -- 172.21.15.28:0/3750906482 shutdown_connections 2024-04-21T08:51:25.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.940+0000 7f4d56a6d640 1 --2- 172.21.15.28:0/3750906482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d50107260 0x7f4d5010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.940+0000 7f4d56a6d640 1 --2- 172.21.15.28:0/3750906482 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d50102b40 0x7f4d50106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.940+0000 7f4d56a6d640 1 --2- 172.21.15.28:0/3750906482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50102190 0x7f4d50102570 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.940+0000 7f4d56a6d640 1 -- 172.21.15.28:0/3750906482 >> 172.21.15.28:0/3750906482 conn(0x7f4d500fd990 msgr2=0x7f4d500ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.941+0000 7f4d56a6d640 1 -- 172.21.15.28:0/3750906482 shutdown_connections 2024-04-21T08:51:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.941+0000 7f4d56a6d640 1 -- 172.21.15.28:0/3750906482 wait complete. 2024-04-21T08:51:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.941+0000 7f4d56a6d640 1 Processor -- start 2024-04-21T08:51:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.941+0000 7f4d56a6d640 1 -- start start 2024-04-21T08:51:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.941+0000 7f4d56a6d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50102190 0x7f4d50074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.941+0000 7f4d56a6d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d50107260 0x7f4d50074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.941+0000 7f4d56a6d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d50074cd0 0x7f4d501a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.941+0000 7f4d56a6d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d501121a0 con 0x7f4d50102190 2024-04-21T08:51:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.941+0000 7f4d56a6d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4d50112020 con 0x7f4d50074cd0 2024-04-21T08:51:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.941+0000 7f4d56a6d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4d50112320 con 0x7f4d50107260 2024-04-21T08:51:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.941+0000 7f4d54809640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50102190 0x7f4d50074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.941+0000 7f4d54809640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50102190 0x7f4d50074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44518/0 (socket says 172.21.15.28:44518) 2024-04-21T08:51:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.941+0000 7f4d54809640 1 -- 172.21.15.28:0/3781685349 learned_addr learned my addr 172.21.15.28:0/3781685349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.942+0000 7f4d4ffff640 1 --2- 172.21.15.28:0/3781685349 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d50107260 0x7f4d50074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.942+0000 7f4d54809640 1 -- 172.21.15.28:0/3781685349 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d50107260 msgr2=0x7f4d50074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.942+0000 7f4d54809640 1 --2- 172.21.15.28:0/3781685349 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d50107260 0x7f4d50074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.942+0000 7f4d5500a640 1 --2- 172.21.15.28:0/3781685349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d50074cd0 0x7f4d501a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.942+0000 7f4d54809640 1 -- 172.21.15.28:0/3781685349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d50074cd0 msgr2=0x7f4d501a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.942+0000 7f4d54809640 1 --2- 172.21.15.28:0/3781685349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d50074cd0 0x7f4d501a6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.942+0000 7f4d54809640 1 -- 172.21.15.28:0/3781685349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d501a6f30 con 0x7f4d50102190 2024-04-21T08:51:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.942+0000 7f4d54809640 1 --2- 172.21.15.28:0/3781685349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50102190 0x7f4d50074250 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4d40005940 tx=0x7f4d4002c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.942+0000 7f4d4dffb640 1 -- 172.21.15.28:0/3781685349 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40043020 con 0x7f4d50102190 2024-04-21T08:51:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.942+0000 7f4d56a6d640 1 -- 172.21.15.28:0/3781685349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d501a71c0 con 0x7f4d50102190 2024-04-21T08:51:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.942+0000 7f4d4dffb640 1 -- 172.21.15.28:0/3781685349 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d4002cc80 con 0x7f4d50102190 2024-04-21T08:51:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.942+0000 7f4d4dffb640 1 -- 172.21.15.28:0/3781685349 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40003c30 con 0x7f4d50102190 2024-04-21T08:51:25.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.942+0000 7f4d56a6d640 1 -- 172.21.15.28:0/3781685349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d501a7760 con 0x7f4d50102190 2024-04-21T08:51:25.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.943+0000 7f4d56a6d640 1 -- 172.21.15.28:0/3781685349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d500691c0 con 0x7f4d50102190 2024-04-21T08:51:25.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.945+0000 7f4d4dffb640 1 -- 172.21.15.28:0/3781685349 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f4d40046020 con 0x7f4d50102190 2024-04-21T08:51:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.945+0000 7f4d4dffb640 1 --2- 172.21.15.28:0/3781685349 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4d3c0a3e40 0x7f4d3c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.945+0000 7f4d4dffb640 1 -- 172.21.15.28:0/3781685349 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f4d400f3810 con 0x7f4d50102190 2024-04-21T08:51:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.945+0000 7f4d4ffff640 1 --2- 172.21.15.28:0/3781685349 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4d3c0a3e40 0x7f4d3c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.946+0000 7f4d4ffff640 1 --2- 172.21.15.28:0/3781685349 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4d3c0a3e40 0x7f4d3c0a6300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4d38005e10 tx=0x7f4d38005da0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:25.949+0000 7f4d4dffb640 1 -- 172.21.15.28:0/3781685349 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4d4004d020 con 0x7f4d50102190 2024-04-21T08:51:26.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:25 smithi028 ceph-mon[28003]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:26.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:25 smithi028 ceph-mon[28003]: from='client.15801 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:26.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:25 smithi028 ceph-mon[28003]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-21T08:51:26.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:25 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/832391549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:26.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:25 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/96299866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:26.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:25 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1146826309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:26.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.134+0000 7f4d56a6d640 1 -- 172.21.15.28:0/3781685349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4d5010a490 con 0x7f4d50102190 2024-04-21T08:51:26.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.135+0000 7f4d4dffb640 1 -- 172.21.15.28:0/3781685349 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f4d4003e450 con 0x7f4d50102190 2024-04-21T08:51:26.135 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:51:26.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.137+0000 7f4d56a6d640 1 -- 172.21.15.28:0/3781685349 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4d3c0a3e40 msgr2=0x7f4d3c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.137+0000 7f4d56a6d640 1 --2- 172.21.15.28:0/3781685349 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4d3c0a3e40 0x7f4d3c0a6300 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4d38005e10 tx=0x7f4d38005da0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.137+0000 7f4d56a6d640 1 -- 172.21.15.28:0/3781685349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50102190 msgr2=0x7f4d50074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.137+0000 7f4d56a6d640 1 --2- 172.21.15.28:0/3781685349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50102190 0x7f4d50074250 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4d40005940 tx=0x7f4d4002c880 comp rx=0 tx=0).stop 2024-04-21T08:51:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.137+0000 7f4d56a6d640 1 -- 172.21.15.28:0/3781685349 shutdown_connections 2024-04-21T08:51:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.137+0000 7f4d56a6d640 1 --2- 172.21.15.28:0/3781685349 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4d3c0a3e40 0x7f4d3c0a6300 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.137+0000 7f4d56a6d640 1 --2- 172.21.15.28:0/3781685349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4d50074cd0 0x7f4d501a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.137+0000 7f4d56a6d640 1 --2- 172.21.15.28:0/3781685349 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d50107260 0x7f4d50074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.137+0000 7f4d56a6d640 1 --2- 172.21.15.28:0/3781685349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50102190 0x7f4d50074250 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.137+0000 7f4d56a6d640 1 -- 172.21.15.28:0/3781685349 >> 172.21.15.28:0/3781685349 conn(0x7f4d500fd990 msgr2=0x7f4d500ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.137+0000 7f4d56a6d640 1 -- 172.21.15.28:0/3781685349 shutdown_connections 2024-04-21T08:51:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.137+0000 7f4d56a6d640 1 -- 172.21.15.28:0/3781685349 wait complete. 2024-04-21T08:51:26.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:51:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.254+0000 7fc44348c640 1 -- 172.21.15.28:0/1039416470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43c103ad0 msgr2=0x7fc43c10bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.254+0000 7fc44348c640 1 --2- 172.21.15.28:0/1039416470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43c103ad0 0x7fc43c10bfb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009900 tx=0x7fc42c02bcd0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.254+0000 7fc44348c640 1 -- 172.21.15.28:0/1039416470 shutdown_connections 2024-04-21T08:51:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.254+0000 7fc44348c640 1 --2- 172.21.15.28:0/1039416470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc43c10c580 0x7fc43c10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.254+0000 7fc44348c640 1 --2- 172.21.15.28:0/1039416470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43c103ad0 0x7fc43c10bfb0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.254+0000 7fc44348c640 1 --2- 172.21.15.28:0/1039416470 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc43c1031b0 0x7fc43c103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.254+0000 7fc44348c640 1 -- 172.21.15.28:0/1039416470 >> 172.21.15.28:0/1039416470 conn(0x7fc43c0771d0 msgr2=0x7fc43c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.254+0000 7fc44348c640 1 -- 172.21.15.28:0/1039416470 shutdown_connections 2024-04-21T08:51:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.254+0000 7fc44348c640 1 -- 172.21.15.28:0/1039416470 wait complete. 2024-04-21T08:51:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.254+0000 7fc44348c640 1 Processor -- start 2024-04-21T08:51:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.254+0000 7fc44348c640 1 -- start start 2024-04-21T08:51:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc44348c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc43c1031b0 0x7fc43c19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc44348c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43c103ad0 0x7fc43c19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc44348c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc43c10c580 0x7fc43c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc44348c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc43c1121a0 con 0x7fc43c103ad0 2024-04-21T08:51:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc44348c640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc43c112020 con 0x7fc43c1031b0 2024-04-21T08:51:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc44348c640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc43c112320 con 0x7fc43c10c580 2024-04-21T08:51:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc440a27640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43c103ad0 0x7fc43c19e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc440a27640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43c103ad0 0x7fc43c19e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44534/0 (socket says 172.21.15.28:44534) 2024-04-21T08:51:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc440a27640 1 -- 172.21.15.28:0/425123260 learned_addr learned my addr 172.21.15.28:0/425123260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc441a29640 1 --2- 172.21.15.28:0/425123260 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc43c10c580 0x7fc43c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc441228640 1 --2- 172.21.15.28:0/425123260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc43c1031b0 0x7fc43c19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc440a27640 1 -- 172.21.15.28:0/425123260 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc43c10c580 msgr2=0x7fc43c1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc440a27640 1 --2- 172.21.15.28:0/425123260 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc43c10c580 0x7fc43c1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc440a27640 1 -- 172.21.15.28:0/425123260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc43c1031b0 msgr2=0x7fc43c19e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc440a27640 1 --2- 172.21.15.28:0/425123260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc43c1031b0 0x7fc43c19e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc440a27640 1 -- 172.21.15.28:0/425123260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc43c198d20 con 0x7fc43c103ad0 2024-04-21T08:51:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.255+0000 7fc440a27640 1 --2- 172.21.15.28:0/425123260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43c103ad0 0x7fc43c19e930 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc42c005940 tx=0x7fc42c02d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.256+0000 7fc4327fc640 1 -- 172.21.15.28:0/425123260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c043020 con 0x7fc43c103ad0 2024-04-21T08:51:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.256+0000 7fc44348c640 1 -- 172.21.15.28:0/425123260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc43c198fb0 con 0x7fc43c103ad0 2024-04-21T08:51:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.256+0000 7fc4327fc640 1 -- 172.21.15.28:0/425123260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc42c02dc90 con 0x7fc43c103ad0 2024-04-21T08:51:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.256+0000 7fc4327fc640 1 -- 172.21.15.28:0/425123260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c003ab0 con 0x7fc43c103ad0 2024-04-21T08:51:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.256+0000 7fc44348c640 1 -- 172.21.15.28:0/425123260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc43c1a52b0 con 0x7fc43c103ad0 2024-04-21T08:51:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.257+0000 7fc44348c640 1 -- 172.21.15.28:0/425123260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc43c06ad70 con 0x7fc43c103ad0 2024-04-21T08:51:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.258+0000 7fc4327fc640 1 -- 172.21.15.28:0/425123260 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fc42c046020 con 0x7fc43c103ad0 2024-04-21T08:51:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.258+0000 7fc4327fc640 1 --2- 172.21.15.28:0/425123260 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4280a3e40 0x7fc4280a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.258+0000 7fc441228640 1 --2- 172.21.15.28:0/425123260 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4280a3e40 0x7fc4280a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.258+0000 7fc4327fc640 1 -- 172.21.15.28:0/425123260 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fc42c0f2df0 con 0x7fc43c103ad0 2024-04-21T08:51:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.259+0000 7fc441228640 1 --2- 172.21.15.28:0/425123260 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4280a3e40 0x7fc4280a6300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc438005fd0 tx=0x7fc438009760 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.263+0000 7fc4327fc640 1 -- 172.21.15.28:0/425123260 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc42c04d020 con 0x7fc43c103ad0 2024-04-21T08:51:26.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:25 smithi152 ceph-mon[31401]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:26.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:25 smithi152 ceph-mon[31401]: from='client.15801 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:26.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:25 smithi152 ceph-mon[31401]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-21T08:51:26.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:25 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/832391549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:26.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:25 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/96299866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:26.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:25 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1146826309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:26.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:25 smithi171 ceph-mon[31358]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:26.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:25 smithi171 ceph-mon[31358]: from='client.15801 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:51:26.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:25 smithi171 ceph-mon[31358]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-21T08:51:26.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:25 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/832391549' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:26.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:25 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/96299866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:26.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:25 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1146826309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:51:26.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.447+0000 7fc44348c640 1 -- 172.21.15.28:0/425123260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc43c1076d0 con 0x7fc43c103ad0 2024-04-21T08:51:26.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.448+0000 7fc4327fc640 1 -- 172.21.15.28:0/425123260 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fc42c0befb0 con 0x7fc43c103ad0 2024-04-21T08:51:26.448 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-21T08:51:26.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.450+0000 7fc44348c640 1 -- 172.21.15.28:0/425123260 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4280a3e40 msgr2=0x7fc4280a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.450+0000 7fc44348c640 1 --2- 172.21.15.28:0/425123260 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4280a3e40 0x7fc4280a6300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc438005fd0 tx=0x7fc438009760 comp rx=0 tx=0).stop 2024-04-21T08:51:26.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.450+0000 7fc44348c640 1 -- 172.21.15.28:0/425123260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43c103ad0 msgr2=0x7fc43c19e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.450+0000 7fc44348c640 1 --2- 172.21.15.28:0/425123260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43c103ad0 0x7fc43c19e930 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc42c005940 tx=0x7fc42c02d890 comp rx=0 tx=0).stop 2024-04-21T08:51:26.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.450+0000 7fc44348c640 1 -- 172.21.15.28:0/425123260 shutdown_connections 2024-04-21T08:51:26.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.450+0000 7fc44348c640 1 --2- 172.21.15.28:0/425123260 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4280a3e40 0x7fc4280a6300 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.450+0000 7fc44348c640 1 --2- 172.21.15.28:0/425123260 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc43c10c580 0x7fc43c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.450+0000 7fc44348c640 1 --2- 172.21.15.28:0/425123260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc43c103ad0 0x7fc43c19e930 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.450+0000 7fc44348c640 1 --2- 172.21.15.28:0/425123260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc43c1031b0 0x7fc43c19e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.450+0000 7fc44348c640 1 -- 172.21.15.28:0/425123260 >> 172.21.15.28:0/425123260 conn(0x7fc43c0771d0 msgr2=0x7fc43c0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.450+0000 7fc44348c640 1 -- 172.21.15.28:0/425123260 shutdown_connections 2024-04-21T08:51:26.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.450+0000 7fc44348c640 1 -- 172.21.15.28:0/425123260 wait complete. 2024-04-21T08:51:26.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:51:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.566+0000 7fe77ad60640 1 -- 172.21.15.28:0/3989013880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe774107260 msgr2=0x7fe77410ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.566+0000 7fe77ad60640 1 --2- 172.21.15.28:0/3989013880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe774107260 0x7fe77410ee30 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009ca0 tx=0x7fe76c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.566+0000 7fe77ad60640 1 -- 172.21.15.28:0/3989013880 shutdown_connections 2024-04-21T08:51:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.566+0000 7fe77ad60640 1 --2- 172.21.15.28:0/3989013880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe774107260 0x7fe77410ee30 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.566+0000 7fe77ad60640 1 --2- 172.21.15.28:0/3989013880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe774102b40 0x7fe774106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.566+0000 7fe77ad60640 1 --2- 172.21.15.28:0/3989013880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe774102190 0x7fe774102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.566+0000 7fe77ad60640 1 -- 172.21.15.28:0/3989013880 >> 172.21.15.28:0/3989013880 conn(0x7fe7740793a0 msgr2=0x7fe7740797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.566+0000 7fe77ad60640 1 -- 172.21.15.28:0/3989013880 shutdown_connections 2024-04-21T08:51:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.566+0000 7fe77ad60640 1 -- 172.21.15.28:0/3989013880 wait complete. 2024-04-21T08:51:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe77ad60640 1 Processor -- start 2024-04-21T08:51:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe77ad60640 1 -- start start 2024-04-21T08:51:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe77ad60640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe774102190 0x7fe774198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe77ad60640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe774107260 0x7fe774198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe77ad60640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe77419cfd0 0x7fe77419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe77ad60640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7741121b0 con 0x7fe774107260 2024-04-21T08:51:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe77ad60640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe774112030 con 0x7fe774102190 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe77ad60640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe774112330 con 0x7fe77419cfd0 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe76bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe774107260 0x7fe774198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe76bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe774107260 0x7fe774198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44556/0 (socket says 172.21.15.28:44556) 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe76bfff640 1 -- 172.21.15.28:0/3402173311 learned_addr learned my addr 172.21.15.28:0/3402173311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe778afc640 1 --2- 172.21.15.28:0/3402173311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe774102190 0x7fe774198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe7792fd640 1 --2- 172.21.15.28:0/3402173311 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe77419cfd0 0x7fe77419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe76bfff640 1 -- 172.21.15.28:0/3402173311 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe77419cfd0 msgr2=0x7fe77419d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe76bfff640 1 --2- 172.21.15.28:0/3402173311 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe77419cfd0 0x7fe77419d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe76bfff640 1 -- 172.21.15.28:0/3402173311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe774102190 msgr2=0x7fe774198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe76bfff640 1 --2- 172.21.15.28:0/3402173311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe774102190 0x7fe774198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe76bfff640 1 -- 172.21.15.28:0/3402173311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe77419dac0 con 0x7fe774107260 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.567+0000 7fe778afc640 1 --2- 172.21.15.28:0/3402173311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe774102190 0x7fe774198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.568+0000 7fe76bfff640 1 --2- 172.21.15.28:0/3402173311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe774107260 0x7fe774198c50 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe764005fd0 tx=0x7fe76400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.568+0000 7fe769ffb640 1 -- 172.21.15.28:0/3402173311 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe764017070 con 0x7fe774107260 2024-04-21T08:51:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.568+0000 7fe769ffb640 1 -- 172.21.15.28:0/3402173311 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe764009c30 con 0x7fe774107260 2024-04-21T08:51:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.568+0000 7fe77ad60640 1 -- 172.21.15.28:0/3402173311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe77419ddb0 con 0x7fe774107260 2024-04-21T08:51:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.568+0000 7fe769ffb640 1 -- 172.21.15.28:0/3402173311 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe764003d30 con 0x7fe774107260 2024-04-21T08:51:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.568+0000 7fe77ad60640 1 -- 172.21.15.28:0/3402173311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7741a55f0 con 0x7fe774107260 2024-04-21T08:51:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.569+0000 7fe77ad60640 1 -- 172.21.15.28:0/3402173311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7740691c0 con 0x7fe774107260 2024-04-21T08:51:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.570+0000 7fe769ffb640 1 -- 172.21.15.28:0/3402173311 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fe764003ed0 con 0x7fe774107260 2024-04-21T08:51:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.570+0000 7fe769ffb640 1 --2- 172.21.15.28:0/3402173311 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe75c0a3ec0 0x7fe75c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.571+0000 7fe778afc640 1 --2- 172.21.15.28:0/3402173311 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe75c0a3ec0 0x7fe75c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.571+0000 7fe769ffb640 1 -- 172.21.15.28:0/3402173311 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe7640d0aa0 con 0x7fe774107260 2024-04-21T08:51:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.571+0000 7fe778afc640 1 --2- 172.21.15.28:0/3402173311 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe75c0a3ec0 0x7fe75c0a6380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe7700058c0 tx=0x7fe770005850 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.575+0000 7fe769ffb640 1 -- 172.21.15.28:0/3402173311 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe764020030 con 0x7fe774107260 2024-04-21T08:51:26.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.761+0000 7fe77ad60640 1 -- 172.21.15.28:0/3402173311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe774102b40 con 0x7fe774107260 2024-04-21T08:51:26.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.762+0000 7fe769ffb640 1 -- 172.21.15.28:0/3402173311 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fe76409cb50 con 0x7fe774107260 2024-04-21T08:51:26.762 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T08:51:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.764+0000 7fe77ad60640 1 -- 172.21.15.28:0/3402173311 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe75c0a3ec0 msgr2=0x7fe75c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.764+0000 7fe77ad60640 1 --2- 172.21.15.28:0/3402173311 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe75c0a3ec0 0x7fe75c0a6380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe7700058c0 tx=0x7fe770005850 comp rx=0 tx=0).stop 2024-04-21T08:51:26.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.764+0000 7fe77ad60640 1 -- 172.21.15.28:0/3402173311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe774107260 msgr2=0x7fe774198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.764+0000 7fe77ad60640 1 --2- 172.21.15.28:0/3402173311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe774107260 0x7fe774198c50 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe764005fd0 tx=0x7fe76400dde0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.764+0000 7fe77ad60640 1 -- 172.21.15.28:0/3402173311 shutdown_connections 2024-04-21T08:51:26.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.764+0000 7fe77ad60640 1 --2- 172.21.15.28:0/3402173311 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe75c0a3ec0 0x7fe75c0a6380 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.764+0000 7fe77ad60640 1 --2- 172.21.15.28:0/3402173311 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe77419cfd0 0x7fe77419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.764+0000 7fe77ad60640 1 --2- 172.21.15.28:0/3402173311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe774107260 0x7fe774198c50 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.764+0000 7fe77ad60640 1 --2- 172.21.15.28:0/3402173311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe774102190 0x7fe774198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.764+0000 7fe77ad60640 1 -- 172.21.15.28:0/3402173311 >> 172.21.15.28:0/3402173311 conn(0x7fe7740793a0 msgr2=0x7fe7740ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.764+0000 7fe77ad60640 1 -- 172.21.15.28:0/3402173311 shutdown_connections 2024-04-21T08:51:26.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.764+0000 7fe77ad60640 1 -- 172.21.15.28:0/3402173311 wait complete. 2024-04-21T08:51:26.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:51:26.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.879+0000 7f1afef9f640 1 -- 172.21.15.28:0/4218302982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af810c5a0 msgr2=0x7f1af810e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.879+0000 7f1afef9f640 1 --2- 172.21.15.28:0/4218302982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af810c5a0 0x7f1af810e9d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1af0009ca0 tx=0x7f1af002c180 comp rx=0 tx=0).stop 2024-04-21T08:51:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.879+0000 7f1afef9f640 1 -- 172.21.15.28:0/4218302982 shutdown_connections 2024-04-21T08:51:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.879+0000 7f1afef9f640 1 --2- 172.21.15.28:0/4218302982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af810c5a0 0x7f1af810e9d0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.879+0000 7f1afef9f640 1 --2- 172.21.15.28:0/4218302982 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1af8103af0 0x7f1af810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.879+0000 7f1afef9f640 1 --2- 172.21.15.28:0/4218302982 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1af81031d0 0x7f1af81035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.879+0000 7f1afef9f640 1 -- 172.21.15.28:0/4218302982 >> 172.21.15.28:0/4218302982 conn(0x7f1af80771d0 msgr2=0x7f1af80775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.879+0000 7f1afef9f640 1 -- 172.21.15.28:0/4218302982 shutdown_connections 2024-04-21T08:51:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.879+0000 7f1afef9f640 1 -- 172.21.15.28:0/4218302982 wait complete. 2024-04-21T08:51:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.880+0000 7f1afef9f640 1 Processor -- start 2024-04-21T08:51:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.880+0000 7f1afef9f640 1 -- start start 2024-04-21T08:51:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.880+0000 7f1afef9f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1af81031d0 0x7f1af8100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.880+0000 7f1afef9f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1af8103af0 0x7f1af8101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.880+0000 7f1afef9f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af810c5a0 0x7f1af81018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.880+0000 7f1afef9f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1af81121a0 con 0x7f1af810c5a0 2024-04-21T08:51:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.880+0000 7f1afef9f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1af8112020 con 0x7f1af8103af0 2024-04-21T08:51:26.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.880+0000 7f1afef9f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1af8112320 con 0x7f1af81031d0 2024-04-21T08:51:26.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.880+0000 7f1afd53c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af810c5a0 0x7f1af81018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.880+0000 7f1afd53c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af810c5a0 0x7f1af81018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44574/0 (socket says 172.21.15.28:44574) 2024-04-21T08:51:26.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.880+0000 7f1afd53c640 1 -- 172.21.15.28:0/1162094011 learned_addr learned my addr 172.21.15.28:0/1162094011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:26.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.880+0000 7f1afcd3b640 1 --2- 172.21.15.28:0/1162094011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1af81031d0 0x7f1af8100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.880+0000 7f1af7fff640 1 --2- 172.21.15.28:0/1162094011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1af8103af0 0x7f1af8101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.880+0000 7f1afd53c640 1 -- 172.21.15.28:0/1162094011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1af81031d0 msgr2=0x7f1af8100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.881+0000 7f1afd53c640 1 --2- 172.21.15.28:0/1162094011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1af81031d0 0x7f1af8100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.881+0000 7f1afd53c640 1 -- 172.21.15.28:0/1162094011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1af8103af0 msgr2=0x7f1af8101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.881+0000 7f1afd53c640 1 --2- 172.21.15.28:0/1162094011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1af8103af0 0x7f1af8101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.881+0000 7f1afd53c640 1 -- 172.21.15.28:0/1162094011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af81a6b00 con 0x7f1af810c5a0 2024-04-21T08:51:26.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.881+0000 7f1afd53c640 1 --2- 172.21.15.28:0/1162094011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af810c5a0 0x7f1af81018a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1af0004af0 tx=0x7f1af0003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.882+0000 7f1af5ffb640 1 -- 172.21.15.28:0/1162094011 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0039040 con 0x7f1af810c5a0 2024-04-21T08:51:26.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.882+0000 7f1af5ffb640 1 -- 172.21.15.28:0/1162094011 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1af0044070 con 0x7f1af810c5a0 2024-04-21T08:51:26.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.882+0000 7f1afef9f640 1 -- 172.21.15.28:0/1162094011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1af81a6d90 con 0x7f1af810c5a0 2024-04-21T08:51:26.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.882+0000 7f1af5ffb640 1 -- 172.21.15.28:0/1162094011 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af00327c0 con 0x7f1af810c5a0 2024-04-21T08:51:26.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.883+0000 7f1afef9f640 1 -- 172.21.15.28:0/1162094011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1af81a7280 con 0x7f1af810c5a0 2024-04-21T08:51:26.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.884+0000 7f1afef9f640 1 -- 172.21.15.28:0/1162094011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1af8069230 con 0x7f1af810c5a0 2024-04-21T08:51:26.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.884+0000 7f1af5ffb640 1 -- 172.21.15.28:0/1162094011 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f1af0004040 con 0x7f1af810c5a0 2024-04-21T08:51:26.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.885+0000 7f1af5ffb640 1 --2- 172.21.15.28:0/1162094011 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ae40a3e70 0x7f1ae40a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.885+0000 7f1af5ffb640 1 -- 172.21.15.28:0/1162094011 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1af00f31e0 con 0x7f1af810c5a0 2024-04-21T08:51:26.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.885+0000 7f1afcd3b640 1 --2- 172.21.15.28:0/1162094011 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ae40a3e70 0x7f1ae40a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.886+0000 7f1afcd3b640 1 --2- 172.21.15.28:0/1162094011 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ae40a3e70 0x7f1ae40a6330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ae000bbe0 tx=0x7f1ae000b390 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:26.890+0000 7f1af5ffb640 1 -- 172.21.15.28:0/1162094011 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1af00bf3a0 con 0x7f1af810c5a0 2024-04-21T08:51:27.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:26 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3781685349' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:27.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:26 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/425123260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:27.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:26 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3402173311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.117+0000 7f1afef9f640 1 -- 172.21.15.28:0/1162094011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1af8107690 con 0x7f1af810c5a0 2024-04-21T08:51:27.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.117+0000 7f1af5ffb640 1 -- 172.21.15.28:0/1162094011 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7f1af00beca0 con 0x7f1af810c5a0 2024-04-21T08:51:27.117 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:27.117 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:51:09.181803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:51:27.118 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-04-21T08:51:27.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.119+0000 7f1afef9f640 1 -- 172.21.15.28:0/1162094011 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ae40a3e70 msgr2=0x7f1ae40a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.119+0000 7f1afef9f640 1 --2- 172.21.15.28:0/1162094011 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ae40a3e70 0x7f1ae40a6330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ae000bbe0 tx=0x7f1ae000b390 comp rx=0 tx=0).stop 2024-04-21T08:51:27.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.119+0000 7f1afef9f640 1 -- 172.21.15.28:0/1162094011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af810c5a0 msgr2=0x7f1af81018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.119+0000 7f1afef9f640 1 --2- 172.21.15.28:0/1162094011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af810c5a0 0x7f1af81018a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1af0004af0 tx=0x7f1af0003ba0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.120+0000 7f1afef9f640 1 -- 172.21.15.28:0/1162094011 shutdown_connections 2024-04-21T08:51:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.120+0000 7f1afef9f640 1 --2- 172.21.15.28:0/1162094011 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ae40a3e70 0x7f1ae40a6330 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.120+0000 7f1afef9f640 1 --2- 172.21.15.28:0/1162094011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af810c5a0 0x7f1af81018a0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.120+0000 7f1afef9f640 1 --2- 172.21.15.28:0/1162094011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1af8103af0 0x7f1af8101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.120+0000 7f1afef9f640 1 --2- 172.21.15.28:0/1162094011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1af81031d0 0x7f1af8100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.120+0000 7f1afef9f640 1 -- 172.21.15.28:0/1162094011 >> 172.21.15.28:0/1162094011 conn(0x7f1af80771d0 msgr2=0x7f1af80fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.120+0000 7f1afef9f640 1 -- 172.21.15.28:0/1162094011 shutdown_connections 2024-04-21T08:51:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.120+0000 7f1afef9f640 1 -- 172.21.15.28:0/1162094011 wait complete. 2024-04-21T08:51:27.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:51:27.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.236+0000 7f6ce1a2e640 1 -- 172.21.15.28:0/2255226604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc06b260 msgr2=0x7f6cdc106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.236+0000 7f6ce1a2e640 1 --2- 172.21.15.28:0/2255226604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc06b260 0x7f6cdc106c10 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0009900 tx=0x7f6cd002bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:27.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.236+0000 7f6ce1a2e640 1 -- 172.21.15.28:0/2255226604 shutdown_connections 2024-04-21T08:51:27.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.236+0000 7f6ce1a2e640 1 --2- 172.21.15.28:0/2255226604 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6cdc107340 0x7f6cdc10eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.236+0000 7f6ce1a2e640 1 --2- 172.21.15.28:0/2255226604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc06b260 0x7f6cdc106c10 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.236+0000 7f6ce1a2e640 1 --2- 172.21.15.28:0/2255226604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6cdc06a8b0 0x7f6cdc06ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.236+0000 7f6ce1a2e640 1 -- 172.21.15.28:0/2255226604 >> 172.21.15.28:0/2255226604 conn(0x7f6cdc077e30 msgr2=0x7f6cdc078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:27.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.236+0000 7f6ce1a2e640 1 -- 172.21.15.28:0/2255226604 shutdown_connections 2024-04-21T08:51:27.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.236+0000 7f6ce1a2e640 1 -- 172.21.15.28:0/2255226604 wait complete. 2024-04-21T08:51:27.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.236+0000 7f6ce1a2e640 1 Processor -- start 2024-04-21T08:51:27.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.236+0000 7f6ce1a2e640 1 -- start start 2024-04-21T08:51:27.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6ce1a2e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6cdc06a8b0 0x7f6cdc19e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6ce1a2e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc107340 0x7f6cdc19e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6ce1a2e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6cdc198660 0x7f6cdc198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6ce1a2e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc112240 con 0x7f6cdc107340 2024-04-21T08:51:27.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6ce1a2e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc1120c0 con 0x7f6cdc06a8b0 2024-04-21T08:51:27.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6ce1a2e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc1123c0 con 0x7f6cdc198660 2024-04-21T08:51:27.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6cda7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc107340 0x7f6cdc19e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6cda7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc107340 0x7f6cdc19e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44594/0 (socket says 172.21.15.28:44594) 2024-04-21T08:51:27.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6cda7fc640 1 -- 172.21.15.28:0/2628167307 learned_addr learned my addr 172.21.15.28:0/2628167307 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:27.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6cdaffd640 1 --2- 172.21.15.28:0/2628167307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6cdc06a8b0 0x7f6cdc19e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6cdb7fe640 1 --2- 172.21.15.28:0/2628167307 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6cdc198660 0x7f6cdc198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6cda7fc640 1 -- 172.21.15.28:0/2628167307 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6cdc198660 msgr2=0x7f6cdc198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6cda7fc640 1 --2- 172.21.15.28:0/2628167307 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6cdc198660 0x7f6cdc198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6cda7fc640 1 -- 172.21.15.28:0/2628167307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6cdc06a8b0 msgr2=0x7f6cdc19e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6cda7fc640 1 --2- 172.21.15.28:0/2628167307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6cdc06a8b0 0x7f6cdc19e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6cda7fc640 1 -- 172.21.15.28:0/2628167307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cdc199180 con 0x7f6cdc107340 2024-04-21T08:51:27.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.237+0000 7f6cda7fc640 1 --2- 172.21.15.28:0/2628167307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc107340 0x7f6cdc19e9a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0005ab0 tx=0x7f6cd002c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:27.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.238+0000 7f6ce0a2c640 1 -- 172.21.15.28:0/2628167307 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0043020 con 0x7f6cdc107340 2024-04-21T08:51:27.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.238+0000 7f6ce1a2e640 1 -- 172.21.15.28:0/2628167307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cdc199410 con 0x7f6cdc107340 2024-04-21T08:51:27.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.238+0000 7f6ce0a2c640 1 -- 172.21.15.28:0/2628167307 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cd002caf0 con 0x7f6cdc107340 2024-04-21T08:51:27.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.238+0000 7f6ce0a2c640 1 -- 172.21.15.28:0/2628167307 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd00039f0 con 0x7f6cdc107340 2024-04-21T08:51:27.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.238+0000 7f6ce1a2e640 1 -- 172.21.15.28:0/2628167307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cdc1a5610 con 0x7f6cdc107340 2024-04-21T08:51:27.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.239+0000 7f6ce1a2e640 1 -- 172.21.15.28:0/2628167307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cdc069230 con 0x7f6cdc107340 2024-04-21T08:51:27.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.240+0000 7f6ce0a2c640 1 -- 172.21.15.28:0/2628167307 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f6cd0046020 con 0x7f6cdc107340 2024-04-21T08:51:27.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.241+0000 7f6ce0a2c640 1 --2- 172.21.15.28:0/2628167307 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f6cc40a3e40 0x7f6cc40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.241+0000 7f6ce0a2c640 1 -- 172.21.15.28:0/2628167307 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f6cd00f2d00 con 0x7f6cdc107340 2024-04-21T08:51:27.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.241+0000 7f6cdaffd640 1 --2- 172.21.15.28:0/2628167307 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f6cc40a3e40 0x7f6cc40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.242+0000 7f6cdaffd640 1 --2- 172.21.15.28:0/2628167307 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f6cc40a3e40 0x7f6cc40a6300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0095f0 tx=0x7f6ccc005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:27.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.246+0000 7f6ce0a2c640 1 -- 172.21.15.28:0/2628167307 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6cd004d020 con 0x7f6cdc107340 2024-04-21T08:51:27.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:26 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3781685349' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:27.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:26 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/425123260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:27.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:26 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3402173311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:27.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:26 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3781685349' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:51:27.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:26 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/425123260' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:51:27.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:26 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3402173311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:51:27.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.436+0000 7f6ce1a2e640 1 -- 172.21.15.28:0/2628167307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6cdc10a460 con 0x7f6cdc107340 2024-04-21T08:51:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.438+0000 7f6ce0a2c640 1 -- 172.21.15.28:0/2628167307 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25160 (secure 0 0 0) 0x7f6cd00beec0 con 0x7f6cdc107340 2024-04-21T08:51:27.437 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:27.438 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T08:51:10.823421+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1076282159":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6800/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/926389767":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/892059008":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/2132574390":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:51:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.440+0000 7f6ce1a2e640 1 -- 172.21.15.28:0/2628167307 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f6cc40a3e40 msgr2=0x7f6cc40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.440+0000 7f6ce1a2e640 1 --2- 172.21.15.28:0/2628167307 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f6cc40a3e40 0x7f6cc40a6300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0095f0 tx=0x7f6ccc005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.440+0000 7f6ce1a2e640 1 -- 172.21.15.28:0/2628167307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc107340 msgr2=0x7f6cdc19e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.440+0000 7f6ce1a2e640 1 --2- 172.21.15.28:0/2628167307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc107340 0x7f6cdc19e9a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0005ab0 tx=0x7f6cd002c660 comp rx=0 tx=0).stop 2024-04-21T08:51:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.440+0000 7f6ce1a2e640 1 -- 172.21.15.28:0/2628167307 shutdown_connections 2024-04-21T08:51:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.440+0000 7f6ce1a2e640 1 --2- 172.21.15.28:0/2628167307 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f6cc40a3e40 0x7f6cc40a6300 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.440+0000 7f6ce1a2e640 1 --2- 172.21.15.28:0/2628167307 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6cdc198660 0x7f6cdc198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.440+0000 7f6ce1a2e640 1 --2- 172.21.15.28:0/2628167307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc107340 0x7f6cdc19e9a0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.440+0000 7f6ce1a2e640 1 --2- 172.21.15.28:0/2628167307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6cdc06a8b0 0x7f6cdc19e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.440+0000 7f6ce1a2e640 1 -- 172.21.15.28:0/2628167307 >> 172.21.15.28:0/2628167307 conn(0x7f6cdc077e30 msgr2=0x7f6cdc0ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.440+0000 7f6ce1a2e640 1 -- 172.21.15.28:0/2628167307 shutdown_connections 2024-04-21T08:51:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.440+0000 7f6ce1a2e640 1 -- 172.21.15.28:0/2628167307 wait complete. 2024-04-21T08:51:27.451 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with random seed 11984 2024-04-21T08:51:27.451 DEBUG:tasks.quiescer:Yielding 2024-04-21T08:51:27.451 INFO:teuthology.run_tasks:Running task workunit... 2024-04-21T08:51:27.465 INFO:tasks.workunit:Pulling workunits from ref 4f6bc5273643aaf5660909e85400647a8951c10b 2024-04-21T08:51:27.465 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-21T08:51:27.466 INFO:tasks.workunit:timeout=3h 2024-04-21T08:51:27.466 INFO:tasks.workunit:cleanup=True 2024-04-21T08:51:27.466 DEBUG:teuthology.orchestra.run.smithi028:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-21T08:51:27.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:51:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-21T08:51:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-21T08:51:27.483 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-21T08:51:27.484 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:51:27.484 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-21T08:51:27.484 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-21 08:49:14.292900674 +0000 2024-04-21T08:51:27.484 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-21 08:49:14.292900674 +0000 2024-04-21T08:51:27.484 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-21 08:51:08.438657052 +0000 2024-04-21T08:51:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-21 08:49:14.292900674 +0000 2024-04-21T08:51:27.484 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-21T08:51:27.484 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-21T08:51:27.555 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/leonid-s-usov/ceph-ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4f6bc5273643aaf5660909e85400647a8951c10b 2024-04-21T08:51:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.572+0000 7fbf50bb6640 1 -- 172.21.15.28:0/2928049685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4c1011a0 msgr2=0x7fbf4c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.572+0000 7fbf50bb6640 1 --2- 172.21.15.28:0/2928049685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4c1011a0 0x7fbf4c101620 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009930 tx=0x7fbf3402bc20 comp rx=0 tx=0).stop 2024-04-21T08:51:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.572+0000 7fbf50bb6640 1 -- 172.21.15.28:0/2928049685 shutdown_connections 2024-04-21T08:51:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.572+0000 7fbf50bb6640 1 --2- 172.21.15.28:0/2928049685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf4c101b60 0x7fbf4c10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.572+0000 7fbf50bb6640 1 --2- 172.21.15.28:0/2928049685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4c1011a0 0x7fbf4c101620 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.572+0000 7fbf50bb6640 1 --2- 172.21.15.28:0/2928049685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf4c105e30 0x7fbf4c106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.572+0000 7fbf50bb6640 1 -- 172.21.15.28:0/2928049685 >> 172.21.15.28:0/2928049685 conn(0x7fbf4c077e30 msgr2=0x7fbf4c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.572+0000 7fbf50bb6640 1 -- 172.21.15.28:0/2928049685 shutdown_connections 2024-04-21T08:51:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.572+0000 7fbf50bb6640 1 -- 172.21.15.28:0/2928049685 wait complete. 2024-04-21T08:51:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.573+0000 7fbf50bb6640 1 Processor -- start 2024-04-21T08:51:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.573+0000 7fbf50bb6640 1 -- start start 2024-04-21T08:51:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.573+0000 7fbf50bb6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4c1011a0 0x7fbf4c0feaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.573+0000 7fbf50bb6640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf4c101b60 0x7fbf4c0fefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.573+0000 7fbf50bb6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf4c105e30 0x7fbf4c0ff520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.573+0000 7fbf50bb6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4c111110 con 0x7fbf4c1011a0 2024-04-21T08:51:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.573+0000 7fbf50bb6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4c110f90 con 0x7fbf4c105e30 2024-04-21T08:51:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.573+0000 7fbf50bb6640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4c111290 con 0x7fbf4c101b60 2024-04-21T08:51:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.573+0000 7fbf4a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4c1011a0 0x7fbf4c0feaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.573+0000 7fbf4a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4c1011a0 0x7fbf4c0feaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44620/0 (socket says 172.21.15.28:44620) 2024-04-21T08:51:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.573+0000 7fbf4a59c640 1 -- 172.21.15.28:0/2799053202 learned_addr learned my addr 172.21.15.28:0/2799053202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:27.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.574+0000 7fbf49d9b640 1 --2- 172.21.15.28:0/2799053202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf4c101b60 0x7fbf4c0fefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.574+0000 7fbf4ad9d640 1 --2- 172.21.15.28:0/2799053202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf4c105e30 0x7fbf4c0ff520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.574+0000 7fbf4a59c640 1 -- 172.21.15.28:0/2799053202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf4c101b60 msgr2=0x7fbf4c0fefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.574+0000 7fbf4a59c640 1 --2- 172.21.15.28:0/2799053202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf4c101b60 0x7fbf4c0fefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.574+0000 7fbf4a59c640 1 -- 172.21.15.28:0/2799053202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf4c105e30 msgr2=0x7fbf4c0ff520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.574+0000 7fbf4a59c640 1 --2- 172.21.15.28:0/2799053202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf4c105e30 0x7fbf4c0ff520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.574+0000 7fbf4a59c640 1 -- 172.21.15.28:0/2799053202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf4c1a6b10 con 0x7fbf4c1011a0 2024-04-21T08:51:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.574+0000 7fbf4a59c640 1 --2- 172.21.15.28:0/2799053202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4c1011a0 0x7fbf4c0feaa0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00c920 tx=0x7fbf3c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.574+0000 7fbf437fe640 1 -- 172.21.15.28:0/2799053202 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c007c70 con 0x7fbf4c1011a0 2024-04-21T08:51:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.574+0000 7fbf50bb6640 1 -- 172.21.15.28:0/2799053202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf4c1a6e00 con 0x7fbf4c1011a0 2024-04-21T08:51:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.574+0000 7fbf437fe640 1 -- 172.21.15.28:0/2799053202 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf3c007e10 con 0x7fbf4c1011a0 2024-04-21T08:51:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.574+0000 7fbf437fe640 1 -- 172.21.15.28:0/2799053202 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c018820 con 0x7fbf4c1011a0 2024-04-21T08:51:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.574+0000 7fbf50bb6640 1 -- 172.21.15.28:0/2799053202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf4c1a7310 con 0x7fbf4c1011a0 2024-04-21T08:51:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.575+0000 7fbf50bb6640 1 -- 172.21.15.28:0/2799053202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf4c068de0 con 0x7fbf4c1011a0 2024-04-21T08:51:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.577+0000 7fbf437fe640 1 -- 172.21.15.28:0/2799053202 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fbf3c00f450 con 0x7fbf4c1011a0 2024-04-21T08:51:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.577+0000 7fbf437fe640 1 --2- 172.21.15.28:0/2799053202 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbf380a3e70 0x7fbf380a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.578+0000 7fbf437fe640 1 -- 172.21.15.28:0/2799053202 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbf3c0d11d0 con 0x7fbf4c1011a0 2024-04-21T08:51:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.578+0000 7fbf49d9b640 1 --2- 172.21.15.28:0/2799053202 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbf380a3e70 0x7fbf380a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.578+0000 7fbf49d9b640 1 --2- 172.21.15.28:0/2799053202 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbf380a3e70 0x7fbf380a6330 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbf34036040 tx=0x7fbf34005fd0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.581+0000 7fbf437fe640 1 -- 172.21.15.28:0/2799053202 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf3c014030 con 0x7fbf4c1011a0 2024-04-21T08:51:27.589 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-21T08:51:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.810+0000 7fbf50bb6640 1 -- 172.21.15.28:0/2799053202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf4c0fff80 con 0x7fbf4c1011a0 2024-04-21T08:51:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.811+0000 7fbf437fe640 1 -- 172.21.15.28:0/2799053202 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7fbf3c09d310 con 0x7fbf4c1011a0 2024-04-21T08:51:27.810 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:27.811 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:51:09.181803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:51:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-04-21T08:51:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.813+0000 7fbf50bb6640 1 -- 172.21.15.28:0/2799053202 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbf380a3e70 msgr2=0x7fbf380a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.813+0000 7fbf50bb6640 1 --2- 172.21.15.28:0/2799053202 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbf380a3e70 0x7fbf380a6330 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbf34036040 tx=0x7fbf34005fd0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.813+0000 7fbf50bb6640 1 -- 172.21.15.28:0/2799053202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4c1011a0 msgr2=0x7fbf4c0feaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.813+0000 7fbf50bb6640 1 --2- 172.21.15.28:0/2799053202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4c1011a0 0x7fbf4c0feaa0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00c920 tx=0x7fbf3c00cde0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.813+0000 7fbf50bb6640 1 -- 172.21.15.28:0/2799053202 shutdown_connections 2024-04-21T08:51:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.813+0000 7fbf50bb6640 1 --2- 172.21.15.28:0/2799053202 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbf380a3e70 0x7fbf380a6330 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.813+0000 7fbf50bb6640 1 --2- 172.21.15.28:0/2799053202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf4c105e30 0x7fbf4c0ff520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.813+0000 7fbf50bb6640 1 --2- 172.21.15.28:0/2799053202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf4c101b60 0x7fbf4c0fefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.813+0000 7fbf50bb6640 1 --2- 172.21.15.28:0/2799053202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4c1011a0 0x7fbf4c0feaa0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.813+0000 7fbf50bb6640 1 -- 172.21.15.28:0/2799053202 >> 172.21.15.28:0/2799053202 conn(0x7fbf4c077e30 msgr2=0x7fbf4c069360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:27.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.813+0000 7fbf50bb6640 1 -- 172.21.15.28:0/2799053202 shutdown_connections 2024-04-21T08:51:27.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.813+0000 7fbf50bb6640 1 -- 172.21.15.28:0/2799053202 wait complete. 2024-04-21T08:51:27.825 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T08:48:45.911185+0000', 'modified': '2024-04-21T08:51:09.181803+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24488, 'mds_1': 24487, 'mds_2': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24419': {'gid': 24419, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.152:6835/3982885739', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6834', 'nonce': 3982885739}, {'type': 'v1', 'addr': '172.21.15.152:6835', 'nonce': 3982885739}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24452': {'gid': 24452, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.152:6837/1261226890', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6836', 'nonce': 1261226890}, {'type': 'v1', 'addr': '172.21.15.152:6837', 'nonce': 1261226890}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.171:6835/663606808', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6834', 'nonce': 663606808}, {'type': 'v1', 'addr': '172.21.15.171:6835', 'nonce': 663606808}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.171:6837/895402983', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6836', 'nonce': 895402983}, {'type': 'v1', 'addr': '172.21.15.171:6837', 'nonce': 895402983}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.171:6839/3423264764', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6838', 'nonce': 3423264764}, {'type': 'v1', 'addr': '172.21.15.171:6839', 'nonce': 3423264764}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.152:6839/3821365920', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6838', 'nonce': 3821365920}, {'type': 'v1', 'addr': '172.21.15.152:6839', 'nonce': 3821365920}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24488, 'qdb_cluster': [24488, 24463, 24487]} 2024-04-21T08:51:27.825 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-21T08:51:27.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-21T08:51:27.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.971+0000 7f68b6a70640 1 -- 172.21.15.28:0/2159365871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b01072e0 msgr2=0x7f68b010eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.971+0000 7f68b6a70640 1 --2- 172.21.15.28:0/2159365871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b01072e0 0x7f68b010eeb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009ca0 tx=0x7f68a802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.971+0000 7f68b6a70640 1 -- 172.21.15.28:0/2159365871 shutdown_connections 2024-04-21T08:51:27.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.971+0000 7f68b6a70640 1 --2- 172.21.15.28:0/2159365871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b01072e0 0x7f68b010eeb0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.971+0000 7f68b6a70640 1 --2- 172.21.15.28:0/2159365871 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68b0102bc0 0x7f68b0106c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.971+0000 7f68b6a70640 1 --2- 172.21.15.28:0/2159365871 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f68b0102210 0x7f68b01025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.971+0000 7f68b6a70640 1 -- 172.21.15.28:0/2159365871 >> 172.21.15.28:0/2159365871 conn(0x7f68b00fdab0 msgr2=0x7f68b00ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.971+0000 7f68b6a70640 1 -- 172.21.15.28:0/2159365871 shutdown_connections 2024-04-21T08:51:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.971+0000 7f68b6a70640 1 -- 172.21.15.28:0/2159365871 wait complete. 2024-04-21T08:51:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.971+0000 7f68b6a70640 1 Processor -- start 2024-04-21T08:51:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.971+0000 7f68b6a70640 1 -- start start 2024-04-21T08:51:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68b6a70640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f68b0102210 0x7f68b0074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68b6a70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b01072e0 0x7f68b0074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68b6a70640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68b0074cd0 0x7f68b01a6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68b6a70640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f68b0112250 con 0x7f68b01072e0 2024-04-21T08:51:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68b6a70640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f68b01120d0 con 0x7f68b0074cd0 2024-04-21T08:51:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68b6a70640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f68b01123d0 con 0x7f68b0102210 2024-04-21T08:51:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b01072e0 0x7f68b0074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b01072e0 0x7f68b0074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44632/0 (socket says 172.21.15.28:44632) 2024-04-21T08:51:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68a7fff640 1 -- 172.21.15.28:0/2443536463 learned_addr learned my addr 172.21.15.28:0/2443536463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68b500d640 1 --2- 172.21.15.28:0/2443536463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68b0074cd0 0x7f68b01a6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68b480c640 1 --2- 172.21.15.28:0/2443536463 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f68b0102210 0x7f68b0074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68a7fff640 1 -- 172.21.15.28:0/2443536463 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f68b0102210 msgr2=0x7f68b0074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68a7fff640 1 --2- 172.21.15.28:0/2443536463 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f68b0102210 0x7f68b0074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68a7fff640 1 -- 172.21.15.28:0/2443536463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68b0074cd0 msgr2=0x7f68b01a6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68a7fff640 1 --2- 172.21.15.28:0/2443536463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68b0074cd0 0x7f68b01a6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68a7fff640 1 -- 172.21.15.28:0/2443536463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68b01a70e0 con 0x7f68b01072e0 2024-04-21T08:51:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68a7fff640 1 --2- 172.21.15.28:0/2443536463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b01072e0 0x7f68b0074790 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f68a000e9c0 tx=0x7f68a000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68a5ffb640 1 -- 172.21.15.28:0/2443536463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a000cd80 con 0x7f68b01072e0 2024-04-21T08:51:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68a5ffb640 1 -- 172.21.15.28:0/2443536463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68a00054d0 con 0x7f68b01072e0 2024-04-21T08:51:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68b6a70640 1 -- 172.21.15.28:0/2443536463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68b01a73d0 con 0x7f68b01072e0 2024-04-21T08:51:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.972+0000 7f68a5ffb640 1 -- 172.21.15.28:0/2443536463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0010430 con 0x7f68b01072e0 2024-04-21T08:51:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.973+0000 7f68b6a70640 1 -- 172.21.15.28:0/2443536463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68b01a7970 con 0x7f68b01072e0 2024-04-21T08:51:27.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.975+0000 7f68a5ffb640 1 -- 172.21.15.28:0/2443536463 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f68a0004fd0 con 0x7f68b01072e0 2024-04-21T08:51:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.975+0000 7f68b965b740 1 Processor -- start 2024-04-21T08:51:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.975+0000 7f68b965b740 1 -- start start 2024-04-21T08:51:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.975+0000 7f68b965b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9fe7b0 0x56127e9feb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.975+0000 7f68b965b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127e9ff160 0x56127ea07710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.975+0000 7f68b965b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127ea07ce0 0x56127ea0a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.975+0000 7f68b965b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56127e962b10 con 0x56127e9ff160 2024-04-21T08:51:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.975+0000 7f68b965b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56127e900240 con 0x56127ea07ce0 2024-04-21T08:51:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.975+0000 7f68b965b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56127e95ead0 con 0x56127e9fe7b0 2024-04-21T08:51:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68b480c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127e9ff160 0x56127ea07710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68b480c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127e9ff160 0x56127ea07710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44638/0 (socket says 172.21.15.28:44638) 2024-04-21T08:51:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68b480c640 1 -- 172.21.15.28:0/1219786913 learned_addr learned my addr 172.21.15.28:0/1219786913 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68a5ffb640 1 --2- 172.21.15.28:0/2443536463 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f68980a3ec0 0x7f68980a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68a7fff640 1 --2- 172.21.15.28:0/1219786913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127ea07ce0 0x56127ea0a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68b500d640 1 --2- 172.21.15.28:0/1219786913 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9fe7b0 0x56127e9feb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68b500d640 1 --2- 172.21.15.28:0/2443536463 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f68980a3ec0 0x7f68980a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68b480c640 1 -- 172.21.15.28:0/1219786913 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9fe7b0 msgr2=0x56127e9feb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68b480c640 1 --2- 172.21.15.28:0/1219786913 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9fe7b0 0x56127e9feb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68b480c640 1 -- 172.21.15.28:0/1219786913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127ea07ce0 msgr2=0x56127ea0a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68b480c640 1 --2- 172.21.15.28:0/1219786913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127ea07ce0 0x56127ea0a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68b480c640 1 -- 172.21.15.28:0/1219786913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56127e6752d0 con 0x56127e9ff160 2024-04-21T08:51:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68a7fff640 1 --2- 172.21.15.28:0/1219786913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127ea07ce0 0x56127ea0a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68a5ffb640 1 -- 172.21.15.28:0/2443536463 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f68a00d1300 con 0x7f68b01072e0 2024-04-21T08:51:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68b480c640 1 --2- 172.21.15.28:0/1219786913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127e9ff160 0x56127ea07710 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009c90 tx=0x7f68ac0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e0a84db11462ff server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f6896ffd640 1 -- 172.21.15.28:0/1219786913 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac012450 con 0x56127e9ff160 2024-04-21T08:51:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.976+0000 7f68b500d640 1 --2- 172.21.15.28:0/2443536463 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f68980a3ec0 0x7f68980a6380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009430 tx=0x7f68a80095c0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.977+0000 7f6896ffd640 1 -- 172.21.15.28:0/1219786913 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68ac012aa0 con 0x56127e9ff160 2024-04-21T08:51:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.977+0000 7f6896ffd640 1 -- 172.21.15.28:0/1219786913 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac00fab0 con 0x56127e9ff160 2024-04-21T08:51:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.977+0000 7f68b965b740 1 -- 172.21.15.28:0/1219786913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127e9ff160 msgr2=0x56127ea07710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.977+0000 7f68b965b740 1 --2- 172.21.15.28:0/1219786913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127e9ff160 0x56127ea07710 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009c90 tx=0x7f68ac0117b0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.977+0000 7f68b965b740 1 -- 172.21.15.28:0/1219786913 shutdown_connections 2024-04-21T08:51:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.977+0000 7f68b965b740 1 --2- 172.21.15.28:0/1219786913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127ea07ce0 0x56127ea0a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.977+0000 7f68b965b740 1 --2- 172.21.15.28:0/1219786913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127e9ff160 0x56127ea07710 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.977+0000 7f68b965b740 1 --2- 172.21.15.28:0/1219786913 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9fe7b0 0x56127e9feb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.977+0000 7f68b965b740 1 -- 172.21.15.28:0/1219786913 >> 172.21.15.28:0/1219786913 conn(0x56127e9f8a60 msgr2=0x56127e9f8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.977+0000 7f68b965b740 1 -- 172.21.15.28:0/1219786913 shutdown_connections 2024-04-21T08:51:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.977+0000 7f68b965b740 1 -- 172.21.15.28:0/1219786913 wait complete. 2024-04-21T08:51:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.978+0000 7f68b965b740 1 -- start start 2024-04-21T08:51:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.978+0000 7f68b965b740 1 Processor -- start 2024-04-21T08:51:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.978+0000 7f68b965b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127e9fe7b0 0x56127ea04c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.978+0000 7f68b965b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9ff160 0x56127ea05140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.978+0000 7f68b965b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127ea07ce0 0x56127ea05680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.978+0000 7f68b965b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56127ea0ac50 con 0x56127e9fe7b0 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.978+0000 7f68b965b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56127e8eb190 con 0x56127e9ff160 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.978+0000 7f68b965b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56127ea0add0 con 0x56127ea07ce0 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.978+0000 7f68b965b740 10 client.0 _ll_register_callbacks cb 0x56127e8e68b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.978+0000 7f68b480c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127e9fe7b0 0x56127ea04c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.978+0000 7f68b480c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127e9fe7b0 0x56127ea04c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44648/0 (socket says 172.21.15.28:44648) 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.978+0000 7f68b480c640 1 -- 172.21.15.28:0/1253659670 learned_addr learned my addr 172.21.15.28:0/1253659670 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.978+0000 7f68a7fff640 1 --2- 172.21.15.28:0/1253659670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9ff160 0x56127ea05140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.978+0000 7f68b500d640 1 --2- 172.21.15.28:0/1253659670 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127ea07ce0 0x56127ea05680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.979+0000 7f68b480c640 1 -- 172.21.15.28:0/1253659670 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127ea07ce0 msgr2=0x56127ea05680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.979+0000 7f68b480c640 1 --2- 172.21.15.28:0/1253659670 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127ea07ce0 0x56127ea05680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.979+0000 7f68b480c640 1 -- 172.21.15.28:0/1253659670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9ff160 msgr2=0x56127ea05140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.979+0000 7f68b480c640 1 --2- 172.21.15.28:0/1253659670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9ff160 0x56127ea05140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.979+0000 7f68b480c640 1 -- 172.21.15.28:0/1253659670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56127e73f6b0 con 0x56127e9fe7b0 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.979+0000 7f68a7fff640 1 --2- 172.21.15.28:0/1253659670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9ff160 0x56127ea05140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.979+0000 7f68b480c640 1 --2- 172.21.15.28:0/1253659670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127e9fe7b0 0x56127ea04c00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f68ac01b000 tx=0x7f68ac012700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18622a93485e742 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.979+0000 7f688ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.979+0000 7f688ffff640 1 -- 172.21.15.28:0/1253659670 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ac028020 con 0x56127e9fe7b0 2024-04-21T08:51:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.979+0000 7f688ffff640 1 -- 172.21.15.28:0/1253659670 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f68ac00fa00 con 0x56127e9fe7b0 2024-04-21T08:51:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.979+0000 7f68b965b740 1 -- 172.21.15.28:0/1253659670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68ac012aa0 con 0x56127e9fe7b0 2024-04-21T08:51:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.979+0000 7f688ffff640 1 -- 172.21.15.28:0/1253659670 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f68ac033050 con 0x56127e9fe7b0 2024-04-21T08:51:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.979+0000 7f68b965b740 10 client.15867 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.979+0000 7f68b965b740 1 -- 172.21.15.28:0/1253659670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56127ea16ea0 con 0x56127e9fe7b0 2024-04-21T08:51:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.979+0000 7f688ffff640 1 -- 172.21.15.28:0/1253659670 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68ac012aa0 con 0x56127e9fe7b0 2024-04-21T08:51:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.980+0000 7f688ffff640 1 -- 172.21.15.28:0/1253659670 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f68ac0046c0 con 0x56127e9fe7b0 2024-04-21T08:51:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.980+0000 7f68b965b740 10 client.15867 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.980+0000 7f68b965b740 10 client.15867 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-21T08:51:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.980+0000 7f68b965b740 20 client.15867 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.980+0000 7f68b965b740 1 --2- 172.21.15.28:0/1253659670 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56127ea19180 0x56127ea1a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.980+0000 7f68b965b740 4 client.15867 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.980+0000 7f68b965b740 1 -- 172.21.15.28:0/1253659670 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56127e970530 con 0x56127ea19180 2024-04-21T08:51:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.980+0000 7f68a7fff640 1 --2- 172.21.15.28:0/1253659670 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56127ea19180 0x56127ea1a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.980+0000 7f688ffff640 10 client.15867 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.980+0000 7f68a7fff640 1 --2- 172.21.15.28:0/1253659670 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56127ea19180 0x56127ea1a5f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=99a2638a01e2cd06 server_cookie=8ea56d3162b442ac in_seq=0 out_seq=0 2024-04-21T08:51:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.981+0000 7f688ffff640 1 -- 172.21.15.28:0/1253659670 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56127e970530 con 0x56127ea19180 2024-04-21T08:51:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.981+0000 7f688ffff640 10 client.15867 handle_command_reply: tid=0 2024-04-21T08:51:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.981+0000 7f68b965b740 1 client.15867 shutdown 2024-04-21T08:51:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.981+0000 7f68b965b740 1 -- 172.21.15.28:0/1253659670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127e9fe7b0 msgr2=0x56127ea04c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.981+0000 7f68b965b740 1 --2- 172.21.15.28:0/1253659670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127e9fe7b0 0x56127ea04c00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f68ac01b000 tx=0x7f68ac012700 comp rx=0 tx=0).stop 2024-04-21T08:51:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.981+0000 7f68b965b740 1 -- 172.21.15.28:0/1253659670 shutdown_connections 2024-04-21T08:51:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.981+0000 7f68b965b740 1 --2- 172.21.15.28:0/1253659670 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56127ea19180 0x56127ea1a5f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.981+0000 7f68b965b740 1 --2- 172.21.15.28:0/1253659670 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127ea07ce0 0x56127ea05680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.981+0000 7f68b965b740 1 --2- 172.21.15.28:0/1253659670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9ff160 0x56127ea05140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.981+0000 7f68b965b740 1 --2- 172.21.15.28:0/1253659670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127e9fe7b0 0x56127ea04c00 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:27.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.981+0000 7f688ffff640 0 client.15867 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:27.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.981+0000 7f68b965b740 1 -- 172.21.15.28:0/1253659670 >> 172.21.15.28:0/1253659670 conn(0x56127e9f8a60 msgr2=0x56127e9fabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:27.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.982+0000 7f68b965b740 1 -- 172.21.15.28:0/1253659670 shutdown_connections 2024-04-21T08:51:27.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.982+0000 7f68b965b740 1 -- 172.21.15.28:0/1253659670 wait complete. 2024-04-21T08:51:27.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:27.982+0000 7f68b965b740 20 client.15867 trim_cache size 0 max 16384 2024-04-21T08:51:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b965b740 1 Processor -- start 2024-04-21T08:51:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b965b740 1 -- start start 2024-04-21T08:51:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b965b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9fe7b0 0x56127ea07330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b965b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9ff160 0x56127ea18ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b965b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127ea07ce0 0x56127ea054a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b965b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56127ea18930 con 0x56127ea07ce0 2024-04-21T08:51:28.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b965b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56127ea19000 con 0x56127e9fe7b0 2024-04-21T08:51:28.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b965b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56127ea196d0 con 0x56127e9ff160 2024-04-21T08:51:28.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b500d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127ea07ce0 0x56127ea054a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b500d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127ea07ce0 0x56127ea054a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44662/0 (socket says 172.21.15.28:44662) 2024-04-21T08:51:28.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b500d640 1 -- 172.21.15.28:0/252680892 learned_addr learned my addr 172.21.15.28:0/252680892 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:28.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68a7fff640 1 --2- 172.21.15.28:0/252680892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9ff160 0x56127ea18ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b480c640 1 --2- 172.21.15.28:0/252680892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9fe7b0 0x56127ea07330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b500d640 1 -- 172.21.15.28:0/252680892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9ff160 msgr2=0x56127ea18ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b500d640 1 --2- 172.21.15.28:0/252680892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9ff160 0x56127ea18ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b500d640 1 -- 172.21.15.28:0/252680892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9fe7b0 msgr2=0x56127ea07330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b500d640 1 --2- 172.21.15.28:0/252680892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9fe7b0 0x56127ea07330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b500d640 1 -- 172.21.15.28:0/252680892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56127ea007e0 con 0x56127ea07ce0 2024-04-21T08:51:28.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68a7fff640 1 --2- 172.21.15.28:0/252680892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9ff160 0x56127ea18ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:28.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.000+0000 7f68b480c640 1 --2- 172.21.15.28:0/252680892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9fe7b0 0x56127ea07330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:28.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.001+0000 7f68b500d640 1 --2- 172.21.15.28:0/252680892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127ea07ce0 0x56127ea054a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f68a8037040 tx=0x7f68a803f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca71333ef1f56d5b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:28.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.001+0000 7f68967fc640 1 -- 172.21.15.28:0/252680892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a8053070 con 0x56127ea07ce0 2024-04-21T08:51:28.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.001+0000 7f68967fc640 1 -- 172.21.15.28:0/252680892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68a8003e20 con 0x56127ea07ce0 2024-04-21T08:51:28.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.001+0000 7f68967fc640 1 -- 172.21.15.28:0/252680892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a8039040 con 0x56127ea07ce0 2024-04-21T08:51:28.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.002+0000 7f68b965b740 1 -- 172.21.15.28:0/252680892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127ea07ce0 msgr2=0x56127ea054a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.002+0000 7f68b965b740 1 --2- 172.21.15.28:0/252680892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127ea07ce0 0x56127ea054a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f68a8037040 tx=0x7f68a803f000 comp rx=0 tx=0).stop 2024-04-21T08:51:28.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.002+0000 7f68b965b740 1 -- 172.21.15.28:0/252680892 shutdown_connections 2024-04-21T08:51:28.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.002+0000 7f68b965b740 1 --2- 172.21.15.28:0/252680892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127ea07ce0 0x56127ea054a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.002+0000 7f68b965b740 1 --2- 172.21.15.28:0/252680892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9ff160 0x56127ea18ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.002+0000 7f68b965b740 1 --2- 172.21.15.28:0/252680892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9fe7b0 0x56127ea07330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.002+0000 7f68b965b740 1 -- 172.21.15.28:0/252680892 >> 172.21.15.28:0/252680892 conn(0x56127e9f8a60 msgr2=0x56127e9f3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:28.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.002+0000 7f68b965b740 1 -- 172.21.15.28:0/252680892 shutdown_connections 2024-04-21T08:51:28.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.002+0000 7f68b965b740 1 -- 172.21.15.28:0/252680892 wait complete. 2024-04-21T08:51:28.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.002+0000 7f68b965b740 1 -- start start 2024-04-21T08:51:28.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.002+0000 7f68b965b740 1 Processor -- start 2024-04-21T08:51:28.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b965b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9fe7b0 0x56127ea01b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b965b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9ff160 0x56127ea02070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b965b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127ea07ce0 0x56127ea025b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b965b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56127ea1a470 con 0x56127ea07ce0 2024-04-21T08:51:28.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b965b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56127ea19da0 con 0x56127e9ff160 2024-04-21T08:51:28.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b965b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56127ea1ab40 con 0x56127e9fe7b0 2024-04-21T08:51:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b965b740 10 client.0 _ll_register_callbacks cb 0x7f68ac009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b500d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127ea07ce0 0x56127ea025b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b500d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127ea07ce0 0x56127ea025b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44672/0 (socket says 172.21.15.28:44672) 2024-04-21T08:51:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b500d640 1 -- 172.21.15.28:0/171568419 learned_addr learned my addr 172.21.15.28:0/171568419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b480c640 1 --2- 172.21.15.28:0/171568419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9fe7b0 0x56127ea01b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68a7fff640 1 --2- 172.21.15.28:0/171568419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9ff160 0x56127ea02070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b500d640 1 -- 172.21.15.28:0/171568419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9fe7b0 msgr2=0x56127ea01b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b500d640 1 --2- 172.21.15.28:0/171568419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9fe7b0 0x56127ea01b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b500d640 1 -- 172.21.15.28:0/171568419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9ff160 msgr2=0x56127ea02070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b500d640 1 --2- 172.21.15.28:0/171568419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9ff160 0x56127ea02070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b500d640 1 -- 172.21.15.28:0/171568419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56127ea20ea0 con 0x56127ea07ce0 2024-04-21T08:51:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f68b500d640 1 --2- 172.21.15.28:0/171568419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127ea07ce0 0x56127ea025b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f68a8006010 tx=0x7f68a80098d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e038fa9ff8be594 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f688ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.003+0000 7f688ffff640 1 -- 172.21.15.28:0/171568419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a8053070 con 0x56127ea07ce0 2024-04-21T08:51:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.004+0000 7f688ffff640 1 -- 172.21.15.28:0/171568419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f68a8004200 con 0x56127ea07ce0 2024-04-21T08:51:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.004+0000 7f68b965b740 1 -- 172.21.15.28:0/171568419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68a8003e20 con 0x56127ea07ce0 2024-04-21T08:51:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.004+0000 7f688ffff640 1 -- 172.21.15.28:0/171568419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f68a8039040 con 0x56127ea07ce0 2024-04-21T08:51:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.004+0000 7f68b965b740 10 client.15873 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.004+0000 7f68b965b740 1 -- 172.21.15.28:0/171568419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56127ea7f580 con 0x56127ea07ce0 2024-04-21T08:51:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.004+0000 7f688ffff640 1 -- 172.21.15.28:0/171568419 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68a8003e20 con 0x56127ea07ce0 2024-04-21T08:51:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.004+0000 7f688ffff640 1 -- 172.21.15.28:0/171568419 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f68a8034440 con 0x56127ea07ce0 2024-04-21T08:51:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.004+0000 7f68b965b740 10 client.15873 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.004+0000 7f68b965b740 10 client.15873 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-21T08:51:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.004+0000 7f68b965b740 20 client.15873 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.004+0000 7f68b965b740 1 --2- 172.21.15.28:0/171568419 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56127ea83c00 0x56127ea82910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.004+0000 7f68b965b740 4 client.15873 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:51:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.004+0000 7f68b965b740 1 -- 172.21.15.28:0/171568419 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "status"}) -- 0x56127e96a3a0 con 0x56127ea83c00 2024-04-21T08:51:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.004+0000 7f68b480c640 1 --2- 172.21.15.28:0/171568419 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56127ea83c00 0x56127ea82910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f68b480c640 1 --2- 172.21.15.28:0/171568419 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56127ea83c00 0x56127ea82910 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a2b39031eab3b6e3 server_cookie=ec44f0f0933c7b82 in_seq=0 out_seq=0 2024-04-21T08:51:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f688ffff640 10 client.15873 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f688ffff640 1 -- 172.21.15.28:0/171568419 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56127e96a3a0 con 0x56127ea83c00 2024-04-21T08:51:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f688ffff640 10 client.15873 handle_command_reply: tid=0 2024-04-21T08:51:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f68b965b740 1 client.15873 shutdown 2024-04-21T08:51:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f68b965b740 1 -- 172.21.15.28:0/171568419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127ea07ce0 msgr2=0x56127ea025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f68b965b740 1 --2- 172.21.15.28:0/171568419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127ea07ce0 0x56127ea025b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f68a8006010 tx=0x7f68a80098d0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f68b965b740 1 -- 172.21.15.28:0/171568419 shutdown_connections 2024-04-21T08:51:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f68b965b740 1 --2- 172.21.15.28:0/171568419 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56127ea83c00 0x56127ea82910 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f68b965b740 1 --2- 172.21.15.28:0/171568419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56127ea07ce0 0x56127ea025b0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f68b965b740 1 --2- 172.21.15.28:0/171568419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56127e9ff160 0x56127ea02070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f68b965b740 1 --2- 172.21.15.28:0/171568419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56127e9fe7b0 0x56127ea01b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f68b965b740 1 -- 172.21.15.28:0/171568419 >> 172.21.15.28:0/171568419 conn(0x56127e9f8a60 msgr2=0x56127e9f7170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:28.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f688ffff640 0 client.15873 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:28.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f68b965b740 1 -- 172.21.15.28:0/171568419 shutdown_connections 2024-04-21T08:51:28.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f68b965b740 1 -- 172.21.15.28:0/171568419 wait complete. 2024-04-21T08:51:28.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.005+0000 7f68b965b740 20 client.15873 trim_cache size 0 max 16384 2024-04-21T08:51:28.014 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:51:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "b51a76c8-ffba-11ee-bc93-c7b262605968", 2024-04-21T08:51:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-04-21T08:51:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24463, 2024-04-21T08:51:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-21T08:51:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-21T08:51:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-21T08:51:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 136.170522557, 2024-04-21T08:51:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 25, 2024-04-21T08:51:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 94, 2024-04-21T08:51:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 93, 2024-04-21T08:51:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 180.89322507599999 2024-04-21T08:51:28.015 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:51:28.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.006+0000 7f68b6a70640 1 -- 172.21.15.28:0/2443536463 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f68980a3ec0 msgr2=0x7f68980a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.006+0000 7f68b6a70640 1 --2- 172.21.15.28:0/2443536463 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f68980a3ec0 0x7f68980a6380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009430 tx=0x7f68a80095c0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.006+0000 7f68b6a70640 1 -- 172.21.15.28:0/2443536463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b01072e0 msgr2=0x7f68b0074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.006+0000 7f68b6a70640 1 --2- 172.21.15.28:0/2443536463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b01072e0 0x7f68b0074790 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f68a000e9c0 tx=0x7f68a000ee80 comp rx=0 tx=0).stop 2024-04-21T08:51:28.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.006+0000 7f68b6a70640 1 -- 172.21.15.28:0/2443536463 shutdown_connections 2024-04-21T08:51:28.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.006+0000 7f68b6a70640 1 --2- 172.21.15.28:0/2443536463 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f68980a3ec0 0x7f68980a6380 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.006+0000 7f68b6a70640 1 --2- 172.21.15.28:0/2443536463 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68b0074cd0 0x7f68b01a6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.006+0000 7f68b6a70640 1 --2- 172.21.15.28:0/2443536463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b01072e0 0x7f68b0074790 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.006+0000 7f68b6a70640 1 --2- 172.21.15.28:0/2443536463 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f68b0102210 0x7f68b0074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.006+0000 7f68b6a70640 1 -- 172.21.15.28:0/2443536463 >> 172.21.15.28:0/2443536463 conn(0x7f68b00fdab0 msgr2=0x7f68b00ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:28.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.006+0000 7f68b6a70640 1 -- 172.21.15.28:0/2443536463 shutdown_connections 2024-04-21T08:51:28.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.006+0000 7f68b6a70640 1 -- 172.21.15.28:0/2443536463 wait complete. 2024-04-21T08:51:28.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-21T08:51:28.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:27 smithi028 ceph-mon[28003]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-21T08:51:28.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:27 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1162094011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:28.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:27 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2628167307' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:28.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:27 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2799053202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.124+0000 7fb98e350640 1 -- 172.21.15.28:0/3171410934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb988106160 msgr2=0x7fb98810dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.125+0000 7fb98e350640 1 --2- 172.21.15.28:0/3171410934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb988106160 0x7fb98810dd30 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb980009ca0 tx=0x7fb98002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.125+0000 7fb98e350640 1 -- 172.21.15.28:0/3171410934 shutdown_connections 2024-04-21T08:51:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.125+0000 7fb98e350640 1 --2- 172.21.15.28:0/3171410934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb988106160 0x7fb98810dd30 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.125+0000 7fb98e350640 1 --2- 172.21.15.28:0/3171410934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb988101a40 0x7fb988105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.125+0000 7fb98e350640 1 --2- 172.21.15.28:0/3171410934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb988101090 0x7fb988101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.125+0000 7fb98e350640 1 -- 172.21.15.28:0/3171410934 >> 172.21.15.28:0/3171410934 conn(0x7fb988077f50 msgr2=0x7fb988078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.125+0000 7fb98e350640 1 -- 172.21.15.28:0/3171410934 shutdown_connections 2024-04-21T08:51:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.125+0000 7fb98e350640 1 -- 172.21.15.28:0/3171410934 wait complete. 2024-04-21T08:51:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.125+0000 7fb98e350640 1 Processor -- start 2024-04-21T08:51:28.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.125+0000 7fb98e350640 1 -- start start 2024-04-21T08:51:28.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb98e350640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb988101090 0x7fb9880fec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb98e350640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb988106160 0x7fb9880ff1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb98e350640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9880ff700 0x7fb9881a6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb98e350640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9881110b0 con 0x7fb988106160 2024-04-21T08:51:28.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb98e350640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb988110f30 con 0x7fb9880ff700 2024-04-21T08:51:28.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb98e350640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb988111230 con 0x7fb988101090 2024-04-21T08:51:28.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb9877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb988106160 0x7fb9880ff1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb9877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb988106160 0x7fb9880ff1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44684/0 (socket says 172.21.15.28:44684) 2024-04-21T08:51:28.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb987fff640 1 --2- 172.21.15.28:0/2285543933 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb988101090 0x7fb9880fec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb9877fe640 1 -- 172.21.15.28:0/2285543933 learned_addr learned my addr 172.21.15.28:0/2285543933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:28.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb98c8ed640 1 --2- 172.21.15.28:0/2285543933 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9880ff700 0x7fb9881a6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb98c8ed640 1 -- 172.21.15.28:0/2285543933 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb988101090 msgr2=0x7fb9880fec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb98c8ed640 1 --2- 172.21.15.28:0/2285543933 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb988101090 0x7fb9880fec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb98c8ed640 1 -- 172.21.15.28:0/2285543933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb988106160 msgr2=0x7fb9880ff1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb98c8ed640 1 --2- 172.21.15.28:0/2285543933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb988106160 0x7fb9880ff1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb98c8ed640 1 -- 172.21.15.28:0/2285543933 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9881a6f30 con 0x7fb9880ff700 2024-04-21T08:51:28.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb9877fe640 1 --2- 172.21.15.28:0/2285543933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb988106160 0x7fb9880ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:28.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.126+0000 7fb98c8ed640 1 --2- 172.21.15.28:0/2285543933 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9880ff700 0x7fb9881a6830 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb98002c700 tx=0x7fb980034760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:28.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.127+0000 7fb9857fa640 1 -- 172.21.15.28:0/2285543933 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb980039040 con 0x7fb9880ff700 2024-04-21T08:51:28.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.127+0000 7fb98e350640 1 -- 172.21.15.28:0/2285543933 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9881a71c0 con 0x7fb9880ff700 2024-04-21T08:51:28.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.127+0000 7fb9857fa640 1 -- 172.21.15.28:0/2285543933 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb980044070 con 0x7fb9880ff700 2024-04-21T08:51:28.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.127+0000 7fb9857fa640 1 -- 172.21.15.28:0/2285543933 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb980004040 con 0x7fb9880ff700 2024-04-21T08:51:28.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.127+0000 7fb98e350640 1 -- 172.21.15.28:0/2285543933 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9881a76d0 con 0x7fb9880ff700 2024-04-21T08:51:28.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.129+0000 7fb99101e740 1 Processor -- start 2024-04-21T08:51:28.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.129+0000 7fb9857fa640 1 -- 172.21.15.28:0/2285543933 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fb9800324c0 con 0x7fb9880ff700 2024-04-21T08:51:28.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.129+0000 7fb99101e740 1 -- start start 2024-04-21T08:51:28.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb99101e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55891245b7b0 0x55891245bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb99101e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912464710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb99101e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558912464ce0 0x558912467130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb99101e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5589123bfb10 con 0x55891245c160 2024-04-21T08:51:28.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb99101e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55891235d240 con 0x55891245b7b0 2024-04-21T08:51:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb99101e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5589123bbad0 con 0x558912464ce0 2024-04-21T08:51:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb987fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912464710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb987fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912464710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44686/0 (socket says 172.21.15.28:44686) 2024-04-21T08:51:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb987fff640 1 -- 172.21.15.28:0/1747373978 learned_addr learned my addr 172.21.15.28:0/1747373978 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb9877fe640 1 --2- 172.21.15.28:0/1747373978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55891245b7b0 0x55891245bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb9877fe640 1 --2- 172.21.15.28:0/1747373978 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558912464ce0 0x558912467130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb987fff640 1 -- 172.21.15.28:0/1747373978 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558912464ce0 msgr2=0x558912467130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb987fff640 1 --2- 172.21.15.28:0/1747373978 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558912464ce0 0x558912467130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb987fff640 1 -- 172.21.15.28:0/1747373978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55891245b7b0 msgr2=0x55891245bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb987fff640 1 --2- 172.21.15.28:0/1747373978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55891245b7b0 0x55891245bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb987fff640 1 -- 172.21.15.28:0/1747373978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589120d22d0 con 0x55891245c160 2024-04-21T08:51:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb9857fa640 1 --2- 172.21.15.28:0/2285543933 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb9700a3e70 0x7fb9700a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb9877fe640 1 --2- 172.21.15.28:0/1747373978 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558912464ce0 0x558912467130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb98c8ed640 1 --2- 172.21.15.28:0/2285543933 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb9700a3e70 0x7fb9700a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb987fff640 1 --2- 172.21.15.28:0/1747373978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912464710 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb97800ad00 tx=0x7fb97800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24c022daf160cd4d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb9857fa640 1 -- 172.21.15.28:0/2285543933 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fb9800f2da0 con 0x7fb9880ff700 2024-04-21T08:51:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.130+0000 7fb96e7fc640 1 -- 172.21.15.28:0/1747373978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97801d070 con 0x55891245c160 2024-04-21T08:51:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.131+0000 7fb96e7fc640 1 -- 172.21.15.28:0/1747373978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb978018af0 con 0x55891245c160 2024-04-21T08:51:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.131+0000 7fb96e7fc640 1 -- 172.21.15.28:0/1747373978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb978010b30 con 0x55891245c160 2024-04-21T08:51:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.131+0000 7fb99101e740 1 -- 172.21.15.28:0/1747373978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 msgr2=0x558912464710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.131+0000 7fb98c8ed640 1 --2- 172.21.15.28:0/2285543933 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb9700a3e70 0x7fb9700a6330 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb9800f00e0 tx=0x7fb9800b3c90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.131+0000 7fb99101e740 1 --2- 172.21.15.28:0/1747373978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912464710 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb97800ad00 tx=0x7fb97800f790 comp rx=0 tx=0).stop 2024-04-21T08:51:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.131+0000 7fb99101e740 1 -- 172.21.15.28:0/1747373978 shutdown_connections 2024-04-21T08:51:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.131+0000 7fb99101e740 1 --2- 172.21.15.28:0/1747373978 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558912464ce0 0x558912467130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.131+0000 7fb99101e740 1 --2- 172.21.15.28:0/1747373978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912464710 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.131+0000 7fb99101e740 1 --2- 172.21.15.28:0/1747373978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55891245b7b0 0x55891245bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.131+0000 7fb99101e740 1 -- 172.21.15.28:0/1747373978 >> 172.21.15.28:0/1747373978 conn(0x558912455a60 msgr2=0x558912455e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.131+0000 7fb99101e740 1 -- 172.21.15.28:0/1747373978 shutdown_connections 2024-04-21T08:51:28.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.131+0000 7fb99101e740 1 -- 172.21.15.28:0/1747373978 wait complete. 2024-04-21T08:51:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb99101e740 1 -- start start 2024-04-21T08:51:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb99101e740 1 Processor -- start 2024-04-21T08:51:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb99101e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55891245b7b0 0x558912461d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb99101e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912462250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb99101e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558912464ce0 0x558912462790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb99101e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558912467c50 con 0x55891245c160 2024-04-21T08:51:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb99101e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558912348190 con 0x558912464ce0 2024-04-21T08:51:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb99101e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558912467dd0 con 0x55891245b7b0 2024-04-21T08:51:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb99101e740 10 client.0 _ll_register_callbacks cb 0x5589123438b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb987fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912462250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb987fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912462250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44702/0 (socket says 172.21.15.28:44702) 2024-04-21T08:51:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb987fff640 1 -- 172.21.15.28:0/184369379 learned_addr learned my addr 172.21.15.28:0/184369379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb9877fe640 1 --2- 172.21.15.28:0/184369379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558912464ce0 0x558912462790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb9877fe640 1 --2- 172.21.15.28:0/184369379 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55891245b7b0 0x558912461d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb987fff640 1 -- 172.21.15.28:0/184369379 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55891245b7b0 msgr2=0x558912461d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.132+0000 7fb987fff640 1 --2- 172.21.15.28:0/184369379 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55891245b7b0 0x558912461d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.133+0000 7fb987fff640 1 -- 172.21.15.28:0/184369379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558912464ce0 msgr2=0x558912462790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.133+0000 7fb987fff640 1 --2- 172.21.15.28:0/184369379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558912464ce0 0x558912462790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.133+0000 7fb987fff640 1 -- 172.21.15.28:0/184369379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55891219c6b0 con 0x55891245c160 2024-04-21T08:51:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.133+0000 7fb9877fe640 1 --2- 172.21.15.28:0/184369379 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55891245b7b0 0x558912461d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.133+0000 7fb987fff640 1 --2- 172.21.15.28:0/184369379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912462250 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb978000c00 tx=0x7fb9780186f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcfe934ae6f9ca53 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.133+0000 7fb9677fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.133+0000 7fb9677fe640 1 -- 172.21.15.28:0/184369379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97801d040 con 0x55891245c160 2024-04-21T08:51:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.133+0000 7fb9677fe640 1 -- 172.21.15.28:0/184369379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fb978010870 con 0x55891245c160 2024-04-21T08:51:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.133+0000 7fb99101e740 1 -- 172.21.15.28:0/184369379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb978018af0 con 0x55891245c160 2024-04-21T08:51:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.133+0000 7fb9677fe640 1 -- 172.21.15.28:0/184369379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb978031050 con 0x55891245c160 2024-04-21T08:51:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.133+0000 7fb99101e740 10 client.15885 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.133+0000 7fb99101e740 1 -- 172.21.15.28:0/184369379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558912473ef0 con 0x55891245c160 2024-04-21T08:51:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.133+0000 7fb9677fe640 1 -- 172.21.15.28:0/184369379 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb978018af0 con 0x55891245c160 2024-04-21T08:51:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.133+0000 7fb9677fe640 1 -- 172.21.15.28:0/184369379 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb978020030 con 0x55891245c160 2024-04-21T08:51:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.133+0000 7fb99101e740 10 client.15885 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.134+0000 7fb99101e740 10 client.15885 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T08:51:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.134+0000 7fb99101e740 20 client.15885 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.134+0000 7fb99101e740 1 --2- 172.21.15.28:0/184369379 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558912476100 0x558912477570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.134+0000 7fb99101e740 4 client.15885 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.134+0000 7fb99101e740 1 -- 172.21.15.28:0/184369379 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589123cd530 con 0x558912476100 2024-04-21T08:51:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.134+0000 7fb9877fe640 1 --2- 172.21.15.28:0/184369379 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558912476100 0x558912477570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.134+0000 7fb9877fe640 1 --2- 172.21.15.28:0/184369379 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558912476100 0x558912477570 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e035d569dbc8651d server_cookie=a3b64a991ae09058 in_seq=0 out_seq=0 2024-04-21T08:51:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.134+0000 7fb9677fe640 10 client.15885 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:28.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.135+0000 7fb9677fe640 1 -- 172.21.15.28:0/184369379 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5589123cd530 con 0x558912476100 2024-04-21T08:51:28.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.135+0000 7fb9677fe640 10 client.15885 handle_command_reply: tid=0 2024-04-21T08:51:28.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.135+0000 7fb99101e740 1 client.15885 shutdown 2024-04-21T08:51:28.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.135+0000 7fb99101e740 1 -- 172.21.15.28:0/184369379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 msgr2=0x558912462250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.135+0000 7fb99101e740 1 --2- 172.21.15.28:0/184369379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912462250 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb978000c00 tx=0x7fb9780186f0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.135+0000 7fb99101e740 1 -- 172.21.15.28:0/184369379 shutdown_connections 2024-04-21T08:51:28.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.135+0000 7fb99101e740 1 --2- 172.21.15.28:0/184369379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558912464ce0 0x558912462790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.135+0000 7fb99101e740 1 --2- 172.21.15.28:0/184369379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912462250 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.135+0000 7fb99101e740 1 --2- 172.21.15.28:0/184369379 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558912476100 0x558912477570 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.135+0000 7fb99101e740 1 --2- 172.21.15.28:0/184369379 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55891245b7b0 0x558912461d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.135+0000 7fb99101e740 1 -- 172.21.15.28:0/184369379 >> 172.21.15.28:0/184369379 conn(0x558912455a60 msgr2=0x558912457bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.135+0000 7fb9677fe640 0 client.15885 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.136+0000 7fb99101e740 1 -- 172.21.15.28:0/184369379 shutdown_connections 2024-04-21T08:51:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.136+0000 7fb99101e740 1 -- 172.21.15.28:0/184369379 wait complete. 2024-04-21T08:51:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.136+0000 7fb99101e740 20 client.15885 trim_cache size 0 max 16384 2024-04-21T08:51:28.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.154+0000 7fb99101e740 1 Processor -- start 2024-04-21T08:51:28.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.154+0000 7fb99101e740 1 -- start start 2024-04-21T08:51:28.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.154+0000 7fb99101e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558912455a60 0x558912464440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.154+0000 7fb99101e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55891245b7b0 0x558912475a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.154+0000 7fb99101e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x5589124625b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.154+0000 7fb99101e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5589124758b0 con 0x55891245c160 2024-04-21T08:51:28.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.154+0000 7fb99101e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558912475f80 con 0x55891245b7b0 2024-04-21T08:51:28.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.154+0000 7fb99101e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558912476650 con 0x558912455a60 2024-04-21T08:51:28.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.154+0000 7fb9877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x5589124625b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.154+0000 7fb9877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x5589124625b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44706/0 (socket says 172.21.15.28:44706) 2024-04-21T08:51:28.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.154+0000 7fb9877fe640 1 -- 172.21.15.28:0/2671827395 learned_addr learned my addr 172.21.15.28:0/2671827395 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:28.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.155+0000 7fb9877fe640 1 --2- 172.21.15.28:0/2671827395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558912455a60 0x558912464440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.155+0000 7fb9877fe640 1 -- 172.21.15.28:0/2671827395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558912455a60 msgr2=0x558912464440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.155+0000 7fb9877fe640 1 --2- 172.21.15.28:0/2671827395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558912455a60 0x558912464440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.155+0000 7fb9877fe640 1 -- 172.21.15.28:0/2671827395 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55891245b7b0 msgr2=0x558912475a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.155+0000 7fb987fff640 1 --2- 172.21.15.28:0/2671827395 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55891245b7b0 0x558912475a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.155+0000 7fb9877fe640 1 --2- 172.21.15.28:0/2671827395 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55891245b7b0 0x558912475a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.155+0000 7fb9877fe640 1 -- 172.21.15.28:0/2671827395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558912460260 con 0x55891245c160 2024-04-21T08:51:28.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.155+0000 7fb987fff640 1 --2- 172.21.15.28:0/2671827395 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55891245b7b0 0x558912475a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:28.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.155+0000 7fb9877fe640 1 --2- 172.21.15.28:0/2671827395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x5589124625b0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00c010 tx=0x7fb97c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=686b0856f436a180 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.155+0000 7fb96dffb640 1 -- 172.21.15.28:0/2671827395 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c003c00 con 0x55891245c160 2024-04-21T08:51:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.155+0000 7fb96dffb640 1 -- 172.21.15.28:0/2671827395 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb97c003da0 con 0x55891245c160 2024-04-21T08:51:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.155+0000 7fb96dffb640 1 -- 172.21.15.28:0/2671827395 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c004da0 con 0x55891245c160 2024-04-21T08:51:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.156+0000 7fb99101e740 1 -- 172.21.15.28:0/2671827395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 msgr2=0x5589124625b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.156+0000 7fb99101e740 1 --2- 172.21.15.28:0/2671827395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x5589124625b0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00c010 tx=0x7fb97c00b480 comp rx=0 tx=0).stop 2024-04-21T08:51:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.156+0000 7fb99101e740 1 -- 172.21.15.28:0/2671827395 shutdown_connections 2024-04-21T08:51:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.156+0000 7fb99101e740 1 --2- 172.21.15.28:0/2671827395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x5589124625b0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.156+0000 7fb99101e740 1 --2- 172.21.15.28:0/2671827395 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55891245b7b0 0x558912475a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.156+0000 7fb99101e740 1 --2- 172.21.15.28:0/2671827395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558912455a60 0x558912464440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.156+0000 7fb99101e740 1 -- 172.21.15.28:0/2671827395 >> 172.21.15.28:0/2671827395 conn(0x55891245c8b0 msgr2=0x558912450800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.156+0000 7fb99101e740 1 -- 172.21.15.28:0/2671827395 shutdown_connections 2024-04-21T08:51:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.156+0000 7fb99101e740 1 -- 172.21.15.28:0/2671827395 wait complete. 2024-04-21T08:51:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.156+0000 7fb99101e740 1 -- start start 2024-04-21T08:51:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.156+0000 7fb99101e740 1 Processor -- start 2024-04-21T08:51:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb99101e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558912455a60 0x5589124791d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb99101e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55891245b7b0 0x558912479710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb99101e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912479c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb99101e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5589124773f0 con 0x55891245c160 2024-04-21T08:51:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb99101e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558912476d20 con 0x558912455a60 2024-04-21T08:51:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb99101e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558912477ac0 con 0x55891245b7b0 2024-04-21T08:51:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb99101e740 10 client.0 _ll_register_callbacks cb 0x7fb97801b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb9877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912479c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb9877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912479c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44712/0 (socket says 172.21.15.28:44712) 2024-04-21T08:51:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb9877fe640 1 -- 172.21.15.28:0/246658638 learned_addr learned my addr 172.21.15.28:0/246658638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb987fff640 1 --2- 172.21.15.28:0/246658638 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55891245b7b0 0x558912479710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb9877fe640 1 --2- 172.21.15.28:0/246658638 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558912455a60 0x5589124791d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb9877fe640 1 -- 172.21.15.28:0/246658638 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55891245b7b0 msgr2=0x558912479710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb9877fe640 1 --2- 172.21.15.28:0/246658638 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55891245b7b0 0x558912479710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb9877fe640 1 -- 172.21.15.28:0/246658638 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558912455a60 msgr2=0x5589124791d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb9877fe640 1 --2- 172.21.15.28:0/246658638 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558912455a60 0x5589124791d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb9877fe640 1 -- 172.21.15.28:0/246658638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55891247d990 con 0x55891245c160 2024-04-21T08:51:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb9877fe640 1 --2- 172.21.15.28:0/246658638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912479c50 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb97c003d60 tx=0x7fb97c01da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb0be38d270824f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb9677fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb9677fe640 1 -- 172.21.15.28:0/246658638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c012660 con 0x55891245c160 2024-04-21T08:51:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb9677fe640 1 -- 172.21.15.28:0/246658638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fb97c012800 con 0x55891245c160 2024-04-21T08:51:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.157+0000 7fb99101e740 1 -- 172.21.15.28:0/246658638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb97c003da0 con 0x55891245c160 2024-04-21T08:51:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.158+0000 7fb9677fe640 1 -- 172.21.15.28:0/246658638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb97c03b050 con 0x55891245c160 2024-04-21T08:51:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.158+0000 7fb99101e740 10 client.15891 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.158+0000 7fb99101e740 1 -- 172.21.15.28:0/246658638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5589124dc290 con 0x55891245c160 2024-04-21T08:51:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.158+0000 7fb9677fe640 1 -- 172.21.15.28:0/246658638 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb97c003da0 con 0x55891245c160 2024-04-21T08:51:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.158+0000 7fb9677fe640 1 -- 172.21.15.28:0/246658638 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb97c029910 con 0x55891245c160 2024-04-21T08:51:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.158+0000 7fb99101e740 10 client.15891 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.158+0000 7fb99101e740 10 client.15891 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-21T08:51:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.158+0000 7fb99101e740 20 client.15891 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.158+0000 7fb99101e740 1 --2- 172.21.15.28:0/246658638 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558912464ce0 0x5589124df730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.158+0000 7fb99101e740 4 client.15891 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:51:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.158+0000 7fb99101e740 1 -- 172.21.15.28:0/246658638 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "status"}) -- 0x5589123c73a0 con 0x558912464ce0 2024-04-21T08:51:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.158+0000 7fb987fff640 1 --2- 172.21.15.28:0/246658638 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558912464ce0 0x5589124df730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb9677fe640 10 client.15891 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb987fff640 1 --2- 172.21.15.28:0/246658638 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558912464ce0 0x5589124df730 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=699de076d1813a55 server_cookie=f1955eb544a80ae8 in_seq=0 out_seq=0 2024-04-21T08:51:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb9677fe640 1 -- 172.21.15.28:0/246658638 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5589123c73a0 con 0x558912464ce0 2024-04-21T08:51:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb9677fe640 10 client.15891 handle_command_reply: tid=0 2024-04-21T08:51:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb99101e740 1 client.15891 shutdown 2024-04-21T08:51:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb99101e740 1 -- 172.21.15.28:0/246658638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 msgr2=0x558912479c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb99101e740 1 --2- 172.21.15.28:0/246658638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912479c50 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb97c003d60 tx=0x7fb97c01da40 comp rx=0 tx=0).stop 2024-04-21T08:51:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb99101e740 1 -- 172.21.15.28:0/246658638 shutdown_connections 2024-04-21T08:51:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb99101e740 1 --2- 172.21.15.28:0/246658638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55891245c160 0x558912479c50 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb99101e740 1 --2- 172.21.15.28:0/246658638 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558912464ce0 0x5589124df730 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb99101e740 1 --2- 172.21.15.28:0/246658638 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55891245b7b0 0x558912479710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb99101e740 1 --2- 172.21.15.28:0/246658638 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558912455a60 0x5589124791d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb9677fe640 0 client.15891 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb99101e740 1 -- 172.21.15.28:0/246658638 >> 172.21.15.28:0/246658638 conn(0x55891245c8b0 msgr2=0x5589124540e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb99101e740 1 -- 172.21.15.28:0/246658638 shutdown_connections 2024-04-21T08:51:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb99101e740 1 -- 172.21.15.28:0/246658638 wait complete. 2024-04-21T08:51:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.159+0000 7fb99101e740 20 client.15891 trim_cache size 0 max 16384 2024-04-21T08:51:28.165 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:51:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "b51a76c8-ffba-11ee-bc93-c7b262605968", 2024-04-21T08:51:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-04-21T08:51:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24487, 2024-04-21T08:51:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-21T08:51:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-21T08:51:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-21T08:51:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 137.336238267, 2024-04-21T08:51:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 25, 2024-04-21T08:51:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 94, 2024-04-21T08:51:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 93, 2024-04-21T08:51:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 175.22438693000001 2024-04-21T08:51:28.166 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:51:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.160+0000 7fb98e350640 1 -- 172.21.15.28:0/2285543933 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb9700a3e70 msgr2=0x7fb9700a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.160+0000 7fb98e350640 1 --2- 172.21.15.28:0/2285543933 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb9700a3e70 0x7fb9700a6330 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb9800f00e0 tx=0x7fb9800b3c90 comp rx=0 tx=0).stop 2024-04-21T08:51:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.160+0000 7fb98e350640 1 -- 172.21.15.28:0/2285543933 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9880ff700 msgr2=0x7fb9881a6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.160+0000 7fb98e350640 1 --2- 172.21.15.28:0/2285543933 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9880ff700 0x7fb9881a6830 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb98002c700 tx=0x7fb980034760 comp rx=0 tx=0).stop 2024-04-21T08:51:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.160+0000 7fb98e350640 1 -- 172.21.15.28:0/2285543933 shutdown_connections 2024-04-21T08:51:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.160+0000 7fb98e350640 1 --2- 172.21.15.28:0/2285543933 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb9700a3e70 0x7fb9700a6330 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.160+0000 7fb98e350640 1 --2- 172.21.15.28:0/2285543933 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb9880ff700 0x7fb9881a6830 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.160+0000 7fb98e350640 1 --2- 172.21.15.28:0/2285543933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb988106160 0x7fb9880ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.160+0000 7fb98e350640 1 --2- 172.21.15.28:0/2285543933 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb988101090 0x7fb9880fec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.160+0000 7fb98e350640 1 -- 172.21.15.28:0/2285543933 >> 172.21.15.28:0/2285543933 conn(0x7fb988077f50 msgr2=0x7fb988069400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.160+0000 7fb98e350640 1 -- 172.21.15.28:0/2285543933 shutdown_connections 2024-04-21T08:51:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.160+0000 7fb98e350640 1 -- 172.21.15.28:0/2285543933 wait complete. 2024-04-21T08:51:28.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:51:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.274+0000 7fbef4138640 1 -- 172.21.15.28:0/1841155539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec106160 msgr2=0x7fbeec10dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.274+0000 7fbef4138640 1 --2- 172.21.15.28:0/1841155539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec106160 0x7fbeec10dd30 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009ca0 tx=0x7fbee402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.274+0000 7fbef4138640 1 -- 172.21.15.28:0/1841155539 shutdown_connections 2024-04-21T08:51:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.274+0000 7fbef4138640 1 --2- 172.21.15.28:0/1841155539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec106160 0x7fbeec10dd30 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.274+0000 7fbef4138640 1 --2- 172.21.15.28:0/1841155539 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbeec101a40 0x7fbeec105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.274+0000 7fbef4138640 1 --2- 172.21.15.28:0/1841155539 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbeec101090 0x7fbeec101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.274+0000 7fbef4138640 1 -- 172.21.15.28:0/1841155539 >> 172.21.15.28:0/1841155539 conn(0x7fbeec077f50 msgr2=0x7fbeec078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.275+0000 7fbef4138640 1 -- 172.21.15.28:0/1841155539 shutdown_connections 2024-04-21T08:51:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.275+0000 7fbef4138640 1 -- 172.21.15.28:0/1841155539 wait complete. 2024-04-21T08:51:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.275+0000 7fbef4138640 1 Processor -- start 2024-04-21T08:51:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.275+0000 7fbef4138640 1 -- start start 2024-04-21T08:51:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef4138640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbeec101090 0x7fbeec197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef4138640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec106160 0x7fbeec197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef4138640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbeec19bed0 0x7fbeec19c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef4138640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec1110d0 con 0x7fbeec106160 2024-04-21T08:51:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef4138640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec110f50 con 0x7fbeec19bed0 2024-04-21T08:51:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef4138640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec111250 con 0x7fbeec101090 2024-04-21T08:51:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef16d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec106160 0x7fbeec197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef16d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec106160 0x7fbeec197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44730/0 (socket says 172.21.15.28:44730) 2024-04-21T08:51:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef16d3640 1 -- 172.21.15.28:0/1553990319 learned_addr learned my addr 172.21.15.28:0/1553990319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef26d5640 1 --2- 172.21.15.28:0/1553990319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbeec19bed0 0x7fbeec19c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef1ed4640 1 --2- 172.21.15.28:0/1553990319 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbeec101090 0x7fbeec197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef16d3640 1 -- 172.21.15.28:0/1553990319 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbeec101090 msgr2=0x7fbeec197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef16d3640 1 --2- 172.21.15.28:0/1553990319 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbeec101090 0x7fbeec197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef16d3640 1 -- 172.21.15.28:0/1553990319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbeec19bed0 msgr2=0x7fbeec19c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef16d3640 1 --2- 172.21.15.28:0/1553990319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbeec19bed0 0x7fbeec19c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef16d3640 1 -- 172.21.15.28:0/1553990319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeec19c9c0 con 0x7fbeec106160 2024-04-21T08:51:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef16d3640 1 --2- 172.21.15.28:0/1553990319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec106160 0x7fbeec197b50 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00c970 tx=0x7fbedc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbee2ffd640 1 -- 172.21.15.28:0/1553990319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc007bb0 con 0x7fbeec106160 2024-04-21T08:51:28.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef4138640 1 -- 172.21.15.28:0/1553990319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeec19ccb0 con 0x7fbeec106160 2024-04-21T08:51:28.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbee2ffd640 1 -- 172.21.15.28:0/1553990319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbedc007d50 con 0x7fbeec106160 2024-04-21T08:51:28.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbee2ffd640 1 -- 172.21.15.28:0/1553990319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc018730 con 0x7fbeec106160 2024-04-21T08:51:28.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.276+0000 7fbef4138640 1 -- 172.21.15.28:0/1553990319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeec1a4450 con 0x7fbeec106160 2024-04-21T08:51:28.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.279+0000 7fbee2ffd640 1 -- 172.21.15.28:0/1553990319 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fbedc00f990 con 0x7fbeec106160 2024-04-21T08:51:28.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.279+0000 7fbef6dfd740 1 Processor -- start 2024-04-21T08:51:28.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.279+0000 7fbef6dfd740 1 -- start start 2024-04-21T08:51:28.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.279+0000 7fbef6dfd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638207947b0 0x563820794b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.279+0000 7fbef6dfd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563820795160 0x56382079d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.279+0000 7fbef6dfd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56382079dce0 0x5638207a0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.279+0000 7fbef6dfd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5638206f8b10 con 0x563820795160 2024-04-21T08:51:28.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.279+0000 7fbef6dfd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563820696240 con 0x5638207947b0 2024-04-21T08:51:28.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.279+0000 7fbef6dfd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5638206f4ad0 con 0x56382079dce0 2024-04-21T08:51:28.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbef1ed4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563820795160 0x56382079d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbef1ed4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563820795160 0x56382079d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44738/0 (socket says 172.21.15.28:44738) 2024-04-21T08:51:28.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbef1ed4640 1 -- 172.21.15.28:0/3044937727 learned_addr learned my addr 172.21.15.28:0/3044937727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:28.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbef26d5640 1 --2- 172.21.15.28:0/3044937727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638207947b0 0x563820794b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbee2ffd640 1 --2- 172.21.15.28:0/1553990319 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbed80a3e40 0x7fbed80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbef16d3640 1 --2- 172.21.15.28:0/3044937727 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56382079dce0 0x5638207a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbef1ed4640 1 -- 172.21.15.28:0/3044937727 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56382079dce0 msgr2=0x5638207a0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbef1ed4640 1 --2- 172.21.15.28:0/3044937727 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56382079dce0 0x5638207a0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbef1ed4640 1 -- 172.21.15.28:0/3044937727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638207947b0 msgr2=0x563820794b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbef26d5640 1 --2- 172.21.15.28:0/1553990319 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbed80a3e40 0x7fbed80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbef1ed4640 1 --2- 172.21.15.28:0/3044937727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638207947b0 0x563820794b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbef1ed4640 1 -- 172.21.15.28:0/3044937727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56382040b2d0 con 0x563820795160 2024-04-21T08:51:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbee2ffd640 1 -- 172.21.15.28:0/1553990319 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbedc0d0e40 con 0x7fbeec106160 2024-04-21T08:51:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbef1ed4640 1 --2- 172.21.15.28:0/3044937727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563820795160 0x56382079d710 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbee800bcb0 tx=0x7fbee800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a76692e1e198cd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbed7fff640 1 -- 172.21.15.28:0/3044937727 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8015070 con 0x563820795160 2024-04-21T08:51:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbef26d5640 1 --2- 172.21.15.28:0/1553990319 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbed80a3e40 0x7fbed80a6300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbee4004af0 tx=0x7fbee402c0f0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbed7fff640 1 -- 172.21.15.28:0/3044937727 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbee801a910 con 0x563820795160 2024-04-21T08:51:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.280+0000 7fbed7fff640 1 -- 172.21.15.28:0/3044937727 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8011980 con 0x563820795160 2024-04-21T08:51:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.281+0000 7fbef6dfd740 1 -- 172.21.15.28:0/3044937727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563820795160 msgr2=0x56382079d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.281+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/3044937727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563820795160 0x56382079d710 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbee800bcb0 tx=0x7fbee800f730 comp rx=0 tx=0).stop 2024-04-21T08:51:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.281+0000 7fbef6dfd740 1 -- 172.21.15.28:0/3044937727 shutdown_connections 2024-04-21T08:51:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.281+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/3044937727 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56382079dce0 0x5638207a0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.281+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/3044937727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563820795160 0x56382079d710 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.281+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/3044937727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638207947b0 0x563820794b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.281+0000 7fbef6dfd740 1 -- 172.21.15.28:0/3044937727 >> 172.21.15.28:0/3044937727 conn(0x56382078ea60 msgr2=0x56382078ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:28.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.281+0000 7fbef6dfd740 1 -- 172.21.15.28:0/3044937727 shutdown_connections 2024-04-21T08:51:28.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.281+0000 7fbef6dfd740 1 -- 172.21.15.28:0/3044937727 wait complete. 2024-04-21T08:51:28.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.282+0000 7fbef6dfd740 1 -- start start 2024-04-21T08:51:28.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.282+0000 7fbef6dfd740 1 Processor -- start 2024-04-21T08:51:28.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.282+0000 7fbef6dfd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638207947b0 0x56382079acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.282+0000 7fbef6dfd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563820795160 0x56382079b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.282+0000 7fbef6dfd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56382079dce0 0x56382079b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.282+0000 7fbef6dfd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5638207a2d70 con 0x5638207947b0 2024-04-21T08:51:28.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.282+0000 7fbef6dfd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563820681190 con 0x56382079dce0 2024-04-21T08:51:28.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.282+0000 7fbef6dfd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5638207a2ef0 con 0x563820795160 2024-04-21T08:51:28.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.282+0000 7fbef6dfd740 10 client.0 _ll_register_callbacks cb 0x56382067c8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:28.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.282+0000 7fbef1ed4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638207947b0 0x56382079acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.282+0000 7fbef1ed4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638207947b0 0x56382079acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44740/0 (socket says 172.21.15.28:44740) 2024-04-21T08:51:28.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.282+0000 7fbef1ed4640 1 -- 172.21.15.28:0/1589170527 learned_addr learned my addr 172.21.15.28:0/1589170527 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:28.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.282+0000 7fbef16d3640 1 --2- 172.21.15.28:0/1589170527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563820795160 0x56382079b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbef26d5640 1 --2- 172.21.15.28:0/1589170527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56382079dce0 0x56382079b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbef1ed4640 1 -- 172.21.15.28:0/1589170527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563820795160 msgr2=0x56382079b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbef1ed4640 1 --2- 172.21.15.28:0/1589170527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563820795160 0x56382079b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbef1ed4640 1 -- 172.21.15.28:0/1589170527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56382079dce0 msgr2=0x56382079b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbef1ed4640 1 --2- 172.21.15.28:0/1589170527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56382079dce0 0x56382079b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbef1ed4640 1 -- 172.21.15.28:0/1589170527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638204d56b0 con 0x5638207947b0 2024-04-21T08:51:28.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbef16d3640 1 --2- 172.21.15.28:0/1589170527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563820795160 0x56382079b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:28.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbef26d5640 1 --2- 172.21.15.28:0/1589170527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56382079dce0 0x56382079b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:28.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbef1ed4640 1 --2- 172.21.15.28:0/1589170527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638207947b0 0x56382079acf0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fbee8011520 tx=0x7fbee800bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed5ab5c00746058 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:28.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbed57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:28.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbed57fa640 1 -- 172.21.15.28:0/1589170527 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8028020 con 0x5638207947b0 2024-04-21T08:51:28.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbed57fa640 1 -- 172.21.15.28:0/1589170527 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbee801aac0 con 0x5638207947b0 2024-04-21T08:51:28.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1589170527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbee801a910 con 0x5638207947b0 2024-04-21T08:51:28.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbed57fa640 1 -- 172.21.15.28:0/1589170527 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbee8030050 con 0x5638207947b0 2024-04-21T08:51:28.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbef6dfd740 10 client.15903 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:28.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1589170527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5638207af140 con 0x5638207947b0 2024-04-21T08:51:28.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbed57fa640 1 -- 172.21.15.28:0/1589170527 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbee801a910 con 0x5638207947b0 2024-04-21T08:51:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.283+0000 7fbed57fa640 1 -- 172.21.15.28:0/1589170527 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fbee8023070 con 0x5638207947b0 2024-04-21T08:51:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.284+0000 7fbef6dfd740 10 client.15903 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.284+0000 7fbef6dfd740 10 client.15903 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:51:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.284+0000 7fbef6dfd740 20 client.15903 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.284+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1589170527 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5638207b1320 0x5638207b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.284+0000 7fbef6dfd740 4 client.15903 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.284+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1589170527 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563820706530 con 0x5638207b1320 2024-04-21T08:51:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.284+0000 7fbef16d3640 1 --2- 172.21.15.28:0/1589170527 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5638207b1320 0x5638207b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.284+0000 7fbed57fa640 10 client.15903 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.284+0000 7fbef16d3640 1 --2- 172.21.15.28:0/1589170527 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5638207b1320 0x5638207b2790 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8aa8b0e6e74bb18c server_cookie=94be5e85e96848da in_seq=0 out_seq=0 2024-04-21T08:51:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.285+0000 7fbed57fa640 1 -- 172.21.15.28:0/1589170527 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563820706530 con 0x5638207b1320 2024-04-21T08:51:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.285+0000 7fbed57fa640 10 client.15903 handle_command_reply: tid=0 2024-04-21T08:51:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.285+0000 7fbef6dfd740 1 client.15903 shutdown 2024-04-21T08:51:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.285+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1589170527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638207947b0 msgr2=0x56382079acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.285+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1589170527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638207947b0 0x56382079acf0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fbee8011520 tx=0x7fbee800bfd0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.285+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1589170527 shutdown_connections 2024-04-21T08:51:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.285+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1589170527 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5638207b1320 0x5638207b2790 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.285+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1589170527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56382079dce0 0x56382079b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.285+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1589170527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563820795160 0x56382079b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.285+0000 7fbed57fa640 0 client.15903 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.285+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1589170527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638207947b0 0x56382079acf0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.285+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1589170527 >> 172.21.15.28:0/1589170527 conn(0x56382078ea60 msgr2=0x563820790bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.285+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1589170527 shutdown_connections 2024-04-21T08:51:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.285+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1589170527 wait complete. 2024-04-21T08:51:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.285+0000 7fbef6dfd740 20 client.15903 trim_cache size 0 max 16384 2024-04-21T08:51:28.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef6dfd740 1 Processor -- start 2024-04-21T08:51:28.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef6dfd740 1 -- start start 2024-04-21T08:51:28.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef6dfd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638207947b0 0x56382079d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef6dfd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563820795160 0x56382078d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef6dfd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56382079dce0 0x563820789800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef6dfd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5638207b0ad0 con 0x5638207947b0 2024-04-21T08:51:28.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef6dfd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5638207b11a0 con 0x563820795160 2024-04-21T08:51:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef6dfd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5638207b1870 con 0x56382079dce0 2024-04-21T08:51:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef1ed4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638207947b0 0x56382079d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef1ed4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638207947b0 0x56382079d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44748/0 (socket says 172.21.15.28:44748) 2024-04-21T08:51:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef1ed4640 1 -- 172.21.15.28:0/1739182130 learned_addr learned my addr 172.21.15.28:0/1739182130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef16d3640 1 --2- 172.21.15.28:0/1739182130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563820795160 0x56382078d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef26d5640 1 --2- 172.21.15.28:0/1739182130 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56382079dce0 0x563820789800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef1ed4640 1 -- 172.21.15.28:0/1739182130 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56382079dce0 msgr2=0x563820789800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef1ed4640 1 --2- 172.21.15.28:0/1739182130 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56382079dce0 0x563820789800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef1ed4640 1 -- 172.21.15.28:0/1739182130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563820795160 msgr2=0x56382078d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef1ed4640 1 --2- 172.21.15.28:0/1739182130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563820795160 0x56382078d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef1ed4640 1 -- 172.21.15.28:0/1739182130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638207b1d20 con 0x5638207947b0 2024-04-21T08:51:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.304+0000 7fbef1ed4640 1 --2- 172.21.15.28:0/1739182130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638207947b0 0x56382079d420 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbee800fc30 tx=0x7fbee800bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb2d8e030b1fb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.305+0000 7fbed77fe640 1 -- 172.21.15.28:0/1739182130 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8028040 con 0x5638207947b0 2024-04-21T08:51:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.305+0000 7fbed77fe640 1 -- 172.21.15.28:0/1739182130 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbee80248b0 con 0x5638207947b0 2024-04-21T08:51:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.305+0000 7fbed77fe640 1 -- 172.21.15.28:0/1739182130 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8015050 con 0x5638207947b0 2024-04-21T08:51:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.305+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1739182130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638207947b0 msgr2=0x56382079d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.305+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1739182130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638207947b0 0x56382079d420 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbee800fc30 tx=0x7fbee800bb00 comp rx=0 tx=0).stop 2024-04-21T08:51:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.305+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1739182130 shutdown_connections 2024-04-21T08:51:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.305+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1739182130 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56382079dce0 0x563820789800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.305+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1739182130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563820795160 0x56382078d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.305+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1739182130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5638207947b0 0x56382079d420 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.305+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1739182130 >> 172.21.15.28:0/1739182130 conn(0x56382078ea60 msgr2=0x56382078ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.305+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1739182130 shutdown_connections 2024-04-21T08:51:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.305+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1739182130 wait complete. 2024-04-21T08:51:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef6dfd740 1 -- start start 2024-04-21T08:51:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef6dfd740 1 Processor -- start 2024-04-21T08:51:28.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef6dfd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638207947b0 0x5638207baab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef6dfd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563820795160 0x5638207b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef6dfd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56382079dce0 0x5638207b4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef6dfd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5638207b2610 con 0x563820795160 2024-04-21T08:51:28.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef6dfd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5638207b1f40 con 0x5638207947b0 2024-04-21T08:51:28.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef6dfd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5638207b2ce0 con 0x56382079dce0 2024-04-21T08:51:28.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef6dfd740 10 client.0 _ll_register_callbacks cb 0x7fbee4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:28.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef16d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563820795160 0x5638207b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef16d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563820795160 0x5638207b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44760/0 (socket says 172.21.15.28:44760) 2024-04-21T08:51:28.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef16d3640 1 -- 172.21.15.28:0/1871800382 learned_addr learned my addr 172.21.15.28:0/1871800382 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:28.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef1ed4640 1 --2- 172.21.15.28:0/1871800382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638207947b0 0x5638207baab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef16d3640 1 -- 172.21.15.28:0/1871800382 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56382079dce0 msgr2=0x5638207b4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef16d3640 1 --2- 172.21.15.28:0/1871800382 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56382079dce0 0x5638207b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef16d3640 1 -- 172.21.15.28:0/1871800382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638207947b0 msgr2=0x5638207baab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:28.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef16d3640 1 --2- 172.21.15.28:0/1871800382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638207947b0 0x5638207baab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.306+0000 7fbef16d3640 1 -- 172.21.15.28:0/1871800382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638207b49b0 con 0x563820795160 2024-04-21T08:51:28.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbef16d3640 1 --2- 172.21.15.28:0/1871800382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563820795160 0x5638207b3f30 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0ce180 tx=0x7fbedc00fd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d669a12988043eeb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:28.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbed57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:28.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbed57fa640 1 -- 172.21.15.28:0/1871800382 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0b9020 con 0x563820795160 2024-04-21T08:51:28.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbed57fa640 1 -- 172.21.15.28:0/1871800382 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbedc098e60 con 0x563820795160 2024-04-21T08:51:28.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1871800382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbee80248b0 con 0x563820795160 2024-04-21T08:51:28.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbed57fa640 1 -- 172.21.15.28:0/1871800382 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbedc09c7f0 con 0x563820795160 2024-04-21T08:51:28.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbef6dfd740 10 client.15909 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:28.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1871800382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5638207b9290 con 0x563820795160 2024-04-21T08:51:28.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbed57fa640 1 -- 172.21.15.28:0/1871800382 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbee80248b0 con 0x563820795160 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbed57fa640 1 -- 172.21.15.28:0/1871800382 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fbedc0d2040 con 0x563820795160 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbef6dfd740 10 client.15909 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbef6dfd740 10 client.15909 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbef6dfd740 20 client.15909 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1871800382 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56382081bb80 0x56382081a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbef6dfd740 4 client.15909 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.307+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1871800382 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "status"}) -- 0x5638207003a0 con 0x56382081bb80 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.308+0000 7fbef1ed4640 1 --2- 172.21.15.28:0/1871800382 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56382081bb80 0x56382081a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.308+0000 7fbed57fa640 10 client.15909 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.308+0000 7fbef1ed4640 1 --2- 172.21.15.28:0/1871800382 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56382081bb80 0x56382081a890 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccadad880a05adb server_cookie=91913c77e5ef438c in_seq=0 out_seq=0 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.308+0000 7fbed57fa640 1 -- 172.21.15.28:0/1871800382 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5638207003a0 con 0x56382081bb80 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.308+0000 7fbed57fa640 10 client.15909 handle_command_reply: tid=0 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.308+0000 7fbef6dfd740 1 client.15909 shutdown 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.308+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1871800382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563820795160 msgr2=0x5638207b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.308+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1871800382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563820795160 0x5638207b3f30 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0ce180 tx=0x7fbedc00fd30 comp rx=0 tx=0).stop 2024-04-21T08:51:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1871800382 shutdown_connections 2024-04-21T08:51:28.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1871800382 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56382081bb80 0x56382081a890 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1871800382 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56382079dce0 0x5638207b4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1871800382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563820795160 0x5638207b3f30 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef6dfd740 1 --2- 172.21.15.28:0/1871800382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5638207947b0 0x5638207baab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbed57fa640 0 client.15909 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:28.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1871800382 >> 172.21.15.28:0/1871800382 conn(0x56382078ea60 msgr2=0x563820799d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:28.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1871800382 shutdown_connections 2024-04-21T08:51:28.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef6dfd740 1 -- 172.21.15.28:0/1871800382 wait complete. 2024-04-21T08:51:28.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef6dfd740 20 client.15909 trim_cache size 0 max 16384 2024-04-21T08:51:28.314 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:51:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "b51a76c8-ffba-11ee-bc93-c7b262605968", 2024-04-21T08:51:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-21T08:51:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24488, 2024-04-21T08:51:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-21T08:51:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-21T08:51:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-21T08:51:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 161.57638793999999, 2024-04-21T08:51:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 25, 2024-04-21T08:51:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 94, 2024-04-21T08:51:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 93, 2024-04-21T08:51:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 166.74859337000001 2024-04-21T08:51:28.315 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:51:28.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef4138640 1 -- 172.21.15.28:0/1553990319 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbed80a3e40 msgr2=0x7fbed80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef4138640 1 --2- 172.21.15.28:0/1553990319 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbed80a3e40 0x7fbed80a6300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbee4004af0 tx=0x7fbee402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef4138640 1 -- 172.21.15.28:0/1553990319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec106160 msgr2=0x7fbeec197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:28.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef4138640 1 --2- 172.21.15.28:0/1553990319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec106160 0x7fbeec197b50 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00c970 tx=0x7fbedc00ce30 comp rx=0 tx=0).stop 2024-04-21T08:51:28.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef4138640 1 -- 172.21.15.28:0/1553990319 shutdown_connections 2024-04-21T08:51:28.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef4138640 1 --2- 172.21.15.28:0/1553990319 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbed80a3e40 0x7fbed80a6300 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef4138640 1 --2- 172.21.15.28:0/1553990319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbeec19bed0 0x7fbeec19c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef4138640 1 --2- 172.21.15.28:0/1553990319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec106160 0x7fbeec197b50 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef4138640 1 --2- 172.21.15.28:0/1553990319 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbeec101090 0x7fbeec197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:28.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef4138640 1 -- 172.21.15.28:0/1553990319 >> 172.21.15.28:0/1553990319 conn(0x7fbeec077f50 msgr2=0x7fbeec0fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:28.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef4138640 1 -- 172.21.15.28:0/1553990319 shutdown_connections 2024-04-21T08:51:28.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:28.309+0000 7fbef4138640 1 -- 172.21.15.28:0/1553990319 wait complete. 2024-04-21T08:51:28.320 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 120 sec 2024-04-21T08:51:28.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:27 smithi152 ceph-mon[31401]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-21T08:51:28.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:27 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1162094011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:28.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:27 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2628167307' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:28.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:27 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2799053202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:28.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:27 smithi171 ceph-mon[31358]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-21T08:51:28.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:27 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1162094011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:28.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:27 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2628167307' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:51:28.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:27 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2799053202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:30.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:29 smithi152 ceph-mon[31401]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-04-21T08:51:30.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:29 smithi028 ceph-mon[28003]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-04-21T08:51:30.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:29 smithi171 ceph-mon[31358]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-04-21T08:51:32.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:31 smithi152 ceph-mon[31401]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-04-21T08:51:32.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:31 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:51:32.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:31 smithi028 ceph-mon[28003]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-04-21T08:51:32.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:31 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:51:32.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:31 smithi171 ceph-mon[31358]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-04-21T08:51:32.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:31 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:51:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.740+0000 7f43be848640 1 -- 172.21.15.28:0/3956602691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43b81072b0 msgr2=0x7f43b810ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.740+0000 7f43be848640 1 --2- 172.21.15.28:0/3956602691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43b81072b0 0x7f43b810ee80 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f43b0009ca0 tx=0x7f43b002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.740+0000 7f43be848640 1 -- 172.21.15.28:0/3956602691 shutdown_connections 2024-04-21T08:51:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.740+0000 7f43be848640 1 --2- 172.21.15.28:0/3956602691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43b81072b0 0x7f43b810ee80 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.740+0000 7f43be848640 1 --2- 172.21.15.28:0/3956602691 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43b8102b90 0x7f43b8106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.740+0000 7f43be848640 1 --2- 172.21.15.28:0/3956602691 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43b81021e0 0x7f43b81025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.740+0000 7f43be848640 1 -- 172.21.15.28:0/3956602691 >> 172.21.15.28:0/3956602691 conn(0x7f43b80fda00 msgr2=0x7f43b80ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.740+0000 7f43be848640 1 -- 172.21.15.28:0/3956602691 shutdown_connections 2024-04-21T08:51:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.740+0000 7f43be848640 1 -- 172.21.15.28:0/3956602691 wait complete. 2024-04-21T08:51:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.741+0000 7f43be848640 1 Processor -- start 2024-04-21T08:51:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.741+0000 7f43be848640 1 -- start start 2024-04-21T08:51:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.741+0000 7f43be848640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43b81021e0 0x7f43b8074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.741+0000 7f43be848640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43b81072b0 0x7f43b8074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.741+0000 7f43be848640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43b8074cd0 0x7f43b81a6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.741+0000 7f43be848640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43b8112220 con 0x7f43b81021e0 2024-04-21T08:51:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.741+0000 7f43be848640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f43b81120a0 con 0x7f43b81072b0 2024-04-21T08:51:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.741+0000 7f43be848640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f43b81123a0 con 0x7f43b8074cd0 2024-04-21T08:51:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.741+0000 7f43b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43b81021e0 0x7f43b8074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.741+0000 7f43b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43b81021e0 0x7f43b8074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59848/0 (socket says 172.21.15.28:59848) 2024-04-21T08:51:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.741+0000 7f43b7fff640 1 -- 172.21.15.28:0/1855371183 learned_addr learned my addr 172.21.15.28:0/1855371183 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.741+0000 7f43b77fe640 1 --2- 172.21.15.28:0/1855371183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43b81072b0 0x7f43b8074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.742+0000 7f43bcde5640 1 --2- 172.21.15.28:0/1855371183 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43b8074cd0 0x7f43b81a6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.742+0000 7f43b7fff640 1 -- 172.21.15.28:0/1855371183 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43b8074cd0 msgr2=0x7f43b81a6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.742+0000 7f43b7fff640 1 --2- 172.21.15.28:0/1855371183 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43b8074cd0 0x7f43b81a6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.742+0000 7f43b7fff640 1 -- 172.21.15.28:0/1855371183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43b81072b0 msgr2=0x7f43b8074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.742+0000 7f43b7fff640 1 --2- 172.21.15.28:0/1855371183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43b81072b0 0x7f43b8074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.742+0000 7f43b7fff640 1 -- 172.21.15.28:0/1855371183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b81a7030 con 0x7f43b81021e0 2024-04-21T08:51:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.742+0000 7f43bcde5640 1 --2- 172.21.15.28:0/1855371183 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43b8074cd0 0x7f43b81a6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.742+0000 7f43b7fff640 1 --2- 172.21.15.28:0/1855371183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43b81021e0 0x7f43b8074250 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f43a800e9f0 tx=0x7f43a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.742+0000 7f43b57fa640 1 -- 172.21.15.28:0/1855371183 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a800cd80 con 0x7f43b81021e0 2024-04-21T08:51:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.742+0000 7f43be848640 1 -- 172.21.15.28:0/1855371183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43b81a7320 con 0x7f43b81021e0 2024-04-21T08:51:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.742+0000 7f43b57fa640 1 -- 172.21.15.28:0/1855371183 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43a80108e0 con 0x7f43b81021e0 2024-04-21T08:51:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.742+0000 7f43b57fa640 1 -- 172.21.15.28:0/1855371183 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a8018980 con 0x7f43b81021e0 2024-04-21T08:51:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.742+0000 7f43be848640 1 -- 172.21.15.28:0/1855371183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43b81a7870 con 0x7f43b81021e0 2024-04-21T08:51:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.745+0000 7f43b57fa640 1 -- 172.21.15.28:0/1855371183 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f43a8010430 con 0x7f43b81021e0 2024-04-21T08:51:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.745+0000 7f43b57fa640 1 --2- 172.21.15.28:0/1855371183 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f43a00a3f10 0x7f43a00a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.745+0000 7f43c1432740 1 Processor -- start 2024-04-21T08:51:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.745+0000 7f43c1432740 1 -- start start 2024-04-21T08:51:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43b57fa640 1 -- 172.21.15.28:0/1855371183 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f43a80d11f0 con 0x7f43b81021e0 2024-04-21T08:51:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43b77fe640 1 --2- 172.21.15.28:0/1855371183 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f43a00a3f10 0x7f43a00a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43c1432740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8b8d0 0x55e585d8a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43c1432740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8ab60 0x55e585d8afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43c1432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d940e0 0x55e585d964d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43c1432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e585cef8d0 con 0x55e585d940e0 2024-04-21T08:51:33.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43c1432740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e585c8ce80 con 0x55e585d8ab60 2024-04-21T08:51:33.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43c1432740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55e585ceb890 con 0x55e585d8b8d0 2024-04-21T08:51:33.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43bcde5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d940e0 0x55e585d964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43bcde5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d940e0 0x55e585d964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59852/0 (socket says 172.21.15.28:59852) 2024-04-21T08:51:33.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43bcde5640 1 -- 172.21.15.28:0/2307357921 learned_addr learned my addr 172.21.15.28:0/2307357921 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43b7fff640 1 --2- 172.21.15.28:0/2307357921 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8b8d0 0x55e585d8a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43b77fe640 1 --2- 172.21.15.28:0/2307357921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8ab60 0x55e585d8afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43b77fe640 1 --2- 172.21.15.28:0/1855371183 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f43a00a3f10 0x7f43a00a63d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f43b8075ce0 tx=0x7f43ac009f70 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43bcde5640 1 -- 172.21.15.28:0/2307357921 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8b8d0 msgr2=0x55e585d8a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43bcde5640 1 --2- 172.21.15.28:0/2307357921 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8b8d0 0x55e585d8a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43bcde5640 1 -- 172.21.15.28:0/2307357921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8ab60 msgr2=0x55e585d8afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43bcde5640 1 --2- 172.21.15.28:0/2307357921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8ab60 0x55e585d8afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43bcde5640 1 -- 172.21.15.28:0/2307357921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e585cae320 con 0x55e585d940e0 2024-04-21T08:51:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43b77fe640 1 --2- 172.21.15.28:0/2307357921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8ab60 0x55e585d8afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.746+0000 7f43bcde5640 1 --2- 172.21.15.28:0/2307357921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d940e0 0x55e585d964d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f43b0009ca0 tx=0x7f43b0005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5101a7cf4395c3b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.747+0000 7f439e7fc640 1 -- 172.21.15.28:0/2307357921 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b0043020 con 0x55e585d940e0 2024-04-21T08:51:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.747+0000 7f439e7fc640 1 -- 172.21.15.28:0/2307357921 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43b0032970 con 0x55e585d940e0 2024-04-21T08:51:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.747+0000 7f439e7fc640 1 -- 172.21.15.28:0/2307357921 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b0039040 con 0x55e585d940e0 2024-04-21T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.747+0000 7f43c1432740 1 -- 172.21.15.28:0/2307357921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d940e0 msgr2=0x55e585d964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.747+0000 7f43c1432740 1 --2- 172.21.15.28:0/2307357921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d940e0 0x55e585d964d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f43b0009ca0 tx=0x7f43b0005fa0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.747+0000 7f43c1432740 1 -- 172.21.15.28:0/2307357921 shutdown_connections 2024-04-21T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.747+0000 7f43c1432740 1 --2- 172.21.15.28:0/2307357921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d940e0 0x55e585d964d0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.747+0000 7f43c1432740 1 --2- 172.21.15.28:0/2307357921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8ab60 0x55e585d8afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.747+0000 7f43c1432740 1 --2- 172.21.15.28:0/2307357921 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8b8d0 0x55e585d8a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.747+0000 7f43c1432740 1 -- 172.21.15.28:0/2307357921 >> 172.21.15.28:0/2307357921 conn(0x55e585d51c90 msgr2=0x55e585d85850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.747+0000 7f43c1432740 1 -- 172.21.15.28:0/2307357921 shutdown_connections 2024-04-21T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.747+0000 7f43c1432740 1 -- 172.21.15.28:0/2307357921 wait complete. 2024-04-21T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.748+0000 7f43c1432740 1 -- start start 2024-04-21T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.748+0000 7f43c1432740 1 Processor -- start 2024-04-21T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.748+0000 7f43c1432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d8ab60 0x55e585d93a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.748+0000 7f43c1432740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8b8d0 0x55e585d90e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.748+0000 7f43c1432740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d940e0 0x55e585d91380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.748+0000 7f43c1432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e585d96db0 con 0x55e585d8ab60 2024-04-21T08:51:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.748+0000 7f43c1432740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e585c77eb0 con 0x55e585d940e0 2024-04-21T08:51:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.748+0000 7f43c1432740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55e585d96f30 con 0x55e585d8b8d0 2024-04-21T08:51:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.748+0000 7f43c1432740 10 client.0 _ll_register_callbacks cb 0x55e585ca4b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.748+0000 7f43b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d8ab60 0x55e585d93a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.748+0000 7f43b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d8ab60 0x55e585d93a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59866/0 (socket says 172.21.15.28:59866) 2024-04-21T08:51:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.748+0000 7f43b7fff640 1 -- 172.21.15.28:0/3732310504 learned_addr learned my addr 172.21.15.28:0/3732310504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.748+0000 7f43b77fe640 1 --2- 172.21.15.28:0/3732310504 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8b8d0 0x55e585d90e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43bcde5640 1 --2- 172.21.15.28:0/3732310504 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d940e0 0x55e585d91380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43b7fff640 1 -- 172.21.15.28:0/3732310504 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8b8d0 msgr2=0x55e585d90e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43b7fff640 1 --2- 172.21.15.28:0/3732310504 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8b8d0 0x55e585d90e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43b7fff640 1 -- 172.21.15.28:0/3732310504 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d940e0 msgr2=0x55e585d91380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43b7fff640 1 --2- 172.21.15.28:0/3732310504 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d940e0 0x55e585d91380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43b7fff640 1 -- 172.21.15.28:0/3732310504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e585a02210 con 0x55e585d8ab60 2024-04-21T08:51:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43bcde5640 1 --2- 172.21.15.28:0/3732310504 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d940e0 0x55e585d91380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43b7fff640 1 --2- 172.21.15.28:0/3732310504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d8ab60 0x55e585d93a70 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f43a8000c00 tx=0x7f43a809d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c9342ca1dbf4d00 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43977fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43977fe640 1 -- 172.21.15.28:0/3732310504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a80a5810 con 0x55e585d8ab60 2024-04-21T08:51:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43977fe640 1 -- 172.21.15.28:0/3732310504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f43a80ce700 con 0x55e585d8ab60 2024-04-21T08:51:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43c1432740 1 -- 172.21.15.28:0/3732310504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43b0032970 con 0x55e585d8ab60 2024-04-21T08:51:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43977fe640 1 -- 172.21.15.28:0/3732310504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f43a80ce900 con 0x55e585d8ab60 2024-04-21T08:51:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43c1432740 10 client.15921 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43c1432740 1 -- 172.21.15.28:0/3732310504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e585ace620 con 0x55e585d8ab60 2024-04-21T08:51:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.749+0000 7f43977fe640 1 -- 172.21.15.28:0/3732310504 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43b0032970 con 0x55e585d8ab60 2024-04-21T08:51:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.750+0000 7f43977fe640 1 -- 172.21.15.28:0/3732310504 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f43a80988f0 con 0x55e585d8ab60 2024-04-21T08:51:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.750+0000 7f43c1432740 10 client.15921 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.750+0000 7f43c1432740 10 client.15921 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.750+0000 7f43c1432740 20 client.15921 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.750+0000 7f43c1432740 1 --2- 172.21.15.28:0/3732310504 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e585da4120 0x55e585da6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.750+0000 7f43c1432740 4 client.15921 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.750+0000 7f43c1432740 1 -- 172.21.15.28:0/3732310504 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e585d03880 con 0x55e585da4120 2024-04-21T08:51:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.750+0000 7f43b77fe640 1 --2- 172.21.15.28:0/3732310504 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e585da4120 0x55e585da6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.750+0000 7f43977fe640 10 client.15921 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.750+0000 7f43b77fe640 1 --2- 172.21.15.28:0/3732310504 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e585da4120 0x55e585da6510 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0be4dd7d8db7f75 server_cookie=33058e4217679cb6 in_seq=0 out_seq=0 2024-04-21T08:51:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.751+0000 7f43977fe640 1 -- 172.21.15.28:0/3732310504 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e585d03880 con 0x55e585da4120 2024-04-21T08:51:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.751+0000 7f43977fe640 10 client.15921 handle_command_reply: tid=0 2024-04-21T08:51:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.751+0000 7f43c1432740 1 client.15921 shutdown 2024-04-21T08:51:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.751+0000 7f43c1432740 1 -- 172.21.15.28:0/3732310504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d8ab60 msgr2=0x55e585d93a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.751+0000 7f43c1432740 1 --2- 172.21.15.28:0/3732310504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d8ab60 0x55e585d93a70 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f43a8000c00 tx=0x7f43a809d2a0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.751+0000 7f43c1432740 1 -- 172.21.15.28:0/3732310504 shutdown_connections 2024-04-21T08:51:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.751+0000 7f43c1432740 1 --2- 172.21.15.28:0/3732310504 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e585da4120 0x55e585da6510 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.751+0000 7f43c1432740 1 --2- 172.21.15.28:0/3732310504 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d940e0 0x55e585d91380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.751+0000 7f43c1432740 1 --2- 172.21.15.28:0/3732310504 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8b8d0 0x55e585d90e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.751+0000 7f43c1432740 1 --2- 172.21.15.28:0/3732310504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d8ab60 0x55e585d93a70 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.751+0000 7f43977fe640 0 client.15921 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.751+0000 7f43c1432740 1 -- 172.21.15.28:0/3732310504 >> 172.21.15.28:0/3732310504 conn(0x55e585d51c90 msgr2=0x55e585d875a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.751+0000 7f43c1432740 1 -- 172.21.15.28:0/3732310504 shutdown_connections 2024-04-21T08:51:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.752+0000 7f43c1432740 1 -- 172.21.15.28:0/3732310504 wait complete. 2024-04-21T08:51:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.752+0000 7f43c1432740 20 client.15921 trim_cache size 0 max 16384 2024-04-21T08:51:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.766+0000 7f43c1432740 1 Processor -- start 2024-04-21T08:51:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.766+0000 7f43c1432740 1 -- start start 2024-04-21T08:51:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43c1432740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8ab60 0x55e585d9fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43c1432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d8b8d0 0x55e585d93530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43c1432740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d940e0 0x55e585d93a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43c1432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43b81b3dd0 con 0x55e585d8b8d0 2024-04-21T08:51:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43c1432740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f43b81b16a0 con 0x55e585d8ab60 2024-04-21T08:51:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43c1432740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f43b81af4f0 con 0x55e585d940e0 2024-04-21T08:51:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d8b8d0 0x55e585d93530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d8b8d0 0x55e585d93530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59870/0 (socket says 172.21.15.28:59870) 2024-04-21T08:51:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43b77fe640 1 -- 172.21.15.28:0/4046730140 learned_addr learned my addr 172.21.15.28:0/4046730140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43b7fff640 1 --2- 172.21.15.28:0/4046730140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8ab60 0x55e585d9fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43bcde5640 1 --2- 172.21.15.28:0/4046730140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d940e0 0x55e585d93a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43b77fe640 1 -- 172.21.15.28:0/4046730140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d940e0 msgr2=0x55e585d93a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43b77fe640 1 --2- 172.21.15.28:0/4046730140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d940e0 0x55e585d93a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43b77fe640 1 -- 172.21.15.28:0/4046730140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8ab60 msgr2=0x55e585d9fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43b77fe640 1 --2- 172.21.15.28:0/4046730140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8ab60 0x55e585d9fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43b77fe640 1 -- 172.21.15.28:0/4046730140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e585d80f30 con 0x55e585d8b8d0 2024-04-21T08:51:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43bcde5640 1 --2- 172.21.15.28:0/4046730140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d940e0 0x55e585d93a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f43b77fe640 1 --2- 172.21.15.28:0/4046730140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d8b8d0 0x55e585d93530 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f43b806b5b0 tx=0x7f43ac01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead10892d7d3767e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f439dffb640 1 -- 172.21.15.28:0/4046730140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac017030 con 0x55e585d8b8d0 2024-04-21T08:51:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.767+0000 7f439dffb640 1 -- 172.21.15.28:0/4046730140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43ac019050 con 0x55e585d8b8d0 2024-04-21T08:51:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.768+0000 7f439dffb640 1 -- 172.21.15.28:0/4046730140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac004d30 con 0x55e585d8b8d0 2024-04-21T08:51:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.768+0000 7f43c1432740 1 -- 172.21.15.28:0/4046730140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d8b8d0 msgr2=0x55e585d93530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.768+0000 7f43c1432740 1 --2- 172.21.15.28:0/4046730140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d8b8d0 0x55e585d93530 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f43b806b5b0 tx=0x7f43ac01a000 comp rx=0 tx=0).stop 2024-04-21T08:51:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.768+0000 7f43c1432740 1 -- 172.21.15.28:0/4046730140 shutdown_connections 2024-04-21T08:51:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.768+0000 7f43c1432740 1 --2- 172.21.15.28:0/4046730140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d940e0 0x55e585d93a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.768+0000 7f43c1432740 1 --2- 172.21.15.28:0/4046730140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d8b8d0 0x55e585d93530 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.768+0000 7f43c1432740 1 --2- 172.21.15.28:0/4046730140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8ab60 0x55e585d9fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.768+0000 7f43c1432740 1 -- 172.21.15.28:0/4046730140 >> 172.21.15.28:0/4046730140 conn(0x55e585d51c90 msgr2=0x55e585d8c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.768+0000 7f43c1432740 1 -- 172.21.15.28:0/4046730140 shutdown_connections 2024-04-21T08:51:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.768+0000 7f43c1432740 1 -- 172.21.15.28:0/4046730140 wait complete. 2024-04-21T08:51:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.768+0000 7f43c1432740 1 -- start start 2024-04-21T08:51:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43c1432740 1 Processor -- start 2024-04-21T08:51:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43c1432740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8ab60 0x55e585d97ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43c1432740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8b8d0 0x55e585d98530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43c1432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d940e0 0x55e585e095d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43c1432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43b81afb60 con 0x55e585d940e0 2024-04-21T08:51:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43c1432740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f43b81b0fd0 con 0x55e585d8b8d0 2024-04-21T08:51:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43c1432740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f43b81b0230 con 0x55e585d8ab60 2024-04-21T08:51:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43c1432740 10 client.0 _ll_register_callbacks cb 0x7f43a809c6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43bcde5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d940e0 0x55e585e095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43bcde5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d940e0 0x55e585e095d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59878/0 (socket says 172.21.15.28:59878) 2024-04-21T08:51:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43bcde5640 1 -- 172.21.15.28:0/3855502639 learned_addr learned my addr 172.21.15.28:0/3855502639 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43b77fe640 1 --2- 172.21.15.28:0/3855502639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8b8d0 0x55e585d98530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43b7fff640 1 --2- 172.21.15.28:0/3855502639 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8ab60 0x55e585d97ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43bcde5640 1 -- 172.21.15.28:0/3855502639 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8ab60 msgr2=0x55e585d97ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43bcde5640 1 --2- 172.21.15.28:0/3855502639 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8ab60 0x55e585d97ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43bcde5640 1 -- 172.21.15.28:0/3855502639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8b8d0 msgr2=0x55e585d98530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43bcde5640 1 --2- 172.21.15.28:0/3855502639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8b8d0 0x55e585d98530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43bcde5640 1 -- 172.21.15.28:0/3855502639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e585e09b10 con 0x55e585d940e0 2024-04-21T08:51:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43b77fe640 1 --2- 172.21.15.28:0/3855502639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8b8d0 0x55e585d98530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.769+0000 7f43b7fff640 1 --2- 172.21.15.28:0/3855502639 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8ab60 0x55e585d97ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43bcde5640 1 --2- 172.21.15.28:0/3855502639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d940e0 0x55e585e095d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f43b0004af0 tx=0x7f43b0009980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4026167a8884774f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43977fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43977fe640 1 -- 172.21.15.28:0/3855502639 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b004c070 con 0x55e585d940e0 2024-04-21T08:51:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43977fe640 1 -- 172.21.15.28:0/3855502639 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f43b00328d0 con 0x55e585d940e0 2024-04-21T08:51:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43c1432740 1 -- 172.21.15.28:0/3855502639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43ac019050 con 0x55e585d940e0 2024-04-21T08:51:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43977fe640 1 -- 172.21.15.28:0/3855502639 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f43b0058050 con 0x55e585d940e0 2024-04-21T08:51:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43c1432740 10 client.15927 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43c1432740 1 -- 172.21.15.28:0/3855502639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e585e0ac80 con 0x55e585d940e0 2024-04-21T08:51:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43977fe640 1 -- 172.21.15.28:0/3855502639 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43ac019050 con 0x55e585d940e0 2024-04-21T08:51:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43977fe640 1 -- 172.21.15.28:0/3855502639 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f43b0004620 con 0x55e585d940e0 2024-04-21T08:51:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43c1432740 10 client.15927 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43c1432740 10 client.15927 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43c1432740 20 client.15927 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43c1432740 1 --2- 172.21.15.28:0/3855502639 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e585e0b060 0x55e585e0d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43c1432740 4 client.15927 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:51:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.770+0000 7f43c1432740 1 -- 172.21.15.28:0/3855502639 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e585c8f3d0 con 0x55e585e0b060 2024-04-21T08:51:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.771+0000 7f43b7fff640 1 --2- 172.21.15.28:0/3855502639 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e585e0b060 0x55e585e0d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.771+0000 7f43b7fff640 1 --2- 172.21.15.28:0/3855502639 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e585e0b060 0x55e585e0d450 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f0caf2a5750851 server_cookie=f3bcfc4da9c0630d in_seq=0 out_seq=0 2024-04-21T08:51:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.771+0000 7f43977fe640 10 client.15927 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.784+0000 7f43977fe640 1 -- 172.21.15.28:0/3855502639 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e585c8f3d0 con 0x55e585e0b060 2024-04-21T08:51:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.784+0000 7f43977fe640 10 client.15927 handle_command_reply: tid=0 2024-04-21T08:51:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.784+0000 7f43c1432740 1 client.15927 shutdown 2024-04-21T08:51:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.784+0000 7f43c1432740 1 -- 172.21.15.28:0/3855502639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d940e0 msgr2=0x55e585e095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.784+0000 7f43c1432740 1 --2- 172.21.15.28:0/3855502639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d940e0 0x55e585e095d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f43b0004af0 tx=0x7f43b0009980 comp rx=0 tx=0).stop 2024-04-21T08:51:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.784+0000 7f43c1432740 1 -- 172.21.15.28:0/3855502639 shutdown_connections 2024-04-21T08:51:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.784+0000 7f43c1432740 1 --2- 172.21.15.28:0/3855502639 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e585e0b060 0x55e585e0d450 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.784+0000 7f43c1432740 1 --2- 172.21.15.28:0/3855502639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e585d940e0 0x55e585e095d0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.784+0000 7f43977fe640 0 client.15927 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.784+0000 7f43c1432740 1 --2- 172.21.15.28:0/3855502639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e585d8b8d0 0x55e585d98530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.784+0000 7f43c1432740 1 --2- 172.21.15.28:0/3855502639 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e585d8ab60 0x55e585d97ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.784+0000 7f43c1432740 1 -- 172.21.15.28:0/3855502639 >> 172.21.15.28:0/3855502639 conn(0x55e585d51c90 msgr2=0x55e585d86ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.784+0000 7f43c1432740 1 -- 172.21.15.28:0/3855502639 shutdown_connections 2024-04-21T08:51:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.784+0000 7f43c1432740 1 -- 172.21.15.28:0/3855502639 wait complete. 2024-04-21T08:51:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.784+0000 7f43c1432740 20 client.15927 trim_cache size 0 max 16384 2024-04-21T08:51:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:51:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:51:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:51:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:51:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.785+0000 7f43be848640 1 -- 172.21.15.28:0/1855371183 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f43a00a3f10 msgr2=0x7f43a00a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.785+0000 7f43be848640 1 --2- 172.21.15.28:0/1855371183 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f43a00a3f10 0x7f43a00a63d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f43b8075ce0 tx=0x7f43ac009f70 comp rx=0 tx=0).stop 2024-04-21T08:51:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.785+0000 7f43be848640 1 -- 172.21.15.28:0/1855371183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43b81021e0 msgr2=0x7f43b8074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.785+0000 7f43be848640 1 --2- 172.21.15.28:0/1855371183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43b81021e0 0x7f43b8074250 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f43a800e9f0 tx=0x7f43a800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.785+0000 7f43be848640 1 -- 172.21.15.28:0/1855371183 shutdown_connections 2024-04-21T08:51:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.785+0000 7f43be848640 1 --2- 172.21.15.28:0/1855371183 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f43a00a3f10 0x7f43a00a63d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.785+0000 7f43be848640 1 --2- 172.21.15.28:0/1855371183 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43b8074cd0 0x7f43b81a6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.785+0000 7f43be848640 1 --2- 172.21.15.28:0/1855371183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43b81072b0 0x7f43b8074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.785+0000 7f43be848640 1 --2- 172.21.15.28:0/1855371183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43b81021e0 0x7f43b8074250 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.785+0000 7f43be848640 1 -- 172.21.15.28:0/1855371183 >> 172.21.15.28:0/1855371183 conn(0x7f43b80fda00 msgr2=0x7f43b80ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.785+0000 7f43be848640 1 -- 172.21.15.28:0/1855371183 shutdown_connections 2024-04-21T08:51:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:33.785+0000 7f43be848640 1 -- 172.21.15.28:0/1855371183 wait complete. 2024-04-21T08:51:34.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:33 smithi152 ceph-mon[31401]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-21T08:51:34.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:33 smithi028 ceph-mon[28003]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-21T08:51:34.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:33 smithi171 ceph-mon[31358]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-21T08:51:36.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:35 smithi152 ceph-mon[31401]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-21T08:51:36.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:35 smithi028 ceph-mon[28003]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-21T08:51:36.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:35 smithi171 ceph-mon[31358]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-21T08:51:38.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:37 smithi152 ceph-mon[31401]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-04-21T08:51:38.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:37 smithi028 ceph-mon[28003]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-04-21T08:51:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:37 smithi171 ceph-mon[31358]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-04-21T08:51:40.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:39 smithi152 ceph-mon[31401]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-21T08:51:40.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:39 smithi028 ceph-mon[28003]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-21T08:51:40.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:39 smithi171 ceph-mon[31358]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-21T08:51:42.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:41 smithi152 ceph-mon[31401]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-21T08:51:42.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:41 smithi028 ceph-mon[28003]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-21T08:51:42.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:41 smithi171 ceph-mon[31358]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-21T08:51:44.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:43 smithi152 ceph-mon[31401]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-04-21T08:51:44.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:43 smithi028 ceph-mon[28003]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-04-21T08:51:44.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:43 smithi171 ceph-mon[31358]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-04-21T08:51:46.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:45 smithi152 ceph-mon[31401]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-21T08:51:46.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:45 smithi028 ceph-mon[28003]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-21T08:51:46.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:45 smithi171 ceph-mon[31358]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-21T08:51:46.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.880+0000 7fc31ff9d640 1 -- 172.21.15.28:0/1109105988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3181031e0 msgr2=0x7fc3181035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:46.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.880+0000 7fc31ff9d640 1 --2- 172.21.15.28:0/1109105988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3181031e0 0x7fc3181035c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fc314009900 tx=0x7fc31402bd60 comp rx=0 tx=0).stop 2024-04-21T08:51:46.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.880+0000 7fc31ff9d640 1 -- 172.21.15.28:0/1109105988 shutdown_connections 2024-04-21T08:51:46.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.880+0000 7fc31ff9d640 1 --2- 172.21.15.28:0/1109105988 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc31810c520 0x7fc31810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.880+0000 7fc31ff9d640 1 --2- 172.21.15.28:0/1109105988 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc318103b00 0x7fc31810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.880+0000 7fc31ff9d640 1 --2- 172.21.15.28:0/1109105988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3181031e0 0x7fc3181035c0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.880+0000 7fc31ff9d640 1 -- 172.21.15.28:0/1109105988 >> 172.21.15.28:0/1109105988 conn(0x7fc318077fd0 msgr2=0x7fc3180783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:46.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.880+0000 7fc31ff9d640 1 -- 172.21.15.28:0/1109105988 shutdown_connections 2024-04-21T08:51:46.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.880+0000 7fc31ff9d640 1 -- 172.21.15.28:0/1109105988 wait complete. 2024-04-21T08:51:46.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.881+0000 7fc31ff9d640 1 Processor -- start 2024-04-21T08:51:46.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.881+0000 7fc31ff9d640 1 -- start start 2024-04-21T08:51:46.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.881+0000 7fc31ff9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3181031e0 0x7fc318198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.881+0000 7fc31ff9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc318103b00 0x7fc318198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc31dd39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3181031e0 0x7fc318198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc31ff9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc31810c520 0x7fc31819d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc31ff9d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc318112210 con 0x7fc3181031e0 2024-04-21T08:51:46.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc31ff9d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc318112090 con 0x7fc31810c520 2024-04-21T08:51:46.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc31dd39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3181031e0 0x7fc318198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41968/0 (socket says 172.21.15.28:41968) 2024-04-21T08:51:46.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc31ff9d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc318112390 con 0x7fc318103b00 2024-04-21T08:51:46.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc31dd39640 1 -- 172.21.15.28:0/3459022269 learned_addr learned my addr 172.21.15.28:0/3459022269 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:46.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc31d538640 1 --2- 172.21.15.28:0/3459022269 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc318103b00 0x7fc318198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc31e53a640 1 --2- 172.21.15.28:0/3459022269 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc31810c520 0x7fc31819d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc31dd39640 1 -- 172.21.15.28:0/3459022269 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc318103b00 msgr2=0x7fc318198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:46.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc31dd39640 1 --2- 172.21.15.28:0/3459022269 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc318103b00 0x7fc318198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc31dd39640 1 -- 172.21.15.28:0/3459022269 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc31810c520 msgr2=0x7fc31819d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:46.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc31dd39640 1 --2- 172.21.15.28:0/3459022269 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc31810c520 0x7fc31819d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc31dd39640 1 -- 172.21.15.28:0/3459022269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31819d740 con 0x7fc3181031e0 2024-04-21T08:51:46.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc31dd39640 1 --2- 172.21.15.28:0/3459022269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3181031e0 0x7fc318198750 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc314005940 tx=0x7fc31402d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:46.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.882+0000 7fc30affd640 1 -- 172.21.15.28:0/3459022269 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc314043020 con 0x7fc3181031e0 2024-04-21T08:51:46.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.883+0000 7fc31ff9d640 1 -- 172.21.15.28:0/3459022269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31819d9d0 con 0x7fc3181031e0 2024-04-21T08:51:46.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.883+0000 7fc30affd640 1 -- 172.21.15.28:0/3459022269 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc31402dd20 con 0x7fc3181031e0 2024-04-21T08:51:46.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.883+0000 7fc31ff9d640 1 -- 172.21.15.28:0/3459022269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3181a52c0 con 0x7fc3181031e0 2024-04-21T08:51:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.883+0000 7fc30affd640 1 -- 172.21.15.28:0/3459022269 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3140039f0 con 0x7fc3181031e0 2024-04-21T08:51:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.886+0000 7fc322b8e740 1 Processor -- start 2024-04-21T08:51:46.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.886+0000 7fc30affd640 1 -- 172.21.15.28:0/3459022269 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fc314046020 con 0x7fc3181031e0 2024-04-21T08:51:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.886+0000 7fc322b8e740 1 -- start start 2024-04-21T08:51:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.886+0000 7fc322b8e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 0x5605b76b7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.886+0000 7fc322b8e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b7b60 0x5605b76b7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.886+0000 7fc322b8e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76c34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.886+0000 7fc322b8e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5605b761c8d0 con 0x5605b76c10e0 2024-04-21T08:51:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.886+0000 7fc322b8e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5605b75b9e80 con 0x5605b76b88d0 2024-04-21T08:51:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.886+0000 7fc322b8e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5605b7618890 con 0x5605b76b7b60 2024-04-21T08:51:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.887+0000 7fc31d538640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76c34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.887+0000 7fc31d538640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76c34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41974/0 (socket says 172.21.15.28:41974) 2024-04-21T08:51:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.887+0000 7fc31d538640 1 -- 172.21.15.28:0/1166440619 learned_addr learned my addr 172.21.15.28:0/1166440619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.887+0000 7fc31e53a640 1 --2- 172.21.15.28:0/1166440619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 0x5605b76b7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.887+0000 7fc31dd39640 1 --2- 172.21.15.28:0/1166440619 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b7b60 0x5605b76b7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.887+0000 7fc30affd640 1 --2- 172.21.15.28:0/3459022269 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc3040a3ee0 0x7fc3040a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.887+0000 7fc31d538640 1 -- 172.21.15.28:0/1166440619 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b7b60 msgr2=0x5605b76b7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.887+0000 7fc31d538640 1 --2- 172.21.15.28:0/1166440619 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b7b60 0x5605b76b7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.887+0000 7fc31d538640 1 -- 172.21.15.28:0/1166440619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 msgr2=0x5605b76b7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.887+0000 7fc31d538640 1 --2- 172.21.15.28:0/1166440619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 0x5605b76b7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.887+0000 7fc31d538640 1 -- 172.21.15.28:0/1166440619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605b75db320 con 0x5605b76c10e0 2024-04-21T08:51:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.887+0000 7fc31e53a640 1 --2- 172.21.15.28:0/3459022269 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc3040a3ee0 0x7fc3040a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.887+0000 7fc30affd640 1 -- 172.21.15.28:0/3459022269 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fc3140f2ed0 con 0x7fc3181031e0 2024-04-21T08:51:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.888+0000 7fc31d538640 1 --2- 172.21.15.28:0/1166440619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76c34d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc30c009a40 tx=0x7fc30c009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e610b4448d16e76d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.888+0000 7fc303fff640 1 -- 172.21.15.28:0/1166440619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c017070 con 0x5605b76c10e0 2024-04-21T08:51:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.888+0000 7fc31e53a640 1 --2- 172.21.15.28:0/3459022269 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc3040a3ee0 0x7fc3040a63a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc310005fd0 tx=0x7fc310004a40 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.888+0000 7fc303fff640 1 -- 172.21.15.28:0/1166440619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc30c013470 con 0x5605b76c10e0 2024-04-21T08:51:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.888+0000 7fc303fff640 1 -- 172.21.15.28:0/1166440619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c012610 con 0x5605b76c10e0 2024-04-21T08:51:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.889+0000 7fc322b8e740 1 -- 172.21.15.28:0/1166440619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 msgr2=0x5605b76c34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.889+0000 7fc322b8e740 1 --2- 172.21.15.28:0/1166440619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76c34d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc30c009a40 tx=0x7fc30c009f00 comp rx=0 tx=0).stop 2024-04-21T08:51:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.889+0000 7fc322b8e740 1 -- 172.21.15.28:0/1166440619 shutdown_connections 2024-04-21T08:51:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.889+0000 7fc322b8e740 1 --2- 172.21.15.28:0/1166440619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76c34d0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.889+0000 7fc322b8e740 1 --2- 172.21.15.28:0/1166440619 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b7b60 0x5605b76b7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.889+0000 7fc322b8e740 1 --2- 172.21.15.28:0/1166440619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 0x5605b76b7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.889+0000 7fc322b8e740 1 -- 172.21.15.28:0/1166440619 >> 172.21.15.28:0/1166440619 conn(0x5605b767ec90 msgr2=0x5605b76b2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.889+0000 7fc322b8e740 1 -- 172.21.15.28:0/1166440619 shutdown_connections 2024-04-21T08:51:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.889+0000 7fc322b8e740 1 -- 172.21.15.28:0/1166440619 wait complete. 2024-04-21T08:51:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.890+0000 7fc322b8e740 1 -- start start 2024-04-21T08:51:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.890+0000 7fc322b8e740 1 Processor -- start 2024-04-21T08:51:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.890+0000 7fc322b8e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b7b60 0x5605b76c09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc322b8e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b88d0 0x5605b76bdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc322b8e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76be2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc322b8e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5605b76c3e10 con 0x5605b76c10e0 2024-04-21T08:51:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc322b8e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5605b75a4eb0 con 0x5605b76b7b60 2024-04-21T08:51:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc31e53a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76be2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc31dd39640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b7b60 0x5605b76c09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc31dd39640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b7b60 0x5605b76c09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:39834/0 (socket says 172.21.15.28:39834) 2024-04-21T08:51:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc31dd39640 1 -- 172.21.15.28:0/445038218 learned_addr learned my addr 172.21.15.28:0/445038218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc31e53a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76be2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41976/0 (socket says 172.21.15.28:41976) 2024-04-21T08:51:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc322b8e740 1 -- 172.21.15.28:0/445038218 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5605b76c3f90 con 0x5605b76b88d0 2024-04-21T08:51:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc31d538640 1 --2- 172.21.15.28:0/445038218 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b88d0 0x5605b76bdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc322b8e740 10 client.0 _ll_register_callbacks cb 0x5605b75d1b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc31dd39640 1 -- 172.21.15.28:0/445038218 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b88d0 msgr2=0x5605b76bdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc31dd39640 1 --2- 172.21.15.28:0/445038218 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b88d0 0x5605b76bdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc31dd39640 1 -- 172.21.15.28:0/445038218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 msgr2=0x5605b76be2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc31dd39640 1 --2- 172.21.15.28:0/445038218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76be2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc31dd39640 1 -- 172.21.15.28:0/445038218 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605b732f210 con 0x5605b76b7b60 2024-04-21T08:51:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.891+0000 7fc31d538640 1 --2- 172.21.15.28:0/445038218 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b88d0 0x5605b76bdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.892+0000 7fc31dd39640 1 --2- 172.21.15.28:0/445038218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b7b60 0x5605b76c09e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc3140be920 tx=0x7fc3140bacf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6326490940a5f0f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.892+0000 7fc3017fa640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:51:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.892+0000 7fc3017fa640 1 -- 172.21.15.28:0/445038218 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3140be2f0 con 0x5605b76b7b60 2024-04-21T08:51:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.892+0000 7fc322b8e740 1 -- 172.21.15.28:0/445038218 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc30c013470 con 0x5605b76b7b60 2024-04-21T08:51:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.892+0000 7fc3017fa640 1 -- 172.21.15.28:0/445038218 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fc3140a9310 con 0x5605b76b7b60 2024-04-21T08:51:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.892+0000 7fc3017fa640 1 -- 172.21.15.28:0/445038218 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc3140bf090 con 0x5605b76b7b60 2024-04-21T08:51:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.892+0000 7fc322b8e740 10 client.25406 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.892+0000 7fc322b8e740 1 -- 172.21.15.28:0/445038218 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5605b73fb620 con 0x5605b76b7b60 2024-04-21T08:51:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.892+0000 7fc3017fa640 1 -- 172.21.15.28:0/445038218 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc30c013470 con 0x5605b76b7b60 2024-04-21T08:51:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.893+0000 7fc3017fa640 1 -- 172.21.15.28:0/445038218 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc3140c7170 con 0x5605b76b7b60 2024-04-21T08:51:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.893+0000 7fc322b8e740 10 client.25406 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.893+0000 7fc322b8e740 10 client.25406 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.893+0000 7fc322b8e740 20 client.25406 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.893+0000 7fc322b8e740 1 --2- 172.21.15.28:0/445038218 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5605b76d1180 0x5605b76d3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.893+0000 7fc322b8e740 4 client.25406 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.893+0000 7fc322b8e740 1 -- 172.21.15.28:0/445038218 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605b7630880 con 0x5605b76d1180 2024-04-21T08:51:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.893+0000 7fc31d538640 1 --2- 172.21.15.28:0/445038218 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5605b76d1180 0x5605b76d3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.893+0000 7fc3017fa640 10 client.25406 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.893+0000 7fc31d538640 1 --2- 172.21.15.28:0/445038218 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5605b76d1180 0x5605b76d3640 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=faafd73d71fc838e server_cookie=c3a24ee49b4c4fd in_seq=0 out_seq=0 2024-04-21T08:51:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.894+0000 7fc3017fa640 1 -- 172.21.15.28:0/445038218 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5605b7630880 con 0x5605b76d1180 2024-04-21T08:51:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.894+0000 7fc3017fa640 10 client.25406 handle_command_reply: tid=0 2024-04-21T08:51:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.894+0000 7fc322b8e740 1 client.25406 shutdown 2024-04-21T08:51:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.894+0000 7fc322b8e740 1 -- 172.21.15.28:0/445038218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b7b60 msgr2=0x5605b76c09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.894+0000 7fc322b8e740 1 --2- 172.21.15.28:0/445038218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b7b60 0x5605b76c09e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc3140be920 tx=0x7fc3140bacf0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.894+0000 7fc322b8e740 1 -- 172.21.15.28:0/445038218 shutdown_connections 2024-04-21T08:51:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.895+0000 7fc322b8e740 1 --2- 172.21.15.28:0/445038218 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5605b76d1180 0x5605b76d3640 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.895+0000 7fc3017fa640 0 client.25406 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.895+0000 7fc322b8e740 1 --2- 172.21.15.28:0/445038218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76be2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.895+0000 7fc322b8e740 1 --2- 172.21.15.28:0/445038218 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b88d0 0x5605b76bdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.895+0000 7fc322b8e740 1 --2- 172.21.15.28:0/445038218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b7b60 0x5605b76c09e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.895+0000 7fc322b8e740 1 -- 172.21.15.28:0/445038218 >> 172.21.15.28:0/445038218 conn(0x5605b767ec90 msgr2=0x5605b76b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.895+0000 7fc322b8e740 1 -- 172.21.15.28:0/445038218 shutdown_connections 2024-04-21T08:51:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.895+0000 7fc322b8e740 1 -- 172.21.15.28:0/445038218 wait complete. 2024-04-21T08:51:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.895+0000 7fc322b8e740 20 client.25406 trim_cache size 0 max 16384 2024-04-21T08:51:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.910+0000 7fc322b8e740 1 Processor -- start 2024-04-21T08:51:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.910+0000 7fc322b8e740 1 -- start start 2024-04-21T08:51:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.910+0000 7fc322b8e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b7b60 0x5605b76ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.910+0000 7fc322b8e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 0x5605b76c04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.910+0000 7fc322b8e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76c09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.910+0000 7fc322b8e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc0127b0 con 0x5605b76c10e0 2024-04-21T08:51:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.910+0000 7fc322b8e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc010080 con 0x5605b76b88d0 2024-04-21T08:51:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.910+0000 7fc322b8e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc00ded0 con 0x5605b76b7b60 2024-04-21T08:51:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.910+0000 7fc31e53a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76c09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.910+0000 7fc31e53a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76c09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41978/0 (socket says 172.21.15.28:41978) 2024-04-21T08:51:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.910+0000 7fc31e53a640 1 -- 172.21.15.28:0/3269584508 learned_addr learned my addr 172.21.15.28:0/3269584508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.910+0000 7fc31dd39640 1 --2- 172.21.15.28:0/3269584508 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b7b60 0x5605b76ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.910+0000 7fc31d538640 1 --2- 172.21.15.28:0/3269584508 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 0x5605b76c04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.911+0000 7fc31e53a640 1 -- 172.21.15.28:0/3269584508 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b7b60 msgr2=0x5605b76ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.911+0000 7fc31e53a640 1 --2- 172.21.15.28:0/3269584508 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b7b60 0x5605b76ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.911+0000 7fc31e53a640 1 -- 172.21.15.28:0/3269584508 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 msgr2=0x5605b76c04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.911+0000 7fc31e53a640 1 --2- 172.21.15.28:0/3269584508 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 0x5605b76c04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.911+0000 7fc31e53a640 1 -- 172.21.15.28:0/3269584508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605b76b6e30 con 0x5605b76c10e0 2024-04-21T08:51:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.911+0000 7fc31e53a640 1 --2- 172.21.15.28:0/3269584508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76c09e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc310012900 tx=0x7fc310014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b909f04e12da2b8a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.911+0000 7fc3037fe640 1 -- 172.21.15.28:0/3269584508 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31001e8d0 con 0x5605b76c10e0 2024-04-21T08:51:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.911+0000 7fc3037fe640 1 -- 172.21.15.28:0/3269584508 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc310016070 con 0x5605b76c10e0 2024-04-21T08:51:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.911+0000 7fc3037fe640 1 -- 172.21.15.28:0/3269584508 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31001d450 con 0x5605b76c10e0 2024-04-21T08:51:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.912+0000 7fc322b8e740 1 -- 172.21.15.28:0/3269584508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 msgr2=0x5605b76c09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.912+0000 7fc322b8e740 1 --2- 172.21.15.28:0/3269584508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76c09e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc310012900 tx=0x7fc310014000 comp rx=0 tx=0).stop 2024-04-21T08:51:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.912+0000 7fc322b8e740 1 -- 172.21.15.28:0/3269584508 shutdown_connections 2024-04-21T08:51:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.912+0000 7fc322b8e740 1 --2- 172.21.15.28:0/3269584508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b76c09e0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.912+0000 7fc322b8e740 1 --2- 172.21.15.28:0/3269584508 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 0x5605b76c04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.912+0000 7fc322b8e740 1 --2- 172.21.15.28:0/3269584508 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b7b60 0x5605b76ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.912+0000 7fc322b8e740 1 -- 172.21.15.28:0/3269584508 >> 172.21.15.28:0/3269584508 conn(0x5605b767ec90 msgr2=0x5605b76bdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.912+0000 7fc322b8e740 1 -- 172.21.15.28:0/3269584508 shutdown_connections 2024-04-21T08:51:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.912+0000 7fc322b8e740 1 -- 172.21.15.28:0/3269584508 wait complete. 2024-04-21T08:51:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.912+0000 7fc322b8e740 1 -- start start 2024-04-21T08:51:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.913+0000 7fc322b8e740 1 Processor -- start 2024-04-21T08:51:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.913+0000 7fc322b8e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b7b60 0x5605b76c4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.913+0000 7fc322b8e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 0x5605b76c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.913+0000 7fc322b8e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b772f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.913+0000 7fc322b8e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc00e540 con 0x5605b76c10e0 2024-04-21T08:51:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.913+0000 7fc322b8e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc00f9b0 con 0x5605b76b88d0 2024-04-21T08:51:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.913+0000 7fc322b8e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc00ec10 con 0x5605b76b7b60 2024-04-21T08:51:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.913+0000 7fc322b8e740 10 client.0 _ll_register_callbacks cb 0x7fc3140057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.913+0000 7fc31d538640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 0x5605b76c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.913+0000 7fc31d538640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 0x5605b76c4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:39860/0 (socket says 172.21.15.28:39860) 2024-04-21T08:51:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.913+0000 7fc31d538640 1 -- 172.21.15.28:0/2907789867 learned_addr learned my addr 172.21.15.28:0/2907789867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.913+0000 7fc31e53a640 1 --2- 172.21.15.28:0/2907789867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b772f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.913+0000 7fc31dd39640 1 --2- 172.21.15.28:0/2907789867 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b7b60 0x5605b76c4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.914+0000 7fc31d538640 1 -- 172.21.15.28:0/2907789867 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b7b60 msgr2=0x5605b76c4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.914+0000 7fc31d538640 1 --2- 172.21.15.28:0/2907789867 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b7b60 0x5605b76c4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.914+0000 7fc31d538640 1 -- 172.21.15.28:0/2907789867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 msgr2=0x5605b772f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.914+0000 7fc31d538640 1 --2- 172.21.15.28:0/2907789867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b772f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.914+0000 7fc31d538640 1 -- 172.21.15.28:0/2907789867 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605b772fdf0 con 0x5605b76b88d0 2024-04-21T08:51:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.914+0000 7fc31e53a640 1 --2- 172.21.15.28:0/2907789867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b772f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.914+0000 7fc31d538640 1 --2- 172.21.15.28:0/2907789867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 0x5605b76c4a50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc30c008010 tx=0x7fc30c013e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a56e0e88a342c3d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.914+0000 7fc3017fa640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:51:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.914+0000 7fc3017fa640 1 -- 172.21.15.28:0/2907789867 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c017040 con 0x5605b76b88d0 2024-04-21T08:51:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.914+0000 7fc322b8e740 1 -- 172.21.15.28:0/2907789867 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc310016070 con 0x5605b76b88d0 2024-04-21T08:51:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.914+0000 7fc3017fa640 1 -- 172.21.15.28:0/2907789867 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fc30c005bc0 con 0x5605b76b88d0 2024-04-21T08:51:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.918+0000 7fc3017fa640 1 -- 172.21.15.28:0/2907789867 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc30c034050 con 0x5605b76b88d0 2024-04-21T08:51:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.918+0000 7fc322b8e740 10 client.25412 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.918+0000 7fc322b8e740 1 -- 172.21.15.28:0/2907789867 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5605b773f9d0 con 0x5605b76b88d0 2024-04-21T08:51:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.918+0000 7fc3017fa640 1 -- 172.21.15.28:0/2907789867 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc310016070 con 0x5605b76b88d0 2024-04-21T08:51:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.919+0000 7fc3017fa640 1 -- 172.21.15.28:0/2907789867 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc30c0137f0 con 0x5605b76b88d0 2024-04-21T08:51:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.919+0000 7fc322b8e740 10 client.25412 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.919+0000 7fc322b8e740 10 client.25412 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.919+0000 7fc322b8e740 20 client.25412 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.919+0000 7fc322b8e740 1 --2- 172.21.15.28:0/2907789867 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5605b76d1180 0x5605b7741f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.919+0000 7fc322b8e740 4 client.25412 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:51:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.919+0000 7fc322b8e740 1 -- 172.21.15.28:0/2907789867 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5605b75bc3d0 con 0x5605b76d1180 2024-04-21T08:51:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.919+0000 7fc31dd39640 1 --2- 172.21.15.28:0/2907789867 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5605b76d1180 0x5605b7741f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.920+0000 7fc3017fa640 10 client.25412 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.920+0000 7fc31dd39640 1 --2- 172.21.15.28:0/2907789867 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5605b76d1180 0x5605b7741f10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9932ae0a37b81739 server_cookie=f81761fdadeb1e0a in_seq=0 out_seq=0 2024-04-21T08:51:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.924+0000 7fc3017fa640 1 -- 172.21.15.28:0/2907789867 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5605b75bc3d0 con 0x5605b76d1180 2024-04-21T08:51:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.924+0000 7fc3017fa640 10 client.25412 handle_command_reply: tid=0 2024-04-21T08:51:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.924+0000 7fc322b8e740 1 client.25412 shutdown 2024-04-21T08:51:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.924+0000 7fc322b8e740 1 -- 172.21.15.28:0/2907789867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 msgr2=0x5605b76c4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.924+0000 7fc322b8e740 1 --2- 172.21.15.28:0/2907789867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 0x5605b76c4a50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc30c008010 tx=0x7fc30c013e20 comp rx=0 tx=0).stop 2024-04-21T08:51:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.924+0000 7fc322b8e740 1 -- 172.21.15.28:0/2907789867 shutdown_connections 2024-04-21T08:51:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.924+0000 7fc322b8e740 1 --2- 172.21.15.28:0/2907789867 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5605b76d1180 0x5605b7741f10 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.924+0000 7fc3017fa640 0 client.25412 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.924+0000 7fc322b8e740 1 --2- 172.21.15.28:0/2907789867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605b76c10e0 0x5605b772f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.924+0000 7fc322b8e740 1 --2- 172.21.15.28:0/2907789867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605b76b88d0 0x5605b76c4a50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.924+0000 7fc322b8e740 1 --2- 172.21.15.28:0/2907789867 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5605b76b7b60 0x5605b76c4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.924+0000 7fc322b8e740 1 -- 172.21.15.28:0/2907789867 >> 172.21.15.28:0/2907789867 conn(0x5605b767ec90 msgr2=0x5605b76b8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.924+0000 7fc322b8e740 1 -- 172.21.15.28:0/2907789867 shutdown_connections 2024-04-21T08:51:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.924+0000 7fc322b8e740 1 -- 172.21.15.28:0/2907789867 wait complete. 2024-04-21T08:51:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.925+0000 7fc322b8e740 20 client.25412 trim_cache size 0 max 16384 2024-04-21T08:51:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:51:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:51:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:51:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:51:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.925+0000 7fc31ff9d640 1 -- 172.21.15.28:0/3459022269 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc3040a3ee0 msgr2=0x7fc3040a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.925+0000 7fc31ff9d640 1 --2- 172.21.15.28:0/3459022269 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc3040a3ee0 0x7fc3040a63a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc310005fd0 tx=0x7fc310004a40 comp rx=0 tx=0).stop 2024-04-21T08:51:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.925+0000 7fc31ff9d640 1 -- 172.21.15.28:0/3459022269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3181031e0 msgr2=0x7fc318198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.925+0000 7fc31ff9d640 1 --2- 172.21.15.28:0/3459022269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3181031e0 0x7fc318198750 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc314005940 tx=0x7fc31402d920 comp rx=0 tx=0).stop 2024-04-21T08:51:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.925+0000 7fc31ff9d640 1 -- 172.21.15.28:0/3459022269 shutdown_connections 2024-04-21T08:51:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.925+0000 7fc31ff9d640 1 --2- 172.21.15.28:0/3459022269 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc3040a3ee0 0x7fc3040a63a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.925+0000 7fc31ff9d640 1 --2- 172.21.15.28:0/3459022269 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc31810c520 0x7fc31819d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.925+0000 7fc31ff9d640 1 --2- 172.21.15.28:0/3459022269 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc318103b00 0x7fc318198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.925+0000 7fc31ff9d640 1 --2- 172.21.15.28:0/3459022269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3181031e0 0x7fc318198750 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.925+0000 7fc31ff9d640 1 -- 172.21.15.28:0/3459022269 >> 172.21.15.28:0/3459022269 conn(0x7fc318077fd0 msgr2=0x7fc3180fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.926+0000 7fc31ff9d640 1 -- 172.21.15.28:0/3459022269 shutdown_connections 2024-04-21T08:51:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:51:46.926+0000 7fc31ff9d640 1 -- 172.21.15.28:0/3459022269 wait complete. 2024-04-21T08:51:48.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:47 smithi152 ceph-mon[31401]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-21T08:51:48.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:47 smithi028 ceph-mon[28003]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-21T08:51:48.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:47 smithi171 ceph-mon[31358]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-21T08:51:49.300 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:51:49.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:51:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.410+0000 7fc7484c1640 1 -- 172.21.15.28:0/103453573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc740101080 msgr2=0x7fc740101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.410+0000 7fc7484c1640 1 --2- 172.21.15.28:0/103453573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc740101080 0x7fc740101460 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc73c009900 tx=0x7fc73c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.410+0000 7fc7484c1640 1 -- 172.21.15.28:0/103453573 shutdown_connections 2024-04-21T08:51:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.410+0000 7fc7484c1640 1 --2- 172.21.15.28:0/103453573 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc740106150 0x7fc74010dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.410+0000 7fc7484c1640 1 --2- 172.21.15.28:0/103453573 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc740101a30 0x7fc740105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.410+0000 7fc7484c1640 1 --2- 172.21.15.28:0/103453573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc740101080 0x7fc740101460 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.410+0000 7fc7484c1640 1 -- 172.21.15.28:0/103453573 >> 172.21.15.28:0/103453573 conn(0x7fc740077f30 msgr2=0x7fc740078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.410+0000 7fc7484c1640 1 -- 172.21.15.28:0/103453573 shutdown_connections 2024-04-21T08:51:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.410+0000 7fc7484c1640 1 -- 172.21.15.28:0/103453573 wait complete. 2024-04-21T08:51:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.411+0000 7fc7484c1640 1 Processor -- start 2024-04-21T08:51:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.411+0000 7fc7484c1640 1 -- start start 2024-04-21T08:51:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.411+0000 7fc7484c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc740101080 0x7fc740199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.411+0000 7fc7484c1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc740106150 0x7fc740199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.411+0000 7fc7484c1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc74019a090 0x7fc7401a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.411+0000 7fc7484c1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7401110b0 con 0x7fc740101080 2024-04-21T08:51:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.411+0000 7fc7484c1640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc740110f30 con 0x7fc740106150 2024-04-21T08:51:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.411+0000 7fc7484c1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc740111230 con 0x7fc74019a090 2024-04-21T08:51:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.411+0000 7fc74625d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc740101080 0x7fc740199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.411+0000 7fc74625d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc740101080 0x7fc740199570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42000/0 (socket says 172.21.15.28:42000) 2024-04-21T08:51:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.411+0000 7fc74625d640 1 -- 172.21.15.28:0/1295549122 learned_addr learned my addr 172.21.15.28:0/1295549122 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.411+0000 7fc745a5c640 1 --2- 172.21.15.28:0/1295549122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc740106150 0x7fc740199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.411+0000 7fc746a5e640 1 --2- 172.21.15.28:0/1295549122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc74019a090 0x7fc7401a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.412+0000 7fc74625d640 1 -- 172.21.15.28:0/1295549122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc74019a090 msgr2=0x7fc7401a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.412+0000 7fc74625d640 1 --2- 172.21.15.28:0/1295549122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc74019a090 0x7fc7401a45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.412+0000 7fc74625d640 1 -- 172.21.15.28:0/1295549122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc740106150 msgr2=0x7fc740199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.412+0000 7fc74625d640 1 --2- 172.21.15.28:0/1295549122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc740106150 0x7fc740199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.412+0000 7fc74625d640 1 -- 172.21.15.28:0/1295549122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7401a4e10 con 0x7fc740101080 2024-04-21T08:51:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.412+0000 7fc74625d640 1 --2- 172.21.15.28:0/1295549122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc740101080 0x7fc740199570 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc73c005940 tx=0x7fc73c02c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.412+0000 7fc7377fe640 1 -- 172.21.15.28:0/1295549122 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c043020 con 0x7fc740101080 2024-04-21T08:51:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.412+0000 7fc7377fe640 1 -- 172.21.15.28:0/1295549122 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc73c02ca70 con 0x7fc740101080 2024-04-21T08:51:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.412+0000 7fc7484c1640 1 -- 172.21.15.28:0/1295549122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7401a50a0 con 0x7fc740101080 2024-04-21T08:51:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.412+0000 7fc7377fe640 1 -- 172.21.15.28:0/1295549122 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c003970 con 0x7fc740101080 2024-04-21T08:51:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.412+0000 7fc7484c1640 1 -- 172.21.15.28:0/1295549122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7401a55a0 con 0x7fc740101080 2024-04-21T08:51:49.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.415+0000 7fc74b192740 1 Processor -- start 2024-04-21T08:51:49.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.415+0000 7fc74b192740 1 -- start start 2024-04-21T08:51:49.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.416+0000 7fc74b192740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee1a390 0x564d5ee1a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.416+0000 7fc74b192740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1ad40 0x564d5ee27950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.416+0000 7fc74b192740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee27e90 0x564d5ee2a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.416+0000 7fc74b192740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564d5ed7f1e0 con 0x564d5ee1ad40 2024-04-21T08:51:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.416+0000 7fc74b192740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564d5ed1c390 con 0x564d5ee1a390 2024-04-21T08:51:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.416+0000 7fc74b192740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564d5ed7b1a0 con 0x564d5ee27e90 2024-04-21T08:51:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.416+0000 7fc7377fe640 1 -- 172.21.15.28:0/1295549122 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fc73c046020 con 0x7fc740101080 2024-04-21T08:51:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.416+0000 7fc746a5e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee1a390 0x564d5ee1a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.416+0000 7fc745a5c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee27e90 0x564d5ee2a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.416+0000 7fc746a5e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee1a390 0x564d5ee1a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:39878/0 (socket says 172.21.15.28:39878) 2024-04-21T08:51:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.416+0000 7fc746a5e640 1 -- 172.21.15.28:0/2783858848 learned_addr learned my addr 172.21.15.28:0/2783858848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.416+0000 7fc746a5e640 1 -- 172.21.15.28:0/2783858848 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee27e90 msgr2=0x564d5ee2a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.416+0000 7fc746a5e640 1 --2- 172.21.15.28:0/2783858848 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee27e90 0x564d5ee2a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.416+0000 7fc746a5e640 1 -- 172.21.15.28:0/2783858848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1ad40 msgr2=0x564d5ee27950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:49.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.417+0000 7fc74625d640 1 --2- 172.21.15.28:0/2783858848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1ad40 0x564d5ee27950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.417+0000 7fc746a5e640 1 --2- 172.21.15.28:0/2783858848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1ad40 0x564d5ee27950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.417+0000 7fc746a5e640 1 -- 172.21.15.28:0/2783858848 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d5eb5b7d0 con 0x564d5ee1a390 2024-04-21T08:51:49.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.417+0000 7fc74625d640 1 --2- 172.21.15.28:0/2783858848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1ad40 0x564d5ee27950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:49.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.417+0000 7fc746a5e640 1 --2- 172.21.15.28:0/2783858848 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee1a390 0x564d5ee1a770 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc738009f90 tx=0x7fc7380118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96e9a60f2f865553 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:49.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.417+0000 7fc7377fe640 1 --2- 172.21.15.28:0/1295549122 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc72c0a3e40 0x7fc72c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.417+0000 7fc734bfe640 1 -- 172.21.15.28:0/2783858848 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73801e070 con 0x564d5ee1a390 2024-04-21T08:51:49.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.417+0000 7fc734bfe640 1 -- 172.21.15.28:0/2783858848 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc738003c30 con 0x564d5ee1a390 2024-04-21T08:51:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.417+0000 7fc734bfe640 1 -- 172.21.15.28:0/2783858848 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73800f8a0 con 0x564d5ee1a390 2024-04-21T08:51:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.417+0000 7fc7377fe640 1 -- 172.21.15.28:0/1295549122 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fc73c0f2d80 con 0x7fc740101080 2024-04-21T08:51:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.417+0000 7fc74625d640 1 --2- 172.21.15.28:0/1295549122 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc72c0a3e40 0x7fc72c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.418+0000 7fc74625d640 1 --2- 172.21.15.28:0/1295549122 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc72c0a3e40 0x7fc72c0a6300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc73c0f0120 tx=0x7fc73c02bc30 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.418+0000 7fc74b192740 1 -- 172.21.15.28:0/2783858848 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee1a390 msgr2=0x564d5ee1a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.418+0000 7fc74b192740 1 --2- 172.21.15.28:0/2783858848 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee1a390 0x564d5ee1a770 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc738009f90 tx=0x7fc7380118d0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.418+0000 7fc74b192740 1 -- 172.21.15.28:0/2783858848 shutdown_connections 2024-04-21T08:51:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.418+0000 7fc74b192740 1 --2- 172.21.15.28:0/2783858848 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee27e90 0x564d5ee2a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.418+0000 7fc74b192740 1 --2- 172.21.15.28:0/2783858848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1ad40 0x564d5ee27950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.418+0000 7fc74b192740 1 --2- 172.21.15.28:0/2783858848 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee1a390 0x564d5ee1a770 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.418+0000 7fc74b192740 1 -- 172.21.15.28:0/2783858848 >> 172.21.15.28:0/2783858848 conn(0x564d5ee14cb0 msgr2=0x564d5ee15090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.418+0000 7fc74b192740 1 -- 172.21.15.28:0/2783858848 shutdown_connections 2024-04-21T08:51:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.418+0000 7fc74b192740 1 -- 172.21.15.28:0/2783858848 wait complete. 2024-04-21T08:51:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.419+0000 7fc74b192740 1 -- start start 2024-04-21T08:51:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.419+0000 7fc74b192740 1 Processor -- start 2024-04-21T08:51:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.419+0000 7fc74b192740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1a390 0x564d5ee24f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.419+0000 7fc74b192740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1ad40 0x564d5ee25450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.419+0000 7fc745a5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1ad40 0x564d5ee25450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.419+0000 7fc745a5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1ad40 0x564d5ee25450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42030/0 (socket says 172.21.15.28:42030) 2024-04-21T08:51:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.419+0000 7fc745a5c640 1 -- 172.21.15.28:0/3219715314 learned_addr learned my addr 172.21.15.28:0/3219715314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.419+0000 7fc746a5e640 1 --2- 172.21.15.28:0/3219715314 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1a390 0x564d5ee24f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.419+0000 7fc74b192740 1 --2- 172.21.15.28:0/3219715314 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee27e90 0x564d5ee25990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.419+0000 7fc74b192740 1 -- 172.21.15.28:0/3219715314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564d5ee2ac50 con 0x564d5ee1ad40 2024-04-21T08:51:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.420+0000 7fc74b192740 1 -- 172.21.15.28:0/3219715314 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564d5ed382f0 con 0x564d5ee27e90 2024-04-21T08:51:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.420+0000 7fc74b192740 1 -- 172.21.15.28:0/3219715314 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564d5ee2add0 con 0x564d5ee1a390 2024-04-21T08:51:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.420+0000 7fc746a5e640 1 -- 172.21.15.28:0/3219715314 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee27e90 msgr2=0x564d5ee25990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.420+0000 7fc746a5e640 1 --2- 172.21.15.28:0/3219715314 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee27e90 0x564d5ee25990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.420+0000 7fc746a5e640 1 -- 172.21.15.28:0/3219715314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1ad40 msgr2=0x564d5ee25450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.420+0000 7fc746a5e640 1 --2- 172.21.15.28:0/3219715314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1ad40 0x564d5ee25450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.420+0000 7fc746a5e640 1 -- 172.21.15.28:0/3219715314 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d5ea913f0 con 0x564d5ee1a390 2024-04-21T08:51:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.420+0000 7fc745a5c640 1 --2- 172.21.15.28:0/3219715314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1ad40 0x564d5ee25450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.420+0000 7fc74b192740 10 client.0 _ll_register_callbacks cb 0x564d5ed45580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.420+0000 7fc74b192740 1 -- 172.21.15.28:0/3219715314 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc738003c30 con 0x564d5ee1a390 2024-04-21T08:51:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.420+0000 7fc746a5e640 1 --2- 172.21.15.28:0/3219715314 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1a390 0x564d5ee24f10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc738006010 tx=0x7fc73800f810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77731e0e6fa9de76 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.420+0000 7fc729ffb640 10 client.25408 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:51:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.421+0000 7fc729ffb640 1 -- 172.21.15.28:0/3219715314 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc738027030 con 0x564d5ee1a390 2024-04-21T08:51:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.421+0000 7fc729ffb640 1 -- 172.21.15.28:0/3219715314 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fc738003a30 con 0x564d5ee1a390 2024-04-21T08:51:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.421+0000 7fc729ffb640 1 -- 172.21.15.28:0/3219715314 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc73801e040 con 0x564d5ee1a390 2024-04-21T08:51:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.421+0000 7fc74b192740 10 client.25408 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.421+0000 7fc74b192740 1 -- 172.21.15.28:0/3219715314 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({fsmap=25}) -- 0x564d5ea510f0 con 0x564d5ee1a390 2024-04-21T08:51:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.421+0000 7fc729ffb640 1 -- 172.21.15.28:0/3219715314 <== mon.1 v2:172.21.15.171:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc738004a40 con 0x564d5ee1a390 2024-04-21T08:51:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.421+0000 7fc74b192740 10 client.25408 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.421+0000 7fc74b192740 10 client.25408 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.421+0000 7fc74b192740 20 client.25408 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.421+0000 7fc74b192740 1 --2- 172.21.15.28:0/3219715314 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564d5ee37a40 0x564d5ee39e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.421+0000 7fc74b192740 4 client.25408 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.421+0000 7fc74b192740 1 -- 172.21.15.28:0/3219715314 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d5ed8cc00 con 0x564d5ee37a40 2024-04-21T08:51:49.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.422+0000 7fc745a5c640 1 --2- 172.21.15.28:0/3219715314 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564d5ee37a40 0x564d5ee39e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.422+0000 7fc745a5c640 1 --2- 172.21.15.28:0/3219715314 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564d5ee37a40 0x564d5ee39e30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66cddb4f8ebcc52d server_cookie=9da278e8b4d89b97 in_seq=0 out_seq=0 2024-04-21T08:51:49.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.422+0000 7fc729ffb640 10 client.25408 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:49.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.423+0000 7fc729ffb640 1 -- 172.21.15.28:0/3219715314 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d5ed8cc00 con 0x564d5ee37a40 2024-04-21T08:51:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.423+0000 7fc729ffb640 10 client.25408 handle_command_reply: tid=0 2024-04-21T08:51:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.423+0000 7fc74b192740 1 client.25408 shutdown 2024-04-21T08:51:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.426+0000 7fc74b192740 1 -- 172.21.15.28:0/3219715314 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1a390 msgr2=0x564d5ee24f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.426+0000 7fc74b192740 1 --2- 172.21.15.28:0/3219715314 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1a390 0x564d5ee24f10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc738006010 tx=0x7fc73800f810 comp rx=0 tx=0).stop 2024-04-21T08:51:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.427+0000 7fc74b192740 1 -- 172.21.15.28:0/3219715314 shutdown_connections 2024-04-21T08:51:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.427+0000 7fc74b192740 1 --2- 172.21.15.28:0/3219715314 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564d5ee37a40 0x564d5ee39e30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.427+0000 7fc729ffb640 0 client.25408 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.427+0000 7fc74b192740 1 --2- 172.21.15.28:0/3219715314 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee27e90 0x564d5ee25990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.427+0000 7fc74b192740 1 --2- 172.21.15.28:0/3219715314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1ad40 0x564d5ee25450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.427+0000 7fc74b192740 1 --2- 172.21.15.28:0/3219715314 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1a390 0x564d5ee24f10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.427+0000 7fc74b192740 1 -- 172.21.15.28:0/3219715314 >> 172.21.15.28:0/3219715314 conn(0x564d5ee14cb0 msgr2=0x564d5ee16a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.427+0000 7fc74b192740 1 -- 172.21.15.28:0/3219715314 shutdown_connections 2024-04-21T08:51:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.427+0000 7fc74b192740 1 -- 172.21.15.28:0/3219715314 wait complete. 2024-04-21T08:51:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.427+0000 7fc74b192740 20 client.25408 trim_cache size 0 max 16384 2024-04-21T08:51:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.444+0000 7fc74b192740 1 Processor -- start 2024-04-21T08:51:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.444+0000 7fc74b192740 1 -- start start 2024-04-21T08:51:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc74b192740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1a390 0x564d5ee23e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc74b192740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1ad40 0x564d5ee33b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc74b192740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee27e90 0x564d5ee257b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc74b192740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7401af930 con 0x564d5ee1a390 2024-04-21T08:51:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc74b192740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc7401ad200 con 0x564d5ee27e90 2024-04-21T08:51:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc74b192740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc7401ab050 con 0x564d5ee1ad40 2024-04-21T08:51:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc746a5e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1a390 0x564d5ee23e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc746a5e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1a390 0x564d5ee23e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42034/0 (socket says 172.21.15.28:42034) 2024-04-21T08:51:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc746a5e640 1 -- 172.21.15.28:0/1455179259 learned_addr learned my addr 172.21.15.28:0/1455179259 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc745a5c640 1 --2- 172.21.15.28:0/1455179259 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1ad40 0x564d5ee33b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc746a5e640 1 -- 172.21.15.28:0/1455179259 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1ad40 msgr2=0x564d5ee33b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc746a5e640 1 --2- 172.21.15.28:0/1455179259 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1ad40 0x564d5ee33b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc746a5e640 1 -- 172.21.15.28:0/1455179259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee27e90 msgr2=0x564d5ee257b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:51:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc746a5e640 1 --2- 172.21.15.28:0/1455179259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee27e90 0x564d5ee257b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc746a5e640 1 -- 172.21.15.28:0/1455179259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d5ee15a40 con 0x564d5ee1a390 2024-04-21T08:51:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.445+0000 7fc745a5c640 1 --2- 172.21.15.28:0/1455179259 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1ad40 0x564d5ee33b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.446+0000 7fc746a5e640 1 --2- 172.21.15.28:0/1455179259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1a390 0x564d5ee23e50 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc7380045f0 tx=0x7fc738023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc9283537b907f41 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.446+0000 7fc72bfff640 1 -- 172.21.15.28:0/1455179259 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc738027040 con 0x564d5ee1a390 2024-04-21T08:51:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.446+0000 7fc72bfff640 1 -- 172.21.15.28:0/1455179259 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc738003c30 con 0x564d5ee1a390 2024-04-21T08:51:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.446+0000 7fc72bfff640 1 -- 172.21.15.28:0/1455179259 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73800fca0 con 0x564d5ee1a390 2024-04-21T08:51:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.447+0000 7fc74b192740 1 -- 172.21.15.28:0/1455179259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1a390 msgr2=0x564d5ee23e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.447+0000 7fc74b192740 1 --2- 172.21.15.28:0/1455179259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1a390 0x564d5ee23e50 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc7380045f0 tx=0x7fc738023000 comp rx=0 tx=0).stop 2024-04-21T08:51:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.447+0000 7fc74b192740 1 -- 172.21.15.28:0/1455179259 shutdown_connections 2024-04-21T08:51:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.447+0000 7fc74b192740 1 --2- 172.21.15.28:0/1455179259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee27e90 0x564d5ee257b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.447+0000 7fc74b192740 1 --2- 172.21.15.28:0/1455179259 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1ad40 0x564d5ee33b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.447+0000 7fc74b192740 1 --2- 172.21.15.28:0/1455179259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee1a390 0x564d5ee23e50 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.447+0000 7fc74b192740 1 -- 172.21.15.28:0/1455179259 >> 172.21.15.28:0/1455179259 conn(0x564d5ee14cb0 msgr2=0x564d5ee168d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:49.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.447+0000 7fc74b192740 1 -- 172.21.15.28:0/1455179259 shutdown_connections 2024-04-21T08:51:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.447+0000 7fc74b192740 1 -- 172.21.15.28:0/1455179259 wait complete. 2024-04-21T08:51:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.447+0000 7fc74b192740 1 -- start start 2024-04-21T08:51:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.447+0000 7fc74b192740 1 Processor -- start 2024-04-21T08:51:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc74b192740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee1a390 0x564d5ee30330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc74b192740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1ad40 0x564d5ee21840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc74b192740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee27e90 0x564d5ee21d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc74b192740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7401ab6c0 con 0x564d5ee27e90 2024-04-21T08:51:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc74b192740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc7401acb30 con 0x564d5ee1a390 2024-04-21T08:51:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc74b192740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc7401abd90 con 0x564d5ee1ad40 2024-04-21T08:51:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc74b192740 10 client.0 _ll_register_callbacks cb 0x7fc73801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc746a5e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee27e90 0x564d5ee21d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc745a5c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1ad40 0x564d5ee21840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc745a5c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1ad40 0x564d5ee21840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:43860/0 (socket says 172.21.15.28:43860) 2024-04-21T08:51:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc746a5e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee27e90 0x564d5ee21d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42038/0 (socket says 172.21.15.28:42038) 2024-04-21T08:51:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc745a5c640 1 -- 172.21.15.28:0/102599914 learned_addr learned my addr 172.21.15.28:0/102599914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:49.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc745a5c640 1 -- 172.21.15.28:0/102599914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee1a390 msgr2=0x564d5ee30330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:51:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc745a5c640 1 --2- 172.21.15.28:0/102599914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee1a390 0x564d5ee30330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc745a5c640 1 -- 172.21.15.28:0/102599914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee27e90 msgr2=0x564d5ee21d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc745a5c640 1 --2- 172.21.15.28:0/102599914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee27e90 0x564d5ee21d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc745a5c640 1 -- 172.21.15.28:0/102599914 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d5ee224c0 con 0x564d5ee1ad40 2024-04-21T08:51:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.448+0000 7fc746a5e640 1 --2- 172.21.15.28:0/102599914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee27e90 0x564d5ee21d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.449+0000 7fc745a5c640 1 --2- 172.21.15.28:0/102599914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1ad40 0x564d5ee21840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x564d5ee264e0 tx=0x7fc730012cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6ebdc90920e7c17 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.449+0000 7fc729ffb640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:51:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.449+0000 7fc729ffb640 1 -- 172.21.15.28:0/102599914 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc730004230 con 0x564d5ee1ad40 2024-04-21T08:51:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.449+0000 7fc729ffb640 1 -- 172.21.15.28:0/102599914 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fc7300043d0 con 0x564d5ee1ad40 2024-04-21T08:51:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.449+0000 7fc74b192740 1 -- 172.21.15.28:0/102599914 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc738003c30 con 0x564d5ee1ad40 2024-04-21T08:51:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.449+0000 7fc729ffb640 1 -- 172.21.15.28:0/102599914 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc730032050 con 0x564d5ee1ad40 2024-04-21T08:51:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.449+0000 7fc74b192740 10 client.25414 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.449+0000 7fc74b192740 1 -- 172.21.15.28:0/102599914 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564d5ee9dec0 con 0x564d5ee1ad40 2024-04-21T08:51:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.450+0000 7fc729ffb640 1 -- 172.21.15.28:0/102599914 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc738003c30 con 0x564d5ee1ad40 2024-04-21T08:51:49.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.450+0000 7fc729ffb640 1 -- 172.21.15.28:0/102599914 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc730003b50 con 0x564d5ee1ad40 2024-04-21T08:51:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.450+0000 7fc74b192740 10 client.25414 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.450+0000 7fc74b192740 10 client.25414 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.450+0000 7fc74b192740 20 client.25414 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.450+0000 7fc74b192740 1 --2- 172.21.15.28:0/102599914 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564d5ee37a40 0x564d5eea0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.450+0000 7fc74b192740 4 client.25414 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:51:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.450+0000 7fc74b192740 1 -- 172.21.15.28:0/102599914 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564d5ed86a70 con 0x564d5ee37a40 2024-04-21T08:51:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.450+0000 7fc746a5e640 1 --2- 172.21.15.28:0/102599914 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564d5ee37a40 0x564d5eea0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.451+0000 7fc729ffb640 10 client.25414 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.451+0000 7fc746a5e640 1 --2- 172.21.15.28:0/102599914 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564d5ee37a40 0x564d5eea0680 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25a645447782bd38 server_cookie=e022f44a2d20336f in_seq=0 out_seq=0 2024-04-21T08:51:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.451+0000 7fc729ffb640 1 -- 172.21.15.28:0/102599914 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564d5ed86a70 con 0x564d5ee37a40 2024-04-21T08:51:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.451+0000 7fc729ffb640 10 client.25414 handle_command_reply: tid=0 2024-04-21T08:51:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.451+0000 7fc74b192740 1 client.25414 shutdown 2024-04-21T08:51:49.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.451+0000 7fc74b192740 1 -- 172.21.15.28:0/102599914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1ad40 msgr2=0x564d5ee21840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:49.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.451+0000 7fc74b192740 1 --2- 172.21.15.28:0/102599914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1ad40 0x564d5ee21840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x564d5ee264e0 tx=0x7fc730012cf0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.451+0000 7fc74b192740 1 -- 172.21.15.28:0/102599914 shutdown_connections 2024-04-21T08:51:49.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.451+0000 7fc74b192740 1 --2- 172.21.15.28:0/102599914 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564d5ee37a40 0x564d5eea0680 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.451+0000 7fc74b192740 1 --2- 172.21.15.28:0/102599914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d5ee27e90 0x564d5ee21d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.451+0000 7fc729ffb640 0 client.25414 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:49.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.451+0000 7fc74b192740 1 --2- 172.21.15.28:0/102599914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564d5ee1ad40 0x564d5ee21840 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.451+0000 7fc74b192740 1 --2- 172.21.15.28:0/102599914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564d5ee1a390 0x564d5ee30330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.451+0000 7fc74b192740 1 -- 172.21.15.28:0/102599914 >> 172.21.15.28:0/102599914 conn(0x564d5ee14cb0 msgr2=0x564d5ee16040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:49.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.452+0000 7fc74b192740 1 -- 172.21.15.28:0/102599914 shutdown_connections 2024-04-21T08:51:49.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.452+0000 7fc74b192740 1 -- 172.21.15.28:0/102599914 wait complete. 2024-04-21T08:51:49.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.452+0000 7fc74b192740 20 client.25414 trim_cache size 0 max 16384 2024-04-21T08:51:49.459 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:51:49.459 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T08:51:49.459 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T08:51:49.459 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:51:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.452+0000 7fc7484c1640 1 -- 172.21.15.28:0/1295549122 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc72c0a3e40 msgr2=0x7fc72c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.452+0000 7fc7484c1640 1 --2- 172.21.15.28:0/1295549122 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc72c0a3e40 0x7fc72c0a6300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc73c0f0120 tx=0x7fc73c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.452+0000 7fc7484c1640 1 -- 172.21.15.28:0/1295549122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc740101080 msgr2=0x7fc740199570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.452+0000 7fc7484c1640 1 --2- 172.21.15.28:0/1295549122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc740101080 0x7fc740199570 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc73c005940 tx=0x7fc73c02c5f0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.452+0000 7fc7484c1640 1 -- 172.21.15.28:0/1295549122 shutdown_connections 2024-04-21T08:51:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.452+0000 7fc7484c1640 1 --2- 172.21.15.28:0/1295549122 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc72c0a3e40 0x7fc72c0a6300 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.453+0000 7fc7484c1640 1 --2- 172.21.15.28:0/1295549122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc74019a090 0x7fc7401a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.453+0000 7fc7484c1640 1 --2- 172.21.15.28:0/1295549122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc740106150 0x7fc740199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.453+0000 7fc7484c1640 1 --2- 172.21.15.28:0/1295549122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc740101080 0x7fc740199570 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.453+0000 7fc7484c1640 1 -- 172.21.15.28:0/1295549122 >> 172.21.15.28:0/1295549122 conn(0x7fc740077f30 msgr2=0x7fc7400ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.453+0000 7fc7484c1640 1 -- 172.21.15.28:0/1295549122 shutdown_connections 2024-04-21T08:51:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.453+0000 7fc7484c1640 1 -- 172.21.15.28:0/1295549122 wait complete. 2024-04-21T08:51:49.463 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:51:49.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:51:49.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.609+0000 7f172f425640 1 -- 172.21.15.28:0/2797885738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1728074250 msgr2=0x7f1728074630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:49.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.609+0000 7f172f425640 1 --2- 172.21.15.28:0/2797885738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1728074250 0x7f1728074630 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1724009900 tx=0x7f172402bcd0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.610+0000 7f172f425640 1 -- 172.21.15.28:0/2797885738 shutdown_connections 2024-04-21T08:51:49.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.610+0000 7f172f425640 1 --2- 172.21.15.28:0/2797885738 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f172810e800 0x7f1728110bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.610+0000 7f172f425640 1 --2- 172.21.15.28:0/2797885738 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1728074b70 0x7f172810e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.610+0000 7f172f425640 1 --2- 172.21.15.28:0/2797885738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1728074250 0x7f1728074630 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.610+0000 7f172f425640 1 -- 172.21.15.28:0/2797885738 >> 172.21.15.28:0/2797885738 conn(0x7f17280793e0 msgr2=0x7f17280797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:49.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.610+0000 7f172f425640 1 -- 172.21.15.28:0/2797885738 shutdown_connections 2024-04-21T08:51:49.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.610+0000 7f172f425640 1 -- 172.21.15.28:0/2797885738 wait complete. 2024-04-21T08:51:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.610+0000 7f172f425640 1 Processor -- start 2024-04-21T08:51:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.610+0000 7f172f425640 1 -- start start 2024-04-21T08:51:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172f425640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1728074250 0x7f17281a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172f425640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1728074b70 0x7f17281a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172f425640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f172810e800 0x7f172819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172f425640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17281143b0 con 0x7f1728074250 2024-04-21T08:51:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172f425640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1728114230 con 0x7f1728074b70 2024-04-21T08:51:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172f425640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1728114530 con 0x7f172810e800 2024-04-21T08:51:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172d1c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1728074250 0x7f17281a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172d1c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1728074250 0x7f17281a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42060/0 (socket says 172.21.15.28:42060) 2024-04-21T08:51:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172d1c1640 1 -- 172.21.15.28:0/412957867 learned_addr learned my addr 172.21.15.28:0/412957867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172c9c0640 1 --2- 172.21.15.28:0/412957867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1728074b70 0x7f17281a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172d9c2640 1 --2- 172.21.15.28:0/412957867 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f172810e800 0x7f172819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172d1c1640 1 -- 172.21.15.28:0/412957867 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f172810e800 msgr2=0x7f172819a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172d1c1640 1 --2- 172.21.15.28:0/412957867 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f172810e800 0x7f172819a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172d1c1640 1 -- 172.21.15.28:0/412957867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1728074b70 msgr2=0x7f17281a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172d1c1640 1 --2- 172.21.15.28:0/412957867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1728074b70 0x7f17281a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.611+0000 7f172d1c1640 1 -- 172.21.15.28:0/412957867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f172819af70 con 0x7f1728074250 2024-04-21T08:51:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.612+0000 7f172d9c2640 1 --2- 172.21.15.28:0/412957867 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f172810e800 0x7f172819a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.612+0000 7f172d1c1640 1 --2- 172.21.15.28:0/412957867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1728074250 0x7f17281a05f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1724005940 tx=0x7f172402c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.612+0000 7f171e7fc640 1 -- 172.21.15.28:0/412957867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724043020 con 0x7f1728074250 2024-04-21T08:51:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.612+0000 7f172f425640 1 -- 172.21.15.28:0/412957867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f172819b200 con 0x7f1728074250 2024-04-21T08:51:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.612+0000 7f171e7fc640 1 -- 172.21.15.28:0/412957867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1724004e00 con 0x7f1728074250 2024-04-21T08:51:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.612+0000 7f172f425640 1 -- 172.21.15.28:0/412957867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17281a7570 con 0x7f1728074250 2024-04-21T08:51:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.612+0000 7f171e7fc640 1 -- 172.21.15.28:0/412957867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1724003970 con 0x7f1728074250 2024-04-21T08:51:49.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.613+0000 7f172f425640 1 -- 172.21.15.28:0/412957867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1728075990 con 0x7f1728074250 2024-04-21T08:51:49.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.615+0000 7f171e7fc640 1 -- 172.21.15.28:0/412957867 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f1724046020 con 0x7f1728074250 2024-04-21T08:51:49.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.616+0000 7f171e7fc640 1 --2- 172.21.15.28:0/412957867 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f17140a3e40 0x7f17140a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:49.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.616+0000 7f171e7fc640 1 -- 172.21.15.28:0/412957867 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f17240f33e0 con 0x7f1728074250 2024-04-21T08:51:49.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.617+0000 7f172c9c0640 1 --2- 172.21.15.28:0/412957867 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f17140a3e40 0x7f17140a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:49.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.617+0000 7f172c9c0640 1 --2- 172.21.15.28:0/412957867 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f17140a3e40 0x7f17140a6300 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f172819bf00 tx=0x7f1718005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:49.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.620+0000 7f171e7fc640 1 -- 172.21.15.28:0/412957867 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f172404d020 con 0x7f1728074250 2024-04-21T08:51:49.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.878+0000 7f172f425640 1 -- 172.21.15.28:0/412957867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1728109980 con 0x7f1728074250 2024-04-21T08:51:49.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.879+0000 7f171e7fc640 1 -- 172.21.15.28:0/412957867 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7f17240bf5a0 con 0x7f1728074250 2024-04-21T08:51:49.879 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:49.879 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:51:09.181803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:51:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-04-21T08:51:49.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.881+0000 7f172f425640 1 -- 172.21.15.28:0/412957867 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f17140a3e40 msgr2=0x7f17140a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:49.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.881+0000 7f172f425640 1 --2- 172.21.15.28:0/412957867 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f17140a3e40 0x7f17140a6300 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f172819bf00 tx=0x7f1718005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:49.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.881+0000 7f172f425640 1 -- 172.21.15.28:0/412957867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1728074250 msgr2=0x7f17281a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:49.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.881+0000 7f172f425640 1 --2- 172.21.15.28:0/412957867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1728074250 0x7f17281a05f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1724005940 tx=0x7f172402c690 comp rx=0 tx=0).stop 2024-04-21T08:51:49.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.882+0000 7f172f425640 1 -- 172.21.15.28:0/412957867 shutdown_connections 2024-04-21T08:51:49.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.882+0000 7f172f425640 1 --2- 172.21.15.28:0/412957867 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f17140a3e40 0x7f17140a6300 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.882+0000 7f172f425640 1 --2- 172.21.15.28:0/412957867 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f172810e800 0x7f172819a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.882+0000 7f172f425640 1 --2- 172.21.15.28:0/412957867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1728074b70 0x7f17281a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.882+0000 7f172f425640 1 --2- 172.21.15.28:0/412957867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1728074250 0x7f17281a05f0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:49.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.882+0000 7f172f425640 1 -- 172.21.15.28:0/412957867 >> 172.21.15.28:0/412957867 conn(0x7f17280793e0 msgr2=0x7f17280fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.882+0000 7f172f425640 1 -- 172.21.15.28:0/412957867 shutdown_connections 2024-04-21T08:51:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:49.882+0000 7f172f425640 1 -- 172.21.15.28:0/412957867 wait complete. 2024-04-21T08:51:49.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:51:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.041+0000 7ff70b146640 1 -- 172.21.15.28:0/2250327490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff704101090 msgr2=0x7ff704101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.041+0000 7ff70b146640 1 --2- 172.21.15.28:0/2250327490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff704101090 0x7ff704101470 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8009900 tx=0x7ff6f802bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.041+0000 7ff70b146640 1 -- 172.21.15.28:0/2250327490 shutdown_connections 2024-04-21T08:51:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.041+0000 7ff70b146640 1 --2- 172.21.15.28:0/2250327490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff704106160 0x7ff70410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.041+0000 7ff70b146640 1 --2- 172.21.15.28:0/2250327490 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff704101a40 0x7ff704105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.041+0000 7ff70b146640 1 --2- 172.21.15.28:0/2250327490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff704101090 0x7ff704101470 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.041+0000 7ff70b146640 1 -- 172.21.15.28:0/2250327490 >> 172.21.15.28:0/2250327490 conn(0x7ff704077f50 msgr2=0x7ff704078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.041+0000 7ff70b146640 1 -- 172.21.15.28:0/2250327490 shutdown_connections 2024-04-21T08:51:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.041+0000 7ff70b146640 1 -- 172.21.15.28:0/2250327490 wait complete. 2024-04-21T08:51:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.041+0000 7ff70b146640 1 Processor -- start 2024-04-21T08:51:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff70b146640 1 -- start start 2024-04-21T08:51:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff70b146640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff704101090 0x7ff704198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff70b146640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff704106160 0x7ff704198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff70b146640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff70419cff0 0x7ff70419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff70b146640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff7041110f0 con 0x7ff704101090 2024-04-21T08:51:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff70b146640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff704110f70 con 0x7ff704106160 2024-04-21T08:51:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff70b146640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff704111270 con 0x7ff70419cff0 2024-04-21T08:51:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff708ee2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff704101090 0x7ff704198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff708ee2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff704101090 0x7ff704198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42074/0 (socket says 172.21.15.28:42074) 2024-04-21T08:51:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff708ee2640 1 -- 172.21.15.28:0/1120254485 learned_addr learned my addr 172.21.15.28:0/1120254485 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff703fff640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff704106160 0x7ff704198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff7096e3640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff70419cff0 0x7ff70419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff708ee2640 1 -- 172.21.15.28:0/1120254485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff70419cff0 msgr2=0x7ff70419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff708ee2640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff70419cff0 0x7ff70419d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff708ee2640 1 -- 172.21.15.28:0/1120254485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff704106160 msgr2=0x7ff704198cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.042+0000 7ff708ee2640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff704106160 0x7ff704198cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.043+0000 7ff708ee2640 1 -- 172.21.15.28:0/1120254485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff70419dae0 con 0x7ff704101090 2024-04-21T08:51:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.043+0000 7ff7096e3640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff70419cff0 0x7ff70419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.043+0000 7ff703fff640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff704106160 0x7ff704198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.043+0000 7ff708ee2640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff704101090 0x7ff704198780 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8000c00 tx=0x7ff6f802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.043+0000 7ff701ffb640 1 -- 172.21.15.28:0/1120254485 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8043020 con 0x7ff704101090 2024-04-21T08:51:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.043+0000 7ff701ffb640 1 -- 172.21.15.28:0/1120254485 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6f802cbe0 con 0x7ff704101090 2024-04-21T08:51:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.043+0000 7ff701ffb640 1 -- 172.21.15.28:0/1120254485 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8003b80 con 0x7ff704101090 2024-04-21T08:51:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.044+0000 7ff70b146640 1 -- 172.21.15.28:0/1120254485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff70419dd70 con 0x7ff704101090 2024-04-21T08:51:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.044+0000 7ff70b146640 1 -- 172.21.15.28:0/1120254485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7041a54e0 con 0x7ff704101090 2024-04-21T08:51:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.047+0000 7ff70de0c740 1 Processor -- start 2024-04-21T08:51:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.047+0000 7ff70de0c740 1 -- start start 2024-04-21T08:51:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.047+0000 7ff70de0c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f994d0 0x5651c6f998b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.047+0000 7ff70de0c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6f99e80 0x5651c6fa2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.047+0000 7ff70de0c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6fa2a00 0x5651c6fa4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.047+0000 7ff70de0c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5651c6efe1f0 con 0x5651c6f994d0 2024-04-21T08:51:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.047+0000 7ff70de0c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5651c6e9b4b0 con 0x5651c6f99e80 2024-04-21T08:51:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.047+0000 7ff70de0c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5651c6efa1b0 con 0x5651c6fa2a00 2024-04-21T08:51:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.047+0000 7ff7096e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f994d0 0x5651c6f998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.047+0000 7ff7096e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f994d0 0x5651c6f998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42082/0 (socket says 172.21.15.28:42082) 2024-04-21T08:51:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.047+0000 7ff7096e3640 1 -- 172.21.15.28:0/3747095685 learned_addr learned my addr 172.21.15.28:0/3747095685 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.048+0000 7ff701ffb640 1 -- 172.21.15.28:0/1120254485 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7ff6f8046020 con 0x7ff704101090 2024-04-21T08:51:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.048+0000 7ff703fff640 1 --2- 172.21.15.28:0/3747095685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6fa2a00 0x5651c6fa4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.048+0000 7ff708ee2640 1 --2- 172.21.15.28:0/3747095685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6f99e80 0x5651c6fa2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.048+0000 7ff7096e3640 1 -- 172.21.15.28:0/3747095685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6fa2a00 msgr2=0x5651c6fa4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.048+0000 7ff7096e3640 1 --2- 172.21.15.28:0/3747095685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6fa2a00 0x5651c6fa4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.048+0000 7ff7096e3640 1 -- 172.21.15.28:0/3747095685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6f99e80 msgr2=0x5651c6fa2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.048+0000 7ff7096e3640 1 --2- 172.21.15.28:0/3747095685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6f99e80 0x5651c6fa2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.048+0000 7ff7096e3640 1 -- 172.21.15.28:0/3747095685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651c6e8d1b0 con 0x5651c6f994d0 2024-04-21T08:51:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.048+0000 7ff703fff640 1 --2- 172.21.15.28:0/3747095685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6fa2a00 0x5651c6fa4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.048+0000 7ff708ee2640 1 --2- 172.21.15.28:0/3747095685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6f99e80 0x5651c6fa2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.048+0000 7ff7096e3640 1 --2- 172.21.15.28:0/3747095685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f994d0 0x5651c6f998b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc010710 tx=0x7ff6fc010bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e92ad0d64f55602 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.048+0000 7ff701ffb640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6f40a3ec0 0x7ff6f40a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.048+0000 7ff6eeffd640 1 -- 172.21.15.28:0/3747095685 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc00e8d0 con 0x5651c6f994d0 2024-04-21T08:51:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.048+0000 7ff6eeffd640 1 -- 172.21.15.28:0/3747095685 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6fc012450 con 0x5651c6f994d0 2024-04-21T08:51:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.048+0000 7ff6eeffd640 1 -- 172.21.15.28:0/3747095685 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc01c670 con 0x5651c6f994d0 2024-04-21T08:51:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.049+0000 7ff701ffb640 1 -- 172.21.15.28:0/1120254485 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ff6f80f2830 con 0x7ff704101090 2024-04-21T08:51:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.049+0000 7ff708ee2640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6f40a3ec0 0x7ff6f40a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.049+0000 7ff70de0c740 1 -- 172.21.15.28:0/3747095685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f994d0 msgr2=0x5651c6f998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.049+0000 7ff70de0c740 1 --2- 172.21.15.28:0/3747095685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f994d0 0x5651c6f998b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc010710 tx=0x7ff6fc010bd0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.049+0000 7ff708ee2640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6f40a3ec0 0x7ff6f40a6380 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80efbd0 tx=0x7ff6f8005e20 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.049+0000 7ff70de0c740 1 -- 172.21.15.28:0/3747095685 shutdown_connections 2024-04-21T08:51:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.049+0000 7ff70de0c740 1 --2- 172.21.15.28:0/3747095685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6fa2a00 0x5651c6fa4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.049+0000 7ff70de0c740 1 --2- 172.21.15.28:0/3747095685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6f99e80 0x5651c6fa2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.049+0000 7ff70de0c740 1 --2- 172.21.15.28:0/3747095685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f994d0 0x5651c6f998b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.049+0000 7ff70de0c740 1 -- 172.21.15.28:0/3747095685 >> 172.21.15.28:0/3747095685 conn(0x5651c6f93ba0 msgr2=0x5651c6f93f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.050+0000 7ff70de0c740 1 -- 172.21.15.28:0/3747095685 shutdown_connections 2024-04-21T08:51:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.050+0000 7ff70de0c740 1 -- 172.21.15.28:0/3747095685 wait complete. 2024-04-21T08:51:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.050+0000 7ff70de0c740 1 -- start start 2024-04-21T08:51:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.050+0000 7ff70de0c740 1 Processor -- start 2024-04-21T08:51:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.050+0000 7ff70de0c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6f994d0 0x5651c6fa2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.050+0000 7ff70de0c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f99e80 0x5651c6f9f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.050+0000 7ff70de0c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6fa2a00 0x5651c6f9fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.050+0000 7ff70de0c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5651c6fa5970 con 0x5651c6f99e80 2024-04-21T08:51:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.050+0000 7ff70de0c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5651c6eb7500 con 0x5651c6fa2a00 2024-04-21T08:51:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.050+0000 7ff70de0c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5651c6fa5af0 con 0x5651c6f994d0 2024-04-21T08:51:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.050+0000 7ff70de0c740 10 client.0 _ll_register_callbacks cb 0x5651c6e85b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff703fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f99e80 0x5651c6f9f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff703fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f99e80 0x5651c6f9f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42098/0 (socket says 172.21.15.28:42098) 2024-04-21T08:51:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff703fff640 1 -- 172.21.15.28:0/676889335 learned_addr learned my addr 172.21.15.28:0/676889335 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff7096e3640 1 --2- 172.21.15.28:0/676889335 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6fa2a00 0x5651c6f9fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff7096e3640 1 --2- 172.21.15.28:0/676889335 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6f994d0 0x5651c6fa2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff703fff640 1 -- 172.21.15.28:0/676889335 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6f994d0 msgr2=0x5651c6fa2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff703fff640 1 --2- 172.21.15.28:0/676889335 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6f994d0 0x5651c6fa2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff703fff640 1 -- 172.21.15.28:0/676889335 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6fa2a00 msgr2=0x5651c6f9fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff703fff640 1 --2- 172.21.15.28:0/676889335 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6fa2a00 0x5651c6f9fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff703fff640 1 -- 172.21.15.28:0/676889335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651c6c103f0 con 0x5651c6f99e80 2024-04-21T08:51:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff7096e3640 1 --2- 172.21.15.28:0/676889335 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6f994d0 0x5651c6fa2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff703fff640 1 --2- 172.21.15.28:0/676889335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f99e80 0x5651c6f9f6e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0012ca0 tx=0x7ff6f00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e693edba5c58ad server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff6e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff6e3fff640 1 -- 172.21.15.28:0/676889335 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0005220 con 0x5651c6f99e80 2024-04-21T08:51:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff6e3fff640 1 -- 172.21.15.28:0/676889335 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ff6f00053c0 con 0x5651c6f99e80 2024-04-21T08:51:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.051+0000 7ff70de0c740 1 -- 172.21.15.28:0/676889335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6fc012450 con 0x5651c6f99e80 2024-04-21T08:51:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.052+0000 7ff6e3fff640 1 -- 172.21.15.28:0/676889335 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff6f0025050 con 0x5651c6f99e80 2024-04-21T08:51:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.052+0000 7ff70de0c740 10 client.15978 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.052+0000 7ff70de0c740 1 -- 172.21.15.28:0/676889335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5651c6cda7d0 con 0x5651c6f99e80 2024-04-21T08:51:50.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.052+0000 7ff6e3fff640 1 -- 172.21.15.28:0/676889335 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6fc012450 con 0x5651c6f99e80 2024-04-21T08:51:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.052+0000 7ff6e3fff640 1 -- 172.21.15.28:0/676889335 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff6f000e860 con 0x5651c6f99e80 2024-04-21T08:51:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.052+0000 7ff70de0c740 10 client.15978 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.052+0000 7ff70de0c740 10 client.15978 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:51:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.052+0000 7ff70de0c740 20 client.15978 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.052+0000 7ff70de0c740 1 --2- 172.21.15.28:0/676889335 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5651c6fb2d10 0x5651c6fb5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.052+0000 7ff70de0c740 4 client.15978 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.052+0000 7ff70de0c740 1 -- 172.21.15.28:0/676889335 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651c6e9d930 con 0x5651c6fb2d10 2024-04-21T08:51:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.052+0000 7ff7096e3640 1 --2- 172.21.15.28:0/676889335 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5651c6fb2d10 0x5651c6fb5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.053+0000 7ff7096e3640 1 --2- 172.21.15.28:0/676889335 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5651c6fb2d10 0x5651c6fb5100 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5f088d7a58dffcfd server_cookie=38389c0cb8387fba in_seq=0 out_seq=0 2024-04-21T08:51:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.053+0000 7ff6e3fff640 10 client.15978 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.054+0000 7ff6e3fff640 1 -- 172.21.15.28:0/676889335 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5651c6e9d930 con 0x5651c6fb2d10 2024-04-21T08:51:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.054+0000 7ff6e3fff640 10 client.15978 handle_command_reply: tid=0 2024-04-21T08:51:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.054+0000 7ff70de0c740 1 client.15978 shutdown 2024-04-21T08:51:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.054+0000 7ff70de0c740 1 -- 172.21.15.28:0/676889335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f99e80 msgr2=0x5651c6f9f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.054+0000 7ff70de0c740 1 --2- 172.21.15.28:0/676889335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f99e80 0x5651c6f9f6e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0012ca0 tx=0x7ff6f00107a0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.054+0000 7ff70de0c740 1 -- 172.21.15.28:0/676889335 shutdown_connections 2024-04-21T08:51:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.054+0000 7ff70de0c740 1 --2- 172.21.15.28:0/676889335 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5651c6fb2d10 0x5651c6fb5100 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.054+0000 7ff70de0c740 1 --2- 172.21.15.28:0/676889335 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6fa2a00 0x5651c6f9fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.054+0000 7ff6e3fff640 0 client.15978 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.054+0000 7ff70de0c740 1 --2- 172.21.15.28:0/676889335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f99e80 0x5651c6f9f6e0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.054+0000 7ff70de0c740 1 --2- 172.21.15.28:0/676889335 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6f994d0 0x5651c6fa2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.054+0000 7ff70de0c740 1 -- 172.21.15.28:0/676889335 >> 172.21.15.28:0/676889335 conn(0x5651c6f93ba0 msgr2=0x5651c6f958f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.054+0000 7ff70de0c740 1 -- 172.21.15.28:0/676889335 shutdown_connections 2024-04-21T08:51:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.054+0000 7ff70de0c740 1 -- 172.21.15.28:0/676889335 wait complete. 2024-04-21T08:51:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.054+0000 7ff70de0c740 20 client.15978 trim_cache size 0 max 16384 2024-04-21T08:51:50.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.067+0000 7ff70de0c740 1 Processor -- start 2024-04-21T08:51:50.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff70de0c740 1 -- start start 2024-04-21T08:51:50.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff70de0c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6f994d0 0x5651c6fae830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff70de0c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f99e80 0x5651c6fa1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff70de0c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6fa2a00 0x5651c6fa2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff70de0c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff7041b1c10 con 0x5651c6f99e80 2024-04-21T08:51:50.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff70de0c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff7041af4e0 con 0x5651c6fa2a00 2024-04-21T08:51:50.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff70de0c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff7041ad330 con 0x5651c6f994d0 2024-04-21T08:51:50.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff703fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f99e80 0x5651c6fa1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff703fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f99e80 0x5651c6fa1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42100/0 (socket says 172.21.15.28:42100) 2024-04-21T08:51:50.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff703fff640 1 -- 172.21.15.28:0/1136931210 learned_addr learned my addr 172.21.15.28:0/1136931210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:50.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff7096e3640 1 --2- 172.21.15.28:0/1136931210 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6f994d0 0x5651c6fae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff7096e3640 1 --2- 172.21.15.28:0/1136931210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6fa2a00 0x5651c6fa2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff703fff640 1 -- 172.21.15.28:0/1136931210 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6f994d0 msgr2=0x5651c6fae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff703fff640 1 --2- 172.21.15.28:0/1136931210 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6f994d0 0x5651c6fae830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff703fff640 1 -- 172.21.15.28:0/1136931210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6fa2a00 msgr2=0x5651c6fa2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff703fff640 1 --2- 172.21.15.28:0/1136931210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6fa2a00 0x5651c6fa2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff703fff640 1 -- 172.21.15.28:0/1136931210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651c6fb5470 con 0x5651c6f99e80 2024-04-21T08:51:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.068+0000 7ff7096e3640 1 --2- 172.21.15.28:0/1136931210 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6f994d0 0x5651c6fae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.069+0000 7ff703fff640 1 --2- 172.21.15.28:0/1136931210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f99e80 0x5651c6fa1e10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0000c00 tx=0x7ff6f0005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fa7dc882c611216 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.069+0000 7ff6ee7fc640 1 -- 172.21.15.28:0/1136931210 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f002d070 con 0x5651c6f99e80 2024-04-21T08:51:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.069+0000 7ff6ee7fc640 1 -- 172.21.15.28:0/1136931210 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6f000e5c0 con 0x5651c6f99e80 2024-04-21T08:51:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.069+0000 7ff6ee7fc640 1 -- 172.21.15.28:0/1136931210 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f001acf0 con 0x5651c6f99e80 2024-04-21T08:51:50.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.069+0000 7ff70de0c740 1 -- 172.21.15.28:0/1136931210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f99e80 msgr2=0x5651c6fa1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.069+0000 7ff70de0c740 1 --2- 172.21.15.28:0/1136931210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f99e80 0x5651c6fa1e10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0000c00 tx=0x7ff6f0005b10 comp rx=0 tx=0).stop 2024-04-21T08:51:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.070+0000 7ff70de0c740 1 -- 172.21.15.28:0/1136931210 shutdown_connections 2024-04-21T08:51:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.070+0000 7ff70de0c740 1 --2- 172.21.15.28:0/1136931210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6fa2a00 0x5651c6fa2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.070+0000 7ff70de0c740 1 --2- 172.21.15.28:0/1136931210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f99e80 0x5651c6fa1e10 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.070+0000 7ff70de0c740 1 --2- 172.21.15.28:0/1136931210 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6f994d0 0x5651c6fae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.070+0000 7ff70de0c740 1 -- 172.21.15.28:0/1136931210 >> 172.21.15.28:0/1136931210 conn(0x5651c6f93ba0 msgr2=0x5651c6f9ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.070+0000 7ff70de0c740 1 -- 172.21.15.28:0/1136931210 shutdown_connections 2024-04-21T08:51:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.070+0000 7ff70de0c740 1 -- 172.21.15.28:0/1136931210 wait complete. 2024-04-21T08:51:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.070+0000 7ff70de0c740 1 -- start start 2024-04-21T08:51:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.070+0000 7ff70de0c740 1 Processor -- start 2024-04-21T08:51:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff70de0c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f994d0 0x5651c6fab010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff70de0c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6f99e80 0x5651c6f9c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff70de0c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6fa2a00 0x5651c6f9c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff70de0c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff7041ad9a0 con 0x5651c6f994d0 2024-04-21T08:51:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff70de0c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff7041aee10 con 0x5651c6f99e80 2024-04-21T08:51:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff70de0c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff7041ae070 con 0x5651c6fa2a00 2024-04-21T08:51:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff7096e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f994d0 0x5651c6fab010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff70de0c740 10 client.0 _ll_register_callbacks cb 0x7ff6f0012830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff7096e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f994d0 0x5651c6fab010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42116/0 (socket says 172.21.15.28:42116) 2024-04-21T08:51:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff7096e3640 1 -- 172.21.15.28:0/1529023502 learned_addr learned my addr 172.21.15.28:0/1529023502 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff703fff640 1 --2- 172.21.15.28:0/1529023502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6f99e80 0x5651c6f9c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff7096e3640 1 -- 172.21.15.28:0/1529023502 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6fa2a00 msgr2=0x5651c6f9c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff7096e3640 1 --2- 172.21.15.28:0/1529023502 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6fa2a00 0x5651c6f9c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff7096e3640 1 -- 172.21.15.28:0/1529023502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6f99e80 msgr2=0x5651c6f9c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff7096e3640 1 --2- 172.21.15.28:0/1529023502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6f99e80 0x5651c6f9c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff7096e3640 1 -- 172.21.15.28:0/1529023502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651c6f9cc90 con 0x5651c6f994d0 2024-04-21T08:51:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff703fff640 1 --2- 172.21.15.28:0/1529023502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6f99e80 0x5651c6f9c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff7096e3640 1 --2- 172.21.15.28:0/1529023502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f994d0 0x5651c6fab010 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc012d50 tx=0x7ff6fc009df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb580dc9b68527c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.071+0000 7ff6e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.072+0000 7ff6e3fff640 1 -- 172.21.15.28:0/1529023502 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc032070 con 0x5651c6f994d0 2024-04-21T08:51:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.072+0000 7ff6e3fff640 1 -- 172.21.15.28:0/1529023502 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ff6fc013e00 con 0x5651c6f994d0 2024-04-21T08:51:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.072+0000 7ff70de0c740 1 -- 172.21.15.28:0/1529023502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6f000e5c0 con 0x5651c6f994d0 2024-04-21T08:51:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.072+0000 7ff6e3fff640 1 -- 172.21.15.28:0/1529023502 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff6fc0124d0 con 0x5651c6f994d0 2024-04-21T08:51:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.072+0000 7ff70de0c740 10 client.15984 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.072+0000 7ff70de0c740 1 -- 172.21.15.28:0/1529023502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5651c70191b0 con 0x5651c6f994d0 2024-04-21T08:51:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.072+0000 7ff6e3fff640 1 -- 172.21.15.28:0/1529023502 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6fc0049a0 con 0x5651c6f994d0 2024-04-21T08:51:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.072+0000 7ff6e3fff640 1 -- 172.21.15.28:0/1529023502 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff6fc02cce0 con 0x5651c6f994d0 2024-04-21T08:51:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.072+0000 7ff70de0c740 10 client.15984 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.072+0000 7ff70de0c740 10 client.15984 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:51:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.072+0000 7ff70de0c740 20 client.15984 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.072+0000 7ff70de0c740 1 --2- 172.21.15.28:0/1529023502 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5651c6fb2d10 0x5651c701b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.073+0000 7ff70de0c740 4 client.15984 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:51:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.073+0000 7ff70de0c740 1 -- 172.21.15.28:0/1529023502 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7ff6f000e5c0 con 0x5651c6fb2d10 2024-04-21T08:51:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.073+0000 7ff703fff640 1 --2- 172.21.15.28:0/1529023502 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5651c6fb2d10 0x5651c701b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.073+0000 7ff6e3fff640 10 client.15984 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.073+0000 7ff703fff640 1 --2- 172.21.15.28:0/1529023502 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5651c6fb2d10 0x5651c701b6f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ede40b4d75b7b30e server_cookie=a952d1be534795cb in_seq=0 out_seq=0 2024-04-21T08:51:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.073+0000 7ff6e3fff640 1 -- 172.21.15.28:0/1529023502 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7ff6f000e5c0 con 0x5651c6fb2d10 2024-04-21T08:51:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.073+0000 7ff6e3fff640 10 client.15984 handle_command_reply: tid=0 2024-04-21T08:51:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.073+0000 7ff70de0c740 1 client.15984 shutdown 2024-04-21T08:51:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.074+0000 7ff70de0c740 1 -- 172.21.15.28:0/1529023502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f994d0 msgr2=0x5651c6fab010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.074+0000 7ff70de0c740 1 --2- 172.21.15.28:0/1529023502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f994d0 0x5651c6fab010 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc012d50 tx=0x7ff6fc009df0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.074+0000 7ff70de0c740 1 -- 172.21.15.28:0/1529023502 shutdown_connections 2024-04-21T08:51:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.074+0000 7ff70de0c740 1 --2- 172.21.15.28:0/1529023502 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5651c6fb2d10 0x5651c701b6f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.074+0000 7ff70de0c740 1 --2- 172.21.15.28:0/1529023502 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651c6fa2a00 0x5651c6f9c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.074+0000 7ff6e3fff640 0 client.15984 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.074+0000 7ff70de0c740 1 --2- 172.21.15.28:0/1529023502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651c6f99e80 0x5651c6f9c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.074+0000 7ff70de0c740 1 --2- 172.21.15.28:0/1529023502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651c6f994d0 0x5651c6fab010 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.074+0000 7ff70de0c740 1 -- 172.21.15.28:0/1529023502 >> 172.21.15.28:0/1529023502 conn(0x5651c6f93ba0 msgr2=0x5651c6f92370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.074+0000 7ff70de0c740 1 -- 172.21.15.28:0/1529023502 shutdown_connections 2024-04-21T08:51:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.074+0000 7ff70de0c740 1 -- 172.21.15.28:0/1529023502 wait complete. 2024-04-21T08:51:50.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.074+0000 7ff70de0c740 20 client.15984 trim_cache size 0 max 16384 2024-04-21T08:51:50.079 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:51:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.075+0000 7ff70b146640 1 -- 172.21.15.28:0/1120254485 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6f40a3ec0 msgr2=0x7ff6f40a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.075+0000 7ff70b146640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6f40a3ec0 0x7ff6f40a6380 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80efbd0 tx=0x7ff6f8005e20 comp rx=0 tx=0).stop 2024-04-21T08:51:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.075+0000 7ff70b146640 1 -- 172.21.15.28:0/1120254485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff704101090 msgr2=0x7ff704198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.075+0000 7ff70b146640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff704101090 0x7ff704198780 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8000c00 tx=0x7ff6f802c7e0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.075+0000 7ff70b146640 1 -- 172.21.15.28:0/1120254485 shutdown_connections 2024-04-21T08:51:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.075+0000 7ff70b146640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6f40a3ec0 0x7ff6f40a6380 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.075+0000 7ff70b146640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff70419cff0 0x7ff70419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.075+0000 7ff70b146640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff704106160 0x7ff704198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.075+0000 7ff70b146640 1 --2- 172.21.15.28:0/1120254485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff704101090 0x7ff704198780 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.075+0000 7ff70b146640 1 -- 172.21.15.28:0/1120254485 >> 172.21.15.28:0/1120254485 conn(0x7ff704077f50 msgr2=0x7ff7040694c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.075+0000 7ff70b146640 1 -- 172.21.15.28:0/1120254485 shutdown_connections 2024-04-21T08:51:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.075+0000 7ff70b146640 1 -- 172.21.15.28:0/1120254485 wait complete. 2024-04-21T08:51:50.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:51:50.172 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:49 smithi171 ceph-mon[31358]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-21T08:51:50.172 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:49 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/412957867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.197+0000 7fe21eda4640 1 -- 172.21.15.28:0/3236933628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe218101090 msgr2=0x7fe218101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.197+0000 7fe21eda4640 1 --2- 172.21.15.28:0/3236933628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe218101090 0x7fe218101470 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009900 tx=0x7fe20c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.197+0000 7fe21eda4640 1 -- 172.21.15.28:0/3236933628 shutdown_connections 2024-04-21T08:51:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.197+0000 7fe21eda4640 1 --2- 172.21.15.28:0/3236933628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe218106160 0x7fe21810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.197+0000 7fe21eda4640 1 --2- 172.21.15.28:0/3236933628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe218101a40 0x7fe218105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.197+0000 7fe21eda4640 1 --2- 172.21.15.28:0/3236933628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe218101090 0x7fe218101470 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.197+0000 7fe21eda4640 1 -- 172.21.15.28:0/3236933628 >> 172.21.15.28:0/3236933628 conn(0x7fe218077f50 msgr2=0x7fe218078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.197+0000 7fe21eda4640 1 -- 172.21.15.28:0/3236933628 shutdown_connections 2024-04-21T08:51:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.197+0000 7fe21eda4640 1 -- 172.21.15.28:0/3236933628 wait complete. 2024-04-21T08:51:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.198+0000 7fe21eda4640 1 Processor -- start 2024-04-21T08:51:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.198+0000 7fe21eda4640 1 -- start start 2024-04-21T08:51:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.198+0000 7fe21eda4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe218106160 0x7fe2181997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.198+0000 7fe21eda4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe218199ce0 0x7fe21819a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.198+0000 7fe21eda4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2181a2810 0x7fe2181a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.198+0000 7fe21eda4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe218111110 con 0x7fe2181a2810 2024-04-21T08:51:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.198+0000 7fe21eda4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe218110f90 con 0x7fe218106160 2024-04-21T08:51:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.198+0000 7fe21eda4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe218111290 con 0x7fe218199ce0 2024-04-21T08:51:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.198+0000 7fe21d341640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2181a2810 0x7fe2181a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.199+0000 7fe21d341640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2181a2810 0x7fe2181a4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42120/0 (socket says 172.21.15.28:42120) 2024-04-21T08:51:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.199+0000 7fe21d341640 1 -- 172.21.15.28:0/757685076 learned_addr learned my addr 172.21.15.28:0/757685076 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.199+0000 7fe21cb40640 1 --2- 172.21.15.28:0/757685076 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe218106160 0x7fe2181997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.199+0000 7fe217fff640 1 --2- 172.21.15.28:0/757685076 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe218199ce0 0x7fe21819a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.199+0000 7fe21d341640 1 -- 172.21.15.28:0/757685076 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe218199ce0 msgr2=0x7fe21819a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.199+0000 7fe21d341640 1 --2- 172.21.15.28:0/757685076 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe218199ce0 0x7fe21819a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.199+0000 7fe21d341640 1 -- 172.21.15.28:0/757685076 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe218106160 msgr2=0x7fe2181997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.199+0000 7fe21d341640 1 --2- 172.21.15.28:0/757685076 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe218106160 0x7fe2181997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.199+0000 7fe21d341640 1 -- 172.21.15.28:0/757685076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2181a5420 con 0x7fe2181a2810 2024-04-21T08:51:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.199+0000 7fe21cb40640 1 --2- 172.21.15.28:0/757685076 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe218106160 0x7fe2181997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.199+0000 7fe21d341640 1 --2- 172.21.15.28:0/757685076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2181a2810 0x7fe2181a4c00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe21000efd0 tx=0x7fe21000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.199+0000 7fe215ffb640 1 -- 172.21.15.28:0/757685076 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe210003b70 con 0x7fe2181a2810 2024-04-21T08:51:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.199+0000 7fe215ffb640 1 -- 172.21.15.28:0/757685076 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe210003d10 con 0x7fe2181a2810 2024-04-21T08:51:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.199+0000 7fe215ffb640 1 -- 172.21.15.28:0/757685076 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2100108a0 con 0x7fe2181a2810 2024-04-21T08:51:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.200+0000 7fe21eda4640 1 -- 172.21.15.28:0/757685076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2181a5710 con 0x7fe2181a2810 2024-04-21T08:51:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.200+0000 7fe21eda4640 1 -- 172.21.15.28:0/757685076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2181a5c60 con 0x7fe2181a2810 2024-04-21T08:51:50.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.202+0000 7fe215ffb640 1 -- 172.21.15.28:0/757685076 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fe210004020 con 0x7fe2181a2810 2024-04-21T08:51:50.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.203+0000 7fe221990740 1 Processor -- start 2024-04-21T08:51:50.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.203+0000 7fe221990740 1 -- start start 2024-04-21T08:51:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.203+0000 7fe215ffb640 1 --2- 172.21.15.28:0/757685076 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe2080a3ec0 0x7fe2080a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.203+0000 7fe221990740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18250 0x55f1f5e18630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.203+0000 7fe221990740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e18c00 0x55f1f5e2a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.203+0000 7fe221990740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e2a570 0x55f1f5e2c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.203+0000 7fe221990740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f1f5d7d1f0 con 0x55f1f5e2a570 2024-04-21T08:51:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.203+0000 7fe221990740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f1f5d1a4b0 con 0x55f1f5e18250 2024-04-21T08:51:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.203+0000 7fe221990740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f1f5d791b0 con 0x55f1f5e18c00 2024-04-21T08:51:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.203+0000 7fe217fff640 1 --2- 172.21.15.28:0/757685076 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe2080a3ec0 0x7fe2080a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.203+0000 7fe21d341640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e2a570 0x55f1f5e2c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.203+0000 7fe21d341640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e2a570 0x55f1f5e2c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42126/0 (socket says 172.21.15.28:42126) 2024-04-21T08:51:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.203+0000 7fe21d341640 1 -- 172.21.15.28:0/4159878510 learned_addr learned my addr 172.21.15.28:0/4159878510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.204+0000 7fe215ffb640 1 -- 172.21.15.28:0/757685076 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe2100d11b0 con 0x7fe2181a2810 2024-04-21T08:51:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.204+0000 7fe21cb40640 1 --2- 172.21.15.28:0/4159878510 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18250 0x55f1f5e18630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.204+0000 7fe21d341640 1 -- 172.21.15.28:0/4159878510 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e18c00 msgr2=0x55f1f5e2a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.204+0000 7fe21d341640 1 --2- 172.21.15.28:0/4159878510 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e18c00 0x55f1f5e2a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.204+0000 7fe21d341640 1 -- 172.21.15.28:0/4159878510 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18250 msgr2=0x55f1f5e18630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.204+0000 7fe21d341640 1 --2- 172.21.15.28:0/4159878510 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18250 0x55f1f5e18630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.204+0000 7fe21d341640 1 -- 172.21.15.28:0/4159878510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1f5d0c1b0 con 0x55f1f5e2a570 2024-04-21T08:51:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.204+0000 7fe21cb40640 1 --2- 172.21.15.28:0/4159878510 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18250 0x55f1f5e18630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.204+0000 7fe217fff640 1 --2- 172.21.15.28:0/757685076 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe2080a3ec0 0x7fe2080a6380 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe21806b6e0 tx=0x7fe204011440 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.204+0000 7fe21d341640 1 --2- 172.21.15.28:0/4159878510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e2a570 0x55f1f5e2c960 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe21001b000 tx=0x7fe21009c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=852eb65c66d41ebf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.204+0000 7fe202ffd640 1 -- 172.21.15.28:0/4159878510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21009d260 con 0x55f1f5e2a570 2024-04-21T08:51:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.205+0000 7fe202ffd640 1 -- 172.21.15.28:0/4159878510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe210070a10 con 0x55f1f5e2a570 2024-04-21T08:51:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.205+0000 7fe202ffd640 1 -- 172.21.15.28:0/4159878510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe210004560 con 0x55f1f5e2a570 2024-04-21T08:51:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.205+0000 7fe221990740 1 -- 172.21.15.28:0/4159878510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e2a570 msgr2=0x55f1f5e2c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.205+0000 7fe221990740 1 --2- 172.21.15.28:0/4159878510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e2a570 0x55f1f5e2c960 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe21001b000 tx=0x7fe21009c4c0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.205+0000 7fe221990740 1 -- 172.21.15.28:0/4159878510 shutdown_connections 2024-04-21T08:51:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.205+0000 7fe221990740 1 --2- 172.21.15.28:0/4159878510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e2a570 0x55f1f5e2c960 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.205+0000 7fe221990740 1 --2- 172.21.15.28:0/4159878510 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e18c00 0x55f1f5e2a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.205+0000 7fe221990740 1 --2- 172.21.15.28:0/4159878510 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18250 0x55f1f5e18630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.205+0000 7fe221990740 1 -- 172.21.15.28:0/4159878510 >> 172.21.15.28:0/4159878510 conn(0x55f1f5e12ba0 msgr2=0x55f1f5e12f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.206+0000 7fe221990740 1 -- 172.21.15.28:0/4159878510 shutdown_connections 2024-04-21T08:51:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.206+0000 7fe221990740 1 -- 172.21.15.28:0/4159878510 wait complete. 2024-04-21T08:51:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.206+0000 7fe221990740 1 -- start start 2024-04-21T08:51:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.206+0000 7fe221990740 1 Processor -- start 2024-04-21T08:51:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe221990740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18250 0x55f1f5e211f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe221990740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e18c00 0x55f1f5e1e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe221990740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e2a570 0x55f1f5e1eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe221990740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f1f5e2d0e0 con 0x55f1f5e2a570 2024-04-21T08:51:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe221990740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f1f5d36500 con 0x55f1f5e18250 2024-04-21T08:51:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe221990740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f1f5e2d260 con 0x55f1f5e18c00 2024-04-21T08:51:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe217fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e18c00 0x55f1f5e1e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe21d341640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e2a570 0x55f1f5e1eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe21cb40640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18250 0x55f1f5e211f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe217fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e18c00 0x55f1f5e1e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:43980/0 (socket says 172.21.15.28:43980) 2024-04-21T08:51:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe21d341640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e2a570 0x55f1f5e1eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42132/0 (socket says 172.21.15.28:42132) 2024-04-21T08:51:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe217fff640 1 -- 172.21.15.28:0/1136203979 learned_addr learned my addr 172.21.15.28:0/1136203979 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe221990740 10 client.0 _ll_register_callbacks cb 0x55f1f5d04b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe21d341640 1 -- 172.21.15.28:0/1136203979 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e18c00 msgr2=0x55f1f5e1e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe21d341640 1 --2- 172.21.15.28:0/1136203979 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e18c00 0x55f1f5e1e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe21d341640 1 -- 172.21.15.28:0/1136203979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18250 msgr2=0x55f1f5e211f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe21d341640 1 --2- 172.21.15.28:0/1136203979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18250 0x55f1f5e211f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe21d341640 1 -- 172.21.15.28:0/1136203979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1f5a8f3f0 con 0x55f1f5e2a570 2024-04-21T08:51:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe21cb40640 1 --2- 172.21.15.28:0/1136203979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18250 0x55f1f5e211f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe221990740 1 -- 172.21.15.28:0/1136203979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe210070a10 con 0x55f1f5e2a570 2024-04-21T08:51:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe21d341640 1 --2- 172.21.15.28:0/1136203979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e2a570 0x55f1f5e1eb90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe210099530 tx=0x7fe210019fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9980bd074258c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.207+0000 7fe1f7fff640 10 client.15996 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.208+0000 7fe1f7fff640 1 -- 172.21.15.28:0/1136203979 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2100b6060 con 0x55f1f5e2a570 2024-04-21T08:51:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.208+0000 7fe1f7fff640 1 -- 172.21.15.28:0/1136203979 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe2100ce6c0 con 0x55f1f5e2a570 2024-04-21T08:51:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.208+0000 7fe1f7fff640 1 -- 172.21.15.28:0/1136203979 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe210074070 con 0x55f1f5e2a570 2024-04-21T08:51:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.208+0000 7fe221990740 10 client.15996 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.208+0000 7fe221990740 1 -- 172.21.15.28:0/1136203979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=25}) -- 0x55f1f5b597d0 con 0x55f1f5e2a570 2024-04-21T08:51:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.208+0000 7fe1f7fff640 1 -- 172.21.15.28:0/1136203979 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe21001f020 con 0x55f1f5e2a570 2024-04-21T08:51:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.208+0000 7fe221990740 10 client.15996 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.208+0000 7fe221990740 10 client.15996 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:51:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.208+0000 7fe221990740 20 client.15996 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.208+0000 7fe221990740 1 --2- 172.21.15.28:0/1136203979 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55f1f5e3a110 0x55f1f5e3c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.208+0000 7fe221990740 4 client.15996 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.208+0000 7fe221990740 1 -- 172.21.15.28:0/1136203979 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1f5d1c930 con 0x55f1f5e3a110 2024-04-21T08:51:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.208+0000 7fe21cb40640 1 --2- 172.21.15.28:0/1136203979 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55f1f5e3a110 0x55f1f5e3c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.209+0000 7fe21cb40640 1 --2- 172.21.15.28:0/1136203979 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55f1f5e3a110 0x55f1f5e3c500 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eb9badf0c5061cd4 server_cookie=317aa1570e02de33 in_seq=0 out_seq=0 2024-04-21T08:51:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.209+0000 7fe1f7fff640 10 client.15996 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.210+0000 7fe1f7fff640 1 -- 172.21.15.28:0/1136203979 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f1f5d1c930 con 0x55f1f5e3a110 2024-04-21T08:51:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.210+0000 7fe1f7fff640 10 client.15996 handle_command_reply: tid=0 2024-04-21T08:51:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.210+0000 7fe221990740 1 client.15996 shutdown 2024-04-21T08:51:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.210+0000 7fe221990740 1 -- 172.21.15.28:0/1136203979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e2a570 msgr2=0x55f1f5e1eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.210+0000 7fe221990740 1 --2- 172.21.15.28:0/1136203979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e2a570 0x55f1f5e1eb90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe210099530 tx=0x7fe210019fe0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.210+0000 7fe221990740 1 -- 172.21.15.28:0/1136203979 shutdown_connections 2024-04-21T08:51:50.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.210+0000 7fe221990740 1 --2- 172.21.15.28:0/1136203979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e2a570 0x55f1f5e1eb90 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.210+0000 7fe221990740 1 --2- 172.21.15.28:0/1136203979 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55f1f5e3a110 0x55f1f5e3c500 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.210+0000 7fe221990740 1 --2- 172.21.15.28:0/1136203979 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e18c00 0x55f1f5e1e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.210+0000 7fe1f7fff640 0 client.15996 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.210+0000 7fe221990740 1 --2- 172.21.15.28:0/1136203979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18250 0x55f1f5e211f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.210+0000 7fe221990740 1 -- 172.21.15.28:0/1136203979 >> 172.21.15.28:0/1136203979 conn(0x55f1f5e12ba0 msgr2=0x55f1f5e148f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.210+0000 7fe221990740 1 -- 172.21.15.28:0/1136203979 shutdown_connections 2024-04-21T08:51:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.210+0000 7fe221990740 1 -- 172.21.15.28:0/1136203979 wait complete. 2024-04-21T08:51:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.210+0000 7fe221990740 20 client.15996 trim_cache size 0 max 16384 2024-04-21T08:51:50.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe221990740 1 Processor -- start 2024-04-21T08:51:50.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe221990740 1 -- start start 2024-04-21T08:51:50.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe221990740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e18250 0x55f1f5e20cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe221990740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18c00 0x55f1f5e211f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe221990740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e2a570 0x55f1f5e1b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe21cb40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e18250 0x55f1f5e20cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe221990740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2181b0020 con 0x55f1f5e18250 2024-04-21T08:51:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe21cb40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e18250 0x55f1f5e20cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42136/0 (socket says 172.21.15.28:42136) 2024-04-21T08:51:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe221990740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe2181ad8f0 con 0x55f1f5e18c00 2024-04-21T08:51:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe21cb40640 1 -- 172.21.15.28:0/2930795073 learned_addr learned my addr 172.21.15.28:0/2930795073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe221990740 1 -- 172.21.15.28:0/2930795073 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe2181ab740 con 0x55f1f5e2a570 2024-04-21T08:51:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe217fff640 1 --2- 172.21.15.28:0/2930795073 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18c00 0x55f1f5e211f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe21cb40640 1 -- 172.21.15.28:0/2930795073 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e2a570 msgr2=0x55f1f5e1b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe21cb40640 1 --2- 172.21.15.28:0/2930795073 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e2a570 0x55f1f5e1b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe21cb40640 1 -- 172.21.15.28:0/2930795073 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18c00 msgr2=0x55f1f5e211f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe21cb40640 1 --2- 172.21.15.28:0/2930795073 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18c00 0x55f1f5e211f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.224+0000 7fe21cb40640 1 -- 172.21.15.28:0/2930795073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1f5e2a3a0 con 0x55f1f5e18250 2024-04-21T08:51:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.225+0000 7fe21cb40640 1 --2- 172.21.15.28:0/2930795073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e18250 0x55f1f5e20cb0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe20c02c240 tx=0x7fe20c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e946fdb4c281628 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.225+0000 7fe2027fc640 1 -- 172.21.15.28:0/2930795073 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20c050070 con 0x55f1f5e18250 2024-04-21T08:51:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.225+0000 7fe2027fc640 1 -- 172.21.15.28:0/2930795073 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe20c02cc90 con 0x55f1f5e18250 2024-04-21T08:51:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.225+0000 7fe2027fc640 1 -- 172.21.15.28:0/2930795073 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20c000e60 con 0x55f1f5e18250 2024-04-21T08:51:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.226+0000 7fe221990740 1 -- 172.21.15.28:0/2930795073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e18250 msgr2=0x55f1f5e20cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.226+0000 7fe221990740 1 --2- 172.21.15.28:0/2930795073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e18250 0x55f1f5e20cb0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe20c02c240 tx=0x7fe20c005a20 comp rx=0 tx=0).stop 2024-04-21T08:51:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.226+0000 7fe221990740 1 -- 172.21.15.28:0/2930795073 shutdown_connections 2024-04-21T08:51:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.226+0000 7fe221990740 1 --2- 172.21.15.28:0/2930795073 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e2a570 0x55f1f5e1b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.226+0000 7fe221990740 1 --2- 172.21.15.28:0/2930795073 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e18c00 0x55f1f5e211f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.226+0000 7fe221990740 1 --2- 172.21.15.28:0/2930795073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e18250 0x55f1f5e20cb0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.226+0000 7fe221990740 1 -- 172.21.15.28:0/2930795073 >> 172.21.15.28:0/2930795073 conn(0x55f1f5e12ba0 msgr2=0x55f1f5e12f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.226+0000 7fe221990740 1 -- 172.21.15.28:0/2930795073 shutdown_connections 2024-04-21T08:51:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.226+0000 7fe221990740 1 -- 172.21.15.28:0/2930795073 wait complete. 2024-04-21T08:51:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.226+0000 7fe221990740 1 -- start start 2024-04-21T08:51:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.227+0000 7fe221990740 1 Processor -- start 2024-04-21T08:51:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.227+0000 7fe221990740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e18250 0x55f1f5e3f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.227+0000 7fe221990740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e18c00 0x55f1f5e3f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.227+0000 7fe221990740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e2a570 0x55f1f5e3c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.227+0000 7fe221990740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2181abdb0 con 0x55f1f5e18250 2024-04-21T08:51:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.227+0000 7fe221990740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe2181ad220 con 0x55f1f5e2a570 2024-04-21T08:51:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.227+0000 7fe221990740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe2181ac480 con 0x55f1f5e18c00 2024-04-21T08:51:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.227+0000 7fe221990740 10 client.0 _ll_register_callbacks cb 0x7fe2100ce350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.227+0000 7fe21cb40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e18250 0x55f1f5e3f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.227+0000 7fe21cb40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e18250 0x55f1f5e3f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42146/0 (socket says 172.21.15.28:42146) 2024-04-21T08:51:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.227+0000 7fe21cb40640 1 -- 172.21.15.28:0/3160877460 learned_addr learned my addr 172.21.15.28:0/3160877460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.227+0000 7fe217fff640 1 --2- 172.21.15.28:0/3160877460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e18c00 0x55f1f5e3f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.227+0000 7fe21d341640 1 --2- 172.21.15.28:0/3160877460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e2a570 0x55f1f5e3c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.228+0000 7fe21cb40640 1 -- 172.21.15.28:0/3160877460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e18c00 msgr2=0x55f1f5e3f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.228+0000 7fe21cb40640 1 --2- 172.21.15.28:0/3160877460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e18c00 0x55f1f5e3f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.228+0000 7fe21cb40640 1 -- 172.21.15.28:0/3160877460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e2a570 msgr2=0x55f1f5e3c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.228+0000 7fe21cb40640 1 --2- 172.21.15.28:0/3160877460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e2a570 0x55f1f5e3c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.228+0000 7fe21cb40640 1 -- 172.21.15.28:0/3160877460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1f5e3cf00 con 0x55f1f5e18250 2024-04-21T08:51:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.228+0000 7fe21cb40640 1 --2- 172.21.15.28:0/3160877460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e18250 0x55f1f5e3f2e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009690 tx=0x7fe20c009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7758535a32312a53 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.228+0000 7fe1f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.228+0000 7fe1f7fff640 1 -- 172.21.15.28:0/3160877460 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20c05c030 con 0x55f1f5e18250 2024-04-21T08:51:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.228+0000 7fe1f7fff640 1 -- 172.21.15.28:0/3160877460 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe20c04c440 con 0x55f1f5e18250 2024-04-21T08:51:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.228+0000 7fe221990740 1 -- 172.21.15.28:0/3160877460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe20c02cc90 con 0x55f1f5e18250 2024-04-21T08:51:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.228+0000 7fe1f7fff640 1 -- 172.21.15.28:0/3160877460 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe20c066050 con 0x55f1f5e18250 2024-04-21T08:51:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.228+0000 7fe221990740 10 client.16002 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.228+0000 7fe221990740 1 -- 172.21.15.28:0/3160877460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f1f5e9fdf0 con 0x55f1f5e18250 2024-04-21T08:51:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.228+0000 7fe1f7fff640 1 -- 172.21.15.28:0/3160877460 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe20c02cc90 con 0x55f1f5e18250 2024-04-21T08:51:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.229+0000 7fe1f7fff640 1 -- 172.21.15.28:0/3160877460 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe20c055450 con 0x55f1f5e18250 2024-04-21T08:51:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.229+0000 7fe221990740 10 client.16002 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.229+0000 7fe221990740 10 client.16002 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:51:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.229+0000 7fe221990740 20 client.16002 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.229+0000 7fe221990740 1 --2- 172.21.15.28:0/3160877460 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55f1f5e3a110 0x55f1f5ea24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.229+0000 7fe221990740 4 client.16002 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:51:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.229+0000 7fe221990740 1 -- 172.21.15.28:0/3160877460 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f1f5d8ac10 con 0x55f1f5e3a110 2024-04-21T08:51:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.229+0000 7fe217fff640 1 --2- 172.21.15.28:0/3160877460 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55f1f5e3a110 0x55f1f5ea24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.229+0000 7fe1f7fff640 10 client.16002 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.229+0000 7fe217fff640 1 --2- 172.21.15.28:0/3160877460 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55f1f5e3a110 0x55f1f5ea24e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=57b1aba044301e72 server_cookie=10a4dcbd2c9cd85b in_seq=0 out_seq=0 2024-04-21T08:51:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.230+0000 7fe1f7fff640 1 -- 172.21.15.28:0/3160877460 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f1f5d8ac10 con 0x55f1f5e3a110 2024-04-21T08:51:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.230+0000 7fe1f7fff640 10 client.16002 handle_command_reply: tid=0 2024-04-21T08:51:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.230+0000 7fe221990740 1 client.16002 shutdown 2024-04-21T08:51:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.230+0000 7fe221990740 1 -- 172.21.15.28:0/3160877460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e18250 msgr2=0x55f1f5e3f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.230+0000 7fe221990740 1 --2- 172.21.15.28:0/3160877460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e18250 0x55f1f5e3f2e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009690 tx=0x7fe20c009230 comp rx=0 tx=0).stop 2024-04-21T08:51:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.230+0000 7fe221990740 1 -- 172.21.15.28:0/3160877460 shutdown_connections 2024-04-21T08:51:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.230+0000 7fe221990740 1 --2- 172.21.15.28:0/3160877460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1f5e2a570 0x55f1f5e3c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.230+0000 7fe221990740 1 --2- 172.21.15.28:0/3160877460 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55f1f5e3a110 0x55f1f5ea24e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.230+0000 7fe1f7fff640 0 client.16002 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.230+0000 7fe221990740 1 --2- 172.21.15.28:0/3160877460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1f5e18c00 0x55f1f5e3f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.230+0000 7fe221990740 1 --2- 172.21.15.28:0/3160877460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1f5e18250 0x55f1f5e3f2e0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.230+0000 7fe221990740 1 -- 172.21.15.28:0/3160877460 >> 172.21.15.28:0/3160877460 conn(0x55f1f5e12ba0 msgr2=0x55f1f5e354b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.230+0000 7fe221990740 1 -- 172.21.15.28:0/3160877460 shutdown_connections 2024-04-21T08:51:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.230+0000 7fe221990740 1 -- 172.21.15.28:0/3160877460 wait complete. 2024-04-21T08:51:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.230+0000 7fe221990740 20 client.16002 trim_cache size 0 max 16384 2024-04-21T08:51:50.234 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:51:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.231+0000 7fe21eda4640 1 -- 172.21.15.28:0/757685076 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe2080a3ec0 msgr2=0x7fe2080a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.231+0000 7fe21eda4640 1 --2- 172.21.15.28:0/757685076 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe2080a3ec0 0x7fe2080a6380 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe21806b6e0 tx=0x7fe204011440 comp rx=0 tx=0).stop 2024-04-21T08:51:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.231+0000 7fe21eda4640 1 -- 172.21.15.28:0/757685076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2181a2810 msgr2=0x7fe2181a4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.231+0000 7fe21eda4640 1 --2- 172.21.15.28:0/757685076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2181a2810 0x7fe2181a4c00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe21000efd0 tx=0x7fe21000c6a0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.231+0000 7fe21eda4640 1 -- 172.21.15.28:0/757685076 shutdown_connections 2024-04-21T08:51:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.231+0000 7fe21eda4640 1 --2- 172.21.15.28:0/757685076 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe2080a3ec0 0x7fe2080a6380 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.231+0000 7fe21eda4640 1 --2- 172.21.15.28:0/757685076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2181a2810 0x7fe2181a4c00 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.231+0000 7fe21eda4640 1 --2- 172.21.15.28:0/757685076 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe218199ce0 0x7fe21819a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.231+0000 7fe21eda4640 1 --2- 172.21.15.28:0/757685076 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe218106160 0x7fe2181997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.231+0000 7fe21eda4640 1 -- 172.21.15.28:0/757685076 >> 172.21.15.28:0/757685076 conn(0x7fe218077f50 msgr2=0x7fe2180ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.231+0000 7fe21eda4640 1 -- 172.21.15.28:0/757685076 shutdown_connections 2024-04-21T08:51:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.231+0000 7fe21eda4640 1 -- 172.21.15.28:0/757685076 wait complete. 2024-04-21T08:51:50.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:49 smithi028 ceph-mon[28003]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-21T08:51:50.242 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:49 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/412957867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:50.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:51:50.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.351+0000 7f745750d640 1 -- 172.21.15.28:0/2553681620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7450102190 msgr2=0x7f7450102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.351+0000 7f745750d640 1 --2- 172.21.15.28:0/2553681620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7450102190 0x7f7450102570 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f744c009900 tx=0x7f744c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:50.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.351+0000 7f745750d640 1 -- 172.21.15.28:0/2553681620 shutdown_connections 2024-04-21T08:51:50.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.351+0000 7f745750d640 1 --2- 172.21.15.28:0/2553681620 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7450107260 0x7f745010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.351+0000 7f745750d640 1 --2- 172.21.15.28:0/2553681620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7450102b40 0x7f7450106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.351+0000 7f745750d640 1 --2- 172.21.15.28:0/2553681620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7450102190 0x7f7450102570 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.351+0000 7f745750d640 1 -- 172.21.15.28:0/2553681620 >> 172.21.15.28:0/2553681620 conn(0x7f74500793a0 msgr2=0x7f74500797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.351+0000 7f745750d640 1 -- 172.21.15.28:0/2553681620 shutdown_connections 2024-04-21T08:51:50.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.351+0000 7f745750d640 1 -- 172.21.15.28:0/2553681620 wait complete. 2024-04-21T08:51:50.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.352+0000 7f745750d640 1 Processor -- start 2024-04-21T08:51:50.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.352+0000 7f745750d640 1 -- start start 2024-04-21T08:51:50.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.352+0000 7f745750d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7450102190 0x7f745019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.352+0000 7f745750d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7450107260 0x7f745019e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.352+0000 7f745750d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74501985c0 0x7f7450198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.352+0000 7f745750d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7450112240 con 0x7f7450102190 2024-04-21T08:51:50.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.352+0000 7f745750d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f74501120c0 con 0x7f7450107260 2024-04-21T08:51:50.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.352+0000 7f745750d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f74501123c0 con 0x7f74501985c0 2024-04-21T08:51:50.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.352+0000 7f74552a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7450102190 0x7f745019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.352+0000 7f74552a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7450102190 0x7f745019e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42164/0 (socket says 172.21.15.28:42164) 2024-04-21T08:51:50.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.352+0000 7f74552a9640 1 -- 172.21.15.28:0/1829867454 learned_addr learned my addr 172.21.15.28:0/1829867454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:50.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.352+0000 7f7455aaa640 1 --2- 172.21.15.28:0/1829867454 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74501985c0 0x7f7450198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.352+0000 7f7454aa8640 1 --2- 172.21.15.28:0/1829867454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7450107260 0x7f745019e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.353+0000 7f74552a9640 1 -- 172.21.15.28:0/1829867454 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74501985c0 msgr2=0x7f7450198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.353+0000 7f74552a9640 1 --2- 172.21.15.28:0/1829867454 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74501985c0 0x7f7450198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.353+0000 7f74552a9640 1 -- 172.21.15.28:0/1829867454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7450107260 msgr2=0x7f745019e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.353+0000 7f74552a9640 1 --2- 172.21.15.28:0/1829867454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7450107260 0x7f745019e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.353+0000 7f74552a9640 1 -- 172.21.15.28:0/1829867454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74501990e0 con 0x7f7450102190 2024-04-21T08:51:50.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.353+0000 7f7454aa8640 1 --2- 172.21.15.28:0/1829867454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7450107260 0x7f745019e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:50.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.353+0000 7f74552a9640 1 --2- 172.21.15.28:0/1829867454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7450102190 0x7f745019e410 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f744c000c00 tx=0x7f744c02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.353+0000 7f74467fc640 1 -- 172.21.15.28:0/1829867454 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c043020 con 0x7f7450102190 2024-04-21T08:51:50.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.353+0000 7f745750d640 1 -- 172.21.15.28:0/1829867454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7450199370 con 0x7f7450102190 2024-04-21T08:51:50.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.353+0000 7f74467fc640 1 -- 172.21.15.28:0/1829867454 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f744c0052b0 con 0x7f7450102190 2024-04-21T08:51:50.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.353+0000 7f74467fc640 1 -- 172.21.15.28:0/1829867454 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c003c60 con 0x7f7450102190 2024-04-21T08:51:50.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.353+0000 7f745750d640 1 -- 172.21.15.28:0/1829867454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74501a5610 con 0x7f7450102190 2024-04-21T08:51:50.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:49 smithi152 ceph-mon[31401]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-21T08:51:50.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:49 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/412957867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:50.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.356+0000 7f74467fc640 1 -- 172.21.15.28:0/1829867454 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f744c046020 con 0x7f7450102190 2024-04-21T08:51:50.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.356+0000 7f74467fc640 1 --2- 172.21.15.28:0/1829867454 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f743c0a3e40 0x7f743c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.356+0000 7f74467fc640 1 -- 172.21.15.28:0/1829867454 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f744c0f2f80 con 0x7f7450102190 2024-04-21T08:51:50.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.356+0000 7f7454aa8640 1 --2- 172.21.15.28:0/1829867454 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f743c0a3e40 0x7f743c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.357+0000 7f7454aa8640 1 --2- 172.21.15.28:0/1829867454 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f743c0a3e40 0x7f743c0a6300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f74400058c0 tx=0x7f7440005850 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.357+0000 7f745a1ce740 1 Processor -- start 2024-04-21T08:51:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.357+0000 7f745a1ce740 1 -- start start 2024-04-21T08:51:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f745a1ce740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db1443b4d0 0x55db1443b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f745a1ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443be80 0x55db14444430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f745a1ce740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 0x55db14446e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f745a1ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db143a01f0 con 0x55db1443be80 2024-04-21T08:51:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f745a1ce740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55db1433d4b0 con 0x55db14444a00 2024-04-21T08:51:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f745a1ce740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55db1439c1b0 con 0x55db1443b4d0 2024-04-21T08:51:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f7454aa8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443be80 0x55db14444430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f7454aa8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443be80 0x55db14444430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42166/0 (socket says 172.21.15.28:42166) 2024-04-21T08:51:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f7454aa8640 1 -- 172.21.15.28:0/1085465512 learned_addr learned my addr 172.21.15.28:0/1085465512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f7455aaa640 1 --2- 172.21.15.28:0/1085465512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 0x55db14446e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f74552a9640 1 --2- 172.21.15.28:0/1085465512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db1443b4d0 0x55db1443b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f7454aa8640 1 -- 172.21.15.28:0/1085465512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db1443b4d0 msgr2=0x55db1443b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f7454aa8640 1 --2- 172.21.15.28:0/1085465512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db1443b4d0 0x55db1443b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f7454aa8640 1 -- 172.21.15.28:0/1085465512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 msgr2=0x55db14446e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f7454aa8640 1 --2- 172.21.15.28:0/1085465512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 0x55db14446e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f7454aa8640 1 -- 172.21.15.28:0/1085465512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db1432f1b0 con 0x55db1443be80 2024-04-21T08:51:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f7455aaa640 1 --2- 172.21.15.28:0/1085465512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 0x55db14446e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.358+0000 7f7454aa8640 1 --2- 172.21.15.28:0/1085465512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443be80 0x55db14444430 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f744000bfc0 tx=0x7f7440012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf777e8ebe7a3fd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.359+0000 7f743b7fe640 1 -- 172.21.15.28:0/1085465512 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7440022070 con 0x55db1443be80 2024-04-21T08:51:50.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.359+0000 7f743b7fe640 1 -- 172.21.15.28:0/1085465512 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7440011e50 con 0x55db1443be80 2024-04-21T08:51:50.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.359+0000 7f743b7fe640 1 -- 172.21.15.28:0/1085465512 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74400109e0 con 0x55db1443be80 2024-04-21T08:51:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.360+0000 7f745a1ce740 1 -- 172.21.15.28:0/1085465512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443be80 msgr2=0x55db14444430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.360+0000 7f745a1ce740 1 --2- 172.21.15.28:0/1085465512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443be80 0x55db14444430 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f744000bfc0 tx=0x7f7440012000 comp rx=0 tx=0).stop 2024-04-21T08:51:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.360+0000 7f745a1ce740 1 -- 172.21.15.28:0/1085465512 shutdown_connections 2024-04-21T08:51:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.360+0000 7f745a1ce740 1 --2- 172.21.15.28:0/1085465512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 0x55db14446e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.360+0000 7f745a1ce740 1 --2- 172.21.15.28:0/1085465512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443be80 0x55db14444430 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.360+0000 7f745a1ce740 1 --2- 172.21.15.28:0/1085465512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db1443b4d0 0x55db1443b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.360+0000 7f745a1ce740 1 -- 172.21.15.28:0/1085465512 >> 172.21.15.28:0/1085465512 conn(0x55db14435ba0 msgr2=0x55db14435f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.360+0000 7f745a1ce740 1 -- 172.21.15.28:0/1085465512 shutdown_connections 2024-04-21T08:51:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.360+0000 7f745a1ce740 1 -- 172.21.15.28:0/1085465512 wait complete. 2024-04-21T08:51:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.360+0000 7f745a1ce740 1 -- start start 2024-04-21T08:51:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.360+0000 7f745a1ce740 1 Processor -- start 2024-04-21T08:51:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f745a1ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443b4d0 0x55db14444360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f745a1ce740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db1443be80 0x55db144416f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f745a1ce740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 0x55db14441d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f745a1ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db14447970 con 0x55db1443b4d0 2024-04-21T08:51:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f745a1ce740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55db14359500 con 0x55db14444a00 2024-04-21T08:51:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f745a1ce740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55db14447af0 con 0x55db1443be80 2024-04-21T08:51:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f74552a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443b4d0 0x55db14444360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f74552a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443b4d0 0x55db14444360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42182/0 (socket says 172.21.15.28:42182) 2024-04-21T08:51:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f74552a9640 1 -- 172.21.15.28:0/270387117 learned_addr learned my addr 172.21.15.28:0/270387117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f745a1ce740 10 client.0 _ll_register_callbacks cb 0x55db14327b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:50.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f7455aaa640 1 --2- 172.21.15.28:0/270387117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 0x55db14441d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f7454aa8640 1 --2- 172.21.15.28:0/270387117 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db1443be80 0x55db144416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f74552a9640 1 -- 172.21.15.28:0/270387117 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db1443be80 msgr2=0x55db144416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f74552a9640 1 --2- 172.21.15.28:0/270387117 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db1443be80 0x55db144416f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f74552a9640 1 -- 172.21.15.28:0/270387117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 msgr2=0x55db14441d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f74552a9640 1 --2- 172.21.15.28:0/270387117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 0x55db14441d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.361+0000 7f74552a9640 1 -- 172.21.15.28:0/270387117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db140b23f0 con 0x55db1443b4d0 2024-04-21T08:51:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.362+0000 7f74552a9640 1 --2- 172.21.15.28:0/270387117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443b4d0 0x55db14444360 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f744c0be740 tx=0x7f744c003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4703a9e567377b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.362+0000 7f7438ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.362+0000 7f7438ff9640 1 -- 172.21.15.28:0/270387117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c0df0b0 con 0x55db1443b4d0 2024-04-21T08:51:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.362+0000 7f7438ff9640 1 -- 172.21.15.28:0/270387117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f744c0c7220 con 0x55db1443b4d0 2024-04-21T08:51:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.362+0000 7f745a1ce740 1 -- 172.21.15.28:0/270387117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7440011e50 con 0x55db1443b4d0 2024-04-21T08:51:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.362+0000 7f7438ff9640 1 -- 172.21.15.28:0/270387117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f744c0bf140 con 0x55db1443b4d0 2024-04-21T08:51:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.362+0000 7f745a1ce740 10 client.16014 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.362+0000 7f745a1ce740 1 -- 172.21.15.28:0/270387117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55db1417c7d0 con 0x55db1443b4d0 2024-04-21T08:51:50.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.362+0000 7f7438ff9640 1 -- 172.21.15.28:0/270387117 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f744c0bea40 con 0x55db1443b4d0 2024-04-21T08:51:50.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.362+0000 7f7438ff9640 1 -- 172.21.15.28:0/270387117 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f744c0f05b0 con 0x55db1443b4d0 2024-04-21T08:51:50.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.362+0000 7f745a1ce740 10 client.16014 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:50.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.362+0000 7f745a1ce740 10 client.16014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:50.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.363+0000 7f745a1ce740 20 client.16014 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:50.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.363+0000 7f745a1ce740 1 --2- 172.21.15.28:0/270387117 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55db14454bd0 0x55db14456fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.363+0000 7f745a1ce740 4 client.16014 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:50.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.363+0000 7f745a1ce740 1 -- 172.21.15.28:0/270387117 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7440011e50 con 0x55db14454bd0 2024-04-21T08:51:50.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.363+0000 7f7454aa8640 1 --2- 172.21.15.28:0/270387117 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55db14454bd0 0x55db14456fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.363+0000 7f7438ff9640 10 client.16014 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:50.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.363+0000 7f7454aa8640 1 --2- 172.21.15.28:0/270387117 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55db14454bd0 0x55db14456fc0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3715b2e80707fe8 server_cookie=6706055e5c7e5917 in_seq=0 out_seq=0 2024-04-21T08:51:50.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.364+0000 7f7438ff9640 1 -- 172.21.15.28:0/270387117 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f7440011e50 con 0x55db14454bd0 2024-04-21T08:51:50.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.364+0000 7f7438ff9640 10 client.16014 handle_command_reply: tid=0 2024-04-21T08:51:50.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.364+0000 7f745a1ce740 1 client.16014 shutdown 2024-04-21T08:51:50.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.364+0000 7f745a1ce740 1 -- 172.21.15.28:0/270387117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443b4d0 msgr2=0x55db14444360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.364+0000 7f745a1ce740 1 --2- 172.21.15.28:0/270387117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443b4d0 0x55db14444360 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f744c0be740 tx=0x7f744c003760 comp rx=0 tx=0).stop 2024-04-21T08:51:50.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.364+0000 7f745a1ce740 1 -- 172.21.15.28:0/270387117 shutdown_connections 2024-04-21T08:51:50.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.364+0000 7f745a1ce740 1 --2- 172.21.15.28:0/270387117 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55db14454bd0 0x55db14456fc0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.364+0000 7f745a1ce740 1 --2- 172.21.15.28:0/270387117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 0x55db14441d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.364+0000 7f745a1ce740 1 --2- 172.21.15.28:0/270387117 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db1443be80 0x55db144416f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.364+0000 7f745a1ce740 1 --2- 172.21.15.28:0/270387117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443b4d0 0x55db14444360 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.364+0000 7f745a1ce740 1 -- 172.21.15.28:0/270387117 >> 172.21.15.28:0/270387117 conn(0x55db14435ba0 msgr2=0x55db144378f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.364+0000 7f7438ff9640 0 client.16014 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:50.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.364+0000 7f745a1ce740 1 -- 172.21.15.28:0/270387117 shutdown_connections 2024-04-21T08:51:50.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.365+0000 7f745a1ce740 1 -- 172.21.15.28:0/270387117 wait complete. 2024-04-21T08:51:50.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.365+0000 7f745a1ce740 20 client.16014 trim_cache size 0 max 16384 2024-04-21T08:51:50.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.378+0000 7f745a1ce740 1 Processor -- start 2024-04-21T08:51:50.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.378+0000 7f745a1ce740 1 -- start start 2024-04-21T08:51:50.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f745a1ce740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db1443b4d0 0x55db14450860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f745a1ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443be80 0x55db14434a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f745a1ce740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 0x55db14443e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f745a1ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74501af9e0 con 0x55db1443be80 2024-04-21T08:51:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f745a1ce740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f74501ad2b0 con 0x55db14444a00 2024-04-21T08:51:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f745a1ce740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f74501ab100 con 0x55db1443b4d0 2024-04-21T08:51:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f7454aa8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443be80 0x55db14434a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f7454aa8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443be80 0x55db14434a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42192/0 (socket says 172.21.15.28:42192) 2024-04-21T08:51:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f7454aa8640 1 -- 172.21.15.28:0/695980802 learned_addr learned my addr 172.21.15.28:0/695980802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f7455aaa640 1 --2- 172.21.15.28:0/695980802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 0x55db14443e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f74552a9640 1 --2- 172.21.15.28:0/695980802 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db1443b4d0 0x55db14450860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f7454aa8640 1 -- 172.21.15.28:0/695980802 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db1443b4d0 msgr2=0x55db14450860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f7454aa8640 1 --2- 172.21.15.28:0/695980802 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db1443b4d0 0x55db14450860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f7454aa8640 1 -- 172.21.15.28:0/695980802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 msgr2=0x55db14443e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f7454aa8640 1 --2- 172.21.15.28:0/695980802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 0x55db14443e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f7454aa8640 1 -- 172.21.15.28:0/695980802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db14457620 con 0x55db1443be80 2024-04-21T08:51:50.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.379+0000 7f7454aa8640 1 --2- 172.21.15.28:0/695980802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443be80 0x55db14434a80 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x55db1443afa0 tx=0x7f7440015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6cadecfe6be386 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.380+0000 7f743affd640 1 -- 172.21.15.28:0/695980802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744003a070 con 0x55db1443be80 2024-04-21T08:51:50.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.380+0000 7f743affd640 1 -- 172.21.15.28:0/695980802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7440004680 con 0x55db1443be80 2024-04-21T08:51:50.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.380+0000 7f743affd640 1 -- 172.21.15.28:0/695980802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7440022040 con 0x55db1443be80 2024-04-21T08:51:50.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.381+0000 7f745a1ce740 1 -- 172.21.15.28:0/695980802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443be80 msgr2=0x55db14434a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.381+0000 7f745a1ce740 1 --2- 172.21.15.28:0/695980802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443be80 0x55db14434a80 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x55db1443afa0 tx=0x7f7440015000 comp rx=0 tx=0).stop 2024-04-21T08:51:50.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.381+0000 7f745a1ce740 1 -- 172.21.15.28:0/695980802 shutdown_connections 2024-04-21T08:51:50.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.381+0000 7f745a1ce740 1 --2- 172.21.15.28:0/695980802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db14444a00 0x55db14443e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.381+0000 7f745a1ce740 1 --2- 172.21.15.28:0/695980802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443be80 0x55db14434a80 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.381+0000 7f745a1ce740 1 --2- 172.21.15.28:0/695980802 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db1443b4d0 0x55db14450860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.381+0000 7f745a1ce740 1 -- 172.21.15.28:0/695980802 >> 172.21.15.28:0/695980802 conn(0x55db14435ba0 msgr2=0x55db14434370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.381+0000 7f745a1ce740 1 -- 172.21.15.28:0/695980802 shutdown_connections 2024-04-21T08:51:50.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.381+0000 7f745a1ce740 1 -- 172.21.15.28:0/695980802 wait complete. 2024-04-21T08:51:50.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.381+0000 7f745a1ce740 1 -- start start 2024-04-21T08:51:50.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.382+0000 7f745a1ce740 1 Processor -- start 2024-04-21T08:51:50.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.382+0000 7f745a1ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443b4d0 0x55db1444d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.382+0000 7f745a1ce740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db1443be80 0x55db1443e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.382+0000 7f745a1ce740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db14444a00 0x55db1443e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.382+0000 7f745a1ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74501ab770 con 0x55db1443b4d0 2024-04-21T08:51:50.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.382+0000 7f74552a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443b4d0 0x55db1444d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.383+0000 7f74552a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443b4d0 0x55db1444d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42196/0 (socket says 172.21.15.28:42196) 2024-04-21T08:51:50.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.383+0000 7f74552a9640 1 -- 172.21.15.28:0/3967541261 learned_addr learned my addr 172.21.15.28:0/3967541261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:50.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.383+0000 7f7454aa8640 1 --2- 172.21.15.28:0/3967541261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db1443be80 0x55db1443e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.383+0000 7f745a1ce740 1 -- 172.21.15.28:0/3967541261 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f74501acbe0 con 0x55db1443be80 2024-04-21T08:51:50.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.383+0000 7f745a1ce740 1 -- 172.21.15.28:0/3967541261 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f74501abe40 con 0x55db14444a00 2024-04-21T08:51:50.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.383+0000 7f745a1ce740 10 client.0 _ll_register_callbacks cb 0x7f744c0be810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:50.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.383+0000 7f7455aaa640 1 --2- 172.21.15.28:0/3967541261 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db14444a00 0x55db1443e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.383+0000 7f7454aa8640 1 -- 172.21.15.28:0/3967541261 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db14444a00 msgr2=0x55db1443e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.383+0000 7f7454aa8640 1 --2- 172.21.15.28:0/3967541261 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db14444a00 0x55db1443e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.383+0000 7f7454aa8640 1 -- 172.21.15.28:0/3967541261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443b4d0 msgr2=0x55db1444d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.383+0000 7f7454aa8640 1 --2- 172.21.15.28:0/3967541261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443b4d0 0x55db1444d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.383+0000 7f7454aa8640 1 -- 172.21.15.28:0/3967541261 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db1443ee10 con 0x55db1443be80 2024-04-21T08:51:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.383+0000 7f74552a9640 1 --2- 172.21.15.28:0/3967541261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443b4d0 0x55db1444d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.383+0000 7f7454aa8640 1 --2- 172.21.15.28:0/3967541261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db1443be80 0x55db1443e0a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55db14451c70 tx=0x7f744000b580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d193542bf8ac402e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.383+0000 7f7438ff9640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:51:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.384+0000 7f7438ff9640 1 -- 172.21.15.28:0/3967541261 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7440043060 con 0x55db1443be80 2024-04-21T08:51:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.384+0000 7f7438ff9640 1 -- 172.21.15.28:0/3967541261 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f7440004a30 con 0x55db1443be80 2024-04-21T08:51:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.384+0000 7f745a1ce740 1 -- 172.21.15.28:0/3967541261 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7440004680 con 0x55db1443be80 2024-04-21T08:51:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.384+0000 7f7438ff9640 1 -- 172.21.15.28:0/3967541261 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f744003a040 con 0x55db1443be80 2024-04-21T08:51:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.384+0000 7f745a1ce740 10 client.25475 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.384+0000 7f745a1ce740 1 -- 172.21.15.28:0/3967541261 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55db144bb140 con 0x55db1443be80 2024-04-21T08:51:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.384+0000 7f7438ff9640 1 -- 172.21.15.28:0/3967541261 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7440004680 con 0x55db1443be80 2024-04-21T08:51:50.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.384+0000 7f7438ff9640 1 -- 172.21.15.28:0/3967541261 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f744003d070 con 0x55db1443be80 2024-04-21T08:51:50.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.384+0000 7f745a1ce740 10 client.25475 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:50.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.384+0000 7f745a1ce740 10 client.25475 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:50.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.384+0000 7f745a1ce740 20 client.25475 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:50.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.384+0000 7f745a1ce740 1 --2- 172.21.15.28:0/3967541261 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55db144bb670 0x55db144bda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:50.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.384+0000 7f745a1ce740 4 client.25475 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:51:50.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.384+0000 7f745a1ce740 1 -- 172.21.15.28:0/3967541261 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55db1433f930 con 0x55db144bb670 2024-04-21T08:51:50.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.385+0000 7f74552a9640 1 --2- 172.21.15.28:0/3967541261 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55db144bb670 0x55db144bda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:50.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.385+0000 7f74552a9640 1 --2- 172.21.15.28:0/3967541261 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55db144bb670 0x55db144bda90 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ae2f67d09bde66c server_cookie=f58b3e883b43baf6 in_seq=0 out_seq=0 2024-04-21T08:51:50.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.385+0000 7f7438ff9640 10 client.25475 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:50.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.385+0000 7f7438ff9640 1 -- 172.21.15.28:0/3967541261 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55db1433f930 con 0x55db144bb670 2024-04-21T08:51:50.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.385+0000 7f7438ff9640 10 client.25475 handle_command_reply: tid=0 2024-04-21T08:51:50.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.385+0000 7f745a1ce740 1 client.25475 shutdown 2024-04-21T08:51:50.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f745a1ce740 1 -- 172.21.15.28:0/3967541261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db1443be80 msgr2=0x55db1443e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f745a1ce740 1 --2- 172.21.15.28:0/3967541261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db1443be80 0x55db1443e0a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55db14451c70 tx=0x7f744000b580 comp rx=0 tx=0).stop 2024-04-21T08:51:50.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f745a1ce740 1 -- 172.21.15.28:0/3967541261 shutdown_connections 2024-04-21T08:51:50.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f745a1ce740 1 --2- 172.21.15.28:0/3967541261 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55db144bb670 0x55db144bda90 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f745a1ce740 1 --2- 172.21.15.28:0/3967541261 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55db14444a00 0x55db1443e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f745a1ce740 1 --2- 172.21.15.28:0/3967541261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db1443be80 0x55db1443e0a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f745a1ce740 1 --2- 172.21.15.28:0/3967541261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db1443b4d0 0x55db1444d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f7438ff9640 0 client.25475 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:50.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f745a1ce740 1 -- 172.21.15.28:0/3967541261 >> 172.21.15.28:0/3967541261 conn(0x55db14435ba0 msgr2=0x55db1443caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f745a1ce740 1 -- 172.21.15.28:0/3967541261 shutdown_connections 2024-04-21T08:51:50.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f745a1ce740 1 -- 172.21.15.28:0/3967541261 wait complete. 2024-04-21T08:51:50.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f745a1ce740 20 client.25475 trim_cache size 0 max 16384 2024-04-21T08:51:50.391 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:51:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f745750d640 1 -- 172.21.15.28:0/1829867454 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f743c0a3e40 msgr2=0x7f743c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f745750d640 1 --2- 172.21.15.28:0/1829867454 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f743c0a3e40 0x7f743c0a6300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f74400058c0 tx=0x7f7440005850 comp rx=0 tx=0).stop 2024-04-21T08:51:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f745750d640 1 -- 172.21.15.28:0/1829867454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7450102190 msgr2=0x7f745019e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.386+0000 7f745750d640 1 --2- 172.21.15.28:0/1829867454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7450102190 0x7f745019e410 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f744c000c00 tx=0x7f744c02c7e0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.387+0000 7f745750d640 1 -- 172.21.15.28:0/1829867454 shutdown_connections 2024-04-21T08:51:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.387+0000 7f745750d640 1 --2- 172.21.15.28:0/1829867454 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f743c0a3e40 0x7f743c0a6300 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.387+0000 7f745750d640 1 --2- 172.21.15.28:0/1829867454 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74501985c0 0x7f7450198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.387+0000 7f745750d640 1 --2- 172.21.15.28:0/1829867454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7450107260 0x7f745019e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.387+0000 7f745750d640 1 --2- 172.21.15.28:0/1829867454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7450102190 0x7f745019e410 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.387+0000 7f745750d640 1 -- 172.21.15.28:0/1829867454 >> 172.21.15.28:0/1829867454 conn(0x7f74500793a0 msgr2=0x7f7450100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.387+0000 7f745750d640 1 -- 172.21.15.28:0/1829867454 shutdown_connections 2024-04-21T08:51:50.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:50.387+0000 7f745750d640 1 -- 172.21.15.28:0/1829867454 wait complete. 2024-04-21T08:51:51.398 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:51:51.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:51:51.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:51:51.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.508+0000 7f20fd206640 1 -- 172.21.15.28:0/4168848276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20f8103af0 msgr2=0x7f20f810bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.508+0000 7f20fd206640 1 --2- 172.21.15.28:0/4168848276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20f8103af0 0x7f20f810bfd0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f20ec009900 tx=0x7f20ec02bd60 comp rx=0 tx=0).stop 2024-04-21T08:51:51.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.508+0000 7f20fd206640 1 -- 172.21.15.28:0/4168848276 shutdown_connections 2024-04-21T08:51:51.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.508+0000 7f20fd206640 1 --2- 172.21.15.28:0/4168848276 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f20f810c510 0x7f20f810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.508+0000 7f20fd206640 1 --2- 172.21.15.28:0/4168848276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20f8103af0 0x7f20f810bfd0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.508+0000 7f20fd206640 1 --2- 172.21.15.28:0/4168848276 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f20f81031d0 0x7f20f81035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.508+0000 7f20fd206640 1 -- 172.21.15.28:0/4168848276 >> 172.21.15.28:0/4168848276 conn(0x7f20f8077f50 msgr2=0x7f20f8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:51.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.509+0000 7f20fd206640 1 -- 172.21.15.28:0/4168848276 shutdown_connections 2024-04-21T08:51:51.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.509+0000 7f20fd206640 1 -- 172.21.15.28:0/4168848276 wait complete. 2024-04-21T08:51:51.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.510+0000 7f20fd206640 1 Processor -- start 2024-04-21T08:51:51.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.510+0000 7f20fd206640 1 -- start start 2024-04-21T08:51:51.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.510+0000 7f20fd206640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f20f81031d0 0x7f20f819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.510+0000 7f20fd206640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20f810c510 0x7f20f819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.510+0000 7f20f659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20f810c510 0x7f20f819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.510+0000 7f20f659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20f810c510 0x7f20f819eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42216/0 (socket says 172.21.15.28:42216) 2024-04-21T08:51:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.510+0000 7f20f659c640 1 -- 172.21.15.28:0/982538764 learned_addr learned my addr 172.21.15.28:0/982538764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.510+0000 7f20fd206640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f20f8198790 0x7f20f8198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.510+0000 7f20fd206640 1 -- 172.21.15.28:0/982538764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20f81121a0 con 0x7f20f810c510 2024-04-21T08:51:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.510+0000 7f20fd206640 1 -- 172.21.15.28:0/982538764 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f20f8112020 con 0x7f20f8198790 2024-04-21T08:51:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.510+0000 7f20fd206640 1 -- 172.21.15.28:0/982538764 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f20f8112320 con 0x7f20f81031d0 2024-04-21T08:51:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.510+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f20f81031d0 0x7f20f819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.511+0000 7f20f759e640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f20f8198790 0x7f20f8198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.511+0000 7f20f659c640 1 -- 172.21.15.28:0/982538764 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f20f81031d0 msgr2=0x7f20f819e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.511+0000 7f20f659c640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f20f81031d0 0x7f20f819e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.511+0000 7f20f659c640 1 -- 172.21.15.28:0/982538764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f20f8198790 msgr2=0x7f20f8198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.511+0000 7f20f659c640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f20f8198790 0x7f20f8198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.511+0000 7f20f659c640 1 -- 172.21.15.28:0/982538764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f8199320 con 0x7f20f810c510 2024-04-21T08:51:51.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.511+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f20f81031d0 0x7f20f819e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:51.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.511+0000 7f20f659c640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20f810c510 0x7f20f819eb60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f20ec035000 tx=0x7f20ec0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:51.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.511+0000 7f20dffff640 1 -- 172.21.15.28:0/982538764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec041020 con 0x7f20f810c510 2024-04-21T08:51:51.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.511+0000 7f20dffff640 1 -- 172.21.15.28:0/982538764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20ec004e00 con 0x7f20f810c510 2024-04-21T08:51:51.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.511+0000 7f20dffff640 1 -- 172.21.15.28:0/982538764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec0038e0 con 0x7f20f810c510 2024-04-21T08:51:51.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.511+0000 7f20fd206640 1 -- 172.21.15.28:0/982538764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f81995b0 con 0x7f20f810c510 2024-04-21T08:51:51.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.512+0000 7f20fd206640 1 -- 172.21.15.28:0/982538764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f81a5740 con 0x7f20f810c510 2024-04-21T08:51:51.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.515+0000 7f20dffff640 1 -- 172.21.15.28:0/982538764 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f20ec044020 con 0x7f20f810c510 2024-04-21T08:51:51.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.515+0000 7f20dffff640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f20e00a3e40 0x7f20e00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.515+0000 7f20dffff640 1 -- 172.21.15.28:0/982538764 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f20ec0f22e0 con 0x7f20f810c510 2024-04-21T08:51:51.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.515+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f20e00a3e40 0x7f20e00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.516+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f20e00a3e40 0x7f20e00a6300 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f20e8009fc0 tx=0x7f20e8005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20ffdf5740 1 Processor -- start 2024-04-21T08:51:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20ffdf5740 1 -- start start 2024-04-21T08:51:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20ffdf5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e2238e0 0x56006e223cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20ffdf5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e224290 0x56006e22c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20ffdf5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e22ce10 0x56006e22f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20ffdf5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56006e187f00 con 0x56006e22ce10 2024-04-21T08:51:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20ffdf5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56006e125590 con 0x56006e224290 2024-04-21T08:51:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20ffdf5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56006e183ec0 con 0x56006e2238e0 2024-04-21T08:51:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20f759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e22ce10 0x56006e22f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20f759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e22ce10 0x56006e22f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42230/0 (socket says 172.21.15.28:42230) 2024-04-21T08:51:51.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20f759e640 1 -- 172.21.15.28:0/3447151598 learned_addr learned my addr 172.21.15.28:0/3447151598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:51.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/3447151598 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e2238e0 0x56006e223cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20f659c640 1 --2- 172.21.15.28:0/3447151598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e224290 0x56006e22c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20f6d9d640 1 -- 172.21.15.28:0/3447151598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e224290 msgr2=0x56006e22c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:51.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/3447151598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e224290 0x56006e22c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20f6d9d640 1 -- 172.21.15.28:0/3447151598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e22ce10 msgr2=0x56006e22f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:51.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/3447151598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e22ce10 0x56006e22f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20f6d9d640 1 -- 172.21.15.28:0/3447151598 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56006df627e0 con 0x56006e2238e0 2024-04-21T08:51:51.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.517+0000 7f20f759e640 1 --2- 172.21.15.28:0/3447151598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e22ce10 0x56006e22f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:51.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.518+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/3447151598 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e2238e0 0x56006e223cc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f20e8011ca0 tx=0x7f20e8014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24129a717269619c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.518+0000 7f20dd7fa640 1 -- 172.21.15.28:0/3447151598 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e800f480 con 0x56006e2238e0 2024-04-21T08:51:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.518+0000 7f20dd7fa640 1 -- 172.21.15.28:0/3447151598 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20e800fad0 con 0x56006e2238e0 2024-04-21T08:51:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.518+0000 7f20dd7fa640 1 -- 172.21.15.28:0/3447151598 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e801ca70 con 0x56006e2238e0 2024-04-21T08:51:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.519+0000 7f20ffdf5740 1 -- 172.21.15.28:0/3447151598 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e2238e0 msgr2=0x56006e223cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.519+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/3447151598 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e2238e0 0x56006e223cc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f20e8011ca0 tx=0x7f20e8014000 comp rx=0 tx=0).stop 2024-04-21T08:51:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.519+0000 7f20ffdf5740 1 -- 172.21.15.28:0/3447151598 shutdown_connections 2024-04-21T08:51:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.519+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/3447151598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e22ce10 0x56006e22f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.519+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/3447151598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e224290 0x56006e22c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.519+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/3447151598 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e2238e0 0x56006e223cc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.519+0000 7f20ffdf5740 1 -- 172.21.15.28:0/3447151598 >> 172.21.15.28:0/3447151598 conn(0x56006e21dfb0 msgr2=0x56006e21e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.519+0000 7f20ffdf5740 1 -- 172.21.15.28:0/3447151598 shutdown_connections 2024-04-21T08:51:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.519+0000 7f20ffdf5740 1 -- 172.21.15.28:0/3447151598 wait complete. 2024-04-21T08:51:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.519+0000 7f20ffdf5740 1 -- start start 2024-04-21T08:51:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20ffdf5740 1 Processor -- start 2024-04-21T08:51:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20ffdf5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e2238e0 0x56006e229e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20ffdf5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e224290 0x56006e22a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20ffdf5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e22ce10 0x56006e22a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20ffdf5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56006e22fd80 con 0x56006e224290 2024-04-21T08:51:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20ffdf5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56006e141580 con 0x56006e22ce10 2024-04-21T08:51:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20ffdf5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56006e22ff00 con 0x56006e2238e0 2024-04-21T08:51:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20f659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e224290 0x56006e22a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20ffdf5740 10 client.0 _ll_register_callbacks cb 0x56006e147650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20f659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e224290 0x56006e22a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42244/0 (socket says 172.21.15.28:42244) 2024-04-21T08:51:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20f659c640 1 -- 172.21.15.28:0/449044852 learned_addr learned my addr 172.21.15.28:0/449044852 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20f759e640 1 --2- 172.21.15.28:0/449044852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e22ce10 0x56006e22a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/449044852 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e2238e0 0x56006e229e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20f659c640 1 -- 172.21.15.28:0/449044852 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e2238e0 msgr2=0x56006e229e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20f659c640 1 --2- 172.21.15.28:0/449044852 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e2238e0 0x56006e229e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20f659c640 1 -- 172.21.15.28:0/449044852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e22ce10 msgr2=0x56006e22a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20f659c640 1 --2- 172.21.15.28:0/449044852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e22ce10 0x56006e22a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20f659c640 1 -- 172.21.15.28:0/449044852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56006dfc2f70 con 0x56006e224290 2024-04-21T08:51:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.520+0000 7f20f759e640 1 --2- 172.21.15.28:0/449044852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e22ce10 0x56006e22a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20f659c640 1 --2- 172.21.15.28:0/449044852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e224290 0x56006e22a360 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0bdd30 tx=0x7f20ec0be560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f670139c16724e03 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20daff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20daff5640 1 -- 172.21.15.28:0/449044852 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec0d7060 con 0x56006e224290 2024-04-21T08:51:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20daff5640 1 -- 172.21.15.28:0/449044852 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f20ec0d7200 con 0x56006e224290 2024-04-21T08:51:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20ffdf5740 1 -- 172.21.15.28:0/449044852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20e800fad0 con 0x56006e224290 2024-04-21T08:51:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20daff5640 1 -- 172.21.15.28:0/449044852 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f20ec0c1070 con 0x56006e224290 2024-04-21T08:51:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20ffdf5740 10 client.16032 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20ffdf5740 1 -- 172.21.15.28:0/449044852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56006de5a170 con 0x56006e224290 2024-04-21T08:51:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20daff5640 1 -- 172.21.15.28:0/449044852 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20e800fad0 con 0x56006e224290 2024-04-21T08:51:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20daff5640 1 -- 172.21.15.28:0/449044852 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f20ec0c6580 con 0x56006e224290 2024-04-21T08:51:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20ffdf5740 10 client.16032 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20ffdf5740 10 client.16032 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20ffdf5740 20 client.16032 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/449044852 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56006e23d2f0 0x56006e23f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20ffdf5740 4 client.16032 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.521+0000 7f20ffdf5740 1 -- 172.21.15.28:0/449044852 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56006e195920 con 0x56006e23d2f0 2024-04-21T08:51:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.522+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/449044852 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56006e23d2f0 0x56006e23f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.522+0000 7f20daff5640 10 client.16032 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.522+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/449044852 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56006e23d2f0 0x56006e23f720 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f64cc3cd7a6690ae server_cookie=ee6c9ea58098530a in_seq=0 out_seq=0 2024-04-21T08:51:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.523+0000 7f20daff5640 1 -- 172.21.15.28:0/449044852 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56006e195920 con 0x56006e23d2f0 2024-04-21T08:51:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.523+0000 7f20daff5640 10 client.16032 handle_command_reply: tid=0 2024-04-21T08:51:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.523+0000 7f20ffdf5740 1 client.16032 shutdown 2024-04-21T08:51:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.523+0000 7f20ffdf5740 1 -- 172.21.15.28:0/449044852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e224290 msgr2=0x56006e22a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.523+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/449044852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e224290 0x56006e22a360 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0bdd30 tx=0x7f20ec0be560 comp rx=0 tx=0).stop 2024-04-21T08:51:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.523+0000 7f20ffdf5740 1 -- 172.21.15.28:0/449044852 shutdown_connections 2024-04-21T08:51:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.523+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/449044852 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56006e23d2f0 0x56006e23f720 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.523+0000 7f20daff5640 0 client.16032 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.523+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/449044852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e22ce10 0x56006e22a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.523+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/449044852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e224290 0x56006e22a360 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.523+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/449044852 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e2238e0 0x56006e229e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.523+0000 7f20ffdf5740 1 -- 172.21.15.28:0/449044852 >> 172.21.15.28:0/449044852 conn(0x56006e21dfb0 msgr2=0x56006e21fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.523+0000 7f20ffdf5740 1 -- 172.21.15.28:0/449044852 shutdown_connections 2024-04-21T08:51:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.523+0000 7f20ffdf5740 1 -- 172.21.15.28:0/449044852 wait complete. 2024-04-21T08:51:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.523+0000 7f20ffdf5740 20 client.16032 trim_cache size 0 max 16384 2024-04-21T08:51:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.541+0000 7f20ffdf5740 1 Processor -- start 2024-04-21T08:51:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.541+0000 7f20ffdf5740 1 -- start start 2024-04-21T08:51:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20ffdf5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e2238e0 0x56006e22c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20ffdf5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e224290 0x56006e238c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20ffdf5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e22ce10 0x56006e228d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20ffdf5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20f81b2280 con 0x56006e22ce10 2024-04-21T08:51:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20ffdf5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f20f81afb50 con 0x56006e2238e0 2024-04-21T08:51:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20ffdf5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f20f81ad9a0 con 0x56006e224290 2024-04-21T08:51:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20f759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e22ce10 0x56006e228d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20f759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e22ce10 0x56006e228d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42260/0 (socket says 172.21.15.28:42260) 2024-04-21T08:51:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20f759e640 1 -- 172.21.15.28:0/1163612046 learned_addr learned my addr 172.21.15.28:0/1163612046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/1163612046 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e2238e0 0x56006e22c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20f659c640 1 --2- 172.21.15.28:0/1163612046 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e224290 0x56006e238c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20f6d9d640 1 -- 172.21.15.28:0/1163612046 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e224290 msgr2=0x56006e238c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/1163612046 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e224290 0x56006e238c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20f6d9d640 1 -- 172.21.15.28:0/1163612046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e22ce10 msgr2=0x56006e228d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/1163612046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e22ce10 0x56006e228d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20f6d9d640 1 -- 172.21.15.28:0/1163612046 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56006e2253d0 con 0x56006e2238e0 2024-04-21T08:51:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20f759e640 1 --2- 172.21.15.28:0/1163612046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e22ce10 0x56006e228d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.542+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/1163612046 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e2238e0 0x56006e22c550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f20e8003a80 tx=0x7f20e80091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5851c77b60ebc23 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.543+0000 7f20dcff9640 1 -- 172.21.15.28:0/1163612046 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e801c3f0 con 0x56006e2238e0 2024-04-21T08:51:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.543+0000 7f20dcff9640 1 -- 172.21.15.28:0/1163612046 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20e80054f0 con 0x56006e2238e0 2024-04-21T08:51:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.543+0000 7f20dcff9640 1 -- 172.21.15.28:0/1163612046 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e8033910 con 0x56006e2238e0 2024-04-21T08:51:51.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.543+0000 7f20ffdf5740 1 -- 172.21.15.28:0/1163612046 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e2238e0 msgr2=0x56006e22c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.543+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/1163612046 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e2238e0 0x56006e22c550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f20e8003a80 tx=0x7f20e80091e0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.543+0000 7f20ffdf5740 1 -- 172.21.15.28:0/1163612046 shutdown_connections 2024-04-21T08:51:51.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.543+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/1163612046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e22ce10 0x56006e228d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.543+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/1163612046 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e224290 0x56006e238c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.543+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/1163612046 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e2238e0 0x56006e22c550 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.543+0000 7f20ffdf5740 1 -- 172.21.15.28:0/1163612046 >> 172.21.15.28:0/1163612046 conn(0x56006e21dfb0 msgr2=0x56006e22a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:51.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20ffdf5740 1 -- 172.21.15.28:0/1163612046 shutdown_connections 2024-04-21T08:51:51.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20ffdf5740 1 -- 172.21.15.28:0/1163612046 wait complete. 2024-04-21T08:51:51.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20ffdf5740 1 -- start start 2024-04-21T08:51:51.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20ffdf5740 1 Processor -- start 2024-04-21T08:51:51.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20ffdf5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e2238e0 0x56006e2353e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20ffdf5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e224290 0x56006e29e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20ffdf5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e22ce10 0x56006e29edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20ffdf5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20f81ae010 con 0x56006e2238e0 2024-04-21T08:51:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20ffdf5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f20f81af480 con 0x56006e224290 2024-04-21T08:51:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20ffdf5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f20f81ae6e0 con 0x56006e22ce10 2024-04-21T08:51:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20ffdf5740 10 client.0 _ll_register_callbacks cb 0x7f20ec0bdb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20f6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e2238e0 0x56006e2353e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20f659c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e224290 0x56006e29e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20f6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e2238e0 0x56006e2353e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42268/0 (socket says 172.21.15.28:42268) 2024-04-21T08:51:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20f6d9d640 1 -- 172.21.15.28:0/225448500 learned_addr learned my addr 172.21.15.28:0/225448500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.544+0000 7f20f759e640 1 --2- 172.21.15.28:0/225448500 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e22ce10 0x56006e29edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.545+0000 7f20f659c640 1 -- 172.21.15.28:0/225448500 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e22ce10 msgr2=0x56006e29edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.545+0000 7f20f659c640 1 --2- 172.21.15.28:0/225448500 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e22ce10 0x56006e29edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.545+0000 7f20f659c640 1 -- 172.21.15.28:0/225448500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e2238e0 msgr2=0x56006e2353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.545+0000 7f20f659c640 1 --2- 172.21.15.28:0/225448500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e2238e0 0x56006e2353e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.545+0000 7f20f659c640 1 -- 172.21.15.28:0/225448500 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56006e29f490 con 0x56006e224290 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.545+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/225448500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e2238e0 0x56006e2353e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.545+0000 7f20f659c640 1 --2- 172.21.15.28:0/225448500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e224290 0x56006e29e8b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f20ec038040 tx=0x7f20ec0c22d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce0593e9e850e375 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.545+0000 7f20daff5640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.545+0000 7f20daff5640 1 -- 172.21.15.28:0/225448500 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec0bd700 con 0x56006e224290 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.545+0000 7f20ffdf5740 1 -- 172.21.15.28:0/225448500 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20e80054f0 con 0x56006e224290 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.545+0000 7f20daff5640 1 -- 172.21.15.28:0/225448500 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f20ec0d7260 con 0x56006e224290 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.545+0000 7f20daff5640 1 -- 172.21.15.28:0/225448500 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f20ec09cc60 con 0x56006e224290 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.545+0000 7f20ffdf5740 10 client.25493 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.545+0000 7f20ffdf5740 1 -- 172.21.15.28:0/225448500 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56006e2341b0 con 0x56006e224290 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.545+0000 7f20daff5640 1 -- 172.21.15.28:0/225448500 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20e80054f0 con 0x56006e224290 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.546+0000 7f20daff5640 1 -- 172.21.15.28:0/225448500 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f20ec0bd380 con 0x56006e224290 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.546+0000 7f20ffdf5740 10 client.25493 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.546+0000 7f20ffdf5740 10 client.25493 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.546+0000 7f20ffdf5740 20 client.25493 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.546+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/225448500 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56006e23d2f0 0x56006e234730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.546+0000 7f20ffdf5740 4 client.25493 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:51:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.546+0000 7f20ffdf5740 1 -- 172.21.15.28:0/225448500 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56006e18f790 con 0x56006e23d2f0 2024-04-21T08:51:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.546+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/225448500 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56006e23d2f0 0x56006e234730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.546+0000 7f20daff5640 10 client.25493 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.546+0000 7f20f6d9d640 1 --2- 172.21.15.28:0/225448500 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56006e23d2f0 0x56006e234730 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e6af8530800f0ae server_cookie=1511fb34cabf1396 in_seq=0 out_seq=0 2024-04-21T08:51:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.547+0000 7f20daff5640 1 -- 172.21.15.28:0/225448500 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56006e18f790 con 0x56006e23d2f0 2024-04-21T08:51:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.547+0000 7f20daff5640 10 client.25493 handle_command_reply: tid=0 2024-04-21T08:51:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.547+0000 7f20ffdf5740 1 client.25493 shutdown 2024-04-21T08:51:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.547+0000 7f20ffdf5740 1 -- 172.21.15.28:0/225448500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e224290 msgr2=0x56006e29e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.547+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/225448500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e224290 0x56006e29e8b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f20ec038040 tx=0x7f20ec0c22d0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.548+0000 7f20ffdf5740 1 -- 172.21.15.28:0/225448500 shutdown_connections 2024-04-21T08:51:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.548+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/225448500 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56006e23d2f0 0x56006e234730 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.548+0000 7f20daff5640 0 client.25493 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.548+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/225448500 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56006e22ce10 0x56006e29edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.548+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/225448500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56006e224290 0x56006e29e8b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.548+0000 7f20ffdf5740 1 --2- 172.21.15.28:0/225448500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56006e2238e0 0x56006e2353e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.548+0000 7f20ffdf5740 1 -- 172.21.15.28:0/225448500 >> 172.21.15.28:0/225448500 conn(0x56006e21dfb0 msgr2=0x56006e229de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.548+0000 7f20ffdf5740 1 -- 172.21.15.28:0/225448500 shutdown_connections 2024-04-21T08:51:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.548+0000 7f20ffdf5740 1 -- 172.21.15.28:0/225448500 wait complete. 2024-04-21T08:51:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.548+0000 7f20ffdf5740 20 client.25493 trim_cache size 0 max 16384 2024-04-21T08:51:51.552 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:51:51.552 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:51:51.552 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "c11d434c-65c8-4f2a-ba04-258d7c90df05", 2024-04-21T08:51:51.552 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:51:51.552 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:51:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.548+0000 7f20fd206640 1 -- 172.21.15.28:0/982538764 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f20e00a3e40 msgr2=0x7f20e00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.548+0000 7f20fd206640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f20e00a3e40 0x7f20e00a6300 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f20e8009fc0 tx=0x7f20e8005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.548+0000 7f20fd206640 1 -- 172.21.15.28:0/982538764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20f810c510 msgr2=0x7f20f819eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.548+0000 7f20fd206640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20f810c510 0x7f20f819eb60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f20ec035000 tx=0x7f20ec0045e0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.549+0000 7f20fd206640 1 -- 172.21.15.28:0/982538764 shutdown_connections 2024-04-21T08:51:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.549+0000 7f20fd206640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f20e00a3e40 0x7f20e00a6300 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.549+0000 7f20fd206640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f20f8198790 0x7f20f8198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.549+0000 7f20fd206640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20f810c510 0x7f20f819eb60 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.549+0000 7f20fd206640 1 --2- 172.21.15.28:0/982538764 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f20f81031d0 0x7f20f819e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.549+0000 7f20fd206640 1 -- 172.21.15.28:0/982538764 >> 172.21.15.28:0/982538764 conn(0x7f20f8077f50 msgr2=0x7f20f80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.549+0000 7f20fd206640 1 -- 172.21.15.28:0/982538764 shutdown_connections 2024-04-21T08:51:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.549+0000 7f20fd206640 1 -- 172.21.15.28:0/982538764 wait complete. 2024-04-21T08:51:51.563 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:51:51.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:51:51.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.672+0000 7f1acde31640 1 -- 172.21.15.28:0/2963457040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac810c500 msgr2=0x7f1ac810e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.672+0000 7f1acde31640 1 --2- 172.21.15.28:0/2963457040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac810c500 0x7f1ac810e9a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009ca0 tx=0x7f1ac0030640 comp rx=0 tx=0).stop 2024-04-21T08:51:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.672+0000 7f1acde31640 1 -- 172.21.15.28:0/2963457040 shutdown_connections 2024-04-21T08:51:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.672+0000 7f1acde31640 1 --2- 172.21.15.28:0/2963457040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac810c500 0x7f1ac810e9a0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.672+0000 7f1acde31640 1 --2- 172.21.15.28:0/2963457040 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ac8103ae0 0x7f1ac810bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.672+0000 7f1acde31640 1 --2- 172.21.15.28:0/2963457040 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ac81031c0 0x7f1ac81035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.672+0000 7f1acde31640 1 -- 172.21.15.28:0/2963457040 >> 172.21.15.28:0/2963457040 conn(0x7f1ac8078000 msgr2=0x7f1ac8078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.673+0000 7f1acde31640 1 -- 172.21.15.28:0/2963457040 shutdown_connections 2024-04-21T08:51:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.673+0000 7f1acde31640 1 -- 172.21.15.28:0/2963457040 wait complete. 2024-04-21T08:51:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.673+0000 7f1acde31640 1 Processor -- start 2024-04-21T08:51:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.673+0000 7f1acde31640 1 -- start start 2024-04-21T08:51:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.673+0000 7f1acde31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac81031c0 0x7f1ac81964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.673+0000 7f1acde31640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ac8103ae0 0x7f1ac8196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1acde31640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ac810c500 0x7f1ac819ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1acde31640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac810fff0 con 0x7f1ac81031c0 2024-04-21T08:51:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1acde31640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac810fe70 con 0x7f1ac8103ae0 2024-04-21T08:51:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1acde31640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac8110170 con 0x7f1ac810c500 2024-04-21T08:51:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1ac77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac81031c0 0x7f1ac81964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1ac77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac81031c0 0x7f1ac81964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42278/0 (socket says 172.21.15.28:42278) 2024-04-21T08:51:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1ac77fe640 1 -- 172.21.15.28:0/2118938307 learned_addr learned my addr 172.21.15.28:0/2118938307 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1ac6ffd640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ac8103ae0 0x7f1ac8196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1ac7fff640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ac810c500 0x7f1ac819ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1ac77fe640 1 -- 172.21.15.28:0/2118938307 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ac810c500 msgr2=0x7f1ac819ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1ac77fe640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ac810c500 0x7f1ac819ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1ac77fe640 1 -- 172.21.15.28:0/2118938307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ac8103ae0 msgr2=0x7f1ac8196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1ac77fe640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ac8103ae0 0x7f1ac8196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1ac77fe640 1 -- 172.21.15.28:0/2118938307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac819b4c0 con 0x7f1ac81031c0 2024-04-21T08:51:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1ac7fff640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ac810c500 0x7f1ac819ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1ac6ffd640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ac8103ae0 0x7f1ac8196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.674+0000 7f1ac77fe640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac81031c0 0x7f1ac81964d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1ab800e9f0 tx=0x7f1ab800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.675+0000 7f1ac4ff9640 1 -- 172.21.15.28:0/2118938307 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab800cd80 con 0x7f1ac81031c0 2024-04-21T08:51:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.675+0000 7f1ac4ff9640 1 -- 172.21.15.28:0/2118938307 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ab80054d0 con 0x7f1ac81031c0 2024-04-21T08:51:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.675+0000 7f1ac4ff9640 1 -- 172.21.15.28:0/2118938307 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8010690 con 0x7f1ac81031c0 2024-04-21T08:51:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.675+0000 7f1acde31640 1 -- 172.21.15.28:0/2118938307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ac819b7b0 con 0x7f1ac81031c0 2024-04-21T08:51:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.675+0000 7f1acde31640 1 -- 172.21.15.28:0/2118938307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ac81a3040 con 0x7f1ac81031c0 2024-04-21T08:51:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.678+0000 7f1ad0a23740 1 Processor -- start 2024-04-21T08:51:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.678+0000 7f1ad0a23740 1 -- start start 2024-04-21T08:51:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.678+0000 7f1ad0a23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5a580 0x55bb88d5a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.678+0000 7f1ad0a23740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5aea0 0x55bb88d63540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.678+0000 7f1ad0a23740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d63a80 0x55bb88d65e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.678+0000 7f1ad0a23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bb88cbf1e0 con 0x55bb88d5a580 2024-04-21T08:51:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.678+0000 7f1ad0a23740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55bb88c5c390 con 0x55bb88d63a80 2024-04-21T08:51:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.678+0000 7f1ad0a23740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55bb88cbb1a0 con 0x55bb88d5aea0 2024-04-21T08:51:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.678+0000 7f1ac7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5a580 0x55bb88d5a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.678+0000 7f1ac7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5a580 0x55bb88d5a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42282/0 (socket says 172.21.15.28:42282) 2024-04-21T08:51:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.678+0000 7f1ac7fff640 1 -- 172.21.15.28:0/125097215 learned_addr learned my addr 172.21.15.28:0/125097215 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.678+0000 7f1ac4ff9640 1 -- 172.21.15.28:0/2118938307 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f1ab8004e90 con 0x7f1ac81031c0 2024-04-21T08:51:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.678+0000 7f1ac6ffd640 1 --2- 172.21.15.28:0/125097215 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d63a80 0x55bb88d65e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.679+0000 7f1ac77fe640 1 --2- 172.21.15.28:0/125097215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5aea0 0x55bb88d63540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.679+0000 7f1ac7fff640 1 -- 172.21.15.28:0/125097215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5aea0 msgr2=0x55bb88d63540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.679+0000 7f1ac7fff640 1 --2- 172.21.15.28:0/125097215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5aea0 0x55bb88d63540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.679+0000 7f1ac7fff640 1 -- 172.21.15.28:0/125097215 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d63a80 msgr2=0x55bb88d65e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.679+0000 7f1ac7fff640 1 --2- 172.21.15.28:0/125097215 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d63a80 0x55bb88d65e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.679+0000 7f1ac7fff640 1 -- 172.21.15.28:0/125097215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb88a9b7d0 con 0x55bb88d5a580 2024-04-21T08:51:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.679+0000 7f1ac4ff9640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ab00a3ec0 0x7f1ab00a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.679+0000 7f1ac7fff640 1 --2- 172.21.15.28:0/125097215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5a580 0x55bb88d5a960 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009dc0 tx=0x7f1ac0010410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b578b2193bc699 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.679+0000 7f1ac4ff9640 1 -- 172.21.15.28:0/2118938307 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1ab80d12d0 con 0x7f1ac81031c0 2024-04-21T08:51:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.680+0000 7f1ac77fe640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ab00a3ec0 0x7f1ab00a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.680+0000 7f1aadffb640 1 -- 172.21.15.28:0/125097215 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0045070 con 0x55bb88d5a580 2024-04-21T08:51:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.680+0000 7f1aadffb640 1 -- 172.21.15.28:0/125097215 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ac0003e20 con 0x55bb88d5a580 2024-04-21T08:51:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.680+0000 7f1aadffb640 1 -- 172.21.15.28:0/125097215 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0004040 con 0x55bb88d5a580 2024-04-21T08:51:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.680+0000 7f1ad0a23740 1 -- 172.21.15.28:0/125097215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5a580 msgr2=0x55bb88d5a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.680+0000 7f1ac77fe640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ab00a3ec0 0x7f1ab00a6380 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80ce500 tx=0x7f1ab8004a90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.680+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/125097215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5a580 0x55bb88d5a960 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009dc0 tx=0x7f1ac0010410 comp rx=0 tx=0).stop 2024-04-21T08:51:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.681+0000 7f1ad0a23740 1 -- 172.21.15.28:0/125097215 shutdown_connections 2024-04-21T08:51:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.681+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/125097215 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d63a80 0x55bb88d65e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.681+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/125097215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5aea0 0x55bb88d63540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.681+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/125097215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5a580 0x55bb88d5a960 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.681+0000 7f1ad0a23740 1 -- 172.21.15.28:0/125097215 >> 172.21.15.28:0/125097215 conn(0x55bb88d54cb0 msgr2=0x55bb88d55090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.681+0000 7f1ad0a23740 1 -- 172.21.15.28:0/125097215 shutdown_connections 2024-04-21T08:51:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.681+0000 7f1ad0a23740 1 -- 172.21.15.28:0/125097215 wait complete. 2024-04-21T08:51:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.681+0000 7f1ad0a23740 1 -- start start 2024-04-21T08:51:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.681+0000 7f1ad0a23740 1 Processor -- start 2024-04-21T08:51:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ad0a23740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5a580 0x55bb88d60b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ad0a23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5aea0 0x55bb88d61090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ac6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5aea0 0x55bb88d61090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ac6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5aea0 0x55bb88d61090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42286/0 (socket says 172.21.15.28:42286) 2024-04-21T08:51:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ac7fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5a580 0x55bb88d60b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ac7fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5a580 0x55bb88d60b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:44150/0 (socket says 172.21.15.28:44150) 2024-04-21T08:51:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ac6ffd640 1 -- 172.21.15.28:0/4154448215 learned_addr learned my addr 172.21.15.28:0/4154448215 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/4154448215 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d63a80 0x55bb88d615d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4154448215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bb88d66810 con 0x55bb88d5aea0 2024-04-21T08:51:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4154448215 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55bb88c782f0 con 0x55bb88d63a80 2024-04-21T08:51:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4154448215 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55bb88d66990 con 0x55bb88d5a580 2024-04-21T08:51:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ac6ffd640 1 -- 172.21.15.28:0/4154448215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5a580 msgr2=0x55bb88d60b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ac6ffd640 1 --2- 172.21.15.28:0/4154448215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5a580 0x55bb88d60b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ac6ffd640 1 -- 172.21.15.28:0/4154448215 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d63a80 msgr2=0x55bb88d615d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:51:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ac6ffd640 1 --2- 172.21.15.28:0/4154448215 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d63a80 0x55bb88d615d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.682+0000 7f1ac6ffd640 1 -- 172.21.15.28:0/4154448215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb889d13f0 con 0x55bb88d5aea0 2024-04-21T08:51:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.683+0000 7f1ac6ffd640 1 --2- 172.21.15.28:0/4154448215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5aea0 0x55bb88d61090 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00b590 tx=0x7f1abc00ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b67d4a88dd9e8c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.683+0000 7f1aa6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.683+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/4154448215 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc023020 con 0x55bb88d5aea0 2024-04-21T08:51:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.683+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/4154448215 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1abc004460 con 0x55bb88d5aea0 2024-04-21T08:51:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.683+0000 7f1ad0a23740 10 client.0 _ll_register_callbacks cb 0x55bb88c85580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.683+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4154448215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ac0003e20 con 0x55bb88d5aea0 2024-04-21T08:51:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.683+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/4154448215 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1abc02e050 con 0x55bb88d5aea0 2024-04-21T08:51:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.683+0000 7f1ad0a23740 10 client.16050 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.683+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4154448215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=25}) -- 0x55bb889910f0 con 0x55bb88d5aea0 2024-04-21T08:51:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.684+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/4154448215 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1abc0035f0 con 0x55bb88d5aea0 2024-04-21T08:51:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.684+0000 7f1ad0a23740 10 client.16050 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.684+0000 7f1ad0a23740 10 client.16050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.684+0000 7f1ad0a23740 20 client.16050 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.684+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/4154448215 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55bb88d73820 0x55bb88d75ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.684+0000 7f1ad0a23740 4 client.16050 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.684+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4154448215 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1ac0003e20 con 0x55bb88d73820 2024-04-21T08:51:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.684+0000 7f1ac7fff640 1 --2- 172.21.15.28:0/4154448215 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55bb88d73820 0x55bb88d75ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.684+0000 7f1ac7fff640 1 --2- 172.21.15.28:0/4154448215 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55bb88d73820 0x55bb88d75ce0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7491866ea9dd2741 server_cookie=24301ae993adec6a in_seq=0 out_seq=0 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.684+0000 7f1aa6ffd640 10 client.16050 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.685+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/4154448215 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f1ac0003e20 con 0x55bb88d73820 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.685+0000 7f1aa6ffd640 10 client.16050 handle_command_reply: tid=0 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.685+0000 7f1ad0a23740 1 client.16050 shutdown 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.685+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4154448215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5aea0 msgr2=0x55bb88d61090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.685+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/4154448215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5aea0 0x55bb88d61090 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00b590 tx=0x7f1abc00ba50 comp rx=0 tx=0).stop 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.685+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4154448215 shutdown_connections 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.685+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/4154448215 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55bb88d73820 0x55bb88d75ce0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.685+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/4154448215 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d63a80 0x55bb88d615d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.685+0000 7f1aa6ffd640 0 client.16050 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.685+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/4154448215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5aea0 0x55bb88d61090 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.685+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/4154448215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5a580 0x55bb88d60b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.685+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4154448215 >> 172.21.15.28:0/4154448215 conn(0x55bb88d54cb0 msgr2=0x55bb88d56a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.686+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4154448215 shutdown_connections 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.686+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4154448215 wait complete. 2024-04-21T08:51:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.686+0000 7f1ad0a23740 20 client.16050 trim_cache size 0 max 16384 2024-04-21T08:51:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.720+0000 7f1ad0a23740 1 Processor -- start 2024-04-21T08:51:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.720+0000 7f1ad0a23740 1 -- start start 2024-04-21T08:51:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.720+0000 7f1ad0a23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5a580 0x55bb88d6f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.720+0000 7f1ad0a23740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5aea0 0x55bb88d5fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.720+0000 7f1ad0a23740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d63a80 0x55bb88d5c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.720+0000 7f1ad0a23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac81af920 con 0x55bb88d5a580 2024-04-21T08:51:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.720+0000 7f1ad0a23740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac81ad1f0 con 0x55bb88d63a80 2024-04-21T08:51:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.720+0000 7f1ad0a23740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac81ab040 con 0x55bb88d5aea0 2024-04-21T08:51:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.721+0000 7f1ac7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5a580 0x55bb88d6f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.721+0000 7f1ac7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5a580 0x55bb88d6f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42298/0 (socket says 172.21.15.28:42298) 2024-04-21T08:51:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.721+0000 7f1ac7fff640 1 -- 172.21.15.28:0/4231384189 learned_addr learned my addr 172.21.15.28:0/4231384189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.721+0000 7f1ac7fff640 1 --2- 172.21.15.28:0/4231384189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d63a80 0x55bb88d5c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.721+0000 7f1ac6ffd640 1 --2- 172.21.15.28:0/4231384189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5aea0 0x55bb88d5fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.721+0000 7f1ac7fff640 1 -- 172.21.15.28:0/4231384189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5aea0 msgr2=0x55bb88d5fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.721+0000 7f1ac7fff640 1 --2- 172.21.15.28:0/4231384189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5aea0 0x55bb88d5fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.721+0000 7f1ac7fff640 1 -- 172.21.15.28:0/4231384189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d63a80 msgr2=0x55bb88d5c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.721+0000 7f1ac7fff640 1 --2- 172.21.15.28:0/4231384189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d63a80 0x55bb88d5c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.721+0000 7f1ac7fff640 1 -- 172.21.15.28:0/4231384189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb88d55a40 con 0x55bb88d5a580 2024-04-21T08:51:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.722+0000 7f1ac7fff640 1 --2- 172.21.15.28:0/4231384189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5a580 0x55bb88d6f6d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0030680 tx=0x7f1ac00095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a5348233a3c76f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.722+0000 7f1aad7fa640 1 -- 172.21.15.28:0/4231384189 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0045040 con 0x55bb88d5a580 2024-04-21T08:51:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.722+0000 7f1aad7fa640 1 -- 172.21.15.28:0/4231384189 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ac0003e20 con 0x55bb88d5a580 2024-04-21T08:51:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.722+0000 7f1aad7fa640 1 -- 172.21.15.28:0/4231384189 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac00598a0 con 0x55bb88d5a580 2024-04-21T08:51:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.722+0000 7f1ac6ffd640 1 --2- 172.21.15.28:0/4231384189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5aea0 0x55bb88d5fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.723+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4231384189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5a580 msgr2=0x55bb88d6f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.723+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/4231384189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5a580 0x55bb88d6f6d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0030680 tx=0x7f1ac00095a0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.723+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4231384189 shutdown_connections 2024-04-21T08:51:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.723+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/4231384189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d63a80 0x55bb88d5c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.723+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/4231384189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d5aea0 0x55bb88d5fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.723+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/4231384189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5a580 0x55bb88d6f6d0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.723+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4231384189 >> 172.21.15.28:0/4231384189 conn(0x55bb88d54cb0 msgr2=0x55bb88d53d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.723+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4231384189 shutdown_connections 2024-04-21T08:51:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.723+0000 7f1ad0a23740 1 -- 172.21.15.28:0/4231384189 wait complete. 2024-04-21T08:51:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.723+0000 7f1ad0a23740 1 -- start start 2024-04-21T08:51:51.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.723+0000 7f1ad0a23740 1 Processor -- start 2024-04-21T08:51:51.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ad0a23740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d5a580 0x55bb88d67510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ad0a23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5aea0 0x55bb88d67a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ad0a23740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d63a80 0x55bb88d67f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ac6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5aea0 0x55bb88d67a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ac6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5aea0 0x55bb88d67a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42314/0 (socket says 172.21.15.28:42314) 2024-04-21T08:51:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ac6ffd640 1 -- 172.21.15.28:0/367385316 learned_addr learned my addr 172.21.15.28:0/367385316 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ad0a23740 1 -- 172.21.15.28:0/367385316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac81ab6b0 con 0x55bb88d5aea0 2024-04-21T08:51:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ac7fff640 1 --2- 172.21.15.28:0/367385316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d5a580 0x55bb88d67510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ad0a23740 1 -- 172.21.15.28:0/367385316 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac81acb20 con 0x55bb88d5a580 2024-04-21T08:51:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ad0a23740 1 -- 172.21.15.28:0/367385316 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac81abd80 con 0x55bb88d63a80 2024-04-21T08:51:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ad0a23740 10 client.0 _ll_register_callbacks cb 0x7f1abc0099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ac7fff640 1 -- 172.21.15.28:0/367385316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d63a80 msgr2=0x55bb88d67f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ac7fff640 1 --2- 172.21.15.28:0/367385316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d63a80 0x55bb88d67f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ac7fff640 1 -- 172.21.15.28:0/367385316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5aea0 msgr2=0x55bb88d67a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ac7fff640 1 --2- 172.21.15.28:0/367385316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5aea0 0x55bb88d67a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ac7fff640 1 -- 172.21.15.28:0/367385316 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb88dd8e60 con 0x55bb88d5a580 2024-04-21T08:51:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.724+0000 7f1ac6ffd640 1 --2- 172.21.15.28:0/367385316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5aea0 0x55bb88d67a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1ac7fff640 1 --2- 172.21.15.28:0/367385316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d5a580 0x55bb88d67510 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0037af0 tx=0x7f1ac0009360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17e49b48e38b9843 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1aa6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:51:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/367385316 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0068060 con 0x55bb88d5a580 2024-04-21T08:51:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1ad0a23740 1 -- 172.21.15.28:0/367385316 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ac0003e20 con 0x55bb88d5a580 2024-04-21T08:51:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/367385316 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1ac0010240 con 0x55bb88d5a580 2024-04-21T08:51:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/367385316 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1ac006f050 con 0x55bb88d5a580 2024-04-21T08:51:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1ad0a23740 10 client.25508 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1ad0a23740 1 -- 172.21.15.28:0/367385316 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bb88dda010 con 0x55bb88d5a580 2024-04-21T08:51:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/367385316 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ac0003e20 con 0x55bb88d5a580 2024-04-21T08:51:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/367385316 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1ac00376d0 con 0x55bb88d5a580 2024-04-21T08:51:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1ad0a23740 10 client.25508 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1ad0a23740 10 client.25508 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1ad0a23740 20 client.25508 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/367385316 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55bb88d73820 0x55bb88ddc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1ad0a23740 4 client.25508 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:51:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.725+0000 7f1ad0a23740 1 -- 172.21.15.28:0/367385316 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bb88cccc00 con 0x55bb88d73820 2024-04-21T08:51:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.726+0000 7f1ac6ffd640 1 --2- 172.21.15.28:0/367385316 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55bb88d73820 0x55bb88ddc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.726+0000 7f1aa6ffd640 10 client.25508 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.726+0000 7f1ac6ffd640 1 --2- 172.21.15.28:0/367385316 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55bb88d73820 0x55bb88ddc1f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=245fa517752540b server_cookie=87d5e096b5c245e1 in_seq=0 out_seq=0 2024-04-21T08:51:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.727+0000 7f1aa6ffd640 1 -- 172.21.15.28:0/367385316 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bb88cccc00 con 0x55bb88d73820 2024-04-21T08:51:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.727+0000 7f1aa6ffd640 10 client.25508 handle_command_reply: tid=0 2024-04-21T08:51:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.727+0000 7f1ad0a23740 1 client.25508 shutdown 2024-04-21T08:51:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.727+0000 7f1ad0a23740 1 -- 172.21.15.28:0/367385316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d5a580 msgr2=0x55bb88d67510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.727+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/367385316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d5a580 0x55bb88d67510 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0037af0 tx=0x7f1ac0009360 comp rx=0 tx=0).stop 2024-04-21T08:51:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.727+0000 7f1ad0a23740 1 -- 172.21.15.28:0/367385316 shutdown_connections 2024-04-21T08:51:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.727+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/367385316 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55bb88d73820 0x55bb88ddc1f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.727+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/367385316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55bb88d63a80 0x55bb88d67f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.727+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/367385316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb88d5aea0 0x55bb88d67a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.727+0000 7f1aa6ffd640 0 client.25508 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.727+0000 7f1ad0a23740 1 --2- 172.21.15.28:0/367385316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55bb88d5a580 0x55bb88d67510 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.727+0000 7f1ad0a23740 1 -- 172.21.15.28:0/367385316 >> 172.21.15.28:0/367385316 conn(0x55bb88d54cb0 msgr2=0x55bb88d55e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.727+0000 7f1ad0a23740 1 -- 172.21.15.28:0/367385316 shutdown_connections 2024-04-21T08:51:51.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.727+0000 7f1ad0a23740 1 -- 172.21.15.28:0/367385316 wait complete. 2024-04-21T08:51:51.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.727+0000 7f1ad0a23740 20 client.25508 trim_cache size 0 max 16384 2024-04-21T08:51:51.731 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:51:51.731 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T08:51:51.731 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T08:51:51.731 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:51:51.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.728+0000 7f1acde31640 1 -- 172.21.15.28:0/2118938307 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ab00a3ec0 msgr2=0x7f1ab00a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.728+0000 7f1acde31640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ab00a3ec0 0x7f1ab00a6380 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80ce500 tx=0x7f1ab8004a90 comp rx=0 tx=0).stop 2024-04-21T08:51:51.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.728+0000 7f1acde31640 1 -- 172.21.15.28:0/2118938307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac81031c0 msgr2=0x7f1ac81964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.728+0000 7f1acde31640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac81031c0 0x7f1ac81964d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1ab800e9f0 tx=0x7f1ab800edf0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.728+0000 7f1acde31640 1 -- 172.21.15.28:0/2118938307 shutdown_connections 2024-04-21T08:51:51.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.728+0000 7f1acde31640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ab00a3ec0 0x7f1ab00a6380 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.728+0000 7f1acde31640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ac810c500 0x7f1ac819ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.728+0000 7f1acde31640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ac8103ae0 0x7f1ac8196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.728+0000 7f1acde31640 1 --2- 172.21.15.28:0/2118938307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac81031c0 0x7f1ac81964d0 secure :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1ab800e9f0 tx=0x7f1ab800edf0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.728+0000 7f1acde31640 1 -- 172.21.15.28:0/2118938307 >> 172.21.15.28:0/2118938307 conn(0x7f1ac8078000 msgr2=0x7f1ac80fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.728+0000 7f1acde31640 1 -- 172.21.15.28:0/2118938307 shutdown_connections 2024-04-21T08:51:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.728+0000 7f1acde31640 1 -- 172.21.15.28:0/2118938307 wait complete. 2024-04-21T08:51:51.741 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:51:51.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:51:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.847+0000 7f78a8a14640 1 -- 172.21.15.28:0/798383502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78a41011a0 msgr2=0x7f78a4101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.848+0000 7f78a8a14640 1 --2- 172.21.15.28:0/798383502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78a41011a0 0x7f78a4101620 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7898009860 tx=0x7f789802bbd0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.848+0000 7f78a8a14640 1 -- 172.21.15.28:0/798383502 shutdown_connections 2024-04-21T08:51:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.848+0000 7f78a8a14640 1 --2- 172.21.15.28:0/798383502 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78a4101b60 0x7f78a410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.848+0000 7f78a8a14640 1 --2- 172.21.15.28:0/798383502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78a41011a0 0x7f78a4101620 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.848+0000 7f78a8a14640 1 --2- 172.21.15.28:0/798383502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78a4105ee0 0x7f78a41062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.848+0000 7f78a8a14640 1 -- 172.21.15.28:0/798383502 >> 172.21.15.28:0/798383502 conn(0x7f78a4077c20 msgr2=0x7f78a4078030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.848+0000 7f78a8a14640 1 -- 172.21.15.28:0/798383502 shutdown_connections 2024-04-21T08:51:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.848+0000 7f78a8a14640 1 -- 172.21.15.28:0/798383502 wait complete. 2024-04-21T08:51:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.849+0000 7f78a8a14640 1 Processor -- start 2024-04-21T08:51:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.849+0000 7f78a8a14640 1 -- start start 2024-04-21T08:51:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.849+0000 7f78a8a14640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78a41011a0 0x7f78a4197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a8a14640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78a4101b60 0x7f78a41979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a8a14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78a4105ee0 0x7f78a419bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a8a14640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41110a0 con 0x7f78a4105ee0 2024-04-21T08:51:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a8a14640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f78a4110f20 con 0x7f78a4101b60 2024-04-21T08:51:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a8a14640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f78a4111220 con 0x7f78a41011a0 2024-04-21T08:51:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78a4105ee0 0x7f78a419bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a1d9b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78a4101b60 0x7f78a41979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78a4105ee0 0x7f78a419bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42320/0 (socket says 172.21.15.28:42320) 2024-04-21T08:51:51.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a2d9d640 1 -- 172.21.15.28:0/3715741612 learned_addr learned my addr 172.21.15.28:0/3715741612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a259c640 1 --2- 172.21.15.28:0/3715741612 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78a41011a0 0x7f78a4197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a1d9b640 1 -- 172.21.15.28:0/3715741612 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78a41011a0 msgr2=0x7f78a4197460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a1d9b640 1 --2- 172.21.15.28:0/3715741612 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78a41011a0 0x7f78a4197460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a1d9b640 1 -- 172.21.15.28:0/3715741612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78a4105ee0 msgr2=0x7f78a419bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a1d9b640 1 --2- 172.21.15.28:0/3715741612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78a4105ee0 0x7f78a419bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a1d9b640 1 -- 172.21.15.28:0/3715741612 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a419c400 con 0x7f78a4101b60 2024-04-21T08:51:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.850+0000 7f78a2d9d640 1 --2- 172.21.15.28:0/3715741612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78a4105ee0 0x7f78a419bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.851+0000 7f78a1d9b640 1 --2- 172.21.15.28:0/3715741612 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78a4101b60 0x7f78a41979a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f789802c110 tx=0x7f7898035c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.851+0000 7f78937fe640 1 -- 172.21.15.28:0/3715741612 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7898039040 con 0x7f78a4101b60 2024-04-21T08:51:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.851+0000 7f78937fe640 1 -- 172.21.15.28:0/3715741612 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7898035e40 con 0x7f78a4101b60 2024-04-21T08:51:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.851+0000 7f78937fe640 1 -- 172.21.15.28:0/3715741612 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789803f440 con 0x7f78a4101b60 2024-04-21T08:51:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.851+0000 7f78a8a14640 1 -- 172.21.15.28:0/3715741612 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a419c690 con 0x7f78a4101b60 2024-04-21T08:51:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.851+0000 7f78a8a14640 1 -- 172.21.15.28:0/3715741612 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a419ca70 con 0x7f78a4101b60 2024-04-21T08:51:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.852+0000 7f78a8a14640 1 -- 172.21.15.28:0/3715741612 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a4102570 con 0x7f78a4101b60 2024-04-21T08:51:51.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.853+0000 7f78937fe640 1 -- 172.21.15.28:0/3715741612 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f7898046030 con 0x7f78a4101b60 2024-04-21T08:51:51.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.854+0000 7f78937fe640 1 --2- 172.21.15.28:0/3715741612 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f788c0a3ec0 0x7f788c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:51.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.854+0000 7f78937fe640 1 -- 172.21.15.28:0/3715741612 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f78980f21d0 con 0x7f78a4101b60 2024-04-21T08:51:51.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.854+0000 7f78a259c640 1 --2- 172.21.15.28:0/3715741612 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f788c0a3ec0 0x7f788c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:51.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.855+0000 7f78a259c640 1 --2- 172.21.15.28:0/3715741612 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f788c0a3ec0 0x7f788c0a6380 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7894005fd0 tx=0x7f7894009260 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:51.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:51.859+0000 7f78937fe640 1 -- 172.21.15.28:0/3715741612 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f789804d060 con 0x7f78a4101b60 2024-04-21T08:51:52.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.093+0000 7f78a8a14640 1 -- 172.21.15.28:0/3715741612 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78a4069230 con 0x7f78a4101b60 2024-04-21T08:51:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.094+0000 7f78937fe640 1 -- 172.21.15.28:0/3715741612 <== mon.2 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11882 (secure 0 0 0) 0x7f78980be420 con 0x7f78a4101b60 2024-04-21T08:51:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:51:52.094 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:51:09.181803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:51:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-04-21T08:51:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.097+0000 7f78a8a14640 1 -- 172.21.15.28:0/3715741612 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f788c0a3ec0 msgr2=0x7f788c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.097+0000 7f78a8a14640 1 --2- 172.21.15.28:0/3715741612 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f788c0a3ec0 0x7f788c0a6380 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7894005fd0 tx=0x7f7894009260 comp rx=0 tx=0).stop 2024-04-21T08:51:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.097+0000 7f78a8a14640 1 -- 172.21.15.28:0/3715741612 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78a4101b60 msgr2=0x7f78a41979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.097+0000 7f78a8a14640 1 --2- 172.21.15.28:0/3715741612 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78a4101b60 0x7f78a41979a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f789802c110 tx=0x7f7898035c50 comp rx=0 tx=0).stop 2024-04-21T08:51:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.097+0000 7f78a8a14640 1 -- 172.21.15.28:0/3715741612 shutdown_connections 2024-04-21T08:51:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.097+0000 7f78a8a14640 1 --2- 172.21.15.28:0/3715741612 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f788c0a3ec0 0x7f788c0a6380 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.097+0000 7f78a8a14640 1 --2- 172.21.15.28:0/3715741612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78a4105ee0 0x7f78a419bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.097+0000 7f78a8a14640 1 --2- 172.21.15.28:0/3715741612 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78a4101b60 0x7f78a41979a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.097+0000 7f78a8a14640 1 --2- 172.21.15.28:0/3715741612 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78a41011a0 0x7f78a4197460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.097+0000 7f78a8a14640 1 -- 172.21.15.28:0/3715741612 >> 172.21.15.28:0/3715741612 conn(0x7f78a4077c20 msgr2=0x7f78a40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.097+0000 7f78a8a14640 1 -- 172.21.15.28:0/3715741612 shutdown_connections 2024-04-21T08:51:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.097+0000 7f78a8a14640 1 -- 172.21.15.28:0/3715741612 wait complete. 2024-04-21T08:51:52.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:51 smithi028 ceph-mon[28003]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:51:52.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:51 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:51:52.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:51 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:52.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:51 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:51:52.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:51 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:52.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:51 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:51:52.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:51:52.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.217+0000 7fe46195f640 1 -- 172.21.15.28:0/3191301858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe45c1072e0 msgr2=0x7fe45c10eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.217+0000 7fe46195f640 1 --2- 172.21.15.28:0/3191301858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe45c1072e0 0x7fe45c10eeb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe454009ca0 tx=0x7fe45402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.217+0000 7fe46195f640 1 -- 172.21.15.28:0/3191301858 shutdown_connections 2024-04-21T08:51:52.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.217+0000 7fe46195f640 1 --2- 172.21.15.28:0/3191301858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe45c1072e0 0x7fe45c10eeb0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.217+0000 7fe46195f640 1 --2- 172.21.15.28:0/3191301858 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe45c102bc0 0x7fe45c106c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.217+0000 7fe46195f640 1 --2- 172.21.15.28:0/3191301858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe45c102210 0x7fe45c1025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.217+0000 7fe46195f640 1 -- 172.21.15.28:0/3191301858 >> 172.21.15.28:0/3191301858 conn(0x7fe45c0fdab0 msgr2=0x7fe45c0ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.217+0000 7fe46195f640 1 -- 172.21.15.28:0/3191301858 shutdown_connections 2024-04-21T08:51:52.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.217+0000 7fe46195f640 1 -- 172.21.15.28:0/3191301858 wait complete. 2024-04-21T08:51:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.218+0000 7fe46195f640 1 Processor -- start 2024-04-21T08:51:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.218+0000 7fe46195f640 1 -- start start 2024-04-21T08:51:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.218+0000 7fe46195f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe45c102210 0x7fe45c19e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.218+0000 7fe46195f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe45c1072e0 0x7fe45c19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.218+0000 7fe46195f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe45c198630 0x7fe45c198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.218+0000 7fe46195f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c112250 con 0x7fe45c198630 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.218+0000 7fe46195f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c1120d0 con 0x7fe45c1072e0 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.218+0000 7fe46195f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c1123d0 con 0x7fe45c102210 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe45a7fc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe45c1072e0 0x7fe45c19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe45b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe45c198630 0x7fe45c198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe45a7fc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe45c1072e0 0x7fe45c19e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:40188/0 (socket says 172.21.15.28:40188) 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe45a7fc640 1 -- 172.21.15.28:0/2795204114 learned_addr learned my addr 172.21.15.28:0/2795204114 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe45affd640 1 --2- 172.21.15.28:0/2795204114 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe45c102210 0x7fe45c19e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe45a7fc640 1 -- 172.21.15.28:0/2795204114 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe45c102210 msgr2=0x7fe45c19e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe45a7fc640 1 --2- 172.21.15.28:0/2795204114 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe45c102210 0x7fe45c19e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe45a7fc640 1 -- 172.21.15.28:0/2795204114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe45c198630 msgr2=0x7fe45c198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe45a7fc640 1 --2- 172.21.15.28:0/2795204114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe45c198630 0x7fe45c198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe45a7fc640 1 -- 172.21.15.28:0/2795204114 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe45c199150 con 0x7fe45c1072e0 2024-04-21T08:51:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe45b7fe640 1 --2- 172.21.15.28:0/2795204114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe45c198630 0x7fe45c198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe45affd640 1 --2- 172.21.15.28:0/2795204114 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe45c102210 0x7fe45c19e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe45a7fc640 1 --2- 172.21.15.28:0/2795204114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe45c1072e0 0x7fe45c19e970 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe45000e9a0 tx=0x7fe45000ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe46095d640 1 -- 172.21.15.28:0/2795204114 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45000cd80 con 0x7fe45c1072e0 2024-04-21T08:51:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe46095d640 1 -- 172.21.15.28:0/2795204114 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4500035f0 con 0x7fe45c1072e0 2024-04-21T08:51:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.219+0000 7fe46195f640 1 -- 172.21.15.28:0/2795204114 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe45c199440 con 0x7fe45c1072e0 2024-04-21T08:51:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.220+0000 7fe46195f640 1 -- 172.21.15.28:0/2795204114 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe45c074520 con 0x7fe45c1072e0 2024-04-21T08:51:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.220+0000 7fe46095d640 1 -- 172.21.15.28:0/2795204114 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe450010690 con 0x7fe45c1072e0 2024-04-21T08:51:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.222+0000 7fe46095d640 1 -- 172.21.15.28:0/2795204114 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fe450004000 con 0x7fe45c1072e0 2024-04-21T08:51:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.223+0000 7fe46095d640 1 --2- 172.21.15.28:0/2795204114 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4440a3ec0 0x7fe4440a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.223+0000 7fe46462a740 1 Processor -- start 2024-04-21T08:51:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.223+0000 7fe46095d640 1 -- 172.21.15.28:0/2795204114 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe4500d1030 con 0x7fe45c1072e0 2024-04-21T08:51:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.223+0000 7fe46462a740 1 -- start start 2024-04-21T08:51:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.223+0000 7fe45affd640 1 --2- 172.21.15.28:0/2795204114 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4440a3ec0 0x7fe4440a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe46462a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d7444d0 0x56229d7448b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe46462a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d744e80 0x56229d74d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe46462a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d74da00 0x56229d74fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe46462a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56229d6a91f0 con 0x56229d74da00 2024-04-21T08:51:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe46462a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56229d6464b0 con 0x56229d744e80 2024-04-21T08:51:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe46462a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56229d6a51b0 con 0x56229d7444d0 2024-04-21T08:51:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe45b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d74da00 0x56229d74fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe45a7fc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d744e80 0x56229d74d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe45a7fc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d744e80 0x56229d74d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:40200/0 (socket says 172.21.15.28:40200) 2024-04-21T08:51:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe45a7fc640 1 -- 172.21.15.28:0/3081765039 learned_addr learned my addr 172.21.15.28:0/3081765039 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe45affd640 1 --2- 172.21.15.28:0/3081765039 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d7444d0 0x56229d7448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe45affd640 1 --2- 172.21.15.28:0/2795204114 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4440a3ec0 0x7fe4440a6380 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe44c0058c0 tx=0x7fe44c005850 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe45a7fc640 1 -- 172.21.15.28:0/3081765039 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d7444d0 msgr2=0x56229d7448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe45a7fc640 1 --2- 172.21.15.28:0/3081765039 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d7444d0 0x56229d7448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe45a7fc640 1 -- 172.21.15.28:0/3081765039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d74da00 msgr2=0x56229d74fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe45a7fc640 1 --2- 172.21.15.28:0/3081765039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d74da00 0x56229d74fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe45a7fc640 1 -- 172.21.15.28:0/3081765039 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56229d6381b0 con 0x56229d744e80 2024-04-21T08:51:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.224+0000 7fe45b7fe640 1 --2- 172.21.15.28:0/3081765039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d74da00 0x56229d74fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.225+0000 7fe45a7fc640 1 --2- 172.21.15.28:0/3081765039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d744e80 0x56229d74d430 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe4500a5120 tx=0x7fe45009c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e04cbcb49ac2a1e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.225+0000 7fe441ffb640 1 -- 172.21.15.28:0/3081765039 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45009d0e0 con 0x56229d744e80 2024-04-21T08:51:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.225+0000 7fe441ffb640 1 -- 172.21.15.28:0/3081765039 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe45009c9e0 con 0x56229d744e80 2024-04-21T08:51:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.225+0000 7fe441ffb640 1 -- 172.21.15.28:0/3081765039 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe450099050 con 0x56229d744e80 2024-04-21T08:51:52.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.225+0000 7fe46462a740 1 -- 172.21.15.28:0/3081765039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d744e80 msgr2=0x56229d74d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.225+0000 7fe46462a740 1 --2- 172.21.15.28:0/3081765039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d744e80 0x56229d74d430 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe4500a5120 tx=0x7fe45009c520 comp rx=0 tx=0).stop 2024-04-21T08:51:52.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.225+0000 7fe46462a740 1 -- 172.21.15.28:0/3081765039 shutdown_connections 2024-04-21T08:51:52.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.225+0000 7fe46462a740 1 --2- 172.21.15.28:0/3081765039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d74da00 0x56229d74fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.225+0000 7fe46462a740 1 --2- 172.21.15.28:0/3081765039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d744e80 0x56229d74d430 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.226+0000 7fe46462a740 1 --2- 172.21.15.28:0/3081765039 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d7444d0 0x56229d7448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.226+0000 7fe46462a740 1 -- 172.21.15.28:0/3081765039 >> 172.21.15.28:0/3081765039 conn(0x56229d73eba0 msgr2=0x56229d73ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.226+0000 7fe46462a740 1 -- 172.21.15.28:0/3081765039 shutdown_connections 2024-04-21T08:51:52.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.226+0000 7fe46462a740 1 -- 172.21.15.28:0/3081765039 wait complete. 2024-04-21T08:51:52.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.226+0000 7fe46462a740 1 -- start start 2024-04-21T08:51:52.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.226+0000 7fe46462a740 1 Processor -- start 2024-04-21T08:51:52.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.226+0000 7fe46462a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d7444d0 0x56229d74d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.226+0000 7fe46462a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d744e80 0x56229d74a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.226+0000 7fe46462a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 0x56229d74ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.226+0000 7fe46462a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56229d750970 con 0x56229d7444d0 2024-04-21T08:51:52.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.226+0000 7fe46462a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56229d662500 con 0x56229d74da00 2024-04-21T08:51:52.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.226+0000 7fe46462a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56229d750af0 con 0x56229d744e80 2024-04-21T08:51:52.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe46462a740 10 client.0 _ll_register_callbacks cb 0x56229d630b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:52.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe45affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d7444d0 0x56229d74d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe45affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d7444d0 0x56229d74d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42342/0 (socket says 172.21.15.28:42342) 2024-04-21T08:51:52.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe45affd640 1 -- 172.21.15.28:0/689334365 learned_addr learned my addr 172.21.15.28:0/689334365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:52.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe45b7fe640 1 --2- 172.21.15.28:0/689334365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 0x56229d74ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe45a7fc640 1 --2- 172.21.15.28:0/689334365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d744e80 0x56229d74a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe45affd640 1 -- 172.21.15.28:0/689334365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d744e80 msgr2=0x56229d74a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe45affd640 1 --2- 172.21.15.28:0/689334365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d744e80 0x56229d74a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe45affd640 1 -- 172.21.15.28:0/689334365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 msgr2=0x56229d74ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe45affd640 1 --2- 172.21.15.28:0/689334365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 0x56229d74ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe45affd640 1 -- 172.21.15.28:0/689334365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56229d3bb3f0 con 0x56229d7444d0 2024-04-21T08:51:52.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe45a7fc640 1 --2- 172.21.15.28:0/689334365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d744e80 0x56229d74a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:52.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe45affd640 1 --2- 172.21.15.28:0/689334365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d7444d0 0x56229d74d370 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe44c01e670 tx=0x7fe44c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba4fd784b680778 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe43affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:52.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe43affd640 1 -- 172.21.15.28:0/689334365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44c011a50 con 0x56229d7444d0 2024-04-21T08:51:52.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.227+0000 7fe43affd640 1 -- 172.21.15.28:0/689334365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe44c011bf0 con 0x56229d7444d0 2024-04-21T08:51:52.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.228+0000 7fe46462a740 1 -- 172.21.15.28:0/689334365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe45009c9e0 con 0x56229d7444d0 2024-04-21T08:51:52.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.228+0000 7fe43affd640 1 -- 172.21.15.28:0/689334365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe44c031050 con 0x56229d7444d0 2024-04-21T08:51:52.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.228+0000 7fe46462a740 10 client.16074 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:52.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.228+0000 7fe46462a740 1 -- 172.21.15.28:0/689334365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56229d4857d0 con 0x56229d7444d0 2024-04-21T08:51:52.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.228+0000 7fe43affd640 1 -- 172.21.15.28:0/689334365 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe45009c9e0 con 0x56229d7444d0 2024-04-21T08:51:52.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.228+0000 7fe43affd640 1 -- 172.21.15.28:0/689334365 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe44c025030 con 0x56229d7444d0 2024-04-21T08:51:52.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.228+0000 7fe46462a740 10 client.16074 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:52.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.228+0000 7fe46462a740 10 client.16074 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:51:52.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.228+0000 7fe46462a740 20 client.16074 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:52.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.229+0000 7fe46462a740 1 --2- 172.21.15.28:0/689334365 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56229d75dcb0 0x56229d7600e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.229+0000 7fe46462a740 4 client.16074 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:52.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.229+0000 7fe46462a740 1 -- 172.21.15.28:0/689334365 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56229d648930 con 0x56229d75dcb0 2024-04-21T08:51:52.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.229+0000 7fe45a7fc640 1 --2- 172.21.15.28:0/689334365 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56229d75dcb0 0x56229d7600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.229+0000 7fe45a7fc640 1 --2- 172.21.15.28:0/689334365 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56229d75dcb0 0x56229d7600e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=24a5bcfb679fc7b server_cookie=7afbd3b36a21d2c8 in_seq=0 out_seq=0 2024-04-21T08:51:52.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.229+0000 7fe43affd640 10 client.16074 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:52.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.230+0000 7fe43affd640 1 -- 172.21.15.28:0/689334365 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56229d648930 con 0x56229d75dcb0 2024-04-21T08:51:52.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.230+0000 7fe43affd640 10 client.16074 handle_command_reply: tid=0 2024-04-21T08:51:52.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.230+0000 7fe46462a740 1 client.16074 shutdown 2024-04-21T08:51:52.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.230+0000 7fe46462a740 1 -- 172.21.15.28:0/689334365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d7444d0 msgr2=0x56229d74d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.230+0000 7fe46462a740 1 --2- 172.21.15.28:0/689334365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d7444d0 0x56229d74d370 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe44c01e670 tx=0x7fe44c0046c0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.231+0000 7fe46462a740 1 -- 172.21.15.28:0/689334365 shutdown_connections 2024-04-21T08:51:52.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.231+0000 7fe46462a740 1 --2- 172.21.15.28:0/689334365 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56229d75dcb0 0x56229d7600e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.231+0000 7fe46462a740 1 --2- 172.21.15.28:0/689334365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 0x56229d74ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.231+0000 7fe46462a740 1 --2- 172.21.15.28:0/689334365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d744e80 0x56229d74a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.231+0000 7fe46462a740 1 --2- 172.21.15.28:0/689334365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d7444d0 0x56229d74d370 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.231+0000 7fe46462a740 1 -- 172.21.15.28:0/689334365 >> 172.21.15.28:0/689334365 conn(0x56229d73eba0 msgr2=0x56229d7408f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.231+0000 7fe43affd640 0 client.16074 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:52.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.231+0000 7fe46462a740 1 -- 172.21.15.28:0/689334365 shutdown_connections 2024-04-21T08:51:52.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.231+0000 7fe46462a740 1 -- 172.21.15.28:0/689334365 wait complete. 2024-04-21T08:51:52.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.231+0000 7fe46462a740 20 client.16074 trim_cache size 0 max 16384 2024-04-21T08:51:52.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.257+0000 7fe46462a740 1 Processor -- start 2024-04-21T08:51:52.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.257+0000 7fe46462a740 1 -- start start 2024-04-21T08:51:52.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.257+0000 7fe46462a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d7444d0 0x56229d759830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.257+0000 7fe46462a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d744e80 0x56229d749860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.257+0000 7fe46462a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 0x56229d73da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.257+0000 7fe46462a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c1b3f10 con 0x56229d7444d0 2024-04-21T08:51:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.257+0000 7fe46462a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c1b17e0 con 0x56229d74da00 2024-04-21T08:51:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.257+0000 7fe46462a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c1af630 con 0x56229d744e80 2024-04-21T08:51:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.257+0000 7fe45affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d7444d0 0x56229d759830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.258+0000 7fe45affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d7444d0 0x56229d759830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42346/0 (socket says 172.21.15.28:42346) 2024-04-21T08:51:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.258+0000 7fe45affd640 1 -- 172.21.15.28:0/403527264 learned_addr learned my addr 172.21.15.28:0/403527264 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.258+0000 7fe45b7fe640 1 --2- 172.21.15.28:0/403527264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 0x56229d73da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.258+0000 7fe45a7fc640 1 --2- 172.21.15.28:0/403527264 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d744e80 0x56229d749860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.258+0000 7fe45b7fe640 1 -- 172.21.15.28:0/403527264 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d744e80 msgr2=0x56229d749860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.258+0000 7fe45b7fe640 1 --2- 172.21.15.28:0/403527264 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d744e80 0x56229d749860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.258+0000 7fe45b7fe640 1 -- 172.21.15.28:0/403527264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d7444d0 msgr2=0x56229d759830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.258+0000 7fe45b7fe640 1 --2- 172.21.15.28:0/403527264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d7444d0 0x56229d759830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.258+0000 7fe45b7fe640 1 -- 172.21.15.28:0/403527264 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56229d748c50 con 0x56229d74da00 2024-04-21T08:51:52.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.258+0000 7fe45affd640 1 --2- 172.21.15.28:0/403527264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d7444d0 0x56229d759830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:52.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.258+0000 7fe45b7fe640 1 --2- 172.21.15.28:0/403527264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 0x56229d73da80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe45402c700 tx=0x7fe454005c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=298e5b945c04245a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.258+0000 7fe4417fa640 1 -- 172.21.15.28:0/403527264 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe454045070 con 0x56229d74da00 2024-04-21T08:51:52.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.259+0000 7fe4417fa640 1 -- 172.21.15.28:0/403527264 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe454032970 con 0x56229d74da00 2024-04-21T08:51:52.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.259+0000 7fe4417fa640 1 -- 172.21.15.28:0/403527264 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe454039040 con 0x56229d74da00 2024-04-21T08:51:52.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.259+0000 7fe46462a740 1 -- 172.21.15.28:0/403527264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 msgr2=0x56229d73da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.259+0000 7fe46462a740 1 --2- 172.21.15.28:0/403527264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 0x56229d73da80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe45402c700 tx=0x7fe454005c50 comp rx=0 tx=0).stop 2024-04-21T08:51:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.259+0000 7fe46462a740 1 -- 172.21.15.28:0/403527264 shutdown_connections 2024-04-21T08:51:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.259+0000 7fe46462a740 1 --2- 172.21.15.28:0/403527264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 0x56229d73da80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.259+0000 7fe46462a740 1 --2- 172.21.15.28:0/403527264 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d744e80 0x56229d749860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.259+0000 7fe46462a740 1 --2- 172.21.15.28:0/403527264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d7444d0 0x56229d759830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.259+0000 7fe46462a740 1 -- 172.21.15.28:0/403527264 >> 172.21.15.28:0/403527264 conn(0x56229d73eba0 msgr2=0x56229d73d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.259+0000 7fe46462a740 1 -- 172.21.15.28:0/403527264 shutdown_connections 2024-04-21T08:51:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.259+0000 7fe46462a740 1 -- 172.21.15.28:0/403527264 wait complete. 2024-04-21T08:51:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.260+0000 7fe46462a740 1 -- start start 2024-04-21T08:51:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.260+0000 7fe46462a740 1 Processor -- start 2024-04-21T08:51:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe46462a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d7444d0 0x56229d755b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe46462a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d744e80 0x56229d756050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe46462a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 0x56229d7510f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe46462a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c1afca0 con 0x56229d744e80 2024-04-21T08:51:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe46462a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c1b1110 con 0x56229d74da00 2024-04-21T08:51:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe46462a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c1b0370 con 0x56229d7444d0 2024-04-21T08:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe45a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d744e80 0x56229d756050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe45a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d744e80 0x56229d756050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42356/0 (socket says 172.21.15.28:42356) 2024-04-21T08:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe45a7fc640 1 -- 172.21.15.28:0/277033661 learned_addr learned my addr 172.21.15.28:0/277033661 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe45affd640 1 --2- 172.21.15.28:0/277033661 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d7444d0 0x56229d755b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe46462a740 10 client.0 _ll_register_callbacks cb 0x7fe44c00be20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe45b7fe640 1 --2- 172.21.15.28:0/277033661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 0x56229d7510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe45a7fc640 1 -- 172.21.15.28:0/277033661 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d7444d0 msgr2=0x56229d755b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe45a7fc640 1 --2- 172.21.15.28:0/277033661 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d7444d0 0x56229d755b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe45a7fc640 1 -- 172.21.15.28:0/277033661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 msgr2=0x56229d7510f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe45a7fc640 1 --2- 172.21.15.28:0/277033661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 0x56229d7510f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe45a7fc640 1 -- 172.21.15.28:0/277033661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56229d751920 con 0x56229d744e80 2024-04-21T08:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe45affd640 1 --2- 172.21.15.28:0/277033661 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d7444d0 0x56229d755b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe45a7fc640 1 --2- 172.21.15.28:0/277033661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d744e80 0x56229d756050 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe45000c6a0 tx=0x7fe450000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884f871e1923c009 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.261+0000 7fe43affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.262+0000 7fe43affd640 1 -- 172.21.15.28:0/277033661 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45009d0e0 con 0x56229d744e80 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.262+0000 7fe43affd640 1 -- 172.21.15.28:0/277033661 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe4500037a0 con 0x56229d744e80 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.262+0000 7fe46462a740 1 -- 172.21.15.28:0/277033661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe454032970 con 0x56229d744e80 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.262+0000 7fe43affd640 1 -- 172.21.15.28:0/277033661 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe4500a5650 con 0x56229d744e80 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.262+0000 7fe46462a740 10 client.16080 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.262+0000 7fe46462a740 1 -- 172.21.15.28:0/277033661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56229d7c3cf0 con 0x56229d744e80 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.262+0000 7fe43affd640 1 -- 172.21.15.28:0/277033661 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe454032970 con 0x56229d744e80 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.262+0000 7fe43affd640 1 -- 172.21.15.28:0/277033661 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe4500a51c0 con 0x56229d744e80 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.262+0000 7fe46462a740 10 client.16080 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.262+0000 7fe46462a740 10 client.16080 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.262+0000 7fe46462a740 20 client.16080 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.262+0000 7fe46462a740 1 --2- 172.21.15.28:0/277033661 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56229d7c3f80 0x56229d7c6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.262+0000 7fe46462a740 4 client.16080 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.262+0000 7fe46462a740 1 -- 172.21.15.28:0/277033661 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56229d6b6c10 con 0x56229d7c3f80 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.263+0000 7fe45affd640 1 --2- 172.21.15.28:0/277033661 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56229d7c3f80 0x56229d7c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.263+0000 7fe43affd640 10 client.16080 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.263+0000 7fe45affd640 1 --2- 172.21.15.28:0/277033661 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56229d7c3f80 0x56229d7c6370 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e4f4fa8a0b08c6e1 server_cookie=97942998221f769c in_seq=0 out_seq=0 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.263+0000 7fe43affd640 1 -- 172.21.15.28:0/277033661 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56229d6b6c10 con 0x56229d7c3f80 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.263+0000 7fe43affd640 10 client.16080 handle_command_reply: tid=0 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.263+0000 7fe46462a740 1 client.16080 shutdown 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.264+0000 7fe46462a740 1 -- 172.21.15.28:0/277033661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d744e80 msgr2=0x56229d756050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.264+0000 7fe46462a740 1 --2- 172.21.15.28:0/277033661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d744e80 0x56229d756050 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fe45000c6a0 tx=0x7fe450000bc0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.264+0000 7fe46462a740 1 -- 172.21.15.28:0/277033661 shutdown_connections 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.264+0000 7fe46462a740 1 --2- 172.21.15.28:0/277033661 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x56229d7c3f80 0x56229d7c6370 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.264+0000 7fe46462a740 1 --2- 172.21.15.28:0/277033661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56229d74da00 0x56229d7510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.264+0000 7fe43affd640 0 client.16080 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.264+0000 7fe46462a740 1 --2- 172.21.15.28:0/277033661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56229d744e80 0x56229d756050 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.264+0000 7fe46462a740 1 --2- 172.21.15.28:0/277033661 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56229d7444d0 0x56229d755b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.264+0000 7fe46462a740 1 -- 172.21.15.28:0/277033661 >> 172.21.15.28:0/277033661 conn(0x56229d73eba0 msgr2=0x56229d73d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.264+0000 7fe46462a740 1 -- 172.21.15.28:0/277033661 shutdown_connections 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.264+0000 7fe46462a740 1 -- 172.21.15.28:0/277033661 wait complete. 2024-04-21T08:51:52.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.264+0000 7fe46462a740 20 client.16080 trim_cache size 0 max 16384 2024-04-21T08:51:52.269 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:51:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.265+0000 7fe46195f640 1 -- 172.21.15.28:0/2795204114 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4440a3ec0 msgr2=0x7fe4440a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.265+0000 7fe46195f640 1 --2- 172.21.15.28:0/2795204114 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4440a3ec0 0x7fe4440a6380 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe44c0058c0 tx=0x7fe44c005850 comp rx=0 tx=0).stop 2024-04-21T08:51:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.265+0000 7fe46195f640 1 -- 172.21.15.28:0/2795204114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe45c1072e0 msgr2=0x7fe45c19e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.265+0000 7fe46195f640 1 --2- 172.21.15.28:0/2795204114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe45c1072e0 0x7fe45c19e970 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe45000e9a0 tx=0x7fe45000ee60 comp rx=0 tx=0).stop 2024-04-21T08:51:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.265+0000 7fe46195f640 1 -- 172.21.15.28:0/2795204114 shutdown_connections 2024-04-21T08:51:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.265+0000 7fe46195f640 1 --2- 172.21.15.28:0/2795204114 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4440a3ec0 0x7fe4440a6380 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.265+0000 7fe46195f640 1 --2- 172.21.15.28:0/2795204114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe45c198630 0x7fe45c198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.265+0000 7fe46195f640 1 --2- 172.21.15.28:0/2795204114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe45c1072e0 0x7fe45c19e970 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.265+0000 7fe46195f640 1 --2- 172.21.15.28:0/2795204114 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe45c102210 0x7fe45c19e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.265+0000 7fe46195f640 1 -- 172.21.15.28:0/2795204114 >> 172.21.15.28:0/2795204114 conn(0x7fe45c0fdab0 msgr2=0x7fe45c0ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.265+0000 7fe46195f640 1 -- 172.21.15.28:0/2795204114 shutdown_connections 2024-04-21T08:51:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.265+0000 7fe46195f640 1 -- 172.21.15.28:0/2795204114 wait complete. 2024-04-21T08:51:52.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:51:52.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:51 smithi152 ceph-mon[31401]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:51:52.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:51 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:51:52.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:51 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:52.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:51 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:51:52.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:51 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:52.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:51 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:51:52.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:51 smithi171 ceph-mon[31358]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:51:52.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:51 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:51:52.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:51 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:52.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:51 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:51:52.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:51 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:52.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:51 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:51:52.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.389+0000 7f873ad2e640 1 -- 172.21.15.28:0/3674876334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734106160 msgr2=0x7f873410dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.389+0000 7f873ad2e640 1 --2- 172.21.15.28:0/3674876334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734106160 0x7f873410dd30 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f872c009ca0 tx=0x7f872c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.389+0000 7f873ad2e640 1 -- 172.21.15.28:0/3674876334 shutdown_connections 2024-04-21T08:51:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.389+0000 7f873ad2e640 1 --2- 172.21.15.28:0/3674876334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734106160 0x7f873410dd30 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.389+0000 7f873ad2e640 1 --2- 172.21.15.28:0/3674876334 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8734101a40 0x7f8734105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.389+0000 7f873ad2e640 1 --2- 172.21.15.28:0/3674876334 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8734101090 0x7f8734101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.389+0000 7f873ad2e640 1 -- 172.21.15.28:0/3674876334 >> 172.21.15.28:0/3674876334 conn(0x7f8734077f50 msgr2=0x7f8734078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.389+0000 7f873ad2e640 1 -- 172.21.15.28:0/3674876334 shutdown_connections 2024-04-21T08:51:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.389+0000 7f873ad2e640 1 -- 172.21.15.28:0/3674876334 wait complete. 2024-04-21T08:51:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.389+0000 7f873ad2e640 1 Processor -- start 2024-04-21T08:51:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f873ad2e640 1 -- start start 2024-04-21T08:51:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f873ad2e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101090 0x7f8734198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f873ad2e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8734106160 0x7f8734198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f873ad2e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f873419cfb0 0x7f873419d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f873ad2e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87341110d0 con 0x7f8734101090 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f873ad2e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8734110f50 con 0x7f873419cfb0 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f873ad2e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8734111250 con 0x7f8734106160 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f8738aca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101090 0x7f8734198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f8738aca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101090 0x7f8734198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42378/0 (socket says 172.21.15.28:42378) 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f8738aca640 1 -- 172.21.15.28:0/2865022281 learned_addr learned my addr 172.21.15.28:0/2865022281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f8733fff640 1 --2- 172.21.15.28:0/2865022281 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8734106160 0x7f8734198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f87392cb640 1 --2- 172.21.15.28:0/2865022281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f873419cfb0 0x7f873419d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f8738aca640 1 -- 172.21.15.28:0/2865022281 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8734106160 msgr2=0x7f8734198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f8738aca640 1 --2- 172.21.15.28:0/2865022281 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8734106160 0x7f8734198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f8738aca640 1 -- 172.21.15.28:0/2865022281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f873419cfb0 msgr2=0x7f873419d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f8738aca640 1 --2- 172.21.15.28:0/2865022281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f873419cfb0 0x7f873419d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.390+0000 7f8738aca640 1 -- 172.21.15.28:0/2865022281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f873419daa0 con 0x7f8734101090 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.391+0000 7f8738aca640 1 --2- 172.21.15.28:0/2865022281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101090 0x7f8734198740 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f872800e9f0 tx=0x7f872800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.391+0000 7f8731ffb640 1 -- 172.21.15.28:0/2865022281 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872800cd80 con 0x7f8734101090 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.391+0000 7f8731ffb640 1 -- 172.21.15.28:0/2865022281 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87280054d0 con 0x7f8734101090 2024-04-21T08:51:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.391+0000 7f8731ffb640 1 -- 172.21.15.28:0/2865022281 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8728010690 con 0x7f8734101090 2024-04-21T08:51:52.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.391+0000 7f873ad2e640 1 -- 172.21.15.28:0/2865022281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f873419dd90 con 0x7f8734101090 2024-04-21T08:51:52.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.391+0000 7f873ad2e640 1 -- 172.21.15.28:0/2865022281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87341a5580 con 0x7f8734101090 2024-04-21T08:51:52.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.394+0000 7f873d9f5740 1 Processor -- start 2024-04-21T08:51:52.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.394+0000 7f873d9f5740 1 -- start start 2024-04-21T08:51:52.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.394+0000 7f873d9f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935f370 0x55953935f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.394+0000 7f873d9f5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953935fd20 0x55953936c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.394+0000 7f873d9f5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953936ce10 0x55953936f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.394+0000 7f873d9f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5595392c41f0 con 0x55953935f370 2024-04-21T08:51:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.394+0000 7f873d9f5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5595392614b0 con 0x55953935fd20 2024-04-21T08:51:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.394+0000 7f873d9f5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5595392c01b0 con 0x55953936ce10 2024-04-21T08:51:52.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.394+0000 7f87392cb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935f370 0x55953935f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.394+0000 7f87392cb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935f370 0x55953935f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42386/0 (socket says 172.21.15.28:42386) 2024-04-21T08:51:52.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.394+0000 7f87392cb640 1 -- 172.21.15.28:0/859171948 learned_addr learned my addr 172.21.15.28:0/859171948 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:52.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.394+0000 7f8731ffb640 1 -- 172.21.15.28:0/2865022281 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f8728004e90 con 0x7f8734101090 2024-04-21T08:51:52.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.394+0000 7f8733fff640 1 --2- 172.21.15.28:0/859171948 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953936ce10 0x55953936f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.395+0000 7f87392cb640 1 -- 172.21.15.28:0/859171948 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953936ce10 msgr2=0x55953936f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.395+0000 7f87392cb640 1 --2- 172.21.15.28:0/859171948 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953936ce10 0x55953936f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.395+0000 7f87392cb640 1 -- 172.21.15.28:0/859171948 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953935fd20 msgr2=0x55953936c840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:51:52.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.395+0000 7f87392cb640 1 --2- 172.21.15.28:0/859171948 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953935fd20 0x55953936c840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.395+0000 7f87392cb640 1 -- 172.21.15.28:0/859171948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595392531b0 con 0x55953935f370 2024-04-21T08:51:52.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.395+0000 7f87392cb640 1 --2- 172.21.15.28:0/859171948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935f370 0x55953935f750 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f872c02c700 tx=0x7f872c032340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df00e03fe907984 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.395+0000 7f8731ffb640 1 --2- 172.21.15.28:0/2865022281 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f87240a3f40 0x7f87240a6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.395+0000 7f871effd640 1 -- 172.21.15.28:0/859171948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c043020 con 0x55953935f370 2024-04-21T08:51:52.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.395+0000 7f871effd640 1 -- 172.21.15.28:0/859171948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f872c032970 con 0x55953935f370 2024-04-21T08:51:52.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.395+0000 7f871effd640 1 -- 172.21.15.28:0/859171948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c039040 con 0x55953935f370 2024-04-21T08:51:52.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.395+0000 7f8731ffb640 1 -- 172.21.15.28:0/2865022281 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f87280d1300 con 0x7f8734101090 2024-04-21T08:51:52.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.396+0000 7f8738aca640 1 --2- 172.21.15.28:0/2865022281 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f87240a3f40 0x7f87240a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.397+0000 7f873d9f5740 1 -- 172.21.15.28:0/859171948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935f370 msgr2=0x55953935f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.397+0000 7f873d9f5740 1 --2- 172.21.15.28:0/859171948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935f370 0x55953935f750 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f872c02c700 tx=0x7f872c032340 comp rx=0 tx=0).stop 2024-04-21T08:51:52.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.397+0000 7f873d9f5740 1 -- 172.21.15.28:0/859171948 shutdown_connections 2024-04-21T08:51:52.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.397+0000 7f873d9f5740 1 --2- 172.21.15.28:0/859171948 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953936ce10 0x55953936f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.397+0000 7f873d9f5740 1 --2- 172.21.15.28:0/859171948 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953935fd20 0x55953936c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.397+0000 7f873d9f5740 1 --2- 172.21.15.28:0/859171948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935f370 0x55953935f750 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.397+0000 7f873d9f5740 1 -- 172.21.15.28:0/859171948 >> 172.21.15.28:0/859171948 conn(0x559539359ba0 msgr2=0x559539359f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.397+0000 7f8738aca640 1 --2- 172.21.15.28:0/2865022281 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f87240a3f40 0x7f87240a6400 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f87280ce530 tx=0x7f87280ce560 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.397+0000 7f873d9f5740 1 -- 172.21.15.28:0/859171948 shutdown_connections 2024-04-21T08:51:52.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.397+0000 7f873d9f5740 1 -- 172.21.15.28:0/859171948 wait complete. 2024-04-21T08:51:52.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.397+0000 7f873d9f5740 1 -- start start 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.397+0000 7f873d9f5740 1 Processor -- start 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f873d9f5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953935f370 0x55953936c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f873d9f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935fd20 0x559539369a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f873d9f5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953936ce10 0x559539369fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f873d9f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55953936fce0 con 0x55953935fd20 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f873d9f5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55953927d500 con 0x55953936ce10 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f873d9f5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55953936fe60 con 0x55953935f370 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f873d9f5740 10 client.0 _ll_register_callbacks cb 0x55953924bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f8733fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935fd20 0x559539369a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f8733fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935fd20 0x559539369a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42394/0 (socket says 172.21.15.28:42394) 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f8733fff640 1 -- 172.21.15.28:0/284450187 learned_addr learned my addr 172.21.15.28:0/284450187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f87392cb640 1 --2- 172.21.15.28:0/284450187 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953935f370 0x55953936c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f87392cb640 1 --2- 172.21.15.28:0/284450187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953936ce10 0x559539369fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f8733fff640 1 -- 172.21.15.28:0/284450187 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953935f370 msgr2=0x55953936c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f8733fff640 1 --2- 172.21.15.28:0/284450187 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953935f370 0x55953936c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f8733fff640 1 -- 172.21.15.28:0/284450187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953936ce10 msgr2=0x559539369fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f8733fff640 1 --2- 172.21.15.28:0/284450187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953936ce10 0x559539369fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f8733fff640 1 -- 172.21.15.28:0/284450187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559538fd63f0 con 0x55953935fd20 2024-04-21T08:51:52.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.398+0000 7f87392cb640 1 --2- 172.21.15.28:0/284450187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953936ce10 0x559539369fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:52.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.399+0000 7f8733fff640 1 --2- 172.21.15.28:0/284450187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935fd20 0x559539369a60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f872000b9f0 tx=0x7f872000beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6296101d49cd89f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.399+0000 7f8713fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:52.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.399+0000 7f8713fff640 1 -- 172.21.15.28:0/284450187 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8720003e20 con 0x55953935fd20 2024-04-21T08:51:52.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.399+0000 7f8713fff640 1 -- 172.21.15.28:0/284450187 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f8720005020 con 0x55953935fd20 2024-04-21T08:51:52.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.399+0000 7f873d9f5740 1 -- 172.21.15.28:0/284450187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f872c032970 con 0x55953935fd20 2024-04-21T08:51:52.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.399+0000 7f8713fff640 1 -- 172.21.15.28:0/284450187 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8720005220 con 0x55953935fd20 2024-04-21T08:51:52.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.399+0000 7f873d9f5740 10 client.16092 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:52.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.399+0000 7f873d9f5740 1 -- 172.21.15.28:0/284450187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5595390a07d0 con 0x55953935fd20 2024-04-21T08:51:52.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.399+0000 7f8713fff640 1 -- 172.21.15.28:0/284450187 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f872c032970 con 0x55953935fd20 2024-04-21T08:51:52.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.400+0000 7f8713fff640 1 -- 172.21.15.28:0/284450187 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f872001b630 con 0x55953935fd20 2024-04-21T08:51:52.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.400+0000 7f873d9f5740 10 client.16092 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:52.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.400+0000 7f873d9f5740 10 client.16092 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:51:52.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.400+0000 7f873d9f5740 20 client.16092 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:52.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.400+0000 7f873d9f5740 1 --2- 172.21.15.28:0/284450187 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55953937d030 0x55953937f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.400+0000 7f873d9f5740 4 client.16092 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:52.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.400+0000 7f873d9f5740 1 -- 172.21.15.28:0/284450187 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559539263930 con 0x55953937d030 2024-04-21T08:51:52.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.400+0000 7f87392cb640 1 --2- 172.21.15.28:0/284450187 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55953937d030 0x55953937f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.401+0000 7f8713fff640 10 client.16092 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:52.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.401+0000 7f87392cb640 1 --2- 172.21.15.28:0/284450187 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55953937d030 0x55953937f440 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=75bd67f1f4f2a2f8 server_cookie=88187658dba3d639 in_seq=0 out_seq=0 2024-04-21T08:51:52.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.401+0000 7f8713fff640 1 -- 172.21.15.28:0/284450187 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559539263930 con 0x55953937d030 2024-04-21T08:51:52.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.401+0000 7f8713fff640 10 client.16092 handle_command_reply: tid=0 2024-04-21T08:51:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.401+0000 7f873d9f5740 1 client.16092 shutdown 2024-04-21T08:51:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.402+0000 7f873d9f5740 1 -- 172.21.15.28:0/284450187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935fd20 msgr2=0x559539369a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.402+0000 7f873d9f5740 1 --2- 172.21.15.28:0/284450187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935fd20 0x559539369a60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f872000b9f0 tx=0x7f872000beb0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.402+0000 7f873d9f5740 1 -- 172.21.15.28:0/284450187 shutdown_connections 2024-04-21T08:51:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.402+0000 7f873d9f5740 1 --2- 172.21.15.28:0/284450187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953936ce10 0x559539369fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.402+0000 7f873d9f5740 1 --2- 172.21.15.28:0/284450187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935fd20 0x559539369a60 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.402+0000 7f873d9f5740 1 --2- 172.21.15.28:0/284450187 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55953937d030 0x55953937f440 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.402+0000 7f8713fff640 0 client.16092 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.402+0000 7f873d9f5740 1 --2- 172.21.15.28:0/284450187 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953935f370 0x55953936c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.402+0000 7f873d9f5740 1 -- 172.21.15.28:0/284450187 >> 172.21.15.28:0/284450187 conn(0x559539359ba0 msgr2=0x55953935b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.402+0000 7f873d9f5740 1 -- 172.21.15.28:0/284450187 shutdown_connections 2024-04-21T08:51:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.402+0000 7f873d9f5740 1 -- 172.21.15.28:0/284450187 wait complete. 2024-04-21T08:51:52.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.402+0000 7f873d9f5740 20 client.16092 trim_cache size 0 max 16384 2024-04-21T08:51:52.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.428+0000 7f873d9f5740 1 Processor -- start 2024-04-21T08:51:52.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.428+0000 7f873d9f5740 1 -- start start 2024-04-21T08:51:52.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f873d9f5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953935f370 0x559539378be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f873d9f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935fd20 0x559539364ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f873d9f5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953936ce10 0x55953936c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f873d9f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87341af9b0 con 0x55953935fd20 2024-04-21T08:51:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f873d9f5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f87341ad280 con 0x55953935f370 2024-04-21T08:51:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f873d9f5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f87341ab0d0 con 0x55953936ce10 2024-04-21T08:51:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f8733fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935fd20 0x559539364ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f8733fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935fd20 0x559539364ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42396/0 (socket says 172.21.15.28:42396) 2024-04-21T08:51:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f8733fff640 1 -- 172.21.15.28:0/3999672450 learned_addr learned my addr 172.21.15.28:0/3999672450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f87392cb640 1 --2- 172.21.15.28:0/3999672450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953935f370 0x559539378be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f87392cb640 1 --2- 172.21.15.28:0/3999672450 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953936ce10 0x55953936c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f8733fff640 1 -- 172.21.15.28:0/3999672450 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953936ce10 msgr2=0x55953936c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f8733fff640 1 --2- 172.21.15.28:0/3999672450 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953936ce10 0x55953936c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f8733fff640 1 -- 172.21.15.28:0/3999672450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953935f370 msgr2=0x559539378be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f8733fff640 1 --2- 172.21.15.28:0/3999672450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953935f370 0x559539378be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f8733fff640 1 -- 172.21.15.28:0/3999672450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559539367fb0 con 0x55953935fd20 2024-04-21T08:51:52.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.429+0000 7f8733fff640 1 --2- 172.21.15.28:0/3999672450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935fd20 0x559539364ef0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8720001eb0 tx=0x7f8720003e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f9b73064ef383c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.430+0000 7f871e7fc640 1 -- 172.21.15.28:0/3999672450 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872002c020 con 0x55953935fd20 2024-04-21T08:51:52.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.430+0000 7f871e7fc640 1 -- 172.21.15.28:0/3999672450 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87200053c0 con 0x55953935fd20 2024-04-21T08:51:52.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.430+0000 7f871e7fc640 1 -- 172.21.15.28:0/3999672450 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8720004ca0 con 0x55953935fd20 2024-04-21T08:51:52.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.430+0000 7f873d9f5740 1 -- 172.21.15.28:0/3999672450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935fd20 msgr2=0x559539364ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.430+0000 7f873d9f5740 1 --2- 172.21.15.28:0/3999672450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935fd20 0x559539364ef0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8720001eb0 tx=0x7f8720003e20 comp rx=0 tx=0).stop 2024-04-21T08:51:52.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.430+0000 7f873d9f5740 1 -- 172.21.15.28:0/3999672450 shutdown_connections 2024-04-21T08:51:52.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.430+0000 7f873d9f5740 1 --2- 172.21.15.28:0/3999672450 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953936ce10 0x55953936c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.430+0000 7f873d9f5740 1 --2- 172.21.15.28:0/3999672450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935fd20 0x559539364ef0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.430+0000 7f873d9f5740 1 --2- 172.21.15.28:0/3999672450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953935f370 0x559539378be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.430+0000 7f873d9f5740 1 -- 172.21.15.28:0/3999672450 >> 172.21.15.28:0/3999672450 conn(0x559539359ba0 msgr2=0x5595393647e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.431+0000 7f873d9f5740 1 -- 172.21.15.28:0/3999672450 shutdown_connections 2024-04-21T08:51:52.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.431+0000 7f873d9f5740 1 -- 172.21.15.28:0/3999672450 wait complete. 2024-04-21T08:51:52.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.431+0000 7f873d9f5740 1 -- start start 2024-04-21T08:51:52.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.431+0000 7f873d9f5740 1 Processor -- start 2024-04-21T08:51:52.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.431+0000 7f873d9f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935f370 0x5595393754f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.431+0000 7f873d9f5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953935fd20 0x559539370860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.431+0000 7f873d9f5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953936ce10 0x559539370da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.431+0000 7f873d9f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87341ab740 con 0x55953935f370 2024-04-21T08:51:52.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.431+0000 7f873d9f5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f87341acbb0 con 0x55953935fd20 2024-04-21T08:51:52.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.431+0000 7f873d9f5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f87341abe10 con 0x55953936ce10 2024-04-21T08:51:52.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.431+0000 7f873d9f5740 10 client.0 _ll_register_callbacks cb 0x7f87200044a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:52.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.432+0000 7f87392cb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935f370 0x5595393754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.432+0000 7f8733fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953935fd20 0x559539370860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.432+0000 7f87392cb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935f370 0x5595393754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42404/0 (socket says 172.21.15.28:42404) 2024-04-21T08:51:52.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.432+0000 7f87392cb640 1 -- 172.21.15.28:0/2602761680 learned_addr learned my addr 172.21.15.28:0/2602761680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.432+0000 7f87392cb640 1 -- 172.21.15.28:0/2602761680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953936ce10 msgr2=0x559539370da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.432+0000 7f87392cb640 1 --2- 172.21.15.28:0/2602761680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953936ce10 0x559539370da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.432+0000 7f87392cb640 1 -- 172.21.15.28:0/2602761680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953935fd20 msgr2=0x559539370860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.432+0000 7f87392cb640 1 --2- 172.21.15.28:0/2602761680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953935fd20 0x559539370860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.432+0000 7f87392cb640 1 -- 172.21.15.28:0/2602761680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595393715d0 con 0x55953935f370 2024-04-21T08:51:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.432+0000 7f87392cb640 1 --2- 172.21.15.28:0/2602761680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935f370 0x5595393754f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f872c009430 tx=0x7f872c02c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb392cf5c08c62e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.432+0000 7f8713fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.432+0000 7f8713fff640 1 -- 172.21.15.28:0/2602761680 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872c054020 con 0x55953935f370 2024-04-21T08:51:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.432+0000 7f8713fff640 1 -- 172.21.15.28:0/2602761680 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f872c0324c0 con 0x55953935f370 2024-04-21T08:51:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.432+0000 7f873d9f5740 1 -- 172.21.15.28:0/2602761680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87200053c0 con 0x55953935f370 2024-04-21T08:51:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.433+0000 7f8713fff640 1 -- 172.21.15.28:0/2602761680 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f872c05f050 con 0x55953935f370 2024-04-21T08:51:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.433+0000 7f873d9f5740 10 client.16098 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.433+0000 7f873d9f5740 1 -- 172.21.15.28:0/2602761680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5595393e35e0 con 0x55953935f370 2024-04-21T08:51:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.433+0000 7f8713fff640 1 -- 172.21.15.28:0/2602761680 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f872c005cb0 con 0x55953935f370 2024-04-21T08:51:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.433+0000 7f8713fff640 1 -- 172.21.15.28:0/2602761680 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f872c03fc20 con 0x55953935f370 2024-04-21T08:51:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.433+0000 7f873d9f5740 10 client.16098 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.433+0000 7f873d9f5740 10 client.16098 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:51:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.433+0000 7f873d9f5740 20 client.16098 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.433+0000 7f873d9f5740 1 --2- 172.21.15.28:0/2602761680 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55953937d030 0x5595393e5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.433+0000 7f873d9f5740 4 client.16098 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:51:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.433+0000 7f873d9f5740 1 -- 172.21.15.28:0/2602761680 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f87200053c0 con 0x55953937d030 2024-04-21T08:51:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.433+0000 7f8733fff640 1 --2- 172.21.15.28:0/2602761680 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55953937d030 0x5595393e5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.434+0000 7f8713fff640 10 client.16098 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.434+0000 7f8733fff640 1 --2- 172.21.15.28:0/2602761680 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55953937d030 0x5595393e5980 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=835c00260cadcf45 server_cookie=cedbc77c429f8284 in_seq=0 out_seq=0 2024-04-21T08:51:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.434+0000 7f8713fff640 1 -- 172.21.15.28:0/2602761680 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f87200053c0 con 0x55953937d030 2024-04-21T08:51:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.434+0000 7f8713fff640 10 client.16098 handle_command_reply: tid=0 2024-04-21T08:51:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.434+0000 7f873d9f5740 1 client.16098 shutdown 2024-04-21T08:51:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.434+0000 7f873d9f5740 1 -- 172.21.15.28:0/2602761680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935f370 msgr2=0x5595393754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.434+0000 7f873d9f5740 1 --2- 172.21.15.28:0/2602761680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935f370 0x5595393754f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f872c009430 tx=0x7f872c02c7f0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.434+0000 7f873d9f5740 1 -- 172.21.15.28:0/2602761680 shutdown_connections 2024-04-21T08:51:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.434+0000 7f873d9f5740 1 --2- 172.21.15.28:0/2602761680 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55953937d030 0x5595393e5980 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.434+0000 7f8713fff640 0 client.16098 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:51:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.434+0000 7f873d9f5740 1 --2- 172.21.15.28:0/2602761680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55953936ce10 0x559539370da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.434+0000 7f873d9f5740 1 --2- 172.21.15.28:0/2602761680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55953935fd20 0x559539370860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.434+0000 7f873d9f5740 1 --2- 172.21.15.28:0/2602761680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55953935f370 0x5595393754f0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.434+0000 7f873d9f5740 1 -- 172.21.15.28:0/2602761680 >> 172.21.15.28:0/2602761680 conn(0x559539359ba0 msgr2=0x55953935b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.434+0000 7f873d9f5740 1 -- 172.21.15.28:0/2602761680 shutdown_connections 2024-04-21T08:51:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.435+0000 7f873d9f5740 1 -- 172.21.15.28:0/2602761680 wait complete. 2024-04-21T08:51:52.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.435+0000 7f873d9f5740 20 client.16098 trim_cache size 0 max 16384 2024-04-21T08:51:52.440 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:51:52.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.435+0000 7f873ad2e640 1 -- 172.21.15.28:0/2865022281 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f87240a3f40 msgr2=0x7f87240a6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.435+0000 7f873ad2e640 1 --2- 172.21.15.28:0/2865022281 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f87240a3f40 0x7f87240a6400 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f87280ce530 tx=0x7f87280ce560 comp rx=0 tx=0).stop 2024-04-21T08:51:52.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.435+0000 7f873ad2e640 1 -- 172.21.15.28:0/2865022281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101090 msgr2=0x7f8734198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.435+0000 7f873ad2e640 1 --2- 172.21.15.28:0/2865022281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101090 0x7f8734198740 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f872800e9f0 tx=0x7f872800edf0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.435+0000 7f873ad2e640 1 -- 172.21.15.28:0/2865022281 shutdown_connections 2024-04-21T08:51:52.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.435+0000 7f873ad2e640 1 --2- 172.21.15.28:0/2865022281 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f87240a3f40 0x7f87240a6400 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.435+0000 7f873ad2e640 1 --2- 172.21.15.28:0/2865022281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f873419cfb0 0x7f873419d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.435+0000 7f873ad2e640 1 --2- 172.21.15.28:0/2865022281 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8734106160 0x7f8734198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.435+0000 7f873ad2e640 1 --2- 172.21.15.28:0/2865022281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8734101090 0x7f8734198740 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.435+0000 7f873ad2e640 1 -- 172.21.15.28:0/2865022281 >> 172.21.15.28:0/2865022281 conn(0x7f8734077f50 msgr2=0x7f8734104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.436+0000 7f873ad2e640 1 -- 172.21.15.28:0/2865022281 shutdown_connections 2024-04-21T08:51:52.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.436+0000 7f873ad2e640 1 -- 172.21.15.28:0/2865022281 wait complete. 2024-04-21T08:51:52.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:51:52.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.554+0000 7f7844a06640 1 -- 172.21.15.28:0/2561224469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f784010e960 msgr2=0x7f7840110d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.554+0000 7f7844a06640 1 --2- 172.21.15.28:0/2561224469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f784010e960 0x7f7840110d50 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7838009ca0 tx=0x7f783802c100 comp rx=0 tx=0).stop 2024-04-21T08:51:52.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.554+0000 7f7844a06640 1 -- 172.21.15.28:0/2561224469 shutdown_connections 2024-04-21T08:51:52.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.554+0000 7f7844a06640 1 --2- 172.21.15.28:0/2561224469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f784010e960 0x7f7840110d50 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.554+0000 7f7844a06640 1 --2- 172.21.15.28:0/2561224469 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f78401016d0 0x7f784010e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.554+0000 7f7844a06640 1 --2- 172.21.15.28:0/2561224469 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7840100db0 0x7f7840101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.554+0000 7f7844a06640 1 -- 172.21.15.28:0/2561224469 >> 172.21.15.28:0/2561224469 conn(0x7f7840077fb0 msgr2=0x7f78400783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.554+0000 7f7844a06640 1 -- 172.21.15.28:0/2561224469 shutdown_connections 2024-04-21T08:51:52.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.554+0000 7f7844a06640 1 -- 172.21.15.28:0/2561224469 wait complete. 2024-04-21T08:51:52.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.555+0000 7f7844a06640 1 Processor -- start 2024-04-21T08:51:52.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.555+0000 7f7844a06640 1 -- start start 2024-04-21T08:51:52.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f7844a06640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7840100db0 0x7f78401a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f7844a06640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78401016d0 0x7f78401a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f7844a06640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f784010e960 0x7f784019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f7844a06640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78401143b0 con 0x7f7840100db0 2024-04-21T08:51:52.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f7844a06640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7840114230 con 0x7f78401016d0 2024-04-21T08:51:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f783e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7840100db0 0x7f78401a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f7844a06640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7840114530 con 0x7f784010e960 2024-04-21T08:51:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f783e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7840100db0 0x7f78401a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42414/0 (socket says 172.21.15.28:42414) 2024-04-21T08:51:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f783e59c640 1 -- 172.21.15.28:0/3274120991 learned_addr learned my addr 172.21.15.28:0/3274120991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f783ed9d640 1 --2- 172.21.15.28:0/3274120991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f784010e960 0x7f784019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f783dd9b640 1 --2- 172.21.15.28:0/3274120991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78401016d0 0x7f78401a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f783e59c640 1 -- 172.21.15.28:0/3274120991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f784010e960 msgr2=0x7f784019a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f783e59c640 1 --2- 172.21.15.28:0/3274120991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f784010e960 0x7f784019a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f783e59c640 1 -- 172.21.15.28:0/3274120991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78401016d0 msgr2=0x7f78401a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f783e59c640 1 --2- 172.21.15.28:0/3274120991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78401016d0 0x7f78401a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f783e59c640 1 -- 172.21.15.28:0/3274120991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f784019aff0 con 0x7f7840100db0 2024-04-21T08:51:52.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.556+0000 7f783e59c640 1 --2- 172.21.15.28:0/3274120991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7840100db0 0x7f78401a0670 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f783000bea0 tx=0x7f783000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.557+0000 7f782f7fe640 1 -- 172.21.15.28:0/3274120991 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783001c070 con 0x7f7840100db0 2024-04-21T08:51:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.557+0000 7f782f7fe640 1 -- 172.21.15.28:0/3274120991 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7830009cd0 con 0x7f7840100db0 2024-04-21T08:51:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.557+0000 7f782f7fe640 1 -- 172.21.15.28:0/3274120991 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783001fc90 con 0x7f7840100db0 2024-04-21T08:51:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.557+0000 7f7844a06640 1 -- 172.21.15.28:0/3274120991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f784019b280 con 0x7f7840100db0 2024-04-21T08:51:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.557+0000 7f7844a06640 1 -- 172.21.15.28:0/3274120991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7840102fc0 con 0x7f7840100db0 2024-04-21T08:51:52.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.560+0000 7f782f7fe640 1 -- 172.21.15.28:0/3274120991 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f783001f3b0 con 0x7f7840100db0 2024-04-21T08:51:52.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.560+0000 7f78475f8740 1 Processor -- start 2024-04-21T08:51:52.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f78475f8740 1 -- start start 2024-04-21T08:51:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f782f7fe640 1 --2- 172.21.15.28:0/3274120991 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f78280a3ec0 0x7f78280a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f782f7fe640 1 -- 172.21.15.28:0/3274120991 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f78300d8f20 con 0x7f7840100db0 2024-04-21T08:51:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f78475f8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1257494d0 0x55f1257498b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f78475f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125749e80 0x55f125752430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f78475f8740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f125752a00 0x55f125754e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f78475f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f1256ae1f0 con 0x55f125749e80 2024-04-21T08:51:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f78475f8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f12564b4b0 con 0x55f1257494d0 2024-04-21T08:51:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f78475f8740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f1256aa1b0 con 0x55f125752a00 2024-04-21T08:51:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f783ed9d640 1 --2- 172.21.15.28:0/3274120991 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f78280a3ec0 0x7f78280a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f783e59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1257494d0 0x55f1257498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f783e59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1257494d0 0x55f1257498b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:40300/0 (socket says 172.21.15.28:40300) 2024-04-21T08:51:52.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f783e59c640 1 -- 172.21.15.28:0/2551661070 learned_addr learned my addr 172.21.15.28:0/2551661070 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:52.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f783dd9b640 1 --2- 172.21.15.28:0/2551661070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125749e80 0x55f125752430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f783dd9b640 1 -- 172.21.15.28:0/2551661070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f125752a00 msgr2=0x55f125754e50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:51:52.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f783dd9b640 1 --2- 172.21.15.28:0/2551661070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f125752a00 0x55f125754e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f783dd9b640 1 -- 172.21.15.28:0/2551661070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1257494d0 msgr2=0x55f1257498b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f783dd9b640 1 --2- 172.21.15.28:0/2551661070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1257494d0 0x55f1257498b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.561+0000 7f783dd9b640 1 -- 172.21.15.28:0/2551661070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f12563d1b0 con 0x55f125749e80 2024-04-21T08:51:52.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.562+0000 7f783ed9d640 1 --2- 172.21.15.28:0/3274120991 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f78280a3ec0 0x7f78280a6380 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7838009bf0 tx=0x7f7838005d80 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.562+0000 7f783dd9b640 1 --2- 172.21.15.28:0/2551661070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125749e80 0x55f125752430 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f783400bb30 tx=0x7f783400f360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ba20ac6686c3d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.562+0000 7f782cff9640 1 -- 172.21.15.28:0/2551661070 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7834015070 con 0x55f125749e80 2024-04-21T08:51:52.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.562+0000 7f782cff9640 1 -- 172.21.15.28:0/2551661070 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f783400fd90 con 0x55f125749e80 2024-04-21T08:51:52.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.562+0000 7f782cff9640 1 -- 172.21.15.28:0/2551661070 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7834011440 con 0x55f125749e80 2024-04-21T08:51:52.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.563+0000 7f78475f8740 1 -- 172.21.15.28:0/2551661070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125749e80 msgr2=0x55f125752430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.563+0000 7f78475f8740 1 --2- 172.21.15.28:0/2551661070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125749e80 0x55f125752430 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f783400bb30 tx=0x7f783400f360 comp rx=0 tx=0).stop 2024-04-21T08:51:52.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.563+0000 7f78475f8740 1 -- 172.21.15.28:0/2551661070 shutdown_connections 2024-04-21T08:51:52.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.563+0000 7f78475f8740 1 --2- 172.21.15.28:0/2551661070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f125752a00 0x55f125754e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.563+0000 7f78475f8740 1 --2- 172.21.15.28:0/2551661070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125749e80 0x55f125752430 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.563+0000 7f78475f8740 1 --2- 172.21.15.28:0/2551661070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1257494d0 0x55f1257498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.563+0000 7f78475f8740 1 -- 172.21.15.28:0/2551661070 >> 172.21.15.28:0/2551661070 conn(0x55f125743ba0 msgr2=0x55f125743f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.563+0000 7f78475f8740 1 -- 172.21.15.28:0/2551661070 shutdown_connections 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.563+0000 7f78475f8740 1 -- 172.21.15.28:0/2551661070 wait complete. 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.563+0000 7f78475f8740 1 -- start start 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.563+0000 7f78475f8740 1 Processor -- start 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.563+0000 7f78475f8740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1257494d0 0x55f1257522f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f78475f8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125749e80 0x55f12574f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f78475f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125752a00 0x55f12574fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f78475f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f125755970 con 0x55f125752a00 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f78475f8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f125667500 con 0x55f125749e80 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f78475f8740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f125755af0 con 0x55f1257494d0 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f78475f8740 10 client.0 _ll_register_callbacks cb 0x55f125635b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f783e59c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1257494d0 0x55f1257522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f783e59c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1257494d0 0x55f1257522f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:44302/0 (socket says 172.21.15.28:44302) 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f783e59c640 1 -- 172.21.15.28:0/3146766473 learned_addr learned my addr 172.21.15.28:0/3146766473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f783dd9b640 1 --2- 172.21.15.28:0/3146766473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125749e80 0x55f12574f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f783ed9d640 1 --2- 172.21.15.28:0/3146766473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125752a00 0x55f12574fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f783e59c640 1 -- 172.21.15.28:0/3146766473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125749e80 msgr2=0x55f12574f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f783e59c640 1 --2- 172.21.15.28:0/3146766473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125749e80 0x55f12574f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f783e59c640 1 -- 172.21.15.28:0/3146766473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125752a00 msgr2=0x55f12574fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f783e59c640 1 --2- 172.21.15.28:0/3146766473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125752a00 0x55f12574fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f783e59c640 1 -- 172.21.15.28:0/3146766473 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1253c03f0 con 0x55f1257494d0 2024-04-21T08:51:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f783ed9d640 1 --2- 172.21.15.28:0/3146766473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125752a00 0x55f12574fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f783e59c640 1 --2- 172.21.15.28:0/3146766473 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1257494d0 0x55f1257522f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f78300d6370 tx=0x7f78300c5190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9369a151aa4a7f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.564+0000 7f7825ffb640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:51:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.565+0000 7f7825ffb640 1 -- 172.21.15.28:0/3146766473 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78300a8d90 con 0x55f1257494d0 2024-04-21T08:51:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.565+0000 7f78475f8740 1 -- 172.21.15.28:0/3146766473 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f783400fd90 con 0x55f1257494d0 2024-04-21T08:51:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.565+0000 7f7825ffb640 1 -- 172.21.15.28:0/3146766473 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f78300c37e0 con 0x55f1257494d0 2024-04-21T08:51:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.565+0000 7f7825ffb640 1 -- 172.21.15.28:0/3146766473 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7830088050 con 0x55f1257494d0 2024-04-21T08:51:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.565+0000 7f78475f8740 10 client.25534 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.565+0000 7f78475f8740 1 -- 172.21.15.28:0/3146766473 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f12548a7d0 con 0x55f1257494d0 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.565+0000 7f7825ffb640 1 -- 172.21.15.28:0/3146766473 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f783400fd90 con 0x55f1257494d0 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.565+0000 7f7825ffb640 1 -- 172.21.15.28:0/3146766473 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f78300d6430 con 0x55f1257494d0 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.565+0000 7f78475f8740 10 client.25534 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.565+0000 7f78475f8740 10 client.25534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.565+0000 7f78475f8740 20 client.25534 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.565+0000 7f78475f8740 1 --2- 172.21.15.28:0/3146766473 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f125763060 0x55f1257654b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.565+0000 7f78475f8740 4 client.25534 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.565+0000 7f78475f8740 1 -- 172.21.15.28:0/3146766473 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f12564d930 con 0x55f125763060 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.566+0000 7f783dd9b640 1 --2- 172.21.15.28:0/3146766473 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f125763060 0x55f1257654b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.566+0000 7f783dd9b640 1 --2- 172.21.15.28:0/3146766473 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f125763060 0x55f1257654b0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=549626fac95b12eb server_cookie=bec2e32cde97ffa in_seq=0 out_seq=0 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.566+0000 7f7825ffb640 10 client.25534 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.567+0000 7f7825ffb640 1 -- 172.21.15.28:0/3146766473 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f12564d930 con 0x55f125763060 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.567+0000 7f7825ffb640 10 client.25534 handle_command_reply: tid=0 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.567+0000 7f78475f8740 1 client.25534 shutdown 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.567+0000 7f78475f8740 1 -- 172.21.15.28:0/3146766473 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1257494d0 msgr2=0x55f1257522f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.567+0000 7f78475f8740 1 --2- 172.21.15.28:0/3146766473 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1257494d0 0x55f1257522f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f78300d6370 tx=0x7f78300c5190 comp rx=0 tx=0).stop 2024-04-21T08:51:52.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.567+0000 7f78475f8740 1 -- 172.21.15.28:0/3146766473 shutdown_connections 2024-04-21T08:51:52.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.567+0000 7f78475f8740 1 --2- 172.21.15.28:0/3146766473 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f125763060 0x55f1257654b0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.567+0000 7f78475f8740 1 --2- 172.21.15.28:0/3146766473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125752a00 0x55f12574fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.567+0000 7f7825ffb640 0 client.25534 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:52.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.567+0000 7f78475f8740 1 --2- 172.21.15.28:0/3146766473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125749e80 0x55f12574f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.567+0000 7f78475f8740 1 --2- 172.21.15.28:0/3146766473 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1257494d0 0x55f1257522f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.567+0000 7f78475f8740 1 -- 172.21.15.28:0/3146766473 >> 172.21.15.28:0/3146766473 conn(0x55f125743ba0 msgr2=0x55f1257458f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.567+0000 7f78475f8740 1 -- 172.21.15.28:0/3146766473 shutdown_connections 2024-04-21T08:51:52.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.567+0000 7f78475f8740 1 -- 172.21.15.28:0/3146766473 wait complete. 2024-04-21T08:51:52.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.567+0000 7f78475f8740 20 client.25534 trim_cache size 0 max 16384 2024-04-21T08:51:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.580+0000 7f78475f8740 1 Processor -- start 2024-04-21T08:51:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.580+0000 7f78475f8740 1 -- start start 2024-04-21T08:51:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f78475f8740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1257494d0 0x55f12575e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f78475f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125749e80 0x55f125745700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f78475f8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125752a00 0x55f125751db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f78475f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78401afa80 con 0x55f125749e80 2024-04-21T08:51:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f78475f8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f78401ad350 con 0x55f125752a00 2024-04-21T08:51:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f78475f8740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f78401ab1a0 con 0x55f1257494d0 2024-04-21T08:51:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f783dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125749e80 0x55f125745700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f783dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125749e80 0x55f125745700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42436/0 (socket says 172.21.15.28:42436) 2024-04-21T08:51:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f783dd9b640 1 -- 172.21.15.28:0/793738936 learned_addr learned my addr 172.21.15.28:0/793738936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f783e59c640 1 --2- 172.21.15.28:0/793738936 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1257494d0 0x55f12575e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f783ed9d640 1 --2- 172.21.15.28:0/793738936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125752a00 0x55f125751db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f783dd9b640 1 -- 172.21.15.28:0/793738936 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1257494d0 msgr2=0x55f12575e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f783dd9b640 1 --2- 172.21.15.28:0/793738936 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1257494d0 0x55f12575e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f783dd9b640 1 -- 172.21.15.28:0/793738936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125752a00 msgr2=0x55f125751db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f783dd9b640 1 --2- 172.21.15.28:0/793738936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125752a00 0x55f125751db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f783dd9b640 1 -- 172.21.15.28:0/793738936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f12574e630 con 0x55f125749e80 2024-04-21T08:51:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.581+0000 7f783ed9d640 1 --2- 172.21.15.28:0/793738936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125752a00 0x55f125751db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.582+0000 7f783dd9b640 1 --2- 172.21.15.28:0/793738936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125749e80 0x55f125745700 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x55f125748fa0 tx=0x7f783400b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af547c84ea4c3f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.582+0000 7f7827fff640 1 -- 172.21.15.28:0/793738936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783402f070 con 0x55f125749e80 2024-04-21T08:51:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.582+0000 7f7827fff640 1 -- 172.21.15.28:0/793738936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7834028030 con 0x55f125749e80 2024-04-21T08:51:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.582+0000 7f7827fff640 1 -- 172.21.15.28:0/793738936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7834015040 con 0x55f125749e80 2024-04-21T08:51:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.583+0000 7f78475f8740 1 -- 172.21.15.28:0/793738936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125749e80 msgr2=0x55f125745700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.583+0000 7f78475f8740 1 --2- 172.21.15.28:0/793738936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125749e80 0x55f125745700 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x55f125748fa0 tx=0x7f783400b6b0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.583+0000 7f78475f8740 1 -- 172.21.15.28:0/793738936 shutdown_connections 2024-04-21T08:51:52.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.583+0000 7f78475f8740 1 --2- 172.21.15.28:0/793738936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125752a00 0x55f125751db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.583+0000 7f78475f8740 1 --2- 172.21.15.28:0/793738936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f125749e80 0x55f125745700 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.583+0000 7f78475f8740 1 --2- 172.21.15.28:0/793738936 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1257494d0 0x55f12575e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.583+0000 7f78475f8740 1 -- 172.21.15.28:0/793738936 >> 172.21.15.28:0/793738936 conn(0x55f125743ba0 msgr2=0x55f125744ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.583+0000 7f78475f8740 1 -- 172.21.15.28:0/793738936 shutdown_connections 2024-04-21T08:51:52.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.583+0000 7f78475f8740 1 -- 172.21.15.28:0/793738936 wait complete. 2024-04-21T08:51:52.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.584+0000 7f78475f8740 1 -- start start 2024-04-21T08:51:52.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.584+0000 7f78475f8740 1 Processor -- start 2024-04-21T08:51:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.584+0000 7f78475f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1257494d0 0x55f12575ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.584+0000 7f78475f8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125749e80 0x55f12575b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.584+0000 7f78475f8740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f125752a00 0x55f12574b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.584+0000 7f78475f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78401ab810 con 0x55f1257494d0 2024-04-21T08:51:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.584+0000 7f78475f8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f78401acc80 con 0x55f125749e80 2024-04-21T08:51:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.584+0000 7f78475f8740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f78401abee0 con 0x55f125752a00 2024-04-21T08:51:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.584+0000 7f78475f8740 10 client.0 _ll_register_callbacks cb 0x7f78300a4340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.584+0000 7f783e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1257494d0 0x55f12575ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.584+0000 7f783e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1257494d0 0x55f12575ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42442/0 (socket says 172.21.15.28:42442) 2024-04-21T08:51:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.584+0000 7f783e59c640 1 -- 172.21.15.28:0/447120900 learned_addr learned my addr 172.21.15.28:0/447120900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.584+0000 7f783dd9b640 1 --2- 172.21.15.28:0/447120900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125749e80 0x55f12575b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.585+0000 7f783ed9d640 1 --2- 172.21.15.28:0/447120900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f125752a00 0x55f12574b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.585+0000 7f783e59c640 1 -- 172.21.15.28:0/447120900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f125752a00 msgr2=0x55f12574b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.585+0000 7f783e59c640 1 --2- 172.21.15.28:0/447120900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f125752a00 0x55f12574b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.585+0000 7f783e59c640 1 -- 172.21.15.28:0/447120900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125749e80 msgr2=0x55f12575b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.585+0000 7f783e59c640 1 --2- 172.21.15.28:0/447120900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125749e80 0x55f12575b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.585+0000 7f783e59c640 1 -- 172.21.15.28:0/447120900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f12574be30 con 0x55f1257494d0 2024-04-21T08:51:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.585+0000 7f783e59c640 1 --2- 172.21.15.28:0/447120900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1257494d0 0x55f12575ad40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f78300a4970 tx=0x7f78300a0db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faf21244e9566a7d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.585+0000 7f7825ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.585+0000 7f7825ffb640 1 -- 172.21.15.28:0/447120900 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78300d65b0 con 0x55f1257494d0 2024-04-21T08:51:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.585+0000 7f7825ffb640 1 -- 172.21.15.28:0/447120900 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f78300c3760 con 0x55f1257494d0 2024-04-21T08:51:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.585+0000 7f78475f8740 1 -- 172.21.15.28:0/447120900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7834028030 con 0x55f1257494d0 2024-04-21T08:51:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.586+0000 7f7825ffb640 1 -- 172.21.15.28:0/447120900 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f78300c3d40 con 0x55f1257494d0 2024-04-21T08:51:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.586+0000 7f78475f8740 10 client.16116 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.586+0000 7f78475f8740 1 -- 172.21.15.28:0/447120900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f1257c8d40 con 0x55f1257494d0 2024-04-21T08:51:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.586+0000 7f7825ffb640 1 -- 172.21.15.28:0/447120900 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7834028030 con 0x55f1257494d0 2024-04-21T08:51:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.586+0000 7f7825ffb640 1 -- 172.21.15.28:0/447120900 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f783008f0b0 con 0x55f1257494d0 2024-04-21T08:51:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.586+0000 7f78475f8740 10 client.16116 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.586+0000 7f78475f8740 10 client.16116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.586+0000 7f78475f8740 20 client.16116 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.586+0000 7f78475f8740 1 --2- 172.21.15.28:0/447120900 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f1257c94f0 0x55f1257cb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.587+0000 7f78475f8740 4 client.16116 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:51:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.587+0000 7f78475f8740 1 -- 172.21.15.28:0/447120900 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f1256bbc10 con 0x55f1257c94f0 2024-04-21T08:51:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.587+0000 7f783dd9b640 1 --2- 172.21.15.28:0/447120900 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f1257c94f0 0x55f1257cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.587+0000 7f783dd9b640 1 --2- 172.21.15.28:0/447120900 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f1257c94f0 0x55f1257cb910 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f79c36f4834de42 server_cookie=39b4aba3220f0641 in_seq=0 out_seq=0 2024-04-21T08:51:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.587+0000 7f7825ffb640 10 client.16116 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.587+0000 7f7825ffb640 1 -- 172.21.15.28:0/447120900 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f1256bbc10 con 0x55f1257c94f0 2024-04-21T08:51:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.587+0000 7f7825ffb640 10 client.16116 handle_command_reply: tid=0 2024-04-21T08:51:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.587+0000 7f78475f8740 1 client.16116 shutdown 2024-04-21T08:51:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.587+0000 7f78475f8740 1 -- 172.21.15.28:0/447120900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1257494d0 msgr2=0x55f12575ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.587+0000 7f78475f8740 1 --2- 172.21.15.28:0/447120900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1257494d0 0x55f12575ad40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f78300a4970 tx=0x7f78300a0db0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.588+0000 7f78475f8740 1 -- 172.21.15.28:0/447120900 shutdown_connections 2024-04-21T08:51:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.588+0000 7f78475f8740 1 --2- 172.21.15.28:0/447120900 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f1257c94f0 0x55f1257cb910 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.588+0000 7f78475f8740 1 --2- 172.21.15.28:0/447120900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f125752a00 0x55f12574b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.588+0000 7f78475f8740 1 --2- 172.21.15.28:0/447120900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f125749e80 0x55f12575b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.588+0000 7f78475f8740 1 --2- 172.21.15.28:0/447120900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1257494d0 0x55f12575ad40 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.588+0000 7f78475f8740 1 -- 172.21.15.28:0/447120900 >> 172.21.15.28:0/447120900 conn(0x55f125743ba0 msgr2=0x55f125742c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.588+0000 7f7825ffb640 0 client.16116 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.588+0000 7f78475f8740 1 -- 172.21.15.28:0/447120900 shutdown_connections 2024-04-21T08:51:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.588+0000 7f78475f8740 1 -- 172.21.15.28:0/447120900 wait complete. 2024-04-21T08:51:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.588+0000 7f78475f8740 20 client.16116 trim_cache size 0 max 16384 2024-04-21T08:51:52.593 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:51:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.589+0000 7f7844a06640 1 -- 172.21.15.28:0/3274120991 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f78280a3ec0 msgr2=0x7f78280a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.589+0000 7f7844a06640 1 --2- 172.21.15.28:0/3274120991 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f78280a3ec0 0x7f78280a6380 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7838009bf0 tx=0x7f7838005d80 comp rx=0 tx=0).stop 2024-04-21T08:51:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.589+0000 7f7844a06640 1 -- 172.21.15.28:0/3274120991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7840100db0 msgr2=0x7f78401a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.589+0000 7f7844a06640 1 --2- 172.21.15.28:0/3274120991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7840100db0 0x7f78401a0670 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f783000bea0 tx=0x7f783000bf90 comp rx=0 tx=0).stop 2024-04-21T08:51:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.589+0000 7f7844a06640 1 -- 172.21.15.28:0/3274120991 shutdown_connections 2024-04-21T08:51:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.589+0000 7f7844a06640 1 --2- 172.21.15.28:0/3274120991 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f78280a3ec0 0x7f78280a6380 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.589+0000 7f7844a06640 1 --2- 172.21.15.28:0/3274120991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f784010e960 0x7f784019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.589+0000 7f7844a06640 1 --2- 172.21.15.28:0/3274120991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f78401016d0 0x7f78401a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.589+0000 7f7844a06640 1 --2- 172.21.15.28:0/3274120991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7840100db0 0x7f78401a0670 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.589+0000 7f7844a06640 1 -- 172.21.15.28:0/3274120991 >> 172.21.15.28:0/3274120991 conn(0x7f7840077fb0 msgr2=0x7f784010ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.589+0000 7f7844a06640 1 -- 172.21.15.28:0/3274120991 shutdown_connections 2024-04-21T08:51:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:52.589+0000 7f7844a06640 1 -- 172.21.15.28:0/3274120991 wait complete. 2024-04-21T08:51:53.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:52 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3715741612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:53.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:52 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3715741612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:53.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:52 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3715741612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:53.601 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:51:53.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:51:53.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:51:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.721+0000 7f9e830e0640 1 -- 172.21.15.28:0/731930485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7c10e900 msgr2=0x7f9e7c110cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.721+0000 7f9e830e0640 1 --2- 172.21.15.28:0/731930485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7c10e900 0x7f9e7c110cf0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9e74009ca0 tx=0x7f9e7402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.721+0000 7f9e830e0640 1 -- 172.21.15.28:0/731930485 shutdown_connections 2024-04-21T08:51:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.721+0000 7f9e830e0640 1 --2- 172.21.15.28:0/731930485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7c10e900 0x7f9e7c110cf0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.721+0000 7f9e830e0640 1 --2- 172.21.15.28:0/731930485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9e7c101670 0x7f9e7c10e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.721+0000 7f9e830e0640 1 --2- 172.21.15.28:0/731930485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e7c100d50 0x7f9e7c101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.721+0000 7f9e830e0640 1 -- 172.21.15.28:0/731930485 >> 172.21.15.28:0/731930485 conn(0x7f9e7c077f50 msgr2=0x7f9e7c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.721+0000 7f9e830e0640 1 -- 172.21.15.28:0/731930485 shutdown_connections 2024-04-21T08:51:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.721+0000 7f9e830e0640 1 -- 172.21.15.28:0/731930485 wait complete. 2024-04-21T08:51:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.721+0000 7f9e830e0640 1 Processor -- start 2024-04-21T08:51:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.722+0000 7f9e830e0640 1 -- start start 2024-04-21T08:51:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.722+0000 7f9e830e0640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9e7c100d50 0x7f9e7c1a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.722+0000 7f9e830e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7c101670 0x7f9e7c1a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.722+0000 7f9e830e0640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e7c10e900 0x7f9e7c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.722+0000 7f9e830e0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c1143c0 con 0x7f9e7c101670 2024-04-21T08:51:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.722+0000 7f9e830e0640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c114240 con 0x7f9e7c100d50 2024-04-21T08:51:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.722+0000 7f9e830e0640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c114540 con 0x7f9e7c10e900 2024-04-21T08:51:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.722+0000 7f9e7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7c101670 0x7f9e7c1a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.722+0000 7f9e7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7c101670 0x7f9e7c1a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54882/0 (socket says 172.21.15.28:54882) 2024-04-21T08:51:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.722+0000 7f9e7bfff640 1 -- 172.21.15.28:0/2549463850 learned_addr learned my addr 172.21.15.28:0/2549463850 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.722+0000 7f9e80e7c640 1 --2- 172.21.15.28:0/2549463850 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9e7c100d50 0x7f9e7c1a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.722+0000 7f9e8167d640 1 --2- 172.21.15.28:0/2549463850 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e7c10e900 0x7f9e7c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.723+0000 7f9e7bfff640 1 -- 172.21.15.28:0/2549463850 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e7c10e900 msgr2=0x7f9e7c19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.723+0000 7f9e7bfff640 1 --2- 172.21.15.28:0/2549463850 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e7c10e900 0x7f9e7c19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.723+0000 7f9e7bfff640 1 -- 172.21.15.28:0/2549463850 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9e7c100d50 msgr2=0x7f9e7c1a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.723+0000 7f9e7bfff640 1 --2- 172.21.15.28:0/2549463850 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9e7c100d50 0x7f9e7c1a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.723+0000 7f9e7bfff640 1 -- 172.21.15.28:0/2549463850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e7c19b040 con 0x7f9e7c101670 2024-04-21T08:51:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.723+0000 7f9e8167d640 1 --2- 172.21.15.28:0/2549463850 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e7c10e900 0x7f9e7c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.723+0000 7f9e7bfff640 1 --2- 172.21.15.28:0/2549463850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7c101670 0x7f9e7c1a0bb0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800d950 tx=0x7f9e6800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.723+0000 7f9e79ffb640 1 -- 172.21.15.28:0/2549463850 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e68017070 con 0x7f9e7c101670 2024-04-21T08:51:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.723+0000 7f9e79ffb640 1 -- 172.21.15.28:0/2549463850 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e68009c30 con 0x7f9e7c101670 2024-04-21T08:51:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.723+0000 7f9e830e0640 1 -- 172.21.15.28:0/2549463850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e7c19b330 con 0x7f9e7c101670 2024-04-21T08:51:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.723+0000 7f9e79ffb640 1 -- 172.21.15.28:0/2549463850 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6800f620 con 0x7f9e7c101670 2024-04-21T08:51:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.723+0000 7f9e830e0640 1 -- 172.21.15.28:0/2549463850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e7c1a75f0 con 0x7f9e7c101670 2024-04-21T08:51:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.726+0000 7f9e85dad740 1 Processor -- start 2024-04-21T08:51:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.726+0000 7f9e85dad740 1 -- start start 2024-04-21T08:51:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.726+0000 7f9e85dad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28dd8e0 0x55c2e28ddcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.726+0000 7f9e85dad740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28de290 0x55c2e28e6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.726+0000 7f9e85dad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28e6e10 0x55c2e28e9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.726+0000 7f9e85dad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c2e2841f00 con 0x55c2e28e6e10 2024-04-21T08:51:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.726+0000 7f9e85dad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c2e27df590 con 0x55c2e28dd8e0 2024-04-21T08:51:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.726+0000 7f9e85dad740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55c2e283dec0 con 0x55c2e28de290 2024-04-21T08:51:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.726+0000 7f9e79ffb640 1 -- 172.21.15.28:0/2549463850 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f9e68010460 con 0x7f9e7c101670 2024-04-21T08:51:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.727+0000 7f9e80e7c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28de290 0x55c2e28e6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.727+0000 7f9e80e7c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28de290 0x55c2e28e6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:35682/0 (socket says 172.21.15.28:35682) 2024-04-21T08:51:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.727+0000 7f9e80e7c640 1 -- 172.21.15.28:0/179307196 learned_addr learned my addr 172.21.15.28:0/179307196 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.727+0000 7f9e8167d640 1 --2- 172.21.15.28:0/179307196 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28dd8e0 0x55c2e28ddcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.727+0000 7f9e7bfff640 1 --2- 172.21.15.28:0/179307196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28e6e10 0x55c2e28e9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.727+0000 7f9e80e7c640 1 -- 172.21.15.28:0/179307196 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28dd8e0 msgr2=0x55c2e28ddcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.727+0000 7f9e80e7c640 1 --2- 172.21.15.28:0/179307196 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28dd8e0 0x55c2e28ddcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.727+0000 7f9e80e7c640 1 -- 172.21.15.28:0/179307196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28e6e10 msgr2=0x55c2e28e9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.727+0000 7f9e80e7c640 1 --2- 172.21.15.28:0/179307196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28e6e10 0x55c2e28e9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.727+0000 7f9e80e7c640 1 -- 172.21.15.28:0/179307196 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2e261c7e0 con 0x55c2e28de290 2024-04-21T08:51:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.727+0000 7f9e7bfff640 1 --2- 172.21.15.28:0/179307196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28e6e10 0x55c2e28e9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.727+0000 7f9e79ffb640 1 --2- 172.21.15.28:0/2549463850 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9e6c0a3e40 0x7f9e6c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.727+0000 7f9e8167d640 1 --2- 172.21.15.28:0/2549463850 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9e6c0a3e40 0x7f9e6c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.727+0000 7f9e79ffb640 1 -- 172.21.15.28:0/2549463850 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f9e680d0ab0 con 0x7f9e7c101670 2024-04-21T08:51:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.727+0000 7f9e80e7c640 1 --2- 172.21.15.28:0/179307196 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28de290 0x55c2e28e6840 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9e7000bcf0 tx=0x7f9e7000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dee158a3bd830084 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:53.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.728+0000 7f9e66ffd640 1 -- 172.21.15.28:0/179307196 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e70015070 con 0x55c2e28de290 2024-04-21T08:51:53.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.728+0000 7f9e66ffd640 1 -- 172.21.15.28:0/179307196 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e700118f0 con 0x55c2e28de290 2024-04-21T08:51:53.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.728+0000 7f9e66ffd640 1 -- 172.21.15.28:0/179307196 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e700107f0 con 0x55c2e28de290 2024-04-21T08:51:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.729+0000 7f9e8167d640 1 --2- 172.21.15.28:0/2549463850 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9e6c0a3e40 0x7f9e6c0a6300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9e7402c630 tx=0x7f9e74004a40 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.729+0000 7f9e85dad740 1 -- 172.21.15.28:0/179307196 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28de290 msgr2=0x55c2e28e6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.729+0000 7f9e85dad740 1 --2- 172.21.15.28:0/179307196 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28de290 0x55c2e28e6840 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9e7000bcf0 tx=0x7f9e7000f730 comp rx=0 tx=0).stop 2024-04-21T08:51:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.729+0000 7f9e85dad740 1 -- 172.21.15.28:0/179307196 shutdown_connections 2024-04-21T08:51:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.729+0000 7f9e85dad740 1 --2- 172.21.15.28:0/179307196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28e6e10 0x55c2e28e9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.729+0000 7f9e85dad740 1 --2- 172.21.15.28:0/179307196 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28de290 0x55c2e28e6840 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.729+0000 7f9e85dad740 1 --2- 172.21.15.28:0/179307196 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28dd8e0 0x55c2e28ddcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.729+0000 7f9e85dad740 1 -- 172.21.15.28:0/179307196 >> 172.21.15.28:0/179307196 conn(0x55c2e28d7fb0 msgr2=0x55c2e28d8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.729+0000 7f9e85dad740 1 -- 172.21.15.28:0/179307196 shutdown_connections 2024-04-21T08:51:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.729+0000 7f9e85dad740 1 -- 172.21.15.28:0/179307196 wait complete. 2024-04-21T08:51:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.729+0000 7f9e85dad740 1 -- start start 2024-04-21T08:51:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.729+0000 7f9e85dad740 1 Processor -- start 2024-04-21T08:51:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.730+0000 7f9e85dad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28dd8e0 0x55c2e28e3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.730+0000 7f9e85dad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28de290 0x55c2e28e4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.730+0000 7f9e85dad740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28e6e10 0x55c2e28e48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.730+0000 7f9e85dad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c2e28e9d80 con 0x55c2e28dd8e0 2024-04-21T08:51:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.730+0000 7f9e85dad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c2e27fb580 con 0x55c2e28de290 2024-04-21T08:51:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.730+0000 7f9e85dad740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55c2e28e9f00 con 0x55c2e28e6e10 2024-04-21T08:51:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.730+0000 7f9e85dad740 10 client.0 _ll_register_callbacks cb 0x55c2e2801650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.730+0000 7f9e80e7c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28dd8e0 0x55c2e28e3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.730+0000 7f9e8167d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28e6e10 0x55c2e28e48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.730+0000 7f9e80e7c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28dd8e0 0x55c2e28e3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54898/0 (socket says 172.21.15.28:54898) 2024-04-21T08:51:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.730+0000 7f9e80e7c640 1 -- 172.21.15.28:0/4140939302 learned_addr learned my addr 172.21.15.28:0/4140939302 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.730+0000 7f9e7bfff640 1 --2- 172.21.15.28:0/4140939302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28de290 0x55c2e28e4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.731+0000 7f9e80e7c640 1 -- 172.21.15.28:0/4140939302 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28e6e10 msgr2=0x55c2e28e48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.731+0000 7f9e80e7c640 1 --2- 172.21.15.28:0/4140939302 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28e6e10 0x55c2e28e48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.731+0000 7f9e80e7c640 1 -- 172.21.15.28:0/4140939302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28de290 msgr2=0x55c2e28e4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.731+0000 7f9e80e7c640 1 --2- 172.21.15.28:0/4140939302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28de290 0x55c2e28e4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.731+0000 7f9e80e7c640 1 -- 172.21.15.28:0/4140939302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2e267cf70 con 0x55c2e28dd8e0 2024-04-21T08:51:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.731+0000 7f9e8167d640 1 --2- 172.21.15.28:0/4140939302 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28e6e10 0x55c2e28e48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.731+0000 7f9e80e7c640 1 --2- 172.21.15.28:0/4140939302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28dd8e0 0x55c2e28e3e50 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f9e70010410 tx=0x7f9e70011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465068eef8f1f0cf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.731+0000 7f9e5bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.731+0000 7f9e5bfff640 1 -- 172.21.15.28:0/4140939302 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e70028060 con 0x55c2e28dd8e0 2024-04-21T08:51:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.731+0000 7f9e5bfff640 1 -- 172.21.15.28:0/4140939302 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f9e70011aa0 con 0x55c2e28dd8e0 2024-04-21T08:51:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.731+0000 7f9e85dad740 1 -- 172.21.15.28:0/4140939302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e700118f0 con 0x55c2e28dd8e0 2024-04-21T08:51:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.732+0000 7f9e5bfff640 1 -- 172.21.15.28:0/4140939302 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9e7002f050 con 0x55c2e28dd8e0 2024-04-21T08:51:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.732+0000 7f9e85dad740 10 client.16128 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.732+0000 7f9e85dad740 1 -- 172.21.15.28:0/4140939302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c2e2514170 con 0x55c2e28dd8e0 2024-04-21T08:51:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.732+0000 7f9e5bfff640 1 -- 172.21.15.28:0/4140939302 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e700118f0 con 0x55c2e28dd8e0 2024-04-21T08:51:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.732+0000 7f9e5bfff640 1 -- 172.21.15.28:0/4140939302 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f9e70004840 con 0x55c2e28dd8e0 2024-04-21T08:51:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.732+0000 7f9e85dad740 10 client.16128 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.732+0000 7f9e85dad740 10 client.16128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.732+0000 7f9e85dad740 20 client.16128 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.732+0000 7f9e85dad740 1 --2- 172.21.15.28:0/4140939302 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c2e28f7310 0x55c2e28f9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.732+0000 7f9e85dad740 4 client.16128 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.732+0000 7f9e85dad740 1 -- 172.21.15.28:0/4140939302 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2e284f920 con 0x55c2e28f7310 2024-04-21T08:51:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.733+0000 7f9e7bfff640 1 --2- 172.21.15.28:0/4140939302 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c2e28f7310 0x55c2e28f9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.733+0000 7f9e7bfff640 1 --2- 172.21.15.28:0/4140939302 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c2e28f7310 0x55c2e28f9700 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2257b280f57c server_cookie=5ba9d363664469e2 in_seq=0 out_seq=0 2024-04-21T08:51:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.733+0000 7f9e5bfff640 10 client.16128 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.734+0000 7f9e5bfff640 1 -- 172.21.15.28:0/4140939302 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c2e284f920 con 0x55c2e28f7310 2024-04-21T08:51:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.734+0000 7f9e5bfff640 10 client.16128 handle_command_reply: tid=0 2024-04-21T08:51:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.734+0000 7f9e85dad740 1 client.16128 shutdown 2024-04-21T08:51:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.734+0000 7f9e85dad740 1 -- 172.21.15.28:0/4140939302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28dd8e0 msgr2=0x55c2e28e3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.734+0000 7f9e85dad740 1 --2- 172.21.15.28:0/4140939302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28dd8e0 0x55c2e28e3e50 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f9e70010410 tx=0x7f9e70011440 comp rx=0 tx=0).stop 2024-04-21T08:51:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.734+0000 7f9e85dad740 1 -- 172.21.15.28:0/4140939302 shutdown_connections 2024-04-21T08:51:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.734+0000 7f9e85dad740 1 --2- 172.21.15.28:0/4140939302 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c2e28f7310 0x55c2e28f9700 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.734+0000 7f9e85dad740 1 --2- 172.21.15.28:0/4140939302 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28e6e10 0x55c2e28e48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.734+0000 7f9e85dad740 1 --2- 172.21.15.28:0/4140939302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28de290 0x55c2e28e4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.734+0000 7f9e85dad740 1 --2- 172.21.15.28:0/4140939302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28dd8e0 0x55c2e28e3e50 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.734+0000 7f9e85dad740 1 -- 172.21.15.28:0/4140939302 >> 172.21.15.28:0/4140939302 conn(0x55c2e28d7fb0 msgr2=0x55c2e28d9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:53.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.734+0000 7f9e5bfff640 0 client.16128 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:53.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.734+0000 7f9e85dad740 1 -- 172.21.15.28:0/4140939302 shutdown_connections 2024-04-21T08:51:53.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.734+0000 7f9e85dad740 1 -- 172.21.15.28:0/4140939302 wait complete. 2024-04-21T08:51:53.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.735+0000 7f9e85dad740 20 client.16128 trim_cache size 0 max 16384 2024-04-21T08:51:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.770+0000 7f9e85dad740 1 Processor -- start 2024-04-21T08:51:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.770+0000 7f9e85dad740 1 -- start start 2024-04-21T08:51:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.770+0000 7f9e85dad740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28dd8e0 0x55c2e28e6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.770+0000 7f9e85dad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28de290 0x55c2e28f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.770+0000 7f9e85dad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28e6e10 0x55c2e28e3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.770+0000 7f9e85dad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c1b3e20 con 0x55c2e28e6e10 2024-04-21T08:51:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.770+0000 7f9e85dad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c1b16f0 con 0x55c2e28de290 2024-04-21T08:51:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.770+0000 7f9e85dad740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c1af540 con 0x55c2e28dd8e0 2024-04-21T08:51:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.770+0000 7f9e8167d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28e6e10 0x55c2e28e3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.771+0000 7f9e8167d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28e6e10 0x55c2e28e3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54914/0 (socket says 172.21.15.28:54914) 2024-04-21T08:51:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.771+0000 7f9e8167d640 1 -- 172.21.15.28:0/2504239653 learned_addr learned my addr 172.21.15.28:0/2504239653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.771+0000 7f9e7bfff640 1 --2- 172.21.15.28:0/2504239653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28de290 0x55c2e28f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.771+0000 7f9e80e7c640 1 --2- 172.21.15.28:0/2504239653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28dd8e0 0x55c2e28e6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.771+0000 7f9e8167d640 1 -- 172.21.15.28:0/2504239653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28dd8e0 msgr2=0x55c2e28e6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.771+0000 7f9e8167d640 1 --2- 172.21.15.28:0/2504239653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28dd8e0 0x55c2e28e6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.771+0000 7f9e8167d640 1 -- 172.21.15.28:0/2504239653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28de290 msgr2=0x55c2e28f2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.771+0000 7f9e8167d640 1 --2- 172.21.15.28:0/2504239653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28de290 0x55c2e28f2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.771+0000 7f9e8167d640 1 -- 172.21.15.28:0/2504239653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2e28d32d0 con 0x55c2e28e6e10 2024-04-21T08:51:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.771+0000 7f9e80e7c640 1 --2- 172.21.15.28:0/2504239653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28dd8e0 0x55c2e28e6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.771+0000 7f9e7bfff640 1 --2- 172.21.15.28:0/2504239653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28de290 0x55c2e28f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:53.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.771+0000 7f9e8167d640 1 --2- 172.21.15.28:0/2504239653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28e6e10 0x55c2e28e3730 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9e74006010 tx=0x7f9e74040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e12f37044391a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:53.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.771+0000 7f9e667fc640 1 -- 172.21.15.28:0/2504239653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e74039040 con 0x55c2e28e6e10 2024-04-21T08:51:53.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.771+0000 7f9e667fc640 1 -- 172.21.15.28:0/2504239653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e74052070 con 0x55c2e28e6e10 2024-04-21T08:51:53.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.771+0000 7f9e667fc640 1 -- 172.21.15.28:0/2504239653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e74004810 con 0x55c2e28e6e10 2024-04-21T08:51:53.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.772+0000 7f9e85dad740 1 -- 172.21.15.28:0/2504239653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28e6e10 msgr2=0x55c2e28e3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.772+0000 7f9e85dad740 1 --2- 172.21.15.28:0/2504239653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28e6e10 0x55c2e28e3730 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9e74006010 tx=0x7f9e74040000 comp rx=0 tx=0).stop 2024-04-21T08:51:53.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.772+0000 7f9e85dad740 1 -- 172.21.15.28:0/2504239653 shutdown_connections 2024-04-21T08:51:53.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.772+0000 7f9e85dad740 1 --2- 172.21.15.28:0/2504239653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28e6e10 0x55c2e28e3730 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.772+0000 7f9e85dad740 1 --2- 172.21.15.28:0/2504239653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28de290 0x55c2e28f2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.772+0000 7f9e85dad740 1 --2- 172.21.15.28:0/2504239653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28dd8e0 0x55c2e28e6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.772+0000 7f9e85dad740 1 -- 172.21.15.28:0/2504239653 >> 172.21.15.28:0/2504239653 conn(0x55c2e28d7fb0 msgr2=0x55c2e28d7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.773+0000 7f9e85dad740 1 -- 172.21.15.28:0/2504239653 shutdown_connections 2024-04-21T08:51:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.773+0000 7f9e85dad740 1 -- 172.21.15.28:0/2504239653 wait complete. 2024-04-21T08:51:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.773+0000 7f9e85dad740 1 -- start start 2024-04-21T08:51:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.773+0000 7f9e85dad740 1 Processor -- start 2024-04-21T08:51:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.773+0000 7f9e85dad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28dd8e0 0x55c2e28ef1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.773+0000 7f9e85dad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28de290 0x55c2e28ef700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.773+0000 7f9e85dad740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28e6e10 0x55c2e28ea730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e85dad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c1afbb0 con 0x55c2e28de290 2024-04-21T08:51:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e85dad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c1b1020 con 0x55c2e28dd8e0 2024-04-21T08:51:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e85dad740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c1b0280 con 0x55c2e28e6e10 2024-04-21T08:51:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e85dad740 10 client.0 _ll_register_callbacks cb 0x7f9e74037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28de290 0x55c2e28ef700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28de290 0x55c2e28ef700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54920/0 (socket says 172.21.15.28:54920) 2024-04-21T08:51:53.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e7bfff640 1 -- 172.21.15.28:0/3846957851 learned_addr learned my addr 172.21.15.28:0/3846957851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e80e7c640 1 --2- 172.21.15.28:0/3846957851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28dd8e0 0x55c2e28ef1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e8167d640 1 --2- 172.21.15.28:0/3846957851 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28e6e10 0x55c2e28ea730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e7bfff640 1 -- 172.21.15.28:0/3846957851 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28e6e10 msgr2=0x55c2e28ea730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e7bfff640 1 --2- 172.21.15.28:0/3846957851 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28e6e10 0x55c2e28ea730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e7bfff640 1 -- 172.21.15.28:0/3846957851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28dd8e0 msgr2=0x55c2e28ef1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e7bfff640 1 --2- 172.21.15.28:0/3846957851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28dd8e0 0x55c2e28ef1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e7bfff640 1 -- 172.21.15.28:0/3846957851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2e28eaf30 con 0x55c2e28de290 2024-04-21T08:51:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e7bfff640 1 --2- 172.21.15.28:0/3846957851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28de290 0x55c2e28ef700 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9e680cddf0 tx=0x7f9e6800bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55fc872d142c78ce server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e5bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:51:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.774+0000 7f9e5bfff640 1 -- 172.21.15.28:0/3846957851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e68020040 con 0x55c2e28de290 2024-04-21T08:51:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.775+0000 7f9e5bfff640 1 -- 172.21.15.28:0/3846957851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f9e680cdfc0 con 0x55c2e28de290 2024-04-21T08:51:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.775+0000 7f9e85dad740 1 -- 172.21.15.28:0/3846957851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e74052070 con 0x55c2e28de290 2024-04-21T08:51:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.775+0000 7f9e5bfff640 1 -- 172.21.15.28:0/3846957851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9e680a0810 con 0x55c2e28de290 2024-04-21T08:51:53.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.775+0000 7f9e85dad740 10 client.16134 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.775+0000 7f9e85dad740 1 -- 172.21.15.28:0/3846957851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c2e295d210 con 0x55c2e28de290 2024-04-21T08:51:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.775+0000 7f9e5bfff640 1 -- 172.21.15.28:0/3846957851 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e74052070 con 0x55c2e28de290 2024-04-21T08:51:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.776+0000 7f9e5bfff640 1 -- 172.21.15.28:0/3846957851 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f9e68098340 con 0x55c2e28de290 2024-04-21T08:51:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.776+0000 7f9e85dad740 10 client.16134 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.776+0000 7f9e85dad740 10 client.16134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.776+0000 7f9e85dad740 20 client.16134 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.776+0000 7f9e85dad740 1 --2- 172.21.15.28:0/3846957851 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c2e295d940 0x55c2e295fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.776+0000 7f9e85dad740 4 client.16134 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:51:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.776+0000 7f9e85dad740 1 -- 172.21.15.28:0/3846957851 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c2e2849790 con 0x55c2e295d940 2024-04-21T08:51:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.776+0000 7f9e80e7c640 1 --2- 172.21.15.28:0/3846957851 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c2e295d940 0x55c2e295fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.777+0000 7f9e80e7c640 1 --2- 172.21.15.28:0/3846957851 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c2e295d940 0x55c2e295fd60 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eaf02f69d1b4741e server_cookie=bfcdca243fefedfc in_seq=0 out_seq=0 2024-04-21T08:51:53.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.777+0000 7f9e5bfff640 10 client.16134 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:53.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.778+0000 7f9e5bfff640 1 -- 172.21.15.28:0/3846957851 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c2e2849790 con 0x55c2e295d940 2024-04-21T08:51:53.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.778+0000 7f9e5bfff640 10 client.16134 handle_command_reply: tid=0 2024-04-21T08:51:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.778+0000 7f9e85dad740 1 client.16134 shutdown 2024-04-21T08:51:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.778+0000 7f9e85dad740 1 -- 172.21.15.28:0/3846957851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28de290 msgr2=0x55c2e28ef700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.778+0000 7f9e85dad740 1 --2- 172.21.15.28:0/3846957851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28de290 0x55c2e28ef700 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9e680cddf0 tx=0x7f9e6800bf80 comp rx=0 tx=0).stop 2024-04-21T08:51:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.778+0000 7f9e85dad740 1 -- 172.21.15.28:0/3846957851 shutdown_connections 2024-04-21T08:51:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.778+0000 7f9e85dad740 1 --2- 172.21.15.28:0/3846957851 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c2e295d940 0x55c2e295fd60 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.778+0000 7f9e85dad740 1 --2- 172.21.15.28:0/3846957851 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c2e28e6e10 0x55c2e28ea730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.778+0000 7f9e85dad740 1 --2- 172.21.15.28:0/3846957851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2e28de290 0x55c2e28ef700 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.778+0000 7f9e85dad740 1 --2- 172.21.15.28:0/3846957851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2e28dd8e0 0x55c2e28ef1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.778+0000 7f9e5bfff640 0 client.16134 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.778+0000 7f9e85dad740 1 -- 172.21.15.28:0/3846957851 >> 172.21.15.28:0/3846957851 conn(0x55c2e28d7fb0 msgr2=0x55c2e28e3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.778+0000 7f9e85dad740 1 -- 172.21.15.28:0/3846957851 shutdown_connections 2024-04-21T08:51:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.778+0000 7f9e85dad740 1 -- 172.21.15.28:0/3846957851 wait complete. 2024-04-21T08:51:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.779+0000 7f9e85dad740 20 client.16134 trim_cache size 0 max 16384 2024-04-21T08:51:53.782 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:51:53.782 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:51:53.782 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "25e90a3e-4366-4a3e-aab4-171469b773f4", 2024-04-21T08:51:53.782 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:51:53.782 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:51:53.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.779+0000 7f9e830e0640 1 -- 172.21.15.28:0/2549463850 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9e6c0a3e40 msgr2=0x7f9e6c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.779+0000 7f9e830e0640 1 --2- 172.21.15.28:0/2549463850 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9e6c0a3e40 0x7f9e6c0a6300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9e7402c630 tx=0x7f9e74004a40 comp rx=0 tx=0).stop 2024-04-21T08:51:53.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.779+0000 7f9e830e0640 1 -- 172.21.15.28:0/2549463850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7c101670 msgr2=0x7f9e7c1a0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.779+0000 7f9e830e0640 1 --2- 172.21.15.28:0/2549463850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7c101670 0x7f9e7c1a0bb0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800d950 tx=0x7f9e6800de10 comp rx=0 tx=0).stop 2024-04-21T08:51:53.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.780+0000 7f9e830e0640 1 -- 172.21.15.28:0/2549463850 shutdown_connections 2024-04-21T08:51:53.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.780+0000 7f9e830e0640 1 --2- 172.21.15.28:0/2549463850 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9e6c0a3e40 0x7f9e6c0a6300 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.780+0000 7f9e830e0640 1 --2- 172.21.15.28:0/2549463850 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e7c10e900 0x7f9e7c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.780+0000 7f9e830e0640 1 --2- 172.21.15.28:0/2549463850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e7c101670 0x7f9e7c1a0bb0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.780+0000 7f9e830e0640 1 --2- 172.21.15.28:0/2549463850 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9e7c100d50 0x7f9e7c1a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.780+0000 7f9e830e0640 1 -- 172.21.15.28:0/2549463850 >> 172.21.15.28:0/2549463850 conn(0x7f9e7c077f50 msgr2=0x7f9e7c10ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:53.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.780+0000 7f9e830e0640 1 -- 172.21.15.28:0/2549463850 shutdown_connections 2024-04-21T08:51:53.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.780+0000 7f9e830e0640 1 -- 172.21.15.28:0/2549463850 wait complete. 2024-04-21T08:51:53.794 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:51:53.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:51:53.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:53 smithi028 ceph-mon[28003]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T08:51:53.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.944+0000 7fb251ddb640 1 -- 172.21.15.28:0/2348418150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c101a40 msgr2=0x7fb24c10dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.944+0000 7fb251ddb640 1 --2- 172.21.15.28:0/2348418150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c101a40 0x7fb24c10dc70 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb244009ca0 tx=0x7fb24402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.944+0000 7fb251ddb640 1 -- 172.21.15.28:0/2348418150 shutdown_connections 2024-04-21T08:51:53.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.944+0000 7fb251ddb640 1 --2- 172.21.15.28:0/2348418150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c101a40 0x7fb24c10dc70 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.944+0000 7fb251ddb640 1 --2- 172.21.15.28:0/2348418150 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb24c1010a0 0x7fb24c101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.944+0000 7fb251ddb640 1 --2- 172.21.15.28:0/2348418150 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb24c106ed0 0x7fb24c1072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.944+0000 7fb251ddb640 1 -- 172.21.15.28:0/2348418150 >> 172.21.15.28:0/2348418150 conn(0x7fb24c077c40 msgr2=0x7fb24c078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:53.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.945+0000 7fb251ddb640 1 -- 172.21.15.28:0/2348418150 shutdown_connections 2024-04-21T08:51:53.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.945+0000 7fb251ddb640 1 -- 172.21.15.28:0/2348418150 wait complete. 2024-04-21T08:51:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.945+0000 7fb251ddb640 1 Processor -- start 2024-04-21T08:51:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.946+0000 7fb251ddb640 1 -- start start 2024-04-21T08:51:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.946+0000 7fb251ddb640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb24c1010a0 0x7fb24c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.946+0000 7fb251ddb640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb24c101a40 0x7fb24c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.946+0000 7fb251ddb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c106ed0 0x7fb24c19d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.946+0000 7fb251ddb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c112220 con 0x7fb24c106ed0 2024-04-21T08:51:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.946+0000 7fb24bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c106ed0 0x7fb24c19d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.946+0000 7fb251ddb640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c1120a0 con 0x7fb24c101a40 2024-04-21T08:51:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.946+0000 7fb24bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c106ed0 0x7fb24c19d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54942/0 (socket says 172.21.15.28:54942) 2024-04-21T08:51:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.946+0000 7fb24b7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb24c1010a0 0x7fb24c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.946+0000 7fb24bfff640 1 -- 172.21.15.28:0/646262364 learned_addr learned my addr 172.21.15.28:0/646262364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.946+0000 7fb251ddb640 1 -- 172.21.15.28:0/646262364 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c1123a0 con 0x7fb24c1010a0 2024-04-21T08:51:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.946+0000 7fb24affd640 1 --2- 172.21.15.28:0/646262364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb24c101a40 0x7fb24c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.946+0000 7fb24affd640 1 -- 172.21.15.28:0/646262364 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb24c1010a0 msgr2=0x7fb24c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.946+0000 7fb24affd640 1 --2- 172.21.15.28:0/646262364 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb24c1010a0 0x7fb24c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.947+0000 7fb24affd640 1 -- 172.21.15.28:0/646262364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c106ed0 msgr2=0x7fb24c19d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.947+0000 7fb24affd640 1 --2- 172.21.15.28:0/646262364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c106ed0 0x7fb24c19d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.947+0000 7fb24affd640 1 -- 172.21.15.28:0/646262364 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24c19d730 con 0x7fb24c101a40 2024-04-21T08:51:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.947+0000 7fb24bfff640 1 --2- 172.21.15.28:0/646262364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c106ed0 0x7fb24c19d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.947+0000 7fb24b7fe640 1 --2- 172.21.15.28:0/646262364 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb24c1010a0 0x7fb24c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.947+0000 7fb24affd640 1 --2- 172.21.15.28:0/646262364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb24c101a40 0x7fb24c198c80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb24000e9f0 tx=0x7fb24000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.947+0000 7fb248ff9640 1 -- 172.21.15.28:0/646262364 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24000ccc0 con 0x7fb24c101a40 2024-04-21T08:51:53.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.947+0000 7fb251ddb640 1 -- 172.21.15.28:0/646262364 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb24c19da20 con 0x7fb24c101a40 2024-04-21T08:51:53.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.947+0000 7fb248ff9640 1 -- 172.21.15.28:0/646262364 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb240005020 con 0x7fb24c101a40 2024-04-21T08:51:53.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.947+0000 7fb251ddb640 1 -- 172.21.15.28:0/646262364 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb24c1a52b0 con 0x7fb24c101a40 2024-04-21T08:51:53.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.947+0000 7fb248ff9640 1 -- 172.21.15.28:0/646262364 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb240004a00 con 0x7fb24c101a40 2024-04-21T08:51:53.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.949+0000 7fb248ff9640 1 -- 172.21.15.28:0/646262364 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7fb240004ba0 con 0x7fb24c101a40 2024-04-21T08:51:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.950+0000 7fb248ff9640 1 --2- 172.21.15.28:0/646262364 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb2340a3e40 0x7fb2340a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.951+0000 7fb248ff9640 1 -- 172.21.15.28:0/646262364 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fb2400d0dc0 con 0x7fb24c101a40 2024-04-21T08:51:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.951+0000 7fb24b7fe640 1 --2- 172.21.15.28:0/646262364 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb2340a3e40 0x7fb2340a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.951+0000 7fb2549ce740 1 Processor -- start 2024-04-21T08:51:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.951+0000 7fb2549ce740 1 -- start start 2024-04-21T08:51:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.951+0000 7fb2549ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.951+0000 7fb2549ce740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145adbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.951+0000 7fb2549ce740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 0x564a145c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.951+0000 7fb2549ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564a145181e0 con 0x564a145bfc10 2024-04-21T08:51:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.951+0000 7fb2549ce740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564a144b5390 con 0x564a145ae120 2024-04-21T08:51:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.951+0000 7fb2549ce740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564a145141a0 con 0x564a145ad760 2024-04-21T08:51:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.951+0000 7fb24b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.951+0000 7fb24b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145bfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54954/0 (socket says 172.21.15.28:54954) 2024-04-21T08:51:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.951+0000 7fb24b7fe640 1 -- 172.21.15.28:0/2426446337 learned_addr learned my addr 172.21.15.28:0/2426446337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.951+0000 7fb24b7fe640 1 --2- 172.21.15.28:0/646262364 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb2340a3e40 0x7fb2340a6300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009f40 tx=0x7fb23c005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.951+0000 7fb24affd640 1 --2- 172.21.15.28:0/2426446337 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145adbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.952+0000 7fb24b7fe640 1 -- 172.21.15.28:0/2426446337 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 msgr2=0x564a145adbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.952+0000 7fb24b7fe640 1 --2- 172.21.15.28:0/2426446337 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145adbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.952+0000 7fb24b7fe640 1 -- 172.21.15.28:0/2426446337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 msgr2=0x564a145c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.952+0000 7fb24bfff640 1 --2- 172.21.15.28:0/2426446337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 0x564a145c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.952+0000 7fb24b7fe640 1 --2- 172.21.15.28:0/2426446337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 0x564a145c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.952+0000 7fb24b7fe640 1 -- 172.21.15.28:0/2426446337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a142f47d0 con 0x564a145bfc10 2024-04-21T08:51:53.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.952+0000 7fb24bfff640 1 --2- 172.21.15.28:0/2426446337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 0x564a145c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:53.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.952+0000 7fb24b7fe640 1 --2- 172.21.15.28:0/2426446337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145bfff0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb23c004240 tx=0x7fb23c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c660c8b1deb033c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.952+0000 7fb231ffb640 1 -- 172.21.15.28:0/2426446337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c026470 con 0x564a145bfc10 2024-04-21T08:51:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.952+0000 7fb231ffb640 1 -- 172.21.15.28:0/2426446337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb23c021070 con 0x564a145bfc10 2024-04-21T08:51:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.952+0000 7fb231ffb640 1 -- 172.21.15.28:0/2426446337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c01d970 con 0x564a145bfc10 2024-04-21T08:51:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.953+0000 7fb2549ce740 1 -- 172.21.15.28:0/2426446337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 msgr2=0x564a145bfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.953+0000 7fb2549ce740 1 --2- 172.21.15.28:0/2426446337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145bfff0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb23c004240 tx=0x7fb23c018000 comp rx=0 tx=0).stop 2024-04-21T08:51:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.954+0000 7fb2549ce740 1 -- 172.21.15.28:0/2426446337 shutdown_connections 2024-04-21T08:51:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.954+0000 7fb2549ce740 1 --2- 172.21.15.28:0/2426446337 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 0x564a145c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.954+0000 7fb2549ce740 1 --2- 172.21.15.28:0/2426446337 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145adbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.954+0000 7fb2549ce740 1 --2- 172.21.15.28:0/2426446337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145bfff0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.954+0000 7fb2549ce740 1 -- 172.21.15.28:0/2426446337 >> 172.21.15.28:0/2426446337 conn(0x564a145b0220 msgr2=0x564a145b0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.954+0000 7fb2549ce740 1 -- 172.21.15.28:0/2426446337 shutdown_connections 2024-04-21T08:51:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.954+0000 7fb2549ce740 1 -- 172.21.15.28:0/2426446337 wait complete. 2024-04-21T08:51:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.954+0000 7fb2549ce740 1 -- start start 2024-04-21T08:51:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.954+0000 7fb2549ce740 1 Processor -- start 2024-04-21T08:51:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb2549ce740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb2549ce740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 0x564a145c5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb2549ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb2549ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564a145c8140 con 0x564a145bfc10 2024-04-21T08:51:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb2549ce740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564a144d12f0 con 0x564a145ae120 2024-04-21T08:51:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb2549ce740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564a145c82c0 con 0x564a145ad760 2024-04-21T08:51:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb24bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb2549ce740 10 client.0 _ll_register_callbacks cb 0x564a144de580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb24b7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb24b7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145c5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:35738/0 (socket says 172.21.15.28:35738) 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb24bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54962/0 (socket says 172.21.15.28:54962) 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb24b7fe640 1 -- 172.21.15.28:0/1458933104 learned_addr learned my addr 172.21.15.28:0/1458933104 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb24affd640 1 --2- 172.21.15.28:0/1458933104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 0x564a145c5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb24b7fe640 1 -- 172.21.15.28:0/1458933104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 msgr2=0x564a145c5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb24b7fe640 1 --2- 172.21.15.28:0/1458933104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 0x564a145c5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.955+0000 7fb24b7fe640 1 -- 172.21.15.28:0/1458933104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 msgr2=0x564a145c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.956+0000 7fb24b7fe640 1 --2- 172.21.15.28:0/1458933104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.956+0000 7fb24b7fe640 1 -- 172.21.15.28:0/1458933104 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a1422a3f0 con 0x564a145ad760 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.956+0000 7fb24bfff640 1 --2- 172.21.15.28:0/1458933104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.956+0000 7fb24b7fe640 1 --2- 172.21.15.28:0/1458933104 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145c5010 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb23c01d590 tx=0x7fb23c003f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c020f0f264eadf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.956+0000 7fb22affd640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.956+0000 7fb22affd640 1 -- 172.21.15.28:0/1458933104 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c026780 con 0x564a145ad760 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.956+0000 7fb2549ce740 1 -- 172.21.15.28:0/1458933104 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb23c021070 con 0x564a145ad760 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.956+0000 7fb22affd640 1 -- 172.21.15.28:0/1458933104 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fb23c026920 con 0x564a145ad760 2024-04-21T08:51:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.956+0000 7fb22affd640 1 -- 172.21.15.28:0/1458933104 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb23c037050 con 0x564a145ad760 2024-04-21T08:51:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.956+0000 7fb2549ce740 10 client.25567 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.956+0000 7fb2549ce740 1 -- 172.21.15.28:0/1458933104 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564a141ea0f0 con 0x564a145ad760 2024-04-21T08:51:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.957+0000 7fb22affd640 1 -- 172.21.15.28:0/1458933104 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb23c021070 con 0x564a145ad760 2024-04-21T08:51:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.957+0000 7fb22affd640 1 -- 172.21.15.28:0/1458933104 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb23c00fd20 con 0x564a145ad760 2024-04-21T08:51:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.957+0000 7fb2549ce740 10 client.25567 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.957+0000 7fb2549ce740 10 client.25567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.957+0000 7fb2549ce740 20 client.25567 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.957+0000 7fb2549ce740 1 --2- 172.21.15.28:0/1458933104 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564a145d5130 0x564a145d7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.957+0000 7fb2549ce740 4 client.25567 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.957+0000 7fb2549ce740 1 -- 172.21.15.28:0/1458933104 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a14525c00 con 0x564a145d5130 2024-04-21T08:51:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.957+0000 7fb24affd640 1 --2- 172.21.15.28:0/1458933104 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564a145d5130 0x564a145d7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.958+0000 7fb22affd640 10 client.25567 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.958+0000 7fb24affd640 1 --2- 172.21.15.28:0/1458933104 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564a145d5130 0x564a145d7520 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=454e0ab3f80a81b8 server_cookie=5d24a529ad1da248 in_seq=0 out_seq=0 2024-04-21T08:51:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.958+0000 7fb22affd640 1 -- 172.21.15.28:0/1458933104 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a14525c00 con 0x564a145d5130 2024-04-21T08:51:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.958+0000 7fb22affd640 10 client.25567 handle_command_reply: tid=0 2024-04-21T08:51:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.958+0000 7fb2549ce740 1 client.25567 shutdown 2024-04-21T08:51:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.959+0000 7fb2549ce740 1 -- 172.21.15.28:0/1458933104 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 msgr2=0x564a145c5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.959+0000 7fb2549ce740 1 --2- 172.21.15.28:0/1458933104 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145c5010 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb23c01d590 tx=0x7fb23c003f20 comp rx=0 tx=0).stop 2024-04-21T08:51:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.959+0000 7fb2549ce740 1 -- 172.21.15.28:0/1458933104 shutdown_connections 2024-04-21T08:51:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.959+0000 7fb2549ce740 1 --2- 172.21.15.28:0/1458933104 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564a145d5130 0x564a145d7520 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.959+0000 7fb22affd640 0 client.25567 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.959+0000 7fb2549ce740 1 --2- 172.21.15.28:0/1458933104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.959+0000 7fb2549ce740 1 --2- 172.21.15.28:0/1458933104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 0x564a145c5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.959+0000 7fb2549ce740 1 --2- 172.21.15.28:0/1458933104 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145c5010 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.959+0000 7fb2549ce740 1 -- 172.21.15.28:0/1458933104 >> 172.21.15.28:0/1458933104 conn(0x564a145b0220 msgr2=0x564a145c12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:53.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.959+0000 7fb2549ce740 1 -- 172.21.15.28:0/1458933104 shutdown_connections 2024-04-21T08:51:53.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.959+0000 7fb2549ce740 1 -- 172.21.15.28:0/1458933104 wait complete. 2024-04-21T08:51:53.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.959+0000 7fb2549ce740 20 client.25567 trim_cache size 0 max 16384 2024-04-21T08:51:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.977+0000 7fb2549ce740 1 Processor -- start 2024-04-21T08:51:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.977+0000 7fb2549ce740 1 -- start start 2024-04-21T08:51:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.977+0000 7fb2549ce740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ad760 0x564a145c7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.977+0000 7fb2549ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145ae120 0x564a145d1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.977+0000 7fb2549ce740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145bfc10 0x564a145bd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.977+0000 7fb2549ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c1afae0 con 0x564a145ae120 2024-04-21T08:51:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.977+0000 7fb2549ce740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c1ad3b0 con 0x564a145ad760 2024-04-21T08:51:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.977+0000 7fb2549ce740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c1ab200 con 0x564a145bfc10 2024-04-21T08:51:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.977+0000 7fb24affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145ae120 0x564a145d1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.977+0000 7fb24affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145ae120 0x564a145d1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54968/0 (socket says 172.21.15.28:54968) 2024-04-21T08:51:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.977+0000 7fb24affd640 1 -- 172.21.15.28:0/1187272194 learned_addr learned my addr 172.21.15.28:0/1187272194 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.977+0000 7fb24b7fe640 1 --2- 172.21.15.28:0/1187272194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ad760 0x564a145c7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.977+0000 7fb24bfff640 1 --2- 172.21.15.28:0/1187272194 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145bfc10 0x564a145bd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.977+0000 7fb24affd640 1 -- 172.21.15.28:0/1187272194 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145bfc10 msgr2=0x564a145bd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.978+0000 7fb24affd640 1 --2- 172.21.15.28:0/1187272194 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145bfc10 0x564a145bd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.978+0000 7fb24affd640 1 -- 172.21.15.28:0/1187272194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ad760 msgr2=0x564a145c7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.978+0000 7fb24affd640 1 --2- 172.21.15.28:0/1187272194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ad760 0x564a145c7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.978+0000 7fb24affd640 1 -- 172.21.15.28:0/1187272194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a145bc1d0 con 0x564a145ae120 2024-04-21T08:51:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.978+0000 7fb24b7fe640 1 --2- 172.21.15.28:0/1187272194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ad760 0x564a145c7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.978+0000 7fb24affd640 1 --2- 172.21.15.28:0/1187272194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145ae120 0x564a145d1000 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb2400ce100 tx=0x7fb2400d0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118822a4bb14c655 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.978+0000 7fb2317fa640 1 -- 172.21.15.28:0/1187272194 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24001c040 con 0x564a145ae120 2024-04-21T08:51:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.978+0000 7fb2317fa640 1 -- 172.21.15.28:0/1187272194 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2400a53e0 con 0x564a145ae120 2024-04-21T08:51:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.978+0000 7fb2317fa640 1 -- 172.21.15.28:0/1187272194 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2400a4f50 con 0x564a145ae120 2024-04-21T08:51:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.979+0000 7fb2549ce740 1 -- 172.21.15.28:0/1187272194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145ae120 msgr2=0x564a145d1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.979+0000 7fb2549ce740 1 --2- 172.21.15.28:0/1187272194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145ae120 0x564a145d1000 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb2400ce100 tx=0x7fb2400d0040 comp rx=0 tx=0).stop 2024-04-21T08:51:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.979+0000 7fb2549ce740 1 -- 172.21.15.28:0/1187272194 shutdown_connections 2024-04-21T08:51:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.979+0000 7fb2549ce740 1 --2- 172.21.15.28:0/1187272194 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145bfc10 0x564a145bd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.979+0000 7fb2549ce740 1 --2- 172.21.15.28:0/1187272194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145ae120 0x564a145d1000 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.979+0000 7fb2549ce740 1 --2- 172.21.15.28:0/1187272194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ad760 0x564a145c7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.979+0000 7fb2549ce740 1 -- 172.21.15.28:0/1187272194 >> 172.21.15.28:0/1187272194 conn(0x564a145b0220 msgr2=0x564a145af8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.979+0000 7fb2549ce740 1 -- 172.21.15.28:0/1187272194 shutdown_connections 2024-04-21T08:51:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.979+0000 7fb2549ce740 1 -- 172.21.15.28:0/1187272194 wait complete. 2024-04-21T08:51:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.979+0000 7fb2549ce740 1 -- start start 2024-04-21T08:51:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.980+0000 7fb2549ce740 1 Processor -- start 2024-04-21T08:51:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.980+0000 7fb2549ce740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.980+0000 7fb2549ce740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 0x564a145c9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.980+0000 7fb2549ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.980+0000 7fb2549ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c1ab870 con 0x564a145bfc10 2024-04-21T08:51:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.980+0000 7fb2549ce740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c1acce0 con 0x564a145ae120 2024-04-21T08:51:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.980+0000 7fb2549ce740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c1abf40 con 0x564a145ad760 2024-04-21T08:51:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.980+0000 7fb2549ce740 10 client.0 _ll_register_callbacks cb 0x7fb240004e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.980+0000 7fb24b7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.980+0000 7fb24bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.980+0000 7fb24b7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145c8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:35748/0 (socket says 172.21.15.28:35748) 2024-04-21T08:51:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.980+0000 7fb24b7fe640 1 -- 172.21.15.28:0/2511188676 learned_addr learned my addr 172.21.15.28:0/2511188676 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:51:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.980+0000 7fb24affd640 1 --2- 172.21.15.28:0/2511188676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 0x564a145c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.981+0000 7fb24b7fe640 1 -- 172.21.15.28:0/2511188676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 msgr2=0x564a145c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.981+0000 7fb24b7fe640 1 --2- 172.21.15.28:0/2511188676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 0x564a145c9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.981+0000 7fb24b7fe640 1 -- 172.21.15.28:0/2511188676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 msgr2=0x564a145c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.981+0000 7fb24b7fe640 1 --2- 172.21.15.28:0/2511188676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145c98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.981+0000 7fb24b7fe640 1 -- 172.21.15.28:0/2511188676 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a1463a4d0 con 0x564a145ad760 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.981+0000 7fb24bfff640 1 --2- 172.21.15.28:0/2511188676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145c98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.981+0000 7fb24b7fe640 1 --2- 172.21.15.28:0/2511188676 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145c8e50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb23c01d590 tx=0x7fb23c01d4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faff0d4c918bd135 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.981+0000 7fb22affd640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.981+0000 7fb22affd640 1 -- 172.21.15.28:0/2511188676 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c003a80 con 0x564a145ad760 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.981+0000 7fb2549ce740 1 -- 172.21.15.28:0/2511188676 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2400a53e0 con 0x564a145ad760 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.981+0000 7fb22affd640 1 -- 172.21.15.28:0/2511188676 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fb23c01c8f0 con 0x564a145ad760 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.982+0000 7fb22affd640 1 -- 172.21.15.28:0/2511188676 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb23c01da70 con 0x564a145ad760 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.982+0000 7fb2549ce740 10 client.25570 fetch_fsmap learned FSMap version 25 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.982+0000 7fb2549ce740 1 -- 172.21.15.28:0/2511188676 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564a1463b6f0 con 0x564a145ad760 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.982+0000 7fb22affd640 1 -- 172.21.15.28:0/2511188676 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2400a53e0 con 0x564a145ad760 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.982+0000 7fb22affd640 1 -- 172.21.15.28:0/2511188676 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fb23c026780 con 0x564a145ad760 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.982+0000 7fb2549ce740 10 client.25570 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.982+0000 7fb2549ce740 10 client.25570 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.982+0000 7fb2549ce740 20 client.25570 populate_metadata read hostname 'smithi028' 2024-04-21T08:51:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.982+0000 7fb2549ce740 1 --2- 172.21.15.28:0/2511188676 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564a1463be60 0x564a1463e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.982+0000 7fb2549ce740 4 client.25570 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:51:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.982+0000 7fb2549ce740 1 -- 172.21.15.28:0/2511188676 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564a1451fa70 con 0x564a1463be60 2024-04-21T08:51:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.982+0000 7fb24affd640 1 --2- 172.21.15.28:0/2511188676 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564a1463be60 0x564a1463e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.983+0000 7fb22affd640 10 client.25570 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.983+0000 7fb24affd640 1 --2- 172.21.15.28:0/2511188676 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564a1463be60 0x564a1463e250 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d93b83683f8796d server_cookie=abdeb653a5b02c1f in_seq=0 out_seq=0 2024-04-21T08:51:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.983+0000 7fb22affd640 1 -- 172.21.15.28:0/2511188676 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564a1451fa70 con 0x564a1463be60 2024-04-21T08:51:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.983+0000 7fb22affd640 10 client.25570 handle_command_reply: tid=0 2024-04-21T08:51:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.983+0000 7fb2549ce740 1 client.25570 shutdown 2024-04-21T08:51:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.984+0000 7fb2549ce740 1 -- 172.21.15.28:0/2511188676 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 msgr2=0x564a145c8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.984+0000 7fb2549ce740 1 --2- 172.21.15.28:0/2511188676 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145c8e50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb23c01d590 tx=0x7fb23c01d4c0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.984+0000 7fb2549ce740 1 -- 172.21.15.28:0/2511188676 shutdown_connections 2024-04-21T08:51:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.984+0000 7fb2549ce740 1 --2- 172.21.15.28:0/2511188676 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564a1463be60 0x564a1463e250 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.984+0000 7fb22affd640 0 client.25570 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:51:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.984+0000 7fb2549ce740 1 --2- 172.21.15.28:0/2511188676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a145bfc10 0x564a145c98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.984+0000 7fb2549ce740 1 --2- 172.21.15.28:0/2511188676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a145ae120 0x564a145c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.984+0000 7fb2549ce740 1 --2- 172.21.15.28:0/2511188676 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564a145ad760 0x564a145c8e50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.984+0000 7fb2549ce740 1 -- 172.21.15.28:0/2511188676 >> 172.21.15.28:0/2511188676 conn(0x564a145b0220 msgr2=0x564a145c57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:53.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.984+0000 7fb2549ce740 1 -- 172.21.15.28:0/2511188676 shutdown_connections 2024-04-21T08:51:53.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.984+0000 7fb2549ce740 1 -- 172.21.15.28:0/2511188676 wait complete. 2024-04-21T08:51:53.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.984+0000 7fb2549ce740 20 client.25570 trim_cache size 0 max 16384 2024-04-21T08:51:53.989 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:51:53.989 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:51:53.989 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:51:53.989 INFO:teuthology.orchestra.run.smithi028.stdout: "25e90a3e-4366-4a3e-aab4-171469b773f4": { 2024-04-21T08:51:53.989 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:51:53.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "25e90a3e-4366-4a3e-aab4-171469b773f4", 2024-04-21T08:51:53.990 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:51:53.990 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:51:53.990 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:51:53.990 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:51:53.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.985+0000 7fb251ddb640 1 -- 172.21.15.28:0/646262364 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb2340a3e40 msgr2=0x7fb2340a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.985+0000 7fb251ddb640 1 --2- 172.21.15.28:0/646262364 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb2340a3e40 0x7fb2340a6300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009f40 tx=0x7fb23c005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:53.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.985+0000 7fb251ddb640 1 -- 172.21.15.28:0/646262364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb24c101a40 msgr2=0x7fb24c198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:53.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.985+0000 7fb251ddb640 1 --2- 172.21.15.28:0/646262364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb24c101a40 0x7fb24c198c80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb24000e9f0 tx=0x7fb24000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.985+0000 7fb251ddb640 1 -- 172.21.15.28:0/646262364 shutdown_connections 2024-04-21T08:51:53.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.985+0000 7fb251ddb640 1 --2- 172.21.15.28:0/646262364 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb2340a3e40 0x7fb2340a6300 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.985+0000 7fb251ddb640 1 --2- 172.21.15.28:0/646262364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c106ed0 0x7fb24c19d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.985+0000 7fb251ddb640 1 --2- 172.21.15.28:0/646262364 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb24c101a40 0x7fb24c198c80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.985+0000 7fb251ddb640 1 --2- 172.21.15.28:0/646262364 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb24c1010a0 0x7fb24c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:53.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.985+0000 7fb251ddb640 1 -- 172.21.15.28:0/646262364 >> 172.21.15.28:0/646262364 conn(0x7fb24c077c40 msgr2=0x7fb24c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:53.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.985+0000 7fb251ddb640 1 -- 172.21.15.28:0/646262364 shutdown_connections 2024-04-21T08:51:53.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:51:53.985+0000 7fb251ddb640 1 -- 172.21.15.28:0/646262364 wait complete. 2024-04-21T08:51:53.995 INFO:tasks.cephfs.filesystem:scrub status for tag:25e90a3e-4366-4a3e-aab4-171469b773f4 - {'path': '/', 'tag': '25e90a3e-4366-4a3e-aab4-171469b773f4', 'options': 'recursive,force'} 2024-04-21T08:51:54.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:53 smithi152 ceph-mon[31401]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T08:51:54.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:53 smithi171 ceph-mon[31358]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-21T08:51:55.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:54 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:51:55.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:54 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:55.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:54 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:51:55.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:54 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:55.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:54 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:51:55.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:54 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:55.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:54 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:51:55.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:54 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:54 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:51:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:54 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:54 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:51:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:54 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:56.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:55 smithi152 ceph-mon[31401]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T08:51:56.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:55 smithi028 ceph-mon[28003]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T08:51:56.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:55 smithi171 ceph-mon[31358]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-04-21T08:51:57.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:56 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:51:57.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:56 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:51:57.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:56 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:51:58.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:57 smithi028 ceph-mon[28003]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-21T08:51:58.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:57 smithi152 ceph-mon[31401]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-21T08:51:58.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:57 smithi171 ceph-mon[31358]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-21T08:52:00.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.048+0000 7f92b0e3a640 1 -- 172.21.15.28:0/1374109632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ac1055c0 msgr2=0x7f92ac1059a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.048+0000 7f92b0e3a640 1 --2- 172.21.15.28:0/1374109632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ac1055c0 0x7f92ac1059a0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f929c009900 tx=0x7f929c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:52:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.048+0000 7f92b0e3a640 1 -- 172.21.15.28:0/1374109632 shutdown_connections 2024-04-21T08:52:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.048+0000 7f92b0e3a640 1 --2- 172.21.15.28:0/1374109632 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92ac1011e0 0x7f92ac10d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.048+0000 7f92b0e3a640 1 --2- 172.21.15.28:0/1374109632 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92ac100820 0x7f92ac100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.048+0000 7f92b0e3a640 1 --2- 172.21.15.28:0/1374109632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ac1055c0 0x7f92ac1059a0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.048+0000 7f92b0e3a640 1 -- 172.21.15.28:0/1374109632 >> 172.21.15.28:0/1374109632 conn(0x7f92ac0761d0 msgr2=0x7f92ac0765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.048+0000 7f92b0e3a640 1 -- 172.21.15.28:0/1374109632 shutdown_connections 2024-04-21T08:52:00.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.048+0000 7f92b0e3a640 1 -- 172.21.15.28:0/1374109632 wait complete. 2024-04-21T08:52:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.049+0000 7f92b0e3a640 1 Processor -- start 2024-04-21T08:52:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.049+0000 7f92b0e3a640 1 -- start start 2024-04-21T08:52:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92b0e3a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92ac100820 0x7f92ac1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92b0e3a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92ac1011e0 0x7f92ac198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92b0e3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ac1055c0 0x7f92ac19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92b0e3a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92ac1122f0 con 0x7f92ac1055c0 2024-04-21T08:52:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92b0e3a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f92ac112170 con 0x7f92ac100820 2024-04-21T08:52:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92b0e3a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f92ac112470 con 0x7f92ac1011e0 2024-04-21T08:52:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92aad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ac1055c0 0x7f92ac19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92a9d9b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92ac1011e0 0x7f92ac198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92aad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ac1055c0 0x7f92ac19cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55008/0 (socket says 172.21.15.28:55008) 2024-04-21T08:52:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92aad9d640 1 -- 172.21.15.28:0/1422934087 learned_addr learned my addr 172.21.15.28:0/1422934087 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92aa59c640 1 --2- 172.21.15.28:0/1422934087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92ac100820 0x7f92ac1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92aad9d640 1 -- 172.21.15.28:0/1422934087 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92ac1011e0 msgr2=0x7f92ac198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92aad9d640 1 --2- 172.21.15.28:0/1422934087 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92ac1011e0 0x7f92ac198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92aad9d640 1 -- 172.21.15.28:0/1422934087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92ac100820 msgr2=0x7f92ac1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92aad9d640 1 --2- 172.21.15.28:0/1422934087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92ac100820 0x7f92ac1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92aad9d640 1 -- 172.21.15.28:0/1422934087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92ac19d6e0 con 0x7f92ac1055c0 2024-04-21T08:52:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.050+0000 7f92aad9d640 1 --2- 172.21.15.28:0/1422934087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ac1055c0 0x7f92ac19cfb0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f92a400eb10 tx=0x7f92a400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.051+0000 7f929b7fe640 1 -- 172.21.15.28:0/1422934087 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a4004b20 con 0x7f92ac1055c0 2024-04-21T08:52:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.051+0000 7f92b0e3a640 1 -- 172.21.15.28:0/1422934087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92ac19d9d0 con 0x7f92ac1055c0 2024-04-21T08:52:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.051+0000 7f92b0e3a640 1 -- 172.21.15.28:0/1422934087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92ac1a5260 con 0x7f92ac1055c0 2024-04-21T08:52:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.051+0000 7f929b7fe640 1 -- 172.21.15.28:0/1422934087 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a4004cc0 con 0x7f92ac1055c0 2024-04-21T08:52:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.051+0000 7f929b7fe640 1 -- 172.21.15.28:0/1422934087 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a4010670 con 0x7f92ac1055c0 2024-04-21T08:52:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.053+0000 7f92b3a20740 1 Processor -- start 2024-04-21T08:52:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.054+0000 7f929b7fe640 1 -- 172.21.15.28:0/1422934087 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138873+0+0 (secure 0 0 0) 0x7f92a4005020 con 0x7f92ac1055c0 2024-04-21T08:52:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.054+0000 7f92b3a20740 1 -- start start 2024-04-21T08:52:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92b3a20740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2e8d0 0x55799ab2d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92b3a20740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab2db60 0x55799ab2dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92b3a20740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab370e0 0x55799ab394d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92b3a20740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55799aa928d0 con 0x55799ab2e8d0 2024-04-21T08:52:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92b3a20740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55799aa2fe80 con 0x55799ab370e0 2024-04-21T08:52:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92b3a20740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55799aa8e890 con 0x55799ab2db60 2024-04-21T08:52:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2e8d0 0x55799ab2d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2e8d0 0x55799ab2d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55012/0 (socket says 172.21.15.28:55012) 2024-04-21T08:52:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92a9d9b640 1 -- 172.21.15.28:0/1745633213 learned_addr learned my addr 172.21.15.28:0/1745633213 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92aa59c640 1 --2- 172.21.15.28:0/1745633213 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab2db60 0x55799ab2dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92a9d9b640 1 -- 172.21.15.28:0/1745633213 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab2db60 msgr2=0x55799ab2dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f929b7fe640 1 --2- 172.21.15.28:0/1422934087 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f92940a3f60 0x7f92940a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92a9d9b640 1 --2- 172.21.15.28:0/1745633213 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab2db60 0x55799ab2dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92a9d9b640 1 -- 172.21.15.28:0/1745633213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab370e0 msgr2=0x55799ab394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92a9d9b640 1 --2- 172.21.15.28:0/1745633213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab370e0 0x55799ab394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92a9d9b640 1 -- 172.21.15.28:0/1745633213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55799aa51320 con 0x55799ab2e8d0 2024-04-21T08:52:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92aad9d640 1 --2- 172.21.15.28:0/1422934087 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f92940a3f60 0x7f92940a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f92a9d9b640 1 --2- 172.21.15.28:0/1745633213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2e8d0 0x55799ab2d620 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f92a000bd50 tx=0x7f92a00127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e146c4b555213488 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.055+0000 7f929b7fe640 1 -- 172.21.15.28:0/1422934087 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f92a40d1270 con 0x7f92ac1055c0 2024-04-21T08:52:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.056+0000 7f9298ff9640 1 -- 172.21.15.28:0/1745633213 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a00120d0 con 0x55799ab2e8d0 2024-04-21T08:52:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.056+0000 7f9298ff9640 1 -- 172.21.15.28:0/1745633213 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a0016070 con 0x55799ab2e8d0 2024-04-21T08:52:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.056+0000 7f9298ff9640 1 -- 172.21.15.28:0/1745633213 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a00107a0 con 0x55799ab2e8d0 2024-04-21T08:52:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.057+0000 7f92aad9d640 1 --2- 172.21.15.28:0/1422934087 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f92940a3f60 0x7f92940a6420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f92a40ce500 tx=0x7f92a40672d0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.057+0000 7f92b3a20740 1 -- 172.21.15.28:0/1745633213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2e8d0 msgr2=0x55799ab2d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.057+0000 7f92b3a20740 1 --2- 172.21.15.28:0/1745633213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2e8d0 0x55799ab2d620 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f92a000bd50 tx=0x7f92a00127c0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.057+0000 7f92b3a20740 1 -- 172.21.15.28:0/1745633213 shutdown_connections 2024-04-21T08:52:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.057+0000 7f92b3a20740 1 --2- 172.21.15.28:0/1745633213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab370e0 0x55799ab394d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.057+0000 7f92b3a20740 1 --2- 172.21.15.28:0/1745633213 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab2db60 0x55799ab2dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.057+0000 7f92b3a20740 1 --2- 172.21.15.28:0/1745633213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2e8d0 0x55799ab2d620 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.057+0000 7f92b3a20740 1 -- 172.21.15.28:0/1745633213 >> 172.21.15.28:0/1745633213 conn(0x55799aaf4c90 msgr2=0x55799ab28850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.057+0000 7f92b3a20740 1 -- 172.21.15.28:0/1745633213 shutdown_connections 2024-04-21T08:52:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.057+0000 7f92b3a20740 1 -- 172.21.15.28:0/1745633213 wait complete. 2024-04-21T08:52:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.057+0000 7f92b3a20740 1 -- start start 2024-04-21T08:52:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.057+0000 7f92b3a20740 1 Processor -- start 2024-04-21T08:52:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92b3a20740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab36a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92b3a20740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab2e8d0 0x55799ab33e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92b3a20740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab370e0 0x55799ab343b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92b3a20740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55799ab39e10 con 0x55799ab2db60 2024-04-21T08:52:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92b3a20740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55799aa1aeb0 con 0x55799ab370e0 2024-04-21T08:52:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92b3a20740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55799ab39f90 con 0x55799ab2e8d0 2024-04-21T08:52:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92b3a20740 10 client.0 _ll_register_callbacks cb 0x55799aa47b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab36a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab36a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55028/0 (socket says 172.21.15.28:55028) 2024-04-21T08:52:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92a9d9b640 1 -- 172.21.15.28:0/4063407409 learned_addr learned my addr 172.21.15.28:0/4063407409 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92aa59c640 1 --2- 172.21.15.28:0/4063407409 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab2e8d0 0x55799ab33e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92a9d9b640 1 --2- 172.21.15.28:0/4063407409 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab370e0 0x55799ab343b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92a9d9b640 1 -- 172.21.15.28:0/4063407409 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab2e8d0 msgr2=0x55799ab33e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92a9d9b640 1 --2- 172.21.15.28:0/4063407409 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab2e8d0 0x55799ab33e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92a9d9b640 1 -- 172.21.15.28:0/4063407409 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab370e0 msgr2=0x55799ab343b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92a9d9b640 1 --2- 172.21.15.28:0/4063407409 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab370e0 0x55799ab343b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92a9d9b640 1 -- 172.21.15.28:0/4063407409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55799a7a5210 con 0x55799ab2db60 2024-04-21T08:52:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f92a9d9b640 1 --2- 172.21.15.28:0/4063407409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab36a80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f92a0005940 tx=0x7f92a0005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6830e155219db1e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.058+0000 7f9291ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.059+0000 7f9291ffb640 1 -- 172.21.15.28:0/4063407409 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a0004450 con 0x55799ab2db60 2024-04-21T08:52:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.059+0000 7f9291ffb640 1 -- 172.21.15.28:0/4063407409 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f92a0005020 con 0x55799ab2db60 2024-04-21T08:52:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.059+0000 7f92b3a20740 1 -- 172.21.15.28:0/4063407409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92a0016070 con 0x55799ab2db60 2024-04-21T08:52:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.059+0000 7f9291ffb640 1 -- 172.21.15.28:0/4063407409 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f92a0030050 con 0x55799ab2db60 2024-04-21T08:52:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.059+0000 7f92b3a20740 10 client.16164 fetch_fsmap learned FSMap version 25 2024-04-21T08:52:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.059+0000 7f92b3a20740 1 -- 172.21.15.28:0/4063407409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55799a871620 con 0x55799ab2db60 2024-04-21T08:52:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.059+0000 7f9291ffb640 1 -- 172.21.15.28:0/4063407409 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a0016070 con 0x55799ab2db60 2024-04-21T08:52:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.059+0000 7f9291ffb640 1 -- 172.21.15.28:0/4063407409 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f92a00046c0 con 0x55799ab2db60 2024-04-21T08:52:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.059+0000 7f92b3a20740 10 client.16164 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:52:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.059+0000 7f92b3a20740 10 client.16164 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.059+0000 7f92b3a20740 20 client.16164 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.059+0000 7f92b3a20740 1 --2- 172.21.15.28:0/4063407409 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55799ab47130 0x55799ab49520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.059+0000 7f92b3a20740 4 client.16164 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.059+0000 7f92b3a20740 1 -- 172.21.15.28:0/4063407409 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55799aaa6880 con 0x55799ab47130 2024-04-21T08:52:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.060+0000 7f92aa59c640 1 --2- 172.21.15.28:0/4063407409 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55799ab47130 0x55799ab49520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.060+0000 7f92aa59c640 1 --2- 172.21.15.28:0/4063407409 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55799ab47130 0x55799ab49520 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85bbe868d040fc6d server_cookie=fbc1b76f66e0e91b in_seq=0 out_seq=0 2024-04-21T08:52:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.060+0000 7f9291ffb640 10 client.16164 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.061+0000 7f9291ffb640 1 -- 172.21.15.28:0/4063407409 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55799aaa6880 con 0x55799ab47130 2024-04-21T08:52:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.061+0000 7f9291ffb640 10 client.16164 handle_command_reply: tid=0 2024-04-21T08:52:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.061+0000 7f92b3a20740 1 client.16164 shutdown 2024-04-21T08:52:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.061+0000 7f92b3a20740 1 -- 172.21.15.28:0/4063407409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 msgr2=0x55799ab36a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.061+0000 7f92b3a20740 1 --2- 172.21.15.28:0/4063407409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab36a80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f92a0005940 tx=0x7f92a0005850 comp rx=0 tx=0).stop 2024-04-21T08:52:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.061+0000 7f92b3a20740 1 -- 172.21.15.28:0/4063407409 shutdown_connections 2024-04-21T08:52:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.061+0000 7f92b3a20740 1 --2- 172.21.15.28:0/4063407409 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55799ab47130 0x55799ab49520 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.061+0000 7f9291ffb640 0 client.16164 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.061+0000 7f92b3a20740 1 --2- 172.21.15.28:0/4063407409 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab370e0 0x55799ab343b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.061+0000 7f92b3a20740 1 --2- 172.21.15.28:0/4063407409 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab2e8d0 0x55799ab33e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.061+0000 7f92b3a20740 1 --2- 172.21.15.28:0/4063407409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab36a80 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.061+0000 7f92b3a20740 1 -- 172.21.15.28:0/4063407409 >> 172.21.15.28:0/4063407409 conn(0x55799aaf4c90 msgr2=0x55799ab2a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.061+0000 7f92b3a20740 1 -- 172.21.15.28:0/4063407409 shutdown_connections 2024-04-21T08:52:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.061+0000 7f92b3a20740 1 -- 172.21.15.28:0/4063407409 wait complete. 2024-04-21T08:52:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.062+0000 7f92b3a20740 20 client.16164 trim_cache size 0 max 16384 2024-04-21T08:52:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.090+0000 7f92b3a20740 1 Processor -- start 2024-04-21T08:52:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.090+0000 7f92b3a20740 1 -- start start 2024-04-21T08:52:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.090+0000 7f92b3a20740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab2d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.090+0000 7f92b3a20740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab2e8d0 0x55799ab32f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.090+0000 7f92b3a20740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab370e0 0x55799ab2a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.090+0000 7f92b3a20740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f928c012800 con 0x55799ab2db60 2024-04-21T08:52:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.090+0000 7f92b3a20740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f928c0100d0 con 0x55799ab2e8d0 2024-04-21T08:52:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.090+0000 7f92b3a20740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f928c00df20 con 0x55799ab370e0 2024-04-21T08:52:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.091+0000 7f92a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab2d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.091+0000 7f92a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab2d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55044/0 (socket says 172.21.15.28:55044) 2024-04-21T08:52:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.091+0000 7f92a9d9b640 1 -- 172.21.15.28:0/2620604782 learned_addr learned my addr 172.21.15.28:0/2620604782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.091+0000 7f92aa59c640 1 --2- 172.21.15.28:0/2620604782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab2e8d0 0x55799ab32f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.091+0000 7f92a9d9b640 1 --2- 172.21.15.28:0/2620604782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab370e0 0x55799ab2a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.091+0000 7f92a9d9b640 1 -- 172.21.15.28:0/2620604782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab370e0 msgr2=0x55799ab2a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.091+0000 7f92a9d9b640 1 --2- 172.21.15.28:0/2620604782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab370e0 0x55799ab2a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.091+0000 7f92a9d9b640 1 -- 172.21.15.28:0/2620604782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab2e8d0 msgr2=0x55799ab32f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.091+0000 7f92a9d9b640 1 --2- 172.21.15.28:0/2620604782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab2e8d0 0x55799ab32f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.091+0000 7f92a9d9b640 1 -- 172.21.15.28:0/2620604782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55799ab34720 con 0x55799ab2db60 2024-04-21T08:52:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.091+0000 7f92a9d9b640 1 --2- 172.21.15.28:0/2620604782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab2d4e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f92a000b7f0 tx=0x7f92a000b570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3d04fa993145be server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.091+0000 7f9293fff640 1 -- 172.21.15.28:0/2620604782 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a0039070 con 0x55799ab2db60 2024-04-21T08:52:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.091+0000 7f9293fff640 1 -- 172.21.15.28:0/2620604782 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a0016070 con 0x55799ab2db60 2024-04-21T08:52:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.091+0000 7f9293fff640 1 -- 172.21.15.28:0/2620604782 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a0004890 con 0x55799ab2db60 2024-04-21T08:52:00.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.092+0000 7f92b3a20740 1 -- 172.21.15.28:0/2620604782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 msgr2=0x55799ab2d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.092+0000 7f92b3a20740 1 --2- 172.21.15.28:0/2620604782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab2d4e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f92a000b7f0 tx=0x7f92a000b570 comp rx=0 tx=0).stop 2024-04-21T08:52:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.092+0000 7f92b3a20740 1 -- 172.21.15.28:0/2620604782 shutdown_connections 2024-04-21T08:52:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.092+0000 7f92b3a20740 1 --2- 172.21.15.28:0/2620604782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab370e0 0x55799ab2a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.092+0000 7f92b3a20740 1 --2- 172.21.15.28:0/2620604782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab2e8d0 0x55799ab32f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.092+0000 7f92b3a20740 1 --2- 172.21.15.28:0/2620604782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab2d4e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.092+0000 7f92b3a20740 1 -- 172.21.15.28:0/2620604782 >> 172.21.15.28:0/2620604782 conn(0x55799aaf4c90 msgr2=0x55799ab29c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.092+0000 7f92b3a20740 1 -- 172.21.15.28:0/2620604782 shutdown_connections 2024-04-21T08:52:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.092+0000 7f92b3a20740 1 -- 172.21.15.28:0/2620604782 wait complete. 2024-04-21T08:52:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.093+0000 7f92b3a20740 1 -- start start 2024-04-21T08:52:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.093+0000 7f92b3a20740 1 Processor -- start 2024-04-21T08:52:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.093+0000 7f92b3a20740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab3f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.093+0000 7f92b3a20740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab2e8d0 0x55799ab3a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.093+0000 7f92b3a20740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab370e0 0x55799ab3ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.093+0000 7f92b3a20740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f928c00e590 con 0x55799ab2db60 2024-04-21T08:52:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.093+0000 7f92b3a20740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f928c00fa00 con 0x55799ab2e8d0 2024-04-21T08:52:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.093+0000 7f92b3a20740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f928c00ec60 con 0x55799ab370e0 2024-04-21T08:52:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.093+0000 7f92b3a20740 10 client.0 _ll_register_callbacks cb 0x7f92a000bc50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.093+0000 7f92a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab3f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.093+0000 7f92a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab3f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55046/0 (socket says 172.21.15.28:55046) 2024-04-21T08:52:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.093+0000 7f92a9d9b640 1 -- 172.21.15.28:0/125795545 learned_addr learned my addr 172.21.15.28:0/125795545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.093+0000 7f92aa59c640 1 --2- 172.21.15.28:0/125795545 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab2e8d0 0x55799ab3a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.094+0000 7f92a9d9b640 1 -- 172.21.15.28:0/125795545 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab370e0 msgr2=0x55799ab3ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.094+0000 7f92a9d9b640 1 --2- 172.21.15.28:0/125795545 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab370e0 0x55799ab3ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.094+0000 7f92a9d9b640 1 -- 172.21.15.28:0/125795545 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab2e8d0 msgr2=0x55799ab3a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.094+0000 7f92a9d9b640 1 --2- 172.21.15.28:0/125795545 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab2e8d0 0x55799ab3a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.094+0000 7f92a9d9b640 1 -- 172.21.15.28:0/125795545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55799ab3b310 con 0x55799ab2db60 2024-04-21T08:52:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.094+0000 7f92a9d9b640 1 --2- 172.21.15.28:0/125795545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab3f3d0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f92a000b8a0 tx=0x7f92a000b400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df919a5a223c992 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.094+0000 7f9291ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.094+0000 7f9291ffb640 1 -- 172.21.15.28:0/125795545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a0039070 con 0x55799ab2db60 2024-04-21T08:52:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.094+0000 7f9291ffb640 1 -- 172.21.15.28:0/125795545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f92a0004610 con 0x55799ab2db60 2024-04-21T08:52:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.094+0000 7f92b3a20740 1 -- 172.21.15.28:0/125795545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92a0016070 con 0x55799ab2db60 2024-04-21T08:52:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.094+0000 7f9291ffb640 1 -- 172.21.15.28:0/125795545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f92a001b470 con 0x55799ab2db60 2024-04-21T08:52:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.094+0000 7f92b3a20740 10 client.16170 fetch_fsmap learned FSMap version 25 2024-04-21T08:52:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.094+0000 7f92b3a20740 1 -- 172.21.15.28:0/125795545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55799abad120 con 0x55799ab2db60 2024-04-21T08:52:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.095+0000 7f9291ffb640 1 -- 172.21.15.28:0/125795545 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92a001b610 con 0x55799ab2db60 2024-04-21T08:52:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.095+0000 7f9291ffb640 1 -- 172.21.15.28:0/125795545 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f92a003e030 con 0x55799ab2db60 2024-04-21T08:52:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.095+0000 7f92b3a20740 10 client.16170 fetch_fsmap finished waiting for FSMap version 25 2024-04-21T08:52:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.095+0000 7f92b3a20740 10 client.16170 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.095+0000 7f92b3a20740 20 client.16170 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.095+0000 7f92b3a20740 1 --2- 172.21.15.28:0/125795545 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55799abad8d0 0x55799abafcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.095+0000 7f92b3a20740 4 client.16170 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:52:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.095+0000 7f92b3a20740 1 -- 172.21.15.28:0/125795545 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f92a0016070 con 0x55799abad8d0 2024-04-21T08:52:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.095+0000 7f92aa59c640 1 --2- 172.21.15.28:0/125795545 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55799abad8d0 0x55799abafcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.096+0000 7f92aa59c640 1 --2- 172.21.15.28:0/125795545 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55799abad8d0 0x55799abafcf0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de89d094d995fed4 server_cookie=d154aaaad5d23da4 in_seq=0 out_seq=0 2024-04-21T08:52:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.096+0000 7f9291ffb640 10 client.16170 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:00.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.103+0000 7f9291ffb640 1 -- 172.21.15.28:0/125795545 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f92a0016070 con 0x55799abad8d0 2024-04-21T08:52:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.103+0000 7f9291ffb640 10 client.16170 handle_command_reply: tid=0 2024-04-21T08:52:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.103+0000 7f92b3a20740 1 client.16170 shutdown 2024-04-21T08:52:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.103+0000 7f92b3a20740 1 -- 172.21.15.28:0/125795545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 msgr2=0x55799ab3f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.103+0000 7f92b3a20740 1 --2- 172.21.15.28:0/125795545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab3f3d0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f92a000b8a0 tx=0x7f92a000b400 comp rx=0 tx=0).stop 2024-04-21T08:52:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.103+0000 7f92b3a20740 1 -- 172.21.15.28:0/125795545 shutdown_connections 2024-04-21T08:52:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.103+0000 7f92b3a20740 1 --2- 172.21.15.28:0/125795545 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55799abad8d0 0x55799abafcf0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.103+0000 7f92b3a20740 1 --2- 172.21.15.28:0/125795545 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55799ab370e0 0x55799ab3ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.103+0000 7f9291ffb640 0 client.16170 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.103+0000 7f92b3a20740 1 --2- 172.21.15.28:0/125795545 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55799ab2e8d0 0x55799ab3a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.103+0000 7f92b3a20740 1 --2- 172.21.15.28:0/125795545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55799ab2db60 0x55799ab3f3d0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.103+0000 7f92b3a20740 1 -- 172.21.15.28:0/125795545 >> 172.21.15.28:0/125795545 conn(0x55799aaf4c90 msgr2=0x55799ab29b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.104+0000 7f92b3a20740 1 -- 172.21.15.28:0/125795545 shutdown_connections 2024-04-21T08:52:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.104+0000 7f92b3a20740 1 -- 172.21.15.28:0/125795545 wait complete. 2024-04-21T08:52:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.104+0000 7f92b3a20740 20 client.16170 trim_cache size 0 max 16384 2024-04-21T08:52:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:52:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:52:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:52:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:52:00.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.104+0000 7f92b0e3a640 1 -- 172.21.15.28:0/1422934087 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f92940a3f60 msgr2=0x7f92940a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:00.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.104+0000 7f92b0e3a640 1 --2- 172.21.15.28:0/1422934087 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f92940a3f60 0x7f92940a6420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f92a40ce500 tx=0x7f92a40672d0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.104+0000 7f92b0e3a640 1 -- 172.21.15.28:0/1422934087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ac1055c0 msgr2=0x7f92ac19cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:00.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.104+0000 7f92b0e3a640 1 --2- 172.21.15.28:0/1422934087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ac1055c0 0x7f92ac19cfb0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f92a400eb10 tx=0x7f92a400c2d0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.104+0000 7f92b0e3a640 1 -- 172.21.15.28:0/1422934087 shutdown_connections 2024-04-21T08:52:00.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.104+0000 7f92b0e3a640 1 --2- 172.21.15.28:0/1422934087 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f92940a3f60 0x7f92940a6420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.104+0000 7f92b0e3a640 1 --2- 172.21.15.28:0/1422934087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92ac1055c0 0x7f92ac19cfb0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.104+0000 7f92b0e3a640 1 --2- 172.21.15.28:0/1422934087 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92ac1011e0 0x7f92ac198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.104+0000 7f92b0e3a640 1 --2- 172.21.15.28:0/1422934087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92ac100820 0x7f92ac1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.104+0000 7f92b0e3a640 1 -- 172.21.15.28:0/1422934087 >> 172.21.15.28:0/1422934087 conn(0x7f92ac0761d0 msgr2=0x7f92ac0fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.104+0000 7f92b0e3a640 1 -- 172.21.15.28:0/1422934087 shutdown_connections 2024-04-21T08:52:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:00.105+0000 7f92b0e3a640 1 -- 172.21.15.28:0/1422934087 wait complete. 2024-04-21T08:52:00.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:51:59 smithi152 ceph-mon[31401]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-21T08:52:00.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:51:59 smithi028 ceph-mon[28003]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-21T08:52:00.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:51:59 smithi171 ceph-mon[31358]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-21T08:52:01.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:52:00 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:52:00.849+0000 7f635ba0a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:52:01.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:52:00 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:52:00.850+0000 7f635ba0a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:52:01.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:52:00 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:52:00.850+0000 7f635ba0a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:52:01.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:52:00 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:52:00.850+0000 7f635ba0a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:52:01.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 08:52:00 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x[28237]: 2024-04-21T08:52:00.850+0000 7f635ba0a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:52:02.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:01 smithi152 ceph-mon[31401]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T08:52:02.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:01 smithi028 ceph-mon[28003]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T08:52:02.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:01 smithi171 ceph-mon[31358]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-21T08:52:04.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:03 smithi152 ceph-mon[31401]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-04-21T08:52:04.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:03 smithi152 ceph-mon[31401]: mgrmap e25: x(active, since 52s), standbys: y, z 2024-04-21T08:52:04.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:03 smithi028 ceph-mon[28003]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-04-21T08:52:04.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:03 smithi028 ceph-mon[28003]: mgrmap e25: x(active, since 52s), standbys: y, z 2024-04-21T08:52:04.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:03 smithi171 ceph-mon[31358]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 24 KiB/s wr, 27 op/s 2024-04-21T08:52:04.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:03 smithi171 ceph-mon[31358]: mgrmap e25: x(active, since 52s), standbys: y, z 2024-04-21T08:52:05.938 INFO:tasks.workunit.client.0.smithi028.stderr:Note: switching to '4f6bc5273643aaf5660909e85400647a8951c10b'. 2024-04-21T08:52:05.938 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-21T08:52:05.938 INFO:tasks.workunit.client.0.smithi028.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-21T08:52:05.938 INFO:tasks.workunit.client.0.smithi028.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-21T08:52:05.938 INFO:tasks.workunit.client.0.smithi028.stderr:state without impacting any branches by switching back to a branch. 2024-04-21T08:52:05.938 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-21T08:52:05.938 INFO:tasks.workunit.client.0.smithi028.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-21T08:52:05.938 INFO:tasks.workunit.client.0.smithi028.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-21T08:52:05.938 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-21T08:52:05.938 INFO:tasks.workunit.client.0.smithi028.stderr: git switch -c 2024-04-21T08:52:05.939 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-21T08:52:05.939 INFO:tasks.workunit.client.0.smithi028.stderr:Or undo this operation with: 2024-04-21T08:52:05.939 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-21T08:52:05.939 INFO:tasks.workunit.client.0.smithi028.stderr: git switch - 2024-04-21T08:52:05.939 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-21T08:52:05.939 INFO:tasks.workunit.client.0.smithi028.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-21T08:52:05.939 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-21T08:52:05.939 INFO:tasks.workunit.client.0.smithi028.stderr:HEAD is now at 4f6bc527364 use ubuntu for fs/workload tests 2024-04-21T08:52:05.942 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-21T08:52:06.194 INFO:tasks.workunit.client.0.smithi028.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-21T08:52:06.196 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-21T08:52:06.196 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-21T08:52:06.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:05 smithi152 ceph-mon[31401]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-21T08:52:06.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:05 smithi028 ceph-mon[28003]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-21T08:52:06.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:05 smithi171 ceph-mon[31358]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-21T08:52:06.791 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-21T08:52:06.848 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-21T08:52:06.890 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-21T08:52:06.892 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-21T08:52:06.892 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-21T08:52:06.932 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-21T08:52:06.936 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T08:52:06.936 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-21T08:52:06.952 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-04-21T08:52:06.953 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-04-21T08:52:06.953 DEBUG:teuthology.orchestra.run.smithi028:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f6bc5273643aaf5660909e85400647a8951c10b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-21T08:52:07.015 INFO:tasks.workunit.client.0.smithi028.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-21T08:52:07.015 INFO:tasks.workunit.client.0.smithi028.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-04-21T08:52:07.015 INFO:tasks.workunit.client.0.smithi028.stderr:+ trap -- retry EXIT 2024-04-21T08:52:07.016 INFO:tasks.workunit.client.0.smithi028.stderr:+ rm -rf ffsb 2024-04-21T08:52:07.016 INFO:tasks.workunit.client.0.smithi028.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-04-21T08:52:07.019 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into 'ffsb'... 2024-04-21T08:52:08.019 INFO:tasks.workunit.client.0.smithi028.stderr:+ trap - EXIT 2024-04-21T08:52:08.019 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd ffsb 2024-04-21T08:52:08.019 INFO:tasks.workunit.client.0.smithi028.stderr:+ ./configure 2024-04-21T08:52:08.174 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-21T08:52:08.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:07 smithi028 ceph-mon[28003]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-04-21T08:52:08.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:07 smithi152 ceph-mon[31401]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-04-21T08:52:08.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:07 smithi171 ceph-mon[31358]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 26 op/s 2024-04-21T08:52:09.182 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether build environment is sane... yes 2024-04-21T08:52:09.193 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-04-21T08:52:09.193 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gawk... gawk 2024-04-21T08:52:09.210 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether make sets $(MAKE)... yes 2024-04-21T08:52:09.243 INFO:tasks.workunit.client.0.smithi028.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-21T08:52:09.243 INFO:tasks.workunit.client.0.smithi028.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-21T08:52:09.256 INFO:tasks.workunit.client.0.smithi028.stdout:checking for style of include used by make... GNU 2024-04-21T08:52:09.260 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... gcc 2024-04-21T08:52:09.317 INFO:tasks.workunit.client.0.smithi028.stdout:checking for C compiler default output file name... a.out 2024-04-21T08:52:09.318 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether the C compiler works... yes 2024-04-21T08:52:09.321 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are cross compiling... no 2024-04-21T08:52:09.360 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of executables... 2024-04-21T08:52:09.394 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of object files... o 2024-04-21T08:52:09.427 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... yes 2024-04-21T08:52:09.459 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... yes 2024-04-21T08:52:09.529 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-21T08:52:09.634 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... gcc3 2024-04-21T08:52:09.692 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to run the C preprocessor... gcc -E 2024-04-21T08:52:09.736 INFO:tasks.workunit.client.0.smithi028.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-21T08:52:09.737 INFO:tasks.workunit.client.0.smithi028.stdout:checking for egrep... /usr/bin/grep -E 2024-04-21T08:52:09.747 INFO:tasks.workunit.client.0.smithi028.stdout:checking for AIX... no 2024-04-21T08:52:09.770 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... (cached) gcc 2024-04-21T08:52:09.777 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-21T08:52:09.779 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-21T08:52:09.779 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-21T08:52:09.779 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-21T08:52:09.834 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lm... yes 2024-04-21T08:52:09.886 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lpthread... yes 2024-04-21T08:52:10.048 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ANSI C header files... yes 2024-04-21T08:52:10.103 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-21T08:52:10.149 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/types.h... yes 2024-04-21T08:52:10.196 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/stat.h... yes 2024-04-21T08:52:10.243 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdlib.h... yes 2024-04-21T08:52:10.290 INFO:tasks.workunit.client.0.smithi028.stdout:checking for string.h... yes 2024-04-21T08:52:10.339 INFO:tasks.workunit.client.0.smithi028.stdout:checking for memory.h... yes 2024-04-21T08:52:10.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:09 smithi152 ceph-mon[31401]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T08:52:10.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:09 smithi028 ceph-mon[28003]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T08:52:10.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:09 smithi171 ceph-mon[31358]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T08:52:10.386 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strings.h... yes 2024-04-21T08:52:10.434 INFO:tasks.workunit.client.0.smithi028.stdout:checking for inttypes.h... yes 2024-04-21T08:52:10.482 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... yes 2024-04-21T08:52:10.531 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... yes 2024-04-21T08:52:10.586 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h usability... yes 2024-04-21T08:52:10.608 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h presence... yes 2024-04-21T08:52:10.608 INFO:tasks.workunit.client.0.smithi028.stdout:checking for pthread.h... yes 2024-04-21T08:52:10.661 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h usability... yes 2024-04-21T08:52:10.682 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h presence... yes 2024-04-21T08:52:10.682 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fcntl.h... yes 2024-04-21T08:52:10.734 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h usability... yes 2024-04-21T08:52:10.754 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h presence... yes 2024-04-21T08:52:10.754 INFO:tasks.workunit.client.0.smithi028.stdout:checking for limits.h... yes 2024-04-21T08:52:10.769 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... (cached) yes 2024-04-21T08:52:10.811 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h usability... yes 2024-04-21T08:52:10.832 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h presence... yes 2024-04-21T08:52:10.832 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/time.h... yes 2024-04-21T08:52:10.847 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... (cached) yes 2024-04-21T08:52:10.890 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h usability... yes 2024-04-21T08:52:10.910 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h presence... yes 2024-04-21T08:52:10.910 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/vfs.h... yes 2024-04-21T08:52:10.965 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h usability... no 2024-04-21T08:52:10.991 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h presence... no 2024-04-21T08:52:10.991 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/limits.h... no 2024-04-21T08:52:11.025 INFO:tasks.workunit.client.0.smithi028.stdout:checking for an ANSI C-conforming const... yes 2024-04-21T08:52:11.060 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-21T08:52:11.060 INFO:tasks.workunit.client.0.smithi028.stdout:checking for function prototypes... yes 2024-04-21T08:52:11.091 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether setvbuf arguments are reversed... no 2024-04-21T08:52:11.173 INFO:tasks.workunit.client.0.smithi028.stdout:checking for system... yes 2024-04-21T08:52:11.237 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gettimeofday... yes 2024-04-21T08:52:11.300 INFO:tasks.workunit.client.0.smithi028.stdout:checking for mkdir... yes 2024-04-21T08:52:11.366 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strerror... yes 2024-04-21T08:52:11.429 INFO:tasks.workunit.client.0.smithi028.stdout:checking for open64... yes 2024-04-21T08:52:11.491 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stat64... yes 2024-04-21T08:52:11.566 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fseeko64... yes 2024-04-21T08:52:11.626 INFO:tasks.workunit.client.0.smithi028.stdout:checking for lrand48_r... yes 2024-04-21T08:52:11.688 INFO:tasks.workunit.client.0.smithi028.stdout:checking for srand48_r... yes 2024-04-21T08:52:11.713 INFO:tasks.workunit.client.0.smithi028.stdout:configure: creating ./config.status 2024-04-21T08:52:11.812 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating Makefile 2024-04-21T08:52:11.826 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating config.h 2024-04-21T08:52:11.843 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing depfiles commands 2024-04-21T08:52:11.892 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing default commands 2024-04-21T08:52:11.940 INFO:tasks.workunit.client.0.smithi028.stderr:+ make 2024-04-21T08:52:11.947 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-04-21T08:52:11.949 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-04-21T08:52:11.949 INFO:tasks.workunit.client.0.smithi028.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-04-21T08:52:11.949 INFO:tasks.workunit.client.0.smithi028.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-04-21T08:52:11.950 INFO:tasks.workunit.client.0.smithi028.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-04-21T08:52:11.950 INFO:tasks.workunit.client.0.smithi028.stderr: any GNU archive site. 2024-04-21T08:52:11.951 INFO:tasks.workunit.client.0.smithi028.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-04-21T08:52:11.952 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-04-21T08:52:11.953 INFO:tasks.workunit.client.0.smithi028.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-04-21T08:52:11.953 INFO:tasks.workunit.client.0.smithi028.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-04-21T08:52:11.953 INFO:tasks.workunit.client.0.smithi028.stderr: You might want to install the `Automake' and `Perl' packages. 2024-04-21T08:52:11.954 INFO:tasks.workunit.client.0.smithi028.stderr: Grab them from any GNU archive site. 2024-04-21T08:52:12.076 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-04-21T08:52:12.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:11 smithi152 ceph-mon[31401]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T08:52:12.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:11 smithi028 ceph-mon[28003]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T08:52:12.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:11 smithi171 ceph-mon[31358]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-21T08:52:12.386 INFO:tasks.workunit.client.0.smithi028.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-04-21T08:52:12.386 INFO:tasks.workunit.client.0.smithi028.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-04-21T08:52:12.386 INFO:tasks.workunit.client.0.smithi028.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-04-21T08:52:12.386 INFO:tasks.workunit.client.0.smithi028.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-04-21T08:52:12.509 INFO:tasks.workunit.client.0.smithi028.stdout:/bin/sh ./config.status --recheck 2024-04-21T08:52:12.682 INFO:tasks.workunit.client.0.smithi028.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-04-21T08:52:12.784 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-21T08:52:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.198+0000 7f1b0504e640 1 -- 172.21.15.28:0/4040495965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b001031e0 msgr2=0x7f1b001035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.198+0000 7f1b0504e640 1 --2- 172.21.15.28:0/4040495965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b001031e0 0x7f1b001035c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f1af0009ca0 tx=0x7f1af002c230 comp rx=0 tx=0).stop 2024-04-21T08:52:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.198+0000 7f1b0504e640 1 -- 172.21.15.28:0/4040495965 shutdown_connections 2024-04-21T08:52:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.198+0000 7f1b0504e640 1 --2- 172.21.15.28:0/4040495965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b0010c520 0x7f1b0010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.198+0000 7f1b0504e640 1 --2- 172.21.15.28:0/4040495965 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b00103b00 0x7f1b0010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.198+0000 7f1b0504e640 1 --2- 172.21.15.28:0/4040495965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b001031e0 0x7f1b001035c0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.198+0000 7f1b0504e640 1 -- 172.21.15.28:0/4040495965 >> 172.21.15.28:0/4040495965 conn(0x7f1b00077fd0 msgr2=0x7f1b000783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.198+0000 7f1b0504e640 1 -- 172.21.15.28:0/4040495965 shutdown_connections 2024-04-21T08:52:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.198+0000 7f1b0504e640 1 -- 172.21.15.28:0/4040495965 wait complete. 2024-04-21T08:52:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.199+0000 7f1b0504e640 1 Processor -- start 2024-04-21T08:52:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.199+0000 7f1b0504e640 1 -- start start 2024-04-21T08:52:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.199+0000 7f1b0504e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b001031e0 0x7f1b00198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.199+0000 7f1b0504e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b00103b00 0x7f1b00198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.199+0000 7f1b0504e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b0010c520 0x7f1b0019cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.199+0000 7f1b0504e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001121b0 con 0x7f1b00103b00 2024-04-21T08:52:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.199+0000 7f1b0504e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1b00112030 con 0x7f1b001031e0 2024-04-21T08:52:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.199+0000 7f1b0504e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1b00112330 con 0x7f1b0010c520 2024-04-21T08:52:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.199+0000 7f1afe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b00103b00 0x7f1b00198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.199+0000 7f1afe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b00103b00 0x7f1b00198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43926/0 (socket says 172.21.15.28:43926) 2024-04-21T08:52:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.199+0000 7f1afe59c640 1 -- 172.21.15.28:0/2994221057 learned_addr learned my addr 172.21.15.28:0/2994221057 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.199+0000 7f1aff59e640 1 --2- 172.21.15.28:0/2994221057 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b0010c520 0x7f1b0019cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.199+0000 7f1afed9d640 1 --2- 172.21.15.28:0/2994221057 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b001031e0 0x7f1b00198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.200+0000 7f1afe59c640 1 -- 172.21.15.28:0/2994221057 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b0010c520 msgr2=0x7f1b0019cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.200+0000 7f1afe59c640 1 --2- 172.21.15.28:0/2994221057 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b0010c520 0x7f1b0019cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.200+0000 7f1afe59c640 1 -- 172.21.15.28:0/2994221057 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b001031e0 msgr2=0x7f1b00198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.200+0000 7f1afe59c640 1 --2- 172.21.15.28:0/2994221057 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b001031e0 0x7f1b00198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.200+0000 7f1afe59c640 1 -- 172.21.15.28:0/2994221057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b0019d6f0 con 0x7f1b00103b00 2024-04-21T08:52:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.200+0000 7f1afe59c640 1 --2- 172.21.15.28:0/2994221057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b00103b00 0x7f1b00198c90 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1af400e9f0 tx=0x7f1af400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.200+0000 7f1ae7fff640 1 -- 172.21.15.28:0/2994221057 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af400cd80 con 0x7f1b00103b00 2024-04-21T08:52:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.200+0000 7f1ae7fff640 1 -- 172.21.15.28:0/2994221057 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1af40054d0 con 0x7f1b00103b00 2024-04-21T08:52:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.200+0000 7f1b0504e640 1 -- 172.21.15.28:0/2994221057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b0019d9e0 con 0x7f1b00103b00 2024-04-21T08:52:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.200+0000 7f1ae7fff640 1 -- 172.21.15.28:0/2994221057 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af4010690 con 0x7f1b00103b00 2024-04-21T08:52:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.200+0000 7f1b0504e640 1 -- 172.21.15.28:0/2994221057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b001a5270 con 0x7f1b00103b00 2024-04-21T08:52:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.202+0000 7f1ae7fff640 1 -- 172.21.15.28:0/2994221057 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f1af4004e90 con 0x7f1b00103b00 2024-04-21T08:52:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1b07c42740 1 Processor -- start 2024-04-21T08:52:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1b07c42740 1 -- start start 2024-04-21T08:52:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1b07c42740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 0x55ab853e4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1b07c42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853e4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1b07c42740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853f24f0 0x55ab853f48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1b07c42740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ab853498d0 con 0x55ab853e4b60 2024-04-21T08:52:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1b07c42740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55ab852e6e80 con 0x55ab853e58d0 2024-04-21T08:52:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1b07c42740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55ab85345890 con 0x55ab853f24f0 2024-04-21T08:52:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1afed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853e4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1ae7fff640 1 --2- 172.21.15.28:0/2994221057 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ae80a3f70 0x7f1ae80a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1afed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853e4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43932/0 (socket says 172.21.15.28:43932) 2024-04-21T08:52:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1afed9d640 1 -- 172.21.15.28:0/3539876826 learned_addr learned my addr 172.21.15.28:0/3539876826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1afe59c640 1 --2- 172.21.15.28:0/3539876826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853f24f0 0x55ab853f48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1aff59e640 1 --2- 172.21.15.28:0/2994221057 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ae80a3f70 0x7f1ae80a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1aff59e640 1 --2- 172.21.15.28:0/3539876826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 0x55ab853e4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1ae7fff640 1 -- 172.21.15.28:0/2994221057 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1af40d1270 con 0x7f1b00103b00 2024-04-21T08:52:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1afed9d640 1 -- 172.21.15.28:0/3539876826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853f24f0 msgr2=0x55ab853f48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.203+0000 7f1afed9d640 1 --2- 172.21.15.28:0/3539876826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853f24f0 0x55ab853f48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.204+0000 7f1afed9d640 1 -- 172.21.15.28:0/3539876826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 msgr2=0x55ab853e4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.204+0000 7f1afed9d640 1 --2- 172.21.15.28:0/3539876826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 0x55ab853e4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.204+0000 7f1afed9d640 1 -- 172.21.15.28:0/3539876826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab85308320 con 0x55ab853e4b60 2024-04-21T08:52:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.204+0000 7f1afed9d640 1 --2- 172.21.15.28:0/3539876826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853e4fe0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f1af002c840 tx=0x7f1af00098c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95c6a800736736d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.204+0000 7f1aff59e640 1 --2- 172.21.15.28:0/2994221057 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ae80a3f70 0x7f1ae80a6430 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1af8012890 tx=0x7f1af80046c0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.204+0000 7f1ae57fa640 1 -- 172.21.15.28:0/3539876826 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0043070 con 0x55ab853e4b60 2024-04-21T08:52:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.204+0000 7f1ae57fa640 1 -- 172.21.15.28:0/3539876826 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1af0003e20 con 0x55ab853e4b60 2024-04-21T08:52:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.204+0000 7f1ae57fa640 1 -- 172.21.15.28:0/3539876826 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0039040 con 0x55ab853e4b60 2024-04-21T08:52:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 -- 172.21.15.28:0/3539876826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 msgr2=0x55ab853e4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 --2- 172.21.15.28:0/3539876826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853e4fe0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f1af002c840 tx=0x7f1af00098c0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 -- 172.21.15.28:0/3539876826 shutdown_connections 2024-04-21T08:52:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 --2- 172.21.15.28:0/3539876826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853f24f0 0x55ab853f48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 --2- 172.21.15.28:0/3539876826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853e4fe0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 --2- 172.21.15.28:0/3539876826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 0x55ab853e4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 -- 172.21.15.28:0/3539876826 >> 172.21.15.28:0/3539876826 conn(0x55ab853abc90 msgr2=0x55ab853df850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 -- 172.21.15.28:0/3539876826 shutdown_connections 2024-04-21T08:52:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 -- 172.21.15.28:0/3539876826 wait complete. 2024-04-21T08:52:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 -- start start 2024-04-21T08:52:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 Processor -- start 2024-04-21T08:52:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e4b60 0x55ab853f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853e58d0 0x55ab853ef230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853f24f0 0x55ab853ef7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ab853f51c0 con 0x55ab853f24f0 2024-04-21T08:52:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55ab852d1eb0 con 0x55ab853e4b60 2024-04-21T08:52:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.205+0000 7f1b07c42740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55ab853f5340 con 0x55ab853e58d0 2024-04-21T08:52:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1b07c42740 10 client.0 _ll_register_callbacks cb 0x55ab852feb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:13.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1aff59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853f24f0 0x55ab853ef7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1aff59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853f24f0 0x55ab853ef7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43944/0 (socket says 172.21.15.28:43944) 2024-04-21T08:52:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1aff59e640 1 -- 172.21.15.28:0/657949788 learned_addr learned my addr 172.21.15.28:0/657949788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1afed9d640 1 --2- 172.21.15.28:0/657949788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e4b60 0x55ab853f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1afe59c640 1 --2- 172.21.15.28:0/657949788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853e58d0 0x55ab853ef230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1aff59e640 1 -- 172.21.15.28:0/657949788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853e58d0 msgr2=0x55ab853ef230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1aff59e640 1 --2- 172.21.15.28:0/657949788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853e58d0 0x55ab853ef230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1aff59e640 1 -- 172.21.15.28:0/657949788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e4b60 msgr2=0x55ab853f1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1aff59e640 1 --2- 172.21.15.28:0/657949788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e4b60 0x55ab853f1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1aff59e640 1 -- 172.21.15.28:0/657949788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab8505c210 con 0x55ab853f24f0 2024-04-21T08:52:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1afed9d640 1 --2- 172.21.15.28:0/657949788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e4b60 0x55ab853f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:52:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1afe59c640 1 --2- 172.21.15.28:0/657949788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853e58d0 0x55ab853ef230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:52:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1aff59e640 1 --2- 172.21.15.28:0/657949788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853f24f0 0x55ab853ef7d0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f1af8012440 tx=0x7f1af800b2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=187279b3a55ea4c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:13.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1ae2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1ae2ff5640 1 -- 172.21.15.28:0/657949788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af802d060 con 0x55ab853f24f0 2024-04-21T08:52:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1ae2ff5640 1 -- 172.21.15.28:0/657949788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1af801b070 con 0x55ab853f24f0 2024-04-21T08:52:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1b07c42740 1 -- 172.21.15.28:0/657949788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1af0003e20 con 0x55ab853f24f0 2024-04-21T08:52:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1ae2ff5640 1 -- 172.21.15.28:0/657949788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1af8012e20 con 0x55ab853f24f0 2024-04-21T08:52:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1b07c42740 10 client.16188 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.206+0000 7f1b07c42740 1 -- 172.21.15.28:0/657949788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ab85128620 con 0x55ab853f24f0 2024-04-21T08:52:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.207+0000 7f1ae2ff5640 1 -- 172.21.15.28:0/657949788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1af0003e20 con 0x55ab853f24f0 2024-04-21T08:52:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.207+0000 7f1ae2ff5640 1 -- 172.21.15.28:0/657949788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f1af8027070 con 0x55ab853f24f0 2024-04-21T08:52:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.207+0000 7f1b07c42740 10 client.16188 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.207+0000 7f1b07c42740 10 client.16188 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.207+0000 7f1b07c42740 20 client.16188 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.207+0000 7f1b07c42740 1 --2- 172.21.15.28:0/657949788 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab85402550 0x55ab85404960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.207+0000 7f1b07c42740 4 client.16188 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.207+0000 7f1b07c42740 1 -- 172.21.15.28:0/657949788 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab8535d880 con 0x55ab85402550 2024-04-21T08:52:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.207+0000 7f1afed9d640 1 --2- 172.21.15.28:0/657949788 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab85402550 0x55ab85404960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.208+0000 7f1afed9d640 1 --2- 172.21.15.28:0/657949788 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab85402550 0x55ab85404960 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a4f05ab4abf252f server_cookie=a37a3ddecf0172b1 in_seq=0 out_seq=0 2024-04-21T08:52:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.208+0000 7f1ae2ff5640 10 client.16188 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.208+0000 7f1ae2ff5640 1 -- 172.21.15.28:0/657949788 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ab8535d880 con 0x55ab85402550 2024-04-21T08:52:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.208+0000 7f1ae2ff5640 10 client.16188 handle_command_reply: tid=0 2024-04-21T08:52:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.208+0000 7f1b07c42740 1 client.16188 shutdown 2024-04-21T08:52:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.208+0000 7f1b07c42740 1 -- 172.21.15.28:0/657949788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853f24f0 msgr2=0x55ab853ef7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.208+0000 7f1b07c42740 1 --2- 172.21.15.28:0/657949788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853f24f0 0x55ab853ef7d0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f1af8012440 tx=0x7f1af800b2e0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.208+0000 7f1b07c42740 1 -- 172.21.15.28:0/657949788 shutdown_connections 2024-04-21T08:52:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.208+0000 7f1b07c42740 1 --2- 172.21.15.28:0/657949788 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab85402550 0x55ab85404960 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.208+0000 7f1b07c42740 1 --2- 172.21.15.28:0/657949788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853f24f0 0x55ab853ef7d0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.208+0000 7f1ae2ff5640 0 client.16188 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:13.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.208+0000 7f1b07c42740 1 --2- 172.21.15.28:0/657949788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853e58d0 0x55ab853ef230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.208+0000 7f1b07c42740 1 --2- 172.21.15.28:0/657949788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e4b60 0x55ab853f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.208+0000 7f1b07c42740 1 -- 172.21.15.28:0/657949788 >> 172.21.15.28:0/657949788 conn(0x55ab853abc90 msgr2=0x55ab853e15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:13.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.209+0000 7f1b07c42740 1 -- 172.21.15.28:0/657949788 shutdown_connections 2024-04-21T08:52:13.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.209+0000 7f1b07c42740 1 -- 172.21.15.28:0/657949788 wait complete. 2024-04-21T08:52:13.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.209+0000 7f1b07c42740 20 client.16188 trim_cache size 0 max 16384 2024-04-21T08:52:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.223+0000 7f1b07c42740 1 Processor -- start 2024-04-21T08:52:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.223+0000 7f1b07c42740 1 -- start start 2024-04-21T08:52:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1b07c42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853fe0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1b07c42740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 0x55ab853f1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1b07c42740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853f24f0 0x55ab853f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1b07c42740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001af9d0 con 0x55ab853e4b60 2024-04-21T08:52:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1b07c42740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001ad2a0 con 0x55ab853e58d0 2024-04-21T08:52:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1b07c42740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001ab0f0 con 0x55ab853f24f0 2024-04-21T08:52:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1afed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853fe0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1afed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853fe0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43956/0 (socket says 172.21.15.28:43956) 2024-04-21T08:52:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1afed9d640 1 -- 172.21.15.28:0/511606344 learned_addr learned my addr 172.21.15.28:0/511606344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1aff59e640 1 --2- 172.21.15.28:0/511606344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853f24f0 0x55ab853f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1afe59c640 1 --2- 172.21.15.28:0/511606344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 0x55ab853f1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1afed9d640 1 -- 172.21.15.28:0/511606344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853f24f0 msgr2=0x55ab853f1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1afed9d640 1 --2- 172.21.15.28:0/511606344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853f24f0 0x55ab853f1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1afed9d640 1 -- 172.21.15.28:0/511606344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 msgr2=0x55ab853f1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1afed9d640 1 --2- 172.21.15.28:0/511606344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 0x55ab853f1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1afed9d640 1 -- 172.21.15.28:0/511606344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab853fec30 con 0x55ab853e4b60 2024-04-21T08:52:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1afe59c640 1 --2- 172.21.15.28:0/511606344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 0x55ab853f1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:52:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1afed9d640 1 --2- 172.21.15.28:0/511606344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853fe0c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1af0005cc0 tx=0x7f1af002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a560bc5ace5b787c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1ae4ff9640 1 -- 172.21.15.28:0/511606344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0039040 con 0x55ab853e4b60 2024-04-21T08:52:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1ae4ff9640 1 -- 172.21.15.28:0/511606344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1af004fd50 con 0x55ab853e4b60 2024-04-21T08:52:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.224+0000 7f1ae4ff9640 1 -- 172.21.15.28:0/511606344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0043040 con 0x55ab853e4b60 2024-04-21T08:52:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 -- 172.21.15.28:0/511606344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 msgr2=0x55ab853fe0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 --2- 172.21.15.28:0/511606344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853fe0c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1af0005cc0 tx=0x7f1af002e040 comp rx=0 tx=0).stop 2024-04-21T08:52:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 -- 172.21.15.28:0/511606344 shutdown_connections 2024-04-21T08:52:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 --2- 172.21.15.28:0/511606344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853f24f0 0x55ab853f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 --2- 172.21.15.28:0/511606344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 0x55ab853f1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 --2- 172.21.15.28:0/511606344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853fe0c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 -- 172.21.15.28:0/511606344 >> 172.21.15.28:0/511606344 conn(0x55ab853abc90 msgr2=0x55ab853df380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 -- 172.21.15.28:0/511606344 shutdown_connections 2024-04-21T08:52:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 -- 172.21.15.28:0/511606344 wait complete. 2024-04-21T08:52:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 -- start start 2024-04-21T08:52:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 Processor -- start 2024-04-21T08:52:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853faa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 0x55ab853ec700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853f24f0 0x55ab853ecd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001ab760 con 0x55ab853e4b60 2024-04-21T08:52:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001acbd0 con 0x55ab853e58d0 2024-04-21T08:52:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.225+0000 7f1b07c42740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1b001abe30 con 0x55ab853f24f0 2024-04-21T08:52:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1b07c42740 10 client.0 _ll_register_callbacks cb 0x7f1af800bce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1afed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853faa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1afed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853faa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43972/0 (socket says 172.21.15.28:43972) 2024-04-21T08:52:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1afed9d640 1 -- 172.21.15.28:0/3901265889 learned_addr learned my addr 172.21.15.28:0/3901265889 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1afe59c640 1 --2- 172.21.15.28:0/3901265889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 0x55ab853ec700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1aff59e640 1 --2- 172.21.15.28:0/3901265889 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853f24f0 0x55ab853ecd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1afed9d640 1 -- 172.21.15.28:0/3901265889 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853f24f0 msgr2=0x55ab853ecd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1afed9d640 1 --2- 172.21.15.28:0/3901265889 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853f24f0 0x55ab853ecd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1afed9d640 1 -- 172.21.15.28:0/3901265889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 msgr2=0x55ab853ec700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1afed9d640 1 --2- 172.21.15.28:0/3901265889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 0x55ab853ec700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1afed9d640 1 -- 172.21.15.28:0/3901265889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab853f31d0 con 0x55ab853e4b60 2024-04-21T08:52:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1afe59c640 1 --2- 172.21.15.28:0/3901265889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 0x55ab853ec700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:52:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1afed9d640 1 --2- 172.21.15.28:0/3901265889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853faa60 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1af002e010 tx=0x7f1af0000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b868a7375bd486e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1ae2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1ae2ff5640 1 -- 172.21.15.28:0/3901265889 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0055020 con 0x55ab853e4b60 2024-04-21T08:52:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1ae2ff5640 1 -- 172.21.15.28:0/3901265889 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1af004f320 con 0x55ab853e4b60 2024-04-21T08:52:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1b07c42740 1 -- 172.21.15.28:0/3901265889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1af004fd50 con 0x55ab853e4b60 2024-04-21T08:52:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1ae2ff5640 1 -- 172.21.15.28:0/3901265889 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1af0039040 con 0x55ab853e4b60 2024-04-21T08:52:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1b07c42740 10 client.16194 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1b07c42740 1 -- 172.21.15.28:0/3901265889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ab85468d90 con 0x55ab853e4b60 2024-04-21T08:52:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.226+0000 7f1ae2ff5640 1 -- 172.21.15.28:0/3901265889 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1af004fd50 con 0x55ab853e4b60 2024-04-21T08:52:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.227+0000 7f1ae2ff5640 1 -- 172.21.15.28:0/3901265889 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f1af0048450 con 0x55ab853e4b60 2024-04-21T08:52:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.227+0000 7f1b07c42740 10 client.16194 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.227+0000 7f1b07c42740 10 client.16194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.227+0000 7f1b07c42740 20 client.16194 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.227+0000 7f1b07c42740 1 --2- 172.21.15.28:0/3901265889 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab85469120 0x55ab8546b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.227+0000 7f1b07c42740 4 client.16194 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:52:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.227+0000 7f1b07c42740 1 -- 172.21.15.28:0/3901265889 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ab852e93d0 con 0x55ab85469120 2024-04-21T08:52:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.227+0000 7f1afe59c640 1 --2- 172.21.15.28:0/3901265889 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab85469120 0x55ab8546b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.227+0000 7f1ae2ff5640 10 client.16194 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.227+0000 7f1afe59c640 1 --2- 172.21.15.28:0/3901265889 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab85469120 0x55ab8546b510 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3254225599e46423 server_cookie=5ca266ec619e2b7b in_seq=0 out_seq=0 2024-04-21T08:52:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.349+0000 7f1ae2ff5640 1 -- 172.21.15.28:0/3901265889 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ab852e93d0 con 0x55ab85469120 2024-04-21T08:52:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.349+0000 7f1ae2ff5640 10 client.16194 handle_command_reply: tid=0 2024-04-21T08:52:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.349+0000 7f1b07c42740 1 client.16194 shutdown 2024-04-21T08:52:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.349+0000 7f1b07c42740 1 -- 172.21.15.28:0/3901265889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 msgr2=0x55ab853faa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.349+0000 7f1b07c42740 1 --2- 172.21.15.28:0/3901265889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853faa60 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1af002e010 tx=0x7f1af0000f40 comp rx=0 tx=0).stop 2024-04-21T08:52:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.349+0000 7f1b07c42740 1 -- 172.21.15.28:0/3901265889 shutdown_connections 2024-04-21T08:52:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.349+0000 7f1b07c42740 1 --2- 172.21.15.28:0/3901265889 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab85469120 0x55ab8546b510 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.349+0000 7f1b07c42740 1 --2- 172.21.15.28:0/3901265889 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab853f24f0 0x55ab853ecd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.349+0000 7f1ae2ff5640 0 client.16194 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.349+0000 7f1b07c42740 1 --2- 172.21.15.28:0/3901265889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab853e58d0 0x55ab853ec700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.349+0000 7f1b07c42740 1 --2- 172.21.15.28:0/3901265889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab853e4b60 0x55ab853faa60 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.349+0000 7f1b07c42740 1 -- 172.21.15.28:0/3901265889 >> 172.21.15.28:0/3901265889 conn(0x55ab853abc90 msgr2=0x55ab853ed6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.349+0000 7f1b07c42740 1 -- 172.21.15.28:0/3901265889 shutdown_connections 2024-04-21T08:52:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.349+0000 7f1b07c42740 1 -- 172.21.15.28:0/3901265889 wait complete. 2024-04-21T08:52:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.349+0000 7f1b07c42740 20 client.16194 trim_cache size 0 max 16384 2024-04-21T08:52:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:52:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:52:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:52:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:52:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.350+0000 7f1b0504e640 1 -- 172.21.15.28:0/2994221057 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ae80a3f70 msgr2=0x7f1ae80a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.350+0000 7f1b0504e640 1 --2- 172.21.15.28:0/2994221057 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ae80a3f70 0x7f1ae80a6430 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1af8012890 tx=0x7f1af80046c0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.350+0000 7f1b0504e640 1 -- 172.21.15.28:0/2994221057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b00103b00 msgr2=0x7f1b00198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.350+0000 7f1b0504e640 1 --2- 172.21.15.28:0/2994221057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b00103b00 0x7f1b00198c90 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1af400e9f0 tx=0x7f1af400edf0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.350+0000 7f1b0504e640 1 -- 172.21.15.28:0/2994221057 shutdown_connections 2024-04-21T08:52:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.350+0000 7f1b0504e640 1 --2- 172.21.15.28:0/2994221057 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1ae80a3f70 0x7f1ae80a6430 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.350+0000 7f1b0504e640 1 --2- 172.21.15.28:0/2994221057 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b0010c520 0x7f1b0019cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.350+0000 7f1b0504e640 1 --2- 172.21.15.28:0/2994221057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b00103b00 0x7f1b00198c90 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.350+0000 7f1b0504e640 1 --2- 172.21.15.28:0/2994221057 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b001031e0 0x7f1b00198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.350+0000 7f1b0504e640 1 -- 172.21.15.28:0/2994221057 >> 172.21.15.28:0/2994221057 conn(0x7f1b00077fd0 msgr2=0x7f1b000fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.350+0000 7f1b0504e640 1 -- 172.21.15.28:0/2994221057 shutdown_connections 2024-04-21T08:52:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:13.350+0000 7f1b0504e640 1 -- 172.21.15.28:0/2994221057 wait complete. 2024-04-21T08:52:13.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:12 smithi152 ceph-mon[31401]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:52:13.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:12 smithi028 ceph-mon[28003]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:52:13.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:12 smithi171 ceph-mon[31358]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:52:13.791 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether build environment is sane... yes 2024-04-21T08:52:13.796 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-04-21T08:52:13.796 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gawk... gawk 2024-04-21T08:52:13.810 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether make sets $(MAKE)... yes 2024-04-21T08:52:13.841 INFO:tasks.workunit.client.0.smithi028.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-21T08:52:13.842 INFO:tasks.workunit.client.0.smithi028.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-21T08:52:13.853 INFO:tasks.workunit.client.0.smithi028.stdout:checking for style of include used by make... GNU 2024-04-21T08:52:13.858 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... gcc 2024-04-21T08:52:13.946 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether the C compiler works... yes 2024-04-21T08:52:13.947 INFO:tasks.workunit.client.0.smithi028.stdout:checking for C compiler default output file name... a.out 2024-04-21T08:52:13.985 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of executables... 2024-04-21T08:52:14.023 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are cross compiling... no 2024-04-21T08:52:14.061 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of object files... o 2024-04-21T08:52:14.086 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... yes 2024-04-21T08:52:14.110 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... yes 2024-04-21T08:52:14.147 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-21T08:52:14.253 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... gcc3 2024-04-21T08:52:14.301 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to run the C preprocessor... gcc -E 2024-04-21T08:52:14.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:13 smithi152 ceph-mon[31401]: pgmap v34: 129 pgs: 129 active+clean; 4.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 277 KiB/s wr, 36 op/s 2024-04-21T08:52:14.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:13 smithi028 ceph-mon[28003]: pgmap v34: 129 pgs: 129 active+clean; 4.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 277 KiB/s wr, 36 op/s 2024-04-21T08:52:14.358 INFO:tasks.workunit.client.0.smithi028.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-21T08:52:14.359 INFO:tasks.workunit.client.0.smithi028.stdout:checking for egrep... /usr/bin/grep -E 2024-04-21T08:52:14.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:13 smithi171 ceph-mon[31358]: pgmap v34: 129 pgs: 129 active+clean; 4.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 277 KiB/s wr, 36 op/s 2024-04-21T08:52:14.499 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ANSI C header files... yes 2024-04-21T08:52:14.536 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/types.h... yes 2024-04-21T08:52:14.578 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/stat.h... yes 2024-04-21T08:52:14.620 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdlib.h... yes 2024-04-21T08:52:14.663 INFO:tasks.workunit.client.0.smithi028.stdout:checking for string.h... yes 2024-04-21T08:52:14.708 INFO:tasks.workunit.client.0.smithi028.stdout:checking for memory.h... yes 2024-04-21T08:52:14.750 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strings.h... yes 2024-04-21T08:52:14.795 INFO:tasks.workunit.client.0.smithi028.stdout:checking for inttypes.h... yes 2024-04-21T08:52:14.840 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... yes 2024-04-21T08:52:14.885 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... yes 2024-04-21T08:52:14.929 INFO:tasks.workunit.client.0.smithi028.stdout:checking minix/config.h usability... no 2024-04-21T08:52:14.959 INFO:tasks.workunit.client.0.smithi028.stdout:checking minix/config.h presence... no 2024-04-21T08:52:14.959 INFO:tasks.workunit.client.0.smithi028.stdout:checking for minix/config.h... no 2024-04-21T08:52:14.995 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-04-21T08:52:14.995 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... (cached) gcc 2024-04-21T08:52:15.030 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-21T08:52:15.030 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-21T08:52:15.030 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-21T08:52:15.031 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-21T08:52:15.073 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lm... yes 2024-04-21T08:52:15.117 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lpthread... yes 2024-04-21T08:52:15.117 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ANSI C header files... (cached) yes 2024-04-21T08:52:15.155 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-21T08:52:15.236 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h usability... yes 2024-04-21T08:52:15.255 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h presence... yes 2024-04-21T08:52:15.255 INFO:tasks.workunit.client.0.smithi028.stdout:checking for pthread.h... yes 2024-04-21T08:52:15.303 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h usability... yes 2024-04-21T08:52:15.320 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h presence... yes 2024-04-21T08:52:15.320 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fcntl.h... yes 2024-04-21T08:52:15.369 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h usability... yes 2024-04-21T08:52:15.386 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h presence... yes 2024-04-21T08:52:15.386 INFO:tasks.workunit.client.0.smithi028.stdout:checking for limits.h... yes 2024-04-21T08:52:15.401 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... (cached) yes 2024-04-21T08:52:15.438 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h usability... yes 2024-04-21T08:52:15.456 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h presence... yes 2024-04-21T08:52:15.456 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/time.h... yes 2024-04-21T08:52:15.470 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... (cached) yes 2024-04-21T08:52:15.508 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h usability... yes 2024-04-21T08:52:15.525 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h presence... yes 2024-04-21T08:52:15.525 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/vfs.h... yes 2024-04-21T08:52:15.576 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h usability... no 2024-04-21T08:52:15.605 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h presence... no 2024-04-21T08:52:15.606 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/limits.h... no 2024-04-21T08:52:15.632 INFO:tasks.workunit.client.0.smithi028.stdout:checking for an ANSI C-conforming const... yes 2024-04-21T08:52:15.661 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-21T08:52:15.725 INFO:tasks.workunit.client.0.smithi028.stdout:checking for system... yes 2024-04-21T08:52:15.791 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gettimeofday... yes 2024-04-21T08:52:15.856 INFO:tasks.workunit.client.0.smithi028.stdout:checking for mkdir... yes 2024-04-21T08:52:15.917 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strerror... yes 2024-04-21T08:52:15.977 INFO:tasks.workunit.client.0.smithi028.stdout:checking for open64... yes 2024-04-21T08:52:16.038 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stat64... yes 2024-04-21T08:52:16.097 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fseeko64... yes 2024-04-21T08:52:16.157 INFO:tasks.workunit.client.0.smithi028.stdout:checking for lrand48_r... yes 2024-04-21T08:52:16.218 INFO:tasks.workunit.client.0.smithi028.stdout:checking for srand48_r... yes 2024-04-21T08:52:16.243 INFO:tasks.workunit.client.0.smithi028.stdout:configure: creating ./config.status 2024-04-21T08:52:16.314 INFO:tasks.workunit.client.0.smithi028.stdout: /bin/sh ./config.status 2024-04-21T08:52:16.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:15 smithi152 ceph-mon[31401]: pgmap v35: 129 pgs: 129 active+clean; 4.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 252 KiB/s wr, 16 op/s 2024-04-21T08:52:16.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:15 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:52:16.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:15 smithi028 ceph-mon[28003]: pgmap v35: 129 pgs: 129 active+clean; 4.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 252 KiB/s wr, 16 op/s 2024-04-21T08:52:16.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:15 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:52:16.364 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating Makefile 2024-04-21T08:52:16.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:15 smithi171 ceph-mon[31358]: pgmap v35: 129 pgs: 129 active+clean; 4.2 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 252 KiB/s wr, 16 op/s 2024-04-21T08:52:16.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:15 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:52:16.375 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating config.h 2024-04-21T08:52:16.398 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing depfiles commands 2024-04-21T08:52:16.408 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing default commands 2024-04-21T08:52:16.437 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-04-21T08:52:16.592 INFO:tasks.workunit.client.0.smithi028.stdout:rm -f stamp-h1 2024-04-21T08:52:16.596 INFO:tasks.workunit.client.0.smithi028.stdout:touch config.h.in 2024-04-21T08:52:16.597 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh ./config.status config.h 2024-04-21T08:52:16.646 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating config.h 2024-04-21T08:52:16.678 INFO:tasks.workunit.client.0.smithi028.stdout:make all-am 2024-04-21T08:52:16.684 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-21T08:52:16.685 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-04-21T08:52:16.866 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-04-21T08:52:16.869 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-04-21T08:52:16.976 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-04-21T08:52:16.979 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-04-21T08:52:17.066 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-04-21T08:52:17.069 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-04-21T08:52:17.159 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-04-21T08:52:17.162 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-04-21T08:52:17.277 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-04-21T08:52:17.280 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-04-21T08:52:17.331 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-04-21T08:52:17.333 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-04-21T08:52:17.375 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-04-21T08:52:17.378 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-04-21T08:52:17.426 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-04-21T08:52:17.429 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-04-21T08:52:17.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:17 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:17 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:17 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:17 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:17 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:17 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:17 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:52:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:17 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:52:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:17 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:17 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:17 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:17 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:17 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:17 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:17 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:17 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:52:17.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:17 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:52:17.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:17 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:17 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:17 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:17 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:17 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:17 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:17 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:17 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:52:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:17 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:52:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:17 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:52:17.640 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-04-21T08:52:17.643 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-04-21T08:52:17.769 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-04-21T08:52:17.774 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-04-21T08:52:17.917 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-04-21T08:52:17.920 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-04-21T08:52:17.989 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-04-21T08:52:17.992 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-04-21T08:52:18.063 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-04-21T08:52:18.066 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-04-21T08:52:18.170 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-04-21T08:52:18.173 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-04-21T08:52:18.202 INFO:tasks.workunit.client.0.smithi028.stderr:In file included from parser.h:22, 2024-04-21T08:52:18.202 INFO:tasks.workunit.client.0.smithi028.stderr: from parser.c:25: 2024-04-21T08:52:18.202 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘init_filesys’: 2024-04-21T08:52:18.202 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-21T08:52:18.202 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-21T08:52:18.202 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-21T08:52:18.202 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-21T08:52:18.202 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-21T08:52:18.202 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-21T08:52:18.202 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-21T08:52:18.202 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-21T08:52:18.202 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-21T08:52:18.203 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-21T08:52:18.203 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-21T08:52:18.203 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-21T08:52:18.203 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-21T08:52:18.203 INFO:tasks.workunit.client.0.smithi028.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-21T08:52:18.203 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-21T08:52:18.203 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-21T08:52:18.203 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-21T08:52:18.203 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-21T08:52:18.203 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-21T08:52:18.203 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-21T08:52:18.203 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-21T08:52:18.203 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-21T08:52:18.204 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-21T08:52:18.204 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-21T08:52:18.204 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-21T08:52:18.204 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-21T08:52:18.204 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-21T08:52:18.204 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-21T08:52:18.204 INFO:tasks.workunit.client.0.smithi028.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-21T08:52:18.204 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-21T08:52:18.204 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-21T08:52:18.204 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-21T08:52:18.204 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-21T08:52:18.204 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-21T08:52:18.204 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-21T08:52:18.204 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-21T08:52:18.205 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-21T08:52:18.205 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-21T08:52:18.205 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-21T08:52:18.205 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-21T08:52:18.205 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-21T08:52:18.205 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-21T08:52:18.205 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-21T08:52:18.205 INFO:tasks.workunit.client.0.smithi028.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-21T08:52:18.205 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-21T08:52:18.205 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-21T08:52:18.205 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-21T08:52:18.205 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-21T08:52:18.205 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-21T08:52:18.205 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-21T08:52:18.206 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-21T08:52:18.206 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-21T08:52:18.206 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-21T08:52:18.206 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-21T08:52:18.206 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-21T08:52:18.206 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-21T08:52:18.206 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-21T08:52:18.206 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-21T08:52:18.206 INFO:tasks.workunit.client.0.smithi028.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-21T08:52:18.206 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-21T08:52:18.206 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘init_tg_stats’: 2024-04-21T08:52:18.206 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-21T08:52:18.206 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-21T08:52:18.206 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-21T08:52:18.207 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-21T08:52:18.208 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-21T08:52:18.209 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-21T08:52:18.210 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-21T08:52:18.210 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-21T08:52:18.210 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-21T08:52:18.210 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-21T08:52:18.210 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-21T08:52:18.210 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-21T08:52:18.210 INFO:tasks.workunit.client.0.smithi028.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-21T08:52:18.210 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-21T08:52:18.250 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘get_optstr’: 2024-04-21T08:52:18.250 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-04-21T08:52:18.250 INFO:tasks.workunit.client.0.smithi028.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-04-21T08:52:18.250 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-21T08:52:18.250 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:200:14: note: source object declared here 2024-04-21T08:52:18.250 INFO:tasks.workunit.client.0.smithi028.stderr: 200 | char temp[BUFSIZE]; 2024-04-21T08:52:18.250 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~ 2024-04-21T08:52:18.256 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘size64_convert’: 2024-04-21T08:52:18.256 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-04-21T08:52:18.256 INFO:tasks.workunit.client.0.smithi028.stderr: 119 | memcpy(unit, "\0", 3); 2024-04-21T08:52:18.257 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-04-21T08:52:18.551 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-04-21T08:52:18.555 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-04-21T08:52:18.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:18 smithi152 ceph-mon[31401]: pgmap v36: 129 pgs: 129 active+clean; 7.7 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 630 KiB/s wr, 145 op/s 2024-04-21T08:52:18.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:18 smithi028 ceph-mon[28003]: pgmap v36: 129 pgs: 129 active+clean; 7.7 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 630 KiB/s wr, 145 op/s 2024-04-21T08:52:18.611 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-04-21T08:52:18.615 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-04-21T08:52:18.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:18 smithi171 ceph-mon[31358]: pgmap v36: 129 pgs: 129 active+clean; 7.7 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 630 KiB/s wr, 145 op/s 2024-04-21T08:52:18.705 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-04-21T08:52:18.709 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-04-21T08:52:18.739 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-04-21T08:52:18.743 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-04-21T08:52:18.777 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-21T08:52:18.778 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd .. 2024-04-21T08:52:18.778 INFO:tasks.workunit.client.0.smithi028.stderr:+ mkdir tmp 2024-04-21T08:52:18.780 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd tmp 2024-04-21T08:52:18.780 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in $mydir/*.ffsb 2024-04-21T08:52:18.780 INFO:tasks.workunit.client.0.smithi028.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-04-21T08:52:18.780 INFO:tasks.workunit.client.0.smithi028.stdout:FFSB version 6.0-RC2 started 2024-04-21T08:52:18.780 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-21T08:52:18.783 INFO:tasks.workunit.client.0.smithi028.stdout:benchmark time = 300 2024-04-21T08:52:18.783 INFO:tasks.workunit.client.0.smithi028.stdout:ThreadGroup 0 2024-04-21T08:52:18.783 INFO:tasks.workunit.client.0.smithi028.stdout:================ 2024-04-21T08:52:18.784 INFO:tasks.workunit.client.0.smithi028.stdout: num_threads = 32 2024-04-21T08:52:18.784 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-21T08:52:18.784 INFO:tasks.workunit.client.0.smithi028.stdout: read_random = off 2024-04-21T08:52:18.784 INFO:tasks.workunit.client.0.smithi028.stdout: read_size = 0 (0B) 2024-04-21T08:52:18.784 INFO:tasks.workunit.client.0.smithi028.stdout: read_blocksize = 0 (0B) 2024-04-21T08:52:18.784 INFO:tasks.workunit.client.0.smithi028.stdout: read_skip = off 2024-04-21T08:52:18.784 INFO:tasks.workunit.client.0.smithi028.stdout: read_skipsize = 0 (0B) 2024-04-21T08:52:18.784 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-21T08:52:18.784 INFO:tasks.workunit.client.0.smithi028.stdout: write_random = on 2024-04-21T08:52:18.784 INFO:tasks.workunit.client.0.smithi028.stdout: write_size = 5242880 (5MB) 2024-04-21T08:52:18.784 INFO:tasks.workunit.client.0.smithi028.stdout: fsync_file = 0 2024-04-21T08:52:18.784 INFO:tasks.workunit.client.0.smithi028.stdout: write_blocksize = 4096 (4KB) 2024-04-21T08:52:18.784 INFO:tasks.workunit.client.0.smithi028.stdout: wait time = 0 2024-04-21T08:52:18.784 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-21T08:52:18.784 INFO:tasks.workunit.client.0.smithi028.stdout: op weights 2024-04-21T08:52:18.785 INFO:tasks.workunit.client.0.smithi028.stdout: read = 0 (0.00%) 2024-04-21T08:52:18.785 INFO:tasks.workunit.client.0.smithi028.stdout: readall = 0 (0.00%) 2024-04-21T08:52:18.785 INFO:tasks.workunit.client.0.smithi028.stdout: write = 1 (100.00%) 2024-04-21T08:52:18.785 INFO:tasks.workunit.client.0.smithi028.stdout: create = 0 (0.00%) 2024-04-21T08:52:18.785 INFO:tasks.workunit.client.0.smithi028.stdout: append = 0 (0.00%) 2024-04-21T08:52:18.785 INFO:tasks.workunit.client.0.smithi028.stdout: delete = 0 (0.00%) 2024-04-21T08:52:18.785 INFO:tasks.workunit.client.0.smithi028.stdout: metaop = 0 (0.00%) 2024-04-21T08:52:18.785 INFO:tasks.workunit.client.0.smithi028.stdout: createdir = 0 (0.00%) 2024-04-21T08:52:18.785 INFO:tasks.workunit.client.0.smithi028.stdout: stat = 0 (0.00%) 2024-04-21T08:52:18.785 INFO:tasks.workunit.client.0.smithi028.stdout: writeall = 0 (0.00%) 2024-04-21T08:52:18.785 INFO:tasks.workunit.client.0.smithi028.stdout: writeall_fsync = 0 (0.00%) 2024-04-21T08:52:18.785 INFO:tasks.workunit.client.0.smithi028.stdout: open_close = 0 (0.00%) 2024-04-21T08:52:18.785 INFO:tasks.workunit.client.0.smithi028.stdout: write_fsync = 0 (0.00%) 2024-04-21T08:52:18.785 INFO:tasks.workunit.client.0.smithi028.stdout: create_fsync = 0 (0.00%) 2024-04-21T08:52:18.785 INFO:tasks.workunit.client.0.smithi028.stdout: append_fsync = 0 (0.00%) 2024-04-21T08:52:18.786 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-21T08:52:18.786 INFO:tasks.workunit.client.0.smithi028.stdout:FileSystem . 2024-04-21T08:52:18.786 INFO:tasks.workunit.client.0.smithi028.stdout:========== 2024-04-21T08:52:18.786 INFO:tasks.workunit.client.0.smithi028.stdout: num_dirs = 0 2024-04-21T08:52:18.786 INFO:tasks.workunit.client.0.smithi028.stdout: starting files = 128 2024-04-21T08:52:18.786 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-21T08:52:18.786 INFO:tasks.workunit.client.0.smithi028.stdout: min file size = 104857600 (100MB) 2024-04-21T08:52:18.786 INFO:tasks.workunit.client.0.smithi028.stdout: max file size = 104857600 (100MB) 2024-04-21T08:52:18.786 INFO:tasks.workunit.client.0.smithi028.stdout: directio = off 2024-04-21T08:52:18.786 INFO:tasks.workunit.client.0.smithi028.stdout: alignedio = on 2024-04-21T08:52:18.786 INFO:tasks.workunit.client.0.smithi028.stdout: bufferedio = off 2024-04-21T08:52:18.786 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-21T08:52:18.786 INFO:tasks.workunit.client.0.smithi028.stdout: aging is off 2024-04-21T08:52:18.786 INFO:tasks.workunit.client.0.smithi028.stdout: current utilization = 0.04% 2024-04-21T08:52:18.786 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-21T08:52:18.787 INFO:tasks.workunit.client.0.smithi028.stdout:checking existing fs: . 2024-04-21T08:52:18.787 INFO:tasks.workunit.client.0.smithi028.stdout:recreating new fileset 2024-04-21T08:52:18.787 INFO:tasks.workunit.client.0.smithi028.stderr:opendir: No such file or directory 2024-04-21T08:52:20.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:20 smithi152 ceph-mon[31401]: pgmap v37: 129 pgs: 129 active+clean; 7.7 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 630 KiB/s wr, 142 op/s 2024-04-21T08:52:20.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:20 smithi028 ceph-mon[28003]: pgmap v37: 129 pgs: 129 active+clean; 7.7 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 630 KiB/s wr, 142 op/s 2024-04-21T08:52:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:20 smithi171 ceph-mon[31358]: pgmap v37: 129 pgs: 129 active+clean; 7.7 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 630 KiB/s wr, 142 op/s 2024-04-21T08:52:22.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:22 smithi152 ceph-mon[31401]: pgmap v38: 129 pgs: 129 active+clean; 7.7 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 630 KiB/s wr, 142 op/s 2024-04-21T08:52:22.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:22 smithi028 ceph-mon[28003]: pgmap v38: 129 pgs: 129 active+clean; 7.7 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 630 KiB/s wr, 142 op/s 2024-04-21T08:52:22.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:22 smithi171 ceph-mon[31358]: pgmap v38: 129 pgs: 129 active+clean; 7.7 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 630 KiB/s wr, 142 op/s 2024-04-21T08:52:23.997 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:52:23.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:52:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.177+0000 7f538e3c3640 1 -- 172.21.15.28:0/1877283822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5388068850 msgr2=0x7f5388106670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.177+0000 7f538e3c3640 1 --2- 172.21.15.28:0/1877283822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5388068850 0x7f5388106670 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f5378009900 tx=0x7f537802bc30 comp rx=0 tx=0).stop 2024-04-21T08:52:24.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.177+0000 7f538e3c3640 1 -- 172.21.15.28:0/1877283822 shutdown_connections 2024-04-21T08:52:24.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.177+0000 7f538e3c3640 1 --2- 172.21.15.28:0/1877283822 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f53881005b0 0x7f5388100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.177+0000 7f538e3c3640 1 --2- 172.21.15.28:0/1877283822 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5388068cc0 0x7f5388106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.177+0000 7f538e3c3640 1 --2- 172.21.15.28:0/1877283822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5388068850 0x7f5388106670 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.177+0000 7f538e3c3640 1 -- 172.21.15.28:0/1877283822 >> 172.21.15.28:0/1877283822 conn(0x7f53880773b0 msgr2=0x7f53880fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:24.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.178+0000 7f538e3c3640 1 -- 172.21.15.28:0/1877283822 shutdown_connections 2024-04-21T08:52:24.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.178+0000 7f538e3c3640 1 -- 172.21.15.28:0/1877283822 wait complete. 2024-04-21T08:52:24.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.179+0000 7f538e3c3640 1 Processor -- start 2024-04-21T08:52:24.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.179+0000 7f538e3c3640 1 -- start start 2024-04-21T08:52:24.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.180+0000 7f538e3c3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5388068850 0x7f53881963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.180+0000 7f538e3c3640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5388068cc0 0x7f5388196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.180+0000 7f538e3c3640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f53881005b0 0x7f538819acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.180+0000 7f538e3c3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f538810ff90 con 0x7f5388068850 2024-04-21T08:52:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.180+0000 7f538e3c3640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f538810fe10 con 0x7f5388068cc0 2024-04-21T08:52:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.180+0000 7f538e3c3640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5388110110 con 0x7f53881005b0 2024-04-21T08:52:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.180+0000 7f5387fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5388068850 0x7f53881963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.180+0000 7f5387fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5388068850 0x7f53881963f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56240/0 (socket says 172.21.15.28:56240) 2024-04-21T08:52:24.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.180+0000 7f5387fff640 1 -- 172.21.15.28:0/1899655771 learned_addr learned my addr 172.21.15.28:0/1899655771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:24.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.180+0000 7f538c960640 1 --2- 172.21.15.28:0/1899655771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f53881005b0 0x7f538819acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.180+0000 7f53877fe640 1 --2- 172.21.15.28:0/1899655771 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5388068cc0 0x7f5388196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.181+0000 7f5387fff640 1 -- 172.21.15.28:0/1899655771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f53881005b0 msgr2=0x7f538819acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:24.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.181+0000 7f5387fff640 1 --2- 172.21.15.28:0/1899655771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f53881005b0 0x7f538819acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.181+0000 7f5387fff640 1 -- 172.21.15.28:0/1899655771 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5388068cc0 msgr2=0x7f5388196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:24.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.181+0000 7f5387fff640 1 --2- 172.21.15.28:0/1899655771 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5388068cc0 0x7f5388196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.181+0000 7f5387fff640 1 -- 172.21.15.28:0/1899655771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f538819b3e0 con 0x7f5388068850 2024-04-21T08:52:24.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.181+0000 7f5387fff640 1 --2- 172.21.15.28:0/1899655771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5388068850 0x7f53881963f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f5378005ab0 tx=0x7f537802c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:24.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.182+0000 7f53857fa640 1 -- 172.21.15.28:0/1899655771 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378043020 con 0x7f5388068850 2024-04-21T08:52:24.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.183+0000 7f53857fa640 1 -- 172.21.15.28:0/1899655771 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53780052b0 con 0x7f5388068850 2024-04-21T08:52:24.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.183+0000 7f53857fa640 1 -- 172.21.15.28:0/1899655771 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378003bc0 con 0x7f5388068850 2024-04-21T08:52:24.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.183+0000 7f538e3c3640 1 -- 172.21.15.28:0/1899655771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f538819b670 con 0x7f5388068850 2024-04-21T08:52:24.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.185+0000 7f538e3c3640 1 -- 172.21.15.28:0/1899655771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53881a2f60 con 0x7f5388068850 2024-04-21T08:52:24.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.188+0000 7f53857fa640 1 -- 172.21.15.28:0/1899655771 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f5378046020 con 0x7f5388068850 2024-04-21T08:52:24.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.189+0000 7f53857fa640 1 --2- 172.21.15.28:0/1899655771 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f53700a6160 0x7f53700a8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.190+0000 7f53857fa640 1 -- 172.21.15.28:0/1899655771 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f53780f2d20 con 0x7f5388068850 2024-04-21T08:52:24.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.190+0000 7f5390fb0740 1 Processor -- start 2024-04-21T08:52:24.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.190+0000 7f5390fb0740 1 -- start start 2024-04-21T08:52:24.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.190+0000 7f5390fb0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d9e5e0 0x560425d9e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.190+0000 7f5390fb0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425d92be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.190+0000 7f5390fb0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d93120 0x560425d99fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.190+0000 7f53877fe640 1 --2- 172.21.15.28:0/1899655771 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f53700a6160 0x7f53700a8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.190+0000 7f5390fb0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560425cfd1e0 con 0x560425d93120 2024-04-21T08:52:24.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.190+0000 7f5390fb0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x560425c9a390 con 0x560425d9e5e0 2024-04-21T08:52:24.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.190+0000 7f5390fb0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x560425cf91a0 con 0x560425d92760 2024-04-21T08:52:24.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.191+0000 7f53877fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425d92be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.191+0000 7f53877fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425d92be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:51280/0 (socket says 172.21.15.28:51280) 2024-04-21T08:52:24.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.191+0000 7f53877fe640 1 -- 172.21.15.28:0/3970065467 learned_addr learned my addr 172.21.15.28:0/3970065467 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:24.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.191+0000 7f53877fe640 1 -- 172.21.15.28:0/3970065467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d9e5e0 msgr2=0x560425d9e9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:52:24.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.191+0000 7f53877fe640 1 --2- 172.21.15.28:0/3970065467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d9e5e0 0x560425d9e9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.191+0000 7f53877fe640 1 -- 172.21.15.28:0/3970065467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d93120 msgr2=0x560425d99fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:24.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.191+0000 7f53877fe640 1 --2- 172.21.15.28:0/3970065467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d93120 0x560425d99fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.191+0000 7f53877fe640 1 -- 172.21.15.28:0/3970065467 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560425ad97d0 con 0x560425d92760 2024-04-21T08:52:24.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.192+0000 7f53877fe640 1 --2- 172.21.15.28:0/3970065467 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425d92be0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f537c0146d0 tx=0x7f537c014b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=901837fa0ba77b9d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:24.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.192+0000 7f536e7fc640 1 -- 172.21.15.28:0/3970065467 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c022070 con 0x560425d92760 2024-04-21T08:52:24.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.192+0000 7f53877fe640 1 --2- 172.21.15.28:0/1899655771 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f53700a6160 0x7f53700a8620 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f537c011750 tx=0x7f537c01c000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:24.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.193+0000 7f536e7fc640 1 -- 172.21.15.28:0/3970065467 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f537c005500 con 0x560425d92760 2024-04-21T08:52:24.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.193+0000 7f536e7fc640 1 -- 172.21.15.28:0/3970065467 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c012ac0 con 0x560425d92760 2024-04-21T08:52:24.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.194+0000 7f5390fb0740 1 -- 172.21.15.28:0/3970065467 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 msgr2=0x560425d92be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.194+0000 7f5390fb0740 1 --2- 172.21.15.28:0/3970065467 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425d92be0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f537c0146d0 tx=0x7f537c014b90 comp rx=0 tx=0).stop 2024-04-21T08:52:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.194+0000 7f5390fb0740 1 -- 172.21.15.28:0/3970065467 shutdown_connections 2024-04-21T08:52:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.194+0000 7f5390fb0740 1 --2- 172.21.15.28:0/3970065467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d93120 0x560425d99fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.194+0000 7f5390fb0740 1 --2- 172.21.15.28:0/3970065467 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425d92be0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.194+0000 7f5390fb0740 1 --2- 172.21.15.28:0/3970065467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d9e5e0 0x560425d9e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.194+0000 7f5390fb0740 1 -- 172.21.15.28:0/3970065467 >> 172.21.15.28:0/3970065467 conn(0x560425d95280 msgr2=0x560425d95660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.194+0000 7f5390fb0740 1 -- 172.21.15.28:0/3970065467 shutdown_connections 2024-04-21T08:52:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.195+0000 7f5390fb0740 1 -- 172.21.15.28:0/3970065467 wait complete. 2024-04-21T08:52:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.195+0000 7f5390fb0740 1 -- start start 2024-04-21T08:52:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.196+0000 7f5390fb0740 1 Processor -- start 2024-04-21T08:52:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.196+0000 7f5390fb0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425db88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.196+0000 7f5390fb0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d93120 0x560425db8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.196+0000 7f5390fb0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d9e5e0 0x560425db2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.196+0000 7f5390fb0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560425d9a760 con 0x560425d9e5e0 2024-04-21T08:52:24.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.196+0000 7f5390fb0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x560425cb62f0 con 0x560425d93120 2024-04-21T08:52:24.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.196+0000 7f5390fb0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x560425d9a8e0 con 0x560425d92760 2024-04-21T08:52:24.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.196+0000 7f5390fb0740 10 client.0 _ll_register_callbacks cb 0x560425cc3580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:24.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.197+0000 7f538c960640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d9e5e0 0x560425db2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.197+0000 7f5387fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425db88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.197+0000 7f5387fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425db88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:51282/0 (socket says 172.21.15.28:51282) 2024-04-21T08:52:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.197+0000 7f5387fff640 1 -- 172.21.15.28:0/511584089 learned_addr learned my addr 172.21.15.28:0/511584089 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.197+0000 7f538c960640 1 -- 172.21.15.28:0/511584089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 msgr2=0x560425db88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.197+0000 7f538c960640 1 --2- 172.21.15.28:0/511584089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425db88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.198+0000 7f538c960640 1 -- 172.21.15.28:0/511584089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d93120 msgr2=0x560425db8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.198+0000 7f538c960640 1 --2- 172.21.15.28:0/511584089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d93120 0x560425db8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.198+0000 7f538c960640 1 -- 172.21.15.28:0/511584089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560425a0f3f0 con 0x560425d9e5e0 2024-04-21T08:52:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.198+0000 7f538c960640 1 --2- 172.21.15.28:0/511584089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d9e5e0 0x560425db2cc0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f538000e7c0 tx=0x7f538000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1f60c1db95fcff server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.198+0000 7f53677fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.198+0000 7f53677fe640 1 -- 172.21.15.28:0/511584089 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538000ccc0 con 0x560425d9e5e0 2024-04-21T08:52:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.199+0000 7f5390fb0740 1 -- 172.21.15.28:0/511584089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f537c005500 con 0x560425d9e5e0 2024-04-21T08:52:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.199+0000 7f53677fe640 1 -- 172.21.15.28:0/511584089 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f5380003970 con 0x560425d9e5e0 2024-04-21T08:52:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.199+0000 7f53677fe640 1 -- 172.21.15.28:0/511584089 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5380022050 con 0x560425d9e5e0 2024-04-21T08:52:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.199+0000 7f5390fb0740 10 client.16203 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.199+0000 7f5390fb0740 1 -- 172.21.15.28:0/511584089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5604259cf0f0 con 0x560425d9e5e0 2024-04-21T08:52:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.201+0000 7f53677fe640 1 -- 172.21.15.28:0/511584089 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f537c005500 con 0x560425d9e5e0 2024-04-21T08:52:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.201+0000 7f53677fe640 1 -- 172.21.15.28:0/511584089 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5380010430 con 0x560425d9e5e0 2024-04-21T08:52:24.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.201+0000 7f5390fb0740 10 client.16203 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:24.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.201+0000 7f5390fb0740 10 client.16203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:24.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.201+0000 7f5390fb0740 20 client.16203 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:24.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.201+0000 7f5390fb0740 1 --2- 172.21.15.28:0/511584089 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560425dc2fc0 0x560425dc5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.201+0000 7f5390fb0740 4 client.16203 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:24.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.201+0000 7f5390fb0740 1 -- 172.21.15.28:0/511584089 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560425d0ac00 con 0x560425dc2fc0 2024-04-21T08:52:24.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.201+0000 7f5387fff640 1 --2- 172.21.15.28:0/511584089 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560425dc2fc0 0x560425dc5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.203+0000 7f53677fe640 10 client.16203 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:24.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.203+0000 7f5387fff640 1 --2- 172.21.15.28:0/511584089 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560425dc2fc0 0x560425dc5400 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d16e3b7df25213bc server_cookie=210d8444f897ee80 in_seq=0 out_seq=0 2024-04-21T08:52:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.204+0000 7f53677fe640 1 -- 172.21.15.28:0/511584089 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560425d0ac00 con 0x560425dc2fc0 2024-04-21T08:52:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.204+0000 7f53677fe640 10 client.16203 handle_command_reply: tid=0 2024-04-21T08:52:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.204+0000 7f5390fb0740 1 client.16203 shutdown 2024-04-21T08:52:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.205+0000 7f5390fb0740 1 -- 172.21.15.28:0/511584089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d9e5e0 msgr2=0x560425db2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.205+0000 7f5390fb0740 1 --2- 172.21.15.28:0/511584089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d9e5e0 0x560425db2cc0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f538000e7c0 tx=0x7f538000ec80 comp rx=0 tx=0).stop 2024-04-21T08:52:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.205+0000 7f5390fb0740 1 -- 172.21.15.28:0/511584089 shutdown_connections 2024-04-21T08:52:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.205+0000 7f5390fb0740 1 --2- 172.21.15.28:0/511584089 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560425dc2fc0 0x560425dc5400 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.205+0000 7f5390fb0740 1 --2- 172.21.15.28:0/511584089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d9e5e0 0x560425db2cc0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.205+0000 7f5390fb0740 1 --2- 172.21.15.28:0/511584089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d93120 0x560425db8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.205+0000 7f5390fb0740 1 --2- 172.21.15.28:0/511584089 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425db88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.205+0000 7f5390fb0740 1 -- 172.21.15.28:0/511584089 >> 172.21.15.28:0/511584089 conn(0x560425d95280 msgr2=0x560425d94980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.205+0000 7f53677fe640 0 client.16203 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.205+0000 7f5390fb0740 1 -- 172.21.15.28:0/511584089 shutdown_connections 2024-04-21T08:52:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.205+0000 7f5390fb0740 1 -- 172.21.15.28:0/511584089 wait complete. 2024-04-21T08:52:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.205+0000 7f5390fb0740 20 client.16203 trim_cache size 0 max 16384 2024-04-21T08:52:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.240+0000 7f5390fb0740 1 Processor -- start 2024-04-21T08:52:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.240+0000 7f5390fb0740 1 -- start start 2024-04-21T08:52:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.241+0000 7f5390fb0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425db2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.241+0000 7f5390fb0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d93120 0x560425d99f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.241+0000 7f5390fb0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d9e5e0 0x560425db8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.241+0000 7f5390fb0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53881af970 con 0x560425d9e5e0 2024-04-21T08:52:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.241+0000 7f5390fb0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f53881ad240 con 0x560425d93120 2024-04-21T08:52:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.241+0000 7f5390fb0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f53881ab090 con 0x560425d92760 2024-04-21T08:52:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.241+0000 7f53877fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d93120 0x560425d99f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.241+0000 7f53877fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d93120 0x560425d99f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38150/0 (socket says 172.21.15.28:38150) 2024-04-21T08:52:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.241+0000 7f53877fe640 1 -- 172.21.15.28:0/3908899223 learned_addr learned my addr 172.21.15.28:0/3908899223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.241+0000 7f5387fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425db2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.241+0000 7f538c960640 1 --2- 172.21.15.28:0/3908899223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d9e5e0 0x560425db8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.242+0000 7f53877fe640 1 -- 172.21.15.28:0/3908899223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 msgr2=0x560425db2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.242+0000 7f53877fe640 1 --2- 172.21.15.28:0/3908899223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425db2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.242+0000 7f53877fe640 1 -- 172.21.15.28:0/3908899223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d9e5e0 msgr2=0x560425db8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.242+0000 7f53877fe640 1 --2- 172.21.15.28:0/3908899223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d9e5e0 0x560425db8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.242+0000 7f53877fe640 1 -- 172.21.15.28:0/3908899223 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560425d9a8d0 con 0x560425d93120 2024-04-21T08:52:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.242+0000 7f5387fff640 1 --2- 172.21.15.28:0/3908899223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425db2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.242+0000 7f53877fe640 1 --2- 172.21.15.28:0/3908899223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d93120 0x560425d99f80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f537c00b890 tx=0x7f537c012c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c17a0b5b612fbac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.243+0000 7f536dffb640 1 -- 172.21.15.28:0/3908899223 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c022040 con 0x560425d93120 2024-04-21T08:52:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.243+0000 7f536dffb640 1 -- 172.21.15.28:0/3908899223 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f537c0127c0 con 0x560425d93120 2024-04-21T08:52:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.243+0000 7f536dffb640 1 -- 172.21.15.28:0/3908899223 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c0052b0 con 0x560425d93120 2024-04-21T08:52:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.244+0000 7f5390fb0740 1 -- 172.21.15.28:0/3908899223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d93120 msgr2=0x560425d99f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.244+0000 7f5390fb0740 1 --2- 172.21.15.28:0/3908899223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d93120 0x560425d99f80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f537c00b890 tx=0x7f537c012c30 comp rx=0 tx=0).stop 2024-04-21T08:52:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.244+0000 7f5390fb0740 1 -- 172.21.15.28:0/3908899223 shutdown_connections 2024-04-21T08:52:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.244+0000 7f5390fb0740 1 --2- 172.21.15.28:0/3908899223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d9e5e0 0x560425db8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.244+0000 7f5390fb0740 1 --2- 172.21.15.28:0/3908899223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d93120 0x560425d99f80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.244+0000 7f5390fb0740 1 --2- 172.21.15.28:0/3908899223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d92760 0x560425db2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.244+0000 7f5390fb0740 1 -- 172.21.15.28:0/3908899223 >> 172.21.15.28:0/3908899223 conn(0x560425d95280 msgr2=0x560425db8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.245+0000 7f5390fb0740 1 -- 172.21.15.28:0/3908899223 shutdown_connections 2024-04-21T08:52:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.245+0000 7f5390fb0740 1 -- 172.21.15.28:0/3908899223 wait complete. 2024-04-21T08:52:24.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.246+0000 7f5390fb0740 1 -- start start 2024-04-21T08:52:24.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.247+0000 7f5390fb0740 1 Processor -- start 2024-04-21T08:52:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.248+0000 7f5390fb0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d92760 0x560425dbb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.248+0000 7f5390fb0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d93120 0x560425dbbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.248+0000 7f5390fb0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d9e5e0 0x560425dbc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.248+0000 7f5390fb0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53881ab700 con 0x560425d92760 2024-04-21T08:52:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.248+0000 7f5390fb0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f53881acb70 con 0x560425d9e5e0 2024-04-21T08:52:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.248+0000 7f5390fb0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f53881abdd0 con 0x560425d93120 2024-04-21T08:52:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.248+0000 7f5390fb0740 10 client.0 _ll_register_callbacks cb 0x7f5380007460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.249+0000 7f538c960640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d9e5e0 0x560425dbc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.249+0000 7f538c960640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d9e5e0 0x560425dbc110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38160/0 (socket says 172.21.15.28:38160) 2024-04-21T08:52:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.249+0000 7f538c960640 1 -- 172.21.15.28:0/3866348354 learned_addr learned my addr 172.21.15.28:0/3866348354 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:24.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.249+0000 7f53877fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d93120 0x560425dbbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.249+0000 7f5387fff640 1 --2- 172.21.15.28:0/3866348354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d92760 0x560425dbb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.250+0000 7f538c960640 1 -- 172.21.15.28:0/3866348354 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d93120 msgr2=0x560425dbbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.250+0000 7f538c960640 1 --2- 172.21.15.28:0/3866348354 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d93120 0x560425dbbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.250+0000 7f538c960640 1 -- 172.21.15.28:0/3866348354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d92760 msgr2=0x560425dbb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.250+0000 7f538c960640 1 --2- 172.21.15.28:0/3866348354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d92760 0x560425dbb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.250+0000 7f538c960640 1 -- 172.21.15.28:0/3866348354 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560425e39590 con 0x560425d9e5e0 2024-04-21T08:52:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.251+0000 7f538c960640 1 --2- 172.21.15.28:0/3866348354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d9e5e0 0x560425dbc110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5380007640 tx=0x7f5380010660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e21817edaf4d2dd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.252+0000 7f53677fe640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:52:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.252+0000 7f53677fe640 1 -- 172.21.15.28:0/3866348354 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538000ee70 con 0x560425d9e5e0 2024-04-21T08:52:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.252+0000 7f53677fe640 1 -- 172.21.15.28:0/3866348354 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f538000cac0 con 0x560425d9e5e0 2024-04-21T08:52:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.252+0000 7f5390fb0740 1 -- 172.21.15.28:0/3866348354 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f537c0127c0 con 0x560425d9e5e0 2024-04-21T08:52:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.252+0000 7f53677fe640 1 -- 172.21.15.28:0/3866348354 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5380037050 con 0x560425d9e5e0 2024-04-21T08:52:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.252+0000 7f5390fb0740 10 client.25637 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.252+0000 7f5390fb0740 1 -- 172.21.15.28:0/3866348354 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560425e3a770 con 0x560425d9e5e0 2024-04-21T08:52:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.254+0000 7f53677fe640 1 -- 172.21.15.28:0/3866348354 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f537c0127c0 con 0x560425d9e5e0 2024-04-21T08:52:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.254+0000 7f53677fe640 1 -- 172.21.15.28:0/3866348354 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f53800051f0 con 0x560425d9e5e0 2024-04-21T08:52:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.254+0000 7f5390fb0740 10 client.25637 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.254+0000 7f5390fb0740 10 client.25637 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.254+0000 7f5390fb0740 20 client.25637 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.254+0000 7f5390fb0740 1 --2- 172.21.15.28:0/3866348354 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560425e3aec0 0x560425e3d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.254+0000 7f5390fb0740 4 client.25637 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:52:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.254+0000 7f5390fb0740 1 -- 172.21.15.28:0/3866348354 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560425d04a70 con 0x560425e3aec0 2024-04-21T08:52:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.254+0000 7f5387fff640 1 --2- 172.21.15.28:0/3866348354 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560425e3aec0 0x560425e3d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.258+0000 7f53677fe640 10 client.25637 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.258+0000 7f5387fff640 1 --2- 172.21.15.28:0/3866348354 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560425e3aec0 0x560425e3d2e0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23b9607daff6c4b8 server_cookie=8ffbe20e837f22ed in_seq=0 out_seq=0 2024-04-21T08:52:24.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.263+0000 7f53677fe640 1 -- 172.21.15.28:0/3866348354 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560425d04a70 con 0x560425e3aec0 2024-04-21T08:52:24.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.263+0000 7f53677fe640 10 client.25637 handle_command_reply: tid=0 2024-04-21T08:52:24.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.263+0000 7f5390fb0740 1 client.25637 shutdown 2024-04-21T08:52:24.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.265+0000 7f5390fb0740 1 -- 172.21.15.28:0/3866348354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d9e5e0 msgr2=0x560425dbc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:24.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.265+0000 7f5390fb0740 1 --2- 172.21.15.28:0/3866348354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d9e5e0 0x560425dbc110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5380007640 tx=0x7f5380010660 comp rx=0 tx=0).stop 2024-04-21T08:52:24.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.266+0000 7f5390fb0740 1 -- 172.21.15.28:0/3866348354 shutdown_connections 2024-04-21T08:52:24.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.266+0000 7f5390fb0740 1 --2- 172.21.15.28:0/3866348354 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560425e3aec0 0x560425e3d2e0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.266+0000 7f5390fb0740 1 --2- 172.21.15.28:0/3866348354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560425d9e5e0 0x560425dbc110 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.266+0000 7f5390fb0740 1 --2- 172.21.15.28:0/3866348354 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560425d93120 0x560425dbbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.266+0000 7f5390fb0740 1 --2- 172.21.15.28:0/3866348354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560425d92760 0x560425dbb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.266+0000 7f5390fb0740 1 -- 172.21.15.28:0/3866348354 >> 172.21.15.28:0/3866348354 conn(0x560425d95280 msgr2=0x560425db2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:24.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.266+0000 7f53677fe640 0 client.25637 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:24.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.267+0000 7f5390fb0740 1 -- 172.21.15.28:0/3866348354 shutdown_connections 2024-04-21T08:52:24.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.267+0000 7f5390fb0740 1 -- 172.21.15.28:0/3866348354 wait complete. 2024-04-21T08:52:24.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.267+0000 7f5390fb0740 20 client.25637 trim_cache size 0 max 16384 2024-04-21T08:52:24.268 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:52:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T08:52:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T08:52:24.269 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:52:24.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.270+0000 7f538e3c3640 1 -- 172.21.15.28:0/1899655771 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f53700a6160 msgr2=0x7f53700a8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:24.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.270+0000 7f538e3c3640 1 --2- 172.21.15.28:0/1899655771 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f53700a6160 0x7f53700a8620 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f537c011750 tx=0x7f537c01c000 comp rx=0 tx=0).stop 2024-04-21T08:52:24.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.270+0000 7f538e3c3640 1 -- 172.21.15.28:0/1899655771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5388068850 msgr2=0x7f53881963f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:24.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.270+0000 7f538e3c3640 1 --2- 172.21.15.28:0/1899655771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5388068850 0x7f53881963f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f5378005ab0 tx=0x7f537802c860 comp rx=0 tx=0).stop 2024-04-21T08:52:24.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.273+0000 7f538e3c3640 1 -- 172.21.15.28:0/1899655771 shutdown_connections 2024-04-21T08:52:24.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.273+0000 7f538e3c3640 1 --2- 172.21.15.28:0/1899655771 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f53700a6160 0x7f53700a8620 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.273+0000 7f538e3c3640 1 --2- 172.21.15.28:0/1899655771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f53881005b0 0x7f538819acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.273+0000 7f538e3c3640 1 --2- 172.21.15.28:0/1899655771 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5388068cc0 0x7f5388196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.273+0000 7f538e3c3640 1 --2- 172.21.15.28:0/1899655771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5388068850 0x7f53881963f0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.273+0000 7f538e3c3640 1 -- 172.21.15.28:0/1899655771 >> 172.21.15.28:0/1899655771 conn(0x7f53880773b0 msgr2=0x7f53880fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:24.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.275+0000 7f538e3c3640 1 -- 172.21.15.28:0/1899655771 shutdown_connections 2024-04-21T08:52:24.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.275+0000 7f538e3c3640 1 -- 172.21.15.28:0/1899655771 wait complete. 2024-04-21T08:52:24.309 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:52:24.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:52:24.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.583+0000 7fef93def640 1 -- 172.21.15.28:0/214090391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fef8c109750 msgr2=0x7fef8c1153d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:24.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.583+0000 7fef93def640 1 --2- 172.21.15.28:0/214090391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fef8c109750 0x7fef8c1153d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009930 tx=0x7fef7c02bc40 comp rx=0 tx=0).stop 2024-04-21T08:52:24.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.583+0000 7fef93def640 1 -- 172.21.15.28:0/214090391 shutdown_connections 2024-04-21T08:52:24.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.583+0000 7fef93def640 1 --2- 172.21.15.28:0/214090391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fef8c109750 0x7fef8c1153d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.583+0000 7fef93def640 1 --2- 172.21.15.28:0/214090391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef8c108d90 0x7fef8c109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.583+0000 7fef93def640 1 --2- 172.21.15.28:0/214090391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef8c10e5c0 0x7fef8c10e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.583+0000 7fef93def640 1 -- 172.21.15.28:0/214090391 >> 172.21.15.28:0/214090391 conn(0x7fef8c077be0 msgr2=0x7fef8c07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.589+0000 7fef93def640 1 -- 172.21.15.28:0/214090391 shutdown_connections 2024-04-21T08:52:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.591+0000 7fef93def640 1 -- 172.21.15.28:0/214090391 wait complete. 2024-04-21T08:52:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.592+0000 7fef93def640 1 Processor -- start 2024-04-21T08:52:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.592+0000 7fef93def640 1 -- start start 2024-04-21T08:52:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.593+0000 7fef93def640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef8c108d90 0x7fef8c084680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.593+0000 7fef93def640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fef8c109750 0x7fef8c07e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.593+0000 7fef93def640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef8c10e5c0 0x7fef8c07ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.593+0000 7fef93def640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c07d0c0 con 0x7fef8c108d90 2024-04-21T08:52:24.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.593+0000 7fef93def640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c07cf40 con 0x7fef8c10e5c0 2024-04-21T08:52:24.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.593+0000 7fef93def640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c07d240 con 0x7fef8c109750 2024-04-21T08:52:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.594+0000 7fef9238c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef8c10e5c0 0x7fef8c07ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.594+0000 7fef9238c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef8c10e5c0 0x7fef8c07ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38182/0 (socket says 172.21.15.28:38182) 2024-04-21T08:52:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.594+0000 7fef9238c640 1 -- 172.21.15.28:0/1086036065 learned_addr learned my addr 172.21.15.28:0/1086036065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.594+0000 7fef9238c640 1 -- 172.21.15.28:0/1086036065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fef8c109750 msgr2=0x7fef8c07e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.594+0000 7fef9238c640 1 --2- 172.21.15.28:0/1086036065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fef8c109750 0x7fef8c07e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.595+0000 7fef9238c640 1 -- 172.21.15.28:0/1086036065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef8c108d90 msgr2=0x7fef8c084680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.595+0000 7fef9238c640 1 --2- 172.21.15.28:0/1086036065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef8c108d90 0x7fef8c084680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.595+0000 7fef9238c640 1 -- 172.21.15.28:0/1086036065 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8c07f2e0 con 0x7fef8c10e5c0 2024-04-21T08:52:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.595+0000 7fef9238c640 1 --2- 172.21.15.28:0/1086036065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef8c10e5c0 0x7fef8c07ec30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fef7c02c1d0 tx=0x7fef7c02c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.597+0000 7fef82ffd640 1 -- 172.21.15.28:0/1086036065 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c043020 con 0x7fef8c10e5c0 2024-04-21T08:52:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.597+0000 7fef82ffd640 1 -- 172.21.15.28:0/1086036065 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef7c0052b0 con 0x7fef8c10e5c0 2024-04-21T08:52:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.597+0000 7fef82ffd640 1 -- 172.21.15.28:0/1086036065 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c003b20 con 0x7fef8c10e5c0 2024-04-21T08:52:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.597+0000 7fef93def640 1 -- 172.21.15.28:0/1086036065 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef8c06b970 con 0x7fef8c10e5c0 2024-04-21T08:52:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.598+0000 7fef93def640 1 -- 172.21.15.28:0/1086036065 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef8c06be70 con 0x7fef8c10e5c0 2024-04-21T08:52:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.601+0000 7fef93def640 1 -- 172.21.15.28:0/1086036065 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef70005290 con 0x7fef8c10e5c0 2024-04-21T08:52:24.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:24 smithi152 ceph-mon[31401]: pgmap v39: 129 pgs: 129 active+clean; 1.1 GiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 96 MiB/s wr, 235 op/s 2024-04-21T08:52:24.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.606+0000 7fef82ffd640 1 -- 172.21.15.28:0/1086036065 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fef7c046020 con 0x7fef8c10e5c0 2024-04-21T08:52:24.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:24 smithi028 ceph-mon[28003]: pgmap v39: 129 pgs: 129 active+clean; 1.1 GiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 96 MiB/s wr, 235 op/s 2024-04-21T08:52:24.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.607+0000 7fef82ffd640 1 --2- 172.21.15.28:0/1086036065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fef780a4050 0x7fef780a6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:24.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.608+0000 7fef82ffd640 1 -- 172.21.15.28:0/1086036065 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fef7c0bd4f0 con 0x7fef8c10e5c0 2024-04-21T08:52:24.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.609+0000 7fef91b8b640 1 --2- 172.21.15.28:0/1086036065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fef780a4050 0x7fef780a6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:24.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.610+0000 7fef91b8b640 1 --2- 172.21.15.28:0/1086036065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fef780a4050 0x7fef780a6510 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fef840092f0 tx=0x7fef84009240 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:24.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.612+0000 7fef82ffd640 1 -- 172.21.15.28:0/1086036065 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fef7c04d020 con 0x7fef8c10e5c0 2024-04-21T08:52:24.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:24 smithi171 ceph-mon[31358]: pgmap v39: 129 pgs: 129 active+clean; 1.1 GiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 96 MiB/s wr, 235 op/s 2024-04-21T08:52:24.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.992+0000 7fef93def640 1 -- 172.21.15.28:0/1086036065 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef70005580 con 0x7fef8c10e5c0 2024-04-21T08:52:24.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.993+0000 7fef82ffd640 1 -- 172.21.15.28:0/1086036065 <== mon.2 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7fef7c04d020 con 0x7fef8c10e5c0 2024-04-21T08:52:24.994 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:52:24.994 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:52:11.568185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24487,24488,24463]},"id":1}]} 2024-04-21T08:52:24.995 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-21T08:52:25.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.998+0000 7fef93def640 1 -- 172.21.15.28:0/1086036065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fef780a4050 msgr2=0x7fef780a6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.998+0000 7fef93def640 1 --2- 172.21.15.28:0/1086036065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fef780a4050 0x7fef780a6510 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fef840092f0 tx=0x7fef84009240 comp rx=0 tx=0).stop 2024-04-21T08:52:25.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.998+0000 7fef93def640 1 -- 172.21.15.28:0/1086036065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef8c10e5c0 msgr2=0x7fef8c07ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.998+0000 7fef93def640 1 --2- 172.21.15.28:0/1086036065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef8c10e5c0 0x7fef8c07ec30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fef7c02c1d0 tx=0x7fef7c02c9c0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.999+0000 7fef93def640 1 -- 172.21.15.28:0/1086036065 shutdown_connections 2024-04-21T08:52:25.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.999+0000 7fef93def640 1 --2- 172.21.15.28:0/1086036065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fef780a4050 0x7fef780a6510 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.999+0000 7fef93def640 1 --2- 172.21.15.28:0/1086036065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fef8c10e5c0 0x7fef8c07ec30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.999+0000 7fef93def640 1 --2- 172.21.15.28:0/1086036065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fef8c109750 0x7fef8c07e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.999+0000 7fef93def640 1 --2- 172.21.15.28:0/1086036065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef8c108d90 0x7fef8c084680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.999+0000 7fef93def640 1 -- 172.21.15.28:0/1086036065 >> 172.21.15.28:0/1086036065 conn(0x7fef8c077be0 msgr2=0x7fef8c07a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.999+0000 7fef93def640 1 -- 172.21.15.28:0/1086036065 shutdown_connections 2024-04-21T08:52:25.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:24.999+0000 7fef93def640 1 -- 172.21.15.28:0/1086036065 wait complete. 2024-04-21T08:52:25.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:52:25.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.191+0000 7f93f359e640 1 -- 172.21.15.28:0/2836746039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f4109520 msgr2=0x7f93f41099a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.191+0000 7f93f359e640 1 --2- 172.21.15.28:0/2836746039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f4109520 0x7f93f41099a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f93e4009900 tx=0x7f93e402bc50 comp rx=0 tx=0).stop 2024-04-21T08:52:25.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.191+0000 7f93f359e640 1 -- 172.21.15.28:0/2836746039 shutdown_connections 2024-04-21T08:52:25.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.191+0000 7f93f359e640 1 --2- 172.21.15.28:0/2836746039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93f4117080 0x7f93f4119470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.191+0000 7f93f359e640 1 --2- 172.21.15.28:0/2836746039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f4109520 0x7f93f41099a0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.191+0000 7f93f359e640 1 --2- 172.21.15.28:0/2836746039 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f93f410c6c0 0x7f93f4108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.191+0000 7f93f359e640 1 -- 172.21.15.28:0/2836746039 >> 172.21.15.28:0/2836746039 conn(0x7f93f406d330 msgr2=0x7f93f406d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.191+0000 7f93f359e640 1 -- 172.21.15.28:0/2836746039 shutdown_connections 2024-04-21T08:52:25.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.191+0000 7f93f359e640 1 -- 172.21.15.28:0/2836746039 wait complete. 2024-04-21T08:52:25.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.192+0000 7f93f359e640 1 Processor -- start 2024-04-21T08:52:25.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.192+0000 7f93f359e640 1 -- start start 2024-04-21T08:52:25.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.193+0000 7f93f359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f4109520 0x7f93f41a3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.193+0000 7f93f359e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f93f410c6c0 0x7f93f41a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.193+0000 7f93f359e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93f4117080 0x7f93f41a7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.193+0000 7f93f359e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f93f411cbe0 con 0x7f93f4109520 2024-04-21T08:52:25.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.193+0000 7f93f359e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f93f411ca60 con 0x7f93f4117080 2024-04-21T08:52:25.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.193+0000 7f93f359e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f93f411cd60 con 0x7f93f410c6c0 2024-04-21T08:52:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.193+0000 7f93f2d9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93f4117080 0x7f93f41a7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.194+0000 7f93f2d9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93f4117080 0x7f93f41a7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38206/0 (socket says 172.21.15.28:38206) 2024-04-21T08:52:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.194+0000 7f93f2d9d640 1 -- 172.21.15.28:0/2637165912 learned_addr learned my addr 172.21.15.28:0/2637165912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:25.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.195+0000 7f93f1d9b640 1 --2- 172.21.15.28:0/2637165912 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f93f410c6c0 0x7f93f41a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.195+0000 7f93f2d9d640 1 -- 172.21.15.28:0/2637165912 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f93f410c6c0 msgr2=0x7f93f41a36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.195+0000 7f93f2d9d640 1 --2- 172.21.15.28:0/2637165912 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f93f410c6c0 0x7f93f41a36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.195+0000 7f93f2d9d640 1 -- 172.21.15.28:0/2637165912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f4109520 msgr2=0x7f93f41a3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.195+0000 7f93f2d9d640 1 --2- 172.21.15.28:0/2637165912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f4109520 0x7f93f41a3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.195+0000 7f93f2d9d640 1 -- 172.21.15.28:0/2637165912 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f41a8150 con 0x7f93f4117080 2024-04-21T08:52:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.196+0000 7f93f2d9d640 1 --2- 172.21.15.28:0/2637165912 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93f4117080 0x7f93f41a7a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00cad0 tx=0x7f93ec00cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.196+0000 7f93e37fe640 1 -- 172.21.15.28:0/2637165912 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ec00ae20 con 0x7f93f4117080 2024-04-21T08:52:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.197+0000 7f93f359e640 1 -- 172.21.15.28:0/2637165912 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93f41a8440 con 0x7f93f4117080 2024-04-21T08:52:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.197+0000 7f93f359e640 1 -- 172.21.15.28:0/2637165912 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93f410cb00 con 0x7f93f4117080 2024-04-21T08:52:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.201+0000 7f93e37fe640 1 -- 172.21.15.28:0/2637165912 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93ec003e40 con 0x7f93f4117080 2024-04-21T08:52:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.201+0000 7f93e37fe640 1 -- 172.21.15.28:0/2637165912 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ec00f6b0 con 0x7f93f4117080 2024-04-21T08:52:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.202+0000 7f93fbb87740 1 Processor -- start 2024-04-21T08:52:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.203+0000 7f93fbb87740 1 -- start start 2024-04-21T08:52:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.203+0000 7f93fbb87740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b06d0 0x55629e8b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.203+0000 7f93fbb87740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b1080 0x55629e8b9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.203+0000 7f93fbb87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b9c00 0x55629e8bc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.203+0000 7f93fbb87740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55629e8151f0 con 0x55629e8b9c00 2024-04-21T08:52:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.203+0000 7f93fbb87740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55629e7b24b0 con 0x55629e8b1080 2024-04-21T08:52:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.203+0000 7f93fbb87740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55629e8111b0 con 0x55629e8b06d0 2024-04-21T08:52:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.204+0000 7f93f1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b9c00 0x55629e8bc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.204+0000 7f93f1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b9c00 0x55629e8bc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56344/0 (socket says 172.21.15.28:56344) 2024-04-21T08:52:25.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.204+0000 7f93f1d9b640 1 -- 172.21.15.28:0/1145184806 learned_addr learned my addr 172.21.15.28:0/1145184806 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:25.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.204+0000 7f93f259c640 1 --2- 172.21.15.28:0/1145184806 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b1080 0x55629e8b9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.205+0000 7f93f1d9b640 1 --2- 172.21.15.28:0/1145184806 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b06d0 0x55629e8b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.205+0000 7f93f1d9b640 1 -- 172.21.15.28:0/1145184806 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b06d0 msgr2=0x55629e8b0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.205+0000 7f93f1d9b640 1 --2- 172.21.15.28:0/1145184806 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b06d0 0x55629e8b0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.205+0000 7f93f1d9b640 1 -- 172.21.15.28:0/1145184806 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b1080 msgr2=0x55629e8b9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.205+0000 7f93f1d9b640 1 --2- 172.21.15.28:0/1145184806 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b1080 0x55629e8b9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.205+0000 7f93f1d9b640 1 -- 172.21.15.28:0/1145184806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55629e7a41b0 con 0x55629e8b9c00 2024-04-21T08:52:25.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.206+0000 7f93f1d9b640 1 --2- 172.21.15.28:0/1145184806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b9c00 0x55629e8bc050 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f93e4005940 tx=0x7f93e40058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4020dc145a1f2228 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.206+0000 7f93e0ff9640 1 -- 172.21.15.28:0/1145184806 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e4004030 con 0x55629e8b9c00 2024-04-21T08:52:25.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.206+0000 7f93e37fe640 1 -- 172.21.15.28:0/2637165912 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f93ec003990 con 0x7f93f4117080 2024-04-21T08:52:25.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.206+0000 7f93e0ff9640 1 -- 172.21.15.28:0/1145184806 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93e4046070 con 0x55629e8b9c00 2024-04-21T08:52:25.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.206+0000 7f93e0ff9640 1 -- 172.21.15.28:0/1145184806 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e4004920 con 0x55629e8b9c00 2024-04-21T08:52:25.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.207+0000 7f93fbb87740 1 -- 172.21.15.28:0/1145184806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b9c00 msgr2=0x55629e8bc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.208+0000 7f93fbb87740 1 --2- 172.21.15.28:0/1145184806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b9c00 0x55629e8bc050 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f93e4005940 tx=0x7f93e40058d0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.208+0000 7f93fbb87740 1 -- 172.21.15.28:0/1145184806 shutdown_connections 2024-04-21T08:52:25.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.208+0000 7f93fbb87740 1 --2- 172.21.15.28:0/1145184806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b9c00 0x55629e8bc050 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.208+0000 7f93fbb87740 1 --2- 172.21.15.28:0/1145184806 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b1080 0x55629e8b9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.208+0000 7f93fbb87740 1 --2- 172.21.15.28:0/1145184806 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b06d0 0x55629e8b0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.208+0000 7f93fbb87740 1 -- 172.21.15.28:0/1145184806 >> 172.21.15.28:0/1145184806 conn(0x55629e8aada0 msgr2=0x55629e8ab180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.208+0000 7f93fbb87740 1 -- 172.21.15.28:0/1145184806 shutdown_connections 2024-04-21T08:52:25.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.208+0000 7f93fbb87740 1 -- 172.21.15.28:0/1145184806 wait complete. 2024-04-21T08:52:25.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.208+0000 7f93e37fe640 1 --2- 172.21.15.28:0/2637165912 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f93dc0a3ee0 0x7f93dc0a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.208+0000 7f93f259c640 1 --2- 172.21.15.28:0/2637165912 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f93dc0a3ee0 0x7f93dc0a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.209+0000 7f93e37fe640 1 -- 172.21.15.28:0/2637165912 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f93ec0d0e30 con 0x7f93f4117080 2024-04-21T08:52:25.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.209+0000 7f93fbb87740 1 -- start start 2024-04-21T08:52:25.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.209+0000 7f93fbb87740 1 Processor -- start 2024-04-21T08:52:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.210+0000 7f93fbb87740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b06d0 0x55629e8b94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.210+0000 7f93f259c640 1 --2- 172.21.15.28:0/2637165912 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f93dc0a3ee0 0x7f93dc0a63a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x55629e8abe10 tx=0x7f93e80090e0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.211+0000 7f93fbb87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b1080 0x55629e8b6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.211+0000 7f93fbb87740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b9c00 0x55629e8b6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.211+0000 7f93fbb87740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55629e8bcb10 con 0x55629e8b1080 2024-04-21T08:52:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.211+0000 7f93fbb87740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55629e7ce500 con 0x55629e8b06d0 2024-04-21T08:52:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.211+0000 7f93fbb87740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55629e8bcc90 con 0x55629e8b9c00 2024-04-21T08:52:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.211+0000 7f93f1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b1080 0x55629e8b6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.211+0000 7f93f259c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b06d0 0x55629e8b94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.211+0000 7f93f1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b1080 0x55629e8b6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56360/0 (socket says 172.21.15.28:56360) 2024-04-21T08:52:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.211+0000 7f93f1d9b640 1 -- 172.21.15.28:0/4209646980 learned_addr learned my addr 172.21.15.28:0/4209646980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.211+0000 7f93fbb87740 10 client.0 _ll_register_callbacks cb 0x55629e79cb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.212+0000 7f93f2d9d640 1 --2- 172.21.15.28:0/4209646980 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b9c00 0x55629e8b6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.212+0000 7f93f1d9b640 1 -- 172.21.15.28:0/4209646980 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b9c00 msgr2=0x55629e8b6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.212+0000 7f93f1d9b640 1 --2- 172.21.15.28:0/4209646980 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b9c00 0x55629e8b6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.212+0000 7f93f1d9b640 1 -- 172.21.15.28:0/4209646980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b06d0 msgr2=0x55629e8b94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.212+0000 7f93f1d9b640 1 --2- 172.21.15.28:0/4209646980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b06d0 0x55629e8b94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.212+0000 7f93f1d9b640 1 -- 172.21.15.28:0/4209646980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55629e5273f0 con 0x55629e8b1080 2024-04-21T08:52:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.212+0000 7f93f2d9d640 1 --2- 172.21.15.28:0/4209646980 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b9c00 0x55629e8b6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:52:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.212+0000 7f93f259c640 1 --2- 172.21.15.28:0/4209646980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b06d0 0x55629e8b94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.213+0000 7f93f1d9b640 1 --2- 172.21.15.28:0/4209646980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b1080 0x55629e8b6830 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f93e4000c00 tx=0x7f93e402bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78df776b4af3d77 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.213+0000 7f93d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.214+0000 7f93d9ffb640 1 -- 172.21.15.28:0/4209646980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e4050070 con 0x55629e8b1080 2024-04-21T08:52:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.214+0000 7f93d9ffb640 1 -- 172.21.15.28:0/4209646980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f93e4004040 con 0x55629e8b1080 2024-04-21T08:52:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.214+0000 7f93fbb87740 1 -- 172.21.15.28:0/4209646980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93e4046070 con 0x55629e8b1080 2024-04-21T08:52:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.214+0000 7f93d9ffb640 1 -- 172.21.15.28:0/4209646980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f93e4004240 con 0x55629e8b1080 2024-04-21T08:52:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.214+0000 7f93fbb87740 10 client.16221 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.214+0000 7f93fbb87740 1 -- 172.21.15.28:0/4209646980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55629e5f17d0 con 0x55629e8b1080 2024-04-21T08:52:25.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.215+0000 7f93d9ffb640 1 -- 172.21.15.28:0/4209646980 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93e4046070 con 0x55629e8b1080 2024-04-21T08:52:25.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.215+0000 7f93d9ffb640 1 -- 172.21.15.28:0/4209646980 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f93e40035f0 con 0x55629e8b1080 2024-04-21T08:52:25.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.216+0000 7f93fbb87740 10 client.16221 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:25.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.216+0000 7f93fbb87740 10 client.16221 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:52:25.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.216+0000 7f93fbb87740 20 client.16221 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:25.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.216+0000 7f93fbb87740 1 --2- 172.21.15.28:0/4209646980 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55629e8c9c50 0x55629e8cc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.216+0000 7f93fbb87740 4 client.16221 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:25.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.216+0000 7f93fbb87740 1 -- 172.21.15.28:0/4209646980 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55629e7b4930 con 0x55629e8c9c50 2024-04-21T08:52:25.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.217+0000 7f93f259c640 1 --2- 172.21.15.28:0/4209646980 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55629e8c9c50 0x55629e8cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.219+0000 7f93d9ffb640 10 client.16221 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:52:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.219+0000 7f93f259c640 1 --2- 172.21.15.28:0/4209646980 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55629e8c9c50 0x55629e8cc040 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=efb9c6f6dc87b06f server_cookie=af74ce0c7d36b647 in_seq=0 out_seq=0 2024-04-21T08:52:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.221+0000 7f93d9ffb640 1 -- 172.21.15.28:0/4209646980 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55629e7b4930 con 0x55629e8c9c50 2024-04-21T08:52:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.221+0000 7f93d9ffb640 10 client.16221 handle_command_reply: tid=0 2024-04-21T08:52:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.221+0000 7f93fbb87740 1 client.16221 shutdown 2024-04-21T08:52:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.222+0000 7f93fbb87740 1 -- 172.21.15.28:0/4209646980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b1080 msgr2=0x55629e8b6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.222+0000 7f93fbb87740 1 --2- 172.21.15.28:0/4209646980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b1080 0x55629e8b6830 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f93e4000c00 tx=0x7f93e402bc50 comp rx=0 tx=0).stop 2024-04-21T08:52:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.222+0000 7f93fbb87740 1 -- 172.21.15.28:0/4209646980 shutdown_connections 2024-04-21T08:52:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.222+0000 7f93fbb87740 1 --2- 172.21.15.28:0/4209646980 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55629e8c9c50 0x55629e8cc040 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.222+0000 7f93d9ffb640 0 client.16221 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:52:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.222+0000 7f93fbb87740 1 --2- 172.21.15.28:0/4209646980 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b9c00 0x55629e8b6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.222+0000 7f93fbb87740 1 --2- 172.21.15.28:0/4209646980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b1080 0x55629e8b6830 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.222+0000 7f93fbb87740 1 --2- 172.21.15.28:0/4209646980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b06d0 0x55629e8b94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.222+0000 7f93fbb87740 1 -- 172.21.15.28:0/4209646980 >> 172.21.15.28:0/4209646980 conn(0x55629e8aada0 msgr2=0x55629e8acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.222+0000 7f93fbb87740 1 -- 172.21.15.28:0/4209646980 shutdown_connections 2024-04-21T08:52:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.222+0000 7f93fbb87740 1 -- 172.21.15.28:0/4209646980 wait complete. 2024-04-21T08:52:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.222+0000 7f93fbb87740 20 client.16221 trim_cache size 0 max 16384 2024-04-21T08:52:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.244+0000 7f93fbb87740 1 Processor -- start 2024-04-21T08:52:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.244+0000 7f93fbb87740 1 -- start start 2024-04-21T08:52:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.244+0000 7f93fbb87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b06d0 0x55629e8b5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.244+0000 7f93fbb87740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b1080 0x55629e8c5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.244+0000 7f93fbb87740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b9c00 0x55629e8b8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.244+0000 7f93fbb87740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f93d40127f0 con 0x55629e8b06d0 2024-04-21T08:52:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.244+0000 7f93fbb87740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f93d40100c0 con 0x55629e8b9c00 2024-04-21T08:52:25.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.244+0000 7f93f259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b06d0 0x55629e8b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.244+0000 7f93fbb87740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f93d400ded0 con 0x55629e8b1080 2024-04-21T08:52:25.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.245+0000 7f93f259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b06d0 0x55629e8b5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56362/0 (socket says 172.21.15.28:56362) 2024-04-21T08:52:25.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.245+0000 7f93f259c640 1 -- 172.21.15.28:0/3399433823 learned_addr learned my addr 172.21.15.28:0/3399433823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:25.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.245+0000 7f93f259c640 1 -- 172.21.15.28:0/3399433823 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b1080 msgr2=0x55629e8c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.245+0000 7f93f259c640 1 --2- 172.21.15.28:0/3399433823 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b1080 0x55629e8c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.245+0000 7f93f259c640 1 -- 172.21.15.28:0/3399433823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b9c00 msgr2=0x55629e8b8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.245+0000 7f93f259c640 1 --2- 172.21.15.28:0/3399433823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b9c00 0x55629e8b8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.245+0000 7f93f259c640 1 -- 172.21.15.28:0/3399433823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55629e8b0e20 con 0x55629e8b06d0 2024-04-21T08:52:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.245+0000 7f93f259c640 1 --2- 172.21.15.28:0/3399433823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b06d0 0x55629e8b5990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x55629e8b01a0 tx=0x7f93e8011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8fef43870af5360 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.246+0000 7f93dbfff640 1 -- 172.21.15.28:0/3399433823 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e801b040 con 0x55629e8b06d0 2024-04-21T08:52:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.246+0000 7f93dbfff640 1 -- 172.21.15.28:0/3399433823 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93e802b030 con 0x55629e8b06d0 2024-04-21T08:52:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.246+0000 7f93dbfff640 1 -- 172.21.15.28:0/3399433823 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e80136b0 con 0x55629e8b06d0 2024-04-21T08:52:25.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.247+0000 7f93fbb87740 1 -- 172.21.15.28:0/3399433823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b06d0 msgr2=0x55629e8b5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.247+0000 7f93fbb87740 1 --2- 172.21.15.28:0/3399433823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b06d0 0x55629e8b5990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x55629e8b01a0 tx=0x7f93e8011000 comp rx=0 tx=0).stop 2024-04-21T08:52:25.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.248+0000 7f93fbb87740 1 -- 172.21.15.28:0/3399433823 shutdown_connections 2024-04-21T08:52:25.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.248+0000 7f93fbb87740 1 --2- 172.21.15.28:0/3399433823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b9c00 0x55629e8b8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.248+0000 7f93fbb87740 1 --2- 172.21.15.28:0/3399433823 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b1080 0x55629e8c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.248+0000 7f93fbb87740 1 --2- 172.21.15.28:0/3399433823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b06d0 0x55629e8b5990 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.248+0000 7f93fbb87740 1 -- 172.21.15.28:0/3399433823 >> 172.21.15.28:0/3399433823 conn(0x55629e8aada0 msgr2=0x55629e8c4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.248+0000 7f93fbb87740 1 -- 172.21.15.28:0/3399433823 shutdown_connections 2024-04-21T08:52:25.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.249+0000 7f93fbb87740 1 -- 172.21.15.28:0/3399433823 wait complete. 2024-04-21T08:52:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.249+0000 7f93fbb87740 1 -- start start 2024-04-21T08:52:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.250+0000 7f93fbb87740 1 Processor -- start 2024-04-21T08:52:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.251+0000 7f93fbb87740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b06d0 0x55629e8cc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.251+0000 7f93fbb87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b1080 0x55629e8cc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.251+0000 7f93f1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b1080 0x55629e8cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.251+0000 7f93f1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b1080 0x55629e8cc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56378/0 (socket says 172.21.15.28:56378) 2024-04-21T08:52:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.251+0000 7f93f1d9b640 1 -- 172.21.15.28:0/100711251 learned_addr learned my addr 172.21.15.28:0/100711251 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:25.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.252+0000 7f93fbb87740 1 --2- 172.21.15.28:0/100711251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b9c00 0x55629e8ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.252+0000 7f93fbb87740 1 -- 172.21.15.28:0/100711251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f93d400e580 con 0x55629e8b1080 2024-04-21T08:52:25.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.252+0000 7f93f259c640 1 --2- 172.21.15.28:0/100711251 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b06d0 0x55629e8cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.252+0000 7f93fbb87740 1 -- 172.21.15.28:0/100711251 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f93d400f9f0 con 0x55629e8b9c00 2024-04-21T08:52:25.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.252+0000 7f93fbb87740 1 -- 172.21.15.28:0/100711251 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f93d400ec50 con 0x55629e8b06d0 2024-04-21T08:52:25.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.252+0000 7f93fbb87740 10 client.0 _ll_register_callbacks cb 0x7f93e402ca70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:25.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.253+0000 7f93f2d9d640 1 --2- 172.21.15.28:0/100711251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b9c00 0x55629e8ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.253+0000 7f93f1d9b640 1 -- 172.21.15.28:0/100711251 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b06d0 msgr2=0x55629e8cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.253+0000 7f93f1d9b640 1 --2- 172.21.15.28:0/100711251 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b06d0 0x55629e8cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.253+0000 7f93f1d9b640 1 -- 172.21.15.28:0/100711251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b9c00 msgr2=0x55629e8ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.253+0000 7f93f1d9b640 1 --2- 172.21.15.28:0/100711251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b9c00 0x55629e8ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.253+0000 7f93f1d9b640 1 -- 172.21.15.28:0/100711251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55629e92e0f0 con 0x55629e8b1080 2024-04-21T08:52:25.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.253+0000 7f93f2d9d640 1 --2- 172.21.15.28:0/100711251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b9c00 0x55629e8ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:52:25.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.254+0000 7f93f1d9b640 1 --2- 172.21.15.28:0/100711251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b1080 0x55629e8cc580 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f93e4057040 tx=0x7f93e4004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc8ae28a324d639 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.254+0000 7f93d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.254+0000 7f93d9ffb640 1 -- 172.21.15.28:0/100711251 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e4059070 con 0x55629e8b1080 2024-04-21T08:52:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.254+0000 7f93d9ffb640 1 -- 172.21.15.28:0/100711251 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f93e4005310 con 0x55629e8b1080 2024-04-21T08:52:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.254+0000 7f93fbb87740 1 -- 172.21.15.28:0/100711251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93e802b030 con 0x55629e8b1080 2024-04-21T08:52:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.255+0000 7f93d9ffb640 1 -- 172.21.15.28:0/100711251 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f93e4004240 con 0x55629e8b1080 2024-04-21T08:52:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.255+0000 7f93fbb87740 10 client.16227 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.255+0000 7f93fbb87740 1 -- 172.21.15.28:0/100711251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55629e92f3d0 con 0x55629e8b1080 2024-04-21T08:52:25.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.255+0000 7f93d9ffb640 1 -- 172.21.15.28:0/100711251 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93e405f070 con 0x55629e8b1080 2024-04-21T08:52:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.256+0000 7f93d9ffb640 1 -- 172.21.15.28:0/100711251 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f93e4041470 con 0x55629e8b1080 2024-04-21T08:52:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.256+0000 7f93fbb87740 10 client.16227 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.256+0000 7f93fbb87740 10 client.16227 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:52:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.256+0000 7f93fbb87740 20 client.16227 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.256+0000 7f93fbb87740 1 --2- 172.21.15.28:0/100711251 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55629e8c9c50 0x55629e931a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.256+0000 7f93fbb87740 4 client.16227 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:52:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.256+0000 7f93fbb87740 1 -- 172.21.15.28:0/100711251 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55629e822c10 con 0x55629e8c9c50 2024-04-21T08:52:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.258+0000 7f93f259c640 1 --2- 172.21.15.28:0/100711251 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55629e8c9c50 0x55629e931a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.261+0000 7f93d9ffb640 10 client.16227 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:52:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.261+0000 7f93f259c640 1 --2- 172.21.15.28:0/100711251 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55629e8c9c50 0x55629e931a20 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=db24b761a277999b server_cookie=36d4a2752899d202 in_seq=0 out_seq=0 2024-04-21T08:52:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.261+0000 7f93d9ffb640 1 -- 172.21.15.28:0/100711251 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55629e822c10 con 0x55629e8c9c50 2024-04-21T08:52:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.261+0000 7f93d9ffb640 10 client.16227 handle_command_reply: tid=0 2024-04-21T08:52:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.261+0000 7f93fbb87740 1 client.16227 shutdown 2024-04-21T08:52:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.261+0000 7f93fbb87740 1 -- 172.21.15.28:0/100711251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b1080 msgr2=0x55629e8cc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.261+0000 7f93fbb87740 1 --2- 172.21.15.28:0/100711251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b1080 0x55629e8cc580 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f93e4057040 tx=0x7f93e4004610 comp rx=0 tx=0).stop 2024-04-21T08:52:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.262+0000 7f93fbb87740 1 -- 172.21.15.28:0/100711251 shutdown_connections 2024-04-21T08:52:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.262+0000 7f93fbb87740 1 --2- 172.21.15.28:0/100711251 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55629e8c9c50 0x55629e931a20 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.262+0000 7f93fbb87740 1 --2- 172.21.15.28:0/100711251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55629e8b9c00 0x55629e8ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.262+0000 7f93fbb87740 1 --2- 172.21.15.28:0/100711251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55629e8b1080 0x55629e8cc580 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.262+0000 7f93fbb87740 1 --2- 172.21.15.28:0/100711251 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55629e8b06d0 0x55629e8cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.262+0000 7f93fbb87740 1 -- 172.21.15.28:0/100711251 >> 172.21.15.28:0/100711251 conn(0x55629e8aada0 msgr2=0x55629e8a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.262+0000 7f93d9ffb640 0 client.16227 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:52:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.262+0000 7f93fbb87740 1 -- 172.21.15.28:0/100711251 shutdown_connections 2024-04-21T08:52:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.262+0000 7f93fbb87740 1 -- 172.21.15.28:0/100711251 wait complete. 2024-04-21T08:52:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.262+0000 7f93fbb87740 20 client.16227 trim_cache size 0 max 16384 2024-04-21T08:52:25.263 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:52:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.263+0000 7f93f359e640 1 -- 172.21.15.28:0/2637165912 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f93dc0a3ee0 msgr2=0x7f93dc0a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.263+0000 7f93f359e640 1 --2- 172.21.15.28:0/2637165912 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f93dc0a3ee0 0x7f93dc0a63a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x55629e8abe10 tx=0x7f93e80090e0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.263+0000 7f93f359e640 1 -- 172.21.15.28:0/2637165912 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93f4117080 msgr2=0x7f93f41a7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.263+0000 7f93f359e640 1 --2- 172.21.15.28:0/2637165912 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93f4117080 0x7f93f41a7a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00cad0 tx=0x7f93ec00cf90 comp rx=0 tx=0).stop 2024-04-21T08:52:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.264+0000 7f93f359e640 1 -- 172.21.15.28:0/2637165912 shutdown_connections 2024-04-21T08:52:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.264+0000 7f93f359e640 1 --2- 172.21.15.28:0/2637165912 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f93dc0a3ee0 0x7f93dc0a63a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.264+0000 7f93f359e640 1 --2- 172.21.15.28:0/2637165912 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93f4117080 0x7f93f41a7a20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.264+0000 7f93f359e640 1 --2- 172.21.15.28:0/2637165912 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f93f410c6c0 0x7f93f41a36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.264+0000 7f93f359e640 1 --2- 172.21.15.28:0/2637165912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93f4109520 0x7f93f41a3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.264+0000 7f93f359e640 1 -- 172.21.15.28:0/2637165912 >> 172.21.15.28:0/2637165912 conn(0x7f93f406d330 msgr2=0x7f93f40728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.264+0000 7f93f359e640 1 -- 172.21.15.28:0/2637165912 shutdown_connections 2024-04-21T08:52:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.264+0000 7f93f359e640 1 -- 172.21.15.28:0/2637165912 wait complete. 2024-04-21T08:52:25.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:52:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.486+0000 7ff054a5c640 1 -- 172.21.15.28:0/2930928608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff050101a40 msgr2=0x7ff050105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.486+0000 7ff054a5c640 1 --2- 172.21.15.28:0/2930928608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff050101a40 0x7ff050105a80 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ff038009900 tx=0x7ff03802bc30 comp rx=0 tx=0).stop 2024-04-21T08:52:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.486+0000 7ff054a5c640 1 -- 172.21.15.28:0/2930928608 shutdown_connections 2024-04-21T08:52:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.486+0000 7ff054a5c640 1 --2- 172.21.15.28:0/2930928608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff050106160 0x7ff05010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.486+0000 7ff054a5c640 1 --2- 172.21.15.28:0/2930928608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff050101a40 0x7ff050105a80 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.486+0000 7ff054a5c640 1 --2- 172.21.15.28:0/2930928608 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff050101090 0x7ff050101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.486+0000 7ff054a5c640 1 -- 172.21.15.28:0/2930928608 >> 172.21.15.28:0/2930928608 conn(0x7ff050077f50 msgr2=0x7ff050078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.486+0000 7ff054a5c640 1 -- 172.21.15.28:0/2930928608 shutdown_connections 2024-04-21T08:52:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.486+0000 7ff054a5c640 1 -- 172.21.15.28:0/2930928608 wait complete. 2024-04-21T08:52:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.486+0000 7ff054a5c640 1 Processor -- start 2024-04-21T08:52:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff054a5c640 1 -- start start 2024-04-21T08:52:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff054a5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff050101090 0x7ff050102b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff054a5c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff050106160 0x7ff050103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff054a5c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0501035a0 0x7ff0501a6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff054a5c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff050111080 con 0x7ff050101090 2024-04-21T08:52:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff054a5c640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff050110f00 con 0x7ff0501035a0 2024-04-21T08:52:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff054a5c640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff050111200 con 0x7ff050106160 2024-04-21T08:52:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff04e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff050101090 0x7ff050102b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff04e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff050101090 0x7ff050102b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56388/0 (socket says 172.21.15.28:56388) 2024-04-21T08:52:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff04e59c640 1 -- 172.21.15.28:0/3435986152 learned_addr learned my addr 172.21.15.28:0/3435986152 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff047fff640 1 --2- 172.21.15.28:0/3435986152 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff050106160 0x7ff050103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff04ed9d640 1 --2- 172.21.15.28:0/3435986152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0501035a0 0x7ff0501a6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff04e59c640 1 -- 172.21.15.28:0/3435986152 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff050106160 msgr2=0x7ff050103060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff04e59c640 1 --2- 172.21.15.28:0/3435986152 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff050106160 0x7ff050103060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff04e59c640 1 -- 172.21.15.28:0/3435986152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0501035a0 msgr2=0x7ff0501a6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff04e59c640 1 --2- 172.21.15.28:0/3435986152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0501035a0 0x7ff0501a6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff04e59c640 1 -- 172.21.15.28:0/3435986152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0501a7000 con 0x7ff050101090 2024-04-21T08:52:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff04ed9d640 1 --2- 172.21.15.28:0/3435986152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0501035a0 0x7ff0501a6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:52:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.487+0000 7ff04e59c640 1 --2- 172.21.15.28:0/3435986152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff050101090 0x7ff050102b20 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff04000d950 tx=0x7ff04000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.488+0000 7ff0477fe640 1 -- 172.21.15.28:0/3435986152 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff040017070 con 0x7ff050101090 2024-04-21T08:52:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.488+0000 7ff054a5c640 1 -- 172.21.15.28:0/3435986152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0501a72f0 con 0x7ff050101090 2024-04-21T08:52:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.488+0000 7ff0477fe640 1 -- 172.21.15.28:0/3435986152 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff040009c30 con 0x7ff050101090 2024-04-21T08:52:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.488+0000 7ff0477fe640 1 -- 172.21.15.28:0/3435986152 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04000f620 con 0x7ff050101090 2024-04-21T08:52:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.488+0000 7ff054a5c640 1 -- 172.21.15.28:0/3435986152 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0501a7810 con 0x7ff050101090 2024-04-21T08:52:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.491+0000 7ff0477fe640 1 -- 172.21.15.28:0/3435986152 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7ff040010460 con 0x7ff050101090 2024-04-21T08:52:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.491+0000 7ff057655740 1 Processor -- start 2024-04-21T08:52:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.491+0000 7ff057655740 1 -- start start 2024-04-21T08:52:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.491+0000 7ff057655740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0eece0 0x564acc0ef0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.491+0000 7ff057655740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0ef690 0x564acc0f7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.491+0000 7ff057655740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0f8210 0x564acc0fa660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.491+0000 7ff057655740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564acc04b1f0 con 0x564acc0ef690 2024-04-21T08:52:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.491+0000 7ff057655740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564acbfe84b0 con 0x564acc0eece0 2024-04-21T08:52:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.491+0000 7ff057655740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564acc0471b0 con 0x564acc0f8210 2024-04-21T08:52:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.492+0000 7ff04ed9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0eece0 0x564acc0ef0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.492+0000 7ff04ed9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0eece0 0x564acc0ef0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38272/0 (socket says 172.21.15.28:38272) 2024-04-21T08:52:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.492+0000 7ff04ed9d640 1 -- 172.21.15.28:0/3269203278 learned_addr learned my addr 172.21.15.28:0/3269203278 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.492+0000 7ff047fff640 1 --2- 172.21.15.28:0/3269203278 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0f8210 0x564acc0fa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.492+0000 7ff04e59c640 1 --2- 172.21.15.28:0/3269203278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0ef690 0x564acc0f7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.492+0000 7ff04ed9d640 1 -- 172.21.15.28:0/3269203278 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0f8210 msgr2=0x564acc0fa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.492+0000 7ff04ed9d640 1 --2- 172.21.15.28:0/3269203278 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0f8210 0x564acc0fa660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.492+0000 7ff04ed9d640 1 -- 172.21.15.28:0/3269203278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0ef690 msgr2=0x564acc0f7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.492+0000 7ff04ed9d640 1 --2- 172.21.15.28:0/3269203278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0ef690 0x564acc0f7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.492+0000 7ff0477fe640 1 --2- 172.21.15.28:0/3435986152 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff03c0a3f00 0x7ff03c0a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.492+0000 7ff04ed9d640 1 -- 172.21.15.28:0/3269203278 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564acbfda1b0 con 0x564acc0eece0 2024-04-21T08:52:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.492+0000 7ff047fff640 1 --2- 172.21.15.28:0/3269203278 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0f8210 0x564acc0fa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.492+0000 7ff047fff640 1 --2- 172.21.15.28:0/3435986152 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff03c0a3f00 0x7ff03c0a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.492+0000 7ff0477fe640 1 -- 172.21.15.28:0/3435986152 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ff0400d0a90 con 0x7ff050101090 2024-04-21T08:52:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.492+0000 7ff04ed9d640 1 --2- 172.21.15.28:0/3269203278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0eece0 0x564acc0ef0c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff048009f30 tx=0x7ff048010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=604520ecb3d634db server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.493+0000 7ff044ff9640 1 -- 172.21.15.28:0/3269203278 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff048016070 con 0x564acc0eece0 2024-04-21T08:52:25.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.493+0000 7ff044ff9640 1 -- 172.21.15.28:0/3269203278 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff04801bb10 con 0x564acc0eece0 2024-04-21T08:52:25.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.493+0000 7ff044ff9640 1 -- 172.21.15.28:0/3269203278 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff048012c10 con 0x564acc0eece0 2024-04-21T08:52:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.494+0000 7ff057655740 1 -- 172.21.15.28:0/3269203278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0eece0 msgr2=0x564acc0ef0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.494+0000 7ff047fff640 1 --2- 172.21.15.28:0/3435986152 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff03c0a3f00 0x7ff03c0a63c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff038009230 tx=0x7ff038005fd0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.494+0000 7ff057655740 1 --2- 172.21.15.28:0/3269203278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0eece0 0x564acc0ef0c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff048009f30 tx=0x7ff048010800 comp rx=0 tx=0).stop 2024-04-21T08:52:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.494+0000 7ff057655740 1 -- 172.21.15.28:0/3269203278 shutdown_connections 2024-04-21T08:52:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.494+0000 7ff057655740 1 --2- 172.21.15.28:0/3269203278 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0f8210 0x564acc0fa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.494+0000 7ff057655740 1 --2- 172.21.15.28:0/3269203278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0ef690 0x564acc0f7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.494+0000 7ff057655740 1 --2- 172.21.15.28:0/3269203278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0eece0 0x564acc0ef0c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.494+0000 7ff057655740 1 -- 172.21.15.28:0/3269203278 >> 172.21.15.28:0/3269203278 conn(0x564acc0e93b0 msgr2=0x564acc0e9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.494+0000 7ff057655740 1 -- 172.21.15.28:0/3269203278 shutdown_connections 2024-04-21T08:52:25.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.494+0000 7ff057655740 1 -- 172.21.15.28:0/3269203278 wait complete. 2024-04-21T08:52:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.494+0000 7ff057655740 1 -- start start 2024-04-21T08:52:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.495+0000 7ff057655740 1 Processor -- start 2024-04-21T08:52:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.495+0000 7ff057655740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0eece0 0x564acc0f7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.495+0000 7ff057655740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0ef690 0x564acc0f4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.495+0000 7ff057655740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0f8210 0x564acc0f5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.495+0000 7ff057655740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564acc0fb180 con 0x564acc0f8210 2024-04-21T08:52:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.495+0000 7ff057655740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564acc004500 con 0x564acc0eece0 2024-04-21T08:52:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.495+0000 7ff057655740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564acc0fb300 con 0x564acc0ef690 2024-04-21T08:52:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.495+0000 7ff057655740 10 client.0 _ll_register_callbacks cb 0x564acbfd2b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.495+0000 7ff04e59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0eece0 0x564acc0f7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.495+0000 7ff04e59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0eece0 0x564acc0f7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38284/0 (socket says 172.21.15.28:38284) 2024-04-21T08:52:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.495+0000 7ff04e59c640 1 -- 172.21.15.28:0/3517998185 learned_addr learned my addr 172.21.15.28:0/3517998185 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.495+0000 7ff04ed9d640 1 --2- 172.21.15.28:0/3517998185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0f8210 0x564acc0f5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.495+0000 7ff047fff640 1 --2- 172.21.15.28:0/3517998185 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0ef690 0x564acc0f4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.495+0000 7ff04e59c640 1 -- 172.21.15.28:0/3517998185 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0ef690 msgr2=0x564acc0f4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.495+0000 7ff04e59c640 1 --2- 172.21.15.28:0/3517998185 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0ef690 0x564acc0f4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.496+0000 7ff04e59c640 1 -- 172.21.15.28:0/3517998185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0f8210 msgr2=0x564acc0f5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.496+0000 7ff04e59c640 1 --2- 172.21.15.28:0/3517998185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0f8210 0x564acc0f5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.496+0000 7ff04e59c640 1 -- 172.21.15.28:0/3517998185 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564acbd5d3f0 con 0x564acc0eece0 2024-04-21T08:52:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.496+0000 7ff04e59c640 1 --2- 172.21.15.28:0/3517998185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0eece0 0x564acc0f7b10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff04009c150 tx=0x7ff040005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3154f6ad3ccacc8e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.496+0000 7ff035ffb640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:52:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.496+0000 7ff035ffb640 1 -- 172.21.15.28:0/3517998185 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff040020070 con 0x564acc0eece0 2024-04-21T08:52:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.496+0000 7ff035ffb640 1 -- 172.21.15.28:0/3517998185 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ff04009b8a0 con 0x564acc0eece0 2024-04-21T08:52:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.496+0000 7ff057655740 1 -- 172.21.15.28:0/3517998185 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff04801bb10 con 0x564acc0eece0 2024-04-21T08:52:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.496+0000 7ff035ffb640 1 -- 172.21.15.28:0/3517998185 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff04009bf90 con 0x564acc0eece0 2024-04-21T08:52:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.496+0000 7ff057655740 10 client.25664 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.496+0000 7ff057655740 1 -- 172.21.15.28:0/3517998185 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564acbe277d0 con 0x564acc0eece0 2024-04-21T08:52:25.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.496+0000 7ff035ffb640 1 -- 172.21.15.28:0/3517998185 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff04801bb10 con 0x564acc0eece0 2024-04-21T08:52:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.497+0000 7ff035ffb640 1 -- 172.21.15.28:0/3517998185 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7ff040010750 con 0x564acc0eece0 2024-04-21T08:52:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.497+0000 7ff057655740 10 client.25664 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.497+0000 7ff057655740 10 client.25664 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:52:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.497+0000 7ff057655740 20 client.25664 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.497+0000 7ff057655740 1 --2- 172.21.15.28:0/3517998185 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x564acc1082e0 0x564acc10a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.497+0000 7ff057655740 4 client.25664 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.497+0000 7ff057655740 1 -- 172.21.15.28:0/3517998185 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564acbfea930 con 0x564acc1082e0 2024-04-21T08:52:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.497+0000 7ff047fff640 1 --2- 172.21.15.28:0/3517998185 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x564acc1082e0 0x564acc10a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.498+0000 7ff047fff640 1 --2- 172.21.15.28:0/3517998185 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x564acc1082e0 0x564acc10a710 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1792247098e473d9 server_cookie=1a72ee0a75088b51 in_seq=0 out_seq=0 2024-04-21T08:52:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.498+0000 7ff035ffb640 10 client.25664 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:52:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.499+0000 7ff035ffb640 1 -- 172.21.15.28:0/3517998185 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564acbfea930 con 0x564acc1082e0 2024-04-21T08:52:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.499+0000 7ff035ffb640 10 client.25664 handle_command_reply: tid=0 2024-04-21T08:52:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.499+0000 7ff057655740 1 client.25664 shutdown 2024-04-21T08:52:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.499+0000 7ff057655740 1 -- 172.21.15.28:0/3517998185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0eece0 msgr2=0x564acc0f7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.499+0000 7ff057655740 1 --2- 172.21.15.28:0/3517998185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0eece0 0x564acc0f7b10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff04009c150 tx=0x7ff040005bd0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.499+0000 7ff057655740 1 -- 172.21.15.28:0/3517998185 shutdown_connections 2024-04-21T08:52:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.499+0000 7ff057655740 1 --2- 172.21.15.28:0/3517998185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0f8210 0x564acc0f5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.499+0000 7ff057655740 1 --2- 172.21.15.28:0/3517998185 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x564acc1082e0 0x564acc10a710 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.499+0000 7ff057655740 1 --2- 172.21.15.28:0/3517998185 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0ef690 0x564acc0f4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.499+0000 7ff035ffb640 0 client.25664 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:52:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.499+0000 7ff057655740 1 --2- 172.21.15.28:0/3517998185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0eece0 0x564acc0f7b10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.499+0000 7ff057655740 1 -- 172.21.15.28:0/3517998185 >> 172.21.15.28:0/3517998185 conn(0x564acc0e93b0 msgr2=0x564acc0eb100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.500+0000 7ff057655740 1 -- 172.21.15.28:0/3517998185 shutdown_connections 2024-04-21T08:52:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.500+0000 7ff057655740 1 -- 172.21.15.28:0/3517998185 wait complete. 2024-04-21T08:52:25.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.500+0000 7ff057655740 20 client.25664 trim_cache size 0 max 16384 2024-04-21T08:52:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.530+0000 7ff057655740 1 Processor -- start 2024-04-21T08:52:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.530+0000 7ff057655740 1 -- start start 2024-04-21T08:52:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.531+0000 7ff057655740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0eece0 0x564acc0dfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.531+0000 7ff04e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0eece0 0x564acc0dfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.531+0000 7ff04e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0eece0 0x564acc0dfc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56420/0 (socket says 172.21.15.28:56420) 2024-04-21T08:52:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.532+0000 7ff057655740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0ef690 0x564acc10ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.532+0000 7ff04e59c640 1 -- 172.21.15.28:0/3226409270 learned_addr learned my addr 172.21.15.28:0/3226409270 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.532+0000 7ff057655740 1 --2- 172.21.15.28:0/3226409270 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0f8210 0x564acc110520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.532+0000 7ff057655740 1 -- 172.21.15.28:0/3226409270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501af990 con 0x564acc0eece0 2024-04-21T08:52:25.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.532+0000 7ff057655740 1 -- 172.21.15.28:0/3226409270 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501ad260 con 0x564acc0ef690 2024-04-21T08:52:25.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.532+0000 7ff057655740 1 -- 172.21.15.28:0/3226409270 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501ab0b0 con 0x564acc0f8210 2024-04-21T08:52:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.533+0000 7ff04e59c640 1 -- 172.21.15.28:0/3226409270 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0f8210 msgr2=0x564acc110520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.533+0000 7ff04e59c640 1 --2- 172.21.15.28:0/3226409270 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0f8210 0x564acc110520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.533+0000 7ff04e59c640 1 -- 172.21.15.28:0/3226409270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0ef690 msgr2=0x564acc10ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.533+0000 7ff04e59c640 1 --2- 172.21.15.28:0/3226409270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0ef690 0x564acc10ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.533+0000 7ff04e59c640 1 -- 172.21.15.28:0/3226409270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564acc0f0a80 con 0x564acc0eece0 2024-04-21T08:52:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.534+0000 7ff04e59c640 1 --2- 172.21.15.28:0/3226409270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0eece0 0x564acc0dfc00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff040005c40 tx=0x7ff0400d45f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98ebf34ef1ed3f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.534+0000 7ff037fff640 1 -- 172.21.15.28:0/3226409270 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff040020070 con 0x564acc0eece0 2024-04-21T08:52:25.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.535+0000 7ff037fff640 1 -- 172.21.15.28:0/3226409270 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff040015070 con 0x564acc0eece0 2024-04-21T08:52:25.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.535+0000 7ff037fff640 1 -- 172.21.15.28:0/3226409270 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04009bdb0 con 0x564acc0eece0 2024-04-21T08:52:25.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.536+0000 7ff057655740 1 -- 172.21.15.28:0/3226409270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0eece0 msgr2=0x564acc0dfc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.536+0000 7ff057655740 1 --2- 172.21.15.28:0/3226409270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0eece0 0x564acc0dfc00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff040005c40 tx=0x7ff0400d45f0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.540+0000 7ff057655740 1 -- 172.21.15.28:0/3226409270 shutdown_connections 2024-04-21T08:52:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.540+0000 7ff057655740 1 --2- 172.21.15.28:0/3226409270 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0f8210 0x564acc110520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.540+0000 7ff057655740 1 --2- 172.21.15.28:0/3226409270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0ef690 0x564acc10ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.540+0000 7ff057655740 1 --2- 172.21.15.28:0/3226409270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0eece0 0x564acc0dfc00 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.540+0000 7ff057655740 1 -- 172.21.15.28:0/3226409270 >> 172.21.15.28:0/3226409270 conn(0x564acc0e93b0 msgr2=0x564acc0ea6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.540+0000 7ff057655740 1 -- 172.21.15.28:0/3226409270 shutdown_connections 2024-04-21T08:52:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.540+0000 7ff057655740 1 -- 172.21.15.28:0/3226409270 wait complete. 2024-04-21T08:52:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.541+0000 7ff057655740 1 -- start start 2024-04-21T08:52:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.542+0000 7ff057655740 1 Processor -- start 2024-04-21T08:52:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.543+0000 7ff057655740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0eece0 0x564acc0fc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.543+0000 7ff057655740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0ef690 0x564acc0fc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.544+0000 7ff057655740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0f8210 0x564acc0fcbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.544+0000 7ff057655740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501ab720 con 0x564acc0ef690 2024-04-21T08:52:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.544+0000 7ff057655740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501acb90 con 0x564acc0f8210 2024-04-21T08:52:25.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.544+0000 7ff057655740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff0501abdf0 con 0x564acc0eece0 2024-04-21T08:52:25.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.544+0000 7ff057655740 10 client.0 _ll_register_callbacks cb 0x7ff0400980b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:25.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.545+0000 7ff04e59c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0eece0 0x564acc0fc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.545+0000 7ff047fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0ef690 0x564acc0fc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.545+0000 7ff047fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0ef690 0x564acc0fc690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56434/0 (socket says 172.21.15.28:56434) 2024-04-21T08:52:25.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.545+0000 7ff047fff640 1 -- 172.21.15.28:0/1936400615 learned_addr learned my addr 172.21.15.28:0/1936400615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:25.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.545+0000 7ff04ed9d640 1 --2- 172.21.15.28:0/1936400615 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0f8210 0x564acc0fcbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.547+0000 7ff047fff640 1 -- 172.21.15.28:0/1936400615 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0eece0 msgr2=0x564acc0fc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.547+0000 7ff047fff640 1 --2- 172.21.15.28:0/1936400615 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0eece0 0x564acc0fc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.547+0000 7ff047fff640 1 -- 172.21.15.28:0/1936400615 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0f8210 msgr2=0x564acc0fcbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.547+0000 7ff047fff640 1 --2- 172.21.15.28:0/1936400615 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0f8210 0x564acc0fcbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.548+0000 7ff047fff640 1 -- 172.21.15.28:0/1936400615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564acc17ea30 con 0x564acc0ef690 2024-04-21T08:52:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.549+0000 7ff047fff640 1 --2- 172.21.15.28:0/1936400615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0ef690 0x564acc0fc690 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7ff038009900 tx=0x7ff03802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655d7ee324f686d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.549+0000 7ff035ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.549+0000 7ff035ffb640 1 -- 172.21.15.28:0/1936400615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03802cc70 con 0x564acc0ef690 2024-04-21T08:52:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.550+0000 7ff035ffb640 1 -- 172.21.15.28:0/1936400615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ff038005040 con 0x564acc0ef690 2024-04-21T08:52:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.550+0000 7ff057655740 1 -- 172.21.15.28:0/1936400615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff040015070 con 0x564acc0ef690 2024-04-21T08:52:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.552+0000 7ff035ffb640 1 -- 172.21.15.28:0/1936400615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff038005280 con 0x564acc0ef690 2024-04-21T08:52:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.552+0000 7ff057655740 10 client.16245 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.552+0000 7ff057655740 1 -- 172.21.15.28:0/1936400615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564acc17fcf0 con 0x564acc0ef690 2024-04-21T08:52:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.554+0000 7ff035ffb640 1 -- 172.21.15.28:0/1936400615 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff040015070 con 0x564acc0ef690 2024-04-21T08:52:25.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.554+0000 7ff035ffb640 1 -- 172.21.15.28:0/1936400615 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7ff038004970 con 0x564acc0ef690 2024-04-21T08:52:25.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.556+0000 7ff057655740 10 client.16245 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:25.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.556+0000 7ff057655740 10 client.16245 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:52:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.556+0000 7ff057655740 20 client.16245 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.556+0000 7ff057655740 1 --2- 172.21.15.28:0/1936400615 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x564acc1082e0 0x564acc1824b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.556+0000 7ff057655740 4 client.16245 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:52:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.556+0000 7ff057655740 1 -- 172.21.15.28:0/1936400615 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564acc058c10 con 0x564acc1082e0 2024-04-21T08:52:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.558+0000 7ff04e59c640 1 --2- 172.21.15.28:0/1936400615 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x564acc1082e0 0x564acc1824b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.560+0000 7ff035ffb640 10 client.16245 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:52:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.560+0000 7ff04e59c640 1 --2- 172.21.15.28:0/1936400615 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x564acc1082e0 0x564acc1824b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=beedf86076e5fe6e server_cookie=813451de9f06d5e4 in_seq=0 out_seq=0 2024-04-21T08:52:25.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.561+0000 7ff035ffb640 1 -- 172.21.15.28:0/1936400615 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564acc058c10 con 0x564acc1082e0 2024-04-21T08:52:25.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.561+0000 7ff035ffb640 10 client.16245 handle_command_reply: tid=0 2024-04-21T08:52:25.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.561+0000 7ff057655740 1 client.16245 shutdown 2024-04-21T08:52:25.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.562+0000 7ff057655740 1 -- 172.21.15.28:0/1936400615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0ef690 msgr2=0x564acc0fc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.562+0000 7ff057655740 1 --2- 172.21.15.28:0/1936400615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0ef690 0x564acc0fc690 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7ff038009900 tx=0x7ff03802bc30 comp rx=0 tx=0).stop 2024-04-21T08:52:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.562+0000 7ff057655740 1 -- 172.21.15.28:0/1936400615 shutdown_connections 2024-04-21T08:52:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.562+0000 7ff057655740 1 --2- 172.21.15.28:0/1936400615 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564acc0f8210 0x564acc0fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.562+0000 7ff057655740 1 --2- 172.21.15.28:0/1936400615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564acc0ef690 0x564acc0fc690 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.562+0000 7ff057655740 1 --2- 172.21.15.28:0/1936400615 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x564acc1082e0 0x564acc1824b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.562+0000 7ff057655740 1 --2- 172.21.15.28:0/1936400615 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564acc0eece0 0x564acc0fc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.562+0000 7ff035ffb640 0 client.16245 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:52:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.562+0000 7ff057655740 1 -- 172.21.15.28:0/1936400615 >> 172.21.15.28:0/1936400615 conn(0x564acc0e93b0 msgr2=0x564acc0f0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.563+0000 7ff057655740 1 -- 172.21.15.28:0/1936400615 shutdown_connections 2024-04-21T08:52:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.563+0000 7ff057655740 1 -- 172.21.15.28:0/1936400615 wait complete. 2024-04-21T08:52:25.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.563+0000 7ff057655740 20 client.16245 trim_cache size 0 max 16384 2024-04-21T08:52:25.564 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:52:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.571+0000 7ff054a5c640 1 -- 172.21.15.28:0/3435986152 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff03c0a3f00 msgr2=0x7ff03c0a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.571+0000 7ff054a5c640 1 --2- 172.21.15.28:0/3435986152 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff03c0a3f00 0x7ff03c0a63c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff038009230 tx=0x7ff038005fd0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.571+0000 7ff054a5c640 1 -- 172.21.15.28:0/3435986152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff050101090 msgr2=0x7ff050102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.572+0000 7ff054a5c640 1 --2- 172.21.15.28:0/3435986152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff050101090 0x7ff050102b20 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff04000d950 tx=0x7ff04000de10 comp rx=0 tx=0).stop 2024-04-21T08:52:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.572+0000 7ff054a5c640 1 -- 172.21.15.28:0/3435986152 shutdown_connections 2024-04-21T08:52:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.572+0000 7ff054a5c640 1 --2- 172.21.15.28:0/3435986152 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff03c0a3f00 0x7ff03c0a63c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.572+0000 7ff054a5c640 1 --2- 172.21.15.28:0/3435986152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff0501035a0 0x7ff0501a6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.572+0000 7ff054a5c640 1 --2- 172.21.15.28:0/3435986152 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff050106160 0x7ff050103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.572+0000 7ff054a5c640 1 --2- 172.21.15.28:0/3435986152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff050101090 0x7ff050102b20 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.572+0000 7ff054a5c640 1 -- 172.21.15.28:0/3435986152 >> 172.21.15.28:0/3435986152 conn(0x7ff050077f50 msgr2=0x7ff050100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.572+0000 7ff054a5c640 1 -- 172.21.15.28:0/3435986152 shutdown_connections 2024-04-21T08:52:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.572+0000 7ff054a5c640 1 -- 172.21.15.28:0/3435986152 wait complete. 2024-04-21T08:52:25.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:25 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1086036065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:52:25.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:52:25.619 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:25 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1086036065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:52:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.753+0000 7f96c6550640 1 -- 172.21.15.28:0/3117764629 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96c010ddc0 msgr2=0x7f96c010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.753+0000 7f96c6550640 1 --2- 172.21.15.28:0/3117764629 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96c010ddc0 0x7f96c010e1a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009860 tx=0x7f96b002bbb0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.753+0000 7f96c6550640 1 -- 172.21.15.28:0/3117764629 shutdown_connections 2024-04-21T08:52:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.753+0000 7f96c6550640 1 --2- 172.21.15.28:0/3117764629 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96c01105e0 0x7f96c0108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.753+0000 7f96c6550640 1 --2- 172.21.15.28:0/3117764629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96c010fc20 0x7f96c01100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.753+0000 7f96c6550640 1 --2- 172.21.15.28:0/3117764629 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96c010ddc0 0x7f96c010e1a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.753+0000 7f96c6550640 1 -- 172.21.15.28:0/3117764629 >> 172.21.15.28:0/3117764629 conn(0x7f96c006d060 msgr2=0x7f96c006d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.754+0000 7f96c6550640 1 -- 172.21.15.28:0/3117764629 shutdown_connections 2024-04-21T08:52:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.754+0000 7f96c6550640 1 -- 172.21.15.28:0/3117764629 wait complete. 2024-04-21T08:52:25.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.755+0000 7f96c6550640 1 Processor -- start 2024-04-21T08:52:25.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.755+0000 7f96c6550640 1 -- start start 2024-04-21T08:52:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.756+0000 7f96c6550640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96c010ddc0 0x7f96c019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.756+0000 7f96c6550640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96c010fc20 0x7f96c019f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.757+0000 7f96c6550640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96c01105e0 0x7f96c01a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.757+0000 7f96c6550640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96c0112720 con 0x7f96c01105e0 2024-04-21T08:52:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.757+0000 7f96c6550640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f96c01125a0 con 0x7f96c010fc20 2024-04-21T08:52:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.757+0000 7f96c6550640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f96c01128a0 con 0x7f96c010ddc0 2024-04-21T08:52:25.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.758+0000 7f96c4aed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96c01105e0 0x7f96c01a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.758+0000 7f96c4aed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96c01105e0 0x7f96c01a35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56450/0 (socket says 172.21.15.28:56450) 2024-04-21T08:52:25.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.758+0000 7f96c4aed640 1 -- 172.21.15.28:0/3676845687 learned_addr learned my addr 172.21.15.28:0/3676845687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:25.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.758+0000 7f96b7fff640 1 --2- 172.21.15.28:0/3676845687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96c010fc20 0x7f96c019f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.759+0000 7f96bffff640 1 --2- 172.21.15.28:0/3676845687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96c010ddc0 0x7f96c019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.760+0000 7f96c4aed640 1 -- 172.21.15.28:0/3676845687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96c010ddc0 msgr2=0x7f96c019ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.760+0000 7f96c4aed640 1 --2- 172.21.15.28:0/3676845687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96c010ddc0 0x7f96c019ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.760+0000 7f96c4aed640 1 -- 172.21.15.28:0/3676845687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96c010fc20 msgr2=0x7f96c019f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.760+0000 7f96c4aed640 1 --2- 172.21.15.28:0/3676845687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96c010fc20 0x7f96c019f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.760+0000 7f96c4aed640 1 -- 172.21.15.28:0/3676845687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96c01a3d20 con 0x7f96c01105e0 2024-04-21T08:52:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.761+0000 7f96c4aed640 1 --2- 172.21.15.28:0/3676845687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96c01105e0 0x7f96c01a35f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f96b800eda0 tx=0x7f96b800c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.762+0000 7f96bdffb640 1 -- 172.21.15.28:0/3676845687 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b8003b70 con 0x7f96c01105e0 2024-04-21T08:52:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.762+0000 7f96bdffb640 1 -- 172.21.15.28:0/3676845687 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96b8003d10 con 0x7f96c01105e0 2024-04-21T08:52:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.762+0000 7f96bdffb640 1 -- 172.21.15.28:0/3676845687 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b8017920 con 0x7f96c01105e0 2024-04-21T08:52:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.763+0000 7f96c6550640 1 -- 172.21.15.28:0/3676845687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96c01a4010 con 0x7f96c01105e0 2024-04-21T08:52:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.763+0000 7f96c6550640 1 -- 172.21.15.28:0/3676845687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96c01ab8a0 con 0x7f96c01105e0 2024-04-21T08:52:25.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.768+0000 7f96bdffb640 1 -- 172.21.15.28:0/3676845687 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f96b8017ac0 con 0x7f96c01105e0 2024-04-21T08:52:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.768+0000 7f96c921c740 1 Processor -- start 2024-04-21T08:52:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.768+0000 7f96c921c740 1 -- start start 2024-04-21T08:52:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.768+0000 7f96c921c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c9344d0 0x55fe3c9348b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.768+0000 7f96c921c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c934e80 0x55fe3c93d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.769+0000 7f96c921c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c93da00 0x55fe3c93fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.769+0000 7f96c921c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fe3c8991f0 con 0x55fe3c934e80 2024-04-21T08:52:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.769+0000 7f96c921c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55fe3c8364b0 con 0x55fe3c93da00 2024-04-21T08:52:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.769+0000 7f96c921c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55fe3c8951b0 con 0x55fe3c9344d0 2024-04-21T08:52:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.769+0000 7f96bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c934e80 0x55fe3c93d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.769+0000 7f96bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c934e80 0x55fe3c93d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56452/0 (socket says 172.21.15.28:56452) 2024-04-21T08:52:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.769+0000 7f96bffff640 1 -- 172.21.15.28:0/1947006407 learned_addr learned my addr 172.21.15.28:0/1947006407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.769+0000 7f96b7fff640 1 --2- 172.21.15.28:0/1947006407 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c9344d0 0x55fe3c9348b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.769+0000 7f96bffff640 1 -- 172.21.15.28:0/1947006407 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c9344d0 msgr2=0x55fe3c9348b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.769+0000 7f96bffff640 1 --2- 172.21.15.28:0/1947006407 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c9344d0 0x55fe3c9348b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.769+0000 7f96bffff640 1 -- 172.21.15.28:0/1947006407 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c93da00 msgr2=0x55fe3c93fe50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:52:25.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.769+0000 7f96bffff640 1 --2- 172.21.15.28:0/1947006407 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c93da00 0x55fe3c93fe50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.769+0000 7f96bffff640 1 -- 172.21.15.28:0/1947006407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe3c8281b0 con 0x55fe3c934e80 2024-04-21T08:52:25.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.769+0000 7f96bdffb640 1 --2- 172.21.15.28:0/3676845687 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f96ac0a8430 0x7f96ac0aa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.770+0000 7f96bdffb640 1 -- 172.21.15.28:0/3676845687 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f96b80d0cb0 con 0x7f96c01105e0 2024-04-21T08:52:25.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.770+0000 7f96b7fff640 1 --2- 172.21.15.28:0/3676845687 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f96ac0a8430 0x7f96ac0aa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.770+0000 7f96bffff640 1 --2- 172.21.15.28:0/1947006407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c934e80 0x55fe3c93d430 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f96b002c1c0 tx=0x7f96b0009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a9c1ab4d2cb16bd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.771+0000 7f96b67fc640 1 -- 172.21.15.28:0/1947006407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0039040 con 0x55fe3c934e80 2024-04-21T08:52:25.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.771+0000 7f96b67fc640 1 -- 172.21.15.28:0/1947006407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96b00346c0 con 0x55fe3c934e80 2024-04-21T08:52:25.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.771+0000 7f96b67fc640 1 -- 172.21.15.28:0/1947006407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b002c840 con 0x55fe3c934e80 2024-04-21T08:52:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.772+0000 7f96c921c740 1 -- 172.21.15.28:0/1947006407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c934e80 msgr2=0x55fe3c93d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.772+0000 7f96c921c740 1 --2- 172.21.15.28:0/1947006407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c934e80 0x55fe3c93d430 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f96b002c1c0 tx=0x7f96b0009480 comp rx=0 tx=0).stop 2024-04-21T08:52:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.773+0000 7f96b7fff640 1 --2- 172.21.15.28:0/3676845687 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f96ac0a8430 0x7f96ac0aa8f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f96c0070d90 tx=0x7f96a8005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.773+0000 7f96c921c740 1 -- 172.21.15.28:0/1947006407 shutdown_connections 2024-04-21T08:52:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.773+0000 7f96c921c740 1 --2- 172.21.15.28:0/1947006407 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c93da00 0x55fe3c93fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.773+0000 7f96c921c740 1 --2- 172.21.15.28:0/1947006407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c934e80 0x55fe3c93d430 secure :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f96b002c1c0 tx=0x7f96b0009480 comp rx=0 tx=0).stop 2024-04-21T08:52:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.773+0000 7f96c921c740 1 --2- 172.21.15.28:0/1947006407 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c9344d0 0x55fe3c9348b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.773+0000 7f96c921c740 1 -- 172.21.15.28:0/1947006407 >> 172.21.15.28:0/1947006407 conn(0x55fe3c92eba0 msgr2=0x55fe3c92ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.774+0000 7f96c921c740 1 -- 172.21.15.28:0/1947006407 shutdown_connections 2024-04-21T08:52:25.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.774+0000 7f96c921c740 1 -- 172.21.15.28:0/1947006407 wait complete. 2024-04-21T08:52:25.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.775+0000 7f96c921c740 1 -- start start 2024-04-21T08:52:25.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.775+0000 7f96c921c740 1 Processor -- start 2024-04-21T08:52:25.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.776+0000 7f96c921c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c9344d0 0x55fe3c93d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.776+0000 7f96c921c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c93da00 0x55fe3c93a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.776+0000 7f96c921c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c93ad30 0x55fe3c947e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.776+0000 7f96c921c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fe3c940910 con 0x55fe3c93ad30 2024-04-21T08:52:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.776+0000 7f96c921c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55fe3c852500 con 0x55fe3c9344d0 2024-04-21T08:52:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.776+0000 7f96c921c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55fe3c940a90 con 0x55fe3c93da00 2024-04-21T08:52:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.776+0000 7f96c921c740 10 client.0 _ll_register_callbacks cb 0x55fe3c820b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.777+0000 7f96c4aed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c93ad30 0x55fe3c947e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.777+0000 7f96c4aed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c93ad30 0x55fe3c947e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56468/0 (socket says 172.21.15.28:56468) 2024-04-21T08:52:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.777+0000 7f96c4aed640 1 -- 172.21.15.28:0/2550763348 learned_addr learned my addr 172.21.15.28:0/2550763348 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.778+0000 7f96c4aed640 1 -- 172.21.15.28:0/2550763348 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c93da00 msgr2=0x55fe3c93a690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:52:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.778+0000 7f96c4aed640 1 --2- 172.21.15.28:0/2550763348 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c93da00 0x55fe3c93a690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.778+0000 7f96c4aed640 1 -- 172.21.15.28:0/2550763348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c9344d0 msgr2=0x55fe3c93d3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:52:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.778+0000 7f96c4aed640 1 --2- 172.21.15.28:0/2550763348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c9344d0 0x55fe3c93d3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.778+0000 7f96c4aed640 1 -- 172.21.15.28:0/2550763348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe3c5ab3f0 con 0x55fe3c93ad30 2024-04-21T08:52:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.779+0000 7f96c4aed640 1 --2- 172.21.15.28:0/2550763348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c93ad30 0x55fe3c947e00 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f96b80174b0 tx=0x7f96b809c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a468d41f2adbe3cb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.779+0000 7f969f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.780+0000 7f969f7fe640 1 -- 172.21.15.28:0/2550763348 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b80a53f0 con 0x55fe3c93ad30 2024-04-21T08:52:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.780+0000 7f96c921c740 1 -- 172.21.15.28:0/2550763348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96b00346c0 con 0x55fe3c93ad30 2024-04-21T08:52:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.780+0000 7f969f7fe640 1 -- 172.21.15.28:0/2550763348 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f96b809bc80 con 0x55fe3c93ad30 2024-04-21T08:52:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.782+0000 7f969f7fe640 1 -- 172.21.15.28:0/2550763348 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f96b809be80 con 0x55fe3c93ad30 2024-04-21T08:52:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.782+0000 7f96c921c740 10 client.16257 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.782+0000 7f96c921c740 1 -- 172.21.15.28:0/2550763348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fe3c6757d0 con 0x55fe3c93ad30 2024-04-21T08:52:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.784+0000 7f969f7fe640 1 -- 172.21.15.28:0/2550763348 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96b00346c0 con 0x55fe3c93ad30 2024-04-21T08:52:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.784+0000 7f969f7fe640 1 -- 172.21.15.28:0/2550763348 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f96b800b030 con 0x55fe3c93ad30 2024-04-21T08:52:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.784+0000 7f96c921c740 10 client.16257 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.784+0000 7f96c921c740 10 client.16257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.785+0000 7f96c921c740 20 client.16257 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.785+0000 7f96c921c740 1 --2- 172.21.15.28:0/2550763348 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fe3c94c3b0 0x55fe3c94e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.785+0000 7f96c921c740 4 client.16257 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.785+0000 7f96c921c740 1 -- 172.21.15.28:0/2550763348 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe3c838930 con 0x55fe3c94c3b0 2024-04-21T08:52:25.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.794+0000 7f96bffff640 1 --2- 172.21.15.28:0/2550763348 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fe3c94c3b0 0x55fe3c94e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.813+0000 7f96bffff640 1 --2- 172.21.15.28:0/2550763348 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fe3c94c3b0 0x55fe3c94e870 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14a81aa17e38ae06 server_cookie=10103f5b751e7514 in_seq=0 out_seq=0 2024-04-21T08:52:25.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.814+0000 7f969f7fe640 10 client.16257 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:25.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.819+0000 7f969f7fe640 1 -- 172.21.15.28:0/2550763348 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fe3c838930 con 0x55fe3c94c3b0 2024-04-21T08:52:25.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.819+0000 7f969f7fe640 10 client.16257 handle_command_reply: tid=0 2024-04-21T08:52:25.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.819+0000 7f96c921c740 1 client.16257 shutdown 2024-04-21T08:52:25.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.820+0000 7f96c921c740 1 -- 172.21.15.28:0/2550763348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c93ad30 msgr2=0x55fe3c947e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.820+0000 7f96c921c740 1 --2- 172.21.15.28:0/2550763348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c93ad30 0x55fe3c947e00 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f96b80174b0 tx=0x7f96b809c830 comp rx=0 tx=0).stop 2024-04-21T08:52:25.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.820+0000 7f96c921c740 1 -- 172.21.15.28:0/2550763348 shutdown_connections 2024-04-21T08:52:25.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.820+0000 7f96c921c740 1 --2- 172.21.15.28:0/2550763348 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fe3c94c3b0 0x55fe3c94e870 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.820+0000 7f96c921c740 1 --2- 172.21.15.28:0/2550763348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c93ad30 0x55fe3c947e00 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.820+0000 7f96c921c740 1 --2- 172.21.15.28:0/2550763348 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c93da00 0x55fe3c93a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.820+0000 7f96c921c740 1 --2- 172.21.15.28:0/2550763348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c9344d0 0x55fe3c93d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.820+0000 7f96c921c740 1 -- 172.21.15.28:0/2550763348 >> 172.21.15.28:0/2550763348 conn(0x55fe3c92eba0 msgr2=0x55fe3c9308f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.820+0000 7f969f7fe640 0 client.16257 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.821+0000 7f96c921c740 1 -- 172.21.15.28:0/2550763348 shutdown_connections 2024-04-21T08:52:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.821+0000 7f96c921c740 1 -- 172.21.15.28:0/2550763348 wait complete. 2024-04-21T08:52:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.822+0000 7f96c921c740 20 client.16257 trim_cache size 0 max 16384 2024-04-21T08:52:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.853+0000 7f96c921c740 1 Processor -- start 2024-04-21T08:52:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.854+0000 7f96c921c740 1 -- start start 2024-04-21T08:52:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.854+0000 7f96c921c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c9344d0 0x55fe3c92da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.854+0000 7f96c921c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c93ad30 0x55fe3c947d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.854+0000 7f96c921c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c93da00 0x55fe3c93ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.854+0000 7f96c921c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96a00128d0 con 0x55fe3c9344d0 2024-04-21T08:52:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.854+0000 7f96c921c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f96a00101a0 con 0x55fe3c93ad30 2024-04-21T08:52:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.854+0000 7f96c921c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f96a000dff0 con 0x55fe3c93da00 2024-04-21T08:52:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.854+0000 7f96bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c9344d0 0x55fe3c92da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.854+0000 7f96bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c9344d0 0x55fe3c92da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56470/0 (socket says 172.21.15.28:56470) 2024-04-21T08:52:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.854+0000 7f96bffff640 1 -- 172.21.15.28:0/2404289222 learned_addr learned my addr 172.21.15.28:0/2404289222 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.855+0000 7f96bffff640 1 -- 172.21.15.28:0/2404289222 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c93da00 msgr2=0x55fe3c93ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:52:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.855+0000 7f96bffff640 1 --2- 172.21.15.28:0/2404289222 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c93da00 0x55fe3c93ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.855+0000 7f96bffff640 1 -- 172.21.15.28:0/2404289222 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c93ad30 msgr2=0x55fe3c947d70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:52:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.855+0000 7f96bffff640 1 --2- 172.21.15.28:0/2404289222 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c93ad30 0x55fe3c947d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.855+0000 7f96bffff640 1 -- 172.21.15.28:0/2404289222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe3c92f930 con 0x55fe3c9344d0 2024-04-21T08:52:25.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.855+0000 7f96bffff640 1 --2- 172.21.15.28:0/2404289222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c9344d0 0x55fe3c92da80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f96b0005b60 tx=0x7f96b0009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6654a390642ba8d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:25 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1086036065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:52:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.855+0000 7f96b5ffb640 1 -- 172.21.15.28:0/2404289222 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0039040 con 0x55fe3c9344d0 2024-04-21T08:52:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.855+0000 7f96b5ffb640 1 -- 172.21.15.28:0/2404289222 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96b0051dd0 con 0x55fe3c9344d0 2024-04-21T08:52:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.855+0000 7f96b5ffb640 1 -- 172.21.15.28:0/2404289222 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0051620 con 0x55fe3c9344d0 2024-04-21T08:52:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.856+0000 7f96c921c740 1 -- 172.21.15.28:0/2404289222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c9344d0 msgr2=0x55fe3c92da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.856+0000 7f96c921c740 1 --2- 172.21.15.28:0/2404289222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c9344d0 0x55fe3c92da80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f96b0005b60 tx=0x7f96b0009480 comp rx=0 tx=0).stop 2024-04-21T08:52:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.857+0000 7f96c921c740 1 -- 172.21.15.28:0/2404289222 shutdown_connections 2024-04-21T08:52:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.857+0000 7f96c921c740 1 --2- 172.21.15.28:0/2404289222 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c93da00 0x55fe3c93ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.857+0000 7f96c921c740 1 --2- 172.21.15.28:0/2404289222 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c93ad30 0x55fe3c947d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.857+0000 7f96c921c740 1 --2- 172.21.15.28:0/2404289222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c9344d0 0x55fe3c92da80 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.857+0000 7f96c921c740 1 -- 172.21.15.28:0/2404289222 >> 172.21.15.28:0/2404289222 conn(0x55fe3c92eba0 msgr2=0x55fe3c92d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.857+0000 7f96c921c740 1 -- 172.21.15.28:0/2404289222 shutdown_connections 2024-04-21T08:52:25.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.857+0000 7f96c921c740 1 -- 172.21.15.28:0/2404289222 wait complete. 2024-04-21T08:52:25.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.857+0000 7f96c921c740 1 -- start start 2024-04-21T08:52:25.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.858+0000 7f96c921c740 1 Processor -- start 2024-04-21T08:52:25.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.858+0000 7f96c921c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c9344d0 0x55fe3c943440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.858+0000 7f96c921c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c93ad30 0x55fe3c943980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.859+0000 7f96c921c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c93da00 0x55fe3c943ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.859+0000 7f96c921c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96a000e660 con 0x55fe3c93da00 2024-04-21T08:52:25.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.859+0000 7f96c921c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f96a000fad0 con 0x55fe3c93ad30 2024-04-21T08:52:25.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.859+0000 7f96c921c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f96a000ed30 con 0x55fe3c9344d0 2024-04-21T08:52:25.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.859+0000 7f96c4aed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c93da00 0x55fe3c943ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.859+0000 7f96c4aed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c93da00 0x55fe3c943ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56480/0 (socket says 172.21.15.28:56480) 2024-04-21T08:52:25.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.859+0000 7f96c4aed640 1 -- 172.21.15.28:0/1419231712 learned_addr learned my addr 172.21.15.28:0/1419231712 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:25.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.859+0000 7f96c921c740 10 client.0 _ll_register_callbacks cb 0x7f96b809c370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:25.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.859+0000 7f96bffff640 1 --2- 172.21.15.28:0/1419231712 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c9344d0 0x55fe3c943440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.859+0000 7f96b7fff640 1 --2- 172.21.15.28:0/1419231712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c93ad30 0x55fe3c943980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.859+0000 7f96c4aed640 1 -- 172.21.15.28:0/1419231712 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c9344d0 msgr2=0x55fe3c943440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.859+0000 7f96c4aed640 1 --2- 172.21.15.28:0/1419231712 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c9344d0 0x55fe3c943440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.859+0000 7f96c4aed640 1 -- 172.21.15.28:0/1419231712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c93ad30 msgr2=0x55fe3c943980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:25.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.859+0000 7f96c4aed640 1 --2- 172.21.15.28:0/1419231712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c93ad30 0x55fe3c943980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.859+0000 7f96c4aed640 1 -- 172.21.15.28:0/1419231712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe3c9d5760 con 0x55fe3c93da00 2024-04-21T08:52:25.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.860+0000 7f96c4aed640 1 --2- 172.21.15.28:0/1419231712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c93da00 0x55fe3c943ec0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f96b800ed70 tx=0x7f96b809c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75d45a61b9dd371 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:25.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.860+0000 7f969f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:25.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.861+0000 7f969f7fe640 1 -- 172.21.15.28:0/1419231712 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b809c600 con 0x55fe3c93da00 2024-04-21T08:52:25.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.861+0000 7f969f7fe640 1 -- 172.21.15.28:0/1419231712 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f96b80ce130 con 0x55fe3c93da00 2024-04-21T08:52:25.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.861+0000 7f96c921c740 1 -- 172.21.15.28:0/1419231712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96b0051dd0 con 0x55fe3c93da00 2024-04-21T08:52:25.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.862+0000 7f969f7fe640 1 -- 172.21.15.28:0/1419231712 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f96b80a53f0 con 0x55fe3c93da00 2024-04-21T08:52:25.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.862+0000 7f96c921c740 10 client.16263 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:25.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.862+0000 7f96c921c740 1 -- 172.21.15.28:0/1419231712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fe3c9d69e0 con 0x55fe3c93da00 2024-04-21T08:52:25.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.863+0000 7f969f7fe640 1 -- 172.21.15.28:0/1419231712 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96b0051dd0 con 0x55fe3c93da00 2024-04-21T08:52:25.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.863+0000 7f969f7fe640 1 -- 172.21.15.28:0/1419231712 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f96b8004a40 con 0x55fe3c93da00 2024-04-21T08:52:25.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.863+0000 7f96c921c740 10 client.16263 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:25.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.863+0000 7f96c921c740 10 client.16263 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:25.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.863+0000 7f96c921c740 20 client.16263 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:25.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.863+0000 7f96c921c740 1 --2- 172.21.15.28:0/1419231712 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fe3c9d6f40 0x55fe3c9d9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:25.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.863+0000 7f96c921c740 4 client.16263 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:52:25.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.863+0000 7f96c921c740 1 -- 172.21.15.28:0/1419231712 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fe3c8a6c10 con 0x55fe3c9d6f40 2024-04-21T08:52:25.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.865+0000 7f96bffff640 1 --2- 172.21.15.28:0/1419231712 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fe3c9d6f40 0x55fe3c9d9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:25.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.869+0000 7f96bffff640 1 --2- 172.21.15.28:0/1419231712 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fe3c9d6f40 0x55fe3c9d9360 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2777ffe5d0a3149f server_cookie=1f3ec16ba51e8eec in_seq=0 out_seq=0 2024-04-21T08:52:25.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.869+0000 7f969f7fe640 10 client.16263 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:25.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.870+0000 7f969f7fe640 1 -- 172.21.15.28:0/1419231712 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fe3c8a6c10 con 0x55fe3c9d6f40 2024-04-21T08:52:25.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.870+0000 7f969f7fe640 10 client.16263 handle_command_reply: tid=0 2024-04-21T08:52:25.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.870+0000 7f96c921c740 1 client.16263 shutdown 2024-04-21T08:52:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.871+0000 7f96c921c740 1 -- 172.21.15.28:0/1419231712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c93da00 msgr2=0x55fe3c943ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.871+0000 7f96c921c740 1 --2- 172.21.15.28:0/1419231712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c93da00 0x55fe3c943ec0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f96b800ed70 tx=0x7f96b809c420 comp rx=0 tx=0).stop 2024-04-21T08:52:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.871+0000 7f96c921c740 1 -- 172.21.15.28:0/1419231712 shutdown_connections 2024-04-21T08:52:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.871+0000 7f96c921c740 1 --2- 172.21.15.28:0/1419231712 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fe3c9d6f40 0x55fe3c9d9360 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.871+0000 7f96c921c740 1 --2- 172.21.15.28:0/1419231712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe3c93da00 0x55fe3c943ec0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.871+0000 7f96c921c740 1 --2- 172.21.15.28:0/1419231712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe3c93ad30 0x55fe3c943980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.871+0000 7f96c921c740 1 --2- 172.21.15.28:0/1419231712 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fe3c9344d0 0x55fe3c943440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.871+0000 7f96c921c740 1 -- 172.21.15.28:0/1419231712 >> 172.21.15.28:0/1419231712 conn(0x55fe3c92eba0 msgr2=0x55fe3c92dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.871+0000 7f969f7fe640 0 client.16263 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:25.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.871+0000 7f96c921c740 1 -- 172.21.15.28:0/1419231712 shutdown_connections 2024-04-21T08:52:25.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.872+0000 7f96c921c740 1 -- 172.21.15.28:0/1419231712 wait complete. 2024-04-21T08:52:25.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.872+0000 7f96c921c740 20 client.16263 trim_cache size 0 max 16384 2024-04-21T08:52:25.872 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:52:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.873+0000 7f96c6550640 1 -- 172.21.15.28:0/3676845687 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f96ac0a8430 msgr2=0x7f96ac0aa8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.873+0000 7f96c6550640 1 --2- 172.21.15.28:0/3676845687 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f96ac0a8430 0x7f96ac0aa8f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f96c0070d90 tx=0x7f96a8005f90 comp rx=0 tx=0).stop 2024-04-21T08:52:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.873+0000 7f96c6550640 1 -- 172.21.15.28:0/3676845687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96c01105e0 msgr2=0x7f96c01a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.873+0000 7f96c6550640 1 --2- 172.21.15.28:0/3676845687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96c01105e0 0x7f96c01a35f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f96b800eda0 tx=0x7f96b800c6e0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.873+0000 7f96c6550640 1 -- 172.21.15.28:0/3676845687 shutdown_connections 2024-04-21T08:52:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.873+0000 7f96c6550640 1 --2- 172.21.15.28:0/3676845687 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f96ac0a8430 0x7f96ac0aa8f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.873+0000 7f96c6550640 1 --2- 172.21.15.28:0/3676845687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96c01105e0 0x7f96c01a35f0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.873+0000 7f96c6550640 1 --2- 172.21.15.28:0/3676845687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f96c010fc20 0x7f96c019f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.873+0000 7f96c6550640 1 --2- 172.21.15.28:0/3676845687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96c010ddc0 0x7f96c019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:25.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.873+0000 7f96c6550640 1 -- 172.21.15.28:0/3676845687 >> 172.21.15.28:0/3676845687 conn(0x7f96c006d060 msgr2=0x7f96c0072a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:25.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.873+0000 7f96c6550640 1 -- 172.21.15.28:0/3676845687 shutdown_connections 2024-04-21T08:52:25.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:25.873+0000 7f96c6550640 1 -- 172.21.15.28:0/3676845687 wait complete. 2024-04-21T08:52:26.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.531+0000 7fa955d9f640 1 -- 172.21.15.28:0/3895232190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9501041f0 msgr2=0x7fa9501045d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:26.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.531+0000 7fa955d9f640 1 --2- 172.21.15.28:0/3895232190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9501041f0 0x7fa9501045d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa940009900 tx=0x7fa94002bc40 comp rx=0 tx=0).stop 2024-04-21T08:52:26.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.531+0000 7fa955d9f640 1 -- 172.21.15.28:0/3895232190 shutdown_connections 2024-04-21T08:52:26.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.531+0000 7fa955d9f640 1 --2- 172.21.15.28:0/3895232190 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9501092c0 0x7fa950110e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.531+0000 7fa955d9f640 1 --2- 172.21.15.28:0/3895232190 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa950104ba0 0x7fa950108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.531+0000 7fa955d9f640 1 --2- 172.21.15.28:0/3895232190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9501041f0 0x7fa9501045d0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.531+0000 7fa955d9f640 1 -- 172.21.15.28:0/3895232190 >> 172.21.15.28:0/3895232190 conn(0x7fa9500ffa30 msgr2=0x7fa950101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:26.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.531+0000 7fa955d9f640 1 -- 172.21.15.28:0/3895232190 shutdown_connections 2024-04-21T08:52:26.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.531+0000 7fa955d9f640 1 -- 172.21.15.28:0/3895232190 wait complete. 2024-04-21T08:52:26.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.532+0000 7fa955d9f640 1 Processor -- start 2024-04-21T08:52:26.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.533+0000 7fa955d9f640 1 -- start start 2024-04-21T08:52:26.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.533+0000 7fa955d9f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9501041f0 0x7fa950076460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.533+0000 7fa955d9f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9501092c0 0x7fa9500769a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.533+0000 7fa955d9f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa950076ee0 0x7fa9501aadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.533+0000 7fa955d9f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa950114200 con 0x7fa9501092c0 2024-04-21T08:52:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.533+0000 7fa955d9f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa950114080 con 0x7fa9501041f0 2024-04-21T08:52:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.533+0000 7fa955d9f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa950114380 con 0x7fa950076ee0 2024-04-21T08:52:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.533+0000 7fa94ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9501092c0 0x7fa9500769a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.533+0000 7fa94ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9501092c0 0x7fa9500769a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56498/0 (socket says 172.21.15.28:56498) 2024-04-21T08:52:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.533+0000 7fa94ffff640 1 -- 172.21.15.28:0/182137186 learned_addr learned my addr 172.21.15.28:0/182137186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.533+0000 7fa95559e640 1 --2- 172.21.15.28:0/182137186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa950076ee0 0x7fa9501aadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.534+0000 7fa94ffff640 1 -- 172.21.15.28:0/182137186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa950076ee0 msgr2=0x7fa9501aadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.534+0000 7fa94ffff640 1 --2- 172.21.15.28:0/182137186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa950076ee0 0x7fa9501aadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.534+0000 7fa94ffff640 1 -- 172.21.15.28:0/182137186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9501041f0 msgr2=0x7fa950076460 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:52:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.534+0000 7fa94ffff640 1 --2- 172.21.15.28:0/182137186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9501041f0 0x7fa950076460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.534+0000 7fa94ffff640 1 -- 172.21.15.28:0/182137186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9501ab330 con 0x7fa9501092c0 2024-04-21T08:52:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.534+0000 7fa94ffff640 1 --2- 172.21.15.28:0/182137186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9501092c0 0x7fa9500769a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa94400e9c0 tx=0x7fa94400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:26.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.534+0000 7fa94dffb640 1 -- 172.21.15.28:0/182137186 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94400cd80 con 0x7fa9501092c0 2024-04-21T08:52:26.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.534+0000 7fa955d9f640 1 -- 172.21.15.28:0/182137186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9501ab5c0 con 0x7fa9501092c0 2024-04-21T08:52:26.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.535+0000 7fa955d9f640 1 -- 172.21.15.28:0/182137186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9501abb30 con 0x7fa9501092c0 2024-04-21T08:52:26.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.535+0000 7fa94dffb640 1 -- 172.21.15.28:0/182137186 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9440054d0 con 0x7fa9501092c0 2024-04-21T08:52:26.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.535+0000 7fa94dffb640 1 -- 172.21.15.28:0/182137186 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa944010430 con 0x7fa9501092c0 2024-04-21T08:52:26.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.539+0000 7fa959bf1740 1 Processor -- start 2024-04-21T08:52:26.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.540+0000 7fa959bf1740 1 -- start start 2024-04-21T08:52:26.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.540+0000 7fa959bf1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae62ac0 0x55fecae61810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.540+0000 7fa95559e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae62ac0 0x55fecae61810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:26.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.540+0000 7fa95559e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae62ac0 0x55fecae61810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56502/0 (socket says 172.21.15.28:56502) 2024-04-21T08:52:26.540 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.541+0000 7fa959bf1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 0x55fecae621d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.541+0000 7fa94dffb640 1 -- 172.21.15.28:0/182137186 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7fa9440046c0 con 0x7fa9501092c0 2024-04-21T08:52:26.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.541+0000 7fa959bf1740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae6b2d0 0x55fecae6d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.541+0000 7fa959bf1740 1 -- 172.21.15.28:0/4228176992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fecadc68d0 con 0x55fecae62ac0 2024-04-21T08:52:26.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.541+0000 7fa959bf1740 1 -- 172.21.15.28:0/4228176992 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55fecad63e80 con 0x55fecae61d50 2024-04-21T08:52:26.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.541+0000 7fa959bf1740 1 -- 172.21.15.28:0/4228176992 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55fecadc2890 con 0x55fecae6b2d0 2024-04-21T08:52:26.541 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.541+0000 7fa95559e640 1 -- 172.21.15.28:0/4228176992 learned_addr learned my addr 172.21.15.28:0/4228176992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.541+0000 7fa95559e640 1 -- 172.21.15.28:0/4228176992 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae6b2d0 msgr2=0x55fecae6d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.542+0000 7fa95559e640 1 --2- 172.21.15.28:0/4228176992 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae6b2d0 0x55fecae6d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.542+0000 7fa95559e640 1 -- 172.21.15.28:0/4228176992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 msgr2=0x55fecae621d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.542+0000 7fa954d9d640 1 --2- 172.21.15.28:0/4228176992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 0x55fecae621d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.542+0000 7fa95559e640 1 --2- 172.21.15.28:0/4228176992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 0x55fecae621d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.542+0000 7fa95559e640 1 -- 172.21.15.28:0/4228176992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fecad85320 con 0x55fecae62ac0 2024-04-21T08:52:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.542+0000 7fa954d9d640 1 --2- 172.21.15.28:0/4228176992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 0x55fecae621d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:52:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.542+0000 7fa95559e640 1 --2- 172.21.15.28:0/4228176992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae62ac0 0x55fecae61810 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa948009e60 tx=0x7fa948010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2829a414eb28dac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.543+0000 7fa94dffb640 1 --2- 172.21.15.28:0/182137186 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa9380a3fc0 0x7fa9380a6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.543+0000 7fa936ffd640 1 -- 172.21.15.28:0/4228176992 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa948016070 con 0x55fecae62ac0 2024-04-21T08:52:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.543+0000 7fa936ffd640 1 -- 172.21.15.28:0/4228176992 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa948004d20 con 0x55fecae62ac0 2024-04-21T08:52:26.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.543+0000 7fa936ffd640 1 -- 172.21.15.28:0/4228176992 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9480127d0 con 0x55fecae62ac0 2024-04-21T08:52:26.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.543+0000 7fa954d9d640 1 --2- 172.21.15.28:0/182137186 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa9380a3fc0 0x7fa9380a6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:26.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.544+0000 7fa959bf1740 1 -- 172.21.15.28:0/4228176992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae62ac0 msgr2=0x55fecae61810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:26.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.544+0000 7fa959bf1740 1 --2- 172.21.15.28:0/4228176992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae62ac0 0x55fecae61810 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa948009e60 tx=0x7fa948010740 comp rx=0 tx=0).stop 2024-04-21T08:52:26.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.544+0000 7fa94dffb640 1 -- 172.21.15.28:0/182137186 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fa9440d1260 con 0x7fa9501092c0 2024-04-21T08:52:26.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.544+0000 7fa959bf1740 1 -- 172.21.15.28:0/4228176992 shutdown_connections 2024-04-21T08:52:26.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.544+0000 7fa959bf1740 1 --2- 172.21.15.28:0/4228176992 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae6b2d0 0x55fecae6d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.544+0000 7fa959bf1740 1 --2- 172.21.15.28:0/4228176992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 0x55fecae621d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.544+0000 7fa959bf1740 1 --2- 172.21.15.28:0/4228176992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae62ac0 0x55fecae61810 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.544+0000 7fa959bf1740 1 -- 172.21.15.28:0/4228176992 >> 172.21.15.28:0/4228176992 conn(0x55fecae28c90 msgr2=0x55fecae5ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:26.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.544+0000 7fa954d9d640 1 --2- 172.21.15.28:0/182137186 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa9380a3fc0 0x7fa9380a6480 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa940005ec0 tx=0x7fa940004040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:26.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.544+0000 7fa959bf1740 1 -- 172.21.15.28:0/4228176992 shutdown_connections 2024-04-21T08:52:26.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.545+0000 7fa959bf1740 1 -- 172.21.15.28:0/4228176992 wait complete. 2024-04-21T08:52:26.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.545+0000 7fa959bf1740 1 -- start start 2024-04-21T08:52:26.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.545+0000 7fa959bf1740 1 Processor -- start 2024-04-21T08:52:26.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.546+0000 7fa959bf1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 0x55fecae6ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.546+0000 7fa959bf1740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae62ac0 0x55fecae68000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.546+0000 7fa959bf1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae6b2d0 0x55fecae685a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.546+0000 7fa959bf1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fecae6e000 con 0x55fecae6b2d0 2024-04-21T08:52:26.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.546+0000 7fa959bf1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55fecad4eeb0 con 0x55fecae61d50 2024-04-21T08:52:26.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.546+0000 7fa959bf1740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55fecae6e180 con 0x55fecae62ac0 2024-04-21T08:52:26.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.546+0000 7fa95559e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae6b2d0 0x55fecae685a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:26.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.546+0000 7fa95559e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae6b2d0 0x55fecae685a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56518/0 (socket says 172.21.15.28:56518) 2024-04-21T08:52:26.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.546+0000 7fa95559e640 1 -- 172.21.15.28:0/3410558075 learned_addr learned my addr 172.21.15.28:0/3410558075 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:26.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.546+0000 7fa959bf1740 10 client.0 _ll_register_callbacks cb 0x55fecad7bb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:26.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.547+0000 7fa954d9d640 1 --2- 172.21.15.28:0/3410558075 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 0x55fecae6ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:26.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.548+0000 7fa95559e640 1 -- 172.21.15.28:0/3410558075 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae62ac0 msgr2=0x55fecae68000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:26.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.548+0000 7fa95559e640 1 --2- 172.21.15.28:0/3410558075 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae62ac0 0x55fecae68000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.548+0000 7fa95559e640 1 -- 172.21.15.28:0/3410558075 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 msgr2=0x55fecae6ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:26.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.548+0000 7fa95559e640 1 --2- 172.21.15.28:0/3410558075 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 0x55fecae6ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.548+0000 7fa95559e640 1 -- 172.21.15.28:0/3410558075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fecaad9210 con 0x55fecae6b2d0 2024-04-21T08:52:26.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.548+0000 7fa95559e640 1 --2- 172.21.15.28:0/3410558075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae6b2d0 0x55fecae685a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fa948011010 tx=0x7fa948005100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a9a8d87871b05ac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:26.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.548+0000 7fa92ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:26.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.549+0000 7fa92ffff640 1 -- 172.21.15.28:0/3410558075 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa948028060 con 0x55fecae6b2d0 2024-04-21T08:52:26.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.549+0000 7fa92ffff640 1 -- 172.21.15.28:0/3410558075 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fa948005530 con 0x55fecae6b2d0 2024-04-21T08:52:26.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.549+0000 7fa959bf1740 1 -- 172.21.15.28:0/3410558075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa948004d20 con 0x55fecae6b2d0 2024-04-21T08:52:26.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.549+0000 7fa92ffff640 1 -- 172.21.15.28:0/3410558075 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa94802f050 con 0x55fecae6b2d0 2024-04-21T08:52:26.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.550+0000 7fa959bf1740 10 client.16275 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:26.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.550+0000 7fa959bf1740 1 -- 172.21.15.28:0/3410558075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fecaba5620 con 0x55fecae6b2d0 2024-04-21T08:52:26.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.550+0000 7fa92ffff640 1 -- 172.21.15.28:0/3410558075 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa948004d20 con 0x55fecae6b2d0 2024-04-21T08:52:26.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.550+0000 7fa92ffff640 1 -- 172.21.15.28:0/3410558075 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fa94800f040 con 0x55fecae6b2d0 2024-04-21T08:52:26.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.550+0000 7fa959bf1740 10 client.16275 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:26.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.550+0000 7fa959bf1740 10 client.16275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:26.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.550+0000 7fa959bf1740 20 client.16275 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:26.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.550+0000 7fa959bf1740 1 --2- 172.21.15.28:0/3410558075 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fecae7af40 0x55fecae7d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.550+0000 7fa959bf1740 4 client.16275 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:26.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.550+0000 7fa959bf1740 1 -- 172.21.15.28:0/3410558075 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fecadda880 con 0x55fecae7af40 2024-04-21T08:52:26.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.551+0000 7fa954d9d640 1 --2- 172.21.15.28:0/3410558075 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fecae7af40 0x55fecae7d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:26.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.554+0000 7fa92ffff640 10 client.16275 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:26.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.554+0000 7fa954d9d640 1 --2- 172.21.15.28:0/3410558075 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fecae7af40 0x55fecae7d330 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dbff33752f421e6 server_cookie=b25241eea386de8b in_seq=0 out_seq=0 2024-04-21T08:52:26.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.556+0000 7fa92ffff640 1 -- 172.21.15.28:0/3410558075 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fecadda880 con 0x55fecae7af40 2024-04-21T08:52:26.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.556+0000 7fa92ffff640 10 client.16275 handle_command_reply: tid=0 2024-04-21T08:52:26.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.557+0000 7fa959bf1740 1 client.16275 shutdown 2024-04-21T08:52:26.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.557+0000 7fa959bf1740 1 -- 172.21.15.28:0/3410558075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae6b2d0 msgr2=0x55fecae685a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.557+0000 7fa959bf1740 1 --2- 172.21.15.28:0/3410558075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae6b2d0 0x55fecae685a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fa948011010 tx=0x7fa948005100 comp rx=0 tx=0).stop 2024-04-21T08:52:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.557+0000 7fa959bf1740 1 -- 172.21.15.28:0/3410558075 shutdown_connections 2024-04-21T08:52:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.557+0000 7fa959bf1740 1 --2- 172.21.15.28:0/3410558075 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fecae7af40 0x55fecae7d330 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.557+0000 7fa959bf1740 1 --2- 172.21.15.28:0/3410558075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae6b2d0 0x55fecae685a0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.557+0000 7fa959bf1740 1 --2- 172.21.15.28:0/3410558075 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae62ac0 0x55fecae68000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.557+0000 7fa959bf1740 1 --2- 172.21.15.28:0/3410558075 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 0x55fecae6ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.557+0000 7fa959bf1740 1 -- 172.21.15.28:0/3410558075 >> 172.21.15.28:0/3410558075 conn(0x55fecae28c90 msgr2=0x55fecae5e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.557+0000 7fa92ffff640 0 client.16275 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.557+0000 7fa959bf1740 1 -- 172.21.15.28:0/3410558075 shutdown_connections 2024-04-21T08:52:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.557+0000 7fa959bf1740 1 -- 172.21.15.28:0/3410558075 wait complete. 2024-04-21T08:52:26.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.558+0000 7fa959bf1740 20 client.16275 trim_cache size 0 max 16384 2024-04-21T08:52:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.592+0000 7fa959bf1740 1 Processor -- start 2024-04-21T08:52:26.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.597+0000 7fa959bf1740 1 -- start start 2024-04-21T08:52:26.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.597+0000 7fa959bf1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae61d50 0x55fecae76f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.597+0000 7fa959bf1740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae62ac0 0x55fecae6a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.597+0000 7fa959bf1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae6b2d0 0x55fecae6ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.598+0000 7fa959bf1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c012880 con 0x55fecae61d50 2024-04-21T08:52:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.598+0000 7fa959bf1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c010150 con 0x55fecae6b2d0 2024-04-21T08:52:26.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.598+0000 7fa959bf1740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c00dfa0 con 0x55fecae62ac0 2024-04-21T08:52:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.600+0000 7fa954d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae61d50 0x55fecae76f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.600+0000 7fa954d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae61d50 0x55fecae76f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56532/0 (socket says 172.21.15.28:56532) 2024-04-21T08:52:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.600+0000 7fa954d9d640 1 -- 172.21.15.28:0/1699019203 learned_addr learned my addr 172.21.15.28:0/1699019203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.601+0000 7fa954d9d640 1 -- 172.21.15.28:0/1699019203 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae62ac0 msgr2=0x55fecae6a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.601+0000 7fa954d9d640 1 --2- 172.21.15.28:0/1699019203 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae62ac0 0x55fecae6a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.601+0000 7fa954d9d640 1 -- 172.21.15.28:0/1699019203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae6b2d0 msgr2=0x55fecae6ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.601+0000 7fa954d9d640 1 --2- 172.21.15.28:0/1699019203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae6b2d0 0x55fecae6ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.601+0000 7fa954d9d640 1 -- 172.21.15.28:0/1699019203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fecae61020 con 0x55fecae61d50 2024-04-21T08:52:26.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.601+0000 7fa954d9d640 1 --2- 172.21.15.28:0/1699019203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae61d50 0x55fecae76f30 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x55fecae63220 tx=0x7fa94003c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd02b3e884dac9c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.604+0000 7fa9367fc640 1 -- 172.21.15.28:0/1699019203 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940046040 con 0x55fecae61d50 2024-04-21T08:52:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.604+0000 7fa9367fc640 1 -- 172.21.15.28:0/1699019203 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa940055030 con 0x55fecae61d50 2024-04-21T08:52:26.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.604+0000 7fa9367fc640 1 -- 172.21.15.28:0/1699019203 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940004cf0 con 0x55fecae61d50 2024-04-21T08:52:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.605+0000 7fa959bf1740 1 -- 172.21.15.28:0/1699019203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae61d50 msgr2=0x55fecae76f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:26.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.605+0000 7fa959bf1740 1 --2- 172.21.15.28:0/1699019203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae61d50 0x55fecae76f30 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x55fecae63220 tx=0x7fa94003c000 comp rx=0 tx=0).stop 2024-04-21T08:52:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.605+0000 7fa959bf1740 1 -- 172.21.15.28:0/1699019203 shutdown_connections 2024-04-21T08:52:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.605+0000 7fa959bf1740 1 --2- 172.21.15.28:0/1699019203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae6b2d0 0x55fecae6ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.605+0000 7fa959bf1740 1 --2- 172.21.15.28:0/1699019203 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae62ac0 0x55fecae6a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.605+0000 7fa959bf1740 1 --2- 172.21.15.28:0/1699019203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae61d50 0x55fecae76f30 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.605+0000 7fa959bf1740 1 -- 172.21.15.28:0/1699019203 >> 172.21.15.28:0/1699019203 conn(0x55fecae28c90 msgr2=0x55fecae63270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.606+0000 7fa959bf1740 1 -- 172.21.15.28:0/1699019203 shutdown_connections 2024-04-21T08:52:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.606+0000 7fa959bf1740 1 -- 172.21.15.28:0/1699019203 wait complete. 2024-04-21T08:52:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.607+0000 7fa959bf1740 1 -- start start 2024-04-21T08:52:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.607+0000 7fa959bf1740 1 Processor -- start 2024-04-21T08:52:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.608+0000 7fa959bf1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 0x55fecae736a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.608+0000 7fa959bf1740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae62ac0 0x55fecae64870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.608+0000 7fa959bf1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae6b2d0 0x55fecae64db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.608+0000 7fa959bf1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c00e610 con 0x55fecae6b2d0 2024-04-21T08:52:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.608+0000 7fa959bf1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c00fa80 con 0x55fecae61d50 2024-04-21T08:52:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.608+0000 7fa959bf1740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa93c00ece0 con 0x55fecae62ac0 2024-04-21T08:52:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.608+0000 7fa959bf1740 10 client.0 _ll_register_callbacks cb 0x7fa948009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.609+0000 7fa95559e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae6b2d0 0x55fecae64db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.609+0000 7fa95559e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae6b2d0 0x55fecae64db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56538/0 (socket says 172.21.15.28:56538) 2024-04-21T08:52:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.609+0000 7fa95559e640 1 -- 172.21.15.28:0/4028101519 learned_addr learned my addr 172.21.15.28:0/4028101519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.609+0000 7fa954d9d640 1 --2- 172.21.15.28:0/4028101519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 0x55fecae736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.610+0000 7fa95559e640 1 -- 172.21.15.28:0/4028101519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae62ac0 msgr2=0x55fecae64870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.610+0000 7fa94ffff640 1 --2- 172.21.15.28:0/4028101519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae62ac0 0x55fecae64870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.610+0000 7fa95559e640 1 --2- 172.21.15.28:0/4028101519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae62ac0 0x55fecae64870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.610+0000 7fa95559e640 1 -- 172.21.15.28:0/4028101519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 msgr2=0x55fecae736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.610+0000 7fa95559e640 1 --2- 172.21.15.28:0/4028101519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 0x55fecae736a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.610+0000 7fa95559e640 1 -- 172.21.15.28:0/4028101519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fecae65450 con 0x55fecae6b2d0 2024-04-21T08:52:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.610+0000 7fa954d9d640 1 --2- 172.21.15.28:0/4028101519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 0x55fecae736a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:52:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.610+0000 7fa95559e640 1 --2- 172.21.15.28:0/4028101519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae6b2d0 0x55fecae64db0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa948005760 tx=0x7fa9480050e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfff8211ff6e4575 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.610+0000 7fa92ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.615+0000 7fa92ffff640 1 -- 172.21.15.28:0/4028101519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa948028040 con 0x55fecae6b2d0 2024-04-21T08:52:26.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.615+0000 7fa959bf1740 1 -- 172.21.15.28:0/4028101519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa940055030 con 0x55fecae6b2d0 2024-04-21T08:52:26.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.616+0000 7fa92ffff640 1 -- 172.21.15.28:0/4028101519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fa948020070 con 0x55fecae6b2d0 2024-04-21T08:52:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.619+0000 7fa92ffff640 1 -- 172.21.15.28:0/4028101519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa948016070 con 0x55fecae6b2d0 2024-04-21T08:52:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.619+0000 7fa959bf1740 10 client.16281 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:26.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.619+0000 7fa959bf1740 1 -- 172.21.15.28:0/4028101519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fecaef20b0 con 0x55fecae6b2d0 2024-04-21T08:52:26.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:26 smithi171 ceph-mon[31358]: pgmap v40: 129 pgs: 129 active+clean; 1.1 GiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 96 MiB/s wr, 221 op/s 2024-04-21T08:52:26.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.621+0000 7fa92ffff640 1 -- 172.21.15.28:0/4028101519 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa940055030 con 0x55fecae6b2d0 2024-04-21T08:52:26.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.621+0000 7fa92ffff640 1 -- 172.21.15.28:0/4028101519 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fa948032020 con 0x55fecae6b2d0 2024-04-21T08:52:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.622+0000 7fa959bf1740 10 client.16281 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.622+0000 7fa959bf1740 10 client.16281 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.622+0000 7fa959bf1740 20 client.16281 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.622+0000 7fa959bf1740 1 --2- 172.21.15.28:0/4028101519 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fecaef2800 0x55fecaef4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.622+0000 7fa959bf1740 4 client.16281 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:52:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.622+0000 7fa959bf1740 1 -- 172.21.15.28:0/4028101519 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fecad663d0 con 0x55fecaef2800 2024-04-21T08:52:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.624+0000 7fa954d9d640 1 --2- 172.21.15.28:0/4028101519 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fecaef2800 0x55fecaef4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.650+0000 7fa954d9d640 1 --2- 172.21.15.28:0/4028101519 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fecaef2800 0x55fecaef4c20 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=650e2980f556b231 server_cookie=871fe8686328bc35 in_seq=0 out_seq=0 2024-04-21T08:52:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.650+0000 7fa92ffff640 10 client.16281 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.748+0000 7fa92ffff640 1 -- 172.21.15.28:0/4028101519 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fecad663d0 con 0x55fecaef2800 2024-04-21T08:52:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.748+0000 7fa92ffff640 10 client.16281 handle_command_reply: tid=0 2024-04-21T08:52:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.748+0000 7fa959bf1740 1 client.16281 shutdown 2024-04-21T08:52:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.748+0000 7fa959bf1740 1 -- 172.21.15.28:0/4028101519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae6b2d0 msgr2=0x55fecae64db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.748+0000 7fa959bf1740 1 --2- 172.21.15.28:0/4028101519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae6b2d0 0x55fecae64db0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa948005760 tx=0x7fa9480050e0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.748+0000 7fa959bf1740 1 -- 172.21.15.28:0/4028101519 shutdown_connections 2024-04-21T08:52:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.748+0000 7fa959bf1740 1 --2- 172.21.15.28:0/4028101519 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55fecaef2800 0x55fecaef4c20 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.748+0000 7fa959bf1740 1 --2- 172.21.15.28:0/4028101519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fecae6b2d0 0x55fecae64db0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.748+0000 7fa959bf1740 1 --2- 172.21.15.28:0/4028101519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55fecae62ac0 0x55fecae64870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.748+0000 7fa92ffff640 0 client.16281 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.748+0000 7fa959bf1740 1 --2- 172.21.15.28:0/4028101519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fecae61d50 0x55fecae736a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.748+0000 7fa959bf1740 1 -- 172.21.15.28:0/4028101519 >> 172.21.15.28:0/4028101519 conn(0x55fecae28c90 msgr2=0x55fecae5b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.749+0000 7fa959bf1740 1 -- 172.21.15.28:0/4028101519 shutdown_connections 2024-04-21T08:52:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.749+0000 7fa959bf1740 1 -- 172.21.15.28:0/4028101519 wait complete. 2024-04-21T08:52:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.749+0000 7fa959bf1740 20 client.16281 trim_cache size 0 max 16384 2024-04-21T08:52:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:52:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:52:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:52:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:52:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.750+0000 7fa955d9f640 1 -- 172.21.15.28:0/182137186 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa9380a3fc0 msgr2=0x7fa9380a6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.750+0000 7fa955d9f640 1 --2- 172.21.15.28:0/182137186 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa9380a3fc0 0x7fa9380a6480 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa940005ec0 tx=0x7fa940004040 comp rx=0 tx=0).stop 2024-04-21T08:52:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.750+0000 7fa955d9f640 1 -- 172.21.15.28:0/182137186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9501092c0 msgr2=0x7fa9500769a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.750+0000 7fa955d9f640 1 --2- 172.21.15.28:0/182137186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9501092c0 0x7fa9500769a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa94400e9c0 tx=0x7fa94400ee80 comp rx=0 tx=0).stop 2024-04-21T08:52:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.750+0000 7fa955d9f640 1 -- 172.21.15.28:0/182137186 shutdown_connections 2024-04-21T08:52:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.750+0000 7fa955d9f640 1 --2- 172.21.15.28:0/182137186 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa9380a3fc0 0x7fa9380a6480 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.750+0000 7fa955d9f640 1 --2- 172.21.15.28:0/182137186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa950076ee0 0x7fa9501aadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.750+0000 7fa955d9f640 1 --2- 172.21.15.28:0/182137186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9501092c0 0x7fa9500769a0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.750+0000 7fa955d9f640 1 --2- 172.21.15.28:0/182137186 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9501041f0 0x7fa950076460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:26.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.750+0000 7fa955d9f640 1 -- 172.21.15.28:0/182137186 >> 172.21.15.28:0/182137186 conn(0x7fa9500ffa30 msgr2=0x7fa950100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:26.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.750+0000 7fa955d9f640 1 -- 172.21.15.28:0/182137186 shutdown_connections 2024-04-21T08:52:26.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:26.750+0000 7fa955d9f640 1 -- 172.21.15.28:0/182137186 wait complete. 2024-04-21T08:52:26.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:26 smithi152 ceph-mon[31401]: pgmap v40: 129 pgs: 129 active+clean; 1.1 GiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 96 MiB/s wr, 221 op/s 2024-04-21T08:52:26.854 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:26 smithi028 ceph-mon[28003]: pgmap v40: 129 pgs: 129 active+clean; 1.1 GiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 96 MiB/s wr, 221 op/s 2024-04-21T08:52:26.893 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:52:26.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:52:26.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:52:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.083+0000 7f70cf811640 1 -- 172.21.15.28:0/70366209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c806ace0 msgr2=0x7f70c810ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.083+0000 7f70cf811640 1 --2- 172.21.15.28:0/70366209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c806ace0 0x7f70c810ee00 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f70c0009cd0 tx=0x7f70c002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.083+0000 7f70cf811640 1 -- 172.21.15.28:0/70366209 shutdown_connections 2024-04-21T08:52:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.083+0000 7f70cf811640 1 --2- 172.21.15.28:0/70366209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c806ace0 0x7f70c810ee00 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.083+0000 7f70cf811640 1 --2- 172.21.15.28:0/70366209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70c806a360 0x7f70c806a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.083+0000 7f70cf811640 1 --2- 172.21.15.28:0/70366209 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70c8107fd0 0x7f70c81083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.083+0000 7f70cf811640 1 -- 172.21.15.28:0/70366209 >> 172.21.15.28:0/70366209 conn(0x7f70c8077ef0 msgr2=0x7f70c8078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.083+0000 7f70cf811640 1 -- 172.21.15.28:0/70366209 shutdown_connections 2024-04-21T08:52:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.083+0000 7f70cf811640 1 -- 172.21.15.28:0/70366209 wait complete. 2024-04-21T08:52:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cf811640 1 Processor -- start 2024-04-21T08:52:27.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cf811640 1 -- start start 2024-04-21T08:52:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cf811640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70c806a360 0x7f70c819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cf811640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70c806ace0 0x7f70c819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cf811640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c8107fd0 0x7f70c8198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cf811640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f70c81121b0 con 0x7f70c8107fd0 2024-04-21T08:52:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cf811640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f70c8112030 con 0x7f70c806a360 2024-04-21T08:52:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cf811640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f70c8112330 con 0x7f70c806ace0 2024-04-21T08:52:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cddae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c8107fd0 0x7f70c8198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cddae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c8107fd0 0x7f70c8198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56560/0 (socket says 172.21.15.28:56560) 2024-04-21T08:52:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cddae640 1 -- 172.21.15.28:0/3198582653 learned_addr learned my addr 172.21.15.28:0/3198582653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cd5ad640 1 --2- 172.21.15.28:0/3198582653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70c806a360 0x7f70c819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70ccdac640 1 --2- 172.21.15.28:0/3198582653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70c806ace0 0x7f70c819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cddae640 1 -- 172.21.15.28:0/3198582653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70c806ace0 msgr2=0x7f70c819e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cddae640 1 --2- 172.21.15.28:0/3198582653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70c806ace0 0x7f70c819e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cddae640 1 -- 172.21.15.28:0/3198582653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70c806a360 msgr2=0x7f70c819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cddae640 1 --2- 172.21.15.28:0/3198582653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70c806a360 0x7f70c819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cddae640 1 -- 172.21.15.28:0/3198582653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c8198d10 con 0x7f70c8107fd0 2024-04-21T08:52:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.084+0000 7f70cddae640 1 --2- 172.21.15.28:0/3198582653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c8107fd0 0x7f70c8198590 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f70c0004af0 tx=0x7f70c00346e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.085+0000 7f70ba7fc640 1 -- 172.21.15.28:0/3198582653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c0041020 con 0x7f70c8107fd0 2024-04-21T08:52:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.085+0000 7f70cf811640 1 -- 172.21.15.28:0/3198582653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c8198fa0 con 0x7f70c8107fd0 2024-04-21T08:52:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.085+0000 7f70ba7fc640 1 -- 172.21.15.28:0/3198582653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70c0032970 con 0x7f70c8107fd0 2024-04-21T08:52:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.085+0000 7f70ba7fc640 1 -- 172.21.15.28:0/3198582653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c0039040 con 0x7f70c8107fd0 2024-04-21T08:52:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.085+0000 7f70cf811640 1 -- 172.21.15.28:0/3198582653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c81a52a0 con 0x7f70c8107fd0 2024-04-21T08:52:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.087+0000 7f70ba7fc640 1 -- 172.21.15.28:0/3198582653 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f70c0044020 con 0x7f70c8107fd0 2024-04-21T08:52:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.087+0000 7f70d2404740 1 Processor -- start 2024-04-21T08:52:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.087+0000 7f70d2404740 1 -- start start 2024-04-21T08:52:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70d2404740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a87c0 0x5582d57a8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70d2404740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a9170 0x5582d57b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70d2404740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57b2270 0x5582d57b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70d2404740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5582d570cf00 con 0x5582d57b2270 2024-04-21T08:52:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70d2404740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5582d56aa590 con 0x5582d57a9170 2024-04-21T08:52:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70d2404740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5582d5708ec0 con 0x5582d57a87c0 2024-04-21T08:52:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70ccdac640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a87c0 0x5582d57a8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70cd5ad640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a9170 0x5582d57b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70cd5ad640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a9170 0x5582d57b1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38440/0 (socket says 172.21.15.28:38440) 2024-04-21T08:52:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70ccdac640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a87c0 0x5582d57a8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:51572/0 (socket says 172.21.15.28:51572) 2024-04-21T08:52:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70cd5ad640 1 -- 172.21.15.28:0/3543364023 learned_addr learned my addr 172.21.15.28:0/3543364023 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70ccdac640 1 --2- 172.21.15.28:0/3543364023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57b2270 0x5582d57b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70ba7fc640 1 --2- 172.21.15.28:0/3198582653 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f70b40a3ed0 0x7f70b40a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70ccdac640 1 -- 172.21.15.28:0/3543364023 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a9170 msgr2=0x5582d57b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70ccdac640 1 --2- 172.21.15.28:0/3543364023 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a9170 0x5582d57b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70ccdac640 1 -- 172.21.15.28:0/3543364023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57b2270 msgr2=0x5582d57b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70ccdac640 1 --2- 172.21.15.28:0/3543364023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57b2270 0x5582d57b26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70ccdac640 1 -- 172.21.15.28:0/3543364023 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582d54e77e0 con 0x5582d57a87c0 2024-04-21T08:52:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70cd5ad640 1 --2- 172.21.15.28:0/3543364023 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a9170 0x5582d57b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70cddae640 1 --2- 172.21.15.28:0/3198582653 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f70b40a3ed0 0x7f70b40a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.088+0000 7f70ccdac640 1 --2- 172.21.15.28:0/3543364023 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a87c0 0x5582d57a8ba0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f70c80ff390 tx=0x7f70bc00a640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6170f4caf823f5a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.089+0000 7f70ba7fc640 1 -- 172.21.15.28:0/3198582653 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f70c00f2aa0 con 0x7f70c8107fd0 2024-04-21T08:52:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.089+0000 7f70b37fe640 1 -- 172.21.15.28:0/3543364023 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc004020 con 0x5582d57a87c0 2024-04-21T08:52:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.089+0000 7f70b37fe640 1 -- 172.21.15.28:0/3543364023 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70bc0054d0 con 0x5582d57a87c0 2024-04-21T08:52:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.089+0000 7f70b37fe640 1 -- 172.21.15.28:0/3543364023 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc004c40 con 0x5582d57a87c0 2024-04-21T08:52:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.089+0000 7f70d2404740 1 -- 172.21.15.28:0/3543364023 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a87c0 msgr2=0x5582d57a8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.089+0000 7f70cddae640 1 --2- 172.21.15.28:0/3198582653 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f70b40a3ed0 0x7f70b40a6390 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f70c00efd80 tx=0x7f70c00a90a0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.089+0000 7f70d2404740 1 --2- 172.21.15.28:0/3543364023 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a87c0 0x5582d57a8ba0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f70c80ff390 tx=0x7f70bc00a640 comp rx=0 tx=0).stop 2024-04-21T08:52:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 1 -- 172.21.15.28:0/3543364023 shutdown_connections 2024-04-21T08:52:27.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 1 --2- 172.21.15.28:0/3543364023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57b2270 0x5582d57b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 1 --2- 172.21.15.28:0/3543364023 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a9170 0x5582d57b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 1 --2- 172.21.15.28:0/3543364023 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a87c0 0x5582d57a8ba0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 1 -- 172.21.15.28:0/3543364023 >> 172.21.15.28:0/3543364023 conn(0x5582d57a2fb0 msgr2=0x5582d57a3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 1 -- 172.21.15.28:0/3543364023 shutdown_connections 2024-04-21T08:52:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 1 -- 172.21.15.28:0/3543364023 wait complete. 2024-04-21T08:52:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 1 -- start start 2024-04-21T08:52:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 1 Processor -- start 2024-04-21T08:52:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a87c0 0x5582d57b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a9170 0x5582d57b6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57b2270 0x5582d57b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5582d57b91b0 con 0x5582d57b2270 2024-04-21T08:52:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5582d56c6580 con 0x5582d57a9170 2024-04-21T08:52:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5582d57b9330 con 0x5582d57a87c0 2024-04-21T08:52:27.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.090+0000 7f70d2404740 10 client.0 _ll_register_callbacks cb 0x5582d56cc650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70ccdac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57b2270 0x5582d57b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70cd5ad640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a9170 0x5582d57b6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70ccdac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57b2270 0x5582d57b67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56578/0 (socket says 172.21.15.28:56578) 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70ccdac640 1 -- 172.21.15.28:0/2546787321 learned_addr learned my addr 172.21.15.28:0/2546787321 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70ccdac640 1 --2- 172.21.15.28:0/2546787321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a87c0 0x5582d57b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70cd5ad640 1 -- 172.21.15.28:0/2546787321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a87c0 msgr2=0x5582d57b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70cd5ad640 1 --2- 172.21.15.28:0/2546787321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a87c0 0x5582d57b5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70cd5ad640 1 -- 172.21.15.28:0/2546787321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57b2270 msgr2=0x5582d57b67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70cd5ad640 1 --2- 172.21.15.28:0/2546787321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57b2270 0x5582d57b67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70cd5ad640 1 -- 172.21.15.28:0/2546787321 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582d5547f70 con 0x5582d57a9170 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70ccdac640 1 --2- 172.21.15.28:0/2546787321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a87c0 0x5582d57b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70cd5ad640 1 --2- 172.21.15.28:0/2546787321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a9170 0x5582d57b6290 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f70c4005850 tx=0x7f70c400be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa3b9170b0a757d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70b0ff9640 1 -- 172.21.15.28:0/2546787321 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c4003eb0 con 0x5582d57a9170 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70d2404740 1 -- 172.21.15.28:0/2546787321 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70bc0054d0 con 0x5582d57a9170 2024-04-21T08:52:27.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.091+0000 7f70b0ff9640 1 -- 172.21.15.28:0/2546787321 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f70c4005020 con 0x5582d57a9170 2024-04-21T08:52:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.092+0000 7f70b0ff9640 1 -- 172.21.15.28:0/2546787321 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f70c4024050 con 0x5582d57a9170 2024-04-21T08:52:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.092+0000 7f70d2404740 10 client.25691 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.092+0000 7f70d2404740 1 -- 172.21.15.28:0/2546787321 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5582d53df170 con 0x5582d57a9170 2024-04-21T08:52:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.092+0000 7f70b0ff9640 1 -- 172.21.15.28:0/2546787321 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70bc0054d0 con 0x5582d57a9170 2024-04-21T08:52:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.092+0000 7f70b0ff9640 1 -- 172.21.15.28:0/2546787321 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f70c401b470 con 0x5582d57a9170 2024-04-21T08:52:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.092+0000 7f70d2404740 10 client.25691 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.092+0000 7f70d2404740 10 client.25691 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.092+0000 7f70d2404740 20 client.25691 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.092+0000 7f70d2404740 1 --2- 172.21.15.28:0/2546787321 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5582d57c6810 0x5582d57c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.092+0000 7f70d2404740 4 client.25691 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.092+0000 7f70d2404740 1 -- 172.21.15.28:0/2546787321 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582d571a920 con 0x5582d57c6810 2024-04-21T08:52:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.092+0000 7f70ccdac640 1 --2- 172.21.15.28:0/2546787321 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5582d57c6810 0x5582d57c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.093+0000 7f70b0ff9640 10 client.25691 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.093+0000 7f70ccdac640 1 --2- 172.21.15.28:0/2546787321 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5582d57c6810 0x5582d57c8c00 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7dc4e67c1d059ca server_cookie=15269186599e3bba in_seq=0 out_seq=0 2024-04-21T08:52:27.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.093+0000 7f70b0ff9640 1 -- 172.21.15.28:0/2546787321 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5582d571a920 con 0x5582d57c6810 2024-04-21T08:52:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.093+0000 7f70b0ff9640 10 client.25691 handle_command_reply: tid=0 2024-04-21T08:52:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.093+0000 7f70d2404740 1 client.25691 shutdown 2024-04-21T08:52:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.093+0000 7f70d2404740 1 -- 172.21.15.28:0/2546787321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a9170 msgr2=0x5582d57b6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.093+0000 7f70d2404740 1 --2- 172.21.15.28:0/2546787321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a9170 0x5582d57b6290 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f70c4005850 tx=0x7f70c400be10 comp rx=0 tx=0).stop 2024-04-21T08:52:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.093+0000 7f70d2404740 1 -- 172.21.15.28:0/2546787321 shutdown_connections 2024-04-21T08:52:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.093+0000 7f70d2404740 1 --2- 172.21.15.28:0/2546787321 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5582d57c6810 0x5582d57c8c00 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.093+0000 7f70d2404740 1 --2- 172.21.15.28:0/2546787321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57b2270 0x5582d57b67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.093+0000 7f70d2404740 1 --2- 172.21.15.28:0/2546787321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a9170 0x5582d57b6290 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.093+0000 7f70d2404740 1 --2- 172.21.15.28:0/2546787321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a87c0 0x5582d57b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.093+0000 7f70b0ff9640 0 client.25691 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.093+0000 7f70d2404740 1 -- 172.21.15.28:0/2546787321 >> 172.21.15.28:0/2546787321 conn(0x5582d57a2fb0 msgr2=0x5582d57a4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.094+0000 7f70d2404740 1 -- 172.21.15.28:0/2546787321 shutdown_connections 2024-04-21T08:52:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.094+0000 7f70d2404740 1 -- 172.21.15.28:0/2546787321 wait complete. 2024-04-21T08:52:27.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.094+0000 7f70d2404740 20 client.25691 trim_cache size 0 max 16384 2024-04-21T08:52:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.112+0000 7f70d2404740 1 Processor -- start 2024-04-21T08:52:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.112+0000 7f70d2404740 1 -- start start 2024-04-21T08:52:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.112+0000 7f70d2404740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a87c0 0x5582d57b8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.112+0000 7f70d2404740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57a9170 0x5582d57c20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.112+0000 7f70d2404740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57b2270 0x5582d57b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.112+0000 7f70d2404740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f70c81afb00 con 0x5582d57a9170 2024-04-21T08:52:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.112+0000 7f70d2404740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f70c81ad3d0 con 0x5582d57a87c0 2024-04-21T08:52:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.112+0000 7f70d2404740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f70c81ab220 con 0x5582d57b2270 2024-04-21T08:52:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.112+0000 7f70cd5ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57a9170 0x5582d57c20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.112+0000 7f70ccdac640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a87c0 0x5582d57b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.112+0000 7f70cd5ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57a9170 0x5582d57c20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56580/0 (socket says 172.21.15.28:56580) 2024-04-21T08:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.112+0000 7f70ccdac640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a87c0 0x5582d57b8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38452/0 (socket says 172.21.15.28:38452) 2024-04-21T08:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.112+0000 7f70cd5ad640 1 -- 172.21.15.28:0/190924189 learned_addr learned my addr 172.21.15.28:0/190924189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.112+0000 7f70ccdac640 1 --2- 172.21.15.28:0/190924189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57b2270 0x5582d57b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.113+0000 7f70cd5ad640 1 -- 172.21.15.28:0/190924189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57b2270 msgr2=0x5582d57b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.113+0000 7f70cd5ad640 1 --2- 172.21.15.28:0/190924189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57b2270 0x5582d57b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.113+0000 7f70cd5ad640 1 -- 172.21.15.28:0/190924189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a87c0 msgr2=0x5582d57b8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.113+0000 7f70cd5ad640 1 --2- 172.21.15.28:0/190924189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a87c0 0x5582d57b8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.113+0000 7f70cd5ad640 1 -- 172.21.15.28:0/190924189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582d579e880 con 0x5582d57a9170 2024-04-21T08:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.113+0000 7f70cd5ad640 1 --2- 172.21.15.28:0/190924189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57a9170 0x5582d57c20e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f70c401ba30 tx=0x7f70c401ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32fe8bacff0e9e9d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.113+0000 7f70b2ffd640 1 -- 172.21.15.28:0/190924189 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c4003f30 con 0x5582d57a9170 2024-04-21T08:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.113+0000 7f70b2ffd640 1 -- 172.21.15.28:0/190924189 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70c402d070 con 0x5582d57a9170 2024-04-21T08:52:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.113+0000 7f70b2ffd640 1 -- 172.21.15.28:0/190924189 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c4005180 con 0x5582d57a9170 2024-04-21T08:52:27.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.114+0000 7f70d2404740 1 -- 172.21.15.28:0/190924189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57a9170 msgr2=0x5582d57c20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.114+0000 7f70d2404740 1 --2- 172.21.15.28:0/190924189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57a9170 0x5582d57c20e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f70c401ba30 tx=0x7f70c401ce70 comp rx=0 tx=0).stop 2024-04-21T08:52:27.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.114+0000 7f70d2404740 1 -- 172.21.15.28:0/190924189 shutdown_connections 2024-04-21T08:52:27.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.114+0000 7f70d2404740 1 --2- 172.21.15.28:0/190924189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57b2270 0x5582d57b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.114+0000 7f70d2404740 1 --2- 172.21.15.28:0/190924189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57a9170 0x5582d57c20e0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.114+0000 7f70d2404740 1 --2- 172.21.15.28:0/190924189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57a87c0 0x5582d57b8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.114+0000 7f70d2404740 1 -- 172.21.15.28:0/190924189 >> 172.21.15.28:0/190924189 conn(0x5582d57a2fb0 msgr2=0x5582d57a4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.114+0000 7f70d2404740 1 -- 172.21.15.28:0/190924189 shutdown_connections 2024-04-21T08:52:27.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.114+0000 7f70d2404740 1 -- 172.21.15.28:0/190924189 wait complete. 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.114+0000 7f70d2404740 1 -- start start 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.114+0000 7f70d2404740 1 Processor -- start 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70d2404740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57a87c0 0x5582d57be5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70d2404740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a9170 0x5582d57beb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70d2404740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57b2270 0x5582d57bb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70d2404740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f70c81ab890 con 0x5582d57a87c0 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70d2404740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f70c81acd00 con 0x5582d57b2270 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70d2404740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f70c81abf60 con 0x5582d57a9170 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70d2404740 10 client.0 _ll_register_callbacks cb 0x7f70c40042d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70ccdac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57a87c0 0x5582d57be5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70ccdac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57a87c0 0x5582d57be5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56582/0 (socket says 172.21.15.28:56582) 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70ccdac640 1 -- 172.21.15.28:0/1119485873 learned_addr learned my addr 172.21.15.28:0/1119485873 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70ccdac640 1 --2- 172.21.15.28:0/1119485873 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57b2270 0x5582d57bb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70cd5ad640 1 --2- 172.21.15.28:0/1119485873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a9170 0x5582d57beb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70ccdac640 1 -- 172.21.15.28:0/1119485873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a9170 msgr2=0x5582d57beb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70ccdac640 1 --2- 172.21.15.28:0/1119485873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a9170 0x5582d57beb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70ccdac640 1 -- 172.21.15.28:0/1119485873 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57b2270 msgr2=0x5582d57bb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70ccdac640 1 --2- 172.21.15.28:0/1119485873 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57b2270 0x5582d57bb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.115+0000 7f70ccdac640 1 -- 172.21.15.28:0/1119485873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582d57bbb80 con 0x5582d57a87c0 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70ccdac640 1 --2- 172.21.15.28:0/1119485873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57a87c0 0x5582d57be5f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00a6c0 tx=0x7f70bc016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa3db9cb6beeaa25 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70b0ff9640 1 -- 172.21.15.28:0/1119485873 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc02e680 con 0x5582d57a87c0 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70b0ff9640 1 -- 172.21.15.28:0/1119485873 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f70bc037040 con 0x5582d57a87c0 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70d2404740 1 -- 172.21.15.28:0/1119485873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70c402d070 con 0x5582d57a87c0 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70b0ff9640 1 -- 172.21.15.28:0/1119485873 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f70bc02e820 con 0x5582d57a87c0 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70d2404740 10 client.16299 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70d2404740 1 -- 172.21.15.28:0/1119485873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5582d582d610 con 0x5582d57a87c0 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70b0ff9640 1 -- 172.21.15.28:0/1119485873 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70c402d070 con 0x5582d57a87c0 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70b0ff9640 1 -- 172.21.15.28:0/1119485873 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f70bc02ece0 con 0x5582d57a87c0 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70d2404740 10 client.16299 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70d2404740 10 client.16299 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70d2404740 20 client.16299 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70d2404740 1 --2- 172.21.15.28:0/1119485873 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5582d582db60 0x5582d582ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70d2404740 4 client.16299 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:52:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.116+0000 7f70d2404740 1 -- 172.21.15.28:0/1119485873 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5582d5714790 con 0x5582d582db60 2024-04-21T08:52:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.117+0000 7f70cd5ad640 1 --2- 172.21.15.28:0/1119485873 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5582d582db60 0x5582d582ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.117+0000 7f70b0ff9640 10 client.16299 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.117+0000 7f70cd5ad640 1 --2- 172.21.15.28:0/1119485873 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5582d582db60 0x5582d582ff80 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f30db58e45e23224 server_cookie=11b91bf358d002b8 in_seq=0 out_seq=0 2024-04-21T08:52:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.118+0000 7f70b0ff9640 1 -- 172.21.15.28:0/1119485873 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5582d5714790 con 0x5582d582db60 2024-04-21T08:52:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.118+0000 7f70b0ff9640 10 client.16299 handle_command_reply: tid=0 2024-04-21T08:52:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.118+0000 7f70d2404740 1 client.16299 shutdown 2024-04-21T08:52:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.118+0000 7f70d2404740 1 -- 172.21.15.28:0/1119485873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57a87c0 msgr2=0x5582d57be5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.118+0000 7f70d2404740 1 --2- 172.21.15.28:0/1119485873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57a87c0 0x5582d57be5f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00a6c0 tx=0x7f70bc016040 comp rx=0 tx=0).stop 2024-04-21T08:52:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.118+0000 7f70d2404740 1 -- 172.21.15.28:0/1119485873 shutdown_connections 2024-04-21T08:52:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.118+0000 7f70d2404740 1 --2- 172.21.15.28:0/1119485873 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5582d582db60 0x5582d582ff80 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.118+0000 7f70d2404740 1 --2- 172.21.15.28:0/1119485873 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5582d57b2270 0x5582d57bb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.118+0000 7f70d2404740 1 --2- 172.21.15.28:0/1119485873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5582d57a9170 0x5582d57beb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.118+0000 7f70b0ff9640 0 client.16299 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.118+0000 7f70d2404740 1 --2- 172.21.15.28:0/1119485873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582d57a87c0 0x5582d57be5f0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.118+0000 7f70d2404740 1 -- 172.21.15.28:0/1119485873 >> 172.21.15.28:0/1119485873 conn(0x5582d57a2fb0 msgr2=0x5582d57b65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.118+0000 7f70d2404740 1 -- 172.21.15.28:0/1119485873 shutdown_connections 2024-04-21T08:52:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.118+0000 7f70d2404740 1 -- 172.21.15.28:0/1119485873 wait complete. 2024-04-21T08:52:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.119+0000 7f70d2404740 20 client.16299 trim_cache size 0 max 16384 2024-04-21T08:52:27.123 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:52:27.123 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:52:27.123 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "26286e22-1c74-42a9-96e0-fefda5508b40", 2024-04-21T08:52:27.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:52:27.123 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:52:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.119+0000 7f70cf811640 1 -- 172.21.15.28:0/3198582653 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f70b40a3ed0 msgr2=0x7f70b40a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.119+0000 7f70cf811640 1 --2- 172.21.15.28:0/3198582653 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f70b40a3ed0 0x7f70b40a6390 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f70c00efd80 tx=0x7f70c00a90a0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.119+0000 7f70cf811640 1 -- 172.21.15.28:0/3198582653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c8107fd0 msgr2=0x7f70c8198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.119+0000 7f70cf811640 1 --2- 172.21.15.28:0/3198582653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c8107fd0 0x7f70c8198590 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f70c0004af0 tx=0x7f70c00346e0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.119+0000 7f70cf811640 1 -- 172.21.15.28:0/3198582653 shutdown_connections 2024-04-21T08:52:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.119+0000 7f70cf811640 1 --2- 172.21.15.28:0/3198582653 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f70b40a3ed0 0x7f70b40a6390 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.119+0000 7f70cf811640 1 --2- 172.21.15.28:0/3198582653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c8107fd0 0x7f70c8198590 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.119+0000 7f70cf811640 1 --2- 172.21.15.28:0/3198582653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70c806ace0 0x7f70c819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.119+0000 7f70cf811640 1 --2- 172.21.15.28:0/3198582653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70c806a360 0x7f70c819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.119+0000 7f70cf811640 1 -- 172.21.15.28:0/3198582653 >> 172.21.15.28:0/3198582653 conn(0x7f70c8077ef0 msgr2=0x7f70c80ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.119+0000 7f70cf811640 1 -- 172.21.15.28:0/3198582653 shutdown_connections 2024-04-21T08:52:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.119+0000 7f70cf811640 1 -- 172.21.15.28:0/3198582653 wait complete. 2024-04-21T08:52:27.132 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:52:27.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:52:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.307+0000 7f0046c8c640 1 -- 172.21.15.28:0/572252259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00401090a0 msgr2=0x7f0040109480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.307+0000 7f0046c8c640 1 --2- 172.21.15.28:0/572252259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00401090a0 0x7f0040109480 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f002c009900 tx=0x7f002c02bc50 comp rx=0 tx=0).stop 2024-04-21T08:52:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.308+0000 7f0046c8c640 1 -- 172.21.15.28:0/572252259 shutdown_connections 2024-04-21T08:52:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.308+0000 7f0046c8c640 1 --2- 172.21.15.28:0/572252259 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f00401150a0 0x7f0040117490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.308+0000 7f0046c8c640 1 --2- 172.21.15.28:0/572252259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00401099c0 0x7f0040114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.308+0000 7f0046c8c640 1 --2- 172.21.15.28:0/572252259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00401090a0 0x7f0040109480 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.308+0000 7f0046c8c640 1 -- 172.21.15.28:0/572252259 >> 172.21.15.28:0/572252259 conn(0x7f004006d070 msgr2=0x7f004006d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.308+0000 7f0046c8c640 1 -- 172.21.15.28:0/572252259 shutdown_connections 2024-04-21T08:52:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.308+0000 7f0046c8c640 1 -- 172.21.15.28:0/572252259 wait complete. 2024-04-21T08:52:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.309+0000 7f0046c8c640 1 Processor -- start 2024-04-21T08:52:27.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.309+0000 7f0046c8c640 1 -- start start 2024-04-21T08:52:27.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.309+0000 7f0046c8c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f00401090a0 0x7f004010a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.309+0000 7f0046c8c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00401099c0 0x7f004010af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.309+0000 7f0046c8c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00401150a0 0x7f004010b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.309+0000 7f0046c8c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f004011ab00 con 0x7f00401099c0 2024-04-21T08:52:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.309+0000 7f0046c8c640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f004011a980 con 0x7f00401150a0 2024-04-21T08:52:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.309+0000 7f0046c8c640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f004011ac80 con 0x7f00401090a0 2024-04-21T08:52:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.310+0000 7f003ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00401099c0 0x7f004010af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.310+0000 7f003ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00401099c0 0x7f004010af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56600/0 (socket says 172.21.15.28:56600) 2024-04-21T08:52:27.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.310+0000 7f003ffff640 1 -- 172.21.15.28:0/139089686 learned_addr learned my addr 172.21.15.28:0/139089686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.310+0000 7f003ffff640 1 -- 172.21.15.28:0/139089686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f00401090a0 msgr2=0x7f004010a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.310+0000 7f003ffff640 1 --2- 172.21.15.28:0/139089686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f00401090a0 0x7f004010a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.310+0000 7f003ffff640 1 -- 172.21.15.28:0/139089686 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00401150a0 msgr2=0x7f004010b440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:52:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.310+0000 7f003ffff640 1 --2- 172.21.15.28:0/139089686 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00401150a0 0x7f004010b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.310+0000 7f003ffff640 1 -- 172.21.15.28:0/139089686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00401b39c0 con 0x7f00401099c0 2024-04-21T08:52:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.311+0000 7f003ffff640 1 --2- 172.21.15.28:0/139089686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00401099c0 0x7f004010af00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f003400b450 tx=0x7f003400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.311+0000 7f003dffb640 1 -- 172.21.15.28:0/139089686 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0034015020 con 0x7f00401099c0 2024-04-21T08:52:27.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.311+0000 7f0046c8c640 1 -- 172.21.15.28:0/139089686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00401b3bf0 con 0x7f00401099c0 2024-04-21T08:52:27.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.312+0000 7f0046c8c640 1 -- 172.21.15.28:0/139089686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00401b4190 con 0x7f00401099c0 2024-04-21T08:52:27.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.312+0000 7f003dffb640 1 -- 172.21.15.28:0/139089686 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0034011d40 con 0x7f00401099c0 2024-04-21T08:52:27.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.312+0000 7f003dffb640 1 -- 172.21.15.28:0/139089686 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003400f340 con 0x7f00401099c0 2024-04-21T08:52:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.314+0000 7f0049522740 1 Processor -- start 2024-04-21T08:52:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.314+0000 7f0049522740 1 -- start start 2024-04-21T08:52:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.315+0000 7f0049522740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bccb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.315+0000 7f0049522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bcd4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.315+0000 7f0049522740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bcd4a80 0x56198bcd6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.315+0000 7f0049522740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56198bc301e0 con 0x56198bccbea0 2024-04-21T08:52:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.315+0000 7f0049522740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56198bbcd390 con 0x56198bccb580 2024-04-21T08:52:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.315+0000 7f0049522740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56198bc2c1a0 con 0x56198bcd4a80 2024-04-21T08:52:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.315+0000 7f0044a28640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bcd4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.315+0000 7f0044a28640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bcd4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56612/0 (socket says 172.21.15.28:56612) 2024-04-21T08:52:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.315+0000 7f0044a28640 1 -- 172.21.15.28:0/3039762507 learned_addr learned my addr 172.21.15.28:0/3039762507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.315+0000 7f0044a28640 1 -- 172.21.15.28:0/3039762507 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bcd4a80 msgr2=0x56198bcd6e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:52:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.315+0000 7f0044a28640 1 --2- 172.21.15.28:0/3039762507 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bcd4a80 0x56198bcd6e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.315+0000 7f0044a28640 1 -- 172.21.15.28:0/3039762507 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 msgr2=0x56198bccb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.315+0000 7f0045229640 1 --2- 172.21.15.28:0/3039762507 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bccb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.315+0000 7f0044a28640 1 --2- 172.21.15.28:0/3039762507 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bccb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.315+0000 7f0044a28640 1 -- 172.21.15.28:0/3039762507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56198ba0c7d0 con 0x56198bccbea0 2024-04-21T08:52:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.316+0000 7f0045229640 1 --2- 172.21.15.28:0/3039762507 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bccb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:52:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.316+0000 7f0044a28640 1 --2- 172.21.15.28:0/3039762507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bcd4540 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f002c009a20 tx=0x7f002c02c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e498d3e6c89ff0bf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.316+0000 7f002affd640 1 -- 172.21.15.28:0/3039762507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c041020 con 0x56198bccbea0 2024-04-21T08:52:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.316+0000 7f002affd640 1 -- 172.21.15.28:0/3039762507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f002c02ccb0 con 0x56198bccbea0 2024-04-21T08:52:27.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.316+0000 7f002affd640 1 -- 172.21.15.28:0/3039762507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c003cf0 con 0x56198bccbea0 2024-04-21T08:52:27.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.316+0000 7f003dffb640 1 -- 172.21.15.28:0/139089686 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7f0034018020 con 0x7f00401099c0 2024-04-21T08:52:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.317+0000 7f0049522740 1 -- 172.21.15.28:0/3039762507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 msgr2=0x56198bcd4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.317+0000 7f0049522740 1 --2- 172.21.15.28:0/3039762507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bcd4540 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f002c009a20 tx=0x7f002c02c720 comp rx=0 tx=0).stop 2024-04-21T08:52:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.317+0000 7f0049522740 1 -- 172.21.15.28:0/3039762507 shutdown_connections 2024-04-21T08:52:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.317+0000 7f0049522740 1 --2- 172.21.15.28:0/3039762507 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bcd4a80 0x56198bcd6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.317+0000 7f0049522740 1 --2- 172.21.15.28:0/3039762507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bcd4540 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.317+0000 7f0049522740 1 --2- 172.21.15.28:0/3039762507 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bccb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.317+0000 7f0049522740 1 -- 172.21.15.28:0/3039762507 >> 172.21.15.28:0/3039762507 conn(0x56198bcc5cb0 msgr2=0x56198bcc6090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.318+0000 7f003dffb640 1 --2- 172.21.15.28:0/139089686 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00300a3ed0 0x7f00300a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.318+0000 7f0044a28640 1 --2- 172.21.15.28:0/139089686 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00300a3ed0 0x7f00300a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.318+0000 7f003dffb640 1 -- 172.21.15.28:0/139089686 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f003409a2c0 con 0x7f00401099c0 2024-04-21T08:52:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.318+0000 7f0049522740 1 -- 172.21.15.28:0/3039762507 shutdown_connections 2024-04-21T08:52:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.318+0000 7f0044a28640 1 --2- 172.21.15.28:0/139089686 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00300a3ed0 0x7f00300a6390 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f002c0039b0 tx=0x7f002c005fd0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.319+0000 7f0049522740 1 -- 172.21.15.28:0/3039762507 wait complete. 2024-04-21T08:52:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.319+0000 7f0049522740 1 -- start start 2024-04-21T08:52:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.319+0000 7f0049522740 1 Processor -- start 2024-04-21T08:52:27.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.320+0000 7f0049522740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bcd1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.321+0000 7f0049522740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bccbea0 0x56198bcd1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.321+0000 7f0049522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bcd2480 0x56198bce0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.321+0000 7f0049522740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56198bcd7810 con 0x56198bcd2480 2024-04-21T08:52:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.321+0000 7f0049522740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56198bbe92f0 con 0x56198bccb580 2024-04-21T08:52:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.321+0000 7f0049522740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56198bcd7990 con 0x56198bccbea0 2024-04-21T08:52:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.321+0000 7f0049522740 10 client.0 _ll_register_callbacks cb 0x56198bbf6580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.321+0000 7f0045229640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bcd2480 0x56198bce0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.321+0000 7f0045229640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bcd2480 0x56198bce0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56618/0 (socket says 172.21.15.28:56618) 2024-04-21T08:52:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.321+0000 7f0045229640 1 -- 172.21.15.28:0/1104040760 learned_addr learned my addr 172.21.15.28:0/1104040760 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.322+0000 7f003ffff640 1 --2- 172.21.15.28:0/1104040760 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bccbea0 0x56198bcd1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.322+0000 7f0045229640 1 -- 172.21.15.28:0/1104040760 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bccbea0 msgr2=0x56198bcd1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.322+0000 7f0045229640 1 --2- 172.21.15.28:0/1104040760 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bccbea0 0x56198bcd1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.322+0000 7f0045229640 1 -- 172.21.15.28:0/1104040760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 msgr2=0x56198bcd1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.322+0000 7f0045229640 1 --2- 172.21.15.28:0/1104040760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bcd1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.322+0000 7f0045229640 1 -- 172.21.15.28:0/1104040760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56198b9423f0 con 0x56198bcd2480 2024-04-21T08:52:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.322+0000 7f003ffff640 1 --2- 172.21.15.28:0/1104040760 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bccbea0 0x56198bcd1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:52:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.322+0000 7f0045229640 1 --2- 172.21.15.28:0/1104040760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bcd2480 0x56198bce0d10 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f003800c920 tx=0x7f003800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b0fce2e292af79 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.322+0000 7f001ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.323+0000 7f001ffff640 1 -- 172.21.15.28:0/1104040760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0038008040 con 0x56198bcd2480 2024-04-21T08:52:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.323+0000 7f001ffff640 1 -- 172.21.15.28:0/1104040760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f003800a9e0 con 0x56198bcd2480 2024-04-21T08:52:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.323+0000 7f0049522740 1 -- 172.21.15.28:0/1104040760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f002c02ccb0 con 0x56198bcd2480 2024-04-21T08:52:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.323+0000 7f001ffff640 1 -- 172.21.15.28:0/1104040760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f003800abe0 con 0x56198bcd2480 2024-04-21T08:52:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.323+0000 7f0049522740 10 client.16311 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.323+0000 7f0049522740 1 -- 172.21.15.28:0/1104040760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56198b9020f0 con 0x56198bcd2480 2024-04-21T08:52:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.324+0000 7f001ffff640 1 -- 172.21.15.28:0/1104040760 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f003800ad80 con 0x56198bcd2480 2024-04-21T08:52:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.324+0000 7f001ffff640 1 -- 172.21.15.28:0/1104040760 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f003800e040 con 0x56198bcd2480 2024-04-21T08:52:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.324+0000 7f0049522740 10 client.16311 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.324+0000 7f0049522740 10 client.16311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.324+0000 7f0049522740 20 client.16311 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.324+0000 7f0049522740 1 --2- 172.21.15.28:0/1104040760 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56198bce4ed0 0x56198bce7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.324+0000 7f0049522740 4 client.16311 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:27.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.324+0000 7f0049522740 1 -- 172.21.15.28:0/1104040760 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f002c02ccb0 con 0x56198bce4ed0 2024-04-21T08:52:27.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.327+0000 7f0044a28640 1 --2- 172.21.15.28:0/1104040760 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56198bce4ed0 0x56198bce7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.330+0000 7f0044a28640 1 --2- 172.21.15.28:0/1104040760 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56198bce4ed0 0x56198bce7300 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e5cd92b49f97077 server_cookie=c620d7009386c35d in_seq=0 out_seq=0 2024-04-21T08:52:27.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.331+0000 7f001ffff640 10 client.16311 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.332+0000 7f001ffff640 1 -- 172.21.15.28:0/1104040760 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f002c02ccb0 con 0x56198bce4ed0 2024-04-21T08:52:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.332+0000 7f001ffff640 10 client.16311 handle_command_reply: tid=0 2024-04-21T08:52:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.333+0000 7f0049522740 1 client.16311 shutdown 2024-04-21T08:52:27.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.333+0000 7f0049522740 1 -- 172.21.15.28:0/1104040760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bcd2480 msgr2=0x56198bce0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.333+0000 7f0049522740 1 --2- 172.21.15.28:0/1104040760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bcd2480 0x56198bce0d10 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f003800c920 tx=0x7f003800cde0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.335+0000 7f0049522740 1 -- 172.21.15.28:0/1104040760 shutdown_connections 2024-04-21T08:52:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.335+0000 7f0049522740 1 --2- 172.21.15.28:0/1104040760 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56198bce4ed0 0x56198bce7300 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.335+0000 7f001ffff640 0 client.16311 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.335+0000 7f0049522740 1 --2- 172.21.15.28:0/1104040760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bcd2480 0x56198bce0d10 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.335+0000 7f0049522740 1 --2- 172.21.15.28:0/1104040760 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bccbea0 0x56198bcd1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.335+0000 7f0049522740 1 --2- 172.21.15.28:0/1104040760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bcd1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.336+0000 7f0049522740 1 -- 172.21.15.28:0/1104040760 >> 172.21.15.28:0/1104040760 conn(0x56198bcc5cb0 msgr2=0x56198bcc7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.337+0000 7f0049522740 1 -- 172.21.15.28:0/1104040760 shutdown_connections 2024-04-21T08:52:27.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.337+0000 7f0049522740 1 -- 172.21.15.28:0/1104040760 wait complete. 2024-04-21T08:52:27.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.337+0000 7f0049522740 20 client.16311 trim_cache size 0 max 16384 2024-04-21T08:52:27.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.411+0000 7f0049522740 1 Processor -- start 2024-04-21T08:52:27.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.412+0000 7f0049522740 1 -- start start 2024-04-21T08:52:27.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.413+0000 7f0049522740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bcecc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.413+0000 7f0049522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bced150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.413+0000 7f0049522740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bcd2480 0x56198bcdcc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.413+0000 7f0049522740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00240128d0 con 0x56198bccbea0 2024-04-21T08:52:27.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.413+0000 7f0049522740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f00240101a0 con 0x56198bccb580 2024-04-21T08:52:27.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.413+0000 7f0049522740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f002400dff0 con 0x56198bcd2480 2024-04-21T08:52:27.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.414+0000 7f003ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bced150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.414+0000 7f0044a28640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bcecc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.414+0000 7f003ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bced150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56622/0 (socket says 172.21.15.28:56622) 2024-04-21T08:52:27.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.414+0000 7f003ffff640 1 -- 172.21.15.28:0/4100476979 learned_addr learned my addr 172.21.15.28:0/4100476979 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:27.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.415+0000 7f003ffff640 1 -- 172.21.15.28:0/4100476979 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bcd2480 msgr2=0x56198bcdcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.415+0000 7f003ffff640 1 --2- 172.21.15.28:0/4100476979 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bcd2480 0x56198bcdcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.415+0000 7f003ffff640 1 -- 172.21.15.28:0/4100476979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 msgr2=0x56198bcecc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.415+0000 7f003ffff640 1 --2- 172.21.15.28:0/4100476979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bcecc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.415+0000 7f003ffff640 1 -- 172.21.15.28:0/4100476979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56198bcc5ae0 con 0x56198bccbea0 2024-04-21T08:52:27.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.415+0000 7f003ffff640 1 --2- 172.21.15.28:0/4100476979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bced150 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0034098bf0 tx=0x7f003409bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af86f03eca08711 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.416+0000 7f002a7fc640 1 -- 172.21.15.28:0/4100476979 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00340b4070 con 0x56198bccbea0 2024-04-21T08:52:27.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.416+0000 7f002a7fc640 1 -- 172.21.15.28:0/4100476979 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0034098890 con 0x56198bccbea0 2024-04-21T08:52:27.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.416+0000 7f002a7fc640 1 -- 172.21.15.28:0/4100476979 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003409b670 con 0x56198bccbea0 2024-04-21T08:52:27.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.417+0000 7f0049522740 1 -- 172.21.15.28:0/4100476979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 msgr2=0x56198bced150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.417+0000 7f0049522740 1 --2- 172.21.15.28:0/4100476979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bced150 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0034098bf0 tx=0x7f003409bcb0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.417+0000 7f0049522740 1 -- 172.21.15.28:0/4100476979 shutdown_connections 2024-04-21T08:52:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.417+0000 7f0049522740 1 --2- 172.21.15.28:0/4100476979 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bcd2480 0x56198bcdcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.417+0000 7f0049522740 1 --2- 172.21.15.28:0/4100476979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bced150 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.417+0000 7f0049522740 1 --2- 172.21.15.28:0/4100476979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bcecc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.417+0000 7f0049522740 1 -- 172.21.15.28:0/4100476979 >> 172.21.15.28:0/4100476979 conn(0x56198bcc5cb0 msgr2=0x56198bcd02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.417+0000 7f0049522740 1 -- 172.21.15.28:0/4100476979 shutdown_connections 2024-04-21T08:52:27.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.417+0000 7f0049522740 1 -- 172.21.15.28:0/4100476979 wait complete. 2024-04-21T08:52:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.418+0000 7f0049522740 1 -- start start 2024-04-21T08:52:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.418+0000 7f0049522740 1 Processor -- start 2024-04-21T08:52:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.418+0000 7f0049522740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bcda620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.418+0000 7f0049522740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bcdab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.418+0000 7f0049522740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bcd2480 0x56198bcdb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.419+0000 7f0049522740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f002400e660 con 0x56198bccbea0 2024-04-21T08:52:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.419+0000 7f0049522740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f002400fad0 con 0x56198bccb580 2024-04-21T08:52:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.419+0000 7f0049522740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f002400ed30 con 0x56198bcd2480 2024-04-21T08:52:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.419+0000 7f0049522740 10 client.0 _ll_register_callbacks cb 0x7f0038007570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.419+0000 7f003ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bcdab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.419+0000 7f0044a28640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bcda620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.419+0000 7f0044a28640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bcda620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38504/0 (socket says 172.21.15.28:38504) 2024-04-21T08:52:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.419+0000 7f0044a28640 1 -- 172.21.15.28:0/3920808904 learned_addr learned my addr 172.21.15.28:0/3920808904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.420+0000 7f0044a28640 1 -- 172.21.15.28:0/3920808904 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bcd2480 msgr2=0x56198bcdb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.420+0000 7f0044a28640 1 --2- 172.21.15.28:0/3920808904 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bcd2480 0x56198bcdb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.420+0000 7f0044a28640 1 -- 172.21.15.28:0/3920808904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 msgr2=0x56198bcdab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.420+0000 7f0044a28640 1 --2- 172.21.15.28:0/3920808904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bcdab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.420+0000 7f0044a28640 1 -- 172.21.15.28:0/3920808904 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56198bd4a490 con 0x56198bccb580 2024-04-21T08:52:27.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.422+0000 7f0044a28640 1 --2- 172.21.15.28:0/3920808904 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bcda620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f002c0095d0 tx=0x7f002c02c720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed765a68347d1fc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.422+0000 7f001ffff640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:52:27.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.423+0000 7f001ffff640 1 -- 172.21.15.28:0/3920808904 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c060060 con 0x56198bccb580 2024-04-21T08:52:27.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.423+0000 7f001ffff640 1 -- 172.21.15.28:0/3920808904 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f002c0039e0 con 0x56198bccb580 2024-04-21T08:52:27.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.423+0000 7f0049522740 1 -- 172.21.15.28:0/3920808904 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0034098890 con 0x56198bccb580 2024-04-21T08:52:27.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.427+0000 7f001ffff640 1 -- 172.21.15.28:0/3920808904 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f002c041030 con 0x56198bccb580 2024-04-21T08:52:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.429+0000 7f0049522740 10 client.25703 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.429+0000 7f0049522740 1 -- 172.21.15.28:0/3920808904 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56198bd4b700 con 0x56198bccb580 2024-04-21T08:52:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.430+0000 7f001ffff640 1 -- 172.21.15.28:0/3920808904 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0034098890 con 0x56198bccb580 2024-04-21T08:52:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.431+0000 7f001ffff640 1 -- 172.21.15.28:0/3920808904 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f002c05a480 con 0x56198bccb580 2024-04-21T08:52:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.431+0000 7f0049522740 10 client.25703 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.431+0000 7f0049522740 10 client.25703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:27.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.431+0000 7f0049522740 20 client.25703 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:27.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.431+0000 7f0049522740 1 --2- 172.21.15.28:0/3920808904 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56198bce4ed0 0x56198bd4db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.431+0000 7f0049522740 4 client.25703 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:52:27.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.431+0000 7f0049522740 1 -- 172.21.15.28:0/3920808904 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56198bc3dc00 con 0x56198bce4ed0 2024-04-21T08:52:27.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.433+0000 7f003ffff640 1 --2- 172.21.15.28:0/3920808904 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56198bce4ed0 0x56198bd4db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.441+0000 7f001ffff640 10 client.25703 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.441+0000 7f003ffff640 1 --2- 172.21.15.28:0/3920808904 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56198bce4ed0 0x56198bd4db30 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76b110186a6cf293 server_cookie=9cf1656964aeb986 in_seq=0 out_seq=0 2024-04-21T08:52:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.442+0000 7f001ffff640 1 -- 172.21.15.28:0/3920808904 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56198bc3dc00 con 0x56198bce4ed0 2024-04-21T08:52:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.442+0000 7f001ffff640 10 client.25703 handle_command_reply: tid=0 2024-04-21T08:52:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.442+0000 7f0049522740 1 client.25703 shutdown 2024-04-21T08:52:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.443+0000 7f0049522740 1 -- 172.21.15.28:0/3920808904 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 msgr2=0x56198bcda620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.443+0000 7f0049522740 1 --2- 172.21.15.28:0/3920808904 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bcda620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f002c0095d0 tx=0x7f002c02c720 comp rx=0 tx=0).stop 2024-04-21T08:52:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.443+0000 7f0049522740 1 -- 172.21.15.28:0/3920808904 shutdown_connections 2024-04-21T08:52:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.443+0000 7f0049522740 1 --2- 172.21.15.28:0/3920808904 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56198bce4ed0 0x56198bd4db30 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.443+0000 7f001ffff640 0 client.25703 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.443+0000 7f0049522740 1 --2- 172.21.15.28:0/3920808904 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56198bcd2480 0x56198bcdb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.443+0000 7f0049522740 1 --2- 172.21.15.28:0/3920808904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56198bccbea0 0x56198bcdab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.443+0000 7f0049522740 1 --2- 172.21.15.28:0/3920808904 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56198bccb580 0x56198bcda620 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.443+0000 7f0049522740 1 -- 172.21.15.28:0/3920808904 >> 172.21.15.28:0/3920808904 conn(0x56198bcc5cb0 msgr2=0x56198bcc7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.445+0000 7f0049522740 1 -- 172.21.15.28:0/3920808904 shutdown_connections 2024-04-21T08:52:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.445+0000 7f0049522740 1 -- 172.21.15.28:0/3920808904 wait complete. 2024-04-21T08:52:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.446+0000 7f0049522740 20 client.25703 trim_cache size 0 max 16384 2024-04-21T08:52:27.447 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:52:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:52:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:52:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "26286e22-1c74-42a9-96e0-fefda5508b40": { 2024-04-21T08:52:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:52:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "26286e22-1c74-42a9-96e0-fefda5508b40", 2024-04-21T08:52:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:52:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:52:27.448 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:52:27.448 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:52:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.449+0000 7f0046c8c640 1 -- 172.21.15.28:0/139089686 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00300a3ed0 msgr2=0x7f00300a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.449+0000 7f0046c8c640 1 --2- 172.21.15.28:0/139089686 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00300a3ed0 0x7f00300a6390 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f002c0039b0 tx=0x7f002c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.450+0000 7f0046c8c640 1 -- 172.21.15.28:0/139089686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00401099c0 msgr2=0x7f004010af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.450+0000 7f0046c8c640 1 --2- 172.21.15.28:0/139089686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00401099c0 0x7f004010af00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f003400b450 tx=0x7f003400b910 comp rx=0 tx=0).stop 2024-04-21T08:52:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.450+0000 7f0046c8c640 1 -- 172.21.15.28:0/139089686 shutdown_connections 2024-04-21T08:52:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.450+0000 7f0046c8c640 1 --2- 172.21.15.28:0/139089686 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00300a3ed0 0x7f00300a6390 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f002c0039b0 tx=0x7f002c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.450+0000 7f0046c8c640 1 --2- 172.21.15.28:0/139089686 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00401150a0 0x7f004010b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.450+0000 7f0046c8c640 1 --2- 172.21.15.28:0/139089686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00401099c0 0x7f004010af00 secure :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f003400b450 tx=0x7f003400b910 comp rx=0 tx=0).stop 2024-04-21T08:52:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.450+0000 7f0046c8c640 1 --2- 172.21.15.28:0/139089686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f00401090a0 0x7f004010a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.450+0000 7f0046c8c640 1 -- 172.21.15.28:0/139089686 >> 172.21.15.28:0/139089686 conn(0x7f004006d070 msgr2=0x7f0040070c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.451+0000 7f0046c8c640 1 -- 172.21.15.28:0/139089686 shutdown_connections 2024-04-21T08:52:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:27.451+0000 7f0046c8c640 1 -- 172.21.15.28:0/139089686 wait complete. 2024-04-21T08:52:27.472 INFO:tasks.cephfs.filesystem:scrub status for tag:26286e22-1c74-42a9-96e0-fefda5508b40 - {'path': '/', 'tag': '26286e22-1c74-42a9-96e0-fefda5508b40', 'options': 'recursive,force'} 2024-04-21T08:52:27.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:27 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:52:27.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:27 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:27.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:27 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:52:27.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:27 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:27.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:27 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:52:27.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:27 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:27.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:27 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:52:27.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:27 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:27.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:27 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:52:27.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:27 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:27.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:27 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:52:27.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:27 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:28 smithi152 ceph-mon[31401]: pgmap v41: 129 pgs: 129 active+clean; 4.7 GiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 404 MiB/s wr, 354 op/s 2024-04-21T08:52:28.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:28 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:52:28.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:28 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:28 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:52:28.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:28 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:52:28.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:28 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:28 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:52:28.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:28 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:28 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:52:28.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:28 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:28 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:52:28.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:28 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:28 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:28 smithi028 ceph-mon[28003]: pgmap v41: 129 pgs: 129 active+clean; 4.7 GiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 404 MiB/s wr, 354 op/s 2024-04-21T08:52:28.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:28 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:52:28.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:28 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:28 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:52:28.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:28 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:52:28.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:28 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:28 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:52:28.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:28 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:28 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:52:28.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:28 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:28 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:52:28.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:28 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:28 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:28 smithi171 ceph-mon[31358]: pgmap v41: 129 pgs: 129 active+clean; 4.7 GiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 404 MiB/s wr, 354 op/s 2024-04-21T08:52:28.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:28 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:52:28.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:28 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:28 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:52:28.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:28 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:52:28.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:28 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:28 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:52:28.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:28 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:28 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:52:28.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:28 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:28 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:52:28.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:28 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:28.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:28 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:30.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:30 smithi152 ceph-mon[31401]: pgmap v42: 129 pgs: 129 active+clean; 5.5 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 466 MiB/s wr, 243 op/s 2024-04-21T08:52:30.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:30 smithi028 ceph-mon[28003]: pgmap v42: 129 pgs: 129 active+clean; 5.5 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 466 MiB/s wr, 243 op/s 2024-04-21T08:52:30.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:30 smithi171 ceph-mon[31358]: pgmap v42: 129 pgs: 129 active+clean; 5.5 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 466 MiB/s wr, 243 op/s 2024-04-21T08:52:31.608 INFO:tasks.workunit.client.0.smithi028.stdout:fs setup took 12 secs 2024-04-21T08:52:32.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:32 smithi152 ceph-mon[31401]: pgmap v43: 129 pgs: 129 active+clean; 5.5 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 466 MiB/s wr, 243 op/s 2024-04-21T08:52:32.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:32 smithi028 ceph-mon[28003]: pgmap v43: 129 pgs: 129 active+clean; 5.5 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 466 MiB/s wr, 243 op/s 2024-04-21T08:52:32.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:32 smithi171 ceph-mon[31358]: pgmap v43: 129 pgs: 129 active+clean; 5.5 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 466 MiB/s wr, 243 op/s 2024-04-21T08:52:34.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:34 smithi152 ceph-mon[31401]: pgmap v44: 129 pgs: 129 active+clean; 9.8 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 840 MiB/s wr, 402 op/s 2024-04-21T08:52:34.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:34 smithi028 ceph-mon[28003]: pgmap v44: 129 pgs: 129 active+clean; 9.8 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 840 MiB/s wr, 402 op/s 2024-04-21T08:52:34.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:34 smithi171 ceph-mon[31358]: pgmap v44: 129 pgs: 129 active+clean; 9.8 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 840 MiB/s wr, 402 op/s 2024-04-21T08:52:35.358 INFO:tasks.workunit.client.0.smithi028.stdout:Syncing()...3 sec 2024-04-21T08:52:35.359 INFO:tasks.workunit.client.0.smithi028.stdout:Starting Actual Benchmark At: Sun Apr 21 08:52:35 2024 2024-04-21T08:52:35.359 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-21T08:52:36.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:36 smithi152 ceph-mon[31401]: pgmap v45: 129 pgs: 129 active+clean; 9.8 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 745 MiB/s wr, 310 op/s 2024-04-21T08:52:36.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:36 smithi028 ceph-mon[28003]: pgmap v45: 129 pgs: 129 active+clean; 9.8 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 745 MiB/s wr, 310 op/s 2024-04-21T08:52:36.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:36 smithi171 ceph-mon[31358]: pgmap v45: 129 pgs: 129 active+clean; 9.8 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 745 MiB/s wr, 310 op/s 2024-04-21T08:52:37.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:37 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:52:37.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:37 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:52:37.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:37 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:52:37.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:37 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:52:37.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:37 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:52:37.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:37 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:52:37.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:37 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:52:37.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:37 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:52:37.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:37 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:52:38.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:38 smithi152 ceph-mon[31401]: pgmap v46: 129 pgs: 129 active+clean; 9.9 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 754 MiB/s wr, 405 op/s 2024-04-21T08:52:38.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:38 smithi028 ceph-mon[28003]: pgmap v46: 129 pgs: 129 active+clean; 9.9 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 754 MiB/s wr, 405 op/s 2024-04-21T08:52:38.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:38 smithi171 ceph-mon[31358]: pgmap v46: 129 pgs: 129 active+clean; 9.9 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 754 MiB/s wr, 405 op/s 2024-04-21T08:52:39.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.906+0000 7faac42eb640 1 -- 172.21.15.28:0/1117637851 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faabc106170 msgr2=0x7faabc10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:39.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.906+0000 7faac42eb640 1 --2- 172.21.15.28:0/1117637851 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faabc106170 0x7faabc10dd40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faab4009ca0 tx=0x7faab402c100 comp rx=0 tx=0).stop 2024-04-21T08:52:39.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.906+0000 7faac42eb640 1 -- 172.21.15.28:0/1117637851 shutdown_connections 2024-04-21T08:52:39.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.906+0000 7faac42eb640 1 --2- 172.21.15.28:0/1117637851 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faabc106170 0x7faabc10dd40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.906+0000 7faac42eb640 1 --2- 172.21.15.28:0/1117637851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faabc101aa0 0x7faabc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.906+0000 7faac42eb640 1 --2- 172.21.15.28:0/1117637851 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faabc1010f0 0x7faabc1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.906+0000 7faac42eb640 1 -- 172.21.15.28:0/1117637851 >> 172.21.15.28:0/1117637851 conn(0x7faabc077fd0 msgr2=0x7faabc0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:39.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.907+0000 7faac42eb640 1 -- 172.21.15.28:0/1117637851 shutdown_connections 2024-04-21T08:52:39.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.907+0000 7faac42eb640 1 -- 172.21.15.28:0/1117637851 wait complete. 2024-04-21T08:52:39.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.908+0000 7faac42eb640 1 Processor -- start 2024-04-21T08:52:39.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.908+0000 7faac42eb640 1 -- start start 2024-04-21T08:52:39.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.908+0000 7faac42eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faabc1010f0 0x7faabc10dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:39.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.908+0000 7faac42eb640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faabc106170 0x7faabc10ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:39.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.908+0000 7faac42eb640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faabc10b240 0x7faabc10b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:39.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.908+0000 7faac42eb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faabc1143c0 con 0x7faabc1010f0 2024-04-21T08:52:39.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.908+0000 7faac42eb640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7faabc114240 con 0x7faabc106170 2024-04-21T08:52:39.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.908+0000 7faac42eb640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7faabc114540 con 0x7faabc10b240 2024-04-21T08:52:39.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.909+0000 7faac1886640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faabc106170 0x7faabc10ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:39.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.909+0000 7faac2087640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faabc1010f0 0x7faabc10dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:39.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.909+0000 7faac2087640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faabc1010f0 0x7faabc10dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38288/0 (socket says 172.21.15.28:38288) 2024-04-21T08:52:39.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.909+0000 7faac1886640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faabc106170 0x7faabc10ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:45426/0 (socket says 172.21.15.28:45426) 2024-04-21T08:52:39.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.909+0000 7faac2087640 1 -- 172.21.15.28:0/4179360344 learned_addr learned my addr 172.21.15.28:0/4179360344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:39.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.909+0000 7faac2888640 1 --2- 172.21.15.28:0/4179360344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faabc10b240 0x7faabc10b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:39.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.909+0000 7faac1886640 1 -- 172.21.15.28:0/4179360344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faabc10b240 msgr2=0x7faabc10b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:39.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.909+0000 7faac1886640 1 --2- 172.21.15.28:0/4179360344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faabc10b240 0x7faabc10b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.909+0000 7faac1886640 1 -- 172.21.15.28:0/4179360344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faabc1010f0 msgr2=0x7faabc10dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:39.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.909+0000 7faac1886640 1 --2- 172.21.15.28:0/4179360344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faabc1010f0 0x7faabc10dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.909+0000 7faac1886640 1 -- 172.21.15.28:0/4179360344 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faabc1a0b20 con 0x7faabc106170 2024-04-21T08:52:39.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.910+0000 7faac1886640 1 --2- 172.21.15.28:0/4179360344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faabc106170 0x7faabc10ad00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faaac00e9c0 tx=0x7faaac00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:39.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.910+0000 7faab37fe640 1 -- 172.21.15.28:0/4179360344 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac00ccc0 con 0x7faabc106170 2024-04-21T08:52:39.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.911+0000 7faab37fe640 1 -- 172.21.15.28:0/4179360344 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faaac0108e0 con 0x7faabc106170 2024-04-21T08:52:39.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.911+0000 7faab37fe640 1 -- 172.21.15.28:0/4179360344 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac017740 con 0x7faabc106170 2024-04-21T08:52:39.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.911+0000 7faac42eb640 1 -- 172.21.15.28:0/4179360344 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faabc1a0e10 con 0x7faabc106170 2024-04-21T08:52:39.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.911+0000 7faac42eb640 1 -- 172.21.15.28:0/4179360344 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faabc1a1310 con 0x7faabc106170 2024-04-21T08:52:39.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.915+0000 7faac6fb8740 1 Processor -- start 2024-04-21T08:52:39.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.916+0000 7faab37fe640 1 -- 172.21.15.28:0/4179360344 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 25) ==== 138930+0+0 (secure 0 0 0) 0x7faaac010430 con 0x7faabc106170 2024-04-21T08:52:39.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.917+0000 7faac6fb8740 1 -- start start 2024-04-21T08:52:39.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.917+0000 7faac6fb8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a0b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:39.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.917+0000 7faac6fb8740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a0bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:39.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.917+0000 7faac6fb8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a150e0 0x55c9c7a174d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:39.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.917+0000 7faac6fb8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c9c79708d0 con 0x55c9c7a150e0 2024-04-21T08:52:39.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.917+0000 7faac6fb8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c9c790de80 con 0x55c9c7a0c8d0 2024-04-21T08:52:39.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.917+0000 7faac6fb8740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55c9c796c890 con 0x55c9c7a0bb60 2024-04-21T08:52:39.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.917+0000 7faac1886640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a150e0 0x55c9c7a174d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:39.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.918+0000 7faac2087640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a0bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:39.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.918+0000 7faac2087640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a0bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:56952/0 (socket says 172.21.15.28:56952) 2024-04-21T08:52:39.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.918+0000 7faac2087640 1 -- 172.21.15.28:0/2982156746 learned_addr learned my addr 172.21.15.28:0/2982156746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:39.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.918+0000 7faac2888640 1 --2- 172.21.15.28:0/2982156746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a0b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:39.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.918+0000 7faab37fe640 1 --2- 172.21.15.28:0/4179360344 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faaa80a4190 0x7faaa80a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.918+0000 7faac2087640 1 -- 172.21.15.28:0/2982156746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0c8d0 msgr2=0x55c9c7a0b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.918+0000 7faac2087640 1 --2- 172.21.15.28:0/2982156746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a0b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.918+0000 7faac2087640 1 -- 172.21.15.28:0/2982156746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a150e0 msgr2=0x55c9c7a174d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.918+0000 7faac2087640 1 --2- 172.21.15.28:0/2982156746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a150e0 0x55c9c7a174d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.918+0000 7faac2087640 1 -- 172.21.15.28:0/2982156746 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9c792f320 con 0x55c9c7a0bb60 2024-04-21T08:52:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.918+0000 7faac2888640 1 --2- 172.21.15.28:0/2982156746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a0b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.918+0000 7faac2888640 1 --2- 172.21.15.28:0/4179360344 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faaa80a4190 0x7faaa80a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.919+0000 7faab37fe640 1 -- 172.21.15.28:0/4179360344 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7faaac0d0fd0 con 0x7faabc106170 2024-04-21T08:52:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.919+0000 7faac2087640 1 --2- 172.21.15.28:0/2982156746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a0bfe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faab800bda0 tx=0x7faab80107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb714091a2cb3442 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.919+0000 7faab0bfe640 1 -- 172.21.15.28:0/2982156746 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab801c680 con 0x55c9c7a0bb60 2024-04-21T08:52:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.919+0000 7faab0bfe640 1 -- 172.21.15.28:0/2982156746 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faab80118f0 con 0x55c9c7a0bb60 2024-04-21T08:52:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.919+0000 7faab0bfe640 1 -- 172.21.15.28:0/2982156746 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab801a800 con 0x55c9c7a0bb60 2024-04-21T08:52:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.921+0000 7faac6fb8740 1 -- 172.21.15.28:0/2982156746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a0bb60 msgr2=0x55c9c7a0bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.921+0000 7faac6fb8740 1 --2- 172.21.15.28:0/2982156746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a0bfe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faab800bda0 tx=0x7faab80107a0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.922+0000 7faac6fb8740 1 -- 172.21.15.28:0/2982156746 shutdown_connections 2024-04-21T08:52:39.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.922+0000 7faac6fb8740 1 --2- 172.21.15.28:0/2982156746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a150e0 0x55c9c7a174d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.922+0000 7faac6fb8740 1 --2- 172.21.15.28:0/2982156746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a0bfe0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.923+0000 7faac6fb8740 1 --2- 172.21.15.28:0/2982156746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a0b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.923+0000 7faac6fb8740 1 -- 172.21.15.28:0/2982156746 >> 172.21.15.28:0/2982156746 conn(0x55c9c79d2c90 msgr2=0x55c9c7a06850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:39.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.924+0000 7faac6fb8740 1 -- 172.21.15.28:0/2982156746 shutdown_connections 2024-04-21T08:52:39.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.924+0000 7faac6fb8740 1 -- 172.21.15.28:0/2982156746 wait complete. 2024-04-21T08:52:39.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.924+0000 7faac2888640 1 --2- 172.21.15.28:0/4179360344 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faaa80a4190 0x7faaa80a6650 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faab4005e30 tx=0x7faab402e040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:39.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.924+0000 7faac6fb8740 1 -- start start 2024-04-21T08:52:39.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.925+0000 7faac6fb8740 1 Processor -- start 2024-04-21T08:52:39.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.926+0000 7faac6fb8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a14a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:39.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.926+0000 7faac6fb8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a11e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:39.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.926+0000 7faac6fb8740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a150e0 0x55c9c7a12380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:39.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.926+0000 7faac6fb8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c9c7a17e10 con 0x55c9c7a0c8d0 2024-04-21T08:52:39.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.926+0000 7faac6fb8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c9c78f8eb0 con 0x55c9c7a0bb60 2024-04-21T08:52:39.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.926+0000 7faac6fb8740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55c9c7a17f90 con 0x55c9c7a150e0 2024-04-21T08:52:39.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.926+0000 7faac6fb8740 10 client.0 _ll_register_callbacks cb 0x55c9c7925b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:39.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.927+0000 7faac1886640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a11e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:39.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.927+0000 7faac2087640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a14a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:39.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.927+0000 7faac1886640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a11e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38296/0 (socket says 172.21.15.28:38296) 2024-04-21T08:52:39.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.927+0000 7faac1886640 1 -- 172.21.15.28:0/1830900090 learned_addr learned my addr 172.21.15.28:0/1830900090 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:39.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.929+0000 7faac2888640 1 --2- 172.21.15.28:0/1830900090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a150e0 0x55c9c7a12380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:39.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.929+0000 7faac2087640 1 -- 172.21.15.28:0/1830900090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a150e0 msgr2=0x55c9c7a12380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:39.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.929+0000 7faac2087640 1 --2- 172.21.15.28:0/1830900090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a150e0 0x55c9c7a12380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.929+0000 7faac2087640 1 -- 172.21.15.28:0/1830900090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a0c8d0 msgr2=0x55c9c7a11e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:39.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.929+0000 7faac2087640 1 --2- 172.21.15.28:0/1830900090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a11e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.929+0000 7faac2087640 1 -- 172.21.15.28:0/1830900090 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9c7683210 con 0x55c9c7a0bb60 2024-04-21T08:52:39.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.929+0000 7faac2888640 1 --2- 172.21.15.28:0/1830900090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a150e0 0x55c9c7a12380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:52:39.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.930+0000 7faac2087640 1 --2- 172.21.15.28:0/1830900090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a14a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faab800bd70 tx=0x7faab8011670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cdaed743ecc54f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:39.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.930+0000 7faaa5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:52:39.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.931+0000 7faaa5ffb640 1 -- 172.21.15.28:0/1830900090 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab8005040 con 0x55c9c7a0bb60 2024-04-21T08:52:39.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.931+0000 7faaa5ffb640 1 -- 172.21.15.28:0/1830900090 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7faab80051e0 con 0x55c9c7a0bb60 2024-04-21T08:52:39.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.931+0000 7faac6fb8740 1 -- 172.21.15.28:0/1830900090 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faab80118f0 con 0x55c9c7a0bb60 2024-04-21T08:52:39.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.931+0000 7faaa5ffb640 1 -- 172.21.15.28:0/1830900090 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faab8032050 con 0x55c9c7a0bb60 2024-04-21T08:52:39.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.931+0000 7faac6fb8740 10 client.25715 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:39.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.931+0000 7faac6fb8740 1 -- 172.21.15.28:0/1830900090 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c9c774f620 con 0x55c9c7a0bb60 2024-04-21T08:52:39.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.932+0000 7faaa5ffb640 1 -- 172.21.15.28:0/1830900090 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faab80118f0 con 0x55c9c7a0bb60 2024-04-21T08:52:39.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.932+0000 7faaa5ffb640 1 -- 172.21.15.28:0/1830900090 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7faab802d030 con 0x55c9c7a0bb60 2024-04-21T08:52:39.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.934+0000 7faac6fb8740 10 client.25715 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:39.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.934+0000 7faac6fb8740 10 client.25715 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:39.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.934+0000 7faac6fb8740 20 client.25715 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:39.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.934+0000 7faac6fb8740 1 --2- 172.21.15.28:0/1830900090 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c9c7a24ff0 0x55c9c7a273e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:39.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.934+0000 7faac6fb8740 4 client.25715 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:39.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.934+0000 7faac6fb8740 1 -- 172.21.15.28:0/1830900090 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9c7984880 con 0x55c9c7a24ff0 2024-04-21T08:52:39.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.935+0000 7faac1886640 1 --2- 172.21.15.28:0/1830900090 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c9c7a24ff0 0x55c9c7a273e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:39.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.939+0000 7faac1886640 1 --2- 172.21.15.28:0/1830900090 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c9c7a24ff0 0x55c9c7a273e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cb5ec332476698e server_cookie=904576803fba622d in_seq=0 out_seq=0 2024-04-21T08:52:39.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.939+0000 7faaa5ffb640 10 client.25715 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:39.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.941+0000 7faaa5ffb640 1 -- 172.21.15.28:0/1830900090 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c9c7984880 con 0x55c9c7a24ff0 2024-04-21T08:52:39.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.941+0000 7faaa5ffb640 10 client.25715 handle_command_reply: tid=0 2024-04-21T08:52:39.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.941+0000 7faac6fb8740 1 client.25715 shutdown 2024-04-21T08:52:39.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.949+0000 7faac6fb8740 1 -- 172.21.15.28:0/1830900090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0bb60 msgr2=0x55c9c7a14a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:39.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.949+0000 7faac6fb8740 1 --2- 172.21.15.28:0/1830900090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a14a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faab800bd70 tx=0x7faab8011670 comp rx=0 tx=0).stop 2024-04-21T08:52:39.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.950+0000 7faac6fb8740 1 -- 172.21.15.28:0/1830900090 shutdown_connections 2024-04-21T08:52:39.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.950+0000 7faac6fb8740 1 --2- 172.21.15.28:0/1830900090 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c9c7a24ff0 0x55c9c7a273e0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.950+0000 7faac6fb8740 1 --2- 172.21.15.28:0/1830900090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a150e0 0x55c9c7a12380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.950+0000 7faac6fb8740 1 --2- 172.21.15.28:0/1830900090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a11e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.950+0000 7faac6fb8740 1 --2- 172.21.15.28:0/1830900090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a14a00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.950+0000 7faac6fb8740 1 -- 172.21.15.28:0/1830900090 >> 172.21.15.28:0/1830900090 conn(0x55c9c79d2c90 msgr2=0x55c9c7a085a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:39.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.950+0000 7faaa5ffb640 0 client.25715 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:39.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.951+0000 7faac6fb8740 1 -- 172.21.15.28:0/1830900090 shutdown_connections 2024-04-21T08:52:39.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.952+0000 7faac6fb8740 1 -- 172.21.15.28:0/1830900090 wait complete. 2024-04-21T08:52:39.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.953+0000 7faac6fb8740 20 client.25715 trim_cache size 0 max 16384 2024-04-21T08:52:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.989+0000 7faac6fb8740 1 Processor -- start 2024-04-21T08:52:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.989+0000 7faac6fb8740 1 -- start start 2024-04-21T08:52:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.990+0000 7faac6fb8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a20d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.990+0000 7faac6fb8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a14530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:39.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.990+0000 7faac2087640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a20d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.990+0000 7faac2087640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a20d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:45462/0 (socket says 172.21.15.28:45462) 2024-04-21T08:52:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.990+0000 7faac2087640 1 -- 172.21.15.28:0/3984252422 learned_addr learned my addr 172.21.15.28:0/3984252422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.990+0000 7faac6fb8740 1 --2- 172.21.15.28:0/3984252422 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a150e0 0x55c9c7a2ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.991+0000 7faac6fb8740 1 -- 172.21.15.28:0/3984252422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faaa0012880 con 0x55c9c7a0c8d0 2024-04-21T08:52:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.991+0000 7faac6fb8740 1 -- 172.21.15.28:0/3984252422 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7faaa0010150 con 0x55c9c7a0bb60 2024-04-21T08:52:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.991+0000 7faac6fb8740 1 -- 172.21.15.28:0/3984252422 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7faaa000dfa0 con 0x55c9c7a150e0 2024-04-21T08:52:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.991+0000 7faac2888640 1 --2- 172.21.15.28:0/3984252422 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a150e0 0x55c9c7a2ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.991+0000 7faac1886640 1 --2- 172.21.15.28:0/3984252422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a14530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.992+0000 7faac2888640 1 -- 172.21.15.28:0/3984252422 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0bb60 msgr2=0x55c9c7a20d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:39.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.992+0000 7faac2888640 1 --2- 172.21.15.28:0/3984252422 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a20d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.992+0000 7faac2888640 1 -- 172.21.15.28:0/3984252422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a0c8d0 msgr2=0x55c9c7a14530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:39.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.992+0000 7faac2888640 1 --2- 172.21.15.28:0/3984252422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a14530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:39.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.992+0000 7faac2888640 1 -- 172.21.15.28:0/3984252422 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9c7a0ae30 con 0x55c9c7a150e0 2024-04-21T08:52:39.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.993+0000 7faac2087640 1 --2- 172.21.15.28:0/3984252422 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a20d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:52:39.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.993+0000 7faac2888640 1 --2- 172.21.15.28:0/3984252422 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a150e0 0x55c9c7a2ca70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faab4009bf0 tx=0x7faab4040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd0e929dbb405cda server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:39.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.995+0000 7faaa7fff640 1 -- 172.21.15.28:0/3984252422 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab4039040 con 0x55c9c7a150e0 2024-04-21T08:52:39.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.995+0000 7faaa7fff640 1 -- 172.21.15.28:0/3984252422 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faab4052070 con 0x55c9c7a150e0 2024-04-21T08:52:39.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.995+0000 7faaa7fff640 1 -- 172.21.15.28:0/3984252422 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab4034440 con 0x55c9c7a150e0 2024-04-21T08:52:39.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.999+0000 7faac6fb8740 1 -- 172.21.15.28:0/3984252422 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a150e0 msgr2=0x55c9c7a2ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:39.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:39.999+0000 7faac6fb8740 1 --2- 172.21.15.28:0/3984252422 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a150e0 0x55c9c7a2ca70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faab4009bf0 tx=0x7faab4040000 comp rx=0 tx=0).stop 2024-04-21T08:52:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.000+0000 7faac6fb8740 1 -- 172.21.15.28:0/3984252422 shutdown_connections 2024-04-21T08:52:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.000+0000 7faac6fb8740 1 --2- 172.21.15.28:0/3984252422 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a150e0 0x55c9c7a2ca70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.000+0000 7faac6fb8740 1 --2- 172.21.15.28:0/3984252422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a14530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.000+0000 7faac6fb8740 1 --2- 172.21.15.28:0/3984252422 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a20d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:40.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.000+0000 7faac6fb8740 1 -- 172.21.15.28:0/3984252422 >> 172.21.15.28:0/3984252422 conn(0x55c9c79d2c90 msgr2=0x55c9c7a11dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.001+0000 7faac6fb8740 1 -- 172.21.15.28:0/3984252422 shutdown_connections 2024-04-21T08:52:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.001+0000 7faac6fb8740 1 -- 172.21.15.28:0/3984252422 wait complete. 2024-04-21T08:52:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.002+0000 7faac6fb8740 1 -- start start 2024-04-21T08:52:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.002+0000 7faac6fb8740 1 Processor -- start 2024-04-21T08:52:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.002+0000 7faac6fb8740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a1d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.003+0000 7faac6fb8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a1d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.003+0000 7faac6fb8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a150e0 0x55c9c7a0e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.003+0000 7faac6fb8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faaa000e610 con 0x55c9c7a150e0 2024-04-21T08:52:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.003+0000 7faac2888640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a150e0 0x55c9c7a0e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.003+0000 7faac2888640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a150e0 0x55c9c7a0e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38320/0 (socket says 172.21.15.28:38320) 2024-04-21T08:52:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.003+0000 7faac2888640 1 -- 172.21.15.28:0/909207042 learned_addr learned my addr 172.21.15.28:0/909207042 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.003+0000 7faac6fb8740 1 -- 172.21.15.28:0/909207042 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7faaa000fa80 con 0x55c9c7a0c8d0 2024-04-21T08:52:40.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.003+0000 7faac6fb8740 1 -- 172.21.15.28:0/909207042 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7faaa000ece0 con 0x55c9c7a0bb60 2024-04-21T08:52:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.003+0000 7faac2087640 1 --2- 172.21.15.28:0/909207042 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a1d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.003+0000 7faac6fb8740 10 client.0 _ll_register_callbacks cb 0x7faaac005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.004+0000 7faac2888640 1 -- 172.21.15.28:0/909207042 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a0bb60 msgr2=0x55c9c7a1d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.004+0000 7faac2888640 1 --2- 172.21.15.28:0/909207042 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a1d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.004+0000 7faac2888640 1 -- 172.21.15.28:0/909207042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0c8d0 msgr2=0x55c9c7a1d790 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:52:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.004+0000 7faac2888640 1 --2- 172.21.15.28:0/909207042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a1d790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.004+0000 7faac2888640 1 -- 172.21.15.28:0/909207042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9c7a0eab0 con 0x55c9c7a150e0 2024-04-21T08:52:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.004+0000 7faac2888640 1 --2- 172.21.15.28:0/909207042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a150e0 0x55c9c7a0e2b0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7faab4032af0 tx=0x7faab4032fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452e6a533fd4828d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.005+0000 7faac2087640 1 --2- 172.21.15.28:0/909207042 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a1d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.005+0000 7faaa5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.005+0000 7faaa5ffb640 1 -- 172.21.15.28:0/909207042 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab4039040 con 0x55c9c7a150e0 2024-04-21T08:52:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.005+0000 7faaa5ffb640 1 -- 172.21.15.28:0/909207042 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7faab4003d80 con 0x55c9c7a150e0 2024-04-21T08:52:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.005+0000 7faac6fb8740 1 -- 172.21.15.28:0/909207042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faab4052070 con 0x55c9c7a150e0 2024-04-21T08:52:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.006+0000 7faaa5ffb640 1 -- 172.21.15.28:0/909207042 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faab4066050 con 0x55c9c7a150e0 2024-04-21T08:52:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.007+0000 7faac6fb8740 10 client.16332 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:40.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.007+0000 7faac6fb8740 1 -- 172.21.15.28:0/909207042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c9c7a8ad00 con 0x55c9c7a150e0 2024-04-21T08:52:40.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.010+0000 7faaa5ffb640 1 -- 172.21.15.28:0/909207042 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faab4052070 con 0x55c9c7a150e0 2024-04-21T08:52:40.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.010+0000 7faaa5ffb640 1 -- 172.21.15.28:0/909207042 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7faab403f040 con 0x55c9c7a150e0 2024-04-21T08:52:40.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.010+0000 7faac6fb8740 10 client.16332 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:40.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.010+0000 7faac6fb8740 10 client.16332 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:40.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.010+0000 7faac6fb8740 20 client.16332 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:40.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.010+0000 7faac6fb8740 1 --2- 172.21.15.28:0/909207042 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c9c7a8b470 0x55c9c7a8d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:40.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.010+0000 7faac6fb8740 4 client.16332 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:52:40.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.010+0000 7faac6fb8740 1 -- 172.21.15.28:0/909207042 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c9c79103d0 con 0x55c9c7a8b470 2024-04-21T08:52:40.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.011+0000 7faac2087640 1 --2- 172.21.15.28:0/909207042 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c9c7a8b470 0x55c9c7a8d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:40.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.013+0000 7faac2087640 1 --2- 172.21.15.28:0/909207042 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c9c7a8b470 0x55c9c7a8d890 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4afd4855493bbb16 server_cookie=b71f7b668b41e828 in_seq=0 out_seq=0 2024-04-21T08:52:40.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:40.015+0000 7faaa5ffb640 10 client.16332 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:40.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:40 smithi152 ceph-mon[31401]: pgmap v47: 129 pgs: 129 active+clean; 13 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 671 MiB/s wr, 1.66k op/s 2024-04-21T08:52:40.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:40 smithi028 ceph-mon[28003]: pgmap v47: 129 pgs: 129 active+clean; 13 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 671 MiB/s wr, 1.66k op/s 2024-04-21T08:52:40.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:40 smithi171 ceph-mon[31358]: pgmap v47: 129 pgs: 129 active+clean; 13 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 671 MiB/s wr, 1.66k op/s 2024-04-21T08:52:42.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:42 smithi152 ceph-mon[31401]: pgmap v48: 129 pgs: 129 active+clean; 13 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 609 MiB/s wr, 1.64k op/s 2024-04-21T08:52:42.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:42 smithi028 ceph-mon[28003]: pgmap v48: 129 pgs: 129 active+clean; 13 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 609 MiB/s wr, 1.64k op/s 2024-04-21T08:52:42.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:42 smithi171 ceph-mon[31358]: pgmap v48: 129 pgs: 129 active+clean; 13 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 609 MiB/s wr, 1.64k op/s 2024-04-21T08:52:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:52:43.461+0000 7faab37fe640 1 -- 172.21.15.28:0/4179360344 <== mon.2 v2:172.21.15.152:3300/0 6 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faaac099000 con 0x7faabc106170 2024-04-21T08:52:44.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:44 smithi152 ceph-mon[31401]: pgmap v49: 129 pgs: 129 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 659 MiB/s wr, 8.46k op/s 2024-04-21T08:52:44.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:44 smithi152 ceph-mon[31401]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-21T08:52:44.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:44 smithi028 ceph-mon[28003]: pgmap v49: 129 pgs: 129 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 659 MiB/s wr, 8.46k op/s 2024-04-21T08:52:44.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:44 smithi028 ceph-mon[28003]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-21T08:52:44.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:44 smithi171 ceph-mon[31358]: pgmap v49: 129 pgs: 129 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 659 MiB/s wr, 8.46k op/s 2024-04-21T08:52:44.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:44 smithi171 ceph-mon[31358]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-21T08:52:46.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:46 smithi152 ceph-mon[31401]: pgmap v50: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 289 MiB/s wr, 8.90k op/s 2024-04-21T08:52:46.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:46 smithi028 ceph-mon[28003]: pgmap v50: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 289 MiB/s wr, 8.90k op/s 2024-04-21T08:52:46.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:46 smithi171 ceph-mon[31358]: pgmap v50: 129 pgs: 129 active+clean; 13 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 289 MiB/s wr, 8.90k op/s 2024-04-21T08:52:48.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:48 smithi152 ceph-mon[31401]: pgmap v51: 129 pgs: 129 active+clean; 13 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 290 MiB/s wr, 9.14k op/s 2024-04-21T08:52:48.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:48 smithi028 ceph-mon[28003]: pgmap v51: 129 pgs: 129 active+clean; 13 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 290 MiB/s wr, 9.14k op/s 2024-04-21T08:52:48.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:48 smithi171 ceph-mon[31358]: pgmap v51: 129 pgs: 129 active+clean; 13 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 290 MiB/s wr, 9.14k op/s 2024-04-21T08:52:49.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:49 smithi152 ceph-mon[31401]: pgmap v52: 129 pgs: 129 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 336 MiB/s wr, 16.16k op/s 2024-04-21T08:52:49.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:49 smithi028 ceph-mon[28003]: pgmap v52: 129 pgs: 129 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 336 MiB/s wr, 16.16k op/s 2024-04-21T08:52:49.869 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:49 smithi171 ceph-mon[31358]: pgmap v52: 129 pgs: 129 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 336 MiB/s wr, 16.16k op/s 2024-04-21T08:52:52.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:51 smithi152 ceph-mon[31401]: pgmap v53: 129 pgs: 129 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 110 MiB/s wr, 14.77k op/s 2024-04-21T08:52:52.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:51 smithi028 ceph-mon[28003]: pgmap v53: 129 pgs: 129 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 110 MiB/s wr, 14.77k op/s 2024-04-21T08:52:52.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:51 smithi171 ceph-mon[31358]: pgmap v53: 129 pgs: 129 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 110 MiB/s wr, 14.77k op/s 2024-04-21T08:52:54.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:53 smithi152 ceph-mon[31401]: pgmap v54: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 170 MiB/s wr, 21.91k op/s 2024-04-21T08:52:54.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:53 smithi028 ceph-mon[28003]: pgmap v54: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 170 MiB/s wr, 21.91k op/s 2024-04-21T08:52:54.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:53 smithi171 ceph-mon[31358]: pgmap v54: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 170 MiB/s wr, 21.91k op/s 2024-04-21T08:52:56.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:55 smithi152 ceph-mon[31401]: pgmap v55: 129 pgs: 129 active+clean; 13 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 126 MiB/s wr, 15.75k op/s 2024-04-21T08:52:56.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:55 smithi028 ceph-mon[28003]: pgmap v55: 129 pgs: 129 active+clean; 13 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 126 MiB/s wr, 15.75k op/s 2024-04-21T08:52:56.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:55 smithi171 ceph-mon[31358]: pgmap v55: 129 pgs: 129 active+clean; 13 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 126 MiB/s wr, 15.75k op/s 2024-04-21T08:52:57.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:52:57.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:52:57.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.683+0000 7f5a7692b640 1 -- 172.21.15.28:0/1982886043 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a70074b70 msgr2=0x7f5a70112aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:57.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.683+0000 7f5a7692b640 1 --2- 172.21.15.28:0/1982886043 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a70074b70 0x7f5a70112aa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5a64009930 tx=0x7f5a6402bcf0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.685+0000 7f5a7692b640 1 -- 172.21.15.28:0/1982886043 shutdown_connections 2024-04-21T08:52:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.685+0000 7f5a7692b640 1 --2- 172.21.15.28:0/1982886043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a70112fe0 0x7f5a701153d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.685+0000 7f5a7692b640 1 --2- 172.21.15.28:0/1982886043 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a70074b70 0x7f5a70112aa0 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5a64009930 tx=0x7f5a6402bcf0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.685+0000 7f5a7692b640 1 --2- 172.21.15.28:0/1982886043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a70074250 0x7f5a70074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.685+0000 7f5a7692b640 1 -- 172.21.15.28:0/1982886043 >> 172.21.15.28:0/1982886043 conn(0x7f5a700793e0 msgr2=0x7f5a700797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:57.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.687+0000 7f5a7692b640 1 -- 172.21.15.28:0/1982886043 shutdown_connections 2024-04-21T08:52:57.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.690+0000 7f5a7692b640 1 -- 172.21.15.28:0/1982886043 wait complete. 2024-04-21T08:52:57.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.691+0000 7f5a7692b640 1 Processor -- start 2024-04-21T08:52:57.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.691+0000 7f5a7692b640 1 -- start start 2024-04-21T08:52:57.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.691+0000 7f5a7692b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a70074250 0x7f5a70112960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.691+0000 7f5a7692b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a70112fe0 0x7f5a7010c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.691+0000 7f5a7692b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a7010cf10 0x7f5a7010d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.691+0000 7f5a7692b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a70106b70 con 0x7f5a7010cf10 2024-04-21T08:52:57.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.691+0000 7f5a7692b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5a701069f0 con 0x7f5a70074250 2024-04-21T08:52:57.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.691+0000 7f5a7692b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5a70106cf0 con 0x7f5a70112fe0 2024-04-21T08:52:57.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.692+0000 7f5a74ec8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a7010cf10 0x7f5a7010d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.692+0000 7f5a74ec8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a7010cf10 0x7f5a7010d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46260/0 (socket says 172.21.15.28:46260) 2024-04-21T08:52:57.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.692+0000 7f5a74ec8640 1 -- 172.21.15.28:0/2330859361 learned_addr learned my addr 172.21.15.28:0/2330859361 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:57.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.693+0000 7f5a74ec8640 1 -- 172.21.15.28:0/2330859361 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a70112fe0 msgr2=0x7f5a7010c9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:52:57.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.693+0000 7f5a74ec8640 1 --2- 172.21.15.28:0/2330859361 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a70112fe0 0x7f5a7010c9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.693+0000 7f5a74ec8640 1 -- 172.21.15.28:0/2330859361 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a70074250 msgr2=0x7f5a70112960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:52:57.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.693+0000 7f5a74ec8640 1 --2- 172.21.15.28:0/2330859361 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a70074250 0x7f5a70112960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.693+0000 7f5a74ec8640 1 -- 172.21.15.28:0/2330859361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a701b3f60 con 0x7f5a7010cf10 2024-04-21T08:52:57.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.693+0000 7f5a74ec8640 1 --2- 172.21.15.28:0/2330859361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a7010cf10 0x7f5a7010d370 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800eb40 tx=0x7f5a6800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:57.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.696+0000 7f5a6d7fa640 1 -- 172.21.15.28:0/2330859361 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68004b20 con 0x7f5a7010cf10 2024-04-21T08:52:57.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.696+0000 7f5a6d7fa640 1 -- 172.21.15.28:0/2330859361 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a68004cc0 con 0x7f5a7010cf10 2024-04-21T08:52:57.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.696+0000 7f5a6d7fa640 1 -- 172.21.15.28:0/2330859361 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a680107b0 con 0x7f5a7010cf10 2024-04-21T08:52:57.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.697+0000 7f5a7692b640 1 -- 172.21.15.28:0/2330859361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a701b4250 con 0x7f5a7010cf10 2024-04-21T08:52:57.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.697+0000 7f5a7692b640 1 -- 172.21.15.28:0/2330859361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a701b4750 con 0x7f5a7010cf10 2024-04-21T08:52:57.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.703+0000 7f5a795f8740 1 Processor -- start 2024-04-21T08:52:57.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.707+0000 7f5a795f8740 1 -- start start 2024-04-21T08:52:57.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.707+0000 7f5a795f8740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11ba2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.707+0000 7f5a795f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11ba2ea0 0x559f11bab540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.707+0000 7f5a795f8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11baba80 0x559f11bade70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.707+0000 7f5a795f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559f11b071e0 con 0x559f11ba2ea0 2024-04-21T08:52:57.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.707+0000 7f5a795f8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x559f11aa4390 con 0x559f11baba80 2024-04-21T08:52:57.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.707+0000 7f5a795f8740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x559f11b031a0 con 0x559f11ba2580 2024-04-21T08:52:57.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.708+0000 7f5a6ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11ba2ea0 0x559f11bab540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.708+0000 7f5a6ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11ba2ea0 0x559f11bab540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46274/0 (socket says 172.21.15.28:46274) 2024-04-21T08:52:57.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.708+0000 7f5a6ffff640 1 -- 172.21.15.28:0/3052334071 learned_addr learned my addr 172.21.15.28:0/3052334071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:57.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.708+0000 7f5a6f7fe640 1 --2- 172.21.15.28:0/3052334071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11baba80 0x559f11bade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.708+0000 7f5a6d7fa640 1 -- 172.21.15.28:0/2330859361 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5a68005020 con 0x7f5a7010cf10 2024-04-21T08:52:57.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.708+0000 7f5a6f7fe640 1 --2- 172.21.15.28:0/3052334071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11ba2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.708+0000 7f5a6ffff640 1 -- 172.21.15.28:0/3052334071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 msgr2=0x559f11ba2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:57.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.708+0000 7f5a6ffff640 1 --2- 172.21.15.28:0/3052334071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11ba2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.708+0000 7f5a6ffff640 1 -- 172.21.15.28:0/3052334071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11baba80 msgr2=0x559f11bade70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:57.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.708+0000 7f5a6ffff640 1 --2- 172.21.15.28:0/3052334071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11baba80 0x559f11bade70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.708+0000 7f5a6ffff640 1 -- 172.21.15.28:0/3052334071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f118e37d0 con 0x559f11ba2ea0 2024-04-21T08:52:57.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.709+0000 7f5a6f7fe640 1 --2- 172.21.15.28:0/3052334071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11baba80 0x559f11bade70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:52:57.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.710+0000 7f5a6d7fa640 1 --2- 172.21.15.28:0/2330859361 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5a580a3f50 0x7f5a580a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.710+0000 7f5a6f7fe640 1 --2- 172.21.15.28:0/2330859361 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5a580a3f50 0x7f5a580a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.711+0000 7f5a6d7fa640 1 -- 172.21.15.28:0/2330859361 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f5a680d12a0 con 0x7f5a7010cf10 2024-04-21T08:52:57.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.711+0000 7f5a6ffff640 1 --2- 172.21.15.28:0/3052334071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11ba2ea0 0x559f11bab540 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009770 tx=0x7f5a60009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d004fffeb087595 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:57.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.712+0000 7f5a567fc640 1 -- 172.21.15.28:0/3052334071 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a60015020 con 0x559f11ba2ea0 2024-04-21T08:52:57.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.712+0000 7f5a567fc640 1 -- 172.21.15.28:0/3052334071 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a60011420 con 0x559f11ba2ea0 2024-04-21T08:52:57.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.713+0000 7f5a567fc640 1 -- 172.21.15.28:0/3052334071 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a600105c0 con 0x559f11ba2ea0 2024-04-21T08:52:57.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.713+0000 7f5a795f8740 1 -- 172.21.15.28:0/3052334071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11ba2ea0 msgr2=0x559f11bab540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:57.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.713+0000 7f5a795f8740 1 --2- 172.21.15.28:0/3052334071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11ba2ea0 0x559f11bab540 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f5a60009770 tx=0x7f5a60009c30 comp rx=0 tx=0).stop 2024-04-21T08:52:57.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.716+0000 7f5a795f8740 1 -- 172.21.15.28:0/3052334071 shutdown_connections 2024-04-21T08:52:57.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.716+0000 7f5a795f8740 1 --2- 172.21.15.28:0/3052334071 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11baba80 0x559f11bade70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.716+0000 7f5a795f8740 1 --2- 172.21.15.28:0/3052334071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11ba2ea0 0x559f11bab540 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.716+0000 7f5a795f8740 1 --2- 172.21.15.28:0/3052334071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11ba2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.716+0000 7f5a795f8740 1 -- 172.21.15.28:0/3052334071 >> 172.21.15.28:0/3052334071 conn(0x559f11b9ccb0 msgr2=0x559f11b9d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:57.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.716+0000 7f5a6f7fe640 1 --2- 172.21.15.28:0/2330859361 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5a580a3f50 0x7f5a580a6410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5a7010e190 tx=0x7f5a64000bc0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:57.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.719+0000 7f5a795f8740 1 -- 172.21.15.28:0/3052334071 shutdown_connections 2024-04-21T08:52:57.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.719+0000 7f5a795f8740 1 -- 172.21.15.28:0/3052334071 wait complete. 2024-04-21T08:52:57.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.720+0000 7f5a795f8740 1 -- start start 2024-04-21T08:52:57.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.720+0000 7f5a795f8740 1 Processor -- start 2024-04-21T08:52:57.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.721+0000 7f5a795f8740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11ba8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.721+0000 7f5a795f8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11ba2ea0 0x559f11ba8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.721+0000 7f5a795f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11baba80 0x559f11ba9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.721+0000 7f5a795f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559f11bae8c0 con 0x559f11baba80 2024-04-21T08:52:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.721+0000 7f5a795f8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x559f11ac02f0 con 0x559f11ba2ea0 2024-04-21T08:52:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.721+0000 7f5a795f8740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x559f11baea40 con 0x559f11ba2580 2024-04-21T08:52:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.721+0000 7f5a795f8740 10 client.0 _ll_register_callbacks cb 0x559f11acd580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.722+0000 7f5a74ec8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11baba80 0x559f11ba9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.722+0000 7f5a74ec8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11baba80 0x559f11ba9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46278/0 (socket says 172.21.15.28:46278) 2024-04-21T08:52:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.722+0000 7f5a74ec8640 1 -- 172.21.15.28:0/3931511267 learned_addr learned my addr 172.21.15.28:0/3931511267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:52:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.722+0000 7f5a6f7fe640 1 --2- 172.21.15.28:0/3931511267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11ba2ea0 0x559f11ba8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.723+0000 7f5a6ffff640 1 --2- 172.21.15.28:0/3931511267 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11ba8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.726+0000 7f5a74ec8640 1 -- 172.21.15.28:0/3931511267 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 msgr2=0x559f11ba8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:57.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.726+0000 7f5a74ec8640 1 --2- 172.21.15.28:0/3931511267 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11ba8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.726+0000 7f5a74ec8640 1 -- 172.21.15.28:0/3931511267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11ba2ea0 msgr2=0x559f11ba8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:57.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.726+0000 7f5a74ec8640 1 --2- 172.21.15.28:0/3931511267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11ba2ea0 0x559f11ba8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:57.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.726+0000 7f5a74ec8640 1 -- 172.21.15.28:0/3931511267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f118193f0 con 0x559f11baba80 2024-04-21T08:52:57.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.726+0000 7f5a74ec8640 1 --2- 172.21.15.28:0/3931511267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11baba80 0x559f11ba9410 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800eb10 tx=0x7f5a6809d600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a383796b8b46e53 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:57.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.727+0000 7f5a4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:52:57.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.727+0000 7f5a4f7fe640 1 -- 172.21.15.28:0/3931511267 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6809f070 con 0x559f11baba80 2024-04-21T08:52:57.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.728+0000 7f5a4f7fe640 1 -- 172.21.15.28:0/3931511267 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f5a680b6260 con 0x559f11baba80 2024-04-21T08:52:57.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.729+0000 7f5a795f8740 1 -- 172.21.15.28:0/3931511267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a60011420 con 0x559f11baba80 2024-04-21T08:52:57.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.730+0000 7f5a4f7fe640 1 -- 172.21.15.28:0/3931511267 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5a680bb050 con 0x559f11baba80 2024-04-21T08:52:57.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.730+0000 7f5a795f8740 10 client.16341 fetch_fsmap learned FSMap version 26 2024-04-21T08:52:57.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.731+0000 7f5a795f8740 1 -- 172.21.15.28:0/3931511267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559f117d90f0 con 0x559f11baba80 2024-04-21T08:52:57.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.733+0000 7f5a4f7fe640 1 -- 172.21.15.28:0/3931511267 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a60011420 con 0x559f11baba80 2024-04-21T08:52:57.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.733+0000 7f5a4f7fe640 1 -- 172.21.15.28:0/3931511267 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5a680a5620 con 0x559f11baba80 2024-04-21T08:52:57.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.734+0000 7f5a795f8740 10 client.16341 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:52:57.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.734+0000 7f5a795f8740 10 client.16341 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:57.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.734+0000 7f5a795f8740 20 client.16341 populate_metadata read hostname 'smithi028' 2024-04-21T08:52:57.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.734+0000 7f5a795f8740 1 --2- 172.21.15.28:0/3931511267 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559f11bbba20 0x559f11bbde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:57.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.734+0000 7f5a795f8740 4 client.16341 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:57.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.734+0000 7f5a795f8740 1 -- 172.21.15.28:0/3931511267 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f11b14c00 con 0x559f11bbba20 2024-04-21T08:52:57.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.737+0000 7f5a6ffff640 1 --2- 172.21.15.28:0/3931511267 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559f11bbba20 0x559f11bbde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:57.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.744+0000 7f5a6ffff640 1 --2- 172.21.15.28:0/3931511267 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559f11bbba20 0x559f11bbde80 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=554866ae337a5805 server_cookie=db1eda4ba0a36f39 in_seq=0 out_seq=0 2024-04-21T08:52:57.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:52:57.744+0000 7f5a4f7fe640 10 client.16341 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:52:58.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:57 smithi152 ceph-mon[31401]: pgmap v56: 129 pgs: 129 active+clean; 13 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 124 MiB/s wr, 15.44k op/s 2024-04-21T08:52:58.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:57 smithi028 ceph-mon[28003]: pgmap v56: 129 pgs: 129 active+clean; 13 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 124 MiB/s wr, 15.44k op/s 2024-04-21T08:52:58.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:57 smithi171 ceph-mon[31358]: pgmap v56: 129 pgs: 129 active+clean; 13 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 124 MiB/s wr, 15.44k op/s 2024-04-21T08:53:00.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:52:59 smithi152 ceph-mon[31401]: pgmap v57: 129 pgs: 129 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 180 MiB/s wr, 22.30k op/s 2024-04-21T08:53:00.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:52:59 smithi028 ceph-mon[28003]: pgmap v57: 129 pgs: 129 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 180 MiB/s wr, 22.30k op/s 2024-04-21T08:53:00.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:52:59 smithi171 ceph-mon[31358]: pgmap v57: 129 pgs: 129 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 180 MiB/s wr, 22.30k op/s 2024-04-21T08:53:02.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:01 smithi152 ceph-mon[31401]: pgmap v58: 129 pgs: 129 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 125 MiB/s wr, 15.19k op/s 2024-04-21T08:53:02.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:01 smithi028 ceph-mon[28003]: pgmap v58: 129 pgs: 129 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 125 MiB/s wr, 15.19k op/s 2024-04-21T08:53:02.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:01 smithi171 ceph-mon[31358]: pgmap v58: 129 pgs: 129 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 125 MiB/s wr, 15.19k op/s 2024-04-21T08:53:03.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.190+0000 7faaa5ffb640 1 -- 172.21.15.28:0/909207042 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c9c79103d0 con 0x55c9c7a8b470 2024-04-21T08:53:03.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.190+0000 7faaa5ffb640 10 client.16332 handle_command_reply: tid=0 2024-04-21T08:53:03.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.190+0000 7faac6fb8740 1 client.16332 shutdown 2024-04-21T08:53:03.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.191+0000 7faac6fb8740 1 -- 172.21.15.28:0/909207042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a150e0 msgr2=0x55c9c7a0e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.191+0000 7faac6fb8740 1 --2- 172.21.15.28:0/909207042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a150e0 0x55c9c7a0e2b0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7faab4032af0 tx=0x7faab4032fc0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.191+0000 7f5a4f7fe640 1 -- 172.21.15.28:0/3931511267 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f11b14c00 con 0x559f11bbba20 2024-04-21T08:53:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.191+0000 7f5a4f7fe640 10 client.16341 handle_command_reply: tid=0 2024-04-21T08:53:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.191+0000 7f5a795f8740 1 client.16341 shutdown 2024-04-21T08:53:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.191+0000 7faac6fb8740 1 -- 172.21.15.28:0/909207042 shutdown_connections 2024-04-21T08:53:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.191+0000 7faac6fb8740 1 --2- 172.21.15.28:0/909207042 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c9c7a8b470 0x55c9c7a8d890 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.191+0000 7faac6fb8740 1 --2- 172.21.15.28:0/909207042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c9c7a150e0 0x55c9c7a0e2b0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.191+0000 7faac6fb8740 1 --2- 172.21.15.28:0/909207042 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c9c7a0c8d0 0x55c9c7a1d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.191+0000 7faac6fb8740 1 --2- 172.21.15.28:0/909207042 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c9c7a0bb60 0x55c9c7a1d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.191+0000 7faac6fb8740 1 -- 172.21.15.28:0/909207042 >> 172.21.15.28:0/909207042 conn(0x55c9c79d2c90 msgr2=0x55c9c7a06380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.191+0000 7faaa5ffb640 0 client.16332 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:03.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.191+0000 7faac6fb8740 1 -- 172.21.15.28:0/909207042 shutdown_connections 2024-04-21T08:53:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.191+0000 7f5a795f8740 1 -- 172.21.15.28:0/3931511267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11baba80 msgr2=0x559f11ba9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.191+0000 7f5a795f8740 1 --2- 172.21.15.28:0/3931511267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11baba80 0x559f11ba9410 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800eb10 tx=0x7f5a6809d600 comp rx=0 tx=0).stop 2024-04-21T08:53:03.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.191+0000 7faac6fb8740 1 -- 172.21.15.28:0/909207042 wait complete. 2024-04-21T08:53:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.191+0000 7f5a795f8740 1 -- 172.21.15.28:0/3931511267 shutdown_connections 2024-04-21T08:53:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.191+0000 7f5a795f8740 1 --2- 172.21.15.28:0/3931511267 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559f11bbba20 0x559f11bbde80 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.191+0000 7f5a795f8740 1 --2- 172.21.15.28:0/3931511267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11baba80 0x559f11ba9410 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.191+0000 7f5a795f8740 1 --2- 172.21.15.28:0/3931511267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11ba2ea0 0x559f11ba8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.191+0000 7f5a795f8740 1 --2- 172.21.15.28:0/3931511267 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11ba8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.191+0000 7f5a795f8740 1 -- 172.21.15.28:0/3931511267 >> 172.21.15.28:0/3931511267 conn(0x559f11b9ccb0 msgr2=0x559f11b9ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.191+0000 7f5a4f7fe640 0 client.16341 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:03.194 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.192+0000 7faac6fb8740 20 client.16332 trim_cache size 0 max 16384 2024-04-21T08:53:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.192+0000 7f5a795f8740 1 -- 172.21.15.28:0/3931511267 shutdown_connections 2024-04-21T08:53:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.192+0000 7f5a795f8740 1 -- 172.21.15.28:0/3931511267 wait complete. 2024-04-21T08:53:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.192+0000 7f5a795f8740 20 client.16341 trim_cache size 0 max 16384 2024-04-21T08:53:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:53:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:53:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:53:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:53:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.199+0000 7faac42eb640 1 -- 172.21.15.28:0/4179360344 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faaa80a4190 msgr2=0x7faaa80a6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.199+0000 7faac42eb640 1 --2- 172.21.15.28:0/4179360344 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faaa80a4190 0x7faaa80a6650 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faab4005e30 tx=0x7faab402e040 comp rx=0 tx=0).stop 2024-04-21T08:53:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.199+0000 7faac42eb640 1 -- 172.21.15.28:0/4179360344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faabc106170 msgr2=0x7faabc10ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.199+0000 7faac42eb640 1 --2- 172.21.15.28:0/4179360344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faabc106170 0x7faabc10ad00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faaac00e9c0 tx=0x7faaac00ee80 comp rx=0 tx=0).stop 2024-04-21T08:53:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.200+0000 7faac42eb640 1 -- 172.21.15.28:0/4179360344 shutdown_connections 2024-04-21T08:53:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.200+0000 7faac42eb640 1 --2- 172.21.15.28:0/4179360344 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faaa80a4190 0x7faaa80a6650 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.200+0000 7faac42eb640 1 --2- 172.21.15.28:0/4179360344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faabc10b240 0x7faabc10b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.200+0000 7faac42eb640 1 --2- 172.21.15.28:0/4179360344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faabc106170 0x7faabc10ad00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.200+0000 7faac42eb640 1 --2- 172.21.15.28:0/4179360344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faabc1010f0 0x7faabc10dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.200+0000 7faac42eb640 1 -- 172.21.15.28:0/4179360344 >> 172.21.15.28:0/4179360344 conn(0x7faabc077fd0 msgr2=0x7faabc0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.200+0000 7faac42eb640 1 -- 172.21.15.28:0/4179360344 shutdown_connections 2024-04-21T08:53:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:03.200+0000 7faac42eb640 1 -- 172.21.15.28:0/4179360344 wait complete. 2024-04-21T08:53:03.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.231+0000 7f5a795f8740 1 Processor -- start 2024-04-21T08:53:03.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.231+0000 7f5a795f8740 1 -- start start 2024-04-21T08:53:03.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.231+0000 7f5a795f8740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11bb7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.232+0000 7f5a795f8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11ba2ea0 0x559f11ba9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.232+0000 7f5a795f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11baba80 0x559f11b9bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.232+0000 7f5a6ffff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11bb7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.232+0000 7f5a6ffff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11bb7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:48622/0 (socket says 172.21.15.28:48622) 2024-04-21T08:53:03.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.232+0000 7f5a6f7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11ba2ea0 0x559f11ba9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.232+0000 7f5a6f7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11ba2ea0 0x559f11ba9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:53320/0 (socket says 172.21.15.28:53320) 2024-04-21T08:53:03.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.232+0000 7f5a6ffff640 1 -- 172.21.15.28:0/651288330 learned_addr learned my addr 172.21.15.28:0/651288330 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:03.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.232+0000 7f5a795f8740 1 -- 172.21.15.28:0/651288330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c0127b0 con 0x559f11baba80 2024-04-21T08:53:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.232+0000 7f5a795f8740 1 -- 172.21.15.28:0/651288330 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c010080 con 0x559f11ba2ea0 2024-04-21T08:53:03.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.232+0000 7f5a795f8740 1 -- 172.21.15.28:0/651288330 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c00ded0 con 0x559f11ba2580 2024-04-21T08:53:03.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.234+0000 7f5a74ec8640 1 --2- 172.21.15.28:0/651288330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11baba80 0x559f11b9bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.234+0000 7f5a6f7fe640 1 -- 172.21.15.28:0/651288330 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 msgr2=0x559f11bb7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:03.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.234+0000 7f5a6f7fe640 1 --2- 172.21.15.28:0/651288330 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11bb7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.234+0000 7f5a6f7fe640 1 -- 172.21.15.28:0/651288330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11baba80 msgr2=0x559f11b9bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:03.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.234+0000 7f5a6f7fe640 1 --2- 172.21.15.28:0/651288330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11baba80 0x559f11b9bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.234+0000 7f5a6f7fe640 1 -- 172.21.15.28:0/651288330 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f11b9cae0 con 0x559f11ba2ea0 2024-04-21T08:53:03.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.234+0000 7f5a6ffff640 1 --2- 172.21.15.28:0/651288330 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11bb7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:53:03.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.235+0000 7f5a6f7fe640 1 --2- 172.21.15.28:0/651288330 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11ba2ea0 0x559f11ba9230 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5a64008000 tx=0x7f5a6403c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26f7389e456fe2f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.237+0000 7f5a55ffb640 1 -- 172.21.15.28:0/651288330 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a64052070 con 0x559f11ba2ea0 2024-04-21T08:53:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.237+0000 7f5a55ffb640 1 -- 172.21.15.28:0/651288330 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a640055b0 con 0x559f11ba2ea0 2024-04-21T08:53:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.237+0000 7f5a55ffb640 1 -- 172.21.15.28:0/651288330 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a640426c0 con 0x559f11ba2ea0 2024-04-21T08:53:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.238+0000 7f5a795f8740 1 -- 172.21.15.28:0/651288330 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11ba2ea0 msgr2=0x559f11ba9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.238+0000 7f5a795f8740 1 --2- 172.21.15.28:0/651288330 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11ba2ea0 0x559f11ba9230 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5a64008000 tx=0x7f5a6403c000 comp rx=0 tx=0).stop 2024-04-21T08:53:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.239+0000 7f5a795f8740 1 -- 172.21.15.28:0/651288330 shutdown_connections 2024-04-21T08:53:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.239+0000 7f5a795f8740 1 --2- 172.21.15.28:0/651288330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11baba80 0x559f11b9bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.239+0000 7f5a795f8740 1 --2- 172.21.15.28:0/651288330 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11ba2ea0 0x559f11ba9230 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.239+0000 7f5a795f8740 1 --2- 172.21.15.28:0/651288330 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11bb7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.240+0000 7f5a795f8740 1 -- 172.21.15.28:0/651288330 >> 172.21.15.28:0/651288330 conn(0x559f11b9ccb0 msgr2=0x559f11ba78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:03.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.242+0000 7f5a795f8740 1 -- 172.21.15.28:0/651288330 shutdown_connections 2024-04-21T08:53:03.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.243+0000 7f5a795f8740 1 -- 172.21.15.28:0/651288330 wait complete. 2024-04-21T08:53:03.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.243+0000 7f5a795f8740 1 -- start start 2024-04-21T08:53:03.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.244+0000 7f5a795f8740 1 Processor -- start 2024-04-21T08:53:03.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.244+0000 7f5a795f8740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11bb4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.245+0000 7f5a795f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11ba2ea0 0x559f11ba5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.245+0000 7f5a795f8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11baba80 0x559f11ba5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.245+0000 7f5a795f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c00e540 con 0x559f11ba2ea0 2024-04-21T08:53:03.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.245+0000 7f5a6f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11ba2ea0 0x559f11ba5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.245+0000 7f5a795f8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c00f9b0 con 0x559f11baba80 2024-04-21T08:53:03.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.245+0000 7f5a795f8740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c00ec10 con 0x559f11ba2580 2024-04-21T08:53:03.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.245+0000 7f5a6f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11ba2ea0 0x559f11ba5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46294/0 (socket says 172.21.15.28:46294) 2024-04-21T08:53:03.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.245+0000 7f5a6f7fe640 1 -- 172.21.15.28:0/1403355908 learned_addr learned my addr 172.21.15.28:0/1403355908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:03.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.245+0000 7f5a795f8740 10 client.0 _ll_register_callbacks cb 0x7f5a68004e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:03.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.246+0000 7f5a6f7fe640 1 -- 172.21.15.28:0/1403355908 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 msgr2=0x559f11bb4220 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:03.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.246+0000 7f5a6f7fe640 1 --2- 172.21.15.28:0/1403355908 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11bb4220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.246+0000 7f5a6f7fe640 1 -- 172.21.15.28:0/1403355908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11baba80 msgr2=0x559f11ba5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.246+0000 7f5a6f7fe640 1 --2- 172.21.15.28:0/1403355908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11baba80 0x559f11ba5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.246+0000 7f5a6f7fe640 1 -- 172.21.15.28:0/1403355908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f11ba61c0 con 0x559f11ba2ea0 2024-04-21T08:53:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.247+0000 7f5a6f7fe640 1 --2- 172.21.15.28:0/1403355908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11ba2ea0 0x559f11ba5410 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5a64036040 tx=0x7f5a640048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44f698209f21e83b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.247+0000 7f5a4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.248+0000 7f5a4f7fe640 1 -- 172.21.15.28:0/1403355908 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6405e070 con 0x559f11ba2ea0 2024-04-21T08:53:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.248+0000 7f5a4f7fe640 1 -- 172.21.15.28:0/1403355908 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f5a64004020 con 0x559f11ba2ea0 2024-04-21T08:53:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.248+0000 7f5a795f8740 1 -- 172.21.15.28:0/1403355908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a640055b0 con 0x559f11ba2ea0 2024-04-21T08:53:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.248+0000 7f5a4f7fe640 1 -- 172.21.15.28:0/1403355908 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5a64067050 con 0x559f11ba2ea0 2024-04-21T08:53:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.249+0000 7f5a795f8740 10 client.16347 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.249+0000 7f5a795f8740 1 -- 172.21.15.28:0/1403355908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559f11c220a0 con 0x559f11ba2ea0 2024-04-21T08:53:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.250+0000 7f5a4f7fe640 1 -- 172.21.15.28:0/1403355908 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a640055b0 con 0x559f11ba2ea0 2024-04-21T08:53:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.250+0000 7f5a4f7fe640 1 -- 172.21.15.28:0/1403355908 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5a64061020 con 0x559f11ba2ea0 2024-04-21T08:53:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.250+0000 7f5a795f8740 10 client.16347 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.250+0000 7f5a795f8740 10 client.16347 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.250+0000 7f5a795f8740 20 client.16347 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.250+0000 7f5a795f8740 1 --2- 172.21.15.28:0/1403355908 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559f11c22780 0x559f11c24ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.250+0000 7f5a795f8740 4 client.16347 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:53:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.250+0000 7f5a795f8740 1 -- 172.21.15.28:0/1403355908 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559f11b0ea70 con 0x559f11c22780 2024-04-21T08:53:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.251+0000 7f5a6ffff640 1 --2- 172.21.15.28:0/1403355908 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559f11c22780 0x559f11c24ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.261+0000 7f5a6ffff640 1 --2- 172.21.15.28:0/1403355908 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559f11c22780 0x559f11c24ba0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10d1ee8349bc8f6b server_cookie=d41b0e1c280dd226 in_seq=0 out_seq=0 2024-04-21T08:53:03.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.261+0000 7f5a4f7fe640 10 client.16347 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:03.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.262+0000 7f5a4f7fe640 1 -- 172.21.15.28:0/1403355908 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559f11b0ea70 con 0x559f11c22780 2024-04-21T08:53:03.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.263+0000 7f5a4f7fe640 10 client.16347 handle_command_reply: tid=0 2024-04-21T08:53:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.263+0000 7f5a795f8740 1 client.16347 shutdown 2024-04-21T08:53:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.263+0000 7f5a795f8740 1 -- 172.21.15.28:0/1403355908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11ba2ea0 msgr2=0x559f11ba5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.263+0000 7f5a795f8740 1 --2- 172.21.15.28:0/1403355908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11ba2ea0 0x559f11ba5410 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5a64036040 tx=0x7f5a640048f0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.263+0000 7f5a795f8740 1 -- 172.21.15.28:0/1403355908 shutdown_connections 2024-04-21T08:53:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.263+0000 7f5a795f8740 1 --2- 172.21.15.28:0/1403355908 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559f11c22780 0x559f11c24ba0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.264+0000 7f5a795f8740 1 --2- 172.21.15.28:0/1403355908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559f11baba80 0x559f11ba5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.264+0000 7f5a795f8740 1 --2- 172.21.15.28:0/1403355908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559f11ba2ea0 0x559f11ba5410 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.264+0000 7f5a4f7fe640 0 client.16347 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.264+0000 7f5a795f8740 1 --2- 172.21.15.28:0/1403355908 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559f11ba2580 0x559f11bb4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.264+0000 7f5a795f8740 1 -- 172.21.15.28:0/1403355908 >> 172.21.15.28:0/1403355908 conn(0x559f11b9ccb0 msgr2=0x559f11b9bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.265+0000 7f5a795f8740 1 -- 172.21.15.28:0/1403355908 shutdown_connections 2024-04-21T08:53:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.266+0000 7f5a795f8740 1 -- 172.21.15.28:0/1403355908 wait complete. 2024-04-21T08:53:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.266+0000 7f5a795f8740 20 client.16347 trim_cache size 0 max 16384 2024-04-21T08:53:03.267 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:53:03.267 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T08:53:03.267 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T08:53:03.267 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:53:03.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.267+0000 7f5a7692b640 1 -- 172.21.15.28:0/2330859361 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5a580a3f50 msgr2=0x7f5a580a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.267+0000 7f5a7692b640 1 --2- 172.21.15.28:0/2330859361 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5a580a3f50 0x7f5a580a6410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5a7010e190 tx=0x7f5a64000bc0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.267+0000 7f5a7692b640 1 -- 172.21.15.28:0/2330859361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a7010cf10 msgr2=0x7f5a7010d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.267+0000 7f5a7692b640 1 --2- 172.21.15.28:0/2330859361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a7010cf10 0x7f5a7010d370 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800eb40 tx=0x7f5a6800c2d0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.268+0000 7f5a7692b640 1 -- 172.21.15.28:0/2330859361 shutdown_connections 2024-04-21T08:53:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.268+0000 7f5a7692b640 1 --2- 172.21.15.28:0/2330859361 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5a580a3f50 0x7f5a580a6410 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.268+0000 7f5a7692b640 1 --2- 172.21.15.28:0/2330859361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a7010cf10 0x7f5a7010d370 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.268+0000 7f5a7692b640 1 --2- 172.21.15.28:0/2330859361 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5a70112fe0 0x7f5a7010c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.268+0000 7f5a7692b640 1 --2- 172.21.15.28:0/2330859361 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5a70074250 0x7f5a70112960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.268+0000 7f5a7692b640 1 -- 172.21.15.28:0/2330859361 >> 172.21.15.28:0/2330859361 conn(0x7f5a700793e0 msgr2=0x7f5a70101500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.269+0000 7f5a7692b640 1 -- 172.21.15.28:0/2330859361 shutdown_connections 2024-04-21T08:53:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.269+0000 7f5a7692b640 1 -- 172.21.15.28:0/2330859361 wait complete. 2024-04-21T08:53:03.290 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:53:03.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:53:03.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.487+0000 7f5ba8ff4640 1 -- 172.21.15.28:0/475938964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba410c680 msgr2=0x7f5ba4108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.487+0000 7f5ba8ff4640 1 --2- 172.21.15.28:0/475938964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba410c680 0x7f5ba4108f50 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5b94009900 tx=0x7f5b9402bd10 comp rx=0 tx=0).stop 2024-04-21T08:53:03.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.489+0000 7f5ba8ff4640 1 -- 172.21.15.28:0/475938964 shutdown_connections 2024-04-21T08:53:03.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.489+0000 7f5ba8ff4640 1 --2- 172.21.15.28:0/475938964 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ba4116fe0 0x7f5ba41193d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.489+0000 7f5ba8ff4640 1 --2- 172.21.15.28:0/475938964 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5ba4109490 0x7f5ba4109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.489+0000 7f5ba8ff4640 1 --2- 172.21.15.28:0/475938964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba410c680 0x7f5ba4108f50 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.489+0000 7f5ba8ff4640 1 -- 172.21.15.28:0/475938964 >> 172.21.15.28:0/475938964 conn(0x7f5ba406e410 msgr2=0x7f5ba40719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.492+0000 7f5ba8ff4640 1 -- 172.21.15.28:0/475938964 shutdown_connections 2024-04-21T08:53:03.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.492+0000 7f5ba8ff4640 1 -- 172.21.15.28:0/475938964 wait complete. 2024-04-21T08:53:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.493+0000 7f5ba8ff4640 1 Processor -- start 2024-04-21T08:53:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.493+0000 7f5ba8ff4640 1 -- start start 2024-04-21T08:53:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.493+0000 7f5ba8ff4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba4109490 0x7f5ba4116910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.493+0000 7f5ba8ff4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ba410c680 0x7f5ba41139b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.493+0000 7f5ba8ff4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5ba4116fe0 0x7f5ba4113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.493+0000 7f5ba8ff4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba411cc60 con 0x7f5ba4109490 2024-04-21T08:53:03.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.493+0000 7f5ba8ff4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba411cae0 con 0x7f5ba410c680 2024-04-21T08:53:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.493+0000 7f5ba8ff4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5ba411cde0 con 0x7f5ba4116fe0 2024-04-21T08:53:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.494+0000 7f5ba259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba4109490 0x7f5ba4116910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.494+0000 7f5ba259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba4109490 0x7f5ba4116910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46322/0 (socket says 172.21.15.28:46322) 2024-04-21T08:53:03.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.494+0000 7f5ba259c640 1 -- 172.21.15.28:0/1465307774 learned_addr learned my addr 172.21.15.28:0/1465307774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.495+0000 7f5ba1d9b640 1 --2- 172.21.15.28:0/1465307774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ba410c680 0x7f5ba41139b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.495+0000 7f5ba2d9d640 1 --2- 172.21.15.28:0/1465307774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5ba4116fe0 0x7f5ba4113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.496+0000 7f5ba2d9d640 1 -- 172.21.15.28:0/1465307774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ba410c680 msgr2=0x7f5ba41139b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.496+0000 7f5ba2d9d640 1 --2- 172.21.15.28:0/1465307774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ba410c680 0x7f5ba41139b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.496+0000 7f5ba2d9d640 1 -- 172.21.15.28:0/1465307774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba4109490 msgr2=0x7f5ba4116910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.496+0000 7f5ba2d9d640 1 --2- 172.21.15.28:0/1465307774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba4109490 0x7f5ba4116910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.496+0000 7f5ba2d9d640 1 -- 172.21.15.28:0/1465307774 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ba41a9140 con 0x7f5ba4116fe0 2024-04-21T08:53:03.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.496+0000 7f5ba2d9d640 1 --2- 172.21.15.28:0/1465307774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5ba4116fe0 0x7f5ba4113f50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5ba406b440 tx=0x7f5b9800dae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.501+0000 7f5b937fe640 1 -- 172.21.15.28:0/1465307774 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b98017070 con 0x7f5ba4116fe0 2024-04-21T08:53:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.501+0000 7f5b937fe640 1 -- 172.21.15.28:0/1465307774 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b980047a0 con 0x7f5ba4116fe0 2024-04-21T08:53:03.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.501+0000 7f5b937fe640 1 -- 172.21.15.28:0/1465307774 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b98009c00 con 0x7f5ba4116fe0 2024-04-21T08:53:03.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.502+0000 7f5ba8ff4640 1 -- 172.21.15.28:0/1465307774 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ba41a9430 con 0x7f5ba4116fe0 2024-04-21T08:53:03.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.502+0000 7f5ba8ff4640 1 -- 172.21.15.28:0/1465307774 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ba41a9930 con 0x7f5ba4116fe0 2024-04-21T08:53:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.504+0000 7f5ba8ff4640 1 -- 172.21.15.28:0/1465307774 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ba4073e40 con 0x7f5ba4116fe0 2024-04-21T08:53:03.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.510+0000 7f5b937fe640 1 -- 172.21.15.28:0/1465307774 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5b98004950 con 0x7f5ba4116fe0 2024-04-21T08:53:03.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.511+0000 7f5b937fe640 1 --2- 172.21.15.28:0/1465307774 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5b8c0a3f20 0x7f5b8c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.511+0000 7f5b937fe640 1 -- 172.21.15.28:0/1465307774 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f5b980d0510 con 0x7f5ba4116fe0 2024-04-21T08:53:03.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.513+0000 7f5ba259c640 1 --2- 172.21.15.28:0/1465307774 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5b8c0a3f20 0x7f5b8c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.517+0000 7f5b937fe640 1 -- 172.21.15.28:0/1465307774 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b9809c8b0 con 0x7f5ba4116fe0 2024-04-21T08:53:03.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.519+0000 7f5ba259c640 1 --2- 172.21.15.28:0/1465307774 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5b8c0a3f20 0x7f5b8c0a63e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5b94000c00 tx=0x7f5b940095f0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:03.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.997+0000 7f5ba8ff4640 1 -- 172.21.15.28:0/1465307774 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ba41a9c50 con 0x7f5ba4116fe0 2024-04-21T08:53:03.998 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:53:03.999 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:52:11.568185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24487,24488,24463]},"id":1}]} 2024-04-21T08:53:03.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:03.999+0000 7f5b937fe640 1 -- 172.21.15.28:0/1465307774 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7f5b9809c1b0 con 0x7f5ba4116fe0 2024-04-21T08:53:03.999 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-21T08:53:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.008+0000 7f5ba8ff4640 1 -- 172.21.15.28:0/1465307774 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5b8c0a3f20 msgr2=0x7f5b8c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.008+0000 7f5ba8ff4640 1 --2- 172.21.15.28:0/1465307774 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5b8c0a3f20 0x7f5b8c0a63e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5b94000c00 tx=0x7f5b940095f0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.008+0000 7f5ba8ff4640 1 -- 172.21.15.28:0/1465307774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5ba4116fe0 msgr2=0x7f5ba4113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.008+0000 7f5ba8ff4640 1 --2- 172.21.15.28:0/1465307774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5ba4116fe0 0x7f5ba4113f50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5ba406b440 tx=0x7f5b9800dae0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.008+0000 7f5ba8ff4640 1 -- 172.21.15.28:0/1465307774 shutdown_connections 2024-04-21T08:53:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.008+0000 7f5ba8ff4640 1 --2- 172.21.15.28:0/1465307774 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5b8c0a3f20 0x7f5b8c0a63e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.008+0000 7f5ba8ff4640 1 --2- 172.21.15.28:0/1465307774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5ba4116fe0 0x7f5ba4113f50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.008+0000 7f5ba8ff4640 1 --2- 172.21.15.28:0/1465307774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ba410c680 0x7f5ba41139b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.008+0000 7f5ba8ff4640 1 --2- 172.21.15.28:0/1465307774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba4109490 0x7f5ba4116910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.008+0000 7f5ba8ff4640 1 -- 172.21.15.28:0/1465307774 >> 172.21.15.28:0/1465307774 conn(0x7f5ba406e410 msgr2=0x7f5ba41175d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.009+0000 7f5ba8ff4640 1 -- 172.21.15.28:0/1465307774 shutdown_connections 2024-04-21T08:53:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.009+0000 7f5ba8ff4640 1 -- 172.21.15.28:0/1465307774 wait complete. 2024-04-21T08:53:04.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:53:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.232+0000 7f0558611640 1 -- 172.21.15.28:0/488480125 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05501107a0 msgr2=0x7f055010e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.232+0000 7f0558611640 1 --2- 172.21.15.28:0/488480125 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05501107a0 0x7f055010e290 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0540009930 tx=0x7f054002bc40 comp rx=0 tx=0).stop 2024-04-21T08:53:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.233+0000 7f0558611640 1 -- 172.21.15.28:0/488480125 shutdown_connections 2024-04-21T08:53:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.233+0000 7f0558611640 1 --2- 172.21.15.28:0/488480125 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f055011b730 0x7f055011db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.233+0000 7f0558611640 1 --2- 172.21.15.28:0/488480125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055010e7d0 0x7f055010ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.233+0000 7f0558611640 1 --2- 172.21.15.28:0/488480125 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05501107a0 0x7f055010e290 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.233+0000 7f0558611640 1 -- 172.21.15.28:0/488480125 >> 172.21.15.28:0/488480125 conn(0x7f055006d060 msgr2=0x7f055006d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.235+0000 7f0558611640 1 -- 172.21.15.28:0/488480125 shutdown_connections 2024-04-21T08:53:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.235+0000 7f0558611640 1 -- 172.21.15.28:0/488480125 wait complete. 2024-04-21T08:53:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.236+0000 7f0558611640 1 Processor -- start 2024-04-21T08:53:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.237+0000 7f0558611640 1 -- start start 2024-04-21T08:53:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.237+0000 7f0558611640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055010e7d0 0x7f05501a7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.237+0000 7f0558611640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05501107a0 0x7f05501a7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.237+0000 7f0558611640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f055011b730 0x7f05501abe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.237+0000 7f0558611640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0550120f40 con 0x7f055010e7d0 2024-04-21T08:53:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.237+0000 7f0558611640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0550120dc0 con 0x7f055011b730 2024-04-21T08:53:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.237+0000 7f0558611640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f05501210c0 con 0x7f05501107a0 2024-04-21T08:53:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.238+0000 7f0555bac640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05501107a0 0x7f05501a7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.238+0000 7f0555bac640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05501107a0 0x7f05501a7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:57568/0 (socket says 172.21.15.28:57568) 2024-04-21T08:53:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.238+0000 7f0555bac640 1 -- 172.21.15.28:0/936404249 learned_addr learned my addr 172.21.15.28:0/936404249 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.238+0000 7f0556bae640 1 --2- 172.21.15.28:0/936404249 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f055011b730 0x7f05501abe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.238+0000 7f05563ad640 1 --2- 172.21.15.28:0/936404249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055010e7d0 0x7f05501a7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.238+0000 7f0556bae640 1 -- 172.21.15.28:0/936404249 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05501107a0 msgr2=0x7f05501a7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.238+0000 7f0556bae640 1 --2- 172.21.15.28:0/936404249 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05501107a0 0x7f05501a7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.238+0000 7f0556bae640 1 -- 172.21.15.28:0/936404249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055010e7d0 msgr2=0x7f05501a7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.238+0000 7f0556bae640 1 --2- 172.21.15.28:0/936404249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055010e7d0 0x7f05501a7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.238+0000 7f0556bae640 1 -- 172.21.15.28:0/936404249 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05501ac550 con 0x7f055011b730 2024-04-21T08:53:04.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.239+0000 7f0556bae640 1 --2- 172.21.15.28:0/936404249 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f055011b730 0x7f05501abe50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f054800e970 tx=0x7f054800ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.239+0000 7f05477fe640 1 -- 172.21.15.28:0/936404249 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0548004d20 con 0x7f055011b730 2024-04-21T08:53:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.239+0000 7f05477fe640 1 -- 172.21.15.28:0/936404249 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0548005020 con 0x7f055011b730 2024-04-21T08:53:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.240+0000 7f05477fe640 1 -- 172.21.15.28:0/936404249 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05480116d0 con 0x7f055011b730 2024-04-21T08:53:04.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.240+0000 7f0558611640 1 -- 172.21.15.28:0/936404249 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05501ac840 con 0x7f055011b730 2024-04-21T08:53:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.240+0000 7f0558611640 1 -- 172.21.15.28:0/936404249 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0550110f10 con 0x7f055011b730 2024-04-21T08:53:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.244+0000 7f055b1fe740 1 Processor -- start 2024-04-21T08:53:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.246+0000 7f055b1fe740 1 -- start start 2024-04-21T08:53:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.246+0000 7f055b1fe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c084d0 0x5643c9c088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.246+0000 7f055b1fe740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c08e80 0x5643c9c11430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.246+0000 7f055b1fe740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c11a00 0x5643c9c13e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.246+0000 7f055b1fe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5643c9b6d1f0 con 0x5643c9c084d0 2024-04-21T08:53:04.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.246+0000 7f055b1fe740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5643c9b0a4b0 con 0x5643c9c11a00 2024-04-21T08:53:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.246+0000 7f055b1fe740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5643c9b691b0 con 0x5643c9c08e80 2024-04-21T08:53:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.248+0000 7f05563ad640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c08e80 0x5643c9c11430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.248+0000 7f0555bac640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c11a00 0x5643c9c13e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.248+0000 7f0555bac640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c11a00 0x5643c9c13e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:58498/0 (socket says 172.21.15.28:58498) 2024-04-21T08:53:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.248+0000 7f0555bac640 1 -- 172.21.15.28:0/2169256551 learned_addr learned my addr 172.21.15.28:0/2169256551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.249+0000 7f0555bac640 1 --2- 172.21.15.28:0/2169256551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c084d0 0x5643c9c088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.251+0000 7f0555bac640 1 -- 172.21.15.28:0/2169256551 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c08e80 msgr2=0x5643c9c11430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.251+0000 7f0555bac640 1 --2- 172.21.15.28:0/2169256551 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c08e80 0x5643c9c11430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.251+0000 7f0555bac640 1 -- 172.21.15.28:0/2169256551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c084d0 msgr2=0x5643c9c088b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.251+0000 7f0555bac640 1 --2- 172.21.15.28:0/2169256551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c084d0 0x5643c9c088b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.251+0000 7f0555bac640 1 -- 172.21.15.28:0/2169256551 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643c9afc1b0 con 0x5643c9c11a00 2024-04-21T08:53:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.251+0000 7f05477fe640 1 -- 172.21.15.28:0/936404249 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f054800cb20 con 0x7f055011b730 2024-04-21T08:53:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.251+0000 7f0555bac640 1 --2- 172.21.15.28:0/2169256551 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c11a00 0x5643c9c13e50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f054c033910 tx=0x7f054c009530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33f23ec47a256ae2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.252+0000 7f05477fe640 1 --2- 172.21.15.28:0/936404249 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f053c0a3ef0 0x7f053c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.252+0000 7f05477fe640 1 -- 172.21.15.28:0/936404249 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f05480d0ae0 con 0x7f055011b730 2024-04-21T08:53:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.252+0000 7f0544bfe640 1 -- 172.21.15.28:0/2169256551 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c0317d0 con 0x5643c9c11a00 2024-04-21T08:53:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.253+0000 7f0544bfe640 1 -- 172.21.15.28:0/2169256551 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f054c031e20 con 0x5643c9c11a00 2024-04-21T08:53:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.254+0000 7f055b1fe740 1 -- 172.21.15.28:0/2169256551 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c11a00 msgr2=0x5643c9c13e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.254+0000 7f055b1fe740 1 --2- 172.21.15.28:0/2169256551 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c11a00 0x5643c9c13e50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f054c033910 tx=0x7f054c009530 comp rx=0 tx=0).stop 2024-04-21T08:53:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.254+0000 7f055b1fe740 1 -- 172.21.15.28:0/2169256551 shutdown_connections 2024-04-21T08:53:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.254+0000 7f055b1fe740 1 --2- 172.21.15.28:0/2169256551 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c11a00 0x5643c9c13e50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.254+0000 7f055b1fe740 1 --2- 172.21.15.28:0/2169256551 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c08e80 0x5643c9c11430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.254+0000 7f055b1fe740 1 --2- 172.21.15.28:0/2169256551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c084d0 0x5643c9c088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.254+0000 7f055b1fe740 1 -- 172.21.15.28:0/2169256551 >> 172.21.15.28:0/2169256551 conn(0x5643c9c02ba0 msgr2=0x5643c9c02f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.256+0000 7f05563ad640 1 --2- 172.21.15.28:0/936404249 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f053c0a3ef0 0x7f053c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.259+0000 7f05563ad640 1 --2- 172.21.15.28:0/936404249 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f053c0a3ef0 0x7f053c0a63b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0540009900 tx=0x7f0540005d50 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.260+0000 7f055b1fe740 1 -- 172.21.15.28:0/2169256551 shutdown_connections 2024-04-21T08:53:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.260+0000 7f055b1fe740 1 -- 172.21.15.28:0/2169256551 wait complete. 2024-04-21T08:53:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.260+0000 7f055b1fe740 1 -- start start 2024-04-21T08:53:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.261+0000 7f055b1fe740 1 Processor -- start 2024-04-21T08:53:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.262+0000 7f055b1fe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c084d0 0x5643c9c11240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.262+0000 7f055b1fe740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c08e80 0x5643c9c0e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.262+0000 7f055b1fe740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c11a00 0x5643c9c0eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.262+0000 7f055b1fe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5643c9c149c0 con 0x5643c9c084d0 2024-04-21T08:53:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.262+0000 7f055b1fe740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5643c9b26500 con 0x5643c9c11a00 2024-04-21T08:53:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.262+0000 7f055b1fe740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5643c9c14b40 con 0x5643c9c08e80 2024-04-21T08:53:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.263+0000 7f055b1fe740 10 client.0 _ll_register_callbacks cb 0x5643c9af4b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:04.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.263+0000 7f05563ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c084d0 0x5643c9c11240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.263+0000 7f05563ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c084d0 0x5643c9c11240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50218/0 (socket says 172.21.15.28:50218) 2024-04-21T08:53:04.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.263+0000 7f05563ad640 1 -- 172.21.15.28:0/3804097643 learned_addr learned my addr 172.21.15.28:0/3804097643 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:04.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.265+0000 7f0555bac640 1 --2- 172.21.15.28:0/3804097643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c08e80 0x5643c9c0e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.266+0000 7f05563ad640 1 -- 172.21.15.28:0/3804097643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c08e80 msgr2=0x5643c9c0e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:04.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.266+0000 7f05563ad640 1 --2- 172.21.15.28:0/3804097643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c08e80 0x5643c9c0e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.266+0000 7f05563ad640 1 -- 172.21.15.28:0/3804097643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c11a00 msgr2=0x5643c9c0eb10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:04.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.266+0000 7f05563ad640 1 --2- 172.21.15.28:0/3804097643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c11a00 0x5643c9c0eb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.266+0000 7f05563ad640 1 -- 172.21.15.28:0/3804097643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643c987f3f0 con 0x5643c9c084d0 2024-04-21T08:53:04.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.266+0000 7f05563ad640 1 --2- 172.21.15.28:0/3804097643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c084d0 0x5643c9c11240 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f0540006010 tx=0x7f0540009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77c61fd301bd52e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.266+0000 7f0539ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:04.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.267+0000 7f0539ffb640 1 -- 172.21.15.28:0/3804097643 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0540005610 con 0x5643c9c084d0 2024-04-21T08:53:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.267+0000 7f0539ffb640 1 -- 172.21.15.28:0/3804097643 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f054002dd50 con 0x5643c9c084d0 2024-04-21T08:53:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.267+0000 7f055b1fe740 1 -- 172.21.15.28:0/3804097643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f054c031e20 con 0x5643c9c084d0 2024-04-21T08:53:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.269+0000 7f0539ffb640 1 -- 172.21.15.28:0/3804097643 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0540034440 con 0x5643c9c084d0 2024-04-21T08:53:04.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.270+0000 7f055b1fe740 10 client.16365 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:04.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.270+0000 7f055b1fe740 1 -- 172.21.15.28:0/3804097643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5643c99497d0 con 0x5643c9c084d0 2024-04-21T08:53:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.271+0000 7f0539ffb640 1 -- 172.21.15.28:0/3804097643 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0540034a90 con 0x5643c9c084d0 2024-04-21T08:53:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.271+0000 7f0539ffb640 1 -- 172.21.15.28:0/3804097643 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f05400408a0 con 0x5643c9c084d0 2024-04-21T08:53:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.271+0000 7f055b1fe740 10 client.16365 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.271+0000 7f055b1fe740 10 client.16365 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:53:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.271+0000 7f055b1fe740 20 client.16365 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.272+0000 7f055b1fe740 1 --2- 172.21.15.28:0/3804097643 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5643c9c21c80 0x5643c9c24110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.272+0000 7f055b1fe740 4 client.16365 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.272+0000 7f055b1fe740 1 -- 172.21.15.28:0/3804097643 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f054c031e20 con 0x5643c9c21c80 2024-04-21T08:53:04.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.275+0000 7f0555bac640 1 --2- 172.21.15.28:0/3804097643 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5643c9c21c80 0x5643c9c24110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.278+0000 7f0555bac640 1 --2- 172.21.15.28:0/3804097643 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5643c9c21c80 0x5643c9c24110 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a0f2a0f42e8214d0 server_cookie=8207e0fd1e6ad10e in_seq=0 out_seq=0 2024-04-21T08:53:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.278+0000 7f0539ffb640 10 client.16365 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:53:04.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.283+0000 7f0539ffb640 1 -- 172.21.15.28:0/3804097643 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f054c031e20 con 0x5643c9c21c80 2024-04-21T08:53:04.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.283+0000 7f0539ffb640 10 client.16365 handle_command_reply: tid=0 2024-04-21T08:53:04.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.284+0000 7f055b1fe740 1 client.16365 shutdown 2024-04-21T08:53:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.286+0000 7f055b1fe740 1 -- 172.21.15.28:0/3804097643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c084d0 msgr2=0x5643c9c11240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.286+0000 7f055b1fe740 1 --2- 172.21.15.28:0/3804097643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c084d0 0x5643c9c11240 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f0540006010 tx=0x7f0540009260 comp rx=0 tx=0).stop 2024-04-21T08:53:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.286+0000 7f055b1fe740 1 -- 172.21.15.28:0/3804097643 shutdown_connections 2024-04-21T08:53:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.287+0000 7f055b1fe740 1 --2- 172.21.15.28:0/3804097643 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5643c9c21c80 0x5643c9c24110 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.287+0000 7f055b1fe740 1 --2- 172.21.15.28:0/3804097643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c11a00 0x5643c9c0eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.287+0000 7f055b1fe740 1 --2- 172.21.15.28:0/3804097643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c08e80 0x5643c9c0e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.287+0000 7f055b1fe740 1 --2- 172.21.15.28:0/3804097643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c084d0 0x5643c9c11240 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.287+0000 7f055b1fe740 1 -- 172.21.15.28:0/3804097643 >> 172.21.15.28:0/3804097643 conn(0x5643c9c02ba0 msgr2=0x5643c9c048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.287+0000 7f0539ffb640 0 client.16365 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:53:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.287+0000 7f055b1fe740 1 -- 172.21.15.28:0/3804097643 shutdown_connections 2024-04-21T08:53:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.287+0000 7f055b1fe740 1 -- 172.21.15.28:0/3804097643 wait complete. 2024-04-21T08:53:04.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.287+0000 7f055b1fe740 20 client.16365 trim_cache size 0 max 16384 2024-04-21T08:53:04.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.315+0000 7f055b1fe740 1 Processor -- start 2024-04-21T08:53:04.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.315+0000 7f055b1fe740 1 -- start start 2024-04-21T08:53:04.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.315+0000 7f055b1fe740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c084d0 0x5643c9c1d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.315+0000 7f055b1fe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c08e80 0x5643c9c09af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.316+0000 7f05563ad640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c084d0 0x5643c9c1d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.316+0000 7f05563ad640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c084d0 0x5643c9c1d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:57580/0 (socket says 172.21.15.28:57580) 2024-04-21T08:53:04.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.316+0000 7f055b1fe740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c11a00 0x5643c9c10d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.316+0000 7f05563ad640 1 -- 172.21.15.28:0/2677514269 learned_addr learned my addr 172.21.15.28:0/2677514269 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:04.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.316+0000 7f055b1fe740 1 -- 172.21.15.28:0/2677514269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0534012910 con 0x5643c9c08e80 2024-04-21T08:53:04.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.316+0000 7f055b1fe740 1 -- 172.21.15.28:0/2677514269 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f05340101e0 con 0x5643c9c11a00 2024-04-21T08:53:04.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.316+0000 7f055b1fe740 1 -- 172.21.15.28:0/2677514269 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f053400e030 con 0x5643c9c084d0 2024-04-21T08:53:04.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.316+0000 7f0555bac640 1 --2- 172.21.15.28:0/2677514269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c08e80 0x5643c9c09af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.316+0000 7f05563ad640 1 -- 172.21.15.28:0/2677514269 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c11a00 msgr2=0x5643c9c10d00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:04.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.316+0000 7f05563ad640 1 --2- 172.21.15.28:0/2677514269 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c11a00 0x5643c9c10d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.316+0000 7f05563ad640 1 -- 172.21.15.28:0/2677514269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c08e80 msgr2=0x5643c9c09af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.316+0000 7f05563ad640 1 --2- 172.21.15.28:0/2677514269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c08e80 0x5643c9c09af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.316+0000 7f05563ad640 1 -- 172.21.15.28:0/2677514269 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643c9c029d0 con 0x5643c9c084d0 2024-04-21T08:53:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.317+0000 7f05563ad640 1 --2- 172.21.15.28:0/2677514269 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c084d0 0x5643c9c1d8d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f054002c180 tx=0x7f054002bc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2d7e10a8a9220c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.318+0000 7f053bfff640 1 -- 172.21.15.28:0/2677514269 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0540005610 con 0x5643c9c084d0 2024-04-21T08:53:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.318+0000 7f053bfff640 1 -- 172.21.15.28:0/2677514269 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f054005f420 con 0x5643c9c084d0 2024-04-21T08:53:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.318+0000 7f053bfff640 1 -- 172.21.15.28:0/2677514269 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054005f970 con 0x5643c9c084d0 2024-04-21T08:53:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.319+0000 7f055b1fe740 1 -- 172.21.15.28:0/2677514269 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c084d0 msgr2=0x5643c9c1d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.319+0000 7f055b1fe740 1 --2- 172.21.15.28:0/2677514269 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c084d0 0x5643c9c1d8d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f054002c180 tx=0x7f054002bc40 comp rx=0 tx=0).stop 2024-04-21T08:53:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.320+0000 7f055b1fe740 1 -- 172.21.15.28:0/2677514269 shutdown_connections 2024-04-21T08:53:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.320+0000 7f055b1fe740 1 --2- 172.21.15.28:0/2677514269 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c11a00 0x5643c9c10d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.320+0000 7f055b1fe740 1 --2- 172.21.15.28:0/2677514269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c08e80 0x5643c9c09af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.320+0000 7f055b1fe740 1 --2- 172.21.15.28:0/2677514269 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c084d0 0x5643c9c1d8d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.320+0000 7f055b1fe740 1 -- 172.21.15.28:0/2677514269 >> 172.21.15.28:0/2677514269 conn(0x5643c9c02ba0 msgr2=0x5643c9c01b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.321+0000 7f055b1fe740 1 -- 172.21.15.28:0/2677514269 shutdown_connections 2024-04-21T08:53:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.321+0000 7f055b1fe740 1 -- 172.21.15.28:0/2677514269 wait complete. 2024-04-21T08:53:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.322+0000 7f055b1fe740 1 -- start start 2024-04-21T08:53:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.322+0000 7f055b1fe740 1 Processor -- start 2024-04-21T08:53:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.322+0000 7f055b1fe740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c084d0 0x5643c9c1a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.322+0000 7f055b1fe740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c08e80 0x5643c9c0af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.322+0000 7f055b1fe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c11a00 0x5643c9c0b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.322+0000 7f055b1fe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f053400e6a0 con 0x5643c9c11a00 2024-04-21T08:53:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.322+0000 7f055b1fe740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f053400fb10 con 0x5643c9c08e80 2024-04-21T08:53:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.322+0000 7f055b1fe740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f053400ed70 con 0x5643c9c084d0 2024-04-21T08:53:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.322+0000 7f055b1fe740 10 client.0 _ll_register_callbacks cb 0x7f0540004e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.323+0000 7f0555bac640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c08e80 0x5643c9c0af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.323+0000 7f0555bac640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c08e80 0x5643c9c0af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:58530/0 (socket says 172.21.15.28:58530) 2024-04-21T08:53:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.323+0000 7f0555bac640 1 -- 172.21.15.28:0/1905081239 learned_addr learned my addr 172.21.15.28:0/1905081239 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.323+0000 7f05563ad640 1 --2- 172.21.15.28:0/1905081239 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c084d0 0x5643c9c1a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.324+0000 7f0556bae640 1 --2- 172.21.15.28:0/1905081239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c11a00 0x5643c9c0b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.324+0000 7f0555bac640 1 -- 172.21.15.28:0/1905081239 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c084d0 msgr2=0x5643c9c1a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.324+0000 7f0555bac640 1 --2- 172.21.15.28:0/1905081239 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c084d0 0x5643c9c1a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.324+0000 7f0555bac640 1 -- 172.21.15.28:0/1905081239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c11a00 msgr2=0x5643c9c0b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.324+0000 7f0555bac640 1 --2- 172.21.15.28:0/1905081239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c11a00 0x5643c9c0b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.324+0000 7f0555bac640 1 -- 172.21.15.28:0/1905081239 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643c9c0bd00 con 0x5643c9c08e80 2024-04-21T08:53:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.324+0000 7f0555bac640 1 --2- 172.21.15.28:0/1905081239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c08e80 0x5643c9c0af90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f054c009250 tx=0x7f054c040770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3aea367ae1cee19 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.325+0000 7f0539ffb640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:53:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.325+0000 7f0539ffb640 1 -- 172.21.15.28:0/1905081239 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c04e070 con 0x5643c9c08e80 2024-04-21T08:53:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.325+0000 7f0539ffb640 1 -- 172.21.15.28:0/1905081239 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f054c009ab0 con 0x5643c9c08e80 2024-04-21T08:53:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.325+0000 7f055b1fe740 1 -- 172.21.15.28:0/1905081239 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f054005f420 con 0x5643c9c08e80 2024-04-21T08:53:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.325+0000 7f0539ffb640 1 -- 172.21.15.28:0/1905081239 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f054c05c050 con 0x5643c9c08e80 2024-04-21T08:53:04.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.326+0000 7f055b1fe740 10 client.25745 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:04.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.326+0000 7f055b1fe740 1 -- 172.21.15.28:0/1905081239 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5643c9caa300 con 0x5643c9c08e80 2024-04-21T08:53:04.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.329+0000 7f0539ffb640 1 -- 172.21.15.28:0/1905081239 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f054005f420 con 0x5643c9c08e80 2024-04-21T08:53:04.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.331+0000 7f0539ffb640 1 -- 172.21.15.28:0/1905081239 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f054c0317a0 con 0x5643c9c08e80 2024-04-21T08:53:04.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.331+0000 7f055b1fe740 10 client.25745 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:04.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.331+0000 7f055b1fe740 10 client.25745 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:53:04.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.331+0000 7f055b1fe740 20 client.25745 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:04.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.331+0000 7f055b1fe740 1 --2- 172.21.15.28:0/1905081239 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5643c9caa840 0x5643c9cacc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.331+0000 7f055b1fe740 4 client.25745 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:53:04.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.331+0000 7f055b1fe740 1 -- 172.21.15.28:0/1905081239 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5643c9b0c930 con 0x5643c9caa840 2024-04-21T08:53:04.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.333+0000 7f05563ad640 1 --2- 172.21.15.28:0/1905081239 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5643c9caa840 0x5643c9cacc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.334+0000 7f05563ad640 1 --2- 172.21.15.28:0/1905081239 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5643c9caa840 0x5643c9cacc30 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e4d1e0398a6c7d5 server_cookie=ad760f030dd19efc in_seq=0 out_seq=0 2024-04-21T08:53:04.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.335+0000 7f0539ffb640 10 client.25745 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:53:04.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.335+0000 7f0539ffb640 1 -- 172.21.15.28:0/1905081239 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5643c9b0c930 con 0x5643c9caa840 2024-04-21T08:53:04.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.335+0000 7f0539ffb640 10 client.25745 handle_command_reply: tid=0 2024-04-21T08:53:04.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.335+0000 7f055b1fe740 1 client.25745 shutdown 2024-04-21T08:53:04.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.336+0000 7f055b1fe740 1 -- 172.21.15.28:0/1905081239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c08e80 msgr2=0x5643c9c0af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.336+0000 7f055b1fe740 1 --2- 172.21.15.28:0/1905081239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c08e80 0x5643c9c0af90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f054c009250 tx=0x7f054c040770 comp rx=0 tx=0).stop 2024-04-21T08:53:04.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.336+0000 7f055b1fe740 1 -- 172.21.15.28:0/1905081239 shutdown_connections 2024-04-21T08:53:04.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.336+0000 7f055b1fe740 1 --2- 172.21.15.28:0/1905081239 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5643c9caa840 0x5643c9cacc30 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.336+0000 7f055b1fe740 1 --2- 172.21.15.28:0/1905081239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643c9c11a00 0x5643c9c0b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.336+0000 7f055b1fe740 1 --2- 172.21.15.28:0/1905081239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5643c9c08e80 0x5643c9c0af90 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f054c009250 tx=0x7f054c040770 comp rx=0 tx=0).stop 2024-04-21T08:53:04.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.336+0000 7f055b1fe740 1 --2- 172.21.15.28:0/1905081239 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5643c9c084d0 0x5643c9c1a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.336+0000 7f055b1fe740 1 -- 172.21.15.28:0/1905081239 >> 172.21.15.28:0/1905081239 conn(0x5643c9c02ba0 msgr2=0x5643c9c09af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:04.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.336+0000 7f0539ffb640 0 client.25745 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:53:04.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.342+0000 7f055b1fe740 1 -- 172.21.15.28:0/1905081239 shutdown_connections 2024-04-21T08:53:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.344+0000 7f055b1fe740 1 -- 172.21.15.28:0/1905081239 wait complete. 2024-04-21T08:53:04.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.345+0000 7f055b1fe740 20 client.25745 trim_cache size 0 max 16384 2024-04-21T08:53:04.345 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:53:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.350+0000 7f0558611640 1 -- 172.21.15.28:0/936404249 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f053c0a3ef0 msgr2=0x7f053c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.350+0000 7f0558611640 1 --2- 172.21.15.28:0/936404249 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f053c0a3ef0 0x7f053c0a63b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0540009900 tx=0x7f0540005d50 comp rx=0 tx=0).stop 2024-04-21T08:53:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.350+0000 7f0558611640 1 -- 172.21.15.28:0/936404249 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f055011b730 msgr2=0x7f05501abe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.350+0000 7f0558611640 1 --2- 172.21.15.28:0/936404249 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f055011b730 0x7f05501abe50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f054800e970 tx=0x7f054800ee30 comp rx=0 tx=0).stop 2024-04-21T08:53:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.350+0000 7f0558611640 1 -- 172.21.15.28:0/936404249 shutdown_connections 2024-04-21T08:53:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.350+0000 7f0558611640 1 --2- 172.21.15.28:0/936404249 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f053c0a3ef0 0x7f053c0a63b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.350+0000 7f0558611640 1 --2- 172.21.15.28:0/936404249 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f055011b730 0x7f05501abe50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.350+0000 7f0558611640 1 --2- 172.21.15.28:0/936404249 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05501107a0 0x7f05501a7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.350+0000 7f0558611640 1 --2- 172.21.15.28:0/936404249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055010e7d0 0x7f05501a7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.350+0000 7f0558611640 1 -- 172.21.15.28:0/936404249 >> 172.21.15.28:0/936404249 conn(0x7f055006d060 msgr2=0x7f0550072a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:04.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.351+0000 7f0558611640 1 -- 172.21.15.28:0/936404249 shutdown_connections 2024-04-21T08:53:04.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.351+0000 7f0558611640 1 -- 172.21.15.28:0/936404249 wait complete. 2024-04-21T08:53:04.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:03 smithi152 ceph-mon[31401]: pgmap v59: 129 pgs: 129 active+clean; 13 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 178 MiB/s wr, 21.66k op/s 2024-04-21T08:53:04.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:03 smithi028 ceph-mon[28003]: pgmap v59: 129 pgs: 129 active+clean; 13 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 178 MiB/s wr, 21.66k op/s 2024-04-21T08:53:04.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:03 smithi171 ceph-mon[31358]: pgmap v59: 129 pgs: 129 active+clean; 13 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 178 MiB/s wr, 21.66k op/s 2024-04-21T08:53:04.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:53:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.576+0000 7f567aa01640 1 -- 172.21.15.28:0/455614493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5674108f40 msgr2=0x7f56741153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.576+0000 7f567aa01640 1 --2- 172.21.15.28:0/455614493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5674108f40 0x7f56741153b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f566c009be0 tx=0x7f566c02bf10 comp rx=0 tx=0).stop 2024-04-21T08:53:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.579+0000 7f567aa01640 1 -- 172.21.15.28:0/455614493 shutdown_connections 2024-04-21T08:53:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.579+0000 7f567aa01640 1 --2- 172.21.15.28:0/455614493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5674108f40 0x7f56741153b0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.579+0000 7f567aa01640 1 --2- 172.21.15.28:0/455614493 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5674108580 0x7f5674108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.579+0000 7f567aa01640 1 --2- 172.21.15.28:0/455614493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f567410e5f0 0x7f567410e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.579+0000 7f567aa01640 1 -- 172.21.15.28:0/455614493 >> 172.21.15.28:0/455614493 conn(0x7f5674076490 msgr2=0x7f56740788b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:04.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.584+0000 7f567aa01640 1 -- 172.21.15.28:0/455614493 shutdown_connections 2024-04-21T08:53:04.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.585+0000 7f567aa01640 1 -- 172.21.15.28:0/455614493 wait complete. 2024-04-21T08:53:04.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.585+0000 7f567aa01640 1 Processor -- start 2024-04-21T08:53:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.587+0000 7f567aa01640 1 -- start start 2024-04-21T08:53:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.587+0000 7f567aa01640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5674108580 0x7f56740737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.587+0000 7f567aa01640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5674108f40 0x7f567406c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.587+0000 7f567aa01640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f567410e5f0 0x7f567406cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.587+0000 7f567aa01640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5674118760 con 0x7f5674108f40 2024-04-21T08:53:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.587+0000 7f567aa01640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f56741185e0 con 0x7f567410e5f0 2024-04-21T08:53:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.587+0000 7f567aa01640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f56741188e0 con 0x7f5674108580 2024-04-21T08:53:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.588+0000 7f5678f9e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f567410e5f0 0x7f567406cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.588+0000 7f5678f9e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f567410e5f0 0x7f567406cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:58544/0 (socket says 172.21.15.28:58544) 2024-04-21T08:53:04.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.588+0000 7f5678f9e640 1 -- 172.21.15.28:0/2676537252 learned_addr learned my addr 172.21.15.28:0/2676537252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:04.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.589+0000 7f5678f9e640 1 -- 172.21.15.28:0/2676537252 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5674108580 msgr2=0x7f56740737b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.589+0000 7f5673fff640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5674108580 0x7f56740737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.589+0000 7f56737fe640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5674108f40 0x7f567406c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.590+0000 7f5678f9e640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5674108580 0x7f56740737b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.590+0000 7f5678f9e640 1 -- 172.21.15.28:0/2676537252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5674108f40 msgr2=0x7f567406c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.590+0000 7f5678f9e640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5674108f40 0x7f567406c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.590+0000 7f5678f9e640 1 -- 172.21.15.28:0/2676537252 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f567406d3d0 con 0x7f567410e5f0 2024-04-21T08:53:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.590+0000 7f5673fff640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5674108580 0x7f56740737b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:53:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.591+0000 7f5678f9e640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f567410e5f0 0x7f567406cdc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f566c02cc20 tx=0x7f566c033750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.591+0000 7f56737fe640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5674108f40 0x7f567406c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:53:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.591+0000 7f56717fa640 1 -- 172.21.15.28:0/2676537252 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566c043020 con 0x7f567410e5f0 2024-04-21T08:53:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.591+0000 7f56717fa640 1 -- 172.21.15.28:0/2676537252 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f566c003b80 con 0x7f567410e5f0 2024-04-21T08:53:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.592+0000 7f56717fa640 1 -- 172.21.15.28:0/2676537252 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566c034690 con 0x7f567410e5f0 2024-04-21T08:53:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.592+0000 7f567aa01640 1 -- 172.21.15.28:0/2676537252 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f567406d660 con 0x7f567410e5f0 2024-04-21T08:53:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.592+0000 7f567aa01640 1 -- 172.21.15.28:0/2676537252 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56740715a0 con 0x7f567410e5f0 2024-04-21T08:53:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.596+0000 7f567d5f0740 1 Processor -- start 2024-04-21T08:53:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.600+0000 7f56717fa640 1 -- 172.21.15.28:0/2676537252 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f566c046020 con 0x7f567410e5f0 2024-04-21T08:53:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.601+0000 7f56717fa640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f565c0a3f20 0x7f565c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.601+0000 7f56717fa640 1 -- 172.21.15.28:0/2676537252 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f566c0f3400 con 0x7f567410e5f0 2024-04-21T08:53:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.601+0000 7f567d5f0740 1 -- start start 2024-04-21T08:53:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.602+0000 7f567d5f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd34d0 0x5602d2cd38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.602+0000 7f567d5f0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cd3e80 0x5602d2cdc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.602+0000 7f567d5f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cdca00 0x5602d2cdee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.602+0000 7f567d5f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5602d2c381f0 con 0x5602d2cd34d0 2024-04-21T08:53:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.602+0000 7f567d5f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5602d2bd54b0 con 0x5602d2cdca00 2024-04-21T08:53:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.602+0000 7f567d5f0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5602d2c341b0 con 0x5602d2cd3e80 2024-04-21T08:53:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.602+0000 7f5678f9e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cdca00 0x5602d2cdee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.602+0000 7f5678f9e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cdca00 0x5602d2cdee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:58548/0 (socket says 172.21.15.28:58548) 2024-04-21T08:53:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.602+0000 7f5678f9e640 1 -- 172.21.15.28:0/3537260255 learned_addr learned my addr 172.21.15.28:0/3537260255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.602+0000 7f5678f9e640 1 -- 172.21.15.28:0/3537260255 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cd3e80 msgr2=0x5602d2cdc430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.602+0000 7f5678f9e640 1 --2- 172.21.15.28:0/3537260255 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cd3e80 0x5602d2cdc430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.602+0000 7f5678f9e640 1 -- 172.21.15.28:0/3537260255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd34d0 msgr2=0x5602d2cd38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.602+0000 7f5678f9e640 1 --2- 172.21.15.28:0/3537260255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd34d0 0x5602d2cd38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.602+0000 7f5678f9e640 1 -- 172.21.15.28:0/3537260255 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602d2bc71b0 con 0x5602d2cdca00 2024-04-21T08:53:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.603+0000 7f56737fe640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f565c0a3f20 0x7f565c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.603+0000 7f5678f9e640 1 --2- 172.21.15.28:0/3537260255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cdca00 0x5602d2cdee50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f566c0bf040 tx=0x7f566c02bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=678a6c85c38d618a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.604+0000 7f56737fe640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f565c0a3f20 0x7f565c0a63e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5664005d00 tx=0x7f5664005e50 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.604+0000 7f565a7fc640 1 -- 172.21.15.28:0/3537260255 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566c0d8070 con 0x5602d2cdca00 2024-04-21T08:53:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.604+0000 7f565a7fc640 1 -- 172.21.15.28:0/3537260255 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f566c0c3460 con 0x5602d2cdca00 2024-04-21T08:53:04.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.605+0000 7f567d5f0740 1 -- 172.21.15.28:0/3537260255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cdca00 msgr2=0x5602d2cdee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.605+0000 7f567d5f0740 1 --2- 172.21.15.28:0/3537260255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cdca00 0x5602d2cdee50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f566c0bf040 tx=0x7f566c02bf10 comp rx=0 tx=0).stop 2024-04-21T08:53:04.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.606+0000 7f567d5f0740 1 -- 172.21.15.28:0/3537260255 shutdown_connections 2024-04-21T08:53:04.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.606+0000 7f567d5f0740 1 --2- 172.21.15.28:0/3537260255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cdca00 0x5602d2cdee50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.606+0000 7f567d5f0740 1 --2- 172.21.15.28:0/3537260255 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cd3e80 0x5602d2cdc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.606+0000 7f567d5f0740 1 --2- 172.21.15.28:0/3537260255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd34d0 0x5602d2cd38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.606+0000 7f567d5f0740 1 -- 172.21.15.28:0/3537260255 >> 172.21.15.28:0/3537260255 conn(0x5602d2ccdba0 msgr2=0x5602d2ccdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:04.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.608+0000 7f567d5f0740 1 -- 172.21.15.28:0/3537260255 shutdown_connections 2024-04-21T08:53:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.609+0000 7f567d5f0740 1 -- 172.21.15.28:0/3537260255 wait complete. 2024-04-21T08:53:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.610+0000 7f567d5f0740 1 -- start start 2024-04-21T08:53:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.610+0000 7f567d5f0740 1 Processor -- start 2024-04-21T08:53:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.612+0000 7f567d5f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd34d0 0x5602d2cdc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.612+0000 7f567d5f0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cd3e80 0x5602d2cd95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.612+0000 7f567d5f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cdca00 0x5602d2cd9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.612+0000 7f567d5f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5602d2bd54b0 con 0x5602d2cd34d0 2024-04-21T08:53:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.612+0000 7f567d5f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5602d2bf1500 con 0x5602d2cdca00 2024-04-21T08:53:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.612+0000 7f567d5f0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5602d2cdf730 con 0x5602d2cd3e80 2024-04-21T08:53:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.612+0000 7f567d5f0740 10 client.0 _ll_register_callbacks cb 0x5602d2bbfb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.612+0000 7f5673fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd34d0 0x5602d2cdc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.612+0000 7f5673fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd34d0 0x5602d2cdc240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50280/0 (socket says 172.21.15.28:50280) 2024-04-21T08:53:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.613+0000 7f5673fff640 1 -- 172.21.15.28:0/1335987837 learned_addr learned my addr 172.21.15.28:0/1335987837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.615+0000 7f5673fff640 1 -- 172.21.15.28:0/1335987837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cd3e80 msgr2=0x5602d2cd95d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.615+0000 7f5673fff640 1 --2- 172.21.15.28:0/1335987837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cd3e80 0x5602d2cd95d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.615+0000 7f5673fff640 1 -- 172.21.15.28:0/1335987837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cdca00 msgr2=0x5602d2cd9b10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.615+0000 7f5673fff640 1 --2- 172.21.15.28:0/1335987837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cdca00 0x5602d2cd9b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.615+0000 7f5673fff640 1 -- 172.21.15.28:0/1335987837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602d294a3f0 con 0x5602d2cd34d0 2024-04-21T08:53:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.615+0000 7f5673fff640 1 --2- 172.21.15.28:0/1335987837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd34d0 0x5602d2cdc240 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f5668009e90 tx=0x7f566800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558c897ea4fcd86a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.615+0000 7f56537fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.616+0000 7f56537fe640 1 -- 172.21.15.28:0/1335987837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5668004230 con 0x5602d2cd34d0 2024-04-21T08:53:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.616+0000 7f567d5f0740 1 -- 172.21.15.28:0/1335987837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f566c0c3460 con 0x5602d2cd34d0 2024-04-21T08:53:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.616+0000 7f56537fe640 1 -- 172.21.15.28:0/1335987837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f56680043d0 con 0x5602d2cd34d0 2024-04-21T08:53:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.616+0000 7f56537fe640 1 -- 172.21.15.28:0/1335987837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5668023050 con 0x5602d2cd34d0 2024-04-21T08:53:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.617+0000 7f567d5f0740 10 client.16377 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.617+0000 7f567d5f0740 1 -- 172.21.15.28:0/1335987837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5602d2a147d0 con 0x5602d2cd34d0 2024-04-21T08:53:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.618+0000 7f56537fe640 1 -- 172.21.15.28:0/1335987837 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f566c0c3460 con 0x5602d2cd34d0 2024-04-21T08:53:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.619+0000 7f56537fe640 1 -- 172.21.15.28:0/1335987837 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5668003940 con 0x5602d2cd34d0 2024-04-21T08:53:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.619+0000 7f567d5f0740 10 client.16377 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.619+0000 7f567d5f0740 10 client.16377 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:53:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.619+0000 7f567d5f0740 20 client.16377 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.619+0000 7f567d5f0740 1 --2- 172.21.15.28:0/1335987837 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5602d2cecca0 0x5602d2cef160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.619+0000 7f567d5f0740 4 client.16377 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:04.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.619+0000 7f567d5f0740 1 -- 172.21.15.28:0/1335987837 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602d2bd7930 con 0x5602d2cecca0 2024-04-21T08:53:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.622+0000 7f56737fe640 1 --2- 172.21.15.28:0/1335987837 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5602d2cecca0 0x5602d2cef160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.626+0000 7f56737fe640 1 --2- 172.21.15.28:0/1335987837 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5602d2cecca0 0x5602d2cef160 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fc6d2c4bd53d8708 server_cookie=d0e64a5c3aa1f95c in_seq=0 out_seq=0 2024-04-21T08:53:04.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.626+0000 7f56537fe640 10 client.16377 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:53:04.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.628+0000 7f56537fe640 1 -- 172.21.15.28:0/1335987837 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602d2bd7930 con 0x5602d2cecca0 2024-04-21T08:53:04.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.628+0000 7f56537fe640 10 client.16377 handle_command_reply: tid=0 2024-04-21T08:53:04.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.631+0000 7f567d5f0740 1 client.16377 shutdown 2024-04-21T08:53:04.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.633+0000 7f567d5f0740 1 -- 172.21.15.28:0/1335987837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd34d0 msgr2=0x5602d2cdc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.633+0000 7f567d5f0740 1 --2- 172.21.15.28:0/1335987837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd34d0 0x5602d2cdc240 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f5668009e90 tx=0x7f566800f730 comp rx=0 tx=0).stop 2024-04-21T08:53:04.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.634+0000 7f567d5f0740 1 -- 172.21.15.28:0/1335987837 shutdown_connections 2024-04-21T08:53:04.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.634+0000 7f567d5f0740 1 --2- 172.21.15.28:0/1335987837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cdca00 0x5602d2cd9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.634+0000 7f567d5f0740 1 --2- 172.21.15.28:0/1335987837 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5602d2cecca0 0x5602d2cef160 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.634+0000 7f567d5f0740 1 --2- 172.21.15.28:0/1335987837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cd3e80 0x5602d2cd95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.634+0000 7f567d5f0740 1 --2- 172.21.15.28:0/1335987837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd34d0 0x5602d2cdc240 secure :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f5668009e90 tx=0x7f566800f730 comp rx=0 tx=0).stop 2024-04-21T08:53:04.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.634+0000 7f567d5f0740 1 -- 172.21.15.28:0/1335987837 >> 172.21.15.28:0/1335987837 conn(0x5602d2ccdba0 msgr2=0x5602d2ccf8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:04.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.635+0000 7f56537fe640 0 client.16377 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:53:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.635+0000 7f567d5f0740 1 -- 172.21.15.28:0/1335987837 shutdown_connections 2024-04-21T08:53:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.636+0000 7f567d5f0740 1 -- 172.21.15.28:0/1335987837 wait complete. 2024-04-21T08:53:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.636+0000 7f567d5f0740 20 client.16377 trim_cache size 0 max 16384 2024-04-21T08:53:04.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.668+0000 7f567d5f0740 1 Processor -- start 2024-04-21T08:53:04.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.669+0000 7f567d5f0740 1 -- start start 2024-04-21T08:53:04.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.669+0000 7f567d5f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cd3e80 0x5602d2ce8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.669+0000 7f567d5f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cdca00 0x5602d2cccc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.669+0000 7f567d5f0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cdbd00 0x5602d2cdc180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.669+0000 7f567d5f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5660012860 con 0x5602d2cdca00 2024-04-21T08:53:04.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.670+0000 7f567d5f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5660010130 con 0x5602d2cd3e80 2024-04-21T08:53:04.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.670+0000 7f567d5f0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f566000df80 con 0x5602d2cdbd00 2024-04-21T08:53:04.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.671+0000 7f56737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cdca00 0x5602d2cccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.671+0000 7f56737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cdca00 0x5602d2cccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50296/0 (socket says 172.21.15.28:50296) 2024-04-21T08:53:04.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.671+0000 7f56737fe640 1 -- 172.21.15.28:0/1103419815 learned_addr learned my addr 172.21.15.28:0/1103419815 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:04.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.671+0000 7f5678f9e640 1 --2- 172.21.15.28:0/1103419815 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cdbd00 0x5602d2cdc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.673+0000 7f56737fe640 1 -- 172.21.15.28:0/1103419815 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cdbd00 msgr2=0x5602d2cdc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.673+0000 7f56737fe640 1 --2- 172.21.15.28:0/1103419815 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cdbd00 0x5602d2cdc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.673+0000 7f56737fe640 1 -- 172.21.15.28:0/1103419815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cd3e80 msgr2=0x5602d2ce8470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.673+0000 7f56737fe640 1 --2- 172.21.15.28:0/1103419815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cd3e80 0x5602d2ce8470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.673+0000 7f56737fe640 1 -- 172.21.15.28:0/1103419815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602d2ceb3d0 con 0x5602d2cdca00 2024-04-21T08:53:04.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.674+0000 7f56737fe640 1 --2- 172.21.15.28:0/1103419815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cdca00 0x5602d2cccc00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x5602d2cd2fa0 tx=0x7f5664011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98366cebd97aa7e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.679+0000 7f5659ffb640 1 -- 172.21.15.28:0/1103419815 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56640241d0 con 0x5602d2cdca00 2024-04-21T08:53:04.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.679+0000 7f5659ffb640 1 -- 172.21.15.28:0/1103419815 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5664017070 con 0x5602d2cdca00 2024-04-21T08:53:04.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.679+0000 7f5659ffb640 1 -- 172.21.15.28:0/1103419815 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566401fa50 con 0x5602d2cdca00 2024-04-21T08:53:04.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.680+0000 7f567d5f0740 1 -- 172.21.15.28:0/1103419815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cdca00 msgr2=0x5602d2cccc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.680+0000 7f567d5f0740 1 --2- 172.21.15.28:0/1103419815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cdca00 0x5602d2cccc00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x5602d2cd2fa0 tx=0x7f5664011000 comp rx=0 tx=0).stop 2024-04-21T08:53:04.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.681+0000 7f567d5f0740 1 -- 172.21.15.28:0/1103419815 shutdown_connections 2024-04-21T08:53:04.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.682+0000 7f567d5f0740 1 --2- 172.21.15.28:0/1103419815 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cdbd00 0x5602d2cdc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.682+0000 7f567d5f0740 1 --2- 172.21.15.28:0/1103419815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cdca00 0x5602d2cccc00 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.682+0000 7f567d5f0740 1 --2- 172.21.15.28:0/1103419815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cd3e80 0x5602d2ce8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.682+0000 7f567d5f0740 1 -- 172.21.15.28:0/1103419815 >> 172.21.15.28:0/1103419815 conn(0x5602d2ccdba0 msgr2=0x5602d2cd4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:04.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.687+0000 7f567d5f0740 1 -- 172.21.15.28:0/1103419815 shutdown_connections 2024-04-21T08:53:04.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.689+0000 7f567d5f0740 1 -- 172.21.15.28:0/1103419815 wait complete. 2024-04-21T08:53:04.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.690+0000 7f567d5f0740 1 -- start start 2024-04-21T08:53:04.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.690+0000 7f567d5f0740 1 Processor -- start 2024-04-21T08:53:04.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.691+0000 7f567d5f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd3e80 0x5602d2ce4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.691+0000 7f567d5f0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cdbd00 0x5602d2cd5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.691+0000 7f5673fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd3e80 0x5602d2ce4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.691+0000 7f5673fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd3e80 0x5602d2ce4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50304/0 (socket says 172.21.15.28:50304) 2024-04-21T08:53:04.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.691+0000 7f5673fff640 1 -- 172.21.15.28:0/635283706 learned_addr learned my addr 172.21.15.28:0/635283706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:04.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.691+0000 7f567d5f0740 1 --2- 172.21.15.28:0/635283706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cdca00 0x5602d2cd64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.692+0000 7f567d5f0740 1 -- 172.21.15.28:0/635283706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f566000e5f0 con 0x5602d2cd3e80 2024-04-21T08:53:04.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.692+0000 7f567d5f0740 1 -- 172.21.15.28:0/635283706 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f566000fa60 con 0x5602d2cdca00 2024-04-21T08:53:04.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.692+0000 7f567d5f0740 1 -- 172.21.15.28:0/635283706 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f566000ecc0 con 0x5602d2cdbd00 2024-04-21T08:53:04.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.692+0000 7f567d5f0740 10 client.0 _ll_register_callbacks cb 0x7f5668009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:04.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.692+0000 7f5673fff640 1 -- 172.21.15.28:0/635283706 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cdbd00 msgr2=0x5602d2cd5fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:04.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.692+0000 7f5673fff640 1 --2- 172.21.15.28:0/635283706 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cdbd00 0x5602d2cd5fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.692+0000 7f5673fff640 1 -- 172.21.15.28:0/635283706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cdca00 msgr2=0x5602d2cd64e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:04.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.692+0000 7f5673fff640 1 --2- 172.21.15.28:0/635283706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cdca00 0x5602d2cd64e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.692+0000 7f5673fff640 1 -- 172.21.15.28:0/635283706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602d2cd6ce0 con 0x5602d2cd3e80 2024-04-21T08:53:04.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.693+0000 7f567d5f0740 1 -- 172.21.15.28:0/635283706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5664017070 con 0x5602d2cd3e80 2024-04-21T08:53:04.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.694+0000 7f5673fff640 1 --2- 172.21.15.28:0/635283706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd3e80 0x5602d2ce4db0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5668003f20 tx=0x7f566801b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca12c46c03f808b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.694+0000 7f56537fe640 10 client.16383 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:04.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.696+0000 7f56537fe640 1 -- 172.21.15.28:0/635283706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5668035050 con 0x5602d2cd3e80 2024-04-21T08:53:04.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.696+0000 7f56537fe640 1 -- 172.21.15.28:0/635283706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f56680054d0 con 0x5602d2cd3e80 2024-04-21T08:53:04.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.696+0000 7f56537fe640 1 -- 172.21.15.28:0/635283706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f566802b070 con 0x5602d2cd3e80 2024-04-21T08:53:04.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.696+0000 7f567d5f0740 10 client.16383 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:04.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.696+0000 7f567d5f0740 1 -- 172.21.15.28:0/635283706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=26}) -- 0x5602d2d53190 con 0x5602d2cd3e80 2024-04-21T08:53:04.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.698+0000 7f56537fe640 1 -- 172.21.15.28:0/635283706 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f56680035f0 con 0x5602d2cd3e80 2024-04-21T08:53:04.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.700+0000 7f567d5f0740 10 client.16383 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:04.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.700+0000 7f567d5f0740 10 client.16383 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:53:04.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.700+0000 7f567d5f0740 20 client.16383 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:04.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.700+0000 7f567d5f0740 1 --2- 172.21.15.28:0/635283706 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5602d2d537a0 0x5602d2d55c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.700+0000 7f567d5f0740 4 client.16383 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:53:04.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.700+0000 7f567d5f0740 1 -- 172.21.15.28:0/635283706 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5602d2c45c10 con 0x5602d2d537a0 2024-04-21T08:53:04.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.701+0000 7f56737fe640 1 --2- 172.21.15.28:0/635283706 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5602d2d537a0 0x5602d2d55c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.707+0000 7f56737fe640 1 --2- 172.21.15.28:0/635283706 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5602d2d537a0 0x5602d2d55c30 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7c720f0956abfd24 server_cookie=f65db35129dc58e8 in_seq=0 out_seq=0 2024-04-21T08:53:04.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.708+0000 7f56537fe640 10 client.16383 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:53:04.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.708+0000 7f56537fe640 1 -- 172.21.15.28:0/635283706 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5602d2c45c10 con 0x5602d2d537a0 2024-04-21T08:53:04.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.708+0000 7f56537fe640 10 client.16383 handle_command_reply: tid=0 2024-04-21T08:53:04.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.708+0000 7f567d5f0740 1 client.16383 shutdown 2024-04-21T08:53:04.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.711+0000 7f567d5f0740 1 -- 172.21.15.28:0/635283706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd3e80 msgr2=0x5602d2ce4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.712+0000 7f567d5f0740 1 --2- 172.21.15.28:0/635283706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd3e80 0x5602d2ce4db0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5668003f20 tx=0x7f566801b630 comp rx=0 tx=0).stop 2024-04-21T08:53:04.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.715+0000 7f567d5f0740 1 -- 172.21.15.28:0/635283706 shutdown_connections 2024-04-21T08:53:04.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.715+0000 7f567d5f0740 1 --2- 172.21.15.28:0/635283706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602d2cdca00 0x5602d2cd64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.715+0000 7f567d5f0740 1 --2- 172.21.15.28:0/635283706 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5602d2d537a0 0x5602d2d55c30 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.715+0000 7f567d5f0740 1 --2- 172.21.15.28:0/635283706 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602d2cdbd00 0x5602d2cd5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.715+0000 7f567d5f0740 1 --2- 172.21.15.28:0/635283706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d2cd3e80 0x5602d2ce4db0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.715+0000 7f567d5f0740 1 -- 172.21.15.28:0/635283706 >> 172.21.15.28:0/635283706 conn(0x5602d2ccdba0 msgr2=0x5602d2cdb510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:04.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.715+0000 7f56537fe640 0 client.16383 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:53:04.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.720+0000 7f567d5f0740 1 -- 172.21.15.28:0/635283706 shutdown_connections 2024-04-21T08:53:04.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.724+0000 7f567d5f0740 1 -- 172.21.15.28:0/635283706 wait complete. 2024-04-21T08:53:04.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.724+0000 7f567d5f0740 20 client.16383 trim_cache size 0 max 16384 2024-04-21T08:53:04.727 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:53:04.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.728+0000 7f567aa01640 1 -- 172.21.15.28:0/2676537252 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f565c0a3f20 msgr2=0x7f565c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.728+0000 7f567aa01640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f565c0a3f20 0x7f565c0a63e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5664005d00 tx=0x7f5664005e50 comp rx=0 tx=0).stop 2024-04-21T08:53:04.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.728+0000 7f567aa01640 1 -- 172.21.15.28:0/2676537252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f567410e5f0 msgr2=0x7f567406cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.728+0000 7f567aa01640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f567410e5f0 0x7f567406cdc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f566c02cc20 tx=0x7f566c033750 comp rx=0 tx=0).stop 2024-04-21T08:53:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.730+0000 7f567aa01640 1 -- 172.21.15.28:0/2676537252 shutdown_connections 2024-04-21T08:53:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.731+0000 7f567aa01640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f565c0a3f20 0x7f565c0a63e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.731+0000 7f567aa01640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f567410e5f0 0x7f567406cdc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.731+0000 7f567aa01640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5674108f40 0x7f567406c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.731+0000 7f567aa01640 1 --2- 172.21.15.28:0/2676537252 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5674108580 0x7f56740737b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.731+0000 7f567aa01640 1 -- 172.21.15.28:0/2676537252 >> 172.21.15.28:0/2676537252 conn(0x7f5674076490 msgr2=0x7f5674077e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.733+0000 7f567aa01640 1 -- 172.21.15.28:0/2676537252 shutdown_connections 2024-04-21T08:53:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.734+0000 7f567aa01640 1 -- 172.21.15.28:0/2676537252 wait complete. 2024-04-21T08:53:04.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:53:04.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.955+0000 7f2c277fe640 1 -- 172.21.15.28:0/3279947849 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1802cc10 con 0x7f2c28104d50 2024-04-21T08:53:04.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.956+0000 7f2c2f211640 1 -- 172.21.15.28:0/3279947849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c28104d50 msgr2=0x7f2c28108d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.956+0000 7f2c2f211640 1 --2- 172.21.15.28:0/3279947849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c28104d50 0x7f2c28108d90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f2c18009900 tx=0x7f2c1802bc60 comp rx=0 tx=0).stop 2024-04-21T08:53:04.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.958+0000 7f2c2f211640 1 -- 172.21.15.28:0/3279947849 shutdown_connections 2024-04-21T08:53:04.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.958+0000 7f2c2f211640 1 --2- 172.21.15.28:0/3279947849 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c28109470 0x7f2c28111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.958+0000 7f2c2f211640 1 --2- 172.21.15.28:0/3279947849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c28104d50 0x7f2c28108d90 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.958+0000 7f2c2f211640 1 --2- 172.21.15.28:0/3279947849 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c281043a0 0x7f2c28104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.958+0000 7f2c2f211640 1 -- 172.21.15.28:0/3279947849 >> 172.21.15.28:0/3279947849 conn(0x7f2c280ffbc0 msgr2=0x7f2c28101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:04.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.960+0000 7f2c2f211640 1 -- 172.21.15.28:0/3279947849 shutdown_connections 2024-04-21T08:53:04.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.961+0000 7f2c2f211640 1 -- 172.21.15.28:0/3279947849 wait complete. 2024-04-21T08:53:04.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.962+0000 7f2c2f211640 1 Processor -- start 2024-04-21T08:53:04.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.962+0000 7f2c2f211640 1 -- start start 2024-04-21T08:53:04.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.962+0000 7f2c2f211640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c281043a0 0x7f2c2810fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.962+0000 7f2c2f211640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c28109470 0x7f2c281100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.962+0000 7f2c2f211640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c281a4a20 0x7f2c281a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.962+0000 7f2c2f211640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2c28077440 con 0x7f2c28109470 2024-04-21T08:53:04.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.962+0000 7f2c2f211640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2c280772c0 con 0x7f2c281a4a20 2024-04-21T08:53:04.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.962+0000 7f2c2f211640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2c280775c0 con 0x7f2c281043a0 2024-04-21T08:53:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.963+0000 7f2c2cfad640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c281043a0 0x7f2c2810fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.963+0000 7f2c2cfad640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c281043a0 0x7f2c2810fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:57654/0 (socket says 172.21.15.28:57654) 2024-04-21T08:53:04.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.963+0000 7f2c2cfad640 1 -- 172.21.15.28:0/3683665790 learned_addr learned my addr 172.21.15.28:0/3683665790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:04.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.964+0000 7f2c27fff640 1 --2- 172.21.15.28:0/3683665790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c28109470 0x7f2c281100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.964+0000 7f2c2d7ae640 1 --2- 172.21.15.28:0/3683665790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c281a4a20 0x7f2c281a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.965+0000 7f2c27fff640 1 -- 172.21.15.28:0/3683665790 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c281043a0 msgr2=0x7f2c2810fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.965+0000 7f2c27fff640 1 --2- 172.21.15.28:0/3683665790 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c281043a0 0x7f2c2810fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.965+0000 7f2c27fff640 1 -- 172.21.15.28:0/3683665790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c281a4a20 msgr2=0x7f2c281a4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.965+0000 7f2c27fff640 1 --2- 172.21.15.28:0/3683665790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c281a4a20 0x7f2c281a4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.965+0000 7f2c27fff640 1 -- 172.21.15.28:0/3683665790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c281a5540 con 0x7f2c28109470 2024-04-21T08:53:04.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.965+0000 7f2c27fff640 1 --2- 172.21.15.28:0/3683665790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c28109470 0x7f2c281100d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2c18036040 tx=0x7f2c1802c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.967+0000 7f2c25ffb640 1 -- 172.21.15.28:0/3683665790 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c18043020 con 0x7f2c28109470 2024-04-21T08:53:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.967+0000 7f2c2f211640 1 -- 172.21.15.28:0/3683665790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c281a57d0 con 0x7f2c28109470 2024-04-21T08:53:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.967+0000 7f2c2f211640 1 -- 172.21.15.28:0/3683665790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c2806bcb0 con 0x7f2c28109470 2024-04-21T08:53:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.970+0000 7f2c25ffb640 1 -- 172.21.15.28:0/3683665790 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c18003aa0 con 0x7f2c28109470 2024-04-21T08:53:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.970+0000 7f2c25ffb640 1 -- 172.21.15.28:0/3683665790 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c18005090 con 0x7f2c28109470 2024-04-21T08:53:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.974+0000 7f2c31e00740 1 Processor -- start 2024-04-21T08:53:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.974+0000 7f2c25ffb640 1 -- 172.21.15.28:0/3683665790 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2c18046020 con 0x7f2c28109470 2024-04-21T08:53:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.975+0000 7f2c25ffb640 1 --2- 172.21.15.28:0/3683665790 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2c100a3e20 0x7f2c100a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.975+0000 7f2c31e00740 1 -- start start 2024-04-21T08:53:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.975+0000 7f2c25ffb640 1 -- 172.21.15.28:0/3683665790 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f2c180f2560 con 0x7f2c28109470 2024-04-21T08:53:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.976+0000 7f2c31e00740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a434d0 0x561678a438b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.976+0000 7f2c31e00740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a43e80 0x561678a4c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.976+0000 7f2c31e00740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a4ca00 0x561678a4ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.976+0000 7f2c31e00740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5616789a81f0 con 0x561678a43e80 2024-04-21T08:53:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.976+0000 7f2c31e00740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5616789454b0 con 0x561678a434d0 2024-04-21T08:53:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.976+0000 7f2c31e00740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5616789a41b0 con 0x561678a4ca00 2024-04-21T08:53:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.976+0000 7f2c27fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a43e80 0x561678a4c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.977+0000 7f2c27fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a43e80 0x561678a4c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50326/0 (socket says 172.21.15.28:50326) 2024-04-21T08:53:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.977+0000 7f2c27fff640 1 -- 172.21.15.28:0/3156890413 learned_addr learned my addr 172.21.15.28:0/3156890413 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.978+0000 7f2c2d7ae640 1 --2- 172.21.15.28:0/3683665790 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2c100a3e20 0x7f2c100a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.978+0000 7f2c27fff640 1 -- 172.21.15.28:0/3156890413 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a4ca00 msgr2=0x561678a4ee50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.978+0000 7f2c27fff640 1 --2- 172.21.15.28:0/3156890413 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a4ca00 0x561678a4ee50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.978+0000 7f2c27fff640 1 -- 172.21.15.28:0/3156890413 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a434d0 msgr2=0x561678a438b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.978+0000 7f2c27fff640 1 --2- 172.21.15.28:0/3156890413 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a434d0 0x561678a438b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.978+0000 7f2c27fff640 1 -- 172.21.15.28:0/3156890413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616789371b0 con 0x561678a43e80 2024-04-21T08:53:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.981+0000 7f2c27fff640 1 --2- 172.21.15.28:0/3156890413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a43e80 0x561678a4c430 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f2c180bbc10 tx=0x7f2c180beba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518c2768caaf7999 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.982+0000 7f2c0effd640 1 -- 172.21.15.28:0/3156890413 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c180c35f0 con 0x561678a43e80 2024-04-21T08:53:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.982+0000 7f2c0effd640 1 -- 172.21.15.28:0/3156890413 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c180c7eb0 con 0x561678a43e80 2024-04-21T08:53:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.983+0000 7f2c31e00740 1 -- 172.21.15.28:0/3156890413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a43e80 msgr2=0x561678a4c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.983+0000 7f2c31e00740 1 --2- 172.21.15.28:0/3156890413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a43e80 0x561678a4c430 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f2c180bbc10 tx=0x7f2c180beba0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.984+0000 7f2c31e00740 1 -- 172.21.15.28:0/3156890413 shutdown_connections 2024-04-21T08:53:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.984+0000 7f2c31e00740 1 --2- 172.21.15.28:0/3156890413 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a4ca00 0x561678a4ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.984+0000 7f2c31e00740 1 --2- 172.21.15.28:0/3156890413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a43e80 0x561678a4c430 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.984+0000 7f2c31e00740 1 --2- 172.21.15.28:0/3156890413 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a434d0 0x561678a438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.984+0000 7f2c31e00740 1 -- 172.21.15.28:0/3156890413 >> 172.21.15.28:0/3156890413 conn(0x561678a3dba0 msgr2=0x561678a3df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.984+0000 7f2c2d7ae640 1 --2- 172.21.15.28:0/3683665790 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2c100a3e20 0x7f2c100a62e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000a4c0 tx=0x7f2c2000d040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.986+0000 7f2c31e00740 1 -- 172.21.15.28:0/3156890413 shutdown_connections 2024-04-21T08:53:04.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.986+0000 7f2c31e00740 1 -- 172.21.15.28:0/3156890413 wait complete. 2024-04-21T08:53:04.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.987+0000 7f2c31e00740 1 -- start start 2024-04-21T08:53:04.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.987+0000 7f2c31e00740 1 Processor -- start 2024-04-21T08:53:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.989+0000 7f2c31e00740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a434d0 0x561678a4c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.989+0000 7f2c31e00740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a43e80 0x561678a49700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.989+0000 7f2c31e00740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a4ca00 0x561678a49d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.990+0000 7f2c31e00740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561678a4f730 con 0x561678a434d0 2024-04-21T08:53:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.990+0000 7f2c2cfad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a434d0 0x561678a4c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.990+0000 7f2c2cfad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a434d0 0x561678a4c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50332/0 (socket says 172.21.15.28:50332) 2024-04-21T08:53:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.990+0000 7f2c2cfad640 1 -- 172.21.15.28:0/676879727 learned_addr learned my addr 172.21.15.28:0/676879727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.990+0000 7f2c31e00740 1 -- 172.21.15.28:0/676879727 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561678961500 con 0x561678a4ca00 2024-04-21T08:53:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.990+0000 7f2c2d7ae640 1 --2- 172.21.15.28:0/676879727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a4ca00 0x561678a49d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.990+0000 7f2c31e00740 1 -- 172.21.15.28:0/676879727 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x561678a4f8b0 con 0x561678a43e80 2024-04-21T08:53:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.990+0000 7f2c31e00740 10 client.0 _ll_register_callbacks cb 0x56167892fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.990+0000 7f2c27fff640 1 --2- 172.21.15.28:0/676879727 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a43e80 0x561678a49700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.991+0000 7f2c2d7ae640 1 -- 172.21.15.28:0/676879727 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a43e80 msgr2=0x561678a49700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.991+0000 7f2c2d7ae640 1 --2- 172.21.15.28:0/676879727 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a43e80 0x561678a49700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.991+0000 7f2c2d7ae640 1 -- 172.21.15.28:0/676879727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a434d0 msgr2=0x561678a4c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.991+0000 7f2c2d7ae640 1 --2- 172.21.15.28:0/676879727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a434d0 0x561678a4c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.991+0000 7f2c2d7ae640 1 -- 172.21.15.28:0/676879727 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616786ba3f0 con 0x561678a4ca00 2024-04-21T08:53:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.992+0000 7f2c2d7ae640 1 --2- 172.21.15.28:0/676879727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a4ca00 0x561678a49d10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009aa0 tx=0x7f2c20004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c49caebe82b958bb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:04.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.995+0000 7f2c07fff640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:53:04.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.995+0000 7f2c07fff640 1 -- 172.21.15.28:0/676879727 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c200205f0 con 0x561678a4ca00 2024-04-21T08:53:04.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.995+0000 7f2c07fff640 1 -- 172.21.15.28:0/676879727 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f2c20020790 con 0x561678a4ca00 2024-04-21T08:53:04.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.997+0000 7f2c31e00740 1 -- 172.21.15.28:0/676879727 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561678947930 con 0x561678a4ca00 2024-04-21T08:53:04.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.998+0000 7f2c07fff640 1 -- 172.21.15.28:0/676879727 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2c20020990 con 0x561678a4ca00 2024-04-21T08:53:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.998+0000 7f2c31e00740 10 client.25760 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:04.998+0000 7f2c31e00740 1 -- 172.21.15.28:0/676879727 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5616787847d0 con 0x561678a4ca00 2024-04-21T08:53:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.000+0000 7f2c07fff640 1 -- 172.21.15.28:0/676879727 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561678947930 con 0x561678a4ca00 2024-04-21T08:53:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.000+0000 7f2c07fff640 1 -- 172.21.15.28:0/676879727 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f2c200289b0 con 0x561678a4ca00 2024-04-21T08:53:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.001+0000 7f2c31e00740 10 client.25760 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.001+0000 7f2c31e00740 10 client.25760 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.001+0000 7f2c31e00740 20 client.25760 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.001+0000 7f2c31e00740 1 --2- 172.21.15.28:0/676879727 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561678a5d100 0x561678a5f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.001+0000 7f2c31e00740 4 client.25760 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.001+0000 7f2c31e00740 1 -- 172.21.15.28:0/676879727 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616789b5c10 con 0x561678a5d100 2024-04-21T08:53:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.003+0000 7f2c2cfad640 1 --2- 172.21.15.28:0/676879727 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561678a5d100 0x561678a5f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:05.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.004+0000 7f2c07fff640 10 client.25760 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:05.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.004+0000 7f2c2cfad640 1 --2- 172.21.15.28:0/676879727 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561678a5d100 0x561678a5f560 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f299fff4ee85ff9 server_cookie=1248cdac8a4b0dc5 in_seq=0 out_seq=0 2024-04-21T08:53:05.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.006+0000 7f2c07fff640 1 -- 172.21.15.28:0/676879727 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5616789b5c10 con 0x561678a5d100 2024-04-21T08:53:05.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.006+0000 7f2c07fff640 10 client.25760 handle_command_reply: tid=0 2024-04-21T08:53:05.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.006+0000 7f2c31e00740 1 client.25760 shutdown 2024-04-21T08:53:05.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.013+0000 7f2c31e00740 1 -- 172.21.15.28:0/676879727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a4ca00 msgr2=0x561678a49d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:05.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.013+0000 7f2c31e00740 1 --2- 172.21.15.28:0/676879727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a4ca00 0x561678a49d10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009aa0 tx=0x7f2c20004ad0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.017+0000 7f2c31e00740 1 -- 172.21.15.28:0/676879727 shutdown_connections 2024-04-21T08:53:05.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.017+0000 7f2c31e00740 1 --2- 172.21.15.28:0/676879727 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561678a5d100 0x561678a5f560 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.017+0000 7f2c31e00740 1 --2- 172.21.15.28:0/676879727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a4ca00 0x561678a49d10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.017+0000 7f2c31e00740 1 --2- 172.21.15.28:0/676879727 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a43e80 0x561678a49700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.017+0000 7f2c31e00740 1 --2- 172.21.15.28:0/676879727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a434d0 0x561678a4c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.017+0000 7f2c31e00740 1 -- 172.21.15.28:0/676879727 >> 172.21.15.28:0/676879727 conn(0x561678a3dba0 msgr2=0x561678a3f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:05.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.017+0000 7f2c07fff640 0 client.25760 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:05.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.018+0000 7f2c31e00740 1 -- 172.21.15.28:0/676879727 shutdown_connections 2024-04-21T08:53:05.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.019+0000 7f2c31e00740 1 -- 172.21.15.28:0/676879727 wait complete. 2024-04-21T08:53:05.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.020+0000 7f2c31e00740 20 client.25760 trim_cache size 0 max 16384 2024-04-21T08:53:05.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.051+0000 7f2c31e00740 1 Processor -- start 2024-04-21T08:53:05.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.051+0000 7f2c31e00740 1 -- start start 2024-04-21T08:53:05.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.052+0000 7f2c31e00740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a434d0 0x561678a586c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.052+0000 7f2c31e00740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a43e80 0x561678a4be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.052+0000 7f2c31e00740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a4ca00 0x561678a4c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.052+0000 7f2c2cfad640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a434d0 0x561678a586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.052+0000 7f2c2cfad640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a434d0 0x561678a586c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:57682/0 (socket says 172.21.15.28:57682) 2024-04-21T08:53:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.052+0000 7f2c27fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a43e80 0x561678a4be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.052+0000 7f2c27fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a43e80 0x561678a4be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:58642/0 (socket says 172.21.15.28:58642) 2024-04-21T08:53:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.053+0000 7f2c2d7ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a4ca00 0x561678a4c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.053+0000 7f2c2d7ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a4ca00 0x561678a4c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50336/0 (socket says 172.21.15.28:50336) 2024-04-21T08:53:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.053+0000 7f2c2cfad640 1 -- 172.21.15.28:0/477997063 learned_addr learned my addr 172.21.15.28:0/477997063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.053+0000 7f2c31e00740 1 -- 172.21.15.28:0/477997063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2c140128c0 con 0x561678a4ca00 2024-04-21T08:53:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.053+0000 7f2c31e00740 1 -- 172.21.15.28:0/477997063 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2c14010190 con 0x561678a43e80 2024-04-21T08:53:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.053+0000 7f2c31e00740 1 -- 172.21.15.28:0/477997063 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1400dfe0 con 0x561678a434d0 2024-04-21T08:53:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.056+0000 7f2c27fff640 1 -- 172.21.15.28:0/477997063 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a434d0 msgr2=0x561678a586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.056+0000 7f2c27fff640 1 --2- 172.21.15.28:0/477997063 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a434d0 0x561678a586c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.056+0000 7f2c27fff640 1 -- 172.21.15.28:0/477997063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a4ca00 msgr2=0x561678a4c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.056+0000 7f2c27fff640 1 --2- 172.21.15.28:0/477997063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a4ca00 0x561678a4c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.056+0000 7f2c27fff640 1 -- 172.21.15.28:0/477997063 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561678a5b700 con 0x561678a43e80 2024-04-21T08:53:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.056+0000 7f2c2d7ae640 1 --2- 172.21.15.28:0/477997063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a4ca00 0x561678a4c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:53:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.056+0000 7f2c27fff640 1 --2- 172.21.15.28:0/477997063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a43e80 0x561678a4be30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x561678a3ec10 tx=0x7f2c180becd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=170d70ce0a9c5ed8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.057+0000 7f2c0e7fc640 1 -- 172.21.15.28:0/477997063 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c180c35f0 con 0x561678a43e80 2024-04-21T08:53:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.057+0000 7f2c0e7fc640 1 -- 172.21.15.28:0/477997063 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c180c7eb0 con 0x561678a43e80 2024-04-21T08:53:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.057+0000 7f2c0e7fc640 1 -- 172.21.15.28:0/477997063 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c180c7ca0 con 0x561678a43e80 2024-04-21T08:53:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.058+0000 7f2c31e00740 1 -- 172.21.15.28:0/477997063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a43e80 msgr2=0x561678a4be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.059+0000 7f2c31e00740 1 --2- 172.21.15.28:0/477997063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a43e80 0x561678a4be30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x561678a3ec10 tx=0x7f2c180becd0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.060+0000 7f2c31e00740 1 -- 172.21.15.28:0/477997063 shutdown_connections 2024-04-21T08:53:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.060+0000 7f2c31e00740 1 --2- 172.21.15.28:0/477997063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a4ca00 0x561678a4c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.060+0000 7f2c31e00740 1 --2- 172.21.15.28:0/477997063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a43e80 0x561678a4be30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.060+0000 7f2c31e00740 1 --2- 172.21.15.28:0/477997063 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a434d0 0x561678a586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.060+0000 7f2c31e00740 1 -- 172.21.15.28:0/477997063 >> 172.21.15.28:0/477997063 conn(0x561678a3dba0 msgr2=0x561678a3f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:05.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.062+0000 7f2c31e00740 1 -- 172.21.15.28:0/477997063 shutdown_connections 2024-04-21T08:53:05.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.063+0000 7f2c31e00740 1 -- 172.21.15.28:0/477997063 wait complete. 2024-04-21T08:53:05.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.064+0000 7f2c31e00740 1 -- start start 2024-04-21T08:53:05.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.064+0000 7f2c31e00740 1 Processor -- start 2024-04-21T08:53:05.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.067+0000 7f2c31e00740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a434d0 0x561678a54dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:05.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.067+0000 7f2c31e00740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a43e80 0x561678a45fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.067+0000 7f2c31e00740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a4ca00 0x561678a464f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.067+0000 7f2c31e00740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1400e650 con 0x561678a434d0 2024-04-21T08:53:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.067+0000 7f2c27fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a43e80 0x561678a45fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.067+0000 7f2c27fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a43e80 0x561678a45fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:57684/0 (socket says 172.21.15.28:57684) 2024-04-21T08:53:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.067+0000 7f2c27fff640 1 -- 172.21.15.28:0/2235077960 learned_addr learned my addr 172.21.15.28:0/2235077960 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.068+0000 7f2c31e00740 1 -- 172.21.15.28:0/2235077960 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1400fac0 con 0x561678a4ca00 2024-04-21T08:53:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.068+0000 7f2c31e00740 1 -- 172.21.15.28:0/2235077960 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1400ed20 con 0x561678a43e80 2024-04-21T08:53:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.068+0000 7f2c2d7ae640 1 --2- 172.21.15.28:0/2235077960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a4ca00 0x561678a464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.068+0000 7f2c31e00740 10 client.0 _ll_register_callbacks cb 0x7f2c20016a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:05.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.068+0000 7f2c27fff640 1 -- 172.21.15.28:0/2235077960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a4ca00 msgr2=0x561678a464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:05.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.068+0000 7f2c27fff640 1 --2- 172.21.15.28:0/2235077960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a4ca00 0x561678a464f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.068+0000 7f2c27fff640 1 -- 172.21.15.28:0/2235077960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a434d0 msgr2=0x561678a54dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:05.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.068+0000 7f2c27fff640 1 --2- 172.21.15.28:0/2235077960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a434d0 0x561678a54dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.068+0000 7f2c27fff640 1 -- 172.21.15.28:0/2235077960 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561678a46a30 con 0x561678a43e80 2024-04-21T08:53:05.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.068+0000 7f2c31e00740 1 -- 172.21.15.28:0/2235077960 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c180c7eb0 con 0x561678a43e80 2024-04-21T08:53:05.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.069+0000 7f2c27fff640 1 --2- 172.21.15.28:0/2235077960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a43e80 0x561678a45fb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2c18004650 tx=0x7f2c1802bc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fae628964d509280 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:05.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.069+0000 7f2c07fff640 10 client.25756 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:53:05.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.070+0000 7f2c07fff640 1 -- 172.21.15.28:0/2235077960 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c180c35f0 con 0x561678a43e80 2024-04-21T08:53:05.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.070+0000 7f2c07fff640 1 -- 172.21.15.28:0/2235077960 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f2c180c7a20 con 0x561678a43e80 2024-04-21T08:53:05.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.070+0000 7f2c07fff640 1 -- 172.21.15.28:0/2235077960 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2c180bd540 con 0x561678a43e80 2024-04-21T08:53:05.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.073+0000 7f2c31e00740 10 client.25756 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:05.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.073+0000 7f2c31e00740 1 -- 172.21.15.28:0/2235077960 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({fsmap=26}) -- 0x561678ac3150 con 0x561678a43e80 2024-04-21T08:53:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.074+0000 7f2c07fff640 1 -- 172.21.15.28:0/2235077960 <== mon.1 v2:172.21.15.171:3300/0 4 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f2c180c3120 con 0x561678a43e80 2024-04-21T08:53:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.076+0000 7f2c31e00740 10 client.25756 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.076+0000 7f2c31e00740 10 client.25756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.076+0000 7f2c31e00740 20 client.25756 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:05.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.076+0000 7f2c31e00740 1 --2- 172.21.15.28:0/2235077960 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561678ac31c0 0x561678ac55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:05.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.076+0000 7f2c31e00740 4 client.25756 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:53:05.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.076+0000 7f2c31e00740 1 -- 172.21.15.28:0/2235077960 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f2c180c7eb0 con 0x561678ac31c0 2024-04-21T08:53:05.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.076+0000 7f2c2cfad640 1 --2- 172.21.15.28:0/2235077960 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561678ac31c0 0x561678ac55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:05.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.082+0000 7f2c2cfad640 1 --2- 172.21.15.28:0/2235077960 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561678ac31c0 0x561678ac55d0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19548f9717ec2248 server_cookie=6fd9e42fd4c240da in_seq=0 out_seq=0 2024-04-21T08:53:05.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.082+0000 7f2c07fff640 10 client.25756 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.083+0000 7f2c07fff640 1 -- 172.21.15.28:0/2235077960 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f2c180c7eb0 con 0x561678ac31c0 2024-04-21T08:53:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.083+0000 7f2c07fff640 10 client.25756 handle_command_reply: tid=0 2024-04-21T08:53:05.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.084+0000 7f2c31e00740 1 client.25756 shutdown 2024-04-21T08:53:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.085+0000 7f2c31e00740 1 -- 172.21.15.28:0/2235077960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a43e80 msgr2=0x561678a45fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.085+0000 7f2c31e00740 1 --2- 172.21.15.28:0/2235077960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a43e80 0x561678a45fb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2c18004650 tx=0x7f2c1802bc60 comp rx=0 tx=0).stop 2024-04-21T08:53:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.086+0000 7f2c31e00740 1 -- 172.21.15.28:0/2235077960 shutdown_connections 2024-04-21T08:53:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.086+0000 7f2c31e00740 1 --2- 172.21.15.28:0/2235077960 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561678ac31c0 0x561678ac55d0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.086+0000 7f2c31e00740 1 --2- 172.21.15.28:0/2235077960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561678a4ca00 0x561678a464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.086+0000 7f2c31e00740 1 --2- 172.21.15.28:0/2235077960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561678a43e80 0x561678a45fb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.086+0000 7f2c31e00740 1 --2- 172.21.15.28:0/2235077960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561678a434d0 0x561678a54dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.086+0000 7f2c31e00740 1 -- 172.21.15.28:0/2235077960 >> 172.21.15.28:0/2235077960 conn(0x561678a3dba0 msgr2=0x561678a3eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.086+0000 7f2c07fff640 0 client.25756 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.086+0000 7f2c31e00740 1 -- 172.21.15.28:0/2235077960 shutdown_connections 2024-04-21T08:53:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.086+0000 7f2c31e00740 1 -- 172.21.15.28:0/2235077960 wait complete. 2024-04-21T08:53:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.086+0000 7f2c31e00740 20 client.25756 trim_cache size 0 max 16384 2024-04-21T08:53:05.088 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:53:05.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.091+0000 7f2c2f211640 1 -- 172.21.15.28:0/3683665790 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2c100a3e20 msgr2=0x7f2c100a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:05.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.091+0000 7f2c2f211640 1 --2- 172.21.15.28:0/3683665790 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2c100a3e20 0x7f2c100a62e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000a4c0 tx=0x7f2c2000d040 comp rx=0 tx=0).stop 2024-04-21T08:53:05.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.091+0000 7f2c2f211640 1 -- 172.21.15.28:0/3683665790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c28109470 msgr2=0x7f2c281100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:05.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.091+0000 7f2c2f211640 1 --2- 172.21.15.28:0/3683665790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c28109470 0x7f2c281100d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2c18036040 tx=0x7f2c1802c890 comp rx=0 tx=0).stop 2024-04-21T08:53:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.092+0000 7f2c2f211640 1 -- 172.21.15.28:0/3683665790 shutdown_connections 2024-04-21T08:53:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.092+0000 7f2c2f211640 1 --2- 172.21.15.28:0/3683665790 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2c100a3e20 0x7f2c100a62e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.092+0000 7f2c2f211640 1 --2- 172.21.15.28:0/3683665790 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2c281a4a20 0x7f2c281a4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.092+0000 7f2c2f211640 1 --2- 172.21.15.28:0/3683665790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c28109470 0x7f2c281100d0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.092+0000 7f2c2f211640 1 --2- 172.21.15.28:0/3683665790 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2c281043a0 0x7f2c2810fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.092+0000 7f2c2f211640 1 -- 172.21.15.28:0/3683665790 >> 172.21.15.28:0/3683665790 conn(0x7f2c280ffbc0 msgr2=0x7f2c28101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.092+0000 7f2c2f211640 1 -- 172.21.15.28:0/3683665790 shutdown_connections 2024-04-21T08:53:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:05.093+0000 7f2c2f211640 1 -- 172.21.15.28:0/3683665790 wait complete. 2024-04-21T08:53:05.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:04 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1465307774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:53:05.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:04 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1465307774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:53:05.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:04 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1465307774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:53:06.116 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:53:06.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:53:06.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:53:06.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.326+0000 7faf836d3640 1 -- 172.21.15.28:0/2662861922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf7c100e20 msgr2=0x7faf7c104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.326+0000 7faf836d3640 1 --2- 172.21.15.28:0/2662861922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf7c100e20 0x7faf7c104990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7faf78009860 tx=0x7faf7802bb70 comp rx=0 tx=0).stop 2024-04-21T08:53:06.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.327+0000 7faf836d3640 1 -- 172.21.15.28:0/2662861922 shutdown_connections 2024-04-21T08:53:06.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.327+0000 7faf836d3640 1 --2- 172.21.15.28:0/2662861922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf7c100e20 0x7faf7c104990 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.327+0000 7faf836d3640 1 --2- 172.21.15.28:0/2662861922 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faf7c100460 0x7faf7c1008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.327+0000 7faf836d3640 1 --2- 172.21.15.28:0/2662861922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf7c106f30 0x7faf7c107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.327+0000 7faf836d3640 1 -- 172.21.15.28:0/2662861922 >> 172.21.15.28:0/2662861922 conn(0x7faf7c0773f0 msgr2=0x7faf7c0fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:06.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.328+0000 7faf836d3640 1 -- 172.21.15.28:0/2662861922 shutdown_connections 2024-04-21T08:53:06.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.329+0000 7faf836d3640 1 -- 172.21.15.28:0/2662861922 wait complete. 2024-04-21T08:53:06.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.330+0000 7faf836d3640 1 Processor -- start 2024-04-21T08:53:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.330+0000 7faf836d3640 1 -- start start 2024-04-21T08:53:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.330+0000 7faf836d3640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faf7c100460 0x7faf7c072a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.330+0000 7faf836d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf7c100e20 0x7faf7c072f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.330+0000 7faf836d3640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf7c106f30 0x7faf7c1a3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.330+0000 7faf836d3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c109f70 con 0x7faf7c100e20 2024-04-21T08:53:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.330+0000 7faf836d3640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c109df0 con 0x7faf7c106f30 2024-04-21T08:53:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.330+0000 7faf836d3640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c10a0f0 con 0x7faf7c100460 2024-04-21T08:53:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.330+0000 7faf80c6e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf7c100e20 0x7faf7c072f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.331+0000 7faf80c6e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf7c100e20 0x7faf7c072f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50362/0 (socket says 172.21.15.28:50362) 2024-04-21T08:53:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.331+0000 7faf80c6e640 1 -- 172.21.15.28:0/476820610 learned_addr learned my addr 172.21.15.28:0/476820610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.331+0000 7faf8146f640 1 --2- 172.21.15.28:0/476820610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faf7c100460 0x7faf7c072a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.331+0000 7faf81c70640 1 --2- 172.21.15.28:0/476820610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf7c106f30 0x7faf7c1a3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.331+0000 7faf80c6e640 1 -- 172.21.15.28:0/476820610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faf7c100460 msgr2=0x7faf7c072a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.331+0000 7faf80c6e640 1 --2- 172.21.15.28:0/476820610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faf7c100460 0x7faf7c072a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.331+0000 7faf80c6e640 1 -- 172.21.15.28:0/476820610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf7c106f30 msgr2=0x7faf7c1a3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.331+0000 7faf80c6e640 1 --2- 172.21.15.28:0/476820610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf7c106f30 0x7faf7c1a3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.331+0000 7faf80c6e640 1 -- 172.21.15.28:0/476820610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf7c1a3c60 con 0x7faf7c100e20 2024-04-21T08:53:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.331+0000 7faf81c70640 1 --2- 172.21.15.28:0/476820610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf7c106f30 0x7faf7c1a3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:53:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.332+0000 7faf80c6e640 1 --2- 172.21.15.28:0/476820610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf7c100e20 0x7faf7c072f90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7faf7400eab0 tx=0x7faf7400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.334+0000 7faf727fc640 1 -- 172.21.15.28:0/476820610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7400ce40 con 0x7faf7c100e20 2024-04-21T08:53:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.334+0000 7faf836d3640 1 -- 172.21.15.28:0/476820610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf7c1a3f50 con 0x7faf7c100e20 2024-04-21T08:53:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.334+0000 7faf836d3640 1 -- 172.21.15.28:0/476820610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf7c1ab7e0 con 0x7faf7c100e20 2024-04-21T08:53:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.334+0000 7faf727fc640 1 -- 172.21.15.28:0/476820610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf74011920 con 0x7faf7c100e20 2024-04-21T08:53:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.334+0000 7faf727fc640 1 -- 172.21.15.28:0/476820610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf74010a30 con 0x7faf7c100e20 2024-04-21T08:53:06.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.338+0000 7faf8639d740 1 Processor -- start 2024-04-21T08:53:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.338+0000 7faf727fc640 1 -- 172.21.15.28:0/476820610 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faf74010bd0 con 0x7faf7c100e20 2024-04-21T08:53:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.339+0000 7faf727fc640 1 --2- 172.21.15.28:0/476820610 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faf680a3e20 0x7faf680a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.340+0000 7faf727fc640 1 -- 172.21.15.28:0/476820610 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7faf740d0fc0 con 0x7faf7c100e20 2024-04-21T08:53:06.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.340+0000 7faf8639d740 1 -- start start 2024-04-21T08:53:06.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.340+0000 7faf81c70640 1 --2- 172.21.15.28:0/476820610 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faf680a3e20 0x7faf680a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf8639d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea58e0 0x56087dea5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf8639d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087dea6290 0x56087deae840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf8639d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087deaee10 0x56087deb1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf8639d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56087de09f00 con 0x56087dea6290 2024-04-21T08:53:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf8639d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56087dda7590 con 0x56087dea58e0 2024-04-21T08:53:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf8639d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56087de05ec0 con 0x56087deaee10 2024-04-21T08:53:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf80c6e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087dea6290 0x56087deae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf80c6e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087dea6290 0x56087deae840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50364/0 (socket says 172.21.15.28:50364) 2024-04-21T08:53:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf80c6e640 1 -- 172.21.15.28:0/518362980 learned_addr learned my addr 172.21.15.28:0/518362980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf81c70640 1 --2- 172.21.15.28:0/476820610 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faf680a3e20 0x7faf680a62e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7faf780057c0 tx=0x7faf78005710 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf81c70640 1 --2- 172.21.15.28:0/518362980 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087deaee10 0x56087deb1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf80c6e640 1 -- 172.21.15.28:0/518362980 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087deaee10 msgr2=0x56087deb1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf80c6e640 1 --2- 172.21.15.28:0/518362980 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087deaee10 0x56087deb1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf80c6e640 1 -- 172.21.15.28:0/518362980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea58e0 msgr2=0x56087dea5cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf80c6e640 1 --2- 172.21.15.28:0/518362980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea58e0 0x56087dea5cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.341+0000 7faf80c6e640 1 -- 172.21.15.28:0/518362980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56087dbe47e0 con 0x56087dea6290 2024-04-21T08:53:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.342+0000 7faf80c6e640 1 --2- 172.21.15.28:0/518362980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087dea6290 0x56087deae840 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7faf7400ea80 tx=0x7faf7409ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895dcc5ab4b1ba1b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.342+0000 7faf677fe640 1 -- 172.21.15.28:0/518362980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf740a57d0 con 0x56087dea6290 2024-04-21T08:53:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.342+0000 7faf677fe640 1 -- 172.21.15.28:0/518362980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf740bd0c0 con 0x56087dea6290 2024-04-21T08:53:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.343+0000 7faf677fe640 1 -- 172.21.15.28:0/518362980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf740a5340 con 0x56087dea6290 2024-04-21T08:53:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.343+0000 7faf8639d740 1 -- 172.21.15.28:0/518362980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087dea6290 msgr2=0x56087deae840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.343+0000 7faf8639d740 1 --2- 172.21.15.28:0/518362980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087dea6290 0x56087deae840 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7faf7400ea80 tx=0x7faf7409ca60 comp rx=0 tx=0).stop 2024-04-21T08:53:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.344+0000 7faf8639d740 1 -- 172.21.15.28:0/518362980 shutdown_connections 2024-04-21T08:53:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.344+0000 7faf8639d740 1 --2- 172.21.15.28:0/518362980 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087deaee10 0x56087deb1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.344+0000 7faf8639d740 1 --2- 172.21.15.28:0/518362980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087dea6290 0x56087deae840 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.344+0000 7faf8639d740 1 --2- 172.21.15.28:0/518362980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea58e0 0x56087dea5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.344+0000 7faf8639d740 1 -- 172.21.15.28:0/518362980 >> 172.21.15.28:0/518362980 conn(0x56087de9ffb0 msgr2=0x56087dea0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.346+0000 7faf8639d740 1 -- 172.21.15.28:0/518362980 shutdown_connections 2024-04-21T08:53:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.346+0000 7faf8639d740 1 -- 172.21.15.28:0/518362980 wait complete. 2024-04-21T08:53:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.346+0000 7faf8639d740 1 -- start start 2024-04-21T08:53:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.346+0000 7faf8639d740 1 Processor -- start 2024-04-21T08:53:06.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.347+0000 7faf8639d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087dea58e0 0x56087deabca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.347+0000 7faf8639d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea6290 0x56087deac1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.347+0000 7faf8639d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087deaee10 0x56087deac720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.347+0000 7faf8639d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56087deb1e20 con 0x56087deaee10 2024-04-21T08:53:06.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.347+0000 7faf8639d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56087ddc3580 con 0x56087dea6290 2024-04-21T08:53:06.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.347+0000 7faf8639d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56087deb1fa0 con 0x56087dea58e0 2024-04-21T08:53:06.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.347+0000 7faf8639d740 10 client.0 _ll_register_callbacks cb 0x56087ddc9650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:06.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.348+0000 7faf81c70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087deaee10 0x56087deac720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.348+0000 7faf81c70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087deaee10 0x56087deac720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50374/0 (socket says 172.21.15.28:50374) 2024-04-21T08:53:06.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.348+0000 7faf81c70640 1 -- 172.21.15.28:0/2111503922 learned_addr learned my addr 172.21.15.28:0/2111503922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:06.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.348+0000 7faf81c70640 1 -- 172.21.15.28:0/2111503922 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087dea58e0 msgr2=0x56087deabca0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:06.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.348+0000 7faf81c70640 1 --2- 172.21.15.28:0/2111503922 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087dea58e0 0x56087deabca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.348+0000 7faf81c70640 1 -- 172.21.15.28:0/2111503922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea6290 msgr2=0x56087deac1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:06.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.348+0000 7faf81c70640 1 --2- 172.21.15.28:0/2111503922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea6290 0x56087deac1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.348+0000 7faf81c70640 1 -- 172.21.15.28:0/2111503922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56087dc44f70 con 0x56087deaee10 2024-04-21T08:53:06.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.349+0000 7faf81c70640 1 --2- 172.21.15.28:0/2111503922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087deaee10 0x56087deac720 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7faf78009830 tx=0x7faf78005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f097dafd410ccf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:06.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.349+0000 7faf64ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:06.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.349+0000 7faf64ff9640 1 -- 172.21.15.28:0/2111503922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7804f070 con 0x56087deaee10 2024-04-21T08:53:06.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.349+0000 7faf64ff9640 1 -- 172.21.15.28:0/2111503922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7faf780035f0 con 0x56087deaee10 2024-04-21T08:53:06.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.349+0000 7faf8639d740 1 -- 172.21.15.28:0/2111503922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf740bd0c0 con 0x56087deaee10 2024-04-21T08:53:06.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.349+0000 7faf64ff9640 1 -- 172.21.15.28:0/2111503922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faf7805b050 con 0x56087deaee10 2024-04-21T08:53:06.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.349+0000 7faf8639d740 10 client.16407 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:06.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.349+0000 7faf8639d740 1 -- 172.21.15.28:0/2111503922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56087dadc170 con 0x56087deaee10 2024-04-21T08:53:06.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.350+0000 7faf64ff9640 1 -- 172.21.15.28:0/2111503922 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf740bd0c0 con 0x56087deaee10 2024-04-21T08:53:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.350+0000 7faf64ff9640 1 -- 172.21.15.28:0/2111503922 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7faf7803f820 con 0x56087deaee10 2024-04-21T08:53:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.350+0000 7faf8639d740 10 client.16407 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.350+0000 7faf8639d740 10 client.16407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.350+0000 7faf8639d740 20 client.16407 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.350+0000 7faf8639d740 1 --2- 172.21.15.28:0/2111503922 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56087debf390 0x56087dec1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.350+0000 7faf8639d740 4 client.16407 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.350+0000 7faf8639d740 1 -- 172.21.15.28:0/2111503922 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56087de17920 con 0x56087debf390 2024-04-21T08:53:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.351+0000 7faf8146f640 1 --2- 172.21.15.28:0/2111503922 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56087debf390 0x56087dec1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.352+0000 7faf8146f640 1 --2- 172.21.15.28:0/2111503922 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56087debf390 0x56087dec1830 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=481c38a249249564 server_cookie=99a1c87eb25f5ad8 in_seq=0 out_seq=0 2024-04-21T08:53:06.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.352+0000 7faf64ff9640 10 client.16407 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:06.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.353+0000 7faf64ff9640 1 -- 172.21.15.28:0/2111503922 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56087de17920 con 0x56087debf390 2024-04-21T08:53:06.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.353+0000 7faf64ff9640 10 client.16407 handle_command_reply: tid=0 2024-04-21T08:53:06.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.353+0000 7faf8639d740 1 client.16407 shutdown 2024-04-21T08:53:06.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.353+0000 7faf8639d740 1 -- 172.21.15.28:0/2111503922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087deaee10 msgr2=0x56087deac720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.353+0000 7faf8639d740 1 --2- 172.21.15.28:0/2111503922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087deaee10 0x56087deac720 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7faf78009830 tx=0x7faf78005f90 comp rx=0 tx=0).stop 2024-04-21T08:53:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.353+0000 7faf8639d740 1 -- 172.21.15.28:0/2111503922 shutdown_connections 2024-04-21T08:53:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.353+0000 7faf8639d740 1 --2- 172.21.15.28:0/2111503922 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56087debf390 0x56087dec1830 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.353+0000 7faf8639d740 1 --2- 172.21.15.28:0/2111503922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087deaee10 0x56087deac720 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.353+0000 7faf8639d740 1 --2- 172.21.15.28:0/2111503922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea6290 0x56087deac1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.353+0000 7faf8639d740 1 --2- 172.21.15.28:0/2111503922 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087dea58e0 0x56087deabca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.353+0000 7faf64ff9640 0 client.16407 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.353+0000 7faf8639d740 1 -- 172.21.15.28:0/2111503922 >> 172.21.15.28:0/2111503922 conn(0x56087de9ffb0 msgr2=0x56087dea1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.354+0000 7faf8639d740 1 -- 172.21.15.28:0/2111503922 shutdown_connections 2024-04-21T08:53:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.354+0000 7faf8639d740 1 -- 172.21.15.28:0/2111503922 wait complete. 2024-04-21T08:53:06.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.354+0000 7faf8639d740 20 client.16407 trim_cache size 0 max 16384 2024-04-21T08:53:06.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:05 smithi152 ceph-mon[31401]: pgmap v60: 129 pgs: 129 active+clean; 13 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 123 MiB/s wr, 15.19k op/s 2024-04-21T08:53:06.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:05 smithi028 ceph-mon[28003]: pgmap v60: 129 pgs: 129 active+clean; 13 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 123 MiB/s wr, 15.19k op/s 2024-04-21T08:53:06.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:05 smithi171 ceph-mon[31358]: pgmap v60: 129 pgs: 129 active+clean; 13 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 123 MiB/s wr, 15.19k op/s 2024-04-21T08:53:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.391+0000 7faf8639d740 1 Processor -- start 2024-04-21T08:53:06.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.391+0000 7faf8639d740 1 -- start start 2024-04-21T08:53:06.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.391+0000 7faf8639d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087dea58e0 0x56087deae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.392+0000 7faf8639d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea6290 0x56087debadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.392+0000 7faf8639d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087deaee10 0x56087dea1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.392+0000 7faf8639d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf600127a0 con 0x56087dea58e0 2024-04-21T08:53:06.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.392+0000 7faf8639d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7faf60010070 con 0x56087dea6290 2024-04-21T08:53:06.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.392+0000 7faf8639d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7faf6000dec0 con 0x56087deaee10 2024-04-21T08:53:06.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.392+0000 7faf81c70640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087deaee10 0x56087dea1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.392+0000 7faf81c70640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087deaee10 0x56087dea1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:57708/0 (socket says 172.21.15.28:57708) 2024-04-21T08:53:06.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.392+0000 7faf81c70640 1 -- 172.21.15.28:0/2168204833 learned_addr learned my addr 172.21.15.28:0/2168204833 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:06.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.393+0000 7faf8146f640 1 --2- 172.21.15.28:0/2168204833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087dea58e0 0x56087deae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.393+0000 7faf81c70640 1 -- 172.21.15.28:0/2168204833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea6290 msgr2=0x56087debadc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:06.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.393+0000 7faf81c70640 1 --2- 172.21.15.28:0/2168204833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea6290 0x56087debadc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.393+0000 7faf81c70640 1 -- 172.21.15.28:0/2168204833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087dea58e0 msgr2=0x56087deae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:06.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.393+0000 7faf81c70640 1 --2- 172.21.15.28:0/2168204833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087dea58e0 0x56087deae3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.393+0000 7faf81c70640 1 -- 172.21.15.28:0/2168204833 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56087deb16c0 con 0x56087deaee10 2024-04-21T08:53:06.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.394+0000 7faf81c70640 1 --2- 172.21.15.28:0/2168204833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087deaee10 0x56087dea1a70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faf78009860 tx=0x7faf7802bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=536ef16ee39ef344 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:06.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.394+0000 7faf66ffd640 1 -- 172.21.15.28:0/2168204833 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7804f040 con 0x56087deaee10 2024-04-21T08:53:06.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.394+0000 7faf66ffd640 1 -- 172.21.15.28:0/2168204833 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf78058020 con 0x56087deaee10 2024-04-21T08:53:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.395+0000 7faf8639d740 1 -- 172.21.15.28:0/2168204833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087deaee10 msgr2=0x56087dea1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.395+0000 7faf8639d740 1 --2- 172.21.15.28:0/2168204833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087deaee10 0x56087dea1a70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faf78009860 tx=0x7faf7802bb70 comp rx=0 tx=0).stop 2024-04-21T08:53:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.396+0000 7faf8639d740 1 -- 172.21.15.28:0/2168204833 shutdown_connections 2024-04-21T08:53:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.396+0000 7faf8639d740 1 --2- 172.21.15.28:0/2168204833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087deaee10 0x56087dea1a70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.396+0000 7faf8639d740 1 --2- 172.21.15.28:0/2168204833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea6290 0x56087debadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.396+0000 7faf8639d740 1 --2- 172.21.15.28:0/2168204833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087dea58e0 0x56087deae3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.396+0000 7faf8639d740 1 -- 172.21.15.28:0/2168204833 >> 172.21.15.28:0/2168204833 conn(0x56087de9ffb0 msgr2=0x56087dea1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:06.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.398+0000 7faf8639d740 1 -- 172.21.15.28:0/2168204833 shutdown_connections 2024-04-21T08:53:06.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.399+0000 7faf8639d740 1 -- 172.21.15.28:0/2168204833 wait complete. 2024-04-21T08:53:06.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.399+0000 7faf8639d740 1 -- start start 2024-04-21T08:53:06.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.399+0000 7faf8639d740 1 Processor -- start 2024-04-21T08:53:06.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.402+0000 7faf8639d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087dea58e0 0x56087deb72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.402+0000 7faf8639d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea6290 0x56087deb7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.402+0000 7faf8639d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087deaee10 0x56087deb42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.402+0000 7faf8639d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf6000e530 con 0x56087deaee10 2024-04-21T08:53:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.402+0000 7faf8639d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7faf6000f9a0 con 0x56087dea6290 2024-04-21T08:53:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.402+0000 7faf8639d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7faf6000ec00 con 0x56087dea58e0 2024-04-21T08:53:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.402+0000 7faf8639d740 10 client.0 _ll_register_callbacks cb 0x7faf780096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:06.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.402+0000 7faf81c70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087deaee10 0x56087deb42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.402+0000 7faf81c70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087deaee10 0x56087deb42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50398/0 (socket says 172.21.15.28:50398) 2024-04-21T08:53:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.402+0000 7faf81c70640 1 -- 172.21.15.28:0/218496026 learned_addr learned my addr 172.21.15.28:0/218496026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.402+0000 7faf8146f640 1 --2- 172.21.15.28:0/218496026 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087dea58e0 0x56087deb72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.403+0000 7faf81c70640 1 -- 172.21.15.28:0/218496026 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087dea58e0 msgr2=0x56087deb72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.403+0000 7faf81c70640 1 --2- 172.21.15.28:0/218496026 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087dea58e0 0x56087deb72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.403+0000 7faf81c70640 1 -- 172.21.15.28:0/218496026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea6290 msgr2=0x56087deb7810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.403+0000 7faf81c70640 1 --2- 172.21.15.28:0/218496026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea6290 0x56087deb7810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.403+0000 7faf81c70640 1 -- 172.21.15.28:0/218496026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56087dea6030 con 0x56087deaee10 2024-04-21T08:53:06.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.403+0000 7faf81c70640 1 --2- 172.21.15.28:0/218496026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087deaee10 0x56087deb42a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7faf7802e010 tx=0x7faf78003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3196e3a48e149f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:06.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.403+0000 7faf64ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:06.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.404+0000 7faf64ff9640 1 -- 172.21.15.28:0/218496026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7804f040 con 0x56087deaee10 2024-04-21T08:53:06.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.404+0000 7faf8639d740 1 -- 172.21.15.28:0/218496026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf78058020 con 0x56087deaee10 2024-04-21T08:53:06.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.404+0000 7faf64ff9640 1 -- 172.21.15.28:0/218496026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7faf7803f850 con 0x56087deaee10 2024-04-21T08:53:06.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.405+0000 7faf64ff9640 1 -- 172.21.15.28:0/218496026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faf78070050 con 0x56087deaee10 2024-04-21T08:53:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.405+0000 7faf8639d740 10 client.16413 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:06.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.405+0000 7faf8639d740 1 -- 172.21.15.28:0/218496026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56087df260e0 con 0x56087deaee10 2024-04-21T08:53:06.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.410+0000 7faf64ff9640 1 -- 172.21.15.28:0/218496026 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf78058020 con 0x56087deaee10 2024-04-21T08:53:06.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.410+0000 7faf64ff9640 1 -- 172.21.15.28:0/218496026 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7faf7806a4c0 con 0x56087deaee10 2024-04-21T08:53:06.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.410+0000 7faf8639d740 10 client.16413 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:06.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.410+0000 7faf8639d740 10 client.16413 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:06.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.410+0000 7faf8639d740 20 client.16413 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:06.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.410+0000 7faf8639d740 1 --2- 172.21.15.28:0/218496026 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56087df26670 0x56087df28a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.410+0000 7faf8639d740 4 client.16413 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:53:06.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.410+0000 7faf8639d740 1 -- 172.21.15.28:0/218496026 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56087de11790 con 0x56087df26670 2024-04-21T08:53:06.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.411+0000 7faf8146f640 1 --2- 172.21.15.28:0/218496026 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56087df26670 0x56087df28a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.417+0000 7faf8146f640 1 --2- 172.21.15.28:0/218496026 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56087df26670 0x56087df28a90 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f02289d896896a89 server_cookie=a639efbeb2708043 in_seq=0 out_seq=0 2024-04-21T08:53:06.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.418+0000 7faf64ff9640 10 client.16413 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:06.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.420+0000 7faf64ff9640 1 -- 172.21.15.28:0/218496026 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56087de11790 con 0x56087df26670 2024-04-21T08:53:06.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.420+0000 7faf64ff9640 10 client.16413 handle_command_reply: tid=0 2024-04-21T08:53:06.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.422+0000 7faf8639d740 1 client.16413 shutdown 2024-04-21T08:53:06.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.423+0000 7faf8639d740 1 -- 172.21.15.28:0/218496026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087deaee10 msgr2=0x56087deb42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.423+0000 7faf8639d740 1 --2- 172.21.15.28:0/218496026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087deaee10 0x56087deb42a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7faf7802e010 tx=0x7faf78003820 comp rx=0 tx=0).stop 2024-04-21T08:53:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.426+0000 7faf8639d740 1 -- 172.21.15.28:0/218496026 shutdown_connections 2024-04-21T08:53:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.426+0000 7faf8639d740 1 --2- 172.21.15.28:0/218496026 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56087df26670 0x56087df28a90 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.426+0000 7faf8639d740 1 --2- 172.21.15.28:0/218496026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56087deaee10 0x56087deb42a0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.426+0000 7faf8639d740 1 --2- 172.21.15.28:0/218496026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56087dea6290 0x56087deb7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.426+0000 7faf8639d740 1 --2- 172.21.15.28:0/218496026 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56087dea58e0 0x56087deb72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.426+0000 7faf8639d740 1 -- 172.21.15.28:0/218496026 >> 172.21.15.28:0/218496026 conn(0x56087de9ffb0 msgr2=0x56087dea1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:06.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.427+0000 7faf64ff9640 0 client.16413 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:06.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.428+0000 7faf8639d740 1 -- 172.21.15.28:0/218496026 shutdown_connections 2024-04-21T08:53:06.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.428+0000 7faf8639d740 1 -- 172.21.15.28:0/218496026 wait complete. 2024-04-21T08:53:06.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.428+0000 7faf8639d740 20 client.16413 trim_cache size 0 max 16384 2024-04-21T08:53:06.428 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:53:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:53:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "6e8ee957-6f89-4b05-b5a0-dbca9e615546", 2024-04-21T08:53:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:53:06.428 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:53:06.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.433+0000 7faf836d3640 1 -- 172.21.15.28:0/476820610 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faf680a3e20 msgr2=0x7faf680a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.433+0000 7faf836d3640 1 --2- 172.21.15.28:0/476820610 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faf680a3e20 0x7faf680a62e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7faf780057c0 tx=0x7faf78005710 comp rx=0 tx=0).stop 2024-04-21T08:53:06.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.433+0000 7faf836d3640 1 -- 172.21.15.28:0/476820610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf7c100e20 msgr2=0x7faf7c072f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.433+0000 7faf836d3640 1 --2- 172.21.15.28:0/476820610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf7c100e20 0x7faf7c072f90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7faf7400eab0 tx=0x7faf7400ef70 comp rx=0 tx=0).stop 2024-04-21T08:53:06.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.437+0000 7faf836d3640 1 -- 172.21.15.28:0/476820610 shutdown_connections 2024-04-21T08:53:06.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.437+0000 7faf836d3640 1 --2- 172.21.15.28:0/476820610 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faf680a3e20 0x7faf680a62e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.437+0000 7faf836d3640 1 --2- 172.21.15.28:0/476820610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faf7c106f30 0x7faf7c1a3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.437+0000 7faf836d3640 1 --2- 172.21.15.28:0/476820610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf7c100e20 0x7faf7c072f90 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.437+0000 7faf836d3640 1 --2- 172.21.15.28:0/476820610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faf7c100460 0x7faf7c072a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.437+0000 7faf836d3640 1 -- 172.21.15.28:0/476820610 >> 172.21.15.28:0/476820610 conn(0x7faf7c0773f0 msgr2=0x7faf7c102580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.438+0000 7faf836d3640 1 -- 172.21.15.28:0/476820610 shutdown_connections 2024-04-21T08:53:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.438+0000 7faf836d3640 1 -- 172.21.15.28:0/476820610 wait complete. 2024-04-21T08:53:06.461 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:53:06.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:53:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.586+0000 7ff672681640 1 -- 172.21.15.28:0/2610389275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c1031c0 msgr2=0x7ff66c1035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.586+0000 7ff672681640 1 --2- 172.21.15.28:0/2610389275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c1031c0 0x7ff66c1035a0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009900 tx=0x7ff65c02bd60 comp rx=0 tx=0).stop 2024-04-21T08:53:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.586+0000 7ff672681640 1 -- 172.21.15.28:0/2610389275 shutdown_connections 2024-04-21T08:53:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.586+0000 7ff672681640 1 --2- 172.21.15.28:0/2610389275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff66c10c500 0x7ff66c10e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.586+0000 7ff672681640 1 --2- 172.21.15.28:0/2610389275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff66c103ae0 0x7ff66c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.586+0000 7ff672681640 1 --2- 172.21.15.28:0/2610389275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c1031c0 0x7ff66c1035a0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.586+0000 7ff672681640 1 -- 172.21.15.28:0/2610389275 >> 172.21.15.28:0/2610389275 conn(0x7ff66c078000 msgr2=0x7ff66c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.586+0000 7ff672681640 1 -- 172.21.15.28:0/2610389275 shutdown_connections 2024-04-21T08:53:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.586+0000 7ff672681640 1 -- 172.21.15.28:0/2610389275 wait complete. 2024-04-21T08:53:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff672681640 1 Processor -- start 2024-04-21T08:53:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff672681640 1 -- start start 2024-04-21T08:53:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff672681640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff66c1031c0 0x7ff66c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff672681640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff66c103ae0 0x7ff66c198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff672681640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c10c500 0x7ff66c19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff672681640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c112170 con 0x7ff66c10c500 2024-04-21T08:53:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff672681640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c111ff0 con 0x7ff66c1031c0 2024-04-21T08:53:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff672681640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c1122f0 con 0x7ff66c103ae0 2024-04-21T08:53:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff670c1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c10c500 0x7ff66c19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff670c1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c10c500 0x7ff66c19cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50412/0 (socket says 172.21.15.28:50412) 2024-04-21T08:53:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff670c1e640 1 -- 172.21.15.28:0/3462516065 learned_addr learned my addr 172.21.15.28:0/3462516065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff66bfff640 1 --2- 172.21.15.28:0/3462516065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff66c1031c0 0x7ff66c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff66b7fe640 1 --2- 172.21.15.28:0/3462516065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff66c103ae0 0x7ff66c198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff670c1e640 1 -- 172.21.15.28:0/3462516065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff66c103ae0 msgr2=0x7ff66c198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff670c1e640 1 --2- 172.21.15.28:0/3462516065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff66c103ae0 0x7ff66c198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff670c1e640 1 -- 172.21.15.28:0/3462516065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff66c1031c0 msgr2=0x7ff66c1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff670c1e640 1 --2- 172.21.15.28:0/3462516065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff66c1031c0 0x7ff66c1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff670c1e640 1 -- 172.21.15.28:0/3462516065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff66c19d6e0 con 0x7ff66c10c500 2024-04-21T08:53:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.587+0000 7ff66bfff640 1 --2- 172.21.15.28:0/3462516065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff66c1031c0 0x7ff66c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:53:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.588+0000 7ff670c1e640 1 --2- 172.21.15.28:0/3462516065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c10c500 0x7ff66c19cfb0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ff664009fd0 tx=0x7ff66400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.588+0000 7ff6697fa640 1 -- 172.21.15.28:0/3462516065 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff664017070 con 0x7ff66c10c500 2024-04-21T08:53:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.588+0000 7ff6697fa640 1 -- 172.21.15.28:0/3462516065 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff664003e20 con 0x7ff66c10c500 2024-04-21T08:53:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.588+0000 7ff672681640 1 -- 172.21.15.28:0/3462516065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff66c19d9d0 con 0x7ff66c10c500 2024-04-21T08:53:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.588+0000 7ff6697fa640 1 -- 172.21.15.28:0/3462516065 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff664010460 con 0x7ff66c10c500 2024-04-21T08:53:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.588+0000 7ff672681640 1 -- 172.21.15.28:0/3462516065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff66c1a5260 con 0x7ff66c10c500 2024-04-21T08:53:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.590+0000 7ff6697fa640 1 -- 172.21.15.28:0/3462516065 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff664003970 con 0x7ff66c10c500 2024-04-21T08:53:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.590+0000 7ff675270740 1 Processor -- start 2024-04-21T08:53:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.590+0000 7ff675270740 1 -- start start 2024-04-21T08:53:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff675270740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfbf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff675270740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfbfea0 0x56465dfc8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff675270740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfc8a80 0x56465dfcae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff675270740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56465df241e0 con 0x56465dfbf580 2024-04-21T08:53:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff675270740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56465dec1390 con 0x56465dfbfea0 2024-04-21T08:53:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff675270740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56465df201a0 con 0x56465dfc8a80 2024-04-21T08:53:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfbf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfbf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50422/0 (socket says 172.21.15.28:50422) 2024-04-21T08:53:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff66b7fe640 1 -- 172.21.15.28:0/3608331290 learned_addr learned my addr 172.21.15.28:0/3608331290 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff6697fa640 1 --2- 172.21.15.28:0/3462516065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6540a3da0 0x7ff6540a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff66bfff640 1 --2- 172.21.15.28:0/3608331290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfbfea0 0x56465dfc8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff66b7fe640 1 --2- 172.21.15.28:0/3608331290 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfc8a80 0x56465dfcae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff670c1e640 1 --2- 172.21.15.28:0/3462516065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6540a3da0 0x7ff6540a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff6697fa640 1 -- 172.21.15.28:0/3462516065 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ff6640d08e0 con 0x7ff66c10c500 2024-04-21T08:53:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff66b7fe640 1 -- 172.21.15.28:0/3608331290 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfc8a80 msgr2=0x56465dfcae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff66b7fe640 1 --2- 172.21.15.28:0/3608331290 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfc8a80 0x56465dfcae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff66b7fe640 1 -- 172.21.15.28:0/3608331290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfbfea0 msgr2=0x56465dfc8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff66b7fe640 1 --2- 172.21.15.28:0/3608331290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfbfea0 0x56465dfc8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff66b7fe640 1 -- 172.21.15.28:0/3608331290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56465dd007d0 con 0x56465dfbf580 2024-04-21T08:53:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff66b7fe640 1 --2- 172.21.15.28:0/3608331290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfbf960 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff660013550 tx=0x7ff66000b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc87f0158f2976f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff670c1e640 1 --2- 172.21.15.28:0/3462516065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6540a3da0 0x7ff6540a6260 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff6640ce290 tx=0x7ff664091970 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:06.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.591+0000 7ff6527fc640 1 -- 172.21.15.28:0/3608331290 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660019030 con 0x56465dfbf580 2024-04-21T08:53:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.592+0000 7ff6527fc640 1 -- 172.21.15.28:0/3608331290 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff660011970 con 0x56465dfbf580 2024-04-21T08:53:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.592+0000 7ff6527fc640 1 -- 172.21.15.28:0/3608331290 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66001d7d0 con 0x56465dfbf580 2024-04-21T08:53:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.592+0000 7ff675270740 1 -- 172.21.15.28:0/3608331290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 msgr2=0x56465dfbf960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.592+0000 7ff675270740 1 --2- 172.21.15.28:0/3608331290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfbf960 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff660013550 tx=0x7ff66000b670 comp rx=0 tx=0).stop 2024-04-21T08:53:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.592+0000 7ff675270740 1 -- 172.21.15.28:0/3608331290 shutdown_connections 2024-04-21T08:53:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.592+0000 7ff675270740 1 --2- 172.21.15.28:0/3608331290 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfc8a80 0x56465dfcae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.592+0000 7ff675270740 1 --2- 172.21.15.28:0/3608331290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfbfea0 0x56465dfc8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.592+0000 7ff675270740 1 --2- 172.21.15.28:0/3608331290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfbf960 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.592+0000 7ff675270740 1 -- 172.21.15.28:0/3608331290 >> 172.21.15.28:0/3608331290 conn(0x56465dfb9cb0 msgr2=0x56465dfba090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.592+0000 7ff675270740 1 -- 172.21.15.28:0/3608331290 shutdown_connections 2024-04-21T08:53:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.592+0000 7ff675270740 1 -- 172.21.15.28:0/3608331290 wait complete. 2024-04-21T08:53:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.593+0000 7ff675270740 1 -- start start 2024-04-21T08:53:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.593+0000 7ff675270740 1 Processor -- start 2024-04-21T08:53:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.593+0000 7ff675270740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfbf580 0x56465dfc5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.593+0000 7ff675270740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbfea0 0x56465dfc6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.593+0000 7ff675270740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfc8a80 0x56465dfc6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.593+0000 7ff675270740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56465dfcb870 con 0x56465dfbfea0 2024-04-21T08:53:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.593+0000 7ff675270740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56465dedd2f0 con 0x56465dfc8a80 2024-04-21T08:53:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.593+0000 7ff675270740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56465dfcb9f0 con 0x56465dfbf580 2024-04-21T08:53:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.593+0000 7ff675270740 10 client.0 _ll_register_callbacks cb 0x56465deea580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.593+0000 7ff66bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbfea0 0x56465dfc6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.593+0000 7ff66bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbfea0 0x56465dfc6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50434/0 (socket says 172.21.15.28:50434) 2024-04-21T08:53:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.593+0000 7ff66bfff640 1 -- 172.21.15.28:0/3961216987 learned_addr learned my addr 172.21.15.28:0/3961216987 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.593+0000 7ff66b7fe640 1 --2- 172.21.15.28:0/3961216987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfc8a80 0x56465dfc6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.594+0000 7ff66b7fe640 1 --2- 172.21.15.28:0/3961216987 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfbf580 0x56465dfc5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.594+0000 7ff66bfff640 1 -- 172.21.15.28:0/3961216987 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfbf580 msgr2=0x56465dfc5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.594+0000 7ff66bfff640 1 --2- 172.21.15.28:0/3961216987 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfbf580 0x56465dfc5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.594+0000 7ff66bfff640 1 -- 172.21.15.28:0/3961216987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfc8a80 msgr2=0x56465dfc6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:06.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.594+0000 7ff66bfff640 1 --2- 172.21.15.28:0/3961216987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfc8a80 0x56465dfc6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.594+0000 7ff66bfff640 1 -- 172.21.15.28:0/3961216987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56465dc363f0 con 0x56465dfbfea0 2024-04-21T08:53:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.594+0000 7ff66bfff640 1 --2- 172.21.15.28:0/3961216987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbfea0 0x56465dfc6030 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009900 tx=0x7ff65c02dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee87a7805e18f4a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.594+0000 7ff64b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.594+0000 7ff64b7fe640 1 -- 172.21.15.28:0/3961216987 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65c043030 con 0x56465dfbfea0 2024-04-21T08:53:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.594+0000 7ff64b7fe640 1 -- 172.21.15.28:0/3961216987 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ff65c005250 con 0x56465dfbfea0 2024-04-21T08:53:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.594+0000 7ff675270740 1 -- 172.21.15.28:0/3961216987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff660011970 con 0x56465dfbfea0 2024-04-21T08:53:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.594+0000 7ff64b7fe640 1 -- 172.21.15.28:0/3961216987 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff65c050050 con 0x56465dfbfea0 2024-04-21T08:53:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.594+0000 7ff675270740 10 client.16425 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.594+0000 7ff675270740 1 -- 172.21.15.28:0/3961216987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56465dbf60f0 con 0x56465dfbfea0 2024-04-21T08:53:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.595+0000 7ff64b7fe640 1 -- 172.21.15.28:0/3961216987 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff660011970 con 0x56465dfbfea0 2024-04-21T08:53:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.595+0000 7ff64b7fe640 1 -- 172.21.15.28:0/3961216987 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7ff65c0035f0 con 0x56465dfbfea0 2024-04-21T08:53:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.595+0000 7ff675270740 10 client.16425 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.595+0000 7ff675270740 10 client.16425 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:06.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.595+0000 7ff675270740 20 client.16425 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.595+0000 7ff675270740 1 --2- 172.21.15.28:0/3961216987 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56465dfd8b90 0x56465dfdafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.595+0000 7ff675270740 4 client.16425 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.595+0000 7ff675270740 1 -- 172.21.15.28:0/3961216987 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56465df31c00 con 0x56465dfd8b90 2024-04-21T08:53:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.595+0000 7ff66b7fe640 1 --2- 172.21.15.28:0/3961216987 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56465dfd8b90 0x56465dfdafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.595+0000 7ff66b7fe640 1 --2- 172.21.15.28:0/3961216987 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56465dfd8b90 0x56465dfdafd0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=598fd306fc90f77d server_cookie=25d664fca5a71099 in_seq=0 out_seq=0 2024-04-21T08:53:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.595+0000 7ff64b7fe640 10 client.16425 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.597+0000 7ff64b7fe640 1 -- 172.21.15.28:0/3961216987 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56465df31c00 con 0x56465dfd8b90 2024-04-21T08:53:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.597+0000 7ff64b7fe640 10 client.16425 handle_command_reply: tid=0 2024-04-21T08:53:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.597+0000 7ff675270740 1 client.16425 shutdown 2024-04-21T08:53:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.597+0000 7ff675270740 1 -- 172.21.15.28:0/3961216987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbfea0 msgr2=0x56465dfc6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.597+0000 7ff675270740 1 --2- 172.21.15.28:0/3961216987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbfea0 0x56465dfc6030 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009900 tx=0x7ff65c02dbb0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.597+0000 7ff675270740 1 -- 172.21.15.28:0/3961216987 shutdown_connections 2024-04-21T08:53:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.597+0000 7ff675270740 1 --2- 172.21.15.28:0/3961216987 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56465dfd8b90 0x56465dfdafd0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.597+0000 7ff675270740 1 --2- 172.21.15.28:0/3961216987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfc8a80 0x56465dfc6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.597+0000 7ff675270740 1 --2- 172.21.15.28:0/3961216987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbfea0 0x56465dfc6030 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.597+0000 7ff675270740 1 --2- 172.21.15.28:0/3961216987 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfbf580 0x56465dfc5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.597+0000 7ff64b7fe640 0 client.16425 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:06.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.597+0000 7ff675270740 1 -- 172.21.15.28:0/3961216987 >> 172.21.15.28:0/3961216987 conn(0x56465dfb9cb0 msgr2=0x56465dfbba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:06.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.597+0000 7ff675270740 1 -- 172.21.15.28:0/3961216987 shutdown_connections 2024-04-21T08:53:06.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.597+0000 7ff675270740 1 -- 172.21.15.28:0/3961216987 wait complete. 2024-04-21T08:53:06.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.597+0000 7ff675270740 20 client.16425 trim_cache size 0 max 16384 2024-04-21T08:53:06.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff675270740 1 Processor -- start 2024-04-21T08:53:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff675270740 1 -- start start 2024-04-21T08:53:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff675270740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfd4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff675270740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfbfea0 0x56465dfc4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff675270740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfc8a80 0x56465dfc6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff675270740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c1b1c20 con 0x56465dfbf580 2024-04-21T08:53:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff675270740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c1af4f0 con 0x56465dfbfea0 2024-04-21T08:53:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff675270740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c1ad340 con 0x56465dfc8a80 2024-04-21T08:53:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfd4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfd4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50450/0 (socket says 172.21.15.28:50450) 2024-04-21T08:53:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff66b7fe640 1 -- 172.21.15.28:0/3177755649 learned_addr learned my addr 172.21.15.28:0/3177755649 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff66b7fe640 1 -- 172.21.15.28:0/3177755649 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfc8a80 msgr2=0x56465dfc6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff66b7fe640 1 --2- 172.21.15.28:0/3177755649 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfc8a80 0x56465dfc6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff66b7fe640 1 -- 172.21.15.28:0/3177755649 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfbfea0 msgr2=0x56465dfc4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff66b7fe640 1 --2- 172.21.15.28:0/3177755649 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfbfea0 0x56465dfc4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.615+0000 7ff66b7fe640 1 -- 172.21.15.28:0/3177755649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56465dfcb1a0 con 0x56465dfbf580 2024-04-21T08:53:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.616+0000 7ff66b7fe640 1 --2- 172.21.15.28:0/3177755649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfd4730 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff660005e00 tx=0x7ff66000b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e4cb039406320a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.616+0000 7ff651ffb640 1 -- 172.21.15.28:0/3177755649 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660019040 con 0x56465dfbf580 2024-04-21T08:53:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.616+0000 7ff651ffb640 1 -- 172.21.15.28:0/3177755649 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff66000be40 con 0x56465dfbf580 2024-04-21T08:53:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.616+0000 7ff651ffb640 1 -- 172.21.15.28:0/3177755649 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6600054c0 con 0x56465dfbf580 2024-04-21T08:53:06.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 -- 172.21.15.28:0/3177755649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 msgr2=0x56465dfd4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 --2- 172.21.15.28:0/3177755649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfd4730 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff660005e00 tx=0x7ff66000b670 comp rx=0 tx=0).stop 2024-04-21T08:53:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 -- 172.21.15.28:0/3177755649 shutdown_connections 2024-04-21T08:53:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 --2- 172.21.15.28:0/3177755649 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfc8a80 0x56465dfc6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 --2- 172.21.15.28:0/3177755649 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfbfea0 0x56465dfc4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 --2- 172.21.15.28:0/3177755649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfd4730 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 -- 172.21.15.28:0/3177755649 >> 172.21.15.28:0/3177755649 conn(0x56465dfb9cb0 msgr2=0x56465dfbb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 -- 172.21.15.28:0/3177755649 shutdown_connections 2024-04-21T08:53:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 -- 172.21.15.28:0/3177755649 wait complete. 2024-04-21T08:53:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 -- start start 2024-04-21T08:53:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 Processor -- start 2024-04-21T08:53:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfd0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfbfea0 0x56465dfd1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfc8a80 0x56465dfcc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c1ad9b0 con 0x56465dfbf580 2024-04-21T08:53:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c1aee20 con 0x56465dfc8a80 2024-04-21T08:53:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.617+0000 7ff675270740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c1ae080 con 0x56465dfbfea0 2024-04-21T08:53:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff675270740 10 client.0 _ll_register_callbacks cb 0x7ff65c009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfd0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff66b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfd0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50464/0 (socket says 172.21.15.28:50464) 2024-04-21T08:53:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff66b7fe640 1 -- 172.21.15.28:0/3619910289 learned_addr learned my addr 172.21.15.28:0/3619910289 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff66bfff640 1 --2- 172.21.15.28:0/3619910289 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfbfea0 0x56465dfd1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff66b7fe640 1 -- 172.21.15.28:0/3619910289 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfbfea0 msgr2=0x56465dfd1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff66b7fe640 1 --2- 172.21.15.28:0/3619910289 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfbfea0 0x56465dfd1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff66b7fe640 1 -- 172.21.15.28:0/3619910289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfc8a80 msgr2=0x56465dfcc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff66b7fe640 1 --2- 172.21.15.28:0/3619910289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfc8a80 0x56465dfcc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff66b7fe640 1 -- 172.21.15.28:0/3619910289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56465dfccbc0 con 0x56465dfbf580 2024-04-21T08:53:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff66b7fe640 1 --2- 172.21.15.28:0/3619910289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfd0c50 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff6600050d0 tx=0x7ff660005100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965aabe3dc2e036 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff64b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff64b7fe640 1 -- 172.21.15.28:0/3619910289 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660019050 con 0x56465dfbf580 2024-04-21T08:53:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff64b7fe640 1 -- 172.21.15.28:0/3619910289 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ff660005560 con 0x56465dfbf580 2024-04-21T08:53:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff675270740 1 -- 172.21.15.28:0/3619910289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff66000be40 con 0x56465dfbf580 2024-04-21T08:53:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff64b7fe640 1 -- 172.21.15.28:0/3619910289 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff660043050 con 0x56465dfbf580 2024-04-21T08:53:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff675270740 10 client.16431 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.618+0000 7ff675270740 1 -- 172.21.15.28:0/3619910289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56465e03edd0 con 0x56465dfbf580 2024-04-21T08:53:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.619+0000 7ff64b7fe640 1 -- 172.21.15.28:0/3619910289 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff66000be40 con 0x56465dfbf580 2024-04-21T08:53:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.619+0000 7ff64b7fe640 1 -- 172.21.15.28:0/3619910289 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7ff66002dce0 con 0x56465dfbf580 2024-04-21T08:53:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.619+0000 7ff675270740 10 client.16431 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.619+0000 7ff675270740 10 client.16431 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.619+0000 7ff675270740 20 client.16431 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.619+0000 7ff675270740 1 --2- 172.21.15.28:0/3619910289 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56465e03f320 0x56465e041740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.619+0000 7ff675270740 4 client.16431 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:53:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.619+0000 7ff675270740 1 -- 172.21.15.28:0/3619910289 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56465df2ba70 con 0x56465e03f320 2024-04-21T08:53:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.619+0000 7ff66bfff640 1 --2- 172.21.15.28:0/3619910289 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56465e03f320 0x56465e041740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.620+0000 7ff66bfff640 1 --2- 172.21.15.28:0/3619910289 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56465e03f320 0x56465e041740 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48287b8792eb5c48 server_cookie=f7be972ba47706fb in_seq=0 out_seq=0 2024-04-21T08:53:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.620+0000 7ff64b7fe640 10 client.16431 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.620+0000 7ff64b7fe640 1 -- 172.21.15.28:0/3619910289 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x7ff65c04d120 con 0x56465e03f320 2024-04-21T08:53:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.620+0000 7ff64b7fe640 10 client.16431 handle_command_reply: tid=0 2024-04-21T08:53:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.620+0000 7ff675270740 1 client.16431 shutdown 2024-04-21T08:53:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.620+0000 7ff675270740 1 -- 172.21.15.28:0/3619910289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 msgr2=0x56465dfd0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff675270740 1 --2- 172.21.15.28:0/3619910289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfd0c50 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff6600050d0 tx=0x7ff660005100 comp rx=0 tx=0).stop 2024-04-21T08:53:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff675270740 1 -- 172.21.15.28:0/3619910289 shutdown_connections 2024-04-21T08:53:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff675270740 1 --2- 172.21.15.28:0/3619910289 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56465e03f320 0x56465e041740 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff675270740 1 --2- 172.21.15.28:0/3619910289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56465dfc8a80 0x56465dfcc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff675270740 1 --2- 172.21.15.28:0/3619910289 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56465dfbfea0 0x56465dfd1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff64b7fe640 0 client.16431 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff675270740 1 --2- 172.21.15.28:0/3619910289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56465dfbf580 0x56465dfd0c50 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff675270740 1 -- 172.21.15.28:0/3619910289 >> 172.21.15.28:0/3619910289 conn(0x56465dfb9cb0 msgr2=0x56465dfb8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff675270740 1 -- 172.21.15.28:0/3619910289 shutdown_connections 2024-04-21T08:53:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff675270740 1 -- 172.21.15.28:0/3619910289 wait complete. 2024-04-21T08:53:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff675270740 20 client.16431 trim_cache size 0 max 16384 2024-04-21T08:53:06.627 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:53:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (87 inodes in the stack)", 2024-04-21T08:53:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:53:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "6e8ee957-6f89-4b05-b5a0-dbca9e615546": { 2024-04-21T08:53:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:53:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "6e8ee957-6f89-4b05-b5a0-dbca9e615546", 2024-04-21T08:53:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:53:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:53:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:53:06.628 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:53:06.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff672681640 1 -- 172.21.15.28:0/3462516065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6540a3da0 msgr2=0x7ff6540a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff672681640 1 --2- 172.21.15.28:0/3462516065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6540a3da0 0x7ff6540a6260 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff6640ce290 tx=0x7ff664091970 comp rx=0 tx=0).stop 2024-04-21T08:53:06.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff672681640 1 -- 172.21.15.28:0/3462516065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c10c500 msgr2=0x7ff66c19cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff672681640 1 --2- 172.21.15.28:0/3462516065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c10c500 0x7ff66c19cfb0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ff664009fd0 tx=0x7ff66400b260 comp rx=0 tx=0).stop 2024-04-21T08:53:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff672681640 1 -- 172.21.15.28:0/3462516065 shutdown_connections 2024-04-21T08:53:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff672681640 1 --2- 172.21.15.28:0/3462516065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6540a3da0 0x7ff6540a6260 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff672681640 1 --2- 172.21.15.28:0/3462516065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff66c10c500 0x7ff66c19cfb0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff672681640 1 --2- 172.21.15.28:0/3462516065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff66c103ae0 0x7ff66c198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff672681640 1 --2- 172.21.15.28:0/3462516065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff66c1031c0 0x7ff66c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff672681640 1 -- 172.21.15.28:0/3462516065 >> 172.21.15.28:0/3462516065 conn(0x7ff66c078000 msgr2=0x7ff66c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff672681640 1 -- 172.21.15.28:0/3462516065 shutdown_connections 2024-04-21T08:53:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:06.621+0000 7ff672681640 1 -- 172.21.15.28:0/3462516065 wait complete. 2024-04-21T08:53:06.632 INFO:tasks.cephfs.filesystem:scrub status for tag:6e8ee957-6f89-4b05-b5a0-dbca9e615546 - {'path': '/', 'tag': '6e8ee957-6f89-4b05-b5a0-dbca9e615546', 'options': 'recursive,force'} 2024-04-21T08:53:07.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:06 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:53:07.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:06 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:07.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:06 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:53:07.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:06 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:53:07.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:06 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:07.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:06 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:53:07.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:06 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:53:07.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:06 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:07.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:06 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:53:08.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:08 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:53:08.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:08 smithi152 ceph-mon[31401]: pgmap v61: 129 pgs: 129 active+clean; 13 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 120 MiB/s wr, 14.70k op/s 2024-04-21T08:53:08.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:08 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:53:08.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:08 smithi028 ceph-mon[28003]: pgmap v61: 129 pgs: 129 active+clean; 13 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 120 MiB/s wr, 14.70k op/s 2024-04-21T08:53:08.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:08 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:53:08.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:08 smithi171 ceph-mon[31358]: pgmap v61: 129 pgs: 129 active+clean; 13 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 120 MiB/s wr, 14.70k op/s 2024-04-21T08:53:10.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:10 smithi152 ceph-mon[31401]: pgmap v62: 129 pgs: 129 active+clean; 13 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 168 MiB/s wr, 20.04k op/s 2024-04-21T08:53:10.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:10 smithi028 ceph-mon[28003]: pgmap v62: 129 pgs: 129 active+clean; 13 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 168 MiB/s wr, 20.04k op/s 2024-04-21T08:53:10.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:10 smithi171 ceph-mon[31358]: pgmap v62: 129 pgs: 129 active+clean; 13 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 168 MiB/s wr, 20.04k op/s 2024-04-21T08:53:12.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:12 smithi028 ceph-mon[28003]: pgmap v63: 129 pgs: 129 active+clean; 13 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 117 MiB/s wr, 13.65k op/s 2024-04-21T08:53:12.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:12 smithi152 ceph-mon[31401]: pgmap v63: 129 pgs: 129 active+clean; 13 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 117 MiB/s wr, 13.65k op/s 2024-04-21T08:53:12.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:12 smithi171 ceph-mon[31358]: pgmap v63: 129 pgs: 129 active+clean; 13 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 117 MiB/s wr, 13.65k op/s 2024-04-21T08:53:14.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:14 smithi152 ceph-mon[31401]: pgmap v64: 129 pgs: 129 active+clean; 13 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 170 MiB/s wr, 19.36k op/s 2024-04-21T08:53:14.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:14 smithi028 ceph-mon[28003]: pgmap v64: 129 pgs: 129 active+clean; 13 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 170 MiB/s wr, 19.36k op/s 2024-04-21T08:53:14.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:14 smithi171 ceph-mon[31358]: pgmap v64: 129 pgs: 129 active+clean; 13 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 170 MiB/s wr, 19.36k op/s 2024-04-21T08:53:16.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:16 smithi152 ceph-mon[31401]: pgmap v65: 129 pgs: 129 active+clean; 13 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 130 MiB/s wr, 14.43k op/s 2024-04-21T08:53:16.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:16 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:53:16.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:16 smithi028 ceph-mon[28003]: pgmap v65: 129 pgs: 129 active+clean; 13 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 130 MiB/s wr, 14.43k op/s 2024-04-21T08:53:16.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:16 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:53:16.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:16 smithi171 ceph-mon[31358]: pgmap v65: 129 pgs: 129 active+clean; 13 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 130 MiB/s wr, 14.43k op/s 2024-04-21T08:53:16.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:16 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:53:16.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.377+0000 7f16f7715640 1 -- 172.21.15.28:0/576843367 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f0110740 msgr2=0x7f16f0108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:16.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.377+0000 7f16f7715640 1 --2- 172.21.15.28:0/576843367 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f0110740 0x7f16f0108f50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009930 tx=0x7f16e002bd10 comp rx=0 tx=0).stop 2024-04-21T08:53:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.381+0000 7f16f7715640 1 -- 172.21.15.28:0/576843367 shutdown_connections 2024-04-21T08:53:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.381+0000 7f16f7715640 1 --2- 172.21.15.28:0/576843367 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16f011b730 0x7f16f011db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.381+0000 7f16f7715640 1 --2- 172.21.15.28:0/576843367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f0110b20 0x7f16f0109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.381+0000 7f16f7715640 1 --2- 172.21.15.28:0/576843367 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f0110740 0x7f16f0108f50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.381+0000 7f16f7715640 1 -- 172.21.15.28:0/576843367 >> 172.21.15.28:0/576843367 conn(0x7f16f006d020 msgr2=0x7f16f006d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.381+0000 7f16f7715640 1 -- 172.21.15.28:0/576843367 shutdown_connections 2024-04-21T08:53:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.381+0000 7f16f7715640 1 -- 172.21.15.28:0/576843367 wait complete. 2024-04-21T08:53:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.382+0000 7f16f7715640 1 Processor -- start 2024-04-21T08:53:16.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.382+0000 7f16f7715640 1 -- start start 2024-04-21T08:53:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.383+0000 7f16f7715640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f0110740 0x7f16f01a7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.383+0000 7f16f7715640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16f011b730 0x7f16f01a7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.383+0000 7f16f7715640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f01abf50 0x7f16f01ac400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.383+0000 7f16f54b1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f0110740 0x7f16f01a7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.383+0000 7f16f54b1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f0110740 0x7f16f01a7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:42060/0 (socket says 172.21.15.28:42060) 2024-04-21T08:53:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.384+0000 7f16f54b1640 1 -- 172.21.15.28:0/3531474293 learned_addr learned my addr 172.21.15.28:0/3531474293 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.384+0000 7f16f5cb2640 1 --2- 172.21.15.28:0/3531474293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f01abf50 0x7f16f01ac400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.384+0000 7f16f7715640 1 -- 172.21.15.28:0/3531474293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16f01210d0 con 0x7f16f01abf50 2024-04-21T08:53:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.384+0000 7f16f7715640 1 -- 172.21.15.28:0/3531474293 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f16f0120f50 con 0x7f16f0110740 2024-04-21T08:53:16.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.384+0000 7f16f7715640 1 -- 172.21.15.28:0/3531474293 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f16f0121250 con 0x7f16f011b730 2024-04-21T08:53:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.385+0000 7f16f4cb0640 1 --2- 172.21.15.28:0/3531474293 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16f011b730 0x7f16f01a7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.386+0000 7f16f5cb2640 1 -- 172.21.15.28:0/3531474293 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16f011b730 msgr2=0x7f16f01a7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.386+0000 7f16f5cb2640 1 --2- 172.21.15.28:0/3531474293 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16f011b730 0x7f16f01a7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.386+0000 7f16f5cb2640 1 -- 172.21.15.28:0/3531474293 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f0110740 msgr2=0x7f16f01a7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.386+0000 7f16f5cb2640 1 --2- 172.21.15.28:0/3531474293 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f0110740 0x7f16f01a7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.386+0000 7f16f5cb2640 1 -- 172.21.15.28:0/3531474293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f01aca40 con 0x7f16f01abf50 2024-04-21T08:53:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.387+0000 7f16f5cb2640 1 --2- 172.21.15.28:0/3531474293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f01abf50 0x7f16f01ac400 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f16e800bdf0 tx=0x7f16e8009ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.387+0000 7f16e67fc640 1 -- 172.21.15.28:0/3531474293 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e8019070 con 0x7f16f01abf50 2024-04-21T08:53:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.387+0000 7f16e67fc640 1 -- 172.21.15.28:0/3531474293 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16e80148f0 con 0x7f16f01abf50 2024-04-21T08:53:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.388+0000 7f16f7715640 1 -- 172.21.15.28:0/3531474293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f01acd30 con 0x7f16f01abf50 2024-04-21T08:53:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.388+0000 7f16f7715640 1 -- 172.21.15.28:0/3531474293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f01b44d0 con 0x7f16f01abf50 2024-04-21T08:53:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.389+0000 7f16e67fc640 1 -- 172.21.15.28:0/3531474293 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e8012880 con 0x7f16f01abf50 2024-04-21T08:53:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.395+0000 7f16fa3e3740 1 Processor -- start 2024-04-21T08:53:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.395+0000 7f16e67fc640 1 -- 172.21.15.28:0/3531474293 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f16e801c070 con 0x7f16f01abf50 2024-04-21T08:53:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.395+0000 7f16fa3e3740 1 -- start start 2024-04-21T08:53:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.396+0000 7f16fa3e3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d50288e0 0x55f1d5027630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.396+0000 7f16fa3e3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d5027b70 0x55f1d5027ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.396+0000 7f16fa3e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50310f0 0x55f1d50334e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.396+0000 7f16fa3e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f1d4f8c8d0 con 0x55f1d50310f0 2024-04-21T08:53:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.396+0000 7f16f54b1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d5027b70 0x55f1d5027ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.396+0000 7f16e67fc640 1 --2- 172.21.15.28:0/3531474293 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f16dc0a3e90 0x7f16dc0a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.396+0000 7f16f54b1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d5027b70 0x55f1d5027ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:42062/0 (socket says 172.21.15.28:42062) 2024-04-21T08:53:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.397+0000 7f16f54b1640 1 -- 172.21.15.28:0/2280880475 learned_addr learned my addr 172.21.15.28:0/2280880475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.397+0000 7f16fa3e3740 1 -- 172.21.15.28:0/2280880475 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f1d4f29e80 con 0x55f1d5027b70 2024-04-21T08:53:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.397+0000 7f16fa3e3740 1 -- 172.21.15.28:0/2280880475 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f1d4f88890 con 0x55f1d50288e0 2024-04-21T08:53:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.397+0000 7f16e67fc640 1 -- 172.21.15.28:0/3531474293 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f16e80d03c0 con 0x7f16f01abf50 2024-04-21T08:53:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.398+0000 7f16f5cb2640 1 --2- 172.21.15.28:0/3531474293 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f16dc0a3e90 0x7f16dc0a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.398+0000 7f16f4cb0640 1 --2- 172.21.15.28:0/2280880475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50310f0 0x55f1d50334e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.398+0000 7f16f4cb0640 1 --2- 172.21.15.28:0/2280880475 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d50288e0 0x55f1d5027630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.398+0000 7f16f54b1640 1 -- 172.21.15.28:0/2280880475 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d50288e0 msgr2=0x55f1d5027630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.398+0000 7f16f54b1640 1 --2- 172.21.15.28:0/2280880475 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d50288e0 0x55f1d5027630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.398+0000 7f16f54b1640 1 -- 172.21.15.28:0/2280880475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50310f0 msgr2=0x55f1d50334e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.398+0000 7f16f54b1640 1 --2- 172.21.15.28:0/2280880475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50310f0 0x55f1d50334e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.398+0000 7f16f54b1640 1 -- 172.21.15.28:0/2280880475 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1d4f4b320 con 0x55f1d5027b70 2024-04-21T08:53:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.399+0000 7f16f54b1640 1 --2- 172.21.15.28:0/2280880475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d5027b70 0x55f1d5027ff0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009850 tx=0x7f16e0009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a8b7e350de93f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.400+0000 7f16db7fe640 1 -- 172.21.15.28:0/2280880475 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e0043070 con 0x55f1d5027b70 2024-04-21T08:53:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.400+0000 7f16db7fe640 1 -- 172.21.15.28:0/2280880475 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16e002c860 con 0x55f1d5027b70 2024-04-21T08:53:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.401+0000 7f16fa3e3740 1 -- 172.21.15.28:0/2280880475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d5027b70 msgr2=0x55f1d5027ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.401+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/2280880475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d5027b70 0x55f1d5027ff0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009850 tx=0x7f16e0009620 comp rx=0 tx=0).stop 2024-04-21T08:53:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.401+0000 7f16f5cb2640 1 --2- 172.21.15.28:0/3531474293 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f16dc0a3e90 0x7f16dc0a6350 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f16e8098580 tx=0x7f16e800ce70 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.403+0000 7f16fa3e3740 1 -- 172.21.15.28:0/2280880475 shutdown_connections 2024-04-21T08:53:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.403+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/2280880475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50310f0 0x55f1d50334e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.403+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/2280880475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d5027b70 0x55f1d5027ff0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.403+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/2280880475 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d50288e0 0x55f1d5027630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.403+0000 7f16fa3e3740 1 -- 172.21.15.28:0/2280880475 >> 172.21.15.28:0/2280880475 conn(0x55f1d4feec90 msgr2=0x55f1d5022860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.407+0000 7f16fa3e3740 1 -- 172.21.15.28:0/2280880475 shutdown_connections 2024-04-21T08:53:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.407+0000 7f16fa3e3740 1 -- 172.21.15.28:0/2280880475 wait complete. 2024-04-21T08:53:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.408+0000 7f16fa3e3740 1 -- start start 2024-04-21T08:53:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.408+0000 7f16fa3e3740 1 Processor -- start 2024-04-21T08:53:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.410+0000 7f16fa3e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d5027b70 0x55f1d50309f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.410+0000 7f16fa3e3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d50288e0 0x55f1d502dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.410+0000 7f16fa3e3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d50310f0 0x55f1d502e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.410+0000 7f16fa3e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f1d5033e10 con 0x55f1d5027b70 2024-04-21T08:53:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.410+0000 7f16fa3e3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f1d4f14eb0 con 0x55f1d50288e0 2024-04-21T08:53:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.410+0000 7f16fa3e3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f1d5033f90 con 0x55f1d50310f0 2024-04-21T08:53:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.410+0000 7f16fa3e3740 10 client.0 _ll_register_callbacks cb 0x55f1d4f41b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:16.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.411+0000 7f16f54b1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d50288e0 0x55f1d502dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.411+0000 7f16f54b1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d50288e0 0x55f1d502dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:42066/0 (socket says 172.21.15.28:42066) 2024-04-21T08:53:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.411+0000 7f16f54b1640 1 -- 172.21.15.28:0/4056761736 learned_addr learned my addr 172.21.15.28:0/4056761736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.412+0000 7f16f4cb0640 1 --2- 172.21.15.28:0/4056761736 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d50310f0 0x55f1d502e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.412+0000 7f16f54b1640 1 -- 172.21.15.28:0/4056761736 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d50310f0 msgr2=0x55f1d502e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.412+0000 7f16f54b1640 1 --2- 172.21.15.28:0/4056761736 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d50310f0 0x55f1d502e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.412+0000 7f16f4cb0640 1 --2- 172.21.15.28:0/4056761736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d5027b70 0x55f1d50309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.412+0000 7f16f54b1640 1 -- 172.21.15.28:0/4056761736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d5027b70 msgr2=0x55f1d50309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.412+0000 7f16f54b1640 1 --2- 172.21.15.28:0/4056761736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d5027b70 0x55f1d50309f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.412+0000 7f16f54b1640 1 -- 172.21.15.28:0/4056761736 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1d4c9f210 con 0x55f1d50288e0 2024-04-21T08:53:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.413+0000 7f16f4cb0640 1 --2- 172.21.15.28:0/4056761736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d5027b70 0x55f1d50309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:53:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.413+0000 7f16f54b1640 1 --2- 172.21.15.28:0/4056761736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d50288e0 0x55f1d502dd80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f16e002c2a0 tx=0x7f16e0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b269d8973c70e67e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.414+0000 7f16d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:53:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.414+0000 7f16d8ff9640 1 -- 172.21.15.28:0/4056761736 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e0043050 con 0x55f1d50288e0 2024-04-21T08:53:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.414+0000 7f16fa3e3740 1 -- 172.21.15.28:0/4056761736 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16e002c860 con 0x55f1d50288e0 2024-04-21T08:53:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.414+0000 7f16d8ff9640 1 -- 172.21.15.28:0/4056761736 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f16e0004ad0 con 0x55f1d50288e0 2024-04-21T08:53:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.414+0000 7f16d8ff9640 1 -- 172.21.15.28:0/4056761736 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f16e005b050 con 0x55f1d50288e0 2024-04-21T08:53:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.414+0000 7f16fa3e3740 10 client.25793 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.414+0000 7f16fa3e3740 1 -- 172.21.15.28:0/4056761736 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f1d4d6b620 con 0x55f1d50288e0 2024-04-21T08:53:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.415+0000 7f16d8ff9640 1 -- 172.21.15.28:0/4056761736 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16e002c860 con 0x55f1d50288e0 2024-04-21T08:53:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.415+0000 7f16d8ff9640 1 -- 172.21.15.28:0/4056761736 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f16e0048450 con 0x55f1d50288e0 2024-04-21T08:53:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.416+0000 7f16fa3e3740 10 client.25793 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.416+0000 7f16fa3e3740 10 client.25793 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.416+0000 7f16fa3e3740 20 client.25793 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.416+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/4056761736 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f1d5041260 0x55f1d50436e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.416+0000 7f16fa3e3740 4 client.25793 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.416+0000 7f16fa3e3740 1 -- 172.21.15.28:0/4056761736 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1d4fa0880 con 0x55f1d5041260 2024-04-21T08:53:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.419+0000 7f16f4cb0640 1 --2- 172.21.15.28:0/4056761736 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f1d5041260 0x55f1d50436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.428+0000 7f16d8ff9640 10 client.25793 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.428+0000 7f16f4cb0640 1 --2- 172.21.15.28:0/4056761736 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f1d5041260 0x55f1d50436e0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f7f6f0db0ab5a8b server_cookie=2db85d630f1559d0 in_seq=0 out_seq=0 2024-04-21T08:53:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.430+0000 7f16d8ff9640 1 -- 172.21.15.28:0/4056761736 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f1d4fa0880 con 0x55f1d5041260 2024-04-21T08:53:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.430+0000 7f16d8ff9640 10 client.25793 handle_command_reply: tid=0 2024-04-21T08:53:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.430+0000 7f16fa3e3740 1 client.25793 shutdown 2024-04-21T08:53:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.433+0000 7f16fa3e3740 1 -- 172.21.15.28:0/4056761736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d50288e0 msgr2=0x55f1d502dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.433+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/4056761736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d50288e0 0x55f1d502dd80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f16e002c2a0 tx=0x7f16e0009260 comp rx=0 tx=0).stop 2024-04-21T08:53:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.435+0000 7f16fa3e3740 1 -- 172.21.15.28:0/4056761736 shutdown_connections 2024-04-21T08:53:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.435+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/4056761736 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f1d5041260 0x55f1d50436e0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.435+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/4056761736 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d50310f0 0x55f1d502e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.435+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/4056761736 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d50288e0 0x55f1d502dd80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.435+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/4056761736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d5027b70 0x55f1d50309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.435+0000 7f16fa3e3740 1 -- 172.21.15.28:0/4056761736 >> 172.21.15.28:0/4056761736 conn(0x55f1d4feec90 msgr2=0x55f1d50245b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.435+0000 7f16d8ff9640 0 client.25793 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.436+0000 7f16fa3e3740 1 -- 172.21.15.28:0/4056761736 shutdown_connections 2024-04-21T08:53:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.436+0000 7f16fa3e3740 1 -- 172.21.15.28:0/4056761736 wait complete. 2024-04-21T08:53:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.436+0000 7f16fa3e3740 20 client.25793 trim_cache size 0 max 16384 2024-04-21T08:53:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.467+0000 7f16fa3e3740 1 Processor -- start 2024-04-21T08:53:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.467+0000 7f16fa3e3740 1 -- start start 2024-04-21T08:53:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.467+0000 7f16fa3e3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d5027b70 0x55f1d503cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.467+0000 7f16fa3e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50288e0 0x55f1d50242e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.468+0000 7f16fa3e3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d50310f0 0x55f1d50304b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.468+0000 7f16fa3e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16f01c0b40 con 0x55f1d50288e0 2024-04-21T08:53:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.468+0000 7f16fa3e3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f16f01be410 con 0x55f1d5027b70 2024-04-21T08:53:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.468+0000 7f16fa3e3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f16f01bc260 con 0x55f1d50310f0 2024-04-21T08:53:16.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.469+0000 7f16f54b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50288e0 0x55f1d50242e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.469+0000 7f16f54b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50288e0 0x55f1d50242e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53130/0 (socket says 172.21.15.28:53130) 2024-04-21T08:53:16.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.469+0000 7f16f54b1640 1 -- 172.21.15.28:0/3240718547 learned_addr learned my addr 172.21.15.28:0/3240718547 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:16.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.470+0000 7f16f4cb0640 1 --2- 172.21.15.28:0/3240718547 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d5027b70 0x55f1d503cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.470+0000 7f16f4cb0640 1 --2- 172.21.15.28:0/3240718547 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d50310f0 0x55f1d50304b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.471+0000 7f16f54b1640 1 -- 172.21.15.28:0/3240718547 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d50310f0 msgr2=0x55f1d50304b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.471+0000 7f16f54b1640 1 --2- 172.21.15.28:0/3240718547 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d50310f0 0x55f1d50304b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.471+0000 7f16f54b1640 1 -- 172.21.15.28:0/3240718547 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d5027b70 msgr2=0x55f1d503cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.471+0000 7f16f54b1640 1 --2- 172.21.15.28:0/3240718547 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d5027b70 0x55f1d503cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.471+0000 7f16f54b1640 1 -- 172.21.15.28:0/3240718547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1d5026e40 con 0x55f1d50288e0 2024-04-21T08:53:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.471+0000 7f16f4cb0640 1 --2- 172.21.15.28:0/3240718547 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d5027b70 0x55f1d503cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:53:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.471+0000 7f16f54b1640 1 --2- 172.21.15.28:0/3240718547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50288e0 0x55f1d50242e0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f16e0048990 tx=0x7f16e0005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1db59c148590c20 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:16.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.474+0000 7f16daffd640 1 -- 172.21.15.28:0/3240718547 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e0064070 con 0x55f1d50288e0 2024-04-21T08:53:16.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.474+0000 7f16daffd640 1 -- 172.21.15.28:0/3240718547 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16e002c860 con 0x55f1d50288e0 2024-04-21T08:53:16.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.474+0000 7f16daffd640 1 -- 172.21.15.28:0/3240718547 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e0043040 con 0x55f1d50288e0 2024-04-21T08:53:16.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.475+0000 7f16fa3e3740 1 -- 172.21.15.28:0/3240718547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50288e0 msgr2=0x55f1d50242e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:16.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.475+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/3240718547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50288e0 0x55f1d50242e0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f16e0048990 tx=0x7f16e0005a00 comp rx=0 tx=0).stop 2024-04-21T08:53:16.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.475+0000 7f16fa3e3740 1 -- 172.21.15.28:0/3240718547 shutdown_connections 2024-04-21T08:53:16.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.475+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/3240718547 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d50310f0 0x55f1d50304b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.475+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/3240718547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50288e0 0x55f1d50242e0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.475+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/3240718547 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d5027b70 0x55f1d503cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.475+0000 7f16fa3e3740 1 -- 172.21.15.28:0/3240718547 >> 172.21.15.28:0/3240718547 conn(0x55f1d4feec90 msgr2=0x55f1d5023bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:16.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.475+0000 7f16fa3e3740 1 -- 172.21.15.28:0/3240718547 shutdown_connections 2024-04-21T08:53:16.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.475+0000 7f16fa3e3740 1 -- 172.21.15.28:0/3240718547 wait complete. 2024-04-21T08:53:16.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.476+0000 7f16fa3e3740 1 -- start start 2024-04-21T08:53:16.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.476+0000 7f16fa3e3740 1 Processor -- start 2024-04-21T08:53:16.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.476+0000 7f16fa3e3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d5027b70 0x55f1d5039550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.476+0000 7f16fa3e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50288e0 0x55f1d5031ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.476+0000 7f16fa3e3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d50310f0 0x55f1d5031ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.476+0000 7f16fa3e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16f01bc8d0 con 0x55f1d50288e0 2024-04-21T08:53:16.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.476+0000 7f16fa3e3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f16f01bdd40 con 0x55f1d50310f0 2024-04-21T08:53:16.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.476+0000 7f16fa3e3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f16f01bcfa0 con 0x55f1d5027b70 2024-04-21T08:53:16.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.476+0000 7f16fa3e3740 10 client.0 _ll_register_callbacks cb 0x7f16e0009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:16.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.477+0000 7f16f54b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50288e0 0x55f1d5031ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.477+0000 7f16f54b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50288e0 0x55f1d5031ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53146/0 (socket says 172.21.15.28:53146) 2024-04-21T08:53:16.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.477+0000 7f16f54b1640 1 -- 172.21.15.28:0/726159327 learned_addr learned my addr 172.21.15.28:0/726159327 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:16.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.477+0000 7f16f4cb0640 1 --2- 172.21.15.28:0/726159327 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d5027b70 0x55f1d5039550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.477+0000 7f16f4cb0640 1 --2- 172.21.15.28:0/726159327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d50310f0 0x55f1d5031ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.477+0000 7f16f54b1640 1 -- 172.21.15.28:0/726159327 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d5027b70 msgr2=0x55f1d5039550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:16.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.477+0000 7f16f54b1640 1 --2- 172.21.15.28:0/726159327 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d5027b70 0x55f1d5039550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.477+0000 7f16f54b1640 1 -- 172.21.15.28:0/726159327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d50310f0 msgr2=0x55f1d5031ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:16.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.477+0000 7f16f54b1640 1 --2- 172.21.15.28:0/726159327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d50310f0 0x55f1d5031ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:16.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.477+0000 7f16f54b1640 1 -- 172.21.15.28:0/726159327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1d5032730 con 0x55f1d50288e0 2024-04-21T08:53:16.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.478+0000 7f16f54b1640 1 --2- 172.21.15.28:0/726159327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50288e0 0x55f1d5031ab0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f16e0062040 tx=0x7f16e0005a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3834a4f72fde4fd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:16.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.478+0000 7f16d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.482+0000 7f16d8ff9640 1 -- 172.21.15.28:0/726159327 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e0064030 con 0x55f1d50288e0 2024-04-21T08:53:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.483+0000 7f16d8ff9640 1 -- 172.21.15.28:0/726159327 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f16e002ce00 con 0x55f1d50288e0 2024-04-21T08:53:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.483+0000 7f16fa3e3740 1 -- 172.21.15.28:0/726159327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16e002c860 con 0x55f1d50288e0 2024-04-21T08:53:16.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.487+0000 7f16d8ff9640 1 -- 172.21.15.28:0/726159327 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f16e0043040 con 0x55f1d50288e0 2024-04-21T08:53:16.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.489+0000 7f16fa3e3740 10 client.16446 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:16.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.489+0000 7f16fa3e3740 1 -- 172.21.15.28:0/726159327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f1d50b8830 con 0x55f1d50288e0 2024-04-21T08:53:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.494+0000 7f16d8ff9640 1 -- 172.21.15.28:0/726159327 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16e002c860 con 0x55f1d50288e0 2024-04-21T08:53:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.494+0000 7f16d8ff9640 1 -- 172.21.15.28:0/726159327 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f16e00035f0 con 0x55f1d50288e0 2024-04-21T08:53:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.494+0000 7f16fa3e3740 10 client.16446 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.494+0000 7f16fa3e3740 10 client.16446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.494+0000 7f16fa3e3740 20 client.16446 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.494+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/726159327 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f1d50b8c50 0x55f1d50bb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.494+0000 7f16fa3e3740 4 client.16446 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:53:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.494+0000 7f16fa3e3740 1 -- 172.21.15.28:0/726159327 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f1d4f2c3d0 con 0x55f1d50b8c50 2024-04-21T08:53:16.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.496+0000 7f16f4cb0640 1 --2- 172.21.15.28:0/726159327 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f1d50b8c50 0x55f1d50bb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:16.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.497+0000 7f16f4cb0640 1 --2- 172.21.15.28:0/726159327 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f1d50b8c50 0x55f1d50bb070 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3de7194026ba10f1 server_cookie=c50e3b1bbe5f8311 in_seq=0 out_seq=0 2024-04-21T08:53:16.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:16.498+0000 7f16d8ff9640 10 client.16446 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:18.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:18 smithi152 ceph-mon[31401]: pgmap v66: 129 pgs: 129 active+clean; 13 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 125 MiB/s wr, 13.75k op/s 2024-04-21T08:53:18.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:18 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:18.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:18 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:53:18.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:18 smithi028 ceph-mon[28003]: pgmap v66: 129 pgs: 129 active+clean; 13 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 125 MiB/s wr, 13.75k op/s 2024-04-21T08:53:18.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:18 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:18.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:18 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:53:18.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:18 smithi171 ceph-mon[31358]: pgmap v66: 129 pgs: 129 active+clean; 13 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 125 MiB/s wr, 13.75k op/s 2024-04-21T08:53:18.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:18 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:18.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:18 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:53:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.395+0000 7f16d8ff9640 1 -- 172.21.15.28:0/726159327 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f1d4f2c3d0 con 0x55f1d50b8c50 2024-04-21T08:53:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.395+0000 7f16d8ff9640 10 client.16446 handle_command_reply: tid=0 2024-04-21T08:53:18.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.395+0000 7f16fa3e3740 1 client.16446 shutdown 2024-04-21T08:53:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.395+0000 7f16fa3e3740 1 -- 172.21.15.28:0/726159327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50288e0 msgr2=0x55f1d5031ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.395+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/726159327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50288e0 0x55f1d5031ab0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f16e0062040 tx=0x7f16e0005a70 comp rx=0 tx=0).stop 2024-04-21T08:53:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.395+0000 7f16fa3e3740 1 -- 172.21.15.28:0/726159327 shutdown_connections 2024-04-21T08:53:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.395+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/726159327 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f1d50b8c50 0x55f1d50bb070 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.395+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/726159327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f1d50310f0 0x55f1d5031ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.395+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/726159327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1d50288e0 0x55f1d5031ab0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.395+0000 7f16fa3e3740 1 --2- 172.21.15.28:0/726159327 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f1d5027b70 0x55f1d5039550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.395+0000 7f16fa3e3740 1 -- 172.21.15.28:0/726159327 >> 172.21.15.28:0/726159327 conn(0x55f1d4feec90 msgr2=0x55f1d5021980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.395+0000 7f16d8ff9640 0 client.16446 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.396+0000 7f16fa3e3740 1 -- 172.21.15.28:0/726159327 shutdown_connections 2024-04-21T08:53:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.396+0000 7f16fa3e3740 1 -- 172.21.15.28:0/726159327 wait complete. 2024-04-21T08:53:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.396+0000 7f16fa3e3740 20 client.16446 trim_cache size 0 max 16384 2024-04-21T08:53:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:53:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:53:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:53:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:53:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.397+0000 7f16f7715640 1 -- 172.21.15.28:0/3531474293 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f16dc0a3e90 msgr2=0x7f16dc0a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.397+0000 7f16f7715640 1 --2- 172.21.15.28:0/3531474293 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f16dc0a3e90 0x7f16dc0a6350 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f16e8098580 tx=0x7f16e800ce70 comp rx=0 tx=0).stop 2024-04-21T08:53:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.397+0000 7f16f7715640 1 -- 172.21.15.28:0/3531474293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f01abf50 msgr2=0x7f16f01ac400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.397+0000 7f16f7715640 1 --2- 172.21.15.28:0/3531474293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f01abf50 0x7f16f01ac400 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f16e800bdf0 tx=0x7f16e8009ef0 comp rx=0 tx=0).stop 2024-04-21T08:53:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.398+0000 7f16f7715640 1 -- 172.21.15.28:0/3531474293 shutdown_connections 2024-04-21T08:53:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.398+0000 7f16f7715640 1 --2- 172.21.15.28:0/3531474293 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f16dc0a3e90 0x7f16dc0a6350 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.398+0000 7f16f7715640 1 --2- 172.21.15.28:0/3531474293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16f01abf50 0x7f16f01ac400 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.398+0000 7f16f7715640 1 --2- 172.21.15.28:0/3531474293 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f16f011b730 0x7f16f01a7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:18.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.398+0000 7f16f7715640 1 --2- 172.21.15.28:0/3531474293 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f0110740 0x7f16f01a7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:18.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.399+0000 7f16f7715640 1 -- 172.21.15.28:0/3531474293 >> 172.21.15.28:0/3531474293 conn(0x7f16f006d020 msgr2=0x7f16f010e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:18.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.399+0000 7f16f7715640 1 -- 172.21.15.28:0/3531474293 shutdown_connections 2024-04-21T08:53:18.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:18.399+0000 7f16f7715640 1 -- 172.21.15.28:0/3531474293 wait complete. 2024-04-21T08:53:19.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:19 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:53:19.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:19 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:53:19.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:19 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:53:19.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:19 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:53:19.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:19 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:53:19.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:19 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:53:19.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:19 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:53:19.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:19 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:53:19.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:19 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:53:20.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:20 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:20.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:20 smithi152 ceph-mon[31401]: pgmap v67: 129 pgs: 129 active+clean; 13 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 178 MiB/s wr, 19.75k op/s 2024-04-21T08:53:20.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:20 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:20.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:20 smithi028 ceph-mon[28003]: pgmap v67: 129 pgs: 129 active+clean; 13 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 178 MiB/s wr, 19.75k op/s 2024-04-21T08:53:20.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:20 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:20.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:20 smithi171 ceph-mon[31358]: pgmap v67: 129 pgs: 129 active+clean; 13 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 178 MiB/s wr, 19.75k op/s 2024-04-21T08:53:22.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:22 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:53:22.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:22 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:22.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:22 smithi152 ceph-mon[31401]: pgmap v68: 129 pgs: 129 active+clean; 13 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 132 MiB/s wr, 14.76k op/s 2024-04-21T08:53:22.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:22 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:53:22.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:22 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:22.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:22 smithi028 ceph-mon[28003]: pgmap v68: 129 pgs: 129 active+clean; 13 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 132 MiB/s wr, 14.76k op/s 2024-04-21T08:53:22.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:22 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:53:22.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:22 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:22.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:22 smithi171 ceph-mon[31358]: pgmap v68: 129 pgs: 129 active+clean; 13 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 132 MiB/s wr, 14.76k op/s 2024-04-21T08:53:24.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:24 smithi152 ceph-mon[31401]: pgmap v69: 129 pgs: 129 active+clean; 13 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 164 MiB/s wr, 18.71k op/s 2024-04-21T08:53:24.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:24 smithi028 ceph-mon[28003]: pgmap v69: 129 pgs: 129 active+clean; 13 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 164 MiB/s wr, 18.71k op/s 2024-04-21T08:53:24.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:24 smithi171 ceph-mon[31358]: pgmap v69: 129 pgs: 129 active+clean; 13 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 164 MiB/s wr, 18.71k op/s 2024-04-21T08:53:25.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:25 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:25.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:25 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:25.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:25 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:26.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:26 smithi152 ceph-mon[31401]: pgmap v70: 129 pgs: 129 active+clean; 13 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 115 MiB/s wr, 13.51k op/s 2024-04-21T08:53:26.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:26 smithi028 ceph-mon[28003]: pgmap v70: 129 pgs: 129 active+clean; 13 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 115 MiB/s wr, 13.51k op/s 2024-04-21T08:53:26.369 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:26 smithi171 ceph-mon[31358]: pgmap v70: 129 pgs: 129 active+clean; 13 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 115 MiB/s wr, 13.51k op/s 2024-04-21T08:53:27.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:27 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:53:27.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:27 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:53:27.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:27 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:53:27.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:27 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:53:27.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:27 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:53:27.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:27 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:53:28.321 INFO:tasks.quiescer.fs.[cephfs]:Total duration (49.0) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-21T08:53:28.321 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.5 2024-04-21T08:53:28.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:53:28.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:28 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:53:28.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:28 smithi152 ceph-mon[31401]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 108 MiB/s wr, 12.78k op/s 2024-04-21T08:53:28.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:28 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:53:28.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:28 smithi028 ceph-mon[28003]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 108 MiB/s wr, 12.78k op/s 2024-04-21T08:53:28.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:28 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:53:28.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:28 smithi171 ceph-mon[31358]: pgmap v71: 129 pgs: 129 active+clean; 13 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 108 MiB/s wr, 12.78k op/s 2024-04-21T08:53:28.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.519+0000 7f4e2bca5640 1 -- 172.21.15.28:0/1247094369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4e241106c0 msgr2=0x7f4e24108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:28.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.519+0000 7f4e2bca5640 1 --2- 172.21.15.28:0/1247094369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4e241106c0 0x7f4e24108f50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4e20009900 tx=0x7f4e2002bc80 comp rx=0 tx=0).stop 2024-04-21T08:53:28.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.520+0000 7f4e2bca5640 1 -- 172.21.15.28:0/1247094369 shutdown_connections 2024-04-21T08:53:28.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.520+0000 7f4e2bca5640 1 --2- 172.21.15.28:0/1247094369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e24109b70 0x7f4e2411d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:28.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.520+0000 7f4e2bca5640 1 --2- 172.21.15.28:0/1247094369 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e24110b30 0x7f4e24109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:28.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.520+0000 7f4e2bca5640 1 --2- 172.21.15.28:0/1247094369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4e241106c0 0x7f4e24108f50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:28.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.520+0000 7f4e2bca5640 1 -- 172.21.15.28:0/1247094369 >> 172.21.15.28:0/1247094369 conn(0x7f4e2406e410 msgr2=0x7f4e240719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:28.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.521+0000 7f4e2bca5640 1 -- 172.21.15.28:0/1247094369 shutdown_connections 2024-04-21T08:53:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.522+0000 7f4e2bca5640 1 -- 172.21.15.28:0/1247094369 wait complete. 2024-04-21T08:53:28.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.523+0000 7f4e2bca5640 1 Processor -- start 2024-04-21T08:53:28.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.523+0000 7f4e2bca5640 1 -- start start 2024-04-21T08:53:28.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.524+0000 7f4e2bca5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e241106c0 0x7f4e241a7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:28.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.524+0000 7f4e2bca5640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4e24110b30 0x7f4e241a7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:28.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.524+0000 7f4e2bca5640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e241abe10 0x7f4e241ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:28.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.524+0000 7f4e2bca5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e24120ff0 con 0x7f4e241106c0 2024-04-21T08:53:28.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.524+0000 7f4e2bca5640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4e24120e70 con 0x7f4e24110b30 2024-04-21T08:53:28.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.524+0000 7f4e2bca5640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4e24121170 con 0x7f4e241abe10 2024-04-21T08:53:28.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.526+0000 7f4e2a242640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e241abe10 0x7f4e241ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:28.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.526+0000 7f4e2a242640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e241abe10 0x7f4e241ac2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:37886/0 (socket says 172.21.15.28:37886) 2024-04-21T08:53:28.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.526+0000 7f4e2a242640 1 -- 172.21.15.28:0/347413090 learned_addr learned my addr 172.21.15.28:0/347413090 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:28.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.527+0000 7f4e2a242640 1 -- 172.21.15.28:0/347413090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4e24110b30 msgr2=0x7f4e241a7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:28.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.527+0000 7f4e2a242640 1 --2- 172.21.15.28:0/347413090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4e24110b30 0x7f4e241a7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:28.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.527+0000 7f4e2a242640 1 -- 172.21.15.28:0/347413090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e241106c0 msgr2=0x7f4e241a7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:28.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.527+0000 7f4e2a242640 1 --2- 172.21.15.28:0/347413090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e241106c0 0x7f4e241a7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:28.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.527+0000 7f4e2a242640 1 -- 172.21.15.28:0/347413090 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e241ac900 con 0x7f4e241abe10 2024-04-21T08:53:28.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.528+0000 7f4e2a242640 1 --2- 172.21.15.28:0/347413090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e241abe10 0x7f4e241ac2c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c00eb10 tx=0x7f4e1c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:28.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.530+0000 7f4e1affd640 1 -- 172.21.15.28:0/347413090 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c004b20 con 0x7f4e241abe10 2024-04-21T08:53:28.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.530+0000 7f4e1affd640 1 -- 172.21.15.28:0/347413090 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e1c004cc0 con 0x7f4e241abe10 2024-04-21T08:53:28.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.530+0000 7f4e1affd640 1 -- 172.21.15.28:0/347413090 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c010670 con 0x7f4e241abe10 2024-04-21T08:53:28.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.530+0000 7f4e2bca5640 1 -- 172.21.15.28:0/347413090 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e241acbf0 con 0x7f4e241abe10 2024-04-21T08:53:28.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.530+0000 7f4e2bca5640 1 -- 172.21.15.28:0/347413090 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e241111e0 con 0x7f4e241abe10 2024-04-21T08:53:28.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.539+0000 7f4e2bca5640 1 -- 172.21.15.28:0/347413090 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e08005290 con 0x7f4e241abe10 2024-04-21T08:53:28.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.543+0000 7f4e1affd640 1 -- 172.21.15.28:0/347413090 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4e1c010810 con 0x7f4e241abe10 2024-04-21T08:53:28.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.544+0000 7f4e1affd640 1 --2- 172.21.15.28:0/347413090 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4e100a3f30 0x7f4e100a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:28.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.544+0000 7f4e1affd640 1 -- 172.21.15.28:0/347413090 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f4e1c0d1490 con 0x7f4e241abe10 2024-04-21T08:53:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.545+0000 7f4e29a41640 1 --2- 172.21.15.28:0/347413090 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4e100a3f30 0x7f4e100a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.546+0000 7f4e29a41640 1 --2- 172.21.15.28:0/347413090 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4e100a3f30 0x7f4e100a63f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4e200096c0 tx=0x7f4e20000bc0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:28.553+0000 7f4e1affd640 1 -- 172.21.15.28:0/347413090 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e1c09d7b0 con 0x7f4e241abe10 2024-04-21T08:53:29.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.030+0000 7f4e2bca5640 1 -- 172.21.15.28:0/347413090 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e08005580 con 0x7f4e241abe10 2024-04-21T08:53:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.032+0000 7f4e1affd640 1 -- 172.21.15.28:0/347413090 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7f4e1c09d0b0 con 0x7f4e241abe10 2024-04-21T08:53:29.032 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:53:29.032 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:52:11.568185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24487,24488,24463]},"id":1}]} 2024-04-21T08:53:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-21T08:53:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.036+0000 7f4e2bca5640 1 -- 172.21.15.28:0/347413090 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4e100a3f30 msgr2=0x7f4e100a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.036+0000 7f4e2bca5640 1 --2- 172.21.15.28:0/347413090 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4e100a3f30 0x7f4e100a63f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4e200096c0 tx=0x7f4e20000bc0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.036+0000 7f4e2bca5640 1 -- 172.21.15.28:0/347413090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e241abe10 msgr2=0x7f4e241ac2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.036+0000 7f4e2bca5640 1 --2- 172.21.15.28:0/347413090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e241abe10 0x7f4e241ac2c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c00eb10 tx=0x7f4e1c00c2d0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.037+0000 7f4e2bca5640 1 -- 172.21.15.28:0/347413090 shutdown_connections 2024-04-21T08:53:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.037+0000 7f4e2bca5640 1 --2- 172.21.15.28:0/347413090 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4e100a3f30 0x7f4e100a63f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.037+0000 7f4e2bca5640 1 --2- 172.21.15.28:0/347413090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e241abe10 0x7f4e241ac2c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.037+0000 7f4e2bca5640 1 --2- 172.21.15.28:0/347413090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4e24110b30 0x7f4e241a7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.037+0000 7f4e2bca5640 1 --2- 172.21.15.28:0/347413090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e241106c0 0x7f4e241a7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.037+0000 7f4e2bca5640 1 -- 172.21.15.28:0/347413090 >> 172.21.15.28:0/347413090 conn(0x7f4e2406e410 msgr2=0x7f4e24119af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.037+0000 7f4e2bca5640 1 -- 172.21.15.28:0/347413090 shutdown_connections 2024-04-21T08:53:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.037+0000 7f4e2bca5640 1 -- 172.21.15.28:0/347413090 wait complete. 2024-04-21T08:53:29.055 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24488 quiesce db / --timeout 30 --expiration 84.5 --await' 2024-04-21T08:53:29.055 DEBUG:teuthology.orchestra.run.smithi028:> timeout 45 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24488 quiesce db / --timeout 30 --expiration 84.5 --await 2024-04-21T08:53:29.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:29 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/347413090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:53:29.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:29 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/347413090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:53:29.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:29 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/347413090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:53:29.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.764+0000 7f377ebf4640 1 -- 172.21.15.28:0/1663085195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f377810eeb0 msgr2=0x7f377811d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:29.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.764+0000 7f377ebf4640 1 --2- 172.21.15.28:0/1663085195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f377810eeb0 0x7f377811d7a0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f376c009900 tx=0x7f376c02bc60 comp rx=0 tx=0).stop 2024-04-21T08:53:29.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.764+0000 7f377ebf4640 1 -- 172.21.15.28:0/1663085195 shutdown_connections 2024-04-21T08:53:29.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.764+0000 7f377ebf4640 1 --2- 172.21.15.28:0/1663085195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f377810eeb0 0x7f377811d7a0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.764+0000 7f377ebf4640 1 --2- 172.21.15.28:0/1663085195 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3778110b20 0x7f377810e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.764+0000 7f377ebf4640 1 --2- 172.21.15.28:0/1663085195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37781106b0 0x7f377810e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.764+0000 7f377ebf4640 1 -- 172.21.15.28:0/1663085195 >> 172.21.15.28:0/1663085195 conn(0x7f377806d070 msgr2=0x7f377806d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:29.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.768+0000 7f377ebf4640 1 -- 172.21.15.28:0/1663085195 shutdown_connections 2024-04-21T08:53:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.771+0000 7f377ebf4640 1 -- 172.21.15.28:0/1663085195 wait complete. 2024-04-21T08:53:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.772+0000 7f377ebf4640 1 Processor -- start 2024-04-21T08:53:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.774+0000 7f377ebf4640 1 -- start start 2024-04-21T08:53:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.774+0000 7f377ebf4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37781106b0 0x7f37781a76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.774+0000 7f377ebf4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37781a7c00 0x7f37781abf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.774+0000 7f377ebf4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37781ac4c0 0x7f37781ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.774+0000 7f377ebf4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f37781210e0 con 0x7f37781106b0 2024-04-21T08:53:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.774+0000 7f377ebf4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3778120f60 con 0x7f37781ac4c0 2024-04-21T08:53:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.774+0000 7f377ebf4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3778121260 con 0x7f37781a7c00 2024-04-21T08:53:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.775+0000 7f377d191640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37781ac4c0 0x7f37781ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.775+0000 7f3777fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37781a7c00 0x7f37781abf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.775+0000 7f3777fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37781a7c00 0x7f37781abf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:37908/0 (socket says 172.21.15.28:37908) 2024-04-21T08:53:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.775+0000 7f3777fff640 1 -- 172.21.15.28:0/727886785 learned_addr learned my addr 172.21.15.28:0/727886785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.775+0000 7f3777fff640 1 -- 172.21.15.28:0/727886785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37781ac4c0 msgr2=0x7f37781ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.775+0000 7f3777fff640 1 --2- 172.21.15.28:0/727886785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37781ac4c0 0x7f37781ac970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.775+0000 7f3777fff640 1 -- 172.21.15.28:0/727886785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37781106b0 msgr2=0x7f37781a76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.775+0000 7f3777fff640 1 --2- 172.21.15.28:0/727886785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37781106b0 0x7f37781a76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.775+0000 7f3777fff640 1 -- 172.21.15.28:0/727886785 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37781b4230 con 0x7f37781a7c00 2024-04-21T08:53:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.776+0000 7f3777fff640 1 --2- 172.21.15.28:0/727886785 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37781a7c00 0x7f37781abf80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f376800b4a0 tx=0x7f376800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:29.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.776+0000 7f3775ffb640 1 -- 172.21.15.28:0/727886785 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3768014070 con 0x7f37781a7c00 2024-04-21T08:53:29.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.776+0000 7f377ebf4640 1 -- 172.21.15.28:0/727886785 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37781b4460 con 0x7f37781a7c00 2024-04-21T08:53:29.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.777+0000 7f377ebf4640 1 -- 172.21.15.28:0/727886785 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37781b49d0 con 0x7f37781a7c00 2024-04-21T08:53:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.780+0000 7f37817ed740 1 Processor -- start 2024-04-21T08:53:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.781+0000 7f3775ffb640 1 -- 172.21.15.28:0/727886785 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3768010ac0 con 0x7f37781a7c00 2024-04-21T08:53:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.781+0000 7f3775ffb640 1 -- 172.21.15.28:0/727886785 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3768017c20 con 0x7f37781a7c00 2024-04-21T08:53:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.781+0000 7f37817ed740 1 -- start start 2024-04-21T08:53:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.781+0000 7f37817ed740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc7443ae0 0x561bc7443ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.781+0000 7f37817ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc744cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.781+0000 7f37817ed740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc744d1d0 0x561bc744f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.781+0000 7f37817ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561bc73a40b0 con 0x561bc7444400 2024-04-21T08:53:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.781+0000 7f37817ed740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561bc7355680 con 0x561bc744d1d0 2024-04-21T08:53:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.781+0000 7f37817ed740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x561bc7361c60 con 0x561bc7443ae0 2024-04-21T08:53:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.782+0000 7f377d191640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc7443ae0 0x561bc7443ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.782+0000 7f377d191640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc7443ae0 0x561bc7443ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:37914/0 (socket says 172.21.15.28:37914) 2024-04-21T08:53:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.782+0000 7f377d191640 1 -- 172.21.15.28:0/541878550 learned_addr learned my addr 172.21.15.28:0/541878550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.782+0000 7f377d191640 1 -- 172.21.15.28:0/541878550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc744d1d0 msgr2=0x561bc744f5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.782+0000 7f377d191640 1 --2- 172.21.15.28:0/541878550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc744d1d0 0x561bc744f5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.782+0000 7f377d191640 1 -- 172.21.15.28:0/541878550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 msgr2=0x561bc744cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.782+0000 7f377d191640 1 --2- 172.21.15.28:0/541878550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc744cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.783+0000 7f377d191640 1 -- 172.21.15.28:0/541878550 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bc7186c90 con 0x561bc7443ae0 2024-04-21T08:53:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.783+0000 7f377d191640 1 --2- 172.21.15.28:0/541878550 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc7443ae0 0x561bc7443ec0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f376c02c270 tx=0x7f376c02c6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87f693bb7ae1fdb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:29.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.784+0000 7f377ebf4640 1 -- 172.21.15.28:0/541878550 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376c044020 con 0x561bc7443ae0 2024-04-21T08:53:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.784+0000 7f377ebf4640 1 -- 172.21.15.28:0/541878550 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f376c02ccc0 con 0x561bc7443ae0 2024-04-21T08:53:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.784+0000 7f377ebf4640 1 -- 172.21.15.28:0/541878550 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376c0046c0 con 0x561bc7443ae0 2024-04-21T08:53:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.784+0000 7f3775ffb640 1 -- 172.21.15.28:0/727886785 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3768017070 con 0x7f37781a7c00 2024-04-21T08:53:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.785+0000 7f37817ed740 1 -- 172.21.15.28:0/541878550 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc7443ae0 msgr2=0x561bc7443ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.785+0000 7f37817ed740 1 --2- 172.21.15.28:0/541878550 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc7443ae0 0x561bc7443ec0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f376c02c270 tx=0x7f376c02c6b0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.786+0000 7f37817ed740 1 -- 172.21.15.28:0/541878550 shutdown_connections 2024-04-21T08:53:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.786+0000 7f37817ed740 1 --2- 172.21.15.28:0/541878550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc744d1d0 0x561bc744f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.786+0000 7f37817ed740 1 --2- 172.21.15.28:0/541878550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc744cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.786+0000 7f37817ed740 1 --2- 172.21.15.28:0/541878550 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc7443ae0 0x561bc7443ec0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.786+0000 7f37817ed740 1 -- 172.21.15.28:0/541878550 >> 172.21.15.28:0/541878550 conn(0x561bc743e400 msgr2=0x561bc743e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.789+0000 7f3775ffb640 1 --2- 172.21.15.28:0/727886785 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f37600a3d30 0x7f37600a61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.790+0000 7f3775ffb640 1 -- 172.21.15.28:0/727886785 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f37680cfc80 con 0x7f37781a7c00 2024-04-21T08:53:29.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.791+0000 7f37817ed740 1 -- 172.21.15.28:0/541878550 shutdown_connections 2024-04-21T08:53:29.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.795+0000 7f377c990640 1 --2- 172.21.15.28:0/727886785 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f37600a3d30 0x7f37600a61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:29.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.796+0000 7f37817ed740 1 -- 172.21.15.28:0/541878550 wait complete. 2024-04-21T08:53:29.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.796+0000 7f37817ed740 1 -- start start 2024-04-21T08:53:29.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.796+0000 7f37817ed740 1 Processor -- start 2024-04-21T08:53:29.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.797+0000 7f37817ed740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc7443ae0 0x561bc744a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.797+0000 7f37817ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc744a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.797+0000 7f37817ed740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc744d1d0 0x561bc744ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.797+0000 7f37817ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561bc73a7400 con 0x561bc7444400 2024-04-21T08:53:29.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.797+0000 7f37817ed740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561bc7363840 con 0x561bc744d1d0 2024-04-21T08:53:29.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.797+0000 7f37817ed740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x561bc73ac720 con 0x561bc7443ae0 2024-04-21T08:53:29.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.797+0000 7f37817ed740 10 client.0 _ll_register_callbacks cb 0x561bc73b4ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.798+0000 7f377d191640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc744d1d0 0x561bc744ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.798+0000 7f3777fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc744a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.798+0000 7f3777fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc744a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58584/0 (socket says 172.21.15.28:58584) 2024-04-21T08:53:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.798+0000 7f3777fff640 1 -- 172.21.15.28:0/3881091795 learned_addr learned my addr 172.21.15.28:0/3881091795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.800+0000 7f3777fff640 1 -- 172.21.15.28:0/3881091795 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc7443ae0 msgr2=0x561bc744a260 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.800+0000 7f3777fff640 1 --2- 172.21.15.28:0/3881091795 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc7443ae0 0x561bc744a260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.800+0000 7f3777fff640 1 -- 172.21.15.28:0/3881091795 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc744d1d0 msgr2=0x561bc744ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.800+0000 7f3777fff640 1 --2- 172.21.15.28:0/3881091795 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc744d1d0 0x561bc744ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.800+0000 7f3777fff640 1 -- 172.21.15.28:0/3881091795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bc71e7420 con 0x561bc7444400 2024-04-21T08:53:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.802+0000 7f3777fff640 1 --2- 172.21.15.28:0/3881091795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc744a7a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3768000c00 tx=0x7f376809ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f7d4ea6b978933a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.802+0000 7f3757fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.804+0000 7f377c990640 1 --2- 172.21.15.28:0/727886785 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f37600a3d30 0x7f37600a61f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x561bc7368460 tx=0x7f3770009230 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.804+0000 7f3757fff640 1 -- 172.21.15.28:0/3881091795 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376801e070 con 0x561bc7444400 2024-04-21T08:53:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.804+0000 7f37817ed740 1 -- 172.21.15.28:0/3881091795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f376c02ccc0 con 0x561bc7444400 2024-04-21T08:53:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.804+0000 7f3757fff640 1 -- 172.21.15.28:0/3881091795 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f3768010610 con 0x561bc7444400 2024-04-21T08:53:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.805+0000 7f3757fff640 1 -- 172.21.15.28:0/3881091795 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f37680cd250 con 0x561bc7444400 2024-04-21T08:53:29.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.807+0000 7f37817ed740 10 client.16455 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:29.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.807+0000 7f37817ed740 1 -- 172.21.15.28:0/3881091795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561bc707a580 con 0x561bc7444400 2024-04-21T08:53:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.809+0000 7f3757fff640 1 -- 172.21.15.28:0/3881091795 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f376c02ccc0 con 0x561bc7444400 2024-04-21T08:53:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.809+0000 7f3757fff640 1 -- 172.21.15.28:0/3881091795 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f37680baaf0 con 0x561bc7444400 2024-04-21T08:53:29.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.810+0000 7f37817ed740 10 client.16455 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:29.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.810+0000 7f37817ed740 10 client.16455 resolve_mds: validated gid 24488 aka daemon mds.b 2024-04-21T08:53:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.810+0000 7f37817ed740 20 client.16455 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.810+0000 7f37817ed740 1 --2- 172.21.15.28:0/3881091795 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561bc747ed20 0x561bc7481140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.810+0000 7f37817ed740 4 client.16455 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.810+0000 7f37817ed740 1 -- 172.21.15.28:0/3881091795 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bc73b5b10 con 0x561bc747ed20 2024-04-21T08:53:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.812+0000 7f377c990640 1 --2- 172.21.15.28:0/3881091795 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561bc747ed20 0x561bc7481140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.819+0000 7f377c990640 1 --2- 172.21.15.28:0/3881091795 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561bc747ed20 0x561bc7481140 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef863cc26de8df8d server_cookie=21eae55681423498 in_seq=0 out_seq=0 2024-04-21T08:53:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.819+0000 7f3757fff640 10 client.16455 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.822+0000 7f3757fff640 1 -- 172.21.15.28:0/3881091795 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561bc73b5b10 con 0x561bc747ed20 2024-04-21T08:53:29.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.822+0000 7f3757fff640 10 client.16455 handle_command_reply: tid=0 2024-04-21T08:53:29.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.822+0000 7f37817ed740 1 client.16455 shutdown 2024-04-21T08:53:29.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.824+0000 7f37817ed740 1 -- 172.21.15.28:0/3881091795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 msgr2=0x561bc744a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:29.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.824+0000 7f37817ed740 1 --2- 172.21.15.28:0/3881091795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc744a7a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3768000c00 tx=0x7f376809ba70 comp rx=0 tx=0).stop 2024-04-21T08:53:29.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.824+0000 7f37817ed740 1 -- 172.21.15.28:0/3881091795 shutdown_connections 2024-04-21T08:53:29.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.824+0000 7f37817ed740 1 --2- 172.21.15.28:0/3881091795 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561bc747ed20 0x561bc7481140 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.824+0000 7f37817ed740 1 --2- 172.21.15.28:0/3881091795 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc744d1d0 0x561bc744ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.824+0000 7f37817ed740 1 --2- 172.21.15.28:0/3881091795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc744a7a0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.824+0000 7f37817ed740 1 --2- 172.21.15.28:0/3881091795 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc7443ae0 0x561bc744a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.824+0000 7f37817ed740 1 -- 172.21.15.28:0/3881091795 >> 172.21.15.28:0/3881091795 conn(0x561bc743e400 msgr2=0x561bc7440150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:29.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.825+0000 7f3757fff640 0 client.16455 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:29.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.826+0000 7f37817ed740 1 -- 172.21.15.28:0/3881091795 shutdown_connections 2024-04-21T08:53:29.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.827+0000 7f37817ed740 1 -- 172.21.15.28:0/3881091795 wait complete. 2024-04-21T08:53:29.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.827+0000 7f37817ed740 20 client.16455 trim_cache size 0 max 16384 2024-04-21T08:53:29.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.866+0000 7f37817ed740 1 Processor -- start 2024-04-21T08:53:29.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.866+0000 7f37817ed740 1 -- start start 2024-04-21T08:53:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.867+0000 7f37817ed740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc7443ae0 0x561bc747a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.867+0000 7f37817ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc74491a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.867+0000 7f37817ed740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc744d1d0 0x561bc7445780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.867+0000 7f37817ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f37640101d0 con 0x561bc7444400 2024-04-21T08:53:29.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.867+0000 7f37817ed740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f376400e020 con 0x561bc7443ae0 2024-04-21T08:53:29.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.868+0000 7f37817ed740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f376400fb00 con 0x561bc744d1d0 2024-04-21T08:53:29.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.868+0000 7f3777fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc74491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.868+0000 7f3777fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc74491a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58586/0 (socket says 172.21.15.28:58586) 2024-04-21T08:53:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.868+0000 7f3777fff640 1 -- 172.21.15.28:0/2592127793 learned_addr learned my addr 172.21.15.28:0/2592127793 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.868+0000 7f377c990640 1 --2- 172.21.15.28:0/2592127793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc7443ae0 0x561bc747a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.868+0000 7f377d191640 1 --2- 172.21.15.28:0/2592127793 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc744d1d0 0x561bc7445780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.869+0000 7f377c990640 1 -- 172.21.15.28:0/2592127793 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc744d1d0 msgr2=0x561bc7445780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.869+0000 7f377c990640 1 --2- 172.21.15.28:0/2592127793 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc744d1d0 0x561bc7445780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.869+0000 7f377c990640 1 -- 172.21.15.28:0/2592127793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 msgr2=0x561bc74491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.869+0000 7f377c990640 1 --2- 172.21.15.28:0/2592127793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc74491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.869+0000 7f377c990640 1 -- 172.21.15.28:0/2592127793 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bc7444230 con 0x561bc7443ae0 2024-04-21T08:53:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.870+0000 7f377c990640 1 --2- 172.21.15.28:0/2592127793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc7443ae0 0x561bc747a8e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3768099be0 tx=0x7f377000f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71c3fd3aad3ddf59 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.871+0000 7f375e7fc640 1 -- 172.21.15.28:0/2592127793 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3770027070 con 0x561bc7443ae0 2024-04-21T08:53:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.871+0000 7f375e7fc640 1 -- 172.21.15.28:0/2592127793 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37700222d0 con 0x561bc7443ae0 2024-04-21T08:53:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.871+0000 7f375e7fc640 1 -- 172.21.15.28:0/2592127793 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377000d880 con 0x561bc7443ae0 2024-04-21T08:53:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.873+0000 7f37817ed740 1 -- 172.21.15.28:0/2592127793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc7443ae0 msgr2=0x561bc747a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.873+0000 7f37817ed740 1 --2- 172.21.15.28:0/2592127793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc7443ae0 0x561bc747a8e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3768099be0 tx=0x7f377000f000 comp rx=0 tx=0).stop 2024-04-21T08:53:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.873+0000 7f37817ed740 1 -- 172.21.15.28:0/2592127793 shutdown_connections 2024-04-21T08:53:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.873+0000 7f37817ed740 1 --2- 172.21.15.28:0/2592127793 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc744d1d0 0x561bc7445780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.873+0000 7f37817ed740 1 --2- 172.21.15.28:0/2592127793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc74491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.873+0000 7f37817ed740 1 --2- 172.21.15.28:0/2592127793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc7443ae0 0x561bc747a8e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.873+0000 7f37817ed740 1 -- 172.21.15.28:0/2592127793 >> 172.21.15.28:0/2592127793 conn(0x561bc743e400 msgr2=0x561bc743d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.873+0000 7f37817ed740 1 -- 172.21.15.28:0/2592127793 shutdown_connections 2024-04-21T08:53:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.874+0000 7f37817ed740 1 -- 172.21.15.28:0/2592127793 wait complete. 2024-04-21T08:53:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.874+0000 7f37817ed740 1 -- start start 2024-04-21T08:53:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.874+0000 7f37817ed740 1 Processor -- start 2024-04-21T08:53:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.875+0000 7f37817ed740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc7443ae0 0x561bc74729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.875+0000 7f37817ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc7472f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.875+0000 7f37817ed740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc744d1d0 0x561bc7473450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.875+0000 7f37817ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f376400ed60 con 0x561bc7444400 2024-04-21T08:53:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.875+0000 7f37817ed740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f376400e690 con 0x561bc7443ae0 2024-04-21T08:53:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.875+0000 7f37817ed740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f376400f430 con 0x561bc744d1d0 2024-04-21T08:53:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.875+0000 7f37817ed740 10 client.0 _ll_register_callbacks cb 0x7f3768097bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.875+0000 7f377d191640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc744d1d0 0x561bc7473450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.875+0000 7f377d191640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc744d1d0 0x561bc7473450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:37940/0 (socket says 172.21.15.28:37940) 2024-04-21T08:53:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.875+0000 7f377d191640 1 -- 172.21.15.28:0/1335715265 learned_addr learned my addr 172.21.15.28:0/1335715265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.875+0000 7f377c990640 1 --2- 172.21.15.28:0/1335715265 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc7443ae0 0x561bc74729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.876+0000 7f377d191640 1 -- 172.21.15.28:0/1335715265 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc7443ae0 msgr2=0x561bc74729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.876+0000 7f377d191640 1 --2- 172.21.15.28:0/1335715265 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc7443ae0 0x561bc74729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.876+0000 7f377d191640 1 -- 172.21.15.28:0/1335715265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 msgr2=0x561bc7472f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.876+0000 7f377d191640 1 --2- 172.21.15.28:0/1335715265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc7472f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.876+0000 7f377d191640 1 -- 172.21.15.28:0/1335715265 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bc74e4430 con 0x561bc744d1d0 2024-04-21T08:53:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.876+0000 7f377d191640 1 --2- 172.21.15.28:0/1335715265 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc744d1d0 0x561bc7473450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f376c02c270 tx=0x7f376c02bc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44a2ed8cb6a8dd1d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.877+0000 7f3757fff640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:53:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.877+0000 7f3757fff640 1 -- 172.21.15.28:0/1335715265 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376c052060 con 0x561bc744d1d0 2024-04-21T08:53:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.878+0000 7f3757fff640 1 -- 172.21.15.28:0/1335715265 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f376c0037f0 con 0x561bc744d1d0 2024-04-21T08:53:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.878+0000 7f37817ed740 1 -- 172.21.15.28:0/1335715265 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37700222d0 con 0x561bc744d1d0 2024-04-21T08:53:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.879+0000 7f3757fff640 1 -- 172.21.15.28:0/1335715265 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f376c044070 con 0x561bc744d1d0 2024-04-21T08:53:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.879+0000 7f37817ed740 10 client.25816 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.879+0000 7f37817ed740 1 -- 172.21.15.28:0/1335715265 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561bc74e56d0 con 0x561bc744d1d0 2024-04-21T08:53:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.881+0000 7f3757fff640 1 -- 172.21.15.28:0/1335715265 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37700222d0 con 0x561bc744d1d0 2024-04-21T08:53:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.881+0000 7f3757fff640 1 -- 172.21.15.28:0/1335715265 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f376c040860 con 0x561bc744d1d0 2024-04-21T08:53:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.881+0000 7f37817ed740 10 client.25816 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.881+0000 7f37817ed740 10 client.25816 resolve_mds: validated gid 24488 aka daemon mds.b 2024-04-21T08:53:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.881+0000 7f37817ed740 20 client.25816 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.882+0000 7f37817ed740 1 --2- 172.21.15.28:0/1335715265 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561bc74e5960 0x561bc74e7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.882+0000 7f37817ed740 4 client.25816 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 30.0, "expiration": 84.5, "await": true}] 2024-04-21T08:53:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.882+0000 7f37817ed740 1 -- 172.21.15.28:0/1335715265 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 30.0, "expiration": 84.5, "await": true}) -- 0x561bc73af980 con 0x561bc74e5960 2024-04-21T08:53:29.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.883+0000 7f377c990640 1 --2- 172.21.15.28:0/1335715265 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561bc74e5960 0x561bc74e7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.888+0000 7f377c990640 1 --2- 172.21.15.28:0/1335715265 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561bc74e5960 0x561bc74e7d50 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59d7e274f14ab793 server_cookie=ad6088c903e92422 in_seq=0 out_seq=0 2024-04-21T08:53:29.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:29.888+0000 7f3757fff640 10 client.25816 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:30.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:30 smithi152 ceph-mon[31401]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 130 MiB/s wr, 16.34k op/s 2024-04-21T08:53:30.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:30 smithi028 ceph-mon[28003]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 130 MiB/s wr, 16.34k op/s 2024-04-21T08:53:30.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:30 smithi171 ceph-mon[31358]: pgmap v72: 129 pgs: 129 active+clean; 13 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 130 MiB/s wr, 16.34k op/s 2024-04-21T08:53:32.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:32 smithi152 ceph-mon[31401]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 79 MiB/s wr, 10.78k op/s 2024-04-21T08:53:32.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:32 smithi028 ceph-mon[28003]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 79 MiB/s wr, 10.78k op/s 2024-04-21T08:53:32.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:32 smithi171 ceph-mon[31358]: pgmap v73: 129 pgs: 129 active+clean; 13 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 79 MiB/s wr, 10.78k op/s 2024-04-21T08:53:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.898+0000 7f92d0f50640 1 -- 172.21.15.28:0/899872379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92c40aa2a0 msgr2=0x7f92c40b1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.898+0000 7f92d0f50640 1 --2- 172.21.15.28:0/899872379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92c40aa2a0 0x7f92c40b1e70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f92b80074e0 tx=0x7f92b802c8d0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.898+0000 7f92d0f50640 1 -- 172.21.15.28:0/899872379 shutdown_connections 2024-04-21T08:53:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.898+0000 7f92d0f50640 1 --2- 172.21.15.28:0/899872379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92c40aa2a0 0x7f92c40b1e70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.898+0000 7f92d0f50640 1 --2- 172.21.15.28:0/899872379 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92c40a5b80 0x7f92c40a9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.898+0000 7f92d0f50640 1 --2- 172.21.15.28:0/899872379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c40a51d0 0x7f92c40a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.898+0000 7f92d0f50640 1 -- 172.21.15.28:0/899872379 >> 172.21.15.28:0/899872379 conn(0x7f92c401aa70 msgr2=0x7f92c401ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.904+0000 7f92d0f50640 1 -- 172.21.15.28:0/899872379 shutdown_connections 2024-04-21T08:53:33.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.906+0000 7f92d0f50640 1 -- 172.21.15.28:0/899872379 wait complete. 2024-04-21T08:53:33.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.907+0000 7f92d0f50640 1 Processor -- start 2024-04-21T08:53:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.908+0000 7f92d0f50640 1 -- start start 2024-04-21T08:53:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.909+0000 7f92d0f50640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92c40a51d0 0x7f92c41498b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.909+0000 7f92d0f50640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92c40aa2a0 0x7f92c4149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.909+0000 7f92d0f50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c4143a60 0x7f92c4143f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.909+0000 7f92d0f50640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92c40b4e70 con 0x7f92c4143a60 2024-04-21T08:53:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.909+0000 7f92d0f50640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f92c40b4cf0 con 0x7f92c40aa2a0 2024-04-21T08:53:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.909+0000 7f92d0f50640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f92c40b4ff0 con 0x7f92c40a51d0 2024-04-21T08:53:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.910+0000 7f92caffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92c40aa2a0 0x7f92c4149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.910+0000 7f92cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c4143a60 0x7f92c4143f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.910+0000 7f92caffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92c40aa2a0 0x7f92c4149df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:33344/0 (socket says 172.21.15.28:33344) 2024-04-21T08:53:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.910+0000 7f92caffd640 1 -- 172.21.15.28:0/1196250425 learned_addr learned my addr 172.21.15.28:0/1196250425 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.911+0000 7f92caffd640 1 -- 172.21.15.28:0/1196250425 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92c40a51d0 msgr2=0x7f92c41498b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:53:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.911+0000 7f92caffd640 1 --2- 172.21.15.28:0/1196250425 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92c40a51d0 0x7f92c41498b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.911+0000 7f92caffd640 1 -- 172.21.15.28:0/1196250425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c4143a60 msgr2=0x7f92c4143f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.911+0000 7f92caffd640 1 --2- 172.21.15.28:0/1196250425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c4143a60 0x7f92c4143f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.911+0000 7f92caffd640 1 -- 172.21.15.28:0/1196250425 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c4144580 con 0x7f92c40aa2a0 2024-04-21T08:53:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.914+0000 7f92caffd640 1 --2- 172.21.15.28:0/1196250425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92c40aa2a0 0x7f92c4149df0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f92cc067c00 tx=0x7f92cc070890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:33.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.917+0000 7f92c8ff9640 1 -- 172.21.15.28:0/1196250425 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc075590 con 0x7f92c40aa2a0 2024-04-21T08:53:33.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.917+0000 7f92d0f50640 1 -- 172.21.15.28:0/1196250425 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92c4144870 con 0x7f92c40aa2a0 2024-04-21T08:53:33.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.917+0000 7f92d0f50640 1 -- 172.21.15.28:0/1196250425 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92c4150a70 con 0x7f92c40aa2a0 2024-04-21T08:53:33.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.917+0000 7f92c8ff9640 1 -- 172.21.15.28:0/1196250425 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92cc075be0 con 0x7f92c40aa2a0 2024-04-21T08:53:33.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.917+0000 7f92c8ff9640 1 -- 172.21.15.28:0/1196250425 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc076ca0 con 0x7f92c40aa2a0 2024-04-21T08:53:33.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.927+0000 7f92c8ff9640 1 -- 172.21.15.28:0/1196250425 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f92cc07d030 con 0x7f92c40aa2a0 2024-04-21T08:53:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.930+0000 7f92d4da0740 1 Processor -- start 2024-04-21T08:53:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.930+0000 7f92c8ff9640 1 --2- 172.21.15.28:0/1196250425 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f92bc0a3ef0 0x7f92bc0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.930+0000 7f92c8ff9640 1 -- 172.21.15.28:0/1196250425 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f92cc133500 con 0x7f92c40aa2a0 2024-04-21T08:53:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.932+0000 7f92d4da0740 1 -- start start 2024-04-21T08:53:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.932+0000 7f92d4da0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be5fac0 0x56219be5e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:33.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.932+0000 7f92d4da0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 0x56219be5f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:33.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.932+0000 7f92d4da0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be682d0 0x56219be6a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:33.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.932+0000 7f92d4da0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56219bdc38d0 con 0x56219be5ed50 2024-04-21T08:53:33.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.932+0000 7f92d4da0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56219bd60e80 con 0x56219be5fac0 2024-04-21T08:53:33.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.932+0000 7f92d4da0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56219bdbf890 con 0x56219be682d0 2024-04-21T08:53:33.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.932+0000 7f92cbfff640 1 --2- 172.21.15.28:0/1196250425 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f92bc0a3ef0 0x7f92bc0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.933+0000 7f92cbfff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be682d0 0x56219be6a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.933+0000 7f92cbfff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be682d0 0x56219be6a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:50256/0 (socket says 172.21.15.28:50256) 2024-04-21T08:53:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.933+0000 7f92cbfff640 1 -- 172.21.15.28:0/741830048 learned_addr learned my addr 172.21.15.28:0/741830048 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.933+0000 7f92caffd640 1 --2- 172.21.15.28:0/741830048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 0x56219be5f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.934+0000 7f92cbfff640 1 -- 172.21.15.28:0/741830048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be5fac0 msgr2=0x56219be5e810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.934+0000 7f92cbfff640 1 --2- 172.21.15.28:0/741830048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be5fac0 0x56219be5e810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.934+0000 7f92cbfff640 1 -- 172.21.15.28:0/741830048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 msgr2=0x56219be5f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.934+0000 7f92cbfff640 1 --2- 172.21.15.28:0/741830048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 0x56219be5f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.934+0000 7f92cbfff640 1 -- 172.21.15.28:0/741830048 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56219bd82320 con 0x56219be682d0 2024-04-21T08:53:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.934+0000 7f92cbfff640 1 --2- 172.21.15.28:0/1196250425 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f92bc0a3ef0 0x7f92bc0a63b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f92b8034000 tx=0x7f92b8007bf0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:33.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.935+0000 7f92cbfff640 1 --2- 172.21.15.28:0/741830048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be682d0 0x56219be6a6c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f92b8007c60 tx=0x7f92b8047000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7cc43a1c7aa8e9d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.936+0000 7f92b5ffb640 1 -- 172.21.15.28:0/741830048 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b8036040 con 0x56219be682d0 2024-04-21T08:53:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.936+0000 7f92b5ffb640 1 -- 172.21.15.28:0/741830048 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92b804c070 con 0x56219be682d0 2024-04-21T08:53:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.936+0000 7f92b5ffb640 1 -- 172.21.15.28:0/741830048 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b80082a0 con 0x56219be682d0 2024-04-21T08:53:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.938+0000 7f92d4da0740 1 -- 172.21.15.28:0/741830048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be682d0 msgr2=0x56219be6a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.938+0000 7f92d4da0740 1 --2- 172.21.15.28:0/741830048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be682d0 0x56219be6a6c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f92b8007c60 tx=0x7f92b8047000 comp rx=0 tx=0).stop 2024-04-21T08:53:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.941+0000 7f92d4da0740 1 -- 172.21.15.28:0/741830048 shutdown_connections 2024-04-21T08:53:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.941+0000 7f92d4da0740 1 --2- 172.21.15.28:0/741830048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be682d0 0x56219be6a6c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.941+0000 7f92d4da0740 1 --2- 172.21.15.28:0/741830048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 0x56219be5f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.941+0000 7f92d4da0740 1 --2- 172.21.15.28:0/741830048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be5fac0 0x56219be5e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.941+0000 7f92d4da0740 1 -- 172.21.15.28:0/741830048 >> 172.21.15.28:0/741830048 conn(0x56219be25c90 msgr2=0x56219be59a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:33.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.944+0000 7f92d4da0740 1 -- 172.21.15.28:0/741830048 shutdown_connections 2024-04-21T08:53:33.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.945+0000 7f92d4da0740 1 -- 172.21.15.28:0/741830048 wait complete. 2024-04-21T08:53:33.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.945+0000 7f92d4da0740 1 -- start start 2024-04-21T08:53:33.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.945+0000 7f92d4da0740 1 Processor -- start 2024-04-21T08:53:33.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.946+0000 7f92d4da0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 0x56219be67c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:33.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.946+0000 7f92d4da0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be5fac0 0x56219be64ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:33.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.946+0000 7f92d4da0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be682d0 0x56219be65570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:33.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.946+0000 7f92d4da0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56219be6afa0 con 0x56219be5ed50 2024-04-21T08:53:33.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.946+0000 7f92d4da0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56219bd4beb0 con 0x56219be5fac0 2024-04-21T08:53:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.946+0000 7f92d4da0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56219be6b120 con 0x56219be682d0 2024-04-21T08:53:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.946+0000 7f92d4da0740 10 client.0 _ll_register_callbacks cb 0x56219bd78b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.947+0000 7f92cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 0x56219be67c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.947+0000 7f92cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 0x56219be67c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45604/0 (socket says 172.21.15.28:45604) 2024-04-21T08:53:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.947+0000 7f92cb7fe640 1 -- 172.21.15.28:0/2408055930 learned_addr learned my addr 172.21.15.28:0/2408055930 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.947+0000 7f92cb7fe640 1 -- 172.21.15.28:0/2408055930 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be682d0 msgr2=0x56219be65570 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.947+0000 7f92cb7fe640 1 --2- 172.21.15.28:0/2408055930 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be682d0 0x56219be65570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.947+0000 7f92cb7fe640 1 -- 172.21.15.28:0/2408055930 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be5fac0 msgr2=0x56219be64ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.947+0000 7f92cb7fe640 1 --2- 172.21.15.28:0/2408055930 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be5fac0 0x56219be64ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.947+0000 7f92cb7fe640 1 -- 172.21.15.28:0/2408055930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56219bad6210 con 0x56219be5ed50 2024-04-21T08:53:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.948+0000 7f92cb7fe640 1 --2- 172.21.15.28:0/2408055930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 0x56219be67c60 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f92c0009e00 tx=0x7f92c000b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed0d21233dc44268 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.948+0000 7f92aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.948+0000 7f92aaffd640 1 -- 172.21.15.28:0/2408055930 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c0015020 con 0x56219be5ed50 2024-04-21T08:53:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.948+0000 7f92d4da0740 1 -- 172.21.15.28:0/2408055930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92b804c070 con 0x56219be5ed50 2024-04-21T08:53:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.948+0000 7f92aaffd640 1 -- 172.21.15.28:0/2408055930 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f92c0011620 con 0x56219be5ed50 2024-04-21T08:53:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.949+0000 7f92aaffd640 1 -- 172.21.15.28:0/2408055930 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f92c0020050 con 0x56219be5ed50 2024-04-21T08:53:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.950+0000 7f92d4da0740 10 client.16467 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.950+0000 7f92d4da0740 1 -- 172.21.15.28:0/2408055930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56219bba2620 con 0x56219be5ed50 2024-04-21T08:53:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.950+0000 7f92aaffd640 1 -- 172.21.15.28:0/2408055930 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92b804c070 con 0x56219be5ed50 2024-04-21T08:53:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.952+0000 7f92aaffd640 1 -- 172.21.15.28:0/2408055930 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f92c0019880 con 0x56219be5ed50 2024-04-21T08:53:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.952+0000 7f92d4da0740 10 client.16467 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.952+0000 7f92d4da0740 10 client.16467 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.952+0000 7f92d4da0740 20 client.16467 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.952+0000 7f92d4da0740 1 --2- 172.21.15.28:0/2408055930 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56219be89850 0x56219be8bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.952+0000 7f92d4da0740 4 client.16467 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.952+0000 7f92d4da0740 1 -- 172.21.15.28:0/2408055930 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56219bdd7880 con 0x56219be89850 2024-04-21T08:53:33.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.956+0000 7f92caffd640 1 --2- 172.21.15.28:0/2408055930 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56219be89850 0x56219be8bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:33.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.960+0000 7f92caffd640 1 --2- 172.21.15.28:0/2408055930 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56219be89850 0x56219be8bd10 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cee4326c93c77d92 server_cookie=cd4e0c61c8e8dbc3 in_seq=0 out_seq=0 2024-04-21T08:53:33.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.960+0000 7f92aaffd640 10 client.16467 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:33.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.965+0000 7f92aaffd640 1 -- 172.21.15.28:0/2408055930 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56219bdd7880 con 0x56219be89850 2024-04-21T08:53:33.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.965+0000 7f92aaffd640 10 client.16467 handle_command_reply: tid=0 2024-04-21T08:53:33.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.966+0000 7f92d4da0740 1 client.16467 shutdown 2024-04-21T08:53:33.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.967+0000 7f92d4da0740 1 -- 172.21.15.28:0/2408055930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 msgr2=0x56219be67c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:33.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.967+0000 7f92d4da0740 1 --2- 172.21.15.28:0/2408055930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 0x56219be67c60 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f92c0009e00 tx=0x7f92c000b4b0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.968+0000 7f92d4da0740 1 -- 172.21.15.28:0/2408055930 shutdown_connections 2024-04-21T08:53:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.968+0000 7f92d4da0740 1 --2- 172.21.15.28:0/2408055930 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56219be89850 0x56219be8bd10 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.968+0000 7f92d4da0740 1 --2- 172.21.15.28:0/2408055930 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be682d0 0x56219be65570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.968+0000 7f92d4da0740 1 --2- 172.21.15.28:0/2408055930 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be5fac0 0x56219be64ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.968+0000 7f92d4da0740 1 --2- 172.21.15.28:0/2408055930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 0x56219be67c60 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.968+0000 7f92d4da0740 1 -- 172.21.15.28:0/2408055930 >> 172.21.15.28:0/2408055930 conn(0x56219be25c90 msgr2=0x56219be5b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.968+0000 7f92aaffd640 0 client.16467 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:33.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.970+0000 7f92d4da0740 1 -- 172.21.15.28:0/2408055930 shutdown_connections 2024-04-21T08:53:33.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.973+0000 7f92d4da0740 1 -- 172.21.15.28:0/2408055930 wait complete. 2024-04-21T08:53:33.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:33.974+0000 7f92d4da0740 20 client.16467 trim_cache size 0 max 16384 2024-04-21T08:53:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.008+0000 7f92d4da0740 1 Processor -- start 2024-04-21T08:53:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.008+0000 7f92d4da0740 1 -- start start 2024-04-21T08:53:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.008+0000 7f92d4da0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 0x56219be73f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.009+0000 7f92d4da0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be5fac0 0x56219be67720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.009+0000 7f92d4da0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be682d0 0x56219be67c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.009+0000 7f92d4da0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92b0012880 con 0x56219be5ed50 2024-04-21T08:53:34.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.009+0000 7f92caffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be5fac0 0x56219be67720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.009+0000 7f92caffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be5fac0 0x56219be67720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:33366/0 (socket says 172.21.15.28:33366) 2024-04-21T08:53:34.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.009+0000 7f92caffd640 1 -- 172.21.15.28:0/634059927 learned_addr learned my addr 172.21.15.28:0/634059927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:34.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.009+0000 7f92d4da0740 1 -- 172.21.15.28:0/634059927 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f92b0010150 con 0x56219be5fac0 2024-04-21T08:53:34.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.009+0000 7f92d4da0740 1 -- 172.21.15.28:0/634059927 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f92b000dfa0 con 0x56219be682d0 2024-04-21T08:53:34.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.012+0000 7f92cb7fe640 1 --2- 172.21.15.28:0/634059927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 0x56219be73f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.012+0000 7f92caffd640 1 -- 172.21.15.28:0/634059927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be682d0 msgr2=0x56219be67c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:34.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.012+0000 7f92caffd640 1 --2- 172.21.15.28:0/634059927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be682d0 0x56219be67c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.012+0000 7f92caffd640 1 -- 172.21.15.28:0/634059927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 msgr2=0x56219be73f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.012+0000 7f92caffd640 1 --2- 172.21.15.28:0/634059927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 0x56219be73f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.012+0000 7f92caffd640 1 -- 172.21.15.28:0/634059927 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56219be658e0 con 0x56219be5fac0 2024-04-21T08:53:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.012+0000 7f92cb7fe640 1 --2- 172.21.15.28:0/634059927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 0x56219be73f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:53:34.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.012+0000 7f92caffd640 1 --2- 172.21.15.28:0/634059927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be5fac0 0x56219be67720 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f92cc02db50 tx=0x7f92cc132040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb320effba2929cd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.018+0000 7f92b57fa640 1 -- 172.21.15.28:0/634059927 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc107e20 con 0x56219be5fac0 2024-04-21T08:53:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.018+0000 7f92b57fa640 1 -- 172.21.15.28:0/634059927 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92cc0fd540 con 0x56219be5fac0 2024-04-21T08:53:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.018+0000 7f92b57fa640 1 -- 172.21.15.28:0/634059927 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc130ad0 con 0x56219be5fac0 2024-04-21T08:53:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.019+0000 7f92d4da0740 1 -- 172.21.15.28:0/634059927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be5fac0 msgr2=0x56219be67720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.019+0000 7f92d4da0740 1 --2- 172.21.15.28:0/634059927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be5fac0 0x56219be67720 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f92cc02db50 tx=0x7f92cc132040 comp rx=0 tx=0).stop 2024-04-21T08:53:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.021+0000 7f92d4da0740 1 -- 172.21.15.28:0/634059927 shutdown_connections 2024-04-21T08:53:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.021+0000 7f92d4da0740 1 --2- 172.21.15.28:0/634059927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be682d0 0x56219be67c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.021+0000 7f92d4da0740 1 --2- 172.21.15.28:0/634059927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be5fac0 0x56219be67720 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f92cc02db50 tx=0x7f92cc132040 comp rx=0 tx=0).stop 2024-04-21T08:53:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.021+0000 7f92d4da0740 1 --2- 172.21.15.28:0/634059927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be5ed50 0x56219be73f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.021+0000 7f92d4da0740 1 -- 172.21.15.28:0/634059927 >> 172.21.15.28:0/634059927 conn(0x56219be25c90 msgr2=0x56219be638c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:34.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.024+0000 7f92d4da0740 1 -- 172.21.15.28:0/634059927 shutdown_connections 2024-04-21T08:53:34.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.024+0000 7f92d4da0740 1 -- 172.21.15.28:0/634059927 wait complete. 2024-04-21T08:53:34.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.024+0000 7f92d4da0740 1 -- start start 2024-04-21T08:53:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.025+0000 7f92d4da0740 1 Processor -- start 2024-04-21T08:53:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.025+0000 7f92d4da0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be5ed50 0x56219be70440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.025+0000 7f92d4da0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be682d0 0x56219be70980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.025+0000 7f92d4da0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be6ba40 0x56219be6bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.025+0000 7f92d4da0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92b000e610 con 0x56219be6ba40 2024-04-21T08:53:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.025+0000 7f92d4da0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f92b000fa80 con 0x56219be682d0 2024-04-21T08:53:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.025+0000 7f92d4da0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f92b000ece0 con 0x56219be5ed50 2024-04-21T08:53:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.025+0000 7f92d4da0740 10 client.0 _ll_register_callbacks cb 0x7f92c0006ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.026+0000 7f92cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be6ba40 0x56219be6bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.026+0000 7f92cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be6ba40 0x56219be6bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45620/0 (socket says 172.21.15.28:45620) 2024-04-21T08:53:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.026+0000 7f92cbfff640 1 -- 172.21.15.28:0/1880639534 learned_addr learned my addr 172.21.15.28:0/1880639534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.026+0000 7f92cb7fe640 1 --2- 172.21.15.28:0/1880639534 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be5ed50 0x56219be70440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.026+0000 7f92cbfff640 1 -- 172.21.15.28:0/1880639534 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be5ed50 msgr2=0x56219be70440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:34.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.026+0000 7f92cbfff640 1 --2- 172.21.15.28:0/1880639534 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be5ed50 0x56219be70440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.026+0000 7f92cbfff640 1 -- 172.21.15.28:0/1880639534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be682d0 msgr2=0x56219be70980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:34.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.026+0000 7f92cbfff640 1 --2- 172.21.15.28:0/1880639534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be682d0 0x56219be70980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.026+0000 7f92cbfff640 1 -- 172.21.15.28:0/1880639534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56219be6c6f0 con 0x56219be6ba40 2024-04-21T08:53:34.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.027+0000 7f92cbfff640 1 --2- 172.21.15.28:0/1880639534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be6ba40 0x56219be6bec0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f92b8007320 tx=0x7f92b802cfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d704f1c5b04b6a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:34.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.029+0000 7f92aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:34.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.029+0000 7f92aaffd640 1 -- 172.21.15.28:0/1880639534 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b8036030 con 0x56219be6ba40 2024-04-21T08:53:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.030+0000 7f92d4da0740 1 -- 172.21.15.28:0/1880639534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92cc0fd540 con 0x56219be6ba40 2024-04-21T08:53:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.030+0000 7f92aaffd640 1 -- 172.21.15.28:0/1880639534 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f92b800a4e0 con 0x56219be6ba40 2024-04-21T08:53:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.031+0000 7f92aaffd640 1 -- 172.21.15.28:0/1880639534 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f92b8061050 con 0x56219be6ba40 2024-04-21T08:53:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.031+0000 7f92d4da0740 10 client.16470 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.031+0000 7f92d4da0740 1 -- 172.21.15.28:0/1880639534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56219beefe40 con 0x56219be6ba40 2024-04-21T08:53:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.034+0000 7f92aaffd640 1 -- 172.21.15.28:0/1880639534 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92cc0fd540 con 0x56219be6ba40 2024-04-21T08:53:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.034+0000 7f92aaffd640 1 -- 172.21.15.28:0/1880639534 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f92b803b040 con 0x56219be6ba40 2024-04-21T08:53:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.034+0000 7f92d4da0740 10 client.16470 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.034+0000 7f92d4da0740 10 client.16470 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.034+0000 7f92d4da0740 20 client.16470 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.034+0000 7f92d4da0740 1 --2- 172.21.15.28:0/1880639534 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56219bef0460 0x56219bef2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.034+0000 7f92d4da0740 4 client.16470 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:53:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.034+0000 7f92d4da0740 1 -- 172.21.15.28:0/1880639534 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56219bd633d0 con 0x56219bef0460 2024-04-21T08:53:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.036+0000 7f92cb7fe640 1 --2- 172.21.15.28:0/1880639534 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56219bef0460 0x56219bef2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.037+0000 7f92cb7fe640 1 --2- 172.21.15.28:0/1880639534 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56219bef0460 0x56219bef2880 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1979d314343add36 server_cookie=5fe40500667a1c9a in_seq=0 out_seq=0 2024-04-21T08:53:34.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:34.038+0000 7f92aaffd640 10 client.16470 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:34.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:34 smithi152 ceph-mon[31401]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 MiB/s wr, 13.91k op/s 2024-04-21T08:53:34.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:34 smithi028 ceph-mon[28003]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 MiB/s wr, 13.91k op/s 2024-04-21T08:53:34.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:34 smithi171 ceph-mon[31358]: pgmap v74: 129 pgs: 129 active+clean; 13 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 MiB/s wr, 13.91k op/s 2024-04-21T08:53:36.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:36 smithi152 ceph-mon[31401]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 62 MiB/s wr, 10.03k op/s 2024-04-21T08:53:36.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:36 smithi028 ceph-mon[28003]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 62 MiB/s wr, 10.03k op/s 2024-04-21T08:53:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:36 smithi171 ceph-mon[31358]: pgmap v75: 129 pgs: 129 active+clean; 13 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 62 MiB/s wr, 10.03k op/s 2024-04-21T08:53:36.633 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:53:36.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:53:38.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.008+0000 7fb1c4d2b640 1 -- 172.21.15.28:0/1513864900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1c010e7b0 msgr2=0x7fb1c010ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:38.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.008+0000 7fb1c4d2b640 1 --2- 172.21.15.28:0/1513864900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1c010e7b0 0x7fb1c010ec10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009930 tx=0x7fb1b402bc50 comp rx=0 tx=0).stop 2024-04-21T08:53:38.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.010+0000 7fb1c4d2b640 1 -- 172.21.15.28:0/1513864900 shutdown_connections 2024-04-21T08:53:38.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.010+0000 7fb1c4d2b640 1 --2- 172.21.15.28:0/1513864900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb1c011b730 0x7fb1c011db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.010+0000 7fb1c4d2b640 1 --2- 172.21.15.28:0/1513864900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1c010e7b0 0x7fb1c010ec10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.010+0000 7fb1c4d2b640 1 --2- 172.21.15.28:0/1513864900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c01106d0 0x7fb1c010e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.010+0000 7fb1c4d2b640 1 -- 172.21.15.28:0/1513864900 >> 172.21.15.28:0/1513864900 conn(0x7fb1c006d070 msgr2=0x7fb1c006d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:38.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.014+0000 7fb1c4d2b640 1 -- 172.21.15.28:0/1513864900 shutdown_connections 2024-04-21T08:53:38.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.015+0000 7fb1c4d2b640 1 -- 172.21.15.28:0/1513864900 wait complete. 2024-04-21T08:53:38.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.016+0000 7fb1c4d2b640 1 Processor -- start 2024-04-21T08:53:38.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.016+0000 7fb1c4d2b640 1 -- start start 2024-04-21T08:53:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.017+0000 7fb1c4d2b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb1c010e7b0 0x7fb1c01a75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.017+0000 7fb1c4d2b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1c01106d0 0x7fb1c01a7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.017+0000 7fb1c4d2b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c011b730 0x7fb1c01abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.017+0000 7fb1be59c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb1c010e7b0 0x7fb1c01a75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.017+0000 7fb1c4d2b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c0121060 con 0x7fb1c011b730 2024-04-21T08:53:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.017+0000 7fb1c4d2b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c0120ee0 con 0x7fb1c01106d0 2024-04-21T08:53:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.017+0000 7fb1c4d2b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c01211e0 con 0x7fb1c010e7b0 2024-04-21T08:53:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.017+0000 7fb1be59c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb1c010e7b0 0x7fb1c01a75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:50268/0 (socket says 172.21.15.28:50268) 2024-04-21T08:53:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.017+0000 7fb1be59c640 1 -- 172.21.15.28:0/2016087122 learned_addr learned my addr 172.21.15.28:0/2016087122 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:38.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.018+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/2016087122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c011b730 0x7fb1c01abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:38.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.018+0000 7fb1bdd9b640 1 --2- 172.21.15.28:0/2016087122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1c01106d0 0x7fb1c01a7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:38.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.019+0000 7fb1bed9d640 1 -- 172.21.15.28:0/2016087122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb1c010e7b0 msgr2=0x7fb1c01a75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:38.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.019+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/2016087122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb1c010e7b0 0x7fb1c01a75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.019+0000 7fb1bed9d640 1 -- 172.21.15.28:0/2016087122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1c01106d0 msgr2=0x7fb1c01a7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.019+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/2016087122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1c01106d0 0x7fb1c01a7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.019+0000 7fb1bed9d640 1 -- 172.21.15.28:0/2016087122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c01ac590 con 0x7fb1c011b730 2024-04-21T08:53:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.019+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/2016087122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c011b730 0x7fb1c01abe60 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800bdd0 tx=0x7fb1b8009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.019+0000 7fb1af7fe640 1 -- 172.21.15.28:0/2016087122 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8019070 con 0x7fb1c011b730 2024-04-21T08:53:38.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.020+0000 7fb1c4d2b640 1 -- 172.21.15.28:0/2016087122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c01ac880 con 0x7fb1c011b730 2024-04-21T08:53:38.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.020+0000 7fb1c4d2b640 1 -- 172.21.15.28:0/2016087122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c01b4110 con 0x7fb1c011b730 2024-04-21T08:53:38.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.020+0000 7fb1af7fe640 1 -- 172.21.15.28:0/2016087122 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1b800cdf0 con 0x7fb1c011b730 2024-04-21T08:53:38.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.021+0000 7fb1af7fe640 1 -- 172.21.15.28:0/2016087122 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8023060 con 0x7fb1c011b730 2024-04-21T08:53:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.024+0000 7fb1af7fe640 1 -- 172.21.15.28:0/2016087122 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb1b801c070 con 0x7fb1c011b730 2024-04-21T08:53:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.025+0000 7fb1af7fe640 1 --2- 172.21.15.28:0/2016087122 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb1a80a3d60 0x7fb1a80a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.026+0000 7fb1be59c640 1 --2- 172.21.15.28:0/2016087122 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb1a80a3d60 0x7fb1a80a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.026+0000 7fb1c79fb740 1 Processor -- start 2024-04-21T08:53:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.026+0000 7fb1c79fb740 1 -- start start 2024-04-21T08:53:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.027+0000 7fb1c79fb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873df8580 0x55c873df8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.027+0000 7fb1c79fb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873df8ea0 0x55c873e01540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.027+0000 7fb1c79fb740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873e01a80 0x55c873e03e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.027+0000 7fb1c79fb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c873d5d1e0 con 0x55c873df8ea0 2024-04-21T08:53:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.027+0000 7fb1c79fb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c873cfa390 con 0x55c873df8580 2024-04-21T08:53:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.027+0000 7fb1c79fb740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55c873d591a0 con 0x55c873e01a80 2024-04-21T08:53:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.027+0000 7fb1bed9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873e01a80 0x55c873e03e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.027+0000 7fb1bed9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873e01a80 0x55c873e03e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:50272/0 (socket says 172.21.15.28:50272) 2024-04-21T08:53:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.027+0000 7fb1bed9d640 1 -- 172.21.15.28:0/3336247202 learned_addr learned my addr 172.21.15.28:0/3336247202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.028+0000 7fb1bed9d640 1 -- 172.21.15.28:0/3336247202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873df8580 msgr2=0x55c873df8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.028+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/3336247202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873df8580 0x55c873df8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.028+0000 7fb1bed9d640 1 -- 172.21.15.28:0/3336247202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873df8ea0 msgr2=0x55c873e01540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.028+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/3336247202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873df8ea0 0x55c873e01540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.028+0000 7fb1bed9d640 1 -- 172.21.15.28:0/3336247202 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c873b397d0 con 0x55c873e01a80 2024-04-21T08:53:38.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.028+0000 7fb1af7fe640 1 -- 172.21.15.28:0/2016087122 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fb1b809a290 con 0x7fb1c011b730 2024-04-21T08:53:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.031+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/3336247202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873e01a80 0x55c873e03e70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8098960 tx=0x7fb1b80a0350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14af3bad26311de0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.033+0000 7fb1acff9640 1 -- 172.21.15.28:0/3336247202 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b80cd970 con 0x55c873e01a80 2024-04-21T08:53:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.033+0000 7fb1acff9640 1 -- 172.21.15.28:0/3336247202 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1b80a4630 con 0x55c873e01a80 2024-04-21T08:53:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.033+0000 7fb1acff9640 1 -- 172.21.15.28:0/3336247202 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8096d60 con 0x55c873e01a80 2024-04-21T08:53:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.034+0000 7fb1c79fb740 1 -- 172.21.15.28:0/3336247202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873e01a80 msgr2=0x55c873e03e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.035+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/3336247202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873e01a80 0x55c873e03e70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8098960 tx=0x7fb1b80a0350 comp rx=0 tx=0).stop 2024-04-21T08:53:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.035+0000 7fb1c79fb740 1 -- 172.21.15.28:0/3336247202 shutdown_connections 2024-04-21T08:53:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.035+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/3336247202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873e01a80 0x55c873e03e70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.035+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/3336247202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873df8ea0 0x55c873e01540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.035+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/3336247202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873df8580 0x55c873df8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.035+0000 7fb1c79fb740 1 -- 172.21.15.28:0/3336247202 >> 172.21.15.28:0/3336247202 conn(0x55c873df2cb0 msgr2=0x55c873df3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.038+0000 7fb1be59c640 1 --2- 172.21.15.28:0/2016087122 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb1a80a3d60 0x7fb1a80a6220 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009e00 tx=0x7fb1b00098f0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.040+0000 7fb1c79fb740 1 -- 172.21.15.28:0/3336247202 shutdown_connections 2024-04-21T08:53:38.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.040+0000 7fb1c79fb740 1 -- 172.21.15.28:0/3336247202 wait complete. 2024-04-21T08:53:38.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.041+0000 7fb1c79fb740 1 -- start start 2024-04-21T08:53:38.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.041+0000 7fb1c79fb740 1 Processor -- start 2024-04-21T08:53:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.044+0000 7fb1c79fb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873df8580 0x55c873dfeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.044+0000 7fb1c79fb740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873df8ea0 0x55c873dff030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.044+0000 7fb1c79fb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873e01a80 0x55c873dff570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.044+0000 7fb1c79fb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c873e04870 con 0x55c873e01a80 2024-04-21T08:53:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.044+0000 7fb1c79fb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c873d162f0 con 0x55c873df8580 2024-04-21T08:53:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.044+0000 7fb1c79fb740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55c873e049f0 con 0x55c873df8ea0 2024-04-21T08:53:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.044+0000 7fb1c79fb740 10 client.0 _ll_register_callbacks cb 0x55c873d23580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.046+0000 7fb1be59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873df8580 0x55c873dfeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.046+0000 7fb1be59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873df8580 0x55c873dfeaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:33410/0 (socket says 172.21.15.28:33410) 2024-04-21T08:53:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.046+0000 7fb1be59c640 1 -- 172.21.15.28:0/2028610832 learned_addr learned my addr 172.21.15.28:0/2028610832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.046+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/2028610832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873e01a80 0x55c873dff570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.047+0000 7fb1bed9d640 1 -- 172.21.15.28:0/2028610832 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873df8ea0 msgr2=0x55c873dff030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.047+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/2028610832 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873df8ea0 0x55c873dff030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.047+0000 7fb1bed9d640 1 -- 172.21.15.28:0/2028610832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873df8580 msgr2=0x55c873dfeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.047+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/2028610832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873df8580 0x55c873dfeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.047+0000 7fb1bed9d640 1 -- 172.21.15.28:0/2028610832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c873a6f3f0 con 0x55c873e01a80 2024-04-21T08:53:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.048+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/2028610832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873e01a80 0x55c873dff570 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8098960 tx=0x7fb1b804fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4650cf292d5f2b93 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.048+0000 7fb1a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.049+0000 7fb1a5ffb640 1 -- 172.21.15.28:0/2028610832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b80b4070 con 0x55c873e01a80 2024-04-21T08:53:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.049+0000 7fb1c79fb740 1 -- 172.21.15.28:0/2028610832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1b80a4630 con 0x55c873e01a80 2024-04-21T08:53:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.049+0000 7fb1a5ffb640 1 -- 172.21.15.28:0/2028610832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fb1b80b5260 con 0x55c873e01a80 2024-04-21T08:53:38.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.049+0000 7fb1a5ffb640 1 -- 172.21.15.28:0/2028610832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb1b8096b00 con 0x55c873e01a80 2024-04-21T08:53:38.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.049+0000 7fb1c79fb740 10 client.16479 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:38.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.049+0000 7fb1c79fb740 1 -- 172.21.15.28:0/2028610832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c873a2f0f0 con 0x55c873e01a80 2024-04-21T08:53:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.050+0000 7fb1a5ffb640 1 -- 172.21.15.28:0/2028610832 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1b80a4630 con 0x55c873e01a80 2024-04-21T08:53:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.050+0000 7fb1a5ffb640 1 -- 172.21.15.28:0/2028610832 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb1b809af60 con 0x55c873e01a80 2024-04-21T08:53:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.051+0000 7fb1c79fb740 10 client.16479 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.051+0000 7fb1c79fb740 10 client.16479 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.051+0000 7fb1c79fb740 20 client.16479 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.051+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/2028610832 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c873e11a50 0x55c873e13eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.051+0000 7fb1c79fb740 4 client.16479 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.051+0000 7fb1c79fb740 1 -- 172.21.15.28:0/2028610832 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c873d6ac00 con 0x55c873e11a50 2024-04-21T08:53:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.054+0000 7fb1be59c640 1 --2- 172.21.15.28:0/2028610832 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c873e11a50 0x55c873e13eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.062+0000 7fb1be59c640 1 --2- 172.21.15.28:0/2028610832 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c873e11a50 0x55c873e13eb0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6ace138c53c0e0c server_cookie=d174f49364f9947a in_seq=0 out_seq=0 2024-04-21T08:53:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:38.062+0000 7fb1a5ffb640 10 client.16479 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:38.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:38 smithi028 ceph-mon[28003]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 59 MiB/s wr, 9.52k op/s 2024-04-21T08:53:38.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:38 smithi152 ceph-mon[31401]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 59 MiB/s wr, 9.52k op/s 2024-04-21T08:53:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:38 smithi171 ceph-mon[31358]: pgmap v76: 129 pgs: 129 active+clean; 13 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 59 MiB/s wr, 9.52k op/s 2024-04-21T08:53:40.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:40 smithi152 ceph-mon[31401]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 74 MiB/s wr, 13.02k op/s 2024-04-21T08:53:40.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:40 smithi028 ceph-mon[28003]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 74 MiB/s wr, 13.02k op/s 2024-04-21T08:53:40.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:40 smithi171 ceph-mon[31358]: pgmap v77: 129 pgs: 129 active+clean; 13 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 74 MiB/s wr, 13.02k op/s 2024-04-21T08:53:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.689+0000 7f92aaffd640 1 -- 172.21.15.28:0/1880639534 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56219bd633d0 con 0x56219bef0460 2024-04-21T08:53:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.689+0000 7f92aaffd640 10 client.16470 handle_command_reply: tid=0 2024-04-21T08:53:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.689+0000 7f92d4da0740 1 client.16470 shutdown 2024-04-21T08:53:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.690+0000 7f92d4da0740 1 -- 172.21.15.28:0/1880639534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be6ba40 msgr2=0x56219be6bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.690+0000 7f92d4da0740 1 --2- 172.21.15.28:0/1880639534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be6ba40 0x56219be6bec0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f92b8007320 tx=0x7f92b802cfd0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.690+0000 7f92d4da0740 1 -- 172.21.15.28:0/1880639534 shutdown_connections 2024-04-21T08:53:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.690+0000 7f92d4da0740 1 --2- 172.21.15.28:0/1880639534 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56219bef0460 0x56219bef2880 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.690+0000 7f92d4da0740 1 --2- 172.21.15.28:0/1880639534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56219be6ba40 0x56219be6bec0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.690+0000 7f92d4da0740 1 --2- 172.21.15.28:0/1880639534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56219be682d0 0x56219be70980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.690+0000 7f92d4da0740 1 --2- 172.21.15.28:0/1880639534 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56219be5ed50 0x56219be70440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.690+0000 7f92d4da0740 1 -- 172.21.15.28:0/1880639534 >> 172.21.15.28:0/1880639534 conn(0x56219be25c90 msgr2=0x56219be58250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.690+0000 7f92aaffd640 0 client.16470 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:40.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.690+0000 7fb1a5ffb640 1 -- 172.21.15.28:0/2028610832 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c873d6ac00 con 0x55c873e11a50 2024-04-21T08:53:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.690+0000 7f92d4da0740 1 -- 172.21.15.28:0/1880639534 shutdown_connections 2024-04-21T08:53:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.690+0000 7f92d4da0740 1 -- 172.21.15.28:0/1880639534 wait complete. 2024-04-21T08:53:40.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.690+0000 7f92d4da0740 20 client.16470 trim_cache size 0 max 16384 2024-04-21T08:53:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.690+0000 7fb1a5ffb640 10 client.16479 handle_command_reply: tid=0 2024-04-21T08:53:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.690+0000 7fb1c79fb740 1 client.16479 shutdown 2024-04-21T08:53:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:53:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:53:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:53:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:53:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.691+0000 7fb1c79fb740 1 -- 172.21.15.28:0/2028610832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873e01a80 msgr2=0x55c873dff570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.691+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/2028610832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873e01a80 0x55c873dff570 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8098960 tx=0x7fb1b804fe20 comp rx=0 tx=0).stop 2024-04-21T08:53:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.691+0000 7fb1c79fb740 1 -- 172.21.15.28:0/2028610832 shutdown_connections 2024-04-21T08:53:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.691+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/2028610832 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c873e11a50 0x55c873e13eb0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.691+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/2028610832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873e01a80 0x55c873dff570 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.691+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/2028610832 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873df8ea0 0x55c873dff030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.691+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/2028610832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873df8580 0x55c873dfeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.691+0000 7fb1c79fb740 1 -- 172.21.15.28:0/2028610832 >> 172.21.15.28:0/2028610832 conn(0x55c873df2cb0 msgr2=0x55c873df4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.691+0000 7fb1a5ffb640 0 client.16479 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.691+0000 7fb1c79fb740 1 -- 172.21.15.28:0/2028610832 shutdown_connections 2024-04-21T08:53:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.691+0000 7fb1c79fb740 1 -- 172.21.15.28:0/2028610832 wait complete. 2024-04-21T08:53:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.691+0000 7f92d0f50640 1 -- 172.21.15.28:0/1196250425 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f92bc0a3ef0 msgr2=0x7f92bc0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.691+0000 7f92d0f50640 1 --2- 172.21.15.28:0/1196250425 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f92bc0a3ef0 0x7f92bc0a63b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f92b8034000 tx=0x7f92b8007bf0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.691+0000 7f92d0f50640 1 -- 172.21.15.28:0/1196250425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92c40aa2a0 msgr2=0x7f92c4149df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.691+0000 7f92d0f50640 1 --2- 172.21.15.28:0/1196250425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92c40aa2a0 0x7f92c4149df0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f92cc067c00 tx=0x7f92cc070890 comp rx=0 tx=0).stop 2024-04-21T08:53:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.691+0000 7f92d0f50640 1 -- 172.21.15.28:0/1196250425 shutdown_connections 2024-04-21T08:53:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.691+0000 7f92d0f50640 1 --2- 172.21.15.28:0/1196250425 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f92bc0a3ef0 0x7f92bc0a63b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.692+0000 7f92d0f50640 1 --2- 172.21.15.28:0/1196250425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c4143a60 0x7f92c4143f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.692+0000 7f92d0f50640 1 --2- 172.21.15.28:0/1196250425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f92c40aa2a0 0x7f92c4149df0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.692+0000 7f92d0f50640 1 --2- 172.21.15.28:0/1196250425 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f92c40a51d0 0x7f92c41498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.692+0000 7f92d0f50640 1 -- 172.21.15.28:0/1196250425 >> 172.21.15.28:0/1196250425 conn(0x7f92c401aa70 msgr2=0x7f92c40a3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.692+0000 7f92d0f50640 1 -- 172.21.15.28:0/1196250425 shutdown_connections 2024-04-21T08:53:40.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:40.692+0000 7f92d0f50640 1 -- 172.21.15.28:0/1196250425 wait complete. 2024-04-21T08:53:40.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.692+0000 7fb1c79fb740 20 client.16479 trim_cache size 0 max 16384 2024-04-21T08:53:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.733+0000 7fb1c79fb740 1 Processor -- start 2024-04-21T08:53:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.733+0000 7fb1c79fb740 1 -- start start 2024-04-21T08:53:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.733+0000 7fb1c79fb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873df8580 0x55c873e0d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.733+0000 7fb1c79fb740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873df8ea0 0x55c873dff390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.733+0000 7fb1c79fb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873e01a80 0x55c873df4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.733+0000 7fb1c79fb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a0012910 con 0x55c873df8580 2024-04-21T08:53:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.733+0000 7fb1c79fb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a00101e0 con 0x55c873e01a80 2024-04-21T08:53:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.733+0000 7fb1c79fb740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a000e030 con 0x55c873df8ea0 2024-04-21T08:53:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.734+0000 7fb1be59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873df8580 0x55c873e0d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.734+0000 7fb1be59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873df8580 0x55c873e0d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45666/0 (socket says 172.21.15.28:45666) 2024-04-21T08:53:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.734+0000 7fb1be59c640 1 -- 172.21.15.28:0/4268369832 learned_addr learned my addr 172.21.15.28:0/4268369832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.734+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/4268369832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873e01a80 0x55c873df4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.734+0000 7fb1be59c640 1 -- 172.21.15.28:0/4268369832 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873df8ea0 msgr2=0x55c873dff390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.734+0000 7fb1be59c640 1 --2- 172.21.15.28:0/4268369832 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873df8ea0 0x55c873dff390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.734+0000 7fb1be59c640 1 -- 172.21.15.28:0/4268369832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873e01a80 msgr2=0x55c873df4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.734+0000 7fb1be59c640 1 --2- 172.21.15.28:0/4268369832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873e01a80 0x55c873df4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.734+0000 7fb1be59c640 1 -- 172.21.15.28:0/4268369832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c873dfa6d0 con 0x55c873df8580 2024-04-21T08:53:40.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.735+0000 7fb1be59c640 1 --2- 172.21.15.28:0/4268369832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873df8580 0x55c873e0d7b0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0012190 tx=0x7fb1b0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b83439dffc839 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:40.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.735+0000 7fb1a7fff640 1 -- 172.21.15.28:0/4268369832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b00239e0 con 0x55c873df8580 2024-04-21T08:53:40.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.736+0000 7fb1a7fff640 1 -- 172.21.15.28:0/4268369832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1b001a040 con 0x55c873df8580 2024-04-21T08:53:40.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.737+0000 7fb1c79fb740 1 -- 172.21.15.28:0/4268369832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873df8580 msgr2=0x55c873e0d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:40.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.737+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/4268369832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873df8580 0x55c873e0d7b0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0012190 tx=0x7fb1b0012000 comp rx=0 tx=0).stop 2024-04-21T08:53:40.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.738+0000 7fb1c79fb740 1 -- 172.21.15.28:0/4268369832 shutdown_connections 2024-04-21T08:53:40.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.738+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/4268369832 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873e01a80 0x55c873df4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.738+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/4268369832 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873df8ea0 0x55c873dff390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.738+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/4268369832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873df8580 0x55c873e0d7b0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.738+0000 7fb1c79fb740 1 -- 172.21.15.28:0/4268369832 >> 172.21.15.28:0/4268369832 conn(0x55c873df2cb0 msgr2=0x55c873df4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:40.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.742+0000 7fb1c79fb740 1 -- 172.21.15.28:0/4268369832 shutdown_connections 2024-04-21T08:53:40.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.742+0000 7fb1c79fb740 1 -- 172.21.15.28:0/4268369832 wait complete. 2024-04-21T08:53:40.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.742+0000 7fb1c79fb740 1 -- start start 2024-04-21T08:53:40.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.743+0000 7fb1c79fb740 1 Processor -- start 2024-04-21T08:53:40.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.743+0000 7fb1c79fb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873df8580 0x55c873e09d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:40.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.743+0000 7fb1c79fb740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873df8ea0 0x55c873e0a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:40.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.743+0000 7fb1c79fb740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873e01a80 0x55c873e05450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:40.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.743+0000 7fb1c79fb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a000e6a0 con 0x55c873df8580 2024-04-21T08:53:40.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.743+0000 7fb1c79fb740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a000fb10 con 0x55c873e01a80 2024-04-21T08:53:40.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.743+0000 7fb1c79fb740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a000ed70 con 0x55c873df8ea0 2024-04-21T08:53:40.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.743+0000 7fb1c79fb740 10 client.0 _ll_register_callbacks cb 0x7fb1b809b9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:40.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.744+0000 7fb1bdd9b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873df8ea0 0x55c873e0a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:40.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.744+0000 7fb1bdd9b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873df8ea0 0x55c873e0a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:50300/0 (socket says 172.21.15.28:50300) 2024-04-21T08:53:40.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.745+0000 7fb1bdd9b640 1 -- 172.21.15.28:0/1861186704 learned_addr learned my addr 172.21.15.28:0/1861186704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:40.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.745+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/1861186704 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873e01a80 0x55c873e05450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:40.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.746+0000 7fb1bed9d640 1 -- 172.21.15.28:0/1861186704 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873df8ea0 msgr2=0x55c873e0a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:40.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.746+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/1861186704 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873df8ea0 0x55c873e0a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.746+0000 7fb1bed9d640 1 -- 172.21.15.28:0/1861186704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873df8580 msgr2=0x55c873e09d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:40.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.746+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/1861186704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873df8580 0x55c873e09d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.746+0000 7fb1bed9d640 1 -- 172.21.15.28:0/1861186704 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c873e05c80 con 0x55c873e01a80 2024-04-21T08:53:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.747+0000 7fb1bed9d640 1 --2- 172.21.15.28:0/1861186704 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873e01a80 0x55c873e05450 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb1b80a4590 tx=0x7fb1b80a0350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44c3ffcfac906d9d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.747+0000 7fb1a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:53:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.747+0000 7fb1a5ffb640 1 -- 172.21.15.28:0/1861186704 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b80b4040 con 0x55c873e01a80 2024-04-21T08:53:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.747+0000 7fb1c79fb740 1 -- 172.21.15.28:0/1861186704 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c873d64a70 con 0x55c873e01a80 2024-04-21T08:53:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.747+0000 7fb1a5ffb640 1 -- 172.21.15.28:0/1861186704 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fb1b809ae50 con 0x55c873e01a80 2024-04-21T08:53:40.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.748+0000 7fb1a5ffb640 1 -- 172.21.15.28:0/1861186704 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb1b8098230 con 0x55c873e01a80 2024-04-21T08:53:40.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.749+0000 7fb1c79fb740 10 client.25835 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.749+0000 7fb1c79fb740 1 -- 172.21.15.28:0/1861186704 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c873e77cf0 con 0x55c873e01a80 2024-04-21T08:53:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.750+0000 7fb1a5ffb640 1 -- 172.21.15.28:0/1861186704 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55c873d64a70 con 0x55c873e01a80 2024-04-21T08:53:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.750+0000 7fb1a5ffb640 1 -- 172.21.15.28:0/1861186704 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fb1b809a550 con 0x55c873e01a80 2024-04-21T08:53:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.751+0000 7fb1c79fb740 10 client.25835 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.751+0000 7fb1c79fb740 10 client.25835 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.751+0000 7fb1c79fb740 20 client.25835 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:40.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.751+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/1861186704 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c873e78300 0x55c873e7a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.751+0000 7fb1c79fb740 4 client.25835 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:53:40.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.751+0000 7fb1c79fb740 1 -- 172.21.15.28:0/1861186704 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c873ce0590 con 0x55c873e78300 2024-04-21T08:53:40.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.753+0000 7fb1be59c640 1 --2- 172.21.15.28:0/1861186704 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c873e78300 0x55c873e7a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.754+0000 7fb1be59c640 1 --2- 172.21.15.28:0/1861186704 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c873e78300 0x55c873e7a720 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13975cf31e518cfd server_cookie=48e40a5ac3389f06 in_seq=0 out_seq=0 2024-04-21T08:53:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.754+0000 7fb1a5ffb640 10 client.25835 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:40.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.756+0000 7fb1a5ffb640 1 -- 172.21.15.28:0/1861186704 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c873ce0590 con 0x55c873e78300 2024-04-21T08:53:40.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.756+0000 7fb1a5ffb640 10 client.25835 handle_command_reply: tid=0 2024-04-21T08:53:40.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.758+0000 7fb1c79fb740 1 client.25835 shutdown 2024-04-21T08:53:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.762+0000 7fb1c79fb740 1 -- 172.21.15.28:0/1861186704 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873e01a80 msgr2=0x55c873e05450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.762+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/1861186704 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873e01a80 0x55c873e05450 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb1b80a4590 tx=0x7fb1b80a0350 comp rx=0 tx=0).stop 2024-04-21T08:53:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.763+0000 7fb1c79fb740 1 -- 172.21.15.28:0/1861186704 shutdown_connections 2024-04-21T08:53:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.763+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/1861186704 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c873e78300 0x55c873e7a720 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.763+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/1861186704 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c873e01a80 0x55c873e05450 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.763+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/1861186704 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c873df8ea0 0x55c873e0a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.763+0000 7fb1c79fb740 1 --2- 172.21.15.28:0/1861186704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c873df8580 0x55c873e09d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.763+0000 7fb1c79fb740 1 -- 172.21.15.28:0/1861186704 >> 172.21.15.28:0/1861186704 conn(0x55c873df2cb0 msgr2=0x55c873df1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.764+0000 7fb1a5ffb640 0 client.25835 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:40.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.767+0000 7fb1c79fb740 1 -- 172.21.15.28:0/1861186704 shutdown_connections 2024-04-21T08:53:40.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.767+0000 7fb1c79fb740 1 -- 172.21.15.28:0/1861186704 wait complete. 2024-04-21T08:53:40.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.769+0000 7fb1c79fb740 20 client.25835 trim_cache size 0 max 16384 2024-04-21T08:53:40.770 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:53:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T08:53:40.771 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T08:53:40.771 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:53:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.772+0000 7fb1c4d2b640 1 -- 172.21.15.28:0/2016087122 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb1a80a3d60 msgr2=0x7fb1a80a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.772+0000 7fb1c4d2b640 1 --2- 172.21.15.28:0/2016087122 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb1a80a3d60 0x7fb1a80a6220 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009e00 tx=0x7fb1b00098f0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.772+0000 7fb1c4d2b640 1 -- 172.21.15.28:0/2016087122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c011b730 msgr2=0x7fb1c01abe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.772+0000 7fb1c4d2b640 1 --2- 172.21.15.28:0/2016087122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c011b730 0x7fb1c01abe60 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800bdd0 tx=0x7fb1b8009ed0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.772+0000 7fb1c4d2b640 1 -- 172.21.15.28:0/2016087122 shutdown_connections 2024-04-21T08:53:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.772+0000 7fb1c4d2b640 1 --2- 172.21.15.28:0/2016087122 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb1a80a3d60 0x7fb1a80a6220 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.772+0000 7fb1c4d2b640 1 --2- 172.21.15.28:0/2016087122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1c011b730 0x7fb1c01abe60 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.772+0000 7fb1c4d2b640 1 --2- 172.21.15.28:0/2016087122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb1c01106d0 0x7fb1c01a7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.772+0000 7fb1c4d2b640 1 --2- 172.21.15.28:0/2016087122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb1c010e7b0 0x7fb1c01a75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.772+0000 7fb1c4d2b640 1 -- 172.21.15.28:0/2016087122 >> 172.21.15.28:0/2016087122 conn(0x7fb1c006d070 msgr2=0x7fb1c0070a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.774+0000 7fb1c4d2b640 1 -- 172.21.15.28:0/2016087122 shutdown_connections 2024-04-21T08:53:40.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.774+0000 7fb1c4d2b640 1 -- 172.21.15.28:0/2016087122 wait complete. 2024-04-21T08:53:40.793 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:53:40.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:53:40.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.992+0000 7fe487006640 1 -- 172.21.15.28:0/1576717243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe480101130 msgr2=0x7fe4801015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:40.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.992+0000 7fe487006640 1 --2- 172.21.15.28:0/1576717243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe480101130 0x7fe4801015b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe470009900 tx=0x7fe47002bc60 comp rx=0 tx=0).stop 2024-04-21T08:53:40.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.995+0000 7fe487006640 1 -- 172.21.15.28:0/1576717243 shutdown_connections 2024-04-21T08:53:40.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.995+0000 7fe487006640 1 --2- 172.21.15.28:0/1576717243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe480101af0 0x7fe48010dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.996+0000 7fe487006640 1 --2- 172.21.15.28:0/1576717243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe480101130 0x7fe4801015b0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.996+0000 7fe487006640 1 --2- 172.21.15.28:0/1576717243 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe480105e80 0x7fe480106260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:40.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.996+0000 7fe487006640 1 -- 172.21.15.28:0/1576717243 >> 172.21.15.28:0/1576717243 conn(0x7fe480077e30 msgr2=0x7fe480078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:40.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:40.998+0000 7fe487006640 1 -- 172.21.15.28:0/1576717243 shutdown_connections 2024-04-21T08:53:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.000+0000 7fe487006640 1 -- 172.21.15.28:0/1576717243 wait complete. 2024-04-21T08:53:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.002+0000 7fe487006640 1 Processor -- start 2024-04-21T08:53:41.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.002+0000 7fe487006640 1 -- start start 2024-04-21T08:53:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.003+0000 7fe487006640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe480101130 0x7fe480116440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.003+0000 7fe487006640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe480101af0 0x7fe480110500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.003+0000 7fe487006640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe480105e80 0x7fe480110a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.003+0000 7fe487006640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe48010eeb0 con 0x7fe480101af0 2024-04-21T08:53:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.003+0000 7fe487006640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe48010ed30 con 0x7fe480101130 2024-04-21T08:53:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.003+0000 7fe4855a3640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe480105e80 0x7fe480110a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.004+0000 7fe4855a3640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe480105e80 0x7fe480110a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:50326/0 (socket says 172.21.15.28:50326) 2024-04-21T08:53:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.004+0000 7fe4855a3640 1 -- 172.21.15.28:0/1876573367 learned_addr learned my addr 172.21.15.28:0/1876573367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.004+0000 7fe487006640 1 -- 172.21.15.28:0/1876573367 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe48010f030 con 0x7fe480105e80 2024-04-21T08:53:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.005+0000 7fe47ffff640 1 --2- 172.21.15.28:0/1876573367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe480101af0 0x7fe480110500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.005+0000 7fe47ffff640 1 -- 172.21.15.28:0/1876573367 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe480105e80 msgr2=0x7fe480110a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.005+0000 7fe47ffff640 1 --2- 172.21.15.28:0/1876573367 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe480105e80 0x7fe480110a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.005+0000 7fe47ffff640 1 -- 172.21.15.28:0/1876573367 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe480101130 msgr2=0x7fe480116440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:53:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.005+0000 7fe47ffff640 1 --2- 172.21.15.28:0/1876573367 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe480101130 0x7fe480116440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.006+0000 7fe47ffff640 1 -- 172.21.15.28:0/1876573367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe480111080 con 0x7fe480101af0 2024-04-21T08:53:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.006+0000 7fe47ffff640 1 --2- 172.21.15.28:0/1876573367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe480101af0 0x7fe480110500 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe470009230 tx=0x7fe47002c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.007+0000 7fe47dffb640 1 -- 172.21.15.28:0/1876573367 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe470043020 con 0x7fe480101af0 2024-04-21T08:53:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.007+0000 7fe47dffb640 1 -- 172.21.15.28:0/1876573367 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4700052b0 con 0x7fe480101af0 2024-04-21T08:53:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.008+0000 7fe487006640 1 -- 172.21.15.28:0/1876573367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe480111310 con 0x7fe480101af0 2024-04-21T08:53:41.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.008+0000 7fe487006640 1 -- 172.21.15.28:0/1876573367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4801b4190 con 0x7fe480101af0 2024-04-21T08:53:41.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.009+0000 7fe47dffb640 1 -- 172.21.15.28:0/1876573367 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe470005000 con 0x7fe480101af0 2024-04-21T08:53:41.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.016+0000 7fe47dffb640 1 -- 172.21.15.28:0/1876573367 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe470046020 con 0x7fe480101af0 2024-04-21T08:53:41.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.018+0000 7fe47dffb640 1 --2- 172.21.15.28:0/1876573367 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4640a3df0 0x7fe4640a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.019+0000 7fe47dffb640 1 -- 172.21.15.28:0/1876573367 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe4700f27b0 con 0x7fe480101af0 2024-04-21T08:53:41.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.019+0000 7fe487006640 1 -- 172.21.15.28:0/1876573367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4801025f0 con 0x7fe480101af0 2024-04-21T08:53:41.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.023+0000 7fe484da2640 1 --2- 172.21.15.28:0/1876573367 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4640a3df0 0x7fe4640a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.028+0000 7fe484da2640 1 --2- 172.21.15.28:0/1876573367 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4640a3df0 0x7fe4640a62b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe478005fd0 tx=0x7fe478005f00 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:41.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.031+0000 7fe47dffb640 1 -- 172.21.15.28:0/1876573367 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe47004d070 con 0x7fe480101af0 2024-04-21T08:53:41.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.504+0000 7fe487006640 1 -- 172.21.15.28:0/1876573367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe480068550 con 0x7fe480101af0 2024-04-21T08:53:41.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.505+0000 7fe47dffb640 1 -- 172.21.15.28:0/1876573367 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7fe480068550 con 0x7fe480101af0 2024-04-21T08:53:41.505 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:53:41.505 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:52:11.568185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24487,24488,24463]},"id":1}]} 2024-04-21T08:53:41.506 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-21T08:53:41.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.510+0000 7fe487006640 1 -- 172.21.15.28:0/1876573367 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4640a3df0 msgr2=0x7fe4640a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:41.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.510+0000 7fe487006640 1 --2- 172.21.15.28:0/1876573367 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4640a3df0 0x7fe4640a62b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe478005fd0 tx=0x7fe478005f00 comp rx=0 tx=0).stop 2024-04-21T08:53:41.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.510+0000 7fe487006640 1 -- 172.21.15.28:0/1876573367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe480101af0 msgr2=0x7fe480110500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.510+0000 7fe487006640 1 --2- 172.21.15.28:0/1876573367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe480101af0 0x7fe480110500 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe470009230 tx=0x7fe47002c9e0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.511+0000 7fe487006640 1 -- 172.21.15.28:0/1876573367 shutdown_connections 2024-04-21T08:53:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.511+0000 7fe487006640 1 --2- 172.21.15.28:0/1876573367 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4640a3df0 0x7fe4640a62b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.511+0000 7fe487006640 1 --2- 172.21.15.28:0/1876573367 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe480105e80 0x7fe480110a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.511+0000 7fe487006640 1 --2- 172.21.15.28:0/1876573367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe480101af0 0x7fe480110500 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.511+0000 7fe487006640 1 --2- 172.21.15.28:0/1876573367 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe480101130 0x7fe480116440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.511+0000 7fe487006640 1 -- 172.21.15.28:0/1876573367 >> 172.21.15.28:0/1876573367 conn(0x7fe480077e30 msgr2=0x7fe4800fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.511+0000 7fe487006640 1 -- 172.21.15.28:0/1876573367 shutdown_connections 2024-04-21T08:53:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.511+0000 7fe487006640 1 -- 172.21.15.28:0/1876573367 wait complete. 2024-04-21T08:53:41.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:53:41.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.731+0000 7fcb0e443640 1 -- 172.21.15.28:0/3989465003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb0810e980 msgr2=0x7fcb0810ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:41.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.731+0000 7fcb0e443640 1 --2- 172.21.15.28:0/3989465003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb0810e980 0x7fcb0810ed60 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fcaf800b9f0 tx=0x7fcaf802d980 comp rx=0 tx=0).stop 2024-04-21T08:53:41.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.734+0000 7fcb0e443640 1 -- 172.21.15.28:0/3989465003 shutdown_connections 2024-04-21T08:53:41.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.734+0000 7fcb0e443640 1 --2- 172.21.15.28:0/3989465003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb08109910 0x7fcb081157d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.734+0000 7fcb0e443640 1 --2- 172.21.15.28:0/3989465003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcb08108f50 0x7fcb081093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.734+0000 7fcb0e443640 1 --2- 172.21.15.28:0/3989465003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb0810e980 0x7fcb0810ed60 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.734+0000 7fcb0e443640 1 -- 172.21.15.28:0/3989465003 >> 172.21.15.28:0/3989465003 conn(0x7fcb0806d4c0 msgr2=0x7fcb0806d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:41.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.736+0000 7fcb0e443640 1 -- 172.21.15.28:0/3989465003 shutdown_connections 2024-04-21T08:53:41.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.739+0000 7fcb0e443640 1 -- 172.21.15.28:0/3989465003 wait complete. 2024-04-21T08:53:41.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.740+0000 7fcb0e443640 1 Processor -- start 2024-04-21T08:53:41.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.740+0000 7fcb0e443640 1 -- start start 2024-04-21T08:53:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.740+0000 7fcb0e443640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcb08108f50 0x7fcb081141e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.740+0000 7fcb0e443640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb08109910 0x7fcb08114720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.740+0000 7fcb0e443640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb0810e980 0x7fcb08114c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.740+0000 7fcb0e443640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcb08117e30 con 0x7fcb08109910 2024-04-21T08:53:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.740+0000 7fcb0e443640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fcb08117cb0 con 0x7fcb0810e980 2024-04-21T08:53:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.740+0000 7fcb0e443640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcb08117fb0 con 0x7fcb08108f50 2024-04-21T08:53:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.741+0000 7fcb07fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcb08108f50 0x7fcb081141e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.741+0000 7fcb07fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcb08108f50 0x7fcb081141e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:50342/0 (socket says 172.21.15.28:50342) 2024-04-21T08:53:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.741+0000 7fcb07fff640 1 -- 172.21.15.28:0/1369979531 learned_addr learned my addr 172.21.15.28:0/1369979531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.741+0000 7fcb0c9e0640 1 --2- 172.21.15.28:0/1369979531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb0810e980 0x7fcb08114c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.741+0000 7fcb0c9e0640 1 -- 172.21.15.28:0/1369979531 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcb08108f50 msgr2=0x7fcb081141e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.741+0000 7fcb0c9e0640 1 --2- 172.21.15.28:0/1369979531 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcb08108f50 0x7fcb081141e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.741+0000 7fcb0c9e0640 1 -- 172.21.15.28:0/1369979531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb08109910 msgr2=0x7fcb08114720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.741+0000 7fcb0c9e0640 1 --2- 172.21.15.28:0/1369979531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb08109910 0x7fcb08114720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.741+0000 7fcb0c9e0640 1 -- 172.21.15.28:0/1369979531 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb081bc130 con 0x7fcb0810e980 2024-04-21T08:53:41.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.743+0000 7fcb0c9e0640 1 --2- 172.21.15.28:0/1369979531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb0810e980 0x7fcb08114c60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcb00005fd0 tx=0x7fcb000047a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:41.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.744+0000 7fcb057fa640 1 -- 172.21.15.28:0/1369979531 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0000e710 con 0x7fcb0810e980 2024-04-21T08:53:41.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.744+0000 7fcb057fa640 1 -- 172.21.15.28:0/1369979531 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb00019070 con 0x7fcb0810e980 2024-04-21T08:53:41.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.744+0000 7fcb057fa640 1 -- 172.21.15.28:0/1369979531 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb00008da0 con 0x7fcb0810e980 2024-04-21T08:53:41.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.745+0000 7fcb0e443640 1 -- 172.21.15.28:0/1369979531 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb081bc360 con 0x7fcb0810e980 2024-04-21T08:53:41.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.746+0000 7fcb0e443640 1 -- 172.21.15.28:0/1369979531 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb081bc830 con 0x7fcb0810e980 2024-04-21T08:53:41.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.750+0000 7fcb11034740 1 Processor -- start 2024-04-21T08:53:41.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.757+0000 7fcb11034740 1 -- start start 2024-04-21T08:53:41.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.758+0000 7fcb11034740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadb4d0 0x55b1cbadb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.758+0000 7fcb11034740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbadbe80 0x55b1cbae4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.758+0000 7fcb11034740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbae4a00 0x55b1cbae6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.758+0000 7fcb11034740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b1cba401f0 con 0x55b1cbadbe80 2024-04-21T08:53:41.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.758+0000 7fcb11034740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55b1cb9dd4b0 con 0x55b1cbae4a00 2024-04-21T08:53:41.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.758+0000 7fcb11034740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55b1cba3c1b0 con 0x55b1cbadb4d0 2024-04-21T08:53:41.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.758+0000 7fcb057fa640 1 -- 172.21.15.28:0/1369979531 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcb0000e8b0 con 0x7fcb0810e980 2024-04-21T08:53:41.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.759+0000 7fcb057fa640 1 --2- 172.21.15.28:0/1369979531 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcaf00a3f20 0x7fcaf00a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.761+0000 7fcb0c9e0640 1 --2- 172.21.15.28:0/1369979531 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcaf00a3f20 0x7fcaf00a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.761+0000 7fcb077fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadb4d0 0x55b1cbadb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.761+0000 7fcb077fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadb4d0 0x55b1cbadb8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:50344/0 (socket says 172.21.15.28:50344) 2024-04-21T08:53:41.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.761+0000 7fcb077fe640 1 -- 172.21.15.28:0/765801357 learned_addr learned my addr 172.21.15.28:0/765801357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:41.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.761+0000 7fcb057fa640 1 -- 172.21.15.28:0/1369979531 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fcb000d03f0 con 0x7fcb0810e980 2024-04-21T08:53:41.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.762+0000 7fcb077fe640 1 --2- 172.21.15.28:0/765801357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbae4a00 0x55b1cbae6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.762+0000 7fcb07fff640 1 --2- 172.21.15.28:0/765801357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbadbe80 0x55b1cbae4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.763+0000 7fcb07fff640 1 -- 172.21.15.28:0/765801357 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadb4d0 msgr2=0x55b1cbadb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.763+0000 7fcb07fff640 1 --2- 172.21.15.28:0/765801357 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadb4d0 0x55b1cbadb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.763+0000 7fcb07fff640 1 -- 172.21.15.28:0/765801357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbae4a00 msgr2=0x55b1cbae6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.763+0000 7fcb07fff640 1 --2- 172.21.15.28:0/765801357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbae4a00 0x55b1cbae6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.763+0000 7fcb07fff640 1 -- 172.21.15.28:0/765801357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1cb9cf1b0 con 0x55b1cbadbe80 2024-04-21T08:53:41.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.765+0000 7fcb0c9e0640 1 --2- 172.21.15.28:0/1369979531 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcaf00a3f20 0x7fcaf00a63e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcb00098430 tx=0x7fcb00097b90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:41.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.765+0000 7fcb07fff640 1 --2- 172.21.15.28:0/765801357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbadbe80 0x55b1cbae4430 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fcaf800e090 tx=0x7fcaf8009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56da4be6d02d80a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.766+0000 7fcaee7fc640 1 -- 172.21.15.28:0/765801357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8036040 con 0x55b1cbadbe80 2024-04-21T08:53:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.766+0000 7fcaee7fc640 1 -- 172.21.15.28:0/765801357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcaf8009e80 con 0x55b1cbadbe80 2024-04-21T08:53:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.767+0000 7fcb11034740 1 -- 172.21.15.28:0/765801357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbadbe80 msgr2=0x55b1cbae4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.768+0000 7fcaee7fc640 1 -- 172.21.15.28:0/765801357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8036040 con 0x55b1cbadbe80 2024-04-21T08:53:41.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.768+0000 7fcb11034740 1 --2- 172.21.15.28:0/765801357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbadbe80 0x55b1cbae4430 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fcaf800e090 tx=0x7fcaf8009d10 comp rx=0 tx=0).stop 2024-04-21T08:53:41.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.770+0000 7fcb11034740 1 -- 172.21.15.28:0/765801357 shutdown_connections 2024-04-21T08:53:41.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.770+0000 7fcb11034740 1 --2- 172.21.15.28:0/765801357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbae4a00 0x55b1cbae6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.770+0000 7fcb11034740 1 --2- 172.21.15.28:0/765801357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbadbe80 0x55b1cbae4430 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.770+0000 7fcb11034740 1 --2- 172.21.15.28:0/765801357 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadb4d0 0x55b1cbadb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.770+0000 7fcb11034740 1 -- 172.21.15.28:0/765801357 >> 172.21.15.28:0/765801357 conn(0x55b1cbad5ba0 msgr2=0x55b1cbad5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.774+0000 7fcb11034740 1 -- 172.21.15.28:0/765801357 shutdown_connections 2024-04-21T08:53:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.774+0000 7fcb11034740 1 -- 172.21.15.28:0/765801357 wait complete. 2024-04-21T08:53:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.774+0000 7fcb11034740 1 -- start start 2024-04-21T08:53:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.774+0000 7fcb11034740 1 Processor -- start 2024-04-21T08:53:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.775+0000 7fcb11034740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbadb4d0 0x55b1cbae42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.775+0000 7fcb11034740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbadbe80 0x55b1cbae1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.775+0000 7fcb11034740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbae4a00 0x55b1cbae1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.775+0000 7fcb11034740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b1cbae7900 con 0x55b1cbadb4d0 2024-04-21T08:53:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.775+0000 7fcb11034740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55b1cb9f9500 con 0x55b1cbadbe80 2024-04-21T08:53:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.775+0000 7fcb11034740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55b1cbae7a80 con 0x55b1cbae4a00 2024-04-21T08:53:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.775+0000 7fcb11034740 10 client.0 _ll_register_callbacks cb 0x55b1cb9c7b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.776+0000 7fcb077fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbae4a00 0x55b1cbae1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.776+0000 7fcb07fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbadbe80 0x55b1cbae1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.776+0000 7fcb077fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbae4a00 0x55b1cbae1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:50354/0 (socket says 172.21.15.28:50354) 2024-04-21T08:53:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.776+0000 7fcb077fe640 1 -- 172.21.15.28:0/3397679965 learned_addr learned my addr 172.21.15.28:0/3397679965 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.776+0000 7fcb077fe640 1 --2- 172.21.15.28:0/3397679965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbadb4d0 0x55b1cbae42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.776+0000 7fcb077fe640 1 -- 172.21.15.28:0/3397679965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbadbe80 msgr2=0x55b1cbae1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.776+0000 7fcb077fe640 1 --2- 172.21.15.28:0/3397679965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbadbe80 0x55b1cbae1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.776+0000 7fcb077fe640 1 -- 172.21.15.28:0/3397679965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbadb4d0 msgr2=0x55b1cbae42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.776+0000 7fcb077fe640 1 --2- 172.21.15.28:0/3397679965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbadb4d0 0x55b1cbae42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.776+0000 7fcb077fe640 1 -- 172.21.15.28:0/3397679965 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1cb7523f0 con 0x55b1cbae4a00 2024-04-21T08:53:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.780+0000 7fcb077fe640 1 --2- 172.21.15.28:0/3397679965 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbae4a00 0x55b1cbae1b70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcafc013f90 tx=0x7fcafc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6e6c6e1567d586d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.782+0000 7fcae77fe640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:53:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.785+0000 7fcae77fe640 1 -- 172.21.15.28:0/3397679965 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc01f640 con 0x55b1cbae4a00 2024-04-21T08:53:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.785+0000 7fcae77fe640 1 -- 172.21.15.28:0/3397679965 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fcafc01f7e0 con 0x55b1cbae4a00 2024-04-21T08:53:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.786+0000 7fcb11034740 1 -- 172.21.15.28:0/3397679965 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcaf8009e80 con 0x55b1cbae4a00 2024-04-21T08:53:41.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.790+0000 7fcae77fe640 1 -- 172.21.15.28:0/3397679965 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcafc01f9e0 con 0x55b1cbae4a00 2024-04-21T08:53:41.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.791+0000 7fcb11034740 10 client.25843 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:41.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.791+0000 7fcb11034740 1 -- 172.21.15.28:0/3397679965 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b1cb81c7d0 con 0x55b1cbae4a00 2024-04-21T08:53:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.793+0000 7fcae77fe640 1 -- 172.21.15.28:0/3397679965 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcaf8009e80 con 0x55b1cbae4a00 2024-04-21T08:53:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.793+0000 7fcae77fe640 1 -- 172.21.15.28:0/3397679965 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fcafc01d020 con 0x55b1cbae4a00 2024-04-21T08:53:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.794+0000 7fcb11034740 10 client.25843 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.794+0000 7fcb11034740 10 client.25843 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:53:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.794+0000 7fcb11034740 20 client.25843 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.794+0000 7fcb11034740 1 --2- 172.21.15.28:0/3397679965 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b1cbb16c80 0x55b1cbb19070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.794+0000 7fcb11034740 4 client.25843 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.794+0000 7fcb11034740 1 -- 172.21.15.28:0/3397679965 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1cb9df930 con 0x55b1cbb16c80 2024-04-21T08:53:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.795+0000 7fcb07fff640 1 --2- 172.21.15.28:0/3397679965 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b1cbb16c80 0x55b1cbb19070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.796+0000 7fcb07fff640 1 --2- 172.21.15.28:0/3397679965 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b1cbb16c80 0x55b1cbb19070 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=da1bce4346999a3a server_cookie=bc3de5359631b1e4 in_seq=0 out_seq=0 2024-04-21T08:53:41.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.797+0000 7fcae77fe640 10 client.25843 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:53:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.803+0000 7fcae77fe640 1 -- 172.21.15.28:0/3397679965 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b1cb9df930 con 0x55b1cbb16c80 2024-04-21T08:53:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.803+0000 7fcae77fe640 10 client.25843 handle_command_reply: tid=0 2024-04-21T08:53:41.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.806+0000 7fcb11034740 1 client.25843 shutdown 2024-04-21T08:53:41.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.813+0000 7fcb11034740 1 -- 172.21.15.28:0/3397679965 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbae4a00 msgr2=0x55b1cbae1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:41.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.813+0000 7fcb11034740 1 --2- 172.21.15.28:0/3397679965 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbae4a00 0x55b1cbae1b70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcafc013f90 tx=0x7fcafc0046c0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.815+0000 7fcb11034740 1 -- 172.21.15.28:0/3397679965 shutdown_connections 2024-04-21T08:53:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.815+0000 7fcb11034740 1 --2- 172.21.15.28:0/3397679965 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b1cbb16c80 0x55b1cbb19070 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.815+0000 7fcb11034740 1 --2- 172.21.15.28:0/3397679965 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbae4a00 0x55b1cbae1b70 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcafc013f90 tx=0x7fcafc0046c0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.815+0000 7fcb11034740 1 --2- 172.21.15.28:0/3397679965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbadbe80 0x55b1cbae1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.815+0000 7fcb11034740 1 --2- 172.21.15.28:0/3397679965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbadb4d0 0x55b1cbae42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.815+0000 7fcb11034740 1 -- 172.21.15.28:0/3397679965 >> 172.21.15.28:0/3397679965 conn(0x55b1cbad5ba0 msgr2=0x55b1cbad78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.815+0000 7fcae77fe640 0 client.25843 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:53:41.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.823+0000 7fcb11034740 1 -- 172.21.15.28:0/3397679965 shutdown_connections 2024-04-21T08:53:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.825+0000 7fcb11034740 1 -- 172.21.15.28:0/3397679965 wait complete. 2024-04-21T08:53:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.825+0000 7fcb11034740 20 client.25843 trim_cache size 0 max 16384 2024-04-21T08:53:41.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.854+0000 7fcb11034740 1 Processor -- start 2024-04-21T08:53:41.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.854+0000 7fcb11034740 1 -- start start 2024-04-21T08:53:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.854+0000 7fcb11034740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbadb4d0 0x55b1cbaf0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.855+0000 7fcb11034740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadbe80 0x55b1cbad4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.855+0000 7fcb077fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbadb4d0 0x55b1cbaf0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.855+0000 7fcb077fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbadb4d0 0x55b1cbaf0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:33490/0 (socket says 172.21.15.28:33490) 2024-04-21T08:53:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.855+0000 7fcb11034740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbae3d60 0x55b1cbae41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.855+0000 7fcb077fe640 1 -- 172.21.15.28:0/1245001635 learned_addr learned my addr 172.21.15.28:0/1245001635 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.855+0000 7fcb11034740 1 -- 172.21.15.28:0/1245001635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081b5720 con 0x55b1cbae3d60 2024-04-21T08:53:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.855+0000 7fcb11034740 1 -- 172.21.15.28:0/1245001635 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081b2ff0 con 0x55b1cbadb4d0 2024-04-21T08:53:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.855+0000 7fcb11034740 1 -- 172.21.15.28:0/1245001635 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081b0e40 con 0x55b1cbadbe80 2024-04-21T08:53:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.856+0000 7fcb07fff640 1 --2- 172.21.15.28:0/1245001635 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadbe80 0x55b1cbad4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.856+0000 7fcb077fe640 1 --2- 172.21.15.28:0/1245001635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbae3d60 0x55b1cbae41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.857+0000 7fcb077fe640 1 -- 172.21.15.28:0/1245001635 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadbe80 msgr2=0x55b1cbad4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:41.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.857+0000 7fcb077fe640 1 --2- 172.21.15.28:0/1245001635 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadbe80 0x55b1cbad4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.857+0000 7fcb077fe640 1 -- 172.21.15.28:0/1245001635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbae3d60 msgr2=0x55b1cbae41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:41.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.857+0000 7fcb077fe640 1 --2- 172.21.15.28:0/1245001635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbae3d60 0x55b1cbae41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.857+0000 7fcb077fe640 1 -- 172.21.15.28:0/1245001635 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1cbad53f0 con 0x55b1cbadb4d0 2024-04-21T08:53:41.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.857+0000 7fcb077fe640 1 --2- 172.21.15.28:0/1245001635 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbadb4d0 0x55b1cbaf0840 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fcafc0105c0 tx=0x7fcafc013c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcddebe427deaae server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.859+0000 7fcaedffb640 1 -- 172.21.15.28:0/1245001635 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc0109b0 con 0x55b1cbadb4d0 2024-04-21T08:53:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.859+0000 7fcaedffb640 1 -- 172.21.15.28:0/1245001635 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcafc005e70 con 0x55b1cbadb4d0 2024-04-21T08:53:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.859+0000 7fcaedffb640 1 -- 172.21.15.28:0/1245001635 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc0043d0 con 0x55b1cbadb4d0 2024-04-21T08:53:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.861+0000 7fcb11034740 1 -- 172.21.15.28:0/1245001635 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbadb4d0 msgr2=0x55b1cbaf0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.861+0000 7fcb11034740 1 --2- 172.21.15.28:0/1245001635 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbadb4d0 0x55b1cbaf0840 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fcafc0105c0 tx=0x7fcafc013c80 comp rx=0 tx=0).stop 2024-04-21T08:53:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.862+0000 7fcb11034740 1 -- 172.21.15.28:0/1245001635 shutdown_connections 2024-04-21T08:53:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.862+0000 7fcb11034740 1 --2- 172.21.15.28:0/1245001635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbae3d60 0x55b1cbae41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.862+0000 7fcb11034740 1 --2- 172.21.15.28:0/1245001635 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadbe80 0x55b1cbad4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.862+0000 7fcb11034740 1 --2- 172.21.15.28:0/1245001635 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbadb4d0 0x55b1cbaf0840 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.862+0000 7fcb11034740 1 -- 172.21.15.28:0/1245001635 >> 172.21.15.28:0/1245001635 conn(0x55b1cbad5ba0 msgr2=0x55b1cbad4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.864+0000 7fcb11034740 1 -- 172.21.15.28:0/1245001635 shutdown_connections 2024-04-21T08:53:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.865+0000 7fcb11034740 1 -- 172.21.15.28:0/1245001635 wait complete. 2024-04-21T08:53:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.865+0000 7fcb11034740 1 -- start start 2024-04-21T08:53:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.866+0000 7fcb11034740 1 Processor -- start 2024-04-21T08:53:41.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.866+0000 7fcb11034740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbadb4d0 0x55b1cbae8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.867+0000 7fcb11034740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadbe80 0x55b1cbae8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.867+0000 7fcb11034740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbae3d60 0x55b1cbae9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.867+0000 7fcb11034740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081b14b0 con 0x55b1cbadb4d0 2024-04-21T08:53:41.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.867+0000 7fcb11034740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081b2920 con 0x55b1cbae3d60 2024-04-21T08:53:41.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.867+0000 7fcb11034740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcb081b1b80 con 0x55b1cbadbe80 2024-04-21T08:53:41.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.867+0000 7fcb11034740 10 client.0 _ll_register_callbacks cb 0x7fcafc013bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:41.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.868+0000 7fcb07fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadbe80 0x55b1cbae8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.868+0000 7fcb07fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadbe80 0x55b1cbae8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:50384/0 (socket says 172.21.15.28:50384) 2024-04-21T08:53:41.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.868+0000 7fcb07fff640 1 -- 172.21.15.28:0/77969795 learned_addr learned my addr 172.21.15.28:0/77969795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:41.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.869+0000 7fcb07fff640 1 -- 172.21.15.28:0/77969795 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbae3d60 msgr2=0x55b1cbae9380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:41.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.869+0000 7fcb07fff640 1 --2- 172.21.15.28:0/77969795 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbae3d60 0x55b1cbae9380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.869+0000 7fcb07fff640 1 -- 172.21.15.28:0/77969795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbadb4d0 msgr2=0x55b1cbae8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:41.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.869+0000 7fcb07fff640 1 --2- 172.21.15.28:0/77969795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbadb4d0 0x55b1cbae8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.869+0000 7fcb07fff640 1 -- 172.21.15.28:0/77969795 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1cbb7c290 con 0x55b1cbadbe80 2024-04-21T08:53:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.875+0000 7fcb07fff640 1 --2- 172.21.15.28:0/77969795 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadbe80 0x55b1cbae8e40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8006010 tx=0x7fcaf8005a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef658284a5254f7f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.875+0000 7fcae77fe640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:53:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.875+0000 7fcae77fe640 1 -- 172.21.15.28:0/77969795 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8036040 con 0x55b1cbadbe80 2024-04-21T08:53:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.876+0000 7fcb11034740 1 -- 172.21.15.28:0/77969795 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcafc005e70 con 0x55b1cbadbe80 2024-04-21T08:53:41.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.876+0000 7fcae77fe640 1 -- 172.21.15.28:0/77969795 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fcaf80092b0 con 0x55b1cbadbe80 2024-04-21T08:53:41.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.876+0000 7fcae77fe640 1 -- 172.21.15.28:0/77969795 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcaf800b660 con 0x55b1cbadbe80 2024-04-21T08:53:41.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.876+0000 7fcb11034740 10 client.25849 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:41.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.876+0000 7fcb11034740 1 -- 172.21.15.28:0/77969795 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b1cbb7d5b0 con 0x55b1cbadbe80 2024-04-21T08:53:41.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.880+0000 7fcae77fe640 1 -- 172.21.15.28:0/77969795 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcafc005e70 con 0x55b1cbadbe80 2024-04-21T08:53:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.881+0000 7fcae77fe640 1 -- 172.21.15.28:0/77969795 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fcaf8032960 con 0x55b1cbadbe80 2024-04-21T08:53:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.883+0000 7fcb11034740 10 client.25849 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.883+0000 7fcb11034740 10 client.25849 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:53:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.883+0000 7fcb11034740 20 client.25849 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.883+0000 7fcb11034740 1 --2- 172.21.15.28:0/77969795 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b1cbb7d8f0 0x55b1cbb7fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.883+0000 7fcb11034740 4 client.25849 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:53:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.883+0000 7fcb11034740 1 -- 172.21.15.28:0/77969795 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b1cbae1f00 con 0x55b1cbb7d8f0 2024-04-21T08:53:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.891+0000 7fcb077fe640 1 --2- 172.21.15.28:0/77969795 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b1cbb7d8f0 0x55b1cbb7fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.895+0000 7fcb077fe640 1 --2- 172.21.15.28:0/77969795 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b1cbb7d8f0 0x55b1cbb7fd20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8642a4c9e99955a7 server_cookie=512cd0c3e5c9fe85 in_seq=0 out_seq=0 2024-04-21T08:53:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.895+0000 7fcae77fe640 10 client.25849 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:53:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.895+0000 7fcae77fe640 1 -- 172.21.15.28:0/77969795 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b1cbae1f00 con 0x55b1cbb7d8f0 2024-04-21T08:53:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.895+0000 7fcae77fe640 10 client.25849 handle_command_reply: tid=0 2024-04-21T08:53:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.895+0000 7fcb11034740 1 client.25849 shutdown 2024-04-21T08:53:41.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.904+0000 7fcb11034740 1 -- 172.21.15.28:0/77969795 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadbe80 msgr2=0x55b1cbae8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:41.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.904+0000 7fcb11034740 1 --2- 172.21.15.28:0/77969795 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadbe80 0x55b1cbae8e40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8006010 tx=0x7fcaf8005a90 comp rx=0 tx=0).stop 2024-04-21T08:53:41.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.905+0000 7fcb11034740 1 -- 172.21.15.28:0/77969795 shutdown_connections 2024-04-21T08:53:41.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.905+0000 7fcb11034740 1 --2- 172.21.15.28:0/77969795 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b1cbb7d8f0 0x55b1cbb7fd20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.905+0000 7fcb11034740 1 --2- 172.21.15.28:0/77969795 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b1cbae3d60 0x55b1cbae9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.905+0000 7fcb11034740 1 --2- 172.21.15.28:0/77969795 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b1cbadbe80 0x55b1cbae8e40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.905+0000 7fcb11034740 1 --2- 172.21.15.28:0/77969795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1cbadb4d0 0x55b1cbae8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.905+0000 7fcb11034740 1 -- 172.21.15.28:0/77969795 >> 172.21.15.28:0/77969795 conn(0x55b1cbad5ba0 msgr2=0x55b1cbadfad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:41.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.905+0000 7fcae77fe640 0 client.25849 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:53:41.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.910+0000 7fcb11034740 1 -- 172.21.15.28:0/77969795 shutdown_connections 2024-04-21T08:53:41.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.913+0000 7fcb11034740 1 -- 172.21.15.28:0/77969795 wait complete. 2024-04-21T08:53:41.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.915+0000 7fcb11034740 20 client.25849 trim_cache size 0 max 16384 2024-04-21T08:53:41.917 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:53:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.919+0000 7fcb0e443640 1 -- 172.21.15.28:0/1369979531 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcaf00a3f20 msgr2=0x7fcaf00a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.919+0000 7fcb0e443640 1 --2- 172.21.15.28:0/1369979531 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcaf00a3f20 0x7fcaf00a63e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcb00098430 tx=0x7fcb00097b90 comp rx=0 tx=0).stop 2024-04-21T08:53:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.919+0000 7fcb0e443640 1 -- 172.21.15.28:0/1369979531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb0810e980 msgr2=0x7fcb08114c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.919+0000 7fcb0e443640 1 --2- 172.21.15.28:0/1369979531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb0810e980 0x7fcb08114c60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcb00005fd0 tx=0x7fcb000047a0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.919+0000 7fcb0e443640 1 -- 172.21.15.28:0/1369979531 shutdown_connections 2024-04-21T08:53:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.919+0000 7fcb0e443640 1 --2- 172.21.15.28:0/1369979531 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcaf00a3f20 0x7fcaf00a63e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.919+0000 7fcb0e443640 1 --2- 172.21.15.28:0/1369979531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb0810e980 0x7fcb08114c60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.919+0000 7fcb0e443640 1 --2- 172.21.15.28:0/1369979531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb08109910 0x7fcb08114720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.919+0000 7fcb0e443640 1 --2- 172.21.15.28:0/1369979531 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcb08108f50 0x7fcb081141e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.919+0000 7fcb0e443640 1 -- 172.21.15.28:0/1369979531 >> 172.21.15.28:0/1369979531 conn(0x7fcb0806d4c0 msgr2=0x7fcb08073350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:41.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.921+0000 7fcb0e443640 1 -- 172.21.15.28:0/1369979531 shutdown_connections 2024-04-21T08:53:41.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:41.922+0000 7fcb0e443640 1 -- 172.21.15.28:0/1369979531 wait complete. 2024-04-21T08:53:41.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:53:42.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.146+0000 7fd73af73640 1 -- 172.21.15.28:0/828619806 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd72c0a4de0 msgr2=0x7fd72c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.146+0000 7fd73af73640 1 --2- 172.21.15.28:0/828619806 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd72c0a4de0 0x7fd72c0a51c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd730009860 tx=0x7fd73002bc10 comp rx=0 tx=0).stop 2024-04-21T08:53:42.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.146+0000 7fd73af73640 1 -- 172.21.15.28:0/828619806 shutdown_connections 2024-04-21T08:53:42.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.146+0000 7fd73af73640 1 --2- 172.21.15.28:0/828619806 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd72c0b7a70 0x7fd72c0b9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.146+0000 7fd73af73640 1 --2- 172.21.15.28:0/828619806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c0a5700 0x7fd72c0b7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.146+0000 7fd73af73640 1 --2- 172.21.15.28:0/828619806 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd72c0a4de0 0x7fd72c0a51c0 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd730009860 tx=0x7fd73002bc10 comp rx=0 tx=0).stop 2024-04-21T08:53:42.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.146+0000 7fd73af73640 1 -- 172.21.15.28:0/828619806 >> 172.21.15.28:0/828619806 conn(0x7fd72c01a640 msgr2=0x7fd72c01aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.147+0000 7fd73af73640 1 -- 172.21.15.28:0/828619806 shutdown_connections 2024-04-21T08:53:42.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.149+0000 7fd73af73640 1 -- 172.21.15.28:0/828619806 wait complete. 2024-04-21T08:53:42.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.150+0000 7fd73af73640 1 Processor -- start 2024-04-21T08:53:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.153+0000 7fd73af73640 1 -- start start 2024-04-21T08:53:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.153+0000 7fd73af73640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd72c0a5700 0x7fd72c143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.153+0000 7fd73af73640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd72c0b7a70 0x7fd72c144260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.153+0000 7fd73af73640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c1485e0 0x7fd72c148a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.153+0000 7fd73af73640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c0bd5b0 con 0x7fd72c1485e0 2024-04-21T08:53:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.153+0000 7fd73af73640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c0bd430 con 0x7fd72c0b7a70 2024-04-21T08:53:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.153+0000 7fd73af73640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c0bd730 con 0x7fd72c0a5700 2024-04-21T08:53:42.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.154+0000 7fd739f71640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd72c0a5700 0x7fd72c143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.154+0000 7fd739f71640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd72c0a5700 0x7fd72c143d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:50388/0 (socket says 172.21.15.28:50388) 2024-04-21T08:53:42.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.154+0000 7fd73a772640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c1485e0 0x7fd72c148a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.154+0000 7fd73a772640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c1485e0 0x7fd72c148a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45746/0 (socket says 172.21.15.28:45746) 2024-04-21T08:53:42.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.154+0000 7fd73a772640 1 -- 172.21.15.28:0/3724188917 learned_addr learned my addr 172.21.15.28:0/3724188917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:42.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.156+0000 7fd73a772640 1 -- 172.21.15.28:0/3724188917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd72c0a5700 msgr2=0x7fd72c143d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.156+0000 7fd73a772640 1 --2- 172.21.15.28:0/3724188917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd72c0a5700 0x7fd72c143d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.156+0000 7fd73a772640 1 -- 172.21.15.28:0/3724188917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd72c0b7a70 msgr2=0x7fd72c144260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.156+0000 7fd73a772640 1 --2- 172.21.15.28:0/3724188917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd72c0b7a70 0x7fd72c144260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.156+0000 7fd73a772640 1 -- 172.21.15.28:0/3724188917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c1490d0 con 0x7fd72c1485e0 2024-04-21T08:53:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.157+0000 7fd73a772640 1 --2- 172.21.15.28:0/3724188917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c1485e0 0x7fd72c148a90 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd734067840 tx=0x7fd734072a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.158+0000 7fd72affd640 1 -- 172.21.15.28:0/3724188917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd734079020 con 0x7fd72c1485e0 2024-04-21T08:53:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.158+0000 7fd73af73640 1 -- 172.21.15.28:0/3724188917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd72c1493c0 con 0x7fd72c1485e0 2024-04-21T08:53:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.158+0000 7fd73af73640 1 -- 172.21.15.28:0/3724188917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd72c150b30 con 0x7fd72c1485e0 2024-04-21T08:53:42.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.162+0000 7fd73edc7740 1 Processor -- start 2024-04-21T08:53:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.162+0000 7fd73edc7740 1 -- start start 2024-04-21T08:53:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.162+0000 7fd72affd640 1 -- 172.21.15.28:0/3724188917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd734074b30 con 0x7fd72c1485e0 2024-04-21T08:53:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.163+0000 7fd73edc7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef77f6d0 0x55acef77fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.163+0000 7fd73edc7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef780080 0x55acef788630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.163+0000 7fd73edc7740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef788c00 0x55acef78b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.163+0000 7fd73edc7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55acef6e41f0 con 0x55acef780080 2024-04-21T08:53:42.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.163+0000 7fd73edc7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55acef6814b0 con 0x55acef77f6d0 2024-04-21T08:53:42.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.163+0000 7fd73edc7740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55acef6e01b0 con 0x55acef788c00 2024-04-21T08:53:42.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.163+0000 7fd72affd640 1 -- 172.21.15.28:0/3724188917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7340759d0 con 0x7fd72c1485e0 2024-04-21T08:53:42.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.163+0000 7fd739770640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef788c00 0x55acef78b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.163+0000 7fd739770640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef788c00 0x55acef78b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:50394/0 (socket says 172.21.15.28:50394) 2024-04-21T08:53:42.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.163+0000 7fd739770640 1 -- 172.21.15.28:0/4050727866 learned_addr learned my addr 172.21.15.28:0/4050727866 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:42.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.163+0000 7fd739770640 1 --2- 172.21.15.28:0/4050727866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef77f6d0 0x55acef77fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.164+0000 7fd739770640 1 -- 172.21.15.28:0/4050727866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef77f6d0 msgr2=0x55acef77fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:42.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.164+0000 7fd739770640 1 --2- 172.21.15.28:0/4050727866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef77f6d0 0x55acef77fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.164+0000 7fd739770640 1 -- 172.21.15.28:0/4050727866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef780080 msgr2=0x55acef788630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:42.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.164+0000 7fd739770640 1 --2- 172.21.15.28:0/4050727866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef780080 0x55acef788630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.164+0000 7fd739770640 1 -- 172.21.15.28:0/4050727866 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acef6731b0 con 0x55acef788c00 2024-04-21T08:53:42.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.164+0000 7fd739770640 1 --2- 172.21.15.28:0/4050727866 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef788c00 0x55acef78b050 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd724013800 tx=0x7fd724009650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a34a4c69b896c7ae server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:42.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.166+0000 7fd71ffff640 1 -- 172.21.15.28:0/4050727866 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7240132d0 con 0x55acef788c00 2024-04-21T08:53:42.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.167+0000 7fd72affd640 1 -- 172.21.15.28:0/3724188917 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd73407c020 con 0x7fd72c1485e0 2024-04-21T08:53:42.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.168+0000 7fd72affd640 1 --2- 172.21.15.28:0/3724188917 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd7200a3d60 0x7fd7200a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.168+0000 7fd739f71640 1 --2- 172.21.15.28:0/3724188917 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd7200a3d60 0x7fd7200a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.168+0000 7fd72affd640 1 -- 172.21.15.28:0/3724188917 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fd7341347d0 con 0x7fd72c1485e0 2024-04-21T08:53:42.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.169+0000 7fd71ffff640 1 -- 172.21.15.28:0/4050727866 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd724013e40 con 0x55acef788c00 2024-04-21T08:53:42.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.169+0000 7fd71ffff640 1 -- 172.21.15.28:0/4050727866 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd724011970 con 0x55acef788c00 2024-04-21T08:53:42.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.170+0000 7fd73edc7740 1 -- 172.21.15.28:0/4050727866 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef788c00 msgr2=0x55acef78b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.170+0000 7fd73edc7740 1 --2- 172.21.15.28:0/4050727866 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef788c00 0x55acef78b050 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd724013800 tx=0x7fd724009650 comp rx=0 tx=0).stop 2024-04-21T08:53:42.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.170+0000 7fd73edc7740 1 -- 172.21.15.28:0/4050727866 shutdown_connections 2024-04-21T08:53:42.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.170+0000 7fd73edc7740 1 --2- 172.21.15.28:0/4050727866 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef788c00 0x55acef78b050 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd724013800 tx=0x7fd724009650 comp rx=0 tx=0).stop 2024-04-21T08:53:42.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.170+0000 7fd73edc7740 1 --2- 172.21.15.28:0/4050727866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef780080 0x55acef788630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.170+0000 7fd73edc7740 1 --2- 172.21.15.28:0/4050727866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef77f6d0 0x55acef77fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.170+0000 7fd73edc7740 1 -- 172.21.15.28:0/4050727866 >> 172.21.15.28:0/4050727866 conn(0x55acef779da0 msgr2=0x55acef77a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:42.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.170+0000 7fd739f71640 1 --2- 172.21.15.28:0/3724188917 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd7200a3d60 0x7fd7200a6220 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd730009780 tx=0x7fd730005a90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:42.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.171+0000 7fd73edc7740 1 -- 172.21.15.28:0/4050727866 shutdown_connections 2024-04-21T08:53:42.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.171+0000 7fd73edc7740 1 -- 172.21.15.28:0/4050727866 wait complete. 2024-04-21T08:53:42.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.171+0000 7fd73edc7740 1 -- start start 2024-04-21T08:53:42.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.172+0000 7fd73edc7740 1 Processor -- start 2024-04-21T08:53:42.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.173+0000 7fd73edc7740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef77f6d0 0x55acef7885c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.173+0000 7fd73edc7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef780080 0x55acef7858b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.173+0000 7fd73edc7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef785f50 0x55acef794f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.173+0000 7fd73edc7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55acef78bb10 con 0x55acef785f50 2024-04-21T08:53:42.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.173+0000 7fd73edc7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55acef69d500 con 0x55acef780080 2024-04-21T08:53:42.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.173+0000 7fd73edc7740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55acef78bc90 con 0x55acef77f6d0 2024-04-21T08:53:42.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.173+0000 7fd73edc7740 10 client.0 _ll_register_callbacks cb 0x55acef66bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:42.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.173+0000 7fd73a772640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef785f50 0x55acef794f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.173+0000 7fd73a772640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef785f50 0x55acef794f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45772/0 (socket says 172.21.15.28:45772) 2024-04-21T08:53:42.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.173+0000 7fd73a772640 1 -- 172.21.15.28:0/2627755975 learned_addr learned my addr 172.21.15.28:0/2627755975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:42.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.173+0000 7fd739f71640 1 --2- 172.21.15.28:0/2627755975 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef77f6d0 0x55acef7885c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.173+0000 7fd739770640 1 --2- 172.21.15.28:0/2627755975 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef780080 0x55acef7858b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.174+0000 7fd73a772640 1 -- 172.21.15.28:0/2627755975 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef77f6d0 msgr2=0x55acef7885c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:42.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.174+0000 7fd73a772640 1 --2- 172.21.15.28:0/2627755975 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef77f6d0 0x55acef7885c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.174+0000 7fd73a772640 1 -- 172.21.15.28:0/2627755975 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef780080 msgr2=0x55acef7858b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:42.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.174+0000 7fd73a772640 1 --2- 172.21.15.28:0/2627755975 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef780080 0x55acef7858b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.174+0000 7fd73a772640 1 -- 172.21.15.28:0/2627755975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acef3f63f0 con 0x55acef785f50 2024-04-21T08:53:42.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.174+0000 7fd73a772640 1 --2- 172.21.15.28:0/2627755975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef785f50 0x55acef794f80 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd7340fc860 tx=0x7fd7340ffd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f0cd951eceff19 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:42.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.175+0000 7fd71d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:42.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.175+0000 7fd71d7fa640 1 -- 172.21.15.28:0/2627755975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7341047b0 con 0x55acef785f50 2024-04-21T08:53:42.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.175+0000 7fd73edc7740 1 -- 172.21.15.28:0/2627755975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55acef683930 con 0x55acef785f50 2024-04-21T08:53:42.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.176+0000 7fd71d7fa640 1 -- 172.21.15.28:0/2627755975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fd734074ce0 con 0x55acef785f50 2024-04-21T08:53:42.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.176+0000 7fd71d7fa640 1 -- 172.21.15.28:0/2627755975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd734108fe0 con 0x55acef785f50 2024-04-21T08:53:42.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.176+0000 7fd73edc7740 10 client.16509 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:42.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.176+0000 7fd73edc7740 1 -- 172.21.15.28:0/2627755975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55acef4c07d0 con 0x55acef785f50 2024-04-21T08:53:42.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.178+0000 7fd71d7fa640 1 -- 172.21.15.28:0/2627755975 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55acef683930 con 0x55acef785f50 2024-04-21T08:53:42.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.178+0000 7fd71d7fa640 1 -- 172.21.15.28:0/2627755975 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd734075b70 con 0x55acef785f50 2024-04-21T08:53:42.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.178+0000 7fd73edc7740 10 client.16509 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:42.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.178+0000 7fd73edc7740 10 client.16509 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:53:42.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.178+0000 7fd73edc7740 20 client.16509 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:42.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.178+0000 7fd73edc7740 1 --2- 172.21.15.28:0/2627755975 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55acef799670 0x55acef79baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.179+0000 7fd73edc7740 4 client.16509 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:42.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.179+0000 7fd73edc7740 1 -- 172.21.15.28:0/2627755975 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55acef6f1c10 con 0x55acef799670 2024-04-21T08:53:42.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.179+0000 7fd739f71640 1 --2- 172.21.15.28:0/2627755975 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55acef799670 0x55acef79baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.182+0000 7fd739f71640 1 --2- 172.21.15.28:0/2627755975 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55acef799670 0x55acef79baf0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6710ae1e53ed60a2 server_cookie=974445e2a981c353 in_seq=0 out_seq=0 2024-04-21T08:53:42.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.182+0000 7fd71d7fa640 10 client.16509 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:53:42.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.183+0000 7fd71d7fa640 1 -- 172.21.15.28:0/2627755975 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55acef6f1c10 con 0x55acef799670 2024-04-21T08:53:42.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.183+0000 7fd71d7fa640 10 client.16509 handle_command_reply: tid=0 2024-04-21T08:53:42.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.183+0000 7fd73edc7740 1 client.16509 shutdown 2024-04-21T08:53:42.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.184+0000 7fd73edc7740 1 -- 172.21.15.28:0/2627755975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef785f50 msgr2=0x55acef794f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.184+0000 7fd73edc7740 1 --2- 172.21.15.28:0/2627755975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef785f50 0x55acef794f80 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd7340fc860 tx=0x7fd7340ffd60 comp rx=0 tx=0).stop 2024-04-21T08:53:42.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.184+0000 7fd73edc7740 1 -- 172.21.15.28:0/2627755975 shutdown_connections 2024-04-21T08:53:42.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.184+0000 7fd73edc7740 1 --2- 172.21.15.28:0/2627755975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef785f50 0x55acef794f80 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.184+0000 7fd73edc7740 1 --2- 172.21.15.28:0/2627755975 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef780080 0x55acef7858b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.184+0000 7fd73edc7740 1 --2- 172.21.15.28:0/2627755975 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55acef799670 0x55acef79baf0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.184+0000 7fd73edc7740 1 --2- 172.21.15.28:0/2627755975 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef77f6d0 0x55acef7885c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.184+0000 7fd73edc7740 1 -- 172.21.15.28:0/2627755975 >> 172.21.15.28:0/2627755975 conn(0x55acef779da0 msgr2=0x55acef77baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:42.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.184+0000 7fd71d7fa640 0 client.16509 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:53:42.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.185+0000 7fd73edc7740 1 -- 172.21.15.28:0/2627755975 shutdown_connections 2024-04-21T08:53:42.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.185+0000 7fd73edc7740 1 -- 172.21.15.28:0/2627755975 wait complete. 2024-04-21T08:53:42.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.185+0000 7fd73edc7740 20 client.16509 trim_cache size 0 max 16384 2024-04-21T08:53:42.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.214+0000 7fd73edc7740 1 Processor -- start 2024-04-21T08:53:42.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.215+0000 7fd73edc7740 1 -- start start 2024-04-21T08:53:42.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.215+0000 7fd73edc7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef77f6d0 0x55acef7880f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.215+0000 7fd73edc7740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef780080 0x55acef788630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.216+0000 7fd73edc7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef784600 0x55acef7a1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.216+0000 7fd73edc7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd718012830 con 0x55acef77f6d0 2024-04-21T08:53:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.216+0000 7fd739770640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef780080 0x55acef788630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.216+0000 7fd739770640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef780080 0x55acef788630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:50408/0 (socket says 172.21.15.28:50408) 2024-04-21T08:53:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.216+0000 7fd739770640 1 -- 172.21.15.28:0/1856209 learned_addr learned my addr 172.21.15.28:0/1856209 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.216+0000 7fd73a772640 1 --2- 172.21.15.28:0/1856209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef784600 0x55acef7a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.216+0000 7fd73edc7740 1 -- 172.21.15.28:0/1856209 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd718010100 con 0x55acef784600 2024-04-21T08:53:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.216+0000 7fd73edc7740 1 -- 172.21.15.28:0/1856209 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd71800df50 con 0x55acef780080 2024-04-21T08:53:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.216+0000 7fd739f71640 1 --2- 172.21.15.28:0/1856209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef77f6d0 0x55acef7880f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.217+0000 7fd73a772640 1 -- 172.21.15.28:0/1856209 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef780080 msgr2=0x55acef788630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.217+0000 7fd73a772640 1 --2- 172.21.15.28:0/1856209 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef780080 0x55acef788630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.217+0000 7fd73a772640 1 -- 172.21.15.28:0/1856209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef77f6d0 msgr2=0x55acef7880f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.217+0000 7fd73a772640 1 --2- 172.21.15.28:0/1856209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef77f6d0 0x55acef7880f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.217+0000 7fd73a772640 1 -- 172.21.15.28:0/1856209 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acef783e20 con 0x55acef784600 2024-04-21T08:53:42.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.217+0000 7fd73a772640 1 --2- 172.21.15.28:0/1856209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef784600 0x55acef7a1400 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd734050a90 tx=0x7fd7340fe910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b161d61256d61bfd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:42.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.218+0000 7fd71f7fe640 1 -- 172.21.15.28:0/1856209 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7340ffa60 con 0x55acef784600 2024-04-21T08:53:42.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.218+0000 7fd71f7fe640 1 -- 172.21.15.28:0/1856209 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd734134380 con 0x55acef784600 2024-04-21T08:53:42.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.219+0000 7fd71f7fe640 1 -- 172.21.15.28:0/1856209 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd734074ce0 con 0x55acef784600 2024-04-21T08:53:42.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.219+0000 7fd73edc7740 1 -- 172.21.15.28:0/1856209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef784600 msgr2=0x55acef7a1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.219+0000 7fd73edc7740 1 --2- 172.21.15.28:0/1856209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef784600 0x55acef7a1400 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd734050a90 tx=0x7fd7340fe910 comp rx=0 tx=0).stop 2024-04-21T08:53:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.222+0000 7fd73edc7740 1 -- 172.21.15.28:0/1856209 shutdown_connections 2024-04-21T08:53:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.222+0000 7fd73edc7740 1 --2- 172.21.15.28:0/1856209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef784600 0x55acef7a1400 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.222+0000 7fd73edc7740 1 --2- 172.21.15.28:0/1856209 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef780080 0x55acef788630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.222+0000 7fd73edc7740 1 --2- 172.21.15.28:0/1856209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef77f6d0 0x55acef7880f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.222+0000 7fd73edc7740 1 -- 172.21.15.28:0/1856209 >> 172.21.15.28:0/1856209 conn(0x55acef779da0 msgr2=0x55acef778e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:42.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.223+0000 7fd73edc7740 1 -- 172.21.15.28:0/1856209 shutdown_connections 2024-04-21T08:53:42.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.223+0000 7fd73edc7740 1 -- 172.21.15.28:0/1856209 wait complete. 2024-04-21T08:53:42.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.224+0000 7fd73edc7740 1 -- start start 2024-04-21T08:53:42.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.224+0000 7fd73edc7740 1 Processor -- start 2024-04-21T08:53:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.224+0000 7fd73edc7740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef77f6d0 0x55acef7913c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.224+0000 7fd73edc7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef780080 0x55acef782310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.224+0000 7fd73edc7740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef784600 0x55acef7828b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.224+0000 7fd73edc7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd71800e5c0 con 0x55acef780080 2024-04-21T08:53:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.224+0000 7fd73edc7740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd71800fa30 con 0x55acef784600 2024-04-21T08:53:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.224+0000 7fd73edc7740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd71800ec90 con 0x55acef77f6d0 2024-04-21T08:53:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.224+0000 7fd73edc7740 10 client.0 _ll_register_callbacks cb 0x7fd7341002a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.225+0000 7fd739f71640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef77f6d0 0x55acef7913c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.225+0000 7fd739f71640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef77f6d0 0x55acef7913c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:50416/0 (socket says 172.21.15.28:50416) 2024-04-21T08:53:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.225+0000 7fd739f71640 1 -- 172.21.15.28:0/4042097419 learned_addr learned my addr 172.21.15.28:0/4042097419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.225+0000 7fd73a772640 1 --2- 172.21.15.28:0/4042097419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef784600 0x55acef7828b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.225+0000 7fd739770640 1 --2- 172.21.15.28:0/4042097419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef780080 0x55acef782310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.226+0000 7fd739770640 1 -- 172.21.15.28:0/4042097419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef77f6d0 msgr2=0x55acef7913c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.226+0000 7fd739770640 1 --2- 172.21.15.28:0/4042097419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef77f6d0 0x55acef7913c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.226+0000 7fd739770640 1 -- 172.21.15.28:0/4042097419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef784600 msgr2=0x55acef7828b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.226+0000 7fd739770640 1 --2- 172.21.15.28:0/4042097419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef784600 0x55acef7828b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.226+0000 7fd739770640 1 -- 172.21.15.28:0/4042097419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acef7830e0 con 0x55acef780080 2024-04-21T08:53:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.226+0000 7fd739770640 1 --2- 172.21.15.28:0/4042097419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef780080 0x55acef782310 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd724009a00 tx=0x7fd7240130d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdef7041edac0fb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.227+0000 7fd71d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:42.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.227+0000 7fd71d7fa640 1 -- 172.21.15.28:0/4042097419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd724013450 con 0x55acef780080 2024-04-21T08:53:42.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.227+0000 7fd73edc7740 1 -- 172.21.15.28:0/4042097419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd734134380 con 0x55acef780080 2024-04-21T08:53:42.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.227+0000 7fd71d7fa640 1 -- 172.21.15.28:0/4042097419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fd724004040 con 0x55acef780080 2024-04-21T08:53:42.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.228+0000 7fd71d7fa640 1 -- 172.21.15.28:0/4042097419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd7240042a0 con 0x55acef780080 2024-04-21T08:53:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.229+0000 7fd73edc7740 10 client.16515 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.229+0000 7fd73edc7740 1 -- 172.21.15.28:0/4042097419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55acef7ffce0 con 0x55acef780080 2024-04-21T08:53:42.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.234+0000 7fd71d7fa640 1 -- 172.21.15.28:0/4042097419 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd734134380 con 0x55acef780080 2024-04-21T08:53:42.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.234+0000 7fd71d7fa640 1 -- 172.21.15.28:0/4042097419 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd724003700 con 0x55acef780080 2024-04-21T08:53:42.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.235+0000 7fd73edc7740 10 client.16515 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:42.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.235+0000 7fd73edc7740 10 client.16515 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:53:42.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.235+0000 7fd73edc7740 20 client.16515 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:42.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.235+0000 7fd73edc7740 1 --2- 172.21.15.28:0/4042097419 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55acef800110 0x55acef802500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.235+0000 7fd73edc7740 4 client.16515 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:53:42.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.235+0000 7fd73edc7740 1 -- 172.21.15.28:0/4042097419 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55acef6eba80 con 0x55acef800110 2024-04-21T08:53:42.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.236+0000 7fd739f71640 1 --2- 172.21.15.28:0/4042097419 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55acef800110 0x55acef802500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.244+0000 7fd739f71640 1 --2- 172.21.15.28:0/4042097419 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55acef800110 0x55acef802500 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f2a3e1657fb9e2dd server_cookie=31e7e2af1607ca13 in_seq=0 out_seq=0 2024-04-21T08:53:42.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.245+0000 7fd71d7fa640 10 client.16515 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:53:42.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.247+0000 7fd71d7fa640 1 -- 172.21.15.28:0/4042097419 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55acef6eba80 con 0x55acef800110 2024-04-21T08:53:42.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.247+0000 7fd71d7fa640 10 client.16515 handle_command_reply: tid=0 2024-04-21T08:53:42.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.247+0000 7fd73edc7740 1 client.16515 shutdown 2024-04-21T08:53:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.253+0000 7fd73edc7740 1 -- 172.21.15.28:0/4042097419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef780080 msgr2=0x55acef782310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.253+0000 7fd73edc7740 1 --2- 172.21.15.28:0/4042097419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef780080 0x55acef782310 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd724009a00 tx=0x7fd7240130d0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.255+0000 7fd73edc7740 1 -- 172.21.15.28:0/4042097419 shutdown_connections 2024-04-21T08:53:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.255+0000 7fd73edc7740 1 --2- 172.21.15.28:0/4042097419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55acef784600 0x55acef7828b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.255+0000 7fd73edc7740 1 --2- 172.21.15.28:0/4042097419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55acef780080 0x55acef782310 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.255+0000 7fd73edc7740 1 --2- 172.21.15.28:0/4042097419 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55acef800110 0x55acef802500 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.255+0000 7fd73edc7740 1 --2- 172.21.15.28:0/4042097419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55acef77f6d0 0x55acef7913c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.255+0000 7fd73edc7740 1 -- 172.21.15.28:0/4042097419 >> 172.21.15.28:0/4042097419 conn(0x55acef779da0 msgr2=0x55acef77b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.255+0000 7fd71d7fa640 0 client.16515 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:53:42.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.257+0000 7fd73edc7740 1 -- 172.21.15.28:0/4042097419 shutdown_connections 2024-04-21T08:53:42.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.258+0000 7fd73edc7740 1 -- 172.21.15.28:0/4042097419 wait complete. 2024-04-21T08:53:42.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.260+0000 7fd73edc7740 20 client.16515 trim_cache size 0 max 16384 2024-04-21T08:53:42.262 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:53:42.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.264+0000 7fd73af73640 1 -- 172.21.15.28:0/3724188917 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd7200a3d60 msgr2=0x7fd7200a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.264+0000 7fd73af73640 1 --2- 172.21.15.28:0/3724188917 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd7200a3d60 0x7fd7200a6220 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd730009780 tx=0x7fd730005a90 comp rx=0 tx=0).stop 2024-04-21T08:53:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.264+0000 7fd73af73640 1 -- 172.21.15.28:0/3724188917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c1485e0 msgr2=0x7fd72c148a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.264+0000 7fd73af73640 1 --2- 172.21.15.28:0/3724188917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c1485e0 0x7fd72c148a90 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd734067840 tx=0x7fd734072a20 comp rx=0 tx=0).stop 2024-04-21T08:53:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.265+0000 7fd73af73640 1 -- 172.21.15.28:0/3724188917 shutdown_connections 2024-04-21T08:53:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.265+0000 7fd73af73640 1 --2- 172.21.15.28:0/3724188917 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd7200a3d60 0x7fd7200a6220 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.265+0000 7fd73af73640 1 --2- 172.21.15.28:0/3724188917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c1485e0 0x7fd72c148a90 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.265+0000 7fd73af73640 1 --2- 172.21.15.28:0/3724188917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd72c0b7a70 0x7fd72c144260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.265+0000 7fd73af73640 1 --2- 172.21.15.28:0/3724188917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd72c0a5700 0x7fd72c143d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.265+0000 7fd73af73640 1 -- 172.21.15.28:0/3724188917 >> 172.21.15.28:0/3724188917 conn(0x7fd72c01a640 msgr2=0x7fd72c0b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.266+0000 7fd73af73640 1 -- 172.21.15.28:0/3724188917 shutdown_connections 2024-04-21T08:53:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.267+0000 7fd73af73640 1 -- 172.21.15.28:0/3724188917 wait complete. 2024-04-21T08:53:42.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:42 smithi028 ceph-mon[28003]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 56 MiB/s wr, 9.95k op/s 2024-04-21T08:53:42.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:42 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1876573367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:53:42.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:53:42.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:42 smithi171 ceph-mon[31358]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 56 MiB/s wr, 9.95k op/s 2024-04-21T08:53:42.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:42 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1876573367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:53:42.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.494+0000 7f14f91d5640 1 -- 172.21.15.28:0/3952180218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f41018e0 msgr2=0x7f14f4105da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.494+0000 7f14f91d5640 1 --2- 172.21.15.28:0/3952180218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f41018e0 0x7f14f4105da0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009ca0 tx=0x7f14ec02c210 comp rx=0 tx=0).stop 2024-04-21T08:53:42.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.495+0000 7f14f91d5640 1 -- 172.21.15.28:0/3952180218 shutdown_connections 2024-04-21T08:53:42.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.495+0000 7f14f91d5640 1 --2- 172.21.15.28:0/3952180218 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14f41062e0 0x7f14f4106770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.495+0000 7f14f91d5640 1 --2- 172.21.15.28:0/3952180218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f41018e0 0x7f14f4105da0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.495+0000 7f14f91d5640 1 --2- 172.21.15.28:0/3952180218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14f4100fc0 0x7f14f41013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.495+0000 7f14f91d5640 1 -- 172.21.15.28:0/3952180218 >> 172.21.15.28:0/3952180218 conn(0x7f14f4077f50 msgr2=0x7f14f4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:42.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.497+0000 7f14f91d5640 1 -- 172.21.15.28:0/3952180218 shutdown_connections 2024-04-21T08:53:42.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.497+0000 7f14f91d5640 1 -- 172.21.15.28:0/3952180218 wait complete. 2024-04-21T08:53:42.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.498+0000 7f14f91d5640 1 Processor -- start 2024-04-21T08:53:42.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.499+0000 7f14f91d5640 1 -- start start 2024-04-21T08:53:42.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.499+0000 7f14f91d5640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14f4100fc0 0x7f14f419ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.499+0000 7f14f91d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f41018e0 0x7f14f419f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.499+0000 7f14f91d5640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14f41062e0 0x7f14f41a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.499+0000 7f14f91d5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14f41187d0 con 0x7f14f41018e0 2024-04-21T08:53:42.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.499+0000 7f14f91d5640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f14f4118650 con 0x7f14f41062e0 2024-04-21T08:53:42.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.499+0000 7f14f91d5640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f14f4118950 con 0x7f14f4100fc0 2024-04-21T08:53:42.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.500+0000 7f14f259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f41018e0 0x7f14f419f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.500+0000 7f14f259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f41018e0 0x7f14f419f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45818/0 (socket says 172.21.15.28:45818) 2024-04-21T08:53:42.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.500+0000 7f14f359e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14f41062e0 0x7f14f41a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.500+0000 7f14f359e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14f41062e0 0x7f14f41a35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:33560/0 (socket says 172.21.15.28:33560) 2024-04-21T08:53:42.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.500+0000 7f14f359e640 1 -- 172.21.15.28:0/3662510618 learned_addr learned my addr 172.21.15.28:0/3662510618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:42.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.501+0000 7f14f259c640 1 -- 172.21.15.28:0/3662510618 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14f4100fc0 msgr2=0x7f14f419ed70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:53:42.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.501+0000 7f14f259c640 1 --2- 172.21.15.28:0/3662510618 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14f4100fc0 0x7f14f419ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.501+0000 7f14f259c640 1 -- 172.21.15.28:0/3662510618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14f41062e0 msgr2=0x7f14f41a35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.501+0000 7f14f259c640 1 --2- 172.21.15.28:0/3662510618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14f41062e0 0x7f14f41a35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.501+0000 7f14f259c640 1 -- 172.21.15.28:0/3662510618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f41a3ce0 con 0x7f14f41018e0 2024-04-21T08:53:42.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.502+0000 7f14f259c640 1 --2- 172.21.15.28:0/3662510618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f41018e0 0x7f14f419f2b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f14ec02c820 tx=0x7f14ec003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:42.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.507+0000 7f14ebfff640 1 -- 172.21.15.28:0/3662510618 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec039040 con 0x7f14f41018e0 2024-04-21T08:53:42.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.507+0000 7f14ebfff640 1 -- 172.21.15.28:0/3662510618 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14ec044070 con 0x7f14f41018e0 2024-04-21T08:53:42.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.507+0000 7f14ebfff640 1 -- 172.21.15.28:0/3662510618 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec032830 con 0x7f14f41018e0 2024-04-21T08:53:42.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.507+0000 7f14f91d5640 1 -- 172.21.15.28:0/3662510618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14f41a3f70 con 0x7f14f41018e0 2024-04-21T08:53:42.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.507+0000 7f14f91d5640 1 -- 172.21.15.28:0/3662510618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14f41a43e0 con 0x7f14f41018e0 2024-04-21T08:53:42.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.511+0000 7f14fbdca740 1 Processor -- start 2024-04-21T08:53:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.514+0000 7f14fbdca740 1 -- start start 2024-04-21T08:53:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.514+0000 7f14fbdca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b2084d0 0x55a56b2088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.514+0000 7f14fbdca740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b208e80 0x55a56b211430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.515+0000 7f14fbdca740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b211a00 0x55a56b213e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.515+0000 7f14fbdca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a56b16d1f0 con 0x55a56b2084d0 2024-04-21T08:53:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.515+0000 7f14ebfff640 1 -- 172.21.15.28:0/3662510618 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f14ec0329d0 con 0x7f14f41018e0 2024-04-21T08:53:42.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.515+0000 7f14fbdca740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55a56b10a4b0 con 0x55a56b208e80 2024-04-21T08:53:42.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.515+0000 7f14fbdca740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55a56b1691b0 con 0x55a56b211a00 2024-04-21T08:53:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.515+0000 7f14f2d9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b208e80 0x55a56b211430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.515+0000 7f14f2d9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b208e80 0x55a56b211430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:33572/0 (socket says 172.21.15.28:33572) 2024-04-21T08:53:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.515+0000 7f14f2d9d640 1 -- 172.21.15.28:0/2295785188 learned_addr learned my addr 172.21.15.28:0/2295785188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.516+0000 7f14ebfff640 1 --2- 172.21.15.28:0/3662510618 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f14e00a3f20 0x7f14e00a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.516+0000 7f14ebfff640 1 -- 172.21.15.28:0/3662510618 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f14ec0f2260 con 0x7f14f41018e0 2024-04-21T08:53:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.516+0000 7f14f359e640 1 --2- 172.21.15.28:0/2295785188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b2084d0 0x55a56b2088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.517+0000 7f14f259c640 1 --2- 172.21.15.28:0/2295785188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b211a00 0x55a56b213e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.518+0000 7f14f2d9d640 1 -- 172.21.15.28:0/2295785188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b211a00 msgr2=0x55a56b213e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.518+0000 7f14f2d9d640 1 --2- 172.21.15.28:0/2295785188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b211a00 0x55a56b213e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.518+0000 7f14f2d9d640 1 -- 172.21.15.28:0/2295785188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b2084d0 msgr2=0x55a56b2088b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.518+0000 7f14f2d9d640 1 --2- 172.21.15.28:0/2295785188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b2084d0 0x55a56b2088b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.518+0000 7f14f2d9d640 1 -- 172.21.15.28:0/2295785188 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a56b0fc1b0 con 0x55a56b208e80 2024-04-21T08:53:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.518+0000 7f14f359e640 1 --2- 172.21.15.28:0/3662510618 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f14e00a3f20 0x7f14e00a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.519+0000 7f14f2d9d640 1 --2- 172.21.15.28:0/2295785188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b208e80 0x55a56b211430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f14dc00aa20 tx=0x7f14dc00aee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bff5448fff078f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.519+0000 7f14e97fa640 1 -- 172.21.15.28:0/2295785188 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14dc00ce40 con 0x55a56b208e80 2024-04-21T08:53:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.521+0000 7f14e97fa640 1 -- 172.21.15.28:0/2295785188 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14dc00f900 con 0x55a56b208e80 2024-04-21T08:53:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.521+0000 7f14e97fa640 1 -- 172.21.15.28:0/2295785188 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14dc018980 con 0x55a56b208e80 2024-04-21T08:53:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.523+0000 7f14f359e640 1 --2- 172.21.15.28:0/3662510618 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f14e00a3f20 0x7f14e00a63e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f14e4005bc0 tx=0x7f14e4004730 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.524+0000 7f14fbdca740 1 -- 172.21.15.28:0/2295785188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b208e80 msgr2=0x55a56b211430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.524+0000 7f14fbdca740 1 --2- 172.21.15.28:0/2295785188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b208e80 0x55a56b211430 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f14dc00aa20 tx=0x7f14dc00aee0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.526+0000 7f14fbdca740 1 -- 172.21.15.28:0/2295785188 shutdown_connections 2024-04-21T08:53:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.526+0000 7f14fbdca740 1 --2- 172.21.15.28:0/2295785188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b211a00 0x55a56b213e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.526+0000 7f14fbdca740 1 --2- 172.21.15.28:0/2295785188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b208e80 0x55a56b211430 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.526+0000 7f14fbdca740 1 --2- 172.21.15.28:0/2295785188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b2084d0 0x55a56b2088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.526+0000 7f14fbdca740 1 -- 172.21.15.28:0/2295785188 >> 172.21.15.28:0/2295785188 conn(0x55a56b202ba0 msgr2=0x55a56b202f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.526+0000 7f14fbdca740 1 -- 172.21.15.28:0/2295785188 shutdown_connections 2024-04-21T08:53:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.526+0000 7f14fbdca740 1 -- 172.21.15.28:0/2295785188 wait complete. 2024-04-21T08:53:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.526+0000 7f14fbdca740 1 -- start start 2024-04-21T08:53:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.526+0000 7f14fbdca740 1 Processor -- start 2024-04-21T08:53:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.527+0000 7f14fbdca740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b2084d0 0x55a56b2112a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.527+0000 7f14fbdca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b208e80 0x55a56b20e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.527+0000 7f14fbdca740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b211a00 0x55a56b20eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.527+0000 7f14fbdca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a56b214970 con 0x55a56b208e80 2024-04-21T08:53:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.527+0000 7f14fbdca740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55a56b126500 con 0x55a56b211a00 2024-04-21T08:53:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.527+0000 7f14fbdca740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55a56b214af0 con 0x55a56b2084d0 2024-04-21T08:53:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.527+0000 7f14fbdca740 10 client.0 _ll_register_callbacks cb 0x55a56b0f4b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.527+0000 7f14f259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b208e80 0x55a56b20e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.528+0000 7f14f259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b208e80 0x55a56b20e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45848/0 (socket says 172.21.15.28:45848) 2024-04-21T08:53:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.528+0000 7f14f259c640 1 -- 172.21.15.28:0/3488549301 learned_addr learned my addr 172.21.15.28:0/3488549301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.528+0000 7f14f359e640 1 --2- 172.21.15.28:0/3488549301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b211a00 0x55a56b20eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.528+0000 7f14f2d9d640 1 --2- 172.21.15.28:0/3488549301 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b2084d0 0x55a56b2112a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.528+0000 7f14f259c640 1 -- 172.21.15.28:0/3488549301 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b2084d0 msgr2=0x55a56b2112a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.528+0000 7f14f259c640 1 --2- 172.21.15.28:0/3488549301 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b2084d0 0x55a56b2112a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.529+0000 7f14f259c640 1 -- 172.21.15.28:0/3488549301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b211a00 msgr2=0x55a56b20eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:42.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.529+0000 7f14f259c640 1 --2- 172.21.15.28:0/3488549301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b211a00 0x55a56b20eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.529+0000 7f14f259c640 1 -- 172.21.15.28:0/3488549301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a56ae7f3f0 con 0x55a56b208e80 2024-04-21T08:53:42.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.529+0000 7f14f2d9d640 1 --2- 172.21.15.28:0/3488549301 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b2084d0 0x55a56b2112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:53:42.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.529+0000 7f14f259c640 1 --2- 172.21.15.28:0/3488549301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b208e80 0x55a56b20e630 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f14ec000f80 tx=0x7f14ec0bda50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d083f929d13bf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:42.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.530+0000 7f14da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:42.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.530+0000 7f14f359e640 1 --2- 172.21.15.28:0/3488549301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b211a00 0x55a56b20eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:53:42.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.530+0000 7f14da7fc640 1 -- 172.21.15.28:0/3488549301 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0d7060 con 0x55a56b208e80 2024-04-21T08:53:42.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.530+0000 7f14fbdca740 1 -- 172.21.15.28:0/3488549301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14dc00f900 con 0x55a56b208e80 2024-04-21T08:53:42.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.530+0000 7f14da7fc640 1 -- 172.21.15.28:0/3488549301 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f14ec0be170 con 0x55a56b208e80 2024-04-21T08:53:42.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.531+0000 7f14da7fc640 1 -- 172.21.15.28:0/3488549301 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f14ec0be610 con 0x55a56b208e80 2024-04-21T08:53:42.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.533+0000 7f14fbdca740 10 client.16527 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:42.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.533+0000 7f14fbdca740 1 -- 172.21.15.28:0/3488549301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a56af497d0 con 0x55a56b208e80 2024-04-21T08:53:42.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.534+0000 7f14da7fc640 1 -- 172.21.15.28:0/3488549301 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14dc00f900 con 0x55a56b208e80 2024-04-21T08:53:42.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.534+0000 7f14da7fc640 1 -- 172.21.15.28:0/3488549301 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f14ec03c030 con 0x55a56b208e80 2024-04-21T08:53:42.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.534+0000 7f14fbdca740 10 client.16527 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:42.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.534+0000 7f14fbdca740 10 client.16527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:42.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.534+0000 7f14fbdca740 20 client.16527 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:42.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.534+0000 7f14fbdca740 1 --2- 172.21.15.28:0/3488549301 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a56b243e90 0x55a56b2462f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.534+0000 7f14fbdca740 4 client.16527 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:42.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.534+0000 7f14fbdca740 1 -- 172.21.15.28:0/3488549301 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a56b10c930 con 0x55a56b243e90 2024-04-21T08:53:42.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.537+0000 7f14f2d9d640 1 --2- 172.21.15.28:0/3488549301 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a56b243e90 0x55a56b2462f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.542+0000 7f14f2d9d640 1 --2- 172.21.15.28:0/3488549301 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a56b243e90 0x55a56b2462f0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec1feebd9869bf9e server_cookie=f760ea2619db3fc in_seq=0 out_seq=0 2024-04-21T08:53:42.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.542+0000 7f14da7fc640 10 client.16527 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:42.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.544+0000 7f14da7fc640 1 -- 172.21.15.28:0/3488549301 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a56b10c930 con 0x55a56b243e90 2024-04-21T08:53:42.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.544+0000 7f14da7fc640 10 client.16527 handle_command_reply: tid=0 2024-04-21T08:53:42.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.544+0000 7f14fbdca740 1 client.16527 shutdown 2024-04-21T08:53:42.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.544+0000 7f14fbdca740 1 -- 172.21.15.28:0/3488549301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b208e80 msgr2=0x55a56b20e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.544+0000 7f14fbdca740 1 --2- 172.21.15.28:0/3488549301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b208e80 0x55a56b20e630 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f14ec000f80 tx=0x7f14ec0bda50 comp rx=0 tx=0).stop 2024-04-21T08:53:42.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.545+0000 7f14fbdca740 1 -- 172.21.15.28:0/3488549301 shutdown_connections 2024-04-21T08:53:42.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.545+0000 7f14fbdca740 1 --2- 172.21.15.28:0/3488549301 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a56b243e90 0x55a56b2462f0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.545+0000 7f14fbdca740 1 --2- 172.21.15.28:0/3488549301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b211a00 0x55a56b20eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.545+0000 7f14fbdca740 1 --2- 172.21.15.28:0/3488549301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b208e80 0x55a56b20e630 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.545+0000 7f14fbdca740 1 --2- 172.21.15.28:0/3488549301 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b2084d0 0x55a56b2112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.545+0000 7f14fbdca740 1 -- 172.21.15.28:0/3488549301 >> 172.21.15.28:0/3488549301 conn(0x55a56b202ba0 msgr2=0x55a56b2048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:42.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.546+0000 7f14da7fc640 0 client.16527 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:42.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.546+0000 7f14fbdca740 1 -- 172.21.15.28:0/3488549301 shutdown_connections 2024-04-21T08:53:42.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.549+0000 7f14fbdca740 1 -- 172.21.15.28:0/3488549301 wait complete. 2024-04-21T08:53:42.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.550+0000 7f14fbdca740 20 client.16527 trim_cache size 0 max 16384 2024-04-21T08:53:42.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.577+0000 7f14fbdca740 1 Processor -- start 2024-04-21T08:53:42.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.578+0000 7f14fbdca740 1 -- start start 2024-04-21T08:53:42.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.578+0000 7f14fbdca740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b2084d0 0x55a56b21d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.578+0000 7f14fbdca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b211a00 0x55a56b204700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.578+0000 7f14fbdca740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b20ec60 0x55a56b210d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.578+0000 7f14fbdca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14d40127e0 con 0x55a56b211a00 2024-04-21T08:53:42.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.578+0000 7f14fbdca740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f14d40100b0 con 0x55a56b2084d0 2024-04-21T08:53:42.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.578+0000 7f14fbdca740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f14d400df00 con 0x55a56b20ec60 2024-04-21T08:53:42.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.579+0000 7f14f2d9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b2084d0 0x55a56b21d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.579+0000 7f14f2d9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b2084d0 0x55a56b21d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:33596/0 (socket says 172.21.15.28:33596) 2024-04-21T08:53:42.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.579+0000 7f14f2d9d640 1 -- 172.21.15.28:0/4288285277 learned_addr learned my addr 172.21.15.28:0/4288285277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:42.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.579+0000 7f14f359e640 1 --2- 172.21.15.28:0/4288285277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b20ec60 0x55a56b210d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.580+0000 7f14f2d9d640 1 -- 172.21.15.28:0/4288285277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b20ec60 msgr2=0x55a56b210d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:42.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.580+0000 7f14f2d9d640 1 --2- 172.21.15.28:0/4288285277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b20ec60 0x55a56b210d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.580+0000 7f14f2d9d640 1 -- 172.21.15.28:0/4288285277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b211a00 msgr2=0x55a56b204700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:42.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.580+0000 7f14f2d9d640 1 --2- 172.21.15.28:0/4288285277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b211a00 0x55a56b204700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.580+0000 7f14f2d9d640 1 -- 172.21.15.28:0/4288285277 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a56b203dd0 con 0x55a56b2084d0 2024-04-21T08:53:42.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.587+0000 7f14f2d9d640 1 --2- 172.21.15.28:0/4288285277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b2084d0 0x55a56b21d8c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f14dc00aa20 tx=0x7f14dc0185e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbf61d73463a7279 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:42.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.592+0000 7f14e8ff9640 1 -- 172.21.15.28:0/4288285277 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14dc02d020 con 0x55a56b2084d0 2024-04-21T08:53:42.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.592+0000 7f14e8ff9640 1 -- 172.21.15.28:0/4288285277 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14dc005580 con 0x55a56b2084d0 2024-04-21T08:53:42.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.592+0000 7f14e8ff9640 1 -- 172.21.15.28:0/4288285277 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14dc004c20 con 0x55a56b2084d0 2024-04-21T08:53:42.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.594+0000 7f14fbdca740 1 -- 172.21.15.28:0/4288285277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b2084d0 msgr2=0x55a56b21d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.594+0000 7f14fbdca740 1 --2- 172.21.15.28:0/4288285277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b2084d0 0x55a56b21d8c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f14dc00aa20 tx=0x7f14dc0185e0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.598+0000 7f14fbdca740 1 -- 172.21.15.28:0/4288285277 shutdown_connections 2024-04-21T08:53:42.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.598+0000 7f14fbdca740 1 --2- 172.21.15.28:0/4288285277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b20ec60 0x55a56b210d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.598+0000 7f14fbdca740 1 --2- 172.21.15.28:0/4288285277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b211a00 0x55a56b204700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.598+0000 7f14fbdca740 1 --2- 172.21.15.28:0/4288285277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b2084d0 0x55a56b21d8c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.598+0000 7f14fbdca740 1 -- 172.21.15.28:0/4288285277 >> 172.21.15.28:0/4288285277 conn(0x55a56b202ba0 msgr2=0x55a56b203ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:42.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.603+0000 7f14fbdca740 1 -- 172.21.15.28:0/4288285277 shutdown_connections 2024-04-21T08:53:42.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:42 smithi152 ceph-mon[31401]: pgmap v78: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 56 MiB/s wr, 9.95k op/s 2024-04-21T08:53:42.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:42 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1876573367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:53:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.605+0000 7f14fbdca740 1 -- 172.21.15.28:0/4288285277 wait complete. 2024-04-21T08:53:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.605+0000 7f14fbdca740 1 -- start start 2024-04-21T08:53:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.605+0000 7f14fbdca740 1 Processor -- start 2024-04-21T08:53:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.606+0000 7f14fbdca740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b2084d0 0x55a56b2a24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.606+0000 7f14fbdca740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b20ec60 0x55a56b2a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.606+0000 7f14fbdca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b211a00 0x55a56b2a2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.606+0000 7f14fbdca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14d400e570 con 0x55a56b211a00 2024-04-21T08:53:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.606+0000 7f14fbdca740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f14d400f9e0 con 0x55a56b20ec60 2024-04-21T08:53:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.606+0000 7f14fbdca740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f14d400ec40 con 0x55a56b2084d0 2024-04-21T08:53:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.606+0000 7f14fbdca740 10 client.0 _ll_register_callbacks cb 0x7f14ec009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.606+0000 7f14f359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b211a00 0x55a56b2a2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.606+0000 7f14f359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b211a00 0x55a56b2a2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45858/0 (socket says 172.21.15.28:45858) 2024-04-21T08:53:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.606+0000 7f14f359e640 1 -- 172.21.15.28:0/925460 learned_addr learned my addr 172.21.15.28:0/925460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.607+0000 7f14f2d9d640 1 --2- 172.21.15.28:0/925460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b2084d0 0x55a56b2a24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.607+0000 7f14f359e640 1 -- 172.21.15.28:0/925460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b2084d0 msgr2=0x55a56b2a24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.607+0000 7f14f359e640 1 --2- 172.21.15.28:0/925460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b2084d0 0x55a56b2a24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.607+0000 7f14f359e640 1 -- 172.21.15.28:0/925460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b20ec60 msgr2=0x55a56b2a2a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.607+0000 7f14f359e640 1 --2- 172.21.15.28:0/925460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b20ec60 0x55a56b2a2a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.607+0000 7f14f359e640 1 -- 172.21.15.28:0/925460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a56b218f70 con 0x55a56b211a00 2024-04-21T08:53:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.607+0000 7f14f2d9d640 1 --2- 172.21.15.28:0/925460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b2084d0 0x55a56b2a24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:53:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.607+0000 7f14f359e640 1 --2- 172.21.15.28:0/925460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b211a00 0x55a56b2a2f70 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f14e4013d80 tx=0x7f14e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24607f7b6521732a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.607+0000 7f14da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.608+0000 7f14da7fc640 1 -- 172.21.15.28:0/925460 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e4030060 con 0x55a56b211a00 2024-04-21T08:53:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.608+0000 7f14fbdca740 1 -- 172.21.15.28:0/925460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14dc005580 con 0x55a56b211a00 2024-04-21T08:53:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.608+0000 7f14da7fc640 1 -- 172.21.15.28:0/925460 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f14e4021b10 con 0x55a56b211a00 2024-04-21T08:53:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.609+0000 7f14da7fc640 1 -- 172.21.15.28:0/925460 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f14e4037050 con 0x55a56b211a00 2024-04-21T08:53:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.612+0000 7f14fbdca740 10 client.16530 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.612+0000 7f14fbdca740 1 -- 172.21.15.28:0/925460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a56b21a290 con 0x55a56b211a00 2024-04-21T08:53:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.615+0000 7f14da7fc640 1 -- 172.21.15.28:0/925460 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14dc005580 con 0x55a56b211a00 2024-04-21T08:53:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.615+0000 7f14da7fc640 1 -- 172.21.15.28:0/925460 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f14e402a030 con 0x55a56b211a00 2024-04-21T08:53:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.616+0000 7f14fbdca740 10 client.16530 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.616+0000 7f14fbdca740 10 client.16530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.616+0000 7f14fbdca740 20 client.16530 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.616+0000 7f14fbdca740 1 --2- 172.21.15.28:0/925460 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a56b2bb620 0x55a56b2bda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.616+0000 7f14fbdca740 4 client.16530 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:53:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.616+0000 7f14fbdca740 1 -- 172.21.15.28:0/925460 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a56b17ac10 con 0x55a56b2bb620 2024-04-21T08:53:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.616+0000 7f14f2d9d640 1 --2- 172.21.15.28:0/925460 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a56b2bb620 0x55a56b2bda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:42.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.621+0000 7f14da7fc640 10 client.16530 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:42.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.621+0000 7f14f2d9d640 1 --2- 172.21.15.28:0/925460 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a56b2bb620 0x55a56b2bda40 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f4d6bc82a892338 server_cookie=9fd9ace39867156b in_seq=0 out_seq=0 2024-04-21T08:53:42.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.622+0000 7f14da7fc640 1 -- 172.21.15.28:0/925460 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a56b17ac10 con 0x55a56b2bb620 2024-04-21T08:53:42.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.622+0000 7f14da7fc640 10 client.16530 handle_command_reply: tid=0 2024-04-21T08:53:42.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.622+0000 7f14fbdca740 1 client.16530 shutdown 2024-04-21T08:53:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.626+0000 7f14fbdca740 1 -- 172.21.15.28:0/925460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b211a00 msgr2=0x55a56b2a2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.626+0000 7f14fbdca740 1 --2- 172.21.15.28:0/925460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b211a00 0x55a56b2a2f70 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f14e4013d80 tx=0x7f14e40046c0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.627+0000 7f14fbdca740 1 -- 172.21.15.28:0/925460 shutdown_connections 2024-04-21T08:53:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.627+0000 7f14fbdca740 1 --2- 172.21.15.28:0/925460 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a56b2bb620 0x55a56b2bda40 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.627+0000 7f14fbdca740 1 --2- 172.21.15.28:0/925460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a56b211a00 0x55a56b2a2f70 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.627+0000 7f14fbdca740 1 --2- 172.21.15.28:0/925460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a56b20ec60 0x55a56b2a2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.627+0000 7f14fbdca740 1 --2- 172.21.15.28:0/925460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a56b2084d0 0x55a56b2a24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.627+0000 7f14fbdca740 1 -- 172.21.15.28:0/925460 >> 172.21.15.28:0/925460 conn(0x55a56b202ba0 msgr2=0x55a56b204860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.631+0000 7f14da7fc640 0 client.16530 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.631+0000 7f14fbdca740 1 -- 172.21.15.28:0/925460 shutdown_connections 2024-04-21T08:53:42.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.631+0000 7f14fbdca740 1 -- 172.21.15.28:0/925460 wait complete. 2024-04-21T08:53:42.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.632+0000 7f14fbdca740 20 client.16530 trim_cache size 0 max 16384 2024-04-21T08:53:42.635 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:53:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.636+0000 7f14f91d5640 1 -- 172.21.15.28:0/3662510618 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f14e00a3f20 msgr2=0x7f14e00a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.636+0000 7f14f91d5640 1 --2- 172.21.15.28:0/3662510618 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f14e00a3f20 0x7f14e00a63e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f14e4005bc0 tx=0x7f14e4004730 comp rx=0 tx=0).stop 2024-04-21T08:53:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.636+0000 7f14f91d5640 1 -- 172.21.15.28:0/3662510618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f41018e0 msgr2=0x7f14f419f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.636+0000 7f14f91d5640 1 --2- 172.21.15.28:0/3662510618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f41018e0 0x7f14f419f2b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f14ec02c820 tx=0x7f14ec003ba0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.637+0000 7f14f91d5640 1 -- 172.21.15.28:0/3662510618 shutdown_connections 2024-04-21T08:53:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.637+0000 7f14f91d5640 1 --2- 172.21.15.28:0/3662510618 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f14e00a3f20 0x7f14e00a63e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.638+0000 7f14f91d5640 1 --2- 172.21.15.28:0/3662510618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14f41062e0 0x7f14f41a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.638+0000 7f14f91d5640 1 --2- 172.21.15.28:0/3662510618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f41018e0 0x7f14f419f2b0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.638+0000 7f14f91d5640 1 --2- 172.21.15.28:0/3662510618 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f14f4100fc0 0x7f14f419ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.638+0000 7f14f91d5640 1 -- 172.21.15.28:0/3662510618 >> 172.21.15.28:0/3662510618 conn(0x7f14f4077f50 msgr2=0x7f14f4103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.639+0000 7f14f91d5640 1 -- 172.21.15.28:0/3662510618 shutdown_connections 2024-04-21T08:53:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:42.639+0000 7f14f91d5640 1 -- 172.21.15.28:0/3662510618 wait complete. 2024-04-21T08:53:43.659 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:53:43.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:53:43.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:53:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.852+0000 7fbf99888640 1 -- 172.21.15.28:0/4008210826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9406a8a0 msgr2=0x7fbf9406ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.852+0000 7fbf99888640 1 --2- 172.21.15.28:0/4008210826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9406a8a0 0x7fbf9406ad00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009900 tx=0x7fbf8402bc80 comp rx=0 tx=0).stop 2024-04-21T08:53:43.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.852+0000 7fbf99888640 1 -- 172.21.15.28:0/4008210826 shutdown_connections 2024-04-21T08:53:43.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.852+0000 7fbf99888640 1 --2- 172.21.15.28:0/4008210826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf94104980 0x7fbf94104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.852+0000 7fbf99888640 1 --2- 172.21.15.28:0/4008210826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9406a8a0 0x7fbf9406ad00 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.852+0000 7fbf99888640 1 --2- 172.21.15.28:0/4008210826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf94106e80 0x7fbf9406a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.852+0000 7fbf99888640 1 -- 172.21.15.28:0/4008210826 >> 172.21.15.28:0/4008210826 conn(0x7fbf94077f80 msgr2=0x7fbf94078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:43.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.853+0000 7fbf99888640 1 -- 172.21.15.28:0/4008210826 shutdown_connections 2024-04-21T08:53:43.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.853+0000 7fbf99888640 1 -- 172.21.15.28:0/4008210826 wait complete. 2024-04-21T08:53:43.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.854+0000 7fbf99888640 1 Processor -- start 2024-04-21T08:53:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.854+0000 7fbf99888640 1 -- start start 2024-04-21T08:53:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.854+0000 7fbf99888640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9406a8a0 0x7fbf94073840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.854+0000 7fbf99888640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf94104980 0x7fbf9406c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.854+0000 7fbf99888640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf94106e80 0x7fbf9406ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.854+0000 7fbf99888640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf941188c0 con 0x7fbf9406a8a0 2024-04-21T08:53:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.854+0000 7fbf99888640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbf94118740 con 0x7fbf94106e80 2024-04-21T08:53:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.854+0000 7fbf99888640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbf94118a40 con 0x7fbf94104980 2024-04-21T08:53:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.855+0000 7fbf937fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf94106e80 0x7fbf9406ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.855+0000 7fbf937fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf94106e80 0x7fbf9406ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:37422/0 (socket says 172.21.15.28:37422) 2024-04-21T08:53:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.855+0000 7fbf937fe640 1 -- 172.21.15.28:0/1878585581 learned_addr learned my addr 172.21.15.28:0/1878585581 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.855+0000 7fbf92ffd640 1 --2- 172.21.15.28:0/1878585581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9406a8a0 0x7fbf94073840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.856+0000 7fbf937fe640 1 -- 172.21.15.28:0/1878585581 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf94104980 msgr2=0x7fbf9406c930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:53:43.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.856+0000 7fbf937fe640 1 --2- 172.21.15.28:0/1878585581 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf94104980 0x7fbf9406c930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.856+0000 7fbf937fe640 1 -- 172.21.15.28:0/1878585581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9406a8a0 msgr2=0x7fbf94073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:43.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.856+0000 7fbf937fe640 1 --2- 172.21.15.28:0/1878585581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9406a8a0 0x7fbf94073840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.856+0000 7fbf937fe640 1 -- 172.21.15.28:0/1878585581 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf9406d5a0 con 0x7fbf94106e80 2024-04-21T08:53:43.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.856+0000 7fbf937fe640 1 --2- 172.21.15.28:0/1878585581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf94106e80 0x7fbf9406ce70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00b970 tx=0x7fbf8c00be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:43.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.857+0000 7fbf98886640 1 -- 172.21.15.28:0/1878585581 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c019070 con 0x7fbf94106e80 2024-04-21T08:53:43.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.857+0000 7fbf99888640 1 -- 172.21.15.28:0/1878585581 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf940713d0 con 0x7fbf94106e80 2024-04-21T08:53:43.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.857+0000 7fbf98886640 1 -- 172.21.15.28:0/1878585581 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf8c014440 con 0x7fbf94106e80 2024-04-21T08:53:43.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.857+0000 7fbf98886640 1 -- 172.21.15.28:0/1878585581 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c0075f0 con 0x7fbf94106e80 2024-04-21T08:53:43.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.857+0000 7fbf99888640 1 -- 172.21.15.28:0/1878585581 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf94071970 con 0x7fbf94106e80 2024-04-21T08:53:43.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.861+0000 7fbf9c47e740 1 Processor -- start 2024-04-21T08:53:43.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.863+0000 7fbf98886640 1 -- 172.21.15.28:0/1878585581 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbf8c01c070 con 0x7fbf94106e80 2024-04-21T08:53:43.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.863+0000 7fbf9c47e740 1 -- start start 2024-04-21T08:53:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.863+0000 7fbf9c47e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f188e0 0x564534f18cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.863+0000 7fbf9c47e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f19290 0x564534f21840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.864+0000 7fbf9c47e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f21e10 0x564534f24260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.864+0000 7fbf9c47e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564534e7cf00 con 0x564534f19290 2024-04-21T08:53:43.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.864+0000 7fbf9c47e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564534e1a590 con 0x564534f21e10 2024-04-21T08:53:43.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.864+0000 7fbf9c47e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564534e78ec0 con 0x564534f188e0 2024-04-21T08:53:43.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.864+0000 7fbf927fc640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f188e0 0x564534f18cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.864+0000 7fbf927fc640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f188e0 0x564534f18cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:38556/0 (socket says 172.21.15.28:38556) 2024-04-21T08:53:43.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.864+0000 7fbf927fc640 1 -- 172.21.15.28:0/2433308898 learned_addr learned my addr 172.21.15.28:0/2433308898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:43.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.864+0000 7fbf927fc640 1 --2- 172.21.15.28:0/2433308898 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f21e10 0x564534f24260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.864+0000 7fbf98886640 1 --2- 172.21.15.28:0/1878585581 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbf7c0a3e20 0x7fbf7c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.865+0000 7fbf927fc640 1 -- 172.21.15.28:0/2433308898 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f21e10 msgr2=0x564534f24260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:43.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.865+0000 7fbf927fc640 1 --2- 172.21.15.28:0/2433308898 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f21e10 0x564534f24260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.865+0000 7fbf927fc640 1 -- 172.21.15.28:0/2433308898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f19290 msgr2=0x564534f21840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:43.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.865+0000 7fbf927fc640 1 --2- 172.21.15.28:0/2433308898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f19290 0x564534f21840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.865+0000 7fbf927fc640 1 -- 172.21.15.28:0/2433308898 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564534c577e0 con 0x564534f188e0 2024-04-21T08:53:43.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.865+0000 7fbf927fc640 1 --2- 172.21.15.28:0/2433308898 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f188e0 0x564534f18cc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbf84005940 tx=0x7fbf8402e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eaec489cd17cf04 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:43.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.866+0000 7fbf937fe640 1 --2- 172.21.15.28:0/1878585581 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbf7c0a3e20 0x7fbf7c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.866+0000 7fbf79ffb640 1 -- 172.21.15.28:0/2433308898 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8402cc30 con 0x564534f188e0 2024-04-21T08:53:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.866+0000 7fbf79ffb640 1 -- 172.21.15.28:0/2433308898 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf84004020 con 0x564534f188e0 2024-04-21T08:53:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.866+0000 7fbf79ffb640 1 -- 172.21.15.28:0/2433308898 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf840049f0 con 0x564534f188e0 2024-04-21T08:53:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.866+0000 7fbf98886640 1 -- 172.21.15.28:0/1878585581 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbf8c0d0190 con 0x7fbf94106e80 2024-04-21T08:53:43.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.867+0000 7fbf9c47e740 1 -- 172.21.15.28:0/2433308898 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f188e0 msgr2=0x564534f18cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:43.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.867+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/2433308898 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f188e0 0x564534f18cc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbf84005940 tx=0x7fbf8402e040 comp rx=0 tx=0).stop 2024-04-21T08:53:43.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.867+0000 7fbf9c47e740 1 -- 172.21.15.28:0/2433308898 shutdown_connections 2024-04-21T08:53:43.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.867+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/2433308898 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f21e10 0x564534f24260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.867+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/2433308898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f19290 0x564534f21840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.867+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/2433308898 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f188e0 0x564534f18cc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.867+0000 7fbf9c47e740 1 -- 172.21.15.28:0/2433308898 >> 172.21.15.28:0/2433308898 conn(0x564534f12fb0 msgr2=0x564534f13390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:43.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.868+0000 7fbf937fe640 1 --2- 172.21.15.28:0/1878585581 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbf7c0a3e20 0x7fbf7c0a62e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c09bb40 tx=0x7fbf8c09bc10 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:43.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.871+0000 7fbf9c47e740 1 -- 172.21.15.28:0/2433308898 shutdown_connections 2024-04-21T08:53:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.873+0000 7fbf9c47e740 1 -- 172.21.15.28:0/2433308898 wait complete. 2024-04-21T08:53:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.873+0000 7fbf9c47e740 1 -- start start 2024-04-21T08:53:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.874+0000 7fbf9c47e740 1 Processor -- start 2024-04-21T08:53:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.874+0000 7fbf9c47e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f1eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.874+0000 7fbf9c47e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f19290 0x564534f1f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.874+0000 7fbf9c47e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f21e10 0x564534f1f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.874+0000 7fbf9c47e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564534f24d20 con 0x564534f188e0 2024-04-21T08:53:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.874+0000 7fbf9c47e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564534e36580 con 0x564534f19290 2024-04-21T08:53:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.874+0000 7fbf9c47e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564534f24ea0 con 0x564534f21e10 2024-04-21T08:53:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.874+0000 7fbf9c47e740 10 client.0 _ll_register_callbacks cb 0x564534e3c650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:43.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.875+0000 7fbf92ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f19290 0x564534f1f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.875+0000 7fbf927fc640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f21e10 0x564534f1f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.875+0000 7fbf927fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f1eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.875+0000 7fbf927fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f1eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35972/0 (socket says 172.21.15.28:35972) 2024-04-21T08:53:43.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.875+0000 7fbf927fc640 1 -- 172.21.15.28:0/1206182949 learned_addr learned my addr 172.21.15.28:0/1206182949 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:43.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.877+0000 7fbf927fc640 1 -- 172.21.15.28:0/1206182949 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f21e10 msgr2=0x564534f1f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:43.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.877+0000 7fbf927fc640 1 --2- 172.21.15.28:0/1206182949 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f21e10 0x564534f1f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.877+0000 7fbf927fc640 1 -- 172.21.15.28:0/1206182949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f19290 msgr2=0x564534f1f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:43.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.877+0000 7fbf927fc640 1 --2- 172.21.15.28:0/1206182949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f19290 0x564534f1f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.877+0000 7fbf927fc640 1 -- 172.21.15.28:0/1206182949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564534cb7f70 con 0x564534f188e0 2024-04-21T08:53:43.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.877+0000 7fbf927fc640 1 --2- 172.21.15.28:0/1206182949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f1eea0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fbf840048d0 tx=0x7fbf84005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c208b3028a215dbe server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:43.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.878+0000 7fbf72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:43.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.879+0000 7fbf72ffd640 1 -- 172.21.15.28:0/1206182949 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf84053070 con 0x564534f188e0 2024-04-21T08:53:43.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.879+0000 7fbf72ffd640 1 -- 172.21.15.28:0/1206182949 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbf8402cc60 con 0x564534f188e0 2024-04-21T08:53:43.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.879+0000 7fbf9c47e740 1 -- 172.21.15.28:0/1206182949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf84004020 con 0x564534f188e0 2024-04-21T08:53:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.879+0000 7fbf72ffd640 1 -- 172.21.15.28:0/1206182949 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbf84005200 con 0x564534f188e0 2024-04-21T08:53:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.881+0000 7fbf9c47e740 10 client.16539 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.881+0000 7fbf9c47e740 1 -- 172.21.15.28:0/1206182949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564534b4f170 con 0x564534f188e0 2024-04-21T08:53:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.883+0000 7fbf72ffd640 1 -- 172.21.15.28:0/1206182949 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf84004020 con 0x564534f188e0 2024-04-21T08:53:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.883+0000 7fbf72ffd640 1 -- 172.21.15.28:0/1206182949 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fbf84040040 con 0x564534f188e0 2024-04-21T08:53:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.883+0000 7fbf9c47e740 10 client.16539 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.883+0000 7fbf9c47e740 10 client.16539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.883+0000 7fbf9c47e740 20 client.16539 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:43.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.883+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/1206182949 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564534f32260 0x564534f34650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.883+0000 7fbf9c47e740 4 client.16539 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:43.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.883+0000 7fbf9c47e740 1 -- 172.21.15.28:0/1206182949 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564534e8a920 con 0x564534f32260 2024-04-21T08:53:43.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.885+0000 7fbf92ffd640 1 --2- 172.21.15.28:0/1206182949 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564534f32260 0x564534f34650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.886+0000 7fbf72ffd640 10 client.16539 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.886+0000 7fbf92ffd640 1 --2- 172.21.15.28:0/1206182949 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564534f32260 0x564534f34650 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfc729bcd9f0998a server_cookie=a1f43cf5cbcdf2c7 in_seq=0 out_seq=0 2024-04-21T08:53:43.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.892+0000 7fbf72ffd640 1 -- 172.21.15.28:0/1206182949 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564534e8a920 con 0x564534f32260 2024-04-21T08:53:43.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.892+0000 7fbf72ffd640 10 client.16539 handle_command_reply: tid=0 2024-04-21T08:53:43.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.892+0000 7fbf9c47e740 1 client.16539 shutdown 2024-04-21T08:53:43.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.893+0000 7fbf9c47e740 1 -- 172.21.15.28:0/1206182949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 msgr2=0x564534f1eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:43.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.893+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/1206182949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f1eea0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fbf840048d0 tx=0x7fbf84005fd0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.893+0000 7fbf9c47e740 1 -- 172.21.15.28:0/1206182949 shutdown_connections 2024-04-21T08:53:43.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.893+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/1206182949 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564534f32260 0x564534f34650 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.893+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/1206182949 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f21e10 0x564534f1f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.893+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/1206182949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f19290 0x564534f1f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.893+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/1206182949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f1eea0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.893+0000 7fbf9c47e740 1 -- 172.21.15.28:0/1206182949 >> 172.21.15.28:0/1206182949 conn(0x564534f12fb0 msgr2=0x564534f14d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:43.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.894+0000 7fbf72ffd640 0 client.16539 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:43.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.894+0000 7fbf9c47e740 1 -- 172.21.15.28:0/1206182949 shutdown_connections 2024-04-21T08:53:43.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.895+0000 7fbf9c47e740 1 -- 172.21.15.28:0/1206182949 wait complete. 2024-04-21T08:53:43.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.895+0000 7fbf9c47e740 20 client.16539 trim_cache size 0 max 16384 2024-04-21T08:53:43.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.935+0000 7fbf9c47e740 1 Processor -- start 2024-04-21T08:53:43.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.935+0000 7fbf9c47e740 1 -- start start 2024-04-21T08:53:43.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.935+0000 7fbf9c47e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f215d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.936+0000 7fbf9c47e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f19290 0x564534f1f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.936+0000 7fbf9c47e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f21e10 0x564534f1c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.936+0000 7fbf9c47e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf941c93d0 con 0x564534f188e0 2024-04-21T08:53:43.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.936+0000 7fbf9c47e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbf941c6ca0 con 0x564534f21e10 2024-04-21T08:53:43.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.936+0000 7fbf9c47e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbf940684e0 con 0x564534f19290 2024-04-21T08:53:43.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.936+0000 7fbf92ffd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f19290 0x564534f1f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.936+0000 7fbf927fc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f21e10 0x564534f1c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.936+0000 7fbf927fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f215d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.936+0000 7fbf927fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f215d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35974/0 (socket says 172.21.15.28:35974) 2024-04-21T08:53:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.936+0000 7fbf927fc640 1 -- 172.21.15.28:0/2109754823 learned_addr learned my addr 172.21.15.28:0/2109754823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.938+0000 7fbf927fc640 1 -- 172.21.15.28:0/2109754823 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f19290 msgr2=0x564534f1f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.938+0000 7fbf927fc640 1 --2- 172.21.15.28:0/2109754823 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f19290 0x564534f1f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.938+0000 7fbf927fc640 1 -- 172.21.15.28:0/2109754823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 msgr2=0x564534f215d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.938+0000 7fbf927fc640 1 --2- 172.21.15.28:0/2109754823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f215d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.938+0000 7fbf927fc640 1 -- 172.21.15.28:0/2109754823 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564534f1d3f0 con 0x564534f21e10 2024-04-21T08:53:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.939+0000 7fbf927fc640 1 --2- 172.21.15.28:0/2109754823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f21e10 0x564534f1c100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbf8402c7d0 tx=0x7fbf84005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a1237e85be07f1a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:43.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.941+0000 7fbf797fa640 1 -- 172.21.15.28:0/2109754823 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf84067070 con 0x564534f21e10 2024-04-21T08:53:43.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.941+0000 7fbf797fa640 1 -- 172.21.15.28:0/2109754823 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf84009520 con 0x564534f21e10 2024-04-21T08:53:43.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.941+0000 7fbf797fa640 1 -- 172.21.15.28:0/2109754823 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf84053030 con 0x564534f21e10 2024-04-21T08:53:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.943+0000 7fbf9c47e740 1 -- 172.21.15.28:0/2109754823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f21e10 msgr2=0x564534f1c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.943+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/2109754823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f21e10 0x564534f1c100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbf8402c7d0 tx=0x7fbf84005fd0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.943+0000 7fbf9c47e740 1 -- 172.21.15.28:0/2109754823 shutdown_connections 2024-04-21T08:53:43.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.943+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/2109754823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f21e10 0x564534f1c100 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.943+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/2109754823 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f19290 0x564534f1f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.943+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/2109754823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f215d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.943+0000 7fbf9c47e740 1 -- 172.21.15.28:0/2109754823 >> 172.21.15.28:0/2109754823 conn(0x564534f12fb0 msgr2=0x564534f1ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:43.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.943+0000 7fbf9c47e740 1 -- 172.21.15.28:0/2109754823 shutdown_connections 2024-04-21T08:53:43.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.943+0000 7fbf9c47e740 1 -- 172.21.15.28:0/2109754823 wait complete. 2024-04-21T08:53:43.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.943+0000 7fbf9c47e740 1 -- start start 2024-04-21T08:53:43.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.944+0000 7fbf9c47e740 1 Processor -- start 2024-04-21T08:53:43.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.944+0000 7fbf9c47e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f370e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.944+0000 7fbf9c47e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f19290 0x564534f37620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.944+0000 7fbf9c47e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f21e10 0x564534f35540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.944+0000 7fbf9c47e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9410ef60 con 0x564534f188e0 2024-04-21T08:53:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.944+0000 7fbf9c47e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbf941c65d0 con 0x564534f19290 2024-04-21T08:53:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.944+0000 7fbf9c47e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9410f630 con 0x564534f21e10 2024-04-21T08:53:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.944+0000 7fbf9c47e740 10 client.0 _ll_register_callbacks cb 0x7fbf8402c6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.945+0000 7fbf92ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f19290 0x564534f37620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.945+0000 7fbf92ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f19290 0x564534f37620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:37438/0 (socket says 172.21.15.28:37438) 2024-04-21T08:53:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.945+0000 7fbf92ffd640 1 -- 172.21.15.28:0/492604999 learned_addr learned my addr 172.21.15.28:0/492604999 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.946+0000 7fbf927fc640 1 --2- 172.21.15.28:0/492604999 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f21e10 0x564534f35540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.946+0000 7fbf927fc640 1 --2- 172.21.15.28:0/492604999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f370e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.947+0000 7fbf927fc640 1 -- 172.21.15.28:0/492604999 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f21e10 msgr2=0x564534f35540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:43.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.947+0000 7fbf927fc640 1 --2- 172.21.15.28:0/492604999 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f21e10 0x564534f35540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.947+0000 7fbf927fc640 1 -- 172.21.15.28:0/492604999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f19290 msgr2=0x564534f37620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:43.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.947+0000 7fbf927fc640 1 --2- 172.21.15.28:0/492604999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f19290 0x564534f37620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.947+0000 7fbf927fc640 1 -- 172.21.15.28:0/492604999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564534f35d40 con 0x564534f188e0 2024-04-21T08:53:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.949+0000 7fbf927fc640 1 --2- 172.21.15.28:0/492604999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f370e0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fbf8402c7d0 tx=0x7fbf8402bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93400b78568d64b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.949+0000 7fbf72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.949+0000 7fbf72ffd640 1 -- 172.21.15.28:0/492604999 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf84067030 con 0x564534f188e0 2024-04-21T08:53:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.950+0000 7fbf72ffd640 1 -- 172.21.15.28:0/492604999 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbf84004850 con 0x564534f188e0 2024-04-21T08:53:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.950+0000 7fbf9c47e740 1 -- 172.21.15.28:0/492604999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf84009520 con 0x564534f188e0 2024-04-21T08:53:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.951+0000 7fbf72ffd640 1 -- 172.21.15.28:0/492604999 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbf8402ccd0 con 0x564534f188e0 2024-04-21T08:53:43.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.952+0000 7fbf9c47e740 10 client.16545 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:43.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.952+0000 7fbf9c47e740 1 -- 172.21.15.28:0/492604999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564534f98fa0 con 0x564534f188e0 2024-04-21T08:53:43.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.953+0000 7fbf72ffd640 1 -- 172.21.15.28:0/492604999 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf84009520 con 0x564534f188e0 2024-04-21T08:53:43.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.954+0000 7fbf72ffd640 1 -- 172.21.15.28:0/492604999 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fbf8406d020 con 0x564534f188e0 2024-04-21T08:53:43.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.954+0000 7fbf9c47e740 10 client.16545 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:43.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.954+0000 7fbf9c47e740 10 client.16545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:43.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.954+0000 7fbf9c47e740 20 client.16545 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:43.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.954+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/492604999 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564534f994d0 0x564534f9b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:43.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.954+0000 7fbf9c47e740 4 client.16545 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:53:43.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.954+0000 7fbf9c47e740 1 -- 172.21.15.28:0/492604999 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564534e84790 con 0x564534f994d0 2024-04-21T08:53:43.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.955+0000 7fbf92ffd640 1 --2- 172.21.15.28:0/492604999 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564534f994d0 0x564534f9b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:43.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.956+0000 7fbf92ffd640 1 --2- 172.21.15.28:0/492604999 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564534f994d0 0x564534f9b8c0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9e92fce90e24fac server_cookie=c1b965155e544e45 in_seq=0 out_seq=0 2024-04-21T08:53:43.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.956+0000 7fbf72ffd640 10 client.16545 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.957+0000 7fbf72ffd640 1 -- 172.21.15.28:0/492604999 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564534e84790 con 0x564534f994d0 2024-04-21T08:53:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.957+0000 7fbf72ffd640 10 client.16545 handle_command_reply: tid=0 2024-04-21T08:53:43.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.959+0000 7fbf9c47e740 1 client.16545 shutdown 2024-04-21T08:53:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.962+0000 7fbf9c47e740 1 -- 172.21.15.28:0/492604999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 msgr2=0x564534f370e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.962+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/492604999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f370e0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fbf8402c7d0 tx=0x7fbf8402bc80 comp rx=0 tx=0).stop 2024-04-21T08:53:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.963+0000 7fbf9c47e740 1 -- 172.21.15.28:0/492604999 shutdown_connections 2024-04-21T08:53:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.963+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/492604999 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564534f994d0 0x564534f9b8c0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.963+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/492604999 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564534f21e10 0x564534f35540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.963+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/492604999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564534f19290 0x564534f37620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.963+0000 7fbf9c47e740 1 --2- 172.21.15.28:0/492604999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564534f188e0 0x564534f370e0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.963+0000 7fbf9c47e740 1 -- 172.21.15.28:0/492604999 >> 172.21.15.28:0/492604999 conn(0x564534f12fb0 msgr2=0x564534f14b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:43.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.964+0000 7fbf72ffd640 0 client.16545 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:43.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.964+0000 7fbf9c47e740 1 -- 172.21.15.28:0/492604999 shutdown_connections 2024-04-21T08:53:43.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.965+0000 7fbf9c47e740 1 -- 172.21.15.28:0/492604999 wait complete. 2024-04-21T08:53:43.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.965+0000 7fbf9c47e740 20 client.16545 trim_cache size 0 max 16384 2024-04-21T08:53:43.965 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:53:43.965 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:53:43.965 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "d29fd45f-62ab-465d-a301-0ac2e916a57c", 2024-04-21T08:53:43.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:53:43.966 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:53:43.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.966+0000 7fbf99888640 1 -- 172.21.15.28:0/1878585581 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbf7c0a3e20 msgr2=0x7fbf7c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:43.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.966+0000 7fbf99888640 1 --2- 172.21.15.28:0/1878585581 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbf7c0a3e20 0x7fbf7c0a62e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c09bb40 tx=0x7fbf8c09bc10 comp rx=0 tx=0).stop 2024-04-21T08:53:43.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.966+0000 7fbf99888640 1 -- 172.21.15.28:0/1878585581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf94106e80 msgr2=0x7fbf9406ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:43.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.966+0000 7fbf99888640 1 --2- 172.21.15.28:0/1878585581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf94106e80 0x7fbf9406ce70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00b970 tx=0x7fbf8c00be30 comp rx=0 tx=0).stop 2024-04-21T08:53:43.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.968+0000 7fbf99888640 1 -- 172.21.15.28:0/1878585581 shutdown_connections 2024-04-21T08:53:43.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.968+0000 7fbf99888640 1 --2- 172.21.15.28:0/1878585581 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbf7c0a3e20 0x7fbf7c0a62e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.968+0000 7fbf99888640 1 --2- 172.21.15.28:0/1878585581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbf94106e80 0x7fbf9406ce70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.968+0000 7fbf99888640 1 --2- 172.21.15.28:0/1878585581 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbf94104980 0x7fbf9406c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.968+0000 7fbf99888640 1 --2- 172.21.15.28:0/1878585581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9406a8a0 0x7fbf94073840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:43.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.968+0000 7fbf99888640 1 -- 172.21.15.28:0/1878585581 >> 172.21.15.28:0/1878585581 conn(0x7fbf94077f80 msgr2=0x7fbf940fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:43.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.969+0000 7fbf99888640 1 -- 172.21.15.28:0/1878585581 shutdown_connections 2024-04-21T08:53:43.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:43.969+0000 7fbf99888640 1 -- 172.21.15.28:0/1878585581 wait complete. 2024-04-21T08:53:43.990 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:53:43.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:53:44.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.183+0000 7feb9a25c640 1 -- 172.21.15.28:0/3343460720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feb9410e620 msgr2=0x7feb9410ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:44.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.183+0000 7feb9a25c640 1 --2- 172.21.15.28:0/3343460720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feb9410e620 0x7feb9410ea00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feb7c009890 tx=0x7feb7c02bbb0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.184+0000 7feb9a25c640 1 -- 172.21.15.28:0/3343460720 shutdown_connections 2024-04-21T08:53:44.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.184+0000 7feb9a25c640 1 --2- 172.21.15.28:0/3343460720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb94108f20 0x7feb941153e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.184+0000 7feb9a25c640 1 --2- 172.21.15.28:0/3343460720 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb94108560 0x7feb941089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.184+0000 7feb9a25c640 1 --2- 172.21.15.28:0/3343460720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feb9410e620 0x7feb9410ea00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.184+0000 7feb9a25c640 1 -- 172.21.15.28:0/3343460720 >> 172.21.15.28:0/3343460720 conn(0x7feb94077d80 msgr2=0x7feb94106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:44.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.186+0000 7feb9a25c640 1 -- 172.21.15.28:0/3343460720 shutdown_connections 2024-04-21T08:53:44.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.186+0000 7feb9a25c640 1 -- 172.21.15.28:0/3343460720 wait complete. 2024-04-21T08:53:44.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.187+0000 7feb9a25c640 1 Processor -- start 2024-04-21T08:53:44.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.188+0000 7feb9a25c640 1 -- start start 2024-04-21T08:53:44.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.188+0000 7feb9a25c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb94108560 0x7feb94073740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.188+0000 7feb9a25c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feb94108f20 0x7feb9406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.188+0000 7feb9a25c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9410e620 0x7feb9406cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.188+0000 7feb9a25c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feb94118830 con 0x7feb9410e620 2024-04-21T08:53:44.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.188+0000 7feb9a25c640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7feb941186b0 con 0x7feb94108f20 2024-04-21T08:53:44.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.188+0000 7feb9a25c640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7feb941189b0 con 0x7feb94108560 2024-04-21T08:53:44.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.189+0000 7feb92ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feb94108f20 0x7feb9406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.189+0000 7feb937fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb94108560 0x7feb94073740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.190+0000 7feb92ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feb94108f20 0x7feb9406c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:37460/0 (socket says 172.21.15.28:37460) 2024-04-21T08:53:44.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.190+0000 7feb92ffd640 1 -- 172.21.15.28:0/1153617914 learned_addr learned my addr 172.21.15.28:0/1153617914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:44.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.190+0000 7feb93fff640 1 --2- 172.21.15.28:0/1153617914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9410e620 0x7feb9406cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.191+0000 7feb92ffd640 1 -- 172.21.15.28:0/1153617914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb94108560 msgr2=0x7feb94073740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.191+0000 7feb92ffd640 1 --2- 172.21.15.28:0/1153617914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb94108560 0x7feb94073740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.191+0000 7feb92ffd640 1 -- 172.21.15.28:0/1153617914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9410e620 msgr2=0x7feb9406cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.191+0000 7feb92ffd640 1 --2- 172.21.15.28:0/1153617914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9410e620 0x7feb9406cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.191+0000 7feb92ffd640 1 -- 172.21.15.28:0/1153617914 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9406d2c0 con 0x7feb94108f20 2024-04-21T08:53:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.192+0000 7feb92ffd640 1 --2- 172.21.15.28:0/1153617914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feb94108f20 0x7feb9406c7c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7feb8400ba50 tx=0x7feb8400bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.192+0000 7feb90ff9640 1 -- 172.21.15.28:0/1153617914 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb84019070 con 0x7feb94108f20 2024-04-21T08:53:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.192+0000 7feb90ff9640 1 -- 172.21.15.28:0/1153617914 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb8400cbe0 con 0x7feb94108f20 2024-04-21T08:53:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.192+0000 7feb9a25c640 1 -- 172.21.15.28:0/1153617914 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb9406d5b0 con 0x7feb94108f20 2024-04-21T08:53:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.192+0000 7feb90ff9640 1 -- 172.21.15.28:0/1153617914 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb840075d0 con 0x7feb94108f20 2024-04-21T08:53:44.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.192+0000 7feb9a25c640 1 -- 172.21.15.28:0/1153617914 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb94071590 con 0x7feb94108f20 2024-04-21T08:53:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.196+0000 7feb9ce50740 1 Processor -- start 2024-04-21T08:53:44.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.196+0000 7feb9ce50740 1 -- start start 2024-04-21T08:53:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.196+0000 7feb9ce50740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a0c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.196+0000 7feb9ce50740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a0cea0 0x560028a15540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.196+0000 7feb9ce50740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a15a80 0x560028a17e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.196+0000 7feb9ce50740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5600289711e0 con 0x560028a15a80 2024-04-21T08:53:44.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.196+0000 7feb9ce50740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56002890e390 con 0x560028a0cea0 2024-04-21T08:53:44.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.196+0000 7feb9ce50740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56002896d1a0 con 0x560028a0c580 2024-04-21T08:53:44.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.197+0000 7feb93fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a0c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.197+0000 7feb93fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a0c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:38602/0 (socket says 172.21.15.28:38602) 2024-04-21T08:53:44.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.197+0000 7feb93fff640 1 -- 172.21.15.28:0/3673617351 learned_addr learned my addr 172.21.15.28:0/3673617351 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:44.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.197+0000 7feb92ffd640 1 --2- 172.21.15.28:0/3673617351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a15a80 0x560028a17e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.197+0000 7feb90ff9640 1 -- 172.21.15.28:0/1153617914 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7feb8401c070 con 0x7feb94108f20 2024-04-21T08:53:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.198+0000 7feb937fe640 1 --2- 172.21.15.28:0/3673617351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a0cea0 0x560028a15540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.198+0000 7feb93fff640 1 -- 172.21.15.28:0/3673617351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a0cea0 msgr2=0x560028a15540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.198+0000 7feb93fff640 1 --2- 172.21.15.28:0/3673617351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a0cea0 0x560028a15540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.198+0000 7feb93fff640 1 -- 172.21.15.28:0/3673617351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a15a80 msgr2=0x560028a17e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.198+0000 7feb93fff640 1 --2- 172.21.15.28:0/3673617351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a15a80 0x560028a17e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.198+0000 7feb93fff640 1 -- 172.21.15.28:0/3673617351 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56002874d7d0 con 0x560028a0c580 2024-04-21T08:53:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.198+0000 7feb937fe640 1 --2- 172.21.15.28:0/3673617351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a0cea0 0x560028a15540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:53:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.199+0000 7feb93fff640 1 --2- 172.21.15.28:0/3673617351 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a0c960 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009c50 tx=0x7feb8c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de95dacf864d24ef server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.199+0000 7feb89ffb640 1 -- 172.21.15.28:0/3673617351 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c004020 con 0x560028a0c580 2024-04-21T08:53:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.199+0000 7feb90ff9640 1 --2- 172.21.15.28:0/1153617914 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7feb800a3e20 0x7feb800a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.199+0000 7feb89ffb640 1 -- 172.21.15.28:0/3673617351 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb8c0041c0 con 0x560028a0c580 2024-04-21T08:53:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.199+0000 7feb89ffb640 1 -- 172.21.15.28:0/3673617351 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c0197a0 con 0x560028a0c580 2024-04-21T08:53:44.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.200+0000 7feb90ff9640 1 -- 172.21.15.28:0/1153617914 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7feb840d0090 con 0x7feb94108f20 2024-04-21T08:53:44.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.200+0000 7feb937fe640 1 --2- 172.21.15.28:0/1153617914 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7feb800a3e20 0x7feb800a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.200+0000 7feb9ce50740 1 -- 172.21.15.28:0/3673617351 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 msgr2=0x560028a0c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.200+0000 7feb9ce50740 1 --2- 172.21.15.28:0/3673617351 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a0c960 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009c50 tx=0x7feb8c011810 comp rx=0 tx=0).stop 2024-04-21T08:53:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.200+0000 7feb9ce50740 1 -- 172.21.15.28:0/3673617351 shutdown_connections 2024-04-21T08:53:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.200+0000 7feb9ce50740 1 --2- 172.21.15.28:0/3673617351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a15a80 0x560028a17e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.200+0000 7feb9ce50740 1 --2- 172.21.15.28:0/3673617351 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a0cea0 0x560028a15540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.200+0000 7feb9ce50740 1 --2- 172.21.15.28:0/3673617351 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a0c960 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.200+0000 7feb9ce50740 1 -- 172.21.15.28:0/3673617351 >> 172.21.15.28:0/3673617351 conn(0x560028a06cb0 msgr2=0x560028a07090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.201+0000 7feb9ce50740 1 -- 172.21.15.28:0/3673617351 shutdown_connections 2024-04-21T08:53:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.202+0000 7feb9ce50740 1 -- 172.21.15.28:0/3673617351 wait complete. 2024-04-21T08:53:44.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.202+0000 7feb937fe640 1 --2- 172.21.15.28:0/1153617914 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7feb800a3e20 0x7feb800a62e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7feb7c02c1c0 tx=0x7feb7c034e00 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:44.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.202+0000 7feb9ce50740 1 -- start start 2024-04-21T08:53:44.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.203+0000 7feb9ce50740 1 Processor -- start 2024-04-21T08:53:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.203+0000 7feb9ce50740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a12a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.203+0000 7feb9ce50740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 0x560028a12f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.203+0000 7feb9ce50740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a15a80 0x560028a134a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.203+0000 7feb9ce50740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560028a18870 con 0x560028a0cea0 2024-04-21T08:53:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.204+0000 7feb9ce50740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56002892a2f0 con 0x560028a15a80 2024-04-21T08:53:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.204+0000 7feb937fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a12a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.204+0000 7feb937fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a12a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:38610/0 (socket says 172.21.15.28:38610) 2024-04-21T08:53:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.204+0000 7feb937fe640 1 -- 172.21.15.28:0/3939305809 learned_addr learned my addr 172.21.15.28:0/3939305809 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.204+0000 7feb93fff640 1 --2- 172.21.15.28:0/3939305809 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a15a80 0x560028a134a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.204+0000 7feb9ce50740 1 -- 172.21.15.28:0/3939305809 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x560028a189f0 con 0x560028a0c580 2024-04-21T08:53:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.204+0000 7feb9ce50740 10 client.0 _ll_register_callbacks cb 0x560028937580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.204+0000 7feb937fe640 1 -- 172.21.15.28:0/3939305809 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a15a80 msgr2=0x560028a134a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.204+0000 7feb937fe640 1 --2- 172.21.15.28:0/3939305809 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a15a80 0x560028a134a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.204+0000 7feb937fe640 1 -- 172.21.15.28:0/3939305809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 msgr2=0x560028a12f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:44.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.204+0000 7feb937fe640 1 --2- 172.21.15.28:0/3939305809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 0x560028a12f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.204+0000 7feb937fe640 1 -- 172.21.15.28:0/3939305809 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600286833f0 con 0x560028a0c580 2024-04-21T08:53:44.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.206+0000 7feb937fe640 1 --2- 172.21.15.28:0/3939305809 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a12a20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feb7c036000 tx=0x7feb7c02bbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e5cad56137f7a6e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:44.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.206+0000 7feb72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:53:44.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.209+0000 7feb72ffd640 1 -- 172.21.15.28:0/3939305809 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb7c04f020 con 0x560028a0c580 2024-04-21T08:53:44.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.209+0000 7feb9ce50740 1 -- 172.21.15.28:0/3939305809 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb8c0041c0 con 0x560028a0c580 2024-04-21T08:53:44.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.209+0000 7feb72ffd640 1 -- 172.21.15.28:0/3939305809 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7feb7c005040 con 0x560028a0c580 2024-04-21T08:53:44.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.210+0000 7feb72ffd640 1 -- 172.21.15.28:0/3939305809 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7feb7c05a050 con 0x560028a0c580 2024-04-21T08:53:44.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.210+0000 7feb9ce50740 10 client.25894 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:44.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.211+0000 7feb9ce50740 1 -- 172.21.15.28:0/3939305809 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5600286430f0 con 0x560028a0c580 2024-04-21T08:53:44.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.216+0000 7feb72ffd640 1 -- 172.21.15.28:0/3939305809 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb8c0041c0 con 0x560028a0c580 2024-04-21T08:53:44.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.216+0000 7feb72ffd640 1 -- 172.21.15.28:0/3939305809 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7feb7c0047d0 con 0x560028a0c580 2024-04-21T08:53:44.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.216+0000 7feb9ce50740 10 client.25894 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:44.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.216+0000 7feb9ce50740 10 client.25894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:44.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.216+0000 7feb9ce50740 20 client.25894 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:44.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.216+0000 7feb9ce50740 1 --2- 172.21.15.28:0/3939305809 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560028a25760 0x560028a27be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.216+0000 7feb9ce50740 4 client.25894 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:44.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.216+0000 7feb9ce50740 1 -- 172.21.15.28:0/3939305809 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56002897ec00 con 0x560028a25760 2024-04-21T08:53:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.220+0000 7feb92ffd640 1 --2- 172.21.15.28:0/3939305809 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560028a25760 0x560028a27be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.225+0000 7feb92ffd640 1 --2- 172.21.15.28:0/3939305809 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560028a25760 0x560028a27be0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9e64747d30de2d0 server_cookie=9e0b3381ea9ad73c in_seq=0 out_seq=0 2024-04-21T08:53:44.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.225+0000 7feb72ffd640 10 client.25894 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.232+0000 7feb72ffd640 1 -- 172.21.15.28:0/3939305809 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56002897ec00 con 0x560028a25760 2024-04-21T08:53:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.232+0000 7feb72ffd640 10 client.25894 handle_command_reply: tid=0 2024-04-21T08:53:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.232+0000 7feb9ce50740 1 client.25894 shutdown 2024-04-21T08:53:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.233+0000 7feb9ce50740 1 -- 172.21.15.28:0/3939305809 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 msgr2=0x560028a12a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.233+0000 7feb9ce50740 1 --2- 172.21.15.28:0/3939305809 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a12a20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7feb7c036000 tx=0x7feb7c02bbb0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.237+0000 7feb9ce50740 1 -- 172.21.15.28:0/3939305809 shutdown_connections 2024-04-21T08:53:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.237+0000 7feb9ce50740 1 --2- 172.21.15.28:0/3939305809 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560028a25760 0x560028a27be0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.237+0000 7feb9ce50740 1 --2- 172.21.15.28:0/3939305809 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a15a80 0x560028a134a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.237+0000 7feb9ce50740 1 --2- 172.21.15.28:0/3939305809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 0x560028a12f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.237+0000 7feb9ce50740 1 --2- 172.21.15.28:0/3939305809 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a12a20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.237+0000 7feb9ce50740 1 -- 172.21.15.28:0/3939305809 >> 172.21.15.28:0/3939305809 conn(0x560028a06cb0 msgr2=0x560028a08a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:44.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.238+0000 7feb72ffd640 0 client.25894 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:44.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.238+0000 7feb9ce50740 1 -- 172.21.15.28:0/3939305809 shutdown_connections 2024-04-21T08:53:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.239+0000 7feb9ce50740 1 -- 172.21.15.28:0/3939305809 wait complete. 2024-04-21T08:53:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.240+0000 7feb9ce50740 20 client.25894 trim_cache size 0 max 16384 2024-04-21T08:53:44.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.276+0000 7feb9ce50740 1 Processor -- start 2024-04-21T08:53:44.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.276+0000 7feb9ce50740 1 -- start start 2024-04-21T08:53:44.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.277+0000 7feb9ce50740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a11960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.277+0000 7feb9ce50740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 0x560028a21730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.277+0000 7feb9ce50740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a15a80 0x560028a132c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.277+0000 7feb9ce50740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feb941c0a90 con 0x560028a0cea0 2024-04-21T08:53:44.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.277+0000 7feb9ce50740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7feb941be360 con 0x560028a15a80 2024-04-21T08:53:44.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.278+0000 7feb9ce50740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7feb941bc1b0 con 0x560028a0c580 2024-04-21T08:53:44.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.278+0000 7feb92ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 0x560028a21730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.278+0000 7feb93fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a15a80 0x560028a132c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.278+0000 7feb92ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 0x560028a21730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36022/0 (socket says 172.21.15.28:36022) 2024-04-21T08:53:44.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.278+0000 7feb92ffd640 1 -- 172.21.15.28:0/3270618413 learned_addr learned my addr 172.21.15.28:0/3270618413 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:44.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.278+0000 7feb937fe640 1 --2- 172.21.15.28:0/3270618413 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a11960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.279+0000 7feb92ffd640 1 -- 172.21.15.28:0/3270618413 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 msgr2=0x560028a11960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:44.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.279+0000 7feb92ffd640 1 --2- 172.21.15.28:0/3270618413 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a11960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.279+0000 7feb92ffd640 1 -- 172.21.15.28:0/3270618413 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a15a80 msgr2=0x560028a132c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:44.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.279+0000 7feb92ffd640 1 --2- 172.21.15.28:0/3270618413 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a15a80 0x560028a132c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.279+0000 7feb92ffd640 1 -- 172.21.15.28:0/3270618413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560028a0ccd0 con 0x560028a0cea0 2024-04-21T08:53:44.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.279+0000 7feb92ffd640 1 --2- 172.21.15.28:0/3270618413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 0x560028a21730 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7feb84014c50 tx=0x7feb840d2520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c2350fa34a6fd05 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:44.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.283+0000 7feb897fa640 1 -- 172.21.15.28:0/3270618413 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8409b930 con 0x560028a0cea0 2024-04-21T08:53:44.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.283+0000 7feb897fa640 1 -- 172.21.15.28:0/3270618413 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb8409bdf0 con 0x560028a0cea0 2024-04-21T08:53:44.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.283+0000 7feb897fa640 1 -- 172.21.15.28:0/3270618413 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8409b750 con 0x560028a0cea0 2024-04-21T08:53:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.284+0000 7feb9ce50740 1 -- 172.21.15.28:0/3270618413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 msgr2=0x560028a21730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.284+0000 7feb9ce50740 1 --2- 172.21.15.28:0/3270618413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 0x560028a21730 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7feb84014c50 tx=0x7feb840d2520 comp rx=0 tx=0).stop 2024-04-21T08:53:44.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.286+0000 7feb9ce50740 1 -- 172.21.15.28:0/3270618413 shutdown_connections 2024-04-21T08:53:44.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.286+0000 7feb9ce50740 1 --2- 172.21.15.28:0/3270618413 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a15a80 0x560028a132c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.286+0000 7feb9ce50740 1 --2- 172.21.15.28:0/3270618413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 0x560028a21730 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.286+0000 7feb9ce50740 1 --2- 172.21.15.28:0/3270618413 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a11960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.286+0000 7feb9ce50740 1 -- 172.21.15.28:0/3270618413 >> 172.21.15.28:0/3270618413 conn(0x560028a06cb0 msgr2=0x560028a05d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:44.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.286+0000 7feb9ce50740 1 -- 172.21.15.28:0/3270618413 shutdown_connections 2024-04-21T08:53:44.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.286+0000 7feb9ce50740 1 -- 172.21.15.28:0/3270618413 wait complete. 2024-04-21T08:53:44.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.286+0000 7feb9ce50740 1 -- start start 2024-04-21T08:53:44.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.287+0000 7feb9ce50740 1 Processor -- start 2024-04-21T08:53:44.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.287+0000 7feb9ce50740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a1e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.287+0000 7feb9ce50740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 0x560028a0f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.287+0000 7feb9ce50740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a15a80 0x560028a0fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.287+0000 7feb9ce50740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feb941bc820 con 0x560028a0cea0 2024-04-21T08:53:44.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.287+0000 7feb9ce50740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7feb941bdc90 con 0x560028a15a80 2024-04-21T08:53:44.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.287+0000 7feb9ce50740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7feb941bcef0 con 0x560028a0c580 2024-04-21T08:53:44.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.287+0000 7feb9ce50740 10 client.0 _ll_register_callbacks cb 0x7feb7c034c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:44.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.289+0000 7feb93fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a15a80 0x560028a0fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.289+0000 7feb92ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 0x560028a0f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.289+0000 7feb92ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 0x560028a0f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36036/0 (socket says 172.21.15.28:36036) 2024-04-21T08:53:44.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.289+0000 7feb92ffd640 1 -- 172.21.15.28:0/701754072 learned_addr learned my addr 172.21.15.28:0/701754072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:44.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.289+0000 7feb937fe640 1 --2- 172.21.15.28:0/701754072 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a1e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.291+0000 7feb92ffd640 1 -- 172.21.15.28:0/701754072 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 msgr2=0x560028a1e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:44.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.291+0000 7feb92ffd640 1 --2- 172.21.15.28:0/701754072 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a1e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.291+0000 7feb92ffd640 1 -- 172.21.15.28:0/701754072 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a15a80 msgr2=0x560028a0fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:44.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.291+0000 7feb92ffd640 1 --2- 172.21.15.28:0/701754072 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a15a80 0x560028a0fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.291+0000 7feb92ffd640 1 -- 172.21.15.28:0/701754072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560028a10340 con 0x560028a0cea0 2024-04-21T08:53:44.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.292+0000 7feb92ffd640 1 --2- 172.21.15.28:0/701754072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 0x560028a0f500 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7feb84009fd0 tx=0x7feb8409b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a5f31e17614b9d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:44.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.294+0000 7feb72ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:53:44.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.295+0000 7feb72ffd640 1 -- 172.21.15.28:0/701754072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb840a49b0 con 0x560028a0cea0 2024-04-21T08:53:44.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.295+0000 7feb72ffd640 1 -- 172.21.15.28:0/701754072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7feb8409a0f0 con 0x560028a0cea0 2024-04-21T08:53:44.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.295+0000 7feb9ce50740 1 -- 172.21.15.28:0/701754072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb8409bdf0 con 0x560028a0cea0 2024-04-21T08:53:44.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.296+0000 7feb72ffd640 1 -- 172.21.15.28:0/701754072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7feb84098460 con 0x560028a0cea0 2024-04-21T08:53:44.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.296+0000 7feb9ce50740 10 client.16560 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:44.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.296+0000 7feb9ce50740 1 -- 172.21.15.28:0/701754072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560028a9ce40 con 0x560028a0cea0 2024-04-21T08:53:44.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.297+0000 7feb72ffd640 1 -- 172.21.15.28:0/701754072 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb8409bdf0 con 0x560028a0cea0 2024-04-21T08:53:44.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.298+0000 7feb72ffd640 1 -- 172.21.15.28:0/701754072 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7feb840a4520 con 0x560028a0cea0 2024-04-21T08:53:44.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.298+0000 7feb9ce50740 10 client.16560 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:44.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.298+0000 7feb9ce50740 10 client.16560 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:44.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.298+0000 7feb9ce50740 20 client.16560 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:44.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.298+0000 7feb9ce50740 1 --2- 172.21.15.28:0/701754072 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560028a9d4a0 0x560028a9f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:44.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.298+0000 7feb9ce50740 4 client.16560 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:53:44.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.298+0000 7feb9ce50740 1 -- 172.21.15.28:0/701754072 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560028978a70 con 0x560028a9d4a0 2024-04-21T08:53:44.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.300+0000 7feb937fe640 1 --2- 172.21.15.28:0/701754072 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560028a9d4a0 0x560028a9f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:44.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.306+0000 7feb72ffd640 10 client.16560 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:44.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.306+0000 7feb937fe640 1 --2- 172.21.15.28:0/701754072 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560028a9d4a0 0x560028a9f8c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3be09e6d400159e server_cookie=59f60e3a6a696f49 in_seq=0 out_seq=0 2024-04-21T08:53:44.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.306+0000 7feb72ffd640 1 -- 172.21.15.28:0/701754072 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x560028978a70 con 0x560028a9d4a0 2024-04-21T08:53:44.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.306+0000 7feb72ffd640 10 client.16560 handle_command_reply: tid=0 2024-04-21T08:53:44.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.306+0000 7feb9ce50740 1 client.16560 shutdown 2024-04-21T08:53:44.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.310+0000 7feb9ce50740 1 -- 172.21.15.28:0/701754072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 msgr2=0x560028a0f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:44.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.310+0000 7feb9ce50740 1 --2- 172.21.15.28:0/701754072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 0x560028a0f500 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7feb84009fd0 tx=0x7feb8409b7c0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.310+0000 7feb9ce50740 1 -- 172.21.15.28:0/701754072 shutdown_connections 2024-04-21T08:53:44.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.310+0000 7feb9ce50740 1 --2- 172.21.15.28:0/701754072 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560028a9d4a0 0x560028a9f8c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.310+0000 7feb9ce50740 1 --2- 172.21.15.28:0/701754072 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560028a15a80 0x560028a0fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.310+0000 7feb9ce50740 1 --2- 172.21.15.28:0/701754072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560028a0cea0 0x560028a0f500 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.310+0000 7feb9ce50740 1 --2- 172.21.15.28:0/701754072 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560028a0c580 0x560028a1e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.310+0000 7feb9ce50740 1 -- 172.21.15.28:0/701754072 >> 172.21.15.28:0/701754072 conn(0x560028a06cb0 msgr2=0x560028a07cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:44.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.310+0000 7feb72ffd640 0 client.16560 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:44.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.311+0000 7feb9ce50740 1 -- 172.21.15.28:0/701754072 shutdown_connections 2024-04-21T08:53:44.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.311+0000 7feb9ce50740 1 -- 172.21.15.28:0/701754072 wait complete. 2024-04-21T08:53:44.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.311+0000 7feb9ce50740 20 client.16560 trim_cache size 0 max 16384 2024-04-21T08:53:44.311 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:53:44.311 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (3 inodes in the stack)", 2024-04-21T08:53:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:53:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "d29fd45f-62ab-465d-a301-0ac2e916a57c": { 2024-04-21T08:53:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:53:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "d29fd45f-62ab-465d-a301-0ac2e916a57c", 2024-04-21T08:53:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:53:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:53:44.312 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:53:44.312 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:53:44.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.313+0000 7feb9a25c640 1 -- 172.21.15.28:0/1153617914 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7feb800a3e20 msgr2=0x7feb800a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:44.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.313+0000 7feb9a25c640 1 --2- 172.21.15.28:0/1153617914 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7feb800a3e20 0x7feb800a62e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7feb7c02c1c0 tx=0x7feb7c034e00 comp rx=0 tx=0).stop 2024-04-21T08:53:44.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.313+0000 7feb9a25c640 1 -- 172.21.15.28:0/1153617914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feb94108f20 msgr2=0x7feb9406c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:44.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.313+0000 7feb9a25c640 1 --2- 172.21.15.28:0/1153617914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feb94108f20 0x7feb9406c7c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7feb8400ba50 tx=0x7feb8400bf10 comp rx=0 tx=0).stop 2024-04-21T08:53:44.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.314+0000 7feb9a25c640 1 -- 172.21.15.28:0/1153617914 shutdown_connections 2024-04-21T08:53:44.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.314+0000 7feb9a25c640 1 --2- 172.21.15.28:0/1153617914 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7feb800a3e20 0x7feb800a62e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.314+0000 7feb9a25c640 1 --2- 172.21.15.28:0/1153617914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9410e620 0x7feb9406cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.314+0000 7feb9a25c640 1 --2- 172.21.15.28:0/1153617914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feb94108f20 0x7feb9406c7c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.314+0000 7feb9a25c640 1 --2- 172.21.15.28:0/1153617914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7feb94108560 0x7feb94073740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:44.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.314+0000 7feb9a25c640 1 -- 172.21.15.28:0/1153617914 >> 172.21.15.28:0/1153617914 conn(0x7feb94077d80 msgr2=0x7feb94078dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:44.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.315+0000 7feb9a25c640 1 -- 172.21.15.28:0/1153617914 shutdown_connections 2024-04-21T08:53:44.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:44.315+0000 7feb9a25c640 1 -- 172.21.15.28:0/1153617914 wait complete. 2024-04-21T08:53:44.335 INFO:tasks.cephfs.filesystem:scrub status for tag:d29fd45f-62ab-465d-a301-0ac2e916a57c - {'path': '/', 'tag': 'd29fd45f-62ab-465d-a301-0ac2e916a57c', 'options': 'recursive,force'} 2024-04-21T08:53:44.336 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:44 smithi028 ceph-mon[28003]: pgmap v79: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 64 MiB/s wr, 11.74k op/s 2024-04-21T08:53:44.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:44 smithi171 ceph-mon[31358]: pgmap v79: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 64 MiB/s wr, 11.74k op/s 2024-04-21T08:53:44.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:44 smithi152 ceph-mon[31401]: pgmap v79: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 64 MiB/s wr, 11.74k op/s 2024-04-21T08:53:45.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:45 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:53:45.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:45 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:45.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:45 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:53:45.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:45 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:53:45.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:45 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:45.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:45 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:53:45.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:45 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:53:45.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:45 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:45.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:45 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:53:46.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:46 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:53:46.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:46 smithi171 ceph-mon[31358]: pgmap v80: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 46 MiB/s wr, 9.10k op/s 2024-04-21T08:53:46.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:46 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:53:46.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:46 smithi152 ceph-mon[31401]: pgmap v80: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 46 MiB/s wr, 9.10k op/s 2024-04-21T08:53:46.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:46 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:53:46.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:46 smithi028 ceph-mon[28003]: pgmap v80: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 46 MiB/s wr, 9.10k op/s 2024-04-21T08:53:48.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:48 smithi152 ceph-mon[31401]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 41 MiB/s wr, 8.32k op/s 2024-04-21T08:53:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:48 smithi028 ceph-mon[28003]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 41 MiB/s wr, 8.32k op/s 2024-04-21T08:53:48.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:48 smithi171 ceph-mon[31358]: pgmap v81: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 41 MiB/s wr, 8.32k op/s 2024-04-21T08:53:50.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:50 smithi152 ceph-mon[31401]: pgmap v82: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 57 MiB/s wr, 11.49k op/s 2024-04-21T08:53:50.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:50 smithi028 ceph-mon[28003]: pgmap v82: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 57 MiB/s wr, 11.49k op/s 2024-04-21T08:53:50.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:50 smithi171 ceph-mon[31358]: pgmap v82: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 57 MiB/s wr, 11.49k op/s 2024-04-21T08:53:52.406 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:52 smithi152 ceph-mon[31401]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 39 MiB/s wr, 7.86k op/s 2024-04-21T08:53:52.407 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:52 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:53:52.407 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:52 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:52.421 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:52 smithi171 ceph-mon[31358]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 39 MiB/s wr, 7.86k op/s 2024-04-21T08:53:52.422 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:52 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:53:52.422 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:52 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:52.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:52 smithi028 ceph-mon[28003]: pgmap v83: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 39 MiB/s wr, 7.86k op/s 2024-04-21T08:53:52.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:52 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:53:52.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:52 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:53.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.900+0000 7fbc88b8e640 1 -- 172.21.15.28:0/2832533452 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc84108f50 msgr2=0x7fbc841093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:53.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.900+0000 7fbc88b8e640 1 --2- 172.21.15.28:0/2832533452 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc84108f50 0x7fbc841093b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009900 tx=0x7fbc6c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:53:53.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.900+0000 7fbc88b8e640 1 -- 172.21.15.28:0/2832533452 shutdown_connections 2024-04-21T08:53:53.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.900+0000 7fbc88b8e640 1 --2- 172.21.15.28:0/2832533452 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc841098f0 0x7fbc84115840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.900+0000 7fbc88b8e640 1 --2- 172.21.15.28:0/2832533452 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc84108f50 0x7fbc841093b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.900+0000 7fbc88b8e640 1 --2- 172.21.15.28:0/2832533452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8410e980 0x7fbc8410ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.900+0000 7fbc88b8e640 1 -- 172.21.15.28:0/2832533452 >> 172.21.15.28:0/2832533452 conn(0x7fbc8406d4b0 msgr2=0x7fbc8406d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:53.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.900+0000 7fbc88b8e640 1 -- 172.21.15.28:0/2832533452 shutdown_connections 2024-04-21T08:53:53.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.901+0000 7fbc88b8e640 1 -- 172.21.15.28:0/2832533452 wait complete. 2024-04-21T08:53:53.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.902+0000 7fbc88b8e640 1 Processor -- start 2024-04-21T08:53:53.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.902+0000 7fbc88b8e640 1 -- start start 2024-04-21T08:53:53.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.902+0000 7fbc88b8e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc84108f50 0x7fbc841a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.902+0000 7fbc88b8e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc841098f0 0x7fbc841a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.902+0000 7fbc88b8e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc8410e980 0x7fbc841a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.902+0000 7fbc88b8e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8411a9f0 con 0x7fbc841098f0 2024-04-21T08:53:53.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.902+0000 7fbc88b8e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8411a870 con 0x7fbc84108f50 2024-04-21T08:53:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.902+0000 7fbc88b8e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8411ab70 con 0x7fbc8410e980 2024-04-21T08:53:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.903+0000 7fbc79d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc841098f0 0x7fbc841a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.903+0000 7fbc82d9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc8410e980 0x7fbc841a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.903+0000 7fbc79d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc841098f0 0x7fbc841a13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39358/0 (socket says 172.21.15.28:39358) 2024-04-21T08:53:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.903+0000 7fbc79d9b640 1 -- 172.21.15.28:0/2775815580 learned_addr learned my addr 172.21.15.28:0/2775815580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.903+0000 7fbc8259c640 1 --2- 172.21.15.28:0/2775815580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc84108f50 0x7fbc841a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.903+0000 7fbc79d9b640 1 -- 172.21.15.28:0/2775815580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc8410e980 msgr2=0x7fbc841a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.903+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/2775815580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc8410e980 0x7fbc841a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.903+0000 7fbc79d9b640 1 -- 172.21.15.28:0/2775815580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc84108f50 msgr2=0x7fbc841a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.903+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/2775815580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc84108f50 0x7fbc841a0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.903+0000 7fbc79d9b640 1 -- 172.21.15.28:0/2775815580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc841a5e60 con 0x7fbc841098f0 2024-04-21T08:53:53.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.903+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/2775815580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc841098f0 0x7fbc841a13b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c005940 tx=0x7fbc6c004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:53.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.904+0000 7fbc7bfff640 1 -- 172.21.15.28:0/2775815580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c054020 con 0x7fbc841098f0 2024-04-21T08:53:53.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.904+0000 7fbc88b8e640 1 -- 172.21.15.28:0/2775815580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc841a60f0 con 0x7fbc841098f0 2024-04-21T08:53:53.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.904+0000 7fbc88b8e640 1 -- 172.21.15.28:0/2775815580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc84117720 con 0x7fbc841098f0 2024-04-21T08:53:53.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.904+0000 7fbc7bfff640 1 -- 172.21.15.28:0/2775815580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc6c02caf0 con 0x7fbc841098f0 2024-04-21T08:53:53.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.904+0000 7fbc7bfff640 1 -- 172.21.15.28:0/2775815580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c004e20 con 0x7fbc841098f0 2024-04-21T08:53:53.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.907+0000 7fbc7bfff640 1 -- 172.21.15.28:0/2775815580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbc6c057020 con 0x7fbc841098f0 2024-04-21T08:53:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.907+0000 7fbc8b776740 1 Processor -- start 2024-04-21T08:53:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.908+0000 7fbc8b776740 1 -- start start 2024-04-21T08:53:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.908+0000 7fbc8b776740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e60553b8d0 0x55e60553a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.908+0000 7fbc8b776740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553ab60 0x55e60553afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.908+0000 7fbc7bfff640 1 --2- 172.21.15.28:0/2775815580 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbc700a3dc0 0x7fbc700a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.908+0000 7fbc8b776740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6055440e0 0x55e6055464d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.908+0000 7fbc8b776740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e60549f8d0 con 0x55e6055440e0 2024-04-21T08:53:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.908+0000 7fbc8b776740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e60543ce80 con 0x55e60553ab60 2024-04-21T08:53:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.908+0000 7fbc8b776740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55e60549b890 con 0x55e60553b8d0 2024-04-21T08:53:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.908+0000 7fbc7bfff640 1 -- 172.21.15.28:0/2775815580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbc6c103a80 con 0x7fbc841098f0 2024-04-21T08:53:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.908+0000 7fbc82d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6055440e0 0x55e6055464d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.908+0000 7fbc82d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6055440e0 0x55e6055464d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39364/0 (socket says 172.21.15.28:39364) 2024-04-21T08:53:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.908+0000 7fbc82d9d640 1 -- 172.21.15.28:0/197468800 learned_addr learned my addr 172.21.15.28:0/197468800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.908+0000 7fbc82d9d640 1 --2- 172.21.15.28:0/197468800 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e60553b8d0 0x55e60553a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.908+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/197468800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553ab60 0x55e60553afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.908+0000 7fbc8259c640 1 --2- 172.21.15.28:0/2775815580 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbc700a3dc0 0x7fbc700a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.909+0000 7fbc79d9b640 1 -- 172.21.15.28:0/197468800 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e60553b8d0 msgr2=0x55e60553a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.909+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/197468800 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e60553b8d0 0x55e60553a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.909+0000 7fbc79d9b640 1 -- 172.21.15.28:0/197468800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6055440e0 msgr2=0x55e6055464d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:53.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.909+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/197468800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6055440e0 0x55e6055464d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.909+0000 7fbc79d9b640 1 -- 172.21.15.28:0/197468800 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e60545e320 con 0x55e60553ab60 2024-04-21T08:53:53.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.909+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/197468800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553ab60 0x55e60553afe0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c100f40 tx=0x7fbc6c0cf390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2dc04b4ea05120a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:53.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.909+0000 7fbc8259c640 1 --2- 172.21.15.28:0/2775815580 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbc700a3dc0 0x7fbc700a6280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbc74005fd0 tx=0x7fbc7400e3a0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:53.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.910+0000 7fbc78d99640 1 -- 172.21.15.28:0/197468800 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c0d3b40 con 0x55e60553ab60 2024-04-21T08:53:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.910+0000 7fbc78d99640 1 -- 172.21.15.28:0/197468800 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc6c0d8400 con 0x55e60553ab60 2024-04-21T08:53:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.910+0000 7fbc78d99640 1 -- 172.21.15.28:0/197468800 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c0d81d0 con 0x55e60553ab60 2024-04-21T08:53:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.911+0000 7fbc8b776740 1 -- 172.21.15.28:0/197468800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553ab60 msgr2=0x55e60553afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.911+0000 7fbc8b776740 1 --2- 172.21.15.28:0/197468800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553ab60 0x55e60553afe0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c100f40 tx=0x7fbc6c0cf390 comp rx=0 tx=0).stop 2024-04-21T08:53:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.911+0000 7fbc8b776740 1 -- 172.21.15.28:0/197468800 shutdown_connections 2024-04-21T08:53:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.911+0000 7fbc8b776740 1 --2- 172.21.15.28:0/197468800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6055440e0 0x55e6055464d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.911+0000 7fbc8b776740 1 --2- 172.21.15.28:0/197468800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553ab60 0x55e60553afe0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.912+0000 7fbc8b776740 1 --2- 172.21.15.28:0/197468800 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e60553b8d0 0x55e60553a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.912+0000 7fbc8b776740 1 -- 172.21.15.28:0/197468800 >> 172.21.15.28:0/197468800 conn(0x55e605501c90 msgr2=0x55e605535850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:53.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.912+0000 7fbc8b776740 1 -- 172.21.15.28:0/197468800 shutdown_connections 2024-04-21T08:53:53.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.912+0000 7fbc8b776740 1 -- 172.21.15.28:0/197468800 wait complete. 2024-04-21T08:53:53.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.913+0000 7fbc8b776740 1 -- start start 2024-04-21T08:53:53.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.913+0000 7fbc8b776740 1 Processor -- start 2024-04-21T08:53:53.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.913+0000 7fbc8b776740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553ab60 0x55e605543b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.913+0000 7fbc8b776740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553b8d0 0x55e605540ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.913+0000 7fbc8b776740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e6055440e0 0x55e605541630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.913+0000 7fbc8b776740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e605546bd0 con 0x55e60553b8d0 2024-04-21T08:53:53.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.913+0000 7fbc8b776740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e605427eb0 con 0x55e60553ab60 2024-04-21T08:53:53.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.913+0000 7fbc8b776740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55e605546d50 con 0x55e6055440e0 2024-04-21T08:53:53.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.913+0000 7fbc8b776740 10 client.0 _ll_register_callbacks cb 0x55e605454b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:53.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.914+0000 7fbc79d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553b8d0 0x55e605540ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.914+0000 7fbc8259c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553ab60 0x55e605543b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.914+0000 7fbc79d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553b8d0 0x55e605540ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39376/0 (socket says 172.21.15.28:39376) 2024-04-21T08:53:53.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.914+0000 7fbc79d9b640 1 -- 172.21.15.28:0/4263961260 learned_addr learned my addr 172.21.15.28:0/4263961260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:53.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.914+0000 7fbc82d9d640 1 --2- 172.21.15.28:0/4263961260 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e6055440e0 0x55e605541630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.914+0000 7fbc8259c640 1 -- 172.21.15.28:0/4263961260 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e6055440e0 msgr2=0x55e605541630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:53.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.914+0000 7fbc8259c640 1 --2- 172.21.15.28:0/4263961260 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e6055440e0 0x55e605541630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.914+0000 7fbc8259c640 1 -- 172.21.15.28:0/4263961260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553b8d0 msgr2=0x55e605540ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:53.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.914+0000 7fbc8259c640 1 --2- 172.21.15.28:0/4263961260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553b8d0 0x55e605540ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.914+0000 7fbc8259c640 1 -- 172.21.15.28:0/4263961260 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6051b2210 con 0x55e60553ab60 2024-04-21T08:53:53.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.914+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/4263961260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553b8d0 0x55e605540ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:53:53.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.914+0000 7fbc8259c640 1 --2- 172.21.15.28:0/4263961260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553ab60 0x55e605543b60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400f630 tx=0x7fbc74004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d51c193cfb0cedb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:53.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.915+0000 7fbc69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:53:53.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.915+0000 7fbc69ffb640 1 -- 172.21.15.28:0/4263961260 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc74016910 con 0x55e60553ab60 2024-04-21T08:53:53.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.915+0000 7fbc69ffb640 1 -- 172.21.15.28:0/4263961260 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbc74016ab0 con 0x55e60553ab60 2024-04-21T08:53:53.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.916+0000 7fbc8b776740 1 -- 172.21.15.28:0/4263961260 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc6c0d8400 con 0x55e60553ab60 2024-04-21T08:53:53.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.916+0000 7fbc69ffb640 1 -- 172.21.15.28:0/4263961260 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbc74016cb0 con 0x55e60553ab60 2024-04-21T08:53:53.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.918+0000 7fbc8b776740 10 client.25913 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:53.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.918+0000 7fbc8b776740 1 -- 172.21.15.28:0/4263961260 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e60527e620 con 0x55e60553ab60 2024-04-21T08:53:53.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.918+0000 7fbc69ffb640 1 -- 172.21.15.28:0/4263961260 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc6c0d8400 con 0x55e60553ab60 2024-04-21T08:53:53.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.918+0000 7fbc69ffb640 1 -- 172.21.15.28:0/4263961260 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fbc74016040 con 0x55e60553ab60 2024-04-21T08:53:53.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.918+0000 7fbc8b776740 10 client.25913 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:53.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.918+0000 7fbc8b776740 10 client.25913 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:53.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.918+0000 7fbc8b776740 20 client.25913 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:53.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.918+0000 7fbc8b776740 1 --2- 172.21.15.28:0/4263961260 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e605553f80 0x55e6055563b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.919+0000 7fbc8b776740 4 client.25913 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:53.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.919+0000 7fbc8b776740 1 -- 172.21.15.28:0/4263961260 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6054b3880 con 0x55e605553f80 2024-04-21T08:53:53.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.919+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/4263961260 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e605553f80 0x55e6055563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.919+0000 7fbc69ffb640 10 client.25913 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:53.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.920+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/4263961260 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e605553f80 0x55e6055563b0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68ed385138656e49 server_cookie=f45bd932a2b56ce2 in_seq=0 out_seq=0 2024-04-21T08:53:53.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.920+0000 7fbc69ffb640 1 -- 172.21.15.28:0/4263961260 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e6054b3880 con 0x55e605553f80 2024-04-21T08:53:53.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.920+0000 7fbc69ffb640 10 client.25913 handle_command_reply: tid=0 2024-04-21T08:53:53.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.921+0000 7fbc8b776740 1 client.25913 shutdown 2024-04-21T08:53:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.922+0000 7fbc8b776740 1 -- 172.21.15.28:0/4263961260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553ab60 msgr2=0x55e605543b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.922+0000 7fbc8b776740 1 --2- 172.21.15.28:0/4263961260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553ab60 0x55e605543b60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400f630 tx=0x7fbc74004ab0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.922+0000 7fbc8b776740 1 -- 172.21.15.28:0/4263961260 shutdown_connections 2024-04-21T08:53:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.922+0000 7fbc8b776740 1 --2- 172.21.15.28:0/4263961260 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e605553f80 0x55e6055563b0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.922+0000 7fbc8b776740 1 --2- 172.21.15.28:0/4263961260 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e6055440e0 0x55e605541630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.922+0000 7fbc8b776740 1 --2- 172.21.15.28:0/4263961260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553b8d0 0x55e605540ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.922+0000 7fbc8b776740 1 --2- 172.21.15.28:0/4263961260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553ab60 0x55e605543b60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.922+0000 7fbc8b776740 1 -- 172.21.15.28:0/4263961260 >> 172.21.15.28:0/4263961260 conn(0x55e605501c90 msgr2=0x55e6055375a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.922+0000 7fbc69ffb640 0 client.25913 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.922+0000 7fbc8b776740 1 -- 172.21.15.28:0/4263961260 shutdown_connections 2024-04-21T08:53:53.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.922+0000 7fbc8b776740 1 -- 172.21.15.28:0/4263961260 wait complete. 2024-04-21T08:53:53.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.922+0000 7fbc8b776740 20 client.25913 trim_cache size 0 max 16384 2024-04-21T08:53:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.949+0000 7fbc8b776740 1 Processor -- start 2024-04-21T08:53:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.949+0000 7fbc8b776740 1 -- start start 2024-04-21T08:53:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.949+0000 7fbc8b776740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553ab60 0x55e60553c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.949+0000 7fbc8b776740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553b8d0 0x55e60554fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.949+0000 7fbc8b776740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e6055440e0 0x55e60553a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.950+0000 7fbc8259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553ab60 0x55e60553c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.950+0000 7fbc8b776740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc64012880 con 0x55e60553ab60 2024-04-21T08:53:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.950+0000 7fbc8259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553ab60 0x55e60553c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39380/0 (socket says 172.21.15.28:39380) 2024-04-21T08:53:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.950+0000 7fbc8259c640 1 -- 172.21.15.28:0/1384262094 learned_addr learned my addr 172.21.15.28:0/1384262094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.950+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/1384262094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553b8d0 0x55e60554fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.950+0000 7fbc8b776740 1 -- 172.21.15.28:0/1384262094 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbc64010150 con 0x55e60553b8d0 2024-04-21T08:53:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.950+0000 7fbc8b776740 1 -- 172.21.15.28:0/1384262094 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6400dfa0 con 0x55e6055440e0 2024-04-21T08:53:53.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.950+0000 7fbc82d9d640 1 --2- 172.21.15.28:0/1384262094 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e6055440e0 0x55e60553a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.950+0000 7fbc79d9b640 1 -- 172.21.15.28:0/1384262094 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e6055440e0 msgr2=0x55e60553a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:53.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.950+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/1384262094 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e6055440e0 0x55e60553a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.950+0000 7fbc79d9b640 1 -- 172.21.15.28:0/1384262094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553ab60 msgr2=0x55e60553c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:53.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.950+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/1384262094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553ab60 0x55e60553c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.950+0000 7fbc79d9b640 1 -- 172.21.15.28:0/1384262094 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e60552fce0 con 0x55e60553b8d0 2024-04-21T08:53:53.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.951+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/1384262094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553b8d0 0x55e60554fa90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0cf490 tx=0x7fbc6c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6399f04c9078cd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:53.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.951+0000 7fbc6bfff640 1 -- 172.21.15.28:0/1384262094 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c0d8400 con 0x55e60553b8d0 2024-04-21T08:53:53.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.951+0000 7fbc6bfff640 1 -- 172.21.15.28:0/1384262094 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc6c0d3b40 con 0x55e60553b8d0 2024-04-21T08:53:53.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.951+0000 7fbc6bfff640 1 -- 172.21.15.28:0/1384262094 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c0cb740 con 0x55e60553b8d0 2024-04-21T08:53:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.952+0000 7fbc8b776740 1 -- 172.21.15.28:0/1384262094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553b8d0 msgr2=0x55e60554fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.952+0000 7fbc8b776740 1 --2- 172.21.15.28:0/1384262094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553b8d0 0x55e60554fa90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0cf490 tx=0x7fbc6c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:53:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.952+0000 7fbc8b776740 1 -- 172.21.15.28:0/1384262094 shutdown_connections 2024-04-21T08:53:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.952+0000 7fbc8b776740 1 --2- 172.21.15.28:0/1384262094 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e6055440e0 0x55e60553a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.952+0000 7fbc8b776740 1 --2- 172.21.15.28:0/1384262094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e60553b8d0 0x55e60554fa90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.952+0000 7fbc8b776740 1 --2- 172.21.15.28:0/1384262094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553ab60 0x55e60553c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.952+0000 7fbc8b776740 1 -- 172.21.15.28:0/1384262094 >> 172.21.15.28:0/1384262094 conn(0x55e605501c90 msgr2=0x55e60553c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.952+0000 7fbc8b776740 1 -- 172.21.15.28:0/1384262094 shutdown_connections 2024-04-21T08:53:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.952+0000 7fbc8b776740 1 -- 172.21.15.28:0/1384262094 wait complete. 2024-04-21T08:53:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.953+0000 7fbc8b776740 1 -- start start 2024-04-21T08:53:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.953+0000 7fbc8b776740 1 Processor -- start 2024-04-21T08:53:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.953+0000 7fbc8b776740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553ab60 0x55e60554c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.954+0000 7fbc8b776740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e60553b8d0 0x55e6055456b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.954+0000 7fbc8b776740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e6055440e0 0x55e605545bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.954+0000 7fbc79d9b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e60553b8d0 0x55e6055456b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.954+0000 7fbc79d9b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e60553b8d0 0x55e6055456b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:46508/0 (socket says 172.21.15.28:46508) 2024-04-21T08:53:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.954+0000 7fbc79d9b640 1 -- 172.21.15.28:0/164202605 learned_addr learned my addr 172.21.15.28:0/164202605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:53:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.954+0000 7fbc8b776740 1 -- 172.21.15.28:0/164202605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6400e610 con 0x55e60553ab60 2024-04-21T08:53:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.954+0000 7fbc82d9d640 1 --2- 172.21.15.28:0/164202605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e6055440e0 0x55e605545bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.954+0000 7fbc8b776740 1 -- 172.21.15.28:0/164202605 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6400fa80 con 0x55e6055440e0 2024-04-21T08:53:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.954+0000 7fbc8b776740 1 -- 172.21.15.28:0/164202605 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6400ece0 con 0x55e60553b8d0 2024-04-21T08:53:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.954+0000 7fbc8b776740 10 client.0 _ll_register_callbacks cb 0x7fbc74009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.955+0000 7fbc8259c640 1 --2- 172.21.15.28:0/164202605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553ab60 0x55e60554c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.955+0000 7fbc79d9b640 1 -- 172.21.15.28:0/164202605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e6055440e0 msgr2=0x55e605545bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.955+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/164202605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e6055440e0 0x55e605545bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.955+0000 7fbc79d9b640 1 -- 172.21.15.28:0/164202605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553ab60 msgr2=0x55e60554c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.955+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/164202605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553ab60 0x55e60554c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.955+0000 7fbc79d9b640 1 -- 172.21.15.28:0/164202605 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6055444f0 con 0x55e60553b8d0 2024-04-21T08:53:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.955+0000 7fbc79d9b640 1 --2- 172.21.15.28:0/164202605 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e60553b8d0 0x55e6055456b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c006010 tx=0x7fbc6c0d81a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5e96e0308c81da7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.955+0000 7fbc69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:53:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.955+0000 7fbc69ffb640 1 -- 172.21.15.28:0/164202605 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c0d8400 con 0x55e60553b8d0 2024-04-21T08:53:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.955+0000 7fbc69ffb640 1 -- 172.21.15.28:0/164202605 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbc6c0cedf0 con 0x55e60553b8d0 2024-04-21T08:53:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.955+0000 7fbc8b776740 1 -- 172.21.15.28:0/164202605 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc6c0d3b40 con 0x55e60553b8d0 2024-04-21T08:53:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.956+0000 7fbc69ffb640 1 -- 172.21.15.28:0/164202605 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbc6c0cb4e0 con 0x55e60553b8d0 2024-04-21T08:53:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.956+0000 7fbc8b776740 10 client.25918 fetch_fsmap learned FSMap version 26 2024-04-21T08:53:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.957+0000 7fbc8b776740 1 -- 172.21.15.28:0/164202605 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e60554b2a0 con 0x55e60553b8d0 2024-04-21T08:53:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.957+0000 7fbc69ffb640 1 -- 172.21.15.28:0/164202605 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc6c0d3b40 con 0x55e60553b8d0 2024-04-21T08:53:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.957+0000 7fbc69ffb640 1 -- 172.21.15.28:0/164202605 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fbc6c0ce650 con 0x55e60553b8d0 2024-04-21T08:53:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.957+0000 7fbc8b776740 10 client.25918 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:53:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.957+0000 7fbc8b776740 10 client.25918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.957+0000 7fbc8b776740 20 client.25918 populate_metadata read hostname 'smithi028' 2024-04-21T08:53:53.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.957+0000 7fbc8b776740 1 --2- 172.21.15.28:0/164202605 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e60554b640 0x55e60554ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:53.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.957+0000 7fbc8b776740 4 client.25918 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:53:53.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.957+0000 7fbc8b776740 1 -- 172.21.15.28:0/164202605 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e60543f3d0 con 0x55e60554b640 2024-04-21T08:53:53.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.958+0000 7fbc8259c640 1 --2- 172.21.15.28:0/164202605 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e60554b640 0x55e60554ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:53.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.958+0000 7fbc69ffb640 10 client.25918 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:53.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.958+0000 7fbc8259c640 1 --2- 172.21.15.28:0/164202605 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e60554b640 0x55e60554ba20 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25677d464505b1e6 server_cookie=577848aad5f5e2c0 in_seq=0 out_seq=0 2024-04-21T08:53:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.990+0000 7fbc69ffb640 1 -- 172.21.15.28:0/164202605 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e60543f3d0 con 0x55e60554b640 2024-04-21T08:53:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.990+0000 7fbc69ffb640 10 client.25918 handle_command_reply: tid=0 2024-04-21T08:53:53.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.990+0000 7fbc8b776740 1 client.25918 shutdown 2024-04-21T08:53:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.992+0000 7fbc8b776740 1 -- 172.21.15.28:0/164202605 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e60553b8d0 msgr2=0x55e6055456b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.992+0000 7fbc8b776740 1 --2- 172.21.15.28:0/164202605 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e60553b8d0 0x55e6055456b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c006010 tx=0x7fbc6c0d81a0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.992+0000 7fbc8b776740 1 -- 172.21.15.28:0/164202605 shutdown_connections 2024-04-21T08:53:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.992+0000 7fbc8b776740 1 --2- 172.21.15.28:0/164202605 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e60554b640 0x55e60554ba20 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.992+0000 7fbc8b776740 1 --2- 172.21.15.28:0/164202605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e6055440e0 0x55e605545bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.992+0000 7fbc8b776740 1 --2- 172.21.15.28:0/164202605 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e60553b8d0 0x55e6055456b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.992+0000 7fbc8b776740 1 --2- 172.21.15.28:0/164202605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e60553ab60 0x55e60554c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.992+0000 7fbc8b776740 1 -- 172.21.15.28:0/164202605 >> 172.21.15.28:0/164202605 conn(0x55e605501c90 msgr2=0x55e60553f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.992+0000 7fbc69ffb640 0 client.25918 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.992+0000 7fbc8b776740 1 -- 172.21.15.28:0/164202605 shutdown_connections 2024-04-21T08:53:53.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.992+0000 7fbc8b776740 1 -- 172.21.15.28:0/164202605 wait complete. 2024-04-21T08:53:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.993+0000 7fbc8b776740 20 client.25918 trim_cache size 0 max 16384 2024-04-21T08:53:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:53:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:53:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:53:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:53:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.996+0000 7fbc88b8e640 1 -- 172.21.15.28:0/2775815580 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbc700a3dc0 msgr2=0x7fbc700a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.996+0000 7fbc88b8e640 1 --2- 172.21.15.28:0/2775815580 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbc700a3dc0 0x7fbc700a6280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbc74005fd0 tx=0x7fbc7400e3a0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.996+0000 7fbc88b8e640 1 -- 172.21.15.28:0/2775815580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc841098f0 msgr2=0x7fbc841a13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.996+0000 7fbc88b8e640 1 --2- 172.21.15.28:0/2775815580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc841098f0 0x7fbc841a13b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c005940 tx=0x7fbc6c004bb0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.996+0000 7fbc88b8e640 1 -- 172.21.15.28:0/2775815580 shutdown_connections 2024-04-21T08:53:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.996+0000 7fbc88b8e640 1 --2- 172.21.15.28:0/2775815580 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbc700a3dc0 0x7fbc700a6280 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.996+0000 7fbc88b8e640 1 --2- 172.21.15.28:0/2775815580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc8410e980 0x7fbc841a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.996+0000 7fbc88b8e640 1 --2- 172.21.15.28:0/2775815580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc841098f0 0x7fbc841a13b0 secure :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c005940 tx=0x7fbc6c004bb0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.996+0000 7fbc88b8e640 1 --2- 172.21.15.28:0/2775815580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc84108f50 0x7fbc841a0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.996+0000 7fbc88b8e640 1 -- 172.21.15.28:0/2775815580 >> 172.21.15.28:0/2775815580 conn(0x7fbc8406d4b0 msgr2=0x7fbc84072680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.997+0000 7fbc88b8e640 1 -- 172.21.15.28:0/2775815580 shutdown_connections 2024-04-21T08:53:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:53:53.997+0000 7fbc88b8e640 1 -- 172.21.15.28:0/2775815580 wait complete. 2024-04-21T08:53:54.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:54 smithi152 ceph-mon[31401]: pgmap v84: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 41 MiB/s wr, 8.35k op/s 2024-04-21T08:53:54.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:54 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:54.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:54 smithi028 ceph-mon[28003]: pgmap v84: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 41 MiB/s wr, 8.35k op/s 2024-04-21T08:53:54.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:54 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:54.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:54 smithi171 ceph-mon[31358]: pgmap v84: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 41 MiB/s wr, 8.35k op/s 2024-04-21T08:53:54.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:54 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:55.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:55 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:55.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:55 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:55.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:55 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:56.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:56 smithi152 ceph-mon[31401]: pgmap v85: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 38 MiB/s wr, 7.75k op/s 2024-04-21T08:53:56.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:56 smithi028 ceph-mon[28003]: pgmap v85: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 38 MiB/s wr, 7.75k op/s 2024-04-21T08:53:56.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:56 smithi171 ceph-mon[31358]: pgmap v85: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 38 MiB/s wr, 7.75k op/s 2024-04-21T08:53:57.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.140+0000 7f3757fff640 1 -- 172.21.15.28:0/1335715265 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x561bc73af980 con 0x561bc74e5960 2024-04-21T08:53:57.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.140+0000 7f3757fff640 10 client.25816 handle_command_reply: tid=0 2024-04-21T08:53:57.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.140+0000 7f37817ed740 1 client.25816 shutdown 2024-04-21T08:53:57.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.140+0000 7f37817ed740 1 -- 172.21.15.28:0/1335715265 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc744d1d0 msgr2=0x561bc7473450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:57.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.140+0000 7f37817ed740 1 --2- 172.21.15.28:0/1335715265 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc744d1d0 0x561bc7473450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f376c02c270 tx=0x7f376c02bc60 comp rx=0 tx=0).stop 2024-04-21T08:53:57.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.141+0000 7f37817ed740 1 -- 172.21.15.28:0/1335715265 shutdown_connections 2024-04-21T08:53:57.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.141+0000 7f37817ed740 1 --2- 172.21.15.28:0/1335715265 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561bc74e5960 0x561bc74e7d50 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.141+0000 7f37817ed740 1 --2- 172.21.15.28:0/1335715265 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561bc744d1d0 0x561bc7473450 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.141+0000 7f37817ed740 1 --2- 172.21.15.28:0/1335715265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc7444400 0x561bc7472f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.141+0000 7f37817ed740 1 --2- 172.21.15.28:0/1335715265 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561bc7443ae0 0x561bc74729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.141+0000 7f3757fff640 0 client.25816 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:53:57.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.141+0000 7f37817ed740 1 -- 172.21.15.28:0/1335715265 >> 172.21.15.28:0/1335715265 conn(0x561bc743e400 msgr2=0x561bc743f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:57.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.141+0000 7f37817ed740 1 -- 172.21.15.28:0/1335715265 shutdown_connections 2024-04-21T08:53:57.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.141+0000 7f37817ed740 1 -- 172.21.15.28:0/1335715265 wait complete. 2024-04-21T08:53:57.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.141+0000 7f37817ed740 20 client.25816 trim_cache size 0 max 16384 2024-04-21T08:53:57.142 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:53:57.142 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 26, 2024-04-21T08:53:57.142 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24488, 2024-04-21T08:53:57.142 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 2, 2024-04-21T08:53:57.142 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-04-21T08:53:57.143 INFO:teuthology.orchestra.run.smithi028.stdout: "ec0d6068": { 2024-04-21T08:53:57.143 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 2, 2024-04-21T08:53:57.143 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-04-21T08:53:57.143 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-21T08:53:57.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-04-21T08:53:57.143 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-21T08:53:57.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T08:53:57.143 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 30.0, 2024-04-21T08:53:57.143 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 84.5, 2024-04-21T08:53:57.143 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-04-21T08:53:57.143 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-04-21T08:53:57.143 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-04-21T08:53:57.143 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-21T08:53:57.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-04-21T08:53:57.143 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-21T08:53:57.144 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:53:57.144 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:53:57.144 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:53:57.144 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:53:57.144 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:53:57.144 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:53:57.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.142+0000 7f375f7fe640 1 -- 172.21.15.28:0/727886785 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f37600a3d30 msgr2=0x7f37600a61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:57.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.142+0000 7f375f7fe640 1 --2- 172.21.15.28:0/727886785 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f37600a3d30 0x7f37600a61f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x561bc7368460 tx=0x7f3770009230 comp rx=0 tx=0).stop 2024-04-21T08:53:57.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.142+0000 7f375f7fe640 1 -- 172.21.15.28:0/727886785 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37781a7c00 msgr2=0x7f37781abf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:57.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.142+0000 7f375f7fe640 1 --2- 172.21.15.28:0/727886785 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37781a7c00 0x7f37781abf80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f376800b4a0 tx=0x7f376800b960 comp rx=0 tx=0).stop 2024-04-21T08:53:57.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.142+0000 7f375f7fe640 1 -- 172.21.15.28:0/727886785 shutdown_connections 2024-04-21T08:53:57.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.142+0000 7f375f7fe640 1 --2- 172.21.15.28:0/727886785 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f37600a3d30 0x7f37600a61f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.142+0000 7f375f7fe640 1 --2- 172.21.15.28:0/727886785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37781ac4c0 0x7f37781ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.142+0000 7f375f7fe640 1 --2- 172.21.15.28:0/727886785 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37781a7c00 0x7f37781abf80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.142+0000 7f375f7fe640 1 --2- 172.21.15.28:0/727886785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37781106b0 0x7f37781a76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.142+0000 7f375f7fe640 1 -- 172.21.15.28:0/727886785 >> 172.21.15.28:0/727886785 conn(0x7f377806d070 msgr2=0x7f3778071550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:57.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.142+0000 7f375f7fe640 1 -- 172.21.15.28:0/727886785 shutdown_connections 2024-04-21T08:53:57.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:53:57.142+0000 7f375f7fe640 1 -- 172.21.15.28:0/727886785 wait complete. 2024-04-21T08:53:57.156 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: ec0d6068, quiesce duration: 24.5 2024-04-21T08:53:58.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:53:58 smithi152 ceph-mon[31401]: pgmap v86: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 32 MiB/s wr, 6.62k op/s 2024-04-21T08:53:58.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:53:58 smithi028 ceph-mon[28003]: pgmap v86: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 32 MiB/s wr, 6.62k op/s 2024-04-21T08:53:58.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:53:58 smithi171 ceph-mon[31358]: pgmap v86: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 32 MiB/s wr, 6.62k op/s 2024-04-21T08:54:00.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:00 smithi028 ceph-mon[28003]: pgmap v87: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 33 MiB/s wr, 6.69k op/s 2024-04-21T08:54:00.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:00 smithi152 ceph-mon[31401]: pgmap v87: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 33 MiB/s wr, 6.69k op/s 2024-04-21T08:54:00.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:00 smithi171 ceph-mon[31358]: pgmap v87: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 33 MiB/s wr, 6.69k op/s 2024-04-21T08:54:02.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:02 smithi152 ceph-mon[31401]: pgmap v88: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 17 MiB/s wr, 3.53k op/s 2024-04-21T08:54:02.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:02 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:54:02.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:02 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:54:02.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:02 smithi028 ceph-mon[28003]: pgmap v88: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 17 MiB/s wr, 3.53k op/s 2024-04-21T08:54:02.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:02 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:54:02.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:02 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:54:02.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:02 smithi171 ceph-mon[31358]: pgmap v88: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 17 MiB/s wr, 3.53k op/s 2024-04-21T08:54:02.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:02 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:54:02.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:02 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:54:03.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:03 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:54:03.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:03 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:54:03.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:03 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:54:04.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:04 smithi152 ceph-mon[31401]: pgmap v89: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 14 MiB/s wr, 2.85k op/s 2024-04-21T08:54:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:04 smithi028 ceph-mon[28003]: pgmap v89: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 14 MiB/s wr, 2.85k op/s 2024-04-21T08:54:04.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:04 smithi171 ceph-mon[31358]: pgmap v89: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 14 MiB/s wr, 2.85k op/s 2024-04-21T08:54:06.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:06 smithi152 ceph-mon[31401]: pgmap v90: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.8 MiB/s wr, 1.88k op/s 2024-04-21T08:54:06.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:06 smithi028 ceph-mon[28003]: pgmap v90: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.8 MiB/s wr, 1.88k op/s 2024-04-21T08:54:06.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:06 smithi171 ceph-mon[31358]: pgmap v90: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.8 MiB/s wr, 1.88k op/s 2024-04-21T08:54:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.104+0000 7fe116dca640 1 -- 172.21.15.28:0/3236362250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe110106190 msgr2=0x7fe11010dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.104+0000 7fe116dca640 1 --2- 172.21.15.28:0/3236362250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe110106190 0x7fe11010dd60 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fe108009900 tx=0x7fe10802bc30 comp rx=0 tx=0).stop 2024-04-21T08:54:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.104+0000 7fe116dca640 1 -- 172.21.15.28:0/3236362250 shutdown_connections 2024-04-21T08:54:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.104+0000 7fe116dca640 1 --2- 172.21.15.28:0/3236362250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe110106190 0x7fe11010dd60 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.104+0000 7fe116dca640 1 --2- 172.21.15.28:0/3236362250 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe110101a70 0x7fe110105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.104+0000 7fe116dca640 1 --2- 172.21.15.28:0/3236362250 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1101010c0 0x7fe1101014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.104+0000 7fe116dca640 1 -- 172.21.15.28:0/3236362250 >> 172.21.15.28:0/3236362250 conn(0x7fe110077f60 msgr2=0x7fe110078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.104+0000 7fe116dca640 1 -- 172.21.15.28:0/3236362250 shutdown_connections 2024-04-21T08:54:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.104+0000 7fe116dca640 1 -- 172.21.15.28:0/3236362250 wait complete. 2024-04-21T08:54:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe116dca640 1 Processor -- start 2024-04-21T08:54:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe116dca640 1 -- start start 2024-04-21T08:54:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe116dca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1101010c0 0x7fe110199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe116dca640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe110106190 0x7fe110199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe116dca640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe110199fd0 0x7fe1101a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe116dca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe110111100 con 0x7fe1101010c0 2024-04-21T08:54:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe116dca640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe110110f80 con 0x7fe110106190 2024-04-21T08:54:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe116dca640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe110111280 con 0x7fe110199fd0 2024-04-21T08:54:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe114b66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1101010c0 0x7fe110199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe114b66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1101010c0 0x7fe110199500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38130/0 (socket says 172.21.15.28:38130) 2024-04-21T08:54:07.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe114b66640 1 -- 172.21.15.28:0/1301912179 learned_addr learned my addr 172.21.15.28:0/1301912179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe10ffff640 1 --2- 172.21.15.28:0/1301912179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe110106190 0x7fe110199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe115367640 1 --2- 172.21.15.28:0/1301912179 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe110199fd0 0x7fe1101a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe114b66640 1 -- 172.21.15.28:0/1301912179 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe110199fd0 msgr2=0x7fe1101a4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe114b66640 1 --2- 172.21.15.28:0/1301912179 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe110199fd0 0x7fe1101a4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe114b66640 1 -- 172.21.15.28:0/1301912179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe110106190 msgr2=0x7fe110199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe114b66640 1 --2- 172.21.15.28:0/1301912179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe110106190 0x7fe110199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe114b66640 1 -- 172.21.15.28:0/1301912179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1101a4d50 con 0x7fe1101010c0 2024-04-21T08:54:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.105+0000 7fe10ffff640 1 --2- 172.21.15.28:0/1301912179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe110106190 0x7fe110199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.106+0000 7fe114b66640 1 --2- 172.21.15.28:0/1301912179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1101010c0 0x7fe110199500 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe0f800e9f0 tx=0x7fe0f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.106+0000 7fe10dffb640 1 -- 172.21.15.28:0/1301912179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f800cd80 con 0x7fe1101010c0 2024-04-21T08:54:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.106+0000 7fe10dffb640 1 -- 172.21.15.28:0/1301912179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0f80108e0 con 0x7fe1101010c0 2024-04-21T08:54:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.106+0000 7fe116dca640 1 -- 172.21.15.28:0/1301912179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1101a4fe0 con 0x7fe1101010c0 2024-04-21T08:54:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.106+0000 7fe10dffb640 1 -- 172.21.15.28:0/1301912179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f8018980 con 0x7fe1101010c0 2024-04-21T08:54:07.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.106+0000 7fe116dca640 1 -- 172.21.15.28:0/1301912179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1101a5500 con 0x7fe1101010c0 2024-04-21T08:54:07.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.108+0000 7fe10dffb640 1 -- 172.21.15.28:0/1301912179 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe0f8010430 con 0x7fe1101010c0 2024-04-21T08:54:07.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe1199b6740 1 Processor -- start 2024-04-21T08:54:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe1199b6740 1 -- start start 2024-04-21T08:54:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe1199b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea08d0 0x5579a2e9f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe1199b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2e9fb60 0x5579a2e9ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe1199b6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2ea90e0 0x5579a2eab4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe1199b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5579a2e048d0 con 0x5579a2ea08d0 2024-04-21T08:54:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe1199b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5579a2da1e80 con 0x5579a2e9fb60 2024-04-21T08:54:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe1199b6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5579a2e00890 con 0x5579a2ea90e0 2024-04-21T08:54:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe115367640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea08d0 0x5579a2e9f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe115367640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea08d0 0x5579a2e9f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38140/0 (socket says 172.21.15.28:38140) 2024-04-21T08:54:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe115367640 1 -- 172.21.15.28:0/3356984886 learned_addr learned my addr 172.21.15.28:0/3356984886 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe114b66640 1 --2- 172.21.15.28:0/3356984886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2e9fb60 0x5579a2e9ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe10dffb640 1 --2- 172.21.15.28:0/1301912179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0fc0a3dc0 0x7fe0fc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe10ffff640 1 --2- 172.21.15.28:0/3356984886 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2ea90e0 0x5579a2eab4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe115367640 1 -- 172.21.15.28:0/3356984886 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2ea90e0 msgr2=0x5579a2eab4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe115367640 1 --2- 172.21.15.28:0/3356984886 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2ea90e0 0x5579a2eab4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe115367640 1 -- 172.21.15.28:0/3356984886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2e9fb60 msgr2=0x5579a2e9ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe115367640 1 --2- 172.21.15.28:0/3356984886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2e9fb60 0x5579a2e9ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.109+0000 7fe115367640 1 -- 172.21.15.28:0/3356984886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579a2dc3320 con 0x5579a2ea08d0 2024-04-21T08:54:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.110+0000 7fe114b66640 1 --2- 172.21.15.28:0/3356984886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2e9fb60 0x5579a2e9ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.110+0000 7fe115367640 1 --2- 172.21.15.28:0/1301912179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0fc0a3dc0 0x7fe0fc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.110+0000 7fe10dffb640 1 -- 172.21.15.28:0/1301912179 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe0f80d0f10 con 0x7fe1101010c0 2024-04-21T08:54:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.110+0000 7fe115367640 1 --2- 172.21.15.28:0/3356984886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea08d0 0x5579a2e9f620 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe108036040 tx=0x7fe1080095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4144f3b4a6a51939 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.110+0000 7fe106ffd640 1 -- 172.21.15.28:0/3356984886 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe108046070 con 0x5579a2ea08d0 2024-04-21T08:54:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.110+0000 7fe106ffd640 1 -- 172.21.15.28:0/3356984886 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1080044a0 con 0x5579a2ea08d0 2024-04-21T08:54:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.110+0000 7fe106ffd640 1 -- 172.21.15.28:0/3356984886 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe108004c80 con 0x5579a2ea08d0 2024-04-21T08:54:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.111+0000 7fe1199b6740 1 -- 172.21.15.28:0/3356984886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea08d0 msgr2=0x5579a2e9f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.111+0000 7fe1199b6740 1 --2- 172.21.15.28:0/3356984886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea08d0 0x5579a2e9f620 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe108036040 tx=0x7fe1080095e0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.111+0000 7fe115367640 1 --2- 172.21.15.28:0/1301912179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0fc0a3dc0 0x7fe0fc0a6280 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe10804c000 tx=0x7fe108049000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.111+0000 7fe1199b6740 1 -- 172.21.15.28:0/3356984886 shutdown_connections 2024-04-21T08:54:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.111+0000 7fe1199b6740 1 --2- 172.21.15.28:0/3356984886 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2ea90e0 0x5579a2eab4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.111+0000 7fe1199b6740 1 --2- 172.21.15.28:0/3356984886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2e9fb60 0x5579a2e9ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.111+0000 7fe1199b6740 1 --2- 172.21.15.28:0/3356984886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea08d0 0x5579a2e9f620 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.111+0000 7fe1199b6740 1 -- 172.21.15.28:0/3356984886 >> 172.21.15.28:0/3356984886 conn(0x5579a2e66c90 msgr2=0x5579a2e9a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.111+0000 7fe1199b6740 1 -- 172.21.15.28:0/3356984886 shutdown_connections 2024-04-21T08:54:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.111+0000 7fe1199b6740 1 -- 172.21.15.28:0/3356984886 wait complete. 2024-04-21T08:54:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.111+0000 7fe1199b6740 1 -- start start 2024-04-21T08:54:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.111+0000 7fe1199b6740 1 Processor -- start 2024-04-21T08:54:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe1199b6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2e9fb60 0x5579a2ea8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe1199b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea08d0 0x5579a2ea5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe1199b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2ea90e0 0x5579a2ea6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe1199b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5579a2eabe10 con 0x5579a2ea08d0 2024-04-21T08:54:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe1199b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5579a2d8ceb0 con 0x5579a2ea90e0 2024-04-21T08:54:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe1199b6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5579a2eabf90 con 0x5579a2e9fb60 2024-04-21T08:54:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe1199b6740 10 client.0 _ll_register_callbacks cb 0x5579a2db9b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe10ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea08d0 0x5579a2ea5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe10ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea08d0 0x5579a2ea5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38148/0 (socket says 172.21.15.28:38148) 2024-04-21T08:54:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe10ffff640 1 -- 172.21.15.28:0/2728137771 learned_addr learned my addr 172.21.15.28:0/2728137771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe114b66640 1 --2- 172.21.15.28:0/2728137771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2e9fb60 0x5579a2ea8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe115367640 1 --2- 172.21.15.28:0/2728137771 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2ea90e0 0x5579a2ea6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe10ffff640 1 -- 172.21.15.28:0/2728137771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2e9fb60 msgr2=0x5579a2ea8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe10ffff640 1 --2- 172.21.15.28:0/2728137771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2e9fb60 0x5579a2ea8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe10ffff640 1 -- 172.21.15.28:0/2728137771 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2ea90e0 msgr2=0x5579a2ea6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe10ffff640 1 --2- 172.21.15.28:0/2728137771 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2ea90e0 0x5579a2ea6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe10ffff640 1 -- 172.21.15.28:0/2728137771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579a2b17210 con 0x5579a2ea08d0 2024-04-21T08:54:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe10ffff640 1 --2- 172.21.15.28:0/2728137771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea08d0 0x5579a2ea5dd0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe100012960 tx=0x7fe100012e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15eb1774a2f5bd4f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.112+0000 7fe0effff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.113+0000 7fe0effff640 1 -- 172.21.15.28:0/2728137771 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe100010d30 con 0x5579a2ea08d0 2024-04-21T08:54:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.113+0000 7fe0effff640 1 -- 172.21.15.28:0/2728137771 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe10001b3c0 con 0x5579a2ea08d0 2024-04-21T08:54:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.113+0000 7fe1199b6740 1 -- 172.21.15.28:0/2728137771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1080044a0 con 0x5579a2ea08d0 2024-04-21T08:54:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.113+0000 7fe0effff640 1 -- 172.21.15.28:0/2728137771 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe10001b5c0 con 0x5579a2ea08d0 2024-04-21T08:54:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.113+0000 7fe1199b6740 10 client.16590 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.113+0000 7fe1199b6740 1 -- 172.21.15.28:0/2728137771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5579a2be3620 con 0x5579a2ea08d0 2024-04-21T08:54:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.113+0000 7fe0effff640 1 -- 172.21.15.28:0/2728137771 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1080044a0 con 0x5579a2ea08d0 2024-04-21T08:54:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.113+0000 7fe0effff640 1 -- 172.21.15.28:0/2728137771 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fe10001c7f0 con 0x5579a2ea08d0 2024-04-21T08:54:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.113+0000 7fe1199b6740 10 client.16590 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.113+0000 7fe1199b6740 10 client.16590 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:07.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.113+0000 7fe1199b6740 20 client.16590 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.113+0000 7fe1199b6740 1 --2- 172.21.15.28:0/2728137771 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5579a2eb9180 0x5579a2ebb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.113+0000 7fe1199b6740 4 client.16590 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.113+0000 7fe1199b6740 1 -- 172.21.15.28:0/2728137771 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579a2e18880 con 0x5579a2eb9180 2024-04-21T08:54:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.114+0000 7fe114b66640 1 --2- 172.21.15.28:0/2728137771 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5579a2eb9180 0x5579a2ebb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.114+0000 7fe114b66640 1 --2- 172.21.15.28:0/2728137771 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5579a2eb9180 0x5579a2ebb5b0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=533347f2caa5812f server_cookie=4049e280e444832f in_seq=0 out_seq=0 2024-04-21T08:54:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.114+0000 7fe0effff640 10 client.16590 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.115+0000 7fe0effff640 1 -- 172.21.15.28:0/2728137771 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5579a2e18880 con 0x5579a2eb9180 2024-04-21T08:54:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.115+0000 7fe0effff640 10 client.16590 handle_command_reply: tid=0 2024-04-21T08:54:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.115+0000 7fe1199b6740 1 client.16590 shutdown 2024-04-21T08:54:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.115+0000 7fe1199b6740 1 -- 172.21.15.28:0/2728137771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea08d0 msgr2=0x5579a2ea5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.115+0000 7fe1199b6740 1 --2- 172.21.15.28:0/2728137771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea08d0 0x5579a2ea5dd0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe100012960 tx=0x7fe100012e20 comp rx=0 tx=0).stop 2024-04-21T08:54:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.115+0000 7fe1199b6740 1 -- 172.21.15.28:0/2728137771 shutdown_connections 2024-04-21T08:54:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.115+0000 7fe1199b6740 1 --2- 172.21.15.28:0/2728137771 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5579a2eb9180 0x5579a2ebb5b0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.115+0000 7fe1199b6740 1 --2- 172.21.15.28:0/2728137771 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2ea90e0 0x5579a2ea6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.115+0000 7fe0effff640 0 client.16590 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:07.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.115+0000 7fe1199b6740 1 --2- 172.21.15.28:0/2728137771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea08d0 0x5579a2ea5dd0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.115+0000 7fe1199b6740 1 --2- 172.21.15.28:0/2728137771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2e9fb60 0x5579a2ea8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.115+0000 7fe1199b6740 1 -- 172.21.15.28:0/2728137771 >> 172.21.15.28:0/2728137771 conn(0x5579a2e66c90 msgr2=0x5579a2e9c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.115+0000 7fe1199b6740 1 -- 172.21.15.28:0/2728137771 shutdown_connections 2024-04-21T08:54:07.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.115+0000 7fe1199b6740 1 -- 172.21.15.28:0/2728137771 wait complete. 2024-04-21T08:54:07.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.115+0000 7fe1199b6740 20 client.16590 trim_cache size 0 max 16384 2024-04-21T08:54:07.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.130+0000 7fe1199b6740 1 Processor -- start 2024-04-21T08:54:07.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.130+0000 7fe1199b6740 1 -- start start 2024-04-21T08:54:07.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe1199b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2e9fb60 0x5579a2eb1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe1199b6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2ea08d0 0x5579a2e9c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe1199b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea90e0 0x5579a2ea8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe1199b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101b1b50 con 0x5579a2ea90e0 2024-04-21T08:54:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe1199b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101af420 con 0x5579a2e9fb60 2024-04-21T08:54:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe1199b6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101ad270 con 0x5579a2ea08d0 2024-04-21T08:54:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe115367640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea90e0 0x5579a2ea8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe115367640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea90e0 0x5579a2ea8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38156/0 (socket says 172.21.15.28:38156) 2024-04-21T08:54:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe115367640 1 -- 172.21.15.28:0/391011275 learned_addr learned my addr 172.21.15.28:0/391011275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe114b66640 1 --2- 172.21.15.28:0/391011275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2e9fb60 0x5579a2eb1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe10ffff640 1 --2- 172.21.15.28:0/391011275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2ea08d0 0x5579a2e9c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe115367640 1 -- 172.21.15.28:0/391011275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2ea08d0 msgr2=0x5579a2e9c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe115367640 1 --2- 172.21.15.28:0/391011275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2ea08d0 0x5579a2e9c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe115367640 1 -- 172.21.15.28:0/391011275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2e9fb60 msgr2=0x5579a2eb1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe115367640 1 --2- 172.21.15.28:0/391011275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2e9fb60 0x5579a2eb1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe115367640 1 -- 172.21.15.28:0/391011275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579a2e95f30 con 0x5579a2ea90e0 2024-04-21T08:54:07.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe114b66640 1 --2- 172.21.15.28:0/391011275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2e9fb60 0x5579a2eb1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe115367640 1 --2- 172.21.15.28:0/391011275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea90e0 0x5579a2ea8500 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe108036040 tx=0x7fe108051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68f7e4ff31d862a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe1067fc640 1 -- 172.21.15.28:0/391011275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe108046070 con 0x5579a2ea90e0 2024-04-21T08:54:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe1067fc640 1 -- 172.21.15.28:0/391011275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1080044a0 con 0x5579a2ea90e0 2024-04-21T08:54:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.131+0000 7fe1067fc640 1 -- 172.21.15.28:0/391011275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe108005310 con 0x5579a2ea90e0 2024-04-21T08:54:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.132+0000 7fe1199b6740 1 -- 172.21.15.28:0/391011275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea90e0 msgr2=0x5579a2ea8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.132+0000 7fe1199b6740 1 --2- 172.21.15.28:0/391011275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea90e0 0x5579a2ea8500 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe108036040 tx=0x7fe108051000 comp rx=0 tx=0).stop 2024-04-21T08:54:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.132+0000 7fe1199b6740 1 -- 172.21.15.28:0/391011275 shutdown_connections 2024-04-21T08:54:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.132+0000 7fe1199b6740 1 --2- 172.21.15.28:0/391011275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2ea90e0 0x5579a2ea8500 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.132+0000 7fe1199b6740 1 --2- 172.21.15.28:0/391011275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2ea08d0 0x5579a2e9c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.132+0000 7fe1199b6740 1 --2- 172.21.15.28:0/391011275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2e9fb60 0x5579a2eb1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.132+0000 7fe1199b6740 1 -- 172.21.15.28:0/391011275 >> 172.21.15.28:0/391011275 conn(0x5579a2e66c90 msgr2=0x5579a2e9bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.132+0000 7fe1199b6740 1 -- 172.21.15.28:0/391011275 shutdown_connections 2024-04-21T08:54:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.132+0000 7fe1199b6740 1 -- 172.21.15.28:0/391011275 wait complete. 2024-04-21T08:54:07.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.132+0000 7fe1199b6740 1 -- start start 2024-04-21T08:54:07.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe1199b6740 1 Processor -- start 2024-04-21T08:54:07.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe1199b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2e9fb60 0x5579a2ea2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe1199b6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2ea08d0 0x5579a2ea2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe1199b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2ea90e0 0x5579a2ea2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe1199b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101ad8e0 con 0x5579a2e9fb60 2024-04-21T08:54:07.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe1199b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101aed50 con 0x5579a2ea90e0 2024-04-21T08:54:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe1199b6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101adfb0 con 0x5579a2ea08d0 2024-04-21T08:54:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe1199b6740 10 client.0 _ll_register_callbacks cb 0x7fe100009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe114b66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2e9fb60 0x5579a2ea2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe114b66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2e9fb60 0x5579a2ea2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38172/0 (socket says 172.21.15.28:38172) 2024-04-21T08:54:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe114b66640 1 -- 172.21.15.28:0/4020545783 learned_addr learned my addr 172.21.15.28:0/4020545783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe10ffff640 1 --2- 172.21.15.28:0/4020545783 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2ea08d0 0x5579a2ea2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe115367640 1 --2- 172.21.15.28:0/4020545783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2ea90e0 0x5579a2ea2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe114b66640 1 -- 172.21.15.28:0/4020545783 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2ea08d0 msgr2=0x5579a2ea2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe114b66640 1 --2- 172.21.15.28:0/4020545783 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2ea08d0 0x5579a2ea2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe114b66640 1 -- 172.21.15.28:0/4020545783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2ea90e0 msgr2=0x5579a2ea2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe114b66640 1 --2- 172.21.15.28:0/4020545783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2ea90e0 0x5579a2ea2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.133+0000 7fe114b66640 1 -- 172.21.15.28:0/4020545783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579a2ea3520 con 0x5579a2e9fb60 2024-04-21T08:54:07.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe114b66640 1 --2- 172.21.15.28:0/4020545783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2e9fb60 0x5579a2ea2230 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe0f80ce320 tx=0x7fe0f800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0d56a88d35526c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe0effff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe0effff640 1 -- 172.21.15.28:0/4020545783 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f80a0ec0 con 0x5579a2e9fb60 2024-04-21T08:54:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe0effff640 1 -- 172.21.15.28:0/4020545783 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe0f8010a90 con 0x5579a2e9fb60 2024-04-21T08:54:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe1199b6740 1 -- 172.21.15.28:0/4020545783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1080044a0 con 0x5579a2e9fb60 2024-04-21T08:54:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe0effff640 1 -- 172.21.15.28:0/4020545783 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe0f80a5780 con 0x5579a2e9fb60 2024-04-21T08:54:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe1199b6740 10 client.16596 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe1199b6740 1 -- 172.21.15.28:0/4020545783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5579a2f24010 con 0x5579a2e9fb60 2024-04-21T08:54:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe0effff640 1 -- 172.21.15.28:0/4020545783 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1080044a0 con 0x5579a2e9fb60 2024-04-21T08:54:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe0effff640 1 -- 172.21.15.28:0/4020545783 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fe0f809bf60 con 0x5579a2e9fb60 2024-04-21T08:54:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe1199b6740 10 client.16596 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe1199b6740 10 client.16596 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe1199b6740 20 client.16596 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe1199b6740 1 --2- 172.21.15.28:0/4020545783 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5579a2f24730 0x5579a2f26b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe1199b6740 4 client.16596 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:54:07.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.134+0000 7fe1199b6740 1 -- 172.21.15.28:0/4020545783 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5579a2da43d0 con 0x5579a2f24730 2024-04-21T08:54:07.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.135+0000 7fe10ffff640 1 --2- 172.21.15.28:0/4020545783 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5579a2f24730 0x5579a2f26b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.135+0000 7fe10ffff640 1 --2- 172.21.15.28:0/4020545783 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5579a2f24730 0x5579a2f26b20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f2e8f03f6a516da server_cookie=2eadd8f17444fd7d in_seq=0 out_seq=0 2024-04-21T08:54:07.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.135+0000 7fe0effff640 10 client.16596 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.149+0000 7fe0effff640 1 -- 172.21.15.28:0/4020545783 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5579a2da43d0 con 0x5579a2f24730 2024-04-21T08:54:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.149+0000 7fe0effff640 10 client.16596 handle_command_reply: tid=0 2024-04-21T08:54:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.149+0000 7fe1199b6740 1 client.16596 shutdown 2024-04-21T08:54:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.149+0000 7fe1199b6740 1 -- 172.21.15.28:0/4020545783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2e9fb60 msgr2=0x5579a2ea2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.149+0000 7fe1199b6740 1 --2- 172.21.15.28:0/4020545783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2e9fb60 0x5579a2ea2230 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe0f80ce320 tx=0x7fe0f800cf20 comp rx=0 tx=0).stop 2024-04-21T08:54:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.149+0000 7fe1199b6740 1 -- 172.21.15.28:0/4020545783 shutdown_connections 2024-04-21T08:54:07.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.149+0000 7fe1199b6740 1 --2- 172.21.15.28:0/4020545783 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5579a2f24730 0x5579a2f26b20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.149+0000 7fe1199b6740 1 --2- 172.21.15.28:0/4020545783 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5579a2ea90e0 0x5579a2ea2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.149+0000 7fe1199b6740 1 --2- 172.21.15.28:0/4020545783 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5579a2ea08d0 0x5579a2ea2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.149+0000 7fe0effff640 0 client.16596 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.149+0000 7fe1199b6740 1 --2- 172.21.15.28:0/4020545783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5579a2e9fb60 0x5579a2ea2230 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.149+0000 7fe1199b6740 1 -- 172.21.15.28:0/4020545783 >> 172.21.15.28:0/4020545783 conn(0x5579a2e66c90 msgr2=0x5579a2e998d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.149+0000 7fe1199b6740 1 -- 172.21.15.28:0/4020545783 shutdown_connections 2024-04-21T08:54:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.149+0000 7fe1199b6740 1 -- 172.21.15.28:0/4020545783 wait complete. 2024-04-21T08:54:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.149+0000 7fe1199b6740 20 client.16596 trim_cache size 0 max 16384 2024-04-21T08:54:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:54:07.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:54:07.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:54:07.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:54:07.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.150+0000 7fe116dca640 1 -- 172.21.15.28:0/1301912179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0fc0a3dc0 msgr2=0x7fe0fc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.150+0000 7fe116dca640 1 --2- 172.21.15.28:0/1301912179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0fc0a3dc0 0x7fe0fc0a6280 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe10804c000 tx=0x7fe108049000 comp rx=0 tx=0).stop 2024-04-21T08:54:07.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.150+0000 7fe116dca640 1 -- 172.21.15.28:0/1301912179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1101010c0 msgr2=0x7fe110199500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.150+0000 7fe116dca640 1 --2- 172.21.15.28:0/1301912179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1101010c0 0x7fe110199500 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe0f800e9f0 tx=0x7fe0f800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.150+0000 7fe116dca640 1 -- 172.21.15.28:0/1301912179 shutdown_connections 2024-04-21T08:54:07.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.150+0000 7fe116dca640 1 --2- 172.21.15.28:0/1301912179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0fc0a3dc0 0x7fe0fc0a6280 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.150+0000 7fe116dca640 1 --2- 172.21.15.28:0/1301912179 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe110199fd0 0x7fe1101a4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.150+0000 7fe116dca640 1 --2- 172.21.15.28:0/1301912179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe110106190 0x7fe110199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.150+0000 7fe116dca640 1 --2- 172.21.15.28:0/1301912179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1101010c0 0x7fe110199500 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.150+0000 7fe116dca640 1 -- 172.21.15.28:0/1301912179 >> 172.21.15.28:0/1301912179 conn(0x7fe110077f60 msgr2=0x7fe1100fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.150+0000 7fe116dca640 1 -- 172.21.15.28:0/1301912179 shutdown_connections 2024-04-21T08:54:07.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:07.150+0000 7fe116dca640 1 -- 172.21.15.28:0/1301912179 wait complete. 2024-04-21T08:54:08.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:08 smithi152 ceph-mon[31401]: pgmap v91: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 351 KiB/s wr, 90 op/s 2024-04-21T08:54:08.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:08 smithi028 ceph-mon[28003]: pgmap v91: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 351 KiB/s wr, 90 op/s 2024-04-21T08:54:08.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:08 smithi171 ceph-mon[31358]: pgmap v91: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 351 KiB/s wr, 90 op/s 2024-04-21T08:54:10.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:10 smithi152 ceph-mon[31401]: pgmap v92: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 351 KiB/s wr, 90 op/s 2024-04-21T08:54:10.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:10 smithi028 ceph-mon[28003]: pgmap v92: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 351 KiB/s wr, 90 op/s 2024-04-21T08:54:10.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:10 smithi171 ceph-mon[31358]: pgmap v92: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 351 KiB/s wr, 90 op/s 2024-04-21T08:54:12.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:12 smithi152 ceph-mon[31401]: pgmap v93: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 42 KiB/s wr, 22 op/s 2024-04-21T08:54:12.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:12 smithi028 ceph-mon[28003]: pgmap v93: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 42 KiB/s wr, 22 op/s 2024-04-21T08:54:12.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:12 smithi171 ceph-mon[31358]: pgmap v93: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 42 KiB/s wr, 22 op/s 2024-04-21T08:54:14.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:54:14.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:54:14.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.455+0000 7f3acba88640 1 -- 172.21.15.28:0/3471214698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac4101660 msgr2=0x7f3ac410e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.455+0000 7f3acba88640 1 --2- 172.21.15.28:0/3471214698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac4101660 0x7f3ac410e3b0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4009900 tx=0x7f3ab402bc30 comp rx=0 tx=0).stop 2024-04-21T08:54:14.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.455+0000 7f3acba88640 1 -- 172.21.15.28:0/3471214698 shutdown_connections 2024-04-21T08:54:14.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.455+0000 7f3acba88640 1 --2- 172.21.15.28:0/3471214698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ac410e8f0 0x7f3ac4110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.455+0000 7f3acba88640 1 --2- 172.21.15.28:0/3471214698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac4101660 0x7f3ac410e3b0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.455+0000 7f3acba88640 1 --2- 172.21.15.28:0/3471214698 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ac4100d40 0x7f3ac4101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.455+0000 7f3acba88640 1 -- 172.21.15.28:0/3471214698 >> 172.21.15.28:0/3471214698 conn(0x7f3ac4078000 msgr2=0x7f3ac4078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:14.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.456+0000 7f3acba88640 1 -- 172.21.15.28:0/3471214698 shutdown_connections 2024-04-21T08:54:14.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.456+0000 7f3acba88640 1 -- 172.21.15.28:0/3471214698 wait complete. 2024-04-21T08:54:14.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.456+0000 7f3acba88640 1 Processor -- start 2024-04-21T08:54:14.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.456+0000 7f3acba88640 1 -- start start 2024-04-21T08:54:14.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.456+0000 7f3acba88640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ac4100d40 0x7f3ac41a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.456+0000 7f3acba88640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac4101660 0x7f3ac41a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.456+0000 7f3acba88640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ac410e8f0 0x7f3ac419a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.456+0000 7f3acba88640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac41143d0 con 0x7f3ac4101660 2024-04-21T08:54:14.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.456+0000 7f3acba88640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac4114250 con 0x7f3ac4100d40 2024-04-21T08:54:14.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.456+0000 7f3acba88640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac4114550 con 0x7f3ac410e8f0 2024-04-21T08:54:14.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.456+0000 7f3ac9023640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac4101660 0x7f3ac41a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.457+0000 7f3ac9023640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac4101660 0x7f3ac41a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48952/0 (socket says 172.21.15.28:48952) 2024-04-21T08:54:14.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.457+0000 7f3ac9023640 1 -- 172.21.15.28:0/2101884885 learned_addr learned my addr 172.21.15.28:0/2101884885 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:14.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.457+0000 7f3ac9824640 1 --2- 172.21.15.28:0/2101884885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ac4100d40 0x7f3ac41a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.457+0000 7f3aca025640 1 --2- 172.21.15.28:0/2101884885 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ac410e8f0 0x7f3ac419a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.457+0000 7f3ac9023640 1 -- 172.21.15.28:0/2101884885 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ac410e8f0 msgr2=0x7f3ac419a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.457+0000 7f3ac9023640 1 --2- 172.21.15.28:0/2101884885 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ac410e8f0 0x7f3ac419a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.457+0000 7f3ac9023640 1 -- 172.21.15.28:0/2101884885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ac4100d40 msgr2=0x7f3ac41a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.457+0000 7f3ac9023640 1 --2- 172.21.15.28:0/2101884885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ac4100d40 0x7f3ac41a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.457+0000 7f3ac9023640 1 -- 172.21.15.28:0/2101884885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac419af80 con 0x7f3ac4101660 2024-04-21T08:54:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.457+0000 7f3ac9023640 1 --2- 172.21.15.28:0/2101884885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac4101660 0x7f3ac41a0b80 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4005940 tx=0x7f3ab4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.457+0000 7f3abaffd640 1 -- 172.21.15.28:0/2101884885 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4043020 con 0x7f3ac4101660 2024-04-21T08:54:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.457+0000 7f3abaffd640 1 -- 172.21.15.28:0/2101884885 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ab4004440 con 0x7f3ac4101660 2024-04-21T08:54:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.457+0000 7f3acba88640 1 -- 172.21.15.28:0/2101884885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac419b210 con 0x7f3ac4101660 2024-04-21T08:54:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.457+0000 7f3abaffd640 1 -- 172.21.15.28:0/2101884885 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4003c20 con 0x7f3ac4101660 2024-04-21T08:54:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.457+0000 7f3acba88640 1 -- 172.21.15.28:0/2101884885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac41a7490 con 0x7f3ac4101660 2024-04-21T08:54:14.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.459+0000 7f3abaffd640 1 -- 172.21.15.28:0/2101884885 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3ab4046020 con 0x7f3ac4101660 2024-04-21T08:54:14.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.460+0000 7f3ace67f740 1 Processor -- start 2024-04-21T08:54:14.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.460+0000 7f3ace67f740 1 -- start start 2024-04-21T08:54:14.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.460+0000 7f3ace67f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390580 0x5587b8390960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.460+0000 7f3ace67f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8390ea0 0x5587b8399540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.460+0000 7f3ace67f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8399a80 0x5587b839be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.460+0000 7f3ace67f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5587b82f51e0 con 0x5587b8390580 2024-04-21T08:54:14.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.460+0000 7f3ace67f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5587b8292390 con 0x5587b8399a80 2024-04-21T08:54:14.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.460+0000 7f3ace67f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5587b82f11a0 con 0x5587b8390ea0 2024-04-21T08:54:14.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.460+0000 7f3abaffd640 1 --2- 172.21.15.28:0/2101884885 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3ab00a3da0 0x7f3ab00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3aca025640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390580 0x5587b8390960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3aca025640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390580 0x5587b8390960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48966/0 (socket says 172.21.15.28:48966) 2024-04-21T08:54:14.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3aca025640 1 -- 172.21.15.28:0/2357102991 learned_addr learned my addr 172.21.15.28:0/2357102991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:14.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3aca025640 1 --2- 172.21.15.28:0/2101884885 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3ab00a3da0 0x7f3ab00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3abaffd640 1 -- 172.21.15.28:0/2101884885 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f3ab40f2d40 con 0x7f3ac4101660 2024-04-21T08:54:14.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3ac9023640 1 --2- 172.21.15.28:0/2357102991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8399a80 0x5587b839be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3ac9824640 1 --2- 172.21.15.28:0/2357102991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8390ea0 0x5587b8399540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3aca025640 1 -- 172.21.15.28:0/2357102991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8390ea0 msgr2=0x5587b8399540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3aca025640 1 --2- 172.21.15.28:0/2357102991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8390ea0 0x5587b8399540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3aca025640 1 -- 172.21.15.28:0/2357102991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8399a80 msgr2=0x5587b839be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3aca025640 1 --2- 172.21.15.28:0/2357102991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8399a80 0x5587b839be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3aca025640 1 -- 172.21.15.28:0/2357102991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587b80d17d0 con 0x5587b8390580 2024-04-21T08:54:14.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3ac9824640 1 --2- 172.21.15.28:0/2357102991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8390ea0 0x5587b8399540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:14.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3aca025640 1 --2- 172.21.15.28:0/2357102991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390580 0x5587b8390960 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f3abc013d40 tx=0x7f3abc0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236bbe2a5fa0b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3aca025640 1 --2- 172.21.15.28:0/2101884885 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3ab00a3da0 0x7f3ab00a6260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3abc005cc0 tx=0x7f3abc01b000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.461+0000 7f3aaffff640 1 -- 172.21.15.28:0/2357102991 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc021070 con 0x5587b8390580 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.462+0000 7f3aaffff640 1 -- 172.21.15.28:0/2357102991 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3abc00aac0 con 0x5587b8390580 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.462+0000 7f3aaffff640 1 -- 172.21.15.28:0/2357102991 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc004020 con 0x5587b8390580 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.462+0000 7f3ace67f740 1 -- 172.21.15.28:0/2357102991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390580 msgr2=0x5587b8390960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.462+0000 7f3ace67f740 1 --2- 172.21.15.28:0/2357102991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390580 0x5587b8390960 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f3abc013d40 tx=0x7f3abc0107a0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.462+0000 7f3ace67f740 1 -- 172.21.15.28:0/2357102991 shutdown_connections 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.462+0000 7f3ace67f740 1 --2- 172.21.15.28:0/2357102991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8399a80 0x5587b839be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.462+0000 7f3ace67f740 1 --2- 172.21.15.28:0/2357102991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8390ea0 0x5587b8399540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.462+0000 7f3ace67f740 1 --2- 172.21.15.28:0/2357102991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390580 0x5587b8390960 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.462+0000 7f3ace67f740 1 -- 172.21.15.28:0/2357102991 >> 172.21.15.28:0/2357102991 conn(0x5587b838acb0 msgr2=0x5587b838b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.462+0000 7f3ace67f740 1 -- 172.21.15.28:0/2357102991 shutdown_connections 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.462+0000 7f3ace67f740 1 -- 172.21.15.28:0/2357102991 wait complete. 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.463+0000 7f3ace67f740 1 -- start start 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.463+0000 7f3ace67f740 1 Processor -- start 2024-04-21T08:54:14.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.463+0000 7f3ace67f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8390580 0x5587b8396aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.463+0000 7f3ace67f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390ea0 0x5587b8396fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.463+0000 7f3ace67f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8399a80 0x5587b8397520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.463+0000 7f3ace67f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5587b839c810 con 0x5587b8390ea0 2024-04-21T08:54:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.463+0000 7f3ace67f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5587b82ae2f0 con 0x5587b8390580 2024-04-21T08:54:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.463+0000 7f3ace67f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5587b839c990 con 0x5587b8399a80 2024-04-21T08:54:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.463+0000 7f3ace67f740 10 client.0 _ll_register_callbacks cb 0x5587b82bb580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.463+0000 7f3ac9023640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390ea0 0x5587b8396fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.463+0000 7f3ac9023640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390ea0 0x5587b8396fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48982/0 (socket says 172.21.15.28:48982) 2024-04-21T08:54:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.463+0000 7f3ac9023640 1 -- 172.21.15.28:0/3066176234 learned_addr learned my addr 172.21.15.28:0/3066176234 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.463+0000 7f3ac9824640 1 --2- 172.21.15.28:0/3066176234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8390580 0x5587b8396aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.464+0000 7f3aca025640 1 --2- 172.21.15.28:0/3066176234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8399a80 0x5587b8397520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.464+0000 7f3ac9023640 1 -- 172.21.15.28:0/3066176234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8399a80 msgr2=0x5587b8397520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.464+0000 7f3ac9023640 1 --2- 172.21.15.28:0/3066176234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8399a80 0x5587b8397520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.464+0000 7f3ac9023640 1 -- 172.21.15.28:0/3066176234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8390580 msgr2=0x5587b8396aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.464+0000 7f3ac9023640 1 --2- 172.21.15.28:0/3066176234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8390580 0x5587b8396aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.464+0000 7f3ac9023640 1 -- 172.21.15.28:0/3066176234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587b80073f0 con 0x5587b8390ea0 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.464+0000 7f3ac9023640 1 --2- 172.21.15.28:0/3066176234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390ea0 0x5587b8396fe0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3ab40bae50 tx=0x7f3ab40036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11648757f8517aee server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.464+0000 7f3aad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.464+0000 7f3aad7fa640 1 -- 172.21.15.28:0/3066176234 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab40d9060 con 0x5587b8390ea0 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.464+0000 7f3aad7fa640 1 -- 172.21.15.28:0/3066176234 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f3ab40f0310 con 0x5587b8390ea0 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.464+0000 7f3ace67f740 1 -- 172.21.15.28:0/3066176234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3abc00aac0 con 0x5587b8390ea0 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.464+0000 7f3aad7fa640 1 -- 172.21.15.28:0/3066176234 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3ab40f0510 con 0x5587b8390ea0 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.464+0000 7f3ace67f740 10 client.16608 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.464+0000 7f3ace67f740 1 -- 172.21.15.28:0/3066176234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5587b7fc70f0 con 0x5587b8390ea0 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.464+0000 7f3aad7fa640 1 -- 172.21.15.28:0/3066176234 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3abc00aac0 con 0x5587b8390ea0 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.465+0000 7f3aad7fa640 1 -- 172.21.15.28:0/3066176234 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3ab40c71d0 con 0x5587b8390ea0 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.465+0000 7f3ace67f740 10 client.16608 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.465+0000 7f3ace67f740 10 client.16608 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.465+0000 7f3ace67f740 20 client.16608 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.465+0000 7f3ace67f740 1 --2- 172.21.15.28:0/3066176234 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5587b83a9900 0x5587b83abd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.465+0000 7f3ace67f740 4 client.16608 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.465+0000 7f3ace67f740 1 -- 172.21.15.28:0/3066176234 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587b8302c00 con 0x5587b83a9900 2024-04-21T08:54:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.465+0000 7f3ac9824640 1 --2- 172.21.15.28:0/3066176234 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5587b83a9900 0x5587b83abd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.465+0000 7f3aad7fa640 10 client.16608 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.465+0000 7f3ac9824640 1 --2- 172.21.15.28:0/3066176234 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5587b83a9900 0x5587b83abd60 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1efca93dbfa4075c server_cookie=87beb1d55221bd5a in_seq=0 out_seq=0 2024-04-21T08:54:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.466+0000 7f3aad7fa640 1 -- 172.21.15.28:0/3066176234 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5587b8302c00 con 0x5587b83a9900 2024-04-21T08:54:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.466+0000 7f3aad7fa640 10 client.16608 handle_command_reply: tid=0 2024-04-21T08:54:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.466+0000 7f3ace67f740 1 client.16608 shutdown 2024-04-21T08:54:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.466+0000 7f3ace67f740 1 -- 172.21.15.28:0/3066176234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390ea0 msgr2=0x5587b8396fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.466+0000 7f3ace67f740 1 --2- 172.21.15.28:0/3066176234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390ea0 0x5587b8396fe0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3ab40bae50 tx=0x7f3ab40036f0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.466+0000 7f3ace67f740 1 -- 172.21.15.28:0/3066176234 shutdown_connections 2024-04-21T08:54:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.466+0000 7f3ace67f740 1 --2- 172.21.15.28:0/3066176234 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5587b83a9900 0x5587b83abd60 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.466+0000 7f3ace67f740 1 --2- 172.21.15.28:0/3066176234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8399a80 0x5587b8397520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.466+0000 7f3ace67f740 1 --2- 172.21.15.28:0/3066176234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390ea0 0x5587b8396fe0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.466+0000 7f3aad7fa640 0 client.16608 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:14.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.466+0000 7f3ace67f740 1 --2- 172.21.15.28:0/3066176234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8390580 0x5587b8396aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.466+0000 7f3ace67f740 1 -- 172.21.15.28:0/3066176234 >> 172.21.15.28:0/3066176234 conn(0x5587b838acb0 msgr2=0x5587b838ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:14.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.467+0000 7f3ace67f740 1 -- 172.21.15.28:0/3066176234 shutdown_connections 2024-04-21T08:54:14.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.467+0000 7f3ace67f740 1 -- 172.21.15.28:0/3066176234 wait complete. 2024-04-21T08:54:14.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.467+0000 7f3ace67f740 20 client.16608 trim_cache size 0 max 16384 2024-04-21T08:54:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3ace67f740 1 Processor -- start 2024-04-21T08:54:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3ace67f740 1 -- start start 2024-04-21T08:54:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3ace67f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8390580 0x5587b83991d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3ace67f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8390ea0 0x5587b83959e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3ace67f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8399a80 0x5587b83a5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3ace67f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac41b3d20 con 0x5587b8399a80 2024-04-21T08:54:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3ace67f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac41b15f0 con 0x5587b8390ea0 2024-04-21T08:54:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3ace67f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac41af440 con 0x5587b8390580 2024-04-21T08:54:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3aca025640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8399a80 0x5587b83a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3aca025640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8399a80 0x5587b83a5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48994/0 (socket says 172.21.15.28:48994) 2024-04-21T08:54:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3aca025640 1 -- 172.21.15.28:0/1475563833 learned_addr learned my addr 172.21.15.28:0/1475563833 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3ac9824640 1 --2- 172.21.15.28:0/1475563833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8390580 0x5587b83991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3aca025640 1 -- 172.21.15.28:0/1475563833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8390580 msgr2=0x5587b83991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3aca025640 1 --2- 172.21.15.28:0/1475563833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8390580 0x5587b83991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3aca025640 1 -- 172.21.15.28:0/1475563833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8390ea0 msgr2=0x5587b83959e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3aca025640 1 --2- 172.21.15.28:0/1475563833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8390ea0 0x5587b83959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3aca025640 1 -- 172.21.15.28:0/1475563833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587b83ac3e0 con 0x5587b8399a80 2024-04-21T08:54:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3ac9824640 1 --2- 172.21.15.28:0/1475563833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8390580 0x5587b83991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.485+0000 7f3aca025640 1 --2- 172.21.15.28:0/1475563833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8399a80 0x5587b83a5700 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00a300 tx=0x7f3abc00a470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1eb27462314df83 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.486+0000 7f3aaf7fe640 1 -- 172.21.15.28:0/1475563833 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc029030 con 0x5587b8399a80 2024-04-21T08:54:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.486+0000 7f3aaf7fe640 1 -- 172.21.15.28:0/1475563833 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3abc00aac0 con 0x5587b8399a80 2024-04-21T08:54:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.486+0000 7f3aaf7fe640 1 -- 172.21.15.28:0/1475563833 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc021050 con 0x5587b8399a80 2024-04-21T08:54:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.486+0000 7f3ace67f740 1 -- 172.21.15.28:0/1475563833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8399a80 msgr2=0x5587b83a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.486+0000 7f3ace67f740 1 --2- 172.21.15.28:0/1475563833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8399a80 0x5587b83a5700 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3abc00a300 tx=0x7f3abc00a470 comp rx=0 tx=0).stop 2024-04-21T08:54:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.486+0000 7f3ace67f740 1 -- 172.21.15.28:0/1475563833 shutdown_connections 2024-04-21T08:54:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.486+0000 7f3ace67f740 1 --2- 172.21.15.28:0/1475563833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8399a80 0x5587b83a5700 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.486+0000 7f3ace67f740 1 --2- 172.21.15.28:0/1475563833 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8390ea0 0x5587b83959e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.486+0000 7f3ace67f740 1 --2- 172.21.15.28:0/1475563833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8390580 0x5587b83991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.486+0000 7f3ace67f740 1 -- 172.21.15.28:0/1475563833 >> 172.21.15.28:0/1475563833 conn(0x5587b838acb0 msgr2=0x5587b838c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.486+0000 7f3ace67f740 1 -- 172.21.15.28:0/1475563833 shutdown_connections 2024-04-21T08:54:14.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.486+0000 7f3ace67f740 1 -- 172.21.15.28:0/1475563833 wait complete. 2024-04-21T08:54:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ace67f740 1 -- start start 2024-04-21T08:54:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ace67f740 1 Processor -- start 2024-04-21T08:54:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ace67f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8390580 0x5587b83a1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ace67f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390ea0 0x5587b83a2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ace67f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8399a80 0x5587b839ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ace67f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac41afab0 con 0x5587b8390ea0 2024-04-21T08:54:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ace67f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac41b0f20 con 0x5587b8390580 2024-04-21T08:54:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ace67f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac41b0180 con 0x5587b8399a80 2024-04-21T08:54:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ace67f740 10 client.0 _ll_register_callbacks cb 0x7f3ab40df0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ac9023640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390ea0 0x5587b83a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ac9023640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390ea0 0x5587b83a2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49004/0 (socket says 172.21.15.28:49004) 2024-04-21T08:54:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ac9023640 1 -- 172.21.15.28:0/4074313314 learned_addr learned my addr 172.21.15.28:0/4074313314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ac9824640 1 --2- 172.21.15.28:0/4074313314 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8390580 0x5587b83a1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3aca025640 1 --2- 172.21.15.28:0/4074313314 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8399a80 0x5587b839ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ac9023640 1 -- 172.21.15.28:0/4074313314 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8399a80 msgr2=0x5587b839ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ac9023640 1 --2- 172.21.15.28:0/4074313314 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8399a80 0x5587b839ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ac9023640 1 -- 172.21.15.28:0/4074313314 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8390580 msgr2=0x5587b83a1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.487+0000 7f3ac9023640 1 --2- 172.21.15.28:0/4074313314 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8390580 0x5587b83a1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3ac9023640 1 -- 172.21.15.28:0/4074313314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587b839f280 con 0x5587b8390ea0 2024-04-21T08:54:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3aca025640 1 --2- 172.21.15.28:0/4074313314 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8399a80 0x5587b839ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3ac9023640 1 --2- 172.21.15.28:0/4074313314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390ea0 0x5587b83a2150 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3ab40ddcf0 tx=0x7f3ab402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fad062b922e69fe server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3aad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3aad7fa640 1 -- 172.21.15.28:0/4074313314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab40be530 con 0x5587b8390ea0 2024-04-21T08:54:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3aad7fa640 1 -- 172.21.15.28:0/4074313314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f3ab40f0480 con 0x5587b8390ea0 2024-04-21T08:54:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3ace67f740 1 -- 172.21.15.28:0/4074313314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3abc00aac0 con 0x5587b8390ea0 2024-04-21T08:54:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3aad7fa640 1 -- 172.21.15.28:0/4074313314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3ab40bac90 con 0x5587b8390ea0 2024-04-21T08:54:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3ace67f740 10 client.16614 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:14.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3ace67f740 1 -- 172.21.15.28:0/4074313314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5587b84108a0 con 0x5587b8390ea0 2024-04-21T08:54:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3aad7fa640 1 -- 172.21.15.28:0/4074313314 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3abc00aac0 con 0x5587b8390ea0 2024-04-21T08:54:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3aad7fa640 1 -- 172.21.15.28:0/4074313314 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3ab40ba740 con 0x5587b8390ea0 2024-04-21T08:54:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3ace67f740 10 client.16614 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3ace67f740 10 client.16614 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3ace67f740 20 client.16614 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3ace67f740 1 --2- 172.21.15.28:0/4074313314 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5587b8410eb0 0x5587b84132a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3ace67f740 4 client.16614 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:54:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.488+0000 7f3ace67f740 1 -- 172.21.15.28:0/4074313314 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5587b82fca70 con 0x5587b8410eb0 2024-04-21T08:54:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.489+0000 7f3ac9824640 1 --2- 172.21.15.28:0/4074313314 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5587b8410eb0 0x5587b84132a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.489+0000 7f3ac9824640 1 --2- 172.21.15.28:0/4074313314 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5587b8410eb0 0x5587b84132a0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48f8e9f8f7af5e6d server_cookie=287785d75c0a61f4 in_seq=0 out_seq=0 2024-04-21T08:54:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.489+0000 7f3aad7fa640 10 client.16614 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.489+0000 7f3aad7fa640 1 -- 172.21.15.28:0/4074313314 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5587b82fca70 con 0x5587b8410eb0 2024-04-21T08:54:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.489+0000 7f3aad7fa640 10 client.16614 handle_command_reply: tid=0 2024-04-21T08:54:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.489+0000 7f3ace67f740 1 client.16614 shutdown 2024-04-21T08:54:14.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.489+0000 7f3ace67f740 1 -- 172.21.15.28:0/4074313314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390ea0 msgr2=0x5587b83a2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.489+0000 7f3ace67f740 1 --2- 172.21.15.28:0/4074313314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390ea0 0x5587b83a2150 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3ab40ddcf0 tx=0x7f3ab402bc30 comp rx=0 tx=0).stop 2024-04-21T08:54:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.489+0000 7f3ace67f740 1 -- 172.21.15.28:0/4074313314 shutdown_connections 2024-04-21T08:54:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3ace67f740 1 --2- 172.21.15.28:0/4074313314 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5587b8410eb0 0x5587b84132a0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3ace67f740 1 --2- 172.21.15.28:0/4074313314 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5587b8399a80 0x5587b839ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3ace67f740 1 --2- 172.21.15.28:0/4074313314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587b8390ea0 0x5587b83a2150 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3aad7fa640 0 client.16614 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3ace67f740 1 --2- 172.21.15.28:0/4074313314 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5587b8390580 0x5587b83a1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3ace67f740 1 -- 172.21.15.28:0/4074313314 >> 172.21.15.28:0/4074313314 conn(0x5587b838acb0 msgr2=0x5587b8396a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3ace67f740 1 -- 172.21.15.28:0/4074313314 shutdown_connections 2024-04-21T08:54:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3ace67f740 1 -- 172.21.15.28:0/4074313314 wait complete. 2024-04-21T08:54:14.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3ace67f740 20 client.16614 trim_cache size 0 max 16384 2024-04-21T08:54:14.496 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:54:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T08:54:14.497 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T08:54:14.497 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:54:14.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3acba88640 1 -- 172.21.15.28:0/2101884885 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3ab00a3da0 msgr2=0x7f3ab00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3acba88640 1 --2- 172.21.15.28:0/2101884885 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3ab00a3da0 0x7f3ab00a6260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3abc005cc0 tx=0x7f3abc01b000 comp rx=0 tx=0).stop 2024-04-21T08:54:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3acba88640 1 -- 172.21.15.28:0/2101884885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac4101660 msgr2=0x7f3ac41a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3acba88640 1 --2- 172.21.15.28:0/2101884885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac4101660 0x7f3ac41a0b80 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4005940 tx=0x7f3ab4004040 comp rx=0 tx=0).stop 2024-04-21T08:54:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3acba88640 1 -- 172.21.15.28:0/2101884885 shutdown_connections 2024-04-21T08:54:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3acba88640 1 --2- 172.21.15.28:0/2101884885 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3ab00a3da0 0x7f3ab00a6260 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3acba88640 1 --2- 172.21.15.28:0/2101884885 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ac410e8f0 0x7f3ac419a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3acba88640 1 --2- 172.21.15.28:0/2101884885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac4101660 0x7f3ac41a0b80 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3acba88640 1 --2- 172.21.15.28:0/2101884885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ac4100d40 0x7f3ac41a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3acba88640 1 -- 172.21.15.28:0/2101884885 >> 172.21.15.28:0/2101884885 conn(0x7f3ac4078000 msgr2=0x7f3ac410ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3acba88640 1 -- 172.21.15.28:0/2101884885 shutdown_connections 2024-04-21T08:54:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.490+0000 7f3acba88640 1 -- 172.21.15.28:0/2101884885 wait complete. 2024-04-21T08:54:14.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:14 smithi028 ceph-mon[28003]: pgmap v94: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T08:54:14.501 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:54:14.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:54:14.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:14 smithi152 ceph-mon[31401]: pgmap v94: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T08:54:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.604+0000 7f2995443640 1 -- 172.21.15.28:0/341162945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29901031d0 msgr2=0x7f29901035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.604+0000 7f2995443640 1 --2- 172.21.15.28:0/341162945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29901031d0 0x7f29901035b0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f2980009900 tx=0x7f298002bcd0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.604+0000 7f2995443640 1 -- 172.21.15.28:0/341162945 shutdown_connections 2024-04-21T08:54:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.604+0000 7f2995443640 1 --2- 172.21.15.28:0/341162945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f299010c5a0 0x7f299010e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.604+0000 7f2995443640 1 --2- 172.21.15.28:0/341162945 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2990103af0 0x7f299010bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.604+0000 7f2995443640 1 --2- 172.21.15.28:0/341162945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29901031d0 0x7f29901035b0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.604+0000 7f2995443640 1 -- 172.21.15.28:0/341162945 >> 172.21.15.28:0/341162945 conn(0x7f29900771d0 msgr2=0x7f29900775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.605+0000 7f2995443640 1 -- 172.21.15.28:0/341162945 shutdown_connections 2024-04-21T08:54:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.605+0000 7f2995443640 1 -- 172.21.15.28:0/341162945 wait complete. 2024-04-21T08:54:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.605+0000 7f2995443640 1 Processor -- start 2024-04-21T08:54:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.605+0000 7f2995443640 1 -- start start 2024-04-21T08:54:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.605+0000 7f2995443640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29901031d0 0x7f2990198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.605+0000 7f2995443640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990103af0 0x7f2990198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.605+0000 7f2995443640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f299010c5a0 0x7f299019cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.605+0000 7f2995443640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2990112220 con 0x7f2990103af0 2024-04-21T08:54:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.605+0000 7f2995443640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f29901120a0 con 0x7f29901031d0 2024-04-21T08:54:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.605+0000 7f2995443640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f29901123a0 con 0x7f299010c5a0 2024-04-21T08:54:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f298e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990103af0 0x7f2990198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f298e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990103af0 0x7f2990198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49028/0 (socket says 172.21.15.28:49028) 2024-04-21T08:54:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f298e7fc640 1 -- 172.21.15.28:0/2624495781 learned_addr learned my addr 172.21.15.28:0/2624495781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f298effd640 1 --2- 172.21.15.28:0/2624495781 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29901031d0 0x7f2990198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f298f7fe640 1 --2- 172.21.15.28:0/2624495781 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f299010c5a0 0x7f299019cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f298e7fc640 1 -- 172.21.15.28:0/2624495781 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f299010c5a0 msgr2=0x7f299019cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f298e7fc640 1 --2- 172.21.15.28:0/2624495781 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f299010c5a0 0x7f299019cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f298e7fc640 1 -- 172.21.15.28:0/2624495781 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29901031d0 msgr2=0x7f2990198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f298e7fc640 1 --2- 172.21.15.28:0/2624495781 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29901031d0 0x7f2990198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f298e7fc640 1 -- 172.21.15.28:0/2624495781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299019d710 con 0x7f2990103af0 2024-04-21T08:54:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f298e7fc640 1 --2- 172.21.15.28:0/2624495781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990103af0 0x7f2990198c60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f298400c890 tx=0x7f298400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f2977fff640 1 -- 172.21.15.28:0/2624495781 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984004220 con 0x7f2990103af0 2024-04-21T08:54:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f2995443640 1 -- 172.21.15.28:0/2624495781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299019da00 con 0x7f2990103af0 2024-04-21T08:54:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f2977fff640 1 -- 172.21.15.28:0/2624495781 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29840043c0 con 0x7f2990103af0 2024-04-21T08:54:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f2977fff640 1 -- 172.21.15.28:0/2624495781 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984004a00 con 0x7f2990103af0 2024-04-21T08:54:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.606+0000 7f2995443640 1 -- 172.21.15.28:0/2624495781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29901a5290 con 0x7f2990103af0 2024-04-21T08:54:14.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.608+0000 7f2995443640 1 -- 172.21.15.28:0/2624495781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2990069230 con 0x7f2990103af0 2024-04-21T08:54:14.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.609+0000 7f2977fff640 1 -- 172.21.15.28:0/2624495781 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f29840079b0 con 0x7f2990103af0 2024-04-21T08:54:14.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.609+0000 7f2977fff640 1 --2- 172.21.15.28:0/2624495781 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f29780a3d20 0x7f29780a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.609+0000 7f298effd640 1 --2- 172.21.15.28:0/2624495781 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f29780a3d20 0x7f29780a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.609+0000 7f2977fff640 1 -- 172.21.15.28:0/2624495781 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f29840d0840 con 0x7f2990103af0 2024-04-21T08:54:14.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.610+0000 7f298effd640 1 --2- 172.21.15.28:0/2624495781 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f29780a3d20 0x7f29780a61e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2980005ab0 tx=0x7f2980005a20 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:14.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.614+0000 7f2977fff640 1 -- 172.21.15.28:0/2624495781 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2984014030 con 0x7f2990103af0 2024-04-21T08:54:14.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:14 smithi171 ceph-mon[31358]: pgmap v94: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T08:54:14.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.841+0000 7f2995443640 1 -- 172.21.15.28:0/2624495781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2990107100 con 0x7f2990103af0 2024-04-21T08:54:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.841+0000 7f2977fff640 1 -- 172.21.15.28:0/2624495781 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7f298409c4f0 con 0x7f2990103af0 2024-04-21T08:54:14.841 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:54:14.842 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:52:11.568185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24487,24488,24463]},"id":1}]} 2024-04-21T08:54:14.842 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-21T08:54:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.843+0000 7f2995443640 1 -- 172.21.15.28:0/2624495781 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f29780a3d20 msgr2=0x7f29780a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.843+0000 7f2995443640 1 --2- 172.21.15.28:0/2624495781 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f29780a3d20 0x7f29780a61e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2980005ab0 tx=0x7f2980005a20 comp rx=0 tx=0).stop 2024-04-21T08:54:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.843+0000 7f2995443640 1 -- 172.21.15.28:0/2624495781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990103af0 msgr2=0x7f2990198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.843+0000 7f2995443640 1 --2- 172.21.15.28:0/2624495781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990103af0 0x7f2990198c60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f298400c890 tx=0x7f298400cd50 comp rx=0 tx=0).stop 2024-04-21T08:54:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.843+0000 7f2995443640 1 -- 172.21.15.28:0/2624495781 shutdown_connections 2024-04-21T08:54:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.843+0000 7f2995443640 1 --2- 172.21.15.28:0/2624495781 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f29780a3d20 0x7f29780a61e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.843+0000 7f2995443640 1 --2- 172.21.15.28:0/2624495781 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f299010c5a0 0x7f299019cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.844+0000 7f2995443640 1 --2- 172.21.15.28:0/2624495781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2990103af0 0x7f2990198c60 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.844+0000 7f2995443640 1 --2- 172.21.15.28:0/2624495781 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29901031d0 0x7f2990198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.844+0000 7f2995443640 1 -- 172.21.15.28:0/2624495781 >> 172.21.15.28:0/2624495781 conn(0x7f29900771d0 msgr2=0x7f29900fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.844+0000 7f2995443640 1 -- 172.21.15.28:0/2624495781 shutdown_connections 2024-04-21T08:54:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.844+0000 7f2995443640 1 -- 172.21.15.28:0/2624495781 wait complete. 2024-04-21T08:54:14.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:54:14.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.960+0000 7f502099d640 1 -- 172.21.15.28:0/4160080758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c107260 msgr2=0x7f501c10ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.960+0000 7f502099d640 1 --2- 172.21.15.28:0/4160080758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c107260 0x7f501c10ee30 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f500c009ca0 tx=0x7f500c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.960+0000 7f502099d640 1 -- 172.21.15.28:0/4160080758 shutdown_connections 2024-04-21T08:54:14.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.960+0000 7f502099d640 1 --2- 172.21.15.28:0/4160080758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c107260 0x7f501c10ee30 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.960+0000 7f502099d640 1 --2- 172.21.15.28:0/4160080758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f501c102b40 0x7f501c106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.960+0000 7f502099d640 1 --2- 172.21.15.28:0/4160080758 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f501c102190 0x7f501c102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.960+0000 7f502099d640 1 -- 172.21.15.28:0/4160080758 >> 172.21.15.28:0/4160080758 conn(0x7f501c0793a0 msgr2=0x7f501c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.960+0000 7f502099d640 1 -- 172.21.15.28:0/4160080758 shutdown_connections 2024-04-21T08:54:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.960+0000 7f502099d640 1 -- 172.21.15.28:0/4160080758 wait complete. 2024-04-21T08:54:14.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f502099d640 1 Processor -- start 2024-04-21T08:54:14.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f502099d640 1 -- start start 2024-04-21T08:54:14.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f502099d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c102190 0x7f501c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f502099d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f501c107260 0x7f501c198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f502099d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f501c19cfa0 0x7f501c19d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f502099d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f501c1121d0 con 0x7f501c102190 2024-04-21T08:54:14.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f502099d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f501c112050 con 0x7f501c19cfa0 2024-04-21T08:54:14.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f502099d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f501c112350 con 0x7f501c107260 2024-04-21T08:54:14.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f501a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c102190 0x7f501c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f501a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c102190 0x7f501c198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49060/0 (socket says 172.21.15.28:49060) 2024-04-21T08:54:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f501a59c640 1 -- 172.21.15.28:0/2612866541 learned_addr learned my addr 172.21.15.28:0/2612866541 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f501ad9d640 1 --2- 172.21.15.28:0/2612866541 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f501c19cfa0 0x7f501c19d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f5019d9b640 1 --2- 172.21.15.28:0/2612866541 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f501c107260 0x7f501c198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f501a59c640 1 -- 172.21.15.28:0/2612866541 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f501c107260 msgr2=0x7f501c198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f501a59c640 1 --2- 172.21.15.28:0/2612866541 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f501c107260 0x7f501c198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f501a59c640 1 -- 172.21.15.28:0/2612866541 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f501c19cfa0 msgr2=0x7f501c19d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f501a59c640 1 --2- 172.21.15.28:0/2612866541 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f501c19cfa0 0x7f501c19d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f501a59c640 1 -- 172.21.15.28:0/2612866541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501c19da90 con 0x7f501c102190 2024-04-21T08:54:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.961+0000 7f5019d9b640 1 --2- 172.21.15.28:0/2612866541 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f501c107260 0x7f501c198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.962+0000 7f501a59c640 1 --2- 172.21.15.28:0/2612866541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c102190 0x7f501c198730 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f501400c970 tx=0x7f501400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.962+0000 7f50137fe640 1 -- 172.21.15.28:0/2612866541 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5014007bb0 con 0x7f501c102190 2024-04-21T08:54:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.962+0000 7f50137fe640 1 -- 172.21.15.28:0/2612866541 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5014007d50 con 0x7f501c102190 2024-04-21T08:54:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.962+0000 7f502099d640 1 -- 172.21.15.28:0/2612866541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f501c19dd80 con 0x7f501c102190 2024-04-21T08:54:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.962+0000 7f50137fe640 1 -- 172.21.15.28:0/2612866541 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501400f450 con 0x7f501c102190 2024-04-21T08:54:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.962+0000 7f502099d640 1 -- 172.21.15.28:0/2612866541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f501c1a5520 con 0x7f501c102190 2024-04-21T08:54:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.964+0000 7f50137fe640 1 -- 172.21.15.28:0/2612866541 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5014004f20 con 0x7f501c102190 2024-04-21T08:54:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.964+0000 7f502358e740 1 Processor -- start 2024-04-21T08:54:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.965+0000 7f502358e740 1 -- start start 2024-04-21T08:54:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.965+0000 7f502358e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c028b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.965+0000 7f502358e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c02e80 0x556046c0b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.965+0000 7f502358e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c0ba00 0x556046c0de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.965+0000 7f502358e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556046b671f0 con 0x556046c02e80 2024-04-21T08:54:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.965+0000 7f502358e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556046b044b0 con 0x556046c0ba00 2024-04-21T08:54:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.965+0000 7f502358e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x556046b631b0 con 0x556046c024d0 2024-04-21T08:54:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.965+0000 7f501a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c02e80 0x556046c0b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.965+0000 7f50137fe640 1 --2- 172.21.15.28:0/2612866541 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f50080a3d20 0x7f50080a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.965+0000 7f501a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c02e80 0x556046c0b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49064/0 (socket says 172.21.15.28:49064) 2024-04-21T08:54:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.965+0000 7f501a59c640 1 -- 172.21.15.28:0/546831271 learned_addr learned my addr 172.21.15.28:0/546831271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.965+0000 7f501ad9d640 1 --2- 172.21.15.28:0/2612866541 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f50080a3d20 0x7f50080a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.965+0000 7f5019d9b640 1 --2- 172.21.15.28:0/546831271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c0ba00 0x556046c0de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.965+0000 7f501ad9d640 1 --2- 172.21.15.28:0/546831271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c028b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.966+0000 7f50137fe640 1 -- 172.21.15.28:0/2612866541 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f50140d0980 con 0x7f501c102190 2024-04-21T08:54:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.966+0000 7f501a59c640 1 -- 172.21.15.28:0/546831271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 msgr2=0x556046c028b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.966+0000 7f501a59c640 1 --2- 172.21.15.28:0/546831271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c028b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.966+0000 7f501a59c640 1 -- 172.21.15.28:0/546831271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c0ba00 msgr2=0x556046c0de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.966+0000 7f501a59c640 1 --2- 172.21.15.28:0/546831271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c0ba00 0x556046c0de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.966+0000 7f501a59c640 1 -- 172.21.15.28:0/546831271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556046af61b0 con 0x556046c02e80 2024-04-21T08:54:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.966+0000 7f501ad9d640 1 --2- 172.21.15.28:0/546831271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.966+0000 7f501a59c640 1 --2- 172.21.15.28:0/546831271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c02e80 0x556046c0b430 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f50140cdc70 tx=0x7f5014004980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83df677bf08ab0a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.966+0000 7f501ad9d640 1 --2- 172.21.15.28:0/2612866541 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f50080a3d20 0x7f50080a61e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f500c0098d0 tx=0x7f500c02e040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.966+0000 7f5010ff9640 1 -- 172.21.15.28:0/546831271 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501401d070 con 0x556046c02e80 2024-04-21T08:54:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.966+0000 7f5010ff9640 1 -- 172.21.15.28:0/546831271 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f501409c520 con 0x556046c02e80 2024-04-21T08:54:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.966+0000 7f5010ff9640 1 -- 172.21.15.28:0/546831271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5014098d20 con 0x556046c02e80 2024-04-21T08:54:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.967+0000 7f502358e740 1 -- 172.21.15.28:0/546831271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c02e80 msgr2=0x556046c0b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.967+0000 7f502358e740 1 --2- 172.21.15.28:0/546831271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c02e80 0x556046c0b430 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f50140cdc70 tx=0x7f5014004980 comp rx=0 tx=0).stop 2024-04-21T08:54:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.967+0000 7f502358e740 1 -- 172.21.15.28:0/546831271 shutdown_connections 2024-04-21T08:54:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.967+0000 7f502358e740 1 --2- 172.21.15.28:0/546831271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c0ba00 0x556046c0de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.967+0000 7f502358e740 1 --2- 172.21.15.28:0/546831271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c02e80 0x556046c0b430 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.967+0000 7f502358e740 1 --2- 172.21.15.28:0/546831271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.967+0000 7f502358e740 1 -- 172.21.15.28:0/546831271 >> 172.21.15.28:0/546831271 conn(0x556046bfcba0 msgr2=0x556046bfcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.967+0000 7f502358e740 1 -- 172.21.15.28:0/546831271 shutdown_connections 2024-04-21T08:54:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.967+0000 7f502358e740 1 -- 172.21.15.28:0/546831271 wait complete. 2024-04-21T08:54:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.967+0000 7f502358e740 1 -- start start 2024-04-21T08:54:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.967+0000 7f502358e740 1 Processor -- start 2024-04-21T08:54:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f502358e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c04830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f502358e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c02e80 0x556046c04d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f502358e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c0ba00 0x556046c052b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f502358e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556046c0e6f0 con 0x556046c0ba00 2024-04-21T08:54:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f502358e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556046b20500 con 0x556046c02e80 2024-04-21T08:54:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f502358e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x556046c0e870 con 0x556046c024d0 2024-04-21T08:54:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f502358e740 10 client.0 _ll_register_callbacks cb 0x556046aeeb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f501ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c0ba00 0x556046c052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f501ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c0ba00 0x556046c052b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49070/0 (socket says 172.21.15.28:49070) 2024-04-21T08:54:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f501ad9d640 1 -- 172.21.15.28:0/2430522046 learned_addr learned my addr 172.21.15.28:0/2430522046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f5019d9b640 1 --2- 172.21.15.28:0/2430522046 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c02e80 0x556046c04d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f501a59c640 1 --2- 172.21.15.28:0/2430522046 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c04830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f501ad9d640 1 -- 172.21.15.28:0/2430522046 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 msgr2=0x556046c04830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f501ad9d640 1 --2- 172.21.15.28:0/2430522046 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c04830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f501ad9d640 1 -- 172.21.15.28:0/2430522046 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c02e80 msgr2=0x556046c04d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f501ad9d640 1 --2- 172.21.15.28:0/2430522046 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c02e80 0x556046c04d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f501ad9d640 1 -- 172.21.15.28:0/2430522046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560468793f0 con 0x556046c0ba00 2024-04-21T08:54:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f501a59c640 1 --2- 172.21.15.28:0/2430522046 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c04830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f501ad9d640 1 --2- 172.21.15.28:0/2430522046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c0ba00 0x556046c052b0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f500c036000 tx=0x7f500c02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9c5c69067dc99c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.968+0000 7f5001ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.969+0000 7f5001ffb640 1 -- 172.21.15.28:0/2430522046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c051020 con 0x556046c0ba00 2024-04-21T08:54:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.969+0000 7f5001ffb640 1 -- 172.21.15.28:0/2430522046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f500c004450 con 0x556046c0ba00 2024-04-21T08:54:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.969+0000 7f502358e740 1 -- 172.21.15.28:0/2430522046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f501409c520 con 0x556046c0ba00 2024-04-21T08:54:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.969+0000 7f5001ffb640 1 -- 172.21.15.28:0/2430522046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f500c039040 con 0x556046c0ba00 2024-04-21T08:54:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.969+0000 7f502358e740 10 client.16632 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.969+0000 7f502358e740 1 -- 172.21.15.28:0/2430522046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5560469437d0 con 0x556046c0ba00 2024-04-21T08:54:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.969+0000 7f5001ffb640 1 -- 172.21.15.28:0/2430522046 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f501409c520 con 0x556046c0ba00 2024-04-21T08:54:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.969+0000 7f5001ffb640 1 -- 172.21.15.28:0/2430522046 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f500c03f070 con 0x556046c0ba00 2024-04-21T08:54:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.969+0000 7f502358e740 10 client.16632 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.969+0000 7f502358e740 10 client.16632 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:54:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.969+0000 7f502358e740 20 client.16632 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.969+0000 7f502358e740 1 --2- 172.21.15.28:0/2430522046 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x556046c24360 0x556046c267c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.969+0000 7f502358e740 4 client.16632 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.969+0000 7f502358e740 1 -- 172.21.15.28:0/2430522046 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556046b06930 con 0x556046c24360 2024-04-21T08:54:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.969+0000 7f501a59c640 1 --2- 172.21.15.28:0/2430522046 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x556046c24360 0x556046c267c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.970+0000 7f5001ffb640 10 client.16632 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:54:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.970+0000 7f501a59c640 1 --2- 172.21.15.28:0/2430522046 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x556046c24360 0x556046c267c0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=51b1ae2b097db3d3 server_cookie=26221819f830efee in_seq=0 out_seq=0 2024-04-21T08:54:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.971+0000 7f5001ffb640 1 -- 172.21.15.28:0/2430522046 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556046b06930 con 0x556046c24360 2024-04-21T08:54:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.971+0000 7f5001ffb640 10 client.16632 handle_command_reply: tid=0 2024-04-21T08:54:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.971+0000 7f502358e740 1 client.16632 shutdown 2024-04-21T08:54:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.971+0000 7f502358e740 1 -- 172.21.15.28:0/2430522046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c0ba00 msgr2=0x556046c052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.971+0000 7f502358e740 1 --2- 172.21.15.28:0/2430522046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c0ba00 0x556046c052b0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f500c036000 tx=0x7f500c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.971+0000 7f502358e740 1 -- 172.21.15.28:0/2430522046 shutdown_connections 2024-04-21T08:54:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.971+0000 7f502358e740 1 --2- 172.21.15.28:0/2430522046 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x556046c24360 0x556046c267c0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.971+0000 7f502358e740 1 --2- 172.21.15.28:0/2430522046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c0ba00 0x556046c052b0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.971+0000 7f5001ffb640 0 client.16632 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:54:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.971+0000 7f502358e740 1 --2- 172.21.15.28:0/2430522046 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c02e80 0x556046c04d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.971+0000 7f502358e740 1 --2- 172.21.15.28:0/2430522046 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c04830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.971+0000 7f502358e740 1 -- 172.21.15.28:0/2430522046 >> 172.21.15.28:0/2430522046 conn(0x556046bfcba0 msgr2=0x556046bfe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.971+0000 7f502358e740 1 -- 172.21.15.28:0/2430522046 shutdown_connections 2024-04-21T08:54:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.971+0000 7f502358e740 1 -- 172.21.15.28:0/2430522046 wait complete. 2024-04-21T08:54:14.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.971+0000 7f502358e740 20 client.16632 trim_cache size 0 max 16384 2024-04-21T08:54:14.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.984+0000 7f502358e740 1 Processor -- start 2024-04-21T08:54:14.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.984+0000 7f502358e740 1 -- start start 2024-04-21T08:54:14.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.984+0000 7f502358e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c0aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.984+0000 7f502358e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c02e80 0x556046bfbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.984+0000 7f502358e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c0ba00 0x556046c0f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.984+0000 7f502358e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f501c1af9f0 con 0x556046c02e80 2024-04-21T08:54:14.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.984+0000 7f502358e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f501c1ad2c0 con 0x556046c0ba00 2024-04-21T08:54:14.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.984+0000 7f502358e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f501c1ab110 con 0x556046c024d0 2024-04-21T08:54:14.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.984+0000 7f5019d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c02e80 0x556046bfbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.984+0000 7f5019d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c02e80 0x556046bfbc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49076/0 (socket says 172.21.15.28:49076) 2024-04-21T08:54:14.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.984+0000 7f5019d9b640 1 -- 172.21.15.28:0/196466544 learned_addr learned my addr 172.21.15.28:0/196466544 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.985+0000 7f501a59c640 1 --2- 172.21.15.28:0/196466544 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c0aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.985+0000 7f5019d9b640 1 -- 172.21.15.28:0/196466544 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 msgr2=0x556046c0aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.985+0000 7f5019d9b640 1 --2- 172.21.15.28:0/196466544 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c0aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.985+0000 7f5019d9b640 1 -- 172.21.15.28:0/196466544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c0ba00 msgr2=0x556046c0f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.985+0000 7f501ad9d640 1 --2- 172.21.15.28:0/196466544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c0ba00 0x556046c0f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.985+0000 7f5019d9b640 1 --2- 172.21.15.28:0/196466544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c0ba00 0x556046c0f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.985+0000 7f5019d9b640 1 -- 172.21.15.28:0/196466544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556046c02c20 con 0x556046c02e80 2024-04-21T08:54:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.985+0000 7f5019d9b640 1 --2- 172.21.15.28:0/196466544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c02e80 0x556046bfbc00 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f5004008cf0 tx=0x7f5004011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1c6f616d38de3c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.985+0000 7f5003fff640 1 -- 172.21.15.28:0/196466544 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5004004050 con 0x556046c02e80 2024-04-21T08:54:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.985+0000 7f5003fff640 1 -- 172.21.15.28:0/196466544 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50040054d0 con 0x556046c02e80 2024-04-21T08:54:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.985+0000 7f5003fff640 1 -- 172.21.15.28:0/196466544 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5004004d10 con 0x556046c02e80 2024-04-21T08:54:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 -- 172.21.15.28:0/196466544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c02e80 msgr2=0x556046bfbc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 --2- 172.21.15.28:0/196466544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c02e80 0x556046bfbc00 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f5004008cf0 tx=0x7f5004011810 comp rx=0 tx=0).stop 2024-04-21T08:54:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 -- 172.21.15.28:0/196466544 shutdown_connections 2024-04-21T08:54:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 --2- 172.21.15.28:0/196466544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c0ba00 0x556046c0f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 --2- 172.21.15.28:0/196466544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c02e80 0x556046bfbc00 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 --2- 172.21.15.28:0/196466544 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c0aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 -- 172.21.15.28:0/196466544 >> 172.21.15.28:0/196466544 conn(0x556046bfcba0 msgr2=0x556046c0ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 -- 172.21.15.28:0/196466544 shutdown_connections 2024-04-21T08:54:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 -- 172.21.15.28:0/196466544 wait complete. 2024-04-21T08:54:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 -- start start 2024-04-21T08:54:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 Processor -- start 2024-04-21T08:54:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c1d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c02e80 0x556046c1da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c0ba00 0x556046c89900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f501c1ab780 con 0x556046c0ba00 2024-04-21T08:54:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f501c1acbf0 con 0x556046c02e80 2024-04-21T08:54:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f501c1abe50 con 0x556046c024d0 2024-04-21T08:54:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f502358e740 10 client.0 _ll_register_callbacks cb 0x7f500c0328a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f501ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c0ba00 0x556046c89900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f501ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c0ba00 0x556046c89900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49086/0 (socket says 172.21.15.28:49086) 2024-04-21T08:54:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f501ad9d640 1 -- 172.21.15.28:0/3176593837 learned_addr learned my addr 172.21.15.28:0/3176593837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f5019d9b640 1 --2- 172.21.15.28:0/3176593837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c02e80 0x556046c1da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.986+0000 7f501a59c640 1 --2- 172.21.15.28:0/3176593837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c1d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.987+0000 7f501ad9d640 1 -- 172.21.15.28:0/3176593837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 msgr2=0x556046c1d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.987+0000 7f501ad9d640 1 --2- 172.21.15.28:0/3176593837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c1d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.987+0000 7f501ad9d640 1 -- 172.21.15.28:0/3176593837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c02e80 msgr2=0x556046c1da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.987+0000 7f501ad9d640 1 --2- 172.21.15.28:0/3176593837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c02e80 0x556046c1da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.987+0000 7f501ad9d640 1 -- 172.21.15.28:0/3176593837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556046c89e40 con 0x556046c0ba00 2024-04-21T08:54:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.987+0000 7f501ad9d640 1 --2- 172.21.15.28:0/3176593837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c0ba00 0x556046c89900 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f500c02c630 tx=0x7f500c032810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6ed9174f253cf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.987+0000 7f5001ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.987+0000 7f5001ffb640 1 -- 172.21.15.28:0/3176593837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c051040 con 0x556046c0ba00 2024-04-21T08:54:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.987+0000 7f5001ffb640 1 -- 172.21.15.28:0/3176593837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f500c034850 con 0x556046c0ba00 2024-04-21T08:54:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.987+0000 7f502358e740 1 -- 172.21.15.28:0/3176593837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50040054d0 con 0x556046c0ba00 2024-04-21T08:54:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.987+0000 7f5001ffb640 1 -- 172.21.15.28:0/3176593837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f500c032950 con 0x556046c0ba00 2024-04-21T08:54:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.987+0000 7f502358e740 10 client.16638 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.987+0000 7f502358e740 1 -- 172.21.15.28:0/3176593837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556046c8afe0 con 0x556046c0ba00 2024-04-21T08:54:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.987+0000 7f5001ffb640 1 -- 172.21.15.28:0/3176593837 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50040054d0 con 0x556046c0ba00 2024-04-21T08:54:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.987+0000 7f5001ffb640 1 -- 172.21.15.28:0/3176593837 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f500c060c50 con 0x556046c0ba00 2024-04-21T08:54:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.988+0000 7f502358e740 10 client.16638 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.988+0000 7f502358e740 10 client.16638 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:54:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.988+0000 7f502358e740 20 client.16638 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.988+0000 7f502358e740 1 --2- 172.21.15.28:0/3176593837 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x556046c8b5c0 0x556046c8d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.988+0000 7f502358e740 4 client.16638 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:54:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.988+0000 7f502358e740 1 -- 172.21.15.28:0/3176593837 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556046b74c10 con 0x556046c8b5c0 2024-04-21T08:54:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.988+0000 7f501a59c640 1 --2- 172.21.15.28:0/3176593837 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x556046c8b5c0 0x556046c8d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.988+0000 7f5001ffb640 10 client.16638 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:54:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.988+0000 7f501a59c640 1 --2- 172.21.15.28:0/3176593837 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x556046c8b5c0 0x556046c8d9b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e559f5d654d14f40 server_cookie=5a07773a582cc95b in_seq=0 out_seq=0 2024-04-21T08:54:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.988+0000 7f5001ffb640 1 -- 172.21.15.28:0/3176593837 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556046b74c10 con 0x556046c8b5c0 2024-04-21T08:54:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.988+0000 7f5001ffb640 10 client.16638 handle_command_reply: tid=0 2024-04-21T08:54:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.988+0000 7f502358e740 1 client.16638 shutdown 2024-04-21T08:54:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.988+0000 7f502358e740 1 -- 172.21.15.28:0/3176593837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c0ba00 msgr2=0x556046c89900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.988+0000 7f502358e740 1 --2- 172.21.15.28:0/3176593837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c0ba00 0x556046c89900 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f500c02c630 tx=0x7f500c032810 comp rx=0 tx=0).stop 2024-04-21T08:54:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502358e740 1 -- 172.21.15.28:0/3176593837 shutdown_connections 2024-04-21T08:54:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502358e740 1 --2- 172.21.15.28:0/3176593837 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x556046c8b5c0 0x556046c8d9b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502358e740 1 --2- 172.21.15.28:0/3176593837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556046c0ba00 0x556046c89900 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502358e740 1 --2- 172.21.15.28:0/3176593837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556046c02e80 0x556046c1da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f5001ffb640 0 client.16638 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:54:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502358e740 1 --2- 172.21.15.28:0/3176593837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556046c024d0 0x556046c1d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502358e740 1 -- 172.21.15.28:0/3176593837 >> 172.21.15.28:0/3176593837 conn(0x556046bfcba0 msgr2=0x556046c0f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502358e740 1 -- 172.21.15.28:0/3176593837 shutdown_connections 2024-04-21T08:54:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502358e740 1 -- 172.21.15.28:0/3176593837 wait complete. 2024-04-21T08:54:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502358e740 20 client.16638 trim_cache size 0 max 16384 2024-04-21T08:54:14.996 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:54:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502099d640 1 -- 172.21.15.28:0/2612866541 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f50080a3d20 msgr2=0x7f50080a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502099d640 1 --2- 172.21.15.28:0/2612866541 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f50080a3d20 0x7f50080a61e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f500c0098d0 tx=0x7f500c02e040 comp rx=0 tx=0).stop 2024-04-21T08:54:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502099d640 1 -- 172.21.15.28:0/2612866541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c102190 msgr2=0x7f501c198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502099d640 1 --2- 172.21.15.28:0/2612866541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c102190 0x7f501c198730 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f501400c970 tx=0x7f501400ce30 comp rx=0 tx=0).stop 2024-04-21T08:54:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502099d640 1 -- 172.21.15.28:0/2612866541 shutdown_connections 2024-04-21T08:54:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502099d640 1 --2- 172.21.15.28:0/2612866541 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f50080a3d20 0x7f50080a61e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502099d640 1 --2- 172.21.15.28:0/2612866541 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f501c19cfa0 0x7f501c19d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502099d640 1 --2- 172.21.15.28:0/2612866541 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f501c107260 0x7f501c198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502099d640 1 --2- 172.21.15.28:0/2612866541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f501c102190 0x7f501c198730 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502099d640 1 -- 172.21.15.28:0/2612866541 >> 172.21.15.28:0/2612866541 conn(0x7f501c0793a0 msgr2=0x7f501c0ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502099d640 1 -- 172.21.15.28:0/2612866541 shutdown_connections 2024-04-21T08:54:14.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:14.989+0000 7f502099d640 1 -- 172.21.15.28:0/2612866541 wait complete. 2024-04-21T08:54:15.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:54:15.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.103+0000 7fe42999d640 1 -- 172.21.15.28:0/2414640192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4241031d0 msgr2=0x7fe4241035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.103+0000 7fe42999d640 1 --2- 172.21.15.28:0/2414640192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4241031d0 0x7fe4241035b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe41c009900 tx=0x7fe41c02bd60 comp rx=0 tx=0).stop 2024-04-21T08:54:15.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.104+0000 7fe42999d640 1 -- 172.21.15.28:0/2414640192 shutdown_connections 2024-04-21T08:54:15.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.104+0000 7fe42999d640 1 --2- 172.21.15.28:0/2414640192 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe42410c510 0x7fe42410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.104+0000 7fe42999d640 1 --2- 172.21.15.28:0/2414640192 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe424103af0 0x7fe42410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.104+0000 7fe42999d640 1 --2- 172.21.15.28:0/2414640192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4241031d0 0x7fe4241035b0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.104+0000 7fe42999d640 1 -- 172.21.15.28:0/2414640192 >> 172.21.15.28:0/2414640192 conn(0x7fe424077f50 msgr2=0x7fe424078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:15.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.104+0000 7fe42999d640 1 -- 172.21.15.28:0/2414640192 shutdown_connections 2024-04-21T08:54:15.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.104+0000 7fe42999d640 1 -- 172.21.15.28:0/2414640192 wait complete. 2024-04-21T08:54:15.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.104+0000 7fe42999d640 1 Processor -- start 2024-04-21T08:54:15.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.104+0000 7fe42999d640 1 -- start start 2024-04-21T08:54:15.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe42999d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4241031d0 0x7fe42419e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe42999d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe424103af0 0x7fe42419e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe42999d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42410c510 0x7fe424198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe42999d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241121a0 con 0x7fe42410c510 2024-04-21T08:54:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe42999d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe424112020 con 0x7fe424103af0 2024-04-21T08:54:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe42999d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe424112320 con 0x7fe4241031d0 2024-04-21T08:54:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe4237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42410c510 0x7fe424198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe4237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42410c510 0x7fe424198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49110/0 (socket says 172.21.15.28:49110) 2024-04-21T08:54:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe4237fe640 1 -- 172.21.15.28:0/4234836985 learned_addr learned my addr 172.21.15.28:0/4234836985 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe422ffd640 1 --2- 172.21.15.28:0/4234836985 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4241031d0 0x7fe42419e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe4227fc640 1 --2- 172.21.15.28:0/4234836985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe424103af0 0x7fe42419e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe4237fe640 1 -- 172.21.15.28:0/4234836985 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4241031d0 msgr2=0x7fe42419e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe4237fe640 1 --2- 172.21.15.28:0/4234836985 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4241031d0 0x7fe42419e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe4237fe640 1 -- 172.21.15.28:0/4234836985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe424103af0 msgr2=0x7fe42419e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe4237fe640 1 --2- 172.21.15.28:0/4234836985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe424103af0 0x7fe42419e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe4237fe640 1 -- 172.21.15.28:0/4234836985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe424198d90 con 0x7fe42410c510 2024-04-21T08:54:15.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe422ffd640 1 --2- 172.21.15.28:0/4234836985 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4241031d0 0x7fe42419e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:15.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe4237fe640 1 --2- 172.21.15.28:0/4234836985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42410c510 0x7fe424198610 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fe41400eaa0 tx=0x7fe41400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:15.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.105+0000 7fe42899b640 1 -- 172.21.15.28:0/4234836985 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41400cdb0 con 0x7fe42410c510 2024-04-21T08:54:15.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.106+0000 7fe42999d640 1 -- 172.21.15.28:0/4234836985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe424199080 con 0x7fe42410c510 2024-04-21T08:54:15.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.106+0000 7fe42899b640 1 -- 172.21.15.28:0/4234836985 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe414005020 con 0x7fe42410c510 2024-04-21T08:54:15.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.106+0000 7fe42899b640 1 -- 172.21.15.28:0/4234836985 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4140116d0 con 0x7fe42410c510 2024-04-21T08:54:15.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.106+0000 7fe42999d640 1 -- 172.21.15.28:0/4234836985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4241a5320 con 0x7fe42410c510 2024-04-21T08:54:15.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.108+0000 7fe42899b640 1 -- 172.21.15.28:0/4234836985 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe4140054a0 con 0x7fe42410c510 2024-04-21T08:54:15.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.108+0000 7fe42c58d740 1 Processor -- start 2024-04-21T08:54:15.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.108+0000 7fe42c58d740 1 -- start start 2024-04-21T08:54:15.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.108+0000 7fe42c58d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 0x55a5019b8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.108+0000 7fe42c58d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019b8d60 0x55a5019c1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.108+0000 7fe42c58d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019c1e60 0x55a5019c22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.108+0000 7fe42c58d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a50191d1f0 con 0x55a5019b8d60 2024-04-21T08:54:15.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.108+0000 7fe42c58d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55a5018ba4b0 con 0x55a5019b83b0 2024-04-21T08:54:15.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.108+0000 7fe42c58d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55a5019191b0 con 0x55a5019c1e60 2024-04-21T08:54:15.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe422ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019b8d60 0x55a5019c1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe422ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019b8d60 0x55a5019c1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49114/0 (socket says 172.21.15.28:49114) 2024-04-21T08:54:15.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe422ffd640 1 -- 172.21.15.28:0/172731906 learned_addr learned my addr 172.21.15.28:0/172731906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:15.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe4227fc640 1 --2- 172.21.15.28:0/172731906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 0x55a5019b8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe42899b640 1 --2- 172.21.15.28:0/4234836985 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe40c0a3d20 0x7fe40c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe422ffd640 1 -- 172.21.15.28:0/172731906 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019c1e60 msgr2=0x55a5019c22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe422ffd640 1 --2- 172.21.15.28:0/172731906 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019c1e60 0x55a5019c22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe422ffd640 1 -- 172.21.15.28:0/172731906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 msgr2=0x55a5019b8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe422ffd640 1 --2- 172.21.15.28:0/172731906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 0x55a5019b8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe422ffd640 1 -- 172.21.15.28:0/172731906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5018ac1b0 con 0x55a5019b8d60 2024-04-21T08:54:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe4227fc640 1 --2- 172.21.15.28:0/172731906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 0x55a5019b8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe4237fe640 1 --2- 172.21.15.28:0/4234836985 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe40c0a3d20 0x7fe40c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe42899b640 1 -- 172.21.15.28:0/4234836985 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe4140d0c20 con 0x7fe42410c510 2024-04-21T08:54:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe422ffd640 1 --2- 172.21.15.28:0/172731906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019b8d60 0x55a5019c1890 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fe41c009900 tx=0x7fe41c02c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=148d536cc28b3b3e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe409ffb640 1 -- 172.21.15.28:0/172731906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c043070 con 0x55a5019b8d60 2024-04-21T08:54:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe409ffb640 1 -- 172.21.15.28:0/172731906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe41c02c910 con 0x55a5019b8d60 2024-04-21T08:54:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.109+0000 7fe409ffb640 1 -- 172.21.15.28:0/172731906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c003b30 con 0x55a5019b8d60 2024-04-21T08:54:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.110+0000 7fe42c58d740 1 -- 172.21.15.28:0/172731906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019b8d60 msgr2=0x55a5019c1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.110+0000 7fe4237fe640 1 --2- 172.21.15.28:0/4234836985 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe40c0a3d20 0x7fe40c0a61e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe4140ce080 tx=0x7fe414015810 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.110+0000 7fe42c58d740 1 --2- 172.21.15.28:0/172731906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019b8d60 0x55a5019c1890 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fe41c009900 tx=0x7fe41c02c7d0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.110+0000 7fe42c58d740 1 -- 172.21.15.28:0/172731906 shutdown_connections 2024-04-21T08:54:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.110+0000 7fe42c58d740 1 --2- 172.21.15.28:0/172731906 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019c1e60 0x55a5019c22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.110+0000 7fe42c58d740 1 --2- 172.21.15.28:0/172731906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019b8d60 0x55a5019c1890 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.110+0000 7fe42c58d740 1 --2- 172.21.15.28:0/172731906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 0x55a5019b8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.110+0000 7fe42c58d740 1 -- 172.21.15.28:0/172731906 >> 172.21.15.28:0/172731906 conn(0x55a5019b2ba0 msgr2=0x55a5019b2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.110+0000 7fe42c58d740 1 -- 172.21.15.28:0/172731906 shutdown_connections 2024-04-21T08:54:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.110+0000 7fe42c58d740 1 -- 172.21.15.28:0/172731906 wait complete. 2024-04-21T08:54:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.110+0000 7fe42c58d740 1 -- start start 2024-04-21T08:54:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe42c58d740 1 Processor -- start 2024-04-21T08:54:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe42c58d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019b83b0 0x55a5019c8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe42c58d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b8d60 0x55a5019c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe42c58d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe42c58d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a5019c8d40 con 0x55a5019c1e60 2024-04-21T08:54:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe42c58d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55a5018d6500 con 0x55a5019b8d60 2024-04-21T08:54:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe42c58d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55a5019c8ec0 con 0x55a5019b83b0 2024-04-21T08:54:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe42c58d740 10 client.0 _ll_register_callbacks cb 0x55a5018a4b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe4227fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe4227fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49126/0 (socket says 172.21.15.28:49126) 2024-04-21T08:54:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe4227fc640 1 -- 172.21.15.28:0/1949953135 learned_addr learned my addr 172.21.15.28:0/1949953135 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe422ffd640 1 --2- 172.21.15.28:0/1949953135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b8d60 0x55a5019c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe4227fc640 1 --2- 172.21.15.28:0/1949953135 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019b83b0 0x55a5019c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe4227fc640 1 -- 172.21.15.28:0/1949953135 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019b83b0 msgr2=0x55a5019c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe4227fc640 1 --2- 172.21.15.28:0/1949953135 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019b83b0 0x55a5019c8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe4227fc640 1 -- 172.21.15.28:0/1949953135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b8d60 msgr2=0x55a5019c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe4227fc640 1 --2- 172.21.15.28:0/1949953135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b8d60 0x55a5019c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe4227fc640 1 -- 172.21.15.28:0/1949953135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a50162f3f0 con 0x55a5019c1e60 2024-04-21T08:54:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe422ffd640 1 --2- 172.21.15.28:0/1949953135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b8d60 0x55a5019c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe4227fc640 1 --2- 172.21.15.28:0/1949953135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c5ae0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fe418005790 tx=0x7fe4180050f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe861b23b2ab1b9f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.111+0000 7fe402ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.112+0000 7fe402ffd640 1 -- 172.21.15.28:0/1949953135 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418004220 con 0x55a5019c1e60 2024-04-21T08:54:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.112+0000 7fe402ffd640 1 -- 172.21.15.28:0/1949953135 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe4180043c0 con 0x55a5019c1e60 2024-04-21T08:54:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.112+0000 7fe42c58d740 1 -- 172.21.15.28:0/1949953135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe41c02c910 con 0x55a5019c1e60 2024-04-21T08:54:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.112+0000 7fe402ffd640 1 -- 172.21.15.28:0/1949953135 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe418024050 con 0x55a5019c1e60 2024-04-21T08:54:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.112+0000 7fe42c58d740 10 client.16650 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.112+0000 7fe42c58d740 1 -- 172.21.15.28:0/1949953135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a5016f97d0 con 0x55a5019c1e60 2024-04-21T08:54:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.112+0000 7fe402ffd640 1 -- 172.21.15.28:0/1949953135 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe41c02c910 con 0x55a5019c1e60 2024-04-21T08:54:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.112+0000 7fe402ffd640 1 -- 172.21.15.28:0/1949953135 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fe4180036a0 con 0x55a5019c1e60 2024-04-21T08:54:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.112+0000 7fe42c58d740 10 client.16650 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.112+0000 7fe42c58d740 10 client.16650 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:54:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.112+0000 7fe42c58d740 20 client.16650 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.112+0000 7fe42c58d740 1 --2- 172.21.15.28:0/1949953135 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55a5019d6060 0x55a5019d8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.112+0000 7fe42c58d740 4 client.16650 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:15.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.112+0000 7fe42c58d740 1 -- 172.21.15.28:0/1949953135 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5018bc930 con 0x55a5019d6060 2024-04-21T08:54:15.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.112+0000 7fe422ffd640 1 --2- 172.21.15.28:0/1949953135 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55a5019d6060 0x55a5019d8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.113+0000 7fe422ffd640 1 --2- 172.21.15.28:0/1949953135 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55a5019d6060 0x55a5019d8480 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4f5033fb77ca1127 server_cookie=b62ee1dea727c01 in_seq=0 out_seq=0 2024-04-21T08:54:15.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.113+0000 7fe402ffd640 10 client.16650 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:54:15.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.114+0000 7fe402ffd640 1 -- 172.21.15.28:0/1949953135 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a5018bc930 con 0x55a5019d6060 2024-04-21T08:54:15.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.114+0000 7fe402ffd640 10 client.16650 handle_command_reply: tid=0 2024-04-21T08:54:15.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.114+0000 7fe42c58d740 1 client.16650 shutdown 2024-04-21T08:54:15.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.114+0000 7fe42c58d740 1 -- 172.21.15.28:0/1949953135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 msgr2=0x55a5019c5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.114+0000 7fe42c58d740 1 --2- 172.21.15.28:0/1949953135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c5ae0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fe418005790 tx=0x7fe4180050f0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.114+0000 7fe42c58d740 1 -- 172.21.15.28:0/1949953135 shutdown_connections 2024-04-21T08:54:15.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.114+0000 7fe42c58d740 1 --2- 172.21.15.28:0/1949953135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c5ae0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.114+0000 7fe42c58d740 1 --2- 172.21.15.28:0/1949953135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b8d60 0x55a5019c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.114+0000 7fe42c58d740 1 --2- 172.21.15.28:0/1949953135 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55a5019d6060 0x55a5019d8480 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.114+0000 7fe42c58d740 1 --2- 172.21.15.28:0/1949953135 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019b83b0 0x55a5019c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.114+0000 7fe402ffd640 0 client.16650 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:54:15.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.114+0000 7fe42c58d740 1 -- 172.21.15.28:0/1949953135 >> 172.21.15.28:0/1949953135 conn(0x55a5019b2ba0 msgr2=0x55a5019b48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:15.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.114+0000 7fe42c58d740 1 -- 172.21.15.28:0/1949953135 shutdown_connections 2024-04-21T08:54:15.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.114+0000 7fe42c58d740 1 -- 172.21.15.28:0/1949953135 wait complete. 2024-04-21T08:54:15.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.114+0000 7fe42c58d740 20 client.16650 trim_cache size 0 max 16384 2024-04-21T08:54:15.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.127+0000 7fe42c58d740 1 Processor -- start 2024-04-21T08:54:15.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.127+0000 7fe42c58d740 1 -- start start 2024-04-21T08:54:15.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.127+0000 7fe42c58d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 0x55a5019d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.127+0000 7fe42c58d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019b8d60 0x55a5019c4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.127+0000 7fe42c58d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.127+0000 7fe42c58d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241afa00 con 0x55a5019c1e60 2024-04-21T08:54:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.127+0000 7fe42c58d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241ad2d0 con 0x55a5019b83b0 2024-04-21T08:54:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.127+0000 7fe42c58d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241ab120 con 0x55a5019b8d60 2024-04-21T08:54:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.128+0000 7fe4227fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.128+0000 7fe4227fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49130/0 (socket says 172.21.15.28:49130) 2024-04-21T08:54:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.128+0000 7fe4227fc640 1 -- 172.21.15.28:0/3275409883 learned_addr learned my addr 172.21.15.28:0/3275409883 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.128+0000 7fe4227fc640 1 --2- 172.21.15.28:0/3275409883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 0x55a5019d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.128+0000 7fe422ffd640 1 --2- 172.21.15.28:0/3275409883 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019b8d60 0x55a5019c4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.128+0000 7fe4227fc640 1 -- 172.21.15.28:0/3275409883 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019b8d60 msgr2=0x55a5019c4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.128+0000 7fe4227fc640 1 --2- 172.21.15.28:0/3275409883 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019b8d60 0x55a5019c4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.128+0000 7fe4227fc640 1 -- 172.21.15.28:0/3275409883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 msgr2=0x55a5019d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.128+0000 7fe4227fc640 1 --2- 172.21.15.28:0/3275409883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 0x55a5019d1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.128+0000 7fe4227fc640 1 -- 172.21.15.28:0/3275409883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5019b8b00 con 0x55a5019c1e60 2024-04-21T08:54:15.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.128+0000 7fe4227fc640 1 --2- 172.21.15.28:0/3275409883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c7cd0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x55a5019b75d0 tx=0x7fe4180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25fe52881ffb18c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.128+0000 7fe4097fa640 1 -- 172.21.15.28:0/3275409883 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418003da0 con 0x55a5019c1e60 2024-04-21T08:54:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.128+0000 7fe4097fa640 1 -- 172.21.15.28:0/3275409883 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4180044d0 con 0x55a5019c1e60 2024-04-21T08:54:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.128+0000 7fe4097fa640 1 -- 172.21.15.28:0/3275409883 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe418011850 con 0x55a5019c1e60 2024-04-21T08:54:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.129+0000 7fe42c58d740 1 -- 172.21.15.28:0/3275409883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 msgr2=0x55a5019c7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.129+0000 7fe42c58d740 1 --2- 172.21.15.28:0/3275409883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c7cd0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x55a5019b75d0 tx=0x7fe4180046c0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.129+0000 7fe42c58d740 1 -- 172.21.15.28:0/3275409883 shutdown_connections 2024-04-21T08:54:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.129+0000 7fe42c58d740 1 --2- 172.21.15.28:0/3275409883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c7cd0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.129+0000 7fe42c58d740 1 --2- 172.21.15.28:0/3275409883 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019b8d60 0x55a5019c4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.129+0000 7fe42c58d740 1 --2- 172.21.15.28:0/3275409883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 0x55a5019d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.129+0000 7fe42c58d740 1 -- 172.21.15.28:0/3275409883 >> 172.21.15.28:0/3275409883 conn(0x55a5019b2ba0 msgr2=0x55a5019b4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.129+0000 7fe42c58d740 1 -- 172.21.15.28:0/3275409883 shutdown_connections 2024-04-21T08:54:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.129+0000 7fe42c58d740 1 -- 172.21.15.28:0/3275409883 wait complete. 2024-04-21T08:54:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.129+0000 7fe42c58d740 1 -- start start 2024-04-21T08:54:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.129+0000 7fe42c58d740 1 Processor -- start 2024-04-21T08:54:15.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe42c58d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 0x55a5019ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe42c58d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019b8d60 0x55a5019ce6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe42c58d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe42c58d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241ab790 con 0x55a5019c1e60 2024-04-21T08:54:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe42c58d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241acc00 con 0x55a5019b83b0 2024-04-21T08:54:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe42c58d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241abe60 con 0x55a5019b8d60 2024-04-21T08:54:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe42c58d740 10 client.0 _ll_register_callbacks cb 0x7fe41800b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe4227fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe4227fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49140/0 (socket says 172.21.15.28:49140) 2024-04-21T08:54:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe4227fc640 1 -- 172.21.15.28:0/3025713976 learned_addr learned my addr 172.21.15.28:0/3025713976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe422ffd640 1 --2- 172.21.15.28:0/3025713976 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019b8d60 0x55a5019ce6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe4227fc640 1 --2- 172.21.15.28:0/3025713976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 0x55a5019ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe4227fc640 1 -- 172.21.15.28:0/3025713976 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019b8d60 msgr2=0x55a5019ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe4227fc640 1 --2- 172.21.15.28:0/3025713976 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019b8d60 0x55a5019ce6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe4227fc640 1 -- 172.21.15.28:0/3025713976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 msgr2=0x55a5019ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe4227fc640 1 --2- 172.21.15.28:0/3025713976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 0x55a5019ce1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe4227fc640 1 -- 172.21.15.28:0/3025713976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5019c7450 con 0x55a5019c1e60 2024-04-21T08:54:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe4227fc640 1 --2- 172.21.15.28:0/3025713976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c6c50 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fe41800b7d0 tx=0x7fe41802d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb9517b224525e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe402ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe402ffd640 1 -- 172.21.15.28:0/3025713976 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4180035f0 con 0x55a5019c1e60 2024-04-21T08:54:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe402ffd640 1 -- 172.21.15.28:0/3025713976 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe418003790 con 0x55a5019c1e60 2024-04-21T08:54:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.130+0000 7fe42c58d740 1 -- 172.21.15.28:0/3025713976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4180044d0 con 0x55a5019c1e60 2024-04-21T08:54:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.131+0000 7fe402ffd640 1 -- 172.21.15.28:0/3025713976 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe418004020 con 0x55a5019c1e60 2024-04-21T08:54:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.131+0000 7fe42c58d740 10 client.16656 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.131+0000 7fe42c58d740 1 -- 172.21.15.28:0/3025713976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a501a3c040 con 0x55a5019c1e60 2024-04-21T08:54:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.131+0000 7fe402ffd640 1 -- 172.21.15.28:0/3025713976 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4180044d0 con 0x55a5019c1e60 2024-04-21T08:54:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.131+0000 7fe402ffd640 1 -- 172.21.15.28:0/3025713976 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fe418037030 con 0x55a5019c1e60 2024-04-21T08:54:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.131+0000 7fe42c58d740 10 client.16656 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.131+0000 7fe42c58d740 10 client.16656 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:54:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.131+0000 7fe42c58d740 20 client.16656 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.131+0000 7fe42c58d740 1 --2- 172.21.15.28:0/3025713976 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55a501a3c6d0 0x55a501a3eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.131+0000 7fe42c58d740 4 client.16656 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:54:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.131+0000 7fe42c58d740 1 -- 172.21.15.28:0/3025713976 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a50192ac10 con 0x55a501a3c6d0 2024-04-21T08:54:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.131+0000 7fe422ffd640 1 --2- 172.21.15.28:0/3025713976 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55a501a3c6d0 0x55a501a3eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe402ffd640 10 client.16656 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:54:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe422ffd640 1 --2- 172.21.15.28:0/3025713976 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55a501a3c6d0 0x55a501a3eaf0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=463d47bc296d3efd server_cookie=4f405f3f595a5ebf in_seq=0 out_seq=0 2024-04-21T08:54:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe402ffd640 1 -- 172.21.15.28:0/3025713976 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a50192ac10 con 0x55a501a3c6d0 2024-04-21T08:54:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe402ffd640 10 client.16656 handle_command_reply: tid=0 2024-04-21T08:54:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe42c58d740 1 client.16656 shutdown 2024-04-21T08:54:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe42c58d740 1 -- 172.21.15.28:0/3025713976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 msgr2=0x55a5019c6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe42c58d740 1 --2- 172.21.15.28:0/3025713976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c6c50 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fe41800b7d0 tx=0x7fe41802d040 comp rx=0 tx=0).stop 2024-04-21T08:54:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe42c58d740 1 -- 172.21.15.28:0/3025713976 shutdown_connections 2024-04-21T08:54:15.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe42c58d740 1 --2- 172.21.15.28:0/3025713976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a5019c1e60 0x55a5019c6c50 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe42c58d740 1 --2- 172.21.15.28:0/3025713976 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55a501a3c6d0 0x55a501a3eaf0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe42c58d740 1 --2- 172.21.15.28:0/3025713976 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a5019b8d60 0x55a5019ce6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe42c58d740 1 --2- 172.21.15.28:0/3025713976 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a5019b83b0 0x55a5019ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe42c58d740 1 -- 172.21.15.28:0/3025713976 >> 172.21.15.28:0/3025713976 conn(0x55a5019b2ba0 msgr2=0x55a5019b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe402ffd640 0 client.16656 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:54:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe42c58d740 1 -- 172.21.15.28:0/3025713976 shutdown_connections 2024-04-21T08:54:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe42c58d740 1 -- 172.21.15.28:0/3025713976 wait complete. 2024-04-21T08:54:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.132+0000 7fe42c58d740 20 client.16656 trim_cache size 0 max 16384 2024-04-21T08:54:15.138 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:54:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.133+0000 7fe42999d640 1 -- 172.21.15.28:0/4234836985 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe40c0a3d20 msgr2=0x7fe40c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.133+0000 7fe42999d640 1 --2- 172.21.15.28:0/4234836985 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe40c0a3d20 0x7fe40c0a61e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe4140ce080 tx=0x7fe414015810 comp rx=0 tx=0).stop 2024-04-21T08:54:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.133+0000 7fe42999d640 1 -- 172.21.15.28:0/4234836985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42410c510 msgr2=0x7fe424198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.133+0000 7fe42999d640 1 --2- 172.21.15.28:0/4234836985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42410c510 0x7fe424198610 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fe41400eaa0 tx=0x7fe41400ef60 comp rx=0 tx=0).stop 2024-04-21T08:54:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.133+0000 7fe42999d640 1 -- 172.21.15.28:0/4234836985 shutdown_connections 2024-04-21T08:54:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.133+0000 7fe42999d640 1 --2- 172.21.15.28:0/4234836985 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe40c0a3d20 0x7fe40c0a61e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.133+0000 7fe42999d640 1 --2- 172.21.15.28:0/4234836985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe42410c510 0x7fe424198610 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.133+0000 7fe42999d640 1 --2- 172.21.15.28:0/4234836985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe424103af0 0x7fe42419e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.133+0000 7fe42999d640 1 --2- 172.21.15.28:0/4234836985 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4241031d0 0x7fe42419e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.133+0000 7fe42999d640 1 -- 172.21.15.28:0/4234836985 >> 172.21.15.28:0/4234836985 conn(0x7fe424077f50 msgr2=0x7fe4240ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.133+0000 7fe42999d640 1 -- 172.21.15.28:0/4234836985 shutdown_connections 2024-04-21T08:54:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.133+0000 7fe42999d640 1 -- 172.21.15.28:0/4234836985 wait complete. 2024-04-21T08:54:15.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:54:15.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.248+0000 7f692d8f1640 1 -- 172.21.15.28:0/2587302260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6928106160 msgr2=0x7f692810dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.248+0000 7f692d8f1640 1 --2- 172.21.15.28:0/2587302260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6928106160 0x7f692810dd30 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6920009ca0 tx=0x7f692002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.248+0000 7f692d8f1640 1 -- 172.21.15.28:0/2587302260 shutdown_connections 2024-04-21T08:54:15.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.248+0000 7f692d8f1640 1 --2- 172.21.15.28:0/2587302260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6928106160 0x7f692810dd30 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.248+0000 7f692d8f1640 1 --2- 172.21.15.28:0/2587302260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6928101a40 0x7f6928105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.248+0000 7f692d8f1640 1 --2- 172.21.15.28:0/2587302260 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6928101090 0x7f6928101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.248+0000 7f692d8f1640 1 -- 172.21.15.28:0/2587302260 >> 172.21.15.28:0/2587302260 conn(0x7f6928077f50 msgr2=0x7f6928078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.248+0000 7f692d8f1640 1 -- 172.21.15.28:0/2587302260 shutdown_connections 2024-04-21T08:54:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.248+0000 7f692d8f1640 1 -- 172.21.15.28:0/2587302260 wait complete. 2024-04-21T08:54:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.249+0000 7f692d8f1640 1 Processor -- start 2024-04-21T08:54:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.249+0000 7f692d8f1640 1 -- start start 2024-04-21T08:54:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.249+0000 7f692d8f1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6928101090 0x7f6928198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.249+0000 7f692d8f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6928106160 0x7f6928198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.249+0000 7f692d8f1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f692819cf00 0x7f692819d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.249+0000 7f692d8f1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69281121e0 con 0x7f6928106160 2024-04-21T08:54:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.249+0000 7f692d8f1640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6928112060 con 0x7f692819cf00 2024-04-21T08:54:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.249+0000 7f692d8f1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6928112360 con 0x7f6928101090 2024-04-21T08:54:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.249+0000 7f691ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6928106160 0x7f6928198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.249+0000 7f691ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6928106160 0x7f6928198b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49160/0 (socket says 172.21.15.28:49160) 2024-04-21T08:54:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.249+0000 7f691ffff640 1 -- 172.21.15.28:0/487874814 learned_addr learned my addr 172.21.15.28:0/487874814 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.249+0000 7f69277fe640 1 --2- 172.21.15.28:0/487874814 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f692819cf00 0x7f692819d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.250+0000 7f6926ffd640 1 --2- 172.21.15.28:0/487874814 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6928101090 0x7f6928198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.250+0000 7f691ffff640 1 -- 172.21.15.28:0/487874814 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6928101090 msgr2=0x7f6928198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.250+0000 7f691ffff640 1 --2- 172.21.15.28:0/487874814 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6928101090 0x7f6928198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.250+0000 7f691ffff640 1 -- 172.21.15.28:0/487874814 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f692819cf00 msgr2=0x7f692819d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.250+0000 7f691ffff640 1 --2- 172.21.15.28:0/487874814 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f692819cf00 0x7f692819d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.250+0000 7f691ffff640 1 -- 172.21.15.28:0/487874814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692819d9f0 con 0x7f6928106160 2024-04-21T08:54:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.250+0000 7f691ffff640 1 --2- 172.21.15.28:0/487874814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6928106160 0x7f6928198b80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f691000c970 tx=0x7f691000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.250+0000 7f692c8ef640 1 -- 172.21.15.28:0/487874814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6910007bb0 con 0x7f6928106160 2024-04-21T08:54:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.250+0000 7f692c8ef640 1 -- 172.21.15.28:0/487874814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6910007d50 con 0x7f6928106160 2024-04-21T08:54:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.250+0000 7f692d8f1640 1 -- 172.21.15.28:0/487874814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f692819dce0 con 0x7f6928106160 2024-04-21T08:54:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.250+0000 7f692c8ef640 1 -- 172.21.15.28:0/487874814 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691000f450 con 0x7f6928106160 2024-04-21T08:54:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.250+0000 7f692d8f1640 1 -- 172.21.15.28:0/487874814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69281a5520 con 0x7f6928106160 2024-04-21T08:54:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.252+0000 7f692c8ef640 1 -- 172.21.15.28:0/487874814 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6910004f20 con 0x7f6928106160 2024-04-21T08:54:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69305b6740 1 Processor -- start 2024-04-21T08:54:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69305b6740 1 -- start start 2024-04-21T08:54:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69305b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dcc64d0 0x56215dcc68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69305b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc6e80 0x56215dccf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69305b6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dccfa00 0x56215dcd1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69305b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56215dc2b1f0 con 0x56215dcc64d0 2024-04-21T08:54:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69305b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56215dbc84b0 con 0x56215dcc6e80 2024-04-21T08:54:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69305b6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56215dc271b0 con 0x56215dccfa00 2024-04-21T08:54:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dcc64d0 0x56215dcc68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dcc64d0 0x56215dcc68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49164/0 (socket says 172.21.15.28:49164) 2024-04-21T08:54:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69277fe640 1 -- 172.21.15.28:0/1944586137 learned_addr learned my addr 172.21.15.28:0/1944586137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f6926ffd640 1 --2- 172.21.15.28:0/1944586137 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc6e80 0x56215dccf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f692c8ef640 1 --2- 172.21.15.28:0/487874814 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f69140a3da0 0x7f69140a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f691ffff640 1 --2- 172.21.15.28:0/1944586137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dccfa00 0x56215dcd1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69277fe640 1 -- 172.21.15.28:0/1944586137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dccfa00 msgr2=0x56215dcd1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69277fe640 1 --2- 172.21.15.28:0/1944586137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dccfa00 0x56215dcd1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69277fe640 1 -- 172.21.15.28:0/1944586137 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc6e80 msgr2=0x56215dccf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69277fe640 1 --2- 172.21.15.28:0/1944586137 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc6e80 0x56215dccf430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69277fe640 1 -- 172.21.15.28:0/1944586137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56215dbba1b0 con 0x56215dcc64d0 2024-04-21T08:54:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f691ffff640 1 --2- 172.21.15.28:0/1944586137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dccfa00 0x56215dcd1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f692c8ef640 1 -- 172.21.15.28:0/487874814 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f69100d0b50 con 0x7f6928106160 2024-04-21T08:54:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.253+0000 7f69277fe640 1 --2- 172.21.15.28:0/487874814 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f69140a3da0 0x7f69140a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.254+0000 7f69277fe640 1 --2- 172.21.15.28:0/1944586137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dcc64d0 0x56215dcc68b0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f6920009ca0 tx=0x7f6920005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9495311bcb43ea0c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.254+0000 7f691dffb640 1 -- 172.21.15.28:0/1944586137 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6920039040 con 0x56215dcc64d0 2024-04-21T08:54:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.254+0000 7f691dffb640 1 -- 172.21.15.28:0/1944586137 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6920003b70 con 0x56215dcc64d0 2024-04-21T08:54:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.254+0000 7f691dffb640 1 -- 172.21.15.28:0/1944586137 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6920034440 con 0x56215dcc64d0 2024-04-21T08:54:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.254+0000 7f69305b6740 1 -- 172.21.15.28:0/1944586137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dcc64d0 msgr2=0x56215dcc68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.254+0000 7f69305b6740 1 --2- 172.21.15.28:0/1944586137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dcc64d0 0x56215dcc68b0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f6920009ca0 tx=0x7f6920005fa0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.254+0000 7f69277fe640 1 --2- 172.21.15.28:0/487874814 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f69140a3da0 0x7f69140a6260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f692004c000 tx=0x7f6920045000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 1 -- 172.21.15.28:0/1944586137 shutdown_connections 2024-04-21T08:54:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 1 --2- 172.21.15.28:0/1944586137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dccfa00 0x56215dcd1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 1 --2- 172.21.15.28:0/1944586137 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc6e80 0x56215dccf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 1 --2- 172.21.15.28:0/1944586137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dcc64d0 0x56215dcc68b0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 1 -- 172.21.15.28:0/1944586137 >> 172.21.15.28:0/1944586137 conn(0x56215dcc0ba0 msgr2=0x56215dcc0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 1 -- 172.21.15.28:0/1944586137 shutdown_connections 2024-04-21T08:54:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 1 -- 172.21.15.28:0/1944586137 wait complete. 2024-04-21T08:54:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 1 -- start start 2024-04-21T08:54:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 1 Processor -- start 2024-04-21T08:54:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc64d0 0x56215dccf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 0x56215dccc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dccfa00 0x56215dccce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56215dcd2970 con 0x56215dccfa00 2024-04-21T08:54:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56215dbe4500 con 0x56215dcc64d0 2024-04-21T08:54:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56215dcd2af0 con 0x56215dcc6e80 2024-04-21T08:54:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.255+0000 7f69305b6740 10 client.0 _ll_register_callbacks cb 0x56215dbb2b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f69277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dccfa00 0x56215dccce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f69277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dccfa00 0x56215dccce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49178/0 (socket says 172.21.15.28:49178) 2024-04-21T08:54:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f69277fe640 1 -- 172.21.15.28:0/2303539953 learned_addr learned my addr 172.21.15.28:0/2303539953 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f691ffff640 1 --2- 172.21.15.28:0/2303539953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 0x56215dccc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f6926ffd640 1 --2- 172.21.15.28:0/2303539953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc64d0 0x56215dccf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f69277fe640 1 -- 172.21.15.28:0/2303539953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 msgr2=0x56215dccc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f69277fe640 1 --2- 172.21.15.28:0/2303539953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 0x56215dccc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f69277fe640 1 -- 172.21.15.28:0/2303539953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc64d0 msgr2=0x56215dccf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f69277fe640 1 --2- 172.21.15.28:0/2303539953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc64d0 0x56215dccf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f69277fe640 1 -- 172.21.15.28:0/2303539953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56215d93d3f0 con 0x56215dccfa00 2024-04-21T08:54:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f691ffff640 1 --2- 172.21.15.28:0/2303539953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 0x56215dccc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f69277fe640 1 --2- 172.21.15.28:0/2303539953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dccfa00 0x56215dccce80 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f69200323c0 tx=0x7f6920005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a8cb48cad48ddb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f6906ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f6906ffd640 1 -- 172.21.15.28:0/2303539953 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6920039070 con 0x56215dccfa00 2024-04-21T08:54:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f6906ffd640 1 -- 172.21.15.28:0/2303539953 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f6920003d20 con 0x56215dccfa00 2024-04-21T08:54:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f69305b6740 1 -- 172.21.15.28:0/2303539953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6920003b70 con 0x56215dccfa00 2024-04-21T08:54:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f6906ffd640 1 -- 172.21.15.28:0/2303539953 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6920004040 con 0x56215dccfa00 2024-04-21T08:54:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f69305b6740 10 client.16668 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.256+0000 7f69305b6740 1 -- 172.21.15.28:0/2303539953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56215da077d0 con 0x56215dccfa00 2024-04-21T08:54:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.257+0000 7f6906ffd640 1 -- 172.21.15.28:0/2303539953 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6920003b70 con 0x56215dccfa00 2024-04-21T08:54:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.257+0000 7f6906ffd640 1 -- 172.21.15.28:0/2303539953 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f692003f040 con 0x56215dccfa00 2024-04-21T08:54:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.257+0000 7f69305b6740 10 client.16668 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.257+0000 7f69305b6740 10 client.16668 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.257+0000 7f69305b6740 20 client.16668 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.257+0000 7f69305b6740 1 --2- 172.21.15.28:0/2303539953 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56215dcdfad0 0x56215dce1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.257+0000 7f69305b6740 4 client.16668 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.257+0000 7f69305b6740 1 -- 172.21.15.28:0/2303539953 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56215dbca930 con 0x56215dcdfad0 2024-04-21T08:54:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.257+0000 7f6926ffd640 1 --2- 172.21.15.28:0/2303539953 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56215dcdfad0 0x56215dce1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.257+0000 7f6906ffd640 10 client.16668 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.257+0000 7f6926ffd640 1 --2- 172.21.15.28:0/2303539953 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56215dcdfad0 0x56215dce1ec0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eea66a6f24955804 server_cookie=673949fda76dcde7 in_seq=0 out_seq=0 2024-04-21T08:54:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.258+0000 7f6906ffd640 1 -- 172.21.15.28:0/2303539953 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56215dbca930 con 0x56215dcdfad0 2024-04-21T08:54:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.258+0000 7f6906ffd640 10 client.16668 handle_command_reply: tid=0 2024-04-21T08:54:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.258+0000 7f69305b6740 1 client.16668 shutdown 2024-04-21T08:54:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.258+0000 7f69305b6740 1 -- 172.21.15.28:0/2303539953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dccfa00 msgr2=0x56215dccce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.258+0000 7f69305b6740 1 --2- 172.21.15.28:0/2303539953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dccfa00 0x56215dccce80 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f69200323c0 tx=0x7f6920005c80 comp rx=0 tx=0).stop 2024-04-21T08:54:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.258+0000 7f69305b6740 1 -- 172.21.15.28:0/2303539953 shutdown_connections 2024-04-21T08:54:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.258+0000 7f69305b6740 1 --2- 172.21.15.28:0/2303539953 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56215dcdfad0 0x56215dce1ec0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.258+0000 7f69305b6740 1 --2- 172.21.15.28:0/2303539953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dccfa00 0x56215dccce80 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.258+0000 7f69305b6740 1 --2- 172.21.15.28:0/2303539953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 0x56215dccc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.258+0000 7f6906ffd640 0 client.16668 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.258+0000 7f69305b6740 1 --2- 172.21.15.28:0/2303539953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc64d0 0x56215dccf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.258+0000 7f69305b6740 1 -- 172.21.15.28:0/2303539953 >> 172.21.15.28:0/2303539953 conn(0x56215dcc0ba0 msgr2=0x56215dcc28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.259+0000 7f69305b6740 1 -- 172.21.15.28:0/2303539953 shutdown_connections 2024-04-21T08:54:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.259+0000 7f69305b6740 1 -- 172.21.15.28:0/2303539953 wait complete. 2024-04-21T08:54:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.259+0000 7f69305b6740 20 client.16668 trim_cache size 0 max 16384 2024-04-21T08:54:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f69305b6740 1 Processor -- start 2024-04-21T08:54:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f69305b6740 1 -- start start 2024-04-21T08:54:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f69305b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dcc64d0 0x56215dccb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f69305b6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 0x56215dccee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f69305b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dccfa00 0x56215dccf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f69305b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69281b1b10 con 0x56215dcc64d0 2024-04-21T08:54:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f69305b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f69281af3e0 con 0x56215dccfa00 2024-04-21T08:54:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f69305b6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f69281ad230 con 0x56215dcc6e80 2024-04-21T08:54:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f6926ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dcc64d0 0x56215dccb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f6926ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dcc64d0 0x56215dccb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49194/0 (socket says 172.21.15.28:49194) 2024-04-21T08:54:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f6926ffd640 1 -- 172.21.15.28:0/4215153180 learned_addr learned my addr 172.21.15.28:0/4215153180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f691ffff640 1 --2- 172.21.15.28:0/4215153180 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 0x56215dccee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f69277fe640 1 --2- 172.21.15.28:0/4215153180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dccfa00 0x56215dccf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f6926ffd640 1 -- 172.21.15.28:0/4215153180 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 msgr2=0x56215dccee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f6926ffd640 1 --2- 172.21.15.28:0/4215153180 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 0x56215dccee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f6926ffd640 1 -- 172.21.15.28:0/4215153180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dccfa00 msgr2=0x56215dccf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f6926ffd640 1 --2- 172.21.15.28:0/4215153180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dccfa00 0x56215dccf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f6926ffd640 1 -- 172.21.15.28:0/4215153180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56215dcbc470 con 0x56215dcc64d0 2024-04-21T08:54:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.272+0000 7f69277fe640 1 --2- 172.21.15.28:0/4215153180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dccfa00 0x56215dccf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.273+0000 7f6926ffd640 1 --2- 172.21.15.28:0/4215153180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dcc64d0 0x56215dccb9d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x56215dc2b300 tx=0x7f691800ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22150ff92143af3c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.273+0000 7f691d7fa640 1 -- 172.21.15.28:0/4215153180 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6918003d00 con 0x56215dcc64d0 2024-04-21T08:54:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.273+0000 7f691d7fa640 1 -- 172.21.15.28:0/4215153180 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6918003ea0 con 0x56215dcc64d0 2024-04-21T08:54:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.273+0000 7f691d7fa640 1 -- 172.21.15.28:0/4215153180 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6918004a00 con 0x56215dcc64d0 2024-04-21T08:54:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.273+0000 7f69305b6740 1 -- 172.21.15.28:0/4215153180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dcc64d0 msgr2=0x56215dccb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.273+0000 7f69305b6740 1 --2- 172.21.15.28:0/4215153180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dcc64d0 0x56215dccb9d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x56215dc2b300 tx=0x7f691800ceb0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.273+0000 7f69305b6740 1 -- 172.21.15.28:0/4215153180 shutdown_connections 2024-04-21T08:54:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.273+0000 7f69305b6740 1 --2- 172.21.15.28:0/4215153180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dccfa00 0x56215dccf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.273+0000 7f69305b6740 1 --2- 172.21.15.28:0/4215153180 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 0x56215dccee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.273+0000 7f69305b6740 1 --2- 172.21.15.28:0/4215153180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dcc64d0 0x56215dccb9d0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.273+0000 7f69305b6740 1 -- 172.21.15.28:0/4215153180 >> 172.21.15.28:0/4215153180 conn(0x56215dcc0ba0 msgr2=0x56215dcc2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.273+0000 7f69305b6740 1 -- 172.21.15.28:0/4215153180 shutdown_connections 2024-04-21T08:54:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.273+0000 7f69305b6740 1 -- 172.21.15.28:0/4215153180 wait complete. 2024-04-21T08:54:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.274+0000 7f69305b6740 1 -- start start 2024-04-21T08:54:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.274+0000 7f69305b6740 1 Processor -- start 2024-04-21T08:54:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.274+0000 7f69305b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc64d0 0x56215dce31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.274+0000 7f69305b6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 0x56215dce36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.274+0000 7f69305b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dccfa00 0x56215dd3ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.274+0000 7f69305b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69281ad8a0 con 0x56215dccfa00 2024-04-21T08:54:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.274+0000 7f69305b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f69281aed10 con 0x56215dcc64d0 2024-04-21T08:54:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.274+0000 7f69305b6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f69281adf70 con 0x56215dcc6e80 2024-04-21T08:54:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.274+0000 7f69305b6740 10 client.0 _ll_register_callbacks cb 0x7f6920032bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.274+0000 7f69277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dccfa00 0x56215dd3ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.274+0000 7f69277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dccfa00 0x56215dd3ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49198/0 (socket says 172.21.15.28:49198) 2024-04-21T08:54:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.274+0000 7f69277fe640 1 -- 172.21.15.28:0/1353937567 learned_addr learned my addr 172.21.15.28:0/1353937567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.274+0000 7f691ffff640 1 --2- 172.21.15.28:0/1353937567 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 0x56215dce36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.274+0000 7f6926ffd640 1 --2- 172.21.15.28:0/1353937567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc64d0 0x56215dce31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f69277fe640 1 -- 172.21.15.28:0/1353937567 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 msgr2=0x56215dce36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f69277fe640 1 --2- 172.21.15.28:0/1353937567 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 0x56215dce36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f69277fe640 1 -- 172.21.15.28:0/1353937567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc64d0 msgr2=0x56215dce31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f69277fe640 1 --2- 172.21.15.28:0/1353937567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc64d0 0x56215dce31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f69277fe640 1 -- 172.21.15.28:0/1353937567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56215dd3f2c0 con 0x56215dccfa00 2024-04-21T08:54:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f6926ffd640 1 --2- 172.21.15.28:0/1353937567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc64d0 0x56215dce31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f69277fe640 1 --2- 172.21.15.28:0/1353937567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dccfa00 0x56215dd3ec80 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f692002c630 tx=0x7f6920009940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47bd55a29f90044 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f6906ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f6906ffd640 1 -- 172.21.15.28:0/1353937567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692006d060 con 0x56215dccfa00 2024-04-21T08:54:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f6906ffd640 1 -- 172.21.15.28:0/1353937567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f6920003a30 con 0x56215dccfa00 2024-04-21T08:54:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f69305b6740 1 -- 172.21.15.28:0/1353937567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6918003ea0 con 0x56215dccfa00 2024-04-21T08:54:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f6906ffd640 1 -- 172.21.15.28:0/1353937567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6920039040 con 0x56215dccfa00 2024-04-21T08:54:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f69305b6740 10 client.16674 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f69305b6740 1 -- 172.21.15.28:0/1353937567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56215dd465a0 con 0x56215dccfa00 2024-04-21T08:54:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f6906ffd640 1 -- 172.21.15.28:0/1353937567 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6918003ea0 con 0x56215dccfa00 2024-04-21T08:54:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f6906ffd640 1 -- 172.21.15.28:0/1353937567 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f69200347d0 con 0x56215dccfa00 2024-04-21T08:54:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f69305b6740 10 client.16674 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f69305b6740 10 client.16674 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f69305b6740 20 client.16674 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f69305b6740 1 --2- 172.21.15.28:0/1353937567 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56215dd46c10 0x56215dd49030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f69305b6740 4 client.16674 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:54:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.275+0000 7f69305b6740 1 -- 172.21.15.28:0/1353937567 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56215dc38c10 con 0x56215dd46c10 2024-04-21T08:54:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.276+0000 7f6926ffd640 1 --2- 172.21.15.28:0/1353937567 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56215dd46c10 0x56215dd49030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.276+0000 7f6926ffd640 1 --2- 172.21.15.28:0/1353937567 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56215dd46c10 0x56215dd49030 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=900d3703e6fb5276 server_cookie=7146d0c8fb50f18a in_seq=0 out_seq=0 2024-04-21T08:54:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.276+0000 7f6906ffd640 10 client.16674 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.276+0000 7f6906ffd640 1 -- 172.21.15.28:0/1353937567 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56215dc38c10 con 0x56215dd46c10 2024-04-21T08:54:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.276+0000 7f6906ffd640 10 client.16674 handle_command_reply: tid=0 2024-04-21T08:54:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.276+0000 7f69305b6740 1 client.16674 shutdown 2024-04-21T08:54:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f69305b6740 1 -- 172.21.15.28:0/1353937567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dccfa00 msgr2=0x56215dd3ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f69305b6740 1 --2- 172.21.15.28:0/1353937567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dccfa00 0x56215dd3ec80 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f692002c630 tx=0x7f6920009940 comp rx=0 tx=0).stop 2024-04-21T08:54:15.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f69305b6740 1 -- 172.21.15.28:0/1353937567 shutdown_connections 2024-04-21T08:54:15.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f69305b6740 1 --2- 172.21.15.28:0/1353937567 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56215dd46c10 0x56215dd49030 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f69305b6740 1 --2- 172.21.15.28:0/1353937567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56215dccfa00 0x56215dd3ec80 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f69305b6740 1 --2- 172.21.15.28:0/1353937567 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56215dcc6e80 0x56215dce36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f6906ffd640 0 client.16674 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:15.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f69305b6740 1 --2- 172.21.15.28:0/1353937567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56215dcc64d0 0x56215dce31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f69305b6740 1 -- 172.21.15.28:0/1353937567 >> 172.21.15.28:0/1353937567 conn(0x56215dcc0ba0 msgr2=0x56215dcbf370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:15.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f69305b6740 1 -- 172.21.15.28:0/1353937567 shutdown_connections 2024-04-21T08:54:15.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f69305b6740 1 -- 172.21.15.28:0/1353937567 wait complete. 2024-04-21T08:54:15.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f69305b6740 20 client.16674 trim_cache size 0 max 16384 2024-04-21T08:54:15.283 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:54:15.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f692d8f1640 1 -- 172.21.15.28:0/487874814 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f69140a3da0 msgr2=0x7f69140a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f692d8f1640 1 --2- 172.21.15.28:0/487874814 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f69140a3da0 0x7f69140a6260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f692004c000 tx=0x7f6920045000 comp rx=0 tx=0).stop 2024-04-21T08:54:15.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f692d8f1640 1 -- 172.21.15.28:0/487874814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6928106160 msgr2=0x7f6928198b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:15.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f692d8f1640 1 --2- 172.21.15.28:0/487874814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6928106160 0x7f6928198b80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f691000c970 tx=0x7f691000ce30 comp rx=0 tx=0).stop 2024-04-21T08:54:15.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f692d8f1640 1 -- 172.21.15.28:0/487874814 shutdown_connections 2024-04-21T08:54:15.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f692d8f1640 1 --2- 172.21.15.28:0/487874814 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f69140a3da0 0x7f69140a6260 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f692d8f1640 1 --2- 172.21.15.28:0/487874814 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f692819cf00 0x7f692819d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f692d8f1640 1 --2- 172.21.15.28:0/487874814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6928106160 0x7f6928198b80 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f692d8f1640 1 --2- 172.21.15.28:0/487874814 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6928101090 0x7f6928198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:15.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f692d8f1640 1 -- 172.21.15.28:0/487874814 >> 172.21.15.28:0/487874814 conn(0x7f6928077f50 msgr2=0x7f69280feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:15.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f692d8f1640 1 -- 172.21.15.28:0/487874814 shutdown_connections 2024-04-21T08:54:15.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:15.277+0000 7f692d8f1640 1 -- 172.21.15.28:0/487874814 wait complete. 2024-04-21T08:54:15.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:15 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2624495781' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:15.880 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:15 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2624495781' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:15.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:15 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2624495781' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:16.288 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:54:16.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:54:16.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:54:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.409+0000 7fa19b843640 1 -- 172.21.15.28:0/3407705428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa194102b40 msgr2=0x7fa194106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.409+0000 7fa19b843640 1 --2- 172.21.15.28:0/3407705428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa194102b40 0x7fa194106b80 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa184009900 tx=0x7fa18402bc30 comp rx=0 tx=0).stop 2024-04-21T08:54:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.409+0000 7fa19b843640 1 -- 172.21.15.28:0/3407705428 shutdown_connections 2024-04-21T08:54:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.410+0000 7fa19b843640 1 --2- 172.21.15.28:0/3407705428 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa194107260 0x7fa19410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.410+0000 7fa19b843640 1 --2- 172.21.15.28:0/3407705428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa194102b40 0x7fa194106b80 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.410+0000 7fa19b843640 1 --2- 172.21.15.28:0/3407705428 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa194102190 0x7fa194102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.410+0000 7fa19b843640 1 -- 172.21.15.28:0/3407705428 >> 172.21.15.28:0/3407705428 conn(0x7fa1940fd990 msgr2=0x7fa1940ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.410+0000 7fa19b843640 1 -- 172.21.15.28:0/3407705428 shutdown_connections 2024-04-21T08:54:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.410+0000 7fa19b843640 1 -- 172.21.15.28:0/3407705428 wait complete. 2024-04-21T08:54:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.410+0000 7fa19b843640 1 Processor -- start 2024-04-21T08:54:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.410+0000 7fa19b843640 1 -- start start 2024-04-21T08:54:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.410+0000 7fa19b843640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa194102190 0x7fa1941a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa19b843640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa194107260 0x7fa1941a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa19b843640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa19419a7c0 0x7fa19419ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa19b843640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1941121a0 con 0x7fa194102190 2024-04-21T08:54:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa19b843640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa194112020 con 0x7fa19419a7c0 2024-04-21T08:54:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa19b843640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa194112320 con 0x7fa194107260 2024-04-21T08:54:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa1995df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa194102190 0x7fa1941a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa1995df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa194102190 0x7fa1941a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49214/0 (socket says 172.21.15.28:49214) 2024-04-21T08:54:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa1995df640 1 -- 172.21.15.28:0/1150243290 learned_addr learned my addr 172.21.15.28:0/1150243290 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa198dde640 1 --2- 172.21.15.28:0/1150243290 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa194107260 0x7fa1941a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa199de0640 1 --2- 172.21.15.28:0/1150243290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa19419a7c0 0x7fa19419ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa1995df640 1 -- 172.21.15.28:0/1150243290 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa194107260 msgr2=0x7fa1941a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa1995df640 1 --2- 172.21.15.28:0/1150243290 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa194107260 0x7fa1941a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa1995df640 1 -- 172.21.15.28:0/1150243290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa19419a7c0 msgr2=0x7fa19419ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa1995df640 1 --2- 172.21.15.28:0/1150243290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa19419a7c0 0x7fa19419ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa1995df640 1 -- 172.21.15.28:0/1150243290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa19419b350 con 0x7fa194102190 2024-04-21T08:54:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa199de0640 1 --2- 172.21.15.28:0/1150243290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa19419a7c0 0x7fa19419ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa1995df640 1 --2- 172.21.15.28:0/1150243290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa194102190 0x7fa1941a0650 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa19000c970 tx=0x7fa19000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa18a7fc640 1 -- 172.21.15.28:0/1150243290 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa190007bb0 con 0x7fa194102190 2024-04-21T08:54:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa18a7fc640 1 -- 172.21.15.28:0/1150243290 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa190007d50 con 0x7fa194102190 2024-04-21T08:54:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa18a7fc640 1 -- 172.21.15.28:0/1150243290 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19000f450 con 0x7fa194102190 2024-04-21T08:54:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa19b843640 1 -- 172.21.15.28:0/1150243290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa19419b640 con 0x7fa194102190 2024-04-21T08:54:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.411+0000 7fa19b843640 1 -- 172.21.15.28:0/1150243290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1941a77a0 con 0x7fa194102190 2024-04-21T08:54:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.414+0000 7fa19e434740 1 Processor -- start 2024-04-21T08:54:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.414+0000 7fa18a7fc640 1 -- 172.21.15.28:0/1150243290 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa190004f20 con 0x7fa194102190 2024-04-21T08:54:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.414+0000 7fa19e434740 1 -- start start 2024-04-21T08:54:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.414+0000 7fa19e434740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 0x5629b3c48cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.414+0000 7fa19e434740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c51840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.414+0000 7fa19e434740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c54260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.414+0000 7fa19e434740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5629b3bacf00 con 0x5629b3c51e10 2024-04-21T08:54:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.414+0000 7fa19e434740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5629b3b4a590 con 0x5629b3c49290 2024-04-21T08:54:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.414+0000 7fa19e434740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5629b3ba8ec0 con 0x5629b3c488e0 2024-04-21T08:54:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa198dde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c54260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa198dde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c54260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49220/0 (socket says 172.21.15.28:49220) 2024-04-21T08:54:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa198dde640 1 -- 172.21.15.28:0/3246097118 learned_addr learned my addr 172.21.15.28:0/3246097118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:16.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa1995df640 1 --2- 172.21.15.28:0/3246097118 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c51840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa199de0640 1 --2- 172.21.15.28:0/3246097118 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 0x5629b3c48cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa198dde640 1 -- 172.21.15.28:0/3246097118 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 msgr2=0x5629b3c48cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa198dde640 1 --2- 172.21.15.28:0/3246097118 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 0x5629b3c48cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa198dde640 1 -- 172.21.15.28:0/3246097118 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 msgr2=0x5629b3c51840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa198dde640 1 --2- 172.21.15.28:0/3246097118 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c51840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa198dde640 1 -- 172.21.15.28:0/3246097118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629b39877e0 con 0x5629b3c51e10 2024-04-21T08:54:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa1995df640 1 --2- 172.21.15.28:0/3246097118 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c51840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa18a7fc640 1 --2- 172.21.15.28:0/1150243290 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa1800a3da0 0x7fa1800a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa198dde640 1 --2- 172.21.15.28:0/3246097118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c54260 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa19406b5d0 tx=0x7fa18402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aed662fc3ea7726 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa199de0640 1 --2- 172.21.15.28:0/1150243290 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa1800a3da0 0x7fa1800a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa18a7fc640 1 -- 172.21.15.28:0/1150243290 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fa1900d0940 con 0x7fa194102190 2024-04-21T08:54:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa17f7fe640 1 -- 172.21.15.28:0/3246097118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184044020 con 0x5629b3c51e10 2024-04-21T08:54:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa17f7fe640 1 -- 172.21.15.28:0/3246097118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa18402cc90 con 0x5629b3c51e10 2024-04-21T08:54:16.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.415+0000 7fa17f7fe640 1 -- 172.21.15.28:0/3246097118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1840046c0 con 0x5629b3c51e10 2024-04-21T08:54:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.416+0000 7fa19e434740 1 -- 172.21.15.28:0/3246097118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 msgr2=0x5629b3c54260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.416+0000 7fa19e434740 1 --2- 172.21.15.28:0/3246097118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c54260 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa19406b5d0 tx=0x7fa18402c650 comp rx=0 tx=0).stop 2024-04-21T08:54:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.416+0000 7fa199de0640 1 --2- 172.21.15.28:0/1150243290 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa1800a3da0 0x7fa1800a6260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa18c009370 tx=0x7fa18c009520 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.416+0000 7fa19e434740 1 -- 172.21.15.28:0/3246097118 shutdown_connections 2024-04-21T08:54:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.416+0000 7fa19e434740 1 --2- 172.21.15.28:0/3246097118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c54260 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.416+0000 7fa19e434740 1 --2- 172.21.15.28:0/3246097118 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c51840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.416+0000 7fa19e434740 1 --2- 172.21.15.28:0/3246097118 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 0x5629b3c48cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.416+0000 7fa19e434740 1 -- 172.21.15.28:0/3246097118 >> 172.21.15.28:0/3246097118 conn(0x5629b3c42fb0 msgr2=0x5629b3c43390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.416+0000 7fa19e434740 1 -- 172.21.15.28:0/3246097118 shutdown_connections 2024-04-21T08:54:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.416+0000 7fa19e434740 1 -- 172.21.15.28:0/3246097118 wait complete. 2024-04-21T08:54:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.416+0000 7fa19e434740 1 -- start start 2024-04-21T08:54:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa19e434740 1 Processor -- start 2024-04-21T08:54:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa19e434740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c488e0 0x5629b3c4edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa19e434740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c4f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa19e434740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c51e10 0x5629b3c4f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa19e434740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5629b3c54d20 con 0x5629b3c488e0 2024-04-21T08:54:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa19e434740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5629b3b66580 con 0x5629b3c49290 2024-04-21T08:54:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa19e434740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5629b3c54ea0 con 0x5629b3c51e10 2024-04-21T08:54:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa19e434740 10 client.0 _ll_register_callbacks cb 0x5629b3b6c650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa1995df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c488e0 0x5629b3c4edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa1995df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c488e0 0x5629b3c4edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49224/0 (socket says 172.21.15.28:49224) 2024-04-21T08:54:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa1995df640 1 -- 172.21.15.28:0/2505481137 learned_addr learned my addr 172.21.15.28:0/2505481137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa199de0640 1 --2- 172.21.15.28:0/2505481137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c51e10 0x5629b3c4f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa198dde640 1 --2- 172.21.15.28:0/2505481137 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c4f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa1995df640 1 -- 172.21.15.28:0/2505481137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c51e10 msgr2=0x5629b3c4f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa1995df640 1 --2- 172.21.15.28:0/2505481137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c51e10 0x5629b3c4f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa1995df640 1 -- 172.21.15.28:0/2505481137 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 msgr2=0x5629b3c4f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa1995df640 1 --2- 172.21.15.28:0/2505481137 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c4f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa1995df640 1 -- 172.21.15.28:0/2505481137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629b39e7f70 con 0x5629b3c488e0 2024-04-21T08:54:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.417+0000 7fa199de0640 1 --2- 172.21.15.28:0/2505481137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c51e10 0x5629b3c4f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa1995df640 1 --2- 172.21.15.28:0/2505481137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c488e0 0x5629b3c4edd0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa19009c240 tx=0x7fa1900988e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167ff7cbfccb5db3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa17cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa17cff9640 1 -- 172.21.15.28:0/2505481137 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19001d040 con 0x5629b3c488e0 2024-04-21T08:54:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa17cff9640 1 -- 172.21.15.28:0/2505481137 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fa1900046c0 con 0x5629b3c488e0 2024-04-21T08:54:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa19e434740 1 -- 172.21.15.28:0/2505481137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa18402cc90 con 0x5629b3c488e0 2024-04-21T08:54:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa17cff9640 1 -- 172.21.15.28:0/2505481137 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa1900055f0 con 0x5629b3c488e0 2024-04-21T08:54:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa19e434740 10 client.16686 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa19e434740 1 -- 172.21.15.28:0/2505481137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5629b387f170 con 0x5629b3c488e0 2024-04-21T08:54:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa17cff9640 1 -- 172.21.15.28:0/2505481137 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa18402cc90 con 0x5629b3c488e0 2024-04-21T08:54:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa17cff9640 1 -- 172.21.15.28:0/2505481137 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fa190098bb0 con 0x5629b3c488e0 2024-04-21T08:54:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa19e434740 10 client.16686 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:16.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa19e434740 10 client.16686 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa19e434740 20 client.16686 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa19e434740 1 --2- 172.21.15.28:0/2505481137 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5629b3c62330 0x5629b3c64780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa19e434740 4 client.16686 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.418+0000 7fa19e434740 1 -- 172.21.15.28:0/2505481137 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629b3bba920 con 0x5629b3c62330 2024-04-21T08:54:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.419+0000 7fa198dde640 1 --2- 172.21.15.28:0/2505481137 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5629b3c62330 0x5629b3c64780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.419+0000 7fa17cff9640 10 client.16686 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.419+0000 7fa198dde640 1 --2- 172.21.15.28:0/2505481137 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5629b3c62330 0x5629b3c64780 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4e865809720cefd server_cookie=66849e427f537820 in_seq=0 out_seq=0 2024-04-21T08:54:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.420+0000 7fa17cff9640 1 -- 172.21.15.28:0/2505481137 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5629b3bba920 con 0x5629b3c62330 2024-04-21T08:54:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.420+0000 7fa17cff9640 10 client.16686 handle_command_reply: tid=0 2024-04-21T08:54:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.420+0000 7fa19e434740 1 client.16686 shutdown 2024-04-21T08:54:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.420+0000 7fa19e434740 1 -- 172.21.15.28:0/2505481137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c488e0 msgr2=0x5629b3c4edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.420+0000 7fa19e434740 1 --2- 172.21.15.28:0/2505481137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c488e0 0x5629b3c4edd0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa19009c240 tx=0x7fa1900988e0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.420+0000 7fa19e434740 1 -- 172.21.15.28:0/2505481137 shutdown_connections 2024-04-21T08:54:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.420+0000 7fa19e434740 1 --2- 172.21.15.28:0/2505481137 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5629b3c62330 0x5629b3c64780 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.420+0000 7fa19e434740 1 --2- 172.21.15.28:0/2505481137 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c51e10 0x5629b3c4f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.420+0000 7fa19e434740 1 --2- 172.21.15.28:0/2505481137 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c4f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.420+0000 7fa19e434740 1 --2- 172.21.15.28:0/2505481137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c488e0 0x5629b3c4edd0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.420+0000 7fa17cff9640 0 client.16686 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.420+0000 7fa19e434740 1 -- 172.21.15.28:0/2505481137 >> 172.21.15.28:0/2505481137 conn(0x5629b3c42fb0 msgr2=0x5629b3c44d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.420+0000 7fa19e434740 1 -- 172.21.15.28:0/2505481137 shutdown_connections 2024-04-21T08:54:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.420+0000 7fa19e434740 1 -- 172.21.15.28:0/2505481137 wait complete. 2024-04-21T08:54:16.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.420+0000 7fa19e434740 20 client.16686 trim_cache size 0 max 16384 2024-04-21T08:54:16.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.438+0000 7fa19e434740 1 Processor -- start 2024-04-21T08:54:16.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.438+0000 7fa19e434740 1 -- start start 2024-04-21T08:54:16.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.438+0000 7fa19e434740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 0x5629b3c51500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.438+0000 7fa19e434740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c4dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.438+0000 7fa19e434740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c5dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.438+0000 7fa19e434740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1941b3d70 con 0x5629b3c51e10 2024-04-21T08:54:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.438+0000 7fa19e434740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa1941b1640 con 0x5629b3c49290 2024-04-21T08:54:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.438+0000 7fa19e434740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa1941af490 con 0x5629b3c488e0 2024-04-21T08:54:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.438+0000 7fa199de0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c5dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.438+0000 7fa199de0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c5dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49232/0 (socket says 172.21.15.28:49232) 2024-04-21T08:54:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.438+0000 7fa199de0640 1 -- 172.21.15.28:0/1094459084 learned_addr learned my addr 172.21.15.28:0/1094459084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.438+0000 7fa198dde640 1 --2- 172.21.15.28:0/1094459084 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c4dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.438+0000 7fa1995df640 1 --2- 172.21.15.28:0/1094459084 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 0x5629b3c51500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.439+0000 7fa199de0640 1 -- 172.21.15.28:0/1094459084 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 msgr2=0x5629b3c51500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.439+0000 7fa199de0640 1 --2- 172.21.15.28:0/1094459084 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 0x5629b3c51500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.439+0000 7fa199de0640 1 -- 172.21.15.28:0/1094459084 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 msgr2=0x5629b3c4dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.439+0000 7fa199de0640 1 --2- 172.21.15.28:0/1094459084 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c4dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.439+0000 7fa199de0640 1 -- 172.21.15.28:0/1094459084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629b3c4d320 con 0x5629b3c51e10 2024-04-21T08:54:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.439+0000 7fa1995df640 1 --2- 172.21.15.28:0/1094459084 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 0x5629b3c51500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:16.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.439+0000 7fa199de0640 1 --2- 172.21.15.28:0/1094459084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c5dbe0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fa18c012820 tx=0x7fa18c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b6c33382b81e49 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:16.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.439+0000 7fa17effd640 1 -- 172.21.15.28:0/1094459084 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c011960 con 0x5629b3c51e10 2024-04-21T08:54:16.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.439+0000 7fa17effd640 1 -- 172.21.15.28:0/1094459084 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa18c020430 con 0x5629b3c51e10 2024-04-21T08:54:16.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.439+0000 7fa17effd640 1 -- 172.21.15.28:0/1094459084 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c028660 con 0x5629b3c51e10 2024-04-21T08:54:16.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 -- 172.21.15.28:0/1094459084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 msgr2=0x5629b3c5dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 --2- 172.21.15.28:0/1094459084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c5dbe0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fa18c012820 tx=0x7fa18c013000 comp rx=0 tx=0).stop 2024-04-21T08:54:16.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 -- 172.21.15.28:0/1094459084 shutdown_connections 2024-04-21T08:54:16.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 --2- 172.21.15.28:0/1094459084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c5dbe0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 --2- 172.21.15.28:0/1094459084 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c4dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 --2- 172.21.15.28:0/1094459084 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 0x5629b3c51500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 -- 172.21.15.28:0/1094459084 >> 172.21.15.28:0/1094459084 conn(0x5629b3c42fb0 msgr2=0x5629b3c42010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 -- 172.21.15.28:0/1094459084 shutdown_connections 2024-04-21T08:54:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 -- 172.21.15.28:0/1094459084 wait complete. 2024-04-21T08:54:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 -- start start 2024-04-21T08:54:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 Processor -- start 2024-04-21T08:54:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 0x5629b3c5a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c5a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c570c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1941afb00 con 0x5629b3c51e10 2024-04-21T08:54:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa1941b0f70 con 0x5629b3c49290 2024-04-21T08:54:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.440+0000 7fa19e434740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa1941b01d0 con 0x5629b3c488e0 2024-04-21T08:54:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa19e434740 10 client.0 _ll_register_callbacks cb 0x7fa190007f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa199de0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa199de0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c570c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49246/0 (socket says 172.21.15.28:49246) 2024-04-21T08:54:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa199de0640 1 -- 172.21.15.28:0/990035280 learned_addr learned my addr 172.21.15.28:0/990035280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa198dde640 1 --2- 172.21.15.28:0/990035280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c5a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa1995df640 1 --2- 172.21.15.28:0/990035280 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 0x5629b3c5a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa199de0640 1 -- 172.21.15.28:0/990035280 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 msgr2=0x5629b3c5a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa199de0640 1 --2- 172.21.15.28:0/990035280 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 0x5629b3c5a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa199de0640 1 -- 172.21.15.28:0/990035280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 msgr2=0x5629b3c5a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa199de0640 1 --2- 172.21.15.28:0/990035280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c5a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa199de0640 1 -- 172.21.15.28:0/990035280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629b3c57800 con 0x5629b3c51e10 2024-04-21T08:54:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa199de0640 1 --2- 172.21.15.28:0/990035280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c570c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fa18c020e30 tx=0x7fa18c012f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c3df4763e28969 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa17cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa17cff9640 1 -- 172.21.15.28:0/990035280 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c038060 con 0x5629b3c51e10 2024-04-21T08:54:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa17cff9640 1 -- 172.21.15.28:0/990035280 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fa18c0207f0 con 0x5629b3c51e10 2024-04-21T08:54:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa19e434740 1 -- 172.21.15.28:0/990035280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa18c020430 con 0x5629b3c51e10 2024-04-21T08:54:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.441+0000 7fa17cff9640 1 -- 172.21.15.28:0/990035280 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa18c03f050 con 0x5629b3c51e10 2024-04-21T08:54:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.442+0000 7fa19e434740 10 client.16692 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.442+0000 7fa19e434740 1 -- 172.21.15.28:0/990035280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5629b3cc9310 con 0x5629b3c51e10 2024-04-21T08:54:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.442+0000 7fa17cff9640 1 -- 172.21.15.28:0/990035280 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa18c020430 con 0x5629b3c51e10 2024-04-21T08:54:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.442+0000 7fa17cff9640 1 -- 172.21.15.28:0/990035280 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fa18c00f040 con 0x5629b3c51e10 2024-04-21T08:54:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.442+0000 7fa19e434740 10 client.16692 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.442+0000 7fa19e434740 10 client.16692 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.442+0000 7fa19e434740 20 client.16692 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.442+0000 7fa19e434740 1 --2- 172.21.15.28:0/990035280 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5629b3cc99b0 0x5629b3ccbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.442+0000 7fa19e434740 4 client.16692 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:54:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.442+0000 7fa19e434740 1 -- 172.21.15.28:0/990035280 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5629b3bb4790 con 0x5629b3cc99b0 2024-04-21T08:54:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.442+0000 7fa1995df640 1 --2- 172.21.15.28:0/990035280 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5629b3cc99b0 0x5629b3ccbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.443+0000 7fa1995df640 1 --2- 172.21.15.28:0/990035280 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5629b3cc99b0 0x5629b3ccbdd0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bee2ca91bb3a1532 server_cookie=324ef7dbf3a9f07f in_seq=0 out_seq=0 2024-04-21T08:54:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.443+0000 7fa17cff9640 10 client.16692 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.444+0000 7fa17cff9640 1 -- 172.21.15.28:0/990035280 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5629b3bb4790 con 0x5629b3cc99b0 2024-04-21T08:54:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.444+0000 7fa17cff9640 10 client.16692 handle_command_reply: tid=0 2024-04-21T08:54:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.444+0000 7fa19e434740 1 client.16692 shutdown 2024-04-21T08:54:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.444+0000 7fa19e434740 1 -- 172.21.15.28:0/990035280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 msgr2=0x5629b3c570c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.444+0000 7fa19e434740 1 --2- 172.21.15.28:0/990035280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c570c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fa18c020e30 tx=0x7fa18c012f10 comp rx=0 tx=0).stop 2024-04-21T08:54:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.444+0000 7fa19e434740 1 -- 172.21.15.28:0/990035280 shutdown_connections 2024-04-21T08:54:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.444+0000 7fa19e434740 1 --2- 172.21.15.28:0/990035280 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5629b3cc99b0 0x5629b3ccbdd0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.444+0000 7fa19e434740 1 --2- 172.21.15.28:0/990035280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629b3c51e10 0x5629b3c570c0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.444+0000 7fa19e434740 1 --2- 172.21.15.28:0/990035280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5629b3c49290 0x5629b3c5a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.444+0000 7fa19e434740 1 --2- 172.21.15.28:0/990035280 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5629b3c488e0 0x5629b3c5a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.444+0000 7fa19e434740 1 -- 172.21.15.28:0/990035280 >> 172.21.15.28:0/990035280 conn(0x5629b3c42fb0 msgr2=0x5629b3c41780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.444+0000 7fa17cff9640 0 client.16692 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.444+0000 7fa19e434740 1 -- 172.21.15.28:0/990035280 shutdown_connections 2024-04-21T08:54:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.444+0000 7fa19e434740 1 -- 172.21.15.28:0/990035280 wait complete. 2024-04-21T08:54:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.444+0000 7fa19e434740 20 client.16692 trim_cache size 0 max 16384 2024-04-21T08:54:16.450 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:54:16.451 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:54:16.451 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "2ba7b196-0f7c-4fe9-87a2-a633a57d659d", 2024-04-21T08:54:16.451 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:54:16.451 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:54:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.445+0000 7fa19b843640 1 -- 172.21.15.28:0/1150243290 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa1800a3da0 msgr2=0x7fa1800a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.445+0000 7fa19b843640 1 --2- 172.21.15.28:0/1150243290 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa1800a3da0 0x7fa1800a6260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa18c009370 tx=0x7fa18c009520 comp rx=0 tx=0).stop 2024-04-21T08:54:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.445+0000 7fa19b843640 1 -- 172.21.15.28:0/1150243290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa194102190 msgr2=0x7fa1941a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.445+0000 7fa19b843640 1 --2- 172.21.15.28:0/1150243290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa194102190 0x7fa1941a0650 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa19000c970 tx=0x7fa19000ce30 comp rx=0 tx=0).stop 2024-04-21T08:54:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.445+0000 7fa19b843640 1 -- 172.21.15.28:0/1150243290 shutdown_connections 2024-04-21T08:54:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.445+0000 7fa19b843640 1 --2- 172.21.15.28:0/1150243290 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa1800a3da0 0x7fa1800a6260 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.445+0000 7fa19b843640 1 --2- 172.21.15.28:0/1150243290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa19419a7c0 0x7fa19419ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.445+0000 7fa19b843640 1 --2- 172.21.15.28:0/1150243290 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa194107260 0x7fa1941a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.445+0000 7fa19b843640 1 --2- 172.21.15.28:0/1150243290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa194102190 0x7fa1941a0650 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.445+0000 7fa19b843640 1 -- 172.21.15.28:0/1150243290 >> 172.21.15.28:0/1150243290 conn(0x7fa1940fd990 msgr2=0x7fa1940fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.445+0000 7fa19b843640 1 -- 172.21.15.28:0/1150243290 shutdown_connections 2024-04-21T08:54:16.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.445+0000 7fa19b843640 1 -- 172.21.15.28:0/1150243290 wait complete. 2024-04-21T08:54:16.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:16 smithi028 ceph-mon[28003]: pgmap v95: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 32 KiB/s wr, 24 op/s 2024-04-21T08:54:16.457 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:54:16.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:54:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.565+0000 7f0eebfbd640 1 -- 172.21.15.28:0/283761440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee4101080 msgr2=0x7f0ee4101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.565+0000 7f0eebfbd640 1 --2- 172.21.15.28:0/283761440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee4101080 0x7f0ee4101460 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0009ca0 tx=0x7f0ee002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.565+0000 7f0eebfbd640 1 -- 172.21.15.28:0/283761440 shutdown_connections 2024-04-21T08:54:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.565+0000 7f0eebfbd640 1 --2- 172.21.15.28:0/283761440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ee4106150 0x7f0ee410dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.565+0000 7f0eebfbd640 1 --2- 172.21.15.28:0/283761440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ee4101a30 0x7f0ee4105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.565+0000 7f0eebfbd640 1 --2- 172.21.15.28:0/283761440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee4101080 0x7f0ee4101460 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.565+0000 7f0eebfbd640 1 -- 172.21.15.28:0/283761440 >> 172.21.15.28:0/283761440 conn(0x7f0ee4078000 msgr2=0x7f0ee4078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.565+0000 7f0eebfbd640 1 -- 172.21.15.28:0/283761440 shutdown_connections 2024-04-21T08:54:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.565+0000 7f0eebfbd640 1 -- 172.21.15.28:0/283761440 wait complete. 2024-04-21T08:54:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.565+0000 7f0eebfbd640 1 Processor -- start 2024-04-21T08:54:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0eebfbd640 1 -- start start 2024-04-21T08:54:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0eebfbd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ee4101080 0x7f0ee4199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0eebfbd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee4106150 0x7f0ee4199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0eebfbd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ee419a000 0x7f0ee41a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0eebfbd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee4111090 con 0x7f0ee4106150 2024-04-21T08:54:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0eebfbd640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee4110f10 con 0x7f0ee419a000 2024-04-21T08:54:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0eebfbd640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee4111210 con 0x7f0ee4101080 2024-04-21T08:54:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0ee9558640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee4106150 0x7f0ee4199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0ee9558640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee4106150 0x7f0ee4199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49262/0 (socket says 172.21.15.28:49262) 2024-04-21T08:54:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0ee9558640 1 -- 172.21.15.28:0/1067897860 learned_addr learned my addr 172.21.15.28:0/1067897860 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0ee9d59640 1 --2- 172.21.15.28:0/1067897860 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ee4101080 0x7f0ee4199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0eea55a640 1 --2- 172.21.15.28:0/1067897860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ee419a000 0x7f0ee41a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0ee9558640 1 -- 172.21.15.28:0/1067897860 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ee4101080 msgr2=0x7f0ee4199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0ee9558640 1 --2- 172.21.15.28:0/1067897860 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ee4101080 0x7f0ee4199530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0ee9558640 1 -- 172.21.15.28:0/1067897860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ee419a000 msgr2=0x7f0ee41a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0ee9558640 1 --2- 172.21.15.28:0/1067897860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ee419a000 0x7f0ee41a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0ee9558640 1 -- 172.21.15.28:0/1067897860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee41a4db0 con 0x7f0ee4106150 2024-04-21T08:54:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0ee9558640 1 --2- 172.21.15.28:0/1067897860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee4106150 0x7f0ee4199a70 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f0ed400d770 tx=0x7f0ed400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.566+0000 7f0edaffd640 1 -- 172.21.15.28:0/1067897860 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4017070 con 0x7f0ee4106150 2024-04-21T08:54:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.567+0000 7f0edaffd640 1 -- 172.21.15.28:0/1067897860 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ed4009c30 con 0x7f0ee4106150 2024-04-21T08:54:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.567+0000 7f0eebfbd640 1 -- 172.21.15.28:0/1067897860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee41a50a0 con 0x7f0ee4106150 2024-04-21T08:54:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.567+0000 7f0edaffd640 1 -- 172.21.15.28:0/1067897860 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed400f620 con 0x7f0ee4106150 2024-04-21T08:54:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.567+0000 7f0eebfbd640 1 -- 172.21.15.28:0/1067897860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee41a55f0 con 0x7f0ee4106150 2024-04-21T08:54:16.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.569+0000 7f0edaffd640 1 -- 172.21.15.28:0/1067897860 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0ed401a070 con 0x7f0ee4106150 2024-04-21T08:54:16.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.569+0000 7f0eeebb1740 1 Processor -- start 2024-04-21T08:54:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.569+0000 7f0eeebb1740 1 -- start start 2024-04-21T08:54:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.569+0000 7f0eeebb1740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5c580 0x559d6bc5c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.569+0000 7f0eeebb1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc5cea0 0x559d6bc65540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.569+0000 7f0eeebb1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc65a80 0x559d6bc67e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.569+0000 7f0eeebb1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559d6bbc11e0 con 0x559d6bc5cea0 2024-04-21T08:54:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.569+0000 7f0eeebb1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x559d6bb5e390 con 0x559d6bc65a80 2024-04-21T08:54:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0eeebb1740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x559d6bbbd1a0 con 0x559d6bc5c580 2024-04-21T08:54:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0ee9d59640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc5cea0 0x559d6bc65540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0ee9d59640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc5cea0 0x559d6bc65540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49272/0 (socket says 172.21.15.28:49272) 2024-04-21T08:54:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0ee9d59640 1 -- 172.21.15.28:0/1325729405 learned_addr learned my addr 172.21.15.28:0/1325729405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0ee9558640 1 --2- 172.21.15.28:0/1325729405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc65a80 0x559d6bc67e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0eea55a640 1 --2- 172.21.15.28:0/1325729405 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5c580 0x559d6bc5c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0edaffd640 1 --2- 172.21.15.28:0/1067897860 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0ed00a3d20 0x7f0ed00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0eea55a640 1 --2- 172.21.15.28:0/1067897860 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0ed00a3d20 0x7f0ed00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0ee9d59640 1 -- 172.21.15.28:0/1325729405 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5c580 msgr2=0x559d6bc5c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0ee9d59640 1 --2- 172.21.15.28:0/1325729405 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5c580 0x559d6bc5c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0ee9d59640 1 -- 172.21.15.28:0/1325729405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc65a80 msgr2=0x559d6bc67e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0ee9d59640 1 --2- 172.21.15.28:0/1325729405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc65a80 0x559d6bc67e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0ee9d59640 1 -- 172.21.15.28:0/1325729405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d6b99d7d0 con 0x559d6bc5cea0 2024-04-21T08:54:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0edaffd640 1 -- 172.21.15.28:0/1067897860 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f0ed40d0820 con 0x7f0ee4106150 2024-04-21T08:54:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0ee9d59640 1 --2- 172.21.15.28:0/1325729405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc5cea0 0x559d6bc65540 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0009ca0 tx=0x7f0ee00323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d32ef3e192624ccb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0ecffff640 1 -- 172.21.15.28:0/1325729405 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0043020 con 0x559d6bc5cea0 2024-04-21T08:54:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0eea55a640 1 --2- 172.21.15.28:0/1067897860 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0ed00a3d20 0x7f0ed00a61e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0edc004120 tx=0x7f0edc004730 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0ecffff640 1 -- 172.21.15.28:0/1325729405 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ee0032970 con 0x559d6bc5cea0 2024-04-21T08:54:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.570+0000 7f0ecffff640 1 -- 172.21.15.28:0/1325729405 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0039040 con 0x559d6bc5cea0 2024-04-21T08:54:16.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.571+0000 7f0eeebb1740 1 -- 172.21.15.28:0/1325729405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc5cea0 msgr2=0x559d6bc65540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.571+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/1325729405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc5cea0 0x559d6bc65540 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0009ca0 tx=0x7f0ee00323c0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.571+0000 7f0eeebb1740 1 -- 172.21.15.28:0/1325729405 shutdown_connections 2024-04-21T08:54:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.571+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/1325729405 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc65a80 0x559d6bc67e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.571+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/1325729405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc5cea0 0x559d6bc65540 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.571+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/1325729405 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5c580 0x559d6bc5c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.571+0000 7f0eeebb1740 1 -- 172.21.15.28:0/1325729405 >> 172.21.15.28:0/1325729405 conn(0x559d6bc56cb0 msgr2=0x559d6bc57090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.571+0000 7f0eeebb1740 1 -- 172.21.15.28:0/1325729405 shutdown_connections 2024-04-21T08:54:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.571+0000 7f0eeebb1740 1 -- 172.21.15.28:0/1325729405 wait complete. 2024-04-21T08:54:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.572+0000 7f0eeebb1740 1 -- start start 2024-04-21T08:54:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.572+0000 7f0eeebb1740 1 Processor -- start 2024-04-21T08:54:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.572+0000 7f0eeebb1740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5c580 0x559d6bc62aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.572+0000 7f0eeebb1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc5cea0 0x559d6bc62fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.572+0000 7f0eeebb1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc65a80 0x559d6bc63520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.572+0000 7f0eeebb1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559d6bc68810 con 0x559d6bc5cea0 2024-04-21T08:54:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.572+0000 7f0eeebb1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x559d6bb7a2f0 con 0x559d6bc65a80 2024-04-21T08:54:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.572+0000 7f0eeebb1740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x559d6bc68990 con 0x559d6bc5c580 2024-04-21T08:54:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.572+0000 7f0eeebb1740 10 client.0 _ll_register_callbacks cb 0x559d6bb87580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.572+0000 7f0ee9558640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc5cea0 0x559d6bc62fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.572+0000 7f0ee9d59640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5c580 0x559d6bc62aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.572+0000 7f0ee9558640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc5cea0 0x559d6bc62fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49276/0 (socket says 172.21.15.28:49276) 2024-04-21T08:54:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.572+0000 7f0ee9558640 1 -- 172.21.15.28:0/3014765145 learned_addr learned my addr 172.21.15.28:0/3014765145 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.572+0000 7f0eea55a640 1 --2- 172.21.15.28:0/3014765145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc65a80 0x559d6bc63520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0ee9558640 1 -- 172.21.15.28:0/3014765145 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5c580 msgr2=0x559d6bc62aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0ee9558640 1 --2- 172.21.15.28:0/3014765145 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5c580 0x559d6bc62aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0ee9558640 1 -- 172.21.15.28:0/3014765145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc65a80 msgr2=0x559d6bc63520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0ee9558640 1 --2- 172.21.15.28:0/3014765145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc65a80 0x559d6bc63520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0ee9558640 1 -- 172.21.15.28:0/3014765145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d6b8d33f0 con 0x559d6bc5cea0 2024-04-21T08:54:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0ee9558640 1 --2- 172.21.15.28:0/3014765145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc5cea0 0x559d6bc62fe0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0ed400d740 tx=0x7f0ed4010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e74969e76c70635 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0ecd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0ecd7fa640 1 -- 172.21.15.28:0/3014765145 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4021040 con 0x559d6bc5cea0 2024-04-21T08:54:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0ecd7fa640 1 -- 172.21.15.28:0/3014765145 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f0ed40cddf0 con 0x559d6bc5cea0 2024-04-21T08:54:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0eeebb1740 1 -- 172.21.15.28:0/3014765145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ee0032970 con 0x559d6bc5cea0 2024-04-21T08:54:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0ecd7fa640 1 -- 172.21.15.28:0/3014765145 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0ed40b0e60 con 0x559d6bc5cea0 2024-04-21T08:54:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0eeebb1740 10 client.16704 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0eeebb1740 1 -- 172.21.15.28:0/3014765145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559d6b8930f0 con 0x559d6bc5cea0 2024-04-21T08:54:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0ecd7fa640 1 -- 172.21.15.28:0/3014765145 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ee0032970 con 0x559d6bc5cea0 2024-04-21T08:54:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0ecd7fa640 1 -- 172.21.15.28:0/3014765145 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f0ed40d0400 con 0x559d6bc5cea0 2024-04-21T08:54:16.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0eeebb1740 10 client.16704 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0eeebb1740 10 client.16704 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0eeebb1740 20 client.16704 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/3014765145 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559d6bc75a40 0x559d6bc77e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0eeebb1740 4 client.16704 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.573+0000 7f0eeebb1740 1 -- 172.21.15.28:0/3014765145 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d6bbcec00 con 0x559d6bc75a40 2024-04-21T08:54:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.574+0000 7f0ee9d59640 1 --2- 172.21.15.28:0/3014765145 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559d6bc75a40 0x559d6bc77e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.574+0000 7f0ecd7fa640 10 client.16704 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.574+0000 7f0ee9d59640 1 --2- 172.21.15.28:0/3014765145 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559d6bc75a40 0x559d6bc77e70 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d40569918e996287 server_cookie=2877010f97f1e9c9 in_seq=0 out_seq=0 2024-04-21T08:54:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.575+0000 7f0ecd7fa640 1 -- 172.21.15.28:0/3014765145 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559d6bbcec00 con 0x559d6bc75a40 2024-04-21T08:54:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.575+0000 7f0ecd7fa640 10 client.16704 handle_command_reply: tid=0 2024-04-21T08:54:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.575+0000 7f0eeebb1740 1 client.16704 shutdown 2024-04-21T08:54:16.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.575+0000 7f0eeebb1740 1 -- 172.21.15.28:0/3014765145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc5cea0 msgr2=0x559d6bc62fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.575+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/3014765145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc5cea0 0x559d6bc62fe0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0ed400d740 tx=0x7f0ed4010c00 comp rx=0 tx=0).stop 2024-04-21T08:54:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.575+0000 7f0eeebb1740 1 -- 172.21.15.28:0/3014765145 shutdown_connections 2024-04-21T08:54:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.575+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/3014765145 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559d6bc75a40 0x559d6bc77e70 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.575+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/3014765145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc65a80 0x559d6bc63520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.575+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/3014765145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc5cea0 0x559d6bc62fe0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.575+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/3014765145 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5c580 0x559d6bc62aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.575+0000 7f0ecd7fa640 0 client.16704 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.575+0000 7f0eeebb1740 1 -- 172.21.15.28:0/3014765145 >> 172.21.15.28:0/3014765145 conn(0x559d6bc56cb0 msgr2=0x559d6bc58a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.575+0000 7f0eeebb1740 1 -- 172.21.15.28:0/3014765145 shutdown_connections 2024-04-21T08:54:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.575+0000 7f0eeebb1740 1 -- 172.21.15.28:0/3014765145 wait complete. 2024-04-21T08:54:16.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.575+0000 7f0eeebb1740 20 client.16704 trim_cache size 0 max 16384 2024-04-21T08:54:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eeebb1740 1 Processor -- start 2024-04-21T08:54:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eeebb1740 1 -- start start 2024-04-21T08:54:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eeebb1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc5c580 0x559d6bc716d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eeebb1740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5cea0 0x559d6bc55bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eeebb1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc65a80 0x559d6bc7d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eeebb1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee41b1bb0 con 0x559d6bc65a80 2024-04-21T08:54:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eeebb1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee41af480 con 0x559d6bc5c580 2024-04-21T08:54:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eeebb1740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee41ad2d0 con 0x559d6bc5cea0 2024-04-21T08:54:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eea55a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc65a80 0x559d6bc7d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eea55a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc65a80 0x559d6bc7d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49292/0 (socket says 172.21.15.28:49292) 2024-04-21T08:54:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eea55a640 1 -- 172.21.15.28:0/3667171451 learned_addr learned my addr 172.21.15.28:0/3667171451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0ee9d59640 1 --2- 172.21.15.28:0/3667171451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc5c580 0x559d6bc716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0ee9558640 1 --2- 172.21.15.28:0/3667171451 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5cea0 0x559d6bc55bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eea55a640 1 -- 172.21.15.28:0/3667171451 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5cea0 msgr2=0x559d6bc55bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eea55a640 1 --2- 172.21.15.28:0/3667171451 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5cea0 0x559d6bc55bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eea55a640 1 -- 172.21.15.28:0/3667171451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc5c580 msgr2=0x559d6bc716d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eea55a640 1 --2- 172.21.15.28:0/3667171451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc5c580 0x559d6bc716d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0eea55a640 1 -- 172.21.15.28:0/3667171451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d6bc67e70 con 0x559d6bc65a80 2024-04-21T08:54:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.593+0000 7f0ee9558640 1 --2- 172.21.15.28:0/3667171451 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5cea0 0x559d6bc55bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.594+0000 7f0eea55a640 1 --2- 172.21.15.28:0/3667171451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc65a80 0x559d6bc7d740 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0edc003e90 tx=0x7f0edc009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81badcaa1f28bd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.594+0000 7f0ecf7fe640 1 -- 172.21.15.28:0/3667171451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc005220 con 0x559d6bc65a80 2024-04-21T08:54:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.594+0000 7f0ecf7fe640 1 -- 172.21.15.28:0/3667171451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0edc004c50 con 0x559d6bc65a80 2024-04-21T08:54:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.594+0000 7f0ecf7fe640 1 -- 172.21.15.28:0/3667171451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc00f6e0 con 0x559d6bc65a80 2024-04-21T08:54:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.594+0000 7f0eeebb1740 1 -- 172.21.15.28:0/3667171451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc65a80 msgr2=0x559d6bc7d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.594+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/3667171451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc65a80 0x559d6bc7d740 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0edc003e90 tx=0x7f0edc009650 comp rx=0 tx=0).stop 2024-04-21T08:54:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.594+0000 7f0eeebb1740 1 -- 172.21.15.28:0/3667171451 shutdown_connections 2024-04-21T08:54:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.594+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/3667171451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc65a80 0x559d6bc7d740 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.594+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/3667171451 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5cea0 0x559d6bc55bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.594+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/3667171451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc5c580 0x559d6bc716d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.594+0000 7f0eeebb1740 1 -- 172.21.15.28:0/3667171451 >> 172.21.15.28:0/3667171451 conn(0x559d6bc56cb0 msgr2=0x559d6bc554e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.594+0000 7f0eeebb1740 1 -- 172.21.15.28:0/3667171451 shutdown_connections 2024-04-21T08:54:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.594+0000 7f0eeebb1740 1 -- 172.21.15.28:0/3667171451 wait complete. 2024-04-21T08:54:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.595+0000 7f0eeebb1740 1 -- start start 2024-04-21T08:54:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.595+0000 7f0eeebb1740 1 Processor -- start 2024-04-21T08:54:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.595+0000 7f0eeebb1740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5c580 0x559d6bc69660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.595+0000 7f0eeebb1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc5cea0 0x559d6bc69ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.595+0000 7f0eeebb1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc65a80 0x559d6bc6a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.595+0000 7f0eeebb1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee41ad940 con 0x559d6bc65a80 2024-04-21T08:54:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.595+0000 7f0eeebb1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee41aedb0 con 0x559d6bc5cea0 2024-04-21T08:54:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.595+0000 7f0eeebb1740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee41ae010 con 0x559d6bc5c580 2024-04-21T08:54:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.595+0000 7f0eeebb1740 10 client.0 _ll_register_callbacks cb 0x7f0ed4010b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:16.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.595+0000 7f0eea55a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc65a80 0x559d6bc6a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.595+0000 7f0eea55a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc65a80 0x559d6bc6a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49302/0 (socket says 172.21.15.28:49302) 2024-04-21T08:54:16.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.595+0000 7f0eea55a640 1 -- 172.21.15.28:0/523235628 learned_addr learned my addr 172.21.15.28:0/523235628 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:16.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.595+0000 7f0ee9558640 1 --2- 172.21.15.28:0/523235628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc5cea0 0x559d6bc69ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0ee9d59640 1 --2- 172.21.15.28:0/523235628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5c580 0x559d6bc69660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0eea55a640 1 -- 172.21.15.28:0/523235628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5c580 msgr2=0x559d6bc69660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0eea55a640 1 --2- 172.21.15.28:0/523235628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5c580 0x559d6bc69660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0eea55a640 1 -- 172.21.15.28:0/523235628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc5cea0 msgr2=0x559d6bc69ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:16.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0eea55a640 1 --2- 172.21.15.28:0/523235628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc5cea0 0x559d6bc69ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0eea55a640 1 -- 172.21.15.28:0/523235628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d6bcdb000 con 0x559d6bc65a80 2024-04-21T08:54:16.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0eea55a640 1 --2- 172.21.15.28:0/523235628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc65a80 0x559d6bc6a0e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f0edc006010 tx=0x7f0edc009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12580f8bdd6976f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:16.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0ecd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:16.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0ecd7fa640 1 -- 172.21.15.28:0/523235628 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc005010 con 0x559d6bc65a80 2024-04-21T08:54:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0ecd7fa640 1 -- 172.21.15.28:0/523235628 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f0edc0051b0 con 0x559d6bc65a80 2024-04-21T08:54:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0eeebb1740 1 -- 172.21.15.28:0/523235628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0edc004c50 con 0x559d6bc65a80 2024-04-21T08:54:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0ecd7fa640 1 -- 172.21.15.28:0/523235628 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0edc040050 con 0x559d6bc65a80 2024-04-21T08:54:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0eeebb1740 10 client.16710 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0eeebb1740 1 -- 172.21.15.28:0/523235628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559d6bcdc310 con 0x559d6bc65a80 2024-04-21T08:54:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0ecd7fa640 1 -- 172.21.15.28:0/523235628 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0edc004c50 con 0x559d6bc65a80 2024-04-21T08:54:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0ecd7fa640 1 -- 172.21.15.28:0/523235628 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f0edc013b90 con 0x559d6bc65a80 2024-04-21T08:54:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.596+0000 7f0eeebb1740 10 client.16710 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.597+0000 7f0eeebb1740 10 client.16710 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.597+0000 7f0eeebb1740 20 client.16710 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.597+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/523235628 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559d6bcdc9b0 0x559d6bcdedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.597+0000 7f0eeebb1740 4 client.16710 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:54:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.597+0000 7f0eeebb1740 1 -- 172.21.15.28:0/523235628 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559d6bbc8a70 con 0x559d6bcdc9b0 2024-04-21T08:54:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.597+0000 7f0ee9d59640 1 --2- 172.21.15.28:0/523235628 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559d6bcdc9b0 0x559d6bcdedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.597+0000 7f0ee9d59640 1 --2- 172.21.15.28:0/523235628 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559d6bcdc9b0 0x559d6bcdedd0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de18304b93b84e46 server_cookie=9e8c602fa9d11af in_seq=0 out_seq=0 2024-04-21T08:54:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.597+0000 7f0ecd7fa640 10 client.16710 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.597+0000 7f0ecd7fa640 1 -- 172.21.15.28:0/523235628 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559d6bbc8a70 con 0x559d6bcdc9b0 2024-04-21T08:54:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.597+0000 7f0ecd7fa640 10 client.16710 handle_command_reply: tid=0 2024-04-21T08:54:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.597+0000 7f0eeebb1740 1 client.16710 shutdown 2024-04-21T08:54:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.597+0000 7f0eeebb1740 1 -- 172.21.15.28:0/523235628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc65a80 msgr2=0x559d6bc6a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.597+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/523235628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc65a80 0x559d6bc6a0e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f0edc006010 tx=0x7f0edc009cc0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eeebb1740 1 -- 172.21.15.28:0/523235628 shutdown_connections 2024-04-21T08:54:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/523235628 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559d6bcdc9b0 0x559d6bcdedd0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/523235628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d6bc65a80 0x559d6bc6a0e0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0ecd7fa640 0 client.16710 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/523235628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559d6bc5cea0 0x559d6bc69ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eeebb1740 1 --2- 172.21.15.28:0/523235628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559d6bc5c580 0x559d6bc69660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eeebb1740 1 -- 172.21.15.28:0/523235628 >> 172.21.15.28:0/523235628 conn(0x559d6bc56cb0 msgr2=0x559d6bc55d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eeebb1740 1 -- 172.21.15.28:0/523235628 shutdown_connections 2024-04-21T08:54:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eeebb1740 1 -- 172.21.15.28:0/523235628 wait complete. 2024-04-21T08:54:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eeebb1740 20 client.16710 trim_cache size 0 max 16384 2024-04-21T08:54:16.605 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:54:16.605 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:54:16.605 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:54:16.605 INFO:teuthology.orchestra.run.smithi028.stdout: "2ba7b196-0f7c-4fe9-87a2-a633a57d659d": { 2024-04-21T08:54:16.605 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:54:16.605 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "2ba7b196-0f7c-4fe9-87a2-a633a57d659d", 2024-04-21T08:54:16.605 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:54:16.605 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:16.606 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:16.606 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:54:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eebfbd640 1 -- 172.21.15.28:0/1067897860 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0ed00a3d20 msgr2=0x7f0ed00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eebfbd640 1 --2- 172.21.15.28:0/1067897860 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0ed00a3d20 0x7f0ed00a61e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0edc004120 tx=0x7f0edc004730 comp rx=0 tx=0).stop 2024-04-21T08:54:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eebfbd640 1 -- 172.21.15.28:0/1067897860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee4106150 msgr2=0x7f0ee4199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eebfbd640 1 --2- 172.21.15.28:0/1067897860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee4106150 0x7f0ee4199a70 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f0ed400d770 tx=0x7f0ed400dc30 comp rx=0 tx=0).stop 2024-04-21T08:54:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eebfbd640 1 -- 172.21.15.28:0/1067897860 shutdown_connections 2024-04-21T08:54:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eebfbd640 1 --2- 172.21.15.28:0/1067897860 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0ed00a3d20 0x7f0ed00a61e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eebfbd640 1 --2- 172.21.15.28:0/1067897860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0ee419a000 0x7f0ee41a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eebfbd640 1 --2- 172.21.15.28:0/1067897860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ee4106150 0x7f0ee4199a70 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eebfbd640 1 --2- 172.21.15.28:0/1067897860 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ee4101080 0x7f0ee4199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eebfbd640 1 -- 172.21.15.28:0/1067897860 >> 172.21.15.28:0/1067897860 conn(0x7f0ee4078000 msgr2=0x7f0ee40ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eebfbd640 1 -- 172.21.15.28:0/1067897860 shutdown_connections 2024-04-21T08:54:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:16.598+0000 7f0eebfbd640 1 -- 172.21.15.28:0/1067897860 wait complete. 2024-04-21T08:54:16.608 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:16 smithi152 ceph-mon[31401]: pgmap v95: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 32 KiB/s wr, 24 op/s 2024-04-21T08:54:16.609 INFO:tasks.cephfs.filesystem:scrub status for tag:2ba7b196-0f7c-4fe9-87a2-a633a57d659d - {'path': '/', 'tag': '2ba7b196-0f7c-4fe9-87a2-a633a57d659d', 'options': 'recursive,force'} 2024-04-21T08:54:16.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:16 smithi171 ceph-mon[31358]: pgmap v95: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 32 KiB/s wr, 24 op/s 2024-04-21T08:54:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:54:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:17.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:17.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:17.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:17.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:17.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:17.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:17.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:17 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:54:17.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:17.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:17.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:17.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:17.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:17.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:17.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:17.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:17.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:17.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:17.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:17.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:17 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:54:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:17.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:17.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:17.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:17.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:17.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:17.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:17.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:17.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:17 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:18.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:18 smithi152 ceph-mon[31401]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-21T08:54:18.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:18 smithi028 ceph-mon[28003]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-21T08:54:18.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:18 smithi171 ceph-mon[31358]: pgmap v96: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-04-21T08:54:19.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:19 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:54:19.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:19 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:54:19.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:19 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:54:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.252+0000 7fe065262640 1 -- 172.21.15.28:0/3856152102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0601006c0 msgr2=0x7fe060100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.252+0000 7fe065262640 1 --2- 172.21.15.28:0/3856152102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0601006c0 0x7fe060100aa0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fe050009900 tx=0x7fe05002bc30 comp rx=0 tx=0).stop 2024-04-21T08:54:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.252+0000 7fe065262640 1 -- 172.21.15.28:0/3856152102 shutdown_connections 2024-04-21T08:54:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.252+0000 7fe065262640 1 --2- 172.21.15.28:0/3856152102 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe060107280 0x7fe06010ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.252+0000 7fe065262640 1 --2- 172.21.15.28:0/3856152102 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe060101070 0x7fe060106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.252+0000 7fe065262640 1 --2- 172.21.15.28:0/3856152102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0601006c0 0x7fe060100aa0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.252+0000 7fe065262640 1 -- 172.21.15.28:0/3856152102 >> 172.21.15.28:0/3856152102 conn(0x7fe060077390 msgr2=0x7fe0600fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.252+0000 7fe065262640 1 -- 172.21.15.28:0/3856152102 shutdown_connections 2024-04-21T08:54:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.252+0000 7fe065262640 1 -- 172.21.15.28:0/3856152102 wait complete. 2024-04-21T08:54:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.253+0000 7fe065262640 1 Processor -- start 2024-04-21T08:54:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.253+0000 7fe065262640 1 -- start start 2024-04-21T08:54:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.253+0000 7fe065262640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0601006c0 0x7fe06019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.253+0000 7fe065262640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe060107280 0x7fe06019e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.253+0000 7fe065262640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0601985c0 0x7fe060198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.253+0000 7fe065262640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0601121e0 con 0x7fe060107280 2024-04-21T08:54:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.253+0000 7fe065262640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe060112060 con 0x7fe0601985c0 2024-04-21T08:54:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.253+0000 7fe065262640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe060112360 con 0x7fe0601006c0 2024-04-21T08:54:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.253+0000 7fe05e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe060107280 0x7fe06019e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.253+0000 7fe05e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe060107280 0x7fe06019e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49316/0 (socket says 172.21.15.28:49316) 2024-04-21T08:54:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.253+0000 7fe05e7fc640 1 -- 172.21.15.28:0/425091525 learned_addr learned my addr 172.21.15.28:0/425091525 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.253+0000 7fe05effd640 1 --2- 172.21.15.28:0/425091525 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0601006c0 0x7fe06019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.254+0000 7fe05f7fe640 1 --2- 172.21.15.28:0/425091525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0601985c0 0x7fe060198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.254+0000 7fe05e7fc640 1 -- 172.21.15.28:0/425091525 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0601006c0 msgr2=0x7fe06019e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.254+0000 7fe05e7fc640 1 --2- 172.21.15.28:0/425091525 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0601006c0 0x7fe06019e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.254+0000 7fe05e7fc640 1 -- 172.21.15.28:0/425091525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0601985c0 msgr2=0x7fe060198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.254+0000 7fe05e7fc640 1 --2- 172.21.15.28:0/425091525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0601985c0 0x7fe060198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.254+0000 7fe05e7fc640 1 -- 172.21.15.28:0/425091525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0601990e0 con 0x7fe060107280 2024-04-21T08:54:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.254+0000 7fe05f7fe640 1 --2- 172.21.15.28:0/425091525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0601985c0 0x7fe060198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.254+0000 7fe05e7fc640 1 --2- 172.21.15.28:0/425091525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe060107280 0x7fe06019e950 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fe05400d740 tx=0x7fe05400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.254+0000 7fe047fff640 1 -- 172.21.15.28:0/425091525 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe054017070 con 0x7fe060107280 2024-04-21T08:54:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.254+0000 7fe065262640 1 -- 172.21.15.28:0/425091525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0601993d0 con 0x7fe060107280 2024-04-21T08:54:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.254+0000 7fe047fff640 1 -- 172.21.15.28:0/425091525 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe05400de50 con 0x7fe060107280 2024-04-21T08:54:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.254+0000 7fe047fff640 1 -- 172.21.15.28:0/425091525 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe054004ce0 con 0x7fe060107280 2024-04-21T08:54:20.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.254+0000 7fe065262640 1 -- 172.21.15.28:0/425091525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0601a55d0 con 0x7fe060107280 2024-04-21T08:54:20.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.257+0000 7fe047fff640 1 -- 172.21.15.28:0/425091525 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe054004e80 con 0x7fe060107280 2024-04-21T08:54:20.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.257+0000 7fe067e4b740 1 Processor -- start 2024-04-21T08:54:20.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.257+0000 7fe067e4b740 1 -- start start 2024-04-21T08:54:20.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.257+0000 7fe067e4b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b58d0 0x556c683b4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.257+0000 7fe067e4b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683b4b60 0x556c683b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.257+0000 7fe067e4b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 0x556c683c04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.257+0000 7fe067e4b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556c683198d0 con 0x556c683b58d0 2024-04-21T08:54:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.257+0000 7fe067e4b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556c682b6e80 con 0x556c683be0e0 2024-04-21T08:54:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.257+0000 7fe067e4b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x556c68315890 con 0x556c683b4b60 2024-04-21T08:54:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe047fff640 1 --2- 172.21.15.28:0/425091525 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0480a3dc0 0x7fe0480a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe05f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b58d0 0x556c683b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe05f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b58d0 0x556c683b4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49320/0 (socket says 172.21.15.28:49320) 2024-04-21T08:54:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe05f7fe640 1 -- 172.21.15.28:0/3975508960 learned_addr learned my addr 172.21.15.28:0/3975508960 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe05e7fc640 1 --2- 172.21.15.28:0/3975508960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 0x556c683c04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe05f7fe640 1 --2- 172.21.15.28:0/425091525 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0480a3dc0 0x7fe0480a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe05effd640 1 --2- 172.21.15.28:0/3975508960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683b4b60 0x556c683b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe047fff640 1 -- 172.21.15.28:0/425091525 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe0540d05f0 con 0x7fe060107280 2024-04-21T08:54:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe05f7fe640 1 -- 172.21.15.28:0/3975508960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683b4b60 msgr2=0x556c683b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe05f7fe640 1 --2- 172.21.15.28:0/3975508960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683b4b60 0x556c683b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe05f7fe640 1 -- 172.21.15.28:0/3975508960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 msgr2=0x556c683c04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe05f7fe640 1 --2- 172.21.15.28:0/3975508960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 0x556c683c04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe05f7fe640 1 -- 172.21.15.28:0/3975508960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c682d8320 con 0x556c683b58d0 2024-04-21T08:54:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe05f7fe640 1 --2- 172.21.15.28:0/3975508960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b58d0 0x556c683b4620 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe058014ea0 tx=0x7fe058014f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901cbec05dee0483 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe05f7fe640 1 --2- 172.21.15.28:0/425091525 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0480a3dc0 0x7fe0480a6280 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe058011c40 tx=0x7fe05801c000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.258+0000 7fe0457fa640 1 -- 172.21.15.28:0/3975508960 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe058022070 con 0x556c683b58d0 2024-04-21T08:54:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.259+0000 7fe0457fa640 1 -- 172.21.15.28:0/3975508960 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe058003e20 con 0x556c683b58d0 2024-04-21T08:54:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.259+0000 7fe0457fa640 1 -- 172.21.15.28:0/3975508960 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe058004290 con 0x556c683b58d0 2024-04-21T08:54:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.259+0000 7fe067e4b740 1 -- 172.21.15.28:0/3975508960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b58d0 msgr2=0x556c683b4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.259+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3975508960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b58d0 0x556c683b4620 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe058014ea0 tx=0x7fe058014f90 comp rx=0 tx=0).stop 2024-04-21T08:54:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.259+0000 7fe067e4b740 1 -- 172.21.15.28:0/3975508960 shutdown_connections 2024-04-21T08:54:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.259+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3975508960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 0x556c683c04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.259+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3975508960 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683b4b60 0x556c683b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.259+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3975508960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b58d0 0x556c683b4620 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.259+0000 7fe067e4b740 1 -- 172.21.15.28:0/3975508960 >> 172.21.15.28:0/3975508960 conn(0x556c6837bc90 msgr2=0x556c683af850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.259+0000 7fe067e4b740 1 -- 172.21.15.28:0/3975508960 shutdown_connections 2024-04-21T08:54:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.259+0000 7fe067e4b740 1 -- 172.21.15.28:0/3975508960 wait complete. 2024-04-21T08:54:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.260+0000 7fe067e4b740 1 -- start start 2024-04-21T08:54:20.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.260+0000 7fe067e4b740 1 Processor -- start 2024-04-21T08:54:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.260+0000 7fe067e4b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683bdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.260+0000 7fe067e4b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683b58d0 0x556c683baee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.260+0000 7fe067e4b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 0x556c683bb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.260+0000 7fe067e4b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556c683c0db0 con 0x556c683b4b60 2024-04-21T08:54:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.260+0000 7fe067e4b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556c682a1eb0 con 0x556c683be0e0 2024-04-21T08:54:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.260+0000 7fe067e4b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x556c683c0f30 con 0x556c683b58d0 2024-04-21T08:54:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.260+0000 7fe067e4b740 10 client.0 _ll_register_callbacks cb 0x556c682ceb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.260+0000 7fe05effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683bdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.260+0000 7fe05effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683bdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49326/0 (socket says 172.21.15.28:49326) 2024-04-21T08:54:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.260+0000 7fe05effd640 1 -- 172.21.15.28:0/3313089674 learned_addr learned my addr 172.21.15.28:0/3313089674 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.260+0000 7fe05e7fc640 1 --2- 172.21.15.28:0/3313089674 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683b58d0 0x556c683baee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.260+0000 7fe05f7fe640 1 --2- 172.21.15.28:0/3313089674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 0x556c683bb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.261+0000 7fe05effd640 1 -- 172.21.15.28:0/3313089674 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683b58d0 msgr2=0x556c683baee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.261+0000 7fe05effd640 1 --2- 172.21.15.28:0/3313089674 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683b58d0 0x556c683baee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.261+0000 7fe05effd640 1 -- 172.21.15.28:0/3313089674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 msgr2=0x556c683bb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.261+0000 7fe05effd640 1 --2- 172.21.15.28:0/3313089674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 0x556c683bb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.261+0000 7fe05effd640 1 -- 172.21.15.28:0/3313089674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c6802c210 con 0x556c683b4b60 2024-04-21T08:54:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.261+0000 7fe05effd640 1 --2- 172.21.15.28:0/3313089674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683bdb50 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe050009900 tx=0x7fe05002c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704a08b857deaa42 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.261+0000 7fe042ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.261+0000 7fe042ff5640 1 -- 172.21.15.28:0/3313089674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05002c610 con 0x556c683b4b60 2024-04-21T08:54:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.261+0000 7fe042ff5640 1 -- 172.21.15.28:0/3313089674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe050005030 con 0x556c683b4b60 2024-04-21T08:54:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.261+0000 7fe067e4b740 1 -- 172.21.15.28:0/3313089674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe058003e20 con 0x556c683b4b60 2024-04-21T08:54:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.261+0000 7fe042ff5640 1 -- 172.21.15.28:0/3313089674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe050052050 con 0x556c683b4b60 2024-04-21T08:54:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.261+0000 7fe067e4b740 10 client.16722 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.261+0000 7fe067e4b740 1 -- 172.21.15.28:0/3313089674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556c680f8620 con 0x556c683b4b60 2024-04-21T08:54:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.261+0000 7fe042ff5640 1 -- 172.21.15.28:0/3313089674 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe058003e20 con 0x556c683b4b60 2024-04-21T08:54:20.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.262+0000 7fe042ff5640 1 -- 172.21.15.28:0/3313089674 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fe0500035f0 con 0x556c683b4b60 2024-04-21T08:54:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.262+0000 7fe067e4b740 10 client.16722 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.262+0000 7fe067e4b740 10 client.16722 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.262+0000 7fe067e4b740 20 client.16722 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.262+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3313089674 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556c683ce210 0x556c683d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.262+0000 7fe067e4b740 4 client.16722 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.262+0000 7fe067e4b740 1 -- 172.21.15.28:0/3313089674 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c6832d880 con 0x556c683ce210 2024-04-21T08:54:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.262+0000 7fe05e7fc640 1 --2- 172.21.15.28:0/3313089674 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556c683ce210 0x556c683d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.262+0000 7fe042ff5640 10 client.16722 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.262+0000 7fe05e7fc640 1 --2- 172.21.15.28:0/3313089674 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556c683ce210 0x556c683d0640 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50484bacb3a07e4f server_cookie=6f577fa3759d4a37 in_seq=0 out_seq=0 2024-04-21T08:54:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.263+0000 7fe042ff5640 1 -- 172.21.15.28:0/3313089674 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556c6832d880 con 0x556c683ce210 2024-04-21T08:54:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.263+0000 7fe042ff5640 10 client.16722 handle_command_reply: tid=0 2024-04-21T08:54:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.263+0000 7fe067e4b740 1 client.16722 shutdown 2024-04-21T08:54:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.263+0000 7fe067e4b740 1 -- 172.21.15.28:0/3313089674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 msgr2=0x556c683bdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:20.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.263+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3313089674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683bdb50 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe050009900 tx=0x7fe05002c9f0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.263+0000 7fe067e4b740 1 -- 172.21.15.28:0/3313089674 shutdown_connections 2024-04-21T08:54:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.263+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3313089674 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556c683ce210 0x556c683d0640 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.263+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3313089674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 0x556c683bb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.263+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3313089674 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683b58d0 0x556c683baee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.263+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3313089674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683bdb50 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.263+0000 7fe067e4b740 1 -- 172.21.15.28:0/3313089674 >> 172.21.15.28:0/3313089674 conn(0x556c6837bc90 msgr2=0x556c683b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.263+0000 7fe042ff5640 0 client.16722 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.263+0000 7fe067e4b740 1 -- 172.21.15.28:0/3313089674 shutdown_connections 2024-04-21T08:54:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.263+0000 7fe067e4b740 1 -- 172.21.15.28:0/3313089674 wait complete. 2024-04-21T08:54:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.263+0000 7fe067e4b740 20 client.16722 trim_cache size 0 max 16384 2024-04-21T08:54:20.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.278+0000 7fe067e4b740 1 Processor -- start 2024-04-21T08:54:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.278+0000 7fe067e4b740 1 -- start start 2024-04-21T08:54:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.278+0000 7fe067e4b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.278+0000 7fe067e4b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683b58d0 0x556c683b9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.278+0000 7fe067e4b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683be0e0 0x556c683bd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe067e4b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0601b1ac0 con 0x556c683b4b60 2024-04-21T08:54:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe067e4b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe0601af390 con 0x556c683b58d0 2024-04-21T08:54:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe067e4b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe0601ad1e0 con 0x556c683be0e0 2024-04-21T08:54:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe05effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe05effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49328/0 (socket says 172.21.15.28:49328) 2024-04-21T08:54:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe05effd640 1 -- 172.21.15.28:0/1061119878 learned_addr learned my addr 172.21.15.28:0/1061119878 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe05effd640 1 -- 172.21.15.28:0/1061119878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683be0e0 msgr2=0x556c683bd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:20.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe05f7fe640 1 --2- 172.21.15.28:0/1061119878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683be0e0 0x556c683bd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe05effd640 1 --2- 172.21.15.28:0/1061119878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683be0e0 0x556c683bd610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe05e7fc640 1 --2- 172.21.15.28:0/1061119878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683b58d0 0x556c683b9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe05effd640 1 -- 172.21.15.28:0/1061119878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683b58d0 msgr2=0x556c683b9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe05effd640 1 --2- 172.21.15.28:0/1061119878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683b58d0 0x556c683b9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe05effd640 1 -- 172.21.15.28:0/1061119878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c683b3e30 con 0x556c683b4b60 2024-04-21T08:54:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe05effd640 1 --2- 172.21.15.28:0/1061119878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683c9cb0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe05002cdf0 tx=0x7fe05002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e4566ab83cb2f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe044ff9640 1 -- 172.21.15.28:0/1061119878 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05002c610 con 0x556c683b4b60 2024-04-21T08:54:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe044ff9640 1 -- 172.21.15.28:0/1061119878 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe050005520 con 0x556c683b4b60 2024-04-21T08:54:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.279+0000 7fe044ff9640 1 -- 172.21.15.28:0/1061119878 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe050004920 con 0x556c683b4b60 2024-04-21T08:54:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.280+0000 7fe067e4b740 1 -- 172.21.15.28:0/1061119878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 msgr2=0x556c683c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.280+0000 7fe067e4b740 1 --2- 172.21.15.28:0/1061119878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683c9cb0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe05002cdf0 tx=0x7fe05002bc30 comp rx=0 tx=0).stop 2024-04-21T08:54:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.280+0000 7fe067e4b740 1 -- 172.21.15.28:0/1061119878 shutdown_connections 2024-04-21T08:54:20.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.280+0000 7fe067e4b740 1 --2- 172.21.15.28:0/1061119878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683be0e0 0x556c683bd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.280+0000 7fe067e4b740 1 --2- 172.21.15.28:0/1061119878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683b58d0 0x556c683b9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.280+0000 7fe067e4b740 1 --2- 172.21.15.28:0/1061119878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683c9cb0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.280+0000 7fe067e4b740 1 -- 172.21.15.28:0/1061119878 >> 172.21.15.28:0/1061119878 conn(0x556c6837bc90 msgr2=0x556c683b97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.280+0000 7fe067e4b740 1 -- 172.21.15.28:0/1061119878 shutdown_connections 2024-04-21T08:54:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.280+0000 7fe067e4b740 1 -- 172.21.15.28:0/1061119878 wait complete. 2024-04-21T08:54:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.280+0000 7fe067e4b740 1 -- start start 2024-04-21T08:54:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.280+0000 7fe067e4b740 1 Processor -- start 2024-04-21T08:54:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe067e4b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe067e4b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683b58d0 0x556c683c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe067e4b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 0x556c683bf140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe067e4b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0601ad850 con 0x556c683b4b60 2024-04-21T08:54:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe067e4b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe0601aecc0 con 0x556c683be0e0 2024-04-21T08:54:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe067e4b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe0601adf20 con 0x556c683b58d0 2024-04-21T08:54:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe067e4b740 10 client.0 _ll_register_callbacks cb 0x7fe050009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe05effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe05effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683b8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49340/0 (socket says 172.21.15.28:49340) 2024-04-21T08:54:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe05effd640 1 -- 172.21.15.28:0/3753138122 learned_addr learned my addr 172.21.15.28:0/3753138122 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe05effd640 1 -- 172.21.15.28:0/3753138122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683b58d0 msgr2=0x556c683c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe05effd640 1 --2- 172.21.15.28:0/3753138122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683b58d0 0x556c683c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe05effd640 1 -- 172.21.15.28:0/3753138122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 msgr2=0x556c683bf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe05f7fe640 1 --2- 172.21.15.28:0/3753138122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 0x556c683bf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe05effd640 1 --2- 172.21.15.28:0/3753138122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 0x556c683bf140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe05effd640 1 -- 172.21.15.28:0/3753138122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c683bf680 con 0x556c683b4b60 2024-04-21T08:54:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe05f7fe640 1 --2- 172.21.15.28:0/3753138122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 0x556c683bf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe05effd640 1 --2- 172.21.15.28:0/3753138122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683b8bc0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fe050058040 tx=0x7fe050003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d87e33af290362e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe042ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe042ff5640 1 -- 172.21.15.28:0/3753138122 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05002c610 con 0x556c683b4b60 2024-04-21T08:54:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe042ff5640 1 -- 172.21.15.28:0/3753138122 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe05005f470 con 0x556c683b4b60 2024-04-21T08:54:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.281+0000 7fe067e4b740 1 -- 172.21.15.28:0/3753138122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe050005520 con 0x556c683b4b60 2024-04-21T08:54:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.282+0000 7fe042ff5640 1 -- 172.21.15.28:0/3753138122 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe050005850 con 0x556c683b4b60 2024-04-21T08:54:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.282+0000 7fe067e4b740 10 client.16728 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.282+0000 7fe067e4b740 1 -- 172.21.15.28:0/3753138122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556c68434eb0 con 0x556c683b4b60 2024-04-21T08:54:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.282+0000 7fe042ff5640 1 -- 172.21.15.28:0/3753138122 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe050005520 con 0x556c683b4b60 2024-04-21T08:54:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.282+0000 7fe042ff5640 1 -- 172.21.15.28:0/3753138122 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fe0500344d0 con 0x556c683b4b60 2024-04-21T08:54:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.282+0000 7fe067e4b740 10 client.16728 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.282+0000 7fe067e4b740 10 client.16728 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.282+0000 7fe067e4b740 20 client.16728 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.282+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3753138122 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556c68435540 0x556c684379a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.282+0000 7fe067e4b740 4 client.16728 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:54:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.282+0000 7fe067e4b740 1 -- 172.21.15.28:0/3753138122 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556c682b93d0 con 0x556c68435540 2024-04-21T08:54:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.282+0000 7fe05e7fc640 1 --2- 172.21.15.28:0/3753138122 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556c68435540 0x556c684379a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:20.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.283+0000 7fe05e7fc640 1 --2- 172.21.15.28:0/3753138122 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556c68435540 0x556c684379a0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e688516c65158bd2 server_cookie=3eba20e460a33331 in_seq=0 out_seq=0 2024-04-21T08:54:20.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.283+0000 7fe042ff5640 10 client.16728 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.293+0000 7fe042ff5640 1 -- 172.21.15.28:0/3753138122 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556c682b93d0 con 0x556c68435540 2024-04-21T08:54:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.293+0000 7fe042ff5640 10 client.16728 handle_command_reply: tid=0 2024-04-21T08:54:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.293+0000 7fe067e4b740 1 client.16728 shutdown 2024-04-21T08:54:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.294+0000 7fe067e4b740 1 -- 172.21.15.28:0/3753138122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 msgr2=0x556c683b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.294+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3753138122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683b8bc0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fe050058040 tx=0x7fe050003f20 comp rx=0 tx=0).stop 2024-04-21T08:54:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.294+0000 7fe067e4b740 1 -- 172.21.15.28:0/3753138122 shutdown_connections 2024-04-21T08:54:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.294+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3753138122 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556c68435540 0x556c684379a0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.294+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3753138122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556c683be0e0 0x556c683bf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.294+0000 7fe042ff5640 0 client.16728 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.294+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3753138122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556c683b58d0 0x556c683c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.294+0000 7fe067e4b740 1 --2- 172.21.15.28:0/3753138122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c683b4b60 0x556c683b8bc0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.294+0000 7fe067e4b740 1 -- 172.21.15.28:0/3753138122 >> 172.21.15.28:0/3753138122 conn(0x556c6837bc90 msgr2=0x556c683ba040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.294+0000 7fe067e4b740 1 -- 172.21.15.28:0/3753138122 shutdown_connections 2024-04-21T08:54:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.294+0000 7fe067e4b740 1 -- 172.21.15.28:0/3753138122 wait complete. 2024-04-21T08:54:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.294+0000 7fe067e4b740 20 client.16728 trim_cache size 0 max 16384 2024-04-21T08:54:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:54:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:54:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:54:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:54:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.295+0000 7fe065262640 1 -- 172.21.15.28:0/425091525 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0480a3dc0 msgr2=0x7fe0480a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.295+0000 7fe065262640 1 --2- 172.21.15.28:0/425091525 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0480a3dc0 0x7fe0480a6280 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe058011c40 tx=0x7fe05801c000 comp rx=0 tx=0).stop 2024-04-21T08:54:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.295+0000 7fe065262640 1 -- 172.21.15.28:0/425091525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe060107280 msgr2=0x7fe06019e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.295+0000 7fe065262640 1 --2- 172.21.15.28:0/425091525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe060107280 0x7fe06019e950 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fe05400d740 tx=0x7fe05400dc00 comp rx=0 tx=0).stop 2024-04-21T08:54:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.295+0000 7fe065262640 1 -- 172.21.15.28:0/425091525 shutdown_connections 2024-04-21T08:54:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.295+0000 7fe065262640 1 --2- 172.21.15.28:0/425091525 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0480a3dc0 0x7fe0480a6280 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.295+0000 7fe065262640 1 --2- 172.21.15.28:0/425091525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0601985c0 0x7fe060198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.295+0000 7fe065262640 1 --2- 172.21.15.28:0/425091525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe060107280 0x7fe06019e950 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.295+0000 7fe065262640 1 --2- 172.21.15.28:0/425091525 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0601006c0 0x7fe06019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.295+0000 7fe065262640 1 -- 172.21.15.28:0/425091525 >> 172.21.15.28:0/425091525 conn(0x7fe060077390 msgr2=0x7fe0601054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.295+0000 7fe065262640 1 -- 172.21.15.28:0/425091525 shutdown_connections 2024-04-21T08:54:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:20.295+0000 7fe065262640 1 -- 172.21.15.28:0/425091525 wait complete. 2024-04-21T08:54:20.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:20 smithi152 ceph-mon[31401]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.6 KiB/s wr, 30 op/s 2024-04-21T08:54:20.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:20 smithi028 ceph-mon[28003]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.6 KiB/s wr, 30 op/s 2024-04-21T08:54:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:20 smithi171 ceph-mon[31358]: pgmap v97: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.6 KiB/s wr, 30 op/s 2024-04-21T08:54:21.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:54:21.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.773+0000 7fc65ad50640 1 -- 172.21.15.28:0/782607107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6541010b0 msgr2=0x7fc654101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:21.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.773+0000 7fc65ad50640 1 --2- 172.21.15.28:0/782607107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6541010b0 0x7fc654101490 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fc648009900 tx=0x7fc64802bc30 comp rx=0 tx=0).stop 2024-04-21T08:54:21.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.773+0000 7fc65ad50640 1 -- 172.21.15.28:0/782607107 shutdown_connections 2024-04-21T08:54:21.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.773+0000 7fc65ad50640 1 --2- 172.21.15.28:0/782607107 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc654106180 0x7fc65410dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.773+0000 7fc65ad50640 1 --2- 172.21.15.28:0/782607107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc654101a60 0x7fc654105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.773+0000 7fc65ad50640 1 --2- 172.21.15.28:0/782607107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6541010b0 0x7fc654101490 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.773+0000 7fc65ad50640 1 -- 172.21.15.28:0/782607107 >> 172.21.15.28:0/782607107 conn(0x7fc6540771d0 msgr2=0x7fc6540775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.774+0000 7fc65ad50640 1 -- 172.21.15.28:0/782607107 shutdown_connections 2024-04-21T08:54:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.774+0000 7fc65ad50640 1 -- 172.21.15.28:0/782607107 wait complete. 2024-04-21T08:54:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.774+0000 7fc65ad50640 1 Processor -- start 2024-04-21T08:54:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.774+0000 7fc65ad50640 1 -- start start 2024-04-21T08:54:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.774+0000 7fc65ad50640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6541010b0 0x7fc654197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.774+0000 7fc65ad50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc654106180 0x7fc654197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.774+0000 7fc65ad50640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc65419bef0 0x7fc65419c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.774+0000 7fc65ad50640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6541110f0 con 0x7fc654106180 2024-04-21T08:54:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.774+0000 7fc65ad50640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc654110f70 con 0x7fc6541010b0 2024-04-21T08:54:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.774+0000 7fc65ad50640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc654111270 con 0x7fc65419bef0 2024-04-21T08:54:21.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.774+0000 7fc653fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc654106180 0x7fc654197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.775+0000 7fc653fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc654106180 0x7fc654197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49352/0 (socket says 172.21.15.28:49352) 2024-04-21T08:54:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.775+0000 7fc653fff640 1 -- 172.21.15.28:0/3561351095 learned_addr learned my addr 172.21.15.28:0/3561351095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.775+0000 7fc6592ed640 1 --2- 172.21.15.28:0/3561351095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc65419bef0 0x7fc65419c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.775+0000 7fc658aec640 1 --2- 172.21.15.28:0/3561351095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6541010b0 0x7fc654197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.775+0000 7fc653fff640 1 -- 172.21.15.28:0/3561351095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc65419bef0 msgr2=0x7fc65419c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.775+0000 7fc653fff640 1 --2- 172.21.15.28:0/3561351095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc65419bef0 0x7fc65419c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.775+0000 7fc653fff640 1 -- 172.21.15.28:0/3561351095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6541010b0 msgr2=0x7fc654197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.775+0000 7fc653fff640 1 --2- 172.21.15.28:0/3561351095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6541010b0 0x7fc654197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.775+0000 7fc653fff640 1 -- 172.21.15.28:0/3561351095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc65419c9e0 con 0x7fc654106180 2024-04-21T08:54:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.775+0000 7fc653fff640 1 --2- 172.21.15.28:0/3561351095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc654106180 0x7fc654197b70 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc64000d770 tx=0x7fc64000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.775+0000 7fc651ffb640 1 -- 172.21.15.28:0/3561351095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc640017070 con 0x7fc654106180 2024-04-21T08:54:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.775+0000 7fc65ad50640 1 -- 172.21.15.28:0/3561351095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc65419ccd0 con 0x7fc654106180 2024-04-21T08:54:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.775+0000 7fc651ffb640 1 -- 172.21.15.28:0/3561351095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc640009c30 con 0x7fc654106180 2024-04-21T08:54:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.775+0000 7fc651ffb640 1 -- 172.21.15.28:0/3561351095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64000f620 con 0x7fc654106180 2024-04-21T08:54:21.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.776+0000 7fc65ad50640 1 -- 172.21.15.28:0/3561351095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6541a44c0 con 0x7fc654106180 2024-04-21T08:54:21.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.777+0000 7fc65ad50640 1 -- 172.21.15.28:0/3561351095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc654069230 con 0x7fc654106180 2024-04-21T08:54:21.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.778+0000 7fc651ffb640 1 -- 172.21.15.28:0/3561351095 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc64001a070 con 0x7fc654106180 2024-04-21T08:54:21.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.778+0000 7fc651ffb640 1 --2- 172.21.15.28:0/3561351095 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc6440a3d20 0x7fc6440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:21.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.778+0000 7fc658aec640 1 --2- 172.21.15.28:0/3561351095 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc6440a3d20 0x7fc6440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:21.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.778+0000 7fc651ffb640 1 -- 172.21.15.28:0/3561351095 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fc6400d0660 con 0x7fc654106180 2024-04-21T08:54:21.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.779+0000 7fc658aec640 1 --2- 172.21.15.28:0/3561351095 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc6440a3d20 0x7fc6440a61e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc648006010 tx=0x7fc648005e40 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:21.783+0000 7fc651ffb640 1 -- 172.21.15.28:0/3561351095 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc640009de0 con 0x7fc654106180 2024-04-21T08:54:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.012+0000 7fc65ad50640 1 -- 172.21.15.28:0/3561351095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc654101a60 con 0x7fc654106180 2024-04-21T08:54:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.013+0000 7fc651ffb640 1 -- 172.21.15.28:0/3561351095 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7fc64009c310 con 0x7fc654106180 2024-04-21T08:54:22.013 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:54:22.013 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:52:11.568185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24487,24488,24463]},"id":1}]} 2024-04-21T08:54:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-21T08:54:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.015+0000 7fc65ad50640 1 -- 172.21.15.28:0/3561351095 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc6440a3d20 msgr2=0x7fc6440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.015+0000 7fc65ad50640 1 --2- 172.21.15.28:0/3561351095 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc6440a3d20 0x7fc6440a61e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc648006010 tx=0x7fc648005e40 comp rx=0 tx=0).stop 2024-04-21T08:54:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.015+0000 7fc65ad50640 1 -- 172.21.15.28:0/3561351095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc654106180 msgr2=0x7fc654197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.015+0000 7fc65ad50640 1 --2- 172.21.15.28:0/3561351095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc654106180 0x7fc654197b70 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fc64000d770 tx=0x7fc64000dc30 comp rx=0 tx=0).stop 2024-04-21T08:54:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.015+0000 7fc65ad50640 1 -- 172.21.15.28:0/3561351095 shutdown_connections 2024-04-21T08:54:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.015+0000 7fc65ad50640 1 --2- 172.21.15.28:0/3561351095 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc6440a3d20 0x7fc6440a61e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.015+0000 7fc65ad50640 1 --2- 172.21.15.28:0/3561351095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc65419bef0 0x7fc65419c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.015+0000 7fc65ad50640 1 --2- 172.21.15.28:0/3561351095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc654106180 0x7fc654197b70 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.015+0000 7fc65ad50640 1 --2- 172.21.15.28:0/3561351095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc6541010b0 0x7fc654197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.015+0000 7fc65ad50640 1 -- 172.21.15.28:0/3561351095 >> 172.21.15.28:0/3561351095 conn(0x7fc6540771d0 msgr2=0x7fc6540fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.015+0000 7fc65ad50640 1 -- 172.21.15.28:0/3561351095 shutdown_connections 2024-04-21T08:54:22.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.015+0000 7fc65ad50640 1 -- 172.21.15.28:0/3561351095 wait complete. 2024-04-21T08:54:22.026 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24488 quiesce db --set-id ec0d6068 --release --await' 2024-04-21T08:54:22.026 DEBUG:teuthology.orchestra.run.smithi028:> timeout 45 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24488 quiesce db --set-id ec0d6068 --release --await 2024-04-21T08:54:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.136+0000 7fd0df643640 1 -- 172.21.15.28:0/1874958796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d8101a40 msgr2=0x7fd0d8105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.136+0000 7fd0df643640 1 --2- 172.21.15.28:0/1874958796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d8101a40 0x7fd0d8105a80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8009900 tx=0x7fd0c802bc30 comp rx=0 tx=0).stop 2024-04-21T08:54:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.136+0000 7fd0df643640 1 -- 172.21.15.28:0/1874958796 shutdown_connections 2024-04-21T08:54:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.136+0000 7fd0df643640 1 --2- 172.21.15.28:0/1874958796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0d8106160 0x7fd0d810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.136+0000 7fd0df643640 1 --2- 172.21.15.28:0/1874958796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d8101a40 0x7fd0d8105a80 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.136+0000 7fd0df643640 1 --2- 172.21.15.28:0/1874958796 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0d8101090 0x7fd0d8101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.136+0000 7fd0df643640 1 -- 172.21.15.28:0/1874958796 >> 172.21.15.28:0/1874958796 conn(0x7fd0d8077f50 msgr2=0x7fd0d8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.136+0000 7fd0df643640 1 -- 172.21.15.28:0/1874958796 shutdown_connections 2024-04-21T08:54:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.136+0000 7fd0df643640 1 -- 172.21.15.28:0/1874958796 wait complete. 2024-04-21T08:54:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0df643640 1 Processor -- start 2024-04-21T08:54:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0df643640 1 -- start start 2024-04-21T08:54:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0df643640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0d8101090 0x7fd0d8197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0df643640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0d8106160 0x7fd0d8197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0df643640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d819bee0 0x7fd0d819c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0df643640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d8111080 con 0x7fd0d819bee0 2024-04-21T08:54:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0df643640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d8110f00 con 0x7fd0d8101090 2024-04-21T08:54:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0df643640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d8111200 con 0x7fd0d8106160 2024-04-21T08:54:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0ddbe0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d819bee0 0x7fd0d819c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0ddbe0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d819bee0 0x7fd0d819c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49360/0 (socket says 172.21.15.28:49360) 2024-04-21T08:54:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0ddbe0640 1 -- 172.21.15.28:0/3479695852 learned_addr learned my addr 172.21.15.28:0/3479695852 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/3479695852 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0d8106160 0x7fd0d8197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0dd3df640 1 --2- 172.21.15.28:0/3479695852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0d8101090 0x7fd0d8197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0ddbe0640 1 -- 172.21.15.28:0/3479695852 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0d8106160 msgr2=0x7fd0d8197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0ddbe0640 1 --2- 172.21.15.28:0/3479695852 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0d8106160 0x7fd0d8197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0ddbe0640 1 -- 172.21.15.28:0/3479695852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0d8101090 msgr2=0x7fd0d8197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0ddbe0640 1 --2- 172.21.15.28:0/3479695852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0d8101090 0x7fd0d8197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.137+0000 7fd0ddbe0640 1 -- 172.21.15.28:0/3479695852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d819c9d0 con 0x7fd0d819bee0 2024-04-21T08:54:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.138+0000 7fd0ddbe0640 1 --2- 172.21.15.28:0/3479695852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d819bee0 0x7fd0d819c390 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000eb10 tx=0x7fd0d000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.138+0000 7fd0ce7fc640 1 -- 172.21.15.28:0/3479695852 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0004b20 con 0x7fd0d819bee0 2024-04-21T08:54:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.138+0000 7fd0ce7fc640 1 -- 172.21.15.28:0/3479695852 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0d0004cc0 con 0x7fd0d819bee0 2024-04-21T08:54:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.138+0000 7fd0df643640 1 -- 172.21.15.28:0/3479695852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0d819ccc0 con 0x7fd0d819bee0 2024-04-21T08:54:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.138+0000 7fd0ce7fc640 1 -- 172.21.15.28:0/3479695852 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0010670 con 0x7fd0d819bee0 2024-04-21T08:54:22.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.138+0000 7fd0df643640 1 -- 172.21.15.28:0/3479695852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0d81a4430 con 0x7fd0d819bee0 2024-04-21T08:54:22.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.140+0000 7fd0ce7fc640 1 -- 172.21.15.28:0/3479695852 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd0d0005020 con 0x7fd0d819bee0 2024-04-21T08:54:22.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.140+0000 7fd0e2239740 1 Processor -- start 2024-04-21T08:54:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0e2239740 1 -- start start 2024-04-21T08:54:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0e2239740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd39fc0 0x55ecebd3a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0e2239740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd3a8e0 0x55ecebd454e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0e2239740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd45a20 0x55ecebd47e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0e2239740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ecebc9f500 con 0x55ecebd45a20 2024-04-21T08:54:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0e2239740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55ecebc456a0 con 0x55ecebd3a8e0 2024-04-21T08:54:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0e2239740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55ecebc9b4c0 con 0x55ecebd39fc0 2024-04-21T08:54:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0ce7fc640 1 --2- 172.21.15.28:0/3479695852 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd0c40a3d20 0x7fd0c40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0dcbde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd45a20 0x55ecebd47e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0dcbde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd45a20 0x55ecebd47e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49368/0 (socket says 172.21.15.28:49368) 2024-04-21T08:54:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0dcbde640 1 -- 172.21.15.28:0/2999154440 learned_addr learned my addr 172.21.15.28:0/2999154440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/2999154440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd39fc0 0x55ecebd3a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0ddbe0640 1 --2- 172.21.15.28:0/3479695852 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd0c40a3d20 0x7fd0c40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0ce7fc640 1 -- 172.21.15.28:0/3479695852 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fd0d00d10a0 con 0x7fd0d819bee0 2024-04-21T08:54:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0dd3df640 1 --2- 172.21.15.28:0/2999154440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd3a8e0 0x55ecebd454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0dcbde640 1 -- 172.21.15.28:0/2999154440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd39fc0 msgr2=0x55ecebd3a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/2999154440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd39fc0 0x55ecebd3a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0dcbde640 1 -- 172.21.15.28:0/2999154440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd3a8e0 msgr2=0x55ecebd454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/2999154440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd3a8e0 0x55ecebd454e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0dcbde640 1 -- 172.21.15.28:0/2999154440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecebc645e0 con 0x55ecebd45a20 2024-04-21T08:54:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.141+0000 7fd0dd3df640 1 --2- 172.21.15.28:0/2999154440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd3a8e0 0x55ecebd454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.142+0000 7fd0ddbe0640 1 --2- 172.21.15.28:0/3479695852 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd0c40a3d20 0x7fd0c40a61e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd0d00ce500 tx=0x7fd0d0092130 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.142+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/2999154440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd45a20 0x55ecebd47e10 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd0d806b6e0 tx=0x7fd0c8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7eb343c65c50a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.142+0000 7fd0c37fe640 1 -- 172.21.15.28:0/2999154440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c804a070 con 0x55ecebd45a20 2024-04-21T08:54:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.142+0000 7fd0c37fe640 1 -- 172.21.15.28:0/2999154440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0c802cc90 con 0x55ecebd45a20 2024-04-21T08:54:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.142+0000 7fd0c37fe640 1 -- 172.21.15.28:0/2999154440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8003db0 con 0x55ecebd45a20 2024-04-21T08:54:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.143+0000 7fd0e2239740 1 -- 172.21.15.28:0/2999154440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd45a20 msgr2=0x55ecebd47e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.143+0000 7fd0e2239740 1 --2- 172.21.15.28:0/2999154440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd45a20 0x55ecebd47e10 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd0d806b6e0 tx=0x7fd0c8005e80 comp rx=0 tx=0).stop 2024-04-21T08:54:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.143+0000 7fd0e2239740 1 -- 172.21.15.28:0/2999154440 shutdown_connections 2024-04-21T08:54:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.143+0000 7fd0e2239740 1 --2- 172.21.15.28:0/2999154440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd45a20 0x55ecebd47e10 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.143+0000 7fd0e2239740 1 --2- 172.21.15.28:0/2999154440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd3a8e0 0x55ecebd454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.143+0000 7fd0e2239740 1 --2- 172.21.15.28:0/2999154440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd39fc0 0x55ecebd3a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.143+0000 7fd0e2239740 1 -- 172.21.15.28:0/2999154440 >> 172.21.15.28:0/2999154440 conn(0x55ecebd34a40 msgr2=0x55ecebd34e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.143+0000 7fd0e2239740 1 -- 172.21.15.28:0/2999154440 shutdown_connections 2024-04-21T08:54:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.143+0000 7fd0e2239740 1 -- 172.21.15.28:0/2999154440 wait complete. 2024-04-21T08:54:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.143+0000 7fd0e2239740 1 -- start start 2024-04-21T08:54:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.143+0000 7fd0e2239740 1 Processor -- start 2024-04-21T08:54:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0e2239740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd39fc0 0x55ecebd454b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0e2239740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd3a8e0 0x55ecebd42840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0e2239740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd45a20 0x55ecebd42f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0e2239740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ecebd48800 con 0x55ecebd3a8e0 2024-04-21T08:54:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0e2239740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55ecebc58a60 con 0x55ecebd39fc0 2024-04-21T08:54:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0e2239740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55ecebd48980 con 0x55ecebd45a20 2024-04-21T08:54:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0e2239740 10 client.0 _ll_register_callbacks cb 0x55eceb9b1110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0dd3df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd3a8e0 0x55ecebd42840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0dd3df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd3a8e0 0x55ecebd42840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49378/0 (socket says 172.21.15.28:49378) 2024-04-21T08:54:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0dd3df640 1 -- 172.21.15.28:0/2991039349 learned_addr learned my addr 172.21.15.28:0/2991039349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/2991039349 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd45a20 0x55ecebd42f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/2991039349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd39fc0 0x55ecebd454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0dd3df640 1 -- 172.21.15.28:0/2991039349 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd45a20 msgr2=0x55ecebd42f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0dd3df640 1 --2- 172.21.15.28:0/2991039349 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd45a20 0x55ecebd42f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0dd3df640 1 -- 172.21.15.28:0/2991039349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd39fc0 msgr2=0x55ecebd454b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0dd3df640 1 --2- 172.21.15.28:0/2991039349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd39fc0 0x55ecebd454b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.144+0000 7fd0dd3df640 1 -- 172.21.15.28:0/2991039349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eceb9b1920 con 0x55ecebd3a8e0 2024-04-21T08:54:22.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0dd3df640 1 --2- 172.21.15.28:0/2991039349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd3a8e0 0x55ecebd42840 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400ba20 tx=0x7fd0d400bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ef5fa877274278 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:22.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:22.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0c0ff9640 1 -- 172.21.15.28:0/2991039349 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d401b640 con 0x55ecebd3a8e0 2024-04-21T08:54:22.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0c0ff9640 1 -- 172.21.15.28:0/2991039349 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fd0d401b7e0 con 0x55ecebd3a8e0 2024-04-21T08:54:22.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0e2239740 1 -- 172.21.15.28:0/2991039349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0c802cc90 con 0x55ecebd3a8e0 2024-04-21T08:54:22.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0c0ff9640 1 -- 172.21.15.28:0/2991039349 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd0d4024050 con 0x55ecebd3a8e0 2024-04-21T08:54:22.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0e2239740 10 client.16746 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:22.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0e2239740 1 -- 172.21.15.28:0/2991039349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55eceba7dd30 con 0x55ecebd3a8e0 2024-04-21T08:54:22.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0c0ff9640 1 -- 172.21.15.28:0/2991039349 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0c802cc90 con 0x55ecebd3a8e0 2024-04-21T08:54:22.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0c0ff9640 1 -- 172.21.15.28:0/2991039349 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd0d4019020 con 0x55ecebd3a8e0 2024-04-21T08:54:22.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0e2239740 10 client.16746 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:22.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0e2239740 10 client.16746 resolve_mds: validated gid 24488 aka daemon mds.b 2024-04-21T08:54:22.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0e2239740 20 client.16746 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:22.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0e2239740 1 --2- 172.21.15.28:0/2991039349 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ecebd55ca0 0x55ecebd580d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0e2239740 4 client.16746 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:22.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.145+0000 7fd0e2239740 1 -- 172.21.15.28:0/2991039349 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecebc72a10 con 0x55ecebd55ca0 2024-04-21T08:54:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.146+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/2991039349 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ecebd55ca0 0x55ecebd580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.146+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/2991039349 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ecebd55ca0 0x55ecebd580d0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f34d0fd6c36564da server_cookie=7d543001cee8af61 in_seq=0 out_seq=0 2024-04-21T08:54:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.146+0000 7fd0c0ff9640 10 client.16746 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.147+0000 7fd0c0ff9640 1 -- 172.21.15.28:0/2991039349 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ecebc72a10 con 0x55ecebd55ca0 2024-04-21T08:54:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.147+0000 7fd0c0ff9640 10 client.16746 handle_command_reply: tid=0 2024-04-21T08:54:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.147+0000 7fd0e2239740 1 client.16746 shutdown 2024-04-21T08:54:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.147+0000 7fd0e2239740 1 -- 172.21.15.28:0/2991039349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd3a8e0 msgr2=0x55ecebd42840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.147+0000 7fd0e2239740 1 --2- 172.21.15.28:0/2991039349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd3a8e0 0x55ecebd42840 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400ba20 tx=0x7fd0d400bee0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.147+0000 7fd0e2239740 1 -- 172.21.15.28:0/2991039349 shutdown_connections 2024-04-21T08:54:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.147+0000 7fd0e2239740 1 --2- 172.21.15.28:0/2991039349 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ecebd55ca0 0x55ecebd580d0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.147+0000 7fd0e2239740 1 --2- 172.21.15.28:0/2991039349 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd45a20 0x55ecebd42f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.147+0000 7fd0e2239740 1 --2- 172.21.15.28:0/2991039349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd3a8e0 0x55ecebd42840 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.147+0000 7fd0e2239740 1 --2- 172.21.15.28:0/2991039349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd39fc0 0x55ecebd454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.147+0000 7fd0e2239740 1 -- 172.21.15.28:0/2991039349 >> 172.21.15.28:0/2991039349 conn(0x55ecebd34a40 msgr2=0x55ecebd36790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.147+0000 7fd0c0ff9640 0 client.16746 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.147+0000 7fd0e2239740 1 -- 172.21.15.28:0/2991039349 shutdown_connections 2024-04-21T08:54:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.147+0000 7fd0e2239740 1 -- 172.21.15.28:0/2991039349 wait complete. 2024-04-21T08:54:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.147+0000 7fd0e2239740 20 client.16746 trim_cache size 0 max 16384 2024-04-21T08:54:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0e2239740 1 Processor -- start 2024-04-21T08:54:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0e2239740 1 -- start start 2024-04-21T08:54:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0e2239740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd39fc0 0x55ecebd41110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0e2239740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd3a8e0 0x55ecebd44f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0e2239740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd45a20 0x55ecebd454b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0e2239740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d81afa00 con 0x55ecebd39fc0 2024-04-21T08:54:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0e2239740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d81ad2d0 con 0x55ecebd3a8e0 2024-04-21T08:54:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0e2239740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d81ab120 con 0x55ecebd45a20 2024-04-21T08:54:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0dcbde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd39fc0 0x55ecebd41110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0dcbde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd39fc0 0x55ecebd41110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49394/0 (socket says 172.21.15.28:49394) 2024-04-21T08:54:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0dcbde640 1 -- 172.21.15.28:0/3575748404 learned_addr learned my addr 172.21.15.28:0/3575748404 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0dd3df640 1 --2- 172.21.15.28:0/3575748404 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd3a8e0 0x55ecebd44f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/3575748404 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd45a20 0x55ecebd454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0dcbde640 1 -- 172.21.15.28:0/3575748404 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd45a20 msgr2=0x55ecebd454b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/3575748404 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd45a20 0x55ecebd454b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0dcbde640 1 -- 172.21.15.28:0/3575748404 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd3a8e0 msgr2=0x55ecebd44f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/3575748404 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd3a8e0 0x55ecebd44f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.165+0000 7fd0dcbde640 1 -- 172.21.15.28:0/3575748404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecebd417f0 con 0x55ecebd39fc0 2024-04-21T08:54:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.166+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/3575748404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd39fc0 0x55ecebd41110 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8000c00 tx=0x7fd0c80043a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26270c3382888bd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.166+0000 7fd0c2ffd640 1 -- 172.21.15.28:0/3575748404 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c804a040 con 0x55ecebd39fc0 2024-04-21T08:54:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.166+0000 7fd0c2ffd640 1 -- 172.21.15.28:0/3575748404 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0c804b040 con 0x55ecebd39fc0 2024-04-21T08:54:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.166+0000 7fd0c2ffd640 1 -- 172.21.15.28:0/3575748404 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8003ac0 con 0x55ecebd39fc0 2024-04-21T08:54:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.166+0000 7fd0e2239740 1 -- 172.21.15.28:0/3575748404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd39fc0 msgr2=0x55ecebd41110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.166+0000 7fd0e2239740 1 --2- 172.21.15.28:0/3575748404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd39fc0 0x55ecebd41110 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8000c00 tx=0x7fd0c80043a0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.166+0000 7fd0e2239740 1 -- 172.21.15.28:0/3575748404 shutdown_connections 2024-04-21T08:54:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.166+0000 7fd0e2239740 1 --2- 172.21.15.28:0/3575748404 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd45a20 0x55ecebd454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.166+0000 7fd0e2239740 1 --2- 172.21.15.28:0/3575748404 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd3a8e0 0x55ecebd44f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.166+0000 7fd0e2239740 1 --2- 172.21.15.28:0/3575748404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd39fc0 0x55ecebd41110 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.166+0000 7fd0e2239740 1 -- 172.21.15.28:0/3575748404 >> 172.21.15.28:0/3575748404 conn(0x55ecebd34a40 msgr2=0x55ecebd33b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.167+0000 7fd0e2239740 1 -- 172.21.15.28:0/3575748404 shutdown_connections 2024-04-21T08:54:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.167+0000 7fd0e2239740 1 -- 172.21.15.28:0/3575748404 wait complete. 2024-04-21T08:54:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.167+0000 7fd0e2239740 1 -- start start 2024-04-21T08:54:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.167+0000 7fd0e2239740 1 Processor -- start 2024-04-21T08:54:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.167+0000 7fd0e2239740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd39fc0 0x55ecebd4dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.167+0000 7fd0e2239740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd3a8e0 0x55ecebd4a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.167+0000 7fd0e2239740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd45a20 0x55ecebd4a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.167+0000 7fd0e2239740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d81ab790 con 0x55ecebd3a8e0 2024-04-21T08:54:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.167+0000 7fd0e2239740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d81acc00 con 0x55ecebd39fc0 2024-04-21T08:54:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.167+0000 7fd0e2239740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d81abe60 con 0x55ecebd45a20 2024-04-21T08:54:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.167+0000 7fd0e2239740 10 client.0 _ll_register_callbacks cb 0x7fd0d40040c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.167+0000 7fd0dd3df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd3a8e0 0x55ecebd4a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0dd3df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd3a8e0 0x55ecebd4a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49404/0 (socket says 172.21.15.28:49404) 2024-04-21T08:54:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0dd3df640 1 -- 172.21.15.28:0/3968449416 learned_addr learned my addr 172.21.15.28:0/3968449416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/3968449416 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd39fc0 0x55ecebd4dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/3968449416 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd45a20 0x55ecebd4a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0dd3df640 1 -- 172.21.15.28:0/3968449416 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd45a20 msgr2=0x55ecebd4a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0dd3df640 1 --2- 172.21.15.28:0/3968449416 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd45a20 0x55ecebd4a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0dd3df640 1 -- 172.21.15.28:0/3968449416 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd39fc0 msgr2=0x55ecebd4dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0dd3df640 1 --2- 172.21.15.28:0/3968449416 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd39fc0 0x55ecebd4dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0dd3df640 1 -- 172.21.15.28:0/3968449416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecebd3a710 con 0x55ecebd3a8e0 2024-04-21T08:54:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/3968449416 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd39fc0 0x55ecebd4dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0dd3df640 1 --2- 172.21.15.28:0/3968449416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd3a8e0 0x55ecebd4a0d0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x55ecebd420a0 tx=0x7fd0d4012fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10af71ab19e1af32 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0c0ff9640 1 -- 172.21.15.28:0/3968449416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4003a80 con 0x55ecebd3a8e0 2024-04-21T08:54:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0e2239740 1 -- 172.21.15.28:0/3968449416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0c804b040 con 0x55ecebd3a8e0 2024-04-21T08:54:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0c0ff9640 1 -- 172.21.15.28:0/3968449416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fd0d4021070 con 0x55ecebd3a8e0 2024-04-21T08:54:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.168+0000 7fd0c0ff9640 1 -- 172.21.15.28:0/3968449416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd0d4037050 con 0x55ecebd3a8e0 2024-04-21T08:54:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.169+0000 7fd0e2239740 10 client.16752 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.169+0000 7fd0e2239740 1 -- 172.21.15.28:0/3968449416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ecebdbce20 con 0x55ecebd3a8e0 2024-04-21T08:54:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.169+0000 7fd0c0ff9640 1 -- 172.21.15.28:0/3968449416 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0c804b040 con 0x55ecebd3a8e0 2024-04-21T08:54:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.169+0000 7fd0c0ff9640 1 -- 172.21.15.28:0/3968449416 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd0d4019070 con 0x55ecebd3a8e0 2024-04-21T08:54:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.169+0000 7fd0e2239740 10 client.16752 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.169+0000 7fd0e2239740 10 client.16752 resolve_mds: validated gid 24488 aka daemon mds.b 2024-04-21T08:54:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.169+0000 7fd0e2239740 20 client.16752 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.169+0000 7fd0e2239740 1 --2- 172.21.15.28:0/3968449416 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ecebdbd620 0x55ecebdbfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.169+0000 7fd0e2239740 4 client.16752 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ec0d6068", "release": true, "await": true}] 2024-04-21T08:54:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.169+0000 7fd0e2239740 1 -- 172.21.15.28:0/3968449416 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ec0d6068", "release": true, "await": true}) -- 0x55ecebcacf20 con 0x55ecebdbd620 2024-04-21T08:54:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.169+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/3968449416 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ecebdbd620 0x55ecebdbfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.170+0000 7fd0dcbde640 1 --2- 172.21.15.28:0/3968449416 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ecebdbd620 0x55ecebdbfa40 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19a826fff690a1f8 server_cookie=2e8a356dad4b2d30 in_seq=0 out_seq=0 2024-04-21T08:54:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.170+0000 7fd0c0ff9640 10 client.16752 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.188+0000 7fd0c0ff9640 1 -- 172.21.15.28:0/3968449416 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55ecebcacf20 con 0x55ecebdbd620 2024-04-21T08:54:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.188+0000 7fd0c0ff9640 10 client.16752 handle_command_reply: tid=0 2024-04-21T08:54:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.188+0000 7fd0e2239740 1 client.16752 shutdown 2024-04-21T08:54:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.189+0000 7fd0e2239740 1 -- 172.21.15.28:0/3968449416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd3a8e0 msgr2=0x55ecebd4a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.189+0000 7fd0e2239740 1 --2- 172.21.15.28:0/3968449416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd3a8e0 0x55ecebd4a0d0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x55ecebd420a0 tx=0x7fd0d4012fa0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.189+0000 7fd0e2239740 1 -- 172.21.15.28:0/3968449416 shutdown_connections 2024-04-21T08:54:22.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.189+0000 7fd0e2239740 1 --2- 172.21.15.28:0/3968449416 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ecebdbd620 0x55ecebdbfa40 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.189+0000 7fd0e2239740 1 --2- 172.21.15.28:0/3968449416 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ecebd45a20 0x55ecebd4a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.189+0000 7fd0e2239740 1 --2- 172.21.15.28:0/3968449416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecebd3a8e0 0x55ecebd4a0d0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.189+0000 7fd0e2239740 1 --2- 172.21.15.28:0/3968449416 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ecebd39fc0 0x55ecebd4dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.189+0000 7fd0e2239740 1 -- 172.21.15.28:0/3968449416 >> 172.21.15.28:0/3968449416 conn(0x55ecebd34a40 msgr2=0x55ecebd33b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:22.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.191+0000 7fd0c0ff9640 0 client.16752 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.192+0000 7fd0e2239740 1 -- 172.21.15.28:0/3968449416 shutdown_connections 2024-04-21T08:54:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.192+0000 7fd0e2239740 1 -- 172.21.15.28:0/3968449416 wait complete. 2024-04-21T08:54:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.192+0000 7fd0e2239740 20 client.16752 trim_cache size 0 max 16384 2024-04-21T08:54:22.192 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:54:22.192 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 26, 2024-04-21T08:54:22.192 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24488, 2024-04-21T08:54:22.192 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 4, 2024-04-21T08:54:22.192 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-04-21T08:54:22.192 INFO:teuthology.orchestra.run.smithi028.stdout: "ec0d6068": { 2024-04-21T08:54:22.192 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 4, 2024-04-21T08:54:22.192 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-04-21T08:54:22.192 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-21T08:54:22.193 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-04-21T08:54:22.193 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-21T08:54:22.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T08:54:22.193 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 30.0, 2024-04-21T08:54:22.193 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 84.5, 2024-04-21T08:54:22.193 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-04-21T08:54:22.193 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-04-21T08:54:22.193 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-04-21T08:54:22.193 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-21T08:54:22.193 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-04-21T08:54:22.193 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-21T08:54:22.193 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:22.193 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:22.193 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:22.193 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:22.194 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:22.194 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:54:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.193+0000 7fd0df643640 1 -- 172.21.15.28:0/3479695852 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd0c40a3d20 msgr2=0x7fd0c40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.193+0000 7fd0df643640 1 --2- 172.21.15.28:0/3479695852 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd0c40a3d20 0x7fd0c40a61e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd0d00ce500 tx=0x7fd0d0092130 comp rx=0 tx=0).stop 2024-04-21T08:54:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.193+0000 7fd0df643640 1 -- 172.21.15.28:0/3479695852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d819bee0 msgr2=0x7fd0d819c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.193+0000 7fd0df643640 1 --2- 172.21.15.28:0/3479695852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d819bee0 0x7fd0d819c390 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000eb10 tx=0x7fd0d000c2d0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.193+0000 7fd0df643640 1 -- 172.21.15.28:0/3479695852 shutdown_connections 2024-04-21T08:54:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.193+0000 7fd0df643640 1 --2- 172.21.15.28:0/3479695852 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd0c40a3d20 0x7fd0c40a61e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.193+0000 7fd0df643640 1 --2- 172.21.15.28:0/3479695852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0d819bee0 0x7fd0d819c390 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.193+0000 7fd0df643640 1 --2- 172.21.15.28:0/3479695852 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0d8106160 0x7fd0d8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.193+0000 7fd0df643640 1 --2- 172.21.15.28:0/3479695852 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0d8101090 0x7fd0d8197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.193+0000 7fd0df643640 1 -- 172.21.15.28:0/3479695852 >> 172.21.15.28:0/3479695852 conn(0x7fd0d8077f50 msgr2=0x7fd0d80ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.193+0000 7fd0df643640 1 -- 172.21.15.28:0/3479695852 shutdown_connections 2024-04-21T08:54:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.193+0000 7fd0df643640 1 -- 172.21.15.28:0/3479695852 wait complete. 2024-04-21T08:54:22.212 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: ec0d6068, seconds elapsed: 53.9 2024-04-21T08:54:22.212 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.5 2024-04-21T08:54:22.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:54:22.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:22 smithi028 ceph-mon[28003]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.1 KiB/s wr, 31 op/s 2024-04-21T08:54:22.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:22 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:54:22.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:22 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:54:22.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:22 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:54:22.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:22 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3561351095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:22.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:22 smithi152 ceph-mon[31401]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.1 KiB/s wr, 31 op/s 2024-04-21T08:54:22.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:22 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:54:22.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:22 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:54:22.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:22 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:54:22.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:22 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3561351095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:22.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:22 smithi171 ceph-mon[31358]: pgmap v98: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.1 KiB/s wr, 31 op/s 2024-04-21T08:54:22.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:22 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:54:22.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:22 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:54:22.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:22 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:54:22.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:22 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3561351095' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:22.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.920+0000 7f6ee1c06640 1 -- 172.21.15.28:0/2437092024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6edc110550 msgr2=0x7f6edc110930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:22.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.920+0000 7f6ee1c06640 1 --2- 172.21.15.28:0/2437092024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6edc110550 0x7f6edc110930 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009ca0 tx=0x7f6ed402c190 comp rx=0 tx=0).stop 2024-04-21T08:54:22.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.928+0000 7f6ee1c06640 1 -- 172.21.15.28:0/2437092024 shutdown_connections 2024-04-21T08:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.928+0000 7f6ee1c06640 1 --2- 172.21.15.28:0/2437092024 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6edc10eb40 0x7f6edc11d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.928+0000 7f6ee1c06640 1 --2- 172.21.15.28:0/2437092024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc10e0c0 0x7f6edc10e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.928+0000 7f6ee1c06640 1 --2- 172.21.15.28:0/2437092024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6edc110550 0x7f6edc110930 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.928+0000 7f6ee1c06640 1 -- 172.21.15.28:0/2437092024 >> 172.21.15.28:0/2437092024 conn(0x7f6edc06ce80 msgr2=0x7f6edc06d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:22.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.939+0000 7f6ee1c06640 1 -- 172.21.15.28:0/2437092024 shutdown_connections 2024-04-21T08:54:22.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.939+0000 7f6ee1c06640 1 -- 172.21.15.28:0/2437092024 wait complete. 2024-04-21T08:54:22.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.940+0000 7f6ee1c06640 1 Processor -- start 2024-04-21T08:54:22.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.940+0000 7f6ee1c06640 1 -- start start 2024-04-21T08:54:22.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.941+0000 7f6ee1c06640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6edc10e0c0 0x7f6edc1a7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.941+0000 7f6ee1c06640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc10eb40 0x7f6edc1a79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.941+0000 7f6ee1c06640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6edc110550 0x7f6edc1abd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.941+0000 7f6ee1c06640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc120f20 con 0x7f6edc10eb40 2024-04-21T08:54:22.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.941+0000 7f6ee1c06640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc120da0 con 0x7f6edc110550 2024-04-21T08:54:22.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.941+0000 7f6ee1c06640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc1210a0 con 0x7f6edc10e0c0 2024-04-21T08:54:22.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.941+0000 7f6edaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc10eb40 0x7f6edc1a79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.942+0000 7f6edaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc10eb40 0x7f6edc1a79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49416/0 (socket says 172.21.15.28:49416) 2024-04-21T08:54:22.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.942+0000 7f6edaffd640 1 -- 172.21.15.28:0/3750901636 learned_addr learned my addr 172.21.15.28:0/3750901636 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:22.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.946+0000 7f6edbfff640 1 --2- 172.21.15.28:0/3750901636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6edc110550 0x7f6edc1abd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.946+0000 7f6edb7fe640 1 --2- 172.21.15.28:0/3750901636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6edc10e0c0 0x7f6edc1a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.948+0000 7f6edaffd640 1 -- 172.21.15.28:0/3750901636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6edc10e0c0 msgr2=0x7f6edc1a7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.948+0000 7f6edaffd640 1 --2- 172.21.15.28:0/3750901636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6edc10e0c0 0x7f6edc1a7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.948+0000 7f6edaffd640 1 -- 172.21.15.28:0/3750901636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6edc110550 msgr2=0x7f6edc1abd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.948+0000 7f6edaffd640 1 --2- 172.21.15.28:0/3750901636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6edc110550 0x7f6edc1abd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.948+0000 7f6edaffd640 1 -- 172.21.15.28:0/3750901636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6edc1ac420 con 0x7f6edc10eb40 2024-04-21T08:54:22.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.953+0000 7f6edaffd640 1 --2- 172.21.15.28:0/3750901636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc10eb40 0x7f6edc1a79a0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc00b450 tx=0x7f6ecc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.957+0000 7f6ed8ff9640 1 -- 172.21.15.28:0/3750901636 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc015020 con 0x7f6edc10eb40 2024-04-21T08:54:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.957+0000 7f6ed8ff9640 1 -- 172.21.15.28:0/3750901636 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ecc011d40 con 0x7f6edc10eb40 2024-04-21T08:54:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.957+0000 7f6ed8ff9640 1 -- 172.21.15.28:0/3750901636 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc00f340 con 0x7f6edc10eb40 2024-04-21T08:54:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.958+0000 7f6ee1c06640 1 -- 172.21.15.28:0/3750901636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6edc1ac710 con 0x7f6edc10eb40 2024-04-21T08:54:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.958+0000 7f6ee1c06640 1 -- 172.21.15.28:0/3750901636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6edc110f10 con 0x7f6edc10eb40 2024-04-21T08:54:22.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.964+0000 7f6ed8ff9640 1 -- 172.21.15.28:0/3750901636 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6ecc018020 con 0x7f6edc10eb40 2024-04-21T08:54:22.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.965+0000 7f6ed8ff9640 1 --2- 172.21.15.28:0/3750901636 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f6ec40a3f30 0x7f6ec40a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:22.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.965+0000 7f6ed8ff9640 1 -- 172.21.15.28:0/3750901636 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f6ecc0d1430 con 0x7f6edc10eb40 2024-04-21T08:54:22.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.968+0000 7f6edb7fe640 1 --2- 172.21.15.28:0/3750901636 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f6ec40a3f30 0x7f6ec40a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.977+0000 7f6eca7fc640 1 -- 172.21.15.28:0/3750901636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ec0005290 con 0x7f6edc10eb40 2024-04-21T08:54:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.994+0000 7f6ed8ff9640 1 -- 172.21.15.28:0/3750901636 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ecc01f020 con 0x7f6edc10eb40 2024-04-21T08:54:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:22.996+0000 7f6edb7fe640 1 --2- 172.21.15.28:0/3750901636 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f6ec40a3f30 0x7f6ec40a63f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4032500 tx=0x7f6ed4000f40 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.668+0000 7f6eca7fc640 1 -- 172.21.15.28:0/3750901636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ec0005850 con 0x7f6edc10eb40 2024-04-21T08:54:23.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.672+0000 7f6ed8ff9640 1 -- 172.21.15.28:0/3750901636 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7f6ecc00d040 con 0x7f6edc10eb40 2024-04-21T08:54:23.674 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:54:23.674 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:52:11.568185+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24487,24488,24463]},"id":1}]} 2024-04-21T08:54:23.674 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-21T08:54:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.678+0000 7f6eca7fc640 1 -- 172.21.15.28:0/3750901636 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f6ec40a3f30 msgr2=0x7f6ec40a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.678+0000 7f6eca7fc640 1 --2- 172.21.15.28:0/3750901636 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f6ec40a3f30 0x7f6ec40a63f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4032500 tx=0x7f6ed4000f40 comp rx=0 tx=0).stop 2024-04-21T08:54:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.678+0000 7f6eca7fc640 1 -- 172.21.15.28:0/3750901636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc10eb40 msgr2=0x7f6edc1a79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.679+0000 7f6eca7fc640 1 --2- 172.21.15.28:0/3750901636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc10eb40 0x7f6edc1a79a0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc00b450 tx=0x7f6ecc00b910 comp rx=0 tx=0).stop 2024-04-21T08:54:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.681+0000 7f6eca7fc640 1 -- 172.21.15.28:0/3750901636 shutdown_connections 2024-04-21T08:54:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.681+0000 7f6eca7fc640 1 --2- 172.21.15.28:0/3750901636 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f6ec40a3f30 0x7f6ec40a63f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.681+0000 7f6eca7fc640 1 --2- 172.21.15.28:0/3750901636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6edc110550 0x7f6edc1abd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.682+0000 7f6eca7fc640 1 --2- 172.21.15.28:0/3750901636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc10eb40 0x7f6edc1a79a0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.682+0000 7f6eca7fc640 1 --2- 172.21.15.28:0/3750901636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6edc10e0c0 0x7f6edc1a7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.682+0000 7f6eca7fc640 1 -- 172.21.15.28:0/3750901636 >> 172.21.15.28:0/3750901636 conn(0x7f6edc06ce80 msgr2=0x7f6edc071280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.682+0000 7f6eca7fc640 1 -- 172.21.15.28:0/3750901636 shutdown_connections 2024-04-21T08:54:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.682+0000 7f6eca7fc640 1 -- 172.21.15.28:0/3750901636 wait complete. 2024-04-21T08:54:23.703 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24488 quiesce db / --timeout 30 --expiration 84.5 --await' 2024-04-21T08:54:23.703 DEBUG:teuthology.orchestra.run.smithi028:> timeout 45 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24488 quiesce db / --timeout 30 --expiration 84.5 --await 2024-04-21T08:54:23.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.903+0000 7f45b3154640 1 -- 172.21.15.28:0/3027743600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a40a9190 msgr2=0x7f45a40a9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:23.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.903+0000 7f45b3154640 1 --2- 172.21.15.28:0/3027743600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a40a9190 0x7f45a40a9570 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009900 tx=0x7f45a802bc60 comp rx=0 tx=0).stop 2024-04-21T08:54:23.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.904+0000 7f45b3154640 1 -- 172.21.15.28:0/3027743600 shutdown_connections 2024-04-21T08:54:23.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.904+0000 7f45b3154640 1 --2- 172.21.15.28:0/3027743600 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45a40ae260 0x7f45a40b5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.904+0000 7f45b3154640 1 --2- 172.21.15.28:0/3027743600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45a40a9b40 0x7f45a40adb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.904+0000 7f45b3154640 1 --2- 172.21.15.28:0/3027743600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a40a9190 0x7f45a40a9570 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.904+0000 7f45b3154640 1 -- 172.21.15.28:0/3027743600 >> 172.21.15.28:0/3027743600 conn(0x7f45a40a49d0 msgr2=0x7f45a40a6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.905+0000 7f45b3154640 1 -- 172.21.15.28:0/3027743600 shutdown_connections 2024-04-21T08:54:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.905+0000 7f45b3154640 1 -- 172.21.15.28:0/3027743600 wait complete. 2024-04-21T08:54:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.905+0000 7f45b3154640 1 Processor -- start 2024-04-21T08:54:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.906+0000 7f45b3154640 1 -- start start 2024-04-21T08:54:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.909+0000 7f45b3154640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45a40a9190 0x7f45a401ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.909+0000 7f45b3154640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45a40ae260 0x7f45a401b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.909+0000 7f45b3154640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a401b7e0 0x7f45a414fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.909+0000 7f45b3154640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45a40b9270 con 0x7f45a401b7e0 2024-04-21T08:54:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.909+0000 7f45b3154640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f45a40b90f0 con 0x7f45a40ae260 2024-04-21T08:54:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.909+0000 7f45b3154640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f45a40b93f0 con 0x7f45a40a9190 2024-04-21T08:54:23.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.910+0000 7f45b1951640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45a40ae260 0x7f45a401b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.910+0000 7f45b2953640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a401b7e0 0x7f45a414fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.910+0000 7f45b1951640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45a40ae260 0x7f45a401b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:45068/0 (socket says 172.21.15.28:45068) 2024-04-21T08:54:23.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.910+0000 7f45b1951640 1 -- 172.21.15.28:0/290197714 learned_addr learned my addr 172.21.15.28:0/290197714 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:23.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.910+0000 7f45b2152640 1 --2- 172.21.15.28:0/290197714 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45a40a9190 0x7f45a401ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.910+0000 7f45b1951640 1 -- 172.21.15.28:0/290197714 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45a40a9190 msgr2=0x7f45a401ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:23.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.910+0000 7f45b1951640 1 --2- 172.21.15.28:0/290197714 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45a40a9190 0x7f45a401ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.911+0000 7f45b1951640 1 -- 172.21.15.28:0/290197714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a401b7e0 msgr2=0x7f45a414fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:23.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.911+0000 7f45b1951640 1 --2- 172.21.15.28:0/290197714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a401b7e0 0x7f45a414fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.911+0000 7f45b1951640 1 -- 172.21.15.28:0/290197714 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a41502c0 con 0x7f45a40ae260 2024-04-21T08:54:23.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.911+0000 7f45b1951640 1 --2- 172.21.15.28:0/290197714 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45a40ae260 0x7f45a401b2a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f459c00b470 tx=0x7f459c00b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:23.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.911+0000 7f45a37fe640 1 -- 172.21.15.28:0/290197714 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c015020 con 0x7f45a40ae260 2024-04-21T08:54:23.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.911+0000 7f45b3154640 1 -- 172.21.15.28:0/290197714 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a41505b0 con 0x7f45a40ae260 2024-04-21T08:54:23.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.912+0000 7f45b3154640 1 -- 172.21.15.28:0/290197714 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a4150ad0 con 0x7f45a40ae260 2024-04-21T08:54:23.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.912+0000 7f45a37fe640 1 -- 172.21.15.28:0/290197714 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f459c011d40 con 0x7f45a40ae260 2024-04-21T08:54:23.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.912+0000 7f45a37fe640 1 -- 172.21.15.28:0/290197714 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c00f340 con 0x7f45a40ae260 2024-04-21T08:54:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.917+0000 7f45b6fad740 1 Processor -- start 2024-04-21T08:54:23.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.918+0000 7f45a37fe640 1 -- 172.21.15.28:0/290197714 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f459c018020 con 0x7f45a40ae260 2024-04-21T08:54:23.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.918+0000 7f45b6fad740 1 -- start start 2024-04-21T08:54:23.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.918+0000 7f45b6fad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a7b90 0x555f961a7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.918+0000 7f45b6fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961a84b0 0x555f961b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.918+0000 7f45b6fad740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961b1090 0x555f961b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.918+0000 7f45b6fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555f961080b0 con 0x555f961a84b0 2024-04-21T08:54:23.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.918+0000 7f45b6fad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x555f960b9680 con 0x555f961a7b90 2024-04-21T08:54:23.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.918+0000 7f45b6fad740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x555f960c5c60 con 0x555f961b1090 2024-04-21T08:54:23.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.919+0000 7f45a37fe640 1 --2- 172.21.15.28:0/290197714 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f45980a3e60 0x7f45980a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.919+0000 7f45a37fe640 1 -- 172.21.15.28:0/290197714 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f459c0d03c0 con 0x7f45a40ae260 2024-04-21T08:54:23.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.919+0000 7f45b2953640 1 --2- 172.21.15.28:0/290197714 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f45980a3e60 0x7f45980a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.919+0000 7f45b1951640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961b1090 0x555f961b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.919+0000 7f45b1951640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961b1090 0x555f961b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:44710/0 (socket says 172.21.15.28:44710) 2024-04-21T08:54:23.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.920+0000 7f45b1951640 1 -- 172.21.15.28:0/4225085449 learned_addr learned my addr 172.21.15.28:0/4225085449 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:23.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.920+0000 7f45b2152640 1 --2- 172.21.15.28:0/4225085449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961a84b0 0x555f961b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.920+0000 7f45b2152640 1 -- 172.21.15.28:0/4225085449 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961b1090 msgr2=0x555f961b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.920+0000 7f45b2953640 1 --2- 172.21.15.28:0/290197714 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f45980a3e60 0x7f45980a6320 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f45ac050760 tx=0x7f45ac067800 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.920+0000 7f45b2152640 1 --2- 172.21.15.28:0/4225085449 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961b1090 0x555f961b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.920+0000 7f45b2152640 1 -- 172.21.15.28:0/4225085449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a7b90 msgr2=0x555f961a7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.920+0000 7f45b2152640 1 --2- 172.21.15.28:0/4225085449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a7b90 0x555f961a7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.920+0000 7f45b2152640 1 -- 172.21.15.28:0/4225085449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f95eeac90 con 0x555f961a84b0 2024-04-21T08:54:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.921+0000 7f45b1951640 1 --2- 172.21.15.28:0/4225085449 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961b1090 0x555f961b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.921+0000 7f45b2152640 1 --2- 172.21.15.28:0/4225085449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961a84b0 0x555f961b0b50 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009850 tx=0x7f45a80095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01acd8d7ef1432c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:23.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.922+0000 7f45a0bfe640 1 -- 172.21.15.28:0/4225085449 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a8044020 con 0x555f961a84b0 2024-04-21T08:54:23.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.922+0000 7f45a0bfe640 1 -- 172.21.15.28:0/4225085449 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45a802ccc0 con 0x555f961a84b0 2024-04-21T08:54:23.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.922+0000 7f45a0bfe640 1 -- 172.21.15.28:0/4225085449 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a80046c0 con 0x555f961a84b0 2024-04-21T08:54:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.923+0000 7f45b6fad740 1 -- 172.21.15.28:0/4225085449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961a84b0 msgr2=0x555f961b0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.923+0000 7f45b6fad740 1 --2- 172.21.15.28:0/4225085449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961a84b0 0x555f961b0b50 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009850 tx=0x7f45a80095f0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.924+0000 7f45b6fad740 1 -- 172.21.15.28:0/4225085449 shutdown_connections 2024-04-21T08:54:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.924+0000 7f45b6fad740 1 --2- 172.21.15.28:0/4225085449 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961b1090 0x555f961b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.924+0000 7f45b6fad740 1 --2- 172.21.15.28:0/4225085449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961a84b0 0x555f961b0b50 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.924+0000 7f45b6fad740 1 --2- 172.21.15.28:0/4225085449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a7b90 0x555f961a7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.924+0000 7f45b6fad740 1 -- 172.21.15.28:0/4225085449 >> 172.21.15.28:0/4225085449 conn(0x555f961a22c0 msgr2=0x555f961a26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.924+0000 7f45b6fad740 1 -- 172.21.15.28:0/4225085449 shutdown_connections 2024-04-21T08:54:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.924+0000 7f45b6fad740 1 -- 172.21.15.28:0/4225085449 wait complete. 2024-04-21T08:54:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.925+0000 7f45b6fad740 1 -- start start 2024-04-21T08:54:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.925+0000 7f45b6fad740 1 Processor -- start 2024-04-21T08:54:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.925+0000 7f45b6fad740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961a7b90 0x555f961ae060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.925+0000 7f45b6fad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a84b0 0x555f961ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.925+0000 7f45b6fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961b1090 0x555f961aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.925+0000 7f45b6fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555f9610b400 con 0x555f961b1090 2024-04-21T08:54:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.925+0000 7f45b6fad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x555f960c7840 con 0x555f961a84b0 2024-04-21T08:54:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.925+0000 7f45b6fad740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x555f96110720 con 0x555f961a7b90 2024-04-21T08:54:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.925+0000 7f45b6fad740 10 client.0 _ll_register_callbacks cb 0x555f96118ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.926+0000 7f45b2152640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961a7b90 0x555f961ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.926+0000 7f45b2953640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961b1090 0x555f961aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.926+0000 7f45b2152640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961a7b90 0x555f961ae060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:44716/0 (socket says 172.21.15.28:44716) 2024-04-21T08:54:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.926+0000 7f45b2152640 1 -- 172.21.15.28:0/4232197079 learned_addr learned my addr 172.21.15.28:0/4232197079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.926+0000 7f45b1951640 1 --2- 172.21.15.28:0/4232197079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a84b0 0x555f961ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.926+0000 7f45b2152640 1 -- 172.21.15.28:0/4232197079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a84b0 msgr2=0x555f961ae5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.926+0000 7f45b2152640 1 --2- 172.21.15.28:0/4232197079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a84b0 0x555f961ae5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.926+0000 7f45b2152640 1 -- 172.21.15.28:0/4232197079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961b1090 msgr2=0x555f961aeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.926+0000 7f45b2152640 1 --2- 172.21.15.28:0/4232197079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961b1090 0x555f961aeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.926+0000 7f45b2152640 1 -- 172.21.15.28:0/4232197079 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f95f4b420 con 0x555f961a7b90 2024-04-21T08:54:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.927+0000 7f45b2152640 1 --2- 172.21.15.28:0/4232197079 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961a7b90 0x555f961ae060 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f45a80045f0 tx=0x7f45a802cb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8728321ea381ecf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.927+0000 7f4595ffb640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:54:23.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.928+0000 7f4595ffb640 1 -- 172.21.15.28:0/4232197079 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a8044030 con 0x555f961a7b90 2024-04-21T08:54:23.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.928+0000 7f4595ffb640 1 -- 172.21.15.28:0/4232197079 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f45a80054d0 con 0x555f961a7b90 2024-04-21T08:54:23.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.928+0000 7f45b6fad740 1 -- 172.21.15.28:0/4232197079 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45a802ccc0 con 0x555f961a7b90 2024-04-21T08:54:23.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.929+0000 7f4595ffb640 1 -- 172.21.15.28:0/4232197079 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f45a80046c0 con 0x555f961a7b90 2024-04-21T08:54:23.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.929+0000 7f45b6fad740 10 client.26080 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.929+0000 7f45b6fad740 1 -- 172.21.15.28:0/4232197079 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555f95dde580 con 0x555f961a7b90 2024-04-21T08:54:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.929+0000 7f4595ffb640 1 -- 172.21.15.28:0/4232197079 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45a802ccc0 con 0x555f961a7b90 2024-04-21T08:54:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.929+0000 7f4595ffb640 1 -- 172.21.15.28:0/4232197079 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f45a8003a00 con 0x555f961a7b90 2024-04-21T08:54:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.929+0000 7f45b6fad740 10 client.26080 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.929+0000 7f45b6fad740 10 client.26080 resolve_mds: validated gid 24488 aka daemon mds.b 2024-04-21T08:54:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.929+0000 7f45b6fad740 20 client.26080 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.929+0000 7f45b6fad740 1 --2- 172.21.15.28:0/4232197079 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555f961c0970 0x555f961c2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.929+0000 7f45b6fad740 4 client.26080 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.929+0000 7f45b6fad740 1 -- 172.21.15.28:0/4232197079 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f96119b10 con 0x555f961c0970 2024-04-21T08:54:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.930+0000 7f45b1951640 1 --2- 172.21.15.28:0/4232197079 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555f961c0970 0x555f961c2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.932+0000 7f4595ffb640 10 client.26080 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:23.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.932+0000 7f45b1951640 1 --2- 172.21.15.28:0/4232197079 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555f961c0970 0x555f961c2d60 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10d76b769a4306ae server_cookie=1b531fa06f36a393 in_seq=0 out_seq=0 2024-04-21T08:54:23.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.934+0000 7f4595ffb640 1 -- 172.21.15.28:0/4232197079 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555f96119b10 con 0x555f961c0970 2024-04-21T08:54:23.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.934+0000 7f4595ffb640 10 client.26080 handle_command_reply: tid=0 2024-04-21T08:54:23.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.934+0000 7f45b6fad740 1 client.26080 shutdown 2024-04-21T08:54:23.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.934+0000 7f45b6fad740 1 -- 172.21.15.28:0/4232197079 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961a7b90 msgr2=0x555f961ae060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:23.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.934+0000 7f45b6fad740 1 --2- 172.21.15.28:0/4232197079 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961a7b90 0x555f961ae060 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f45a80045f0 tx=0x7f45a802cb20 comp rx=0 tx=0).stop 2024-04-21T08:54:23.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.934+0000 7f45b6fad740 1 -- 172.21.15.28:0/4232197079 shutdown_connections 2024-04-21T08:54:23.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.934+0000 7f45b6fad740 1 --2- 172.21.15.28:0/4232197079 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555f961c0970 0x555f961c2d60 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.934+0000 7f45b6fad740 1 --2- 172.21.15.28:0/4232197079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961b1090 0x555f961aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.934+0000 7f45b6fad740 1 --2- 172.21.15.28:0/4232197079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a84b0 0x555f961ae5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.934+0000 7f45b6fad740 1 --2- 172.21.15.28:0/4232197079 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961a7b90 0x555f961ae060 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.934+0000 7f45b6fad740 1 -- 172.21.15.28:0/4232197079 >> 172.21.15.28:0/4232197079 conn(0x555f961a22c0 msgr2=0x555f961a4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:23.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.934+0000 7f4595ffb640 0 client.26080 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.934+0000 7f45b6fad740 1 -- 172.21.15.28:0/4232197079 shutdown_connections 2024-04-21T08:54:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.935+0000 7f45b6fad740 1 -- 172.21.15.28:0/4232197079 wait complete. 2024-04-21T08:54:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.936+0000 7f45b6fad740 20 client.26080 trim_cache size 0 max 16384 2024-04-21T08:54:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.967+0000 7f45b6fad740 1 Processor -- start 2024-04-21T08:54:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.968+0000 7f45b6fad740 1 -- start start 2024-04-21T08:54:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.968+0000 7f45b6fad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a7b90 0x555f961b0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.968+0000 7f45b6fad740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961a84b0 0x555f961ae900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.968+0000 7f45b6fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961b1090 0x555f961c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.968+0000 7f45b6fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4590010080 con 0x555f961b1090 2024-04-21T08:54:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.968+0000 7f45b6fad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f459000ded0 con 0x555f961a7b90 2024-04-21T08:54:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.968+0000 7f45b6fad740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f459000f9b0 con 0x555f961a84b0 2024-04-21T08:54:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.969+0000 7f45b2953640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961b1090 0x555f961c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.969+0000 7f45b2953640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961b1090 0x555f961c83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51174/0 (socket says 172.21.15.28:51174) 2024-04-21T08:54:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.969+0000 7f45b2953640 1 -- 172.21.15.28:0/2585264365 learned_addr learned my addr 172.21.15.28:0/2585264365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.969+0000 7f45b2152640 1 --2- 172.21.15.28:0/2585264365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a7b90 0x555f961b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.969+0000 7f45b1951640 1 --2- 172.21.15.28:0/2585264365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961a84b0 0x555f961ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.969+0000 7f45b2953640 1 -- 172.21.15.28:0/2585264365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961a84b0 msgr2=0x555f961ae900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.969+0000 7f45b2953640 1 --2- 172.21.15.28:0/2585264365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961a84b0 0x555f961ae900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.969+0000 7f45b2953640 1 -- 172.21.15.28:0/2585264365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a7b90 msgr2=0x555f961b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.969+0000 7f45b2953640 1 --2- 172.21.15.28:0/2585264365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a7b90 0x555f961b0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.969+0000 7f45b2953640 1 -- 172.21.15.28:0/2585264365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f961ac5b0 con 0x555f961b1090 2024-04-21T08:54:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.970+0000 7f45b2953640 1 --2- 172.21.15.28:0/2585264365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961b1090 0x555f961c83c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f45ac067e60 tx=0x7f45ac07b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db327b6031bd7cba server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.970+0000 7f4597fff640 1 -- 172.21.15.28:0/2585264365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac076db0 con 0x555f961b1090 2024-04-21T08:54:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.970+0000 7f4597fff640 1 -- 172.21.15.28:0/2585264365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45ac088030 con 0x555f961b1090 2024-04-21T08:54:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.970+0000 7f4597fff640 1 -- 172.21.15.28:0/2585264365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac084bc0 con 0x555f961b1090 2024-04-21T08:54:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.971+0000 7f45b6fad740 1 -- 172.21.15.28:0/2585264365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961b1090 msgr2=0x555f961c83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.971+0000 7f45b6fad740 1 --2- 172.21.15.28:0/2585264365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961b1090 0x555f961c83c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f45ac067e60 tx=0x7f45ac07b000 comp rx=0 tx=0).stop 2024-04-21T08:54:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.971+0000 7f45b6fad740 1 -- 172.21.15.28:0/2585264365 shutdown_connections 2024-04-21T08:54:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.971+0000 7f45b6fad740 1 --2- 172.21.15.28:0/2585264365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961b1090 0x555f961c83c0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.971+0000 7f45b6fad740 1 --2- 172.21.15.28:0/2585264365 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961a84b0 0x555f961ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.971+0000 7f45b6fad740 1 --2- 172.21.15.28:0/2585264365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a7b90 0x555f961b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.971+0000 7f45b6fad740 1 -- 172.21.15.28:0/2585264365 >> 172.21.15.28:0/2585264365 conn(0x555f961a22c0 msgr2=0x555f961a34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.971+0000 7f45b6fad740 1 -- 172.21.15.28:0/2585264365 shutdown_connections 2024-04-21T08:54:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.971+0000 7f45b6fad740 1 -- 172.21.15.28:0/2585264365 wait complete. 2024-04-21T08:54:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.972+0000 7f45b6fad740 1 -- start start 2024-04-21T08:54:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.972+0000 7f45b6fad740 1 Processor -- start 2024-04-21T08:54:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.972+0000 7f45b6fad740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a7b90 0x555f961b8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.972+0000 7f45b6fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961a84b0 0x555f961b9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.972+0000 7f45b6fad740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961b1090 0x555f9621fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.972+0000 7f45b6fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f459000ec10 con 0x555f961a84b0 2024-04-21T08:54:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.972+0000 7f45b6fad740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f459000e540 con 0x555f961a7b90 2024-04-21T08:54:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.972+0000 7f45b6fad740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f459000f2e0 con 0x555f961b1090 2024-04-21T08:54:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.972+0000 7f45b6fad740 10 client.0 _ll_register_callbacks cb 0x7f459c00cee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.973+0000 7f45b2953640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961b1090 0x555f9621fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.973+0000 7f45b2953640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961b1090 0x555f9621fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:44742/0 (socket says 172.21.15.28:44742) 2024-04-21T08:54:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.973+0000 7f45b2953640 1 -- 172.21.15.28:0/2450457418 learned_addr learned my addr 172.21.15.28:0/2450457418 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.973+0000 7f45b2152640 1 --2- 172.21.15.28:0/2450457418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a7b90 0x555f961b8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.973+0000 7f45b1951640 1 --2- 172.21.15.28:0/2450457418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961a84b0 0x555f961b9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.973+0000 7f45b2953640 1 -- 172.21.15.28:0/2450457418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a7b90 msgr2=0x555f961b8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.973+0000 7f45b2953640 1 --2- 172.21.15.28:0/2450457418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a7b90 0x555f961b8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.973+0000 7f45b2953640 1 -- 172.21.15.28:0/2450457418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961a84b0 msgr2=0x555f961b9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.973+0000 7f45b2953640 1 --2- 172.21.15.28:0/2450457418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961a84b0 0x555f961b9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.973+0000 7f45b2953640 1 -- 172.21.15.28:0/2450457418 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f962202f0 con 0x555f961b1090 2024-04-21T08:54:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.974+0000 7f45b2953640 1 --2- 172.21.15.28:0/2450457418 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961b1090 0x555f9621fcb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f45ac065470 tx=0x7f45ac076db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eae49e49631920d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.974+0000 7f4595ffb640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:54:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.974+0000 7f4595ffb640 1 -- 172.21.15.28:0/2450457418 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac08d620 con 0x555f961b1090 2024-04-21T08:54:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.974+0000 7f4595ffb640 1 -- 172.21.15.28:0/2450457418 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f45ac08d7c0 con 0x555f961b1090 2024-04-21T08:54:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.974+0000 7f45b6fad740 1 -- 172.21.15.28:0/2450457418 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45ac088030 con 0x555f961b1090 2024-04-21T08:54:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.975+0000 7f4595ffb640 1 -- 172.21.15.28:0/2450457418 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f45ac09f050 con 0x555f961b1090 2024-04-21T08:54:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.975+0000 7f45b6fad740 10 client.26083 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.975+0000 7f45b6fad740 1 -- 172.21.15.28:0/2450457418 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555f96238520 con 0x555f961b1090 2024-04-21T08:54:23.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.976+0000 7f4595ffb640 1 -- 172.21.15.28:0/2450457418 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45ac088030 con 0x555f961b1090 2024-04-21T08:54:23.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.976+0000 7f4595ffb640 1 -- 172.21.15.28:0/2450457418 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f45ac084a50 con 0x555f961b1090 2024-04-21T08:54:23.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.977+0000 7f45b6fad740 10 client.26083 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:23.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.977+0000 7f45b6fad740 10 client.26083 resolve_mds: validated gid 24488 aka daemon mds.b 2024-04-21T08:54:23.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.977+0000 7f45b6fad740 20 client.26083 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:23.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.977+0000 7f45b6fad740 1 --2- 172.21.15.28:0/2450457418 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555f96238c80 0x555f9623b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:23.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.977+0000 7f45b6fad740 4 client.26083 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 30.0, "expiration": 84.5, "await": true}] 2024-04-21T08:54:23.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.977+0000 7f45b6fad740 1 -- 172.21.15.28:0/2450457418 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 30.0, "expiration": 84.5, "await": true}) -- 0x555f96113980 con 0x555f96238c80 2024-04-21T08:54:23.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.977+0000 7f45b2152640 1 --2- 172.21.15.28:0/2450457418 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555f96238c80 0x555f9623b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:23.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.978+0000 7f4595ffb640 10 client.26083 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:23.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:23.978+0000 7f45b2152640 1 --2- 172.21.15.28:0/2450457418 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555f96238c80 0x555f9623b0a0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38500915bbc99f6c server_cookie=537288ec3b2df60f in_seq=0 out_seq=0 2024-04-21T08:54:24.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:24 smithi152 ceph-mon[31401]: pgmap v99: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 4.1 KiB/s wr, 32 op/s 2024-04-21T08:54:24.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:24 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3750901636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:24.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:24 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:54:24.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:24 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:54:24.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:24 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:54:24.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:24 smithi028 ceph-mon[28003]: pgmap v99: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 4.1 KiB/s wr, 32 op/s 2024-04-21T08:54:24.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:24 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3750901636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:24.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:24 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:54:24.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:24 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:54:24.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:24 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:54:24.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:24 smithi171 ceph-mon[31358]: pgmap v99: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 4.1 KiB/s wr, 32 op/s 2024-04-21T08:54:24.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:24 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3750901636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:24.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:24 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:54:24.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:24 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:54:24.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:24 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:54:26.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:26 smithi152 ceph-mon[31401]: pgmap v100: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.6 MiB/s wr, 298 op/s 2024-04-21T08:54:26.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:26 smithi028 ceph-mon[28003]: pgmap v100: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.6 MiB/s wr, 298 op/s 2024-04-21T08:54:26.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:26 smithi171 ceph-mon[31358]: pgmap v100: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.6 MiB/s wr, 298 op/s 2024-04-21T08:54:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.992+0000 7f4595ffb640 1 -- 172.21.15.28:0/2450457418 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x555f96113980 con 0x555f96238c80 2024-04-21T08:54:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.992+0000 7f4595ffb640 10 client.26083 handle_command_reply: tid=0 2024-04-21T08:54:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.992+0000 7f45b6fad740 1 client.26083 shutdown 2024-04-21T08:54:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.992+0000 7f45b6fad740 1 -- 172.21.15.28:0/2450457418 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961b1090 msgr2=0x555f9621fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.992+0000 7f45b6fad740 1 --2- 172.21.15.28:0/2450457418 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961b1090 0x555f9621fcb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f45ac065470 tx=0x7f45ac076db0 comp rx=0 tx=0).stop 2024-04-21T08:54:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.992+0000 7f45b6fad740 1 -- 172.21.15.28:0/2450457418 shutdown_connections 2024-04-21T08:54:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.992+0000 7f45b6fad740 1 --2- 172.21.15.28:0/2450457418 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555f96238c80 0x555f9623b0a0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.992+0000 7f45b6fad740 1 --2- 172.21.15.28:0/2450457418 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555f961b1090 0x555f9621fcb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.992+0000 7f45b6fad740 1 --2- 172.21.15.28:0/2450457418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f961a84b0 0x555f961b9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.992+0000 7f45b6fad740 1 --2- 172.21.15.28:0/2450457418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555f961a7b90 0x555f961b8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.992+0000 7f45b6fad740 1 -- 172.21.15.28:0/2450457418 >> 172.21.15.28:0/2450457418 conn(0x555f961a22c0 msgr2=0x555f961a3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.992+0000 7f4595ffb640 0 client.26083 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.992+0000 7f45b6fad740 1 -- 172.21.15.28:0/2450457418 shutdown_connections 2024-04-21T08:54:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.992+0000 7f45b6fad740 1 -- 172.21.15.28:0/2450457418 wait complete. 2024-04-21T08:54:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.992+0000 7f45b6fad740 20 client.26083 trim_cache size 0 max 16384 2024-04-21T08:54:27.995 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:54:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 26, 2024-04-21T08:54:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24488, 2024-04-21T08:54:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 6, 2024-04-21T08:54:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-04-21T08:54:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "7459dea4": { 2024-04-21T08:54:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 6, 2024-04-21T08:54:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-04-21T08:54:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-21T08:54:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-04-21T08:54:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-21T08:54:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T08:54:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 30.0, 2024-04-21T08:54:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 84.5, 2024-04-21T08:54:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-04-21T08:54:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-04-21T08:54:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-04-21T08:54:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-21T08:54:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "QUIESCED", 2024-04-21T08:54:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-21T08:54:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:27.996 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:54:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.993+0000 7f45b3154640 1 -- 172.21.15.28:0/290197714 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f45980a3e60 msgr2=0x7f45980a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.993+0000 7f45b3154640 1 --2- 172.21.15.28:0/290197714 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f45980a3e60 0x7f45980a6320 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f45ac050760 tx=0x7f45ac067800 comp rx=0 tx=0).stop 2024-04-21T08:54:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.993+0000 7f45b3154640 1 -- 172.21.15.28:0/290197714 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45a40ae260 msgr2=0x7f45a401b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.993+0000 7f45b3154640 1 --2- 172.21.15.28:0/290197714 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45a40ae260 0x7f45a401b2a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f459c00b470 tx=0x7f459c00b930 comp rx=0 tx=0).stop 2024-04-21T08:54:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.993+0000 7f45b3154640 1 -- 172.21.15.28:0/290197714 shutdown_connections 2024-04-21T08:54:27.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.993+0000 7f45b3154640 1 --2- 172.21.15.28:0/290197714 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f45980a3e60 0x7f45980a6320 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:27.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.993+0000 7f45b3154640 1 --2- 172.21.15.28:0/290197714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45a401b7e0 0x7f45a414fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:27.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.993+0000 7f45b3154640 1 --2- 172.21.15.28:0/290197714 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45a40ae260 0x7f45a401b2a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:27.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.993+0000 7f45b3154640 1 --2- 172.21.15.28:0/290197714 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45a40a9190 0x7f45a401ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:27.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.993+0000 7f45b3154640 1 -- 172.21.15.28:0/290197714 >> 172.21.15.28:0/290197714 conn(0x7f45a40a49d0 msgr2=0x7f45a40a5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:27.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.994+0000 7f45b3154640 1 -- 172.21.15.28:0/290197714 shutdown_connections 2024-04-21T08:54:27.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:27.994+0000 7f45b3154640 1 -- 172.21.15.28:0/290197714 wait complete. 2024-04-21T08:54:28.013 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: 7459dea4, quiesce duration: 24.5 2024-04-21T08:54:28.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:28 smithi028 ceph-mon[28003]: pgmap v101: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.6 MiB/s wr, 297 op/s 2024-04-21T08:54:28.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:28 smithi152 ceph-mon[31401]: pgmap v101: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.6 MiB/s wr, 297 op/s 2024-04-21T08:54:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:28 smithi171 ceph-mon[31358]: pgmap v101: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.6 MiB/s wr, 297 op/s 2024-04-21T08:54:30.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:30 smithi152 ceph-mon[31401]: pgmap v102: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 15 MiB/s wr, 2.61k op/s 2024-04-21T08:54:30.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:30 smithi028 ceph-mon[28003]: pgmap v102: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 15 MiB/s wr, 2.61k op/s 2024-04-21T08:54:30.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:30 smithi171 ceph-mon[31358]: pgmap v102: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 15 MiB/s wr, 2.61k op/s 2024-04-21T08:54:32.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:32 smithi152 ceph-mon[31401]: pgmap v103: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 21 MiB/s wr, 3.67k op/s 2024-04-21T08:54:32.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:32 smithi028 ceph-mon[28003]: pgmap v103: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 21 MiB/s wr, 3.67k op/s 2024-04-21T08:54:32.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:32 smithi171 ceph-mon[31358]: pgmap v103: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 21 MiB/s wr, 3.67k op/s 2024-04-21T08:54:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.395+0000 7fd10329a640 1 -- 172.21.15.28:0/3465339857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0fc105e20 msgr2=0x7fd0fc106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.395+0000 7fd10329a640 1 --2- 172.21.15.28:0/3465339857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0fc105e20 0x7fd0fc106200 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8009900 tx=0x7fd0f802bc80 comp rx=0 tx=0).stop 2024-04-21T08:54:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.395+0000 7fd10329a640 1 -- 172.21.15.28:0/3465339857 shutdown_connections 2024-04-21T08:54:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.395+0000 7fd10329a640 1 --2- 172.21.15.28:0/3465339857 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0fc101b40 0x7fd0fc10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.395+0000 7fd10329a640 1 --2- 172.21.15.28:0/3465339857 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0fc1011a0 0x7fd0fc101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.395+0000 7fd10329a640 1 --2- 172.21.15.28:0/3465339857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0fc105e20 0x7fd0fc106200 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.395+0000 7fd10329a640 1 -- 172.21.15.28:0/3465339857 >> 172.21.15.28:0/3465339857 conn(0x7fd0fc076c50 msgr2=0x7fd0fc077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.395+0000 7fd10329a640 1 -- 172.21.15.28:0/3465339857 shutdown_connections 2024-04-21T08:54:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.395+0000 7fd10329a640 1 -- 172.21.15.28:0/3465339857 wait complete. 2024-04-21T08:54:33.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.395+0000 7fd10329a640 1 Processor -- start 2024-04-21T08:54:33.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd10329a640 1 -- start start 2024-04-21T08:54:33.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd10329a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0fc1011a0 0x7fd0fc10e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd10329a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0fc101b40 0x7fd0fc10ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd10329a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0fc105e20 0x7fd0fc10f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd10329a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc1122f0 con 0x7fd0fc1011a0 2024-04-21T08:54:33.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd10329a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc112170 con 0x7fd0fc105e20 2024-04-21T08:54:33.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd10329a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc112470 con 0x7fd0fc101b40 2024-04-21T08:54:33.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd101036640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0fc1011a0 0x7fd0fc10e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd101036640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0fc1011a0 0x7fd0fc10e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51208/0 (socket says 172.21.15.28:51208) 2024-04-21T08:54:33.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd101036640 1 -- 172.21.15.28:0/1829906953 learned_addr learned my addr 172.21.15.28:0/1829906953 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:33.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd101837640 1 --2- 172.21.15.28:0/1829906953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0fc105e20 0x7fd0fc10f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd100835640 1 --2- 172.21.15.28:0/1829906953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0fc101b40 0x7fd0fc10ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd101036640 1 -- 172.21.15.28:0/1829906953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0fc101b40 msgr2=0x7fd0fc10ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd101036640 1 --2- 172.21.15.28:0/1829906953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0fc101b40 0x7fd0fc10ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd101036640 1 -- 172.21.15.28:0/1829906953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0fc105e20 msgr2=0x7fd0fc10f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd101036640 1 --2- 172.21.15.28:0/1829906953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0fc105e20 0x7fd0fc10f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd101036640 1 -- 172.21.15.28:0/1829906953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0fc1a6bd0 con 0x7fd0fc1011a0 2024-04-21T08:54:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.396+0000 7fd101837640 1 --2- 172.21.15.28:0/1829906953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0fc105e20 0x7fd0fc10f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.397+0000 7fd101036640 1 --2- 172.21.15.28:0/1829906953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0fc1011a0 0x7fd0fc10e7d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8006010 tx=0x7fd0f8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.397+0000 7fd0f27fc640 1 -- 172.21.15.28:0/1829906953 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8043020 con 0x7fd0fc1011a0 2024-04-21T08:54:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.397+0000 7fd0f27fc640 1 -- 172.21.15.28:0/1829906953 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0f8004440 con 0x7fd0fc1011a0 2024-04-21T08:54:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.397+0000 7fd10329a640 1 -- 172.21.15.28:0/1829906953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0fc1a6e60 con 0x7fd0fc1011a0 2024-04-21T08:54:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.397+0000 7fd0f27fc640 1 -- 172.21.15.28:0/1829906953 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8003b50 con 0x7fd0fc1011a0 2024-04-21T08:54:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.397+0000 7fd10329a640 1 -- 172.21.15.28:0/1829906953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0fc1a7320 con 0x7fd0fc1011a0 2024-04-21T08:54:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.399+0000 7fd0f27fc640 1 -- 172.21.15.28:0/1829906953 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd0f8046020 con 0x7fd0fc1011a0 2024-04-21T08:54:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd105f6b740 1 Processor -- start 2024-04-21T08:54:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd105f6b740 1 -- start start 2024-04-21T08:54:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd0f27fc640 1 --2- 172.21.15.28:0/1829906953 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd0e80a3dc0 0x7fd0e80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd105f6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4698d0 0x561f0e468620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd105f6b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e468b60 0x561f0e468fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd105f6b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 0x561f0e4744d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd105f6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561f0e3cd8d0 con 0x561f0e4698d0 2024-04-21T08:54:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd105f6b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561f0e36ae80 con 0x561f0e468b60 2024-04-21T08:54:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd105f6b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x561f0e3c9890 con 0x561f0e4720e0 2024-04-21T08:54:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd101837640 1 --2- 172.21.15.28:0/1829906953 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd0e80a3dc0 0x7fd0e80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd101036640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4698d0 0x561f0e468620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd0f27fc640 1 -- 172.21.15.28:0/1829906953 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fd0f80f2800 con 0x7fd0fc1011a0 2024-04-21T08:54:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd101036640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4698d0 0x561f0e468620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51214/0 (socket says 172.21.15.28:51214) 2024-04-21T08:54:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd101036640 1 -- 172.21.15.28:0/2772071297 learned_addr learned my addr 172.21.15.28:0/2772071297 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd101837640 1 --2- 172.21.15.28:0/2772071297 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 0x561f0e4744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd100835640 1 --2- 172.21.15.28:0/2772071297 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e468b60 0x561f0e468fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.400+0000 7fd101036640 1 -- 172.21.15.28:0/2772071297 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 msgr2=0x561f0e4744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.401+0000 7fd101036640 1 --2- 172.21.15.28:0/2772071297 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 0x561f0e4744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.401+0000 7fd101036640 1 -- 172.21.15.28:0/2772071297 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e468b60 msgr2=0x561f0e468fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.401+0000 7fd101036640 1 --2- 172.21.15.28:0/2772071297 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e468b60 0x561f0e468fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.401+0000 7fd101036640 1 -- 172.21.15.28:0/2772071297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f0e38c320 con 0x561f0e4698d0 2024-04-21T08:54:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.401+0000 7fd101036640 1 --2- 172.21.15.28:0/2772071297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4698d0 0x561f0e468620 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd0f80efd20 tx=0x7fd0f8003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7846e18f0a1fb5f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.401+0000 7fd101837640 1 --2- 172.21.15.28:0/1829906953 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd0e80a3dc0 0x7fd0e80a6280 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd0f40126c0 tx=0x7fd0f4012ac0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.401+0000 7fd0e77fe640 1 -- 172.21.15.28:0/2772071297 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f80bde70 con 0x561f0e4698d0 2024-04-21T08:54:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.401+0000 7fd0e77fe640 1 -- 172.21.15.28:0/2772071297 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0f80dee50 con 0x561f0e4698d0 2024-04-21T08:54:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.401+0000 7fd0e77fe640 1 -- 172.21.15.28:0/2772071297 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8046310 con 0x561f0e4698d0 2024-04-21T08:54:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.402+0000 7fd105f6b740 1 -- 172.21.15.28:0/2772071297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4698d0 msgr2=0x561f0e468620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.402+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2772071297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4698d0 0x561f0e468620 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd0f80efd20 tx=0x7fd0f8003680 comp rx=0 tx=0).stop 2024-04-21T08:54:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.402+0000 7fd105f6b740 1 -- 172.21.15.28:0/2772071297 shutdown_connections 2024-04-21T08:54:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.402+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2772071297 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 0x561f0e4744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.402+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2772071297 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e468b60 0x561f0e468fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.402+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2772071297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4698d0 0x561f0e468620 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.402+0000 7fd105f6b740 1 -- 172.21.15.28:0/2772071297 >> 172.21.15.28:0/2772071297 conn(0x561f0e42fc90 msgr2=0x561f0e463850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.402+0000 7fd105f6b740 1 -- 172.21.15.28:0/2772071297 shutdown_connections 2024-04-21T08:54:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.402+0000 7fd105f6b740 1 -- 172.21.15.28:0/2772071297 wait complete. 2024-04-21T08:54:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.402+0000 7fd105f6b740 1 -- start start 2024-04-21T08:54:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.402+0000 7fd105f6b740 1 Processor -- start 2024-04-21T08:54:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd105f6b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e468b60 0x561f0e471ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd105f6b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4698d0 0x561f0e46ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd105f6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4720e0 0x561f0e46f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd105f6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561f0e3cd8d0 con 0x561f0e4720e0 2024-04-21T08:54:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd105f6b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561f0e355eb0 con 0x561f0e468b60 2024-04-21T08:54:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd105f6b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x561f0e474bd0 con 0x561f0e4698d0 2024-04-21T08:54:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd105f6b740 10 client.0 _ll_register_callbacks cb 0x561f0e382b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd101837640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4720e0 0x561f0e46f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd101036640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e468b60 0x561f0e471ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd101837640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4720e0 0x561f0e46f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51226/0 (socket says 172.21.15.28:51226) 2024-04-21T08:54:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd101837640 1 -- 172.21.15.28:0/2637220279 learned_addr learned my addr 172.21.15.28:0/2637220279 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd100835640 1 --2- 172.21.15.28:0/2637220279 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4698d0 0x561f0e46ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd101837640 1 -- 172.21.15.28:0/2637220279 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4698d0 msgr2=0x561f0e46ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd101837640 1 --2- 172.21.15.28:0/2637220279 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4698d0 0x561f0e46ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd101837640 1 -- 172.21.15.28:0/2637220279 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e468b60 msgr2=0x561f0e471ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd101837640 1 --2- 172.21.15.28:0/2637220279 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e468b60 0x561f0e471ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd101837640 1 -- 172.21.15.28:0/2637220279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f0e0e0210 con 0x561f0e4720e0 2024-04-21T08:54:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd101837640 1 --2- 172.21.15.28:0/2637220279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4720e0 0x561f0e46f460 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4010a90 tx=0x7fd0f4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e839a1009fbb4a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.403+0000 7fd0e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.404+0000 7fd0e4ff9640 1 -- 172.21.15.28:0/2637220279 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4024020 con 0x561f0e4720e0 2024-04-21T08:54:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.404+0000 7fd0e4ff9640 1 -- 172.21.15.28:0/2637220279 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fd0f4011d30 con 0x561f0e4720e0 2024-04-21T08:54:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.404+0000 7fd105f6b740 1 -- 172.21.15.28:0/2637220279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0f80dee50 con 0x561f0e4720e0 2024-04-21T08:54:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.404+0000 7fd0e4ff9640 1 -- 172.21.15.28:0/2637220279 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd0f402f050 con 0x561f0e4720e0 2024-04-21T08:54:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.404+0000 7fd105f6b740 10 client.16785 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.404+0000 7fd105f6b740 1 -- 172.21.15.28:0/2637220279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561f0e1ac620 con 0x561f0e4720e0 2024-04-21T08:54:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.404+0000 7fd0e4ff9640 1 -- 172.21.15.28:0/2637220279 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0f80dee50 con 0x561f0e4720e0 2024-04-21T08:54:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.404+0000 7fd0e4ff9640 1 -- 172.21.15.28:0/2637220279 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd0f401e500 con 0x561f0e4720e0 2024-04-21T08:54:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.404+0000 7fd105f6b740 10 client.16785 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.404+0000 7fd105f6b740 10 client.16785 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.404+0000 7fd105f6b740 20 client.16785 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.404+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2637220279 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561f0e481e40 0x561f0e4842e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.404+0000 7fd105f6b740 4 client.16785 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:33.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.404+0000 7fd105f6b740 1 -- 172.21.15.28:0/2637220279 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f0e3e1880 con 0x561f0e481e40 2024-04-21T08:54:33.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.405+0000 7fd101036640 1 --2- 172.21.15.28:0/2637220279 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561f0e481e40 0x561f0e4842e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.405+0000 7fd0e4ff9640 10 client.16785 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:33.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.405+0000 7fd101036640 1 --2- 172.21.15.28:0/2637220279 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561f0e481e40 0x561f0e4842e0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed9f3aba2b17eda4 server_cookie=c1a1754697b250fc in_seq=0 out_seq=0 2024-04-21T08:54:33.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.406+0000 7fd0e4ff9640 1 -- 172.21.15.28:0/2637220279 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561f0e3e1880 con 0x561f0e481e40 2024-04-21T08:54:33.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.406+0000 7fd0e4ff9640 10 client.16785 handle_command_reply: tid=0 2024-04-21T08:54:33.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.406+0000 7fd105f6b740 1 client.16785 shutdown 2024-04-21T08:54:33.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.406+0000 7fd105f6b740 1 -- 172.21.15.28:0/2637220279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4720e0 msgr2=0x561f0e46f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:33.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.406+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2637220279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4720e0 0x561f0e46f460 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4010a90 tx=0x7fd0f4004ab0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.406+0000 7fd105f6b740 1 -- 172.21.15.28:0/2637220279 shutdown_connections 2024-04-21T08:54:33.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.406+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2637220279 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561f0e481e40 0x561f0e4842e0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.406+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2637220279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4720e0 0x561f0e46f460 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.406+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2637220279 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4698d0 0x561f0e46ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.406+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2637220279 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e468b60 0x561f0e471ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.406+0000 7fd0e4ff9640 0 client.16785 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:33.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.406+0000 7fd105f6b740 1 -- 172.21.15.28:0/2637220279 >> 172.21.15.28:0/2637220279 conn(0x561f0e42fc90 msgr2=0x561f0e4655a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:33.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.406+0000 7fd105f6b740 1 -- 172.21.15.28:0/2637220279 shutdown_connections 2024-04-21T08:54:33.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.406+0000 7fd105f6b740 1 -- 172.21.15.28:0/2637220279 wait complete. 2024-04-21T08:54:33.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.406+0000 7fd105f6b740 20 client.16785 trim_cache size 0 max 16384 2024-04-21T08:54:33.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd105f6b740 1 Processor -- start 2024-04-21T08:54:33.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd105f6b740 1 -- start start 2024-04-21T08:54:33.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd105f6b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e468b60 0x561f0e46dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd105f6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4698d0 0x561f0e471580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd105f6b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 0x561f0e471ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd105f6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc1b3c60 con 0x561f0e4698d0 2024-04-21T08:54:33.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd105f6b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc1b1530 con 0x561f0e468b60 2024-04-21T08:54:33.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd105f6b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc1af380 con 0x561f0e4720e0 2024-04-21T08:54:33.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd100835640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4698d0 0x561f0e471580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd100835640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4698d0 0x561f0e471580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51242/0 (socket says 172.21.15.28:51242) 2024-04-21T08:54:33.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd100835640 1 -- 172.21.15.28:0/2366225863 learned_addr learned my addr 172.21.15.28:0/2366225863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:33.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd101036640 1 --2- 172.21.15.28:0/2366225863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e468b60 0x561f0e46dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd101837640 1 --2- 172.21.15.28:0/2366225863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 0x561f0e471ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd100835640 1 -- 172.21.15.28:0/2366225863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 msgr2=0x561f0e471ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:33.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd100835640 1 --2- 172.21.15.28:0/2366225863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 0x561f0e471ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd100835640 1 -- 172.21.15.28:0/2366225863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e468b60 msgr2=0x561f0e46dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd100835640 1 --2- 172.21.15.28:0/2366225863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e468b60 0x561f0e46dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.422+0000 7fd100835640 1 -- 172.21.15.28:0/2366225863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f0e467e30 con 0x561f0e4698d0 2024-04-21T08:54:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.423+0000 7fd100835640 1 --2- 172.21.15.28:0/2366225863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4698d0 0x561f0e471580 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00bcb0 tx=0x7fd0ec012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2baf77ab6b2f39d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.423+0000 7fd0e6ffd640 1 -- 172.21.15.28:0/2366225863 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec00bda0 con 0x561f0e4698d0 2024-04-21T08:54:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.423+0000 7fd0e6ffd640 1 -- 172.21.15.28:0/2366225863 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0ec0035f0 con 0x561f0e4698d0 2024-04-21T08:54:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.423+0000 7fd0e6ffd640 1 -- 172.21.15.28:0/2366225863 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec01c680 con 0x561f0e4698d0 2024-04-21T08:54:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.423+0000 7fd105f6b740 1 -- 172.21.15.28:0/2366225863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4698d0 msgr2=0x561f0e471580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.423+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2366225863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4698d0 0x561f0e471580 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00bcb0 tx=0x7fd0ec012820 comp rx=0 tx=0).stop 2024-04-21T08:54:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.423+0000 7fd105f6b740 1 -- 172.21.15.28:0/2366225863 shutdown_connections 2024-04-21T08:54:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.423+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2366225863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 0x561f0e471ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.423+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2366225863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e4698d0 0x561f0e471580 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.423+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2366225863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e468b60 0x561f0e46dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.423+0000 7fd105f6b740 1 -- 172.21.15.28:0/2366225863 >> 172.21.15.28:0/2366225863 conn(0x561f0e42fc90 msgr2=0x561f0e46d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:33.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.423+0000 7fd105f6b740 1 -- 172.21.15.28:0/2366225863 shutdown_connections 2024-04-21T08:54:33.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.423+0000 7fd105f6b740 1 -- 172.21.15.28:0/2366225863 wait complete. 2024-04-21T08:54:33.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.424+0000 7fd105f6b740 1 -- start start 2024-04-21T08:54:33.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.424+0000 7fd105f6b740 1 Processor -- start 2024-04-21T08:54:33.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.424+0000 7fd105f6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e468b60 0x561f0e47a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.424+0000 7fd105f6b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e4698d0 0x561f0e475480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.424+0000 7fd105f6b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 0x561f0e4759c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.424+0000 7fd105f6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc1af9f0 con 0x561f0e468b60 2024-04-21T08:54:33.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.424+0000 7fd105f6b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc1b0e60 con 0x561f0e4698d0 2024-04-21T08:54:33.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.424+0000 7fd105f6b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc1b00c0 con 0x561f0e4720e0 2024-04-21T08:54:33.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.424+0000 7fd105f6b740 10 client.0 _ll_register_callbacks cb 0x7fd0fc07a280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:33.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.424+0000 7fd101036640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e468b60 0x561f0e47a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.424+0000 7fd101036640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e468b60 0x561f0e47a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51246/0 (socket says 172.21.15.28:51246) 2024-04-21T08:54:33.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.424+0000 7fd101036640 1 -- 172.21.15.28:0/2190134919 learned_addr learned my addr 172.21.15.28:0/2190134919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:33.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd101837640 1 --2- 172.21.15.28:0/2190134919 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 0x561f0e4759c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd100835640 1 --2- 172.21.15.28:0/2190134919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e4698d0 0x561f0e475480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd101036640 1 -- 172.21.15.28:0/2190134919 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 msgr2=0x561f0e4759c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd101036640 1 --2- 172.21.15.28:0/2190134919 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 0x561f0e4759c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd101036640 1 -- 172.21.15.28:0/2190134919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e4698d0 msgr2=0x561f0e475480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd101036640 1 --2- 172.21.15.28:0/2190134919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e4698d0 0x561f0e475480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd101036640 1 -- 172.21.15.28:0/2190134919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f0e476100 con 0x561f0e468b60 2024-04-21T08:54:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd100835640 1 --2- 172.21.15.28:0/2190134919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e4698d0 0x561f0e475480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd101837640 1 --2- 172.21.15.28:0/2190134919 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 0x561f0e4759c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd101036640 1 --2- 172.21.15.28:0/2190134919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e468b60 0x561f0e47a110 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fd0f80efd70 tx=0x7fd0f802bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3bbc8064ba8ddf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd0e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd0e4ff9640 1 -- 172.21.15.28:0/2190134919 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f80c28c0 con 0x561f0e468b60 2024-04-21T08:54:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd0e4ff9640 1 -- 172.21.15.28:0/2190134919 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fd0f8003cf0 con 0x561f0e468b60 2024-04-21T08:54:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd105f6b740 1 -- 172.21.15.28:0/2190134919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0ec0035f0 con 0x561f0e468b60 2024-04-21T08:54:33.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd0e4ff9640 1 -- 172.21.15.28:0/2190134919 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd0f80c7180 con 0x561f0e468b60 2024-04-21T08:54:33.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd105f6b740 10 client.16791 fetch_fsmap learned FSMap version 26 2024-04-21T08:54:33.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd105f6b740 1 -- 172.21.15.28:0/2190134919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561f0e4e8360 con 0x561f0e468b60 2024-04-21T08:54:33.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.425+0000 7fd0e4ff9640 1 -- 172.21.15.28:0/2190134919 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0ec0035f0 con 0x561f0e468b60 2024-04-21T08:54:33.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.426+0000 7fd0e4ff9640 1 -- 172.21.15.28:0/2190134919 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd0f80efda0 con 0x561f0e468b60 2024-04-21T08:54:33.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.426+0000 7fd105f6b740 10 client.16791 fetch_fsmap finished waiting for FSMap version 26 2024-04-21T08:54:33.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.426+0000 7fd105f6b740 10 client.16791 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:33.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.426+0000 7fd105f6b740 20 client.16791 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:33.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.426+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2190134919 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561f0e4e88b0 0x561f0e4eacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:33.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.426+0000 7fd105f6b740 4 client.16791 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:54:33.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.426+0000 7fd105f6b740 1 -- 172.21.15.28:0/2190134919 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f0e36d3d0 con 0x561f0e4e88b0 2024-04-21T08:54:33.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.426+0000 7fd100835640 1 --2- 172.21.15.28:0/2190134919 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561f0e4e88b0 0x561f0e4eacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:33.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.426+0000 7fd0e4ff9640 10 client.16791 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:33.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.426+0000 7fd100835640 1 --2- 172.21.15.28:0/2190134919 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561f0e4e88b0 0x561f0e4eacd0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66975e979d2a8fc5 server_cookie=c7e05291911bb012 in_seq=0 out_seq=0 2024-04-21T08:54:33.435 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.435+0000 7fd0e4ff9640 1 -- 172.21.15.28:0/2190134919 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f0e36d3d0 con 0x561f0e4e88b0 2024-04-21T08:54:33.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.435+0000 7fd0e4ff9640 10 client.16791 handle_command_reply: tid=0 2024-04-21T08:54:33.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.435+0000 7fd105f6b740 1 client.16791 shutdown 2024-04-21T08:54:33.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.435+0000 7fd105f6b740 1 -- 172.21.15.28:0/2190134919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e468b60 msgr2=0x561f0e47a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:33.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.435+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2190134919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e468b60 0x561f0e47a110 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fd0f80efd70 tx=0x7fd0f802bc80 comp rx=0 tx=0).stop 2024-04-21T08:54:33.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.435+0000 7fd105f6b740 1 -- 172.21.15.28:0/2190134919 shutdown_connections 2024-04-21T08:54:33.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.435+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2190134919 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561f0e4e88b0 0x561f0e4eacd0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.435+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2190134919 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561f0e4720e0 0x561f0e4759c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.435+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2190134919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561f0e4698d0 0x561f0e475480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.435+0000 7fd105f6b740 1 --2- 172.21.15.28:0/2190134919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f0e468b60 0x561f0e47a110 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.435+0000 7fd0e4ff9640 0 client.16791 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:33.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.435+0000 7fd105f6b740 1 -- 172.21.15.28:0/2190134919 >> 172.21.15.28:0/2190134919 conn(0x561f0e42fc90 msgr2=0x561f0e46de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:33.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.435+0000 7fd105f6b740 1 -- 172.21.15.28:0/2190134919 shutdown_connections 2024-04-21T08:54:33.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.435+0000 7fd105f6b740 1 -- 172.21.15.28:0/2190134919 wait complete. 2024-04-21T08:54:33.436 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.436+0000 7fd105f6b740 20 client.16791 trim_cache size 0 max 16384 2024-04-21T08:54:33.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:54:33.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:54:33.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:54:33.437 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:54:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.436+0000 7fd10329a640 1 -- 172.21.15.28:0/1829906953 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd0e80a3dc0 msgr2=0x7fd0e80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.436+0000 7fd10329a640 1 --2- 172.21.15.28:0/1829906953 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd0e80a3dc0 0x7fd0e80a6280 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd0f40126c0 tx=0x7fd0f4012ac0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.436+0000 7fd10329a640 1 -- 172.21.15.28:0/1829906953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0fc1011a0 msgr2=0x7fd0fc10e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.436+0000 7fd10329a640 1 --2- 172.21.15.28:0/1829906953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0fc1011a0 0x7fd0fc10e7d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8006010 tx=0x7fd0f8004040 comp rx=0 tx=0).stop 2024-04-21T08:54:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.436+0000 7fd10329a640 1 -- 172.21.15.28:0/1829906953 shutdown_connections 2024-04-21T08:54:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.436+0000 7fd10329a640 1 --2- 172.21.15.28:0/1829906953 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd0e80a3dc0 0x7fd0e80a6280 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.436+0000 7fd10329a640 1 --2- 172.21.15.28:0/1829906953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0fc105e20 0x7fd0fc10f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.436+0000 7fd10329a640 1 --2- 172.21.15.28:0/1829906953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd0fc101b40 0x7fd0fc10ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.436+0000 7fd10329a640 1 --2- 172.21.15.28:0/1829906953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0fc1011a0 0x7fd0fc10e7d0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:33.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.436+0000 7fd10329a640 1 -- 172.21.15.28:0/1829906953 >> 172.21.15.28:0/1829906953 conn(0x7fd0fc076c50 msgr2=0x7fd0fc0fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:33.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.436+0000 7fd10329a640 1 -- 172.21.15.28:0/1829906953 shutdown_connections 2024-04-21T08:54:33.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:33.436+0000 7fd10329a640 1 -- 172.21.15.28:0/1829906953 wait complete. 2024-04-21T08:54:34.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:34 smithi152 ceph-mon[31401]: pgmap v104: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 29 MiB/s wr, 5.00k op/s 2024-04-21T08:54:34.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:34 smithi028 ceph-mon[28003]: pgmap v104: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 29 MiB/s wr, 5.00k op/s 2024-04-21T08:54:34.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:34 smithi171 ceph-mon[31358]: pgmap v104: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 29 MiB/s wr, 5.00k op/s 2024-04-21T08:54:36.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:36 smithi152 ceph-mon[31401]: pgmap v105: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 36 MiB/s wr, 6.00k op/s 2024-04-21T08:54:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:36 smithi028 ceph-mon[28003]: pgmap v105: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 36 MiB/s wr, 6.00k op/s 2024-04-21T08:54:36.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:36 smithi171 ceph-mon[31358]: pgmap v105: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 36 MiB/s wr, 6.00k op/s 2024-04-21T08:54:38.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:38 smithi152 ceph-mon[31401]: pgmap v106: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 35 MiB/s wr, 5.74k op/s 2024-04-21T08:54:38.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:38 smithi152 ceph-mon[31401]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:54:38.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:38 smithi028 ceph-mon[28003]: pgmap v106: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 35 MiB/s wr, 5.74k op/s 2024-04-21T08:54:38.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:38 smithi028 ceph-mon[28003]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:54:38.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:38 smithi171 ceph-mon[31358]: pgmap v106: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 35 MiB/s wr, 5.74k op/s 2024-04-21T08:54:38.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:38 smithi171 ceph-mon[31358]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:54:40.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:40 smithi152 ceph-mon[31401]: pgmap v107: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 45 MiB/s wr, 7.48k op/s 2024-04-21T08:54:40.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:40 smithi028 ceph-mon[28003]: pgmap v107: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 45 MiB/s wr, 7.48k op/s 2024-04-21T08:54:40.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:40 smithi171 ceph-mon[31358]: pgmap v107: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 45 MiB/s wr, 7.48k op/s 2024-04-21T08:54:42.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:42 smithi152 ceph-mon[31401]: pgmap v108: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 40 MiB/s wr, 6.45k op/s 2024-04-21T08:54:42.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:42 smithi028 ceph-mon[28003]: pgmap v108: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 40 MiB/s wr, 6.45k op/s 2024-04-21T08:54:42.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:42 smithi171 ceph-mon[31358]: pgmap v108: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 40 MiB/s wr, 6.45k op/s 2024-04-21T08:54:44.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:44 smithi152 ceph-mon[31401]: pgmap v109: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 43 MiB/s wr, 7.01k op/s 2024-04-21T08:54:44.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:44 smithi028 ceph-mon[28003]: pgmap v109: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 43 MiB/s wr, 7.01k op/s 2024-04-21T08:54:44.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:44 smithi171 ceph-mon[31358]: pgmap v109: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 43 MiB/s wr, 7.01k op/s 2024-04-21T08:54:46.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:46 smithi152 ceph-mon[31401]: pgmap v110: 129 pgs: 129 active+clean; 14 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 43 MiB/s wr, 7.11k op/s 2024-04-21T08:54:46.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:46 smithi028 ceph-mon[28003]: pgmap v110: 129 pgs: 129 active+clean; 14 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 43 MiB/s wr, 7.11k op/s 2024-04-21T08:54:46.609 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:54:46.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:54:46.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:46 smithi171 ceph-mon[31358]: pgmap v110: 129 pgs: 129 active+clean; 14 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 43 MiB/s wr, 7.11k op/s 2024-04-21T08:54:46.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.965+0000 7f9eee8f2640 1 -- 172.21.15.28:0/2698381509 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ee8108f20 msgr2=0x7f9ee81153e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:46.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.965+0000 7f9eee8f2640 1 --2- 172.21.15.28:0/2698381509 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ee8108f20 0x7f9ee81153e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9ee001ace0 tx=0x7f9ee003d150 comp rx=0 tx=0).stop 2024-04-21T08:54:46.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.965+0000 7f9eee8f2640 1 -- 172.21.15.28:0/2698381509 shutdown_connections 2024-04-21T08:54:46.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.965+0000 7f9eee8f2640 1 --2- 172.21.15.28:0/2698381509 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ee8108f20 0x7f9ee81153e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:46.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.965+0000 7f9eee8f2640 1 --2- 172.21.15.28:0/2698381509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ee8108560 0x7f9ee81089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:46.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.965+0000 7f9eee8f2640 1 --2- 172.21.15.28:0/2698381509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee810e620 0x7f9ee810ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:46.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.965+0000 7f9eee8f2640 1 -- 172.21.15.28:0/2698381509 >> 172.21.15.28:0/2698381509 conn(0x7f9ee80764c0 msgr2=0x7f9ee80788e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:46.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.968+0000 7f9eee8f2640 1 -- 172.21.15.28:0/2698381509 shutdown_connections 2024-04-21T08:54:46.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.968+0000 7f9eee8f2640 1 -- 172.21.15.28:0/2698381509 wait complete. 2024-04-21T08:54:46.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.969+0000 7f9eee8f2640 1 Processor -- start 2024-04-21T08:54:46.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.969+0000 7f9eee8f2640 1 -- start start 2024-04-21T08:54:46.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.970+0000 7f9eee8f2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ee8108560 0x7f9ee8114e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:46.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.970+0000 7f9eee8f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8108f20 0x7f9ee81153d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:46.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.970+0000 7f9eee8f2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ee810e620 0x7f9ee80827f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:46.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.970+0000 7f9eee8f2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee806e1b0 con 0x7f9ee8108f20 2024-04-21T08:54:46.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.970+0000 7f9eee8f2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee806e030 con 0x7f9ee810e620 2024-04-21T08:54:46.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.970+0000 7f9eee8f2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee806e330 con 0x7f9ee8108560 2024-04-21T08:54:46.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.970+0000 7f9eece8f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ee810e620 0x7f9ee80827f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:46.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.970+0000 7f9eece8f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ee810e620 0x7f9ee80827f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:48442/0 (socket says 172.21.15.28:48442) 2024-04-21T08:54:46.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.970+0000 7f9eece8f640 1 -- 172.21.15.28:0/448616300 learned_addr learned my addr 172.21.15.28:0/448616300 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:46.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.970+0000 7f9ee7fff640 1 --2- 172.21.15.28:0/448616300 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ee8108560 0x7f9ee8114e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:46.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.970+0000 7f9ee77fe640 1 --2- 172.21.15.28:0/448616300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8108f20 0x7f9ee81153d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:46.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.971+0000 7fa189ed9640 1 -- 172.21.15.28:0/2778696363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa18410a0b0 msgr2=0x7fa18410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:46.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.971+0000 7fa189ed9640 1 --2- 172.21.15.28:0/2778696363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa18410a0b0 0x7fa18410e570 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa174009900 tx=0x7fa17402bd10 comp rx=0 tx=0).stop 2024-04-21T08:54:46.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.971+0000 7f9eece8f640 1 -- 172.21.15.28:0/448616300 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ee8108560 msgr2=0x7f9ee8114e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:46.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.971+0000 7f9eece8f640 1 --2- 172.21.15.28:0/448616300 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ee8108560 0x7f9ee8114e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:46.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.971+0000 7f9eece8f640 1 -- 172.21.15.28:0/448616300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8108f20 msgr2=0x7f9ee81153d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:46.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.971+0000 7f9eece8f640 1 --2- 172.21.15.28:0/448616300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8108f20 0x7f9ee81153d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:46.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.971+0000 7f9eece8f640 1 -- 172.21.15.28:0/448616300 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee81111d0 con 0x7f9ee810e620 2024-04-21T08:54:46.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.972+0000 7f9eece8f640 1 --2- 172.21.15.28:0/448616300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ee810e620 0x7f9ee80827f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9ee001a440 tx=0x7f9ee00045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:46.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.973+0000 7f9ee57fa640 1 -- 172.21.15.28:0/448616300 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee004a040 con 0x7f9ee810e620 2024-04-21T08:54:46.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.973+0000 7fa189ed9640 1 -- 172.21.15.28:0/2778696363 shutdown_connections 2024-04-21T08:54:46.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.973+0000 7fa189ed9640 1 --2- 172.21.15.28:0/2778696363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa18410eb90 0x7fa18410f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:46.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.973+0000 7fa189ed9640 1 --2- 172.21.15.28:0/2778696363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa18410a0b0 0x7fa18410e570 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:46.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.973+0000 7fa189ed9640 1 --2- 172.21.15.28:0/2778696363 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa184109790 0x7fa184109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:46.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.973+0000 7fa189ed9640 1 -- 172.21.15.28:0/2778696363 >> 172.21.15.28:0/2778696363 conn(0x7fa18406bd30 msgr2=0x7fa18406e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:46.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.973+0000 7fa189ed9640 1 -- 172.21.15.28:0/2778696363 shutdown_connections 2024-04-21T08:54:46.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.973+0000 7fa189ed9640 1 -- 172.21.15.28:0/2778696363 wait complete. 2024-04-21T08:54:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.973+0000 7f9eee8f2640 1 -- 172.21.15.28:0/448616300 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee8111460 con 0x7f9ee810e620 2024-04-21T08:54:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.973+0000 7f9eee8f2640 1 -- 172.21.15.28:0/448616300 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee81119d0 con 0x7f9ee810e620 2024-04-21T08:54:46.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.974+0000 7f9ee57fa640 1 -- 172.21.15.28:0/448616300 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ee0055070 con 0x7f9ee810e620 2024-04-21T08:54:46.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.975+0000 7f9ee57fa640 1 -- 172.21.15.28:0/448616300 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0050850 con 0x7f9ee810e620 2024-04-21T08:54:46.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.974+0000 7fa189ed9640 1 Processor -- start 2024-04-21T08:54:46.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.975+0000 7fa189ed9640 1 -- start start 2024-04-21T08:54:46.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.976+0000 7fa189ed9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa184109790 0x7fa1841a7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:46.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.976+0000 7fa189ed9640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa18410a0b0 0x7fa1841a7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:46.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.976+0000 7fa189ed9640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa18410eb90 0x7fa1841abec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:46.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.976+0000 7fa189ed9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1841210e0 con 0x7fa184109790 2024-04-21T08:54:46.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.976+0000 7fa189ed9640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa184120f60 con 0x7fa18410a0b0 2024-04-21T08:54:46.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.976+0000 7fa189ed9640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa184121260 con 0x7fa18410eb90 2024-04-21T08:54:46.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.977+0000 7fa1837fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa184109790 0x7fa1841a7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:46.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.977+0000 7fa1837fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa184109790 0x7fa1841a7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34058/0 (socket says 172.21.15.28:34058) 2024-04-21T08:54:46.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.977+0000 7fa1837fe640 1 -- 172.21.15.28:0/291137286 learned_addr learned my addr 172.21.15.28:0/291137286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:46.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.977+0000 7fa183fff640 1 --2- 172.21.15.28:0/291137286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa18410eb90 0x7fa1841abec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:46.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.979+0000 7fa183fff640 1 -- 172.21.15.28:0/291137286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa18410a0b0 msgr2=0x7fa1841a7b40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:54:46.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.979+0000 7fa183fff640 1 --2- 172.21.15.28:0/291137286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa18410a0b0 0x7fa1841a7b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:46.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.979+0000 7fa183fff640 1 -- 172.21.15.28:0/291137286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa184109790 msgr2=0x7fa1841a7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:46.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.979+0000 7fa183fff640 1 --2- 172.21.15.28:0/291137286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa184109790 0x7fa1841a7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:46.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.979+0000 7fa183fff640 1 -- 172.21.15.28:0/291137286 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1841ac5f0 con 0x7fa18410eb90 2024-04-21T08:54:46.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.979+0000 7fa183fff640 1 --2- 172.21.15.28:0/291137286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa18410eb90 0x7fa1841abec0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa17800b410 tx=0x7fa17800b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:46.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.979+0000 7f9ee57fa640 1 -- 172.21.15.28:0/448616300 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9ee00454e0 con 0x7f9ee810e620 2024-04-21T08:54:46.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.982+0000 7fa180ff9640 1 -- 172.21.15.28:0/291137286 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa178015020 con 0x7fa18410eb90 2024-04-21T08:54:46.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.980+0000 7f9ee57fa640 1 --2- 172.21.15.28:0/448616300 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9ed00a3d50 0x7f9ed00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:46.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.980+0000 7f9ee57fa640 1 -- 172.21.15.28:0/448616300 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f9ee0103bd0 con 0x7f9ee810e620 2024-04-21T08:54:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.982+0000 7fa180ff9640 1 -- 172.21.15.28:0/291137286 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa178011ce0 con 0x7fa18410eb90 2024-04-21T08:54:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.982+0000 7fa180ff9640 1 -- 172.21.15.28:0/291137286 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa178010db0 con 0x7fa18410eb90 2024-04-21T08:54:46.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.983+0000 7fa189ed9640 1 -- 172.21.15.28:0/291137286 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1841ac8e0 con 0x7fa18410eb90 2024-04-21T08:54:46.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.983+0000 7fa189ed9640 1 -- 172.21.15.28:0/291137286 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa18407a1b0 con 0x7fa18410eb90 2024-04-21T08:54:46.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.986+0000 7f9ee7fff640 1 --2- 172.21.15.28:0/448616300 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9ed00a3d50 0x7f9ed00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:46.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.988+0000 7fa180ff9640 1 -- 172.21.15.28:0/291137286 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa178018020 con 0x7fa18410eb90 2024-04-21T08:54:46.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.989+0000 7fa180ff9640 1 --2- 172.21.15.28:0/291137286 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa16c0a3f20 0x7fa16c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:46.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.989+0000 7fa180ff9640 1 -- 172.21.15.28:0/291137286 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fa1780d0150 con 0x7fa18410eb90 2024-04-21T08:54:46.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:46.992+0000 7fa1837fe640 1 --2- 172.21.15.28:0/291137286 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa16c0a3f20 0x7fa16c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:46.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:46.995+0000 7f9ee7fff640 1 --2- 172.21.15.28:0/448616300 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9ed00a3d50 0x7f9ed00a6210 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9edc009890 tx=0x7f9edc008040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:46.999 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.000+0000 7fa1837fe640 1 --2- 172.21.15.28:0/291137286 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa16c0a3f20 0x7fa16c0a63e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00e200 tx=0x7fa17c00b040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:47.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.043+0000 7f9ef15be740 1 Processor -- start 2024-04-21T08:54:47.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.045+0000 7fa18cba9740 1 Processor -- start 2024-04-21T08:54:47.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.045+0000 7f9ef15be740 1 -- start start 2024-04-21T08:54:47.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.046+0000 7f9ef15be740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927a7580 0x5555927a7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.046+0000 7f9ef15be740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927a7ea0 0x5555927b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.046+0000 7f9ef15be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927b0a80 0x5555927b2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.046+0000 7f9ef15be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55559270c1e0 con 0x5555927b0a80 2024-04-21T08:54:47.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.046+0000 7f9ef15be740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5555926a9390 con 0x5555927a7ea0 2024-04-21T08:54:47.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.046+0000 7f9ef15be740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5555927081a0 con 0x5555927a7580 2024-04-21T08:54:47.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.046+0000 7fa18cba9740 1 -- start start 2024-04-21T08:54:47.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.046+0000 7fa18cba9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ece4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.046+0000 7fa18cba9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 0x562d9ece4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.046+0000 7fa18cba9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ecee0e0 0x562d9ecf04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.046+0000 7fa18cba9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562d9ec498d0 con 0x562d9ecee0e0 2024-04-21T08:54:47.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.046+0000 7fa18cba9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562d9ebe6e80 con 0x562d9ece4b60 2024-04-21T08:54:47.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.046+0000 7fa18cba9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x562d9ec45890 con 0x562d9ece58d0 2024-04-21T08:54:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.046+0000 7f9eece8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927b0a80 0x5555927b2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.046+0000 7f9eece8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927b0a80 0x5555927b2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34066/0 (socket says 172.21.15.28:34066) 2024-04-21T08:54:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.046+0000 7f9eece8f640 1 -- 172.21.15.28:0/3474284113 learned_addr learned my addr 172.21.15.28:0/3474284113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.047+0000 7f9ee77fe640 1 --2- 172.21.15.28:0/3474284113 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927a7ea0 0x5555927b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.047+0000 7f9ee7fff640 1 --2- 172.21.15.28:0/3474284113 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927a7580 0x5555927a7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.048+0000 7f9ee77fe640 1 -- 172.21.15.28:0/3474284113 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927a7580 msgr2=0x5555927a7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.048+0000 7f9ee77fe640 1 --2- 172.21.15.28:0/3474284113 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927a7580 0x5555927a7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.048+0000 7f9ee77fe640 1 -- 172.21.15.28:0/3474284113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927b0a80 msgr2=0x5555927b2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.048+0000 7f9ee77fe640 1 --2- 172.21.15.28:0/3474284113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927b0a80 0x5555927b2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.048+0000 7f9ee77fe640 1 -- 172.21.15.28:0/3474284113 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555924e87d0 con 0x5555927a7ea0 2024-04-21T08:54:47.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.049+0000 7f9ee77fe640 1 --2- 172.21.15.28:0/3474284113 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927a7ea0 0x5555927b0540 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8112770 tx=0x7f9ed800f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3a3910b008449d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.047+0000 7fa182ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 0x562d9ece4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.047+0000 7fa182ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 0x562d9ece4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:48458/0 (socket says 172.21.15.28:48458) 2024-04-21T08:54:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.047+0000 7fa182ffd640 1 -- 172.21.15.28:0/3459310747 learned_addr learned my addr 172.21.15.28:0/3459310747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.048+0000 7fa1837fe640 1 --2- 172.21.15.28:0/3459310747 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ece4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.050+0000 7fa182ffd640 1 -- 172.21.15.28:0/3459310747 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 msgr2=0x562d9ece4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.050+0000 7fa182ffd640 1 --2- 172.21.15.28:0/3459310747 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ece4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.050+0000 7fa182ffd640 1 -- 172.21.15.28:0/3459310747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ecee0e0 msgr2=0x562d9ecf04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.050+0000 7fa182ffd640 1 --2- 172.21.15.28:0/3459310747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ecee0e0 0x562d9ecf04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.050+0000 7fa182ffd640 1 -- 172.21.15.28:0/3459310747 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d9ec08320 con 0x562d9ece4b60 2024-04-21T08:54:47.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.051+0000 7f9ece7fc640 1 -- 172.21.15.28:0/3474284113 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8004020 con 0x5555927a7ea0 2024-04-21T08:54:47.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.051+0000 7f9ece7fc640 1 -- 172.21.15.28:0/3474284113 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ed80041c0 con 0x5555927a7ea0 2024-04-21T08:54:47.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.051+0000 7f9ece7fc640 1 -- 172.21.15.28:0/3474284113 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8018730 con 0x5555927a7ea0 2024-04-21T08:54:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.052+0000 7fa182ffd640 1 --2- 172.21.15.28:0/3459310747 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 0x562d9ece4fe0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa174005c20 tx=0x7fa17402c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30a4f0aef6d7d4c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:47.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.052+0000 7f9ef15be740 1 -- 172.21.15.28:0/3474284113 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927a7ea0 msgr2=0x5555927b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:47.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.052+0000 7f9ef15be740 1 --2- 172.21.15.28:0/3474284113 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927a7ea0 0x5555927b0540 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8112770 tx=0x7f9ed800f3c0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.054+0000 7fa169ffb640 1 -- 172.21.15.28:0/3459310747 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa174041020 con 0x562d9ece4b60 2024-04-21T08:54:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.054+0000 7fa169ffb640 1 -- 172.21.15.28:0/3459310747 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa17402cd40 con 0x562d9ece4b60 2024-04-21T08:54:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.054+0000 7fa169ffb640 1 -- 172.21.15.28:0/3459310747 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1740046c0 con 0x562d9ece4b60 2024-04-21T08:54:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.055+0000 7fa18cba9740 1 -- 172.21.15.28:0/3459310747 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 msgr2=0x562d9ece4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.055+0000 7fa18cba9740 1 --2- 172.21.15.28:0/3459310747 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 0x562d9ece4fe0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa174005c20 tx=0x7fa17402c7f0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.057+0000 7f9ef15be740 1 -- 172.21.15.28:0/3474284113 shutdown_connections 2024-04-21T08:54:47.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.057+0000 7f9ef15be740 1 --2- 172.21.15.28:0/3474284113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927b0a80 0x5555927b2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.057+0000 7f9ef15be740 1 --2- 172.21.15.28:0/3474284113 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927a7ea0 0x5555927b0540 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.057+0000 7f9ef15be740 1 --2- 172.21.15.28:0/3474284113 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927a7580 0x5555927a7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.057+0000 7f9ef15be740 1 -- 172.21.15.28:0/3474284113 >> 172.21.15.28:0/3474284113 conn(0x5555927a1cb0 msgr2=0x5555927a2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.058+0000 7fa18cba9740 1 -- 172.21.15.28:0/3459310747 shutdown_connections 2024-04-21T08:54:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.058+0000 7fa18cba9740 1 --2- 172.21.15.28:0/3459310747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ecee0e0 0x562d9ecf04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.058+0000 7fa18cba9740 1 --2- 172.21.15.28:0/3459310747 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 0x562d9ece4fe0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.058+0000 7fa18cba9740 1 --2- 172.21.15.28:0/3459310747 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ece4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.058+0000 7fa18cba9740 1 -- 172.21.15.28:0/3459310747 >> 172.21.15.28:0/3459310747 conn(0x562d9ecabc90 msgr2=0x562d9ecdf850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:47.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.064+0000 7fa18cba9740 1 -- 172.21.15.28:0/3459310747 shutdown_connections 2024-04-21T08:54:47.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.063+0000 7f9ef15be740 1 -- 172.21.15.28:0/3474284113 shutdown_connections 2024-04-21T08:54:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.064+0000 7fa18cba9740 1 -- 172.21.15.28:0/3459310747 wait complete. 2024-04-21T08:54:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.064+0000 7fa18cba9740 1 -- start start 2024-04-21T08:54:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.064+0000 7fa18cba9740 1 Processor -- start 2024-04-21T08:54:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.065+0000 7fa18cba9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece4b60 0x562d9eceda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.065+0000 7fa18cba9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ece58d0 0x562d9eceae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.065+0000 7fa18cba9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ecee0e0 0x562d9eceb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.065+0000 7fa18cba9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562d9ecf0db0 con 0x562d9ece58d0 2024-04-21T08:54:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.065+0000 7fa18cba9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562d9ebd1eb0 con 0x562d9ecee0e0 2024-04-21T08:54:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.065+0000 7fa18cba9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x562d9ecf0f30 con 0x562d9ece4b60 2024-04-21T08:54:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.065+0000 7fa18cba9740 10 client.0 _ll_register_callbacks cb 0x562d9ebfeb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.066+0000 7fa182ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ece58d0 0x562d9eceae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.066+0000 7fa182ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ece58d0 0x562d9eceae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34092/0 (socket says 172.21.15.28:34092) 2024-04-21T08:54:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.066+0000 7fa182ffd640 1 -- 172.21.15.28:0/7780026 learned_addr learned my addr 172.21.15.28:0/7780026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.066+0000 7fa183fff640 1 --2- 172.21.15.28:0/7780026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ecee0e0 0x562d9eceb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.067+0000 7fa183fff640 1 -- 172.21.15.28:0/7780026 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece4b60 msgr2=0x562d9eceda90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:54:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.068+0000 7fa183fff640 1 --2- 172.21.15.28:0/7780026 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece4b60 0x562d9eceda90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.068+0000 7fa183fff640 1 -- 172.21.15.28:0/7780026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ece58d0 msgr2=0x562d9eceae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:47.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.068+0000 7fa183fff640 1 --2- 172.21.15.28:0/7780026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ece58d0 0x562d9eceae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.068+0000 7fa183fff640 1 -- 172.21.15.28:0/7780026 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d9e95c210 con 0x562d9ecee0e0 2024-04-21T08:54:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.067+0000 7f9ef15be740 1 -- 172.21.15.28:0/3474284113 wait complete. 2024-04-21T08:54:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.068+0000 7f9ef15be740 1 -- start start 2024-04-21T08:54:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.068+0000 7f9ef15be740 1 Processor -- start 2024-04-21T08:54:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.069+0000 7f9ef15be740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927a7580 0x5555927ada40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.069+0000 7f9ef15be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 0x5555927adf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.069+0000 7f9ef15be740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927b0a80 0x5555927ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.069+0000 7f9ef15be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5555927b3870 con 0x5555927a7ea0 2024-04-21T08:54:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.069+0000 7f9ef15be740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5555926c52f0 con 0x5555927a7580 2024-04-21T08:54:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.069+0000 7f9ef15be740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5555927b39f0 con 0x5555927b0a80 2024-04-21T08:54:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.069+0000 7f9ef15be740 10 client.0 _ll_register_callbacks cb 0x5555926d2580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:47.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.069+0000 7f9eece8f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927b0a80 0x5555927ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.070+0000 7f9eece8f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927b0a80 0x5555927ae4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:60614/0 (socket says 172.21.15.28:60614) 2024-04-21T08:54:47.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.070+0000 7f9eece8f640 1 -- 172.21.15.28:0/3388791355 learned_addr learned my addr 172.21.15.28:0/3388791355 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.070+0000 7f9ee77fe640 1 --2- 172.21.15.28:0/3388791355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 0x5555927adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.071+0000 7f9eece8f640 1 -- 172.21.15.28:0/3388791355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927a7580 msgr2=0x5555927ada40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:54:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.071+0000 7f9eece8f640 1 --2- 172.21.15.28:0/3388791355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927a7580 0x5555927ada40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.071+0000 7f9eece8f640 1 -- 172.21.15.28:0/3388791355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 msgr2=0x5555927adf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.071+0000 7f9eece8f640 1 --2- 172.21.15.28:0/3388791355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 0x5555927adf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.071+0000 7f9eece8f640 1 -- 172.21.15.28:0/3388791355 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55559241e3f0 con 0x5555927b0a80 2024-04-21T08:54:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.071+0000 7f9eece8f640 1 --2- 172.21.15.28:0/3388791355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927b0a80 0x5555927ae4c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9ee00cbce0 tx=0x7f9ee00cf2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a54173504302729 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.072+0000 7f9ec77fe640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:54:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.072+0000 7f9ec77fe640 1 -- 172.21.15.28:0/3388791355 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee00d3c30 con 0x5555927b0a80 2024-04-21T08:54:47.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.071+0000 7fa183fff640 1 --2- 172.21.15.28:0/7780026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ecee0e0 0x562d9eceb3c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa1780cd5b0 tx=0x7fa17809ba00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f26646bee7c7fafd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:47.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.071+0000 7fa162ffd640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:54:47.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.072+0000 7fa162ffd640 1 -- 172.21.15.28:0/7780026 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17809be00 con 0x562d9ecee0e0 2024-04-21T08:54:47.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.072+0000 7fa18cba9740 1 -- 172.21.15.28:0/7780026 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa17402cd40 con 0x562d9ecee0e0 2024-04-21T08:54:47.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.072+0000 7f9ef15be740 1 -- 172.21.15.28:0/3388791355 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ed80041c0 con 0x5555927b0a80 2024-04-21T08:54:47.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.072+0000 7f9ec77fe640 1 -- 172.21.15.28:0/3388791355 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f9ee01011a0 con 0x5555927b0a80 2024-04-21T08:54:47.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.072+0000 7fa162ffd640 1 -- 172.21.15.28:0/7780026 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fa17809f070 con 0x562d9ecee0e0 2024-04-21T08:54:47.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.074+0000 7fa162ffd640 1 -- 172.21.15.28:0/7780026 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa1780a01b0 con 0x562d9ecee0e0 2024-04-21T08:54:47.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.074+0000 7fa18cba9740 10 client.26099 fetch_fsmap learned FSMap version 27 2024-04-21T08:54:47.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.074+0000 7fa18cba9740 1 -- 172.21.15.28:0/7780026 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562d9ea28620 con 0x562d9ecee0e0 2024-04-21T08:54:47.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.075+0000 7fa162ffd640 1 -- 172.21.15.28:0/7780026 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa17402cd40 con 0x562d9ecee0e0 2024-04-21T08:54:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.074+0000 7f9ec77fe640 1 -- 172.21.15.28:0/3388791355 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9ee01013a0 con 0x5555927b0a80 2024-04-21T08:54:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.075+0000 7f9ef15be740 10 client.26116 fetch_fsmap learned FSMap version 27 2024-04-21T08:54:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.075+0000 7f9ef15be740 1 -- 172.21.15.28:0/3388791355 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5555923de0f0 con 0x5555927b0a80 2024-04-21T08:54:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.076+0000 7fa162ffd640 1 -- 172.21.15.28:0/7780026 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7fa17809b250 con 0x562d9ecee0e0 2024-04-21T08:54:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.076+0000 7fa18cba9740 10 client.26099 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T08:54:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.076+0000 7fa18cba9740 10 client.26099 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.076+0000 7fa18cba9740 20 client.26099 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.076+0000 7fa18cba9740 1 --2- 172.21.15.28:0/7780026 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562d9ecfe5e0 0x562d9ed00a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.076+0000 7fa18cba9740 4 client.26099 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.076+0000 7fa18cba9740 1 -- 172.21.15.28:0/7780026 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d9ec5d880 con 0x562d9ecfe5e0 2024-04-21T08:54:47.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.079+0000 7f9ec77fe640 1 -- 172.21.15.28:0/3388791355 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ed80041c0 con 0x5555927b0a80 2024-04-21T08:54:47.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.080+0000 7f9ec77fe640 1 -- 172.21.15.28:0/3388791355 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7f9ee00d0bb0 con 0x5555927b0a80 2024-04-21T08:54:47.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.080+0000 7f9ef15be740 10 client.26116 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T08:54:47.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.080+0000 7f9ef15be740 10 client.26116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:47.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.080+0000 7f9ef15be740 20 client.26116 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:47.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.080+0000 7f9ef15be740 1 --2- 172.21.15.28:0/3388791355 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555927c0af0 0x5555927c2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.080+0000 7f9ef15be740 4 client.26116 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:47.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.080+0000 7f9ef15be740 1 -- 172.21.15.28:0/3388791355 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555592719c00 con 0x5555927c0af0 2024-04-21T08:54:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.085+0000 7fa1837fe640 1 --2- 172.21.15.28:0/7780026 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562d9ecfe5e0 0x562d9ed00a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.085+0000 7f9ee7fff640 1 --2- 172.21.15.28:0/3388791355 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555927c0af0 0x5555927c2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.089+0000 7fa1837fe640 1 --2- 172.21.15.28:0/7780026 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562d9ecfe5e0 0x562d9ed00a20 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18d35070cf3f6ec8 server_cookie=8f74c7bd39f9d253 in_seq=0 out_seq=0 2024-04-21T08:54:47.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.089+0000 7fa162ffd640 10 client.26099 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:47.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.090+0000 7fa162ffd640 1 -- 172.21.15.28:0/7780026 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562d9ec5d880 con 0x562d9ecfe5e0 2024-04-21T08:54:47.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.090+0000 7fa162ffd640 10 client.26099 handle_command_reply: tid=0 2024-04-21T08:54:47.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.093+0000 7fa18cba9740 1 client.26099 shutdown 2024-04-21T08:54:47.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.090+0000 7f9ee7fff640 1 --2- 172.21.15.28:0/3388791355 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555927c0af0 0x5555927c2f70 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b6b214bbf03eeef server_cookie=353fae217a84341a in_seq=0 out_seq=0 2024-04-21T08:54:47.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.090+0000 7f9ec77fe640 10 client.26116 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:47.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.092+0000 7f9ec77fe640 1 -- 172.21.15.28:0/3388791355 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555592719c00 con 0x5555927c0af0 2024-04-21T08:54:47.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.092+0000 7f9ec77fe640 10 client.26116 handle_command_reply: tid=0 2024-04-21T08:54:47.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.092+0000 7f9ef15be740 1 client.26116 shutdown 2024-04-21T08:54:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.093+0000 7fa18cba9740 1 -- 172.21.15.28:0/7780026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ecee0e0 msgr2=0x562d9eceb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.093+0000 7fa18cba9740 1 --2- 172.21.15.28:0/7780026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ecee0e0 0x562d9eceb3c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa1780cd5b0 tx=0x7fa17809ba00 comp rx=0 tx=0).stop 2024-04-21T08:54:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.095+0000 7fa18cba9740 1 -- 172.21.15.28:0/7780026 shutdown_connections 2024-04-21T08:54:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.095+0000 7fa18cba9740 1 --2- 172.21.15.28:0/7780026 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562d9ecfe5e0 0x562d9ed00a20 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.095+0000 7fa18cba9740 1 --2- 172.21.15.28:0/7780026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ecee0e0 0x562d9eceb3c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.095+0000 7fa18cba9740 1 --2- 172.21.15.28:0/7780026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ece58d0 0x562d9eceae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.095+0000 7fa18cba9740 1 --2- 172.21.15.28:0/7780026 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece4b60 0x562d9eceda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.095+0000 7fa18cba9740 1 -- 172.21.15.28:0/7780026 >> 172.21.15.28:0/7780026 conn(0x562d9ecabc90 msgr2=0x562d9ece15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:47.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.097+0000 7f9ef15be740 1 -- 172.21.15.28:0/3388791355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927b0a80 msgr2=0x5555927ae4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:47.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.097+0000 7f9ef15be740 1 --2- 172.21.15.28:0/3388791355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927b0a80 0x5555927ae4c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9ee00cbce0 tx=0x7f9ee00cf2a0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.096+0000 7fa162ffd640 0 client.26099 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.098+0000 7fa18cba9740 1 -- 172.21.15.28:0/7780026 shutdown_connections 2024-04-21T08:54:47.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.099+0000 7fa18cba9740 1 -- 172.21.15.28:0/7780026 wait complete. 2024-04-21T08:54:47.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.101+0000 7f9ef15be740 1 -- 172.21.15.28:0/3388791355 shutdown_connections 2024-04-21T08:54:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.101+0000 7f9ef15be740 1 --2- 172.21.15.28:0/3388791355 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555927c0af0 0x5555927c2f70 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.101+0000 7f9ef15be740 1 --2- 172.21.15.28:0/3388791355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927b0a80 0x5555927ae4c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.101+0000 7f9ef15be740 1 --2- 172.21.15.28:0/3388791355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 0x5555927adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.101+0000 7f9ef15be740 1 --2- 172.21.15.28:0/3388791355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927a7580 0x5555927ada40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.101+0000 7f9ef15be740 1 -- 172.21.15.28:0/3388791355 >> 172.21.15.28:0/3388791355 conn(0x5555927a1cb0 msgr2=0x5555927a3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.102+0000 7fa18cba9740 20 client.26099 trim_cache size 0 max 16384 2024-04-21T08:54:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.102+0000 7f9ec77fe640 0 client.26116 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.102+0000 7f9ef15be740 1 -- 172.21.15.28:0/3388791355 shutdown_connections 2024-04-21T08:54:47.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.104+0000 7f9ef15be740 1 -- 172.21.15.28:0/3388791355 wait complete. 2024-04-21T08:54:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.105+0000 7f9ef15be740 20 client.26116 trim_cache size 0 max 16384 2024-04-21T08:54:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.139+0000 7fa18cba9740 1 Processor -- start 2024-04-21T08:54:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.140+0000 7fa18cba9740 1 -- start start 2024-04-21T08:54:47.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.140+0000 7fa18cba9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 0x562d9ecf9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.140+0000 7fa18cba9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ed062f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.141+0000 7fa1837fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 0x562d9ecf9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.141+0000 7fa1837fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 0x562d9ecf9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:48494/0 (socket says 172.21.15.28:48494) 2024-04-21T08:54:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.141+0000 7fa182ffd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ed062f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.141+0000 7fa182ffd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ed062f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:60624/0 (socket says 172.21.15.28:60624) 2024-04-21T08:54:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.141+0000 7fa1837fe640 1 -- 172.21.15.28:0/2640431117 learned_addr learned my addr 172.21.15.28:0/2640431117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.141+0000 7fa18cba9740 1 --2- 172.21.15.28:0/2640431117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ecee0e0 0x562d9ed06830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.141+0000 7fa18cba9740 1 -- 172.21.15.28:0/2640431117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1700126a0 con 0x562d9ecee0e0 2024-04-21T08:54:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.141+0000 7fa18cba9740 1 -- 172.21.15.28:0/2640431117 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa17000ff70 con 0x562d9ece4b60 2024-04-21T08:54:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.141+0000 7fa18cba9740 1 -- 172.21.15.28:0/2640431117 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa17000de70 con 0x562d9ece58d0 2024-04-21T08:54:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.142+0000 7fa183fff640 1 --2- 172.21.15.28:0/2640431117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ecee0e0 0x562d9ed06830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.142+0000 7fa183fff640 1 -- 172.21.15.28:0/2640431117 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 msgr2=0x562d9ed062f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.142+0000 7fa183fff640 1 --2- 172.21.15.28:0/2640431117 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ed062f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.142+0000 7fa183fff640 1 -- 172.21.15.28:0/2640431117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 msgr2=0x562d9ecf9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.142+0000 7fa183fff640 1 --2- 172.21.15.28:0/2640431117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 0x562d9ecf9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.142+0000 7fa183fff640 1 -- 172.21.15.28:0/2640431117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d9ece09c0 con 0x562d9ecee0e0 2024-04-21T08:54:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.143+0000 7fa183fff640 1 --2- 172.21.15.28:0/2640431117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ecee0e0 0x562d9ed06830 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa1780987d0 tx=0x7fa17809a310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c14ea5ab25ca1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.143+0000 7fa1837fe640 1 --2- 172.21.15.28:0/2640431117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 0x562d9ecf9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:47.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.144+0000 7fa182ffd640 1 --2- 172.21.15.28:0/2640431117 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ed062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:47.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.144+0000 7fa1697fa640 1 -- 172.21.15.28:0/2640431117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1780985d0 con 0x562d9ecee0e0 2024-04-21T08:54:47.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.144+0000 7fa1697fa640 1 -- 172.21.15.28:0/2640431117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa17809be00 con 0x562d9ecee0e0 2024-04-21T08:54:47.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.145+0000 7fa18cba9740 1 -- 172.21.15.28:0/2640431117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ecee0e0 msgr2=0x562d9ed06830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:47.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.145+0000 7fa18cba9740 1 --2- 172.21.15.28:0/2640431117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ecee0e0 0x562d9ed06830 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa1780987d0 tx=0x7fa17809a310 comp rx=0 tx=0).stop 2024-04-21T08:54:47.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.145+0000 7fa18cba9740 1 -- 172.21.15.28:0/2640431117 shutdown_connections 2024-04-21T08:54:47.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.145+0000 7fa18cba9740 1 --2- 172.21.15.28:0/2640431117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ecee0e0 0x562d9ed06830 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.145+0000 7fa18cba9740 1 --2- 172.21.15.28:0/2640431117 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ed062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.145+0000 7fa18cba9740 1 --2- 172.21.15.28:0/2640431117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 0x562d9ecf9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.145+0000 7fa18cba9740 1 -- 172.21.15.28:0/2640431117 >> 172.21.15.28:0/2640431117 conn(0x562d9ecabc90 msgr2=0x562d9ecdf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.145+0000 7fa18cba9740 1 -- 172.21.15.28:0/2640431117 shutdown_connections 2024-04-21T08:54:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.146+0000 7fa18cba9740 1 -- 172.21.15.28:0/2640431117 wait complete. 2024-04-21T08:54:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.146+0000 7fa18cba9740 1 -- start start 2024-04-21T08:54:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.146+0000 7fa18cba9740 1 Processor -- start 2024-04-21T08:54:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.145+0000 7f9ef15be740 1 Processor -- start 2024-04-21T08:54:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.145+0000 7f9ef15be740 1 -- start start 2024-04-21T08:54:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.146+0000 7f9ef15be740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927a7580 0x5555927b0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.146+0000 7f9ef15be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 0x5555927bc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.146+0000 7f9ef15be740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927b0a80 0x5555927a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.146+0000 7f9ef15be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed40127b0 con 0x5555927a7ea0 2024-04-21T08:54:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.146+0000 7f9ef15be740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed4010080 con 0x5555927a7580 2024-04-21T08:54:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.146+0000 7f9ef15be740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed400ded0 con 0x5555927b0a80 2024-04-21T08:54:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.147+0000 7f9eece8f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927b0a80 0x5555927a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.147+0000 7f9eece8f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927b0a80 0x5555927a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:60630/0 (socket says 172.21.15.28:60630) 2024-04-21T08:54:47.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.147+0000 7f9eece8f640 1 -- 172.21.15.28:0/3391422922 learned_addr learned my addr 172.21.15.28:0/3391422922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.147+0000 7fa18cba9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 0x562d9ecf65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.147+0000 7fa18cba9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ecf1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.147+0000 7fa18cba9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ecee0e0 0x562d9ecf1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.147+0000 7fa18cba9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa17000e430 con 0x562d9ecee0e0 2024-04-21T08:54:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.147+0000 7fa18cba9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa17000f8a0 con 0x562d9ece4b60 2024-04-21T08:54:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.147+0000 7fa18cba9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa17000eb00 con 0x562d9ece58d0 2024-04-21T08:54:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.147+0000 7fa18cba9740 10 client.0 _ll_register_callbacks cb 0x7fa178009210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.147+0000 7fa182ffd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ecf1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.147+0000 7fa182ffd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ecf1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:60642/0 (socket says 172.21.15.28:60642) 2024-04-21T08:54:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.147+0000 7fa182ffd640 1 -- 172.21.15.28:0/692513745 learned_addr learned my addr 172.21.15.28:0/692513745 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:47.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.147+0000 7f9ee77fe640 1 --2- 172.21.15.28:0/3391422922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 0x5555927bc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.148+0000 7f9eece8f640 1 -- 172.21.15.28:0/3391422922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927a7580 msgr2=0x5555927b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:47.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.148+0000 7f9eece8f640 1 --2- 172.21.15.28:0/3391422922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927a7580 0x5555927b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.148+0000 7f9eece8f640 1 -- 172.21.15.28:0/3391422922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 msgr2=0x5555927bc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:47.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.148+0000 7f9eece8f640 1 --2- 172.21.15.28:0/3391422922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 0x5555927bc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.148+0000 7f9eece8f640 1 -- 172.21.15.28:0/3391422922 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555927b2e70 con 0x5555927b0a80 2024-04-21T08:54:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.148+0000 7fa183fff640 1 --2- 172.21.15.28:0/692513745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ecee0e0 0x562d9ecf1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.148+0000 7fa182ffd640 1 -- 172.21.15.28:0/692513745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 msgr2=0x562d9ecf65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.148+0000 7fa182ffd640 1 --2- 172.21.15.28:0/692513745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 0x562d9ecf65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.148+0000 7fa182ffd640 1 -- 172.21.15.28:0/692513745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ecee0e0 msgr2=0x562d9ecf1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.148+0000 7fa182ffd640 1 --2- 172.21.15.28:0/692513745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ecee0e0 0x562d9ecf1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.148+0000 7fa182ffd640 1 -- 172.21.15.28:0/692513745 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d9ecf2670 con 0x562d9ece58d0 2024-04-21T08:54:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.148+0000 7fa182ffd640 1 --2- 172.21.15.28:0/692513745 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ecf1920 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa17402c250 tx=0x7fa17402cac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb768ad7d95377eb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.149+0000 7fa162ffd640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:54:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.149+0000 7fa162ffd640 1 -- 172.21.15.28:0/692513745 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa174041040 con 0x562d9ece58d0 2024-04-21T08:54:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.149+0000 7fa162ffd640 1 -- 172.21.15.28:0/692513745 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fa174005250 con 0x562d9ece58d0 2024-04-21T08:54:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.149+0000 7fa18cba9740 1 -- 172.21.15.28:0/692513745 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa17809be00 con 0x562d9ece58d0 2024-04-21T08:54:47.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.152+0000 7fa162ffd640 1 -- 172.21.15.28:0/692513745 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa174058050 con 0x562d9ece58d0 2024-04-21T08:54:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.152+0000 7f9eece8f640 1 --2- 172.21.15.28:0/3391422922 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927b0a80 0x5555927a0bf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9ee00d3df0 tx=0x7f9ee00cfa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d07590606af2358 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.155+0000 7f9ecdffb640 1 -- 172.21.15.28:0/3391422922 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee00d84f0 con 0x5555927b0a80 2024-04-21T08:54:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.155+0000 7f9ecdffb640 1 -- 172.21.15.28:0/3391422922 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ee00cee90 con 0x5555927b0a80 2024-04-21T08:54:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.155+0000 7f9ecdffb640 1 -- 172.21.15.28:0/3391422922 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee00d82c0 con 0x5555927b0a80 2024-04-21T08:54:47.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.157+0000 7f9ef15be740 1 -- 172.21.15.28:0/3391422922 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927b0a80 msgr2=0x5555927a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:47.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.157+0000 7f9ef15be740 1 --2- 172.21.15.28:0/3391422922 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927b0a80 0x5555927a0bf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9ee00d3df0 tx=0x7f9ee00cfa00 comp rx=0 tx=0).stop 2024-04-21T08:54:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.157+0000 7f9ef15be740 1 -- 172.21.15.28:0/3391422922 shutdown_connections 2024-04-21T08:54:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.157+0000 7f9ef15be740 1 --2- 172.21.15.28:0/3391422922 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927b0a80 0x5555927a0bf0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.157+0000 7f9ef15be740 1 --2- 172.21.15.28:0/3391422922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 0x5555927bc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.157+0000 7f9ef15be740 1 --2- 172.21.15.28:0/3391422922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927a7580 0x5555927b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.157+0000 7f9ef15be740 1 -- 172.21.15.28:0/3391422922 >> 172.21.15.28:0/3391422922 conn(0x5555927a1cb0 msgr2=0x5555927a04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:47.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.158+0000 7fa18cba9740 10 client.26122 fetch_fsmap learned FSMap version 27 2024-04-21T08:54:47.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.158+0000 7fa18cba9740 1 -- 172.21.15.28:0/692513745 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562d9ed64ab0 con 0x562d9ece58d0 2024-04-21T08:54:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.162+0000 7f9ef15be740 1 -- 172.21.15.28:0/3391422922 shutdown_connections 2024-04-21T08:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.161+0000 7fa162ffd640 1 -- 172.21.15.28:0/692513745 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa17809be00 con 0x562d9ece58d0 2024-04-21T08:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.162+0000 7fa162ffd640 1 -- 172.21.15.28:0/692513745 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7fa174004d10 con 0x562d9ece58d0 2024-04-21T08:54:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.162+0000 7f9ef15be740 1 -- 172.21.15.28:0/3391422922 wait complete. 2024-04-21T08:54:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.162+0000 7f9ef15be740 1 -- start start 2024-04-21T08:54:47.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.163+0000 7f9ef15be740 1 Processor -- start 2024-04-21T08:54:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.163+0000 7fa18cba9740 10 client.26122 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T08:54:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.163+0000 7fa18cba9740 10 client.26122 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.163+0000 7fa18cba9740 20 client.26122 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.163+0000 7fa18cba9740 1 --2- 172.21.15.28:0/692513745 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562d9ed64da0 0x562d9ed67190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.163+0000 7fa18cba9740 4 client.26122 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:54:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.163+0000 7fa18cba9740 1 -- 172.21.15.28:0/692513745 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562d9ebe93d0 con 0x562d9ed64da0 2024-04-21T08:54:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.163+0000 7f9ef15be740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927a7580 0x5555927aa580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.163+0000 7f9ef15be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 0x5555927aaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.163+0000 7f9ef15be740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927b0a80 0x5555927ab000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.163+0000 7f9ef15be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed400e540 con 0x5555927a7ea0 2024-04-21T08:54:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.163+0000 7f9ef15be740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed400f9b0 con 0x5555927b0a80 2024-04-21T08:54:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.163+0000 7f9ef15be740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed400ec10 con 0x5555927a7580 2024-04-21T08:54:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.163+0000 7f9ef15be740 10 client.0 _ll_register_callbacks cb 0x7f9ee0100fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.164+0000 7f9eece8f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927b0a80 0x5555927ab000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.165+0000 7f9ee77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 0x5555927aaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.165+0000 7f9eece8f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927b0a80 0x5555927ab000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:48516/0 (socket says 172.21.15.28:48516) 2024-04-21T08:54:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.165+0000 7f9eece8f640 1 -- 172.21.15.28:0/2359991180 learned_addr learned my addr 172.21.15.28:0/2359991180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.165+0000 7f9ee7fff640 1 --2- 172.21.15.28:0/2359991180 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927a7580 0x5555927aa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.165+0000 7f9ee77fe640 1 -- 172.21.15.28:0/2359991180 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927a7580 msgr2=0x5555927aa580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.165+0000 7f9ee77fe640 1 --2- 172.21.15.28:0/2359991180 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927a7580 0x5555927aa580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.165+0000 7f9ee77fe640 1 -- 172.21.15.28:0/2359991180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927b0a80 msgr2=0x5555927ab000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:47.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.165+0000 7f9ee77fe640 1 --2- 172.21.15.28:0/2359991180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927b0a80 0x5555927ab000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.165+0000 7f9ee77fe640 1 -- 172.21.15.28:0/2359991180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555928260f0 con 0x5555927a7ea0 2024-04-21T08:54:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.166+0000 7fa1837fe640 1 --2- 172.21.15.28:0/692513745 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562d9ed64da0 0x562d9ed67190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.166+0000 7f9ee77fe640 1 --2- 172.21.15.28:0/2359991180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 0x5555927aaac0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800a740 tx=0x7f9ed800a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878c5a5c80d1cad2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:47.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.166+0000 7f9ec77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:47.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.167+0000 7fa1837fe640 1 --2- 172.21.15.28:0/692513745 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562d9ed64da0 0x562d9ed67190 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69100e2a95e58bcc server_cookie=7edf74c7855be433 in_seq=0 out_seq=0 2024-04-21T08:54:47.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.167+0000 7fa162ffd640 10 client.26122 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.169+0000 7f9ec77fe640 1 -- 172.21.15.28:0/2359991180 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed800a770 con 0x5555927a7ea0 2024-04-21T08:54:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.170+0000 7f9ef15be740 1 -- 172.21.15.28:0/2359991180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ee00cee90 con 0x5555927a7ea0 2024-04-21T08:54:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.170+0000 7f9ec77fe640 1 -- 172.21.15.28:0/2359991180 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f9ed8005040 con 0x5555927a7ea0 2024-04-21T08:54:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.170+0000 7f9ec77fe640 1 -- 172.21.15.28:0/2359991180 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9ed8005240 con 0x5555927a7ea0 2024-04-21T08:54:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.170+0000 7f9ef15be740 10 client.16818 fetch_fsmap learned FSMap version 27 2024-04-21T08:54:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.170+0000 7f9ef15be740 1 -- 172.21.15.28:0/2359991180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5555928272e0 con 0x5555927a7ea0 2024-04-21T08:54:47.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.172+0000 7f9ec77fe640 1 -- 172.21.15.28:0/2359991180 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ee00cee90 con 0x5555927a7ea0 2024-04-21T08:54:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.174+0000 7f9ec77fe640 1 -- 172.21.15.28:0/2359991180 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7f9ed80046e0 con 0x5555927a7ea0 2024-04-21T08:54:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.174+0000 7f9ef15be740 10 client.16818 fetch_fsmap finished waiting for FSMap version 27 2024-04-21T08:54:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.174+0000 7f9ef15be740 10 client.16818 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.174+0000 7f9ef15be740 20 client.16818 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.174+0000 7f9ef15be740 1 --2- 172.21.15.28:0/2359991180 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555928279e0 0x555592829e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.174+0000 7f9ef15be740 4 client.16818 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:54:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.174+0000 7f9ef15be740 1 -- 172.21.15.28:0/2359991180 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555592713a70 con 0x5555928279e0 2024-04-21T08:54:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.174+0000 7f9ee7fff640 1 --2- 172.21.15.28:0/2359991180 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555928279e0 0x555592829e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.175+0000 7f9ee7fff640 1 --2- 172.21.15.28:0/2359991180 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555928279e0 0x555592829e00 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=923b8660bafadc95 server_cookie=773398a7dc09db14 in_seq=0 out_seq=0 2024-04-21T08:54:47.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.175+0000 7f9ec77fe640 10 client.16818 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.523+0000 7fa162ffd640 1 -- 172.21.15.28:0/692513745 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562d9ebe93d0 con 0x562d9ed64da0 2024-04-21T08:54:47.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.523+0000 7fa162ffd640 10 client.26122 handle_command_reply: tid=0 2024-04-21T08:54:47.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.523+0000 7f9ec77fe640 1 -- 172.21.15.28:0/2359991180 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555592713a70 con 0x5555928279e0 2024-04-21T08:54:47.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.523+0000 7fa18cba9740 1 client.26122 shutdown 2024-04-21T08:54:47.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.523+0000 7f9ec77fe640 10 client.16818 handle_command_reply: tid=0 2024-04-21T08:54:47.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.524+0000 7f9ef15be740 1 client.16818 shutdown 2024-04-21T08:54:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.524+0000 7fa18cba9740 1 -- 172.21.15.28:0/692513745 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 msgr2=0x562d9ecf1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.524+0000 7fa18cba9740 1 --2- 172.21.15.28:0/692513745 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ecf1920 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa17402c250 tx=0x7fa17402cac0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.524+0000 7fa18cba9740 1 -- 172.21.15.28:0/692513745 shutdown_connections 2024-04-21T08:54:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.524+0000 7fa18cba9740 1 --2- 172.21.15.28:0/692513745 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562d9ed64da0 0x562d9ed67190 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.524+0000 7fa18cba9740 1 --2- 172.21.15.28:0/692513745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9ecee0e0 0x562d9ecf1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.524+0000 7fa18cba9740 1 --2- 172.21.15.28:0/692513745 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562d9ece58d0 0x562d9ecf1920 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.524+0000 7fa18cba9740 1 --2- 172.21.15.28:0/692513745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d9ece4b60 0x562d9ecf65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.524+0000 7fa18cba9740 1 -- 172.21.15.28:0/692513745 >> 172.21.15.28:0/692513745 conn(0x562d9ecabc90 msgr2=0x562d9ece92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.524+0000 7fa162ffd640 0 client.26122 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.524+0000 7fa18cba9740 1 -- 172.21.15.28:0/692513745 shutdown_connections 2024-04-21T08:54:47.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.524+0000 7fa18cba9740 1 -- 172.21.15.28:0/692513745 wait complete. 2024-04-21T08:54:47.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.524+0000 7fa18cba9740 20 client.26122 trim_cache size 0 max 16384 2024-04-21T08:54:47.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.524+0000 7f9ef15be740 1 -- 172.21.15.28:0/2359991180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 msgr2=0x5555927aaac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:47.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.524+0000 7f9ef15be740 1 --2- 172.21.15.28:0/2359991180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 0x5555927aaac0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800a740 tx=0x7f9ed800a5b0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.524+0000 7f9ef15be740 1 -- 172.21.15.28:0/2359991180 shutdown_connections 2024-04-21T08:54:47.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.524+0000 7f9ef15be740 1 --2- 172.21.15.28:0/2359991180 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555928279e0 0x555592829e00 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.524+0000 7f9ef15be740 1 --2- 172.21.15.28:0/2359991180 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555927b0a80 0x5555927ab000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.524+0000 7f9ef15be740 1 --2- 172.21.15.28:0/2359991180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555927a7ea0 0x5555927aaac0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.524+0000 7f9ef15be740 1 --2- 172.21.15.28:0/2359991180 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555927a7580 0x5555927aa580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.524+0000 7f9ef15be740 1 -- 172.21.15.28:0/2359991180 >> 172.21.15.28:0/2359991180 conn(0x5555927a1cb0 msgr2=0x5555927a0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:47.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.524+0000 7f9ec77fe640 0 client.16818 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:54:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:54:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:54:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:54:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.525+0000 7fa189ed9640 1 -- 172.21.15.28:0/291137286 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa16c0a3f20 msgr2=0x7fa16c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.525+0000 7fa189ed9640 1 --2- 172.21.15.28:0/291137286 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa16c0a3f20 0x7fa16c0a63e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00e200 tx=0x7fa17c00b040 comp rx=0 tx=0).stop 2024-04-21T08:54:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.525+0000 7fa189ed9640 1 -- 172.21.15.28:0/291137286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa18410eb90 msgr2=0x7fa1841abec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.525+0000 7fa189ed9640 1 --2- 172.21.15.28:0/291137286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa18410eb90 0x7fa1841abec0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa17800b410 tx=0x7fa17800b8d0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.525+0000 7fa189ed9640 1 -- 172.21.15.28:0/291137286 shutdown_connections 2024-04-21T08:54:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.525+0000 7fa189ed9640 1 --2- 172.21.15.28:0/291137286 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa16c0a3f20 0x7fa16c0a63e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.525+0000 7fa189ed9640 1 --2- 172.21.15.28:0/291137286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa18410eb90 0x7fa1841abec0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.525+0000 7fa189ed9640 1 --2- 172.21.15.28:0/291137286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa18410a0b0 0x7fa1841a7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.525+0000 7fa189ed9640 1 --2- 172.21.15.28:0/291137286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa184109790 0x7fa1841a7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.525+0000 7fa189ed9640 1 -- 172.21.15.28:0/291137286 >> 172.21.15.28:0/291137286 conn(0x7fa18406bd30 msgr2=0x7fa18406df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:47.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.525+0000 7f9ef15be740 1 -- 172.21.15.28:0/2359991180 shutdown_connections 2024-04-21T08:54:47.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.525+0000 7f9ef15be740 1 -- 172.21.15.28:0/2359991180 wait complete. 2024-04-21T08:54:47.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.525+0000 7f9ef15be740 20 client.16818 trim_cache size 0 max 16384 2024-04-21T08:54:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.526+0000 7fa189ed9640 1 -- 172.21.15.28:0/291137286 shutdown_connections 2024-04-21T08:54:47.529 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:54:47.529 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T08:54:47.529 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T08:54:47.529 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:54:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:54:47.526+0000 7fa189ed9640 1 -- 172.21.15.28:0/291137286 wait complete. 2024-04-21T08:54:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.526+0000 7f9eee8f2640 1 -- 172.21.15.28:0/448616300 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9ed00a3d50 msgr2=0x7f9ed00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.526+0000 7f9eee8f2640 1 --2- 172.21.15.28:0/448616300 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9ed00a3d50 0x7f9ed00a6210 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9edc009890 tx=0x7f9edc008040 comp rx=0 tx=0).stop 2024-04-21T08:54:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.526+0000 7f9eee8f2640 1 -- 172.21.15.28:0/448616300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ee810e620 msgr2=0x7f9ee80827f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.526+0000 7f9eee8f2640 1 --2- 172.21.15.28:0/448616300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ee810e620 0x7f9ee80827f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9ee001a440 tx=0x7f9ee00045e0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.527+0000 7f9eee8f2640 1 -- 172.21.15.28:0/448616300 shutdown_connections 2024-04-21T08:54:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.527+0000 7f9eee8f2640 1 --2- 172.21.15.28:0/448616300 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9ed00a3d50 0x7f9ed00a6210 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.527+0000 7f9eee8f2640 1 --2- 172.21.15.28:0/448616300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9ee810e620 0x7f9ee80827f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.527+0000 7f9eee8f2640 1 --2- 172.21.15.28:0/448616300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee8108f20 0x7f9ee81153d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.527+0000 7f9eee8f2640 1 --2- 172.21.15.28:0/448616300 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9ee8108560 0x7f9ee8114e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.527+0000 7f9eee8f2640 1 -- 172.21.15.28:0/448616300 >> 172.21.15.28:0/448616300 conn(0x7f9ee80764c0 msgr2=0x7f9ee8077c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:47.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.528+0000 7f9eee8f2640 1 -- 172.21.15.28:0/448616300 shutdown_connections 2024-04-21T08:54:47.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.529+0000 7f9eee8f2640 1 -- 172.21.15.28:0/448616300 wait complete. 2024-04-21T08:54:47.549 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:54:47.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:54:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.741+0000 7f48dc2b3640 1 -- 172.21.15.28:0/3125218584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48d410e980 msgr2=0x7f48d410ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.741+0000 7f48dc2b3640 1 --2- 172.21.15.28:0/3125218584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48d410e980 0x7f48d410ed60 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f48cc005fd0 tx=0x7f48cc02cce0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.742+0000 7f48dc2b3640 1 -- 172.21.15.28:0/3125218584 shutdown_connections 2024-04-21T08:54:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.742+0000 7f48dc2b3640 1 --2- 172.21.15.28:0/3125218584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48d4109910 0x7f48d41157d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.742+0000 7f48dc2b3640 1 --2- 172.21.15.28:0/3125218584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48d4108f50 0x7f48d41093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.742+0000 7f48dc2b3640 1 --2- 172.21.15.28:0/3125218584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48d410e980 0x7f48d410ed60 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.742+0000 7f48dc2b3640 1 -- 172.21.15.28:0/3125218584 >> 172.21.15.28:0/3125218584 conn(0x7f48d406e410 msgr2=0x7f48d40719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:47.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.742+0000 7f48dc2b3640 1 -- 172.21.15.28:0/3125218584 shutdown_connections 2024-04-21T08:54:47.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.742+0000 7f48dc2b3640 1 -- 172.21.15.28:0/3125218584 wait complete. 2024-04-21T08:54:47.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.743+0000 7f48dc2b3640 1 Processor -- start 2024-04-21T08:54:47.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.743+0000 7f48dc2b3640 1 -- start start 2024-04-21T08:54:47.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.743+0000 7f48dc2b3640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48d4108f50 0x7f48d41a3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.743+0000 7f48dc2b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48d4109910 0x7f48d41a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.743+0000 7f48dc2b3640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48d410e980 0x7f48d41a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.743+0000 7f48dc2b3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48d4118760 con 0x7f48d4109910 2024-04-21T08:54:47.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.743+0000 7f48dc2b3640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f48d41185e0 con 0x7f48d4108f50 2024-04-21T08:54:47.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.743+0000 7f48dc2b3640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f48d41188e0 con 0x7f48d410e980 2024-04-21T08:54:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.743+0000 7f48d984e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48d4109910 0x7f48d41a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.743+0000 7f48da04f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48d4108f50 0x7f48d41a3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.744+0000 7f48d984e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48d4109910 0x7f48d41a35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34152/0 (socket says 172.21.15.28:34152) 2024-04-21T08:54:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.744+0000 7f48da04f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48d4108f50 0x7f48d41a3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:48534/0 (socket says 172.21.15.28:48534) 2024-04-21T08:54:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.744+0000 7f48d984e640 1 -- 172.21.15.28:0/3777736593 learned_addr learned my addr 172.21.15.28:0/3777736593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.744+0000 7f48da850640 1 --2- 172.21.15.28:0/3777736593 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48d410e980 0x7f48d41a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.744+0000 7f48da850640 1 -- 172.21.15.28:0/3777736593 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48d4108f50 msgr2=0x7f48d41a3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.744+0000 7f48da850640 1 --2- 172.21.15.28:0/3777736593 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48d4108f50 0x7f48d41a3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.744+0000 7f48da850640 1 -- 172.21.15.28:0/3777736593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48d4109910 msgr2=0x7f48d41a35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.744+0000 7f48da850640 1 --2- 172.21.15.28:0/3777736593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48d4109910 0x7f48d41a35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.744+0000 7f48da850640 1 -- 172.21.15.28:0/3777736593 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d41a8080 con 0x7f48d410e980 2024-04-21T08:54:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.744+0000 7f48d984e640 1 --2- 172.21.15.28:0/3777736593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48d4109910 0x7f48d41a35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.744+0000 7f48da850640 1 --2- 172.21.15.28:0/3777736593 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48d410e980 0x7f48d41a7950 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f48d000c710 tx=0x7f48d000cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.745+0000 7f48cb7fe640 1 -- 172.21.15.28:0/3777736593 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d0004220 con 0x7f48d410e980 2024-04-21T08:54:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.745+0000 7f48dc2b3640 1 -- 172.21.15.28:0/3777736593 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d41a8370 con 0x7f48d410e980 2024-04-21T08:54:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.745+0000 7f48cb7fe640 1 -- 172.21.15.28:0/3777736593 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48d000ce20 con 0x7f48d410e980 2024-04-21T08:54:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.745+0000 7f48dc2b3640 1 -- 172.21.15.28:0/3777736593 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d41afc00 con 0x7f48d410e980 2024-04-21T08:54:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.745+0000 7f48cb7fe640 1 -- 172.21.15.28:0/3777736593 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d0003d50 con 0x7f48d410e980 2024-04-21T08:54:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.746+0000 7f48dc2b3640 1 -- 172.21.15.28:0/3777736593 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48d4068de0 con 0x7f48d410e980 2024-04-21T08:54:47.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.749+0000 7f48cb7fe640 1 -- 172.21.15.28:0/3777736593 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f48d00079b0 con 0x7f48d410e980 2024-04-21T08:54:47.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.750+0000 7f48cb7fe640 1 --2- 172.21.15.28:0/3777736593 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48c00a3dd0 0x7f48c00a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:47.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.750+0000 7f48cb7fe640 1 -- 172.21.15.28:0/3777736593 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f48d00d0770 con 0x7f48d410e980 2024-04-21T08:54:47.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.751+0000 7f48da04f640 1 --2- 172.21.15.28:0/3777736593 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48c00a3dd0 0x7f48c00a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:47.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.753+0000 7f48da04f640 1 --2- 172.21.15.28:0/3777736593 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48c00a3dd0 0x7f48c00a6290 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f48cc02d2f0 tx=0x7f48cc02f040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:47.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:47.757+0000 7f48cb7fe640 1 -- 172.21.15.28:0/3777736593 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f48d0043020 con 0x7f48d410e980 2024-04-21T08:54:48.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.213+0000 7f48dc2b3640 1 -- 172.21.15.28:0/3777736593 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48d41afe90 con 0x7f48d410e980 2024-04-21T08:54:48.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.214+0000 7f48cb7fe640 1 -- 172.21.15.28:0/3777736593 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7f48d0014030 con 0x7f48d410e980 2024-04-21T08:54:48.214 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:54:48.214 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:54:48.214 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T08:54:48.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.218+0000 7f48dc2b3640 1 -- 172.21.15.28:0/3777736593 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48c00a3dd0 msgr2=0x7f48c00a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.218+0000 7f48dc2b3640 1 --2- 172.21.15.28:0/3777736593 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48c00a3dd0 0x7f48c00a6290 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f48cc02d2f0 tx=0x7f48cc02f040 comp rx=0 tx=0).stop 2024-04-21T08:54:48.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.218+0000 7f48dc2b3640 1 -- 172.21.15.28:0/3777736593 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48d410e980 msgr2=0x7f48d41a7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.218+0000 7f48dc2b3640 1 --2- 172.21.15.28:0/3777736593 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48d410e980 0x7f48d41a7950 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f48d000c710 tx=0x7f48d000cbd0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.218+0000 7f48dc2b3640 1 -- 172.21.15.28:0/3777736593 shutdown_connections 2024-04-21T08:54:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.218+0000 7f48dc2b3640 1 --2- 172.21.15.28:0/3777736593 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48c00a3dd0 0x7f48c00a6290 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.218+0000 7f48dc2b3640 1 --2- 172.21.15.28:0/3777736593 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48d410e980 0x7f48d41a7950 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.218+0000 7f48dc2b3640 1 --2- 172.21.15.28:0/3777736593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48d4109910 0x7f48d41a35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.218+0000 7f48dc2b3640 1 --2- 172.21.15.28:0/3777736593 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48d4108f50 0x7f48d41a3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.218+0000 7f48dc2b3640 1 -- 172.21.15.28:0/3777736593 >> 172.21.15.28:0/3777736593 conn(0x7f48d406e410 msgr2=0x7f48d4071c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.219+0000 7f48dc2b3640 1 -- 172.21.15.28:0/3777736593 shutdown_connections 2024-04-21T08:54:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.219+0000 7f48dc2b3640 1 -- 172.21.15.28:0/3777736593 wait complete. 2024-04-21T08:54:48.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:54:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.425+0000 7f8414854640 1 -- 172.21.15.28:0/321978181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8408095f80 msgr2=0x7f8408096360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.425+0000 7f8414854640 1 --2- 172.21.15.28:0/321978181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8408095f80 0x7f8408096360 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f8404009900 tx=0x7f8404030060 comp rx=0 tx=0).stop 2024-04-21T08:54:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.426+0000 7f8414854640 1 -- 172.21.15.28:0/321978181 shutdown_connections 2024-04-21T08:54:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.426+0000 7f8414854640 1 --2- 172.21.15.28:0/321978181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f840809b050 0x7f84080a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.427+0000 7f8414854640 1 --2- 172.21.15.28:0/321978181 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8408096930 0x7f840809a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.427+0000 7f8414854640 1 --2- 172.21.15.28:0/321978181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8408095f80 0x7f8408096360 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.427+0000 7f8414854640 1 -- 172.21.15.28:0/321978181 >> 172.21.15.28:0/321978181 conn(0x7f840800ce50 msgr2=0x7f840800d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.431+0000 7f8414854640 1 -- 172.21.15.28:0/321978181 shutdown_connections 2024-04-21T08:54:48.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.432+0000 7f8414854640 1 -- 172.21.15.28:0/321978181 wait complete. 2024-04-21T08:54:48.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.433+0000 7f8414854640 1 Processor -- start 2024-04-21T08:54:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.436+0000 7f8414854640 1 -- start start 2024-04-21T08:54:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.436+0000 7f8414854640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8408095f80 0x7f8408138910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.436+0000 7f8414854640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f840809b050 0x7f8408138e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.436+0000 7f8414854640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8408132ac0 0x7f8408132f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.436+0000 7f8414854640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84080a3ed0 con 0x7f8408095f80 2024-04-21T08:54:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.436+0000 7f8414854640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f84080a3d50 con 0x7f8408132ac0 2024-04-21T08:54:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.436+0000 7f8414854640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f84080a4050 con 0x7f840809b050 2024-04-21T08:54:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.436+0000 7f840f59e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8408132ac0 0x7f8408132f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.437+0000 7f840f59e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8408132ac0 0x7f8408132f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:48556/0 (socket says 172.21.15.28:48556) 2024-04-21T08:54:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.437+0000 7f840f59e640 1 -- 172.21.15.28:0/1418721750 learned_addr learned my addr 172.21.15.28:0/1418721750 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.437+0000 7f840f59e640 1 -- 172.21.15.28:0/1418721750 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f840809b050 msgr2=0x7f8408138e50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:54:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.437+0000 7f840f59e640 1 --2- 172.21.15.28:0/1418721750 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f840809b050 0x7f8408138e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.437+0000 7f840f59e640 1 -- 172.21.15.28:0/1418721750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8408095f80 msgr2=0x7f8408138910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.437+0000 7f840f59e640 1 --2- 172.21.15.28:0/1418721750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8408095f80 0x7f8408138910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.437+0000 7f840f59e640 1 -- 172.21.15.28:0/1418721750 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84081335e0 con 0x7f8408132ac0 2024-04-21T08:54:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.438+0000 7f840f59e640 1 --2- 172.21.15.28:0/1418721750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8408132ac0 0x7f8408132f70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f840000b3f0 tx=0x7f840000b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.441+0000 7f83f7fff640 1 -- 172.21.15.28:0/1418721750 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400015020 con 0x7f8408132ac0 2024-04-21T08:54:48.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.441+0000 7f83f7fff640 1 -- 172.21.15.28:0/1418721750 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8400011ce0 con 0x7f8408132ac0 2024-04-21T08:54:48.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.442+0000 7f83f7fff640 1 -- 172.21.15.28:0/1418721750 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400010dd0 con 0x7f8408132ac0 2024-04-21T08:54:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.445+0000 7f8414854640 1 -- 172.21.15.28:0/1418721750 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84081338d0 con 0x7f8408132ac0 2024-04-21T08:54:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.445+0000 7f8414854640 1 -- 172.21.15.28:0/1418721750 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f840812a4e0 con 0x7f8408132ac0 2024-04-21T08:54:48.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.450+0000 7f83f7fff640 1 -- 172.21.15.28:0/1418721750 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8400018020 con 0x7f8408132ac0 2024-04-21T08:54:48.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.451+0000 7f8417c47740 1 Processor -- start 2024-04-21T08:54:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.451+0000 7f8417c47740 1 -- start start 2024-04-21T08:54:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.451+0000 7f8417c47740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e036d0 0x561d98e03ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.451+0000 7f8417c47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 0x561d98e0c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.451+0000 7f8417c47740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e0cc00 0x561d98e0f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.451+0000 7f8417c47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561d98d681f0 con 0x561d98e04080 2024-04-21T08:54:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.451+0000 7f8417c47740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561d98d054b0 con 0x561d98e0cc00 2024-04-21T08:54:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.451+0000 7f8417c47740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x561d98d641b0 con 0x561d98e036d0 2024-04-21T08:54:48.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.452+0000 7f83f7fff640 1 --2- 172.21.15.28:0/1418721750 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f83f80a3d80 0x7f83f80a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.453+0000 7f83f7fff640 1 -- 172.21.15.28:0/1418721750 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f84000d0150 con 0x7f8408132ac0 2024-04-21T08:54:48.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.453+0000 7f840ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 0x561d98e0c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.453+0000 7f840ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 0x561d98e0c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34184/0 (socket says 172.21.15.28:34184) 2024-04-21T08:54:48.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.453+0000 7f840ed9d640 1 -- 172.21.15.28:0/1111078471 learned_addr learned my addr 172.21.15.28:0/1111078471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:48.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.453+0000 7f840ed9d640 1 -- 172.21.15.28:0/1111078471 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e036d0 msgr2=0x561d98e03ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:54:48.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.453+0000 7f840ed9d640 1 --2- 172.21.15.28:0/1111078471 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e036d0 0x561d98e03ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.454+0000 7f840ed9d640 1 -- 172.21.15.28:0/1111078471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e0cc00 msgr2=0x561d98e0f050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:54:48.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.454+0000 7f840ed9d640 1 --2- 172.21.15.28:0/1111078471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e0cc00 0x561d98e0f050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.454+0000 7f840ed9d640 1 -- 172.21.15.28:0/1111078471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d98cf71b0 con 0x561d98e04080 2024-04-21T08:54:48.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.454+0000 7f840ed9d640 1 --2- 172.21.15.28:0/1111078471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 0x561d98e0c630 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f840403a040 tx=0x7f84040384f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4544582d7ee2134d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.455+0000 7f83f57fa640 1 -- 172.21.15.28:0/1111078471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8404046070 con 0x561d98e04080 2024-04-21T08:54:48.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.455+0000 7f83f57fa640 1 -- 172.21.15.28:0/1111078471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8404010520 con 0x561d98e04080 2024-04-21T08:54:48.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.455+0000 7f83f57fa640 1 -- 172.21.15.28:0/1111078471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8404046070 con 0x561d98e04080 2024-04-21T08:54:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.457+0000 7f840f59e640 1 --2- 172.21.15.28:0/1418721750 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f83f80a3d80 0x7f83f80a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.457+0000 7f8417c47740 1 -- 172.21.15.28:0/1111078471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 msgr2=0x561d98e0c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.457+0000 7f8417c47740 1 --2- 172.21.15.28:0/1111078471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 0x561d98e0c630 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f840403a040 tx=0x7f84040384f0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.457+0000 7f8417c47740 1 -- 172.21.15.28:0/1111078471 shutdown_connections 2024-04-21T08:54:48.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.457+0000 7f8417c47740 1 --2- 172.21.15.28:0/1111078471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e0cc00 0x561d98e0f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.457+0000 7f8417c47740 1 --2- 172.21.15.28:0/1111078471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 0x561d98e0c630 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.457+0000 7f8417c47740 1 --2- 172.21.15.28:0/1111078471 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e036d0 0x561d98e03ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.457+0000 7f8417c47740 1 -- 172.21.15.28:0/1111078471 >> 172.21.15.28:0/1111078471 conn(0x561d98dfdda0 msgr2=0x561d98dfe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.457+0000 7f840f59e640 1 --2- 172.21.15.28:0/1418721750 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f83f80a3d80 0x7f83f80a6240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f84000982c0 tx=0x7f84000047d0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.464+0000 7f8417c47740 1 -- 172.21.15.28:0/1111078471 shutdown_connections 2024-04-21T08:54:48.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.464+0000 7f8417c47740 1 -- 172.21.15.28:0/1111078471 wait complete. 2024-04-21T08:54:48.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.464+0000 7f8417c47740 1 -- start start 2024-04-21T08:54:48.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.465+0000 7f8417c47740 1 Processor -- start 2024-04-21T08:54:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.465+0000 7f8417c47740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e036d0 0x561d98e0c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.465+0000 7f8417c47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 0x561d98e09820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.465+0000 7f8417c47740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e0cc00 0x561d98e09d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.465+0000 7f8417c47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561d98e0fb10 con 0x561d98e04080 2024-04-21T08:54:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.465+0000 7f8417c47740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561d98d21500 con 0x561d98e0cc00 2024-04-21T08:54:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.465+0000 7f8417c47740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x561d98e0fc90 con 0x561d98e036d0 2024-04-21T08:54:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.465+0000 7f8417c47740 10 client.0 _ll_register_callbacks cb 0x561d98cefb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.466+0000 7f840e59c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e036d0 0x561d98e0c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.466+0000 7f840e59c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e036d0 0x561d98e0c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:60698/0 (socket says 172.21.15.28:60698) 2024-04-21T08:54:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.466+0000 7f840e59c640 1 -- 172.21.15.28:0/1138784713 learned_addr learned my addr 172.21.15.28:0/1138784713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.466+0000 7f840e59c640 1 --2- 172.21.15.28:0/1138784713 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e0cc00 0x561d98e09d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.466+0000 7f840ed9d640 1 --2- 172.21.15.28:0/1138784713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 0x561d98e09820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.466+0000 7f840e59c640 1 -- 172.21.15.28:0/1138784713 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e0cc00 msgr2=0x561d98e09d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.466+0000 7f840e59c640 1 --2- 172.21.15.28:0/1138784713 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e0cc00 0x561d98e09d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.467+0000 7f840e59c640 1 -- 172.21.15.28:0/1138784713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 msgr2=0x561d98e09820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.467+0000 7f840e59c640 1 --2- 172.21.15.28:0/1138784713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 0x561d98e09820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.467+0000 7f840e59c640 1 -- 172.21.15.28:0/1138784713 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d98a7a3f0 con 0x561d98e036d0 2024-04-21T08:54:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.467+0000 7f83f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:54:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.467+0000 7f840e59c640 1 --2- 172.21.15.28:0/1138784713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e036d0 0x561d98e0c490 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f84100676b0 tx=0x7f84100682e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=351f1ccf46bd3c45 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.468+0000 7f83f2ff5640 1 -- 172.21.15.28:0/1138784713 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8410077dc0 con 0x561d98e036d0 2024-04-21T08:54:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.468+0000 7f83f2ff5640 1 -- 172.21.15.28:0/1138784713 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f8410088060 con 0x561d98e036d0 2024-04-21T08:54:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.468+0000 7f8417c47740 1 -- 172.21.15.28:0/1138784713 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561d98d07930 con 0x561d98e036d0 2024-04-21T08:54:48.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.471+0000 7f83f2ff5640 1 -- 172.21.15.28:0/1138784713 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8410077dc0 con 0x561d98e036d0 2024-04-21T08:54:48.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.473+0000 7f8417c47740 10 client.26131 fetch_fsmap learned FSMap version 28 2024-04-21T08:54:48.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.473+0000 7f8417c47740 1 -- 172.21.15.28:0/1138784713 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561d98b447d0 con 0x561d98e036d0 2024-04-21T08:54:48.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.477+0000 7f83f2ff5640 1 -- 172.21.15.28:0/1138784713 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561d98d07930 con 0x561d98e036d0 2024-04-21T08:54:48.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.477+0000 7f83f2ff5640 1 -- 172.21.15.28:0/1138784713 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f841007e440 con 0x561d98e036d0 2024-04-21T08:54:48.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.478+0000 7f8417c47740 10 client.26131 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:54:48.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.478+0000 7f8417c47740 10 client.26131 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:54:48.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.478+0000 7f8417c47740 20 client.26131 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:48.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.478+0000 7f8417c47740 1 --2- 172.21.15.28:0/1138784713 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x561d98e1cd00 0x561d98e1f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.479+0000 7f8417c47740 4 client.26131 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:48.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.479+0000 7f8417c47740 1 -- 172.21.15.28:0/1138784713 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d98d75c10 con 0x561d98e1cd00 2024-04-21T08:54:48.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.480+0000 7f840ed9d640 1 --2- 172.21.15.28:0/1138784713 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x561d98e1cd00 0x561d98e1f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.481+0000 7f83f2ff5640 10 client.26131 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:54:48.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.481+0000 7f840ed9d640 1 --2- 172.21.15.28:0/1138784713 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x561d98e1cd00 0x561d98e1f130 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f079d19f597eff69 server_cookie=c4b701ef02e7365b in_seq=0 out_seq=0 2024-04-21T08:54:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.483+0000 7f83f2ff5640 1 -- 172.21.15.28:0/1138784713 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561d98d75c10 con 0x561d98e1cd00 2024-04-21T08:54:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.483+0000 7f83f2ff5640 10 client.26131 handle_command_reply: tid=0 2024-04-21T08:54:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.483+0000 7f8417c47740 1 client.26131 shutdown 2024-04-21T08:54:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.483+0000 7f8417c47740 1 -- 172.21.15.28:0/1138784713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e036d0 msgr2=0x561d98e0c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.483+0000 7f8417c47740 1 --2- 172.21.15.28:0/1138784713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e036d0 0x561d98e0c490 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f84100676b0 tx=0x7f84100682e0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.484+0000 7f8417c47740 1 -- 172.21.15.28:0/1138784713 shutdown_connections 2024-04-21T08:54:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.484+0000 7f8417c47740 1 --2- 172.21.15.28:0/1138784713 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x561d98e1cd00 0x561d98e1f130 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.484+0000 7f83f2ff5640 0 client.26131 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:54:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.484+0000 7f8417c47740 1 --2- 172.21.15.28:0/1138784713 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e0cc00 0x561d98e09d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.484+0000 7f8417c47740 1 --2- 172.21.15.28:0/1138784713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 0x561d98e09820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.484+0000 7f8417c47740 1 --2- 172.21.15.28:0/1138784713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e036d0 0x561d98e0c490 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.484+0000 7f8417c47740 1 -- 172.21.15.28:0/1138784713 >> 172.21.15.28:0/1138784713 conn(0x561d98dfdda0 msgr2=0x561d98dffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.486+0000 7f8417c47740 1 -- 172.21.15.28:0/1138784713 shutdown_connections 2024-04-21T08:54:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.486+0000 7f8417c47740 1 -- 172.21.15.28:0/1138784713 wait complete. 2024-04-21T08:54:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.487+0000 7f8417c47740 20 client.26131 trim_cache size 0 max 16384 2024-04-21T08:54:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.513+0000 7f8417c47740 1 Processor -- start 2024-04-21T08:54:48.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.513+0000 7f8417c47740 1 -- start start 2024-04-21T08:54:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f8417c47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e036d0 0x561d98e18a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f840e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e036d0 0x561d98e18a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f840e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e036d0 0x561d98e18a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34194/0 (socket says 172.21.15.28:34194) 2024-04-21T08:54:48.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f8417c47740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e04080 0x561d98e04bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f840e59c640 1 -- 172.21.15.28:0/3528452246 learned_addr learned my addr 172.21.15.28:0/3528452246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:48.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f8417c47740 1 --2- 172.21.15.28:0/3528452246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e0cc00 0x561d98e0bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f8417c47740 1 -- 172.21.15.28:0/3528452246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84081546f0 con 0x561d98e036d0 2024-04-21T08:54:48.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f8417c47740 1 -- 172.21.15.28:0/3528452246 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8408151fc0 con 0x561d98e0cc00 2024-04-21T08:54:48.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f8417c47740 1 -- 172.21.15.28:0/3528452246 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8408097450 con 0x561d98e04080 2024-04-21T08:54:48.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f840ed9d640 1 --2- 172.21.15.28:0/3528452246 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e04080 0x561d98e04bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f840e59c640 1 --2- 172.21.15.28:0/3528452246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e0cc00 0x561d98e0bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f840e59c640 1 -- 172.21.15.28:0/3528452246 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e04080 msgr2=0x561d98e04bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f840e59c640 1 --2- 172.21.15.28:0/3528452246 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e04080 0x561d98e04bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f840e59c640 1 -- 172.21.15.28:0/3528452246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e0cc00 msgr2=0x561d98e0bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f840e59c640 1 --2- 172.21.15.28:0/3528452246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e0cc00 0x561d98e0bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.514+0000 7f840e59c640 1 -- 172.21.15.28:0/3528452246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d98dfdbd0 con 0x561d98e036d0 2024-04-21T08:54:48.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.515+0000 7f840e59c640 1 --2- 172.21.15.28:0/3528452246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e036d0 0x561d98e18a60 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f8410067ac0 tx=0x7f84100676e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=274608a430aca0f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.515+0000 7f83f4ff9640 1 -- 172.21.15.28:0/3528452246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841007e440 con 0x561d98e036d0 2024-04-21T08:54:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.515+0000 7f83f4ff9640 1 -- 172.21.15.28:0/3528452246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8410075040 con 0x561d98e036d0 2024-04-21T08:54:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.515+0000 7f83f4ff9640 1 -- 172.21.15.28:0/3528452246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841006a590 con 0x561d98e036d0 2024-04-21T08:54:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.516+0000 7f8417c47740 1 -- 172.21.15.28:0/3528452246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e036d0 msgr2=0x561d98e18a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.516+0000 7f8417c47740 1 --2- 172.21.15.28:0/3528452246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e036d0 0x561d98e18a60 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f8410067ac0 tx=0x7f84100676e0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.517+0000 7f8417c47740 1 -- 172.21.15.28:0/3528452246 shutdown_connections 2024-04-21T08:54:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.517+0000 7f8417c47740 1 --2- 172.21.15.28:0/3528452246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e0cc00 0x561d98e0bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.517+0000 7f8417c47740 1 --2- 172.21.15.28:0/3528452246 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e04080 0x561d98e04bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.517+0000 7f8417c47740 1 --2- 172.21.15.28:0/3528452246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e036d0 0x561d98e18a60 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.517+0000 7f8417c47740 1 -- 172.21.15.28:0/3528452246 >> 172.21.15.28:0/3528452246 conn(0x561d98dfdda0 msgr2=0x561d98e044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.517+0000 7f8417c47740 1 -- 172.21.15.28:0/3528452246 shutdown_connections 2024-04-21T08:54:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.517+0000 7f8417c47740 1 -- 172.21.15.28:0/3528452246 wait complete. 2024-04-21T08:54:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.517+0000 7f8417c47740 1 -- start start 2024-04-21T08:54:48.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.517+0000 7f8417c47740 1 Processor -- start 2024-04-21T08:54:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f8417c47740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e036d0 0x561d98e153e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f8417c47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 0x561d98e10750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f8417c47740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e0cc00 0x561d98e10cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f8417c47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8408132680 con 0x561d98e04080 2024-04-21T08:54:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f840e59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e036d0 0x561d98e153e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f840e59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e036d0 0x561d98e153e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:48590/0 (socket says 172.21.15.28:48590) 2024-04-21T08:54:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f840e59c640 1 -- 172.21.15.28:0/4030741129 learned_addr learned my addr 172.21.15.28:0/4030741129 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f840ed9d640 1 --2- 172.21.15.28:0/4030741129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 0x561d98e10750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f8417c47740 1 -- 172.21.15.28:0/4030741129 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f84081518f0 con 0x561d98e036d0 2024-04-21T08:54:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f8417c47740 1 -- 172.21.15.28:0/4030741129 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8408150b50 con 0x561d98e0cc00 2024-04-21T08:54:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f8417c47740 10 client.0 _ll_register_callbacks cb 0x7f8410080c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f840e59c640 1 --2- 172.21.15.28:0/4030741129 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e0cc00 0x561d98e10cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f840e59c640 1 -- 172.21.15.28:0/4030741129 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e0cc00 msgr2=0x561d98e10cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f840e59c640 1 --2- 172.21.15.28:0/4030741129 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e0cc00 0x561d98e10cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f840e59c640 1 -- 172.21.15.28:0/4030741129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 msgr2=0x561d98e10750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f840e59c640 1 --2- 172.21.15.28:0/4030741129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 0x561d98e10750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.518+0000 7f840e59c640 1 -- 172.21.15.28:0/4030741129 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d98e11520 con 0x561d98e036d0 2024-04-21T08:54:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.519+0000 7f840e59c640 1 --2- 172.21.15.28:0/4030741129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e036d0 0x561d98e153e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f84100515e0 tx=0x7f8410067960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b6bbd2caef32433 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.519+0000 7f83f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:54:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.519+0000 7f83f2ff5640 1 -- 172.21.15.28:0/4030741129 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841009a670 con 0x561d98e036d0 2024-04-21T08:54:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.519+0000 7f83f2ff5640 1 -- 172.21.15.28:0/4030741129 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f841009a810 con 0x561d98e036d0 2024-04-21T08:54:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.519+0000 7f8417c47740 1 -- 172.21.15.28:0/4030741129 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8410075040 con 0x561d98e036d0 2024-04-21T08:54:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.519+0000 7f83f2ff5640 1 -- 172.21.15.28:0/4030741129 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8410085020 con 0x561d98e036d0 2024-04-21T08:54:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.519+0000 7f8417c47740 10 client.26117 fetch_fsmap learned FSMap version 28 2024-04-21T08:54:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.519+0000 7f8417c47740 1 -- 172.21.15.28:0/4030741129 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561d98e83340 con 0x561d98e036d0 2024-04-21T08:54:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.519+0000 7f83f2ff5640 1 -- 172.21.15.28:0/4030741129 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8410075040 con 0x561d98e036d0 2024-04-21T08:54:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.520+0000 7f83f2ff5640 1 -- 172.21.15.28:0/4030741129 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f8410081810 con 0x561d98e036d0 2024-04-21T08:54:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.520+0000 7f8417c47740 10 client.26117 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:54:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.520+0000 7f8417c47740 10 client.26117 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:54:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.520+0000 7f8417c47740 20 client.26117 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.520+0000 7f8417c47740 1 --2- 172.21.15.28:0/4030741129 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x561d98e839b0 0x561d98e85da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.520+0000 7f8417c47740 4 client.26117 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:54:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.520+0000 7f8417c47740 1 -- 172.21.15.28:0/4030741129 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561d98d6fa80 con 0x561d98e839b0 2024-04-21T08:54:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.520+0000 7f840ed9d640 1 --2- 172.21.15.28:0/4030741129 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x561d98e839b0 0x561d98e85da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.520+0000 7f83f2ff5640 10 client.26117 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:54:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.520+0000 7f840ed9d640 1 --2- 172.21.15.28:0/4030741129 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x561d98e839b0 0x561d98e85da0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c637d17b5c57da5d server_cookie=ae80208f01b55685 in_seq=0 out_seq=0 2024-04-21T08:54:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.520+0000 7f83f2ff5640 1 -- 172.21.15.28:0/4030741129 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561d98d6fa80 con 0x561d98e839b0 2024-04-21T08:54:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.520+0000 7f83f2ff5640 10 client.26117 handle_command_reply: tid=0 2024-04-21T08:54:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.520+0000 7f8417c47740 1 client.26117 shutdown 2024-04-21T08:54:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.521+0000 7f8417c47740 1 -- 172.21.15.28:0/4030741129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e036d0 msgr2=0x561d98e153e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.521+0000 7f8417c47740 1 --2- 172.21.15.28:0/4030741129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e036d0 0x561d98e153e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f84100515e0 tx=0x7f8410067960 comp rx=0 tx=0).stop 2024-04-21T08:54:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.521+0000 7f8417c47740 1 -- 172.21.15.28:0/4030741129 shutdown_connections 2024-04-21T08:54:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.521+0000 7f8417c47740 1 --2- 172.21.15.28:0/4030741129 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x561d98e839b0 0x561d98e85da0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.521+0000 7f83f2ff5640 0 client.26117 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:54:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.521+0000 7f8417c47740 1 --2- 172.21.15.28:0/4030741129 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x561d98e0cc00 0x561d98e10cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.521+0000 7f8417c47740 1 --2- 172.21.15.28:0/4030741129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d98e04080 0x561d98e10750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.521+0000 7f8417c47740 1 --2- 172.21.15.28:0/4030741129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561d98e036d0 0x561d98e153e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.521+0000 7f8417c47740 1 -- 172.21.15.28:0/4030741129 >> 172.21.15.28:0/4030741129 conn(0x561d98dfdda0 msgr2=0x561d98e04d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.521+0000 7f8417c47740 1 -- 172.21.15.28:0/4030741129 shutdown_connections 2024-04-21T08:54:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.521+0000 7f8417c47740 1 -- 172.21.15.28:0/4030741129 wait complete. 2024-04-21T08:54:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.521+0000 7f8417c47740 20 client.26117 trim_cache size 0 max 16384 2024-04-21T08:54:48.528 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:54:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.522+0000 7f8414854640 1 -- 172.21.15.28:0/1418721750 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f83f80a3d80 msgr2=0x7f83f80a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.522+0000 7f8414854640 1 --2- 172.21.15.28:0/1418721750 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f83f80a3d80 0x7f83f80a6240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f84000982c0 tx=0x7f84000047d0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.522+0000 7f8414854640 1 -- 172.21.15.28:0/1418721750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8408132ac0 msgr2=0x7f8408132f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.522+0000 7f8414854640 1 --2- 172.21.15.28:0/1418721750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8408132ac0 0x7f8408132f70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f840000b3f0 tx=0x7f840000b8b0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.522+0000 7f8414854640 1 -- 172.21.15.28:0/1418721750 shutdown_connections 2024-04-21T08:54:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.522+0000 7f8414854640 1 --2- 172.21.15.28:0/1418721750 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f83f80a3d80 0x7f83f80a6240 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.522+0000 7f8414854640 1 --2- 172.21.15.28:0/1418721750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8408132ac0 0x7f8408132f70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.522+0000 7f8414854640 1 --2- 172.21.15.28:0/1418721750 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f840809b050 0x7f8408138e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.522+0000 7f8414854640 1 --2- 172.21.15.28:0/1418721750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8408095f80 0x7f8408138910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.522+0000 7f8414854640 1 -- 172.21.15.28:0/1418721750 >> 172.21.15.28:0/1418721750 conn(0x7f840800ce50 msgr2=0x7f8408093fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.522+0000 7f8414854640 1 -- 172.21.15.28:0/1418721750 shutdown_connections 2024-04-21T08:54:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.522+0000 7f8414854640 1 -- 172.21.15.28:0/1418721750 wait complete. 2024-04-21T08:54:48.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:54:48.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:48 smithi028 ceph-mon[28003]: pgmap v111: 129 pgs: 129 active+clean; 14 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 36 MiB/s wr, 6.10k op/s 2024-04-21T08:54:48.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:48 smithi028 ceph-mon[28003]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:54:48.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:48 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3777736593' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:48.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:48 smithi152 ceph-mon[31401]: pgmap v111: 129 pgs: 129 active+clean; 14 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 36 MiB/s wr, 6.10k op/s 2024-04-21T08:54:48.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:48 smithi152 ceph-mon[31401]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:54:48.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:48 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3777736593' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:48.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:48 smithi171 ceph-mon[31358]: pgmap v111: 129 pgs: 129 active+clean; 14 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 36 MiB/s wr, 6.10k op/s 2024-04-21T08:54:48.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:48 smithi171 ceph-mon[31358]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-21T08:54:48.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:48 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3777736593' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:48.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.700+0000 7f8b7677e640 1 -- 172.21.15.28:0/494610422 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b70106160 msgr2=0x7f8b7010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.700+0000 7f8b7677e640 1 --2- 172.21.15.28:0/494610422 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b70106160 0x7f8b7010dd30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8b68009ca0 tx=0x7f8b6802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.700+0000 7f8b7677e640 1 -- 172.21.15.28:0/494610422 shutdown_connections 2024-04-21T08:54:48.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.700+0000 7f8b7677e640 1 --2- 172.21.15.28:0/494610422 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b70106160 0x7f8b7010dd30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.700+0000 7f8b7677e640 1 --2- 172.21.15.28:0/494610422 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b70101a40 0x7f8b70105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.700+0000 7f8b7677e640 1 --2- 172.21.15.28:0/494610422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70101090 0x7f8b70101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.700+0000 7f8b7677e640 1 -- 172.21.15.28:0/494610422 >> 172.21.15.28:0/494610422 conn(0x7f8b70077f50 msgr2=0x7f8b70078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.700+0000 7f8b7677e640 1 -- 172.21.15.28:0/494610422 shutdown_connections 2024-04-21T08:54:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.700+0000 7f8b7677e640 1 -- 172.21.15.28:0/494610422 wait complete. 2024-04-21T08:54:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.701+0000 7f8b7677e640 1 Processor -- start 2024-04-21T08:54:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.701+0000 7f8b7677e640 1 -- start start 2024-04-21T08:54:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.701+0000 7f8b7677e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70101090 0x7f8b70198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.701+0000 7f8b7677e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b70106160 0x7f8b70198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.701+0000 7f8b7677e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b7019cff0 0x7f8b7019d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.701+0000 7f8b7677e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b701121d0 con 0x7f8b70101090 2024-04-21T08:54:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.701+0000 7f8b7677e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8b70112050 con 0x7f8b70106160 2024-04-21T08:54:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.701+0000 7f8b7677e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8b70112350 con 0x7f8b7019cff0 2024-04-21T08:54:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.701+0000 7f8b6ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70101090 0x7f8b70198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.701+0000 7f8b6f7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b70106160 0x7f8b70198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.701+0000 7f8b74d1b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b7019cff0 0x7f8b7019d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.701+0000 7f8b6ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70101090 0x7f8b70198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34230/0 (socket says 172.21.15.28:34230) 2024-04-21T08:54:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.701+0000 7f8b6f7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b70106160 0x7f8b70198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:48610/0 (socket says 172.21.15.28:48610) 2024-04-21T08:54:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.701+0000 7f8b6ffff640 1 -- 172.21.15.28:0/1546785516 learned_addr learned my addr 172.21.15.28:0/1546785516 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.702+0000 7f8b6f7fe640 1 -- 172.21.15.28:0/1546785516 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b7019cff0 msgr2=0x7f8b7019d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.702+0000 7f8b6f7fe640 1 --2- 172.21.15.28:0/1546785516 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b7019cff0 0x7f8b7019d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.702+0000 7f8b6f7fe640 1 -- 172.21.15.28:0/1546785516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70101090 msgr2=0x7f8b70198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.702+0000 7f8b6f7fe640 1 --2- 172.21.15.28:0/1546785516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70101090 0x7f8b70198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.702+0000 7f8b6f7fe640 1 -- 172.21.15.28:0/1546785516 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b7019dae0 con 0x7f8b70106160 2024-04-21T08:54:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.702+0000 7f8b74d1b640 1 --2- 172.21.15.28:0/1546785516 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b7019cff0 0x7f8b7019d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.702+0000 7f8b6f7fe640 1 --2- 172.21.15.28:0/1546785516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b70106160 0x7f8b70198c70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8b6400e9c0 tx=0x7f8b6400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.702+0000 7f8b6d7fa640 1 -- 172.21.15.28:0/1546785516 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6400ccc0 con 0x7f8b70106160 2024-04-21T08:54:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.702+0000 7f8b7677e640 1 -- 172.21.15.28:0/1546785516 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b7019ddd0 con 0x7f8b70106160 2024-04-21T08:54:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.702+0000 7f8b6d7fa640 1 -- 172.21.15.28:0/1546785516 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b64005020 con 0x7f8b70106160 2024-04-21T08:54:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.702+0000 7f8b6d7fa640 1 -- 172.21.15.28:0/1546785516 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64004a00 con 0x7f8b70106160 2024-04-21T08:54:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.702+0000 7f8b7677e640 1 -- 172.21.15.28:0/1546785516 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b701a55c0 con 0x7f8b70106160 2024-04-21T08:54:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.705+0000 7f8b6d7fa640 1 -- 172.21.15.28:0/1546785516 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8b640051d0 con 0x7f8b70106160 2024-04-21T08:54:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.705+0000 7f8b6d7fa640 1 --2- 172.21.15.28:0/1546785516 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8b580a3da0 0x7f8b580a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.705+0000 7f8b7936f740 1 Processor -- start 2024-04-21T08:54:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.706+0000 7f8b7936f740 1 -- start start 2024-04-21T08:54:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.706+0000 7f8b6d7fa640 1 -- 172.21.15.28:0/1546785516 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f8b640d0a40 con 0x7f8b70106160 2024-04-21T08:54:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.706+0000 7f8b7936f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5563343704d0 0x5563343708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.706+0000 7f8b7936f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556334370e80 0x556334379430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.706+0000 7f8b6ffff640 1 --2- 172.21.15.28:0/1546785516 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8b580a3da0 0x7f8b580a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.706+0000 7f8b7936f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334379a00 0x55633437be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.706+0000 7f8b7936f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5563342d51f0 con 0x556334370e80 2024-04-21T08:54:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.706+0000 7f8b7936f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5563342724b0 con 0x556334379a00 2024-04-21T08:54:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.706+0000 7f8b7936f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5563342d11b0 con 0x5563343704d0 2024-04-21T08:54:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.706+0000 7f8b6f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556334370e80 0x556334379430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.706+0000 7f8b6f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556334370e80 0x556334379430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34240/0 (socket says 172.21.15.28:34240) 2024-04-21T08:54:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.706+0000 7f8b6f7fe640 1 -- 172.21.15.28:0/3104036924 learned_addr learned my addr 172.21.15.28:0/3104036924 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:48.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.706+0000 7f8b6ffff640 1 --2- 172.21.15.28:0/3104036924 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5563343704d0 0x5563343708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.706+0000 7f8b74d1b640 1 --2- 172.21.15.28:0/3104036924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334379a00 0x55633437be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.707+0000 7f8b6f7fe640 1 -- 172.21.15.28:0/3104036924 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5563343704d0 msgr2=0x5563343708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.707+0000 7f8b6f7fe640 1 --2- 172.21.15.28:0/3104036924 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5563343704d0 0x5563343708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.707+0000 7f8b6f7fe640 1 -- 172.21.15.28:0/3104036924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334379a00 msgr2=0x55633437be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.707+0000 7f8b6f7fe640 1 --2- 172.21.15.28:0/3104036924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334379a00 0x55633437be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.707+0000 7f8b6f7fe640 1 -- 172.21.15.28:0/3104036924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563342641b0 con 0x556334370e80 2024-04-21T08:54:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.707+0000 7f8b74d1b640 1 --2- 172.21.15.28:0/3104036924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334379a00 0x55633437be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.707+0000 7f8b6f7fe640 1 --2- 172.21.15.28:0/3104036924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556334370e80 0x556334379430 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f8b640a4d80 tx=0x7f8b6409ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=386fc49981ea1e0f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.707+0000 7f8b6ffff640 1 --2- 172.21.15.28:0/1546785516 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8b580a3da0 0x7f8b580a6260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8b600139f0 tx=0x7f8b600134e0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.707+0000 7f8b567fc640 1 -- 172.21.15.28:0/3104036924 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6401c040 con 0x556334370e80 2024-04-21T08:54:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.707+0000 7f8b567fc640 1 -- 172.21.15.28:0/3104036924 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b640b3020 con 0x556334370e80 2024-04-21T08:54:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.707+0000 7f8b567fc640 1 -- 172.21.15.28:0/3104036924 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64098cb0 con 0x556334370e80 2024-04-21T08:54:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.708+0000 7f8b7936f740 1 -- 172.21.15.28:0/3104036924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556334370e80 msgr2=0x556334379430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.708+0000 7f8b7936f740 1 --2- 172.21.15.28:0/3104036924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556334370e80 0x556334379430 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f8b640a4d80 tx=0x7f8b6409ce00 comp rx=0 tx=0).stop 2024-04-21T08:54:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.708+0000 7f8b7936f740 1 -- 172.21.15.28:0/3104036924 shutdown_connections 2024-04-21T08:54:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.708+0000 7f8b7936f740 1 --2- 172.21.15.28:0/3104036924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334379a00 0x55633437be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.708+0000 7f8b7936f740 1 --2- 172.21.15.28:0/3104036924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556334370e80 0x556334379430 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.708+0000 7f8b7936f740 1 --2- 172.21.15.28:0/3104036924 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5563343704d0 0x5563343708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.708+0000 7f8b7936f740 1 -- 172.21.15.28:0/3104036924 >> 172.21.15.28:0/3104036924 conn(0x55633436aba0 msgr2=0x55633436af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.708+0000 7f8b7936f740 1 -- 172.21.15.28:0/3104036924 shutdown_connections 2024-04-21T08:54:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.709+0000 7f8b7936f740 1 -- 172.21.15.28:0/3104036924 wait complete. 2024-04-21T08:54:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.709+0000 7f8b7936f740 1 -- start start 2024-04-21T08:54:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.709+0000 7f8b7936f740 1 Processor -- start 2024-04-21T08:54:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.710+0000 7f8b7936f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563343704d0 0x5563343790e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.710+0000 7f8b7936f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334370e80 0x5563343766d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.710+0000 7f8b7936f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334379a00 0x556334376c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.710+0000 7f8b7936f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55633437c970 con 0x5563343704d0 2024-04-21T08:54:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.710+0000 7f8b74d1b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334379a00 0x556334376c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.710+0000 7f8b74d1b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334379a00 0x556334376c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:60752/0 (socket says 172.21.15.28:60752) 2024-04-21T08:54:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.710+0000 7f8b74d1b640 1 -- 172.21.15.28:0/2633986955 learned_addr learned my addr 172.21.15.28:0/2633986955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.710+0000 7f8b7936f740 1 -- 172.21.15.28:0/2633986955 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55633428e500 con 0x556334370e80 2024-04-21T08:54:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.710+0000 7f8b7936f740 1 -- 172.21.15.28:0/2633986955 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55633437caf0 con 0x556334379a00 2024-04-21T08:54:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.710+0000 7f8b6ffff640 1 --2- 172.21.15.28:0/2633986955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563343704d0 0x5563343790e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.710+0000 7f8b6f7fe640 1 --2- 172.21.15.28:0/2633986955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334370e80 0x5563343766d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.710+0000 7f8b7936f740 10 client.0 _ll_register_callbacks cb 0x55633425cb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.711+0000 7f8b74d1b640 1 -- 172.21.15.28:0/2633986955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334370e80 msgr2=0x5563343766d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.711+0000 7f8b74d1b640 1 --2- 172.21.15.28:0/2633986955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334370e80 0x5563343766d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.711+0000 7f8b74d1b640 1 -- 172.21.15.28:0/2633986955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563343704d0 msgr2=0x5563343790e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.711+0000 7f8b74d1b640 1 --2- 172.21.15.28:0/2633986955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563343704d0 0x5563343790e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.711+0000 7f8b74d1b640 1 -- 172.21.15.28:0/2633986955 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556333fe73f0 con 0x556334379a00 2024-04-21T08:54:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.711+0000 7f8b74d1b640 1 --2- 172.21.15.28:0/2633986955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334379a00 0x556334376c10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8b68009ca0 tx=0x7f8b680326f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8754eca693a7f1fd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.711+0000 7f8b4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:54:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.711+0000 7f8b4f7fe640 1 -- 172.21.15.28:0/2633986955 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b68043070 con 0x556334379a00 2024-04-21T08:54:48.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.711+0000 7f8b4f7fe640 1 -- 172.21.15.28:0/2633986955 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f8b68003970 con 0x556334379a00 2024-04-21T08:54:48.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.712+0000 7f8b7936f740 1 -- 172.21.15.28:0/2633986955 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b640b3020 con 0x556334379a00 2024-04-21T08:54:48.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.712+0000 7f8b4f7fe640 1 -- 172.21.15.28:0/2633986955 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8b6804f050 con 0x556334379a00 2024-04-21T08:54:48.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.712+0000 7f8b7936f740 10 client.26149 fetch_fsmap learned FSMap version 28 2024-04-21T08:54:48.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.712+0000 7f8b7936f740 1 -- 172.21.15.28:0/2633986955 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5563340b17d0 con 0x556334379a00 2024-04-21T08:54:48.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.712+0000 7f8b4f7fe640 1 -- 172.21.15.28:0/2633986955 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b640b3020 con 0x556334379a00 2024-04-21T08:54:48.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.712+0000 7f8b4f7fe640 1 -- 172.21.15.28:0/2633986955 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f8b68004690 con 0x556334379a00 2024-04-21T08:54:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.712+0000 7f8b7936f740 10 client.26149 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:54:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.712+0000 7f8b7936f740 10 client.26149 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:54:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.712+0000 7f8b7936f740 20 client.26149 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.713+0000 7f8b7936f740 1 --2- 172.21.15.28:0/2633986955 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x556334392370 0x556334394760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.713+0000 7f8b7936f740 4 client.26149 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.713+0000 7f8b7936f740 1 -- 172.21.15.28:0/2633986955 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556334274930 con 0x556334392370 2024-04-21T08:54:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.713+0000 7f8b6ffff640 1 --2- 172.21.15.28:0/2633986955 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x556334392370 0x556334394760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.713+0000 7f8b6ffff640 1 --2- 172.21.15.28:0/2633986955 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x556334392370 0x556334394760 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2bb88a75123294f7 server_cookie=8e8aac9caa6c587e in_seq=0 out_seq=0 2024-04-21T08:54:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.713+0000 7f8b4f7fe640 10 client.26149 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:54:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.714+0000 7f8b4f7fe640 1 -- 172.21.15.28:0/2633986955 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556334274930 con 0x556334392370 2024-04-21T08:54:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.714+0000 7f8b4f7fe640 10 client.26149 handle_command_reply: tid=0 2024-04-21T08:54:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.714+0000 7f8b7936f740 1 client.26149 shutdown 2024-04-21T08:54:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.715+0000 7f8b7936f740 1 -- 172.21.15.28:0/2633986955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334379a00 msgr2=0x556334376c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.715+0000 7f8b7936f740 1 --2- 172.21.15.28:0/2633986955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334379a00 0x556334376c10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8b68009ca0 tx=0x7f8b680326f0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.715+0000 7f8b7936f740 1 -- 172.21.15.28:0/2633986955 shutdown_connections 2024-04-21T08:54:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.715+0000 7f8b7936f740 1 --2- 172.21.15.28:0/2633986955 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x556334392370 0x556334394760 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.715+0000 7f8b7936f740 1 --2- 172.21.15.28:0/2633986955 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334379a00 0x556334376c10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.715+0000 7f8b7936f740 1 --2- 172.21.15.28:0/2633986955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334370e80 0x5563343766d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.715+0000 7f8b7936f740 1 --2- 172.21.15.28:0/2633986955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563343704d0 0x5563343790e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.715+0000 7f8b4f7fe640 0 client.26149 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:54:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.715+0000 7f8b7936f740 1 -- 172.21.15.28:0/2633986955 >> 172.21.15.28:0/2633986955 conn(0x55633436aba0 msgr2=0x55633436c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.715+0000 7f8b7936f740 1 -- 172.21.15.28:0/2633986955 shutdown_connections 2024-04-21T08:54:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.715+0000 7f8b7936f740 1 -- 172.21.15.28:0/2633986955 wait complete. 2024-04-21T08:54:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.715+0000 7f8b7936f740 20 client.26149 trim_cache size 0 max 16384 2024-04-21T08:54:48.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.736+0000 7f8b7936f740 1 Processor -- start 2024-04-21T08:54:48.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.736+0000 7f8b7936f740 1 -- start start 2024-04-21T08:54:48.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.736+0000 7f8b7936f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563343704d0 0x556334371b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.736+0000 7f8b7936f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334370e80 0x556334378e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.736+0000 7f8b7936f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334379a00 0x55633439a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.737+0000 7f8b7936f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b701af9a0 con 0x5563343704d0 2024-04-21T08:54:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.737+0000 7f8b7936f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8b701ad270 con 0x556334379a00 2024-04-21T08:54:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.737+0000 7f8b7936f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8b701ab0c0 con 0x556334370e80 2024-04-21T08:54:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.737+0000 7f8b6f7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334370e80 0x556334378e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.737+0000 7f8b6f7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334370e80 0x556334378e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:60760/0 (socket says 172.21.15.28:60760) 2024-04-21T08:54:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.737+0000 7f8b6f7fe640 1 -- 172.21.15.28:0/3271634486 learned_addr learned my addr 172.21.15.28:0/3271634486 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.737+0000 7f8b74d1b640 1 --2- 172.21.15.28:0/3271634486 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334379a00 0x55633439a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.737+0000 7f8b6ffff640 1 --2- 172.21.15.28:0/3271634486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563343704d0 0x556334371b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.737+0000 7f8b6f7fe640 1 -- 172.21.15.28:0/3271634486 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334379a00 msgr2=0x55633439a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.737+0000 7f8b6f7fe640 1 --2- 172.21.15.28:0/3271634486 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334379a00 0x55633439a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.737+0000 7f8b6f7fe640 1 -- 172.21.15.28:0/3271634486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563343704d0 msgr2=0x556334371b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.737+0000 7f8b6f7fe640 1 --2- 172.21.15.28:0/3271634486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563343704d0 0x556334371b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.737+0000 7f8b6f7fe640 1 -- 172.21.15.28:0/3271634486 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556334394ad0 con 0x556334370e80 2024-04-21T08:54:48.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.738+0000 7f8b6f7fe640 1 --2- 172.21.15.28:0/3271634486 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334370e80 0x556334378e00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8b64099010 tx=0x7f8b6400efb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5853143e0b60f51e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.738+0000 7f8b55ffb640 1 -- 172.21.15.28:0/3271634486 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6401c040 con 0x556334370e80 2024-04-21T08:54:48.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.738+0000 7f8b55ffb640 1 -- 172.21.15.28:0/3271634486 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b640b3070 con 0x556334370e80 2024-04-21T08:54:48.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.738+0000 7f8b55ffb640 1 -- 172.21.15.28:0/3271634486 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b640ce200 con 0x556334370e80 2024-04-21T08:54:48.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.739+0000 7f8b7936f740 1 -- 172.21.15.28:0/3271634486 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334370e80 msgr2=0x556334378e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.739+0000 7f8b7936f740 1 --2- 172.21.15.28:0/3271634486 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334370e80 0x556334378e00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8b64099010 tx=0x7f8b6400efb0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.739+0000 7f8b7936f740 1 -- 172.21.15.28:0/3271634486 shutdown_connections 2024-04-21T08:54:48.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.739+0000 7f8b7936f740 1 --2- 172.21.15.28:0/3271634486 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556334379a00 0x55633439a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.739+0000 7f8b7936f740 1 --2- 172.21.15.28:0/3271634486 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334370e80 0x556334378e00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.739+0000 7f8b7936f740 1 --2- 172.21.15.28:0/3271634486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5563343704d0 0x556334371b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.739+0000 7f8b7936f740 1 -- 172.21.15.28:0/3271634486 >> 172.21.15.28:0/3271634486 conn(0x55633436aba0 msgr2=0x556334375f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.739+0000 7f8b7936f740 1 -- 172.21.15.28:0/3271634486 shutdown_connections 2024-04-21T08:54:48.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.739+0000 7f8b7936f740 1 -- 172.21.15.28:0/3271634486 wait complete. 2024-04-21T08:54:48.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.740+0000 7f8b7936f740 1 -- start start 2024-04-21T08:54:48.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.740+0000 7f8b7936f740 1 Processor -- start 2024-04-21T08:54:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.740+0000 7f8b7936f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563343704d0 0x55633438e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.740+0000 7f8b7936f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556334370e80 0x55633438ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b7936f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334379a00 0x55633438b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b7936f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b701ab730 con 0x556334370e80 2024-04-21T08:54:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b7936f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8b701acba0 con 0x5563343704d0 2024-04-21T08:54:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b7936f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8b701abe00 con 0x556334379a00 2024-04-21T08:54:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b7936f740 10 client.0 _ll_register_callbacks cb 0x7f8b68009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b6f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556334370e80 0x55633438ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b6f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556334370e80 0x55633438ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34274/0 (socket says 172.21.15.28:34274) 2024-04-21T08:54:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b6f7fe640 1 -- 172.21.15.28:0/868284388 learned_addr learned my addr 172.21.15.28:0/868284388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b74d1b640 1 --2- 172.21.15.28:0/868284388 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334379a00 0x55633438b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b6f7fe640 1 -- 172.21.15.28:0/868284388 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334379a00 msgr2=0x55633438b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b6f7fe640 1 --2- 172.21.15.28:0/868284388 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334379a00 0x55633438b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b6f7fe640 1 -- 172.21.15.28:0/868284388 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563343704d0 msgr2=0x55633438e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b6ffff640 1 --2- 172.21.15.28:0/868284388 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563343704d0 0x55633438e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b6f7fe640 1 --2- 172.21.15.28:0/868284388 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563343704d0 0x55633438e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b6f7fe640 1 -- 172.21.15.28:0/868284388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55633438f5e0 con 0x556334370e80 2024-04-21T08:54:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.741+0000 7f8b6ffff640 1 --2- 172.21.15.28:0/868284388 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563343704d0 0x55633438e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.742+0000 7f8b6f7fe640 1 --2- 172.21.15.28:0/868284388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556334370e80 0x55633438ef20 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8b64098aa0 tx=0x7f8b64005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cfd3947d583bc62 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.742+0000 7f8b4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.742+0000 7f8b4f7fe640 1 -- 172.21.15.28:0/868284388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6401c070 con 0x556334370e80 2024-04-21T08:54:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.742+0000 7f8b4f7fe640 1 -- 172.21.15.28:0/868284388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f8b640a5030 con 0x556334370e80 2024-04-21T08:54:48.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.742+0000 7f8b7936f740 1 -- 172.21.15.28:0/868284388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b640b3070 con 0x556334370e80 2024-04-21T08:54:48.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.742+0000 7f8b4f7fe640 1 -- 172.21.15.28:0/868284388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8b640ce1e0 con 0x556334370e80 2024-04-21T08:54:48.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.742+0000 7f8b7936f740 10 client.16857 fetch_fsmap learned FSMap version 28 2024-04-21T08:54:48.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.742+0000 7f8b7936f740 1 -- 172.21.15.28:0/868284388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5563343f8e80 con 0x556334370e80 2024-04-21T08:54:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.743+0000 7f8b4f7fe640 1 -- 172.21.15.28:0/868284388 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b64004ba0 con 0x556334370e80 2024-04-21T08:54:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.743+0000 7f8b4f7fe640 1 -- 172.21.15.28:0/868284388 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f8b6409ba90 con 0x556334370e80 2024-04-21T08:54:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.743+0000 7f8b7936f740 10 client.16857 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:54:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.743+0000 7f8b7936f740 10 client.16857 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:54:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.743+0000 7f8b7936f740 20 client.16857 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.743+0000 7f8b7936f740 1 --2- 172.21.15.28:0/868284388 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5563343f9100 0x5563343fb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.743+0000 7f8b7936f740 4 client.16857 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:54:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.743+0000 7f8b7936f740 1 -- 172.21.15.28:0/868284388 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f8b640b3070 con 0x5563343f9100 2024-04-21T08:54:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.743+0000 7f8b6ffff640 1 --2- 172.21.15.28:0/868284388 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5563343f9100 0x5563343fb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.744+0000 7f8b6ffff640 1 --2- 172.21.15.28:0/868284388 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5563343f9100 0x5563343fb4f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7e6b5dd77c1030b6 server_cookie=da63125fd30de040 in_seq=0 out_seq=0 2024-04-21T08:54:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.744+0000 7f8b4f7fe640 10 client.16857 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:54:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.744+0000 7f8b4f7fe640 1 -- 172.21.15.28:0/868284388 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f8b640b3070 con 0x5563343f9100 2024-04-21T08:54:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.744+0000 7f8b4f7fe640 10 client.16857 handle_command_reply: tid=0 2024-04-21T08:54:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.744+0000 7f8b7936f740 1 client.16857 shutdown 2024-04-21T08:54:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.744+0000 7f8b7936f740 1 -- 172.21.15.28:0/868284388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556334370e80 msgr2=0x55633438ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.744+0000 7f8b7936f740 1 --2- 172.21.15.28:0/868284388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556334370e80 0x55633438ef20 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8b64098aa0 tx=0x7f8b64005720 comp rx=0 tx=0).stop 2024-04-21T08:54:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.744+0000 7f8b7936f740 1 -- 172.21.15.28:0/868284388 shutdown_connections 2024-04-21T08:54:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.744+0000 7f8b7936f740 1 --2- 172.21.15.28:0/868284388 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5563343f9100 0x5563343fb4f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.745+0000 7f8b7936f740 1 --2- 172.21.15.28:0/868284388 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556334379a00 0x55633438b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.745+0000 7f8b7936f740 1 --2- 172.21.15.28:0/868284388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556334370e80 0x55633438ef20 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.745+0000 7f8b7936f740 1 --2- 172.21.15.28:0/868284388 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5563343704d0 0x55633438e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.745+0000 7f8b7936f740 1 -- 172.21.15.28:0/868284388 >> 172.21.15.28:0/868284388 conn(0x55633436aba0 msgr2=0x556334371990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.745+0000 7f8b4f7fe640 0 client.16857 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:54:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.745+0000 7f8b7936f740 1 -- 172.21.15.28:0/868284388 shutdown_connections 2024-04-21T08:54:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.745+0000 7f8b7936f740 1 -- 172.21.15.28:0/868284388 wait complete. 2024-04-21T08:54:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.745+0000 7f8b7936f740 20 client.16857 trim_cache size 0 max 16384 2024-04-21T08:54:48.747 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:54:48.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.745+0000 7f8b7677e640 1 -- 172.21.15.28:0/1546785516 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8b580a3da0 msgr2=0x7f8b580a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.746+0000 7f8b7677e640 1 --2- 172.21.15.28:0/1546785516 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8b580a3da0 0x7f8b580a6260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8b600139f0 tx=0x7f8b600134e0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.746+0000 7f8b7677e640 1 -- 172.21.15.28:0/1546785516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b70106160 msgr2=0x7f8b70198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.746+0000 7f8b7677e640 1 --2- 172.21.15.28:0/1546785516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b70106160 0x7f8b70198c70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8b6400e9c0 tx=0x7f8b6400ee80 comp rx=0 tx=0).stop 2024-04-21T08:54:48.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.746+0000 7f8b7677e640 1 -- 172.21.15.28:0/1546785516 shutdown_connections 2024-04-21T08:54:48.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.746+0000 7f8b7677e640 1 --2- 172.21.15.28:0/1546785516 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8b580a3da0 0x7f8b580a6260 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.746+0000 7f8b7677e640 1 --2- 172.21.15.28:0/1546785516 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8b7019cff0 0x7f8b7019d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.746+0000 7f8b7677e640 1 --2- 172.21.15.28:0/1546785516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b70106160 0x7f8b70198c70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.746+0000 7f8b7677e640 1 --2- 172.21.15.28:0/1546785516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70101090 0x7f8b70198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.746+0000 7f8b7677e640 1 -- 172.21.15.28:0/1546785516 >> 172.21.15.28:0/1546785516 conn(0x7f8b70077f50 msgr2=0x7f8b700fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.747+0000 7f8b7677e640 1 -- 172.21.15.28:0/1546785516 shutdown_connections 2024-04-21T08:54:48.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.747+0000 7f8b7677e640 1 -- 172.21.15.28:0/1546785516 wait complete. 2024-04-21T08:54:48.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:54:48.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.885+0000 7f35b9255640 1 -- 172.21.15.28:0/1341099167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b4103af0 msgr2=0x7f35b410bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.885+0000 7f35b9255640 1 --2- 172.21.15.28:0/1341099167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b4103af0 0x7f35b410bfd0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f359c009900 tx=0x7f359c02bd60 comp rx=0 tx=0).stop 2024-04-21T08:54:48.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.885+0000 7f35b9255640 1 -- 172.21.15.28:0/1341099167 shutdown_connections 2024-04-21T08:54:48.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.885+0000 7f35b9255640 1 --2- 172.21.15.28:0/1341099167 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35b410c510 0x7f35b410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.885+0000 7f35b9255640 1 --2- 172.21.15.28:0/1341099167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b4103af0 0x7f35b410bfd0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.885+0000 7f35b9255640 1 --2- 172.21.15.28:0/1341099167 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f35b41031d0 0x7f35b41035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.885+0000 7f35b9255640 1 -- 172.21.15.28:0/1341099167 >> 172.21.15.28:0/1341099167 conn(0x7f35b4077f50 msgr2=0x7f35b4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.885+0000 7f35b9255640 1 -- 172.21.15.28:0/1341099167 shutdown_connections 2024-04-21T08:54:48.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.885+0000 7f35b9255640 1 -- 172.21.15.28:0/1341099167 wait complete. 2024-04-21T08:54:48.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.886+0000 7f35b9255640 1 Processor -- start 2024-04-21T08:54:48.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.886+0000 7f35b9255640 1 -- start start 2024-04-21T08:54:48.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.886+0000 7f35b9255640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b41031d0 0x7f35b41986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.886+0000 7f35b9255640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f35b4103af0 0x7f35b4198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.886+0000 7f35b9255640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35b410c510 0x7f35b419cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.886+0000 7f35b9255640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35b4112180 con 0x7f35b41031d0 2024-04-21T08:54:48.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.886+0000 7f35b9255640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f35b4112000 con 0x7f35b4103af0 2024-04-21T08:54:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.886+0000 7f35b9255640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f35b4112300 con 0x7f35b410c510 2024-04-21T08:54:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.886+0000 7f35b2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b41031d0 0x7f35b41986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35b2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b41031d0 0x7f35b41986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34288/0 (socket says 172.21.15.28:34288) 2024-04-21T08:54:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35b2d9d640 1 -- 172.21.15.28:0/4101070376 learned_addr learned my addr 172.21.15.28:0/4101070376 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35b259c640 1 --2- 172.21.15.28:0/4101070376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f35b4103af0 0x7f35b4198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35b359e640 1 --2- 172.21.15.28:0/4101070376 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35b410c510 0x7f35b419cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35b2d9d640 1 -- 172.21.15.28:0/4101070376 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35b410c510 msgr2=0x7f35b419cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35b2d9d640 1 --2- 172.21.15.28:0/4101070376 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35b410c510 0x7f35b419cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35b2d9d640 1 -- 172.21.15.28:0/4101070376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f35b4103af0 msgr2=0x7f35b4198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35b2d9d640 1 --2- 172.21.15.28:0/4101070376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f35b4103af0 0x7f35b4198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35b2d9d640 1 -- 172.21.15.28:0/4101070376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b419d6d0 con 0x7f35b41031d0 2024-04-21T08:54:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35b359e640 1 --2- 172.21.15.28:0/4101070376 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35b410c510 0x7f35b419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35b2d9d640 1 --2- 172.21.15.28:0/4101070376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b41031d0 0x7f35b41986e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f35a400d950 tx=0x7f35a400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35abfff640 1 -- 172.21.15.28:0/4101070376 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a4017070 con 0x7f35b41031d0 2024-04-21T08:54:48.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35b9255640 1 -- 172.21.15.28:0/4101070376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35b419d9c0 con 0x7f35b41031d0 2024-04-21T08:54:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35abfff640 1 -- 172.21.15.28:0/4101070376 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35a4009c30 con 0x7f35b41031d0 2024-04-21T08:54:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35abfff640 1 -- 172.21.15.28:0/4101070376 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a400f620 con 0x7f35b41031d0 2024-04-21T08:54:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.887+0000 7f35b9255640 1 -- 172.21.15.28:0/4101070376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35b41a5250 con 0x7f35b41031d0 2024-04-21T08:54:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.890+0000 7f35abfff640 1 -- 172.21.15.28:0/4101070376 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f35a4010460 con 0x7f35b41031d0 2024-04-21T08:54:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.890+0000 7f35bbe46740 1 Processor -- start 2024-04-21T08:54:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.890+0000 7f35bbe46740 1 -- start start 2024-04-21T08:54:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.890+0000 7f35bbe46740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.890+0000 7f35bbe46740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f3e80 0x55d0a37fc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.890+0000 7f35bbe46740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37fca00 0x55d0a37fee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.890+0000 7f35bbe46740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d0a37581f0 con 0x55d0a37f34d0 2024-04-21T08:54:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.890+0000 7f35bbe46740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55d0a36f54b0 con 0x55d0a37fca00 2024-04-21T08:54:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.890+0000 7f35bbe46740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55d0a37541b0 con 0x55d0a37f3e80 2024-04-21T08:54:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.890+0000 7f35abfff640 1 --2- 172.21.15.28:0/4101070376 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f35a00a3d20 0x7f35a00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.890+0000 7f35b359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35b359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37f38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34292/0 (socket says 172.21.15.28:34292) 2024-04-21T08:54:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35b359e640 1 -- 172.21.15.28:0/2857428870 learned_addr learned my addr 172.21.15.28:0/2857428870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35b259c640 1 --2- 172.21.15.28:0/2857428870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37fca00 0x55d0a37fee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35b2d9d640 1 --2- 172.21.15.28:0/2857428870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f3e80 0x55d0a37fc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35b359e640 1 --2- 172.21.15.28:0/4101070376 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f35a00a3d20 0x7f35a00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35abfff640 1 -- 172.21.15.28:0/4101070376 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f35a40d0800 con 0x7f35b41031d0 2024-04-21T08:54:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35b359e640 1 -- 172.21.15.28:0/2857428870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f3e80 msgr2=0x55d0a37fc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35b359e640 1 --2- 172.21.15.28:0/2857428870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f3e80 0x55d0a37fc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35b359e640 1 -- 172.21.15.28:0/2857428870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37fca00 msgr2=0x55d0a37fee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35b359e640 1 --2- 172.21.15.28:0/2857428870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37fca00 0x55d0a37fee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35b359e640 1 -- 172.21.15.28:0/2857428870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0a36e71b0 con 0x55d0a37f34d0 2024-04-21T08:54:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35b259c640 1 --2- 172.21.15.28:0/2857428870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37fca00 0x55d0a37fee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35b359e640 1 --2- 172.21.15.28:0/2857428870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37f38b0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f35ac012960 tx=0x7f35ac012e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89df16a9034026f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35a97fa640 1 -- 172.21.15.28:0/2857428870 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac004d40 con 0x55d0a37f34d0 2024-04-21T08:54:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35a97fa640 1 -- 172.21.15.28:0/2857428870 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35ac007590 con 0x55d0a37f34d0 2024-04-21T08:54:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35b359e640 1 --2- 172.21.15.28:0/4101070376 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f35a00a3d20 0x7f35a00a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f35ac01f000 tx=0x7f35ac01d000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.891+0000 7f35a97fa640 1 -- 172.21.15.28:0/2857428870 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac005250 con 0x55d0a37f34d0 2024-04-21T08:54:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.892+0000 7f35bbe46740 1 -- 172.21.15.28:0/2857428870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 msgr2=0x55d0a37f38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.892+0000 7f35bbe46740 1 --2- 172.21.15.28:0/2857428870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37f38b0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f35ac012960 tx=0x7f35ac012e20 comp rx=0 tx=0).stop 2024-04-21T08:54:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.892+0000 7f35bbe46740 1 -- 172.21.15.28:0/2857428870 shutdown_connections 2024-04-21T08:54:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.892+0000 7f35bbe46740 1 --2- 172.21.15.28:0/2857428870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37fca00 0x55d0a37fee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.892+0000 7f35bbe46740 1 --2- 172.21.15.28:0/2857428870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f3e80 0x55d0a37fc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.892+0000 7f35bbe46740 1 --2- 172.21.15.28:0/2857428870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37f38b0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.892+0000 7f35bbe46740 1 -- 172.21.15.28:0/2857428870 >> 172.21.15.28:0/2857428870 conn(0x55d0a37edba0 msgr2=0x55d0a37edf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.892+0000 7f35bbe46740 1 -- 172.21.15.28:0/2857428870 shutdown_connections 2024-04-21T08:54:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.892+0000 7f35bbe46740 1 -- 172.21.15.28:0/2857428870 wait complete. 2024-04-21T08:54:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.892+0000 7f35bbe46740 1 -- start start 2024-04-21T08:54:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.892+0000 7f35bbe46740 1 Processor -- start 2024-04-21T08:54:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35bbe46740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35bbe46740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37f3e80 0x55d0a37f9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35bbe46740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37fca00 0x55d0a37f9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35bbe46740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d0a37ff970 con 0x55d0a37f34d0 2024-04-21T08:54:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35bbe46740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55d0a3711500 con 0x55d0a37f3e80 2024-04-21T08:54:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35bbe46740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55d0a37ffaf0 con 0x55d0a37fca00 2024-04-21T08:54:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35bbe46740 10 client.0 _ll_register_callbacks cb 0x55d0a36dfb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35b2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35b2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37fc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34308/0 (socket says 172.21.15.28:34308) 2024-04-21T08:54:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35b2d9d640 1 -- 172.21.15.28:0/87203881 learned_addr learned my addr 172.21.15.28:0/87203881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35b259c640 1 --2- 172.21.15.28:0/87203881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37f3e80 0x55d0a37f9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35b359e640 1 --2- 172.21.15.28:0/87203881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37fca00 0x55d0a37f9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35b2d9d640 1 -- 172.21.15.28:0/87203881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37fca00 msgr2=0x55d0a37f9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35b2d9d640 1 --2- 172.21.15.28:0/87203881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37fca00 0x55d0a37f9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35b2d9d640 1 -- 172.21.15.28:0/87203881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37f3e80 msgr2=0x55d0a37f9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35b2d9d640 1 --2- 172.21.15.28:0/87203881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37f3e80 0x55d0a37f9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35b2d9d640 1 -- 172.21.15.28:0/87203881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0a346a3f0 con 0x55d0a37f34d0 2024-04-21T08:54:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35b359e640 1 --2- 172.21.15.28:0/87203881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37fca00 0x55d0a37f9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35b2d9d640 1 --2- 172.21.15.28:0/87203881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37fc2f0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f35a4098dd0 tx=0x7f35a4018ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e163096ad48c82b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f359a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f359a7fc640 1 -- 172.21.15.28:0/87203881 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a4020030 con 0x55d0a37f34d0 2024-04-21T08:54:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f359a7fc640 1 -- 172.21.15.28:0/87203881 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f35a4018d10 con 0x55d0a37f34d0 2024-04-21T08:54:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.893+0000 7f35bbe46740 1 -- 172.21.15.28:0/87203881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35ac007590 con 0x55d0a37f34d0 2024-04-21T08:54:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.894+0000 7f359a7fc640 1 -- 172.21.15.28:0/87203881 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f35a4009de0 con 0x55d0a37f34d0 2024-04-21T08:54:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.894+0000 7f35bbe46740 10 client.16869 fetch_fsmap learned FSMap version 28 2024-04-21T08:54:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.894+0000 7f35bbe46740 1 -- 172.21.15.28:0/87203881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d0a35347d0 con 0x55d0a37f34d0 2024-04-21T08:54:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.894+0000 7f359a7fc640 1 -- 172.21.15.28:0/87203881 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35ac007590 con 0x55d0a37f34d0 2024-04-21T08:54:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.894+0000 7f359a7fc640 1 -- 172.21.15.28:0/87203881 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f35a4016170 con 0x55d0a37f34d0 2024-04-21T08:54:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.894+0000 7f35bbe46740 10 client.16869 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:54:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.894+0000 7f35bbe46740 10 client.16869 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.894+0000 7f35bbe46740 20 client.16869 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.894+0000 7f35bbe46740 1 --2- 172.21.15.28:0/87203881 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d0a380cb40 0x55d0a380ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.894+0000 7f35bbe46740 4 client.16869 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.894+0000 7f35bbe46740 1 -- 172.21.15.28:0/87203881 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0a36f7930 con 0x55d0a380cb40 2024-04-21T08:54:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.894+0000 7f35b259c640 1 --2- 172.21.15.28:0/87203881 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d0a380cb40 0x55d0a380ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.895+0000 7f35b259c640 1 --2- 172.21.15.28:0/87203881 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d0a380cb40 0x55d0a380ef90 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87a1d2090343a845 server_cookie=fa81340406fae83f in_seq=0 out_seq=0 2024-04-21T08:54:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.895+0000 7f359a7fc640 10 client.16869 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.895+0000 7f359a7fc640 1 -- 172.21.15.28:0/87203881 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d0a36f7930 con 0x55d0a380cb40 2024-04-21T08:54:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.895+0000 7f359a7fc640 10 client.16869 handle_command_reply: tid=0 2024-04-21T08:54:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.896+0000 7f35bbe46740 1 client.16869 shutdown 2024-04-21T08:54:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.896+0000 7f35bbe46740 1 -- 172.21.15.28:0/87203881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 msgr2=0x55d0a37fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.896+0000 7f35bbe46740 1 --2- 172.21.15.28:0/87203881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37fc2f0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f35a4098dd0 tx=0x7f35a4018ca0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.896+0000 7f35bbe46740 1 -- 172.21.15.28:0/87203881 shutdown_connections 2024-04-21T08:54:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.896+0000 7f35bbe46740 1 --2- 172.21.15.28:0/87203881 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d0a380cb40 0x55d0a380ef90 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.896+0000 7f35bbe46740 1 --2- 172.21.15.28:0/87203881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37fca00 0x55d0a37f9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.896+0000 7f35bbe46740 1 --2- 172.21.15.28:0/87203881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37f3e80 0x55d0a37f9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.896+0000 7f359a7fc640 0 client.16869 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.896+0000 7f35bbe46740 1 --2- 172.21.15.28:0/87203881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37fc2f0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.896+0000 7f35bbe46740 1 -- 172.21.15.28:0/87203881 >> 172.21.15.28:0/87203881 conn(0x55d0a37edba0 msgr2=0x55d0a37ef8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.896+0000 7f35bbe46740 1 -- 172.21.15.28:0/87203881 shutdown_connections 2024-04-21T08:54:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.896+0000 7f35bbe46740 1 -- 172.21.15.28:0/87203881 wait complete. 2024-04-21T08:54:48.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.896+0000 7f35bbe46740 20 client.16869 trim_cache size 0 max 16384 2024-04-21T08:54:48.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.909+0000 7f35bbe46740 1 Processor -- start 2024-04-21T08:54:48.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35bbe46740 1 -- start start 2024-04-21T08:54:48.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35bbe46740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f34d0 0x55d0a3808830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35bbe46740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37f3e80 0x55d0a37fbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35bbe46740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37fca00 0x55d0a37fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35bbe46740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35b41b1bd0 con 0x55d0a37fca00 2024-04-21T08:54:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35bbe46740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f35b41af4a0 con 0x55d0a37f3e80 2024-04-21T08:54:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35bbe46740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f35b41ad2f0 con 0x55d0a37f34d0 2024-04-21T08:54:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35b359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37fca00 0x55d0a37fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35b2d9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f34d0 0x55d0a3808830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35b359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37fca00 0x55d0a37fc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34312/0 (socket says 172.21.15.28:34312) 2024-04-21T08:54:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35b359e640 1 -- 172.21.15.28:0/189948568 learned_addr learned my addr 172.21.15.28:0/189948568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35b259c640 1 --2- 172.21.15.28:0/189948568 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37f3e80 0x55d0a37fbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35b359e640 1 -- 172.21.15.28:0/189948568 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f34d0 msgr2=0x55d0a3808830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35b359e640 1 --2- 172.21.15.28:0/189948568 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f34d0 0x55d0a3808830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35b359e640 1 -- 172.21.15.28:0/189948568 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37f3e80 msgr2=0x55d0a37fbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35b359e640 1 --2- 172.21.15.28:0/189948568 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37f3e80 0x55d0a37fbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35b359e640 1 -- 172.21.15.28:0/189948568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0a37f9f70 con 0x55d0a37fca00 2024-04-21T08:54:48.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35b259c640 1 --2- 172.21.15.28:0/189948568 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37f3e80 0x55d0a37fbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:48.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.910+0000 7f35b2d9d640 1 --2- 172.21.15.28:0/189948568 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f34d0 0x55d0a3808830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:48.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.911+0000 7f35b359e640 1 --2- 172.21.15.28:0/189948568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37fca00 0x55d0a37fc2f0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f35ac007fd0 tx=0x7f35ac0108d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7f4bb9748f44d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.911+0000 7f35a8ff9640 1 -- 172.21.15.28:0/189948568 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac005050 con 0x55d0a37fca00 2024-04-21T08:54:48.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.911+0000 7f35a8ff9640 1 -- 172.21.15.28:0/189948568 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35ac035070 con 0x55d0a37fca00 2024-04-21T08:54:48.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.911+0000 7f35a8ff9640 1 -- 172.21.15.28:0/189948568 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac005a10 con 0x55d0a37fca00 2024-04-21T08:54:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 -- 172.21.15.28:0/189948568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37fca00 msgr2=0x55d0a37fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 --2- 172.21.15.28:0/189948568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37fca00 0x55d0a37fc2f0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f35ac007fd0 tx=0x7f35ac0108d0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 -- 172.21.15.28:0/189948568 shutdown_connections 2024-04-21T08:54:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 --2- 172.21.15.28:0/189948568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37fca00 0x55d0a37fc2f0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 --2- 172.21.15.28:0/189948568 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37f3e80 0x55d0a37fbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 --2- 172.21.15.28:0/189948568 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f34d0 0x55d0a3808830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 -- 172.21.15.28:0/189948568 >> 172.21.15.28:0/189948568 conn(0x55d0a37edba0 msgr2=0x55d0a37ecc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 -- 172.21.15.28:0/189948568 shutdown_connections 2024-04-21T08:54:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 -- 172.21.15.28:0/189948568 wait complete. 2024-04-21T08:54:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 -- start start 2024-04-21T08:54:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 Processor -- start 2024-04-21T08:54:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f3e80 0x55d0a37f68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37fca00 0x55d0a37f6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35b41ad960 con 0x55d0a37f34d0 2024-04-21T08:54:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f35b41aedd0 con 0x55d0a37fca00 2024-04-21T08:54:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f35b41ae030 con 0x55d0a37f3e80 2024-04-21T08:54:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35bbe46740 10 client.0 _ll_register_callbacks cb 0x7f35a409c2a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.912+0000 7f35b2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f35b2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37f63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34314/0 (socket says 172.21.15.28:34314) 2024-04-21T08:54:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f35b2d9d640 1 -- 172.21.15.28:0/838577868 learned_addr learned my addr 172.21.15.28:0/838577868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f35b259c640 1 --2- 172.21.15.28:0/838577868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f3e80 0x55d0a37f68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f35b359e640 1 --2- 172.21.15.28:0/838577868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37fca00 0x55d0a37f6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f35b2d9d640 1 -- 172.21.15.28:0/838577868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f3e80 msgr2=0x55d0a37f68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f35b2d9d640 1 --2- 172.21.15.28:0/838577868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f3e80 0x55d0a37f68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f35b2d9d640 1 -- 172.21.15.28:0/838577868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37fca00 msgr2=0x55d0a37f6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f35b2d9d640 1 --2- 172.21.15.28:0/838577868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37fca00 0x55d0a37f6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f35b2d9d640 1 -- 172.21.15.28:0/838577868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0a3872170 con 0x55d0a37f34d0 2024-04-21T08:54:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f35b259c640 1 --2- 172.21.15.28:0/838577868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f3e80 0x55d0a37f68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f35b2d9d640 1 --2- 172.21.15.28:0/838577868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37f63b0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f35a4000c00 tx=0x7f35a409c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4bdde70e7b9ecd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f359a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f359a7fc640 1 -- 172.21.15.28:0/838577868 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a4020040 con 0x55d0a37f34d0 2024-04-21T08:54:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f359a7fc640 1 -- 172.21.15.28:0/838577868 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f35a4018d10 con 0x55d0a37f34d0 2024-04-21T08:54:48.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f35bbe46740 1 -- 172.21.15.28:0/838577868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35ac035070 con 0x55d0a37f34d0 2024-04-21T08:54:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f359a7fc640 1 -- 172.21.15.28:0/838577868 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f35a4050050 con 0x55d0a37f34d0 2024-04-21T08:54:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f35bbe46740 10 client.16875 fetch_fsmap learned FSMap version 28 2024-04-21T08:54:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.913+0000 7f35bbe46740 1 -- 172.21.15.28:0/838577868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d0a3873470 con 0x55d0a37f34d0 2024-04-21T08:54:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.914+0000 7f359a7fc640 1 -- 172.21.15.28:0/838577868 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35ac035070 con 0x55d0a37f34d0 2024-04-21T08:54:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.914+0000 7f359a7fc640 1 -- 172.21.15.28:0/838577868 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f35a4051340 con 0x55d0a37f34d0 2024-04-21T08:54:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.914+0000 7f35bbe46740 10 client.16875 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:54:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.914+0000 7f35bbe46740 10 client.16875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.914+0000 7f35bbe46740 20 client.16875 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.914+0000 7f35bbe46740 1 --2- 172.21.15.28:0/838577868 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d0a3873c10 0x55d0a3876030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.914+0000 7f35bbe46740 4 client.16875 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:54:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.914+0000 7f35bbe46740 1 -- 172.21.15.28:0/838577868 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d0a3765c10 con 0x55d0a3873c10 2024-04-21T08:54:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.914+0000 7f35b259c640 1 --2- 172.21.15.28:0/838577868 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d0a3873c10 0x55d0a3876030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.914+0000 7f35b259c640 1 --2- 172.21.15.28:0/838577868 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d0a3873c10 0x55d0a3876030 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c26307505d616501 server_cookie=668e5c117e454300 in_seq=0 out_seq=0 2024-04-21T08:54:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.914+0000 7f359a7fc640 10 client.16875 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:48.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.915+0000 7f359a7fc640 1 -- 172.21.15.28:0/838577868 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d0a3765c10 con 0x55d0a3873c10 2024-04-21T08:54:48.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.915+0000 7f359a7fc640 10 client.16875 handle_command_reply: tid=0 2024-04-21T08:54:48.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.915+0000 7f35bbe46740 1 client.16875 shutdown 2024-04-21T08:54:48.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.915+0000 7f35bbe46740 1 -- 172.21.15.28:0/838577868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 msgr2=0x55d0a37f63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.915+0000 7f35bbe46740 1 --2- 172.21.15.28:0/838577868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37f63b0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f35a4000c00 tx=0x7f35a409c1d0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.915+0000 7f35bbe46740 1 -- 172.21.15.28:0/838577868 shutdown_connections 2024-04-21T08:54:48.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.915+0000 7f35bbe46740 1 --2- 172.21.15.28:0/838577868 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d0a3873c10 0x55d0a3876030 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.915+0000 7f35bbe46740 1 --2- 172.21.15.28:0/838577868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0a37fca00 0x55d0a37f6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.915+0000 7f359a7fc640 0 client.16875 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:48.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.915+0000 7f35bbe46740 1 --2- 172.21.15.28:0/838577868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d0a37f3e80 0x55d0a37f68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.915+0000 7f35bbe46740 1 --2- 172.21.15.28:0/838577868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d0a37f34d0 0x55d0a37f63b0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.915+0000 7f35bbe46740 1 -- 172.21.15.28:0/838577868 >> 172.21.15.28:0/838577868 conn(0x55d0a37edba0 msgr2=0x55d0a37eee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.915+0000 7f35bbe46740 1 -- 172.21.15.28:0/838577868 shutdown_connections 2024-04-21T08:54:48.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.915+0000 7f35bbe46740 1 -- 172.21.15.28:0/838577868 wait complete. 2024-04-21T08:54:48.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.915+0000 7f35bbe46740 20 client.16875 trim_cache size 0 max 16384 2024-04-21T08:54:48.921 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:54:48.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.916+0000 7f35b9255640 1 -- 172.21.15.28:0/4101070376 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f35a00a3d20 msgr2=0x7f35a00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.916+0000 7f35b9255640 1 --2- 172.21.15.28:0/4101070376 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f35a00a3d20 0x7f35a00a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f35ac01f000 tx=0x7f35ac01d000 comp rx=0 tx=0).stop 2024-04-21T08:54:48.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.916+0000 7f35b9255640 1 -- 172.21.15.28:0/4101070376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b41031d0 msgr2=0x7f35b41986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:48.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.916+0000 7f35b9255640 1 --2- 172.21.15.28:0/4101070376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b41031d0 0x7f35b41986e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f35a400d950 tx=0x7f35a400de10 comp rx=0 tx=0).stop 2024-04-21T08:54:48.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.916+0000 7f35b9255640 1 -- 172.21.15.28:0/4101070376 shutdown_connections 2024-04-21T08:54:48.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.916+0000 7f35b9255640 1 --2- 172.21.15.28:0/4101070376 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f35a00a3d20 0x7f35a00a61e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.916+0000 7f35b9255640 1 --2- 172.21.15.28:0/4101070376 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35b410c510 0x7f35b419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.916+0000 7f35b9255640 1 --2- 172.21.15.28:0/4101070376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f35b4103af0 0x7f35b4198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.916+0000 7f35b9255640 1 --2- 172.21.15.28:0/4101070376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35b41031d0 0x7f35b41986e0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:48.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.916+0000 7f35b9255640 1 -- 172.21.15.28:0/4101070376 >> 172.21.15.28:0/4101070376 conn(0x7f35b4077f50 msgr2=0x7f35b40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:48.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.916+0000 7f35b9255640 1 -- 172.21.15.28:0/4101070376 shutdown_connections 2024-04-21T08:54:48.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:48.916+0000 7f35b9255640 1 -- 172.21.15.28:0/4101070376 wait complete. 2024-04-21T08:54:49.929 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:54:49.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:54:49.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:54:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.047+0000 7fe87b9b6640 1 -- 172.21.15.28:0/1248895554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe874102190 msgr2=0x7fe874102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.047+0000 7fe87b9b6640 1 --2- 172.21.15.28:0/1248895554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe874102190 0x7fe874102570 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fe870009900 tx=0x7fe87002bc30 comp rx=0 tx=0).stop 2024-04-21T08:54:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.047+0000 7fe87b9b6640 1 -- 172.21.15.28:0/1248895554 shutdown_connections 2024-04-21T08:54:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.047+0000 7fe87b9b6640 1 --2- 172.21.15.28:0/1248895554 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe874107260 0x7fe87410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.047+0000 7fe87b9b6640 1 --2- 172.21.15.28:0/1248895554 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe874102b40 0x7fe874106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.047+0000 7fe87b9b6640 1 --2- 172.21.15.28:0/1248895554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe874102190 0x7fe874102570 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.047+0000 7fe87b9b6640 1 -- 172.21.15.28:0/1248895554 >> 172.21.15.28:0/1248895554 conn(0x7fe8740793a0 msgr2=0x7fe8740797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.047+0000 7fe87b9b6640 1 -- 172.21.15.28:0/1248895554 shutdown_connections 2024-04-21T08:54:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.047+0000 7fe87b9b6640 1 -- 172.21.15.28:0/1248895554 wait complete. 2024-04-21T08:54:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.048+0000 7fe87b9b6640 1 Processor -- start 2024-04-21T08:54:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.048+0000 7fe87b9b6640 1 -- start start 2024-04-21T08:54:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.048+0000 7fe87b9b6640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe874102190 0x7fe874198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.048+0000 7fe87b9b6640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe874107260 0x7fe874198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.048+0000 7fe87b9b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87419d020 0x7fe87419d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.048+0000 7fe87b9b6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8741121c0 con 0x7fe87419d020 2024-04-21T08:54:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.048+0000 7fe87b9b6640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe874112040 con 0x7fe874102190 2024-04-21T08:54:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.048+0000 7fe87b9b6640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe874112340 con 0x7fe874107260 2024-04-21T08:54:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.048+0000 7fe879f53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87419d020 0x7fe87419d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.048+0000 7fe879f53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87419d020 0x7fe87419d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34328/0 (socket says 172.21.15.28:34328) 2024-04-21T08:54:50.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.048+0000 7fe879f53640 1 -- 172.21.15.28:0/1958636725 learned_addr learned my addr 172.21.15.28:0/1958636725 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.048+0000 7fe879752640 1 --2- 172.21.15.28:0/1958636725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe874102190 0x7fe874198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.048+0000 7fe878f51640 1 --2- 172.21.15.28:0/1958636725 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe874107260 0x7fe874198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.048+0000 7fe879f53640 1 -- 172.21.15.28:0/1958636725 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe874107260 msgr2=0x7fe874198d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.048+0000 7fe879f53640 1 --2- 172.21.15.28:0/1958636725 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe874107260 0x7fe874198d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.049+0000 7fe879f53640 1 -- 172.21.15.28:0/1958636725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe874102190 msgr2=0x7fe874198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.049+0000 7fe879f53640 1 --2- 172.21.15.28:0/1958636725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe874102190 0x7fe874198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.049+0000 7fe879f53640 1 -- 172.21.15.28:0/1958636725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87419db10 con 0x7fe87419d020 2024-04-21T08:54:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.049+0000 7fe878f51640 1 --2- 172.21.15.28:0/1958636725 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe874107260 0x7fe874198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.049+0000 7fe879f53640 1 --2- 172.21.15.28:0/1958636725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87419d020 0x7fe87419d4d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009fd0 tx=0x7fe86c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.049+0000 7fe86a7fc640 1 -- 172.21.15.28:0/1958636725 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c017070 con 0x7fe87419d020 2024-04-21T08:54:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.049+0000 7fe87b9b6640 1 -- 172.21.15.28:0/1958636725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe87419de00 con 0x7fe87419d020 2024-04-21T08:54:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.049+0000 7fe86a7fc640 1 -- 172.21.15.28:0/1958636725 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe86c003e20 con 0x7fe87419d020 2024-04-21T08:54:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.049+0000 7fe86a7fc640 1 -- 172.21.15.28:0/1958636725 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c010460 con 0x7fe87419d020 2024-04-21T08:54:50.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.049+0000 7fe87b9b6640 1 -- 172.21.15.28:0/1958636725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8741a55a0 con 0x7fe87419d020 2024-04-21T08:54:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.051+0000 7fe86a7fc640 1 -- 172.21.15.28:0/1958636725 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe86c003970 con 0x7fe87419d020 2024-04-21T08:54:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.052+0000 7fe87e5a9740 1 Processor -- start 2024-04-21T08:54:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.052+0000 7fe87e5a9740 1 -- start start 2024-04-21T08:54:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.052+0000 7fe87e5a9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c6971378e0 0x55c697137cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.052+0000 7fe87e5a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697138290 0x55c697140840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.052+0000 7fe87e5a9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c697140e10 0x55c697143260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.052+0000 7fe87e5a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c69709bf00 con 0x55c697138290 2024-04-21T08:54:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.052+0000 7fe87e5a9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c697039590 con 0x55c697140e10 2024-04-21T08:54:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.052+0000 7fe87e5a9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55c697097ec0 con 0x55c6971378e0 2024-04-21T08:54:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.052+0000 7fe879752640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697138290 0x55c697140840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.052+0000 7fe878f51640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c697140e10 0x55c697143260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.052+0000 7fe879752640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697138290 0x55c697140840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34338/0 (socket says 172.21.15.28:34338) 2024-04-21T08:54:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.052+0000 7fe879752640 1 -- 172.21.15.28:0/612863900 learned_addr learned my addr 172.21.15.28:0/612863900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.052+0000 7fe86a7fc640 1 --2- 172.21.15.28:0/1958636725 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe8600a3d20 0x7fe8600a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.052+0000 7fe878f51640 1 --2- 172.21.15.28:0/612863900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c6971378e0 0x55c697137cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.052+0000 7fe879752640 1 -- 172.21.15.28:0/612863900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c6971378e0 msgr2=0x55c697137cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.053+0000 7fe879752640 1 --2- 172.21.15.28:0/612863900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c6971378e0 0x55c697137cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.053+0000 7fe879f53640 1 --2- 172.21.15.28:0/1958636725 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe8600a3d20 0x7fe8600a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.053+0000 7fe879752640 1 -- 172.21.15.28:0/612863900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c697140e10 msgr2=0x55c697143260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.053+0000 7fe879752640 1 --2- 172.21.15.28:0/612863900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c697140e10 0x55c697143260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.053+0000 7fe879752640 1 -- 172.21.15.28:0/612863900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c696e767e0 con 0x55c697138290 2024-04-21T08:54:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.053+0000 7fe878f51640 1 --2- 172.21.15.28:0/612863900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c697140e10 0x55c697143260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.053+0000 7fe86a7fc640 1 -- 172.21.15.28:0/1958636725 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe86c0d08e0 con 0x7fe87419d020 2024-04-21T08:54:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.053+0000 7fe879752640 1 --2- 172.21.15.28:0/612863900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697138290 0x55c697140840 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fe870009900 tx=0x7fe87002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5501b7327373389 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.053+0000 7fe879f53640 1 --2- 172.21.15.28:0/1958636725 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe8600a3d20 0x7fe8600a61e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0cdd40 tx=0x7fe86c06bdc0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.053+0000 7fe85f7fe640 1 -- 172.21.15.28:0/612863900 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe870045020 con 0x55c697138290 2024-04-21T08:54:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.053+0000 7fe85f7fe640 1 -- 172.21.15.28:0/612863900 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe87002c7e0 con 0x55c697138290 2024-04-21T08:54:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.053+0000 7fe85f7fe640 1 -- 172.21.15.28:0/612863900 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8700046c0 con 0x55c697138290 2024-04-21T08:54:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.054+0000 7fe87e5a9740 1 -- 172.21.15.28:0/612863900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697138290 msgr2=0x55c697140840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.054+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/612863900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697138290 0x55c697140840 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fe870009900 tx=0x7fe87002c6c0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.054+0000 7fe87e5a9740 1 -- 172.21.15.28:0/612863900 shutdown_connections 2024-04-21T08:54:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.054+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/612863900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c697140e10 0x55c697143260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.054+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/612863900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697138290 0x55c697140840 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.054+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/612863900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c6971378e0 0x55c697137cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.054+0000 7fe87e5a9740 1 -- 172.21.15.28:0/612863900 >> 172.21.15.28:0/612863900 conn(0x55c697131fb0 msgr2=0x55c697132390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:50.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.054+0000 7fe87e5a9740 1 -- 172.21.15.28:0/612863900 shutdown_connections 2024-04-21T08:54:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.054+0000 7fe87e5a9740 1 -- 172.21.15.28:0/612863900 wait complete. 2024-04-21T08:54:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.054+0000 7fe87e5a9740 1 -- start start 2024-04-21T08:54:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.054+0000 7fe87e5a9740 1 Processor -- start 2024-04-21T08:54:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe87e5a9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6971378e0 0x55c69713de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe87e5a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697138290 0x55c69713e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe87e5a9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697140e10 0x55c69713e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe87e5a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c697143d20 con 0x55c697138290 2024-04-21T08:54:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe87e5a9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c697055580 con 0x55c6971378e0 2024-04-21T08:54:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe87e5a9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55c697143ea0 con 0x55c697140e10 2024-04-21T08:54:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe87e5a9740 10 client.0 _ll_register_callbacks cb 0x55c69705b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe879752640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697138290 0x55c69713e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe879752640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697138290 0x55c69713e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34344/0 (socket says 172.21.15.28:34344) 2024-04-21T08:54:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe879752640 1 -- 172.21.15.28:0/1810710997 learned_addr learned my addr 172.21.15.28:0/1810710997 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe878f51640 1 --2- 172.21.15.28:0/1810710997 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697140e10 0x55c69713e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe878f51640 1 --2- 172.21.15.28:0/1810710997 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6971378e0 0x55c69713de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe879752640 1 -- 172.21.15.28:0/1810710997 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697140e10 msgr2=0x55c69713e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe879752640 1 --2- 172.21.15.28:0/1810710997 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697140e10 0x55c69713e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe879752640 1 -- 172.21.15.28:0/1810710997 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6971378e0 msgr2=0x55c69713de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe879752640 1 --2- 172.21.15.28:0/1810710997 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6971378e0 0x55c69713de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe879752640 1 -- 172.21.15.28:0/1810710997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c696ed6f70 con 0x55c697138290 2024-04-21T08:54:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe878f51640 1 --2- 172.21.15.28:0/1810710997 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697140e10 0x55c69713e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe879752640 1 --2- 172.21.15.28:0/1810710997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697138290 0x55c69713e350 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fe87002c170 tx=0x7fe87002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17e9816d66f2307 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe85cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe85cff9640 1 -- 172.21.15.28:0/1810710997 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe870045030 con 0x55c697138290 2024-04-21T08:54:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe85cff9640 1 -- 172.21.15.28:0/1810710997 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe8700035f0 con 0x55c697138290 2024-04-21T08:54:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.055+0000 7fe87e5a9740 1 -- 172.21.15.28:0/1810710997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe87002c7e0 con 0x55c697138290 2024-04-21T08:54:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.056+0000 7fe85cff9640 1 -- 172.21.15.28:0/1810710997 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe87005c050 con 0x55c697138290 2024-04-21T08:54:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.056+0000 7fe87e5a9740 10 client.16887 fetch_fsmap learned FSMap version 28 2024-04-21T08:54:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.056+0000 7fe87e5a9740 1 -- 172.21.15.28:0/1810710997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c696d6e170 con 0x55c697138290 2024-04-21T08:54:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.056+0000 7fe85cff9640 1 -- 172.21.15.28:0/1810710997 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe87002c7e0 con 0x55c697138290 2024-04-21T08:54:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.056+0000 7fe85cff9640 1 -- 172.21.15.28:0/1810710997 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe870004bf0 con 0x55c697138290 2024-04-21T08:54:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.056+0000 7fe87e5a9740 10 client.16887 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:54:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.056+0000 7fe87e5a9740 10 client.16887 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.056+0000 7fe87e5a9740 20 client.16887 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.056+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/1810710997 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c697151190 0x55c697153580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.056+0000 7fe87e5a9740 4 client.16887 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.056+0000 7fe87e5a9740 1 -- 172.21.15.28:0/1810710997 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6970a9920 con 0x55c697151190 2024-04-21T08:54:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.056+0000 7fe878f51640 1 --2- 172.21.15.28:0/1810710997 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c697151190 0x55c697153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.057+0000 7fe878f51640 1 --2- 172.21.15.28:0/1810710997 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c697151190 0x55c697153580 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b12c736ef612582 server_cookie=74536c04d0658803 in_seq=0 out_seq=0 2024-04-21T08:54:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.057+0000 7fe85cff9640 10 client.16887 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.058+0000 7fe85cff9640 1 -- 172.21.15.28:0/1810710997 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c6970a9920 con 0x55c697151190 2024-04-21T08:54:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.058+0000 7fe85cff9640 10 client.16887 handle_command_reply: tid=0 2024-04-21T08:54:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.058+0000 7fe87e5a9740 1 client.16887 shutdown 2024-04-21T08:54:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.058+0000 7fe87e5a9740 1 -- 172.21.15.28:0/1810710997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697138290 msgr2=0x55c69713e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.058+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/1810710997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697138290 0x55c69713e350 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fe87002c170 tx=0x7fe87002bc30 comp rx=0 tx=0).stop 2024-04-21T08:54:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.058+0000 7fe87e5a9740 1 -- 172.21.15.28:0/1810710997 shutdown_connections 2024-04-21T08:54:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.058+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/1810710997 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c697151190 0x55c697153580 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.058+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/1810710997 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697140e10 0x55c69713e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.058+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/1810710997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697138290 0x55c69713e350 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.058+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/1810710997 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6971378e0 0x55c69713de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.058+0000 7fe85cff9640 0 client.16887 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.058+0000 7fe87e5a9740 1 -- 172.21.15.28:0/1810710997 >> 172.21.15.28:0/1810710997 conn(0x55c697131fb0 msgr2=0x55c697133d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.058+0000 7fe87e5a9740 1 -- 172.21.15.28:0/1810710997 shutdown_connections 2024-04-21T08:54:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.058+0000 7fe87e5a9740 1 -- 172.21.15.28:0/1810710997 wait complete. 2024-04-21T08:54:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.058+0000 7fe87e5a9740 20 client.16887 trim_cache size 0 max 16384 2024-04-21T08:54:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.076+0000 7fe87e5a9740 1 Processor -- start 2024-04-21T08:54:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.076+0000 7fe87e5a9740 1 -- start start 2024-04-21T08:54:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.076+0000 7fe87e5a9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6971378e0 0x55c697139560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.076+0000 7fe87e5a9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697138290 0x55c69714cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.076+0000 7fe87e5a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697140e10 0x55c69713d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.076+0000 7fe87e5a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8741b1ad0 con 0x55c697140e10 2024-04-21T08:54:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.076+0000 7fe87e5a9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe8741af3a0 con 0x55c6971378e0 2024-04-21T08:54:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.076+0000 7fe87e5a9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe8741ad1f0 con 0x55c697138290 2024-04-21T08:54:50.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.076+0000 7fe878f51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697140e10 0x55c69713d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.076+0000 7fe878f51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697140e10 0x55c69713d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34356/0 (socket says 172.21.15.28:34356) 2024-04-21T08:54:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.076+0000 7fe878f51640 1 -- 172.21.15.28:0/1673027642 learned_addr learned my addr 172.21.15.28:0/1673027642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.076+0000 7fe878f51640 1 --2- 172.21.15.28:0/1673027642 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6971378e0 0x55c697139560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.076+0000 7fe879752640 1 --2- 172.21.15.28:0/1673027642 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697138290 0x55c69714cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.077+0000 7fe878f51640 1 -- 172.21.15.28:0/1673027642 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697138290 msgr2=0x55c69714cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.077+0000 7fe878f51640 1 --2- 172.21.15.28:0/1673027642 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697138290 0x55c69714cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.077+0000 7fe878f51640 1 -- 172.21.15.28:0/1673027642 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6971378e0 msgr2=0x55c697139560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.077+0000 7fe878f51640 1 --2- 172.21.15.28:0/1673027642 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6971378e0 0x55c697139560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.077+0000 7fe878f51640 1 -- 172.21.15.28:0/1673027642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c697138bf0 con 0x55c697140e10 2024-04-21T08:54:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.077+0000 7fe878f51640 1 --2- 172.21.15.28:0/1673027642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697140e10 0x55c69713d6f0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fe874101950 tx=0x7fe8640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7a6e031baf9d226 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.077+0000 7fe85effd640 1 -- 172.21.15.28:0/1673027642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe864011700 con 0x55c697140e10 2024-04-21T08:54:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.077+0000 7fe85effd640 1 -- 172.21.15.28:0/1673027642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe864011d50 con 0x55c697140e10 2024-04-21T08:54:50.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.077+0000 7fe85effd640 1 -- 172.21.15.28:0/1673027642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86400fdb0 con 0x55c697140e10 2024-04-21T08:54:50.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.078+0000 7fe87e5a9740 1 -- 172.21.15.28:0/1673027642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697140e10 msgr2=0x55c69713d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.078+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/1673027642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697140e10 0x55c69713d6f0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fe874101950 tx=0x7fe8640046c0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.078+0000 7fe87e5a9740 1 -- 172.21.15.28:0/1673027642 shutdown_connections 2024-04-21T08:54:50.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.078+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/1673027642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697140e10 0x55c69713d6f0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.078+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/1673027642 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697138290 0x55c69714cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.078+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/1673027642 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6971378e0 0x55c697139560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.078+0000 7fe87e5a9740 1 -- 172.21.15.28:0/1673027642 >> 172.21.15.28:0/1673027642 conn(0x55c697131fb0 msgr2=0x55c69713ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:50.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.078+0000 7fe87e5a9740 1 -- 172.21.15.28:0/1673027642 shutdown_connections 2024-04-21T08:54:50.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.078+0000 7fe87e5a9740 1 -- 172.21.15.28:0/1673027642 wait complete. 2024-04-21T08:54:50.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.078+0000 7fe87e5a9740 1 -- start start 2024-04-21T08:54:50.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.078+0000 7fe87e5a9740 1 Processor -- start 2024-04-21T08:54:50.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.078+0000 7fe87e5a9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6971378e0 0x55c6971495c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.078+0000 7fe87e5a9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697138290 0x55c697144980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.078+0000 7fe87e5a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697140e10 0x55c697144f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe87e5a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8741ad860 con 0x55c697140e10 2024-04-21T08:54:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe87e5a9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe8741aecd0 con 0x55c6971378e0 2024-04-21T08:54:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe87e5a9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe8741adf30 con 0x55c697138290 2024-04-21T08:54:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe87e5a9740 10 client.0 _ll_register_callbacks cb 0x7fe86400c700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe878f51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697140e10 0x55c697144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe878f51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697140e10 0x55c697144f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34360/0 (socket says 172.21.15.28:34360) 2024-04-21T08:54:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe878f51640 1 -- 172.21.15.28:0/301779163 learned_addr learned my addr 172.21.15.28:0/301779163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe878f51640 1 --2- 172.21.15.28:0/301779163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6971378e0 0x55c6971495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe879752640 1 --2- 172.21.15.28:0/301779163 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697138290 0x55c697144980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe878f51640 1 -- 172.21.15.28:0/301779163 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697138290 msgr2=0x55c697144980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe878f51640 1 --2- 172.21.15.28:0/301779163 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697138290 0x55c697144980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe878f51640 1 -- 172.21.15.28:0/301779163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6971378e0 msgr2=0x55c6971495c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe878f51640 1 --2- 172.21.15.28:0/301779163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6971378e0 0x55c6971495c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe878f51640 1 -- 172.21.15.28:0/301779163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c697145790 con 0x55c697140e10 2024-04-21T08:54:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe878f51640 1 --2- 172.21.15.28:0/301779163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697140e10 0x55c697144f90 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe8640214d0 tx=0x7fe86400f990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4899c719ad91a47 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe85cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe85cff9640 1 -- 172.21.15.28:0/301779163 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe864035040 con 0x55c697140e10 2024-04-21T08:54:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe85cff9640 1 -- 172.21.15.28:0/301779163 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe86400fa30 con 0x55c697140e10 2024-04-21T08:54:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe87e5a9740 1 -- 172.21.15.28:0/301779163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe864011d50 con 0x55c697140e10 2024-04-21T08:54:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe85cff9640 1 -- 172.21.15.28:0/301779163 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe86400fc30 con 0x55c697140e10 2024-04-21T08:54:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe87e5a9740 10 client.16893 fetch_fsmap learned FSMap version 28 2024-04-21T08:54:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.079+0000 7fe87e5a9740 1 -- 172.21.15.28:0/301779163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c6971b74d0 con 0x55c697140e10 2024-04-21T08:54:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.080+0000 7fe85cff9640 1 -- 172.21.15.28:0/301779163 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe864011d50 con 0x55c697140e10 2024-04-21T08:54:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.080+0000 7fe85cff9640 1 -- 172.21.15.28:0/301779163 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe86402c540 con 0x55c697140e10 2024-04-21T08:54:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.080+0000 7fe87e5a9740 10 client.16893 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:54:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.080+0000 7fe87e5a9740 10 client.16893 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.080+0000 7fe87e5a9740 20 client.16893 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.080+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/301779163 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c6971b7b10 0x55c6971b9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.080+0000 7fe87e5a9740 4 client.16893 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:54:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.080+0000 7fe87e5a9740 1 -- 172.21.15.28:0/301779163 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c6970a3790 con 0x55c6971b7b10 2024-04-21T08:54:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.080+0000 7fe879752640 1 --2- 172.21.15.28:0/301779163 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c6971b7b10 0x55c6971b9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.081+0000 7fe85cff9640 10 client.16893 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.081+0000 7fe879752640 1 --2- 172.21.15.28:0/301779163 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c6971b7b10 0x55c6971b9f30 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcd14fad304aa61e server_cookie=1dff386745a3a4d7 in_seq=0 out_seq=0 2024-04-21T08:54:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.082+0000 7fe85cff9640 1 -- 172.21.15.28:0/301779163 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c6970a3790 con 0x55c6971b7b10 2024-04-21T08:54:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.082+0000 7fe85cff9640 10 client.16893 handle_command_reply: tid=0 2024-04-21T08:54:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.082+0000 7fe87e5a9740 1 client.16893 shutdown 2024-04-21T08:54:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.082+0000 7fe87e5a9740 1 -- 172.21.15.28:0/301779163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697140e10 msgr2=0x55c697144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.082+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/301779163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697140e10 0x55c697144f90 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe8640214d0 tx=0x7fe86400f990 comp rx=0 tx=0).stop 2024-04-21T08:54:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.082+0000 7fe87e5a9740 1 -- 172.21.15.28:0/301779163 shutdown_connections 2024-04-21T08:54:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.082+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/301779163 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c6971b7b10 0x55c6971b9f30 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.082+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/301779163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c697140e10 0x55c697144f90 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.082+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/301779163 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c697138290 0x55c697144980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.082+0000 7fe87e5a9740 1 --2- 172.21.15.28:0/301779163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6971378e0 0x55c6971495c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.082+0000 7fe87e5a9740 1 -- 172.21.15.28:0/301779163 >> 172.21.15.28:0/301779163 conn(0x55c697131fb0 msgr2=0x55c69713e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:50.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.082+0000 7fe85cff9640 0 client.16893 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:50.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.082+0000 7fe87e5a9740 1 -- 172.21.15.28:0/301779163 shutdown_connections 2024-04-21T08:54:50.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.082+0000 7fe87e5a9740 1 -- 172.21.15.28:0/301779163 wait complete. 2024-04-21T08:54:50.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.082+0000 7fe87e5a9740 20 client.16893 trim_cache size 0 max 16384 2024-04-21T08:54:50.087 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:54:50.087 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:54:50.087 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "8b485e2a-5339-4b65-a825-69e1706aba4a", 2024-04-21T08:54:50.087 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:54:50.087 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:54:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.083+0000 7fe87b9b6640 1 -- 172.21.15.28:0/1958636725 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe8600a3d20 msgr2=0x7fe8600a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.083+0000 7fe87b9b6640 1 --2- 172.21.15.28:0/1958636725 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe8600a3d20 0x7fe8600a61e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe86c0cdd40 tx=0x7fe86c06bdc0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.083+0000 7fe87b9b6640 1 -- 172.21.15.28:0/1958636725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87419d020 msgr2=0x7fe87419d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.083+0000 7fe87b9b6640 1 --2- 172.21.15.28:0/1958636725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87419d020 0x7fe87419d4d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009fd0 tx=0x7fe86c00b260 comp rx=0 tx=0).stop 2024-04-21T08:54:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.083+0000 7fe87b9b6640 1 -- 172.21.15.28:0/1958636725 shutdown_connections 2024-04-21T08:54:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.083+0000 7fe87b9b6640 1 --2- 172.21.15.28:0/1958636725 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe8600a3d20 0x7fe8600a61e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.083+0000 7fe87b9b6640 1 --2- 172.21.15.28:0/1958636725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87419d020 0x7fe87419d4d0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.083+0000 7fe87b9b6640 1 --2- 172.21.15.28:0/1958636725 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe874107260 0x7fe874198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.083+0000 7fe87b9b6640 1 --2- 172.21.15.28:0/1958636725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe874102190 0x7fe874198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.083+0000 7fe87b9b6640 1 -- 172.21.15.28:0/1958636725 >> 172.21.15.28:0/1958636725 conn(0x7fe8740793a0 msgr2=0x7fe8741000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.083+0000 7fe87b9b6640 1 -- 172.21.15.28:0/1958636725 shutdown_connections 2024-04-21T08:54:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.083+0000 7fe87b9b6640 1 -- 172.21.15.28:0/1958636725 wait complete. 2024-04-21T08:54:50.095 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:54:50.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:54:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.202+0000 7f2005d7e640 1 -- 172.21.15.28:0/1126076734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2000101080 msgr2=0x7f2000101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.202+0000 7f2005d7e640 1 --2- 172.21.15.28:0/1126076734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2000101080 0x7f2000101460 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0009900 tx=0x7f1ff002bc30 comp rx=0 tx=0).stop 2024-04-21T08:54:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.202+0000 7f2005d7e640 1 -- 172.21.15.28:0/1126076734 shutdown_connections 2024-04-21T08:54:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.202+0000 7f2005d7e640 1 --2- 172.21.15.28:0/1126076734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2000106150 0x7f200010dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.202+0000 7f2005d7e640 1 --2- 172.21.15.28:0/1126076734 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2000101a30 0x7f2000105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.202+0000 7f2005d7e640 1 --2- 172.21.15.28:0/1126076734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2000101080 0x7f2000101460 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.202+0000 7f2005d7e640 1 -- 172.21.15.28:0/1126076734 >> 172.21.15.28:0/1126076734 conn(0x7f2000078000 msgr2=0x7f2000078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.203+0000 7f2005d7e640 1 -- 172.21.15.28:0/1126076734 shutdown_connections 2024-04-21T08:54:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.203+0000 7f2005d7e640 1 -- 172.21.15.28:0/1126076734 wait complete. 2024-04-21T08:54:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.203+0000 7f2005d7e640 1 Processor -- start 2024-04-21T08:54:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.203+0000 7f2005d7e640 1 -- start start 2024-04-21T08:54:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f2005d7e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2000101080 0x7f20001987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f2005d7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2000106150 0x7f2000198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f2005d7e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f200019bfb0 0x7f200019c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f2005d7e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20001110b0 con 0x7f2000106150 2024-04-21T08:54:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f2005d7e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2000110f30 con 0x7f2000101080 2024-04-21T08:54:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f2005d7e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2000111230 con 0x7f200019bfb0 2024-04-21T08:54:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f1ffeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2000106150 0x7f2000198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f1ffeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2000106150 0x7f2000198ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34376/0 (socket says 172.21.15.28:34376) 2024-04-21T08:54:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f1ffeffd640 1 -- 172.21.15.28:0/4078479653 learned_addr learned my addr 172.21.15.28:0/4078479653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f1ffffff640 1 --2- 172.21.15.28:0/4078479653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f200019bfb0 0x7f200019c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f1fff7fe640 1 --2- 172.21.15.28:0/4078479653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2000101080 0x7f20001987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f1ffeffd640 1 -- 172.21.15.28:0/4078479653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f200019bfb0 msgr2=0x7f200019c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/4078479653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f200019bfb0 0x7f200019c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f1ffeffd640 1 -- 172.21.15.28:0/4078479653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2000101080 msgr2=0x7f20001987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/4078479653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2000101080 0x7f20001987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f1ffeffd640 1 -- 172.21.15.28:0/4078479653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200019caa0 con 0x7f2000106150 2024-04-21T08:54:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/4078479653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2000106150 0x7f2000198ce0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f1ff400e9f0 tx=0x7f1ff400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f1ffcff9640 1 -- 172.21.15.28:0/4078479653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff400ccc0 con 0x7f2000106150 2024-04-21T08:54:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f1ffcff9640 1 -- 172.21.15.28:0/4078479653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff40054d0 con 0x7f2000106150 2024-04-21T08:54:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f1ffcff9640 1 -- 172.21.15.28:0/4078479653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4010430 con 0x7f2000106150 2024-04-21T08:54:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.204+0000 7f2005d7e640 1 -- 172.21.15.28:0/4078479653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f200019cd90 con 0x7f2000106150 2024-04-21T08:54:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.205+0000 7f2005d7e640 1 -- 172.21.15.28:0/4078479653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20001a55e0 con 0x7f2000106150 2024-04-21T08:54:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.207+0000 7f1ffcff9640 1 -- 172.21.15.28:0/4078479653 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1ff40046c0 con 0x7f2000106150 2024-04-21T08:54:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.207+0000 7f200896c740 1 Processor -- start 2024-04-21T08:54:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.207+0000 7f200896c740 1 -- start start 2024-04-21T08:54:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.207+0000 7f200896c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a8b580 0x563445a8b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.207+0000 7f200896c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8bea0 0x563445a94540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.207+0000 7f200896c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a94a80 0x563445a96e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.207+0000 7f200896c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5634459f01e0 con 0x563445a94a80 2024-04-21T08:54:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.207+0000 7f200896c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56344598d390 con 0x563445a8bea0 2024-04-21T08:54:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.207+0000 7f200896c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5634459ec1a0 con 0x563445a8b580 2024-04-21T08:54:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.207+0000 7f1ffcff9640 1 --2- 172.21.15.28:0/4078479653 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1fe80a3d20 0x7f1fe80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.207+0000 7f1ffeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a94a80 0x563445a96e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.207+0000 7f1ffeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a94a80 0x563445a96e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34390/0 (socket says 172.21.15.28:34390) 2024-04-21T08:54:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.207+0000 7f1ffeffd640 1 -- 172.21.15.28:0/3947986990 learned_addr learned my addr 172.21.15.28:0/3947986990 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.207+0000 7f1ffffff640 1 --2- 172.21.15.28:0/3947986990 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a8b580 0x563445a8b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.207+0000 7f1fff7fe640 1 --2- 172.21.15.28:0/3947986990 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8bea0 0x563445a94540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.208+0000 7f1ffffff640 1 --2- 172.21.15.28:0/4078479653 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1fe80a3d20 0x7f1fe80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.208+0000 7f1ffcff9640 1 -- 172.21.15.28:0/4078479653 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1ff40d0ed0 con 0x7f2000106150 2024-04-21T08:54:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.208+0000 7f1ffeffd640 1 -- 172.21.15.28:0/3947986990 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a8b580 msgr2=0x563445a8b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.208+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/3947986990 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a8b580 0x563445a8b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.208+0000 7f1ffeffd640 1 -- 172.21.15.28:0/3947986990 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8bea0 msgr2=0x563445a94540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.208+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/3947986990 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8bea0 0x563445a94540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.208+0000 7f1ffeffd640 1 -- 172.21.15.28:0/3947986990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634457cc7d0 con 0x563445a94a80 2024-04-21T08:54:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.208+0000 7f1ffffff640 1 --2- 172.21.15.28:0/3947986990 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a8b580 0x563445a8b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.208+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/3947986990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a94a80 0x563445a96e70 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40a5210 tx=0x7f1ff4004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6bd72b9a4827fd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.208+0000 7f1fe5ffb640 1 -- 172.21.15.28:0/3947986990 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff409cad0 con 0x563445a94a80 2024-04-21T08:54:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.208+0000 7f1ffffff640 1 --2- 172.21.15.28:0/4078479653 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1fe80a3d20 0x7f1fe80a61e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8009fd0 tx=0x7f1ff80096a0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.208+0000 7f1fe5ffb640 1 -- 172.21.15.28:0/3947986990 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff4005680 con 0x563445a94a80 2024-04-21T08:54:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.208+0000 7f1fe5ffb640 1 -- 172.21.15.28:0/3947986990 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4099140 con 0x563445a94a80 2024-04-21T08:54:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.209+0000 7f200896c740 1 -- 172.21.15.28:0/3947986990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a94a80 msgr2=0x563445a96e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.209+0000 7f200896c740 1 --2- 172.21.15.28:0/3947986990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a94a80 0x563445a96e70 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40a5210 tx=0x7f1ff4004a60 comp rx=0 tx=0).stop 2024-04-21T08:54:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.209+0000 7f200896c740 1 -- 172.21.15.28:0/3947986990 shutdown_connections 2024-04-21T08:54:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.209+0000 7f200896c740 1 --2- 172.21.15.28:0/3947986990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a94a80 0x563445a96e70 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.209+0000 7f200896c740 1 --2- 172.21.15.28:0/3947986990 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8bea0 0x563445a94540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.209+0000 7f200896c740 1 --2- 172.21.15.28:0/3947986990 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a8b580 0x563445a8b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.209+0000 7f200896c740 1 -- 172.21.15.28:0/3947986990 >> 172.21.15.28:0/3947986990 conn(0x563445a85cb0 msgr2=0x563445a86090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.209+0000 7f200896c740 1 -- 172.21.15.28:0/3947986990 shutdown_connections 2024-04-21T08:54:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.209+0000 7f200896c740 1 -- 172.21.15.28:0/3947986990 wait complete. 2024-04-21T08:54:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.209+0000 7f200896c740 1 -- start start 2024-04-21T08:54:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.209+0000 7f200896c740 1 Processor -- start 2024-04-21T08:54:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f200896c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a8b580 0x563445a91ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f200896c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8bea0 0x563445a92000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f200896c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a94a80 0x563445a92540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f200896c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563445a97870 con 0x563445a8bea0 2024-04-21T08:54:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f200896c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5634459a92f0 con 0x563445a94a80 2024-04-21T08:54:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f200896c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x563445a979f0 con 0x563445a8b580 2024-04-21T08:54:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f200896c740 10 client.0 _ll_register_callbacks cb 0x5634459b6580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f1ffeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8bea0 0x563445a92000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f1ffeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8bea0 0x563445a92000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34398/0 (socket says 172.21.15.28:34398) 2024-04-21T08:54:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f1ffeffd640 1 -- 172.21.15.28:0/192283070 learned_addr learned my addr 172.21.15.28:0/192283070 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f1fff7fe640 1 --2- 172.21.15.28:0/192283070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a8b580 0x563445a91ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f1ffffff640 1 --2- 172.21.15.28:0/192283070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a94a80 0x563445a92540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f1ffeffd640 1 -- 172.21.15.28:0/192283070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a8b580 msgr2=0x563445a91ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/192283070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a8b580 0x563445a91ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f1ffeffd640 1 -- 172.21.15.28:0/192283070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a94a80 msgr2=0x563445a92540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/192283070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a94a80 0x563445a92540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f1ffeffd640 1 -- 172.21.15.28:0/192283070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634457023f0 con 0x563445a8bea0 2024-04-21T08:54:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f1ffffff640 1 --2- 172.21.15.28:0/192283070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a94a80 0x563445a92540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/192283070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8bea0 0x563445a92000 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f1ff409c7d0 tx=0x7f1ff40ce270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1133bc807505e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:50.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f1fdeffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.210+0000 7f1fdeffd640 1 -- 172.21.15.28:0/192283070 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff409cf30 con 0x563445a8bea0 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.211+0000 7f1fdeffd640 1 -- 172.21.15.28:0/192283070 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1ff4005000 con 0x563445a8bea0 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.211+0000 7f200896c740 1 -- 172.21.15.28:0/192283070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ff4005680 con 0x563445a8bea0 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.211+0000 7f1fdeffd640 1 -- 172.21.15.28:0/192283070 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1ff409d1d0 con 0x563445a8bea0 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.211+0000 7f200896c740 10 client.16905 fetch_fsmap learned FSMap version 28 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.211+0000 7f200896c740 1 -- 172.21.15.28:0/192283070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5634456c20f0 con 0x563445a8bea0 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.211+0000 7f1fdeffd640 1 -- 172.21.15.28:0/192283070 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff4005680 con 0x563445a8bea0 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.211+0000 7f1fdeffd640 1 -- 172.21.15.28:0/192283070 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f1ff4098820 con 0x563445a8bea0 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.211+0000 7f200896c740 10 client.16905 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.211+0000 7f200896c740 10 client.16905 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.211+0000 7f200896c740 20 client.16905 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.211+0000 7f200896c740 1 --2- 172.21.15.28:0/192283070 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563445aa4900 0x563445aa6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.211+0000 7f200896c740 4 client.16905 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.211+0000 7f200896c740 1 -- 172.21.15.28:0/192283070 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634459fdc00 con 0x563445aa4900 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.211+0000 7f1fff7fe640 1 --2- 172.21.15.28:0/192283070 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563445aa4900 0x563445aa6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.212+0000 7f1fff7fe640 1 --2- 172.21.15.28:0/192283070 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563445aa4900 0x563445aa6cf0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5f92b36db270edf server_cookie=a8d03977599b3bbf in_seq=0 out_seq=0 2024-04-21T08:54:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.212+0000 7f1fdeffd640 10 client.16905 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.212+0000 7f1fdeffd640 1 -- 172.21.15.28:0/192283070 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5634459fdc00 con 0x563445aa4900 2024-04-21T08:54:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.212+0000 7f1fdeffd640 10 client.16905 handle_command_reply: tid=0 2024-04-21T08:54:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.213+0000 7f200896c740 1 client.16905 shutdown 2024-04-21T08:54:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.213+0000 7f200896c740 1 -- 172.21.15.28:0/192283070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8bea0 msgr2=0x563445a92000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.213+0000 7f200896c740 1 --2- 172.21.15.28:0/192283070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8bea0 0x563445a92000 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f1ff409c7d0 tx=0x7f1ff40ce270 comp rx=0 tx=0).stop 2024-04-21T08:54:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.213+0000 7f200896c740 1 -- 172.21.15.28:0/192283070 shutdown_connections 2024-04-21T08:54:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.213+0000 7f200896c740 1 --2- 172.21.15.28:0/192283070 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563445aa4900 0x563445aa6cf0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.213+0000 7f200896c740 1 --2- 172.21.15.28:0/192283070 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a94a80 0x563445a92540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.213+0000 7f200896c740 1 --2- 172.21.15.28:0/192283070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8bea0 0x563445a92000 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.213+0000 7f1fdeffd640 0 client.16905 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.213+0000 7f200896c740 1 --2- 172.21.15.28:0/192283070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a8b580 0x563445a91ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.213+0000 7f200896c740 1 -- 172.21.15.28:0/192283070 >> 172.21.15.28:0/192283070 conn(0x563445a85cb0 msgr2=0x563445a87a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.213+0000 7f200896c740 1 -- 172.21.15.28:0/192283070 shutdown_connections 2024-04-21T08:54:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.213+0000 7f200896c740 1 -- 172.21.15.28:0/192283070 wait complete. 2024-04-21T08:54:50.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.213+0000 7f200896c740 20 client.16905 trim_cache size 0 max 16384 2024-04-21T08:54:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.230+0000 7f200896c740 1 Processor -- start 2024-04-21T08:54:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.230+0000 7f200896c740 1 -- start start 2024-04-21T08:54:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.230+0000 7f200896c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8b580 0x563445aaa100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.230+0000 7f200896c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8bea0 0x563445a8ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.230+0000 7f200896c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a94a80 0x563445a8f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.230+0000 7f200896c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20001b1c50 con 0x563445a8bea0 2024-04-21T08:54:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.230+0000 7f200896c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f20001af520 con 0x563445a8b580 2024-04-21T08:54:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.230+0000 7f200896c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f20001ad370 con 0x563445a94a80 2024-04-21T08:54:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.231+0000 7f1ffeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8bea0 0x563445a8ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.231+0000 7f1ffeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8bea0 0x563445a8ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34410/0 (socket says 172.21.15.28:34410) 2024-04-21T08:54:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.231+0000 7f1ffeffd640 1 -- 172.21.15.28:0/3883568122 learned_addr learned my addr 172.21.15.28:0/3883568122 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.231+0000 7f1fff7fe640 1 --2- 172.21.15.28:0/3883568122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8b580 0x563445aaa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.231+0000 7f1ffffff640 1 --2- 172.21.15.28:0/3883568122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a94a80 0x563445a8f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.231+0000 7f1ffeffd640 1 -- 172.21.15.28:0/3883568122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a94a80 msgr2=0x563445a8f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.231+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/3883568122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a94a80 0x563445a8f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.231+0000 7f1ffeffd640 1 -- 172.21.15.28:0/3883568122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8b580 msgr2=0x563445aaa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.231+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/3883568122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8b580 0x563445aaa100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.231+0000 7f1ffeffd640 1 -- 172.21.15.28:0/3883568122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563445a96e70 con 0x563445a8bea0 2024-04-21T08:54:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.231+0000 7f1fff7fe640 1 --2- 172.21.15.28:0/3883568122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8b580 0x563445aaa100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.231+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/3883568122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8bea0 0x563445a8ed20 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40ce880 tx=0x7f1ff4004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae7890c021294f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.231+0000 7f1fe57fa640 1 -- 172.21.15.28:0/3883568122 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4005680 con 0x563445a8bea0 2024-04-21T08:54:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.231+0000 7f1fe57fa640 1 -- 172.21.15.28:0/3883568122 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff409cad0 con 0x563445a8bea0 2024-04-21T08:54:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.231+0000 7f1fe57fa640 1 -- 172.21.15.28:0/3883568122 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4005240 con 0x563445a8bea0 2024-04-21T08:54:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.232+0000 7f200896c740 1 -- 172.21.15.28:0/3883568122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8bea0 msgr2=0x563445a8ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.232+0000 7f200896c740 1 --2- 172.21.15.28:0/3883568122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8bea0 0x563445a8ed20 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40ce880 tx=0x7f1ff4004a60 comp rx=0 tx=0).stop 2024-04-21T08:54:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.232+0000 7f200896c740 1 -- 172.21.15.28:0/3883568122 shutdown_connections 2024-04-21T08:54:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.232+0000 7f200896c740 1 --2- 172.21.15.28:0/3883568122 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a94a80 0x563445a8f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.232+0000 7f200896c740 1 --2- 172.21.15.28:0/3883568122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8bea0 0x563445a8ed20 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.232+0000 7f200896c740 1 --2- 172.21.15.28:0/3883568122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8b580 0x563445aaa100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.232+0000 7f200896c740 1 -- 172.21.15.28:0/3883568122 >> 172.21.15.28:0/3883568122 conn(0x563445a85cb0 msgr2=0x563445a87990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.232+0000 7f200896c740 1 -- 172.21.15.28:0/3883568122 shutdown_connections 2024-04-21T08:54:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.232+0000 7f200896c740 1 -- 172.21.15.28:0/3883568122 wait complete. 2024-04-21T08:54:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.232+0000 7f200896c740 1 -- start start 2024-04-21T08:54:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.232+0000 7f200896c740 1 Processor -- start 2024-04-21T08:54:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f200896c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8b580 0x563445b074e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f200896c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8bea0 0x563445b07a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f200896c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a94a80 0x563445b018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f200896c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20001ad9e0 con 0x563445a8b580 2024-04-21T08:54:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f200896c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f20001aee50 con 0x563445a8bea0 2024-04-21T08:54:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f200896c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f20001ae0b0 con 0x563445a94a80 2024-04-21T08:54:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f200896c740 10 client.0 _ll_register_callbacks cb 0x7f1ff0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f1fff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8b580 0x563445b074e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f1fff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8b580 0x563445b074e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34414/0 (socket says 172.21.15.28:34414) 2024-04-21T08:54:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f1fff7fe640 1 -- 172.21.15.28:0/1719990217 learned_addr learned my addr 172.21.15.28:0/1719990217 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/1719990217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8bea0 0x563445b07a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f1ffffff640 1 --2- 172.21.15.28:0/1719990217 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a94a80 0x563445b018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f1fff7fe640 1 -- 172.21.15.28:0/1719990217 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a94a80 msgr2=0x563445b018f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f1fff7fe640 1 --2- 172.21.15.28:0/1719990217 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a94a80 0x563445b018f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f1fff7fe640 1 -- 172.21.15.28:0/1719990217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8bea0 msgr2=0x563445b07a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f1fff7fe640 1 --2- 172.21.15.28:0/1719990217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8bea0 0x563445b07a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f1fff7fe640 1 -- 172.21.15.28:0/1719990217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563445b01fe0 con 0x563445a8b580 2024-04-21T08:54:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/1719990217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8bea0 0x563445b07a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f1fff7fe640 1 --2- 172.21.15.28:0/1719990217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8b580 0x563445b074e0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f1ff002c170 tx=0x7f1ff0004490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbbcd5b725f61f6e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f1fdeffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f1fdeffd640 1 -- 172.21.15.28:0/1719990217 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff002c7e0 con 0x563445a8b580 2024-04-21T08:54:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f200896c740 1 -- 172.21.15.28:0/1719990217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ff409cad0 con 0x563445a8b580 2024-04-21T08:54:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.233+0000 7f1fdeffd640 1 -- 172.21.15.28:0/1719990217 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1ff002c980 con 0x563445a8b580 2024-04-21T08:54:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.234+0000 7f1fdeffd640 1 -- 172.21.15.28:0/1719990217 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1ff0059050 con 0x563445a8b580 2024-04-21T08:54:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.234+0000 7f200896c740 10 client.16911 fetch_fsmap learned FSMap version 28 2024-04-21T08:54:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.234+0000 7f200896c740 1 -- 172.21.15.28:0/1719990217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563445b0ed50 con 0x563445a8b580 2024-04-21T08:54:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.234+0000 7f1fdeffd640 1 -- 172.21.15.28:0/1719990217 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff409cad0 con 0x563445a8b580 2024-04-21T08:54:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.234+0000 7f1fdeffd640 1 -- 172.21.15.28:0/1719990217 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f1ff004b3f0 con 0x563445a8b580 2024-04-21T08:54:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.234+0000 7f200896c740 10 client.16911 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:54:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.234+0000 7f200896c740 10 client.16911 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.234+0000 7f200896c740 20 client.16911 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.234+0000 7f200896c740 1 --2- 172.21.15.28:0/1719990217 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563445b0f4d0 0x563445b118f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.234+0000 7f200896c740 4 client.16911 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:54:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.234+0000 7f200896c740 1 -- 172.21.15.28:0/1719990217 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5634459f7a70 con 0x563445b0f4d0 2024-04-21T08:54:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.234+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/1719990217 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563445b0f4d0 0x563445b118f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f1ffeffd640 1 --2- 172.21.15.28:0/1719990217 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563445b0f4d0 0x563445b118f0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d87a2c5b18121a6 server_cookie=f59ac600b2138d5b in_seq=0 out_seq=0 2024-04-21T08:54:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f1fdeffd640 10 client.16911 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f1fdeffd640 1 -- 172.21.15.28:0/1719990217 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5634459f7a70 con 0x563445b0f4d0 2024-04-21T08:54:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f1fdeffd640 10 client.16911 handle_command_reply: tid=0 2024-04-21T08:54:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f200896c740 1 client.16911 shutdown 2024-04-21T08:54:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f200896c740 1 -- 172.21.15.28:0/1719990217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8b580 msgr2=0x563445b074e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f200896c740 1 --2- 172.21.15.28:0/1719990217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8b580 0x563445b074e0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f1ff002c170 tx=0x7f1ff0004490 comp rx=0 tx=0).stop 2024-04-21T08:54:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f200896c740 1 -- 172.21.15.28:0/1719990217 shutdown_connections 2024-04-21T08:54:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f200896c740 1 --2- 172.21.15.28:0/1719990217 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563445b0f4d0 0x563445b118f0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f200896c740 1 --2- 172.21.15.28:0/1719990217 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563445a94a80 0x563445b018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f1fdeffd640 0 client.16911 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f200896c740 1 --2- 172.21.15.28:0/1719990217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563445a8bea0 0x563445b07a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f200896c740 1 --2- 172.21.15.28:0/1719990217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563445a8b580 0x563445b074e0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f200896c740 1 -- 172.21.15.28:0/1719990217 >> 172.21.15.28:0/1719990217 conn(0x563445a85cb0 msgr2=0x563445a9fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f200896c740 1 -- 172.21.15.28:0/1719990217 shutdown_connections 2024-04-21T08:54:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f200896c740 1 -- 172.21.15.28:0/1719990217 wait complete. 2024-04-21T08:54:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.235+0000 7f200896c740 20 client.16911 trim_cache size 0 max 16384 2024-04-21T08:54:50.244 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:54:50.244 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:54:50.244 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:54:50.244 INFO:teuthology.orchestra.run.smithi028.stdout: "8b485e2a-5339-4b65-a825-69e1706aba4a": { 2024-04-21T08:54:50.244 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:54:50.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "8b485e2a-5339-4b65-a825-69e1706aba4a", 2024-04-21T08:54:50.244 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:54:50.244 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:50.244 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:50.245 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:54:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.236+0000 7f2005d7e640 1 -- 172.21.15.28:0/4078479653 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1fe80a3d20 msgr2=0x7f1fe80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.236+0000 7f2005d7e640 1 --2- 172.21.15.28:0/4078479653 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1fe80a3d20 0x7f1fe80a61e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8009fd0 tx=0x7f1ff80096a0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.236+0000 7f2005d7e640 1 -- 172.21.15.28:0/4078479653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2000106150 msgr2=0x7f2000198ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.236+0000 7f2005d7e640 1 --2- 172.21.15.28:0/4078479653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2000106150 0x7f2000198ce0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f1ff400e9f0 tx=0x7f1ff400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.236+0000 7f2005d7e640 1 -- 172.21.15.28:0/4078479653 shutdown_connections 2024-04-21T08:54:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.236+0000 7f2005d7e640 1 --2- 172.21.15.28:0/4078479653 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1fe80a3d20 0x7f1fe80a61e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.236+0000 7f2005d7e640 1 --2- 172.21.15.28:0/4078479653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f200019bfb0 0x7f200019c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.236+0000 7f2005d7e640 1 --2- 172.21.15.28:0/4078479653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2000106150 0x7f2000198ce0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.236+0000 7f2005d7e640 1 --2- 172.21.15.28:0/4078479653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2000101080 0x7f20001987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.236+0000 7f2005d7e640 1 -- 172.21.15.28:0/4078479653 >> 172.21.15.28:0/4078479653 conn(0x7f2000078000 msgr2=0x7f2000069480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.236+0000 7f2005d7e640 1 -- 172.21.15.28:0/4078479653 shutdown_connections 2024-04-21T08:54:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:50.236+0000 7f2005d7e640 1 -- 172.21.15.28:0/4078479653 wait complete. 2024-04-21T08:54:50.247 INFO:tasks.cephfs.filesystem:scrub status for tag:8b485e2a-5339-4b65-a825-69e1706aba4a - {'path': '/', 'tag': '8b485e2a-5339-4b65-a825-69e1706aba4a', 'options': 'recursive,force'} 2024-04-21T08:54:50.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:50 smithi152 ceph-mon[31401]: pgmap v112: 129 pgs: 129 active+clean; 14 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 46 MiB/s wr, 7.90k op/s 2024-04-21T08:54:50.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:50 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:50.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:50 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:50 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:50.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:50 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:50 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:50.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:50 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:50 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:50.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:50 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:50 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:50.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:50 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:50 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:50.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:50 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:50 smithi028 ceph-mon[28003]: pgmap v112: 129 pgs: 129 active+clean; 14 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 46 MiB/s wr, 7.90k op/s 2024-04-21T08:54:50.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:50 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:50.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:50 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:50 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:50.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:50 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:50 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:50.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:50 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:50 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:50.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:50 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:50 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:50.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:50 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:50 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:50.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:50 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:50 smithi171 ceph-mon[31358]: pgmap v112: 129 pgs: 129 active+clean; 14 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 46 MiB/s wr, 7.90k op/s 2024-04-21T08:54:50.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:50 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:50.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:50 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:50 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:50.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:50 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:50 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:50.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:50 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:50 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:50.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:50 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:50 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:50.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:50 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:50.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:50 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:50.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:50 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:51 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:54:51.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:51 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:51 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:51.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:51 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:51.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:51 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:51 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:51.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:51 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:51 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:51.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:51 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:51 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:54:51.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:51 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:51 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:51 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:54:51.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:51 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:51 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:51.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:51 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:51.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:51 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:51 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:51.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:51 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:51 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:51.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:51 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:51 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:54:51.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:51 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:51 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:51 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:54:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:51 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:51 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:51 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:51 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:51 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:51 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:51 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:51.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:51 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:51 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:54:51.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:51 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:51.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:51 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:52.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:54:52.557 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:52 smithi028 ceph-mon[28003]: pgmap v113: 129 pgs: 129 active+clean; 14 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 41 MiB/s wr, 6.96k op/s 2024-04-21T08:54:52.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:52 smithi152 ceph-mon[31401]: pgmap v113: 129 pgs: 129 active+clean; 14 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 41 MiB/s wr, 6.96k op/s 2024-04-21T08:54:52.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:52 smithi171 ceph-mon[31358]: pgmap v113: 129 pgs: 129 active+clean; 14 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 41 MiB/s wr, 6.96k op/s 2024-04-21T08:54:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.698+0000 7fc430c5c640 1 -- 172.21.15.28:0/3144104141 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc42c106420 msgr2=0x7fc42c1068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.698+0000 7fc430c5c640 1 --2- 172.21.15.28:0/3144104141 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc42c106420 0x7fc42c1068d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc41c009860 tx=0x7fc41c02bc50 comp rx=0 tx=0).stop 2024-04-21T08:54:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.698+0000 7fc430c5c640 1 -- 172.21.15.28:0/3144104141 shutdown_connections 2024-04-21T08:54:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.698+0000 7fc430c5c640 1 --2- 172.21.15.28:0/3144104141 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc42c106420 0x7fc42c1068d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.698+0000 7fc430c5c640 1 --2- 172.21.15.28:0/3144104141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc42c101940 0x7fc42c105e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.698+0000 7fc430c5c640 1 --2- 172.21.15.28:0/3144104141 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc42c101020 0x7fc42c101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.698+0000 7fc430c5c640 1 -- 172.21.15.28:0/3144104141 >> 172.21.15.28:0/3144104141 conn(0x7fc42c077fb0 msgr2=0x7fc42c0783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:52.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.702+0000 7fc430c5c640 1 -- 172.21.15.28:0/3144104141 shutdown_connections 2024-04-21T08:54:52.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.702+0000 7fc430c5c640 1 -- 172.21.15.28:0/3144104141 wait complete. 2024-04-21T08:54:52.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.705+0000 7fc430c5c640 1 Processor -- start 2024-04-21T08:54:52.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.707+0000 7fc430c5c640 1 -- start start 2024-04-21T08:54:52.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.708+0000 7fc430c5c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc42c101020 0x7fc42c19eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.708+0000 7fc430c5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc42c101940 0x7fc42c19f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.708+0000 7fc430c5c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc42c106420 0x7fc42c1a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.708+0000 7fc430c5c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc42c1188b0 con 0x7fc42c101940 2024-04-21T08:54:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.708+0000 7fc430c5c640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc42c118730 con 0x7fc42c106420 2024-04-21T08:54:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.709+0000 7fc430c5c640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc42c118a30 con 0x7fc42c101020 2024-04-21T08:54:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.709+0000 7fc42a59c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc42c101020 0x7fc42c19eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.709+0000 7fc42ad9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc42c106420 0x7fc42c1a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.710+0000 7fc42ad9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc42c106420 0x7fc42c1a3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:48784/0 (socket says 172.21.15.28:48784) 2024-04-21T08:54:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.710+0000 7fc42a59c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc42c101020 0x7fc42c19eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:60956/0 (socket says 172.21.15.28:60956) 2024-04-21T08:54:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.710+0000 7fc42ad9d640 1 -- 172.21.15.28:0/875699412 learned_addr learned my addr 172.21.15.28:0/875699412 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.710+0000 7fc429d9b640 1 --2- 172.21.15.28:0/875699412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc42c101940 0x7fc42c19f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.711+0000 7fc42a59c640 1 -- 172.21.15.28:0/875699412 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc42c106420 msgr2=0x7fc42c1a3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.711+0000 7fc42a59c640 1 --2- 172.21.15.28:0/875699412 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc42c106420 0x7fc42c1a3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.711+0000 7fc42a59c640 1 -- 172.21.15.28:0/875699412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc42c101940 msgr2=0x7fc42c19f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:52.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.711+0000 7fc42a59c640 1 --2- 172.21.15.28:0/875699412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc42c101940 0x7fc42c19f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:52.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.711+0000 7fc42a59c640 1 -- 172.21.15.28:0/875699412 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc42c1a3ea0 con 0x7fc42c101020 2024-04-21T08:54:52.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.711+0000 7fc42ad9d640 1 --2- 172.21.15.28:0/875699412 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc42c106420 0x7fc42c1a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:52.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.711+0000 7fc429d9b640 1 --2- 172.21.15.28:0/875699412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc42c101940 0x7fc42c19f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:52.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.712+0000 7fc42a59c640 1 --2- 172.21.15.28:0/875699412 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc42c101020 0x7fc42c19eeb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc42000e790 tx=0x7fc42000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:52.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.714+0000 7fc41b7fe640 1 -- 172.21.15.28:0/875699412 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42000eea0 con 0x7fc42c101020 2024-04-21T08:54:52.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.714+0000 7fc41b7fe640 1 -- 172.21.15.28:0/875699412 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc42000ccc0 con 0x7fc42c101020 2024-04-21T08:54:52.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.715+0000 7fc41b7fe640 1 -- 172.21.15.28:0/875699412 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc420018680 con 0x7fc42c101020 2024-04-21T08:54:52.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.717+0000 7fc430c5c640 1 -- 172.21.15.28:0/875699412 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc42c1a4190 con 0x7fc42c101020 2024-04-21T08:54:52.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.717+0000 7fc430c5c640 1 -- 172.21.15.28:0/875699412 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc42c1a45a0 con 0x7fc42c101020 2024-04-21T08:54:52.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.718+0000 7fc430c5c640 1 -- 172.21.15.28:0/875699412 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc42c06ab10 con 0x7fc42c101020 2024-04-21T08:54:52.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.724+0000 7fc41b7fe640 1 -- 172.21.15.28:0/875699412 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc42001f020 con 0x7fc42c101020 2024-04-21T08:54:52.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.725+0000 7fc41b7fe640 1 --2- 172.21.15.28:0/875699412 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4140a3e00 0x7fc4140a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:52.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.725+0000 7fc41b7fe640 1 -- 172.21.15.28:0/875699412 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fc4200d0c40 con 0x7fc42c101020 2024-04-21T08:54:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.728+0000 7fc429d9b640 1 --2- 172.21.15.28:0/875699412 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4140a3e00 0x7fc4140a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:52.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.730+0000 7fc41b7fe640 1 -- 172.21.15.28:0/875699412 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc42c1a45a0 con 0x7fc42c101020 2024-04-21T08:54:52.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:52.731+0000 7fc429d9b640 1 --2- 172.21.15.28:0/875699412 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4140a3e00 0x7fc4140a62c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc42c078900 tx=0x7fc424009550 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:53.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.186+0000 7fc430c5c640 1 -- 172.21.15.28:0/875699412 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc42c1a0200 con 0x7fc42c101020 2024-04-21T08:54:53.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.187+0000 7fc41b7fe640 1 -- 172.21.15.28:0/875699412 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7fc42009cfe0 con 0x7fc42c101020 2024-04-21T08:54:53.187 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:54:53.187 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:54:53.187 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T08:54:53.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.191+0000 7fc430c5c640 1 -- 172.21.15.28:0/875699412 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4140a3e00 msgr2=0x7fc4140a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:53.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.191+0000 7fc430c5c640 1 --2- 172.21.15.28:0/875699412 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4140a3e00 0x7fc4140a62c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc42c078900 tx=0x7fc424009550 comp rx=0 tx=0).stop 2024-04-21T08:54:53.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.191+0000 7fc430c5c640 1 -- 172.21.15.28:0/875699412 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc42c101020 msgr2=0x7fc42c19eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:53.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.191+0000 7fc430c5c640 1 --2- 172.21.15.28:0/875699412 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc42c101020 0x7fc42c19eeb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc42000e790 tx=0x7fc42000ec50 comp rx=0 tx=0).stop 2024-04-21T08:54:53.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.191+0000 7fc430c5c640 1 -- 172.21.15.28:0/875699412 shutdown_connections 2024-04-21T08:54:53.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.191+0000 7fc430c5c640 1 --2- 172.21.15.28:0/875699412 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4140a3e00 0x7fc4140a62c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.191+0000 7fc430c5c640 1 --2- 172.21.15.28:0/875699412 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc42c106420 0x7fc42c1a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.191+0000 7fc430c5c640 1 --2- 172.21.15.28:0/875699412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc42c101940 0x7fc42c19f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.191+0000 7fc430c5c640 1 --2- 172.21.15.28:0/875699412 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc42c101020 0x7fc42c19eeb0 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc42000e790 tx=0x7fc42000ec50 comp rx=0 tx=0).stop 2024-04-21T08:54:53.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.191+0000 7fc430c5c640 1 -- 172.21.15.28:0/875699412 >> 172.21.15.28:0/875699412 conn(0x7fc42c077fb0 msgr2=0x7fc42c103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:53.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.192+0000 7fc430c5c640 1 -- 172.21.15.28:0/875699412 shutdown_connections 2024-04-21T08:54:53.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.192+0000 7fc430c5c640 1 -- 172.21.15.28:0/875699412 wait complete. 2024-04-21T08:54:53.211 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24488 quiesce db --set-id 7459dea4 --release --await' 2024-04-21T08:54:53.211 DEBUG:teuthology.orchestra.run.smithi028:> timeout 45 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24488 quiesce db --set-id 7459dea4 --release --await 2024-04-21T08:54:53.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.375+0000 7f431b225640 1 -- 172.21.15.28:0/2068377002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4314074300 msgr2=0x7f43140746e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:53.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.375+0000 7f431b225640 1 --2- 172.21.15.28:0/2068377002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4314074300 0x7f43140746e0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4310009900 tx=0x7f431002bcd0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.375+0000 7f431b225640 1 -- 172.21.15.28:0/2068377002 shutdown_connections 2024-04-21T08:54:53.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.375+0000 7f431b225640 1 --2- 172.21.15.28:0/2068377002 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f431410e7e0 0x7f4314110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.375+0000 7f431b225640 1 --2- 172.21.15.28:0/2068377002 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4314074c20 0x7f431410e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.375+0000 7f431b225640 1 --2- 172.21.15.28:0/2068377002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4314074300 0x7f43140746e0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.375+0000 7f431b225640 1 -- 172.21.15.28:0/2068377002 >> 172.21.15.28:0/2068377002 conn(0x7f43140fd990 msgr2=0x7f43140ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:53.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.375+0000 7f431b225640 1 -- 172.21.15.28:0/2068377002 shutdown_connections 2024-04-21T08:54:53.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.375+0000 7f431b225640 1 -- 172.21.15.28:0/2068377002 wait complete. 2024-04-21T08:54:53.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f431b225640 1 Processor -- start 2024-04-21T08:54:53.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f431b225640 1 -- start start 2024-04-21T08:54:53.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f431b225640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4314074300 0x7f43141a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f431b225640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4314074c20 0x7f43141a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f431b225640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f431410e7e0 0x7f431419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f431b225640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4314114390 con 0x7f431410e7e0 2024-04-21T08:54:53.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f431b225640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4314114210 con 0x7f4314074c20 2024-04-21T08:54:53.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f431b225640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4314114510 con 0x7f4314074300 2024-04-21T08:54:53.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f43197c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f431410e7e0 0x7f431419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f4318fc1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4314074300 0x7f43141a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f43197c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f431410e7e0 0x7f431419a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34434/0 (socket says 172.21.15.28:34434) 2024-04-21T08:54:53.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f43197c2640 1 -- 172.21.15.28:0/3137374116 learned_addr learned my addr 172.21.15.28:0/3137374116 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:53.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f430bfff640 1 --2- 172.21.15.28:0/3137374116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4314074c20 0x7f43141a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f43197c2640 1 -- 172.21.15.28:0/3137374116 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4314074300 msgr2=0x7f43141a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:53.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f43197c2640 1 --2- 172.21.15.28:0/3137374116 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4314074300 0x7f43141a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f43197c2640 1 -- 172.21.15.28:0/3137374116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4314074c20 msgr2=0x7f43141a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:53.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f43197c2640 1 --2- 172.21.15.28:0/3137374116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4314074c20 0x7f43141a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.376+0000 7f43197c2640 1 -- 172.21.15.28:0/3137374116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f431419aff0 con 0x7f431410e7e0 2024-04-21T08:54:53.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.377+0000 7f43197c2640 1 --2- 172.21.15.28:0/3137374116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f431410e7e0 0x7f431419a730 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f430c00cce0 tx=0x7f430c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:53.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.377+0000 7f4309ffb640 1 -- 172.21.15.28:0/3137374116 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c00f450 con 0x7f431410e7e0 2024-04-21T08:54:53.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.377+0000 7f431b225640 1 -- 172.21.15.28:0/3137374116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f431419b2e0 con 0x7f431410e7e0 2024-04-21T08:54:53.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.377+0000 7f4309ffb640 1 -- 172.21.15.28:0/3137374116 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f430c00faa0 con 0x7f431410e7e0 2024-04-21T08:54:53.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.377+0000 7f4309ffb640 1 -- 172.21.15.28:0/3137374116 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c018b70 con 0x7f431410e7e0 2024-04-21T08:54:53.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.377+0000 7f431b225640 1 -- 172.21.15.28:0/3137374116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43141a75f0 con 0x7f431410e7e0 2024-04-21T08:54:53.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.379+0000 7f4309ffb640 1 -- 172.21.15.28:0/3137374116 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f430c018d10 con 0x7f431410e7e0 2024-04-21T08:54:53.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.379+0000 7f431de13740 1 Processor -- start 2024-04-21T08:54:53.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.379+0000 7f431de13740 1 -- start start 2024-04-21T08:54:53.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f4309ffb640 1 --2- 172.21.15.28:0/3137374116 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f42fc0a3d20 0x7f42fc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f431de13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91310 0x562bd4a916f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f431de13740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91c30 0x562bd4a9a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f431de13740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a9a810 0x562bd4a9cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f431de13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562bd49f6500 con 0x562bd4a91310 2024-04-21T08:54:53.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f431de13740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562bd499c6a0 con 0x562bd4a91c30 2024-04-21T08:54:53.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f431de13740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x562bd49f24c0 con 0x562bd4a9a810 2024-04-21T08:54:53.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f4318fc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91310 0x562bd4a916f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f430bfff640 1 --2- 172.21.15.28:0/3137374116 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f42fc0a3d20 0x7f42fc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f4309ffb640 1 -- 172.21.15.28:0/3137374116 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f430c0d0f60 con 0x7f431410e7e0 2024-04-21T08:54:53.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f4318fc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91310 0x562bd4a916f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34448/0 (socket says 172.21.15.28:34448) 2024-04-21T08:54:53.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f4318fc1640 1 -- 172.21.15.28:0/2971980970 learned_addr learned my addr 172.21.15.28:0/2971980970 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:53.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f43197c2640 1 --2- 172.21.15.28:0/2971980970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a9a810 0x562bd4a9cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f430bfff640 1 --2- 172.21.15.28:0/2971980970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91c30 0x562bd4a9a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f43197c2640 1 -- 172.21.15.28:0/2971980970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91c30 msgr2=0x562bd4a9a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:53.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f43197c2640 1 --2- 172.21.15.28:0/2971980970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91c30 0x562bd4a9a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f43197c2640 1 -- 172.21.15.28:0/2971980970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91310 msgr2=0x562bd4a916f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:53.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f43197c2640 1 --2- 172.21.15.28:0/2971980970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91310 0x562bd4a916f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f43197c2640 1 -- 172.21.15.28:0/2971980970 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bd49bb5e0 con 0x562bd4a9a810 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f4318fc1640 1 --2- 172.21.15.28:0/2971980970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91310 0x562bd4a916f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f430bfff640 1 --2- 172.21.15.28:0/3137374116 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f42fc0a3d20 0x7f42fc0a61e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f430400b620 tx=0x7f430400bb10 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.380+0000 7f43197c2640 1 --2- 172.21.15.28:0/2971980970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a9a810 0x562bd4a9cc00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f430c0ce2b0 tx=0x7f430c00af30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f011161023617ae server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.381+0000 7f42faffd640 1 -- 172.21.15.28:0/2971980970 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c019000 con 0x562bd4a9a810 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.381+0000 7f42faffd640 1 -- 172.21.15.28:0/2971980970 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f430c09d200 con 0x562bd4a9a810 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.381+0000 7f42faffd640 1 -- 172.21.15.28:0/2971980970 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c00fc50 con 0x562bd4a9a810 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.381+0000 7f431de13740 1 -- 172.21.15.28:0/2971980970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a9a810 msgr2=0x562bd4a9cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.381+0000 7f431de13740 1 --2- 172.21.15.28:0/2971980970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a9a810 0x562bd4a9cc00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f430c0ce2b0 tx=0x7f430c00af30 comp rx=0 tx=0).stop 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.381+0000 7f431de13740 1 -- 172.21.15.28:0/2971980970 shutdown_connections 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.381+0000 7f431de13740 1 --2- 172.21.15.28:0/2971980970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a9a810 0x562bd4a9cc00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.381+0000 7f431de13740 1 --2- 172.21.15.28:0/2971980970 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91c30 0x562bd4a9a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.381+0000 7f431de13740 1 --2- 172.21.15.28:0/2971980970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91310 0x562bd4a916f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.381+0000 7f431de13740 1 -- 172.21.15.28:0/2971980970 >> 172.21.15.28:0/2971980970 conn(0x562bd4a8ba40 msgr2=0x562bd4a8be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.381+0000 7f431de13740 1 -- 172.21.15.28:0/2971980970 shutdown_connections 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.381+0000 7f431de13740 1 -- 172.21.15.28:0/2971980970 wait complete. 2024-04-21T08:54:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f431de13740 1 -- start start 2024-04-21T08:54:53.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f431de13740 1 Processor -- start 2024-04-21T08:54:53.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f431de13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91310 0x562bd4a9a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f431de13740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a91c30 0x562bd4a97650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f431de13740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a9a810 0x562bd4a97d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f431de13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562bd4a9d600 con 0x562bd4a91310 2024-04-21T08:54:53.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f431de13740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562bd49afa60 con 0x562bd4a9a810 2024-04-21T08:54:53.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f431de13740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x562bd4a9d780 con 0x562bd4a91c30 2024-04-21T08:54:53.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f431de13740 10 client.0 _ll_register_callbacks cb 0x562bd4708110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:53.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f4318fc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91310 0x562bd4a9a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f4318fc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91310 0x562bd4a9a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34460/0 (socket says 172.21.15.28:34460) 2024-04-21T08:54:53.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f4318fc1640 1 -- 172.21.15.28:0/2421376748 learned_addr learned my addr 172.21.15.28:0/2421376748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:53.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f430bfff640 1 --2- 172.21.15.28:0/2421376748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a91c30 0x562bd4a97650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f43197c2640 1 --2- 172.21.15.28:0/2421376748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a9a810 0x562bd4a97d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f4318fc1640 1 -- 172.21.15.28:0/2421376748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a91c30 msgr2=0x562bd4a97650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:53.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f4318fc1640 1 --2- 172.21.15.28:0/2421376748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a91c30 0x562bd4a97650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f4318fc1640 1 -- 172.21.15.28:0/2421376748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a9a810 msgr2=0x562bd4a97d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:53.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f4318fc1640 1 --2- 172.21.15.28:0/2421376748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a9a810 0x562bd4a97d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f4318fc1640 1 -- 172.21.15.28:0/2421376748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bd4708920 con 0x562bd4a91310 2024-04-21T08:54:53.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.382+0000 7f43197c2640 1 --2- 172.21.15.28:0/2421376748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a9a810 0x562bd4a97d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:53.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f4318fc1640 1 --2- 172.21.15.28:0/2421376748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91310 0x562bd4a9a2c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4310009230 tx=0x7f4310005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f6c63222c47aa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:53.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f42f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:53.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f42f3fff640 1 -- 172.21.15.28:0/2421376748 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4310043070 con 0x562bd4a91310 2024-04-21T08:54:53.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f42f3fff640 1 -- 172.21.15.28:0/2421376748 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f431002c880 con 0x562bd4a91310 2024-04-21T08:54:53.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f431de13740 1 -- 172.21.15.28:0/2421376748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f430c09d200 con 0x562bd4a91310 2024-04-21T08:54:53.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f42f3fff640 1 -- 172.21.15.28:0/2421376748 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f431004f050 con 0x562bd4a91310 2024-04-21T08:54:53.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f431de13740 10 client.16926 fetch_fsmap learned FSMap version 28 2024-04-21T08:54:53.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f431de13740 1 -- 172.21.15.28:0/2421376748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562bd47d4d30 con 0x562bd4a91310 2024-04-21T08:54:53.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f42f3fff640 1 -- 172.21.15.28:0/2421376748 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f430c09d200 con 0x562bd4a91310 2024-04-21T08:54:53.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f42f3fff640 1 -- 172.21.15.28:0/2421376748 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f43100035f0 con 0x562bd4a91310 2024-04-21T08:54:53.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f431de13740 10 client.16926 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:54:53.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f431de13740 10 client.16926 resolve_mds: validated gid 24488 aka daemon mds.b 2024-04-21T08:54:53.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f431de13740 20 client.16926 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:53.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f431de13740 1 --2- 172.21.15.28:0/2421376748 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562bd4aaa940 0x562bd4aacdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f431de13740 4 client.16926 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:53.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.383+0000 7f431de13740 1 -- 172.21.15.28:0/2421376748 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bd49c9a10 con 0x562bd4aaa940 2024-04-21T08:54:53.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.384+0000 7f430bfff640 1 --2- 172.21.15.28:0/2421376748 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562bd4aaa940 0x562bd4aacdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.384+0000 7f430bfff640 1 --2- 172.21.15.28:0/2421376748 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562bd4aaa940 0x562bd4aacdc0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95c9d2f9935097c server_cookie=65783cf55bf4ae25 in_seq=0 out_seq=0 2024-04-21T08:54:53.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.384+0000 7f42f3fff640 10 client.16926 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:53.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.385+0000 7f42f3fff640 1 -- 172.21.15.28:0/2421376748 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562bd49c9a10 con 0x562bd4aaa940 2024-04-21T08:54:53.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.385+0000 7f42f3fff640 10 client.16926 handle_command_reply: tid=0 2024-04-21T08:54:53.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.385+0000 7f431de13740 1 client.16926 shutdown 2024-04-21T08:54:53.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.385+0000 7f431de13740 1 -- 172.21.15.28:0/2421376748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91310 msgr2=0x562bd4a9a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:53.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.385+0000 7f431de13740 1 --2- 172.21.15.28:0/2421376748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91310 0x562bd4a9a2c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4310009230 tx=0x7f4310005270 comp rx=0 tx=0).stop 2024-04-21T08:54:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.385+0000 7f431de13740 1 -- 172.21.15.28:0/2421376748 shutdown_connections 2024-04-21T08:54:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.385+0000 7f431de13740 1 --2- 172.21.15.28:0/2421376748 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562bd4aaa940 0x562bd4aacdc0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.385+0000 7f431de13740 1 --2- 172.21.15.28:0/2421376748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a9a810 0x562bd4a97d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.385+0000 7f42f3fff640 0 client.16926 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.385+0000 7f431de13740 1 --2- 172.21.15.28:0/2421376748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a91c30 0x562bd4a97650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.385+0000 7f431de13740 1 --2- 172.21.15.28:0/2421376748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91310 0x562bd4a9a2c0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.385+0000 7f431de13740 1 -- 172.21.15.28:0/2421376748 >> 172.21.15.28:0/2421376748 conn(0x562bd4a8ba40 msgr2=0x562bd4a8d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.385+0000 7f431de13740 1 -- 172.21.15.28:0/2421376748 shutdown_connections 2024-04-21T08:54:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.385+0000 7f431de13740 1 -- 172.21.15.28:0/2421376748 wait complete. 2024-04-21T08:54:53.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.385+0000 7f431de13740 20 client.16926 trim_cache size 0 max 16384 2024-04-21T08:54:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f431de13740 1 Processor -- start 2024-04-21T08:54:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f431de13740 1 -- start start 2024-04-21T08:54:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f431de13740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91310 0x562bd4a8d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f431de13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91c30 0x562bd4a99d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f431de13740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a9a810 0x562bd4a9a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f431de13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43141b3cf0 con 0x562bd4a91c30 2024-04-21T08:54:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f431de13740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f43141b15c0 con 0x562bd4a91310 2024-04-21T08:54:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f431de13740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f43141af410 con 0x562bd4a9a810 2024-04-21T08:54:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f430bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91c30 0x562bd4a99d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f430bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91c30 0x562bd4a99d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34468/0 (socket says 172.21.15.28:34468) 2024-04-21T08:54:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f430bfff640 1 -- 172.21.15.28:0/2828002176 learned_addr learned my addr 172.21.15.28:0/2828002176 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f4318fc1640 1 --2- 172.21.15.28:0/2828002176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91310 0x562bd4a8d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f43197c2640 1 --2- 172.21.15.28:0/2828002176 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a9a810 0x562bd4a9a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f430bfff640 1 -- 172.21.15.28:0/2828002176 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a9a810 msgr2=0x562bd4a9a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f430bfff640 1 --2- 172.21.15.28:0/2828002176 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a9a810 0x562bd4a9a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f430bfff640 1 -- 172.21.15.28:0/2828002176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91310 msgr2=0x562bd4a8d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f430bfff640 1 --2- 172.21.15.28:0/2828002176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91310 0x562bd4a8d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f430bfff640 1 -- 172.21.15.28:0/2828002176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bd4a95ba0 con 0x562bd4a91c30 2024-04-21T08:54:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f4318fc1640 1 --2- 172.21.15.28:0/2828002176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91310 0x562bd4a8d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f43197c2640 1 --2- 172.21.15.28:0/2828002176 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a9a810 0x562bd4a9a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.403+0000 7f430bfff640 1 --2- 172.21.15.28:0/2828002176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91c30 0x562bd4a99d80 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f43040058a0 tx=0x7f430401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f62703771e23efb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.404+0000 7f42fa7fc640 1 -- 172.21.15.28:0/2828002176 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4304010b00 con 0x562bd4a91c30 2024-04-21T08:54:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.404+0000 7f42fa7fc640 1 -- 172.21.15.28:0/2828002176 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4304031070 con 0x562bd4a91c30 2024-04-21T08:54:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.404+0000 7f42fa7fc640 1 -- 172.21.15.28:0/2828002176 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4304021830 con 0x562bd4a91c30 2024-04-21T08:54:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.404+0000 7f431de13740 1 -- 172.21.15.28:0/2828002176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91c30 msgr2=0x562bd4a99d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.404+0000 7f431de13740 1 --2- 172.21.15.28:0/2828002176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91c30 0x562bd4a99d80 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f43040058a0 tx=0x7f430401f000 comp rx=0 tx=0).stop 2024-04-21T08:54:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.404+0000 7f431de13740 1 -- 172.21.15.28:0/2828002176 shutdown_connections 2024-04-21T08:54:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.404+0000 7f431de13740 1 --2- 172.21.15.28:0/2828002176 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a9a810 0x562bd4a9a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.404+0000 7f431de13740 1 --2- 172.21.15.28:0/2828002176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a91c30 0x562bd4a99d80 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.404+0000 7f431de13740 1 --2- 172.21.15.28:0/2828002176 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91310 0x562bd4a8d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.404+0000 7f431de13740 1 -- 172.21.15.28:0/2828002176 >> 172.21.15.28:0/2828002176 conn(0x562bd4a8ba40 msgr2=0x562bd4a8ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.404+0000 7f431de13740 1 -- 172.21.15.28:0/2828002176 shutdown_connections 2024-04-21T08:54:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.404+0000 7f431de13740 1 -- 172.21.15.28:0/2828002176 wait complete. 2024-04-21T08:54:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f431de13740 1 -- start start 2024-04-21T08:54:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f431de13740 1 Processor -- start 2024-04-21T08:54:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f431de13740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a91310 0x562bd4aa2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f431de13740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91c30 0x562bd4a9d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f431de13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a9a810 0x562bd4a9ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f431de13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43141afa80 con 0x562bd4a9a810 2024-04-21T08:54:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f431de13740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f43141b0ef0 con 0x562bd4a91c30 2024-04-21T08:54:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f431de13740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f43141b0150 con 0x562bd4a91310 2024-04-21T08:54:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f431de13740 10 client.0 _ll_register_callbacks cb 0x7f431002c6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f43197c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a9a810 0x562bd4a9ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f43197c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a9a810 0x562bd4a9ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34484/0 (socket says 172.21.15.28:34484) 2024-04-21T08:54:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f43197c2640 1 -- 172.21.15.28:0/2357763308 learned_addr learned my addr 172.21.15.28:0/2357763308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:54:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f430bfff640 1 --2- 172.21.15.28:0/2357763308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91c30 0x562bd4a9d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f4318fc1640 1 --2- 172.21.15.28:0/2357763308 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a91310 0x562bd4aa2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f43197c2640 1 -- 172.21.15.28:0/2357763308 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a91310 msgr2=0x562bd4aa2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f43197c2640 1 --2- 172.21.15.28:0/2357763308 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a91310 0x562bd4aa2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.405+0000 7f43197c2640 1 -- 172.21.15.28:0/2357763308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91c30 msgr2=0x562bd4a9d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f43197c2640 1 --2- 172.21.15.28:0/2357763308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91c30 0x562bd4a9d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f43197c2640 1 -- 172.21.15.28:0/2357763308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bd4a9e540 con 0x562bd4a9a810 2024-04-21T08:54:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f4318fc1640 1 --2- 172.21.15.28:0/2357763308 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a91310 0x562bd4aa2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f43197c2640 1 --2- 172.21.15.28:0/2357763308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a9a810 0x562bd4a9ded0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f430c011010 tx=0x7f430c005c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad0eb2048b813a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f42f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:54:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f42f3fff640 1 -- 172.21.15.28:0/2357763308 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c0b6070 con 0x562bd4a9a810 2024-04-21T08:54:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f42f3fff640 1 -- 172.21.15.28:0/2357763308 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f430c0bb900 con 0x562bd4a9a810 2024-04-21T08:54:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f431de13740 1 -- 172.21.15.28:0/2357763308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4304031070 con 0x562bd4a9a810 2024-04-21T08:54:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f42f3fff640 1 -- 172.21.15.28:0/2357763308 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f430c09cb00 con 0x562bd4a9a810 2024-04-21T08:54:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f431de13740 10 client.16932 fetch_fsmap learned FSMap version 28 2024-04-21T08:54:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f431de13740 1 -- 172.21.15.28:0/2357763308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562bd4b11d40 con 0x562bd4a9a810 2024-04-21T08:54:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f42f3fff640 1 -- 172.21.15.28:0/2357763308 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4304031070 con 0x562bd4a9a810 2024-04-21T08:54:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f42f3fff640 1 -- 172.21.15.28:0/2357763308 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f430c0ce530 con 0x562bd4a9a810 2024-04-21T08:54:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f431de13740 10 client.16932 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:54:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f431de13740 10 client.16932 resolve_mds: validated gid 24488 aka daemon mds.b 2024-04-21T08:54:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f431de13740 20 client.16932 populate_metadata read hostname 'smithi028' 2024-04-21T08:54:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f431de13740 1 --2- 172.21.15.28:0/2357763308 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562bd4b11fd0 0x562bd4b143c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f431de13740 4 client.16932 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "7459dea4", "release": true, "await": true}] 2024-04-21T08:54:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.406+0000 7f431de13740 1 -- 172.21.15.28:0/2357763308 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "quiesce db", "set_id": "7459dea4", "release": true, "await": true}) -- 0x562bd4a03f20 con 0x562bd4b11fd0 2024-04-21T08:54:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.407+0000 7f4318fc1640 1 --2- 172.21.15.28:0/2357763308 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562bd4b11fd0 0x562bd4b143c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.407+0000 7f4318fc1640 1 --2- 172.21.15.28:0/2357763308 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562bd4b11fd0 0x562bd4b143c0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e558a7b8a97dc9d server_cookie=3d7a4070d4513606 in_seq=0 out_seq=0 2024-04-21T08:54:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.407+0000 7f42f3fff640 10 client.16932 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:53.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.422+0000 7f42f3fff640 1 -- 172.21.15.28:0/2357763308 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x562bd4a03f20 con 0x562bd4b11fd0 2024-04-21T08:54:53.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.422+0000 7f42f3fff640 10 client.16932 handle_command_reply: tid=0 2024-04-21T08:54:53.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.422+0000 7f431de13740 1 client.16932 shutdown 2024-04-21T08:54:53.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.422+0000 7f431de13740 1 -- 172.21.15.28:0/2357763308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a9a810 msgr2=0x562bd4a9ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:53.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.422+0000 7f431de13740 1 --2- 172.21.15.28:0/2357763308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a9a810 0x562bd4a9ded0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f430c011010 tx=0x7f430c005c10 comp rx=0 tx=0).stop 2024-04-21T08:54:53.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.422+0000 7f431de13740 1 -- 172.21.15.28:0/2357763308 shutdown_connections 2024-04-21T08:54:53.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.422+0000 7f431de13740 1 --2- 172.21.15.28:0/2357763308 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562bd4b11fd0 0x562bd4b143c0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.422+0000 7f431de13740 1 --2- 172.21.15.28:0/2357763308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bd4a9a810 0x562bd4a9ded0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.422+0000 7f42f3fff640 0 client.16932 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:54:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.422+0000 7f431de13740 1 --2- 172.21.15.28:0/2357763308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562bd4a91c30 0x562bd4a9d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.422+0000 7f431de13740 1 --2- 172.21.15.28:0/2357763308 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562bd4a91310 0x562bd4aa2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.422+0000 7f431de13740 1 -- 172.21.15.28:0/2357763308 >> 172.21.15.28:0/2357763308 conn(0x562bd4a8ba40 msgr2=0x562bd4a8ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:53.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.432+0000 7f431de13740 1 -- 172.21.15.28:0/2357763308 shutdown_connections 2024-04-21T08:54:53.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.436+0000 7f431de13740 1 -- 172.21.15.28:0/2357763308 wait complete. 2024-04-21T08:54:53.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.438+0000 7f431de13740 20 client.16932 trim_cache size 0 max 16384 2024-04-21T08:54:53.492 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:54:53.492 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 28, 2024-04-21T08:54:53.492 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24488, 2024-04-21T08:54:53.493 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 8, 2024-04-21T08:54:53.493 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": { 2024-04-21T08:54:53.493 INFO:teuthology.orchestra.run.smithi028.stdout: "7459dea4": { 2024-04-21T08:54:53.493 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 8, 2024-04-21T08:54:53.493 INFO:teuthology.orchestra.run.smithi028.stdout: "age_ref": 0.0, 2024-04-21T08:54:53.493 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-21T08:54:53.493 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-04-21T08:54:53.493 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-21T08:54:53.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T08:54:53.493 INFO:teuthology.orchestra.run.smithi028.stdout: "timeout": 30.0, 2024-04-21T08:54:53.493 INFO:teuthology.orchestra.run.smithi028.stdout: "expiration": 84.5, 2024-04-21T08:54:53.493 INFO:teuthology.orchestra.run.smithi028.stdout: "members": { 2024-04-21T08:54:53.493 INFO:teuthology.orchestra.run.smithi028.stdout: "file:/": { 2024-04-21T08:54:53.493 INFO:teuthology.orchestra.run.smithi028.stdout: "excluded": false, 2024-04-21T08:54:53.494 INFO:teuthology.orchestra.run.smithi028.stdout: "state": { 2024-04-21T08:54:53.494 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RELEASED", 2024-04-21T08:54:53.494 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0 2024-04-21T08:54:53.494 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:53.494 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:53.494 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:53.494 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:53.494 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:54:53.494 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:54:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.538+0000 7f431b225640 1 -- 172.21.15.28:0/3137374116 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f42fc0a3d20 msgr2=0x7f42fc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.538+0000 7f431b225640 1 --2- 172.21.15.28:0/3137374116 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f42fc0a3d20 0x7f42fc0a61e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f430400b620 tx=0x7f430400bb10 comp rx=0 tx=0).stop 2024-04-21T08:54:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.538+0000 7f431b225640 1 -- 172.21.15.28:0/3137374116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f431410e7e0 msgr2=0x7f431419a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.538+0000 7f431b225640 1 --2- 172.21.15.28:0/3137374116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f431410e7e0 0x7f431419a730 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f430c00cce0 tx=0x7f430c00a620 comp rx=0 tx=0).stop 2024-04-21T08:54:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.539+0000 7f431b225640 1 -- 172.21.15.28:0/3137374116 shutdown_connections 2024-04-21T08:54:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.539+0000 7f431b225640 1 --2- 172.21.15.28:0/3137374116 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f42fc0a3d20 0x7f42fc0a61e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.539+0000 7f431b225640 1 --2- 172.21.15.28:0/3137374116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f431410e7e0 0x7f431419a730 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.539+0000 7f431b225640 1 --2- 172.21.15.28:0/3137374116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4314074c20 0x7f43141a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.539+0000 7f431b225640 1 --2- 172.21.15.28:0/3137374116 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4314074300 0x7f43141a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:53.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.539+0000 7f431b225640 1 -- 172.21.15.28:0/3137374116 >> 172.21.15.28:0/3137374116 conn(0x7f43140fd990 msgr2=0x7f43140fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:53.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.541+0000 7f431b225640 1 -- 172.21.15.28:0/3137374116 shutdown_connections 2024-04-21T08:54:53.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:54:53.541+0000 7f431b225640 1 -- 172.21.15.28:0/3137374116 wait complete. 2024-04-21T08:54:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:53 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/875699412' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:53.568 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set_id: 7459dea4, seconds elapsed: 31.4 2024-04-21T08:54:53.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:53 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/875699412' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:53.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:53 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/875699412' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:54.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:54 smithi152 ceph-mon[31401]: pgmap v114: 129 pgs: 129 active+clean; 14 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 41 MiB/s wr, 6.99k op/s 2024-04-21T08:54:54.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:54 smithi028 ceph-mon[28003]: pgmap v114: 129 pgs: 129 active+clean; 14 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 41 MiB/s wr, 6.99k op/s 2024-04-21T08:54:54.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:54 smithi171 ceph-mon[31358]: pgmap v114: 129 pgs: 129 active+clean; 14 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 41 MiB/s wr, 6.99k op/s 2024-04-21T08:54:56.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:56 smithi152 ceph-mon[31401]: pgmap v115: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 38 MiB/s wr, 6.39k op/s 2024-04-21T08:54:56.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:56 smithi028 ceph-mon[28003]: pgmap v115: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 38 MiB/s wr, 6.39k op/s 2024-04-21T08:54:56.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:56 smithi171 ceph-mon[31358]: pgmap v115: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 38 MiB/s wr, 6.39k op/s 2024-04-21T08:54:57.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:57 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:54:57.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:57 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:54:57.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:57 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:54:57.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:57 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:54:57.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:57 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:54:57.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:57 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:54:57.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:57 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:54:57.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:57 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:54:57.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:57 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:54:58.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:54:58 smithi152 ceph-mon[31401]: pgmap v116: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 32 MiB/s wr, 5.26k op/s 2024-04-21T08:54:58.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:54:58 smithi028 ceph-mon[28003]: pgmap v116: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 32 MiB/s wr, 5.26k op/s 2024-04-21T08:54:58.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:54:58 smithi171 ceph-mon[31358]: pgmap v116: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 32 MiB/s wr, 5.26k op/s 2024-04-21T08:55:00.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:00 smithi152 ceph-mon[31401]: pgmap v117: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 51 MiB/s wr, 8.21k op/s 2024-04-21T08:55:00.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:00 smithi028 ceph-mon[28003]: pgmap v117: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 51 MiB/s wr, 8.21k op/s 2024-04-21T08:55:00.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:00 smithi171 ceph-mon[31358]: pgmap v117: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 51 MiB/s wr, 8.21k op/s 2024-04-21T08:55:00.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.694+0000 7fdad3c77640 1 -- 172.21.15.28:0/1426743948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdacc11b730 msgr2=0x7fdacc11db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:00.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.694+0000 7fdad3c77640 1 --2- 172.21.15.28:0/1426743948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdacc11b730 0x7fdacc11db20 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fdac401ace0 tx=0x7fdac403d1b0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.698+0000 7fdad3c77640 1 -- 172.21.15.28:0/1426743948 shutdown_connections 2024-04-21T08:55:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.698+0000 7fdad3c77640 1 --2- 172.21.15.28:0/1426743948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdacc11b730 0x7fdacc11db20 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.698+0000 7fdad3c77640 1 --2- 172.21.15.28:0/1426743948 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdacc110b20 0x7fdacc109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.698+0000 7fdad3c77640 1 --2- 172.21.15.28:0/1426743948 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdacc110740 0x7fdacc108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.698+0000 7fdad3c77640 1 -- 172.21.15.28:0/1426743948 >> 172.21.15.28:0/1426743948 conn(0x7fdacc06d020 msgr2=0x7fdacc06d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.701+0000 7fdad3c77640 1 -- 172.21.15.28:0/1426743948 shutdown_connections 2024-04-21T08:55:00.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.703+0000 7fdad3c77640 1 -- 172.21.15.28:0/1426743948 wait complete. 2024-04-21T08:55:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.703+0000 7fdad3c77640 1 Processor -- start 2024-04-21T08:55:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.705+0000 7fdad3c77640 1 -- start start 2024-04-21T08:55:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.705+0000 7fdad3c77640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdacc110740 0x7fdacc11ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.705+0000 7fdad3c77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdacc11b730 0x7fdacc116e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.705+0000 7fdad3c77640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdacc117360 0x7fdacc117810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.705+0000 7fdad3c77640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdacc113750 con 0x7fdacc11b730 2024-04-21T08:55:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.705+0000 7fdad3c77640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fdacc1135d0 con 0x7fdacc117360 2024-04-21T08:55:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.705+0000 7fdad3c77640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdacc1138d0 con 0x7fdacc110740 2024-04-21T08:55:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.705+0000 7fdad1a13640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdacc110740 0x7fdacc11ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.705+0000 7fdad1a13640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdacc110740 0x7fdacc11ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:41234/0 (socket says 172.21.15.28:41234) 2024-04-21T08:55:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.705+0000 7fdad1a13640 1 -- 172.21.15.28:0/3641025346 learned_addr learned my addr 172.21.15.28:0/3641025346 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.705+0000 7fdad1212640 1 --2- 172.21.15.28:0/3641025346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdacc11b730 0x7fdacc116e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.706+0000 7fdad1212640 1 -- 172.21.15.28:0/3641025346 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdacc110740 msgr2=0x7fdacc11ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:00.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.706+0000 7fdad1212640 1 --2- 172.21.15.28:0/3641025346 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdacc110740 0x7fdacc11ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.706+0000 7fdad1212640 1 -- 172.21.15.28:0/3641025346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdacc117360 msgr2=0x7fdacc117810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:55:00.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.706+0000 7fdad1212640 1 --2- 172.21.15.28:0/3641025346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdacc117360 0x7fdacc117810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.706+0000 7fdad1212640 1 -- 172.21.15.28:0/3641025346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdacc1ab520 con 0x7fdacc11b730 2024-04-21T08:55:00.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.706+0000 7fdad1212640 1 --2- 172.21.15.28:0/3641025346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdacc11b730 0x7fdacc116e20 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00d6a0 tx=0x7fdabc00db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:00.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.710+0000 7fdac2ffd640 1 -- 172.21.15.28:0/3641025346 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc039070 con 0x7fdacc11b730 2024-04-21T08:55:00.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.710+0000 7fdad3c77640 1 -- 172.21.15.28:0/3641025346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdacc1ab750 con 0x7fdacc11b730 2024-04-21T08:55:00.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.710+0000 7fdad3c77640 1 -- 172.21.15.28:0/3641025346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdacc1abc20 con 0x7fdacc11b730 2024-04-21T08:55:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.714+0000 7fdac2ffd640 1 -- 172.21.15.28:0/3641025346 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdabc009820 con 0x7fdacc11b730 2024-04-21T08:55:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.714+0000 7fdac2ffd640 1 -- 172.21.15.28:0/3641025346 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc004b90 con 0x7fdacc11b730 2024-04-21T08:55:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.714+0000 7fdad6862740 1 Processor -- start 2024-04-21T08:55:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.715+0000 7fdad6862740 1 -- start start 2024-04-21T08:55:00.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.715+0000 7fdad6862740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea7368d0 0x55f7ea735620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.716+0000 7fdad6862740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea735b60 0x55f7ea735fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.716+0000 7fdad2214640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea7368d0 0x55f7ea735620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.716+0000 7fdad2214640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea7368d0 0x55f7ea735620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:39552/0 (socket says 172.21.15.28:39552) 2024-04-21T08:55:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.716+0000 7fdad2214640 1 -- 172.21.15.28:0/126630626 learned_addr learned my addr 172.21.15.28:0/126630626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.716+0000 7fdad6862740 1 --2- 172.21.15.28:0/126630626 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea73f0e0 0x55f7ea7414d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.716+0000 7fdad6862740 1 -- 172.21.15.28:0/126630626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f7ea69a8d0 con 0x55f7ea735b60 2024-04-21T08:55:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.716+0000 7fdad6862740 1 -- 172.21.15.28:0/126630626 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f7ea637e80 con 0x55f7ea7368d0 2024-04-21T08:55:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.716+0000 7fdad6862740 1 -- 172.21.15.28:0/126630626 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f7ea696890 con 0x55f7ea73f0e0 2024-04-21T08:55:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.718+0000 7fdad1a13640 1 --2- 172.21.15.28:0/126630626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea735b60 0x55f7ea735fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.718+0000 7fdad1212640 1 --2- 172.21.15.28:0/126630626 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea73f0e0 0x55f7ea7414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.719+0000 7fdad2214640 1 -- 172.21.15.28:0/126630626 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea73f0e0 msgr2=0x55f7ea7414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.719+0000 7fdad2214640 1 --2- 172.21.15.28:0/126630626 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea73f0e0 0x55f7ea7414d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.719+0000 7fdad2214640 1 -- 172.21.15.28:0/126630626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea735b60 msgr2=0x55f7ea735fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.719+0000 7fdad2214640 1 --2- 172.21.15.28:0/126630626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea735b60 0x55f7ea735fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.719+0000 7fdad2214640 1 -- 172.21.15.28:0/126630626 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7ea659320 con 0x55f7ea7368d0 2024-04-21T08:55:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.720+0000 7fdac2ffd640 1 -- 172.21.15.28:0/3641025346 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdabc005220 con 0x7fdacc11b730 2024-04-21T08:55:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.721+0000 7fdac2ffd640 1 --2- 172.21.15.28:0/3641025346 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdab80a3ff0 0x7fdab80a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.721+0000 7fdac2ffd640 1 -- 172.21.15.28:0/3641025346 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fdabc0f7910 con 0x7fdacc11b730 2024-04-21T08:55:00.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.722+0000 7fdad1a13640 1 --2- 172.21.15.28:0/3641025346 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdab80a3ff0 0x7fdab80a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:00.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.723+0000 7fdad1a13640 1 --2- 172.21.15.28:0/3641025346 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdab80a3ff0 0x7fdab80a64b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x55f7ea737030 tx=0x7fdac800b480 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:00.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.724+0000 7fdad2214640 1 --2- 172.21.15.28:0/126630626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea7368d0 0x55f7ea735620 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdac4048040 tx=0x7fdac401a810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=877c3030b6911609 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:00.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.724+0000 7fdab7fff640 1 -- 172.21.15.28:0/126630626 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac4010020 con 0x55f7ea7368d0 2024-04-21T08:55:00.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.724+0000 7fdab7fff640 1 -- 172.21.15.28:0/126630626 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdac4003b70 con 0x55f7ea7368d0 2024-04-21T08:55:00.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.725+0000 7fdab7fff640 1 -- 172.21.15.28:0/126630626 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac400c6a0 con 0x55f7ea7368d0 2024-04-21T08:55:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.726+0000 7fdad6862740 1 -- 172.21.15.28:0/126630626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea7368d0 msgr2=0x55f7ea735620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:00.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.726+0000 7fdad6862740 1 --2- 172.21.15.28:0/126630626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea7368d0 0x55f7ea735620 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdac4048040 tx=0x7fdac401a810 comp rx=0 tx=0).stop 2024-04-21T08:55:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.732+0000 7fdad6862740 1 -- 172.21.15.28:0/126630626 shutdown_connections 2024-04-21T08:55:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.732+0000 7fdad6862740 1 --2- 172.21.15.28:0/126630626 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea73f0e0 0x55f7ea7414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.732+0000 7fdad6862740 1 --2- 172.21.15.28:0/126630626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea735b60 0x55f7ea735fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.732+0000 7fdad6862740 1 --2- 172.21.15.28:0/126630626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea7368d0 0x55f7ea735620 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.732+0000 7fdad6862740 1 -- 172.21.15.28:0/126630626 >> 172.21.15.28:0/126630626 conn(0x55f7ea6fcc90 msgr2=0x55f7ea730850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:00.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.734+0000 7fdad6862740 1 -- 172.21.15.28:0/126630626 shutdown_connections 2024-04-21T08:55:00.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.735+0000 7fdad6862740 1 -- 172.21.15.28:0/126630626 wait complete. 2024-04-21T08:55:00.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.736+0000 7fdad6862740 1 -- start start 2024-04-21T08:55:00.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.736+0000 7fdad6862740 1 Processor -- start 2024-04-21T08:55:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.736+0000 7fdad6862740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea735b60 0x55f7ea73e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.737+0000 7fdad6862740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea7368d0 0x55f7ea73bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.737+0000 7fdad6862740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea73f0e0 0x55f7ea73c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.737+0000 7fdad6862740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f7ea741e00 con 0x55f7ea73f0e0 2024-04-21T08:55:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.737+0000 7fdad6862740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f7ea622eb0 con 0x55f7ea735b60 2024-04-21T08:55:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.737+0000 7fdad6862740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f7ea741f80 con 0x55f7ea7368d0 2024-04-21T08:55:00.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.737+0000 7fdad6862740 10 client.0 _ll_register_callbacks cb 0x55f7ea64fb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.738+0000 7fdad1a13640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea735b60 0x55f7ea73e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.738+0000 7fdad2214640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea73f0e0 0x55f7ea73c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.738+0000 7fdad1a13640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea735b60 0x55f7ea73e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:39568/0 (socket says 172.21.15.28:39568) 2024-04-21T08:55:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.738+0000 7fdad2214640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea73f0e0 0x55f7ea73c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60900/0 (socket says 172.21.15.28:60900) 2024-04-21T08:55:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.738+0000 7fdad1a13640 1 -- 172.21.15.28:0/339390151 learned_addr learned my addr 172.21.15.28:0/339390151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.738+0000 7fdad2214640 1 -- 172.21.15.28:0/339390151 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea7368d0 msgr2=0x55f7ea73bd60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.738+0000 7fdad2214640 1 --2- 172.21.15.28:0/339390151 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea7368d0 0x55f7ea73bd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.738+0000 7fdad2214640 1 -- 172.21.15.28:0/339390151 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea735b60 msgr2=0x55f7ea73e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.738+0000 7fdad2214640 1 --2- 172.21.15.28:0/339390151 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea735b60 0x55f7ea73e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.738+0000 7fdad2214640 1 -- 172.21.15.28:0/339390151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7ea3ad210 con 0x55f7ea73f0e0 2024-04-21T08:55:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.739+0000 7fdad2214640 1 --2- 172.21.15.28:0/339390151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea73f0e0 0x55f7ea73c2a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fdac4006010 tx=0x7fdac40044e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f45c8ca200fe23 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.741+0000 7fdab57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:55:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.741+0000 7fdab57fa640 1 -- 172.21.15.28:0/339390151 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac4010070 con 0x55f7ea73f0e0 2024-04-21T08:55:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.741+0000 7fdab57fa640 1 -- 172.21.15.28:0/339390151 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fdac400c650 con 0x55f7ea73f0e0 2024-04-21T08:55:00.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.743+0000 7fdad6862740 1 -- 172.21.15.28:0/339390151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdac4003b70 con 0x55f7ea73f0e0 2024-04-21T08:55:00.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.745+0000 7fdab57fa640 1 -- 172.21.15.28:0/339390151 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdac4054050 con 0x55f7ea73f0e0 2024-04-21T08:55:00.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.746+0000 7fdad6862740 10 client.16941 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:00.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.746+0000 7fdad6862740 1 -- 172.21.15.28:0/339390151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f7ea479620 con 0x55f7ea73f0e0 2024-04-21T08:55:00.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.749+0000 7fdab57fa640 1 -- 172.21.15.28:0/339390151 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdac4003b70 con 0x55f7ea73f0e0 2024-04-21T08:55:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.749+0000 7fdab57fa640 1 -- 172.21.15.28:0/339390151 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fdac404f980 con 0x55f7ea73f0e0 2024-04-21T08:55:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.750+0000 7fdad6862740 10 client.16941 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.750+0000 7fdad6862740 10 client.16941 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.750+0000 7fdad6862740 20 client.16941 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.750+0000 7fdad6862740 1 --2- 172.21.15.28:0/339390151 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f7ea74ed40 0x55f7ea751130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.750+0000 7fdad6862740 4 client.16941 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.750+0000 7fdad6862740 1 -- 172.21.15.28:0/339390151 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7ea6ae880 con 0x55f7ea74ed40 2024-04-21T08:55:00.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.751+0000 7fdad1a13640 1 --2- 172.21.15.28:0/339390151 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f7ea74ed40 0x55f7ea751130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:00.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.758+0000 7fdad1a13640 1 --2- 172.21.15.28:0/339390151 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f7ea74ed40 0x55f7ea751130 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28160ad2dde984c3 server_cookie=bfdf20bea4440637 in_seq=0 out_seq=0 2024-04-21T08:55:00.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.758+0000 7fdab57fa640 10 client.16941 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.760+0000 7fdab57fa640 1 -- 172.21.15.28:0/339390151 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f7ea6ae880 con 0x55f7ea74ed40 2024-04-21T08:55:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.760+0000 7fdab57fa640 10 client.16941 handle_command_reply: tid=0 2024-04-21T08:55:00.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.760+0000 7fdad6862740 1 client.16941 shutdown 2024-04-21T08:55:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.760+0000 7fdad6862740 1 -- 172.21.15.28:0/339390151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea73f0e0 msgr2=0x55f7ea73c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.760+0000 7fdad6862740 1 --2- 172.21.15.28:0/339390151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea73f0e0 0x55f7ea73c2a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fdac4006010 tx=0x7fdac40044e0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.760+0000 7fdad6862740 1 -- 172.21.15.28:0/339390151 shutdown_connections 2024-04-21T08:55:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.760+0000 7fdad6862740 1 --2- 172.21.15.28:0/339390151 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f7ea74ed40 0x55f7ea751130 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.760+0000 7fdad6862740 1 --2- 172.21.15.28:0/339390151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea73f0e0 0x55f7ea73c2a0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.760+0000 7fdad6862740 1 --2- 172.21.15.28:0/339390151 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea7368d0 0x55f7ea73bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.760+0000 7fdad6862740 1 --2- 172.21.15.28:0/339390151 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea735b60 0x55f7ea73e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.760+0000 7fdad6862740 1 -- 172.21.15.28:0/339390151 >> 172.21.15.28:0/339390151 conn(0x55f7ea6fcc90 msgr2=0x55f7ea7325a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.760+0000 7fdab57fa640 0 client.16941 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:00.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.762+0000 7fdad6862740 1 -- 172.21.15.28:0/339390151 shutdown_connections 2024-04-21T08:55:00.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.762+0000 7fdad6862740 1 -- 172.21.15.28:0/339390151 wait complete. 2024-04-21T08:55:00.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.762+0000 7fdad6862740 20 client.16941 trim_cache size 0 max 16384 2024-04-21T08:55:00.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.797+0000 7fdad6862740 1 Processor -- start 2024-04-21T08:55:00.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.797+0000 7fdad6862740 1 -- start start 2024-04-21T08:55:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.797+0000 7fdad6862740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea735b60 0x55f7ea74ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.797+0000 7fdad6862740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea7368d0 0x55f7ea72f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.798+0000 7fdad6862740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea73f0e0 0x55f7ea73e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.798+0000 7fdad6862740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdab00128d0 con 0x55f7ea7368d0 2024-04-21T08:55:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.798+0000 7fdad6862740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fdab00101a0 con 0x55f7ea735b60 2024-04-21T08:55:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.798+0000 7fdad6862740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdab000dff0 con 0x55f7ea73f0e0 2024-04-21T08:55:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.798+0000 7fdad2214640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea73f0e0 0x55f7ea73e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.799+0000 7fdad2214640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea73f0e0 0x55f7ea73e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:41262/0 (socket says 172.21.15.28:41262) 2024-04-21T08:55:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.799+0000 7fdad2214640 1 -- 172.21.15.28:0/1852299234 learned_addr learned my addr 172.21.15.28:0/1852299234 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.800+0000 7fdad1212640 1 --2- 172.21.15.28:0/1852299234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea7368d0 0x55f7ea72f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:00.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.800+0000 7fdad1a13640 1 --2- 172.21.15.28:0/1852299234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea735b60 0x55f7ea74ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:00.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.801+0000 7fdad2214640 1 -- 172.21.15.28:0/1852299234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea735b60 msgr2=0x55f7ea74ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:00.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.801+0000 7fdad2214640 1 --2- 172.21.15.28:0/1852299234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea735b60 0x55f7ea74ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.801+0000 7fdad2214640 1 -- 172.21.15.28:0/1852299234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea7368d0 msgr2=0x55f7ea72f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:00.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.801+0000 7fdad2214640 1 --2- 172.21.15.28:0/1852299234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea7368d0 0x55f7ea72f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.801+0000 7fdad2214640 1 -- 172.21.15.28:0/1852299234 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7ea741ea0 con 0x55f7ea73f0e0 2024-04-21T08:55:00.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.803+0000 7fdad2214640 1 --2- 172.21.15.28:0/1852299234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea73f0e0 0x55f7ea73e490 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdac4005e30 tx=0x7fdac403d1b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f34c30bc5115032 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.805+0000 7fdab77fe640 1 -- 172.21.15.28:0/1852299234 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac4010070 con 0x55f7ea73f0e0 2024-04-21T08:55:00.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.805+0000 7fdab77fe640 1 -- 172.21.15.28:0/1852299234 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdac4003b70 con 0x55f7ea73f0e0 2024-04-21T08:55:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.806+0000 7fdad6862740 1 -- 172.21.15.28:0/1852299234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea73f0e0 msgr2=0x55f7ea73e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:00.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.806+0000 7fdad6862740 1 --2- 172.21.15.28:0/1852299234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea73f0e0 0x55f7ea73e490 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdac4005e30 tx=0x7fdac403d1b0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.808+0000 7fdad6862740 1 -- 172.21.15.28:0/1852299234 shutdown_connections 2024-04-21T08:55:00.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.808+0000 7fdad6862740 1 --2- 172.21.15.28:0/1852299234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea73f0e0 0x55f7ea73e490 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.808+0000 7fdad6862740 1 --2- 172.21.15.28:0/1852299234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea7368d0 0x55f7ea72f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.808+0000 7fdad6862740 1 --2- 172.21.15.28:0/1852299234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea735b60 0x55f7ea74ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.808+0000 7fdad6862740 1 -- 172.21.15.28:0/1852299234 >> 172.21.15.28:0/1852299234 conn(0x55f7ea6fcc90 msgr2=0x55f7ea72f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:00.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.810+0000 7fdad6862740 1 -- 172.21.15.28:0/1852299234 shutdown_connections 2024-04-21T08:55:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.812+0000 7fdad6862740 1 -- 172.21.15.28:0/1852299234 wait complete. 2024-04-21T08:55:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.812+0000 7fdad6862740 1 -- start start 2024-04-21T08:55:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.812+0000 7fdad6862740 1 Processor -- start 2024-04-21T08:55:00.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.813+0000 7fdad6862740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea735b60 0x55f7ea747680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.813+0000 7fdad6862740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea7368d0 0x55f7ea738740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.813+0000 7fdad6862740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea73f0e0 0x55f7ea738c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.813+0000 7fdad6862740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdab000e660 con 0x55f7ea73f0e0 2024-04-21T08:55:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.813+0000 7fdad6862740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fdab000fad0 con 0x55f7ea7368d0 2024-04-21T08:55:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.813+0000 7fdad6862740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdab000ed30 con 0x55f7ea735b60 2024-04-21T08:55:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.813+0000 7fdad6862740 10 client.0 _ll_register_callbacks cb 0x7fdac401ab30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:00.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.814+0000 7fdad2214640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea73f0e0 0x55f7ea738c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.814+0000 7fdad2214640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea73f0e0 0x55f7ea738c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60922/0 (socket says 172.21.15.28:60922) 2024-04-21T08:55:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.814+0000 7fdad2214640 1 -- 172.21.15.28:0/2140982363 learned_addr learned my addr 172.21.15.28:0/2140982363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.814+0000 7fdad1212640 1 --2- 172.21.15.28:0/2140982363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea7368d0 0x55f7ea738740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.814+0000 7fdad1212640 1 -- 172.21.15.28:0/2140982363 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea735b60 msgr2=0x55f7ea747680 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.814+0000 7fdad1212640 1 --2- 172.21.15.28:0/2140982363 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea735b60 0x55f7ea747680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.814+0000 7fdad1212640 1 -- 172.21.15.28:0/2140982363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea73f0e0 msgr2=0x55f7ea738c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.814+0000 7fdad1212640 1 --2- 172.21.15.28:0/2140982363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea73f0e0 0x55f7ea738c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:00.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.815+0000 7fdad1212640 1 -- 172.21.15.28:0/2140982363 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7ea739460 con 0x55f7ea7368d0 2024-04-21T08:55:00.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.816+0000 7fdad1212640 1 --2- 172.21.15.28:0/2140982363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea7368d0 0x55f7ea738740 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdabc0f4c60 tx=0x7fdabc0c2ff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=921c5170b9debf15 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.816+0000 7fdab57fa640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:55:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.817+0000 7fdab57fa640 1 -- 172.21.15.28:0/2140982363 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc042040 con 0x55f7ea7368d0 2024-04-21T08:55:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.817+0000 7fdad6862740 1 -- 172.21.15.28:0/2140982363 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdac4003b70 con 0x55f7ea7368d0 2024-04-21T08:55:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.817+0000 7fdab57fa640 1 -- 172.21.15.28:0/2140982363 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fdabc0f4ee0 con 0x55f7ea7368d0 2024-04-21T08:55:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.817+0000 7fdab57fa640 1 -- 172.21.15.28:0/2140982363 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdabc0f50e0 con 0x55f7ea7368d0 2024-04-21T08:55:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.817+0000 7fdad6862740 10 client.26213 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:00.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.817+0000 7fdad6862740 1 -- 172.21.15.28:0/2140982363 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f7ea7b4f90 con 0x55f7ea7368d0 2024-04-21T08:55:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.818+0000 7fdab57fa640 1 -- 172.21.15.28:0/2140982363 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdac4003b70 con 0x55f7ea7368d0 2024-04-21T08:55:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.819+0000 7fdab57fa640 1 -- 172.21.15.28:0/2140982363 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fdabc0bf170 con 0x55f7ea7368d0 2024-04-21T08:55:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.820+0000 7fdad6862740 10 client.26213 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.820+0000 7fdad6862740 10 client.26213 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.820+0000 7fdad6862740 20 client.26213 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.820+0000 7fdad6862740 1 --2- 172.21.15.28:0/2140982363 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f7ea7b5740 0x55f7ea7b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.820+0000 7fdad6862740 4 client.26213 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:55:00.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.820+0000 7fdad6862740 1 -- 172.21.15.28:0/2140982363 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f7ea63a3d0 con 0x55f7ea7b5740 2024-04-21T08:55:00.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.821+0000 7fdad1a13640 1 --2- 172.21.15.28:0/2140982363 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f7ea7b5740 0x55f7ea7b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.825+0000 7fdad1a13640 1 --2- 172.21.15.28:0/2140982363 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f7ea7b5740 0x55f7ea7b7b60 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=356fd509ac586d01 server_cookie=aa8575f315f22d58 in_seq=0 out_seq=0 2024-04-21T08:55:00.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:00.825+0000 7fdab57fa640 10 client.26213 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.519+0000 7fdab57fa640 1 -- 172.21.15.28:0/2140982363 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f7ea63a3d0 con 0x55f7ea7b5740 2024-04-21T08:55:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.519+0000 7fdab57fa640 10 client.26213 handle_command_reply: tid=0 2024-04-21T08:55:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.520+0000 7fdad6862740 1 client.26213 shutdown 2024-04-21T08:55:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.520+0000 7fdad6862740 1 -- 172.21.15.28:0/2140982363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea7368d0 msgr2=0x55f7ea738740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.520+0000 7fdad6862740 1 --2- 172.21.15.28:0/2140982363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea7368d0 0x55f7ea738740 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdabc0f4c60 tx=0x7fdabc0c2ff0 comp rx=0 tx=0).stop 2024-04-21T08:55:01.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.520+0000 7fdad6862740 1 -- 172.21.15.28:0/2140982363 shutdown_connections 2024-04-21T08:55:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.520+0000 7fdad6862740 1 --2- 172.21.15.28:0/2140982363 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f7ea7b5740 0x55f7ea7b7b60 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.520+0000 7fdad6862740 1 --2- 172.21.15.28:0/2140982363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7ea73f0e0 0x55f7ea738c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.520+0000 7fdad6862740 1 --2- 172.21.15.28:0/2140982363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7ea7368d0 0x55f7ea738740 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.520+0000 7fdad6862740 1 --2- 172.21.15.28:0/2140982363 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f7ea735b60 0x55f7ea747680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.520+0000 7fdad6862740 1 -- 172.21.15.28:0/2140982363 >> 172.21.15.28:0/2140982363 conn(0x55f7ea6fcc90 msgr2=0x55f7ea72f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.520+0000 7fdab57fa640 0 client.26213 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.520+0000 7fdad6862740 1 -- 172.21.15.28:0/2140982363 shutdown_connections 2024-04-21T08:55:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.521+0000 7fdad6862740 1 -- 172.21.15.28:0/2140982363 wait complete. 2024-04-21T08:55:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.521+0000 7fdad6862740 20 client.26213 trim_cache size 0 max 16384 2024-04-21T08:55:01.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:55:01.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:55:01.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:55:01.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:55:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.522+0000 7fdad3c77640 1 -- 172.21.15.28:0/3641025346 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdab80a3ff0 msgr2=0x7fdab80a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.522+0000 7fdad3c77640 1 --2- 172.21.15.28:0/3641025346 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdab80a3ff0 0x7fdab80a64b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x55f7ea737030 tx=0x7fdac800b480 comp rx=0 tx=0).stop 2024-04-21T08:55:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.522+0000 7fdad3c77640 1 -- 172.21.15.28:0/3641025346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdacc11b730 msgr2=0x7fdacc116e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.522+0000 7fdad3c77640 1 --2- 172.21.15.28:0/3641025346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdacc11b730 0x7fdacc116e20 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00d6a0 tx=0x7fdabc00db60 comp rx=0 tx=0).stop 2024-04-21T08:55:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.522+0000 7fdad3c77640 1 -- 172.21.15.28:0/3641025346 shutdown_connections 2024-04-21T08:55:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.522+0000 7fdad3c77640 1 --2- 172.21.15.28:0/3641025346 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdab80a3ff0 0x7fdab80a64b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.522+0000 7fdad3c77640 1 --2- 172.21.15.28:0/3641025346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdacc117360 0x7fdacc117810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.522+0000 7fdad3c77640 1 --2- 172.21.15.28:0/3641025346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdacc11b730 0x7fdacc116e20 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.522+0000 7fdad3c77640 1 --2- 172.21.15.28:0/3641025346 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdacc110740 0x7fdacc11ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.522+0000 7fdad3c77640 1 -- 172.21.15.28:0/3641025346 >> 172.21.15.28:0/3641025346 conn(0x7fdacc06d020 msgr2=0x7fdacc11bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.522+0000 7fdad3c77640 1 -- 172.21.15.28:0/3641025346 shutdown_connections 2024-04-21T08:55:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:01.522+0000 7fdad3c77640 1 -- 172.21.15.28:0/3641025346 wait complete. 2024-04-21T08:55:02.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:02 smithi152 ceph-mon[31401]: pgmap v118: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 48 MiB/s wr, 7.70k op/s 2024-04-21T08:55:02.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:02 smithi028 ceph-mon[28003]: pgmap v118: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 48 MiB/s wr, 7.70k op/s 2024-04-21T08:55:02.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:02 smithi171 ceph-mon[31358]: pgmap v118: 129 pgs: 129 active+clean; 14 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 48 MiB/s wr, 7.70k op/s 2024-04-21T08:55:04.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:04 smithi152 ceph-mon[31401]: pgmap v119: 129 pgs: 129 active+clean; 14 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 58 MiB/s wr, 9.20k op/s 2024-04-21T08:55:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:04 smithi028 ceph-mon[28003]: pgmap v119: 129 pgs: 129 active+clean; 14 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 58 MiB/s wr, 9.20k op/s 2024-04-21T08:55:04.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:04 smithi171 ceph-mon[31358]: pgmap v119: 129 pgs: 129 active+clean; 14 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 58 MiB/s wr, 9.20k op/s 2024-04-21T08:55:06.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:06 smithi152 ceph-mon[31401]: pgmap v120: 129 pgs: 129 active+clean; 14 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 55 MiB/s wr, 8.73k op/s 2024-04-21T08:55:06.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:06 smithi028 ceph-mon[28003]: pgmap v120: 129 pgs: 129 active+clean; 14 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 55 MiB/s wr, 8.73k op/s 2024-04-21T08:55:06.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:06 smithi171 ceph-mon[31358]: pgmap v120: 129 pgs: 129 active+clean; 14 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 55 MiB/s wr, 8.73k op/s 2024-04-21T08:55:08.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:08 smithi152 ceph-mon[31401]: pgmap v121: 129 pgs: 129 active+clean; 14 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 54 MiB/s wr, 8.53k op/s 2024-04-21T08:55:08.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:08 smithi028 ceph-mon[28003]: pgmap v121: 129 pgs: 129 active+clean; 14 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 54 MiB/s wr, 8.53k op/s 2024-04-21T08:55:08.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:08 smithi171 ceph-mon[31358]: pgmap v121: 129 pgs: 129 active+clean; 14 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 54 MiB/s wr, 8.53k op/s 2024-04-21T08:55:10.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:10 smithi171 ceph-mon[31358]: pgmap v122: 129 pgs: 129 active+clean; 15 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 66 MiB/s wr, 10.24k op/s 2024-04-21T08:55:10.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:10 smithi152 ceph-mon[31401]: pgmap v122: 129 pgs: 129 active+clean; 15 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 66 MiB/s wr, 10.24k op/s 2024-04-21T08:55:10.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:10 smithi028 ceph-mon[28003]: pgmap v122: 129 pgs: 129 active+clean; 15 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 66 MiB/s wr, 10.24k op/s 2024-04-21T08:55:12.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:12 smithi171 ceph-mon[31358]: pgmap v123: 129 pgs: 129 active+clean; 15 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 57 MiB/s wr, 8.65k op/s 2024-04-21T08:55:12.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:12 smithi152 ceph-mon[31401]: pgmap v123: 129 pgs: 129 active+clean; 15 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 57 MiB/s wr, 8.65k op/s 2024-04-21T08:55:12.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:12 smithi028 ceph-mon[28003]: pgmap v123: 129 pgs: 129 active+clean; 15 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 57 MiB/s wr, 8.65k op/s 2024-04-21T08:55:14.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:14 smithi171 ceph-mon[31358]: pgmap v124: 129 pgs: 129 active+clean; 15 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 62 MiB/s wr, 9.08k op/s 2024-04-21T08:55:14.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.684+0000 7febcaa43640 1 -- 172.21.15.28:0/2453706400 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7febc41010a0 msgr2=0x7febc4101520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:14.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.684+0000 7febcaa43640 1 --2- 172.21.15.28:0/2453706400 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7febc41010a0 0x7febc4101520 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7febb4009860 tx=0x7febb402bb90 comp rx=0 tx=0).stop 2024-04-21T08:55:14.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.688+0000 7febcaa43640 1 -- 172.21.15.28:0/2453706400 shutdown_connections 2024-04-21T08:55:14.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.688+0000 7febcaa43640 1 --2- 172.21.15.28:0/2453706400 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7febc4101a60 0x7febc410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.688+0000 7febcaa43640 1 --2- 172.21.15.28:0/2453706400 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7febc41010a0 0x7febc4101520 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.688+0000 7febcaa43640 1 --2- 172.21.15.28:0/2453706400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc4105de0 0x7febc41061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.688+0000 7febcaa43640 1 -- 172.21.15.28:0/2453706400 >> 172.21.15.28:0/2453706400 conn(0x7febc4077d00 msgr2=0x7febc4078110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:14.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.688+0000 7febcaa43640 1 -- 172.21.15.28:0/2453706400 shutdown_connections 2024-04-21T08:55:14.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.689+0000 7febcaa43640 1 -- 172.21.15.28:0/2453706400 wait complete. 2024-04-21T08:55:14.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.690+0000 7febcaa43640 1 Processor -- start 2024-04-21T08:55:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.690+0000 7febcaa43640 1 -- start start 2024-04-21T08:55:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.690+0000 7febcaa43640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7febc41010a0 0x7febc419e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.690+0000 7febcaa43640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc4101a60 0x7febc419e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.690+0000 7febcaa43640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7febc4105de0 0x7febc41966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.690+0000 7febcaa43640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7febc41111a0 con 0x7febc4101a60 2024-04-21T08:55:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.690+0000 7febcaa43640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7febc4111020 con 0x7febc4105de0 2024-04-21T08:55:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.690+0000 7febcaa43640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7febc4111320 con 0x7febc41010a0 2024-04-21T08:55:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.691+0000 7febc37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc4101a60 0x7febc419e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.691+0000 7febc37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc4101a60 0x7febc419e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44540/0 (socket says 172.21.15.28:44540) 2024-04-21T08:55:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.691+0000 7febc37fe640 1 -- 172.21.15.28:0/3505458974 learned_addr learned my addr 172.21.15.28:0/3505458974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.691+0000 7febc37fe640 1 -- 172.21.15.28:0/3505458974 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7febc41010a0 msgr2=0x7febc419e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.691+0000 7febc37fe640 1 --2- 172.21.15.28:0/3505458974 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7febc41010a0 0x7febc419e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.691+0000 7febc37fe640 1 -- 172.21.15.28:0/3505458974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7febc4105de0 msgr2=0x7febc41966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.691+0000 7febc37fe640 1 --2- 172.21.15.28:0/3505458974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7febc4105de0 0x7febc41966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.691+0000 7febc37fe640 1 -- 172.21.15.28:0/3505458974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc4196f90 con 0x7febc4101a60 2024-04-21T08:55:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.692+0000 7febc37fe640 1 --2- 172.21.15.28:0/3505458974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc4101a60 0x7febc419e880 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7febb402c120 tx=0x7febb4035990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.692+0000 7febc17fa640 1 -- 172.21.15.28:0/3505458974 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb4039040 con 0x7febc4101a60 2024-04-21T08:55:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.692+0000 7febcaa43640 1 -- 172.21.15.28:0/3505458974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febc4197220 con 0x7febc4101a60 2024-04-21T08:55:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.692+0000 7febcaa43640 1 -- 172.21.15.28:0/3505458974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febc41b4090 con 0x7febc4101a60 2024-04-21T08:55:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.693+0000 7febc17fa640 1 -- 172.21.15.28:0/3505458974 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febb4042070 con 0x7febc4101a60 2024-04-21T08:55:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.693+0000 7febc17fa640 1 -- 172.21.15.28:0/3505458974 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb402c9a0 con 0x7febc4101a60 2024-04-21T08:55:14.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.696+0000 7febc17fa640 1 -- 172.21.15.28:0/3505458974 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7febb40046c0 con 0x7febc4101a60 2024-04-21T08:55:14.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.702+0000 7febcd63b740 1 Processor -- start 2024-04-21T08:55:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.703+0000 7febcd63b740 1 -- start start 2024-04-21T08:55:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.703+0000 7febcd63b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 0x562e019eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.703+0000 7febcd63b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 0x562e019ebfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.703+0000 7febcd63b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019f50e0 0x562e019f74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.703+0000 7febcd63b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e019508d0 con 0x562e019ebb60 2024-04-21T08:55:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.703+0000 7febcd63b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562e018ede80 con 0x562e019ec8d0 2024-04-21T08:55:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.703+0000 7febcd63b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x562e0194c890 con 0x562e019f50e0 2024-04-21T08:55:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.703+0000 7febc8fe0640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 0x562e019eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.703+0000 7febc8fe0640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 0x562e019eb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38218/0 (socket says 172.21.15.28:38218) 2024-04-21T08:55:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.703+0000 7febc8fe0640 1 -- 172.21.15.28:0/2522293870 learned_addr learned my addr 172.21.15.28:0/2522293870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.707+0000 7febc3fff640 1 --2- 172.21.15.28:0/2522293870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 0x562e019ebfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.707+0000 7febc17fa640 1 --2- 172.21.15.28:0/3505458974 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7febb00a3e90 0x7febb00a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.708+0000 7febc37fe640 1 --2- 172.21.15.28:0/2522293870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019f50e0 0x562e019f74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.708+0000 7febc8fe0640 1 -- 172.21.15.28:0/2522293870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019f50e0 msgr2=0x562e019f74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.709+0000 7febc8fe0640 1 --2- 172.21.15.28:0/2522293870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019f50e0 0x562e019f74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.709+0000 7febc8fe0640 1 -- 172.21.15.28:0/2522293870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 msgr2=0x562e019ebfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.709+0000 7febc8fe0640 1 --2- 172.21.15.28:0/2522293870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 0x562e019ebfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.709+0000 7febc8fe0640 1 -- 172.21.15.28:0/2522293870 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e0190f320 con 0x562e019ec8d0 2024-04-21T08:55:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.709+0000 7febc8fe0640 1 --2- 172.21.15.28:0/3505458974 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7febb00a3e90 0x7febb00a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.709+0000 7febc37fe640 1 --2- 172.21.15.28:0/2522293870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019f50e0 0x562e019f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:55:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.709+0000 7febc8fe0640 1 --2- 172.21.15.28:0/2522293870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 0x562e019eb620 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7febac009dc0 tx=0x7febac00e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=966eea4e3df84c0b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.709+0000 7febc17fa640 1 -- 172.21.15.28:0/3505458974 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7febb40f20a0 con 0x7febc4101a60 2024-04-21T08:55:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.710+0000 7febba7fc640 1 -- 172.21.15.28:0/2522293870 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac012bc0 con 0x562e019ec8d0 2024-04-21T08:55:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.710+0000 7febba7fc640 1 -- 172.21.15.28:0/2522293870 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febac012d60 con 0x562e019ec8d0 2024-04-21T08:55:14.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.710+0000 7febba7fc640 1 -- 172.21.15.28:0/2522293870 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac019600 con 0x562e019ec8d0 2024-04-21T08:55:14.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.713+0000 7febcd63b740 1 -- 172.21.15.28:0/2522293870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 msgr2=0x562e019eb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:14.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.713+0000 7febcd63b740 1 --2- 172.21.15.28:0/2522293870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 0x562e019eb620 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7febac009dc0 tx=0x7febac00e720 comp rx=0 tx=0).stop 2024-04-21T08:55:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.714+0000 7febc8fe0640 1 --2- 172.21.15.28:0/3505458974 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7febb00a3e90 0x7febb00a6350 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7febac020000 tx=0x7febac021000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.717+0000 7febcd63b740 1 -- 172.21.15.28:0/2522293870 shutdown_connections 2024-04-21T08:55:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.717+0000 7febcd63b740 1 --2- 172.21.15.28:0/2522293870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019f50e0 0x562e019f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.717+0000 7febcd63b740 1 --2- 172.21.15.28:0/2522293870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 0x562e019ebfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.717+0000 7febcd63b740 1 --2- 172.21.15.28:0/2522293870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 0x562e019eb620 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.717+0000 7febcd63b740 1 -- 172.21.15.28:0/2522293870 >> 172.21.15.28:0/2522293870 conn(0x562e019b2c90 msgr2=0x562e019e6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:14.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.719+0000 7febcd63b740 1 -- 172.21.15.28:0/2522293870 shutdown_connections 2024-04-21T08:55:14.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.723+0000 7febcd63b740 1 -- 172.21.15.28:0/2522293870 wait complete. 2024-04-21T08:55:14.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.724+0000 7febcd63b740 1 -- start start 2024-04-21T08:55:14.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.724+0000 7febcd63b740 1 Processor -- start 2024-04-21T08:55:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.727+0000 7febcd63b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 0x562e019f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.727+0000 7febcd63b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019ec8d0 0x562e019f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.727+0000 7febcd63b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019f50e0 0x562e019f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.727+0000 7febcd63b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e019f7e10 con 0x562e019ebb60 2024-04-21T08:55:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.727+0000 7febcd63b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562e018d8eb0 con 0x562e019f50e0 2024-04-21T08:55:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.727+0000 7febcd63b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x562e019f7f90 con 0x562e019ec8d0 2024-04-21T08:55:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.727+0000 7febcd63b740 10 client.0 _ll_register_callbacks cb 0x562e01905b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.728+0000 7febc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 0x562e019f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.728+0000 7febc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 0x562e019f4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44570/0 (socket says 172.21.15.28:44570) 2024-04-21T08:55:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.728+0000 7febc3fff640 1 -- 172.21.15.28:0/1671333529 learned_addr learned my addr 172.21.15.28:0/1671333529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.728+0000 7febc3fff640 1 -- 172.21.15.28:0/1671333529 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019ec8d0 msgr2=0x562e019f1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.728+0000 7febc3fff640 1 --2- 172.21.15.28:0/1671333529 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019ec8d0 0x562e019f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.728+0000 7febc3fff640 1 -- 172.21.15.28:0/1671333529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019f50e0 msgr2=0x562e019f22f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.728+0000 7febc3fff640 1 --2- 172.21.15.28:0/1671333529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019f50e0 0x562e019f22f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.728+0000 7febc3fff640 1 -- 172.21.15.28:0/1671333529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e01663210 con 0x562e019ebb60 2024-04-21T08:55:14.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.729+0000 7febc3fff640 1 --2- 172.21.15.28:0/1671333529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 0x562e019f4a20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7febbc010a80 tx=0x7febbc010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a8b485412799b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.731+0000 7feba37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:55:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.732+0000 7feba37fe640 1 -- 172.21.15.28:0/1671333529 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febbc00eea0 con 0x562e019ebb60 2024-04-21T08:55:14.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.732+0000 7febcd63b740 1 -- 172.21.15.28:0/1671333529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febac012d60 con 0x562e019ebb60 2024-04-21T08:55:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.732+0000 7feba37fe640 1 -- 172.21.15.28:0/1671333529 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7febbc004b20 con 0x562e019ebb60 2024-04-21T08:55:14.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.732+0000 7feba37fe640 1 -- 172.21.15.28:0/1671333529 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7febbc004d20 con 0x562e019ebb60 2024-04-21T08:55:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.734+0000 7febcd63b740 10 client.16959 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:14.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.734+0000 7febcd63b740 1 -- 172.21.15.28:0/1671333529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562e0172f620 con 0x562e019ebb60 2024-04-21T08:55:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.735+0000 7feba37fe640 1 -- 172.21.15.28:0/1671333529 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febac012d60 con 0x562e019ebb60 2024-04-21T08:55:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.735+0000 7feba37fe640 1 -- 172.21.15.28:0/1671333529 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7febbc00b040 con 0x562e019ebb60 2024-04-21T08:55:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.735+0000 7febcd63b740 10 client.16959 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.735+0000 7febcd63b740 10 client.16959 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.736+0000 7febcd63b740 20 client.16959 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.736+0000 7febcd63b740 1 --2- 172.21.15.28:0/1671333529 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e01a05520 0x562e01a079e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.736+0000 7febcd63b740 4 client.16959 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.736+0000 7febcd63b740 1 -- 172.21.15.28:0/1671333529 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e01964880 con 0x562e01a05520 2024-04-21T08:55:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.736+0000 7febc37fe640 1 --2- 172.21.15.28:0/1671333529 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e01a05520 0x562e01a079e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.745+0000 7febc37fe640 1 --2- 172.21.15.28:0/1671333529 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e01a05520 0x562e01a079e0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=105b767b8813d1e server_cookie=eaccbbe52358422d in_seq=0 out_seq=0 2024-04-21T08:55:14.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.745+0000 7feba37fe640 10 client.16959 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.746+0000 7feba37fe640 1 -- 172.21.15.28:0/1671333529 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e01964880 con 0x562e01a05520 2024-04-21T08:55:14.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.746+0000 7feba37fe640 10 client.16959 handle_command_reply: tid=0 2024-04-21T08:55:14.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.749+0000 7febcd63b740 1 client.16959 shutdown 2024-04-21T08:55:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.751+0000 7febcd63b740 1 -- 172.21.15.28:0/1671333529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 msgr2=0x562e019f4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:14.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.751+0000 7febcd63b740 1 --2- 172.21.15.28:0/1671333529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 0x562e019f4a20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7febbc010a80 tx=0x7febbc010f40 comp rx=0 tx=0).stop 2024-04-21T08:55:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.757+0000 7febcd63b740 1 -- 172.21.15.28:0/1671333529 shutdown_connections 2024-04-21T08:55:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.757+0000 7febcd63b740 1 --2- 172.21.15.28:0/1671333529 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e01a05520 0x562e01a079e0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.757+0000 7febcd63b740 1 --2- 172.21.15.28:0/1671333529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019f50e0 0x562e019f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.757+0000 7febcd63b740 1 --2- 172.21.15.28:0/1671333529 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019ec8d0 0x562e019f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.757+0000 7febcd63b740 1 --2- 172.21.15.28:0/1671333529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 0x562e019f4a20 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.757+0000 7febcd63b740 1 -- 172.21.15.28:0/1671333529 >> 172.21.15.28:0/1671333529 conn(0x562e019b2c90 msgr2=0x562e019e85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.757+0000 7feba37fe640 0 client.16959 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.760+0000 7febcd63b740 1 -- 172.21.15.28:0/1671333529 shutdown_connections 2024-04-21T08:55:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.761+0000 7febcd63b740 1 -- 172.21.15.28:0/1671333529 wait complete. 2024-04-21T08:55:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.761+0000 7febcd63b740 20 client.16959 trim_cache size 0 max 16384 2024-04-21T08:55:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.798+0000 7febcd63b740 1 Processor -- start 2024-04-21T08:55:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.802+0000 7febcd63b740 1 -- start start 2024-04-21T08:55:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.802+0000 7febcd63b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019ebb60 0x562e01a00d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.802+0000 7febcd63b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 0x562e019f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.802+0000 7febcd63b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019f50e0 0x562e019f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.802+0000 7febcd63b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feba40128b0 con 0x562e019f50e0 2024-04-21T08:55:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.802+0000 7febcd63b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7feba4010180 con 0x562e019ec8d0 2024-04-21T08:55:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.802+0000 7febcd63b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7feba400dfd0 con 0x562e019ebb60 2024-04-21T08:55:14.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.803+0000 7febc8fe0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019f50e0 0x562e019f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.803+0000 7febc8fe0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019f50e0 0x562e019f4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44578/0 (socket says 172.21.15.28:44578) 2024-04-21T08:55:14.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.803+0000 7febc8fe0640 1 -- 172.21.15.28:0/304459457 learned_addr learned my addr 172.21.15.28:0/304459457 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:14.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.803+0000 7febc37fe640 1 --2- 172.21.15.28:0/304459457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 0x562e019f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.803+0000 7febc3fff640 1 --2- 172.21.15.28:0/304459457 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019ebb60 0x562e01a00d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.804+0000 7febc37fe640 1 -- 172.21.15.28:0/304459457 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019ebb60 msgr2=0x562e01a00d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:14.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.804+0000 7febc37fe640 1 --2- 172.21.15.28:0/304459457 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019ebb60 0x562e01a00d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.804+0000 7febc37fe640 1 -- 172.21.15.28:0/304459457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019f50e0 msgr2=0x562e019f4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:14.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.804+0000 7febc37fe640 1 --2- 172.21.15.28:0/304459457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019f50e0 0x562e019f4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.804+0000 7febc37fe640 1 -- 172.21.15.28:0/304459457 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e019e69f0 con 0x562e019ec8d0 2024-04-21T08:55:14.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.804+0000 7febc37fe640 1 --2- 172.21.15.28:0/304459457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 0x562e019f44e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7febb40ef560 tx=0x7febb40f4040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=574f275fbe08e1e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.805+0000 7febb9ffb640 1 -- 172.21.15.28:0/304459457 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb40be4b0 con 0x562e019ec8d0 2024-04-21T08:55:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.805+0000 7febb9ffb640 1 -- 172.21.15.28:0/304459457 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febb40bddb0 con 0x562e019ec8d0 2024-04-21T08:55:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.805+0000 7febb9ffb640 1 -- 172.21.15.28:0/304459457 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb40bc0b0 con 0x562e019ec8d0 2024-04-21T08:55:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.806+0000 7febcd63b740 1 -- 172.21.15.28:0/304459457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 msgr2=0x562e019f44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:14.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.806+0000 7febcd63b740 1 --2- 172.21.15.28:0/304459457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 0x562e019f44e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7febb40ef560 tx=0x7febb40f4040 comp rx=0 tx=0).stop 2024-04-21T08:55:14.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.809+0000 7febcd63b740 1 -- 172.21.15.28:0/304459457 shutdown_connections 2024-04-21T08:55:14.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.809+0000 7febcd63b740 1 --2- 172.21.15.28:0/304459457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019f50e0 0x562e019f4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.809+0000 7febcd63b740 1 --2- 172.21.15.28:0/304459457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 0x562e019f44e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.809+0000 7febcd63b740 1 --2- 172.21.15.28:0/304459457 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019ebb60 0x562e01a00d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.809+0000 7febcd63b740 1 -- 172.21.15.28:0/304459457 >> 172.21.15.28:0/304459457 conn(0x562e019b2c90 msgr2=0x562e019f1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.811+0000 7febcd63b740 1 -- 172.21.15.28:0/304459457 shutdown_connections 2024-04-21T08:55:14.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.811+0000 7febcd63b740 1 -- 172.21.15.28:0/304459457 wait complete. 2024-04-21T08:55:14.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.812+0000 7febcd63b740 1 -- start start 2024-04-21T08:55:14.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.812+0000 7febcd63b740 1 Processor -- start 2024-04-21T08:55:14.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.812+0000 7febcd63b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 0x562e019fd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.812+0000 7febcd63b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 0x562e019f8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.812+0000 7febcd63b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019f50e0 0x562e019f8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.812+0000 7febcd63b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feba400e640 con 0x562e019ebb60 2024-04-21T08:55:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.812+0000 7febcd63b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7feba400fab0 con 0x562e019ec8d0 2024-04-21T08:55:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.812+0000 7febcd63b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7feba400ed10 con 0x562e019f50e0 2024-04-21T08:55:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.812+0000 7febcd63b740 10 client.0 _ll_register_callbacks cb 0x7febbc009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.813+0000 7febc8fe0640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019f50e0 0x562e019f8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.813+0000 7febc8fe0640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019f50e0 0x562e019f8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:53128/0 (socket says 172.21.15.28:53128) 2024-04-21T08:55:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.813+0000 7febc8fe0640 1 -- 172.21.15.28:0/4045698223 learned_addr learned my addr 172.21.15.28:0/4045698223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.813+0000 7febc37fe640 1 --2- 172.21.15.28:0/4045698223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 0x562e019f8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.815+0000 7febc3fff640 1 --2- 172.21.15.28:0/4045698223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 0x562e019fd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.816+0000 7febc8fe0640 1 -- 172.21.15.28:0/4045698223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 msgr2=0x562e019f8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:14.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.816+0000 7febc8fe0640 1 --2- 172.21.15.28:0/4045698223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 0x562e019f8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.816+0000 7febc8fe0640 1 -- 172.21.15.28:0/4045698223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 msgr2=0x562e019fd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:14.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.816+0000 7febc8fe0640 1 --2- 172.21.15.28:0/4045698223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 0x562e019fd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.816+0000 7febc8fe0640 1 -- 172.21.15.28:0/4045698223 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e019f8fa0 con 0x562e019f50e0 2024-04-21T08:55:14.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.816+0000 7febc8fe0640 1 --2- 172.21.15.28:0/4045698223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019f50e0 0x562e019f8770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7febac01d040 tx=0x7febac004620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bb8d4c8d8511da9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:14.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.816+0000 7feba37fe640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:55:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.817+0000 7feba37fe640 1 -- 172.21.15.28:0/4045698223 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac02b040 con 0x562e019f50e0 2024-04-21T08:55:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.817+0000 7feba37fe640 1 -- 172.21.15.28:0/4045698223 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7febac0193a0 con 0x562e019f50e0 2024-04-21T08:55:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.817+0000 7febcd63b740 1 -- 172.21.15.28:0/4045698223 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febb40bddb0 con 0x562e019f50e0 2024-04-21T08:55:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.820+0000 7feba37fe640 1 -- 172.21.15.28:0/4045698223 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7febac012dc0 con 0x562e019f50e0 2024-04-21T08:55:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.820+0000 7febcd63b740 10 client.26242 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.820+0000 7febcd63b740 1 -- 172.21.15.28:0/4045698223 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562e01a7bc00 con 0x562e019f50e0 2024-04-21T08:55:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.820+0000 7feba37fe640 1 -- 172.21.15.28:0/4045698223 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febb40bddb0 con 0x562e019f50e0 2024-04-21T08:55:14.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.821+0000 7feba37fe640 1 -- 172.21.15.28:0/4045698223 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7febac0319a0 con 0x562e019f50e0 2024-04-21T08:55:14.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.821+0000 7febcd63b740 10 client.26242 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:14.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.821+0000 7febcd63b740 10 client.26242 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:14.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.821+0000 7febcd63b740 20 client.26242 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:14.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.821+0000 7febcd63b740 1 --2- 172.21.15.28:0/4045698223 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e01a7c2c0 0x562e01a7e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.821+0000 7febcd63b740 4 client.26242 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:55:14.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.821+0000 7febcd63b740 1 -- 172.21.15.28:0/4045698223 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e018f03d0 con 0x562e01a7c2c0 2024-04-21T08:55:14.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.824+0000 7febc3fff640 1 --2- 172.21.15.28:0/4045698223 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e01a7c2c0 0x562e01a7e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.833+0000 7febc3fff640 1 --2- 172.21.15.28:0/4045698223 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e01a7c2c0 0x562e01a7e6e0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c461ac00bf66757 server_cookie=bd8d27ce0631cf0 in_seq=0 out_seq=0 2024-04-21T08:55:14.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:14.833+0000 7feba37fe640 10 client.26242 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:14.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:14 smithi152 ceph-mon[31401]: pgmap v124: 129 pgs: 129 active+clean; 15 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 62 MiB/s wr, 9.08k op/s 2024-04-21T08:55:14.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:14 smithi028 ceph-mon[28003]: pgmap v124: 129 pgs: 129 active+clean; 15 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 62 MiB/s wr, 9.08k op/s 2024-04-21T08:55:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.005+0000 7feba37fe640 1 -- 172.21.15.28:0/4045698223 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562e018f03d0 con 0x562e01a7c2c0 2024-04-21T08:55:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.005+0000 7feba37fe640 10 client.26242 handle_command_reply: tid=0 2024-04-21T08:55:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.005+0000 7febcd63b740 1 client.26242 shutdown 2024-04-21T08:55:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.005+0000 7febcd63b740 1 -- 172.21.15.28:0/4045698223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019f50e0 msgr2=0x562e019f8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.005+0000 7febcd63b740 1 --2- 172.21.15.28:0/4045698223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019f50e0 0x562e019f8770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7febac01d040 tx=0x7febac004620 comp rx=0 tx=0).stop 2024-04-21T08:55:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.005+0000 7febcd63b740 1 -- 172.21.15.28:0/4045698223 shutdown_connections 2024-04-21T08:55:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.005+0000 7febcd63b740 1 --2- 172.21.15.28:0/4045698223 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e01a7c2c0 0x562e01a7e6e0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.005+0000 7febcd63b740 1 --2- 172.21.15.28:0/4045698223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e019f50e0 0x562e019f8770 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.005+0000 7febcd63b740 1 --2- 172.21.15.28:0/4045698223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e019ec8d0 0x562e019f8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.005+0000 7febcd63b740 1 --2- 172.21.15.28:0/4045698223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e019ebb60 0x562e019fd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.006+0000 7febcd63b740 1 -- 172.21.15.28:0/4045698223 >> 172.21.15.28:0/4045698223 conn(0x562e019b2c90 msgr2=0x562e019ed080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.006+0000 7feba37fe640 0 client.26242 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:15.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.006+0000 7febcd63b740 1 -- 172.21.15.28:0/4045698223 shutdown_connections 2024-04-21T08:55:15.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.006+0000 7febcd63b740 1 -- 172.21.15.28:0/4045698223 wait complete. 2024-04-21T08:55:15.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.006+0000 7febcd63b740 20 client.26242 trim_cache size 0 max 16384 2024-04-21T08:55:15.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:55:15.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:55:15.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:55:15.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:55:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.007+0000 7febcaa43640 1 -- 172.21.15.28:0/3505458974 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7febb00a3e90 msgr2=0x7febb00a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.007+0000 7febcaa43640 1 --2- 172.21.15.28:0/3505458974 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7febb00a3e90 0x7febb00a6350 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7febac020000 tx=0x7febac021000 comp rx=0 tx=0).stop 2024-04-21T08:55:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.007+0000 7febcaa43640 1 -- 172.21.15.28:0/3505458974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc4101a60 msgr2=0x7febc419e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.007+0000 7febcaa43640 1 --2- 172.21.15.28:0/3505458974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc4101a60 0x7febc419e880 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7febb402c120 tx=0x7febb4035990 comp rx=0 tx=0).stop 2024-04-21T08:55:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.007+0000 7febcaa43640 1 -- 172.21.15.28:0/3505458974 shutdown_connections 2024-04-21T08:55:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.007+0000 7febcaa43640 1 --2- 172.21.15.28:0/3505458974 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7febb00a3e90 0x7febb00a6350 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.007+0000 7febcaa43640 1 --2- 172.21.15.28:0/3505458974 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7febc4105de0 0x7febc41966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.007+0000 7febcaa43640 1 --2- 172.21.15.28:0/3505458974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc4101a60 0x7febc419e880 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.007+0000 7febcaa43640 1 --2- 172.21.15.28:0/3505458974 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7febc41010a0 0x7febc419e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.007+0000 7febcaa43640 1 -- 172.21.15.28:0/3505458974 >> 172.21.15.28:0/3505458974 conn(0x7febc4077d00 msgr2=0x7febc410cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.007+0000 7febcaa43640 1 -- 172.21.15.28:0/3505458974 shutdown_connections 2024-04-21T08:55:15.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:15.007+0000 7febcaa43640 1 -- 172.21.15.28:0/3505458974 wait complete. 2024-04-21T08:55:16.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:16 smithi152 ceph-mon[31401]: pgmap v125: 129 pgs: 129 active+clean; 15 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 57 MiB/s wr, 8.20k op/s 2024-04-21T08:55:16.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:16 smithi028 ceph-mon[28003]: pgmap v125: 129 pgs: 129 active+clean; 15 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 57 MiB/s wr, 8.20k op/s 2024-04-21T08:55:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:16 smithi171 ceph-mon[31358]: pgmap v125: 129 pgs: 129 active+clean; 15 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 57 MiB/s wr, 8.20k op/s 2024-04-21T08:55:18.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:18 smithi152 ceph-mon[31401]: pgmap v126: 129 pgs: 129 active+clean; 15 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 57 MiB/s wr, 8.10k op/s 2024-04-21T08:55:18.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:18 smithi028 ceph-mon[28003]: pgmap v126: 129 pgs: 129 active+clean; 15 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 57 MiB/s wr, 8.10k op/s 2024-04-21T08:55:18.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:18 smithi171 ceph-mon[31358]: pgmap v126: 129 pgs: 129 active+clean; 15 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 57 MiB/s wr, 8.10k op/s 2024-04-21T08:55:20.248 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:55:20.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:55:20.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.441+0000 7f4b0c4f1640 1 -- 172.21.15.28:0/3634384011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b04100610 msgr2=0x7f4b04100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:20.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.441+0000 7f4b0c4f1640 1 --2- 172.21.15.28:0/3634384011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b04100610 0x7f4b04100a90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4af400aa40 tx=0x7f4af402cf60 comp rx=0 tx=0).stop 2024-04-21T08:55:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.443+0000 7f4b0c4f1640 1 -- 172.21.15.28:0/3634384011 shutdown_connections 2024-04-21T08:55:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.443+0000 7f4b0c4f1640 1 --2- 172.21.15.28:0/3634384011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100fd0 0x7f4b0410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.443+0000 7f4b0c4f1640 1 --2- 172.21.15.28:0/3634384011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b04100610 0x7f4b04100a90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.443+0000 7f4b0c4f1640 1 --2- 172.21.15.28:0/3634384011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b04106680 0x7f4b04106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.443+0000 7f4b0c4f1640 1 -- 172.21.15.28:0/3634384011 >> 172.21.15.28:0/3634384011 conn(0x7f4b04077360 msgr2=0x7f4b040fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.443+0000 7f4b0c4f1640 1 -- 172.21.15.28:0/3634384011 shutdown_connections 2024-04-21T08:55:20.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.443+0000 7f4b0c4f1640 1 -- 172.21.15.28:0/3634384011 wait complete. 2024-04-21T08:55:20.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.444+0000 7f4b0c4f1640 1 Processor -- start 2024-04-21T08:55:20.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.444+0000 7f4b0c4f1640 1 -- start start 2024-04-21T08:55:20.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.445+0000 7f4b0c4f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100610 0x7f4b04073710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.445+0000 7f4b0c4f1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b04100fd0 0x7f4b0406c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.445+0000 7f4b0c4f1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b04106680 0x7f4b0406ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.445+0000 7f4b0c4f1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b04110040 con 0x7f4b04100610 2024-04-21T08:55:20.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.445+0000 7f4b0c4f1640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0410fec0 con 0x7f4b04100fd0 2024-04-21T08:55:20.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.445+0000 7f4b0c4f1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4b041101c0 con 0x7f4b04106680 2024-04-21T08:55:20.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.445+0000 7f4b0aa8e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b04106680 0x7f4b0406ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.445+0000 7f4b0aa8e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b04106680 0x7f4b0406ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:53134/0 (socket says 172.21.15.28:53134) 2024-04-21T08:55:20.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.445+0000 7f4b0aa8e640 1 -- 172.21.15.28:0/575290426 learned_addr learned my addr 172.21.15.28:0/575290426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:20.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.445+0000 7f4b0a28d640 1 --2- 172.21.15.28:0/575290426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100610 0x7f4b04073710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.446+0000 7f4b09a8c640 1 --2- 172.21.15.28:0/575290426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b04100fd0 0x7f4b0406c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.446+0000 7f4b0a28d640 1 -- 172.21.15.28:0/575290426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b04106680 msgr2=0x7f4b0406ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.446+0000 7f4b0a28d640 1 --2- 172.21.15.28:0/575290426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b04106680 0x7f4b0406ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.446+0000 7f4b0a28d640 1 -- 172.21.15.28:0/575290426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b04100fd0 msgr2=0x7f4b0406c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.446+0000 7f4b0a28d640 1 --2- 172.21.15.28:0/575290426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b04100fd0 0x7f4b0406c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.446+0000 7f4b0a28d640 1 -- 172.21.15.28:0/575290426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0406d240 con 0x7f4b04100610 2024-04-21T08:55:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.447+0000 7f4b0a28d640 1 --2- 172.21.15.28:0/575290426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100610 0x7f4b04073710 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f4afc010e10 tx=0x7f4afc00b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.447+0000 7f4b037fe640 1 -- 172.21.15.28:0/575290426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc01e070 con 0x7f4b04100610 2024-04-21T08:55:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.447+0000 7f4b0c4f1640 1 -- 172.21.15.28:0/575290426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b0406d530 con 0x7f4b04100610 2024-04-21T08:55:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.447+0000 7f4b0c4f1640 1 -- 172.21.15.28:0/575290426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b04071520 con 0x7f4b04100610 2024-04-21T08:55:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.447+0000 7f4b037fe640 1 -- 172.21.15.28:0/575290426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4afc0179d0 con 0x7f4b04100610 2024-04-21T08:55:20.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.448+0000 7f4b037fe640 1 -- 172.21.15.28:0/575290426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc019950 con 0x7f4b04100610 2024-04-21T08:55:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.451+0000 7f4b037fe640 1 -- 172.21.15.28:0/575290426 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4afc021070 con 0x7f4b04100610 2024-04-21T08:55:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.453+0000 7f4b0f1b6740 1 Processor -- start 2024-04-21T08:55:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.453+0000 7f4b037fe640 1 --2- 172.21.15.28:0/575290426 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4aec0a3d50 0x7f4aec0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.453+0000 7f4b0f1b6740 1 -- start start 2024-04-21T08:55:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.453+0000 7f4b037fe640 1 -- 172.21.15.28:0/575290426 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f4afc0d4420 con 0x7f4b04100610 2024-04-21T08:55:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.453+0000 7f4b0f1b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd122a50 0x5617dd122e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.453+0000 7f4b0f1b6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd123370 0x5617dd11c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.453+0000 7f4b0f1b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 0x5617dd1370f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.453+0000 7f4b0f1b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5617dd07f1e0 con 0x5617dd134d00 2024-04-21T08:55:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.453+0000 7f4b0f1b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5617dd01c390 con 0x5617dd122a50 2024-04-21T08:55:20.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.453+0000 7f4b0f1b6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5617dd07b1a0 con 0x5617dd123370 2024-04-21T08:55:20.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.454+0000 7f4b0a28d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd122a50 0x5617dd122e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.454+0000 7f4b0a28d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd122a50 0x5617dd122e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38278/0 (socket says 172.21.15.28:38278) 2024-04-21T08:55:20.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.454+0000 7f4b0a28d640 1 -- 172.21.15.28:0/1689902087 learned_addr learned my addr 172.21.15.28:0/1689902087 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:20.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.454+0000 7f4b0aa8e640 1 --2- 172.21.15.28:0/1689902087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 0x5617dd1370f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.454+0000 7f4b0aa8e640 1 --2- 172.21.15.28:0/575290426 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4aec0a3d50 0x7f4aec0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.455+0000 7f4b0aa8e640 1 -- 172.21.15.28:0/1689902087 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd123370 msgr2=0x5617dd11c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:20.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.455+0000 7f4b0aa8e640 1 --2- 172.21.15.28:0/1689902087 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd123370 0x5617dd11c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.455+0000 7f4b0aa8e640 1 -- 172.21.15.28:0/1689902087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd122a50 msgr2=0x5617dd122e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:20.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.455+0000 7f4b0aa8e640 1 --2- 172.21.15.28:0/1689902087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd122a50 0x5617dd122e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.455+0000 7f4b0aa8e640 1 -- 172.21.15.28:0/1689902087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617dce5b7d0 con 0x5617dd134d00 2024-04-21T08:55:20.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.455+0000 7f4b0aa8e640 1 --2- 172.21.15.28:0/575290426 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4aec0a3d50 0x7f4aec0a6210 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4af800e800 tx=0x7f4af8009810 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:20.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.456+0000 7f4b0aa8e640 1 --2- 172.21.15.28:0/1689902087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 0x5617dd1370f0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f4af80057b0 tx=0x7f4af801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9408d0274d3c506 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:20.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.456+0000 7f4b00bfe640 1 -- 172.21.15.28:0/1689902087 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af802a070 con 0x5617dd134d00 2024-04-21T08:55:20.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.457+0000 7f4b00bfe640 1 -- 172.21.15.28:0/1689902087 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4af800ee40 con 0x5617dd134d00 2024-04-21T08:55:20.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.457+0000 7f4b00bfe640 1 -- 172.21.15.28:0/1689902087 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af80259b0 con 0x5617dd134d00 2024-04-21T08:55:20.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.457+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/1689902087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 msgr2=0x5617dd1370f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:20.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.457+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/1689902087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 0x5617dd1370f0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f4af80057b0 tx=0x7f4af801c000 comp rx=0 tx=0).stop 2024-04-21T08:55:20.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.458+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/1689902087 shutdown_connections 2024-04-21T08:55:20.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.458+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/1689902087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 0x5617dd1370f0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.458+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/1689902087 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd123370 0x5617dd11c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.458+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/1689902087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd122a50 0x5617dd122e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.458+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/1689902087 >> 172.21.15.28:0/1689902087 conn(0x5617dd11d4c0 msgr2=0x5617dd11d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:20.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.462+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/1689902087 shutdown_connections 2024-04-21T08:55:20.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.463+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/1689902087 wait complete. 2024-04-21T08:55:20.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.463+0000 7f4b0f1b6740 1 -- start start 2024-04-21T08:55:20.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.464+0000 7f4b0f1b6740 1 Processor -- start 2024-04-21T08:55:20.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.464+0000 7f4b0f1b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd122a50 0x5617dd119ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.464+0000 7f4b0f1b6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd123370 0x5617dd119ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.464+0000 7f4b0f1b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd134d00 0x5617dd11a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.464+0000 7f4b0f1b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5617dd1378c0 con 0x5617dd122a50 2024-04-21T08:55:20.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.464+0000 7f4b0f1b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5617dd0382f0 con 0x5617dd134d00 2024-04-21T08:55:20.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.464+0000 7f4b0f1b6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5617dd137a40 con 0x5617dd123370 2024-04-21T08:55:20.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.465+0000 7f4b0f1b6740 10 client.0 _ll_register_callbacks cb 0x5617dd045580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:20.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.465+0000 7f4b0a28d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd122a50 0x5617dd119ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.465+0000 7f4b0a28d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd122a50 0x5617dd119ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44632/0 (socket says 172.21.15.28:44632) 2024-04-21T08:55:20.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.465+0000 7f4b0a28d640 1 -- 172.21.15.28:0/745648335 learned_addr learned my addr 172.21.15.28:0/745648335 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:20.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.465+0000 7f4b09a8c640 1 --2- 172.21.15.28:0/745648335 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd123370 0x5617dd119ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.465+0000 7f4b0aa8e640 1 --2- 172.21.15.28:0/745648335 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd134d00 0x5617dd11a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.465+0000 7f4b0a28d640 1 -- 172.21.15.28:0/745648335 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd123370 msgr2=0x5617dd119ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:20.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.465+0000 7f4b0a28d640 1 --2- 172.21.15.28:0/745648335 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd123370 0x5617dd119ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.465+0000 7f4b0a28d640 1 -- 172.21.15.28:0/745648335 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd134d00 msgr2=0x5617dd11a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:20.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.465+0000 7f4b0a28d640 1 --2- 172.21.15.28:0/745648335 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd134d00 0x5617dd11a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.465+0000 7f4b0a28d640 1 -- 172.21.15.28:0/745648335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617dcd913f0 con 0x5617dd122a50 2024-04-21T08:55:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.466+0000 7f4b0a28d640 1 --2- 172.21.15.28:0/745648335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd122a50 0x5617dd119ab0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f4afc09ca40 tx=0x7f4afc0a4370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff3e8354c5858a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.467+0000 7f4ae9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:55:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.467+0000 7f4ae9ffb640 1 -- 172.21.15.28:0/745648335 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc0d1bf0 con 0x5617dd122a50 2024-04-21T08:55:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.467+0000 7f4ae9ffb640 1 -- 172.21.15.28:0/745648335 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f4afc0a87a0 con 0x5617dd122a50 2024-04-21T08:55:20.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.468+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/745648335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5617dd08cc00 con 0x5617dd122a50 2024-04-21T08:55:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.470+0000 7f4ae9ffb640 1 -- 172.21.15.28:0/745648335 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4afc0a89a0 con 0x5617dd122a50 2024-04-21T08:55:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.471+0000 7f4b0f1b6740 10 client.16977 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.471+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/745648335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5617dcd510f0 con 0x5617dd122a50 2024-04-21T08:55:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.472+0000 7f4ae9ffb640 1 -- 172.21.15.28:0/745648335 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5617dd08cc00 con 0x5617dd122a50 2024-04-21T08:55:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.472+0000 7f4ae9ffb640 1 -- 172.21.15.28:0/745648335 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f4afc0a2030 con 0x5617dd122a50 2024-04-21T08:55:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.472+0000 7f4b0f1b6740 10 client.16977 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.472+0000 7f4b0f1b6740 10 client.16977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.472+0000 7f4b0f1b6740 20 client.16977 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.472+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/745648335 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5617dd144ac0 0x5617dd146f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.472+0000 7f4b0f1b6740 4 client.16977 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.472+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/745648335 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617dd086a70 con 0x5617dd144ac0 2024-04-21T08:55:20.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.475+0000 7f4b09a8c640 1 --2- 172.21.15.28:0/745648335 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5617dd144ac0 0x5617dd146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.478+0000 7f4b09a8c640 1 --2- 172.21.15.28:0/745648335 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5617dd144ac0 0x5617dd146f20 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba9171aa5b37e759 server_cookie=48e0dc39a1b26679 in_seq=0 out_seq=0 2024-04-21T08:55:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.478+0000 7f4ae9ffb640 10 client.16977 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.479+0000 7f4ae9ffb640 1 -- 172.21.15.28:0/745648335 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5617dd086a70 con 0x5617dd144ac0 2024-04-21T08:55:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.479+0000 7f4ae9ffb640 10 client.16977 handle_command_reply: tid=0 2024-04-21T08:55:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.479+0000 7f4b0f1b6740 1 client.16977 shutdown 2024-04-21T08:55:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.480+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/745648335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd122a50 msgr2=0x5617dd119ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.480+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/745648335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd122a50 0x5617dd119ab0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f4afc09ca40 tx=0x7f4afc0a4370 comp rx=0 tx=0).stop 2024-04-21T08:55:20.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.482+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/745648335 shutdown_connections 2024-04-21T08:55:20.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.482+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/745648335 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5617dd144ac0 0x5617dd146f20 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.482+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/745648335 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd134d00 0x5617dd11a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.482+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/745648335 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd123370 0x5617dd119ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.482+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/745648335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd122a50 0x5617dd119ab0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.482+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/745648335 >> 172.21.15.28:0/745648335 conn(0x5617dd11d4c0 msgr2=0x5617dd11f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.483+0000 7f4ae9ffb640 0 client.16977 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.483+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/745648335 shutdown_connections 2024-04-21T08:55:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.483+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/745648335 wait complete. 2024-04-21T08:55:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.484+0000 7f4b0f1b6740 20 client.16977 trim_cache size 0 max 16384 2024-04-21T08:55:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.521+0000 7f4b0f1b6740 1 Processor -- start 2024-04-21T08:55:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.521+0000 7f4b0f1b6740 1 -- start start 2024-04-21T08:55:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.521+0000 7f4b0f1b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd122a50 0x5617dd140870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.521+0000 7f4b0f1b6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd123370 0x5617dd113b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.522+0000 7f4b0f1b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 0x5617dd118c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.522+0000 7f4b0f1b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4af0012870 con 0x5617dd134d00 2024-04-21T08:55:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.522+0000 7f4b0f1b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4af0010140 con 0x5617dd122a50 2024-04-21T08:55:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.522+0000 7f4b0f1b6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4af000df90 con 0x5617dd123370 2024-04-21T08:55:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.522+0000 7f4b0aa8e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 0x5617dd118c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.522+0000 7f4b0aa8e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 0x5617dd118c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44636/0 (socket says 172.21.15.28:44636) 2024-04-21T08:55:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.522+0000 7f4b0aa8e640 1 -- 172.21.15.28:0/1885991316 learned_addr learned my addr 172.21.15.28:0/1885991316 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.522+0000 7f4b0aa8e640 1 -- 172.21.15.28:0/1885991316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd123370 msgr2=0x5617dd113b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.522+0000 7f4b0aa8e640 1 --2- 172.21.15.28:0/1885991316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd123370 0x5617dd113b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.522+0000 7f4b0aa8e640 1 -- 172.21.15.28:0/1885991316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd122a50 msgr2=0x5617dd140870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.522+0000 7f4b0aa8e640 1 --2- 172.21.15.28:0/1885991316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd122a50 0x5617dd140870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.522+0000 7f4b0aa8e640 1 -- 172.21.15.28:0/1885991316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617dd119150 con 0x5617dd134d00 2024-04-21T08:55:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.523+0000 7f4b0aa8e640 1 --2- 172.21.15.28:0/1885991316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 0x5617dd118c10 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f4af8028040 tx=0x7f4af80254f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0072eaf8efb1593 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.523+0000 7f4aebfff640 1 -- 172.21.15.28:0/1885991316 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af802a040 con 0x5617dd134d00 2024-04-21T08:55:20.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.523+0000 7f4aebfff640 1 -- 172.21.15.28:0/1885991316 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4af8035070 con 0x5617dd134d00 2024-04-21T08:55:20.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.524+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/1885991316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 msgr2=0x5617dd118c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:20.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.524+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/1885991316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 0x5617dd118c10 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f4af8028040 tx=0x7f4af80254f0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.525+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/1885991316 shutdown_connections 2024-04-21T08:55:20.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.525+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/1885991316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 0x5617dd118c10 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.525+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/1885991316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd123370 0x5617dd113b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.525+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/1885991316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd122a50 0x5617dd140870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.525+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/1885991316 >> 172.21.15.28:0/1885991316 conn(0x5617dd11d4c0 msgr2=0x5617dd11e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.525+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/1885991316 shutdown_connections 2024-04-21T08:55:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.526+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/1885991316 wait complete. 2024-04-21T08:55:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.526+0000 7f4b0f1b6740 1 -- start start 2024-04-21T08:55:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.526+0000 7f4b0f1b6740 1 Processor -- start 2024-04-21T08:55:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.527+0000 7f4b0f1b6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd122a50 0x5617dd13caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.527+0000 7f4b0f1b6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd123370 0x5617dd13cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.527+0000 7f4b0f1b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 0x5617dd138f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.527+0000 7f4b0f1b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4af000e600 con 0x5617dd134d00 2024-04-21T08:55:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.527+0000 7f4b0f1b6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4af000fa70 con 0x5617dd123370 2024-04-21T08:55:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.527+0000 7f4b0f1b6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4af000ecd0 con 0x5617dd122a50 2024-04-21T08:55:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.527+0000 7f4b0f1b6740 10 client.0 _ll_register_callbacks cb 0x7f4afc00be60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.527+0000 7f4b09a8c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd123370 0x5617dd13cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.527+0000 7f4b09a8c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd123370 0x5617dd13cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38304/0 (socket says 172.21.15.28:38304) 2024-04-21T08:55:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.527+0000 7f4b09a8c640 1 -- 172.21.15.28:0/869172993 learned_addr learned my addr 172.21.15.28:0/869172993 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.527+0000 7f4b0a28d640 1 --2- 172.21.15.28:0/869172993 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd122a50 0x5617dd13caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.528+0000 7f4b09a8c640 1 -- 172.21.15.28:0/869172993 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd122a50 msgr2=0x5617dd13caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.528+0000 7f4b09a8c640 1 --2- 172.21.15.28:0/869172993 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd122a50 0x5617dd13caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.528+0000 7f4b09a8c640 1 -- 172.21.15.28:0/869172993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 msgr2=0x5617dd138f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.528+0000 7f4b09a8c640 1 --2- 172.21.15.28:0/869172993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 0x5617dd138f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.528+0000 7f4b09a8c640 1 -- 172.21.15.28:0/869172993 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617dd139700 con 0x5617dd123370 2024-04-21T08:55:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.528+0000 7f4b09a8c640 1 --2- 172.21.15.28:0/869172993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd123370 0x5617dd13cfe0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4af402f010 tx=0x7f4af402dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c80dfcc47c66fbff server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.530+0000 7f4ae9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:55:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.530+0000 7f4ae9ffb640 1 -- 172.21.15.28:0/869172993 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af402ddb0 con 0x5617dd123370 2024-04-21T08:55:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.530+0000 7f4ae9ffb640 1 -- 172.21.15.28:0/869172993 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f4af4048bc0 con 0x5617dd123370 2024-04-21T08:55:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.531+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/869172993 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4af8035070 con 0x5617dd123370 2024-04-21T08:55:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.533+0000 7f4ae9ffb640 1 -- 172.21.15.28:0/869172993 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4af4048dc0 con 0x5617dd123370 2024-04-21T08:55:20.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.533+0000 7f4b0f1b6740 10 client.26240 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:20.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.533+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/869172993 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5617dd1abb90 con 0x5617dd123370 2024-04-21T08:55:20.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.534+0000 7f4ae9ffb640 1 -- 172.21.15.28:0/869172993 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4af8035070 con 0x5617dd123370 2024-04-21T08:55:20.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.534+0000 7f4ae9ffb640 1 -- 172.21.15.28:0/869172993 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f4af400b040 con 0x5617dd123370 2024-04-21T08:55:20.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.535+0000 7f4b0f1b6740 10 client.26240 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:20.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.535+0000 7f4b0f1b6740 10 client.26240 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.535+0000 7f4b0f1b6740 20 client.26240 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.535+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/869172993 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5617dd1ac310 0x5617dd1ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.535+0000 7f4b0f1b6740 4 client.26240 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:55:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.535+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/869172993 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5617dd002590 con 0x5617dd1ac310 2024-04-21T08:55:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.538+0000 7f4b0a28d640 1 --2- 172.21.15.28:0/869172993 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5617dd1ac310 0x5617dd1ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.539+0000 7f4b0a28d640 1 --2- 172.21.15.28:0/869172993 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5617dd1ac310 0x5617dd1ae730 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddf0203d07ebfcc6 server_cookie=d02c617a62b942ef in_seq=0 out_seq=0 2024-04-21T08:55:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.539+0000 7f4ae9ffb640 10 client.26240 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.541+0000 7f4ae9ffb640 1 -- 172.21.15.28:0/869172993 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5617dd002590 con 0x5617dd1ac310 2024-04-21T08:55:20.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.541+0000 7f4ae9ffb640 10 client.26240 handle_command_reply: tid=0 2024-04-21T08:55:20.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.542+0000 7f4b0f1b6740 1 client.26240 shutdown 2024-04-21T08:55:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.544+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/869172993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd123370 msgr2=0x5617dd13cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.544+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/869172993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd123370 0x5617dd13cfe0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4af402f010 tx=0x7f4af402dd40 comp rx=0 tx=0).stop 2024-04-21T08:55:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.544+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/869172993 shutdown_connections 2024-04-21T08:55:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.544+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/869172993 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5617dd1ac310 0x5617dd1ae730 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.544+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/869172993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617dd134d00 0x5617dd138f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.544+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/869172993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5617dd123370 0x5617dd13cfe0 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4af402f010 tx=0x7f4af402dd40 comp rx=0 tx=0).stop 2024-04-21T08:55:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.544+0000 7f4b0f1b6740 1 --2- 172.21.15.28:0/869172993 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5617dd122a50 0x5617dd13caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.544+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/869172993 >> 172.21.15.28:0/869172993 conn(0x5617dd11d4c0 msgr2=0x5617dd11f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.545+0000 7f4ae9ffb640 0 client.26240 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:20.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.548+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/869172993 shutdown_connections 2024-04-21T08:55:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.550+0000 7f4b0f1b6740 1 -- 172.21.15.28:0/869172993 wait complete. 2024-04-21T08:55:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.550+0000 7f4b0f1b6740 20 client.26240 trim_cache size 0 max 16384 2024-04-21T08:55:20.552 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:55:20.553 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T08:55:20.553 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T08:55:20.553 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:55:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.553+0000 7f4b0c4f1640 1 -- 172.21.15.28:0/575290426 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4aec0a3d50 msgr2=0x7f4aec0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.553+0000 7f4b0c4f1640 1 --2- 172.21.15.28:0/575290426 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4aec0a3d50 0x7f4aec0a6210 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4af800e800 tx=0x7f4af8009810 comp rx=0 tx=0).stop 2024-04-21T08:55:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.553+0000 7f4b0c4f1640 1 -- 172.21.15.28:0/575290426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100610 msgr2=0x7f4b04073710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.553+0000 7f4b0c4f1640 1 --2- 172.21.15.28:0/575290426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100610 0x7f4b04073710 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f4afc010e10 tx=0x7f4afc00b690 comp rx=0 tx=0).stop 2024-04-21T08:55:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.555+0000 7f4b0c4f1640 1 -- 172.21.15.28:0/575290426 shutdown_connections 2024-04-21T08:55:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.555+0000 7f4b0c4f1640 1 --2- 172.21.15.28:0/575290426 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4aec0a3d50 0x7f4aec0a6210 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.555+0000 7f4b0c4f1640 1 --2- 172.21.15.28:0/575290426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b04106680 0x7f4b0406ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.555+0000 7f4b0c4f1640 1 --2- 172.21.15.28:0/575290426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b04100fd0 0x7f4b0406c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.555+0000 7f4b0c4f1640 1 --2- 172.21.15.28:0/575290426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100610 0x7f4b04073710 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.555+0000 7f4b0c4f1640 1 -- 172.21.15.28:0/575290426 >> 172.21.15.28:0/575290426 conn(0x7f4b04077360 msgr2=0x7f4b040feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.558+0000 7f4b0c4f1640 1 -- 172.21.15.28:0/575290426 shutdown_connections 2024-04-21T08:55:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.558+0000 7f4b0c4f1640 1 -- 172.21.15.28:0/575290426 wait complete. 2024-04-21T08:55:20.579 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:55:20.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:55:20.580 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:20 smithi028 ceph-mon[28003]: pgmap v127: 129 pgs: 129 active+clean; 15 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 68 MiB/s wr, 9.66k op/s 2024-04-21T08:55:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.734+0000 7fb8b763e640 1 -- 172.21.15.28:0/1954361136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b0103af0 msgr2=0x7fb8b010bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.734+0000 7fb8b763e640 1 --2- 172.21.15.28:0/1954361136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b0103af0 0x7fb8b010bfd0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0009900 tx=0x7fb8a002bcd0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.734+0000 7fb8b763e640 1 -- 172.21.15.28:0/1954361136 shutdown_connections 2024-04-21T08:55:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.734+0000 7fb8b763e640 1 --2- 172.21.15.28:0/1954361136 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8b010c5a0 0x7fb8b010e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.734+0000 7fb8b763e640 1 --2- 172.21.15.28:0/1954361136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b0103af0 0x7fb8b010bfd0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.734+0000 7fb8b763e640 1 --2- 172.21.15.28:0/1954361136 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8b01031d0 0x7fb8b01035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.734+0000 7fb8b763e640 1 -- 172.21.15.28:0/1954361136 >> 172.21.15.28:0/1954361136 conn(0x7fb8b00771d0 msgr2=0x7fb8b00775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.734+0000 7fb8b763e640 1 -- 172.21.15.28:0/1954361136 shutdown_connections 2024-04-21T08:55:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.734+0000 7fb8b763e640 1 -- 172.21.15.28:0/1954361136 wait complete. 2024-04-21T08:55:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.735+0000 7fb8b763e640 1 Processor -- start 2024-04-21T08:55:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.735+0000 7fb8b763e640 1 -- start start 2024-04-21T08:55:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.735+0000 7fb8b763e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b01031d0 0x7fb8b019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.735+0000 7fb8b763e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8b0103af0 0x7fb8b019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.735+0000 7fb8b763e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8b010c5a0 0x7fb8b01985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.735+0000 7fb8b763e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b01121c0 con 0x7fb8b01031d0 2024-04-21T08:55:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.735+0000 7fb8b763e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b0112040 con 0x7fb8b0103af0 2024-04-21T08:55:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.735+0000 7fb8b763e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b0112340 con 0x7fb8b010c5a0 2024-04-21T08:55:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.735+0000 7fb8b53da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b01031d0 0x7fb8b019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.735+0000 7fb8b53da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b01031d0 0x7fb8b019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44674/0 (socket says 172.21.15.28:44674) 2024-04-21T08:55:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.735+0000 7fb8b53da640 1 -- 172.21.15.28:0/3261278883 learned_addr learned my addr 172.21.15.28:0/3261278883 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.735+0000 7fb8b4bd9640 1 --2- 172.21.15.28:0/3261278883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8b0103af0 0x7fb8b019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.735+0000 7fb8b5bdb640 1 --2- 172.21.15.28:0/3261278883 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8b010c5a0 0x7fb8b01985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.736+0000 7fb8b53da640 1 -- 172.21.15.28:0/3261278883 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8b010c5a0 msgr2=0x7fb8b01985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.736+0000 7fb8b53da640 1 --2- 172.21.15.28:0/3261278883 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8b010c5a0 0x7fb8b01985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.736+0000 7fb8b53da640 1 -- 172.21.15.28:0/3261278883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8b0103af0 msgr2=0x7fb8b019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.736+0000 7fb8b53da640 1 --2- 172.21.15.28:0/3261278883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8b0103af0 0x7fb8b019e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.736+0000 7fb8b53da640 1 -- 172.21.15.28:0/3261278883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b0198d20 con 0x7fb8b01031d0 2024-04-21T08:55:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.736+0000 7fb8b5bdb640 1 --2- 172.21.15.28:0/3261278883 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8b010c5a0 0x7fb8b01985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:55:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.736+0000 7fb8b53da640 1 --2- 172.21.15.28:0/3261278883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b01031d0 0x7fb8b019e3f0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00d950 tx=0x7fb8ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.736+0000 7fb8a67fc640 1 -- 172.21.15.28:0/3261278883 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac017070 con 0x7fb8b01031d0 2024-04-21T08:55:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.736+0000 7fb8a67fc640 1 -- 172.21.15.28:0/3261278883 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8ac009c30 con 0x7fb8b01031d0 2024-04-21T08:55:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.736+0000 7fb8a67fc640 1 -- 172.21.15.28:0/3261278883 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac00f620 con 0x7fb8b01031d0 2024-04-21T08:55:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.736+0000 7fb8b763e640 1 -- 172.21.15.28:0/3261278883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b0199010 con 0x7fb8b01031d0 2024-04-21T08:55:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.736+0000 7fb8b763e640 1 -- 172.21.15.28:0/3261278883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b01a52b0 con 0x7fb8b01031d0 2024-04-21T08:55:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.737+0000 7fb8b763e640 1 -- 172.21.15.28:0/3261278883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b0069230 con 0x7fb8b01031d0 2024-04-21T08:55:20.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.739+0000 7fb8a67fc640 1 -- 172.21.15.28:0/3261278883 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb8ac010460 con 0x7fb8b01031d0 2024-04-21T08:55:20.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.740+0000 7fb8a67fc640 1 --2- 172.21.15.28:0/3261278883 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb89c0a3da0 0x7fb89c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:20.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.740+0000 7fb8b4bd9640 1 --2- 172.21.15.28:0/3261278883 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb89c0a3da0 0x7fb89c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:20.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.740+0000 7fb8a67fc640 1 -- 172.21.15.28:0/3261278883 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fb8ac0d07c0 con 0x7fb8b01031d0 2024-04-21T08:55:20.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.740+0000 7fb8b4bd9640 1 --2- 172.21.15.28:0/3261278883 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb89c0a3da0 0x7fb89c0a6260 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0006010 tx=0x7fb8a0005e40 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:20.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:20.745+0000 7fb8a67fc640 1 -- 172.21.15.28:0/3261278883 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb8ac09ca60 con 0x7fb8b01031d0 2024-04-21T08:55:20.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:20 smithi152 ceph-mon[31401]: pgmap v127: 129 pgs: 129 active+clean; 15 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 68 MiB/s wr, 9.66k op/s 2024-04-21T08:55:20.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:20 smithi171 ceph-mon[31358]: pgmap v127: 129 pgs: 129 active+clean; 15 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 68 MiB/s wr, 9.66k op/s 2024-04-21T08:55:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.176+0000 7fb8b763e640 1 -- 172.21.15.28:0/3261278883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8b0199ad0 con 0x7fb8b01031d0 2024-04-21T08:55:21.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.178+0000 7fb8a67fc640 1 -- 172.21.15.28:0/3261278883 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7fb8ac009de0 con 0x7fb8b01031d0 2024-04-21T08:55:21.178 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:55:21.179 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:55:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T08:55:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.182+0000 7fb8b763e640 1 -- 172.21.15.28:0/3261278883 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb89c0a3da0 msgr2=0x7fb89c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.182+0000 7fb8b763e640 1 --2- 172.21.15.28:0/3261278883 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb89c0a3da0 0x7fb89c0a6260 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0006010 tx=0x7fb8a0005e40 comp rx=0 tx=0).stop 2024-04-21T08:55:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.182+0000 7fb8b763e640 1 -- 172.21.15.28:0/3261278883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b01031d0 msgr2=0x7fb8b019e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.182+0000 7fb8b763e640 1 --2- 172.21.15.28:0/3261278883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b01031d0 0x7fb8b019e3f0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00d950 tx=0x7fb8ac00de10 comp rx=0 tx=0).stop 2024-04-21T08:55:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.183+0000 7fb8b763e640 1 -- 172.21.15.28:0/3261278883 shutdown_connections 2024-04-21T08:55:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.183+0000 7fb8b763e640 1 --2- 172.21.15.28:0/3261278883 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb89c0a3da0 0x7fb89c0a6260 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0006010 tx=0x7fb8a0005e40 comp rx=0 tx=0).stop 2024-04-21T08:55:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.183+0000 7fb8b763e640 1 --2- 172.21.15.28:0/3261278883 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8b010c5a0 0x7fb8b01985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.183+0000 7fb8b763e640 1 --2- 172.21.15.28:0/3261278883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb8b0103af0 0x7fb8b019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.183+0000 7fb8b763e640 1 --2- 172.21.15.28:0/3261278883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8b01031d0 0x7fb8b019e3f0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.183+0000 7fb8b763e640 1 -- 172.21.15.28:0/3261278883 >> 172.21.15.28:0/3261278883 conn(0x7fb8b00771d0 msgr2=0x7fb8b00fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.184+0000 7fb8b763e640 1 -- 172.21.15.28:0/3261278883 shutdown_connections 2024-04-21T08:55:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.185+0000 7fb8b763e640 1 -- 172.21.15.28:0/3261278883 wait complete. 2024-04-21T08:55:21.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:55:21.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.402+0000 7fee1ceab640 1 -- 172.21.15.28:0/2515229734 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1002cb60 con 0x7fee181099d0 2024-04-21T08:55:21.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.403+0000 7fee1eeaf640 1 -- 172.21.15.28:0/2515229734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fee181099d0 msgr2=0x7fee1811d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.403+0000 7fee1eeaf640 1 --2- 172.21.15.28:0/2515229734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fee181099d0 0x7fee1811d740 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fee10009be0 tx=0x7fee1002bfc0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.403+0000 7fee1eeaf640 1 -- 172.21.15.28:0/2515229734 shutdown_connections 2024-04-21T08:55:21.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.403+0000 7fee1eeaf640 1 --2- 172.21.15.28:0/2515229734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fee181099d0 0x7fee1811d740 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.403+0000 7fee1eeaf640 1 --2- 172.21.15.28:0/2515229734 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee18108f50 0x7fee181093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.403+0000 7fee1eeaf640 1 --2- 172.21.15.28:0/2515229734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee181105f0 0x7fee181109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.403+0000 7fee1eeaf640 1 -- 172.21.15.28:0/2515229734 >> 172.21.15.28:0/2515229734 conn(0x7fee1806ced0 msgr2=0x7fee1806d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:21.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.403+0000 7fee1eeaf640 1 -- 172.21.15.28:0/2515229734 shutdown_connections 2024-04-21T08:55:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.403+0000 7fee1eeaf640 1 -- 172.21.15.28:0/2515229734 wait complete. 2024-04-21T08:55:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.404+0000 7fee1eeaf640 1 Processor -- start 2024-04-21T08:55:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.404+0000 7fee1eeaf640 1 -- start start 2024-04-21T08:55:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.405+0000 7fee1eeaf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee18108f50 0x7fee1811d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.405+0000 7fee1eeaf640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fee181099d0 0x7fee181164a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.405+0000 7fee1eeaf640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee181105f0 0x7fee181169e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.405+0000 7fee1eeaf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee18120ee0 con 0x7fee18108f50 2024-04-21T08:55:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.405+0000 7fee1eeaf640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fee18120d60 con 0x7fee181099d0 2024-04-21T08:55:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.405+0000 7fee1eeaf640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fee18121060 con 0x7fee181105f0 2024-04-21T08:55:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.405+0000 7fee1d6ac640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fee181099d0 0x7fee181164a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.405+0000 7fee1d6ac640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fee181099d0 0x7fee181164a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38348/0 (socket says 172.21.15.28:38348) 2024-04-21T08:55:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.405+0000 7fee1d6ac640 1 -- 172.21.15.28:0/2238509202 learned_addr learned my addr 172.21.15.28:0/2238509202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.405+0000 7fee1e6ae640 1 --2- 172.21.15.28:0/2238509202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee181105f0 0x7fee181169e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.405+0000 7fee1e6ae640 1 -- 172.21.15.28:0/2238509202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fee181099d0 msgr2=0x7fee181164a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.405+0000 7fee1e6ae640 1 --2- 172.21.15.28:0/2238509202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fee181099d0 0x7fee181164a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.405+0000 7fee1e6ae640 1 -- 172.21.15.28:0/2238509202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee18108f50 msgr2=0x7fee1811d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.405+0000 7fee1e6ae640 1 --2- 172.21.15.28:0/2238509202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee18108f50 0x7fee1811d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.405+0000 7fee1e6ae640 1 -- 172.21.15.28:0/2238509202 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee18116ff0 con 0x7fee181105f0 2024-04-21T08:55:21.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.406+0000 7fee1e6ae640 1 --2- 172.21.15.28:0/2238509202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee181105f0 0x7fee181169e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fee10009810 tx=0x7fee10033c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:21.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.409+0000 7fee0effd640 1 -- 172.21.15.28:0/2238509202 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee10043020 con 0x7fee181105f0 2024-04-21T08:55:21.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.410+0000 7fee0effd640 1 -- 172.21.15.28:0/2238509202 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee10033e40 con 0x7fee181105f0 2024-04-21T08:55:21.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.410+0000 7fee0effd640 1 -- 172.21.15.28:0/2238509202 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1002cb60 con 0x7fee181105f0 2024-04-21T08:55:21.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.410+0000 7fee1eeaf640 1 -- 172.21.15.28:0/2238509202 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee18117280 con 0x7fee181105f0 2024-04-21T08:55:21.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.410+0000 7fee1eeaf640 1 -- 172.21.15.28:0/2238509202 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee1811b140 con 0x7fee181105f0 2024-04-21T08:55:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.416+0000 7fee22dd6740 1 Processor -- start 2024-04-21T08:55:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.417+0000 7fee0effd640 1 -- 172.21.15.28:0/2238509202 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fee10046020 con 0x7fee181105f0 2024-04-21T08:55:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.417+0000 7fee22dd6740 1 -- start start 2024-04-21T08:55:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.417+0000 7fee22dd6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc00d6d0 0x55cfbc00dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.417+0000 7fee22dd6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00e080 0x55cfbc016630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.417+0000 7fee22dd6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc016c00 0x55cfbc019050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.417+0000 7fee22dd6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cfbbf721f0 con 0x55cfbc00e080 2024-04-21T08:55:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.417+0000 7fee22dd6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55cfbbf0f4b0 con 0x55cfbc016c00 2024-04-21T08:55:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.417+0000 7fee22dd6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55cfbbf6e1b0 con 0x55cfbc00d6d0 2024-04-21T08:55:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.417+0000 7fee1d6ac640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc00d6d0 0x55cfbc00dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.417+0000 7fee1d6ac640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc00d6d0 0x55cfbc00dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:53222/0 (socket says 172.21.15.28:53222) 2024-04-21T08:55:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.417+0000 7fee1d6ac640 1 -- 172.21.15.28:0/3723489589 learned_addr learned my addr 172.21.15.28:0/3723489589 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.417+0000 7fee1d6ac640 1 --2- 172.21.15.28:0/3723489589 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc016c00 0x55cfbc019050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.418+0000 7fee1d6ac640 1 -- 172.21.15.28:0/3723489589 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc016c00 msgr2=0x55cfbc019050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.418+0000 7fee1d6ac640 1 --2- 172.21.15.28:0/3723489589 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc016c00 0x55cfbc019050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.418+0000 7fee1d6ac640 1 -- 172.21.15.28:0/3723489589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00e080 msgr2=0x55cfbc016630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.418+0000 7fee1d6ac640 1 --2- 172.21.15.28:0/3723489589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00e080 0x55cfbc016630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.418+0000 7fee1d6ac640 1 -- 172.21.15.28:0/3723489589 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfbbf011b0 con 0x55cfbc00d6d0 2024-04-21T08:55:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.418+0000 7fee1d6ac640 1 --2- 172.21.15.28:0/3723489589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc00d6d0 0x55cfbc00dab0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fee14014a20 tx=0x7fee1400b520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88ffb500738bebb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.418+0000 7fee0effd640 1 --2- 172.21.15.28:0/2238509202 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fee040a3ef0 0x7fee040a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.419+0000 7fee03fff640 1 -- 172.21.15.28:0/3723489589 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee140142d0 con 0x55cfbc00d6d0 2024-04-21T08:55:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.419+0000 7fee03fff640 1 -- 172.21.15.28:0/3723489589 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee14005020 con 0x55cfbc00d6d0 2024-04-21T08:55:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.419+0000 7fee0effd640 1 -- 172.21.15.28:0/2238509202 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fee100f30c0 con 0x7fee181105f0 2024-04-21T08:55:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.420+0000 7fee22dd6740 1 -- 172.21.15.28:0/3723489589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc00d6d0 msgr2=0x55cfbc00dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.420+0000 7fee22dd6740 1 --2- 172.21.15.28:0/3723489589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc00d6d0 0x55cfbc00dab0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fee14014a20 tx=0x7fee1400b520 comp rx=0 tx=0).stop 2024-04-21T08:55:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.422+0000 7fee1dead640 1 --2- 172.21.15.28:0/2238509202 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fee040a3ef0 0x7fee040a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.429+0000 7fee22dd6740 1 -- 172.21.15.28:0/3723489589 shutdown_connections 2024-04-21T08:55:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.429+0000 7fee22dd6740 1 --2- 172.21.15.28:0/3723489589 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc016c00 0x55cfbc019050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.429+0000 7fee22dd6740 1 --2- 172.21.15.28:0/3723489589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00e080 0x55cfbc016630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.429+0000 7fee22dd6740 1 --2- 172.21.15.28:0/3723489589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc00d6d0 0x55cfbc00dab0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.429+0000 7fee22dd6740 1 -- 172.21.15.28:0/3723489589 >> 172.21.15.28:0/3723489589 conn(0x55cfbc007da0 msgr2=0x55cfbc008180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:21.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.433+0000 7fee1dead640 1 --2- 172.21.15.28:0/2238509202 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fee040a3ef0 0x7fee040a63b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fee080058c0 tx=0x7fee08009660 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:21.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.435+0000 7fee22dd6740 1 -- 172.21.15.28:0/3723489589 shutdown_connections 2024-04-21T08:55:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.437+0000 7fee22dd6740 1 -- 172.21.15.28:0/3723489589 wait complete. 2024-04-21T08:55:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.438+0000 7fee22dd6740 1 -- start start 2024-04-21T08:55:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.438+0000 7fee22dd6740 1 Processor -- start 2024-04-21T08:55:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.439+0000 7fee22dd6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00d6d0 0x55cfbc0163d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.439+0000 7fee22dd6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc00e080 0x55cfbc013760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.439+0000 7fee22dd6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc016c00 0x55cfbc013ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.439+0000 7fee22dd6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cfbc019b60 con 0x55cfbc00d6d0 2024-04-21T08:55:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.439+0000 7fee22dd6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55cfbbf2b500 con 0x55cfbc00e080 2024-04-21T08:55:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.439+0000 7fee22dd6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55cfbc019ce0 con 0x55cfbc016c00 2024-04-21T08:55:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.439+0000 7fee22dd6740 10 client.0 _ll_register_callbacks cb 0x55cfbbef9b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.440+0000 7fee1dead640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00d6d0 0x55cfbc0163d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.440+0000 7fee1dead640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00d6d0 0x55cfbc0163d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44702/0 (socket says 172.21.15.28:44702) 2024-04-21T08:55:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.440+0000 7fee1dead640 1 -- 172.21.15.28:0/637715884 learned_addr learned my addr 172.21.15.28:0/637715884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.440+0000 7fee1e6ae640 1 --2- 172.21.15.28:0/637715884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc016c00 0x55cfbc013ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.440+0000 7fee1d6ac640 1 --2- 172.21.15.28:0/637715884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc00e080 0x55cfbc013760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.441+0000 7fee1dead640 1 -- 172.21.15.28:0/637715884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc016c00 msgr2=0x55cfbc013ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.441+0000 7fee1dead640 1 --2- 172.21.15.28:0/637715884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc016c00 0x55cfbc013ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.441+0000 7fee1dead640 1 -- 172.21.15.28:0/637715884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc00e080 msgr2=0x55cfbc013760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.441+0000 7fee1dead640 1 --2- 172.21.15.28:0/637715884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc00e080 0x55cfbc013760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.441+0000 7fee1dead640 1 -- 172.21.15.28:0/637715884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfbbc843f0 con 0x55cfbc00d6d0 2024-04-21T08:55:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.445+0000 7fee1dead640 1 --2- 172.21.15.28:0/637715884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00d6d0 0x55cfbc0163d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fee08009fd0 tx=0x7fee08009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6981abf1dad12e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.447+0000 7fee017fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:55:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.448+0000 7fee017fa640 1 -- 172.21.15.28:0/637715884 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee08020070 con 0x55cfbc00d6d0 2024-04-21T08:55:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.448+0000 7fee22dd6740 1 -- 172.21.15.28:0/637715884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee14005020 con 0x55cfbc00d6d0 2024-04-21T08:55:21.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.448+0000 7fee017fa640 1 -- 172.21.15.28:0/637715884 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fee0801c8f0 con 0x55cfbc00d6d0 2024-04-21T08:55:21.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.457+0000 7fee017fa640 1 -- 172.21.15.28:0/637715884 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fee0801caf0 con 0x55cfbc00d6d0 2024-04-21T08:55:21.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.461+0000 7fee22dd6740 10 client.16992 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:21.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.461+0000 7fee22dd6740 1 -- 172.21.15.28:0/637715884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cfbbd4e7d0 con 0x55cfbc00d6d0 2024-04-21T08:55:21.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.462+0000 7fee017fa640 1 -- 172.21.15.28:0/637715884 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee14005020 con 0x55cfbc00d6d0 2024-04-21T08:55:21.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.462+0000 7fee017fa640 1 -- 172.21.15.28:0/637715884 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fee0802f420 con 0x55cfbc00d6d0 2024-04-21T08:55:21.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.462+0000 7fee22dd6740 10 client.16992 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:21.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.462+0000 7fee22dd6740 10 client.16992 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:55:21.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.462+0000 7fee22dd6740 20 client.16992 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:21.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.462+0000 7fee22dd6740 1 --2- 172.21.15.28:0/637715884 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55cfbc027240 0x55cfbc029680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.462+0000 7fee22dd6740 4 client.16992 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:21.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.462+0000 7fee22dd6740 1 -- 172.21.15.28:0/637715884 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfbbf11930 con 0x55cfbc027240 2024-04-21T08:55:21.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.463+0000 7fee1d6ac640 1 --2- 172.21.15.28:0/637715884 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55cfbc027240 0x55cfbc029680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.468+0000 7fee1d6ac640 1 --2- 172.21.15.28:0/637715884 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55cfbc027240 0x55cfbc029680 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=940218da5b556206 server_cookie=6d6ddfd95b50b9dd in_seq=0 out_seq=0 2024-04-21T08:55:21.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.470+0000 7fee017fa640 10 client.16992 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:55:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.471+0000 7fee017fa640 1 -- 172.21.15.28:0/637715884 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cfbbf11930 con 0x55cfbc027240 2024-04-21T08:55:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.471+0000 7fee017fa640 10 client.16992 handle_command_reply: tid=0 2024-04-21T08:55:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.471+0000 7fee22dd6740 1 client.16992 shutdown 2024-04-21T08:55:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.473+0000 7fee22dd6740 1 -- 172.21.15.28:0/637715884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00d6d0 msgr2=0x55cfbc0163d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.473+0000 7fee22dd6740 1 --2- 172.21.15.28:0/637715884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00d6d0 0x55cfbc0163d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fee08009fd0 tx=0x7fee08009260 comp rx=0 tx=0).stop 2024-04-21T08:55:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.473+0000 7fee22dd6740 1 -- 172.21.15.28:0/637715884 shutdown_connections 2024-04-21T08:55:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.474+0000 7fee22dd6740 1 --2- 172.21.15.28:0/637715884 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55cfbc027240 0x55cfbc029680 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.474+0000 7fee22dd6740 1 --2- 172.21.15.28:0/637715884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc016c00 0x55cfbc013ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.474+0000 7fee22dd6740 1 --2- 172.21.15.28:0/637715884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc00e080 0x55cfbc013760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.474+0000 7fee22dd6740 1 --2- 172.21.15.28:0/637715884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00d6d0 0x55cfbc0163d0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.474+0000 7fee22dd6740 1 -- 172.21.15.28:0/637715884 >> 172.21.15.28:0/637715884 conn(0x55cfbc007da0 msgr2=0x55cfbc009af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.474+0000 7fee017fa640 0 client.16992 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:55:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.474+0000 7fee22dd6740 1 -- 172.21.15.28:0/637715884 shutdown_connections 2024-04-21T08:55:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.474+0000 7fee22dd6740 1 -- 172.21.15.28:0/637715884 wait complete. 2024-04-21T08:55:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.478+0000 7fee22dd6740 20 client.16992 trim_cache size 0 max 16384 2024-04-21T08:55:21.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.509+0000 7fee22dd6740 1 Processor -- start 2024-04-21T08:55:21.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.509+0000 7fee22dd6740 1 -- start start 2024-04-21T08:55:21.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.510+0000 7fee22dd6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00d6d0 0x55cfbc022b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.510+0000 7fee22dd6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc00e080 0x55cfbc015e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.510+0000 7fee22dd6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc016c00 0x55cfbc0163d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.510+0000 7fee22dd6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc012910 con 0x55cfbc00d6d0 2024-04-21T08:55:21.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.510+0000 7fee22dd6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc0101e0 con 0x55cfbc00e080 2024-04-21T08:55:21.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.510+0000 7fee22dd6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc00e030 con 0x55cfbc016c00 2024-04-21T08:55:21.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.510+0000 7fee1dead640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00d6d0 0x55cfbc022b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.510+0000 7fee1dead640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00d6d0 0x55cfbc022b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44708/0 (socket says 172.21.15.28:44708) 2024-04-21T08:55:21.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.510+0000 7fee1dead640 1 -- 172.21.15.28:0/3965468754 learned_addr learned my addr 172.21.15.28:0/3965468754 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:21.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.511+0000 7fee1d6ac640 1 --2- 172.21.15.28:0/3965468754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc00e080 0x55cfbc015e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.511+0000 7fee1dead640 1 -- 172.21.15.28:0/3965468754 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc016c00 msgr2=0x55cfbc0163d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:21.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.511+0000 7fee1dead640 1 --2- 172.21.15.28:0/3965468754 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc016c00 0x55cfbc0163d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.511+0000 7fee1dead640 1 -- 172.21.15.28:0/3965468754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc00e080 msgr2=0x55cfbc015e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:21.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.511+0000 7fee1dead640 1 --2- 172.21.15.28:0/3965468754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc00e080 0x55cfbc015e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.511+0000 7fee1dead640 1 -- 172.21.15.28:0/3965468754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfbc003480 con 0x55cfbc00d6d0 2024-04-21T08:55:21.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.511+0000 7fee1dead640 1 --2- 172.21.15.28:0/3965468754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00d6d0 0x55cfbc022b00 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fee08000c00 tx=0x7fee08004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce451b64fd01fe44 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:21.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.515+0000 7fee037fe640 1 -- 172.21.15.28:0/3965468754 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee08020040 con 0x55cfbc00d6d0 2024-04-21T08:55:21.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.515+0000 7fee037fe640 1 -- 172.21.15.28:0/3965468754 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee0802f620 con 0x55cfbc00d6d0 2024-04-21T08:55:21.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.516+0000 7fee22dd6740 1 -- 172.21.15.28:0/3965468754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00d6d0 msgr2=0x55cfbc022b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.516+0000 7fee22dd6740 1 --2- 172.21.15.28:0/3965468754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00d6d0 0x55cfbc022b00 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fee08000c00 tx=0x7fee08004680 comp rx=0 tx=0).stop 2024-04-21T08:55:21.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.517+0000 7fee22dd6740 1 -- 172.21.15.28:0/3965468754 shutdown_connections 2024-04-21T08:55:21.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.517+0000 7fee22dd6740 1 --2- 172.21.15.28:0/3965468754 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc016c00 0x55cfbc0163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.517+0000 7fee22dd6740 1 --2- 172.21.15.28:0/3965468754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc00e080 0x55cfbc015e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.517+0000 7fee22dd6740 1 --2- 172.21.15.28:0/3965468754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00d6d0 0x55cfbc022b00 secure :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fee08000c00 tx=0x7fee08004680 comp rx=0 tx=0).stop 2024-04-21T08:55:21.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.517+0000 7fee22dd6740 1 -- 172.21.15.28:0/3965468754 >> 172.21.15.28:0/3965468754 conn(0x55cfbc007da0 msgr2=0x55cfbc006e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:21.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.518+0000 7fee22dd6740 1 -- 172.21.15.28:0/3965468754 shutdown_connections 2024-04-21T08:55:21.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.520+0000 7fee22dd6740 1 -- 172.21.15.28:0/3965468754 wait complete. 2024-04-21T08:55:21.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.520+0000 7fee22dd6740 1 -- start start 2024-04-21T08:55:21.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.520+0000 7fee22dd6740 1 Processor -- start 2024-04-21T08:55:21.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.521+0000 7fee22dd6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00e080 0x55cfbc01f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.521+0000 7fee22dd6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc016c00 0x55cfbc01a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.521+0000 7fee22dd6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc01ab10 0x55cfbc01af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.521+0000 7fee22dd6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc00e6a0 con 0x55cfbc00e080 2024-04-21T08:55:21.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.521+0000 7fee22dd6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc00fb10 con 0x55cfbc016c00 2024-04-21T08:55:21.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.521+0000 7fee22dd6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fedfc00ed70 con 0x55cfbc01ab10 2024-04-21T08:55:21.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.521+0000 7fee22dd6740 10 client.0 _ll_register_callbacks cb 0x7fee0800ee10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.522+0000 7fee1e6ae640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc01ab10 0x55cfbc01af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.522+0000 7fee1e6ae640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc01ab10 0x55cfbc01af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:53244/0 (socket says 172.21.15.28:53244) 2024-04-21T08:55:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.522+0000 7fee1e6ae640 1 -- 172.21.15.28:0/542061927 learned_addr learned my addr 172.21.15.28:0/542061927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.522+0000 7fee1dead640 1 --2- 172.21.15.28:0/542061927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00e080 0x55cfbc01f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.522+0000 7fee1dead640 1 -- 172.21.15.28:0/542061927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc01ab10 msgr2=0x55cfbc01af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.522+0000 7fee1dead640 1 --2- 172.21.15.28:0/542061927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc01ab10 0x55cfbc01af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.522+0000 7fee1dead640 1 -- 172.21.15.28:0/542061927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc016c00 msgr2=0x55cfbc01a5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.522+0000 7fee1dead640 1 --2- 172.21.15.28:0/542061927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc016c00 0x55cfbc01a5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.522+0000 7fee1dead640 1 -- 172.21.15.28:0/542061927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfbc08c800 con 0x55cfbc00e080 2024-04-21T08:55:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.523+0000 7fee1dead640 1 --2- 172.21.15.28:0/542061927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00e080 0x55cfbc01f100 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x55cfbc023670 tx=0x7fee08005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c461734f22301bc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.523+0000 7fee017fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:55:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.524+0000 7fee017fa640 1 -- 172.21.15.28:0/542061927 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee08047020 con 0x55cfbc00e080 2024-04-21T08:55:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.524+0000 7fee22dd6740 1 -- 172.21.15.28:0/542061927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee0802f620 con 0x55cfbc00e080 2024-04-21T08:55:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.524+0000 7fee017fa640 1 -- 172.21.15.28:0/542061927 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fee0802fa10 con 0x55cfbc00e080 2024-04-21T08:55:21.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.525+0000 7fee017fa640 1 -- 172.21.15.28:0/542061927 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fee08020040 con 0x55cfbc00e080 2024-04-21T08:55:21.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.525+0000 7fee22dd6740 10 client.16998 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:21.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.525+0000 7fee22dd6740 1 -- 172.21.15.28:0/542061927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cfbc08d970 con 0x55cfbc00e080 2024-04-21T08:55:21.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.526+0000 7fee017fa640 1 -- 172.21.15.28:0/542061927 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee0802f620 con 0x55cfbc00e080 2024-04-21T08:55:21.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.526+0000 7fee017fa640 1 -- 172.21.15.28:0/542061927 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fee08042070 con 0x55cfbc00e080 2024-04-21T08:55:21.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.526+0000 7fee22dd6740 10 client.16998 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:21.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.526+0000 7fee22dd6740 10 client.16998 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:55:21.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.526+0000 7fee22dd6740 20 client.16998 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:21.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.526+0000 7fee22dd6740 1 --2- 172.21.15.28:0/542061927 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55cfbc08de90 0x55cfbc090280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.526+0000 7fee22dd6740 4 client.16998 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:55:21.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.526+0000 7fee22dd6740 1 -- 172.21.15.28:0/542061927 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cfbbf7fc10 con 0x55cfbc08de90 2024-04-21T08:55:21.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.528+0000 7fee1d6ac640 1 --2- 172.21.15.28:0/542061927 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55cfbc08de90 0x55cfbc090280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.530+0000 7fee1d6ac640 1 --2- 172.21.15.28:0/542061927 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55cfbc08de90 0x55cfbc090280 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=150431b1b8a4caaa server_cookie=6d7e0039a08b35ad in_seq=0 out_seq=0 2024-04-21T08:55:21.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.530+0000 7fee017fa640 10 client.16998 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:55:21.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.531+0000 7fee017fa640 1 -- 172.21.15.28:0/542061927 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cfbbf7fc10 con 0x55cfbc08de90 2024-04-21T08:55:21.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.531+0000 7fee017fa640 10 client.16998 handle_command_reply: tid=0 2024-04-21T08:55:21.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.532+0000 7fee22dd6740 1 client.16998 shutdown 2024-04-21T08:55:21.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.532+0000 7fee22dd6740 1 -- 172.21.15.28:0/542061927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00e080 msgr2=0x55cfbc01f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.532+0000 7fee22dd6740 1 --2- 172.21.15.28:0/542061927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00e080 0x55cfbc01f100 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x55cfbc023670 tx=0x7fee08005850 comp rx=0 tx=0).stop 2024-04-21T08:55:21.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.538+0000 7fee22dd6740 1 -- 172.21.15.28:0/542061927 shutdown_connections 2024-04-21T08:55:21.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.538+0000 7fee22dd6740 1 --2- 172.21.15.28:0/542061927 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55cfbc08de90 0x55cfbc090280 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.538+0000 7fee22dd6740 1 --2- 172.21.15.28:0/542061927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55cfbc01ab10 0x55cfbc01af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.538+0000 7fee22dd6740 1 --2- 172.21.15.28:0/542061927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cfbc016c00 0x55cfbc01a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.538+0000 7fee22dd6740 1 --2- 172.21.15.28:0/542061927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cfbc00e080 0x55cfbc01f100 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.538+0000 7fee22dd6740 1 -- 172.21.15.28:0/542061927 >> 172.21.15.28:0/542061927 conn(0x55cfbc007da0 msgr2=0x55cfbc0090a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:21.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.538+0000 7fee017fa640 0 client.16998 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:55:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.542+0000 7fee22dd6740 1 -- 172.21.15.28:0/542061927 shutdown_connections 2024-04-21T08:55:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.544+0000 7fee22dd6740 1 -- 172.21.15.28:0/542061927 wait complete. 2024-04-21T08:55:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.544+0000 7fee22dd6740 20 client.16998 trim_cache size 0 max 16384 2024-04-21T08:55:21.545 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:55:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.548+0000 7fee1eeaf640 1 -- 172.21.15.28:0/2238509202 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fee040a3ef0 msgr2=0x7fee040a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.548+0000 7fee1eeaf640 1 --2- 172.21.15.28:0/2238509202 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fee040a3ef0 0x7fee040a63b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fee080058c0 tx=0x7fee08009660 comp rx=0 tx=0).stop 2024-04-21T08:55:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.549+0000 7fee1eeaf640 1 -- 172.21.15.28:0/2238509202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee181105f0 msgr2=0x7fee181169e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.549+0000 7fee1eeaf640 1 --2- 172.21.15.28:0/2238509202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee181105f0 0x7fee181169e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fee10009810 tx=0x7fee10033c50 comp rx=0 tx=0).stop 2024-04-21T08:55:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.550+0000 7fee1eeaf640 1 -- 172.21.15.28:0/2238509202 shutdown_connections 2024-04-21T08:55:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.550+0000 7fee1eeaf640 1 --2- 172.21.15.28:0/2238509202 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fee040a3ef0 0x7fee040a63b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.550+0000 7fee1eeaf640 1 --2- 172.21.15.28:0/2238509202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fee181105f0 0x7fee181169e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.550+0000 7fee1eeaf640 1 --2- 172.21.15.28:0/2238509202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fee181099d0 0x7fee181164a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.550+0000 7fee1eeaf640 1 --2- 172.21.15.28:0/2238509202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee18108f50 0x7fee1811d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.550+0000 7fee1eeaf640 1 -- 172.21.15.28:0/2238509202 >> 172.21.15.28:0/2238509202 conn(0x7fee1806ced0 msgr2=0x7fee18070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:21.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.554+0000 7fee1eeaf640 1 -- 172.21.15.28:0/2238509202 shutdown_connections 2024-04-21T08:55:21.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.555+0000 7fee1eeaf640 1 -- 172.21.15.28:0/2238509202 wait complete. 2024-04-21T08:55:21.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:55:21.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:21 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3261278883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:55:21.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.762+0000 7fe0be3b1640 1 -- 172.21.15.28:0/3626877107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0b80742f0 msgr2=0x7fe0b80746d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.762+0000 7fe0be3b1640 1 --2- 172.21.15.28:0/3626877107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0b80742f0 0x7fe0b80746d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8009900 tx=0x7fe0a802bcd0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.763+0000 7fe0be3b1640 1 -- 172.21.15.28:0/3626877107 shutdown_connections 2024-04-21T08:55:21.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.763+0000 7fe0be3b1640 1 --2- 172.21.15.28:0/3626877107 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0b8112f20 0x7fe0b8115310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.763+0000 7fe0be3b1640 1 --2- 172.21.15.28:0/3626877107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b8074c10 0x7fe0b81129e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.763+0000 7fe0be3b1640 1 --2- 172.21.15.28:0/3626877107 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0b80742f0 0x7fe0b80746d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.763+0000 7fe0be3b1640 1 -- 172.21.15.28:0/3626877107 >> 172.21.15.28:0/3626877107 conn(0x7fe0b80fd9c0 msgr2=0x7fe0b80ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:21.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.764+0000 7fe0be3b1640 1 -- 172.21.15.28:0/3626877107 shutdown_connections 2024-04-21T08:55:21.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.764+0000 7fe0be3b1640 1 -- 172.21.15.28:0/3626877107 wait complete. 2024-04-21T08:55:21.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.765+0000 7fe0be3b1640 1 Processor -- start 2024-04-21T08:55:21.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.765+0000 7fe0be3b1640 1 -- start start 2024-04-21T08:55:21.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.766+0000 7fe0be3b1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0b80742f0 0x7fe0b81a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.766+0000 7fe0be3b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b8074c10 0x7fe0b81a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.766+0000 7fe0be3b1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0b8112f20 0x7fe0b819ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.766+0000 7fe0be3b1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b8118870 con 0x7fe0b8074c10 2024-04-21T08:55:21.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.766+0000 7fe0be3b1640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81186f0 con 0x7fe0b8112f20 2024-04-21T08:55:21.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.766+0000 7fe0be3b1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81189f0 con 0x7fe0b80742f0 2024-04-21T08:55:21.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.767+0000 7fe0b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b8074c10 0x7fe0b81a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.768+0000 7fe0b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b8074c10 0x7fe0b81a4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44748/0 (socket says 172.21.15.28:44748) 2024-04-21T08:55:21.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.768+0000 7fe0b77fe640 1 -- 172.21.15.28:0/1042204658 learned_addr learned my addr 172.21.15.28:0/1042204658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:21.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.769+0000 7fe0bc94e640 1 --2- 172.21.15.28:0/1042204658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0b8112f20 0x7fe0b819ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.769+0000 7fe0b7fff640 1 --2- 172.21.15.28:0/1042204658 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0b80742f0 0x7fe0b81a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.770+0000 7fe0b77fe640 1 -- 172.21.15.28:0/1042204658 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0b80742f0 msgr2=0x7fe0b81a4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.770+0000 7fe0b77fe640 1 --2- 172.21.15.28:0/1042204658 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0b80742f0 0x7fe0b81a4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.770+0000 7fe0b77fe640 1 -- 172.21.15.28:0/1042204658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0b8112f20 msgr2=0x7fe0b819ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.770+0000 7fe0b77fe640 1 --2- 172.21.15.28:0/1042204658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0b8112f20 0x7fe0b819ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.770+0000 7fe0b77fe640 1 -- 172.21.15.28:0/1042204658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b819f210 con 0x7fe0b8074c10 2024-04-21T08:55:21.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.770+0000 7fe0b77fe640 1 --2- 172.21.15.28:0/1042204658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b8074c10 0x7fe0b81a4dd0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00b4a0 tx=0x7fe0ac00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:21.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.772+0000 7fe0bc94e640 1 --2- 172.21.15.28:0/1042204658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0b8112f20 0x7fe0b819ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:55:21.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.772+0000 7fe0b57fa640 1 -- 172.21.15.28:0/1042204658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac014070 con 0x7fe0b8074c10 2024-04-21T08:55:21.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.772+0000 7fe0b57fa640 1 -- 172.21.15.28:0/1042204658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0ac010ac0 con 0x7fe0b8074c10 2024-04-21T08:55:21.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.772+0000 7fe0b57fa640 1 -- 172.21.15.28:0/1042204658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac014070 con 0x7fe0b8074c10 2024-04-21T08:55:21.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.772+0000 7fe0be3b1640 1 -- 172.21.15.28:0/1042204658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0b819f500 con 0x7fe0b8074c10 2024-04-21T08:55:21.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.772+0000 7fe0be3b1640 1 -- 172.21.15.28:0/1042204658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0b81ab750 con 0x7fe0b8074c10 2024-04-21T08:55:21.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.777+0000 7fe0b57fa640 1 -- 172.21.15.28:0/1042204658 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe0ac017070 con 0x7fe0b8074c10 2024-04-21T08:55:21.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.778+0000 7fe0b57fa640 1 --2- 172.21.15.28:0/1042204658 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0a00a4120 0x7fe0a00a65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.779+0000 7fe0c0f9b740 1 Processor -- start 2024-04-21T08:55:21.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.779+0000 7fe0c0f9b740 1 -- start start 2024-04-21T08:55:21.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.779+0000 7fe0c0f9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed04d0 0x560f5aed08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.779+0000 7fe0c0f9b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 0x560f5aed9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.779+0000 7fe0c0f9b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aedbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.779+0000 7fe0c0f9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560f5ae351f0 con 0x560f5aed04d0 2024-04-21T08:55:21.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.779+0000 7fe0c0f9b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x560f5add24b0 con 0x560f5aed0e80 2024-04-21T08:55:21.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.779+0000 7fe0c0f9b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x560f5ae311b0 con 0x560f5aed9a00 2024-04-21T08:55:21.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.780+0000 7fe0b57fa640 1 -- 172.21.15.28:0/1042204658 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe0ac00d040 con 0x7fe0b8074c10 2024-04-21T08:55:21.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.780+0000 7fe0b7fff640 1 --2- 172.21.15.28:0/1042204658 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0a00a4120 0x7fe0a00a65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.780+0000 7fe0b77fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 0x560f5aed9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.780+0000 7fe0bc94e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aedbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.780+0000 7fe0bc94e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aedbe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:53270/0 (socket says 172.21.15.28:53270) 2024-04-21T08:55:21.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.780+0000 7fe0bc94e640 1 -- 172.21.15.28:0/2457356918 learned_addr learned my addr 172.21.15.28:0/2457356918 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:21.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.780+0000 7fe0b7fff640 1 --2- 172.21.15.28:0/2457356918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed04d0 0x560f5aed08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.781+0000 7fe0b77fe640 1 -- 172.21.15.28:0/2457356918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 msgr2=0x560f5aedbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:21.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.781+0000 7fe0b77fe640 1 --2- 172.21.15.28:0/2457356918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aedbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.781+0000 7fe0b77fe640 1 -- 172.21.15.28:0/2457356918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed04d0 msgr2=0x560f5aed08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.781+0000 7fe0b77fe640 1 --2- 172.21.15.28:0/2457356918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed04d0 0x560f5aed08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.781+0000 7fe0b77fe640 1 -- 172.21.15.28:0/2457356918 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f5adc41b0 con 0x560f5aed0e80 2024-04-21T08:55:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.781+0000 7fe0b77fe640 1 --2- 172.21.15.28:0/2457356918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 0x560f5aed9430 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac0cdcf0 tx=0x7fe0ac0a1a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95a6a24087344d31 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.781+0000 7fe09e7fc640 1 -- 172.21.15.28:0/2457356918 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac098f70 con 0x560f5aed0e80 2024-04-21T08:55:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.782+0000 7fe0b7fff640 1 --2- 172.21.15.28:0/1042204658 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0a00a4120 0x7fe0a00a65e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe0a80057c0 tx=0x7fe0a80042e0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.782+0000 7fe09e7fc640 1 -- 172.21.15.28:0/2457356918 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0ac00ee50 con 0x560f5aed0e80 2024-04-21T08:55:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.783+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/2457356918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 msgr2=0x560f5aed9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.783+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/2457356918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 0x560f5aed9430 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac0cdcf0 tx=0x7fe0ac0a1a10 comp rx=0 tx=0).stop 2024-04-21T08:55:21.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.784+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/2457356918 shutdown_connections 2024-04-21T08:55:21.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.784+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/2457356918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aedbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.784+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/2457356918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 0x560f5aed9430 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.784+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/2457356918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed04d0 0x560f5aed08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.784+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/2457356918 >> 172.21.15.28:0/2457356918 conn(0x560f5aecaba0 msgr2=0x560f5aecaf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:21.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.784+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/2457356918 shutdown_connections 2024-04-21T08:55:21.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.785+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/2457356918 wait complete. 2024-04-21T08:55:21.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.785+0000 7fe0c0f9b740 1 -- start start 2024-04-21T08:55:21.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.786+0000 7fe0c0f9b740 1 Processor -- start 2024-04-21T08:55:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.786+0000 7fe0c0f9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed04d0 0x560f5aed91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.786+0000 7fe0c0f9b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 0x560f5aed6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.786+0000 7fe0c0f9b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aed6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.787+0000 7fe0c0f9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560f5aeed8e0 con 0x560f5aed04d0 2024-04-21T08:55:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.787+0000 7fe0c0f9b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x560f5adee500 con 0x560f5aed0e80 2024-04-21T08:55:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.787+0000 7fe0c0f9b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x560f5aeeda60 con 0x560f5aed9a00 2024-04-21T08:55:21.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.787+0000 7fe0bc94e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aed6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.787+0000 7fe0bc94e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aed6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:53274/0 (socket says 172.21.15.28:53274) 2024-04-21T08:55:21.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.787+0000 7fe0bc94e640 1 -- 172.21.15.28:0/3235759396 learned_addr learned my addr 172.21.15.28:0/3235759396 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:21.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.787+0000 7fe0c0f9b740 10 client.0 _ll_register_callbacks cb 0x560f5adbcb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:21.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.788+0000 7fe0b77fe640 1 --2- 172.21.15.28:0/3235759396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 0x560f5aed6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.790+0000 7fe0bc94e640 1 -- 172.21.15.28:0/3235759396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 msgr2=0x560f5aed6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:21.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.790+0000 7fe0bc94e640 1 --2- 172.21.15.28:0/3235759396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 0x560f5aed6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.790+0000 7fe0bc94e640 1 -- 172.21.15.28:0/3235759396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed04d0 msgr2=0x560f5aed91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:21.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.790+0000 7fe0bc94e640 1 --2- 172.21.15.28:0/3235759396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed04d0 0x560f5aed91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.790+0000 7fe0bc94e640 1 -- 172.21.15.28:0/3235759396 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f5ab473f0 con 0x560f5aed9a00 2024-04-21T08:55:21.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.790+0000 7fe0b77fe640 1 --2- 172.21.15.28:0/3235759396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 0x560f5aed6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:55:21.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.790+0000 7fe0bc94e640 1 --2- 172.21.15.28:0/3235759396 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aed6a70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0010a50 tx=0x7fe0b0010f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a38063bff62aca67 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:21.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.790+0000 7fe0977fe640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:55:21.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.792+0000 7fe0977fe640 1 -- 172.21.15.28:0/3235759396 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b001d060 con 0x560f5aed9a00 2024-04-21T08:55:21.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.793+0000 7fe0977fe640 1 -- 172.21.15.28:0/3235759396 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe0b000ed00 con 0x560f5aed9a00 2024-04-21T08:55:21.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.793+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/3235759396 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0ac00ee50 con 0x560f5aed9a00 2024-04-21T08:55:21.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.797+0000 7fe0977fe640 1 -- 172.21.15.28:0/3235759396 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe0b0018410 con 0x560f5aed9a00 2024-04-21T08:55:21.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.797+0000 7fe0c0f9b740 10 client.26281 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:21.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.797+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/3235759396 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560f5ac117d0 con 0x560f5aed9a00 2024-04-21T08:55:21.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.798+0000 7fe0977fe640 1 -- 172.21.15.28:0/3235759396 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0ac00ee50 con 0x560f5aed9a00 2024-04-21T08:55:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.798+0000 7fe0977fe640 1 -- 172.21.15.28:0/3235759396 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe0b0021670 con 0x560f5aed9a00 2024-04-21T08:55:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.798+0000 7fe0c0f9b740 10 client.26281 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.798+0000 7fe0c0f9b740 10 client.26281 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:55:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.798+0000 7fe0c0f9b740 20 client.26281 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.799+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/3235759396 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x560f5aefaa50 0x560f5aefced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.799+0000 7fe0c0f9b740 4 client.26281 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.799+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/3235759396 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f5add4930 con 0x560f5aefaa50 2024-04-21T08:55:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.799+0000 7fe0b7fff640 1 --2- 172.21.15.28:0/3235759396 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x560f5aefaa50 0x560f5aefced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.800+0000 7fe0b7fff640 1 --2- 172.21.15.28:0/3235759396 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x560f5aefaa50 0x560f5aefced0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7b6424fb69e12a03 server_cookie=598d21f29b2c8bb3 in_seq=0 out_seq=0 2024-04-21T08:55:21.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.800+0000 7fe0977fe640 10 client.26281 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:55:21.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.803+0000 7fe0977fe640 1 -- 172.21.15.28:0/3235759396 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f5add4930 con 0x560f5aefaa50 2024-04-21T08:55:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.803+0000 7fe0977fe640 10 client.26281 handle_command_reply: tid=0 2024-04-21T08:55:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.803+0000 7fe0c0f9b740 1 client.26281 shutdown 2024-04-21T08:55:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.807+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/3235759396 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 msgr2=0x560f5aed6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.807+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/3235759396 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aed6a70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0010a50 tx=0x7fe0b0010f10 comp rx=0 tx=0).stop 2024-04-21T08:55:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.807+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/3235759396 shutdown_connections 2024-04-21T08:55:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.807+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/3235759396 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x560f5aefaa50 0x560f5aefced0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.807+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/3235759396 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aed6a70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.807+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/3235759396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 0x560f5aed6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.807+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/3235759396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed04d0 0x560f5aed91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.807+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/3235759396 >> 172.21.15.28:0/3235759396 conn(0x560f5aecaba0 msgr2=0x560f5aecc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.807+0000 7fe0977fe640 0 client.26281 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:55:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.808+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/3235759396 shutdown_connections 2024-04-21T08:55:21.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.810+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/3235759396 wait complete. 2024-04-21T08:55:21.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.810+0000 7fe0c0f9b740 20 client.26281 trim_cache size 0 max 16384 2024-04-21T08:55:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.836+0000 7fe0c0f9b740 1 Processor -- start 2024-04-21T08:55:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.837+0000 7fe0c0f9b740 1 -- start start 2024-04-21T08:55:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.837+0000 7fe0c0f9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed04d0 0x560f5aef6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.837+0000 7fe0c0f9b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 0x560f5aed1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.837+0000 7fe0c0f9b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aed8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.837+0000 7fe0c0f9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a4012910 con 0x560f5aed04d0 2024-04-21T08:55:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.837+0000 7fe0c0f9b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a40101e0 con 0x560f5aed0e80 2024-04-21T08:55:21.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.837+0000 7fe0c0f9b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a400e030 con 0x560f5aed9a00 2024-04-21T08:55:21.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.837+0000 7fe0bc94e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aed8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.837+0000 7fe0bc94e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aed8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:53276/0 (socket says 172.21.15.28:53276) 2024-04-21T08:55:21.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.837+0000 7fe0bc94e640 1 -- 172.21.15.28:0/1344553001 learned_addr learned my addr 172.21.15.28:0/1344553001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:21.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.838+0000 7fe0b7fff640 1 --2- 172.21.15.28:0/1344553001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed04d0 0x560f5aef6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.838+0000 7fe0bc94e640 1 -- 172.21.15.28:0/1344553001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 msgr2=0x560f5aed1ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.838+0000 7fe0bc94e640 1 --2- 172.21.15.28:0/1344553001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 0x560f5aed1ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.838+0000 7fe0bc94e640 1 -- 172.21.15.28:0/1344553001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed04d0 msgr2=0x560f5aef6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.838+0000 7fe0bc94e640 1 --2- 172.21.15.28:0/1344553001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed04d0 0x560f5aef6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.838+0000 7fe0bc94e640 1 -- 172.21.15.28:0/1344553001 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f5aed97a0 con 0x560f5aed9a00 2024-04-21T08:55:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.839+0000 7fe0bc94e640 1 --2- 172.21.15.28:0/1344553001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aed8c60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe0b000b3c0 tx=0x7fe0b00021d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d8ff0b113c77c52 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.839+0000 7fe09dffb640 1 -- 172.21.15.28:0/1344553001 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b001d020 con 0x560f5aed9a00 2024-04-21T08:55:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.839+0000 7fe09dffb640 1 -- 172.21.15.28:0/1344553001 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0b0021850 con 0x560f5aed9a00 2024-04-21T08:55:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.839+0000 7fe09dffb640 1 -- 172.21.15.28:0/1344553001 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b0027080 con 0x560f5aed9a00 2024-04-21T08:55:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.840+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/1344553001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 msgr2=0x560f5aed8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.840+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/1344553001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aed8c60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe0b000b3c0 tx=0x7fe0b00021d0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.840+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/1344553001 shutdown_connections 2024-04-21T08:55:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.840+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/1344553001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed9a00 0x560f5aed8c60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.840+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/1344553001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed0e80 0x560f5aed1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.840+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/1344553001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed04d0 0x560f5aef6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.840+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/1344553001 >> 172.21.15.28:0/1344553001 conn(0x560f5aecaba0 msgr2=0x560f5aed13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.841+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/1344553001 shutdown_connections 2024-04-21T08:55:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.841+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/1344553001 wait complete. 2024-04-21T08:55:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.841+0000 7fe0c0f9b740 1 -- start start 2024-04-21T08:55:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.842+0000 7fe0c0f9b740 1 Processor -- start 2024-04-21T08:55:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.842+0000 7fe0c0f9b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed04d0 0x560f5aef31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.842+0000 7fe0c0f9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed0e80 0x560f5aed3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.842+0000 7fe0c0f9b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed9a00 0x560f5aed3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.842+0000 7fe0c0f9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a400e6a0 con 0x560f5aed0e80 2024-04-21T08:55:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.842+0000 7fe0c0f9b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a400fb10 con 0x560f5aed9a00 2024-04-21T08:55:21.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.842+0000 7fe0c0f9b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a400ed70 con 0x560f5aed04d0 2024-04-21T08:55:21.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.842+0000 7fe0c0f9b740 10 client.0 _ll_register_callbacks cb 0x7fe0b0010660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:21.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.843+0000 7fe0b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed0e80 0x560f5aed3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.843+0000 7fe0b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed0e80 0x560f5aed3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44774/0 (socket says 172.21.15.28:44774) 2024-04-21T08:55:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.844+0000 7fe0b77fe640 1 -- 172.21.15.28:0/2614014367 learned_addr learned my addr 172.21.15.28:0/2614014367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.844+0000 7fe0b7fff640 1 --2- 172.21.15.28:0/2614014367 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed04d0 0x560f5aef31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.844+0000 7fe0bc94e640 1 --2- 172.21.15.28:0/2614014367 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed9a00 0x560f5aed3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.846+0000 7fe0bc94e640 1 -- 172.21.15.28:0/2614014367 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed04d0 msgr2=0x560f5aef31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.846+0000 7fe0bc94e640 1 --2- 172.21.15.28:0/2614014367 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed04d0 0x560f5aef31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.846+0000 7fe0bc94e640 1 -- 172.21.15.28:0/2614014367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed0e80 msgr2=0x560f5aed3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.846+0000 7fe0bc94e640 1 --2- 172.21.15.28:0/2614014367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed0e80 0x560f5aed3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.846+0000 7fe0bc94e640 1 -- 172.21.15.28:0/2614014367 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f5aed3ea0 con 0x560f5aed9a00 2024-04-21T08:55:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.847+0000 7fe0bc94e640 1 --2- 172.21.15.28:0/2614014367 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed9a00 0x560f5aed3670 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x560f5aeedaf0 tx=0x7fe0b000b3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=749a1dcd91ce85c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.847+0000 7fe0977fe640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:55:21.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.850+0000 7fe0977fe640 1 -- 172.21.15.28:0/2614014367 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b001d070 con 0x560f5aed9a00 2024-04-21T08:55:21.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.850+0000 7fe0977fe640 1 -- 172.21.15.28:0/2614014367 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe0b00185b0 con 0x560f5aed9a00 2024-04-21T08:55:21.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.850+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/2614014367 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0b0021850 con 0x560f5aed9a00 2024-04-21T08:55:21.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.851+0000 7fe0977fe640 1 -- 172.21.15.28:0/2614014367 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe0b0027050 con 0x560f5aed9a00 2024-04-21T08:55:21.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.852+0000 7fe0c0f9b740 10 client.26270 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.852+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/2614014367 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560f5af720f0 con 0x560f5aed9a00 2024-04-21T08:55:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.853+0000 7fe0977fe640 1 -- 172.21.15.28:0/2614014367 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0b0021850 con 0x560f5aed9a00 2024-04-21T08:55:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.854+0000 7fe0977fe640 1 -- 172.21.15.28:0/2614014367 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe0b000b970 con 0x560f5aed9a00 2024-04-21T08:55:21.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:21 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3261278883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:55:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.855+0000 7fe0c0f9b740 10 client.26270 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.855+0000 7fe0c0f9b740 10 client.26270 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:55:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.855+0000 7fe0c0f9b740 20 client.26270 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.855+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/2614014367 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x560f5af726a0 0x560f5af74a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.855+0000 7fe0c0f9b740 4 client.26270 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:55:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.855+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/2614014367 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560f5aed6e00 con 0x560f5af726a0 2024-04-21T08:55:21.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.859+0000 7fe0b7fff640 1 --2- 172.21.15.28:0/2614014367 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x560f5af726a0 0x560f5af74a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.864+0000 7fe0977fe640 10 client.26270 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:55:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.864+0000 7fe0b7fff640 1 --2- 172.21.15.28:0/2614014367 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x560f5af726a0 0x560f5af74a90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8da77bdba6cd7ccb server_cookie=253fbed863165bf1 in_seq=0 out_seq=0 2024-04-21T08:55:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.864+0000 7fe0977fe640 1 -- 172.21.15.28:0/2614014367 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560f5aed6e00 con 0x560f5af726a0 2024-04-21T08:55:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.864+0000 7fe0977fe640 10 client.26270 handle_command_reply: tid=0 2024-04-21T08:55:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.865+0000 7fe0c0f9b740 1 client.26270 shutdown 2024-04-21T08:55:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.866+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/2614014367 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed9a00 msgr2=0x560f5aed3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.866+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/2614014367 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed9a00 0x560f5aed3670 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x560f5aeedaf0 tx=0x7fe0b000b3f0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.868+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/2614014367 shutdown_connections 2024-04-21T08:55:21.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.868+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/2614014367 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f5aed9a00 0x560f5aed3670 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.868+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/2614014367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560f5aed0e80 0x560f5aed3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.868+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/2614014367 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x560f5af726a0 0x560f5af74a90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.868+0000 7fe0c0f9b740 1 --2- 172.21.15.28:0/2614014367 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560f5aed04d0 0x560f5aef31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.868+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/2614014367 >> 172.21.15.28:0/2614014367 conn(0x560f5aecaba0 msgr2=0x560f5aec9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.868+0000 7fe0977fe640 0 client.26270 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:55:21.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:21 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3261278883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:55:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.871+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/2614014367 shutdown_connections 2024-04-21T08:55:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.871+0000 7fe0c0f9b740 1 -- 172.21.15.28:0/2614014367 wait complete. 2024-04-21T08:55:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.872+0000 7fe0c0f9b740 20 client.26270 trim_cache size 0 max 16384 2024-04-21T08:55:21.873 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:55:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.875+0000 7fe0be3b1640 1 -- 172.21.15.28:0/1042204658 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0a00a4120 msgr2=0x7fe0a00a65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.875+0000 7fe0be3b1640 1 --2- 172.21.15.28:0/1042204658 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0a00a4120 0x7fe0a00a65e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe0a80057c0 tx=0x7fe0a80042e0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.875+0000 7fe0be3b1640 1 -- 172.21.15.28:0/1042204658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b8074c10 msgr2=0x7fe0b81a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.875+0000 7fe0be3b1640 1 --2- 172.21.15.28:0/1042204658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b8074c10 0x7fe0b81a4dd0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00b4a0 tx=0x7fe0ac00b960 comp rx=0 tx=0).stop 2024-04-21T08:55:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.876+0000 7fe0be3b1640 1 -- 172.21.15.28:0/1042204658 shutdown_connections 2024-04-21T08:55:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.876+0000 7fe0be3b1640 1 --2- 172.21.15.28:0/1042204658 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0a00a4120 0x7fe0a00a65e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.876+0000 7fe0be3b1640 1 --2- 172.21.15.28:0/1042204658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0b8112f20 0x7fe0b819ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.876+0000 7fe0be3b1640 1 --2- 172.21.15.28:0/1042204658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b8074c10 0x7fe0b81a4dd0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.876+0000 7fe0be3b1640 1 --2- 172.21.15.28:0/1042204658 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0b80742f0 0x7fe0b81a4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.876+0000 7fe0be3b1640 1 -- 172.21.15.28:0/1042204658 >> 172.21.15.28:0/1042204658 conn(0x7fe0b80fd9c0 msgr2=0x7fe0b80fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.876+0000 7fe0be3b1640 1 -- 172.21.15.28:0/1042204658 shutdown_connections 2024-04-21T08:55:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:21.876+0000 7fe0be3b1640 1 -- 172.21.15.28:0/1042204658 wait complete. 2024-04-21T08:55:21.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:55:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.095+0000 7fbafaa31640 1 -- 172.21.15.28:0/717949392 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbaf41018e0 msgr2=0x7fbaf4105da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:22.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.095+0000 7fbafaa31640 1 --2- 172.21.15.28:0/717949392 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbaf41018e0 0x7fbaf4105da0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbae4009900 tx=0x7fbae402bd90 comp rx=0 tx=0).stop 2024-04-21T08:55:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.099+0000 7fbafaa31640 1 -- 172.21.15.28:0/717949392 shutdown_connections 2024-04-21T08:55:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.099+0000 7fbafaa31640 1 --2- 172.21.15.28:0/717949392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaf41062e0 0x7fbaf4106770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.099+0000 7fbafaa31640 1 --2- 172.21.15.28:0/717949392 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbaf41018e0 0x7fbaf4105da0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.099+0000 7fbafaa31640 1 --2- 172.21.15.28:0/717949392 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbaf4100fc0 0x7fbaf41013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.099+0000 7fbafaa31640 1 -- 172.21.15.28:0/717949392 >> 172.21.15.28:0/717949392 conn(0x7fbaf4077f50 msgr2=0x7fbaf4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.104+0000 7fbafaa31640 1 -- 172.21.15.28:0/717949392 shutdown_connections 2024-04-21T08:55:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.104+0000 7fbafaa31640 1 -- 172.21.15.28:0/717949392 wait complete. 2024-04-21T08:55:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.105+0000 7fbafaa31640 1 Processor -- start 2024-04-21T08:55:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.105+0000 7fbafaa31640 1 -- start start 2024-04-21T08:55:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.106+0000 7fbafaa31640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbaf4100fc0 0x7fbaf4073860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.106+0000 7fbafaa31640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbaf41018e0 0x7fbaf406d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.106+0000 7fbafaa31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaf41062e0 0x7fbaf406de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.106+0000 7fbaf3fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbaf4100fc0 0x7fbaf4073860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.106+0000 7fbaf3fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbaf4100fc0 0x7fbaf4073860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38432/0 (socket says 172.21.15.28:38432) 2024-04-21T08:55:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.106+0000 7fbaf3fff640 1 -- 172.21.15.28:0/3532222573 learned_addr learned my addr 172.21.15.28:0/3532222573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.106+0000 7fbafaa31640 1 -- 172.21.15.28:0/3532222573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf406c270 con 0x7fbaf41062e0 2024-04-21T08:55:22.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.106+0000 7fbafaa31640 1 -- 172.21.15.28:0/3532222573 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf406c0f0 con 0x7fbaf4100fc0 2024-04-21T08:55:22.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.106+0000 7fbafaa31640 1 -- 172.21.15.28:0/3532222573 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf406c3f0 con 0x7fbaf41018e0 2024-04-21T08:55:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.107+0000 7fbaf37fe640 1 --2- 172.21.15.28:0/3532222573 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbaf41018e0 0x7fbaf406d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.107+0000 7fbaf8fce640 1 --2- 172.21.15.28:0/3532222573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaf41062e0 0x7fbaf406de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.107+0000 7fbaf3fff640 1 -- 172.21.15.28:0/3532222573 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbaf41018e0 msgr2=0x7fbaf406d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.107+0000 7fbaf3fff640 1 --2- 172.21.15.28:0/3532222573 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbaf41018e0 0x7fbaf406d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.107+0000 7fbaf3fff640 1 -- 172.21.15.28:0/3532222573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaf41062e0 msgr2=0x7fbaf406de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.107+0000 7fbaf3fff640 1 --2- 172.21.15.28:0/3532222573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaf41062e0 0x7fbaf406de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.107+0000 7fbaf3fff640 1 -- 172.21.15.28:0/3532222573 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaf406e540 con 0x7fbaf4100fc0 2024-04-21T08:55:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.109+0000 7fbaf3fff640 1 --2- 172.21.15.28:0/3532222573 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbaf4100fc0 0x7fbaf4073860 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbae800e9f0 tx=0x7fbae800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.109+0000 7fbaf17fa640 1 -- 172.21.15.28:0/3532222573 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae800cd80 con 0x7fbaf4100fc0 2024-04-21T08:55:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.109+0000 7fbafaa31640 1 -- 172.21.15.28:0/3532222573 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaf411df80 con 0x7fbaf4100fc0 2024-04-21T08:55:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.110+0000 7fbafaa31640 1 -- 172.21.15.28:0/3532222573 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaf411e3e0 con 0x7fbaf4100fc0 2024-04-21T08:55:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.110+0000 7fbaf17fa640 1 -- 172.21.15.28:0/3532222573 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbae80108e0 con 0x7fbaf4100fc0 2024-04-21T08:55:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.110+0000 7fbaf17fa640 1 -- 172.21.15.28:0/3532222573 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae8018980 con 0x7fbaf4100fc0 2024-04-21T08:55:22.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.115+0000 7fbafd61d740 1 Processor -- start 2024-04-21T08:55:22.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.115+0000 7fbaf17fa640 1 -- 172.21.15.28:0/3532222573 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbae8018b20 con 0x7fbaf4100fc0 2024-04-21T08:55:22.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.115+0000 7fbafd61d740 1 -- start start 2024-04-21T08:55:22.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.115+0000 7fbafd61d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5623899754d0 0x5623899758b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.115+0000 7fbafd61d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562389975e80 0x56238997e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.115+0000 7fbafd61d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 0x562389980e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.115+0000 7fbafd61d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5623898da1f0 con 0x56238997ea00 2024-04-21T08:55:22.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.115+0000 7fbafd61d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5623898774b0 con 0x562389975e80 2024-04-21T08:55:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.115+0000 7fbafd61d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5623898d61b0 con 0x5623899754d0 2024-04-21T08:55:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.116+0000 7fbaf17fa640 1 --2- 172.21.15.28:0/3532222573 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbadc0a3f20 0x7fbadc0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.116+0000 7fbaf17fa640 1 -- 172.21.15.28:0/3532222573 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbae80d0e60 con 0x7fbaf4100fc0 2024-04-21T08:55:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.117+0000 7fbaf3fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562389975e80 0x56238997e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.118+0000 7fbaf8fce640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5623899754d0 0x5623899758b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.118+0000 7fbaf3fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562389975e80 0x56238997e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38446/0 (socket says 172.21.15.28:38446) 2024-04-21T08:55:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.118+0000 7fbaf3fff640 1 -- 172.21.15.28:0/516058903 learned_addr learned my addr 172.21.15.28:0/516058903 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.118+0000 7fbaf37fe640 1 --2- 172.21.15.28:0/516058903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 0x562389980e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.119+0000 7fbaf37fe640 1 -- 172.21.15.28:0/516058903 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5623899754d0 msgr2=0x5623899758b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.119+0000 7fbaf37fe640 1 --2- 172.21.15.28:0/516058903 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5623899754d0 0x5623899758b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.119+0000 7fbaf37fe640 1 -- 172.21.15.28:0/516058903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562389975e80 msgr2=0x56238997e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:22.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.119+0000 7fbaf37fe640 1 --2- 172.21.15.28:0/516058903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562389975e80 0x56238997e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.119+0000 7fbaf37fe640 1 -- 172.21.15.28:0/516058903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623898691b0 con 0x56238997ea00 2024-04-21T08:55:22.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.119+0000 7fbaf37fe640 1 --2- 172.21.15.28:0/516058903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 0x562389980e50 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fbae402c3a0 tx=0x7fbae4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cfa6e8966f5a5ce server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.120+0000 7fbaf8fce640 1 --2- 172.21.15.28:0/3532222573 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbadc0a3f20 0x7fbadc0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.121+0000 7fbada7fc640 1 -- 172.21.15.28:0/516058903 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae4042020 con 0x56238997ea00 2024-04-21T08:55:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.121+0000 7fbada7fc640 1 -- 172.21.15.28:0/516058903 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbae4004940 con 0x56238997ea00 2024-04-21T08:55:22.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.122+0000 7fbafd61d740 1 -- 172.21.15.28:0/516058903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 msgr2=0x562389980e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:22.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.122+0000 7fbafd61d740 1 --2- 172.21.15.28:0/516058903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 0x562389980e50 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fbae402c3a0 tx=0x7fbae4004750 comp rx=0 tx=0).stop 2024-04-21T08:55:22.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.122+0000 7fbaf8fce640 1 --2- 172.21.15.28:0/3532222573 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbadc0a3f20 0x7fbadc0a63e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbaec009fa0 tx=0x7fbaec004a40 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:22.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.124+0000 7fbafd61d740 1 -- 172.21.15.28:0/516058903 shutdown_connections 2024-04-21T08:55:22.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.124+0000 7fbafd61d740 1 --2- 172.21.15.28:0/516058903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 0x562389980e50 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.124+0000 7fbafd61d740 1 --2- 172.21.15.28:0/516058903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562389975e80 0x56238997e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.124+0000 7fbafd61d740 1 --2- 172.21.15.28:0/516058903 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5623899754d0 0x5623899758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.124+0000 7fbafd61d740 1 -- 172.21.15.28:0/516058903 >> 172.21.15.28:0/516058903 conn(0x56238996fba0 msgr2=0x56238996ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:22.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.124+0000 7fbafd61d740 1 -- 172.21.15.28:0/516058903 shutdown_connections 2024-04-21T08:55:22.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.124+0000 7fbafd61d740 1 -- 172.21.15.28:0/516058903 wait complete. 2024-04-21T08:55:22.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.124+0000 7fbafd61d740 1 -- start start 2024-04-21T08:55:22.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.125+0000 7fbafd61d740 1 Processor -- start 2024-04-21T08:55:22.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.125+0000 7fbafd61d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 0x56238997e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.125+0000 7fbafd61d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562389975e80 0x56238997b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.125+0000 7fbafd61d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56238997ea00 0x56238997b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.125+0000 7fbafd61d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562389981960 con 0x562389975e80 2024-04-21T08:55:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.125+0000 7fbafd61d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562389893500 con 0x5623899754d0 2024-04-21T08:55:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.125+0000 7fbafd61d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x562389981ae0 con 0x56238997ea00 2024-04-21T08:55:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.125+0000 7fbafd61d740 10 client.0 _ll_register_callbacks cb 0x562389861b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.126+0000 7fbaf8fce640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56238997ea00 0x56238997b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.126+0000 7fbaf8fce640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56238997ea00 0x56238997b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:53336/0 (socket says 172.21.15.28:53336) 2024-04-21T08:55:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.126+0000 7fbaf8fce640 1 -- 172.21.15.28:0/2971458402 learned_addr learned my addr 172.21.15.28:0/2971458402 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.126+0000 7fbaf3fff640 1 --2- 172.21.15.28:0/2971458402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 0x56238997e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.126+0000 7fbaf37fe640 1 --2- 172.21.15.28:0/2971458402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562389975e80 0x56238997b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.126+0000 7fbaf8fce640 1 -- 172.21.15.28:0/2971458402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 msgr2=0x56238997e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.126+0000 7fbaf8fce640 1 --2- 172.21.15.28:0/2971458402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 0x56238997e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.126+0000 7fbaf8fce640 1 -- 172.21.15.28:0/2971458402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562389975e80 msgr2=0x56238997b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.126+0000 7fbaf8fce640 1 --2- 172.21.15.28:0/2971458402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562389975e80 0x56238997b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.126+0000 7fbaf8fce640 1 -- 172.21.15.28:0/2971458402 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623895ec3f0 con 0x56238997ea00 2024-04-21T08:55:22.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.127+0000 7fbaf8fce640 1 --2- 172.21.15.28:0/2971458402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56238997ea00 0x56238997b9f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbaec01ae00 tx=0x7fbaec004b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6beec442e174bb53 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:22.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.127+0000 7fbad37fe640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:55:22.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.128+0000 7fbad37fe640 1 -- 172.21.15.28:0/2971458402 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec027060 con 0x56238997ea00 2024-04-21T08:55:22.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.128+0000 7fbafd61d740 1 -- 172.21.15.28:0/2971458402 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbae4004940 con 0x56238997ea00 2024-04-21T08:55:22.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.128+0000 7fbad37fe640 1 -- 172.21.15.28:0/2971458402 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbaec018460 con 0x56238997ea00 2024-04-21T08:55:22.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.128+0000 7fbad37fe640 1 -- 172.21.15.28:0/2971458402 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbaec02e050 con 0x56238997ea00 2024-04-21T08:55:22.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.129+0000 7fbafd61d740 10 client.26293 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:22.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.129+0000 7fbafd61d740 1 -- 172.21.15.28:0/2971458402 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5623896b67d0 con 0x56238997ea00 2024-04-21T08:55:22.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.130+0000 7fbad37fe640 1 -- 172.21.15.28:0/2971458402 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbae4004940 con 0x56238997ea00 2024-04-21T08:55:22.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.130+0000 7fbad37fe640 1 -- 172.21.15.28:0/2971458402 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fbaec016040 con 0x56238997ea00 2024-04-21T08:55:22.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.135+0000 7fbafd61d740 10 client.26293 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:22.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.135+0000 7fbafd61d740 10 client.26293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:22.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.135+0000 7fbafd61d740 20 client.26293 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:22.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.135+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2971458402 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56238999fb10 0x5623899a1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.135+0000 7fbafd61d740 4 client.26293 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:22.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.135+0000 7fbafd61d740 1 -- 172.21.15.28:0/2971458402 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562389879930 con 0x56238999fb10 2024-04-21T08:55:22.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.137+0000 7fbaf3fff640 1 --2- 172.21.15.28:0/2971458402 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56238999fb10 0x5623899a1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.138+0000 7fbaf3fff640 1 --2- 172.21.15.28:0/2971458402 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56238999fb10 0x5623899a1f70 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e797511c604285c8 server_cookie=a690d0bf40b0cc41 in_seq=0 out_seq=0 2024-04-21T08:55:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.138+0000 7fbad37fe640 10 client.26293 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.139+0000 7fbad37fe640 1 -- 172.21.15.28:0/2971458402 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562389879930 con 0x56238999fb10 2024-04-21T08:55:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.139+0000 7fbad37fe640 10 client.26293 handle_command_reply: tid=0 2024-04-21T08:55:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.139+0000 7fbafd61d740 1 client.26293 shutdown 2024-04-21T08:55:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.140+0000 7fbafd61d740 1 -- 172.21.15.28:0/2971458402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56238997ea00 msgr2=0x56238997b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.140+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2971458402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56238997ea00 0x56238997b9f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbaec01ae00 tx=0x7fbaec004b50 comp rx=0 tx=0).stop 2024-04-21T08:55:22.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.140+0000 7fbafd61d740 1 -- 172.21.15.28:0/2971458402 shutdown_connections 2024-04-21T08:55:22.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.140+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2971458402 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56238999fb10 0x5623899a1f70 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.140+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2971458402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56238997ea00 0x56238997b9f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.140+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2971458402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562389975e80 0x56238997b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.140+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2971458402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 0x56238997e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.140+0000 7fbafd61d740 1 -- 172.21.15.28:0/2971458402 >> 172.21.15.28:0/2971458402 conn(0x56238996fba0 msgr2=0x5623899718f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:22.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.140+0000 7fbad37fe640 0 client.26293 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.140+0000 7fbafd61d740 1 -- 172.21.15.28:0/2971458402 shutdown_connections 2024-04-21T08:55:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.141+0000 7fbafd61d740 1 -- 172.21.15.28:0/2971458402 wait complete. 2024-04-21T08:55:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.141+0000 7fbafd61d740 20 client.26293 trim_cache size 0 max 16384 2024-04-21T08:55:22.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.172+0000 7fbafd61d740 1 Processor -- start 2024-04-21T08:55:22.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.178+0000 7fbafd61d740 1 -- start start 2024-04-21T08:55:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.178+0000 7fbafd61d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 0x56238998a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.178+0000 7fbafd61d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562389975e80 0x562389971700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.178+0000 7fbafd61d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 0x56238997dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.178+0000 7fbafd61d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbae0012900 con 0x56238997ea00 2024-04-21T08:55:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.178+0000 7fbafd61d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbae00101d0 con 0x5623899754d0 2024-04-21T08:55:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.178+0000 7fbafd61d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbae000e020 con 0x562389975e80 2024-04-21T08:55:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.179+0000 7fbaf8fce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 0x56238997dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.179+0000 7fbaf8fce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 0x56238997dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44818/0 (socket says 172.21.15.28:44818) 2024-04-21T08:55:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.179+0000 7fbaf8fce640 1 -- 172.21.15.28:0/2073465482 learned_addr learned my addr 172.21.15.28:0/2073465482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.179+0000 7fbaf37fe640 1 --2- 172.21.15.28:0/2073465482 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562389975e80 0x562389971700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.179+0000 7fbaf3fff640 1 --2- 172.21.15.28:0/2073465482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 0x56238998a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.179+0000 7fbaf37fe640 1 -- 172.21.15.28:0/2073465482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 msgr2=0x56238998a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.179+0000 7fbaf37fe640 1 --2- 172.21.15.28:0/2073465482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 0x56238998a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.179+0000 7fbaf37fe640 1 -- 172.21.15.28:0/2073465482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 msgr2=0x56238997dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.179+0000 7fbaf37fe640 1 --2- 172.21.15.28:0/2073465482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 0x56238997dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.179+0000 7fbaf37fe640 1 -- 172.21.15.28:0/2073465482 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562389975c20 con 0x562389975e80 2024-04-21T08:55:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.180+0000 7fbaf37fe640 1 --2- 172.21.15.28:0/2073465482 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562389975e80 0x562389971700 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbae4005c40 tx=0x7fbae402cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3756ebf3f9decc98 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.181+0000 7fbad9ffb640 1 -- 172.21.15.28:0/2073465482 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae4042040 con 0x562389975e80 2024-04-21T08:55:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.181+0000 7fbad9ffb640 1 -- 172.21.15.28:0/2073465482 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbae4004940 con 0x562389975e80 2024-04-21T08:55:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.182+0000 7fbafd61d740 1 -- 172.21.15.28:0/2073465482 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562389975e80 msgr2=0x562389971700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.182+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2073465482 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562389975e80 0x562389971700 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbae4005c40 tx=0x7fbae402cc70 comp rx=0 tx=0).stop 2024-04-21T08:55:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.184+0000 7fbafd61d740 1 -- 172.21.15.28:0/2073465482 shutdown_connections 2024-04-21T08:55:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.185+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2073465482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 0x56238997dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.185+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2073465482 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562389975e80 0x562389971700 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.185+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2073465482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 0x56238998a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.185+0000 7fbafd61d740 1 -- 172.21.15.28:0/2073465482 >> 172.21.15.28:0/2073465482 conn(0x56238996fba0 msgr2=0x562389970ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.188+0000 7fbafd61d740 1 -- 172.21.15.28:0/2073465482 shutdown_connections 2024-04-21T08:55:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.190+0000 7fbafd61d740 1 -- 172.21.15.28:0/2073465482 wait complete. 2024-04-21T08:55:22.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.190+0000 7fbafd61d740 1 -- start start 2024-04-21T08:55:22.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.190+0000 7fbafd61d740 1 Processor -- start 2024-04-21T08:55:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.191+0000 7fbafd61d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 0x562389986de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.191+0000 7fbafd61d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562389975e80 0x562389987320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.191+0000 7fbaf3fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 0x562389986de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.191+0000 7fbaf3fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 0x562389986de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38474/0 (socket says 172.21.15.28:38474) 2024-04-21T08:55:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.192+0000 7fbafd61d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 0x562389983f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.192+0000 7fbaf37fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562389975e80 0x562389987320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.192+0000 7fbaf37fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562389975e80 0x562389987320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:53358/0 (socket says 172.21.15.28:53358) 2024-04-21T08:55:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.192+0000 7fbaf3fff640 1 -- 172.21.15.28:0/2690389079 learned_addr learned my addr 172.21.15.28:0/2690389079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.192+0000 7fbafd61d740 1 -- 172.21.15.28:0/2690389079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbae000e690 con 0x56238997ea00 2024-04-21T08:55:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.192+0000 7fbafd61d740 1 -- 172.21.15.28:0/2690389079 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbae000fb00 con 0x5623899754d0 2024-04-21T08:55:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.192+0000 7fbafd61d740 1 -- 172.21.15.28:0/2690389079 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbae000ed60 con 0x562389975e80 2024-04-21T08:55:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.192+0000 7fbafd61d740 10 client.0 _ll_register_callbacks cb 0x7fbaec01aa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.193+0000 7fbaf3fff640 1 -- 172.21.15.28:0/2690389079 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562389975e80 msgr2=0x562389987320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.193+0000 7fbaf3fff640 1 --2- 172.21.15.28:0/2690389079 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562389975e80 0x562389987320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.193+0000 7fbaf3fff640 1 -- 172.21.15.28:0/2690389079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 msgr2=0x562389983f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.193+0000 7fbaf3fff640 1 --2- 172.21.15.28:0/2690389079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 0x562389983f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.193+0000 7fbaf3fff640 1 -- 172.21.15.28:0/2690389079 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562389984760 con 0x5623899754d0 2024-04-21T08:55:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.194+0000 7fbaf3fff640 1 --2- 172.21.15.28:0/2690389079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 0x562389986de0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbae80bd010 tx=0x7fbae801a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59d8b8ebdf48a1a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.194+0000 7fbad37fe640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:55:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.194+0000 7fbad37fe640 1 -- 172.21.15.28:0/2690389079 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae80ab060 con 0x5623899754d0 2024-04-21T08:55:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.194+0000 7fbafd61d740 1 -- 172.21.15.28:0/2690389079 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbae4004940 con 0x5623899754d0 2024-04-21T08:55:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.194+0000 7fbad37fe640 1 -- 172.21.15.28:0/2690389079 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbae809a120 con 0x5623899754d0 2024-04-21T08:55:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.194+0000 7fbad37fe640 1 -- 172.21.15.28:0/2690389079 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbae80b2050 con 0x5623899754d0 2024-04-21T08:55:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.195+0000 7fbafd61d740 10 client.26288 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.195+0000 7fbafd61d740 1 -- 172.21.15.28:0/2690389079 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562389a06d60 con 0x5623899754d0 2024-04-21T08:55:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.195+0000 7fbad37fe640 1 -- 172.21.15.28:0/2690389079 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbae4004940 con 0x5623899754d0 2024-04-21T08:55:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.196+0000 7fbad37fe640 1 -- 172.21.15.28:0/2690389079 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fbae809cf00 con 0x5623899754d0 2024-04-21T08:55:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.196+0000 7fbafd61d740 10 client.26288 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.196+0000 7fbafd61d740 10 client.26288 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.196+0000 7fbafd61d740 20 client.26288 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.196+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2690389079 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562389a07460 0x562389a098a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.196+0000 7fbafd61d740 4 client.26288 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:55:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.196+0000 7fbafd61d740 1 -- 172.21.15.28:0/2690389079 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5623898e7c10 con 0x562389a07460 2024-04-21T08:55:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.196+0000 7fbaf37fe640 1 --2- 172.21.15.28:0/2690389079 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562389a07460 0x562389a098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.197+0000 7fbaf37fe640 1 --2- 172.21.15.28:0/2690389079 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562389a07460 0x562389a098a0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60b2f3ac066d1f27 server_cookie=d64499cf0a8ccf1c in_seq=0 out_seq=0 2024-04-21T08:55:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.197+0000 7fbad37fe640 10 client.26288 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.197+0000 7fbad37fe640 1 -- 172.21.15.28:0/2690389079 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5623898e7c10 con 0x562389a07460 2024-04-21T08:55:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.197+0000 7fbad37fe640 10 client.26288 handle_command_reply: tid=0 2024-04-21T08:55:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.197+0000 7fbafd61d740 1 client.26288 shutdown 2024-04-21T08:55:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.199+0000 7fbafd61d740 1 -- 172.21.15.28:0/2690389079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 msgr2=0x562389986de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.199+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2690389079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 0x562389986de0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbae80bd010 tx=0x7fbae801a040 comp rx=0 tx=0).stop 2024-04-21T08:55:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.203+0000 7fbafd61d740 1 -- 172.21.15.28:0/2690389079 shutdown_connections 2024-04-21T08:55:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.203+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2690389079 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562389a07460 0x562389a098a0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.203+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2690389079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56238997ea00 0x562389983f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.203+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2690389079 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562389975e80 0x562389987320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.203+0000 7fbafd61d740 1 --2- 172.21.15.28:0/2690389079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5623899754d0 0x562389986de0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.203+0000 7fbafd61d740 1 -- 172.21.15.28:0/2690389079 >> 172.21.15.28:0/2690389079 conn(0x56238996fba0 msgr2=0x56238996ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.203+0000 7fbad37fe640 0 client.26288 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.205+0000 7fbafd61d740 1 -- 172.21.15.28:0/2690389079 shutdown_connections 2024-04-21T08:55:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.206+0000 7fbafd61d740 1 -- 172.21.15.28:0/2690389079 wait complete. 2024-04-21T08:55:22.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.208+0000 7fbafd61d740 20 client.26288 trim_cache size 0 max 16384 2024-04-21T08:55:22.210 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:55:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.211+0000 7fbafaa31640 1 -- 172.21.15.28:0/3532222573 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbadc0a3f20 msgr2=0x7fbadc0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.211+0000 7fbafaa31640 1 --2- 172.21.15.28:0/3532222573 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbadc0a3f20 0x7fbadc0a63e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbaec009fa0 tx=0x7fbaec004a40 comp rx=0 tx=0).stop 2024-04-21T08:55:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.211+0000 7fbafaa31640 1 -- 172.21.15.28:0/3532222573 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbaf4100fc0 msgr2=0x7fbaf4073860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.211+0000 7fbafaa31640 1 --2- 172.21.15.28:0/3532222573 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbaf4100fc0 0x7fbaf4073860 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbae800e9f0 tx=0x7fbae800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.211+0000 7fbafaa31640 1 -- 172.21.15.28:0/3532222573 shutdown_connections 2024-04-21T08:55:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.211+0000 7fbafaa31640 1 --2- 172.21.15.28:0/3532222573 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbadc0a3f20 0x7fbadc0a63e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.211+0000 7fbafaa31640 1 --2- 172.21.15.28:0/3532222573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaf41062e0 0x7fbaf406de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.211+0000 7fbafaa31640 1 --2- 172.21.15.28:0/3532222573 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbaf41018e0 0x7fbaf406d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.211+0000 7fbafaa31640 1 --2- 172.21.15.28:0/3532222573 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbaf4100fc0 0x7fbaf4073860 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:22.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.211+0000 7fbafaa31640 1 -- 172.21.15.28:0/3532222573 >> 172.21.15.28:0/3532222573 conn(0x7fbaf4077f50 msgr2=0x7fbaf4103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:22.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.211+0000 7fbafaa31640 1 -- 172.21.15.28:0/3532222573 shutdown_connections 2024-04-21T08:55:22.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:22.211+0000 7fbafaa31640 1 -- 172.21.15.28:0/3532222573 wait complete. 2024-04-21T08:55:22.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:22 smithi028 ceph-mon[28003]: pgmap v128: 129 pgs: 129 active+clean; 15 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 62 MiB/s wr, 8.87k op/s 2024-04-21T08:55:22.755 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:22 smithi152 ceph-mon[31401]: pgmap v128: 129 pgs: 129 active+clean; 15 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 62 MiB/s wr, 8.87k op/s 2024-04-21T08:55:22.771 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:22 smithi171 ceph-mon[31358]: pgmap v128: 129 pgs: 129 active+clean; 15 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 62 MiB/s wr, 8.87k op/s 2024-04-21T08:55:23.231 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:55:23.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:55:23.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:55:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.437+0000 7f1e66fdb640 1 -- 172.21.15.28:0/275596057 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1e6010e7d0 msgr2=0x7f1e6010ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:23.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.437+0000 7f1e66fdb640 1 --2- 172.21.15.28:0/275596057 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1e6010e7d0 0x7f1e6010ec50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1e58009be0 tx=0x7f1e5802bf30 comp rx=0 tx=0).stop 2024-04-21T08:55:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.440+0000 7f1e66fdb640 1 -- 172.21.15.28:0/275596057 shutdown_connections 2024-04-21T08:55:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.440+0000 7f1e66fdb640 1 --2- 172.21.15.28:0/275596057 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e6011b730 0x7f1e6011db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.440+0000 7f1e66fdb640 1 --2- 172.21.15.28:0/275596057 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1e6010e7d0 0x7f1e6010ec50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.440+0000 7f1e66fdb640 1 --2- 172.21.15.28:0/275596057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e601107a0 0x7f1e6010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.440+0000 7f1e66fdb640 1 -- 172.21.15.28:0/275596057 >> 172.21.15.28:0/275596057 conn(0x7f1e6006d060 msgr2=0x7f1e6006d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:23.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.445+0000 7f1e66fdb640 1 -- 172.21.15.28:0/275596057 shutdown_connections 2024-04-21T08:55:23.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.447+0000 7f1e66fdb640 1 -- 172.21.15.28:0/275596057 wait complete. 2024-04-21T08:55:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.448+0000 7f1e66fdb640 1 Processor -- start 2024-04-21T08:55:23.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.454+0000 7f1e66fdb640 1 -- start start 2024-04-21T08:55:23.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.455+0000 7f1e66fdb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e6010e7d0 0x7f1e60118cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.455+0000 7f1e66fdb640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e601107a0 0x7f1e60111e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.455+0000 7f1e66fdb640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1e6011b730 0x7f1e60112380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.455+0000 7f1e66fdb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e60121030 con 0x7f1e6010e7d0 2024-04-21T08:55:23.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.455+0000 7f1e66fdb640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1e60120eb0 con 0x7f1e601107a0 2024-04-21T08:55:23.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.455+0000 7f1e66fdb640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1e601211b0 con 0x7f1e6011b730 2024-04-21T08:55:23.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.455+0000 7f1e5ffff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e601107a0 0x7f1e60111e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.455+0000 7f1e5ffff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e601107a0 0x7f1e60111e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38494/0 (socket says 172.21.15.28:38494) 2024-04-21T08:55:23.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.455+0000 7f1e5ffff640 1 -- 172.21.15.28:0/1909354608 learned_addr learned my addr 172.21.15.28:0/1909354608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:23.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.455+0000 7f1e5ffff640 1 -- 172.21.15.28:0/1909354608 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1e6011b730 msgr2=0x7f1e60112380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:55:23.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.455+0000 7f1e5ffff640 1 --2- 172.21.15.28:0/1909354608 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1e6011b730 0x7f1e60112380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.455+0000 7f1e5ffff640 1 -- 172.21.15.28:0/1909354608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e6010e7d0 msgr2=0x7f1e60118cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:55:23.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.455+0000 7f1e5ffff640 1 --2- 172.21.15.28:0/1909354608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e6010e7d0 0x7f1e60118cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.455+0000 7f1e5ffff640 1 -- 172.21.15.28:0/1909354608 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e60112940 con 0x7f1e601107a0 2024-04-21T08:55:23.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.456+0000 7f1e5ffff640 1 --2- 172.21.15.28:0/1909354608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e601107a0 0x7f1e60111e40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1e58036040 tx=0x7f1e58034780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:23.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.456+0000 7f1e5dffb640 1 -- 172.21.15.28:0/1909354608 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e58043020 con 0x7f1e601107a0 2024-04-21T08:55:23.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.456+0000 7f1e5dffb640 1 -- 172.21.15.28:0/1909354608 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e58034b80 con 0x7f1e601107a0 2024-04-21T08:55:23.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.456+0000 7f1e5dffb640 1 -- 172.21.15.28:0/1909354608 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5803f400 con 0x7f1e601107a0 2024-04-21T08:55:23.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.460+0000 7f1e66fdb640 1 -- 172.21.15.28:0/1909354608 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e60112bd0 con 0x7f1e601107a0 2024-04-21T08:55:23.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.461+0000 7f1e66fdb640 1 -- 172.21.15.28:0/1909354608 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e60116a70 con 0x7f1e601107a0 2024-04-21T08:55:23.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.464+0000 7f1e69bcf740 1 Processor -- start 2024-04-21T08:55:23.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.468+0000 7f1e5dffb640 1 -- 172.21.15.28:0/1909354608 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1e58046020 con 0x7f1e601107a0 2024-04-21T08:55:23.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.469+0000 7f1e5dffb640 1 --2- 172.21.15.28:0/1909354608 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1e480a3e60 0x7f1e480a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.469+0000 7f1e5dffb640 1 -- 172.21.15.28:0/1909354608 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1e580f2f10 con 0x7f1e601107a0 2024-04-21T08:55:23.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.470+0000 7f1e69bcf740 1 -- start start 2024-04-21T08:55:23.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.470+0000 7f1e69bcf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b1468e0 0x55d39b146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.470+0000 7f1e69bcf740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b147290 0x55d39b14f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.470+0000 7f1e69bcf740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b14fe10 0x55d39b152260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.470+0000 7f1e69bcf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d39b0aaf00 con 0x55d39b1468e0 2024-04-21T08:55:23.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.470+0000 7f1e69bcf740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55d39b048590 con 0x55d39b147290 2024-04-21T08:55:23.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.470+0000 7f1e69bcf740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55d39b0a6ec0 con 0x55d39b14fe10 2024-04-21T08:55:23.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.470+0000 7f1e65578640 1 --2- 172.21.15.28:0/1909354608 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1e480a3e60 0x7f1e480a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.471+0000 7f1e64d77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b1468e0 0x55d39b146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.471+0000 7f1e64d77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b1468e0 0x55d39b146cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44866/0 (socket says 172.21.15.28:44866) 2024-04-21T08:55:23.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.471+0000 7f1e64d77640 1 -- 172.21.15.28:0/3315053775 learned_addr learned my addr 172.21.15.28:0/3315053775 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:23.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.471+0000 7f1e5ffff640 1 --2- 172.21.15.28:0/3315053775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b147290 0x55d39b14f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.471+0000 7f1e65578640 1 --2- 172.21.15.28:0/3315053775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b14fe10 0x55d39b152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.472+0000 7f1e5ffff640 1 -- 172.21.15.28:0/3315053775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b14fe10 msgr2=0x55d39b152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.472+0000 7f1e5ffff640 1 --2- 172.21.15.28:0/3315053775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b14fe10 0x55d39b152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.472+0000 7f1e5ffff640 1 -- 172.21.15.28:0/3315053775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b1468e0 msgr2=0x55d39b146cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.472+0000 7f1e5ffff640 1 --2- 172.21.15.28:0/3315053775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b1468e0 0x55d39b146cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.472+0000 7f1e5ffff640 1 -- 172.21.15.28:0/3315053775 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d39ae857e0 con 0x55d39b147290 2024-04-21T08:55:23.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.475+0000 7f1e65578640 1 --2- 172.21.15.28:0/1909354608 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1e480a3e60 0x7f1e480a6320 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400dc10 tx=0x7f1e5400d280 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:23.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.477+0000 7f1e5ffff640 1 --2- 172.21.15.28:0/3315053775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b147290 0x55d39b14f840 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1e580f0370 tx=0x7f1e5803eb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b43d46c374f5ae5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:23.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.482+0000 7f1e46ffd640 1 -- 172.21.15.28:0/3315053775 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e580bf2c0 con 0x55d39b147290 2024-04-21T08:55:23.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.482+0000 7f1e46ffd640 1 -- 172.21.15.28:0/3315053775 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e580bebc0 con 0x55d39b147290 2024-04-21T08:55:23.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.483+0000 7f1e69bcf740 1 -- 172.21.15.28:0/3315053775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b147290 msgr2=0x55d39b14f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:23.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.483+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/3315053775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b147290 0x55d39b14f840 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1e580f0370 tx=0x7f1e5803eb50 comp rx=0 tx=0).stop 2024-04-21T08:55:23.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.484+0000 7f1e69bcf740 1 -- 172.21.15.28:0/3315053775 shutdown_connections 2024-04-21T08:55:23.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.484+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/3315053775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b14fe10 0x55d39b152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.484+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/3315053775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b147290 0x55d39b14f840 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.484+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/3315053775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b1468e0 0x55d39b146cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.484+0000 7f1e69bcf740 1 -- 172.21.15.28:0/3315053775 >> 172.21.15.28:0/3315053775 conn(0x55d39b140fb0 msgr2=0x55d39b141390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:23.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.485+0000 7f1e69bcf740 1 -- 172.21.15.28:0/3315053775 shutdown_connections 2024-04-21T08:55:23.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.485+0000 7f1e69bcf740 1 -- 172.21.15.28:0/3315053775 wait complete. 2024-04-21T08:55:23.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.486+0000 7f1e69bcf740 1 -- start start 2024-04-21T08:55:23.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.486+0000 7f1e69bcf740 1 Processor -- start 2024-04-21T08:55:23.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.487+0000 7f1e69bcf740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b1468e0 0x55d39b14cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.487+0000 7f1e69bcf740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b147290 0x55d39b14d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.487+0000 7f1e69bcf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b14fe10 0x55d39b14d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.487+0000 7f1e69bcf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d39b048590 con 0x55d39b14fe10 2024-04-21T08:55:23.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.487+0000 7f1e69bcf740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55d39b064580 con 0x55d39b147290 2024-04-21T08:55:23.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.487+0000 7f1e69bcf740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55d39b152b40 con 0x55d39b1468e0 2024-04-21T08:55:23.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.487+0000 7f1e69bcf740 10 client.0 _ll_register_callbacks cb 0x55d39b06a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:23.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.488+0000 7f1e65578640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b14fe10 0x55d39b14d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.488+0000 7f1e65578640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b14fe10 0x55d39b14d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44868/0 (socket says 172.21.15.28:44868) 2024-04-21T08:55:23.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.488+0000 7f1e65578640 1 -- 172.21.15.28:0/374224100 learned_addr learned my addr 172.21.15.28:0/374224100 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:23.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.488+0000 7f1e5ffff640 1 --2- 172.21.15.28:0/374224100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b147290 0x55d39b14d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.488+0000 7f1e64d77640 1 --2- 172.21.15.28:0/374224100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b1468e0 0x55d39b14cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.491+0000 7f1e65578640 1 -- 172.21.15.28:0/374224100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b1468e0 msgr2=0x55d39b14cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.491+0000 7f1e65578640 1 --2- 172.21.15.28:0/374224100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b1468e0 0x55d39b14cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.491+0000 7f1e65578640 1 -- 172.21.15.28:0/374224100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b147290 msgr2=0x55d39b14d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.491+0000 7f1e65578640 1 --2- 172.21.15.28:0/374224100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b147290 0x55d39b14d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.491+0000 7f1e65578640 1 -- 172.21.15.28:0/374224100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d39aee5f70 con 0x55d39b14fe10 2024-04-21T08:55:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.491+0000 7f1e65578640 1 --2- 172.21.15.28:0/374224100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b14fe10 0x55d39b14d760 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f1e540179f0 tx=0x7f1e54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=331eb1527b9d3115 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:23.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.492+0000 7f1e3ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:55:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.492+0000 7f1e3ffff640 1 -- 172.21.15.28:0/374224100 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54016600 con 0x55d39b14fe10 2024-04-21T08:55:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.493+0000 7f1e69bcf740 1 -- 172.21.15.28:0/374224100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e580bebc0 con 0x55d39b14fe10 2024-04-21T08:55:23.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.493+0000 7f1e3ffff640 1 -- 172.21.15.28:0/374224100 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1e540167a0 con 0x55d39b14fe10 2024-04-21T08:55:23.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.493+0000 7f1e3ffff640 1 -- 172.21.15.28:0/374224100 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1e5402f050 con 0x55d39b14fe10 2024-04-21T08:55:23.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.495+0000 7f1e69bcf740 10 client.17031 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:23.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.495+0000 7f1e69bcf740 1 -- 172.21.15.28:0/374224100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d39ad7d170 con 0x55d39b14fe10 2024-04-21T08:55:23.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.498+0000 7f1e3ffff640 1 -- 172.21.15.28:0/374224100 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e580bebc0 con 0x55d39b14fe10 2024-04-21T08:55:23.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.498+0000 7f1e3ffff640 1 -- 172.21.15.28:0/374224100 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f1e54015070 con 0x55d39b14fe10 2024-04-21T08:55:23.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.499+0000 7f1e69bcf740 10 client.17031 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:23.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.499+0000 7f1e69bcf740 10 client.17031 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:23.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.499+0000 7f1e69bcf740 20 client.17031 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:23.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.499+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/374224100 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d39b15ff20 0x55d39b162310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.499+0000 7f1e69bcf740 4 client.17031 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:23.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.499+0000 7f1e69bcf740 1 -- 172.21.15.28:0/374224100 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d39b0b8920 con 0x55d39b15ff20 2024-04-21T08:55:23.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.500+0000 7f1e64d77640 1 --2- 172.21.15.28:0/374224100 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d39b15ff20 0x55d39b162310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.513+0000 7f1e64d77640 1 --2- 172.21.15.28:0/374224100 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d39b15ff20 0x55d39b162310 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91ae67555b4d0031 server_cookie=c8854d362d10e25 in_seq=0 out_seq=0 2024-04-21T08:55:23.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.515+0000 7f1e3ffff640 10 client.17031 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:23.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.516+0000 7f1e3ffff640 1 -- 172.21.15.28:0/374224100 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d39b0b8920 con 0x55d39b15ff20 2024-04-21T08:55:23.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.516+0000 7f1e3ffff640 10 client.17031 handle_command_reply: tid=0 2024-04-21T08:55:23.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.516+0000 7f1e69bcf740 1 client.17031 shutdown 2024-04-21T08:55:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.518+0000 7f1e69bcf740 1 -- 172.21.15.28:0/374224100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b14fe10 msgr2=0x55d39b14d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.518+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/374224100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b14fe10 0x55d39b14d760 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f1e540179f0 tx=0x7f1e54004750 comp rx=0 tx=0).stop 2024-04-21T08:55:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.521+0000 7f1e69bcf740 1 -- 172.21.15.28:0/374224100 shutdown_connections 2024-04-21T08:55:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.521+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/374224100 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d39b15ff20 0x55d39b162310 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.521+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/374224100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b14fe10 0x55d39b14d760 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.521+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/374224100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b147290 0x55d39b14d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.521+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/374224100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b1468e0 0x55d39b14cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.521+0000 7f1e69bcf740 1 -- 172.21.15.28:0/374224100 >> 172.21.15.28:0/374224100 conn(0x55d39b140fb0 msgr2=0x55d39b142d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:23.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.521+0000 7f1e3ffff640 0 client.17031 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.524+0000 7f1e69bcf740 1 -- 172.21.15.28:0/374224100 shutdown_connections 2024-04-21T08:55:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.524+0000 7f1e69bcf740 1 -- 172.21.15.28:0/374224100 wait complete. 2024-04-21T08:55:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.524+0000 7f1e69bcf740 20 client.17031 trim_cache size 0 max 16384 2024-04-21T08:55:23.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.563+0000 7f1e69bcf740 1 Processor -- start 2024-04-21T08:55:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.564+0000 7f1e69bcf740 1 -- start start 2024-04-21T08:55:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.565+0000 7f1e69bcf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b1468e0 0x55d39b14f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.565+0000 7f1e69bcf740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b147290 0x55d39b167be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.565+0000 7f1e64d77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b1468e0 0x55d39b14f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.565+0000 7f1e64d77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b1468e0 0x55d39b14f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44870/0 (socket says 172.21.15.28:44870) 2024-04-21T08:55:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.565+0000 7f1e64d77640 1 -- 172.21.15.28:0/2453361963 learned_addr learned my addr 172.21.15.28:0/2453361963 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.565+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/2453361963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b14fe10 0x55d39b168120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.565+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2453361963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e601c9370 con 0x55d39b1468e0 2024-04-21T08:55:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.565+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2453361963 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1e601c6c40 con 0x55d39b14fe10 2024-04-21T08:55:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.565+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2453361963 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1e601c4a90 con 0x55d39b147290 2024-04-21T08:55:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.565+0000 7f1e65578640 1 --2- 172.21.15.28:0/2453361963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b14fe10 0x55d39b168120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.566+0000 7f1e5ffff640 1 --2- 172.21.15.28:0/2453361963 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b147290 0x55d39b167be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.567+0000 7f1e64d77640 1 -- 172.21.15.28:0/2453361963 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b147290 msgr2=0x55d39b167be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.567+0000 7f1e64d77640 1 --2- 172.21.15.28:0/2453361963 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b147290 0x55d39b167be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.567+0000 7f1e64d77640 1 -- 172.21.15.28:0/2453361963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b14fe10 msgr2=0x55d39b168120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.567+0000 7f1e64d77640 1 --2- 172.21.15.28:0/2453361963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b14fe10 0x55d39b168120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.567+0000 7f1e64d77640 1 -- 172.21.15.28:0/2453361963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d39b147030 con 0x55d39b1468e0 2024-04-21T08:55:23.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.570+0000 7f1e64d77640 1 --2- 172.21.15.28:0/2453361963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b1468e0 0x55d39b14f410 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f1e50004650 tx=0x7f1e50005410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3e7b4c232a356d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:23.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.575+0000 7f1e467fc640 1 -- 172.21.15.28:0/2453361963 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e50023070 con 0x55d39b1468e0 2024-04-21T08:55:23.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.575+0000 7f1e467fc640 1 -- 172.21.15.28:0/2453361963 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e50004c10 con 0x55d39b1468e0 2024-04-21T08:55:23.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.576+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2453361963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b1468e0 msgr2=0x55d39b14f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:23.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.576+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/2453361963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b1468e0 0x55d39b14f410 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f1e50004650 tx=0x7f1e50005410 comp rx=0 tx=0).stop 2024-04-21T08:55:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.577+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2453361963 shutdown_connections 2024-04-21T08:55:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.577+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/2453361963 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b14fe10 0x55d39b168120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.577+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/2453361963 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b147290 0x55d39b167be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.577+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/2453361963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b1468e0 0x55d39b14f410 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.577+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2453361963 >> 172.21.15.28:0/2453361963 conn(0x55d39b140fb0 msgr2=0x55d39b142150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:23.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.580+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2453361963 shutdown_connections 2024-04-21T08:55:23.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.581+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2453361963 wait complete. 2024-04-21T08:55:23.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.582+0000 7f1e69bcf740 1 -- start start 2024-04-21T08:55:23.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.582+0000 7f1e69bcf740 1 Processor -- start 2024-04-21T08:55:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.585+0000 7f1e69bcf740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b1468e0 0x55d39b155a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.585+0000 7f1e69bcf740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b147290 0x55d39b155f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.585+0000 7f1e69bcf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b14fe10 0x55d39b1564a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.585+0000 7f1e69bcf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e601c5100 con 0x55d39b14fe10 2024-04-21T08:55:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.585+0000 7f1e69bcf740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1e601c6570 con 0x55d39b1468e0 2024-04-21T08:55:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.585+0000 7f1e69bcf740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1e601c57d0 con 0x55d39b147290 2024-04-21T08:55:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.585+0000 7f1e69bcf740 10 client.0 _ll_register_callbacks cb 0x7f1e54012ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:23.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.588+0000 7f1e5ffff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b147290 0x55d39b155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.588+0000 7f1e5ffff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b147290 0x55d39b155f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:53410/0 (socket says 172.21.15.28:53410) 2024-04-21T08:55:23.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.588+0000 7f1e5ffff640 1 -- 172.21.15.28:0/2864897906 learned_addr learned my addr 172.21.15.28:0/2864897906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:23.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.588+0000 7f1e64d77640 1 --2- 172.21.15.28:0/2864897906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b1468e0 0x55d39b155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.590+0000 7f1e5ffff640 1 -- 172.21.15.28:0/2864897906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b1468e0 msgr2=0x55d39b155a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.590+0000 7f1e5ffff640 1 --2- 172.21.15.28:0/2864897906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b1468e0 0x55d39b155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.590+0000 7f1e5ffff640 1 -- 172.21.15.28:0/2864897906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b14fe10 msgr2=0x55d39b1564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.590+0000 7f1e5ffff640 1 --2- 172.21.15.28:0/2864897906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b14fe10 0x55d39b1564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.590+0000 7f1e5ffff640 1 -- 172.21.15.28:0/2864897906 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d39b1c56b0 con 0x55d39b147290 2024-04-21T08:55:23.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.593+0000 7f1e5ffff640 1 --2- 172.21.15.28:0/2864897906 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b147290 0x55d39b155f60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1e580beb50 tx=0x7f1e5802cd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=659b60fc618cad4f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:23.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.593+0000 7f1e3ffff640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:55:23.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.596+0000 7f1e3ffff640 1 -- 172.21.15.28:0/2864897906 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e580bb3d0 con 0x55d39b147290 2024-04-21T08:55:23.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.596+0000 7f1e3ffff640 1 -- 172.21.15.28:0/2864897906 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f1e580c73a0 con 0x55d39b147290 2024-04-21T08:55:23.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.596+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2864897906 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e50004c10 con 0x55d39b147290 2024-04-21T08:55:23.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.598+0000 7f1e3ffff640 1 -- 172.21.15.28:0/2864897906 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1e580bebc0 con 0x55d39b147290 2024-04-21T08:55:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.601+0000 7f1e69bcf740 10 client.26314 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:23.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.601+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2864897906 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d39b1c69f0 con 0x55d39b147290 2024-04-21T08:55:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.604+0000 7f1e3ffff640 1 -- 172.21.15.28:0/2864897906 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e50004c10 con 0x55d39b147290 2024-04-21T08:55:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.604+0000 7f1e3ffff640 1 -- 172.21.15.28:0/2864897906 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f1e580be010 con 0x55d39b147290 2024-04-21T08:55:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.605+0000 7f1e69bcf740 10 client.26314 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.605+0000 7f1e69bcf740 10 client.26314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.605+0000 7f1e69bcf740 20 client.26314 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.605+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/2864897906 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d39b1c70d0 0x55d39b1c9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.605+0000 7f1e69bcf740 4 client.26314 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:55:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.605+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2864897906 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d39b0b2790 con 0x55d39b1c70d0 2024-04-21T08:55:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.605+0000 7f1e64d77640 1 --2- 172.21.15.28:0/2864897906 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d39b1c70d0 0x55d39b1c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.607+0000 7f1e64d77640 1 --2- 172.21.15.28:0/2864897906 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d39b1c70d0 0x55d39b1c9510 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c80f5797ae4535fe server_cookie=a5929323ddac69c9 in_seq=0 out_seq=0 2024-04-21T08:55:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.607+0000 7f1e3ffff640 10 client.26314 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.612+0000 7f1e3ffff640 1 -- 172.21.15.28:0/2864897906 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d39b0b2790 con 0x55d39b1c70d0 2024-04-21T08:55:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.612+0000 7f1e3ffff640 10 client.26314 handle_command_reply: tid=0 2024-04-21T08:55:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.613+0000 7f1e69bcf740 1 client.26314 shutdown 2024-04-21T08:55:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.614+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2864897906 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b147290 msgr2=0x55d39b155f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.614+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/2864897906 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b147290 0x55d39b155f60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1e580beb50 tx=0x7f1e5802cd90 comp rx=0 tx=0).stop 2024-04-21T08:55:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.617+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2864897906 shutdown_connections 2024-04-21T08:55:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.617+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/2864897906 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d39b1c70d0 0x55d39b1c9510 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.617+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/2864897906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d39b14fe10 0x55d39b1564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.617+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/2864897906 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d39b147290 0x55d39b155f60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.617+0000 7f1e69bcf740 1 --2- 172.21.15.28:0/2864897906 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d39b1468e0 0x55d39b155a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.617+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2864897906 >> 172.21.15.28:0/2864897906 conn(0x55d39b140fb0 msgr2=0x55d39b1429e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.617+0000 7f1e3ffff640 0 client.26314 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.621+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2864897906 shutdown_connections 2024-04-21T08:55:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.623+0000 7f1e69bcf740 1 -- 172.21.15.28:0/2864897906 wait complete. 2024-04-21T08:55:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.626+0000 7f1e69bcf740 20 client.26314 trim_cache size 0 max 16384 2024-04-21T08:55:23.628 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:55:23.628 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:55:23.628 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "0acf896b-0514-4934-9514-30a1c103351c", 2024-04-21T08:55:23.628 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:55:23.628 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:55:23.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.629+0000 7f1e66fdb640 1 -- 172.21.15.28:0/1909354608 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1e480a3e60 msgr2=0x7f1e480a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:23.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.629+0000 7f1e66fdb640 1 --2- 172.21.15.28:0/1909354608 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1e480a3e60 0x7f1e480a6320 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400dc10 tx=0x7f1e5400d280 comp rx=0 tx=0).stop 2024-04-21T08:55:23.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.629+0000 7f1e66fdb640 1 -- 172.21.15.28:0/1909354608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e601107a0 msgr2=0x7f1e60111e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:23.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.629+0000 7f1e66fdb640 1 --2- 172.21.15.28:0/1909354608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e601107a0 0x7f1e60111e40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1e58036040 tx=0x7f1e58034780 comp rx=0 tx=0).stop 2024-04-21T08:55:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.630+0000 7f1e66fdb640 1 -- 172.21.15.28:0/1909354608 shutdown_connections 2024-04-21T08:55:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.630+0000 7f1e66fdb640 1 --2- 172.21.15.28:0/1909354608 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1e480a3e60 0x7f1e480a6320 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.630+0000 7f1e66fdb640 1 --2- 172.21.15.28:0/1909354608 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1e6011b730 0x7f1e60112380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.630+0000 7f1e66fdb640 1 --2- 172.21.15.28:0/1909354608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e601107a0 0x7f1e60111e40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.630+0000 7f1e66fdb640 1 --2- 172.21.15.28:0/1909354608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e6010e7d0 0x7f1e60118cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.630+0000 7f1e66fdb640 1 -- 172.21.15.28:0/1909354608 >> 172.21.15.28:0/1909354608 conn(0x7f1e6006d060 msgr2=0x7f1e60072a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:23.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.634+0000 7f1e66fdb640 1 -- 172.21.15.28:0/1909354608 shutdown_connections 2024-04-21T08:55:23.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.635+0000 7f1e66fdb640 1 -- 172.21.15.28:0/1909354608 wait complete. 2024-04-21T08:55:23.657 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:55:23.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:55:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.858+0000 7f07c7f75640 1 -- 172.21.15.28:0/3821134836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07c01106d0 msgr2=0x7f07c010e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:23.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.858+0000 7f07c7f75640 1 --2- 172.21.15.28:0/3821134836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07c01106d0 0x7f07c010e270 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009c10 tx=0x7f07b802bf10 comp rx=0 tx=0).stop 2024-04-21T08:55:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.862+0000 7f07c7f75640 1 -- 172.21.15.28:0/3821134836 shutdown_connections 2024-04-21T08:55:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.862+0000 7f07c7f75640 1 --2- 172.21.15.28:0/3821134836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c011b730 0x7f07c011db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.862+0000 7f07c7f75640 1 --2- 172.21.15.28:0/3821134836 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f07c010e7b0 0x7f07c010ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.862+0000 7f07c7f75640 1 --2- 172.21.15.28:0/3821134836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07c01106d0 0x7f07c010e270 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.862+0000 7f07c7f75640 1 -- 172.21.15.28:0/3821134836 >> 172.21.15.28:0/3821134836 conn(0x7f07c006d070 msgr2=0x7f07c006d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:23.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.866+0000 7f07c7f75640 1 -- 172.21.15.28:0/3821134836 shutdown_connections 2024-04-21T08:55:23.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.871+0000 7f07c7f75640 1 -- 172.21.15.28:0/3821134836 wait complete. 2024-04-21T08:55:23.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.871+0000 7f07c7f75640 1 Processor -- start 2024-04-21T08:55:23.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.872+0000 7f07c7f75640 1 -- start start 2024-04-21T08:55:23.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.872+0000 7f07c7f75640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c010e7b0 0x7f07c01a76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.872+0000 7f07c7f75640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f07c01106d0 0x7f07c01a7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.872+0000 7f07c7f75640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07c011b730 0x7f07c01abf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.872+0000 7f07c7f75640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07c0121080 con 0x7f07c010e7b0 2024-04-21T08:55:23.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.872+0000 7f07c7f75640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f07c0120f00 con 0x7f07c011b730 2024-04-21T08:55:23.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.872+0000 7f07c7f75640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f07c0121200 con 0x7f07c01106d0 2024-04-21T08:55:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.874+0000 7f07c6512640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07c011b730 0x7f07c01abf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.875+0000 7f07c6512640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07c011b730 0x7f07c01abf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:40814/0 (socket says 172.21.15.28:40814) 2024-04-21T08:55:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.875+0000 7f07c6512640 1 -- 172.21.15.28:0/1753979393 learned_addr learned my addr 172.21.15.28:0/1753979393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.877+0000 7f07c6512640 1 -- 172.21.15.28:0/1753979393 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f07c01106d0 msgr2=0x7f07c01a7bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:55:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.877+0000 7f07c6512640 1 --2- 172.21.15.28:0/1753979393 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f07c01106d0 0x7f07c01a7bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.877+0000 7f07c6512640 1 -- 172.21.15.28:0/1753979393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c010e7b0 msgr2=0x7f07c01a76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.877+0000 7f07c6512640 1 --2- 172.21.15.28:0/1753979393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c010e7b0 0x7f07c01a76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.877+0000 7f07c6512640 1 -- 172.21.15.28:0/1753979393 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07c01ac6a0 con 0x7f07c011b730 2024-04-21T08:55:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.878+0000 7f07c6512640 1 --2- 172.21.15.28:0/1753979393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07c011b730 0x7f07c01abf70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f07bc00e940 tx=0x7f07bc00ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.879+0000 7f07b6ffd640 1 -- 172.21.15.28:0/1753979393 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07bc00ccc0 con 0x7f07c011b730 2024-04-21T08:55:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.879+0000 7f07b6ffd640 1 -- 172.21.15.28:0/1753979393 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07bc0054d0 con 0x7f07c011b730 2024-04-21T08:55:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.879+0000 7f07c7f75640 1 -- 172.21.15.28:0/1753979393 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07c01ac990 con 0x7f07c011b730 2024-04-21T08:55:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.879+0000 7f07c7f75640 1 -- 172.21.15.28:0/1753979393 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07c01b4220 con 0x7f07c011b730 2024-04-21T08:55:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.881+0000 7f07b6ffd640 1 -- 172.21.15.28:0/1753979393 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07bc010430 con 0x7f07c011b730 2024-04-21T08:55:23.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.884+0000 7f07cab63740 1 Processor -- start 2024-04-21T08:55:23.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.884+0000 7f07cab63740 1 -- start start 2024-04-21T08:55:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.886+0000 7f07cab63740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 0x55a48cdb1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.886+0000 7f07cab63740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1ea0 0x55a48cdba540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.886+0000 7f07cab63740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 0x55a48cdbce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.886+0000 7f07cab63740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a48cd161e0 con 0x55a48cdbaa80 2024-04-21T08:55:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.886+0000 7f07cab63740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55a48ccb3390 con 0x55a48cdb1580 2024-04-21T08:55:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.886+0000 7f07cab63740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55a48cd121a0 con 0x55a48cdb1ea0 2024-04-21T08:55:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.887+0000 7f07b6ffd640 1 -- 172.21.15.28:0/1753979393 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f07bc004f80 con 0x7f07c011b730 2024-04-21T08:55:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.888+0000 7f07b6ffd640 1 --2- 172.21.15.28:0/1753979393 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f07ac0a3ea0 0x7f07ac0a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.888+0000 7f07b6ffd640 1 -- 172.21.15.28:0/1753979393 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f07bc0d0e10 con 0x7f07c011b730 2024-04-21T08:55:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.888+0000 7f07c6512640 1 --2- 172.21.15.28:0/1753979393 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f07ac0a3ea0 0x7f07ac0a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.889+0000 7f07c6512640 1 --2- 172.21.15.28:0/1753979393 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f07ac0a3ea0 0x7f07ac0a6360 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f07bc004a90 tx=0x7f07bc004980 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.890+0000 7f07c5d11640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1ea0 0x55a48cdba540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.893+0000 7f07c5510640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 0x55a48cdb1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.893+0000 7f07c5510640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 0x55a48cdb1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:40822/0 (socket says 172.21.15.28:40822) 2024-04-21T08:55:23.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.893+0000 7f07c5510640 1 -- 172.21.15.28:0/1753209849 learned_addr learned my addr 172.21.15.28:0/1753209849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:23.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.893+0000 7f07c5510640 1 --2- 172.21.15.28:0/1753209849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 0x55a48cdbce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.894+0000 7f07c5d11640 1 -- 172.21.15.28:0/1753209849 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 msgr2=0x55a48cdb1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.894+0000 7f07c5d11640 1 --2- 172.21.15.28:0/1753209849 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 0x55a48cdb1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.894+0000 7f07c5d11640 1 -- 172.21.15.28:0/1753209849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 msgr2=0x55a48cdbce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.894+0000 7f07c5d11640 1 --2- 172.21.15.28:0/1753209849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 0x55a48cdbce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.894+0000 7f07c5d11640 1 -- 172.21.15.28:0/1753209849 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a48caf27d0 con 0x55a48cdb1ea0 2024-04-21T08:55:23.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.899+0000 7f07c5d11640 1 --2- 172.21.15.28:0/1753209849 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1ea0 0x55a48cdba540 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009b30 tx=0x7f07b8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=201b3b4337330e99 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:23.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.901+0000 7f07abfff640 1 -- 172.21.15.28:0/1753209849 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8042020 con 0x55a48cdb1ea0 2024-04-21T08:55:23.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.901+0000 7f07abfff640 1 -- 172.21.15.28:0/1753209849 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07b802c8b0 con 0x55a48cdb1ea0 2024-04-21T08:55:23.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.901+0000 7f07abfff640 1 -- 172.21.15.28:0/1753209849 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b8004390 con 0x55a48cdb1ea0 2024-04-21T08:55:23.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.902+0000 7f07cab63740 1 -- 172.21.15.28:0/1753209849 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1ea0 msgr2=0x55a48cdba540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:23.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.902+0000 7f07cab63740 1 --2- 172.21.15.28:0/1753209849 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1ea0 0x55a48cdba540 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009b30 tx=0x7f07b8009840 comp rx=0 tx=0).stop 2024-04-21T08:55:23.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.905+0000 7f07cab63740 1 -- 172.21.15.28:0/1753209849 shutdown_connections 2024-04-21T08:55:23.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.905+0000 7f07cab63740 1 --2- 172.21.15.28:0/1753209849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 0x55a48cdbce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.905+0000 7f07cab63740 1 --2- 172.21.15.28:0/1753209849 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1ea0 0x55a48cdba540 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.905+0000 7f07cab63740 1 --2- 172.21.15.28:0/1753209849 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 0x55a48cdb1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.905+0000 7f07cab63740 1 -- 172.21.15.28:0/1753209849 >> 172.21.15.28:0/1753209849 conn(0x55a48cdabcb0 msgr2=0x55a48cdac090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:23.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.909+0000 7f07cab63740 1 -- 172.21.15.28:0/1753209849 shutdown_connections 2024-04-21T08:55:23.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.913+0000 7f07cab63740 1 -- 172.21.15.28:0/1753209849 wait complete. 2024-04-21T08:55:23.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.913+0000 7f07cab63740 1 -- start start 2024-04-21T08:55:23.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.913+0000 7f07cab63740 1 Processor -- start 2024-04-21T08:55:23.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.916+0000 7f07cab63740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1580 0x55a48cdb7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.916+0000 7f07cab63740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdb1ea0 0x55a48cdb7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.916+0000 7f07cab63740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdbaa80 0x55a48cdb8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.916+0000 7f07cab63740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a48cdbd860 con 0x55a48cdb1ea0 2024-04-21T08:55:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.916+0000 7f07cab63740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55a48cccf2f0 con 0x55a48cdbaa80 2024-04-21T08:55:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.917+0000 7f07cab63740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55a48cdbd9e0 con 0x55a48cdb1580 2024-04-21T08:55:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.917+0000 7f07cab63740 10 client.0 _ll_register_callbacks cb 0x55a48ccdc580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.917+0000 7f07c5510640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1580 0x55a48cdb7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.917+0000 7f07c5510640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1580 0x55a48cdb7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:58112/0 (socket says 172.21.15.28:58112) 2024-04-21T08:55:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.917+0000 7f07c5510640 1 -- 172.21.15.28:0/634621406 learned_addr learned my addr 172.21.15.28:0/634621406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.917+0000 7f07c5510640 1 --2- 172.21.15.28:0/634621406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdbaa80 0x55a48cdb8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.918+0000 7f07c5510640 1 -- 172.21.15.28:0/634621406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdbaa80 msgr2=0x55a48cdb8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.918+0000 7f07c5510640 1 --2- 172.21.15.28:0/634621406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdbaa80 0x55a48cdb8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.918+0000 7f07c5510640 1 -- 172.21.15.28:0/634621406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdb1ea0 msgr2=0x55a48cdb7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.918+0000 7f07c5510640 1 --2- 172.21.15.28:0/634621406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdb1ea0 0x55a48cdb7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.918+0000 7f07c5510640 1 -- 172.21.15.28:0/634621406 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a48ca283f0 con 0x55a48cdb1580 2024-04-21T08:55:23.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.919+0000 7f07c5510640 1 --2- 172.21.15.28:0/634621406 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1580 0x55a48cdb7a00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f07b0000fe0 tx=0x7f07b0006860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fba8e58c2c564fc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.922+0000 7f07a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:55:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.922+0000 7f07a97fa640 1 -- 172.21.15.28:0/634621406 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b0011040 con 0x55a48cdb1580 2024-04-21T08:55:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.922+0000 7f07cab63740 1 -- 172.21.15.28:0/634621406 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07b802c8b0 con 0x55a48cdb1580 2024-04-21T08:55:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.922+0000 7f07a97fa640 1 -- 172.21.15.28:0/634621406 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f07b0017880 con 0x55a48cdb1580 2024-04-21T08:55:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.927+0000 7f07a97fa640 1 -- 172.21.15.28:0/634621406 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f07b0007b10 con 0x55a48cdb1580 2024-04-21T08:55:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.931+0000 7f07cab63740 10 client.26320 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.931+0000 7f07cab63740 1 -- 172.21.15.28:0/634621406 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a48c9e80f0 con 0x55a48cdb1580 2024-04-21T08:55:23.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.932+0000 7f07a97fa640 1 -- 172.21.15.28:0/634621406 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07b802c8b0 con 0x55a48cdb1580 2024-04-21T08:55:23.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.933+0000 7f07a97fa640 1 -- 172.21.15.28:0/634621406 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f07b0008180 con 0x55a48cdb1580 2024-04-21T08:55:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.933+0000 7f07cab63740 10 client.26320 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.933+0000 7f07cab63740 10 client.26320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.933+0000 7f07cab63740 20 client.26320 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.933+0000 7f07cab63740 1 --2- 172.21.15.28:0/634621406 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a48cddbf10 0x55a48cdde390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.933+0000 7f07cab63740 4 client.26320 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.933+0000 7f07cab63740 1 -- 172.21.15.28:0/634621406 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a48cd23c00 con 0x55a48cddbf10 2024-04-21T08:55:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.934+0000 7f07c5d11640 1 --2- 172.21.15.28:0/634621406 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a48cddbf10 0x55a48cdde390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.941+0000 7f07c5d11640 1 --2- 172.21.15.28:0/634621406 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a48cddbf10 0x55a48cdde390 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe50a470c77932aa server_cookie=92754bf95bcb0465 in_seq=0 out_seq=0 2024-04-21T08:55:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.941+0000 7f07a97fa640 10 client.26320 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.942+0000 7f07a97fa640 1 -- 172.21.15.28:0/634621406 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a48cd23c00 con 0x55a48cddbf10 2024-04-21T08:55:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.942+0000 7f07a97fa640 10 client.26320 handle_command_reply: tid=0 2024-04-21T08:55:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.942+0000 7f07cab63740 1 client.26320 shutdown 2024-04-21T08:55:23.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.943+0000 7f07cab63740 1 -- 172.21.15.28:0/634621406 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1580 msgr2=0x55a48cdb7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:23.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.943+0000 7f07cab63740 1 --2- 172.21.15.28:0/634621406 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1580 0x55a48cdb7a00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f07b0000fe0 tx=0x7f07b0006860 comp rx=0 tx=0).stop 2024-04-21T08:55:23.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.943+0000 7f07cab63740 1 -- 172.21.15.28:0/634621406 shutdown_connections 2024-04-21T08:55:23.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.943+0000 7f07cab63740 1 --2- 172.21.15.28:0/634621406 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a48cddbf10 0x55a48cdde390 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.943+0000 7f07cab63740 1 --2- 172.21.15.28:0/634621406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdbaa80 0x55a48cdb8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.943+0000 7f07cab63740 1 --2- 172.21.15.28:0/634621406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdb1ea0 0x55a48cdb7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.943+0000 7f07cab63740 1 --2- 172.21.15.28:0/634621406 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1580 0x55a48cdb7a00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.943+0000 7f07cab63740 1 -- 172.21.15.28:0/634621406 >> 172.21.15.28:0/634621406 conn(0x55a48cdabcb0 msgr2=0x55a48cdada00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.943+0000 7f07a97fa640 0 client.26320 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.943+0000 7f07cab63740 1 -- 172.21.15.28:0/634621406 shutdown_connections 2024-04-21T08:55:23.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.943+0000 7f07cab63740 1 -- 172.21.15.28:0/634621406 wait complete. 2024-04-21T08:55:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.945+0000 7f07cab63740 20 client.26320 trim_cache size 0 max 16384 2024-04-21T08:55:23.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.981+0000 7f07cab63740 1 Processor -- start 2024-04-21T08:55:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.983+0000 7f07cab63740 1 -- start start 2024-04-21T08:55:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.983+0000 7f07cab63740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 0x55a48cdc67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.983+0000 7f07cab63740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1ea0 0x55a48cdb6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.983+0000 7f07cab63740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 0x55a48cdb82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.983+0000 7f07cab63740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07a4012910 con 0x55a48cdbaa80 2024-04-21T08:55:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.983+0000 7f07cab63740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f07a40101e0 con 0x55a48cdb1580 2024-04-21T08:55:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.983+0000 7f07cab63740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f07a400e030 con 0x55a48cdb1ea0 2024-04-21T08:55:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.987+0000 7f07c5d11640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1ea0 0x55a48cdb6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.987+0000 7f07c5510640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 0x55a48cdb82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.987+0000 7f07c5510640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 0x55a48cdb82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47428/0 (socket says 172.21.15.28:47428) 2024-04-21T08:55:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.987+0000 7f07c5510640 1 -- 172.21.15.28:0/1472484501 learned_addr learned my addr 172.21.15.28:0/1472484501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.987+0000 7f07c5510640 1 --2- 172.21.15.28:0/1472484501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 0x55a48cdc67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.990+0000 7f07c5510640 1 -- 172.21.15.28:0/1472484501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1ea0 msgr2=0x55a48cdb6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.990+0000 7f07c5510640 1 --2- 172.21.15.28:0/1472484501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1ea0 0x55a48cdb6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.990+0000 7f07c5510640 1 -- 172.21.15.28:0/1472484501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 msgr2=0x55a48cdc67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.990+0000 7f07c5510640 1 --2- 172.21.15.28:0/1472484501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 0x55a48cdc67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.990+0000 7f07c5510640 1 -- 172.21.15.28:0/1472484501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a48cdabae0 con 0x55a48cdbaa80 2024-04-21T08:55:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.990+0000 7f07c5510640 1 --2- 172.21.15.28:0/1472484501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 0x55a48cdb82a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f07b000b500 tx=0x7f07b00047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9cfccf70bf6702f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.991+0000 7f07ab7fe640 1 -- 172.21.15.28:0/1472484501 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b0033070 con 0x55a48cdbaa80 2024-04-21T08:55:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.991+0000 7f07ab7fe640 1 -- 172.21.15.28:0/1472484501 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07b0017d30 con 0x55a48cdbaa80 2024-04-21T08:55:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.991+0000 7f07ab7fe640 1 -- 172.21.15.28:0/1472484501 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b00081e0 con 0x55a48cdbaa80 2024-04-21T08:55:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.992+0000 7f07cab63740 1 -- 172.21.15.28:0/1472484501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 msgr2=0x55a48cdb82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.992+0000 7f07cab63740 1 --2- 172.21.15.28:0/1472484501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 0x55a48cdb82a0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f07b000b500 tx=0x7f07b00047d0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.997+0000 7f07cab63740 1 -- 172.21.15.28:0/1472484501 shutdown_connections 2024-04-21T08:55:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.997+0000 7f07cab63740 1 --2- 172.21.15.28:0/1472484501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 0x55a48cdb82a0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.997+0000 7f07cab63740 1 --2- 172.21.15.28:0/1472484501 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1ea0 0x55a48cdb6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.997+0000 7f07cab63740 1 --2- 172.21.15.28:0/1472484501 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 0x55a48cdc67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:23.997+0000 7f07cab63740 1 -- 172.21.15.28:0/1472484501 >> 172.21.15.28:0/1472484501 conn(0x55a48cdabcb0 msgr2=0x55a48cdaad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:24.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.000+0000 7f07cab63740 1 -- 172.21.15.28:0/1472484501 shutdown_connections 2024-04-21T08:55:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.002+0000 7f07cab63740 1 -- 172.21.15.28:0/1472484501 wait complete. 2024-04-21T08:55:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.002+0000 7f07cab63740 1 -- start start 2024-04-21T08:55:24.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.003+0000 7f07cab63740 1 Processor -- start 2024-04-21T08:55:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.003+0000 7f07cab63740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 0x55a48cdc2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.003+0000 7f07cab63740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1ea0 0x55a48cdc31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.003+0000 7f07cab63740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 0x55a48cdb40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.003+0000 7f07cab63740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07a400e6a0 con 0x55a48cdbaa80 2024-04-21T08:55:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.003+0000 7f07cab63740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f07a400fb10 con 0x55a48cdb1580 2024-04-21T08:55:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.003+0000 7f07cab63740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f07a400ed70 con 0x55a48cdb1ea0 2024-04-21T08:55:24.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.003+0000 7f07cab63740 10 client.0 _ll_register_callbacks cb 0x7f07b000ba50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:24.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.006+0000 7f07c5510640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 0x55a48cdc2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:24.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.006+0000 7f07c5510640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 0x55a48cdc2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:40844/0 (socket says 172.21.15.28:40844) 2024-04-21T08:55:24.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.006+0000 7f07c5510640 1 -- 172.21.15.28:0/2289583383 learned_addr learned my addr 172.21.15.28:0/2289583383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:24.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.006+0000 7f07c5510640 1 --2- 172.21.15.28:0/2289583383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 0x55a48cdb40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:24.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.007+0000 7f07c5510640 1 -- 172.21.15.28:0/2289583383 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1ea0 msgr2=0x55a48cdc31f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:24.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.007+0000 7f07c5510640 1 --2- 172.21.15.28:0/2289583383 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1ea0 0x55a48cdc31f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.007+0000 7f07c5510640 1 -- 172.21.15.28:0/2289583383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 msgr2=0x55a48cdb40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:24.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.007+0000 7f07c5510640 1 --2- 172.21.15.28:0/2289583383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 0x55a48cdb40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.007+0000 7f07c5510640 1 -- 172.21.15.28:0/2289583383 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a48cdb48f0 con 0x55a48cdb1580 2024-04-21T08:55:24.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.008+0000 7f07c5510640 1 --2- 172.21.15.28:0/2289583383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 0x55a48cdc2cb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f07b0018010 tx=0x7f07b0007dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a7e5d81f262430f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:24.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.008+0000 7f07a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:55:24.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.009+0000 7f07a97fa640 1 -- 172.21.15.28:0/2289583383 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b0033070 con 0x55a48cdb1580 2024-04-21T08:55:24.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.009+0000 7f07cab63740 1 -- 172.21.15.28:0/2289583383 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07b0017d30 con 0x55a48cdb1580 2024-04-21T08:55:24.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.009+0000 7f07a97fa640 1 -- 172.21.15.28:0/2289583383 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f07b0007f60 con 0x55a48cdb1580 2024-04-21T08:55:24.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.012+0000 7f07a97fa640 1 -- 172.21.15.28:0/2289583383 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f07b0008160 con 0x55a48cdb1580 2024-04-21T08:55:24.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.015+0000 7f07cab63740 10 client.26318 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:24.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.015+0000 7f07cab63740 1 -- 172.21.15.28:0/2289583383 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a48ce42080 con 0x55a48cdb1580 2024-04-21T08:55:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.019+0000 7f07a97fa640 1 -- 172.21.15.28:0/2289583383 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07b0017d30 con 0x55a48cdb1580 2024-04-21T08:55:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.019+0000 7f07a97fa640 1 -- 172.21.15.28:0/2289583383 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f07b0039070 con 0x55a48cdb1580 2024-04-21T08:55:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.020+0000 7f07cab63740 10 client.26318 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.020+0000 7f07cab63740 10 client.26318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:24.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.020+0000 7f07cab63740 20 client.26318 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.020+0000 7f07cab63740 1 --2- 172.21.15.28:0/2289583383 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a48ce426a0 0x55a48ce44ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.020+0000 7f07cab63740 4 client.26318 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:55:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.020+0000 7f07cab63740 1 -- 172.21.15.28:0/2289583383 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a48cd1da70 con 0x55a48ce426a0 2024-04-21T08:55:24.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.023+0000 7f07c5d11640 1 --2- 172.21.15.28:0/2289583383 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a48ce426a0 0x55a48ce44ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:24.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.029+0000 7f07c5d11640 1 --2- 172.21.15.28:0/2289583383 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a48ce426a0 0x55a48ce44ab0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46b058832c45edf9 server_cookie=950087e757b523cd in_seq=0 out_seq=0 2024-04-21T08:55:24.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.030+0000 7f07a97fa640 10 client.26318 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:24.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.032+0000 7f07a97fa640 1 -- 172.21.15.28:0/2289583383 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55a48cd1da70 con 0x55a48ce426a0 2024-04-21T08:55:24.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.033+0000 7f07a97fa640 10 client.26318 handle_command_reply: tid=0 2024-04-21T08:55:24.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.033+0000 7f07cab63740 1 client.26318 shutdown 2024-04-21T08:55:24.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.034+0000 7f07cab63740 1 -- 172.21.15.28:0/2289583383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 msgr2=0x55a48cdc2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:24.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.034+0000 7f07cab63740 1 --2- 172.21.15.28:0/2289583383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 0x55a48cdc2cb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f07b0018010 tx=0x7f07b0007dc0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.038+0000 7f07cab63740 1 -- 172.21.15.28:0/2289583383 shutdown_connections 2024-04-21T08:55:24.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.038+0000 7f07cab63740 1 --2- 172.21.15.28:0/2289583383 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a48ce426a0 0x55a48ce44ab0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.038+0000 7f07cab63740 1 --2- 172.21.15.28:0/2289583383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a48cdbaa80 0x55a48cdb40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.038+0000 7f07cab63740 1 --2- 172.21.15.28:0/2289583383 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a48cdb1ea0 0x55a48cdc31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.038+0000 7f07cab63740 1 --2- 172.21.15.28:0/2289583383 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a48cdb1580 0x55a48cdc2cb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.038+0000 7f07cab63740 1 -- 172.21.15.28:0/2289583383 >> 172.21.15.28:0/2289583383 conn(0x55a48cdabcb0 msgr2=0x55a48cdaccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:24.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.038+0000 7f07a97fa640 0 client.26318 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:24.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.042+0000 7f07cab63740 1 -- 172.21.15.28:0/2289583383 shutdown_connections 2024-04-21T08:55:24.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.044+0000 7f07cab63740 1 -- 172.21.15.28:0/2289583383 wait complete. 2024-04-21T08:55:24.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.044+0000 7f07cab63740 20 client.26318 trim_cache size 0 max 16384 2024-04-21T08:55:24.045 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:55:24.045 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (2 inodes in the stack)", 2024-04-21T08:55:24.045 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:55:24.045 INFO:teuthology.orchestra.run.smithi028.stdout: "0acf896b-0514-4934-9514-30a1c103351c": { 2024-04-21T08:55:24.045 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:55:24.045 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "0acf896b-0514-4934-9514-30a1c103351c", 2024-04-21T08:55:24.045 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:55:24.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:55:24.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:55:24.046 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:55:24.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.047+0000 7f07c7f75640 1 -- 172.21.15.28:0/1753979393 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f07ac0a3ea0 msgr2=0x7f07ac0a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:24.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.047+0000 7f07c7f75640 1 --2- 172.21.15.28:0/1753979393 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f07ac0a3ea0 0x7f07ac0a6360 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f07bc004a90 tx=0x7f07bc004980 comp rx=0 tx=0).stop 2024-04-21T08:55:24.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.047+0000 7f07c7f75640 1 -- 172.21.15.28:0/1753979393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07c011b730 msgr2=0x7f07c01abf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:24.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.047+0000 7f07c7f75640 1 --2- 172.21.15.28:0/1753979393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07c011b730 0x7f07c01abf70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f07bc00e940 tx=0x7f07bc00ee00 comp rx=0 tx=0).stop 2024-04-21T08:55:24.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.047+0000 7f07c7f75640 1 -- 172.21.15.28:0/1753979393 shutdown_connections 2024-04-21T08:55:24.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.047+0000 7f07c7f75640 1 --2- 172.21.15.28:0/1753979393 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f07ac0a3ea0 0x7f07ac0a6360 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.047+0000 7f07c7f75640 1 --2- 172.21.15.28:0/1753979393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07c011b730 0x7f07c01abf70 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.047+0000 7f07c7f75640 1 --2- 172.21.15.28:0/1753979393 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f07c01106d0 0x7f07c01a7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.047+0000 7f07c7f75640 1 --2- 172.21.15.28:0/1753979393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c010e7b0 0x7f07c01a76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:24.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.047+0000 7f07c7f75640 1 -- 172.21.15.28:0/1753979393 >> 172.21.15.28:0/1753979393 conn(0x7f07c006d070 msgr2=0x7f07c0070a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:24.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.050+0000 7f07c7f75640 1 -- 172.21.15.28:0/1753979393 shutdown_connections 2024-04-21T08:55:24.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:24.051+0000 7f07c7f75640 1 -- 172.21.15.28:0/1753979393 wait complete. 2024-04-21T08:55:24.071 INFO:tasks.cephfs.filesystem:scrub status for tag:0acf896b-0514-4934-9514-30a1c103351c - {'path': '/', 'tag': '0acf896b-0514-4934-9514-30a1c103351c', 'options': 'recursive,force'} 2024-04-21T08:55:24.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:24 smithi152 ceph-mon[31401]: pgmap v129: 129 pgs: 129 active+clean; 15 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 61 MiB/s wr, 8.84k op/s 2024-04-21T08:55:24.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:24 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:55:24.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:24 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:24.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:24 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:55:24.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:24 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:55:24.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:24 smithi028 ceph-mon[28003]: pgmap v129: 129 pgs: 129 active+clean; 15 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 61 MiB/s wr, 8.84k op/s 2024-04-21T08:55:24.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:24 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:55:24.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:24 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:24.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:24 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:55:24.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:24 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:55:24.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:24 smithi171 ceph-mon[31358]: pgmap v129: 129 pgs: 129 active+clean; 15 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 61 MiB/s wr, 8.84k op/s 2024-04-21T08:55:24.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:24 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:55:24.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:24 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:24.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:24 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:55:24.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:24 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:55:25.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:25 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:55:25.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:25 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:55:25.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:25 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:55:26.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:26 smithi152 ceph-mon[31401]: pgmap v130: 129 pgs: 129 active+clean; 16 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 61 MiB/s wr, 8.81k op/s 2024-04-21T08:55:26.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:26 smithi028 ceph-mon[28003]: pgmap v130: 129 pgs: 129 active+clean; 16 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 61 MiB/s wr, 8.81k op/s 2024-04-21T08:55:26.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:26 smithi171 ceph-mon[31358]: pgmap v130: 129 pgs: 129 active+clean; 16 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 61 MiB/s wr, 8.81k op/s 2024-04-21T08:55:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.168+0000 7f7bdceb3640 1 -- 172.21.15.28:0/2270489080 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bd811b730 msgr2=0x7f7bd811db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.168+0000 7f7bdceb3640 1 --2- 172.21.15.28:0/2270489080 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bd811b730 0x7f7bd811db20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0009cd0 tx=0x7f7bd002c190 comp rx=0 tx=0).stop 2024-04-21T08:55:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.168+0000 7f7bdceb3640 1 -- 172.21.15.28:0/2270489080 shutdown_connections 2024-04-21T08:55:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.168+0000 7f7bdceb3640 1 --2- 172.21.15.28:0/2270489080 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bd811b730 0x7f7bd811db20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.168+0000 7f7bdceb3640 1 --2- 172.21.15.28:0/2270489080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd8110b20 0x7f7bd8109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.168+0000 7f7bdceb3640 1 --2- 172.21.15.28:0/2270489080 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bd8110740 0x7f7bd8108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.168+0000 7f7bdceb3640 1 -- 172.21.15.28:0/2270489080 >> 172.21.15.28:0/2270489080 conn(0x7f7bd806d020 msgr2=0x7f7bd806d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.169+0000 7f7bdceb3640 1 -- 172.21.15.28:0/2270489080 shutdown_connections 2024-04-21T08:55:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.169+0000 7f7bdceb3640 1 -- 172.21.15.28:0/2270489080 wait complete. 2024-04-21T08:55:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.170+0000 7f7bdceb3640 1 Processor -- start 2024-04-21T08:55:28.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.170+0000 7f7bdceb3640 1 -- start start 2024-04-21T08:55:28.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.171+0000 7f7bdceb3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd8110740 0x7f7bd81a75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.171+0000 7f7bdceb3640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bd811b730 0x7f7bd81a7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.171+0000 7f7bdceb3640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bd81abe80 0x7f7bd81ac330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.171+0000 7f7bdceb3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd8121060 con 0x7f7bd8110740 2024-04-21T08:55:28.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.171+0000 7f7bdceb3640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd8120ee0 con 0x7f7bd811b730 2024-04-21T08:55:28.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.171+0000 7f7bdceb3640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd81211e0 con 0x7f7bd81abe80 2024-04-21T08:55:28.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.171+0000 7f7bd659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd8110740 0x7f7bd81a75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:28.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.171+0000 7f7bd659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd8110740 0x7f7bd81a75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47446/0 (socket says 172.21.15.28:47446) 2024-04-21T08:55:28.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.171+0000 7f7bd659c640 1 -- 172.21.15.28:0/1169234666 learned_addr learned my addr 172.21.15.28:0/1169234666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:28.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.171+0000 7f7bd6d9d640 1 --2- 172.21.15.28:0/1169234666 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bd81abe80 0x7f7bd81ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:28.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.172+0000 7f7bd659c640 1 -- 172.21.15.28:0/1169234666 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bd81abe80 msgr2=0x7f7bd81ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:28.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.172+0000 7f7bd659c640 1 --2- 172.21.15.28:0/1169234666 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bd81abe80 0x7f7bd81ac330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.172+0000 7f7bd659c640 1 -- 172.21.15.28:0/1169234666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bd811b730 msgr2=0x7f7bd81a7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:28.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.172+0000 7f7bd659c640 1 --2- 172.21.15.28:0/1169234666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bd811b730 0x7f7bd81a7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.172+0000 7f7bd659c640 1 -- 172.21.15.28:0/1169234666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd81ac970 con 0x7f7bd8110740 2024-04-21T08:55:28.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.174+0000 7f7bd659c640 1 --2- 172.21.15.28:0/1169234666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd8110740 0x7f7bd81a75c0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800d950 tx=0x7f7bc800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:28.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.175+0000 7f7bc77fe640 1 -- 172.21.15.28:0/1169234666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8017070 con 0x7f7bd8110740 2024-04-21T08:55:28.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.175+0000 7f7bdceb3640 1 -- 172.21.15.28:0/1169234666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bd81acc60 con 0x7f7bd8110740 2024-04-21T08:55:28.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.175+0000 7f7bdceb3640 1 -- 172.21.15.28:0/1169234666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bd8111280 con 0x7f7bd8110740 2024-04-21T08:55:28.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.175+0000 7f7bc77fe640 1 -- 172.21.15.28:0/1169234666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bc8009c30 con 0x7f7bd8110740 2024-04-21T08:55:28.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.175+0000 7f7bc77fe640 1 -- 172.21.15.28:0/1169234666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc800f3c0 con 0x7f7bd8110740 2024-04-21T08:55:28.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.180+0000 7f7bdfb7d740 1 Processor -- start 2024-04-21T08:55:28.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.182+0000 7f7bc77fe640 1 -- 172.21.15.28:0/1169234666 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7bc800f680 con 0x7f7bd8110740 2024-04-21T08:55:28.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.183+0000 7f7bc77fe640 1 --2- 172.21.15.28:0/1169234666 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f7bc00a3f10 0x7f7bc00a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.183+0000 7f7bc77fe640 1 -- 172.21.15.28:0/1169234666 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f7bc80d0680 con 0x7f7bd8110740 2024-04-21T08:55:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.184+0000 7f7bdfb7d740 1 -- start start 2024-04-21T08:55:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.184+0000 7f7bdfb7d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 0x560b4fbee620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.184+0000 7f7bdfb7d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbeeb60 0x560b4fbeefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.184+0000 7f7bdfb7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf80e0 0x560b4fbfa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.184+0000 7f7bdfb7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560b4fb538d0 con 0x560b4fbf80e0 2024-04-21T08:55:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.184+0000 7f7bdfb7d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x560b4faf0e80 con 0x560b4fbeeb60 2024-04-21T08:55:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.184+0000 7f7bdfb7d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x560b4fb4f890 con 0x560b4fbef8d0 2024-04-21T08:55:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.184+0000 7f7bd6d9d640 1 --2- 172.21.15.28:0/1169234666 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f7bc00a3f10 0x7f7bc00a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.184+0000 7f7bd659c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 0x560b4fbee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:28.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.184+0000 7f7bd659c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 0x560b4fbee620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:58166/0 (socket says 172.21.15.28:58166) 2024-04-21T08:55:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.185+0000 7f7bd659c640 1 -- 172.21.15.28:0/1260997732 learned_addr learned my addr 172.21.15.28:0/1260997732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.185+0000 7f7bd659c640 1 -- 172.21.15.28:0/1260997732 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbeeb60 msgr2=0x560b4fbeefe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.185+0000 7f7bd659c640 1 --2- 172.21.15.28:0/1260997732 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbeeb60 0x560b4fbeefe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.185+0000 7f7bd659c640 1 -- 172.21.15.28:0/1260997732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf80e0 msgr2=0x560b4fbfa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.185+0000 7f7bd659c640 1 --2- 172.21.15.28:0/1260997732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf80e0 0x560b4fbfa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.185+0000 7f7bd659c640 1 -- 172.21.15.28:0/1260997732 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b4fb12320 con 0x560b4fbef8d0 2024-04-21T08:55:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.185+0000 7f7bd659c640 1 --2- 172.21.15.28:0/1260997732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 0x560b4fbee620 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7bc80990c0 tx=0x7f7bc809c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3447294606b17928 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.185+0000 7f7bc4ff9640 1 -- 172.21.15.28:0/1260997732 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8020040 con 0x560b4fbef8d0 2024-04-21T08:55:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.186+0000 7f7bc4ff9640 1 -- 172.21.15.28:0/1260997732 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bc809c6f0 con 0x560b4fbef8d0 2024-04-21T08:55:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.187+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/1260997732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 msgr2=0x560b4fbee620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:28.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.187+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/1260997732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 0x560b4fbee620 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7bc80990c0 tx=0x7f7bc809c360 comp rx=0 tx=0).stop 2024-04-21T08:55:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.188+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/1260997732 shutdown_connections 2024-04-21T08:55:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.188+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/1260997732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf80e0 0x560b4fbfa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.188+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/1260997732 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbeeb60 0x560b4fbeefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.188+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/1260997732 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 0x560b4fbee620 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.188+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/1260997732 >> 172.21.15.28:0/1260997732 conn(0x560b4fbb5c90 msgr2=0x560b4fbe9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:28.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.188+0000 7f7bd6d9d640 1 --2- 172.21.15.28:0/1169234666 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f7bc00a3f10 0x7f7bc00a63d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0037040 tx=0x7f7bd00323a0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:28.197 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.195+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/1260997732 shutdown_connections 2024-04-21T08:55:28.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.197+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/1260997732 wait complete. 2024-04-21T08:55:28.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.197+0000 7f7bdfb7d740 1 -- start start 2024-04-21T08:55:28.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.198+0000 7f7bdfb7d740 1 Processor -- start 2024-04-21T08:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.200+0000 7f7bdfb7d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 0x560b4fbf79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.200+0000 7f7bdfb7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf80e0 0x560b4fbf4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.200+0000 7f7bdfb7d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbf51e0 0x560b4fbf5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.200+0000 7f7bdfb7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560b4fbfae50 con 0x560b4fbf80e0 2024-04-21T08:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.200+0000 7f7bdfb7d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x560b4fadbeb0 con 0x560b4fbf51e0 2024-04-21T08:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.200+0000 7f7bdfb7d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x560b4fbfafd0 con 0x560b4fbef8d0 2024-04-21T08:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.200+0000 7f7bdfb7d740 10 client.0 _ll_register_callbacks cb 0x560b4fb08b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:28.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.202+0000 7f7bd5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf80e0 0x560b4fbf4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.202+0000 7f7bd5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf80e0 0x560b4fbf4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47458/0 (socket says 172.21.15.28:47458) 2024-04-21T08:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.202+0000 7f7bd5d9b640 1 -- 172.21.15.28:0/2140164286 learned_addr learned my addr 172.21.15.28:0/2140164286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.202+0000 7f7bd5d9b640 1 -- 172.21.15.28:0/2140164286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 msgr2=0x560b4fbf79b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.202+0000 7f7bd5d9b640 1 --2- 172.21.15.28:0/2140164286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 0x560b4fbf79b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.202+0000 7f7bd5d9b640 1 -- 172.21.15.28:0/2140164286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbf51e0 msgr2=0x560b4fbf5660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.202+0000 7f7bd5d9b640 1 --2- 172.21.15.28:0/2140164286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbf51e0 0x560b4fbf5660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.202+0000 7f7bd5d9b640 1 -- 172.21.15.28:0/2140164286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b4f866210 con 0x560b4fbf80e0 2024-04-21T08:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.203+0000 7f7bd5d9b640 1 --2- 172.21.15.28:0/2140164286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf80e0 0x560b4fbf4ca0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009d60 tx=0x7f7bcc00b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb9c7d112f7be2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.204+0000 7f7bbdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.204+0000 7f7bbdffb640 1 -- 172.21.15.28:0/2140164286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc015020 con 0x560b4fbf80e0 2024-04-21T08:55:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.204+0000 7f7bbdffb640 1 -- 172.21.15.28:0/2140164286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f7bcc011420 con 0x560b4fbf80e0 2024-04-21T08:55:28.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.204+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/2140164286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bc8098ef0 con 0x560b4fbf80e0 2024-04-21T08:55:28.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.205+0000 7f7bbdffb640 1 -- 172.21.15.28:0/2140164286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7bcc011620 con 0x560b4fbf80e0 2024-04-21T08:55:28.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.207+0000 7f7bdfb7d740 10 client.17052 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:28.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.207+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/2140164286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560b4f932620 con 0x560b4fbf80e0 2024-04-21T08:55:28.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.208+0000 7f7bbdffb640 1 -- 172.21.15.28:0/2140164286 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bc8098ef0 con 0x560b4fbf80e0 2024-04-21T08:55:28.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.208+0000 7f7bbdffb640 1 -- 172.21.15.28:0/2140164286 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f7bcc0199b0 con 0x560b4fbf80e0 2024-04-21T08:55:28.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.208+0000 7f7bdfb7d740 10 client.17052 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:28.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.208+0000 7f7bdfb7d740 10 client.17052 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:28.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.208+0000 7f7bdfb7d740 20 client.17052 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:28.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.208+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/2140164286 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560b4fc08a30 0x560b4fc0aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.208+0000 7f7bdfb7d740 4 client.17052 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:28.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.208+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/2140164286 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b4fb67880 con 0x560b4fc08a30 2024-04-21T08:55:28.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.208+0000 7f7bd659c640 1 --2- 172.21.15.28:0/2140164286 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560b4fc08a30 0x560b4fc0aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:28.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.209+0000 7f7bbdffb640 10 client.17052 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:28.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.209+0000 7f7bd659c640 1 --2- 172.21.15.28:0/2140164286 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560b4fc08a30 0x560b4fc0aeb0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=694f147668564205 server_cookie=2258501a9e855db0 in_seq=0 out_seq=0 2024-04-21T08:55:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.213+0000 7f7bbdffb640 1 -- 172.21.15.28:0/2140164286 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560b4fb67880 con 0x560b4fc08a30 2024-04-21T08:55:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.213+0000 7f7bbdffb640 10 client.17052 handle_command_reply: tid=0 2024-04-21T08:55:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.213+0000 7f7bdfb7d740 1 client.17052 shutdown 2024-04-21T08:55:28.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.214+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/2140164286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf80e0 msgr2=0x560b4fbf4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:28.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.214+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/2140164286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf80e0 0x560b4fbf4ca0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009d60 tx=0x7f7bcc00b4b0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.215+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/2140164286 shutdown_connections 2024-04-21T08:55:28.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.215+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/2140164286 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560b4fc08a30 0x560b4fc0aeb0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.215+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/2140164286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbf51e0 0x560b4fbf5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.215+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/2140164286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf80e0 0x560b4fbf4ca0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.215+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/2140164286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 0x560b4fbf79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.215+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/2140164286 >> 172.21.15.28:0/2140164286 conn(0x560b4fbb5c90 msgr2=0x560b4fbeb5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:28.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.215+0000 7f7bbdffb640 0 client.17052 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:28.216 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.217+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/2140164286 shutdown_connections 2024-04-21T08:55:28.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.218+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/2140164286 wait complete. 2024-04-21T08:55:28.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.218+0000 7f7bdfb7d740 20 client.17052 trim_cache size 0 max 16384 2024-04-21T08:55:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.249+0000 7f7bdfb7d740 1 Processor -- start 2024-04-21T08:55:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.249+0000 7f7bdfb7d740 1 -- start start 2024-04-21T08:55:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.251+0000 7f7bdfb7d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 0x560b4fc03e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.251+0000 7f7bdfb7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf51e0 0x560b4fbe4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.251+0000 7f7bdfb7d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbf80e0 0x560b4fbf74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.251+0000 7f7bdfb7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb8034920 con 0x560b4fbf51e0 2024-04-21T08:55:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.251+0000 7f7bdfb7d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb80321f0 con 0x560b4fbf80e0 2024-04-21T08:55:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.251+0000 7f7bdfb7d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb8030040 con 0x560b4fbef8d0 2024-04-21T08:55:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.251+0000 7f7bd659c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 0x560b4fc03e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.251+0000 7f7bd659c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 0x560b4fc03e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:58190/0 (socket says 172.21.15.28:58190) 2024-04-21T08:55:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.251+0000 7f7bd659c640 1 -- 172.21.15.28:0/633204303 learned_addr learned my addr 172.21.15.28:0/633204303 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.251+0000 7f7bd5d9b640 1 --2- 172.21.15.28:0/633204303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf51e0 0x560b4fbe4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.251+0000 7f7bd659c640 1 -- 172.21.15.28:0/633204303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbf80e0 msgr2=0x560b4fbf74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.251+0000 7f7bd659c640 1 --2- 172.21.15.28:0/633204303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbf80e0 0x560b4fbf74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.251+0000 7f7bd659c640 1 -- 172.21.15.28:0/633204303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf51e0 msgr2=0x560b4fbe4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.251+0000 7f7bd659c640 1 --2- 172.21.15.28:0/633204303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf51e0 0x560b4fbe4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.251+0000 7f7bd659c640 1 -- 172.21.15.28:0/633204303 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b4fbe4f30 con 0x560b4fbef8d0 2024-04-21T08:55:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.252+0000 7f7bd659c640 1 --2- 172.21.15.28:0/633204303 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 0x560b4fc03e50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8000c00 tx=0x7f7bc800bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b1bf8b5c65b747b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.252+0000 7f7bbffff640 1 -- 172.21.15.28:0/633204303 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc80ba070 con 0x560b4fbef8d0 2024-04-21T08:55:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.252+0000 7f7bbffff640 1 -- 172.21.15.28:0/633204303 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bc80cde50 con 0x560b4fbef8d0 2024-04-21T08:55:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.252+0000 7f7bbffff640 1 -- 172.21.15.28:0/633204303 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8020040 con 0x560b4fbef8d0 2024-04-21T08:55:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.253+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/633204303 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 msgr2=0x560b4fc03e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.253+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/633204303 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 0x560b4fc03e50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8000c00 tx=0x7f7bc800bf60 comp rx=0 tx=0).stop 2024-04-21T08:55:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.258+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/633204303 shutdown_connections 2024-04-21T08:55:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.258+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/633204303 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbf80e0 0x560b4fbf74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.258+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/633204303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbf51e0 0x560b4fbe4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.258+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/633204303 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbef8d0 0x560b4fc03e50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.258+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/633204303 >> 172.21.15.28:0/633204303 conn(0x560b4fbb5c90 msgr2=0x560b4fbee080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.260+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/633204303 shutdown_connections 2024-04-21T08:55:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.260+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/633204303 wait complete. 2024-04-21T08:55:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.261+0000 7f7bdfb7d740 1 -- start start 2024-04-21T08:55:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.261+0000 7f7bdfb7d740 1 Processor -- start 2024-04-21T08:55:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.262+0000 7f7bdfb7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbef8d0 0x560b4fc00550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.262+0000 7f7bdfb7d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbf51e0 0x560b4fbf1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.262+0000 7f7bdfb7d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbf80e0 0x560b4fbf24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.262+0000 7f7bdfb7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb80306b0 con 0x560b4fbef8d0 2024-04-21T08:55:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.263+0000 7f7bdfb7d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb8031b20 con 0x560b4fbf51e0 2024-04-21T08:55:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.263+0000 7f7bdfb7d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb8030d80 con 0x560b4fbf80e0 2024-04-21T08:55:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.263+0000 7f7bd6d9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbf80e0 0x560b4fbf24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.263+0000 7f7bd6d9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbf80e0 0x560b4fbf24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:58194/0 (socket says 172.21.15.28:58194) 2024-04-21T08:55:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.263+0000 7f7bd6d9d640 1 -- 172.21.15.28:0/1300451326 learned_addr learned my addr 172.21.15.28:0/1300451326 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.263+0000 7f7bdfb7d740 10 client.0 _ll_register_callbacks cb 0x7f7bcc006eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.263+0000 7f7bd6d9d640 1 -- 172.21.15.28:0/1300451326 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbf51e0 msgr2=0x560b4fbf1f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.263+0000 7f7bd6d9d640 1 --2- 172.21.15.28:0/1300451326 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbf51e0 0x560b4fbf1f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.263+0000 7f7bd6d9d640 1 -- 172.21.15.28:0/1300451326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbef8d0 msgr2=0x560b4fc00550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.263+0000 7f7bd6d9d640 1 --2- 172.21.15.28:0/1300451326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbef8d0 0x560b4fc00550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.263+0000 7f7bd6d9d640 1 -- 172.21.15.28:0/1300451326 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b4fbf2a10 con 0x560b4fbf80e0 2024-04-21T08:55:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.264+0000 7f7bd6d9d640 1 --2- 172.21.15.28:0/1300451326 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbf80e0 0x560b4fbf24a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0009cd0 tx=0x7f7bd0032a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9735f983544ad129 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.264+0000 7f7bbdffb640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:55:28.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.267+0000 7f7bbdffb640 1 -- 172.21.15.28:0/1300451326 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd004f030 con 0x560b4fbf80e0 2024-04-21T08:55:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.268+0000 7f7bbdffb640 1 -- 172.21.15.28:0/1300451326 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f7bd0032e30 con 0x560b4fbf80e0 2024-04-21T08:55:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.268+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/1300451326 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bc80cde50 con 0x560b4fbf80e0 2024-04-21T08:55:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.273+0000 7f7bbdffb640 1 -- 172.21.15.28:0/1300451326 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7bd0039040 con 0x560b4fbf80e0 2024-04-21T08:55:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.275+0000 7f7bdfb7d740 10 client.26335 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.275+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/1300451326 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560b4fbf92f0 con 0x560b4fbf80e0 2024-04-21T08:55:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.277+0000 7f7bbdffb640 1 -- 172.21.15.28:0/1300451326 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bc80cde50 con 0x560b4fbf80e0 2024-04-21T08:55:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.277+0000 7f7bbdffb640 1 -- 172.21.15.28:0/1300451326 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f7bd003f070 con 0x560b4fbf80e0 2024-04-21T08:55:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.277+0000 7f7bdfb7d740 10 client.26335 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.277+0000 7f7bdfb7d740 10 client.26335 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.277+0000 7f7bdfb7d740 20 client.26335 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.277+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/1300451326 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560b4fc6f580 0x560b4fc719c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.278+0000 7f7bdfb7d740 4 client.26335 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:55:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.278+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/1300451326 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560b4faf33d0 con 0x560b4fc6f580 2024-04-21T08:55:28.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.280+0000 7f7bd659c640 1 --2- 172.21.15.28:0/1300451326 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560b4fc6f580 0x560b4fc719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.283+0000 7f7bd659c640 1 --2- 172.21.15.28:0/1300451326 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560b4fc6f580 0x560b4fc719c0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30d8c76e8cb7e6d8 server_cookie=204a6606e7b775cc in_seq=0 out_seq=0 2024-04-21T08:55:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.283+0000 7f7bbdffb640 10 client.26335 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.799+0000 7f7bbdffb640 1 -- 172.21.15.28:0/1300451326 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560b4faf33d0 con 0x560b4fc6f580 2024-04-21T08:55:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.799+0000 7f7bbdffb640 10 client.26335 handle_command_reply: tid=0 2024-04-21T08:55:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.799+0000 7f7bdfb7d740 1 client.26335 shutdown 2024-04-21T08:55:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.799+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/1300451326 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbf80e0 msgr2=0x560b4fbf24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:28.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.799+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/1300451326 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbf80e0 0x560b4fbf24a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0009cd0 tx=0x7f7bd0032a70 comp rx=0 tx=0).stop 2024-04-21T08:55:28.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.800+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/1300451326 shutdown_connections 2024-04-21T08:55:28.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.800+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/1300451326 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560b4fc6f580 0x560b4fc719c0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.800+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/1300451326 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560b4fbf80e0 0x560b4fbf24a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.800+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/1300451326 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560b4fbf51e0 0x560b4fbf1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.800+0000 7f7bdfb7d740 1 --2- 172.21.15.28:0/1300451326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b4fbef8d0 0x560b4fc00550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.800+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/1300451326 >> 172.21.15.28:0/1300451326 conn(0x560b4fbb5c90 msgr2=0x560b4fbf3570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:28.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.800+0000 7f7bbdffb640 0 client.26335 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:28.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.800+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/1300451326 shutdown_connections 2024-04-21T08:55:28.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.801+0000 7f7bdfb7d740 1 -- 172.21.15.28:0/1300451326 wait complete. 2024-04-21T08:55:28.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.801+0000 7f7bdfb7d740 20 client.26335 trim_cache size 0 max 16384 2024-04-21T08:55:28.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:55:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:55:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:55:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:55:28.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.802+0000 7f7bdceb3640 1 -- 172.21.15.28:0/1169234666 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f7bc00a3f10 msgr2=0x7f7bc00a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:28.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.802+0000 7f7bdceb3640 1 --2- 172.21.15.28:0/1169234666 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f7bc00a3f10 0x7f7bc00a63d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7bd0037040 tx=0x7f7bd00323a0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.802+0000 7f7bdceb3640 1 -- 172.21.15.28:0/1169234666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd8110740 msgr2=0x7f7bd81a75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:28.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.802+0000 7f7bdceb3640 1 --2- 172.21.15.28:0/1169234666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd8110740 0x7f7bd81a75c0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800d950 tx=0x7f7bc800de10 comp rx=0 tx=0).stop 2024-04-21T08:55:28.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.803+0000 7f7bdceb3640 1 -- 172.21.15.28:0/1169234666 shutdown_connections 2024-04-21T08:55:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.803+0000 7f7bdceb3640 1 --2- 172.21.15.28:0/1169234666 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f7bc00a3f10 0x7f7bc00a63d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.803+0000 7f7bdceb3640 1 --2- 172.21.15.28:0/1169234666 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bd81abe80 0x7f7bd81ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.803+0000 7f7bdceb3640 1 --2- 172.21.15.28:0/1169234666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7bd811b730 0x7f7bd81a7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.803+0000 7f7bdceb3640 1 --2- 172.21.15.28:0/1169234666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bd8110740 0x7f7bd81a75c0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.803+0000 7f7bdceb3640 1 -- 172.21.15.28:0/1169234666 >> 172.21.15.28:0/1169234666 conn(0x7f7bd806d020 msgr2=0x7f7bd810e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.803+0000 7f7bdceb3640 1 -- 172.21.15.28:0/1169234666 shutdown_connections 2024-04-21T08:55:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:28.803+0000 7f7bdceb3640 1 -- 172.21.15.28:0/1169234666 wait complete. 2024-04-21T08:55:28.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:28 smithi152 ceph-mon[31401]: pgmap v131: 129 pgs: 129 active+clean; 16 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 56 MiB/s wr, 8.17k op/s 2024-04-21T08:55:28.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:28 smithi028 ceph-mon[28003]: pgmap v131: 129 pgs: 129 active+clean; 16 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 56 MiB/s wr, 8.17k op/s 2024-04-21T08:55:28.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:28 smithi171 ceph-mon[31358]: pgmap v131: 129 pgs: 129 active+clean; 16 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 56 MiB/s wr, 8.17k op/s 2024-04-21T08:55:30.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:30 smithi152 ceph-mon[31401]: pgmap v132: 129 pgs: 129 active+clean; 16 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 66 MiB/s wr, 9.97k op/s 2024-04-21T08:55:30.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:30 smithi028 ceph-mon[28003]: pgmap v132: 129 pgs: 129 active+clean; 16 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 66 MiB/s wr, 9.97k op/s 2024-04-21T08:55:30.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:30 smithi171 ceph-mon[31358]: pgmap v132: 129 pgs: 129 active+clean; 16 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 66 MiB/s wr, 9.97k op/s 2024-04-21T08:55:32.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:32 smithi152 ceph-mon[31401]: pgmap v133: 129 pgs: 129 active+clean; 16 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 56 MiB/s wr, 8.75k op/s 2024-04-21T08:55:32.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:32 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:55:32.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:32 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:32.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:32 smithi028 ceph-mon[28003]: pgmap v133: 129 pgs: 129 active+clean; 16 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 56 MiB/s wr, 8.75k op/s 2024-04-21T08:55:32.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:32 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:55:32.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:32 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:32.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:32 smithi171 ceph-mon[31358]: pgmap v133: 129 pgs: 129 active+clean; 16 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 56 MiB/s wr, 8.75k op/s 2024-04-21T08:55:32.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:32 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:55:32.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:32 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:33.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:33 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:55:33.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:33 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:55:33.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:33 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:55:33.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:33 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:55:33.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:33 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:55:33.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:33 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:55:33.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:33 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:55:33.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:33 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:55:33.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:33 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:55:34.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:34 smithi152 ceph-mon[31401]: pgmap v134: 129 pgs: 129 active+clean; 16 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 53 MiB/s wr, 8.40k op/s 2024-04-21T08:55:34.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:34 smithi028 ceph-mon[28003]: pgmap v134: 129 pgs: 129 active+clean; 16 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 53 MiB/s wr, 8.40k op/s 2024-04-21T08:55:34.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:34 smithi171 ceph-mon[31358]: pgmap v134: 129 pgs: 129 active+clean; 16 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 53 MiB/s wr, 8.40k op/s 2024-04-21T08:55:35.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:35 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:35.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:35 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:35.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:35 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:36.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:36 smithi152 ceph-mon[31401]: pgmap v135: 129 pgs: 129 active+clean; 16 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 53 MiB/s wr, 8.72k op/s 2024-04-21T08:55:36.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:36 smithi028 ceph-mon[28003]: pgmap v135: 129 pgs: 129 active+clean; 16 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 53 MiB/s wr, 8.72k op/s 2024-04-21T08:55:36.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:36 smithi171 ceph-mon[31358]: pgmap v135: 129 pgs: 129 active+clean; 16 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 53 MiB/s wr, 8.72k op/s 2024-04-21T08:55:38.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:38 smithi152 ceph-mon[31401]: pgmap v136: 129 pgs: 129 active+clean; 16 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 47 MiB/s wr, 7.82k op/s 2024-04-21T08:55:38.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:38 smithi028 ceph-mon[28003]: pgmap v136: 129 pgs: 129 active+clean; 16 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 47 MiB/s wr, 7.82k op/s 2024-04-21T08:55:38.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:38 smithi171 ceph-mon[31358]: pgmap v136: 129 pgs: 129 active+clean; 16 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 47 MiB/s wr, 7.82k op/s 2024-04-21T08:55:39.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:39 smithi152 ceph-mon[31401]: pgmap v137: 129 pgs: 129 active+clean; 16 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 50 MiB/s wr, 8.68k op/s 2024-04-21T08:55:39.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:39 smithi028 ceph-mon[28003]: pgmap v137: 129 pgs: 129 active+clean; 16 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 50 MiB/s wr, 8.68k op/s 2024-04-21T08:55:39.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:39 smithi171 ceph-mon[31358]: pgmap v137: 129 pgs: 129 active+clean; 16 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 50 MiB/s wr, 8.68k op/s 2024-04-21T08:55:42.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.136+0000 7fcf012c8640 1 -- 172.21.15.28:0/4263092388 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcefc1010f0 msgr2=0x7fcefc1014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.136+0000 7fcf012c8640 1 --2- 172.21.15.28:0/4263092388 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcefc1010f0 0x7fcefc1014d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcef4009ca0 tx=0x7fcef402bdf0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.138+0000 7fcf012c8640 1 -- 172.21.15.28:0/4263092388 shutdown_connections 2024-04-21T08:55:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.138+0000 7fcf012c8640 1 --2- 172.21.15.28:0/4263092388 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcefc106170 0x7fcefc10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.138+0000 7fcf012c8640 1 --2- 172.21.15.28:0/4263092388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc101aa0 0x7fcefc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.138+0000 7fcf012c8640 1 --2- 172.21.15.28:0/4263092388 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcefc1010f0 0x7fcefc1014d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.138+0000 7fcf012c8640 1 -- 172.21.15.28:0/4263092388 >> 172.21.15.28:0/4263092388 conn(0x7fcefc077fd0 msgr2=0x7fcefc0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.138+0000 7fcf012c8640 1 -- 172.21.15.28:0/4263092388 shutdown_connections 2024-04-21T08:55:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.139+0000 7fcf012c8640 1 -- 172.21.15.28:0/4263092388 wait complete. 2024-04-21T08:55:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.139+0000 7fcf012c8640 1 Processor -- start 2024-04-21T08:55:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.140+0000 7fcf012c8640 1 -- start start 2024-04-21T08:55:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.140+0000 7fcf012c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc1010f0 0x7fcefc073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.140+0000 7fcf012c8640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcefc106170 0x7fcefc06c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.140+0000 7fcf012c8640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcefc06ce20 0x7fcefc06d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.140+0000 7fcf012c8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcefc10ffd0 con 0x7fcefc1010f0 2024-04-21T08:55:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.140+0000 7fcf012c8640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fcefc10fe50 con 0x7fcefc106170 2024-04-21T08:55:42.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.140+0000 7fcf012c8640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcefc110150 con 0x7fcefc06ce20 2024-04-21T08:55:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.141+0000 7fcefb7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcefc06ce20 0x7fcefc06d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.142+0000 7fcefaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc1010f0 0x7fcefc073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.142+0000 7fcefaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc1010f0 0x7fcefc073850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44172/0 (socket says 172.21.15.28:44172) 2024-04-21T08:55:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.142+0000 7fcefaffd640 1 -- 172.21.15.28:0/3279567911 learned_addr learned my addr 172.21.15.28:0/3279567911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.142+0000 7fcefa7fc640 1 --2- 172.21.15.28:0/3279567911 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcefc106170 0x7fcefc06c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.143+0000 7fcefaffd640 1 -- 172.21.15.28:0/3279567911 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcefc06ce20 msgr2=0x7fcefc06d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.143+0000 7fcefaffd640 1 --2- 172.21.15.28:0/3279567911 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcefc06ce20 0x7fcefc06d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.143+0000 7fcefaffd640 1 -- 172.21.15.28:0/3279567911 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcefc106170 msgr2=0x7fcefc06c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.143+0000 7fcefaffd640 1 --2- 172.21.15.28:0/3279567911 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcefc106170 0x7fcefc06c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.143+0000 7fcefaffd640 1 -- 172.21.15.28:0/3279567911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcefc071370 con 0x7fcefc1010f0 2024-04-21T08:55:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.147+0000 7fcefaffd640 1 --2- 172.21.15.28:0/3279567911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc1010f0 0x7fcefc073850 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fcef4005e30 tx=0x7fcef4003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.150+0000 7fcee3fff640 1 -- 172.21.15.28:0/3279567911 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4039040 con 0x7fcefc1010f0 2024-04-21T08:55:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.150+0000 7fcee3fff640 1 -- 172.21.15.28:0/3279567911 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcef4044070 con 0x7fcefc1010f0 2024-04-21T08:55:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.150+0000 7fcee3fff640 1 -- 172.21.15.28:0/3279567911 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4034440 con 0x7fcefc1010f0 2024-04-21T08:55:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.151+0000 7fcf012c8640 1 -- 172.21.15.28:0/3279567911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcefc071600 con 0x7fcefc1010f0 2024-04-21T08:55:42.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.151+0000 7fcf012c8640 1 -- 172.21.15.28:0/3279567911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcefc071aa0 con 0x7fcefc1010f0 2024-04-21T08:55:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.155+0000 7fcee3fff640 1 -- 172.21.15.28:0/3279567911 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcef4004040 con 0x7fcefc1010f0 2024-04-21T08:55:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.156+0000 7fcee3fff640 1 --2- 172.21.15.28:0/3279567911 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcee40a3fa0 0x7fcee40a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.156+0000 7fcee3fff640 1 -- 172.21.15.28:0/3279567911 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fcef40f2c40 con 0x7fcefc1010f0 2024-04-21T08:55:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.157+0000 7fcefa7fc640 1 --2- 172.21.15.28:0/3279567911 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcee40a3fa0 0x7fcee40a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.158+0000 7fcefa7fc640 1 --2- 172.21.15.28:0/3279567911 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcee40a3fa0 0x7fcee40a6460 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcefc06e090 tx=0x7fcef0008040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.159+0000 7fcf03f98740 1 Processor -- start 2024-04-21T08:55:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.160+0000 7fcf03f98740 1 -- start start 2024-04-21T08:55:42.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.160+0000 7fcf03f98740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fce9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.160+0000 7fcf03f98740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fce9b60 0x560a5fce9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.160+0000 7fcf03f98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fcf30e0 0x560a5fcf54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.160+0000 7fcf03f98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560a5fc4e8d0 con 0x560a5fcf30e0 2024-04-21T08:55:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.160+0000 7fcf03f98740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x560a5fbebe80 con 0x560a5fcea8d0 2024-04-21T08:55:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.160+0000 7fcf03f98740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x560a5fc4a890 con 0x560a5fce9b60 2024-04-21T08:55:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.161+0000 7fcefa7fc640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fce9b60 0x560a5fce9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.161+0000 7fcefa7fc640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fce9b60 0x560a5fce9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:40986/0 (socket says 172.21.15.28:40986) 2024-04-21T08:55:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.161+0000 7fcefa7fc640 1 -- 172.21.15.28:0/374135688 learned_addr learned my addr 172.21.15.28:0/374135688 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:42.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.161+0000 7fcefaffd640 1 --2- 172.21.15.28:0/374135688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fce9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.162+0000 7fcefaffd640 1 -- 172.21.15.28:0/374135688 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fce9b60 msgr2=0x560a5fce9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:42.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.162+0000 7fcefaffd640 1 --2- 172.21.15.28:0/374135688 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fce9b60 0x560a5fce9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.162+0000 7fcefaffd640 1 -- 172.21.15.28:0/374135688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fcf30e0 msgr2=0x560a5fcf54d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:42.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.162+0000 7fcefaffd640 1 --2- 172.21.15.28:0/374135688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fcf30e0 0x560a5fcf54d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.162+0000 7fcefaffd640 1 -- 172.21.15.28:0/374135688 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a5fc0d320 con 0x560a5fcea8d0 2024-04-21T08:55:42.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.162+0000 7fcefaffd640 1 --2- 172.21.15.28:0/374135688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fce9620 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcef40f00a0 tx=0x7fcef4009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9328ee227bea441 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:42.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.163+0000 7fcee17fa640 1 -- 172.21.15.28:0/374135688 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef40bf080 con 0x560a5fcea8d0 2024-04-21T08:55:42.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.163+0000 7fcee17fa640 1 -- 172.21.15.28:0/374135688 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcef40be980 con 0x560a5fcea8d0 2024-04-21T08:55:42.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.164+0000 7fcf03f98740 1 -- 172.21.15.28:0/374135688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 msgr2=0x560a5fce9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:42.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.164+0000 7fcf03f98740 1 --2- 172.21.15.28:0/374135688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fce9620 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcef40f00a0 tx=0x7fcef4009a20 comp rx=0 tx=0).stop 2024-04-21T08:55:42.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.164+0000 7fcee17fa640 1 -- 172.21.15.28:0/374135688 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef40c50d0 con 0x560a5fcea8d0 2024-04-21T08:55:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.164+0000 7fcf03f98740 1 -- 172.21.15.28:0/374135688 shutdown_connections 2024-04-21T08:55:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.164+0000 7fcf03f98740 1 --2- 172.21.15.28:0/374135688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fcf30e0 0x560a5fcf54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.164+0000 7fcf03f98740 1 --2- 172.21.15.28:0/374135688 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fce9b60 0x560a5fce9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.164+0000 7fcf03f98740 1 --2- 172.21.15.28:0/374135688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fce9620 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.164+0000 7fcf03f98740 1 -- 172.21.15.28:0/374135688 >> 172.21.15.28:0/374135688 conn(0x560a5fcb0c90 msgr2=0x560a5fce4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.164+0000 7fcf03f98740 1 -- 172.21.15.28:0/374135688 shutdown_connections 2024-04-21T08:55:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.164+0000 7fcf03f98740 1 -- 172.21.15.28:0/374135688 wait complete. 2024-04-21T08:55:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.165+0000 7fcf03f98740 1 -- start start 2024-04-21T08:55:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.165+0000 7fcf03f98740 1 Processor -- start 2024-04-21T08:55:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.165+0000 7fcf03f98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fce9b60 0x560a5fcf2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.165+0000 7fcf03f98740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fcefd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.165+0000 7fcf03f98740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fcf30e0 0x560a5fcf0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.165+0000 7fcefaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fce9b60 0x560a5fcf2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.165+0000 7fcefaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fce9b60 0x560a5fcf2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44186/0 (socket says 172.21.15.28:44186) 2024-04-21T08:55:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.166+0000 7fcefaffd640 1 -- 172.21.15.28:0/687828512 learned_addr learned my addr 172.21.15.28:0/687828512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.166+0000 7fcf03f98740 1 -- 172.21.15.28:0/687828512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560a5fcf5bd0 con 0x560a5fce9b60 2024-04-21T08:55:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.166+0000 7fcf03f98740 1 -- 172.21.15.28:0/687828512 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x560a5fbd6eb0 con 0x560a5fcea8d0 2024-04-21T08:55:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.166+0000 7fcf03f98740 1 -- 172.21.15.28:0/687828512 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x560a5fcf5d50 con 0x560a5fcf30e0 2024-04-21T08:55:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.166+0000 7fcefa7fc640 1 --2- 172.21.15.28:0/687828512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fcefd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.166+0000 7fcefb7fe640 1 --2- 172.21.15.28:0/687828512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fcf30e0 0x560a5fcf0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.166+0000 7fcefaffd640 1 -- 172.21.15.28:0/687828512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fcf30e0 msgr2=0x560a5fcf0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.166+0000 7fcefaffd640 1 --2- 172.21.15.28:0/687828512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fcf30e0 0x560a5fcf0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.166+0000 7fcefaffd640 1 -- 172.21.15.28:0/687828512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 msgr2=0x560a5fcefd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.166+0000 7fcefaffd640 1 --2- 172.21.15.28:0/687828512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fcefd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.166+0000 7fcefaffd640 1 -- 172.21.15.28:0/687828512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a5f961210 con 0x560a5fce9b60 2024-04-21T08:55:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.166+0000 7fcf03f98740 10 client.0 _ll_register_callbacks cb 0x560a5fc03b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.166+0000 7fcf03f98740 1 -- 172.21.15.28:0/687828512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcef40be980 con 0x560a5fce9b60 2024-04-21T08:55:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.166+0000 7fcefaffd640 1 --2- 172.21.15.28:0/687828512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fce9b60 0x560a5fcf2980 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fcef402c400 tx=0x7fcef4005f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b80a9a6aa1e021 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.167+0000 7fcedeff5640 10 client.17061 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:55:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.167+0000 7fcedeff5640 1 -- 172.21.15.28:0/687828512 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef40db020 con 0x560a5fce9b60 2024-04-21T08:55:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.167+0000 7fcedeff5640 1 -- 172.21.15.28:0/687828512 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fcef40bebe0 con 0x560a5fce9b60 2024-04-21T08:55:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.167+0000 7fcedeff5640 1 -- 172.21.15.28:0/687828512 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcef40bee00 con 0x560a5fce9b60 2024-04-21T08:55:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.167+0000 7fcf03f98740 10 client.17061 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.167+0000 7fcf03f98740 1 -- 172.21.15.28:0/687828512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=28}) -- 0x560a5fa2d620 con 0x560a5fce9b60 2024-04-21T08:55:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.173+0000 7fcedeff5640 1 -- 172.21.15.28:0/687828512 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fcef403c030 con 0x560a5fce9b60 2024-04-21T08:55:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.173+0000 7fcf03f98740 10 client.17061 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.173+0000 7fcf03f98740 10 client.17061 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.173+0000 7fcf03f98740 20 client.17061 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.173+0000 7fcf03f98740 1 --2- 172.21.15.28:0/687828512 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560a5fd026f0 0x560a5fd04b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.173+0000 7fcf03f98740 4 client.17061 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:42.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.173+0000 7fcf03f98740 1 -- 172.21.15.28:0/687828512 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a5fc62880 con 0x560a5fd026f0 2024-04-21T08:55:42.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.174+0000 7fcefa7fc640 1 --2- 172.21.15.28:0/687828512 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560a5fd026f0 0x560a5fd04b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.178+0000 7fcefa7fc640 1 --2- 172.21.15.28:0/687828512 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560a5fd026f0 0x560a5fd04b90 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2aea9fc97796fc11 server_cookie=336028d88eb35014 in_seq=0 out_seq=0 2024-04-21T08:55:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.178+0000 7fcedeff5640 10 client.17061 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.180+0000 7fcedeff5640 1 -- 172.21.15.28:0/687828512 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560a5fc62880 con 0x560a5fd026f0 2024-04-21T08:55:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.180+0000 7fcedeff5640 10 client.17061 handle_command_reply: tid=0 2024-04-21T08:55:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.180+0000 7fcf03f98740 1 client.17061 shutdown 2024-04-21T08:55:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.180+0000 7fcf03f98740 1 -- 172.21.15.28:0/687828512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fce9b60 msgr2=0x560a5fcf2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.180+0000 7fcf03f98740 1 --2- 172.21.15.28:0/687828512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fce9b60 0x560a5fcf2980 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fcef402c400 tx=0x7fcef4005f60 comp rx=0 tx=0).stop 2024-04-21T08:55:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.180+0000 7fcf03f98740 1 -- 172.21.15.28:0/687828512 shutdown_connections 2024-04-21T08:55:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.180+0000 7fcf03f98740 1 --2- 172.21.15.28:0/687828512 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560a5fd026f0 0x560a5fd04b90 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.181+0000 7fcf03f98740 1 --2- 172.21.15.28:0/687828512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fcf30e0 0x560a5fcf0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.181+0000 7fcf03f98740 1 --2- 172.21.15.28:0/687828512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fcefd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.181+0000 7fcf03f98740 1 --2- 172.21.15.28:0/687828512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fce9b60 0x560a5fcf2980 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.181+0000 7fcf03f98740 1 -- 172.21.15.28:0/687828512 >> 172.21.15.28:0/687828512 conn(0x560a5fcb0c90 msgr2=0x560a5fce65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.181+0000 7fcedeff5640 0 client.17061 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.181+0000 7fcf03f98740 1 -- 172.21.15.28:0/687828512 shutdown_connections 2024-04-21T08:55:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.181+0000 7fcf03f98740 1 -- 172.21.15.28:0/687828512 wait complete. 2024-04-21T08:55:42.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.182+0000 7fcf03f98740 20 client.17061 trim_cache size 0 max 16384 2024-04-21T08:55:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.212+0000 7fcf03f98740 1 Processor -- start 2024-04-21T08:55:42.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.213+0000 7fcf03f98740 1 -- start start 2024-04-21T08:55:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.213+0000 7fcf03f98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fce9b60 0x560a5fcfea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.213+0000 7fcf03f98740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fcf2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.213 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.214+0000 7fcf03f98740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fcf30e0 0x560a5fcf2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.214+0000 7fcefa7fc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fcf2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.214+0000 7fcefa7fc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fcf2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:48318/0 (socket says 172.21.15.28:48318) 2024-04-21T08:55:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.214+0000 7fcefb7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fcf30e0 0x560a5fcf2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.214+0000 7fcefb7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fcf30e0 0x560a5fcf2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:41004/0 (socket says 172.21.15.28:41004) 2024-04-21T08:55:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.214+0000 7fcefb7fe640 1 -- 172.21.15.28:0/1292400951 learned_addr learned my addr 172.21.15.28:0/1292400951 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.214+0000 7fcf03f98740 1 -- 172.21.15.28:0/1292400951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcee80127e0 con 0x560a5fce9b60 2024-04-21T08:55:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.214+0000 7fcf03f98740 1 -- 172.21.15.28:0/1292400951 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fcee80100b0 con 0x560a5fcea8d0 2024-04-21T08:55:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.214+0000 7fcf03f98740 1 -- 172.21.15.28:0/1292400951 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcee800df00 con 0x560a5fcf30e0 2024-04-21T08:55:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.214+0000 7fcefaffd640 1 --2- 172.21.15.28:0/1292400951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fce9b60 0x560a5fcfea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.215+0000 7fcefb7fe640 1 -- 172.21.15.28:0/1292400951 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 msgr2=0x560a5fcf2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.215+0000 7fcefb7fe640 1 --2- 172.21.15.28:0/1292400951 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fcf2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.215+0000 7fcefb7fe640 1 -- 172.21.15.28:0/1292400951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fce9b60 msgr2=0x560a5fcfea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.215+0000 7fcefb7fe640 1 --2- 172.21.15.28:0/1292400951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fce9b60 0x560a5fcfea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.215+0000 7fcefb7fe640 1 -- 172.21.15.28:0/1292400951 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a5fce6b50 con 0x560a5fcf30e0 2024-04-21T08:55:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.216+0000 7fcefb7fe640 1 --2- 172.21.15.28:0/1292400951 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fcf30e0 0x560a5fcf2980 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fceec00bb10 tx=0x7fceec00d3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fa1f75f6668b863 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.217+0000 7fcee0ff9640 1 -- 172.21.15.28:0/1292400951 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec00dd30 con 0x560a5fcf30e0 2024-04-21T08:55:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.217+0000 7fcee0ff9640 1 -- 172.21.15.28:0/1292400951 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceec004c50 con 0x560a5fcf30e0 2024-04-21T08:55:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.217+0000 7fcee0ff9640 1 -- 172.21.15.28:0/1292400951 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec017400 con 0x560a5fcf30e0 2024-04-21T08:55:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.219+0000 7fcf03f98740 1 -- 172.21.15.28:0/1292400951 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fcf30e0 msgr2=0x560a5fcf2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.219+0000 7fcf03f98740 1 --2- 172.21.15.28:0/1292400951 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fcf30e0 0x560a5fcf2980 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fceec00bb10 tx=0x7fceec00d3a0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.219+0000 7fcf03f98740 1 -- 172.21.15.28:0/1292400951 shutdown_connections 2024-04-21T08:55:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.219+0000 7fcf03f98740 1 --2- 172.21.15.28:0/1292400951 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fcf30e0 0x560a5fcf2980 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.219+0000 7fcf03f98740 1 --2- 172.21.15.28:0/1292400951 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fcf2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.220+0000 7fcf03f98740 1 --2- 172.21.15.28:0/1292400951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fce9b60 0x560a5fcfea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.220+0000 7fcf03f98740 1 -- 172.21.15.28:0/1292400951 >> 172.21.15.28:0/1292400951 conn(0x560a5fcb0c90 msgr2=0x560a5fce38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.223+0000 7fcf03f98740 1 -- 172.21.15.28:0/1292400951 shutdown_connections 2024-04-21T08:55:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.224+0000 7fcf03f98740 1 -- 172.21.15.28:0/1292400951 wait complete. 2024-04-21T08:55:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.224+0000 7fcf03f98740 1 -- start start 2024-04-21T08:55:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.225+0000 7fcf03f98740 1 Processor -- start 2024-04-21T08:55:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.226+0000 7fcf03f98740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fce9b60 0x560a5fcf6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.226+0000 7fcf03f98740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fcf7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.226+0000 7fcf03f98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fcf30e0 0x560a5fcf7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.226+0000 7fcf03f98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcee800e570 con 0x560a5fcf30e0 2024-04-21T08:55:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.226+0000 7fcf03f98740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fcee800f9e0 con 0x560a5fcea8d0 2024-04-21T08:55:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.226+0000 7fcf03f98740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcee800ec40 con 0x560a5fce9b60 2024-04-21T08:55:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.226+0000 7fcf03f98740 10 client.0 _ll_register_callbacks cb 0x7fcef0009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.227+0000 7fcefb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fcf30e0 0x560a5fcf7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.227+0000 7fcefb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fcf30e0 0x560a5fcf7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44206/0 (socket says 172.21.15.28:44206) 2024-04-21T08:55:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.227+0000 7fcefb7fe640 1 -- 172.21.15.28:0/1018576646 learned_addr learned my addr 172.21.15.28:0/1018576646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.228+0000 7fcefb7fe640 1 -- 172.21.15.28:0/1018576646 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fce9b60 msgr2=0x560a5fcf6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.228+0000 7fcefb7fe640 1 --2- 172.21.15.28:0/1018576646 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fce9b60 0x560a5fcf6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.228+0000 7fcefb7fe640 1 -- 172.21.15.28:0/1018576646 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 msgr2=0x560a5fcf7050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.228+0000 7fcefb7fe640 1 --2- 172.21.15.28:0/1018576646 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fcf7050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.228+0000 7fcefb7fe640 1 -- 172.21.15.28:0/1018576646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a5fd67e30 con 0x560a5fcf30e0 2024-04-21T08:55:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.231+0000 7fcefb7fe640 1 --2- 172.21.15.28:0/1018576646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fcf30e0 0x560a5fcf7590 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fceec00bae0 tx=0x7fceec00ff80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaeafcf82762c432 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.231+0000 7fcedeff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:55:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.232+0000 7fcedeff5640 1 -- 172.21.15.28:0/1018576646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec025020 con 0x560a5fcf30e0 2024-04-21T08:55:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.233+0000 7fcf03f98740 1 -- 172.21.15.28:0/1018576646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fceec004c50 con 0x560a5fcf30e0 2024-04-21T08:55:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.233+0000 7fcedeff5640 1 -- 172.21.15.28:0/1018576646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fceec0048d0 con 0x560a5fcf30e0 2024-04-21T08:55:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.233+0000 7fcedeff5640 1 -- 172.21.15.28:0/1018576646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fceec021420 con 0x560a5fcf30e0 2024-04-21T08:55:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.238+0000 7fcf03f98740 10 client.17067 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:42.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.238+0000 7fcf03f98740 1 -- 172.21.15.28:0/1018576646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560a5fd8b130 con 0x560a5fcf30e0 2024-04-21T08:55:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.239+0000 7fcedeff5640 1 -- 172.21.15.28:0/1018576646 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceec004c50 con 0x560a5fcf30e0 2024-04-21T08:55:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.239+0000 7fcedeff5640 1 -- 172.21.15.28:0/1018576646 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fceec017400 con 0x560a5fcf30e0 2024-04-21T08:55:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.239+0000 7fcf03f98740 10 client.17067 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.239+0000 7fcf03f98740 10 client.17067 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.239+0000 7fcf03f98740 20 client.17067 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.239+0000 7fcf03f98740 1 --2- 172.21.15.28:0/1018576646 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560a5fd8b7b0 0x560a5fd8dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.239+0000 7fcf03f98740 4 client.17067 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:55:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.239+0000 7fcf03f98740 1 -- 172.21.15.28:0/1018576646 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560a5fbee3d0 con 0x560a5fd8b7b0 2024-04-21T08:55:42.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.241+0000 7fcefaffd640 1 --2- 172.21.15.28:0/1018576646 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560a5fd8b7b0 0x560a5fd8dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.248+0000 7fcefaffd640 1 --2- 172.21.15.28:0/1018576646 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560a5fd8b7b0 0x560a5fd8dbf0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e0e7f17bb3b7c6d server_cookie=d24119d5170cdeba in_seq=0 out_seq=0 2024-04-21T08:55:42.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:42.248+0000 7fcedeff5640 10 client.17067 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:42.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:42 smithi152 ceph-mon[31401]: pgmap v138: 129 pgs: 129 active+clean; 16 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 43 MiB/s wr, 7.53k op/s 2024-04-21T08:55:42.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:42 smithi028 ceph-mon[28003]: pgmap v138: 129 pgs: 129 active+clean; 16 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 43 MiB/s wr, 7.53k op/s 2024-04-21T08:55:42.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:42 smithi171 ceph-mon[31358]: pgmap v138: 129 pgs: 129 active+clean; 16 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 43 MiB/s wr, 7.53k op/s 2024-04-21T08:55:44.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.269+0000 7fcedeff5640 1 -- 172.21.15.28:0/1018576646 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560a5fbee3d0 con 0x560a5fd8b7b0 2024-04-21T08:55:44.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.269+0000 7fcedeff5640 10 client.17067 handle_command_reply: tid=0 2024-04-21T08:55:44.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.269+0000 7fcf03f98740 1 client.17067 shutdown 2024-04-21T08:55:44.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.269+0000 7fcf03f98740 1 -- 172.21.15.28:0/1018576646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fcf30e0 msgr2=0x560a5fcf7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:44.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.269+0000 7fcf03f98740 1 --2- 172.21.15.28:0/1018576646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fcf30e0 0x560a5fcf7590 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fceec00bae0 tx=0x7fceec00ff80 comp rx=0 tx=0).stop 2024-04-21T08:55:44.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.269+0000 7fcf03f98740 1 -- 172.21.15.28:0/1018576646 shutdown_connections 2024-04-21T08:55:44.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.269+0000 7fcf03f98740 1 --2- 172.21.15.28:0/1018576646 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x560a5fd8b7b0 0x560a5fd8dbf0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:44.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.269+0000 7fcf03f98740 1 --2- 172.21.15.28:0/1018576646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a5fcf30e0 0x560a5fcf7590 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:44.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.269+0000 7fcf03f98740 1 --2- 172.21.15.28:0/1018576646 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560a5fcea8d0 0x560a5fcf7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:44.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.269+0000 7fcf03f98740 1 --2- 172.21.15.28:0/1018576646 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x560a5fce9b60 0x560a5fcf6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:44.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.269+0000 7fcf03f98740 1 -- 172.21.15.28:0/1018576646 >> 172.21.15.28:0/1018576646 conn(0x560a5fcb0c90 msgr2=0x560a5fce5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:44.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.269+0000 7fcedeff5640 0 client.17067 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:44.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.269+0000 7fcf03f98740 1 -- 172.21.15.28:0/1018576646 shutdown_connections 2024-04-21T08:55:44.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.269+0000 7fcf03f98740 1 -- 172.21.15.28:0/1018576646 wait complete. 2024-04-21T08:55:44.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.269+0000 7fcf03f98740 20 client.17067 trim_cache size 0 max 16384 2024-04-21T08:55:44.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:55:44.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:55:44.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:55:44.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:55:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.270+0000 7fcf012c8640 1 -- 172.21.15.28:0/3279567911 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcee40a3fa0 msgr2=0x7fcee40a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.270+0000 7fcf012c8640 1 --2- 172.21.15.28:0/3279567911 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcee40a3fa0 0x7fcee40a6460 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcefc06e090 tx=0x7fcef0008040 comp rx=0 tx=0).stop 2024-04-21T08:55:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.270+0000 7fcf012c8640 1 -- 172.21.15.28:0/3279567911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc1010f0 msgr2=0x7fcefc073850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.270+0000 7fcf012c8640 1 --2- 172.21.15.28:0/3279567911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc1010f0 0x7fcefc073850 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fcef4005e30 tx=0x7fcef4003f90 comp rx=0 tx=0).stop 2024-04-21T08:55:44.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.270+0000 7fcf012c8640 1 -- 172.21.15.28:0/3279567911 shutdown_connections 2024-04-21T08:55:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.270+0000 7fcf012c8640 1 --2- 172.21.15.28:0/3279567911 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcee40a3fa0 0x7fcee40a6460 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.270+0000 7fcf012c8640 1 --2- 172.21.15.28:0/3279567911 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcefc06ce20 0x7fcefc06d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.270+0000 7fcf012c8640 1 --2- 172.21.15.28:0/3279567911 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcefc106170 0x7fcefc06c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.270+0000 7fcf012c8640 1 --2- 172.21.15.28:0/3279567911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc1010f0 0x7fcefc073850 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.270+0000 7fcf012c8640 1 -- 172.21.15.28:0/3279567911 >> 172.21.15.28:0/3279567911 conn(0x7fcefc077fd0 msgr2=0x7fcefc0ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.270+0000 7fcf012c8640 1 -- 172.21.15.28:0/3279567911 shutdown_connections 2024-04-21T08:55:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:44.270+0000 7fcf012c8640 1 -- 172.21.15.28:0/3279567911 wait complete. 2024-04-21T08:55:44.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:44 smithi152 ceph-mon[31401]: pgmap v139: 129 pgs: 129 active+clean; 16 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 42 MiB/s wr, 7.24k op/s 2024-04-21T08:55:44.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:44 smithi028 ceph-mon[28003]: pgmap v139: 129 pgs: 129 active+clean; 16 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 42 MiB/s wr, 7.24k op/s 2024-04-21T08:55:44.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:44 smithi171 ceph-mon[31358]: pgmap v139: 129 pgs: 129 active+clean; 16 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 42 MiB/s wr, 7.24k op/s 2024-04-21T08:55:46.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:46 smithi152 ceph-mon[31401]: pgmap v140: 129 pgs: 129 active+clean; 16 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 44 MiB/s wr, 7.62k op/s 2024-04-21T08:55:46.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:46 smithi028 ceph-mon[28003]: pgmap v140: 129 pgs: 129 active+clean; 16 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 44 MiB/s wr, 7.62k op/s 2024-04-21T08:55:46.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:46 smithi171 ceph-mon[31358]: pgmap v140: 129 pgs: 129 active+clean; 16 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 44 MiB/s wr, 7.62k op/s 2024-04-21T08:55:47.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:47 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:47.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:47 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:47.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:47 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:48.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:48 smithi152 ceph-mon[31401]: pgmap v141: 129 pgs: 129 active+clean; 16 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 40 MiB/s wr, 6.83k op/s 2024-04-21T08:55:48.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:48 smithi028 ceph-mon[28003]: pgmap v141: 129 pgs: 129 active+clean; 16 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 40 MiB/s wr, 6.83k op/s 2024-04-21T08:55:48.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:48 smithi171 ceph-mon[31358]: pgmap v141: 129 pgs: 129 active+clean; 16 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 40 MiB/s wr, 6.83k op/s 2024-04-21T08:55:50.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:50 smithi152 ceph-mon[31401]: pgmap v142: 129 pgs: 129 active+clean; 17 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 47 MiB/s wr, 7.75k op/s 2024-04-21T08:55:50.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:50 smithi028 ceph-mon[28003]: pgmap v142: 129 pgs: 129 active+clean; 17 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 47 MiB/s wr, 7.75k op/s 2024-04-21T08:55:50.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:50 smithi171 ceph-mon[31358]: pgmap v142: 129 pgs: 129 active+clean; 17 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 47 MiB/s wr, 7.75k op/s 2024-04-21T08:55:52.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:52 smithi152 ceph-mon[31401]: pgmap v143: 129 pgs: 129 active+clean; 17 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 48 MiB/s wr, 7.59k op/s 2024-04-21T08:55:52.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:52 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:55:52.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:52 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:55:52.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:52 smithi028 ceph-mon[28003]: pgmap v143: 129 pgs: 129 active+clean; 17 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 48 MiB/s wr, 7.59k op/s 2024-04-21T08:55:52.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:52 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:55:52.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:52 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:55:52.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:52 smithi171 ceph-mon[31358]: pgmap v143: 129 pgs: 129 active+clean; 17 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 48 MiB/s wr, 7.59k op/s 2024-04-21T08:55:52.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:52 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:55:52.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:52 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:55:53.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:53 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:55:53.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:53 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:55:53.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:53 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:55:54.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:55:54.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:55:54.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.282+0000 7fad3590d640 1 -- 172.21.15.28:0/1277725989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad30101a30 msgr2=0x7fad30105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.282+0000 7fad3590d640 1 --2- 172.21.15.28:0/1277725989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad30101a30 0x7fad30105a70 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fad24009900 tx=0x7fad2402bc40 comp rx=0 tx=0).stop 2024-04-21T08:55:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.286+0000 7fad3590d640 1 -- 172.21.15.28:0/1277725989 shutdown_connections 2024-04-21T08:55:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.286+0000 7fad3590d640 1 --2- 172.21.15.28:0/1277725989 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad30106150 0x7fad3010dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.286+0000 7fad3590d640 1 --2- 172.21.15.28:0/1277725989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad30101a30 0x7fad30105a70 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.286+0000 7fad3590d640 1 --2- 172.21.15.28:0/1277725989 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad30101080 0x7fad30101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.286+0000 7fad3590d640 1 -- 172.21.15.28:0/1277725989 >> 172.21.15.28:0/1277725989 conn(0x7fad30078000 msgr2=0x7fad30078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:54.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.289+0000 7fad3590d640 1 -- 172.21.15.28:0/1277725989 shutdown_connections 2024-04-21T08:55:54.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.291+0000 7fad3590d640 1 -- 172.21.15.28:0/1277725989 wait complete. 2024-04-21T08:55:54.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.292+0000 7fad3590d640 1 Processor -- start 2024-04-21T08:55:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.294+0000 7fad3590d640 1 -- start start 2024-04-21T08:55:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.295+0000 7fad3590d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad30101080 0x7fad30073940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.295+0000 7fad3590d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad30106150 0x7fad30107cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.295+0000 7fad3590d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad30108230 0x7fad301086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.295+0000 7fad3590d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad301110c0 con 0x7fad30101080 2024-04-21T08:55:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.295+0000 7fad3590d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fad30110f40 con 0x7fad30108230 2024-04-21T08:55:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.295+0000 7fad3590d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fad30111240 con 0x7fad30106150 2024-04-21T08:55:54.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.296+0000 7fad2f7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad30108230 0x7fad301086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.296+0000 7fad2f7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad30108230 0x7fad301086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:37750/0 (socket says 172.21.15.28:37750) 2024-04-21T08:55:54.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.296+0000 7fad2f7fe640 1 -- 172.21.15.28:0/560992842 learned_addr learned my addr 172.21.15.28:0/560992842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:54.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.298+0000 7fad2e7fc640 1 --2- 172.21.15.28:0/560992842 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad30106150 0x7fad30107cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.298+0000 7fad2effd640 1 --2- 172.21.15.28:0/560992842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad30101080 0x7fad30073940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.299+0000 7fad2effd640 1 -- 172.21.15.28:0/560992842 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad30106150 msgr2=0x7fad30107cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.299+0000 7fad2effd640 1 --2- 172.21.15.28:0/560992842 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad30106150 0x7fad30107cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.299+0000 7fad2effd640 1 -- 172.21.15.28:0/560992842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad30108230 msgr2=0x7fad301086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.299+0000 7fad2effd640 1 --2- 172.21.15.28:0/560992842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad30108230 0x7fad301086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.299+0000 7fad2effd640 1 -- 172.21.15.28:0/560992842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad30109ca0 con 0x7fad30101080 2024-04-21T08:55:54.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.300+0000 7fad2effd640 1 --2- 172.21.15.28:0/560992842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad30101080 0x7fad30073940 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fad2800e9f0 tx=0x7fad2800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:54.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.302+0000 7fad3490b640 1 -- 172.21.15.28:0/560992842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2800cd80 con 0x7fad30101080 2024-04-21T08:55:54.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.302+0000 7fad3490b640 1 -- 172.21.15.28:0/560992842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad28011920 con 0x7fad30101080 2024-04-21T08:55:54.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.302+0000 7fad3490b640 1 -- 172.21.15.28:0/560992842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad280109c0 con 0x7fad30101080 2024-04-21T08:55:54.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.303+0000 7fad3590d640 1 -- 172.21.15.28:0/560992842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad301b3e90 con 0x7fad30101080 2024-04-21T08:55:54.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.303+0000 7fad3590d640 1 -- 172.21.15.28:0/560992842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad301b4300 con 0x7fad30101080 2024-04-21T08:55:54.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.308+0000 7fad3490b640 1 -- 172.21.15.28:0/560992842 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fad28010b60 con 0x7fad30101080 2024-04-21T08:55:54.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.308+0000 7fad385ce740 1 Processor -- start 2024-04-21T08:55:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.311+0000 7fad385ce740 1 -- start start 2024-04-21T08:55:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.311+0000 7fad385ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fb580 0x55f8dd7fb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.311+0000 7fad385ce740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd7fbea0 0x55f8dd804540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.311+0000 7fad385ce740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd804a80 0x55f8dd806e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.311+0000 7fad385ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f8dd7601e0 con 0x55f8dd7fb580 2024-04-21T08:55:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.311+0000 7fad385ce740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f8dd6fd390 con 0x55f8dd7fbea0 2024-04-21T08:55:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.311+0000 7fad385ce740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f8dd75c1a0 con 0x55f8dd804a80 2024-04-21T08:55:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.312+0000 7fad3490b640 1 --2- 172.21.15.28:0/560992842 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fad180a3f20 0x7fad180a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.313+0000 7fad3490b640 1 -- 172.21.15.28:0/560992842 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fad280d1130 con 0x7fad30101080 2024-04-21T08:55:54.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.314+0000 7fad2effd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd7fbea0 0x55f8dd804540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.315+0000 7fad2f7fe640 1 --2- 172.21.15.28:0/560992842 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fad180a3f20 0x7fad180a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.315+0000 7fad2f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fb580 0x55f8dd7fb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.315+0000 7fad2f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fb580 0x55f8dd7fb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46050/0 (socket says 172.21.15.28:46050) 2024-04-21T08:55:54.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.315+0000 7fad2f7fe640 1 -- 172.21.15.28:0/3339584468 learned_addr learned my addr 172.21.15.28:0/3339584468 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.315+0000 7fad2e7fc640 1 --2- 172.21.15.28:0/3339584468 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd804a80 0x55f8dd806e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.317+0000 7fad2f7fe640 1 -- 172.21.15.28:0/3339584468 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd804a80 msgr2=0x55f8dd806e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.317+0000 7fad2f7fe640 1 --2- 172.21.15.28:0/3339584468 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd804a80 0x55f8dd806e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.317+0000 7fad2f7fe640 1 -- 172.21.15.28:0/3339584468 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd7fbea0 msgr2=0x55f8dd804540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.317+0000 7fad2f7fe640 1 --2- 172.21.15.28:0/3339584468 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd7fbea0 0x55f8dd804540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.317+0000 7fad2f7fe640 1 -- 172.21.15.28:0/3339584468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8dd53c7d0 con 0x55f8dd7fb580 2024-04-21T08:55:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.318+0000 7fad2f7fe640 1 --2- 172.21.15.28:0/560992842 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fad180a3f20 0x7fad180a63e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fad200146e0 tx=0x7fad2000cbb0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.318+0000 7fad2f7fe640 1 --2- 172.21.15.28:0/3339584468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fb580 0x55f8dd7fb960 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fad20014f70 tx=0x7fad2001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91531a85889a5bb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:54.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.322+0000 7fad15ffb640 1 -- 172.21.15.28:0/3339584468 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20023070 con 0x55f8dd7fb580 2024-04-21T08:55:54.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.322+0000 7fad15ffb640 1 -- 172.21.15.28:0/3339584468 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad2000c420 con 0x55f8dd7fb580 2024-04-21T08:55:54.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.322+0000 7fad15ffb640 1 -- 172.21.15.28:0/3339584468 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2000fb50 con 0x55f8dd7fb580 2024-04-21T08:55:54.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.323+0000 7fad385ce740 1 -- 172.21.15.28:0/3339584468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fb580 msgr2=0x55f8dd7fb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.323+0000 7fad385ce740 1 --2- 172.21.15.28:0/3339584468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fb580 0x55f8dd7fb960 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fad20014f70 tx=0x7fad2001c000 comp rx=0 tx=0).stop 2024-04-21T08:55:54.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.324+0000 7fad385ce740 1 -- 172.21.15.28:0/3339584468 shutdown_connections 2024-04-21T08:55:54.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.324+0000 7fad385ce740 1 --2- 172.21.15.28:0/3339584468 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd804a80 0x55f8dd806e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.324+0000 7fad385ce740 1 --2- 172.21.15.28:0/3339584468 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd7fbea0 0x55f8dd804540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.324+0000 7fad385ce740 1 --2- 172.21.15.28:0/3339584468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fb580 0x55f8dd7fb960 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.324+0000 7fad385ce740 1 -- 172.21.15.28:0/3339584468 >> 172.21.15.28:0/3339584468 conn(0x55f8dd7f5cb0 msgr2=0x55f8dd7f6090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:54.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.325+0000 7fad385ce740 1 -- 172.21.15.28:0/3339584468 shutdown_connections 2024-04-21T08:55:54.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.328+0000 7fad385ce740 1 -- 172.21.15.28:0/3339584468 wait complete. 2024-04-21T08:55:54.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.328+0000 7fad385ce740 1 -- start start 2024-04-21T08:55:54.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.328+0000 7fad385ce740 1 Processor -- start 2024-04-21T08:55:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.329+0000 7fad385ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fb580 0x55f8dd801ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.329+0000 7fad385ce740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd7fbea0 0x55f8dd802020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.329+0000 7fad385ce740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd804a80 0x55f8dd802560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.329+0000 7fad385ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f8dd807810 con 0x55f8dd7fb580 2024-04-21T08:55:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.329+0000 7fad385ce740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f8dd7192f0 con 0x55f8dd7fbea0 2024-04-21T08:55:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.329+0000 7fad385ce740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f8dd807990 con 0x55f8dd804a80 2024-04-21T08:55:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.329+0000 7fad385ce740 10 client.0 _ll_register_callbacks cb 0x55f8dd726580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.329+0000 7fad2f7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd804a80 0x55f8dd802560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.329+0000 7fad2f7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd804a80 0x55f8dd802560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:55152/0 (socket says 172.21.15.28:55152) 2024-04-21T08:55:54.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.329+0000 7fad2f7fe640 1 -- 172.21.15.28:0/4023106103 learned_addr learned my addr 172.21.15.28:0/4023106103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:54.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.330+0000 7fad2f7fe640 1 -- 172.21.15.28:0/4023106103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd7fbea0 msgr2=0x55f8dd802020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:54.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.330+0000 7fad2f7fe640 1 --2- 172.21.15.28:0/4023106103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd7fbea0 0x55f8dd802020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.330+0000 7fad2f7fe640 1 -- 172.21.15.28:0/4023106103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fb580 msgr2=0x55f8dd801ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:54.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.330+0000 7fad2f7fe640 1 --2- 172.21.15.28:0/4023106103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fb580 0x55f8dd801ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.330+0000 7fad2f7fe640 1 -- 172.21.15.28:0/4023106103 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8dd4723f0 con 0x55f8dd804a80 2024-04-21T08:55:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.331+0000 7fad2f7fe640 1 --2- 172.21.15.28:0/4023106103 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd804a80 0x55f8dd802560 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fad2000f810 tx=0x7fad20014220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9029937cf2abd27 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.331+0000 7fad0effd640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:55:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.331+0000 7fad0effd640 1 -- 172.21.15.28:0/4023106103 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2002c070 con 0x55f8dd804a80 2024-04-21T08:55:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.331+0000 7fad0effd640 1 -- 172.21.15.28:0/4023106103 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fad2000fac0 con 0x55f8dd804a80 2024-04-21T08:55:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.332+0000 7fad385ce740 1 -- 172.21.15.28:0/4023106103 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad2000c420 con 0x55f8dd804a80 2024-04-21T08:55:54.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.335+0000 7fad0effd640 1 -- 172.21.15.28:0/4023106103 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fad20036050 con 0x55f8dd804a80 2024-04-21T08:55:54.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.337+0000 7fad385ce740 10 client.26362 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:54.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.337+0000 7fad385ce740 1 -- 172.21.15.28:0/4023106103 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f8dd4320f0 con 0x55f8dd804a80 2024-04-21T08:55:54.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.342+0000 7fad0effd640 1 -- 172.21.15.28:0/4023106103 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad2000c420 con 0x55f8dd804a80 2024-04-21T08:55:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.342+0000 7fad0effd640 1 -- 172.21.15.28:0/4023106103 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fad200046c0 con 0x55f8dd804a80 2024-04-21T08:55:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.342+0000 7fad385ce740 10 client.26362 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.342+0000 7fad385ce740 10 client.26362 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.342+0000 7fad385ce740 20 client.26362 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.342+0000 7fad385ce740 1 --2- 172.21.15.28:0/4023106103 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f8dd825a50 0x55f8dd827e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.342+0000 7fad385ce740 4 client.26362 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.342+0000 7fad385ce740 1 -- 172.21.15.28:0/4023106103 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8dd76dc00 con 0x55f8dd825a50 2024-04-21T08:55:54.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.344+0000 7fad2effd640 1 --2- 172.21.15.28:0/4023106103 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f8dd825a50 0x55f8dd827e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.346+0000 7fad2effd640 1 --2- 172.21.15.28:0/4023106103 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f8dd825a50 0x55f8dd827e40 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87e4884db9d165d6 server_cookie=1041ea2cede30a56 in_seq=0 out_seq=0 2024-04-21T08:55:54.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.346+0000 7fad0effd640 10 client.26362 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:54.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.347+0000 7fad0effd640 1 -- 172.21.15.28:0/4023106103 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f8dd76dc00 con 0x55f8dd825a50 2024-04-21T08:55:54.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.347+0000 7fad0effd640 10 client.26362 handle_command_reply: tid=0 2024-04-21T08:55:54.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.347+0000 7fad385ce740 1 client.26362 shutdown 2024-04-21T08:55:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.352+0000 7fad385ce740 1 -- 172.21.15.28:0/4023106103 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd804a80 msgr2=0x55f8dd802560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.352+0000 7fad385ce740 1 --2- 172.21.15.28:0/4023106103 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd804a80 0x55f8dd802560 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fad2000f810 tx=0x7fad20014220 comp rx=0 tx=0).stop 2024-04-21T08:55:54.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.352+0000 7fad385ce740 1 -- 172.21.15.28:0/4023106103 shutdown_connections 2024-04-21T08:55:54.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.352+0000 7fad385ce740 1 --2- 172.21.15.28:0/4023106103 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f8dd825a50 0x55f8dd827e40 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.352+0000 7fad385ce740 1 --2- 172.21.15.28:0/4023106103 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd804a80 0x55f8dd802560 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.352+0000 7fad385ce740 1 --2- 172.21.15.28:0/4023106103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd7fbea0 0x55f8dd802020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.352+0000 7fad385ce740 1 --2- 172.21.15.28:0/4023106103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fb580 0x55f8dd801ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.352+0000 7fad385ce740 1 -- 172.21.15.28:0/4023106103 >> 172.21.15.28:0/4023106103 conn(0x55f8dd7f5cb0 msgr2=0x55f8dd7f7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:54.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.353+0000 7fad0effd640 0 client.26362 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:54.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.353+0000 7fad385ce740 1 -- 172.21.15.28:0/4023106103 shutdown_connections 2024-04-21T08:55:54.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.353+0000 7fad385ce740 1 -- 172.21.15.28:0/4023106103 wait complete. 2024-04-21T08:55:54.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.354+0000 7fad385ce740 20 client.26362 trim_cache size 0 max 16384 2024-04-21T08:55:54.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:54 smithi152 ceph-mon[31401]: pgmap v144: 129 pgs: 129 active+clean; 17 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 46 MiB/s wr, 7.01k op/s 2024-04-21T08:55:54.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:54 smithi028 ceph-mon[28003]: pgmap v144: 129 pgs: 129 active+clean; 17 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 46 MiB/s wr, 7.01k op/s 2024-04-21T08:55:54.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:54 smithi171 ceph-mon[31358]: pgmap v144: 129 pgs: 129 active+clean; 17 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 46 MiB/s wr, 7.01k op/s 2024-04-21T08:55:54.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.395+0000 7fad385ce740 1 Processor -- start 2024-04-21T08:55:54.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.395+0000 7fad385ce740 1 -- start start 2024-04-21T08:55:54.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.395+0000 7fad385ce740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd7fb580 0x55f8dd804210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.395+0000 7fad385ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fbea0 0x55f8dd8106d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.395+0000 7fad385ce740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd804a80 0x55f8dd800c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.395+0000 7fad385ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad301b7b10 con 0x55f8dd7fbea0 2024-04-21T08:55:54.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.395+0000 7fad385ce740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fad301b81e0 con 0x55f8dd804a80 2024-04-21T08:55:54.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.395+0000 7fad385ce740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fad301b5ab0 con 0x55f8dd7fb580 2024-04-21T08:55:54.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.395+0000 7fad2e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fbea0 0x55f8dd8106d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.395+0000 7fad2e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fbea0 0x55f8dd8106d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46052/0 (socket says 172.21.15.28:46052) 2024-04-21T08:55:54.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.395+0000 7fad2e7fc640 1 -- 172.21.15.28:0/2490433028 learned_addr learned my addr 172.21.15.28:0/2490433028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:54.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.395+0000 7fad2f7fe640 1 --2- 172.21.15.28:0/2490433028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd804a80 0x55f8dd800c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.395+0000 7fad2effd640 1 --2- 172.21.15.28:0/2490433028 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd7fb580 0x55f8dd804210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.396+0000 7fad2f7fe640 1 -- 172.21.15.28:0/2490433028 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd7fb580 msgr2=0x55f8dd804210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:54.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.396+0000 7fad2f7fe640 1 --2- 172.21.15.28:0/2490433028 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd7fb580 0x55f8dd804210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.396+0000 7fad2f7fe640 1 -- 172.21.15.28:0/2490433028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fbea0 msgr2=0x55f8dd8106d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:54.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.396+0000 7fad2f7fe640 1 --2- 172.21.15.28:0/2490433028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fbea0 0x55f8dd8106d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.396+0000 7fad2f7fe640 1 -- 172.21.15.28:0/2490433028 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8dd800030 con 0x55f8dd804a80 2024-04-21T08:55:54.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.398+0000 7fad2f7fe640 1 --2- 172.21.15.28:0/2490433028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd804a80 0x55f8dd800c40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fad20004c80 tx=0x7fad2000f840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5aefd9a585b5fe37 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:54.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.398+0000 7fad157fa640 1 -- 172.21.15.28:0/2490433028 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2003f030 con 0x55f8dd804a80 2024-04-21T08:55:54.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.398+0000 7fad157fa640 1 -- 172.21.15.28:0/2490433028 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad20004680 con 0x55f8dd804a80 2024-04-21T08:55:54.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.399+0000 7fad385ce740 1 -- 172.21.15.28:0/2490433028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd804a80 msgr2=0x55f8dd800c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.399+0000 7fad385ce740 1 --2- 172.21.15.28:0/2490433028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd804a80 0x55f8dd800c40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fad20004c80 tx=0x7fad2000f840 comp rx=0 tx=0).stop 2024-04-21T08:55:54.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.400+0000 7fad385ce740 1 -- 172.21.15.28:0/2490433028 shutdown_connections 2024-04-21T08:55:54.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.400+0000 7fad385ce740 1 --2- 172.21.15.28:0/2490433028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd804a80 0x55f8dd800c40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.400+0000 7fad385ce740 1 --2- 172.21.15.28:0/2490433028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd7fbea0 0x55f8dd8106d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.400+0000 7fad385ce740 1 --2- 172.21.15.28:0/2490433028 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd7fb580 0x55f8dd804210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.400+0000 7fad385ce740 1 -- 172.21.15.28:0/2490433028 >> 172.21.15.28:0/2490433028 conn(0x55f8dd7f5cb0 msgr2=0x55f8dd801aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.404+0000 7fad385ce740 1 -- 172.21.15.28:0/2490433028 shutdown_connections 2024-04-21T08:55:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.404+0000 7fad385ce740 1 -- 172.21.15.28:0/2490433028 wait complete. 2024-04-21T08:55:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.404+0000 7fad385ce740 1 -- start start 2024-04-21T08:55:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.405+0000 7fad385ce740 1 Processor -- start 2024-04-21T08:55:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.405+0000 7fad385ce740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd7fb580 0x55f8dd808940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.405+0000 7fad385ce740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd7fbea0 0x55f8dd808e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.405+0000 7fad385ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd804a80 0x55f8dd88add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.405+0000 7fad385ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad301af750 con 0x55f8dd804a80 2024-04-21T08:55:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.405+0000 7fad385ce740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fad301b53e0 con 0x55f8dd7fb580 2024-04-21T08:55:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.405+0000 7fad385ce740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fad301b4640 con 0x55f8dd7fbea0 2024-04-21T08:55:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.405+0000 7fad385ce740 10 client.0 _ll_register_callbacks cb 0x7fad28005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.406+0000 7fad2e7fc640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd7fbea0 0x55f8dd808e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.406+0000 7fad2e7fc640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd7fbea0 0x55f8dd808e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:55174/0 (socket says 172.21.15.28:55174) 2024-04-21T08:55:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.406+0000 7fad2e7fc640 1 -- 172.21.15.28:0/4184624647 learned_addr learned my addr 172.21.15.28:0/4184624647 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.406+0000 7fad2effd640 1 --2- 172.21.15.28:0/4184624647 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd7fb580 0x55f8dd808940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.406+0000 7fad2f7fe640 1 --2- 172.21.15.28:0/4184624647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd804a80 0x55f8dd88add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.406+0000 7fad2e7fc640 1 -- 172.21.15.28:0/4184624647 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd7fb580 msgr2=0x55f8dd808940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.406+0000 7fad2e7fc640 1 --2- 172.21.15.28:0/4184624647 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd7fb580 0x55f8dd808940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.406+0000 7fad2e7fc640 1 -- 172.21.15.28:0/4184624647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd804a80 msgr2=0x55f8dd88add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.406+0000 7fad2e7fc640 1 --2- 172.21.15.28:0/4184624647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd804a80 0x55f8dd88add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.406+0000 7fad2e7fc640 1 -- 172.21.15.28:0/4184624647 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8dd809780 con 0x55f8dd7fbea0 2024-04-21T08:55:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.407+0000 7fad2e7fc640 1 --2- 172.21.15.28:0/4184624647 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd7fbea0 0x55f8dd808e80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fad24009230 tx=0x7fad2402ca20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=614232f37d3700e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.407+0000 7fad0effd640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:55:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.407+0000 7fad0effd640 1 -- 172.21.15.28:0/4184624647 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad24005240 con 0x55f8dd7fbea0 2024-04-21T08:55:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.407+0000 7fad385ce740 1 -- 172.21.15.28:0/4184624647 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad20004680 con 0x55f8dd7fbea0 2024-04-21T08:55:54.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.407+0000 7fad0effd640 1 -- 172.21.15.28:0/4184624647 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fad240053e0 con 0x55f8dd7fbea0 2024-04-21T08:55:54.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.408+0000 7fad0effd640 1 -- 172.21.15.28:0/4184624647 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fad240055e0 con 0x55f8dd7fbea0 2024-04-21T08:55:54.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.412+0000 7fad385ce740 10 client.26368 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:54.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.412+0000 7fad385ce740 1 -- 172.21.15.28:0/4184624647 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f8dd88c450 con 0x55f8dd7fbea0 2024-04-21T08:55:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.414+0000 7fad0effd640 1 -- 172.21.15.28:0/4184624647 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad20004680 con 0x55f8dd7fbea0 2024-04-21T08:55:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.414+0000 7fad0effd640 1 -- 172.21.15.28:0/4184624647 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fad24003820 con 0x55f8dd7fbea0 2024-04-21T08:55:54.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.414+0000 7fad385ce740 10 client.26368 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:54.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.414+0000 7fad385ce740 10 client.26368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:54.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.414+0000 7fad385ce740 20 client.26368 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:54.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.414+0000 7fad385ce740 1 --2- 172.21.15.28:0/4184624647 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f8dd88cbb0 0x55f8dd88efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.414+0000 7fad385ce740 4 client.26368 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:55:54.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.414+0000 7fad385ce740 1 -- 172.21.15.28:0/4184624647 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f8dd767a70 con 0x55f8dd88cbb0 2024-04-21T08:55:54.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.416+0000 7fad2effd640 1 --2- 172.21.15.28:0/4184624647 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f8dd88cbb0 0x55f8dd88efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.419+0000 7fad2effd640 1 --2- 172.21.15.28:0/4184624647 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f8dd88cbb0 0x55f8dd88efd0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd0945cc6368a1b5 server_cookie=5405566204c07b1c in_seq=0 out_seq=0 2024-04-21T08:55:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.420+0000 7fad0effd640 10 client.26368 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.420+0000 7fad0effd640 1 -- 172.21.15.28:0/4184624647 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f8dd767a70 con 0x55f8dd88cbb0 2024-04-21T08:55:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.421+0000 7fad0effd640 10 client.26368 handle_command_reply: tid=0 2024-04-21T08:55:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.421+0000 7fad385ce740 1 client.26368 shutdown 2024-04-21T08:55:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.425+0000 7fad385ce740 1 -- 172.21.15.28:0/4184624647 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd7fbea0 msgr2=0x55f8dd808e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.425+0000 7fad385ce740 1 --2- 172.21.15.28:0/4184624647 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd7fbea0 0x55f8dd808e80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fad24009230 tx=0x7fad2402ca20 comp rx=0 tx=0).stop 2024-04-21T08:55:54.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.427+0000 7fad385ce740 1 -- 172.21.15.28:0/4184624647 shutdown_connections 2024-04-21T08:55:54.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.427+0000 7fad385ce740 1 --2- 172.21.15.28:0/4184624647 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f8dd88cbb0 0x55f8dd88efd0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.427+0000 7fad385ce740 1 --2- 172.21.15.28:0/4184624647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f8dd804a80 0x55f8dd88add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.427+0000 7fad385ce740 1 --2- 172.21.15.28:0/4184624647 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f8dd7fbea0 0x55f8dd808e80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.427+0000 7fad385ce740 1 --2- 172.21.15.28:0/4184624647 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f8dd7fb580 0x55f8dd808940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.427+0000 7fad385ce740 1 -- 172.21.15.28:0/4184624647 >> 172.21.15.28:0/4184624647 conn(0x55f8dd7f5cb0 msgr2=0x55f8dd801aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:54.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.428+0000 7fad0effd640 0 client.26368 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:54.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.428+0000 7fad385ce740 1 -- 172.21.15.28:0/4184624647 shutdown_connections 2024-04-21T08:55:54.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.428+0000 7fad385ce740 1 -- 172.21.15.28:0/4184624647 wait complete. 2024-04-21T08:55:54.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.429+0000 7fad385ce740 20 client.26368 trim_cache size 0 max 16384 2024-04-21T08:55:54.430 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:55:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T08:55:54.430 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T08:55:54.430 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:55:54.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.434+0000 7fad3590d640 1 -- 172.21.15.28:0/560992842 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fad180a3f20 msgr2=0x7fad180a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.434+0000 7fad3590d640 1 --2- 172.21.15.28:0/560992842 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fad180a3f20 0x7fad180a63e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fad200146e0 tx=0x7fad2000cbb0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.434+0000 7fad3590d640 1 -- 172.21.15.28:0/560992842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad30101080 msgr2=0x7fad30073940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.434+0000 7fad3590d640 1 --2- 172.21.15.28:0/560992842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad30101080 0x7fad30073940 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fad2800e9f0 tx=0x7fad2800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.436+0000 7fad3590d640 1 -- 172.21.15.28:0/560992842 shutdown_connections 2024-04-21T08:55:54.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.436+0000 7fad3590d640 1 --2- 172.21.15.28:0/560992842 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fad180a3f20 0x7fad180a63e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.436+0000 7fad3590d640 1 --2- 172.21.15.28:0/560992842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad30108230 0x7fad301086b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.436+0000 7fad3590d640 1 --2- 172.21.15.28:0/560992842 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad30106150 0x7fad30107cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.436+0000 7fad3590d640 1 --2- 172.21.15.28:0/560992842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad30101080 0x7fad30073940 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.436+0000 7fad3590d640 1 -- 172.21.15.28:0/560992842 >> 172.21.15.28:0/560992842 conn(0x7fad30078000 msgr2=0x7fad300ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.437+0000 7fad3590d640 1 -- 172.21.15.28:0/560992842 shutdown_connections 2024-04-21T08:55:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.437+0000 7fad3590d640 1 -- 172.21.15.28:0/560992842 wait complete. 2024-04-21T08:55:54.457 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:55:54.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:55:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.664+0000 7f9d3d2d2640 1 -- 172.21.15.28:0/431769059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d38069cb0 msgr2=0x7f9d38104900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.664+0000 7f9d3d2d2640 1 --2- 172.21.15.28:0/431769059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d38069cb0 0x7f9d38104900 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f9d28009860 tx=0x7f9d2802bb90 comp rx=0 tx=0).stop 2024-04-21T08:55:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.665+0000 7f9d3d2d2640 1 -- 172.21.15.28:0/431769059 shutdown_connections 2024-04-21T08:55:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.665+0000 7f9d3d2d2640 1 --2- 172.21.15.28:0/431769059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d38069cb0 0x7f9d38104900 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.665+0000 7f9d3d2d2640 1 --2- 172.21.15.28:0/431769059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d38107380 0x7f9d38069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.665+0000 7f9d3d2d2640 1 --2- 172.21.15.28:0/431769059 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d38106f10 0x7f9d380690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.665+0000 7f9d3d2d2640 1 -- 172.21.15.28:0/431769059 >> 172.21.15.28:0/431769059 conn(0x7f9d38077e30 msgr2=0x7f9d38078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:54.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.665+0000 7f9d3d2d2640 1 -- 172.21.15.28:0/431769059 shutdown_connections 2024-04-21T08:55:54.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.667+0000 7f9d3d2d2640 1 -- 172.21.15.28:0/431769059 wait complete. 2024-04-21T08:55:54.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.668+0000 7f9d3d2d2640 1 Processor -- start 2024-04-21T08:55:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.668+0000 7f9d3d2d2640 1 -- start start 2024-04-21T08:55:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.668+0000 7f9d3d2d2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d38069cb0 0x7f9d38115690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.668+0000 7f9d3d2d2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d38106f10 0x7f9d3810f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.668+0000 7f9d3d2d2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d38107380 0x7f9d3810fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.668+0000 7f9d3d2d2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d38108080 con 0x7f9d38107380 2024-04-21T08:55:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.668+0000 7f9d3d2d2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9d38107f00 con 0x7f9d38106f10 2024-04-21T08:55:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.668+0000 7f9d3d2d2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9d38108200 con 0x7f9d38069cb0 2024-04-21T08:55:54.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.670+0000 7f9d367fc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d38106f10 0x7f9d3810f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.670+0000 7f9d367fc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d38106f10 0x7f9d3810f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:37796/0 (socket says 172.21.15.28:37796) 2024-04-21T08:55:54.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.670+0000 7f9d367fc640 1 -- 172.21.15.28:0/495455978 learned_addr learned my addr 172.21.15.28:0/495455978 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:54.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.670+0000 7f9d36ffd640 1 --2- 172.21.15.28:0/495455978 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d38069cb0 0x7f9d38115690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.670+0000 7f9d377fe640 1 --2- 172.21.15.28:0/495455978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d38107380 0x7f9d3810fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.670+0000 7f9d36ffd640 1 -- 172.21.15.28:0/495455978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d38106f10 msgr2=0x7f9d3810f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.670+0000 7f9d36ffd640 1 --2- 172.21.15.28:0/495455978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d38106f10 0x7f9d3810f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.670+0000 7f9d36ffd640 1 -- 172.21.15.28:0/495455978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d38107380 msgr2=0x7f9d3810fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.670+0000 7f9d36ffd640 1 --2- 172.21.15.28:0/495455978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d38107380 0x7f9d3810fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.671+0000 7f9d36ffd640 1 -- 172.21.15.28:0/495455978 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d38110360 con 0x7f9d38069cb0 2024-04-21T08:55:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.672+0000 7f9d36ffd640 1 --2- 172.21.15.28:0/495455978 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d38069cb0 0x7f9d38115690 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000ef90 tx=0x7f9d3000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.674+0000 7f9d1ffff640 1 -- 172.21.15.28:0/495455978 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3001b070 con 0x7f9d38069cb0 2024-04-21T08:55:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.674+0000 7f9d1ffff640 1 -- 172.21.15.28:0/495455978 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d3000bae0 con 0x7f9d38069cb0 2024-04-21T08:55:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.674+0000 7f9d1ffff640 1 -- 172.21.15.28:0/495455978 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d30016b80 con 0x7f9d38069cb0 2024-04-21T08:55:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.675+0000 7f9d3d2d2640 1 -- 172.21.15.28:0/495455978 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d3806b970 con 0x7f9d38069cb0 2024-04-21T08:55:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.675+0000 7f9d3d2d2640 1 -- 172.21.15.28:0/495455978 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d3806be70 con 0x7f9d38069cb0 2024-04-21T08:55:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.676+0000 7f9d3d2d2640 1 -- 172.21.15.28:0/495455978 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d38101440 con 0x7f9d38069cb0 2024-04-21T08:55:54.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.680+0000 7f9d1ffff640 1 -- 172.21.15.28:0/495455978 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9d3001e070 con 0x7f9d38069cb0 2024-04-21T08:55:54.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.681+0000 7f9d1ffff640 1 --2- 172.21.15.28:0/495455978 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9d200a3f20 0x7f9d200a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.682+0000 7f9d367fc640 1 --2- 172.21.15.28:0/495455978 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9d200a3f20 0x7f9d200a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.682+0000 7f9d1ffff640 1 -- 172.21.15.28:0/495455978 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f9d300d0e90 con 0x7f9d38069cb0 2024-04-21T08:55:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.682+0000 7f9d367fc640 1 --2- 172.21.15.28:0/495455978 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9d200a3f20 0x7f9d200a63e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9d38110ec0 tx=0x7f9d2c00a2d0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.686+0000 7f9d1ffff640 1 -- 172.21.15.28:0/495455978 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d3009d1b0 con 0x7f9d38069cb0 2024-04-21T08:55:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.990+0000 7f9d3d2d2640 1 -- 172.21.15.28:0/495455978 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d38110c30 con 0x7f9d38069cb0 2024-04-21T08:55:54.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.992+0000 7f9d1ffff640 1 -- 172.21.15.28:0/495455978 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7f9d3009cab0 con 0x7f9d38069cb0 2024-04-21T08:55:54.992 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:55:54.992 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:55:54.992 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T08:55:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.995+0000 7f9d3d2d2640 1 -- 172.21.15.28:0/495455978 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9d200a3f20 msgr2=0x7f9d200a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.995+0000 7f9d3d2d2640 1 --2- 172.21.15.28:0/495455978 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9d200a3f20 0x7f9d200a63e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9d38110ec0 tx=0x7f9d2c00a2d0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.995+0000 7f9d3d2d2640 1 -- 172.21.15.28:0/495455978 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d38069cb0 msgr2=0x7f9d38115690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.996+0000 7f9d3d2d2640 1 --2- 172.21.15.28:0/495455978 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d38069cb0 0x7f9d38115690 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000ef90 tx=0x7f9d3000c800 comp rx=0 tx=0).stop 2024-04-21T08:55:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.997+0000 7f9d3d2d2640 1 -- 172.21.15.28:0/495455978 shutdown_connections 2024-04-21T08:55:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.997+0000 7f9d3d2d2640 1 --2- 172.21.15.28:0/495455978 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9d200a3f20 0x7f9d200a63e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.997+0000 7f9d3d2d2640 1 --2- 172.21.15.28:0/495455978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d38107380 0x7f9d3810fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.997+0000 7f9d3d2d2640 1 --2- 172.21.15.28:0/495455978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d38106f10 0x7f9d3810f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.998+0000 7f9d3d2d2640 1 --2- 172.21.15.28:0/495455978 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d38069cb0 0x7f9d38115690 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:54.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.998+0000 7f9d3d2d2640 1 -- 172.21.15.28:0/495455978 >> 172.21.15.28:0/495455978 conn(0x7f9d38077e30 msgr2=0x7f9d381038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:54.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.998+0000 7f9d3d2d2640 1 -- 172.21.15.28:0/495455978 shutdown_connections 2024-04-21T08:55:54.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:54.998+0000 7f9d3d2d2640 1 -- 172.21.15.28:0/495455978 wait complete. 2024-04-21T08:55:55.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:55:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.214+0000 7ffa272cc640 1 -- 172.21.15.28:0/3373748344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1809d220 msgr2=0x7ffa180a4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.214+0000 7ffa272cc640 1 --2- 172.21.15.28:0/3373748344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1809d220 0x7ffa180a4df0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7ffa20067c50 tx=0x7ffa2008f890 comp rx=0 tx=0).stop 2024-04-21T08:55:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.215+0000 7ffa272cc640 1 -- 172.21.15.28:0/3373748344 shutdown_connections 2024-04-21T08:55:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.215+0000 7ffa272cc640 1 --2- 172.21.15.28:0/3373748344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1809d220 0x7ffa180a4df0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.215+0000 7ffa272cc640 1 --2- 172.21.15.28:0/3373748344 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa18098b00 0x7ffa1809cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.215+0000 7ffa272cc640 1 --2- 172.21.15.28:0/3373748344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa18098150 0x7ffa18098530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.215+0000 7ffa272cc640 1 -- 172.21.15.28:0/3373748344 >> 172.21.15.28:0/3373748344 conn(0x7ffa180939f0 msgr2=0x7ffa18095e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.218+0000 7ffa272cc640 1 -- 172.21.15.28:0/3373748344 shutdown_connections 2024-04-21T08:55:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.218+0000 7ffa272cc640 1 -- 172.21.15.28:0/3373748344 wait complete. 2024-04-21T08:55:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.219+0000 7ffa272cc640 1 Processor -- start 2024-04-21T08:55:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.220+0000 7ffa272cc640 1 -- start start 2024-04-21T08:55:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.220+0000 7ffa272cc640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa18098150 0x7ffa18009040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.220+0000 7ffa272cc640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa1809d220 0x7ffa18009580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.220+0000 7ffa272cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa18009ac0 0x7ffa1813ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.220+0000 7ffa272cc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa180a8110 con 0x7ffa18009ac0 2024-04-21T08:55:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.220+0000 7ffa272cc640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ffa180a7f90 con 0x7ffa1809d220 2024-04-21T08:55:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.220+0000 7ffa272cc640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffa180a8290 con 0x7ffa18098150 2024-04-21T08:55:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.221+0000 7ffa26acb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa18009ac0 0x7ffa1813ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.221+0000 7ffa26acb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa18009ac0 0x7ffa1813ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46098/0 (socket says 172.21.15.28:46098) 2024-04-21T08:55:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.221+0000 7ffa26acb640 1 -- 172.21.15.28:0/2445983808 learned_addr learned my addr 172.21.15.28:0/2445983808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.222+0000 7ffa25ac9640 1 --2- 172.21.15.28:0/2445983808 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa1809d220 0x7ffa18009580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.222+0000 7ffa26acb640 1 -- 172.21.15.28:0/2445983808 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa18098150 msgr2=0x7ffa18009040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.222+0000 7ffa26acb640 1 --2- 172.21.15.28:0/2445983808 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa18098150 0x7ffa18009040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.222+0000 7ffa26acb640 1 -- 172.21.15.28:0/2445983808 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa1809d220 msgr2=0x7ffa18009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.222+0000 7ffa26acb640 1 --2- 172.21.15.28:0/2445983808 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa1809d220 0x7ffa18009580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.222+0000 7ffa26acb640 1 -- 172.21.15.28:0/2445983808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1813efc0 con 0x7ffa18009ac0 2024-04-21T08:55:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.222+0000 7ffa25ac9640 1 --2- 172.21.15.28:0/2445983808 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa1809d220 0x7ffa18009580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:55:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.223+0000 7ffa26acb640 1 --2- 172.21.15.28:0/2445983808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa18009ac0 0x7ffa1813ea80 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ffa20050f50 tx=0x7ffa20068f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.225+0000 7ffa177fe640 1 -- 172.21.15.28:0/2445983808 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa200a4070 con 0x7ffa18009ac0 2024-04-21T08:55:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.225+0000 7ffa177fe640 1 -- 172.21.15.28:0/2445983808 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa20096b10 con 0x7ffa18009ac0 2024-04-21T08:55:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.225+0000 7ffa177fe640 1 -- 172.21.15.28:0/2445983808 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2009b040 con 0x7ffa18009ac0 2024-04-21T08:55:55.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.225+0000 7ffa272cc640 1 -- 172.21.15.28:0/2445983808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa1813f250 con 0x7ffa18009ac0 2024-04-21T08:55:55.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.225+0000 7ffa272cc640 1 -- 172.21.15.28:0/2445983808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa1813f600 con 0x7ffa18009ac0 2024-04-21T08:55:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.230+0000 7ffa2b1f3740 1 Processor -- start 2024-04-21T08:55:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.237+0000 7ffa2b1f3740 1 -- start start 2024-04-21T08:55:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.237+0000 7ffa2b1f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b1496d0 0x55e51b149ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.237+0000 7ffa2b1f3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b14a080 0x55e51b152630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.237+0000 7ffa2b1f3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b152c00 0x55e51b155050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.237+0000 7ffa2b1f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e51b0ae1f0 con 0x55e51b1496d0 2024-04-21T08:55:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.237+0000 7ffa2b1f3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e51b04b4b0 con 0x55e51b152c00 2024-04-21T08:55:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.237+0000 7ffa2b1f3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55e51b0aa1b0 con 0x55e51b14a080 2024-04-21T08:55:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.237+0000 7ffa25ac9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b1496d0 0x55e51b149ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.238+0000 7ffa25ac9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b1496d0 0x55e51b149ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46108/0 (socket says 172.21.15.28:46108) 2024-04-21T08:55:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.238+0000 7ffa25ac9640 1 -- 172.21.15.28:0/235219014 learned_addr learned my addr 172.21.15.28:0/235219014 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.238+0000 7ffa262ca640 1 --2- 172.21.15.28:0/235219014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b14a080 0x55e51b152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.238+0000 7ffa177fe640 1 -- 172.21.15.28:0/2445983808 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffa200a7070 con 0x7ffa18009ac0 2024-04-21T08:55:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.238+0000 7ffa262ca640 1 -- 172.21.15.28:0/235219014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b152c00 msgr2=0x55e51b155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.238+0000 7ffa262ca640 1 --2- 172.21.15.28:0/235219014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b152c00 0x55e51b155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.238+0000 7ffa262ca640 1 -- 172.21.15.28:0/235219014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b1496d0 msgr2=0x55e51b149ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.238+0000 7ffa262ca640 1 --2- 172.21.15.28:0/235219014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b1496d0 0x55e51b149ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.238+0000 7ffa262ca640 1 -- 172.21.15.28:0/235219014 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e51b03d1b0 con 0x55e51b14a080 2024-04-21T08:55:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.239+0000 7ffa262ca640 1 --2- 172.21.15.28:0/235219014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b14a080 0x55e51b152630 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c01ab00 tx=0x7ffa1c01f360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49bbf9868107469c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.240+0000 7ffa14bfe640 1 -- 172.21.15.28:0/235219014 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c011030 con 0x55e51b14a080 2024-04-21T08:55:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.240+0000 7ffa177fe640 1 --2- 172.21.15.28:0/2445983808 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ffa0c0a3f20 0x7ffa0c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.240+0000 7ffa14bfe640 1 -- 172.21.15.28:0/235219014 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa1c01fd20 con 0x55e51b14a080 2024-04-21T08:55:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.240+0000 7ffa14bfe640 1 -- 172.21.15.28:0/235219014 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c004a00 con 0x55e51b14a080 2024-04-21T08:55:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.241+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/235219014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b14a080 msgr2=0x55e51b152630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.241+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/235219014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b14a080 0x55e51b152630 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c01ab00 tx=0x7ffa1c01f360 comp rx=0 tx=0).stop 2024-04-21T08:55:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.241+0000 7ffa177fe640 1 -- 172.21.15.28:0/2445983808 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ffa20155eb0 con 0x7ffa18009ac0 2024-04-21T08:55:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.241+0000 7ffa25ac9640 1 --2- 172.21.15.28:0/2445983808 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ffa0c0a3f20 0x7ffa0c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.242+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/235219014 shutdown_connections 2024-04-21T08:55:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.242+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/235219014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b152c00 0x55e51b155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.242+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/235219014 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b14a080 0x55e51b152630 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.242+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/235219014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b1496d0 0x55e51b149ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.242+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/235219014 >> 172.21.15.28:0/235219014 conn(0x55e51b143da0 msgr2=0x55e51b144180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.242+0000 7ffa25ac9640 1 --2- 172.21.15.28:0/2445983808 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ffa0c0a3f20 0x7ffa0c0a63e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ffa10005f60 tx=0x7ffa100096f0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.246+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/235219014 shutdown_connections 2024-04-21T08:55:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.248+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/235219014 wait complete. 2024-04-21T08:55:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.248+0000 7ffa2b1f3740 1 -- start start 2024-04-21T08:55:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.248+0000 7ffa2b1f3740 1 Processor -- start 2024-04-21T08:55:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.248+0000 7ffa2b1f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b1496d0 0x55e51b152480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.249+0000 7ffa2b1f3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b14a080 0x55e51b14f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.249+0000 7ffa2b1f3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b152c00 0x55e51b14fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.249+0000 7ffa2b1f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e51b155b70 con 0x55e51b1496d0 2024-04-21T08:55:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.249+0000 7ffa2b1f3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e51b067500 con 0x55e51b14a080 2024-04-21T08:55:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.249+0000 7ffa2b1f3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55e51b155cf0 con 0x55e51b152c00 2024-04-21T08:55:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.249+0000 7ffa262ca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b1496d0 0x55e51b152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.249+0000 7ffa262ca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b1496d0 0x55e51b152480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46124/0 (socket says 172.21.15.28:46124) 2024-04-21T08:55:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.249+0000 7ffa262ca640 1 -- 172.21.15.28:0/604287995 learned_addr learned my addr 172.21.15.28:0/604287995 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.249+0000 7ffa2b1f3740 10 client.0 _ll_register_callbacks cb 0x55e51b035b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.249+0000 7ffa262ca640 1 -- 172.21.15.28:0/604287995 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b152c00 msgr2=0x55e51b14fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.249+0000 7ffa262ca640 1 --2- 172.21.15.28:0/604287995 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b152c00 0x55e51b14fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.249+0000 7ffa262ca640 1 -- 172.21.15.28:0/604287995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b14a080 msgr2=0x55e51b14f810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:55.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.249+0000 7ffa262ca640 1 --2- 172.21.15.28:0/604287995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b14a080 0x55e51b14f810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.249+0000 7ffa262ca640 1 -- 172.21.15.28:0/604287995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e51adc03f0 con 0x55e51b1496d0 2024-04-21T08:55:55.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.250+0000 7ffa262ca640 1 --2- 172.21.15.28:0/604287995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b1496d0 0x55e51b152480 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c005330 tx=0x7ffa1c01a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48701da2cf94095f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.252+0000 7ffa09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:55:55.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.252+0000 7ffa09ffb640 1 -- 172.21.15.28:0/604287995 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c048060 con 0x55e51b1496d0 2024-04-21T08:55:55.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.253+0000 7ffa09ffb640 1 -- 172.21.15.28:0/604287995 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ffa1c004430 con 0x55e51b1496d0 2024-04-21T08:55:55.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.253+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/604287995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa1c01fd20 con 0x55e51b1496d0 2024-04-21T08:55:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.257+0000 7ffa09ffb640 1 -- 172.21.15.28:0/604287995 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffa1c0035f0 con 0x55e51b1496d0 2024-04-21T08:55:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.257+0000 7ffa2b1f3740 10 client.17100 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.257+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/604287995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e51ae8a7d0 con 0x55e51b1496d0 2024-04-21T08:55:55.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.258+0000 7ffa09ffb640 1 -- 172.21.15.28:0/604287995 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa1c01fd20 con 0x55e51b1496d0 2024-04-21T08:55:55.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.258+0000 7ffa09ffb640 1 -- 172.21.15.28:0/604287995 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7ffa1c004d10 con 0x55e51b1496d0 2024-04-21T08:55:55.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.261+0000 7ffa2b1f3740 10 client.17100 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:55.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.261+0000 7ffa2b1f3740 10 client.17100 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:55:55.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.261+0000 7ffa2b1f3740 20 client.17100 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:55.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.261+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/604287995 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55e51b162a70 0x55e51b164e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.261+0000 7ffa2b1f3740 4 client.17100 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:55.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.261+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/604287995 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e51b04d930 con 0x55e51b162a70 2024-04-21T08:55:55.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.263+0000 7ffa25ac9640 1 --2- 172.21.15.28:0/604287995 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55e51b162a70 0x55e51b164e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.264+0000 7ffa25ac9640 1 --2- 172.21.15.28:0/604287995 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55e51b162a70 0x55e51b164e60 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5c450fe5831a6e15 server_cookie=b5962103504d1cf7 in_seq=0 out_seq=0 2024-04-21T08:55:55.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.264+0000 7ffa09ffb640 10 client.17100 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:55:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.267+0000 7ffa09ffb640 1 -- 172.21.15.28:0/604287995 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e51b04d930 con 0x55e51b162a70 2024-04-21T08:55:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.268+0000 7ffa09ffb640 10 client.17100 handle_command_reply: tid=0 2024-04-21T08:55:55.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.268+0000 7ffa2b1f3740 1 client.17100 shutdown 2024-04-21T08:55:55.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.272+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/604287995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b1496d0 msgr2=0x55e51b152480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.272+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/604287995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b1496d0 0x55e51b152480 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c005330 tx=0x7ffa1c01a270 comp rx=0 tx=0).stop 2024-04-21T08:55:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.273+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/604287995 shutdown_connections 2024-04-21T08:55:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.273+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/604287995 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55e51b162a70 0x55e51b164e60 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.273+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/604287995 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b152c00 0x55e51b14fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.273+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/604287995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b14a080 0x55e51b14f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.273+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/604287995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b1496d0 0x55e51b152480 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.273+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/604287995 >> 172.21.15.28:0/604287995 conn(0x55e51b143da0 msgr2=0x55e51b145af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.273+0000 7ffa09ffb640 0 client.17100 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:55:55.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.273+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/604287995 shutdown_connections 2024-04-21T08:55:55.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.274+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/604287995 wait complete. 2024-04-21T08:55:55.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.277+0000 7ffa2b1f3740 20 client.17100 trim_cache size 0 max 16384 2024-04-21T08:55:55.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.308+0000 7ffa2b1f3740 1 Processor -- start 2024-04-21T08:55:55.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.311+0000 7ffa2b1f3740 1 -- start start 2024-04-21T08:55:55.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.311+0000 7ffa2b1f3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b1496d0 0x55e51b15eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.311+0000 7ffa2b1f3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b14a080 0x55e51b151f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.311+0000 7ffa2b1f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b152c00 0x55e51b152480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.311+0000 7ffa2b1f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa04012880 con 0x55e51b152c00 2024-04-21T08:55:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.311+0000 7ffa2b1f3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ffa04010150 con 0x55e51b1496d0 2024-04-21T08:55:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.311+0000 7ffa2b1f3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0400dfa0 con 0x55e51b14a080 2024-04-21T08:55:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.311+0000 7ffa262ca640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b1496d0 0x55e51b15eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.311+0000 7ffa262ca640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b1496d0 0x55e51b15eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:37836/0 (socket says 172.21.15.28:37836) 2024-04-21T08:55:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.311+0000 7ffa262ca640 1 -- 172.21.15.28:0/1823014094 learned_addr learned my addr 172.21.15.28:0/1823014094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.311+0000 7ffa25ac9640 1 --2- 172.21.15.28:0/1823014094 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b14a080 0x55e51b151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.312+0000 7ffa262ca640 1 -- 172.21.15.28:0/1823014094 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b14a080 msgr2=0x55e51b151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.312+0000 7ffa262ca640 1 --2- 172.21.15.28:0/1823014094 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b14a080 0x55e51b151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.312+0000 7ffa262ca640 1 -- 172.21.15.28:0/1823014094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b152c00 msgr2=0x55e51b152480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.312+0000 7ffa262ca640 1 --2- 172.21.15.28:0/1823014094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b152c00 0x55e51b152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.312+0000 7ffa262ca640 1 -- 172.21.15.28:0/1823014094 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e51b1500b0 con 0x55e51b1496d0 2024-04-21T08:55:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.315+0000 7ffa262ca640 1 --2- 172.21.15.28:0/1823014094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b1496d0 0x55e51b15eac0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0050b0 tx=0x7ffa1c005df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89ba3de21d842a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.315+0000 7ffa0bfff640 1 -- 172.21.15.28:0/1823014094 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c011070 con 0x55e51b1496d0 2024-04-21T08:55:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.316+0000 7ffa0bfff640 1 -- 172.21.15.28:0/1823014094 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa1c0159c0 con 0x55e51b1496d0 2024-04-21T08:55:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.317+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/1823014094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b1496d0 msgr2=0x55e51b15eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.317+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/1823014094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b1496d0 0x55e51b15eac0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0050b0 tx=0x7ffa1c005df0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.317+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/1823014094 shutdown_connections 2024-04-21T08:55:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.317+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/1823014094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b152c00 0x55e51b152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.317+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/1823014094 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b14a080 0x55e51b151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.317+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/1823014094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b1496d0 0x55e51b15eac0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.317+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/1823014094 >> 172.21.15.28:0/1823014094 conn(0x55e51b143da0 msgr2=0x55e51b142e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.318+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/1823014094 shutdown_connections 2024-04-21T08:55:55.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.318+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/1823014094 wait complete. 2024-04-21T08:55:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.318+0000 7ffa2b1f3740 1 -- start start 2024-04-21T08:55:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.318+0000 7ffa2b1f3740 1 Processor -- start 2024-04-21T08:55:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.319+0000 7ffa2b1f3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b1496d0 0x55e51b15b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.319+0000 7ffa2b1f3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b14a080 0x55e51b15b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.319+0000 7ffa2b1f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b152c00 0x55e51b156580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.319+0000 7ffa2b1f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0400e610 con 0x55e51b152c00 2024-04-21T08:55:55.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.319+0000 7ffa2b1f3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0400fa80 con 0x55e51b14a080 2024-04-21T08:55:55.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.319+0000 7ffa2b1f3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0400ece0 con 0x55e51b1496d0 2024-04-21T08:55:55.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.319+0000 7ffa2b1f3740 10 client.0 _ll_register_callbacks cb 0x7ffa1c01aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:55.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.319+0000 7ffa26acb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b152c00 0x55e51b156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.319+0000 7ffa262ca640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b1496d0 0x55e51b15b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.320+0000 7ffa262ca640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b1496d0 0x55e51b15b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:55252/0 (socket says 172.21.15.28:55252) 2024-04-21T08:55:55.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.320+0000 7ffa262ca640 1 -- 172.21.15.28:0/2758404262 learned_addr learned my addr 172.21.15.28:0/2758404262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:55.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.320+0000 7ffa25ac9640 1 --2- 172.21.15.28:0/2758404262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b14a080 0x55e51b15b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.321+0000 7ffa25ac9640 1 -- 172.21.15.28:0/2758404262 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b1496d0 msgr2=0x55e51b15b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.321+0000 7ffa25ac9640 1 --2- 172.21.15.28:0/2758404262 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b1496d0 0x55e51b15b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.321+0000 7ffa25ac9640 1 -- 172.21.15.28:0/2758404262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b152c00 msgr2=0x55e51b156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.321+0000 7ffa25ac9640 1 --2- 172.21.15.28:0/2758404262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b152c00 0x55e51b156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.321+0000 7ffa25ac9640 1 -- 172.21.15.28:0/2758404262 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e51b156d80 con 0x55e51b14a080 2024-04-21T08:55:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.324+0000 7ffa25ac9640 1 --2- 172.21.15.28:0/2758404262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b14a080 0x55e51b15b540 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x55e51b165490 tx=0x7ffa10015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7cdec9c019e096a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.324+0000 7ffa09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:55:55.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.324+0000 7ffa09ffb640 1 -- 172.21.15.28:0/2758404262 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa100275e0 con 0x55e51b14a080 2024-04-21T08:55:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.324+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/2758404262 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa1c0159c0 con 0x55e51b14a080 2024-04-21T08:55:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.324+0000 7ffa09ffb640 1 -- 172.21.15.28:0/2758404262 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ffa10027780 con 0x55e51b14a080 2024-04-21T08:55:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.325+0000 7ffa09ffb640 1 -- 172.21.15.28:0/2758404262 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffa10039050 con 0x55e51b14a080 2024-04-21T08:55:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.325+0000 7ffa2b1f3740 10 client.26363 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:55.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.325+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/2758404262 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e51b1c8aa0 con 0x55e51b14a080 2024-04-21T08:55:55.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.326+0000 7ffa09ffb640 1 -- 172.21.15.28:0/2758404262 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa1c0159c0 con 0x55e51b14a080 2024-04-21T08:55:55.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.326+0000 7ffa09ffb640 1 -- 172.21.15.28:0/2758404262 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7ffa1002e020 con 0x55e51b14a080 2024-04-21T08:55:55.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.326+0000 7ffa2b1f3740 10 client.26363 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:55.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.326+0000 7ffa2b1f3740 10 client.26363 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:55:55.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.326+0000 7ffa2b1f3740 20 client.26363 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:55.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.326+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/2758404262 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55e51b1c91d0 0x55e51b1cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.326+0000 7ffa2b1f3740 4 client.26363 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:55:55.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.326+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/2758404262 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e51b0bbc10 con 0x55e51b1c91d0 2024-04-21T08:55:55.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.327+0000 7ffa262ca640 1 --2- 172.21.15.28:0/2758404262 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55e51b1c91d0 0x55e51b1cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.327+0000 7ffa09ffb640 10 client.26363 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:55:55.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.328+0000 7ffa262ca640 1 --2- 172.21.15.28:0/2758404262 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55e51b1c91d0 0x55e51b1cb5f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b23d00a93cfbb184 server_cookie=8004b3f6e975779c in_seq=0 out_seq=0 2024-04-21T08:55:55.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.328+0000 7ffa09ffb640 1 -- 172.21.15.28:0/2758404262 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e51b0bbc10 con 0x55e51b1c91d0 2024-04-21T08:55:55.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.328+0000 7ffa09ffb640 10 client.26363 handle_command_reply: tid=0 2024-04-21T08:55:55.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.328+0000 7ffa2b1f3740 1 client.26363 shutdown 2024-04-21T08:55:55.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.329+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/2758404262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b14a080 msgr2=0x55e51b15b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.329+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/2758404262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b14a080 0x55e51b15b540 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x55e51b165490 tx=0x7ffa10015040 comp rx=0 tx=0).stop 2024-04-21T08:55:55.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.331+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/2758404262 shutdown_connections 2024-04-21T08:55:55.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.331+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/2758404262 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55e51b1c91d0 0x55e51b1cb5f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.331+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/2758404262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e51b152c00 0x55e51b156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.331+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/2758404262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e51b14a080 0x55e51b15b540 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.331+0000 7ffa2b1f3740 1 --2- 172.21.15.28:0/2758404262 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e51b1496d0 0x55e51b15b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.331+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/2758404262 >> 172.21.15.28:0/2758404262 conn(0x55e51b143da0 msgr2=0x55e51b142590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.332+0000 7ffa09ffb640 0 client.26363 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:55:55.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.332+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/2758404262 shutdown_connections 2024-04-21T08:55:55.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.332+0000 7ffa2b1f3740 1 -- 172.21.15.28:0/2758404262 wait complete. 2024-04-21T08:55:55.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.332+0000 7ffa2b1f3740 20 client.26363 trim_cache size 0 max 16384 2024-04-21T08:55:55.333 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:55:55.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.334+0000 7ffa272cc640 1 -- 172.21.15.28:0/2445983808 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ffa0c0a3f20 msgr2=0x7ffa0c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.334+0000 7ffa272cc640 1 --2- 172.21.15.28:0/2445983808 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ffa0c0a3f20 0x7ffa0c0a63e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ffa10005f60 tx=0x7ffa100096f0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.334+0000 7ffa272cc640 1 -- 172.21.15.28:0/2445983808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa18009ac0 msgr2=0x7ffa1813ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.334+0000 7ffa272cc640 1 --2- 172.21.15.28:0/2445983808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa18009ac0 0x7ffa1813ea80 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ffa20050f50 tx=0x7ffa20068f20 comp rx=0 tx=0).stop 2024-04-21T08:55:55.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.334+0000 7ffa272cc640 1 -- 172.21.15.28:0/2445983808 shutdown_connections 2024-04-21T08:55:55.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.334+0000 7ffa272cc640 1 --2- 172.21.15.28:0/2445983808 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ffa0c0a3f20 0x7ffa0c0a63e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.334+0000 7ffa272cc640 1 --2- 172.21.15.28:0/2445983808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa18009ac0 0x7ffa1813ea80 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.334+0000 7ffa272cc640 1 --2- 172.21.15.28:0/2445983808 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa1809d220 0x7ffa18009580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.334+0000 7ffa272cc640 1 --2- 172.21.15.28:0/2445983808 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa18098150 0x7ffa18009040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.334+0000 7ffa272cc640 1 -- 172.21.15.28:0/2445983808 >> 172.21.15.28:0/2445983808 conn(0x7ffa180939f0 msgr2=0x7ffa18094410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.335+0000 7ffa272cc640 1 -- 172.21.15.28:0/2445983808 shutdown_connections 2024-04-21T08:55:55.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.335+0000 7ffa272cc640 1 -- 172.21.15.28:0/2445983808 wait complete. 2024-04-21T08:55:55.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:55 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/495455978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:55:55.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:55:55.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:55 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/495455978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:55:55.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:55 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/495455978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:55:55.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.549+0000 7fb93bb24640 1 -- 172.21.15.28:0/863818311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb93410e7d0 msgr2=0x7fb93410ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.549+0000 7fb93bb24640 1 --2- 172.21.15.28:0/863818311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb93410e7d0 0x7fb93410ec50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb924009930 tx=0x7fb92402bc30 comp rx=0 tx=0).stop 2024-04-21T08:55:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.550+0000 7fb93bb24640 1 -- 172.21.15.28:0/863818311 shutdown_connections 2024-04-21T08:55:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.550+0000 7fb93bb24640 1 --2- 172.21.15.28:0/863818311 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb93411b730 0x7fb93411db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.550+0000 7fb93bb24640 1 --2- 172.21.15.28:0/863818311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb93410e7d0 0x7fb93410ec50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.550+0000 7fb93bb24640 1 --2- 172.21.15.28:0/863818311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9341107a0 0x7fb93410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.550+0000 7fb93bb24640 1 -- 172.21.15.28:0/863818311 >> 172.21.15.28:0/863818311 conn(0x7fb93406d060 msgr2=0x7fb93406d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.555+0000 7fb93bb24640 1 -- 172.21.15.28:0/863818311 shutdown_connections 2024-04-21T08:55:55.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.556+0000 7fb93bb24640 1 -- 172.21.15.28:0/863818311 wait complete. 2024-04-21T08:55:55.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.557+0000 7fb93bb24640 1 Processor -- start 2024-04-21T08:55:55.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.557+0000 7fb93bb24640 1 -- start start 2024-04-21T08:55:55.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.557+0000 7fb93bb24640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb93410e7d0 0x7fb9341a75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.557+0000 7fb93bb24640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9341107a0 0x7fb9341a7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.558+0000 7fb93bb24640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb93411b730 0x7fb9341abe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.558+0000 7fb93bb24640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb934120fa0 con 0x7fb93410e7d0 2024-04-21T08:55:55.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.558+0000 7fb93bb24640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb934120e20 con 0x7fb93411b730 2024-04-21T08:55:55.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.558+0000 7fb93bb24640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb934121120 con 0x7fb9341107a0 2024-04-21T08:55:55.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.558+0000 7fb93a0c1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb93411b730 0x7fb9341abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.559+0000 7fb93a0c1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb93411b730 0x7fb9341abe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:37866/0 (socket says 172.21.15.28:37866) 2024-04-21T08:55:55.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.559+0000 7fb93a0c1640 1 -- 172.21.15.28:0/3972509006 learned_addr learned my addr 172.21.15.28:0/3972509006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:55.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.559+0000 7fb9398c0640 1 --2- 172.21.15.28:0/3972509006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb93410e7d0 0x7fb9341a75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.559+0000 7fb93a0c1640 1 -- 172.21.15.28:0/3972509006 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9341107a0 msgr2=0x7fb9341a7ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:55:55.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.559+0000 7fb93a0c1640 1 --2- 172.21.15.28:0/3972509006 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9341107a0 0x7fb9341a7ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.559+0000 7fb93a0c1640 1 -- 172.21.15.28:0/3972509006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb93410e7d0 msgr2=0x7fb9341a75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.559+0000 7fb93a0c1640 1 --2- 172.21.15.28:0/3972509006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb93410e7d0 0x7fb9341a75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.559+0000 7fb93a0c1640 1 -- 172.21.15.28:0/3972509006 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9341ac540 con 0x7fb93411b730 2024-04-21T08:55:55.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.559+0000 7fb93a0c1640 1 --2- 172.21.15.28:0/3972509006 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb93411b730 0x7fb9341abe10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00eaa0 tx=0x7fb92c00ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.560+0000 7fb92affd640 1 -- 172.21.15.28:0/3972509006 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb92c00cdb0 con 0x7fb93411b730 2024-04-21T08:55:55.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.560+0000 7fb93bb24640 1 -- 172.21.15.28:0/3972509006 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9341ac830 con 0x7fb93411b730 2024-04-21T08:55:55.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.560+0000 7fb93bb24640 1 -- 172.21.15.28:0/3972509006 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9341b40c0 con 0x7fb93411b730 2024-04-21T08:55:55.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.560+0000 7fb92affd640 1 -- 172.21.15.28:0/3972509006 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb92c005020 con 0x7fb93411b730 2024-04-21T08:55:55.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.560+0000 7fb92affd640 1 -- 172.21.15.28:0/3972509006 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb92c0116b0 con 0x7fb93411b730 2024-04-21T08:55:55.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.565+0000 7fb93e7f2740 1 Processor -- start 2024-04-21T08:55:55.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.566+0000 7fb93e7f2740 1 -- start start 2024-04-21T08:55:55.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.567+0000 7fb93e7f2740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 0x55ac74c7c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.567+0000 7fb93e7f2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c7ce80 0x55ac74c85430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.567+0000 7fb93e7f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c85a00 0x55ac74c87e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.567+0000 7fb9390bf640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 0x55ac74c7c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.567+0000 7fb9390bf640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 0x55ac74c7c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:55282/0 (socket says 172.21.15.28:55282) 2024-04-21T08:55:55.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.567+0000 7fb9390bf640 1 -- 172.21.15.28:0/3883879222 learned_addr learned my addr 172.21.15.28:0/3883879222 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:55.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.568+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3883879222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ac74be11f0 con 0x55ac74c85a00 2024-04-21T08:55:55.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.568+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3883879222 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55ac74b7e4b0 con 0x55ac74c7ce80 2024-04-21T08:55:55.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.568+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3883879222 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55ac74bdd1b0 con 0x55ac74c7c4d0 2024-04-21T08:55:55.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.568+0000 7fb9390bf640 1 --2- 172.21.15.28:0/3883879222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c85a00 0x55ac74c87e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.568+0000 7fb9390bf640 1 -- 172.21.15.28:0/3883879222 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c7ce80 msgr2=0x55ac74c85430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:55.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.568+0000 7fb9390bf640 1 --2- 172.21.15.28:0/3883879222 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c7ce80 0x55ac74c85430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.568+0000 7fb9390bf640 1 -- 172.21.15.28:0/3883879222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c85a00 msgr2=0x55ac74c87e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.568+0000 7fb9390bf640 1 --2- 172.21.15.28:0/3883879222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c85a00 0x55ac74c87e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.568+0000 7fb9390bf640 1 -- 172.21.15.28:0/3883879222 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac74b701b0 con 0x55ac74c7c4d0 2024-04-21T08:55:55.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.569+0000 7fb9390bf640 1 --2- 172.21.15.28:0/3883879222 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 0x55ac74c7c8b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb924005ab0 tx=0x7fb92402c150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f47ba9ecf3b7326c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.569+0000 7fb92affd640 1 -- 172.21.15.28:0/3972509006 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb92c004a40 con 0x7fb93411b730 2024-04-21T08:55:55.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.570+0000 7fb92affd640 1 --2- 172.21.15.28:0/3972509006 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb9200a3e60 0x7fb9200a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.570+0000 7fb91ffff640 1 -- 172.21.15.28:0/3883879222 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb924004100 con 0x55ac74c7c4d0 2024-04-21T08:55:55.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.570+0000 7fb91ffff640 1 -- 172.21.15.28:0/3883879222 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb92402cc90 con 0x55ac74c7c4d0 2024-04-21T08:55:55.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.570+0000 7fb91ffff640 1 -- 172.21.15.28:0/3883879222 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb924003c40 con 0x55ac74c7c4d0 2024-04-21T08:55:55.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.571+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3883879222 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 msgr2=0x55ac74c7c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.571+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/3883879222 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 0x55ac74c7c8b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb924005ab0 tx=0x7fb92402c150 comp rx=0 tx=0).stop 2024-04-21T08:55:55.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.572+0000 7fb92affd640 1 -- 172.21.15.28:0/3972509006 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fb92c0d0c80 con 0x7fb93411b730 2024-04-21T08:55:55.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.572+0000 7fb9398c0640 1 --2- 172.21.15.28:0/3972509006 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb9200a3e60 0x7fb9200a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.576+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3883879222 shutdown_connections 2024-04-21T08:55:55.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.576+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/3883879222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c85a00 0x55ac74c87e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.576+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/3883879222 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c7ce80 0x55ac74c85430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.576+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/3883879222 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 0x55ac74c7c8b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.576+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3883879222 >> 172.21.15.28:0/3883879222 conn(0x55ac74c76ba0 msgr2=0x55ac74c76f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.580+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3883879222 shutdown_connections 2024-04-21T08:55:55.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.580+0000 7fb9398c0640 1 --2- 172.21.15.28:0/3972509006 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb9200a3e60 0x7fb9200a6320 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x55ac74c77c10 tx=0x7fb930005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.582+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3883879222 wait complete. 2024-04-21T08:55:55.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.583+0000 7fb93e7f2740 1 -- start start 2024-04-21T08:55:55.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.583+0000 7fb93e7f2740 1 Processor -- start 2024-04-21T08:55:55.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.587+0000 7fb93e7f2740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 0x55ac74c85330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.587+0000 7fb93e7f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7ce80 0x55ac74c826c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.587+0000 7fb93e7f2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c85a00 0x55ac74c82cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.587+0000 7fb93e7f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ac74c88910 con 0x55ac74c7ce80 2024-04-21T08:55:55.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.587+0000 7fb93e7f2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55ac74b9a500 con 0x55ac74c85a00 2024-04-21T08:55:55.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.587+0000 7fb93e7f2740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55ac74c88a90 con 0x55ac74c7c4d0 2024-04-21T08:55:55.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.587+0000 7fb93e7f2740 10 client.0 _ll_register_callbacks cb 0x55ac74b68b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:55.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.588+0000 7fb9398c0640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 0x55ac74c85330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.588+0000 7fb9398c0640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 0x55ac74c85330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:55288/0 (socket says 172.21.15.28:55288) 2024-04-21T08:55:55.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.588+0000 7fb9398c0640 1 -- 172.21.15.28:0/3519916872 learned_addr learned my addr 172.21.15.28:0/3519916872 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:55.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.588+0000 7fb93a0c1640 1 --2- 172.21.15.28:0/3519916872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c85a00 0x55ac74c82cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.589+0000 7fb9390bf640 1 --2- 172.21.15.28:0/3519916872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7ce80 0x55ac74c826c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.589+0000 7fb9398c0640 1 -- 172.21.15.28:0/3519916872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c85a00 msgr2=0x55ac74c82cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.589+0000 7fb9398c0640 1 --2- 172.21.15.28:0/3519916872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c85a00 0x55ac74c82cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.589+0000 7fb9398c0640 1 -- 172.21.15.28:0/3519916872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7ce80 msgr2=0x55ac74c826c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.589+0000 7fb9398c0640 1 --2- 172.21.15.28:0/3519916872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7ce80 0x55ac74c826c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.589+0000 7fb9398c0640 1 -- 172.21.15.28:0/3519916872 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac748f33f0 con 0x55ac74c7c4d0 2024-04-21T08:55:55.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.590+0000 7fb9398c0640 1 --2- 172.21.15.28:0/3519916872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 0x55ac74c85330 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb9300127a0 tx=0x7fb930004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92e1eb3410ac3cf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.590+0000 7fb91d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:55:55.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.591+0000 7fb91d7fa640 1 -- 172.21.15.28:0/3519916872 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93000fb50 con 0x55ac74c7c4d0 2024-04-21T08:55:55.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.592+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3519916872 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb92402cc90 con 0x55ac74c7c4d0 2024-04-21T08:55:55.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.592+0000 7fb91d7fa640 1 -- 172.21.15.28:0/3519916872 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fb93000fcf0 con 0x55ac74c7c4d0 2024-04-21T08:55:55.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.596+0000 7fb91d7fa640 1 -- 172.21.15.28:0/3519916872 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb930030050 con 0x55ac74c7c4d0 2024-04-21T08:55:55.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.598+0000 7fb93e7f2740 10 client.26395 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:55.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.598+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3519916872 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ac749bd7d0 con 0x55ac74c7c4d0 2024-04-21T08:55:55.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.598+0000 7fb91d7fa640 1 -- 172.21.15.28:0/3519916872 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb92402cc90 con 0x55ac74c7c4d0 2024-04-21T08:55:55.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.599+0000 7fb91d7fa640 1 -- 172.21.15.28:0/3519916872 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fb930027500 con 0x55ac74c7c4d0 2024-04-21T08:55:55.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.599+0000 7fb93e7f2740 10 client.26395 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:55.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.599+0000 7fb93e7f2740 10 client.26395 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:55:55.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.599+0000 7fb93e7f2740 20 client.26395 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:55.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.599+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/3519916872 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55ac74c95f80 0x55ac74c98400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.599+0000 7fb93e7f2740 4 client.26395 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:55.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.599+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3519916872 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac74b80930 con 0x55ac74c95f80 2024-04-21T08:55:55.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.600+0000 7fb9390bf640 1 --2- 172.21.15.28:0/3519916872 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55ac74c95f80 0x55ac74c98400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.606+0000 7fb9390bf640 1 --2- 172.21.15.28:0/3519916872 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55ac74c95f80 0x55ac74c98400 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=38170ef60352f1f1 server_cookie=9086d47ec5b8c981 in_seq=0 out_seq=0 2024-04-21T08:55:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.606+0000 7fb91d7fa640 10 client.26395 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:55:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.608+0000 7fb91d7fa640 1 -- 172.21.15.28:0/3519916872 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ac74b80930 con 0x55ac74c95f80 2024-04-21T08:55:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.608+0000 7fb91d7fa640 10 client.26395 handle_command_reply: tid=0 2024-04-21T08:55:55.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.608+0000 7fb93e7f2740 1 client.26395 shutdown 2024-04-21T08:55:55.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.611+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3519916872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 msgr2=0x55ac74c85330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.611+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/3519916872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 0x55ac74c85330 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb9300127a0 tx=0x7fb930004730 comp rx=0 tx=0).stop 2024-04-21T08:55:55.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.613+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3519916872 shutdown_connections 2024-04-21T08:55:55.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.613+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/3519916872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c85a00 0x55ac74c82cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.613+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/3519916872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7ce80 0x55ac74c826c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.613+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/3519916872 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55ac74c95f80 0x55ac74c98400 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.613+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/3519916872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 0x55ac74c85330 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.613+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3519916872 >> 172.21.15.28:0/3519916872 conn(0x55ac74c76ba0 msgr2=0x55ac74c788f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.613+0000 7fb91d7fa640 0 client.26395 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:55:55.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.613+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3519916872 shutdown_connections 2024-04-21T08:55:55.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.613+0000 7fb93e7f2740 1 -- 172.21.15.28:0/3519916872 wait complete. 2024-04-21T08:55:55.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.613+0000 7fb93e7f2740 20 client.26395 trim_cache size 0 max 16384 2024-04-21T08:55:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.647+0000 7fb93e7f2740 1 Processor -- start 2024-04-21T08:55:55.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.649+0000 7fb93e7f2740 1 -- start start 2024-04-21T08:55:55.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.649+0000 7fb93e7f2740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 0x55ac74c71ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.649+0000 7fb93e7f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7ce80 0x55ac74c78a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.649+0000 7fb93e7f2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c85a00 0x55ac74c84df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.649+0000 7fb93e7f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb918012820 con 0x55ac74c7ce80 2024-04-21T08:55:55.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.649+0000 7fb93e7f2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb9180100f0 con 0x55ac74c85a00 2024-04-21T08:55:55.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.649+0000 7fb93e7f2740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb91800df40 con 0x55ac74c7c4d0 2024-04-21T08:55:55.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.649+0000 7fb9390bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7ce80 0x55ac74c78a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.649+0000 7fb9390bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7ce80 0x55ac74c78a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46166/0 (socket says 172.21.15.28:46166) 2024-04-21T08:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.649+0000 7fb9390bf640 1 -- 172.21.15.28:0/1264479220 learned_addr learned my addr 172.21.15.28:0/1264479220 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.650+0000 7fb93a0c1640 1 --2- 172.21.15.28:0/1264479220 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c85a00 0x55ac74c84df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.650+0000 7fb9390bf640 1 -- 172.21.15.28:0/1264479220 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 msgr2=0x55ac74c71ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.650+0000 7fb9390bf640 1 --2- 172.21.15.28:0/1264479220 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 0x55ac74c71ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.650+0000 7fb9390bf640 1 -- 172.21.15.28:0/1264479220 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c85a00 msgr2=0x55ac74c84df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.650+0000 7fb9390bf640 1 --2- 172.21.15.28:0/1264479220 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c85a00 0x55ac74c84df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.650+0000 7fb9390bf640 1 -- 172.21.15.28:0/1264479220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac74c83040 con 0x55ac74c7ce80 2024-04-21T08:55:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.650+0000 7fb9390bf640 1 --2- 172.21.15.28:0/1264479220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7ce80 0x55ac74c78a40 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fb924005ab0 tx=0x7fb924009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed4132eef200074 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.651+0000 7fb91f7fe640 1 -- 172.21.15.28:0/1264479220 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb924054070 con 0x55ac74c7ce80 2024-04-21T08:55:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.651+0000 7fb91f7fe640 1 -- 172.21.15.28:0/1264479220 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb92402cc90 con 0x55ac74c7ce80 2024-04-21T08:55:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.651+0000 7fb91f7fe640 1 -- 172.21.15.28:0/1264479220 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb924050790 con 0x55ac74c7ce80 2024-04-21T08:55:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.652+0000 7fb93e7f2740 1 -- 172.21.15.28:0/1264479220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7ce80 msgr2=0x55ac74c78a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.652+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/1264479220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7ce80 0x55ac74c78a40 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fb924005ab0 tx=0x7fb924009520 comp rx=0 tx=0).stop 2024-04-21T08:55:55.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.658+0000 7fb93e7f2740 1 -- 172.21.15.28:0/1264479220 shutdown_connections 2024-04-21T08:55:55.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.658+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/1264479220 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c85a00 0x55ac74c84df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.658+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/1264479220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7ce80 0x55ac74c78a40 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.658+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/1264479220 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c7c4d0 0x55ac74c71ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.658+0000 7fb93e7f2740 1 -- 172.21.15.28:0/1264479220 >> 172.21.15.28:0/1264479220 conn(0x55ac74c76ba0 msgr2=0x55ac74c7daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.660+0000 7fb93e7f2740 1 -- 172.21.15.28:0/1264479220 shutdown_connections 2024-04-21T08:55:55.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.662+0000 7fb93e7f2740 1 -- 172.21.15.28:0/1264479220 wait complete. 2024-04-21T08:55:55.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.662+0000 7fb93e7f2740 1 -- start start 2024-04-21T08:55:55.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.662+0000 7fb93e7f2740 1 Processor -- start 2024-04-21T08:55:55.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.663+0000 7fb93e7f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7c4d0 0x55ac74c8e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.663+0000 7fb93e7f2740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c7ce80 0x55ac74c88d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.663+0000 7fb93e7f2740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c85a00 0x55ac74c89470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.663+0000 7fb93e7f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb91800e5b0 con 0x55ac74c7c4d0 2024-04-21T08:55:55.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.663+0000 7fb93e7f2740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb91800fa20 con 0x55ac74c7ce80 2024-04-21T08:55:55.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.663+0000 7fb93e7f2740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb91800ec80 con 0x55ac74c85a00 2024-04-21T08:55:55.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.663+0000 7fb93e7f2740 10 client.0 _ll_register_callbacks cb 0x7fb930012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.663+0000 7fb93a0c1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c85a00 0x55ac74c89470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.664+0000 7fb93a0c1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c85a00 0x55ac74c89470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:55298/0 (socket says 172.21.15.28:55298) 2024-04-21T08:55:55.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.664+0000 7fb93a0c1640 1 -- 172.21.15.28:0/365519624 learned_addr learned my addr 172.21.15.28:0/365519624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.664+0000 7fb9390bf640 1 --2- 172.21.15.28:0/365519624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c7ce80 0x55ac74c88d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.664+0000 7fb9398c0640 1 --2- 172.21.15.28:0/365519624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7c4d0 0x55ac74c8e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.664+0000 7fb9398c0640 1 -- 172.21.15.28:0/365519624 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c85a00 msgr2=0x55ac74c89470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.664+0000 7fb9398c0640 1 --2- 172.21.15.28:0/365519624 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c85a00 0x55ac74c89470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.664+0000 7fb9398c0640 1 -- 172.21.15.28:0/365519624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c7ce80 msgr2=0x55ac74c88d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.664+0000 7fb9398c0640 1 --2- 172.21.15.28:0/365519624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c7ce80 0x55ac74c88d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.664+0000 7fb9398c0640 1 -- 172.21.15.28:0/365519624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac74d0b540 con 0x55ac74c7c4d0 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.664+0000 7fb9398c0640 1 --2- 172.21.15.28:0/365519624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7c4d0 0x55ac74c8e2f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fb93000f7a0 tx=0x7fb930027da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a3af6f9a191854 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.665+0000 7fb91d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.665+0000 7fb91d7fa640 1 -- 172.21.15.28:0/365519624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb930027610 con 0x55ac74c7c4d0 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.665+0000 7fb91d7fa640 1 -- 172.21.15.28:0/365519624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fb93000f850 con 0x55ac74c7c4d0 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.666+0000 7fb93e7f2740 1 -- 172.21.15.28:0/365519624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb92402cc90 con 0x55ac74c7c4d0 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.666+0000 7fb91d7fa640 1 -- 172.21.15.28:0/365519624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb93000fa50 con 0x55ac74c7c4d0 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.666+0000 7fb93e7f2740 10 client.17121 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.666+0000 7fb93e7f2740 1 -- 172.21.15.28:0/365519624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ac74cf46c0 con 0x55ac74c7c4d0 2024-04-21T08:55:55.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.666+0000 7fb91d7fa640 1 -- 172.21.15.28:0/365519624 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb92402cc90 con 0x55ac74c7c4d0 2024-04-21T08:55:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.666+0000 7fb91d7fa640 1 -- 172.21.15.28:0/365519624 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fb93003db50 con 0x55ac74c7c4d0 2024-04-21T08:55:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.667+0000 7fb93e7f2740 10 client.17121 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.667+0000 7fb93e7f2740 10 client.17121 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:55:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.667+0000 7fb93e7f2740 20 client.17121 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.667+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/365519624 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55ac74cf4d90 0x55ac74cf71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.667+0000 7fb93e7f2740 4 client.17121 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:55:55.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.667+0000 7fb93e7f2740 1 -- 172.21.15.28:0/365519624 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ac74beec10 con 0x55ac74cf4d90 2024-04-21T08:55:55.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.668+0000 7fb9390bf640 1 --2- 172.21.15.28:0/365519624 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55ac74cf4d90 0x55ac74cf71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.675+0000 7fb9390bf640 1 --2- 172.21.15.28:0/365519624 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55ac74cf4d90 0x55ac74cf71b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=64504534129f4c2c server_cookie=aa5b77768c94e7b0 in_seq=0 out_seq=0 2024-04-21T08:55:55.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.675+0000 7fb91d7fa640 10 client.17121 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:55:55.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.675+0000 7fb91d7fa640 1 -- 172.21.15.28:0/365519624 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ac74beec10 con 0x55ac74cf4d90 2024-04-21T08:55:55.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.675+0000 7fb91d7fa640 10 client.17121 handle_command_reply: tid=0 2024-04-21T08:55:55.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.675+0000 7fb93e7f2740 1 client.17121 shutdown 2024-04-21T08:55:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.681+0000 7fb93e7f2740 1 -- 172.21.15.28:0/365519624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7c4d0 msgr2=0x55ac74c8e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.681+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/365519624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7c4d0 0x55ac74c8e2f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fb93000f7a0 tx=0x7fb930027da0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.681+0000 7fb93e7f2740 1 -- 172.21.15.28:0/365519624 shutdown_connections 2024-04-21T08:55:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.681+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/365519624 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55ac74cf4d90 0x55ac74cf71b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.681+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/365519624 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ac74c85a00 0x55ac74c89470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.681+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/365519624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ac74c7ce80 0x55ac74c88d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.681+0000 7fb93e7f2740 1 --2- 172.21.15.28:0/365519624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ac74c7c4d0 0x55ac74c8e2f0 secure :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fb93000f7a0 tx=0x7fb930027da0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.681+0000 7fb91d7fa640 0 client.17121 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:55:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.681+0000 7fb93e7f2740 1 -- 172.21.15.28:0/365519624 >> 172.21.15.28:0/365519624 conn(0x55ac74c76ba0 msgr2=0x55ac74c75370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.686+0000 7fb93e7f2740 1 -- 172.21.15.28:0/365519624 shutdown_connections 2024-04-21T08:55:55.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.687+0000 7fb93e7f2740 1 -- 172.21.15.28:0/365519624 wait complete. 2024-04-21T08:55:55.688 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:55:55.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.687+0000 7fb93e7f2740 20 client.17121 trim_cache size 0 max 16384 2024-04-21T08:55:55.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.693+0000 7fb93bb24640 1 -- 172.21.15.28:0/3972509006 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb9200a3e60 msgr2=0x7fb9200a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.693+0000 7fb93bb24640 1 --2- 172.21.15.28:0/3972509006 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb9200a3e60 0x7fb9200a6320 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x55ac74c77c10 tx=0x7fb930005f90 comp rx=0 tx=0).stop 2024-04-21T08:55:55.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.693+0000 7fb93bb24640 1 -- 172.21.15.28:0/3972509006 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb93411b730 msgr2=0x7fb9341abe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.693+0000 7fb93bb24640 1 --2- 172.21.15.28:0/3972509006 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb93411b730 0x7fb9341abe10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00eaa0 tx=0x7fb92c00ef60 comp rx=0 tx=0).stop 2024-04-21T08:55:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.693+0000 7fb93bb24640 1 -- 172.21.15.28:0/3972509006 shutdown_connections 2024-04-21T08:55:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.693+0000 7fb93bb24640 1 --2- 172.21.15.28:0/3972509006 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb9200a3e60 0x7fb9200a6320 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.693+0000 7fb93bb24640 1 --2- 172.21.15.28:0/3972509006 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb93411b730 0x7fb9341abe10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.693+0000 7fb93bb24640 1 --2- 172.21.15.28:0/3972509006 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9341107a0 0x7fb9341a7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.693+0000 7fb93bb24640 1 --2- 172.21.15.28:0/3972509006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb93410e7d0 0x7fb9341a75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.693+0000 7fb93bb24640 1 -- 172.21.15.28:0/3972509006 >> 172.21.15.28:0/3972509006 conn(0x7fb93406d060 msgr2=0x7fb934072a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.694+0000 7fb93bb24640 1 -- 172.21.15.28:0/3972509006 shutdown_connections 2024-04-21T08:55:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.694+0000 7fb93bb24640 1 -- 172.21.15.28:0/3972509006 wait complete. 2024-04-21T08:55:55.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:55:55.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.875+0000 7f45b392b640 1 -- 172.21.15.28:0/4283569200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac101090 msgr2=0x7f45ac101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.875+0000 7f45b392b640 1 --2- 172.21.15.28:0/4283569200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac101090 0x7f45ac101470 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f459c009900 tx=0x7f459c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:55:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.875+0000 7f45b392b640 1 -- 172.21.15.28:0/4283569200 shutdown_connections 2024-04-21T08:55:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.875+0000 7f45b392b640 1 --2- 172.21.15.28:0/4283569200 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45ac106160 0x7f45ac10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.875+0000 7f45b392b640 1 --2- 172.21.15.28:0/4283569200 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45ac101a40 0x7f45ac105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.875+0000 7f45b392b640 1 --2- 172.21.15.28:0/4283569200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac101090 0x7f45ac101470 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.875+0000 7f45b392b640 1 -- 172.21.15.28:0/4283569200 >> 172.21.15.28:0/4283569200 conn(0x7f45ac077f50 msgr2=0x7f45ac078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.876+0000 7f45b392b640 1 -- 172.21.15.28:0/4283569200 shutdown_connections 2024-04-21T08:55:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.876+0000 7f45b392b640 1 -- 172.21.15.28:0/4283569200 wait complete. 2024-04-21T08:55:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.876+0000 7f45b392b640 1 Processor -- start 2024-04-21T08:55:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.876+0000 7f45b392b640 1 -- start start 2024-04-21T08:55:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.876+0000 7f45b392b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45ac101090 0x7f45ac19e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.876+0000 7f45b392b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac106160 0x7f45ac19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.876+0000 7f45b392b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45ac198590 0x7f45ac198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.876+0000 7f45b392b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1121d0 con 0x7f45ac106160 2024-04-21T08:55:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.876+0000 7f45b392b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac112050 con 0x7f45ac198590 2024-04-21T08:55:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.876+0000 7f45b392b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac112350 con 0x7f45ac101090 2024-04-21T08:55:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45b0ec6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac106160 0x7f45ac19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45b16c7640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45ac101090 0x7f45ac19e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45b0ec6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac106160 0x7f45ac19e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46200/0 (socket says 172.21.15.28:46200) 2024-04-21T08:55:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45b0ec6640 1 -- 172.21.15.28:0/904167007 learned_addr learned my addr 172.21.15.28:0/904167007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45b1ec8640 1 --2- 172.21.15.28:0/904167007 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45ac198590 0x7f45ac198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45b0ec6640 1 -- 172.21.15.28:0/904167007 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45ac101090 msgr2=0x7f45ac19e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45b0ec6640 1 --2- 172.21.15.28:0/904167007 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45ac101090 0x7f45ac19e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45b0ec6640 1 -- 172.21.15.28:0/904167007 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45ac198590 msgr2=0x7f45ac198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45b0ec6640 1 --2- 172.21.15.28:0/904167007 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45ac198590 0x7f45ac198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45b0ec6640 1 -- 172.21.15.28:0/904167007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac1990b0 con 0x7f45ac106160 2024-04-21T08:55:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45b0ec6640 1 --2- 172.21.15.28:0/904167007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac106160 0x7f45ac19e920 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f45a800e9c0 tx=0x7f45a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45a27fc640 1 -- 172.21.15.28:0/904167007 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a800cd80 con 0x7f45ac106160 2024-04-21T08:55:55.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45a27fc640 1 -- 172.21.15.28:0/904167007 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45a80054d0 con 0x7f45ac106160 2024-04-21T08:55:55.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45b392b640 1 -- 172.21.15.28:0/904167007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45ac1993a0 con 0x7f45ac106160 2024-04-21T08:55:55.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45a27fc640 1 -- 172.21.15.28:0/904167007 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a8010430 con 0x7f45ac106160 2024-04-21T08:55:55.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.877+0000 7f45b392b640 1 -- 172.21.15.28:0/904167007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45ac1a5640 con 0x7f45ac106160 2024-04-21T08:55:55.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.880+0000 7f45a27fc640 1 -- 172.21.15.28:0/904167007 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f45a8004fd0 con 0x7f45ac106160 2024-04-21T08:55:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.880+0000 7f45b65f9740 1 Processor -- start 2024-04-21T08:55:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.880+0000 7f45b65f9740 1 -- start start 2024-04-21T08:55:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.880+0000 7f45b65f9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b40160f8e0 0x55b40160fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.880+0000 7f45b65f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401610290 0x55b401618840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.880+0000 7f45b65f9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b401618e10 0x55b40161b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.880+0000 7f45b65f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b4015701f0 con 0x55b401610290 2024-04-21T08:55:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.880+0000 7f45b65f9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55b40150d4b0 con 0x55b40160f8e0 2024-04-21T08:55:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.880+0000 7f45b65f9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55b40156c1b0 con 0x55b401618e10 2024-04-21T08:55:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.880+0000 7f45b16c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401610290 0x55b401618840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.880+0000 7f45b1ec8640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b40160f8e0 0x55b40160fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.880+0000 7f45b16c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401610290 0x55b401618840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46210/0 (socket says 172.21.15.28:46210) 2024-04-21T08:55:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.880+0000 7f45b16c7640 1 -- 172.21.15.28:0/2059870271 learned_addr learned my addr 172.21.15.28:0/2059870271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:55.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.880+0000 7f45b0ec6640 1 --2- 172.21.15.28:0/2059870271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b401618e10 0x55b40161b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.881+0000 7f45a27fc640 1 --2- 172.21.15.28:0/904167007 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f45980a3da0 0x7f45980a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.881+0000 7f45b1ec8640 1 -- 172.21.15.28:0/2059870271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b401618e10 msgr2=0x55b40161b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.881+0000 7f45b1ec8640 1 --2- 172.21.15.28:0/2059870271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b401618e10 0x55b40161b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.881+0000 7f45b1ec8640 1 -- 172.21.15.28:0/2059870271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401610290 msgr2=0x55b401618840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.881+0000 7f45b1ec8640 1 --2- 172.21.15.28:0/2059870271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401610290 0x55b401618840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.881+0000 7f45b1ec8640 1 -- 172.21.15.28:0/2059870271 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4014ff1b0 con 0x55b40160f8e0 2024-04-21T08:55:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.881+0000 7f45b16c7640 1 --2- 172.21.15.28:0/2059870271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401610290 0x55b401618840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:55:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.881+0000 7f45a27fc640 1 -- 172.21.15.28:0/904167007 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f45a80d0ef0 con 0x7f45ac106160 2024-04-21T08:55:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.881+0000 7f45b1ec8640 1 --2- 172.21.15.28:0/904167007 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f45980a3da0 0x7f45980a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.881+0000 7f45b1ec8640 1 --2- 172.21.15.28:0/2059870271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b40160f8e0 0x55b40160fcc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f45a4005f10 tx=0x7f45a4010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b74c4af906903d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.881+0000 7f45977fe640 1 -- 172.21.15.28:0/2059870271 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a4004b20 con 0x55b40160f8e0 2024-04-21T08:55:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.881+0000 7f45977fe640 1 -- 172.21.15.28:0/2059870271 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45a401ddc0 con 0x55b40160f8e0 2024-04-21T08:55:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.881+0000 7f45977fe640 1 -- 172.21.15.28:0/2059870271 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a4011d30 con 0x55b40160f8e0 2024-04-21T08:55:55.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.882+0000 7f45b65f9740 1 -- 172.21.15.28:0/2059870271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b40160f8e0 msgr2=0x55b40160fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.882+0000 7f45b1ec8640 1 --2- 172.21.15.28:0/904167007 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f45980a3da0 0x7f45980a6260 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f45a4021000 tx=0x7f45a401e000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.882+0000 7f45b65f9740 1 --2- 172.21.15.28:0/2059870271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b40160f8e0 0x55b40160fcc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f45a4005f10 tx=0x7f45a4010860 comp rx=0 tx=0).stop 2024-04-21T08:55:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.882+0000 7f45b65f9740 1 -- 172.21.15.28:0/2059870271 shutdown_connections 2024-04-21T08:55:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.882+0000 7f45b65f9740 1 --2- 172.21.15.28:0/2059870271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b401618e10 0x55b40161b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.882+0000 7f45b65f9740 1 --2- 172.21.15.28:0/2059870271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401610290 0x55b401618840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.882+0000 7f45b65f9740 1 --2- 172.21.15.28:0/2059870271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b40160f8e0 0x55b40160fcc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.882+0000 7f45b65f9740 1 -- 172.21.15.28:0/2059870271 >> 172.21.15.28:0/2059870271 conn(0x55b401609fb0 msgr2=0x55b40160a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.882+0000 7f45b65f9740 1 -- 172.21.15.28:0/2059870271 shutdown_connections 2024-04-21T08:55:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.882+0000 7f45b65f9740 1 -- 172.21.15.28:0/2059870271 wait complete. 2024-04-21T08:55:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.882+0000 7f45b65f9740 1 -- start start 2024-04-21T08:55:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.882+0000 7f45b65f9740 1 Processor -- start 2024-04-21T08:55:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b65f9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b40160f8e0 0x55b4016187c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b65f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401610290 0x55b401615b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b65f9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b401618e10 0x55b401616290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b65f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b40161bd80 con 0x55b401610290 2024-04-21T08:55:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b65f9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55b401529500 con 0x55b401618e10 2024-04-21T08:55:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b65f9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55b40161bf00 con 0x55b40160f8e0 2024-04-21T08:55:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b65f9740 10 client.0 _ll_register_callbacks cb 0x55b4014f7b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b0ec6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401610290 0x55b401615b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b0ec6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401610290 0x55b401615b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46220/0 (socket says 172.21.15.28:46220) 2024-04-21T08:55:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b0ec6640 1 -- 172.21.15.28:0/998248282 learned_addr learned my addr 172.21.15.28:0/998248282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b1ec8640 1 --2- 172.21.15.28:0/998248282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b401618e10 0x55b401616290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b16c7640 1 --2- 172.21.15.28:0/998248282 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b40160f8e0 0x55b4016187c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b0ec6640 1 -- 172.21.15.28:0/998248282 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b40160f8e0 msgr2=0x55b4016187c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b0ec6640 1 --2- 172.21.15.28:0/998248282 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b40160f8e0 0x55b4016187c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b0ec6640 1 -- 172.21.15.28:0/998248282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b401618e10 msgr2=0x55b401616290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b0ec6640 1 --2- 172.21.15.28:0/998248282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b401618e10 0x55b401616290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b0ec6640 1 -- 172.21.15.28:0/998248282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4012823f0 con 0x55b401610290 2024-04-21T08:55:55.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f45b0ec6640 1 --2- 172.21.15.28:0/998248282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401610290 0x55b401615b50 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f45a809ca80 tx=0x7f45a80d4630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c20b85fffd022e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.883+0000 7f4594ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:55:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.884+0000 7f4594ff9640 1 -- 172.21.15.28:0/998248282 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a809c630 con 0x55b401610290 2024-04-21T08:55:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.884+0000 7f4594ff9640 1 -- 172.21.15.28:0/998248282 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f45a8099160 con 0x55b401610290 2024-04-21T08:55:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.884+0000 7f45b65f9740 1 -- 172.21.15.28:0/998248282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45a401ddc0 con 0x55b401610290 2024-04-21T08:55:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.884+0000 7f4594ff9640 1 -- 172.21.15.28:0/998248282 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f45a809c450 con 0x55b401610290 2024-04-21T08:55:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.884+0000 7f45b65f9740 10 client.17133 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.884+0000 7f45b65f9740 1 -- 172.21.15.28:0/998248282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b40134c7d0 con 0x55b401610290 2024-04-21T08:55:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.884+0000 7f4594ff9640 1 -- 172.21.15.28:0/998248282 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45a401ddc0 con 0x55b401610290 2024-04-21T08:55:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.884+0000 7f4594ff9640 1 -- 172.21.15.28:0/998248282 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f45a80a52d0 con 0x55b401610290 2024-04-21T08:55:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.884+0000 7f45b65f9740 10 client.17133 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:55.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.884+0000 7f45b65f9740 10 client.17133 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.884+0000 7f45b65f9740 20 client.17133 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.884+0000 7f45b65f9740 1 --2- 172.21.15.28:0/998248282 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55b401629100 0x55b40162b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.884+0000 7f45b65f9740 4 client.17133 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.884+0000 7f45b65f9740 1 -- 172.21.15.28:0/998248282 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b40150f930 con 0x55b401629100 2024-04-21T08:55:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.885+0000 7f45b16c7640 1 --2- 172.21.15.28:0/998248282 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55b401629100 0x55b40162b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.885+0000 7f4594ff9640 10 client.17133 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.885+0000 7f45b16c7640 1 --2- 172.21.15.28:0/998248282 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55b401629100 0x55b40162b510 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c67bb82eaf0edb7 server_cookie=4f89fca41a2c4b97 in_seq=0 out_seq=0 2024-04-21T08:55:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.886+0000 7f4594ff9640 1 -- 172.21.15.28:0/998248282 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b40150f930 con 0x55b401629100 2024-04-21T08:55:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.886+0000 7f4594ff9640 10 client.17133 handle_command_reply: tid=0 2024-04-21T08:55:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.886+0000 7f45b65f9740 1 client.17133 shutdown 2024-04-21T08:55:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.886+0000 7f45b65f9740 1 -- 172.21.15.28:0/998248282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401610290 msgr2=0x55b401615b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.886+0000 7f45b65f9740 1 --2- 172.21.15.28:0/998248282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401610290 0x55b401615b50 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f45a809ca80 tx=0x7f45a80d4630 comp rx=0 tx=0).stop 2024-04-21T08:55:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.886+0000 7f45b65f9740 1 -- 172.21.15.28:0/998248282 shutdown_connections 2024-04-21T08:55:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.886+0000 7f45b65f9740 1 --2- 172.21.15.28:0/998248282 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55b401629100 0x55b40162b510 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.886+0000 7f45b65f9740 1 --2- 172.21.15.28:0/998248282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b401618e10 0x55b401616290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.886+0000 7f4594ff9640 0 client.17133 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.886+0000 7f45b65f9740 1 --2- 172.21.15.28:0/998248282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401610290 0x55b401615b50 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.886+0000 7f45b65f9740 1 --2- 172.21.15.28:0/998248282 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b40160f8e0 0x55b4016187c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.886+0000 7f45b65f9740 1 -- 172.21.15.28:0/998248282 >> 172.21.15.28:0/998248282 conn(0x55b401609fb0 msgr2=0x55b40160bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.886+0000 7f45b65f9740 1 -- 172.21.15.28:0/998248282 shutdown_connections 2024-04-21T08:55:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.886+0000 7f45b65f9740 1 -- 172.21.15.28:0/998248282 wait complete. 2024-04-21T08:55:55.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.887+0000 7f45b65f9740 20 client.17133 trim_cache size 0 max 16384 2024-04-21T08:55:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b65f9740 1 Processor -- start 2024-04-21T08:55:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b65f9740 1 -- start start 2024-04-21T08:55:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b65f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b40160f8e0 0x55b401624c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b65f9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b401610290 0x55b401630de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b65f9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b401618e10 0x55b401631320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b65f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1af9e0 con 0x55b40160f8e0 2024-04-21T08:55:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b65f9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1ad2b0 con 0x55b401618e10 2024-04-21T08:55:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b65f9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1ab100 con 0x55b401610290 2024-04-21T08:55:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b16c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b40160f8e0 0x55b401624c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b16c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b40160f8e0 0x55b401624c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46226/0 (socket says 172.21.15.28:46226) 2024-04-21T08:55:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b16c7640 1 -- 172.21.15.28:0/419178437 learned_addr learned my addr 172.21.15.28:0/419178437 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b0ec6640 1 --2- 172.21.15.28:0/419178437 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b401610290 0x55b401630de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b1ec8640 1 --2- 172.21.15.28:0/419178437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b401618e10 0x55b401631320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b16c7640 1 -- 172.21.15.28:0/419178437 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b401610290 msgr2=0x55b401630de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b16c7640 1 --2- 172.21.15.28:0/419178437 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b401610290 0x55b401630de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b16c7640 1 -- 172.21.15.28:0/419178437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b401618e10 msgr2=0x55b401631320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b16c7640 1 --2- 172.21.15.28:0/419178437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b401618e10 0x55b401631320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.900+0000 7f45b16c7640 1 -- 172.21.15.28:0/419178437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b401614b00 con 0x55b40160f8e0 2024-04-21T08:55:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.901+0000 7f45b16c7640 1 --2- 172.21.15.28:0/419178437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b40160f8e0 0x55b401624c40 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f459c0096c0 tx=0x7f459c004350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27bcd8009f1f5db4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.901+0000 7f4596ffd640 1 -- 172.21.15.28:0/419178437 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c004480 con 0x55b40160f8e0 2024-04-21T08:55:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.901+0000 7f4596ffd640 1 -- 172.21.15.28:0/419178437 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f459c02cc90 con 0x55b40160f8e0 2024-04-21T08:55:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.901+0000 7f4596ffd640 1 -- 172.21.15.28:0/419178437 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c0047b0 con 0x55b40160f8e0 2024-04-21T08:55:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.901+0000 7f45b65f9740 1 -- 172.21.15.28:0/419178437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b40160f8e0 msgr2=0x55b401624c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.901+0000 7f45b65f9740 1 --2- 172.21.15.28:0/419178437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b40160f8e0 0x55b401624c40 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f459c0096c0 tx=0x7f459c004350 comp rx=0 tx=0).stop 2024-04-21T08:55:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.901+0000 7f45b65f9740 1 -- 172.21.15.28:0/419178437 shutdown_connections 2024-04-21T08:55:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.901+0000 7f45b65f9740 1 --2- 172.21.15.28:0/419178437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b401618e10 0x55b401631320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.901+0000 7f45b65f9740 1 --2- 172.21.15.28:0/419178437 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b401610290 0x55b401630de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.901+0000 7f45b65f9740 1 --2- 172.21.15.28:0/419178437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b40160f8e0 0x55b401624c40 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.901+0000 7f45b65f9740 1 -- 172.21.15.28:0/419178437 >> 172.21.15.28:0/419178437 conn(0x55b401609fb0 msgr2=0x55b401604c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.901+0000 7f45b65f9740 1 -- 172.21.15.28:0/419178437 shutdown_connections 2024-04-21T08:55:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.901+0000 7f45b65f9740 1 -- 172.21.15.28:0/419178437 wait complete. 2024-04-21T08:55:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.902+0000 7f45b65f9740 1 -- start start 2024-04-21T08:55:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.902+0000 7f45b65f9740 1 Processor -- start 2024-04-21T08:55:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.902+0000 7f45b65f9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b40160f8e0 0x55b40161d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.902+0000 7f45b65f9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b401610290 0x55b40161d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.902+0000 7f45b65f9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401618e10 0x55b40168e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.902+0000 7f45b65f9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1ab770 con 0x55b401618e10 2024-04-21T08:55:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.902+0000 7f45b65f9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1acbe0 con 0x55b40160f8e0 2024-04-21T08:55:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.902+0000 7f45b65f9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1abe40 con 0x55b401610290 2024-04-21T08:55:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.902+0000 7f45b65f9740 10 client.0 _ll_register_callbacks cb 0x7f45a80048b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.902+0000 7f45b1ec8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401618e10 0x55b40168e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.902+0000 7f45b1ec8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401618e10 0x55b40168e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46236/0 (socket says 172.21.15.28:46236) 2024-04-21T08:55:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.902+0000 7f45b1ec8640 1 -- 172.21.15.28:0/838683030 learned_addr learned my addr 172.21.15.28:0/838683030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.902+0000 7f45b0ec6640 1 --2- 172.21.15.28:0/838683030 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b401610290 0x55b40161d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.902+0000 7f45b16c7640 1 --2- 172.21.15.28:0/838683030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b40160f8e0 0x55b40161d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f45b1ec8640 1 -- 172.21.15.28:0/838683030 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b401610290 msgr2=0x55b40161d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f45b1ec8640 1 --2- 172.21.15.28:0/838683030 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b401610290 0x55b40161d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f45b1ec8640 1 -- 172.21.15.28:0/838683030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b40160f8e0 msgr2=0x55b40161d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f45b1ec8640 1 --2- 172.21.15.28:0/838683030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b40160f8e0 0x55b40161d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f45b1ec8640 1 -- 172.21.15.28:0/838683030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b401616600 con 0x55b401618e10 2024-04-21T08:55:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f45b1ec8640 1 --2- 172.21.15.28:0/838683030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401618e10 0x55b40168e650 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f45a4005c40 tx=0x7f45a4009a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e650bf26652b715d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f4594ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:55:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f4594ff9640 1 -- 172.21.15.28:0/838683030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a4013260 con 0x55b401618e10 2024-04-21T08:55:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f4594ff9640 1 -- 172.21.15.28:0/838683030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f45a4011440 con 0x55b401618e10 2024-04-21T08:55:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f45b65f9740 1 -- 172.21.15.28:0/838683030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f459c02cc90 con 0x55b401618e10 2024-04-21T08:55:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f4594ff9640 1 -- 172.21.15.28:0/838683030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f45a403e050 con 0x55b401618e10 2024-04-21T08:55:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f45b65f9740 10 client.17139 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f45b65f9740 1 -- 172.21.15.28:0/838683030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b40168fcc0 con 0x55b401618e10 2024-04-21T08:55:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f4594ff9640 1 -- 172.21.15.28:0/838683030 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f459c02cc90 con 0x55b401618e10 2024-04-21T08:55:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f4594ff9640 1 -- 172.21.15.28:0/838683030 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f45a4039030 con 0x55b401618e10 2024-04-21T08:55:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f45b65f9740 10 client.17139 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f45b65f9740 10 client.17139 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f45b65f9740 20 client.17139 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f45b65f9740 1 --2- 172.21.15.28:0/838683030 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55b401629100 0x55b401692380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f45b65f9740 4 client.17139 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:55:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.903+0000 7f45b65f9740 1 -- 172.21.15.28:0/838683030 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b40157dc10 con 0x55b401629100 2024-04-21T08:55:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.904+0000 7f45b16c7640 1 --2- 172.21.15.28:0/838683030 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55b401629100 0x55b401692380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.904+0000 7f45b16c7640 1 --2- 172.21.15.28:0/838683030 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55b401629100 0x55b401692380 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2647db11d5dff21f server_cookie=387a50e0eabf4145 in_seq=0 out_seq=0 2024-04-21T08:55:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.904+0000 7f4594ff9640 10 client.17139 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.904+0000 7f4594ff9640 1 -- 172.21.15.28:0/838683030 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b40157dc10 con 0x55b401629100 2024-04-21T08:55:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.904+0000 7f4594ff9640 10 client.17139 handle_command_reply: tid=0 2024-04-21T08:55:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.904+0000 7f45b65f9740 1 client.17139 shutdown 2024-04-21T08:55:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b65f9740 1 -- 172.21.15.28:0/838683030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401618e10 msgr2=0x55b40168e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b65f9740 1 --2- 172.21.15.28:0/838683030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401618e10 0x55b40168e650 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f45a4005c40 tx=0x7f45a4009a00 comp rx=0 tx=0).stop 2024-04-21T08:55:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b65f9740 1 -- 172.21.15.28:0/838683030 shutdown_connections 2024-04-21T08:55:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b65f9740 1 --2- 172.21.15.28:0/838683030 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55b401629100 0x55b401692380 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b65f9740 1 --2- 172.21.15.28:0/838683030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b401618e10 0x55b40168e650 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b65f9740 1 --2- 172.21.15.28:0/838683030 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b401610290 0x55b40161d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b65f9740 1 --2- 172.21.15.28:0/838683030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b40160f8e0 0x55b40161d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f4594ff9640 0 client.17139 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b65f9740 1 -- 172.21.15.28:0/838683030 >> 172.21.15.28:0/838683030 conn(0x55b401609fb0 msgr2=0x55b40160b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b65f9740 1 -- 172.21.15.28:0/838683030 shutdown_connections 2024-04-21T08:55:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b65f9740 1 -- 172.21.15.28:0/838683030 wait complete. 2024-04-21T08:55:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b65f9740 20 client.17139 trim_cache size 0 max 16384 2024-04-21T08:55:55.911 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:55:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b392b640 1 -- 172.21.15.28:0/904167007 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f45980a3da0 msgr2=0x7f45980a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b392b640 1 --2- 172.21.15.28:0/904167007 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f45980a3da0 0x7f45980a6260 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f45a4021000 tx=0x7f45a401e000 comp rx=0 tx=0).stop 2024-04-21T08:55:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b392b640 1 -- 172.21.15.28:0/904167007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac106160 msgr2=0x7f45ac19e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b392b640 1 --2- 172.21.15.28:0/904167007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac106160 0x7f45ac19e920 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f45a800e9c0 tx=0x7f45a800ee80 comp rx=0 tx=0).stop 2024-04-21T08:55:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b392b640 1 -- 172.21.15.28:0/904167007 shutdown_connections 2024-04-21T08:55:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b392b640 1 --2- 172.21.15.28:0/904167007 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f45980a3da0 0x7f45980a6260 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b392b640 1 --2- 172.21.15.28:0/904167007 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45ac198590 0x7f45ac198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b392b640 1 --2- 172.21.15.28:0/904167007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac106160 0x7f45ac19e920 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b392b640 1 --2- 172.21.15.28:0/904167007 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45ac101090 0x7f45ac19e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b392b640 1 -- 172.21.15.28:0/904167007 >> 172.21.15.28:0/904167007 conn(0x7f45ac077f50 msgr2=0x7f45ac0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b392b640 1 -- 172.21.15.28:0/904167007 shutdown_connections 2024-04-21T08:55:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:55.905+0000 7f45b392b640 1 -- 172.21.15.28:0/904167007 wait complete. 2024-04-21T08:55:56.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:56 smithi152 ceph-mon[31401]: pgmap v145: 129 pgs: 129 active+clean; 17 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 45 MiB/s wr, 6.90k op/s 2024-04-21T08:55:56.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:56 smithi028 ceph-mon[28003]: pgmap v145: 129 pgs: 129 active+clean; 17 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 45 MiB/s wr, 6.90k op/s 2024-04-21T08:55:56.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:56 smithi171 ceph-mon[31358]: pgmap v145: 129 pgs: 129 active+clean; 17 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 45 MiB/s wr, 6.90k op/s 2024-04-21T08:55:56.918 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:55:56.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:55:56.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:55:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.133+0000 7f57ae90e640 1 -- 172.21.15.28:0/2671278932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a8106e80 msgr2=0x7f57a8107260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.133+0000 7f57ae90e640 1 --2- 172.21.15.28:0/2671278932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a8106e80 0x7f57a8107260 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f57a001acb0 tx=0x7f57a003d150 comp rx=0 tx=0).stop 2024-04-21T08:55:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.136+0000 7f57ae90e640 1 -- 172.21.15.28:0/2671278932 shutdown_connections 2024-04-21T08:55:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.136+0000 7f57ae90e640 1 --2- 172.21.15.28:0/2671278932 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57a806ad00 0x7f57a8105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.136+0000 7f57ae90e640 1 --2- 172.21.15.28:0/2671278932 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57a806a280 0x7f57a806a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.136+0000 7f57ae90e640 1 --2- 172.21.15.28:0/2671278932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a8106e80 0x7f57a8107260 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.136+0000 7f57ae90e640 1 -- 172.21.15.28:0/2671278932 >> 172.21.15.28:0/2671278932 conn(0x7f57a80769d0 msgr2=0x7f57a8076de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.143+0000 7f57ae90e640 1 -- 172.21.15.28:0/2671278932 shutdown_connections 2024-04-21T08:55:57.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.143+0000 7f57ae90e640 1 -- 172.21.15.28:0/2671278932 wait complete. 2024-04-21T08:55:57.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.145+0000 7f57ae90e640 1 Processor -- start 2024-04-21T08:55:57.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.145+0000 7f57ae90e640 1 -- start start 2024-04-21T08:55:57.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.146+0000 7f57ae90e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a806a280 0x7f57a8072a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.146+0000 7f57ae90e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57a806ad00 0x7f57a8072fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.146+0000 7f57ae90e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57a8106e80 0x7f57a8115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.146+0000 7f57ae90e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57a810a010 con 0x7f57a806a280 2024-04-21T08:55:57.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.146+0000 7f57ae90e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f57a8109e90 con 0x7f57a806ad00 2024-04-21T08:55:57.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.146+0000 7f57ae90e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f57a810a190 con 0x7f57a8106e80 2024-04-21T08:55:57.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.147+0000 7f57aceab640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57a8106e80 0x7f57a8115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.150+0000 7f57a77fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57a806ad00 0x7f57a8072fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.150+0000 7f57a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a806a280 0x7f57a8072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.150+0000 7f57a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a806a280 0x7f57a8072a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46260/0 (socket says 172.21.15.28:46260) 2024-04-21T08:55:57.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.150+0000 7f57a7fff640 1 -- 172.21.15.28:0/2189145675 learned_addr learned my addr 172.21.15.28:0/2189145675 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:57.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.150+0000 7f57a77fe640 1 -- 172.21.15.28:0/2189145675 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57a8106e80 msgr2=0x7f57a8115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.150+0000 7f57a77fe640 1 --2- 172.21.15.28:0/2189145675 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57a8106e80 0x7f57a8115c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.150+0000 7f57a77fe640 1 -- 172.21.15.28:0/2189145675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a806a280 msgr2=0x7f57a8072a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.150+0000 7f57a77fe640 1 --2- 172.21.15.28:0/2189145675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a806a280 0x7f57a8072a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.150+0000 7f57a77fe640 1 -- 172.21.15.28:0/2189145675 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57a810db40 con 0x7f57a806ad00 2024-04-21T08:55:57.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.151+0000 7f57a77fe640 1 --2- 172.21.15.28:0/2189145675 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57a806ad00 0x7f57a8072fc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f579c00d6d0 tx=0x7f579c00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.152+0000 7f57a57fa640 1 -- 172.21.15.28:0/2189145675 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c017070 con 0x7f57a806ad00 2024-04-21T08:55:57.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.152+0000 7f57a57fa640 1 -- 172.21.15.28:0/2189145675 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f579c009820 con 0x7f57a806ad00 2024-04-21T08:55:57.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.152+0000 7f57a57fa640 1 -- 172.21.15.28:0/2189145675 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c004c40 con 0x7f57a806ad00 2024-04-21T08:55:57.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.155+0000 7f57ae90e640 1 -- 172.21.15.28:0/2189145675 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57a810de30 con 0x7f57a806ad00 2024-04-21T08:55:57.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.155+0000 7f57ae90e640 1 -- 172.21.15.28:0/2189145675 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57a810e300 con 0x7f57a806ad00 2024-04-21T08:55:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.159+0000 7f57b15d9740 1 Processor -- start 2024-04-21T08:55:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.162+0000 7f57a57fa640 1 -- 172.21.15.28:0/2189145675 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f579c005220 con 0x7f57a806ad00 2024-04-21T08:55:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.162+0000 7f57a57fa640 1 --2- 172.21.15.28:0/2189145675 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f57900a3f20 0x7f57900a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.163+0000 7f57a57fa640 1 -- 172.21.15.28:0/2189145675 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f579c0d03e0 con 0x7f57a806ad00 2024-04-21T08:55:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.165+0000 7f57b15d9740 1 -- start start 2024-04-21T08:55:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.165+0000 7f57b15d9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b28e0 0x5556bb0b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.165+0000 7f57b15d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0b3290 0x5556bb0bb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.165+0000 7f57b15d9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0bbe10 0x5556bb0be260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.165+0000 7f57b15d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5556bb016f00 con 0x5556bb0b3290 2024-04-21T08:55:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.165+0000 7f57b15d9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5556bafb4590 con 0x5556bb0bbe10 2024-04-21T08:55:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.165+0000 7f57b15d9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5556bb012ec0 con 0x5556bb0b28e0 2024-04-21T08:55:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.166+0000 7f57aceab640 1 --2- 172.21.15.28:0/2189145675 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f57900a3f20 0x7f57900a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.166+0000 7f57a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0b3290 0x5556bb0bb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.166+0000 7f57aceab640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0bbe10 0x5556bb0be260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.166+0000 7f57aceab640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0bbe10 0x5556bb0be260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38000/0 (socket says 172.21.15.28:38000) 2024-04-21T08:55:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.166+0000 7f57aceab640 1 -- 172.21.15.28:0/3625455562 learned_addr learned my addr 172.21.15.28:0/3625455562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.167+0000 7f57aceab640 1 -- 172.21.15.28:0/3625455562 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b28e0 msgr2=0x5556bb0b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.167+0000 7f57aceab640 1 --2- 172.21.15.28:0/3625455562 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b28e0 0x5556bb0b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.167+0000 7f57aceab640 1 -- 172.21.15.28:0/3625455562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0b3290 msgr2=0x5556bb0bb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.167+0000 7f57aceab640 1 --2- 172.21.15.28:0/3625455562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0b3290 0x5556bb0bb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.167+0000 7f57aceab640 1 -- 172.21.15.28:0/3625455562 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556badf17e0 con 0x5556bb0bbe10 2024-04-21T08:55:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.168+0000 7f57aceab640 1 --2- 172.21.15.28:0/2189145675 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f57900a3f20 0x7f57900a63e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5798013820 tx=0x7f57980097b0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.169+0000 7f57aceab640 1 --2- 172.21.15.28:0/3625455562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0bbe10 0x5556bb0be260 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5798009bd0 tx=0x7f579801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54cacb13488b0bb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.169+0000 7f578e7fc640 1 -- 172.21.15.28:0/3625455562 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579800fdf0 con 0x5556bb0bbe10 2024-04-21T08:55:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.170+0000 7f578e7fc640 1 -- 172.21.15.28:0/3625455562 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5798048070 con 0x5556bb0bbe10 2024-04-21T08:55:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.171+0000 7f57b15d9740 1 -- 172.21.15.28:0/3625455562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0bbe10 msgr2=0x5556bb0be260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.171+0000 7f57b15d9740 1 --2- 172.21.15.28:0/3625455562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0bbe10 0x5556bb0be260 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5798009bd0 tx=0x7f579801d000 comp rx=0 tx=0).stop 2024-04-21T08:55:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.175+0000 7f57b15d9740 1 -- 172.21.15.28:0/3625455562 shutdown_connections 2024-04-21T08:55:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.175+0000 7f57b15d9740 1 --2- 172.21.15.28:0/3625455562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0bbe10 0x5556bb0be260 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.175+0000 7f57b15d9740 1 --2- 172.21.15.28:0/3625455562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0b3290 0x5556bb0bb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.175+0000 7f57b15d9740 1 --2- 172.21.15.28:0/3625455562 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b28e0 0x5556bb0b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.175+0000 7f57b15d9740 1 -- 172.21.15.28:0/3625455562 >> 172.21.15.28:0/3625455562 conn(0x5556bb0acfb0 msgr2=0x5556bb0ad390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.182+0000 7f57b15d9740 1 -- 172.21.15.28:0/3625455562 shutdown_connections 2024-04-21T08:55:57.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.186+0000 7f57b15d9740 1 -- 172.21.15.28:0/3625455562 wait complete. 2024-04-21T08:55:57.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.187+0000 7f57b15d9740 1 -- start start 2024-04-21T08:55:57.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.187+0000 7f57b15d9740 1 Processor -- start 2024-04-21T08:55:57.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.190+0000 7f57b15d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0b28e0 0x5556bb0b8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.190+0000 7f57b15d9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b3290 0x5556bb0b9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.190+0000 7f57b15d9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0bbe10 0x5556bb0b9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.190+0000 7f57b15d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5556bb0bedc0 con 0x5556bb0b28e0 2024-04-21T08:55:57.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.190+0000 7f57b15d9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5556bafd0580 con 0x5556bb0bbe10 2024-04-21T08:55:57.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.190+0000 7f57b15d9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5556bb0bef40 con 0x5556bb0b3290 2024-04-21T08:55:57.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.190+0000 7f57b15d9740 10 client.0 _ll_register_callbacks cb 0x5556bafd6650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:57.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.192+0000 7f57a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0b28e0 0x5556bb0b8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.192+0000 7f57a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0b28e0 0x5556bb0b8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46272/0 (socket says 172.21.15.28:46272) 2024-04-21T08:55:57.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.192+0000 7f57a7fff640 1 -- 172.21.15.28:0/3687501550 learned_addr learned my addr 172.21.15.28:0/3687501550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:57.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.193+0000 7f57aceab640 1 --2- 172.21.15.28:0/3687501550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0bbe10 0x5556bb0b9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.193+0000 7f57a7fff640 1 -- 172.21.15.28:0/3687501550 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b3290 msgr2=0x5556bb0b9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.193+0000 7f57a7fff640 1 --2- 172.21.15.28:0/3687501550 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b3290 0x5556bb0b9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.193+0000 7f57a7fff640 1 -- 172.21.15.28:0/3687501550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0bbe10 msgr2=0x5556bb0b9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.193+0000 7f57a7fff640 1 --2- 172.21.15.28:0/3687501550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0bbe10 0x5556bb0b9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.193+0000 7f57a7fff640 1 -- 172.21.15.28:0/3687501550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556bae51f70 con 0x5556bb0b28e0 2024-04-21T08:55:57.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.193+0000 7f57a7fff640 1 --2- 172.21.15.28:0/3687501550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0b28e0 0x5556bb0b8cd0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f57a0048040 tx=0x7f57a001aa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c647ad0ffe4a9280 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.194+0000 7f57877fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:55:57.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.194+0000 7f57877fe640 1 -- 172.21.15.28:0/3687501550 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0013070 con 0x5556bb0b28e0 2024-04-21T08:55:57.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.194+0000 7f57b15d9740 1 -- 172.21.15.28:0/3687501550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5798048070 con 0x5556bb0b28e0 2024-04-21T08:55:57.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.194+0000 7f57877fe640 1 -- 172.21.15.28:0/3687501550 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f57a0043d80 con 0x5556bb0b28e0 2024-04-21T08:55:57.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.195+0000 7f57877fe640 1 -- 172.21.15.28:0/3687501550 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f57a004a020 con 0x5556bb0b28e0 2024-04-21T08:55:57.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.196+0000 7f57b15d9740 10 client.17151 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:57.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.196+0000 7f57b15d9740 1 -- 172.21.15.28:0/3687501550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5556bace9170 con 0x5556bb0b28e0 2024-04-21T08:55:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.197+0000 7f57877fe640 1 -- 172.21.15.28:0/3687501550 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5798048070 con 0x5556bb0b28e0 2024-04-21T08:55:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.197+0000 7f57877fe640 1 -- 172.21.15.28:0/3687501550 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f57a0004620 con 0x5556bb0b28e0 2024-04-21T08:55:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.197+0000 7f57b15d9740 10 client.17151 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.197+0000 7f57b15d9740 10 client.17151 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.197+0000 7f57b15d9740 20 client.17151 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.197+0000 7f57b15d9740 1 --2- 172.21.15.28:0/3687501550 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5556bb0cc680 0x5556bb0ceae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.197+0000 7f57b15d9740 4 client.17151 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.197+0000 7f57b15d9740 1 -- 172.21.15.28:0/3687501550 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556bb024920 con 0x5556bb0cc680 2024-04-21T08:55:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.198+0000 7f57a77fe640 1 --2- 172.21.15.28:0/3687501550 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5556bb0cc680 0x5556bb0ceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.203+0000 7f57a77fe640 1 --2- 172.21.15.28:0/3687501550 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5556bb0cc680 0x5556bb0ceae0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4577c6cbdf5ae72d server_cookie=8a412f5b52159aa9 in_seq=0 out_seq=0 2024-04-21T08:55:57.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.204+0000 7f57877fe640 10 client.17151 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:57.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.206+0000 7f57877fe640 1 -- 172.21.15.28:0/3687501550 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5556bb024920 con 0x5556bb0cc680 2024-04-21T08:55:57.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.206+0000 7f57877fe640 10 client.17151 handle_command_reply: tid=0 2024-04-21T08:55:57.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.206+0000 7f57b15d9740 1 client.17151 shutdown 2024-04-21T08:55:57.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.208+0000 7f57b15d9740 1 -- 172.21.15.28:0/3687501550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0b28e0 msgr2=0x5556bb0b8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.208+0000 7f57b15d9740 1 --2- 172.21.15.28:0/3687501550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0b28e0 0x5556bb0b8cd0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f57a0048040 tx=0x7f57a001aa20 comp rx=0 tx=0).stop 2024-04-21T08:55:57.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.216+0000 7f57b15d9740 1 -- 172.21.15.28:0/3687501550 shutdown_connections 2024-04-21T08:55:57.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.216+0000 7f57b15d9740 1 --2- 172.21.15.28:0/3687501550 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5556bb0cc680 0x5556bb0ceae0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.216+0000 7f57b15d9740 1 --2- 172.21.15.28:0/3687501550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0bbe10 0x5556bb0b9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.216+0000 7f57b15d9740 1 --2- 172.21.15.28:0/3687501550 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b3290 0x5556bb0b9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.216+0000 7f57b15d9740 1 --2- 172.21.15.28:0/3687501550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0b28e0 0x5556bb0b8cd0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.216+0000 7f57b15d9740 1 -- 172.21.15.28:0/3687501550 >> 172.21.15.28:0/3687501550 conn(0x5556bb0acfb0 msgr2=0x5556bb0aed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.217+0000 7f57877fe640 0 client.17151 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:57.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.220+0000 7f57b15d9740 1 -- 172.21.15.28:0/3687501550 shutdown_connections 2024-04-21T08:55:57.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.221+0000 7f57b15d9740 1 -- 172.21.15.28:0/3687501550 wait complete. 2024-04-21T08:55:57.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.222+0000 7f57b15d9740 20 client.17151 trim_cache size 0 max 16384 2024-04-21T08:55:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.260+0000 7f57b15d9740 1 Processor -- start 2024-04-21T08:55:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.261+0000 7f57b15d9740 1 -- start start 2024-04-21T08:55:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.261+0000 7f57b15d9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b28e0 0x5556bb0c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.261+0000 7f57b15d9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0b3290 0x5556bb0b9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.261+0000 7f57b15d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0bbe10 0x5556bb0abe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.261+0000 7f57b15d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57a8122620 con 0x5556bb0bbe10 2024-04-21T08:55:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.261+0000 7f57b15d9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f57a811ff30 con 0x5556bb0b3290 2024-04-21T08:55:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.261+0000 7f57b15d9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f57a8073dd0 con 0x5556bb0b28e0 2024-04-21T08:55:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.263+0000 7f57a7fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b28e0 0x5556bb0c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.263+0000 7f57a7fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b28e0 0x5556bb0c7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:55398/0 (socket says 172.21.15.28:55398) 2024-04-21T08:55:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.263+0000 7f57a7fff640 1 -- 172.21.15.28:0/2465934603 learned_addr learned my addr 172.21.15.28:0/2465934603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.263+0000 7f57a77fe640 1 --2- 172.21.15.28:0/2465934603 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0b3290 0x5556bb0b9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.264+0000 7f57aceab640 1 --2- 172.21.15.28:0/2465934603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0bbe10 0x5556bb0abe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.265+0000 7f57a7fff640 1 -- 172.21.15.28:0/2465934603 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0b3290 msgr2=0x5556bb0b9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.265+0000 7f57a7fff640 1 --2- 172.21.15.28:0/2465934603 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0b3290 0x5556bb0b9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.265+0000 7f57a7fff640 1 -- 172.21.15.28:0/2465934603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0bbe10 msgr2=0x5556bb0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.265+0000 7f57a7fff640 1 --2- 172.21.15.28:0/2465934603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0bbe10 0x5556bb0abe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.265+0000 7f57a7fff640 1 -- 172.21.15.28:0/2465934603 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556bb0af2a0 con 0x5556bb0b28e0 2024-04-21T08:55:57.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.267+0000 7f57a7fff640 1 --2- 172.21.15.28:0/2465934603 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b28e0 0x5556bb0c7cd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f57a0000f80 tx=0x7f57a001a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=374b66879143b5af server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.270+0000 7f578dffb640 1 -- 172.21.15.28:0/2465934603 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0013050 con 0x5556bb0b28e0 2024-04-21T08:55:57.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.270+0000 7f578dffb640 1 -- 172.21.15.28:0/2465934603 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57a0045b00 con 0x5556bb0b28e0 2024-04-21T08:55:57.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.270+0000 7f578dffb640 1 -- 172.21.15.28:0/2465934603 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0003d40 con 0x5556bb0b28e0 2024-04-21T08:55:57.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.271+0000 7f57b15d9740 1 -- 172.21.15.28:0/2465934603 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b28e0 msgr2=0x5556bb0c7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.271+0000 7f57b15d9740 1 --2- 172.21.15.28:0/2465934603 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b28e0 0x5556bb0c7cd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f57a0000f80 tx=0x7f57a001a5d0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.271+0000 7f57b15d9740 1 -- 172.21.15.28:0/2465934603 shutdown_connections 2024-04-21T08:55:57.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.271+0000 7f57b15d9740 1 --2- 172.21.15.28:0/2465934603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0bbe10 0x5556bb0abe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.271+0000 7f57b15d9740 1 --2- 172.21.15.28:0/2465934603 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0b3290 0x5556bb0b9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.271+0000 7f57b15d9740 1 --2- 172.21.15.28:0/2465934603 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b28e0 0x5556bb0c7cd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.271+0000 7f57b15d9740 1 -- 172.21.15.28:0/2465934603 >> 172.21.15.28:0/2465934603 conn(0x5556bb0acfb0 msgr2=0x5556bb0ab780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.275+0000 7f57b15d9740 1 -- 172.21.15.28:0/2465934603 shutdown_connections 2024-04-21T08:55:57.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.278+0000 7f57b15d9740 1 -- 172.21.15.28:0/2465934603 wait complete. 2024-04-21T08:55:57.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.278+0000 7f57b15d9740 1 -- start start 2024-04-21T08:55:57.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.278+0000 7f57b15d9740 1 Processor -- start 2024-04-21T08:55:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.279+0000 7f57b15d9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0b28e0 0x5556bb0c45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.279+0000 7f57b15d9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b3290 0x5556bb0b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.279+0000 7f57b15d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0bbe10 0x5556bb0b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.279+0000 7f57b15d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57a811e3f0 con 0x5556bb0bbe10 2024-04-21T08:55:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.279+0000 7f57b15d9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f57a811f860 con 0x5556bb0b28e0 2024-04-21T08:55:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.279+0000 7f57b15d9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f57a811eac0 con 0x5556bb0b3290 2024-04-21T08:55:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.280+0000 7f57b15d9740 10 client.0 _ll_register_callbacks cb 0x7f5798005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.280+0000 7f57a7fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0b28e0 0x5556bb0c45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.280+0000 7f57a7fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0b28e0 0x5556bb0c45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38024/0 (socket says 172.21.15.28:38024) 2024-04-21T08:55:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.280+0000 7f57a7fff640 1 -- 172.21.15.28:0/1614288985 learned_addr learned my addr 172.21.15.28:0/1614288985 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.280+0000 7f57a77fe640 1 --2- 172.21.15.28:0/1614288985 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b3290 0x5556bb0b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.280+0000 7f57a7fff640 1 -- 172.21.15.28:0/1614288985 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b3290 msgr2=0x5556bb0b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.280+0000 7f57a7fff640 1 --2- 172.21.15.28:0/1614288985 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b3290 0x5556bb0b56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.280+0000 7f57a7fff640 1 -- 172.21.15.28:0/1614288985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0bbe10 msgr2=0x5556bb0b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.280+0000 7f57a7fff640 1 --2- 172.21.15.28:0/1614288985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0bbe10 0x5556bb0b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.280+0000 7f57a7fff640 1 -- 172.21.15.28:0/1614288985 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556bb0b6410 con 0x5556bb0b28e0 2024-04-21T08:55:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.281+0000 7f57a77fe640 1 --2- 172.21.15.28:0/1614288985 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b3290 0x5556bb0b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:55:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.281+0000 7f57a7fff640 1 --2- 172.21.15.28:0/1614288985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0b28e0 0x5556bb0c45b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f57a001a1c0 tx=0x7f57a003d920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc0ea201d925af43 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.281+0000 7f57877fe640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:55:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.283+0000 7f57877fe640 1 -- 172.21.15.28:0/1614288985 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0013070 con 0x5556bb0b28e0 2024-04-21T08:55:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.284+0000 7f57b15d9740 1 -- 172.21.15.28:0/1614288985 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57a0045b00 con 0x5556bb0b28e0 2024-04-21T08:55:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.284+0000 7f57877fe640 1 -- 172.21.15.28:0/1614288985 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f57a0004b40 con 0x5556bb0b28e0 2024-04-21T08:55:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.286+0000 7f57877fe640 1 -- 172.21.15.28:0/1614288985 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f57a0065050 con 0x5556bb0b28e0 2024-04-21T08:55:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.286+0000 7f57b15d9740 10 client.26414 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.286+0000 7f57b15d9740 1 -- 172.21.15.28:0/1614288985 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5556bb132c80 con 0x5556bb0b28e0 2024-04-21T08:55:57.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.290+0000 7f57877fe640 1 -- 172.21.15.28:0/1614288985 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57a0045b00 con 0x5556bb0b28e0 2024-04-21T08:55:57.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.290+0000 7f57877fe640 1 -- 172.21.15.28:0/1614288985 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f57a0004440 con 0x5556bb0b28e0 2024-04-21T08:55:57.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.290+0000 7f57b15d9740 10 client.26414 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:57.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.290+0000 7f57b15d9740 10 client.26414 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:57.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.290+0000 7f57b15d9740 20 client.26414 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:57.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.290+0000 7f57b15d9740 1 --2- 172.21.15.28:0/1614288985 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5556bb133230 0x5556bb135680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.290+0000 7f57b15d9740 4 client.26414 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:55:57.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.290+0000 7f57b15d9740 1 -- 172.21.15.28:0/1614288985 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5556bb01e790 con 0x5556bb133230 2024-04-21T08:55:57.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.294+0000 7f57a77fe640 1 --2- 172.21.15.28:0/1614288985 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5556bb133230 0x5556bb135680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.300+0000 7f57a77fe640 1 --2- 172.21.15.28:0/1614288985 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5556bb133230 0x5556bb135680 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd7d9a530c4a2633 server_cookie=82b56fcd20aa89e0 in_seq=0 out_seq=0 2024-04-21T08:55:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.300+0000 7f57877fe640 10 client.26414 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:57.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.302+0000 7f57877fe640 1 -- 172.21.15.28:0/1614288985 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5556bb01e790 con 0x5556bb133230 2024-04-21T08:55:57.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.302+0000 7f57877fe640 10 client.26414 handle_command_reply: tid=0 2024-04-21T08:55:57.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.302+0000 7f57b15d9740 1 client.26414 shutdown 2024-04-21T08:55:57.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.302+0000 7f57b15d9740 1 -- 172.21.15.28:0/1614288985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0b28e0 msgr2=0x5556bb0c45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.302+0000 7f57b15d9740 1 --2- 172.21.15.28:0/1614288985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0b28e0 0x5556bb0c45b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f57a001a1c0 tx=0x7f57a003d920 comp rx=0 tx=0).stop 2024-04-21T08:55:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.304+0000 7f57b15d9740 1 -- 172.21.15.28:0/1614288985 shutdown_connections 2024-04-21T08:55:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.304+0000 7f57b15d9740 1 --2- 172.21.15.28:0/1614288985 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5556bb133230 0x5556bb135680 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.304+0000 7f57b15d9740 1 --2- 172.21.15.28:0/1614288985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556bb0bbe10 0x5556bb0b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.304+0000 7f57b15d9740 1 --2- 172.21.15.28:0/1614288985 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5556bb0b3290 0x5556bb0b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.304+0000 7f57b15d9740 1 --2- 172.21.15.28:0/1614288985 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5556bb0b28e0 0x5556bb0c45b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.304+0000 7f57b15d9740 1 -- 172.21.15.28:0/1614288985 >> 172.21.15.28:0/1614288985 conn(0x5556bb0acfb0 msgr2=0x5556bb0aeb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.304+0000 7f57877fe640 0 client.26414 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.304+0000 7f57b15d9740 1 -- 172.21.15.28:0/1614288985 shutdown_connections 2024-04-21T08:55:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.305+0000 7f57b15d9740 1 -- 172.21.15.28:0/1614288985 wait complete. 2024-04-21T08:55:57.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.306+0000 7f57b15d9740 20 client.26414 trim_cache size 0 max 16384 2024-04-21T08:55:57.309 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:55:57.309 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:55:57.309 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "dd219396-9868-49d6-9248-7e294e4ebcd5", 2024-04-21T08:55:57.310 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:55:57.310 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:55:57.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.310+0000 7f57ae90e640 1 -- 172.21.15.28:0/2189145675 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f57900a3f20 msgr2=0x7f57900a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.310+0000 7f57ae90e640 1 --2- 172.21.15.28:0/2189145675 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f57900a3f20 0x7f57900a63e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5798013820 tx=0x7f57980097b0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.310+0000 7f57ae90e640 1 -- 172.21.15.28:0/2189145675 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57a806ad00 msgr2=0x7f57a8072fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.310+0000 7f57ae90e640 1 --2- 172.21.15.28:0/2189145675 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57a806ad00 0x7f57a8072fc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f579c00d6d0 tx=0x7f579c00db90 comp rx=0 tx=0).stop 2024-04-21T08:55:57.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.314+0000 7f57ae90e640 1 -- 172.21.15.28:0/2189145675 shutdown_connections 2024-04-21T08:55:57.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.314+0000 7f57ae90e640 1 --2- 172.21.15.28:0/2189145675 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f57900a3f20 0x7f57900a63e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.314+0000 7f57ae90e640 1 --2- 172.21.15.28:0/2189145675 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57a8106e80 0x7f57a8115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.314+0000 7f57ae90e640 1 --2- 172.21.15.28:0/2189145675 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57a806ad00 0x7f57a8072fc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.314+0000 7f57ae90e640 1 --2- 172.21.15.28:0/2189145675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a806a280 0x7f57a8072a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.314+0000 7f57ae90e640 1 -- 172.21.15.28:0/2189145675 >> 172.21.15.28:0/2189145675 conn(0x7f57a80769d0 msgr2=0x7f57a8103a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.315+0000 7f57ae90e640 1 -- 172.21.15.28:0/2189145675 shutdown_connections 2024-04-21T08:55:57.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.315+0000 7f57ae90e640 1 -- 172.21.15.28:0/2189145675 wait complete. 2024-04-21T08:55:57.342 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:55:57.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:55:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.449+0000 7fd323bb2640 1 -- 172.21.15.28:0/4276610062 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd31c110740 msgr2=0x7fd31c108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.449+0000 7fd323bb2640 1 --2- 172.21.15.28:0/4276610062 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd31c110740 0x7fd31c108f50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd314009900 tx=0x7fd31402bcf0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.451+0000 7fd323bb2640 1 -- 172.21.15.28:0/4276610062 shutdown_connections 2024-04-21T08:55:57.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.451+0000 7fd323bb2640 1 --2- 172.21.15.28:0/4276610062 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd31c11b730 0x7fd31c11db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.451+0000 7fd323bb2640 1 --2- 172.21.15.28:0/4276610062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd31c110b20 0x7fd31c109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.451+0000 7fd323bb2640 1 --2- 172.21.15.28:0/4276610062 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd31c110740 0x7fd31c108f50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.451+0000 7fd323bb2640 1 -- 172.21.15.28:0/4276610062 >> 172.21.15.28:0/4276610062 conn(0x7fd31c06d020 msgr2=0x7fd31c06d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.460+0000 7fd323bb2640 1 -- 172.21.15.28:0/4276610062 shutdown_connections 2024-04-21T08:55:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.471+0000 7fd323bb2640 1 -- 172.21.15.28:0/4276610062 wait complete. 2024-04-21T08:55:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.472+0000 7fd323bb2640 1 Processor -- start 2024-04-21T08:55:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.475+0000 7fd323bb2640 1 -- start start 2024-04-21T08:55:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.475+0000 7fd323bb2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd31c110740 0x7fd31c118cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.475+0000 7fd323bb2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd31c11b730 0x7fd31c111d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.475+0000 7fd323bb2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd31c119300 0x7fd31c112360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.475+0000 7fd323bb2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c1210d0 con 0x7fd31c110740 2024-04-21T08:55:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.475+0000 7fd323bb2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c120f50 con 0x7fd31c11b730 2024-04-21T08:55:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.475+0000 7fd323bb2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c121250 con 0x7fd31c119300 2024-04-21T08:55:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.479+0000 7fd32214f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd31c119300 0x7fd31c112360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.479+0000 7fd32214f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd31c119300 0x7fd31c112360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:55426/0 (socket says 172.21.15.28:55426) 2024-04-21T08:55:57.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.479+0000 7fd32214f640 1 -- 172.21.15.28:0/3721089146 learned_addr learned my addr 172.21.15.28:0/3721089146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:57.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.479+0000 7fd32114d640 1 --2- 172.21.15.28:0/3721089146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd31c11b730 0x7fd31c111d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.480+0000 7fd32194e640 1 --2- 172.21.15.28:0/3721089146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd31c110740 0x7fd31c118cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.484+0000 7fd32214f640 1 -- 172.21.15.28:0/3721089146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd31c11b730 msgr2=0x7fd31c111d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.484+0000 7fd32214f640 1 --2- 172.21.15.28:0/3721089146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd31c11b730 0x7fd31c111d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.484+0000 7fd32214f640 1 -- 172.21.15.28:0/3721089146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd31c110740 msgr2=0x7fd31c118cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.484+0000 7fd32214f640 1 --2- 172.21.15.28:0/3721089146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd31c110740 0x7fd31c118cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.484+0000 7fd32214f640 1 -- 172.21.15.28:0/3721089146 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31c1167d0 con 0x7fd31c119300 2024-04-21T08:55:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.485+0000 7fd32214f640 1 --2- 172.21.15.28:0/3721089146 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd31c119300 0x7fd31c112360 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd31000b3c0 tx=0x7fd31000b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.492+0000 7fd31affd640 1 -- 172.21.15.28:0/3721089146 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd310011830 con 0x7fd31c119300 2024-04-21T08:55:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.492+0000 7fd31affd640 1 -- 172.21.15.28:0/3721089146 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd310016070 con 0x7fd31c119300 2024-04-21T08:55:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.492+0000 7fd31affd640 1 -- 172.21.15.28:0/3721089146 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31000f2e0 con 0x7fd31c119300 2024-04-21T08:55:57.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.493+0000 7fd323bb2640 1 -- 172.21.15.28:0/3721089146 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31c116ac0 con 0x7fd31c119300 2024-04-21T08:55:57.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.493+0000 7fd323bb2640 1 -- 172.21.15.28:0/3721089146 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd31c116fc0 con 0x7fd31c119300 2024-04-21T08:55:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.497+0000 7fd3267a1740 1 Processor -- start 2024-04-21T08:55:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.499+0000 7fd31affd640 1 -- 172.21.15.28:0/3721089146 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd3100119d0 con 0x7fd31c119300 2024-04-21T08:55:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.500+0000 7fd3267a1740 1 -- start start 2024-04-21T08:55:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.500+0000 7fd3267a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49afa8d0 0x562e49af9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.500+0000 7fd3267a1740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49af9b60 0x562e49af9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.500+0000 7fd3267a1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49b030e0 0x562e49b054d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.500+0000 7fd3267a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e49a5e8d0 con 0x562e49afa8d0 2024-04-21T08:55:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.500+0000 7fd3267a1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562e499fbe80 con 0x562e49b030e0 2024-04-21T08:55:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.500+0000 7fd3267a1740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x562e49a5a890 con 0x562e49af9b60 2024-04-21T08:55:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.501+0000 7fd31affd640 1 --2- 172.21.15.28:0/3721089146 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd3040a3f70 0x7fd3040a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.501+0000 7fd31affd640 1 -- 172.21.15.28:0/3721089146 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fd3100cff00 con 0x7fd31c119300 2024-04-21T08:55:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.503+0000 7fd32194e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49af9b60 0x562e49af9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.503+0000 7fd32194e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49af9b60 0x562e49af9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:55442/0 (socket says 172.21.15.28:55442) 2024-04-21T08:55:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.503+0000 7fd32194e640 1 -- 172.21.15.28:0/1175907779 learned_addr learned my addr 172.21.15.28:0/1175907779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.503+0000 7fd32214f640 1 --2- 172.21.15.28:0/3721089146 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd3040a3f70 0x7fd3040a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.504+0000 7fd32114d640 1 --2- 172.21.15.28:0/1175907779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49b030e0 0x562e49b054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.504+0000 7fd32114d640 1 --2- 172.21.15.28:0/1175907779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49afa8d0 0x562e49af9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.504+0000 7fd32194e640 1 -- 172.21.15.28:0/1175907779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49b030e0 msgr2=0x562e49b054d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.504+0000 7fd32194e640 1 --2- 172.21.15.28:0/1175907779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49b030e0 0x562e49b054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.504+0000 7fd32194e640 1 -- 172.21.15.28:0/1175907779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49afa8d0 msgr2=0x562e49af9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.504+0000 7fd32194e640 1 --2- 172.21.15.28:0/1175907779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49afa8d0 0x562e49af9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.504+0000 7fd32194e640 1 -- 172.21.15.28:0/1175907779 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e49a1d320 con 0x562e49af9b60 2024-04-21T08:55:57.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.504+0000 7fd32114d640 1 --2- 172.21.15.28:0/1175907779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49afa8d0 0x562e49af9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:55:57.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.512+0000 7fd32194e640 1 --2- 172.21.15.28:0/1175907779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49af9b60 0x562e49af9fe0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd314009230 tx=0x7fd314009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e654cf92fabff7e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.512+0000 7fd32214f640 1 --2- 172.21.15.28:0/3721089146 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd3040a3f70 0x7fd3040a6430 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd31000b390 tx=0x7fd310004700 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.512+0000 7fd303fff640 1 -- 172.21.15.28:0/1175907779 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3140043c0 con 0x562e49af9b60 2024-04-21T08:55:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.512+0000 7fd303fff640 1 -- 172.21.15.28:0/1175907779 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3140054f0 con 0x562e49af9b60 2024-04-21T08:55:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.513+0000 7fd3267a1740 1 -- 172.21.15.28:0/1175907779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49af9b60 msgr2=0x562e49af9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.513+0000 7fd3267a1740 1 --2- 172.21.15.28:0/1175907779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49af9b60 0x562e49af9fe0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd314009230 tx=0x7fd314009610 comp rx=0 tx=0).stop 2024-04-21T08:55:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.514+0000 7fd3267a1740 1 -- 172.21.15.28:0/1175907779 shutdown_connections 2024-04-21T08:55:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.514+0000 7fd3267a1740 1 --2- 172.21.15.28:0/1175907779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49b030e0 0x562e49b054d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.514+0000 7fd3267a1740 1 --2- 172.21.15.28:0/1175907779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49af9b60 0x562e49af9fe0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.514+0000 7fd3267a1740 1 --2- 172.21.15.28:0/1175907779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49afa8d0 0x562e49af9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.514+0000 7fd3267a1740 1 -- 172.21.15.28:0/1175907779 >> 172.21.15.28:0/1175907779 conn(0x562e49ac0c90 msgr2=0x562e49af4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.514+0000 7fd3267a1740 1 -- 172.21.15.28:0/1175907779 shutdown_connections 2024-04-21T08:55:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.514+0000 7fd3267a1740 1 -- 172.21.15.28:0/1175907779 wait complete. 2024-04-21T08:55:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.515+0000 7fd3267a1740 1 -- start start 2024-04-21T08:55:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.515+0000 7fd3267a1740 1 Processor -- start 2024-04-21T08:55:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.515+0000 7fd3267a1740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49afa8d0 0x562e49b027c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.515+0000 7fd3267a1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49b030e0 0x562e49affab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.515+0000 7fd3267a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49affff0 0x562e49b00470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.515+0000 7fd3267a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e49b05e00 con 0x562e49affff0 2024-04-21T08:55:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.515+0000 7fd3267a1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562e499e6eb0 con 0x562e49b030e0 2024-04-21T08:55:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.515+0000 7fd3267a1740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x562e49b05f80 con 0x562e49afa8d0 2024-04-21T08:55:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.515+0000 7fd3267a1740 10 client.0 _ll_register_callbacks cb 0x562e49a13b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.516+0000 7fd32194e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49b030e0 0x562e49affab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.516+0000 7fd32194e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49b030e0 0x562e49affab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38062/0 (socket says 172.21.15.28:38062) 2024-04-21T08:55:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.516+0000 7fd32194e640 1 -- 172.21.15.28:0/955446231 learned_addr learned my addr 172.21.15.28:0/955446231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.516+0000 7fd32194e640 1 -- 172.21.15.28:0/955446231 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49afa8d0 msgr2=0x562e49b027c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.516+0000 7fd32194e640 1 --2- 172.21.15.28:0/955446231 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49afa8d0 0x562e49b027c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.516+0000 7fd32194e640 1 -- 172.21.15.28:0/955446231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49affff0 msgr2=0x562e49b00470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.516+0000 7fd32194e640 1 --2- 172.21.15.28:0/955446231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49affff0 0x562e49b00470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.516+0000 7fd32194e640 1 -- 172.21.15.28:0/955446231 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e49771210 con 0x562e49b030e0 2024-04-21T08:55:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.516+0000 7fd32194e640 1 --2- 172.21.15.28:0/955446231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49b030e0 0x562e49affab0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd31402c7e0 tx=0x7fd31402bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e39f6fec626f7e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.517+0000 7fd3017fa640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:55:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.517+0000 7fd3017fa640 1 -- 172.21.15.28:0/955446231 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31404e060 con 0x562e49b030e0 2024-04-21T08:55:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.517+0000 7fd3017fa640 1 -- 172.21.15.28:0/955446231 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fd3140037f0 con 0x562e49b030e0 2024-04-21T08:55:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.518+0000 7fd3267a1740 1 -- 172.21.15.28:0/955446231 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562e49a72880 con 0x562e49b030e0 2024-04-21T08:55:57.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.522+0000 7fd3017fa640 1 -- 172.21.15.28:0/955446231 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd3140039f0 con 0x562e49b030e0 2024-04-21T08:55:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.525+0000 7fd3267a1740 10 client.26420 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.525+0000 7fd3267a1740 1 -- 172.21.15.28:0/955446231 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562e4983d620 con 0x562e49b030e0 2024-04-21T08:55:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.530+0000 7fd3017fa640 1 -- 172.21.15.28:0/955446231 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562e49a72880 con 0x562e49b030e0 2024-04-21T08:55:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.530+0000 7fd3017fa640 1 -- 172.21.15.28:0/955446231 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fd31403e420 con 0x562e49b030e0 2024-04-21T08:55:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.530+0000 7fd3267a1740 10 client.26420 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.530+0000 7fd3267a1740 10 client.26420 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.530+0000 7fd3267a1740 20 client.26420 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.530+0000 7fd3267a1740 1 --2- 172.21.15.28:0/955446231 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e49b13430 0x562e49b15850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.530+0000 7fd3267a1740 4 client.26420 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.530+0000 7fd3267a1740 1 -- 172.21.15.28:0/955446231 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e499fe3d0 con 0x562e49b13430 2024-04-21T08:55:57.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.531+0000 7fd32114d640 1 --2- 172.21.15.28:0/955446231 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e49b13430 0x562e49b15850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.536+0000 7fd32114d640 1 --2- 172.21.15.28:0/955446231 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e49b13430 0x562e49b15850 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a29a59ea7ca3b144 server_cookie=c554ad17df106a31 in_seq=0 out_seq=0 2024-04-21T08:55:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.536+0000 7fd3017fa640 10 client.26420 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.538+0000 7fd3017fa640 1 -- 172.21.15.28:0/955446231 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e499fe3d0 con 0x562e49b13430 2024-04-21T08:55:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.538+0000 7fd3017fa640 10 client.26420 handle_command_reply: tid=0 2024-04-21T08:55:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.539+0000 7fd3267a1740 1 client.26420 shutdown 2024-04-21T08:55:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.544+0000 7fd3267a1740 1 -- 172.21.15.28:0/955446231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49b030e0 msgr2=0x562e49affab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.544+0000 7fd3267a1740 1 --2- 172.21.15.28:0/955446231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49b030e0 0x562e49affab0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd31402c7e0 tx=0x7fd31402bcf0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.544+0000 7fd3267a1740 1 -- 172.21.15.28:0/955446231 shutdown_connections 2024-04-21T08:55:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.544+0000 7fd3267a1740 1 --2- 172.21.15.28:0/955446231 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e49b13430 0x562e49b15850 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.544+0000 7fd3267a1740 1 --2- 172.21.15.28:0/955446231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49affff0 0x562e49b00470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.544+0000 7fd3267a1740 1 --2- 172.21.15.28:0/955446231 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49b030e0 0x562e49affab0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.544+0000 7fd3267a1740 1 --2- 172.21.15.28:0/955446231 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49afa8d0 0x562e49b027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.544+0000 7fd3267a1740 1 -- 172.21.15.28:0/955446231 >> 172.21.15.28:0/955446231 conn(0x562e49ac0c90 msgr2=0x562e49af3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.544+0000 7fd3017fa640 0 client.26420 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:57.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.549+0000 7fd3267a1740 1 -- 172.21.15.28:0/955446231 shutdown_connections 2024-04-21T08:55:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.553+0000 7fd3267a1740 1 -- 172.21.15.28:0/955446231 wait complete. 2024-04-21T08:55:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.553+0000 7fd3267a1740 20 client.26420 trim_cache size 0 max 16384 2024-04-21T08:55:57.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.594+0000 7fbd94946640 1 -- 172.21.15.28:0/2853505546 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd9010e7b0 msgr2=0x7fbd9010ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.594+0000 7fbd94946640 1 --2- 172.21.15.28:0/2853505546 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd9010e7b0 0x7fbd9010ec10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbd84009900 tx=0x7fbd8402bc80 comp rx=0 tx=0).stop 2024-04-21T08:55:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.595+0000 7fd3267a1740 1 Processor -- start 2024-04-21T08:55:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.595+0000 7fd3267a1740 1 -- start start 2024-04-21T08:55:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.596+0000 7fd3267a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49afa8d0 0x562e49b0ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.597+0000 7fd3267a1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49affff0 0x562e49b044d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.597+0000 7fd3267a1740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49b030e0 0x562e49b022f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.597+0000 7fd3267a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd308012900 con 0x562e49afa8d0 2024-04-21T08:55:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.597+0000 7fd3267a1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd3080101d0 con 0x562e49affff0 2024-04-21T08:55:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.597+0000 7fd3267a1740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd30800e020 con 0x562e49b030e0 2024-04-21T08:55:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.598+0000 7fd32114d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49b030e0 0x562e49b022f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.598+0000 7fd32114d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49afa8d0 0x562e49b0ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.598+0000 7fd32114d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49afa8d0 0x562e49b0ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46354/0 (socket says 172.21.15.28:46354) 2024-04-21T08:55:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.598+0000 7fd32114d640 1 -- 172.21.15.28:0/128344779 learned_addr learned my addr 172.21.15.28:0/128344779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:57.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.598+0000 7fd32194e640 1 --2- 172.21.15.28:0/128344779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49affff0 0x562e49b044d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.599+0000 7fd32114d640 1 -- 172.21.15.28:0/128344779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49affff0 msgr2=0x562e49b044d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.599+0000 7fd32114d640 1 --2- 172.21.15.28:0/128344779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49affff0 0x562e49b044d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.599+0000 7fd32114d640 1 -- 172.21.15.28:0/128344779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49afa8d0 msgr2=0x562e49b0ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.599+0000 7fd32114d640 1 --2- 172.21.15.28:0/128344779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49afa8d0 0x562e49b0ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.599+0000 7fd32114d640 1 -- 172.21.15.28:0/128344779 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e49af8e30 con 0x562e49b030e0 2024-04-21T08:55:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.599+0000 7fd32114d640 1 --2- 172.21.15.28:0/128344779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49b030e0 0x562e49b022f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd30c004a80 tx=0x7fd30c009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c209c6601759cf46 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.600+0000 7fd3037fe640 1 -- 172.21.15.28:0/128344779 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c01a7e0 con 0x562e49b030e0 2024-04-21T08:55:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.601+0000 7fd3037fe640 1 -- 172.21.15.28:0/128344779 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd30c012dd0 con 0x562e49b030e0 2024-04-21T08:55:57.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.601+0000 7fbd94946640 1 -- 172.21.15.28:0/2853505546 shutdown_connections 2024-04-21T08:55:57.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.601+0000 7fbd94946640 1 --2- 172.21.15.28:0/2853505546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd9011b730 0x7fbd9011db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.601+0000 7fbd94946640 1 --2- 172.21.15.28:0/2853505546 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd9010e7b0 0x7fbd9010ec10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.601+0000 7fbd94946640 1 --2- 172.21.15.28:0/2853505546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd901106d0 0x7fbd9010e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.601+0000 7fbd94946640 1 -- 172.21.15.28:0/2853505546 >> 172.21.15.28:0/2853505546 conn(0x7fbd9006d070 msgr2=0x7fbd9006d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.601+0000 7fd3037fe640 1 -- 172.21.15.28:0/128344779 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c024d10 con 0x562e49b030e0 2024-04-21T08:55:57.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.601+0000 7fbd94946640 1 -- 172.21.15.28:0/2853505546 shutdown_connections 2024-04-21T08:55:57.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.601+0000 7fbd94946640 1 -- 172.21.15.28:0/2853505546 wait complete. 2024-04-21T08:55:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.602+0000 7fd3267a1740 1 -- 172.21.15.28:0/128344779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49b030e0 msgr2=0x562e49b022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.602+0000 7fd3267a1740 1 --2- 172.21.15.28:0/128344779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49b030e0 0x562e49b022f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd30c004a80 tx=0x7fd30c009680 comp rx=0 tx=0).stop 2024-04-21T08:55:57.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.602+0000 7fbd94946640 1 Processor -- start 2024-04-21T08:55:57.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.602+0000 7fbd94946640 1 -- start start 2024-04-21T08:55:57.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.603+0000 7fbd94946640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd9010e7b0 0x7fbd901a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.603+0000 7fbd94946640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd901106d0 0x7fbd901a7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.603+0000 7fbd94946640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd9011b730 0x7fbd901abf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.603+0000 7fbd94946640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbd90121100 con 0x7fbd901106d0 2024-04-21T08:55:57.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.603+0000 7fbd94946640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbd90120f80 con 0x7fbd9010e7b0 2024-04-21T08:55:57.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.603+0000 7fbd94946640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbd90121280 con 0x7fbd9011b730 2024-04-21T08:55:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.603+0000 7fd3267a1740 1 -- 172.21.15.28:0/128344779 shutdown_connections 2024-04-21T08:55:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.603+0000 7fd3267a1740 1 --2- 172.21.15.28:0/128344779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49b030e0 0x562e49b022f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.603+0000 7fd3267a1740 1 --2- 172.21.15.28:0/128344779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49affff0 0x562e49b044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.603+0000 7fd3267a1740 1 --2- 172.21.15.28:0/128344779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49afa8d0 0x562e49b0ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.603+0000 7fd3267a1740 1 -- 172.21.15.28:0/128344779 >> 172.21.15.28:0/128344779 conn(0x562e49ac0c90 msgr2=0x562e49af4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.604+0000 7fbd8ed9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd9011b730 0x7fbd901abf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.604+0000 7fbd8ed9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd9011b730 0x7fbd901abf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:55460/0 (socket says 172.21.15.28:55460) 2024-04-21T08:55:57.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.604+0000 7fbd8ed9d640 1 -- 172.21.15.28:0/866252688 learned_addr learned my addr 172.21.15.28:0/866252688 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:57.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.605+0000 7fbd8e59c640 1 --2- 172.21.15.28:0/866252688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd9010e7b0 0x7fbd901a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.605+0000 7fbd8dd9b640 1 --2- 172.21.15.28:0/866252688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd901106d0 0x7fbd901a7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.605+0000 7fbd8ed9d640 1 -- 172.21.15.28:0/866252688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd9010e7b0 msgr2=0x7fbd901a7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.605+0000 7fbd8ed9d640 1 --2- 172.21.15.28:0/866252688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd9010e7b0 0x7fbd901a7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.605+0000 7fbd8ed9d640 1 -- 172.21.15.28:0/866252688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd901106d0 msgr2=0x7fbd901a7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.605+0000 7fbd8ed9d640 1 --2- 172.21.15.28:0/866252688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd901106d0 0x7fbd901a7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.605+0000 7fbd8ed9d640 1 -- 172.21.15.28:0/866252688 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd901ac600 con 0x7fbd9011b730 2024-04-21T08:55:57.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.608+0000 7fd3267a1740 1 -- 172.21.15.28:0/128344779 shutdown_connections 2024-04-21T08:55:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.608+0000 7fd3267a1740 1 -- 172.21.15.28:0/128344779 wait complete. 2024-04-21T08:55:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.608+0000 7fd3267a1740 1 -- start start 2024-04-21T08:55:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.608+0000 7fd3267a1740 1 Processor -- start 2024-04-21T08:55:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.609+0000 7fd3267a1740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49afa8d0 0x562e49afd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.609+0000 7fd3267a1740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49affff0 0x562e49afbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.609+0000 7fd3267a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49b030e0 0x562e49afc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.609+0000 7fd3267a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd30800e690 con 0x562e49b030e0 2024-04-21T08:55:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.609+0000 7fd3267a1740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd30800fb00 con 0x562e49affff0 2024-04-21T08:55:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.609+0000 7fd3267a1740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd30800ed60 con 0x562e49afa8d0 2024-04-21T08:55:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.609+0000 7fd3267a1740 10 client.0 _ll_register_callbacks cb 0x7fd314009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.609+0000 7fd32194e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49affff0 0x562e49afbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.610+0000 7fd32194e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49affff0 0x562e49afbd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:38108/0 (socket says 172.21.15.28:38108) 2024-04-21T08:55:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.610+0000 7fd32194e640 1 -- 172.21.15.28:0/570105687 learned_addr learned my addr 172.21.15.28:0/570105687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:57.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.608+0000 7fbd8ed9d640 1 --2- 172.21.15.28:0/866252688 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd9011b730 0x7fbd901abf00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbd8000b420 tx=0x7fbd8000b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.610+0000 7fd32194e640 1 -- 172.21.15.28:0/570105687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49afa8d0 msgr2=0x562e49afd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.610+0000 7fd32194e640 1 --2- 172.21.15.28:0/570105687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49afa8d0 0x562e49afd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.610+0000 7fd32194e640 1 -- 172.21.15.28:0/570105687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49b030e0 msgr2=0x562e49afc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.610+0000 7fd32194e640 1 --2- 172.21.15.28:0/570105687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49b030e0 0x562e49afc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.610+0000 7fd32194e640 1 -- 172.21.15.28:0/570105687 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e49afca60 con 0x562e49affff0 2024-04-21T08:55:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.610+0000 7fd32194e640 1 --2- 172.21.15.28:0/570105687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49affff0 0x562e49afbd20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x562e49b01080 tx=0x7fd31403ec40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd9df40c438e12de server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.611+0000 7fd3017fa640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:55:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.611+0000 7fd3017fa640 1 -- 172.21.15.28:0/570105687 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31404e040 con 0x562e49affff0 2024-04-21T08:55:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.611+0000 7fd3267a1740 1 -- 172.21.15.28:0/570105687 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd30c012dd0 con 0x562e49affff0 2024-04-21T08:55:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.611+0000 7fd3017fa640 1 -- 172.21.15.28:0/570105687 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fd314003b90 con 0x562e49affff0 2024-04-21T08:55:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.612+0000 7fd3017fa640 1 -- 172.21.15.28:0/570105687 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd314003800 con 0x562e49affff0 2024-04-21T08:55:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.612+0000 7fd3267a1740 10 client.26429 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.612+0000 7fd3267a1740 1 -- 172.21.15.28:0/570105687 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562e49b0b250 con 0x562e49affff0 2024-04-21T08:55:57.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.613+0000 7fd3017fa640 1 -- 172.21.15.28:0/570105687 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd30c012dd0 con 0x562e49affff0 2024-04-21T08:55:57.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.613+0000 7fd3017fa640 1 -- 172.21.15.28:0/570105687 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fd314034a70 con 0x562e49affff0 2024-04-21T08:55:57.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.611+0000 7fbd7f7fe640 1 -- 172.21.15.28:0/866252688 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80011830 con 0x7fbd9011b730 2024-04-21T08:55:57.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.611+0000 7fbd7f7fe640 1 -- 172.21.15.28:0/866252688 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd80016070 con 0x7fbd9011b730 2024-04-21T08:55:57.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.611+0000 7fbd7f7fe640 1 -- 172.21.15.28:0/866252688 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80010dd0 con 0x7fbd9011b730 2024-04-21T08:55:57.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.613+0000 7fbd94946640 1 -- 172.21.15.28:0/866252688 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd901ac8f0 con 0x7fbd9011b730 2024-04-21T08:55:57.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.613+0000 7fbd94946640 1 -- 172.21.15.28:0/866252688 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd90110f10 con 0x7fbd9011b730 2024-04-21T08:55:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.614+0000 7fd3267a1740 10 client.26429 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.614+0000 7fd3267a1740 10 client.26429 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.614+0000 7fd3267a1740 20 client.26429 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.614+0000 7fd3267a1740 1 --2- 172.21.15.28:0/570105687 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e49b0b7f0 0x562e49b7b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.614+0000 7fd3267a1740 4 client.26429 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:55:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.614+0000 7fd3267a1740 1 -- 172.21.15.28:0/570105687 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e49a6c2f0 con 0x562e49b0b7f0 2024-04-21T08:55:57.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.619+0000 7fd32114d640 1 --2- 172.21.15.28:0/570105687 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e49b0b7f0 0x562e49b7b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.618+0000 7fbd9760c740 1 Processor -- start 2024-04-21T08:55:57.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.621+0000 7fbd7f7fe640 1 -- 172.21.15.28:0/866252688 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbd8001a410 con 0x7fbd9011b730 2024-04-21T08:55:57.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.622+0000 7fbd7f7fe640 1 --2- 172.21.15.28:0/866252688 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbd780a3f20 0x7fbd780a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.623+0000 7fd32114d640 1 --2- 172.21.15.28:0/570105687 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e49b0b7f0 0x562e49b7b9c0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9065cc64d50723db server_cookie=2d900e8563133520 in_seq=0 out_seq=0 2024-04-21T08:55:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:55:57.623+0000 7fd3017fa640 10 client.26429 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:57.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.623+0000 7fbd9760c740 1 -- start start 2024-04-21T08:55:57.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.623+0000 7fbd7f7fe640 1 -- 172.21.15.28:0/866252688 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbd800cf860 con 0x7fbd9011b730 2024-04-21T08:55:57.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.624+0000 7fbd8dd9b640 1 --2- 172.21.15.28:0/866252688 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbd780a3f20 0x7fbd780a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.625+0000 7fbd9760c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d7746580 0x5640d7746960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.625+0000 7fbd9760c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746ea0 0x5640d774f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.625+0000 7fbd9760c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d774fa80 0x5640d7751e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.625+0000 7fbd9760c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5640d76ab1e0 con 0x5640d7746ea0 2024-04-21T08:55:57.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.625+0000 7fbd9760c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5640d7648390 con 0x5640d7746580 2024-04-21T08:55:57.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.625+0000 7fbd9760c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5640d76a71a0 con 0x5640d774fa80 2024-04-21T08:55:57.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.626+0000 7fbd8dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746ea0 0x5640d774f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.626+0000 7fbd8dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746ea0 0x5640d774f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46370/0 (socket says 172.21.15.28:46370) 2024-04-21T08:55:57.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.626+0000 7fbd8dd9b640 1 -- 172.21.15.28:0/2735189041 learned_addr learned my addr 172.21.15.28:0/2735189041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.626+0000 7fbd8ed9d640 1 --2- 172.21.15.28:0/2735189041 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d774fa80 0x5640d7751e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.626+0000 7fbd8dd9b640 1 -- 172.21.15.28:0/2735189041 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d774fa80 msgr2=0x5640d7751e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.626+0000 7fbd8dd9b640 1 --2- 172.21.15.28:0/2735189041 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d774fa80 0x5640d7751e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.626+0000 7fbd8dd9b640 1 -- 172.21.15.28:0/2735189041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d7746580 msgr2=0x5640d7746960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:57.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.626+0000 7fbd8dd9b640 1 --2- 172.21.15.28:0/2735189041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d7746580 0x5640d7746960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.626+0000 7fbd8dd9b640 1 -- 172.21.15.28:0/2735189041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640d74877d0 con 0x5640d7746ea0 2024-04-21T08:55:57.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.627+0000 7fbd8dd9b640 1 --2- 172.21.15.28:0/866252688 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbd780a3f20 0x7fbd780a63e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbd84009900 tx=0x7fbd84004350 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.629+0000 7fbd8dd9b640 1 --2- 172.21.15.28:0/2735189041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746ea0 0x5640d774f540 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fbd84009230 tx=0x7fbd84048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cbe7c3e28a5ffd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.631+0000 7fbd7cff9640 1 -- 172.21.15.28:0/2735189041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84005100 con 0x5640d7746ea0 2024-04-21T08:55:57.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.631+0000 7fbd7cff9640 1 -- 172.21.15.28:0/2735189041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd840046c0 con 0x5640d7746ea0 2024-04-21T08:55:57.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.631+0000 7fbd7cff9640 1 -- 172.21.15.28:0/2735189041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84003830 con 0x5640d7746ea0 2024-04-21T08:55:57.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.632+0000 7fbd9760c740 1 -- 172.21.15.28:0/2735189041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746ea0 msgr2=0x5640d774f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.632+0000 7fbd9760c740 1 --2- 172.21.15.28:0/2735189041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746ea0 0x5640d774f540 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fbd84009230 tx=0x7fbd84048000 comp rx=0 tx=0).stop 2024-04-21T08:55:57.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.635+0000 7fbd9760c740 1 -- 172.21.15.28:0/2735189041 shutdown_connections 2024-04-21T08:55:57.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.635+0000 7fbd9760c740 1 --2- 172.21.15.28:0/2735189041 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d774fa80 0x5640d7751e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.635+0000 7fbd9760c740 1 --2- 172.21.15.28:0/2735189041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746ea0 0x5640d774f540 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.635+0000 7fbd9760c740 1 --2- 172.21.15.28:0/2735189041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d7746580 0x5640d7746960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.635+0000 7fbd9760c740 1 -- 172.21.15.28:0/2735189041 >> 172.21.15.28:0/2735189041 conn(0x5640d7740cb0 msgr2=0x5640d7741090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.639+0000 7fbd9760c740 1 -- 172.21.15.28:0/2735189041 shutdown_connections 2024-04-21T08:55:57.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.640+0000 7fbd9760c740 1 -- 172.21.15.28:0/2735189041 wait complete. 2024-04-21T08:55:57.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.640+0000 7fbd9760c740 1 -- start start 2024-04-21T08:55:57.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.640+0000 7fbd9760c740 1 Processor -- start 2024-04-21T08:55:57.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.641+0000 7fbd9760c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 0x5640d774cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.641+0000 7fbd9760c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d7746ea0 0x5640d774d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.641+0000 7fbd9760c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d774fa80 0x5640d774d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.641+0000 7fbd9760c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5640d7752810 con 0x5640d7746580 2024-04-21T08:55:57.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.641+0000 7fbd9760c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5640d76642f0 con 0x5640d774fa80 2024-04-21T08:55:57.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.641+0000 7fbd9760c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5640d7752990 con 0x5640d7746ea0 2024-04-21T08:55:57.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.641+0000 7fbd9760c740 10 client.0 _ll_register_callbacks cb 0x5640d7671580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:57.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.642+0000 7fbd8dd9b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d7746ea0 0x5640d774d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.642+0000 7fbd8dd9b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d7746ea0 0x5640d774d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:55486/0 (socket says 172.21.15.28:55486) 2024-04-21T08:55:57.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.642+0000 7fbd8ed9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d774fa80 0x5640d774d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.642+0000 7fbd8dd9b640 1 -- 172.21.15.28:0/2743346808 learned_addr learned my addr 172.21.15.28:0/2743346808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:55:57.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.643+0000 7fbd8ed9d640 1 -- 172.21.15.28:0/2743346808 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d7746ea0 msgr2=0x5640d774d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.643+0000 7fbd8ed9d640 1 --2- 172.21.15.28:0/2743346808 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d7746ea0 0x5640d774d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.643+0000 7fbd8ed9d640 1 -- 172.21.15.28:0/2743346808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 msgr2=0x5640d774cb80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:57.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.643+0000 7fbd8ed9d640 1 --2- 172.21.15.28:0/2743346808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 0x5640d774cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.643+0000 7fbd8ed9d640 1 -- 172.21.15.28:0/2743346808 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640d73bd3f0 con 0x5640d774fa80 2024-04-21T08:55:57.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.643+0000 7fbd8ed9d640 1 --2- 172.21.15.28:0/2743346808 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d774fa80 0x5640d774d600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbd800cccc0 tx=0x7fbd8009bca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6215fa729984a974 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.643+0000 7fbd75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:55:57.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.644+0000 7fbd75ffb640 1 -- 172.21.15.28:0/2743346808 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd800a4180 con 0x5640d774fa80 2024-04-21T08:55:57.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.644+0000 7fbd75ffb640 1 -- 172.21.15.28:0/2743346808 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbd800cce30 con 0x5640d774fa80 2024-04-21T08:55:57.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.644+0000 7fbd9760c740 1 -- 172.21.15.28:0/2743346808 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd840046c0 con 0x5640d774fa80 2024-04-21T08:55:57.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.645+0000 7fbd75ffb640 1 -- 172.21.15.28:0/2743346808 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbd800cd030 con 0x5640d774fa80 2024-04-21T08:55:57.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.646+0000 7fbd9760c740 10 client.26432 fetch_fsmap learned FSMap version 28 2024-04-21T08:55:57.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.646+0000 7fbd9760c740 1 -- 172.21.15.28:0/2743346808 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5640d737d0f0 con 0x5640d774fa80 2024-04-21T08:55:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.647+0000 7fbd75ffb640 1 -- 172.21.15.28:0/2743346808 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd840046c0 con 0x5640d774fa80 2024-04-21T08:55:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.647+0000 7fbd75ffb640 1 -- 172.21.15.28:0/2743346808 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fbd800a0ae0 con 0x5640d774fa80 2024-04-21T08:55:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.647+0000 7fbd9760c740 10 client.26432 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:55:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.647+0000 7fbd9760c740 10 client.26432 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.647+0000 7fbd9760c740 20 client.26432 populate_metadata read hostname 'smithi028' 2024-04-21T08:55:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.647+0000 7fbd9760c740 1 --2- 172.21.15.28:0/2743346808 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5640d775faf0 0x5640d7761f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.647+0000 7fbd9760c740 4 client.26432 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:57.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.647+0000 7fbd9760c740 1 -- 172.21.15.28:0/2743346808 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640d76b8c00 con 0x5640d775faf0 2024-04-21T08:55:57.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.653+0000 7fbd8e59c640 1 --2- 172.21.15.28:0/2743346808 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5640d775faf0 0x5640d7761f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.666+0000 7fbd8e59c640 1 --2- 172.21.15.28:0/2743346808 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5640d775faf0 0x5640d7761f20 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcd53e8e74aa73b3 server_cookie=2c799dd416ebcc33 in_seq=0 out_seq=0 2024-04-21T08:55:57.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:55:57.666+0000 7fbd75ffb640 10 client.26432 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:55:58.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:58 smithi152 ceph-mon[31401]: pgmap v146: 129 pgs: 129 active+clean; 17 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 45 MiB/s wr, 6.81k op/s 2024-04-21T08:55:58.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:58 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:55:58.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:58 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:58.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:55:58 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:55:58.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:58 smithi028 ceph-mon[28003]: pgmap v146: 129 pgs: 129 active+clean; 17 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 45 MiB/s wr, 6.81k op/s 2024-04-21T08:55:58.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:58 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:55:58.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:58 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:58.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:55:58 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:55:58.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:58 smithi171 ceph-mon[31358]: pgmap v146: 129 pgs: 129 active+clean; 17 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 45 MiB/s wr, 6.81k op/s 2024-04-21T08:55:58.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:58 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:55:58.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:58 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:58.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:55:58 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:56:00.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:00 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:56:00.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:00 smithi152 ceph-mon[31401]: pgmap v147: 129 pgs: 129 active+clean; 17 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 50 MiB/s wr, 7.66k op/s 2024-04-21T08:56:00.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:00 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:56:00.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:00 smithi028 ceph-mon[28003]: pgmap v147: 129 pgs: 129 active+clean; 17 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 50 MiB/s wr, 7.66k op/s 2024-04-21T08:56:00.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:00 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:56:00.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:00 smithi171 ceph-mon[31358]: pgmap v147: 129 pgs: 129 active+clean; 17 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 50 MiB/s wr, 7.66k op/s 2024-04-21T08:56:02.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:02 smithi152 ceph-mon[31401]: pgmap v148: 129 pgs: 129 active+clean; 17 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 44 MiB/s wr, 6.94k op/s 2024-04-21T08:56:02.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:02 smithi028 ceph-mon[28003]: pgmap v148: 129 pgs: 129 active+clean; 17 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 44 MiB/s wr, 6.94k op/s 2024-04-21T08:56:02.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:02 smithi171 ceph-mon[31358]: pgmap v148: 129 pgs: 129 active+clean; 17 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 44 MiB/s wr, 6.94k op/s 2024-04-21T08:56:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.379+0000 7fd3017fa640 1 -- 172.21.15.28:0/570105687 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562e49a6c2f0 con 0x562e49b0b7f0 2024-04-21T08:56:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.379+0000 7fd3017fa640 10 client.26429 handle_command_reply: tid=0 2024-04-21T08:56:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.379+0000 7fd3267a1740 1 client.26429 shutdown 2024-04-21T08:56:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.379+0000 7fd3267a1740 1 -- 172.21.15.28:0/570105687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49affff0 msgr2=0x562e49afbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.379+0000 7fd3267a1740 1 --2- 172.21.15.28:0/570105687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49affff0 0x562e49afbd20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x562e49b01080 tx=0x7fd31403ec40 comp rx=0 tx=0).stop 2024-04-21T08:56:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.379+0000 7fd3267a1740 1 -- 172.21.15.28:0/570105687 shutdown_connections 2024-04-21T08:56:02.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.379+0000 7fd3267a1740 1 --2- 172.21.15.28:0/570105687 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x562e49b0b7f0 0x562e49b7b9c0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.379+0000 7fd3267a1740 1 --2- 172.21.15.28:0/570105687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e49b030e0 0x562e49afc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.379+0000 7fd3267a1740 1 --2- 172.21.15.28:0/570105687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562e49affff0 0x562e49afbd20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.379+0000 7fd3267a1740 1 --2- 172.21.15.28:0/570105687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562e49afa8d0 0x562e49afd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.379+0000 7fd3267a1740 1 -- 172.21.15.28:0/570105687 >> 172.21.15.28:0/570105687 conn(0x562e49ac0c90 msgr2=0x562e49af9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:02.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.379+0000 7fbd75ffb640 1 -- 172.21.15.28:0/2743346808 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5640d76b8c00 con 0x5640d775faf0 2024-04-21T08:56:02.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.379+0000 7fbd75ffb640 10 client.26432 handle_command_reply: tid=0 2024-04-21T08:56:02.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.379+0000 7fbd9760c740 1 client.26432 shutdown 2024-04-21T08:56:02.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fbd9760c740 1 -- 172.21.15.28:0/2743346808 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d774fa80 msgr2=0x5640d774d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:02.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fbd9760c740 1 --2- 172.21.15.28:0/2743346808 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d774fa80 0x5640d774d600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbd800cccc0 tx=0x7fbd8009bca0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fbd9760c740 1 -- 172.21.15.28:0/2743346808 shutdown_connections 2024-04-21T08:56:02.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fbd9760c740 1 --2- 172.21.15.28:0/2743346808 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5640d775faf0 0x5640d7761f20 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fbd9760c740 1 --2- 172.21.15.28:0/2743346808 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d774fa80 0x5640d774d600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fbd9760c740 1 --2- 172.21.15.28:0/2743346808 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d7746ea0 0x5640d774d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fbd9760c740 1 --2- 172.21.15.28:0/2743346808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 0x5640d774cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fbd9760c740 1 -- 172.21.15.28:0/2743346808 >> 172.21.15.28:0/2743346808 conn(0x5640d7740cb0 msgr2=0x5640d7742a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:02.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fbd75ffb640 0 client.26432 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:02.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fd3017fa640 0 client.26429 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:02.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fbd9760c740 1 -- 172.21.15.28:0/2743346808 shutdown_connections 2024-04-21T08:56:02.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fbd9760c740 1 -- 172.21.15.28:0/2743346808 wait complete. 2024-04-21T08:56:02.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fd3267a1740 1 -- 172.21.15.28:0/570105687 shutdown_connections 2024-04-21T08:56:02.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fbd9760c740 20 client.26432 trim_cache size 0 max 16384 2024-04-21T08:56:02.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fd3267a1740 1 -- 172.21.15.28:0/570105687 wait complete. 2024-04-21T08:56:02.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.380+0000 7fd3267a1740 20 client.26429 trim_cache size 0 max 16384 2024-04-21T08:56:02.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:56:02.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:56:02.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:56:02.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:56:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.381+0000 7fd323bb2640 1 -- 172.21.15.28:0/3721089146 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd3040a3f70 msgr2=0x7fd3040a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.381+0000 7fd323bb2640 1 --2- 172.21.15.28:0/3721089146 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd3040a3f70 0x7fd3040a6430 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd31000b390 tx=0x7fd310004700 comp rx=0 tx=0).stop 2024-04-21T08:56:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.381+0000 7fd323bb2640 1 -- 172.21.15.28:0/3721089146 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd31c119300 msgr2=0x7fd31c112360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.381+0000 7fd323bb2640 1 --2- 172.21.15.28:0/3721089146 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd31c119300 0x7fd31c112360 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd31000b3c0 tx=0x7fd31000b880 comp rx=0 tx=0).stop 2024-04-21T08:56:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.382+0000 7fd323bb2640 1 -- 172.21.15.28:0/3721089146 shutdown_connections 2024-04-21T08:56:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.382+0000 7fd323bb2640 1 --2- 172.21.15.28:0/3721089146 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd3040a3f70 0x7fd3040a6430 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.382+0000 7fd323bb2640 1 --2- 172.21.15.28:0/3721089146 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd31c119300 0x7fd31c112360 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.382+0000 7fd323bb2640 1 --2- 172.21.15.28:0/3721089146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd31c11b730 0x7fd31c111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.382+0000 7fd323bb2640 1 --2- 172.21.15.28:0/3721089146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd31c110740 0x7fd31c118cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.382+0000 7fd323bb2640 1 -- 172.21.15.28:0/3721089146 >> 172.21.15.28:0/3721089146 conn(0x7fd31c06d020 msgr2=0x7fd31c10e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.382+0000 7fd323bb2640 1 -- 172.21.15.28:0/3721089146 shutdown_connections 2024-04-21T08:56:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:02.382+0000 7fd323bb2640 1 -- 172.21.15.28:0/3721089146 wait complete. 2024-04-21T08:56:02.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.417+0000 7fbd9760c740 1 Processor -- start 2024-04-21T08:56:02.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.417+0000 7fbd9760c740 1 -- start start 2024-04-21T08:56:02.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.418+0000 7fbd9760c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 0x5640d774f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:02.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.418+0000 7fbd9760c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d7746ea0 0x5640d775b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:02.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.418+0000 7fbd9760c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d774fa80 0x5640d773fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:02.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.418+0000 7fbd9760c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901c9390 con 0x5640d7746580 2024-04-21T08:56:02.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.418+0000 7fbd9760c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901c6c60 con 0x5640d774fa80 2024-04-21T08:56:02.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.418+0000 7fbd9760c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbd90068d70 con 0x5640d7746ea0 2024-04-21T08:56:02.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.418+0000 7fbd8e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 0x5640d774f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:02.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.418+0000 7fbd8e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 0x5640d774f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46384/0 (socket says 172.21.15.28:46384) 2024-04-21T08:56:02.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.418+0000 7fbd8e59c640 1 -- 172.21.15.28:0/3140305509 learned_addr learned my addr 172.21.15.28:0/3140305509 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:02.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.419+0000 7fbd8dd9b640 1 --2- 172.21.15.28:0/3140305509 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d7746ea0 0x5640d775b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:02.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.419+0000 7fbd8ed9d640 1 --2- 172.21.15.28:0/3140305509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d774fa80 0x5640d773fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:02.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.419+0000 7fbd8e59c640 1 -- 172.21.15.28:0/3140305509 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d7746ea0 msgr2=0x5640d775b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:02.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.419+0000 7fbd8e59c640 1 --2- 172.21.15.28:0/3140305509 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d7746ea0 0x5640d775b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.419+0000 7fbd8e59c640 1 -- 172.21.15.28:0/3140305509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d774fa80 msgr2=0x5640d773fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:02.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.419+0000 7fbd8e59c640 1 --2- 172.21.15.28:0/3140305509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d774fa80 0x5640d773fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.419+0000 7fbd8e59c640 1 -- 172.21.15.28:0/3140305509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640d77625a0 con 0x5640d7746580 2024-04-21T08:56:02.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.420+0000 7fbd8e59c640 1 --2- 172.21.15.28:0/3140305509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 0x5640d774f2b0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x5640d7762770 tx=0x7fbd88009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b1f7588b0788e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:02.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.423+0000 7fbd77fff640 1 -- 172.21.15.28:0/3140305509 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd88019390 con 0x5640d7746580 2024-04-21T08:56:02.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.423+0000 7fbd77fff640 1 -- 172.21.15.28:0/3140305509 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd88016430 con 0x5640d7746580 2024-04-21T08:56:02.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.423+0000 7fbd77fff640 1 -- 172.21.15.28:0/3140305509 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd88017670 con 0x5640d7746580 2024-04-21T08:56:02.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.424+0000 7fbd9760c740 1 -- 172.21.15.28:0/3140305509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 msgr2=0x5640d774f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:02.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.424+0000 7fbd9760c740 1 --2- 172.21.15.28:0/3140305509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 0x5640d774f2b0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x5640d7762770 tx=0x7fbd88009390 comp rx=0 tx=0).stop 2024-04-21T08:56:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.427+0000 7fbd9760c740 1 -- 172.21.15.28:0/3140305509 shutdown_connections 2024-04-21T08:56:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.427+0000 7fbd9760c740 1 --2- 172.21.15.28:0/3140305509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d774fa80 0x5640d773fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.427+0000 7fbd9760c740 1 --2- 172.21.15.28:0/3140305509 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d7746ea0 0x5640d775b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.427+0000 7fbd9760c740 1 --2- 172.21.15.28:0/3140305509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 0x5640d774f2b0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.427+0000 7fbd9760c740 1 -- 172.21.15.28:0/3140305509 >> 172.21.15.28:0/3140305509 conn(0x5640d7740cb0 msgr2=0x5640d773f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.430+0000 7fbd9760c740 1 -- 172.21.15.28:0/3140305509 shutdown_connections 2024-04-21T08:56:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.430+0000 7fbd9760c740 1 -- 172.21.15.28:0/3140305509 wait complete. 2024-04-21T08:56:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.431+0000 7fbd9760c740 1 -- start start 2024-04-21T08:56:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.431+0000 7fbd9760c740 1 Processor -- start 2024-04-21T08:56:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.431+0000 7fbd9760c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 0x5640d7753940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.431+0000 7fbd9760c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d7746ea0 0x5640d7753e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:02.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.431+0000 7fbd9760c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d774fa80 0x5640d7754400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:02.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.431+0000 7fbd9760c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901c51c0 con 0x5640d7746580 2024-04-21T08:56:02.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.431+0000 7fbd9760c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901c6590 con 0x5640d7746ea0 2024-04-21T08:56:02.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.431+0000 7fbd9760c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbd901c57f0 con 0x5640d774fa80 2024-04-21T08:56:02.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.431+0000 7fbd9760c740 10 client.0 _ll_register_callbacks cb 0x7fbd800971a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:02.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.432+0000 7fbd8ed9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d774fa80 0x5640d7754400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:02.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.432+0000 7fbd8ed9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d774fa80 0x5640d7754400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:55502/0 (socket says 172.21.15.28:55502) 2024-04-21T08:56:02.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.432+0000 7fbd8ed9d640 1 -- 172.21.15.28:0/1502484601 learned_addr learned my addr 172.21.15.28:0/1502484601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:02.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.432+0000 7fbd8dd9b640 1 --2- 172.21.15.28:0/1502484601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d7746ea0 0x5640d7753e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:02.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.432+0000 7fbd8e59c640 1 --2- 172.21.15.28:0/1502484601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 0x5640d7753940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.433+0000 7fbd8e59c640 1 -- 172.21.15.28:0/1502484601 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d774fa80 msgr2=0x5640d7754400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.433+0000 7fbd8e59c640 1 --2- 172.21.15.28:0/1502484601 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d774fa80 0x5640d7754400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.433+0000 7fbd8e59c640 1 -- 172.21.15.28:0/1502484601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d7746ea0 msgr2=0x5640d7753e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.433+0000 7fbd8e59c640 1 --2- 172.21.15.28:0/1502484601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d7746ea0 0x5640d7753e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.433+0000 7fbd8e59c640 1 -- 172.21.15.28:0/1502484601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640d77c5390 con 0x5640d7746580 2024-04-21T08:56:02.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.435+0000 7fbd8e59c640 1 --2- 172.21.15.28:0/1502484601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 0x5640d7753940 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009440 tx=0x7fbd880165e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2d370b3e3e926b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:02.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.435+0000 7fbd75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:56:02.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.436+0000 7fbd75ffb640 1 -- 172.21.15.28:0/1502484601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd880168e0 con 0x5640d7746580 2024-04-21T08:56:02.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.436+0000 7fbd9760c740 1 -- 172.21.15.28:0/1502484601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd88016430 con 0x5640d7746580 2024-04-21T08:56:02.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.436+0000 7fbd75ffb640 1 -- 172.21.15.28:0/1502484601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbd88016a80 con 0x5640d7746580 2024-04-21T08:56:02.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.436+0000 7fbd75ffb640 1 -- 172.21.15.28:0/1502484601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbd88036050 con 0x5640d7746580 2024-04-21T08:56:02.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.437+0000 7fbd9760c740 10 client.17172 fetch_fsmap learned FSMap version 28 2024-04-21T08:56:02.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.437+0000 7fbd9760c740 1 -- 172.21.15.28:0/1502484601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5640d77c66a0 con 0x5640d7746580 2024-04-21T08:56:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.439+0000 7fbd75ffb640 1 -- 172.21.15.28:0/1502484601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd88016430 con 0x5640d7746580 2024-04-21T08:56:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.439+0000 7fbd75ffb640 1 -- 172.21.15.28:0/1502484601 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fbd88022740 con 0x5640d7746580 2024-04-21T08:56:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.443+0000 7fbd9760c740 10 client.17172 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:56:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.443+0000 7fbd9760c740 10 client.17172 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.443+0000 7fbd9760c740 20 client.17172 populate_metadata read hostname 'smithi028' 2024-04-21T08:56:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.443+0000 7fbd9760c740 1 --2- 172.21.15.28:0/1502484601 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5640d77c6c60 0x5640d77c9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.443+0000 7fbd9760c740 4 client.17172 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:56:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.443+0000 7fbd9760c740 1 -- 172.21.15.28:0/1502484601 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5640d76b2a70 con 0x5640d77c6c60 2024-04-21T08:56:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.444+0000 7fbd8dd9b640 1 --2- 172.21.15.28:0/1502484601 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5640d77c6c60 0x5640d77c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:02.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.448+0000 7fbd8dd9b640 1 --2- 172.21.15.28:0/1502484601 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5640d77c6c60 0x5640d77c9050 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c336ead38828e4c server_cookie=9d719a5e24d767dc in_seq=0 out_seq=0 2024-04-21T08:56:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.448+0000 7fbd75ffb640 10 client.17172 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.449+0000 7fbd75ffb640 1 -- 172.21.15.28:0/1502484601 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x5640d76b2a70 con 0x5640d77c6c60 2024-04-21T08:56:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.449+0000 7fbd75ffb640 10 client.17172 handle_command_reply: tid=0 2024-04-21T08:56:02.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.450+0000 7fbd9760c740 1 client.17172 shutdown 2024-04-21T08:56:02.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.455+0000 7fbd9760c740 1 -- 172.21.15.28:0/1502484601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 msgr2=0x5640d7753940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:02.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.455+0000 7fbd9760c740 1 --2- 172.21.15.28:0/1502484601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 0x5640d7753940 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009440 tx=0x7fbd880165e0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.457+0000 7fbd9760c740 1 -- 172.21.15.28:0/1502484601 shutdown_connections 2024-04-21T08:56:02.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.457+0000 7fbd9760c740 1 --2- 172.21.15.28:0/1502484601 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5640d77c6c60 0x5640d77c9050 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.457+0000 7fbd9760c740 1 --2- 172.21.15.28:0/1502484601 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5640d774fa80 0x5640d7754400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.457+0000 7fbd9760c740 1 --2- 172.21.15.28:0/1502484601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640d7746ea0 0x5640d7753e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.457+0000 7fbd9760c740 1 --2- 172.21.15.28:0/1502484601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5640d7746580 0x5640d7753940 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.457+0000 7fbd9760c740 1 -- 172.21.15.28:0/1502484601 >> 172.21.15.28:0/1502484601 conn(0x5640d7740cb0 msgr2=0x5640d773fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:02.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.457+0000 7fbd75ffb640 0 client.17172 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:02.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.461+0000 7fbd9760c740 1 -- 172.21.15.28:0/1502484601 shutdown_connections 2024-04-21T08:56:02.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.461+0000 7fbd9760c740 1 -- 172.21.15.28:0/1502484601 wait complete. 2024-04-21T08:56:02.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.462+0000 7fbd9760c740 20 client.17172 trim_cache size 0 max 16384 2024-04-21T08:56:02.465 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:56:02.465 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (98 inodes in the stack)", 2024-04-21T08:56:02.465 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:56:02.465 INFO:teuthology.orchestra.run.smithi028.stdout: "dd219396-9868-49d6-9248-7e294e4ebcd5": { 2024-04-21T08:56:02.465 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:56:02.465 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "dd219396-9868-49d6-9248-7e294e4ebcd5", 2024-04-21T08:56:02.465 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:56:02.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:56:02.465 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:56:02.465 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:56:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.465+0000 7fbd94946640 1 -- 172.21.15.28:0/866252688 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbd780a3f20 msgr2=0x7fbd780a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.465+0000 7fbd94946640 1 --2- 172.21.15.28:0/866252688 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbd780a3f20 0x7fbd780a63e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbd84009900 tx=0x7fbd84004350 comp rx=0 tx=0).stop 2024-04-21T08:56:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.465+0000 7fbd94946640 1 -- 172.21.15.28:0/866252688 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd9011b730 msgr2=0x7fbd901abf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.465+0000 7fbd94946640 1 --2- 172.21.15.28:0/866252688 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd9011b730 0x7fbd901abf00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbd8000b420 tx=0x7fbd8000b8e0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.466+0000 7fbd94946640 1 -- 172.21.15.28:0/866252688 shutdown_connections 2024-04-21T08:56:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.466+0000 7fbd94946640 1 --2- 172.21.15.28:0/866252688 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbd780a3f20 0x7fbd780a63e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.466+0000 7fbd94946640 1 --2- 172.21.15.28:0/866252688 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbd9011b730 0x7fbd901abf00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.466+0000 7fbd94946640 1 --2- 172.21.15.28:0/866252688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd901106d0 0x7fbd901a7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.466+0000 7fbd94946640 1 --2- 172.21.15.28:0/866252688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbd9010e7b0 0x7fbd901a7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:02.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.466+0000 7fbd94946640 1 -- 172.21.15.28:0/866252688 >> 172.21.15.28:0/866252688 conn(0x7fbd9006d070 msgr2=0x7fbd9011a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:02.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.467+0000 7fbd94946640 1 -- 172.21.15.28:0/866252688 shutdown_connections 2024-04-21T08:56:02.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:02.467+0000 7fbd94946640 1 -- 172.21.15.28:0/866252688 wait complete. 2024-04-21T08:56:02.488 INFO:tasks.cephfs.filesystem:scrub status for tag:dd219396-9868-49d6-9248-7e294e4ebcd5 - {'path': '/', 'tag': 'dd219396-9868-49d6-9248-7e294e4ebcd5', 'options': 'recursive,force'} 2024-04-21T08:56:04.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:04 smithi152 ceph-mon[31401]: pgmap v149: 129 pgs: 129 active+clean; 17 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 44 MiB/s wr, 6.97k op/s 2024-04-21T08:56:04.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:04 smithi028 ceph-mon[28003]: pgmap v149: 129 pgs: 129 active+clean; 17 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 44 MiB/s wr, 6.97k op/s 2024-04-21T08:56:04.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:04 smithi171 ceph-mon[31358]: pgmap v149: 129 pgs: 129 active+clean; 17 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 44 MiB/s wr, 6.97k op/s 2024-04-21T08:56:06.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:06 smithi152 ceph-mon[31401]: pgmap v150: 129 pgs: 129 active+clean; 17 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 45 MiB/s wr, 7.26k op/s 2024-04-21T08:56:06.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:06 smithi028 ceph-mon[28003]: pgmap v150: 129 pgs: 129 active+clean; 17 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 45 MiB/s wr, 7.26k op/s 2024-04-21T08:56:06.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:06 smithi171 ceph-mon[31358]: pgmap v150: 129 pgs: 129 active+clean; 17 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 45 MiB/s wr, 7.26k op/s 2024-04-21T08:56:06.840 INFO:tasks.workunit.client.0.smithi028.stdout:Wrote -1 instead of 4096 bytes. 2024-04-21T08:56:06.840 INFO:tasks.workunit.client.0.smithi028.stdout:Probably out of disk space 2024-04-21T08:56:06.840 INFO:tasks.workunit.client.0.smithi028.stderr:write: Resource temporarily unavailable 2024-04-21T08:56:08.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:08 smithi152 ceph-mon[31401]: pgmap v151: 129 pgs: 129 active+clean; 17 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 43 MiB/s wr, 6.95k op/s 2024-04-21T08:56:08.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:08 smithi028 ceph-mon[28003]: pgmap v151: 129 pgs: 129 active+clean; 17 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 43 MiB/s wr, 6.95k op/s 2024-04-21T08:56:08.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:08 smithi171 ceph-mon[31358]: pgmap v151: 129 pgs: 129 active+clean; 17 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 43 MiB/s wr, 6.95k op/s 2024-04-21T08:56:10.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:10 smithi171 ceph-mon[31358]: pgmap v152: 129 pgs: 129 active+clean; 18 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 53 MiB/s wr, 8.82k op/s 2024-04-21T08:56:10.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:10 smithi152 ceph-mon[31401]: pgmap v152: 129 pgs: 129 active+clean; 18 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 53 MiB/s wr, 8.82k op/s 2024-04-21T08:56:10.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:10 smithi028 ceph-mon[28003]: pgmap v152: 129 pgs: 129 active+clean; 18 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 53 MiB/s wr, 8.82k op/s 2024-04-21T08:56:12.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:12 smithi171 ceph-mon[31358]: pgmap v153: 129 pgs: 129 active+clean; 18 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 46 MiB/s wr, 7.81k op/s 2024-04-21T08:56:12.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:12 smithi152 ceph-mon[31401]: pgmap v153: 129 pgs: 129 active+clean; 18 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 46 MiB/s wr, 7.81k op/s 2024-04-21T08:56:12.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:12 smithi028 ceph-mon[28003]: pgmap v153: 129 pgs: 129 active+clean; 18 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 46 MiB/s wr, 7.81k op/s 2024-04-21T08:56:14.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:14 smithi171 ceph-mon[31358]: pgmap v154: 129 pgs: 129 active+clean; 18 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 52 MiB/s wr, 8.86k op/s 2024-04-21T08:56:14.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:56:14.400 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-04-21T08:56:14.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-21T08:56:14.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:14 smithi028 ceph-mon[28003]: pgmap v154: 129 pgs: 129 active+clean; 18 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 52 MiB/s wr, 8.86k op/s 2024-04-21T08:56:14.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:14 smithi152 ceph-mon[31401]: pgmap v154: 129 pgs: 129 active+clean; 18 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 52 MiB/s wr, 8.86k op/s 2024-04-21T08:56:15.330 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi028 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f6bc5273643aaf5660909e85400647a8951c10b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-04-21T08:56:15.576 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=dc206c93661a49b2848cffd7a8054c62 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi028 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f6bc5273643aaf5660909e85400647a8951c10b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-04-21T08:56:15.578 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-21T08:56:15.589 INFO:tasks.quiescer:joining Quiescers 2024-04-21T08:56:15.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:56:15.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.536+0000 7fa53212e640 1 -- 172.21.15.28:0/1244120645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa52c109490 msgr2=0x7fa52c1098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.536+0000 7fa53212e640 1 --2- 172.21.15.28:0/1244120645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa52c109490 0x7fa52c1098f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa51c009860 tx=0x7fa51c02bb90 comp rx=0 tx=0).stop 2024-04-21T08:56:15.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.536+0000 7fa53212e640 1 -- 172.21.15.28:0/1244120645 shutdown_connections 2024-04-21T08:56:15.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.536+0000 7fa53212e640 1 --2- 172.21.15.28:0/1244120645 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa52c116fe0 0x7fa52c1193d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.536+0000 7fa53212e640 1 --2- 172.21.15.28:0/1244120645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa52c109490 0x7fa52c1098f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.536+0000 7fa53212e640 1 --2- 172.21.15.28:0/1244120645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa52c10c650 0x7fa52c108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.536+0000 7fa53212e640 1 -- 172.21.15.28:0/1244120645 >> 172.21.15.28:0/1244120645 conn(0x7fa52c06d020 msgr2=0x7fa52c06d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:15.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.537+0000 7fa53212e640 1 -- 172.21.15.28:0/1244120645 shutdown_connections 2024-04-21T08:56:15.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.537+0000 7fa53212e640 1 -- 172.21.15.28:0/1244120645 wait complete. 2024-04-21T08:56:15.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.537+0000 7fa53212e640 1 Processor -- start 2024-04-21T08:56:15.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.538+0000 7fa53212e640 1 -- start start 2024-04-21T08:56:15.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.538+0000 7fa53212e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa52c109490 0x7fa52c1154d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.538+0000 7fa53212e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa52c10c650 0x7fa52c115a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.538+0000 7fa53212e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa52c116fe0 0x7fa52c1ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.538+0000 7fa53212e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c10dca0 con 0x7fa52c116fe0 2024-04-21T08:56:15.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.538+0000 7fa53212e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c10db20 con 0x7fa52c109490 2024-04-21T08:56:15.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.538+0000 7fa53212e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa52c10de20 con 0x7fa52c10c650 2024-04-21T08:56:15.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.538+0000 7fa52affd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa52c10c650 0x7fa52c115a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.538+0000 7fa52affd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa52c10c650 0x7fa52c115a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:35574/0 (socket says 172.21.15.28:35574) 2024-04-21T08:56:15.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.538+0000 7fa52affd640 1 -- 172.21.15.28:0/3227794656 learned_addr learned my addr 172.21.15.28:0/3227794656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:15.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.538+0000 7fa52b7fe640 1 --2- 172.21.15.28:0/3227794656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa52c109490 0x7fa52c1154d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.539+0000 7fa52affd640 1 -- 172.21.15.28:0/3227794656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa52c109490 msgr2=0x7fa52c1154d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.539+0000 7fa52affd640 1 --2- 172.21.15.28:0/3227794656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa52c109490 0x7fa52c1154d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.539+0000 7fa52affd640 1 -- 172.21.15.28:0/3227794656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa52c116fe0 msgr2=0x7fa52c1ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.539+0000 7fa52affd640 1 --2- 172.21.15.28:0/3227794656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa52c116fe0 0x7fa52c1ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.539+0000 7fa52affd640 1 -- 172.21.15.28:0/3227794656 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa52c1ad710 con 0x7fa52c10c650 2024-04-21T08:56:15.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.540+0000 7fa52affd640 1 --2- 172.21.15.28:0/3227794656 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa52c10c650 0x7fa52c115a10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa51c035f90 tx=0x7fa51c004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:15.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.544+0000 7fa528ff9640 1 -- 172.21.15.28:0/3227794656 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c041020 con 0x7fa52c10c650 2024-04-21T08:56:15.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.544+0000 7fa528ff9640 1 -- 172.21.15.28:0/3227794656 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51c02c9e0 con 0x7fa52c10c650 2024-04-21T08:56:15.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.544+0000 7fa528ff9640 1 -- 172.21.15.28:0/3227794656 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c039040 con 0x7fa52c10c650 2024-04-21T08:56:15.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.545+0000 7fa53212e640 1 -- 172.21.15.28:0/3227794656 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa52c1ad9a0 con 0x7fa52c10c650 2024-04-21T08:56:15.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.546+0000 7fa53212e640 1 -- 172.21.15.28:0/3227794656 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa52c1ade40 con 0x7fa52c10c650 2024-04-21T08:56:15.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.554+0000 7fa534df4740 1 Processor -- start 2024-04-21T08:56:15.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.556+0000 7fa528ff9640 1 -- 172.21.15.28:0/3227794656 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa51c044020 con 0x7fa52c10c650 2024-04-21T08:56:15.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.557+0000 7fa528ff9640 1 --2- 172.21.15.28:0/3227794656 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa5140a3fa0 0x7fa5140a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.557+0000 7fa528ff9640 1 -- 172.21.15.28:0/3227794656 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fa51c0f2360 con 0x7fa52c10c650 2024-04-21T08:56:15.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.558+0000 7fa534df4740 1 -- start start 2024-04-21T08:56:15.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.558+0000 7fa534df4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 0x55ab66dca620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.558+0000 7fa534df4740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dcafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.558+0000 7fa534df4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dd40e0 0x55ab66dd64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.558+0000 7fa534df4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ab66d2f8d0 con 0x55ab66dcb8d0 2024-04-21T08:56:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.558+0000 7fa534df4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55ab66ccce80 con 0x55ab66dd40e0 2024-04-21T08:56:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.558+0000 7fa534df4740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55ab66d2b890 con 0x55ab66dcab60 2024-04-21T08:56:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.559+0000 7fa52affd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dcafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.559+0000 7fa52affd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dcafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:35580/0 (socket says 172.21.15.28:35580) 2024-04-21T08:56:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.559+0000 7fa52affd640 1 -- 172.21.15.28:0/1930290817 learned_addr learned my addr 172.21.15.28:0/1930290817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.560+0000 7fa52affd640 1 -- 172.21.15.28:0/1930290817 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dd40e0 msgr2=0x55ab66dd64d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:56:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.560+0000 7fa52affd640 1 --2- 172.21.15.28:0/1930290817 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dd40e0 0x55ab66dd64d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.560+0000 7fa52affd640 1 -- 172.21.15.28:0/1930290817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 msgr2=0x55ab66dca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.560+0000 7fa52affd640 1 --2- 172.21.15.28:0/1930290817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 0x55ab66dca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.560+0000 7fa52affd640 1 -- 172.21.15.28:0/1930290817 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab66cee320 con 0x55ab66dcab60 2024-04-21T08:56:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.561+0000 7fa52bfff640 1 --2- 172.21.15.28:0/3227794656 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa5140a3fa0 0x7fa5140a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.561+0000 7fa52affd640 1 --2- 172.21.15.28:0/1930290817 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dcafe0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa51c0c6750 tx=0x7fa51c0bda30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d227f4bf31cff18 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.562+0000 7fa511ffb640 1 -- 172.21.15.28:0/1930290817 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c0d7070 con 0x55ab66dcab60 2024-04-21T08:56:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.562+0000 7fa511ffb640 1 -- 172.21.15.28:0/1930290817 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51c0c23c0 con 0x55ab66dcab60 2024-04-21T08:56:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.562+0000 7fa511ffb640 1 -- 172.21.15.28:0/1930290817 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c0ba800 con 0x55ab66dcab60 2024-04-21T08:56:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.563+0000 7fa534df4740 1 -- 172.21.15.28:0/1930290817 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 msgr2=0x55ab66dcafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.563+0000 7fa534df4740 1 --2- 172.21.15.28:0/1930290817 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dcafe0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa51c0c6750 tx=0x7fa51c0bda30 comp rx=0 tx=0).stop 2024-04-21T08:56:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.565+0000 7fa52bfff640 1 --2- 172.21.15.28:0/3227794656 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa5140a3fa0 0x7fa5140a6460 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55ab66dc69e0 tx=0x7fa520005c90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.565+0000 7fa534df4740 1 -- 172.21.15.28:0/1930290817 shutdown_connections 2024-04-21T08:56:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.565+0000 7fa534df4740 1 --2- 172.21.15.28:0/1930290817 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dd40e0 0x55ab66dd64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.565+0000 7fa534df4740 1 --2- 172.21.15.28:0/1930290817 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dcafe0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.565+0000 7fa534df4740 1 --2- 172.21.15.28:0/1930290817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 0x55ab66dca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.565+0000 7fa534df4740 1 -- 172.21.15.28:0/1930290817 >> 172.21.15.28:0/1930290817 conn(0x55ab66d91c90 msgr2=0x55ab66dc5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.571+0000 7fa534df4740 1 -- 172.21.15.28:0/1930290817 shutdown_connections 2024-04-21T08:56:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.571+0000 7fa534df4740 1 -- 172.21.15.28:0/1930290817 wait complete. 2024-04-21T08:56:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.571+0000 7fa534df4740 1 -- start start 2024-04-21T08:56:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.572+0000 7fa534df4740 1 Processor -- start 2024-04-21T08:56:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.572+0000 7fa534df4740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dd3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.572+0000 7fa534df4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dcb8d0 0x55ab66dd0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.572+0000 7fa534df4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dd40e0 0x55ab66dd1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.572+0000 7fa534df4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ab66dd6bd0 con 0x55ab66dd40e0 2024-04-21T08:56:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.572+0000 7fa534df4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55ab66cb7eb0 con 0x55ab66dcb8d0 2024-04-21T08:56:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.572+0000 7fa534df4740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55ab66dd6d50 con 0x55ab66dcab60 2024-04-21T08:56:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.572+0000 7fa534df4740 10 client.0 _ll_register_callbacks cb 0x55ab66ce4b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.572+0000 7fa52affd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dcb8d0 0x55ab66dd0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.572+0000 7fa52affd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dcb8d0 0x55ab66dd0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:41984/0 (socket says 172.21.15.28:41984) 2024-04-21T08:56:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.573+0000 7fa52affd640 1 -- 172.21.15.28:0/2048256159 learned_addr learned my addr 172.21.15.28:0/2048256159 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.573+0000 7fa52b7fe640 1 --2- 172.21.15.28:0/2048256159 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dd3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.573+0000 7fa52affd640 1 -- 172.21.15.28:0/2048256159 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 msgr2=0x55ab66dd3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.573+0000 7fa52affd640 1 --2- 172.21.15.28:0/2048256159 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dd3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.573+0000 7fa52affd640 1 -- 172.21.15.28:0/2048256159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dd40e0 msgr2=0x55ab66dd1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.573+0000 7fa52affd640 1 --2- 172.21.15.28:0/2048256159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dd40e0 0x55ab66dd1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.573+0000 7fa52affd640 1 -- 172.21.15.28:0/2048256159 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab66a42210 con 0x55ab66dcb8d0 2024-04-21T08:56:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.573+0000 7fa52b7fe640 1 --2- 172.21.15.28:0/2048256159 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dd3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:56:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.573+0000 7fa52affd640 1 --2- 172.21.15.28:0/2048256159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dcb8d0 0x55ab66dd0e00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa51c0040b0 tx=0x7fa51c0095b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d8adff91ac53 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:15.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.576+0000 7fa50affd640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:56:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.576+0000 7fa50affd640 1 -- 172.21.15.28:0/2048256159 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c0dc060 con 0x55ab66dcb8d0 2024-04-21T08:56:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.576+0000 7fa50affd640 1 -- 172.21.15.28:0/2048256159 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fa51c0efe00 con 0x55ab66dcb8d0 2024-04-21T08:56:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.576+0000 7fa534df4740 1 -- 172.21.15.28:0/2048256159 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa51c0c23c0 con 0x55ab66dcb8d0 2024-04-21T08:56:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.578+0000 7fa50affd640 1 -- 172.21.15.28:0/2048256159 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa51c0d7040 con 0x55ab66dcb8d0 2024-04-21T08:56:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.578+0000 7fa534df4740 10 client.26447 fetch_fsmap learned FSMap version 28 2024-04-21T08:56:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.578+0000 7fa534df4740 1 -- 172.21.15.28:0/2048256159 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ab66b0e620 con 0x55ab66dcb8d0 2024-04-21T08:56:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.581+0000 7fa50affd640 1 -- 172.21.15.28:0/2048256159 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51c0c23c0 con 0x55ab66dcb8d0 2024-04-21T08:56:15.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.582+0000 7fa50affd640 1 -- 172.21.15.28:0/2048256159 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fa51c03c030 con 0x55ab66dcb8d0 2024-04-21T08:56:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.583+0000 7fa534df4740 10 client.26447 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:56:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.583+0000 7fa534df4740 10 client.26447 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.583+0000 7fa534df4740 20 client.26447 populate_metadata read hostname 'smithi028' 2024-04-21T08:56:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.583+0000 7fa534df4740 1 --2- 172.21.15.28:0/2048256159 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab66de3ce0 0x55ab66de60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.583+0000 7fa534df4740 4 client.26447 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:56:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.583+0000 7fa534df4740 1 -- 172.21.15.28:0/2048256159 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab66d43880 con 0x55ab66de3ce0 2024-04-21T08:56:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.584+0000 7fa52b7fe640 1 --2- 172.21.15.28:0/2048256159 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab66de3ce0 0x55ab66de60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.585+0000 7fa52b7fe640 1 --2- 172.21.15.28:0/2048256159 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab66de3ce0 0x55ab66de60d0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5c075ebe5fcd32 server_cookie=11aff4fda73add92 in_seq=0 out_seq=0 2024-04-21T08:56:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.586+0000 7fa50affd640 10 client.26447 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.587+0000 7fa50affd640 1 -- 172.21.15.28:0/2048256159 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ab66d43880 con 0x55ab66de3ce0 2024-04-21T08:56:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.587+0000 7fa50affd640 10 client.26447 handle_command_reply: tid=0 2024-04-21T08:56:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.587+0000 7fa534df4740 1 client.26447 shutdown 2024-04-21T08:56:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.589+0000 7fa534df4740 1 -- 172.21.15.28:0/2048256159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dcb8d0 msgr2=0x55ab66dd0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.589+0000 7fa534df4740 1 --2- 172.21.15.28:0/2048256159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dcb8d0 0x55ab66dd0e00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa51c0040b0 tx=0x7fa51c0095b0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.590+0000 7fa534df4740 1 -- 172.21.15.28:0/2048256159 shutdown_connections 2024-04-21T08:56:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.590+0000 7fa534df4740 1 --2- 172.21.15.28:0/2048256159 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab66de3ce0 0x55ab66de60d0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.590+0000 7fa534df4740 1 --2- 172.21.15.28:0/2048256159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dd40e0 0x55ab66dd1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.590+0000 7fa534df4740 1 --2- 172.21.15.28:0/2048256159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dcb8d0 0x55ab66dd0e00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.590+0000 7fa534df4740 1 --2- 172.21.15.28:0/2048256159 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dd3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.590+0000 7fa534df4740 1 -- 172.21.15.28:0/2048256159 >> 172.21.15.28:0/2048256159 conn(0x55ab66d91c90 msgr2=0x55ab66dc75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.590+0000 7fa50affd640 0 client.26447 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.591+0000 7fa534df4740 1 -- 172.21.15.28:0/2048256159 shutdown_connections 2024-04-21T08:56:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.594+0000 7fa534df4740 1 -- 172.21.15.28:0/2048256159 wait complete. 2024-04-21T08:56:15.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.594+0000 7fa534df4740 20 client.26447 trim_cache size 0 max 16384 2024-04-21T08:56:15.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.627+0000 7fa534df4740 1 Processor -- start 2024-04-21T08:56:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.629+0000 7fa534df4740 1 -- start start 2024-04-21T08:56:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.629+0000 7fa534df4740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66ddfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.629+0000 7fa534df4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 0x55ab66dc4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.629+0000 7fa534df4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dd40e0 0x55ab66dd3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.629+0000 7fa534df4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5180126f0 con 0x55ab66dcb8d0 2024-04-21T08:56:15.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.629+0000 7fa534df4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa51800ffc0 con 0x55ab66dd40e0 2024-04-21T08:56:15.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.629+0000 7fa534df4740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa51800dec0 con 0x55ab66dcab60 2024-04-21T08:56:15.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.630+0000 7fa52b7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66ddfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.630+0000 7fa52affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 0x55ab66dc4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.630+0000 7fa52affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 0x55ab66dc4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50340/0 (socket says 172.21.15.28:50340) 2024-04-21T08:56:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.630+0000 7fa52affd640 1 -- 172.21.15.28:0/595618237 learned_addr learned my addr 172.21.15.28:0/595618237 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.630+0000 7fa52affd640 1 -- 172.21.15.28:0/595618237 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 msgr2=0x55ab66ddfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.630+0000 7fa52affd640 1 --2- 172.21.15.28:0/595618237 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66ddfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.630+0000 7fa52affd640 1 -- 172.21.15.28:0/595618237 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dd40e0 msgr2=0x55ab66dd3530 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:56:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.630+0000 7fa52affd640 1 --2- 172.21.15.28:0/595618237 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dd40e0 0x55ab66dd3530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.630+0000 7fa52affd640 1 -- 172.21.15.28:0/595618237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab66dc9e30 con 0x55ab66dcb8d0 2024-04-21T08:56:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.631+0000 7fa52affd640 1 --2- 172.21.15.28:0/595618237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 0x55ab66dc4750 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fa51c037040 tx=0x7fa51c0bc520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2abba138604be4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.633+0000 7fa5117fa640 1 -- 172.21.15.28:0/595618237 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c0dc070 con 0x55ab66dcb8d0 2024-04-21T08:56:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.633+0000 7fa5117fa640 1 -- 172.21.15.28:0/595618237 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51c0c23c0 con 0x55ab66dcb8d0 2024-04-21T08:56:15.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.633+0000 7fa5117fa640 1 -- 172.21.15.28:0/595618237 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c02cb90 con 0x55ab66dcb8d0 2024-04-21T08:56:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.635+0000 7fa534df4740 1 -- 172.21.15.28:0/595618237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 msgr2=0x55ab66dc4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.635+0000 7fa534df4740 1 --2- 172.21.15.28:0/595618237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 0x55ab66dc4750 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fa51c037040 tx=0x7fa51c0bc520 comp rx=0 tx=0).stop 2024-04-21T08:56:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.636+0000 7fa534df4740 1 -- 172.21.15.28:0/595618237 shutdown_connections 2024-04-21T08:56:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.636+0000 7fa534df4740 1 --2- 172.21.15.28:0/595618237 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dd40e0 0x55ab66dd3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.636+0000 7fa534df4740 1 --2- 172.21.15.28:0/595618237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 0x55ab66dc4750 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.636+0000 7fa534df4740 1 --2- 172.21.15.28:0/595618237 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66ddfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.636+0000 7fa534df4740 1 -- 172.21.15.28:0/595618237 >> 172.21.15.28:0/595618237 conn(0x55ab66d91c90 msgr2=0x55ab66dc4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.636+0000 7fa534df4740 1 -- 172.21.15.28:0/595618237 shutdown_connections 2024-04-21T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.637+0000 7fa534df4740 1 -- 172.21.15.28:0/595618237 wait complete. 2024-04-21T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.638+0000 7fa534df4740 1 -- start start 2024-04-21T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.638+0000 7fa534df4740 1 Processor -- start 2024-04-21T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.639+0000 7fa534df4740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dcdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.639+0000 7fa534df4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 0x55ab66dce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.639+0000 7fa534df4740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dd40e0 0x55ab66dce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.639+0000 7fa534df4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa51800e480 con 0x55ab66dcb8d0 2024-04-21T08:56:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.639+0000 7fa534df4740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fa51800f8f0 con 0x55ab66dd40e0 2024-04-21T08:56:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.639+0000 7fa534df4740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fa51800eb50 con 0x55ab66dcab60 2024-04-21T08:56:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.639+0000 7fa534df4740 10 client.0 _ll_register_callbacks cb 0x7fa51c0d5070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.641+0000 7fa52affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 0x55ab66dce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.641+0000 7fa52affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 0x55ab66dce280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50356/0 (socket says 172.21.15.28:50356) 2024-04-21T08:56:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.641+0000 7fa52affd640 1 -- 172.21.15.28:0/32127402 learned_addr learned my addr 172.21.15.28:0/32127402 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.641+0000 7fa52b7fe640 1 --2- 172.21.15.28:0/32127402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dcdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.641+0000 7fa52affd640 1 -- 172.21.15.28:0/32127402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 msgr2=0x55ab66dcdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.641+0000 7fa52affd640 1 --2- 172.21.15.28:0/32127402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dcdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.641+0000 7fa52affd640 1 -- 172.21.15.28:0/32127402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dd40e0 msgr2=0x55ab66dce7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:56:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.641+0000 7fa52affd640 1 --2- 172.21.15.28:0/32127402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dd40e0 0x55ab66dce7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.641+0000 7fa52affd640 1 -- 172.21.15.28:0/32127402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab66e49320 con 0x55ab66dcb8d0 2024-04-21T08:56:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.641+0000 7fa52b7fe640 1 --2- 172.21.15.28:0/32127402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dcdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:56:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.644+0000 7fa52affd640 1 --2- 172.21.15.28:0/32127402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 0x55ab66dce280 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fa51c0efcb0 tx=0x7fa51c0c6a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f104c3251b40210e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.644+0000 7fa50affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:56:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.644+0000 7fa50affd640 1 -- 172.21.15.28:0/32127402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51c0dc070 con 0x55ab66dcb8d0 2024-04-21T08:56:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.644+0000 7fa50affd640 1 -- 172.21.15.28:0/32127402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fa51c0efe00 con 0x55ab66dcb8d0 2024-04-21T08:56:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.644+0000 7fa534df4740 1 -- 172.21.15.28:0/32127402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa51c0c23c0 con 0x55ab66dcb8d0 2024-04-21T08:56:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.645+0000 7fa50affd640 1 -- 172.21.15.28:0/32127402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa51c0bdb50 con 0x55ab66dcb8d0 2024-04-21T08:56:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.645+0000 7fa534df4740 10 client.17181 fetch_fsmap learned FSMap version 28 2024-04-21T08:56:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.645+0000 7fa534df4740 1 -- 172.21.15.28:0/32127402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ab66e49320 con 0x55ab66dcb8d0 2024-04-21T08:56:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.646+0000 7fa50affd640 1 -- 172.21.15.28:0/32127402 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51c0c23c0 con 0x55ab66dcb8d0 2024-04-21T08:56:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.646+0000 7fa50affd640 1 -- 172.21.15.28:0/32127402 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fa51c0b9d80 con 0x55ab66dcb8d0 2024-04-21T08:56:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.646+0000 7fa534df4740 10 client.17181 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:56:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.646+0000 7fa534df4740 10 client.17181 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.646+0000 7fa534df4740 20 client.17181 populate_metadata read hostname 'smithi028' 2024-04-21T08:56:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.646+0000 7fa534df4740 1 --2- 172.21.15.28:0/32127402 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab66e6c920 0x55ab66e6ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.646+0000 7fa534df4740 4 client.17181 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:56:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.646+0000 7fa534df4740 1 -- 172.21.15.28:0/32127402 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ab66ccf3d0 con 0x55ab66e6c920 2024-04-21T08:56:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.648+0000 7fa52b7fe640 1 --2- 172.21.15.28:0/32127402 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab66e6c920 0x55ab66e6ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.650+0000 7fa52b7fe640 1 --2- 172.21.15.28:0/32127402 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab66e6c920 0x55ab66e6ed10 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0d61ef34fbea188 server_cookie=e5b0ac784d57006d in_seq=0 out_seq=0 2024-04-21T08:56:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:15.650+0000 7fa50affd640 10 client.17181 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.814+0000 7f64ba5ca640 1 -- 172.21.15.28:0/1446871903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64b4106e50 msgr2=0x7f64b4107230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.814+0000 7f64ba5ca640 1 --2- 172.21.15.28:0/1446871903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64b4106e50 0x7f64b4107230 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009860 tx=0x7f64a802bb70 comp rx=0 tx=0).stop 2024-04-21T08:56:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.814+0000 7f64ba5ca640 1 -- 172.21.15.28:0/1446871903 shutdown_connections 2024-04-21T08:56:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.814+0000 7f64ba5ca640 1 --2- 172.21.15.28:0/1446871903 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64b406b230 0x7f64b41049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.814+0000 7f64ba5ca640 1 --2- 172.21.15.28:0/1446871903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b406a6f0 0x7f64b406ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.814+0000 7f64ba5ca640 1 --2- 172.21.15.28:0/1446871903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64b4106e50 0x7f64b4107230 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.814+0000 7f64ba5ca640 1 -- 172.21.15.28:0/1446871903 >> 172.21.15.28:0/1446871903 conn(0x7f64b4077c20 msgr2=0x7f64b4078030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.815+0000 7f64ba5ca640 1 -- 172.21.15.28:0/1446871903 shutdown_connections 2024-04-21T08:56:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.815+0000 7f64ba5ca640 1 -- 172.21.15.28:0/1446871903 wait complete. 2024-04-21T08:56:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.817+0000 7f64ba5ca640 1 Processor -- start 2024-04-21T08:56:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.817+0000 7f64ba5ca640 1 -- start start 2024-04-21T08:56:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.818+0000 7f64ba5ca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b406a6f0 0x7f64b4196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.818+0000 7f64ba5ca640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64b406b230 0x7f64b4196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.818+0000 7f64ba5ca640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64b4106e50 0x7f64b419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.818+0000 7f64ba5ca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64b4109f20 con 0x7f64b406a6f0 2024-04-21T08:56:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.818+0000 7f64ba5ca640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f64b4109da0 con 0x7f64b406b230 2024-04-21T08:56:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.818+0000 7f64ba5ca640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f64b410a0a0 con 0x7f64b4106e50 2024-04-21T08:56:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.818+0000 7f64b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b406a6f0 0x7f64b4196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.818+0000 7f64b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b406a6f0 0x7f64b4196540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50376/0 (socket says 172.21.15.28:50376) 2024-04-21T08:56:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.818+0000 7f64b3fff640 1 -- 172.21.15.28:0/1852808854 learned_addr learned my addr 172.21.15.28:0/1852808854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.819+0000 7f64b8b67640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64b4106e50 0x7f64b419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.819+0000 7f64b37fe640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64b406b230 0x7f64b4196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.819+0000 7f64b3fff640 1 -- 172.21.15.28:0/1852808854 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64b4106e50 msgr2=0x7f64b419ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.819+0000 7f64b3fff640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64b4106e50 0x7f64b419ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.820+0000 7f64b3fff640 1 -- 172.21.15.28:0/1852808854 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64b406b230 msgr2=0x7f64b4196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.820+0000 7f64b3fff640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64b406b230 0x7f64b4196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.820+0000 7f64b3fff640 1 -- 172.21.15.28:0/1852808854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b419b500 con 0x7f64b406a6f0 2024-04-21T08:56:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.820+0000 7f64b37fe640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64b406b230 0x7f64b4196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:56:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.820+0000 7f64b8b67640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64b4106e50 0x7f64b419ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:56:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.821+0000 7f64b3fff640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b406a6f0 0x7f64b4196540 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f64a80057c0 tx=0x7f64a80359a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.821+0000 7f64b17fa640 1 -- 172.21.15.28:0/1852808854 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a8041020 con 0x7f64b406a6f0 2024-04-21T08:56:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.821+0000 7f64b17fa640 1 -- 172.21.15.28:0/1852808854 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64a802c9c0 con 0x7f64b406a6f0 2024-04-21T08:56:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.822+0000 7f64b17fa640 1 -- 172.21.15.28:0/1852808854 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a8039040 con 0x7f64b406a6f0 2024-04-21T08:56:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.822+0000 7f64ba5ca640 1 -- 172.21.15.28:0/1852808854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64b419b790 con 0x7f64b406a6f0 2024-04-21T08:56:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.822+0000 7f64ba5ca640 1 -- 172.21.15.28:0/1852808854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64b419bc30 con 0x7f64b406a6f0 2024-04-21T08:56:15.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.826+0000 7f64b17fa640 1 -- 172.21.15.28:0/1852808854 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f64a8044020 con 0x7f64b406a6f0 2024-04-21T08:56:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.826+0000 7f64ba5ca640 1 -- 172.21.15.28:0/1852808854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64b4069230 con 0x7f64b406a6f0 2024-04-21T08:56:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.829+0000 7f64b17fa640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f649c0a3ef0 0x7f649c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.829+0000 7f64b17fa640 1 -- 172.21.15.28:0/1852808854 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f64a80f22e0 con 0x7f64b406a6f0 2024-04-21T08:56:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.832+0000 7f64b37fe640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f649c0a3ef0 0x7f649c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.833+0000 7f64b37fe640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f649c0a3ef0 0x7f649c0a63b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f64b4078520 tx=0x7f64a401a270 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:15.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:15.841+0000 7f64b17fa640 1 -- 172.21.15.28:0/1852808854 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f64b419bc30 con 0x7f64b406a6f0 2024-04-21T08:56:16.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.310+0000 7f64ba5ca640 1 -- 172.21.15.28:0/1852808854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64b406bac0 con 0x7f64b406a6f0 2024-04-21T08:56:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.312+0000 7f64b17fa640 1 -- 172.21.15.28:0/1852808854 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7f64a804b020 con 0x7f64b406a6f0 2024-04-21T08:56:16.313 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:56:16.313 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:56:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T08:56:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.316+0000 7f64ba5ca640 1 -- 172.21.15.28:0/1852808854 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f649c0a3ef0 msgr2=0x7f649c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.316+0000 7f64ba5ca640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f649c0a3ef0 0x7f649c0a63b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f64b4078520 tx=0x7f64a401a270 comp rx=0 tx=0).stop 2024-04-21T08:56:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.316+0000 7f64ba5ca640 1 -- 172.21.15.28:0/1852808854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b406a6f0 msgr2=0x7f64b4196540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:16.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.316+0000 7f64ba5ca640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b406a6f0 0x7f64b4196540 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f64a80057c0 tx=0x7f64a80359a0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.317+0000 7f64ba5ca640 1 -- 172.21.15.28:0/1852808854 shutdown_connections 2024-04-21T08:56:16.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.317+0000 7f64ba5ca640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f649c0a3ef0 0x7f649c0a63b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.317+0000 7f64ba5ca640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64b4106e50 0x7f64b419ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.317+0000 7f64ba5ca640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64b406b230 0x7f64b4196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.317+0000 7f64ba5ca640 1 --2- 172.21.15.28:0/1852808854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b406a6f0 0x7f64b4196540 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.317+0000 7f64ba5ca640 1 -- 172.21.15.28:0/1852808854 >> 172.21.15.28:0/1852808854 conn(0x7f64b4077c20 msgr2=0x7f64b4101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:16.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.317+0000 7f64ba5ca640 1 -- 172.21.15.28:0/1852808854 shutdown_connections 2024-04-21T08:56:16.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.317+0000 7f64ba5ca640 1 -- 172.21.15.28:0/1852808854 wait complete. 2024-04-21T08:56:16.337 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24488 quiesce db --cancel --all' 2024-04-21T08:56:16.337 DEBUG:teuthology.orchestra.run.smithi028:> timeout 45 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24488 quiesce db --cancel --all 2024-04-21T08:56:16.399 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:16 smithi028 ceph-mon[28003]: pgmap v155: 129 pgs: 129 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 47 MiB/s wr, 8.10k op/s 2024-04-21T08:56:16.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.542+0000 7f8a667a4640 1 -- 172.21.15.28:0/2733214675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a60106160 msgr2=0x7f8a6010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:16.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.542+0000 7f8a667a4640 1 --2- 172.21.15.28:0/2733214675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a60106160 0x7f8a6010dd30 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f8a5801acb0 tx=0x7f8a5803d170 comp rx=0 tx=0).stop 2024-04-21T08:56:16.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.542+0000 7f8a667a4640 1 -- 172.21.15.28:0/2733214675 shutdown_connections 2024-04-21T08:56:16.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.542+0000 7f8a667a4640 1 --2- 172.21.15.28:0/2733214675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a60106160 0x7f8a6010dd30 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.542+0000 7f8a667a4640 1 --2- 172.21.15.28:0/2733214675 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a60101a40 0x7f8a60105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.542+0000 7f8a667a4640 1 --2- 172.21.15.28:0/2733214675 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a60101090 0x7f8a60101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.542+0000 7f8a667a4640 1 -- 172.21.15.28:0/2733214675 >> 172.21.15.28:0/2733214675 conn(0x7f8a600771d0 msgr2=0x7f8a600775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:16.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.546+0000 7f8a667a4640 1 -- 172.21.15.28:0/2733214675 shutdown_connections 2024-04-21T08:56:16.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.548+0000 7f8a667a4640 1 -- 172.21.15.28:0/2733214675 wait complete. 2024-04-21T08:56:16.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.549+0000 7f8a667a4640 1 Processor -- start 2024-04-21T08:56:16.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.552+0000 7f8a667a4640 1 -- start start 2024-04-21T08:56:16.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.552+0000 7f8a667a4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a60101090 0x7f8a60073870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.552+0000 7f8a667a4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a60106160 0x7f8a6006e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.552+0000 7f8a667a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6006ee80 0x7f8a6006f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.552+0000 7f8a667a4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6006d1a0 con 0x7f8a6006ee80 2024-04-21T08:56:16.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.552+0000 7f8a667a4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6006d020 con 0x7f8a60106160 2024-04-21T08:56:16.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.552+0000 7f8a667a4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6006d320 con 0x7f8a60101090 2024-04-21T08:56:16.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.553+0000 7f8a5f7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a60106160 0x7f8a6006e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:16.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.553+0000 7f8a5f7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a60106160 0x7f8a6006e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:42040/0 (socket says 172.21.15.28:42040) 2024-04-21T08:56:16.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.553+0000 7f8a5f7fe640 1 -- 172.21.15.28:0/3956375223 learned_addr learned my addr 172.21.15.28:0/3956375223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:16.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.554+0000 7f8a5ffff640 1 --2- 172.21.15.28:0/3956375223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a60101090 0x7f8a60073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:16.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.554+0000 7f8a64d41640 1 --2- 172.21.15.28:0/3956375223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6006ee80 0x7f8a6006f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:16.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.555+0000 7f8a5f7fe640 1 -- 172.21.15.28:0/3956375223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a60101090 msgr2=0x7f8a60073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.555+0000 7f8a5f7fe640 1 --2- 172.21.15.28:0/3956375223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a60101090 0x7f8a60073870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.555+0000 7f8a5f7fe640 1 -- 172.21.15.28:0/3956375223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6006ee80 msgr2=0x7f8a6006f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.555+0000 7f8a5f7fe640 1 --2- 172.21.15.28:0/3956375223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6006ee80 0x7f8a6006f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.555+0000 7f8a5f7fe640 1 -- 172.21.15.28:0/3956375223 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6019a5d0 con 0x7f8a60106160 2024-04-21T08:56:16.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.559+0000 7f8a5f7fe640 1 --2- 172.21.15.28:0/3956375223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a60106160 0x7f8a6006e940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8a5000b470 tx=0x7f8a5000b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.562+0000 7f8a5d7fa640 1 -- 172.21.15.28:0/3956375223 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a50036070 con 0x7f8a60106160 2024-04-21T08:56:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.562+0000 7f8a5d7fa640 1 -- 172.21.15.28:0/3956375223 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a50032cb0 con 0x7f8a60106160 2024-04-21T08:56:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.563+0000 7f8a5d7fa640 1 -- 172.21.15.28:0/3956375223 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a50039d50 con 0x7f8a60106160 2024-04-21T08:56:16.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.567+0000 7f8a667a4640 1 -- 172.21.15.28:0/3956375223 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a6019a8c0 con 0x7f8a60106160 2024-04-21T08:56:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.567+0000 7f8a667a4640 1 -- 172.21.15.28:0/3956375223 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a6019ad90 con 0x7f8a60106160 2024-04-21T08:56:16.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.576+0000 7f8a5d7fa640 1 -- 172.21.15.28:0/3956375223 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8a50039070 con 0x7f8a60106160 2024-04-21T08:56:16.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.577+0000 7f8a5d7fa640 1 --2- 172.21.15.28:0/3956375223 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8a480a3d80 0x7f8a480a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.577+0000 7f8a5d7fa640 1 -- 172.21.15.28:0/3956375223 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f8a500f1f30 con 0x7f8a60106160 2024-04-21T08:56:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.580+0000 7f8a69396740 1 Processor -- start 2024-04-21T08:56:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.581+0000 7f8a69396740 1 -- start start 2024-04-21T08:56:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.581+0000 7f8a69396740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d05d10 0x55d1e8d060f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.581+0000 7f8a69396740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d066c0 0x55d1e8d0ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.581+0000 7f8a69396740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d0f240 0x55d1e8d11690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.581+0000 7f8a69396740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d1e8c6b000 con 0x55d1e8d0f240 2024-04-21T08:56:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.581+0000 7f8a69396740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55d1e8c07b20 con 0x55d1e8d066c0 2024-04-21T08:56:16.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.581+0000 7f8a69396740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55d1e8c66fc0 con 0x55d1e8d05d10 2024-04-21T08:56:16.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.583+0000 7f8a5f7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d066c0 0x55d1e8d0ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:16.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.583+0000 7f8a5f7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d066c0 0x55d1e8d0ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:42056/0 (socket says 172.21.15.28:42056) 2024-04-21T08:56:16.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.583+0000 7f8a5f7fe640 1 -- 172.21.15.28:0/3108881093 learned_addr learned my addr 172.21.15.28:0/3108881093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:16.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.583+0000 7f8a64d41640 1 --2- 172.21.15.28:0/3108881093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d0f240 0x55d1e8d11690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:16.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.583+0000 7f8a5f7fe640 1 -- 172.21.15.28:0/3108881093 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d05d10 msgr2=0x55d1e8d060f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:56:16.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.583+0000 7f8a5f7fe640 1 --2- 172.21.15.28:0/3108881093 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d05d10 0x55d1e8d060f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.583+0000 7f8a5f7fe640 1 -- 172.21.15.28:0/3108881093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d0f240 msgr2=0x55d1e8d11690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:16.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.583+0000 7f8a5f7fe640 1 --2- 172.21.15.28:0/3108881093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d0f240 0x55d1e8d11690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.583+0000 7f8a5f7fe640 1 -- 172.21.15.28:0/3108881093 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1e8c21ea0 con 0x55d1e8d066c0 2024-04-21T08:56:16.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.584+0000 7f8a5f7fe640 1 --2- 172.21.15.28:0/3108881093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d066c0 0x55d1e8d0ec70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8a500ba5b0 tx=0x7f8a500bdbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2d62535c6f52ed6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:16.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.584+0000 7f8a467fc640 1 -- 172.21.15.28:0/3108881093 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a50032e60 con 0x55d1e8d066c0 2024-04-21T08:56:16.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.584+0000 7f8a467fc640 1 -- 172.21.15.28:0/3108881093 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a500ba250 con 0x55d1e8d066c0 2024-04-21T08:56:16.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.584+0000 7f8a467fc640 1 -- 172.21.15.28:0/3108881093 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a500ef500 con 0x55d1e8d066c0 2024-04-21T08:56:16.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.585+0000 7f8a5ffff640 1 --2- 172.21.15.28:0/3956375223 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8a480a3d80 0x7f8a480a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:16.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.585+0000 7f8a69396740 1 -- 172.21.15.28:0/3108881093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d066c0 msgr2=0x55d1e8d0ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:16.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.585+0000 7f8a69396740 1 --2- 172.21.15.28:0/3108881093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d066c0 0x55d1e8d0ec70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8a500ba5b0 tx=0x7f8a500bdbe0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.586+0000 7f8a69396740 1 -- 172.21.15.28:0/3108881093 shutdown_connections 2024-04-21T08:56:16.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.586+0000 7f8a69396740 1 --2- 172.21.15.28:0/3108881093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d0f240 0x55d1e8d11690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.586+0000 7f8a69396740 1 --2- 172.21.15.28:0/3108881093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d066c0 0x55d1e8d0ec70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.586+0000 7f8a69396740 1 --2- 172.21.15.28:0/3108881093 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d05d10 0x55d1e8d060f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.586+0000 7f8a69396740 1 -- 172.21.15.28:0/3108881093 >> 172.21.15.28:0/3108881093 conn(0x55d1e8d003e0 msgr2=0x55d1e8d007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:16.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.586+0000 7f8a5ffff640 1 --2- 172.21.15.28:0/3956375223 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8a480a3d80 0x7f8a480a6240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8a54006fd0 tx=0x7f8a54008040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:16.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.592+0000 7f8a69396740 1 -- 172.21.15.28:0/3108881093 shutdown_connections 2024-04-21T08:56:16.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.592+0000 7f8a69396740 1 -- 172.21.15.28:0/3108881093 wait complete. 2024-04-21T08:56:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.593+0000 7f8a69396740 1 -- start start 2024-04-21T08:56:16.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.593+0000 7f8a69396740 1 Processor -- start 2024-04-21T08:56:16.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.595+0000 7f8a69396740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d05d10 0x55d1e8d0c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.595+0000 7f8a69396740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d066c0 0x55d1e8d0c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.595+0000 7f8a69396740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d0f240 0x55d1e8d0cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.595+0000 7f8a69396740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d1e8d120c0 con 0x55d1e8d066c0 2024-04-21T08:56:16.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.595+0000 7f8a69396740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55d1e8c05510 con 0x55d1e8d05d10 2024-04-21T08:56:16.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.595+0000 7f8a69396740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55d1e8d12240 con 0x55d1e8d0f240 2024-04-21T08:56:16.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.595+0000 7f8a69396740 10 client.0 _ll_register_callbacks cb 0x55d1e8c04f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:16.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.596+0000 7f8a64d41640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d0f240 0x55d1e8d0cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:16.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.597+0000 7f8a64d41640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d0f240 0x55d1e8d0cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:35666/0 (socket says 172.21.15.28:35666) 2024-04-21T08:56:16.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.597+0000 7f8a64d41640 1 -- 172.21.15.28:0/3453652416 learned_addr learned my addr 172.21.15.28:0/3453652416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.597+0000 7f8a5ffff640 1 --2- 172.21.15.28:0/3453652416 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d05d10 0x55d1e8d0c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.597+0000 7f8a64d41640 1 -- 172.21.15.28:0/3453652416 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d05d10 msgr2=0x55d1e8d0c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.597+0000 7f8a64d41640 1 --2- 172.21.15.28:0/3453652416 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d05d10 0x55d1e8d0c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.597+0000 7f8a64d41640 1 -- 172.21.15.28:0/3453652416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d066c0 msgr2=0x55d1e8d0c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.597+0000 7f8a64d41640 1 --2- 172.21.15.28:0/3453652416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d066c0 0x55d1e8d0c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.597+0000 7f8a64d41640 1 -- 172.21.15.28:0/3453652416 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1e8a49a20 con 0x55d1e8d0f240 2024-04-21T08:56:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.599+0000 7f8a64d41640 1 --2- 172.21.15.28:0/3453652416 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d0f240 0x55d1e8d0cbe0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8a5801acb0 tx=0x7f8a58003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3bb0ffa6d433795 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.599+0000 7f8a3f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:56:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.600+0000 7f8a3f7fe640 1 -- 172.21.15.28:0/3453652416 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a58012070 con 0x55d1e8d0f240 2024-04-21T08:56:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.600+0000 7f8a3f7fe640 1 -- 172.21.15.28:0/3453652416 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f8a58004390 con 0x55d1e8d0f240 2024-04-21T08:56:16.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.601+0000 7f8a69396740 1 -- 172.21.15.28:0/3453652416 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a500ba250 con 0x55d1e8d0f240 2024-04-21T08:56:16.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:16 smithi152 ceph-mon[31401]: pgmap v155: 129 pgs: 129 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 47 MiB/s wr, 8.10k op/s 2024-04-21T08:56:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.605+0000 7f8a3f7fe640 1 -- 172.21.15.28:0/3453652416 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8a5804a020 con 0x55d1e8d0f240 2024-04-21T08:56:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.605+0000 7f8a69396740 10 client.26482 fetch_fsmap learned FSMap version 28 2024-04-21T08:56:16.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.605+0000 7f8a69396740 1 -- 172.21.15.28:0/3453652416 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d1e8aaa1b0 con 0x55d1e8d0f240 2024-04-21T08:56:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.607+0000 7f8a3f7fe640 1 -- 172.21.15.28:0/3453652416 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a500ba250 con 0x55d1e8d0f240 2024-04-21T08:56:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.607+0000 7f8a3f7fe640 1 -- 172.21.15.28:0/3453652416 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f8a5800e410 con 0x55d1e8d0f240 2024-04-21T08:56:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.607+0000 7f8a69396740 10 client.26482 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:56:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.607+0000 7f8a69396740 10 client.26482 resolve_mds: validated gid 24488 aka daemon mds.b 2024-04-21T08:56:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.607+0000 7f8a69396740 20 client.26482 populate_metadata read hostname 'smithi028' 2024-04-21T08:56:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.607+0000 7f8a69396740 1 --2- 172.21.15.28:0/3453652416 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d1e8d1fb40 0x55d1e8d21fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.607+0000 7f8a69396740 4 client.26482 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:56:16.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.607+0000 7f8a69396740 1 -- 172.21.15.28:0/3453652416 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1e8c78a20 con 0x55d1e8d1fb40 2024-04-21T08:56:16.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.608+0000 7f8a5ffff640 1 --2- 172.21.15.28:0/3453652416 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d1e8d1fb40 0x55d1e8d21fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:16.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.612+0000 7f8a5ffff640 1 --2- 172.21.15.28:0/3453652416 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d1e8d1fb40 0x55d1e8d21fc0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=351a18f8b4535cbf server_cookie=fdffedd3e2b211f5 in_seq=0 out_seq=0 2024-04-21T08:56:16.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:16.614+0000 7f8a3f7fe640 10 client.26482 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:16.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:16 smithi171 ceph-mon[31358]: pgmap v155: 129 pgs: 129 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 47 MiB/s wr, 8.10k op/s 2024-04-21T08:56:17.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:17 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1852808854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:56:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:17 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:56:17.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:17 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1852808854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:56:17.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:17 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:56:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:17 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1852808854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:56:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:17 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:56:18.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:18 smithi152 ceph-mon[31401]: pgmap v156: 129 pgs: 129 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 46 MiB/s wr, 7.90k op/s 2024-04-21T08:56:18.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:18 smithi028 ceph-mon[28003]: pgmap v156: 129 pgs: 129 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 46 MiB/s wr, 7.90k op/s 2024-04-21T08:56:18.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:18 smithi171 ceph-mon[31358]: pgmap v156: 129 pgs: 129 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 46 MiB/s wr, 7.90k op/s 2024-04-21T08:56:19.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.678+0000 7fa50affd640 1 -- 172.21.15.28:0/32127402 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ab66ccf3d0 con 0x55ab66e6c920 2024-04-21T08:56:19.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.678+0000 7fa50affd640 10 client.17181 handle_command_reply: tid=0 2024-04-21T08:56:19.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.678+0000 7fa534df4740 1 client.17181 shutdown 2024-04-21T08:56:19.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.679+0000 7fa534df4740 1 -- 172.21.15.28:0/32127402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 msgr2=0x55ab66dce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.679+0000 7fa534df4740 1 --2- 172.21.15.28:0/32127402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 0x55ab66dce280 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fa51c0efcb0 tx=0x7fa51c0c6a20 comp rx=0 tx=0).stop 2024-04-21T08:56:19.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.679+0000 7fa534df4740 1 -- 172.21.15.28:0/32127402 shutdown_connections 2024-04-21T08:56:19.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.679+0000 7fa534df4740 1 --2- 172.21.15.28:0/32127402 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55ab66e6c920 0x55ab66e6ed10 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.679+0000 7fa534df4740 1 --2- 172.21.15.28:0/32127402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ab66dd40e0 0x55ab66dce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.679+0000 7fa534df4740 1 --2- 172.21.15.28:0/32127402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab66dcb8d0 0x55ab66dce280 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.679+0000 7fa534df4740 1 --2- 172.21.15.28:0/32127402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55ab66dcab60 0x55ab66dcdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.679+0000 7fa534df4740 1 -- 172.21.15.28:0/32127402 >> 172.21.15.28:0/32127402 conn(0x55ab66d91c90 msgr2=0x55ab66dc7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:19.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.679+0000 7fa50affd640 0 client.17181 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:19.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.679+0000 7fa534df4740 1 -- 172.21.15.28:0/32127402 shutdown_connections 2024-04-21T08:56:19.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.679+0000 7fa534df4740 1 -- 172.21.15.28:0/32127402 wait complete. 2024-04-21T08:56:19.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.679+0000 7f8a3f7fe640 1 -- 172.21.15.28:0/3453652416 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d1e8c78a20 con 0x55d1e8d1fb40 2024-04-21T08:56:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.679+0000 7f8a3f7fe640 10 client.26482 handle_command_reply: tid=0 2024-04-21T08:56:19.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.679+0000 7fa534df4740 20 client.17181 trim_cache size 0 max 16384 2024-04-21T08:56:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.679+0000 7f8a69396740 1 client.26482 shutdown 2024-04-21T08:56:19.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-04-21T08:56:19.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-04-21T08:56:19.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-04-21T08:56:19.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-04-21T08:56:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.680+0000 7f8a69396740 1 -- 172.21.15.28:0/3453652416 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d0f240 msgr2=0x55d1e8d0cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.680+0000 7f8a69396740 1 --2- 172.21.15.28:0/3453652416 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d0f240 0x55d1e8d0cbe0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8a5801acb0 tx=0x7f8a58003f90 comp rx=0 tx=0).stop 2024-04-21T08:56:19.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.680+0000 7f8a69396740 1 -- 172.21.15.28:0/3453652416 shutdown_connections 2024-04-21T08:56:19.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.680+0000 7f8a69396740 1 --2- 172.21.15.28:0/3453652416 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d1e8d1fb40 0x55d1e8d21fc0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.680+0000 7f8a69396740 1 --2- 172.21.15.28:0/3453652416 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d0f240 0x55d1e8d0cbe0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.680+0000 7f8a69396740 1 --2- 172.21.15.28:0/3453652416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d066c0 0x55d1e8d0c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.680+0000 7f8a69396740 1 --2- 172.21.15.28:0/3453652416 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d05d10 0x55d1e8d0c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.680+0000 7f8a69396740 1 -- 172.21.15.28:0/3453652416 >> 172.21.15.28:0/3453652416 conn(0x55d1e8d003e0 msgr2=0x55d1e8cff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:19.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.680+0000 7f8a3f7fe640 0 client.26482 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.680+0000 7fa53212e640 1 -- 172.21.15.28:0/3227794656 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa5140a3fa0 msgr2=0x7fa5140a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.680+0000 7fa53212e640 1 --2- 172.21.15.28:0/3227794656 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa5140a3fa0 0x7fa5140a6460 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55ab66dc69e0 tx=0x7fa520005c90 comp rx=0 tx=0).stop 2024-04-21T08:56:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.680+0000 7fa53212e640 1 -- 172.21.15.28:0/3227794656 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa52c10c650 msgr2=0x7fa52c115a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.680+0000 7fa53212e640 1 --2- 172.21.15.28:0/3227794656 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa52c10c650 0x7fa52c115a10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa51c035f90 tx=0x7fa51c004040 comp rx=0 tx=0).stop 2024-04-21T08:56:19.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.680+0000 7f8a69396740 1 -- 172.21.15.28:0/3453652416 shutdown_connections 2024-04-21T08:56:19.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.680+0000 7f8a69396740 1 -- 172.21.15.28:0/3453652416 wait complete. 2024-04-21T08:56:19.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.681+0000 7f8a69396740 20 client.26482 trim_cache size 0 max 16384 2024-04-21T08:56:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.681+0000 7fa53212e640 1 -- 172.21.15.28:0/3227794656 shutdown_connections 2024-04-21T08:56:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.681+0000 7fa53212e640 1 --2- 172.21.15.28:0/3227794656 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fa5140a3fa0 0x7fa5140a6460 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.681+0000 7fa53212e640 1 --2- 172.21.15.28:0/3227794656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa52c116fe0 0x7fa52c1ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.681+0000 7fa53212e640 1 --2- 172.21.15.28:0/3227794656 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa52c10c650 0x7fa52c115a10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.681+0000 7fa53212e640 1 --2- 172.21.15.28:0/3227794656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa52c109490 0x7fa52c1154d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.681+0000 7fa53212e640 1 -- 172.21.15.28:0/3227794656 >> 172.21.15.28:0/3227794656 conn(0x7fa52c06d020 msgr2=0x7fa52c1173c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.690+0000 7fa53212e640 1 -- 172.21.15.28:0/3227794656 shutdown_connections 2024-04-21T08:56:19.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-04-21T08:56:19.690+0000 7fa53212e640 1 -- 172.21.15.28:0/3227794656 wait complete. 2024-04-21T08:56:19.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.716+0000 7f8a69396740 1 Processor -- start 2024-04-21T08:56:19.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.716+0000 7f8a69396740 1 -- start start 2024-04-21T08:56:19.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.717+0000 7f8a69396740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d05d10 0x55d1e8d0e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.717+0000 7f8a69396740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d066c0 0x55d1e8cff460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.717+0000 7f8a69396740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d0f240 0x55d1e8d0ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.717+0000 7f8a69396740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a601b80a0 con 0x55d1e8d05d10 2024-04-21T08:56:19.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.717+0000 7f8a69396740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8a601b5970 con 0x55d1e8d0f240 2024-04-21T08:56:19.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.717+0000 7f8a69396740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8a601b37c0 con 0x55d1e8d066c0 2024-04-21T08:56:19.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.717+0000 7f8a5ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d05d10 0x55d1e8d0e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.717+0000 7f8a5ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d05d10 0x55d1e8d0e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50446/0 (socket says 172.21.15.28:50446) 2024-04-21T08:56:19.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.717+0000 7f8a5ffff640 1 -- 172.21.15.28:0/2640709933 learned_addr learned my addr 172.21.15.28:0/2640709933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:19.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.717+0000 7f8a64d41640 1 --2- 172.21.15.28:0/2640709933 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d0f240 0x55d1e8d0ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.717+0000 7f8a5ffff640 1 -- 172.21.15.28:0/2640709933 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d066c0 msgr2=0x55d1e8cff460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:19.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.717+0000 7f8a5ffff640 1 --2- 172.21.15.28:0/2640709933 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d066c0 0x55d1e8cff460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.717+0000 7f8a5ffff640 1 -- 172.21.15.28:0/2640709933 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d0f240 msgr2=0x55d1e8d0ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:19.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.717+0000 7f8a5ffff640 1 --2- 172.21.15.28:0/2640709933 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d0f240 0x55d1e8d0ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.717+0000 7f8a5ffff640 1 -- 172.21.15.28:0/2640709933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1e893d310 con 0x55d1e8d05d10 2024-04-21T08:56:19.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.718+0000 7f8a5ffff640 1 --2- 172.21.15.28:0/2640709933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d05d10 0x55d1e8d0e890 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400da50 tx=0x7f8a54010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef24f4bc09eeb9c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:19.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.719+0000 7f8a45ffb640 1 -- 172.21.15.28:0/2640709933 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a54022960 con 0x55d1e8d05d10 2024-04-21T08:56:19.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.719+0000 7f8a45ffb640 1 -- 172.21.15.28:0/2640709933 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a5402c070 con 0x55d1e8d05d10 2024-04-21T08:56:19.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.719+0000 7f8a45ffb640 1 -- 172.21.15.28:0/2640709933 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a54014430 con 0x55d1e8d05d10 2024-04-21T08:56:19.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.720+0000 7f8a69396740 1 -- 172.21.15.28:0/2640709933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d05d10 msgr2=0x55d1e8d0e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.720+0000 7f8a69396740 1 --2- 172.21.15.28:0/2640709933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d05d10 0x55d1e8d0e890 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400da50 tx=0x7f8a54010000 comp rx=0 tx=0).stop 2024-04-21T08:56:19.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.720+0000 7f8a69396740 1 -- 172.21.15.28:0/2640709933 shutdown_connections 2024-04-21T08:56:19.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.721+0000 7f8a69396740 1 --2- 172.21.15.28:0/2640709933 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d0f240 0x55d1e8d0ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.721+0000 7f8a69396740 1 --2- 172.21.15.28:0/2640709933 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d066c0 0x55d1e8cff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.721+0000 7f8a69396740 1 --2- 172.21.15.28:0/2640709933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d05d10 0x55d1e8d0e890 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.721+0000 7f8a69396740 1 -- 172.21.15.28:0/2640709933 >> 172.21.15.28:0/2640709933 conn(0x55d1e8d003e0 msgr2=0x55d1e8d02030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:19.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.721+0000 7f8a69396740 1 -- 172.21.15.28:0/2640709933 shutdown_connections 2024-04-21T08:56:19.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.721+0000 7f8a69396740 1 -- 172.21.15.28:0/2640709933 wait complete. 2024-04-21T08:56:19.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.722+0000 7f8a69396740 1 -- start start 2024-04-21T08:56:19.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.722+0000 7f8a69396740 1 Processor -- start 2024-04-21T08:56:19.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.723+0000 7f8a69396740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d05d10 0x55d1e8d175c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.723+0000 7f8a69396740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d066c0 0x55d1e8d17b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.724+0000 7f8a5ffff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d05d10 0x55d1e8d175c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.724+0000 7f8a5ffff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d05d10 0x55d1e8d175c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:42096/0 (socket says 172.21.15.28:42096) 2024-04-21T08:56:19.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.724+0000 7f8a5ffff640 1 -- 172.21.15.28:0/985377934 learned_addr learned my addr 172.21.15.28:0/985377934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:19.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.724+0000 7f8a5f7fe640 1 --2- 172.21.15.28:0/985377934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d066c0 0x55d1e8d17b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.724+0000 7f8a69396740 1 --2- 172.21.15.28:0/985377934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d0f240 0x55d1e8d145d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.724+0000 7f8a69396740 1 -- 172.21.15.28:0/985377934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a601b3e30 con 0x55d1e8d0f240 2024-04-21T08:56:19.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.724+0000 7f8a69396740 1 -- 172.21.15.28:0/985377934 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8a601b52a0 con 0x55d1e8d05d10 2024-04-21T08:56:19.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.724+0000 7f8a69396740 1 -- 172.21.15.28:0/985377934 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8a601b4500 con 0x55d1e8d066c0 2024-04-21T08:56:19.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.725+0000 7f8a69396740 10 client.0 _ll_register_callbacks cb 0x7f8a58003d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:19.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.725+0000 7f8a5ffff640 1 -- 172.21.15.28:0/985377934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d066c0 msgr2=0x55d1e8d17b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:19.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.725+0000 7f8a5ffff640 1 --2- 172.21.15.28:0/985377934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d066c0 0x55d1e8d17b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.725+0000 7f8a5ffff640 1 -- 172.21.15.28:0/985377934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d0f240 msgr2=0x55d1e8d145d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:19.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.725+0000 7f8a5ffff640 1 --2- 172.21.15.28:0/985377934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d0f240 0x55d1e8d145d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.725+0000 7f8a5ffff640 1 -- 172.21.15.28:0/985377934 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1e8d14e00 con 0x55d1e8d05d10 2024-04-21T08:56:19.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.725+0000 7f8a5ffff640 1 --2- 172.21.15.28:0/985377934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d05d10 0x55d1e8d175c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8a54008010 tx=0x7f8a54022f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a831e22da52b635e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:19.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.726+0000 7f8a69396740 1 -- 172.21.15.28:0/985377934 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a5402c070 con 0x55d1e8d05d10 2024-04-21T08:56:19.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.726+0000 7f8a3f7fe640 10 client.26468 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:56:19.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.726+0000 7f8a3f7fe640 1 -- 172.21.15.28:0/985377934 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a54036020 con 0x55d1e8d05d10 2024-04-21T08:56:19.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.727+0000 7f8a3f7fe640 1 -- 172.21.15.28:0/985377934 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f8a54026740 con 0x55d1e8d05d10 2024-04-21T08:56:19.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.727+0000 7f8a3f7fe640 1 -- 172.21.15.28:0/985377934 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8a54032440 con 0x55d1e8d05d10 2024-04-21T08:56:19.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.728+0000 7f8a69396740 10 client.26468 fetch_fsmap learned FSMap version 28 2024-04-21T08:56:19.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.728+0000 7f8a69396740 1 -- 172.21.15.28:0/985377934 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55d1e8d86b60 con 0x55d1e8d05d10 2024-04-21T08:56:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.730+0000 7f8a3f7fe640 1 -- 172.21.15.28:0/985377934 <== mon.2 v2:172.21.15.152:3300/0 4 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f8a54043060 con 0x55d1e8d05d10 2024-04-21T08:56:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.730+0000 7f8a69396740 10 client.26468 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:56:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.730+0000 7f8a69396740 10 client.26468 resolve_mds: validated gid 24488 aka daemon mds.b 2024-04-21T08:56:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.730+0000 7f8a69396740 20 client.26468 populate_metadata read hostname 'smithi028' 2024-04-21T08:56:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.730+0000 7f8a69396740 1 --2- 172.21.15.28:0/985377934 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d1e8d87250 0x55d1e8d89670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.730+0000 7f8a69396740 4 client.26468 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-21T08:56:19.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.730+0000 7f8a69396740 1 -- 172.21.15.28:0/985377934 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x7f8a5402c070 con 0x55d1e8d87250 2024-04-21T08:56:19.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.730+0000 7f8a5f7fe640 1 --2- 172.21.15.28:0/985377934 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d1e8d87250 0x55d1e8d89670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.731+0000 7f8a5f7fe640 1 --2- 172.21.15.28:0/985377934 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d1e8d87250 0x55d1e8d89670 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69929ab8098e95b9 server_cookie=f1eb227ac0a0b2d7 in_seq=0 out_seq=0 2024-04-21T08:56:19.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.731+0000 7f8a3f7fe640 10 client.26468 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:19.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.731+0000 7f8a3f7fe640 1 -- 172.21.15.28:0/985377934 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+79 (crc 0 0 0) 0x7f8a5402c070 con 0x55d1e8d87250 2024-04-21T08:56:19.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.731+0000 7f8a3f7fe640 10 client.26468 handle_command_reply: tid=0 2024-04-21T08:56:19.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.731+0000 7f8a69396740 1 client.26468 shutdown 2024-04-21T08:56:19.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.733+0000 7f8a69396740 1 -- 172.21.15.28:0/985377934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d05d10 msgr2=0x55d1e8d175c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.733+0000 7f8a69396740 1 --2- 172.21.15.28:0/985377934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d05d10 0x55d1e8d175c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8a54008010 tx=0x7f8a54022f00 comp rx=0 tx=0).stop 2024-04-21T08:56:19.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.733+0000 7f8a69396740 1 -- 172.21.15.28:0/985377934 shutdown_connections 2024-04-21T08:56:19.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.733+0000 7f8a69396740 1 --2- 172.21.15.28:0/985377934 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d1e8d87250 0x55d1e8d89670 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.733+0000 7f8a69396740 1 --2- 172.21.15.28:0/985377934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d1e8d0f240 0x55d1e8d145d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.733+0000 7f8a69396740 1 --2- 172.21.15.28:0/985377934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d1e8d066c0 0x55d1e8d17b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.733+0000 7f8a69396740 1 --2- 172.21.15.28:0/985377934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1e8d05d10 0x55d1e8d175c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.733+0000 7f8a69396740 1 -- 172.21.15.28:0/985377934 >> 172.21.15.28:0/985377934 conn(0x55d1e8d003e0 msgr2=0x55d1e8d0d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:19.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.733+0000 7f8a3f7fe640 0 client.26468 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:19.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.734+0000 7f8a69396740 1 -- 172.21.15.28:0/985377934 shutdown_connections 2024-04-21T08:56:19.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.734+0000 7f8a69396740 1 -- 172.21.15.28:0/985377934 wait complete. 2024-04-21T08:56:19.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.734+0000 7f8a69396740 20 client.26468 trim_cache size 0 max 16384 2024-04-21T08:56:19.735 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:56:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 28, 2024-04-21T08:56:19.735 INFO:teuthology.orchestra.run.smithi028.stdout: "leader": 24488, 2024-04-21T08:56:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: "set_version": 8, 2024-04-21T08:56:19.736 INFO:teuthology.orchestra.run.smithi028.stdout: "sets": {} 2024-04-21T08:56:19.736 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:56:19.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.734+0000 7f8a667a4640 1 -- 172.21.15.28:0/3956375223 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8a480a3d80 msgr2=0x7f8a480a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.734+0000 7f8a667a4640 1 --2- 172.21.15.28:0/3956375223 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8a480a3d80 0x7f8a480a6240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8a54006fd0 tx=0x7f8a54008040 comp rx=0 tx=0).stop 2024-04-21T08:56:19.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.735+0000 7f8a667a4640 1 -- 172.21.15.28:0/3956375223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a60106160 msgr2=0x7f8a6006e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.735+0000 7f8a667a4640 1 --2- 172.21.15.28:0/3956375223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a60106160 0x7f8a6006e940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8a5000b470 tx=0x7f8a5000b930 comp rx=0 tx=0).stop 2024-04-21T08:56:19.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.735+0000 7f8a667a4640 1 -- 172.21.15.28:0/3956375223 shutdown_connections 2024-04-21T08:56:19.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.735+0000 7f8a667a4640 1 --2- 172.21.15.28:0/3956375223 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8a480a3d80 0x7f8a480a6240 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.735+0000 7f8a667a4640 1 --2- 172.21.15.28:0/3956375223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6006ee80 0x7f8a6006f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.735+0000 7f8a667a4640 1 --2- 172.21.15.28:0/3956375223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a60106160 0x7f8a6006e940 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.735+0000 7f8a667a4640 1 --2- 172.21.15.28:0/3956375223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a60101090 0x7f8a60073870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.735+0000 7f8a667a4640 1 -- 172.21.15.28:0/3956375223 >> 172.21.15.28:0/3956375223 conn(0x7f8a600771d0 msgr2=0x7f8a600feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:19.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.735+0000 7f8a667a4640 1 -- 172.21.15.28:0/3956375223 shutdown_connections 2024-04-21T08:56:19.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:19.735+0000 7f8a667a4640 1 -- 172.21.15.28:0/3956375223 wait complete. 2024-04-21T08:56:19.754 INFO:tasks.quiescer:done joining Quiescers 2024-04-21T08:56:19.754 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-21T08:56:19.765 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi028.front.sepia.ceph.com 2024-04-21T08:56:19.803 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-21T08:56:19.814 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-21T08:56:19.814 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T08:56:19.814 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-21T08:56:19.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-21T08:56:20.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-21T08:56:20.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:20 smithi028 ceph-mon[28003]: pgmap v157: 129 pgs: 129 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 50 MiB/s wr, 8.79k op/s 2024-04-21T08:56:20.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:20 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:56:20.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:20 smithi152 ceph-mon[31401]: pgmap v157: 129 pgs: 129 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 50 MiB/s wr, 8.79k op/s 2024-04-21T08:56:20.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:20 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:56:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:20 smithi171 ceph-mon[31358]: pgmap v157: 129 pgs: 129 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 50 MiB/s wr, 8.79k op/s 2024-04-21T08:56:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:20 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:56:22.406 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:22 smithi152 ceph-mon[31401]: pgmap v158: 129 pgs: 129 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 36 MiB/s wr, 6.43k op/s 2024-04-21T08:56:22.422 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:22 smithi171 ceph-mon[31358]: pgmap v158: 129 pgs: 129 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 36 MiB/s wr, 6.43k op/s 2024-04-21T08:56:22.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:22 smithi028 ceph-mon[28003]: pgmap v158: 129 pgs: 129 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 36 MiB/s wr, 6.43k op/s 2024-04-21T08:56:24.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:24 smithi152 ceph-mon[31401]: pgmap v159: 129 pgs: 129 active+clean; 18 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 MiB/s wr, 6.57k op/s 2024-04-21T08:56:24.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:24 smithi028 ceph-mon[28003]: pgmap v159: 129 pgs: 129 active+clean; 18 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 MiB/s wr, 6.57k op/s 2024-04-21T08:56:24.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:24 smithi171 ceph-mon[31358]: pgmap v159: 129 pgs: 129 active+clean; 18 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 MiB/s wr, 6.57k op/s 2024-04-21T08:56:26.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:26 smithi152 ceph-mon[31401]: pgmap v160: 129 pgs: 129 active+clean; 18 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 34 MiB/s wr, 6.25k op/s 2024-04-21T08:56:26.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:26 smithi028 ceph-mon[28003]: pgmap v160: 129 pgs: 129 active+clean; 18 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 34 MiB/s wr, 6.25k op/s 2024-04-21T08:56:26.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:26 smithi171 ceph-mon[31358]: pgmap v160: 129 pgs: 129 active+clean; 18 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 34 MiB/s wr, 6.25k op/s 2024-04-21T08:56:28.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:28 smithi152 ceph-mon[31401]: pgmap v161: 129 pgs: 129 active+clean; 18 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 MiB/s wr, 6.00k op/s 2024-04-21T08:56:28.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:28 smithi028 ceph-mon[28003]: pgmap v161: 129 pgs: 129 active+clean; 18 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 MiB/s wr, 6.00k op/s 2024-04-21T08:56:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:28 smithi171 ceph-mon[31358]: pgmap v161: 129 pgs: 129 active+clean; 18 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 MiB/s wr, 6.00k op/s 2024-04-21T08:56:30.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:30 smithi152 ceph-mon[31401]: pgmap v162: 129 pgs: 129 active+clean; 18 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 37 MiB/s wr, 7.20k op/s 2024-04-21T08:56:30.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:30 smithi028 ceph-mon[28003]: pgmap v162: 129 pgs: 129 active+clean; 18 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 37 MiB/s wr, 7.20k op/s 2024-04-21T08:56:30.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:30 smithi171 ceph-mon[31358]: pgmap v162: 129 pgs: 129 active+clean; 18 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 37 MiB/s wr, 7.20k op/s 2024-04-21T08:56:32.489 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:56:32.490 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:56:32.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:32 smithi028 ceph-mon[28003]: pgmap v163: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 31 MiB/s wr, 6.18k op/s 2024-04-21T08:56:32.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:32 smithi152 ceph-mon[31401]: pgmap v163: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 31 MiB/s wr, 6.18k op/s 2024-04-21T08:56:32.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:32 smithi171 ceph-mon[31358]: pgmap v163: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 31 MiB/s wr, 6.18k op/s 2024-04-21T08:56:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.689+0000 7f44ea081640 1 -- 172.21.15.28:0/735807436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e4112d40 msgr2=0x7f44e4115130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.689+0000 7f44ea081640 1 --2- 172.21.15.28:0/735807436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e4112d40 0x7f44e4115130 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f44d4009900 tx=0x7f44d402bcc0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.694+0000 7f44ea081640 1 -- 172.21.15.28:0/735807436 shutdown_connections 2024-04-21T08:56:32.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.694+0000 7f44ea081640 1 --2- 172.21.15.28:0/735807436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e4112d40 0x7f44e4115130 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.694+0000 7f44ea081640 1 --2- 172.21.15.28:0/735807436 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44e4100940 0x7f44e4112800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.694+0000 7f44ea081640 1 --2- 172.21.15.28:0/735807436 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44e4100020 0x7f44e4100400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.694+0000 7f44ea081640 1 -- 172.21.15.28:0/735807436 >> 172.21.15.28:0/735807436 conn(0x7f44e4076fa0 msgr2=0x7f44e40fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:32.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.694+0000 7f44ea081640 1 -- 172.21.15.28:0/735807436 shutdown_connections 2024-04-21T08:56:32.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.695+0000 7f44ea081640 1 -- 172.21.15.28:0/735807436 wait complete. 2024-04-21T08:56:32.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.695+0000 7f44ea081640 1 Processor -- start 2024-04-21T08:56:32.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.697+0000 7f44ea081640 1 -- start start 2024-04-21T08:56:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.697+0000 7f44ea081640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44e4100020 0x7f44e41123b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.697+0000 7f44ea081640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44e4100940 0x7f44e410c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.697+0000 7f44ea081640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e4112d40 0x7f44e410c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.698+0000 7f44ea081640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44e41088e0 con 0x7f44e4112d40 2024-04-21T08:56:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.698+0000 7f44ea081640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f44e4108760 con 0x7f44e4100020 2024-04-21T08:56:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.698+0000 7f44ea081640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f44e4108a60 con 0x7f44e4100940 2024-04-21T08:56:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.699+0000 7f44e3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e4112d40 0x7f44e410c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.699+0000 7f44e3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e4112d40 0x7f44e410c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36438/0 (socket says 172.21.15.28:36438) 2024-04-21T08:56:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.699+0000 7f44e3fff640 1 -- 172.21.15.28:0/2665178929 learned_addr learned my addr 172.21.15.28:0/2665178929 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.699+0000 7f44e2ffd640 1 --2- 172.21.15.28:0/2665178929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44e4100940 0x7f44e410c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.700+0000 7f44e2ffd640 1 -- 172.21.15.28:0/2665178929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44e4100020 msgr2=0x7f44e41123b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.700+0000 7f44e2ffd640 1 --2- 172.21.15.28:0/2665178929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44e4100020 0x7f44e41123b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.700+0000 7f44e2ffd640 1 -- 172.21.15.28:0/2665178929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e4112d40 msgr2=0x7f44e410c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.700+0000 7f44e2ffd640 1 --2- 172.21.15.28:0/2665178929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e4112d40 0x7f44e410c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.700+0000 7f44e2ffd640 1 -- 172.21.15.28:0/2665178929 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e410cff0 con 0x7f44e4100940 2024-04-21T08:56:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.701+0000 7f44e3fff640 1 --2- 172.21.15.28:0/2665178929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e4112d40 0x7f44e410c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:56:32.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.704+0000 7f44e2ffd640 1 --2- 172.21.15.28:0/2665178929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44e4100940 0x7f44e410c470 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f44dc00eb00 tx=0x7f44dc00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.705+0000 7f44e0ff9640 1 -- 172.21.15.28:0/2665178929 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44dc003970 con 0x7f44e4100940 2024-04-21T08:56:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.705+0000 7f44e0ff9640 1 -- 172.21.15.28:0/2665178929 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44dc004030 con 0x7f44e4100940 2024-04-21T08:56:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.705+0000 7f44e0ff9640 1 -- 172.21.15.28:0/2665178929 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44dc010670 con 0x7f44e4100940 2024-04-21T08:56:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.706+0000 7f44ea081640 1 -- 172.21.15.28:0/2665178929 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44e406b970 con 0x7f44e4100940 2024-04-21T08:56:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.706+0000 7f44ea081640 1 -- 172.21.15.28:0/2665178929 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44e406be70 con 0x7f44e4100940 2024-04-21T08:56:32.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.710+0000 7f44e0ff9640 1 -- 172.21.15.28:0/2665178929 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f44dc010810 con 0x7f44e4100940 2024-04-21T08:56:32.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.711+0000 7f44e0ff9640 1 --2- 172.21.15.28:0/2665178929 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f44cc0a3ea0 0x7f44cc0a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.712+0000 7f44e37fe640 1 --2- 172.21.15.28:0/2665178929 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f44cc0a3ea0 0x7f44cc0a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:32.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.713+0000 7f44e37fe640 1 --2- 172.21.15.28:0/2665178929 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f44cc0a3ea0 0x7f44cc0a6360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f44d8005fd0 tx=0x7f44d8005850 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:32.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.714+0000 7f44ecc75740 1 Processor -- start 2024-04-21T08:56:32.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.714+0000 7f44e0ff9640 1 -- 172.21.15.28:0/2665178929 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f44dc0d0f00 con 0x7f44e4100940 2024-04-21T08:56:32.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.715+0000 7f44ecc75740 1 -- start start 2024-04-21T08:56:32.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.715+0000 7f44ecc75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945580 0x558128945960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.715+0000 7f44ecc75740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945ea0 0x55812894e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.715+0000 7f44ecc75740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 0x558128950e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.715+0000 7f44ecc75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5581288aa1e0 con 0x558128945580 2024-04-21T08:56:32.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.715+0000 7f44ecc75740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558128847390 con 0x558128945ea0 2024-04-21T08:56:32.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.715+0000 7f44ecc75740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5581288a61a0 con 0x55812894ea80 2024-04-21T08:56:32.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.716+0000 7f44e2ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945ea0 0x55812894e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:32.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.717+0000 7f44e3fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 0x558128950e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:32.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.717+0000 7f44e2ffd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945ea0 0x55812894e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:56016/0 (socket says 172.21.15.28:56016) 2024-04-21T08:56:32.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.717+0000 7f44e2ffd640 1 -- 172.21.15.28:0/2660007878 learned_addr learned my addr 172.21.15.28:0/2660007878 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:32.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.717+0000 7f44e37fe640 1 --2- 172.21.15.28:0/2660007878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945580 0x558128945960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:32.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.718+0000 7f44e3fff640 1 -- 172.21.15.28:0/2660007878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945ea0 msgr2=0x55812894e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:32.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.718+0000 7f44e3fff640 1 --2- 172.21.15.28:0/2660007878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945ea0 0x55812894e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.718+0000 7f44e3fff640 1 -- 172.21.15.28:0/2660007878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945580 msgr2=0x558128945960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:32.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.718+0000 7f44e3fff640 1 --2- 172.21.15.28:0/2660007878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945580 0x558128945960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.718+0000 7f44e3fff640 1 -- 172.21.15.28:0/2660007878 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581286867d0 con 0x55812894ea80 2024-04-21T08:56:32.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.720+0000 7f44e3fff640 1 --2- 172.21.15.28:0/2660007878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 0x558128950e70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f44d4005ab0 tx=0x7f44d4009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce28b769eda3a699 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:32.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.721+0000 7f44c9ffb640 1 -- 172.21.15.28:0/2660007878 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4043070 con 0x55812894ea80 2024-04-21T08:56:32.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.721+0000 7f44c9ffb640 1 -- 172.21.15.28:0/2660007878 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44d402c870 con 0x55812894ea80 2024-04-21T08:56:32.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.721+0000 7f44c9ffb640 1 -- 172.21.15.28:0/2660007878 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d40039f0 con 0x55812894ea80 2024-04-21T08:56:32.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.722+0000 7f44ecc75740 1 -- 172.21.15.28:0/2660007878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 msgr2=0x558128950e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:32.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.722+0000 7f44ecc75740 1 --2- 172.21.15.28:0/2660007878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 0x558128950e70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f44d4005ab0 tx=0x7f44d4009610 comp rx=0 tx=0).stop 2024-04-21T08:56:32.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.723+0000 7f44ecc75740 1 -- 172.21.15.28:0/2660007878 shutdown_connections 2024-04-21T08:56:32.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.723+0000 7f44ecc75740 1 --2- 172.21.15.28:0/2660007878 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 0x558128950e70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.723+0000 7f44ecc75740 1 --2- 172.21.15.28:0/2660007878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945ea0 0x55812894e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.723+0000 7f44ecc75740 1 --2- 172.21.15.28:0/2660007878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945580 0x558128945960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.723+0000 7f44ecc75740 1 -- 172.21.15.28:0/2660007878 >> 172.21.15.28:0/2660007878 conn(0x55812893fcb0 msgr2=0x558128940090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:32.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.724+0000 7f44ecc75740 1 -- 172.21.15.28:0/2660007878 shutdown_connections 2024-04-21T08:56:32.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.726+0000 7f44ecc75740 1 -- 172.21.15.28:0/2660007878 wait complete. 2024-04-21T08:56:32.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.727+0000 7f44ecc75740 1 -- start start 2024-04-21T08:56:32.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.727+0000 7f44ecc75740 1 Processor -- start 2024-04-21T08:56:32.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.727+0000 7f44ecc75740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945580 0x55812894ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.727+0000 7f44ecc75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945ea0 0x55812894bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.727+0000 7f44ecc75740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 0x55812894c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.727+0000 7f44ecc75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558128951810 con 0x558128945ea0 2024-04-21T08:56:32.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.727+0000 7f44ecc75740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5581288632f0 con 0x558128945580 2024-04-21T08:56:32.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.727+0000 7f44ecc75740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558128951990 con 0x55812894ea80 2024-04-21T08:56:32.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.727+0000 7f44ecc75740 10 client.0 _ll_register_callbacks cb 0x558128870580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:32.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.728+0000 7f44e3fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 0x55812894c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:32.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.728+0000 7f44e3fff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 0x55812894c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:51796/0 (socket says 172.21.15.28:51796) 2024-04-21T08:56:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.728+0000 7f44e3fff640 1 -- 172.21.15.28:0/2403124680 learned_addr learned my addr 172.21.15.28:0/2403124680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.729+0000 7f44e3fff640 1 -- 172.21.15.28:0/2403124680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945580 msgr2=0x55812894ba70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:56:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.729+0000 7f44e3fff640 1 --2- 172.21.15.28:0/2403124680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945580 0x55812894ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.729+0000 7f44e3fff640 1 -- 172.21.15.28:0/2403124680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945ea0 msgr2=0x55812894bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.729+0000 7f44e3fff640 1 --2- 172.21.15.28:0/2403124680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945ea0 0x55812894bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.729+0000 7f44e3fff640 1 -- 172.21.15.28:0/2403124680 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581285bc3f0 con 0x55812894ea80 2024-04-21T08:56:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.729+0000 7f44e3fff640 1 --2- 172.21.15.28:0/2403124680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 0x55812894c4f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f44d40092a0 tx=0x7f44d4004370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59c1c71d9ae94b57 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:32.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.729+0000 7f44c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:56:32.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.730+0000 7f44c2ffd640 1 -- 172.21.15.28:0/2403124680 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d404c070 con 0x55812894ea80 2024-04-21T08:56:32.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.730+0000 7f44ecc75740 1 -- 172.21.15.28:0/2403124680 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44d402c870 con 0x55812894ea80 2024-04-21T08:56:32.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.730+0000 7f44c2ffd640 1 -- 172.21.15.28:0/2403124680 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f44d4003bc0 con 0x55812894ea80 2024-04-21T08:56:32.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.730+0000 7f44c2ffd640 1 -- 172.21.15.28:0/2403124680 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f44d4043040 con 0x55812894ea80 2024-04-21T08:56:32.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.731+0000 7f44ecc75740 10 client.26497 fetch_fsmap learned FSMap version 28 2024-04-21T08:56:32.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.731+0000 7f44ecc75740 1 -- 172.21.15.28:0/2403124680 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55812857c0f0 con 0x55812894ea80 2024-04-21T08:56:32.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.732+0000 7f44c2ffd640 1 -- 172.21.15.28:0/2403124680 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44d402c870 con 0x55812894ea80 2024-04-21T08:56:32.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.732+0000 7f44c2ffd640 1 -- 172.21.15.28:0/2403124680 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f44d403f440 con 0x55812894ea80 2024-04-21T08:56:32.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.732+0000 7f44ecc75740 10 client.26497 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:56:32.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.732+0000 7f44ecc75740 10 client.26497 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:32.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.732+0000 7f44ecc75740 20 client.26497 populate_metadata read hostname 'smithi028' 2024-04-21T08:56:32.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.732+0000 7f44ecc75740 1 --2- 172.21.15.28:0/2403124680 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55812895eaa0 0x558128960f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.732+0000 7f44ecc75740 4 client.26497 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:56:32.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.732+0000 7f44ecc75740 1 -- 172.21.15.28:0/2403124680 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581288b7c00 con 0x55812895eaa0 2024-04-21T08:56:32.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.732+0000 7f44e37fe640 1 --2- 172.21.15.28:0/2403124680 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55812895eaa0 0x558128960f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:32.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.733+0000 7f44c2ffd640 10 client.26497 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:32.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.734+0000 7f44e37fe640 1 --2- 172.21.15.28:0/2403124680 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55812895eaa0 0x558128960f60 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e27e94f83ed6fa7 server_cookie=1c2593a6dd6b81cc in_seq=0 out_seq=0 2024-04-21T08:56:32.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.735+0000 7f44c2ffd640 1 -- 172.21.15.28:0/2403124680 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5581288b7c00 con 0x55812895eaa0 2024-04-21T08:56:32.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.735+0000 7f44c2ffd640 10 client.26497 handle_command_reply: tid=0 2024-04-21T08:56:32.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.736+0000 7f44ecc75740 1 client.26497 shutdown 2024-04-21T08:56:32.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.736+0000 7f44ecc75740 1 -- 172.21.15.28:0/2403124680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 msgr2=0x55812894c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:32.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.736+0000 7f44ecc75740 1 --2- 172.21.15.28:0/2403124680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 0x55812894c4f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f44d40092a0 tx=0x7f44d4004370 comp rx=0 tx=0).stop 2024-04-21T08:56:32.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.736+0000 7f44ecc75740 1 -- 172.21.15.28:0/2403124680 shutdown_connections 2024-04-21T08:56:32.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.736+0000 7f44ecc75740 1 --2- 172.21.15.28:0/2403124680 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55812895eaa0 0x558128960f60 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.736+0000 7f44ecc75740 1 --2- 172.21.15.28:0/2403124680 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 0x55812894c4f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.736+0000 7f44ecc75740 1 --2- 172.21.15.28:0/2403124680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945ea0 0x55812894bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.736+0000 7f44ecc75740 1 --2- 172.21.15.28:0/2403124680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945580 0x55812894ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.736+0000 7f44ecc75740 1 -- 172.21.15.28:0/2403124680 >> 172.21.15.28:0/2403124680 conn(0x55812893fcb0 msgr2=0x558128941a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:32.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.736+0000 7f44c2ffd640 0 client.26497 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:32.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.737+0000 7f44ecc75740 1 -- 172.21.15.28:0/2403124680 shutdown_connections 2024-04-21T08:56:32.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.737+0000 7f44ecc75740 1 -- 172.21.15.28:0/2403124680 wait complete. 2024-04-21T08:56:32.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.737+0000 7f44ecc75740 20 client.26497 trim_cache size 0 max 16384 2024-04-21T08:56:32.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.773+0000 7f44ecc75740 1 Processor -- start 2024-04-21T08:56:32.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.773+0000 7f44ecc75740 1 -- start start 2024-04-21T08:56:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44ecc75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945580 0x55812894e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44ecc75740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945ea0 0x55812895a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44ecc75740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 0x558128941780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44ecc75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44d0012880 con 0x558128945580 2024-04-21T08:56:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44ecc75740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f44d0010150 con 0x558128945ea0 2024-04-21T08:56:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44ecc75740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f44d000dfa0 con 0x55812894ea80 2024-04-21T08:56:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945580 0x55812894e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945580 0x55812894e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36466/0 (socket says 172.21.15.28:36466) 2024-04-21T08:56:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44e37fe640 1 -- 172.21.15.28:0/3280843079 learned_addr learned my addr 172.21.15.28:0/3280843079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44e2ffd640 1 --2- 172.21.15.28:0/3280843079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945ea0 0x55812895a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44e3fff640 1 --2- 172.21.15.28:0/3280843079 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 0x558128941780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:32.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44e37fe640 1 -- 172.21.15.28:0/3280843079 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 msgr2=0x558128941780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:32.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44e37fe640 1 --2- 172.21.15.28:0/3280843079 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 0x558128941780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44e37fe640 1 -- 172.21.15.28:0/3280843079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945ea0 msgr2=0x55812895a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:32.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44e37fe640 1 --2- 172.21.15.28:0/3280843079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945ea0 0x55812895a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.774+0000 7f44e37fe640 1 -- 172.21.15.28:0/3280843079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558128949fc0 con 0x558128945580 2024-04-21T08:56:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.775+0000 7f44e37fe640 1 --2- 172.21.15.28:0/3280843079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945580 0x55812894e1a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f44d8012040 tx=0x7f44d8010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d5e393ed4be13a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.775+0000 7f44c97fa640 1 -- 172.21.15.28:0/3280843079 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d802b070 con 0x558128945580 2024-04-21T08:56:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.775+0000 7f44c97fa640 1 -- 172.21.15.28:0/3280843079 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44d8017050 con 0x558128945580 2024-04-21T08:56:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.775+0000 7f44c97fa640 1 -- 172.21.15.28:0/3280843079 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d800f620 con 0x558128945580 2024-04-21T08:56:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.776+0000 7f44ecc75740 1 -- 172.21.15.28:0/3280843079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945580 msgr2=0x55812894e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.776+0000 7f44ecc75740 1 --2- 172.21.15.28:0/3280843079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945580 0x55812894e1a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f44d8012040 tx=0x7f44d8010000 comp rx=0 tx=0).stop 2024-04-21T08:56:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.776+0000 7f44ecc75740 1 -- 172.21.15.28:0/3280843079 shutdown_connections 2024-04-21T08:56:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.776+0000 7f44ecc75740 1 --2- 172.21.15.28:0/3280843079 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55812894ea80 0x558128941780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.776+0000 7f44ecc75740 1 --2- 172.21.15.28:0/3280843079 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945ea0 0x55812895a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.776+0000 7f44ecc75740 1 --2- 172.21.15.28:0/3280843079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558128945580 0x55812894e1a0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.776+0000 7f44ecc75740 1 -- 172.21.15.28:0/3280843079 >> 172.21.15.28:0/3280843079 conn(0x55812893fcb0 msgr2=0x558128941070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:32.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.777+0000 7f44ecc75740 1 -- 172.21.15.28:0/3280843079 shutdown_connections 2024-04-21T08:56:32.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.778+0000 7f44ecc75740 1 -- 172.21.15.28:0/3280843079 wait complete. 2024-04-21T08:56:32.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.778+0000 7f44ecc75740 1 -- start start 2024-04-21T08:56:32.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.778+0000 7f44ecc75740 1 Processor -- start 2024-04-21T08:56:32.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.779+0000 7f44ecc75740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945580 0x558128956dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.779+0000 7f44ecc75740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558128945ea0 0x558128952180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.779+0000 7f44ecc75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55812894ea80 0x5581289526c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.779+0000 7f44ecc75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44d000e610 con 0x55812894ea80 2024-04-21T08:56:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.779+0000 7f44ecc75740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f44d000fa80 con 0x558128945580 2024-04-21T08:56:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.779+0000 7f44ecc75740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f44d000ece0 con 0x558128945ea0 2024-04-21T08:56:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.779+0000 7f44ecc75740 10 client.0 _ll_register_callbacks cb 0x7f44d4009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.779+0000 7f44e2ffd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558128945ea0 0x558128952180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.779+0000 7f44e2ffd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558128945ea0 0x558128952180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:51804/0 (socket says 172.21.15.28:51804) 2024-04-21T08:56:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.779+0000 7f44e2ffd640 1 -- 172.21.15.28:0/3247690828 learned_addr learned my addr 172.21.15.28:0/3247690828 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:56:32.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.780+0000 7f44e2ffd640 1 -- 172.21.15.28:0/3247690828 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945580 msgr2=0x558128956dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:56:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.780+0000 7f44e2ffd640 1 --2- 172.21.15.28:0/3247690828 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945580 0x558128956dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.780+0000 7f44e2ffd640 1 -- 172.21.15.28:0/3247690828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55812894ea80 msgr2=0x5581289526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.780+0000 7f44e2ffd640 1 --2- 172.21.15.28:0/3247690828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55812894ea80 0x5581289526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.780+0000 7f44e2ffd640 1 -- 172.21.15.28:0/3247690828 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558128952c80 con 0x558128945ea0 2024-04-21T08:56:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.780+0000 7f44e2ffd640 1 --2- 172.21.15.28:0/3247690828 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558128945ea0 0x558128952180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f44dc09c800 tx=0x7f44dc0a0eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8268d52cd765b5ac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.780+0000 7f44c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:56:32.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.781+0000 7f44c2ffd640 1 -- 172.21.15.28:0/3247690828 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44dc004d60 con 0x558128945ea0 2024-04-21T08:56:32.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.781+0000 7f44c2ffd640 1 -- 172.21.15.28:0/3247690828 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f44dc0bb900 con 0x558128945ea0 2024-04-21T08:56:32.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.781+0000 7f44ecc75740 1 -- 172.21.15.28:0/3247690828 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44d8017050 con 0x558128945ea0 2024-04-21T08:56:32.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.781+0000 7f44c2ffd640 1 -- 172.21.15.28:0/3247690828 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f44dc0aa050 con 0x558128945ea0 2024-04-21T08:56:32.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.781+0000 7f44ecc75740 10 client.26503 fetch_fsmap learned FSMap version 28 2024-04-21T08:56:32.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.781+0000 7f44ecc75740 1 -- 172.21.15.28:0/3247690828 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5581289c4ea0 con 0x558128945ea0 2024-04-21T08:56:32.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.782+0000 7f44c2ffd640 1 -- 172.21.15.28:0/3247690828 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44d8017050 con 0x558128945ea0 2024-04-21T08:56:32.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.782+0000 7f44c2ffd640 1 -- 172.21.15.28:0/3247690828 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f44dc020020 con 0x558128945ea0 2024-04-21T08:56:32.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.783+0000 7f44ecc75740 10 client.26503 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:56:32.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.783+0000 7f44ecc75740 10 client.26503 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:32.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.783+0000 7f44ecc75740 20 client.26503 populate_metadata read hostname 'smithi028' 2024-04-21T08:56:32.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.783+0000 7f44ecc75740 1 --2- 172.21.15.28:0/3247690828 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5581289c5620 0x5581289c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:32.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.783+0000 7f44ecc75740 4 client.26503 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:56:32.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.783+0000 7f44ecc75740 1 -- 172.21.15.28:0/3247690828 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5581288b1a70 con 0x5581289c5620 2024-04-21T08:56:32.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.783+0000 7f44e37fe640 1 --2- 172.21.15.28:0/3247690828 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5581289c5620 0x5581289c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:32.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.784+0000 7f44c2ffd640 10 client.26503 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:32.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.784+0000 7f44e37fe640 1 --2- 172.21.15.28:0/3247690828 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5581289c5620 0x5581289c7a40 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68d575ad88ed806e server_cookie=cd0fc56dd899bea1 in_seq=0 out_seq=0 2024-04-21T08:56:32.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.784+0000 7f44c2ffd640 1 -- 172.21.15.28:0/3247690828 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x5581288b1a70 con 0x5581289c5620 2024-04-21T08:56:32.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.784+0000 7f44c2ffd640 10 client.26503 handle_command_reply: tid=0 2024-04-21T08:56:32.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.784+0000 7f44ecc75740 1 client.26503 shutdown 2024-04-21T08:56:32.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.785+0000 7f44ecc75740 1 -- 172.21.15.28:0/3247690828 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558128945ea0 msgr2=0x558128952180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:32.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.785+0000 7f44ecc75740 1 --2- 172.21.15.28:0/3247690828 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558128945ea0 0x558128952180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f44dc09c800 tx=0x7f44dc0a0eb0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.786+0000 7f44ecc75740 1 -- 172.21.15.28:0/3247690828 shutdown_connections 2024-04-21T08:56:32.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.786+0000 7f44ecc75740 1 --2- 172.21.15.28:0/3247690828 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5581289c5620 0x5581289c7a40 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.786+0000 7f44ecc75740 1 --2- 172.21.15.28:0/3247690828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55812894ea80 0x5581289526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.786+0000 7f44ecc75740 1 --2- 172.21.15.28:0/3247690828 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558128945ea0 0x558128952180 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.786+0000 7f44ecc75740 1 --2- 172.21.15.28:0/3247690828 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558128945580 0x558128956dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.786+0000 7f44ecc75740 1 -- 172.21.15.28:0/3247690828 >> 172.21.15.28:0/3247690828 conn(0x55812893fcb0 msgr2=0x55812894c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:32.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.786+0000 7f44c2ffd640 0 client.26503 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:56:32.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.786+0000 7f44ecc75740 1 -- 172.21.15.28:0/3247690828 shutdown_connections 2024-04-21T08:56:32.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.786+0000 7f44ecc75740 1 -- 172.21.15.28:0/3247690828 wait complete. 2024-04-21T08:56:32.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.787+0000 7f44ecc75740 20 client.26503 trim_cache size 0 max 16384 2024-04-21T08:56:32.789 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:56:32.789 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-21T08:56:32.789 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:56:32.789 INFO:teuthology.orchestra.run.smithi028.stdout: "dd219396-9868-49d6-9248-7e294e4ebcd5": { 2024-04-21T08:56:32.789 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:56:32.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "dd219396-9868-49d6-9248-7e294e4ebcd5", 2024-04-21T08:56:32.790 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:56:32.790 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:56:32.790 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:56:32.790 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:56:32.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.788+0000 7f44ea081640 1 -- 172.21.15.28:0/2665178929 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f44cc0a3ea0 msgr2=0x7f44cc0a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:32.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.788+0000 7f44ea081640 1 --2- 172.21.15.28:0/2665178929 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f44cc0a3ea0 0x7f44cc0a6360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f44d8005fd0 tx=0x7f44d8005850 comp rx=0 tx=0).stop 2024-04-21T08:56:32.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.788+0000 7f44ea081640 1 -- 172.21.15.28:0/2665178929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44e4100940 msgr2=0x7f44e410c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:32.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.788+0000 7f44ea081640 1 --2- 172.21.15.28:0/2665178929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44e4100940 0x7f44e410c470 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f44dc00eb00 tx=0x7f44dc00c2d0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.789+0000 7f44ea081640 1 -- 172.21.15.28:0/2665178929 shutdown_connections 2024-04-21T08:56:32.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.789+0000 7f44ea081640 1 --2- 172.21.15.28:0/2665178929 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f44cc0a3ea0 0x7f44cc0a6360 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.789+0000 7f44ea081640 1 --2- 172.21.15.28:0/2665178929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e4112d40 0x7f44e410c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.789+0000 7f44ea081640 1 --2- 172.21.15.28:0/2665178929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f44e4100940 0x7f44e410c470 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.789+0000 7f44ea081640 1 --2- 172.21.15.28:0/2665178929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f44e4100020 0x7f44e41123b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:32.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.789+0000 7f44ea081640 1 -- 172.21.15.28:0/2665178929 >> 172.21.15.28:0/2665178929 conn(0x7f44e4076fa0 msgr2=0x7f44e4113fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:32.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.789+0000 7f44ea081640 1 -- 172.21.15.28:0/2665178929 shutdown_connections 2024-04-21T08:56:32.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:56:32.789+0000 7f44ea081640 1 -- 172.21.15.28:0/2665178929 wait complete. 2024-04-21T08:56:32.807 INFO:tasks.cephfs.filesystem:scrub status for tag:dd219396-9868-49d6-9248-7e294e4ebcd5 - {'path': '/', 'tag': 'dd219396-9868-49d6-9248-7e294e4ebcd5', 'options': 'recursive,force'} 2024-04-21T08:56:34.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:34 smithi152 ceph-mon[31401]: pgmap v164: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 6.20k op/s 2024-04-21T08:56:34.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:34 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:56:34.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:34 smithi028 ceph-mon[28003]: pgmap v164: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 6.20k op/s 2024-04-21T08:56:34.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:34 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:56:34.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:34 smithi171 ceph-mon[31358]: pgmap v164: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 6.20k op/s 2024-04-21T08:56:34.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:34 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:56:35.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:35 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:56:35.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:35 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:56:35.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:35 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:56:36.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:36 smithi152 ceph-mon[31401]: pgmap v165: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 29 MiB/s wr, 5.97k op/s 2024-04-21T08:56:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:36 smithi028 ceph-mon[28003]: pgmap v165: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 29 MiB/s wr, 5.97k op/s 2024-04-21T08:56:36.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:36 smithi171 ceph-mon[31358]: pgmap v165: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 29 MiB/s wr, 5.97k op/s 2024-04-21T08:56:38.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:38 smithi152 ceph-mon[31401]: pgmap v166: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 22 MiB/s wr, 4.72k op/s 2024-04-21T08:56:38.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:38 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:56:38.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:38 smithi028 ceph-mon[28003]: pgmap v166: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 22 MiB/s wr, 4.72k op/s 2024-04-21T08:56:38.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:38 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:56:38.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:38 smithi171 ceph-mon[31358]: pgmap v166: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 22 MiB/s wr, 4.72k op/s 2024-04-21T08:56:38.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:38 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:56:40.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:40 smithi152 ceph-mon[31401]: pgmap v167: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 19 MiB/s wr, 4.24k op/s 2024-04-21T08:56:40.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:40 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:56:40.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:40 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:56:40.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:40 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:56:40.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:40 smithi028 ceph-mon[28003]: pgmap v167: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 19 MiB/s wr, 4.24k op/s 2024-04-21T08:56:40.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:40 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:56:40.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:40 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:56:40.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:40 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:56:40.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:40 smithi171 ceph-mon[31358]: pgmap v167: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 19 MiB/s wr, 4.24k op/s 2024-04-21T08:56:40.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:40 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:56:40.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:40 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:56:40.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:40 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:56:42.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:42 smithi152 ceph-mon[31401]: pgmap v168: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 9.4 MiB/s wr, 2.15k op/s 2024-04-21T08:56:42.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:42 smithi028 ceph-mon[28003]: pgmap v168: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 9.4 MiB/s wr, 2.15k op/s 2024-04-21T08:56:42.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:42 smithi171 ceph-mon[31358]: pgmap v168: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 9.4 MiB/s wr, 2.15k op/s 2024-04-21T08:56:44.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:44 smithi152 ceph-mon[31401]: pgmap v169: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 7.8 MiB/s wr, 1.78k op/s 2024-04-21T08:56:44.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:44 smithi028 ceph-mon[28003]: pgmap v169: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 7.8 MiB/s wr, 1.78k op/s 2024-04-21T08:56:44.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:44 smithi171 ceph-mon[31358]: pgmap v169: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 7.8 MiB/s wr, 1.78k op/s 2024-04-21T08:56:46.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:46 smithi152 ceph-mon[31401]: pgmap v170: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 4.9 MiB/s wr, 1.11k op/s 2024-04-21T08:56:46.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:46 smithi028 ceph-mon[28003]: pgmap v170: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 4.9 MiB/s wr, 1.11k op/s 2024-04-21T08:56:46.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:46 smithi171 ceph-mon[31358]: pgmap v170: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 4.9 MiB/s wr, 1.11k op/s 2024-04-21T08:56:47.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:47 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:56:47.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:47 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:56:47.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:47 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:56:47.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:47 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:56:47.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:47 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:56:47.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:47 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:56:47.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:47 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:56:47.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:47 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:56:47.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:47 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:56:48.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:48 smithi152 ceph-mon[31401]: pgmap v171: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 2.2 MiB/s wr, 477 op/s 2024-04-21T08:56:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:48 smithi028 ceph-mon[28003]: pgmap v171: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 2.2 MiB/s wr, 477 op/s 2024-04-21T08:56:48.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:48 smithi171 ceph-mon[31358]: pgmap v171: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 2.2 MiB/s wr, 477 op/s 2024-04-21T08:56:50.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:50 smithi152 ceph-mon[31401]: pgmap v172: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 70 KiB/s wr, 14 op/s 2024-04-21T08:56:50.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:50 smithi028 ceph-mon[28003]: pgmap v172: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 70 KiB/s wr, 14 op/s 2024-04-21T08:56:50.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:50 smithi171 ceph-mon[31358]: pgmap v172: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 70 KiB/s wr, 14 op/s 2024-04-21T08:56:52.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:52 smithi152 ceph-mon[31401]: pgmap v173: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 6 op/s 2024-04-21T08:56:52.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:52 smithi028 ceph-mon[28003]: pgmap v173: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 6 op/s 2024-04-21T08:56:52.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:52 smithi171 ceph-mon[31358]: pgmap v173: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 KiB/s wr, 6 op/s 2024-04-21T08:56:54.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:54 smithi152 ceph-mon[31401]: pgmap v174: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 17 KiB/s wr, 8 op/s 2024-04-21T08:56:54.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:54 smithi028 ceph-mon[28003]: pgmap v174: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 17 KiB/s wr, 8 op/s 2024-04-21T08:56:54.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:54 smithi171 ceph-mon[31358]: pgmap v174: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 17 KiB/s wr, 8 op/s 2024-04-21T08:56:56.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:56 smithi152 ceph-mon[31401]: pgmap v175: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 6 op/s 2024-04-21T08:56:56.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:56 smithi028 ceph-mon[28003]: pgmap v175: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 6 op/s 2024-04-21T08:56:56.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:56 smithi171 ceph-mon[31358]: pgmap v175: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 6 op/s 2024-04-21T08:56:58.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:56:58 smithi152 ceph-mon[31401]: pgmap v176: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-21T08:56:58.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:56:58 smithi028 ceph-mon[28003]: pgmap v176: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-21T08:56:58.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:56:58 smithi171 ceph-mon[31358]: pgmap v176: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-21T08:57:00.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:00 smithi152 ceph-mon[31401]: pgmap v177: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-21T08:57:00.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:00 smithi028 ceph-mon[28003]: pgmap v177: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-21T08:57:00.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:00 smithi171 ceph-mon[31358]: pgmap v177: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-21T08:57:02.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:02 smithi152 ceph-mon[31401]: pgmap v178: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T08:57:02.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:02 smithi028 ceph-mon[28003]: pgmap v178: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T08:57:02.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:02 smithi171 ceph-mon[31358]: pgmap v178: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-21T08:57:02.807 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:57:02.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:57:02.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.926+0000 7fe26939f640 1 -- 172.21.15.28:0/1864748943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264103870 msgr2=0x7fe26410bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:02.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.926+0000 7fe26939f640 1 --2- 172.21.15.28:0/1864748943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264103870 0x7fe26410bd50 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe258009900 tx=0x7fe25802bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:02.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.926+0000 7fe26939f640 1 -- 172.21.15.28:0/1864748943 shutdown_connections 2024-04-21T08:57:02.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.926+0000 7fe26939f640 1 --2- 172.21.15.28:0/1864748943 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe26410c320 0x7fe26410e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.926+0000 7fe26939f640 1 --2- 172.21.15.28:0/1864748943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264103870 0x7fe26410bd50 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.926+0000 7fe26939f640 1 --2- 172.21.15.28:0/1864748943 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe264102f50 0x7fe264103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.926+0000 7fe26939f640 1 -- 172.21.15.28:0/1864748943 >> 172.21.15.28:0/1864748943 conn(0x7fe2640fc6f0 msgr2=0x7fe2640feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.926+0000 7fe26939f640 1 -- 172.21.15.28:0/1864748943 shutdown_connections 2024-04-21T08:57:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.926+0000 7fe26939f640 1 -- 172.21.15.28:0/1864748943 wait complete. 2024-04-21T08:57:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.927+0000 7fe26939f640 1 Processor -- start 2024-04-21T08:57:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.927+0000 7fe26939f640 1 -- start start 2024-04-21T08:57:02.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe26939f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe264102f50 0x7fe2641984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe26939f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe264103870 0x7fe2641989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe26939f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26410c320 0x7fe26419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe26939f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe264111fa0 con 0x7fe26410c320 2024-04-21T08:57:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe26939f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe264111e20 con 0x7fe264102f50 2024-04-21T08:57:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe26939f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe264112120 con 0x7fe264103870 2024-04-21T08:57:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe268b9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26410c320 0x7fe26419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe268b9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26410c320 0x7fe26419cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42530/0 (socket says 172.21.15.28:42530) 2024-04-21T08:57:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe263fff640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe264102f50 0x7fe2641984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe268b9e640 1 -- 172.21.15.28:0/1286612003 learned_addr learned my addr 172.21.15.28:0/1286612003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe2637fe640 1 --2- 172.21.15.28:0/1286612003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe264103870 0x7fe2641989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe268b9e640 1 -- 172.21.15.28:0/1286612003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe264103870 msgr2=0x7fe2641989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe268b9e640 1 --2- 172.21.15.28:0/1286612003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe264103870 0x7fe2641989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe268b9e640 1 -- 172.21.15.28:0/1286612003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe264102f50 msgr2=0x7fe2641984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe268b9e640 1 --2- 172.21.15.28:0/1286612003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe264102f50 0x7fe2641984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe268b9e640 1 -- 172.21.15.28:0/1286612003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26419d490 con 0x7fe26410c320 2024-04-21T08:57:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe268b9e640 1 --2- 172.21.15.28:0/1286612003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26410c320 0x7fe26419cd60 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00eaa0 tx=0x7fe25c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.928+0000 7fe2617fa640 1 -- 172.21.15.28:0/1286612003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c004d20 con 0x7fe26410c320 2024-04-21T08:57:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.929+0000 7fe26939f640 1 -- 172.21.15.28:0/1286612003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe26419d780 con 0x7fe26410c320 2024-04-21T08:57:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.929+0000 7fe2617fa640 1 -- 172.21.15.28:0/1286612003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe25c005020 con 0x7fe26410c320 2024-04-21T08:57:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.929+0000 7fe2617fa640 1 -- 172.21.15.28:0/1286612003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c0116b0 con 0x7fe26410c320 2024-04-21T08:57:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.929+0000 7fe26939f640 1 -- 172.21.15.28:0/1286612003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2641a5010 con 0x7fe26410c320 2024-04-21T08:57:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.931+0000 7fe2617fa640 1 -- 172.21.15.28:0/1286612003 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe25c0051d0 con 0x7fe26410c320 2024-04-21T08:57:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.931+0000 7fe26d1ea740 1 Processor -- start 2024-04-21T08:57:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe26d1ea740 1 -- start start 2024-04-21T08:57:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe26d1ea740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea0710 0x563f72ea0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe26d1ea740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea1030 0x563f72ea96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe26d1ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea9c10 0x563f72eac000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe26d1ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563f72e051e0 con 0x563f72ea9c10 2024-04-21T08:57:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe26d1ea740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563f72da2390 con 0x563f72ea1030 2024-04-21T08:57:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe26d1ea740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x563f72e011a0 con 0x563f72ea0710 2024-04-21T08:57:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe2617fa640 1 --2- 172.21.15.28:0/1286612003 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe24c0a3d20 0x7fe24c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe2637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea9c10 0x563f72eac000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe268b9e640 1 --2- 172.21.15.28:0/1286612003 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe24c0a3d20 0x7fe24c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe2637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea9c10 0x563f72eac000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42534/0 (socket says 172.21.15.28:42534) 2024-04-21T08:57:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe2637fe640 1 -- 172.21.15.28:0/1719100360 learned_addr learned my addr 172.21.15.28:0/1719100360 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe2637fe640 1 --2- 172.21.15.28:0/1719100360 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea0710 0x563f72ea0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe263fff640 1 --2- 172.21.15.28:0/1719100360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea1030 0x563f72ea96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe2617fa640 1 -- 172.21.15.28:0/1286612003 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe25c0d0ce0 con 0x7fe26410c320 2024-04-21T08:57:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe2637fe640 1 -- 172.21.15.28:0/1719100360 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea0710 msgr2=0x563f72ea0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe2637fe640 1 --2- 172.21.15.28:0/1719100360 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea0710 0x563f72ea0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe2637fe640 1 -- 172.21.15.28:0/1719100360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea1030 msgr2=0x563f72ea96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe2637fe640 1 --2- 172.21.15.28:0/1719100360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea1030 0x563f72ea96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe2637fe640 1 -- 172.21.15.28:0/1719100360 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f72be17d0 con 0x563f72ea9c10 2024-04-21T08:57:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe263fff640 1 --2- 172.21.15.28:0/1719100360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea1030 0x563f72ea96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:57:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.932+0000 7fe268b9e640 1 --2- 172.21.15.28:0/1286612003 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe24c0a3d20 0x7fe24c0a61e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe25c06f3a0 tx=0x7fe25c06f3d0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.933+0000 7fe2637fe640 1 --2- 172.21.15.28:0/1719100360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea9c10 0x563f72eac000 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe25802c780 tx=0x7fe258005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a4792c358cb25ae server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.933+0000 7fe24a7fc640 1 -- 172.21.15.28:0/1719100360 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25804a070 con 0x563f72ea9c10 2024-04-21T08:57:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.933+0000 7fe24a7fc640 1 -- 172.21.15.28:0/1719100360 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe258005050 con 0x563f72ea9c10 2024-04-21T08:57:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.933+0000 7fe24a7fc640 1 -- 172.21.15.28:0/1719100360 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258004920 con 0x563f72ea9c10 2024-04-21T08:57:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.933+0000 7fe26d1ea740 1 -- 172.21.15.28:0/1719100360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea9c10 msgr2=0x563f72eac000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.934+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/1719100360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea9c10 0x563f72eac000 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fe25802c780 tx=0x7fe258005fd0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.934+0000 7fe26d1ea740 1 -- 172.21.15.28:0/1719100360 shutdown_connections 2024-04-21T08:57:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.934+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/1719100360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea9c10 0x563f72eac000 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.934+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/1719100360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea1030 0x563f72ea96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.934+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/1719100360 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea0710 0x563f72ea0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.934+0000 7fe26d1ea740 1 -- 172.21.15.28:0/1719100360 >> 172.21.15.28:0/1719100360 conn(0x563f72e9ae40 msgr2=0x563f72e9b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.934+0000 7fe26d1ea740 1 -- 172.21.15.28:0/1719100360 shutdown_connections 2024-04-21T08:57:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.934+0000 7fe26d1ea740 1 -- 172.21.15.28:0/1719100360 wait complete. 2024-04-21T08:57:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.934+0000 7fe26d1ea740 1 -- start start 2024-04-21T08:57:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.934+0000 7fe26d1ea740 1 Processor -- start 2024-04-21T08:57:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe26d1ea740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea0710 0x563f72ea6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe26d1ea740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea1030 0x563f72ea7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe26d1ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea9c10 0x563f72ea7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe26d1ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563f72eac9a0 con 0x563f72ea9c10 2024-04-21T08:57:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe26d1ea740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563f72dbe2f0 con 0x563f72ea0710 2024-04-21T08:57:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe26d1ea740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x563f72eacb20 con 0x563f72ea1030 2024-04-21T08:57:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe26d1ea740 10 client.0 _ll_register_callbacks cb 0x563f72dcb580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe2637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea9c10 0x563f72ea7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe2637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea9c10 0x563f72ea7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42550/0 (socket says 172.21.15.28:42550) 2024-04-21T08:57:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe2637fe640 1 -- 172.21.15.28:0/1771508882 learned_addr learned my addr 172.21.15.28:0/1771508882 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe2637fe640 1 --2- 172.21.15.28:0/1771508882 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea0710 0x563f72ea6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe263fff640 1 --2- 172.21.15.28:0/1771508882 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea1030 0x563f72ea7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe2637fe640 1 -- 172.21.15.28:0/1771508882 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea1030 msgr2=0x563f72ea7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe2637fe640 1 --2- 172.21.15.28:0/1771508882 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea1030 0x563f72ea7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe2637fe640 1 -- 172.21.15.28:0/1771508882 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea0710 msgr2=0x563f72ea6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe2637fe640 1 --2- 172.21.15.28:0/1771508882 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea0710 0x563f72ea6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe2637fe640 1 -- 172.21.15.28:0/1771508882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f72b173f0 con 0x563f72ea9c10 2024-04-21T08:57:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe263fff640 1 --2- 172.21.15.28:0/1771508882 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea1030 0x563f72ea7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe2637fe640 1 --2- 172.21.15.28:0/1771508882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea9c10 0x563f72ea7760 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fe258001eb0 tx=0x7fe25802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4eb6701a1c4414a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe2437fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe2437fe640 1 -- 172.21.15.28:0/1771508882 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25804a040 con 0x563f72ea9c10 2024-04-21T08:57:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe2437fe640 1 -- 172.21.15.28:0/1771508882 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe2580048c0 con 0x563f72ea9c10 2024-04-21T08:57:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.935+0000 7fe26d1ea740 1 -- 172.21.15.28:0/1771508882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe258005050 con 0x563f72ea9c10 2024-04-21T08:57:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.936+0000 7fe2437fe640 1 -- 172.21.15.28:0/1771508882 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe25805a050 con 0x563f72ea9c10 2024-04-21T08:57:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.936+0000 7fe26d1ea740 10 client.17223 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.936+0000 7fe26d1ea740 1 -- 172.21.15.28:0/1771508882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563f72ad70f0 con 0x563f72ea9c10 2024-04-21T08:57:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.936+0000 7fe2437fe640 1 -- 172.21.15.28:0/1771508882 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe258005050 con 0x563f72ea9c10 2024-04-21T08:57:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.936+0000 7fe2437fe640 1 -- 172.21.15.28:0/1771508882 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe25804d050 con 0x563f72ea9c10 2024-04-21T08:57:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.936+0000 7fe26d1ea740 10 client.17223 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.936+0000 7fe26d1ea740 10 client.17223 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.936+0000 7fe26d1ea740 20 client.17223 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.936+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/1771508882 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563f72eb9b20 0x563f72ebbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.936+0000 7fe26d1ea740 4 client.17223 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:57:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.936+0000 7fe26d1ea740 1 -- 172.21.15.28:0/1771508882 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f72e12c00 con 0x563f72eb9b20 2024-04-21T08:57:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.936+0000 7fe263fff640 1 --2- 172.21.15.28:0/1771508882 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563f72eb9b20 0x563f72ebbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.937+0000 7fe263fff640 1 --2- 172.21.15.28:0/1771508882 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563f72eb9b20 0x563f72ebbf10 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dec0e68319d6d1e server_cookie=ec6a804ad6733b52 in_seq=0 out_seq=0 2024-04-21T08:57:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.937+0000 7fe2437fe640 10 client.17223 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.938+0000 7fe2437fe640 1 -- 172.21.15.28:0/1771508882 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f72e12c00 con 0x563f72eb9b20 2024-04-21T08:57:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.938+0000 7fe2437fe640 10 client.17223 handle_command_reply: tid=0 2024-04-21T08:57:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.938+0000 7fe26d1ea740 1 client.17223 shutdown 2024-04-21T08:57:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.938+0000 7fe26d1ea740 1 -- 172.21.15.28:0/1771508882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea9c10 msgr2=0x563f72ea7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.938+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/1771508882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea9c10 0x563f72ea7760 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fe258001eb0 tx=0x7fe25802bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.938+0000 7fe26d1ea740 1 -- 172.21.15.28:0/1771508882 shutdown_connections 2024-04-21T08:57:02.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.938+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/1771508882 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563f72eb9b20 0x563f72ebbf10 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.938+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/1771508882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea9c10 0x563f72ea7760 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.938+0000 7fe2437fe640 0 client.17223 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:02.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.938+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/1771508882 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea1030 0x563f72ea7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.938+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/1771508882 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea0710 0x563f72ea6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.938+0000 7fe26d1ea740 1 -- 172.21.15.28:0/1771508882 >> 172.21.15.28:0/1771508882 conn(0x563f72e9ae40 msgr2=0x563f72e9a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:02.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.938+0000 7fe26d1ea740 1 -- 172.21.15.28:0/1771508882 shutdown_connections 2024-04-21T08:57:02.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.938+0000 7fe26d1ea740 1 -- 172.21.15.28:0/1771508882 wait complete. 2024-04-21T08:57:02.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.938+0000 7fe26d1ea740 20 client.17223 trim_cache size 0 max 16384 2024-04-21T08:57:02.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.956+0000 7fe26d1ea740 1 Processor -- start 2024-04-21T08:57:02.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.956+0000 7fe26d1ea740 1 -- start start 2024-04-21T08:57:02.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.956+0000 7fe26d1ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea0710 0x563f72ea9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.956+0000 7fe26d1ea740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea1030 0x563f72e9b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.956+0000 7fe26d1ea740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea9c10 0x563f72ea5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.956+0000 7fe26d1ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2641b1a10 con 0x563f72ea0710 2024-04-21T08:57:02.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.956+0000 7fe26d1ea740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe2641af2e0 con 0x563f72ea9c10 2024-04-21T08:57:02.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.956+0000 7fe26d1ea740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe2641ad130 con 0x563f72ea1030 2024-04-21T08:57:02.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.957+0000 7fe2637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea0710 0x563f72ea9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.957+0000 7fe2637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea0710 0x563f72ea9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42552/0 (socket says 172.21.15.28:42552) 2024-04-21T08:57:02.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.957+0000 7fe2637fe640 1 -- 172.21.15.28:0/4244498403 learned_addr learned my addr 172.21.15.28:0/4244498403 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:02.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.957+0000 7fe2637fe640 1 -- 172.21.15.28:0/4244498403 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea1030 msgr2=0x563f72e9b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:02.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.957+0000 7fe2637fe640 1 --2- 172.21.15.28:0/4244498403 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea1030 0x563f72e9b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.957+0000 7fe2637fe640 1 -- 172.21.15.28:0/4244498403 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea9c10 msgr2=0x563f72ea5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:02.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.957+0000 7fe2637fe640 1 --2- 172.21.15.28:0/4244498403 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea9c10 0x563f72ea5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.957+0000 7fe2637fe640 1 -- 172.21.15.28:0/4244498403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f72ea0e60 con 0x563f72ea0710 2024-04-21T08:57:02.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.957+0000 7fe2637fe640 1 --2- 172.21.15.28:0/4244498403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea0710 0x563f72ea9410 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe258040460 tx=0x7fe258005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e07ebf73c15c6b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:02.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.957+0000 7fe249ffb640 1 -- 172.21.15.28:0/4244498403 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258057070 con 0x563f72ea0710 2024-04-21T08:57:02.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.957+0000 7fe249ffb640 1 -- 172.21.15.28:0/4244498403 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2580042e0 con 0x563f72ea0710 2024-04-21T08:57:02.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.957+0000 7fe249ffb640 1 -- 172.21.15.28:0/4244498403 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25804a070 con 0x563f72ea0710 2024-04-21T08:57:02.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.958+0000 7fe26d1ea740 1 -- 172.21.15.28:0/4244498403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea0710 msgr2=0x563f72ea9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:02.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.958+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/4244498403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea0710 0x563f72ea9410 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe258040460 tx=0x7fe258005e40 comp rx=0 tx=0).stop 2024-04-21T08:57:02.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.958+0000 7fe26d1ea740 1 -- 172.21.15.28:0/4244498403 shutdown_connections 2024-04-21T08:57:02.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.958+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/4244498403 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea9c10 0x563f72ea5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.958+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/4244498403 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea1030 0x563f72e9b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.958+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/4244498403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea0710 0x563f72ea9410 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.958+0000 7fe26d1ea740 1 -- 172.21.15.28:0/4244498403 >> 172.21.15.28:0/4244498403 conn(0x563f72e9ae40 msgr2=0x563f72e99890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:02.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.958+0000 7fe26d1ea740 1 -- 172.21.15.28:0/4244498403 shutdown_connections 2024-04-21T08:57:02.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.958+0000 7fe26d1ea740 1 -- 172.21.15.28:0/4244498403 wait complete. 2024-04-21T08:57:02.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.958+0000 7fe26d1ea740 1 -- start start 2024-04-21T08:57:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.958+0000 7fe26d1ea740 1 Processor -- start 2024-04-21T08:57:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe26d1ea740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea0710 0x563f72ebc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe26d1ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea1030 0x563f72ebce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe26d1ea740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea9c10 0x563f72ebd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe26d1ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2641ad7a0 con 0x563f72ea1030 2024-04-21T08:57:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe26d1ea740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe2641aec10 con 0x563f72ea9c10 2024-04-21T08:57:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe26d1ea740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe2641ade70 con 0x563f72ea0710 2024-04-21T08:57:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe26d1ea740 10 client.0 _ll_register_callbacks cb 0x7fe258048070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe263fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea1030 0x563f72ebce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe263fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea1030 0x563f72ebce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42560/0 (socket says 172.21.15.28:42560) 2024-04-21T08:57:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe263fff640 1 -- 172.21.15.28:0/666313522 learned_addr learned my addr 172.21.15.28:0/666313522 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:02.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe2637fe640 1 --2- 172.21.15.28:0/666313522 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea9c10 0x563f72ebd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe263fff640 1 -- 172.21.15.28:0/666313522 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea0710 msgr2=0x563f72ebc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe263fff640 1 --2- 172.21.15.28:0/666313522 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea0710 0x563f72ebc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe263fff640 1 -- 172.21.15.28:0/666313522 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea9c10 msgr2=0x563f72ebd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe263fff640 1 --2- 172.21.15.28:0/666313522 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea9c10 0x563f72ebd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe263fff640 1 -- 172.21.15.28:0/666313522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f72f1ed00 con 0x563f72ea1030 2024-04-21T08:57:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe2637fe640 1 --2- 172.21.15.28:0/666313522 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea9c10 0x563f72ebd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe263fff640 1 --2- 172.21.15.28:0/666313522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea1030 0x563f72ebce70 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x563f72eacb90 tx=0x7fe2540108c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac6c306d3430d45 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe2437fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe2437fe640 1 -- 172.21.15.28:0/666313522 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25401d740 con 0x563f72ea1030 2024-04-21T08:57:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe2437fe640 1 -- 172.21.15.28:0/666313522 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fe25401d8e0 con 0x563f72ea1030 2024-04-21T08:57:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.959+0000 7fe26d1ea740 1 -- 172.21.15.28:0/666313522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2580042e0 con 0x563f72ea1030 2024-04-21T08:57:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.960+0000 7fe2437fe640 1 -- 172.21.15.28:0/666313522 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe254005020 con 0x563f72ea1030 2024-04-21T08:57:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.960+0000 7fe26d1ea740 10 client.17229 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.960+0000 7fe26d1ea740 1 -- 172.21.15.28:0/666313522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563f72f20010 con 0x563f72ea1030 2024-04-21T08:57:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.960+0000 7fe2437fe640 1 -- 172.21.15.28:0/666313522 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2580042e0 con 0x563f72ea1030 2024-04-21T08:57:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.960+0000 7fe2437fe640 1 -- 172.21.15.28:0/666313522 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe25400f4e0 con 0x563f72ea1030 2024-04-21T08:57:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.960+0000 7fe26d1ea740 10 client.17229 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.960+0000 7fe26d1ea740 10 client.17229 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.960+0000 7fe26d1ea740 20 client.17229 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.960+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/666313522 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563f72f20810 0x563f72f22c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.960+0000 7fe26d1ea740 4 client.17229 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:57:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.960+0000 7fe26d1ea740 1 -- 172.21.15.28:0/666313522 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563f72e0ca70 con 0x563f72f20810 2024-04-21T08:57:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.960+0000 7fe2637fe640 1 --2- 172.21.15.28:0/666313522 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563f72f20810 0x563f72f22c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe2637fe640 1 --2- 172.21.15.28:0/666313522 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563f72f20810 0x563f72f22c30 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74c5f5f00e14a766 server_cookie=90ab79f910fe0cea in_seq=0 out_seq=0 2024-04-21T08:57:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe2437fe640 10 client.17229 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe2437fe640 1 -- 172.21.15.28:0/666313522 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563f72e0ca70 con 0x563f72f20810 2024-04-21T08:57:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe2437fe640 10 client.17229 handle_command_reply: tid=0 2024-04-21T08:57:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe26d1ea740 1 client.17229 shutdown 2024-04-21T08:57:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe26d1ea740 1 -- 172.21.15.28:0/666313522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea1030 msgr2=0x563f72ebce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/666313522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea1030 0x563f72ebce70 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x563f72eacb90 tx=0x7fe2540108c0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe26d1ea740 1 -- 172.21.15.28:0/666313522 shutdown_connections 2024-04-21T08:57:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/666313522 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x563f72f20810 0x563f72f22c30 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/666313522 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f72ea9c10 0x563f72ebd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/666313522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f72ea1030 0x563f72ebce70 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe26d1ea740 1 --2- 172.21.15.28:0/666313522 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563f72ea0710 0x563f72ebc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe26d1ea740 1 -- 172.21.15.28:0/666313522 >> 172.21.15.28:0/666313522 conn(0x563f72e9ae40 msgr2=0x563f72e9a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe2437fe640 0 client.17229 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe26d1ea740 1 -- 172.21.15.28:0/666313522 shutdown_connections 2024-04-21T08:57:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe26d1ea740 1 -- 172.21.15.28:0/666313522 wait complete. 2024-04-21T08:57:02.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.961+0000 7fe26d1ea740 20 client.17229 trim_cache size 0 max 16384 2024-04-21T08:57:02.972 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:57:02.972 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T08:57:02.972 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T08:57:02.972 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:57:02.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.962+0000 7fe26939f640 1 -- 172.21.15.28:0/1286612003 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe24c0a3d20 msgr2=0x7fe24c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:02.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.962+0000 7fe26939f640 1 --2- 172.21.15.28:0/1286612003 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe24c0a3d20 0x7fe24c0a61e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe25c06f3a0 tx=0x7fe25c06f3d0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.962+0000 7fe26939f640 1 -- 172.21.15.28:0/1286612003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26410c320 msgr2=0x7fe26419cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:02.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.962+0000 7fe26939f640 1 --2- 172.21.15.28:0/1286612003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26410c320 0x7fe26419cd60 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00eaa0 tx=0x7fe25c00eea0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.962+0000 7fe26939f640 1 -- 172.21.15.28:0/1286612003 shutdown_connections 2024-04-21T08:57:02.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.962+0000 7fe26939f640 1 --2- 172.21.15.28:0/1286612003 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe24c0a3d20 0x7fe24c0a61e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.962+0000 7fe26939f640 1 --2- 172.21.15.28:0/1286612003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe26410c320 0x7fe26419cd60 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.962+0000 7fe26939f640 1 --2- 172.21.15.28:0/1286612003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe264103870 0x7fe2641989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.962+0000 7fe26939f640 1 --2- 172.21.15.28:0/1286612003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe264102f50 0x7fe2641984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.962+0000 7fe26939f640 1 -- 172.21.15.28:0/1286612003 >> 172.21.15.28:0/1286612003 conn(0x7fe2640fc6f0 msgr2=0x7fe2640fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.962+0000 7fe26939f640 1 -- 172.21.15.28:0/1286612003 shutdown_connections 2024-04-21T08:57:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:02.962+0000 7fe26939f640 1 -- 172.21.15.28:0/1286612003 wait complete. 2024-04-21T08:57:02.975 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:57:02.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:57:03.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.078+0000 7f5df8d50640 1 -- 172.21.15.28:0/2440547298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df4100d50 msgr2=0x7f5df4101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.078+0000 7f5df8d50640 1 --2- 172.21.15.28:0/2440547298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df4100d50 0x7f5df4101130 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f5de4009900 tx=0x7f5de402bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.078+0000 7f5df8d50640 1 -- 172.21.15.28:0/2440547298 shutdown_connections 2024-04-21T08:57:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.078+0000 7f5df8d50640 1 --2- 172.21.15.28:0/2440547298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5df410e900 0x7f5df4110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.078+0000 7f5df8d50640 1 --2- 172.21.15.28:0/2440547298 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5df4101670 0x7f5df410e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.078+0000 7f5df8d50640 1 --2- 172.21.15.28:0/2440547298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df4100d50 0x7f5df4101130 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.078+0000 7f5df8d50640 1 -- 172.21.15.28:0/2440547298 >> 172.21.15.28:0/2440547298 conn(0x7f5df40771d0 msgr2=0x7f5df40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.078+0000 7f5df8d50640 1 -- 172.21.15.28:0/2440547298 shutdown_connections 2024-04-21T08:57:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.078+0000 7f5df8d50640 1 -- 172.21.15.28:0/2440547298 wait complete. 2024-04-21T08:57:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.079+0000 7f5df8d50640 1 Processor -- start 2024-04-21T08:57:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.079+0000 7f5df8d50640 1 -- start start 2024-04-21T08:57:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.079+0000 7f5df8d50640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5df4100d50 0x7f5df41a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.079+0000 7f5df8d50640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5df4101670 0x7f5df41a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.079+0000 7f5df8d50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df410e900 0x7f5df419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.079+0000 7f5df8d50640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5df4114470 con 0x7f5df410e900 2024-04-21T08:57:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.079+0000 7f5df8d50640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5df41142f0 con 0x7f5df4101670 2024-04-21T08:57:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.079+0000 7f5df8d50640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5df41145f0 con 0x7f5df4100d50 2024-04-21T08:57:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.079+0000 7f5df2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df410e900 0x7f5df419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.079+0000 7f5df2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df410e900 0x7f5df419a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42584/0 (socket says 172.21.15.28:42584) 2024-04-21T08:57:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.079+0000 7f5df2d9d640 1 -- 172.21.15.28:0/318408519 learned_addr learned my addr 172.21.15.28:0/318408519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.079+0000 7f5df259c640 1 --2- 172.21.15.28:0/318408519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5df4100d50 0x7f5df41a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.079+0000 7f5df1d9b640 1 --2- 172.21.15.28:0/318408519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5df4101670 0x7f5df41a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.079+0000 7f5df2d9d640 1 -- 172.21.15.28:0/318408519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5df4100d50 msgr2=0x7f5df41a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.079+0000 7f5df2d9d640 1 --2- 172.21.15.28:0/318408519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5df4100d50 0x7f5df41a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.080+0000 7f5df2d9d640 1 -- 172.21.15.28:0/318408519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5df4101670 msgr2=0x7f5df41a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.080+0000 7f5df2d9d640 1 --2- 172.21.15.28:0/318408519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5df4101670 0x7f5df41a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.080+0000 7f5df2d9d640 1 -- 172.21.15.28:0/318408519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5df419afd0 con 0x7f5df410e900 2024-04-21T08:57:03.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.080+0000 7f5df1d9b640 1 --2- 172.21.15.28:0/318408519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5df4101670 0x7f5df41a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:03.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.080+0000 7f5df2d9d640 1 --2- 172.21.15.28:0/318408519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df410e900 0x7f5df419a710 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f5dec009fd0 tx=0x7f5dec00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.080+0000 7f5de37fe640 1 -- 172.21.15.28:0/318408519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec004d20 con 0x7f5df410e900 2024-04-21T08:57:03.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.080+0000 7f5de37fe640 1 -- 172.21.15.28:0/318408519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5dec005020 con 0x7f5df410e900 2024-04-21T08:57:03.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.080+0000 7f5de37fe640 1 -- 172.21.15.28:0/318408519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec0116b0 con 0x7f5df410e900 2024-04-21T08:57:03.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.080+0000 7f5df8d50640 1 -- 172.21.15.28:0/318408519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5df419b2c0 con 0x7f5df410e900 2024-04-21T08:57:03.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.080+0000 7f5df8d50640 1 -- 172.21.15.28:0/318408519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5df41a75d0 con 0x7f5df410e900 2024-04-21T08:57:03.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.081+0000 7f5df8d50640 1 -- 172.21.15.28:0/318408519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5df406ae60 con 0x7f5df410e900 2024-04-21T08:57:03.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.082+0000 7f5de37fe640 1 -- 172.21.15.28:0/318408519 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5dec0051d0 con 0x7f5df410e900 2024-04-21T08:57:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.083+0000 7f5de37fe640 1 --2- 172.21.15.28:0/318408519 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5ddc0a3d20 0x7f5ddc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.083+0000 7f5de37fe640 1 -- 172.21.15.28:0/318408519 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f5dec0d1110 con 0x7f5df410e900 2024-04-21T08:57:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.083+0000 7f5df259c640 1 --2- 172.21.15.28:0/318408519 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5ddc0a3d20 0x7f5ddc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.083+0000 7f5df259c640 1 --2- 172.21.15.28:0/318408519 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5ddc0a3d20 0x7f5ddc0a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5de4006010 tx=0x7f5de4000bc0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.087+0000 7f5de37fe640 1 -- 172.21.15.28:0/318408519 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5dec016030 con 0x7f5df410e900 2024-04-21T08:57:03.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.314+0000 7f5df8d50640 1 -- 172.21.15.28:0/318408519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5df4069230 con 0x7f5df410e900 2024-04-21T08:57:03.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.314+0000 7f5de37fe640 1 -- 172.21.15.28:0/318408519 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7f5dec09d410 con 0x7f5df410e900 2024-04-21T08:57:03.315 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:57:03.315 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:57:03.315 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T08:57:03.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.316+0000 7f5df8d50640 1 -- 172.21.15.28:0/318408519 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5ddc0a3d20 msgr2=0x7f5ddc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.316+0000 7f5df8d50640 1 --2- 172.21.15.28:0/318408519 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5ddc0a3d20 0x7f5ddc0a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5de4006010 tx=0x7f5de4000bc0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.316+0000 7f5df8d50640 1 -- 172.21.15.28:0/318408519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df410e900 msgr2=0x7f5df419a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.316+0000 7f5df8d50640 1 --2- 172.21.15.28:0/318408519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df410e900 0x7f5df419a710 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f5dec009fd0 tx=0x7f5dec00ee60 comp rx=0 tx=0).stop 2024-04-21T08:57:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.316+0000 7f5df8d50640 1 -- 172.21.15.28:0/318408519 shutdown_connections 2024-04-21T08:57:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.316+0000 7f5df8d50640 1 --2- 172.21.15.28:0/318408519 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5ddc0a3d20 0x7f5ddc0a61e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.316+0000 7f5df8d50640 1 --2- 172.21.15.28:0/318408519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5df410e900 0x7f5df419a710 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.316+0000 7f5df8d50640 1 --2- 172.21.15.28:0/318408519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5df4101670 0x7f5df41a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.316+0000 7f5df8d50640 1 --2- 172.21.15.28:0/318408519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5df4100d50 0x7f5df41a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.316+0000 7f5df8d50640 1 -- 172.21.15.28:0/318408519 >> 172.21.15.28:0/318408519 conn(0x7f5df40771d0 msgr2=0x7f5df410fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.317+0000 7f5df8d50640 1 -- 172.21.15.28:0/318408519 shutdown_connections 2024-04-21T08:57:03.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.317+0000 7f5df8d50640 1 -- 172.21.15.28:0/318408519 wait complete. 2024-04-21T08:57:03.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:57:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.433+0000 7f543a59d640 1 -- 172.21.15.28:0/1200175100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5434103af0 msgr2=0x7f543410bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.433+0000 7f543a59d640 1 --2- 172.21.15.28:0/1200175100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5434103af0 0x7f543410bfd0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f5424009900 tx=0x7f542402bd60 comp rx=0 tx=0).stop 2024-04-21T08:57:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.434+0000 7f543a59d640 1 -- 172.21.15.28:0/1200175100 shutdown_connections 2024-04-21T08:57:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.434+0000 7f543a59d640 1 --2- 172.21.15.28:0/1200175100 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f543410c510 0x7f543410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.434+0000 7f543a59d640 1 --2- 172.21.15.28:0/1200175100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5434103af0 0x7f543410bfd0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.434+0000 7f543a59d640 1 --2- 172.21.15.28:0/1200175100 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54341031d0 0x7f54341035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.434+0000 7f543a59d640 1 -- 172.21.15.28:0/1200175100 >> 172.21.15.28:0/1200175100 conn(0x7f5434077f50 msgr2=0x7f5434078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.434+0000 7f543a59d640 1 -- 172.21.15.28:0/1200175100 shutdown_connections 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.434+0000 7f543a59d640 1 -- 172.21.15.28:0/1200175100 wait complete. 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.434+0000 7f543a59d640 1 Processor -- start 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.434+0000 7f543a59d640 1 -- start start 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f543a59d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54341031d0 0x7f543419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f543a59d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5434103af0 0x7f543419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f543a59d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f543410c510 0x7f54341985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f543a59d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54341121a0 con 0x7f5434103af0 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f543a59d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5434112020 con 0x7f543410c510 2024-04-21T08:57:03.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f543a59d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5434112320 con 0x7f54341031d0 2024-04-21T08:57:03.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f54337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5434103af0 0x7f543419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f54337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5434103af0 0x7f543419e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42600/0 (socket says 172.21.15.28:42600) 2024-04-21T08:57:03.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f54337fe640 1 -- 172.21.15.28:0/3467639726 learned_addr learned my addr 172.21.15.28:0/3467639726 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f5438b3a640 1 --2- 172.21.15.28:0/3467639726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f543410c510 0x7f54341985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f5433fff640 1 --2- 172.21.15.28:0/3467639726 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54341031d0 0x7f543419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f54337fe640 1 -- 172.21.15.28:0/3467639726 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54341031d0 msgr2=0x7f543419e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f54337fe640 1 --2- 172.21.15.28:0/3467639726 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54341031d0 0x7f543419e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f54337fe640 1 -- 172.21.15.28:0/3467639726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f543410c510 msgr2=0x7f54341985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f54337fe640 1 --2- 172.21.15.28:0/3467639726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f543410c510 0x7f54341985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f54337fe640 1 -- 172.21.15.28:0/3467639726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5434198d30 con 0x7f5434103af0 2024-04-21T08:57:03.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f54337fe640 1 --2- 172.21.15.28:0/3467639726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5434103af0 0x7f543419e940 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5424005940 tx=0x7f542402d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f54317fa640 1 -- 172.21.15.28:0/3467639726 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424043020 con 0x7f5434103af0 2024-04-21T08:57:03.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f543a59d640 1 -- 172.21.15.28:0/3467639726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5434198fc0 con 0x7f5434103af0 2024-04-21T08:57:03.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f54317fa640 1 -- 172.21.15.28:0/3467639726 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f542402dd20 con 0x7f5434103af0 2024-04-21T08:57:03.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f54317fa640 1 -- 172.21.15.28:0/3467639726 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424003b80 con 0x7f5434103af0 2024-04-21T08:57:03.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.435+0000 7f543a59d640 1 -- 172.21.15.28:0/3467639726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54341a52c0 con 0x7f5434103af0 2024-04-21T08:57:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f54317fa640 1 -- 172.21.15.28:0/3467639726 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5424046020 con 0x7f5434103af0 2024-04-21T08:57:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f543d18d740 1 Processor -- start 2024-04-21T08:57:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f543d18d740 1 -- start start 2024-04-21T08:57:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f543d18d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b35c4d0 0x55d65b35c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f543d18d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35ce80 0x55d65b365430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f543d18d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b365a00 0x55d65b367e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f543d18d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d65b2c11f0 con 0x55d65b365a00 2024-04-21T08:57:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f543d18d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55d65b25e4b0 con 0x55d65b35ce80 2024-04-21T08:57:03.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f543d18d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55d65b2bd1b0 con 0x55d65b35c4d0 2024-04-21T08:57:03.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f54337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b365a00 0x55d65b367e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f54337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b365a00 0x55d65b367e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42608/0 (socket says 172.21.15.28:42608) 2024-04-21T08:57:03.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f54337fe640 1 -- 172.21.15.28:0/3923628001 learned_addr learned my addr 172.21.15.28:0/3923628001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f54317fa640 1 --2- 172.21.15.28:0/3467639726 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f541c0a3d20 0x7f541c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f5433fff640 1 --2- 172.21.15.28:0/3923628001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35ce80 0x55d65b365430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f54337fe640 1 -- 172.21.15.28:0/3923628001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b35c4d0 msgr2=0x55d65b35c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f5438b3a640 1 --2- 172.21.15.28:0/3923628001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b35c4d0 0x55d65b35c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f54337fe640 1 --2- 172.21.15.28:0/3923628001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b35c4d0 0x55d65b35c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f54337fe640 1 -- 172.21.15.28:0/3923628001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35ce80 msgr2=0x55d65b365430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f54337fe640 1 --2- 172.21.15.28:0/3923628001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35ce80 0x55d65b365430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f54337fe640 1 -- 172.21.15.28:0/3923628001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d65b2501b0 con 0x55d65b365a00 2024-04-21T08:57:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f5433fff640 1 --2- 172.21.15.28:0/3923628001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35ce80 0x55d65b365430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.438+0000 7f5438b3a640 1 --2- 172.21.15.28:0/3923628001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b35c4d0 0x55d65b35c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.439+0000 7f5438b3a640 1 --2- 172.21.15.28:0/3467639726 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f541c0a3d20 0x7f541c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.439+0000 7f54317fa640 1 -- 172.21.15.28:0/3467639726 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f54240f2cb0 con 0x7f5434103af0 2024-04-21T08:57:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.439+0000 7f54337fe640 1 --2- 172.21.15.28:0/3923628001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b365a00 0x55d65b367e50 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f54240f00e0 tx=0x7f542407e580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd49b223ed8443f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.439+0000 7f541a7fc640 1 -- 172.21.15.28:0/3923628001 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54240be790 con 0x55d65b365a00 2024-04-21T08:57:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.439+0000 7f541a7fc640 1 -- 172.21.15.28:0/3923628001 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54240bf0c0 con 0x55d65b365a00 2024-04-21T08:57:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.439+0000 7f541a7fc640 1 -- 172.21.15.28:0/3923628001 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54240be320 con 0x55d65b365a00 2024-04-21T08:57:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 -- 172.21.15.28:0/3923628001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b365a00 msgr2=0x55d65b367e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 --2- 172.21.15.28:0/3923628001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b365a00 0x55d65b367e50 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f54240f00e0 tx=0x7f542407e580 comp rx=0 tx=0).stop 2024-04-21T08:57:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f5438b3a640 1 --2- 172.21.15.28:0/3467639726 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f541c0a3d20 0x7f541c0a61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f542c005fd0 tx=0x7f542c004a40 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 -- 172.21.15.28:0/3923628001 shutdown_connections 2024-04-21T08:57:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 --2- 172.21.15.28:0/3923628001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b365a00 0x55d65b367e50 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 --2- 172.21.15.28:0/3923628001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35ce80 0x55d65b365430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 --2- 172.21.15.28:0/3923628001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b35c4d0 0x55d65b35c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 -- 172.21.15.28:0/3923628001 >> 172.21.15.28:0/3923628001 conn(0x55d65b356ba0 msgr2=0x55d65b356f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 -- 172.21.15.28:0/3923628001 shutdown_connections 2024-04-21T08:57:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 -- 172.21.15.28:0/3923628001 wait complete. 2024-04-21T08:57:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 -- start start 2024-04-21T08:57:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 Processor -- start 2024-04-21T08:57:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35c4d0 0x55d65b365240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35ce80 0x55d65b3625d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b365a00 0x55d65b362b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d65b368730 con 0x55d65b35ce80 2024-04-21T08:57:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55d65b27a500 con 0x55d65b35c4d0 2024-04-21T08:57:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55d65b3688b0 con 0x55d65b365a00 2024-04-21T08:57:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.440+0000 7f543d18d740 10 client.0 _ll_register_callbacks cb 0x55d65b248b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f54337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35ce80 0x55d65b3625d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f54337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35ce80 0x55d65b3625d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42622/0 (socket says 172.21.15.28:42622) 2024-04-21T08:57:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f54337fe640 1 -- 172.21.15.28:0/2139819872 learned_addr learned my addr 172.21.15.28:0/2139819872 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f5433fff640 1 --2- 172.21.15.28:0/2139819872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35c4d0 0x55d65b365240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f54337fe640 1 -- 172.21.15.28:0/2139819872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b365a00 msgr2=0x55d65b362b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f54337fe640 1 --2- 172.21.15.28:0/2139819872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b365a00 0x55d65b362b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f54337fe640 1 -- 172.21.15.28:0/2139819872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35c4d0 msgr2=0x55d65b365240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f54337fe640 1 --2- 172.21.15.28:0/2139819872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35c4d0 0x55d65b365240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f54337fe640 1 -- 172.21.15.28:0/2139819872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d65afd33f0 con 0x55d65b35ce80 2024-04-21T08:57:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f54337fe640 1 --2- 172.21.15.28:0/2139819872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35ce80 0x55d65b3625d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f54240f0660 tx=0x7f54240c2de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd045bb55e3438a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f54137fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f54137fe640 1 -- 172.21.15.28:0/2139819872 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54240be500 con 0x55d65b35ce80 2024-04-21T08:57:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f54137fe640 1 -- 172.21.15.28:0/2139819872 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f5424003d20 con 0x55d65b35ce80 2024-04-21T08:57:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f543d18d740 1 -- 172.21.15.28:0/2139819872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54240bf0c0 con 0x55d65b35ce80 2024-04-21T08:57:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f54137fe640 1 -- 172.21.15.28:0/2139819872 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f54240be320 con 0x55d65b35ce80 2024-04-21T08:57:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f543d18d740 10 client.17247 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f543d18d740 1 -- 172.21.15.28:0/2139819872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d65b09d7d0 con 0x55d65b35ce80 2024-04-21T08:57:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.441+0000 7f54137fe640 1 -- 172.21.15.28:0/2139819872 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54240bf0c0 con 0x55d65b35ce80 2024-04-21T08:57:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.442+0000 7f54137fe640 1 -- 172.21.15.28:0/2139819872 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f54240bb030 con 0x55d65b35ce80 2024-04-21T08:57:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.442+0000 7f543d18d740 10 client.17247 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.442+0000 7f543d18d740 10 client.17247 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:57:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.442+0000 7f543d18d740 20 client.17247 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.442+0000 7f543d18d740 1 --2- 172.21.15.28:0/2139819872 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55d65b375670 0x55d65b377a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.442+0000 7f543d18d740 4 client.17247 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:57:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.442+0000 7f543d18d740 1 -- 172.21.15.28:0/2139819872 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d65b260930 con 0x55d65b375670 2024-04-21T08:57:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.442+0000 7f5433fff640 1 --2- 172.21.15.28:0/2139819872 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55d65b375670 0x55d65b377a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.443+0000 7f54137fe640 10 client.17247 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:57:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.443+0000 7f5433fff640 1 --2- 172.21.15.28:0/2139819872 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55d65b375670 0x55d65b377a80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c716fecc510b67ac server_cookie=7e1953e3694173da in_seq=0 out_seq=0 2024-04-21T08:57:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.443+0000 7f54137fe640 1 -- 172.21.15.28:0/2139819872 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d65b260930 con 0x55d65b375670 2024-04-21T08:57:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.443+0000 7f54137fe640 10 client.17247 handle_command_reply: tid=0 2024-04-21T08:57:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.443+0000 7f543d18d740 1 client.17247 shutdown 2024-04-21T08:57:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.444+0000 7f543d18d740 1 -- 172.21.15.28:0/2139819872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35ce80 msgr2=0x55d65b3625d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.444+0000 7f543d18d740 1 --2- 172.21.15.28:0/2139819872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35ce80 0x55d65b3625d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f54240f0660 tx=0x7f54240c2de0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.444+0000 7f543d18d740 1 -- 172.21.15.28:0/2139819872 shutdown_connections 2024-04-21T08:57:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.444+0000 7f543d18d740 1 --2- 172.21.15.28:0/2139819872 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55d65b375670 0x55d65b377a80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.444+0000 7f543d18d740 1 --2- 172.21.15.28:0/2139819872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b365a00 0x55d65b362b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.444+0000 7f543d18d740 1 --2- 172.21.15.28:0/2139819872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35ce80 0x55d65b3625d0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.444+0000 7f54137fe640 0 client.17247 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:57:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.444+0000 7f543d18d740 1 --2- 172.21.15.28:0/2139819872 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35c4d0 0x55d65b365240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.444+0000 7f543d18d740 1 -- 172.21.15.28:0/2139819872 >> 172.21.15.28:0/2139819872 conn(0x55d65b356ba0 msgr2=0x55d65b3588f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.444+0000 7f543d18d740 1 -- 172.21.15.28:0/2139819872 shutdown_connections 2024-04-21T08:57:03.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.444+0000 7f543d18d740 1 -- 172.21.15.28:0/2139819872 wait complete. 2024-04-21T08:57:03.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.444+0000 7f543d18d740 20 client.17247 trim_cache size 0 max 16384 2024-04-21T08:57:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.457+0000 7f543d18d740 1 Processor -- start 2024-04-21T08:57:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.457+0000 7f543d18d740 1 -- start start 2024-04-21T08:57:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.457+0000 7f543d18d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35c4d0 0x55d65b371560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.457+0000 7f543d18d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35ce80 0x55d65b361730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.457+0000 7f543d18d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b365a00 0x55d65b3586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.457+0000 7f543d18d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54341b1b50 con 0x55d65b35c4d0 2024-04-21T08:57:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.457+0000 7f543d18d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f54341af420 con 0x55d65b35ce80 2024-04-21T08:57:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.457+0000 7f543d18d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f54341ad270 con 0x55d65b365a00 2024-04-21T08:57:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.457+0000 7f5433fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35c4d0 0x55d65b371560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.457+0000 7f5433fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35c4d0 0x55d65b371560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42628/0 (socket says 172.21.15.28:42628) 2024-04-21T08:57:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.457+0000 7f5433fff640 1 -- 172.21.15.28:0/2670224653 learned_addr learned my addr 172.21.15.28:0/2670224653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.458+0000 7f5438b3a640 1 --2- 172.21.15.28:0/2670224653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b365a00 0x55d65b3586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.458+0000 7f54337fe640 1 --2- 172.21.15.28:0/2670224653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35ce80 0x55d65b361730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.458+0000 7f5433fff640 1 -- 172.21.15.28:0/2670224653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b365a00 msgr2=0x55d65b3586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.458+0000 7f5433fff640 1 --2- 172.21.15.28:0/2670224653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b365a00 0x55d65b3586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.458+0000 7f5433fff640 1 -- 172.21.15.28:0/2670224653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35ce80 msgr2=0x55d65b361730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.458+0000 7f5433fff640 1 --2- 172.21.15.28:0/2670224653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35ce80 0x55d65b361730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.458+0000 7f5433fff640 1 -- 172.21.15.28:0/2670224653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d65b3569d0 con 0x55d65b35c4d0 2024-04-21T08:57:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.458+0000 7f5433fff640 1 --2- 172.21.15.28:0/2670224653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35c4d0 0x55d65b371560 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x55d65b35bfa0 tx=0x7f542800d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e2de3dcb500b27 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.458+0000 7f5419ffb640 1 -- 172.21.15.28:0/2670224653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5428013e10 con 0x55d65b35c4d0 2024-04-21T08:57:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.458+0000 7f5419ffb640 1 -- 172.21.15.28:0/2670224653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f542801f030 con 0x55d65b35c4d0 2024-04-21T08:57:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.458+0000 7f5419ffb640 1 -- 172.21.15.28:0/2670224653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5428005280 con 0x55d65b35c4d0 2024-04-21T08:57:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 -- 172.21.15.28:0/2670224653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35c4d0 msgr2=0x55d65b371560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 --2- 172.21.15.28:0/2670224653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35c4d0 0x55d65b371560 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x55d65b35bfa0 tx=0x7f542800d340 comp rx=0 tx=0).stop 2024-04-21T08:57:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 -- 172.21.15.28:0/2670224653 shutdown_connections 2024-04-21T08:57:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 --2- 172.21.15.28:0/2670224653 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b365a00 0x55d65b3586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 --2- 172.21.15.28:0/2670224653 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35ce80 0x55d65b361730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 --2- 172.21.15.28:0/2670224653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35c4d0 0x55d65b371560 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 -- 172.21.15.28:0/2670224653 >> 172.21.15.28:0/2670224653 conn(0x55d65b356ba0 msgr2=0x55d65b357fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 -- 172.21.15.28:0/2670224653 shutdown_connections 2024-04-21T08:57:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 -- 172.21.15.28:0/2670224653 wait complete. 2024-04-21T08:57:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 -- start start 2024-04-21T08:57:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 Processor -- start 2024-04-21T08:57:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35c4d0 0x55d65b36db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35ce80 0x55d65b369020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b365a00 0x55d65b369560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54341ad8e0 con 0x55d65b35ce80 2024-04-21T08:57:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f54341aed50 con 0x55d65b35c4d0 2024-04-21T08:57:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.459+0000 7f543d18d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f54341adfb0 con 0x55d65b365a00 2024-04-21T08:57:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f543d18d740 10 client.0 _ll_register_callbacks cb 0x7f542402ded0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f54337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35ce80 0x55d65b369020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f54337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35ce80 0x55d65b369020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42638/0 (socket says 172.21.15.28:42638) 2024-04-21T08:57:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f54337fe640 1 -- 172.21.15.28:0/4287171569 learned_addr learned my addr 172.21.15.28:0/4287171569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f5438b3a640 1 --2- 172.21.15.28:0/4287171569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b365a00 0x55d65b369560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f5433fff640 1 --2- 172.21.15.28:0/4287171569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35c4d0 0x55d65b36db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f54337fe640 1 -- 172.21.15.28:0/4287171569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b365a00 msgr2=0x55d65b369560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f54337fe640 1 --2- 172.21.15.28:0/4287171569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b365a00 0x55d65b369560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f54337fe640 1 -- 172.21.15.28:0/4287171569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35c4d0 msgr2=0x55d65b36db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f54337fe640 1 --2- 172.21.15.28:0/4287171569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35c4d0 0x55d65b36db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f54337fe640 1 -- 172.21.15.28:0/4287171569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d65b369aa0 con 0x55d65b35ce80 2024-04-21T08:57:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f5438b3a640 1 --2- 172.21.15.28:0/4287171569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b365a00 0x55d65b369560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:57:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f5433fff640 1 --2- 172.21.15.28:0/4287171569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35c4d0 0x55d65b36db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f54337fe640 1 --2- 172.21.15.28:0/4287171569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35ce80 0x55d65b369020 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f54240be6c0 tx=0x7f54240df0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34d68cef104fdf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f54137fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f54137fe640 1 -- 172.21.15.28:0/4287171569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54240be790 con 0x55d65b35ce80 2024-04-21T08:57:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f54137fe640 1 -- 172.21.15.28:0/4287171569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f54240bb030 con 0x55d65b35ce80 2024-04-21T08:57:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f543d18d740 1 -- 172.21.15.28:0/4287171569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f542801f030 con 0x55d65b35ce80 2024-04-21T08:57:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f54137fe640 1 -- 172.21.15.28:0/4287171569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f54240bf0c0 con 0x55d65b35ce80 2024-04-21T08:57:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f543d18d740 10 client.17253 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f543d18d740 1 -- 172.21.15.28:0/4287171569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d65b369aa0 con 0x55d65b35ce80 2024-04-21T08:57:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.460+0000 7f54137fe640 1 -- 172.21.15.28:0/4287171569 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f542801f030 con 0x55d65b35ce80 2024-04-21T08:57:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.461+0000 7f54137fe640 1 -- 172.21.15.28:0/4287171569 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f54240bec70 con 0x55d65b35ce80 2024-04-21T08:57:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.461+0000 7f543d18d740 10 client.17253 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.461+0000 7f543d18d740 10 client.17253 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:57:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.461+0000 7f543d18d740 20 client.17253 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.461+0000 7f543d18d740 1 --2- 172.21.15.28:0/4287171569 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55d65b3dbd80 0x55d65b3de1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.461+0000 7f543d18d740 4 client.17253 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:57:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.461+0000 7f543d18d740 1 -- 172.21.15.28:0/4287171569 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d65b2cec10 con 0x55d65b3dbd80 2024-04-21T08:57:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.461+0000 7f5433fff640 1 --2- 172.21.15.28:0/4287171569 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55d65b3dbd80 0x55d65b3de1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.461+0000 7f54137fe640 10 client.17253 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:57:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.461+0000 7f5433fff640 1 --2- 172.21.15.28:0/4287171569 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55d65b3dbd80 0x55d65b3de1a0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=904356eba8f40218 server_cookie=8897608cf26598 in_seq=0 out_seq=0 2024-04-21T08:57:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f54137fe640 1 -- 172.21.15.28:0/4287171569 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d65b2cec10 con 0x55d65b3dbd80 2024-04-21T08:57:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f54137fe640 10 client.17253 handle_command_reply: tid=0 2024-04-21T08:57:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543d18d740 1 client.17253 shutdown 2024-04-21T08:57:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543d18d740 1 -- 172.21.15.28:0/4287171569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35ce80 msgr2=0x55d65b369020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543d18d740 1 --2- 172.21.15.28:0/4287171569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35ce80 0x55d65b369020 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f54240be6c0 tx=0x7f54240df0b0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543d18d740 1 -- 172.21.15.28:0/4287171569 shutdown_connections 2024-04-21T08:57:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543d18d740 1 --2- 172.21.15.28:0/4287171569 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55d65b3dbd80 0x55d65b3de1a0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543d18d740 1 --2- 172.21.15.28:0/4287171569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d65b365a00 0x55d65b369560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543d18d740 1 --2- 172.21.15.28:0/4287171569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d65b35ce80 0x55d65b369020 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f54137fe640 0 client.17253 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:57:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543d18d740 1 --2- 172.21.15.28:0/4287171569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d65b35c4d0 0x55d65b36db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543d18d740 1 -- 172.21.15.28:0/4287171569 >> 172.21.15.28:0/4287171569 conn(0x55d65b356ba0 msgr2=0x55d65b358860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543d18d740 1 -- 172.21.15.28:0/4287171569 shutdown_connections 2024-04-21T08:57:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543d18d740 1 -- 172.21.15.28:0/4287171569 wait complete. 2024-04-21T08:57:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543d18d740 20 client.17253 trim_cache size 0 max 16384 2024-04-21T08:57:03.473 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:57:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543a59d640 1 -- 172.21.15.28:0/3467639726 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f541c0a3d20 msgr2=0x7f541c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543a59d640 1 --2- 172.21.15.28:0/3467639726 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f541c0a3d20 0x7f541c0a61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f542c005fd0 tx=0x7f542c004a40 comp rx=0 tx=0).stop 2024-04-21T08:57:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543a59d640 1 -- 172.21.15.28:0/3467639726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5434103af0 msgr2=0x7f543419e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.462+0000 7f543a59d640 1 --2- 172.21.15.28:0/3467639726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5434103af0 0x7f543419e940 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5424005940 tx=0x7f542402d920 comp rx=0 tx=0).stop 2024-04-21T08:57:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.463+0000 7f543a59d640 1 -- 172.21.15.28:0/3467639726 shutdown_connections 2024-04-21T08:57:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.463+0000 7f543a59d640 1 --2- 172.21.15.28:0/3467639726 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f541c0a3d20 0x7f541c0a61e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.463+0000 7f543a59d640 1 --2- 172.21.15.28:0/3467639726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f543410c510 0x7f54341985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.463+0000 7f543a59d640 1 --2- 172.21.15.28:0/3467639726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5434103af0 0x7f543419e940 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.463+0000 7f543a59d640 1 --2- 172.21.15.28:0/3467639726 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54341031d0 0x7f543419e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.463+0000 7f543a59d640 1 -- 172.21.15.28:0/3467639726 >> 172.21.15.28:0/3467639726 conn(0x7f5434077f50 msgr2=0x7f54340fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.463+0000 7f543a59d640 1 -- 172.21.15.28:0/3467639726 shutdown_connections 2024-04-21T08:57:03.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.463+0000 7f543a59d640 1 -- 172.21.15.28:0/3467639726 wait complete. 2024-04-21T08:57:03.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:57:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.581+0000 7f48ec663640 1 -- 172.21.15.28:0/447599869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48e4106160 msgr2=0x7f48e410dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.581+0000 7f48ec663640 1 --2- 172.21.15.28:0/447599869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48e4106160 0x7f48e410dd30 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f48dc009900 tx=0x7f48dc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.581+0000 7f48ec663640 1 -- 172.21.15.28:0/447599869 shutdown_connections 2024-04-21T08:57:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.581+0000 7f48ec663640 1 --2- 172.21.15.28:0/447599869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48e4106160 0x7f48e410dd30 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.581+0000 7f48ec663640 1 --2- 172.21.15.28:0/447599869 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48e4101a40 0x7f48e4105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.581+0000 7f48ec663640 1 --2- 172.21.15.28:0/447599869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48e4101090 0x7f48e4101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.581+0000 7f48ec663640 1 -- 172.21.15.28:0/447599869 >> 172.21.15.28:0/447599869 conn(0x7f48e4077f50 msgr2=0x7f48e4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.581+0000 7f48ec663640 1 -- 172.21.15.28:0/447599869 shutdown_connections 2024-04-21T08:57:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.581+0000 7f48ec663640 1 -- 172.21.15.28:0/447599869 wait complete. 2024-04-21T08:57:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48ec663640 1 Processor -- start 2024-04-21T08:57:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48ec663640 1 -- start start 2024-04-21T08:57:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48ec663640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48e4101090 0x7f48e41994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48ec663640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48e4106160 0x7f48e4199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48ec663640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48e4199fa0 0x7f48e41a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48ec663640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48e41110d0 con 0x7f48e4106160 2024-04-21T08:57:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48ec663640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f48e4110f50 con 0x7f48e4101090 2024-04-21T08:57:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48ec663640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f48e4111250 con 0x7f48e4199fa0 2024-04-21T08:57:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48e9bfe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48e4106160 0x7f48e4199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48e9bfe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48e4106160 0x7f48e4199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42662/0 (socket says 172.21.15.28:42662) 2024-04-21T08:57:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48e9bfe640 1 -- 172.21.15.28:0/2564461078 learned_addr learned my addr 172.21.15.28:0/2564461078 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48eac00640 1 --2- 172.21.15.28:0/2564461078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48e4199fa0 0x7f48e41a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48ea3ff640 1 --2- 172.21.15.28:0/2564461078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48e4101090 0x7f48e41994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48e9bfe640 1 -- 172.21.15.28:0/2564461078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48e4199fa0 msgr2=0x7f48e41a4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48e9bfe640 1 --2- 172.21.15.28:0/2564461078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48e4199fa0 0x7f48e41a4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48e9bfe640 1 -- 172.21.15.28:0/2564461078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48e4101090 msgr2=0x7f48e41994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48e9bfe640 1 --2- 172.21.15.28:0/2564461078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48e4101090 0x7f48e41994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48e9bfe640 1 -- 172.21.15.28:0/2564461078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48e41a4d20 con 0x7f48e4106160 2024-04-21T08:57:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.582+0000 7f48ea3ff640 1 --2- 172.21.15.28:0/2564461078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48e4101090 0x7f48e41994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:57:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.583+0000 7f48e9bfe640 1 --2- 172.21.15.28:0/2564461078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48e4106160 0x7f48e4199a10 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f48d800c970 tx=0x7f48d800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.583+0000 7f48e37fe640 1 -- 172.21.15.28:0/2564461078 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d8007bb0 con 0x7f48e4106160 2024-04-21T08:57:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.583+0000 7f48e37fe640 1 -- 172.21.15.28:0/2564461078 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48d8007d50 con 0x7f48e4106160 2024-04-21T08:57:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.583+0000 7f48ec663640 1 -- 172.21.15.28:0/2564461078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48e41a4fb0 con 0x7f48e4106160 2024-04-21T08:57:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.583+0000 7f48e37fe640 1 -- 172.21.15.28:0/2564461078 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d8018730 con 0x7f48e4106160 2024-04-21T08:57:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.583+0000 7f48ec663640 1 -- 172.21.15.28:0/2564461078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48e41a5470 con 0x7f48e4106160 2024-04-21T08:57:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.585+0000 7f48e37fe640 1 -- 172.21.15.28:0/2564461078 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f48d800f990 con 0x7f48e4106160 2024-04-21T08:57:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.585+0000 7f48ef258740 1 Processor -- start 2024-04-21T08:57:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.586+0000 7f48ef258740 1 -- start start 2024-04-21T08:57:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.586+0000 7f48ef258740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491d250 0x563d7491d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.586+0000 7f48ef258740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7491dc00 0x563d7492a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.586+0000 7f48ef258740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7492ae10 0x563d7492d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.586+0000 7f48ef258740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563d748821f0 con 0x563d7491d250 2024-04-21T08:57:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.586+0000 7f48ef258740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563d7481f4b0 con 0x563d7491dc00 2024-04-21T08:57:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.586+0000 7f48ef258740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x563d7487e1b0 con 0x563d7492ae10 2024-04-21T08:57:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.586+0000 7f48e37fe640 1 --2- 172.21.15.28:0/2564461078 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48cc0a3d20 0x7f48cc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.586+0000 7f48eac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491d250 0x563d7491d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.586+0000 7f48eac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491d250 0x563d7491d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42672/0 (socket says 172.21.15.28:42672) 2024-04-21T08:57:03.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.586+0000 7f48eac00640 1 -- 172.21.15.28:0/3545265360 learned_addr learned my addr 172.21.15.28:0/3545265360 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.586+0000 7f48eac00640 1 --2- 172.21.15.28:0/2564461078 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48cc0a3d20 0x7f48cc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.586+0000 7f48e37fe640 1 -- 172.21.15.28:0/2564461078 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f48d80d0b20 con 0x7f48e4106160 2024-04-21T08:57:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.586+0000 7f48e9bfe640 1 --2- 172.21.15.28:0/3545265360 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7492ae10 0x563d7492d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.586+0000 7f48ea3ff640 1 --2- 172.21.15.28:0/3545265360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7491dc00 0x563d7492a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.587+0000 7f48eac00640 1 -- 172.21.15.28:0/3545265360 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7492ae10 msgr2=0x563d7492d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.587+0000 7f48eac00640 1 --2- 172.21.15.28:0/3545265360 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7492ae10 0x563d7492d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.587+0000 7f48eac00640 1 -- 172.21.15.28:0/3545265360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7491dc00 msgr2=0x563d7492a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.587+0000 7f48eac00640 1 --2- 172.21.15.28:0/3545265360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7491dc00 0x563d7492a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.587+0000 7f48eac00640 1 -- 172.21.15.28:0/3545265360 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d748111b0 con 0x563d7491d250 2024-04-21T08:57:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.587+0000 7f48e9bfe640 1 --2- 172.21.15.28:0/3545265360 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7492ae10 0x563d7492d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:57:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.587+0000 7f48ea3ff640 1 --2- 172.21.15.28:0/3545265360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7491dc00 0x563d7492a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:57:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.587+0000 7f48eac00640 1 --2- 172.21.15.28:0/3545265360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491d250 0x563d7491d630 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f48dc006010 tx=0x7f48dc02cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecddf4620c3a8f6e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.587+0000 7f48eac00640 1 --2- 172.21.15.28:0/2564461078 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48cc0a3d20 0x7f48cc0a61e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f48dc005ec0 tx=0x7f48dc03b000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.587+0000 7f48e0bfe640 1 -- 172.21.15.28:0/3545265360 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc005590 con 0x563d7491d250 2024-04-21T08:57:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.587+0000 7f48e0bfe640 1 -- 172.21.15.28:0/3545265360 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48dc051070 con 0x563d7491d250 2024-04-21T08:57:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.587+0000 7f48e0bfe640 1 -- 172.21.15.28:0/3545265360 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc003970 con 0x563d7491d250 2024-04-21T08:57:03.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.588+0000 7f48ef258740 1 -- 172.21.15.28:0/3545265360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491d250 msgr2=0x563d7491d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.588+0000 7f48ef258740 1 --2- 172.21.15.28:0/3545265360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491d250 0x563d7491d630 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f48dc006010 tx=0x7f48dc02cc00 comp rx=0 tx=0).stop 2024-04-21T08:57:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.588+0000 7f48ef258740 1 -- 172.21.15.28:0/3545265360 shutdown_connections 2024-04-21T08:57:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.588+0000 7f48ef258740 1 --2- 172.21.15.28:0/3545265360 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7492ae10 0x563d7492d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.588+0000 7f48ef258740 1 --2- 172.21.15.28:0/3545265360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7491dc00 0x563d7492a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.588+0000 7f48ef258740 1 --2- 172.21.15.28:0/3545265360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491d250 0x563d7491d630 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.588+0000 7f48ef258740 1 -- 172.21.15.28:0/3545265360 >> 172.21.15.28:0/3545265360 conn(0x563d74917ba0 msgr2=0x563d74917f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.588+0000 7f48ef258740 1 -- 172.21.15.28:0/3545265360 shutdown_connections 2024-04-21T08:57:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.588+0000 7f48ef258740 1 -- 172.21.15.28:0/3545265360 wait complete. 2024-04-21T08:57:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.588+0000 7f48ef258740 1 -- start start 2024-04-21T08:57:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.588+0000 7f48ef258740 1 Processor -- start 2024-04-21T08:57:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.588+0000 7f48ef258740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7491d250 0x563d7492a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.588+0000 7f48ef258740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491dc00 0x563d74927b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.588+0000 7f48ef258740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7492ae10 0x563d74928120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48ef258740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563d7492dce0 con 0x563d7491dc00 2024-04-21T08:57:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48ef258740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x563d7483b500 con 0x563d7492ae10 2024-04-21T08:57:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48ef258740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x563d7492de60 con 0x563d7491d250 2024-04-21T08:57:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48ef258740 10 client.0 _ll_register_callbacks cb 0x563d74809b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48e9bfe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491dc00 0x563d74927b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48e9bfe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491dc00 0x563d74927b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42688/0 (socket says 172.21.15.28:42688) 2024-04-21T08:57:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48e9bfe640 1 -- 172.21.15.28:0/3360485261 learned_addr learned my addr 172.21.15.28:0/3360485261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48ea3ff640 1 --2- 172.21.15.28:0/3360485261 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7491d250 0x563d7492a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48eac00640 1 --2- 172.21.15.28:0/3360485261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7492ae10 0x563d74928120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48e9bfe640 1 -- 172.21.15.28:0/3360485261 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7491d250 msgr2=0x563d7492a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48e9bfe640 1 --2- 172.21.15.28:0/3360485261 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7491d250 0x563d7492a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48e9bfe640 1 -- 172.21.15.28:0/3360485261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7492ae10 msgr2=0x563d74928120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48e9bfe640 1 --2- 172.21.15.28:0/3360485261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7492ae10 0x563d74928120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48e9bfe640 1 -- 172.21.15.28:0/3360485261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d745943f0 con 0x563d7491dc00 2024-04-21T08:57:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48eac00640 1 --2- 172.21.15.28:0/3360485261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7492ae10 0x563d74928120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48e9bfe640 1 --2- 172.21.15.28:0/3360485261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491dc00 0x563d74927b10 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f48d80ce4d0 tx=0x7f48d809ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db62029091b0973 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48c9ffb640 1 -- 172.21.15.28:0/3360485261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d80a0ad0 con 0x563d7491dc00 2024-04-21T08:57:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48c9ffb640 1 -- 172.21.15.28:0/3360485261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f48d809bb70 con 0x563d7491dc00 2024-04-21T08:57:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48ef258740 1 -- 172.21.15.28:0/3360485261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48dc051070 con 0x563d7491dc00 2024-04-21T08:57:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48c9ffb640 1 -- 172.21.15.28:0/3360485261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f48d80a5390 con 0x563d7491dc00 2024-04-21T08:57:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48ef258740 10 client.17265 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.589+0000 7f48ef258740 1 -- 172.21.15.28:0/3360485261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563d7465e7d0 con 0x563d7491dc00 2024-04-21T08:57:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.590+0000 7f48c9ffb640 1 -- 172.21.15.28:0/3360485261 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48dc051070 con 0x563d7491dc00 2024-04-21T08:57:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.590+0000 7f48c9ffb640 1 -- 172.21.15.28:0/3360485261 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f48d8020030 con 0x563d7491dc00 2024-04-21T08:57:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.590+0000 7f48ef258740 10 client.17265 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.590+0000 7f48ef258740 10 client.17265 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:57:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.590+0000 7f48ef258740 20 client.17265 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.590+0000 7f48ef258740 1 --2- 172.21.15.28:0/3360485261 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x563d7493afe0 0x563d7493d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.590+0000 7f48ef258740 4 client.17265 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:57:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.590+0000 7f48ef258740 1 -- 172.21.15.28:0/3360485261 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d74821930 con 0x563d7493afe0 2024-04-21T08:57:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.590+0000 7f48ea3ff640 1 --2- 172.21.15.28:0/3360485261 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x563d7493afe0 0x563d7493d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.591+0000 7f48ea3ff640 1 --2- 172.21.15.28:0/3360485261 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x563d7493afe0 0x563d7493d3d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d5aefe0cd099826f server_cookie=2810806d067d92e7 in_seq=0 out_seq=0 2024-04-21T08:57:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.591+0000 7f48c9ffb640 10 client.17265 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:57:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.591+0000 7f48c9ffb640 1 -- 172.21.15.28:0/3360485261 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563d74821930 con 0x563d7493afe0 2024-04-21T08:57:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.591+0000 7f48c9ffb640 10 client.17265 handle_command_reply: tid=0 2024-04-21T08:57:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.591+0000 7f48ef258740 1 client.17265 shutdown 2024-04-21T08:57:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.591+0000 7f48ef258740 1 -- 172.21.15.28:0/3360485261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491dc00 msgr2=0x563d74927b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.591+0000 7f48ef258740 1 --2- 172.21.15.28:0/3360485261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491dc00 0x563d74927b10 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f48d80ce4d0 tx=0x7f48d809ce20 comp rx=0 tx=0).stop 2024-04-21T08:57:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.591+0000 7f48ef258740 1 -- 172.21.15.28:0/3360485261 shutdown_connections 2024-04-21T08:57:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.591+0000 7f48ef258740 1 --2- 172.21.15.28:0/3360485261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7492ae10 0x563d74928120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.591+0000 7f48ef258740 1 --2- 172.21.15.28:0/3360485261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491dc00 0x563d74927b10 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.591+0000 7f48ef258740 1 --2- 172.21.15.28:0/3360485261 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x563d7493afe0 0x563d7493d3d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.591+0000 7f48ef258740 1 --2- 172.21.15.28:0/3360485261 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7491d250 0x563d7492a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.591+0000 7f48ef258740 1 -- 172.21.15.28:0/3360485261 >> 172.21.15.28:0/3360485261 conn(0x563d74917ba0 msgr2=0x563d749198f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.591+0000 7f48c9ffb640 0 client.17265 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:57:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.592+0000 7f48ef258740 1 -- 172.21.15.28:0/3360485261 shutdown_connections 2024-04-21T08:57:03.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.592+0000 7f48ef258740 1 -- 172.21.15.28:0/3360485261 wait complete. 2024-04-21T08:57:03.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.592+0000 7f48ef258740 20 client.17265 trim_cache size 0 max 16384 2024-04-21T08:57:03.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.605+0000 7f48ef258740 1 Processor -- start 2024-04-21T08:57:03.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.605+0000 7f48ef258740 1 -- start start 2024-04-21T08:57:03.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.605+0000 7f48ef258740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7491d250 0x563d74936c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.605+0000 7f48ef258740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491dc00 0x563d7492a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.605+0000 7f48ef258740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7492ae10 0x563d7492a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.605+0000 7f48ef258740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48e41af800 con 0x563d7491dc00 2024-04-21T08:57:03.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.605+0000 7f48ef258740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f48e41ad0d0 con 0x563d7492ae10 2024-04-21T08:57:03.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.605+0000 7f48ef258740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f48e41aaf20 con 0x563d7491d250 2024-04-21T08:57:03.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.605+0000 7f48e9bfe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491dc00 0x563d7492a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.605+0000 7f48e9bfe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491dc00 0x563d7492a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42700/0 (socket says 172.21.15.28:42700) 2024-04-21T08:57:03.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.605+0000 7f48e9bfe640 1 -- 172.21.15.28:0/877335393 learned_addr learned my addr 172.21.15.28:0/877335393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.606+0000 7f48ea3ff640 1 --2- 172.21.15.28:0/877335393 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7491d250 0x563d74936c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.606+0000 7f48e9bfe640 1 -- 172.21.15.28:0/877335393 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7491d250 msgr2=0x563d74936c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.606+0000 7f48e9bfe640 1 --2- 172.21.15.28:0/877335393 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7491d250 0x563d74936c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.606+0000 7f48e9bfe640 1 -- 172.21.15.28:0/877335393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7492ae10 msgr2=0x563d7492a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.606+0000 7f48e9bfe640 1 --2- 172.21.15.28:0/877335393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7492ae10 0x563d7492a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.606+0000 7f48e9bfe640 1 -- 172.21.15.28:0/877335393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d7493d740 con 0x563d7491dc00 2024-04-21T08:57:03.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.606+0000 7f48e9bfe640 1 --2- 172.21.15.28:0/877335393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491dc00 0x563d7492a240 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f48d800c940 tx=0x7f48d809ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82deb9f14d8a6472 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.606+0000 7f48cbfff640 1 -- 172.21.15.28:0/877335393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d8098470 con 0x563d7491dc00 2024-04-21T08:57:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.606+0000 7f48cbfff640 1 -- 172.21.15.28:0/877335393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48d8098d90 con 0x563d7491dc00 2024-04-21T08:57:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.606+0000 7f48cbfff640 1 -- 172.21.15.28:0/877335393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d809bb70 con 0x563d7491dc00 2024-04-21T08:57:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 -- 172.21.15.28:0/877335393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491dc00 msgr2=0x563d7492a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 --2- 172.21.15.28:0/877335393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491dc00 0x563d7492a240 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f48d800c940 tx=0x7f48d809ac30 comp rx=0 tx=0).stop 2024-04-21T08:57:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 -- 172.21.15.28:0/877335393 shutdown_connections 2024-04-21T08:57:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 --2- 172.21.15.28:0/877335393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7492ae10 0x563d7492a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 --2- 172.21.15.28:0/877335393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7491dc00 0x563d7492a240 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 --2- 172.21.15.28:0/877335393 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7491d250 0x563d74936c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 -- 172.21.15.28:0/877335393 >> 172.21.15.28:0/877335393 conn(0x563d74917ba0 msgr2=0x563d74919880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 -- 172.21.15.28:0/877335393 shutdown_connections 2024-04-21T08:57:03.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 -- 172.21.15.28:0/877335393 wait complete. 2024-04-21T08:57:03.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 -- start start 2024-04-21T08:57:03.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 Processor -- start 2024-04-21T08:57:03.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7491d250 0x563d74933120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7491dc00 0x563d74933660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7492ae10 0x563d749300f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48e41ab590 con 0x563d7492ae10 2024-04-21T08:57:03.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f48e41aca00 con 0x563d7491dc00 2024-04-21T08:57:03.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f48e41abc60 con 0x563d7491d250 2024-04-21T08:57:03.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ef258740 10 client.0 _ll_register_callbacks cb 0x7f48d809c080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:03.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48eac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7492ae10 0x563d749300f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48eac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7492ae10 0x563d749300f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42702/0 (socket says 172.21.15.28:42702) 2024-04-21T08:57:03.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48eac00640 1 -- 172.21.15.28:0/1599484212 learned_addr learned my addr 172.21.15.28:0/1599484212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.607+0000 7f48ea3ff640 1 --2- 172.21.15.28:0/1599484212 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7491d250 0x563d74933120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.608+0000 7f48eac00640 1 -- 172.21.15.28:0/1599484212 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7491d250 msgr2=0x563d74933120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.608+0000 7f48eac00640 1 --2- 172.21.15.28:0/1599484212 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7491d250 0x563d74933120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.608+0000 7f48eac00640 1 -- 172.21.15.28:0/1599484212 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7491dc00 msgr2=0x563d74933660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.608+0000 7f48e9bfe640 1 --2- 172.21.15.28:0/1599484212 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7491dc00 0x563d74933660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.608+0000 7f48eac00640 1 --2- 172.21.15.28:0/1599484212 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7491dc00 0x563d74933660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.608+0000 7f48eac00640 1 -- 172.21.15.28:0/1599484212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d74930680 con 0x563d7492ae10 2024-04-21T08:57:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.608+0000 7f48eac00640 1 --2- 172.21.15.28:0/1599484212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7492ae10 0x563d749300f0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f48dc02ce10 tx=0x7f48dc004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c7feada524b858 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.608+0000 7f48c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.608+0000 7f48c9ffb640 1 -- 172.21.15.28:0/1599484212 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc059020 con 0x563d7492ae10 2024-04-21T08:57:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.608+0000 7f48c9ffb640 1 -- 172.21.15.28:0/1599484212 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f48dc0035f0 con 0x563d7492ae10 2024-04-21T08:57:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.608+0000 7f48ef258740 1 -- 172.21.15.28:0/1599484212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48d8098d90 con 0x563d7492ae10 2024-04-21T08:57:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.608+0000 7f48c9ffb640 1 -- 172.21.15.28:0/1599484212 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f48dc0037f0 con 0x563d7492ae10 2024-04-21T08:57:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.608+0000 7f48ef258740 10 client.17271 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.608+0000 7f48ef258740 1 -- 172.21.15.28:0/1599484212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563d749a1da0 con 0x563d7492ae10 2024-04-21T08:57:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.608+0000 7f48c9ffb640 1 -- 172.21.15.28:0/1599484212 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48d8098d90 con 0x563d7492ae10 2024-04-21T08:57:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.609+0000 7f48c9ffb640 1 -- 172.21.15.28:0/1599484212 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f48dc044460 con 0x563d7492ae10 2024-04-21T08:57:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.609+0000 7f48ef258740 10 client.17271 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.609+0000 7f48ef258740 10 client.17271 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:57:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.609+0000 7f48ef258740 20 client.17271 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.609+0000 7f48ef258740 1 --2- 172.21.15.28:0/1599484212 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x563d749a2120 0x563d749a4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.609+0000 7f48ef258740 4 client.17271 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:57:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.609+0000 7f48ef258740 1 -- 172.21.15.28:0/1599484212 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563d7488fc10 con 0x563d749a2120 2024-04-21T08:57:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.609+0000 7f48ea3ff640 1 --2- 172.21.15.28:0/1599484212 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x563d749a2120 0x563d749a4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.609+0000 7f48ea3ff640 1 --2- 172.21.15.28:0/1599484212 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x563d749a2120 0x563d749a4540 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=db8a651747c17c9 server_cookie=86962dd3c0decf3b in_seq=0 out_seq=0 2024-04-21T08:57:03.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.609+0000 7f48c9ffb640 10 client.17271 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:57:03.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48c9ffb640 1 -- 172.21.15.28:0/1599484212 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563d7488fc10 con 0x563d749a2120 2024-04-21T08:57:03.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48c9ffb640 10 client.17271 handle_command_reply: tid=0 2024-04-21T08:57:03.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ef258740 1 client.17271 shutdown 2024-04-21T08:57:03.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ef258740 1 -- 172.21.15.28:0/1599484212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7492ae10 msgr2=0x563d749300f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ef258740 1 --2- 172.21.15.28:0/1599484212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7492ae10 0x563d749300f0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f48dc02ce10 tx=0x7f48dc004bb0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ef258740 1 -- 172.21.15.28:0/1599484212 shutdown_connections 2024-04-21T08:57:03.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ef258740 1 --2- 172.21.15.28:0/1599484212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d7492ae10 0x563d749300f0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ef258740 1 --2- 172.21.15.28:0/1599484212 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563d7491dc00 0x563d74933660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ef258740 1 --2- 172.21.15.28:0/1599484212 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x563d749a2120 0x563d749a4540 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ef258740 1 --2- 172.21.15.28:0/1599484212 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x563d7491d250 0x563d74933120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48c9ffb640 0 client.17271 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:57:03.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ef258740 1 -- 172.21.15.28:0/1599484212 >> 172.21.15.28:0/1599484212 conn(0x563d74917ba0 msgr2=0x563d749227e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ef258740 1 -- 172.21.15.28:0/1599484212 shutdown_connections 2024-04-21T08:57:03.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ef258740 1 -- 172.21.15.28:0/1599484212 wait complete. 2024-04-21T08:57:03.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ef258740 20 client.17271 trim_cache size 0 max 16384 2024-04-21T08:57:03.621 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:57:03.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ec663640 1 -- 172.21.15.28:0/2564461078 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48cc0a3d20 msgr2=0x7f48cc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ec663640 1 --2- 172.21.15.28:0/2564461078 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48cc0a3d20 0x7f48cc0a61e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f48dc005ec0 tx=0x7f48dc03b000 comp rx=0 tx=0).stop 2024-04-21T08:57:03.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ec663640 1 -- 172.21.15.28:0/2564461078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48e4106160 msgr2=0x7f48e4199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ec663640 1 --2- 172.21.15.28:0/2564461078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48e4106160 0x7f48e4199a10 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f48d800c970 tx=0x7f48d800ce30 comp rx=0 tx=0).stop 2024-04-21T08:57:03.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ec663640 1 -- 172.21.15.28:0/2564461078 shutdown_connections 2024-04-21T08:57:03.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ec663640 1 --2- 172.21.15.28:0/2564461078 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48cc0a3d20 0x7f48cc0a61e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ec663640 1 --2- 172.21.15.28:0/2564461078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48e4199fa0 0x7f48e41a4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ec663640 1 --2- 172.21.15.28:0/2564461078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48e4106160 0x7f48e4199a10 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ec663640 1 --2- 172.21.15.28:0/2564461078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f48e4101090 0x7f48e41994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ec663640 1 -- 172.21.15.28:0/2564461078 >> 172.21.15.28:0/2564461078 conn(0x7f48e4077f50 msgr2=0x7f48e40feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ec663640 1 -- 172.21.15.28:0/2564461078 shutdown_connections 2024-04-21T08:57:03.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.610+0000 7f48ec663640 1 -- 172.21.15.28:0/2564461078 wait complete. 2024-04-21T08:57:03.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:57:03.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.729+0000 7f4b4a9ab640 1 -- 172.21.15.28:0/443006223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44101a50 msgr2=0x7f4b44105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.729+0000 7f4b4a9ab640 1 --2- 172.21.15.28:0/443006223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44101a50 0x7f4b44105a90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f4b38009900 tx=0x7f4b3802bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:03.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.729+0000 7f4b4a9ab640 1 -- 172.21.15.28:0/443006223 shutdown_connections 2024-04-21T08:57:03.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.729+0000 7f4b4a9ab640 1 --2- 172.21.15.28:0/443006223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b44106170 0x7f4b4410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.729+0000 7f4b4a9ab640 1 --2- 172.21.15.28:0/443006223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44101a50 0x7f4b44105a90 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.729+0000 7f4b4a9ab640 1 --2- 172.21.15.28:0/443006223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b441010a0 0x7f4b44101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.729+0000 7f4b4a9ab640 1 -- 172.21.15.28:0/443006223 >> 172.21.15.28:0/443006223 conn(0x7f4b44077eb0 msgr2=0x7f4b440782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.729+0000 7f4b4a9ab640 1 -- 172.21.15.28:0/443006223 shutdown_connections 2024-04-21T08:57:03.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.729+0000 7f4b4a9ab640 1 -- 172.21.15.28:0/443006223 wait complete. 2024-04-21T08:57:03.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.729+0000 7f4b4a9ab640 1 Processor -- start 2024-04-21T08:57:03.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.729+0000 7f4b4a9ab640 1 -- start start 2024-04-21T08:57:03.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b4a9ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b441010a0 0x7f4b44198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b4a9ab640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b44106170 0x7f4b44198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b4a9ab640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b4419cfc0 0x7f4b4419d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b4a9ab640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b44111090 con 0x7f4b441010a0 2024-04-21T08:57:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b4a9ab640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4b44110f10 con 0x7f4b44106170 2024-04-21T08:57:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b4a9ab640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4b44111210 con 0x7f4b4419cfc0 2024-04-21T08:57:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b43fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b441010a0 0x7f4b44198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b43fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b441010a0 0x7f4b44198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56632/0 (socket says 172.21.15.28:56632) 2024-04-21T08:57:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b43fff640 1 -- 172.21.15.28:0/137163744 learned_addr learned my addr 172.21.15.28:0/137163744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b48f48640 1 --2- 172.21.15.28:0/137163744 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b4419cfc0 0x7f4b4419d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b437fe640 1 --2- 172.21.15.28:0/137163744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b44106170 0x7f4b44198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b43fff640 1 -- 172.21.15.28:0/137163744 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b4419cfc0 msgr2=0x7f4b4419d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b43fff640 1 --2- 172.21.15.28:0/137163744 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b4419cfc0 0x7f4b4419d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b43fff640 1 -- 172.21.15.28:0/137163744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b44106170 msgr2=0x7f4b44198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b43fff640 1 --2- 172.21.15.28:0/137163744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b44106170 0x7f4b44198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b43fff640 1 -- 172.21.15.28:0/137163744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b4419dab0 con 0x7f4b441010a0 2024-04-21T08:57:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b437fe640 1 --2- 172.21.15.28:0/137163744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b44106170 0x7f4b44198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b43fff640 1 --2- 172.21.15.28:0/137163744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b441010a0 0x7f4b44198700 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400e9f0 tx=0x7f4b3400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b417fa640 1 -- 172.21.15.28:0/137163744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3400cd80 con 0x7f4b441010a0 2024-04-21T08:57:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b417fa640 1 -- 172.21.15.28:0/137163744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b340108e0 con 0x7f4b441010a0 2024-04-21T08:57:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b4a9ab640 1 -- 172.21.15.28:0/137163744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b4419dda0 con 0x7f4b441010a0 2024-04-21T08:57:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b417fa640 1 -- 172.21.15.28:0/137163744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b34018980 con 0x7f4b441010a0 2024-04-21T08:57:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.730+0000 7f4b4a9ab640 1 -- 172.21.15.28:0/137163744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b441a5590 con 0x7f4b441010a0 2024-04-21T08:57:03.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b417fa640 1 -- 172.21.15.28:0/137163744 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4b34010430 con 0x7f4b441010a0 2024-04-21T08:57:03.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b4d59d740 1 Processor -- start 2024-04-21T08:57:03.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b4d59d740 1 -- start start 2024-04-21T08:57:03.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b4d59d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8a9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b4d59d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a9d60 0x55834e8b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b4d59d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8b2e60 0x55834e8b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b4d59d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55834e80e1f0 con 0x55834e8a93b0 2024-04-21T08:57:03.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b4d59d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55834e7ab4b0 con 0x55834e8a9d60 2024-04-21T08:57:03.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b4d59d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55834e80a1b0 con 0x55834e8b2e60 2024-04-21T08:57:03.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b48f48640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8a9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b48f48640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8a9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56646/0 (socket says 172.21.15.28:56646) 2024-04-21T08:57:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b48f48640 1 -- 172.21.15.28:0/916348987 learned_addr learned my addr 172.21.15.28:0/916348987 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b417fa640 1 --2- 172.21.15.28:0/137163744 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4b2c0a3d20 0x7f4b2c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b437fe640 1 --2- 172.21.15.28:0/916348987 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8b2e60 0x55834e8b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b48f48640 1 --2- 172.21.15.28:0/137163744 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4b2c0a3d20 0x7f4b2c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b43fff640 1 --2- 172.21.15.28:0/916348987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a9d60 0x55834e8b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b48f48640 1 -- 172.21.15.28:0/916348987 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8b2e60 msgr2=0x55834e8b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b48f48640 1 --2- 172.21.15.28:0/916348987 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8b2e60 0x55834e8b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.733+0000 7f4b417fa640 1 -- 172.21.15.28:0/137163744 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f4b340d0fe0 con 0x7f4b441010a0 2024-04-21T08:57:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.734+0000 7f4b48f48640 1 -- 172.21.15.28:0/916348987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a9d60 msgr2=0x55834e8b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.734+0000 7f4b48f48640 1 --2- 172.21.15.28:0/916348987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a9d60 0x55834e8b2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.734+0000 7f4b48f48640 1 -- 172.21.15.28:0/916348987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55834e79d1b0 con 0x55834e8a93b0 2024-04-21T08:57:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.734+0000 7f4b437fe640 1 --2- 172.21.15.28:0/916348987 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8b2e60 0x55834e8b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:57:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.734+0000 7f4b43fff640 1 --2- 172.21.15.28:0/916348987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a9d60 0x55834e8b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.734+0000 7f4b48f48640 1 --2- 172.21.15.28:0/916348987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8a9790 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0129f0 tx=0x7f4b3c012eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe6c7f9510aa38d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.734+0000 7f4b2a7fc640 1 -- 172.21.15.28:0/916348987 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c023070 con 0x55834e8a93b0 2024-04-21T08:57:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.734+0000 7f4b2a7fc640 1 -- 172.21.15.28:0/916348987 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b3c0046e0 con 0x55834e8a93b0 2024-04-21T08:57:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.734+0000 7f4b2a7fc640 1 -- 172.21.15.28:0/916348987 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c01f8f0 con 0x55834e8a93b0 2024-04-21T08:57:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 -- 172.21.15.28:0/916348987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 msgr2=0x55834e8a9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b48f48640 1 --2- 172.21.15.28:0/137163744 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4b2c0a3d20 0x7f4b2c0a61e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c022000 tx=0x7f4b3c021000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/916348987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8a9790 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0129f0 tx=0x7f4b3c012eb0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 -- 172.21.15.28:0/916348987 shutdown_connections 2024-04-21T08:57:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/916348987 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8b2e60 0x55834e8b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/916348987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a9d60 0x55834e8b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/916348987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8a9790 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 -- 172.21.15.28:0/916348987 >> 172.21.15.28:0/916348987 conn(0x55834e8a3ba0 msgr2=0x55834e8a3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 -- 172.21.15.28:0/916348987 shutdown_connections 2024-04-21T08:57:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 -- 172.21.15.28:0/916348987 wait complete. 2024-04-21T08:57:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 -- start start 2024-04-21T08:57:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 Processor -- start 2024-04-21T08:57:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8b92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8a9d60 0x55834e8b6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8b2e60 0x55834e8b6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55834e8b9da0 con 0x55834e8a93b0 2024-04-21T08:57:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55834e7c7500 con 0x55834e8b2e60 2024-04-21T08:57:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.735+0000 7f4b4d59d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55834e8b9f20 con 0x55834e8a9d60 2024-04-21T08:57:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b4d59d740 10 client.0 _ll_register_callbacks cb 0x55834e795b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b43fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8b92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b43fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8b92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56650/0 (socket says 172.21.15.28:56650) 2024-04-21T08:57:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b43fff640 1 -- 172.21.15.28:0/3837984295 learned_addr learned my addr 172.21.15.28:0/3837984295 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b437fe640 1 --2- 172.21.15.28:0/3837984295 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8a9d60 0x55834e8b6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b48f48640 1 --2- 172.21.15.28:0/3837984295 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8b2e60 0x55834e8b6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b43fff640 1 -- 172.21.15.28:0/3837984295 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8a9d60 msgr2=0x55834e8b6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b43fff640 1 --2- 172.21.15.28:0/3837984295 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8a9d60 0x55834e8b6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b43fff640 1 -- 172.21.15.28:0/3837984295 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8b2e60 msgr2=0x55834e8b6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b43fff640 1 --2- 172.21.15.28:0/3837984295 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8b2e60 0x55834e8b6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b43fff640 1 -- 172.21.15.28:0/3837984295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55834e5203f0 con 0x55834e8a93b0 2024-04-21T08:57:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b48f48640 1 --2- 172.21.15.28:0/3837984295 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8b2e60 0x55834e8b6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b43fff640 1 --2- 172.21.15.28:0/3837984295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8b92b0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4b340ce440 tx=0x7f4b3401d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad083ffa3d222af1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b237fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b237fe640 1 -- 172.21.15.28:0/3837984295 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b34010720 con 0x55834e8a93b0 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b237fe640 1 -- 172.21.15.28:0/3837984295 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f4b34014d20 con 0x55834e8a93b0 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b4d59d740 1 -- 172.21.15.28:0/3837984295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b3c0046e0 con 0x55834e8a93b0 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b237fe640 1 -- 172.21.15.28:0/3837984295 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4b340b6150 con 0x55834e8a93b0 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b4d59d740 10 client.17283 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.736+0000 7f4b4d59d740 1 -- 172.21.15.28:0/3837984295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55834e5ea7d0 con 0x55834e8a93b0 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.737+0000 7f4b237fe640 1 -- 172.21.15.28:0/3837984295 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b3c0046e0 con 0x55834e8a93b0 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.737+0000 7f4b237fe640 1 -- 172.21.15.28:0/3837984295 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f4b34022ce0 con 0x55834e8a93b0 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.737+0000 7f4b4d59d740 10 client.17283 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.737+0000 7f4b4d59d740 10 client.17283 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.737+0000 7f4b4d59d740 20 client.17283 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.737+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/3837984295 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55834e8c6f50 0x55834e8c9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.737+0000 7f4b4d59d740 4 client.17283 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.737+0000 7f4b4d59d740 1 -- 172.21.15.28:0/3837984295 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55834e7ad930 con 0x55834e8c6f50 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.737+0000 7f4b437fe640 1 --2- 172.21.15.28:0/3837984295 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55834e8c6f50 0x55834e8c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.738+0000 7f4b237fe640 10 client.17283 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.738+0000 7f4b437fe640 1 --2- 172.21.15.28:0/3837984295 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55834e8c6f50 0x55834e8c9340 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff84ff726b2d46d5 server_cookie=a81fc105d8f655bb in_seq=0 out_seq=0 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.738+0000 7f4b237fe640 1 -- 172.21.15.28:0/3837984295 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55834e7ad930 con 0x55834e8c6f50 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.738+0000 7f4b237fe640 10 client.17283 handle_command_reply: tid=0 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.738+0000 7f4b4d59d740 1 client.17283 shutdown 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.739+0000 7f4b4d59d740 1 -- 172.21.15.28:0/3837984295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 msgr2=0x55834e8b92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.739+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/3837984295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8b92b0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f4b340ce440 tx=0x7f4b3401d9e0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.739+0000 7f4b4d59d740 1 -- 172.21.15.28:0/3837984295 shutdown_connections 2024-04-21T08:57:03.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.739+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/3837984295 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55834e8c6f50 0x55834e8c9340 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.739+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/3837984295 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8b2e60 0x55834e8b6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.739+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/3837984295 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8a9d60 0x55834e8b6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.739+0000 7f4b237fe640 0 client.17283 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:03.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.739+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/3837984295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8b92b0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.739+0000 7f4b4d59d740 1 -- 172.21.15.28:0/3837984295 >> 172.21.15.28:0/3837984295 conn(0x55834e8a3ba0 msgr2=0x55834e8a58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.739+0000 7f4b4d59d740 1 -- 172.21.15.28:0/3837984295 shutdown_connections 2024-04-21T08:57:03.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.739+0000 7f4b4d59d740 1 -- 172.21.15.28:0/3837984295 wait complete. 2024-04-21T08:57:03.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.739+0000 7f4b4d59d740 20 client.17283 trim_cache size 0 max 16384 2024-04-21T08:57:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.752+0000 7f4b4d59d740 1 Processor -- start 2024-04-21T08:57:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.752+0000 7f4b4d59d740 1 -- start start 2024-04-21T08:57:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.752+0000 7f4b4d59d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a93b0 0x55834e8b4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.752+0000 7f4b4d59d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a9d60 0x55834e8c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.752+0000 7f4b4d59d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8b2e60 0x55834e8b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.752+0000 7f4b4d59d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b441af9e0 con 0x55834e8a9d60 2024-04-21T08:57:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.752+0000 7f4b4d59d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4b441ad2b0 con 0x55834e8a93b0 2024-04-21T08:57:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.752+0000 7f4b4d59d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4b441ab100 con 0x55834e8b2e60 2024-04-21T08:57:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.752+0000 7f4b437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a9d60 0x55834e8c2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.752+0000 7f4b437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a9d60 0x55834e8c2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56654/0 (socket says 172.21.15.28:56654) 2024-04-21T08:57:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.753+0000 7f4b437fe640 1 -- 172.21.15.28:0/1055045641 learned_addr learned my addr 172.21.15.28:0/1055045641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.753+0000 7f4b43fff640 1 --2- 172.21.15.28:0/1055045641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a93b0 0x55834e8b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.753+0000 7f4b437fe640 1 -- 172.21.15.28:0/1055045641 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8b2e60 msgr2=0x55834e8b57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.753+0000 7f4b437fe640 1 --2- 172.21.15.28:0/1055045641 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8b2e60 0x55834e8b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.753+0000 7f4b437fe640 1 -- 172.21.15.28:0/1055045641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a93b0 msgr2=0x55834e8b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.753+0000 7f4b437fe640 1 --2- 172.21.15.28:0/1055045641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a93b0 0x55834e8b4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.753+0000 7f4b437fe640 1 -- 172.21.15.28:0/1055045641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55834e8a36d0 con 0x55834e8a9d60 2024-04-21T08:57:03.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.753+0000 7f4b437fe640 1 --2- 172.21.15.28:0/1055045641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a9d60 0x55834e8c2ca0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f4b4406a400 tx=0x7f4b38005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6368c9715b8b0be server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.753+0000 7f4b29ffb640 1 -- 172.21.15.28:0/1055045641 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3804d020 con 0x55834e8a9d60 2024-04-21T08:57:03.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.753+0000 7f4b29ffb640 1 -- 172.21.15.28:0/1055045641 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b3802c7e0 con 0x55834e8a9d60 2024-04-21T08:57:03.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.753+0000 7f4b29ffb640 1 -- 172.21.15.28:0/1055045641 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b380055e0 con 0x55834e8a9d60 2024-04-21T08:57:03.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 -- 172.21.15.28:0/1055045641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a9d60 msgr2=0x55834e8c2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/1055045641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a9d60 0x55834e8c2ca0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f4b4406a400 tx=0x7f4b38005b50 comp rx=0 tx=0).stop 2024-04-21T08:57:03.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 -- 172.21.15.28:0/1055045641 shutdown_connections 2024-04-21T08:57:03.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/1055045641 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8b2e60 0x55834e8b57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/1055045641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a9d60 0x55834e8c2ca0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/1055045641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a93b0 0x55834e8b4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 -- 172.21.15.28:0/1055045641 >> 172.21.15.28:0/1055045641 conn(0x55834e8a3ba0 msgr2=0x55834e8b7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 -- 172.21.15.28:0/1055045641 shutdown_connections 2024-04-21T08:57:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 -- 172.21.15.28:0/1055045641 wait complete. 2024-04-21T08:57:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 -- start start 2024-04-21T08:57:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 Processor -- start 2024-04-21T08:57:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8ce1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a9d60 0x55834e8ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8b2e60 0x55834e8cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b441ab770 con 0x55834e8a93b0 2024-04-21T08:57:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4b441acbe0 con 0x55834e8a9d60 2024-04-21T08:57:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4b441abe40 con 0x55834e8b2e60 2024-04-21T08:57:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b4d59d740 10 client.0 _ll_register_callbacks cb 0x7f4b3409c9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.754+0000 7f4b43fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8ce1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b43fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8ce1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56660/0 (socket says 172.21.15.28:56660) 2024-04-21T08:57:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b43fff640 1 -- 172.21.15.28:0/4219028901 learned_addr learned my addr 172.21.15.28:0/4219028901 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b437fe640 1 --2- 172.21.15.28:0/4219028901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a9d60 0x55834e8ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b48f48640 1 --2- 172.21.15.28:0/4219028901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8b2e60 0x55834e8cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b43fff640 1 -- 172.21.15.28:0/4219028901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8b2e60 msgr2=0x55834e8cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b43fff640 1 --2- 172.21.15.28:0/4219028901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8b2e60 0x55834e8cec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b43fff640 1 -- 172.21.15.28:0/4219028901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a9d60 msgr2=0x55834e8ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b43fff640 1 --2- 172.21.15.28:0/4219028901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a9d60 0x55834e8ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b43fff640 1 -- 172.21.15.28:0/4219028901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55834e92c550 con 0x55834e8a93b0 2024-04-21T08:57:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b43fff640 1 --2- 172.21.15.28:0/4219028901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8ce1d0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f4b3409c8e0 tx=0x7f4b340a10b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=401c9545a8a6fef3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b237fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b237fe640 1 -- 172.21.15.28:0/4219028901 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3409c540 con 0x55834e8a93b0 2024-04-21T08:57:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b237fe640 1 -- 172.21.15.28:0/4219028901 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f4b34018b20 con 0x55834e8a93b0 2024-04-21T08:57:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b4d59d740 1 -- 172.21.15.28:0/4219028901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b3802c7e0 con 0x55834e8a93b0 2024-04-21T08:57:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b237fe640 1 -- 172.21.15.28:0/4219028901 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4b340b6150 con 0x55834e8a93b0 2024-04-21T08:57:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b4d59d740 10 client.17289 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b4d59d740 1 -- 172.21.15.28:0/4219028901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55834e92d830 con 0x55834e8a93b0 2024-04-21T08:57:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.755+0000 7f4b237fe640 1 -- 172.21.15.28:0/4219028901 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b3802c7e0 con 0x55834e8a93b0 2024-04-21T08:57:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.756+0000 7f4b237fe640 1 -- 172.21.15.28:0/4219028901 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f4b34098930 con 0x55834e8a93b0 2024-04-21T08:57:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.756+0000 7f4b4d59d740 10 client.17289 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.756+0000 7f4b4d59d740 10 client.17289 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.756+0000 7f4b4d59d740 20 client.17289 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.756+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/4219028901 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55834e92df80 0x55834e9303a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.756+0000 7f4b4d59d740 4 client.17289 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:57:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.756+0000 7f4b4d59d740 1 -- 172.21.15.28:0/4219028901 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55834e81bc10 con 0x55834e92df80 2024-04-21T08:57:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.756+0000 7f4b437fe640 1 --2- 172.21.15.28:0/4219028901 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55834e92df80 0x55834e9303a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.756+0000 7f4b237fe640 10 client.17289 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.756+0000 7f4b437fe640 1 --2- 172.21.15.28:0/4219028901 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55834e92df80 0x55834e9303a0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c96aa70dcc53917e server_cookie=25709441ee694ac8 in_seq=0 out_seq=0 2024-04-21T08:57:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b237fe640 1 -- 172.21.15.28:0/4219028901 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55834e81bc10 con 0x55834e92df80 2024-04-21T08:57:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b237fe640 10 client.17289 handle_command_reply: tid=0 2024-04-21T08:57:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4d59d740 1 client.17289 shutdown 2024-04-21T08:57:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4d59d740 1 -- 172.21.15.28:0/4219028901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 msgr2=0x55834e8ce1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/4219028901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8ce1d0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f4b3409c8e0 tx=0x7f4b340a10b0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4d59d740 1 -- 172.21.15.28:0/4219028901 shutdown_connections 2024-04-21T08:57:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/4219028901 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55834e92df80 0x55834e9303a0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/4219028901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55834e8b2e60 0x55834e8cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/4219028901 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55834e8a9d60 0x55834e8ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b237fe640 0 client.17289 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4d59d740 1 --2- 172.21.15.28:0/4219028901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55834e8a93b0 0x55834e8ce1d0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4d59d740 1 -- 172.21.15.28:0/4219028901 >> 172.21.15.28:0/4219028901 conn(0x55834e8a3ba0 msgr2=0x55834e8a2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4d59d740 1 -- 172.21.15.28:0/4219028901 shutdown_connections 2024-04-21T08:57:03.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4d59d740 1 -- 172.21.15.28:0/4219028901 wait complete. 2024-04-21T08:57:03.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4d59d740 20 client.17289 trim_cache size 0 max 16384 2024-04-21T08:57:03.768 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:57:03.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4a9ab640 1 -- 172.21.15.28:0/137163744 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4b2c0a3d20 msgr2=0x7f4b2c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4a9ab640 1 --2- 172.21.15.28:0/137163744 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4b2c0a3d20 0x7f4b2c0a61e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c022000 tx=0x7f4b3c021000 comp rx=0 tx=0).stop 2024-04-21T08:57:03.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4a9ab640 1 -- 172.21.15.28:0/137163744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b441010a0 msgr2=0x7f4b44198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4a9ab640 1 --2- 172.21.15.28:0/137163744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b441010a0 0x7f4b44198700 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400e9f0 tx=0x7f4b3400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4a9ab640 1 -- 172.21.15.28:0/137163744 shutdown_connections 2024-04-21T08:57:03.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4a9ab640 1 --2- 172.21.15.28:0/137163744 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4b2c0a3d20 0x7f4b2c0a61e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4a9ab640 1 --2- 172.21.15.28:0/137163744 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b4419cfc0 0x7f4b4419d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4a9ab640 1 --2- 172.21.15.28:0/137163744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b44106170 0x7f4b44198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.757+0000 7f4b4a9ab640 1 --2- 172.21.15.28:0/137163744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b441010a0 0x7f4b44198700 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.758+0000 7f4b4a9ab640 1 -- 172.21.15.28:0/137163744 >> 172.21.15.28:0/137163744 conn(0x7f4b44077eb0 msgr2=0x7f4b44104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.758+0000 7f4b4a9ab640 1 -- 172.21.15.28:0/137163744 shutdown_connections 2024-04-21T08:57:03.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:03.758+0000 7f4b4a9ab640 1 -- 172.21.15.28:0/137163744 wait complete. 2024-04-21T08:57:04.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:04 smithi152 ceph-mon[31401]: pgmap v179: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-21T08:57:04.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:04 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/318408519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:04 smithi028 ceph-mon[28003]: pgmap v179: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-21T08:57:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:04 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/318408519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:04.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:04 smithi171 ceph-mon[31358]: pgmap v179: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-21T08:57:04.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:04 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/318408519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:04.772 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:57:04.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:04.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:57:04.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.893+0000 7f4843304640 1 -- 172.21.15.28:0/798660579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c10c510 msgr2=0x7f483c10e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.893+0000 7f4843304640 1 --2- 172.21.15.28:0/798660579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c10c510 0x7f483c10e9b0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f482c009ca0 tx=0x7f482c02c230 comp rx=0 tx=0).stop 2024-04-21T08:57:04.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.893+0000 7f4843304640 1 -- 172.21.15.28:0/798660579 shutdown_connections 2024-04-21T08:57:04.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.893+0000 7f4843304640 1 --2- 172.21.15.28:0/798660579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c10c510 0x7f483c10e9b0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.893+0000 7f4843304640 1 --2- 172.21.15.28:0/798660579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f483c103af0 0x7f483c10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.893+0000 7f4843304640 1 --2- 172.21.15.28:0/798660579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f483c1031d0 0x7f483c1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.893+0000 7f4843304640 1 -- 172.21.15.28:0/798660579 >> 172.21.15.28:0/798660579 conn(0x7f483c077f50 msgr2=0x7f483c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:04.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.893+0000 7f4843304640 1 -- 172.21.15.28:0/798660579 shutdown_connections 2024-04-21T08:57:04.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.893+0000 7f4843304640 1 -- 172.21.15.28:0/798660579 wait complete. 2024-04-21T08:57:04.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.893+0000 7f4843304640 1 Processor -- start 2024-04-21T08:57:04.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f4843304640 1 -- start start 2024-04-21T08:57:04.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f4843304640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f483c1031d0 0x7f483c100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f4843304640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c103af0 0x7f483c101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f4843304640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f483c10c510 0x7f483c1018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f4843304640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f483c1121a0 con 0x7f483c103af0 2024-04-21T08:57:04.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f4843304640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f483c112020 con 0x7f483c1031d0 2024-04-21T08:57:04.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f4843304640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f483c112320 con 0x7f483c10c510 2024-04-21T08:57:04.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f484089f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c103af0 0x7f483c101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f484089f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c103af0 0x7f483c101360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56672/0 (socket says 172.21.15.28:56672) 2024-04-21T08:57:04.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f484089f640 1 -- 172.21.15.28:0/3882882890 learned_addr learned my addr 172.21.15.28:0/3882882890 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:04.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f48418a1640 1 --2- 172.21.15.28:0/3882882890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f483c10c510 0x7f483c1018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f48410a0640 1 --2- 172.21.15.28:0/3882882890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f483c1031d0 0x7f483c100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f484089f640 1 -- 172.21.15.28:0/3882882890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f483c10c510 msgr2=0x7f483c1018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f484089f640 1 --2- 172.21.15.28:0/3882882890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f483c10c510 0x7f483c1018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f484089f640 1 -- 172.21.15.28:0/3882882890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f483c1031d0 msgr2=0x7f483c100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f484089f640 1 --2- 172.21.15.28:0/3882882890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f483c1031d0 0x7f483c100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f484089f640 1 -- 172.21.15.28:0/3882882890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f483c1a6be0 con 0x7f483c103af0 2024-04-21T08:57:04.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.894+0000 7f484089f640 1 --2- 172.21.15.28:0/3882882890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c103af0 0x7f483c101360 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f483000d770 tx=0x7f483000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:04.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.895+0000 7f483a7fc640 1 -- 172.21.15.28:0/3882882890 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830017070 con 0x7f483c103af0 2024-04-21T08:57:04.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.895+0000 7f483a7fc640 1 -- 172.21.15.28:0/3882882890 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48300054d0 con 0x7f483c103af0 2024-04-21T08:57:04.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.895+0000 7f483a7fc640 1 -- 172.21.15.28:0/3882882890 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830004c30 con 0x7f483c103af0 2024-04-21T08:57:04.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.895+0000 7f4843304640 1 -- 172.21.15.28:0/3882882890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f483c1a6ed0 con 0x7f483c103af0 2024-04-21T08:57:04.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.895+0000 7f4843304640 1 -- 172.21.15.28:0/3882882890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f483c1a7340 con 0x7f483c103af0 2024-04-21T08:57:04.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.897+0000 7f483a7fc640 1 -- 172.21.15.28:0/3882882890 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4830004df0 con 0x7f483c103af0 2024-04-21T08:57:04.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f4845fd5740 1 Processor -- start 2024-04-21T08:57:04.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f4845fd5740 1 -- start start 2024-04-21T08:57:04.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f4845fd5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a358e0 0x564c51a35cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f4845fd5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a36290 0x564c51a3e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f4845fd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51a41260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f4845fd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564c51999f00 con 0x564c51a3ee10 2024-04-21T08:57:04.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f4845fd5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564c51937590 con 0x564c51a36290 2024-04-21T08:57:04.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f4845fd5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564c51995ec0 con 0x564c51a358e0 2024-04-21T08:57:04.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f483a7fc640 1 --2- 172.21.15.28:0/3882882890 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48240a3d20 0x7f48240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f484089f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51a41260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f484089f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51a41260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56682/0 (socket says 172.21.15.28:56682) 2024-04-21T08:57:04.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f484089f640 1 -- 172.21.15.28:0/2203168611 learned_addr learned my addr 172.21.15.28:0/2203168611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:04.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f48418a1640 1 --2- 172.21.15.28:0/3882882890 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48240a3d20 0x7f48240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f48410a0640 1 --2- 172.21.15.28:0/2203168611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a36290 0x564c51a3e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f483a7fc640 1 -- 172.21.15.28:0/3882882890 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f48300d0600 con 0x7f483c103af0 2024-04-21T08:57:04.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f48418a1640 1 --2- 172.21.15.28:0/2203168611 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a358e0 0x564c51a35cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f484089f640 1 -- 172.21.15.28:0/2203168611 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a358e0 msgr2=0x564c51a35cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:04.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f484089f640 1 --2- 172.21.15.28:0/2203168611 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a358e0 0x564c51a35cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f484089f640 1 -- 172.21.15.28:0/2203168611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a36290 msgr2=0x564c51a3e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:04.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.898+0000 7f484089f640 1 --2- 172.21.15.28:0/2203168611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a36290 0x564c51a3e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.899+0000 7f484089f640 1 -- 172.21.15.28:0/2203168611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c517747e0 con 0x564c51a3ee10 2024-04-21T08:57:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.899+0000 7f484089f640 1 --2- 172.21.15.28:0/2203168611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51a41260 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f48300cda10 tx=0x7f4830009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda45a096b97b2f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.899+0000 7f48418a1640 1 --2- 172.21.15.28:0/3882882890 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48240a3d20 0x7f48240a61e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f482c004af0 tx=0x7f482c02e040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.899+0000 7f48237fe640 1 -- 172.21.15.28:0/2203168611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830020040 con 0x564c51a3ee10 2024-04-21T08:57:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.899+0000 7f48237fe640 1 -- 172.21.15.28:0/2203168611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f483009c200 con 0x564c51a3ee10 2024-04-21T08:57:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.899+0000 7f48237fe640 1 -- 172.21.15.28:0/2203168611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830098870 con 0x564c51a3ee10 2024-04-21T08:57:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.900+0000 7f4845fd5740 1 -- 172.21.15.28:0/2203168611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 msgr2=0x564c51a41260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.900+0000 7f4845fd5740 1 --2- 172.21.15.28:0/2203168611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51a41260 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f48300cda10 tx=0x7f4830009f90 comp rx=0 tx=0).stop 2024-04-21T08:57:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.900+0000 7f4845fd5740 1 -- 172.21.15.28:0/2203168611 shutdown_connections 2024-04-21T08:57:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.900+0000 7f4845fd5740 1 --2- 172.21.15.28:0/2203168611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51a41260 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.900+0000 7f4845fd5740 1 --2- 172.21.15.28:0/2203168611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a36290 0x564c51a3e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.900+0000 7f4845fd5740 1 --2- 172.21.15.28:0/2203168611 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a358e0 0x564c51a35cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.900+0000 7f4845fd5740 1 -- 172.21.15.28:0/2203168611 >> 172.21.15.28:0/2203168611 conn(0x564c51a2ffb0 msgr2=0x564c51a30390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.900+0000 7f4845fd5740 1 -- 172.21.15.28:0/2203168611 shutdown_connections 2024-04-21T08:57:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.900+0000 7f4845fd5740 1 -- 172.21.15.28:0/2203168611 wait complete. 2024-04-21T08:57:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.900+0000 7f4845fd5740 1 -- start start 2024-04-21T08:57:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.900+0000 7f4845fd5740 1 Processor -- start 2024-04-21T08:57:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f4845fd5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a358e0 0x564c51a3bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f4845fd5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a36290 0x564c51a3c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f4845fd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51a3c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f4845fd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564c51a41d80 con 0x564c51a3ee10 2024-04-21T08:57:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f4845fd5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564c51953580 con 0x564c51a358e0 2024-04-21T08:57:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f4845fd5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564c51a41f00 con 0x564c51a36290 2024-04-21T08:57:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f4845fd5740 10 client.0 _ll_register_callbacks cb 0x564c51959650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f48418a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51a3c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f48418a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51a3c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56698/0 (socket says 172.21.15.28:56698) 2024-04-21T08:57:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f48410a0640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a358e0 0x564c51a3bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f48418a1640 1 -- 172.21.15.28:0/3979244865 learned_addr learned my addr 172.21.15.28:0/3979244865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f484089f640 1 --2- 172.21.15.28:0/3979244865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a36290 0x564c51a3c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f48418a1640 1 -- 172.21.15.28:0/3979244865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a36290 msgr2=0x564c51a3c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f48418a1640 1 --2- 172.21.15.28:0/3979244865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a36290 0x564c51a3c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f48418a1640 1 -- 172.21.15.28:0/3979244865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a358e0 msgr2=0x564c51a3bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f48418a1640 1 --2- 172.21.15.28:0/3979244865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a358e0 0x564c51a3bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f48418a1640 1 -- 172.21.15.28:0/3979244865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c517d4f70 con 0x564c51a3ee10 2024-04-21T08:57:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f48418a1640 1 --2- 172.21.15.28:0/3979244865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51a3c860 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f482c036000 tx=0x7f482c02c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96660fc7b3964da1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f4820ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f4820ff9640 1 -- 172.21.15.28:0/3979244865 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482c051070 con 0x564c51a3ee10 2024-04-21T08:57:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.901+0000 7f4820ff9640 1 -- 172.21.15.28:0/3979244865 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f482c034bd0 con 0x564c51a3ee10 2024-04-21T08:57:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.902+0000 7f4845fd5740 1 -- 172.21.15.28:0/3979244865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f483009c200 con 0x564c51a3ee10 2024-04-21T08:57:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.902+0000 7f4820ff9640 1 -- 172.21.15.28:0/3979244865 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f482c05d050 con 0x564c51a3ee10 2024-04-21T08:57:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.902+0000 7f4845fd5740 10 client.17301 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.902+0000 7f4845fd5740 1 -- 172.21.15.28:0/3979244865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564c5166c170 con 0x564c51a3ee10 2024-04-21T08:57:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.902+0000 7f4820ff9640 1 -- 172.21.15.28:0/3979244865 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f483009c200 con 0x564c51a3ee10 2024-04-21T08:57:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.902+0000 7f4820ff9640 1 -- 172.21.15.28:0/3979244865 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f482c04c070 con 0x564c51a3ee10 2024-04-21T08:57:04.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.902+0000 7f4845fd5740 10 client.17301 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:04.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.902+0000 7f4845fd5740 10 client.17301 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:04.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.902+0000 7f4845fd5740 20 client.17301 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:04.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.902+0000 7f4845fd5740 1 --2- 172.21.15.28:0/3979244865 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564c51a4f330 0x564c51a51780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.902+0000 7f4845fd5740 4 client.17301 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:57:04.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.902+0000 7f4845fd5740 1 -- 172.21.15.28:0/3979244865 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c519a7920 con 0x564c51a4f330 2024-04-21T08:57:04.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.902+0000 7f48410a0640 1 --2- 172.21.15.28:0/3979244865 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564c51a4f330 0x564c51a51780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.903+0000 7f48410a0640 1 --2- 172.21.15.28:0/3979244865 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564c51a4f330 0x564c51a51780 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5b92aa3285a734c server_cookie=9839d97b9b0577e3 in_seq=0 out_seq=0 2024-04-21T08:57:04.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.903+0000 7f4820ff9640 10 client.17301 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:04.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.904+0000 7f4820ff9640 1 -- 172.21.15.28:0/3979244865 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564c519a7920 con 0x564c51a4f330 2024-04-21T08:57:04.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.904+0000 7f4820ff9640 10 client.17301 handle_command_reply: tid=0 2024-04-21T08:57:04.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.904+0000 7f4845fd5740 1 client.17301 shutdown 2024-04-21T08:57:04.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.904+0000 7f4845fd5740 1 -- 172.21.15.28:0/3979244865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 msgr2=0x564c51a3c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.904+0000 7f4845fd5740 1 --2- 172.21.15.28:0/3979244865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51a3c860 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f482c036000 tx=0x7f482c02c230 comp rx=0 tx=0).stop 2024-04-21T08:57:04.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.904+0000 7f4845fd5740 1 -- 172.21.15.28:0/3979244865 shutdown_connections 2024-04-21T08:57:04.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.904+0000 7f4845fd5740 1 --2- 172.21.15.28:0/3979244865 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564c51a4f330 0x564c51a51780 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.904+0000 7f4845fd5740 1 --2- 172.21.15.28:0/3979244865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51a3c860 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.904+0000 7f4845fd5740 1 --2- 172.21.15.28:0/3979244865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a36290 0x564c51a3c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.904+0000 7f4820ff9640 0 client.17301 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.904+0000 7f4845fd5740 1 --2- 172.21.15.28:0/3979244865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a358e0 0x564c51a3bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.904+0000 7f4845fd5740 1 -- 172.21.15.28:0/3979244865 >> 172.21.15.28:0/3979244865 conn(0x564c51a2ffb0 msgr2=0x564c51a31d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.904+0000 7f4845fd5740 1 -- 172.21.15.28:0/3979244865 shutdown_connections 2024-04-21T08:57:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.904+0000 7f4845fd5740 1 -- 172.21.15.28:0/3979244865 wait complete. 2024-04-21T08:57:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.904+0000 7f4845fd5740 20 client.17301 trim_cache size 0 max 16384 2024-04-21T08:57:04.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f4845fd5740 1 Processor -- start 2024-04-21T08:57:04.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f4845fd5740 1 -- start start 2024-04-21T08:57:04.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f4845fd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a358e0 0x564c51a3e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f4845fd5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a36290 0x564c51a4ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f4845fd5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a3ee10 0x564c51a57050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f4845fd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f483c1b3d50 con 0x564c51a358e0 2024-04-21T08:57:04.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f4845fd5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f483c1b1620 con 0x564c51a3ee10 2024-04-21T08:57:04.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f4845fd5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f483c1af470 con 0x564c51a36290 2024-04-21T08:57:04.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f48410a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a358e0 0x564c51a3e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f48410a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a358e0 0x564c51a3e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56700/0 (socket says 172.21.15.28:56700) 2024-04-21T08:57:04.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f48410a0640 1 -- 172.21.15.28:0/4251800477 learned_addr learned my addr 172.21.15.28:0/4251800477 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:04.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f484089f640 1 --2- 172.21.15.28:0/4251800477 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a36290 0x564c51a4ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f48410a0640 1 -- 172.21.15.28:0/4251800477 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a36290 msgr2=0x564c51a4ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f48410a0640 1 --2- 172.21.15.28:0/4251800477 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a36290 0x564c51a4ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f48410a0640 1 -- 172.21.15.28:0/4251800477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a3ee10 msgr2=0x564c51a57050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f48410a0640 1 --2- 172.21.15.28:0/4251800477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a3ee10 0x564c51a57050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.922+0000 7f48410a0640 1 -- 172.21.15.28:0/4251800477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c51a41570 con 0x564c51a358e0 2024-04-21T08:57:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.923+0000 7f48410a0640 1 --2- 172.21.15.28:0/4251800477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a358e0 0x564c51a3e510 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f4834000c00 tx=0x7f4834004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f2ee9b8652b573f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.923+0000 7f4822ffd640 1 -- 172.21.15.28:0/4251800477 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483401d420 con 0x564c51a358e0 2024-04-21T08:57:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.923+0000 7f4822ffd640 1 -- 172.21.15.28:0/4251800477 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4834022070 con 0x564c51a358e0 2024-04-21T08:57:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.923+0000 7f4822ffd640 1 -- 172.21.15.28:0/4251800477 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4834011d70 con 0x564c51a358e0 2024-04-21T08:57:04.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.923+0000 7f4845fd5740 1 -- 172.21.15.28:0/4251800477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a358e0 msgr2=0x564c51a3e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.923+0000 7f4845fd5740 1 --2- 172.21.15.28:0/4251800477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a358e0 0x564c51a3e510 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f4834000c00 tx=0x7f4834004580 comp rx=0 tx=0).stop 2024-04-21T08:57:04.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.923+0000 7f4845fd5740 1 -- 172.21.15.28:0/4251800477 shutdown_connections 2024-04-21T08:57:04.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.923+0000 7f4845fd5740 1 --2- 172.21.15.28:0/4251800477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a3ee10 0x564c51a57050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.923+0000 7f4845fd5740 1 --2- 172.21.15.28:0/4251800477 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a36290 0x564c51a4ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.923+0000 7f4845fd5740 1 --2- 172.21.15.28:0/4251800477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a358e0 0x564c51a3e510 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.923+0000 7f4845fd5740 1 -- 172.21.15.28:0/4251800477 >> 172.21.15.28:0/4251800477 conn(0x564c51a2ffb0 msgr2=0x564c51a3c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.923+0000 7f4845fd5740 1 -- 172.21.15.28:0/4251800477 shutdown_connections 2024-04-21T08:57:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.923+0000 7f4845fd5740 1 -- 172.21.15.28:0/4251800477 wait complete. 2024-04-21T08:57:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.924+0000 7f4845fd5740 1 -- start start 2024-04-21T08:57:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.924+0000 7f4845fd5740 1 Processor -- start 2024-04-21T08:57:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.924+0000 7f4845fd5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a358e0 0x564c51a433b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.924+0000 7f4845fd5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a36290 0x564c51a438f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.924+0000 7f4845fd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51aae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.924+0000 7f4845fd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f483c1afae0 con 0x564c51a3ee10 2024-04-21T08:57:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.924+0000 7f4845fd5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f483c1b0f50 con 0x564c51a36290 2024-04-21T08:57:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.924+0000 7f4845fd5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f483c1b01b0 con 0x564c51a358e0 2024-04-21T08:57:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.924+0000 7f4845fd5740 10 client.0 _ll_register_callbacks cb 0x7f482c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.924+0000 7f48418a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51aae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.924+0000 7f48418a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51aae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56708/0 (socket says 172.21.15.28:56708) 2024-04-21T08:57:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.924+0000 7f48418a1640 1 -- 172.21.15.28:0/2669098424 learned_addr learned my addr 172.21.15.28:0/2669098424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.924+0000 7f48410a0640 1 --2- 172.21.15.28:0/2669098424 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a358e0 0x564c51a433b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f484089f640 1 --2- 172.21.15.28:0/2669098424 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a36290 0x564c51a438f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f48418a1640 1 -- 172.21.15.28:0/2669098424 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a358e0 msgr2=0x564c51a433b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f48418a1640 1 --2- 172.21.15.28:0/2669098424 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a358e0 0x564c51a433b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f48418a1640 1 -- 172.21.15.28:0/2669098424 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a36290 msgr2=0x564c51a438f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f48418a1640 1 --2- 172.21.15.28:0/2669098424 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a36290 0x564c51a438f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f48418a1640 1 -- 172.21.15.28:0/2669098424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c51aaee80 con 0x564c51a3ee10 2024-04-21T08:57:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f484089f640 1 --2- 172.21.15.28:0/2669098424 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a36290 0x564c51a438f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f48418a1640 1 --2- 172.21.15.28:0/2669098424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51aae890 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f482c040eb0 tx=0x7f482c0437d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f50456e60e8bee server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f4820ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f4820ff9640 1 -- 172.21.15.28:0/2669098424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482c05a070 con 0x564c51a3ee10 2024-04-21T08:57:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f4820ff9640 1 -- 172.21.15.28:0/2669098424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f482c034de0 con 0x564c51a3ee10 2024-04-21T08:57:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f4845fd5740 1 -- 172.21.15.28:0/2669098424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4834022070 con 0x564c51a3ee10 2024-04-21T08:57:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f4820ff9640 1 -- 172.21.15.28:0/2669098424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f482c051070 con 0x564c51a3ee10 2024-04-21T08:57:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f4845fd5740 10 client.17307 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f4845fd5740 1 -- 172.21.15.28:0/2669098424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564c51ab62a0 con 0x564c51a3ee10 2024-04-21T08:57:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.925+0000 7f4820ff9640 1 -- 172.21.15.28:0/2669098424 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4834022070 con 0x564c51a3ee10 2024-04-21T08:57:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.926+0000 7f4820ff9640 1 -- 172.21.15.28:0/2669098424 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f482c04c040 con 0x564c51a3ee10 2024-04-21T08:57:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.926+0000 7f4845fd5740 10 client.17307 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.926+0000 7f4845fd5740 10 client.17307 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.926+0000 7f4845fd5740 20 client.17307 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.926+0000 7f4845fd5740 1 --2- 172.21.15.28:0/2669098424 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564c51ab6a40 0x564c51ab8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.926+0000 7f4845fd5740 4 client.17307 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:57:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.926+0000 7f4845fd5740 1 -- 172.21.15.28:0/2669098424 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564c519a1790 con 0x564c51ab6a40 2024-04-21T08:57:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.926+0000 7f48410a0640 1 --2- 172.21.15.28:0/2669098424 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564c51ab6a40 0x564c51ab8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.926+0000 7f48410a0640 1 --2- 172.21.15.28:0/2669098424 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564c51ab6a40 0x564c51ab8e60 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55233eac25437fa5 server_cookie=1643176bc46161b5 in_seq=0 out_seq=0 2024-04-21T08:57:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.926+0000 7f4820ff9640 10 client.17307 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.927+0000 7f4820ff9640 1 -- 172.21.15.28:0/2669098424 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564c519a1790 con 0x564c51ab6a40 2024-04-21T08:57:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.927+0000 7f4820ff9640 10 client.17307 handle_command_reply: tid=0 2024-04-21T08:57:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.927+0000 7f4845fd5740 1 client.17307 shutdown 2024-04-21T08:57:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.927+0000 7f4845fd5740 1 -- 172.21.15.28:0/2669098424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 msgr2=0x564c51aae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.927+0000 7f4845fd5740 1 --2- 172.21.15.28:0/2669098424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51aae890 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f482c040eb0 tx=0x7f482c0437d0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4845fd5740 1 -- 172.21.15.28:0/2669098424 shutdown_connections 2024-04-21T08:57:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4845fd5740 1 --2- 172.21.15.28:0/2669098424 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564c51ab6a40 0x564c51ab8e60 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4845fd5740 1 --2- 172.21.15.28:0/2669098424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c51a3ee10 0x564c51aae890 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4845fd5740 1 --2- 172.21.15.28:0/2669098424 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564c51a36290 0x564c51a438f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4845fd5740 1 --2- 172.21.15.28:0/2669098424 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564c51a358e0 0x564c51a433b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4820ff9640 0 client.17307 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:04.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4845fd5740 1 -- 172.21.15.28:0/2669098424 >> 172.21.15.28:0/2669098424 conn(0x564c51a2ffb0 msgr2=0x564c51a2e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:04.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4845fd5740 1 -- 172.21.15.28:0/2669098424 shutdown_connections 2024-04-21T08:57:04.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4845fd5740 1 -- 172.21.15.28:0/2669098424 wait complete. 2024-04-21T08:57:04.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4845fd5740 20 client.17307 trim_cache size 0 max 16384 2024-04-21T08:57:04.939 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:57:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:57:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "f827cb55-f8a5-4a09-b796-187b09487a19", 2024-04-21T08:57:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:57:04.939 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:57:04.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4843304640 1 -- 172.21.15.28:0/3882882890 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48240a3d20 msgr2=0x7f48240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4843304640 1 --2- 172.21.15.28:0/3882882890 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48240a3d20 0x7f48240a61e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f482c004af0 tx=0x7f482c02e040 comp rx=0 tx=0).stop 2024-04-21T08:57:04.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4843304640 1 -- 172.21.15.28:0/3882882890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c103af0 msgr2=0x7f483c101360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4843304640 1 --2- 172.21.15.28:0/3882882890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c103af0 0x7f483c101360 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f483000d770 tx=0x7f483000dc30 comp rx=0 tx=0).stop 2024-04-21T08:57:04.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4843304640 1 -- 172.21.15.28:0/3882882890 shutdown_connections 2024-04-21T08:57:04.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4843304640 1 --2- 172.21.15.28:0/3882882890 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f48240a3d20 0x7f48240a61e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4843304640 1 --2- 172.21.15.28:0/3882882890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f483c10c510 0x7f483c1018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4843304640 1 --2- 172.21.15.28:0/3882882890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c103af0 0x7f483c101360 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4843304640 1 --2- 172.21.15.28:0/3882882890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f483c1031d0 0x7f483c100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.928+0000 7f4843304640 1 -- 172.21.15.28:0/3882882890 >> 172.21.15.28:0/3882882890 conn(0x7f483c077f50 msgr2=0x7f483c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:04.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.929+0000 7f4843304640 1 -- 172.21.15.28:0/3882882890 shutdown_connections 2024-04-21T08:57:04.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:04.929+0000 7f4843304640 1 -- 172.21.15.28:0/3882882890 wait complete. 2024-04-21T08:57:04.942 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:57:04.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:57:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.051+0000 7f892219f640 1 -- 172.21.15.28:0/3578015230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891c103850 msgr2=0x7f891c10bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.051+0000 7f892219f640 1 --2- 172.21.15.28:0/3578015230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891c103850 0x7f891c10bd30 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f890c009900 tx=0x7f890c02bcd0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.051+0000 7f892219f640 1 -- 172.21.15.28:0/3578015230 shutdown_connections 2024-04-21T08:57:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.051+0000 7f892219f640 1 --2- 172.21.15.28:0/3578015230 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f891c10c300 0x7f891c10e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.051+0000 7f892219f640 1 --2- 172.21.15.28:0/3578015230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891c103850 0x7f891c10bd30 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.051+0000 7f892219f640 1 --2- 172.21.15.28:0/3578015230 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f891c102f30 0x7f891c103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.051+0000 7f892219f640 1 -- 172.21.15.28:0/3578015230 >> 172.21.15.28:0/3578015230 conn(0x7f891c077a30 msgr2=0x7f891c0fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.051+0000 7f892219f640 1 -- 172.21.15.28:0/3578015230 shutdown_connections 2024-04-21T08:57:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.051+0000 7f892219f640 1 -- 172.21.15.28:0/3578015230 wait complete. 2024-04-21T08:57:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.051+0000 7f892219f640 1 Processor -- start 2024-04-21T08:57:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.051+0000 7f892219f640 1 -- start start 2024-04-21T08:57:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892219f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891c102f30 0x7f891c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892219f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f891c103850 0x7f891c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892219f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f891c10c300 0x7f891c19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892219f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f891c111f50 con 0x7f891c102f30 2024-04-21T08:57:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892219f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f891c111dd0 con 0x7f891c10c300 2024-04-21T08:57:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892219f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f891c1120d0 con 0x7f891c103850 2024-04-21T08:57:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892119d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891c102f30 0x7f891c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892119d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891c102f30 0x7f891c198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56724/0 (socket says 172.21.15.28:56724) 2024-04-21T08:57:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892119d640 1 -- 172.21.15.28:0/893503421 learned_addr learned my addr 172.21.15.28:0/893503421 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892099c640 1 --2- 172.21.15.28:0/893503421 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f891c103850 0x7f891c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892199e640 1 --2- 172.21.15.28:0/893503421 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f891c10c300 0x7f891c19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892119d640 1 -- 172.21.15.28:0/893503421 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f891c103850 msgr2=0x7f891c198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892119d640 1 --2- 172.21.15.28:0/893503421 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f891c103850 0x7f891c198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892119d640 1 -- 172.21.15.28:0/893503421 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f891c10c300 msgr2=0x7f891c19cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892119d640 1 --2- 172.21.15.28:0/893503421 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f891c10c300 0x7f891c19cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892119d640 1 -- 172.21.15.28:0/893503421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f891c19d4b0 con 0x7f891c102f30 2024-04-21T08:57:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892199e640 1 --2- 172.21.15.28:0/893503421 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f891c10c300 0x7f891c19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:57:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f892119d640 1 --2- 172.21.15.28:0/893503421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891c102f30 0x7f891c198510 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f891800db80 tx=0x7f891800b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.052+0000 7f89127fc640 1 -- 172.21.15.28:0/893503421 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8918017070 con 0x7f891c102f30 2024-04-21T08:57:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.053+0000 7f892219f640 1 -- 172.21.15.28:0/893503421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f891c19d7a0 con 0x7f891c102f30 2024-04-21T08:57:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.053+0000 7f89127fc640 1 -- 172.21.15.28:0/893503421 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8918003b70 con 0x7f891c102f30 2024-04-21T08:57:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.053+0000 7f89127fc640 1 -- 172.21.15.28:0/893503421 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89180106c0 con 0x7f891c102f30 2024-04-21T08:57:05.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.053+0000 7f892219f640 1 -- 172.21.15.28:0/893503421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f891c1a5030 con 0x7f891c102f30 2024-04-21T08:57:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.055+0000 7f89127fc640 1 -- 172.21.15.28:0/893503421 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8918004020 con 0x7f891c102f30 2024-04-21T08:57:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.055+0000 7f8925ff0740 1 Processor -- start 2024-04-21T08:57:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.055+0000 7f8925ff0740 1 -- start start 2024-04-21T08:57:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.055+0000 7f8925ff0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230b40f0 0x5613230b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.055+0000 7f89127fc640 1 --2- 172.21.15.28:0/893503421 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f89080a3d20 0x7f89080a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f8925ff0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b4aa0 0x5613230c1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f8925ff0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230c2020 0x5613230c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f8925ff0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5613230191e0 con 0x5613230b4aa0 2024-04-21T08:57:05.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f8925ff0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561322fb6390 con 0x5613230c2020 2024-04-21T08:57:05.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f8925ff0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5613230151a0 con 0x5613230b40f0 2024-04-21T08:57:05.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f892099c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b4aa0 0x5613230c1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f892119d640 1 --2- 172.21.15.28:0/893503421 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f89080a3d20 0x7f89080a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f892099c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b4aa0 0x5613230c1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56730/0 (socket says 172.21.15.28:56730) 2024-04-21T08:57:05.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f892099c640 1 -- 172.21.15.28:0/1270710894 learned_addr learned my addr 172.21.15.28:0/1270710894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:05.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f892199e640 1 --2- 172.21.15.28:0/1270710894 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230c2020 0x5613230c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f89127fc640 1 -- 172.21.15.28:0/893503421 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f89180d0be0 con 0x7f891c102f30 2024-04-21T08:57:05.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f892199e640 1 --2- 172.21.15.28:0/1270710894 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230b40f0 0x5613230b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f892099c640 1 -- 172.21.15.28:0/1270710894 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230b40f0 msgr2=0x5613230b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:05.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f892099c640 1 --2- 172.21.15.28:0/1270710894 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230b40f0 0x5613230b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f892099c640 1 -- 172.21.15.28:0/1270710894 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230c2020 msgr2=0x5613230c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:05.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f892099c640 1 --2- 172.21.15.28:0/1270710894 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230c2020 0x5613230c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f892099c640 1 -- 172.21.15.28:0/1270710894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561322df57d0 con 0x5613230b4aa0 2024-04-21T08:57:05.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f892199e640 1 --2- 172.21.15.28:0/1270710894 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230b40f0 0x5613230b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f892119d640 1 --2- 172.21.15.28:0/893503421 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f89080a3d20 0x7f89080a61e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f89180ce040 tx=0x7f8918098b00 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f892099c640 1 --2- 172.21.15.28:0/1270710894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b4aa0 0x5613230c1ae0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f890c009900 tx=0x7f890c02c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d18b19b0c0dfa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.056+0000 7f89077fe640 1 -- 172.21.15.28:0/1270710894 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890c004180 con 0x5613230b4aa0 2024-04-21T08:57:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.057+0000 7f89077fe640 1 -- 172.21.15.28:0/1270710894 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f890c0054f0 con 0x5613230b4aa0 2024-04-21T08:57:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.057+0000 7f89077fe640 1 -- 172.21.15.28:0/1270710894 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890c0046c0 con 0x5613230b4aa0 2024-04-21T08:57:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.057+0000 7f8925ff0740 1 -- 172.21.15.28:0/1270710894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b4aa0 msgr2=0x5613230c1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.057+0000 7f8925ff0740 1 --2- 172.21.15.28:0/1270710894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b4aa0 0x5613230c1ae0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f890c009900 tx=0x7f890c02c760 comp rx=0 tx=0).stop 2024-04-21T08:57:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.057+0000 7f8925ff0740 1 -- 172.21.15.28:0/1270710894 shutdown_connections 2024-04-21T08:57:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.057+0000 7f8925ff0740 1 --2- 172.21.15.28:0/1270710894 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230c2020 0x5613230c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.057+0000 7f8925ff0740 1 --2- 172.21.15.28:0/1270710894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b4aa0 0x5613230c1ae0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.057+0000 7f8925ff0740 1 --2- 172.21.15.28:0/1270710894 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230b40f0 0x5613230b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.057+0000 7f8925ff0740 1 -- 172.21.15.28:0/1270710894 >> 172.21.15.28:0/1270710894 conn(0x5613230aee40 msgr2=0x5613230af220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.057+0000 7f8925ff0740 1 -- 172.21.15.28:0/1270710894 shutdown_connections 2024-04-21T08:57:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.057+0000 7f8925ff0740 1 -- 172.21.15.28:0/1270710894 wait complete. 2024-04-21T08:57:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f8925ff0740 1 -- start start 2024-04-21T08:57:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f8925ff0740 1 Processor -- start 2024-04-21T08:57:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f8925ff0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b40f0 0x5613230bef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f8925ff0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230b4aa0 0x5613230bf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f8925ff0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230c2020 0x5613230bf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f8925ff0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5613230c4d80 con 0x5613230b40f0 2024-04-21T08:57:05.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f8925ff0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x561322fd22f0 con 0x5613230c2020 2024-04-21T08:57:05.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f8925ff0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5613230c4f00 con 0x5613230b4aa0 2024-04-21T08:57:05.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f8925ff0740 10 client.0 _ll_register_callbacks cb 0x561322fdf580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:05.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f892199e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b40f0 0x5613230bef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f892199e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b40f0 0x5613230bef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56736/0 (socket says 172.21.15.28:56736) 2024-04-21T08:57:05.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f892199e640 1 -- 172.21.15.28:0/1930876010 learned_addr learned my addr 172.21.15.28:0/1930876010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:05.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f892199e640 1 --2- 172.21.15.28:0/1930876010 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230c2020 0x5613230bf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f892099c640 1 --2- 172.21.15.28:0/1930876010 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230b4aa0 0x5613230bf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f892199e640 1 -- 172.21.15.28:0/1930876010 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230b4aa0 msgr2=0x5613230bf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:05.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f892199e640 1 --2- 172.21.15.28:0/1930876010 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230b4aa0 0x5613230bf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f892199e640 1 -- 172.21.15.28:0/1930876010 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230c2020 msgr2=0x5613230bf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:05.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f892199e640 1 --2- 172.21.15.28:0/1930876010 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230c2020 0x5613230bf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f892199e640 1 -- 172.21.15.28:0/1930876010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561322d2b3f0 con 0x5613230b40f0 2024-04-21T08:57:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.058+0000 7f892099c640 1 --2- 172.21.15.28:0/1930876010 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230b4aa0 0x5613230bf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f892199e640 1 --2- 172.21.15.28:0/1930876010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b40f0 0x5613230bef50 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f8914014e60 tx=0x7f891400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58e32a9df0abca0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f8904ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f8904ff9640 1 -- 172.21.15.28:0/1930876010 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8914011e90 con 0x5613230b40f0 2024-04-21T08:57:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f8925ff0740 1 -- 172.21.15.28:0/1930876010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f890c0054f0 con 0x5613230b40f0 2024-04-21T08:57:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f8904ff9640 1 -- 172.21.15.28:0/1930876010 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f8914012450 con 0x5613230b40f0 2024-04-21T08:57:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f8904ff9640 1 -- 172.21.15.28:0/1930876010 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8914026050 con 0x5613230b40f0 2024-04-21T08:57:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f8925ff0740 10 client.17319 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:05.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f8925ff0740 1 -- 172.21.15.28:0/1930876010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561322ceb0f0 con 0x5613230b40f0 2024-04-21T08:57:05.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f8904ff9640 1 -- 172.21.15.28:0/1930876010 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f890c0054f0 con 0x5613230b40f0 2024-04-21T08:57:05.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f8904ff9640 1 -- 172.21.15.28:0/1930876010 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f891401f060 con 0x5613230b40f0 2024-04-21T08:57:05.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f8925ff0740 10 client.17319 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:05.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f8925ff0740 10 client.17319 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:05.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f8925ff0740 20 client.17319 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:05.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f8925ff0740 1 --2- 172.21.15.28:0/1930876010 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5613230d2010 0x5613230d4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f8925ff0740 4 client.17319 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:57:05.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.059+0000 7f8925ff0740 1 -- 172.21.15.28:0/1930876010 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561323026c00 con 0x5613230d2010 2024-04-21T08:57:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.060+0000 7f892099c640 1 --2- 172.21.15.28:0/1930876010 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5613230d2010 0x5613230d4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.060+0000 7f892099c640 1 --2- 172.21.15.28:0/1930876010 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5613230d2010 0x5613230d4470 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bb182dfd748d2cf server_cookie=31fd84842e1890bb in_seq=0 out_seq=0 2024-04-21T08:57:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.060+0000 7f8904ff9640 10 client.17319 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.061+0000 7f8904ff9640 1 -- 172.21.15.28:0/1930876010 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561323026c00 con 0x5613230d2010 2024-04-21T08:57:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.061+0000 7f8904ff9640 10 client.17319 handle_command_reply: tid=0 2024-04-21T08:57:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.061+0000 7f8925ff0740 1 client.17319 shutdown 2024-04-21T08:57:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.061+0000 7f8925ff0740 1 -- 172.21.15.28:0/1930876010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b40f0 msgr2=0x5613230bef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.061+0000 7f8925ff0740 1 --2- 172.21.15.28:0/1930876010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b40f0 0x5613230bef50 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f8914014e60 tx=0x7f891400c790 comp rx=0 tx=0).stop 2024-04-21T08:57:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.061+0000 7f8925ff0740 1 -- 172.21.15.28:0/1930876010 shutdown_connections 2024-04-21T08:57:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.061+0000 7f8925ff0740 1 --2- 172.21.15.28:0/1930876010 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5613230d2010 0x5613230d4470 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.061+0000 7f8925ff0740 1 --2- 172.21.15.28:0/1930876010 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230c2020 0x5613230bf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.061+0000 7f8925ff0740 1 --2- 172.21.15.28:0/1930876010 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230b4aa0 0x5613230bf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.061+0000 7f8904ff9640 0 client.17319 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.061+0000 7f8925ff0740 1 --2- 172.21.15.28:0/1930876010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b40f0 0x5613230bef50 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.061+0000 7f8925ff0740 1 -- 172.21.15.28:0/1930876010 >> 172.21.15.28:0/1930876010 conn(0x5613230aee40 msgr2=0x5613230ae120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.061+0000 7f8925ff0740 1 -- 172.21.15.28:0/1930876010 shutdown_connections 2024-04-21T08:57:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.061+0000 7f8925ff0740 1 -- 172.21.15.28:0/1930876010 wait complete. 2024-04-21T08:57:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.061+0000 7f8925ff0740 20 client.17319 trim_cache size 0 max 16384 2024-04-21T08:57:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.078+0000 7f8925ff0740 1 Processor -- start 2024-04-21T08:57:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.078+0000 7f8925ff0740 1 -- start start 2024-04-21T08:57:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f8925ff0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230b40f0 0x5613230c1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f8925ff0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b4aa0 0x5613230cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f8925ff0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230c2020 0x5613230bde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f8925ff0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f891c1af6f0 con 0x5613230b4aa0 2024-04-21T08:57:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f8925ff0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f891c1acfc0 con 0x5613230b40f0 2024-04-21T08:57:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f8925ff0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f891c1aae10 con 0x5613230c2020 2024-04-21T08:57:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f892099c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b4aa0 0x5613230cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f892099c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b4aa0 0x5613230cdd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56748/0 (socket says 172.21.15.28:56748) 2024-04-21T08:57:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f892099c640 1 -- 172.21.15.28:0/1659344339 learned_addr learned my addr 172.21.15.28:0/1659344339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f892099c640 1 -- 172.21.15.28:0/1659344339 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230c2020 msgr2=0x5613230bde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f892099c640 1 --2- 172.21.15.28:0/1659344339 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230c2020 0x5613230bde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f892099c640 1 -- 172.21.15.28:0/1659344339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230b40f0 msgr2=0x5613230c1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f892099c640 1 --2- 172.21.15.28:0/1659344339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230b40f0 0x5613230c1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f892099c640 1 -- 172.21.15.28:0/1659344339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613230b3800 con 0x5613230b4aa0 2024-04-21T08:57:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f892099c640 1 --2- 172.21.15.28:0/1659344339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b4aa0 0x5613230cdd10 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f890c02cb50 tx=0x7f890c02cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85910e4fd676f31 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f8906ffd640 1 -- 172.21.15.28:0/1659344339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890c004040 con 0x5613230b4aa0 2024-04-21T08:57:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f8906ffd640 1 -- 172.21.15.28:0/1659344339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f890c005500 con 0x5613230b4aa0 2024-04-21T08:57:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.079+0000 7f8906ffd640 1 -- 172.21.15.28:0/1659344339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890c004a20 con 0x5613230b4aa0 2024-04-21T08:57:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.080+0000 7f8925ff0740 1 -- 172.21.15.28:0/1659344339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b4aa0 msgr2=0x5613230cdd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.080+0000 7f8925ff0740 1 --2- 172.21.15.28:0/1659344339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b4aa0 0x5613230cdd10 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f890c02cb50 tx=0x7f890c02cb80 comp rx=0 tx=0).stop 2024-04-21T08:57:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.080+0000 7f8925ff0740 1 -- 172.21.15.28:0/1659344339 shutdown_connections 2024-04-21T08:57:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.080+0000 7f8925ff0740 1 --2- 172.21.15.28:0/1659344339 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230c2020 0x5613230bde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.080+0000 7f8925ff0740 1 --2- 172.21.15.28:0/1659344339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b4aa0 0x5613230cdd10 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.080+0000 7f8925ff0740 1 --2- 172.21.15.28:0/1659344339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230b40f0 0x5613230c1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.080+0000 7f8925ff0740 1 -- 172.21.15.28:0/1659344339 >> 172.21.15.28:0/1659344339 conn(0x5613230aee40 msgr2=0x5613230bd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.080+0000 7f8925ff0740 1 -- 172.21.15.28:0/1659344339 shutdown_connections 2024-04-21T08:57:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.080+0000 7f8925ff0740 1 -- 172.21.15.28:0/1659344339 wait complete. 2024-04-21T08:57:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.080+0000 7f8925ff0740 1 -- start start 2024-04-21T08:57:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.080+0000 7f8925ff0740 1 Processor -- start 2024-04-21T08:57:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f8925ff0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b40f0 0x5613230ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f8925ff0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230b4aa0 0x5613230c5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f8925ff0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230c2020 0x5613230c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f8925ff0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f891c1ab480 con 0x5613230b40f0 2024-04-21T08:57:05.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f8925ff0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f891c1ac8f0 con 0x5613230b4aa0 2024-04-21T08:57:05.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f8925ff0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f891c1abb50 con 0x5613230c2020 2024-04-21T08:57:05.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f8925ff0740 10 client.0 _ll_register_callbacks cb 0x7f891400c6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:05.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f892199e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b40f0 0x5613230ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f892199e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b40f0 0x5613230ca5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56752/0 (socket says 172.21.15.28:56752) 2024-04-21T08:57:05.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f892199e640 1 -- 172.21.15.28:0/673595754 learned_addr learned my addr 172.21.15.28:0/673595754 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:05.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f892099c640 1 --2- 172.21.15.28:0/673595754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230b4aa0 0x5613230c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f892199e640 1 --2- 172.21.15.28:0/673595754 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230c2020 0x5613230c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f892199e640 1 -- 172.21.15.28:0/673595754 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230c2020 msgr2=0x5613230c5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:05.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f892199e640 1 --2- 172.21.15.28:0/673595754 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230c2020 0x5613230c5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f892199e640 1 -- 172.21.15.28:0/673595754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230b4aa0 msgr2=0x5613230c5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:05.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f892199e640 1 --2- 172.21.15.28:0/673595754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230b4aa0 0x5613230c5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f892199e640 1 -- 172.21.15.28:0/673595754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613230c6670 con 0x5613230b40f0 2024-04-21T08:57:05.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f892199e640 1 --2- 172.21.15.28:0/673595754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b40f0 0x5613230ca5e0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f891400c810 tx=0x7f8914005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df98e0b8dd331652 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:05.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f8904ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:05.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f8904ff9640 1 -- 172.21.15.28:0/673595754 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8914030030 con 0x5613230b40f0 2024-04-21T08:57:05.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f8904ff9640 1 -- 172.21.15.28:0/673595754 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f8914014110 con 0x5613230b40f0 2024-04-21T08:57:05.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.081+0000 7f8925ff0740 1 -- 172.21.15.28:0/673595754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f890c005500 con 0x5613230b40f0 2024-04-21T08:57:05.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.082+0000 7f8904ff9640 1 -- 172.21.15.28:0/673595754 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f891403c050 con 0x5613230b40f0 2024-04-21T08:57:05.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.082+0000 7f8925ff0740 10 client.17325 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:05.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.082+0000 7f8925ff0740 1 -- 172.21.15.28:0/673595754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561323138610 con 0x5613230b40f0 2024-04-21T08:57:05.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.082+0000 7f8904ff9640 1 -- 172.21.15.28:0/673595754 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f890c005500 con 0x5613230b40f0 2024-04-21T08:57:05.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.082+0000 7f8904ff9640 1 -- 172.21.15.28:0/673595754 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f8914022ce0 con 0x5613230b40f0 2024-04-21T08:57:05.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.082+0000 7f8925ff0740 10 client.17325 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:05.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.082+0000 7f8925ff0740 10 client.17325 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:05.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.082+0000 7f8925ff0740 20 client.17325 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:05.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.082+0000 7f8925ff0740 1 --2- 172.21.15.28:0/673595754 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561323138c50 0x56132313b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:05.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.082+0000 7f8925ff0740 4 client.17325 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:57:05.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.082+0000 7f8925ff0740 1 -- 172.21.15.28:0/673595754 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561323020a70 con 0x561323138c50 2024-04-21T08:57:05.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.082+0000 7f892099c640 1 --2- 172.21.15.28:0/673595754 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561323138c50 0x56132313b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:05.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8904ff9640 10 client.17325 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:05.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f892099c640 1 --2- 172.21.15.28:0/673595754 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561323138c50 0x56132313b070 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f4b58853a145941 server_cookie=b5c790102a6addec in_seq=0 out_seq=0 2024-04-21T08:57:05.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8904ff9640 1 -- 172.21.15.28:0/673595754 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561323020a70 con 0x561323138c50 2024-04-21T08:57:05.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8904ff9640 10 client.17325 handle_command_reply: tid=0 2024-04-21T08:57:05.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8925ff0740 1 client.17325 shutdown 2024-04-21T08:57:05.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8925ff0740 1 -- 172.21.15.28:0/673595754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b40f0 msgr2=0x5613230ca5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:05.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8925ff0740 1 --2- 172.21.15.28:0/673595754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b40f0 0x5613230ca5e0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f891400c810 tx=0x7f8914005850 comp rx=0 tx=0).stop 2024-04-21T08:57:05.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8925ff0740 1 -- 172.21.15.28:0/673595754 shutdown_connections 2024-04-21T08:57:05.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8925ff0740 1 --2- 172.21.15.28:0/673595754 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x561323138c50 0x56132313b070 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8925ff0740 1 --2- 172.21.15.28:0/673595754 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5613230c2020 0x5613230c5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8925ff0740 1 --2- 172.21.15.28:0/673595754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5613230b4aa0 0x5613230c5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8925ff0740 1 --2- 172.21.15.28:0/673595754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613230b40f0 0x5613230ca5e0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8904ff9640 0 client.17325 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:05.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8925ff0740 1 -- 172.21.15.28:0/673595754 >> 172.21.15.28:0/673595754 conn(0x5613230aee40 msgr2=0x5613230b09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:05.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8925ff0740 1 -- 172.21.15.28:0/673595754 shutdown_connections 2024-04-21T08:57:05.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8925ff0740 1 -- 172.21.15.28:0/673595754 wait complete. 2024-04-21T08:57:05.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.083+0000 7f8925ff0740 20 client.17325 trim_cache size 0 max 16384 2024-04-21T08:57:05.110 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:57:05.110 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:57:05.110 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:57:05.110 INFO:teuthology.orchestra.run.smithi028.stdout: "f827cb55-f8a5-4a09-b796-187b09487a19": { 2024-04-21T08:57:05.110 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:57:05.110 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "f827cb55-f8a5-4a09-b796-187b09487a19", 2024-04-21T08:57:05.111 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:57:05.111 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:57:05.111 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:57:05.111 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:57:05.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.084+0000 7f892219f640 1 -- 172.21.15.28:0/893503421 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f89080a3d20 msgr2=0x7f89080a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:05.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.084+0000 7f892219f640 1 --2- 172.21.15.28:0/893503421 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f89080a3d20 0x7f89080a61e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f89180ce040 tx=0x7f8918098b00 comp rx=0 tx=0).stop 2024-04-21T08:57:05.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.084+0000 7f892219f640 1 -- 172.21.15.28:0/893503421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891c102f30 msgr2=0x7f891c198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:05.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.084+0000 7f892219f640 1 --2- 172.21.15.28:0/893503421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891c102f30 0x7f891c198510 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f891800db80 tx=0x7f891800b630 comp rx=0 tx=0).stop 2024-04-21T08:57:05.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.084+0000 7f892219f640 1 -- 172.21.15.28:0/893503421 shutdown_connections 2024-04-21T08:57:05.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.084+0000 7f892219f640 1 --2- 172.21.15.28:0/893503421 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f89080a3d20 0x7f89080a61e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.084+0000 7f892219f640 1 --2- 172.21.15.28:0/893503421 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f891c10c300 0x7f891c19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.084+0000 7f892219f640 1 --2- 172.21.15.28:0/893503421 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f891c103850 0x7f891c198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.084+0000 7f892219f640 1 --2- 172.21.15.28:0/893503421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f891c102f30 0x7f891c198510 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:05.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.084+0000 7f892219f640 1 -- 172.21.15.28:0/893503421 >> 172.21.15.28:0/893503421 conn(0x7f891c077a30 msgr2=0x7f891c0fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.084+0000 7f892219f640 1 -- 172.21.15.28:0/893503421 shutdown_connections 2024-04-21T08:57:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:05.084+0000 7f892219f640 1 -- 172.21.15.28:0/893503421 wait complete. 2024-04-21T08:57:05.114 INFO:tasks.cephfs.filesystem:scrub status for tag:f827cb55-f8a5-4a09-b796-187b09487a19 - {'path': '/', 'tag': 'f827cb55-f8a5-4a09-b796-187b09487a19', 'options': 'recursive,force'} 2024-04-21T08:57:05.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:05.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:05.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:05.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:05.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:05.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:05.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:05.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:05.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:05.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:05.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:05 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:05.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:05.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:05.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:05.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:05.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:05.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:05.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:05.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:05.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:05.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:05 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:05.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:05.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:05.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:05.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:05.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:05.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:05.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:05.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:05.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:05.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:05.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:05 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:06.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:06 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:57:06.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:06 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:06.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:06 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:06.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:06 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:06.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:06 smithi152 ceph-mon[31401]: pgmap v180: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-04-21T08:57:06.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:06 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:57:06.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:06 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:06.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:06 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:06.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:06 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:06.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:06 smithi028 ceph-mon[28003]: pgmap v180: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-04-21T08:57:06.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:06 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:57:06.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:06 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:06.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:06 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:06.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:06 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:06.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:06 smithi171 ceph-mon[31358]: pgmap v180: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 21 op/s 2024-04-21T08:57:08.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:08 smithi152 ceph-mon[31401]: pgmap v181: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-04-21T08:57:08.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:08 smithi028 ceph-mon[28003]: pgmap v181: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-04-21T08:57:08.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:08 smithi171 ceph-mon[31358]: pgmap v181: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-04-21T08:57:10.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:10 smithi152 ceph-mon[31401]: pgmap v182: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 28 KiB/s wr, 41 op/s 2024-04-21T08:57:10.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:10 smithi028 ceph-mon[28003]: pgmap v182: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 28 KiB/s wr, 41 op/s 2024-04-21T08:57:10.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:10 smithi171 ceph-mon[31358]: pgmap v182: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 28 KiB/s wr, 41 op/s 2024-04-21T08:57:12.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:12 smithi152 ceph-mon[31401]: pgmap v183: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 28 KiB/s wr, 43 op/s 2024-04-21T08:57:12.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:12 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:57:12.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:12 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:57:12.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:12 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:57:12.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:12 smithi028 ceph-mon[28003]: pgmap v183: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 28 KiB/s wr, 43 op/s 2024-04-21T08:57:12.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:12 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:57:12.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:12 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:57:12.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:12 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:57:12.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:12 smithi171 ceph-mon[31358]: pgmap v183: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 28 KiB/s wr, 43 op/s 2024-04-21T08:57:12.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:12 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:57:12.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:12 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:57:12.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:12 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:57:14.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:14 smithi152 ceph-mon[31401]: pgmap v184: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 28 KiB/s wr, 45 op/s 2024-04-21T08:57:14.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:14 smithi028 ceph-mon[28003]: pgmap v184: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 28 KiB/s wr, 45 op/s 2024-04-21T08:57:14.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:14 smithi171 ceph-mon[31358]: pgmap v184: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 28 KiB/s wr, 45 op/s 2024-04-21T08:57:16.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:16 smithi152 ceph-mon[31401]: pgmap v185: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 16 KiB/s wr, 42 op/s 2024-04-21T08:57:16.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:16 smithi028 ceph-mon[28003]: pgmap v185: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 16 KiB/s wr, 42 op/s 2024-04-21T08:57:16.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:16 smithi171 ceph-mon[31358]: pgmap v185: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 16 KiB/s wr, 42 op/s 2024-04-21T08:57:18.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:18 smithi152 ceph-mon[31401]: pgmap v186: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 2.6 KiB/s wr, 29 op/s 2024-04-21T08:57:18.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:18 smithi028 ceph-mon[28003]: pgmap v186: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 2.6 KiB/s wr, 29 op/s 2024-04-21T08:57:18.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:18 smithi171 ceph-mon[31358]: pgmap v186: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 2.6 KiB/s wr, 29 op/s 2024-04-21T08:57:20.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:20 smithi152 ceph-mon[31401]: pgmap v187: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.6 KiB/s wr, 25 op/s 2024-04-21T08:57:20.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:20 smithi028 ceph-mon[28003]: pgmap v187: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.6 KiB/s wr, 25 op/s 2024-04-21T08:57:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:20 smithi171 ceph-mon[31358]: pgmap v187: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 2.6 KiB/s wr, 25 op/s 2024-04-21T08:57:22.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:22 smithi152 ceph-mon[31401]: pgmap v188: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 8 op/s 2024-04-21T08:57:22.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:22 smithi028 ceph-mon[28003]: pgmap v188: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 8 op/s 2024-04-21T08:57:22.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:22 smithi171 ceph-mon[31358]: pgmap v188: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 8 op/s 2024-04-21T08:57:24.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:24 smithi152 ceph-mon[31401]: pgmap v189: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:57:24.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:24 smithi028 ceph-mon[28003]: pgmap v189: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:57:24.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:24 smithi171 ceph-mon[31358]: pgmap v189: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:57:26.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:26 smithi152 ceph-mon[31401]: pgmap v190: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:57:26.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:26 smithi028 ceph-mon[28003]: pgmap v190: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:57:26.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:26 smithi171 ceph-mon[31358]: pgmap v190: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:57:28.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:28 smithi152 ceph-mon[31401]: pgmap v191: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:57:28.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:28 smithi028 ceph-mon[28003]: pgmap v191: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:57:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:28 smithi171 ceph-mon[31358]: pgmap v191: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:57:30.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:30 smithi152 ceph-mon[31401]: pgmap v192: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:57:30.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:30 smithi028 ceph-mon[28003]: pgmap v192: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:57:30.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:30 smithi171 ceph-mon[31358]: pgmap v192: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:57:32.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:32 smithi152 ceph-mon[31401]: pgmap v193: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:57:32.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:32 smithi028 ceph-mon[28003]: pgmap v193: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:57:32.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:32 smithi171 ceph-mon[31358]: pgmap v193: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:57:34.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:34 smithi152 ceph-mon[31401]: pgmap v194: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:57:34.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:34 smithi028 ceph-mon[28003]: pgmap v194: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:57:34.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:34 smithi171 ceph-mon[31358]: pgmap v194: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:57:35.115 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:57:35.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:57:35.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.234+0000 7fbaefc31640 1 -- 172.21.15.28:0/2555518580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae8101070 msgr2=0x7fbae8106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.234+0000 7fbaefc31640 1 --2- 172.21.15.28:0/2555518580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae8101070 0x7fbae8106ba0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009900 tx=0x7fbad802bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:35.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.234+0000 7fbaefc31640 1 -- 172.21.15.28:0/2555518580 shutdown_connections 2024-04-21T08:57:35.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.234+0000 7fbaefc31640 1 --2- 172.21.15.28:0/2555518580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbae8107280 0x7fbae810ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.234+0000 7fbaefc31640 1 --2- 172.21.15.28:0/2555518580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae8101070 0x7fbae8106ba0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.234+0000 7fbaefc31640 1 --2- 172.21.15.28:0/2555518580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbae81006c0 0x7fbae8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.234+0000 7fbaefc31640 1 -- 172.21.15.28:0/2555518580 >> 172.21.15.28:0/2555518580 conn(0x7fbae80773a0 msgr2=0x7fbae80fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.234+0000 7fbaefc31640 1 -- 172.21.15.28:0/2555518580 shutdown_connections 2024-04-21T08:57:35.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.234+0000 7fbaefc31640 1 -- 172.21.15.28:0/2555518580 wait complete. 2024-04-21T08:57:35.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.234+0000 7fbaefc31640 1 Processor -- start 2024-04-21T08:57:35.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.234+0000 7fbaefc31640 1 -- start start 2024-04-21T08:57:35.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaefc31640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbae81006c0 0x7fbae8101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaefc31640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbae8107280 0x7fbae8101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaefc31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae81020b0 0x7fbae81a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaefc31640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81121c0 con 0x7fbae81020b0 2024-04-21T08:57:35.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaefc31640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbae8112040 con 0x7fbae81006c0 2024-04-21T08:57:35.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaefc31640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbae8112340 con 0x7fbae8107280 2024-04-21T08:57:35.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaee1ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae81020b0 0x7fbae81a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaee1ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae81020b0 0x7fbae81a68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42984/0 (socket says 172.21.15.28:42984) 2024-04-21T08:57:35.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaee1ce640 1 -- 172.21.15.28:0/3183992108 learned_addr learned my addr 172.21.15.28:0/3183992108 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaed9cd640 1 --2- 172.21.15.28:0/3183992108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbae81006c0 0x7fbae8101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaed1cc640 1 --2- 172.21.15.28:0/3183992108 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbae8107280 0x7fbae8101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaee1ce640 1 -- 172.21.15.28:0/3183992108 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbae8107280 msgr2=0x7fbae8101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaee1ce640 1 --2- 172.21.15.28:0/3183992108 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbae8107280 0x7fbae8101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaee1ce640 1 -- 172.21.15.28:0/3183992108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbae81006c0 msgr2=0x7fbae8101630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaee1ce640 1 --2- 172.21.15.28:0/3183992108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbae81006c0 0x7fbae8101630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaee1ce640 1 -- 172.21.15.28:0/3183992108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae81a6fd0 con 0x7fbae81020b0 2024-04-21T08:57:35.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.235+0000 7fbaee1ce640 1 --2- 172.21.15.28:0/3183992108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae81020b0 0x7fbae81a68d0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fbae000cce0 tx=0x7fbae000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.236+0000 7fbadeffd640 1 -- 172.21.15.28:0/3183992108 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae000f450 con 0x7fbae81020b0 2024-04-21T08:57:35.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.236+0000 7fbaefc31640 1 -- 172.21.15.28:0/3183992108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae81a72c0 con 0x7fbae81020b0 2024-04-21T08:57:35.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.236+0000 7fbadeffd640 1 -- 172.21.15.28:0/3183992108 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbae000faa0 con 0x7fbae81020b0 2024-04-21T08:57:35.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.236+0000 7fbadeffd640 1 -- 172.21.15.28:0/3183992108 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae0018b70 con 0x7fbae81020b0 2024-04-21T08:57:35.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.236+0000 7fbaefc31640 1 -- 172.21.15.28:0/3183992108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae81a7790 con 0x7fbae81020b0 2024-04-21T08:57:35.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.238+0000 7fbadeffd640 1 -- 172.21.15.28:0/3183992108 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbae0018d10 con 0x7fbae81020b0 2024-04-21T08:57:35.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.238+0000 7fbaf2829740 1 Processor -- start 2024-04-21T08:57:35.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaf2829740 1 -- start start 2024-04-21T08:57:35.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaf2829740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373580 0x5576d5373960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaf2829740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d5373ea0 0x5576d537c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaf2829740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d537ca80 0x5576d537ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaf2829740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5576d52d81e0 con 0x5576d5373ea0 2024-04-21T08:57:35.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaf2829740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5576d5275390 con 0x5576d537ca80 2024-04-21T08:57:35.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaf2829740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5576d52d41a0 con 0x5576d5373580 2024-04-21T08:57:35.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaed9cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d5373ea0 0x5576d537c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaed9cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d5373ea0 0x5576d537c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42996/0 (socket says 172.21.15.28:42996) 2024-04-21T08:57:35.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaed9cd640 1 -- 172.21.15.28:0/1986402643 learned_addr learned my addr 172.21.15.28:0/1986402643 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaed1cc640 1 --2- 172.21.15.28:0/1986402643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d537ca80 0x5576d537ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbadeffd640 1 --2- 172.21.15.28:0/3183992108 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbad40a3d50 0x7fbad40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaed1cc640 1 --2- 172.21.15.28:0/1986402643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373580 0x5576d5373960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaed9cd640 1 -- 172.21.15.28:0/1986402643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373580 msgr2=0x5576d5373960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaed9cd640 1 --2- 172.21.15.28:0/1986402643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373580 0x5576d5373960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaed9cd640 1 -- 172.21.15.28:0/1986402643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d537ca80 msgr2=0x5576d537ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaed9cd640 1 --2- 172.21.15.28:0/1986402643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d537ca80 0x5576d537ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaee1ce640 1 --2- 172.21.15.28:0/3183992108 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbad40a3d50 0x7fbad40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaed9cd640 1 -- 172.21.15.28:0/1986402643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576d50b47d0 con 0x5576d5373ea0 2024-04-21T08:57:35.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbaed1cc640 1 --2- 172.21.15.28:0/1986402643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373580 0x5576d5373960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:57:35.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.239+0000 7fbadeffd640 1 -- 172.21.15.28:0/3183992108 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbae00d0f20 con 0x7fbae81020b0 2024-04-21T08:57:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.240+0000 7fbaed9cd640 1 --2- 172.21.15.28:0/1986402643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d5373ea0 0x5576d537c540 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fbae400b760 tx=0x7fbae400bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f25bc2dd0341359e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.240+0000 7fbad3fff640 1 -- 172.21.15.28:0/1986402643 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae4003dd0 con 0x5576d5373ea0 2024-04-21T08:57:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.240+0000 7fbaee1ce640 1 --2- 172.21.15.28:0/3183992108 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbad40a3d50 0x7fbad40a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbae009c7c0 tx=0x7fbae0091f50 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.240+0000 7fbad3fff640 1 -- 172.21.15.28:0/1986402643 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbae4015070 con 0x5576d5373ea0 2024-04-21T08:57:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.240+0000 7fbad3fff640 1 -- 172.21.15.28:0/1986402643 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae4004da0 con 0x5576d5373ea0 2024-04-21T08:57:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.241+0000 7fbaf2829740 1 -- 172.21.15.28:0/1986402643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d5373ea0 msgr2=0x5576d537c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.241+0000 7fbaf2829740 1 --2- 172.21.15.28:0/1986402643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d5373ea0 0x5576d537c540 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fbae400b760 tx=0x7fbae400bc20 comp rx=0 tx=0).stop 2024-04-21T08:57:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.241+0000 7fbaf2829740 1 -- 172.21.15.28:0/1986402643 shutdown_connections 2024-04-21T08:57:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.241+0000 7fbaf2829740 1 --2- 172.21.15.28:0/1986402643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d537ca80 0x5576d537ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.241+0000 7fbaf2829740 1 --2- 172.21.15.28:0/1986402643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d5373ea0 0x5576d537c540 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.241+0000 7fbaf2829740 1 --2- 172.21.15.28:0/1986402643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373580 0x5576d5373960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.241+0000 7fbaf2829740 1 -- 172.21.15.28:0/1986402643 >> 172.21.15.28:0/1986402643 conn(0x5576d536dcb0 msgr2=0x5576d536e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.241+0000 7fbaf2829740 1 -- 172.21.15.28:0/1986402643 shutdown_connections 2024-04-21T08:57:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.241+0000 7fbaf2829740 1 -- 172.21.15.28:0/1986402643 wait complete. 2024-04-21T08:57:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.241+0000 7fbaf2829740 1 -- start start 2024-04-21T08:57:35.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.241+0000 7fbaf2829740 1 Processor -- start 2024-04-21T08:57:35.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaf2829740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373580 0x5576d5379b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaf2829740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373ea0 0x5576d537a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaf2829740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d537ca80 0x5576d537a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaf2829740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5576d537f870 con 0x5576d537ca80 2024-04-21T08:57:35.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaf2829740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5576d52912f0 con 0x5576d5373580 2024-04-21T08:57:35.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaf2829740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5576d537f9f0 con 0x5576d5373ea0 2024-04-21T08:57:35.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaf2829740 10 client.0 _ll_register_callbacks cb 0x5576d529e580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:35.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaed1cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d537ca80 0x5576d537a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaed1cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d537ca80 0x5576d537a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43012/0 (socket says 172.21.15.28:43012) 2024-04-21T08:57:35.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaed1cc640 1 -- 172.21.15.28:0/1158457115 learned_addr learned my addr 172.21.15.28:0/1158457115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaed1cc640 1 --2- 172.21.15.28:0/1158457115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373580 0x5576d5379b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaed9cd640 1 --2- 172.21.15.28:0/1158457115 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373ea0 0x5576d537a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaed1cc640 1 -- 172.21.15.28:0/1158457115 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373ea0 msgr2=0x5576d537a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaed1cc640 1 --2- 172.21.15.28:0/1158457115 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373ea0 0x5576d537a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaed1cc640 1 -- 172.21.15.28:0/1158457115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373580 msgr2=0x5576d5379b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaed1cc640 1 --2- 172.21.15.28:0/1158457115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373580 0x5576d5379b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaed1cc640 1 -- 172.21.15.28:0/1158457115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576d4fea3f0 con 0x5576d537ca80 2024-04-21T08:57:35.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaed9cd640 1 --2- 172.21.15.28:0/1158457115 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373ea0 0x5576d537a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:57:35.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbaed1cc640 1 --2- 172.21.15.28:0/1158457115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d537ca80 0x5576d537a580 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fbae80746d0 tx=0x7fbad8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e97ffba5fcd1eea server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbad17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.242+0000 7fbad17fa640 1 -- 172.21.15.28:0/1158457115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad8005510 con 0x5576d537ca80 2024-04-21T08:57:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.243+0000 7fbad17fa640 1 -- 172.21.15.28:0/1158457115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbad80046c0 con 0x5576d537ca80 2024-04-21T08:57:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.243+0000 7fbaf2829740 1 -- 172.21.15.28:0/1158457115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbae4015070 con 0x5576d537ca80 2024-04-21T08:57:35.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.243+0000 7fbad17fa640 1 -- 172.21.15.28:0/1158457115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbad8057050 con 0x5576d537ca80 2024-04-21T08:57:35.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.243+0000 7fbaf2829740 10 client.17337 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:35.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.243+0000 7fbaf2829740 1 -- 172.21.15.28:0/1158457115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5576d4faa0f0 con 0x5576d537ca80 2024-04-21T08:57:35.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.243+0000 7fbad17fa640 1 -- 172.21.15.28:0/1158457115 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbae4015070 con 0x5576d537ca80 2024-04-21T08:57:35.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.243+0000 7fbad17fa640 1 -- 172.21.15.28:0/1158457115 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fbad804e070 con 0x5576d537ca80 2024-04-21T08:57:35.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.243+0000 7fbaf2829740 10 client.17337 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:35.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.243+0000 7fbaf2829740 10 client.17337 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:35.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.243+0000 7fbaf2829740 20 client.17337 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:35.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.243+0000 7fbaf2829740 1 --2- 172.21.15.28:0/1158457115 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5576d538c9f0 0x5576d538ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.243+0000 7fbaf2829740 4 client.17337 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:57:35.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.243+0000 7fbaf2829740 1 -- 172.21.15.28:0/1158457115 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576d52e5c00 con 0x5576d538c9f0 2024-04-21T08:57:35.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.244+0000 7fbaed9cd640 1 --2- 172.21.15.28:0/1158457115 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5576d538c9f0 0x5576d538ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.244+0000 7fbaed9cd640 1 --2- 172.21.15.28:0/1158457115 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5576d538c9f0 0x5576d538ee70 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dc2789bfdf10e0c server_cookie=e2f8ec31fcd59768 in_seq=0 out_seq=0 2024-04-21T08:57:35.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.244+0000 7fbad17fa640 10 client.17337 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:35.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.245+0000 7fbad17fa640 1 -- 172.21.15.28:0/1158457115 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5576d52e5c00 con 0x5576d538c9f0 2024-04-21T08:57:35.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.245+0000 7fbad17fa640 10 client.17337 handle_command_reply: tid=0 2024-04-21T08:57:35.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.245+0000 7fbaf2829740 1 client.17337 shutdown 2024-04-21T08:57:35.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.245+0000 7fbaf2829740 1 -- 172.21.15.28:0/1158457115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d537ca80 msgr2=0x5576d537a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.245+0000 7fbaf2829740 1 --2- 172.21.15.28:0/1158457115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d537ca80 0x5576d537a580 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fbae80746d0 tx=0x7fbad8009260 comp rx=0 tx=0).stop 2024-04-21T08:57:35.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.245+0000 7fbaf2829740 1 -- 172.21.15.28:0/1158457115 shutdown_connections 2024-04-21T08:57:35.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.245+0000 7fbaf2829740 1 --2- 172.21.15.28:0/1158457115 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5576d538c9f0 0x5576d538ee70 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.245+0000 7fbaf2829740 1 --2- 172.21.15.28:0/1158457115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d537ca80 0x5576d537a580 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.245+0000 7fbaf2829740 1 --2- 172.21.15.28:0/1158457115 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373ea0 0x5576d537a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.245+0000 7fbaf2829740 1 --2- 172.21.15.28:0/1158457115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373580 0x5576d5379b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.245+0000 7fbad17fa640 0 client.17337 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:35.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.245+0000 7fbaf2829740 1 -- 172.21.15.28:0/1158457115 >> 172.21.15.28:0/1158457115 conn(0x5576d536dcb0 msgr2=0x5576d536fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.245+0000 7fbaf2829740 1 -- 172.21.15.28:0/1158457115 shutdown_connections 2024-04-21T08:57:35.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.245+0000 7fbaf2829740 1 -- 172.21.15.28:0/1158457115 wait complete. 2024-04-21T08:57:35.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.245+0000 7fbaf2829740 20 client.17337 trim_cache size 0 max 16384 2024-04-21T08:57:35.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaf2829740 1 Processor -- start 2024-04-21T08:57:35.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaf2829740 1 -- start start 2024-04-21T08:57:35.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaf2829740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373580 0x5576d5388730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaf2829740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d5373ea0 0x5576d53793e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaf2829740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d537ca80 0x5576d537a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaf2829740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81b3ca0 con 0x5576d5373ea0 2024-04-21T08:57:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaf2829740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81b1570 con 0x5576d5373580 2024-04-21T08:57:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaf2829740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81af3c0 con 0x5576d537ca80 2024-04-21T08:57:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaed9cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d5373ea0 0x5576d53793e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaed9cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d5373ea0 0x5576d53793e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43022/0 (socket says 172.21.15.28:43022) 2024-04-21T08:57:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaed9cd640 1 -- 172.21.15.28:0/1618101328 learned_addr learned my addr 172.21.15.28:0/1618101328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaed1cc640 1 --2- 172.21.15.28:0/1618101328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373580 0x5576d5388730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaed1cc640 1 --2- 172.21.15.28:0/1618101328 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d537ca80 0x5576d537a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaed9cd640 1 -- 172.21.15.28:0/1618101328 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d537ca80 msgr2=0x5576d537a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaed9cd640 1 --2- 172.21.15.28:0/1618101328 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d537ca80 0x5576d537a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaed9cd640 1 -- 172.21.15.28:0/1618101328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373580 msgr2=0x5576d5388730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaed9cd640 1 --2- 172.21.15.28:0/1618101328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373580 0x5576d5388730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.263+0000 7fbaed9cd640 1 -- 172.21.15.28:0/1618101328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576d5373cd0 con 0x5576d5373ea0 2024-04-21T08:57:35.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.264+0000 7fbaed9cd640 1 --2- 172.21.15.28:0/1618101328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d5373ea0 0x5576d53793e0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x5576d5373050 tx=0x7fbae40049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6d3167b5d9446c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.264+0000 7fbad37fe640 1 -- 172.21.15.28:0/1618101328 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae4003e50 con 0x5576d5373ea0 2024-04-21T08:57:35.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.264+0000 7fbad37fe640 1 -- 172.21.15.28:0/1618101328 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbae4015050 con 0x5576d5373ea0 2024-04-21T08:57:35.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.264+0000 7fbad37fe640 1 -- 172.21.15.28:0/1618101328 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae40116a0 con 0x5576d5373ea0 2024-04-21T08:57:35.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.264+0000 7fbaf2829740 1 -- 172.21.15.28:0/1618101328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d5373ea0 msgr2=0x5576d53793e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.264+0000 7fbaf2829740 1 --2- 172.21.15.28:0/1618101328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d5373ea0 0x5576d53793e0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x5576d5373050 tx=0x7fbae40049c0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.264+0000 7fbaf2829740 1 -- 172.21.15.28:0/1618101328 shutdown_connections 2024-04-21T08:57:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.264+0000 7fbaf2829740 1 --2- 172.21.15.28:0/1618101328 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d537ca80 0x5576d537a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.264+0000 7fbaf2829740 1 --2- 172.21.15.28:0/1618101328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d5373ea0 0x5576d53793e0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.264+0000 7fbaf2829740 1 --2- 172.21.15.28:0/1618101328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373580 0x5576d5388730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.264+0000 7fbaf2829740 1 -- 172.21.15.28:0/1618101328 >> 172.21.15.28:0/1618101328 conn(0x5576d536dcb0 msgr2=0x5576d5379ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.265+0000 7fbaf2829740 1 -- 172.21.15.28:0/1618101328 shutdown_connections 2024-04-21T08:57:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.265+0000 7fbaf2829740 1 -- 172.21.15.28:0/1618101328 wait complete. 2024-04-21T08:57:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.265+0000 7fbaf2829740 1 -- start start 2024-04-21T08:57:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.265+0000 7fbaf2829740 1 Processor -- start 2024-04-21T08:57:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.265+0000 7fbaf2829740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373580 0x5576d5385060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.265+0000 7fbaf2829740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373ea0 0x5576d5376530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.265+0000 7fbaf2829740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d537ca80 0x5576d5376a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.265+0000 7fbaf2829740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81afa30 con 0x5576d537ca80 2024-04-21T08:57:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.265+0000 7fbaf2829740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81b0ea0 con 0x5576d5373ea0 2024-04-21T08:57:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.265+0000 7fbaf2829740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81b0100 con 0x5576d5373580 2024-04-21T08:57:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.265+0000 7fbaf2829740 10 client.0 _ll_register_callbacks cb 0x7fbad8009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.265+0000 7fbaed1cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d537ca80 0x5576d5376a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbaed1cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d537ca80 0x5576d5376a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43036/0 (socket says 172.21.15.28:43036) 2024-04-21T08:57:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbaed1cc640 1 -- 172.21.15.28:0/44743199 learned_addr learned my addr 172.21.15.28:0/44743199 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbaed9cd640 1 --2- 172.21.15.28:0/44743199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373ea0 0x5576d5376530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbaed1cc640 1 --2- 172.21.15.28:0/44743199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373580 0x5576d5385060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbaed1cc640 1 -- 172.21.15.28:0/44743199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373580 msgr2=0x5576d5385060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbaed1cc640 1 --2- 172.21.15.28:0/44743199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373580 0x5576d5385060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbaed1cc640 1 -- 172.21.15.28:0/44743199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373ea0 msgr2=0x5576d5376530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbaed1cc640 1 --2- 172.21.15.28:0/44743199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373ea0 0x5576d5376530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbaed1cc640 1 -- 172.21.15.28:0/44743199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576d53772f0 con 0x5576d537ca80 2024-04-21T08:57:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbaed9cd640 1 --2- 172.21.15.28:0/44743199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373ea0 0x5576d5376530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:57:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbaed1cc640 1 --2- 172.21.15.28:0/44743199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d537ca80 0x5576d5376a70 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009900 tx=0x7fbad8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03ae60b94c123c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbad17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbad17fa640 1 -- 172.21.15.28:0/44743199 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad80037f0 con 0x5576d537ca80 2024-04-21T08:57:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbad17fa640 1 -- 172.21.15.28:0/44743199 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbad80046c0 con 0x5576d537ca80 2024-04-21T08:57:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbaf2829740 1 -- 172.21.15.28:0/44743199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbae4015050 con 0x5576d537ca80 2024-04-21T08:57:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbad17fa640 1 -- 172.21.15.28:0/44743199 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbad80048c0 con 0x5576d537ca80 2024-04-21T08:57:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbaf2829740 10 client.17343 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.266+0000 7fbaf2829740 1 -- 172.21.15.28:0/44743199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5576d53f3010 con 0x5576d537ca80 2024-04-21T08:57:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.267+0000 7fbad17fa640 1 -- 172.21.15.28:0/44743199 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbae4015050 con 0x5576d537ca80 2024-04-21T08:57:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.267+0000 7fbad17fa640 1 -- 172.21.15.28:0/44743199 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fbad804e040 con 0x5576d537ca80 2024-04-21T08:57:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.267+0000 7fbaf2829740 10 client.17343 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.267+0000 7fbaf2829740 10 client.17343 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.267+0000 7fbaf2829740 20 client.17343 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.267+0000 7fbaf2829740 1 --2- 172.21.15.28:0/44743199 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5576d53f3620 0x5576d53f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.267+0000 7fbaf2829740 4 client.17343 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:57:35.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.267+0000 7fbaf2829740 1 -- 172.21.15.28:0/44743199 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5576d52dfa70 con 0x5576d53f3620 2024-04-21T08:57:35.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.267+0000 7fbaed9cd640 1 --2- 172.21.15.28:0/44743199 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5576d53f3620 0x5576d53f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.267+0000 7fbad17fa640 10 client.17343 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:35.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.267+0000 7fbaed9cd640 1 --2- 172.21.15.28:0/44743199 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5576d53f3620 0x5576d53f5a40 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=290f92add68fdc11 server_cookie=bae26891e0f7bd17 in_seq=0 out_seq=0 2024-04-21T08:57:35.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.268+0000 7fbad17fa640 1 -- 172.21.15.28:0/44743199 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5576d52dfa70 con 0x5576d53f3620 2024-04-21T08:57:35.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.268+0000 7fbad17fa640 10 client.17343 handle_command_reply: tid=0 2024-04-21T08:57:35.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.268+0000 7fbaf2829740 1 client.17343 shutdown 2024-04-21T08:57:35.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.268+0000 7fbaf2829740 1 -- 172.21.15.28:0/44743199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d537ca80 msgr2=0x5576d5376a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.268+0000 7fbaf2829740 1 --2- 172.21.15.28:0/44743199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d537ca80 0x5576d5376a70 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009900 tx=0x7fbad8004040 comp rx=0 tx=0).stop 2024-04-21T08:57:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.268+0000 7fbaf2829740 1 -- 172.21.15.28:0/44743199 shutdown_connections 2024-04-21T08:57:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.268+0000 7fbaf2829740 1 --2- 172.21.15.28:0/44743199 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5576d53f3620 0x5576d53f5a40 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.268+0000 7fbaf2829740 1 --2- 172.21.15.28:0/44743199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576d537ca80 0x5576d5376a70 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.268+0000 7fbaf2829740 1 --2- 172.21.15.28:0/44743199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5576d5373ea0 0x5576d5376530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.268+0000 7fbad17fa640 0 client.17343 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.268+0000 7fbaf2829740 1 --2- 172.21.15.28:0/44743199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5576d5373580 0x5576d5385060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.268+0000 7fbaf2829740 1 -- 172.21.15.28:0/44743199 >> 172.21.15.28:0/44743199 conn(0x5576d536dcb0 msgr2=0x5576d537a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.268+0000 7fbaf2829740 1 -- 172.21.15.28:0/44743199 shutdown_connections 2024-04-21T08:57:35.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.268+0000 7fbaf2829740 1 -- 172.21.15.28:0/44743199 wait complete. 2024-04-21T08:57:35.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.268+0000 7fbaf2829740 20 client.17343 trim_cache size 0 max 16384 2024-04-21T08:57:35.303 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:57:35.303 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T08:57:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T08:57:35.304 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:57:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.269+0000 7fbaefc31640 1 -- 172.21.15.28:0/3183992108 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbad40a3d50 msgr2=0x7fbad40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.269+0000 7fbaefc31640 1 --2- 172.21.15.28:0/3183992108 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbad40a3d50 0x7fbad40a6210 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbae009c7c0 tx=0x7fbae0091f50 comp rx=0 tx=0).stop 2024-04-21T08:57:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.269+0000 7fbaefc31640 1 -- 172.21.15.28:0/3183992108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae81020b0 msgr2=0x7fbae81a68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.269+0000 7fbaefc31640 1 --2- 172.21.15.28:0/3183992108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae81020b0 0x7fbae81a68d0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fbae000cce0 tx=0x7fbae000a620 comp rx=0 tx=0).stop 2024-04-21T08:57:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.269+0000 7fbaefc31640 1 -- 172.21.15.28:0/3183992108 shutdown_connections 2024-04-21T08:57:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.269+0000 7fbaefc31640 1 --2- 172.21.15.28:0/3183992108 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbad40a3d50 0x7fbad40a6210 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.269+0000 7fbaefc31640 1 --2- 172.21.15.28:0/3183992108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbae81020b0 0x7fbae81a68d0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.269+0000 7fbaefc31640 1 --2- 172.21.15.28:0/3183992108 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbae8107280 0x7fbae8101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.269+0000 7fbaefc31640 1 --2- 172.21.15.28:0/3183992108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbae81006c0 0x7fbae8101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.269+0000 7fbaefc31640 1 -- 172.21.15.28:0/3183992108 >> 172.21.15.28:0/3183992108 conn(0x7fbae80773a0 msgr2=0x7fbae8105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.269+0000 7fbaefc31640 1 -- 172.21.15.28:0/3183992108 shutdown_connections 2024-04-21T08:57:35.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.269+0000 7fbaefc31640 1 -- 172.21.15.28:0/3183992108 wait complete. 2024-04-21T08:57:35.307 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:57:35.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:57:35.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.410+0000 7f38707fb640 1 -- 172.21.15.28:0/3896455166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868103af0 msgr2=0x7f386810bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.410+0000 7f38707fb640 1 --2- 172.21.15.28:0/3896455166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868103af0 0x7f386810bfd0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3858009900 tx=0x7f385802bcd0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.410+0000 7f38707fb640 1 -- 172.21.15.28:0/3896455166 shutdown_connections 2024-04-21T08:57:35.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.410+0000 7f38707fb640 1 --2- 172.21.15.28:0/3896455166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f386810c5a0 0x7f386810e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.410+0000 7f38707fb640 1 --2- 172.21.15.28:0/3896455166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868103af0 0x7f386810bfd0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.410+0000 7f38707fb640 1 --2- 172.21.15.28:0/3896455166 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f38681031d0 0x7f38681035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.410+0000 7f38707fb640 1 -- 172.21.15.28:0/3896455166 >> 172.21.15.28:0/3896455166 conn(0x7f38680771d0 msgr2=0x7f38680775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.410+0000 7f38707fb640 1 -- 172.21.15.28:0/3896455166 shutdown_connections 2024-04-21T08:57:35.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.410+0000 7f38707fb640 1 -- 172.21.15.28:0/3896455166 wait complete. 2024-04-21T08:57:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f38707fb640 1 Processor -- start 2024-04-21T08:57:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f38707fb640 1 -- start start 2024-04-21T08:57:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f38707fb640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38681031d0 0x7f386819e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f38707fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868103af0 0x7f386819eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f38707fb640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f386810c5a0 0x7f3868198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f38707fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38681121c0 con 0x7f3868103af0 2024-04-21T08:57:35.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f38707fb640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3868112040 con 0x7f38681031d0 2024-04-21T08:57:35.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f38707fb640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3868112340 con 0x7f386810c5a0 2024-04-21T08:57:35.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f386dd96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868103af0 0x7f386819eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f386dd96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868103af0 0x7f386819eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43046/0 (socket says 172.21.15.28:43046) 2024-04-21T08:57:35.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f386dd96640 1 -- 172.21.15.28:0/169842550 learned_addr learned my addr 172.21.15.28:0/169842550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f386ed98640 1 --2- 172.21.15.28:0/169842550 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f386810c5a0 0x7f3868198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f386e597640 1 --2- 172.21.15.28:0/169842550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38681031d0 0x7f386819e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f386dd96640 1 -- 172.21.15.28:0/169842550 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f386810c5a0 msgr2=0x7f3868198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f386dd96640 1 --2- 172.21.15.28:0/169842550 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f386810c5a0 0x7f3868198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f386dd96640 1 -- 172.21.15.28:0/169842550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38681031d0 msgr2=0x7f386819e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f386dd96640 1 --2- 172.21.15.28:0/169842550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38681031d0 0x7f386819e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f386dd96640 1 -- 172.21.15.28:0/169842550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3868198ef0 con 0x7f3868103af0 2024-04-21T08:57:35.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.411+0000 7f386e597640 1 --2- 172.21.15.28:0/169842550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38681031d0 0x7f386819e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:35.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.412+0000 7f386dd96640 1 --2- 172.21.15.28:0/169842550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868103af0 0x7f386819eab0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f3858005940 tx=0x7f385802d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.412+0000 7f385f7fe640 1 -- 172.21.15.28:0/169842550 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858043020 con 0x7f3868103af0 2024-04-21T08:57:35.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.412+0000 7f385f7fe640 1 -- 172.21.15.28:0/169842550 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f385802dc90 con 0x7f3868103af0 2024-04-21T08:57:35.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.412+0000 7f38707fb640 1 -- 172.21.15.28:0/169842550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3868199180 con 0x7f3868103af0 2024-04-21T08:57:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.412+0000 7f385f7fe640 1 -- 172.21.15.28:0/169842550 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858003ab0 con 0x7f3868103af0 2024-04-21T08:57:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.412+0000 7f38707fb640 1 -- 172.21.15.28:0/169842550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38681a54f0 con 0x7f3868103af0 2024-04-21T08:57:35.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.413+0000 7f38707fb640 1 -- 172.21.15.28:0/169842550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f386806ae60 con 0x7f3868103af0 2024-04-21T08:57:35.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.414+0000 7f385f7fe640 1 -- 172.21.15.28:0/169842550 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3858046020 con 0x7f3868103af0 2024-04-21T08:57:35.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.414+0000 7f385f7fe640 1 --2- 172.21.15.28:0/169842550 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f38540a3da0 0x7f38540a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.415+0000 7f385f7fe640 1 -- 172.21.15.28:0/169842550 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f38580f2c30 con 0x7f3868103af0 2024-04-21T08:57:35.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.415+0000 7f386e597640 1 --2- 172.21.15.28:0/169842550 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f38540a3da0 0x7f38540a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.415+0000 7f386e597640 1 --2- 172.21.15.28:0/169842550 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f38540a3da0 0x7f38540a6260 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3864005d00 tx=0x7f3864009550 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.419+0000 7f385f7fe640 1 -- 172.21.15.28:0/169842550 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f385804d020 con 0x7f3868103af0 2024-04-21T08:57:35.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.645+0000 7f38707fb640 1 -- 172.21.15.28:0/169842550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3868199b60 con 0x7f3868103af0 2024-04-21T08:57:35.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.646+0000 7f385f7fe640 1 -- 172.21.15.28:0/169842550 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7f38580bf040 con 0x7f3868103af0 2024-04-21T08:57:35.646 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:57:35.646 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:57:35.647 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T08:57:35.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.648+0000 7f38707fb640 1 -- 172.21.15.28:0/169842550 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f38540a3da0 msgr2=0x7f38540a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.648+0000 7f38707fb640 1 --2- 172.21.15.28:0/169842550 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f38540a3da0 0x7f38540a6260 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3864005d00 tx=0x7f3864009550 comp rx=0 tx=0).stop 2024-04-21T08:57:35.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.648+0000 7f38707fb640 1 -- 172.21.15.28:0/169842550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868103af0 msgr2=0x7f386819eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.648+0000 7f38707fb640 1 --2- 172.21.15.28:0/169842550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868103af0 0x7f386819eab0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f3858005940 tx=0x7f385802d890 comp rx=0 tx=0).stop 2024-04-21T08:57:35.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.648+0000 7f38707fb640 1 -- 172.21.15.28:0/169842550 shutdown_connections 2024-04-21T08:57:35.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.648+0000 7f38707fb640 1 --2- 172.21.15.28:0/169842550 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f38540a3da0 0x7f38540a6260 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.648+0000 7f38707fb640 1 --2- 172.21.15.28:0/169842550 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f386810c5a0 0x7f3868198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.648+0000 7f38707fb640 1 --2- 172.21.15.28:0/169842550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868103af0 0x7f386819eab0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.648+0000 7f38707fb640 1 --2- 172.21.15.28:0/169842550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f38681031d0 0x7f386819e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.648+0000 7f38707fb640 1 -- 172.21.15.28:0/169842550 >> 172.21.15.28:0/169842550 conn(0x7f38680771d0 msgr2=0x7f38680ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.648+0000 7f38707fb640 1 -- 172.21.15.28:0/169842550 shutdown_connections 2024-04-21T08:57:35.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.648+0000 7f38707fb640 1 -- 172.21.15.28:0/169842550 wait complete. 2024-04-21T08:57:35.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:57:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.764+0000 7ff9e1f7e640 1 -- 172.21.15.28:0/1143985442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9dc06a840 msgr2=0x7ff9dc06ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.764+0000 7ff9e1f7e640 1 --2- 172.21.15.28:0/1143985442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9dc06a840 0x7ff9dc06ac20 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc009900 tx=0x7ff9cc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.764+0000 7ff9e1f7e640 1 -- 172.21.15.28:0/1143985442 shutdown_connections 2024-04-21T08:57:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.764+0000 7ff9e1f7e640 1 --2- 172.21.15.28:0/1143985442 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9dc107270 0x7ff9dc10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.764+0000 7ff9e1f7e640 1 --2- 172.21.15.28:0/1143985442 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9dc06b1f0 0x7ff9dc106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.764+0000 7ff9e1f7e640 1 --2- 172.21.15.28:0/1143985442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9dc06a840 0x7ff9dc06ac20 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.764+0000 7ff9e1f7e640 1 -- 172.21.15.28:0/1143985442 >> 172.21.15.28:0/1143985442 conn(0x7ff9dc077eb0 msgr2=0x7ff9dc0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.764+0000 7ff9e1f7e640 1 -- 172.21.15.28:0/1143985442 shutdown_connections 2024-04-21T08:57:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.764+0000 7ff9e1f7e640 1 -- 172.21.15.28:0/1143985442 wait complete. 2024-04-21T08:57:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.764+0000 7ff9e1f7e640 1 Processor -- start 2024-04-21T08:57:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9e1f7e640 1 -- start start 2024-04-21T08:57:35.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9e1f7e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9dc06a840 0x7ff9dc19e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9e1f7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9dc107270 0x7ff9dc19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9e1f7e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9dc1985e0 0x7ff9dc198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9e1f7e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1122a0 con 0x7ff9dc107270 2024-04-21T08:57:35.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9e1f7e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc112120 con 0x7ff9dc06a840 2024-04-21T08:57:35.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9e1f7e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc112420 con 0x7ff9dc1985e0 2024-04-21T08:57:35.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9dc107270 0x7ff9dc19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9dc107270 0x7ff9dc19e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43066/0 (socket says 172.21.15.28:43066) 2024-04-21T08:57:35.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9d2ffd640 1 -- 172.21.15.28:0/1478305417 learned_addr learned my addr 172.21.15.28:0/1478305417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9db7fe640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9dc06a840 0x7ff9dc19e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9dbfff640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9dc1985e0 0x7ff9dc198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9d2ffd640 1 -- 172.21.15.28:0/1478305417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9dc1985e0 msgr2=0x7ff9dc198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9d2ffd640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9dc1985e0 0x7ff9dc198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9d2ffd640 1 -- 172.21.15.28:0/1478305417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9dc06a840 msgr2=0x7ff9dc19e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9d2ffd640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9dc06a840 0x7ff9dc19e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9d2ffd640 1 -- 172.21.15.28:0/1478305417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9dc199100 con 0x7ff9dc107270 2024-04-21T08:57:35.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9db7fe640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9dc06a840 0x7ff9dc19e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9dbfff640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9dc1985e0 0x7ff9dc198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.765+0000 7ff9d2ffd640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9dc107270 0x7ff9dc19e970 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400d740 tx=0x7ff9c400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.766+0000 7ff9d97fa640 1 -- 172.21.15.28:0/1478305417 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4017070 con 0x7ff9dc107270 2024-04-21T08:57:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.766+0000 7ff9d97fa640 1 -- 172.21.15.28:0/1478305417 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9c400de50 con 0x7ff9dc107270 2024-04-21T08:57:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.766+0000 7ff9e1f7e640 1 -- 172.21.15.28:0/1478305417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9dc1993f0 con 0x7ff9dc107270 2024-04-21T08:57:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.766+0000 7ff9d97fa640 1 -- 172.21.15.28:0/1478305417 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4004ce0 con 0x7ff9dc107270 2024-04-21T08:57:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.766+0000 7ff9e1f7e640 1 -- 172.21.15.28:0/1478305417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9dc1a5640 con 0x7ff9dc107270 2024-04-21T08:57:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.768+0000 7ff9d97fa640 1 -- 172.21.15.28:0/1478305417 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff9c4004e80 con 0x7ff9dc107270 2024-04-21T08:57:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.768+0000 7ff9e4b6b740 1 Processor -- start 2024-04-21T08:57:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.768+0000 7ff9e4b6b740 1 -- start start 2024-04-21T08:57:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.768+0000 7ff9e4b6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bec4d0 0x5589d3bec8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.768+0000 7ff9e4b6b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bece80 0x5589d3bf5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.768+0000 7ff9e4b6b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bf5a00 0x5589d3bf7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.768+0000 7ff9e4b6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5589d3b511f0 con 0x5589d3bec4d0 2024-04-21T08:57:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.768+0000 7ff9e4b6b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5589d3aee4b0 con 0x5589d3bf5a00 2024-04-21T08:57:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.768+0000 7ff9e4b6b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5589d3b4d1b0 con 0x5589d3bece80 2024-04-21T08:57:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bec4d0 0x5589d3bec8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bec4d0 0x5589d3bec8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43070/0 (socket says 172.21.15.28:43070) 2024-04-21T08:57:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9dbfff640 1 -- 172.21.15.28:0/2982057821 learned_addr learned my addr 172.21.15.28:0/2982057821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9d97fa640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff9c80a3da0 0x7ff9c80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9d2ffd640 1 --2- 172.21.15.28:0/2982057821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bf5a00 0x5589d3bf7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9dbfff640 1 -- 172.21.15.28:0/2982057821 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bece80 msgr2=0x5589d3bf5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9db7fe640 1 --2- 172.21.15.28:0/2982057821 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bece80 0x5589d3bf5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9dbfff640 1 --2- 172.21.15.28:0/2982057821 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bece80 0x5589d3bf5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9dbfff640 1 -- 172.21.15.28:0/2982057821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bf5a00 msgr2=0x5589d3bf7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9dbfff640 1 --2- 172.21.15.28:0/2982057821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bf5a00 0x5589d3bf7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9dbfff640 1 -- 172.21.15.28:0/2982057821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589d3ae01b0 con 0x5589d3bec4d0 2024-04-21T08:57:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9d2ffd640 1 --2- 172.21.15.28:0/2982057821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bf5a00 0x5589d3bf7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:57:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9db7fe640 1 --2- 172.21.15.28:0/2982057821 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bece80 0x5589d3bf5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9dbfff640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff9c80a3da0 0x7ff9c80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9d97fa640 1 -- 172.21.15.28:0/1478305417 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ff9c40d0570 con 0x7ff9dc107270 2024-04-21T08:57:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9dbfff640 1 --2- 172.21.15.28:0/2982057821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bec4d0 0x5589d3bec8b0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400c370 tx=0x7ff9d400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fe8ae4cca7638d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9d1ffb640 1 -- 172.21.15.28:0/2982057821 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4016070 con 0x5589d3bec4d0 2024-04-21T08:57:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9d1ffb640 1 -- 172.21.15.28:0/2982057821 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9d4011cd0 con 0x5589d3bec4d0 2024-04-21T08:57:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9d1ffb640 1 -- 172.21.15.28:0/2982057821 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d400fb40 con 0x5589d3bec4d0 2024-04-21T08:57:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.769+0000 7ff9dbfff640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff9c80a3da0 0x7ff9c80a6260 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff9d401d000 tx=0x7ff9d401e000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.770+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/2982057821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bec4d0 msgr2=0x5589d3bec8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.770+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/2982057821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bec4d0 0x5589d3bec8b0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400c370 tx=0x7ff9d400c830 comp rx=0 tx=0).stop 2024-04-21T08:57:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.770+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/2982057821 shutdown_connections 2024-04-21T08:57:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.770+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/2982057821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bf5a00 0x5589d3bf7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.770+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/2982057821 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bece80 0x5589d3bf5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.770+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/2982057821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bec4d0 0x5589d3bec8b0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.770+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/2982057821 >> 172.21.15.28:0/2982057821 conn(0x5589d3be6ba0 msgr2=0x5589d3be6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.770+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/2982057821 shutdown_connections 2024-04-21T08:57:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.770+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/2982057821 wait complete. 2024-04-21T08:57:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.770+0000 7ff9e4b6b740 1 -- start start 2024-04-21T08:57:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9e4b6b740 1 Processor -- start 2024-04-21T08:57:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9e4b6b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bec4d0 0x5589d3bf5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9e4b6b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bece80 0x5589d3bf26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9e4b6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bf5a00 0x5589d3bf2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9e4b6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5589d3bf8970 con 0x5589d3bf5a00 2024-04-21T08:57:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9e4b6b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5589d3b0a500 con 0x5589d3bece80 2024-04-21T08:57:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9e4b6b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5589d3bf8af0 con 0x5589d3bec4d0 2024-04-21T08:57:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9e4b6b740 10 client.0 _ll_register_callbacks cb 0x5589d3ad8b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bf5a00 0x5589d3bf2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bf5a00 0x5589d3bf2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43074/0 (socket says 172.21.15.28:43074) 2024-04-21T08:57:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9dbfff640 1 -- 172.21.15.28:0/658942248 learned_addr learned my addr 172.21.15.28:0/658942248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9db7fe640 1 --2- 172.21.15.28:0/658942248 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bec4d0 0x5589d3bf5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9d2ffd640 1 --2- 172.21.15.28:0/658942248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bece80 0x5589d3bf26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9dbfff640 1 -- 172.21.15.28:0/658942248 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bec4d0 msgr2=0x5589d3bf5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9dbfff640 1 --2- 172.21.15.28:0/658942248 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bec4d0 0x5589d3bf5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9dbfff640 1 -- 172.21.15.28:0/658942248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bece80 msgr2=0x5589d3bf26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9dbfff640 1 --2- 172.21.15.28:0/658942248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bece80 0x5589d3bf26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.771+0000 7ff9dbfff640 1 -- 172.21.15.28:0/658942248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589d38633f0 con 0x5589d3bf5a00 2024-04-21T08:57:35.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9dbfff640 1 --2- 172.21.15.28:0/658942248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bf5a00 0x5589d3bf2d00 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4011040 tx=0x7ff9d40075b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991f323b39bb604f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9baffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:35.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9baffd640 1 -- 172.21.15.28:0/658942248 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4016070 con 0x5589d3bf5a00 2024-04-21T08:57:35.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9baffd640 1 -- 172.21.15.28:0/658942248 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ff9d4011a30 con 0x5589d3bf5a00 2024-04-21T08:57:35.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/658942248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9d4011cd0 con 0x5589d3bf5a00 2024-04-21T08:57:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9baffd640 1 -- 172.21.15.28:0/658942248 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff9d4039050 con 0x5589d3bf5a00 2024-04-21T08:57:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9e4b6b740 10 client.17361 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/658942248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5589d392d7d0 con 0x5589d3bf5a00 2024-04-21T08:57:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9baffd640 1 -- 172.21.15.28:0/658942248 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9d4011cd0 con 0x5589d3bf5a00 2024-04-21T08:57:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9baffd640 1 -- 172.21.15.28:0/658942248 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7ff9d402e020 con 0x5589d3bf5a00 2024-04-21T08:57:35.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9e4b6b740 10 client.17361 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:35.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9e4b6b740 10 client.17361 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:57:35.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9e4b6b740 20 client.17361 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:35.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/658942248 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5589d3c05bc0 0x5589d3c07fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9e4b6b740 4 client.17361 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:57:35.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.772+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/658942248 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589d3af0930 con 0x5589d3c05bc0 2024-04-21T08:57:35.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.773+0000 7ff9db7fe640 1 --2- 172.21.15.28:0/658942248 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5589d3c05bc0 0x5589d3c07fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.773+0000 7ff9db7fe640 1 --2- 172.21.15.28:0/658942248 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5589d3c05bc0 0x5589d3c07fb0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=15012d922ca97b0c server_cookie=503bb2c03dbb4465 in_seq=0 out_seq=0 2024-04-21T08:57:35.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.773+0000 7ff9baffd640 10 client.17361 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:57:35.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.774+0000 7ff9baffd640 1 -- 172.21.15.28:0/658942248 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5589d3af0930 con 0x5589d3c05bc0 2024-04-21T08:57:35.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.774+0000 7ff9baffd640 10 client.17361 handle_command_reply: tid=0 2024-04-21T08:57:35.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.774+0000 7ff9e4b6b740 1 client.17361 shutdown 2024-04-21T08:57:35.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.774+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/658942248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bf5a00 msgr2=0x5589d3bf2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.774+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/658942248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bf5a00 0x5589d3bf2d00 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4011040 tx=0x7ff9d40075b0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.774+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/658942248 shutdown_connections 2024-04-21T08:57:35.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.774+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/658942248 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5589d3c05bc0 0x5589d3c07fb0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.774+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/658942248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bf5a00 0x5589d3bf2d00 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.774+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/658942248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bece80 0x5589d3bf26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.774+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/658942248 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bec4d0 0x5589d3bf5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.774+0000 7ff9baffd640 0 client.17361 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:57:35.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.774+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/658942248 >> 172.21.15.28:0/658942248 conn(0x5589d3be6ba0 msgr2=0x5589d3be88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.774+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/658942248 shutdown_connections 2024-04-21T08:57:35.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.774+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/658942248 wait complete. 2024-04-21T08:57:35.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.774+0000 7ff9e4b6b740 20 client.17361 trim_cache size 0 max 16384 2024-04-21T08:57:35.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9e4b6b740 1 Processor -- start 2024-04-21T08:57:35.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9e4b6b740 1 -- start start 2024-04-21T08:57:35.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9e4b6b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bec4d0 0x5589d3c01830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9e4b6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bece80 0x5589d3bf0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9e4b6b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bf5a00 0x5589d3bfe120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9e4b6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1b1bf0 con 0x5589d3bece80 2024-04-21T08:57:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9e4b6b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1af4c0 con 0x5589d3bf5a00 2024-04-21T08:57:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9e4b6b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1ad310 con 0x5589d3bec4d0 2024-04-21T08:57:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bece80 0x5589d3bf0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bece80 0x5589d3bf0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43084/0 (socket says 172.21.15.28:43084) 2024-04-21T08:57:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9d2ffd640 1 -- 172.21.15.28:0/1685968589 learned_addr learned my addr 172.21.15.28:0/1685968589 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9db7fe640 1 --2- 172.21.15.28:0/1685968589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bec4d0 0x5589d3c01830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9dbfff640 1 --2- 172.21.15.28:0/1685968589 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bf5a00 0x5589d3bfe120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9d2ffd640 1 -- 172.21.15.28:0/1685968589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bec4d0 msgr2=0x5589d3c01830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9d2ffd640 1 --2- 172.21.15.28:0/1685968589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bec4d0 0x5589d3c01830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9d2ffd640 1 -- 172.21.15.28:0/1685968589 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bf5a00 msgr2=0x5589d3bfe120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9d2ffd640 1 --2- 172.21.15.28:0/1685968589 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bf5a00 0x5589d3bfe120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9d2ffd640 1 -- 172.21.15.28:0/1685968589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589d3bf8850 con 0x5589d3bece80 2024-04-21T08:57:35.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.788+0000 7ff9dbfff640 1 --2- 172.21.15.28:0/1685968589 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bf5a00 0x5589d3bfe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:35.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.789+0000 7ff9d2ffd640 1 --2- 172.21.15.28:0/1685968589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bece80 0x5589d3bf0e90 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7ff9c40cdf20 tx=0x7ff9c409bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece0821adb7d31eb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.789+0000 7ff9d17fa640 1 -- 172.21.15.28:0/1685968589 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4020040 con 0x5589d3bece80 2024-04-21T08:57:35.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.789+0000 7ff9d17fa640 1 -- 172.21.15.28:0/1685968589 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9c40a0520 con 0x5589d3bece80 2024-04-21T08:57:35.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.789+0000 7ff9d17fa640 1 -- 172.21.15.28:0/1685968589 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c40d0170 con 0x5589d3bece80 2024-04-21T08:57:35.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.789+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/1685968589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bece80 msgr2=0x5589d3bf0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.789+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/1685968589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bece80 0x5589d3bf0e90 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7ff9c40cdf20 tx=0x7ff9c409bdd0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.789+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/1685968589 shutdown_connections 2024-04-21T08:57:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.789+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/1685968589 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bf5a00 0x5589d3bfe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.789+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/1685968589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bece80 0x5589d3bf0e90 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.789+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/1685968589 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bec4d0 0x5589d3c01830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.789+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/1685968589 >> 172.21.15.28:0/1685968589 conn(0x5589d3be6ba0 msgr2=0x5589d3be87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.789+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/1685968589 shutdown_connections 2024-04-21T08:57:35.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.789+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/1685968589 wait complete. 2024-04-21T08:57:35.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.790+0000 7ff9e4b6b740 1 -- start start 2024-04-21T08:57:35.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.790+0000 7ff9e4b6b740 1 Processor -- start 2024-04-21T08:57:35.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.790+0000 7ff9e4b6b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bec4d0 0x5589d3c64140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.790+0000 7ff9e4b6b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bece80 0x5589d3c64680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.790+0000 7ff9e4b6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bf5a00 0x5589d3c6efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.790+0000 7ff9e4b6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1ad980 con 0x5589d3bf5a00 2024-04-21T08:57:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.790+0000 7ff9e4b6b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1aedf0 con 0x5589d3bec4d0 2024-04-21T08:57:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.790+0000 7ff9e4b6b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1ae050 con 0x5589d3bece80 2024-04-21T08:57:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.790+0000 7ff9e4b6b740 10 client.0 _ll_register_callbacks cb 0x7ff9d4007280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.790+0000 7ff9dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bf5a00 0x5589d3c6efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.790+0000 7ff9dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bf5a00 0x5589d3c6efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43100/0 (socket says 172.21.15.28:43100) 2024-04-21T08:57:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.790+0000 7ff9dbfff640 1 -- 172.21.15.28:0/1316006537 learned_addr learned my addr 172.21.15.28:0/1316006537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.790+0000 7ff9d2ffd640 1 --2- 172.21.15.28:0/1316006537 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bece80 0x5589d3c64680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.790+0000 7ff9db7fe640 1 --2- 172.21.15.28:0/1316006537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bec4d0 0x5589d3c64140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9dbfff640 1 -- 172.21.15.28:0/1316006537 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bece80 msgr2=0x5589d3c64680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9dbfff640 1 --2- 172.21.15.28:0/1316006537 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bece80 0x5589d3c64680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9dbfff640 1 -- 172.21.15.28:0/1316006537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bec4d0 msgr2=0x5589d3c64140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9dbfff640 1 --2- 172.21.15.28:0/1316006537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bec4d0 0x5589d3c64140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9dbfff640 1 -- 172.21.15.28:0/1316006537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589d3c6f510 con 0x5589d3bf5a00 2024-04-21T08:57:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9d2ffd640 1 --2- 172.21.15.28:0/1316006537 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bece80 0x5589d3c64680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:35.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9db7fe640 1 --2- 172.21.15.28:0/1316006537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bec4d0 0x5589d3c64140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:35.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9dbfff640 1 --2- 172.21.15.28:0/1316006537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bf5a00 0x5589d3c6efd0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4030670 tx=0x7ff9d400ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a64ec5486331fe server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9baffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:35.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9baffd640 1 -- 172.21.15.28:0/1316006537 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4016070 con 0x5589d3bf5a00 2024-04-21T08:57:35.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/1316006537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9c40a0520 con 0x5589d3bf5a00 2024-04-21T08:57:35.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9baffd640 1 -- 172.21.15.28:0/1316006537 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7ff9d400f4b0 con 0x5589d3bf5a00 2024-04-21T08:57:35.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9baffd640 1 -- 172.21.15.28:0/1316006537 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff9d4007930 con 0x5589d3bf5a00 2024-04-21T08:57:35.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9e4b6b740 10 client.17367 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:35.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/1316006537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5589d3c70760 con 0x5589d3bf5a00 2024-04-21T08:57:35.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9baffd640 1 -- 172.21.15.28:0/1316006537 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9c40a0520 con 0x5589d3bf5a00 2024-04-21T08:57:35.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9baffd640 1 -- 172.21.15.28:0/1316006537 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7ff9d402e040 con 0x5589d3bf5a00 2024-04-21T08:57:35.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.791+0000 7ff9e4b6b740 10 client.17367 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:35.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.792+0000 7ff9e4b6b740 10 client.17367 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:57:35.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.792+0000 7ff9e4b6b740 20 client.17367 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:35.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.792+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/1316006537 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5589d3c70e20 0x5589d3c73210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.792+0000 7ff9e4b6b740 4 client.17367 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:57:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.792+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/1316006537 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5589d3b5ec10 con 0x5589d3c70e20 2024-04-21T08:57:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.792+0000 7ff9db7fe640 1 --2- 172.21.15.28:0/1316006537 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5589d3c70e20 0x5589d3c73210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.792+0000 7ff9db7fe640 1 --2- 172.21.15.28:0/1316006537 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5589d3c70e20 0x5589d3c73210 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=53c62ac29f2b4040 server_cookie=e402c62ed43ad0f5 in_seq=0 out_seq=0 2024-04-21T08:57:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.792+0000 7ff9baffd640 10 client.17367 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:57:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9baffd640 1 -- 172.21.15.28:0/1316006537 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5589d3b5ec10 con 0x5589d3c70e20 2024-04-21T08:57:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9baffd640 10 client.17367 handle_command_reply: tid=0 2024-04-21T08:57:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e4b6b740 1 client.17367 shutdown 2024-04-21T08:57:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/1316006537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bf5a00 msgr2=0x5589d3c6efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/1316006537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bf5a00 0x5589d3c6efd0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4030670 tx=0x7ff9d400ffd0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/1316006537 shutdown_connections 2024-04-21T08:57:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/1316006537 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5589d3c70e20 0x5589d3c73210 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/1316006537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589d3bf5a00 0x5589d3c6efd0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/1316006537 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5589d3bece80 0x5589d3c64680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9baffd640 0 client.17367 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:57:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e4b6b740 1 --2- 172.21.15.28:0/1316006537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5589d3bec4d0 0x5589d3c64140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/1316006537 >> 172.21.15.28:0/1316006537 conn(0x5589d3be6ba0 msgr2=0x5589d3bed2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/1316006537 shutdown_connections 2024-04-21T08:57:35.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e4b6b740 1 -- 172.21.15.28:0/1316006537 wait complete. 2024-04-21T08:57:35.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e4b6b740 20 client.17367 trim_cache size 0 max 16384 2024-04-21T08:57:35.851 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:57:35.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e1f7e640 1 -- 172.21.15.28:0/1478305417 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff9c80a3da0 msgr2=0x7ff9c80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e1f7e640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff9c80a3da0 0x7ff9c80a6260 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff9d401d000 tx=0x7ff9d401e000 comp rx=0 tx=0).stop 2024-04-21T08:57:35.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e1f7e640 1 -- 172.21.15.28:0/1478305417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9dc107270 msgr2=0x7ff9dc19e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.793+0000 7ff9e1f7e640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9dc107270 0x7ff9dc19e970 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400d740 tx=0x7ff9c400dc00 comp rx=0 tx=0).stop 2024-04-21T08:57:35.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.794+0000 7ff9e1f7e640 1 -- 172.21.15.28:0/1478305417 shutdown_connections 2024-04-21T08:57:35.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.794+0000 7ff9e1f7e640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff9c80a3da0 0x7ff9c80a6260 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.794+0000 7ff9e1f7e640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff9dc1985e0 0x7ff9dc198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.794+0000 7ff9e1f7e640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9dc107270 0x7ff9dc19e970 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.794+0000 7ff9e1f7e640 1 --2- 172.21.15.28:0/1478305417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff9dc06a840 0x7ff9dc19e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.794+0000 7ff9e1f7e640 1 -- 172.21.15.28:0/1478305417 >> 172.21.15.28:0/1478305417 conn(0x7ff9dc077eb0 msgr2=0x7ff9dc100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.794+0000 7ff9e1f7e640 1 -- 172.21.15.28:0/1478305417 shutdown_connections 2024-04-21T08:57:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.794+0000 7ff9e1f7e640 1 -- 172.21.15.28:0/1478305417 wait complete. 2024-04-21T08:57:35.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:57:35.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.959+0000 7fbc7939f640 1 -- 172.21.15.28:0/673649597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc74103af0 msgr2=0x7fbc7410bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.959+0000 7fbc7939f640 1 --2- 172.21.15.28:0/673649597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc74103af0 0x7fbc7410bfd0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fbc68009900 tx=0x7fbc6802bd60 comp rx=0 tx=0).stop 2024-04-21T08:57:35.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.959+0000 7fbc7939f640 1 -- 172.21.15.28:0/673649597 shutdown_connections 2024-04-21T08:57:35.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.959+0000 7fbc7939f640 1 --2- 172.21.15.28:0/673649597 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc7410c510 0x7fbc7410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.959+0000 7fbc7939f640 1 --2- 172.21.15.28:0/673649597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc74103af0 0x7fbc7410bfd0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.959+0000 7fbc7939f640 1 --2- 172.21.15.28:0/673649597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc741031d0 0x7fbc741035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.960+0000 7fbc7939f640 1 -- 172.21.15.28:0/673649597 >> 172.21.15.28:0/673649597 conn(0x7fbc74077f50 msgr2=0x7fbc74078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.960+0000 7fbc7939f640 1 -- 172.21.15.28:0/673649597 shutdown_connections 2024-04-21T08:57:35.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.960+0000 7fbc7939f640 1 -- 172.21.15.28:0/673649597 wait complete. 2024-04-21T08:57:35.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.960+0000 7fbc7939f640 1 Processor -- start 2024-04-21T08:57:35.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.960+0000 7fbc7939f640 1 -- start start 2024-04-21T08:57:35.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.960+0000 7fbc7939f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc741031d0 0x7fbc741986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.960+0000 7fbc7939f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc74103af0 0x7fbc74198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.960+0000 7fbc7939f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc7410c510 0x7fbc7419cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.960+0000 7fbc7939f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc74112180 con 0x7fbc741031d0 2024-04-21T08:57:35.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.960+0000 7fbc7939f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbc74112000 con 0x7fbc7410c510 2024-04-21T08:57:35.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.960+0000 7fbc7939f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbc74112300 con 0x7fbc74103af0 2024-04-21T08:57:35.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc72ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc741031d0 0x7fbc741986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc72ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc741031d0 0x7fbc741986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43110/0 (socket says 172.21.15.28:43110) 2024-04-21T08:57:35.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc72ffd640 1 -- 172.21.15.28:0/545691397 learned_addr learned my addr 172.21.15.28:0/545691397 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc737fe640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc7410c510 0x7fbc7419cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc727fc640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc74103af0 0x7fbc74198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc72ffd640 1 -- 172.21.15.28:0/545691397 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc74103af0 msgr2=0x7fbc74198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc72ffd640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc74103af0 0x7fbc74198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc72ffd640 1 -- 172.21.15.28:0/545691397 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc7410c510 msgr2=0x7fbc7419cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc72ffd640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc7410c510 0x7fbc7419cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc72ffd640 1 -- 172.21.15.28:0/545691397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7419d6d0 con 0x7fbc741031d0 2024-04-21T08:57:35.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc727fc640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc74103af0 0x7fbc74198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:35.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc737fe640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc7410c510 0x7fbc7419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:35.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc72ffd640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc741031d0 0x7fbc741986e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400c920 tx=0x7fbc6400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc5bfff640 1 -- 172.21.15.28:0/545691397 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc64007c70 con 0x7fbc741031d0 2024-04-21T08:57:35.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc5bfff640 1 -- 172.21.15.28:0/545691397 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc64007e10 con 0x7fbc741031d0 2024-04-21T08:57:35.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc7939f640 1 -- 172.21.15.28:0/545691397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7419d9c0 con 0x7fbc741031d0 2024-04-21T08:57:35.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc5bfff640 1 -- 172.21.15.28:0/545691397 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc64018820 con 0x7fbc741031d0 2024-04-21T08:57:35.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.961+0000 7fbc7939f640 1 -- 172.21.15.28:0/545691397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc741a5250 con 0x7fbc741031d0 2024-04-21T08:57:35.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.963+0000 7fbc5bfff640 1 -- 172.21.15.28:0/545691397 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbc6400f450 con 0x7fbc741031d0 2024-04-21T08:57:35.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc7bf8b740 1 Processor -- start 2024-04-21T08:57:35.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc7bf8b740 1 -- start start 2024-04-21T08:57:35.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc7bf8b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f4744d0 0x558d6f4748b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc7bf8b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f474e80 0x558d6f47d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc7bf8b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f47da00 0x558d6f47fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc7bf8b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558d6f3d91f0 con 0x558d6f4744d0 2024-04-21T08:57:35.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc7bf8b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558d6f3764b0 con 0x558d6f47da00 2024-04-21T08:57:35.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc7bf8b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558d6f3d51b0 con 0x558d6f474e80 2024-04-21T08:57:35.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f4744d0 0x558d6f4748b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f4744d0 0x558d6f4748b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43124/0 (socket says 172.21.15.28:43124) 2024-04-21T08:57:35.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc737fe640 1 -- 172.21.15.28:0/2322374781 learned_addr learned my addr 172.21.15.28:0/2322374781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc5bfff640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbc5c0a3d20 0x7fbc5c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc737fe640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbc5c0a3d20 0x7fbc5c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc5bfff640 1 -- 172.21.15.28:0/545691397 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbc640d0d70 con 0x7fbc741031d0 2024-04-21T08:57:35.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc72ffd640 1 --2- 172.21.15.28:0/2322374781 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f474e80 0x558d6f47d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc737fe640 1 -- 172.21.15.28:0/2322374781 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f474e80 msgr2=0x558d6f47d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc737fe640 1 --2- 172.21.15.28:0/2322374781 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f474e80 0x558d6f47d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc737fe640 1 -- 172.21.15.28:0/2322374781 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f47da00 msgr2=0x558d6f47fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc737fe640 1 --2- 172.21.15.28:0/2322374781 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f47da00 0x558d6f47fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.964+0000 7fbc737fe640 1 -- 172.21.15.28:0/2322374781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d6f3681b0 con 0x558d6f4744d0 2024-04-21T08:57:35.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.965+0000 7fbc737fe640 1 --2- 172.21.15.28:0/2322374781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f4744d0 0x558d6f4748b0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c012990 tx=0x7fbc6c012e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c38c3b9dc90e2d19 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.965+0000 7fbc597fa640 1 -- 172.21.15.28:0/2322374781 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c022070 con 0x558d6f4744d0 2024-04-21T08:57:35.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.965+0000 7fbc597fa640 1 -- 172.21.15.28:0/2322374781 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc6c0123b0 con 0x558d6f4744d0 2024-04-21T08:57:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.965+0000 7fbc597fa640 1 -- 172.21.15.28:0/2322374781 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c005740 con 0x558d6f4744d0 2024-04-21T08:57:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.965+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2322374781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f4744d0 msgr2=0x558d6f4748b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.965+0000 7fbc737fe640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbc5c0a3d20 0x7fbc5c0a61e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c01f000 tx=0x7fbc6c01b000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.965+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2322374781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f4744d0 0x558d6f4748b0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c012990 tx=0x7fbc6c012e50 comp rx=0 tx=0).stop 2024-04-21T08:57:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.965+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2322374781 shutdown_connections 2024-04-21T08:57:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.965+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2322374781 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f47da00 0x558d6f47fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.965+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2322374781 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f474e80 0x558d6f47d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.965+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2322374781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f4744d0 0x558d6f4748b0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.965+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2322374781 >> 172.21.15.28:0/2322374781 conn(0x558d6f46eba0 msgr2=0x558d6f46ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2322374781 shutdown_connections 2024-04-21T08:57:35.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2322374781 wait complete. 2024-04-21T08:57:35.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc7bf8b740 1 -- start start 2024-04-21T08:57:35.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc7bf8b740 1 Processor -- start 2024-04-21T08:57:35.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc7bf8b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f4744d0 0x558d6f47d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc7bf8b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f474e80 0x558d6f47a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc7bf8b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f47da00 0x558d6f47ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc7bf8b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558d6f480970 con 0x558d6f474e80 2024-04-21T08:57:35.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc7bf8b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558d6f392500 con 0x558d6f4744d0 2024-04-21T08:57:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc7bf8b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558d6f480af0 con 0x558d6f47da00 2024-04-21T08:57:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc7bf8b740 10 client.0 _ll_register_callbacks cb 0x558d6f360b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc727fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f474e80 0x558d6f47a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc727fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f474e80 0x558d6f47a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43128/0 (socket says 172.21.15.28:43128) 2024-04-21T08:57:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc727fc640 1 -- 172.21.15.28:0/2885071679 learned_addr learned my addr 172.21.15.28:0/2885071679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc727fc640 1 -- 172.21.15.28:0/2885071679 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f47da00 msgr2=0x558d6f47ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc727fc640 1 --2- 172.21.15.28:0/2885071679 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f47da00 0x558d6f47ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc727fc640 1 -- 172.21.15.28:0/2885071679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f4744d0 msgr2=0x558d6f47d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc727fc640 1 --2- 172.21.15.28:0/2885071679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f4744d0 0x558d6f47d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.966+0000 7fbc727fc640 1 -- 172.21.15.28:0/2885071679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d6f0eb3f0 con 0x558d6f474e80 2024-04-21T08:57:35.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.967+0000 7fbc727fc640 1 --2- 172.21.15.28:0/2885071679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f474e80 0x558d6f47a600 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fbc680092a0 tx=0x7fbc6802db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6253d0af50058cb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.967+0000 7fbc56ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:35.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.967+0000 7fbc56ff5640 1 -- 172.21.15.28:0/2885071679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc68042020 con 0x558d6f474e80 2024-04-21T08:57:35.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.967+0000 7fbc56ff5640 1 -- 172.21.15.28:0/2885071679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbc68005250 con 0x558d6f474e80 2024-04-21T08:57:35.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.967+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2885071679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558d6f378930 con 0x558d6f474e80 2024-04-21T08:57:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.967+0000 7fbc56ff5640 1 -- 172.21.15.28:0/2885071679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbc6804d050 con 0x558d6f474e80 2024-04-21T08:57:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.967+0000 7fbc7bf8b740 10 client.17379 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.967+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2885071679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558d6f1b57d0 con 0x558d6f474e80 2024-04-21T08:57:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.967+0000 7fbc56ff5640 1 -- 172.21.15.28:0/2885071679 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558d6f378930 con 0x558d6f474e80 2024-04-21T08:57:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.967+0000 7fbc56ff5640 1 -- 172.21.15.28:0/2885071679 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fbc680035f0 con 0x558d6f474e80 2024-04-21T08:57:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.967+0000 7fbc7bf8b740 10 client.17379 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.967+0000 7fbc7bf8b740 10 client.17379 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:57:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.967+0000 7fbc7bf8b740 20 client.17379 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.967+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2885071679 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558d6f48dbc0 0x558d6f490060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.968+0000 7fbc7bf8b740 4 client.17379 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:57:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.968+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2885071679 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d6f3e6c10 con 0x558d6f48dbc0 2024-04-21T08:57:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.968+0000 7fbc72ffd640 1 --2- 172.21.15.28:0/2885071679 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558d6f48dbc0 0x558d6f490060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.968+0000 7fbc72ffd640 1 --2- 172.21.15.28:0/2885071679 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558d6f48dbc0 0x558d6f490060 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8f3b14a722a7cbbc server_cookie=467c12125d1dbe19 in_seq=0 out_seq=0 2024-04-21T08:57:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.968+0000 7fbc56ff5640 10 client.17379 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:57:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.969+0000 7fbc56ff5640 1 -- 172.21.15.28:0/2885071679 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d6f3e6c10 con 0x558d6f48dbc0 2024-04-21T08:57:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.969+0000 7fbc56ff5640 10 client.17379 handle_command_reply: tid=0 2024-04-21T08:57:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.969+0000 7fbc7bf8b740 1 client.17379 shutdown 2024-04-21T08:57:35.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.969+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2885071679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f474e80 msgr2=0x558d6f47a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.969+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2885071679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f474e80 0x558d6f47a600 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fbc680092a0 tx=0x7fbc6802db50 comp rx=0 tx=0).stop 2024-04-21T08:57:35.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.969+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2885071679 shutdown_connections 2024-04-21T08:57:35.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.969+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2885071679 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558d6f48dbc0 0x558d6f490060 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.969+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2885071679 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f47da00 0x558d6f47ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.969+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2885071679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f474e80 0x558d6f47a600 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.969+0000 7fbc56ff5640 0 client.17379 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:57:35.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.969+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2885071679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f4744d0 0x558d6f47d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.969+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2885071679 >> 172.21.15.28:0/2885071679 conn(0x558d6f46eba0 msgr2=0x558d6f4708f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.969+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2885071679 shutdown_connections 2024-04-21T08:57:35.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.969+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2885071679 wait complete. 2024-04-21T08:57:35.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.970+0000 7fbc7bf8b740 20 client.17379 trim_cache size 0 max 16384 2024-04-21T08:57:35.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc7bf8b740 1 Processor -- start 2024-04-21T08:57:35.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc7bf8b740 1 -- start start 2024-04-21T08:57:35.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc7bf8b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f4744d0 0x558d6f489830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc7bf8b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f474e80 0x558d6f47cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc7bf8b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f47da00 0x558d6f47d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc7bf8b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc741b1c20 con 0x558d6f47da00 2024-04-21T08:57:35.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc7bf8b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbc741af4f0 con 0x558d6f4744d0 2024-04-21T08:57:35.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc7bf8b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbc741ad340 con 0x558d6f474e80 2024-04-21T08:57:35.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f47da00 0x558d6f47d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f47da00 0x558d6f47d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43140/0 (socket says 172.21.15.28:43140) 2024-04-21T08:57:35.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc737fe640 1 -- 172.21.15.28:0/2121898468 learned_addr learned my addr 172.21.15.28:0/2121898468 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc737fe640 1 -- 172.21.15.28:0/2121898468 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f474e80 msgr2=0x558d6f47cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc737fe640 1 --2- 172.21.15.28:0/2121898468 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f474e80 0x558d6f47cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc737fe640 1 -- 172.21.15.28:0/2121898468 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f4744d0 msgr2=0x558d6f489830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc72ffd640 1 --2- 172.21.15.28:0/2121898468 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f4744d0 0x558d6f489830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc737fe640 1 --2- 172.21.15.28:0/2121898468 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f4744d0 0x558d6f489830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc737fe640 1 -- 172.21.15.28:0/2121898468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d6f46f4e0 con 0x558d6f47da00 2024-04-21T08:57:35.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.983+0000 7fbc72ffd640 1 --2- 172.21.15.28:0/2121898468 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f4744d0 0x558d6f489830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:35.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.984+0000 7fbc737fe640 1 --2- 172.21.15.28:0/2121898468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f47da00 0x558d6f47d270 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c006010 tx=0x7fbc6c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67d6b8bee4c82d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.984+0000 7fbc58ff9640 1 -- 172.21.15.28:0/2121898468 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c033070 con 0x558d6f47da00 2024-04-21T08:57:35.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.984+0000 7fbc58ff9640 1 -- 172.21.15.28:0/2121898468 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc6c004980 con 0x558d6f47da00 2024-04-21T08:57:35.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.984+0000 7fbc58ff9640 1 -- 172.21.15.28:0/2121898468 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c022050 con 0x558d6f47da00 2024-04-21T08:57:35.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.984+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2121898468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f47da00 msgr2=0x558d6f47d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.984+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2121898468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f47da00 0x558d6f47d270 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c006010 tx=0x7fbc6c009840 comp rx=0 tx=0).stop 2024-04-21T08:57:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.984+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2121898468 shutdown_connections 2024-04-21T08:57:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.984+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2121898468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f47da00 0x558d6f47d270 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.984+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2121898468 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f474e80 0x558d6f47cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.984+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2121898468 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f4744d0 0x558d6f489830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.984+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2121898468 >> 172.21.15.28:0/2121898468 conn(0x558d6f46eba0 msgr2=0x558d6f46dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:35.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.984+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2121898468 shutdown_connections 2024-04-21T08:57:35.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.984+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2121898468 wait complete. 2024-04-21T08:57:35.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc7bf8b740 1 -- start start 2024-04-21T08:57:35.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc7bf8b740 1 Processor -- start 2024-04-21T08:57:35.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc7bf8b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f4744d0 0x558d6f481910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc7bf8b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f474e80 0x558d6f481e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc7bf8b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f47da00 0x558d6f482390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc7bf8b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc741ad9b0 con 0x558d6f47da00 2024-04-21T08:57:35.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc7bf8b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fbc741aee20 con 0x558d6f474e80 2024-04-21T08:57:35.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc7bf8b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fbc741ae080 con 0x558d6f4744d0 2024-04-21T08:57:35.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc7bf8b740 10 client.0 _ll_register_callbacks cb 0x7fbc68004540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:35.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f47da00 0x558d6f482390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f47da00 0x558d6f482390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43146/0 (socket says 172.21.15.28:43146) 2024-04-21T08:57:35.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc737fe640 1 -- 172.21.15.28:0/2720064773 learned_addr learned my addr 172.21.15.28:0/2720064773 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:35.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc72ffd640 1 --2- 172.21.15.28:0/2720064773 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f4744d0 0x558d6f481910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc727fc640 1 --2- 172.21.15.28:0/2720064773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f474e80 0x558d6f481e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc737fe640 1 -- 172.21.15.28:0/2720064773 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f4744d0 msgr2=0x558d6f481910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc737fe640 1 --2- 172.21.15.28:0/2720064773 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f4744d0 0x558d6f481910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc737fe640 1 -- 172.21.15.28:0/2720064773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f474e80 msgr2=0x558d6f481e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.985+0000 7fbc737fe640 1 --2- 172.21.15.28:0/2720064773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f474e80 0x558d6f481e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc737fe640 1 -- 172.21.15.28:0/2720064773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d6f4f33a0 con 0x558d6f47da00 2024-04-21T08:57:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc727fc640 1 --2- 172.21.15.28:0/2720064773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f474e80 0x558d6f481e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:57:35.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc737fe640 1 --2- 172.21.15.28:0/2720064773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f47da00 0x558d6f482390 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c004de0 tx=0x7fbc6c009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ceb97c4cac4c1cc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc56ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc56ff5640 1 -- 172.21.15.28:0/2720064773 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c03e060 con 0x558d6f47da00 2024-04-21T08:57:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc56ff5640 1 -- 172.21.15.28:0/2720064773 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7fbc6c01db80 con 0x558d6f47da00 2024-04-21T08:57:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2720064773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558d6f47aed0 con 0x558d6f47da00 2024-04-21T08:57:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc56ff5640 1 -- 172.21.15.28:0/2720064773 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbc6c022020 con 0x558d6f47da00 2024-04-21T08:57:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc7bf8b740 10 client.17385 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2720064773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558d6f4f45b0 con 0x558d6f47da00 2024-04-21T08:57:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc56ff5640 1 -- 172.21.15.28:0/2720064773 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558d6f47aed0 con 0x558d6f47da00 2024-04-21T08:57:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc56ff5640 1 -- 172.21.15.28:0/2720064773 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fbc6c026440 con 0x558d6f47da00 2024-04-21T08:57:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc7bf8b740 10 client.17385 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc7bf8b740 10 client.17385 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:57:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc7bf8b740 20 client.17385 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2720064773 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558d6f4f4a20 0x558d6f4f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc7bf8b740 4 client.17385 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:57:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.986+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2720064773 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558d6f3e0a80 con 0x558d6f4f4a20 2024-04-21T08:57:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.987+0000 7fbc72ffd640 1 --2- 172.21.15.28:0/2720064773 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558d6f4f4a20 0x558d6f4f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.987+0000 7fbc72ffd640 1 --2- 172.21.15.28:0/2720064773 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558d6f4f4a20 0x558d6f4f6e10 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1518fc6ee2661c61 server_cookie=bcdef50dafdbcef9 in_seq=0 out_seq=0 2024-04-21T08:57:35.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.987+0000 7fbc56ff5640 10 client.17385 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:57:35.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.987+0000 7fbc56ff5640 1 -- 172.21.15.28:0/2720064773 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558d6f3e0a80 con 0x558d6f4f4a20 2024-04-21T08:57:35.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.987+0000 7fbc56ff5640 10 client.17385 handle_command_reply: tid=0 2024-04-21T08:57:35.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.987+0000 7fbc7bf8b740 1 client.17385 shutdown 2024-04-21T08:57:35.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.987+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2720064773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f47da00 msgr2=0x558d6f482390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:35.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.987+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2720064773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f47da00 0x558d6f482390 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c004de0 tx=0x7fbc6c009520 comp rx=0 tx=0).stop 2024-04-21T08:57:35.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2720064773 shutdown_connections 2024-04-21T08:57:35.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2720064773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d6f47da00 0x558d6f482390 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2720064773 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558d6f474e80 0x558d6f481e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2720064773 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558d6f4f4a20 0x558d6f4f6e10 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7bf8b740 1 --2- 172.21.15.28:0/2720064773 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558d6f4744d0 0x558d6f481910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc56ff5640 0 client.17385 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:57:36.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2720064773 >> 172.21.15.28:0/2720064773 conn(0x558d6f46eba0 msgr2=0x558d6f46ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:36.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2720064773 shutdown_connections 2024-04-21T08:57:36.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7bf8b740 1 -- 172.21.15.28:0/2720064773 wait complete. 2024-04-21T08:57:36.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7bf8b740 20 client.17385 trim_cache size 0 max 16384 2024-04-21T08:57:36.001 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:57:36.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7939f640 1 -- 172.21.15.28:0/545691397 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbc5c0a3d20 msgr2=0x7fbc5c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:36.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7939f640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbc5c0a3d20 0x7fbc5c0a61e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c01f000 tx=0x7fbc6c01b000 comp rx=0 tx=0).stop 2024-04-21T08:57:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7939f640 1 -- 172.21.15.28:0/545691397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc741031d0 msgr2=0x7fbc741986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7939f640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc741031d0 0x7fbc741986e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400c920 tx=0x7fbc6400cde0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7939f640 1 -- 172.21.15.28:0/545691397 shutdown_connections 2024-04-21T08:57:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7939f640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fbc5c0a3d20 0x7fbc5c0a61e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7939f640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbc7410c510 0x7fbc7419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7939f640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc74103af0 0x7fbc74198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7939f640 1 --2- 172.21.15.28:0/545691397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc741031d0 0x7fbc741986e0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7939f640 1 -- 172.21.15.28:0/545691397 >> 172.21.15.28:0/545691397 conn(0x7fbc74077f50 msgr2=0x7fbc740fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:36.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7939f640 1 -- 172.21.15.28:0/545691397 shutdown_connections 2024-04-21T08:57:36.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:35.988+0000 7fbc7939f640 1 -- 172.21.15.28:0/545691397 wait complete. 2024-04-21T08:57:36.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:57:36.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 -- 172.21.15.28:0/1922214377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b0c1031d0 msgr2=0x7f2b0c1035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 --2- 172.21.15.28:0/1922214377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b0c1031d0 0x7f2b0c1035b0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f2afc009900 tx=0x7f2afc02bd60 comp rx=0 tx=0).stop 2024-04-21T08:57:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 -- 172.21.15.28:0/1922214377 shutdown_connections 2024-04-21T08:57:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 --2- 172.21.15.28:0/1922214377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b0c10c510 0x7f2b0c10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 --2- 172.21.15.28:0/1922214377 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b0c103af0 0x7f2b0c10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 --2- 172.21.15.28:0/1922214377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b0c1031d0 0x7f2b0c1035b0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 -- 172.21.15.28:0/1922214377 >> 172.21.15.28:0/1922214377 conn(0x7f2b0c077f50 msgr2=0x7f2b0c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 -- 172.21.15.28:0/1922214377 shutdown_connections 2024-04-21T08:57:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 -- 172.21.15.28:0/1922214377 wait complete. 2024-04-21T08:57:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 Processor -- start 2024-04-21T08:57:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 -- start start 2024-04-21T08:57:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b0c1031d0 0x7f2b0c1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b0c103af0 0x7f2b0c198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b0c10c510 0x7f2b0c19d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c1121d0 con 0x7f2b0c1031d0 2024-04-21T08:57:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c112050 con 0x7f2b0c10c510 2024-04-21T08:57:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.110+0000 7f2b1312b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c112350 con 0x7f2b0c103af0 2024-04-21T08:57:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.111+0000 7f2b10ec7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b0c1031d0 0x7f2b0c1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.111+0000 7f2b10ec7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b0c1031d0 0x7f2b0c1987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43168/0 (socket says 172.21.15.28:43168) 2024-04-21T08:57:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.111+0000 7f2b10ec7640 1 -- 172.21.15.28:0/321691817 learned_addr learned my addr 172.21.15.28:0/321691817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.111+0000 7f2b116c8640 1 --2- 172.21.15.28:0/321691817 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b0c10c510 0x7f2b0c19d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.111+0000 7f2b10ec7640 1 -- 172.21.15.28:0/321691817 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b0c103af0 msgr2=0x7f2b0c198cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.111+0000 7f2b10ec7640 1 --2- 172.21.15.28:0/321691817 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b0c103af0 0x7f2b0c198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.111+0000 7f2b10ec7640 1 -- 172.21.15.28:0/321691817 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b0c10c510 msgr2=0x7f2b0c19d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.111+0000 7f2b10ec7640 1 --2- 172.21.15.28:0/321691817 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b0c10c510 0x7f2b0c19d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.111+0000 7f2b10ec7640 1 -- 172.21.15.28:0/321691817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0c19d750 con 0x7f2b0c1031d0 2024-04-21T08:57:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.111+0000 7f2b10ec7640 1 --2- 172.21.15.28:0/321691817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b0c1031d0 0x7f2b0c1987b0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f2afc000c00 tx=0x7f2afc02d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.111+0000 7f2b09ffb640 1 -- 172.21.15.28:0/321691817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc043020 con 0x7f2b0c1031d0 2024-04-21T08:57:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.111+0000 7f2b09ffb640 1 -- 172.21.15.28:0/321691817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2afc02dde0 con 0x7f2b0c1031d0 2024-04-21T08:57:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.111+0000 7f2b1312b640 1 -- 172.21.15.28:0/321691817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b0c19d9e0 con 0x7f2b0c1031d0 2024-04-21T08:57:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.111+0000 7f2b09ffb640 1 -- 172.21.15.28:0/321691817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc003b80 con 0x7f2b0c1031d0 2024-04-21T08:57:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.111+0000 7f2b1312b640 1 -- 172.21.15.28:0/321691817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b0c1a52d0 con 0x7f2b0c1031d0 2024-04-21T08:57:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.114+0000 7f2b15df6740 1 Processor -- start 2024-04-21T08:57:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.114+0000 7f2b15df6740 1 -- start start 2024-04-21T08:57:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.114+0000 7f2b09ffb640 1 -- 172.21.15.28:0/321691817 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2afc046020 con 0x7f2b0c1031d0 2024-04-21T08:57:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b15df6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910330 0x5602e9910710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b15df6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910ce0 0x5602e9919820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b15df6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e991a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b15df6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5602e98751f0 con 0x5602e9919e40 2024-04-21T08:57:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b15df6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5602e98124b0 con 0x5602e9910330 2024-04-21T08:57:36.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b15df6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5602e98711b0 con 0x5602e9910ce0 2024-04-21T08:57:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b0bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e991a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b0bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e991a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43178/0 (socket says 172.21.15.28:43178) 2024-04-21T08:57:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b0bfff640 1 -- 172.21.15.28:0/747615272 learned_addr learned my addr 172.21.15.28:0/747615272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b116c8640 1 --2- 172.21.15.28:0/747615272 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910330 0x5602e9910710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b10ec7640 1 --2- 172.21.15.28:0/747615272 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910ce0 0x5602e9919820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b0bfff640 1 -- 172.21.15.28:0/747615272 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910ce0 msgr2=0x5602e9919820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b0bfff640 1 --2- 172.21.15.28:0/747615272 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910ce0 0x5602e9919820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b0bfff640 1 -- 172.21.15.28:0/747615272 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910330 msgr2=0x5602e9910710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b0bfff640 1 --2- 172.21.15.28:0/747615272 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910330 0x5602e9910710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b0bfff640 1 -- 172.21.15.28:0/747615272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602e98041b0 con 0x5602e9919e40 2024-04-21T08:57:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b10ec7640 1 --2- 172.21.15.28:0/747615272 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910ce0 0x5602e9919820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b09ffb640 1 --2- 172.21.15.28:0/321691817 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2af40a3d20 0x7f2af40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b116c8640 1 --2- 172.21.15.28:0/321691817 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2af40a3d20 0x7f2af40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b0bfff640 1 --2- 172.21.15.28:0/747615272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e991a2c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f2b0000bae0 tx=0x7f2b0000d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a4ccf54684e95be server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.115+0000 7f2b09ffb640 1 -- 172.21.15.28:0/321691817 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f2afc0f2cf0 con 0x7f2b0c1031d0 2024-04-21T08:57:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.116+0000 7f2af2ffd640 1 -- 172.21.15.28:0/747615272 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b00016070 con 0x5602e9919e40 2024-04-21T08:57:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.116+0000 7f2af2ffd640 1 -- 172.21.15.28:0/747615272 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b000054d0 con 0x5602e9919e40 2024-04-21T08:57:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.116+0000 7f2af2ffd640 1 -- 172.21.15.28:0/747615272 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b00004e60 con 0x5602e9919e40 2024-04-21T08:57:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.116+0000 7f2b15df6740 1 -- 172.21.15.28:0/747615272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 msgr2=0x5602e991a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.116+0000 7f2b116c8640 1 --2- 172.21.15.28:0/321691817 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2af40a3d20 0x7f2af40a61e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2b04005fd0 tx=0x7f2b040096e0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.116+0000 7f2b15df6740 1 --2- 172.21.15.28:0/747615272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e991a2c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f2b0000bae0 tx=0x7f2b0000d2e0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.116+0000 7f2b15df6740 1 -- 172.21.15.28:0/747615272 shutdown_connections 2024-04-21T08:57:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.116+0000 7f2b15df6740 1 --2- 172.21.15.28:0/747615272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e991a2c0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.116+0000 7f2b15df6740 1 --2- 172.21.15.28:0/747615272 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910ce0 0x5602e9919820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.116+0000 7f2b15df6740 1 --2- 172.21.15.28:0/747615272 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910330 0x5602e9910710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.116+0000 7f2b15df6740 1 -- 172.21.15.28:0/747615272 >> 172.21.15.28:0/747615272 conn(0x5602e990aba0 msgr2=0x5602e990af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.116+0000 7f2b15df6740 1 -- 172.21.15.28:0/747615272 shutdown_connections 2024-04-21T08:57:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.116+0000 7f2b15df6740 1 -- 172.21.15.28:0/747615272 wait complete. 2024-04-21T08:57:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.117+0000 7f2b15df6740 1 -- start start 2024-04-21T08:57:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.117+0000 7f2b15df6740 1 Processor -- start 2024-04-21T08:57:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.117+0000 7f2b15df6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910330 0x5602e99201f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.117+0000 7f2b15df6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910ce0 0x5602e991d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.117+0000 7f2b15df6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e991dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.117+0000 7f2b15df6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5602e9920dc0 con 0x5602e9919e40 2024-04-21T08:57:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.117+0000 7f2b15df6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5602e982e500 con 0x5602e9910ce0 2024-04-21T08:57:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.117+0000 7f2b15df6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5602e9920f40 con 0x5602e9910330 2024-04-21T08:57:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.117+0000 7f2b15df6740 10 client.0 _ll_register_callbacks cb 0x5602e97fcb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.117+0000 7f2b116c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e991dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.117+0000 7f2b116c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e991dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43192/0 (socket says 172.21.15.28:43192) 2024-04-21T08:57:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.117+0000 7f2b116c8640 1 -- 172.21.15.28:0/2543937868 learned_addr learned my addr 172.21.15.28:0/2543937868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.117+0000 7f2b10ec7640 1 --2- 172.21.15.28:0/2543937868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910330 0x5602e99201f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2b0bfff640 1 --2- 172.21.15.28:0/2543937868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910ce0 0x5602e991d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2b116c8640 1 -- 172.21.15.28:0/2543937868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910330 msgr2=0x5602e99201f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2b116c8640 1 --2- 172.21.15.28:0/2543937868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910330 0x5602e99201f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2b116c8640 1 -- 172.21.15.28:0/2543937868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910ce0 msgr2=0x5602e991d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2b116c8640 1 --2- 172.21.15.28:0/2543937868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910ce0 0x5602e991d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2b116c8640 1 -- 172.21.15.28:0/2543937868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602e95873f0 con 0x5602e9919e40 2024-04-21T08:57:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2b10ec7640 1 --2- 172.21.15.28:0/2543937868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910330 0x5602e99201f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:57:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2b0bfff640 1 --2- 172.21.15.28:0/2543937868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910ce0 0x5602e991d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2b116c8640 1 --2- 172.21.15.28:0/2543937868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e991dac0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f2b0401d850 tx=0x7f2b04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66822de23f1f665 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2aebfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2aebfff640 1 -- 172.21.15.28:0/2543937868 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0401e820 con 0x5602e9919e40 2024-04-21T08:57:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2b15df6740 1 -- 172.21.15.28:0/2543937868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b000054d0 con 0x5602e9919e40 2024-04-21T08:57:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2aebfff640 1 -- 172.21.15.28:0/2543937868 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f2b0401e9c0 con 0x5602e9919e40 2024-04-21T08:57:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2aebfff640 1 -- 172.21.15.28:0/2543937868 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2b0401ebc0 con 0x5602e9919e40 2024-04-21T08:57:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2b15df6740 10 client.17397 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2b15df6740 1 -- 172.21.15.28:0/2543937868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5602e96517d0 con 0x5602e9919e40 2024-04-21T08:57:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2aebfff640 1 -- 172.21.15.28:0/2543937868 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b000054d0 con 0x5602e9919e40 2024-04-21T08:57:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.118+0000 7f2aebfff640 1 -- 172.21.15.28:0/2543937868 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f2b0400f040 con 0x5602e9919e40 2024-04-21T08:57:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.119+0000 7f2b15df6740 10 client.17397 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.119+0000 7f2b15df6740 10 client.17397 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.119+0000 7f2b15df6740 20 client.17397 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.119+0000 7f2b15df6740 1 --2- 172.21.15.28:0/2543937868 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5602e992dfd0 0x5602e9930450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.119+0000 7f2b15df6740 4 client.17397 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:57:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.119+0000 7f2b15df6740 1 -- 172.21.15.28:0/2543937868 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602e9814930 con 0x5602e992dfd0 2024-04-21T08:57:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.119+0000 7f2b10ec7640 1 --2- 172.21.15.28:0/2543937868 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5602e992dfd0 0x5602e9930450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.119+0000 7f2b10ec7640 1 --2- 172.21.15.28:0/2543937868 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5602e992dfd0 0x5602e9930450 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a91a661364066c7 server_cookie=989f69a131249548 in_seq=0 out_seq=0 2024-04-21T08:57:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.119+0000 7f2aebfff640 10 client.17397 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.120+0000 7f2aebfff640 1 -- 172.21.15.28:0/2543937868 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602e9814930 con 0x5602e992dfd0 2024-04-21T08:57:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.120+0000 7f2aebfff640 10 client.17397 handle_command_reply: tid=0 2024-04-21T08:57:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.120+0000 7f2b15df6740 1 client.17397 shutdown 2024-04-21T08:57:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.120+0000 7f2b15df6740 1 -- 172.21.15.28:0/2543937868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 msgr2=0x5602e991dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.120+0000 7f2b15df6740 1 --2- 172.21.15.28:0/2543937868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e991dac0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f2b0401d850 tx=0x7f2b04004a40 comp rx=0 tx=0).stop 2024-04-21T08:57:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.120+0000 7f2b15df6740 1 -- 172.21.15.28:0/2543937868 shutdown_connections 2024-04-21T08:57:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.120+0000 7f2b15df6740 1 --2- 172.21.15.28:0/2543937868 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5602e992dfd0 0x5602e9930450 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.120+0000 7f2b15df6740 1 --2- 172.21.15.28:0/2543937868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e991dac0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.120+0000 7f2b15df6740 1 --2- 172.21.15.28:0/2543937868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910ce0 0x5602e991d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.120+0000 7f2aebfff640 0 client.17397 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.120+0000 7f2b15df6740 1 --2- 172.21.15.28:0/2543937868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910330 0x5602e99201f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.120+0000 7f2b15df6740 1 -- 172.21.15.28:0/2543937868 >> 172.21.15.28:0/2543937868 conn(0x5602e990aba0 msgr2=0x5602e990c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.120+0000 7f2b15df6740 1 -- 172.21.15.28:0/2543937868 shutdown_connections 2024-04-21T08:57:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.120+0000 7f2b15df6740 1 -- 172.21.15.28:0/2543937868 wait complete. 2024-04-21T08:57:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.120+0000 7f2b15df6740 20 client.17397 trim_cache size 0 max 16384 2024-04-21T08:57:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b15df6740 1 Processor -- start 2024-04-21T08:57:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b15df6740 1 -- start start 2024-04-21T08:57:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b15df6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910330 0x5602e9929cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b15df6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910ce0 0x5602e991fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b15df6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e99201f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b15df6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c1afa10 con 0x5602e9919e40 2024-04-21T08:57:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b15df6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c1ad2e0 con 0x5602e9910330 2024-04-21T08:57:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b15df6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c1ab130 con 0x5602e9910ce0 2024-04-21T08:57:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b116c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e99201f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b116c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e99201f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43200/0 (socket says 172.21.15.28:43200) 2024-04-21T08:57:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b116c8640 1 -- 172.21.15.28:0/273739616 learned_addr learned my addr 172.21.15.28:0/273739616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b0bfff640 1 --2- 172.21.15.28:0/273739616 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910ce0 0x5602e991fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b116c8640 1 -- 172.21.15.28:0/273739616 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910ce0 msgr2=0x5602e991fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:36.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b116c8640 1 --2- 172.21.15.28:0/273739616 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910ce0 0x5602e991fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b116c8640 1 -- 172.21.15.28:0/273739616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910330 msgr2=0x5602e9929cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:36.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b10ec7640 1 --2- 172.21.15.28:0/273739616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910330 0x5602e9929cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b116c8640 1 --2- 172.21.15.28:0/273739616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910330 0x5602e9929cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b116c8640 1 -- 172.21.15.28:0/273739616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602e9919b90 con 0x5602e9919e40 2024-04-21T08:57:36.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.134+0000 7f2b116c8640 1 --2- 172.21.15.28:0/273739616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e99201f0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f2b04013fd0 tx=0x7f2b04004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ee597b6cb1d47d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:36.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.135+0000 7f2af27fc640 1 -- 172.21.15.28:0/273739616 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0403c020 con 0x5602e9919e40 2024-04-21T08:57:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.135+0000 7f2af27fc640 1 -- 172.21.15.28:0/273739616 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b0401e820 con 0x5602e9919e40 2024-04-21T08:57:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.135+0000 7f2af27fc640 1 -- 172.21.15.28:0/273739616 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04029840 con 0x5602e9919e40 2024-04-21T08:57:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.135+0000 7f2b15df6740 1 -- 172.21.15.28:0/273739616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 msgr2=0x5602e99201f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.135+0000 7f2b15df6740 1 --2- 172.21.15.28:0/273739616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e99201f0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f2b04013fd0 tx=0x7f2b04004ab0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.135+0000 7f2b15df6740 1 -- 172.21.15.28:0/273739616 shutdown_connections 2024-04-21T08:57:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.135+0000 7f2b15df6740 1 --2- 172.21.15.28:0/273739616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9919e40 0x5602e99201f0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.135+0000 7f2b15df6740 1 --2- 172.21.15.28:0/273739616 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9910ce0 0x5602e991fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.135+0000 7f2b15df6740 1 --2- 172.21.15.28:0/273739616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910330 0x5602e9929cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.135+0000 7f2b15df6740 1 -- 172.21.15.28:0/273739616 >> 172.21.15.28:0/273739616 conn(0x5602e990aba0 msgr2=0x5602e991d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.135+0000 7f2b15df6740 1 -- 172.21.15.28:0/273739616 shutdown_connections 2024-04-21T08:57:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.135+0000 7f2b15df6740 1 -- 172.21.15.28:0/273739616 wait complete. 2024-04-21T08:57:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.135+0000 7f2b15df6740 1 -- start start 2024-04-21T08:57:36.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b15df6740 1 Processor -- start 2024-04-21T08:57:36.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b15df6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9910330 0x5602e99263c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b15df6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910ce0 0x5602e991ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b15df6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9919e40 0x5602e991f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b15df6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c1ab7a0 con 0x5602e9910330 2024-04-21T08:57:36.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b15df6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c1acc10 con 0x5602e9910ce0 2024-04-21T08:57:36.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b15df6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0c1abe70 con 0x5602e9919e40 2024-04-21T08:57:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b15df6740 10 client.0 _ll_register_callbacks cb 0x7f2b0401d430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b10ec7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9910330 0x5602e99263c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b10ec7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9910330 0x5602e99263c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43204/0 (socket says 172.21.15.28:43204) 2024-04-21T08:57:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b10ec7640 1 -- 172.21.15.28:0/3122056402 learned_addr learned my addr 172.21.15.28:0/3122056402 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b0bfff640 1 --2- 172.21.15.28:0/3122056402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910ce0 0x5602e991ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b116c8640 1 --2- 172.21.15.28:0/3122056402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9919e40 0x5602e991f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b10ec7640 1 -- 172.21.15.28:0/3122056402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9919e40 msgr2=0x5602e991f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b10ec7640 1 --2- 172.21.15.28:0/3122056402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9919e40 0x5602e991f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b10ec7640 1 -- 172.21.15.28:0/3122056402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910ce0 msgr2=0x5602e991ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b10ec7640 1 --2- 172.21.15.28:0/3122056402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910ce0 0x5602e991ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.136+0000 7f2b10ec7640 1 -- 172.21.15.28:0/3122056402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602e991f800 con 0x5602e9910330 2024-04-21T08:57:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2b10ec7640 1 --2- 172.21.15.28:0/3122056402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9910330 0x5602e99263c0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f2afc02c610 tx=0x7f2afc0df040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8910b4472b40966 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2aebfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:36.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2aebfff640 1 -- 172.21.15.28:0/3122056402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2afc0bea00 con 0x5602e9910330 2024-04-21T08:57:36.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2b15df6740 1 -- 172.21.15.28:0/3122056402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b0401e820 con 0x5602e9910330 2024-04-21T08:57:36.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2aebfff640 1 -- 172.21.15.28:0/3122056402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f2afc0f0320 con 0x5602e9910330 2024-04-21T08:57:36.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2aebfff640 1 -- 172.21.15.28:0/3122056402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2afc0c2db0 con 0x5602e9910330 2024-04-21T08:57:36.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2b15df6740 10 client.17403 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:36.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2b15df6740 1 -- 172.21.15.28:0/3122056402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5602e99943c0 con 0x5602e9910330 2024-04-21T08:57:36.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2aebfff640 1 -- 172.21.15.28:0/3122056402 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b0401e820 con 0x5602e9910330 2024-04-21T08:57:36.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2aebfff640 1 -- 172.21.15.28:0/3122056402 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f2afc005490 con 0x5602e9910330 2024-04-21T08:57:36.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2b15df6740 10 client.17403 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:36.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2b15df6740 10 client.17403 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:36.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2b15df6740 20 client.17403 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:36.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2b15df6740 1 --2- 172.21.15.28:0/3122056402 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5602e9994a90 0x5602e9996eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:36.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2b15df6740 4 client.17403 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:57:36.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.137+0000 7f2b15df6740 1 -- 172.21.15.28:0/3122056402 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5602e9882c10 con 0x5602e9994a90 2024-04-21T08:57:36.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2b0bfff640 1 --2- 172.21.15.28:0/3122056402 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5602e9994a90 0x5602e9996eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:36.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2b0bfff640 1 --2- 172.21.15.28:0/3122056402 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5602e9994a90 0x5602e9996eb0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=926bfd6c95e82899 server_cookie=1260fa1af004714f in_seq=0 out_seq=0 2024-04-21T08:57:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2aebfff640 10 client.17403 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2aebfff640 1 -- 172.21.15.28:0/3122056402 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5602e9882c10 con 0x5602e9994a90 2024-04-21T08:57:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2aebfff640 10 client.17403 handle_command_reply: tid=0 2024-04-21T08:57:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2b15df6740 1 client.17403 shutdown 2024-04-21T08:57:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2b15df6740 1 -- 172.21.15.28:0/3122056402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9910330 msgr2=0x5602e99263c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2b15df6740 1 --2- 172.21.15.28:0/3122056402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9910330 0x5602e99263c0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f2afc02c610 tx=0x7f2afc0df040 comp rx=0 tx=0).stop 2024-04-21T08:57:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2b15df6740 1 -- 172.21.15.28:0/3122056402 shutdown_connections 2024-04-21T08:57:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2b15df6740 1 --2- 172.21.15.28:0/3122056402 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5602e9994a90 0x5602e9996eb0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2b15df6740 1 --2- 172.21.15.28:0/3122056402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5602e9919e40 0x5602e991f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2b15df6740 1 --2- 172.21.15.28:0/3122056402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5602e9910ce0 0x5602e991ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2aebfff640 0 client.17403 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2b15df6740 1 --2- 172.21.15.28:0/3122056402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602e9910330 0x5602e99263c0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2b15df6740 1 -- 172.21.15.28:0/3122056402 >> 172.21.15.28:0/3122056402 conn(0x5602e990aba0 msgr2=0x5602e9909370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2b15df6740 1 -- 172.21.15.28:0/3122056402 shutdown_connections 2024-04-21T08:57:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.138+0000 7f2b15df6740 1 -- 172.21.15.28:0/3122056402 wait complete. 2024-04-21T08:57:36.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.139+0000 7f2b15df6740 20 client.17403 trim_cache size 0 max 16384 2024-04-21T08:57:36.152 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:57:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.139+0000 7f2b1312b640 1 -- 172.21.15.28:0/321691817 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2af40a3d20 msgr2=0x7f2af40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.139+0000 7f2b1312b640 1 --2- 172.21.15.28:0/321691817 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2af40a3d20 0x7f2af40a61e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2b04005fd0 tx=0x7f2b040096e0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.139+0000 7f2b1312b640 1 -- 172.21.15.28:0/321691817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b0c1031d0 msgr2=0x7f2b0c1987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.139+0000 7f2b1312b640 1 --2- 172.21.15.28:0/321691817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b0c1031d0 0x7f2b0c1987b0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f2afc000c00 tx=0x7f2afc02d920 comp rx=0 tx=0).stop 2024-04-21T08:57:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.139+0000 7f2b1312b640 1 -- 172.21.15.28:0/321691817 shutdown_connections 2024-04-21T08:57:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.139+0000 7f2b1312b640 1 --2- 172.21.15.28:0/321691817 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2af40a3d20 0x7f2af40a61e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.139+0000 7f2b1312b640 1 --2- 172.21.15.28:0/321691817 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b0c10c510 0x7f2b0c19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.139+0000 7f2b1312b640 1 --2- 172.21.15.28:0/321691817 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b0c103af0 0x7f2b0c198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.139+0000 7f2b1312b640 1 --2- 172.21.15.28:0/321691817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b0c1031d0 0x7f2b0c1987b0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:36.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.139+0000 7f2b1312b640 1 -- 172.21.15.28:0/321691817 >> 172.21.15.28:0/321691817 conn(0x7f2b0c077f50 msgr2=0x7f2b0c0fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:36.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.139+0000 7f2b1312b640 1 -- 172.21.15.28:0/321691817 shutdown_connections 2024-04-21T08:57:36.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:36.139+0000 7f2b1312b640 1 -- 172.21.15.28:0/321691817 wait complete. 2024-04-21T08:57:36.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:36 smithi152 ceph-mon[31401]: pgmap v195: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:57:36.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:36 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/169842550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:36 smithi028 ceph-mon[28003]: pgmap v195: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:57:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:36 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/169842550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:36.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:36 smithi171 ceph-mon[31358]: pgmap v195: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:57:36.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:36 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/169842550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:37.155 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:57:37.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:37.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:57:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.276+0000 7f71b2def640 1 -- 172.21.15.28:0/611326065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac107260 msgr2=0x7f71ac10ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.276+0000 7f71b2def640 1 --2- 172.21.15.28:0/611326065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac107260 0x7f71ac10ee30 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f7194009900 tx=0x7f719402bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.276+0000 7f71b2def640 1 -- 172.21.15.28:0/611326065 shutdown_connections 2024-04-21T08:57:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.276+0000 7f71b2def640 1 --2- 172.21.15.28:0/611326065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac107260 0x7f71ac10ee30 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.276+0000 7f71b2def640 1 --2- 172.21.15.28:0/611326065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71ac102b40 0x7f71ac106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.276+0000 7f71b2def640 1 --2- 172.21.15.28:0/611326065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71ac102190 0x7f71ac102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.276+0000 7f71b2def640 1 -- 172.21.15.28:0/611326065 >> 172.21.15.28:0/611326065 conn(0x7f71ac0fd990 msgr2=0x7f71ac0ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.276+0000 7f71b2def640 1 -- 172.21.15.28:0/611326065 shutdown_connections 2024-04-21T08:57:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.276+0000 7f71b2def640 1 -- 172.21.15.28:0/611326065 wait complete. 2024-04-21T08:57:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.276+0000 7f71b2def640 1 Processor -- start 2024-04-21T08:57:37.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.276+0000 7f71b2def640 1 -- start start 2024-04-21T08:57:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71b2def640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71ac102190 0x7f71ac074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71b2def640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71ac107260 0x7f71ac074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71b2def640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac074cd0 0x7f71ac1a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71b2def640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac1121b0 con 0x7f71ac074cd0 2024-04-21T08:57:37.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71b2def640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac112030 con 0x7f71ac107260 2024-04-21T08:57:37.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71b2def640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac112330 con 0x7f71ac102190 2024-04-21T08:57:37.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71b138c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac074cd0 0x7f71ac1a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71b138c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac074cd0 0x7f71ac1a6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43222/0 (socket says 172.21.15.28:43222) 2024-04-21T08:57:37.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71b138c640 1 -- 172.21.15.28:0/3767746333 learned_addr learned my addr 172.21.15.28:0/3767746333 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:37.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71abfff640 1 --2- 172.21.15.28:0/3767746333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71ac107260 0x7f71ac074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71b0b8b640 1 --2- 172.21.15.28:0/3767746333 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71ac102190 0x7f71ac074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71b138c640 1 -- 172.21.15.28:0/3767746333 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71ac102190 msgr2=0x7f71ac074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71b138c640 1 --2- 172.21.15.28:0/3767746333 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71ac102190 0x7f71ac074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71b138c640 1 -- 172.21.15.28:0/3767746333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71ac107260 msgr2=0x7f71ac074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71b138c640 1 --2- 172.21.15.28:0/3767746333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71ac107260 0x7f71ac074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.277+0000 7f71b138c640 1 -- 172.21.15.28:0/3767746333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71ac1a6f80 con 0x7f71ac074cd0 2024-04-21T08:57:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.278+0000 7f71b138c640 1 --2- 172.21.15.28:0/3767746333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac074cd0 0x7f71ac1a6880 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7194009230 tx=0x7f71940045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.278+0000 7f71a9ffb640 1 -- 172.21.15.28:0/3767746333 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7194043020 con 0x7f71ac074cd0 2024-04-21T08:57:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.278+0000 7f71a9ffb640 1 -- 172.21.15.28:0/3767746333 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f719402cb70 con 0x7f71ac074cd0 2024-04-21T08:57:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.278+0000 7f71b2def640 1 -- 172.21.15.28:0/3767746333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71ac1a7210 con 0x7f71ac074cd0 2024-04-21T08:57:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.278+0000 7f71a9ffb640 1 -- 172.21.15.28:0/3767746333 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7194003a50 con 0x7f71ac074cd0 2024-04-21T08:57:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.278+0000 7f71b2def640 1 -- 172.21.15.28:0/3767746333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71ac1a77b0 con 0x7f71ac074cd0 2024-04-21T08:57:37.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.281+0000 7f71b59d9740 1 Processor -- start 2024-04-21T08:57:37.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.281+0000 7f71b59d9740 1 -- start start 2024-04-21T08:57:37.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71b59d9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 0x55e556ab4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71b59d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556abd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71b59d9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556ac0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71b59d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e556a18f00 con 0x55e556ab5290 2024-04-21T08:57:37.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71b59d9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e5569b6590 con 0x55e556ab48e0 2024-04-21T08:57:37.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71b59d9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55e556a14ec0 con 0x55e556abde10 2024-04-21T08:57:37.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71b0b8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556abd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71b0b8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556abd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43236/0 (socket says 172.21.15.28:43236) 2024-04-21T08:57:37.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71b0b8b640 1 -- 172.21.15.28:0/1899432641 learned_addr learned my addr 172.21.15.28:0/1899432641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:37.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71abfff640 1 --2- 172.21.15.28:0/1899432641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 0x55e556ab4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71abfff640 1 --2- 172.21.15.28:0/1899432641 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556ac0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71b0b8b640 1 -- 172.21.15.28:0/1899432641 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 msgr2=0x55e556ac0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71b0b8b640 1 --2- 172.21.15.28:0/1899432641 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556ac0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71b0b8b640 1 -- 172.21.15.28:0/1899432641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 msgr2=0x55e556ab4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71b0b8b640 1 --2- 172.21.15.28:0/1899432641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 0x55e556ab4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.282+0000 7f71b0b8b640 1 -- 172.21.15.28:0/1899432641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5567f37e0 con 0x55e556ab5290 2024-04-21T08:57:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.283+0000 7f71b0b8b640 1 --2- 172.21.15.28:0/1899432641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556abd840 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f71a4009f60 tx=0x7f71a4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610a8ef4f4361bf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.283+0000 7f71a2ffd640 1 -- 172.21.15.28:0/1899432641 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a4016070 con 0x55e556ab5290 2024-04-21T08:57:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.283+0000 7f71a2ffd640 1 -- 172.21.15.28:0/1899432641 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71a401bb10 con 0x55e556ab5290 2024-04-21T08:57:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.283+0000 7f71a2ffd640 1 -- 172.21.15.28:0/1899432641 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a4012c30 con 0x55e556ab5290 2024-04-21T08:57:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.284+0000 7f71b59d9740 1 -- 172.21.15.28:0/1899432641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 msgr2=0x55e556abd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.284+0000 7f71b59d9740 1 --2- 172.21.15.28:0/1899432641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556abd840 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f71a4009f60 tx=0x7f71a4009f90 comp rx=0 tx=0).stop 2024-04-21T08:57:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.284+0000 7f71a9ffb640 1 -- 172.21.15.28:0/3767746333 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7194046020 con 0x7f71ac074cd0 2024-04-21T08:57:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.284+0000 7f71b59d9740 1 -- 172.21.15.28:0/1899432641 shutdown_connections 2024-04-21T08:57:37.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.284+0000 7f71b59d9740 1 --2- 172.21.15.28:0/1899432641 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556ac0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.284+0000 7f71b59d9740 1 --2- 172.21.15.28:0/1899432641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556abd840 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.284+0000 7f71b59d9740 1 --2- 172.21.15.28:0/1899432641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 0x55e556ab4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.284+0000 7f71b59d9740 1 -- 172.21.15.28:0/1899432641 >> 172.21.15.28:0/1899432641 conn(0x55e556aaefb0 msgr2=0x55e556aaf390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.284+0000 7f71b59d9740 1 -- 172.21.15.28:0/1899432641 shutdown_connections 2024-04-21T08:57:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.284+0000 7f71b59d9740 1 -- 172.21.15.28:0/1899432641 wait complete. 2024-04-21T08:57:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.284+0000 7f71b59d9740 1 -- start start 2024-04-21T08:57:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.284+0000 7f71b59d9740 1 Processor -- start 2024-04-21T08:57:37.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71a9ffb640 1 --2- 172.21.15.28:0/3767746333 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f71980a3ce0 0x7f71980a61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71b59d9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 0x55e556abad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71b59d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556abb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71b59d9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556abb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71b59d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e556ac0d80 con 0x55e556ab5290 2024-04-21T08:57:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71b59d9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e5569d2580 con 0x55e556ab48e0 2024-04-21T08:57:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71b59d9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55e556ac0f00 con 0x55e556abde10 2024-04-21T08:57:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71a9ffb640 1 -- 172.21.15.28:0/3767746333 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f71940f2c40 con 0x7f71ac074cd0 2024-04-21T08:57:37.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71abfff640 1 --2- 172.21.15.28:0/3767746333 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f71980a3ce0 0x7f71980a61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71b59d9740 10 client.0 _ll_register_callbacks cb 0x55e5569d8650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556abb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556abb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43244/0 (socket says 172.21.15.28:43244) 2024-04-21T08:57:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71b0b8b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 0x55e556abad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71abfff640 1 -- 172.21.15.28:0/3009334616 learned_addr learned my addr 172.21.15.28:0/3009334616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71b138c640 1 --2- 172.21.15.28:0/3009334616 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556abb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71abfff640 1 -- 172.21.15.28:0/3009334616 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 msgr2=0x55e556abb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71abfff640 1 --2- 172.21.15.28:0/3009334616 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556abb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71abfff640 1 -- 172.21.15.28:0/3009334616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 msgr2=0x55e556abad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71abfff640 1 --2- 172.21.15.28:0/3009334616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 0x55e556abad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71abfff640 1 -- 172.21.15.28:0/3009334616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e556853f70 con 0x55e556ab5290 2024-04-21T08:57:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71abfff640 1 --2- 172.21.15.28:0/3767746333 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f71980a3ce0 0x7f71980a61a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f719c005940 tx=0x7f719c00ad50 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f71abfff640 1 --2- 172.21.15.28:0/3009334616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556abb290 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f719c00ff80 tx=0x7f719c00a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3087a7eef2268d2a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.285+0000 7f718bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.286+0000 7f718bfff640 1 -- 172.21.15.28:0/3009334616 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c01a040 con 0x55e556ab5290 2024-04-21T08:57:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.286+0000 7f718bfff640 1 -- 172.21.15.28:0/3009334616 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f719c029670 con 0x55e556ab5290 2024-04-21T08:57:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.286+0000 7f71b59d9740 1 -- 172.21.15.28:0/3009334616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71a401bb10 con 0x55e556ab5290 2024-04-21T08:57:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.286+0000 7f718bfff640 1 -- 172.21.15.28:0/3009334616 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f719c0298b0 con 0x55e556ab5290 2024-04-21T08:57:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.286+0000 7f71b59d9740 10 client.17415 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.286+0000 7f71b59d9740 1 -- 172.21.15.28:0/3009334616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e5566eb170 con 0x55e556ab5290 2024-04-21T08:57:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.286+0000 7f718bfff640 1 -- 172.21.15.28:0/3009334616 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71a401bb10 con 0x55e556ab5290 2024-04-21T08:57:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.286+0000 7f718bfff640 1 -- 172.21.15.28:0/3009334616 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f719c020720 con 0x55e556ab5290 2024-04-21T08:57:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.286+0000 7f71b59d9740 10 client.17415 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.286+0000 7f71b59d9740 10 client.17415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.286+0000 7f71b59d9740 20 client.17415 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.286+0000 7f71b59d9740 1 --2- 172.21.15.28:0/3009334616 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e556ace360 0x55e556ad0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.286+0000 7f71b59d9740 4 client.17415 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:57:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.286+0000 7f71b59d9740 1 -- 172.21.15.28:0/3009334616 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e556a26920 con 0x55e556ace360 2024-04-21T08:57:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.287+0000 7f71b0b8b640 1 --2- 172.21.15.28:0/3009334616 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e556ace360 0x55e556ad0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.287+0000 7f71b0b8b640 1 --2- 172.21.15.28:0/3009334616 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e556ace360 0x55e556ad0800 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46544494f37560cf server_cookie=63ca165841db2c0a in_seq=0 out_seq=0 2024-04-21T08:57:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.287+0000 7f718bfff640 10 client.17415 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.288+0000 7f718bfff640 1 -- 172.21.15.28:0/3009334616 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e556a26920 con 0x55e556ace360 2024-04-21T08:57:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.288+0000 7f718bfff640 10 client.17415 handle_command_reply: tid=0 2024-04-21T08:57:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.288+0000 7f71b59d9740 1 client.17415 shutdown 2024-04-21T08:57:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.288+0000 7f71b59d9740 1 -- 172.21.15.28:0/3009334616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 msgr2=0x55e556abb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.288+0000 7f71b59d9740 1 --2- 172.21.15.28:0/3009334616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556abb290 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f719c00ff80 tx=0x7f719c00a2d0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.288+0000 7f71b59d9740 1 -- 172.21.15.28:0/3009334616 shutdown_connections 2024-04-21T08:57:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.288+0000 7f71b59d9740 1 --2- 172.21.15.28:0/3009334616 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e556ace360 0x55e556ad0800 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.288+0000 7f71b59d9740 1 --2- 172.21.15.28:0/3009334616 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556abb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.288+0000 7f71b59d9740 1 --2- 172.21.15.28:0/3009334616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556abb290 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.288+0000 7f718bfff640 0 client.17415 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.288+0000 7f71b59d9740 1 --2- 172.21.15.28:0/3009334616 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 0x55e556abad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.288+0000 7f71b59d9740 1 -- 172.21.15.28:0/3009334616 >> 172.21.15.28:0/3009334616 conn(0x55e556aaefb0 msgr2=0x55e556ab0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.288+0000 7f71b59d9740 1 -- 172.21.15.28:0/3009334616 shutdown_connections 2024-04-21T08:57:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.288+0000 7f71b59d9740 1 -- 172.21.15.28:0/3009334616 wait complete. 2024-04-21T08:57:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.288+0000 7f71b59d9740 20 client.17415 trim_cache size 0 max 16384 2024-04-21T08:57:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71b59d9740 1 Processor -- start 2024-04-21T08:57:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71b59d9740 1 -- start start 2024-04-21T08:57:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71b59d9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 0x55e556abd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71b59d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556ac9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71b59d9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556aade90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71b59d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac1b3dd0 con 0x55e556ab5290 2024-04-21T08:57:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71b59d9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac1b16a0 con 0x55e556ab48e0 2024-04-21T08:57:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71b59d9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac1af4f0 con 0x55e556abde10 2024-04-21T08:57:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556ac9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556ac9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43260/0 (socket says 172.21.15.28:43260) 2024-04-21T08:57:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71abfff640 1 -- 172.21.15.28:0/624607003 learned_addr learned my addr 172.21.15.28:0/624607003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71b138c640 1 --2- 172.21.15.28:0/624607003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556aade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71abfff640 1 -- 172.21.15.28:0/624607003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 msgr2=0x55e556aade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71abfff640 1 --2- 172.21.15.28:0/624607003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556aade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71abfff640 1 -- 172.21.15.28:0/624607003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 msgr2=0x55e556abd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.306+0000 7f71abfff640 1 --2- 172.21.15.28:0/624607003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 0x55e556abd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.307+0000 7f71abfff640 1 -- 172.21.15.28:0/624607003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e556ab4ff0 con 0x55e556ab5290 2024-04-21T08:57:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.307+0000 7f71b138c640 1 --2- 172.21.15.28:0/624607003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556aade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.307+0000 7f71abfff640 1 --2- 172.21.15.28:0/624607003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556ac9ca0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f719c020be0 tx=0x7f719c01feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93dd98612332d86 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.307+0000 7f71a27fc640 1 -- 172.21.15.28:0/624607003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c03b070 con 0x55e556ab5290 2024-04-21T08:57:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.307+0000 7f71a27fc640 1 -- 172.21.15.28:0/624607003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f719c024070 con 0x55e556ab5290 2024-04-21T08:57:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.307+0000 7f71a27fc640 1 -- 172.21.15.28:0/624607003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c0296d0 con 0x55e556ab5290 2024-04-21T08:57:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.307+0000 7f71b59d9740 1 -- 172.21.15.28:0/624607003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 msgr2=0x55e556ac9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.307+0000 7f71b59d9740 1 --2- 172.21.15.28:0/624607003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556ac9ca0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f719c020be0 tx=0x7f719c01feb0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.307+0000 7f71b59d9740 1 -- 172.21.15.28:0/624607003 shutdown_connections 2024-04-21T08:57:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.307+0000 7f71b59d9740 1 --2- 172.21.15.28:0/624607003 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556aade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.307+0000 7f71b59d9740 1 --2- 172.21.15.28:0/624607003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556ac9ca0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.307+0000 7f71b59d9740 1 --2- 172.21.15.28:0/624607003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 0x55e556abd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.307+0000 7f71b59d9740 1 -- 172.21.15.28:0/624607003 >> 172.21.15.28:0/624607003 conn(0x55e556aaefb0 msgr2=0x55e556aad780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.308+0000 7f71b59d9740 1 -- 172.21.15.28:0/624607003 shutdown_connections 2024-04-21T08:57:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.308+0000 7f71b59d9740 1 -- 172.21.15.28:0/624607003 wait complete. 2024-04-21T08:57:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.308+0000 7f71b59d9740 1 -- start start 2024-04-21T08:57:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.308+0000 7f71b59d9740 1 Processor -- start 2024-04-21T08:57:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.308+0000 7f71b59d9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 0x55e556ac1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.308+0000 7f71b59d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556ac2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.308+0000 7f71b59d9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556b33940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.308+0000 7f71b59d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac1afb60 con 0x55e556ab5290 2024-04-21T08:57:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.308+0000 7f71b59d9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac1b0fd0 con 0x55e556ab48e0 2024-04-21T08:57:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.308+0000 7f71b59d9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac1b0230 con 0x55e556abde10 2024-04-21T08:57:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.308+0000 7f71b59d9740 10 client.0 _ll_register_callbacks cb 0x7f719c0105f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.308+0000 7f71abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556ac2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.308+0000 7f71abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556ac2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43276/0 (socket says 172.21.15.28:43276) 2024-04-21T08:57:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.308+0000 7f71abfff640 1 -- 172.21.15.28:0/276879626 learned_addr learned my addr 172.21.15.28:0/276879626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.309+0000 7f71b0b8b640 1 --2- 172.21.15.28:0/276879626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 0x55e556ac1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.309+0000 7f71abfff640 1 -- 172.21.15.28:0/276879626 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 msgr2=0x55e556b33940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.309+0000 7f71abfff640 1 --2- 172.21.15.28:0/276879626 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556b33940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.309+0000 7f71abfff640 1 -- 172.21.15.28:0/276879626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 msgr2=0x55e556ac1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.309+0000 7f71abfff640 1 --2- 172.21.15.28:0/276879626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 0x55e556ac1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.309+0000 7f71abfff640 1 -- 172.21.15.28:0/276879626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e556ac2ce0 con 0x55e556ab5290 2024-04-21T08:57:37.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.309+0000 7f71abfff640 1 --2- 172.21.15.28:0/276879626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556ac2440 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f719c020cd0 tx=0x7f719c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79dc0f8ed5947194 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.309+0000 7f718bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.309+0000 7f718bfff640 1 -- 172.21.15.28:0/276879626 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c03b070 con 0x55e556ab5290 2024-04-21T08:57:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.309+0000 7f718bfff640 1 -- 172.21.15.28:0/276879626 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f719c029c50 con 0x55e556ab5290 2024-04-21T08:57:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.309+0000 7f71b59d9740 1 -- 172.21.15.28:0/276879626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f719c024070 con 0x55e556ab5290 2024-04-21T08:57:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.309+0000 7f718bfff640 1 -- 172.21.15.28:0/276879626 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f719c00abb0 con 0x55e556ab5290 2024-04-21T08:57:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.309+0000 7f71b59d9740 10 client.17421 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.309+0000 7f71b59d9740 1 -- 172.21.15.28:0/276879626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e556b34f00 con 0x55e556ab5290 2024-04-21T08:57:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.309+0000 7f718bfff640 1 -- 172.21.15.28:0/276879626 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f719c024070 con 0x55e556ab5290 2024-04-21T08:57:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.310+0000 7f718bfff640 1 -- 172.21.15.28:0/276879626 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f719c0051e0 con 0x55e556ab5290 2024-04-21T08:57:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.310+0000 7f71b59d9740 10 client.17421 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.310+0000 7f71b59d9740 10 client.17421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.310+0000 7f71b59d9740 20 client.17421 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.310+0000 7f71b59d9740 1 --2- 172.21.15.28:0/276879626 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e556ace360 0x55e556b375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.310+0000 7f71b59d9740 4 client.17421 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:57:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.310+0000 7f71b59d9740 1 -- 172.21.15.28:0/276879626 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e556a20790 con 0x55e556ace360 2024-04-21T08:57:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.310+0000 7f71b0b8b640 1 --2- 172.21.15.28:0/276879626 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e556ace360 0x55e556b375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.310+0000 7f718bfff640 10 client.17421 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.310+0000 7f71b0b8b640 1 --2- 172.21.15.28:0/276879626 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e556ace360 0x55e556b375d0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9898b6de41bb1fe3 server_cookie=936a0322dc4233df in_seq=0 out_seq=0 2024-04-21T08:57:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.312+0000 7f718bfff640 1 -- 172.21.15.28:0/276879626 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e556a20790 con 0x55e556ace360 2024-04-21T08:57:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.312+0000 7f718bfff640 10 client.17421 handle_command_reply: tid=0 2024-04-21T08:57:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.312+0000 7f71b59d9740 1 client.17421 shutdown 2024-04-21T08:57:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.312+0000 7f71b59d9740 1 -- 172.21.15.28:0/276879626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 msgr2=0x55e556ac2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.312+0000 7f71b59d9740 1 --2- 172.21.15.28:0/276879626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556ac2440 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f719c020cd0 tx=0x7f719c00f790 comp rx=0 tx=0).stop 2024-04-21T08:57:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.312+0000 7f71b59d9740 1 -- 172.21.15.28:0/276879626 shutdown_connections 2024-04-21T08:57:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.312+0000 7f71b59d9740 1 --2- 172.21.15.28:0/276879626 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e556ace360 0x55e556b375d0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.312+0000 7f71b59d9740 1 --2- 172.21.15.28:0/276879626 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e556abde10 0x55e556b33940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.312+0000 7f71b59d9740 1 --2- 172.21.15.28:0/276879626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e556ab5290 0x55e556ac2440 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.312+0000 7f71b59d9740 1 --2- 172.21.15.28:0/276879626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e556ab48e0 0x55e556ac1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.312+0000 7f71b59d9740 1 -- 172.21.15.28:0/276879626 >> 172.21.15.28:0/276879626 conn(0x55e556aaefb0 msgr2=0x55e556aae010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.312+0000 7f718bfff640 0 client.17421 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.312+0000 7f71b59d9740 1 -- 172.21.15.28:0/276879626 shutdown_connections 2024-04-21T08:57:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.312+0000 7f71b59d9740 1 -- 172.21.15.28:0/276879626 wait complete. 2024-04-21T08:57:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.312+0000 7f71b59d9740 20 client.17421 trim_cache size 0 max 16384 2024-04-21T08:57:37.322 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:57:37.322 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:57:37.322 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "fdc2d2a7-c5a0-4ec9-abdb-9f31a600776a", 2024-04-21T08:57:37.322 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:57:37.322 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:57:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.313+0000 7f71b2def640 1 -- 172.21.15.28:0/3767746333 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f71980a3ce0 msgr2=0x7f71980a61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.313+0000 7f71b2def640 1 --2- 172.21.15.28:0/3767746333 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f71980a3ce0 0x7f71980a61a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f719c005940 tx=0x7f719c00ad50 comp rx=0 tx=0).stop 2024-04-21T08:57:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.313+0000 7f71b2def640 1 -- 172.21.15.28:0/3767746333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac074cd0 msgr2=0x7f71ac1a6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.313+0000 7f71b2def640 1 --2- 172.21.15.28:0/3767746333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac074cd0 0x7f71ac1a6880 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7194009230 tx=0x7f71940045e0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.313+0000 7f71b2def640 1 -- 172.21.15.28:0/3767746333 shutdown_connections 2024-04-21T08:57:37.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.313+0000 7f71b2def640 1 --2- 172.21.15.28:0/3767746333 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f71980a3ce0 0x7f71980a61a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.313+0000 7f71b2def640 1 --2- 172.21.15.28:0/3767746333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71ac074cd0 0x7f71ac1a6880 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.313+0000 7f71b2def640 1 --2- 172.21.15.28:0/3767746333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71ac107260 0x7f71ac074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.313+0000 7f71b2def640 1 --2- 172.21.15.28:0/3767746333 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f71ac102190 0x7f71ac074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.313+0000 7f71b2def640 1 -- 172.21.15.28:0/3767746333 >> 172.21.15.28:0/3767746333 conn(0x7f71ac0fd990 msgr2=0x7f71ac0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:37.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.313+0000 7f71b2def640 1 -- 172.21.15.28:0/3767746333 shutdown_connections 2024-04-21T08:57:37.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.313+0000 7f71b2def640 1 -- 172.21.15.28:0/3767746333 wait complete. 2024-04-21T08:57:37.325 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:57:37.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:57:37.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.433+0000 7f52fc94b640 1 -- 172.21.15.28:0/1801971407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f81021a0 msgr2=0x7f52f8102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.433+0000 7f52fc94b640 1 --2- 172.21.15.28:0/1801971407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f81021a0 0x7f52f8102580 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009900 tx=0x7f52e802bc50 comp rx=0 tx=0).stop 2024-04-21T08:57:37.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.433+0000 7f52fc94b640 1 -- 172.21.15.28:0/1801971407 shutdown_connections 2024-04-21T08:57:37.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.433+0000 7f52fc94b640 1 --2- 172.21.15.28:0/1801971407 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f52f8107270 0x7f52f810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.433+0000 7f52fc94b640 1 --2- 172.21.15.28:0/1801971407 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52f8102b50 0x7f52f8106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.433+0000 7f52fc94b640 1 --2- 172.21.15.28:0/1801971407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f81021a0 0x7f52f8102580 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.433+0000 7f52fc94b640 1 -- 172.21.15.28:0/1801971407 >> 172.21.15.28:0/1801971407 conn(0x7f52f80793e0 msgr2=0x7f52f80797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:37.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.433+0000 7f52fc94b640 1 -- 172.21.15.28:0/1801971407 shutdown_connections 2024-04-21T08:57:37.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.433+0000 7f52fc94b640 1 -- 172.21.15.28:0/1801971407 wait complete. 2024-04-21T08:57:37.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52fc94b640 1 Processor -- start 2024-04-21T08:57:37.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52fc94b640 1 -- start start 2024-04-21T08:57:37.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52fc94b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f52f81021a0 0x7f52f8074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52fc94b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52f8107270 0x7f52f8074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52fc94b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8074cd0 0x7f52f81a6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52fc94b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52f81121b0 con 0x7f52f8074cd0 2024-04-21T08:57:37.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52fc94b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f52f8112030 con 0x7f52f81021a0 2024-04-21T08:57:37.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52fc94b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f52f8112330 con 0x7f52f8107270 2024-04-21T08:57:37.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52f6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8074cd0 0x7f52f81a6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52f6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8074cd0 0x7f52f81a6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43294/0 (socket says 172.21.15.28:43294) 2024-04-21T08:57:37.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52f6d9d640 1 -- 172.21.15.28:0/511944052 learned_addr learned my addr 172.21.15.28:0/511944052 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:37.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52f5d9b640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52f8107270 0x7f52f8074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52f659c640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f52f81021a0 0x7f52f8074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52f6d9d640 1 -- 172.21.15.28:0/511944052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52f8107270 msgr2=0x7f52f8074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52f6d9d640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52f8107270 0x7f52f8074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.434+0000 7f52f6d9d640 1 -- 172.21.15.28:0/511944052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f52f81021a0 msgr2=0x7f52f8074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.435+0000 7f52f6d9d640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f52f81021a0 0x7f52f8074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.435+0000 7f52f6d9d640 1 -- 172.21.15.28:0/511944052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f81a6f20 con 0x7f52f8074cd0 2024-04-21T08:57:37.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.435+0000 7f52f659c640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f52f81021a0 0x7f52f8074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:57:37.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.435+0000 7f52f5d9b640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52f8107270 0x7f52f8074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:57:37.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.435+0000 7f52f6d9d640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8074cd0 0x7f52f81a6820 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f52f000cce0 tx=0x7f52f000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:37.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.435+0000 7f52ef7fe640 1 -- 172.21.15.28:0/511944052 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f000f450 con 0x7f52f8074cd0 2024-04-21T08:57:37.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.435+0000 7f52ef7fe640 1 -- 172.21.15.28:0/511944052 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52f000faa0 con 0x7f52f8074cd0 2024-04-21T08:57:37.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.435+0000 7f52ef7fe640 1 -- 172.21.15.28:0/511944052 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f0018b70 con 0x7f52f8074cd0 2024-04-21T08:57:37.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.435+0000 7f52fc94b640 1 -- 172.21.15.28:0/511944052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52f81a7210 con 0x7f52f8074cd0 2024-04-21T08:57:37.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.435+0000 7f52fc94b640 1 -- 172.21.15.28:0/511944052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52f81a7710 con 0x7f52f8074cd0 2024-04-21T08:57:37.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.438+0000 7f52ef7fe640 1 -- 172.21.15.28:0/511944052 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f52f0018d10 con 0x7f52f8074cd0 2024-04-21T08:57:37.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.438+0000 7f52ff613740 1 Processor -- start 2024-04-21T08:57:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.438+0000 7f52ff613740 1 -- start start 2024-04-21T08:57:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.438+0000 7f52ff613740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20580 0x5633c1d20960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.438+0000 7f52ff613740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d20ea0 0x5633c1d29540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.438+0000 7f52ff613740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d29a80 0x5633c1d2be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.438+0000 7f52ff613740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5633c1c851e0 con 0x5633c1d20580 2024-04-21T08:57:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.438+0000 7f52ff613740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5633c1c22390 con 0x5633c1d29a80 2024-04-21T08:57:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.438+0000 7f52ff613740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5633c1c811a0 con 0x5633c1d20ea0 2024-04-21T08:57:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.438+0000 7f52f5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20580 0x5633c1d20960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.438+0000 7f52ef7fe640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f52e40a3d50 0x7f52e40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.438+0000 7f52f5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20580 0x5633c1d20960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43302/0 (socket says 172.21.15.28:43302) 2024-04-21T08:57:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.438+0000 7f52f5d9b640 1 -- 172.21.15.28:0/3312039435 learned_addr learned my addr 172.21.15.28:0/3312039435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.438+0000 7f52f659c640 1 --2- 172.21.15.28:0/3312039435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d20ea0 0x5633c1d29540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.438+0000 7f52f6d9d640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f52e40a3d50 0x7f52e40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.439+0000 7f52f5d9b640 1 --2- 172.21.15.28:0/3312039435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d29a80 0x5633c1d2be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.439+0000 7f52f5d9b640 1 -- 172.21.15.28:0/3312039435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d20ea0 msgr2=0x5633c1d29540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.439+0000 7f52f5d9b640 1 --2- 172.21.15.28:0/3312039435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d20ea0 0x5633c1d29540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.439+0000 7f52f5d9b640 1 -- 172.21.15.28:0/3312039435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d29a80 msgr2=0x5633c1d2be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.439+0000 7f52f5d9b640 1 --2- 172.21.15.28:0/3312039435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d29a80 0x5633c1d2be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.439+0000 7f52ef7fe640 1 -- 172.21.15.28:0/511944052 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f52f00d0ff0 con 0x7f52f8074cd0 2024-04-21T08:57:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.439+0000 7f52f5d9b640 1 -- 172.21.15.28:0/3312039435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633c1a617d0 con 0x5633c1d20580 2024-04-21T08:57:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.439+0000 7f52f659c640 1 --2- 172.21.15.28:0/3312039435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d20ea0 0x5633c1d29540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:57:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.439+0000 7f52f5d9b640 1 --2- 172.21.15.28:0/3312039435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20580 0x5633c1d20960 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f52e0008ec0 tx=0x7f52e0008600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aabf7d385870d82b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.439+0000 7f52f6d9d640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f52e40a3d50 0x7f52e40a6210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f52f00ce9a0 tx=0x7f52f006e630 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.439+0000 7f52ecff9640 1 -- 172.21.15.28:0/3312039435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e0010ea0 con 0x5633c1d20580 2024-04-21T08:57:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.439+0000 7f52ecff9640 1 -- 172.21.15.28:0/3312039435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52e0016030 con 0x5633c1d20580 2024-04-21T08:57:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.439+0000 7f52ecff9640 1 -- 172.21.15.28:0/3312039435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e00128c0 con 0x5633c1d20580 2024-04-21T08:57:37.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.440+0000 7f52ff613740 1 -- 172.21.15.28:0/3312039435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20580 msgr2=0x5633c1d20960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.440+0000 7f52ff613740 1 --2- 172.21.15.28:0/3312039435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20580 0x5633c1d20960 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f52e0008ec0 tx=0x7f52e0008600 comp rx=0 tx=0).stop 2024-04-21T08:57:37.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.440+0000 7f52ff613740 1 -- 172.21.15.28:0/3312039435 shutdown_connections 2024-04-21T08:57:37.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.440+0000 7f52ff613740 1 --2- 172.21.15.28:0/3312039435 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d29a80 0x5633c1d2be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.440+0000 7f52ff613740 1 --2- 172.21.15.28:0/3312039435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d20ea0 0x5633c1d29540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.440+0000 7f52ff613740 1 --2- 172.21.15.28:0/3312039435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20580 0x5633c1d20960 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.440+0000 7f52ff613740 1 -- 172.21.15.28:0/3312039435 >> 172.21.15.28:0/3312039435 conn(0x5633c1d1acb0 msgr2=0x5633c1d1b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:37.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.440+0000 7f52ff613740 1 -- 172.21.15.28:0/3312039435 shutdown_connections 2024-04-21T08:57:37.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.440+0000 7f52ff613740 1 -- 172.21.15.28:0/3312039435 wait complete. 2024-04-21T08:57:37.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.440+0000 7f52ff613740 1 -- start start 2024-04-21T08:57:37.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.440+0000 7f52ff613740 1 Processor -- start 2024-04-21T08:57:37.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52ff613740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d20580 0x5633c1d26ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52ff613740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20ea0 0x5633c1d26ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52ff613740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d29a80 0x5633c1d27530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52ff613740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5633c1d2c870 con 0x5633c1d20ea0 2024-04-21T08:57:37.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52ff613740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5633c1c3e2f0 con 0x5633c1d20580 2024-04-21T08:57:37.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52ff613740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5633c1d2c9f0 con 0x5633c1d29a80 2024-04-21T08:57:37.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52ff613740 10 client.0 _ll_register_callbacks cb 0x5633c1c4b580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:37.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52f659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20ea0 0x5633c1d26ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52f659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20ea0 0x5633c1d26ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43308/0 (socket says 172.21.15.28:43308) 2024-04-21T08:57:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52f5d9b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d20580 0x5633c1d26ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52f659c640 1 -- 172.21.15.28:0/202233406 learned_addr learned my addr 172.21.15.28:0/202233406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52f5d9b640 1 --2- 172.21.15.28:0/202233406 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d29a80 0x5633c1d27530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52f659c640 1 -- 172.21.15.28:0/202233406 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d29a80 msgr2=0x5633c1d27530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52f659c640 1 --2- 172.21.15.28:0/202233406 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d29a80 0x5633c1d27530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52f659c640 1 -- 172.21.15.28:0/202233406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d20580 msgr2=0x5633c1d26ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52f659c640 1 --2- 172.21.15.28:0/202233406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d20580 0x5633c1d26ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52f659c640 1 -- 172.21.15.28:0/202233406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633c19973f0 con 0x5633c1d20ea0 2024-04-21T08:57:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52f659c640 1 --2- 172.21.15.28:0/202233406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20ea0 0x5633c1d26ff0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f52e80096c0 tx=0x7f52e802cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f330f3add8dc45e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52ddffb640 1 -- 172.21.15.28:0/202233406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e8005450 con 0x5633c1d20ea0 2024-04-21T08:57:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52ddffb640 1 -- 172.21.15.28:0/202233406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f52e80046c0 con 0x5633c1d20ea0 2024-04-21T08:57:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.441+0000 7f52ff613740 1 -- 172.21.15.28:0/202233406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52e0016030 con 0x5633c1d20ea0 2024-04-21T08:57:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.442+0000 7f52ddffb640 1 -- 172.21.15.28:0/202233406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f52e80055f0 con 0x5633c1d20ea0 2024-04-21T08:57:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.442+0000 7f52ff613740 10 client.17433 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.442+0000 7f52ff613740 1 -- 172.21.15.28:0/202233406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5633c19570f0 con 0x5633c1d20ea0 2024-04-21T08:57:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.442+0000 7f52ddffb640 1 -- 172.21.15.28:0/202233406 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52e0016030 con 0x5633c1d20ea0 2024-04-21T08:57:37.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.442+0000 7f52ddffb640 1 -- 172.21.15.28:0/202233406 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f52e8003a30 con 0x5633c1d20ea0 2024-04-21T08:57:37.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.442+0000 7f52ff613740 10 client.17433 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:37.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.442+0000 7f52ff613740 10 client.17433 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:37.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.442+0000 7f52ff613740 20 client.17433 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:37.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.442+0000 7f52ff613740 1 --2- 172.21.15.28:0/202233406 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5633c1d39a70 0x5633c1d3be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.442+0000 7f52ff613740 4 client.17433 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:57:37.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.442+0000 7f52ff613740 1 -- 172.21.15.28:0/202233406 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633c1c92c00 con 0x5633c1d39a70 2024-04-21T08:57:37.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.442+0000 7f52f5d9b640 1 --2- 172.21.15.28:0/202233406 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5633c1d39a70 0x5633c1d3be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.443+0000 7f52f5d9b640 1 --2- 172.21.15.28:0/202233406 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5633c1d39a70 0x5633c1d3be60 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f36e128fb15497d0 server_cookie=cee3149a813268dd in_seq=0 out_seq=0 2024-04-21T08:57:37.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.443+0000 7f52ddffb640 10 client.17433 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:37.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.443+0000 7f52ddffb640 1 -- 172.21.15.28:0/202233406 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5633c1c92c00 con 0x5633c1d39a70 2024-04-21T08:57:37.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.443+0000 7f52ddffb640 10 client.17433 handle_command_reply: tid=0 2024-04-21T08:57:37.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.443+0000 7f52ff613740 1 client.17433 shutdown 2024-04-21T08:57:37.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.444+0000 7f52ff613740 1 -- 172.21.15.28:0/202233406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20ea0 msgr2=0x5633c1d26ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.444+0000 7f52ff613740 1 --2- 172.21.15.28:0/202233406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20ea0 0x5633c1d26ff0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f52e80096c0 tx=0x7f52e802cab0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.444+0000 7f52ff613740 1 -- 172.21.15.28:0/202233406 shutdown_connections 2024-04-21T08:57:37.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.444+0000 7f52ff613740 1 --2- 172.21.15.28:0/202233406 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5633c1d39a70 0x5633c1d3be60 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.444+0000 7f52ff613740 1 --2- 172.21.15.28:0/202233406 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d29a80 0x5633c1d27530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.444+0000 7f52ff613740 1 --2- 172.21.15.28:0/202233406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20ea0 0x5633c1d26ff0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.444+0000 7f52ddffb640 0 client.17433 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.444+0000 7f52ff613740 1 --2- 172.21.15.28:0/202233406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d20580 0x5633c1d26ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.444+0000 7f52ff613740 1 -- 172.21.15.28:0/202233406 >> 172.21.15.28:0/202233406 conn(0x5633c1d1acb0 msgr2=0x5633c1d1ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.444+0000 7f52ff613740 1 -- 172.21.15.28:0/202233406 shutdown_connections 2024-04-21T08:57:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.444+0000 7f52ff613740 1 -- 172.21.15.28:0/202233406 wait complete. 2024-04-21T08:57:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.444+0000 7f52ff613740 20 client.17433 trim_cache size 0 max 16384 2024-04-21T08:57:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.461+0000 7f52ff613740 1 Processor -- start 2024-04-21T08:57:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.461+0000 7f52ff613740 1 -- start start 2024-04-21T08:57:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.461+0000 7f52ff613740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d20580 0x5633c1d291e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.461+0000 7f52ff613740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20ea0 0x5633c1d259f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.461+0000 7f52ff613740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d29a80 0x5633c1d21fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.461+0000 7f52ff613740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52f81b3c40 con 0x5633c1d20ea0 2024-04-21T08:57:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.461+0000 7f52ff613740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f52f81b1510 con 0x5633c1d20580 2024-04-21T08:57:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.461+0000 7f52ff613740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f52f81af360 con 0x5633c1d29a80 2024-04-21T08:57:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.462+0000 7f52f659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20ea0 0x5633c1d259f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.462+0000 7f52f659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20ea0 0x5633c1d259f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43324/0 (socket says 172.21.15.28:43324) 2024-04-21T08:57:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.462+0000 7f52f659c640 1 -- 172.21.15.28:0/1783421147 learned_addr learned my addr 172.21.15.28:0/1783421147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.462+0000 7f52f5d9b640 1 --2- 172.21.15.28:0/1783421147 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d20580 0x5633c1d291e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.462+0000 7f52f659c640 1 -- 172.21.15.28:0/1783421147 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d29a80 msgr2=0x5633c1d21fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.462+0000 7f52f659c640 1 --2- 172.21.15.28:0/1783421147 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d29a80 0x5633c1d21fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.462+0000 7f52f659c640 1 -- 172.21.15.28:0/1783421147 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d20580 msgr2=0x5633c1d291e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.462+0000 7f52f659c640 1 --2- 172.21.15.28:0/1783421147 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d20580 0x5633c1d291e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.462+0000 7f52f659c640 1 -- 172.21.15.28:0/1783421147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633c1d2be70 con 0x5633c1d20ea0 2024-04-21T08:57:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.462+0000 7f52f659c640 1 --2- 172.21.15.28:0/1783421147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20ea0 0x5633c1d259f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f52e802c190 tx=0x7f52e8005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b39fa7ec9f32c5b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.462+0000 7f52dffff640 1 -- 172.21.15.28:0/1783421147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e8009520 con 0x5633c1d20ea0 2024-04-21T08:57:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.462+0000 7f52dffff640 1 -- 172.21.15.28:0/1783421147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52e80055f0 con 0x5633c1d20ea0 2024-04-21T08:57:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.462+0000 7f52dffff640 1 -- 172.21.15.28:0/1783421147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e80035f0 con 0x5633c1d20ea0 2024-04-21T08:57:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 -- 172.21.15.28:0/1783421147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20ea0 msgr2=0x5633c1d259f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 --2- 172.21.15.28:0/1783421147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20ea0 0x5633c1d259f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f52e802c190 tx=0x7f52e8005a20 comp rx=0 tx=0).stop 2024-04-21T08:57:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 -- 172.21.15.28:0/1783421147 shutdown_connections 2024-04-21T08:57:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 --2- 172.21.15.28:0/1783421147 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d29a80 0x5633c1d21fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 --2- 172.21.15.28:0/1783421147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d20ea0 0x5633c1d259f0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 --2- 172.21.15.28:0/1783421147 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d20580 0x5633c1d291e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 -- 172.21.15.28:0/1783421147 >> 172.21.15.28:0/1783421147 conn(0x5633c1d1acb0 msgr2=0x5633c1d27300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 -- 172.21.15.28:0/1783421147 shutdown_connections 2024-04-21T08:57:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 -- 172.21.15.28:0/1783421147 wait complete. 2024-04-21T08:57:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 -- start start 2024-04-21T08:57:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 Processor -- start 2024-04-21T08:57:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d20580 0x5633c1d32050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d20ea0 0x5633c1d2d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d29a80 0x5633c1d2d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52f81af9d0 con 0x5633c1d29a80 2024-04-21T08:57:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f52f81b0e40 con 0x5633c1d20ea0 2024-04-21T08:57:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f52f81b00a0 con 0x5633c1d20580 2024-04-21T08:57:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.463+0000 7f52ff613740 10 client.0 _ll_register_callbacks cb 0x7f52e8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:57:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52f5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d29a80 0x5633c1d2d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52f5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d29a80 0x5633c1d2d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43334/0 (socket says 172.21.15.28:43334) 2024-04-21T08:57:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52f5d9b640 1 -- 172.21.15.28:0/3934871868 learned_addr learned my addr 172.21.15.28:0/3934871868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:57:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52f659c640 1 --2- 172.21.15.28:0/3934871868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d20ea0 0x5633c1d2d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52f5d9b640 1 --2- 172.21.15.28:0/3934871868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d20580 0x5633c1d32050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52f5d9b640 1 -- 172.21.15.28:0/3934871868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d20580 msgr2=0x5633c1d32050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52f5d9b640 1 --2- 172.21.15.28:0/3934871868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d20580 0x5633c1d32050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52f5d9b640 1 -- 172.21.15.28:0/3934871868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d20ea0 msgr2=0x5633c1d2d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52f5d9b640 1 --2- 172.21.15.28:0/3934871868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d20ea0 0x5633c1d2d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52f5d9b640 1 -- 172.21.15.28:0/3934871868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633c1d2e130 con 0x5633c1d29a80 2024-04-21T08:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52f5d9b640 1 --2- 172.21.15.28:0/3934871868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d29a80 0x5633c1d2d900 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f52e0008ac0 tx=0x7f52e002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e9cc1f21097bd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52ddffb640 1 -- 172.21.15.28:0/3934871868 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e0004840 con 0x5633c1d29a80 2024-04-21T08:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52ff613740 1 -- 172.21.15.28:0/3934871868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52e80055f0 con 0x5633c1d29a80 2024-04-21T08:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52ddffb640 1 -- 172.21.15.28:0/3934871868 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9445+0+0 (secure 0 0 0) 0x7f52e0025040 con 0x5633c1d29a80 2024-04-21T08:57:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52ddffb640 1 -- 172.21.15.28:0/3934871868 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f52e00049e0 con 0x5633c1d29a80 2024-04-21T08:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52ff613740 10 client.17439 fetch_fsmap learned FSMap version 28 2024-04-21T08:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52ff613740 1 -- 172.21.15.28:0/3934871868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5633c1da0060 con 0x5633c1d29a80 2024-04-21T08:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.464+0000 7f52ddffb640 1 -- 172.21.15.28:0/3934871868 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52e80055f0 con 0x5633c1d29a80 2024-04-21T08:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.465+0000 7f52ddffb640 1 -- 172.21.15.28:0/3934871868 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f52e0012780 con 0x5633c1d29a80 2024-04-21T08:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.465+0000 7f52ff613740 10 client.17439 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.465+0000 7f52ff613740 10 client.17439 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.465+0000 7f52ff613740 20 client.17439 populate_metadata read hostname 'smithi028' 2024-04-21T08:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.465+0000 7f52ff613740 1 --2- 172.21.15.28:0/3934871868 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5633c1da05a0 0x5633c1da2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.465+0000 7f52ff613740 4 client.17439 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.465+0000 7f52ff613740 1 -- 172.21.15.28:0/3934871868 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5633c1c8ca70 con 0x5633c1da05a0 2024-04-21T08:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.465+0000 7f52f659c640 1 --2- 172.21.15.28:0/3934871868 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5633c1da05a0 0x5633c1da2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.465+0000 7f52f659c640 1 --2- 172.21.15.28:0/3934871868 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5633c1da05a0 0x5633c1da2990 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77d40da1a64799fd server_cookie=1aaf54b2ef0671c4 in_seq=0 out_seq=0 2024-04-21T08:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.465+0000 7f52ddffb640 10 client.17439 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.465+0000 7f52ddffb640 1 -- 172.21.15.28:0/3934871868 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5633c1c8ca70 con 0x5633c1da05a0 2024-04-21T08:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.465+0000 7f52ddffb640 10 client.17439 handle_command_reply: tid=0 2024-04-21T08:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.465+0000 7f52ff613740 1 client.17439 shutdown 2024-04-21T08:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52ff613740 1 -- 172.21.15.28:0/3934871868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d29a80 msgr2=0x5633c1d2d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52ff613740 1 --2- 172.21.15.28:0/3934871868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d29a80 0x5633c1d2d900 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f52e0008ac0 tx=0x7f52e002e040 comp rx=0 tx=0).stop 2024-04-21T08:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52ff613740 1 -- 172.21.15.28:0/3934871868 shutdown_connections 2024-04-21T08:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52ff613740 1 --2- 172.21.15.28:0/3934871868 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5633c1da05a0 0x5633c1da2990 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52ff613740 1 --2- 172.21.15.28:0/3934871868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5633c1d29a80 0x5633c1d2d900 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52ddffb640 0 client.17439 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52ff613740 1 --2- 172.21.15.28:0/3934871868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5633c1d20ea0 0x5633c1d2d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52ff613740 1 --2- 172.21.15.28:0/3934871868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5633c1d20580 0x5633c1d32050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52ff613740 1 -- 172.21.15.28:0/3934871868 >> 172.21.15.28:0/3934871868 conn(0x5633c1d1acb0 msgr2=0x5633c1d194e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52ff613740 1 -- 172.21.15.28:0/3934871868 shutdown_connections 2024-04-21T08:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52ff613740 1 -- 172.21.15.28:0/3934871868 wait complete. 2024-04-21T08:57:37.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52ff613740 20 client.17439 trim_cache size 0 max 16384 2024-04-21T08:57:37.476 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:57:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:57:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:57:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "fdc2d2a7-c5a0-4ec9-abdb-9f31a600776a": { 2024-04-21T08:57:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:57:37.476 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "fdc2d2a7-c5a0-4ec9-abdb-9f31a600776a", 2024-04-21T08:57:37.477 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:57:37.477 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:57:37.477 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:57:37.477 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:57:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52fc94b640 1 -- 172.21.15.28:0/511944052 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f52e40a3d50 msgr2=0x7f52e40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52fc94b640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f52e40a3d50 0x7f52e40a6210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f52f00ce9a0 tx=0x7f52f006e630 comp rx=0 tx=0).stop 2024-04-21T08:57:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52fc94b640 1 -- 172.21.15.28:0/511944052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8074cd0 msgr2=0x7f52f81a6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52fc94b640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8074cd0 0x7f52f81a6820 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f52f000cce0 tx=0x7f52f000a620 comp rx=0 tx=0).stop 2024-04-21T08:57:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52fc94b640 1 -- 172.21.15.28:0/511944052 shutdown_connections 2024-04-21T08:57:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52fc94b640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f52e40a3d50 0x7f52e40a6210 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52fc94b640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52f8074cd0 0x7f52f81a6820 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52fc94b640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52f8107270 0x7f52f8074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52fc94b640 1 --2- 172.21.15.28:0/511944052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f52f81021a0 0x7f52f8074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52fc94b640 1 -- 172.21.15.28:0/511944052 >> 172.21.15.28:0/511944052 conn(0x7f52f80793e0 msgr2=0x7f52f80ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52fc94b640 1 -- 172.21.15.28:0/511944052 shutdown_connections 2024-04-21T08:57:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:57:37.466+0000 7f52fc94b640 1 -- 172.21.15.28:0/511944052 wait complete. 2024-04-21T08:57:37.480 INFO:tasks.cephfs.filesystem:scrub status for tag:fdc2d2a7-c5a0-4ec9-abdb-9f31a600776a - {'path': '/', 'tag': 'fdc2d2a7-c5a0-4ec9-abdb-9f31a600776a', 'options': 'recursive,force'} 2024-04-21T08:57:38.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: pgmap v196: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:57:38.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:57:38.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:38.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:38.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:38.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:38.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:38.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:38.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:38.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:38.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:38.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.608 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:57:38.608 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.608 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:38 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: pgmap v196: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:57:38.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:57:38.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:38.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:38.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:38.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:38.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:38.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:38.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:38.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:38.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:38.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:57:38.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:38 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: pgmap v196: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:57:38.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:57:38.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:38.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:38.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:38.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:38.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:38.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:38.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:38.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:38.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:38.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:57:38.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:38.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:38 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:57:40.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:40 smithi152 ceph-mon[31401]: pgmap v197: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 op/s 2024-04-21T08:57:40.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:40 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:57:40.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:40 smithi028 ceph-mon[28003]: pgmap v197: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 op/s 2024-04-21T08:57:40.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:40 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:57:40.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:40 smithi171 ceph-mon[31358]: pgmap v197: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 op/s 2024-04-21T08:57:40.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:40 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:57:41.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:41 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:57:41.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:41 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:57:41.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:41 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:57:41.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:41 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:57:41.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:41 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:57:41.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:41 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:57:41.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:41 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:57:41.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:41 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:57:41.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:41 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:57:42.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:42 smithi152 ceph-mon[31401]: pgmap v198: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 op/s 2024-04-21T08:57:42.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:42 smithi028 ceph-mon[28003]: pgmap v198: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 op/s 2024-04-21T08:57:42.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:42 smithi171 ceph-mon[31358]: pgmap v198: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 22 op/s 2024-04-21T08:57:44.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:44 smithi152 ceph-mon[31401]: pgmap v199: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:57:44.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:44 smithi028 ceph-mon[28003]: pgmap v199: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:57:44.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:44 smithi171 ceph-mon[31358]: pgmap v199: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:57:46.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:46 smithi152 ceph-mon[31401]: pgmap v200: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 28 op/s 2024-04-21T08:57:46.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:46 smithi028 ceph-mon[28003]: pgmap v200: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 28 op/s 2024-04-21T08:57:46.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:46 smithi171 ceph-mon[31358]: pgmap v200: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 28 op/s 2024-04-21T08:57:47.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:47 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:57:47.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:47 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:57:47.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:47 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:57:47.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:47 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:57:47.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:47 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:57:47.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:47 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:57:47.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:47 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:57:47.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:47 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:57:47.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:47 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:57:48.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:48 smithi152 ceph-mon[31401]: pgmap v201: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:57:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:48 smithi028 ceph-mon[28003]: pgmap v201: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:57:48.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:48 smithi171 ceph-mon[31358]: pgmap v201: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:57:50.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:50 smithi152 ceph-mon[31401]: pgmap v202: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 29 op/s 2024-04-21T08:57:50.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:50 smithi028 ceph-mon[28003]: pgmap v202: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 29 op/s 2024-04-21T08:57:50.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:50 smithi171 ceph-mon[31358]: pgmap v202: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 29 op/s 2024-04-21T08:57:52.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:52 smithi152 ceph-mon[31401]: pgmap v203: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 op/s 2024-04-21T08:57:52.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:52 smithi028 ceph-mon[28003]: pgmap v203: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 op/s 2024-04-21T08:57:52.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:52 smithi171 ceph-mon[31358]: pgmap v203: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 op/s 2024-04-21T08:57:54.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:54 smithi152 ceph-mon[31401]: pgmap v204: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 11 op/s 2024-04-21T08:57:54.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:54 smithi028 ceph-mon[28003]: pgmap v204: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 11 op/s 2024-04-21T08:57:54.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:54 smithi171 ceph-mon[31358]: pgmap v204: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 11 op/s 2024-04-21T08:57:56.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:56 smithi152 ceph-mon[31401]: pgmap v205: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:57:56.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:56 smithi028 ceph-mon[28003]: pgmap v205: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:57:56.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:56 smithi171 ceph-mon[31358]: pgmap v205: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:57:58.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:57:58 smithi152 ceph-mon[31401]: pgmap v206: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:57:58.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:57:58 smithi028 ceph-mon[28003]: pgmap v206: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:57:58.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:57:58 smithi171 ceph-mon[31358]: pgmap v206: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:58:00.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:00 smithi152 ceph-mon[31401]: pgmap v207: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:58:00.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:00 smithi028 ceph-mon[28003]: pgmap v207: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:58:00.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:00 smithi171 ceph-mon[31358]: pgmap v207: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:58:02.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:02 smithi152 ceph-mon[31401]: pgmap v208: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:58:02.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:02 smithi028 ceph-mon[28003]: pgmap v208: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:58:02.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:02 smithi171 ceph-mon[31358]: pgmap v208: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:58:04.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:04 smithi152 ceph-mon[31401]: pgmap v209: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T08:58:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:04 smithi028 ceph-mon[28003]: pgmap v209: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T08:58:04.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:04 smithi171 ceph-mon[31358]: pgmap v209: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T08:58:06.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:06 smithi152 ceph-mon[31401]: pgmap v210: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T08:58:06.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:06 smithi028 ceph-mon[28003]: pgmap v210: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T08:58:06.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:06 smithi171 ceph-mon[31358]: pgmap v210: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T08:58:07.482 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:58:07.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:58:07.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.599+0000 7f22424fd640 1 -- 172.21.15.28:0/1648031527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223c10c500 msgr2=0x7f223c10e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:07.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.599+0000 7f22424fd640 1 --2- 172.21.15.28:0/1648031527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223c10c500 0x7f223c10e9a0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f2234009ca0 tx=0x7f223402c230 comp rx=0 tx=0).stop 2024-04-21T08:58:07.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.599+0000 7f22424fd640 1 -- 172.21.15.28:0/1648031527 shutdown_connections 2024-04-21T08:58:07.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.599+0000 7f22424fd640 1 --2- 172.21.15.28:0/1648031527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223c10c500 0x7f223c10e9a0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.599+0000 7f22424fd640 1 --2- 172.21.15.28:0/1648031527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f223c103ae0 0x7f223c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.599+0000 7f22424fd640 1 --2- 172.21.15.28:0/1648031527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f223c1031c0 0x7f223c1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.599+0000 7f22424fd640 1 -- 172.21.15.28:0/1648031527 >> 172.21.15.28:0/1648031527 conn(0x7f223c078000 msgr2=0x7f223c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:07.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.599+0000 7f22424fd640 1 -- 172.21.15.28:0/1648031527 shutdown_connections 2024-04-21T08:58:07.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.599+0000 7f22424fd640 1 -- 172.21.15.28:0/1648031527 wait complete. 2024-04-21T08:58:07.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f22424fd640 1 Processor -- start 2024-04-21T08:58:07.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f22424fd640 1 -- start start 2024-04-21T08:58:07.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f22424fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223c1031c0 0x7f223c100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f22424fd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f223c103ae0 0x7f223c101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f22424fd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f223c10c500 0x7f223c101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f22424fd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f223c112170 con 0x7f223c1031c0 2024-04-21T08:58:07.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f22424fd640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f223c111ff0 con 0x7f223c10c500 2024-04-21T08:58:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f22424fd640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1122f0 con 0x7f223c103ae0 2024-04-21T08:58:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f223bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223c1031c0 0x7f223c100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f223bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223c1031c0 0x7f223c100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51688/0 (socket says 172.21.15.28:51688) 2024-04-21T08:58:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f223bfff640 1 -- 172.21.15.28:0/1459411925 learned_addr learned my addr 172.21.15.28:0/1459411925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f223b7fe640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f223c103ae0 0x7f223c101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f2240a9a640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f223c10c500 0x7f223c101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f223bfff640 1 -- 172.21.15.28:0/1459411925 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f223c103ae0 msgr2=0x7f223c101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:07.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f223bfff640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f223c103ae0 0x7f223c101350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f223bfff640 1 -- 172.21.15.28:0/1459411925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f223c10c500 msgr2=0x7f223c101890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:07.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f223bfff640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f223c10c500 0x7f223c101890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f223bfff640 1 -- 172.21.15.28:0/1459411925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223c1a6bb0 con 0x7f223c1031c0 2024-04-21T08:58:07.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f223b7fe640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f223c103ae0 0x7f223c101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:07.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.600+0000 7f2240a9a640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f223c10c500 0x7f223c101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:07.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.601+0000 7f223bfff640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223c1031c0 0x7f223c100e10 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f222c00d950 tx=0x7f222c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:07.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.601+0000 7f22397fa640 1 -- 172.21.15.28:0/1459411925 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c017070 con 0x7f223c1031c0 2024-04-21T08:58:07.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.601+0000 7f22397fa640 1 -- 172.21.15.28:0/1459411925 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f222c009c30 con 0x7f223c1031c0 2024-04-21T08:58:07.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.601+0000 7f22424fd640 1 -- 172.21.15.28:0/1459411925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f223c1a6ea0 con 0x7f223c1031c0 2024-04-21T08:58:07.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.601+0000 7f22397fa640 1 -- 172.21.15.28:0/1459411925 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c00f620 con 0x7f223c1031c0 2024-04-21T08:58:07.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.601+0000 7f22424fd640 1 -- 172.21.15.28:0/1459411925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f223c1a7360 con 0x7f223c1031c0 2024-04-21T08:58:07.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.603+0000 7f22397fa640 1 -- 172.21.15.28:0/1459411925 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f222c010460 con 0x7f223c1031c0 2024-04-21T08:58:07.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.603+0000 7f22451c8740 1 Processor -- start 2024-04-21T08:58:07.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f22451c8740 1 -- start start 2024-04-21T08:58:07.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f22451c8740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24c580 0x55872c24c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f22451c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24cea0 0x55872c255540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f22451c8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c255a80 0x55872c257e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f22451c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55872c1b11e0 con 0x55872c24cea0 2024-04-21T08:58:07.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f22451c8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55872c14e390 con 0x55872c255a80 2024-04-21T08:58:07.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f22451c8740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55872c1ad1a0 con 0x55872c24c580 2024-04-21T08:58:07.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f223bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24cea0 0x55872c255540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f223bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24cea0 0x55872c255540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51694/0 (socket says 172.21.15.28:51694) 2024-04-21T08:58:07.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f223bfff640 1 -- 172.21.15.28:0/589960519 learned_addr learned my addr 172.21.15.28:0/589960519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f2240a9a640 1 --2- 172.21.15.28:0/589960519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24c580 0x55872c24c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f22397fa640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f22240a3d20 0x7f22240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f223b7fe640 1 --2- 172.21.15.28:0/589960519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c255a80 0x55872c257e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f2240a9a640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f22240a3d20 0x7f22240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f22397fa640 1 -- 172.21.15.28:0/1459411925 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7f222c0d0920 con 0x7f223c1031c0 2024-04-21T08:58:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f2240a9a640 1 -- 172.21.15.28:0/589960519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c255a80 msgr2=0x55872c257e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f2240a9a640 1 --2- 172.21.15.28:0/589960519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c255a80 0x55872c257e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f2240a9a640 1 -- 172.21.15.28:0/589960519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24cea0 msgr2=0x55872c255540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f2240a9a640 1 --2- 172.21.15.28:0/589960519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24cea0 0x55872c255540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f2240a9a640 1 -- 172.21.15.28:0/589960519 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55872bf8d7d0 con 0x55872c24c580 2024-04-21T08:58:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.604+0000 7f223bfff640 1 --2- 172.21.15.28:0/589960519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24cea0 0x55872c255540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:58:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.605+0000 7f2240a9a640 1 --2- 172.21.15.28:0/589960519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24c580 0x55872c24c960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2234009430 tx=0x7f2234003b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=254f4c8edd19b642 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.605+0000 7f2240a9a640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f22240a3d20 0x7f22240a61e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2234045000 tx=0x7f2234048000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:07.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.605+0000 7f22227fc640 1 -- 172.21.15.28:0/589960519 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2234039070 con 0x55872c24c580 2024-04-21T08:58:07.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.605+0000 7f22227fc640 1 -- 172.21.15.28:0/589960519 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2234032640 con 0x55872c24c580 2024-04-21T08:58:07.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.605+0000 7f22227fc640 1 -- 172.21.15.28:0/589960519 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22340346a0 con 0x55872c24c580 2024-04-21T08:58:07.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.606+0000 7f22451c8740 1 -- 172.21.15.28:0/589960519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24c580 msgr2=0x55872c24c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:07.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.606+0000 7f22451c8740 1 --2- 172.21.15.28:0/589960519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24c580 0x55872c24c960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2234009430 tx=0x7f2234003b00 comp rx=0 tx=0).stop 2024-04-21T08:58:07.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.606+0000 7f22451c8740 1 -- 172.21.15.28:0/589960519 shutdown_connections 2024-04-21T08:58:07.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.606+0000 7f22451c8740 1 --2- 172.21.15.28:0/589960519 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c255a80 0x55872c257e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.606+0000 7f22451c8740 1 --2- 172.21.15.28:0/589960519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24cea0 0x55872c255540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.606+0000 7f22451c8740 1 --2- 172.21.15.28:0/589960519 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24c580 0x55872c24c960 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.606+0000 7f22451c8740 1 -- 172.21.15.28:0/589960519 >> 172.21.15.28:0/589960519 conn(0x55872c246cb0 msgr2=0x55872c247090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:07.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.606+0000 7f22451c8740 1 -- 172.21.15.28:0/589960519 shutdown_connections 2024-04-21T08:58:07.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.606+0000 7f22451c8740 1 -- 172.21.15.28:0/589960519 wait complete. 2024-04-21T08:58:07.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.606+0000 7f22451c8740 1 -- start start 2024-04-21T08:58:07.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.606+0000 7f22451c8740 1 Processor -- start 2024-04-21T08:58:07.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f22451c8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c24c580 0x55872c252b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f22451c8740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24cea0 0x55872c253090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f22451c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c255a80 0x55872c2535d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f22451c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55872c258810 con 0x55872c255a80 2024-04-21T08:58:07.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f22451c8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55872c16a2f0 con 0x55872c24c580 2024-04-21T08:58:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f22451c8740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55872c258990 con 0x55872c24cea0 2024-04-21T08:58:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f22451c8740 10 client.0 _ll_register_callbacks cb 0x55872c177580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f2240a9a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c255a80 0x55872c2535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f223b7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24cea0 0x55872c253090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f2240a9a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c255a80 0x55872c2535d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51704/0 (socket says 172.21.15.28:51704) 2024-04-21T08:58:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f223b7fe640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24cea0 0x55872c253090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:35990/0 (socket says 172.21.15.28:35990) 2024-04-21T08:58:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f2240a9a640 1 -- 172.21.15.28:0/223262748 learned_addr learned my addr 172.21.15.28:0/223262748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f223bfff640 1 --2- 172.21.15.28:0/223262748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c24c580 0x55872c252b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f2240a9a640 1 -- 172.21.15.28:0/223262748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24cea0 msgr2=0x55872c253090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f2240a9a640 1 --2- 172.21.15.28:0/223262748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24cea0 0x55872c253090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f2240a9a640 1 -- 172.21.15.28:0/223262748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c24c580 msgr2=0x55872c252b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f2240a9a640 1 --2- 172.21.15.28:0/223262748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c24c580 0x55872c252b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f2240a9a640 1 -- 172.21.15.28:0/223262748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55872bec33f0 con 0x55872c255a80 2024-04-21T08:58:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f223b7fe640 1 --2- 172.21.15.28:0/223262748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24cea0 0x55872c253090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:58:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f223bfff640 1 --2- 172.21.15.28:0/223262748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c24c580 0x55872c252b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:58:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f2240a9a640 1 --2- 172.21.15.28:0/223262748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c255a80 0x55872c2535d0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f2234037040 tx=0x7f22340323e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59108c173679e740 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.607+0000 7f221b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.608+0000 7f221b7fe640 1 -- 172.21.15.28:0/223262748 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2234051030 con 0x55872c255a80 2024-04-21T08:58:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.608+0000 7f221b7fe640 1 -- 172.21.15.28:0/223262748 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7f22340046e0 con 0x55872c255a80 2024-04-21T08:58:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.608+0000 7f22451c8740 1 -- 172.21.15.28:0/223262748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2234032640 con 0x55872c255a80 2024-04-21T08:58:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.608+0000 7f221b7fe640 1 -- 172.21.15.28:0/223262748 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2234062050 con 0x55872c255a80 2024-04-21T08:58:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.608+0000 7f22451c8740 10 client.17451 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.608+0000 7f22451c8740 1 -- 172.21.15.28:0/223262748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55872be830f0 con 0x55872c255a80 2024-04-21T08:58:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.608+0000 7f221b7fe640 1 -- 172.21.15.28:0/223262748 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2234032640 con 0x55872c255a80 2024-04-21T08:58:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.608+0000 7f221b7fe640 1 -- 172.21.15.28:0/223262748 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f2234034cc0 con 0x55872c255a80 2024-04-21T08:58:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.608+0000 7f22451c8740 10 client.17451 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.608+0000 7f22451c8740 10 client.17451 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.608+0000 7f22451c8740 20 client.17451 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.608+0000 7f22451c8740 1 --2- 172.21.15.28:0/223262748 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55872c265960 0x55872c267d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.608+0000 7f22451c8740 4 client.17451 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:58:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.608+0000 7f22451c8740 1 -- 172.21.15.28:0/223262748 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55872c1bec00 con 0x55872c265960 2024-04-21T08:58:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.609+0000 7f223bfff640 1 --2- 172.21.15.28:0/223262748 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55872c265960 0x55872c267d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.609+0000 7f223bfff640 1 --2- 172.21.15.28:0/223262748 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55872c265960 0x55872c267d50 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e158aacc830bbde6 server_cookie=9c01b05d75bf3f18 in_seq=0 out_seq=0 2024-04-21T08:58:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.609+0000 7f221b7fe640 10 client.17451 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:07.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.610+0000 7f221b7fe640 1 -- 172.21.15.28:0/223262748 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55872c1bec00 con 0x55872c265960 2024-04-21T08:58:07.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.610+0000 7f221b7fe640 10 client.17451 handle_command_reply: tid=0 2024-04-21T08:58:07.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.610+0000 7f22451c8740 1 client.17451 shutdown 2024-04-21T08:58:07.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.610+0000 7f22451c8740 1 -- 172.21.15.28:0/223262748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c255a80 msgr2=0x55872c2535d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:07.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.610+0000 7f22451c8740 1 --2- 172.21.15.28:0/223262748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c255a80 0x55872c2535d0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f2234037040 tx=0x7f22340323e0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.610+0000 7f22451c8740 1 -- 172.21.15.28:0/223262748 shutdown_connections 2024-04-21T08:58:07.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.610+0000 7f22451c8740 1 --2- 172.21.15.28:0/223262748 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55872c265960 0x55872c267d50 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.610+0000 7f22451c8740 1 --2- 172.21.15.28:0/223262748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c255a80 0x55872c2535d0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.610+0000 7f221b7fe640 0 client.17451 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:07.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.610+0000 7f22451c8740 1 --2- 172.21.15.28:0/223262748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24cea0 0x55872c253090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.610+0000 7f22451c8740 1 --2- 172.21.15.28:0/223262748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c24c580 0x55872c252b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.610+0000 7f22451c8740 1 -- 172.21.15.28:0/223262748 >> 172.21.15.28:0/223262748 conn(0x55872c246cb0 msgr2=0x55872c248a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:07.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.610+0000 7f22451c8740 1 -- 172.21.15.28:0/223262748 shutdown_connections 2024-04-21T08:58:07.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.610+0000 7f22451c8740 1 -- 172.21.15.28:0/223262748 wait complete. 2024-04-21T08:58:07.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.610+0000 7f22451c8740 20 client.17451 trim_cache size 0 max 16384 2024-04-21T08:58:07.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f22451c8740 1 Processor -- start 2024-04-21T08:58:07.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f22451c8740 1 -- start start 2024-04-21T08:58:07.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f22451c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24c580 0x55872c261700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f22451c8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c24cea0 0x55872c251a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f22451c8740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c255a80 0x55872c2533f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f22451c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1b3fb0 con 0x55872c24c580 2024-04-21T08:58:07.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f22451c8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1b1880 con 0x55872c24cea0 2024-04-21T08:58:07.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f22451c8740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1af6f0 con 0x55872c255a80 2024-04-21T08:58:07.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f223bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24c580 0x55872c261700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f223bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24c580 0x55872c261700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51712/0 (socket says 172.21.15.28:51712) 2024-04-21T08:58:07.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f223bfff640 1 -- 172.21.15.28:0/269461678 learned_addr learned my addr 172.21.15.28:0/269461678 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:07.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f223b7fe640 1 --2- 172.21.15.28:0/269461678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c24cea0 0x55872c251a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f223bfff640 1 -- 172.21.15.28:0/269461678 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c255a80 msgr2=0x55872c2533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:07.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f223bfff640 1 --2- 172.21.15.28:0/269461678 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c255a80 0x55872c2533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f223bfff640 1 -- 172.21.15.28:0/269461678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c24cea0 msgr2=0x55872c251a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:07.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f223bfff640 1 --2- 172.21.15.28:0/269461678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c24cea0 0x55872c251a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f223bfff640 1 -- 172.21.15.28:0/269461678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55872c2581e0 con 0x55872c24c580 2024-04-21T08:58:07.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f223b7fe640 1 --2- 172.21.15.28:0/269461678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c24cea0 0x55872c251a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:07.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.628+0000 7f223bfff640 1 --2- 172.21.15.28:0/269461678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24c580 0x55872c261700 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f222c0cdcd0 tx=0x7f222c0d2ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ffbebfd88a2144 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:07.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.629+0000 7f2221ffb640 1 -- 172.21.15.28:0/269461678 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c020040 con 0x55872c24c580 2024-04-21T08:58:07.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.629+0000 7f2221ffb640 1 -- 172.21.15.28:0/269461678 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f222c09c470 con 0x55872c24c580 2024-04-21T08:58:07.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.629+0000 7f2221ffb640 1 -- 172.21.15.28:0/269461678 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c0d0480 con 0x55872c24c580 2024-04-21T08:58:07.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.629+0000 7f22451c8740 1 -- 172.21.15.28:0/269461678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24c580 msgr2=0x55872c261700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:07.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.629+0000 7f22451c8740 1 --2- 172.21.15.28:0/269461678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24c580 0x55872c261700 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f222c0cdcd0 tx=0x7f222c0d2ea0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.629+0000 7f22451c8740 1 -- 172.21.15.28:0/269461678 shutdown_connections 2024-04-21T08:58:07.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.629+0000 7f22451c8740 1 --2- 172.21.15.28:0/269461678 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c255a80 0x55872c2533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.629+0000 7f22451c8740 1 --2- 172.21.15.28:0/269461678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c24cea0 0x55872c251a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.629+0000 7f22451c8740 1 --2- 172.21.15.28:0/269461678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24c580 0x55872c261700 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.629+0000 7f22451c8740 1 -- 172.21.15.28:0/269461678 >> 172.21.15.28:0/269461678 conn(0x55872c246cb0 msgr2=0x55872c248970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:07.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.629+0000 7f22451c8740 1 -- 172.21.15.28:0/269461678 shutdown_connections 2024-04-21T08:58:07.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.629+0000 7f22451c8740 1 -- 172.21.15.28:0/269461678 wait complete. 2024-04-21T08:58:07.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f22451c8740 1 -- start start 2024-04-21T08:58:07.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f22451c8740 1 Processor -- start 2024-04-21T08:58:07.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f22451c8740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24c580 0x55872c25dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f22451c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24cea0 0x55872c25e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f22451c8740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c255a80 0x55872c2593d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f22451c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1afd40 con 0x55872c24cea0 2024-04-21T08:58:07.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f22451c8740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1b11b0 con 0x55872c255a80 2024-04-21T08:58:07.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f22451c8740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f223c1b0410 con 0x55872c24c580 2024-04-21T08:58:07.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f22451c8740 10 client.0 _ll_register_callbacks cb 0x7f222c098790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:07.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f223b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24cea0 0x55872c25e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f223b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24cea0 0x55872c25e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51716/0 (socket says 172.21.15.28:51716) 2024-04-21T08:58:07.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f223b7fe640 1 -- 172.21.15.28:0/2580277011 learned_addr learned my addr 172.21.15.28:0/2580277011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:07.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f223bfff640 1 --2- 172.21.15.28:0/2580277011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24c580 0x55872c25dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f223b7fe640 1 -- 172.21.15.28:0/2580277011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24c580 msgr2=0x55872c25dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:07.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f223b7fe640 1 --2- 172.21.15.28:0/2580277011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24c580 0x55872c25dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f2240a9a640 1 --2- 172.21.15.28:0/2580277011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c255a80 0x55872c2593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f223b7fe640 1 -- 172.21.15.28:0/2580277011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c255a80 msgr2=0x55872c2593d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:07.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f223b7fe640 1 --2- 172.21.15.28:0/2580277011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c255a80 0x55872c2593d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f223b7fe640 1 -- 172.21.15.28:0/2580277011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55872c259ba0 con 0x55872c24cea0 2024-04-21T08:58:07.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.630+0000 7f2240a9a640 1 --2- 172.21.15.28:0/2580277011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c255a80 0x55872c2593d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:07.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f223b7fe640 1 --2- 172.21.15.28:0/2580277011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24cea0 0x55872c25e170 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f2230005fd0 tx=0x7f2230013cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d4097b7225452a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:07.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f221b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:07.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f221b7fe640 1 -- 172.21.15.28:0/2580277011 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2230004030 con 0x55872c24cea0 2024-04-21T08:58:07.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f221b7fe640 1 -- 172.21.15.28:0/2580277011 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7f22300041d0 con 0x55872c24cea0 2024-04-21T08:58:07.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f22451c8740 1 -- 172.21.15.28:0/2580277011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f222c09c470 con 0x55872c24cea0 2024-04-21T08:58:07.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f221b7fe640 1 -- 172.21.15.28:0/2580277011 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f22300043d0 con 0x55872c24cea0 2024-04-21T08:58:07.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f22451c8740 10 client.17457 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:07.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f22451c8740 1 -- 172.21.15.28:0/2580277011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55872c2cb990 con 0x55872c24cea0 2024-04-21T08:58:07.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f221b7fe640 1 -- 172.21.15.28:0/2580277011 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f222c09c470 con 0x55872c24cea0 2024-04-21T08:58:07.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f221b7fe640 1 -- 172.21.15.28:0/2580277011 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f22300039d0 con 0x55872c24cea0 2024-04-21T08:58:07.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f22451c8740 10 client.17457 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:07.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f22451c8740 10 client.17457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:07.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f22451c8740 20 client.17457 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:07.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f22451c8740 1 --2- 172.21.15.28:0/2580277011 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55872c2cc170 0x55872c2ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f22451c8740 4 client.17457 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:58:07.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.631+0000 7f22451c8740 1 -- 172.21.15.28:0/2580277011 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55872c1b8a70 con 0x55872c2cc170 2024-04-21T08:58:07.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.632+0000 7f223bfff640 1 --2- 172.21.15.28:0/2580277011 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55872c2cc170 0x55872c2ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.632+0000 7f221b7fe640 10 client.17457 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:07.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.632+0000 7f223bfff640 1 --2- 172.21.15.28:0/2580277011 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55872c2cc170 0x55872c2ce590 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f13e5fe4a71d6f24 server_cookie=4b27cf9518bc93c2 in_seq=0 out_seq=0 2024-04-21T08:58:07.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.632+0000 7f221b7fe640 1 -- 172.21.15.28:0/2580277011 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55872c1b8a70 con 0x55872c2cc170 2024-04-21T08:58:07.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.632+0000 7f221b7fe640 10 client.17457 handle_command_reply: tid=0 2024-04-21T08:58:07.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.632+0000 7f22451c8740 1 client.17457 shutdown 2024-04-21T08:58:07.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22451c8740 1 -- 172.21.15.28:0/2580277011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24cea0 msgr2=0x55872c25e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:07.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22451c8740 1 --2- 172.21.15.28:0/2580277011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24cea0 0x55872c25e170 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f2230005fd0 tx=0x7f2230013cb0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22451c8740 1 -- 172.21.15.28:0/2580277011 shutdown_connections 2024-04-21T08:58:07.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22451c8740 1 --2- 172.21.15.28:0/2580277011 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55872c2cc170 0x55872c2ce590 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22451c8740 1 --2- 172.21.15.28:0/2580277011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55872c255a80 0x55872c2593d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22451c8740 1 --2- 172.21.15.28:0/2580277011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55872c24cea0 0x55872c25e170 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f221b7fe640 0 client.17457 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:07.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22451c8740 1 --2- 172.21.15.28:0/2580277011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55872c24c580 0x55872c25dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22451c8740 1 -- 172.21.15.28:0/2580277011 >> 172.21.15.28:0/2580277011 conn(0x55872c246cb0 msgr2=0x55872c2454e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:07.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22451c8740 1 -- 172.21.15.28:0/2580277011 shutdown_connections 2024-04-21T08:58:07.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22451c8740 1 -- 172.21.15.28:0/2580277011 wait complete. 2024-04-21T08:58:07.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22451c8740 20 client.17457 trim_cache size 0 max 16384 2024-04-21T08:58:07.643 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:58:07.643 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T08:58:07.644 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T08:58:07.644 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:58:07.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22424fd640 1 -- 172.21.15.28:0/1459411925 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f22240a3d20 msgr2=0x7f22240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:07.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22424fd640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f22240a3d20 0x7f22240a61e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2234045000 tx=0x7f2234048000 comp rx=0 tx=0).stop 2024-04-21T08:58:07.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22424fd640 1 -- 172.21.15.28:0/1459411925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223c1031c0 msgr2=0x7f223c100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:07.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22424fd640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223c1031c0 0x7f223c100e10 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f222c00d950 tx=0x7f222c00de10 comp rx=0 tx=0).stop 2024-04-21T08:58:07.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22424fd640 1 -- 172.21.15.28:0/1459411925 shutdown_connections 2024-04-21T08:58:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22424fd640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f22240a3d20 0x7f22240a61e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22424fd640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f223c10c500 0x7f223c101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22424fd640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f223c103ae0 0x7f223c101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22424fd640 1 --2- 172.21.15.28:0/1459411925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f223c1031c0 0x7f223c100e10 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22424fd640 1 -- 172.21.15.28:0/1459411925 >> 172.21.15.28:0/1459411925 conn(0x7f223c078000 msgr2=0x7f223c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22424fd640 1 -- 172.21.15.28:0/1459411925 shutdown_connections 2024-04-21T08:58:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.633+0000 7f22424fd640 1 -- 172.21.15.28:0/1459411925 wait complete. 2024-04-21T08:58:07.647 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:58:07.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:58:07.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.749+0000 7f064194b640 1 -- 172.21.15.28:0/3125456752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f063c102b50 msgr2=0x7f063c106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:07.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.749+0000 7f064194b640 1 --2- 172.21.15.28:0/3125456752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f063c102b50 0x7f063c106b90 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f0624009900 tx=0x7f062402bc30 comp rx=0 tx=0).stop 2024-04-21T08:58:07.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.749+0000 7f064194b640 1 -- 172.21.15.28:0/3125456752 shutdown_connections 2024-04-21T08:58:07.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.749+0000 7f064194b640 1 --2- 172.21.15.28:0/3125456752 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f063c107270 0x7f063c10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.749+0000 7f064194b640 1 --2- 172.21.15.28:0/3125456752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f063c102b50 0x7f063c106b90 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.749+0000 7f064194b640 1 --2- 172.21.15.28:0/3125456752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f063c1021a0 0x7f063c102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.749+0000 7f064194b640 1 -- 172.21.15.28:0/3125456752 >> 172.21.15.28:0/3125456752 conn(0x7f063c0793a0 msgr2=0x7f063c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.749+0000 7f064194b640 1 -- 172.21.15.28:0/3125456752 shutdown_connections 2024-04-21T08:58:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.749+0000 7f064194b640 1 -- 172.21.15.28:0/3125456752 wait complete. 2024-04-21T08:58:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.750+0000 7f064194b640 1 Processor -- start 2024-04-21T08:58:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.750+0000 7f064194b640 1 -- start start 2024-04-21T08:58:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.750+0000 7f064194b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f063c1021a0 0x7f063c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.750+0000 7f064194b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f063c107270 0x7f063c198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.750+0000 7f064194b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f063c19cfd0 0x7f063c19d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.750+0000 7f064194b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f063c1121b0 con 0x7f063c1021a0 2024-04-21T08:58:07.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.750+0000 7f064194b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f063c112030 con 0x7f063c107270 2024-04-21T08:58:07.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.750+0000 7f064194b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f063c112330 con 0x7f063c19cfd0 2024-04-21T08:58:07.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.750+0000 7f063affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f063c1021a0 0x7f063c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.750+0000 7f063affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f063c1021a0 0x7f063c198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51746/0 (socket says 172.21.15.28:51746) 2024-04-21T08:58:07.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.750+0000 7f063affd640 1 -- 172.21.15.28:0/3319040864 learned_addr learned my addr 172.21.15.28:0/3319040864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.751+0000 7f063b7fe640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f063c19cfd0 0x7f063c19d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.751+0000 7f0633fff640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f063c107270 0x7f063c198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.751+0000 7f063affd640 1 -- 172.21.15.28:0/3319040864 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f063c19cfd0 msgr2=0x7f063c19d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.751+0000 7f063affd640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f063c19cfd0 0x7f063c19d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.751+0000 7f063affd640 1 -- 172.21.15.28:0/3319040864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f063c107270 msgr2=0x7f063c198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.751+0000 7f063affd640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f063c107270 0x7f063c198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.751+0000 7f063affd640 1 -- 172.21.15.28:0/3319040864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f063c19dac0 con 0x7f063c1021a0 2024-04-21T08:58:07.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.751+0000 7f0633fff640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f063c107270 0x7f063c198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.751+0000 7f063b7fe640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f063c19cfd0 0x7f063c19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.751+0000 7f063affd640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f063c1021a0 0x7f063c198710 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f062c00d950 tx=0x7f062c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.751+0000 7f0640949640 1 -- 172.21.15.28:0/3319040864 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062c017070 con 0x7f063c1021a0 2024-04-21T08:58:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.751+0000 7f064194b640 1 -- 172.21.15.28:0/3319040864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f063c19ddb0 con 0x7f063c1021a0 2024-04-21T08:58:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.751+0000 7f0640949640 1 -- 172.21.15.28:0/3319040864 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f062c009c30 con 0x7f063c1021a0 2024-04-21T08:58:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.751+0000 7f0640949640 1 -- 172.21.15.28:0/3319040864 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062c00f620 con 0x7f063c1021a0 2024-04-21T08:58:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.751+0000 7f064194b640 1 -- 172.21.15.28:0/3319040864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f063c1a55f0 con 0x7f063c1021a0 2024-04-21T08:58:07.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.752+0000 7f064194b640 1 -- 172.21.15.28:0/3319040864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f063c069230 con 0x7f063c1021a0 2024-04-21T08:58:07.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.753+0000 7f0640949640 1 -- 172.21.15.28:0/3319040864 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f062c010460 con 0x7f063c1021a0 2024-04-21T08:58:07.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.754+0000 7f0640949640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f06280a3d20 0x7f06280a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.754+0000 7f0633fff640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f06280a3d20 0x7f06280a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.754+0000 7f0640949640 1 -- 172.21.15.28:0/3319040864 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7f062c0d0920 con 0x7f063c1021a0 2024-04-21T08:58:07.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.755+0000 7f0633fff640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f06280a3d20 0x7f06280a61e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0624036040 tx=0x7f0624000bc0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:07.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.759+0000 7f0640949640 1 -- 172.21.15.28:0/3319040864 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f062c09cb70 con 0x7f063c1021a0 2024-04-21T08:58:07.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.986+0000 7f064194b640 1 -- 172.21.15.28:0/3319040864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f063c10a500 con 0x7f063c1021a0 2024-04-21T08:58:07.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.987+0000 7f0640949640 1 -- 172.21.15.28:0/3319040864 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7f062c009de0 con 0x7f063c1021a0 2024-04-21T08:58:07.987 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:58:07.987 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:58:07.988 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T08:58:07.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.989+0000 7f064194b640 1 -- 172.21.15.28:0/3319040864 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f06280a3d20 msgr2=0x7f06280a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:07.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.989+0000 7f064194b640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f06280a3d20 0x7f06280a61e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0624036040 tx=0x7f0624000bc0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.989+0000 7f064194b640 1 -- 172.21.15.28:0/3319040864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f063c1021a0 msgr2=0x7f063c198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:07.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.989+0000 7f064194b640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f063c1021a0 0x7f063c198710 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f062c00d950 tx=0x7f062c00de10 comp rx=0 tx=0).stop 2024-04-21T08:58:07.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.989+0000 7f064194b640 1 -- 172.21.15.28:0/3319040864 shutdown_connections 2024-04-21T08:58:07.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.989+0000 7f064194b640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f06280a3d20 0x7f06280a61e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.989+0000 7f064194b640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f063c19cfd0 0x7f063c19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.989+0000 7f064194b640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f063c107270 0x7f063c198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.989+0000 7f064194b640 1 --2- 172.21.15.28:0/3319040864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f063c1021a0 0x7f063c198710 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:07.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.989+0000 7f064194b640 1 -- 172.21.15.28:0/3319040864 >> 172.21.15.28:0/3319040864 conn(0x7f063c0793a0 msgr2=0x7f063c0fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:07.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.989+0000 7f064194b640 1 -- 172.21.15.28:0/3319040864 shutdown_connections 2024-04-21T08:58:07.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:07.989+0000 7f064194b640 1 -- 172.21.15.28:0/3319040864 wait complete. 2024-04-21T08:58:08.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:58:08.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.105+0000 7fecf9f23640 1 -- 172.21.15.28:0/3416478511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf406b1f0 msgr2=0x7fecf4106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.105+0000 7fecf9f23640 1 --2- 172.21.15.28:0/3416478511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf406b1f0 0x7fecf4106b90 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fece8009900 tx=0x7fece802bc30 comp rx=0 tx=0).stop 2024-04-21T08:58:08.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.105+0000 7fecf9f23640 1 -- 172.21.15.28:0/3416478511 shutdown_connections 2024-04-21T08:58:08.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.105+0000 7fecf9f23640 1 --2- 172.21.15.28:0/3416478511 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fecf4107270 0x7fecf410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.105+0000 7fecf9f23640 1 --2- 172.21.15.28:0/3416478511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf406b1f0 0x7fecf4106b90 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.105+0000 7fecf9f23640 1 --2- 172.21.15.28:0/3416478511 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fecf406a840 0x7fecf406ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.105+0000 7fecf9f23640 1 -- 172.21.15.28:0/3416478511 >> 172.21.15.28:0/3416478511 conn(0x7fecf4077eb0 msgr2=0x7fecf40782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.105+0000 7fecf9f23640 1 -- 172.21.15.28:0/3416478511 shutdown_connections 2024-04-21T08:58:08.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.105+0000 7fecf9f23640 1 -- 172.21.15.28:0/3416478511 wait complete. 2024-04-21T08:58:08.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.105+0000 7fecf9f23640 1 Processor -- start 2024-04-21T08:58:08.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.105+0000 7fecf9f23640 1 -- start start 2024-04-21T08:58:08.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf9f23640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fecf406a840 0x7fecf4198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf9f23640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fecf4107270 0x7fecf4198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf9f23640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf419d010 0x7fecf419d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf9f23640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fecf41121b0 con 0x7fecf419d010 2024-04-21T08:58:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf9f23640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fecf4112030 con 0x7fecf4107270 2024-04-21T08:58:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf9f23640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fecf4112330 con 0x7fecf406a840 2024-04-21T08:58:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf419d010 0x7fecf419d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf419d010 0x7fecf419d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51752/0 (socket says 172.21.15.28:51752) 2024-04-21T08:58:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf3fff640 1 -- 172.21.15.28:0/68117744 learned_addr learned my addr 172.21.15.28:0/68117744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf2ffd640 1 --2- 172.21.15.28:0/68117744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fecf4107270 0x7fecf4198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf37fe640 1 --2- 172.21.15.28:0/68117744 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fecf406a840 0x7fecf4198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf3fff640 1 -- 172.21.15.28:0/68117744 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fecf406a840 msgr2=0x7fecf4198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf3fff640 1 --2- 172.21.15.28:0/68117744 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fecf406a840 0x7fecf4198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf3fff640 1 -- 172.21.15.28:0/68117744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fecf4107270 msgr2=0x7fecf4198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf3fff640 1 --2- 172.21.15.28:0/68117744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fecf4107270 0x7fecf4198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf3fff640 1 -- 172.21.15.28:0/68117744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf419db00 con 0x7fecf419d010 2024-04-21T08:58:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf3fff640 1 --2- 172.21.15.28:0/68117744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf419d010 0x7fecf419d4c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fecec00cad0 tx=0x7fecec00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf0ff9640 1 -- 172.21.15.28:0/68117744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec00ae20 con 0x7fecf419d010 2024-04-21T08:58:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf9f23640 1 -- 172.21.15.28:0/68117744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecf419ddf0 con 0x7fecf419d010 2024-04-21T08:58:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.106+0000 7fecf0ff9640 1 -- 172.21.15.28:0/68117744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecec005020 con 0x7fecf419d010 2024-04-21T08:58:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.107+0000 7fecf0ff9640 1 -- 172.21.15.28:0/68117744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec00f6b0 con 0x7fecf419d010 2024-04-21T08:58:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.107+0000 7fecf9f23640 1 -- 172.21.15.28:0/68117744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecf41a5600 con 0x7fecf419d010 2024-04-21T08:58:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.109+0000 7fecfcbec740 1 Processor -- start 2024-04-21T08:58:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.109+0000 7fecf0ff9640 1 -- 172.21.15.28:0/68117744 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fecec0051d0 con 0x7fecf419d010 2024-04-21T08:58:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.109+0000 7fecfcbec740 1 -- start start 2024-04-21T08:58:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.109+0000 7fecfcbec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12a54d0 0x5632e12a58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.109+0000 7fecfcbec740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12a5e80 0x5632e12ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.109+0000 7fecfcbec740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12aea00 0x5632e12b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.109+0000 7fecfcbec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5632e120a1f0 con 0x5632e12a54d0 2024-04-21T08:58:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.109+0000 7fecfcbec740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5632e11a74b0 con 0x5632e12a5e80 2024-04-21T08:58:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.109+0000 7fecfcbec740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5632e12061b0 con 0x5632e12aea00 2024-04-21T08:58:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.109+0000 7fecf2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12a54d0 0x5632e12a58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.109+0000 7fecf2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12a54d0 0x5632e12a58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51764/0 (socket says 172.21.15.28:51764) 2024-04-21T08:58:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.109+0000 7fecf2ffd640 1 -- 172.21.15.28:0/2872857861 learned_addr learned my addr 172.21.15.28:0/2872857861 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.110+0000 7fecf0ff9640 1 --2- 172.21.15.28:0/68117744 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fecdc0a3d20 0x7fecdc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.110+0000 7fecf2ffd640 1 -- 172.21.15.28:0/2872857861 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12aea00 msgr2=0x5632e12b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.110+0000 7fecf2ffd640 1 --2- 172.21.15.28:0/2872857861 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12aea00 0x5632e12b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.110+0000 7fecf2ffd640 1 -- 172.21.15.28:0/2872857861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12a5e80 msgr2=0x5632e12ae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.110+0000 7fecf2ffd640 1 --2- 172.21.15.28:0/2872857861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12a5e80 0x5632e12ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.110+0000 7fecf2ffd640 1 -- 172.21.15.28:0/2872857861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632e11991b0 con 0x5632e12a54d0 2024-04-21T08:58:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.110+0000 7fecf3fff640 1 --2- 172.21.15.28:0/68117744 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fecdc0a3d20 0x7fecdc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.110+0000 7fecf0ff9640 1 -- 172.21.15.28:0/68117744 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7fecec0d0c20 con 0x7fecf419d010 2024-04-21T08:58:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.110+0000 7fecf2ffd640 1 --2- 172.21.15.28:0/2872857861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12a54d0 0x5632e12a58b0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fecf406a420 tx=0x7fece80041c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=808c5accfb0b3403 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.110+0000 7fecd9ffb640 1 -- 172.21.15.28:0/2872857861 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece8045020 con 0x5632e12a54d0 2024-04-21T08:58:08.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.110+0000 7fecd9ffb640 1 -- 172.21.15.28:0/2872857861 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fece802cc90 con 0x5632e12a54d0 2024-04-21T08:58:08.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.110+0000 7fecd9ffb640 1 -- 172.21.15.28:0/2872857861 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece8004920 con 0x5632e12a54d0 2024-04-21T08:58:08.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.111+0000 7fecfcbec740 1 -- 172.21.15.28:0/2872857861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12a54d0 msgr2=0x5632e12a58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.111+0000 7fecf3fff640 1 --2- 172.21.15.28:0/68117744 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fecdc0a3d20 0x7fecdc0a61e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fecec0ce520 tx=0x7fecec087040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.111+0000 7fecfcbec740 1 --2- 172.21.15.28:0/2872857861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12a54d0 0x5632e12a58b0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fecf406a420 tx=0x7fece80041c0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.111+0000 7fecfcbec740 1 -- 172.21.15.28:0/2872857861 shutdown_connections 2024-04-21T08:58:08.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.111+0000 7fecfcbec740 1 --2- 172.21.15.28:0/2872857861 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12aea00 0x5632e12b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.111+0000 7fecfcbec740 1 --2- 172.21.15.28:0/2872857861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12a5e80 0x5632e12ae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.111+0000 7fecfcbec740 1 --2- 172.21.15.28:0/2872857861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12a54d0 0x5632e12a58b0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.111+0000 7fecfcbec740 1 -- 172.21.15.28:0/2872857861 >> 172.21.15.28:0/2872857861 conn(0x5632e129fba0 msgr2=0x5632e129ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.111+0000 7fecfcbec740 1 -- 172.21.15.28:0/2872857861 shutdown_connections 2024-04-21T08:58:08.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.111+0000 7fecfcbec740 1 -- 172.21.15.28:0/2872857861 wait complete. 2024-04-21T08:58:08.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.111+0000 7fecfcbec740 1 -- start start 2024-04-21T08:58:08.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.111+0000 7fecfcbec740 1 Processor -- start 2024-04-21T08:58:08.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecfcbec740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12a54d0 0x5632e12ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecfcbec740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12a5e80 0x5632e12ab660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecfcbec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12aea00 0x5632e12abbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecfcbec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5632e12b1910 con 0x5632e12aea00 2024-04-21T08:58:08.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecfcbec740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5632e11c3500 con 0x5632e12a54d0 2024-04-21T08:58:08.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecfcbec740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5632e12b1a90 con 0x5632e12a5e80 2024-04-21T08:58:08.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecfcbec740 10 client.0 _ll_register_callbacks cb 0x5632e1191b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:08.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecf2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12aea00 0x5632e12abbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecf2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12aea00 0x5632e12abbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51766/0 (socket says 172.21.15.28:51766) 2024-04-21T08:58:08.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecf2ffd640 1 -- 172.21.15.28:0/269263871 learned_addr learned my addr 172.21.15.28:0/269263871 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:08.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecf2ffd640 1 --2- 172.21.15.28:0/269263871 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12a54d0 0x5632e12ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecf37fe640 1 --2- 172.21.15.28:0/269263871 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12a5e80 0x5632e12ab660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecf2ffd640 1 -- 172.21.15.28:0/269263871 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12a5e80 msgr2=0x5632e12ab660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecf2ffd640 1 --2- 172.21.15.28:0/269263871 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12a5e80 0x5632e12ab660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecf2ffd640 1 -- 172.21.15.28:0/269263871 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12a54d0 msgr2=0x5632e12ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecf2ffd640 1 --2- 172.21.15.28:0/269263871 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12a54d0 0x5632e12ae2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecf2ffd640 1 -- 172.21.15.28:0/269263871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632e0f1c3f0 con 0x5632e12aea00 2024-04-21T08:58:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecf2ffd640 1 --2- 172.21.15.28:0/269263871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12aea00 0x5632e12abbc0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fece8036040 tx=0x7fece802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d42a7b5911699cd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecd2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecd2ffd640 1 -- 172.21.15.28:0/269263871 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece804f020 con 0x5632e12aea00 2024-04-21T08:58:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecd2ffd640 1 -- 172.21.15.28:0/269263871 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7fece8004c80 con 0x5632e12aea00 2024-04-21T08:58:08.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.112+0000 7fecfcbec740 1 -- 172.21.15.28:0/269263871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fece802cc90 con 0x5632e12aea00 2024-04-21T08:58:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.113+0000 7fecd2ffd640 1 -- 172.21.15.28:0/269263871 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fece805d050 con 0x5632e12aea00 2024-04-21T08:58:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.113+0000 7fecfcbec740 10 client.17475 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.113+0000 7fecfcbec740 1 -- 172.21.15.28:0/269263871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5632e0fe67d0 con 0x5632e12aea00 2024-04-21T08:58:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.113+0000 7fecd2ffd640 1 -- 172.21.15.28:0/269263871 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fece802cc90 con 0x5632e12aea00 2024-04-21T08:58:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.113+0000 7fecd2ffd640 1 -- 172.21.15.28:0/269263871 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fece8041480 con 0x5632e12aea00 2024-04-21T08:58:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.113+0000 7fecfcbec740 10 client.17475 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.113+0000 7fecfcbec740 10 client.17475 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:58:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.113+0000 7fecfcbec740 20 client.17475 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:08.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.113+0000 7fecfcbec740 1 --2- 172.21.15.28:0/269263871 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5632e12bea00 0x5632e12c0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.113+0000 7fecfcbec740 4 client.17475 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:58:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.113+0000 7fecfcbec740 1 -- 172.21.15.28:0/269263871 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632e11a9930 con 0x5632e12bea00 2024-04-21T08:58:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.113+0000 7fecf37fe640 1 --2- 172.21.15.28:0/269263871 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5632e12bea00 0x5632e12c0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.114+0000 7fecd2ffd640 10 client.17475 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:58:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.114+0000 7fecf37fe640 1 --2- 172.21.15.28:0/269263871 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5632e12bea00 0x5632e12c0df0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=74ccc5de8c73cdde server_cookie=7efa36963e44957f in_seq=0 out_seq=0 2024-04-21T08:58:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.114+0000 7fecd2ffd640 1 -- 172.21.15.28:0/269263871 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5632e11a9930 con 0x5632e12bea00 2024-04-21T08:58:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.114+0000 7fecd2ffd640 10 client.17475 handle_command_reply: tid=0 2024-04-21T08:58:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.114+0000 7fecfcbec740 1 client.17475 shutdown 2024-04-21T08:58:08.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.115+0000 7fecfcbec740 1 -- 172.21.15.28:0/269263871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12aea00 msgr2=0x5632e12abbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.115+0000 7fecfcbec740 1 --2- 172.21.15.28:0/269263871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12aea00 0x5632e12abbc0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fece8036040 tx=0x7fece802bc30 comp rx=0 tx=0).stop 2024-04-21T08:58:08.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.115+0000 7fecfcbec740 1 -- 172.21.15.28:0/269263871 shutdown_connections 2024-04-21T08:58:08.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.115+0000 7fecfcbec740 1 --2- 172.21.15.28:0/269263871 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5632e12bea00 0x5632e12c0df0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.115+0000 7fecfcbec740 1 --2- 172.21.15.28:0/269263871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12aea00 0x5632e12abbc0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.115+0000 7fecfcbec740 1 --2- 172.21.15.28:0/269263871 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12a5e80 0x5632e12ab660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.115+0000 7fecd2ffd640 0 client.17475 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:58:08.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.115+0000 7fecfcbec740 1 --2- 172.21.15.28:0/269263871 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12a54d0 0x5632e12ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.115+0000 7fecfcbec740 1 -- 172.21.15.28:0/269263871 >> 172.21.15.28:0/269263871 conn(0x5632e129fba0 msgr2=0x5632e12a18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.115+0000 7fecfcbec740 1 -- 172.21.15.28:0/269263871 shutdown_connections 2024-04-21T08:58:08.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.115+0000 7fecfcbec740 1 -- 172.21.15.28:0/269263871 wait complete. 2024-04-21T08:58:08.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.115+0000 7fecfcbec740 20 client.17475 trim_cache size 0 max 16384 2024-04-21T08:58:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.128+0000 7fecfcbec740 1 Processor -- start 2024-04-21T08:58:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.128+0000 7fecfcbec740 1 -- start start 2024-04-21T08:58:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.128+0000 7fecfcbec740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12a54d0 0x5632e12c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.128+0000 7fecfcbec740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12a5e80 0x5632e12add90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.128+0000 7fecfcbec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12aea00 0x5632e12ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.128+0000 7fecfcbec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fecf41b1da0 con 0x5632e12aea00 2024-04-21T08:58:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.128+0000 7fecfcbec740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fecf41af670 con 0x5632e12a54d0 2024-04-21T08:58:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.128+0000 7fecfcbec740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fecf41ad4e0 con 0x5632e12a5e80 2024-04-21T08:58:08.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.128+0000 7fecf2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12aea00 0x5632e12ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.129+0000 7fecf2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12aea00 0x5632e12ae2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51770/0 (socket says 172.21.15.28:51770) 2024-04-21T08:58:08.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.129+0000 7fecf2ffd640 1 -- 172.21.15.28:0/3202383927 learned_addr learned my addr 172.21.15.28:0/3202383927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:08.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.129+0000 7fecf37fe640 1 --2- 172.21.15.28:0/3202383927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12a5e80 0x5632e12add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.129+0000 7fecf2ffd640 1 --2- 172.21.15.28:0/3202383927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12a54d0 0x5632e12c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.129+0000 7fecf2ffd640 1 -- 172.21.15.28:0/3202383927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12a5e80 msgr2=0x5632e12add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.129+0000 7fecf2ffd640 1 --2- 172.21.15.28:0/3202383927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12a5e80 0x5632e12add90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.129+0000 7fecf2ffd640 1 -- 172.21.15.28:0/3202383927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12a54d0 msgr2=0x5632e12c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.129+0000 7fecf2ffd640 1 --2- 172.21.15.28:0/3202383927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12a54d0 0x5632e12c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.129+0000 7fecf2ffd640 1 -- 172.21.15.28:0/3202383927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632e12a5c20 con 0x5632e12aea00 2024-04-21T08:58:08.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.129+0000 7fecf2ffd640 1 --2- 172.21.15.28:0/3202383927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12aea00 0x5632e12ae2d0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fece8041850 tx=0x7fece8001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3bef844ae5d074 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.129+0000 7fecd97fa640 1 -- 172.21.15.28:0/3202383927 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece804f040 con 0x5632e12aea00 2024-04-21T08:58:08.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.129+0000 7fecd97fa640 1 -- 172.21.15.28:0/3202383927 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fece80041a0 con 0x5632e12aea00 2024-04-21T08:58:08.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.129+0000 7fecd97fa640 1 -- 172.21.15.28:0/3202383927 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece8045070 con 0x5632e12aea00 2024-04-21T08:58:08.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.130+0000 7fecfcbec740 1 -- 172.21.15.28:0/3202383927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12aea00 msgr2=0x5632e12ae2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.130+0000 7fecfcbec740 1 --2- 172.21.15.28:0/3202383927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12aea00 0x5632e12ae2d0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fece8041850 tx=0x7fece8001e50 comp rx=0 tx=0).stop 2024-04-21T08:58:08.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.130+0000 7fecfcbec740 1 -- 172.21.15.28:0/3202383927 shutdown_connections 2024-04-21T08:58:08.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.130+0000 7fecfcbec740 1 --2- 172.21.15.28:0/3202383927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12aea00 0x5632e12ae2d0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.130+0000 7fecfcbec740 1 --2- 172.21.15.28:0/3202383927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12a5e80 0x5632e12add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.130+0000 7fecfcbec740 1 --2- 172.21.15.28:0/3202383927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12a54d0 0x5632e12c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.130+0000 7fecfcbec740 1 -- 172.21.15.28:0/3202383927 >> 172.21.15.28:0/3202383927 conn(0x5632e129fba0 msgr2=0x5632e129ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.130+0000 7fecfcbec740 1 -- 172.21.15.28:0/3202383927 shutdown_connections 2024-04-21T08:58:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.130+0000 7fecfcbec740 1 -- 172.21.15.28:0/3202383927 wait complete. 2024-04-21T08:58:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.130+0000 7fecfcbec740 1 -- start start 2024-04-21T08:58:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.130+0000 7fecfcbec740 1 Processor -- start 2024-04-21T08:58:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.130+0000 7fecfcbec740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12a54d0 0x5632e1321040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.130+0000 7fecfcbec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12a5e80 0x5632e1321580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.130+0000 7fecfcbec740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12aea00 0x5632e131b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.130+0000 7fecfcbec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fecf41adb30 con 0x5632e12a5e80 2024-04-21T08:58:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecfcbec740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fecf41aefa0 con 0x5632e12aea00 2024-04-21T08:58:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecfcbec740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fecf41ae200 con 0x5632e12a54d0 2024-04-21T08:58:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecfcbec740 10 client.0 _ll_register_callbacks cb 0x7fece80049d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecf37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12a5e80 0x5632e1321580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecf37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12a5e80 0x5632e1321580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51782/0 (socket says 172.21.15.28:51782) 2024-04-21T08:58:08.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecf37fe640 1 -- 172.21.15.28:0/2558840865 learned_addr learned my addr 172.21.15.28:0/2558840865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:08.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecf2ffd640 1 --2- 172.21.15.28:0/2558840865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12a54d0 0x5632e1321040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecf37fe640 1 -- 172.21.15.28:0/2558840865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12a54d0 msgr2=0x5632e1321040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecf37fe640 1 --2- 172.21.15.28:0/2558840865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12a54d0 0x5632e1321040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecf37fe640 1 -- 172.21.15.28:0/2558840865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12aea00 msgr2=0x5632e131b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecf37fe640 1 --2- 172.21.15.28:0/2558840865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12aea00 0x5632e131b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecf37fe640 1 -- 172.21.15.28:0/2558840865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632e131bb40 con 0x5632e12a5e80 2024-04-21T08:58:08.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecf37fe640 1 --2- 172.21.15.28:0/2558840865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12a5e80 0x5632e1321580 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fece400cb10 tx=0x7fece4005410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae3417c3033a673f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecd2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecd2ffd640 1 -- 172.21.15.28:0/2558840865 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4004220 con 0x5632e12a5e80 2024-04-21T08:58:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecd2ffd640 1 -- 172.21.15.28:0/2558840865 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7fece40043c0 con 0x5632e12a5e80 2024-04-21T08:58:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecfcbec740 1 -- 172.21.15.28:0/2558840865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fece80041a0 con 0x5632e12a5e80 2024-04-21T08:58:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecd2ffd640 1 -- 172.21.15.28:0/2558840865 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fece4005600 con 0x5632e12a5e80 2024-04-21T08:58:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecfcbec740 10 client.17481 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.131+0000 7fecfcbec740 1 -- 172.21.15.28:0/2558840865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5632e1328810 con 0x5632e12a5e80 2024-04-21T08:58:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.132+0000 7fecd2ffd640 1 -- 172.21.15.28:0/2558840865 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fece80041a0 con 0x5632e12a5e80 2024-04-21T08:58:08.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.132+0000 7fecd2ffd640 1 -- 172.21.15.28:0/2558840865 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fece4020070 con 0x5632e12a5e80 2024-04-21T08:58:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.132+0000 7fecfcbec740 10 client.17481 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.132+0000 7fecfcbec740 10 client.17481 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:58:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.132+0000 7fecfcbec740 20 client.17481 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.132+0000 7fecfcbec740 1 --2- 172.21.15.28:0/2558840865 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5632e1328fc0 0x5632e132b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.132+0000 7fecfcbec740 4 client.17481 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:58:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.132+0000 7fecfcbec740 1 -- 172.21.15.28:0/2558840865 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5632e1217c10 con 0x5632e1328fc0 2024-04-21T08:58:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.132+0000 7fecf2ffd640 1 --2- 172.21.15.28:0/2558840865 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5632e1328fc0 0x5632e132b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.132+0000 7fecf2ffd640 1 --2- 172.21.15.28:0/2558840865 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5632e1328fc0 0x5632e132b3e0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=715b872b9aa21198 server_cookie=73065783fe9f9663 in_seq=0 out_seq=0 2024-04-21T08:58:08.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.132+0000 7fecd2ffd640 10 client.17481 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:58:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.133+0000 7fecd2ffd640 1 -- 172.21.15.28:0/2558840865 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5632e1217c10 con 0x5632e1328fc0 2024-04-21T08:58:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.133+0000 7fecd2ffd640 10 client.17481 handle_command_reply: tid=0 2024-04-21T08:58:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.133+0000 7fecfcbec740 1 client.17481 shutdown 2024-04-21T08:58:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.133+0000 7fecfcbec740 1 -- 172.21.15.28:0/2558840865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12a5e80 msgr2=0x5632e1321580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.133+0000 7fecfcbec740 1 --2- 172.21.15.28:0/2558840865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12a5e80 0x5632e1321580 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fece400cb10 tx=0x7fece4005410 comp rx=0 tx=0).stop 2024-04-21T08:58:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.133+0000 7fecfcbec740 1 -- 172.21.15.28:0/2558840865 shutdown_connections 2024-04-21T08:58:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.133+0000 7fecfcbec740 1 --2- 172.21.15.28:0/2558840865 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5632e1328fc0 0x5632e132b3e0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.133+0000 7fecfcbec740 1 --2- 172.21.15.28:0/2558840865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5632e12aea00 0x5632e131b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.133+0000 7fecfcbec740 1 --2- 172.21.15.28:0/2558840865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632e12a5e80 0x5632e1321580 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.133+0000 7fecd2ffd640 0 client.17481 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:58:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.133+0000 7fecfcbec740 1 --2- 172.21.15.28:0/2558840865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5632e12a54d0 0x5632e1321040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.133+0000 7fecfcbec740 1 -- 172.21.15.28:0/2558840865 >> 172.21.15.28:0/2558840865 conn(0x5632e129fba0 msgr2=0x5632e12b9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.133+0000 7fecfcbec740 1 -- 172.21.15.28:0/2558840865 shutdown_connections 2024-04-21T08:58:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.133+0000 7fecfcbec740 1 -- 172.21.15.28:0/2558840865 wait complete. 2024-04-21T08:58:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.133+0000 7fecfcbec740 20 client.17481 trim_cache size 0 max 16384 2024-04-21T08:58:08.143 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:58:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.134+0000 7fecf9f23640 1 -- 172.21.15.28:0/68117744 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fecdc0a3d20 msgr2=0x7fecdc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.134+0000 7fecf9f23640 1 --2- 172.21.15.28:0/68117744 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fecdc0a3d20 0x7fecdc0a61e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fecec0ce520 tx=0x7fecec087040 comp rx=0 tx=0).stop 2024-04-21T08:58:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.134+0000 7fecf9f23640 1 -- 172.21.15.28:0/68117744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf419d010 msgr2=0x7fecf419d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.134+0000 7fecf9f23640 1 --2- 172.21.15.28:0/68117744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf419d010 0x7fecf419d4c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fecec00cad0 tx=0x7fecec00cf90 comp rx=0 tx=0).stop 2024-04-21T08:58:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.134+0000 7fecf9f23640 1 -- 172.21.15.28:0/68117744 shutdown_connections 2024-04-21T08:58:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.134+0000 7fecf9f23640 1 --2- 172.21.15.28:0/68117744 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fecdc0a3d20 0x7fecdc0a61e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.134+0000 7fecf9f23640 1 --2- 172.21.15.28:0/68117744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf419d010 0x7fecf419d4c0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.134+0000 7fecf9f23640 1 --2- 172.21.15.28:0/68117744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fecf4107270 0x7fecf4198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.134+0000 7fecf9f23640 1 --2- 172.21.15.28:0/68117744 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fecf406a840 0x7fecf4198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.134+0000 7fecf9f23640 1 -- 172.21.15.28:0/68117744 >> 172.21.15.28:0/68117744 conn(0x7fecf4077eb0 msgr2=0x7fecf4108440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.134+0000 7fecf9f23640 1 -- 172.21.15.28:0/68117744 shutdown_connections 2024-04-21T08:58:08.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.134+0000 7fecf9f23640 1 -- 172.21.15.28:0/68117744 wait complete. 2024-04-21T08:58:08.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:58:08.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.251+0000 7fab3f2f2640 1 -- 172.21.15.28:0/3628433814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab38101aa0 msgr2=0x7fab38105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.251+0000 7fab3f2f2640 1 --2- 172.21.15.28:0/3628433814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab38101aa0 0x7fab38105a90 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fab28009900 tx=0x7fab2802bc30 comp rx=0 tx=0).stop 2024-04-21T08:58:08.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.251+0000 7fab3f2f2640 1 -- 172.21.15.28:0/3628433814 shutdown_connections 2024-04-21T08:58:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.251+0000 7fab3f2f2640 1 --2- 172.21.15.28:0/3628433814 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fab38106170 0x7fab3810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.251+0000 7fab3f2f2640 1 --2- 172.21.15.28:0/3628433814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab38101aa0 0x7fab38105a90 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.251+0000 7fab3f2f2640 1 --2- 172.21.15.28:0/3628433814 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab381010f0 0x7fab381014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.251+0000 7fab3f2f2640 1 -- 172.21.15.28:0/3628433814 >> 172.21.15.28:0/3628433814 conn(0x7fab38077fb0 msgr2=0x7fab380783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.251+0000 7fab3f2f2640 1 -- 172.21.15.28:0/3628433814 shutdown_connections 2024-04-21T08:58:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.251+0000 7fab3f2f2640 1 -- 172.21.15.28:0/3628433814 wait complete. 2024-04-21T08:58:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3f2f2640 1 Processor -- start 2024-04-21T08:58:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3f2f2640 1 -- start start 2024-04-21T08:58:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3f2f2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fab381010f0 0x7fab38197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3f2f2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab38106170 0x7fab38197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3f2f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab38197ea0 0x7fab381a6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3f2f2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab38111110 con 0x7fab38197ea0 2024-04-21T08:58:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3f2f2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fab38110f90 con 0x7fab38106170 2024-04-21T08:58:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3f2f2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fab38111290 con 0x7fab381010f0 2024-04-21T08:58:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3d88f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab38197ea0 0x7fab381a6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3d88f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab38197ea0 0x7fab381a6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51800/0 (socket says 172.21.15.28:51800) 2024-04-21T08:58:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3d88f640 1 -- 172.21.15.28:0/3636559252 learned_addr learned my addr 172.21.15.28:0/3636559252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3d08e640 1 --2- 172.21.15.28:0/3636559252 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fab381010f0 0x7fab38197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3c88d640 1 --2- 172.21.15.28:0/3636559252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab38106170 0x7fab38197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3d88f640 1 -- 172.21.15.28:0/3636559252 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fab381010f0 msgr2=0x7fab38197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3d88f640 1 --2- 172.21.15.28:0/3636559252 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fab381010f0 0x7fab38197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3d88f640 1 -- 172.21.15.28:0/3636559252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab38106170 msgr2=0x7fab38197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3d88f640 1 --2- 172.21.15.28:0/3636559252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab38106170 0x7fab38197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3d88f640 1 -- 172.21.15.28:0/3636559252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab381a7060 con 0x7fab38197ea0 2024-04-21T08:58:08.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.252+0000 7fab3c88d640 1 --2- 172.21.15.28:0/3636559252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab38106170 0x7fab38197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:08.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.253+0000 7fab3d88f640 1 --2- 172.21.15.28:0/3636559252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab38197ea0 0x7fab381a6aa0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fab3000ba10 tx=0x7fab3000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.253+0000 7fab2e7fc640 1 -- 172.21.15.28:0/3636559252 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab30019070 con 0x7fab38197ea0 2024-04-21T08:58:08.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.253+0000 7fab2e7fc640 1 -- 172.21.15.28:0/3636559252 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab30014440 con 0x7fab38197ea0 2024-04-21T08:58:08.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.253+0000 7fab3f2f2640 1 -- 172.21.15.28:0/3636559252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab381a7350 con 0x7fab38197ea0 2024-04-21T08:58:08.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.253+0000 7fab2e7fc640 1 -- 172.21.15.28:0/3636559252 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab300075d0 con 0x7fab38197ea0 2024-04-21T08:58:08.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.253+0000 7fab3f2f2640 1 -- 172.21.15.28:0/3636559252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab381a7850 con 0x7fab38197ea0 2024-04-21T08:58:08.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.256+0000 7fab41fc0740 1 Processor -- start 2024-04-21T08:58:08.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.256+0000 7fab2e7fc640 1 -- 172.21.15.28:0/3636559252 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fab3001c070 con 0x7fab38197ea0 2024-04-21T08:58:08.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.256+0000 7fab41fc0740 1 -- start start 2024-04-21T08:58:08.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.256+0000 7fab41fc0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf24d0 0x55e75eaf28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.256+0000 7fab41fc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eaf2e80 0x55e75eafb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.256+0000 7fab41fc0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eafba00 0x55e75eafde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.256+0000 7fab41fc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e75ea571f0 con 0x55e75eaf2e80 2024-04-21T08:58:08.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.256+0000 7fab41fc0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e75e9f44b0 con 0x55e75eaf24d0 2024-04-21T08:58:08.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.256+0000 7fab41fc0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55e75ea531b0 con 0x55e75eafba00 2024-04-21T08:58:08.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.256+0000 7fab3d08e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eaf2e80 0x55e75eafb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.256+0000 7fab3d08e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eaf2e80 0x55e75eafb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51808/0 (socket says 172.21.15.28:51808) 2024-04-21T08:58:08.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.256+0000 7fab3d08e640 1 -- 172.21.15.28:0/3441272715 learned_addr learned my addr 172.21.15.28:0/3441272715 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:08.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab2e7fc640 1 --2- 172.21.15.28:0/3636559252 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fab240a3d20 0x7fab240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab3c88d640 1 --2- 172.21.15.28:0/3441272715 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eafba00 0x55e75eafde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab3d08e640 1 -- 172.21.15.28:0/3441272715 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eafba00 msgr2=0x55e75eafde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab3d08e640 1 --2- 172.21.15.28:0/3441272715 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eafba00 0x55e75eafde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab3c88d640 1 --2- 172.21.15.28:0/3441272715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf24d0 0x55e75eaf28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab3d08e640 1 -- 172.21.15.28:0/3441272715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf24d0 msgr2=0x55e75eaf28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab3d08e640 1 --2- 172.21.15.28:0/3441272715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf24d0 0x55e75eaf28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab3d08e640 1 -- 172.21.15.28:0/3441272715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e75e9e61b0 con 0x55e75eaf2e80 2024-04-21T08:58:08.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab3d88f640 1 --2- 172.21.15.28:0/3636559252 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fab240a3d20 0x7fab240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab3c88d640 1 --2- 172.21.15.28:0/3441272715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf24d0 0x55e75eaf28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:08.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab2e7fc640 1 -- 172.21.15.28:0/3636559252 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7fab300d0100 con 0x7fab38197ea0 2024-04-21T08:58:08.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab3d08e640 1 --2- 172.21.15.28:0/3441272715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eaf2e80 0x55e75eafb430 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fab3400dcb0 tx=0x7fab3400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3662cc7fc7355816 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab237fe640 1 -- 172.21.15.28:0/3441272715 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab34011440 con 0x55e75eaf2e80 2024-04-21T08:58:08.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab3d88f640 1 --2- 172.21.15.28:0/3636559252 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fab240a3d20 0x7fab240a61e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fab3000b9e0 tx=0x7fab300bc040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab237fe640 1 -- 172.21.15.28:0/3441272715 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab34011a90 con 0x55e75eaf2e80 2024-04-21T08:58:08.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.257+0000 7fab237fe640 1 -- 172.21.15.28:0/3441272715 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab3401ba10 con 0x55e75eaf2e80 2024-04-21T08:58:08.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.258+0000 7fab41fc0740 1 -- 172.21.15.28:0/3441272715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eaf2e80 msgr2=0x55e75eafb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.258+0000 7fab41fc0740 1 --2- 172.21.15.28:0/3441272715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eaf2e80 0x55e75eafb430 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fab3400dcb0 tx=0x7fab3400f730 comp rx=0 tx=0).stop 2024-04-21T08:58:08.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.258+0000 7fab41fc0740 1 -- 172.21.15.28:0/3441272715 shutdown_connections 2024-04-21T08:58:08.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.258+0000 7fab41fc0740 1 --2- 172.21.15.28:0/3441272715 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eafba00 0x55e75eafde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.258+0000 7fab41fc0740 1 --2- 172.21.15.28:0/3441272715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eaf2e80 0x55e75eafb430 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.258+0000 7fab41fc0740 1 --2- 172.21.15.28:0/3441272715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf24d0 0x55e75eaf28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.258+0000 7fab41fc0740 1 -- 172.21.15.28:0/3441272715 >> 172.21.15.28:0/3441272715 conn(0x55e75eaecba0 msgr2=0x55e75eaecf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.258+0000 7fab41fc0740 1 -- 172.21.15.28:0/3441272715 shutdown_connections 2024-04-21T08:58:08.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.258+0000 7fab41fc0740 1 -- 172.21.15.28:0/3441272715 wait complete. 2024-04-21T08:58:08.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.258+0000 7fab41fc0740 1 -- start start 2024-04-21T08:58:08.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.258+0000 7fab41fc0740 1 Processor -- start 2024-04-21T08:58:08.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab41fc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eaf24d0 0x55e75eafb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab41fc0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf2e80 0x55e75eaf8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab41fc0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eafba00 0x55e75eaf8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab41fc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e75eafe970 con 0x55e75eaf24d0 2024-04-21T08:58:08.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab41fc0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e75ea10500 con 0x55e75eafba00 2024-04-21T08:58:08.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab41fc0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55e75eafeaf0 con 0x55e75eaf2e80 2024-04-21T08:58:08.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab41fc0740 10 client.0 _ll_register_callbacks cb 0x55e75e9deb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:08.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab3c88d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eaf24d0 0x55e75eafb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab3c88d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eaf24d0 0x55e75eafb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51812/0 (socket says 172.21.15.28:51812) 2024-04-21T08:58:08.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab3c88d640 1 -- 172.21.15.28:0/2292097579 learned_addr learned my addr 172.21.15.28:0/2292097579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:08.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab3d08e640 1 --2- 172.21.15.28:0/2292097579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf2e80 0x55e75eaf8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab3c88d640 1 --2- 172.21.15.28:0/2292097579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eafba00 0x55e75eaf8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab3c88d640 1 -- 172.21.15.28:0/2292097579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf2e80 msgr2=0x55e75eaf8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab3c88d640 1 --2- 172.21.15.28:0/2292097579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf2e80 0x55e75eaf8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab3c88d640 1 -- 172.21.15.28:0/2292097579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eafba00 msgr2=0x55e75eaf8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab3c88d640 1 --2- 172.21.15.28:0/2292097579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eafba00 0x55e75eaf8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab3c88d640 1 -- 172.21.15.28:0/2292097579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e75e7693f0 con 0x55e75eaf24d0 2024-04-21T08:58:08.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab3d08e640 1 --2- 172.21.15.28:0/2292097579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf2e80 0x55e75eaf8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:08.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab3c88d640 1 --2- 172.21.15.28:0/2292097579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eaf24d0 0x55e75eafb2f0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fab28005ec0 tx=0x7fab2802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d1cf0201958c3d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.259+0000 7fab20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:08.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.260+0000 7fab20ff9640 1 -- 172.21.15.28:0/2292097579 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab28049060 con 0x55e75eaf24d0 2024-04-21T08:58:08.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.260+0000 7fab20ff9640 1 -- 172.21.15.28:0/2292097579 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7fab28005040 con 0x55e75eaf24d0 2024-04-21T08:58:08.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.260+0000 7fab41fc0740 1 -- 172.21.15.28:0/2292097579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab34011a90 con 0x55e75eaf24d0 2024-04-21T08:58:08.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.260+0000 7fab20ff9640 1 -- 172.21.15.28:0/2292097579 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fab28050050 con 0x55e75eaf24d0 2024-04-21T08:58:08.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.260+0000 7fab41fc0740 10 client.17493 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:08.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.260+0000 7fab41fc0740 1 -- 172.21.15.28:0/2292097579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e75e8337d0 con 0x55e75eaf24d0 2024-04-21T08:58:08.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.260+0000 7fab20ff9640 1 -- 172.21.15.28:0/2292097579 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab34011a90 con 0x55e75eaf24d0 2024-04-21T08:58:08.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.260+0000 7fab20ff9640 1 -- 172.21.15.28:0/2292097579 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fab280035f0 con 0x55e75eaf24d0 2024-04-21T08:58:08.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.260+0000 7fab41fc0740 10 client.17493 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:08.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.260+0000 7fab41fc0740 10 client.17493 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:58:08.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.260+0000 7fab41fc0740 20 client.17493 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:08.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.260+0000 7fab41fc0740 1 --2- 172.21.15.28:0/2292097579 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55e75eb0bb60 0x55e75eb0df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.260+0000 7fab41fc0740 4 client.17493 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:58:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.260+0000 7fab41fc0740 1 -- 172.21.15.28:0/2292097579 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e75e9f6930 con 0x55e75eb0bb60 2024-04-21T08:58:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.261+0000 7fab3d08e640 1 --2- 172.21.15.28:0/2292097579 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55e75eb0bb60 0x55e75eb0df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.261+0000 7fab3d08e640 1 --2- 172.21.15.28:0/2292097579 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55e75eb0bb60 0x55e75eb0df50 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b47529661e0fefc7 server_cookie=e43a11d7afee2b08 in_seq=0 out_seq=0 2024-04-21T08:58:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.261+0000 7fab20ff9640 10 client.17493 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:58:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.262+0000 7fab20ff9640 1 -- 172.21.15.28:0/2292097579 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e75e9f6930 con 0x55e75eb0bb60 2024-04-21T08:58:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.262+0000 7fab20ff9640 10 client.17493 handle_command_reply: tid=0 2024-04-21T08:58:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.262+0000 7fab41fc0740 1 client.17493 shutdown 2024-04-21T08:58:08.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.262+0000 7fab41fc0740 1 -- 172.21.15.28:0/2292097579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eaf24d0 msgr2=0x55e75eafb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.262+0000 7fab41fc0740 1 --2- 172.21.15.28:0/2292097579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eaf24d0 0x55e75eafb2f0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fab28005ec0 tx=0x7fab2802bc30 comp rx=0 tx=0).stop 2024-04-21T08:58:08.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.262+0000 7fab41fc0740 1 -- 172.21.15.28:0/2292097579 shutdown_connections 2024-04-21T08:58:08.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.262+0000 7fab41fc0740 1 --2- 172.21.15.28:0/2292097579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eafba00 0x55e75eaf8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.262+0000 7fab41fc0740 1 --2- 172.21.15.28:0/2292097579 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55e75eb0bb60 0x55e75eb0df50 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.262+0000 7fab41fc0740 1 --2- 172.21.15.28:0/2292097579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf2e80 0x55e75eaf8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.262+0000 7fab41fc0740 1 --2- 172.21.15.28:0/2292097579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eaf24d0 0x55e75eafb2f0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.262+0000 7fab20ff9640 0 client.17493 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:58:08.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.262+0000 7fab41fc0740 1 -- 172.21.15.28:0/2292097579 >> 172.21.15.28:0/2292097579 conn(0x55e75eaecba0 msgr2=0x55e75eaee8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.262+0000 7fab41fc0740 1 -- 172.21.15.28:0/2292097579 shutdown_connections 2024-04-21T08:58:08.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.262+0000 7fab41fc0740 1 -- 172.21.15.28:0/2292097579 wait complete. 2024-04-21T08:58:08.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.262+0000 7fab41fc0740 20 client.17493 trim_cache size 0 max 16384 2024-04-21T08:58:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab41fc0740 1 Processor -- start 2024-04-21T08:58:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab41fc0740 1 -- start start 2024-04-21T08:58:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab41fc0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf24d0 0x55e75eb07830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab41fc0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf2e80 0x55e75eafadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab41fc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eafba00 0x55e75eafb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab41fc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab381b4120 con 0x55e75eafba00 2024-04-21T08:58:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab41fc0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fab381b19f0 con 0x55e75eaf2e80 2024-04-21T08:58:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab41fc0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fab381af860 con 0x55e75eaf24d0 2024-04-21T08:58:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab3c88d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eafba00 0x55e75eafb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab3c88d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eafba00 0x55e75eafb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51816/0 (socket says 172.21.15.28:51816) 2024-04-21T08:58:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab3c88d640 1 -- 172.21.15.28:0/2515862966 learned_addr learned my addr 172.21.15.28:0/2515862966 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab3d08e640 1 --2- 172.21.15.28:0/2515862966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf2e80 0x55e75eafadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab3c88d640 1 --2- 172.21.15.28:0/2515862966 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf24d0 0x55e75eb07830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab3c88d640 1 -- 172.21.15.28:0/2515862966 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf24d0 msgr2=0x55e75eb07830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab3c88d640 1 --2- 172.21.15.28:0/2515862966 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf24d0 0x55e75eb07830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab3c88d640 1 -- 172.21.15.28:0/2515862966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf2e80 msgr2=0x55e75eafadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab3c88d640 1 --2- 172.21.15.28:0/2515862966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf2e80 0x55e75eafadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab3c88d640 1 -- 172.21.15.28:0/2515862966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e75eb0e2c0 con 0x55e75eafba00 2024-04-21T08:58:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab3c88d640 1 --2- 172.21.15.28:0/2515862966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eafba00 0x55e75eafb2f0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fab2802cdc0 tx=0x7fab28005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eea6d44e5b0537f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.276+0000 7fab22ffd640 1 -- 172.21.15.28:0/2515862966 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab28000e60 con 0x55e75eafba00 2024-04-21T08:58:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.277+0000 7fab22ffd640 1 -- 172.21.15.28:0/2515862966 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab280035f0 con 0x55e75eafba00 2024-04-21T08:58:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.277+0000 7fab22ffd640 1 -- 172.21.15.28:0/2515862966 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab28049040 con 0x55e75eafba00 2024-04-21T08:58:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.277+0000 7fab41fc0740 1 -- 172.21.15.28:0/2515862966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eafba00 msgr2=0x55e75eafb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.277+0000 7fab41fc0740 1 --2- 172.21.15.28:0/2515862966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eafba00 0x55e75eafb2f0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fab2802cdc0 tx=0x7fab28005fd0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.277+0000 7fab41fc0740 1 -- 172.21.15.28:0/2515862966 shutdown_connections 2024-04-21T08:58:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.277+0000 7fab41fc0740 1 --2- 172.21.15.28:0/2515862966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eafba00 0x55e75eafb2f0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.277+0000 7fab41fc0740 1 --2- 172.21.15.28:0/2515862966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf2e80 0x55e75eafadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.277+0000 7fab41fc0740 1 --2- 172.21.15.28:0/2515862966 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf24d0 0x55e75eb07830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.277+0000 7fab41fc0740 1 -- 172.21.15.28:0/2515862966 >> 172.21.15.28:0/2515862966 conn(0x55e75eaecba0 msgr2=0x55e75eaebc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.277+0000 7fab41fc0740 1 -- 172.21.15.28:0/2515862966 shutdown_connections 2024-04-21T08:58:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.277+0000 7fab41fc0740 1 -- 172.21.15.28:0/2515862966 wait complete. 2024-04-21T08:58:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.277+0000 7fab41fc0740 1 -- start start 2024-04-21T08:58:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab41fc0740 1 Processor -- start 2024-04-21T08:58:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab41fc0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf24d0 0x55e75eaf5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab41fc0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf2e80 0x55e75eaf5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab41fc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eafba00 0x55e75eb71050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab41fc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab381afeb0 con 0x55e75eafba00 2024-04-21T08:58:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab41fc0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fab381b1320 con 0x55e75eaf24d0 2024-04-21T08:58:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab41fc0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fab381b0580 con 0x55e75eaf2e80 2024-04-21T08:58:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab41fc0740 10 client.0 _ll_register_callbacks cb 0x7fab28004400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab3c88d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eafba00 0x55e75eb71050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab3c88d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eafba00 0x55e75eb71050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51820/0 (socket says 172.21.15.28:51820) 2024-04-21T08:58:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab3c88d640 1 -- 172.21.15.28:0/3102056063 learned_addr learned my addr 172.21.15.28:0/3102056063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab3d08e640 1 --2- 172.21.15.28:0/3102056063 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf2e80 0x55e75eaf5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab3c88d640 1 --2- 172.21.15.28:0/3102056063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf24d0 0x55e75eaf5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab3c88d640 1 -- 172.21.15.28:0/3102056063 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf2e80 msgr2=0x55e75eaf5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab3c88d640 1 --2- 172.21.15.28:0/3102056063 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf2e80 0x55e75eaf5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab3c88d640 1 -- 172.21.15.28:0/3102056063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf24d0 msgr2=0x55e75eaf5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab3c88d640 1 --2- 172.21.15.28:0/3102056063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf24d0 0x55e75eaf5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab3c88d640 1 -- 172.21.15.28:0/3102056063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e75eaf8f70 con 0x55e75eafba00 2024-04-21T08:58:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab3c88d640 1 --2- 172.21.15.28:0/3102056063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eafba00 0x55e75eb71050 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fab28005ab0 tx=0x7fab280044b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b3e827d3cdfef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.278+0000 7fab20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.279+0000 7fab20ff9640 1 -- 172.21.15.28:0/3102056063 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab28049040 con 0x55e75eafba00 2024-04-21T08:58:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.279+0000 7fab41fc0740 1 -- 172.21.15.28:0/3102056063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab280035f0 con 0x55e75eafba00 2024-04-21T08:58:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.279+0000 7fab20ff9640 1 -- 172.21.15.28:0/3102056063 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7fab280048d0 con 0x55e75eafba00 2024-04-21T08:58:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.279+0000 7fab20ff9640 1 -- 172.21.15.28:0/3102056063 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fab280041d0 con 0x55e75eafba00 2024-04-21T08:58:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.279+0000 7fab41fc0740 10 client.17499 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.279+0000 7fab41fc0740 1 -- 172.21.15.28:0/3102056063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e75eb72420 con 0x55e75eafba00 2024-04-21T08:58:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.279+0000 7fab20ff9640 1 -- 172.21.15.28:0/3102056063 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab280035f0 con 0x55e75eafba00 2024-04-21T08:58:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.279+0000 7fab20ff9640 1 -- 172.21.15.28:0/3102056063 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fab28042470 con 0x55e75eafba00 2024-04-21T08:58:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.279+0000 7fab41fc0740 10 client.17499 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.279+0000 7fab41fc0740 10 client.17499 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:58:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.279+0000 7fab41fc0740 20 client.17499 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.279+0000 7fab41fc0740 1 --2- 172.21.15.28:0/3102056063 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55e75eb72ae0 0x55e75eb74ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.279+0000 7fab41fc0740 4 client.17499 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:58:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.279+0000 7fab41fc0740 1 -- 172.21.15.28:0/3102056063 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e75ea64c10 con 0x55e75eb72ae0 2024-04-21T08:58:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.279+0000 7fab3d08e640 1 --2- 172.21.15.28:0/3102056063 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55e75eb72ae0 0x55e75eb74ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab20ff9640 10 client.17499 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:58:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab3d08e640 1 --2- 172.21.15.28:0/3102056063 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55e75eb72ae0 0x55e75eb74ed0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6377dd9d170f14a8 server_cookie=8aa7acede1f82091 in_seq=0 out_seq=0 2024-04-21T08:58:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab20ff9640 1 -- 172.21.15.28:0/3102056063 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e75ea64c10 con 0x55e75eb72ae0 2024-04-21T08:58:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab20ff9640 10 client.17499 handle_command_reply: tid=0 2024-04-21T08:58:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab41fc0740 1 client.17499 shutdown 2024-04-21T08:58:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab41fc0740 1 -- 172.21.15.28:0/3102056063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eafba00 msgr2=0x55e75eb71050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab41fc0740 1 --2- 172.21.15.28:0/3102056063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eafba00 0x55e75eb71050 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fab28005ab0 tx=0x7fab280044b0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab41fc0740 1 -- 172.21.15.28:0/3102056063 shutdown_connections 2024-04-21T08:58:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab41fc0740 1 --2- 172.21.15.28:0/3102056063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e75eafba00 0x55e75eb71050 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab41fc0740 1 --2- 172.21.15.28:0/3102056063 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55e75eb72ae0 0x55e75eb74ed0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab41fc0740 1 --2- 172.21.15.28:0/3102056063 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e75eaf2e80 0x55e75eaf5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab20ff9640 0 client.17499 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:58:08.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab41fc0740 1 --2- 172.21.15.28:0/3102056063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e75eaf24d0 0x55e75eaf5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab41fc0740 1 -- 172.21.15.28:0/3102056063 >> 172.21.15.28:0/3102056063 conn(0x55e75eaecba0 msgr2=0x55e75eaf32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab41fc0740 1 -- 172.21.15.28:0/3102056063 shutdown_connections 2024-04-21T08:58:08.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.280+0000 7fab41fc0740 1 -- 172.21.15.28:0/3102056063 wait complete. 2024-04-21T08:58:08.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.281+0000 7fab41fc0740 20 client.17499 trim_cache size 0 max 16384 2024-04-21T08:58:08.292 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:58:08.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.281+0000 7fab3f2f2640 1 -- 172.21.15.28:0/3636559252 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fab240a3d20 msgr2=0x7fab240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.281+0000 7fab3f2f2640 1 --2- 172.21.15.28:0/3636559252 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fab240a3d20 0x7fab240a61e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fab3000b9e0 tx=0x7fab300bc040 comp rx=0 tx=0).stop 2024-04-21T08:58:08.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.281+0000 7fab3f2f2640 1 -- 172.21.15.28:0/3636559252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab38197ea0 msgr2=0x7fab381a6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.281+0000 7fab3f2f2640 1 --2- 172.21.15.28:0/3636559252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab38197ea0 0x7fab381a6aa0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fab3000ba10 tx=0x7fab3000bed0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.281+0000 7fab3f2f2640 1 -- 172.21.15.28:0/3636559252 shutdown_connections 2024-04-21T08:58:08.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.281+0000 7fab3f2f2640 1 --2- 172.21.15.28:0/3636559252 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fab240a3d20 0x7fab240a61e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.281+0000 7fab3f2f2640 1 --2- 172.21.15.28:0/3636559252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab38197ea0 0x7fab381a6aa0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.281+0000 7fab3f2f2640 1 --2- 172.21.15.28:0/3636559252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab38106170 0x7fab38197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.281+0000 7fab3f2f2640 1 --2- 172.21.15.28:0/3636559252 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fab381010f0 0x7fab38197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.281+0000 7fab3f2f2640 1 -- 172.21.15.28:0/3636559252 >> 172.21.15.28:0/3636559252 conn(0x7fab38077fb0 msgr2=0x7fab380ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.281+0000 7fab3f2f2640 1 -- 172.21.15.28:0/3636559252 shutdown_connections 2024-04-21T08:58:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.281+0000 7fab3f2f2640 1 -- 172.21.15.28:0/3636559252 wait complete. 2024-04-21T08:58:08.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:58:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.401+0000 7ff123c5b640 1 -- 172.21.15.28:0/3625454821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c102b40 msgr2=0x7ff11c106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.401+0000 7ff123c5b640 1 --2- 172.21.15.28:0/3625454821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c102b40 0x7ff11c106b80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7ff10c009900 tx=0x7ff10c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:58:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.401+0000 7ff123c5b640 1 -- 172.21.15.28:0/3625454821 shutdown_connections 2024-04-21T08:58:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.401+0000 7ff123c5b640 1 --2- 172.21.15.28:0/3625454821 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff11c107260 0x7ff11c10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.401+0000 7ff123c5b640 1 --2- 172.21.15.28:0/3625454821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c102b40 0x7ff11c106b80 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.401+0000 7ff123c5b640 1 --2- 172.21.15.28:0/3625454821 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff11c102190 0x7ff11c102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.401+0000 7ff123c5b640 1 -- 172.21.15.28:0/3625454821 >> 172.21.15.28:0/3625454821 conn(0x7ff11c0793a0 msgr2=0x7ff11c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.401+0000 7ff123c5b640 1 -- 172.21.15.28:0/3625454821 shutdown_connections 2024-04-21T08:58:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.401+0000 7ff123c5b640 1 -- 172.21.15.28:0/3625454821 wait complete. 2024-04-21T08:58:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff123c5b640 1 Processor -- start 2024-04-21T08:58:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff123c5b640 1 -- start start 2024-04-21T08:58:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff123c5b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff11c102190 0x7ff11c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff123c5b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c107260 0x7ff11c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff123c5b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff11c074cd0 0x7ff11c1a6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff123c5b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c1121a0 con 0x7ff11c107260 2024-04-21T08:58:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff123c5b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c112020 con 0x7ff11c102190 2024-04-21T08:58:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff123c5b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c112320 con 0x7ff11c074cd0 2024-04-21T08:58:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff1211f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c107260 0x7ff11c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff1211f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c107260 0x7ff11c074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51844/0 (socket says 172.21.15.28:51844) 2024-04-21T08:58:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff1211f6640 1 -- 172.21.15.28:0/76356704 learned_addr learned my addr 172.21.15.28:0/76356704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff1219f7640 1 --2- 172.21.15.28:0/76356704 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff11c102190 0x7ff11c074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff1221f8640 1 --2- 172.21.15.28:0/76356704 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff11c074cd0 0x7ff11c1a6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff1211f6640 1 -- 172.21.15.28:0/76356704 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff11c074cd0 msgr2=0x7ff11c1a6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff1211f6640 1 --2- 172.21.15.28:0/76356704 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff11c074cd0 0x7ff11c1a6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff1211f6640 1 -- 172.21.15.28:0/76356704 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff11c102190 msgr2=0x7ff11c074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff1211f6640 1 --2- 172.21.15.28:0/76356704 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff11c102190 0x7ff11c074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff1211f6640 1 -- 172.21.15.28:0/76356704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11c1a7050 con 0x7ff11c107260 2024-04-21T08:58:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.402+0000 7ff1221f8640 1 --2- 172.21.15.28:0/76356704 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff11c074cd0 0x7ff11c1a6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.403+0000 7ff1211f6640 1 --2- 172.21.15.28:0/76356704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c107260 0x7ff11c074790 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7ff10c036040 tx=0x7ff10c02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.403+0000 7ff112ffd640 1 -- 172.21.15.28:0/76356704 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c043020 con 0x7ff11c107260 2024-04-21T08:58:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.403+0000 7ff112ffd640 1 -- 172.21.15.28:0/76356704 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff10c02cbe0 con 0x7ff11c107260 2024-04-21T08:58:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.403+0000 7ff123c5b640 1 -- 172.21.15.28:0/76356704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff11c1a72e0 con 0x7ff11c107260 2024-04-21T08:58:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.403+0000 7ff112ffd640 1 -- 172.21.15.28:0/76356704 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c003bf0 con 0x7ff11c107260 2024-04-21T08:58:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.403+0000 7ff123c5b640 1 -- 172.21.15.28:0/76356704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff11c1a7830 con 0x7ff11c107260 2024-04-21T08:58:08.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.405+0000 7ff112ffd640 1 -- 172.21.15.28:0/76356704 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff10c046020 con 0x7ff11c107260 2024-04-21T08:58:08.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.405+0000 7ff126845740 1 Processor -- start 2024-04-21T08:58:08.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff126845740 1 -- start start 2024-04-21T08:58:08.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff126845740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff834d0 0x55e06ff838b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff126845740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff83e80 0x55e06ff8c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff126845740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff8ca00 0x55e06ff8ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff126845740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e06fee81f0 con 0x55e06ff8ca00 2024-04-21T08:58:08.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff126845740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e06fe854b0 con 0x55e06ff834d0 2024-04-21T08:58:08.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff126845740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55e06fee41b0 con 0x55e06ff83e80 2024-04-21T08:58:08.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff112ffd640 1 --2- 172.21.15.28:0/76356704 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff1080a3d50 0x7ff1080a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff1211f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff8ca00 0x55e06ff8ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff1211f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff8ca00 0x55e06ff8ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51854/0 (socket says 172.21.15.28:51854) 2024-04-21T08:58:08.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff1211f6640 1 -- 172.21.15.28:0/3091113133 learned_addr learned my addr 172.21.15.28:0/3091113133 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:08.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff1219f7640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff83e80 0x55e06ff8c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff1221f8640 1 --2- 172.21.15.28:0/76356704 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff1080a3d50 0x7ff1080a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff1221f8640 1 --2- 172.21.15.28:0/3091113133 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff834d0 0x55e06ff838b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff112ffd640 1 -- 172.21.15.28:0/76356704 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7ff10c0f2e10 con 0x7ff11c107260 2024-04-21T08:58:08.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff1211f6640 1 -- 172.21.15.28:0/3091113133 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff83e80 msgr2=0x55e06ff8c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.406+0000 7ff1211f6640 1 --2- 172.21.15.28:0/3091113133 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff83e80 0x55e06ff8c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.407+0000 7ff1211f6640 1 -- 172.21.15.28:0/3091113133 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff834d0 msgr2=0x55e06ff838b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.407+0000 7ff1211f6640 1 --2- 172.21.15.28:0/3091113133 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff834d0 0x55e06ff838b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.407+0000 7ff1211f6640 1 -- 172.21.15.28:0/3091113133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e06fe771b0 con 0x55e06ff8ca00 2024-04-21T08:58:08.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.407+0000 7ff1219f7640 1 --2- 172.21.15.28:0/3091113133 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff83e80 0x55e06ff8c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:58:08.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.407+0000 7ff1211f6640 1 --2- 172.21.15.28:0/3091113133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff8ca00 0x55e06ff8ee50 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ff10c0c71b0 tx=0x7ff10c0a95a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=addc108f5e18f782 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.407+0000 7ff1221f8640 1 --2- 172.21.15.28:0/76356704 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff1080a3d50 0x7ff1080a6210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff114009fd0 tx=0x7ff114009290 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.407+0000 7ff107fff640 1 -- 172.21.15.28:0/3091113133 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c0be3d0 con 0x55e06ff8ca00 2024-04-21T08:58:08.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.407+0000 7ff107fff640 1 -- 172.21.15.28:0/3091113133 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff10c0bf170 con 0x55e06ff8ca00 2024-04-21T08:58:08.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.407+0000 7ff107fff640 1 -- 172.21.15.28:0/3091113133 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c0054d0 con 0x55e06ff8ca00 2024-04-21T08:58:08.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.408+0000 7ff126845740 1 -- 172.21.15.28:0/3091113133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff8ca00 msgr2=0x55e06ff8ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.408+0000 7ff126845740 1 --2- 172.21.15.28:0/3091113133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff8ca00 0x55e06ff8ee50 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ff10c0c71b0 tx=0x7ff10c0a95a0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.408+0000 7ff126845740 1 -- 172.21.15.28:0/3091113133 shutdown_connections 2024-04-21T08:58:08.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.408+0000 7ff126845740 1 --2- 172.21.15.28:0/3091113133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff8ca00 0x55e06ff8ee50 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.408+0000 7ff126845740 1 --2- 172.21.15.28:0/3091113133 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff83e80 0x55e06ff8c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.408+0000 7ff126845740 1 --2- 172.21.15.28:0/3091113133 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff834d0 0x55e06ff838b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.408+0000 7ff126845740 1 -- 172.21.15.28:0/3091113133 >> 172.21.15.28:0/3091113133 conn(0x55e06ff7dba0 msgr2=0x55e06ff7df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.408+0000 7ff126845740 1 -- 172.21.15.28:0/3091113133 shutdown_connections 2024-04-21T08:58:08.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.408+0000 7ff126845740 1 -- 172.21.15.28:0/3091113133 wait complete. 2024-04-21T08:58:08.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.408+0000 7ff126845740 1 -- start start 2024-04-21T08:58:08.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.408+0000 7ff126845740 1 Processor -- start 2024-04-21T08:58:08.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff126845740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff834d0 0x55e06ff8c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff126845740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff83e80 0x55e06ff89670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff126845740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff8ca00 0x55e06ff89bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff126845740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e06ff8f730 con 0x55e06ff83e80 2024-04-21T08:58:08.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff126845740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55e06fea1500 con 0x55e06ff8ca00 2024-04-21T08:58:08.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff126845740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55e06ff8f8b0 con 0x55e06ff834d0 2024-04-21T08:58:08.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff126845740 10 client.0 _ll_register_callbacks cb 0x55e06fe6fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:08.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff1211f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff83e80 0x55e06ff89670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff1211f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff83e80 0x55e06ff89670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51868/0 (socket says 172.21.15.28:51868) 2024-04-21T08:58:08.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff1211f6640 1 -- 172.21.15.28:0/2626993309 learned_addr learned my addr 172.21.15.28:0/2626993309 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:08.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff1221f8640 1 --2- 172.21.15.28:0/2626993309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff8ca00 0x55e06ff89bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff1219f7640 1 --2- 172.21.15.28:0/2626993309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff834d0 0x55e06ff8c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff1211f6640 1 -- 172.21.15.28:0/2626993309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff834d0 msgr2=0x55e06ff8c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff1211f6640 1 --2- 172.21.15.28:0/2626993309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff834d0 0x55e06ff8c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff1211f6640 1 -- 172.21.15.28:0/2626993309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff8ca00 msgr2=0x55e06ff89bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff1211f6640 1 --2- 172.21.15.28:0/2626993309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff8ca00 0x55e06ff89bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff1211f6640 1 -- 172.21.15.28:0/2626993309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e06fbfa3f0 con 0x55e06ff83e80 2024-04-21T08:58:08.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff1211f6640 1 --2- 172.21.15.28:0/2626993309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff83e80 0x55e06ff89670 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ff10c0f0710 tx=0x7ff10c0be670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659101548f9a505a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff1057fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:08.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff1057fa640 1 -- 172.21.15.28:0/2626993309 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c0bea70 con 0x55e06ff83e80 2024-04-21T08:58:08.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff1057fa640 1 -- 172.21.15.28:0/2626993309 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7ff10c0bb0e0 con 0x55e06ff83e80 2024-04-21T08:58:08.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.409+0000 7ff126845740 1 -- 172.21.15.28:0/2626993309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff10c0bf170 con 0x55e06ff83e80 2024-04-21T08:58:08.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.410+0000 7ff1057fa640 1 -- 172.21.15.28:0/2626993309 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff10c0c2e20 con 0x55e06ff83e80 2024-04-21T08:58:08.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.410+0000 7ff126845740 10 client.17511 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:08.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.410+0000 7ff126845740 1 -- 172.21.15.28:0/2626993309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e06fcc47d0 con 0x55e06ff83e80 2024-04-21T08:58:08.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.410+0000 7ff1057fa640 1 -- 172.21.15.28:0/2626993309 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff10c0bf170 con 0x55e06ff83e80 2024-04-21T08:58:08.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.410+0000 7ff1057fa640 1 -- 172.21.15.28:0/2626993309 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7ff10c0c7250 con 0x55e06ff83e80 2024-04-21T08:58:08.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.410+0000 7ff126845740 10 client.17511 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:08.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.410+0000 7ff126845740 10 client.17511 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:08.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.410+0000 7ff126845740 20 client.17511 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:08.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.410+0000 7ff126845740 1 --2- 172.21.15.28:0/2626993309 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e06ff9c790 0x55e06ff9eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.410+0000 7ff126845740 4 client.17511 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:58:08.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.410+0000 7ff126845740 1 -- 172.21.15.28:0/2626993309 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e06fe87930 con 0x55e06ff9c790 2024-04-21T08:58:08.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.410+0000 7ff1219f7640 1 --2- 172.21.15.28:0/2626993309 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e06ff9c790 0x55e06ff9eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.411+0000 7ff1057fa640 10 client.17511 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:08.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.411+0000 7ff1219f7640 1 --2- 172.21.15.28:0/2626993309 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e06ff9c790 0x55e06ff9eb80 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97a84bf37585f5fe server_cookie=31f31f16d930782f in_seq=0 out_seq=0 2024-04-21T08:58:08.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.412+0000 7ff1057fa640 1 -- 172.21.15.28:0/2626993309 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e06fe87930 con 0x55e06ff9c790 2024-04-21T08:58:08.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.412+0000 7ff1057fa640 10 client.17511 handle_command_reply: tid=0 2024-04-21T08:58:08.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.412+0000 7ff126845740 1 client.17511 shutdown 2024-04-21T08:58:08.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.412+0000 7ff126845740 1 -- 172.21.15.28:0/2626993309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff83e80 msgr2=0x55e06ff89670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.412+0000 7ff126845740 1 --2- 172.21.15.28:0/2626993309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff83e80 0x55e06ff89670 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ff10c0f0710 tx=0x7ff10c0be670 comp rx=0 tx=0).stop 2024-04-21T08:58:08.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.412+0000 7ff126845740 1 -- 172.21.15.28:0/2626993309 shutdown_connections 2024-04-21T08:58:08.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.412+0000 7ff126845740 1 --2- 172.21.15.28:0/2626993309 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e06ff9c790 0x55e06ff9eb80 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.412+0000 7ff126845740 1 --2- 172.21.15.28:0/2626993309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff8ca00 0x55e06ff89bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.412+0000 7ff126845740 1 --2- 172.21.15.28:0/2626993309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff83e80 0x55e06ff89670 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.412+0000 7ff1057fa640 0 client.17511 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.412+0000 7ff126845740 1 --2- 172.21.15.28:0/2626993309 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff834d0 0x55e06ff8c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.412+0000 7ff126845740 1 -- 172.21.15.28:0/2626993309 >> 172.21.15.28:0/2626993309 conn(0x55e06ff7dba0 msgr2=0x55e06ff7f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.412+0000 7ff126845740 1 -- 172.21.15.28:0/2626993309 shutdown_connections 2024-04-21T08:58:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.412+0000 7ff126845740 1 -- 172.21.15.28:0/2626993309 wait complete. 2024-04-21T08:58:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.412+0000 7ff126845740 20 client.17511 trim_cache size 0 max 16384 2024-04-21T08:58:08.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.425+0000 7ff126845740 1 Processor -- start 2024-04-21T08:58:08.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.425+0000 7ff126845740 1 -- start start 2024-04-21T08:58:08.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.425+0000 7ff126845740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff834d0 0x55e06ff98560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.425+0000 7ff126845740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff83e80 0x55e06ff887d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.425+0000 7ff126845740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff8ca00 0x55e06ff7f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.425+0000 7ff126845740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c1b4080 con 0x55e06ff834d0 2024-04-21T08:58:08.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.425+0000 7ff126845740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c1b1950 con 0x55e06ff83e80 2024-04-21T08:58:08.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.425+0000 7ff126845740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c1af7c0 con 0x55e06ff8ca00 2024-04-21T08:58:08.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.426+0000 7ff1219f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff834d0 0x55e06ff98560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.426+0000 7ff1219f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff834d0 0x55e06ff98560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51882/0 (socket says 172.21.15.28:51882) 2024-04-21T08:58:08.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.426+0000 7ff1219f7640 1 -- 172.21.15.28:0/2972816411 learned_addr learned my addr 172.21.15.28:0/2972816411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:08.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.426+0000 7ff1211f6640 1 --2- 172.21.15.28:0/2972816411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff83e80 0x55e06ff887d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.426+0000 7ff1219f7640 1 -- 172.21.15.28:0/2972816411 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff8ca00 msgr2=0x55e06ff7f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.426+0000 7ff1219f7640 1 --2- 172.21.15.28:0/2972816411 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff8ca00 0x55e06ff7f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.426+0000 7ff1219f7640 1 -- 172.21.15.28:0/2972816411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff83e80 msgr2=0x55e06ff887d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.426+0000 7ff1219f7640 1 --2- 172.21.15.28:0/2972816411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff83e80 0x55e06ff887d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.426+0000 7ff1219f7640 1 -- 172.21.15.28:0/2972816411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e06ff89f40 con 0x55e06ff834d0 2024-04-21T08:58:08.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.426+0000 7ff1219f7640 1 --2- 172.21.15.28:0/2972816411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff834d0 0x55e06ff98560 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x55e06ff9f300 tx=0x7ff118011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392a9eafd586acd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.426+0000 7ff1077fe640 1 -- 172.21.15.28:0/2972816411 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1180042f0 con 0x55e06ff834d0 2024-04-21T08:58:08.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.426+0000 7ff1077fe640 1 -- 172.21.15.28:0/2972816411 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff118022070 con 0x55e06ff834d0 2024-04-21T08:58:08.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.426+0000 7ff1077fe640 1 -- 172.21.15.28:0/2972816411 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff118003d10 con 0x55e06ff834d0 2024-04-21T08:58:08.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.427+0000 7ff126845740 1 -- 172.21.15.28:0/2972816411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff834d0 msgr2=0x55e06ff98560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.427+0000 7ff126845740 1 --2- 172.21.15.28:0/2972816411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff834d0 0x55e06ff98560 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x55e06ff9f300 tx=0x7ff118011980 comp rx=0 tx=0).stop 2024-04-21T08:58:08.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.427+0000 7ff126845740 1 -- 172.21.15.28:0/2972816411 shutdown_connections 2024-04-21T08:58:08.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.427+0000 7ff126845740 1 --2- 172.21.15.28:0/2972816411 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff8ca00 0x55e06ff7f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.427+0000 7ff126845740 1 --2- 172.21.15.28:0/2972816411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff83e80 0x55e06ff887d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.427+0000 7ff126845740 1 --2- 172.21.15.28:0/2972816411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff834d0 0x55e06ff98560 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.427+0000 7ff126845740 1 -- 172.21.15.28:0/2972816411 >> 172.21.15.28:0/2972816411 conn(0x55e06ff7dba0 msgr2=0x55e06ff7eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.427+0000 7ff126845740 1 -- 172.21.15.28:0/2972816411 shutdown_connections 2024-04-21T08:58:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.427+0000 7ff126845740 1 -- 172.21.15.28:0/2972816411 wait complete. 2024-04-21T08:58:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.427+0000 7ff126845740 1 -- start start 2024-04-21T08:58:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.427+0000 7ff126845740 1 Processor -- start 2024-04-21T08:58:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff126845740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff834d0 0x55e0700037c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff126845740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff83e80 0x55e070003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff126845740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff8ca00 0x55e06ff8fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff126845740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c1afe10 con 0x55e06ff834d0 2024-04-21T08:58:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff126845740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c1b1280 con 0x55e06ff83e80 2024-04-21T08:58:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff126845740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c1b04e0 con 0x55e06ff8ca00 2024-04-21T08:58:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff126845740 10 client.0 _ll_register_callbacks cb 0x7ff11800b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff1219f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff834d0 0x55e0700037c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff1219f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff834d0 0x55e0700037c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51884/0 (socket says 172.21.15.28:51884) 2024-04-21T08:58:08.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff1219f7640 1 -- 172.21.15.28:0/4064454625 learned_addr learned my addr 172.21.15.28:0/4064454625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff1211f6640 1 --2- 172.21.15.28:0/4064454625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff83e80 0x55e070003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff1219f7640 1 -- 172.21.15.28:0/4064454625 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff8ca00 msgr2=0x55e06ff8fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff1219f7640 1 --2- 172.21.15.28:0/4064454625 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff8ca00 0x55e06ff8fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff1219f7640 1 -- 172.21.15.28:0/4064454625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff83e80 msgr2=0x55e070003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff1219f7640 1 --2- 172.21.15.28:0/4064454625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff83e80 0x55e070003d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff1219f7640 1 -- 172.21.15.28:0/4064454625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e06ff904e0 con 0x55e06ff834d0 2024-04-21T08:58:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff1219f7640 1 --2- 172.21.15.28:0/4064454625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff834d0 0x55e0700037c0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff1180037e0 tx=0x7ff118003810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=890199eeec3e242a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff1057fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff1057fa640 1 -- 172.21.15.28:0/4064454625 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff118003a90 con 0x55e06ff834d0 2024-04-21T08:58:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff1057fa640 1 -- 172.21.15.28:0/4064454625 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7ff118003c30 con 0x55e06ff834d0 2024-04-21T08:58:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff126845740 1 -- 172.21.15.28:0/4064454625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff118022070 con 0x55e06ff834d0 2024-04-21T08:58:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff1057fa640 1 -- 172.21.15.28:0/4064454625 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff118038050 con 0x55e06ff834d0 2024-04-21T08:58:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff126845740 10 client.17517 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.428+0000 7ff126845740 1 -- 172.21.15.28:0/4064454625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e07000af10 con 0x55e06ff834d0 2024-04-21T08:58:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.429+0000 7ff1057fa640 1 -- 172.21.15.28:0/4064454625 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff118022070 con 0x55e06ff834d0 2024-04-21T08:58:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.429+0000 7ff1057fa640 1 -- 172.21.15.28:0/4064454625 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7ff118010950 con 0x55e06ff834d0 2024-04-21T08:58:08.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.429+0000 7ff126845740 10 client.17517 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.429+0000 7ff126845740 10 client.17517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.429+0000 7ff126845740 20 client.17517 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.429+0000 7ff126845740 1 --2- 172.21.15.28:0/4064454625 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e07000b600 0x55e07000da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.429+0000 7ff126845740 4 client.17517 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:58:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.429+0000 7ff126845740 1 -- 172.21.15.28:0/4064454625 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e06fef5c10 con 0x55e07000b600 2024-04-21T08:58:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.429+0000 7ff1211f6640 1 --2- 172.21.15.28:0/4064454625 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e07000b600 0x55e07000da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff1211f6640 1 --2- 172.21.15.28:0/4064454625 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e07000b600 0x55e07000da40 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76b44aa592520658 server_cookie=8fdf7faf74f4165a in_seq=0 out_seq=0 2024-04-21T08:58:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff1057fa640 10 client.17517 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff1057fa640 1 -- 172.21.15.28:0/4064454625 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e06fef5c10 con 0x55e07000b600 2024-04-21T08:58:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff1057fa640 10 client.17517 handle_command_reply: tid=0 2024-04-21T08:58:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff126845740 1 client.17517 shutdown 2024-04-21T08:58:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff126845740 1 -- 172.21.15.28:0/4064454625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff834d0 msgr2=0x55e0700037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff126845740 1 --2- 172.21.15.28:0/4064454625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff834d0 0x55e0700037c0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff1180037e0 tx=0x7ff118003810 comp rx=0 tx=0).stop 2024-04-21T08:58:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff126845740 1 -- 172.21.15.28:0/4064454625 shutdown_connections 2024-04-21T08:58:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff126845740 1 --2- 172.21.15.28:0/4064454625 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55e07000b600 0x55e07000da40 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff126845740 1 --2- 172.21.15.28:0/4064454625 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55e06ff8ca00 0x55e06ff8fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff126845740 1 --2- 172.21.15.28:0/4064454625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e06ff83e80 0x55e070003d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff126845740 1 --2- 172.21.15.28:0/4064454625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e06ff834d0 0x55e0700037c0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff1057fa640 0 client.17517 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:08.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff126845740 1 -- 172.21.15.28:0/4064454625 >> 172.21.15.28:0/4064454625 conn(0x55e06ff7dba0 msgr2=0x55e06ff7cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff126845740 1 -- 172.21.15.28:0/4064454625 shutdown_connections 2024-04-21T08:58:08.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff126845740 1 -- 172.21.15.28:0/4064454625 wait complete. 2024-04-21T08:58:08.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.430+0000 7ff126845740 20 client.17517 trim_cache size 0 max 16384 2024-04-21T08:58:08.441 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:58:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.431+0000 7ff123c5b640 1 -- 172.21.15.28:0/76356704 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff1080a3d50 msgr2=0x7ff1080a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.431+0000 7ff123c5b640 1 --2- 172.21.15.28:0/76356704 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff1080a3d50 0x7ff1080a6210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff114009fd0 tx=0x7ff114009290 comp rx=0 tx=0).stop 2024-04-21T08:58:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.431+0000 7ff123c5b640 1 -- 172.21.15.28:0/76356704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c107260 msgr2=0x7ff11c074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.431+0000 7ff123c5b640 1 --2- 172.21.15.28:0/76356704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c107260 0x7ff11c074790 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7ff10c036040 tx=0x7ff10c02c7e0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.431+0000 7ff123c5b640 1 -- 172.21.15.28:0/76356704 shutdown_connections 2024-04-21T08:58:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.431+0000 7ff123c5b640 1 --2- 172.21.15.28:0/76356704 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff1080a3d50 0x7ff1080a6210 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.431+0000 7ff123c5b640 1 --2- 172.21.15.28:0/76356704 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff11c074cd0 0x7ff11c1a6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.431+0000 7ff123c5b640 1 --2- 172.21.15.28:0/76356704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11c107260 0x7ff11c074790 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.431+0000 7ff123c5b640 1 --2- 172.21.15.28:0/76356704 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff11c102190 0x7ff11c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:08.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.431+0000 7ff123c5b640 1 -- 172.21.15.28:0/76356704 >> 172.21.15.28:0/76356704 conn(0x7ff11c0793a0 msgr2=0x7ff11c100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:08.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.431+0000 7ff123c5b640 1 -- 172.21.15.28:0/76356704 shutdown_connections 2024-04-21T08:58:08.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:08.431+0000 7ff123c5b640 1 -- 172.21.15.28:0/76356704 wait complete. 2024-04-21T08:58:08.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:08 smithi152 ceph-mon[31401]: pgmap v211: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T08:58:08.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:08 smithi152 ceph-mon[31401]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T08:58:08.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:08 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3319040864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:58:08.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:08 smithi028 ceph-mon[28003]: pgmap v211: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T08:58:08.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:08 smithi028 ceph-mon[28003]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T08:58:08.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:08 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3319040864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:58:08.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:08 smithi171 ceph-mon[31358]: pgmap v211: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T08:58:08.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:08 smithi171 ceph-mon[31358]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T08:58:08.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:08 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3319040864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:58:09.445 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:58:09.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:58:09.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:58:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.566+0000 7ff63b72c640 1 -- 172.21.15.28:0/1506310879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6341011a0 msgr2=0x7ff634101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.566+0000 7ff63b72c640 1 --2- 172.21.15.28:0/1506310879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6341011a0 0x7ff634101600 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ff624009930 tx=0x7ff62402bc30 comp rx=0 tx=0).stop 2024-04-21T08:58:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.566+0000 7ff63b72c640 1 -- 172.21.15.28:0/1506310879 shutdown_connections 2024-04-21T08:58:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.566+0000 7ff63b72c640 1 --2- 172.21.15.28:0/1506310879 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff634101b40 0x7ff63410dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.566+0000 7ff63b72c640 1 --2- 172.21.15.28:0/1506310879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6341011a0 0x7ff634101600 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.566+0000 7ff63b72c640 1 --2- 172.21.15.28:0/1506310879 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff634107030 0x7ff634107410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.566+0000 7ff63b72c640 1 -- 172.21.15.28:0/1506310879 >> 172.21.15.28:0/1506310879 conn(0x7ff634076c50 msgr2=0x7ff634077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.566+0000 7ff63b72c640 1 -- 172.21.15.28:0/1506310879 shutdown_connections 2024-04-21T08:58:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.566+0000 7ff63b72c640 1 -- 172.21.15.28:0/1506310879 wait complete. 2024-04-21T08:58:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.566+0000 7ff63b72c640 1 Processor -- start 2024-04-21T08:58:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff63b72c640 1 -- start start 2024-04-21T08:58:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff63b72c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff6341011a0 0x7ff634197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff63b72c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff634101b40 0x7ff634197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff63b72c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff634107030 0x7ff63419bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff63b72c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff634111110 con 0x7ff634107030 2024-04-21T08:58:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff63b72c640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff634110f90 con 0x7ff634101b40 2024-04-21T08:58:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff63b72c640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff634111290 con 0x7ff6341011a0 2024-04-21T08:58:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff639cc9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff634107030 0x7ff63419bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff639cc9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff634107030 0x7ff63419bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51908/0 (socket says 172.21.15.28:51908) 2024-04-21T08:58:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff639cc9640 1 -- 172.21.15.28:0/3165829421 learned_addr learned my addr 172.21.15.28:0/3165829421 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff638cc7640 1 --2- 172.21.15.28:0/3165829421 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff634101b40 0x7ff634197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff6394c8640 1 --2- 172.21.15.28:0/3165829421 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff6341011a0 0x7ff634197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff639cc9640 1 -- 172.21.15.28:0/3165829421 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff6341011a0 msgr2=0x7ff634197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff639cc9640 1 --2- 172.21.15.28:0/3165829421 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff6341011a0 0x7ff634197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff639cc9640 1 -- 172.21.15.28:0/3165829421 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff634101b40 msgr2=0x7ff634197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff639cc9640 1 --2- 172.21.15.28:0/3165829421 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff634101b40 0x7ff634197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff639cc9640 1 -- 172.21.15.28:0/3165829421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff63419c620 con 0x7ff634107030 2024-04-21T08:58:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.567+0000 7ff639cc9640 1 --2- 172.21.15.28:0/3165829421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff634107030 0x7ff63419bef0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00c970 tx=0x7ff62c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.568+0000 7ff62a7fc640 1 -- 172.21.15.28:0/3165829421 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c007bb0 con 0x7ff634107030 2024-04-21T08:58:09.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.568+0000 7ff63b72c640 1 -- 172.21.15.28:0/3165829421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff63419c910 con 0x7ff634107030 2024-04-21T08:58:09.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.568+0000 7ff62a7fc640 1 -- 172.21.15.28:0/3165829421 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff62c007d50 con 0x7ff634107030 2024-04-21T08:58:09.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.568+0000 7ff62a7fc640 1 -- 172.21.15.28:0/3165829421 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62c018770 con 0x7ff634107030 2024-04-21T08:58:09.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.568+0000 7ff63b72c640 1 -- 172.21.15.28:0/3165829421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6341a41a0 con 0x7ff634107030 2024-04-21T08:58:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff62a7fc640 1 -- 172.21.15.28:0/3165829421 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff62c00f990 con 0x7ff634107030 2024-04-21T08:58:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff63e3f0740 1 Processor -- start 2024-04-21T08:58:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff63e3f0740 1 -- start start 2024-04-21T08:58:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff63e3f0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d265fdaf0 0x556d265fded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff63e3f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d26606a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff63e3f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d26607020 0x556d26609470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff63e3f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d2655ff00 con 0x556d265fe4a0 2024-04-21T08:58:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff63e3f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556d264fd590 con 0x556d26607020 2024-04-21T08:58:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff63e3f0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x556d2655bec0 con 0x556d265fdaf0 2024-04-21T08:58:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff6394c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d26606a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff6394c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d26606a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51916/0 (socket says 172.21.15.28:51916) 2024-04-21T08:58:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff6394c8640 1 -- 172.21.15.28:0/2027355424 learned_addr learned my addr 172.21.15.28:0/2027355424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff638cc7640 1 --2- 172.21.15.28:0/2027355424 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d265fdaf0 0x556d265fded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff638cc7640 1 --2- 172.21.15.28:0/2027355424 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d26607020 0x556d26609470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff62a7fc640 1 --2- 172.21.15.28:0/3165829421 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6200a3d20 0x7ff6200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff6394c8640 1 -- 172.21.15.28:0/2027355424 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d265fdaf0 msgr2=0x556d265fded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff6394c8640 1 --2- 172.21.15.28:0/2027355424 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d265fdaf0 0x556d265fded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff6394c8640 1 -- 172.21.15.28:0/2027355424 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d26607020 msgr2=0x556d26609470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff6394c8640 1 --2- 172.21.15.28:0/2027355424 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d26607020 0x556d26609470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff6394c8640 1 -- 172.21.15.28:0/2027355424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d2633a7e0 con 0x556d265fe4a0 2024-04-21T08:58:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.571+0000 7ff638cc7640 1 --2- 172.21.15.28:0/2027355424 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d265fdaf0 0x556d265fded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.572+0000 7ff639cc9640 1 --2- 172.21.15.28:0/3165829421 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6200a3d20 0x7ff6200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.572+0000 7ff62a7fc640 1 -- 172.21.15.28:0/3165829421 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7ff62c0d0c70 con 0x7ff634107030 2024-04-21T08:58:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.572+0000 7ff6394c8640 1 --2- 172.21.15.28:0/2027355424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d26606a50 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ff63000af40 tx=0x7ff63000f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b278920949aeaa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.572+0000 7ff61f7fe640 1 -- 172.21.15.28:0/2027355424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63001d070 con 0x556d265fe4a0 2024-04-21T08:58:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.572+0000 7ff61f7fe640 1 -- 172.21.15.28:0/2027355424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff630010db0 con 0x556d265fe4a0 2024-04-21T08:58:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.572+0000 7ff61f7fe640 1 -- 172.21.15.28:0/2027355424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff630018da0 con 0x556d265fe4a0 2024-04-21T08:58:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.573+0000 7ff63e3f0740 1 -- 172.21.15.28:0/2027355424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 msgr2=0x556d26606a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.573+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/2027355424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d26606a50 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ff63000af40 tx=0x7ff63000f850 comp rx=0 tx=0).stop 2024-04-21T08:58:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.573+0000 7ff639cc9640 1 --2- 172.21.15.28:0/3165829421 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6200a3d20 0x7ff6200a61e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff62c0ce020 tx=0x7ff62c091c50 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.573+0000 7ff63e3f0740 1 -- 172.21.15.28:0/2027355424 shutdown_connections 2024-04-21T08:58:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.573+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/2027355424 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d26607020 0x556d26609470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.573+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/2027355424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d26606a50 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.573+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/2027355424 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d265fdaf0 0x556d265fded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.573+0000 7ff63e3f0740 1 -- 172.21.15.28:0/2027355424 >> 172.21.15.28:0/2027355424 conn(0x556d265f81c0 msgr2=0x556d265f85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.573+0000 7ff63e3f0740 1 -- 172.21.15.28:0/2027355424 shutdown_connections 2024-04-21T08:58:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.573+0000 7ff63e3f0740 1 -- 172.21.15.28:0/2027355424 wait complete. 2024-04-21T08:58:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.573+0000 7ff63e3f0740 1 -- start start 2024-04-21T08:58:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.573+0000 7ff63e3f0740 1 Processor -- start 2024-04-21T08:58:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff63e3f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fdaf0 0x556d26603fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff63e3f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d265fe4a0 0x556d266044f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff63e3f0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d26607020 0x556d26604a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff63e3f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d26609f90 con 0x556d265fdaf0 2024-04-21T08:58:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff63e3f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556d26519580 con 0x556d265fe4a0 2024-04-21T08:58:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff63e3f0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x556d2660a110 con 0x556d26607020 2024-04-21T08:58:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff63e3f0740 10 client.0 _ll_register_callbacks cb 0x556d2651f650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff638cc7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fdaf0 0x556d26603fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff638cc7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fdaf0 0x556d26603fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51920/0 (socket says 172.21.15.28:51920) 2024-04-21T08:58:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff638cc7640 1 -- 172.21.15.28:0/363750705 learned_addr learned my addr 172.21.15.28:0/363750705 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff6394c8640 1 --2- 172.21.15.28:0/363750705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d265fe4a0 0x556d266044f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff638cc7640 1 --2- 172.21.15.28:0/363750705 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d26607020 0x556d26604a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff6394c8640 1 -- 172.21.15.28:0/363750705 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d26607020 msgr2=0x556d26604a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff6394c8640 1 --2- 172.21.15.28:0/363750705 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d26607020 0x556d26604a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff6394c8640 1 -- 172.21.15.28:0/363750705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fdaf0 msgr2=0x556d26603fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff6394c8640 1 --2- 172.21.15.28:0/363750705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fdaf0 0x556d26603fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff6394c8640 1 -- 172.21.15.28:0/363750705 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d2639af70 con 0x556d265fe4a0 2024-04-21T08:58:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff638cc7640 1 --2- 172.21.15.28:0/363750705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fdaf0 0x556d26603fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:58:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff6394c8640 1 --2- 172.21.15.28:0/363750705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d265fe4a0 0x556d266044f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x556d265f9230 tx=0x7ff63000ffd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7278ffb501fa77 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.574+0000 7ff61cff9640 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-04-21T08:58:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.575+0000 7ff61cff9640 1 -- 172.21.15.28:0/363750705 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63001d050 con 0x556d265fe4a0 2024-04-21T08:58:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.575+0000 7ff63e3f0740 1 -- 172.21.15.28:0/363750705 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff630010db0 con 0x556d265fe4a0 2024-04-21T08:58:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.575+0000 7ff61cff9640 1 -- 172.21.15.28:0/363750705 <== mon.2 v2:172.21.15.152:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7ff630004a40 con 0x556d265fe4a0 2024-04-21T08:58:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.575+0000 7ff61cff9640 1 -- 172.21.15.28:0/363750705 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff630031050 con 0x556d265fe4a0 2024-04-21T08:58:09.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.575+0000 7ff63e3f0740 10 client.26660 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.575+0000 7ff63e3f0740 1 -- 172.21.15.28:0/363750705 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556d26232170 con 0x556d265fe4a0 2024-04-21T08:58:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.575+0000 7ff61cff9640 1 -- 172.21.15.28:0/363750705 <== mon.2 v2:172.21.15.152:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff630010db0 con 0x556d265fe4a0 2024-04-21T08:58:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.575+0000 7ff61cff9640 1 -- 172.21.15.28:0/363750705 <== mon.2 v2:172.21.15.152:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7ff630005160 con 0x556d265fe4a0 2024-04-21T08:58:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.575+0000 7ff63e3f0740 10 client.26660 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.575+0000 7ff63e3f0740 10 client.26660 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.575+0000 7ff63e3f0740 20 client.26660 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.575+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/363750705 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556d26617420 0x556d26619810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.575+0000 7ff63e3f0740 4 client.26660 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:58:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.575+0000 7ff63e3f0740 1 -- 172.21.15.28:0/363750705 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d2656d920 con 0x556d26617420 2024-04-21T08:58:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.576+0000 7ff638cc7640 1 --2- 172.21.15.28:0/363750705 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556d26617420 0x556d26619810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.576+0000 7ff638cc7640 1 --2- 172.21.15.28:0/363750705 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556d26617420 0x556d26619810 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3096adcea67f9e54 server_cookie=73ea964f8cbbe0a0 in_seq=0 out_seq=0 2024-04-21T08:58:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.576+0000 7ff61cff9640 10 client.26660 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.577+0000 7ff61cff9640 1 -- 172.21.15.28:0/363750705 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556d2656d920 con 0x556d26617420 2024-04-21T08:58:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.577+0000 7ff61cff9640 10 client.26660 handle_command_reply: tid=0 2024-04-21T08:58:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.577+0000 7ff63e3f0740 1 client.26660 shutdown 2024-04-21T08:58:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.577+0000 7ff63e3f0740 1 -- 172.21.15.28:0/363750705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d265fe4a0 msgr2=0x556d266044f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.577+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/363750705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d265fe4a0 0x556d266044f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x556d265f9230 tx=0x7ff63000ffd0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.577+0000 7ff63e3f0740 1 -- 172.21.15.28:0/363750705 shutdown_connections 2024-04-21T08:58:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.577+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/363750705 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556d26617420 0x556d26619810 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.577+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/363750705 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d26607020 0x556d26604a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.577+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/363750705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d265fe4a0 0x556d266044f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.577+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/363750705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fdaf0 0x556d26603fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.577+0000 7ff61cff9640 0 client.26660 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.577+0000 7ff63e3f0740 1 -- 172.21.15.28:0/363750705 >> 172.21.15.28:0/363750705 conn(0x556d265f81c0 msgr2=0x556d265f9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.577+0000 7ff63e3f0740 1 -- 172.21.15.28:0/363750705 shutdown_connections 2024-04-21T08:58:09.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.577+0000 7ff63e3f0740 1 -- 172.21.15.28:0/363750705 wait complete. 2024-04-21T08:58:09.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.577+0000 7ff63e3f0740 20 client.26660 trim_cache size 0 max 16384 2024-04-21T08:58:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.595+0000 7ff63e3f0740 1 Processor -- start 2024-04-21T08:58:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.595+0000 7ff63e3f0740 1 -- start start 2024-04-21T08:58:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.595+0000 7ff63e3f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d265fdaf0 0x556d266066e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.595+0000 7ff63e3f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d26612e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.595+0000 7ff63e3f0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d26607020 0x556d265f4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.595+0000 7ff63e3f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6341afc10 con 0x556d265fe4a0 2024-04-21T08:58:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.595+0000 7ff63e3f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff6341ad4e0 con 0x556d265fdaf0 2024-04-21T08:58:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.595+0000 7ff63e3f0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff6341ab350 con 0x556d26607020 2024-04-21T08:58:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.595+0000 7ff6394c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d26612e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.595+0000 7ff6394c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d26612e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51922/0 (socket says 172.21.15.28:51922) 2024-04-21T08:58:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.595+0000 7ff6394c8640 1 -- 172.21.15.28:0/27864694 learned_addr learned my addr 172.21.15.28:0/27864694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.596+0000 7ff6394c8640 1 -- 172.21.15.28:0/27864694 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d26607020 msgr2=0x556d265f4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.596+0000 7ff6394c8640 1 --2- 172.21.15.28:0/27864694 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d26607020 0x556d265f4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.596+0000 7ff6394c8640 1 -- 172.21.15.28:0/27864694 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d265fdaf0 msgr2=0x556d266066e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.596+0000 7ff6394c8640 1 --2- 172.21.15.28:0/27864694 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d265fdaf0 0x556d266066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.596+0000 7ff6394c8640 1 -- 172.21.15.28:0/27864694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d265f03f0 con 0x556d265fe4a0 2024-04-21T08:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.596+0000 7ff6394c8640 1 --2- 172.21.15.28:0/27864694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d26612e80 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff63000af40 tx=0x7ff63000adc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9699a27a26900cef server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.596+0000 7ff61effd640 1 -- 172.21.15.28:0/27864694 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63001d040 con 0x556d265fe4a0 2024-04-21T08:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.596+0000 7ff61effd640 1 -- 172.21.15.28:0/27864694 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff63001e040 con 0x556d265fe4a0 2024-04-21T08:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.596+0000 7ff61effd640 1 -- 172.21.15.28:0/27864694 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff630003970 con 0x556d265fe4a0 2024-04-21T08:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.596+0000 7ff63e3f0740 1 -- 172.21.15.28:0/27864694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 msgr2=0x556d26612e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.596+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/27864694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d26612e80 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff63000af40 tx=0x7ff63000adc0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 1 -- 172.21.15.28:0/27864694 shutdown_connections 2024-04-21T08:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/27864694 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d26607020 0x556d265f4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/27864694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d26612e80 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/27864694 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d265fdaf0 0x556d266066e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 1 -- 172.21.15.28:0/27864694 >> 172.21.15.28:0/27864694 conn(0x556d265f81c0 msgr2=0x556d265f4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:09.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 1 -- 172.21.15.28:0/27864694 shutdown_connections 2024-04-21T08:58:09.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 1 -- 172.21.15.28:0/27864694 wait complete. 2024-04-21T08:58:09.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 1 -- start start 2024-04-21T08:58:09.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 1 Processor -- start 2024-04-21T08:58:09.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d265fdaf0 0x556d2660ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d2660b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d26607020 0x556d2660b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6341ab9a0 con 0x556d265fe4a0 2024-04-21T08:58:09.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ff6341ace10 con 0x556d265fdaf0 2024-04-21T08:58:09.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ff6341ac070 con 0x556d26607020 2024-04-21T08:58:09.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff63e3f0740 10 client.0 _ll_register_callbacks cb 0x7ff63001b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:09.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.597+0000 7ff6394c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d2660b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff6394c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d2660b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51928/0 (socket says 172.21.15.28:51928) 2024-04-21T08:58:09.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff6394c8640 1 -- 172.21.15.28:0/616946526 learned_addr learned my addr 172.21.15.28:0/616946526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:09.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff638cc7640 1 --2- 172.21.15.28:0/616946526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d265fdaf0 0x556d2660ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff638cc7640 1 --2- 172.21.15.28:0/616946526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d26607020 0x556d2660b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff6394c8640 1 -- 172.21.15.28:0/616946526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d26607020 msgr2=0x556d2660b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff6394c8640 1 --2- 172.21.15.28:0/616946526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d26607020 0x556d2660b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff6394c8640 1 -- 172.21.15.28:0/616946526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d265fdaf0 msgr2=0x556d2660ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff6394c8640 1 --2- 172.21.15.28:0/616946526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d265fdaf0 0x556d2660ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff6394c8640 1 -- 172.21.15.28:0/616946526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d2667c680 con 0x556d265fe4a0 2024-04-21T08:58:09.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff638cc7640 1 --2- 172.21.15.28:0/616946526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d26607020 0x556d2660b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:09.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff6394c8640 1 --2- 172.21.15.28:0/616946526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d2660b370 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff63000aa10 tx=0x7ff630005a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f453ea91114dcdab server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:09.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff61cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:09.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff61cff9640 1 -- 172.21.15.28:0/616946526 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63001d070 con 0x556d265fe4a0 2024-04-21T08:58:09.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff61cff9640 1 -- 172.21.15.28:0/616946526 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7ff630003b80 con 0x556d265fe4a0 2024-04-21T08:58:09.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff63e3f0740 1 -- 172.21.15.28:0/616946526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff63001e040 con 0x556d265fe4a0 2024-04-21T08:58:09.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff61cff9640 1 -- 172.21.15.28:0/616946526 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff630018790 con 0x556d265fe4a0 2024-04-21T08:58:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff63e3f0740 10 client.17535 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.598+0000 7ff63e3f0740 1 -- 172.21.15.28:0/616946526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556d2667d9a0 con 0x556d265fe4a0 2024-04-21T08:58:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.599+0000 7ff61cff9640 1 -- 172.21.15.28:0/616946526 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff63001e040 con 0x556d265fe4a0 2024-04-21T08:58:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.599+0000 7ff61cff9640 1 -- 172.21.15.28:0/616946526 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7ff63003b070 con 0x556d265fe4a0 2024-04-21T08:58:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.599+0000 7ff63e3f0740 10 client.17535 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.599+0000 7ff63e3f0740 10 client.17535 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.599+0000 7ff63e3f0740 20 client.17535 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.599+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/616946526 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556d2667e030 0x556d26680450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.599+0000 7ff63e3f0740 4 client.17535 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:58:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.599+0000 7ff63e3f0740 1 -- 172.21.15.28:0/616946526 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556d26567790 con 0x556d2667e030 2024-04-21T08:58:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.599+0000 7ff638cc7640 1 --2- 172.21.15.28:0/616946526 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556d2667e030 0x556d26680450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.599+0000 7ff61cff9640 10 client.17535 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.599+0000 7ff638cc7640 1 --2- 172.21.15.28:0/616946526 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556d2667e030 0x556d26680450 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=953b0d395ad8a3bd server_cookie=ab0cca257d265bfc in_seq=0 out_seq=0 2024-04-21T08:58:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.601+0000 7ff61cff9640 1 -- 172.21.15.28:0/616946526 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556d26567790 con 0x556d2667e030 2024-04-21T08:58:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.601+0000 7ff61cff9640 10 client.17535 handle_command_reply: tid=0 2024-04-21T08:58:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.601+0000 7ff63e3f0740 1 client.17535 shutdown 2024-04-21T08:58:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.601+0000 7ff63e3f0740 1 -- 172.21.15.28:0/616946526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 msgr2=0x556d2660b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.601+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/616946526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d2660b370 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff63000aa10 tx=0x7ff630005a80 comp rx=0 tx=0).stop 2024-04-21T08:58:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.601+0000 7ff63e3f0740 1 -- 172.21.15.28:0/616946526 shutdown_connections 2024-04-21T08:58:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.601+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/616946526 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556d2667e030 0x556d26680450 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.601+0000 7ff61cff9640 0 client.17535 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.601+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/616946526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556d26607020 0x556d2660b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.601+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/616946526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d265fe4a0 0x556d2660b370 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.601+0000 7ff63e3f0740 1 --2- 172.21.15.28:0/616946526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556d265fdaf0 0x556d2660ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.601+0000 7ff63e3f0740 1 -- 172.21.15.28:0/616946526 >> 172.21.15.28:0/616946526 conn(0x556d265f81c0 msgr2=0x556d265f5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.601+0000 7ff63e3f0740 1 -- 172.21.15.28:0/616946526 shutdown_connections 2024-04-21T08:58:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.601+0000 7ff63e3f0740 1 -- 172.21.15.28:0/616946526 wait complete. 2024-04-21T08:58:09.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.601+0000 7ff63e3f0740 20 client.17535 trim_cache size 0 max 16384 2024-04-21T08:58:09.610 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:58:09.610 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:58:09.610 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "f05e0590-b012-4ae6-88e7-35152f074273", 2024-04-21T08:58:09.610 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:58:09.611 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:58:09.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.602+0000 7ff63b72c640 1 -- 172.21.15.28:0/3165829421 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6200a3d20 msgr2=0x7ff6200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.602+0000 7ff63b72c640 1 --2- 172.21.15.28:0/3165829421 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6200a3d20 0x7ff6200a61e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff62c0ce020 tx=0x7ff62c091c50 comp rx=0 tx=0).stop 2024-04-21T08:58:09.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.602+0000 7ff63b72c640 1 -- 172.21.15.28:0/3165829421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff634107030 msgr2=0x7ff63419bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.602+0000 7ff63b72c640 1 --2- 172.21.15.28:0/3165829421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff634107030 0x7ff63419bef0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00c970 tx=0x7ff62c00ce30 comp rx=0 tx=0).stop 2024-04-21T08:58:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.602+0000 7ff63b72c640 1 -- 172.21.15.28:0/3165829421 shutdown_connections 2024-04-21T08:58:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.602+0000 7ff63b72c640 1 --2- 172.21.15.28:0/3165829421 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ff6200a3d20 0x7ff6200a61e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.602+0000 7ff63b72c640 1 --2- 172.21.15.28:0/3165829421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff634107030 0x7ff63419bef0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.602+0000 7ff63b72c640 1 --2- 172.21.15.28:0/3165829421 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff634101b40 0x7ff634197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.602+0000 7ff63b72c640 1 --2- 172.21.15.28:0/3165829421 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff6341011a0 0x7ff634197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.602+0000 7ff63b72c640 1 -- 172.21.15.28:0/3165829421 >> 172.21.15.28:0/3165829421 conn(0x7ff634076c50 msgr2=0x7ff6340ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.602+0000 7ff63b72c640 1 -- 172.21.15.28:0/3165829421 shutdown_connections 2024-04-21T08:58:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.602+0000 7ff63b72c640 1 -- 172.21.15.28:0/3165829421 wait complete. 2024-04-21T08:58:09.614 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:58:09.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:58:09.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.722+0000 7fe05d946640 1 -- 172.21.15.28:0/2192578099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe058103ad0 msgr2=0x7fe05810bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.722+0000 7fe05d946640 1 --2- 172.21.15.28:0/2192578099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe058103ad0 0x7fe05810bfb0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fe040009900 tx=0x7fe04002bcd0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.722+0000 7fe05d946640 1 -- 172.21.15.28:0/2192578099 shutdown_connections 2024-04-21T08:58:09.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.722+0000 7fe05d946640 1 --2- 172.21.15.28:0/2192578099 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe05810c580 0x7fe05810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.722+0000 7fe05d946640 1 --2- 172.21.15.28:0/2192578099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe058103ad0 0x7fe05810bfb0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.722+0000 7fe05d946640 1 --2- 172.21.15.28:0/2192578099 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0581031b0 0x7fe058103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.722+0000 7fe05d946640 1 -- 172.21.15.28:0/2192578099 >> 172.21.15.28:0/2192578099 conn(0x7fe0580771d0 msgr2=0x7fe0580775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:09.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.722+0000 7fe05d946640 1 -- 172.21.15.28:0/2192578099 shutdown_connections 2024-04-21T08:58:09.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.722+0000 7fe05d946640 1 -- 172.21.15.28:0/2192578099 wait complete. 2024-04-21T08:58:09.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe05d946640 1 Processor -- start 2024-04-21T08:58:09.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe05d946640 1 -- start start 2024-04-21T08:58:09.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe05d946640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0581031b0 0x7fe058198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe05d946640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe058103ad0 0x7fe058198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe05d946640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe05810c580 0x7fe05819d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe05d946640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0581121a0 con 0x7fe058103ad0 2024-04-21T08:58:09.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe05d946640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe058112020 con 0x7fe05810c580 2024-04-21T08:58:09.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe05d946640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe058112320 con 0x7fe0581031b0 2024-04-21T08:58:09.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe04ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe058103ad0 0x7fe058198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe04ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe058103ad0 0x7fe058198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51942/0 (socket says 172.21.15.28:51942) 2024-04-21T08:58:09.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe04ffff640 1 -- 172.21.15.28:0/760625717 learned_addr learned my addr 172.21.15.28:0/760625717 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:09.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe0577fe640 1 --2- 172.21.15.28:0/760625717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe05810c580 0x7fe05819d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe056ffd640 1 --2- 172.21.15.28:0/760625717 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0581031b0 0x7fe058198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe04ffff640 1 -- 172.21.15.28:0/760625717 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0581031b0 msgr2=0x7fe058198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe04ffff640 1 --2- 172.21.15.28:0/760625717 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0581031b0 0x7fe058198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe04ffff640 1 -- 172.21.15.28:0/760625717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe05810c580 msgr2=0x7fe05819d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe04ffff640 1 --2- 172.21.15.28:0/760625717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe05810c580 0x7fe05819d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.723+0000 7fe04ffff640 1 -- 172.21.15.28:0/760625717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe05819d750 con 0x7fe058103ad0 2024-04-21T08:58:09.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.724+0000 7fe04ffff640 1 --2- 172.21.15.28:0/760625717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe058103ad0 0x7fe058198ca0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fe040005940 tx=0x7fe04002d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:09.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.724+0000 7fe05c944640 1 -- 172.21.15.28:0/760625717 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe040043020 con 0x7fe058103ad0 2024-04-21T08:58:09.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.724+0000 7fe05c944640 1 -- 172.21.15.28:0/760625717 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe04002dc90 con 0x7fe058103ad0 2024-04-21T08:58:09.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.724+0000 7fe05d946640 1 -- 172.21.15.28:0/760625717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe05819d9e0 con 0x7fe058103ad0 2024-04-21T08:58:09.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.724+0000 7fe05c944640 1 -- 172.21.15.28:0/760625717 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe040003ab0 con 0x7fe058103ad0 2024-04-21T08:58:09.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.724+0000 7fe05d946640 1 -- 172.21.15.28:0/760625717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0581a52d0 con 0x7fe058103ad0 2024-04-21T08:58:09.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.726+0000 7fe05c944640 1 -- 172.21.15.28:0/760625717 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe040046020 con 0x7fe058103ad0 2024-04-21T08:58:09.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.726+0000 7fe06060c740 1 Processor -- start 2024-04-21T08:58:09.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.726+0000 7fe06060c740 1 -- start start 2024-04-21T08:58:09.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.726+0000 7fe06060c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5179580 0x55c3a5179960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.726+0000 7fe06060c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179ea0 0x55c3a5182540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.726+0000 7fe06060c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5182a80 0x55c3a5184e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.726+0000 7fe06060c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c3a50de1e0 con 0x55c3a5179580 2024-04-21T08:58:09.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.726+0000 7fe06060c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c3a507b390 con 0x55c3a5182a80 2024-04-21T08:58:09.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.726+0000 7fe06060c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55c3a50da1a0 con 0x55c3a5179ea0 2024-04-21T08:58:09.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe0577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5179580 0x55c3a5179960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe0577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5179580 0x55c3a5179960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51958/0 (socket says 172.21.15.28:51958) 2024-04-21T08:58:09.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe0577fe640 1 -- 172.21.15.28:0/2531007881 learned_addr learned my addr 172.21.15.28:0/2531007881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:09.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe04ffff640 1 --2- 172.21.15.28:0/2531007881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5182a80 0x55c3a5184e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe056ffd640 1 --2- 172.21.15.28:0/2531007881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179ea0 0x55c3a5182540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe05c944640 1 --2- 172.21.15.28:0/760625717 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0440a3da0 0x7fe0440a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe0577fe640 1 -- 172.21.15.28:0/2531007881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179ea0 msgr2=0x55c3a5182540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe0577fe640 1 --2- 172.21.15.28:0/2531007881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179ea0 0x55c3a5182540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe0577fe640 1 -- 172.21.15.28:0/2531007881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5182a80 msgr2=0x55c3a5184e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe0577fe640 1 --2- 172.21.15.28:0/2531007881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5182a80 0x55c3a5184e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe0577fe640 1 -- 172.21.15.28:0/2531007881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3a4eba7d0 con 0x55c3a5179580 2024-04-21T08:58:09.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe056ffd640 1 --2- 172.21.15.28:0/2531007881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179ea0 0x55c3a5182540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:09.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe0577fe640 1 --2- 172.21.15.28:0/760625717 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0440a3da0 0x7fe0440a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe05c944640 1 -- 172.21.15.28:0/760625717 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7fe0400f2c90 con 0x7fe058103ad0 2024-04-21T08:58:09.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe0577fe640 1 --2- 172.21.15.28:0/2531007881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5179580 0x55c3a5179960 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fe050010470 tx=0x7fe050010930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb3d96f924347ad server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:09.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe04dffb640 1 -- 172.21.15.28:0/2531007881 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05001d400 con 0x55c3a5179580 2024-04-21T08:58:09.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe04dffb640 1 -- 172.21.15.28:0/2531007881 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe05001da50 con 0x55c3a5179580 2024-04-21T08:58:09.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.727+0000 7fe04dffb640 1 -- 172.21.15.28:0/2531007881 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05001c700 con 0x55c3a5179580 2024-04-21T08:58:09.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.728+0000 7fe06060c740 1 -- 172.21.15.28:0/2531007881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5179580 msgr2=0x55c3a5179960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.728+0000 7fe0577fe640 1 --2- 172.21.15.28:0/760625717 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0440a3da0 0x7fe0440a6260 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe050021000 tx=0x7fe05001e000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:09.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.728+0000 7fe06060c740 1 --2- 172.21.15.28:0/2531007881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5179580 0x55c3a5179960 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fe050010470 tx=0x7fe050010930 comp rx=0 tx=0).stop 2024-04-21T08:58:09.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.728+0000 7fe06060c740 1 -- 172.21.15.28:0/2531007881 shutdown_connections 2024-04-21T08:58:09.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.728+0000 7fe06060c740 1 --2- 172.21.15.28:0/2531007881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5182a80 0x55c3a5184e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.728+0000 7fe06060c740 1 --2- 172.21.15.28:0/2531007881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179ea0 0x55c3a5182540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.728+0000 7fe06060c740 1 --2- 172.21.15.28:0/2531007881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5179580 0x55c3a5179960 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.728+0000 7fe06060c740 1 -- 172.21.15.28:0/2531007881 >> 172.21.15.28:0/2531007881 conn(0x55c3a5173cb0 msgr2=0x55c3a5174090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:09.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.728+0000 7fe06060c740 1 -- 172.21.15.28:0/2531007881 shutdown_connections 2024-04-21T08:58:09.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.728+0000 7fe06060c740 1 -- 172.21.15.28:0/2531007881 wait complete. 2024-04-21T08:58:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.728+0000 7fe06060c740 1 -- start start 2024-04-21T08:58:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe06060c740 1 Processor -- start 2024-04-21T08:58:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe06060c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179580 0x55c3a517faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe06060c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5179ea0 0x55c3a517ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe06060c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5182a80 0x55c3a5180520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe06060c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c3a5185870 con 0x55c3a5182a80 2024-04-21T08:58:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe06060c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55c3a50972f0 con 0x55c3a5179ea0 2024-04-21T08:58:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe06060c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55c3a51859f0 con 0x55c3a5179580 2024-04-21T08:58:09.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe06060c740 10 client.0 _ll_register_callbacks cb 0x55c3a50a4580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:09.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe0577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5182a80 0x55c3a5180520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe0577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5182a80 0x55c3a5180520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51974/0 (socket says 172.21.15.28:51974) 2024-04-21T08:58:09.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe0577fe640 1 -- 172.21.15.28:0/192231512 learned_addr learned my addr 172.21.15.28:0/192231512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:09.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe04ffff640 1 --2- 172.21.15.28:0/192231512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5179ea0 0x55c3a517ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe056ffd640 1 --2- 172.21.15.28:0/192231512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179580 0x55c3a517faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe0577fe640 1 -- 172.21.15.28:0/192231512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179580 msgr2=0x55c3a517faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe0577fe640 1 --2- 172.21.15.28:0/192231512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179580 0x55c3a517faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe0577fe640 1 -- 172.21.15.28:0/192231512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5179ea0 msgr2=0x55c3a517ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe0577fe640 1 --2- 172.21.15.28:0/192231512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5179ea0 0x55c3a517ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.729+0000 7fe0577fe640 1 -- 172.21.15.28:0/192231512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3a4df03f0 con 0x55c3a5182a80 2024-04-21T08:58:09.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe0577fe640 1 --2- 172.21.15.28:0/192231512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5182a80 0x55c3a5180520 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fe0500135a0 tx=0x7fe050005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d6d97515627765 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:09.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe036ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:09.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe036ffd640 1 -- 172.21.15.28:0/192231512 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe050036060 con 0x55c3a5182a80 2024-04-21T08:58:09.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe036ffd640 1 -- 172.21.15.28:0/192231512 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7fe05001ca60 con 0x55c3a5182a80 2024-04-21T08:58:09.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe06060c740 1 -- 172.21.15.28:0/192231512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe05001da50 con 0x55c3a5182a80 2024-04-21T08:58:09.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe036ffd640 1 -- 172.21.15.28:0/192231512 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe05001cc60 con 0x55c3a5182a80 2024-04-21T08:58:09.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe06060c740 10 client.17547 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:09.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe06060c740 1 -- 172.21.15.28:0/192231512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c3a4db00f0 con 0x55c3a5182a80 2024-04-21T08:58:09.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe036ffd640 1 -- 172.21.15.28:0/192231512 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe05001da50 con 0x55c3a5182a80 2024-04-21T08:58:09.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe036ffd640 1 -- 172.21.15.28:0/192231512 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe05000f040 con 0x55c3a5182a80 2024-04-21T08:58:09.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe06060c740 10 client.17547 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:09.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe06060c740 10 client.17547 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:09.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe06060c740 20 client.17547 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:09.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe06060c740 1 --2- 172.21.15.28:0/192231512 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c3a5192a80 0x55c3a5194e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe06060c740 4 client.17547 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:58:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.730+0000 7fe06060c740 1 -- 172.21.15.28:0/192231512 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3a50ebc00 con 0x55c3a5192a80 2024-04-21T08:58:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.731+0000 7fe056ffd640 1 --2- 172.21.15.28:0/192231512 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c3a5192a80 0x55c3a5194e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.731+0000 7fe056ffd640 1 --2- 172.21.15.28:0/192231512 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c3a5192a80 0x55c3a5194e70 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dbcbc744cffcffc server_cookie=aa94ca0c72698947 in_seq=0 out_seq=0 2024-04-21T08:58:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.731+0000 7fe036ffd640 10 client.17547 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.731+0000 7fe036ffd640 1 -- 172.21.15.28:0/192231512 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c3a50ebc00 con 0x55c3a5192a80 2024-04-21T08:58:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.732+0000 7fe036ffd640 10 client.17547 handle_command_reply: tid=0 2024-04-21T08:58:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.732+0000 7fe06060c740 1 client.17547 shutdown 2024-04-21T08:58:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.732+0000 7fe06060c740 1 -- 172.21.15.28:0/192231512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5182a80 msgr2=0x55c3a5180520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.732+0000 7fe06060c740 1 --2- 172.21.15.28:0/192231512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5182a80 0x55c3a5180520 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fe0500135a0 tx=0x7fe050005f10 comp rx=0 tx=0).stop 2024-04-21T08:58:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.732+0000 7fe06060c740 1 -- 172.21.15.28:0/192231512 shutdown_connections 2024-04-21T08:58:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.732+0000 7fe06060c740 1 --2- 172.21.15.28:0/192231512 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c3a5192a80 0x55c3a5194e70 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.732+0000 7fe06060c740 1 --2- 172.21.15.28:0/192231512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5182a80 0x55c3a5180520 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.732+0000 7fe06060c740 1 --2- 172.21.15.28:0/192231512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5179ea0 0x55c3a517ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.732+0000 7fe06060c740 1 --2- 172.21.15.28:0/192231512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179580 0x55c3a517faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.732+0000 7fe036ffd640 0 client.17547 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.732+0000 7fe06060c740 1 -- 172.21.15.28:0/192231512 >> 172.21.15.28:0/192231512 conn(0x55c3a5173cb0 msgr2=0x55c3a5175a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.732+0000 7fe06060c740 1 -- 172.21.15.28:0/192231512 shutdown_connections 2024-04-21T08:58:09.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.732+0000 7fe06060c740 1 -- 172.21.15.28:0/192231512 wait complete. 2024-04-21T08:58:09.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.732+0000 7fe06060c740 20 client.17547 trim_cache size 0 max 16384 2024-04-21T08:58:09.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.749+0000 7fe06060c740 1 Processor -- start 2024-04-21T08:58:09.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.749+0000 7fe06060c740 1 -- start start 2024-04-21T08:58:09.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe06060c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5179580 0x55c3a5182160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe06060c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179ea0 0x55c3a518e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe06060c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5182a80 0x55c3a519a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe06060c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0581b1ea0 con 0x55c3a5179580 2024-04-21T08:58:09.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe06060c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe0581af770 con 0x55c3a5182a80 2024-04-21T08:58:09.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe06060c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe0581ad5e0 con 0x55c3a5179ea0 2024-04-21T08:58:09.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe056ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5179580 0x55c3a5182160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe056ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5179580 0x55c3a5182160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51990/0 (socket says 172.21.15.28:51990) 2024-04-21T08:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe056ffd640 1 -- 172.21.15.28:0/3876622610 learned_addr learned my addr 172.21.15.28:0/3876622610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe04ffff640 1 --2- 172.21.15.28:0/3876622610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179ea0 0x55c3a518e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe056ffd640 1 -- 172.21.15.28:0/3876622610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179ea0 msgr2=0x55c3a518e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe056ffd640 1 --2- 172.21.15.28:0/3876622610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179ea0 0x55c3a518e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe056ffd640 1 -- 172.21.15.28:0/3876622610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5182a80 msgr2=0x55c3a519a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe056ffd640 1 --2- 172.21.15.28:0/3876622610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5182a80 0x55c3a519a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe056ffd640 1 -- 172.21.15.28:0/3876622610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3a517b5f0 con 0x55c3a5179580 2024-04-21T08:58:09.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe04ffff640 1 --2- 172.21.15.28:0/3876622610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179ea0 0x55c3a518e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:09.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe056ffd640 1 --2- 172.21.15.28:0/3876622610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5179580 0x55c3a5182160 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fe04800ec00 tx=0x7fe04800e640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac1bc981f8453bf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:09.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe04d7fa640 1 -- 172.21.15.28:0/3876622610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe048013b10 con 0x55c3a5179580 2024-04-21T08:58:09.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe04d7fa640 1 -- 172.21.15.28:0/3876622610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe048013cb0 con 0x55c3a5179580 2024-04-21T08:58:09.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.750+0000 7fe04d7fa640 1 -- 172.21.15.28:0/3876622610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0480035f0 con 0x55c3a5179580 2024-04-21T08:58:09.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.751+0000 7fe06060c740 1 -- 172.21.15.28:0/3876622610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5179580 msgr2=0x55c3a5182160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.751+0000 7fe06060c740 1 --2- 172.21.15.28:0/3876622610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5179580 0x55c3a5182160 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fe04800ec00 tx=0x7fe04800e640 comp rx=0 tx=0).stop 2024-04-21T08:58:09.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.751+0000 7fe06060c740 1 -- 172.21.15.28:0/3876622610 shutdown_connections 2024-04-21T08:58:09.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.751+0000 7fe06060c740 1 --2- 172.21.15.28:0/3876622610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5182a80 0x55c3a519a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.751+0000 7fe06060c740 1 --2- 172.21.15.28:0/3876622610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179ea0 0x55c3a518e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.751+0000 7fe06060c740 1 --2- 172.21.15.28:0/3876622610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5179580 0x55c3a5182160 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.751+0000 7fe06060c740 1 -- 172.21.15.28:0/3876622610 >> 172.21.15.28:0/3876622610 conn(0x55c3a5173cb0 msgr2=0x55c3a517e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:09.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.751+0000 7fe06060c740 1 -- 172.21.15.28:0/3876622610 shutdown_connections 2024-04-21T08:58:09.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.751+0000 7fe06060c740 1 -- 172.21.15.28:0/3876622610 wait complete. 2024-04-21T08:58:09.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.751+0000 7fe06060c740 1 -- start start 2024-04-21T08:58:09.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.751+0000 7fe06060c740 1 Processor -- start 2024-04-21T08:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe06060c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179580 0x55c3a518acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe06060c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5179ea0 0x55c3a5186040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe06060c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5182a80 0x55c3a5186580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe06060c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0581adc30 con 0x55c3a5182a80 2024-04-21T08:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe06060c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe0581af0a0 con 0x55c3a5179ea0 2024-04-21T08:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe06060c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe0581ae300 con 0x55c3a5179580 2024-04-21T08:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe06060c740 10 client.0 _ll_register_callbacks cb 0x7fe050020020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe0577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5182a80 0x55c3a5186580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe0577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5182a80 0x55c3a5186580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52006/0 (socket says 172.21.15.28:52006) 2024-04-21T08:58:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe0577fe640 1 -- 172.21.15.28:0/2652527461 learned_addr learned my addr 172.21.15.28:0/2652527461 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe04ffff640 1 --2- 172.21.15.28:0/2652527461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5179ea0 0x55c3a5186040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe056ffd640 1 --2- 172.21.15.28:0/2652527461 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179580 0x55c3a518acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe0577fe640 1 -- 172.21.15.28:0/2652527461 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179580 msgr2=0x55c3a518acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe0577fe640 1 --2- 172.21.15.28:0/2652527461 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179580 0x55c3a518acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe0577fe640 1 -- 172.21.15.28:0/2652527461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5179ea0 msgr2=0x55c3a5186040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe0577fe640 1 --2- 172.21.15.28:0/2652527461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5179ea0 0x55c3a5186040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe0577fe640 1 -- 172.21.15.28:0/2652527461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3a5186ac0 con 0x55c3a5182a80 2024-04-21T08:58:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe04ffff640 1 --2- 172.21.15.28:0/2652527461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5179ea0 0x55c3a5186040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:58:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe0577fe640 1 --2- 172.21.15.28:0/2652527461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5182a80 0x55c3a5186580 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fe05001c690 tx=0x7fe0500134c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22065d44179167db server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe036ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.752+0000 7fe036ffd640 1 -- 172.21.15.28:0/2652527461 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe050036070 con 0x55c3a5182a80 2024-04-21T08:58:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.753+0000 7fe036ffd640 1 -- 172.21.15.28:0/2652527461 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9621+0+0 (secure 0 0 0) 0x7fe05001ca60 con 0x55c3a5182a80 2024-04-21T08:58:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.753+0000 7fe06060c740 1 -- 172.21.15.28:0/2652527461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe048013cb0 con 0x55c3a5182a80 2024-04-21T08:58:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.753+0000 7fe036ffd640 1 -- 172.21.15.28:0/2652527461 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe050049050 con 0x55c3a5182a80 2024-04-21T08:58:09.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.753+0000 7fe06060c740 10 client.17553 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:09.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.753+0000 7fe06060c740 1 -- 172.21.15.28:0/2652527461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c3a51f8a80 con 0x55c3a5182a80 2024-04-21T08:58:09.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.753+0000 7fe036ffd640 1 -- 172.21.15.28:0/2652527461 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe048013cb0 con 0x55c3a5182a80 2024-04-21T08:58:09.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.753+0000 7fe036ffd640 1 -- 172.21.15.28:0/2652527461 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe05000f040 con 0x55c3a5182a80 2024-04-21T08:58:09.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.753+0000 7fe06060c740 10 client.17553 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:09.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.753+0000 7fe06060c740 10 client.17553 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:09.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.753+0000 7fe06060c740 20 client.17553 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:09.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.753+0000 7fe06060c740 1 --2- 172.21.15.28:0/2652527461 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c3a51f9110 0x55c3a51fb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:09.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.753+0000 7fe06060c740 4 client.17553 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:58:09.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.753+0000 7fe06060c740 1 -- 172.21.15.28:0/2652527461 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c3a50e5a70 con 0x55c3a51f9110 2024-04-21T08:58:09.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.753+0000 7fe056ffd640 1 --2- 172.21.15.28:0/2652527461 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c3a51f9110 0x55c3a51fb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:09.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe036ffd640 10 client.17553 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:09.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe056ffd640 1 --2- 172.21.15.28:0/2652527461 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c3a51f9110 0x55c3a51fb530 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c7a4b3bcfdbdf65 server_cookie=72923c9873494b59 in_seq=0 out_seq=0 2024-04-21T08:58:09.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe036ffd640 1 -- 172.21.15.28:0/2652527461 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c3a50e5a70 con 0x55c3a51f9110 2024-04-21T08:58:09.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe036ffd640 10 client.17553 handle_command_reply: tid=0 2024-04-21T08:58:09.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe06060c740 1 client.17553 shutdown 2024-04-21T08:58:09.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe06060c740 1 -- 172.21.15.28:0/2652527461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5182a80 msgr2=0x55c3a5186580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe06060c740 1 --2- 172.21.15.28:0/2652527461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5182a80 0x55c3a5186580 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fe05001c690 tx=0x7fe0500134c0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe06060c740 1 -- 172.21.15.28:0/2652527461 shutdown_connections 2024-04-21T08:58:09.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe06060c740 1 --2- 172.21.15.28:0/2652527461 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55c3a51f9110 0x55c3a51fb530 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe06060c740 1 --2- 172.21.15.28:0/2652527461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3a5182a80 0x55c3a5186580 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe06060c740 1 --2- 172.21.15.28:0/2652527461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c3a5179ea0 0x55c3a5186040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe036ffd640 0 client.17553 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:09.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe06060c740 1 --2- 172.21.15.28:0/2652527461 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55c3a5179580 0x55c3a518acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe06060c740 1 -- 172.21.15.28:0/2652527461 >> 172.21.15.28:0/2652527461 conn(0x55c3a5173cb0 msgr2=0x55c3a5175100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:09.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe06060c740 1 -- 172.21.15.28:0/2652527461 shutdown_connections 2024-04-21T08:58:09.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe06060c740 1 -- 172.21.15.28:0/2652527461 wait complete. 2024-04-21T08:58:09.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.754+0000 7fe06060c740 20 client.17553 trim_cache size 0 max 16384 2024-04-21T08:58:09.765 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:58:09.765 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:58:09.765 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:58:09.765 INFO:teuthology.orchestra.run.smithi028.stdout: "f05e0590-b012-4ae6-88e7-35152f074273": { 2024-04-21T08:58:09.765 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:58:09.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "f05e0590-b012-4ae6-88e7-35152f074273", 2024-04-21T08:58:09.766 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:58:09.766 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:58:09.766 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:58:09.766 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:58:09.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.755+0000 7fe05d946640 1 -- 172.21.15.28:0/760625717 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0440a3da0 msgr2=0x7fe0440a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.755+0000 7fe05d946640 1 --2- 172.21.15.28:0/760625717 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0440a3da0 0x7fe0440a6260 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe050021000 tx=0x7fe05001e000 comp rx=0 tx=0).stop 2024-04-21T08:58:09.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.755+0000 7fe05d946640 1 -- 172.21.15.28:0/760625717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe058103ad0 msgr2=0x7fe058198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:09.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.755+0000 7fe05d946640 1 --2- 172.21.15.28:0/760625717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe058103ad0 0x7fe058198ca0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fe040005940 tx=0x7fe04002d890 comp rx=0 tx=0).stop 2024-04-21T08:58:09.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.755+0000 7fe05d946640 1 -- 172.21.15.28:0/760625717 shutdown_connections 2024-04-21T08:58:09.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.755+0000 7fe05d946640 1 --2- 172.21.15.28:0/760625717 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe0440a3da0 0x7fe0440a6260 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.755+0000 7fe05d946640 1 --2- 172.21.15.28:0/760625717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe05810c580 0x7fe05819d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.755+0000 7fe05d946640 1 --2- 172.21.15.28:0/760625717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe058103ad0 0x7fe058198ca0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.755+0000 7fe05d946640 1 --2- 172.21.15.28:0/760625717 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0581031b0 0x7fe058198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:09.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.755+0000 7fe05d946640 1 -- 172.21.15.28:0/760625717 >> 172.21.15.28:0/760625717 conn(0x7fe0580771d0 msgr2=0x7fe0580fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:09.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.755+0000 7fe05d946640 1 -- 172.21.15.28:0/760625717 shutdown_connections 2024-04-21T08:58:09.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:09.755+0000 7fe05d946640 1 -- 172.21.15.28:0/760625717 wait complete. 2024-04-21T08:58:09.769 INFO:tasks.cephfs.filesystem:scrub status for tag:f05e0590-b012-4ae6-88e7-35152f074273 - {'path': '/', 'tag': 'f05e0590-b012-4ae6-88e7-35152f074273', 'options': 'recursive,force'} 2024-04-21T08:58:10.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: pgmap v213: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-21T08:58:10.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:58:10.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:10.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:10.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:10.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:10.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:10.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:10.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:10 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: pgmap v213: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-21T08:58:10.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:58:10.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:10.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:10.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:10.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:10.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:10.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:10.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:10.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:10.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:10.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:10.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:10 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: pgmap v213: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-21T08:58:10.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:58:10.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:10.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:10.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:10.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:10.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:10.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:10.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:10.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:10.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:10.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:10.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:10.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:10 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:11.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:11 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:11.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:11 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:11.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:11 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:12.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:12 smithi152 ceph-mon[31401]: pgmap v214: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-21T08:58:12.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:12 smithi028 ceph-mon[28003]: pgmap v214: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-21T08:58:12.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:12 smithi171 ceph-mon[31358]: pgmap v214: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-21T08:58:14.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:14 smithi152 ceph-mon[31401]: pgmap v215: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 38 op/s 2024-04-21T08:58:14.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:14 smithi152 ceph-mon[31401]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T08:58:14.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:14 smithi028 ceph-mon[28003]: pgmap v215: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 38 op/s 2024-04-21T08:58:14.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:14 smithi028 ceph-mon[28003]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T08:58:14.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:14 smithi171 ceph-mon[31358]: pgmap v215: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 38 op/s 2024-04-21T08:58:14.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:14 smithi171 ceph-mon[31358]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T08:58:16.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:16 smithi152 ceph-mon[31401]: pgmap v217: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 255 B/s wr, 40 op/s 2024-04-21T08:58:16.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:16 smithi028 ceph-mon[28003]: pgmap v217: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 255 B/s wr, 40 op/s 2024-04-21T08:58:16.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:16 smithi171 ceph-mon[31358]: pgmap v217: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 255 B/s wr, 40 op/s 2024-04-21T08:58:17.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:17 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:58:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:17 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:58:17.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:17 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:58:17.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:17 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:58:17.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:17 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:58:17.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:17 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:58:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:17 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:58:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:17 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:58:17.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:17 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:58:18.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:18 smithi152 ceph-mon[31401]: pgmap v218: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 105 B/s wr, 34 op/s 2024-04-21T08:58:18.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:18 smithi028 ceph-mon[28003]: pgmap v218: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 105 B/s wr, 34 op/s 2024-04-21T08:58:18.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:18 smithi171 ceph-mon[31358]: pgmap v218: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 105 B/s wr, 34 op/s 2024-04-21T08:58:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:20 smithi171 ceph-mon[31358]: pgmap v219: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 102 B/s wr, 33 op/s 2024-04-21T08:58:20.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:20 smithi152 ceph-mon[31401]: pgmap v219: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 102 B/s wr, 33 op/s 2024-04-21T08:58:20.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:20 smithi028 ceph-mon[28003]: pgmap v219: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 102 B/s wr, 33 op/s 2024-04-21T08:58:22.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:22 smithi171 ceph-mon[31358]: pgmap v220: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 102 B/s wr, 30 op/s 2024-04-21T08:58:22.657 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:22 smithi152 ceph-mon[31401]: pgmap v220: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 102 B/s wr, 30 op/s 2024-04-21T08:58:22.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:22 smithi028 ceph-mon[28003]: pgmap v220: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 102 B/s wr, 30 op/s 2024-04-21T08:58:24.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:24 smithi171 ceph-mon[31358]: pgmap v221: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 op/s 2024-04-21T08:58:24.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:24 smithi152 ceph-mon[31401]: pgmap v221: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 op/s 2024-04-21T08:58:24.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:24 smithi028 ceph-mon[28003]: pgmap v221: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 op/s 2024-04-21T08:58:26.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:26 smithi171 ceph-mon[31358]: pgmap v222: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 7 op/s 2024-04-21T08:58:26.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:26 smithi152 ceph-mon[31401]: pgmap v222: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 7 op/s 2024-04-21T08:58:26.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:26 smithi028 ceph-mon[28003]: pgmap v222: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 7 op/s 2024-04-21T08:58:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:28 smithi171 ceph-mon[31358]: pgmap v223: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:58:28.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:28 smithi028 ceph-mon[28003]: pgmap v223: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:58:28.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:28 smithi152 ceph-mon[31401]: pgmap v223: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:58:30.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:30 smithi152 ceph-mon[31401]: pgmap v224: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:58:30.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:30 smithi028 ceph-mon[28003]: pgmap v224: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:58:30.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:30 smithi171 ceph-mon[31358]: pgmap v224: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:58:32.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:32 smithi152 ceph-mon[31401]: pgmap v225: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:58:32.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:32 smithi028 ceph-mon[28003]: pgmap v225: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:58:32.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:32 smithi171 ceph-mon[31358]: pgmap v225: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:58:34.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:34 smithi152 ceph-mon[31401]: pgmap v226: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:58:34.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:34 smithi028 ceph-mon[28003]: pgmap v226: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:58:34.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:34 smithi171 ceph-mon[31358]: pgmap v226: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:58:36.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:36 smithi152 ceph-mon[31401]: pgmap v227: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:58:36.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:36 smithi028 ceph-mon[28003]: pgmap v227: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:58:36.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:36 smithi171 ceph-mon[31358]: pgmap v227: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:58:38.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:38 smithi152 ceph-mon[31401]: pgmap v228: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:58:38.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:38 smithi028 ceph-mon[28003]: pgmap v228: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:58:38.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:38 smithi171 ceph-mon[31358]: pgmap v228: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:58:39.770 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:58:39.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:58:39.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.889+0000 7f2de2967640 1 -- 172.21.15.28:0/4077616411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ddc101080 msgr2=0x7f2ddc101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:39.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.889+0000 7f2de2967640 1 --2- 172.21.15.28:0/4077616411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ddc101080 0x7f2ddc101460 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009900 tx=0x7f2dcc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:58:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.889+0000 7f2de2967640 1 -- 172.21.15.28:0/4077616411 shutdown_connections 2024-04-21T08:58:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.889+0000 7f2de2967640 1 --2- 172.21.15.28:0/4077616411 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ddc106150 0x7f2ddc10dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.889+0000 7f2de2967640 1 --2- 172.21.15.28:0/4077616411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ddc101a30 0x7f2ddc105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.889+0000 7f2de2967640 1 --2- 172.21.15.28:0/4077616411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ddc101080 0x7f2ddc101460 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.889+0000 7f2de2967640 1 -- 172.21.15.28:0/4077616411 >> 172.21.15.28:0/4077616411 conn(0x7f2ddc078000 msgr2=0x7f2ddc078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.889+0000 7f2de2967640 1 -- 172.21.15.28:0/4077616411 shutdown_connections 2024-04-21T08:58:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.889+0000 7f2de2967640 1 -- 172.21.15.28:0/4077616411 wait complete. 2024-04-21T08:58:39.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.889+0000 7f2de2967640 1 Processor -- start 2024-04-21T08:58:39.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2de2967640 1 -- start start 2024-04-21T08:58:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2de2967640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ddc101080 0x7f2ddc102b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2de2967640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ddc106150 0x7f2ddc103050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2de2967640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ddc103590 0x7f2ddc1a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2de2967640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ddc111110 con 0x7f2ddc101080 2024-04-21T08:58:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2de2967640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2ddc110f90 con 0x7f2ddc103590 2024-04-21T08:58:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2de2967640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2ddc111290 con 0x7f2ddc106150 2024-04-21T08:58:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2ddbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ddc101080 0x7f2ddc102b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2ddbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ddc101080 0x7f2ddc102b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33426/0 (socket says 172.21.15.28:33426) 2024-04-21T08:58:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2ddbfff640 1 -- 172.21.15.28:0/1524179711 learned_addr learned my addr 172.21.15.28:0/1524179711 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2ddb7fe640 1 --2- 172.21.15.28:0/1524179711 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ddc106150 0x7f2ddc103050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2de0f04640 1 --2- 172.21.15.28:0/1524179711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ddc103590 0x7f2ddc1a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2ddbfff640 1 -- 172.21.15.28:0/1524179711 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ddc106150 msgr2=0x7f2ddc103050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2ddbfff640 1 --2- 172.21.15.28:0/1524179711 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ddc106150 0x7f2ddc103050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2ddbfff640 1 -- 172.21.15.28:0/1524179711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ddc103590 msgr2=0x7f2ddc1a6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2ddbfff640 1 --2- 172.21.15.28:0/1524179711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ddc103590 0x7f2ddc1a6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.890+0000 7f2ddbfff640 1 -- 172.21.15.28:0/1524179711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ddc1a7010 con 0x7f2ddc101080 2024-04-21T08:58:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.891+0000 7f2ddbfff640 1 --2- 172.21.15.28:0/1524179711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ddc101080 0x7f2ddc102b10 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc000c00 tx=0x7f2dcc02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.891+0000 7f2dd97fa640 1 -- 172.21.15.28:0/1524179711 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc043020 con 0x7f2ddc101080 2024-04-21T08:58:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.891+0000 7f2dd97fa640 1 -- 172.21.15.28:0/1524179711 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dcc0052b0 con 0x7f2ddc101080 2024-04-21T08:58:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.891+0000 7f2de2967640 1 -- 172.21.15.28:0/1524179711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ddc1a72a0 con 0x7f2ddc101080 2024-04-21T08:58:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.891+0000 7f2dd97fa640 1 -- 172.21.15.28:0/1524179711 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc003c60 con 0x7f2ddc101080 2024-04-21T08:58:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.891+0000 7f2de2967640 1 -- 172.21.15.28:0/1524179711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ddc1a77f0 con 0x7f2ddc101080 2024-04-21T08:58:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.893+0000 7f2dd97fa640 1 -- 172.21.15.28:0/1524179711 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2dcc046020 con 0x7f2ddc101080 2024-04-21T08:58:39.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2de5637740 1 Processor -- start 2024-04-21T08:58:39.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2de5637740 1 -- start start 2024-04-21T08:58:39.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2dd97fa640 1 --2- 172.21.15.28:0/1524179711 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2dc40a3d50 0x7f2dc40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2de5637740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 0x5639c37fe960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2de5637740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c37feea0 0x5639c3807540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2de5637740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c3807a80 0x5639c3809e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2de5637740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5639c37631e0 con 0x5639c3807a80 2024-04-21T08:58:39.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2de5637740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5639c3700390 con 0x5639c37feea0 2024-04-21T08:58:39.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2de5637740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5639c375f1a0 con 0x5639c37fe580 2024-04-21T08:58:39.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2de0f04640 1 --2- 172.21.15.28:0/1524179711 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2dc40a3d50 0x7f2dc40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2dd97fa640 1 -- 172.21.15.28:0/1524179711 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7f2dcc0f2d60 con 0x7f2ddc101080 2024-04-21T08:58:39.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2de0f04640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c3807a80 0x5639c3809e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2ddb7fe640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c37feea0 0x5639c3807540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2de0f04640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c3807a80 0x5639c3809e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33436/0 (socket says 172.21.15.28:33436) 2024-04-21T08:58:39.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2de0f04640 1 -- 172.21.15.28:0/3705380113 learned_addr learned my addr 172.21.15.28:0/3705380113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:39.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.894+0000 7f2ddbfff640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 0x5639c37fe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.895+0000 7f2de0f04640 1 -- 172.21.15.28:0/3705380113 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 msgr2=0x5639c37fe960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:39.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.895+0000 7f2de0f04640 1 --2- 172.21.15.28:0/3705380113 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 0x5639c37fe960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.895+0000 7f2de0f04640 1 -- 172.21.15.28:0/3705380113 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c37feea0 msgr2=0x5639c3807540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:39.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.895+0000 7f2de0f04640 1 --2- 172.21.15.28:0/3705380113 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c37feea0 0x5639c3807540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.895+0000 7f2de0f04640 1 -- 172.21.15.28:0/3705380113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639c353f7d0 con 0x5639c3807a80 2024-04-21T08:58:39.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.895+0000 7f2de0f04640 1 --2- 172.21.15.28:0/1524179711 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2dc40a3d50 0x7f2dc40a6210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4015b70 tx=0x7f2dd4009d90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:39.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.895+0000 7f2de0f04640 1 --2- 172.21.15.28:0/3705380113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c3807a80 0x5639c3809e70 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4015fd0 tx=0x7f2dd401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d32c7848cc72b76 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:39.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.895+0000 7f2dc27fc640 1 -- 172.21.15.28:0/3705380113 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4012d50 con 0x5639c3807a80 2024-04-21T08:58:39.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.895+0000 7f2dc27fc640 1 -- 172.21.15.28:0/3705380113 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dd4010c10 con 0x5639c3807a80 2024-04-21T08:58:39.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.895+0000 7f2dc27fc640 1 -- 172.21.15.28:0/3705380113 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4005370 con 0x5639c3807a80 2024-04-21T08:58:39.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.896+0000 7f2de5637740 1 -- 172.21.15.28:0/3705380113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c3807a80 msgr2=0x5639c3809e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:39.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.896+0000 7f2de5637740 1 --2- 172.21.15.28:0/3705380113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c3807a80 0x5639c3809e70 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4015fd0 tx=0x7f2dd401c000 comp rx=0 tx=0).stop 2024-04-21T08:58:39.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.896+0000 7f2de5637740 1 -- 172.21.15.28:0/3705380113 shutdown_connections 2024-04-21T08:58:39.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.896+0000 7f2de5637740 1 --2- 172.21.15.28:0/3705380113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c3807a80 0x5639c3809e70 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.896+0000 7f2de5637740 1 --2- 172.21.15.28:0/3705380113 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c37feea0 0x5639c3807540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.896+0000 7f2de5637740 1 --2- 172.21.15.28:0/3705380113 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 0x5639c37fe960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.896+0000 7f2de5637740 1 -- 172.21.15.28:0/3705380113 >> 172.21.15.28:0/3705380113 conn(0x5639c37f8cb0 msgr2=0x5639c37f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:39.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.896+0000 7f2de5637740 1 -- 172.21.15.28:0/3705380113 shutdown_connections 2024-04-21T08:58:39.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.896+0000 7f2de5637740 1 -- 172.21.15.28:0/3705380113 wait complete. 2024-04-21T08:58:39.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.896+0000 7f2de5637740 1 -- start start 2024-04-21T08:58:39.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.896+0000 7f2de5637740 1 Processor -- start 2024-04-21T08:58:39.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de5637740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 0x5639c3804a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de5637740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c37feea0 0x5639c3804f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de5637740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c3807a80 0x5639c3805490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de5637740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5639c380a810 con 0x5639c3807a80 2024-04-21T08:58:39.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de5637740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5639c371c2f0 con 0x5639c37feea0 2024-04-21T08:58:39.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de5637740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5639c380a990 con 0x5639c37fe580 2024-04-21T08:58:39.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de5637740 10 client.0 _ll_register_callbacks cb 0x5639c3729580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:39.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de0f04640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c3807a80 0x5639c3805490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de0f04640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c3807a80 0x5639c3805490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33438/0 (socket says 172.21.15.28:33438) 2024-04-21T08:58:39.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de0f04640 1 -- 172.21.15.28:0/4019268331 learned_addr learned my addr 172.21.15.28:0/4019268331 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:39.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2ddb7fe640 1 --2- 172.21.15.28:0/4019268331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c37feea0 0x5639c3804f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2ddbfff640 1 --2- 172.21.15.28:0/4019268331 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 0x5639c3804a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de0f04640 1 -- 172.21.15.28:0/4019268331 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 msgr2=0x5639c3804a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:39.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de0f04640 1 --2- 172.21.15.28:0/4019268331 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 0x5639c3804a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de0f04640 1 -- 172.21.15.28:0/4019268331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c37feea0 msgr2=0x5639c3804f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:39.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de0f04640 1 --2- 172.21.15.28:0/4019268331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c37feea0 0x5639c3804f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de0f04640 1 -- 172.21.15.28:0/4019268331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639c34753f0 con 0x5639c3807a80 2024-04-21T08:58:39.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2ddbfff640 1 --2- 172.21.15.28:0/4019268331 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 0x5639c3804a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:39.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de0f04640 1 --2- 172.21.15.28:0/4019268331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c3807a80 0x5639c3805490 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4012820 tx=0x7f2dd4012970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5380266036afd5a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:39.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2dbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:39.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2dbb7fe640 1 -- 172.21.15.28:0/4019268331 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4005090 con 0x5639c3807a80 2024-04-21T08:58:39.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2dbb7fe640 1 -- 172.21.15.28:0/4019268331 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7f2dd4005230 con 0x5639c3807a80 2024-04-21T08:58:39.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.897+0000 7f2de5637740 1 -- 172.21.15.28:0/4019268331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2dd4010c10 con 0x5639c3807a80 2024-04-21T08:58:39.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.898+0000 7f2dbb7fe640 1 -- 172.21.15.28:0/4019268331 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2dd403b050 con 0x5639c3807a80 2024-04-21T08:58:39.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.898+0000 7f2de5637740 10 client.17565 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:39.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.898+0000 7f2de5637740 1 -- 172.21.15.28:0/4019268331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5639c34350f0 con 0x5639c3807a80 2024-04-21T08:58:39.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.898+0000 7f2dbb7fe640 1 -- 172.21.15.28:0/4019268331 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dd4010c10 con 0x5639c3807a80 2024-04-21T08:58:39.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.898+0000 7f2dbb7fe640 1 -- 172.21.15.28:0/4019268331 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f2dd4029610 con 0x5639c3807a80 2024-04-21T08:58:39.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.898+0000 7f2de5637740 10 client.17565 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:39.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.898+0000 7f2de5637740 10 client.17565 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:39.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.898+0000 7f2de5637740 20 client.17565 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:39.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.898+0000 7f2de5637740 1 --2- 172.21.15.28:0/4019268331 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5639c38179f0 0x5639c3819de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.898+0000 7f2de5637740 4 client.17565 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:58:39.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.898+0000 7f2de5637740 1 -- 172.21.15.28:0/4019268331 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639c3770c00 con 0x5639c38179f0 2024-04-21T08:58:39.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.898+0000 7f2ddbfff640 1 --2- 172.21.15.28:0/4019268331 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5639c38179f0 0x5639c3819de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.899+0000 7f2ddbfff640 1 --2- 172.21.15.28:0/4019268331 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5639c38179f0 0x5639c3819de0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42e810961792d896 server_cookie=a80c989c05beaaa9 in_seq=0 out_seq=0 2024-04-21T08:58:39.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.899+0000 7f2dbb7fe640 10 client.17565 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:39.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.899+0000 7f2dbb7fe640 1 -- 172.21.15.28:0/4019268331 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5639c3770c00 con 0x5639c38179f0 2024-04-21T08:58:39.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.899+0000 7f2dbb7fe640 10 client.17565 handle_command_reply: tid=0 2024-04-21T08:58:39.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.900+0000 7f2de5637740 1 client.17565 shutdown 2024-04-21T08:58:39.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.900+0000 7f2de5637740 1 -- 172.21.15.28:0/4019268331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c3807a80 msgr2=0x5639c3805490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:39.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.900+0000 7f2de5637740 1 --2- 172.21.15.28:0/4019268331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c3807a80 0x5639c3805490 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4012820 tx=0x7f2dd4012970 comp rx=0 tx=0).stop 2024-04-21T08:58:39.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.900+0000 7f2de5637740 1 -- 172.21.15.28:0/4019268331 shutdown_connections 2024-04-21T08:58:39.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.900+0000 7f2de5637740 1 --2- 172.21.15.28:0/4019268331 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5639c38179f0 0x5639c3819de0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.900+0000 7f2de5637740 1 --2- 172.21.15.28:0/4019268331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c3807a80 0x5639c3805490 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.900+0000 7f2de5637740 1 --2- 172.21.15.28:0/4019268331 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c37feea0 0x5639c3804f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.900+0000 7f2dbb7fe640 0 client.17565 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:39.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.900+0000 7f2de5637740 1 --2- 172.21.15.28:0/4019268331 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 0x5639c3804a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.900+0000 7f2de5637740 1 -- 172.21.15.28:0/4019268331 >> 172.21.15.28:0/4019268331 conn(0x5639c37f8cb0 msgr2=0x5639c37faa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:39.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.900+0000 7f2de5637740 1 -- 172.21.15.28:0/4019268331 shutdown_connections 2024-04-21T08:58:39.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.900+0000 7f2de5637740 1 -- 172.21.15.28:0/4019268331 wait complete. 2024-04-21T08:58:39.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.900+0000 7f2de5637740 20 client.17565 trim_cache size 0 max 16384 2024-04-21T08:58:39.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2de5637740 1 Processor -- start 2024-04-21T08:58:39.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2de5637740 1 -- start start 2024-04-21T08:58:39.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2de5637740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 0x5639c3807140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2de5637740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c37feea0 0x5639c3803950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2de5637740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c3807a80 0x5639c38137b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2de5637740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ddc1b3e60 con 0x5639c37feea0 2024-04-21T08:58:39.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2de5637740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2ddc1b1730 con 0x5639c3807a80 2024-04-21T08:58:39.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2de5637740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2ddc1af5a0 con 0x5639c37fe580 2024-04-21T08:58:39.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2ddb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c37feea0 0x5639c3803950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2ddb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c37feea0 0x5639c3803950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33446/0 (socket says 172.21.15.28:33446) 2024-04-21T08:58:39.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2ddb7fe640 1 -- 172.21.15.28:0/4127227163 learned_addr learned my addr 172.21.15.28:0/4127227163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:39.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2ddbfff640 1 --2- 172.21.15.28:0/4127227163 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 0x5639c3807140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2de0f04640 1 --2- 172.21.15.28:0/4127227163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c3807a80 0x5639c38137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2ddb7fe640 1 -- 172.21.15.28:0/4127227163 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 msgr2=0x5639c3807140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:39.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2ddb7fe640 1 --2- 172.21.15.28:0/4127227163 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 0x5639c3807140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2ddb7fe640 1 -- 172.21.15.28:0/4127227163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c3807a80 msgr2=0x5639c38137b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:39.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2ddb7fe640 1 --2- 172.21.15.28:0/4127227163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c3807a80 0x5639c38137b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2ddb7fe640 1 -- 172.21.15.28:0/4127227163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639c37f9630 con 0x5639c37feea0 2024-04-21T08:58:39.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2de0f04640 1 --2- 172.21.15.28:0/4127227163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c3807a80 0x5639c38137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:39.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.918+0000 7f2ddb7fe640 1 --2- 172.21.15.28:0/4127227163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c37feea0 0x5639c3803950 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0005fd0 tx=0x7f2dd000daf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7734fc8012883f85 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:39.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.919+0000 7f2dc1ffb640 1 -- 172.21.15.28:0/4127227163 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0005020 con 0x5639c37feea0 2024-04-21T08:58:39.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.919+0000 7f2dc1ffb640 1 -- 172.21.15.28:0/4127227163 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dd00047a0 con 0x5639c37feea0 2024-04-21T08:58:39.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.919+0000 7f2dc1ffb640 1 -- 172.21.15.28:0/4127227163 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0003ad0 con 0x5639c37feea0 2024-04-21T08:58:39.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.919+0000 7f2de5637740 1 -- 172.21.15.28:0/4127227163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c37feea0 msgr2=0x5639c3803950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:39.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.919+0000 7f2de5637740 1 --2- 172.21.15.28:0/4127227163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c37feea0 0x5639c3803950 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0005fd0 tx=0x7f2dd000daf0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.919+0000 7f2de5637740 1 -- 172.21.15.28:0/4127227163 shutdown_connections 2024-04-21T08:58:39.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.919+0000 7f2de5637740 1 --2- 172.21.15.28:0/4127227163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c3807a80 0x5639c38137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.919+0000 7f2de5637740 1 --2- 172.21.15.28:0/4127227163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c37feea0 0x5639c3803950 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.919+0000 7f2de5637740 1 --2- 172.21.15.28:0/4127227163 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37fe580 0x5639c3807140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.919+0000 7f2de5637740 1 -- 172.21.15.28:0/4127227163 >> 172.21.15.28:0/4127227163 conn(0x5639c37f8cb0 msgr2=0x5639c37f7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:39.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.919+0000 7f2de5637740 1 -- 172.21.15.28:0/4127227163 shutdown_connections 2024-04-21T08:58:39.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.919+0000 7f2de5637740 1 -- 172.21.15.28:0/4127227163 wait complete. 2024-04-21T08:58:39.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.919+0000 7f2de5637740 1 -- start start 2024-04-21T08:58:39.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2de5637740 1 Processor -- start 2024-04-21T08:58:39.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2de5637740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c37fe580 0x5639c380b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2de5637740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37feea0 0x5639c380bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2de5637740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c3807a80 0x5639c380c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2de5637740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ddc1afbf0 con 0x5639c37fe580 2024-04-21T08:58:39.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2de5637740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2ddc1b1060 con 0x5639c3807a80 2024-04-21T08:58:39.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2de5637740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2ddc1b02c0 con 0x5639c37feea0 2024-04-21T08:58:39.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2de5637740 10 client.0 _ll_register_callbacks cb 0x7f2dd000d3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:39.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2ddbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c37fe580 0x5639c380b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2ddbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c37fe580 0x5639c380b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33462/0 (socket says 172.21.15.28:33462) 2024-04-21T08:58:39.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2ddbfff640 1 -- 172.21.15.28:0/4255175863 learned_addr learned my addr 172.21.15.28:0/4255175863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:39.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2de0f04640 1 --2- 172.21.15.28:0/4255175863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c3807a80 0x5639c380c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2ddb7fe640 1 --2- 172.21.15.28:0/4255175863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37feea0 0x5639c380bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2ddbfff640 1 -- 172.21.15.28:0/4255175863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37feea0 msgr2=0x5639c380bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:39.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2ddbfff640 1 --2- 172.21.15.28:0/4255175863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37feea0 0x5639c380bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2ddbfff640 1 -- 172.21.15.28:0/4255175863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c3807a80 msgr2=0x5639c380c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:39.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2ddbfff640 1 --2- 172.21.15.28:0/4255175863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c3807a80 0x5639c380c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2ddbfff640 1 -- 172.21.15.28:0/4255175863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639c387cd70 con 0x5639c37fe580 2024-04-21T08:58:39.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2de0f04640 1 --2- 172.21.15.28:0/4255175863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c3807a80 0x5639c380c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:39.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.920+0000 7f2ddb7fe640 1 --2- 172.21.15.28:0/4255175863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37feea0 0x5639c380bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:39.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2ddbfff640 1 --2- 172.21.15.28:0/4255175863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c37fe580 0x5639c380b620 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0f0230 tx=0x7f2dcc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225cc03f653075c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:39.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2dbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:39.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2dbb7fe640 1 -- 172.21.15.28:0/4255175863 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc0beb40 con 0x5639c37fe580 2024-04-21T08:58:39.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2dbb7fe640 1 -- 172.21.15.28:0/4255175863 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7f2dcc0bce40 con 0x5639c37fe580 2024-04-21T08:58:39.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2de5637740 1 -- 172.21.15.28:0/4255175863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2dd00047a0 con 0x5639c37fe580 2024-04-21T08:58:39.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2dbb7fe640 1 -- 172.21.15.28:0/4255175863 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2dcc0c2ef0 con 0x5639c37fe580 2024-04-21T08:58:39.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2de5637740 10 client.17571 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2de5637740 1 -- 172.21.15.28:0/4255175863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5639c387df90 con 0x5639c37fe580 2024-04-21T08:58:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2dbb7fe640 1 -- 172.21.15.28:0/4255175863 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dd00047a0 con 0x5639c37fe580 2024-04-21T08:58:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2dbb7fe640 1 -- 172.21.15.28:0/4255175863 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f2dcc083be0 con 0x5639c37fe580 2024-04-21T08:58:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2de5637740 10 client.17571 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2de5637740 10 client.17571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2de5637740 20 client.17571 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2de5637740 1 --2- 172.21.15.28:0/4255175863 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5639c387e6b0 0x5639c3880ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:39.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2de5637740 4 client.17571 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:58:39.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.921+0000 7f2de5637740 1 -- 172.21.15.28:0/4255175863 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5639c376aa70 con 0x5639c387e6b0 2024-04-21T08:58:39.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.922+0000 7f2ddb7fe640 1 --2- 172.21.15.28:0/4255175863 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5639c387e6b0 0x5639c3880ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:39.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.922+0000 7f2dbb7fe640 10 client.17571 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:39.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.922+0000 7f2ddb7fe640 1 --2- 172.21.15.28:0/4255175863 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5639c387e6b0 0x5639c3880ad0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30ee04b9f1394141 server_cookie=a7a7d80fda171c24 in_seq=0 out_seq=0 2024-04-21T08:58:39.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.922+0000 7f2dbb7fe640 1 -- 172.21.15.28:0/4255175863 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5639c376aa70 con 0x5639c387e6b0 2024-04-21T08:58:39.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.922+0000 7f2dbb7fe640 10 client.17571 handle_command_reply: tid=0 2024-04-21T08:58:39.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.922+0000 7f2de5637740 1 client.17571 shutdown 2024-04-21T08:58:39.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.922+0000 7f2de5637740 1 -- 172.21.15.28:0/4255175863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c37fe580 msgr2=0x5639c380b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:39.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.922+0000 7f2de5637740 1 --2- 172.21.15.28:0/4255175863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c37fe580 0x5639c380b620 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0f0230 tx=0x7f2dcc009260 comp rx=0 tx=0).stop 2024-04-21T08:58:39.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.922+0000 7f2de5637740 1 -- 172.21.15.28:0/4255175863 shutdown_connections 2024-04-21T08:58:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.922+0000 7f2de5637740 1 --2- 172.21.15.28:0/4255175863 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5639c387e6b0 0x5639c3880ad0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.922+0000 7f2de5637740 1 --2- 172.21.15.28:0/4255175863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5639c3807a80 0x5639c380c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.922+0000 7f2de5637740 1 --2- 172.21.15.28:0/4255175863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5639c37feea0 0x5639c380bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.922+0000 7f2dbb7fe640 0 client.17571 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.922+0000 7f2de5637740 1 --2- 172.21.15.28:0/4255175863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639c37fe580 0x5639c380b620 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.922+0000 7f2de5637740 1 -- 172.21.15.28:0/4255175863 >> 172.21.15.28:0/4255175863 conn(0x5639c37f8cb0 msgr2=0x5639c37f74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.923+0000 7f2de5637740 1 -- 172.21.15.28:0/4255175863 shutdown_connections 2024-04-21T08:58:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.923+0000 7f2de5637740 1 -- 172.21.15.28:0/4255175863 wait complete. 2024-04-21T08:58:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.923+0000 7f2de5637740 20 client.17571 trim_cache size 0 max 16384 2024-04-21T08:58:39.933 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:58:39.933 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T08:58:39.933 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T08:58:39.933 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:58:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.923+0000 7f2de2967640 1 -- 172.21.15.28:0/1524179711 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2dc40a3d50 msgr2=0x7f2dc40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.923+0000 7f2de2967640 1 --2- 172.21.15.28:0/1524179711 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2dc40a3d50 0x7f2dc40a6210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4015b70 tx=0x7f2dd4009d90 comp rx=0 tx=0).stop 2024-04-21T08:58:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.923+0000 7f2de2967640 1 -- 172.21.15.28:0/1524179711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ddc101080 msgr2=0x7f2ddc102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.923+0000 7f2de2967640 1 --2- 172.21.15.28:0/1524179711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ddc101080 0x7f2ddc102b10 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc000c00 tx=0x7f2dcc02c7e0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.923+0000 7f2de2967640 1 -- 172.21.15.28:0/1524179711 shutdown_connections 2024-04-21T08:58:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.923+0000 7f2de2967640 1 --2- 172.21.15.28:0/1524179711 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2dc40a3d50 0x7f2dc40a6210 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.923+0000 7f2de2967640 1 --2- 172.21.15.28:0/1524179711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ddc103590 0x7f2ddc1a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.923+0000 7f2de2967640 1 --2- 172.21.15.28:0/1524179711 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2ddc106150 0x7f2ddc103050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.923+0000 7f2de2967640 1 --2- 172.21.15.28:0/1524179711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ddc101080 0x7f2ddc102b10 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:39.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.923+0000 7f2de2967640 1 -- 172.21.15.28:0/1524179711 >> 172.21.15.28:0/1524179711 conn(0x7f2ddc078000 msgr2=0x7f2ddc069490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:39.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.923+0000 7f2de2967640 1 -- 172.21.15.28:0/1524179711 shutdown_connections 2024-04-21T08:58:39.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:39.923+0000 7f2de2967640 1 -- 172.21.15.28:0/1524179711 wait complete. 2024-04-21T08:58:39.936 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:58:39.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:58:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.038+0000 7faae2f9d640 1 -- 172.21.15.28:0/1382639378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faadc1031d0 msgr2=0x7faadc1035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.038+0000 7faae2f9d640 1 --2- 172.21.15.28:0/1382639378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faadc1031d0 0x7faadc1035b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7faacc009900 tx=0x7faacc02bcd0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae2f9d640 1 -- 172.21.15.28:0/1382639378 shutdown_connections 2024-04-21T08:58:40.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae2f9d640 1 --2- 172.21.15.28:0/1382639378 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faadc10c5a0 0x7faadc10e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae2f9d640 1 --2- 172.21.15.28:0/1382639378 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faadc103af0 0x7faadc10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae2f9d640 1 --2- 172.21.15.28:0/1382639378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faadc1031d0 0x7faadc1035b0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae2f9d640 1 -- 172.21.15.28:0/1382639378 >> 172.21.15.28:0/1382639378 conn(0x7faadc0771d0 msgr2=0x7faadc0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae2f9d640 1 -- 172.21.15.28:0/1382639378 shutdown_connections 2024-04-21T08:58:40.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae2f9d640 1 -- 172.21.15.28:0/1382639378 wait complete. 2024-04-21T08:58:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae2f9d640 1 Processor -- start 2024-04-21T08:58:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae2f9d640 1 -- start start 2024-04-21T08:58:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae2f9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faadc1031d0 0x7faadc100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae2f9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faadc103af0 0x7faadc101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae2f9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faadc10c5a0 0x7faadc1018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae2f9d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faadc1121d0 con 0x7faadc10c5a0 2024-04-21T08:58:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae2f9d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7faadc112050 con 0x7faadc103af0 2024-04-21T08:58:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae2f9d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7faadc112350 con 0x7faadc1031d0 2024-04-21T08:58:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.039+0000 7faae153a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faadc10c5a0 0x7faadc1018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faae153a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faadc10c5a0 0x7faadc1018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33472/0 (socket says 172.21.15.28:33472) 2024-04-21T08:58:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faae153a640 1 -- 172.21.15.28:0/771269837 learned_addr learned my addr 172.21.15.28:0/771269837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faae0d39640 1 --2- 172.21.15.28:0/771269837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faadc1031d0 0x7faadc100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faadbfff640 1 --2- 172.21.15.28:0/771269837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faadc103af0 0x7faadc101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faae153a640 1 -- 172.21.15.28:0/771269837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faadc1031d0 msgr2=0x7faadc100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faae153a640 1 --2- 172.21.15.28:0/771269837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faadc1031d0 0x7faadc100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faae153a640 1 -- 172.21.15.28:0/771269837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faadc103af0 msgr2=0x7faadc101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faae153a640 1 --2- 172.21.15.28:0/771269837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faadc103af0 0x7faadc101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faae153a640 1 -- 172.21.15.28:0/771269837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faadc1a6bb0 con 0x7faadc10c5a0 2024-04-21T08:58:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faadbfff640 1 --2- 172.21.15.28:0/771269837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faadc103af0 0x7faadc101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faae153a640 1 --2- 172.21.15.28:0/771269837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faadc10c5a0 0x7faadc1018a0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7faad400dac0 tx=0x7faad400df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faad9ffb640 1 -- 172.21.15.28:0/771269837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad4017070 con 0x7faadc10c5a0 2024-04-21T08:58:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faad9ffb640 1 -- 172.21.15.28:0/771269837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faad400be30 con 0x7faadc10c5a0 2024-04-21T08:58:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faae2f9d640 1 -- 172.21.15.28:0/771269837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faadc1a6ea0 con 0x7faadc10c5a0 2024-04-21T08:58:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faad9ffb640 1 -- 172.21.15.28:0/771269837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad4010460 con 0x7faadc10c5a0 2024-04-21T08:58:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.040+0000 7faae2f9d640 1 -- 172.21.15.28:0/771269837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faadc1a7370 con 0x7faadc10c5a0 2024-04-21T08:58:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.041+0000 7faae2f9d640 1 -- 172.21.15.28:0/771269837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faadc069230 con 0x7faadc10c5a0 2024-04-21T08:58:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.043+0000 7faad9ffb640 1 -- 172.21.15.28:0/771269837 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faad40040b0 con 0x7faadc10c5a0 2024-04-21T08:58:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.043+0000 7faad9ffb640 1 --2- 172.21.15.28:0/771269837 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faac40a3d20 0x7faac40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.043+0000 7faae0d39640 1 --2- 172.21.15.28:0/771269837 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faac40a3d20 0x7faac40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.043+0000 7faad9ffb640 1 -- 172.21.15.28:0/771269837 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7faad40d0500 con 0x7faadc10c5a0 2024-04-21T08:58:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.044+0000 7faae0d39640 1 --2- 172.21.15.28:0/771269837 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faac40a3d20 0x7faac40a61e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faacc006010 tx=0x7faacc005e40 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.048+0000 7faad9ffb640 1 -- 172.21.15.28:0/771269837 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faad409c7b0 con 0x7faadc10c5a0 2024-04-21T08:58:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.274+0000 7faae2f9d640 1 -- 172.21.15.28:0/771269837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faadc102350 con 0x7faadc10c5a0 2024-04-21T08:58:40.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.274+0000 7faad9ffb640 1 -- 172.21.15.28:0/771269837 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7faad409c0b0 con 0x7faadc10c5a0 2024-04-21T08:58:40.274 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:58:40.275 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:58:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T08:58:40.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.276+0000 7faae2f9d640 1 -- 172.21.15.28:0/771269837 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faac40a3d20 msgr2=0x7faac40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.276+0000 7faae2f9d640 1 --2- 172.21.15.28:0/771269837 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faac40a3d20 0x7faac40a61e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faacc006010 tx=0x7faacc005e40 comp rx=0 tx=0).stop 2024-04-21T08:58:40.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.276+0000 7faae2f9d640 1 -- 172.21.15.28:0/771269837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faadc10c5a0 msgr2=0x7faadc1018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.276+0000 7faae2f9d640 1 --2- 172.21.15.28:0/771269837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faadc10c5a0 0x7faadc1018a0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7faad400dac0 tx=0x7faad400df80 comp rx=0 tx=0).stop 2024-04-21T08:58:40.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.277+0000 7faae2f9d640 1 -- 172.21.15.28:0/771269837 shutdown_connections 2024-04-21T08:58:40.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.277+0000 7faae2f9d640 1 --2- 172.21.15.28:0/771269837 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faac40a3d20 0x7faac40a61e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.277+0000 7faae2f9d640 1 --2- 172.21.15.28:0/771269837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faadc10c5a0 0x7faadc1018a0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.277+0000 7faae2f9d640 1 --2- 172.21.15.28:0/771269837 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faadc103af0 0x7faadc101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.277+0000 7faae2f9d640 1 --2- 172.21.15.28:0/771269837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faadc1031d0 0x7faadc100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.277+0000 7faae2f9d640 1 -- 172.21.15.28:0/771269837 >> 172.21.15.28:0/771269837 conn(0x7faadc0771d0 msgr2=0x7faadc0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.277+0000 7faae2f9d640 1 -- 172.21.15.28:0/771269837 shutdown_connections 2024-04-21T08:58:40.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.277+0000 7faae2f9d640 1 -- 172.21.15.28:0/771269837 wait complete. 2024-04-21T08:58:40.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:58:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.393+0000 7f27e6206640 1 -- 172.21.15.28:0/3888000563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e0106160 msgr2=0x7f27e010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.393+0000 7f27e6206640 1 --2- 172.21.15.28:0/3888000563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e0106160 0x7f27e010dd30 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f27d8009900 tx=0x7f27d802bc30 comp rx=0 tx=0).stop 2024-04-21T08:58:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.393+0000 7f27e6206640 1 -- 172.21.15.28:0/3888000563 shutdown_connections 2024-04-21T08:58:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.393+0000 7f27e6206640 1 --2- 172.21.15.28:0/3888000563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e0106160 0x7f27e010dd30 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.393+0000 7f27e6206640 1 --2- 172.21.15.28:0/3888000563 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27e0101a40 0x7f27e0105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.393+0000 7f27e6206640 1 --2- 172.21.15.28:0/3888000563 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f27e0101090 0x7f27e0101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.393+0000 7f27e6206640 1 -- 172.21.15.28:0/3888000563 >> 172.21.15.28:0/3888000563 conn(0x7f27e0077f50 msgr2=0x7f27e0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.393+0000 7f27e6206640 1 -- 172.21.15.28:0/3888000563 shutdown_connections 2024-04-21T08:58:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.393+0000 7f27e6206640 1 -- 172.21.15.28:0/3888000563 wait complete. 2024-04-21T08:58:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.394+0000 7f27e6206640 1 Processor -- start 2024-04-21T08:58:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.394+0000 7f27e6206640 1 -- start start 2024-04-21T08:58:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.394+0000 7f27e6206640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f27e0101090 0x7f27e0198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.394+0000 7f27e6206640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e0106160 0x7f27e0198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.394+0000 7f27e6206640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27e019cfe0 0x7f27e019d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.394+0000 7f27e6206640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27e0111120 con 0x7f27e0106160 2024-04-21T08:58:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.394+0000 7f27e6206640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f27e0110fa0 con 0x7f27e019cfe0 2024-04-21T08:58:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.394+0000 7f27e6206640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01112a0 con 0x7f27e0101090 2024-04-21T08:58:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27deffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e0106160 0x7f27e0198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27deffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e0106160 0x7f27e0198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33490/0 (socket says 172.21.15.28:33490) 2024-04-21T08:58:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27deffd640 1 -- 172.21.15.28:0/118633456 learned_addr learned my addr 172.21.15.28:0/118633456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27df7fe640 1 --2- 172.21.15.28:0/118633456 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f27e0101090 0x7f27e0198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27dffff640 1 --2- 172.21.15.28:0/118633456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27e019cfe0 0x7f27e019d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27deffd640 1 -- 172.21.15.28:0/118633456 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f27e0101090 msgr2=0x7f27e0198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27deffd640 1 --2- 172.21.15.28:0/118633456 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f27e0101090 0x7f27e0198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27deffd640 1 -- 172.21.15.28:0/118633456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27e019cfe0 msgr2=0x7f27e019d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27deffd640 1 --2- 172.21.15.28:0/118633456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27e019cfe0 0x7f27e019d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27deffd640 1 -- 172.21.15.28:0/118633456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e019dad0 con 0x7f27e0106160 2024-04-21T08:58:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27dffff640 1 --2- 172.21.15.28:0/118633456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27e019cfe0 0x7f27e019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27deffd640 1 --2- 172.21.15.28:0/118633456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e0106160 0x7f27e0198c60 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f27d400d770 tx=0x7f27d400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27dcff9640 1 -- 172.21.15.28:0/118633456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d4017070 con 0x7f27e0106160 2024-04-21T08:58:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27dcff9640 1 -- 172.21.15.28:0/118633456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27d4009c30 con 0x7f27e0106160 2024-04-21T08:58:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27e6206640 1 -- 172.21.15.28:0/118633456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e019ddc0 con 0x7f27e0106160 2024-04-21T08:58:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27dcff9640 1 -- 172.21.15.28:0/118633456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d400f620 con 0x7f27e0106160 2024-04-21T08:58:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.395+0000 7f27e6206640 1 -- 172.21.15.28:0/118633456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e01a55b0 con 0x7f27e0106160 2024-04-21T08:58:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.397+0000 7f27dcff9640 1 -- 172.21.15.28:0/118633456 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f27d401a070 con 0x7f27e0106160 2024-04-21T08:58:40.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27e8df5740 1 Processor -- start 2024-04-21T08:58:40.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27e8df5740 1 -- start start 2024-04-21T08:58:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27dcff9640 1 --2- 172.21.15.28:0/118633456 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f27c80a3d20 0x7f27c80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27e8df5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffa64d0 0x5653bffa68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27e8df5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa6e80 0x5653bffaf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27e8df5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffafa00 0x5653bffb1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27e8df5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5653bff0b1f0 con 0x5653bffa6e80 2024-04-21T08:58:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27e8df5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5653bfea84b0 con 0x5653bffafa00 2024-04-21T08:58:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27e8df5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5653bff071b0 con 0x5653bffa64d0 2024-04-21T08:58:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27deffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa6e80 0x5653bffaf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27dffff640 1 --2- 172.21.15.28:0/118633456 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f27c80a3d20 0x7f27c80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27deffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa6e80 0x5653bffaf430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33494/0 (socket says 172.21.15.28:33494) 2024-04-21T08:58:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27deffd640 1 -- 172.21.15.28:0/1552476241 learned_addr learned my addr 172.21.15.28:0/1552476241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27dcff9640 1 -- 172.21.15.28:0/118633456 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7f27d40d0730 con 0x7f27e0106160 2024-04-21T08:58:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27df7fe640 1 --2- 172.21.15.28:0/1552476241 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffa64d0 0x5653bffa68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.398+0000 7f27dffff640 1 --2- 172.21.15.28:0/1552476241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffafa00 0x5653bffb1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.399+0000 7f27deffd640 1 -- 172.21.15.28:0/1552476241 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffa64d0 msgr2=0x5653bffa68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.399+0000 7f27deffd640 1 --2- 172.21.15.28:0/1552476241 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffa64d0 0x5653bffa68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.399+0000 7f27deffd640 1 -- 172.21.15.28:0/1552476241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffafa00 msgr2=0x5653bffb1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.399+0000 7f27deffd640 1 --2- 172.21.15.28:0/1552476241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffafa00 0x5653bffb1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.399+0000 7f27deffd640 1 -- 172.21.15.28:0/1552476241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653bfe9a1b0 con 0x5653bffa6e80 2024-04-21T08:58:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.399+0000 7f27df7fe640 1 --2- 172.21.15.28:0/1552476241 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffa64d0 0x5653bffa68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:58:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.399+0000 7f27dffff640 1 --2- 172.21.15.28:0/118633456 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f27c80a3d20 0x7f27c80a61e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f27d8006010 tx=0x7f27d80043e0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.399+0000 7f27deffd640 1 --2- 172.21.15.28:0/1552476241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa6e80 0x5653bffaf430 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f27d409c210 tx=0x7f27d400df10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6225d85c3ca94b83 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.399+0000 7f27c5ffb640 1 -- 172.21.15.28:0/1552476241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d4021070 con 0x5653bffa6e80 2024-04-21T08:58:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.399+0000 7f27c5ffb640 1 -- 172.21.15.28:0/1552476241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27d40cf030 con 0x5653bffa6e80 2024-04-21T08:58:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.399+0000 7f27c5ffb640 1 -- 172.21.15.28:0/1552476241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d40a4ef0 con 0x5653bffa6e80 2024-04-21T08:58:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.400+0000 7f27e8df5740 1 -- 172.21.15.28:0/1552476241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa6e80 msgr2=0x5653bffaf430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.400+0000 7f27e8df5740 1 --2- 172.21.15.28:0/1552476241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa6e80 0x5653bffaf430 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f27d409c210 tx=0x7f27d400df10 comp rx=0 tx=0).stop 2024-04-21T08:58:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.400+0000 7f27e8df5740 1 -- 172.21.15.28:0/1552476241 shutdown_connections 2024-04-21T08:58:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.400+0000 7f27e8df5740 1 --2- 172.21.15.28:0/1552476241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffafa00 0x5653bffb1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.400+0000 7f27e8df5740 1 --2- 172.21.15.28:0/1552476241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa6e80 0x5653bffaf430 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.400+0000 7f27e8df5740 1 --2- 172.21.15.28:0/1552476241 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffa64d0 0x5653bffa68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.400+0000 7f27e8df5740 1 -- 172.21.15.28:0/1552476241 >> 172.21.15.28:0/1552476241 conn(0x5653bffa0ba0 msgr2=0x5653bffa0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.400+0000 7f27e8df5740 1 -- 172.21.15.28:0/1552476241 shutdown_connections 2024-04-21T08:58:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.400+0000 7f27e8df5740 1 -- 172.21.15.28:0/1552476241 wait complete. 2024-04-21T08:58:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.400+0000 7f27e8df5740 1 -- start start 2024-04-21T08:58:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.400+0000 7f27e8df5740 1 Processor -- start 2024-04-21T08:58:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27e8df5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa64d0 0x5653bffaf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27e8df5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffa6e80 0x5653bffac6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27e8df5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffafa00 0x5653bffacce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27e8df5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5653bffb2970 con 0x5653bffa64d0 2024-04-21T08:58:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27e8df5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5653bfec4500 con 0x5653bffa6e80 2024-04-21T08:58:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27e8df5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5653bffb2af0 con 0x5653bffafa00 2024-04-21T08:58:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27e8df5740 10 client.0 _ll_register_callbacks cb 0x5653bfe92b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa64d0 0x5653bffaf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa64d0 0x5653bffaf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33502/0 (socket says 172.21.15.28:33502) 2024-04-21T08:58:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27df7fe640 1 -- 172.21.15.28:0/1508531120 learned_addr learned my addr 172.21.15.28:0/1508531120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27deffd640 1 --2- 172.21.15.28:0/1508531120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffa6e80 0x5653bffac6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27dffff640 1 --2- 172.21.15.28:0/1508531120 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffafa00 0x5653bffacce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27df7fe640 1 -- 172.21.15.28:0/1508531120 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffafa00 msgr2=0x5653bffacce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27df7fe640 1 --2- 172.21.15.28:0/1508531120 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffafa00 0x5653bffacce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27df7fe640 1 -- 172.21.15.28:0/1508531120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffa6e80 msgr2=0x5653bffac6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27df7fe640 1 --2- 172.21.15.28:0/1508531120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffa6e80 0x5653bffac6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27df7fe640 1 -- 172.21.15.28:0/1508531120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653bfc1d3f0 con 0x5653bffa64d0 2024-04-21T08:58:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27df7fe640 1 --2- 172.21.15.28:0/1508531120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa64d0 0x5653bffaf340 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f27d0013d20 tx=0x7f27d00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04f1c7df2757876 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27beffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27beffd640 1 -- 172.21.15.28:0/1508531120 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d001c640 con 0x5653bffa64d0 2024-04-21T08:58:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27beffd640 1 -- 172.21.15.28:0/1508531120 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7f27d001b3e0 con 0x5653bffa64d0 2024-04-21T08:58:40.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.401+0000 7f27e8df5740 1 -- 172.21.15.28:0/1508531120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27d40cf030 con 0x5653bffa64d0 2024-04-21T08:58:40.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.402+0000 7f27beffd640 1 -- 172.21.15.28:0/1508531120 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f27d001c7e0 con 0x5653bffa64d0 2024-04-21T08:58:40.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.402+0000 7f27e8df5740 10 client.17589 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:40.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.402+0000 7f27e8df5740 1 -- 172.21.15.28:0/1508531120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5653bfce77d0 con 0x5653bffa64d0 2024-04-21T08:58:40.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.402+0000 7f27beffd640 1 -- 172.21.15.28:0/1508531120 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27d40cf030 con 0x5653bffa64d0 2024-04-21T08:58:40.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.402+0000 7f27beffd640 1 -- 172.21.15.28:0/1508531120 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f27d001a070 con 0x5653bffa64d0 2024-04-21T08:58:40.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.402+0000 7f27e8df5740 10 client.17589 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:40.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.402+0000 7f27e8df5740 10 client.17589 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:58:40.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.402+0000 7f27e8df5740 20 client.17589 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:40.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.402+0000 7f27e8df5740 1 --2- 172.21.15.28:0/1508531120 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5653bffbfd60 0x5653bffc2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.402+0000 7f27e8df5740 4 client.17589 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:58:40.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.402+0000 7f27e8df5740 1 -- 172.21.15.28:0/1508531120 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653bfeaa930 con 0x5653bffbfd60 2024-04-21T08:58:40.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.402+0000 7f27deffd640 1 --2- 172.21.15.28:0/1508531120 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5653bffbfd60 0x5653bffc2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.403+0000 7f27deffd640 1 --2- 172.21.15.28:0/1508531120 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5653bffbfd60 0x5653bffc2190 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a5860bab70bad90d server_cookie=9b3ddf080905fb18 in_seq=0 out_seq=0 2024-04-21T08:58:40.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.403+0000 7f27beffd640 10 client.17589 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:58:40.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.404+0000 7f27beffd640 1 -- 172.21.15.28:0/1508531120 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5653bfeaa930 con 0x5653bffbfd60 2024-04-21T08:58:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.404+0000 7f27beffd640 10 client.17589 handle_command_reply: tid=0 2024-04-21T08:58:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.404+0000 7f27e8df5740 1 client.17589 shutdown 2024-04-21T08:58:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.404+0000 7f27e8df5740 1 -- 172.21.15.28:0/1508531120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa64d0 msgr2=0x5653bffaf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.404+0000 7f27e8df5740 1 --2- 172.21.15.28:0/1508531120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa64d0 0x5653bffaf340 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f27d0013d20 tx=0x7f27d00117b0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.404+0000 7f27e8df5740 1 -- 172.21.15.28:0/1508531120 shutdown_connections 2024-04-21T08:58:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.404+0000 7f27e8df5740 1 --2- 172.21.15.28:0/1508531120 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5653bffbfd60 0x5653bffc2190 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.404+0000 7f27beffd640 0 client.17589 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:58:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.404+0000 7f27e8df5740 1 --2- 172.21.15.28:0/1508531120 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffafa00 0x5653bffacce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.404+0000 7f27e8df5740 1 --2- 172.21.15.28:0/1508531120 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffa6e80 0x5653bffac6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.404+0000 7f27e8df5740 1 --2- 172.21.15.28:0/1508531120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa64d0 0x5653bffaf340 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.404+0000 7f27e8df5740 1 -- 172.21.15.28:0/1508531120 >> 172.21.15.28:0/1508531120 conn(0x5653bffa0ba0 msgr2=0x5653bffa28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.404+0000 7f27e8df5740 1 -- 172.21.15.28:0/1508531120 shutdown_connections 2024-04-21T08:58:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.404+0000 7f27e8df5740 1 -- 172.21.15.28:0/1508531120 wait complete. 2024-04-21T08:58:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.404+0000 7f27e8df5740 20 client.17589 trim_cache size 0 max 16384 2024-04-21T08:58:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.417+0000 7f27e8df5740 1 Processor -- start 2024-04-21T08:58:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.417+0000 7f27e8df5740 1 -- start start 2024-04-21T08:58:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.417+0000 7f27e8df5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffa64d0 0x5653bffbb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.417+0000 7f27e8df5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffa6e80 0x5653bffaee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.417+0000 7f27e8df5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffafa00 0x5653bffaf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.417+0000 7f27e8df5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01b1da0 con 0x5653bffafa00 2024-04-21T08:58:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.417+0000 7f27e8df5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01af670 con 0x5653bffa6e80 2024-04-21T08:58:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.417+0000 7f27e8df5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01ad4e0 con 0x5653bffa64d0 2024-04-21T08:58:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.418+0000 7f27dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffafa00 0x5653bffaf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.418+0000 7f27dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffafa00 0x5653bffaf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33516/0 (socket says 172.21.15.28:33516) 2024-04-21T08:58:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.418+0000 7f27dffff640 1 -- 172.21.15.28:0/616652210 learned_addr learned my addr 172.21.15.28:0/616652210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.418+0000 7f27deffd640 1 --2- 172.21.15.28:0/616652210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffa6e80 0x5653bffaee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.418+0000 7f27dffff640 1 -- 172.21.15.28:0/616652210 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffa64d0 msgr2=0x5653bffbb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.418+0000 7f27dffff640 1 --2- 172.21.15.28:0/616652210 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffa64d0 0x5653bffbb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.418+0000 7f27dffff640 1 -- 172.21.15.28:0/616652210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffa6e80 msgr2=0x5653bffaee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.418+0000 7f27dffff640 1 --2- 172.21.15.28:0/616652210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffa6e80 0x5653bffaee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.418+0000 7f27dffff640 1 -- 172.21.15.28:0/616652210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653bffb2160 con 0x5653bffafa00 2024-04-21T08:58:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.418+0000 7f27dffff640 1 --2- 172.21.15.28:0/616652210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffafa00 0x5653bffaf340 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f27d8009900 tx=0x7f27d8048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0d578355a3ae28b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.418+0000 7f27c57fa640 1 -- 172.21.15.28:0/616652210 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d8053070 con 0x5653bffafa00 2024-04-21T08:58:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.418+0000 7f27c57fa640 1 -- 172.21.15.28:0/616652210 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27d80056e0 con 0x5653bffafa00 2024-04-21T08:58:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.418+0000 7f27c57fa640 1 -- 172.21.15.28:0/616652210 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d8003960 con 0x5653bffafa00 2024-04-21T08:58:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.419+0000 7f27e8df5740 1 -- 172.21.15.28:0/616652210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffafa00 msgr2=0x5653bffaf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.419+0000 7f27e8df5740 1 --2- 172.21.15.28:0/616652210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffafa00 0x5653bffaf340 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f27d8009900 tx=0x7f27d8048000 comp rx=0 tx=0).stop 2024-04-21T08:58:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.419+0000 7f27e8df5740 1 -- 172.21.15.28:0/616652210 shutdown_connections 2024-04-21T08:58:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.419+0000 7f27e8df5740 1 --2- 172.21.15.28:0/616652210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffafa00 0x5653bffaf340 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.419+0000 7f27e8df5740 1 --2- 172.21.15.28:0/616652210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffa6e80 0x5653bffaee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.419+0000 7f27e8df5740 1 --2- 172.21.15.28:0/616652210 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffa64d0 0x5653bffbb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.419+0000 7f27e8df5740 1 -- 172.21.15.28:0/616652210 >> 172.21.15.28:0/616652210 conn(0x5653bffa0ba0 msgr2=0x5653bffa2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.419+0000 7f27e8df5740 1 -- 172.21.15.28:0/616652210 shutdown_connections 2024-04-21T08:58:40.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.419+0000 7f27e8df5740 1 -- 172.21.15.28:0/616652210 wait complete. 2024-04-21T08:58:40.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.419+0000 7f27e8df5740 1 -- start start 2024-04-21T08:58:40.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.419+0000 7f27e8df5740 1 Processor -- start 2024-04-21T08:58:40.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.419+0000 7f27e8df5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa64d0 0x5653bffa92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.419+0000 7f27e8df5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffa6e80 0x5653bffa9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27e8df5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffafa00 0x5653bffa9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27e8df5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01adb30 con 0x5653bffa64d0 2024-04-21T08:58:40.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27e8df5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01aefa0 con 0x5653bffafa00 2024-04-21T08:58:40.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27e8df5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01ae200 con 0x5653bffa6e80 2024-04-21T08:58:40.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27e8df5740 10 client.0 _ll_register_callbacks cb 0x7f27d0013960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:40.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa64d0 0x5653bffa92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa64d0 0x5653bffa92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33528/0 (socket says 172.21.15.28:33528) 2024-04-21T08:58:40.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27df7fe640 1 -- 172.21.15.28:0/661679768 learned_addr learned my addr 172.21.15.28:0/661679768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27dffff640 1 --2- 172.21.15.28:0/661679768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffafa00 0x5653bffa9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27df7fe640 1 -- 172.21.15.28:0/661679768 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffa6e80 msgr2=0x5653bffa9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27df7fe640 1 --2- 172.21.15.28:0/661679768 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffa6e80 0x5653bffa9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27df7fe640 1 -- 172.21.15.28:0/661679768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffafa00 msgr2=0x5653bffa9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27df7fe640 1 --2- 172.21.15.28:0/661679768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffafa00 0x5653bffa9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27df7fe640 1 -- 172.21.15.28:0/661679768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653c00252d0 con 0x5653bffa64d0 2024-04-21T08:58:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27dffff640 1 --2- 172.21.15.28:0/661679768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffafa00 0x5653bffa9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27df7fe640 1 --2- 172.21.15.28:0/661679768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa64d0 0x5653bffa92d0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f27d0009be0 tx=0x7f27d0011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5bbb8d4f7dc2a5b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27beffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27beffd640 1 -- 172.21.15.28:0/661679768 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d001c980 con 0x5653bffa64d0 2024-04-21T08:58:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27beffd640 1 -- 172.21.15.28:0/661679768 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7f27d001cb20 con 0x5653bffa64d0 2024-04-21T08:58:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27e8df5740 1 -- 172.21.15.28:0/661679768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27d80056e0 con 0x5653bffa64d0 2024-04-21T08:58:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27beffd640 1 -- 172.21.15.28:0/661679768 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f27d0038050 con 0x5653bffa64d0 2024-04-21T08:58:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27e8df5740 10 client.17595 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.420+0000 7f27e8df5740 1 -- 172.21.15.28:0/661679768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5653c00264e0 con 0x5653bffa64d0 2024-04-21T08:58:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.421+0000 7f27beffd640 1 -- 172.21.15.28:0/661679768 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27d80056e0 con 0x5653bffa64d0 2024-04-21T08:58:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.421+0000 7f27beffd640 1 -- 172.21.15.28:0/661679768 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f27d001a050 con 0x5653bffa64d0 2024-04-21T08:58:40.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.421+0000 7f27e8df5740 10 client.17595 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:40.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.421+0000 7f27e8df5740 10 client.17595 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:58:40.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.421+0000 7f27e8df5740 20 client.17595 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:40.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.421+0000 7f27e8df5740 1 --2- 172.21.15.28:0/661679768 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5653bffbfd60 0x5653c0028d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.421+0000 7f27e8df5740 4 client.17595 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:58:40.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.421+0000 7f27e8df5740 1 -- 172.21.15.28:0/661679768 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5653bff18c10 con 0x5653bffbfd60 2024-04-21T08:58:40.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.421+0000 7f27deffd640 1 --2- 172.21.15.28:0/661679768 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5653bffbfd60 0x5653c0028d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.421+0000 7f27beffd640 10 client.17595 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:58:40.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.421+0000 7f27deffd640 1 --2- 172.21.15.28:0/661679768 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5653bffbfd60 0x5653c0028d20 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6005fb0ed1383f9 server_cookie=dfd376196892a5f3 in_seq=0 out_seq=0 2024-04-21T08:58:40.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27beffd640 1 -- 172.21.15.28:0/661679768 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5653bff18c10 con 0x5653bffbfd60 2024-04-21T08:58:40.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27beffd640 10 client.17595 handle_command_reply: tid=0 2024-04-21T08:58:40.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e8df5740 1 client.17595 shutdown 2024-04-21T08:58:40.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e8df5740 1 -- 172.21.15.28:0/661679768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa64d0 msgr2=0x5653bffa92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e8df5740 1 --2- 172.21.15.28:0/661679768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa64d0 0x5653bffa92d0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f27d0009be0 tx=0x7f27d0011da0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e8df5740 1 -- 172.21.15.28:0/661679768 shutdown_connections 2024-04-21T08:58:40.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e8df5740 1 --2- 172.21.15.28:0/661679768 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5653bffbfd60 0x5653c0028d20 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e8df5740 1 --2- 172.21.15.28:0/661679768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653bffafa00 0x5653bffa9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e8df5740 1 --2- 172.21.15.28:0/661679768 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5653bffa6e80 0x5653bffa9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27beffd640 0 client.17595 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:58:40.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e8df5740 1 --2- 172.21.15.28:0/661679768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653bffa64d0 0x5653bffa92d0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e8df5740 1 -- 172.21.15.28:0/661679768 >> 172.21.15.28:0/661679768 conn(0x5653bffa0ba0 msgr2=0x5653bff9f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e8df5740 1 -- 172.21.15.28:0/661679768 shutdown_connections 2024-04-21T08:58:40.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e8df5740 1 -- 172.21.15.28:0/661679768 wait complete. 2024-04-21T08:58:40.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e8df5740 20 client.17595 trim_cache size 0 max 16384 2024-04-21T08:58:40.435 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:58:40.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e6206640 1 -- 172.21.15.28:0/118633456 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f27c80a3d20 msgr2=0x7f27c80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e6206640 1 --2- 172.21.15.28:0/118633456 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f27c80a3d20 0x7f27c80a61e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f27d8006010 tx=0x7f27d80043e0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e6206640 1 -- 172.21.15.28:0/118633456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e0106160 msgr2=0x7f27e0198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.422+0000 7f27e6206640 1 --2- 172.21.15.28:0/118633456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e0106160 0x7f27e0198c60 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f27d400d770 tx=0x7f27d400dc30 comp rx=0 tx=0).stop 2024-04-21T08:58:40.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.423+0000 7f27e6206640 1 -- 172.21.15.28:0/118633456 shutdown_connections 2024-04-21T08:58:40.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.423+0000 7f27e6206640 1 --2- 172.21.15.28:0/118633456 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f27c80a3d20 0x7f27c80a61e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.423+0000 7f27e6206640 1 --2- 172.21.15.28:0/118633456 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f27e019cfe0 0x7f27e019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.423+0000 7f27e6206640 1 --2- 172.21.15.28:0/118633456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e0106160 0x7f27e0198c60 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.423+0000 7f27e6206640 1 --2- 172.21.15.28:0/118633456 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f27e0101090 0x7f27e0198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.423+0000 7f27e6206640 1 -- 172.21.15.28:0/118633456 >> 172.21.15.28:0/118633456 conn(0x7f27e0077f50 msgr2=0x7f27e0069410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.423+0000 7f27e6206640 1 -- 172.21.15.28:0/118633456 shutdown_connections 2024-04-21T08:58:40.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.423+0000 7f27e6206640 1 -- 172.21.15.28:0/118633456 wait complete. 2024-04-21T08:58:40.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:58:40.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.543+0000 7f2590cc8640 1 -- 172.21.15.28:0/3553238053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f258c107270 msgr2=0x7f258c10ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.543+0000 7f2590cc8640 1 --2- 172.21.15.28:0/3553238053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f258c107270 0x7f258c10ee40 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f2584009ca0 tx=0x7f258402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.543+0000 7f2590cc8640 1 -- 172.21.15.28:0/3553238053 shutdown_connections 2024-04-21T08:58:40.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.543+0000 7f2590cc8640 1 --2- 172.21.15.28:0/3553238053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f258c107270 0x7f258c10ee40 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.543+0000 7f2590cc8640 1 --2- 172.21.15.28:0/3553238053 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f258c06b1f0 0x7f258c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.543+0000 7f2590cc8640 1 --2- 172.21.15.28:0/3553238053 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f258c06a840 0x7f258c06ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.543+0000 7f2590cc8640 1 -- 172.21.15.28:0/3553238053 >> 172.21.15.28:0/3553238053 conn(0x7f258c077eb0 msgr2=0x7f258c0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.543+0000 7f2590cc8640 1 -- 172.21.15.28:0/3553238053 shutdown_connections 2024-04-21T08:58:40.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.543+0000 7f2590cc8640 1 -- 172.21.15.28:0/3553238053 wait complete. 2024-04-21T08:58:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.544+0000 7f2590cc8640 1 Processor -- start 2024-04-21T08:58:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.544+0000 7f2590cc8640 1 -- start start 2024-04-21T08:58:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.544+0000 7f2590cc8640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f258c06a840 0x7f258c100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.544+0000 7f2590cc8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f258c107270 0x7f258c1013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.544+0000 7f2590cc8640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f258c101900 0x7f258c1a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.544+0000 7f2590cc8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f258c1121d0 con 0x7f258c107270 2024-04-21T08:58:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.544+0000 7f2590cc8640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f258c112050 con 0x7f258c101900 2024-04-21T08:58:40.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.544+0000 7f2590cc8640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f258c112350 con 0x7f258c06a840 2024-04-21T08:58:40.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.544+0000 7f2583fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f258c107270 0x7f258c1013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.544+0000 7f2583fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f258c107270 0x7f258c1013c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33548/0 (socket says 172.21.15.28:33548) 2024-04-21T08:58:40.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.544+0000 7f2583fff640 1 -- 172.21.15.28:0/3973480968 learned_addr learned my addr 172.21.15.28:0/3973480968 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.545+0000 7f258a59c640 1 --2- 172.21.15.28:0/3973480968 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f258c06a840 0x7f258c100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.545+0000 7f258ad9d640 1 --2- 172.21.15.28:0/3973480968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f258c101900 0x7f258c1a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.545+0000 7f2583fff640 1 -- 172.21.15.28:0/3973480968 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f258c06a840 msgr2=0x7f258c100e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.545+0000 7f2583fff640 1 --2- 172.21.15.28:0/3973480968 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f258c06a840 0x7f258c100e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.545+0000 7f2583fff640 1 -- 172.21.15.28:0/3973480968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f258c101900 msgr2=0x7f258c1a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.545+0000 7f2583fff640 1 --2- 172.21.15.28:0/3973480968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f258c101900 0x7f258c1a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.545+0000 7f2583fff640 1 -- 172.21.15.28:0/3973480968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f258c1a6f80 con 0x7f258c107270 2024-04-21T08:58:40.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.545+0000 7f258a59c640 1 --2- 172.21.15.28:0/3973480968 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f258c06a840 0x7f258c100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:40.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.545+0000 7f2583fff640 1 --2- 172.21.15.28:0/3973480968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f258c107270 0x7f258c1013c0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f257400d770 tx=0x7f257400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.545+0000 7f25837fe640 1 -- 172.21.15.28:0/3973480968 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2574017070 con 0x7f258c107270 2024-04-21T08:58:40.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.545+0000 7f2590cc8640 1 -- 172.21.15.28:0/3973480968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f258c1a7270 con 0x7f258c107270 2024-04-21T08:58:40.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.545+0000 7f25837fe640 1 -- 172.21.15.28:0/3973480968 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2574009c30 con 0x7f258c107270 2024-04-21T08:58:40.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.545+0000 7f25837fe640 1 -- 172.21.15.28:0/3973480968 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257400f620 con 0x7f258c107270 2024-04-21T08:58:40.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.545+0000 7f2590cc8640 1 -- 172.21.15.28:0/3973480968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f258c1a77c0 con 0x7f258c107270 2024-04-21T08:58:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.548+0000 7f25837fe640 1 -- 172.21.15.28:0/3973480968 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f257401a070 con 0x7f258c107270 2024-04-21T08:58:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.548+0000 7f2593998740 1 Processor -- start 2024-04-21T08:58:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.548+0000 7f2593998740 1 -- start start 2024-04-21T08:58:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.548+0000 7f2593998740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982c4d0 0x56292982c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.548+0000 7f2593998740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56292982ce80 0x562929835430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.548+0000 7f2593998740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562929835a00 0x562929837e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.548+0000 7f2593998740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5629297911f0 con 0x562929835a00 2024-04-21T08:58:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.548+0000 7f2593998740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56292972e4b0 con 0x56292982ce80 2024-04-21T08:58:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.548+0000 7f2593998740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56292978d1b0 con 0x56292982c4d0 2024-04-21T08:58:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.548+0000 7f2583fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562929835a00 0x562929837e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.548+0000 7f2583fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562929835a00 0x562929837e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33564/0 (socket says 172.21.15.28:33564) 2024-04-21T08:58:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.548+0000 7f2583fff640 1 -- 172.21.15.28:0/2708620433 learned_addr learned my addr 172.21.15.28:0/2708620433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.548+0000 7f258ad9d640 1 --2- 172.21.15.28:0/2708620433 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982c4d0 0x56292982c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.549+0000 7f258a59c640 1 --2- 172.21.15.28:0/2708620433 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56292982ce80 0x562929835430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.549+0000 7f25837fe640 1 --2- 172.21.15.28:0/3973480968 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f25780a3d50 0x7f25780a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.549+0000 7f2583fff640 1 -- 172.21.15.28:0/2708620433 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982c4d0 msgr2=0x56292982c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.549+0000 7f2583fff640 1 --2- 172.21.15.28:0/2708620433 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982c4d0 0x56292982c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.549+0000 7f2583fff640 1 -- 172.21.15.28:0/2708620433 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56292982ce80 msgr2=0x562929835430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.549+0000 7f2583fff640 1 --2- 172.21.15.28:0/2708620433 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56292982ce80 0x562929835430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.549+0000 7f2583fff640 1 -- 172.21.15.28:0/2708620433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629297201b0 con 0x562929835a00 2024-04-21T08:58:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.549+0000 7f258ad9d640 1 --2- 172.21.15.28:0/3973480968 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f25780a3d50 0x7f25780a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.549+0000 7f25837fe640 1 -- 172.21.15.28:0/3973480968 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7f25740d0730 con 0x7f258c107270 2024-04-21T08:58:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.549+0000 7f2583fff640 1 --2- 172.21.15.28:0/2708620433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562929835a00 0x562929837e50 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f25740cda00 tx=0x7f25740b09c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d309368937958935 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.549+0000 7f2580ff9640 1 -- 172.21.15.28:0/2708620433 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2574021030 con 0x562929835a00 2024-04-21T08:58:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.549+0000 7f2580ff9640 1 -- 172.21.15.28:0/2708620433 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2574096e50 con 0x562929835a00 2024-04-21T08:58:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.549+0000 7f2580ff9640 1 -- 172.21.15.28:0/2708620433 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257409bc40 con 0x562929835a00 2024-04-21T08:58:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.549+0000 7f258ad9d640 1 --2- 172.21.15.28:0/3973480968 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f25780a3d50 0x7f25780a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2584005e30 tx=0x7f258402e040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.550+0000 7f2593998740 1 -- 172.21.15.28:0/2708620433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562929835a00 msgr2=0x562929837e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.550+0000 7f2593998740 1 --2- 172.21.15.28:0/2708620433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562929835a00 0x562929837e50 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f25740cda00 tx=0x7f25740b09c0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.550+0000 7f2593998740 1 -- 172.21.15.28:0/2708620433 shutdown_connections 2024-04-21T08:58:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.550+0000 7f2593998740 1 --2- 172.21.15.28:0/2708620433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562929835a00 0x562929837e50 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.550+0000 7f2593998740 1 --2- 172.21.15.28:0/2708620433 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56292982ce80 0x562929835430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.550+0000 7f2593998740 1 --2- 172.21.15.28:0/2708620433 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982c4d0 0x56292982c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.550+0000 7f2593998740 1 -- 172.21.15.28:0/2708620433 >> 172.21.15.28:0/2708620433 conn(0x562929826ba0 msgr2=0x562929826f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.550+0000 7f2593998740 1 -- 172.21.15.28:0/2708620433 shutdown_connections 2024-04-21T08:58:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.550+0000 7f2593998740 1 -- 172.21.15.28:0/2708620433 wait complete. 2024-04-21T08:58:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.550+0000 7f2593998740 1 -- start start 2024-04-21T08:58:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.550+0000 7f2593998740 1 Processor -- start 2024-04-21T08:58:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f2593998740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982c4d0 0x5629298352d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f2593998740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56292982ce80 0x562929832660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f2593998740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562929835a00 0x562929832bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f2593998740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5629298388b0 con 0x562929835a00 2024-04-21T08:58:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f2593998740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56292974a500 con 0x56292982ce80 2024-04-21T08:58:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f2593998740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x562929838a30 con 0x56292982c4d0 2024-04-21T08:58:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f2593998740 10 client.0 _ll_register_callbacks cb 0x562929718b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f258ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562929835a00 0x562929832bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f258ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562929835a00 0x562929832bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33576/0 (socket says 172.21.15.28:33576) 2024-04-21T08:58:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f258ad9d640 1 -- 172.21.15.28:0/3156381702 learned_addr learned my addr 172.21.15.28:0/3156381702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f2583fff640 1 --2- 172.21.15.28:0/3156381702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56292982ce80 0x562929832660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f258a59c640 1 --2- 172.21.15.28:0/3156381702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982c4d0 0x5629298352d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f258ad9d640 1 -- 172.21.15.28:0/3156381702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982c4d0 msgr2=0x5629298352d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f258ad9d640 1 --2- 172.21.15.28:0/3156381702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982c4d0 0x5629298352d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f258ad9d640 1 -- 172.21.15.28:0/3156381702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56292982ce80 msgr2=0x562929832660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f258ad9d640 1 --2- 172.21.15.28:0/3156381702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56292982ce80 0x562929832660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f258ad9d640 1 -- 172.21.15.28:0/3156381702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629294a33f0 con 0x562929835a00 2024-04-21T08:58:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f258ad9d640 1 --2- 172.21.15.28:0/3156381702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562929835a00 0x562929832bc0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f2584009ca0 tx=0x7f2584005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55727b714844218e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f2571ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:40.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f2571ffb640 1 -- 172.21.15.28:0/3156381702 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584051060 con 0x562929835a00 2024-04-21T08:58:40.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.551+0000 7f2571ffb640 1 -- 172.21.15.28:0/3156381702 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7f2584003b70 con 0x562929835a00 2024-04-21T08:58:40.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.552+0000 7f2593998740 1 -- 172.21.15.28:0/3156381702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2574096e50 con 0x562929835a00 2024-04-21T08:58:40.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.552+0000 7f2571ffb640 1 -- 172.21.15.28:0/3156381702 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2584058050 con 0x562929835a00 2024-04-21T08:58:40.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.552+0000 7f2593998740 10 client.17607 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:40.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.552+0000 7f2593998740 1 -- 172.21.15.28:0/3156381702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56292956d7d0 con 0x562929835a00 2024-04-21T08:58:40.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.552+0000 7f2571ffb640 1 -- 172.21.15.28:0/3156381702 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2574096e50 con 0x562929835a00 2024-04-21T08:58:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.552+0000 7f2571ffb640 1 -- 172.21.15.28:0/3156381702 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f258403f040 con 0x562929835a00 2024-04-21T08:58:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.552+0000 7f2593998740 10 client.17607 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.552+0000 7f2593998740 10 client.17607 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:58:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.552+0000 7f2593998740 20 client.17607 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.552+0000 7f2593998740 1 --2- 172.21.15.28:0/3156381702 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x562929845b60 0x562929847f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.552+0000 7f2593998740 4 client.17607 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:58:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.552+0000 7f2593998740 1 -- 172.21.15.28:0/3156381702 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562929730930 con 0x562929845b60 2024-04-21T08:58:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.553+0000 7f258a59c640 1 --2- 172.21.15.28:0/3156381702 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x562929845b60 0x562929847f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.553+0000 7f258a59c640 1 --2- 172.21.15.28:0/3156381702 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x562929845b60 0x562929847f70 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=718c445e9989db88 server_cookie=9bb13608afe4707 in_seq=0 out_seq=0 2024-04-21T08:58:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.553+0000 7f2571ffb640 10 client.17607 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:58:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.554+0000 7f2571ffb640 1 -- 172.21.15.28:0/3156381702 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562929730930 con 0x562929845b60 2024-04-21T08:58:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.554+0000 7f2571ffb640 10 client.17607 handle_command_reply: tid=0 2024-04-21T08:58:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.554+0000 7f2593998740 1 client.17607 shutdown 2024-04-21T08:58:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.554+0000 7f2593998740 1 -- 172.21.15.28:0/3156381702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562929835a00 msgr2=0x562929832bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.554+0000 7f2593998740 1 --2- 172.21.15.28:0/3156381702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562929835a00 0x562929832bc0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f2584009ca0 tx=0x7f2584005d80 comp rx=0 tx=0).stop 2024-04-21T08:58:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.554+0000 7f2593998740 1 -- 172.21.15.28:0/3156381702 shutdown_connections 2024-04-21T08:58:40.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.554+0000 7f2593998740 1 --2- 172.21.15.28:0/3156381702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562929835a00 0x562929832bc0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.554+0000 7f2593998740 1 --2- 172.21.15.28:0/3156381702 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56292982ce80 0x562929832660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.554+0000 7f2593998740 1 --2- 172.21.15.28:0/3156381702 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x562929845b60 0x562929847f70 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.554+0000 7f2593998740 1 --2- 172.21.15.28:0/3156381702 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982c4d0 0x5629298352d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.554+0000 7f2593998740 1 -- 172.21.15.28:0/3156381702 >> 172.21.15.28:0/3156381702 conn(0x562929826ba0 msgr2=0x5629298288f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.554+0000 7f2571ffb640 0 client.17607 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:58:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.554+0000 7f2593998740 1 -- 172.21.15.28:0/3156381702 shutdown_connections 2024-04-21T08:58:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.554+0000 7f2593998740 1 -- 172.21.15.28:0/3156381702 wait complete. 2024-04-21T08:58:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.554+0000 7f2593998740 20 client.17607 trim_cache size 0 max 16384 2024-04-21T08:58:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.567+0000 7f2593998740 1 Processor -- start 2024-04-21T08:58:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.567+0000 7f2593998740 1 -- start start 2024-04-21T08:58:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f2593998740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56292982c4d0 0x562929841770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f2593998740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982ce80 0x562929834d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f2593998740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562929835a00 0x5629298352d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f2593998740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f258c1b3ec0 con 0x56292982c4d0 2024-04-21T08:58:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f2593998740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f258c1b1790 con 0x562929835a00 2024-04-21T08:58:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f2593998740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f258c1af600 con 0x56292982ce80 2024-04-21T08:58:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f258a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56292982c4d0 0x562929841770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f258a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56292982c4d0 0x562929841770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33582/0 (socket says 172.21.15.28:33582) 2024-04-21T08:58:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f258a59c640 1 -- 172.21.15.28:0/3529177615 learned_addr learned my addr 172.21.15.28:0/3529177615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f2583fff640 1 --2- 172.21.15.28:0/3529177615 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982ce80 0x562929834d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f258a59c640 1 -- 172.21.15.28:0/3529177615 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982ce80 msgr2=0x562929834d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f258a59c640 1 --2- 172.21.15.28:0/3529177615 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982ce80 0x562929834d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f258a59c640 1 -- 172.21.15.28:0/3529177615 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562929835a00 msgr2=0x5629298352d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f258a59c640 1 --2- 172.21.15.28:0/3529177615 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562929835a00 0x5629298352d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f258a59c640 1 -- 172.21.15.28:0/3529177615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562929830bb0 con 0x56292982c4d0 2024-04-21T08:58:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f258a59c640 1 --2- 172.21.15.28:0/3529177615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56292982c4d0 0x562929841770 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f257c005940 tx=0x7f257c0128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f8ff8db6bf7fca1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f2573fff640 1 -- 172.21.15.28:0/3529177615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c022030 con 0x56292982c4d0 2024-04-21T08:58:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f2573fff640 1 -- 172.21.15.28:0/3529177615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f257c005020 con 0x56292982c4d0 2024-04-21T08:58:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.568+0000 7f2573fff640 1 -- 172.21.15.28:0/3529177615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c004b20 con 0x56292982c4d0 2024-04-21T08:58:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.569+0000 7f2593998740 1 -- 172.21.15.28:0/3529177615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56292982c4d0 msgr2=0x562929841770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.569+0000 7f2593998740 1 --2- 172.21.15.28:0/3529177615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56292982c4d0 0x562929841770 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f257c005940 tx=0x7f257c0128e0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.569+0000 7f2593998740 1 -- 172.21.15.28:0/3529177615 shutdown_connections 2024-04-21T08:58:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.569+0000 7f2593998740 1 --2- 172.21.15.28:0/3529177615 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562929835a00 0x5629298352d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.569+0000 7f2593998740 1 --2- 172.21.15.28:0/3529177615 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982ce80 0x562929834d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.569+0000 7f2593998740 1 --2- 172.21.15.28:0/3529177615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56292982c4d0 0x562929841770 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.569+0000 7f2593998740 1 -- 172.21.15.28:0/3529177615 >> 172.21.15.28:0/3529177615 conn(0x562929826ba0 msgr2=0x562929828880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.569+0000 7f2593998740 1 -- 172.21.15.28:0/3529177615 shutdown_connections 2024-04-21T08:58:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.569+0000 7f2593998740 1 -- 172.21.15.28:0/3529177615 wait complete. 2024-04-21T08:58:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.569+0000 7f2593998740 1 -- start start 2024-04-21T08:58:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.569+0000 7f2593998740 1 Processor -- start 2024-04-21T08:58:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f2593998740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56292982c4d0 0x562929839820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f2593998740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982ce80 0x562929839d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f2593998740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562929835a00 0x56292983a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f2593998740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f258c1afc50 con 0x56292982c4d0 2024-04-21T08:58:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f2593998740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f258c1b10c0 con 0x562929835a00 2024-04-21T08:58:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f2593998740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f258c1b0320 con 0x56292982ce80 2024-04-21T08:58:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f2593998740 10 client.0 _ll_register_callbacks cb 0x7f2584032f80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f258a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56292982c4d0 0x562929839820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f258a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56292982c4d0 0x562929839820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33588/0 (socket says 172.21.15.28:33588) 2024-04-21T08:58:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f258a59c640 1 -- 172.21.15.28:0/3049380679 learned_addr learned my addr 172.21.15.28:0/3049380679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f2583fff640 1 --2- 172.21.15.28:0/3049380679 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982ce80 0x562929839d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f258ad9d640 1 --2- 172.21.15.28:0/3049380679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562929835a00 0x56292983a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f258a59c640 1 -- 172.21.15.28:0/3049380679 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982ce80 msgr2=0x562929839d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f258a59c640 1 --2- 172.21.15.28:0/3049380679 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982ce80 0x562929839d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f258a59c640 1 -- 172.21.15.28:0/3049380679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562929835a00 msgr2=0x56292983a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f258a59c640 1 --2- 172.21.15.28:0/3049380679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562929835a00 0x56292983a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f258a59c640 1 -- 172.21.15.28:0/3049380679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629298ab260 con 0x56292982c4d0 2024-04-21T08:58:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f2583fff640 1 --2- 172.21.15.28:0/3049380679 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982ce80 0x562929839d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f258a59c640 1 --2- 172.21.15.28:0/3049380679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56292982c4d0 0x562929839820 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f257c01e700 tx=0x7f257c01e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f808f1484611a904 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f2571ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f2571ffb640 1 -- 172.21.15.28:0/3049380679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c022040 con 0x56292982c4d0 2024-04-21T08:58:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f2571ffb640 1 -- 172.21.15.28:0/3049380679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7f257c0053d0 con 0x56292982c4d0 2024-04-21T08:58:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.570+0000 7f2593998740 1 -- 172.21.15.28:0/3049380679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f257c005020 con 0x56292982c4d0 2024-04-21T08:58:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.571+0000 7f2571ffb640 1 -- 172.21.15.28:0/3049380679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f257c038050 con 0x56292982c4d0 2024-04-21T08:58:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.571+0000 7f2593998740 10 client.17613 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.571+0000 7f2593998740 1 -- 172.21.15.28:0/3049380679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5629298ac520 con 0x56292982c4d0 2024-04-21T08:58:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.571+0000 7f2571ffb640 1 -- 172.21.15.28:0/3049380679 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f257c005020 con 0x56292982c4d0 2024-04-21T08:58:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.571+0000 7f2571ffb640 1 -- 172.21.15.28:0/3049380679 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f257c0276a0 con 0x56292982c4d0 2024-04-21T08:58:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.571+0000 7f2593998740 10 client.17613 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.571+0000 7f2593998740 10 client.17613 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:58:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.571+0000 7f2593998740 20 client.17613 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.571+0000 7f2593998740 1 --2- 172.21.15.28:0/3049380679 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5629298acc40 0x5629298af060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.571+0000 7f2593998740 4 client.17613 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:58:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.571+0000 7f2593998740 1 -- 172.21.15.28:0/3049380679 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56292979ec10 con 0x5629298acc40 2024-04-21T08:58:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.571+0000 7f2583fff640 1 --2- 172.21.15.28:0/3049380679 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5629298acc40 0x5629298af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2571ffb640 10 client.17613 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:58:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2583fff640 1 --2- 172.21.15.28:0/3049380679 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5629298acc40 0x5629298af060 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b1cb737b80ea57c3 server_cookie=7249de57d6230aea in_seq=0 out_seq=0 2024-04-21T08:58:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2571ffb640 1 -- 172.21.15.28:0/3049380679 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56292979ec10 con 0x5629298acc40 2024-04-21T08:58:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2571ffb640 10 client.17613 handle_command_reply: tid=0 2024-04-21T08:58:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2593998740 1 client.17613 shutdown 2024-04-21T08:58:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2593998740 1 -- 172.21.15.28:0/3049380679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56292982c4d0 msgr2=0x562929839820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2593998740 1 --2- 172.21.15.28:0/3049380679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56292982c4d0 0x562929839820 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f257c01e700 tx=0x7f257c01e730 comp rx=0 tx=0).stop 2024-04-21T08:58:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2593998740 1 -- 172.21.15.28:0/3049380679 shutdown_connections 2024-04-21T08:58:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2593998740 1 --2- 172.21.15.28:0/3049380679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562929835a00 0x56292983a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2593998740 1 --2- 172.21.15.28:0/3049380679 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5629298acc40 0x5629298af060 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2593998740 1 --2- 172.21.15.28:0/3049380679 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56292982ce80 0x562929839d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2593998740 1 --2- 172.21.15.28:0/3049380679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56292982c4d0 0x562929839820 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2571ffb640 0 client.17613 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:58:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2593998740 1 -- 172.21.15.28:0/3049380679 >> 172.21.15.28:0/3049380679 conn(0x562929826ba0 msgr2=0x56292982d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2593998740 1 -- 172.21.15.28:0/3049380679 shutdown_connections 2024-04-21T08:58:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2593998740 1 -- 172.21.15.28:0/3049380679 wait complete. 2024-04-21T08:58:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.572+0000 7f2593998740 20 client.17613 trim_cache size 0 max 16384 2024-04-21T08:58:40.583 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:58:40.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.573+0000 7f2590cc8640 1 -- 172.21.15.28:0/3973480968 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f25780a3d50 msgr2=0x7f25780a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.573+0000 7f2590cc8640 1 --2- 172.21.15.28:0/3973480968 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f25780a3d50 0x7f25780a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2584005e30 tx=0x7f258402e040 comp rx=0 tx=0).stop 2024-04-21T08:58:40.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.573+0000 7f2590cc8640 1 -- 172.21.15.28:0/3973480968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f258c107270 msgr2=0x7f258c1013c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.573+0000 7f2590cc8640 1 --2- 172.21.15.28:0/3973480968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f258c107270 0x7f258c1013c0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f257400d770 tx=0x7f257400dc30 comp rx=0 tx=0).stop 2024-04-21T08:58:40.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.573+0000 7f2590cc8640 1 -- 172.21.15.28:0/3973480968 shutdown_connections 2024-04-21T08:58:40.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.573+0000 7f2590cc8640 1 --2- 172.21.15.28:0/3973480968 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f25780a3d50 0x7f25780a6210 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.573+0000 7f2590cc8640 1 --2- 172.21.15.28:0/3973480968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f258c101900 0x7f258c1a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.573+0000 7f2590cc8640 1 --2- 172.21.15.28:0/3973480968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f258c107270 0x7f258c1013c0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.573+0000 7f2590cc8640 1 --2- 172.21.15.28:0/3973480968 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f258c06a840 0x7f258c100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.573+0000 7f2590cc8640 1 -- 172.21.15.28:0/3973480968 >> 172.21.15.28:0/3973480968 conn(0x7f258c077eb0 msgr2=0x7f258c0fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.573+0000 7f2590cc8640 1 -- 172.21.15.28:0/3973480968 shutdown_connections 2024-04-21T08:58:40.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.573+0000 7f2590cc8640 1 -- 172.21.15.28:0/3973480968 wait complete. 2024-04-21T08:58:40.585 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:40 smithi028 ceph-mon[28003]: pgmap v229: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:58:40.586 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:40 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/771269837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:58:40.587 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:58:40.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.690+0000 7f6731b32640 1 -- 172.21.15.28:0/3337003475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f672c101a40 msgr2=0x7f672c105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.690+0000 7f6731b32640 1 --2- 172.21.15.28:0/3337003475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f672c101a40 0x7f672c105a80 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f6720009900 tx=0x7f672002bc30 comp rx=0 tx=0).stop 2024-04-21T08:58:40.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.690+0000 7f6731b32640 1 -- 172.21.15.28:0/3337003475 shutdown_connections 2024-04-21T08:58:40.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.690+0000 7f6731b32640 1 --2- 172.21.15.28:0/3337003475 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f672c106160 0x7f672c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.690+0000 7f6731b32640 1 --2- 172.21.15.28:0/3337003475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f672c101a40 0x7f672c105a80 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.691+0000 7f6731b32640 1 --2- 172.21.15.28:0/3337003475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f672c101090 0x7f672c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.691+0000 7f6731b32640 1 -- 172.21.15.28:0/3337003475 >> 172.21.15.28:0/3337003475 conn(0x7f672c077f50 msgr2=0x7f672c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.691+0000 7f6731b32640 1 -- 172.21.15.28:0/3337003475 shutdown_connections 2024-04-21T08:58:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.691+0000 7f6731b32640 1 -- 172.21.15.28:0/3337003475 wait complete. 2024-04-21T08:58:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.691+0000 7f6731b32640 1 Processor -- start 2024-04-21T08:58:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.691+0000 7f6731b32640 1 -- start start 2024-04-21T08:58:40.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f6731b32640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f672c101090 0x7f672c198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f6731b32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f672c106160 0x7f672c198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f6731b32640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f672c19cef0 0x7f672c19d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f6731b32640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f672c1121d0 con 0x7f672c106160 2024-04-21T08:58:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f6731b32640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f672c112050 con 0x7f672c19cef0 2024-04-21T08:58:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f6731b32640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f672c112350 con 0x7f672c101090 2024-04-21T08:58:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f672affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f672c106160 0x7f672c198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f672affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f672c106160 0x7f672c198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33600/0 (socket says 172.21.15.28:33600) 2024-04-21T08:58:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f672affd640 1 -- 172.21.15.28:0/772425345 learned_addr learned my addr 172.21.15.28:0/772425345 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f672bfff640 1 --2- 172.21.15.28:0/772425345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f672c19cef0 0x7f672c19d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f672b7fe640 1 --2- 172.21.15.28:0/772425345 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f672c101090 0x7f672c198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f672affd640 1 -- 172.21.15.28:0/772425345 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f672c101090 msgr2=0x7f672c198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f672affd640 1 --2- 172.21.15.28:0/772425345 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f672c101090 0x7f672c198630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f672affd640 1 -- 172.21.15.28:0/772425345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f672c19cef0 msgr2=0x7f672c19d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f672affd640 1 --2- 172.21.15.28:0/772425345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f672c19cef0 0x7f672c19d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f672affd640 1 -- 172.21.15.28:0/772425345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672c19d9e0 con 0x7f672c106160 2024-04-21T08:58:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f672bfff640 1 --2- 172.21.15.28:0/772425345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f672c19cef0 0x7f672c19d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f672affd640 1 --2- 172.21.15.28:0/772425345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f672c106160 0x7f672c198b70 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f6720005ec0 tx=0x7f672002ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f6728ff9640 1 -- 172.21.15.28:0/772425345 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720043020 con 0x7f672c106160 2024-04-21T08:58:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f6728ff9640 1 -- 172.21.15.28:0/772425345 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f672002cc80 con 0x7f672c106160 2024-04-21T08:58:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f6731b32640 1 -- 172.21.15.28:0/772425345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f672c19dc70 con 0x7f672c106160 2024-04-21T08:58:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f6728ff9640 1 -- 172.21.15.28:0/772425345 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720003c90 con 0x7f672c106160 2024-04-21T08:58:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.692+0000 7f6731b32640 1 -- 172.21.15.28:0/772425345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f672c1a54c0 con 0x7f672c106160 2024-04-21T08:58:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.695+0000 7f6728ff9640 1 -- 172.21.15.28:0/772425345 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6720046020 con 0x7f672c106160 2024-04-21T08:58:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.695+0000 7f6734800740 1 Processor -- start 2024-04-21T08:58:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.695+0000 7f6734800740 1 -- start start 2024-04-21T08:58:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.695+0000 7f6734800740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.695+0000 7f6734800740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cde80 0x556b3e5d6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.695+0000 7f6734800740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e5d8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.695+0000 7f6734800740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556b3e5321f0 con 0x556b3e5cd4d0 2024-04-21T08:58:40.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.695+0000 7f6734800740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556b3e4cf4b0 con 0x556b3e5d6a00 2024-04-21T08:58:40.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.695+0000 7f6734800740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x556b3e52e1b0 con 0x556b3e5cde80 2024-04-21T08:58:40.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.695+0000 7f672bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.695+0000 7f672bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33610/0 (socket says 172.21.15.28:33610) 2024-04-21T08:58:40.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.695+0000 7f672bfff640 1 -- 172.21.15.28:0/2110083646 learned_addr learned my addr 172.21.15.28:0/2110083646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.695+0000 7f6728ff9640 1 --2- 172.21.15.28:0/772425345 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f67140a3da0 0x7f67140a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.695+0000 7f672b7fe640 1 --2- 172.21.15.28:0/2110083646 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cde80 0x556b3e5d6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.695+0000 7f672affd640 1 --2- 172.21.15.28:0/2110083646 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e5d8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.696+0000 7f672bfff640 1 -- 172.21.15.28:0/2110083646 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cde80 msgr2=0x556b3e5d6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.696+0000 7f672bfff640 1 --2- 172.21.15.28:0/2110083646 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cde80 0x556b3e5d6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.696+0000 7f672bfff640 1 -- 172.21.15.28:0/2110083646 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 msgr2=0x556b3e5d8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.696+0000 7f672bfff640 1 --2- 172.21.15.28:0/2110083646 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e5d8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.696+0000 7f672bfff640 1 -- 172.21.15.28:0/2110083646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b3e4c11b0 con 0x556b3e5cd4d0 2024-04-21T08:58:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.696+0000 7f672b7fe640 1 --2- 172.21.15.28:0/2110083646 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cde80 0x556b3e5d6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.696+0000 7f6728ff9640 1 -- 172.21.15.28:0/772425345 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7f67200f2df0 con 0x7f672c106160 2024-04-21T08:58:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.696+0000 7f672bfff640 1 --2- 172.21.15.28:0/772425345 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f67140a3da0 0x7f67140a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.696+0000 7f672bfff640 1 --2- 172.21.15.28:0/2110083646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5cd8b0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f6724008f60 tx=0x7f6724008f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed6e4d3f136d56e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.696+0000 7f6711ffb640 1 -- 172.21.15.28:0/2110083646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672401a8e0 con 0x556b3e5cd4d0 2024-04-21T08:58:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.696+0000 7f6711ffb640 1 -- 172.21.15.28:0/2110083646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6724019410 con 0x556b3e5cd4d0 2024-04-21T08:58:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.696+0000 7f6711ffb640 1 -- 172.21.15.28:0/2110083646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672400e5b0 con 0x556b3e5cd4d0 2024-04-21T08:58:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.696+0000 7f672bfff640 1 --2- 172.21.15.28:0/772425345 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f67140a3da0 0x7f67140a6260 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f672401e000 tx=0x7f672401f000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.697+0000 7f6734800740 1 -- 172.21.15.28:0/2110083646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 msgr2=0x556b3e5cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.697+0000 7f6734800740 1 --2- 172.21.15.28:0/2110083646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5cd8b0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f6724008f60 tx=0x7f6724008f90 comp rx=0 tx=0).stop 2024-04-21T08:58:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.697+0000 7f6734800740 1 -- 172.21.15.28:0/2110083646 shutdown_connections 2024-04-21T08:58:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.697+0000 7f6734800740 1 --2- 172.21.15.28:0/2110083646 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e5d8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.697+0000 7f6734800740 1 --2- 172.21.15.28:0/2110083646 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cde80 0x556b3e5d6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.697+0000 7f6734800740 1 --2- 172.21.15.28:0/2110083646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5cd8b0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.697+0000 7f6734800740 1 -- 172.21.15.28:0/2110083646 >> 172.21.15.28:0/2110083646 conn(0x556b3e5c7ba0 msgr2=0x556b3e5c7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.697+0000 7f6734800740 1 -- 172.21.15.28:0/2110083646 shutdown_connections 2024-04-21T08:58:40.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.697+0000 7f6734800740 1 -- 172.21.15.28:0/2110083646 wait complete. 2024-04-21T08:58:40.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.697+0000 7f6734800740 1 -- start start 2024-04-21T08:58:40.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.697+0000 7f6734800740 1 Processor -- start 2024-04-21T08:58:40.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.697+0000 7f6734800740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5d6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f6734800740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cde80 0x556b3e5d36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f6734800740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e5d3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f6734800740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556b3e5d9970 con 0x556b3e5cd4d0 2024-04-21T08:58:40.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f6734800740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556b3e4eb500 con 0x556b3e5d6a00 2024-04-21T08:58:40.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f6734800740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x556b3e5d9af0 con 0x556b3e5cde80 2024-04-21T08:58:40.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f6734800740 10 client.0 _ll_register_callbacks cb 0x556b3e4b9b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:40.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f672b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5d6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f672b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5d6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33622/0 (socket says 172.21.15.28:33622) 2024-04-21T08:58:40.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f672b7fe640 1 -- 172.21.15.28:0/4042942677 learned_addr learned my addr 172.21.15.28:0/4042942677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f672bfff640 1 --2- 172.21.15.28:0/4042942677 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e5d3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f672b7fe640 1 -- 172.21.15.28:0/4042942677 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cde80 msgr2=0x556b3e5d36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f672b7fe640 1 --2- 172.21.15.28:0/4042942677 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cde80 0x556b3e5d36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f672b7fe640 1 -- 172.21.15.28:0/4042942677 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 msgr2=0x556b3e5d3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f672b7fe640 1 --2- 172.21.15.28:0/4042942677 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e5d3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f672b7fe640 1 -- 172.21.15.28:0/4042942677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b3e2443f0 con 0x556b3e5cd4d0 2024-04-21T08:58:40.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f672bfff640 1 --2- 172.21.15.28:0/4042942677 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e5d3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:40.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f672b7fe640 1 --2- 172.21.15.28:0/4042942677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5d6360 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f671c00f760 tx=0x7f671c00fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd81f532c82e615 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f670affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:40.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f670affd640 1 -- 172.21.15.28:0/4042942677 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c016070 con 0x556b3e5cd4d0 2024-04-21T08:58:40.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f670affd640 1 -- 172.21.15.28:0/4042942677 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7f671c00bcb0 con 0x556b3e5cd4d0 2024-04-21T08:58:40.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f6734800740 1 -- 172.21.15.28:0/4042942677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6724019410 con 0x556b3e5cd4d0 2024-04-21T08:58:40.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f670affd640 1 -- 172.21.15.28:0/4042942677 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f671c00fe70 con 0x556b3e5cd4d0 2024-04-21T08:58:40.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f6734800740 10 client.17625 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:40.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.698+0000 7f6734800740 1 -- 172.21.15.28:0/4042942677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556b3e30e7d0 con 0x556b3e5cd4d0 2024-04-21T08:58:40.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.699+0000 7f670affd640 1 -- 172.21.15.28:0/4042942677 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6724019410 con 0x556b3e5cd4d0 2024-04-21T08:58:40.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.699+0000 7f670affd640 1 -- 172.21.15.28:0/4042942677 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f671c01a4a0 con 0x556b3e5cd4d0 2024-04-21T08:58:40.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.699+0000 7f6734800740 10 client.17625 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:40.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.699+0000 7f6734800740 10 client.17625 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:40.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.699+0000 7f6734800740 20 client.17625 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:40.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.699+0000 7f6734800740 1 --2- 172.21.15.28:0/4042942677 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556b3e5e6cb0 0x556b3e5e9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.699+0000 7f6734800740 4 client.17625 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:58:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.699+0000 7f6734800740 1 -- 172.21.15.28:0/4042942677 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b3e4d1930 con 0x556b3e5e6cb0 2024-04-21T08:58:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.699+0000 7f672affd640 1 --2- 172.21.15.28:0/4042942677 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556b3e5e6cb0 0x556b3e5e9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.700+0000 7f672affd640 1 --2- 172.21.15.28:0/4042942677 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556b3e5e6cb0 0x556b3e5e9150 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=556386391c7555a9 server_cookie=271238f29e6938f9 in_seq=0 out_seq=0 2024-04-21T08:58:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.700+0000 7f670affd640 10 client.17625 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:40.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.700+0000 7f670affd640 1 -- 172.21.15.28:0/4042942677 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556b3e4d1930 con 0x556b3e5e6cb0 2024-04-21T08:58:40.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.700+0000 7f670affd640 10 client.17625 handle_command_reply: tid=0 2024-04-21T08:58:40.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.700+0000 7f6734800740 1 client.17625 shutdown 2024-04-21T08:58:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.701+0000 7f6734800740 1 -- 172.21.15.28:0/4042942677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 msgr2=0x556b3e5d6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.701+0000 7f6734800740 1 --2- 172.21.15.28:0/4042942677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5d6360 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f671c00f760 tx=0x7f671c00fc20 comp rx=0 tx=0).stop 2024-04-21T08:58:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.701+0000 7f6734800740 1 -- 172.21.15.28:0/4042942677 shutdown_connections 2024-04-21T08:58:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.701+0000 7f6734800740 1 --2- 172.21.15.28:0/4042942677 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556b3e5e6cb0 0x556b3e5e9150 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.701+0000 7f6734800740 1 --2- 172.21.15.28:0/4042942677 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e5d3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.701+0000 7f670affd640 0 client.17625 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.701+0000 7f6734800740 1 --2- 172.21.15.28:0/4042942677 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cde80 0x556b3e5d36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.701+0000 7f6734800740 1 --2- 172.21.15.28:0/4042942677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5d6360 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.701+0000 7f6734800740 1 -- 172.21.15.28:0/4042942677 >> 172.21.15.28:0/4042942677 conn(0x556b3e5c7ba0 msgr2=0x556b3e5c98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.701+0000 7f6734800740 1 -- 172.21.15.28:0/4042942677 shutdown_connections 2024-04-21T08:58:40.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.701+0000 7f6734800740 1 -- 172.21.15.28:0/4042942677 wait complete. 2024-04-21T08:58:40.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.701+0000 7f6734800740 20 client.17625 trim_cache size 0 max 16384 2024-04-21T08:58:40.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.714+0000 7f6734800740 1 Processor -- start 2024-04-21T08:58:40.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.714+0000 7f6734800740 1 -- start start 2024-04-21T08:58:40.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.714+0000 7f6734800740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5e2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.714+0000 7f6734800740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cde80 0x556b3e5df120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.714+0000 7f6734800740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e5c6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.714+0000 7f6734800740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f672c1b1c00 con 0x556b3e5cd4d0 2024-04-21T08:58:40.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.714+0000 7f6734800740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f672c1af4d0 con 0x556b3e5d6a00 2024-04-21T08:58:40.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.714+0000 7f6734800740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f672c1ad340 con 0x556b3e5cde80 2024-04-21T08:58:40.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.715+0000 7f672b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5e2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.715+0000 7f672b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5e2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33638/0 (socket says 172.21.15.28:33638) 2024-04-21T08:58:40.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.715+0000 7f672b7fe640 1 -- 172.21.15.28:0/1726188569 learned_addr learned my addr 172.21.15.28:0/1726188569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.715+0000 7f672affd640 1 --2- 172.21.15.28:0/1726188569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cde80 0x556b3e5df120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.715+0000 7f672b7fe640 1 -- 172.21.15.28:0/1726188569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cde80 msgr2=0x556b3e5df120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.715+0000 7f672bfff640 1 --2- 172.21.15.28:0/1726188569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e5c6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.715+0000 7f672b7fe640 1 --2- 172.21.15.28:0/1726188569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cde80 0x556b3e5df120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.715+0000 7f672b7fe640 1 -- 172.21.15.28:0/1726188569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 msgr2=0x556b3e5c6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.715+0000 7f672b7fe640 1 --2- 172.21.15.28:0/1726188569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e5c6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.715+0000 7f672b7fe640 1 -- 172.21.15.28:0/1726188569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b3e5d26a0 con 0x556b3e5cd4d0 2024-04-21T08:58:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.715+0000 7f672bfff640 1 --2- 172.21.15.28:0/1726188569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e5c6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.715+0000 7f672b7fe640 1 --2- 172.21.15.28:0/1726188569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5e2830 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f671c000c00 tx=0x7f671c00d5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0e1f69e02a1b29 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.715+0000 7f67117fa640 1 -- 172.21.15.28:0/1726188569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c016040 con 0x556b3e5cd4d0 2024-04-21T08:58:40.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.715+0000 7f67117fa640 1 -- 172.21.15.28:0/1726188569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f671c01a410 con 0x556b3e5cd4d0 2024-04-21T08:58:40.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.715+0000 7f67117fa640 1 -- 172.21.15.28:0/1726188569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c004640 con 0x556b3e5cd4d0 2024-04-21T08:58:40.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.716+0000 7f6734800740 1 -- 172.21.15.28:0/1726188569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 msgr2=0x556b3e5e2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.716+0000 7f6734800740 1 --2- 172.21.15.28:0/1726188569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5e2830 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f671c000c00 tx=0x7f671c00d5c0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.716+0000 7f6734800740 1 -- 172.21.15.28:0/1726188569 shutdown_connections 2024-04-21T08:58:40.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.716+0000 7f6734800740 1 --2- 172.21.15.28:0/1726188569 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e5c6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.716+0000 7f6734800740 1 --2- 172.21.15.28:0/1726188569 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cde80 0x556b3e5df120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.716+0000 7f6734800740 1 --2- 172.21.15.28:0/1726188569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cd4d0 0x556b3e5e2830 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.716+0000 7f6734800740 1 -- 172.21.15.28:0/1726188569 >> 172.21.15.28:0/1726188569 conn(0x556b3e5c7ba0 msgr2=0x556b3e5c6370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.716+0000 7f6734800740 1 -- 172.21.15.28:0/1726188569 shutdown_connections 2024-04-21T08:58:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.716+0000 7f6734800740 1 -- 172.21.15.28:0/1726188569 wait complete. 2024-04-21T08:58:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.716+0000 7f6734800740 1 -- start start 2024-04-21T08:58:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.716+0000 7f6734800740 1 Processor -- start 2024-04-21T08:58:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f6734800740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cd4d0 0x556b3e645490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f6734800740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cde80 0x556b3e6459d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f6734800740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e650320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f6734800740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f672c1ad990 con 0x556b3e5cde80 2024-04-21T08:58:40.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f6734800740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f672c1aee00 con 0x556b3e5d6a00 2024-04-21T08:58:40.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f6734800740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f672c1ae060 con 0x556b3e5cd4d0 2024-04-21T08:58:40.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f6734800740 10 client.0 _ll_register_callbacks cb 0x7f671c00ded0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:40.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f672affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cde80 0x556b3e6459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f672affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cde80 0x556b3e6459d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33644/0 (socket says 172.21.15.28:33644) 2024-04-21T08:58:40.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f672affd640 1 -- 172.21.15.28:0/2727634599 learned_addr learned my addr 172.21.15.28:0/2727634599 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:40.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f672b7fe640 1 --2- 172.21.15.28:0/2727634599 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cd4d0 0x556b3e645490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f672bfff640 1 --2- 172.21.15.28:0/2727634599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e650320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f672affd640 1 -- 172.21.15.28:0/2727634599 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cd4d0 msgr2=0x556b3e645490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f672affd640 1 --2- 172.21.15.28:0/2727634599 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cd4d0 0x556b3e645490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f672affd640 1 -- 172.21.15.28:0/2727634599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 msgr2=0x556b3e650320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:40.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f672affd640 1 --2- 172.21.15.28:0/2727634599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e650320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f672affd640 1 -- 172.21.15.28:0/2727634599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b3e5d4070 con 0x556b3e5cde80 2024-04-21T08:58:40.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f672affd640 1 --2- 172.21.15.28:0/2727634599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cde80 0x556b3e6459d0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f67200be7b0 tx=0x7f67200904f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2223741102a4e63b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:40.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f670affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:40.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f670affd640 1 -- 172.21.15.28:0/2727634599 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67200beab0 con 0x556b3e5cde80 2024-04-21T08:58:40.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f670affd640 1 -- 172.21.15.28:0/2727634599 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7f67200bcdb0 con 0x556b3e5cde80 2024-04-21T08:58:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.717+0000 7f6734800740 1 -- 172.21.15.28:0/2727634599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f671c01a410 con 0x556b3e5cde80 2024-04-21T08:58:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.718+0000 7f670affd640 1 -- 172.21.15.28:0/2727634599 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f67200c2e60 con 0x556b3e5cde80 2024-04-21T08:58:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.718+0000 7f6734800740 10 client.17631 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.718+0000 7f6734800740 1 -- 172.21.15.28:0/2727634599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556b3e6519e0 con 0x556b3e5cde80 2024-04-21T08:58:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.718+0000 7f670affd640 1 -- 172.21.15.28:0/2727634599 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f671c01a410 con 0x556b3e5cde80 2024-04-21T08:58:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.718+0000 7f670affd640 1 -- 172.21.15.28:0/2727634599 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f67200f03d0 con 0x556b3e5cde80 2024-04-21T08:58:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.718+0000 7f6734800740 10 client.17631 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.718+0000 7f6734800740 10 client.17631 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.718+0000 7f6734800740 20 client.17631 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:40.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.718+0000 7f6734800740 1 --2- 172.21.15.28:0/2727634599 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556b3e5e6cb0 0x556b3e6541a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:40.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.718+0000 7f6734800740 4 client.17631 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:58:40.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.718+0000 7f6734800740 1 -- 172.21.15.28:0/2727634599 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556b3e53fc10 con 0x556b3e5e6cb0 2024-04-21T08:58:40.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.718+0000 7f672b7fe640 1 --2- 172.21.15.28:0/2727634599 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556b3e5e6cb0 0x556b3e6541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:40.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f670affd640 10 client.17631 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:40.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f672b7fe640 1 --2- 172.21.15.28:0/2727634599 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556b3e5e6cb0 0x556b3e6541a0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efb7bb054fd3a81e server_cookie=4c1bee084b1ec1 in_seq=0 out_seq=0 2024-04-21T08:58:40.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f670affd640 1 -- 172.21.15.28:0/2727634599 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556b3e53fc10 con 0x556b3e5e6cb0 2024-04-21T08:58:40.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f670affd640 10 client.17631 handle_command_reply: tid=0 2024-04-21T08:58:40.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f6734800740 1 client.17631 shutdown 2024-04-21T08:58:40.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f6734800740 1 -- 172.21.15.28:0/2727634599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cde80 msgr2=0x556b3e6459d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f6734800740 1 --2- 172.21.15.28:0/2727634599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cde80 0x556b3e6459d0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f67200be7b0 tx=0x7f67200904f0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f6734800740 1 -- 172.21.15.28:0/2727634599 shutdown_connections 2024-04-21T08:58:40.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f6734800740 1 --2- 172.21.15.28:0/2727634599 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556b3e5e6cb0 0x556b3e6541a0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f6734800740 1 --2- 172.21.15.28:0/2727634599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b3e5d6a00 0x556b3e650320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f670affd640 0 client.17631 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:40.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f6734800740 1 --2- 172.21.15.28:0/2727634599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b3e5cde80 0x556b3e6459d0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f6734800740 1 --2- 172.21.15.28:0/2727634599 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556b3e5cd4d0 0x556b3e645490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f6734800740 1 -- 172.21.15.28:0/2727634599 >> 172.21.15.28:0/2727634599 conn(0x556b3e5c7ba0 msgr2=0x556b3e5c97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f6734800740 1 -- 172.21.15.28:0/2727634599 shutdown_connections 2024-04-21T08:58:40.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f6734800740 1 -- 172.21.15.28:0/2727634599 wait complete. 2024-04-21T08:58:40.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.719+0000 7f6734800740 20 client.17631 trim_cache size 0 max 16384 2024-04-21T08:58:40.730 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:58:40.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.720+0000 7f6731b32640 1 -- 172.21.15.28:0/772425345 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f67140a3da0 msgr2=0x7f67140a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.720+0000 7f6731b32640 1 --2- 172.21.15.28:0/772425345 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f67140a3da0 0x7f67140a6260 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f672401e000 tx=0x7f672401f000 comp rx=0 tx=0).stop 2024-04-21T08:58:40.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.720+0000 7f6731b32640 1 -- 172.21.15.28:0/772425345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f672c106160 msgr2=0x7f672c198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:40.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.720+0000 7f6731b32640 1 --2- 172.21.15.28:0/772425345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f672c106160 0x7f672c198b70 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f6720005ec0 tx=0x7f672002ca90 comp rx=0 tx=0).stop 2024-04-21T08:58:40.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.720+0000 7f6731b32640 1 -- 172.21.15.28:0/772425345 shutdown_connections 2024-04-21T08:58:40.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.720+0000 7f6731b32640 1 --2- 172.21.15.28:0/772425345 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f67140a3da0 0x7f67140a6260 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.720+0000 7f6731b32640 1 --2- 172.21.15.28:0/772425345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f672c19cef0 0x7f672c19d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.720+0000 7f6731b32640 1 --2- 172.21.15.28:0/772425345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f672c106160 0x7f672c198b70 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.720+0000 7f6731b32640 1 --2- 172.21.15.28:0/772425345 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f672c101090 0x7f672c198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:40.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.720+0000 7f6731b32640 1 -- 172.21.15.28:0/772425345 >> 172.21.15.28:0/772425345 conn(0x7f672c077f50 msgr2=0x7f672c0fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:40.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.720+0000 7f6731b32640 1 -- 172.21.15.28:0/772425345 shutdown_connections 2024-04-21T08:58:40.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:40.720+0000 7f6731b32640 1 -- 172.21.15.28:0/772425345 wait complete. 2024-04-21T08:58:40.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:40 smithi152 ceph-mon[31401]: pgmap v229: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:58:40.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:40 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/771269837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:58:40.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:40 smithi171 ceph-mon[31358]: pgmap v229: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:58:40.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:40 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/771269837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:58:41.734 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:58:41.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:58:41.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:58:41.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:41 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:58:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.842+0000 7f916c6b2640 1 -- 172.21.15.28:0/1832649233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9164102190 msgr2=0x7f9164102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.842+0000 7f916c6b2640 1 --2- 172.21.15.28:0/1832649233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9164102190 0x7f9164102570 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f9160009900 tx=0x7f916002bc30 comp rx=0 tx=0).stop 2024-04-21T08:58:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.843+0000 7f916c6b2640 1 -- 172.21.15.28:0/1832649233 shutdown_connections 2024-04-21T08:58:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.843+0000 7f916c6b2640 1 --2- 172.21.15.28:0/1832649233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9164107260 0x7f916410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.843+0000 7f916c6b2640 1 --2- 172.21.15.28:0/1832649233 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9164102b40 0x7f9164106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.843+0000 7f916c6b2640 1 --2- 172.21.15.28:0/1832649233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9164102190 0x7f9164102570 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.843+0000 7f916c6b2640 1 -- 172.21.15.28:0/1832649233 >> 172.21.15.28:0/1832649233 conn(0x7f91640fd990 msgr2=0x7f91640ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:41.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.843+0000 7f916c6b2640 1 -- 172.21.15.28:0/1832649233 shutdown_connections 2024-04-21T08:58:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.843+0000 7f916c6b2640 1 -- 172.21.15.28:0/1832649233 wait complete. 2024-04-21T08:58:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.843+0000 7f916c6b2640 1 Processor -- start 2024-04-21T08:58:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.843+0000 7f916c6b2640 1 -- start start 2024-04-21T08:58:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.843+0000 7f916c6b2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9164102190 0x7f916419e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.843+0000 7f916c6b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9164107260 0x7f916419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.843+0000 7f916c6b2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9164198640 0x7f9164198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.843+0000 7f916c6b2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91641121c0 con 0x7f9164107260 2024-04-21T08:58:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.843+0000 7f916c6b2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9164112040 con 0x7f9164198640 2024-04-21T08:58:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.843+0000 7f916c6b2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9164112340 con 0x7f9164102190 2024-04-21T08:58:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f9169c4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9164107260 0x7f916419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f9169c4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9164107260 0x7f916419e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33658/0 (socket says 172.21.15.28:33658) 2024-04-21T08:58:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f9169c4d640 1 -- 172.21.15.28:0/3333323065 learned_addr learned my addr 172.21.15.28:0/3333323065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f916ac4f640 1 --2- 172.21.15.28:0/3333323065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9164198640 0x7f9164198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f916a44e640 1 --2- 172.21.15.28:0/3333323065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9164102190 0x7f916419e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f9169c4d640 1 -- 172.21.15.28:0/3333323065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9164102190 msgr2=0x7f916419e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f9169c4d640 1 --2- 172.21.15.28:0/3333323065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9164102190 0x7f916419e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f9169c4d640 1 -- 172.21.15.28:0/3333323065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9164198640 msgr2=0x7f9164198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f9169c4d640 1 --2- 172.21.15.28:0/3333323065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9164198640 0x7f9164198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f9169c4d640 1 -- 172.21.15.28:0/3333323065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9164199160 con 0x7f9164107260 2024-04-21T08:58:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f916ac4f640 1 --2- 172.21.15.28:0/3333323065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9164198640 0x7f9164198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f9169c4d640 1 --2- 172.21.15.28:0/3333323065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9164107260 0x7f916419e980 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f915400d740 tx=0x7f915400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f915b7fe640 1 -- 172.21.15.28:0/3333323065 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9154017070 con 0x7f9164107260 2024-04-21T08:58:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f916c6b2640 1 -- 172.21.15.28:0/3333323065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9164199450 con 0x7f9164107260 2024-04-21T08:58:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f915b7fe640 1 -- 172.21.15.28:0/3333323065 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f915400de50 con 0x7f9164107260 2024-04-21T08:58:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f916c6b2640 1 -- 172.21.15.28:0/3333323065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91641a5650 con 0x7f9164107260 2024-04-21T08:58:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.844+0000 7f915b7fe640 1 -- 172.21.15.28:0/3333323065 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9154020060 con 0x7f9164107260 2024-04-21T08:58:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.846+0000 7f915b7fe640 1 -- 172.21.15.28:0/3333323065 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9154004c80 con 0x7f9164107260 2024-04-21T08:58:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.847+0000 7f916f376740 1 Processor -- start 2024-04-21T08:58:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.847+0000 7f916f376740 1 -- start start 2024-04-21T08:58:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.847+0000 7f916f376740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 0x558e09d2ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.847+0000 7f916f376740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 0x558e09d37840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.847+0000 7f916f376740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d3a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.847+0000 7f916f376740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558e09c92f00 con 0x558e09d37e10 2024-04-21T08:58:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.847+0000 7f916f376740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558e09c30590 con 0x558e09d2f290 2024-04-21T08:58:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.847+0000 7f916f376740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558e09c8eec0 con 0x558e09d2e8e0 2024-04-21T08:58:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.847+0000 7f916a44e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 0x558e09d37840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.847+0000 7f916a44e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 0x558e09d37840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:47364/0 (socket says 172.21.15.28:47364) 2024-04-21T08:58:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.847+0000 7f916ac4f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 0x558e09d2ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.847+0000 7f916a44e640 1 -- 172.21.15.28:0/715941560 learned_addr learned my addr 172.21.15.28:0/715941560 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.847+0000 7f9169c4d640 1 --2- 172.21.15.28:0/715941560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d3a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.847+0000 7f915b7fe640 1 --2- 172.21.15.28:0/3333323065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f91500a3d20 0x7f91500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.848+0000 7f916a44e640 1 -- 172.21.15.28:0/715941560 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 msgr2=0x558e09d2ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:41.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.848+0000 7f916a44e640 1 --2- 172.21.15.28:0/715941560 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 0x558e09d2ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.848+0000 7f916a44e640 1 -- 172.21.15.28:0/715941560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 msgr2=0x558e09d3a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:41.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.848+0000 7f916a44e640 1 --2- 172.21.15.28:0/715941560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d3a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.848+0000 7f916a44e640 1 -- 172.21.15.28:0/715941560 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e09a6d7e0 con 0x558e09d2f290 2024-04-21T08:58:41.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.848+0000 7f9169c4d640 1 --2- 172.21.15.28:0/715941560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d3a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:41.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.848+0000 7f916ac4f640 1 --2- 172.21.15.28:0/3333323065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f91500a3d20 0x7f91500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.848+0000 7f915b7fe640 1 -- 172.21.15.28:0/3333323065 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7f91540d0710 con 0x7f9164107260 2024-04-21T08:58:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.848+0000 7f916a44e640 1 --2- 172.21.15.28:0/715941560 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 0x558e09d37840 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9160000c00 tx=0x7f916002c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=270cba63cd0f769d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.848+0000 7f9158ff9640 1 -- 172.21.15.28:0/715941560 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160043020 con 0x558e09d2f290 2024-04-21T08:58:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.848+0000 7f9158ff9640 1 -- 172.21.15.28:0/715941560 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f916002c7e0 con 0x558e09d2f290 2024-04-21T08:58:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.848+0000 7f9158ff9640 1 -- 172.21.15.28:0/715941560 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91600055b0 con 0x558e09d2f290 2024-04-21T08:58:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.848+0000 7f916ac4f640 1 --2- 172.21.15.28:0/3333323065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f91500a3d20 0x7f91500a61e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f915c005fd0 tx=0x7f915c004a40 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 -- 172.21.15.28:0/715941560 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 msgr2=0x558e09d37840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 --2- 172.21.15.28:0/715941560 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 0x558e09d37840 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9160000c00 tx=0x7f916002c5d0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 -- 172.21.15.28:0/715941560 shutdown_connections 2024-04-21T08:58:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 --2- 172.21.15.28:0/715941560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d3a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 --2- 172.21.15.28:0/715941560 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 0x558e09d37840 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 --2- 172.21.15.28:0/715941560 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 0x558e09d2ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 -- 172.21.15.28:0/715941560 >> 172.21.15.28:0/715941560 conn(0x558e09d28fb0 msgr2=0x558e09d29390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 -- 172.21.15.28:0/715941560 shutdown_connections 2024-04-21T08:58:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 -- 172.21.15.28:0/715941560 wait complete. 2024-04-21T08:58:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 -- start start 2024-04-21T08:58:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 Processor -- start 2024-04-21T08:58:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 0x558e09d34df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 0x558e09d35330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d35870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558e09d3ad20 con 0x558e09d37e10 2024-04-21T08:58:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558e09c4c580 con 0x558e09d2f290 2024-04-21T08:58:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.849+0000 7f916f376740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558e09d3aea0 con 0x558e09d2e8e0 2024-04-21T08:58:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f916f376740 10 client.0 _ll_register_callbacks cb 0x558e09c52650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f916ac4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d35870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f916ac4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d35870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33670/0 (socket says 172.21.15.28:33670) 2024-04-21T08:58:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f916ac4f640 1 -- 172.21.15.28:0/3351280284 learned_addr learned my addr 172.21.15.28:0/3351280284 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f916a44e640 1 --2- 172.21.15.28:0/3351280284 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 0x558e09d34df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f9169c4d640 1 --2- 172.21.15.28:0/3351280284 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 0x558e09d35330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f916ac4f640 1 -- 172.21.15.28:0/3351280284 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 msgr2=0x558e09d34df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f916ac4f640 1 --2- 172.21.15.28:0/3351280284 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 0x558e09d34df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f916ac4f640 1 -- 172.21.15.28:0/3351280284 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 msgr2=0x558e09d35330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f916ac4f640 1 --2- 172.21.15.28:0/3351280284 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 0x558e09d35330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f916ac4f640 1 -- 172.21.15.28:0/3351280284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e09acdf70 con 0x558e09d37e10 2024-04-21T08:58:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f9169c4d640 1 --2- 172.21.15.28:0/3351280284 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 0x558e09d35330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:58:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f916ac4f640 1 --2- 172.21.15.28:0/3351280284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d35870 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f915c009b60 tx=0x7f915c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4be9c6e6928dc18 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f914dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f914dffb640 1 -- 172.21.15.28:0/3351280284 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915c02b060 con 0x558e09d37e10 2024-04-21T08:58:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f914dffb640 1 -- 172.21.15.28:0/3351280284 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7f915c018070 con 0x558e09d37e10 2024-04-21T08:58:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.850+0000 7f916f376740 1 -- 172.21.15.28:0/3351280284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f916002c7e0 con 0x558e09d37e10 2024-04-21T08:58:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.851+0000 7f914dffb640 1 -- 172.21.15.28:0/3351280284 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f915c032050 con 0x558e09d37e10 2024-04-21T08:58:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.851+0000 7f916f376740 10 client.17643 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.851+0000 7f916f376740 1 -- 172.21.15.28:0/3351280284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558e09965170 con 0x558e09d37e10 2024-04-21T08:58:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.851+0000 7f914dffb640 1 -- 172.21.15.28:0/3351280284 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f916002c7e0 con 0x558e09d37e10 2024-04-21T08:58:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.851+0000 7f914dffb640 1 -- 172.21.15.28:0/3351280284 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f915c00f040 con 0x558e09d37e10 2024-04-21T08:58:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.851+0000 7f916f376740 10 client.17643 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.851+0000 7f916f376740 10 client.17643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.851+0000 7f916f376740 20 client.17643 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.851+0000 7f916f376740 1 --2- 172.21.15.28:0/3351280284 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x558e09d483b0 0x558e09d4a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.851+0000 7f916f376740 4 client.17643 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:58:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.851+0000 7f916f376740 1 -- 172.21.15.28:0/3351280284 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e09ca0920 con 0x558e09d483b0 2024-04-21T08:58:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.851+0000 7f916a44e640 1 --2- 172.21.15.28:0/3351280284 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x558e09d483b0 0x558e09d4a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.852+0000 7f916a44e640 1 --2- 172.21.15.28:0/3351280284 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x558e09d483b0 0x558e09d4a800 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd93139b8f8f4ebe server_cookie=afeed53a122a191 in_seq=0 out_seq=0 2024-04-21T08:58:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.852+0000 7f914dffb640 10 client.17643 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.852+0000 7f914dffb640 1 -- 172.21.15.28:0/3351280284 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558e09ca0920 con 0x558e09d483b0 2024-04-21T08:58:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.852+0000 7f914dffb640 10 client.17643 handle_command_reply: tid=0 2024-04-21T08:58:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.853+0000 7f916f376740 1 client.17643 shutdown 2024-04-21T08:58:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.853+0000 7f916f376740 1 -- 172.21.15.28:0/3351280284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 msgr2=0x558e09d35870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.853+0000 7f916f376740 1 --2- 172.21.15.28:0/3351280284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d35870 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f915c009b60 tx=0x7f915c004ab0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.853+0000 7f916f376740 1 -- 172.21.15.28:0/3351280284 shutdown_connections 2024-04-21T08:58:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.853+0000 7f916f376740 1 --2- 172.21.15.28:0/3351280284 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x558e09d483b0 0x558e09d4a800 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.853+0000 7f916f376740 1 --2- 172.21.15.28:0/3351280284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d35870 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.853+0000 7f916f376740 1 --2- 172.21.15.28:0/3351280284 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 0x558e09d35330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.853+0000 7f914dffb640 0 client.17643 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.853+0000 7f916f376740 1 --2- 172.21.15.28:0/3351280284 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 0x558e09d34df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.853+0000 7f916f376740 1 -- 172.21.15.28:0/3351280284 >> 172.21.15.28:0/3351280284 conn(0x558e09d28fb0 msgr2=0x558e09d2ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.853+0000 7f916f376740 1 -- 172.21.15.28:0/3351280284 shutdown_connections 2024-04-21T08:58:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.853+0000 7f916f376740 1 -- 172.21.15.28:0/3351280284 wait complete. 2024-04-21T08:58:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.853+0000 7f916f376740 20 client.17643 trim_cache size 0 max 16384 2024-04-21T08:58:41.866 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:41 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:58:41.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:41 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:58:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916f376740 1 Processor -- start 2024-04-21T08:58:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916f376740 1 -- start start 2024-04-21T08:58:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916f376740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d2e8e0 0x558e09d37520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916f376740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2f290 0x558e09d43be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916f376740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d37e10 0x558e09d500d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916f376740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91641b1d60 con 0x558e09d2e8e0 2024-04-21T08:58:41.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916f376740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f91641af630 con 0x558e09d37e10 2024-04-21T08:58:41.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916f376740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f91641ad4a0 con 0x558e09d2f290 2024-04-21T08:58:41.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916a44e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d2e8e0 0x558e09d37520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916a44e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d2e8e0 0x558e09d37520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33672/0 (socket says 172.21.15.28:33672) 2024-04-21T08:58:41.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916a44e640 1 -- 172.21.15.28:0/3511168884 learned_addr learned my addr 172.21.15.28:0/3511168884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:41.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f9169c4d640 1 --2- 172.21.15.28:0/3511168884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2f290 0x558e09d43be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916a44e640 1 -- 172.21.15.28:0/3511168884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2f290 msgr2=0x558e09d43be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916a44e640 1 --2- 172.21.15.28:0/3511168884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2f290 0x558e09d43be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916a44e640 1 -- 172.21.15.28:0/3511168884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d37e10 msgr2=0x558e09d500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916a44e640 1 --2- 172.21.15.28:0/3511168884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d37e10 0x558e09d500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.871+0000 7f916a44e640 1 -- 172.21.15.28:0/3511168884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e09d46f50 con 0x558e09d2e8e0 2024-04-21T08:58:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.872+0000 7f916a44e640 1 --2- 172.21.15.28:0/3511168884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d2e8e0 0x558e09d37520 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f9160000c00 tx=0x7f916002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78dfe833848d3d9e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.872+0000 7f914ffff640 1 -- 172.21.15.28:0/3511168884 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160043040 con 0x558e09d2e8e0 2024-04-21T08:58:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.872+0000 7f914ffff640 1 -- 172.21.15.28:0/3511168884 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f916002c970 con 0x558e09d2e8e0 2024-04-21T08:58:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.872+0000 7f914ffff640 1 -- 172.21.15.28:0/3511168884 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160052840 con 0x558e09d2e8e0 2024-04-21T08:58:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.872+0000 7f916f376740 1 -- 172.21.15.28:0/3511168884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d2e8e0 msgr2=0x558e09d37520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:41.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.872+0000 7f916f376740 1 --2- 172.21.15.28:0/3511168884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d2e8e0 0x558e09d37520 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f9160000c00 tx=0x7f916002bc30 comp rx=0 tx=0).stop 2024-04-21T08:58:41.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.872+0000 7f916f376740 1 -- 172.21.15.28:0/3511168884 shutdown_connections 2024-04-21T08:58:41.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.872+0000 7f916f376740 1 --2- 172.21.15.28:0/3511168884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d37e10 0x558e09d500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.872+0000 7f916f376740 1 --2- 172.21.15.28:0/3511168884 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2f290 0x558e09d43be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.872+0000 7f916f376740 1 --2- 172.21.15.28:0/3511168884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d2e8e0 0x558e09d37520 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.872+0000 7f916f376740 1 -- 172.21.15.28:0/3511168884 >> 172.21.15.28:0/3511168884 conn(0x558e09d28fb0 msgr2=0x558e09d28010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:41.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.872+0000 7f916f376740 1 -- 172.21.15.28:0/3511168884 shutdown_connections 2024-04-21T08:58:41.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.872+0000 7f916f376740 1 -- 172.21.15.28:0/3511168884 wait complete. 2024-04-21T08:58:41.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916f376740 1 -- start start 2024-04-21T08:58:41.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916f376740 1 Processor -- start 2024-04-21T08:58:41.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916f376740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 0x558e09da7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916f376740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 0x558e09da8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916f376740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d3c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916f376740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91641adaf0 con 0x558e09d37e10 2024-04-21T08:58:41.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916f376740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f91641aef60 con 0x558e09d2f290 2024-04-21T08:58:41.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916f376740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f91641ae1c0 con 0x558e09d2e8e0 2024-04-21T08:58:41.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916f376740 10 client.0 _ll_register_callbacks cb 0x7f915c009d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:41.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916ac4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d3c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916ac4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d3c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33680/0 (socket says 172.21.15.28:33680) 2024-04-21T08:58:41.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916ac4f640 1 -- 172.21.15.28:0/1688588077 learned_addr learned my addr 172.21.15.28:0/1688588077 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:41.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916a44e640 1 --2- 172.21.15.28:0/1688588077 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 0x558e09da7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916ac4f640 1 -- 172.21.15.28:0/1688588077 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 msgr2=0x558e09da7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:41.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916ac4f640 1 --2- 172.21.15.28:0/1688588077 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 0x558e09da7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916ac4f640 1 -- 172.21.15.28:0/1688588077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 msgr2=0x558e09da8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:41.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916ac4f640 1 --2- 172.21.15.28:0/1688588077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 0x558e09da8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916ac4f640 1 -- 172.21.15.28:0/1688588077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e09d3c930 con 0x558e09d37e10 2024-04-21T08:58:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.873+0000 7f916a44e640 1 --2- 172.21.15.28:0/1688588077 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 0x558e09da7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f916ac4f640 1 --2- 172.21.15.28:0/1688588077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d3c270 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f915c009b30 tx=0x7f915c010f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e582adbc36d1380 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f914dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f914dffb640 1 -- 172.21.15.28:0/1688588077 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f915c02b020 con 0x558e09d37e10 2024-04-21T08:58:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f914dffb640 1 -- 172.21.15.28:0/1688588077 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7f915c01c680 con 0x558e09d37e10 2024-04-21T08:58:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f916f376740 1 -- 172.21.15.28:0/1688588077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f916002c970 con 0x558e09d37e10 2024-04-21T08:58:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f914dffb640 1 -- 172.21.15.28:0/1688588077 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f915c01e6f0 con 0x558e09d37e10 2024-04-21T08:58:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f916f376740 10 client.17649 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f916f376740 1 -- 172.21.15.28:0/1688588077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558e09daf3c0 con 0x558e09d37e10 2024-04-21T08:58:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f914dffb640 1 -- 172.21.15.28:0/1688588077 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f916002c970 con 0x558e09d37e10 2024-04-21T08:58:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f914dffb640 1 -- 172.21.15.28:0/1688588077 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f915c0267f0 con 0x558e09d37e10 2024-04-21T08:58:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f916f376740 10 client.17649 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f916f376740 10 client.17649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f916f376740 20 client.17649 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f916f376740 1 --2- 172.21.15.28:0/1688588077 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x558e09dafb00 0x558e09db1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f916f376740 4 client.17649 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:58:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.874+0000 7f916f376740 1 -- 172.21.15.28:0/1688588077 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558e09c9a790 con 0x558e09dafb00 2024-04-21T08:58:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.875+0000 7f916a44e640 1 --2- 172.21.15.28:0/1688588077 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x558e09dafb00 0x558e09db1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.875+0000 7f916a44e640 1 --2- 172.21.15.28:0/1688588077 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x558e09dafb00 0x558e09db1f20 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fbf7f436b7c1f4b server_cookie=8e6958311298ba6 in_seq=0 out_seq=0 2024-04-21T08:58:41.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.875+0000 7f914dffb640 10 client.17649 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.876+0000 7f914dffb640 1 -- 172.21.15.28:0/1688588077 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558e09c9a790 con 0x558e09dafb00 2024-04-21T08:58:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.876+0000 7f914dffb640 10 client.17649 handle_command_reply: tid=0 2024-04-21T08:58:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.876+0000 7f916f376740 1 client.17649 shutdown 2024-04-21T08:58:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.876+0000 7f916f376740 1 -- 172.21.15.28:0/1688588077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 msgr2=0x558e09d3c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:41.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.876+0000 7f916f376740 1 --2- 172.21.15.28:0/1688588077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d3c270 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f915c009b30 tx=0x7f915c010f30 comp rx=0 tx=0).stop 2024-04-21T08:58:41.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.876+0000 7f916f376740 1 -- 172.21.15.28:0/1688588077 shutdown_connections 2024-04-21T08:58:41.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.876+0000 7f916f376740 1 --2- 172.21.15.28:0/1688588077 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x558e09dafb00 0x558e09db1f20 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.876+0000 7f916f376740 1 --2- 172.21.15.28:0/1688588077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e09d37e10 0x558e09d3c270 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.876+0000 7f916f376740 1 --2- 172.21.15.28:0/1688588077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558e09d2f290 0x558e09da8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.876+0000 7f916f376740 1 --2- 172.21.15.28:0/1688588077 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558e09d2e8e0 0x558e09da7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.876+0000 7f914dffb640 0 client.17649 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:41.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.876+0000 7f916f376740 1 -- 172.21.15.28:0/1688588077 >> 172.21.15.28:0/1688588077 conn(0x558e09d28fb0 msgr2=0x558e09d27780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:41.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.876+0000 7f916f376740 1 -- 172.21.15.28:0/1688588077 shutdown_connections 2024-04-21T08:58:41.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.876+0000 7f916f376740 1 -- 172.21.15.28:0/1688588077 wait complete. 2024-04-21T08:58:41.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.876+0000 7f916f376740 20 client.17649 trim_cache size 0 max 16384 2024-04-21T08:58:41.886 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:58:41.886 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:58:41.886 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "d1b52aae-f4fb-4a24-a149-bfc4213aad47", 2024-04-21T08:58:41.887 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:58:41.887 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:58:41.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.877+0000 7f916c6b2640 1 -- 172.21.15.28:0/3333323065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f91500a3d20 msgr2=0x7f91500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.877+0000 7f916c6b2640 1 --2- 172.21.15.28:0/3333323065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f91500a3d20 0x7f91500a61e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f915c005fd0 tx=0x7f915c004a40 comp rx=0 tx=0).stop 2024-04-21T08:58:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.877+0000 7f916c6b2640 1 -- 172.21.15.28:0/3333323065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9164107260 msgr2=0x7f916419e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.877+0000 7f916c6b2640 1 --2- 172.21.15.28:0/3333323065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9164107260 0x7f916419e980 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f915400d740 tx=0x7f915400dc00 comp rx=0 tx=0).stop 2024-04-21T08:58:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.877+0000 7f916c6b2640 1 -- 172.21.15.28:0/3333323065 shutdown_connections 2024-04-21T08:58:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.877+0000 7f916c6b2640 1 --2- 172.21.15.28:0/3333323065 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f91500a3d20 0x7f91500a61e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.877+0000 7f916c6b2640 1 --2- 172.21.15.28:0/3333323065 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9164198640 0x7f9164198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.877+0000 7f916c6b2640 1 --2- 172.21.15.28:0/3333323065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9164107260 0x7f916419e980 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.877+0000 7f916c6b2640 1 --2- 172.21.15.28:0/3333323065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9164102190 0x7f916419e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.877+0000 7f916c6b2640 1 -- 172.21.15.28:0/3333323065 >> 172.21.15.28:0/3333323065 conn(0x7f91640fd990 msgr2=0x7f91640fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.877+0000 7f916c6b2640 1 -- 172.21.15.28:0/3333323065 shutdown_connections 2024-04-21T08:58:41.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:41.877+0000 7f916c6b2640 1 -- 172.21.15.28:0/3333323065 wait complete. 2024-04-21T08:58:41.890 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:58:41.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:58:42.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.035+0000 7fda23206640 1 -- 172.21.15.28:0/1472755823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda1c101a30 msgr2=0x7fda1c105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.035+0000 7fda23206640 1 --2- 172.21.15.28:0/1472755823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda1c101a30 0x7fda1c105a70 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fda08009900 tx=0x7fda0802bc30 comp rx=0 tx=0).stop 2024-04-21T08:58:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.035+0000 7fda23206640 1 -- 172.21.15.28:0/1472755823 shutdown_connections 2024-04-21T08:58:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.035+0000 7fda23206640 1 --2- 172.21.15.28:0/1472755823 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda1c106150 0x7fda1c10dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.035+0000 7fda23206640 1 --2- 172.21.15.28:0/1472755823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda1c101a30 0x7fda1c105a70 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.035+0000 7fda23206640 1 --2- 172.21.15.28:0/1472755823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda1c101080 0x7fda1c101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.035+0000 7fda23206640 1 -- 172.21.15.28:0/1472755823 >> 172.21.15.28:0/1472755823 conn(0x7fda1c078000 msgr2=0x7fda1c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.035+0000 7fda23206640 1 -- 172.21.15.28:0/1472755823 shutdown_connections 2024-04-21T08:58:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.035+0000 7fda23206640 1 -- 172.21.15.28:0/1472755823 wait complete. 2024-04-21T08:58:42.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda23206640 1 Processor -- start 2024-04-21T08:58:42.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda23206640 1 -- start start 2024-04-21T08:58:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda23206640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda1c101080 0x7fda1c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda23206640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda1c106150 0x7fda1c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda23206640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda1c19d000 0x7fda1c19d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda23206640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c111070 con 0x7fda1c19d000 2024-04-21T08:58:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda23206640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c110ef0 con 0x7fda1c101080 2024-04-21T08:58:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda23206640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c1111f0 con 0x7fda1c106150 2024-04-21T08:58:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda217a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda1c19d000 0x7fda1c19d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda217a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda1c19d000 0x7fda1c19d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33704/0 (socket says 172.21.15.28:33704) 2024-04-21T08:58:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda217a3640 1 -- 172.21.15.28:0/1015813485 learned_addr learned my addr 172.21.15.28:0/1015813485 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda1bfff640 1 --2- 172.21.15.28:0/1015813485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda1c106150 0x7fda1c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda20fa2640 1 --2- 172.21.15.28:0/1015813485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda1c101080 0x7fda1c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:42.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda217a3640 1 -- 172.21.15.28:0/1015813485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda1c106150 msgr2=0x7fda1c198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:42.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda217a3640 1 --2- 172.21.15.28:0/1015813485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda1c106150 0x7fda1c198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda217a3640 1 -- 172.21.15.28:0/1015813485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda1c101080 msgr2=0x7fda1c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:42.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda217a3640 1 --2- 172.21.15.28:0/1015813485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda1c101080 0x7fda1c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda217a3640 1 -- 172.21.15.28:0/1015813485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1c19daf0 con 0x7fda1c19d000 2024-04-21T08:58:42.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.036+0000 7fda20fa2640 1 --2- 172.21.15.28:0/1015813485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda1c101080 0x7fda1c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:58:42.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.037+0000 7fda217a3640 1 --2- 172.21.15.28:0/1015813485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda1c19d000 0x7fda1c19d4b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fda14009fd0 tx=0x7fda1400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:42.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.037+0000 7fda19ffb640 1 -- 172.21.15.28:0/1015813485 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda14017070 con 0x7fda1c19d000 2024-04-21T08:58:42.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.037+0000 7fda23206640 1 -- 172.21.15.28:0/1015813485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda1c19dde0 con 0x7fda1c19d000 2024-04-21T08:58:42.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.037+0000 7fda19ffb640 1 -- 172.21.15.28:0/1015813485 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda14003e20 con 0x7fda1c19d000 2024-04-21T08:58:42.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.037+0000 7fda19ffb640 1 -- 172.21.15.28:0/1015813485 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda14010460 con 0x7fda1c19d000 2024-04-21T08:58:42.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.037+0000 7fda23206640 1 -- 172.21.15.28:0/1015813485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda1c1a5550 con 0x7fda1c19d000 2024-04-21T08:58:42.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.039+0000 7fda19ffb640 1 -- 172.21.15.28:0/1015813485 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fda14003970 con 0x7fda1c19d000 2024-04-21T08:58:42.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.039+0000 7fda25df6740 1 Processor -- start 2024-04-21T08:58:42.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.039+0000 7fda25df6740 1 -- start start 2024-04-21T08:58:42.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda25df6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51be9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda25df6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 0x555c51bf2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda25df6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51bf2a80 0x555c51bf4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda25df6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555c51b4e1e0 con 0x555c51be9580 2024-04-21T08:58:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda25df6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x555c51aeb390 con 0x555c51bf2a80 2024-04-21T08:58:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda25df6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x555c51b4a1a0 con 0x555c51be9ea0 2024-04-21T08:58:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda1bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51be9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda19ffb640 1 --2- 172.21.15.28:0/1015813485 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fda0c0a3d20 0x7fda0c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda1bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51be9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33712/0 (socket says 172.21.15.28:33712) 2024-04-21T08:58:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda1bfff640 1 -- 172.21.15.28:0/2892345600 learned_addr learned my addr 172.21.15.28:0/2892345600 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:42.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda1bfff640 1 --2- 172.21.15.28:0/2892345600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51bf2a80 0x555c51bf4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:42.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda20fa2640 1 --2- 172.21.15.28:0/2892345600 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 0x555c51bf2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:42.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda217a3640 1 --2- 172.21.15.28:0/1015813485 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fda0c0a3d20 0x7fda0c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:42.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda19ffb640 1 -- 172.21.15.28:0/1015813485 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7fda140d0990 con 0x7fda1c19d000 2024-04-21T08:58:42.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda1bfff640 1 -- 172.21.15.28:0/2892345600 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 msgr2=0x555c51bf2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:42.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda1bfff640 1 --2- 172.21.15.28:0/2892345600 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 0x555c51bf2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda1bfff640 1 -- 172.21.15.28:0/2892345600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51bf2a80 msgr2=0x555c51bf4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:42.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda1bfff640 1 --2- 172.21.15.28:0/2892345600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51bf2a80 0x555c51bf4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda1bfff640 1 -- 172.21.15.28:0/2892345600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c5192a7d0 con 0x555c51be9580 2024-04-21T08:58:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda20fa2640 1 --2- 172.21.15.28:0/2892345600 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 0x555c51bf2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:58:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda1bfff640 1 --2- 172.21.15.28:0/2892345600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51be9960 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fda1c06b6f0 tx=0x7fda08005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f18d0aff0114f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.040+0000 7fda217a3640 1 --2- 172.21.15.28:0/1015813485 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fda0c0a3d20 0x7fda0c0a61e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fda140a4c20 tx=0x7fda1406df20 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.041+0000 7fda06ffd640 1 -- 172.21.15.28:0/2892345600 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0804a070 con 0x555c51be9580 2024-04-21T08:58:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.041+0000 7fda06ffd640 1 -- 172.21.15.28:0/2892345600 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda0802cc90 con 0x555c51be9580 2024-04-21T08:58:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.041+0000 7fda06ffd640 1 -- 172.21.15.28:0/2892345600 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda08003e50 con 0x555c51be9580 2024-04-21T08:58:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.041+0000 7fda25df6740 1 -- 172.21.15.28:0/2892345600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 msgr2=0x555c51be9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.041+0000 7fda25df6740 1 --2- 172.21.15.28:0/2892345600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51be9960 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fda1c06b6f0 tx=0x7fda08005e40 comp rx=0 tx=0).stop 2024-04-21T08:58:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.041+0000 7fda25df6740 1 -- 172.21.15.28:0/2892345600 shutdown_connections 2024-04-21T08:58:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.041+0000 7fda25df6740 1 --2- 172.21.15.28:0/2892345600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51bf2a80 0x555c51bf4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.041+0000 7fda25df6740 1 --2- 172.21.15.28:0/2892345600 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 0x555c51bf2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.041+0000 7fda25df6740 1 --2- 172.21.15.28:0/2892345600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51be9960 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.041+0000 7fda25df6740 1 -- 172.21.15.28:0/2892345600 >> 172.21.15.28:0/2892345600 conn(0x555c51be3cb0 msgr2=0x555c51be4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.041+0000 7fda25df6740 1 -- 172.21.15.28:0/2892345600 shutdown_connections 2024-04-21T08:58:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.041+0000 7fda25df6740 1 -- 172.21.15.28:0/2892345600 wait complete. 2024-04-21T08:58:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.042+0000 7fda25df6740 1 -- start start 2024-04-21T08:58:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.042+0000 7fda25df6740 1 Processor -- start 2024-04-21T08:58:42.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.042+0000 7fda25df6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51befb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.042+0000 7fda25df6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 0x555c51bf0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.042+0000 7fda25df6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51bf2a80 0x555c51bf05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.042+0000 7fda25df6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555c51bf5810 con 0x555c51be9580 2024-04-21T08:58:42.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.042+0000 7fda25df6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x555c51b072f0 con 0x555c51bf2a80 2024-04-21T08:58:42.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.042+0000 7fda25df6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x555c51bf5990 con 0x555c51be9ea0 2024-04-21T08:58:42.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.042+0000 7fda25df6740 10 client.0 _ll_register_callbacks cb 0x555c51b14580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:42.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.042+0000 7fda1bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51befb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.042+0000 7fda1bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51befb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33718/0 (socket says 172.21.15.28:33718) 2024-04-21T08:58:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.042+0000 7fda1bfff640 1 -- 172.21.15.28:0/3073765246 learned_addr learned my addr 172.21.15.28:0/3073765246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.042+0000 7fda20fa2640 1 --2- 172.21.15.28:0/3073765246 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 0x555c51bf0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.042+0000 7fda1bfff640 1 --2- 172.21.15.28:0/3073765246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51bf2a80 0x555c51bf05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fda1bfff640 1 -- 172.21.15.28:0/3073765246 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 msgr2=0x555c51bf0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fda1bfff640 1 --2- 172.21.15.28:0/3073765246 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 0x555c51bf0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fda1bfff640 1 -- 172.21.15.28:0/3073765246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51bf2a80 msgr2=0x555c51bf05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fda1bfff640 1 --2- 172.21.15.28:0/3073765246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51bf2a80 0x555c51bf05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fda1bfff640 1 -- 172.21.15.28:0/3073765246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c518603f0 con 0x555c51be9580 2024-04-21T08:58:42.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fda20fa2640 1 --2- 172.21.15.28:0/3073765246 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 0x555c51bf0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:58:42.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fda1bfff640 1 --2- 172.21.15.28:0/3073765246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51befb50 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fda08036040 tx=0x7fda08003a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e18c0d975d73ec23 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:42.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fd9fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:42.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fd9fbfff640 1 -- 172.21.15.28:0/3073765246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0804a070 con 0x555c51be9580 2024-04-21T08:58:42.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fd9fbfff640 1 -- 172.21.15.28:0/3073765246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7fda0802c7e0 con 0x555c51be9580 2024-04-21T08:58:42.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fda25df6740 1 -- 172.21.15.28:0/3073765246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda0802cc90 con 0x555c51be9580 2024-04-21T08:58:42.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fd9fbfff640 1 -- 172.21.15.28:0/3073765246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fda0802c9e0 con 0x555c51be9580 2024-04-21T08:58:42.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fda25df6740 10 client.17661 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fda25df6740 1 -- 172.21.15.28:0/3073765246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555c518200f0 con 0x555c51be9580 2024-04-21T08:58:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fd9fbfff640 1 -- 172.21.15.28:0/3073765246 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda0802cc90 con 0x555c51be9580 2024-04-21T08:58:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.043+0000 7fd9fbfff640 1 -- 172.21.15.28:0/3073765246 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fda08004a60 con 0x555c51be9580 2024-04-21T08:58:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.044+0000 7fda25df6740 10 client.17661 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.044+0000 7fda25df6740 10 client.17661 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.044+0000 7fda25df6740 20 client.17661 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.044+0000 7fda25df6740 1 --2- 172.21.15.28:0/3073765246 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555c51c02930 0x555c51c04d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.044+0000 7fda25df6740 4 client.17661 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:58:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.044+0000 7fda25df6740 1 -- 172.21.15.28:0/3073765246 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c51b5bc00 con 0x555c51c02930 2024-04-21T08:58:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.044+0000 7fda20fa2640 1 --2- 172.21.15.28:0/3073765246 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555c51c02930 0x555c51c04d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.044+0000 7fda20fa2640 1 --2- 172.21.15.28:0/3073765246 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555c51c02930 0x555c51c04d20 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3aec48aaf17d422b server_cookie=6fa34c45553d5759 in_seq=0 out_seq=0 2024-04-21T08:58:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.044+0000 7fd9fbfff640 10 client.17661 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.045+0000 7fd9fbfff640 1 -- 172.21.15.28:0/3073765246 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555c51b5bc00 con 0x555c51c02930 2024-04-21T08:58:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.045+0000 7fd9fbfff640 10 client.17661 handle_command_reply: tid=0 2024-04-21T08:58:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.045+0000 7fda25df6740 1 client.17661 shutdown 2024-04-21T08:58:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.045+0000 7fda25df6740 1 -- 172.21.15.28:0/3073765246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 msgr2=0x555c51befb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.045+0000 7fda25df6740 1 --2- 172.21.15.28:0/3073765246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51befb50 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fda08036040 tx=0x7fda08003a30 comp rx=0 tx=0).stop 2024-04-21T08:58:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.045+0000 7fda25df6740 1 -- 172.21.15.28:0/3073765246 shutdown_connections 2024-04-21T08:58:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.045+0000 7fda25df6740 1 --2- 172.21.15.28:0/3073765246 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555c51c02930 0x555c51c04d20 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.045+0000 7fda25df6740 1 --2- 172.21.15.28:0/3073765246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51bf2a80 0x555c51bf05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.045+0000 7fda25df6740 1 --2- 172.21.15.28:0/3073765246 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 0x555c51bf0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.045+0000 7fda25df6740 1 --2- 172.21.15.28:0/3073765246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51befb50 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.045+0000 7fd9fbfff640 0 client.17661 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.045+0000 7fda25df6740 1 -- 172.21.15.28:0/3073765246 >> 172.21.15.28:0/3073765246 conn(0x555c51be3cb0 msgr2=0x555c51be5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.045+0000 7fda25df6740 1 -- 172.21.15.28:0/3073765246 shutdown_connections 2024-04-21T08:58:42.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.045+0000 7fda25df6740 1 -- 172.21.15.28:0/3073765246 wait complete. 2024-04-21T08:58:42.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.045+0000 7fda25df6740 20 client.17661 trim_cache size 0 max 16384 2024-04-21T08:58:42.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda25df6740 1 Processor -- start 2024-04-21T08:58:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda25df6740 1 -- start start 2024-04-21T08:58:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda25df6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51bf2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda25df6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51be9ea0 0x555c51becdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda25df6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51bf2a80 0x555c51bed2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda25df6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c1b1c90 con 0x555c51be9580 2024-04-21T08:58:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda25df6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c1af560 con 0x555c51be9ea0 2024-04-21T08:58:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda25df6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c1ad3d0 con 0x555c51bf2a80 2024-04-21T08:58:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda1bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51bf2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda1bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51bf2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33728/0 (socket says 172.21.15.28:33728) 2024-04-21T08:58:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda1bfff640 1 -- 172.21.15.28:0/4222039647 learned_addr learned my addr 172.21.15.28:0/4222039647 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda1bfff640 1 -- 172.21.15.28:0/4222039647 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51bf2a80 msgr2=0x555c51bed2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda1bfff640 1 --2- 172.21.15.28:0/4222039647 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51bf2a80 0x555c51bed2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda1bfff640 1 -- 172.21.15.28:0/4222039647 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51be9ea0 msgr2=0x555c51becdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda1bfff640 1 --2- 172.21.15.28:0/4222039647 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51be9ea0 0x555c51becdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.063+0000 7fda1bfff640 1 -- 172.21.15.28:0/4222039647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c51c0ab60 con 0x555c51be9580 2024-04-21T08:58:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.064+0000 7fda1bfff640 1 --2- 172.21.15.28:0/4222039647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51bf2280 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fda08053500 tx=0x7fda08005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0f9e518687d471 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.064+0000 7fda067fc640 1 -- 172.21.15.28:0/4222039647 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0804a070 con 0x555c51be9580 2024-04-21T08:58:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.064+0000 7fda067fc640 1 -- 172.21.15.28:0/4222039647 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda0802cb80 con 0x555c51be9580 2024-04-21T08:58:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.064+0000 7fda067fc640 1 -- 172.21.15.28:0/4222039647 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda080049b0 con 0x555c51be9580 2024-04-21T08:58:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.064+0000 7fda25df6740 1 -- 172.21.15.28:0/4222039647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 msgr2=0x555c51bf2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.064+0000 7fda25df6740 1 --2- 172.21.15.28:0/4222039647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51bf2280 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fda08053500 tx=0x7fda08005e10 comp rx=0 tx=0).stop 2024-04-21T08:58:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.064+0000 7fda25df6740 1 -- 172.21.15.28:0/4222039647 shutdown_connections 2024-04-21T08:58:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.064+0000 7fda25df6740 1 --2- 172.21.15.28:0/4222039647 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51bf2a80 0x555c51bed2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.064+0000 7fda25df6740 1 --2- 172.21.15.28:0/4222039647 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51be9ea0 0x555c51becdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.064+0000 7fda25df6740 1 --2- 172.21.15.28:0/4222039647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51bf2280 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.064+0000 7fda25df6740 1 -- 172.21.15.28:0/4222039647 >> 172.21.15.28:0/4222039647 conn(0x555c51be3cb0 msgr2=0x555c51bfe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.064+0000 7fda25df6740 1 -- 172.21.15.28:0/4222039647 shutdown_connections 2024-04-21T08:58:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.065+0000 7fda25df6740 1 -- 172.21.15.28:0/4222039647 wait complete. 2024-04-21T08:58:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.065+0000 7fda25df6740 1 -- start start 2024-04-21T08:58:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.065+0000 7fda25df6740 1 Processor -- start 2024-04-21T08:58:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.065+0000 7fda25df6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51c07d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.065+0000 7fda25df6740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 0x555c51c082a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.065+0000 7fda25df6740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51bf2a80 0x555c51c052f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.065+0000 7fda25df6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c1ada20 con 0x555c51be9580 2024-04-21T08:58:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.065+0000 7fda25df6740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c1aee90 con 0x555c51bf2a80 2024-04-21T08:58:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.065+0000 7fda25df6740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c1ae0f0 con 0x555c51be9ea0 2024-04-21T08:58:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.065+0000 7fda25df6740 10 client.0 _ll_register_callbacks cb 0x7fda08048070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:58:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.065+0000 7fda1bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51c07d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.065+0000 7fda1bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51c07d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33734/0 (socket says 172.21.15.28:33734) 2024-04-21T08:58:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.065+0000 7fda1bfff640 1 -- 172.21.15.28:0/3703068164 learned_addr learned my addr 172.21.15.28:0/3703068164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:58:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fda20fa2640 1 --2- 172.21.15.28:0/3703068164 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 0x555c51c082a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fda1bfff640 1 -- 172.21.15.28:0/3703068164 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 msgr2=0x555c51c082a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fda1bfff640 1 --2- 172.21.15.28:0/3703068164 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 0x555c51c082a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fda1bfff640 1 -- 172.21.15.28:0/3703068164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51bf2a80 msgr2=0x555c51c052f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:58:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fda1bfff640 1 --2- 172.21.15.28:0/3703068164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51bf2a80 0x555c51c052f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fda1bfff640 1 -- 172.21.15.28:0/3703068164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c51c05af0 con 0x555c51be9580 2024-04-21T08:58:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fda20fa2640 1 --2- 172.21.15.28:0/3703068164 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 0x555c51c082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:58:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fda1bfff640 1 --2- 172.21.15.28:0/3703068164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51c07d60 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fda08009900 tx=0x7fda0802c610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca653f7e50c8cba server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:58:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fd9fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:58:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fd9fbfff640 1 -- 172.21.15.28:0/3703068164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0804a040 con 0x555c51be9580 2024-04-21T08:58:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fd9fbfff640 1 -- 172.21.15.28:0/3703068164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9533+0+0 (secure 0 0 0) 0x7fda08004730 con 0x555c51be9580 2024-04-21T08:58:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fda25df6740 1 -- 172.21.15.28:0/3703068164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda0802cb80 con 0x555c51be9580 2024-04-21T08:58:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fd9fbfff640 1 -- 172.21.15.28:0/3703068164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fda08053670 con 0x555c51be9580 2024-04-21T08:58:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fda25df6740 10 client.17667 fetch_fsmap learned FSMap version 28 2024-04-21T08:58:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fda25df6740 1 -- 172.21.15.28:0/3703068164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555c51c68770 con 0x555c51be9580 2024-04-21T08:58:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.066+0000 7fd9fbfff640 1 -- 172.21.15.28:0/3703068164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda0802cb80 con 0x555c51be9580 2024-04-21T08:58:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.067+0000 7fd9fbfff640 1 -- 172.21.15.28:0/3703068164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fda08004f30 con 0x555c51be9580 2024-04-21T08:58:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.067+0000 7fda25df6740 10 client.17667 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:58:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.067+0000 7fda25df6740 10 client.17667 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:58:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.067+0000 7fda25df6740 20 client.17667 populate_metadata read hostname 'smithi028' 2024-04-21T08:58:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.067+0000 7fda25df6740 1 --2- 172.21.15.28:0/3703068164 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555c51c68d80 0x555c51c6b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:58:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.067+0000 7fda25df6740 4 client.17667 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:58:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.067+0000 7fda25df6740 1 -- 172.21.15.28:0/3703068164 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555c51beaa50 con 0x555c51c68d80 2024-04-21T08:58:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.067+0000 7fda20fa2640 1 --2- 172.21.15.28:0/3703068164 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555c51c68d80 0x555c51c6b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:58:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.067+0000 7fd9fbfff640 10 client.17667 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.067+0000 7fda20fa2640 1 --2- 172.21.15.28:0/3703068164 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555c51c68d80 0x555c51c6b1a0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1296edae1b411304 server_cookie=f47eb46f02f8c203 in_seq=0 out_seq=0 2024-04-21T08:58:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.067+0000 7fd9fbfff640 1 -- 172.21.15.28:0/3703068164 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555c51beaa50 con 0x555c51c68d80 2024-04-21T08:58:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.067+0000 7fd9fbfff640 10 client.17667 handle_command_reply: tid=0 2024-04-21T08:58:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.067+0000 7fda25df6740 1 client.17667 shutdown 2024-04-21T08:58:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda25df6740 1 -- 172.21.15.28:0/3703068164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 msgr2=0x555c51c07d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda25df6740 1 --2- 172.21.15.28:0/3703068164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51c07d60 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fda08009900 tx=0x7fda0802c610 comp rx=0 tx=0).stop 2024-04-21T08:58:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda25df6740 1 -- 172.21.15.28:0/3703068164 shutdown_connections 2024-04-21T08:58:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda25df6740 1 --2- 172.21.15.28:0/3703068164 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x555c51c68d80 0x555c51c6b1a0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda25df6740 1 --2- 172.21.15.28:0/3703068164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555c51bf2a80 0x555c51c052f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fd9fbfff640 0 client.17667 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:58:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda25df6740 1 --2- 172.21.15.28:0/3703068164 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x555c51be9ea0 0x555c51c082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda25df6740 1 --2- 172.21.15.28:0/3703068164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c51be9580 0x555c51c07d60 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda25df6740 1 -- 172.21.15.28:0/3703068164 >> 172.21.15.28:0/3703068164 conn(0x555c51be3cb0 msgr2=0x555c51befb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda25df6740 1 -- 172.21.15.28:0/3703068164 shutdown_connections 2024-04-21T08:58:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda25df6740 1 -- 172.21.15.28:0/3703068164 wait complete. 2024-04-21T08:58:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda25df6740 20 client.17667 trim_cache size 0 max 16384 2024-04-21T08:58:42.078 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:58:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:58:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:58:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "d1b52aae-f4fb-4a24-a149-bfc4213aad47": { 2024-04-21T08:58:42.078 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:58:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "d1b52aae-f4fb-4a24-a149-bfc4213aad47", 2024-04-21T08:58:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:58:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:58:42.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:58:42.079 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:58:42.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda23206640 1 -- 172.21.15.28:0/1015813485 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fda0c0a3d20 msgr2=0x7fda0c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:42.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda23206640 1 --2- 172.21.15.28:0/1015813485 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fda0c0a3d20 0x7fda0c0a61e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fda140a4c20 tx=0x7fda1406df20 comp rx=0 tx=0).stop 2024-04-21T08:58:42.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda23206640 1 -- 172.21.15.28:0/1015813485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda1c19d000 msgr2=0x7fda1c19d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:58:42.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda23206640 1 --2- 172.21.15.28:0/1015813485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda1c19d000 0x7fda1c19d4b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fda14009fd0 tx=0x7fda1400b260 comp rx=0 tx=0).stop 2024-04-21T08:58:42.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda23206640 1 -- 172.21.15.28:0/1015813485 shutdown_connections 2024-04-21T08:58:42.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda23206640 1 --2- 172.21.15.28:0/1015813485 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fda0c0a3d20 0x7fda0c0a61e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda23206640 1 --2- 172.21.15.28:0/1015813485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda1c19d000 0x7fda1c19d4b0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda23206640 1 --2- 172.21.15.28:0/1015813485 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fda1c106150 0x7fda1c198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda23206640 1 --2- 172.21.15.28:0/1015813485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda1c101080 0x7fda1c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:58:42.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.068+0000 7fda23206640 1 -- 172.21.15.28:0/1015813485 >> 172.21.15.28:0/1015813485 conn(0x7fda1c078000 msgr2=0x7fda1c105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:58:42.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.069+0000 7fda23206640 1 -- 172.21.15.28:0/1015813485 shutdown_connections 2024-04-21T08:58:42.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:58:42.069+0000 7fda23206640 1 -- 172.21.15.28:0/1015813485 wait complete. 2024-04-21T08:58:42.082 INFO:tasks.cephfs.filesystem:scrub status for tag:d1b52aae-f4fb-4a24-a149-bfc4213aad47 - {'path': '/', 'tag': 'd1b52aae-f4fb-4a24-a149-bfc4213aad47', 'options': 'recursive,force'} 2024-04-21T08:58:42.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: pgmap v230: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:58:42.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:58:42.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:58:42.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:58:42.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:42.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:42.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:42.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:42.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:42.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:42.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:42.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:42.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:42 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: pgmap v230: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:58:42.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:58:42.857 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:58:42.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:58:42.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:42.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:42.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:42.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:42.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:42.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:42.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:42.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:42.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:42 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: pgmap v230: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:58:42.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:58:42.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:58:42.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:58:42.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:42.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:42.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:42.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:42.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:42.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:42.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:42.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:42.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:42.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:42 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:43.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:43 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:58:43.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:43 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:43.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:43 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:58:43.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:43 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:43.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:43 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:58:43.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:43 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:43.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:43 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:58:43.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:43 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:43.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:43 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:58:43.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:43 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:43.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:43 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:58:43.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:43 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:58:44.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:44 smithi152 ceph-mon[31401]: pgmap v231: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 16 op/s 2024-04-21T08:58:44.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:44 smithi028 ceph-mon[28003]: pgmap v231: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 16 op/s 2024-04-21T08:58:44.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:44 smithi171 ceph-mon[31358]: pgmap v231: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 16 op/s 2024-04-21T08:58:46.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:46 smithi152 ceph-mon[31401]: pgmap v232: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 27 op/s 2024-04-21T08:58:46.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:46 smithi028 ceph-mon[28003]: pgmap v232: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 27 op/s 2024-04-21T08:58:46.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:46 smithi171 ceph-mon[31358]: pgmap v232: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 27 op/s 2024-04-21T08:58:48.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:48 smithi152 ceph-mon[31401]: pgmap v233: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:58:48.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:48 smithi028 ceph-mon[28003]: pgmap v233: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:58:48.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:48 smithi171 ceph-mon[31358]: pgmap v233: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:58:50.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:50 smithi152 ceph-mon[31401]: pgmap v234: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 29 op/s 2024-04-21T08:58:50.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:50 smithi028 ceph-mon[28003]: pgmap v234: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 29 op/s 2024-04-21T08:58:50.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:50 smithi171 ceph-mon[31358]: pgmap v234: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 29 op/s 2024-04-21T08:58:52.672 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:52 smithi171 ceph-mon[31358]: pgmap v235: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 op/s 2024-04-21T08:58:52.672 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:52 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:58:52.672 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:52 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:58:52.672 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:52 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:58:52.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:52 smithi152 ceph-mon[31401]: pgmap v235: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 op/s 2024-04-21T08:58:52.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:52 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:58:52.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:52 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:58:52.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:52 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:58:52.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:52 smithi028 ceph-mon[28003]: pgmap v235: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 op/s 2024-04-21T08:58:52.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:52 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:58:52.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:52 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:58:52.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:52 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:58:54.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:54 smithi152 ceph-mon[31401]: pgmap v236: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:58:54.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:54 smithi028 ceph-mon[28003]: pgmap v236: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:58:54.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:54 smithi171 ceph-mon[31358]: pgmap v236: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:58:56.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:56 smithi152 ceph-mon[31401]: pgmap v237: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 19 op/s 2024-04-21T08:58:56.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:56 smithi028 ceph-mon[28003]: pgmap v237: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 19 op/s 2024-04-21T08:58:56.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:56 smithi171 ceph-mon[31358]: pgmap v237: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 19 op/s 2024-04-21T08:58:58.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:58:58 smithi152 ceph-mon[31401]: pgmap v238: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-04-21T08:58:58.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:58:58 smithi028 ceph-mon[28003]: pgmap v238: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-04-21T08:58:58.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:58:58 smithi171 ceph-mon[31358]: pgmap v238: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-04-21T08:59:00.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:00 smithi152 ceph-mon[31401]: pgmap v239: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:59:00.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:00 smithi028 ceph-mon[28003]: pgmap v239: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:59:00.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:00 smithi171 ceph-mon[31358]: pgmap v239: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:59:02.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:02 smithi152 ceph-mon[31401]: pgmap v240: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:59:02.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:02 smithi028 ceph-mon[28003]: pgmap v240: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:59:02.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:02 smithi171 ceph-mon[31358]: pgmap v240: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:59:04.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:04 smithi152 ceph-mon[31401]: pgmap v241: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T08:59:04.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:04 smithi028 ceph-mon[28003]: pgmap v241: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T08:59:04.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:04 smithi171 ceph-mon[31358]: pgmap v241: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T08:59:06.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:06 smithi152 ceph-mon[31401]: pgmap v242: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T08:59:06.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:06 smithi028 ceph-mon[28003]: pgmap v242: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T08:59:06.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:06 smithi171 ceph-mon[31358]: pgmap v242: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T08:59:08.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:08 smithi152 ceph-mon[31401]: pgmap v243: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T08:59:08.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:08 smithi152 ceph-mon[31401]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T08:59:08.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:08 smithi028 ceph-mon[28003]: pgmap v243: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T08:59:08.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:08 smithi028 ceph-mon[28003]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T08:59:08.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:08 smithi171 ceph-mon[31358]: pgmap v243: 129 pgs: 129 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T08:59:08.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:08 smithi171 ceph-mon[31358]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T08:59:10.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:10 smithi152 ceph-mon[31401]: pgmap v245: 129 pgs: 9 active+clean+snaptrim, 93 active+clean, 27 active+clean+snaptrim_wait; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-21T08:59:10.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:10 smithi028 ceph-mon[28003]: pgmap v245: 129 pgs: 9 active+clean+snaptrim, 93 active+clean, 27 active+clean+snaptrim_wait; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-21T08:59:10.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:10 smithi171 ceph-mon[31358]: pgmap v245: 129 pgs: 9 active+clean+snaptrim, 93 active+clean, 27 active+clean+snaptrim_wait; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-21T08:59:12.083 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:59:12.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:59:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.205+0000 7f64908d9640 1 -- 172.21.15.28:0/1876229098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c101a30 msgr2=0x7f648c105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.205+0000 7f64908d9640 1 --2- 172.21.15.28:0/1876229098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c101a30 0x7f648c105a70 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f6480009900 tx=0x7f648002bc30 comp rx=0 tx=0).stop 2024-04-21T08:59:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.205+0000 7f64908d9640 1 -- 172.21.15.28:0/1876229098 shutdown_connections 2024-04-21T08:59:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.205+0000 7f64908d9640 1 --2- 172.21.15.28:0/1876229098 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f648c106150 0x7f648c10dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.205+0000 7f64908d9640 1 --2- 172.21.15.28:0/1876229098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c101a30 0x7f648c105a70 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.205+0000 7f64908d9640 1 --2- 172.21.15.28:0/1876229098 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f648c101080 0x7f648c101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.205+0000 7f64908d9640 1 -- 172.21.15.28:0/1876229098 >> 172.21.15.28:0/1876229098 conn(0x7f648c078000 msgr2=0x7f648c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.205+0000 7f64908d9640 1 -- 172.21.15.28:0/1876229098 shutdown_connections 2024-04-21T08:59:12.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.205+0000 7f64908d9640 1 -- 172.21.15.28:0/1876229098 wait complete. 2024-04-21T08:59:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.205+0000 7f64908d9640 1 Processor -- start 2024-04-21T08:59:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f64908d9640 1 -- start start 2024-04-21T08:59:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f64908d9640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f648c101080 0x7f648c196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f64908d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c106150 0x7f648c196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f64908d9640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f648c19add0 0x7f648c19b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f64908d9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f648c10ffb0 con 0x7f648c106150 2024-04-21T08:59:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f64908d9640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f648c10fe30 con 0x7f648c19add0 2024-04-21T08:59:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f64908d9640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f648c110130 con 0x7f648c101080 2024-04-21T08:59:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f6489d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c106150 0x7f648c196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f6489d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c106150 0x7f648c196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55512/0 (socket says 172.21.15.28:55512) 2024-04-21T08:59:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f6489d9b640 1 -- 172.21.15.28:0/3376605881 learned_addr learned my addr 172.21.15.28:0/3376605881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f648a59c640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f648c101080 0x7f648c196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f648ad9d640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f648c19add0 0x7f648c19b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f648a59c640 1 -- 172.21.15.28:0/3376605881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f648c19add0 msgr2=0x7f648c19b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f648a59c640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f648c19add0 0x7f648c19b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f648a59c640 1 -- 172.21.15.28:0/3376605881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c106150 msgr2=0x7f648c196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f648a59c640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c106150 0x7f648c196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f648a59c640 1 -- 172.21.15.28:0/3376605881 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c19b8c0 con 0x7f648c101080 2024-04-21T08:59:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f648ad9d640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f648c19add0 0x7f648c19b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.206+0000 7f6489d9b640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c106150 0x7f648c196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:59:12.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.207+0000 7f648a59c640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f648c101080 0x7f648c196510 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f647c00e9f0 tx=0x7f647c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.207+0000 7f647b7fe640 1 -- 172.21.15.28:0/3376605881 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c00ccc0 con 0x7f648c101080 2024-04-21T08:59:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.207+0000 7f64908d9640 1 -- 172.21.15.28:0/3376605881 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f648c19bbb0 con 0x7f648c101080 2024-04-21T08:59:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.207+0000 7f647b7fe640 1 -- 172.21.15.28:0/3376605881 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f647c0108e0 con 0x7f648c101080 2024-04-21T08:59:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.207+0000 7f647b7fe640 1 -- 172.21.15.28:0/3376605881 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c018880 con 0x7f648c101080 2024-04-21T08:59:12.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.207+0000 7f64908d9640 1 -- 172.21.15.28:0/3376605881 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f648c1a33a0 con 0x7f648c101080 2024-04-21T08:59:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.209+0000 7f64935a9740 1 Processor -- start 2024-04-21T08:59:12.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f64935a9740 1 -- start start 2024-04-21T08:59:12.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f64935a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2c02b0 0x56001a2c0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f64935a9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c0bd0 0x56001a2cd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f64935a9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2cde90 0x56001a2d0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f64935a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56001a2251e0 con 0x56001a2c02b0 2024-04-21T08:59:12.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f64935a9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56001a1c2390 con 0x56001a2c0bd0 2024-04-21T08:59:12.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f64935a9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56001a2211a0 con 0x56001a2cde90 2024-04-21T08:59:12.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f648ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2c02b0 0x56001a2c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f648ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2c02b0 0x56001a2c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55520/0 (socket says 172.21.15.28:55520) 2024-04-21T08:59:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f6489d9b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2cde90 0x56001a2d0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f648ad9d640 1 -- 172.21.15.28:0/2024859991 learned_addr learned my addr 172.21.15.28:0/2024859991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f647b7fe640 1 -- 172.21.15.28:0/3376605881 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f647c018a20 con 0x7f648c101080 2024-04-21T08:59:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f6489d9b640 1 -- 172.21.15.28:0/2024859991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c0bd0 msgr2=0x56001a2cd950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:59:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f6489d9b640 1 --2- 172.21.15.28:0/2024859991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c0bd0 0x56001a2cd950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f6489d9b640 1 -- 172.21.15.28:0/2024859991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2c02b0 msgr2=0x56001a2c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f6489d9b640 1 --2- 172.21.15.28:0/2024859991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2c02b0 0x56001a2c0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f6489d9b640 1 -- 172.21.15.28:0/2024859991 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56001a0017d0 con 0x56001a2cde90 2024-04-21T08:59:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.210+0000 7f648ad9d640 1 --2- 172.21.15.28:0/2024859991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2c02b0 0x56001a2c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:59:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.211+0000 7f6489d9b640 1 --2- 172.21.15.28:0/2024859991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2cde90 0x56001a2d0280 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6480005ec0 tx=0x7f648002c5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f5a92db695734f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.211+0000 7f6478ff9640 1 -- 172.21.15.28:0/2024859991 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6480045070 con 0x56001a2cde90 2024-04-21T08:59:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.211+0000 7f6478ff9640 1 -- 172.21.15.28:0/2024859991 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f648002c7e0 con 0x56001a2cde90 2024-04-21T08:59:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.211+0000 7f6478ff9640 1 -- 172.21.15.28:0/2024859991 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6480003b90 con 0x56001a2cde90 2024-04-21T08:59:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.211+0000 7f647b7fe640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f64740a3d20 0x7f64740a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.212+0000 7f648ad9d640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f64740a3d20 0x7f64740a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.212+0000 7f64935a9740 1 -- 172.21.15.28:0/2024859991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2cde90 msgr2=0x56001a2d0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.212+0000 7f64935a9740 1 --2- 172.21.15.28:0/2024859991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2cde90 0x56001a2d0280 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6480005ec0 tx=0x7f648002c5f0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.212+0000 7f64935a9740 1 -- 172.21.15.28:0/2024859991 shutdown_connections 2024-04-21T08:59:12.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.212+0000 7f64935a9740 1 --2- 172.21.15.28:0/2024859991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2cde90 0x56001a2d0280 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.212+0000 7f64935a9740 1 --2- 172.21.15.28:0/2024859991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c0bd0 0x56001a2cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.212+0000 7f64935a9740 1 --2- 172.21.15.28:0/2024859991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2c02b0 0x56001a2c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.212+0000 7f64935a9740 1 -- 172.21.15.28:0/2024859991 >> 172.21.15.28:0/2024859991 conn(0x56001a2bacb0 msgr2=0x56001a2bb090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.212+0000 7f647b7fe640 1 -- 172.21.15.28:0/3376605881 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7f647c0d1030 con 0x7f648c101080 2024-04-21T08:59:12.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.212+0000 7f648ad9d640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f64740a3d20 0x7f64740a61e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6484005f40 tx=0x7f648400d040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.212+0000 7f64935a9740 1 -- 172.21.15.28:0/2024859991 shutdown_connections 2024-04-21T08:59:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.212+0000 7f64935a9740 1 -- 172.21.15.28:0/2024859991 wait complete. 2024-04-21T08:59:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f64935a9740 1 -- start start 2024-04-21T08:59:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f64935a9740 1 Processor -- start 2024-04-21T08:59:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f64935a9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 0x56001a2caec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f64935a9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2c0bd0 0x56001a2cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f64935a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2cde90 0x56001a2cb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f64935a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56001a2d0bf0 con 0x56001a2cde90 2024-04-21T08:59:12.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f64935a9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56001a1de2f0 con 0x56001a2c02b0 2024-04-21T08:59:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f64935a9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56001a2d0d70 con 0x56001a2c0bd0 2024-04-21T08:59:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f64935a9740 10 client.0 _ll_register_callbacks cb 0x56001a1eb580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f6489d9b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2c0bd0 0x56001a2cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f648ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2cde90 0x56001a2cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f648a59c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 0x56001a2caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f6489d9b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2c0bd0 0x56001a2cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:37434/0 (socket says 172.21.15.28:37434) 2024-04-21T08:59:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f6489d9b640 1 -- 172.21.15.28:0/3912992598 learned_addr learned my addr 172.21.15.28:0/3912992598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f6489d9b640 1 -- 172.21.15.28:0/3912992598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 msgr2=0x56001a2caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f6489d9b640 1 --2- 172.21.15.28:0/3912992598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 0x56001a2caec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f6489d9b640 1 -- 172.21.15.28:0/3912992598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2cde90 msgr2=0x56001a2cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.213+0000 7f6489d9b640 1 --2- 172.21.15.28:0/3912992598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2cde90 0x56001a2cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.214+0000 7f6489d9b640 1 -- 172.21.15.28:0/3912992598 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560019f373f0 con 0x56001a2c0bd0 2024-04-21T08:59:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.214+0000 7f6489d9b640 1 --2- 172.21.15.28:0/3912992598 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2c0bd0 0x56001a2cb400 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f648002c780 tx=0x7f648003f6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57f2fcf443af9cfd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.214+0000 7f6471ffb640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:59:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.214+0000 7f6471ffb640 1 -- 172.21.15.28:0/3912992598 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6480045040 con 0x56001a2c0bd0 2024-04-21T08:59:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.214+0000 7f64935a9740 1 -- 172.21.15.28:0/3912992598 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f648002c7e0 con 0x56001a2c0bd0 2024-04-21T08:59:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.214+0000 7f6471ffb640 1 -- 172.21.15.28:0/3912992598 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7f648003fd20 con 0x56001a2c0bd0 2024-04-21T08:59:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.214+0000 7f6471ffb640 1 -- 172.21.15.28:0/3912992598 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f648005a050 con 0x56001a2c0bd0 2024-04-21T08:59:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.214+0000 7f64935a9740 10 client.26824 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.214+0000 7f64935a9740 1 -- 172.21.15.28:0/3912992598 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560019ef70f0 con 0x56001a2c0bd0 2024-04-21T08:59:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.214+0000 7f6471ffb640 1 -- 172.21.15.28:0/3912992598 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f648002c7e0 con 0x56001a2c0bd0 2024-04-21T08:59:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.215+0000 7f6471ffb640 1 -- 172.21.15.28:0/3912992598 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f6480055440 con 0x56001a2c0bd0 2024-04-21T08:59:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.215+0000 7f64935a9740 10 client.26824 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.215+0000 7f64935a9740 10 client.26824 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.215+0000 7f64935a9740 20 client.26824 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.215+0000 7f64935a9740 1 --2- 172.21.15.28:0/3912992598 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56001a2ddcc0 0x56001a2e00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.215+0000 7f64935a9740 4 client.26824 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:59:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.215+0000 7f64935a9740 1 -- 172.21.15.28:0/3912992598 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56001a232c00 con 0x56001a2ddcc0 2024-04-21T08:59:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.215+0000 7f648a59c640 1 --2- 172.21.15.28:0/3912992598 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56001a2ddcc0 0x56001a2e00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.215+0000 7f648a59c640 1 --2- 172.21.15.28:0/3912992598 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56001a2ddcc0 0x56001a2e00b0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25561b0f5abd28ee server_cookie=88aa9b63ced72cfd in_seq=0 out_seq=0 2024-04-21T08:59:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.215+0000 7f6471ffb640 10 client.26824 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.216+0000 7f6471ffb640 1 -- 172.21.15.28:0/3912992598 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56001a232c00 con 0x56001a2ddcc0 2024-04-21T08:59:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.216+0000 7f6471ffb640 10 client.26824 handle_command_reply: tid=0 2024-04-21T08:59:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.216+0000 7f64935a9740 1 client.26824 shutdown 2024-04-21T08:59:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.217+0000 7f64935a9740 1 -- 172.21.15.28:0/3912992598 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2c0bd0 msgr2=0x56001a2cb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.217+0000 7f64935a9740 1 --2- 172.21.15.28:0/3912992598 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2c0bd0 0x56001a2cb400 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f648002c780 tx=0x7f648003f6f0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.217+0000 7f64935a9740 1 -- 172.21.15.28:0/3912992598 shutdown_connections 2024-04-21T08:59:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.217+0000 7f64935a9740 1 --2- 172.21.15.28:0/3912992598 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56001a2ddcc0 0x56001a2e00b0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.217+0000 7f64935a9740 1 --2- 172.21.15.28:0/3912992598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2cde90 0x56001a2cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.217+0000 7f64935a9740 1 --2- 172.21.15.28:0/3912992598 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2c0bd0 0x56001a2cb400 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.217+0000 7f64935a9740 1 --2- 172.21.15.28:0/3912992598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 0x56001a2caec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.217+0000 7f64935a9740 1 -- 172.21.15.28:0/3912992598 >> 172.21.15.28:0/3912992598 conn(0x56001a2bacb0 msgr2=0x56001a2bca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.217+0000 7f6471ffb640 0 client.26824 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:12.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.217+0000 7f64935a9740 1 -- 172.21.15.28:0/3912992598 shutdown_connections 2024-04-21T08:59:12.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.217+0000 7f64935a9740 1 -- 172.21.15.28:0/3912992598 wait complete. 2024-04-21T08:59:12.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.217+0000 7f64935a9740 20 client.26824 trim_cache size 0 max 16384 2024-04-21T08:59:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.238+0000 7f64935a9740 1 Processor -- start 2024-04-21T08:59:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.238+0000 7f64935a9740 1 -- start start 2024-04-21T08:59:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.238+0000 7f64935a9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 0x56001a2cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.238+0000 7f64935a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2c0bd0 0x56001a2d9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.238+0000 7f64935a9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2cde90 0x56001a2ca040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.238+0000 7f64935a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f648c1afc00 con 0x56001a2c0bd0 2024-04-21T08:59:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.238+0000 7f64935a9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f648c1ad4d0 con 0x56001a2c02b0 2024-04-21T08:59:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.238+0000 7f64935a9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f648c1ab340 con 0x56001a2cde90 2024-04-21T08:59:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.239+0000 7f6489d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2c0bd0 0x56001a2d9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.239+0000 7f648ad9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2cde90 0x56001a2ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.239+0000 7f6489d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2c0bd0 0x56001a2d9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55532/0 (socket says 172.21.15.28:55532) 2024-04-21T08:59:12.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.239+0000 7f6489d9b640 1 -- 172.21.15.28:0/1580867610 learned_addr learned my addr 172.21.15.28:0/1580867610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.239+0000 7f648a59c640 1 --2- 172.21.15.28:0/1580867610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 0x56001a2cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.239+0000 7f6489d9b640 1 -- 172.21.15.28:0/1580867610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2cde90 msgr2=0x56001a2ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.239+0000 7f6489d9b640 1 --2- 172.21.15.28:0/1580867610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2cde90 0x56001a2ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.239+0000 7f6489d9b640 1 -- 172.21.15.28:0/1580867610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 msgr2=0x56001a2cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.239+0000 7f6489d9b640 1 --2- 172.21.15.28:0/1580867610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 0x56001a2cd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.239+0000 7f6489d9b640 1 -- 172.21.15.28:0/1580867610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56001a2b51c0 con 0x56001a2c0bd0 2024-04-21T08:59:12.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.239+0000 7f648a59c640 1 --2- 172.21.15.28:0/1580867610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 0x56001a2cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:12.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.239+0000 7f6489d9b640 1 --2- 172.21.15.28:0/1580867610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2c0bd0 0x56001a2d9af0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f6480009230 tx=0x7f6480005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d848669a91722680 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.239+0000 7f6473fff640 1 -- 172.21.15.28:0/1580867610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6480063070 con 0x56001a2c0bd0 2024-04-21T08:59:12.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.239+0000 7f6473fff640 1 -- 172.21.15.28:0/1580867610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6480055e50 con 0x56001a2c0bd0 2024-04-21T08:59:12.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.239+0000 7f6473fff640 1 -- 172.21.15.28:0/1580867610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6480045040 con 0x56001a2c0bd0 2024-04-21T08:59:12.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.240+0000 7f64935a9740 1 -- 172.21.15.28:0/1580867610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2c0bd0 msgr2=0x56001a2d9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.240+0000 7f64935a9740 1 --2- 172.21.15.28:0/1580867610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2c0bd0 0x56001a2d9af0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f6480009230 tx=0x7f6480005b50 comp rx=0 tx=0).stop 2024-04-21T08:59:12.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.240+0000 7f64935a9740 1 -- 172.21.15.28:0/1580867610 shutdown_connections 2024-04-21T08:59:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.240+0000 7f64935a9740 1 --2- 172.21.15.28:0/1580867610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2cde90 0x56001a2ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.240+0000 7f64935a9740 1 --2- 172.21.15.28:0/1580867610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2c0bd0 0x56001a2d9af0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.240+0000 7f64935a9740 1 --2- 172.21.15.28:0/1580867610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 0x56001a2cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.240+0000 7f64935a9740 1 -- 172.21.15.28:0/1580867610 >> 172.21.15.28:0/1580867610 conn(0x56001a2bacb0 msgr2=0x56001a2b94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.240+0000 7f64935a9740 1 -- 172.21.15.28:0/1580867610 shutdown_connections 2024-04-21T08:59:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.240+0000 7f64935a9740 1 -- 172.21.15.28:0/1580867610 wait complete. 2024-04-21T08:59:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.240+0000 7f64935a9740 1 -- start start 2024-04-21T08:59:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f64935a9740 1 Processor -- start 2024-04-21T08:59:12.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f64935a9740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 0x56001a2d64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f64935a9740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2c0bd0 0x56001a2c7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f64935a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2cde90 0x56001a2c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f64935a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f648c1ab990 con 0x56001a2cde90 2024-04-21T08:59:12.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f64935a9740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f648c1ace00 con 0x56001a2c02b0 2024-04-21T08:59:12.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f64935a9740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f648c1ac060 con 0x56001a2c0bd0 2024-04-21T08:59:12.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f64935a9740 10 client.0 _ll_register_callbacks cb 0x7f648400b360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:12.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f648ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2cde90 0x56001a2c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f648ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2cde90 0x56001a2c7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55538/0 (socket says 172.21.15.28:55538) 2024-04-21T08:59:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f6489d9b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2c0bd0 0x56001a2c7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f648ad9d640 1 -- 172.21.15.28:0/3199930152 learned_addr learned my addr 172.21.15.28:0/3199930152 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f648a59c640 1 --2- 172.21.15.28:0/3199930152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 0x56001a2d64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f6489d9b640 1 -- 172.21.15.28:0/3199930152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 msgr2=0x56001a2d64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f6489d9b640 1 --2- 172.21.15.28:0/3199930152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 0x56001a2d64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f6489d9b640 1 -- 172.21.15.28:0/3199930152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2cde90 msgr2=0x56001a2c7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f6489d9b640 1 --2- 172.21.15.28:0/3199930152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2cde90 0x56001a2c7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f6489d9b640 1 -- 172.21.15.28:0/3199930152 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56001a2d0280 con 0x56001a2c0bd0 2024-04-21T08:59:12.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.241+0000 7f648ad9d640 1 --2- 172.21.15.28:0/3199930152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2cde90 0x56001a2c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:59:12.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.242+0000 7f6489d9b640 1 --2- 172.21.15.28:0/3199930152 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2c0bd0 0x56001a2c7920 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x56001a2bbb30 tx=0x7f648002cd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=290c6b20386904a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.242+0000 7f6471ffb640 10 client.0 ms_handle_connect on v2:172.21.15.171:3300/0 2024-04-21T08:59:12.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.242+0000 7f6471ffb640 1 -- 172.21.15.28:0/3199930152 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6480063050 con 0x56001a2c0bd0 2024-04-21T08:59:12.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.242+0000 7f6471ffb640 1 -- 172.21.15.28:0/3199930152 <== mon.1 v2:172.21.15.171:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7f6480055c30 con 0x56001a2c0bd0 2024-04-21T08:59:12.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.242+0000 7f64935a9740 1 -- 172.21.15.28:0/3199930152 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6480055e50 con 0x56001a2c0bd0 2024-04-21T08:59:12.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.242+0000 7f6471ffb640 1 -- 172.21.15.28:0/3199930152 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6480045040 con 0x56001a2c0bd0 2024-04-21T08:59:12.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.242+0000 7f64935a9740 10 client.26830 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:12.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.242+0000 7f64935a9740 1 -- 172.21.15.28:0/3199930152 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56001a343d90 con 0x56001a2c0bd0 2024-04-21T08:59:12.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.242+0000 7f6471ffb640 1 -- 172.21.15.28:0/3199930152 <== mon.1 v2:172.21.15.171:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6480055e50 con 0x56001a2c0bd0 2024-04-21T08:59:12.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.242+0000 7f6471ffb640 1 -- 172.21.15.28:0/3199930152 <== mon.1 v2:172.21.15.171:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f648004a950 con 0x56001a2c0bd0 2024-04-21T08:59:12.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.243+0000 7f64935a9740 10 client.26830 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:12.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.243+0000 7f64935a9740 10 client.26830 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:12.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.243+0000 7f64935a9740 20 client.26830 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:12.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.243+0000 7f64935a9740 1 --2- 172.21.15.28:0/3199930152 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56001a344390 0x56001a346780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.243+0000 7f64935a9740 4 client.26830 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:59:12.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.243+0000 7f64935a9740 1 -- 172.21.15.28:0/3199930152 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56001a22ca70 con 0x56001a344390 2024-04-21T08:59:12.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.243+0000 7f648a59c640 1 --2- 172.21.15.28:0/3199930152 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56001a344390 0x56001a346780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.243+0000 7f6471ffb640 10 client.26830 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:12.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.243+0000 7f648a59c640 1 --2- 172.21.15.28:0/3199930152 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56001a344390 0x56001a346780 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28dd9a44b6005284 server_cookie=bf404315aaea77ba in_seq=0 out_seq=0 2024-04-21T08:59:12.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f6471ffb640 1 -- 172.21.15.28:0/3199930152 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56001a22ca70 con 0x56001a344390 2024-04-21T08:59:12.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f6471ffb640 10 client.26830 handle_command_reply: tid=0 2024-04-21T08:59:12.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64935a9740 1 client.26830 shutdown 2024-04-21T08:59:12.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64935a9740 1 -- 172.21.15.28:0/3199930152 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2c0bd0 msgr2=0x56001a2c7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64935a9740 1 --2- 172.21.15.28:0/3199930152 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2c0bd0 0x56001a2c7920 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x56001a2bbb30 tx=0x7f648002cd20 comp rx=0 tx=0).stop 2024-04-21T08:59:12.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64935a9740 1 -- 172.21.15.28:0/3199930152 shutdown_connections 2024-04-21T08:59:12.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64935a9740 1 --2- 172.21.15.28:0/3199930152 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56001a344390 0x56001a346780 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64935a9740 1 --2- 172.21.15.28:0/3199930152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56001a2cde90 0x56001a2c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f6471ffb640 0 client.26830 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:12.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64935a9740 1 --2- 172.21.15.28:0/3199930152 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56001a2c0bd0 0x56001a2c7920 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64935a9740 1 --2- 172.21.15.28:0/3199930152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56001a2c02b0 0x56001a2d64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64935a9740 1 -- 172.21.15.28:0/3199930152 >> 172.21.15.28:0/3199930152 conn(0x56001a2bacb0 msgr2=0x56001a2b94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64935a9740 1 -- 172.21.15.28:0/3199930152 shutdown_connections 2024-04-21T08:59:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64935a9740 1 -- 172.21.15.28:0/3199930152 wait complete. 2024-04-21T08:59:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64935a9740 20 client.26830 trim_cache size 0 max 16384 2024-04-21T08:59:12.254 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:59:12.255 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T08:59:12.255 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T08:59:12.255 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:59:12.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64908d9640 1 -- 172.21.15.28:0/3376605881 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f64740a3d20 msgr2=0x7f64740a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64908d9640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f64740a3d20 0x7f64740a61e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6484005f40 tx=0x7f648400d040 comp rx=0 tx=0).stop 2024-04-21T08:59:12.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64908d9640 1 -- 172.21.15.28:0/3376605881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f648c101080 msgr2=0x7f648c196510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.244+0000 7f64908d9640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f648c101080 0x7f648c196510 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f647c00e9f0 tx=0x7f647c00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.245+0000 7f64908d9640 1 -- 172.21.15.28:0/3376605881 shutdown_connections 2024-04-21T08:59:12.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.245+0000 7f64908d9640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f64740a3d20 0x7f64740a61e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.245+0000 7f64908d9640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f648c19add0 0x7f648c19b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.245+0000 7f64908d9640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c106150 0x7f648c196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.245+0000 7f64908d9640 1 --2- 172.21.15.28:0/3376605881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f648c101080 0x7f648c196510 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.245+0000 7f64908d9640 1 -- 172.21.15.28:0/3376605881 >> 172.21.15.28:0/3376605881 conn(0x7f648c078000 msgr2=0x7f648c0ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.245+0000 7f64908d9640 1 -- 172.21.15.28:0/3376605881 shutdown_connections 2024-04-21T08:59:12.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.245+0000 7f64908d9640 1 -- 172.21.15.28:0/3376605881 wait complete. 2024-04-21T08:59:12.258 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:59:12.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:59:12.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.383+0000 7fe54d455640 1 -- 172.21.15.28:0/1037460635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54806a340 msgr2=0x7fe54806a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.383+0000 7fe54d455640 1 --2- 172.21.15.28:0/1037460635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54806a340 0x7fe54806a7a0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009930 tx=0x7fe53c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:59:12.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.383+0000 7fe54d455640 1 -- 172.21.15.28:0/1037460635 shutdown_connections 2024-04-21T08:59:12.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.383+0000 7fe54d455640 1 --2- 172.21.15.28:0/1037460635 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54806ace0 0x7fe54810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.383+0000 7fe54d455640 1 --2- 172.21.15.28:0/1037460635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54806a340 0x7fe54806a7a0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.383+0000 7fe54d455640 1 --2- 172.21.15.28:0/1037460635 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe5481080d0 0x7fe5481084b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.383+0000 7fe54d455640 1 -- 172.21.15.28:0/1037460635 >> 172.21.15.28:0/1037460635 conn(0x7fe548077e30 msgr2=0x7fe548078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.383+0000 7fe54d455640 1 -- 172.21.15.28:0/1037460635 shutdown_connections 2024-04-21T08:59:12.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.383+0000 7fe54d455640 1 -- 172.21.15.28:0/1037460635 wait complete. 2024-04-21T08:59:12.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.383+0000 7fe54d455640 1 Processor -- start 2024-04-21T08:59:12.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe54d455640 1 -- start start 2024-04-21T08:59:12.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe54d455640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54806a340 0x7fe548101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe54d455640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe54806ace0 0x7fe548101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe54d455640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5481080d0 0x7fe548101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe54d455640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe548112210 con 0x7fe5481080d0 2024-04-21T08:59:12.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe54d455640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe548112090 con 0x7fe54806a340 2024-04-21T08:59:12.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe54d455640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe548112390 con 0x7fe54806ace0 2024-04-21T08:59:12.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe5477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5481080d0 0x7fe548101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe5477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5481080d0 0x7fe548101c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55566/0 (socket says 172.21.15.28:55566) 2024-04-21T08:59:12.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe5477fe640 1 -- 172.21.15.28:0/3330681807 learned_addr learned my addr 172.21.15.28:0/3330681807 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe546ffd640 1 --2- 172.21.15.28:0/3330681807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54806a340 0x7fe548101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe5467fc640 1 --2- 172.21.15.28:0/3330681807 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe54806ace0 0x7fe548101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe5477fe640 1 -- 172.21.15.28:0/3330681807 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe54806ace0 msgr2=0x7fe548101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe5477fe640 1 --2- 172.21.15.28:0/3330681807 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe54806ace0 0x7fe548101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe5477fe640 1 -- 172.21.15.28:0/3330681807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54806a340 msgr2=0x7fe548101200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe5477fe640 1 --2- 172.21.15.28:0/3330681807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54806a340 0x7fe548101200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.384+0000 7fe5477fe640 1 -- 172.21.15.28:0/3330681807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5481a6bf0 con 0x7fe5481080d0 2024-04-21T08:59:12.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.385+0000 7fe5477fe640 1 --2- 172.21.15.28:0/3330681807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5481080d0 0x7fe548101c80 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fe54000eb10 tx=0x7fe54000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.385+0000 7fe52ffff640 1 -- 172.21.15.28:0/3330681807 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe540004b20 con 0x7fe5481080d0 2024-04-21T08:59:12.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.385+0000 7fe52ffff640 1 -- 172.21.15.28:0/3330681807 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe540004cc0 con 0x7fe5481080d0 2024-04-21T08:59:12.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.385+0000 7fe54d455640 1 -- 172.21.15.28:0/3330681807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5481a6ee0 con 0x7fe5481080d0 2024-04-21T08:59:12.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.385+0000 7fe52ffff640 1 -- 172.21.15.28:0/3330681807 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe540010670 con 0x7fe5481080d0 2024-04-21T08:59:12.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.385+0000 7fe54d455640 1 -- 172.21.15.28:0/3330681807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5481a73b0 con 0x7fe5481080d0 2024-04-21T08:59:12.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.386+0000 7fe54d455640 1 -- 172.21.15.28:0/3330681807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe548068550 con 0x7fe5481080d0 2024-04-21T08:59:12.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.387+0000 7fe52ffff640 1 -- 172.21.15.28:0/3330681807 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe540005020 con 0x7fe5481080d0 2024-04-21T08:59:12.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.388+0000 7fe52ffff640 1 --2- 172.21.15.28:0/3330681807 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe5300a3d20 0x7fe5300a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.388+0000 7fe546ffd640 1 --2- 172.21.15.28:0/3330681807 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe5300a3d20 0x7fe5300a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.388+0000 7fe52ffff640 1 -- 172.21.15.28:0/3330681807 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7fe5400d1270 con 0x7fe5481080d0 2024-04-21T08:59:12.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.388+0000 7fe546ffd640 1 --2- 172.21.15.28:0/3330681807 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe5300a3d20 0x7fe5300a61e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe538009b50 tx=0x7fe538005f90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.393+0000 7fe52ffff640 1 -- 172.21.15.28:0/3330681807 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe54009d4c0 con 0x7fe5481080d0 2024-04-21T08:59:12.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.631+0000 7fe54d455640 1 -- 172.21.15.28:0/3330681807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe54810a4c0 con 0x7fe5481080d0 2024-04-21T08:59:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.632+0000 7fe52ffff640 1 -- 172.21.15.28:0/3330681807 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7fe54009cdc0 con 0x7fe5481080d0 2024-04-21T08:59:12.632 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:59:12.632 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:59:12.633 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T08:59:12.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.634+0000 7fe54d455640 1 -- 172.21.15.28:0/3330681807 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe5300a3d20 msgr2=0x7fe5300a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.634+0000 7fe54d455640 1 --2- 172.21.15.28:0/3330681807 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe5300a3d20 0x7fe5300a61e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe538009b50 tx=0x7fe538005f90 comp rx=0 tx=0).stop 2024-04-21T08:59:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.634+0000 7fe54d455640 1 -- 172.21.15.28:0/3330681807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5481080d0 msgr2=0x7fe548101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.634+0000 7fe54d455640 1 --2- 172.21.15.28:0/3330681807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5481080d0 0x7fe548101c80 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fe54000eb10 tx=0x7fe54000c2d0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.634+0000 7fe54d455640 1 -- 172.21.15.28:0/3330681807 shutdown_connections 2024-04-21T08:59:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.634+0000 7fe54d455640 1 --2- 172.21.15.28:0/3330681807 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe5300a3d20 0x7fe5300a61e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.634+0000 7fe54d455640 1 --2- 172.21.15.28:0/3330681807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5481080d0 0x7fe548101c80 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.634+0000 7fe54d455640 1 --2- 172.21.15.28:0/3330681807 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe54806ace0 0x7fe548101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.634+0000 7fe54d455640 1 --2- 172.21.15.28:0/3330681807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe54806a340 0x7fe548101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.634+0000 7fe54d455640 1 -- 172.21.15.28:0/3330681807 >> 172.21.15.28:0/3330681807 conn(0x7fe548077e30 msgr2=0x7fe5480fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.634+0000 7fe54d455640 1 -- 172.21.15.28:0/3330681807 shutdown_connections 2024-04-21T08:59:12.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.634+0000 7fe54d455640 1 -- 172.21.15.28:0/3330681807 wait complete. 2024-04-21T08:59:12.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:59:12.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.750+0000 7fe4dcb8f640 1 -- 172.21.15.28:0/624111428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d8106170 msgr2=0x7fe4d810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.750+0000 7fe4dcb8f640 1 --2- 172.21.15.28:0/624111428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d8106170 0x7fe4d810dd40 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009ca0 tx=0x7fe4d002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.750+0000 7fe4dcb8f640 1 -- 172.21.15.28:0/624111428 shutdown_connections 2024-04-21T08:59:12.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.750+0000 7fe4dcb8f640 1 --2- 172.21.15.28:0/624111428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d8106170 0x7fe4d810dd40 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.750+0000 7fe4dcb8f640 1 --2- 172.21.15.28:0/624111428 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4d8101a50 0x7fe4d8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.750+0000 7fe4dcb8f640 1 --2- 172.21.15.28:0/624111428 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe4d81010a0 0x7fe4d8101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.750+0000 7fe4dcb8f640 1 -- 172.21.15.28:0/624111428 >> 172.21.15.28:0/624111428 conn(0x7fe4d8077eb0 msgr2=0x7fe4d80782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.750+0000 7fe4dcb8f640 1 -- 172.21.15.28:0/624111428 shutdown_connections 2024-04-21T08:59:12.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.750+0000 7fe4dcb8f640 1 -- 172.21.15.28:0/624111428 wait complete. 2024-04-21T08:59:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.751+0000 7fe4dcb8f640 1 Processor -- start 2024-04-21T08:59:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.751+0000 7fe4dcb8f640 1 -- start start 2024-04-21T08:59:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.751+0000 7fe4dcb8f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4d81010a0 0x7fe4d81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.751+0000 7fe4dcb8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d8106170 0x7fe4d8198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.751+0000 7fe4dcb8f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe4d819cf60 0x7fe4d819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.751+0000 7fe4dcb8f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d81121f0 con 0x7fe4d8106170 2024-04-21T08:59:12.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.751+0000 7fe4dcb8f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d8112070 con 0x7fe4d819cf60 2024-04-21T08:59:12.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.751+0000 7fe4dcb8f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d8112370 con 0x7fe4d81010a0 2024-04-21T08:59:12.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.751+0000 7fe4cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d8106170 0x7fe4d8198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.751+0000 7fe4cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d8106170 0x7fe4d8198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55580/0 (socket says 172.21.15.28:55580) 2024-04-21T08:59:12.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.752+0000 7fe4cffff640 1 -- 172.21.15.28:0/131142041 learned_addr learned my addr 172.21.15.28:0/131142041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.752+0000 7fe4d659c640 1 --2- 172.21.15.28:0/131142041 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4d81010a0 0x7fe4d81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.752+0000 7fe4d6d9d640 1 --2- 172.21.15.28:0/131142041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe4d819cf60 0x7fe4d819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.752+0000 7fe4cffff640 1 -- 172.21.15.28:0/131142041 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4d81010a0 msgr2=0x7fe4d81986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.752+0000 7fe4cffff640 1 --2- 172.21.15.28:0/131142041 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4d81010a0 0x7fe4d81986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.752+0000 7fe4cffff640 1 -- 172.21.15.28:0/131142041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe4d819cf60 msgr2=0x7fe4d819d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.752+0000 7fe4cffff640 1 --2- 172.21.15.28:0/131142041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe4d819cf60 0x7fe4d819d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.752+0000 7fe4cffff640 1 -- 172.21.15.28:0/131142041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d819da50 con 0x7fe4d8106170 2024-04-21T08:59:12.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.752+0000 7fe4cffff640 1 --2- 172.21.15.28:0/131142041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d8106170 0x7fe4d8198c30 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000d740 tx=0x7fe4c000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.752+0000 7fe4cf7fe640 1 -- 172.21.15.28:0/131142041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0017070 con 0x7fe4d8106170 2024-04-21T08:59:12.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.752+0000 7fe4cf7fe640 1 -- 172.21.15.28:0/131142041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4c000de50 con 0x7fe4d8106170 2024-04-21T08:59:12.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.752+0000 7fe4dcb8f640 1 -- 172.21.15.28:0/131142041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4d819dd40 con 0x7fe4d8106170 2024-04-21T08:59:12.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.752+0000 7fe4cf7fe640 1 -- 172.21.15.28:0/131142041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0004ce0 con 0x7fe4d8106170 2024-04-21T08:59:12.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.752+0000 7fe4dcb8f640 1 -- 172.21.15.28:0/131142041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4d81a54e0 con 0x7fe4d8106170 2024-04-21T08:59:12.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.755+0000 7fe4cf7fe640 1 -- 172.21.15.28:0/131142041 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe4c0004e80 con 0x7fe4d8106170 2024-04-21T08:59:12.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.755+0000 7fe4df77d740 1 Processor -- start 2024-04-21T08:59:12.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.755+0000 7fe4df77d740 1 -- start start 2024-04-21T08:59:12.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.755+0000 7fe4df77d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b3634d0 0x55b85b3638b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.755+0000 7fe4df77d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b363e80 0x55b85b36c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.755+0000 7fe4df77d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b36ca00 0x55b85b36ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.755+0000 7fe4df77d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b85b2c81f0 con 0x55b85b36ca00 2024-04-21T08:59:12.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.755+0000 7fe4df77d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55b85b2654b0 con 0x55b85b363e80 2024-04-21T08:59:12.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.755+0000 7fe4df77d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55b85b2c41b0 con 0x55b85b3634d0 2024-04-21T08:59:12.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.755+0000 7fe4cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b36ca00 0x55b85b36ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.755+0000 7fe4cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b36ca00 0x55b85b36ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55582/0 (socket says 172.21.15.28:55582) 2024-04-21T08:59:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.755+0000 7fe4cffff640 1 -- 172.21.15.28:0/2098946193 learned_addr learned my addr 172.21.15.28:0/2098946193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4cf7fe640 1 --2- 172.21.15.28:0/131142041 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4c40a3d20 0x7fe4c40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4d659c640 1 --2- 172.21.15.28:0/2098946193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b363e80 0x55b85b36c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4d6d9d640 1 --2- 172.21.15.28:0/2098946193 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b3634d0 0x55b85b3638b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4d6d9d640 1 --2- 172.21.15.28:0/131142041 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4c40a3d20 0x7fe4c40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4d659c640 1 -- 172.21.15.28:0/2098946193 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b3634d0 msgr2=0x55b85b3638b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4d659c640 1 --2- 172.21.15.28:0/2098946193 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b3634d0 0x55b85b3638b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4d659c640 1 -- 172.21.15.28:0/2098946193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b36ca00 msgr2=0x55b85b36ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4d659c640 1 --2- 172.21.15.28:0/2098946193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b36ca00 0x55b85b36ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4d659c640 1 -- 172.21.15.28:0/2098946193 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b85b2571b0 con 0x55b85b363e80 2024-04-21T08:59:12.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4cffff640 1 --2- 172.21.15.28:0/2098946193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b36ca00 0x55b85b36ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:59:12.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4cf7fe640 1 -- 172.21.15.28:0/131142041 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7fe4c00d04d0 con 0x7fe4d8106170 2024-04-21T08:59:12.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4d659c640 1 --2- 172.21.15.28:0/2098946193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b363e80 0x55b85b36c430 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800bbf0 tx=0x7fe4c80107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d20d25c2f2231b51 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4d6d9d640 1 --2- 172.21.15.28:0/131142041 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4c40a3d20 0x7fe4c40a61e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0004af0 tx=0x7fe4d0005c50 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4ccff9640 1 -- 172.21.15.28:0/2098946193 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8004030 con 0x55b85b363e80 2024-04-21T08:59:12.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4ccff9640 1 -- 172.21.15.28:0/2098946193 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4c80035f0 con 0x55b85b363e80 2024-04-21T08:59:12.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.756+0000 7fe4ccff9640 1 -- 172.21.15.28:0/2098946193 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c80116a0 con 0x55b85b363e80 2024-04-21T08:59:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.757+0000 7fe4df77d740 1 -- 172.21.15.28:0/2098946193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b363e80 msgr2=0x55b85b36c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.757+0000 7fe4df77d740 1 --2- 172.21.15.28:0/2098946193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b363e80 0x55b85b36c430 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe4c800bbf0 tx=0x7fe4c80107a0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.757+0000 7fe4df77d740 1 -- 172.21.15.28:0/2098946193 shutdown_connections 2024-04-21T08:59:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.757+0000 7fe4df77d740 1 --2- 172.21.15.28:0/2098946193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b36ca00 0x55b85b36ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.757+0000 7fe4df77d740 1 --2- 172.21.15.28:0/2098946193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b363e80 0x55b85b36c430 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.757+0000 7fe4df77d740 1 --2- 172.21.15.28:0/2098946193 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b3634d0 0x55b85b3638b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.757+0000 7fe4df77d740 1 -- 172.21.15.28:0/2098946193 >> 172.21.15.28:0/2098946193 conn(0x55b85b35dba0 msgr2=0x55b85b35df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.757+0000 7fe4df77d740 1 -- 172.21.15.28:0/2098946193 shutdown_connections 2024-04-21T08:59:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.757+0000 7fe4df77d740 1 -- 172.21.15.28:0/2098946193 wait complete. 2024-04-21T08:59:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4df77d740 1 -- start start 2024-04-21T08:59:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4df77d740 1 Processor -- start 2024-04-21T08:59:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4df77d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b3634d0 0x55b85b36c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4df77d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b363e80 0x55b85b3696c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4df77d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b36ca00 0x55b85b369cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4df77d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b85b36f970 con 0x55b85b363e80 2024-04-21T08:59:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4df77d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55b85b281500 con 0x55b85b3634d0 2024-04-21T08:59:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4df77d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55b85b36faf0 con 0x55b85b36ca00 2024-04-21T08:59:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4df77d740 10 client.0 _ll_register_callbacks cb 0x55b85b24fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b363e80 0x55b85b3696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b363e80 0x55b85b3696c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55596/0 (socket says 172.21.15.28:55596) 2024-04-21T08:59:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4cffff640 1 -- 172.21.15.28:0/306158460 learned_addr learned my addr 172.21.15.28:0/306158460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4d659c640 1 --2- 172.21.15.28:0/306158460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b3634d0 0x55b85b36c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4d6d9d640 1 --2- 172.21.15.28:0/306158460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b36ca00 0x55b85b369cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4cffff640 1 -- 172.21.15.28:0/306158460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b36ca00 msgr2=0x55b85b369cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4cffff640 1 --2- 172.21.15.28:0/306158460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b36ca00 0x55b85b369cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4cffff640 1 -- 172.21.15.28:0/306158460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b3634d0 msgr2=0x55b85b36c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4cffff640 1 --2- 172.21.15.28:0/306158460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b3634d0 0x55b85b36c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4cffff640 1 -- 172.21.15.28:0/306158460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b85afda3f0 con 0x55b85b363e80 2024-04-21T08:59:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.758+0000 7fe4d6d9d640 1 --2- 172.21.15.28:0/306158460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b36ca00 0x55b85b369cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4cffff640 1 --2- 172.21.15.28:0/306158460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b363e80 0x55b85b3696c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0098aa0 tx=0x7fe4c000b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a963be29014a115 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4bdffb640 1 -- 172.21.15.28:0/306158460 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0020040 con 0x55b85b363e80 2024-04-21T08:59:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4bdffb640 1 -- 172.21.15.28:0/306158460 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7fe4c00a00b0 con 0x55b85b363e80 2024-04-21T08:59:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4df77d740 1 -- 172.21.15.28:0/306158460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4c80035f0 con 0x55b85b363e80 2024-04-21T08:59:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4bdffb640 1 -- 172.21.15.28:0/306158460 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe4c009c0d0 con 0x55b85b363e80 2024-04-21T08:59:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4df77d740 10 client.17703 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4df77d740 1 -- 172.21.15.28:0/306158460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b85b0a47d0 con 0x55b85b363e80 2024-04-21T08:59:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4bdffb640 1 -- 172.21.15.28:0/306158460 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4c80035f0 con 0x55b85b363e80 2024-04-21T08:59:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4bdffb640 1 -- 172.21.15.28:0/306158460 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe4c0098740 con 0x55b85b363e80 2024-04-21T08:59:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4df77d740 10 client.17703 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4df77d740 10 client.17703 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:59:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4df77d740 20 client.17703 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4df77d740 1 --2- 172.21.15.28:0/306158460 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b85b37ccb0 0x55b85b37f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4df77d740 4 client.17703 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:59:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.759+0000 7fe4df77d740 1 -- 172.21.15.28:0/306158460 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b85b267930 con 0x55b85b37ccb0 2024-04-21T08:59:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.760+0000 7fe4d659c640 1 --2- 172.21.15.28:0/306158460 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b85b37ccb0 0x55b85b37f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.760+0000 7fe4bdffb640 10 client.17703 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:59:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.760+0000 7fe4d659c640 1 --2- 172.21.15.28:0/306158460 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b85b37ccb0 0x55b85b37f0a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bd1ff423052d4a4d server_cookie=9688e50090c3d999 in_seq=0 out_seq=0 2024-04-21T08:59:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.761+0000 7fe4bdffb640 1 -- 172.21.15.28:0/306158460 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b85b267930 con 0x55b85b37ccb0 2024-04-21T08:59:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.761+0000 7fe4bdffb640 10 client.17703 handle_command_reply: tid=0 2024-04-21T08:59:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.761+0000 7fe4df77d740 1 client.17703 shutdown 2024-04-21T08:59:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.761+0000 7fe4df77d740 1 -- 172.21.15.28:0/306158460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b363e80 msgr2=0x55b85b3696c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.761+0000 7fe4df77d740 1 --2- 172.21.15.28:0/306158460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b363e80 0x55b85b3696c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0098aa0 tx=0x7fe4c000b4c0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.761+0000 7fe4df77d740 1 -- 172.21.15.28:0/306158460 shutdown_connections 2024-04-21T08:59:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.761+0000 7fe4df77d740 1 --2- 172.21.15.28:0/306158460 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b85b37ccb0 0x55b85b37f0a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.761+0000 7fe4df77d740 1 --2- 172.21.15.28:0/306158460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b36ca00 0x55b85b369cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.761+0000 7fe4bdffb640 0 client.17703 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:59:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.761+0000 7fe4df77d740 1 --2- 172.21.15.28:0/306158460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b363e80 0x55b85b3696c0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.761+0000 7fe4df77d740 1 --2- 172.21.15.28:0/306158460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b3634d0 0x55b85b36c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.761+0000 7fe4df77d740 1 -- 172.21.15.28:0/306158460 >> 172.21.15.28:0/306158460 conn(0x55b85b35dba0 msgr2=0x55b85b35f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.761+0000 7fe4df77d740 1 -- 172.21.15.28:0/306158460 shutdown_connections 2024-04-21T08:59:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.761+0000 7fe4df77d740 1 -- 172.21.15.28:0/306158460 wait complete. 2024-04-21T08:59:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.761+0000 7fe4df77d740 20 client.17703 trim_cache size 0 max 16384 2024-04-21T08:59:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.774+0000 7fe4df77d740 1 Processor -- start 2024-04-21T08:59:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.774+0000 7fe4df77d740 1 -- start start 2024-04-21T08:59:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.774+0000 7fe4df77d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b3634d0 0x55b85b378860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.774+0000 7fe4df77d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b363e80 0x55b85b364ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.774+0000 7fe4df77d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b36ca00 0x55b85b36bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4df77d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d81b1d40 con 0x55b85b363e80 2024-04-21T08:59:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4df77d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d81af610 con 0x55b85b3634d0 2024-04-21T08:59:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4df77d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d81ad480 con 0x55b85b36ca00 2024-04-21T08:59:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b363e80 0x55b85b364ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b363e80 0x55b85b364ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55610/0 (socket says 172.21.15.28:55610) 2024-04-21T08:59:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4cffff640 1 -- 172.21.15.28:0/70885390 learned_addr learned my addr 172.21.15.28:0/70885390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4d6d9d640 1 --2- 172.21.15.28:0/70885390 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b36ca00 0x55b85b36bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4d659c640 1 --2- 172.21.15.28:0/70885390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b3634d0 0x55b85b378860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4cffff640 1 -- 172.21.15.28:0/70885390 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b36ca00 msgr2=0x55b85b36bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4cffff640 1 --2- 172.21.15.28:0/70885390 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b36ca00 0x55b85b36bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4cffff640 1 -- 172.21.15.28:0/70885390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b3634d0 msgr2=0x55b85b378860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4cffff640 1 --2- 172.21.15.28:0/70885390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b3634d0 0x55b85b378860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4cffff640 1 -- 172.21.15.28:0/70885390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b85b362560 con 0x55b85b363e80 2024-04-21T08:59:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4cffff640 1 --2- 172.21.15.28:0/70885390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b363e80 0x55b85b364ae0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fe4c00d0220 tx=0x7fe4c00d0860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259dcc0d9f088e49 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4bffff640 1 -- 172.21.15.28:0/70885390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0020040 con 0x55b85b363e80 2024-04-21T08:59:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4bffff640 1 -- 172.21.15.28:0/70885390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4c00a03d0 con 0x55b85b363e80 2024-04-21T08:59:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.775+0000 7fe4bffff640 1 -- 172.21.15.28:0/70885390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0097ef0 con 0x55b85b363e80 2024-04-21T08:59:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 -- 172.21.15.28:0/70885390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b363e80 msgr2=0x55b85b364ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 --2- 172.21.15.28:0/70885390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b363e80 0x55b85b364ae0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fe4c00d0220 tx=0x7fe4c00d0860 comp rx=0 tx=0).stop 2024-04-21T08:59:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 -- 172.21.15.28:0/70885390 shutdown_connections 2024-04-21T08:59:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 --2- 172.21.15.28:0/70885390 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b36ca00 0x55b85b36bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 --2- 172.21.15.28:0/70885390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b363e80 0x55b85b364ae0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 --2- 172.21.15.28:0/70885390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b3634d0 0x55b85b378860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 -- 172.21.15.28:0/70885390 >> 172.21.15.28:0/70885390 conn(0x55b85b35dba0 msgr2=0x55b85b3643d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 -- 172.21.15.28:0/70885390 shutdown_connections 2024-04-21T08:59:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 -- 172.21.15.28:0/70885390 wait complete. 2024-04-21T08:59:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 -- start start 2024-04-21T08:59:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 Processor -- start 2024-04-21T08:59:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b3634d0 0x55b85b374d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b363e80 0x55b85b375250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b36ca00 0x55b85b3701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d81adad0 con 0x55b85b36ca00 2024-04-21T08:59:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d81aef40 con 0x55b85b3634d0 2024-04-21T08:59:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d81ae1a0 con 0x55b85b363e80 2024-04-21T08:59:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4df77d740 10 client.0 _ll_register_callbacks cb 0x7fe4c00d3d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.776+0000 7fe4d6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b36ca00 0x55b85b3701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4d6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b36ca00 0x55b85b3701f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55620/0 (socket says 172.21.15.28:55620) 2024-04-21T08:59:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4d6d9d640 1 -- 172.21.15.28:0/1869206217 learned_addr learned my addr 172.21.15.28:0/1869206217 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4d659c640 1 --2- 172.21.15.28:0/1869206217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b3634d0 0x55b85b374d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4cffff640 1 --2- 172.21.15.28:0/1869206217 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b363e80 0x55b85b375250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4d6d9d640 1 -- 172.21.15.28:0/1869206217 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b363e80 msgr2=0x55b85b375250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4d6d9d640 1 --2- 172.21.15.28:0/1869206217 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b363e80 0x55b85b375250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4d6d9d640 1 -- 172.21.15.28:0/1869206217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b3634d0 msgr2=0x55b85b374d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4d6d9d640 1 --2- 172.21.15.28:0/1869206217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b3634d0 0x55b85b374d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4d6d9d640 1 -- 172.21.15.28:0/1869206217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b85b370a20 con 0x55b85b36ca00 2024-04-21T08:59:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4d6d9d640 1 --2- 172.21.15.28:0/1869206217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b36ca00 0x55b85b3701f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009ca0 tx=0x7fe4d002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7702dce0e1bffc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4bdffb640 1 -- 172.21.15.28:0/1869206217 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0039040 con 0x55b85b36ca00 2024-04-21T08:59:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4bdffb640 1 -- 172.21.15.28:0/1869206217 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7fe4d0003d00 con 0x55b85b36ca00 2024-04-21T08:59:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4df77d740 1 -- 172.21.15.28:0/1869206217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4c00a03d0 con 0x55b85b36ca00 2024-04-21T08:59:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4bdffb640 1 -- 172.21.15.28:0/1869206217 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe4d0004040 con 0x55b85b36ca00 2024-04-21T08:59:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4df77d740 10 client.17709 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4df77d740 1 -- 172.21.15.28:0/1869206217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b85b3e2d90 con 0x55b85b36ca00 2024-04-21T08:59:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.777+0000 7fe4bdffb640 1 -- 172.21.15.28:0/1869206217 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4c00a03d0 con 0x55b85b36ca00 2024-04-21T08:59:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4bdffb640 1 -- 172.21.15.28:0/1869206217 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe4d0034880 con 0x55b85b36ca00 2024-04-21T08:59:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4df77d740 10 client.17709 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4df77d740 10 client.17709 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:59:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4df77d740 20 client.17709 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4df77d740 1 --2- 172.21.15.28:0/1869206217 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b85b3e3560 0x55b85b3e5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4df77d740 4 client.17709 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:59:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4df77d740 1 -- 172.21.15.28:0/1869206217 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b85b2d5c10 con 0x55b85b3e3560 2024-04-21T08:59:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4d659c640 1 --2- 172.21.15.28:0/1869206217 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b85b3e3560 0x55b85b3e5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4d659c640 1 --2- 172.21.15.28:0/1869206217 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b85b3e3560 0x55b85b3e5980 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d33129d134c66d2 server_cookie=8ab2676e6281cb41 in_seq=0 out_seq=0 2024-04-21T08:59:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4bdffb640 10 client.17709 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:59:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4bdffb640 1 -- 172.21.15.28:0/1869206217 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b85b2d5c10 con 0x55b85b3e3560 2024-04-21T08:59:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4bdffb640 10 client.17709 handle_command_reply: tid=0 2024-04-21T08:59:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4df77d740 1 client.17709 shutdown 2024-04-21T08:59:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4df77d740 1 -- 172.21.15.28:0/1869206217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b36ca00 msgr2=0x55b85b3701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4df77d740 1 --2- 172.21.15.28:0/1869206217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b36ca00 0x55b85b3701f0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009ca0 tx=0x7fe4d002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4df77d740 1 -- 172.21.15.28:0/1869206217 shutdown_connections 2024-04-21T08:59:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4df77d740 1 --2- 172.21.15.28:0/1869206217 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55b85b3e3560 0x55b85b3e5980 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4df77d740 1 --2- 172.21.15.28:0/1869206217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b85b36ca00 0x55b85b3701f0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4df77d740 1 --2- 172.21.15.28:0/1869206217 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55b85b363e80 0x55b85b375250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4bdffb640 0 client.17709 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:59:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4df77d740 1 --2- 172.21.15.28:0/1869206217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b85b3634d0 0x55b85b374d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.778+0000 7fe4df77d740 1 -- 172.21.15.28:0/1869206217 >> 172.21.15.28:0/1869206217 conn(0x55b85b35dba0 msgr2=0x55b85b364c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.779+0000 7fe4df77d740 1 -- 172.21.15.28:0/1869206217 shutdown_connections 2024-04-21T08:59:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.779+0000 7fe4df77d740 1 -- 172.21.15.28:0/1869206217 wait complete. 2024-04-21T08:59:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.779+0000 7fe4df77d740 20 client.17709 trim_cache size 0 max 16384 2024-04-21T08:59:12.791 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:59:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.779+0000 7fe4dcb8f640 1 -- 172.21.15.28:0/131142041 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4c40a3d20 msgr2=0x7fe4c40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.779+0000 7fe4dcb8f640 1 --2- 172.21.15.28:0/131142041 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4c40a3d20 0x7fe4c40a61e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0004af0 tx=0x7fe4d0005c50 comp rx=0 tx=0).stop 2024-04-21T08:59:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.779+0000 7fe4dcb8f640 1 -- 172.21.15.28:0/131142041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d8106170 msgr2=0x7fe4d8198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.779+0000 7fe4dcb8f640 1 --2- 172.21.15.28:0/131142041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d8106170 0x7fe4d8198c30 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000d740 tx=0x7fe4c000dc00 comp rx=0 tx=0).stop 2024-04-21T08:59:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.779+0000 7fe4dcb8f640 1 -- 172.21.15.28:0/131142041 shutdown_connections 2024-04-21T08:59:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.779+0000 7fe4dcb8f640 1 --2- 172.21.15.28:0/131142041 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4c40a3d20 0x7fe4c40a61e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.779+0000 7fe4dcb8f640 1 --2- 172.21.15.28:0/131142041 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe4d819cf60 0x7fe4d819d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.779+0000 7fe4dcb8f640 1 --2- 172.21.15.28:0/131142041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d8106170 0x7fe4d8198c30 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.779+0000 7fe4dcb8f640 1 --2- 172.21.15.28:0/131142041 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4d81010a0 0x7fe4d81986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.779+0000 7fe4dcb8f640 1 -- 172.21.15.28:0/131142041 >> 172.21.15.28:0/131142041 conn(0x7fe4d8077eb0 msgr2=0x7fe4d8100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.779+0000 7fe4dcb8f640 1 -- 172.21.15.28:0/131142041 shutdown_connections 2024-04-21T08:59:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.779+0000 7fe4dcb8f640 1 -- 172.21.15.28:0/131142041 wait complete. 2024-04-21T08:59:12.793 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:12 smithi028 ceph-mon[28003]: pgmap v246: 129 pgs: 15 active+clean+snaptrim, 73 active+clean, 41 active+clean+snaptrim_wait; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-21T08:59:12.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:59:12.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:12 smithi152 ceph-mon[31401]: pgmap v246: 129 pgs: 15 active+clean+snaptrim, 73 active+clean, 41 active+clean+snaptrim_wait; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-21T08:59:12.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:12 smithi171 ceph-mon[31358]: pgmap v246: 129 pgs: 15 active+clean+snaptrim, 73 active+clean, 41 active+clean+snaptrim_wait; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-21T08:59:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.898+0000 7f1e31932640 1 -- 172.21.15.28:0/2146297716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c06a840 msgr2=0x7f1e2c06ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.898+0000 7f1e31932640 1 --2- 172.21.15.28:0/2146297716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c06a840 0x7f1e2c06ac20 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c009900 tx=0x7f1e1c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:59:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.898+0000 7f1e31932640 1 -- 172.21.15.28:0/2146297716 shutdown_connections 2024-04-21T08:59:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.898+0000 7f1e31932640 1 --2- 172.21.15.28:0/2146297716 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1e2c107270 0x7f1e2c10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.898+0000 7f1e31932640 1 --2- 172.21.15.28:0/2146297716 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e2c06b1f0 0x7f1e2c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.898+0000 7f1e31932640 1 --2- 172.21.15.28:0/2146297716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c06a840 0x7f1e2c06ac20 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.898+0000 7f1e31932640 1 -- 172.21.15.28:0/2146297716 >> 172.21.15.28:0/2146297716 conn(0x7f1e2c077eb0 msgr2=0x7f1e2c0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.898+0000 7f1e31932640 1 -- 172.21.15.28:0/2146297716 shutdown_connections 2024-04-21T08:59:12.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.898+0000 7f1e31932640 1 -- 172.21.15.28:0/2146297716 wait complete. 2024-04-21T08:59:12.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.899+0000 7f1e31932640 1 Processor -- start 2024-04-21T08:59:12.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.899+0000 7f1e31932640 1 -- start start 2024-04-21T08:59:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.899+0000 7f1e31932640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1e2c06a840 0x7f1e2c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.899+0000 7f1e31932640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c107270 0x7f1e2c198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.899+0000 7f1e31932640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e2c19cff0 0x7f1e2c19d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.899+0000 7f1e31932640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c1121b0 con 0x7f1e2c107270 2024-04-21T08:59:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.899+0000 7f1e31932640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c112030 con 0x7f1e2c19cff0 2024-04-21T08:59:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.899+0000 7f1e31932640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c112330 con 0x7f1e2c06a840 2024-04-21T08:59:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.899+0000 7f1e2a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c107270 0x7f1e2c198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.899+0000 7f1e2a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c107270 0x7f1e2c198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55638/0 (socket says 172.21.15.28:55638) 2024-04-21T08:59:12.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.899+0000 7f1e2a7fc640 1 -- 172.21.15.28:0/1614551382 learned_addr learned my addr 172.21.15.28:0/1614551382 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.900+0000 7f1e2b7fe640 1 --2- 172.21.15.28:0/1614551382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e2c19cff0 0x7f1e2c19d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.900+0000 7f1e2a7fc640 1 -- 172.21.15.28:0/1614551382 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1e2c06a840 msgr2=0x7f1e2c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.900+0000 7f1e2affd640 1 --2- 172.21.15.28:0/1614551382 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1e2c06a840 0x7f1e2c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.900+0000 7f1e2a7fc640 1 --2- 172.21.15.28:0/1614551382 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1e2c06a840 0x7f1e2c198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.900+0000 7f1e2a7fc640 1 -- 172.21.15.28:0/1614551382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e2c19cff0 msgr2=0x7f1e2c19d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.900+0000 7f1e2a7fc640 1 --2- 172.21.15.28:0/1614551382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e2c19cff0 0x7f1e2c19d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.900+0000 7f1e2a7fc640 1 -- 172.21.15.28:0/1614551382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e2c19dae0 con 0x7f1e2c107270 2024-04-21T08:59:12.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.900+0000 7f1e2a7fc640 1 --2- 172.21.15.28:0/1614551382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c107270 0x7f1e2c198c70 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f1e20005fd0 tx=0x7f1e2000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.900+0000 7f1e30930640 1 -- 172.21.15.28:0/1614551382 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e20017070 con 0x7f1e2c107270 2024-04-21T08:59:12.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.900+0000 7f1e31932640 1 -- 172.21.15.28:0/1614551382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e2c19ddd0 con 0x7f1e2c107270 2024-04-21T08:59:12.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.900+0000 7f1e30930640 1 -- 172.21.15.28:0/1614551382 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e20009c30 con 0x7f1e2c107270 2024-04-21T08:59:12.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.900+0000 7f1e30930640 1 -- 172.21.15.28:0/1614551382 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e20003d30 con 0x7f1e2c107270 2024-04-21T08:59:12.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.900+0000 7f1e31932640 1 -- 172.21.15.28:0/1614551382 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e2c1a5610 con 0x7f1e2c107270 2024-04-21T08:59:12.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.902+0000 7f1e30930640 1 -- 172.21.15.28:0/1614551382 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1e20003ed0 con 0x7f1e2c107270 2024-04-21T08:59:12.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.902+0000 7f1e34604740 1 Processor -- start 2024-04-21T08:59:12.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.902+0000 7f1e34604740 1 -- start start 2024-04-21T08:59:12.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e34604740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17a4d0 0x55768d17a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e34604740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17ae80 0x55768d183430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e34604740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 0x55768d185e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e34604740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55768d0df1f0 con 0x55768d17a4d0 2024-04-21T08:59:12.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e34604740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55768d07c4b0 con 0x55768d17ae80 2024-04-21T08:59:12.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e34604740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55768d0db1b0 con 0x55768d183a00 2024-04-21T08:59:12.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e2b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17a4d0 0x55768d17a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e2b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17a4d0 0x55768d17a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55640/0 (socket says 172.21.15.28:55640) 2024-04-21T08:59:12.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e2b7fe640 1 -- 172.21.15.28:0/1820002607 learned_addr learned my addr 172.21.15.28:0/1820002607 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e2affd640 1 --2- 172.21.15.28:0/1820002607 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17ae80 0x55768d183430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e2a7fc640 1 --2- 172.21.15.28:0/1820002607 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 0x55768d185e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e30930640 1 --2- 172.21.15.28:0/1614551382 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1e140a3d20 0x7f1e140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e2b7fe640 1 -- 172.21.15.28:0/1820002607 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 msgr2=0x55768d185e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e2b7fe640 1 --2- 172.21.15.28:0/1820002607 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 0x55768d185e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e2b7fe640 1 -- 172.21.15.28:0/1820002607 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17ae80 msgr2=0x55768d183430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e2b7fe640 1 --2- 172.21.15.28:0/1820002607 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17ae80 0x55768d183430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e2b7fe640 1 -- 172.21.15.28:0/1820002607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55768d06e1b0 con 0x55768d17a4d0 2024-04-21T08:59:12.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e30930640 1 -- 172.21.15.28:0/1614551382 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7f1e200d0950 con 0x7f1e2c107270 2024-04-21T08:59:12.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e2affd640 1 --2- 172.21.15.28:0/1820002607 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17ae80 0x55768d183430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:59:12.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e2b7fe640 1 --2- 172.21.15.28:0/1614551382 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1e140a3d20 0x7f1e140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.903+0000 7f1e2b7fe640 1 --2- 172.21.15.28:0/1820002607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17a4d0 0x55768d17a8b0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009fd0 tx=0x7f1e2400f870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487556857d32a403 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.904+0000 7f1e11ffb640 1 -- 172.21.15.28:0/1820002607 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e2401c910 con 0x55768d17a4d0 2024-04-21T08:59:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.904+0000 7f1e11ffb640 1 -- 172.21.15.28:0/1820002607 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e2401b440 con 0x55768d17a4d0 2024-04-21T08:59:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.904+0000 7f1e11ffb640 1 -- 172.21.15.28:0/1820002607 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e24010630 con 0x55768d17a4d0 2024-04-21T08:59:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.904+0000 7f1e34604740 1 -- 172.21.15.28:0/1820002607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17a4d0 msgr2=0x55768d17a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.904+0000 7f1e2b7fe640 1 --2- 172.21.15.28:0/1614551382 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1e140a3d20 0x7f1e140a61e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1e24020000 tx=0x7f1e2401d000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.904+0000 7f1e34604740 1 --2- 172.21.15.28:0/1820002607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17a4d0 0x55768d17a8b0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f1e24009fd0 tx=0x7f1e2400f870 comp rx=0 tx=0).stop 2024-04-21T08:59:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.904+0000 7f1e34604740 1 -- 172.21.15.28:0/1820002607 shutdown_connections 2024-04-21T08:59:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.904+0000 7f1e34604740 1 --2- 172.21.15.28:0/1820002607 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 0x55768d185e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.904+0000 7f1e34604740 1 --2- 172.21.15.28:0/1820002607 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17ae80 0x55768d183430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.904+0000 7f1e34604740 1 --2- 172.21.15.28:0/1820002607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17a4d0 0x55768d17a8b0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.904+0000 7f1e34604740 1 -- 172.21.15.28:0/1820002607 >> 172.21.15.28:0/1820002607 conn(0x55768d174ba0 msgr2=0x55768d174f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.904+0000 7f1e34604740 1 -- 172.21.15.28:0/1820002607 shutdown_connections 2024-04-21T08:59:12.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.904+0000 7f1e34604740 1 -- 172.21.15.28:0/1820002607 wait complete. 2024-04-21T08:59:12.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.905+0000 7f1e34604740 1 -- start start 2024-04-21T08:59:12.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.905+0000 7f1e34604740 1 Processor -- start 2024-04-21T08:59:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.905+0000 7f1e34604740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d17a4d0 0x55768d183350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.905+0000 7f1e34604740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17ae80 0x55768d1806e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.905+0000 7f1e34604740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d183a00 0x55768d180cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.905+0000 7f1e34604740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55768d186970 con 0x55768d17ae80 2024-04-21T08:59:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.905+0000 7f1e34604740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55768d098500 con 0x55768d183a00 2024-04-21T08:59:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.905+0000 7f1e34604740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55768d186af0 con 0x55768d17a4d0 2024-04-21T08:59:12.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.905+0000 7f1e34604740 10 client.0 _ll_register_callbacks cb 0x55768d066b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:12.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.905+0000 7f1e2a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17ae80 0x55768d1806e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.905+0000 7f1e2a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17ae80 0x55768d1806e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55652/0 (socket says 172.21.15.28:55652) 2024-04-21T08:59:12.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.905+0000 7f1e2a7fc640 1 -- 172.21.15.28:0/3606829628 learned_addr learned my addr 172.21.15.28:0/3606829628 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.905+0000 7f1e2affd640 1 --2- 172.21.15.28:0/3606829628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d17a4d0 0x55768d183350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e2b7fe640 1 --2- 172.21.15.28:0/3606829628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d183a00 0x55768d180cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e2a7fc640 1 -- 172.21.15.28:0/3606829628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d17a4d0 msgr2=0x55768d183350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e2a7fc640 1 --2- 172.21.15.28:0/3606829628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d17a4d0 0x55768d183350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e2a7fc640 1 -- 172.21.15.28:0/3606829628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d183a00 msgr2=0x55768d180cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e2a7fc640 1 --2- 172.21.15.28:0/3606829628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d183a00 0x55768d180cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e2a7fc640 1 -- 172.21.15.28:0/3606829628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55768cdf13f0 con 0x55768d17ae80 2024-04-21T08:59:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e2b7fe640 1 --2- 172.21.15.28:0/3606829628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d183a00 0x55768d180cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e2a7fc640 1 --2- 172.21.15.28:0/3606829628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17ae80 0x55768d1806e0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f1e200ce250 tx=0x7f1e2009c0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca26beaa42c77afc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e0affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e0affd640 1 -- 172.21.15.28:0/3606829628 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e200a0850 con 0x55768d17ae80 2024-04-21T08:59:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e0affd640 1 -- 172.21.15.28:0/3606829628 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7f1e200cded0 con 0x55768d17ae80 2024-04-21T08:59:12.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e34604740 1 -- 172.21.15.28:0/3606829628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e2401b440 con 0x55768d17ae80 2024-04-21T08:59:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e0affd640 1 -- 172.21.15.28:0/3606829628 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1e200a5110 con 0x55768d17ae80 2024-04-21T08:59:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e34604740 10 client.17721 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e34604740 1 -- 172.21.15.28:0/3606829628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55768cebb7d0 con 0x55768d17ae80 2024-04-21T08:59:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e0affd640 1 -- 172.21.15.28:0/3606829628 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e2401b440 con 0x55768d17ae80 2024-04-21T08:59:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.906+0000 7f1e0affd640 1 -- 172.21.15.28:0/3606829628 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f1e2009b8f0 con 0x55768d17ae80 2024-04-21T08:59:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.907+0000 7f1e34604740 10 client.17721 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.907+0000 7f1e34604740 10 client.17721 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:59:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.907+0000 7f1e34604740 20 client.17721 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:12.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.907+0000 7f1e34604740 1 --2- 172.21.15.28:0/3606829628 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55768d193ca0 0x55768d196090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.907+0000 7f1e34604740 4 client.17721 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:59:12.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.907+0000 7f1e34604740 1 -- 172.21.15.28:0/3606829628 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55768d07e930 con 0x55768d193ca0 2024-04-21T08:59:12.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.907+0000 7f1e2affd640 1 --2- 172.21.15.28:0/3606829628 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55768d193ca0 0x55768d196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.907+0000 7f1e2affd640 1 --2- 172.21.15.28:0/3606829628 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55768d193ca0 0x55768d196090 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=acefdd55bb9b79cb server_cookie=fb0f2f6457a7f436 in_seq=0 out_seq=0 2024-04-21T08:59:12.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.907+0000 7f1e0affd640 10 client.17721 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:59:12.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.908+0000 7f1e0affd640 1 -- 172.21.15.28:0/3606829628 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55768d07e930 con 0x55768d193ca0 2024-04-21T08:59:12.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.908+0000 7f1e0affd640 10 client.17721 handle_command_reply: tid=0 2024-04-21T08:59:12.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.908+0000 7f1e34604740 1 client.17721 shutdown 2024-04-21T08:59:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.908+0000 7f1e34604740 1 -- 172.21.15.28:0/3606829628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17ae80 msgr2=0x55768d1806e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.908+0000 7f1e34604740 1 --2- 172.21.15.28:0/3606829628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17ae80 0x55768d1806e0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f1e200ce250 tx=0x7f1e2009c0a0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.908+0000 7f1e34604740 1 -- 172.21.15.28:0/3606829628 shutdown_connections 2024-04-21T08:59:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.908+0000 7f1e34604740 1 --2- 172.21.15.28:0/3606829628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d183a00 0x55768d180cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.908+0000 7f1e34604740 1 --2- 172.21.15.28:0/3606829628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17ae80 0x55768d1806e0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.908+0000 7f1e34604740 1 --2- 172.21.15.28:0/3606829628 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55768d193ca0 0x55768d196090 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.908+0000 7f1e34604740 1 --2- 172.21.15.28:0/3606829628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d17a4d0 0x55768d183350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.908+0000 7f1e0affd640 0 client.17721 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:59:12.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.908+0000 7f1e34604740 1 -- 172.21.15.28:0/3606829628 >> 172.21.15.28:0/3606829628 conn(0x55768d174ba0 msgr2=0x55768d1768f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.908+0000 7f1e34604740 1 -- 172.21.15.28:0/3606829628 shutdown_connections 2024-04-21T08:59:12.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.908+0000 7f1e34604740 1 -- 172.21.15.28:0/3606829628 wait complete. 2024-04-21T08:59:12.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.908+0000 7f1e34604740 20 client.17721 trim_cache size 0 max 16384 2024-04-21T08:59:12.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e34604740 1 Processor -- start 2024-04-21T08:59:12.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e34604740 1 -- start start 2024-04-21T08:59:12.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e34604740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17a4d0 0x55768d18f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e34604740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17ae80 0x55768d17ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e34604740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 0x55768d182e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e34604740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c1b1d70 con 0x55768d17a4d0 2024-04-21T08:59:12.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e34604740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c1af640 con 0x55768d17ae80 2024-04-21T08:59:12.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e34604740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c1ad4b0 con 0x55768d183a00 2024-04-21T08:59:12.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e2affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17a4d0 0x55768d18f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e2affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17a4d0 0x55768d18f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55664/0 (socket says 172.21.15.28:55664) 2024-04-21T08:59:12.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e2affd640 1 -- 172.21.15.28:0/4033606917 learned_addr learned my addr 172.21.15.28:0/4033606917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e2b7fe640 1 --2- 172.21.15.28:0/4033606917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 0x55768d182e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e2affd640 1 -- 172.21.15.28:0/4033606917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 msgr2=0x55768d182e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e2a7fc640 1 --2- 172.21.15.28:0/4033606917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17ae80 0x55768d17ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e2affd640 1 --2- 172.21.15.28:0/4033606917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 0x55768d182e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e2affd640 1 -- 172.21.15.28:0/4033606917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17ae80 msgr2=0x55768d17ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e2affd640 1 --2- 172.21.15.28:0/4033606917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17ae80 0x55768d17ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e2affd640 1 -- 172.21.15.28:0/4033606917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55768d181060 con 0x55768d17a4d0 2024-04-21T08:59:12.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.922+0000 7f1e2b7fe640 1 --2- 172.21.15.28:0/4033606917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 0x55768d182e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:12.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.923+0000 7f1e2affd640 1 --2- 172.21.15.28:0/4033606917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17a4d0 0x55768d18f830 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c0096c0 tx=0x7f1e1c02c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd9e02b5eb1d1d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.923+0000 7f1e117fa640 1 -- 172.21.15.28:0/4033606917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c004320 con 0x55768d17a4d0 2024-04-21T08:59:12.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.923+0000 7f1e117fa640 1 -- 172.21.15.28:0/4033606917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e1c02cc90 con 0x55768d17a4d0 2024-04-21T08:59:12.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.923+0000 7f1e117fa640 1 -- 172.21.15.28:0/4033606917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c004920 con 0x55768d17a4d0 2024-04-21T08:59:12.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.923+0000 7f1e34604740 1 -- 172.21.15.28:0/4033606917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17a4d0 msgr2=0x55768d18f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.923+0000 7f1e34604740 1 --2- 172.21.15.28:0/4033606917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17a4d0 0x55768d18f830 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c0096c0 tx=0x7f1e1c02c330 comp rx=0 tx=0).stop 2024-04-21T08:59:12.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.923+0000 7f1e34604740 1 -- 172.21.15.28:0/4033606917 shutdown_connections 2024-04-21T08:59:12.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.923+0000 7f1e34604740 1 --2- 172.21.15.28:0/4033606917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 0x55768d182e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.923+0000 7f1e34604740 1 --2- 172.21.15.28:0/4033606917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17ae80 0x55768d17ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.923+0000 7f1e34604740 1 --2- 172.21.15.28:0/4033606917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17a4d0 0x55768d18f830 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.923+0000 7f1e34604740 1 -- 172.21.15.28:0/4033606917 >> 172.21.15.28:0/4033606917 conn(0x55768d174ba0 msgr2=0x55768d173c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.923+0000 7f1e34604740 1 -- 172.21.15.28:0/4033606917 shutdown_connections 2024-04-21T08:59:12.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.923+0000 7f1e34604740 1 -- 172.21.15.28:0/4033606917 wait complete. 2024-04-21T08:59:12.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e34604740 1 -- start start 2024-04-21T08:59:12.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e34604740 1 Processor -- start 2024-04-21T08:59:12.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e34604740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17a4d0 0x55768d17d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e34604740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17ae80 0x55768d17d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e34604740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 0x55768d17df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e34604740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c1adb00 con 0x55768d17ae80 2024-04-21T08:59:12.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e34604740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c1aef70 con 0x55768d17a4d0 2024-04-21T08:59:12.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e34604740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2c1ae1d0 con 0x55768d183a00 2024-04-21T08:59:12.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e34604740 10 client.0 _ll_register_callbacks cb 0x7f1e2009c270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:12.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e2a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17ae80 0x55768d17d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e2a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17ae80 0x55768d17d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55674/0 (socket says 172.21.15.28:55674) 2024-04-21T08:59:12.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e2a7fc640 1 -- 172.21.15.28:0/2229548656 learned_addr learned my addr 172.21.15.28:0/2229548656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:12.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e2affd640 1 --2- 172.21.15.28:0/2229548656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17a4d0 0x55768d17d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e2b7fe640 1 --2- 172.21.15.28:0/2229548656 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 0x55768d17df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e2a7fc640 1 -- 172.21.15.28:0/2229548656 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 msgr2=0x55768d17df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e2a7fc640 1 --2- 172.21.15.28:0/2229548656 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 0x55768d17df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.924+0000 7f1e2a7fc640 1 -- 172.21.15.28:0/2229548656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17a4d0 msgr2=0x55768d17d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:12.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e2a7fc640 1 --2- 172.21.15.28:0/2229548656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17a4d0 0x55768d17d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e2a7fc640 1 -- 172.21.15.28:0/2229548656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55768d1f9280 con 0x55768d17ae80 2024-04-21T08:59:12.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e2a7fc640 1 --2- 172.21.15.28:0/2229548656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17ae80 0x55768d17d9e0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f1e2009bd90 tx=0x7f1e20098520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35034bc402b9ec55 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e0affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e0affd640 1 -- 172.21.15.28:0/2229548656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e200b4070 con 0x55768d17ae80 2024-04-21T08:59:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e0affd640 1 -- 172.21.15.28:0/2229548656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7f1e200043c0 con 0x55768d17ae80 2024-04-21T08:59:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e34604740 1 -- 172.21.15.28:0/2229548656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e1c02cc90 con 0x55768d17ae80 2024-04-21T08:59:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e0affd640 1 -- 172.21.15.28:0/2229548656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1e200a0850 con 0x55768d17ae80 2024-04-21T08:59:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e34604740 10 client.17727 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e34604740 1 -- 172.21.15.28:0/2229548656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55768d1fa560 con 0x55768d17ae80 2024-04-21T08:59:12.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e0affd640 1 -- 172.21.15.28:0/2229548656 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e1c02cc90 con 0x55768d17ae80 2024-04-21T08:59:12.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e0affd640 1 -- 172.21.15.28:0/2229548656 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f1e200bb260 con 0x55768d17ae80 2024-04-21T08:59:12.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e34604740 10 client.17727 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:12.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e34604740 10 client.17727 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:59:12.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e34604740 20 client.17727 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:12.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e34604740 1 --2- 172.21.15.28:0/2229548656 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55768d1facd0 0x55768d1fd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:12.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e34604740 4 client.17727 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:59:12.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.925+0000 7f1e34604740 1 -- 172.21.15.28:0/2229548656 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55768d0ecc10 con 0x55768d1facd0 2024-04-21T08:59:12.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e2affd640 1 --2- 172.21.15.28:0/2229548656 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55768d1facd0 0x55768d1fd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:12.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e0affd640 10 client.17727 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:59:12.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e2affd640 1 --2- 172.21.15.28:0/2229548656 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55768d1facd0 0x55768d1fd0f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7b300bbd31e723f server_cookie=e6340f1faecd3bdf in_seq=0 out_seq=0 2024-04-21T08:59:12.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e0affd640 1 -- 172.21.15.28:0/2229548656 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55768d0ecc10 con 0x55768d1facd0 2024-04-21T08:59:12.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e0affd640 10 client.17727 handle_command_reply: tid=0 2024-04-21T08:59:12.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e34604740 1 client.17727 shutdown 2024-04-21T08:59:12.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e34604740 1 -- 172.21.15.28:0/2229548656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17ae80 msgr2=0x55768d17d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e34604740 1 --2- 172.21.15.28:0/2229548656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17ae80 0x55768d17d9e0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f1e2009bd90 tx=0x7f1e20098520 comp rx=0 tx=0).stop 2024-04-21T08:59:12.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e34604740 1 -- 172.21.15.28:0/2229548656 shutdown_connections 2024-04-21T08:59:12.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e34604740 1 --2- 172.21.15.28:0/2229548656 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x55768d1facd0 0x55768d1fd0f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e34604740 1 --2- 172.21.15.28:0/2229548656 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55768d183a00 0x55768d17df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e34604740 1 --2- 172.21.15.28:0/2229548656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55768d17ae80 0x55768d17d9e0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e0affd640 0 client.17727 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:59:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e34604740 1 --2- 172.21.15.28:0/2229548656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55768d17a4d0 0x55768d17d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e34604740 1 -- 172.21.15.28:0/2229548656 >> 172.21.15.28:0/2229548656 conn(0x55768d174ba0 msgr2=0x55768d175f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e34604740 1 -- 172.21.15.28:0/2229548656 shutdown_connections 2024-04-21T08:59:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e34604740 1 -- 172.21.15.28:0/2229548656 wait complete. 2024-04-21T08:59:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.926+0000 7f1e34604740 20 client.17727 trim_cache size 0 max 16384 2024-04-21T08:59:12.939 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:59:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.927+0000 7f1e31932640 1 -- 172.21.15.28:0/1614551382 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1e140a3d20 msgr2=0x7f1e140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.927+0000 7f1e31932640 1 --2- 172.21.15.28:0/1614551382 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1e140a3d20 0x7f1e140a61e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1e24020000 tx=0x7f1e2401d000 comp rx=0 tx=0).stop 2024-04-21T08:59:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.927+0000 7f1e31932640 1 -- 172.21.15.28:0/1614551382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c107270 msgr2=0x7f1e2c198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.927+0000 7f1e31932640 1 --2- 172.21.15.28:0/1614551382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c107270 0x7f1e2c198c70 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f1e20005fd0 tx=0x7f1e2000dde0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.927+0000 7f1e31932640 1 -- 172.21.15.28:0/1614551382 shutdown_connections 2024-04-21T08:59:12.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.927+0000 7f1e31932640 1 --2- 172.21.15.28:0/1614551382 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1e140a3d20 0x7f1e140a61e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.927+0000 7f1e31932640 1 --2- 172.21.15.28:0/1614551382 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1e2c19cff0 0x7f1e2c19d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.927+0000 7f1e31932640 1 --2- 172.21.15.28:0/1614551382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e2c107270 0x7f1e2c198c70 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.927+0000 7f1e31932640 1 --2- 172.21.15.28:0/1614551382 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1e2c06a840 0x7f1e2c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:12.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.927+0000 7f1e31932640 1 -- 172.21.15.28:0/1614551382 >> 172.21.15.28:0/1614551382 conn(0x7f1e2c077eb0 msgr2=0x7f1e2c108440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:12.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.927+0000 7f1e31932640 1 -- 172.21.15.28:0/1614551382 shutdown_connections 2024-04-21T08:59:12.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:12.927+0000 7f1e31932640 1 -- 172.21.15.28:0/1614551382 wait complete. 2024-04-21T08:59:12.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:59:13.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.047+0000 7f1f9c881640 1 -- 172.21.15.28:0/961291668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f98102190 msgr2=0x7f1f98102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:13.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.047+0000 7f1f9c881640 1 --2- 172.21.15.28:0/961291668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f98102190 0x7f1f98102570 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f1f88009900 tx=0x7f1f8802bc30 comp rx=0 tx=0).stop 2024-04-21T08:59:13.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.047+0000 7f1f9c881640 1 -- 172.21.15.28:0/961291668 shutdown_connections 2024-04-21T08:59:13.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.047+0000 7f1f9c881640 1 --2- 172.21.15.28:0/961291668 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f98107260 0x7f1f9810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.047+0000 7f1f9c881640 1 --2- 172.21.15.28:0/961291668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f98102b40 0x7f1f98106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.047+0000 7f1f9c881640 1 --2- 172.21.15.28:0/961291668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f98102190 0x7f1f98102570 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.047+0000 7f1f9c881640 1 -- 172.21.15.28:0/961291668 >> 172.21.15.28:0/961291668 conn(0x7f1f980fd990 msgr2=0x7f1f980ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:13.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.047+0000 7f1f9c881640 1 -- 172.21.15.28:0/961291668 shutdown_connections 2024-04-21T08:59:13.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.047+0000 7f1f9c881640 1 -- 172.21.15.28:0/961291668 wait complete. 2024-04-21T08:59:13.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f9c881640 1 Processor -- start 2024-04-21T08:59:13.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f9c881640 1 -- start start 2024-04-21T08:59:13.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f9c881640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f98102190 0x7f1f98074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f9c881640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f98107260 0x7f1f98074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f9c881640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f98074cd0 0x7f1f981a6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f9c881640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f981121d0 con 0x7f1f98074cd0 2024-04-21T08:59:13.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f9c881640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1f98112050 con 0x7f1f98102190 2024-04-21T08:59:13.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f9c881640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1f98112350 con 0x7f1f98107260 2024-04-21T08:59:13.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f96d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f98074cd0 0x7f1f981a6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f96d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f98074cd0 0x7f1f981a6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55692/0 (socket says 172.21.15.28:55692) 2024-04-21T08:59:13.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f96d9d640 1 -- 172.21.15.28:0/3698166595 learned_addr learned my addr 172.21.15.28:0/3698166595 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:13.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f9659c640 1 --2- 172.21.15.28:0/3698166595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f98102190 0x7f1f98074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f95d9b640 1 --2- 172.21.15.28:0/3698166595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f98107260 0x7f1f98074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f9659c640 1 -- 172.21.15.28:0/3698166595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f98107260 msgr2=0x7f1f98074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:13.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f9659c640 1 --2- 172.21.15.28:0/3698166595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f98107260 0x7f1f98074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f9659c640 1 -- 172.21.15.28:0/3698166595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f98074cd0 msgr2=0x7f1f981a6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:13.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f9659c640 1 --2- 172.21.15.28:0/3698166595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f98074cd0 0x7f1f981a6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f9659c640 1 -- 172.21.15.28:0/3698166595 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f981a7020 con 0x7f1f98102190 2024-04-21T08:59:13.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.048+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/3698166595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f98074cd0 0x7f1f981a6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:59:13.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.049+0000 7f1f9659c640 1 --2- 172.21.15.28:0/3698166595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f98102190 0x7f1f98074250 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1f88036040 tx=0x7f1f880045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:13.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.049+0000 7f1f877fe640 1 -- 172.21.15.28:0/3698166595 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f88043020 con 0x7f1f98102190 2024-04-21T08:59:13.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.049+0000 7f1f877fe640 1 -- 172.21.15.28:0/3698166595 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f8802cb70 con 0x7f1f98102190 2024-04-21T08:59:13.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.049+0000 7f1f9c881640 1 -- 172.21.15.28:0/3698166595 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f981a72b0 con 0x7f1f98102190 2024-04-21T08:59:13.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.049+0000 7f1f877fe640 1 -- 172.21.15.28:0/3698166595 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f88003b50 con 0x7f1f98102190 2024-04-21T08:59:13.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.049+0000 7f1f9c881640 1 -- 172.21.15.28:0/3698166595 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f981a7800 con 0x7f1f98102190 2024-04-21T08:59:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.052+0000 7f1f9f471740 1 Processor -- start 2024-04-21T08:59:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.052+0000 7f1f9f471740 1 -- start start 2024-04-21T08:59:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.052+0000 7f1f877fe640 1 -- 172.21.15.28:0/3698166595 <== mon.2 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1f88046020 con 0x7f1f98102190 2024-04-21T08:59:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.052+0000 7f1f9f471740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 0x5608ea2a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.052+0000 7f1f9f471740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2ad430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.052+0000 7f1f9f471740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2ada00 0x5608ea2afe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.052+0000 7f1f9f471740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5608ea2091f0 con 0x5608ea2a4e80 2024-04-21T08:59:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.052+0000 7f1f9f471740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5608ea1a64b0 con 0x5608ea2a44d0 2024-04-21T08:59:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.052+0000 7f1f9f471740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5608ea2051b0 con 0x5608ea2ada00 2024-04-21T08:59:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.052+0000 7f1f96d9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 0x5608ea2a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.052+0000 7f1f9659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.052+0000 7f1f96d9d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 0x5608ea2a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.28:39842/0 (socket says 172.21.15.28:39842) 2024-04-21T08:59:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.052+0000 7f1f96d9d640 1 -- 172.21.15.28:0/2966153876 learned_addr learned my addr 172.21.15.28:0/2966153876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.053+0000 7f1f95d9b640 1 --2- 172.21.15.28:0/2966153876 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2ada00 0x5608ea2afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.053+0000 7f1f96d9d640 1 -- 172.21.15.28:0/2966153876 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2ada00 msgr2=0x5608ea2afe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.053+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/2966153876 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2ada00 0x5608ea2afe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.053+0000 7f1f96d9d640 1 -- 172.21.15.28:0/2966153876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 msgr2=0x5608ea2ad430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.053+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/2966153876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2ad430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.053+0000 7f1f96d9d640 1 -- 172.21.15.28:0/2966153876 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608ea1981b0 con 0x5608ea2a44d0 2024-04-21T08:59:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.053+0000 7f1f877fe640 1 --2- 172.21.15.28:0/3698166595 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1f800a3d20 0x7f1f800a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.053+0000 7f1f9659c640 1 --2- 172.21.15.28:0/3698166595 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1f800a3d20 0x7f1f800a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.053+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/2966153876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 0x5608ea2a48b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1f90011780 tx=0x7f1f90011c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=399e007a01eb4faa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:13.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.053+0000 7f1f877fe640 1 -- 172.21.15.28:0/3698166595 <== mon.2 v2:172.21.15.152:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7f1f880f2c20 con 0x7f1f98102190 2024-04-21T08:59:13.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.053+0000 7f1f84ff9640 1 -- 172.21.15.28:0/2966153876 <== mon.2 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9000fb40 con 0x5608ea2a44d0 2024-04-21T08:59:13.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.053+0000 7f1f84ff9640 1 -- 172.21.15.28:0/2966153876 <== mon.2 v2:172.21.15.152:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f90011e90 con 0x5608ea2a44d0 2024-04-21T08:59:13.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.053+0000 7f1f84ff9640 1 -- 172.21.15.28:0/2966153876 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f90012e60 con 0x5608ea2a44d0 2024-04-21T08:59:13.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.054+0000 7f1f9f471740 1 -- 172.21.15.28:0/2966153876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 msgr2=0x5608ea2a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:13.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.054+0000 7f1f9f471740 1 --2- 172.21.15.28:0/2966153876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 0x5608ea2a48b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1f90011780 tx=0x7f1f90011c40 comp rx=0 tx=0).stop 2024-04-21T08:59:13.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.054+0000 7f1f9659c640 1 --2- 172.21.15.28:0/3698166595 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1f800a3d20 0x7f1f800a61e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1f880f00e0 tx=0x7f1f880f6770 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:13.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.054+0000 7f1f9f471740 1 -- 172.21.15.28:0/2966153876 shutdown_connections 2024-04-21T08:59:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.054+0000 7f1f9f471740 1 --2- 172.21.15.28:0/2966153876 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2ada00 0x5608ea2afe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.054+0000 7f1f9f471740 1 --2- 172.21.15.28:0/2966153876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2ad430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.054+0000 7f1f9f471740 1 --2- 172.21.15.28:0/2966153876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 0x5608ea2a48b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.054+0000 7f1f9f471740 1 -- 172.21.15.28:0/2966153876 >> 172.21.15.28:0/2966153876 conn(0x5608ea29eba0 msgr2=0x5608ea29ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.054+0000 7f1f9f471740 1 -- 172.21.15.28:0/2966153876 shutdown_connections 2024-04-21T08:59:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.054+0000 7f1f9f471740 1 -- 172.21.15.28:0/2966153876 wait complete. 2024-04-21T08:59:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.054+0000 7f1f9f471740 1 -- start start 2024-04-21T08:59:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.054+0000 7f1f9f471740 1 Processor -- start 2024-04-21T08:59:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f9f471740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2a44d0 0x5608ea2ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f9f471740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2aa630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f9f471740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2ada00 0x5608ea2aab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f9f471740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5608ea2b0970 con 0x5608ea2a4e80 2024-04-21T08:59:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f9f471740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5608ea1c2500 con 0x5608ea2ada00 2024-04-21T08:59:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f9f471740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5608ea2b0af0 con 0x5608ea2a44d0 2024-04-21T08:59:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f9f471740 10 client.0 _ll_register_callbacks cb 0x5608ea190b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f95d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2aa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f95d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2aa630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55706/0 (socket says 172.21.15.28:55706) 2024-04-21T08:59:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f95d9b640 1 -- 172.21.15.28:0/1907417899 learned_addr learned my addr 172.21.15.28:0/1907417899 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/1907417899 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2a44d0 0x5608ea2ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f95d9b640 1 -- 172.21.15.28:0/1907417899 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2a44d0 msgr2=0x5608ea2ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/1907417899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2ada00 0x5608ea2aab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f95d9b640 1 --2- 172.21.15.28:0/1907417899 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2a44d0 0x5608ea2ad2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f95d9b640 1 -- 172.21.15.28:0/1907417899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2ada00 msgr2=0x5608ea2aab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f95d9b640 1 --2- 172.21.15.28:0/1907417899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2ada00 0x5608ea2aab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f95d9b640 1 -- 172.21.15.28:0/1907417899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608e9f1b3f0 con 0x5608ea2a4e80 2024-04-21T08:59:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/1907417899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2ada00 0x5608ea2aab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f95d9b640 1 --2- 172.21.15.28:0/1907417899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2aa630 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c013710 tx=0x7f1f8c013bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de073febe3ece13 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f7dffb640 1 -- 172.21.15.28:0/1907417899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c011dd0 con 0x5608ea2a4e80 2024-04-21T08:59:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f7dffb640 1 -- 172.21.15.28:0/1907417899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7f1f8c004030 con 0x5608ea2a4e80 2024-04-21T08:59:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.055+0000 7f1f9f471740 1 -- 172.21.15.28:0/1907417899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f90011e90 con 0x5608ea2a4e80 2024-04-21T08:59:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.056+0000 7f1f7dffb640 1 -- 172.21.15.28:0/1907417899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1f8c004270 con 0x5608ea2a4e80 2024-04-21T08:59:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.056+0000 7f1f9f471740 10 client.17739 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.056+0000 7f1f9f471740 1 -- 172.21.15.28:0/1907417899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5608e9fe57d0 con 0x5608ea2a4e80 2024-04-21T08:59:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.056+0000 7f1f7dffb640 1 -- 172.21.15.28:0/1907417899 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f90011e90 con 0x5608ea2a4e80 2024-04-21T08:59:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.056+0000 7f1f7dffb640 1 -- 172.21.15.28:0/1907417899 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f1f8c01a070 con 0x5608ea2a4e80 2024-04-21T08:59:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.056+0000 7f1f9f471740 10 client.17739 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.056+0000 7f1f9f471740 10 client.17739 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.056+0000 7f1f9f471740 20 client.17739 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.056+0000 7f1f9f471740 1 --2- 172.21.15.28:0/1907417899 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5608ea2bdc60 0x5608ea2c00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.056+0000 7f1f9f471740 4 client.17739 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:59:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.056+0000 7f1f9f471740 1 -- 172.21.15.28:0/1907417899 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608ea1a8930 con 0x5608ea2bdc60 2024-04-21T08:59:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.056+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/1907417899 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5608ea2bdc60 0x5608ea2c00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.057+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/1907417899 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5608ea2bdc60 0x5608ea2c00a0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5358c4138400043a server_cookie=d2a6fb7a56f7c97a in_seq=0 out_seq=0 2024-04-21T08:59:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.057+0000 7f1f7dffb640 10 client.17739 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.057+0000 7f1f7dffb640 1 -- 172.21.15.28:0/1907417899 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5608ea1a8930 con 0x5608ea2bdc60 2024-04-21T08:59:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.057+0000 7f1f7dffb640 10 client.17739 handle_command_reply: tid=0 2024-04-21T08:59:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.057+0000 7f1f9f471740 1 client.17739 shutdown 2024-04-21T08:59:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.058+0000 7f1f9f471740 1 -- 172.21.15.28:0/1907417899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 msgr2=0x5608ea2aa630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.058+0000 7f1f9f471740 1 --2- 172.21.15.28:0/1907417899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2aa630 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c013710 tx=0x7f1f8c013bd0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.058+0000 7f1f9f471740 1 -- 172.21.15.28:0/1907417899 shutdown_connections 2024-04-21T08:59:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.058+0000 7f1f9f471740 1 --2- 172.21.15.28:0/1907417899 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5608ea2bdc60 0x5608ea2c00a0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.058+0000 7f1f9f471740 1 --2- 172.21.15.28:0/1907417899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2ada00 0x5608ea2aab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.058+0000 7f1f9f471740 1 --2- 172.21.15.28:0/1907417899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2aa630 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.058+0000 7f1f7dffb640 0 client.17739 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.058+0000 7f1f9f471740 1 --2- 172.21.15.28:0/1907417899 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2a44d0 0x5608ea2ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.058+0000 7f1f9f471740 1 -- 172.21.15.28:0/1907417899 >> 172.21.15.28:0/1907417899 conn(0x5608ea29eba0 msgr2=0x5608ea2a08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.058+0000 7f1f9f471740 1 -- 172.21.15.28:0/1907417899 shutdown_connections 2024-04-21T08:59:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.058+0000 7f1f9f471740 1 -- 172.21.15.28:0/1907417899 wait complete. 2024-04-21T08:59:13.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.058+0000 7f1f9f471740 20 client.17739 trim_cache size 0 max 16384 2024-04-21T08:59:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.071+0000 7f1f9f471740 1 Processor -- start 2024-04-21T08:59:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.071+0000 7f1f9f471740 1 -- start start 2024-04-21T08:59:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.071+0000 7f1f9f471740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 0x5608ea2acd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.071+0000 7f1f9f471740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2a4e80 0x5608ea2ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.071+0000 7f1f9f471740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2ada00 0x5608ea2c5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.071+0000 7f1f9f471740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f981b4010 con 0x5608ea2ada00 2024-04-21T08:59:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.071+0000 7f1f9f471740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1f981b18e0 con 0x5608ea2a44d0 2024-04-21T08:59:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.071+0000 7f1f9f471740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1f981af750 con 0x5608ea2a4e80 2024-04-21T08:59:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.072+0000 7f1f96d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2ada00 0x5608ea2c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.072+0000 7f1f96d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2ada00 0x5608ea2c5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55712/0 (socket says 172.21.15.28:55712) 2024-04-21T08:59:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.072+0000 7f1f96d9d640 1 -- 172.21.15.28:0/1193987685 learned_addr learned my addr 172.21.15.28:0/1193987685 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.072+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/1193987685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 0x5608ea2acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.072+0000 7f1f95d9b640 1 --2- 172.21.15.28:0/1193987685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2a4e80 0x5608ea2ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.072+0000 7f1f96d9d640 1 -- 172.21.15.28:0/1193987685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2a4e80 msgr2=0x5608ea2ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.072+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/1193987685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2a4e80 0x5608ea2ad2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.072+0000 7f1f96d9d640 1 -- 172.21.15.28:0/1193987685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 msgr2=0x5608ea2acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.072+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/1193987685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 0x5608ea2acd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.072+0000 7f1f96d9d640 1 -- 172.21.15.28:0/1193987685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608ea2a4c20 con 0x5608ea2ada00 2024-04-21T08:59:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.072+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/1193987685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2ada00 0x5608ea2c5970 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f1f90005c60 tx=0x7f1f90011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee41c12284da4200 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.072+0000 7f1f7ffff640 1 -- 172.21.15.28:0/1193987685 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f90012e60 con 0x5608ea2ada00 2024-04-21T08:59:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.072+0000 7f1f7ffff640 1 -- 172.21.15.28:0/1193987685 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f90012450 con 0x5608ea2ada00 2024-04-21T08:59:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.072+0000 7f1f7ffff640 1 -- 172.21.15.28:0/1193987685 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9002e390 con 0x5608ea2ada00 2024-04-21T08:59:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 -- 172.21.15.28:0/1193987685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2ada00 msgr2=0x5608ea2c5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 --2- 172.21.15.28:0/1193987685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2ada00 0x5608ea2c5970 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f1f90005c60 tx=0x7f1f90011c40 comp rx=0 tx=0).stop 2024-04-21T08:59:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 -- 172.21.15.28:0/1193987685 shutdown_connections 2024-04-21T08:59:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 --2- 172.21.15.28:0/1193987685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2ada00 0x5608ea2c5970 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 --2- 172.21.15.28:0/1193987685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2a4e80 0x5608ea2ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 --2- 172.21.15.28:0/1193987685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 0x5608ea2acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 -- 172.21.15.28:0/1193987685 >> 172.21.15.28:0/1193987685 conn(0x5608ea29eba0 msgr2=0x5608ea29d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 -- 172.21.15.28:0/1193987685 shutdown_connections 2024-04-21T08:59:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 -- 172.21.15.28:0/1193987685 wait complete. 2024-04-21T08:59:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 -- start start 2024-04-21T08:59:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 Processor -- start 2024-04-21T08:59:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 0x5608ea2b5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2b6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2ada00 0x5608ea2b2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f981afda0 con 0x5608ea2a4e80 2024-04-21T08:59:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f1f981b1210 con 0x5608ea2a44d0 2024-04-21T08:59:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f1f981b0470 con 0x5608ea2ada00 2024-04-21T08:59:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f9f471740 10 client.0 _ll_register_callbacks cb 0x7f1f8c009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.073+0000 7f1f95d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f95d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2b6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55728/0 (socket says 172.21.15.28:55728) 2024-04-21T08:59:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f95d9b640 1 -- 172.21.15.28:0/2334845178 learned_addr learned my addr 172.21.15.28:0/2334845178 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:13.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/2334845178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 0x5608ea2b5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/2334845178 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2ada00 0x5608ea2b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f95d9b640 1 -- 172.21.15.28:0/2334845178 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2ada00 msgr2=0x5608ea2b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:13.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f95d9b640 1 --2- 172.21.15.28:0/2334845178 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2ada00 0x5608ea2b2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f95d9b640 1 -- 172.21.15.28:0/2334845178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 msgr2=0x5608ea2b5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:13.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f95d9b640 1 --2- 172.21.15.28:0/2334845178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 0x5608ea2b5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f95d9b640 1 -- 172.21.15.28:0/2334845178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608ea2b2a70 con 0x5608ea2a4e80 2024-04-21T08:59:13.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f95d9b640 1 --2- 172.21.15.28:0/2334845178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2b6280 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009e10 tx=0x7f1f8c00f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2492b250a3a76595 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f7dffb640 1 -- 172.21.15.28:0/2334845178 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c02f060 con 0x5608ea2a4e80 2024-04-21T08:59:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f7dffb640 1 -- 172.21.15.28:0/2334845178 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7f1f8c005450 con 0x5608ea2a4e80 2024-04-21T08:59:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f9f471740 1 -- 172.21.15.28:0/2334845178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f90012450 con 0x5608ea2a4e80 2024-04-21T08:59:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f7dffb640 1 -- 172.21.15.28:0/2334845178 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1f8c036050 con 0x5608ea2a4e80 2024-04-21T08:59:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f9f471740 10 client.17745 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.074+0000 7f1f9f471740 1 -- 172.21.15.28:0/2334845178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5608ea324e30 con 0x5608ea2a4e80 2024-04-21T08:59:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.075+0000 7f1f7dffb640 1 -- 172.21.15.28:0/2334845178 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f90012450 con 0x5608ea2a4e80 2024-04-21T08:59:13.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.075+0000 7f1f7dffb640 1 -- 172.21.15.28:0/2334845178 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f1f8c01a070 con 0x5608ea2a4e80 2024-04-21T08:59:13.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.075+0000 7f1f9f471740 10 client.17745 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:13.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.075+0000 7f1f9f471740 10 client.17745 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:13.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.075+0000 7f1f9f471740 20 client.17745 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:13.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.075+0000 7f1f9f471740 1 --2- 172.21.15.28:0/2334845178 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5608ea325550 0x5608ea327970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:13.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.075+0000 7f1f9f471740 4 client.17745 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:59:13.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.075+0000 7f1f9f471740 1 -- 172.21.15.28:0/2334845178 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5608ea216c10 con 0x5608ea325550 2024-04-21T08:59:13.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.075+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/2334845178 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5608ea325550 0x5608ea327970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:13.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f96d9d640 1 --2- 172.21.15.28:0/2334845178 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5608ea325550 0x5608ea327970 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cf61bce8f60bf43 server_cookie=ccb65f847f9b076 in_seq=0 out_seq=0 2024-04-21T08:59:13.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f7dffb640 10 client.17745 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:13.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f7dffb640 1 -- 172.21.15.28:0/2334845178 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5608ea216c10 con 0x5608ea325550 2024-04-21T08:59:13.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f7dffb640 10 client.17745 handle_command_reply: tid=0 2024-04-21T08:59:13.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f9f471740 1 client.17745 shutdown 2024-04-21T08:59:13.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f9f471740 1 -- 172.21.15.28:0/2334845178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 msgr2=0x5608ea2b6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:13.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f9f471740 1 --2- 172.21.15.28:0/2334845178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2b6280 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009e10 tx=0x7f1f8c00f540 comp rx=0 tx=0).stop 2024-04-21T08:59:13.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f9f471740 1 -- 172.21.15.28:0/2334845178 shutdown_connections 2024-04-21T08:59:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f9f471740 1 --2- 172.21.15.28:0/2334845178 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5608ea325550 0x5608ea327970 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f9f471740 1 --2- 172.21.15.28:0/2334845178 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5608ea2ada00 0x5608ea2b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f9f471740 1 --2- 172.21.15.28:0/2334845178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608ea2a4e80 0x5608ea2b6280 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f9f471740 1 --2- 172.21.15.28:0/2334845178 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5608ea2a44d0 0x5608ea2b5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f7dffb640 0 client.17745 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f9f471740 1 -- 172.21.15.28:0/2334845178 >> 172.21.15.28:0/2334845178 conn(0x5608ea29eba0 msgr2=0x5608ea2a0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f9f471740 1 -- 172.21.15.28:0/2334845178 shutdown_connections 2024-04-21T08:59:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f9f471740 1 -- 172.21.15.28:0/2334845178 wait complete. 2024-04-21T08:59:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.076+0000 7f1f9f471740 20 client.17745 trim_cache size 0 max 16384 2024-04-21T08:59:13.088 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:59:13.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.077+0000 7f1f9c881640 1 -- 172.21.15.28:0/3698166595 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1f800a3d20 msgr2=0x7f1f800a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:13.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.077+0000 7f1f9c881640 1 --2- 172.21.15.28:0/3698166595 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1f800a3d20 0x7f1f800a61e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1f880f00e0 tx=0x7f1f880f6770 comp rx=0 tx=0).stop 2024-04-21T08:59:13.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.077+0000 7f1f9c881640 1 -- 172.21.15.28:0/3698166595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f98102190 msgr2=0x7f1f98074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.077+0000 7f1f9c881640 1 --2- 172.21.15.28:0/3698166595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f98102190 0x7f1f98074250 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1f88036040 tx=0x7f1f880045e0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.077+0000 7f1f9c881640 1 -- 172.21.15.28:0/3698166595 shutdown_connections 2024-04-21T08:59:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.077+0000 7f1f9c881640 1 --2- 172.21.15.28:0/3698166595 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f1f800a3d20 0x7f1f800a61e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.077+0000 7f1f9c881640 1 --2- 172.21.15.28:0/3698166595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f98074cd0 0x7f1f981a6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.077+0000 7f1f9c881640 1 --2- 172.21.15.28:0/3698166595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f98107260 0x7f1f98074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.077+0000 7f1f9c881640 1 --2- 172.21.15.28:0/3698166595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f98102190 0x7f1f98074250 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.077+0000 7f1f9c881640 1 -- 172.21.15.28:0/3698166595 >> 172.21.15.28:0/3698166595 conn(0x7f1f980fd990 msgr2=0x7f1f980fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.077+0000 7f1f9c881640 1 -- 172.21.15.28:0/3698166595 shutdown_connections 2024-04-21T08:59:13.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:13.077+0000 7f1f9c881640 1 -- 172.21.15.28:0/3698166595 wait complete. 2024-04-21T08:59:13.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:13 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3330681807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:59:13.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:13 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3330681807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:59:13.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:13 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3330681807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:59:14.091 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:59:14.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:59:14.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:59:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.212+0000 7f42ddc50640 1 -- 172.21.15.28:0/714482189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d8107260 msgr2=0x7f42d810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.212+0000 7f42ddc50640 1 --2- 172.21.15.28:0/714482189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d8107260 0x7f42d810ee30 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f42d0009ca0 tx=0x7f42d002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.212+0000 7f42ddc50640 1 -- 172.21.15.28:0/714482189 shutdown_connections 2024-04-21T08:59:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.212+0000 7f42ddc50640 1 --2- 172.21.15.28:0/714482189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d8107260 0x7f42d810ee30 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.212+0000 7f42ddc50640 1 --2- 172.21.15.28:0/714482189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42d8102b40 0x7f42d8106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.212+0000 7f42ddc50640 1 --2- 172.21.15.28:0/714482189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42d8102190 0x7f42d8102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.212+0000 7f42ddc50640 1 -- 172.21.15.28:0/714482189 >> 172.21.15.28:0/714482189 conn(0x7f42d80fd990 msgr2=0x7f42d80ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.212+0000 7f42ddc50640 1 -- 172.21.15.28:0/714482189 shutdown_connections 2024-04-21T08:59:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.212+0000 7f42ddc50640 1 -- 172.21.15.28:0/714482189 wait complete. 2024-04-21T08:59:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.212+0000 7f42ddc50640 1 Processor -- start 2024-04-21T08:59:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42ddc50640 1 -- start start 2024-04-21T08:59:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42ddc50640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42d8102190 0x7f42d8074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42ddc50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d8107260 0x7f42d8074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42ddc50640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42d8074cd0 0x7f42d81a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42ddc50640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42d81121b0 con 0x7f42d8107260 2024-04-21T08:59:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42ddc50640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f42d8112030 con 0x7f42d8074cd0 2024-04-21T08:59:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42ddc50640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f42d8112330 con 0x7f42d8102190 2024-04-21T08:59:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d8107260 0x7f42d8074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d8107260 0x7f42d8074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53704/0 (socket says 172.21.15.28:53704) 2024-04-21T08:59:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42d6ffd640 1 -- 172.21.15.28:0/714511556 learned_addr learned my addr 172.21.15.28:0/714511556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42d77fe640 1 --2- 172.21.15.28:0/714511556 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42d8102190 0x7f42d8074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42d7fff640 1 --2- 172.21.15.28:0/714511556 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42d8074cd0 0x7f42d81a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42d6ffd640 1 -- 172.21.15.28:0/714511556 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42d8102190 msgr2=0x7f42d8074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/714511556 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42d8102190 0x7f42d8074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42d6ffd640 1 -- 172.21.15.28:0/714511556 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42d8074cd0 msgr2=0x7f42d81a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/714511556 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42d8074cd0 0x7f42d81a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42d6ffd640 1 -- 172.21.15.28:0/714511556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d81a6f80 con 0x7f42d8107260 2024-04-21T08:59:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42d7fff640 1 --2- 172.21.15.28:0/714511556 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42d8074cd0 0x7f42d81a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.213+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/714511556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d8107260 0x7f42d8074790 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f42cc00c970 tx=0x7f42cc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.214+0000 7f42d4ff9640 1 -- 172.21.15.28:0/714511556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc014070 con 0x7f42d8107260 2024-04-21T08:59:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.214+0000 7f42d4ff9640 1 -- 172.21.15.28:0/714511556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42cc004440 con 0x7f42d8107260 2024-04-21T08:59:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.214+0000 7f42ddc50640 1 -- 172.21.15.28:0/714511556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d81a7270 con 0x7f42d8107260 2024-04-21T08:59:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.214+0000 7f42d4ff9640 1 -- 172.21.15.28:0/714511556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc003e50 con 0x7f42d8107260 2024-04-21T08:59:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.214+0000 7f42ddc50640 1 -- 172.21.15.28:0/714511556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d81a77c0 con 0x7f42d8107260 2024-04-21T08:59:14.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.216+0000 7f42d4ff9640 1 -- 172.21.15.28:0/714511556 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f42cc017070 con 0x7f42d8107260 2024-04-21T08:59:14.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.216+0000 7f42e083d740 1 Processor -- start 2024-04-21T08:59:14.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.216+0000 7f42e083d740 1 -- start start 2024-04-21T08:59:14.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.216+0000 7f42e083d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 0x556cf42dbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.216+0000 7f42e083d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42dc290 0x556cf42e4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.216+0000 7f42e083d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42e4e10 0x556cf42e7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42e083d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556cf423ff00 con 0x556cf42e4e10 2024-04-21T08:59:14.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42e083d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556cf41dd590 con 0x556cf42dc290 2024-04-21T08:59:14.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42e083d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x556cf423bec0 con 0x556cf42db8e0 2024-04-21T08:59:14.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42d4ff9640 1 --2- 172.21.15.28:0/714511556 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f42c00a3ef0 0x7f42c00a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42e4e10 0x556cf42e7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42e4e10 0x556cf42e7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53718/0 (socket says 172.21.15.28:53718) 2024-04-21T08:59:14.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42d6ffd640 1 -- 172.21.15.28:0/1481900698 learned_addr learned my addr 172.21.15.28:0/1481900698 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:14.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42d7fff640 1 --2- 172.21.15.28:0/714511556 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f42c00a3ef0 0x7f42c00a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42d77fe640 1 --2- 172.21.15.28:0/1481900698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42dc290 0x556cf42e4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42d7fff640 1 --2- 172.21.15.28:0/1481900698 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 0x556cf42dbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42d4ff9640 1 -- 172.21.15.28:0/714511556 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7f42cc0d0790 con 0x7f42d8107260 2024-04-21T08:59:14.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42d6ffd640 1 -- 172.21.15.28:0/1481900698 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 msgr2=0x556cf42dbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/1481900698 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 0x556cf42dbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42d6ffd640 1 -- 172.21.15.28:0/1481900698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42dc290 msgr2=0x556cf42e4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/1481900698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42dc290 0x556cf42e4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42d6ffd640 1 -- 172.21.15.28:0/1481900698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cf401a7e0 con 0x556cf42e4e10 2024-04-21T08:59:14.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42d7fff640 1 --2- 172.21.15.28:0/714511556 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f42c00a3ef0 0x7f42c00a63b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f42d0009a60 tx=0x7f42d00099f0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:14.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.217+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/1481900698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42e4e10 0x556cf42e7260 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0cd9a0 tx=0x7f42cc0038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8d36296f891859b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:14.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.218+0000 7f42bdffb640 1 -- 172.21.15.28:0/1481900698 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0a0630 con 0x556cf42e4e10 2024-04-21T08:59:14.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.218+0000 7f42bdffb640 1 -- 172.21.15.28:0/1481900698 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42cc0cf070 con 0x556cf42e4e10 2024-04-21T08:59:14.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.218+0000 7f42bdffb640 1 -- 172.21.15.28:0/1481900698 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0cdcb0 con 0x556cf42e4e10 2024-04-21T08:59:14.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.218+0000 7f42e083d740 1 -- 172.21.15.28:0/1481900698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42e4e10 msgr2=0x556cf42e7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.218+0000 7f42e083d740 1 --2- 172.21.15.28:0/1481900698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42e4e10 0x556cf42e7260 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0cd9a0 tx=0x7f42cc0038b0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.218+0000 7f42e083d740 1 -- 172.21.15.28:0/1481900698 shutdown_connections 2024-04-21T08:59:14.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.218+0000 7f42e083d740 1 --2- 172.21.15.28:0/1481900698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42e4e10 0x556cf42e7260 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.218+0000 7f42e083d740 1 --2- 172.21.15.28:0/1481900698 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42dc290 0x556cf42e4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.218+0000 7f42e083d740 1 --2- 172.21.15.28:0/1481900698 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 0x556cf42dbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.218+0000 7f42e083d740 1 -- 172.21.15.28:0/1481900698 >> 172.21.15.28:0/1481900698 conn(0x556cf42d5fb0 msgr2=0x556cf42d6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.218+0000 7f42e083d740 1 -- 172.21.15.28:0/1481900698 shutdown_connections 2024-04-21T08:59:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.218+0000 7f42e083d740 1 -- 172.21.15.28:0/1481900698 wait complete. 2024-04-21T08:59:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.219+0000 7f42e083d740 1 -- start start 2024-04-21T08:59:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.219+0000 7f42e083d740 1 Processor -- start 2024-04-21T08:59:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.219+0000 7f42e083d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 0x556cf42e1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.219+0000 7f42e083d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42dc290 0x556cf42e2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.219+0000 7f42e083d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 0x556cf42e2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.219+0000 7f42e083d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556cf42e7d80 con 0x556cf42dc290 2024-04-21T08:59:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.219+0000 7f42e083d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556cf41f9580 con 0x556cf42e4e10 2024-04-21T08:59:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.219+0000 7f42e083d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x556cf42e7f00 con 0x556cf42db8e0 2024-04-21T08:59:14.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.219+0000 7f42e083d740 10 client.0 _ll_register_callbacks cb 0x556cf41ff650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:14.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.219+0000 7f42d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42dc290 0x556cf42e2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.219+0000 7f42d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42dc290 0x556cf42e2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53724/0 (socket says 172.21.15.28:53724) 2024-04-21T08:59:14.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.219+0000 7f42d6ffd640 1 -- 172.21.15.28:0/228747093 learned_addr learned my addr 172.21.15.28:0/228747093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:14.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.219+0000 7f42d77fe640 1 --2- 172.21.15.28:0/228747093 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 0x556cf42e1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.220+0000 7f42d7fff640 1 --2- 172.21.15.28:0/228747093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 0x556cf42e2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.220+0000 7f42d6ffd640 1 -- 172.21.15.28:0/228747093 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 msgr2=0x556cf42e1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.220+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/228747093 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 0x556cf42e1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.220+0000 7f42d6ffd640 1 -- 172.21.15.28:0/228747093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 msgr2=0x556cf42e2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.220+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/228747093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 0x556cf42e2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.220+0000 7f42d6ffd640 1 -- 172.21.15.28:0/228747093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cf407af70 con 0x556cf42dc290 2024-04-21T08:59:14.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.220+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/228747093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42dc290 0x556cf42e2310 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f42cc004630 tx=0x7f42cc0a4f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c404b28d10fcb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:14.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.220+0000 7f42b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:14.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.220+0000 7f42b6ffd640 1 -- 172.21.15.28:0/228747093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0a0630 con 0x556cf42dc290 2024-04-21T08:59:14.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.220+0000 7f42b6ffd640 1 -- 172.21.15.28:0/228747093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7f42cc098960 con 0x556cf42dc290 2024-04-21T08:59:14.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.220+0000 7f42e083d740 1 -- 172.21.15.28:0/228747093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42cc0cf070 con 0x556cf42dc290 2024-04-21T08:59:14.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.220+0000 7f42b6ffd640 1 -- 172.21.15.28:0/228747093 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f42cc0a4a60 con 0x556cf42dc290 2024-04-21T08:59:14.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.220+0000 7f42e083d740 10 client.17757 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:14.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.220+0000 7f42e083d740 1 -- 172.21.15.28:0/228747093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556cf3f12170 con 0x556cf42dc290 2024-04-21T08:59:14.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.220+0000 7f42b6ffd640 1 -- 172.21.15.28:0/228747093 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42cc0cf070 con 0x556cf42dc290 2024-04-21T08:59:14.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.221+0000 7f42b6ffd640 1 -- 172.21.15.28:0/228747093 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f42cc0d0290 con 0x556cf42dc290 2024-04-21T08:59:14.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.221+0000 7f42e083d740 10 client.17757 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:14.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.221+0000 7f42e083d740 10 client.17757 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:14.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.221+0000 7f42e083d740 20 client.17757 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:14.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.221+0000 7f42e083d740 1 --2- 172.21.15.28:0/228747093 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556cf42f51f0 0x556cf42f75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.221+0000 7f42e083d740 4 client.17757 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:59:14.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.221+0000 7f42e083d740 1 -- 172.21.15.28:0/228747093 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cf424d920 con 0x556cf42f51f0 2024-04-21T08:59:14.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.221+0000 7f42d77fe640 1 --2- 172.21.15.28:0/228747093 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556cf42f51f0 0x556cf42f75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.221+0000 7f42b6ffd640 10 client.17757 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:14.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.221+0000 7f42d77fe640 1 --2- 172.21.15.28:0/228747093 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556cf42f51f0 0x556cf42f75e0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1226d0759be4b6bc server_cookie=df3ace2a9e830ee6 in_seq=0 out_seq=0 2024-04-21T08:59:14.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.222+0000 7f42b6ffd640 1 -- 172.21.15.28:0/228747093 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556cf424d920 con 0x556cf42f51f0 2024-04-21T08:59:14.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.222+0000 7f42b6ffd640 10 client.17757 handle_command_reply: tid=0 2024-04-21T08:59:14.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.222+0000 7f42e083d740 1 client.17757 shutdown 2024-04-21T08:59:14.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.222+0000 7f42e083d740 1 -- 172.21.15.28:0/228747093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42dc290 msgr2=0x556cf42e2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.222+0000 7f42e083d740 1 --2- 172.21.15.28:0/228747093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42dc290 0x556cf42e2310 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f42cc004630 tx=0x7f42cc0a4f60 comp rx=0 tx=0).stop 2024-04-21T08:59:14.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.222+0000 7f42e083d740 1 -- 172.21.15.28:0/228747093 shutdown_connections 2024-04-21T08:59:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.222+0000 7f42e083d740 1 --2- 172.21.15.28:0/228747093 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556cf42f51f0 0x556cf42f75e0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.222+0000 7f42e083d740 1 --2- 172.21.15.28:0/228747093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 0x556cf42e2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.222+0000 7f42b6ffd640 0 client.17757 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.222+0000 7f42e083d740 1 --2- 172.21.15.28:0/228747093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42dc290 0x556cf42e2310 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.222+0000 7f42e083d740 1 --2- 172.21.15.28:0/228747093 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 0x556cf42e1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.222+0000 7f42e083d740 1 -- 172.21.15.28:0/228747093 >> 172.21.15.28:0/228747093 conn(0x556cf42d5fb0 msgr2=0x556cf42d7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.223+0000 7f42e083d740 1 -- 172.21.15.28:0/228747093 shutdown_connections 2024-04-21T08:59:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.223+0000 7f42e083d740 1 -- 172.21.15.28:0/228747093 wait complete. 2024-04-21T08:59:14.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.223+0000 7f42e083d740 20 client.17757 trim_cache size 0 max 16384 2024-04-21T08:59:14.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.241+0000 7f42e083d740 1 Processor -- start 2024-04-21T08:59:14.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.241+0000 7f42e083d740 1 -- start start 2024-04-21T08:59:14.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.241+0000 7f42e083d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 0x556cf42e4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.241+0000 7f42e083d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42dc290 0x556cf42d7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.241+0000 7f42e083d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 0x556cf42fa600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.241+0000 7f42e083d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42d81b4080 con 0x556cf42dc290 2024-04-21T08:59:14.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.241+0000 7f42e083d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f42d81b1950 con 0x556cf42e4e10 2024-04-21T08:59:14.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.241+0000 7f42e083d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f42d81af7c0 con 0x556cf42db8e0 2024-04-21T08:59:14.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.241+0000 7f42d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42dc290 0x556cf42d7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.241+0000 7f42d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42dc290 0x556cf42d7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53736/0 (socket says 172.21.15.28:53736) 2024-04-21T08:59:14.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.241+0000 7f42d6ffd640 1 -- 172.21.15.28:0/2998384500 learned_addr learned my addr 172.21.15.28:0/2998384500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:14.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.242+0000 7f42d77fe640 1 --2- 172.21.15.28:0/2998384500 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 0x556cf42e4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.242+0000 7f42d7fff640 1 --2- 172.21.15.28:0/2998384500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 0x556cf42fa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.242+0000 7f42d6ffd640 1 -- 172.21.15.28:0/2998384500 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 msgr2=0x556cf42e4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.242+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/2998384500 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 0x556cf42e4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.242+0000 7f42d6ffd640 1 -- 172.21.15.28:0/2998384500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 msgr2=0x556cf42fa600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.242+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/2998384500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 0x556cf42fa600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.242+0000 7f42d6ffd640 1 -- 172.21.15.28:0/2998384500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cf42d67f0 con 0x556cf42dc290 2024-04-21T08:59:14.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.242+0000 7f42d7fff640 1 --2- 172.21.15.28:0/2998384500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 0x556cf42fa600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:14.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.242+0000 7f42d77fe640 1 --2- 172.21.15.28:0/2998384500 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 0x556cf42e4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:14.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.242+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/2998384500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42dc290 0x556cf42d7b10 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0ce030 tx=0x7f42cc09a790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f096553e791f9d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:14.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.242+0000 7f42bd7fa640 1 -- 172.21.15.28:0/2998384500 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0a0630 con 0x556cf42dc290 2024-04-21T08:59:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.242+0000 7f42bd7fa640 1 -- 172.21.15.28:0/2998384500 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42cc0cf070 con 0x556cf42dc290 2024-04-21T08:59:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.242+0000 7f42bd7fa640 1 -- 172.21.15.28:0/2998384500 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc09b980 con 0x556cf42dc290 2024-04-21T08:59:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.243+0000 7f42e083d740 1 -- 172.21.15.28:0/2998384500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42dc290 msgr2=0x556cf42d7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.243+0000 7f42e083d740 1 --2- 172.21.15.28:0/2998384500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42dc290 0x556cf42d7b10 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0ce030 tx=0x7f42cc09a790 comp rx=0 tx=0).stop 2024-04-21T08:59:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.243+0000 7f42e083d740 1 -- 172.21.15.28:0/2998384500 shutdown_connections 2024-04-21T08:59:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.243+0000 7f42e083d740 1 --2- 172.21.15.28:0/2998384500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 0x556cf42fa600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.243+0000 7f42e083d740 1 --2- 172.21.15.28:0/2998384500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42dc290 0x556cf42d7b10 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.243+0000 7f42e083d740 1 --2- 172.21.15.28:0/2998384500 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42db8e0 0x556cf42e4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.243+0000 7f42e083d740 1 -- 172.21.15.28:0/2998384500 >> 172.21.15.28:0/2998384500 conn(0x556cf42d5fb0 msgr2=0x556cf42d7400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:14.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.243+0000 7f42e083d740 1 -- 172.21.15.28:0/2998384500 shutdown_connections 2024-04-21T08:59:14.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.243+0000 7f42e083d740 1 -- 172.21.15.28:0/2998384500 wait complete. 2024-04-21T08:59:14.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.243+0000 7f42e083d740 1 -- start start 2024-04-21T08:59:14.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42e083d740 1 Processor -- start 2024-04-21T08:59:14.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42e083d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42db8e0 0x556cf42f86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42e083d740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42dc290 0x556cf42f8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42e083d740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 0x556cf435a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42e083d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42d81afe10 con 0x556cf42db8e0 2024-04-21T08:59:14.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42e083d740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f42d81b1280 con 0x556cf42e4e10 2024-04-21T08:59:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42e083d740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f42d81b04e0 con 0x556cf42dc290 2024-04-21T08:59:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42e083d740 10 client.0 _ll_register_callbacks cb 0x7f42c800b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42db8e0 0x556cf42f86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42db8e0 0x556cf42f86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53752/0 (socket says 172.21.15.28:53752) 2024-04-21T08:59:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42d77fe640 1 -- 172.21.15.28:0/1986057362 learned_addr learned my addr 172.21.15.28:0/1986057362 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/1986057362 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42dc290 0x556cf42f8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42d7fff640 1 --2- 172.21.15.28:0/1986057362 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 0x556cf435a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42d77fe640 1 -- 172.21.15.28:0/1986057362 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42dc290 msgr2=0x556cf42f8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42d77fe640 1 --2- 172.21.15.28:0/1986057362 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42dc290 0x556cf42f8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42d77fe640 1 -- 172.21.15.28:0/1986057362 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 msgr2=0x556cf435a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42d77fe640 1 --2- 172.21.15.28:0/1986057362 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 0x556cf435a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42d77fe640 1 -- 172.21.15.28:0/1986057362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cf435a910 con 0x556cf42db8e0 2024-04-21T08:59:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/1986057362 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42dc290 0x556cf42f8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42d7fff640 1 --2- 172.21.15.28:0/1986057362 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 0x556cf435a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42d77fe640 1 --2- 172.21.15.28:0/1986057362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42db8e0 0x556cf42f86c0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f42c80045f0 tx=0x7f42c800bcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b789f8baab6abb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.244+0000 7f42b6ffd640 1 -- 172.21.15.28:0/1986057362 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c8020160 con 0x556cf42db8e0 2024-04-21T08:59:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.245+0000 7f42b6ffd640 1 -- 172.21.15.28:0/1986057362 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7f42c8020300 con 0x556cf42db8e0 2024-04-21T08:59:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.245+0000 7f42e083d740 1 -- 172.21.15.28:0/1986057362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42cc0cf070 con 0x556cf42db8e0 2024-04-21T08:59:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.245+0000 7f42b6ffd640 1 -- 172.21.15.28:0/1986057362 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f42c8030050 con 0x556cf42db8e0 2024-04-21T08:59:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.245+0000 7f42e083d740 10 client.17763 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.245+0000 7f42e083d740 1 -- 172.21.15.28:0/1986057362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556cf435bb20 con 0x556cf42db8e0 2024-04-21T08:59:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.245+0000 7f42b6ffd640 1 -- 172.21.15.28:0/1986057362 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42cc0cf070 con 0x556cf42db8e0 2024-04-21T08:59:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.245+0000 7f42b6ffd640 1 -- 172.21.15.28:0/1986057362 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f42c8011a60 con 0x556cf42db8e0 2024-04-21T08:59:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.245+0000 7f42e083d740 10 client.17763 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.245+0000 7f42e083d740 10 client.17763 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.245+0000 7f42e083d740 20 client.17763 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.245+0000 7f42e083d740 1 --2- 172.21.15.28:0/1986057362 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556cf435c240 0x556cf435e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.245+0000 7f42e083d740 4 client.17763 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:59:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.245+0000 7f42e083d740 1 -- 172.21.15.28:0/1986057362 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556cf4247790 con 0x556cf435c240 2024-04-21T08:59:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.245+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/1986057362 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556cf435c240 0x556cf435e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.246+0000 7f42b6ffd640 10 client.17763 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.246+0000 7f42d6ffd640 1 --2- 172.21.15.28:0/1986057362 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556cf435c240 0x556cf435e660 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac897b0239b252bf server_cookie=c85d74d33dcf1c01 in_seq=0 out_seq=0 2024-04-21T08:59:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.247+0000 7f42b6ffd640 1 -- 172.21.15.28:0/1986057362 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556cf4247790 con 0x556cf435c240 2024-04-21T08:59:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.247+0000 7f42b6ffd640 10 client.17763 handle_command_reply: tid=0 2024-04-21T08:59:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.247+0000 7f42e083d740 1 client.17763 shutdown 2024-04-21T08:59:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.247+0000 7f42e083d740 1 -- 172.21.15.28:0/1986057362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42db8e0 msgr2=0x556cf42f86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.247+0000 7f42e083d740 1 --2- 172.21.15.28:0/1986057362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42db8e0 0x556cf42f86c0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f42c80045f0 tx=0x7f42c800bcc0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.247+0000 7f42e083d740 1 -- 172.21.15.28:0/1986057362 shutdown_connections 2024-04-21T08:59:14.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.247+0000 7f42e083d740 1 --2- 172.21.15.28:0/1986057362 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556cf435c240 0x556cf435e660 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.247+0000 7f42e083d740 1 --2- 172.21.15.28:0/1986057362 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556cf42e4e10 0x556cf435a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.247+0000 7f42e083d740 1 --2- 172.21.15.28:0/1986057362 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556cf42dc290 0x556cf42f8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.247+0000 7f42e083d740 1 --2- 172.21.15.28:0/1986057362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556cf42db8e0 0x556cf42f86c0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.247+0000 7f42b6ffd640 0 client.17763 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:14.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.247+0000 7f42e083d740 1 -- 172.21.15.28:0/1986057362 >> 172.21.15.28:0/1986057362 conn(0x556cf42d5fb0 msgr2=0x556cf42d7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:14.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.247+0000 7f42e083d740 1 -- 172.21.15.28:0/1986057362 shutdown_connections 2024-04-21T08:59:14.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.247+0000 7f42e083d740 1 -- 172.21.15.28:0/1986057362 wait complete. 2024-04-21T08:59:14.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.248+0000 7f42e083d740 20 client.17763 trim_cache size 0 max 16384 2024-04-21T08:59:14.259 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:59:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:59:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "65ef3917-70d1-4caa-8cc5-7bf17a2a89b6", 2024-04-21T08:59:14.260 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:59:14.261 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:59:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.248+0000 7f42ddc50640 1 -- 172.21.15.28:0/714511556 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f42c00a3ef0 msgr2=0x7f42c00a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.248+0000 7f42ddc50640 1 --2- 172.21.15.28:0/714511556 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f42c00a3ef0 0x7f42c00a63b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f42d0009a60 tx=0x7f42d00099f0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.248+0000 7f42ddc50640 1 -- 172.21.15.28:0/714511556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d8107260 msgr2=0x7f42d8074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.248+0000 7f42ddc50640 1 --2- 172.21.15.28:0/714511556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d8107260 0x7f42d8074790 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f42cc00c970 tx=0x7f42cc00ce30 comp rx=0 tx=0).stop 2024-04-21T08:59:14.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.248+0000 7f42ddc50640 1 -- 172.21.15.28:0/714511556 shutdown_connections 2024-04-21T08:59:14.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.248+0000 7f42ddc50640 1 --2- 172.21.15.28:0/714511556 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f42c00a3ef0 0x7f42c00a63b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.248+0000 7f42ddc50640 1 --2- 172.21.15.28:0/714511556 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42d8074cd0 0x7f42d81a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.248+0000 7f42ddc50640 1 --2- 172.21.15.28:0/714511556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42d8107260 0x7f42d8074790 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.248+0000 7f42ddc50640 1 --2- 172.21.15.28:0/714511556 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42d8102190 0x7f42d8074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.248+0000 7f42ddc50640 1 -- 172.21.15.28:0/714511556 >> 172.21.15.28:0/714511556 conn(0x7f42d80fd990 msgr2=0x7f42d80fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:14.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.248+0000 7f42ddc50640 1 -- 172.21.15.28:0/714511556 shutdown_connections 2024-04-21T08:59:14.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.248+0000 7f42ddc50640 1 -- 172.21.15.28:0/714511556 wait complete. 2024-04-21T08:59:14.264 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:59:14.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:59:14.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.371+0000 7f419503f640 1 -- 172.21.15.28:0/1554450988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4190106150 msgr2=0x7f419010dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.371+0000 7f419503f640 1 --2- 172.21.15.28:0/1554450988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4190106150 0x7f419010dd20 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f4188009ca0 tx=0x7f418802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.371+0000 7f419503f640 1 -- 172.21.15.28:0/1554450988 shutdown_connections 2024-04-21T08:59:14.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.371+0000 7f419503f640 1 --2- 172.21.15.28:0/1554450988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4190106150 0x7f419010dd20 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.371+0000 7f419503f640 1 --2- 172.21.15.28:0/1554450988 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4190101a30 0x7f4190105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.371+0000 7f419503f640 1 --2- 172.21.15.28:0/1554450988 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4190101080 0x7f4190101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.372+0000 7f419503f640 1 -- 172.21.15.28:0/1554450988 >> 172.21.15.28:0/1554450988 conn(0x7f4190078000 msgr2=0x7f4190078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:14.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.372+0000 7f419503f640 1 -- 172.21.15.28:0/1554450988 shutdown_connections 2024-04-21T08:59:14.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.372+0000 7f419503f640 1 -- 172.21.15.28:0/1554450988 wait complete. 2024-04-21T08:59:14.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.372+0000 7f419503f640 1 Processor -- start 2024-04-21T08:59:14.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.372+0000 7f419503f640 1 -- start start 2024-04-21T08:59:14.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.372+0000 7f419503f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4190101080 0x7f41901986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.372+0000 7f419503f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4190106150 0x7f4190198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.372+0000 7f419503f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419019cf80 0x7f419019d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.372+0000 7f419503f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41901110a0 con 0x7f419019cf80 2024-04-21T08:59:14.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.372+0000 7f419503f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4190110f20 con 0x7f4190101080 2024-04-21T08:59:14.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.372+0000 7f419503f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4190111220 con 0x7f4190106150 2024-04-21T08:59:14.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f418f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419019cf80 0x7f419019d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f418f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419019cf80 0x7f419019d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53782/0 (socket says 172.21.15.28:53782) 2024-04-21T08:59:14.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f418f59e640 1 -- 172.21.15.28:0/2293672547 learned_addr learned my addr 172.21.15.28:0/2293672547 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f418ed9d640 1 --2- 172.21.15.28:0/2293672547 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4190101080 0x7f41901986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f4187fff640 1 --2- 172.21.15.28:0/2293672547 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4190106150 0x7f4190198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f418f59e640 1 -- 172.21.15.28:0/2293672547 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4190106150 msgr2=0x7f4190198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f418f59e640 1 --2- 172.21.15.28:0/2293672547 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4190106150 0x7f4190198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f418f59e640 1 -- 172.21.15.28:0/2293672547 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4190101080 msgr2=0x7f41901986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f418f59e640 1 --2- 172.21.15.28:0/2293672547 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4190101080 0x7f41901986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f418f59e640 1 -- 172.21.15.28:0/2293672547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419019da70 con 0x7f419019cf80 2024-04-21T08:59:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f4187fff640 1 --2- 172.21.15.28:0/2293672547 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4190106150 0x7f4190198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f418f59e640 1 --2- 172.21.15.28:0/2293672547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419019cf80 0x7f419019d430 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f4188005e30 tx=0x7f4188034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f418cd99640 1 -- 172.21.15.28:0/2293672547 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4188039040 con 0x7f419019cf80 2024-04-21T08:59:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f419503f640 1 -- 172.21.15.28:0/2293672547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f419019dd00 con 0x7f419019cf80 2024-04-21T08:59:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f418cd99640 1 -- 172.21.15.28:0/2293672547 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4188044070 con 0x7f419019cf80 2024-04-21T08:59:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f418cd99640 1 -- 172.21.15.28:0/2293672547 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4188003e60 con 0x7f419019cf80 2024-04-21T08:59:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.373+0000 7f419503f640 1 -- 172.21.15.28:0/2293672547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41901a5520 con 0x7f419019cf80 2024-04-21T08:59:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.376+0000 7f418cd99640 1 -- 172.21.15.28:0/2293672547 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f41880324c0 con 0x7f419019cf80 2024-04-21T08:59:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.376+0000 7f4197c2b740 1 Processor -- start 2024-04-21T08:59:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.376+0000 7f4197c2b740 1 -- start start 2024-04-21T08:59:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f4197c2b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8230 0x5651887f8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f4197c2b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8b50 0x565188805950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f4197c2b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x565188808280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f4197c2b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56518875d1e0 con 0x5651887f8230 2024-04-21T08:59:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f4197c2b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5651886fa390 con 0x565188805e90 2024-04-21T08:59:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f4197c2b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5651887591a0 con 0x5651887f8b50 2024-04-21T08:59:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f4187fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8230 0x5651887f8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f418ed9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8b50 0x565188805950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f418ed9d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8b50 0x565188805950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.28:43396/0 (socket says 172.21.15.28:43396) 2024-04-21T08:59:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f418ed9d640 1 -- 172.21.15.28:0/3563490918 learned_addr learned my addr 172.21.15.28:0/3563490918 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f418cd99640 1 --2- 172.21.15.28:0/2293672547 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f417c0a3d20 0x7f417c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f4187fff640 1 -- 172.21.15.28:0/3563490918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8b50 msgr2=0x565188805950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f4187fff640 1 --2- 172.21.15.28:0/3563490918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8b50 0x565188805950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f4187fff640 1 -- 172.21.15.28:0/3563490918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 msgr2=0x565188808280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f4187fff640 1 --2- 172.21.15.28:0/3563490918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x565188808280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f4187fff640 1 -- 172.21.15.28:0/3563490918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651885397d0 con 0x5651887f8230 2024-04-21T08:59:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f418f59e640 1 --2- 172.21.15.28:0/2293672547 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f417c0a3d20 0x7f417c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f418cd99640 1 -- 172.21.15.28:0/2293672547 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7f41880f2c60 con 0x7f419019cf80 2024-04-21T08:59:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.377+0000 7f4187fff640 1 --2- 172.21.15.28:0/3563490918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8230 0x5651887f8610 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f419006b6f0 tx=0x7f4178010310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eed57c5a4a3b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.378+0000 7f41857fa640 1 -- 172.21.15.28:0/3563490918 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4178016070 con 0x5651887f8230 2024-04-21T08:59:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.378+0000 7f41857fa640 1 -- 172.21.15.28:0/3563490918 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4178010d40 con 0x5651887f8230 2024-04-21T08:59:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.378+0000 7f41857fa640 1 -- 172.21.15.28:0/3563490918 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4178004dd0 con 0x5651887f8230 2024-04-21T08:59:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.378+0000 7f4197c2b740 1 -- 172.21.15.28:0/3563490918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8230 msgr2=0x5651887f8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.378+0000 7f418f59e640 1 --2- 172.21.15.28:0/2293672547 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f417c0a3d20 0x7f417c0a61e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f41880f0120 tx=0x7f41880df040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:14.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.378+0000 7f4197c2b740 1 --2- 172.21.15.28:0/3563490918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8230 0x5651887f8610 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f419006b6f0 tx=0x7f4178010310 comp rx=0 tx=0).stop 2024-04-21T08:59:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 1 -- 172.21.15.28:0/3563490918 shutdown_connections 2024-04-21T08:59:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 1 --2- 172.21.15.28:0/3563490918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x565188808280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 1 --2- 172.21.15.28:0/3563490918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8b50 0x565188805950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 1 --2- 172.21.15.28:0/3563490918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8230 0x5651887f8610 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 1 -- 172.21.15.28:0/3563490918 >> 172.21.15.28:0/3563490918 conn(0x5651887f2cb0 msgr2=0x5651887f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 1 -- 172.21.15.28:0/3563490918 shutdown_connections 2024-04-21T08:59:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 1 -- 172.21.15.28:0/3563490918 wait complete. 2024-04-21T08:59:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 1 -- start start 2024-04-21T08:59:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 1 Processor -- start 2024-04-21T08:59:14.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8230 0x565188802e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8b50 0x565188803350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x565188803890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x565188808cd0 con 0x5651887f8230 2024-04-21T08:59:14.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5651887162f0 con 0x565188805e90 2024-04-21T08:59:14.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x565188808e50 con 0x5651887f8b50 2024-04-21T08:59:14.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4197c2b740 10 client.0 _ll_register_callbacks cb 0x565188723580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:14.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4187fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8230 0x565188802e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4187fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8230 0x565188802e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53808/0 (socket says 172.21.15.28:53808) 2024-04-21T08:59:14.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.379+0000 7f4187fff640 1 -- 172.21.15.28:0/2078009897 learned_addr learned my addr 172.21.15.28:0/2078009897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:14.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f4187fff640 1 --2- 172.21.15.28:0/2078009897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x565188803890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f418ed9d640 1 --2- 172.21.15.28:0/2078009897 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8b50 0x565188803350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f4187fff640 1 -- 172.21.15.28:0/2078009897 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8b50 msgr2=0x565188803350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f4187fff640 1 --2- 172.21.15.28:0/2078009897 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8b50 0x565188803350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f4187fff640 1 -- 172.21.15.28:0/2078009897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 msgr2=0x565188803890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f4187fff640 1 --2- 172.21.15.28:0/2078009897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x565188803890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f4187fff640 1 -- 172.21.15.28:0/2078009897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56518846f3f0 con 0x5651887f8230 2024-04-21T08:59:14.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f4187fff640 1 --2- 172.21.15.28:0/2078009897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8230 0x565188802e10 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f4178009360 tx=0x7f4178010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76424024163801f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:14.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f41767fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:14.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f41767fc640 1 -- 172.21.15.28:0/2078009897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4178016040 con 0x5651887f8230 2024-04-21T08:59:14.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f41767fc640 1 -- 172.21.15.28:0/2078009897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7f41780052e0 con 0x5651887f8230 2024-04-21T08:59:14.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f4197c2b740 1 -- 172.21.15.28:0/2078009897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4178010d40 con 0x5651887f8230 2024-04-21T08:59:14.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f41767fc640 1 -- 172.21.15.28:0/2078009897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f41780054e0 con 0x5651887f8230 2024-04-21T08:59:14.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f4197c2b740 10 client.17775 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:14.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f4197c2b740 1 -- 172.21.15.28:0/2078009897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56518842f0f0 con 0x5651887f8230 2024-04-21T08:59:14.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.380+0000 7f41767fc640 1 -- 172.21.15.28:0/2078009897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4178010d40 con 0x5651887f8230 2024-04-21T08:59:14.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.381+0000 7f41767fc640 1 -- 172.21.15.28:0/2078009897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f4178029020 con 0x5651887f8230 2024-04-21T08:59:14.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.381+0000 7f4197c2b740 10 client.17775 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:14.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.381+0000 7f4197c2b740 10 client.17775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:14.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.381+0000 7f4197c2b740 20 client.17775 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:14.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.381+0000 7f4197c2b740 1 --2- 172.21.15.28:0/2078009897 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x565188815e80 0x565188818270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.381+0000 7f4197c2b740 4 client.17775 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:59:14.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.381+0000 7f4197c2b740 1 -- 172.21.15.28:0/2078009897 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56518876ac00 con 0x565188815e80 2024-04-21T08:59:14.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.381+0000 7f418ed9d640 1 --2- 172.21.15.28:0/2078009897 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x565188815e80 0x565188818270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.381+0000 7f418ed9d640 1 --2- 172.21.15.28:0/2078009897 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x565188815e80 0x565188818270 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=615fc8742944f9a8 server_cookie=7f86c6e5d4befb8a in_seq=0 out_seq=0 2024-04-21T08:59:14.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.381+0000 7f41767fc640 10 client.17775 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:14.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.382+0000 7f41767fc640 1 -- 172.21.15.28:0/2078009897 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56518876ac00 con 0x565188815e80 2024-04-21T08:59:14.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.382+0000 7f41767fc640 10 client.17775 handle_command_reply: tid=0 2024-04-21T08:59:14.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.382+0000 7f4197c2b740 1 client.17775 shutdown 2024-04-21T08:59:14.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.382+0000 7f4197c2b740 1 -- 172.21.15.28:0/2078009897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8230 msgr2=0x565188802e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.382+0000 7f4197c2b740 1 --2- 172.21.15.28:0/2078009897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8230 0x565188802e10 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f4178009360 tx=0x7f4178010c80 comp rx=0 tx=0).stop 2024-04-21T08:59:14.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.382+0000 7f4197c2b740 1 -- 172.21.15.28:0/2078009897 shutdown_connections 2024-04-21T08:59:14.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.382+0000 7f4197c2b740 1 --2- 172.21.15.28:0/2078009897 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x565188815e80 0x565188818270 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.382+0000 7f4197c2b740 1 --2- 172.21.15.28:0/2078009897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x565188803890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.382+0000 7f4197c2b740 1 --2- 172.21.15.28:0/2078009897 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8b50 0x565188803350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.382+0000 7f41767fc640 0 client.17775 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:14.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.382+0000 7f4197c2b740 1 --2- 172.21.15.28:0/2078009897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8230 0x565188802e10 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.382+0000 7f4197c2b740 1 -- 172.21.15.28:0/2078009897 >> 172.21.15.28:0/2078009897 conn(0x5651887f2cb0 msgr2=0x5651887f4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:14.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.382+0000 7f4197c2b740 1 -- 172.21.15.28:0/2078009897 shutdown_connections 2024-04-21T08:59:14.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.382+0000 7f4197c2b740 1 -- 172.21.15.28:0/2078009897 wait complete. 2024-04-21T08:59:14.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.383+0000 7f4197c2b740 20 client.17775 trim_cache size 0 max 16384 2024-04-21T08:59:14.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.400+0000 7f4197c2b740 1 Processor -- start 2024-04-21T08:59:14.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.400+0000 7f4197c2b740 1 -- start start 2024-04-21T08:59:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.400+0000 7f4197c2b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8230 0x565188805540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.400+0000 7f4197c2b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8b50 0x565188811c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.400+0000 7f4197c2b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x56518881d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.400+0000 7f4197c2b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41901afc20 con 0x5651887f8b50 2024-04-21T08:59:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.400+0000 7f4197c2b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f41901ad4f0 con 0x565188805e90 2024-04-21T08:59:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.400+0000 7f4197c2b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f41901ab360 con 0x5651887f8230 2024-04-21T08:59:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.400+0000 7f418ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8b50 0x565188811c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.401+0000 7f418ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8b50 0x565188811c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53812/0 (socket says 172.21.15.28:53812) 2024-04-21T08:59:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.401+0000 7f418ed9d640 1 -- 172.21.15.28:0/506681949 learned_addr learned my addr 172.21.15.28:0/506681949 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.401+0000 7f4187fff640 1 --2- 172.21.15.28:0/506681949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x56518881d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.401+0000 7f418ed9d640 1 -- 172.21.15.28:0/506681949 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8230 msgr2=0x565188805540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.401+0000 7f418ed9d640 1 --2- 172.21.15.28:0/506681949 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8230 0x565188805540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.401+0000 7f418ed9d640 1 -- 172.21.15.28:0/506681949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 msgr2=0x56518881d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.401+0000 7f418ed9d640 1 --2- 172.21.15.28:0/506681949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x56518881d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.401+0000 7f418ed9d640 1 -- 172.21.15.28:0/506681949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651887f2ae0 con 0x5651887f8b50 2024-04-21T08:59:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.401+0000 7f4187fff640 1 --2- 172.21.15.28:0/506681949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x56518881d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.401+0000 7f418ed9d640 1 --2- 172.21.15.28:0/506681949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8b50 0x565188811c60 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x565188802670 tx=0x7f4180010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e55ef7fae65c9b52 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.401+0000 7f4184ff9640 1 -- 172.21.15.28:0/506681949 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180004220 con 0x5651887f8b50 2024-04-21T08:59:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.401+0000 7f4184ff9640 1 -- 172.21.15.28:0/506681949 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f418001c8e0 con 0x5651887f8b50 2024-04-21T08:59:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.401+0000 7f4184ff9640 1 -- 172.21.15.28:0/506681949 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180003d40 con 0x5651887f8b50 2024-04-21T08:59:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 -- 172.21.15.28:0/506681949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8b50 msgr2=0x565188811c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 --2- 172.21.15.28:0/506681949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8b50 0x565188811c60 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x565188802670 tx=0x7f4180010800 comp rx=0 tx=0).stop 2024-04-21T08:59:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 -- 172.21.15.28:0/506681949 shutdown_connections 2024-04-21T08:59:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 --2- 172.21.15.28:0/506681949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x56518881d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 --2- 172.21.15.28:0/506681949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8b50 0x565188811c60 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 --2- 172.21.15.28:0/506681949 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8230 0x565188805540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 -- 172.21.15.28:0/506681949 >> 172.21.15.28:0/506681949 conn(0x5651887f2cb0 msgr2=0x5651887fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 -- 172.21.15.28:0/506681949 shutdown_connections 2024-04-21T08:59:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 -- 172.21.15.28:0/506681949 wait complete. 2024-04-21T08:59:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 -- start start 2024-04-21T08:59:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 Processor -- start 2024-04-21T08:59:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8230 0x5651887ffb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8b50 0x565188800080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x5651888005c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41901ab9b0 con 0x5651887f8b50 2024-04-21T08:59:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f41901ace20 con 0x565188805e90 2024-04-21T08:59:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f41901ac080 con 0x5651887f8230 2024-04-21T08:59:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.402+0000 7f4197c2b740 10 client.0 _ll_register_callbacks cb 0x7f418000c390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f418ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8b50 0x565188800080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f418ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8b50 0x565188800080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53824/0 (socket says 172.21.15.28:53824) 2024-04-21T08:59:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f418ed9d640 1 -- 172.21.15.28:0/1698488766 learned_addr learned my addr 172.21.15.28:0/1698488766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f4187fff640 1 --2- 172.21.15.28:0/1698488766 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8230 0x5651887ffb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f4187fff640 1 --2- 172.21.15.28:0/1698488766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x5651888005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f418ed9d640 1 -- 172.21.15.28:0/1698488766 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8230 msgr2=0x5651887ffb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f418ed9d640 1 --2- 172.21.15.28:0/1698488766 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8230 0x5651887ffb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f418ed9d640 1 -- 172.21.15.28:0/1698488766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 msgr2=0x5651888005c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f418ed9d640 1 --2- 172.21.15.28:0/1698488766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x5651888005c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f418ed9d640 1 -- 172.21.15.28:0/1698488766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565188808280 con 0x5651887f8b50 2024-04-21T08:59:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f4187fff640 1 --2- 172.21.15.28:0/1698488766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x5651888005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f418ed9d640 1 --2- 172.21.15.28:0/1698488766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8b50 0x565188800080 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f4180003a20 tx=0x7f41800037d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=539acebd628c17a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f41767fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f41767fc640 1 -- 172.21.15.28:0/1698488766 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180003ac0 con 0x5651887f8b50 2024-04-21T08:59:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f4197c2b740 1 -- 172.21.15.28:0/1698488766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f418001c8e0 con 0x5651887f8b50 2024-04-21T08:59:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f41767fc640 1 -- 172.21.15.28:0/1698488766 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9621+0+0 (secure 0 0 0) 0x7f4180003c60 con 0x5651887f8b50 2024-04-21T08:59:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f41767fc640 1 -- 172.21.15.28:0/1698488766 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4180036050 con 0x5651887f8b50 2024-04-21T08:59:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f4197c2b740 10 client.17781 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.403+0000 7f4197c2b740 1 -- 172.21.15.28:0/1698488766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56518887c400 con 0x5651887f8b50 2024-04-21T08:59:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.404+0000 7f41767fc640 1 -- 172.21.15.28:0/1698488766 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f418001c8e0 con 0x5651887f8b50 2024-04-21T08:59:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.404+0000 7f41767fc640 1 -- 172.21.15.28:0/1698488766 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f4180025ad0 con 0x5651887f8b50 2024-04-21T08:59:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.404+0000 7f4197c2b740 10 client.17781 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.404+0000 7f4197c2b740 10 client.17781 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.404+0000 7f4197c2b740 20 client.17781 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.404+0000 7f4197c2b740 1 --2- 172.21.15.28:0/1698488766 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56518887cb80 0x56518887efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.404+0000 7f4197c2b740 4 client.17781 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:59:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.404+0000 7f4197c2b740 1 -- 172.21.15.28:0/1698488766 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x565188764a70 con 0x56518887cb80 2024-04-21T08:59:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.404+0000 7f4187fff640 1 --2- 172.21.15.28:0/1698488766 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56518887cb80 0x56518887efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.404+0000 7f41767fc640 10 client.17781 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.404+0000 7f4187fff640 1 --2- 172.21.15.28:0/1698488766 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56518887cb80 0x56518887efa0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78be050923831558 server_cookie=3312d7a7e7ce0640 in_seq=0 out_seq=0 2024-04-21T08:59:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.405+0000 7f41767fc640 1 -- 172.21.15.28:0/1698488766 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x565188764a70 con 0x56518887cb80 2024-04-21T08:59:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.405+0000 7f41767fc640 10 client.17781 handle_command_reply: tid=0 2024-04-21T08:59:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.405+0000 7f4197c2b740 1 client.17781 shutdown 2024-04-21T08:59:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.405+0000 7f4197c2b740 1 -- 172.21.15.28:0/1698488766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8b50 msgr2=0x565188800080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.405+0000 7f4197c2b740 1 --2- 172.21.15.28:0/1698488766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8b50 0x565188800080 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f4180003a20 tx=0x7f41800037d0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.405+0000 7f4197c2b740 1 -- 172.21.15.28:0/1698488766 shutdown_connections 2024-04-21T08:59:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.405+0000 7f4197c2b740 1 --2- 172.21.15.28:0/1698488766 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56518887cb80 0x56518887efa0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.405+0000 7f4197c2b740 1 --2- 172.21.15.28:0/1698488766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565188805e90 0x5651888005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.405+0000 7f41767fc640 0 client.17781 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.405+0000 7f4197c2b740 1 --2- 172.21.15.28:0/1698488766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5651887f8b50 0x565188800080 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.405+0000 7f4197c2b740 1 --2- 172.21.15.28:0/1698488766 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5651887f8230 0x5651887ffb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.405+0000 7f4197c2b740 1 -- 172.21.15.28:0/1698488766 >> 172.21.15.28:0/1698488766 conn(0x5651887f2cb0 msgr2=0x5651887f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.405+0000 7f4197c2b740 1 -- 172.21.15.28:0/1698488766 shutdown_connections 2024-04-21T08:59:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.405+0000 7f4197c2b740 1 -- 172.21.15.28:0/1698488766 wait complete. 2024-04-21T08:59:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.405+0000 7f4197c2b740 20 client.17781 trim_cache size 0 max 16384 2024-04-21T08:59:14.416 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:59:14.416 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:59:14.416 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:59:14.416 INFO:teuthology.orchestra.run.smithi028.stdout: "65ef3917-70d1-4caa-8cc5-7bf17a2a89b6": { 2024-04-21T08:59:14.417 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:59:14.417 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "65ef3917-70d1-4caa-8cc5-7bf17a2a89b6", 2024-04-21T08:59:14.417 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:59:14.417 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:59:14.417 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:59:14.417 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:59:14.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.406+0000 7f419503f640 1 -- 172.21.15.28:0/2293672547 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f417c0a3d20 msgr2=0x7f417c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.406+0000 7f419503f640 1 --2- 172.21.15.28:0/2293672547 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f417c0a3d20 0x7f417c0a61e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f41880f0120 tx=0x7f41880df040 comp rx=0 tx=0).stop 2024-04-21T08:59:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.406+0000 7f419503f640 1 -- 172.21.15.28:0/2293672547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419019cf80 msgr2=0x7f419019d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.406+0000 7f419503f640 1 --2- 172.21.15.28:0/2293672547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419019cf80 0x7f419019d430 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f4188005e30 tx=0x7f4188034760 comp rx=0 tx=0).stop 2024-04-21T08:59:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.406+0000 7f419503f640 1 -- 172.21.15.28:0/2293672547 shutdown_connections 2024-04-21T08:59:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.406+0000 7f419503f640 1 --2- 172.21.15.28:0/2293672547 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f417c0a3d20 0x7f417c0a61e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.406+0000 7f419503f640 1 --2- 172.21.15.28:0/2293672547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419019cf80 0x7f419019d430 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.406+0000 7f419503f640 1 --2- 172.21.15.28:0/2293672547 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4190106150 0x7f4190198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.406+0000 7f419503f640 1 --2- 172.21.15.28:0/2293672547 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4190101080 0x7f41901986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:14.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.406+0000 7f419503f640 1 -- 172.21.15.28:0/2293672547 >> 172.21.15.28:0/2293672547 conn(0x7f4190078000 msgr2=0x7f4190069480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:14.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.406+0000 7f419503f640 1 -- 172.21.15.28:0/2293672547 shutdown_connections 2024-04-21T08:59:14.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:14.406+0000 7f419503f640 1 -- 172.21.15.28:0/2293672547 wait complete. 2024-04-21T08:59:14.420 INFO:tasks.cephfs.filesystem:scrub status for tag:65ef3917-70d1-4caa-8cc5-7bf17a2a89b6 - {'path': '/', 'tag': '65ef3917-70d1-4caa-8cc5-7bf17a2a89b6', 'options': 'recursive,force'} 2024-04-21T08:59:14.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:14 smithi152 ceph-mon[31401]: pgmap v247: 129 pgs: 11 active+clean+snaptrim, 90 active+clean, 28 active+clean+snaptrim_wait; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-04-21T08:59:14.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:14 smithi028 ceph-mon[28003]: pgmap v247: 129 pgs: 11 active+clean+snaptrim, 90 active+clean, 28 active+clean+snaptrim_wait; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-04-21T08:59:14.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:14 smithi171 ceph-mon[31358]: pgmap v247: 129 pgs: 11 active+clean+snaptrim, 90 active+clean, 28 active+clean+snaptrim_wait; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-04-21T08:59:15.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:59:15.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:15.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:15.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:15.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:15.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.856 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:15.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:15.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:15.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:15.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:15.857 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:15.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.858 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:15 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:59:15.858 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:15.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:15.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:15.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:15.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:15.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:15.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:15.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:15.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:15.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:15.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:15 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:59:15.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:15.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:15.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:15.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:15.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:15.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:15.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:15.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:15.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.872 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:15.873 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.873 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:15.873 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:15.873 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:15 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:16.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:16 smithi152 ceph-mon[31401]: pgmap v248: 129 pgs: 2 active+clean+snaptrim, 119 active+clean, 8 active+clean+snaptrim_wait; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 204 B/s wr, 11 op/s 2024-04-21T08:59:16.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:16 smithi028 ceph-mon[28003]: pgmap v248: 129 pgs: 2 active+clean+snaptrim, 119 active+clean, 8 active+clean+snaptrim_wait; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 204 B/s wr, 11 op/s 2024-04-21T08:59:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:16 smithi171 ceph-mon[31358]: pgmap v248: 129 pgs: 2 active+clean+snaptrim, 119 active+clean, 8 active+clean+snaptrim_wait; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 204 B/s wr, 11 op/s 2024-04-21T08:59:18.855 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:18 smithi152 ceph-mon[31401]: pgmap v249: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 15 op/s 2024-04-21T08:59:18.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:18 smithi028 ceph-mon[28003]: pgmap v249: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 15 op/s 2024-04-21T08:59:18.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:18 smithi171 ceph-mon[31358]: pgmap v249: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 15 op/s 2024-04-21T08:59:20.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:19 smithi152 ceph-mon[31401]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T08:59:20.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:19 smithi152 ceph-mon[31401]: pgmap v251: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 102 B/s wr, 33 op/s 2024-04-21T08:59:20.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:19 smithi028 ceph-mon[28003]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T08:59:20.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:19 smithi028 ceph-mon[28003]: pgmap v251: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 102 B/s wr, 33 op/s 2024-04-21T08:59:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:19 smithi171 ceph-mon[31358]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T08:59:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:19 smithi171 ceph-mon[31358]: pgmap v251: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 102 B/s wr, 33 op/s 2024-04-21T08:59:22.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:22 smithi152 ceph-mon[31401]: pgmap v252: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 102 B/s wr, 33 op/s 2024-04-21T08:59:22.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:22 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:59:22.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:22 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:59:22.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:22 smithi028 ceph-mon[28003]: pgmap v252: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 102 B/s wr, 33 op/s 2024-04-21T08:59:22.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:22 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:59:22.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:22 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:59:22.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:22 smithi171 ceph-mon[31358]: pgmap v252: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 102 B/s wr, 33 op/s 2024-04-21T08:59:22.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:22 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:59:22.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:22 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:59:23.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:23 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:59:23.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:23 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:59:23.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:23 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:59:24.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:24 smithi152 ceph-mon[31401]: pgmap v253: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 33 op/s 2024-04-21T08:59:24.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:24 smithi028 ceph-mon[28003]: pgmap v253: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 33 op/s 2024-04-21T08:59:24.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:24 smithi171 ceph-mon[31358]: pgmap v253: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 33 op/s 2024-04-21T08:59:26.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:26 smithi152 ceph-mon[31401]: pgmap v254: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 28 op/s 2024-04-21T08:59:26.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:26 smithi028 ceph-mon[28003]: pgmap v254: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 28 op/s 2024-04-21T08:59:26.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:26 smithi171 ceph-mon[31358]: pgmap v254: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 28 op/s 2024-04-21T08:59:28.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:28 smithi152 ceph-mon[31401]: pgmap v255: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 24 op/s 2024-04-21T08:59:28.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:28 smithi028 ceph-mon[28003]: pgmap v255: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 24 op/s 2024-04-21T08:59:28.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:28 smithi171 ceph-mon[31358]: pgmap v255: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 24 op/s 2024-04-21T08:59:30.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:30 smithi171 ceph-mon[31358]: pgmap v256: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:59:30.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:30 smithi152 ceph-mon[31401]: pgmap v256: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:59:30.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:30 smithi028 ceph-mon[28003]: pgmap v256: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:59:32.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:32 smithi171 ceph-mon[31358]: pgmap v257: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:59:32.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:32 smithi152 ceph-mon[31401]: pgmap v257: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:59:32.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:32 smithi028 ceph-mon[28003]: pgmap v257: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:59:34.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:34 smithi171 ceph-mon[31358]: pgmap v258: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:59:34.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:34 smithi152 ceph-mon[31401]: pgmap v258: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:59:34.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:34 smithi028 ceph-mon[28003]: pgmap v258: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:59:36.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:36 smithi171 ceph-mon[31358]: pgmap v259: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:59:36.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:36 smithi152 ceph-mon[31401]: pgmap v259: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:59:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:36 smithi028 ceph-mon[28003]: pgmap v259: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:59:38.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:38 smithi171 ceph-mon[31358]: pgmap v260: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:59:38.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:38 smithi152 ceph-mon[31401]: pgmap v260: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:59:38.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:38 smithi028 ceph-mon[28003]: pgmap v260: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:59:40.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:40 smithi152 ceph-mon[31401]: pgmap v261: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:59:40.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:40 smithi028 ceph-mon[28003]: pgmap v261: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:59:40.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:40 smithi171 ceph-mon[31358]: pgmap v261: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T08:59:42.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:42 smithi152 ceph-mon[31401]: pgmap v262: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:59:42.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:42 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:59:42.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:42 smithi028 ceph-mon[28003]: pgmap v262: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:59:42.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:42 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:59:42.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:42 smithi171 ceph-mon[31358]: pgmap v262: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T08:59:42.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:42 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:59:43.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:43 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:59:43.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:43 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:59:43.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:43 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:59:43.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:43 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:59:43.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:43 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:59:43.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:43 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:59:43.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:43 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:59:43.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:43 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:59:43.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:43 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T08:59:44.422 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:59:44.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:59:44.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.540+0000 7f70f3643640 1 -- 172.21.15.28:0/2052333558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70ec10e810 msgr2=0x7f70ec110c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:44.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.540+0000 7f70f3643640 1 --2- 172.21.15.28:0/2052333558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70ec10e810 0x7f70ec110c00 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009ca0 tx=0x7f70e402c180 comp rx=0 tx=0).stop 2024-04-21T08:59:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.540+0000 7f70f3643640 1 -- 172.21.15.28:0/2052333558 shutdown_connections 2024-04-21T08:59:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.540+0000 7f70f3643640 1 --2- 172.21.15.28:0/2052333558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70ec10e810 0x7f70ec110c00 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.540+0000 7f70f3643640 1 --2- 172.21.15.28:0/2052333558 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70ec074b40 0x7f70ec10e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.540+0000 7f70f3643640 1 --2- 172.21.15.28:0/2052333558 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70ec074220 0x7f70ec074600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.540+0000 7f70f3643640 1 -- 172.21.15.28:0/2052333558 >> 172.21.15.28:0/2052333558 conn(0x7f70ec0793a0 msgr2=0x7f70ec0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.540+0000 7f70f3643640 1 -- 172.21.15.28:0/2052333558 shutdown_connections 2024-04-21T08:59:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.540+0000 7f70f3643640 1 -- 172.21.15.28:0/2052333558 wait complete. 2024-04-21T08:59:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.541+0000 7f70f3643640 1 Processor -- start 2024-04-21T08:59:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.541+0000 7f70f3643640 1 -- start start 2024-04-21T08:59:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.541+0000 7f70f3643640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70ec074220 0x7f70ec1a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.541+0000 7f70f3643640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70ec074b40 0x7f70ec1a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.541+0000 7f70f3643640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70ec10e810 0x7f70ec19a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.541+0000 7f70f3643640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec114380 con 0x7f70ec074220 2024-04-21T08:59:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.541+0000 7f70f3643640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec114200 con 0x7f70ec074b40 2024-04-21T08:59:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.541+0000 7f70f3643640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec114500 con 0x7f70ec10e810 2024-04-21T08:59:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.541+0000 7f70f13df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70ec074220 0x7f70ec1a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.541+0000 7f70f13df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70ec074220 0x7f70ec1a0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46134/0 (socket says 172.21.15.28:46134) 2024-04-21T08:59:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.541+0000 7f70f13df640 1 -- 172.21.15.28:0/2889753333 learned_addr learned my addr 172.21.15.28:0/2889753333 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.541+0000 7f70f0bde640 1 --2- 172.21.15.28:0/2889753333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70ec074b40 0x7f70ec1a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.541+0000 7f70f1be0640 1 --2- 172.21.15.28:0/2889753333 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70ec10e810 0x7f70ec19a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.541+0000 7f70f13df640 1 -- 172.21.15.28:0/2889753333 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70ec10e810 msgr2=0x7f70ec19a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.542+0000 7f70f13df640 1 --2- 172.21.15.28:0/2889753333 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70ec10e810 0x7f70ec19a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.542+0000 7f70f13df640 1 -- 172.21.15.28:0/2889753333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70ec074b40 msgr2=0x7f70ec1a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.542+0000 7f70f13df640 1 --2- 172.21.15.28:0/2889753333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70ec074b40 0x7f70ec1a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.542+0000 7f70f13df640 1 -- 172.21.15.28:0/2889753333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70ec19ad60 con 0x7f70ec074220 2024-04-21T08:59:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.542+0000 7f70f0bde640 1 --2- 172.21.15.28:0/2889753333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70ec074b40 0x7f70ec1a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.542+0000 7f70f13df640 1 --2- 172.21.15.28:0/2889753333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70ec074220 0x7f70ec1a0430 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f70e800c920 tx=0x7f70e800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.542+0000 7f70e27fc640 1 -- 172.21.15.28:0/2889753333 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e8007c70 con 0x7f70ec074220 2024-04-21T08:59:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.542+0000 7f70e27fc640 1 -- 172.21.15.28:0/2889753333 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70e8007e10 con 0x7f70ec074220 2024-04-21T08:59:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.542+0000 7f70f3643640 1 -- 172.21.15.28:0/2889753333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70ec19b050 con 0x7f70ec074220 2024-04-21T08:59:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.542+0000 7f70e27fc640 1 -- 172.21.15.28:0/2889753333 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e8018820 con 0x7f70ec074220 2024-04-21T08:59:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.542+0000 7f70f3643640 1 -- 172.21.15.28:0/2889753333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70ec1a72f0 con 0x7f70ec074220 2024-04-21T08:59:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.544+0000 7f70e27fc640 1 -- 172.21.15.28:0/2889753333 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f70e800f450 con 0x7f70ec074220 2024-04-21T08:59:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f623c740 1 Processor -- start 2024-04-21T08:59:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f623c740 1 -- start start 2024-04-21T08:59:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f623c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84580 0x5555c7a84960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f623c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a84ea0 0x5555c7a8d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f623c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a8da80 0x5555c7a8fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f623c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5555c79e91e0 con 0x5555c7a84580 2024-04-21T08:59:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f623c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5555c7986390 con 0x5555c7a84ea0 2024-04-21T08:59:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f623c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5555c79e51a0 con 0x5555c7a8da80 2024-04-21T08:59:44.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70e27fc640 1 --2- 172.21.15.28:0/2889753333 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f70d80a3d20 0x7f70d80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f1be0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84580 0x5555c7a84960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f1be0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84580 0x5555c7a84960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46150/0 (socket says 172.21.15.28:46150) 2024-04-21T08:59:44.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f1be0640 1 -- 172.21.15.28:0/939885734 learned_addr learned my addr 172.21.15.28:0/939885734 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:44.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f0bde640 1 --2- 172.21.15.28:0/939885734 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a8da80 0x5555c7a8fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f13df640 1 --2- 172.21.15.28:0/939885734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a84ea0 0x5555c7a8d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f1be0640 1 --2- 172.21.15.28:0/2889753333 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f70d80a3d20 0x7f70d80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70e27fc640 1 -- 172.21.15.28:0/2889753333 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7f70e80d0e20 con 0x7f70ec074220 2024-04-21T08:59:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f1be0640 1 -- 172.21.15.28:0/939885734 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a8da80 msgr2=0x5555c7a8fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f1be0640 1 --2- 172.21.15.28:0/939885734 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a8da80 0x5555c7a8fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f1be0640 1 -- 172.21.15.28:0/939885734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a84ea0 msgr2=0x5555c7a8d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f1be0640 1 --2- 172.21.15.28:0/939885734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a84ea0 0x5555c7a8d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f1be0640 1 -- 172.21.15.28:0/939885734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555c77c57d0 con 0x5555c7a84580 2024-04-21T08:59:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.545+0000 7f70f0bde640 1 --2- 172.21.15.28:0/939885734 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a8da80 0x5555c7a8fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.546+0000 7f70f1be0640 1 --2- 172.21.15.28:0/939885734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84580 0x5555c7a84960 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f70e402c6c0 tx=0x7f70e4003d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8617c857ee674687 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.546+0000 7f70d77fe640 1 -- 172.21.15.28:0/939885734 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e4032560 con 0x5555c7a84580 2024-04-21T08:59:44.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.546+0000 7f70d77fe640 1 -- 172.21.15.28:0/939885734 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70e4032bb0 con 0x5555c7a84580 2024-04-21T08:59:44.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.546+0000 7f70d77fe640 1 -- 172.21.15.28:0/939885734 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e4039070 con 0x5555c7a84580 2024-04-21T08:59:44.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.547+0000 7f70f623c740 1 -- 172.21.15.28:0/939885734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84580 msgr2=0x5555c7a84960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:44.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.547+0000 7f70f1be0640 1 --2- 172.21.15.28:0/2889753333 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f70d80a3d20 0x7f70d80a61e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f70e404c000 tx=0x7f70e4048000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:44.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.547+0000 7f70f623c740 1 --2- 172.21.15.28:0/939885734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84580 0x5555c7a84960 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f70e402c6c0 tx=0x7f70e4003d20 comp rx=0 tx=0).stop 2024-04-21T08:59:44.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.547+0000 7f70f623c740 1 -- 172.21.15.28:0/939885734 shutdown_connections 2024-04-21T08:59:44.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.547+0000 7f70f623c740 1 --2- 172.21.15.28:0/939885734 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a8da80 0x5555c7a8fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.547+0000 7f70f623c740 1 --2- 172.21.15.28:0/939885734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a84ea0 0x5555c7a8d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.547+0000 7f70f623c740 1 --2- 172.21.15.28:0/939885734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84580 0x5555c7a84960 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.547+0000 7f70f623c740 1 -- 172.21.15.28:0/939885734 >> 172.21.15.28:0/939885734 conn(0x5555c7a7ecb0 msgr2=0x5555c7a7f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.547+0000 7f70f623c740 1 -- 172.21.15.28:0/939885734 shutdown_connections 2024-04-21T08:59:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.547+0000 7f70f623c740 1 -- 172.21.15.28:0/939885734 wait complete. 2024-04-21T08:59:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.547+0000 7f70f623c740 1 -- start start 2024-04-21T08:59:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.547+0000 7f70f623c740 1 Processor -- start 2024-04-21T08:59:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.547+0000 7f70f623c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a84580 0x5555c7a8aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f623c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a84ea0 0x5555c7a8b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f623c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a8da80 0x5555c7a8b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f623c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5555c7a90870 con 0x5555c7a8da80 2024-04-21T08:59:44.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f623c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5555c79a22f0 con 0x5555c7a84ea0 2024-04-21T08:59:44.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f623c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5555c7a909f0 con 0x5555c7a84580 2024-04-21T08:59:44.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f623c740 10 client.0 _ll_register_callbacks cb 0x5555c79af580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:44.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f1be0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a8da80 0x5555c7a8b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f1be0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a8da80 0x5555c7a8b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46156/0 (socket says 172.21.15.28:46156) 2024-04-21T08:59:44.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f1be0640 1 -- 172.21.15.28:0/761013774 learned_addr learned my addr 172.21.15.28:0/761013774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:44.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f13df640 1 --2- 172.21.15.28:0/761013774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a84580 0x5555c7a8aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f0bde640 1 --2- 172.21.15.28:0/761013774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a84ea0 0x5555c7a8b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f1be0640 1 -- 172.21.15.28:0/761013774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a84580 msgr2=0x5555c7a8aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:44.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f1be0640 1 --2- 172.21.15.28:0/761013774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a84580 0x5555c7a8aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f1be0640 1 -- 172.21.15.28:0/761013774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a84ea0 msgr2=0x5555c7a8b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:44.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f1be0640 1 --2- 172.21.15.28:0/761013774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a84ea0 0x5555c7a8b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f1be0640 1 -- 172.21.15.28:0/761013774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555c76fb3f0 con 0x5555c7a8da80 2024-04-21T08:59:44.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f13df640 1 --2- 172.21.15.28:0/761013774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a84580 0x5555c7a8aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:44.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f1be0640 1 --2- 172.21.15.28:0/761013774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a8da80 0x5555c7a8b560 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f70e402c1c0 tx=0x7f70e4004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b3d49f01c929d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70d4ff9640 1 -- 172.21.15.28:0/761013774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e405a060 con 0x5555c7a8da80 2024-04-21T08:59:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70d4ff9640 1 -- 172.21.15.28:0/761013774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7f70e4032970 con 0x5555c7a8da80 2024-04-21T08:59:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f623c740 1 -- 172.21.15.28:0/761013774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70e4032bb0 con 0x5555c7a8da80 2024-04-21T08:59:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70d4ff9640 1 -- 172.21.15.28:0/761013774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f70e4061050 con 0x5555c7a8da80 2024-04-21T08:59:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.548+0000 7f70f623c740 10 client.17793 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.549+0000 7f70f623c740 1 -- 172.21.15.28:0/761013774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5555c76bb0f0 con 0x5555c7a8da80 2024-04-21T08:59:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.549+0000 7f70d4ff9640 1 -- 172.21.15.28:0/761013774 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70e4032bb0 con 0x5555c7a8da80 2024-04-21T08:59:44.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.549+0000 7f70d4ff9640 1 -- 172.21.15.28:0/761013774 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f70e4043a00 con 0x5555c7a8da80 2024-04-21T08:59:44.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.549+0000 7f70f623c740 10 client.17793 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:44.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.549+0000 7f70f623c740 10 client.17793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:44.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.549+0000 7f70f623c740 20 client.17793 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:44.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.549+0000 7f70f623c740 1 --2- 172.21.15.28:0/761013774 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555c7a9d750 0x5555c7a9fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.549+0000 7f70f623c740 4 client.17793 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:59:44.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.549+0000 7f70f623c740 1 -- 172.21.15.28:0/761013774 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555c79f6c00 con 0x5555c7a9d750 2024-04-21T08:59:44.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.549+0000 7f70f13df640 1 --2- 172.21.15.28:0/761013774 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555c7a9d750 0x5555c7a9fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.550+0000 7f70f13df640 1 --2- 172.21.15.28:0/761013774 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555c7a9d750 0x5555c7a9fb40 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4db13e662857ad5 server_cookie=ae4a2bcdd5ddd384 in_seq=0 out_seq=0 2024-04-21T08:59:44.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.550+0000 7f70d4ff9640 10 client.17793 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:44.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.550+0000 7f70d4ff9640 1 -- 172.21.15.28:0/761013774 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5555c79f6c00 con 0x5555c7a9d750 2024-04-21T08:59:44.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.550+0000 7f70d4ff9640 10 client.17793 handle_command_reply: tid=0 2024-04-21T08:59:44.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.550+0000 7f70f623c740 1 client.17793 shutdown 2024-04-21T08:59:44.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.550+0000 7f70f623c740 1 -- 172.21.15.28:0/761013774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a8da80 msgr2=0x5555c7a8b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:44.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.551+0000 7f70f623c740 1 --2- 172.21.15.28:0/761013774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a8da80 0x5555c7a8b560 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f70e402c1c0 tx=0x7f70e4004960 comp rx=0 tx=0).stop 2024-04-21T08:59:44.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.551+0000 7f70f623c740 1 -- 172.21.15.28:0/761013774 shutdown_connections 2024-04-21T08:59:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.551+0000 7f70f623c740 1 --2- 172.21.15.28:0/761013774 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555c7a9d750 0x5555c7a9fb40 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.551+0000 7f70f623c740 1 --2- 172.21.15.28:0/761013774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a8da80 0x5555c7a8b560 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.551+0000 7f70f623c740 1 --2- 172.21.15.28:0/761013774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a84ea0 0x5555c7a8b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.551+0000 7f70d4ff9640 0 client.17793 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.551+0000 7f70f623c740 1 --2- 172.21.15.28:0/761013774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a84580 0x5555c7a8aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.551+0000 7f70f623c740 1 -- 172.21.15.28:0/761013774 >> 172.21.15.28:0/761013774 conn(0x5555c7a7ecb0 msgr2=0x5555c7a80a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.551+0000 7f70f623c740 1 -- 172.21.15.28:0/761013774 shutdown_connections 2024-04-21T08:59:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.551+0000 7f70f623c740 1 -- 172.21.15.28:0/761013774 wait complete. 2024-04-21T08:59:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.551+0000 7f70f623c740 20 client.17793 trim_cache size 0 max 16384 2024-04-21T08:59:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f623c740 1 Processor -- start 2024-04-21T08:59:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f623c740 1 -- start start 2024-04-21T08:59:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f623c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a84580 0x5555c7a80970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f623c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84ea0 0x5555c7a8b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f623c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a8da80 0x5555c7a7dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f623c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec1b3dd0 con 0x5555c7a84ea0 2024-04-21T08:59:44.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f623c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec1b16a0 con 0x5555c7a84580 2024-04-21T08:59:44.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f623c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec1af510 con 0x5555c7a8da80 2024-04-21T08:59:44.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f0bde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84ea0 0x5555c7a8b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f0bde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84ea0 0x5555c7a8b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46172/0 (socket says 172.21.15.28:46172) 2024-04-21T08:59:44.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f0bde640 1 -- 172.21.15.28:0/2405399860 learned_addr learned my addr 172.21.15.28:0/2405399860 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:44.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f13df640 1 --2- 172.21.15.28:0/2405399860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a84580 0x5555c7a80970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f1be0640 1 --2- 172.21.15.28:0/2405399860 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a8da80 0x5555c7a7dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f0bde640 1 -- 172.21.15.28:0/2405399860 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a8da80 msgr2=0x5555c7a7dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:44.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f0bde640 1 --2- 172.21.15.28:0/2405399860 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a8da80 0x5555c7a7dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f0bde640 1 -- 172.21.15.28:0/2405399860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a84580 msgr2=0x5555c7a80970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f0bde640 1 --2- 172.21.15.28:0/2405399860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a84580 0x5555c7a80970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.569+0000 7f70f0bde640 1 -- 172.21.15.28:0/2405399860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555c7a89050 con 0x5555c7a84ea0 2024-04-21T08:59:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.570+0000 7f70f0bde640 1 --2- 172.21.15.28:0/2405399860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84ea0 0x5555c7a8b380 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f70dc010a70 tx=0x7f70dc010f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=575c15915fa910c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.570+0000 7f70d6ffd640 1 -- 172.21.15.28:0/2405399860 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70dc004030 con 0x5555c7a84ea0 2024-04-21T08:59:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.570+0000 7f70d6ffd640 1 -- 172.21.15.28:0/2405399860 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70dc0054d0 con 0x5555c7a84ea0 2024-04-21T08:59:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.570+0000 7f70d6ffd640 1 -- 172.21.15.28:0/2405399860 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70dc004e10 con 0x5555c7a84ea0 2024-04-21T08:59:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.570+0000 7f70f623c740 1 -- 172.21.15.28:0/2405399860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84ea0 msgr2=0x5555c7a8b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.570+0000 7f70f623c740 1 --2- 172.21.15.28:0/2405399860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84ea0 0x5555c7a8b380 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f70dc010a70 tx=0x7f70dc010f30 comp rx=0 tx=0).stop 2024-04-21T08:59:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.570+0000 7f70f623c740 1 -- 172.21.15.28:0/2405399860 shutdown_connections 2024-04-21T08:59:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.571+0000 7f70f623c740 1 --2- 172.21.15.28:0/2405399860 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a8da80 0x5555c7a7dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.571+0000 7f70f623c740 1 --2- 172.21.15.28:0/2405399860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84ea0 0x5555c7a8b380 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.571+0000 7f70f623c740 1 --2- 172.21.15.28:0/2405399860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a84580 0x5555c7a80970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.571+0000 7f70f623c740 1 -- 172.21.15.28:0/2405399860 >> 172.21.15.28:0/2405399860 conn(0x5555c7a7ecb0 msgr2=0x5555c7a7d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.571+0000 7f70f623c740 1 -- 172.21.15.28:0/2405399860 shutdown_connections 2024-04-21T08:59:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.571+0000 7f70f623c740 1 -- 172.21.15.28:0/2405399860 wait complete. 2024-04-21T08:59:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.571+0000 7f70f623c740 1 -- start start 2024-04-21T08:59:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.571+0000 7f70f623c740 1 Processor -- start 2024-04-21T08:59:44.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.571+0000 7f70f623c740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a84580 0x5555c7aa0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.571+0000 7f70f623c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84ea0 0x5555c7aa1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.571+0000 7f70f623c740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a8da80 0x5555c7afc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.571+0000 7f70f623c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec1afb60 con 0x5555c7a84ea0 2024-04-21T08:59:44.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.571+0000 7f70f623c740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec1b0fd0 con 0x5555c7a8da80 2024-04-21T08:59:44.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.571+0000 7f70f623c740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec1b0230 con 0x5555c7a84580 2024-04-21T08:59:44.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.571+0000 7f70f623c740 10 client.0 _ll_register_callbacks cb 0x7f70e4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:44.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70f0bde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84ea0 0x5555c7aa1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70f0bde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84ea0 0x5555c7aa1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46178/0 (socket says 172.21.15.28:46178) 2024-04-21T08:59:44.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70f0bde640 1 -- 172.21.15.28:0/1384511286 learned_addr learned my addr 172.21.15.28:0/1384511286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:44.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70f1be0640 1 --2- 172.21.15.28:0/1384511286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a8da80 0x5555c7afc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70f13df640 1 --2- 172.21.15.28:0/1384511286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a84580 0x5555c7aa0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70f0bde640 1 -- 172.21.15.28:0/1384511286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a84580 msgr2=0x5555c7aa0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:44.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70f0bde640 1 --2- 172.21.15.28:0/1384511286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a84580 0x5555c7aa0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70f0bde640 1 -- 172.21.15.28:0/1384511286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a8da80 msgr2=0x5555c7afc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:44.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70f0bde640 1 --2- 172.21.15.28:0/1384511286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a8da80 0x5555c7afc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70f0bde640 1 -- 172.21.15.28:0/1384511286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555c7afcec0 con 0x5555c7a84ea0 2024-04-21T08:59:44.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70f13df640 1 --2- 172.21.15.28:0/1384511286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a84580 0x5555c7aa0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:44.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70f0bde640 1 --2- 172.21.15.28:0/1384511286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84ea0 0x5555c7aa1350 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f70dc004910 tx=0x7f70dc004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e6488fd9b620ce server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:44.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:44.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70d4ff9640 1 -- 172.21.15.28:0/1384511286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70dc026070 con 0x5555c7a84ea0 2024-04-21T08:59:44.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70d4ff9640 1 -- 172.21.15.28:0/1384511286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7f70dc004c20 con 0x5555c7a84ea0 2024-04-21T08:59:44.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70f623c740 1 -- 172.21.15.28:0/1384511286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70dc0054d0 con 0x5555c7a84ea0 2024-04-21T08:59:44.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70d4ff9640 1 -- 172.21.15.28:0/1384511286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f70dc032050 con 0x5555c7a84ea0 2024-04-21T08:59:44.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70f623c740 10 client.17799 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:44.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.572+0000 7f70f623c740 1 -- 172.21.15.28:0/1384511286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5555c7b04260 con 0x5555c7a84ea0 2024-04-21T08:59:44.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.573+0000 7f70d4ff9640 1 -- 172.21.15.28:0/1384511286 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70dc0054d0 con 0x5555c7a84ea0 2024-04-21T08:59:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.573+0000 7f70d4ff9640 1 -- 172.21.15.28:0/1384511286 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f70dc019070 con 0x5555c7a84ea0 2024-04-21T08:59:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.573+0000 7f70f623c740 10 client.17799 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.573+0000 7f70f623c740 10 client.17799 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.573+0000 7f70f623c740 20 client.17799 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.573+0000 7f70f623c740 1 --2- 172.21.15.28:0/1384511286 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555c7b04950 0x5555c7b06d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.573+0000 7f70f623c740 4 client.17799 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:59:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.573+0000 7f70f623c740 1 -- 172.21.15.28:0/1384511286 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5555c79f0a70 con 0x5555c7b04950 2024-04-21T08:59:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.573+0000 7f70f13df640 1 --2- 172.21.15.28:0/1384511286 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555c7b04950 0x5555c7b06d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70f13df640 1 --2- 172.21.15.28:0/1384511286 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555c7b04950 0x5555c7b06d70 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df35dbed151f909a server_cookie=cf0a7829b4875d5b in_seq=0 out_seq=0 2024-04-21T08:59:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70d4ff9640 10 client.17799 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70d4ff9640 1 -- 172.21.15.28:0/1384511286 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5555c79f0a70 con 0x5555c7b04950 2024-04-21T08:59:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70d4ff9640 10 client.17799 handle_command_reply: tid=0 2024-04-21T08:59:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70f623c740 1 client.17799 shutdown 2024-04-21T08:59:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70f623c740 1 -- 172.21.15.28:0/1384511286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84ea0 msgr2=0x5555c7aa1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70f623c740 1 --2- 172.21.15.28:0/1384511286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84ea0 0x5555c7aa1350 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f70dc004910 tx=0x7f70dc004740 comp rx=0 tx=0).stop 2024-04-21T08:59:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70f623c740 1 -- 172.21.15.28:0/1384511286 shutdown_connections 2024-04-21T08:59:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70f623c740 1 --2- 172.21.15.28:0/1384511286 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5555c7b04950 0x5555c7b06d70 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70f623c740 1 --2- 172.21.15.28:0/1384511286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5555c7a8da80 0x5555c7afc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70f623c740 1 --2- 172.21.15.28:0/1384511286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c7a84ea0 0x5555c7aa1350 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70f623c740 1 --2- 172.21.15.28:0/1384511286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5555c7a84580 0x5555c7aa0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70d4ff9640 0 client.17799 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70f623c740 1 -- 172.21.15.28:0/1384511286 >> 172.21.15.28:0/1384511286 conn(0x5555c7a7ecb0 msgr2=0x5555c7a7dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70f623c740 1 -- 172.21.15.28:0/1384511286 shutdown_connections 2024-04-21T08:59:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70f623c740 1 -- 172.21.15.28:0/1384511286 wait complete. 2024-04-21T08:59:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.574+0000 7f70f623c740 20 client.17799 trim_cache size 0 max 16384 2024-04-21T08:59:44.585 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:59:44.585 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T08:59:44.585 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T08:59:44.585 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:59:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.575+0000 7f70f3643640 1 -- 172.21.15.28:0/2889753333 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f70d80a3d20 msgr2=0x7f70d80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.575+0000 7f70f3643640 1 --2- 172.21.15.28:0/2889753333 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f70d80a3d20 0x7f70d80a61e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f70e404c000 tx=0x7f70e4048000 comp rx=0 tx=0).stop 2024-04-21T08:59:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.575+0000 7f70f3643640 1 -- 172.21.15.28:0/2889753333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70ec074220 msgr2=0x7f70ec1a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.575+0000 7f70f3643640 1 --2- 172.21.15.28:0/2889753333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70ec074220 0x7f70ec1a0430 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f70e800c920 tx=0x7f70e800cde0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.575+0000 7f70f3643640 1 -- 172.21.15.28:0/2889753333 shutdown_connections 2024-04-21T08:59:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.575+0000 7f70f3643640 1 --2- 172.21.15.28:0/2889753333 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f70d80a3d20 0x7f70d80a61e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.575+0000 7f70f3643640 1 --2- 172.21.15.28:0/2889753333 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70ec10e810 0x7f70ec19a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.575+0000 7f70f3643640 1 --2- 172.21.15.28:0/2889753333 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70ec074b40 0x7f70ec1a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.575+0000 7f70f3643640 1 --2- 172.21.15.28:0/2889753333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70ec074220 0x7f70ec1a0430 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.575+0000 7f70f3643640 1 -- 172.21.15.28:0/2889753333 >> 172.21.15.28:0/2889753333 conn(0x7f70ec0793a0 msgr2=0x7f70ec0fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.575+0000 7f70f3643640 1 -- 172.21.15.28:0/2889753333 shutdown_connections 2024-04-21T08:59:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.575+0000 7f70f3643640 1 -- 172.21.15.28:0/2889753333 wait complete. 2024-04-21T08:59:44.588 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:44 smithi028 ceph-mon[28003]: pgmap v263: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:59:44.589 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:59:44.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:59:44.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:44 smithi152 ceph-mon[31401]: pgmap v263: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:59:44.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:44 smithi171 ceph-mon[31358]: pgmap v263: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T08:59:44.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.692+0000 7f9c6eca5640 1 -- 172.21.15.28:0/1830227590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68102b50 msgr2=0x7f9c68106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:44.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.692+0000 7f9c6eca5640 1 --2- 172.21.15.28:0/1830227590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68102b50 0x7f9c68106b90 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009900 tx=0x7f9c5802bc30 comp rx=0 tx=0).stop 2024-04-21T08:59:44.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.692+0000 7f9c6eca5640 1 -- 172.21.15.28:0/1830227590 shutdown_connections 2024-04-21T08:59:44.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.692+0000 7f9c6eca5640 1 --2- 172.21.15.28:0/1830227590 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c68107270 0x7f9c6810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.692+0000 7f9c6eca5640 1 --2- 172.21.15.28:0/1830227590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68102b50 0x7f9c68106b90 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.692+0000 7f9c6eca5640 1 --2- 172.21.15.28:0/1830227590 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c681021a0 0x7f9c68102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.692+0000 7f9c6eca5640 1 -- 172.21.15.28:0/1830227590 >> 172.21.15.28:0/1830227590 conn(0x7f9c680793a0 msgr2=0x7f9c680797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:44.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.692+0000 7f9c6eca5640 1 -- 172.21.15.28:0/1830227590 shutdown_connections 2024-04-21T08:59:44.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.692+0000 7f9c6eca5640 1 -- 172.21.15.28:0/1830227590 wait complete. 2024-04-21T08:59:44.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.692+0000 7f9c6eca5640 1 Processor -- start 2024-04-21T08:59:44.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.692+0000 7f9c6eca5640 1 -- start start 2024-04-21T08:59:44.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c6eca5640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c681021a0 0x7f9c68074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c6eca5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68107270 0x7f9c68074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c6eca5640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c68074cd0 0x7f9c681a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c6eca5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c68112190 con 0x7f9c68107270 2024-04-21T08:59:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c6eca5640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9c68112010 con 0x7f9c681021a0 2024-04-21T08:59:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c6eca5640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9c68112310 con 0x7f9c68074cd0 2024-04-21T08:59:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c67fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68107270 0x7f9c68074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c6ca41640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c681021a0 0x7f9c68074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c67fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68107270 0x7f9c68074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46186/0 (socket says 172.21.15.28:46186) 2024-04-21T08:59:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c67fff640 1 -- 172.21.15.28:0/3313464022 learned_addr learned my addr 172.21.15.28:0/3313464022 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c6d242640 1 --2- 172.21.15.28:0/3313464022 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c68074cd0 0x7f9c681a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c67fff640 1 -- 172.21.15.28:0/3313464022 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c68074cd0 msgr2=0x7f9c681a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c67fff640 1 --2- 172.21.15.28:0/3313464022 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c68074cd0 0x7f9c681a68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c67fff640 1 -- 172.21.15.28:0/3313464022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c681021a0 msgr2=0x7f9c68074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c67fff640 1 --2- 172.21.15.28:0/3313464022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c681021a0 0x7f9c68074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c67fff640 1 -- 172.21.15.28:0/3313464022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c681a6fe0 con 0x7f9c68107270 2024-04-21T08:59:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c67fff640 1 --2- 172.21.15.28:0/3313464022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68107270 0x7f9c68074790 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f9c58036040 tx=0x7f9c5802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.693+0000 7f9c65ffb640 1 -- 172.21.15.28:0/3313464022 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c58043020 con 0x7f9c68107270 2024-04-21T08:59:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.694+0000 7f9c65ffb640 1 -- 172.21.15.28:0/3313464022 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c5802cbe0 con 0x7f9c68107270 2024-04-21T08:59:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.694+0000 7f9c6eca5640 1 -- 172.21.15.28:0/3313464022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c681a7270 con 0x7f9c68107270 2024-04-21T08:59:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.694+0000 7f9c65ffb640 1 -- 172.21.15.28:0/3313464022 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c58003bf0 con 0x7f9c68107270 2024-04-21T08:59:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.694+0000 7f9c6eca5640 1 -- 172.21.15.28:0/3313464022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c681a77c0 con 0x7f9c68107270 2024-04-21T08:59:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.695+0000 7f9c6eca5640 1 -- 172.21.15.28:0/3313464022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c68069230 con 0x7f9c68107270 2024-04-21T08:59:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.696+0000 7f9c65ffb640 1 -- 172.21.15.28:0/3313464022 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9c58046020 con 0x7f9c68107270 2024-04-21T08:59:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.696+0000 7f9c65ffb640 1 --2- 172.21.15.28:0/3313464022 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9c540a3d50 0x7f9c540a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.697+0000 7f9c6ca41640 1 --2- 172.21.15.28:0/3313464022 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9c540a3d50 0x7f9c540a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.697+0000 7f9c65ffb640 1 -- 172.21.15.28:0/3313464022 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7f9c580f2d80 con 0x7f9c68107270 2024-04-21T08:59:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.697+0000 7f9c6ca41640 1 --2- 172.21.15.28:0/3313464022 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9c540a3d50 0x7f9c540a6210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9c60005fd0 tx=0x7f9c60009680 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.701+0000 7f9c65ffb640 1 -- 172.21.15.28:0/3313464022 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9c5804d020 con 0x7f9c68107270 2024-04-21T08:59:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.929+0000 7f9c6eca5640 1 -- 172.21.15.28:0/3313464022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c6810a4a0 con 0x7f9c68107270 2024-04-21T08:59:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.930+0000 7f9c65ffb640 1 -- 172.21.15.28:0/3313464022 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7f9c580bf0e0 con 0x7f9c68107270 2024-04-21T08:59:44.930 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T08:59:44.930 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T08:59:44.930 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T08:59:44.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.931+0000 7f9c6eca5640 1 -- 172.21.15.28:0/3313464022 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9c540a3d50 msgr2=0x7f9c540a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.931+0000 7f9c6eca5640 1 --2- 172.21.15.28:0/3313464022 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9c540a3d50 0x7f9c540a6210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9c60005fd0 tx=0x7f9c60009680 comp rx=0 tx=0).stop 2024-04-21T08:59:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.931+0000 7f9c6eca5640 1 -- 172.21.15.28:0/3313464022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68107270 msgr2=0x7f9c68074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.931+0000 7f9c6eca5640 1 --2- 172.21.15.28:0/3313464022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68107270 0x7f9c68074790 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f9c58036040 tx=0x7f9c5802c7e0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.932+0000 7f9c6eca5640 1 -- 172.21.15.28:0/3313464022 shutdown_connections 2024-04-21T08:59:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.932+0000 7f9c6eca5640 1 --2- 172.21.15.28:0/3313464022 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9c540a3d50 0x7f9c540a6210 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.932+0000 7f9c6eca5640 1 --2- 172.21.15.28:0/3313464022 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c68074cd0 0x7f9c681a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.932+0000 7f9c6eca5640 1 --2- 172.21.15.28:0/3313464022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c68107270 0x7f9c68074790 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.932+0000 7f9c6eca5640 1 --2- 172.21.15.28:0/3313464022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c681021a0 0x7f9c68074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:44.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.932+0000 7f9c6eca5640 1 -- 172.21.15.28:0/3313464022 >> 172.21.15.28:0/3313464022 conn(0x7f9c680793a0 msgr2=0x7f9c680fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:44.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.932+0000 7f9c6eca5640 1 -- 172.21.15.28:0/3313464022 shutdown_connections 2024-04-21T08:59:44.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:44.932+0000 7f9c6eca5640 1 -- 172.21.15.28:0/3313464022 wait complete. 2024-04-21T08:59:44.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T08:59:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.047+0000 7fb5aa093640 1 -- 172.21.15.28:0/1548799045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a4107260 msgr2=0x7fb5a410ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.047+0000 7fb5aa093640 1 --2- 172.21.15.28:0/1548799045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a4107260 0x7fb5a410ee30 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009ca0 tx=0x7fb59c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.047+0000 7fb5aa093640 1 -- 172.21.15.28:0/1548799045 shutdown_connections 2024-04-21T08:59:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.047+0000 7fb5aa093640 1 --2- 172.21.15.28:0/1548799045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a4107260 0x7fb5a410ee30 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.047+0000 7fb5aa093640 1 --2- 172.21.15.28:0/1548799045 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb5a4102b40 0x7fb5a4106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.047+0000 7fb5aa093640 1 --2- 172.21.15.28:0/1548799045 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5a4102190 0x7fb5a4102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.047+0000 7fb5aa093640 1 -- 172.21.15.28:0/1548799045 >> 172.21.15.28:0/1548799045 conn(0x7fb5a40fd990 msgr2=0x7fb5a40ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.047+0000 7fb5aa093640 1 -- 172.21.15.28:0/1548799045 shutdown_connections 2024-04-21T08:59:45.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.047+0000 7fb5aa093640 1 -- 172.21.15.28:0/1548799045 wait complete. 2024-04-21T08:59:45.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.048+0000 7fb5aa093640 1 Processor -- start 2024-04-21T08:59:45.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.048+0000 7fb5aa093640 1 -- start start 2024-04-21T08:59:45.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.048+0000 7fb5aa093640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb5a4102190 0x7fb5a4198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.048+0000 7fb5aa093640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5a4107260 0x7fb5a4198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.048+0000 7fb5aa093640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a419cfd0 0x7fb5a419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.048+0000 7fb5aa093640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a41121b0 con 0x7fb5a419cfd0 2024-04-21T08:59:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.048+0000 7fb5aa093640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a4112030 con 0x7fb5a4102190 2024-04-21T08:59:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.048+0000 7fb5aa093640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a4112330 con 0x7fb5a4107260 2024-04-21T08:59:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.048+0000 7fb5a3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a419cfd0 0x7fb5a419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.048+0000 7fb5a3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a419cfd0 0x7fb5a419d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46210/0 (socket says 172.21.15.28:46210) 2024-04-21T08:59:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.048+0000 7fb5a3fff640 1 -- 172.21.15.28:0/675711189 learned_addr learned my addr 172.21.15.28:0/675711189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.048+0000 7fb5a2ffd640 1 --2- 172.21.15.28:0/675711189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5a4107260 0x7fb5a4198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.048+0000 7fb5a37fe640 1 --2- 172.21.15.28:0/675711189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb5a4102190 0x7fb5a4198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.049+0000 7fb5a3fff640 1 -- 172.21.15.28:0/675711189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5a4107260 msgr2=0x7fb5a4198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.049+0000 7fb5a3fff640 1 --2- 172.21.15.28:0/675711189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5a4107260 0x7fb5a4198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.049+0000 7fb5a3fff640 1 -- 172.21.15.28:0/675711189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb5a4102190 msgr2=0x7fb5a4198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.049+0000 7fb5a3fff640 1 --2- 172.21.15.28:0/675711189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb5a4102190 0x7fb5a4198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.049+0000 7fb5a3fff640 1 -- 172.21.15.28:0/675711189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a419dac0 con 0x7fb5a419cfd0 2024-04-21T08:59:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.049+0000 7fb5a3fff640 1 --2- 172.21.15.28:0/675711189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a419cfd0 0x7fb5a419d480 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0098d0 tx=0x7fb59c035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.049+0000 7fb5a0ff9640 1 -- 172.21.15.28:0/675711189 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c039040 con 0x7fb5a419cfd0 2024-04-21T08:59:45.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.049+0000 7fb5aa093640 1 -- 172.21.15.28:0/675711189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a419dd50 con 0x7fb5a419cfd0 2024-04-21T08:59:45.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.049+0000 7fb5a0ff9640 1 -- 172.21.15.28:0/675711189 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb59c044070 con 0x7fb5a419cfd0 2024-04-21T08:59:45.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.049+0000 7fb5a0ff9640 1 -- 172.21.15.28:0/675711189 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c0042a0 con 0x7fb5a419cfd0 2024-04-21T08:59:45.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.049+0000 7fb5aa093640 1 -- 172.21.15.28:0/675711189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a41a5510 con 0x7fb5a419cfd0 2024-04-21T08:59:45.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.052+0000 7fb5a0ff9640 1 -- 172.21.15.28:0/675711189 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb59c004440 con 0x7fb5a419cfd0 2024-04-21T08:59:45.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.052+0000 7fb5acd64740 1 Processor -- start 2024-04-21T08:59:45.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.052+0000 7fb5acd64740 1 -- start start 2024-04-21T08:59:45.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.052+0000 7fb5acd64740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6d8250 0x5559ed6d8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.052+0000 7fb5acd64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8c00 0x5559ed6e5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.052+0000 7fb5acd64740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6e5e10 0x5559ed6e8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.052+0000 7fb5acd64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5559ed63d1f0 con 0x5559ed6d8c00 2024-04-21T08:59:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.052+0000 7fb5acd64740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5559ed5da4b0 con 0x5559ed6e5e10 2024-04-21T08:59:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.052+0000 7fb5acd64740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5559ed6391b0 con 0x5559ed6d8250 2024-04-21T08:59:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.052+0000 7fb5a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8c00 0x5559ed6e5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.052+0000 7fb5a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8c00 0x5559ed6e5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46218/0 (socket says 172.21.15.28:46218) 2024-04-21T08:59:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.052+0000 7fb5a37fe640 1 -- 172.21.15.28:0/1942711119 learned_addr learned my addr 172.21.15.28:0/1942711119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.053+0000 7fb5a2ffd640 1 --2- 172.21.15.28:0/1942711119 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6d8250 0x5559ed6d8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.053+0000 7fb5a2ffd640 1 --2- 172.21.15.28:0/1942711119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6e5e10 0x5559ed6e8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.053+0000 7fb5a37fe640 1 -- 172.21.15.28:0/1942711119 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6d8250 msgr2=0x5559ed6d8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.053+0000 7fb5a37fe640 1 --2- 172.21.15.28:0/1942711119 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6d8250 0x5559ed6d8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.053+0000 7fb5a37fe640 1 -- 172.21.15.28:0/1942711119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6e5e10 msgr2=0x5559ed6e8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.053+0000 7fb5a37fe640 1 --2- 172.21.15.28:0/1942711119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6e5e10 0x5559ed6e8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.053+0000 7fb5a37fe640 1 -- 172.21.15.28:0/1942711119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559ed5cc1b0 con 0x5559ed6d8c00 2024-04-21T08:59:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.053+0000 7fb5a0ff9640 1 --2- 172.21.15.28:0/675711189 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb58c0a3d20 0x7fb58c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.053+0000 7fb5a37fe640 1 --2- 172.21.15.28:0/1942711119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8c00 0x5559ed6e5840 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fb59400cbf0 tx=0x7fb59400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a7d5afa6dcb2acb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.053+0000 7fb5a2ffd640 1 --2- 172.21.15.28:0/675711189 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb58c0a3d20 0x7fb58c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.053+0000 7fb5a0ff9640 1 -- 172.21.15.28:0/675711189 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7fb59c0f3100 con 0x7fb5a419cfd0 2024-04-21T08:59:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.053+0000 7fb589ffb640 1 -- 172.21.15.28:0/1942711119 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb594010630 con 0x5559ed6d8c00 2024-04-21T08:59:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.053+0000 7fb589ffb640 1 -- 172.21.15.28:0/1942711119 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb594010c80 con 0x5559ed6d8c00 2024-04-21T08:59:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.053+0000 7fb589ffb640 1 -- 172.21.15.28:0/1942711119 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb594019b40 con 0x5559ed6d8c00 2024-04-21T08:59:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.054+0000 7fb5acd64740 1 -- 172.21.15.28:0/1942711119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8c00 msgr2=0x5559ed6e5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.054+0000 7fb5acd64740 1 --2- 172.21.15.28:0/1942711119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8c00 0x5559ed6e5840 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fb59400cbf0 tx=0x7fb59400e720 comp rx=0 tx=0).stop 2024-04-21T08:59:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.054+0000 7fb5a2ffd640 1 --2- 172.21.15.28:0/675711189 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb58c0a3d20 0x7fb58c0a61e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb598005fd0 tx=0x7fb5980046c0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.054+0000 7fb5acd64740 1 -- 172.21.15.28:0/1942711119 shutdown_connections 2024-04-21T08:59:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.054+0000 7fb5acd64740 1 --2- 172.21.15.28:0/1942711119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6e5e10 0x5559ed6e8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.054+0000 7fb5acd64740 1 --2- 172.21.15.28:0/1942711119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8c00 0x5559ed6e5840 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.054+0000 7fb5acd64740 1 --2- 172.21.15.28:0/1942711119 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6d8250 0x5559ed6d8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.054+0000 7fb5acd64740 1 -- 172.21.15.28:0/1942711119 >> 172.21.15.28:0/1942711119 conn(0x5559ed6d2ba0 msgr2=0x5559ed6d2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.054+0000 7fb5acd64740 1 -- 172.21.15.28:0/1942711119 shutdown_connections 2024-04-21T08:59:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.054+0000 7fb5acd64740 1 -- 172.21.15.28:0/1942711119 wait complete. 2024-04-21T08:59:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.054+0000 7fb5acd64740 1 -- start start 2024-04-21T08:59:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.054+0000 7fb5acd64740 1 Processor -- start 2024-04-21T08:59:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5acd64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8250 0x5559ed6e5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5acd64740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6d8c00 0x5559ed6e2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5acd64740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 0x5559ed6e2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5acd64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5559ed6e8d40 con 0x5559ed6d8250 2024-04-21T08:59:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5acd64740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5559ed5f6500 con 0x5559ed6d8c00 2024-04-21T08:59:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5acd64740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5559ed6e8ec0 con 0x5559ed6e5e10 2024-04-21T08:59:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5acd64740 10 client.0 _ll_register_callbacks cb 0x5559ed5c4b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8250 0x5559ed6e5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8250 0x5559ed6e5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46226/0 (socket says 172.21.15.28:46226) 2024-04-21T08:59:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5a37fe640 1 -- 172.21.15.28:0/3109159716 learned_addr learned my addr 172.21.15.28:0/3109159716 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5a2ffd640 1 --2- 172.21.15.28:0/3109159716 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6d8c00 0x5559ed6e2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5a3fff640 1 --2- 172.21.15.28:0/3109159716 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 0x5559ed6e2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5a37fe640 1 -- 172.21.15.28:0/3109159716 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 msgr2=0x5559ed6e2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5a37fe640 1 --2- 172.21.15.28:0/3109159716 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 0x5559ed6e2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5a37fe640 1 -- 172.21.15.28:0/3109159716 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6d8c00 msgr2=0x5559ed6e2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5a37fe640 1 --2- 172.21.15.28:0/3109159716 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6d8c00 0x5559ed6e2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5a37fe640 1 -- 172.21.15.28:0/3109159716 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559ed34f3f0 con 0x5559ed6d8250 2024-04-21T08:59:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5a3fff640 1 --2- 172.21.15.28:0/3109159716 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 0x5559ed6e2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5a37fe640 1 --2- 172.21.15.28:0/3109159716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8250 0x5559ed6e5680 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fb5940058c0 tx=0x7fb5940198c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d64ddbd7ca16743 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb582ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb582ffd640 1 -- 172.21.15.28:0/3109159716 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb594019c70 con 0x5559ed6d8250 2024-04-21T08:59:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb582ffd640 1 -- 172.21.15.28:0/3109159716 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7fb594019e10 con 0x5559ed6d8250 2024-04-21T08:59:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.055+0000 7fb5acd64740 1 -- 172.21.15.28:0/3109159716 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb594010c80 con 0x5559ed6d8250 2024-04-21T08:59:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.056+0000 7fb582ffd640 1 -- 172.21.15.28:0/3109159716 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb594005050 con 0x5559ed6d8250 2024-04-21T08:59:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.056+0000 7fb5acd64740 10 client.17817 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.056+0000 7fb5acd64740 1 -- 172.21.15.28:0/3109159716 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5559ed4197d0 con 0x5559ed6d8250 2024-04-21T08:59:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.056+0000 7fb582ffd640 1 -- 172.21.15.28:0/3109159716 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb594010c80 con 0x5559ed6d8250 2024-04-21T08:59:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.056+0000 7fb582ffd640 1 -- 172.21.15.28:0/3109159716 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fb5940046c0 con 0x5559ed6d8250 2024-04-21T08:59:45.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.056+0000 7fb5acd64740 10 client.17817 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:45.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.056+0000 7fb5acd64740 10 client.17817 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:59:45.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.056+0000 7fb5acd64740 20 client.17817 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:45.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.056+0000 7fb5acd64740 1 --2- 172.21.15.28:0/3109159716 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5559ed6f5f50 0x5559ed6f8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.056+0000 7fb5acd64740 4 client.17817 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:59:45.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.056+0000 7fb5acd64740 1 -- 172.21.15.28:0/3109159716 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559ed5dc930 con 0x5559ed6f5f50 2024-04-21T08:59:45.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.056+0000 7fb5a2ffd640 1 --2- 172.21.15.28:0/3109159716 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5559ed6f5f50 0x5559ed6f8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.057+0000 7fb5a2ffd640 1 --2- 172.21.15.28:0/3109159716 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5559ed6f5f50 0x5559ed6f8340 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=66dea1d3292b71c1 server_cookie=ca051b8c82963bd3 in_seq=0 out_seq=0 2024-04-21T08:59:45.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.057+0000 7fb582ffd640 10 client.17817 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:59:45.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.058+0000 7fb582ffd640 1 -- 172.21.15.28:0/3109159716 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5559ed5dc930 con 0x5559ed6f5f50 2024-04-21T08:59:45.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.058+0000 7fb582ffd640 10 client.17817 handle_command_reply: tid=0 2024-04-21T08:59:45.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.058+0000 7fb5acd64740 1 client.17817 shutdown 2024-04-21T08:59:45.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.058+0000 7fb5acd64740 1 -- 172.21.15.28:0/3109159716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8250 msgr2=0x5559ed6e5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.058+0000 7fb5acd64740 1 --2- 172.21.15.28:0/3109159716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8250 0x5559ed6e5680 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fb5940058c0 tx=0x7fb5940198c0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.058+0000 7fb5acd64740 1 -- 172.21.15.28:0/3109159716 shutdown_connections 2024-04-21T08:59:45.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.058+0000 7fb5acd64740 1 --2- 172.21.15.28:0/3109159716 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5559ed6f5f50 0x5559ed6f8340 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.058+0000 7fb5acd64740 1 --2- 172.21.15.28:0/3109159716 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 0x5559ed6e2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.058+0000 7fb5acd64740 1 --2- 172.21.15.28:0/3109159716 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6d8c00 0x5559ed6e2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.058+0000 7fb5acd64740 1 --2- 172.21.15.28:0/3109159716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8250 0x5559ed6e5680 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.058+0000 7fb582ffd640 0 client.17817 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:59:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.058+0000 7fb5acd64740 1 -- 172.21.15.28:0/3109159716 >> 172.21.15.28:0/3109159716 conn(0x5559ed6d2ba0 msgr2=0x5559ed6d48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.058+0000 7fb5acd64740 1 -- 172.21.15.28:0/3109159716 shutdown_connections 2024-04-21T08:59:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.058+0000 7fb5acd64740 1 -- 172.21.15.28:0/3109159716 wait complete. 2024-04-21T08:59:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.058+0000 7fb5acd64740 20 client.17817 trim_cache size 0 max 16384 2024-04-21T08:59:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.071+0000 7fb5acd64740 1 Processor -- start 2024-04-21T08:59:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.071+0000 7fb5acd64740 1 -- start start 2024-04-21T08:59:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.071+0000 7fb5acd64740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6d8250 0x5559ed6f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.071+0000 7fb5acd64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8c00 0x5559ed6d1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.071+0000 7fb5acd64740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 0x5559ed6e5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.071+0000 7fb5acd64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a41afa90 con 0x5559ed6d8c00 2024-04-21T08:59:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.071+0000 7fb5acd64740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a41ad360 con 0x5559ed6d8250 2024-04-21T08:59:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.071+0000 7fb5acd64740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a41ab1d0 con 0x5559ed6e5e10 2024-04-21T08:59:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.072+0000 7fb5a2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8c00 0x5559ed6d1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.072+0000 7fb5a2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8c00 0x5559ed6d1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46238/0 (socket says 172.21.15.28:46238) 2024-04-21T08:59:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.072+0000 7fb5a2ffd640 1 -- 172.21.15.28:0/3644670685 learned_addr learned my addr 172.21.15.28:0/3644670685 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.072+0000 7fb5a2ffd640 1 -- 172.21.15.28:0/3644670685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 msgr2=0x5559ed6e5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.072+0000 7fb5a2ffd640 1 --2- 172.21.15.28:0/3644670685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 0x5559ed6e5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.072+0000 7fb5a2ffd640 1 -- 172.21.15.28:0/3644670685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6d8250 msgr2=0x5559ed6f1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.072+0000 7fb5a2ffd640 1 --2- 172.21.15.28:0/3644670685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6d8250 0x5559ed6f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.072+0000 7fb5a2ffd640 1 -- 172.21.15.28:0/3644670685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559ed6f8970 con 0x5559ed6d8c00 2024-04-21T08:59:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.072+0000 7fb5a2ffd640 1 --2- 172.21.15.28:0/3644670685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8c00 0x5559ed6d1b40 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fb5980058c0 tx=0x7fb598014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6eace59877b41c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.072+0000 7fb5897fa640 1 -- 172.21.15.28:0/3644670685 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59802f070 con 0x5559ed6d8c00 2024-04-21T08:59:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.072+0000 7fb5897fa640 1 -- 172.21.15.28:0/3644670685 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb59802a670 con 0x5559ed6d8c00 2024-04-21T08:59:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.072+0000 7fb5897fa640 1 -- 172.21.15.28:0/3644670685 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59801a5f0 con 0x5559ed6d8c00 2024-04-21T08:59:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 -- 172.21.15.28:0/3644670685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8c00 msgr2=0x5559ed6d1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 --2- 172.21.15.28:0/3644670685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8c00 0x5559ed6d1b40 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fb5980058c0 tx=0x7fb598014000 comp rx=0 tx=0).stop 2024-04-21T08:59:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 -- 172.21.15.28:0/3644670685 shutdown_connections 2024-04-21T08:59:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 --2- 172.21.15.28:0/3644670685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 0x5559ed6e5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 --2- 172.21.15.28:0/3644670685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8c00 0x5559ed6d1b40 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 --2- 172.21.15.28:0/3644670685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6d8250 0x5559ed6f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 -- 172.21.15.28:0/3644670685 >> 172.21.15.28:0/3644670685 conn(0x5559ed6d2ba0 msgr2=0x5559ed6de070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 -- 172.21.15.28:0/3644670685 shutdown_connections 2024-04-21T08:59:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 -- 172.21.15.28:0/3644670685 wait complete. 2024-04-21T08:59:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 -- start start 2024-04-21T08:59:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 Processor -- start 2024-04-21T08:59:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8250 0x5559ed6ee570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6d8c00 0x5559ed6df420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 0x5559ed6df960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a41ab820 con 0x5559ed6d8250 2024-04-21T08:59:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a41acc90 con 0x5559ed6d8c00 2024-04-21T08:59:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.073+0000 7fb5acd64740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a41abef0 con 0x5559ed6e5e10 2024-04-21T08:59:45.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5acd64740 10 client.0 _ll_register_callbacks cb 0x7fb594010ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8250 0x5559ed6ee570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8250 0x5559ed6ee570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46254/0 (socket says 172.21.15.28:46254) 2024-04-21T08:59:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5a37fe640 1 -- 172.21.15.28:0/1981180187 learned_addr learned my addr 172.21.15.28:0/1981180187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5a3fff640 1 --2- 172.21.15.28:0/1981180187 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 0x5559ed6df960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5a2ffd640 1 --2- 172.21.15.28:0/1981180187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6d8c00 0x5559ed6df420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5a37fe640 1 -- 172.21.15.28:0/1981180187 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 msgr2=0x5559ed6df960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5a37fe640 1 --2- 172.21.15.28:0/1981180187 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 0x5559ed6df960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5a37fe640 1 -- 172.21.15.28:0/1981180187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6d8c00 msgr2=0x5559ed6df420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5a37fe640 1 --2- 172.21.15.28:0/1981180187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6d8c00 0x5559ed6df420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5a37fe640 1 -- 172.21.15.28:0/1981180187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559ed6d7560 con 0x5559ed6d8250 2024-04-21T08:59:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5a3fff640 1 --2- 172.21.15.28:0/1981180187 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 0x5559ed6df960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5a2ffd640 1 --2- 172.21.15.28:0/1981180187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6d8c00 0x5559ed6df420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5a37fe640 1 --2- 172.21.15.28:0/1981180187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8250 0x5559ed6ee570 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fb594005d90 tx=0x7fb59400efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9111861c66b3ba16 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb582ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb582ffd640 1 -- 172.21.15.28:0/1981180187 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb594038060 con 0x5559ed6d8250 2024-04-21T08:59:45.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb582ffd640 1 -- 172.21.15.28:0/1981180187 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7fb594004da0 con 0x5559ed6d8250 2024-04-21T08:59:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5acd64740 1 -- 172.21.15.28:0/1981180187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb59802a670 con 0x5559ed6d8250 2024-04-21T08:59:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb582ffd640 1 -- 172.21.15.28:0/1981180187 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb59403f050 con 0x5559ed6d8250 2024-04-21T08:59:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5acd64740 10 client.17823 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.074+0000 7fb5acd64740 1 -- 172.21.15.28:0/1981180187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5559ed75c240 con 0x5559ed6d8250 2024-04-21T08:59:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.075+0000 7fb582ffd640 1 -- 172.21.15.28:0/1981180187 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb59802a670 con 0x5559ed6d8250 2024-04-21T08:59:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.075+0000 7fb582ffd640 1 -- 172.21.15.28:0/1981180187 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fb594010430 con 0x5559ed6d8250 2024-04-21T08:59:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.075+0000 7fb5acd64740 10 client.17823 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.075+0000 7fb5acd64740 10 client.17823 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T08:59:45.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.075+0000 7fb5acd64740 20 client.17823 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.075+0000 7fb5acd64740 1 --2- 172.21.15.28:0/1981180187 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5559ed75c960 0x5559ed75ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.075+0000 7fb5acd64740 4 client.17823 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:59:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.075+0000 7fb5acd64740 1 -- 172.21.15.28:0/1981180187 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5559ed6e32e0 con 0x5559ed75c960 2024-04-21T08:59:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.075+0000 7fb5a2ffd640 1 --2- 172.21.15.28:0/1981180187 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5559ed75c960 0x5559ed75ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.075+0000 7fb582ffd640 10 client.17823 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T08:59:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.075+0000 7fb5a2ffd640 1 --2- 172.21.15.28:0/1981180187 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5559ed75c960 0x5559ed75ed80 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bc7e5abd24f801d1 server_cookie=4a82591c3d48691e in_seq=0 out_seq=0 2024-04-21T08:59:45.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb582ffd640 1 -- 172.21.15.28:0/1981180187 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5559ed6e32e0 con 0x5559ed75c960 2024-04-21T08:59:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb582ffd640 10 client.17823 handle_command_reply: tid=0 2024-04-21T08:59:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5acd64740 1 client.17823 shutdown 2024-04-21T08:59:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5acd64740 1 -- 172.21.15.28:0/1981180187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8250 msgr2=0x5559ed6ee570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5acd64740 1 --2- 172.21.15.28:0/1981180187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8250 0x5559ed6ee570 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fb594005d90 tx=0x7fb59400efd0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5acd64740 1 -- 172.21.15.28:0/1981180187 shutdown_connections 2024-04-21T08:59:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5acd64740 1 --2- 172.21.15.28:0/1981180187 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x5559ed75c960 0x5559ed75ed80 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5acd64740 1 --2- 172.21.15.28:0/1981180187 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5559ed6e5e10 0x5559ed6df960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5acd64740 1 --2- 172.21.15.28:0/1981180187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5559ed6d8c00 0x5559ed6df420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb582ffd640 0 client.17823 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T08:59:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5acd64740 1 --2- 172.21.15.28:0/1981180187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5559ed6d8250 0x5559ed6ee570 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5acd64740 1 -- 172.21.15.28:0/1981180187 >> 172.21.15.28:0/1981180187 conn(0x5559ed6d2ba0 msgr2=0x5559ed6e1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5acd64740 1 -- 172.21.15.28:0/1981180187 shutdown_connections 2024-04-21T08:59:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5acd64740 1 -- 172.21.15.28:0/1981180187 wait complete. 2024-04-21T08:59:45.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5acd64740 20 client.17823 trim_cache size 0 max 16384 2024-04-21T08:59:45.086 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:59:45.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5aa093640 1 -- 172.21.15.28:0/675711189 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb58c0a3d20 msgr2=0x7fb58c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5aa093640 1 --2- 172.21.15.28:0/675711189 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb58c0a3d20 0x7fb58c0a61e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb598005fd0 tx=0x7fb5980046c0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5aa093640 1 -- 172.21.15.28:0/675711189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a419cfd0 msgr2=0x7fb5a419d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5aa093640 1 --2- 172.21.15.28:0/675711189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a419cfd0 0x7fb5a419d480 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0098d0 tx=0x7fb59c035d60 comp rx=0 tx=0).stop 2024-04-21T08:59:45.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5aa093640 1 -- 172.21.15.28:0/675711189 shutdown_connections 2024-04-21T08:59:45.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5aa093640 1 --2- 172.21.15.28:0/675711189 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fb58c0a3d20 0x7fb58c0a61e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5aa093640 1 --2- 172.21.15.28:0/675711189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5a419cfd0 0x7fb5a419d480 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5aa093640 1 --2- 172.21.15.28:0/675711189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5a4107260 0x7fb5a4198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5aa093640 1 --2- 172.21.15.28:0/675711189 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb5a4102190 0x7fb5a4198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.076+0000 7fb5aa093640 1 -- 172.21.15.28:0/675711189 >> 172.21.15.28:0/675711189 conn(0x7fb5a40fd990 msgr2=0x7fb5a40ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.077+0000 7fb5aa093640 1 -- 172.21.15.28:0/675711189 shutdown_connections 2024-04-21T08:59:45.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.077+0000 7fb5aa093640 1 -- 172.21.15.28:0/675711189 wait complete. 2024-04-21T08:59:45.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T08:59:45.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.194+0000 7fe1f3ba4640 1 -- 172.21.15.28:0/2201785339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1ec101670 msgr2=0x7fe1ec10e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.194+0000 7fe1f3ba4640 1 --2- 172.21.15.28:0/2201785339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1ec101670 0x7fe1ec10e3c0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc009900 tx=0x7fe1dc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:59:45.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.194+0000 7fe1f3ba4640 1 -- 172.21.15.28:0/2201785339 shutdown_connections 2024-04-21T08:59:45.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.194+0000 7fe1f3ba4640 1 --2- 172.21.15.28:0/2201785339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1ec10e900 0x7fe1ec110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.194+0000 7fe1f3ba4640 1 --2- 172.21.15.28:0/2201785339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1ec101670 0x7fe1ec10e3c0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.194+0000 7fe1f3ba4640 1 --2- 172.21.15.28:0/2201785339 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe1ec100d50 0x7fe1ec101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.194+0000 7fe1f3ba4640 1 -- 172.21.15.28:0/2201785339 >> 172.21.15.28:0/2201785339 conn(0x7fe1ec077f50 msgr2=0x7fe1ec078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.194+0000 7fe1f3ba4640 1 -- 172.21.15.28:0/2201785339 shutdown_connections 2024-04-21T08:59:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.194+0000 7fe1f3ba4640 1 -- 172.21.15.28:0/2201785339 wait complete. 2024-04-21T08:59:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.194+0000 7fe1f3ba4640 1 Processor -- start 2024-04-21T08:59:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.194+0000 7fe1f3ba4640 1 -- start start 2024-04-21T08:59:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f3ba4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe1ec100d50 0x7fe1ec1a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f3ba4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1ec101670 0x7fe1ec1a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f3ba4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1ec10e900 0x7fe1ec19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f3ba4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ec1143e0 con 0x7fe1ec101670 2024-04-21T08:59:45.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f3ba4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ec114260 con 0x7fe1ec10e900 2024-04-21T08:59:45.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f3ba4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ec114560 con 0x7fe1ec100d50 2024-04-21T08:59:45.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f113f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1ec101670 0x7fe1ec1a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f113f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1ec101670 0x7fe1ec1a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46284/0 (socket says 172.21.15.28:46284) 2024-04-21T08:59:45.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f113f640 1 -- 172.21.15.28:0/312062939 learned_addr learned my addr 172.21.15.28:0/312062939 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:45.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f2141640 1 --2- 172.21.15.28:0/312062939 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1ec10e900 0x7fe1ec19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f1940640 1 --2- 172.21.15.28:0/312062939 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe1ec100d50 0x7fe1ec1a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f113f640 1 -- 172.21.15.28:0/312062939 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe1ec100d50 msgr2=0x7fe1ec1a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f113f640 1 --2- 172.21.15.28:0/312062939 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe1ec100d50 0x7fe1ec1a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f113f640 1 -- 172.21.15.28:0/312062939 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1ec10e900 msgr2=0x7fe1ec19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f113f640 1 --2- 172.21.15.28:0/312062939 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1ec10e900 0x7fe1ec19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f113f640 1 -- 172.21.15.28:0/312062939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1ec19af50 con 0x7fe1ec101670 2024-04-21T08:59:45.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f113f640 1 --2- 172.21.15.28:0/312062939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1ec101670 0x7fe1ec1a0b50 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc005940 tx=0x7fe1dc0040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1e2ffd640 1 -- 172.21.15.28:0/312062939 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc043020 con 0x7fe1ec101670 2024-04-21T08:59:45.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1e2ffd640 1 -- 172.21.15.28:0/312062939 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1dc0052b0 con 0x7fe1ec101670 2024-04-21T08:59:45.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f3ba4640 1 -- 172.21.15.28:0/312062939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1ec19b1e0 con 0x7fe1ec101670 2024-04-21T08:59:45.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1e2ffd640 1 -- 172.21.15.28:0/312062939 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc003d00 con 0x7fe1ec101670 2024-04-21T08:59:45.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.195+0000 7fe1f3ba4640 1 -- 172.21.15.28:0/312062939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1ec1a7460 con 0x7fe1ec101670 2024-04-21T08:59:45.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.198+0000 7fe1e2ffd640 1 -- 172.21.15.28:0/312062939 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe1dc046020 con 0x7fe1ec101670 2024-04-21T08:59:45.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.198+0000 7fe1f679a740 1 Processor -- start 2024-04-21T08:59:45.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.198+0000 7fe1f679a740 1 -- start start 2024-04-21T08:59:45.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.198+0000 7fe1f679a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 0x5567f9a6e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.198+0000 7fe1f679a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a6ee80 0x5567f9a77430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.198+0000 7fe1f679a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a77a00 0x5567f9a79e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.198+0000 7fe1f679a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5567f99d31f0 con 0x5567f9a77a00 2024-04-21T08:59:45.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.198+0000 7fe1f679a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5567f99704b0 con 0x5567f9a6e4d0 2024-04-21T08:59:45.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.198+0000 7fe1f679a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5567f99cf1b0 con 0x5567f9a6ee80 2024-04-21T08:59:45.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.198+0000 7fe1f113f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a77a00 0x5567f9a79e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.198+0000 7fe1f113f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a77a00 0x5567f9a79e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46288/0 (socket says 172.21.15.28:46288) 2024-04-21T08:59:45.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1f113f640 1 -- 172.21.15.28:0/2005808465 learned_addr learned my addr 172.21.15.28:0/2005808465 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:45.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1f2141640 1 --2- 172.21.15.28:0/2005808465 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 0x5567f9a6e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1e2ffd640 1 --2- 172.21.15.28:0/312062939 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe1d80a3da0 0x7fe1d80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1f1940640 1 --2- 172.21.15.28:0/2005808465 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a6ee80 0x5567f9a77430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1f2141640 1 --2- 172.21.15.28:0/312062939 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe1d80a3da0 0x7fe1d80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1e2ffd640 1 -- 172.21.15.28:0/312062939 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7fe1dc0f2e50 con 0x7fe1ec101670 2024-04-21T08:59:45.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1f113f640 1 -- 172.21.15.28:0/2005808465 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a6ee80 msgr2=0x5567f9a77430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1f113f640 1 --2- 172.21.15.28:0/2005808465 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a6ee80 0x5567f9a77430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1f113f640 1 -- 172.21.15.28:0/2005808465 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 msgr2=0x5567f9a6e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1f113f640 1 --2- 172.21.15.28:0/2005808465 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 0x5567f9a6e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1f113f640 1 -- 172.21.15.28:0/2005808465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567f99621b0 con 0x5567f9a77a00 2024-04-21T08:59:45.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1f1940640 1 --2- 172.21.15.28:0/2005808465 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a6ee80 0x5567f9a77430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:45.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1f2141640 1 --2- 172.21.15.28:0/2005808465 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 0x5567f9a6e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:45.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1f113f640 1 --2- 172.21.15.28:0/2005808465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a77a00 0x5567f9a79e50 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0be810 tx=0x7fe1dc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=284638449317fde9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1d7fff640 1 -- 172.21.15.28:0/2005808465 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc0be650 con 0x5567f9a77a00 2024-04-21T08:59:45.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1f2141640 1 --2- 172.21.15.28:0/312062939 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe1d80a3da0 0x7fe1d80a6260 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400a2e0 tx=0x7fe1e4004ab0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1d7fff640 1 -- 172.21.15.28:0/2005808465 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1dc0035f0 con 0x5567f9a77a00 2024-04-21T08:59:45.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.199+0000 7fe1d7fff640 1 -- 172.21.15.28:0/2005808465 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc0044e0 con 0x5567f9a77a00 2024-04-21T08:59:45.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.200+0000 7fe1f679a740 1 -- 172.21.15.28:0/2005808465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a77a00 msgr2=0x5567f9a79e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.200+0000 7fe1f679a740 1 --2- 172.21.15.28:0/2005808465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a77a00 0x5567f9a79e50 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0be810 tx=0x7fe1dc000bc0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.200+0000 7fe1f679a740 1 -- 172.21.15.28:0/2005808465 shutdown_connections 2024-04-21T08:59:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.200+0000 7fe1f679a740 1 --2- 172.21.15.28:0/2005808465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a77a00 0x5567f9a79e50 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.200+0000 7fe1f679a740 1 --2- 172.21.15.28:0/2005808465 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a6ee80 0x5567f9a77430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.200+0000 7fe1f679a740 1 --2- 172.21.15.28:0/2005808465 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 0x5567f9a6e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.200+0000 7fe1f679a740 1 -- 172.21.15.28:0/2005808465 >> 172.21.15.28:0/2005808465 conn(0x5567f9a68ba0 msgr2=0x5567f9a68f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.200+0000 7fe1f679a740 1 -- 172.21.15.28:0/2005808465 shutdown_connections 2024-04-21T08:59:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.200+0000 7fe1f679a740 1 -- 172.21.15.28:0/2005808465 wait complete. 2024-04-21T08:59:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.200+0000 7fe1f679a740 1 -- start start 2024-04-21T08:59:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f679a740 1 Processor -- start 2024-04-21T08:59:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f679a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 0x5567f9a77340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f679a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a6ee80 0x5567f9a746d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f679a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a77a00 0x5567f9a74ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f679a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5567f9a7a730 con 0x5567f9a77a00 2024-04-21T08:59:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f679a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5567f998c500 con 0x5567f9a6e4d0 2024-04-21T08:59:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f679a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5567f9a7a8b0 con 0x5567f9a6ee80 2024-04-21T08:59:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f679a740 10 client.0 _ll_register_callbacks cb 0x5567f995ab80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f2141640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a77a00 0x5567f9a74ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f2141640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a77a00 0x5567f9a74ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46296/0 (socket says 172.21.15.28:46296) 2024-04-21T08:59:45.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f2141640 1 -- 172.21.15.28:0/166842319 learned_addr learned my addr 172.21.15.28:0/166842319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:45.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f113f640 1 --2- 172.21.15.28:0/166842319 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a6ee80 0x5567f9a746d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f1940640 1 --2- 172.21.15.28:0/166842319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 0x5567f9a77340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f2141640 1 -- 172.21.15.28:0/166842319 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a6ee80 msgr2=0x5567f9a746d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f2141640 1 --2- 172.21.15.28:0/166842319 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a6ee80 0x5567f9a746d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f2141640 1 -- 172.21.15.28:0/166842319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 msgr2=0x5567f9a77340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f2141640 1 --2- 172.21.15.28:0/166842319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 0x5567f9a77340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f2141640 1 -- 172.21.15.28:0/166842319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567f96e53f0 con 0x5567f9a77a00 2024-04-21T08:59:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.201+0000 7fe1f1940640 1 --2- 172.21.15.28:0/166842319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 0x5567f9a77340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1f2141640 1 --2- 172.21.15.28:0/166842319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a77a00 0x5567f9a74ce0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4011f70 tx=0x7fe1e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b272153d75db6082 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1d57fa640 1 -- 172.21.15.28:0/166842319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e400fc60 con 0x5567f9a77a00 2024-04-21T08:59:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1d57fa640 1 -- 172.21.15.28:0/166842319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7fe1e400fe00 con 0x5567f9a77a00 2024-04-21T08:59:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1f679a740 1 -- 172.21.15.28:0/166842319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5567f9972930 con 0x5567f9a77a00 2024-04-21T08:59:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1d57fa640 1 -- 172.21.15.28:0/166842319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe1e4027430 con 0x5567f9a77a00 2024-04-21T08:59:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1f679a740 10 client.17835 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1f679a740 1 -- 172.21.15.28:0/166842319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5567f97af7d0 con 0x5567f9a77a00 2024-04-21T08:59:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1d57fa640 1 -- 172.21.15.28:0/166842319 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5567f9972930 con 0x5567f9a77a00 2024-04-21T08:59:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1d57fa640 1 -- 172.21.15.28:0/166842319 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe1e401d680 con 0x5567f9a77a00 2024-04-21T08:59:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1f679a740 10 client.17835 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1f679a740 10 client.17835 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:59:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1f679a740 20 client.17835 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1f679a740 1 --2- 172.21.15.28:0/166842319 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5567f9a879c0 0x5567f9a89e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1f679a740 4 client.17835 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:59:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.202+0000 7fe1f679a740 1 -- 172.21.15.28:0/166842319 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567f99e0c10 con 0x5567f9a879c0 2024-04-21T08:59:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.203+0000 7fe1f1940640 1 --2- 172.21.15.28:0/166842319 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5567f9a879c0 0x5567f9a89e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.203+0000 7fe1f1940640 1 --2- 172.21.15.28:0/166842319 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5567f9a879c0 0x5567f9a89e80 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b9294956a6798345 server_cookie=bdb212fe753488d6 in_seq=0 out_seq=0 2024-04-21T08:59:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.203+0000 7fe1d57fa640 10 client.17835 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:59:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.204+0000 7fe1d57fa640 1 -- 172.21.15.28:0/166842319 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5567f99e0c10 con 0x5567f9a879c0 2024-04-21T08:59:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.204+0000 7fe1d57fa640 10 client.17835 handle_command_reply: tid=0 2024-04-21T08:59:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.204+0000 7fe1f679a740 1 client.17835 shutdown 2024-04-21T08:59:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.204+0000 7fe1f679a740 1 -- 172.21.15.28:0/166842319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a77a00 msgr2=0x5567f9a74ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.204+0000 7fe1f679a740 1 --2- 172.21.15.28:0/166842319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a77a00 0x5567f9a74ce0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fe1e4011f70 tx=0x7fe1e4004a40 comp rx=0 tx=0).stop 2024-04-21T08:59:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.204+0000 7fe1f679a740 1 -- 172.21.15.28:0/166842319 shutdown_connections 2024-04-21T08:59:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.204+0000 7fe1f679a740 1 --2- 172.21.15.28:0/166842319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a77a00 0x5567f9a74ce0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.204+0000 7fe1f679a740 1 --2- 172.21.15.28:0/166842319 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5567f9a879c0 0x5567f9a89e80 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.204+0000 7fe1f679a740 1 --2- 172.21.15.28:0/166842319 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a6ee80 0x5567f9a746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.204+0000 7fe1f679a740 1 --2- 172.21.15.28:0/166842319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 0x5567f9a77340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.204+0000 7fe1d57fa640 0 client.17835 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:59:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.204+0000 7fe1f679a740 1 -- 172.21.15.28:0/166842319 >> 172.21.15.28:0/166842319 conn(0x5567f9a68ba0 msgr2=0x5567f9a6a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.204+0000 7fe1f679a740 1 -- 172.21.15.28:0/166842319 shutdown_connections 2024-04-21T08:59:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.204+0000 7fe1f679a740 1 -- 172.21.15.28:0/166842319 wait complete. 2024-04-21T08:59:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.204+0000 7fe1f679a740 20 client.17835 trim_cache size 0 max 16384 2024-04-21T08:59:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.217+0000 7fe1f679a740 1 Processor -- start 2024-04-21T08:59:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.217+0000 7fe1f679a740 1 -- start start 2024-04-21T08:59:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1f679a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a6e4d0 0x5567f9a83560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1f679a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a6ee80 0x5567f9a76e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1f679a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a77a00 0x5567f9a77340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1f679a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ec1b3ff0 con 0x5567f9a6e4d0 2024-04-21T08:59:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1f679a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ec1b18c0 con 0x5567f9a77a00 2024-04-21T08:59:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1f679a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ec1af730 con 0x5567f9a6ee80 2024-04-21T08:59:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1f1940640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a6e4d0 0x5567f9a83560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1f1940640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a6e4d0 0x5567f9a83560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46302/0 (socket says 172.21.15.28:46302) 2024-04-21T08:59:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1f1940640 1 -- 172.21.15.28:0/1390942972 learned_addr learned my addr 172.21.15.28:0/1390942972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1f1940640 1 -- 172.21.15.28:0/1390942972 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a6ee80 msgr2=0x5567f9a76e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1f1940640 1 --2- 172.21.15.28:0/1390942972 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a6ee80 0x5567f9a76e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1f1940640 1 -- 172.21.15.28:0/1390942972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a77a00 msgr2=0x5567f9a77340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1f1940640 1 --2- 172.21.15.28:0/1390942972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a77a00 0x5567f9a77340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1f1940640 1 -- 172.21.15.28:0/1390942972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567f9a7a160 con 0x5567f9a6e4d0 2024-04-21T08:59:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1f1940640 1 --2- 172.21.15.28:0/1390942972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a6e4d0 0x5567f9a83560 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800ac50 tx=0x7fe1e800a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a565bf8a606fb2bc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1390942972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8013e30 con 0x5567f9a6e4d0 2024-04-21T08:59:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1390942972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1e801b860 con 0x5567f9a6e4d0 2024-04-21T08:59:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.218+0000 7fe1d77fe640 1 -- 172.21.15.28:0/1390942972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8011e90 con 0x5567f9a6e4d0 2024-04-21T08:59:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.219+0000 7fe1f679a740 1 -- 172.21.15.28:0/1390942972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a6e4d0 msgr2=0x5567f9a83560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.219+0000 7fe1f679a740 1 --2- 172.21.15.28:0/1390942972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a6e4d0 0x5567f9a83560 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800ac50 tx=0x7fe1e800a6b0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.219+0000 7fe1f679a740 1 -- 172.21.15.28:0/1390942972 shutdown_connections 2024-04-21T08:59:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.219+0000 7fe1f679a740 1 --2- 172.21.15.28:0/1390942972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a77a00 0x5567f9a77340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.219+0000 7fe1f679a740 1 --2- 172.21.15.28:0/1390942972 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a6ee80 0x5567f9a76e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.219+0000 7fe1f679a740 1 --2- 172.21.15.28:0/1390942972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a6e4d0 0x5567f9a83560 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.219+0000 7fe1f679a740 1 -- 172.21.15.28:0/1390942972 >> 172.21.15.28:0/1390942972 conn(0x5567f9a68ba0 msgr2=0x5567f9a6a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.219+0000 7fe1f679a740 1 -- 172.21.15.28:0/1390942972 shutdown_connections 2024-04-21T08:59:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.219+0000 7fe1f679a740 1 -- 172.21.15.28:0/1390942972 wait complete. 2024-04-21T08:59:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.219+0000 7fe1f679a740 1 -- start start 2024-04-21T08:59:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.219+0000 7fe1f679a740 1 Processor -- start 2024-04-21T08:59:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f679a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 0x5567f9a7b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f679a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a6ee80 0x5567f9a7ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f679a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a77a00 0x5567f9a7bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f679a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ec1afd80 con 0x5567f9a6ee80 2024-04-21T08:59:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f679a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ec1b11f0 con 0x5567f9a6e4d0 2024-04-21T08:59:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f679a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe1ec1b0450 con 0x5567f9a77a00 2024-04-21T08:59:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f679a740 10 client.0 _ll_register_callbacks cb 0x7fe1e4011c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f113f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a6ee80 0x5567f9a7ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f113f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a6ee80 0x5567f9a7ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46316/0 (socket says 172.21.15.28:46316) 2024-04-21T08:59:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f113f640 1 -- 172.21.15.28:0/480477622 learned_addr learned my addr 172.21.15.28:0/480477622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f1940640 1 --2- 172.21.15.28:0/480477622 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 0x5567f9a7b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f2141640 1 --2- 172.21.15.28:0/480477622 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a77a00 0x5567f9a7bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f113f640 1 -- 172.21.15.28:0/480477622 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a77a00 msgr2=0x5567f9a7bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f113f640 1 --2- 172.21.15.28:0/480477622 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a77a00 0x5567f9a7bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f113f640 1 -- 172.21.15.28:0/480477622 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 msgr2=0x5567f9a7b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f113f640 1 --2- 172.21.15.28:0/480477622 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 0x5567f9a7b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f113f640 1 -- 172.21.15.28:0/480477622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567f9aed060 con 0x5567f9a6ee80 2024-04-21T08:59:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f1940640 1 --2- 172.21.15.28:0/480477622 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 0x5567f9a7b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f2141640 1 --2- 172.21.15.28:0/480477622 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a77a00 0x5567f9a7bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f113f640 1 --2- 172.21.15.28:0/480477622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a6ee80 0x5567f9a7ba60 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0baf70 tx=0x7fe1dc0056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8290501b96f4944 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1d57fa640 1 -- 172.21.15.28:0/480477622 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc0d8060 con 0x5567f9a6ee80 2024-04-21T08:59:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1d57fa640 1 -- 172.21.15.28:0/480477622 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7fe1dc0bb180 con 0x5567f9a6ee80 2024-04-21T08:59:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.220+0000 7fe1f679a740 1 -- 172.21.15.28:0/480477622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1e801b860 con 0x5567f9a6ee80 2024-04-21T08:59:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.221+0000 7fe1d57fa640 1 -- 172.21.15.28:0/480477622 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe1dc0be470 con 0x5567f9a6ee80 2024-04-21T08:59:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.221+0000 7fe1f679a740 10 client.17841 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.221+0000 7fe1f679a740 1 -- 172.21.15.28:0/480477622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5567f9aee340 con 0x5567f9a6ee80 2024-04-21T08:59:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.221+0000 7fe1d57fa640 1 -- 172.21.15.28:0/480477622 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1e801b860 con 0x5567f9a6ee80 2024-04-21T08:59:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.221+0000 7fe1d57fa640 1 -- 172.21.15.28:0/480477622 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe1dc0c72f0 con 0x5567f9a6ee80 2024-04-21T08:59:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.221+0000 7fe1f679a740 10 client.17841 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.221+0000 7fe1f679a740 10 client.17841 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T08:59:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.221+0000 7fe1f679a740 20 client.17841 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.221+0000 7fe1f679a740 1 --2- 172.21.15.28:0/480477622 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5567f9aee8a0 0x5567f9af0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.221+0000 7fe1f679a740 4 client.17841 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:59:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.221+0000 7fe1f679a740 1 -- 172.21.15.28:0/480477622 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5567f99daa80 con 0x5567f9aee8a0 2024-04-21T08:59:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.221+0000 7fe1f1940640 1 --2- 172.21.15.28:0/480477622 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5567f9aee8a0 0x5567f9af0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1f1940640 1 --2- 172.21.15.28:0/480477622 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5567f9aee8a0 0x5567f9af0cc0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7b6e6dd6b0228c01 server_cookie=46d70e1f06654139 in_seq=0 out_seq=0 2024-04-21T08:59:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1d57fa640 10 client.17841 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T08:59:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1d57fa640 1 -- 172.21.15.28:0/480477622 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5567f99daa80 con 0x5567f9aee8a0 2024-04-21T08:59:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1d57fa640 10 client.17841 handle_command_reply: tid=0 2024-04-21T08:59:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1f679a740 1 client.17841 shutdown 2024-04-21T08:59:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1f679a740 1 -- 172.21.15.28:0/480477622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a6ee80 msgr2=0x5567f9a7ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1f679a740 1 --2- 172.21.15.28:0/480477622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a6ee80 0x5567f9a7ba60 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0baf70 tx=0x7fe1dc0056f0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1f679a740 1 -- 172.21.15.28:0/480477622 shutdown_connections 2024-04-21T08:59:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1f679a740 1 --2- 172.21.15.28:0/480477622 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x5567f9aee8a0 0x5567f9af0cc0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1f679a740 1 --2- 172.21.15.28:0/480477622 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5567f9a77a00 0x5567f9a7bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1f679a740 1 --2- 172.21.15.28:0/480477622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567f9a6ee80 0x5567f9a7ba60 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1f679a740 1 --2- 172.21.15.28:0/480477622 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5567f9a6e4d0 0x5567f9a7b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1d57fa640 0 client.17841 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T08:59:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1f679a740 1 -- 172.21.15.28:0/480477622 >> 172.21.15.28:0/480477622 conn(0x5567f9a68ba0 msgr2=0x5567f9a6f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1f679a740 1 -- 172.21.15.28:0/480477622 shutdown_connections 2024-04-21T08:59:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1f679a740 1 -- 172.21.15.28:0/480477622 wait complete. 2024-04-21T08:59:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.222+0000 7fe1f679a740 20 client.17841 trim_cache size 0 max 16384 2024-04-21T08:59:45.233 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:59:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.223+0000 7fe1f3ba4640 1 -- 172.21.15.28:0/312062939 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe1d80a3da0 msgr2=0x7fe1d80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.223+0000 7fe1f3ba4640 1 --2- 172.21.15.28:0/312062939 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe1d80a3da0 0x7fe1d80a6260 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400a2e0 tx=0x7fe1e4004ab0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.223+0000 7fe1f3ba4640 1 -- 172.21.15.28:0/312062939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1ec101670 msgr2=0x7fe1ec1a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.223+0000 7fe1f3ba4640 1 --2- 172.21.15.28:0/312062939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1ec101670 0x7fe1ec1a0b50 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc005940 tx=0x7fe1dc0040e0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.223+0000 7fe1f3ba4640 1 -- 172.21.15.28:0/312062939 shutdown_connections 2024-04-21T08:59:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.223+0000 7fe1f3ba4640 1 --2- 172.21.15.28:0/312062939 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe1d80a3da0 0x7fe1d80a6260 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.223+0000 7fe1f3ba4640 1 --2- 172.21.15.28:0/312062939 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1ec10e900 0x7fe1ec19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.223+0000 7fe1f3ba4640 1 --2- 172.21.15.28:0/312062939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1ec101670 0x7fe1ec1a0b50 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.223+0000 7fe1f3ba4640 1 --2- 172.21.15.28:0/312062939 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe1ec100d50 0x7fe1ec1a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.223+0000 7fe1f3ba4640 1 -- 172.21.15.28:0/312062939 >> 172.21.15.28:0/312062939 conn(0x7fe1ec077f50 msgr2=0x7fe1ec1100b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.223+0000 7fe1f3ba4640 1 -- 172.21.15.28:0/312062939 shutdown_connections 2024-04-21T08:59:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.223+0000 7fe1f3ba4640 1 -- 172.21.15.28:0/312062939 wait complete. 2024-04-21T08:59:45.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:59:45.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.341+0000 7eff5a43a640 1 -- 172.21.15.28:0/557146509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff54102190 msgr2=0x7eff54102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.341+0000 7eff5a43a640 1 --2- 172.21.15.28:0/557146509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff54102190 0x7eff54102570 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7eff44009900 tx=0x7eff4402bc30 comp rx=0 tx=0).stop 2024-04-21T08:59:45.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.341+0000 7eff5a43a640 1 -- 172.21.15.28:0/557146509 shutdown_connections 2024-04-21T08:59:45.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.341+0000 7eff5a43a640 1 --2- 172.21.15.28:0/557146509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff54107260 0x7eff5410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.341+0000 7eff5a43a640 1 --2- 172.21.15.28:0/557146509 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff54102b40 0x7eff54106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.341+0000 7eff5a43a640 1 --2- 172.21.15.28:0/557146509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff54102190 0x7eff54102570 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.341+0000 7eff5a43a640 1 -- 172.21.15.28:0/557146509 >> 172.21.15.28:0/557146509 conn(0x7eff540793a0 msgr2=0x7eff540797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.341+0000 7eff5a43a640 1 -- 172.21.15.28:0/557146509 shutdown_connections 2024-04-21T08:59:45.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.341+0000 7eff5a43a640 1 -- 172.21.15.28:0/557146509 wait complete. 2024-04-21T08:59:45.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff5a43a640 1 Processor -- start 2024-04-21T08:59:45.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff5a43a640 1 -- start start 2024-04-21T08:59:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff5a43a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff54102190 0x7eff5419e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff5a43a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff54107260 0x7eff5419e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff5a43a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff54198610 0x7eff54198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff5a43a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff54112270 con 0x7eff54107260 2024-04-21T08:59:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff5a43a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7eff541120f0 con 0x7eff54198610 2024-04-21T08:59:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff5a43a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7eff541123f0 con 0x7eff54102190 2024-04-21T08:59:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff537fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff54107260 0x7eff5419e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff537fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff54107260 0x7eff5419e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46338/0 (socket says 172.21.15.28:46338) 2024-04-21T08:59:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff537fe640 1 -- 172.21.15.28:0/2525369944 learned_addr learned my addr 172.21.15.28:0/2525369944 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff53fff640 1 --2- 172.21.15.28:0/2525369944 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff54102190 0x7eff5419e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff537fe640 1 -- 172.21.15.28:0/2525369944 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff54102190 msgr2=0x7eff5419e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff537fe640 1 --2- 172.21.15.28:0/2525369944 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff54102190 0x7eff5419e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff537fe640 1 -- 172.21.15.28:0/2525369944 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff54198610 msgr2=0x7eff54198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.342+0000 7eff537fe640 1 --2- 172.21.15.28:0/2525369944 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff54198610 0x7eff54198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.343+0000 7eff537fe640 1 -- 172.21.15.28:0/2525369944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff54199130 con 0x7eff54107260 2024-04-21T08:59:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.343+0000 7eff53fff640 1 --2- 172.21.15.28:0/2525369944 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff54102190 0x7eff5419e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.343+0000 7eff537fe640 1 --2- 172.21.15.28:0/2525369944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff54107260 0x7eff5419e9a0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7eff48005fd0 tx=0x7eff4800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.343+0000 7eff517fa640 1 -- 172.21.15.28:0/2525369944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff48017070 con 0x7eff54107260 2024-04-21T08:59:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.343+0000 7eff5a43a640 1 -- 172.21.15.28:0/2525369944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff54199420 con 0x7eff54107260 2024-04-21T08:59:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.343+0000 7eff517fa640 1 -- 172.21.15.28:0/2525369944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff48009c30 con 0x7eff54107260 2024-04-21T08:59:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.343+0000 7eff517fa640 1 -- 172.21.15.28:0/2525369944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff48003d30 con 0x7eff54107260 2024-04-21T08:59:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.343+0000 7eff5a43a640 1 -- 172.21.15.28:0/2525369944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff541a56c0 con 0x7eff54107260 2024-04-21T08:59:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.345+0000 7eff517fa640 1 -- 172.21.15.28:0/2525369944 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7eff48003ed0 con 0x7eff54107260 2024-04-21T08:59:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.345+0000 7eff5d020740 1 Processor -- start 2024-04-21T08:59:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.345+0000 7eff5d020740 1 -- start start 2024-04-21T08:59:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.345+0000 7eff5d020740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f45a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.345+0000 7eff5d020740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f45ae80 0x55637f463430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff5d020740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f463a00 0x55637f465e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff5d020740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55637f3bf1f0 con 0x55637f45a4d0 2024-04-21T08:59:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff5d020740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55637f35c4b0 con 0x55637f463a00 2024-04-21T08:59:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff5d020740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55637f3bb1b0 con 0x55637f45ae80 2024-04-21T08:59:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff589d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f45a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff589d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f45a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46340/0 (socket says 172.21.15.28:46340) 2024-04-21T08:59:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff589d7640 1 -- 172.21.15.28:0/1908294130 learned_addr learned my addr 172.21.15.28:0/1908294130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff517fa640 1 --2- 172.21.15.28:0/2525369944 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7eff3c0a3d20 0x7eff3c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff537fe640 1 --2- 172.21.15.28:0/1908294130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f463a00 0x55637f465e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff53fff640 1 --2- 172.21.15.28:0/1908294130 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f45ae80 0x55637f463430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff589d7640 1 -- 172.21.15.28:0/1908294130 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f45ae80 msgr2=0x55637f463430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff589d7640 1 --2- 172.21.15.28:0/1908294130 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f45ae80 0x55637f463430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff589d7640 1 -- 172.21.15.28:0/1908294130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f463a00 msgr2=0x55637f465e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff589d7640 1 --2- 172.21.15.28:0/1908294130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f463a00 0x55637f465e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff589d7640 1 -- 172.21.15.28:0/1908294130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55637f34e1b0 con 0x55637f45a4d0 2024-04-21T08:59:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff53fff640 1 --2- 172.21.15.28:0/1908294130 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f45ae80 0x55637f463430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff537fe640 1 --2- 172.21.15.28:0/1908294130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f463a00 0x55637f465e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff517fa640 1 -- 172.21.15.28:0/2525369944 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7eff480d0950 con 0x7eff54107260 2024-04-21T08:59:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff589d7640 1 --2- 172.21.15.28:0/2525369944 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7eff3c0a3d20 0x7eff3c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff589d7640 1 --2- 172.21.15.28:0/1908294130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f45a8b0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7eff4c009ec0 tx=0x7eff4c00b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d934df12d77dec5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff3a7fc640 1 -- 172.21.15.28:0/1908294130 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4c019040 con 0x55637f45a4d0 2024-04-21T08:59:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff3a7fc640 1 -- 172.21.15.28:0/1908294130 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff4c01a020 con 0x55637f45a4d0 2024-04-21T08:59:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.346+0000 7eff3a7fc640 1 -- 172.21.15.28:0/1908294130 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4c01f850 con 0x55637f45a4d0 2024-04-21T08:59:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.347+0000 7eff5d020740 1 -- 172.21.15.28:0/1908294130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 msgr2=0x55637f45a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.347+0000 7eff589d7640 1 --2- 172.21.15.28:0/2525369944 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7eff3c0a3d20 0x7eff3c0a61e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7eff4c01b000 tx=0x7eff4c01c000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.347+0000 7eff5d020740 1 --2- 172.21.15.28:0/1908294130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f45a8b0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7eff4c009ec0 tx=0x7eff4c00b420 comp rx=0 tx=0).stop 2024-04-21T08:59:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.347+0000 7eff5d020740 1 -- 172.21.15.28:0/1908294130 shutdown_connections 2024-04-21T08:59:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.347+0000 7eff5d020740 1 --2- 172.21.15.28:0/1908294130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f463a00 0x55637f465e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.347+0000 7eff5d020740 1 --2- 172.21.15.28:0/1908294130 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f45ae80 0x55637f463430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.347+0000 7eff5d020740 1 --2- 172.21.15.28:0/1908294130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f45a8b0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.347+0000 7eff5d020740 1 -- 172.21.15.28:0/1908294130 >> 172.21.15.28:0/1908294130 conn(0x55637f454ba0 msgr2=0x55637f454f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.347+0000 7eff5d020740 1 -- 172.21.15.28:0/1908294130 shutdown_connections 2024-04-21T08:59:45.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.347+0000 7eff5d020740 1 -- 172.21.15.28:0/1908294130 wait complete. 2024-04-21T08:59:45.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff5d020740 1 -- start start 2024-04-21T08:59:45.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff5d020740 1 Processor -- start 2024-04-21T08:59:45.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff5d020740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f4632f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff5d020740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f45ae80 0x55637f460680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff5d020740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f463a00 0x55637f460c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff5d020740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55637f466970 con 0x55637f45a4d0 2024-04-21T08:59:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff5d020740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55637f378500 con 0x55637f463a00 2024-04-21T08:59:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff5d020740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55637f466af0 con 0x55637f45ae80 2024-04-21T08:59:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff5d020740 10 client.0 _ll_register_callbacks cb 0x55637f346b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff53fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f4632f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff53fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f4632f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46356/0 (socket says 172.21.15.28:46356) 2024-04-21T08:59:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff53fff640 1 -- 172.21.15.28:0/2975311552 learned_addr learned my addr 172.21.15.28:0/2975311552 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff589d7640 1 --2- 172.21.15.28:0/2975311552 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f463a00 0x55637f460c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff53fff640 1 -- 172.21.15.28:0/2975311552 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f45ae80 msgr2=0x55637f460680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff537fe640 1 --2- 172.21.15.28:0/2975311552 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f45ae80 0x55637f460680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff53fff640 1 --2- 172.21.15.28:0/2975311552 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f45ae80 0x55637f460680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff53fff640 1 -- 172.21.15.28:0/2975311552 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f463a00 msgr2=0x55637f460c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff53fff640 1 --2- 172.21.15.28:0/2975311552 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f463a00 0x55637f460c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff53fff640 1 -- 172.21.15.28:0/2975311552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55637f0d13f0 con 0x55637f45a4d0 2024-04-21T08:59:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.348+0000 7eff589d7640 1 --2- 172.21.15.28:0/2975311552 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f463a00 0x55637f460c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff53fff640 1 --2- 172.21.15.28:0/2975311552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f4632f0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7eff44005c40 tx=0x7eff4402c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33094a5b01bcc6f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff337fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:45.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff337fe640 1 -- 172.21.15.28:0/2975311552 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44005230 con 0x55637f45a4d0 2024-04-21T08:59:45.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff337fe640 1 -- 172.21.15.28:0/2975311552 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7eff440053d0 con 0x55637f45a4d0 2024-04-21T08:59:45.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff5d020740 1 -- 172.21.15.28:0/2975311552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff4c01a020 con 0x55637f45a4d0 2024-04-21T08:59:45.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff337fe640 1 -- 172.21.15.28:0/2975311552 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7eff4404d050 con 0x55637f45a4d0 2024-04-21T08:59:45.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff5d020740 10 client.17853 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:45.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff5d020740 1 -- 172.21.15.28:0/2975311552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55637f19b7d0 con 0x55637f45a4d0 2024-04-21T08:59:45.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff337fe640 1 -- 172.21.15.28:0/2975311552 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff4c01a020 con 0x55637f45a4d0 2024-04-21T08:59:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff337fe640 1 -- 172.21.15.28:0/2975311552 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7eff440035f0 con 0x55637f45a4d0 2024-04-21T08:59:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff5d020740 10 client.17853 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff5d020740 10 client.17853 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff5d020740 20 client.17853 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff5d020740 1 --2- 172.21.15.28:0/2975311552 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55637f473bb0 0x55637f476000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff5d020740 4 client.17853 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:59:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.349+0000 7eff5d020740 1 -- 172.21.15.28:0/2975311552 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55637f35e930 con 0x55637f473bb0 2024-04-21T08:59:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.350+0000 7eff537fe640 1 --2- 172.21.15.28:0/2975311552 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55637f473bb0 0x55637f476000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.350+0000 7eff337fe640 10 client.17853 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.350+0000 7eff537fe640 1 --2- 172.21.15.28:0/2975311552 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55637f473bb0 0x55637f476000 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bccd988fc89b36c server_cookie=3e4d9019b329b8b9 in_seq=0 out_seq=0 2024-04-21T08:59:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.351+0000 7eff337fe640 1 -- 172.21.15.28:0/2975311552 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55637f35e930 con 0x55637f473bb0 2024-04-21T08:59:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.351+0000 7eff337fe640 10 client.17853 handle_command_reply: tid=0 2024-04-21T08:59:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.351+0000 7eff5d020740 1 client.17853 shutdown 2024-04-21T08:59:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.352+0000 7eff5d020740 1 -- 172.21.15.28:0/2975311552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 msgr2=0x55637f4632f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.352+0000 7eff5d020740 1 --2- 172.21.15.28:0/2975311552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f4632f0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7eff44005c40 tx=0x7eff4402c9f0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.352+0000 7eff5d020740 1 -- 172.21.15.28:0/2975311552 shutdown_connections 2024-04-21T08:59:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.352+0000 7eff5d020740 1 --2- 172.21.15.28:0/2975311552 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55637f473bb0 0x55637f476000 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.352+0000 7eff5d020740 1 --2- 172.21.15.28:0/2975311552 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f463a00 0x55637f460c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.352+0000 7eff5d020740 1 --2- 172.21.15.28:0/2975311552 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f45ae80 0x55637f460680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.352+0000 7eff337fe640 0 client.17853 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.352+0000 7eff5d020740 1 --2- 172.21.15.28:0/2975311552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f4632f0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.352+0000 7eff5d020740 1 -- 172.21.15.28:0/2975311552 >> 172.21.15.28:0/2975311552 conn(0x55637f454ba0 msgr2=0x55637f4568f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.352+0000 7eff5d020740 1 -- 172.21.15.28:0/2975311552 shutdown_connections 2024-04-21T08:59:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.352+0000 7eff5d020740 1 -- 172.21.15.28:0/2975311552 wait complete. 2024-04-21T08:59:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.352+0000 7eff5d020740 20 client.17853 trim_cache size 0 max 16384 2024-04-21T08:59:45.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.365+0000 7eff5d020740 1 Processor -- start 2024-04-21T08:59:45.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.365+0000 7eff5d020740 1 -- start start 2024-04-21T08:59:45.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.365+0000 7eff5d020740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f45a4d0 0x55637f46f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.365+0000 7eff5d020740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45ae80 0x55637f453a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.365+0000 7eff5d020740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f463a00 0x55637f462db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.365+0000 7eff5d020740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff541b1e10 con 0x55637f45ae80 2024-04-21T08:59:45.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.365+0000 7eff5d020740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7eff541af6e0 con 0x55637f45a4d0 2024-04-21T08:59:45.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.365+0000 7eff5d020740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7eff541ad550 con 0x55637f463a00 2024-04-21T08:59:45.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.365+0000 7eff537fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45ae80 0x55637f453a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.365+0000 7eff537fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45ae80 0x55637f453a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46358/0 (socket says 172.21.15.28:46358) 2024-04-21T08:59:45.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.365+0000 7eff537fe640 1 -- 172.21.15.28:0/3843736329 learned_addr learned my addr 172.21.15.28:0/3843736329 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:45.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.366+0000 7eff537fe640 1 -- 172.21.15.28:0/3843736329 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f463a00 msgr2=0x55637f462db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.366+0000 7eff537fe640 1 --2- 172.21.15.28:0/3843736329 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f463a00 0x55637f462db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.366+0000 7eff53fff640 1 --2- 172.21.15.28:0/3843736329 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f45a4d0 0x55637f46f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.366+0000 7eff537fe640 1 -- 172.21.15.28:0/3843736329 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f45a4d0 msgr2=0x55637f46f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.366+0000 7eff537fe640 1 --2- 172.21.15.28:0/3843736329 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f45a4d0 0x55637f46f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.366+0000 7eff537fe640 1 -- 172.21.15.28:0/3843736329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55637f45f630 con 0x55637f45ae80 2024-04-21T08:59:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.366+0000 7eff537fe640 1 --2- 172.21.15.28:0/3843736329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45ae80 0x55637f453a80 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7eff4809c430 tx=0x7eff4809f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c2a556b4df418b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.366+0000 7eff39ffb640 1 -- 172.21.15.28:0/3843736329 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff480a5110 con 0x55637f45ae80 2024-04-21T08:59:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.366+0000 7eff39ffb640 1 -- 172.21.15.28:0/3843736329 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff480a0850 con 0x55637f45ae80 2024-04-21T08:59:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.366+0000 7eff39ffb640 1 -- 172.21.15.28:0/3843736329 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4809c700 con 0x55637f45ae80 2024-04-21T08:59:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.366+0000 7eff5d020740 1 -- 172.21.15.28:0/3843736329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45ae80 msgr2=0x55637f453a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.366+0000 7eff5d020740 1 --2- 172.21.15.28:0/3843736329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45ae80 0x55637f453a80 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7eff4809c430 tx=0x7eff4809f040 comp rx=0 tx=0).stop 2024-04-21T08:59:45.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 1 -- 172.21.15.28:0/3843736329 shutdown_connections 2024-04-21T08:59:45.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 1 --2- 172.21.15.28:0/3843736329 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f463a00 0x55637f462db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 1 --2- 172.21.15.28:0/3843736329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45ae80 0x55637f453a80 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 1 --2- 172.21.15.28:0/3843736329 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f45a4d0 0x55637f46f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 1 -- 172.21.15.28:0/3843736329 >> 172.21.15.28:0/3843736329 conn(0x55637f454ba0 msgr2=0x55637f453370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 1 -- 172.21.15.28:0/3843736329 shutdown_connections 2024-04-21T08:59:45.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 1 -- 172.21.15.28:0/3843736329 wait complete. 2024-04-21T08:59:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 1 -- start start 2024-04-21T08:59:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 1 Processor -- start 2024-04-21T08:59:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f45c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f45ae80 0x55637f45ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f463a00 0x55637f45d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff541adba0 con 0x55637f45a4d0 2024-04-21T08:59:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7eff541af010 con 0x55637f45ae80 2024-04-21T08:59:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7eff541ae270 con 0x55637f463a00 2024-04-21T08:59:45.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff5d020740 10 client.0 _ll_register_callbacks cb 0x7eff44004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:45.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff53fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f45c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.367+0000 7eff53fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f45c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46374/0 (socket says 172.21.15.28:46374) 2024-04-21T08:59:45.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff53fff640 1 -- 172.21.15.28:0/2137048374 learned_addr learned my addr 172.21.15.28:0/2137048374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:45.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff537fe640 1 --2- 172.21.15.28:0/2137048374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f45ae80 0x55637f45ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff53fff640 1 -- 172.21.15.28:0/2137048374 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f463a00 msgr2=0x55637f45d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff53fff640 1 --2- 172.21.15.28:0/2137048374 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f463a00 0x55637f45d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff53fff640 1 -- 172.21.15.28:0/2137048374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f45ae80 msgr2=0x55637f45ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff53fff640 1 --2- 172.21.15.28:0/2137048374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f45ae80 0x55637f45ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff53fff640 1 -- 172.21.15.28:0/2137048374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55637f45ac20 con 0x55637f45a4d0 2024-04-21T08:59:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff537fe640 1 --2- 172.21.15.28:0/2137048374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f45ae80 0x55637f45ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff53fff640 1 --2- 172.21.15.28:0/2137048374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f45c8f0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7eff44004650 tx=0x7eff440038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e933cc22ec09a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:45.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff337fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff337fe640 1 -- 172.21.15.28:0/2137048374 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44056060 con 0x55637f45a4d0 2024-04-21T08:59:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff337fe640 1 -- 172.21.15.28:0/2137048374 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7eff44034490 con 0x55637f45a4d0 2024-04-21T08:59:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff5d020740 1 -- 172.21.15.28:0/2137048374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff480a0850 con 0x55637f45a4d0 2024-04-21T08:59:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff337fe640 1 -- 172.21.15.28:0/2137048374 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7eff4405d050 con 0x55637f45a4d0 2024-04-21T08:59:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff5d020740 10 client.17859 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff5d020740 1 -- 172.21.15.28:0/2137048374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55637f4de370 con 0x55637f45a4d0 2024-04-21T08:59:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.368+0000 7eff337fe640 1 -- 172.21.15.28:0/2137048374 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff480a0850 con 0x55637f45a4d0 2024-04-21T08:59:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.369+0000 7eff337fe640 1 -- 172.21.15.28:0/2137048374 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7eff440474c0 con 0x55637f45a4d0 2024-04-21T08:59:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.369+0000 7eff5d020740 10 client.17859 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.369+0000 7eff5d020740 10 client.17859 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.369+0000 7eff5d020740 20 client.17859 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.369+0000 7eff5d020740 1 --2- 172.21.15.28:0/2137048374 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55637f4deac0 0x55637f4e0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.369+0000 7eff5d020740 4 client.17859 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:59:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.369+0000 7eff5d020740 1 -- 172.21.15.28:0/2137048374 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55637f3ccc10 con 0x55637f4deac0 2024-04-21T08:59:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.369+0000 7eff537fe640 1 --2- 172.21.15.28:0/2137048374 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55637f4deac0 0x55637f4e0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:45.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.369+0000 7eff337fe640 10 client.17859 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.369+0000 7eff537fe640 1 --2- 172.21.15.28:0/2137048374 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55637f4deac0 0x55637f4e0ee0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8327cd8147e6f29 server_cookie=fef1ddf5f682e2c1 in_seq=0 out_seq=0 2024-04-21T08:59:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.369+0000 7eff337fe640 1 -- 172.21.15.28:0/2137048374 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55637f3ccc10 con 0x55637f4deac0 2024-04-21T08:59:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.369+0000 7eff337fe640 10 client.17859 handle_command_reply: tid=0 2024-04-21T08:59:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.369+0000 7eff5d020740 1 client.17859 shutdown 2024-04-21T08:59:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5d020740 1 -- 172.21.15.28:0/2137048374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 msgr2=0x55637f45c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5d020740 1 --2- 172.21.15.28:0/2137048374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f45c8f0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7eff44004650 tx=0x7eff440038a0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5d020740 1 -- 172.21.15.28:0/2137048374 shutdown_connections 2024-04-21T08:59:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5d020740 1 --2- 172.21.15.28:0/2137048374 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55637f4deac0 0x55637f4e0ee0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5d020740 1 --2- 172.21.15.28:0/2137048374 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55637f463a00 0x55637f45d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5d020740 1 --2- 172.21.15.28:0/2137048374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55637f45ae80 0x55637f45ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff337fe640 0 client.17859 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5d020740 1 --2- 172.21.15.28:0/2137048374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55637f45a4d0 0x55637f45c8f0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5d020740 1 -- 172.21.15.28:0/2137048374 >> 172.21.15.28:0/2137048374 conn(0x55637f454ba0 msgr2=0x55637f453c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5d020740 1 -- 172.21.15.28:0/2137048374 shutdown_connections 2024-04-21T08:59:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5d020740 1 -- 172.21.15.28:0/2137048374 wait complete. 2024-04-21T08:59:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5d020740 20 client.17859 trim_cache size 0 max 16384 2024-04-21T08:59:45.387 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T08:59:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5a43a640 1 -- 172.21.15.28:0/2525369944 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7eff3c0a3d20 msgr2=0x7eff3c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5a43a640 1 --2- 172.21.15.28:0/2525369944 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7eff3c0a3d20 0x7eff3c0a61e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7eff4c01b000 tx=0x7eff4c01c000 comp rx=0 tx=0).stop 2024-04-21T08:59:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5a43a640 1 -- 172.21.15.28:0/2525369944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff54107260 msgr2=0x7eff5419e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5a43a640 1 --2- 172.21.15.28:0/2525369944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff54107260 0x7eff5419e9a0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7eff48005fd0 tx=0x7eff4800dde0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5a43a640 1 -- 172.21.15.28:0/2525369944 shutdown_connections 2024-04-21T08:59:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5a43a640 1 --2- 172.21.15.28:0/2525369944 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7eff3c0a3d20 0x7eff3c0a61e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5a43a640 1 --2- 172.21.15.28:0/2525369944 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7eff54198610 0x7eff54198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5a43a640 1 --2- 172.21.15.28:0/2525369944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff54107260 0x7eff5419e9a0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5a43a640 1 --2- 172.21.15.28:0/2525369944 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff54102190 0x7eff5419e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5a43a640 1 -- 172.21.15.28:0/2525369944 >> 172.21.15.28:0/2525369944 conn(0x7eff540793a0 msgr2=0x7eff540fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.370+0000 7eff5a43a640 1 -- 172.21.15.28:0/2525369944 shutdown_connections 2024-04-21T08:59:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:45.371+0000 7eff5a43a640 1 -- 172.21.15.28:0/2525369944 wait complete. 2024-04-21T08:59:45.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:45 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3313464022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:59:45.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:45 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3313464022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:59:45.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:45 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3313464022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:59:46.391 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:59:46.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:59:46.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:59:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.512+0000 7fc91492e640 1 -- 172.21.15.28:0/653006666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc910101090 msgr2=0x7fc910101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.512+0000 7fc91492e640 1 --2- 172.21.15.28:0/653006666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc910101090 0x7fc910101470 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fc900009900 tx=0x7fc90002bc30 comp rx=0 tx=0).stop 2024-04-21T08:59:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.512+0000 7fc91492e640 1 -- 172.21.15.28:0/653006666 shutdown_connections 2024-04-21T08:59:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.512+0000 7fc91492e640 1 --2- 172.21.15.28:0/653006666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc910106160 0x7fc91010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.512+0000 7fc91492e640 1 --2- 172.21.15.28:0/653006666 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc910101a40 0x7fc910105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.512+0000 7fc91492e640 1 --2- 172.21.15.28:0/653006666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc910101090 0x7fc910101470 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.512+0000 7fc91492e640 1 -- 172.21.15.28:0/653006666 >> 172.21.15.28:0/653006666 conn(0x7fc910077f50 msgr2=0x7fc910078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.512+0000 7fc91492e640 1 -- 172.21.15.28:0/653006666 shutdown_connections 2024-04-21T08:59:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.512+0000 7fc91492e640 1 -- 172.21.15.28:0/653006666 wait complete. 2024-04-21T08:59:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.513+0000 7fc91492e640 1 Processor -- start 2024-04-21T08:59:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.513+0000 7fc91492e640 1 -- start start 2024-04-21T08:59:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.513+0000 7fc91492e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc910101090 0x7fc91019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.513+0000 7fc91492e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc910106160 0x7fc91019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.513+0000 7fc91492e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc9101985d0 0x7fc910198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.513+0000 7fc91492e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9101110c0 con 0x7fc910106160 2024-04-21T08:59:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.513+0000 7fc91492e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc910110f40 con 0x7fc9101985d0 2024-04-21T08:59:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.513+0000 7fc91492e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc910111240 con 0x7fc910101090 2024-04-21T08:59:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.513+0000 7fc90dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc910106160 0x7fc91019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.513+0000 7fc90dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc910106160 0x7fc91019e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46390/0 (socket says 172.21.15.28:46390) 2024-04-21T08:59:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.513+0000 7fc90dd9b640 1 -- 172.21.15.28:0/3082852934 learned_addr learned my addr 172.21.15.28:0/3082852934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.513+0000 7fc90ed9d640 1 --2- 172.21.15.28:0/3082852934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc9101985d0 0x7fc910198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.514+0000 7fc90e59c640 1 --2- 172.21.15.28:0/3082852934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc910101090 0x7fc91019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.514+0000 7fc90dd9b640 1 -- 172.21.15.28:0/3082852934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc910101090 msgr2=0x7fc91019e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.514+0000 7fc90dd9b640 1 --2- 172.21.15.28:0/3082852934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc910101090 0x7fc91019e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.514+0000 7fc90dd9b640 1 -- 172.21.15.28:0/3082852934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc9101985d0 msgr2=0x7fc910198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.514+0000 7fc90dd9b640 1 --2- 172.21.15.28:0/3082852934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc9101985d0 0x7fc910198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.514+0000 7fc90dd9b640 1 -- 172.21.15.28:0/3082852934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9101990f0 con 0x7fc910106160 2024-04-21T08:59:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.514+0000 7fc90e59c640 1 --2- 172.21.15.28:0/3082852934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc910101090 0x7fc91019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.514+0000 7fc90dd9b640 1 --2- 172.21.15.28:0/3082852934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc910106160 0x7fc91019e960 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fc90400e9f0 tx=0x7fc90400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.514+0000 7fc8ff7fe640 1 -- 172.21.15.28:0/3082852934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90400ccc0 con 0x7fc910106160 2024-04-21T08:59:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.514+0000 7fc8ff7fe640 1 -- 172.21.15.28:0/3082852934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9040054d0 con 0x7fc910106160 2024-04-21T08:59:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.514+0000 7fc91492e640 1 -- 172.21.15.28:0/3082852934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9101993e0 con 0x7fc910106160 2024-04-21T08:59:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.514+0000 7fc8ff7fe640 1 -- 172.21.15.28:0/3082852934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc904010430 con 0x7fc910106160 2024-04-21T08:59:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.514+0000 7fc91492e640 1 -- 172.21.15.28:0/3082852934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9101a55e0 con 0x7fc910106160 2024-04-21T08:59:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.516+0000 7fc8ff7fe640 1 -- 172.21.15.28:0/3082852934 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc9040046c0 con 0x7fc910106160 2024-04-21T08:59:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc9175f5740 1 Processor -- start 2024-04-21T08:59:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc9175f5740 1 -- start start 2024-04-21T08:59:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc9175f5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c57418e0 0x5570c5741cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc9175f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c5742290 0x5570c574a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc9175f5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c574ae10 0x5570c574d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc9175f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5570c56a5f00 con 0x5570c5742290 2024-04-21T08:59:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc9175f5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5570c5643590 con 0x5570c57418e0 2024-04-21T08:59:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc9175f5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5570c56a1ec0 con 0x5570c574ae10 2024-04-21T08:59:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc8ff7fe640 1 --2- 172.21.15.28:0/3082852934 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc8f80a3d20 0x7fc8f80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc90e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c5742290 0x5570c574a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc90e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c5742290 0x5570c574a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46400/0 (socket says 172.21.15.28:46400) 2024-04-21T08:59:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc90e59c640 1 -- 172.21.15.28:0/2959376275 learned_addr learned my addr 172.21.15.28:0/2959376275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc90ed9d640 1 --2- 172.21.15.28:0/3082852934 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc8f80a3d20 0x7fc8f80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc8ff7fe640 1 -- 172.21.15.28:0/3082852934 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7fc9040d0ff0 con 0x7fc910106160 2024-04-21T08:59:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc90ed9d640 1 --2- 172.21.15.28:0/2959376275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c57418e0 0x5570c5741cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.517+0000 7fc90dd9b640 1 --2- 172.21.15.28:0/2959376275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c574ae10 0x5570c574d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.518+0000 7fc90e59c640 1 -- 172.21.15.28:0/2959376275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c574ae10 msgr2=0x5570c574d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.518+0000 7fc90e59c640 1 --2- 172.21.15.28:0/2959376275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c574ae10 0x5570c574d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.518+0000 7fc90e59c640 1 -- 172.21.15.28:0/2959376275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c57418e0 msgr2=0x5570c5741cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.518+0000 7fc90e59c640 1 --2- 172.21.15.28:0/2959376275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c57418e0 0x5570c5741cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.518+0000 7fc90e59c640 1 -- 172.21.15.28:0/2959376275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570c54807e0 con 0x5570c5742290 2024-04-21T08:59:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.518+0000 7fc90e59c640 1 --2- 172.21.15.28:0/2959376275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c5742290 0x5570c574a840 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fc900009900 tx=0x7fc900005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af653fdf02f65490 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.518+0000 7fc90ed9d640 1 --2- 172.21.15.28:0/3082852934 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc8f80a3d20 0x7fc8f80a61e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc9080098b0 tx=0x7fc908009d90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.518+0000 7fc8fcff9640 1 -- 172.21.15.28:0/2959376275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc900044020 con 0x5570c5742290 2024-04-21T08:59:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.518+0000 7fc8fcff9640 1 -- 172.21.15.28:0/2959376275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc90002cc90 con 0x5570c5742290 2024-04-21T08:59:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.518+0000 7fc8fcff9640 1 -- 172.21.15.28:0/2959376275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9000046c0 con 0x5570c5742290 2024-04-21T08:59:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.519+0000 7fc9175f5740 1 -- 172.21.15.28:0/2959376275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c5742290 msgr2=0x5570c574a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.519+0000 7fc9175f5740 1 --2- 172.21.15.28:0/2959376275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c5742290 0x5570c574a840 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fc900009900 tx=0x7fc900005e50 comp rx=0 tx=0).stop 2024-04-21T08:59:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.519+0000 7fc9175f5740 1 -- 172.21.15.28:0/2959376275 shutdown_connections 2024-04-21T08:59:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.519+0000 7fc9175f5740 1 --2- 172.21.15.28:0/2959376275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c574ae10 0x5570c574d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.519+0000 7fc9175f5740 1 --2- 172.21.15.28:0/2959376275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c5742290 0x5570c574a840 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.519+0000 7fc9175f5740 1 --2- 172.21.15.28:0/2959376275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c57418e0 0x5570c5741cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.519+0000 7fc9175f5740 1 -- 172.21.15.28:0/2959376275 >> 172.21.15.28:0/2959376275 conn(0x5570c573bfb0 msgr2=0x5570c573c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.519+0000 7fc9175f5740 1 -- 172.21.15.28:0/2959376275 shutdown_connections 2024-04-21T08:59:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.519+0000 7fc9175f5740 1 -- 172.21.15.28:0/2959376275 wait complete. 2024-04-21T08:59:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.519+0000 7fc9175f5740 1 -- start start 2024-04-21T08:59:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.519+0000 7fc9175f5740 1 Processor -- start 2024-04-21T08:59:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc9175f5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c57418e0 0x5570c5747df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc9175f5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c5742290 0x5570c5748330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc9175f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c574ae10 0x5570c5748870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc9175f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5570c574dd20 con 0x5570c574ae10 2024-04-21T08:59:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc9175f5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5570c565f580 con 0x5570c57418e0 2024-04-21T08:59:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc9175f5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5570c574dea0 con 0x5570c5742290 2024-04-21T08:59:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc9175f5740 10 client.0 _ll_register_callbacks cb 0x5570c5665650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc90ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c574ae10 0x5570c5748870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc90ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c574ae10 0x5570c5748870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46414/0 (socket says 172.21.15.28:46414) 2024-04-21T08:59:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc90ed9d640 1 -- 172.21.15.28:0/550833021 learned_addr learned my addr 172.21.15.28:0/550833021 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc90e59c640 1 --2- 172.21.15.28:0/550833021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c57418e0 0x5570c5747df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc90dd9b640 1 --2- 172.21.15.28:0/550833021 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c5742290 0x5570c5748330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc90ed9d640 1 -- 172.21.15.28:0/550833021 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c5742290 msgr2=0x5570c5748330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc90ed9d640 1 --2- 172.21.15.28:0/550833021 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c5742290 0x5570c5748330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc90ed9d640 1 -- 172.21.15.28:0/550833021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c57418e0 msgr2=0x5570c5747df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc90ed9d640 1 --2- 172.21.15.28:0/550833021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c57418e0 0x5570c5747df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc90ed9d640 1 -- 172.21.15.28:0/550833021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570c54e0f70 con 0x5570c574ae10 2024-04-21T08:59:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc90ed9d640 1 --2- 172.21.15.28:0/550833021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c574ae10 0x5570c5748870 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fc9080092f0 tx=0x7fc908009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893c800fa1074b4b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc8f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc8f5ffb640 1 -- 172.21.15.28:0/550833021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc908004b20 con 0x5570c574ae10 2024-04-21T08:59:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc8f5ffb640 1 -- 172.21.15.28:0/550833021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7fc908004cc0 con 0x5570c574ae10 2024-04-21T08:59:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.520+0000 7fc9175f5740 1 -- 172.21.15.28:0/550833021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc90002cc90 con 0x5570c574ae10 2024-04-21T08:59:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.521+0000 7fc8f5ffb640 1 -- 172.21.15.28:0/550833021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc908030050 con 0x5570c574ae10 2024-04-21T08:59:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.521+0000 7fc9175f5740 10 client.17871 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.521+0000 7fc9175f5740 1 -- 172.21.15.28:0/550833021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5570c5378170 con 0x5570c574ae10 2024-04-21T08:59:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.521+0000 7fc8f5ffb640 1 -- 172.21.15.28:0/550833021 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc90002cc90 con 0x5570c574ae10 2024-04-21T08:59:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.521+0000 7fc8f5ffb640 1 -- 172.21.15.28:0/550833021 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fc90801f070 con 0x5570c574ae10 2024-04-21T08:59:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.521+0000 7fc9175f5740 10 client.17871 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.521+0000 7fc9175f5740 10 client.17871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.521+0000 7fc9175f5740 20 client.17871 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.521+0000 7fc9175f5740 1 --2- 172.21.15.28:0/550833021 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5570c575b2a0 0x5570c575d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.521+0000 7fc9175f5740 4 client.17871 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:59:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.521+0000 7fc9175f5740 1 -- 172.21.15.28:0/550833021 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570c56b3920 con 0x5570c575b2a0 2024-04-21T08:59:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.521+0000 7fc90e59c640 1 --2- 172.21.15.28:0/550833021 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5570c575b2a0 0x5570c575d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.522+0000 7fc90e59c640 1 --2- 172.21.15.28:0/550833021 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5570c575b2a0 0x5570c575d6f0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34d53dfb645c56b1 server_cookie=6f788639e588355a in_seq=0 out_seq=0 2024-04-21T08:59:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.522+0000 7fc8f5ffb640 10 client.17871 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.522+0000 7fc8f5ffb640 1 -- 172.21.15.28:0/550833021 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5570c56b3920 con 0x5570c575b2a0 2024-04-21T08:59:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.522+0000 7fc8f5ffb640 10 client.17871 handle_command_reply: tid=0 2024-04-21T08:59:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.523+0000 7fc9175f5740 1 client.17871 shutdown 2024-04-21T08:59:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.523+0000 7fc9175f5740 1 -- 172.21.15.28:0/550833021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c574ae10 msgr2=0x5570c5748870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.523+0000 7fc9175f5740 1 --2- 172.21.15.28:0/550833021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c574ae10 0x5570c5748870 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fc9080092f0 tx=0x7fc908009600 comp rx=0 tx=0).stop 2024-04-21T08:59:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.523+0000 7fc9175f5740 1 -- 172.21.15.28:0/550833021 shutdown_connections 2024-04-21T08:59:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.523+0000 7fc9175f5740 1 --2- 172.21.15.28:0/550833021 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5570c575b2a0 0x5570c575d6f0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.523+0000 7fc9175f5740 1 --2- 172.21.15.28:0/550833021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c574ae10 0x5570c5748870 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.523+0000 7fc8f5ffb640 0 client.17871 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:46.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.523+0000 7fc9175f5740 1 --2- 172.21.15.28:0/550833021 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c5742290 0x5570c5748330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.523+0000 7fc9175f5740 1 --2- 172.21.15.28:0/550833021 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c57418e0 0x5570c5747df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.523+0000 7fc9175f5740 1 -- 172.21.15.28:0/550833021 >> 172.21.15.28:0/550833021 conn(0x5570c573bfb0 msgr2=0x5570c573dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:46.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.523+0000 7fc9175f5740 1 -- 172.21.15.28:0/550833021 shutdown_connections 2024-04-21T08:59:46.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.523+0000 7fc9175f5740 1 -- 172.21.15.28:0/550833021 wait complete. 2024-04-21T08:59:46.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.523+0000 7fc9175f5740 20 client.17871 trim_cache size 0 max 16384 2024-04-21T08:59:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.541+0000 7fc9175f5740 1 Processor -- start 2024-04-21T08:59:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.541+0000 7fc9175f5740 1 -- start start 2024-04-21T08:59:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.541+0000 7fc9175f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c57418e0 0x5570c574a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.541+0000 7fc9175f5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c5742290 0x5570c5756be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.541+0000 7fc9175f5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c574ae10 0x5570c5762fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.541+0000 7fc9175f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9101b1d70 con 0x5570c57418e0 2024-04-21T08:59:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.541+0000 7fc9175f5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc9101af640 con 0x5570c5742290 2024-04-21T08:59:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.541+0000 7fc9175f5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc9101ad4b0 con 0x5570c574ae10 2024-04-21T08:59:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.541+0000 7fc90e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c57418e0 0x5570c574a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.541+0000 7fc90e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c57418e0 0x5570c574a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46424/0 (socket says 172.21.15.28:46424) 2024-04-21T08:59:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.542+0000 7fc90e59c640 1 -- 172.21.15.28:0/3811217779 learned_addr learned my addr 172.21.15.28:0/3811217779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.542+0000 7fc90dd9b640 1 --2- 172.21.15.28:0/3811217779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c5742290 0x5570c5756be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.542+0000 7fc90e59c640 1 -- 172.21.15.28:0/3811217779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c574ae10 msgr2=0x5570c5762fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.542+0000 7fc90e59c640 1 --2- 172.21.15.28:0/3811217779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c574ae10 0x5570c5762fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.542+0000 7fc90e59c640 1 -- 172.21.15.28:0/3811217779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c5742290 msgr2=0x5570c5756be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.542+0000 7fc90e59c640 1 --2- 172.21.15.28:0/3811217779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c5742290 0x5570c5756be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.542+0000 7fc90e59c640 1 -- 172.21.15.28:0/3811217779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570c5746340 con 0x5570c57418e0 2024-04-21T08:59:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.542+0000 7fc90dd9b640 1 --2- 172.21.15.28:0/3811217779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c5742290 0x5570c5756be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.542+0000 7fc90e59c640 1 --2- 172.21.15.28:0/3811217779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c57418e0 0x5570c574a520 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fc90002c170 tx=0x7fc900005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7addec1c93fa9c03 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.542+0000 7fc8f7fff640 1 -- 172.21.15.28:0/3811217779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc900044040 con 0x5570c57418e0 2024-04-21T08:59:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.542+0000 7fc8f7fff640 1 -- 172.21.15.28:0/3811217779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc900005600 con 0x5570c57418e0 2024-04-21T08:59:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.542+0000 7fc8f7fff640 1 -- 172.21.15.28:0/3811217779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc900003850 con 0x5570c57418e0 2024-04-21T08:59:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 -- 172.21.15.28:0/3811217779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c57418e0 msgr2=0x5570c574a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 --2- 172.21.15.28:0/3811217779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c57418e0 0x5570c574a520 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fc90002c170 tx=0x7fc900005fd0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 -- 172.21.15.28:0/3811217779 shutdown_connections 2024-04-21T08:59:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 --2- 172.21.15.28:0/3811217779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c574ae10 0x5570c5762fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 --2- 172.21.15.28:0/3811217779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c5742290 0x5570c5756be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 --2- 172.21.15.28:0/3811217779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c57418e0 0x5570c574a520 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 -- 172.21.15.28:0/3811217779 >> 172.21.15.28:0/3811217779 conn(0x5570c573bfb0 msgr2=0x5570c573b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 -- 172.21.15.28:0/3811217779 shutdown_connections 2024-04-21T08:59:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 -- 172.21.15.28:0/3811217779 wait complete. 2024-04-21T08:59:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 -- start start 2024-04-21T08:59:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 Processor -- start 2024-04-21T08:59:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c57418e0 0x5570c574edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c5742290 0x5570c574f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c574ae10 0x5570c574f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9101adb00 con 0x5570c57418e0 2024-04-21T08:59:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc9101aef70 con 0x5570c574ae10 2024-04-21T08:59:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc9101ae1d0 con 0x5570c5742290 2024-04-21T08:59:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc9175f5740 10 client.0 _ll_register_callbacks cb 0x7fc9080125b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.543+0000 7fc90e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c57418e0 0x5570c574edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc90e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c57418e0 0x5570c574edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46428/0 (socket says 172.21.15.28:46428) 2024-04-21T08:59:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc90e59c640 1 -- 172.21.15.28:0/3794708377 learned_addr learned my addr 172.21.15.28:0/3794708377 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc90ed9d640 1 --2- 172.21.15.28:0/3794708377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c574ae10 0x5570c574f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc90e59c640 1 -- 172.21.15.28:0/3794708377 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c5742290 msgr2=0x5570c574f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc90e59c640 1 --2- 172.21.15.28:0/3794708377 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c5742290 0x5570c574f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc90e59c640 1 -- 172.21.15.28:0/3794708377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c574ae10 msgr2=0x5570c574f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc90e59c640 1 --2- 172.21.15.28:0/3794708377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c574ae10 0x5570c574f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc90e59c640 1 -- 172.21.15.28:0/3794708377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570c57c0ad0 con 0x5570c57418e0 2024-04-21T08:59:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc90ed9d640 1 --2- 172.21.15.28:0/3794708377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c574ae10 0x5570c574f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc90e59c640 1 --2- 172.21.15.28:0/3794708377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c57418e0 0x5570c574edb0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fc90002bc70 tx=0x7fc90002c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa15301e01ec0cb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc8f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc8f5ffb640 1 -- 172.21.15.28:0/3794708377 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc900044040 con 0x5570c57418e0 2024-04-21T08:59:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc8f5ffb640 1 -- 172.21.15.28:0/3794708377 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7fc900004450 con 0x5570c57418e0 2024-04-21T08:59:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc9175f5740 1 -- 172.21.15.28:0/3794708377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc900005600 con 0x5570c57418e0 2024-04-21T08:59:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc8f5ffb640 1 -- 172.21.15.28:0/3794708377 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc9000035f0 con 0x5570c57418e0 2024-04-21T08:59:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc9175f5740 10 client.17877 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc9175f5740 1 -- 172.21.15.28:0/3794708377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5570c57c1d80 con 0x5570c57418e0 2024-04-21T08:59:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.544+0000 7fc8f5ffb640 1 -- 172.21.15.28:0/3794708377 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc900005600 con 0x5570c57418e0 2024-04-21T08:59:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.545+0000 7fc8f5ffb640 1 -- 172.21.15.28:0/3794708377 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fc900040950 con 0x5570c57418e0 2024-04-21T08:59:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.545+0000 7fc9175f5740 10 client.17877 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.545+0000 7fc9175f5740 10 client.17877 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.545+0000 7fc9175f5740 20 client.17877 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.545+0000 7fc9175f5740 1 --2- 172.21.15.28:0/3794708377 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5570c57c22f0 0x5570c57c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.545+0000 7fc9175f5740 4 client.17877 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:59:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.545+0000 7fc9175f5740 1 -- 172.21.15.28:0/3794708377 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5570c56ad790 con 0x5570c57c22f0 2024-04-21T08:59:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.545+0000 7fc90dd9b640 1 --2- 172.21.15.28:0/3794708377 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5570c57c22f0 0x5570c57c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.545+0000 7fc90dd9b640 1 --2- 172.21.15.28:0/3794708377 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5570c57c22f0 0x5570c57c46e0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b672bc6f73b4359 server_cookie=30d448cbbe9e245b in_seq=0 out_seq=0 2024-04-21T08:59:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.545+0000 7fc8f5ffb640 10 client.17877 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:46.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.546+0000 7fc8f5ffb640 1 -- 172.21.15.28:0/3794708377 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5570c56ad790 con 0x5570c57c22f0 2024-04-21T08:59:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.546+0000 7fc8f5ffb640 10 client.17877 handle_command_reply: tid=0 2024-04-21T08:59:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc9175f5740 1 client.17877 shutdown 2024-04-21T08:59:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc9175f5740 1 -- 172.21.15.28:0/3794708377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c57418e0 msgr2=0x5570c574edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc9175f5740 1 --2- 172.21.15.28:0/3794708377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c57418e0 0x5570c574edb0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fc90002bc70 tx=0x7fc90002c960 comp rx=0 tx=0).stop 2024-04-21T08:59:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc9175f5740 1 -- 172.21.15.28:0/3794708377 shutdown_connections 2024-04-21T08:59:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc9175f5740 1 --2- 172.21.15.28:0/3794708377 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5570c57c22f0 0x5570c57c46e0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc9175f5740 1 --2- 172.21.15.28:0/3794708377 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5570c574ae10 0x5570c574f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc9175f5740 1 --2- 172.21.15.28:0/3794708377 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5570c5742290 0x5570c574f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc8f5ffb640 0 client.17877 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc9175f5740 1 --2- 172.21.15.28:0/3794708377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c57418e0 0x5570c574edb0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc9175f5740 1 -- 172.21.15.28:0/3794708377 >> 172.21.15.28:0/3794708377 conn(0x5570c573bfb0 msgr2=0x5570c5747db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc9175f5740 1 -- 172.21.15.28:0/3794708377 shutdown_connections 2024-04-21T08:59:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc9175f5740 1 -- 172.21.15.28:0/3794708377 wait complete. 2024-04-21T08:59:46.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc9175f5740 20 client.17877 trim_cache size 0 max 16384 2024-04-21T08:59:46.556 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:59:46.557 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T08:59:46.557 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "4d607d1a-9f7d-44f2-840a-0ddc2e551163", 2024-04-21T08:59:46.557 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T08:59:46.557 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:59:46.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc91492e640 1 -- 172.21.15.28:0/3082852934 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc8f80a3d20 msgr2=0x7fc8f80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc91492e640 1 --2- 172.21.15.28:0/3082852934 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc8f80a3d20 0x7fc8f80a61e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc9080098b0 tx=0x7fc908009d90 comp rx=0 tx=0).stop 2024-04-21T08:59:46.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc91492e640 1 -- 172.21.15.28:0/3082852934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc910106160 msgr2=0x7fc91019e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.547+0000 7fc91492e640 1 --2- 172.21.15.28:0/3082852934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc910106160 0x7fc91019e960 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fc90400e9f0 tx=0x7fc90400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.548+0000 7fc91492e640 1 -- 172.21.15.28:0/3082852934 shutdown_connections 2024-04-21T08:59:46.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.548+0000 7fc91492e640 1 --2- 172.21.15.28:0/3082852934 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc8f80a3d20 0x7fc8f80a61e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.548+0000 7fc91492e640 1 --2- 172.21.15.28:0/3082852934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc9101985d0 0x7fc910198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.548+0000 7fc91492e640 1 --2- 172.21.15.28:0/3082852934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc910106160 0x7fc91019e960 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.548+0000 7fc91492e640 1 --2- 172.21.15.28:0/3082852934 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc910101090 0x7fc91019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.548+0000 7fc91492e640 1 -- 172.21.15.28:0/3082852934 >> 172.21.15.28:0/3082852934 conn(0x7fc910077f50 msgr2=0x7fc9100694c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.548+0000 7fc91492e640 1 -- 172.21.15.28:0/3082852934 shutdown_connections 2024-04-21T08:59:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.548+0000 7fc91492e640 1 -- 172.21.15.28:0/3082852934 wait complete. 2024-04-21T08:59:46.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:59:46.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:59:46.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:46 smithi028 ceph-mon[28003]: pgmap v264: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:59:46.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:46 smithi152 ceph-mon[31401]: pgmap v264: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:59:46.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:46 smithi171 ceph-mon[31358]: pgmap v264: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T08:59:46.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.668+0000 7f0580f98640 1 -- 172.21.15.28:0/696460381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057c06a5b0 msgr2=0x7f057c06aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.668+0000 7f0580f98640 1 --2- 172.21.15.28:0/696460381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057c06a5b0 0x7f057c06aa10 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f0564009ca0 tx=0x7f056402c180 comp rx=0 tx=0).stop 2024-04-21T08:59:46.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.668+0000 7f0580f98640 1 -- 172.21.15.28:0/696460381 shutdown_connections 2024-04-21T08:59:46.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.668+0000 7f0580f98640 1 --2- 172.21.15.28:0/696460381 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f057c06af50 0x7f057c10ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.668+0000 7f0580f98640 1 --2- 172.21.15.28:0/696460381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057c06a5b0 0x7f057c06aa10 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.668+0000 7f0580f98640 1 --2- 172.21.15.28:0/696460381 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f057c108040 0x7f057c108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.668+0000 7f0580f98640 1 -- 172.21.15.28:0/696460381 >> 172.21.15.28:0/696460381 conn(0x7f057c077c40 msgr2=0x7f057c078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:46.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.668+0000 7f0580f98640 1 -- 172.21.15.28:0/696460381 shutdown_connections 2024-04-21T08:59:46.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.668+0000 7f0580f98640 1 -- 172.21.15.28:0/696460381 wait complete. 2024-04-21T08:59:46.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0580f98640 1 Processor -- start 2024-04-21T08:59:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0580f98640 1 -- start start 2024-04-21T08:59:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0580f98640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f057c06a5b0 0x7f057c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0580f98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057c06af50 0x7f057c198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0580f98640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f057c108040 0x7f057c19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0580f98640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f057c112150 con 0x7f057c06af50 2024-04-21T08:59:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0580f98640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f057c111fd0 con 0x7f057c108040 2024-04-21T08:59:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0580f98640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f057c1122d0 con 0x7f057c06a5b0 2024-04-21T08:59:46.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0579d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057c06af50 0x7f057c198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0579d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057c06af50 0x7f057c198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46446/0 (socket says 172.21.15.28:46446) 2024-04-21T08:59:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0579d9b640 1 -- 172.21.15.28:0/2678987494 learned_addr learned my addr 172.21.15.28:0/2678987494 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f057a59c640 1 --2- 172.21.15.28:0/2678987494 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f057c06a5b0 0x7f057c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f057ad9d640 1 --2- 172.21.15.28:0/2678987494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f057c108040 0x7f057c19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0579d9b640 1 -- 172.21.15.28:0/2678987494 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f057c06a5b0 msgr2=0x7f057c198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0579d9b640 1 --2- 172.21.15.28:0/2678987494 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f057c06a5b0 0x7f057c198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0579d9b640 1 -- 172.21.15.28:0/2678987494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f057c108040 msgr2=0x7f057c19cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0579d9b640 1 --2- 172.21.15.28:0/2678987494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f057c108040 0x7f057c19cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f0579d9b640 1 -- 172.21.15.28:0/2678987494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f057c19d680 con 0x7f057c06af50 2024-04-21T08:59:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.669+0000 7f057a59c640 1 --2- 172.21.15.28:0/2678987494 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f057c06a5b0 0x7f057c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.670+0000 7f0579d9b640 1 --2- 172.21.15.28:0/2678987494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057c06af50 0x7f057c198bd0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f0564005e30 tx=0x7f05640039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.670+0000 7f05737fe640 1 -- 172.21.15.28:0/2678987494 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0564039040 con 0x7f057c06af50 2024-04-21T08:59:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.670+0000 7f05737fe640 1 -- 172.21.15.28:0/2678987494 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0564044070 con 0x7f057c06af50 2024-04-21T08:59:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.670+0000 7f0580f98640 1 -- 172.21.15.28:0/2678987494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f057c19d910 con 0x7f057c06af50 2024-04-21T08:59:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.670+0000 7f05737fe640 1 -- 172.21.15.28:0/2678987494 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0564032960 con 0x7f057c06af50 2024-04-21T08:59:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.670+0000 7f0580f98640 1 -- 172.21.15.28:0/2678987494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f057c1a5200 con 0x7f057c06af50 2024-04-21T08:59:46.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.673+0000 7f0583b82740 1 Processor -- start 2024-04-21T08:59:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.673+0000 7f05737fe640 1 -- 172.21.15.28:0/2678987494 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0564004580 con 0x7f057c06af50 2024-04-21T08:59:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.673+0000 7f0583b82740 1 -- start start 2024-04-21T08:59:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.673+0000 7f0583b82740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7580 0x55f258cc7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.673+0000 7f0583b82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cd0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.673+0000 7f0583b82740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 0x55f258cd2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.673+0000 7f0583b82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f258c2c1e0 con 0x55f258cc7ea0 2024-04-21T08:59:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.673+0000 7f0583b82740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f258bc9390 con 0x55f258cc7580 2024-04-21T08:59:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.673+0000 7f0583b82740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f258c281a0 con 0x55f258cd0a80 2024-04-21T08:59:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.673+0000 7f057a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cd0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.673+0000 7f057a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cd0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46450/0 (socket says 172.21.15.28:46450) 2024-04-21T08:59:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.673+0000 7f057a59c640 1 -- 172.21.15.28:0/3820405619 learned_addr learned my addr 172.21.15.28:0/3820405619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.673+0000 7f057ad9d640 1 --2- 172.21.15.28:0/3820405619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7580 0x55f258cc7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.674+0000 7f05737fe640 1 --2- 172.21.15.28:0/2678987494 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f05680a3d20 0x7f05680a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.674+0000 7f0579d9b640 1 --2- 172.21.15.28:0/3820405619 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 0x55f258cd2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.674+0000 7f057a59c640 1 -- 172.21.15.28:0/3820405619 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 msgr2=0x55f258cd2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.674+0000 7f057a59c640 1 --2- 172.21.15.28:0/3820405619 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 0x55f258cd2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.674+0000 7f057a59c640 1 -- 172.21.15.28:0/3820405619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7580 msgr2=0x55f258cc7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.674+0000 7f057a59c640 1 --2- 172.21.15.28:0/3820405619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7580 0x55f258cc7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.674+0000 7f057a59c640 1 -- 172.21.15.28:0/3820405619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f258a087d0 con 0x55f258cc7ea0 2024-04-21T08:59:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.674+0000 7f057ad9d640 1 --2- 172.21.15.28:0/3820405619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7580 0x55f258cc7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.674+0000 7f0579d9b640 1 --2- 172.21.15.28:0/3820405619 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 0x55f258cd2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.674+0000 7f05737fe640 1 -- 172.21.15.28:0/2678987494 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7f05640f31c0 con 0x7f057c06af50 2024-04-21T08:59:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.674+0000 7f057ad9d640 1 --2- 172.21.15.28:0/2678987494 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f05680a3d20 0x7f05680a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.674+0000 7f057a59c640 1 --2- 172.21.15.28:0/3820405619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cd0540 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f056c00bcc0 tx=0x7f056c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d57f4e5d5be9eb03 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.674+0000 7f0570ff9640 1 -- 172.21.15.28:0/3820405619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056c015070 con 0x55f258cc7ea0 2024-04-21T08:59:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.674+0000 7f0570ff9640 1 -- 172.21.15.28:0/3820405619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f056c01a910 con 0x55f258cc7ea0 2024-04-21T08:59:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.674+0000 7f0570ff9640 1 -- 172.21.15.28:0/3820405619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056c011980 con 0x55f258cc7ea0 2024-04-21T08:59:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.675+0000 7f0583b82740 1 -- 172.21.15.28:0/3820405619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 msgr2=0x55f258cd0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.675+0000 7f057ad9d640 1 --2- 172.21.15.28:0/2678987494 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f05680a3d20 0x7f05680a61e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0574005fd0 tx=0x7f05740046c0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.675+0000 7f0583b82740 1 --2- 172.21.15.28:0/3820405619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cd0540 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f056c00bcc0 tx=0x7f056c00f730 comp rx=0 tx=0).stop 2024-04-21T08:59:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.675+0000 7f0583b82740 1 -- 172.21.15.28:0/3820405619 shutdown_connections 2024-04-21T08:59:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.675+0000 7f0583b82740 1 --2- 172.21.15.28:0/3820405619 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 0x55f258cd2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.675+0000 7f0583b82740 1 --2- 172.21.15.28:0/3820405619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cd0540 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.675+0000 7f0583b82740 1 --2- 172.21.15.28:0/3820405619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7580 0x55f258cc7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.675+0000 7f0583b82740 1 -- 172.21.15.28:0/3820405619 >> 172.21.15.28:0/3820405619 conn(0x55f258cc1cb0 msgr2=0x55f258cc2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.675+0000 7f0583b82740 1 -- 172.21.15.28:0/3820405619 shutdown_connections 2024-04-21T08:59:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.675+0000 7f0583b82740 1 -- 172.21.15.28:0/3820405619 wait complete. 2024-04-21T08:59:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.675+0000 7f0583b82740 1 -- start start 2024-04-21T08:59:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.675+0000 7f0583b82740 1 Processor -- start 2024-04-21T08:59:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0583b82740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7580 0x55f258ccdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0583b82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0583b82740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 0x55f258cce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0583b82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f258cd3870 con 0x55f258cc7ea0 2024-04-21T08:59:46.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0583b82740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f258be52f0 con 0x55f258cc7580 2024-04-21T08:59:46.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0583b82740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f258cd39f0 con 0x55f258cd0a80 2024-04-21T08:59:46.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0583b82740 10 client.0 _ll_register_callbacks cb 0x55f258bf2580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:46.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0579d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0579d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46466/0 (socket says 172.21.15.28:46466) 2024-04-21T08:59:46.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0579d9b640 1 -- 172.21.15.28:0/2992291667 learned_addr learned my addr 172.21.15.28:0/2992291667 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:46.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f057a59c640 1 --2- 172.21.15.28:0/2992291667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7580 0x55f258ccdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f057ad9d640 1 --2- 172.21.15.28:0/2992291667 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 0x55f258cce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0579d9b640 1 -- 172.21.15.28:0/2992291667 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 msgr2=0x55f258cce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0579d9b640 1 --2- 172.21.15.28:0/2992291667 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 0x55f258cce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0579d9b640 1 -- 172.21.15.28:0/2992291667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7580 msgr2=0x55f258ccdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0579d9b640 1 --2- 172.21.15.28:0/2992291667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7580 0x55f258ccdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0579d9b640 1 -- 172.21.15.28:0/2992291667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f25893e3f0 con 0x55f258cc7ea0 2024-04-21T08:59:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f057ad9d640 1 --2- 172.21.15.28:0/2992291667 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 0x55f258cce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f057a59c640 1 --2- 172.21.15.28:0/2992291667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7580 0x55f258ccdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0579d9b640 1 --2- 172.21.15.28:0/2992291667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cce010 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f05640f0b20 tx=0x7f056408e760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b761a57dfd45a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0561ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.676+0000 7f0561ffb640 1 -- 172.21.15.28:0/2992291667 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0564089990 con 0x55f258cc7ea0 2024-04-21T08:59:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.677+0000 7f0561ffb640 1 -- 172.21.15.28:0/2992291667 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7f0564004040 con 0x55f258cc7ea0 2024-04-21T08:59:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.677+0000 7f0583b82740 1 -- 172.21.15.28:0/2992291667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f056c01a910 con 0x55f258cc7ea0 2024-04-21T08:59:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.677+0000 7f0561ffb640 1 -- 172.21.15.28:0/2992291667 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f05640bf580 con 0x55f258cc7ea0 2024-04-21T08:59:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.677+0000 7f0583b82740 10 client.17889 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.677+0000 7f0583b82740 1 -- 172.21.15.28:0/2992291667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f2588fe0f0 con 0x55f258cc7ea0 2024-04-21T08:59:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.677+0000 7f0561ffb640 1 -- 172.21.15.28:0/2992291667 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f056c01a910 con 0x55f258cc7ea0 2024-04-21T08:59:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.677+0000 7f0561ffb640 1 -- 172.21.15.28:0/2992291667 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f056403c030 con 0x55f258cc7ea0 2024-04-21T08:59:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.677+0000 7f0583b82740 10 client.17889 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.677+0000 7f0583b82740 10 client.17889 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.677+0000 7f0583b82740 20 client.17889 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.677+0000 7f0583b82740 1 --2- 172.21.15.28:0/2992291667 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f258ce4d90 0x55f258ce7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.677+0000 7f0583b82740 4 client.17889 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:59:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.677+0000 7f0583b82740 1 -- 172.21.15.28:0/2992291667 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f258c39c00 con 0x55f258ce4d90 2024-04-21T08:59:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.677+0000 7f057a59c640 1 --2- 172.21.15.28:0/2992291667 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f258ce4d90 0x55f258ce7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.678+0000 7f0561ffb640 10 client.17889 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.678+0000 7f057a59c640 1 --2- 172.21.15.28:0/2992291667 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f258ce4d90 0x55f258ce7220 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b4ef693cd036da2 server_cookie=144bb9ce9df8b7a9 in_seq=0 out_seq=0 2024-04-21T08:59:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.678+0000 7f0561ffb640 1 -- 172.21.15.28:0/2992291667 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f258c39c00 con 0x55f258ce4d90 2024-04-21T08:59:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.678+0000 7f0561ffb640 10 client.17889 handle_command_reply: tid=0 2024-04-21T08:59:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.678+0000 7f0583b82740 1 client.17889 shutdown 2024-04-21T08:59:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.679+0000 7f0583b82740 1 -- 172.21.15.28:0/2992291667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 msgr2=0x55f258cce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.679+0000 7f0583b82740 1 --2- 172.21.15.28:0/2992291667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cce010 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f05640f0b20 tx=0x7f056408e760 comp rx=0 tx=0).stop 2024-04-21T08:59:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.679+0000 7f0583b82740 1 -- 172.21.15.28:0/2992291667 shutdown_connections 2024-04-21T08:59:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.679+0000 7f0583b82740 1 --2- 172.21.15.28:0/2992291667 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f258ce4d90 0x55f258ce7220 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.679+0000 7f0583b82740 1 --2- 172.21.15.28:0/2992291667 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 0x55f258cce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.679+0000 7f0583b82740 1 --2- 172.21.15.28:0/2992291667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cce010 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.679+0000 7f0583b82740 1 --2- 172.21.15.28:0/2992291667 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7580 0x55f258ccdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.679+0000 7f0561ffb640 0 client.17889 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:46.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.679+0000 7f0583b82740 1 -- 172.21.15.28:0/2992291667 >> 172.21.15.28:0/2992291667 conn(0x55f258cc1cb0 msgr2=0x55f258cc3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:46.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.679+0000 7f0583b82740 1 -- 172.21.15.28:0/2992291667 shutdown_connections 2024-04-21T08:59:46.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.679+0000 7f0583b82740 1 -- 172.21.15.28:0/2992291667 wait complete. 2024-04-21T08:59:46.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.679+0000 7f0583b82740 20 client.17889 trim_cache size 0 max 16384 2024-04-21T08:59:46.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.696+0000 7f0583b82740 1 Processor -- start 2024-04-21T08:59:46.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.696+0000 7f0583b82740 1 -- start start 2024-04-21T08:59:46.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.696+0000 7f0583b82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7580 0x55f258cd0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.696+0000 7f0583b82740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7ea0 0x55f258ce0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.696+0000 7f0583b82740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 0x55f258cc37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.696+0000 7f0583b82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f057c1afa60 con 0x55f258cc7580 2024-04-21T08:59:46.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.696+0000 7f0583b82740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f057c1ad330 con 0x55f258cc7ea0 2024-04-21T08:59:46.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.696+0000 7f0583b82740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f057c1ab1a0 con 0x55f258cd0a80 2024-04-21T08:59:46.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.697+0000 7f057a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7580 0x55f258cd0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.697+0000 7f057a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7580 0x55f258cd0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46474/0 (socket says 172.21.15.28:46474) 2024-04-21T08:59:46.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.697+0000 7f057a59c640 1 -- 172.21.15.28:0/2410822666 learned_addr learned my addr 172.21.15.28:0/2410822666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:46.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.697+0000 7f0579d9b640 1 --2- 172.21.15.28:0/2410822666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7ea0 0x55f258ce0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.697+0000 7f057ad9d640 1 --2- 172.21.15.28:0/2410822666 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 0x55f258cc37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.697+0000 7f057a59c640 1 -- 172.21.15.28:0/2410822666 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 msgr2=0x55f258cc37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.697+0000 7f057a59c640 1 --2- 172.21.15.28:0/2410822666 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 0x55f258cc37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.697+0000 7f057a59c640 1 -- 172.21.15.28:0/2410822666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7ea0 msgr2=0x55f258ce0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.697+0000 7f057a59c640 1 --2- 172.21.15.28:0/2410822666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7ea0 0x55f258ce0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.697+0000 7f057a59c640 1 -- 172.21.15.28:0/2410822666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f258ccc020 con 0x55f258cc7580 2024-04-21T08:59:46.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.697+0000 7f057ad9d640 1 --2- 172.21.15.28:0/2410822666 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 0x55f258cc37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:59:46.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.697+0000 7f057a59c640 1 --2- 172.21.15.28:0/2410822666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7580 0x55f258cd0200 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x55f258cc7050 tx=0x7f056c00b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3c4073bc6713c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:46.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.697+0000 7f0563fff640 1 -- 172.21.15.28:0/2410822666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056c015050 con 0x55f258cc7580 2024-04-21T08:59:46.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.697+0000 7f0563fff640 1 -- 172.21.15.28:0/2410822666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f056c0054d0 con 0x55f258cc7580 2024-04-21T08:59:46.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.697+0000 7f0563fff640 1 -- 172.21.15.28:0/2410822666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056c011a30 con 0x55f258cc7580 2024-04-21T08:59:46.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 -- 172.21.15.28:0/2410822666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7580 msgr2=0x55f258cd0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 --2- 172.21.15.28:0/2410822666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7580 0x55f258cd0200 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x55f258cc7050 tx=0x7f056c00b5e0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 -- 172.21.15.28:0/2410822666 shutdown_connections 2024-04-21T08:59:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 --2- 172.21.15.28:0/2410822666 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cd0a80 0x55f258cc37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 --2- 172.21.15.28:0/2410822666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cc7ea0 0x55f258ce0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 --2- 172.21.15.28:0/2410822666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7580 0x55f258cd0200 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 -- 172.21.15.28:0/2410822666 >> 172.21.15.28:0/2410822666 conn(0x55f258cc1cb0 msgr2=0x55f258cc30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 -- 172.21.15.28:0/2410822666 shutdown_connections 2024-04-21T08:59:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 -- 172.21.15.28:0/2410822666 wait complete. 2024-04-21T08:59:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 -- start start 2024-04-21T08:59:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 Processor -- start 2024-04-21T08:59:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cc7580 0x55f258cdd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cdd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cd0a80 0x55f258cda0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f057c1ab7f0 con 0x55f258cc7ea0 2024-04-21T08:59:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f057c1acc60 con 0x55f258cd0a80 2024-04-21T08:59:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.698+0000 7f0583b82740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f057c1abec0 con 0x55f258cc7580 2024-04-21T08:59:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0583b82740 10 client.0 _ll_register_callbacks cb 0x7f05640bb180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:59:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0579d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cdd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0579d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cdd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46484/0 (socket says 172.21.15.28:46484) 2024-04-21T08:59:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0579d9b640 1 -- 172.21.15.28:0/3837141681 learned_addr learned my addr 172.21.15.28:0/3837141681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T08:59:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f057ad9d640 1 --2- 172.21.15.28:0/3837141681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cd0a80 0x55f258cda0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f057a59c640 1 --2- 172.21.15.28:0/3837141681 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cc7580 0x55f258cdd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0579d9b640 1 -- 172.21.15.28:0/3837141681 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cc7580 msgr2=0x55f258cdd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0579d9b640 1 --2- 172.21.15.28:0/3837141681 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cc7580 0x55f258cdd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0579d9b640 1 -- 172.21.15.28:0/3837141681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cd0a80 msgr2=0x55f258cda0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:59:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0579d9b640 1 --2- 172.21.15.28:0/3837141681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cd0a80 0x55f258cda0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0579d9b640 1 -- 172.21.15.28:0/3837141681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f258cda710 con 0x55f258cc7ea0 2024-04-21T08:59:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f057ad9d640 1 --2- 172.21.15.28:0/3837141681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cd0a80 0x55f258cda0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:59:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0579d9b640 1 --2- 172.21.15.28:0/3837141681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cdd640 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f05640f06e0 tx=0x7f05640be9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb93216082c8a1b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:59:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0561ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T08:59:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0561ffb640 1 -- 172.21.15.28:0/3837141681 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05640c3230 con 0x55f258cc7ea0 2024-04-21T08:59:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0583b82740 1 -- 172.21.15.28:0/3837141681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f056c0054d0 con 0x55f258cc7ea0 2024-04-21T08:59:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0561ffb640 1 -- 172.21.15.28:0/3837141681 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9533+0+0 (secure 0 0 0) 0x7f05640c7660 con 0x55f258cc7ea0 2024-04-21T08:59:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0561ffb640 1 -- 172.21.15.28:0/3837141681 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f05640c7860 con 0x55f258cc7ea0 2024-04-21T08:59:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0583b82740 10 client.17895 fetch_fsmap learned FSMap version 28 2024-04-21T08:59:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.699+0000 7f0583b82740 1 -- 172.21.15.28:0/3837141681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f258d4bc50 con 0x55f258cc7ea0 2024-04-21T08:59:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.700+0000 7f0561ffb640 1 -- 172.21.15.28:0/3837141681 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f056c0054d0 con 0x55f258cc7ea0 2024-04-21T08:59:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.700+0000 7f0561ffb640 1 -- 172.21.15.28:0/3837141681 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f05640be3d0 con 0x55f258cc7ea0 2024-04-21T08:59:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.700+0000 7f0583b82740 10 client.17895 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T08:59:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.700+0000 7f0583b82740 10 client.17895 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:59:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.700+0000 7f0583b82740 20 client.17895 populate_metadata read hostname 'smithi028' 2024-04-21T08:59:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.700+0000 7f0583b82740 1 --2- 172.21.15.28:0/3837141681 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f258d4c1e0 0x55f258d4e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:59:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.700+0000 7f0583b82740 4 client.17895 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:59:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.700+0000 7f0583b82740 1 -- 172.21.15.28:0/3837141681 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f258c33a70 con 0x55f258d4c1e0 2024-04-21T08:59:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.700+0000 7f057a59c640 1 --2- 172.21.15.28:0/3837141681 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f258d4c1e0 0x55f258d4e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:59:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.700+0000 7f0561ffb640 10 client.17895 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.700+0000 7f057a59c640 1 --2- 172.21.15.28:0/3837141681 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f258d4c1e0 0x55f258d4e5d0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53ef9153daa061a server_cookie=17567f7d823e6c5e in_seq=0 out_seq=0 2024-04-21T08:59:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0561ffb640 1 -- 172.21.15.28:0/3837141681 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f258c33a70 con 0x55f258d4c1e0 2024-04-21T08:59:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0561ffb640 10 client.17895 handle_command_reply: tid=0 2024-04-21T08:59:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0583b82740 1 client.17895 shutdown 2024-04-21T08:59:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0583b82740 1 -- 172.21.15.28:0/3837141681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 msgr2=0x55f258cdd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0583b82740 1 --2- 172.21.15.28:0/3837141681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cdd640 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f05640f06e0 tx=0x7f05640be9c0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0583b82740 1 -- 172.21.15.28:0/3837141681 shutdown_connections 2024-04-21T08:59:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0583b82740 1 --2- 172.21.15.28:0/3837141681 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55f258d4c1e0 0x55f258d4e5d0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0583b82740 1 --2- 172.21.15.28:0/3837141681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f258cd0a80 0x55f258cda0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0561ffb640 0 client.17895 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T08:59:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0583b82740 1 --2- 172.21.15.28:0/3837141681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f258cc7ea0 0x55f258cdd640 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0583b82740 1 --2- 172.21.15.28:0/3837141681 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f258cc7580 0x55f258cdd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0583b82740 1 -- 172.21.15.28:0/3837141681 >> 172.21.15.28:0/3837141681 conn(0x55f258cc1cb0 msgr2=0x55f258cc0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0583b82740 1 -- 172.21.15.28:0/3837141681 shutdown_connections 2024-04-21T08:59:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0583b82740 1 -- 172.21.15.28:0/3837141681 wait complete. 2024-04-21T08:59:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0583b82740 20 client.17895 trim_cache size 0 max 16384 2024-04-21T08:59:46.712 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T08:59:46.712 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:59:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T08:59:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: "4d607d1a-9f7d-44f2-840a-0ddc2e551163": { 2024-04-21T08:59:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T08:59:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "4d607d1a-9f7d-44f2-840a-0ddc2e551163", 2024-04-21T08:59:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T08:59:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:59:46.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T08:59:46.713 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T08:59:46.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0580f98640 1 -- 172.21.15.28:0/2678987494 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f05680a3d20 msgr2=0x7f05680a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0580f98640 1 --2- 172.21.15.28:0/2678987494 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f05680a3d20 0x7f05680a61e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0574005fd0 tx=0x7f05740046c0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0580f98640 1 -- 172.21.15.28:0/2678987494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057c06af50 msgr2=0x7f057c198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:59:46.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0580f98640 1 --2- 172.21.15.28:0/2678987494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057c06af50 0x7f057c198bd0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f0564005e30 tx=0x7f05640039f0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0580f98640 1 -- 172.21.15.28:0/2678987494 shutdown_connections 2024-04-21T08:59:46.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0580f98640 1 --2- 172.21.15.28:0/2678987494 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f05680a3d20 0x7f05680a61e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0580f98640 1 --2- 172.21.15.28:0/2678987494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f057c108040 0x7f057c19cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0580f98640 1 --2- 172.21.15.28:0/2678987494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057c06af50 0x7f057c198bd0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0580f98640 1 --2- 172.21.15.28:0/2678987494 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f057c06a5b0 0x7f057c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:59:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0580f98640 1 -- 172.21.15.28:0/2678987494 >> 172.21.15.28:0/2678987494 conn(0x7f057c077c40 msgr2=0x7f057c0ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:59:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0580f98640 1 -- 172.21.15.28:0/2678987494 shutdown_connections 2024-04-21T08:59:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T08:59:46.701+0000 7f0580f98640 1 -- 172.21.15.28:0/2678987494 wait complete. 2024-04-21T08:59:46.716 INFO:tasks.cephfs.filesystem:scrub status for tag:4d607d1a-9f7d-44f2-840a-0ddc2e551163 - {'path': '/', 'tag': '4d607d1a-9f7d-44f2-840a-0ddc2e551163', 'options': 'recursive,force'} 2024-04-21T08:59:47.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T08:59:47.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:47.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:47.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:47.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:47.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:47.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:47.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:47.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:47.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:47.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T08:59:47.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:47 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T08:59:47.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:47.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:47.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:47.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:47.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:47.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:47.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:47.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:47.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:47.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T08:59:47.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:47 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T08:59:47.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:47.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:47.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:47.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:47.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:47.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:47.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:47.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:47.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:47.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T08:59:47.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:47.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:47 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T08:59:48.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:48 smithi152 ceph-mon[31401]: pgmap v265: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 8 op/s 2024-04-21T08:59:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:48 smithi028 ceph-mon[28003]: pgmap v265: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 8 op/s 2024-04-21T08:59:48.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:48 smithi171 ceph-mon[31358]: pgmap v265: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 8 op/s 2024-04-21T08:59:50.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:49 smithi152 ceph-mon[31401]: pgmap v266: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 24 op/s 2024-04-21T08:59:50.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:49 smithi028 ceph-mon[28003]: pgmap v266: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 24 op/s 2024-04-21T08:59:50.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:49 smithi171 ceph-mon[31358]: pgmap v266: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 24 op/s 2024-04-21T08:59:52.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:52 smithi152 ceph-mon[31401]: pgmap v267: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 op/s 2024-04-21T08:59:52.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:52 smithi028 ceph-mon[28003]: pgmap v267: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 op/s 2024-04-21T08:59:52.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:52 smithi171 ceph-mon[31358]: pgmap v267: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 op/s 2024-04-21T08:59:54.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:54 smithi152 ceph-mon[31401]: pgmap v268: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:59:54.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:54 smithi028 ceph-mon[28003]: pgmap v268: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:59:54.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:54 smithi171 ceph-mon[31358]: pgmap v268: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:59:56.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:56 smithi152 ceph-mon[31401]: pgmap v269: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 28 op/s 2024-04-21T08:59:56.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:56 smithi028 ceph-mon[28003]: pgmap v269: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 28 op/s 2024-04-21T08:59:56.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:56 smithi171 ceph-mon[31358]: pgmap v269: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 28 op/s 2024-04-21T08:59:57.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:57 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:59:57.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:57 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:59:57.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:57 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:59:57.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:57 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:59:57.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:57 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:59:57.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:57 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:59:58.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:58 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T08:59:58.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 08:59:58 smithi171 ceph-mon[31358]: pgmap v270: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:59:58.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:58 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T08:59:58.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 08:59:58 smithi028 ceph-mon[28003]: pgmap v270: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T08:59:58.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:58 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T08:59:58.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 08:59:58 smithi152 ceph-mon[31401]: pgmap v270: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 29 op/s 2024-04-21T09:00:00.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:00 smithi171 ceph-mon[31358]: pgmap v271: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 26 op/s 2024-04-21T09:00:00.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:00 smithi171 ceph-mon[31358]: overall HEALTH_OK 2024-04-21T09:00:00.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:00 smithi152 ceph-mon[31401]: pgmap v271: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 26 op/s 2024-04-21T09:00:00.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:00 smithi152 ceph-mon[31401]: overall HEALTH_OK 2024-04-21T09:00:00.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:00 smithi028 ceph-mon[28003]: pgmap v271: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 26 op/s 2024-04-21T09:00:00.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:00 smithi028 ceph-mon[28003]: overall HEALTH_OK 2024-04-21T09:00:02.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:02 smithi171 ceph-mon[31358]: pgmap v272: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 10 op/s 2024-04-21T09:00:02.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:02 smithi152 ceph-mon[31401]: pgmap v272: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 10 op/s 2024-04-21T09:00:02.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:02 smithi028 ceph-mon[28003]: pgmap v272: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 10 op/s 2024-04-21T09:00:04.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:04 smithi171 ceph-mon[31358]: pgmap v273: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T09:00:04.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:04 smithi152 ceph-mon[31401]: pgmap v273: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T09:00:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:04 smithi028 ceph-mon[28003]: pgmap v273: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T09:00:06.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:06 smithi171 ceph-mon[31358]: pgmap v274: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T09:00:06.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:06 smithi152 ceph-mon[31401]: pgmap v274: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T09:00:06.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:06 smithi028 ceph-mon[28003]: pgmap v274: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T09:00:08.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:08 smithi152 ceph-mon[31401]: pgmap v275: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T09:00:08.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:08 smithi152 ceph-mon[31401]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T09:00:08.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:08 smithi028 ceph-mon[28003]: pgmap v275: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T09:00:08.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:08 smithi028 ceph-mon[28003]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T09:00:08.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:08 smithi171 ceph-mon[31358]: pgmap v275: 129 pgs: 129 active+clean; 18 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T09:00:08.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:08 smithi171 ceph-mon[31358]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T09:00:10.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:10 smithi152 ceph-mon[31401]: pgmap v277: 129 pgs: 12 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 85 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-21T09:00:10.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:10 smithi028 ceph-mon[28003]: pgmap v277: 129 pgs: 12 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 85 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-21T09:00:10.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:10 smithi171 ceph-mon[31358]: pgmap v277: 129 pgs: 12 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 85 active+clean; 18 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-21T09:00:12.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:12 smithi152 ceph-mon[31401]: pgmap v278: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 17 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-21T09:00:12.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:12 smithi028 ceph-mon[28003]: pgmap v278: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 17 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-21T09:00:12.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:12 smithi171 ceph-mon[31358]: pgmap v278: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 17 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-21T09:00:14.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:14 smithi152 ceph-mon[31401]: pgmap v279: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 16 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-04-21T09:00:14.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:14 smithi028 ceph-mon[28003]: pgmap v279: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 16 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-04-21T09:00:14.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:14 smithi171 ceph-mon[31358]: pgmap v279: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 16 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-04-21T09:00:16.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:16 smithi152 ceph-mon[31401]: pgmap v280: 129 pgs: 14 active+clean+snaptrim, 43 active+clean+snaptrim_wait, 72 active+clean; 15 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T09:00:16.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:16 smithi028 ceph-mon[28003]: pgmap v280: 129 pgs: 14 active+clean+snaptrim, 43 active+clean+snaptrim_wait, 72 active+clean; 15 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T09:00:16.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:16 smithi171 ceph-mon[31358]: pgmap v280: 129 pgs: 14 active+clean+snaptrim, 43 active+clean+snaptrim_wait, 72 active+clean; 15 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T09:00:16.717 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T09:00:16.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T09:00:16.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.836+0000 7f9524014640 1 -- 172.21.15.28:0/1779230500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c107270 msgr2=0x7f951c10ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:16.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.836+0000 7f9524014640 1 --2- 172.21.15.28:0/1779230500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c107270 0x7f951c10ee40 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f9514009900 tx=0x7f951402bc30 comp rx=0 tx=0).stop 2024-04-21T09:00:16.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.836+0000 7f9524014640 1 -- 172.21.15.28:0/1779230500 shutdown_connections 2024-04-21T09:00:16.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.836+0000 7f9524014640 1 --2- 172.21.15.28:0/1779230500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c107270 0x7f951c10ee40 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.836+0000 7f9524014640 1 --2- 172.21.15.28:0/1779230500 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f951c102b50 0x7f951c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.836+0000 7f9524014640 1 --2- 172.21.15.28:0/1779230500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f951c1021a0 0x7f951c102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.836+0000 7f9524014640 1 -- 172.21.15.28:0/1779230500 >> 172.21.15.28:0/1779230500 conn(0x7f951c0fda40 msgr2=0x7f951c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.837+0000 7f9524014640 1 -- 172.21.15.28:0/1779230500 shutdown_connections 2024-04-21T09:00:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.837+0000 7f9524014640 1 -- 172.21.15.28:0/1779230500 wait complete. 2024-04-21T09:00:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.837+0000 7f9524014640 1 Processor -- start 2024-04-21T09:00:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.837+0000 7f9524014640 1 -- start start 2024-04-21T09:00:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.837+0000 7f9524014640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f951c1021a0 0x7f951c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.837+0000 7f9524014640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c107270 0x7f951c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.837+0000 7f9524014640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f951c074cd0 0x7f951c1a6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.837+0000 7f9524014640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f951c1121e0 con 0x7f951c107270 2024-04-21T09:00:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.837+0000 7f9524014640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f951c112060 con 0x7f951c1021a0 2024-04-21T09:00:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.837+0000 7f9524014640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f951c112360 con 0x7f951c074cd0 2024-04-21T09:00:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f95215af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c107270 0x7f951c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f95215af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c107270 0x7f951c074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38766/0 (socket says 172.21.15.28:38766) 2024-04-21T09:00:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f95215af640 1 -- 172.21.15.28:0/1359905961 learned_addr learned my addr 172.21.15.28:0/1359905961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f9521db0640 1 --2- 172.21.15.28:0/1359905961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f951c1021a0 0x7f951c074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f95225b1640 1 --2- 172.21.15.28:0/1359905961 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f951c074cd0 0x7f951c1a6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f95215af640 1 -- 172.21.15.28:0/1359905961 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f951c074cd0 msgr2=0x7f951c1a6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f95215af640 1 --2- 172.21.15.28:0/1359905961 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f951c074cd0 0x7f951c1a6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f95215af640 1 -- 172.21.15.28:0/1359905961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f951c1021a0 msgr2=0x7f951c074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f95215af640 1 --2- 172.21.15.28:0/1359905961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f951c1021a0 0x7f951c074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f95215af640 1 -- 172.21.15.28:0/1359905961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f951c1a7030 con 0x7f951c107270 2024-04-21T09:00:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f9521db0640 1 --2- 172.21.15.28:0/1359905961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f951c1021a0 0x7f951c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T09:00:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f95215af640 1 --2- 172.21.15.28:0/1359905961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c107270 0x7f951c074790 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f950c00d740 tx=0x7f950c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f9512ffd640 1 -- 172.21.15.28:0/1359905961 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950c017070 con 0x7f951c107270 2024-04-21T09:00:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f9524014640 1 -- 172.21.15.28:0/1359905961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f951c1a7320 con 0x7f951c107270 2024-04-21T09:00:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f9512ffd640 1 -- 172.21.15.28:0/1359905961 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f950c00de50 con 0x7f951c107270 2024-04-21T09:00:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f9512ffd640 1 -- 172.21.15.28:0/1359905961 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950c004ce0 con 0x7f951c107270 2024-04-21T09:00:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.838+0000 7f9524014640 1 -- 172.21.15.28:0/1359905961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f951c1a7870 con 0x7f951c107270 2024-04-21T09:00:16.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.841+0000 7f9512ffd640 1 -- 172.21.15.28:0/1359905961 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f950c004e80 con 0x7f951c107270 2024-04-21T09:00:16.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.841+0000 7f9526c06740 1 Processor -- start 2024-04-21T09:00:16.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.841+0000 7f9526c06740 1 -- start start 2024-04-21T09:00:16.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.841+0000 7f9526c06740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2f1580 0x557e2b2f1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.841+0000 7f9526c06740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1ea0 0x557e2b2fa540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.841+0000 7f9512ffd640 1 --2- 172.21.15.28:0/1359905961 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95080a3d50 0x7f95080a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.841+0000 7f9526c06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2faa80 0x557e2b2fce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.841+0000 7f9526c06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557e2b2561e0 con 0x557e2b2faa80 2024-04-21T09:00:16.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.841+0000 7f9526c06740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x557e2b1f3390 con 0x557e2b2f1ea0 2024-04-21T09:00:16.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.841+0000 7f9526c06740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x557e2b2521a0 con 0x557e2b2f1580 2024-04-21T09:00:16.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.842+0000 7f95225b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2faa80 0x557e2b2fce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.842+0000 7f9521db0640 1 --2- 172.21.15.28:0/1359905961 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95080a3d50 0x7f95080a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.842+0000 7f95225b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2faa80 0x557e2b2fce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38780/0 (socket says 172.21.15.28:38780) 2024-04-21T09:00:16.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.842+0000 7f95225b1640 1 -- 172.21.15.28:0/2436052385 learned_addr learned my addr 172.21.15.28:0/2436052385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:16.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.842+0000 7f9512ffd640 1 -- 172.21.15.28:0/1359905961 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f950c0d04a0 con 0x7f951c107270 2024-04-21T09:00:16.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.842+0000 7f95225b1640 1 --2- 172.21.15.28:0/2436052385 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2f1580 0x557e2b2f1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.842+0000 7f95215af640 1 --2- 172.21.15.28:0/2436052385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1ea0 0x557e2b2fa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.842+0000 7f95225b1640 1 -- 172.21.15.28:0/2436052385 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2f1580 msgr2=0x557e2b2f1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.842+0000 7f95225b1640 1 --2- 172.21.15.28:0/2436052385 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2f1580 0x557e2b2f1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.842+0000 7f95225b1640 1 -- 172.21.15.28:0/2436052385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1ea0 msgr2=0x557e2b2fa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.842+0000 7f95225b1640 1 --2- 172.21.15.28:0/2436052385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1ea0 0x557e2b2fa540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.842+0000 7f95225b1640 1 -- 172.21.15.28:0/2436052385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e2b0327d0 con 0x557e2b2faa80 2024-04-21T09:00:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.842+0000 7f9521db0640 1 --2- 172.21.15.28:0/1359905961 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95080a3d50 0x7f95080a6210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9518005fd0 tx=0x7f9518009500 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.842+0000 7f95225b1640 1 --2- 172.21.15.28:0/2436052385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2faa80 0x557e2b2fce70 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f9514036040 tx=0x7f9514005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1474827d88744769 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.842+0000 7f9507fff640 1 -- 172.21.15.28:0/2436052385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951404a070 con 0x557e2b2faa80 2024-04-21T09:00:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.843+0000 7f9507fff640 1 -- 172.21.15.28:0/2436052385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f951402cc90 con 0x557e2b2faa80 2024-04-21T09:00:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.843+0000 7f9507fff640 1 -- 172.21.15.28:0/2436052385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514003de0 con 0x557e2b2faa80 2024-04-21T09:00:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.843+0000 7f9526c06740 1 -- 172.21.15.28:0/2436052385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2faa80 msgr2=0x557e2b2fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.843+0000 7f9526c06740 1 --2- 172.21.15.28:0/2436052385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2faa80 0x557e2b2fce70 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f9514036040 tx=0x7f9514005e50 comp rx=0 tx=0).stop 2024-04-21T09:00:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.843+0000 7f9526c06740 1 -- 172.21.15.28:0/2436052385 shutdown_connections 2024-04-21T09:00:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.843+0000 7f9526c06740 1 --2- 172.21.15.28:0/2436052385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2faa80 0x557e2b2fce70 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.843+0000 7f9526c06740 1 --2- 172.21.15.28:0/2436052385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1ea0 0x557e2b2fa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.843+0000 7f9526c06740 1 --2- 172.21.15.28:0/2436052385 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2f1580 0x557e2b2f1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.843+0000 7f9526c06740 1 -- 172.21.15.28:0/2436052385 >> 172.21.15.28:0/2436052385 conn(0x557e2b2ebcb0 msgr2=0x557e2b2ec090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:16.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.843+0000 7f9526c06740 1 -- 172.21.15.28:0/2436052385 shutdown_connections 2024-04-21T09:00:16.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.843+0000 7f9526c06740 1 -- 172.21.15.28:0/2436052385 wait complete. 2024-04-21T09:00:16.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.844+0000 7f9526c06740 1 -- start start 2024-04-21T09:00:16.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.844+0000 7f9526c06740 1 Processor -- start 2024-04-21T09:00:16.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.844+0000 7f9526c06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2f1580 0x557e2b2f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.844+0000 7f9526c06740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1ea0 0x557e2b2f80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.844+0000 7f9526c06740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2faa80 0x557e2b2f85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.844+0000 7f9526c06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557e2b2fd810 con 0x557e2b2f1580 2024-04-21T09:00:16.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.844+0000 7f9526c06740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x557e2b20f2f0 con 0x557e2b2f1ea0 2024-04-21T09:00:16.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.844+0000 7f9526c06740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x557e2b2fd990 con 0x557e2b2faa80 2024-04-21T09:00:16.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.844+0000 7f9526c06740 10 client.0 _ll_register_callbacks cb 0x557e2b21c580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:16.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.844+0000 7f9521db0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2f1580 0x557e2b2f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.844+0000 7f9521db0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2f1580 0x557e2b2f7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38786/0 (socket says 172.21.15.28:38786) 2024-04-21T09:00:16.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.844+0000 7f9521db0640 1 -- 172.21.15.28:0/3738110794 learned_addr learned my addr 172.21.15.28:0/3738110794 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:16.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.844+0000 7f95215af640 1 --2- 172.21.15.28:0/3738110794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1ea0 0x557e2b2f80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.845+0000 7f95225b1640 1 --2- 172.21.15.28:0/3738110794 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2faa80 0x557e2b2f85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.845+0000 7f9521db0640 1 -- 172.21.15.28:0/3738110794 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2faa80 msgr2=0x557e2b2f85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.845+0000 7f9521db0640 1 --2- 172.21.15.28:0/3738110794 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2faa80 0x557e2b2f85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.845+0000 7f9521db0640 1 -- 172.21.15.28:0/3738110794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1ea0 msgr2=0x557e2b2f80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.845+0000 7f9521db0640 1 --2- 172.21.15.28:0/3738110794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1ea0 0x557e2b2f80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.845+0000 7f9521db0640 1 -- 172.21.15.28:0/3738110794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e2af683f0 con 0x557e2b2f1580 2024-04-21T09:00:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.845+0000 7f9521db0640 1 --2- 172.21.15.28:0/3738110794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2f1580 0x557e2b2f7b70 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f9518003dc0 tx=0x7f9518004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3002aee54083fd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.845+0000 7f95057fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.845+0000 7f95057fa640 1 -- 172.21.15.28:0/3738110794 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951801e830 con 0x557e2b2f1580 2024-04-21T09:00:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.845+0000 7f95057fa640 1 -- 172.21.15.28:0/3738110794 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f951801e9d0 con 0x557e2b2f1580 2024-04-21T09:00:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.845+0000 7f9526c06740 1 -- 172.21.15.28:0/3738110794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f951402cc90 con 0x557e2b2f1580 2024-04-21T09:00:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.845+0000 7f95057fa640 1 -- 172.21.15.28:0/3738110794 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f951802f050 con 0x557e2b2f1580 2024-04-21T09:00:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.845+0000 7f9526c06740 10 client.17907 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.845+0000 7f9526c06740 1 -- 172.21.15.28:0/3738110794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557e2af280f0 con 0x557e2b2f1580 2024-04-21T09:00:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.846+0000 7f95057fa640 1 -- 172.21.15.28:0/3738110794 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f951402cc90 con 0x557e2b2f1580 2024-04-21T09:00:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.846+0000 7f95057fa640 1 -- 172.21.15.28:0/3738110794 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f9518026c20 con 0x557e2b2f1580 2024-04-21T09:00:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.846+0000 7f9526c06740 10 client.17907 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:16.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.846+0000 7f9526c06740 10 client.17907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:16.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.846+0000 7f9526c06740 20 client.17907 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:16.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.846+0000 7f9526c06740 1 --2- 172.21.15.28:0/3738110794 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x557e2b30ab30 0x557e2b30cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.846+0000 7f9526c06740 4 client.17907 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:00:16.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.846+0000 7f9526c06740 1 -- 172.21.15.28:0/3738110794 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e2b263c00 con 0x557e2b30ab30 2024-04-21T09:00:16.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.846+0000 7f95215af640 1 --2- 172.21.15.28:0/3738110794 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x557e2b30ab30 0x557e2b30cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.847+0000 7f95215af640 1 --2- 172.21.15.28:0/3738110794 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x557e2b30ab30 0x557e2b30cfa0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df321045a2cffbab server_cookie=1aaf1699cf2db38a in_seq=0 out_seq=0 2024-04-21T09:00:16.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.847+0000 7f95057fa640 10 client.17907 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:16.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.847+0000 7f95057fa640 1 -- 172.21.15.28:0/3738110794 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557e2b263c00 con 0x557e2b30ab30 2024-04-21T09:00:16.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.847+0000 7f95057fa640 10 client.17907 handle_command_reply: tid=0 2024-04-21T09:00:16.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.847+0000 7f9526c06740 1 client.17907 shutdown 2024-04-21T09:00:16.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.847+0000 7f9526c06740 1 -- 172.21.15.28:0/3738110794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2f1580 msgr2=0x557e2b2f7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:16.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.847+0000 7f9526c06740 1 --2- 172.21.15.28:0/3738110794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2f1580 0x557e2b2f7b70 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f9518003dc0 tx=0x7f9518004ab0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.847+0000 7f9526c06740 1 -- 172.21.15.28:0/3738110794 shutdown_connections 2024-04-21T09:00:16.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.847+0000 7f9526c06740 1 --2- 172.21.15.28:0/3738110794 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x557e2b30ab30 0x557e2b30cfa0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.847+0000 7f9526c06740 1 --2- 172.21.15.28:0/3738110794 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2faa80 0x557e2b2f85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.847+0000 7f95057fa640 0 client.17907 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:16.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.847+0000 7f9526c06740 1 --2- 172.21.15.28:0/3738110794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1ea0 0x557e2b2f80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.847+0000 7f9526c06740 1 --2- 172.21.15.28:0/3738110794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2f1580 0x557e2b2f7b70 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.847+0000 7f9526c06740 1 -- 172.21.15.28:0/3738110794 >> 172.21.15.28:0/3738110794 conn(0x557e2b2ebcb0 msgr2=0x557e2b2eda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:16.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.848+0000 7f9526c06740 1 -- 172.21.15.28:0/3738110794 shutdown_connections 2024-04-21T09:00:16.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.848+0000 7f9526c06740 1 -- 172.21.15.28:0/3738110794 wait complete. 2024-04-21T09:00:16.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.848+0000 7f9526c06740 20 client.17907 trim_cache size 0 max 16384 2024-04-21T09:00:16.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.865+0000 7f9526c06740 1 Processor -- start 2024-04-21T09:00:16.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.865+0000 7f9526c06740 1 -- start start 2024-04-21T09:00:16.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.865+0000 7f9526c06740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1580 0x557e2b2fa2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f9526c06740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2f1ea0 0x557e2b3066d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f9526c06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2faa80 0x557e2b2eabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f9526c06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f951c1b40c0 con 0x557e2b2faa80 2024-04-21T09:00:16.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f9526c06740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f951c1b1990 con 0x557e2b2f1580 2024-04-21T09:00:16.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f9526c06740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f951c1af800 con 0x557e2b2f1ea0 2024-04-21T09:00:16.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f95225b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2faa80 0x557e2b2eabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f95225b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2faa80 0x557e2b2eabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38790/0 (socket says 172.21.15.28:38790) 2024-04-21T09:00:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f95225b1640 1 -- 172.21.15.28:0/3509601425 learned_addr learned my addr 172.21.15.28:0/3509601425 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f9521db0640 1 --2- 172.21.15.28:0/3509601425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1580 0x557e2b2fa2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f95215af640 1 --2- 172.21.15.28:0/3509601425 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2f1ea0 0x557e2b3066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f95225b1640 1 -- 172.21.15.28:0/3509601425 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2f1ea0 msgr2=0x557e2b3066d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f95225b1640 1 --2- 172.21.15.28:0/3509601425 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2f1ea0 0x557e2b3066d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f95225b1640 1 -- 172.21.15.28:0/3509601425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1580 msgr2=0x557e2b2fa2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f95225b1640 1 --2- 172.21.15.28:0/3509601425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1580 0x557e2b2fa2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f95225b1640 1 -- 172.21.15.28:0/3509601425 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e2b2ecee0 con 0x557e2b2faa80 2024-04-21T09:00:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f95225b1640 1 --2- 172.21.15.28:0/3509601425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2faa80 0x557e2b2eabf0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x557e2b2ece40 tx=0x7f9514005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df26d62760dc8d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f95077fe640 1 -- 172.21.15.28:0/3509601425 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951404a040 con 0x557e2b2faa80 2024-04-21T09:00:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f95077fe640 1 -- 172.21.15.28:0/3509601425 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f951402c9e0 con 0x557e2b2faa80 2024-04-21T09:00:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.866+0000 7f95077fe640 1 -- 172.21.15.28:0/3509601425 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95140049f0 con 0x557e2b2faa80 2024-04-21T09:00:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.867+0000 7f9526c06740 1 -- 172.21.15.28:0/3509601425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2faa80 msgr2=0x557e2b2eabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.867+0000 7f9526c06740 1 --2- 172.21.15.28:0/3509601425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2faa80 0x557e2b2eabf0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x557e2b2ece40 tx=0x7f9514005fd0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.867+0000 7f9526c06740 1 -- 172.21.15.28:0/3509601425 shutdown_connections 2024-04-21T09:00:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.867+0000 7f9526c06740 1 --2- 172.21.15.28:0/3509601425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2faa80 0x557e2b2eabf0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.867+0000 7f9526c06740 1 --2- 172.21.15.28:0/3509601425 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2f1ea0 0x557e2b3066d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.867+0000 7f9526c06740 1 --2- 172.21.15.28:0/3509601425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1580 0x557e2b2fa2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.867+0000 7f9526c06740 1 -- 172.21.15.28:0/3509601425 >> 172.21.15.28:0/3509601425 conn(0x557e2b2ebcb0 msgr2=0x557e2b2ea4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:16.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.867+0000 7f9526c06740 1 -- 172.21.15.28:0/3509601425 shutdown_connections 2024-04-21T09:00:16.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.867+0000 7f9526c06740 1 -- 172.21.15.28:0/3509601425 wait complete. 2024-04-21T09:00:16.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.867+0000 7f9526c06740 1 -- start start 2024-04-21T09:00:16.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9526c06740 1 Processor -- start 2024-04-21T09:00:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9526c06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2f1580 0x557e2b2f48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9526c06740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1ea0 0x557e2b2f4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9526c06740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2faa80 0x557e2b2f5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9526c06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f951c1afe50 con 0x557e2b2f1580 2024-04-21T09:00:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9526c06740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f951c1b12c0 con 0x557e2b2f1ea0 2024-04-21T09:00:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9526c06740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f951c1b0520 con 0x557e2b2faa80 2024-04-21T09:00:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9526c06740 10 client.0 _ll_register_callbacks cb 0x7f951800fa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9521db0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2f1580 0x557e2b2f48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9521db0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2f1580 0x557e2b2f48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38800/0 (socket says 172.21.15.28:38800) 2024-04-21T09:00:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9521db0640 1 -- 172.21.15.28:0/2580446850 learned_addr learned my addr 172.21.15.28:0/2580446850 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f95215af640 1 --2- 172.21.15.28:0/2580446850 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1ea0 0x557e2b2f4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f95225b1640 1 --2- 172.21.15.28:0/2580446850 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2faa80 0x557e2b2f5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9521db0640 1 -- 172.21.15.28:0/2580446850 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2faa80 msgr2=0x557e2b2f5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9521db0640 1 --2- 172.21.15.28:0/2580446850 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2faa80 0x557e2b2f5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9521db0640 1 -- 172.21.15.28:0/2580446850 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1ea0 msgr2=0x557e2b2f4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9521db0640 1 --2- 172.21.15.28:0/2580446850 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1ea0 0x557e2b2f4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f9521db0640 1 -- 172.21.15.28:0/2580446850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e2b370160 con 0x557e2b2f1580 2024-04-21T09:00:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.868+0000 7f95225b1640 1 --2- 172.21.15.28:0/2580446850 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2faa80 0x557e2b2f5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f9521db0640 1 --2- 172.21.15.28:0/2580446850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2f1580 0x557e2b2f48d0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f951801d520 tx=0x7f9518003f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839bd2bfc7e53c65 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f95057fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f95057fa640 1 -- 172.21.15.28:0/2580446850 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f951801d7b0 con 0x557e2b2f1580 2024-04-21T09:00:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f95057fa640 1 -- 172.21.15.28:0/2580446850 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f951802c040 con 0x557e2b2f1580 2024-04-21T09:00:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f9526c06740 1 -- 172.21.15.28:0/2580446850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f951402c9e0 con 0x557e2b2f1580 2024-04-21T09:00:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f95057fa640 1 -- 172.21.15.28:0/2580446850 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9518047050 con 0x557e2b2f1580 2024-04-21T09:00:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f9526c06740 10 client.17913 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f9526c06740 1 -- 172.21.15.28:0/2580446850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557e2b371480 con 0x557e2b2f1580 2024-04-21T09:00:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f95057fa640 1 -- 172.21.15.28:0/2580446850 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f951402c9e0 con 0x557e2b2f1580 2024-04-21T09:00:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f95057fa640 1 -- 172.21.15.28:0/2580446850 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f951801e630 con 0x557e2b2f1580 2024-04-21T09:00:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f9526c06740 10 client.17913 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f9526c06740 10 client.17913 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f9526c06740 20 client.17913 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f9526c06740 1 --2- 172.21.15.28:0/2580446850 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x557e2b371c30 0x557e2b374050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f9526c06740 4 client.17913 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T09:00:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.869+0000 7f9526c06740 1 -- 172.21.15.28:0/2580446850 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557e2b25da70 con 0x557e2b371c30 2024-04-21T09:00:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f95215af640 1 --2- 172.21.15.28:0/2580446850 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x557e2b371c30 0x557e2b374050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f95215af640 1 --2- 172.21.15.28:0/2580446850 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x557e2b371c30 0x557e2b374050 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=879d7ffb4c06c634 server_cookie=ae35124003b0c662 in_seq=0 out_seq=0 2024-04-21T09:00:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f95057fa640 10 client.17913 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f95057fa640 1 -- 172.21.15.28:0/2580446850 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557e2b25da70 con 0x557e2b371c30 2024-04-21T09:00:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f95057fa640 10 client.17913 handle_command_reply: tid=0 2024-04-21T09:00:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f9526c06740 1 client.17913 shutdown 2024-04-21T09:00:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f9526c06740 1 -- 172.21.15.28:0/2580446850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2f1580 msgr2=0x557e2b2f48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f9526c06740 1 --2- 172.21.15.28:0/2580446850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2f1580 0x557e2b2f48d0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f951801d520 tx=0x7f9518003f30 comp rx=0 tx=0).stop 2024-04-21T09:00:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f9526c06740 1 -- 172.21.15.28:0/2580446850 shutdown_connections 2024-04-21T09:00:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f9526c06740 1 --2- 172.21.15.28:0/2580446850 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x557e2b371c30 0x557e2b374050 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f9526c06740 1 --2- 172.21.15.28:0/2580446850 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x557e2b2faa80 0x557e2b2f5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f9526c06740 1 --2- 172.21.15.28:0/2580446850 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557e2b2f1ea0 0x557e2b2f4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f9526c06740 1 --2- 172.21.15.28:0/2580446850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557e2b2f1580 0x557e2b2f48d0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f95057fa640 0 client.17913 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f9526c06740 1 -- 172.21.15.28:0/2580446850 >> 172.21.15.28:0/2580446850 conn(0x557e2b2ebcb0 msgr2=0x557e2b2f83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f9526c06740 1 -- 172.21.15.28:0/2580446850 shutdown_connections 2024-04-21T09:00:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f9526c06740 1 -- 172.21.15.28:0/2580446850 wait complete. 2024-04-21T09:00:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.870+0000 7f9526c06740 20 client.17913 trim_cache size 0 max 16384 2024-04-21T09:00:16.881 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T09:00:16.881 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T09:00:16.882 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T09:00:16.882 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T09:00:16.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.871+0000 7f9524014640 1 -- 172.21.15.28:0/1359905961 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95080a3d50 msgr2=0x7f95080a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:16.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.871+0000 7f9524014640 1 --2- 172.21.15.28:0/1359905961 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95080a3d50 0x7f95080a6210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9518005fd0 tx=0x7f9518009500 comp rx=0 tx=0).stop 2024-04-21T09:00:16.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.871+0000 7f9524014640 1 -- 172.21.15.28:0/1359905961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c107270 msgr2=0x7f951c074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:16.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.871+0000 7f9524014640 1 --2- 172.21.15.28:0/1359905961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c107270 0x7f951c074790 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f950c00d740 tx=0x7f950c00dc00 comp rx=0 tx=0).stop 2024-04-21T09:00:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.871+0000 7f9524014640 1 -- 172.21.15.28:0/1359905961 shutdown_connections 2024-04-21T09:00:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.871+0000 7f9524014640 1 --2- 172.21.15.28:0/1359905961 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95080a3d50 0x7f95080a6210 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.871+0000 7f9524014640 1 --2- 172.21.15.28:0/1359905961 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f951c074cd0 0x7f951c1a6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.871+0000 7f9524014640 1 --2- 172.21.15.28:0/1359905961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c107270 0x7f951c074790 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.871+0000 7f9524014640 1 --2- 172.21.15.28:0/1359905961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f951c1021a0 0x7f951c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.871+0000 7f9524014640 1 -- 172.21.15.28:0/1359905961 >> 172.21.15.28:0/1359905961 conn(0x7f951c0fda40 msgr2=0x7f951c0ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.871+0000 7f9524014640 1 -- 172.21.15.28:0/1359905961 shutdown_connections 2024-04-21T09:00:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.871+0000 7f9524014640 1 -- 172.21.15.28:0/1359905961 wait complete. 2024-04-21T09:00:16.885 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T09:00:16.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T09:00:16.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.988+0000 7f5c54693640 1 -- 172.21.15.28:0/1769508877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c4c074200 msgr2=0x7f5c4c0745e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:16.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.988+0000 7f5c54693640 1 --2- 172.21.15.28:0/1769508877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c4c074200 0x7f5c4c0745e0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009900 tx=0x7f5c3c02bcd0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.988+0000 7f5c54693640 1 -- 172.21.15.28:0/1769508877 shutdown_connections 2024-04-21T09:00:16.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.988+0000 7f5c54693640 1 --2- 172.21.15.28:0/1769508877 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5c4c10e860 0x7f5c4c110c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.988+0000 7f5c54693640 1 --2- 172.21.15.28:0/1769508877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c4c074b20 0x7f5c4c10e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.988+0000 7f5c54693640 1 --2- 172.21.15.28:0/1769508877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c4c074200 0x7f5c4c0745e0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.988+0000 7f5c54693640 1 -- 172.21.15.28:0/1769508877 >> 172.21.15.28:0/1769508877 conn(0x7f5c4c0793a0 msgr2=0x7f5c4c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:16.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.988+0000 7f5c54693640 1 -- 172.21.15.28:0/1769508877 shutdown_connections 2024-04-21T09:00:16.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.988+0000 7f5c54693640 1 -- 172.21.15.28:0/1769508877 wait complete. 2024-04-21T09:00:16.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.989+0000 7f5c54693640 1 Processor -- start 2024-04-21T09:00:16.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.989+0000 7f5c54693640 1 -- start start 2024-04-21T09:00:16.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.989+0000 7f5c54693640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c4c074200 0x7f5c4c1a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.989+0000 7f5c54693640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c4c074b20 0x7f5c4c1a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.989+0000 7f5c54693640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5c4c10e860 0x7f5c4c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.989+0000 7f5c54693640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c1143f0 con 0x7f5c4c074b20 2024-04-21T09:00:16.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.989+0000 7f5c54693640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c114270 con 0x7f5c4c074200 2024-04-21T09:00:16.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.989+0000 7f5c54693640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c114570 con 0x7f5c4c10e860 2024-04-21T09:00:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.989+0000 7f5c51c2e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c4c074b20 0x7f5c4c1a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.989+0000 7f5c51c2e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c4c074b20 0x7f5c4c1a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38818/0 (socket says 172.21.15.28:38818) 2024-04-21T09:00:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.989+0000 7f5c51c2e640 1 -- 172.21.15.28:0/3958497768 learned_addr learned my addr 172.21.15.28:0/3958497768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.989+0000 7f5c52c30640 1 --2- 172.21.15.28:0/3958497768 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5c4c10e860 0x7f5c4c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.989+0000 7f5c5242f640 1 --2- 172.21.15.28:0/3958497768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c4c074200 0x7f5c4c1a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.990+0000 7f5c51c2e640 1 -- 172.21.15.28:0/3958497768 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5c4c10e860 msgr2=0x7f5c4c19a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.990+0000 7f5c51c2e640 1 --2- 172.21.15.28:0/3958497768 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5c4c10e860 0x7f5c4c19a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.990+0000 7f5c51c2e640 1 -- 172.21.15.28:0/3958497768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c4c074200 msgr2=0x7f5c4c1a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.990+0000 7f5c51c2e640 1 --2- 172.21.15.28:0/3958497768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c4c074200 0x7f5c4c1a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.990+0000 7f5c51c2e640 1 -- 172.21.15.28:0/3958497768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c4c19af60 con 0x7f5c4c074b20 2024-04-21T09:00:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.990+0000 7f5c5242f640 1 --2- 172.21.15.28:0/3958497768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c4c074200 0x7f5c4c1a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.990+0000 7f5c51c2e640 1 --2- 172.21.15.28:0/3958497768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c4c074b20 0x7f5c4c1a0b60 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f5c4800e9f0 tx=0x7f5c4800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.990+0000 7f5c437fe640 1 -- 172.21.15.28:0/3958497768 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4800cd80 con 0x7f5c4c074b20 2024-04-21T09:00:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.990+0000 7f5c437fe640 1 -- 172.21.15.28:0/3958497768 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c480108e0 con 0x7f5c4c074b20 2024-04-21T09:00:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.990+0000 7f5c54693640 1 -- 172.21.15.28:0/3958497768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c4c19b250 con 0x7f5c4c074b20 2024-04-21T09:00:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.990+0000 7f5c437fe640 1 -- 172.21.15.28:0/3958497768 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c48018980 con 0x7f5c4c074b20 2024-04-21T09:00:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.990+0000 7f5c54693640 1 -- 172.21.15.28:0/3958497768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c4c1a7470 con 0x7f5c4c074b20 2024-04-21T09:00:16.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.991+0000 7f5c54693640 1 -- 172.21.15.28:0/3958497768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c4c075940 con 0x7f5c4c074b20 2024-04-21T09:00:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.992+0000 7f5c437fe640 1 -- 172.21.15.28:0/3958497768 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5c48010430 con 0x7f5c4c074b20 2024-04-21T09:00:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.993+0000 7f5c437fe640 1 --2- 172.21.15.28:0/3958497768 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5c380a3d20 0x7f5c380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.993+0000 7f5c5242f640 1 --2- 172.21.15.28:0/3958497768 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5c380a3d20 0x7f5c380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.993+0000 7f5c437fe640 1 -- 172.21.15.28:0/3958497768 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f5c480d1040 con 0x7f5c4c074b20 2024-04-21T09:00:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.993+0000 7f5c5242f640 1 --2- 172.21.15.28:0/3958497768 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5c380a3d20 0x7f5c380a61e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c005ab0 tx=0x7f5c3c005a20 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:16.998+0000 7f5c437fe640 1 -- 172.21.15.28:0/3958497768 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5c4809d230 con 0x7f5c4c074b20 2024-04-21T09:00:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.225+0000 7f5c54693640 1 -- 172.21.15.28:0/3958497768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c4c1099e0 con 0x7f5c4c074b20 2024-04-21T09:00:17.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.226+0000 7f5c437fe640 1 -- 172.21.15.28:0/3958497768 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7f5c4809cb30 con 0x7f5c4c074b20 2024-04-21T09:00:17.226 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T09:00:17.226 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T09:00:17.227 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T09:00:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.228+0000 7f5c54693640 1 -- 172.21.15.28:0/3958497768 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5c380a3d20 msgr2=0x7f5c380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.228+0000 7f5c54693640 1 --2- 172.21.15.28:0/3958497768 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5c380a3d20 0x7f5c380a61e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c005ab0 tx=0x7f5c3c005a20 comp rx=0 tx=0).stop 2024-04-21T09:00:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.228+0000 7f5c54693640 1 -- 172.21.15.28:0/3958497768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c4c074b20 msgr2=0x7f5c4c1a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.228+0000 7f5c54693640 1 --2- 172.21.15.28:0/3958497768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c4c074b20 0x7f5c4c1a0b60 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f5c4800e9f0 tx=0x7f5c4800eeb0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.228+0000 7f5c54693640 1 -- 172.21.15.28:0/3958497768 shutdown_connections 2024-04-21T09:00:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.228+0000 7f5c54693640 1 --2- 172.21.15.28:0/3958497768 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5c380a3d20 0x7f5c380a61e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.228+0000 7f5c54693640 1 --2- 172.21.15.28:0/3958497768 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5c4c10e860 0x7f5c4c19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.228+0000 7f5c54693640 1 --2- 172.21.15.28:0/3958497768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c4c074b20 0x7f5c4c1a0b60 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.228+0000 7f5c54693640 1 --2- 172.21.15.28:0/3958497768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c4c074200 0x7f5c4c1a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.228+0000 7f5c54693640 1 -- 172.21.15.28:0/3958497768 >> 172.21.15.28:0/3958497768 conn(0x7f5c4c0793a0 msgr2=0x7f5c4c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.228+0000 7f5c54693640 1 -- 172.21.15.28:0/3958497768 shutdown_connections 2024-04-21T09:00:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.228+0000 7f5c54693640 1 -- 172.21.15.28:0/3958497768 wait complete. 2024-04-21T09:00:17.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T09:00:17.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.343+0000 7f54c36ba640 1 -- 172.21.15.28:0/3175070830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc103af0 msgr2=0x7f54bc10bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.343+0000 7f54c36ba640 1 --2- 172.21.15.28:0/3175070830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc103af0 0x7f54bc10bfd0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009900 tx=0x7f54b002bd60 comp rx=0 tx=0).stop 2024-04-21T09:00:17.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.344+0000 7f54c36ba640 1 -- 172.21.15.28:0/3175070830 shutdown_connections 2024-04-21T09:00:17.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.344+0000 7f54c36ba640 1 --2- 172.21.15.28:0/3175070830 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc10c510 0x7f54bc10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.344+0000 7f54c36ba640 1 --2- 172.21.15.28:0/3175070830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc103af0 0x7f54bc10bfd0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.344+0000 7f54c36ba640 1 --2- 172.21.15.28:0/3175070830 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54bc1031d0 0x7f54bc1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.344+0000 7f54c36ba640 1 -- 172.21.15.28:0/3175070830 >> 172.21.15.28:0/3175070830 conn(0x7f54bc077f50 msgr2=0x7f54bc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.344+0000 7f54c36ba640 1 -- 172.21.15.28:0/3175070830 shutdown_connections 2024-04-21T09:00:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.344+0000 7f54c36ba640 1 -- 172.21.15.28:0/3175070830 wait complete. 2024-04-21T09:00:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.344+0000 7f54c36ba640 1 Processor -- start 2024-04-21T09:00:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.344+0000 7f54c36ba640 1 -- start start 2024-04-21T09:00:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.344+0000 7f54c36ba640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54bc1031d0 0x7f54bc100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.344+0000 7f54c36ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc103af0 0x7f54bc101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.344+0000 7f54c36ba640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc10c510 0x7f54bc1018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.344+0000 7f54c36ba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc1121f0 con 0x7f54bc103af0 2024-04-21T09:00:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.344+0000 7f54c36ba640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc112070 con 0x7f54bc10c510 2024-04-21T09:00:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54c36ba640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc112370 con 0x7f54bc1031d0 2024-04-21T09:00:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54c0c55640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc103af0 0x7f54bc101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54c0c55640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc103af0 0x7f54bc101360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38844/0 (socket says 172.21.15.28:38844) 2024-04-21T09:00:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54c0c55640 1 -- 172.21.15.28:0/3112474746 learned_addr learned my addr 172.21.15.28:0/3112474746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54c1456640 1 --2- 172.21.15.28:0/3112474746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54bc1031d0 0x7f54bc100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54c1c57640 1 --2- 172.21.15.28:0/3112474746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc10c510 0x7f54bc1018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54c0c55640 1 -- 172.21.15.28:0/3112474746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54bc1031d0 msgr2=0x7f54bc100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54c0c55640 1 --2- 172.21.15.28:0/3112474746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54bc1031d0 0x7f54bc100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54c0c55640 1 -- 172.21.15.28:0/3112474746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc10c510 msgr2=0x7f54bc1018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54c0c55640 1 --2- 172.21.15.28:0/3112474746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc10c510 0x7f54bc1018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54c0c55640 1 -- 172.21.15.28:0/3112474746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54bc1a6c10 con 0x7f54bc103af0 2024-04-21T09:00:17.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54c1456640 1 --2- 172.21.15.28:0/3112474746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54bc1031d0 0x7f54bc100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:17.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54c0c55640 1 --2- 172.21.15.28:0/3112474746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc103af0 0x7f54bc101360 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f54b0000c00 tx=0x7f54b002d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54ba7fc640 1 -- 172.21.15.28:0/3112474746 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b0043020 con 0x7f54bc103af0 2024-04-21T09:00:17.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54ba7fc640 1 -- 172.21.15.28:0/3112474746 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54b002dde0 con 0x7f54bc103af0 2024-04-21T09:00:17.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54ba7fc640 1 -- 172.21.15.28:0/3112474746 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b0003b80 con 0x7f54bc103af0 2024-04-21T09:00:17.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54c36ba640 1 -- 172.21.15.28:0/3112474746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54bc1a6ea0 con 0x7f54bc103af0 2024-04-21T09:00:17.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.345+0000 7f54c36ba640 1 -- 172.21.15.28:0/3112474746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54bc1a73b0 con 0x7f54bc103af0 2024-04-21T09:00:17.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.348+0000 7f54ba7fc640 1 -- 172.21.15.28:0/3112474746 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f54b0046020 con 0x7f54bc103af0 2024-04-21T09:00:17.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.348+0000 7f54c6383740 1 Processor -- start 2024-04-21T09:00:17.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.348+0000 7f54c6383740 1 -- start start 2024-04-21T09:00:17.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c6383740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308ef4d0 0x559e308ef8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c6383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308efe80 0x559e308f8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c6383740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308f8a00 0x559e308fae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c6383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559e308541f0 con 0x559e308efe80 2024-04-21T09:00:17.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c6383740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x559e307f14b0 con 0x559e308f8a00 2024-04-21T09:00:17.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c6383740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x559e308501b0 con 0x559e308ef4d0 2024-04-21T09:00:17.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c1456640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308efe80 0x559e308f8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c1456640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308efe80 0x559e308f8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38858/0 (socket says 172.21.15.28:38858) 2024-04-21T09:00:17.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c1456640 1 -- 172.21.15.28:0/1721513477 learned_addr learned my addr 172.21.15.28:0/1721513477 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:17.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c1c57640 1 --2- 172.21.15.28:0/1721513477 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308ef4d0 0x559e308ef8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c0c55640 1 --2- 172.21.15.28:0/1721513477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308f8a00 0x559e308fae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54ba7fc640 1 --2- 172.21.15.28:0/3112474746 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f54a40a3da0 0x7f54a40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c1456640 1 -- 172.21.15.28:0/1721513477 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308ef4d0 msgr2=0x559e308ef8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c1456640 1 --2- 172.21.15.28:0/1721513477 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308ef4d0 0x559e308ef8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c1456640 1 -- 172.21.15.28:0/1721513477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308f8a00 msgr2=0x559e308fae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c1456640 1 --2- 172.21.15.28:0/1721513477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308f8a00 0x559e308fae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c1456640 1 -- 172.21.15.28:0/1721513477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e307e31b0 con 0x559e308efe80 2024-04-21T09:00:17.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c0c55640 1 --2- 172.21.15.28:0/1721513477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308f8a00 0x559e308fae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:00:17.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c1c57640 1 --2- 172.21.15.28:0/3112474746 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f54a40a3da0 0x7f54a40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54ba7fc640 1 -- 172.21.15.28:0/3112474746 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f54b00f2ce0 con 0x7f54bc103af0 2024-04-21T09:00:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.349+0000 7f54c1456640 1 --2- 172.21.15.28:0/1721513477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308efe80 0x559e308f8430 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f54b400bd40 tx=0x7f54b400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8c6122344c93bd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.350+0000 7f54a37fe640 1 -- 172.21.15.28:0/1721513477 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b4015070 con 0x559e308efe80 2024-04-21T09:00:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.350+0000 7f54a37fe640 1 -- 172.21.15.28:0/1721513477 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54b400be30 con 0x559e308efe80 2024-04-21T09:00:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.350+0000 7f54a37fe640 1 -- 172.21.15.28:0/1721513477 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b40106e0 con 0x559e308efe80 2024-04-21T09:00:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.350+0000 7f54c6383740 1 -- 172.21.15.28:0/1721513477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308efe80 msgr2=0x559e308f8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.350+0000 7f54c1c57640 1 --2- 172.21.15.28:0/3112474746 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f54a40a3da0 0x7f54a40a6260 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f54ac005cc0 tx=0x7f54ac004a40 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.350+0000 7f54c6383740 1 --2- 172.21.15.28:0/1721513477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308efe80 0x559e308f8430 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f54b400bd40 tx=0x7f54b400f730 comp rx=0 tx=0).stop 2024-04-21T09:00:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.350+0000 7f54c6383740 1 -- 172.21.15.28:0/1721513477 shutdown_connections 2024-04-21T09:00:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.350+0000 7f54c6383740 1 --2- 172.21.15.28:0/1721513477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308f8a00 0x559e308fae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.350+0000 7f54c6383740 1 --2- 172.21.15.28:0/1721513477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308efe80 0x559e308f8430 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.350+0000 7f54c6383740 1 --2- 172.21.15.28:0/1721513477 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308ef4d0 0x559e308ef8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.350+0000 7f54c6383740 1 -- 172.21.15.28:0/1721513477 >> 172.21.15.28:0/1721513477 conn(0x559e308e9ba0 msgr2=0x559e308e9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.350+0000 7f54c6383740 1 -- 172.21.15.28:0/1721513477 shutdown_connections 2024-04-21T09:00:17.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.351+0000 7f54c6383740 1 -- 172.21.15.28:0/1721513477 wait complete. 2024-04-21T09:00:17.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.351+0000 7f54c6383740 1 -- start start 2024-04-21T09:00:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.351+0000 7f54c6383740 1 Processor -- start 2024-04-21T09:00:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.351+0000 7f54c6383740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308ef4d0 0x559e308f82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.351+0000 7f54c6383740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308efe80 0x559e308f5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.351+0000 7f54c6383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308f8a00 0x559e308f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.351+0000 7f54c6383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559e308fb970 con 0x559e308f8a00 2024-04-21T09:00:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.351+0000 7f54c6383740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x559e3080d500 con 0x559e308ef4d0 2024-04-21T09:00:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.351+0000 7f54c6383740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x559e308fbaf0 con 0x559e308efe80 2024-04-21T09:00:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.351+0000 7f54c6383740 10 client.0 _ll_register_callbacks cb 0x559e307dbb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.351+0000 7f54c1c57640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308f8a00 0x559e308f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.351+0000 7f54c1c57640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308f8a00 0x559e308f5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38870/0 (socket says 172.21.15.28:38870) 2024-04-21T09:00:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.351+0000 7f54c1c57640 1 -- 172.21.15.28:0/2400021482 learned_addr learned my addr 172.21.15.28:0/2400021482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.351+0000 7f54c0c55640 1 --2- 172.21.15.28:0/2400021482 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308efe80 0x559e308f5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54c1456640 1 --2- 172.21.15.28:0/2400021482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308ef4d0 0x559e308f82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54c1c57640 1 -- 172.21.15.28:0/2400021482 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308efe80 msgr2=0x559e308f5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54c1c57640 1 --2- 172.21.15.28:0/2400021482 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308efe80 0x559e308f5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54c1c57640 1 -- 172.21.15.28:0/2400021482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308ef4d0 msgr2=0x559e308f82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54c1c57640 1 --2- 172.21.15.28:0/2400021482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308ef4d0 0x559e308f82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54c1c57640 1 -- 172.21.15.28:0/2400021482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e305663f0 con 0x559e308f8a00 2024-04-21T09:00:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54c1456640 1 --2- 172.21.15.28:0/2400021482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308ef4d0 0x559e308f82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:00:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54c1c57640 1 --2- 172.21.15.28:0/2400021482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308f8a00 0x559e308f5b70 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f54ac01ce40 tx=0x7f54ac004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4c96d08b520f76 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54a0ff9640 1 -- 172.21.15.28:0/2400021482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ac029060 con 0x559e308f8a00 2024-04-21T09:00:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54a0ff9640 1 -- 172.21.15.28:0/2400021482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f54ac019620 con 0x559e308f8a00 2024-04-21T09:00:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54c6383740 1 -- 172.21.15.28:0/2400021482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54b400be30 con 0x559e308f8a00 2024-04-21T09:00:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54a0ff9640 1 -- 172.21.15.28:0/2400021482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f54ac030050 con 0x559e308f8a00 2024-04-21T09:00:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54c6383740 10 client.17931 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54c6383740 1 -- 172.21.15.28:0/2400021482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559e306307d0 con 0x559e308f8a00 2024-04-21T09:00:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.352+0000 7f54a0ff9640 1 -- 172.21.15.28:0/2400021482 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54b400be30 con 0x559e308f8a00 2024-04-21T09:00:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.353+0000 7f54a0ff9640 1 -- 172.21.15.28:0/2400021482 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f54ac0246a0 con 0x559e308f8a00 2024-04-21T09:00:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.353+0000 7f54c6383740 10 client.17931 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.353+0000 7f54c6383740 10 client.17931 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T09:00:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.353+0000 7f54c6383740 20 client.17931 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.353+0000 7f54c6383740 1 --2- 172.21.15.28:0/2400021482 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x559e30908c00 0x559e3090b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.353+0000 7f54c6383740 4 client.17931 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:00:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.353+0000 7f54c6383740 1 -- 172.21.15.28:0/2400021482 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e307f3930 con 0x559e30908c00 2024-04-21T09:00:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.353+0000 7f54c1456640 1 --2- 172.21.15.28:0/2400021482 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x559e30908c00 0x559e3090b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.353+0000 7f54c1456640 1 --2- 172.21.15.28:0/2400021482 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x559e30908c00 0x559e3090b050 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5e23d6f58d5d6c46 server_cookie=691c603a67737479 in_seq=0 out_seq=0 2024-04-21T09:00:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.353+0000 7f54a0ff9640 10 client.17931 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T09:00:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.354+0000 7f54a0ff9640 1 -- 172.21.15.28:0/2400021482 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559e307f3930 con 0x559e30908c00 2024-04-21T09:00:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.354+0000 7f54a0ff9640 10 client.17931 handle_command_reply: tid=0 2024-04-21T09:00:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.354+0000 7f54c6383740 1 client.17931 shutdown 2024-04-21T09:00:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.354+0000 7f54c6383740 1 -- 172.21.15.28:0/2400021482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308f8a00 msgr2=0x559e308f5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.354+0000 7f54c6383740 1 --2- 172.21.15.28:0/2400021482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308f8a00 0x559e308f5b70 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f54ac01ce40 tx=0x7f54ac004ab0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.354+0000 7f54c6383740 1 -- 172.21.15.28:0/2400021482 shutdown_connections 2024-04-21T09:00:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.354+0000 7f54c6383740 1 --2- 172.21.15.28:0/2400021482 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x559e30908c00 0x559e3090b050 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.354+0000 7f54c6383740 1 --2- 172.21.15.28:0/2400021482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308f8a00 0x559e308f5b70 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.354+0000 7f54c6383740 1 --2- 172.21.15.28:0/2400021482 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308efe80 0x559e308f5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.354+0000 7f54a0ff9640 0 client.17931 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T09:00:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.354+0000 7f54c6383740 1 --2- 172.21.15.28:0/2400021482 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308ef4d0 0x559e308f82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.354+0000 7f54c6383740 1 -- 172.21.15.28:0/2400021482 >> 172.21.15.28:0/2400021482 conn(0x559e308e9ba0 msgr2=0x559e308eb8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.354+0000 7f54c6383740 1 -- 172.21.15.28:0/2400021482 shutdown_connections 2024-04-21T09:00:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.354+0000 7f54c6383740 1 -- 172.21.15.28:0/2400021482 wait complete. 2024-04-21T09:00:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.354+0000 7f54c6383740 20 client.17931 trim_cache size 0 max 16384 2024-04-21T09:00:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c6383740 1 Processor -- start 2024-04-21T09:00:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c6383740 1 -- start start 2024-04-21T09:00:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c6383740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308e9ba0 0x559e30904860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c6383740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308ef4d0 0x559e308f09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c6383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308efe80 0x559e308f7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c6383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc1b3f30 con 0x559e308efe80 2024-04-21T09:00:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c6383740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc1b1800 con 0x559e308ef4d0 2024-04-21T09:00:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c6383740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc1af670 con 0x559e308e9ba0 2024-04-21T09:00:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c1c57640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308efe80 0x559e308f7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c1c57640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308efe80 0x559e308f7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38880/0 (socket says 172.21.15.28:38880) 2024-04-21T09:00:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c1c57640 1 -- 172.21.15.28:0/4063841972 learned_addr learned my addr 172.21.15.28:0/4063841972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c1456640 1 --2- 172.21.15.28:0/4063841972 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308e9ba0 0x559e30904860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c0c55640 1 --2- 172.21.15.28:0/4063841972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308ef4d0 0x559e308f09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c1c57640 1 -- 172.21.15.28:0/4063841972 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308e9ba0 msgr2=0x559e30904860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c1c57640 1 --2- 172.21.15.28:0/4063841972 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308e9ba0 0x559e30904860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c1c57640 1 -- 172.21.15.28:0/4063841972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308ef4d0 msgr2=0x559e308f09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c1c57640 1 --2- 172.21.15.28:0/4063841972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308ef4d0 0x559e308f09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c1c57640 1 -- 172.21.15.28:0/4063841972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e309077a0 con 0x559e308efe80 2024-04-21T09:00:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.368+0000 7f54c1c57640 1 --2- 172.21.15.28:0/4063841972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308efe80 0x559e308f7d60 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f54ac024c30 tx=0x7f54ac00bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bfed0753c55234e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.369+0000 7f54a2ffd640 1 -- 172.21.15.28:0/4063841972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ac039070 con 0x559e308efe80 2024-04-21T09:00:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.369+0000 7f54a2ffd640 1 -- 172.21.15.28:0/4063841972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54ac024cb0 con 0x559e308efe80 2024-04-21T09:00:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.369+0000 7f54a2ffd640 1 -- 172.21.15.28:0/4063841972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ac019c70 con 0x559e308efe80 2024-04-21T09:00:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.369+0000 7f54c6383740 1 -- 172.21.15.28:0/4063841972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308efe80 msgr2=0x559e308f7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.369+0000 7f54c6383740 1 --2- 172.21.15.28:0/4063841972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308efe80 0x559e308f7d60 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f54ac024c30 tx=0x7f54ac00bfd0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.369+0000 7f54c6383740 1 -- 172.21.15.28:0/4063841972 shutdown_connections 2024-04-21T09:00:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.369+0000 7f54c6383740 1 --2- 172.21.15.28:0/4063841972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308efe80 0x559e308f7d60 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.369+0000 7f54c6383740 1 --2- 172.21.15.28:0/4063841972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308ef4d0 0x559e308f09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.369+0000 7f54c6383740 1 --2- 172.21.15.28:0/4063841972 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308e9ba0 0x559e30904860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.369+0000 7f54c6383740 1 -- 172.21.15.28:0/4063841972 >> 172.21.15.28:0/4063841972 conn(0x559e308f5cd0 msgr2=0x559e308f02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.369+0000 7f54c6383740 1 -- 172.21.15.28:0/4063841972 shutdown_connections 2024-04-21T09:00:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.369+0000 7f54c6383740 1 -- 172.21.15.28:0/4063841972 wait complete. 2024-04-21T09:00:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.369+0000 7f54c6383740 1 -- start start 2024-04-21T09:00:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c6383740 1 Processor -- start 2024-04-21T09:00:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c6383740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308e9ba0 0x559e30900ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c6383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308ef4d0 0x559e308fc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c6383740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308efe80 0x559e308fc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c6383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc1afcc0 con 0x559e308ef4d0 2024-04-21T09:00:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c6383740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc1b1130 con 0x559e308e9ba0 2024-04-21T09:00:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c6383740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc1b0390 con 0x559e308efe80 2024-04-21T09:00:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c6383740 10 client.0 _ll_register_callbacks cb 0x7f54ac01caf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c0c55640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308ef4d0 0x559e308fc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c0c55640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308ef4d0 0x559e308fc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38884/0 (socket says 172.21.15.28:38884) 2024-04-21T09:00:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c0c55640 1 -- 172.21.15.28:0/721432611 learned_addr learned my addr 172.21.15.28:0/721432611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c1c57640 1 --2- 172.21.15.28:0/721432611 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308efe80 0x559e308fc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c1456640 1 --2- 172.21.15.28:0/721432611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308e9ba0 0x559e30900ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c0c55640 1 -- 172.21.15.28:0/721432611 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308efe80 msgr2=0x559e308fc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c0c55640 1 --2- 172.21.15.28:0/721432611 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308efe80 0x559e308fc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c0c55640 1 -- 172.21.15.28:0/721432611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308e9ba0 msgr2=0x559e30900ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c0c55640 1 --2- 172.21.15.28:0/721432611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308e9ba0 0x559e30900ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.370+0000 7f54c0c55640 1 -- 172.21.15.28:0/721432611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e308fcde0 con 0x559e308ef4d0 2024-04-21T09:00:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54c0c55640 1 --2- 172.21.15.28:0/721432611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308ef4d0 0x559e308fc330 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f54b00f0150 tx=0x7f54b002bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4feb577087f0fd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54a0ff9640 1 -- 172.21.15.28:0/721432611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b00be940 con 0x559e308ef4d0 2024-04-21T09:00:17.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54a0ff9640 1 -- 172.21.15.28:0/721432611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f54b00bafb0 con 0x559e308ef4d0 2024-04-21T09:00:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54c6383740 1 -- 172.21.15.28:0/721432611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54ac024cb0 con 0x559e308ef4d0 2024-04-21T09:00:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54a0ff9640 1 -- 172.21.15.28:0/721432611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f54b00c2cf0 con 0x559e308ef4d0 2024-04-21T09:00:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54c6383740 10 client.17937 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54c6383740 1 -- 172.21.15.28:0/721432611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559e3096ef50 con 0x559e308ef4d0 2024-04-21T09:00:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54a0ff9640 1 -- 172.21.15.28:0/721432611 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54ac024cb0 con 0x559e308ef4d0 2024-04-21T09:00:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54a0ff9640 1 -- 172.21.15.28:0/721432611 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f54b00a90b0 con 0x559e308ef4d0 2024-04-21T09:00:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54c6383740 10 client.17937 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54c6383740 10 client.17937 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T09:00:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54c6383740 20 client.17937 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54c6383740 1 --2- 172.21.15.28:0/721432611 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x559e308f8a00 0x559e30971690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54c6383740 4 client.17937 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T09:00:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.371+0000 7f54c6383740 1 -- 172.21.15.28:0/721432611 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559e30861c10 con 0x559e308f8a00 2024-04-21T09:00:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54c1456640 1 --2- 172.21.15.28:0/721432611 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x559e308f8a00 0x559e30971690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54c1456640 1 --2- 172.21.15.28:0/721432611 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x559e308f8a00 0x559e30971690 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=77017b84a8a485c3 server_cookie=e4bdac5e2e0ba9f0 in_seq=0 out_seq=0 2024-04-21T09:00:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54a0ff9640 10 client.17937 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T09:00:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54a0ff9640 1 -- 172.21.15.28:0/721432611 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559e30861c10 con 0x559e308f8a00 2024-04-21T09:00:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54a0ff9640 10 client.17937 handle_command_reply: tid=0 2024-04-21T09:00:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54c6383740 1 client.17937 shutdown 2024-04-21T09:00:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54c6383740 1 -- 172.21.15.28:0/721432611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308ef4d0 msgr2=0x559e308fc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54c6383740 1 --2- 172.21.15.28:0/721432611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308ef4d0 0x559e308fc330 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f54b00f0150 tx=0x7f54b002bd60 comp rx=0 tx=0).stop 2024-04-21T09:00:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54c6383740 1 -- 172.21.15.28:0/721432611 shutdown_connections 2024-04-21T09:00:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54c6383740 1 --2- 172.21.15.28:0/721432611 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x559e308f8a00 0x559e30971690 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54c6383740 1 --2- 172.21.15.28:0/721432611 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559e308efe80 0x559e308fc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54c6383740 1 --2- 172.21.15.28:0/721432611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e308ef4d0 0x559e308fc330 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54a0ff9640 0 client.17937 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T09:00:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54c6383740 1 --2- 172.21.15.28:0/721432611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e308e9ba0 0x559e30900ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54c6383740 1 -- 172.21.15.28:0/721432611 >> 172.21.15.28:0/721432611 conn(0x559e308f5cd0 msgr2=0x559e308f3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54c6383740 1 -- 172.21.15.28:0/721432611 shutdown_connections 2024-04-21T09:00:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54c6383740 1 -- 172.21.15.28:0/721432611 wait complete. 2024-04-21T09:00:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.372+0000 7f54c6383740 20 client.17937 trim_cache size 0 max 16384 2024-04-21T09:00:17.385 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T09:00:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.373+0000 7f54c36ba640 1 -- 172.21.15.28:0/3112474746 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f54a40a3da0 msgr2=0x7f54a40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.373+0000 7f54c36ba640 1 --2- 172.21.15.28:0/3112474746 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f54a40a3da0 0x7f54a40a6260 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f54ac005cc0 tx=0x7f54ac004a40 comp rx=0 tx=0).stop 2024-04-21T09:00:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.373+0000 7f54c36ba640 1 -- 172.21.15.28:0/3112474746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc103af0 msgr2=0x7f54bc101360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.373+0000 7f54c36ba640 1 --2- 172.21.15.28:0/3112474746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc103af0 0x7f54bc101360 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f54b0000c00 tx=0x7f54b002d920 comp rx=0 tx=0).stop 2024-04-21T09:00:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.373+0000 7f54c36ba640 1 -- 172.21.15.28:0/3112474746 shutdown_connections 2024-04-21T09:00:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.373+0000 7f54c36ba640 1 --2- 172.21.15.28:0/3112474746 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f54a40a3da0 0x7f54a40a6260 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.373+0000 7f54c36ba640 1 --2- 172.21.15.28:0/3112474746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f54bc10c510 0x7f54bc1018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.373+0000 7f54c36ba640 1 --2- 172.21.15.28:0/3112474746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54bc103af0 0x7f54bc101360 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.373+0000 7f54c36ba640 1 --2- 172.21.15.28:0/3112474746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54bc1031d0 0x7f54bc100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.373+0000 7f54c36ba640 1 -- 172.21.15.28:0/3112474746 >> 172.21.15.28:0/3112474746 conn(0x7f54bc077f50 msgr2=0x7f54bc0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.373+0000 7f54c36ba640 1 -- 172.21.15.28:0/3112474746 shutdown_connections 2024-04-21T09:00:17.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.373+0000 7f54c36ba640 1 -- 172.21.15.28:0/3112474746 wait complete. 2024-04-21T09:00:17.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T09:00:17.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.493+0000 7f4ce2826640 1 -- 172.21.15.28:0/4161217969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cdc1010a0 msgr2=0x7f4cdc101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.493+0000 7f4ce2826640 1 --2- 172.21.15.28:0/4161217969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cdc1010a0 0x7f4cdc101480 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009900 tx=0x7f4ccc02bc30 comp rx=0 tx=0).stop 2024-04-21T09:00:17.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.493+0000 7f4ce2826640 1 -- 172.21.15.28:0/4161217969 shutdown_connections 2024-04-21T09:00:17.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.493+0000 7f4ce2826640 1 --2- 172.21.15.28:0/4161217969 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4cdc106170 0x7f4cdc10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.493+0000 7f4ce2826640 1 --2- 172.21.15.28:0/4161217969 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cdc101a50 0x7f4cdc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.493+0000 7f4ce2826640 1 --2- 172.21.15.28:0/4161217969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cdc1010a0 0x7f4cdc101480 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.493+0000 7f4ce2826640 1 -- 172.21.15.28:0/4161217969 >> 172.21.15.28:0/4161217969 conn(0x7f4cdc077eb0 msgr2=0x7f4cdc0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.493+0000 7f4ce2826640 1 -- 172.21.15.28:0/4161217969 shutdown_connections 2024-04-21T09:00:17.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.493+0000 7f4ce2826640 1 -- 172.21.15.28:0/4161217969 wait complete. 2024-04-21T09:00:17.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.493+0000 7f4ce2826640 1 Processor -- start 2024-04-21T09:00:17.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.493+0000 7f4ce2826640 1 -- start start 2024-04-21T09:00:17.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4ce2826640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4cdc1010a0 0x7f4cdc197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4ce2826640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cdc106170 0x7f4cdc197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4ce2826640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cdc19bed0 0x7f4cdc19c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4ce2826640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc1110b0 con 0x7f4cdc106170 2024-04-21T09:00:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4ce2826640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc110f30 con 0x7f4cdc19bed0 2024-04-21T09:00:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4ce2826640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc111230 con 0x7f4cdc1010a0 2024-04-21T09:00:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4cdb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cdc106170 0x7f4cdc197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4cdb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cdc106170 0x7f4cdc197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38908/0 (socket says 172.21.15.28:38908) 2024-04-21T09:00:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4cdb7fe640 1 -- 172.21.15.28:0/215439074 learned_addr learned my addr 172.21.15.28:0/215439074 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:17.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4cdbfff640 1 --2- 172.21.15.28:0/215439074 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4cdc1010a0 0x7f4cdc197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4ce0dc3640 1 --2- 172.21.15.28:0/215439074 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cdc19bed0 0x7f4cdc19c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4cdb7fe640 1 -- 172.21.15.28:0/215439074 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4cdc1010a0 msgr2=0x7f4cdc197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4cdb7fe640 1 --2- 172.21.15.28:0/215439074 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4cdc1010a0 0x7f4cdc197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4cdb7fe640 1 -- 172.21.15.28:0/215439074 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cdc19bed0 msgr2=0x7f4cdc19c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4cdb7fe640 1 --2- 172.21.15.28:0/215439074 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cdc19bed0 0x7f4cdc19c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4cdb7fe640 1 -- 172.21.15.28:0/215439074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cdc19c9c0 con 0x7f4cdc106170 2024-04-21T09:00:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4ce0dc3640 1 --2- 172.21.15.28:0/215439074 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cdc19bed0 0x7f4cdc19c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:00:17.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.494+0000 7f4cdb7fe640 1 --2- 172.21.15.28:0/215439074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cdc106170 0x7f4cdc197b50 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000e9c0 tx=0x7f4cd000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.495+0000 7f4cd97fa640 1 -- 172.21.15.28:0/215439074 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd000cd80 con 0x7f4cdc106170 2024-04-21T09:00:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.495+0000 7f4cd97fa640 1 -- 172.21.15.28:0/215439074 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cd00054d0 con 0x7f4cdc106170 2024-04-21T09:00:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.495+0000 7f4ce2826640 1 -- 172.21.15.28:0/215439074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cdc19ccb0 con 0x7f4cdc106170 2024-04-21T09:00:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.495+0000 7f4cd97fa640 1 -- 172.21.15.28:0/215439074 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0010430 con 0x7f4cdc106170 2024-04-21T09:00:17.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.495+0000 7f4ce2826640 1 -- 172.21.15.28:0/215439074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cdc1a44a0 con 0x7f4cdc106170 2024-04-21T09:00:17.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.497+0000 7f4ce5410740 1 Processor -- start 2024-04-21T09:00:17.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.497+0000 7f4cd97fa640 1 -- 172.21.15.28:0/215439074 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4cd0004fd0 con 0x7f4cdc106170 2024-04-21T09:00:17.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.497+0000 7f4ce5410740 1 -- start start 2024-04-21T09:00:17.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.497+0000 7f4ce5410740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db14d0 0x558891db18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.497+0000 7f4ce5410740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db1e80 0x558891dba430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.497+0000 7f4ce5410740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891dbaa00 0x558891dbce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4ce5410740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558891d161f0 con 0x558891db14d0 2024-04-21T09:00:17.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4ce5410740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558891cb34b0 con 0x558891db1e80 2024-04-21T09:00:17.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4ce5410740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558891d121b0 con 0x558891dbaa00 2024-04-21T09:00:17.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4ce0dc3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db14d0 0x558891db18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4ce0dc3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db14d0 0x558891db18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38914/0 (socket says 172.21.15.28:38914) 2024-04-21T09:00:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4ce0dc3640 1 -- 172.21.15.28:0/1716673473 learned_addr learned my addr 172.21.15.28:0/1716673473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4cdb7fe640 1 --2- 172.21.15.28:0/1716673473 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891dbaa00 0x558891dbce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4cdbfff640 1 --2- 172.21.15.28:0/1716673473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db1e80 0x558891dba430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4ce0dc3640 1 -- 172.21.15.28:0/1716673473 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891dbaa00 msgr2=0x558891dbce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4ce0dc3640 1 --2- 172.21.15.28:0/1716673473 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891dbaa00 0x558891dbce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4ce0dc3640 1 -- 172.21.15.28:0/1716673473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db1e80 msgr2=0x558891dba430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4ce0dc3640 1 --2- 172.21.15.28:0/1716673473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db1e80 0x558891dba430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4ce0dc3640 1 -- 172.21.15.28:0/1716673473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558891ca51b0 con 0x558891db14d0 2024-04-21T09:00:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4cd97fa640 1 --2- 172.21.15.28:0/215439074 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4cc40a3d20 0x7f4cc40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4ce0dc3640 1 --2- 172.21.15.28:0/1716673473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db14d0 0x558891db18b0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009f10 tx=0x7f4cd400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5bea6697846ce33 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4cdbfff640 1 --2- 172.21.15.28:0/215439074 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4cc40a3d20 0x7f4cc40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4cd97fa640 1 -- 172.21.15.28:0/215439074 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f4cd00d1100 con 0x7f4cdc106170 2024-04-21T09:00:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4cc27fc640 1 -- 172.21.15.28:0/1716673473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4003b70 con 0x558891db14d0 2024-04-21T09:00:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4cc27fc640 1 -- 172.21.15.28:0/1716673473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cd4003d10 con 0x558891db14d0 2024-04-21T09:00:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.498+0000 7f4cc27fc640 1 -- 172.21.15.28:0/1716673473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4011940 con 0x558891db14d0 2024-04-21T09:00:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.499+0000 7f4ce5410740 1 -- 172.21.15.28:0/1716673473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db14d0 msgr2=0x558891db18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.499+0000 7f4cdbfff640 1 --2- 172.21.15.28:0/215439074 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4cc40a3d20 0x7f4cc40a61e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc006010 tx=0x7f4ccc005e40 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.499+0000 7f4ce5410740 1 --2- 172.21.15.28:0/1716673473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db14d0 0x558891db18b0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009f10 tx=0x7f4cd400f730 comp rx=0 tx=0).stop 2024-04-21T09:00:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.499+0000 7f4ce5410740 1 -- 172.21.15.28:0/1716673473 shutdown_connections 2024-04-21T09:00:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.499+0000 7f4ce5410740 1 --2- 172.21.15.28:0/1716673473 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891dbaa00 0x558891dbce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.499+0000 7f4ce5410740 1 --2- 172.21.15.28:0/1716673473 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db1e80 0x558891dba430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.499+0000 7f4ce5410740 1 --2- 172.21.15.28:0/1716673473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db14d0 0x558891db18b0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.499+0000 7f4ce5410740 1 -- 172.21.15.28:0/1716673473 >> 172.21.15.28:0/1716673473 conn(0x558891dabba0 msgr2=0x558891dabf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.499+0000 7f4ce5410740 1 -- 172.21.15.28:0/1716673473 shutdown_connections 2024-04-21T09:00:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.499+0000 7f4ce5410740 1 -- 172.21.15.28:0/1716673473 wait complete. 2024-04-21T09:00:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.499+0000 7f4ce5410740 1 -- start start 2024-04-21T09:00:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4ce5410740 1 Processor -- start 2024-04-21T09:00:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4ce5410740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db14d0 0x558891dba320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4ce5410740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db1e80 0x558891db76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4ce5410740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891dbaa00 0x558891db7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4ce5410740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558891dbd970 con 0x558891db1e80 2024-04-21T09:00:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4ce5410740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558891ccf500 con 0x558891db14d0 2024-04-21T09:00:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4ce5410740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558891dbdaf0 con 0x558891dbaa00 2024-04-21T09:00:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4ce5410740 10 client.0 _ll_register_callbacks cb 0x558891c9db80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4cdb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db1e80 0x558891db76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4cdb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db1e80 0x558891db76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38924/0 (socket says 172.21.15.28:38924) 2024-04-21T09:00:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4cdb7fe640 1 -- 172.21.15.28:0/2343671674 learned_addr learned my addr 172.21.15.28:0/2343671674 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4ce0dc3640 1 --2- 172.21.15.28:0/2343671674 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891dbaa00 0x558891db7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4cdbfff640 1 --2- 172.21.15.28:0/2343671674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db14d0 0x558891dba320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4cdb7fe640 1 -- 172.21.15.28:0/2343671674 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891dbaa00 msgr2=0x558891db7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4cdb7fe640 1 --2- 172.21.15.28:0/2343671674 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891dbaa00 0x558891db7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4cdb7fe640 1 -- 172.21.15.28:0/2343671674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db14d0 msgr2=0x558891dba320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4cdb7fe640 1 --2- 172.21.15.28:0/2343671674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db14d0 0x558891dba320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4cdb7fe640 1 -- 172.21.15.28:0/2343671674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558891a283f0 con 0x558891db1e80 2024-04-21T09:00:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.500+0000 7f4cdbfff640 1 --2- 172.21.15.28:0/2343671674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db14d0 0x558891dba320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:00:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4cdb7fe640 1 --2- 172.21.15.28:0/2343671674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db1e80 0x558891db76b0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f4cd009cbe0 tx=0x7f4cd009cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a463ec0aa7f547 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4cbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4cbb7fe640 1 -- 172.21.15.28:0/2343671674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd009d350 con 0x558891db1e80 2024-04-21T09:00:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4cbb7fe640 1 -- 172.21.15.28:0/2343671674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f4cd009ceb0 con 0x558891db1e80 2024-04-21T09:00:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4ce5410740 1 -- 172.21.15.28:0/2343671674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cd4003d10 con 0x558891db1e80 2024-04-21T09:00:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4cbb7fe640 1 -- 172.21.15.28:0/2343671674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4cd009cc50 con 0x558891db1e80 2024-04-21T09:00:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4ce5410740 10 client.17949 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4ce5410740 1 -- 172.21.15.28:0/2343671674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558891af27d0 con 0x558891db1e80 2024-04-21T09:00:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4cbb7fe640 1 -- 172.21.15.28:0/2343671674 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cd4003d10 con 0x558891db1e80 2024-04-21T09:00:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4cbb7fe640 1 -- 172.21.15.28:0/2343671674 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f4cd00a54c0 con 0x558891db1e80 2024-04-21T09:00:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4ce5410740 10 client.17949 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4ce5410740 10 client.17949 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T09:00:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4ce5410740 20 client.17949 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4ce5410740 1 --2- 172.21.15.28:0/2343671674 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558891dcacc0 0x558891dcd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4ce5410740 4 client.17949 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:00:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4ce5410740 1 -- 172.21.15.28:0/2343671674 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558891cb5930 con 0x558891dcacc0 2024-04-21T09:00:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.501+0000 7f4cdbfff640 1 --2- 172.21.15.28:0/2343671674 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558891dcacc0 0x558891dcd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.502+0000 7f4cbb7fe640 10 client.17949 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T09:00:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.502+0000 7f4cdbfff640 1 --2- 172.21.15.28:0/2343671674 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558891dcacc0 0x558891dcd0b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=93ff6b65fbcca0d6 server_cookie=78e717adb3eca68f in_seq=0 out_seq=0 2024-04-21T09:00:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.503+0000 7f4cbb7fe640 1 -- 172.21.15.28:0/2343671674 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558891cb5930 con 0x558891dcacc0 2024-04-21T09:00:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.503+0000 7f4cbb7fe640 10 client.17949 handle_command_reply: tid=0 2024-04-21T09:00:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.503+0000 7f4ce5410740 1 client.17949 shutdown 2024-04-21T09:00:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.503+0000 7f4ce5410740 1 -- 172.21.15.28:0/2343671674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db1e80 msgr2=0x558891db76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.503+0000 7f4ce5410740 1 --2- 172.21.15.28:0/2343671674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db1e80 0x558891db76b0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f4cd009cbe0 tx=0x7f4cd009cae0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.503+0000 7f4ce5410740 1 -- 172.21.15.28:0/2343671674 shutdown_connections 2024-04-21T09:00:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.503+0000 7f4ce5410740 1 --2- 172.21.15.28:0/2343671674 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558891dcacc0 0x558891dcd0b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.503+0000 7f4ce5410740 1 --2- 172.21.15.28:0/2343671674 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891dbaa00 0x558891db7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.503+0000 7f4ce5410740 1 --2- 172.21.15.28:0/2343671674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db1e80 0x558891db76b0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.503+0000 7f4ce5410740 1 --2- 172.21.15.28:0/2343671674 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db14d0 0x558891dba320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.503+0000 7f4ce5410740 1 -- 172.21.15.28:0/2343671674 >> 172.21.15.28:0/2343671674 conn(0x558891dabba0 msgr2=0x558891dad8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.503+0000 7f4cbb7fe640 0 client.17949 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T09:00:17.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.503+0000 7f4ce5410740 1 -- 172.21.15.28:0/2343671674 shutdown_connections 2024-04-21T09:00:17.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.503+0000 7f4ce5410740 1 -- 172.21.15.28:0/2343671674 wait complete. 2024-04-21T09:00:17.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.503+0000 7f4ce5410740 20 client.17949 trim_cache size 0 max 16384 2024-04-21T09:00:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.516+0000 7f4ce5410740 1 Processor -- start 2024-04-21T09:00:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.516+0000 7f4ce5410740 1 -- start start 2024-04-21T09:00:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.516+0000 7f4ce5410740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891db14d0 0x558891db9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.516+0000 7f4ce5410740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db1e80 0x558891dba320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.516+0000 7f4ce5410740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891dbaa00 0x558891dd2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.516+0000 7f4ce5410740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc1b0e10 con 0x558891db1e80 2024-04-21T09:00:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.516+0000 7f4ce5410740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc1ae6e0 con 0x558891dbaa00 2024-04-21T09:00:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.516+0000 7f4ce5410740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc1ac550 con 0x558891db14d0 2024-04-21T09:00:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.517+0000 7f4cdb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db1e80 0x558891dba320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.517+0000 7f4cdb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db1e80 0x558891dba320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38930/0 (socket says 172.21.15.28:38930) 2024-04-21T09:00:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.517+0000 7f4cdb7fe640 1 -- 172.21.15.28:0/1661440182 learned_addr learned my addr 172.21.15.28:0/1661440182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.517+0000 7f4cdbfff640 1 --2- 172.21.15.28:0/1661440182 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891db14d0 0x558891db9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.517+0000 7f4ce0dc3640 1 --2- 172.21.15.28:0/1661440182 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891dbaa00 0x558891dd2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.517+0000 7f4cdb7fe640 1 -- 172.21.15.28:0/1661440182 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891db14d0 msgr2=0x558891db9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.517+0000 7f4cdb7fe640 1 --2- 172.21.15.28:0/1661440182 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891db14d0 0x558891db9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.517+0000 7f4cdb7fe640 1 -- 172.21.15.28:0/1661440182 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891dbaa00 msgr2=0x558891dd2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.517+0000 7f4cdb7fe640 1 --2- 172.21.15.28:0/1661440182 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891dbaa00 0x558891dd2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.517+0000 7f4cdb7fe640 1 -- 172.21.15.28:0/1661440182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558891db0790 con 0x558891db1e80 2024-04-21T09:00:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.517+0000 7f4cdb7fe640 1 --2- 172.21.15.28:0/1661440182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db1e80 0x558891dba320 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0099060 tx=0x7f4cd009b160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c927c3a37ec448f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.517+0000 7f4cc1ffb640 1 -- 172.21.15.28:0/1661440182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd009d350 con 0x558891db1e80 2024-04-21T09:00:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.517+0000 7f4cc1ffb640 1 -- 172.21.15.28:0/1661440182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cd00a1000 con 0x558891db1e80 2024-04-21T09:00:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.517+0000 7f4cc1ffb640 1 -- 172.21.15.28:0/1661440182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd00a5430 con 0x558891db1e80 2024-04-21T09:00:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.518+0000 7f4ce5410740 1 -- 172.21.15.28:0/1661440182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db1e80 msgr2=0x558891dba320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.518+0000 7f4ce5410740 1 --2- 172.21.15.28:0/1661440182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db1e80 0x558891dba320 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0099060 tx=0x7f4cd009b160 comp rx=0 tx=0).stop 2024-04-21T09:00:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.518+0000 7f4ce5410740 1 -- 172.21.15.28:0/1661440182 shutdown_connections 2024-04-21T09:00:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.518+0000 7f4ce5410740 1 --2- 172.21.15.28:0/1661440182 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891dbaa00 0x558891dd2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.518+0000 7f4ce5410740 1 --2- 172.21.15.28:0/1661440182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891db1e80 0x558891dba320 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.518+0000 7f4ce5410740 1 --2- 172.21.15.28:0/1661440182 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891db14d0 0x558891db9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.518+0000 7f4ce5410740 1 -- 172.21.15.28:0/1661440182 >> 172.21.15.28:0/1661440182 conn(0x558891dabba0 msgr2=0x558891dacea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.518+0000 7f4ce5410740 1 -- 172.21.15.28:0/1661440182 shutdown_connections 2024-04-21T09:00:17.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.518+0000 7f4ce5410740 1 -- 172.21.15.28:0/1661440182 wait complete. 2024-04-21T09:00:17.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.518+0000 7f4ce5410740 1 -- start start 2024-04-21T09:00:17.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.518+0000 7f4ce5410740 1 Processor -- start 2024-04-21T09:00:17.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.518+0000 7f4ce5410740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db14d0 0x558891dbeac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.518+0000 7f4ce5410740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891db1e80 0x558891dbf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4ce5410740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891dbaa00 0x558891e345b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4ce5410740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc1acba0 con 0x558891dbaa00 2024-04-21T09:00:17.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4ce5410740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc1ae010 con 0x558891db14d0 2024-04-21T09:00:17.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4ce5410740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc1ad270 con 0x558891db1e80 2024-04-21T09:00:17.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4ce5410740 10 client.0 _ll_register_callbacks cb 0x7f4cd0005680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:17.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4ce0dc3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891dbaa00 0x558891e345b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4ce0dc3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891dbaa00 0x558891e345b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38946/0 (socket says 172.21.15.28:38946) 2024-04-21T09:00:17.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4ce0dc3640 1 -- 172.21.15.28:0/1489934490 learned_addr learned my addr 172.21.15.28:0/1489934490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:17.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4cdbfff640 1 --2- 172.21.15.28:0/1489934490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db14d0 0x558891dbeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4cdb7fe640 1 --2- 172.21.15.28:0/1489934490 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891db1e80 0x558891dbf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4ce0dc3640 1 -- 172.21.15.28:0/1489934490 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891db1e80 msgr2=0x558891dbf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4ce0dc3640 1 --2- 172.21.15.28:0/1489934490 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891db1e80 0x558891dbf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4ce0dc3640 1 -- 172.21.15.28:0/1489934490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db14d0 msgr2=0x558891dbeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4ce0dc3640 1 --2- 172.21.15.28:0/1489934490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db14d0 0x558891dbeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4ce0dc3640 1 -- 172.21.15.28:0/1489934490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558891dbf8d0 con 0x558891dbaa00 2024-04-21T09:00:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4ce0dc3640 1 --2- 172.21.15.28:0/1489934490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891dbaa00 0x558891e345b0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f4cd400fc30 tx=0x7f4cd4009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2789d39af5492f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4cbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4cbb7fe640 1 -- 172.21.15.28:0/1489934490 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd402a060 con 0x558891dbaa00 2024-04-21T09:00:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4ce5410740 1 -- 172.21.15.28:0/1489934490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cd00a1000 con 0x558891dbaa00 2024-04-21T09:00:17.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.519+0000 7f4cbb7fe640 1 -- 172.21.15.28:0/1489934490 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f4cd4003d80 con 0x558891dbaa00 2024-04-21T09:00:17.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.520+0000 7f4cbb7fe640 1 -- 172.21.15.28:0/1489934490 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4cd4031050 con 0x558891dbaa00 2024-04-21T09:00:17.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.520+0000 7f4ce5410740 10 client.17955 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:17.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.520+0000 7f4ce5410740 1 -- 172.21.15.28:0/1489934490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558891e35bf0 con 0x558891dbaa00 2024-04-21T09:00:17.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.520+0000 7f4cbb7fe640 1 -- 172.21.15.28:0/1489934490 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cd00a1000 con 0x558891dbaa00 2024-04-21T09:00:17.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.520+0000 7f4cbb7fe640 1 -- 172.21.15.28:0/1489934490 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f4cd40052e0 con 0x558891dbaa00 2024-04-21T09:00:17.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.520+0000 7f4ce5410740 10 client.17955 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:17.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.520+0000 7f4ce5410740 10 client.17955 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T09:00:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.520+0000 7f4ce5410740 20 client.17955 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.520+0000 7f4ce5410740 1 --2- 172.21.15.28:0/1489934490 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558891e360b0 0x558891e384d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.520+0000 7f4ce5410740 4 client.17955 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T09:00:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.520+0000 7f4ce5410740 1 -- 172.21.15.28:0/1489934490 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558891d23c10 con 0x558891e360b0 2024-04-21T09:00:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.520+0000 7f4cdbfff640 1 --2- 172.21.15.28:0/1489934490 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558891e360b0 0x558891e384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4cdbfff640 1 --2- 172.21.15.28:0/1489934490 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558891e360b0 0x558891e384d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8fe48dd60ca7e100 server_cookie=7a53fa055538dbd0 in_seq=0 out_seq=0 2024-04-21T09:00:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4cbb7fe640 10 client.17955 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T09:00:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4cbb7fe640 1 -- 172.21.15.28:0/1489934490 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558891d23c10 con 0x558891e360b0 2024-04-21T09:00:17.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4cbb7fe640 10 client.17955 handle_command_reply: tid=0 2024-04-21T09:00:17.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4ce5410740 1 client.17955 shutdown 2024-04-21T09:00:17.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4ce5410740 1 -- 172.21.15.28:0/1489934490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891dbaa00 msgr2=0x558891e345b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4ce5410740 1 --2- 172.21.15.28:0/1489934490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891dbaa00 0x558891e345b0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f4cd400fc30 tx=0x7f4cd4009cd0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4ce5410740 1 -- 172.21.15.28:0/1489934490 shutdown_connections 2024-04-21T09:00:17.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4ce5410740 1 --2- 172.21.15.28:0/1489934490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558891dbaa00 0x558891e345b0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4ce5410740 1 --2- 172.21.15.28:0/1489934490 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558891e360b0 0x558891e384d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4ce5410740 1 --2- 172.21.15.28:0/1489934490 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558891db1e80 0x558891dbf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4ce5410740 1 --2- 172.21.15.28:0/1489934490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558891db14d0 0x558891dbeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4cbb7fe640 0 client.17955 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T09:00:17.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4ce5410740 1 -- 172.21.15.28:0/1489934490 >> 172.21.15.28:0/1489934490 conn(0x558891dabba0 msgr2=0x558891dad710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4ce5410740 1 -- 172.21.15.28:0/1489934490 shutdown_connections 2024-04-21T09:00:17.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4ce5410740 1 -- 172.21.15.28:0/1489934490 wait complete. 2024-04-21T09:00:17.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.521+0000 7f4ce5410740 20 client.17955 trim_cache size 0 max 16384 2024-04-21T09:00:17.531 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T09:00:17.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.522+0000 7f4ce2826640 1 -- 172.21.15.28:0/215439074 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4cc40a3d20 msgr2=0x7f4cc40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.522+0000 7f4ce2826640 1 --2- 172.21.15.28:0/215439074 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4cc40a3d20 0x7f4cc40a61e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc006010 tx=0x7f4ccc005e40 comp rx=0 tx=0).stop 2024-04-21T09:00:17.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.522+0000 7f4ce2826640 1 -- 172.21.15.28:0/215439074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cdc106170 msgr2=0x7f4cdc197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.522+0000 7f4ce2826640 1 --2- 172.21.15.28:0/215439074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cdc106170 0x7f4cdc197b50 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000e9c0 tx=0x7f4cd000ee80 comp rx=0 tx=0).stop 2024-04-21T09:00:17.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.522+0000 7f4ce2826640 1 -- 172.21.15.28:0/215439074 shutdown_connections 2024-04-21T09:00:17.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.522+0000 7f4ce2826640 1 --2- 172.21.15.28:0/215439074 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4cc40a3d20 0x7f4cc40a61e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.522+0000 7f4ce2826640 1 --2- 172.21.15.28:0/215439074 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4cdc19bed0 0x7f4cdc19c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.522+0000 7f4ce2826640 1 --2- 172.21.15.28:0/215439074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cdc106170 0x7f4cdc197b50 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.522+0000 7f4ce2826640 1 --2- 172.21.15.28:0/215439074 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4cdc1010a0 0x7f4cdc197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.522+0000 7f4ce2826640 1 -- 172.21.15.28:0/215439074 >> 172.21.15.28:0/215439074 conn(0x7f4cdc077eb0 msgr2=0x7f4cdc0ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.522+0000 7f4ce2826640 1 -- 172.21.15.28:0/215439074 shutdown_connections 2024-04-21T09:00:17.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.522+0000 7f4ce2826640 1 -- 172.21.15.28:0/215439074 wait complete. 2024-04-21T09:00:17.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T09:00:17.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.639+0000 7fe4ae24a640 1 -- 172.21.15.28:0/2792463318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a81021a0 msgr2=0x7fe4a8102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.639+0000 7fe4ae24a640 1 --2- 172.21.15.28:0/2792463318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a81021a0 0x7fe4a8102580 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fe49c009900 tx=0x7fe49c02bc30 comp rx=0 tx=0).stop 2024-04-21T09:00:17.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.639+0000 7fe4ae24a640 1 -- 172.21.15.28:0/2792463318 shutdown_connections 2024-04-21T09:00:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.639+0000 7fe4ae24a640 1 --2- 172.21.15.28:0/2792463318 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4a8107270 0x7fe4a810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.639+0000 7fe4ae24a640 1 --2- 172.21.15.28:0/2792463318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe4a8102b50 0x7fe4a8106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.639+0000 7fe4ae24a640 1 --2- 172.21.15.28:0/2792463318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a81021a0 0x7fe4a8102580 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.639+0000 7fe4ae24a640 1 -- 172.21.15.28:0/2792463318 >> 172.21.15.28:0/2792463318 conn(0x7fe4a8079350 msgr2=0x7fe4a8079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.639+0000 7fe4ae24a640 1 -- 172.21.15.28:0/2792463318 shutdown_connections 2024-04-21T09:00:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.639+0000 7fe4ae24a640 1 -- 172.21.15.28:0/2792463318 wait complete. 2024-04-21T09:00:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.639+0000 7fe4ae24a640 1 Processor -- start 2024-04-21T09:00:17.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.639+0000 7fe4ae24a640 1 -- start start 2024-04-21T09:00:17.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4ae24a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4a81021a0 0x7fe4a819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4ae24a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe4a8107270 0x7fe4a819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4ae24a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8198590 0x7fe4a8198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4ae24a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81121b0 con 0x7fe4a8198590 2024-04-21T09:00:17.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4ae24a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a8112030 con 0x7fe4a8107270 2024-04-21T09:00:17.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4ae24a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a8112330 con 0x7fe4a81021a0 2024-04-21T09:00:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8198590 0x7fe4a8198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8198590 0x7fe4a8198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38970/0 (socket says 172.21.15.28:38970) 2024-04-21T09:00:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4a7fff640 1 -- 172.21.15.28:0/3969377775 learned_addr learned my addr 172.21.15.28:0/3969377775 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4a6ffd640 1 --2- 172.21.15.28:0/3969377775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe4a8107270 0x7fe4a819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4a77fe640 1 --2- 172.21.15.28:0/3969377775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4a81021a0 0x7fe4a819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4a7fff640 1 -- 172.21.15.28:0/3969377775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4a81021a0 msgr2=0x7fe4a819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4a7fff640 1 --2- 172.21.15.28:0/3969377775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4a81021a0 0x7fe4a819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4a7fff640 1 -- 172.21.15.28:0/3969377775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe4a8107270 msgr2=0x7fe4a819e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4a7fff640 1 --2- 172.21.15.28:0/3969377775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe4a8107270 0x7fe4a819e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4a7fff640 1 -- 172.21.15.28:0/3969377775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a81990b0 con 0x7fe4a8198590 2024-04-21T09:00:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4a7fff640 1 --2- 172.21.15.28:0/3969377775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8198590 0x7fe4a8198a40 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000eaa0 tx=0x7fe4a000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.640+0000 7fe4a4ff9640 1 -- 172.21.15.28:0/3969377775 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a000cdb0 con 0x7fe4a8198590 2024-04-21T09:00:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.641+0000 7fe4a4ff9640 1 -- 172.21.15.28:0/3969377775 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4a0005020 con 0x7fe4a8198590 2024-04-21T09:00:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.641+0000 7fe4ae24a640 1 -- 172.21.15.28:0/3969377775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a81993a0 con 0x7fe4a8198590 2024-04-21T09:00:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.641+0000 7fe4a4ff9640 1 -- 172.21.15.28:0/3969377775 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a00116d0 con 0x7fe4a8198590 2024-04-21T09:00:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.641+0000 7fe4ae24a640 1 -- 172.21.15.28:0/3969377775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a81a55f0 con 0x7fe4a8198590 2024-04-21T09:00:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.643+0000 7fe4a4ff9640 1 -- 172.21.15.28:0/3969377775 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe4a00051d0 con 0x7fe4a8198590 2024-04-21T09:00:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.643+0000 7fe4b0e3f740 1 Processor -- start 2024-04-21T09:00:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.643+0000 7fe4b0e3f740 1 -- start start 2024-04-21T09:00:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.643+0000 7fe4b0e3f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e473c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.643+0000 7fe4b0e3f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e473cd60 0x5621e4745890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.643+0000 7fe4b0e3f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e4745e60 0x5621e47462c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.643+0000 7fe4b0e3f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5621e46a11f0 con 0x5621e473c3b0 2024-04-21T09:00:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.643+0000 7fe4b0e3f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5621e463e4b0 con 0x5621e4745e60 2024-04-21T09:00:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.643+0000 7fe4b0e3f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5621e469d1b0 con 0x5621e473cd60 2024-04-21T09:00:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.643+0000 7fe4a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e473c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.643+0000 7fe4a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e473c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38984/0 (socket says 172.21.15.28:38984) 2024-04-21T09:00:17.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.643+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/3323856752 learned_addr learned my addr 172.21.15.28:0/3323856752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:17.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.643+0000 7fe4a4ff9640 1 --2- 172.21.15.28:0/3969377775 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4900a3da0 0x7fe4900a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.643+0000 7fe4a77fe640 1 --2- 172.21.15.28:0/3323856752 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e473cd60 0x5621e4745890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.644+0000 7fe4a7fff640 1 --2- 172.21.15.28:0/3969377775 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4900a3da0 0x7fe4900a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.644+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/3323856752 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e473cd60 msgr2=0x5621e4745890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.644+0000 7fe4a6ffd640 1 --2- 172.21.15.28:0/3323856752 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e473cd60 0x5621e4745890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.644+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/3323856752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e4745e60 msgr2=0x5621e47462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.644+0000 7fe4a6ffd640 1 --2- 172.21.15.28:0/3323856752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e4745e60 0x5621e47462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.644+0000 7fe4a4ff9640 1 -- 172.21.15.28:0/3969377775 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7fe4a00d0ca0 con 0x7fe4a8198590 2024-04-21T09:00:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.644+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/3323856752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621e46301b0 con 0x5621e473c3b0 2024-04-21T09:00:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.644+0000 7fe4a6ffd640 1 --2- 172.21.15.28:0/3323856752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e473c790 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fe49800bcd0 tx=0x7fe4980127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884b12a950423732 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.644+0000 7fe48dffb640 1 -- 172.21.15.28:0/3323856752 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4980120d0 con 0x5621e473c3b0 2024-04-21T09:00:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.644+0000 7fe4a7fff640 1 --2- 172.21.15.28:0/3969377775 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4900a3da0 0x7fe4900a6260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00ce000 tx=0x7fe4a0087040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.644+0000 7fe48dffb640 1 -- 172.21.15.28:0/3323856752 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe498016070 con 0x5621e473c3b0 2024-04-21T09:00:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.644+0000 7fe48dffb640 1 -- 172.21.15.28:0/3323856752 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe498004a00 con 0x5621e473c3b0 2024-04-21T09:00:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.645+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/3323856752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 msgr2=0x5621e473c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.645+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/3323856752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e473c790 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fe49800bcd0 tx=0x7fe4980127c0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.645+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/3323856752 shutdown_connections 2024-04-21T09:00:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.645+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/3323856752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e4745e60 0x5621e47462c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.645+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/3323856752 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e473cd60 0x5621e4745890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.645+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/3323856752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e473c790 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.645+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/3323856752 >> 172.21.15.28:0/3323856752 conn(0x5621e4736ba0 msgr2=0x5621e4736f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.645+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/3323856752 shutdown_connections 2024-04-21T09:00:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.645+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/3323856752 wait complete. 2024-04-21T09:00:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.645+0000 7fe4b0e3f740 1 -- start start 2024-04-21T09:00:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.645+0000 7fe4b0e3f740 1 Processor -- start 2024-04-21T09:00:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.645+0000 7fe4b0e3f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e474c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4b0e3f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e473cd60 0x5621e4749620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4b0e3f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e4745e60 0x5621e4749b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4b0e3f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5621e474cda0 con 0x5621e473c3b0 2024-04-21T09:00:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4b0e3f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5621e465a500 con 0x5621e473cd60 2024-04-21T09:00:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4b0e3f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5621e474cf20 con 0x5621e4745e60 2024-04-21T09:00:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4b0e3f740 10 client.0 _ll_register_callbacks cb 0x5621e4628b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e474c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e474c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38996/0 (socket says 172.21.15.28:38996) 2024-04-21T09:00:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/5745302 learned_addr learned my addr 172.21.15.28:0/5745302 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4a77fe640 1 --2- 172.21.15.28:0/5745302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e473cd60 0x5621e4749620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4a6ffd640 1 --2- 172.21.15.28:0/5745302 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e4745e60 0x5621e4749b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/5745302 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e4745e60 msgr2=0x5621e4749b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4a6ffd640 1 --2- 172.21.15.28:0/5745302 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e4745e60 0x5621e4749b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/5745302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e473cd60 msgr2=0x5621e4749620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4a6ffd640 1 --2- 172.21.15.28:0/5745302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e473cd60 0x5621e4749620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/5745302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621e43b33f0 con 0x5621e473c3b0 2024-04-21T09:00:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4a6ffd640 1 --2- 172.21.15.28:0/5745302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e474c290 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fe498005550 tx=0x7fe498005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54703450d335efa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe486ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe486ffd640 1 -- 172.21.15.28:0/5745302 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe498004220 con 0x5621e473c3b0 2024-04-21T09:00:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/5745302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe498016070 con 0x5621e473c3b0 2024-04-21T09:00:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.646+0000 7fe486ffd640 1 -- 172.21.15.28:0/5745302 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7fe4980043c0 con 0x5621e473c3b0 2024-04-21T09:00:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.647+0000 7fe486ffd640 1 -- 172.21.15.28:0/5745302 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe49802f050 con 0x5621e473c3b0 2024-04-21T09:00:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.647+0000 7fe4b0e3f740 10 client.17967 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.647+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/5745302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5621e447d7d0 con 0x5621e473c3b0 2024-04-21T09:00:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.647+0000 7fe486ffd640 1 -- 172.21.15.28:0/5745302 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe498016070 con 0x5621e473c3b0 2024-04-21T09:00:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.647+0000 7fe486ffd640 1 -- 172.21.15.28:0/5745302 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe498027020 con 0x5621e473c3b0 2024-04-21T09:00:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.647+0000 7fe4b0e3f740 10 client.17967 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.647+0000 7fe4b0e3f740 10 client.17967 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.647+0000 7fe4b0e3f740 20 client.17967 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.647+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/5745302 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5621e4759f80 0x5621e475c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.647+0000 7fe4b0e3f740 4 client.17967 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:00:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.647+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/5745302 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621e4640930 con 0x5621e4759f80 2024-04-21T09:00:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.647+0000 7fe4a77fe640 1 --2- 172.21.15.28:0/5745302 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5621e4759f80 0x5621e475c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.648+0000 7fe4a77fe640 1 --2- 172.21.15.28:0/5745302 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5621e4759f80 0x5621e475c370 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=625cdc9bf3bd0496 server_cookie=a57ff471d1157442 in_seq=0 out_seq=0 2024-04-21T09:00:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.648+0000 7fe486ffd640 10 client.17967 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.648+0000 7fe486ffd640 1 -- 172.21.15.28:0/5745302 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621e4640930 con 0x5621e4759f80 2024-04-21T09:00:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.648+0000 7fe486ffd640 10 client.17967 handle_command_reply: tid=0 2024-04-21T09:00:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.648+0000 7fe4b0e3f740 1 client.17967 shutdown 2024-04-21T09:00:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.649+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/5745302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 msgr2=0x5621e474c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.649+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/5745302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e474c290 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fe498005550 tx=0x7fe498005850 comp rx=0 tx=0).stop 2024-04-21T09:00:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.649+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/5745302 shutdown_connections 2024-04-21T09:00:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.649+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/5745302 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5621e4759f80 0x5621e475c370 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.649+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/5745302 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e4745e60 0x5621e4749b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.649+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/5745302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e473cd60 0x5621e4749620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.649+0000 7fe486ffd640 0 client.17967 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.649+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/5745302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e474c290 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.649+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/5745302 >> 172.21.15.28:0/5745302 conn(0x5621e4736ba0 msgr2=0x5621e47388f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.649+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/5745302 shutdown_connections 2024-04-21T09:00:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.649+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/5745302 wait complete. 2024-04-21T09:00:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.649+0000 7fe4b0e3f740 20 client.17967 trim_cache size 0 max 16384 2024-04-21T09:00:17.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.662+0000 7fe4b0e3f740 1 Processor -- start 2024-04-21T09:00:17.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.662+0000 7fe4b0e3f740 1 -- start start 2024-04-21T09:00:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.662+0000 7fe4b0e3f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e4738a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.662+0000 7fe4b0e3f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e473cd60 0x5621e4748f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.662+0000 7fe4b0e3f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e4745e60 0x5621e4747250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe4b0e3f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81afc90 con 0x5621e473c3b0 2024-04-21T09:00:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe4b0e3f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81ad560 con 0x5621e473cd60 2024-04-21T09:00:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe4b0e3f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81ab3d0 con 0x5621e4745e60 2024-04-21T09:00:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe4a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e4738a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe4a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e4738a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39002/0 (socket says 172.21.15.28:39002) 2024-04-21T09:00:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/28373514 learned_addr learned my addr 172.21.15.28:0/28373514 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe4a77fe640 1 --2- 172.21.15.28:0/28373514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e473cd60 0x5621e4748f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/28373514 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e4745e60 msgr2=0x5621e4747250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe4a6ffd640 1 --2- 172.21.15.28:0/28373514 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e4745e60 0x5621e4747250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/28373514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e473cd60 msgr2=0x5621e4748f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe4a6ffd640 1 --2- 172.21.15.28:0/28373514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e473cd60 0x5621e4748f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/28373514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621e473b560 con 0x5621e473c3b0 2024-04-21T09:00:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe4a6ffd640 1 --2- 172.21.15.28:0/28373514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e4738a50 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fe49800b9e0 tx=0x7fe498005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a69fdcdbf9bab9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe48d7fa640 1 -- 172.21.15.28:0/28373514 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe498012cc0 con 0x5621e473c3b0 2024-04-21T09:00:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe48d7fa640 1 -- 172.21.15.28:0/28373514 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe498016070 con 0x5621e473c3b0 2024-04-21T09:00:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.663+0000 7fe48d7fa640 1 -- 172.21.15.28:0/28373514 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe498010b40 con 0x5621e473c3b0 2024-04-21T09:00:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.664+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/28373514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 msgr2=0x5621e4738a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.664+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/28373514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e4738a50 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fe49800b9e0 tx=0x7fe498005770 comp rx=0 tx=0).stop 2024-04-21T09:00:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.664+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/28373514 shutdown_connections 2024-04-21T09:00:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.664+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/28373514 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e4745e60 0x5621e4747250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.664+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/28373514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e473cd60 0x5621e4748f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.664+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/28373514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e473c3b0 0x5621e4738a50 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.664+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/28373514 >> 172.21.15.28:0/28373514 conn(0x5621e4736ba0 msgr2=0x5621e4735c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.664+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/28373514 shutdown_connections 2024-04-21T09:00:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.664+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/28373514 wait complete. 2024-04-21T09:00:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.664+0000 7fe4b0e3f740 1 -- start start 2024-04-21T09:00:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.664+0000 7fe4b0e3f740 1 Processor -- start 2024-04-21T09:00:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.664+0000 7fe4b0e3f740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e473c3b0 0x5621e47523a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4b0e3f740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e473cd60 0x5621e474ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4b0e3f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e4745e60 0x5621e474b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4b0e3f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81aba20 con 0x5621e4745e60 2024-04-21T09:00:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4b0e3f740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81ace90 con 0x5621e473cd60 2024-04-21T09:00:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4b0e3f740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81ac0f0 con 0x5621e473c3b0 2024-04-21T09:00:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4b0e3f740 10 client.0 _ll_register_callbacks cb 0x7fe498005020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e4745e60 0x5621e474b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e4745e60 0x5621e474b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39012/0 (socket says 172.21.15.28:39012) 2024-04-21T09:00:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/3075561585 learned_addr learned my addr 172.21.15.28:0/3075561585 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4a77fe640 1 --2- 172.21.15.28:0/3075561585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e473cd60 0x5621e474ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4a6ffd640 1 --2- 172.21.15.28:0/3075561585 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e473c3b0 0x5621e47523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/3075561585 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e473c3b0 msgr2=0x5621e47523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4a6ffd640 1 --2- 172.21.15.28:0/3075561585 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e473c3b0 0x5621e47523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/3075561585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e473cd60 msgr2=0x5621e474ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4a6ffd640 1 --2- 172.21.15.28:0/3075561585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e473cd60 0x5621e474ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4a6ffd640 1 -- 172.21.15.28:0/3075561585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621e474b8f0 con 0x5621e4745e60 2024-04-21T09:00:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4a6ffd640 1 --2- 172.21.15.28:0/3075561585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e4745e60 0x5621e474b2b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fe498003c70 tx=0x7fe498014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f94a307f064fcd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe486ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe486ffd640 1 -- 172.21.15.28:0/3075561585 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe498010430 con 0x5621e4745e60 2024-04-21T09:00:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe486ffd640 1 -- 172.21.15.28:0/3075561585 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7fe4980105d0 con 0x5621e4745e60 2024-04-21T09:00:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/3075561585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe498016070 con 0x5621e4745e60 2024-04-21T09:00:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe486ffd640 1 -- 172.21.15.28:0/3075561585 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe4980107d0 con 0x5621e4745e60 2024-04-21T09:00:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4b0e3f740 10 client.17973 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.665+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/3075561585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5621e47c0020 con 0x5621e4745e60 2024-04-21T09:00:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.666+0000 7fe486ffd640 1 -- 172.21.15.28:0/3075561585 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe498016070 con 0x5621e4745e60 2024-04-21T09:00:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.666+0000 7fe486ffd640 1 -- 172.21.15.28:0/3075561585 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fe49803e060 con 0x5621e4745e60 2024-04-21T09:00:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.666+0000 7fe4b0e3f740 10 client.17973 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.666+0000 7fe4b0e3f740 10 client.17973 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.666+0000 7fe4b0e3f740 20 client.17973 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.666+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/3075561585 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5621e47c0580 0x5621e47c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.666+0000 7fe4b0e3f740 4 client.17973 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T09:00:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.666+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/3075561585 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5621e46aec10 con 0x5621e47c0580 2024-04-21T09:00:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.666+0000 7fe4a77fe640 1 --2- 172.21.15.28:0/3075561585 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5621e47c0580 0x5621e47c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.666+0000 7fe486ffd640 10 client.17973 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.666+0000 7fe4a77fe640 1 --2- 172.21.15.28:0/3075561585 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5621e47c0580 0x5621e47c29a0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c00d8ce6730a528 server_cookie=4ec11d5c15d9e7f9 in_seq=0 out_seq=0 2024-04-21T09:00:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe486ffd640 1 -- 172.21.15.28:0/3075561585 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5621e46aec10 con 0x5621e47c0580 2024-04-21T09:00:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe486ffd640 10 client.17973 handle_command_reply: tid=0 2024-04-21T09:00:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4b0e3f740 1 client.17973 shutdown 2024-04-21T09:00:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/3075561585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e4745e60 msgr2=0x5621e474b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/3075561585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e4745e60 0x5621e474b2b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fe498003c70 tx=0x7fe498014040 comp rx=0 tx=0).stop 2024-04-21T09:00:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/3075561585 shutdown_connections 2024-04-21T09:00:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/3075561585 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5621e47c0580 0x5621e47c29a0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/3075561585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e4745e60 0x5621e474b2b0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/3075561585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5621e473cd60 0x5621e474ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4b0e3f740 1 --2- 172.21.15.28:0/3075561585 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5621e473c3b0 0x5621e47523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe486ffd640 0 client.17973 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/3075561585 >> 172.21.15.28:0/3075561585 conn(0x5621e4736ba0 msgr2=0x5621e4735370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/3075561585 shutdown_connections 2024-04-21T09:00:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4b0e3f740 1 -- 172.21.15.28:0/3075561585 wait complete. 2024-04-21T09:00:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4b0e3f740 20 client.17973 trim_cache size 0 max 16384 2024-04-21T09:00:17.677 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T09:00:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4ae24a640 1 -- 172.21.15.28:0/3969377775 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4900a3da0 msgr2=0x7fe4900a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4ae24a640 1 --2- 172.21.15.28:0/3969377775 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4900a3da0 0x7fe4900a6260 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00ce000 tx=0x7fe4a0087040 comp rx=0 tx=0).stop 2024-04-21T09:00:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4ae24a640 1 -- 172.21.15.28:0/3969377775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8198590 msgr2=0x7fe4a8198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.667+0000 7fe4ae24a640 1 --2- 172.21.15.28:0/3969377775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8198590 0x7fe4a8198a40 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000eaa0 tx=0x7fe4a000ef60 comp rx=0 tx=0).stop 2024-04-21T09:00:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.668+0000 7fe4ae24a640 1 -- 172.21.15.28:0/3969377775 shutdown_connections 2024-04-21T09:00:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.668+0000 7fe4ae24a640 1 --2- 172.21.15.28:0/3969377775 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fe4900a3da0 0x7fe4900a6260 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.668+0000 7fe4ae24a640 1 --2- 172.21.15.28:0/3969377775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8198590 0x7fe4a8198a40 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.668+0000 7fe4ae24a640 1 --2- 172.21.15.28:0/3969377775 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe4a8107270 0x7fe4a819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.668+0000 7fe4ae24a640 1 --2- 172.21.15.28:0/3969377775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe4a81021a0 0x7fe4a819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.668+0000 7fe4ae24a640 1 -- 172.21.15.28:0/3969377775 >> 172.21.15.28:0/3969377775 conn(0x7fe4a8079350 msgr2=0x7fe4a81002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.668+0000 7fe4ae24a640 1 -- 172.21.15.28:0/3969377775 shutdown_connections 2024-04-21T09:00:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:17.668+0000 7fe4ae24a640 1 -- 172.21.15.28:0/3969377775 wait complete. 2024-04-21T09:00:18.425 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:18 smithi152 ceph-mon[31401]: pgmap v281: 129 pgs: 10 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 91 active+clean; 14 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T09:00:18.425 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:18 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3958497768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T09:00:18.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:18 smithi028 ceph-mon[28003]: pgmap v281: 129 pgs: 10 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 91 active+clean; 14 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T09:00:18.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:18 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3958497768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T09:00:18.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:18 smithi171 ceph-mon[31358]: pgmap v281: 129 pgs: 10 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 91 active+clean; 14 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T09:00:18.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:18 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3958497768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T09:00:18.681 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T09:00:18.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T09:00:18.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T09:00:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.801+0000 7f9578443640 1 -- 172.21.15.28:0/2172539083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95701004c0 msgr2=0x7f9570100940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.801+0000 7f9578443640 1 --2- 172.21.15.28:0/2172539083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95701004c0 0x7f9570100940 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f9560009930 tx=0x7f956002bc30 comp rx=0 tx=0).stop 2024-04-21T09:00:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.801+0000 7f9578443640 1 -- 172.21.15.28:0/2172539083 shutdown_connections 2024-04-21T09:00:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.801+0000 7f9578443640 1 --2- 172.21.15.28:0/2172539083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9570100e80 0x7f957010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.801+0000 7f9578443640 1 --2- 172.21.15.28:0/2172539083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95701004c0 0x7f9570100940 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.801+0000 7f9578443640 1 --2- 172.21.15.28:0/2172539083 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9570106f50 0x7f9570107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.801+0000 7f9578443640 1 -- 172.21.15.28:0/2172539083 >> 172.21.15.28:0/2172539083 conn(0x7f9570077450 msgr2=0x7f95700fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.801+0000 7f9578443640 1 -- 172.21.15.28:0/2172539083 shutdown_connections 2024-04-21T09:00:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.801+0000 7f9578443640 1 -- 172.21.15.28:0/2172539083 wait complete. 2024-04-21T09:00:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.802+0000 7f9578443640 1 Processor -- start 2024-04-21T09:00:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.802+0000 7f9578443640 1 -- start start 2024-04-21T09:00:18.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.802+0000 7f9578443640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95701004c0 0x7f95701a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.802+0000 7f9578443640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9570100e80 0x7f95701a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.802+0000 7f9578443640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9570106f50 0x7f957019a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.802+0000 7f9578443640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9570102930 con 0x7f9570106f50 2024-04-21T09:00:18.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.802+0000 7f9578443640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f95701027b0 con 0x7f9570100e80 2024-04-21T09:00:18.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.802+0000 7f9578443640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9570102ab0 con 0x7f95701004c0 2024-04-21T09:00:18.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.802+0000 7f95769e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9570106f50 0x7f957019a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.802+0000 7f95769e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9570106f50 0x7f957019a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39034/0 (socket says 172.21.15.28:39034) 2024-04-21T09:00:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.802+0000 7f95769e0640 1 -- 172.21.15.28:0/3226121273 learned_addr learned my addr 172.21.15.28:0/3226121273 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.802+0000 7f95761df640 1 --2- 172.21.15.28:0/3226121273 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95701004c0 0x7f95701a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.802+0000 7f95759de640 1 --2- 172.21.15.28:0/3226121273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9570100e80 0x7f95701a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.803+0000 7f95769e0640 1 -- 172.21.15.28:0/3226121273 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95701004c0 msgr2=0x7f95701a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.803+0000 7f95769e0640 1 --2- 172.21.15.28:0/3226121273 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95701004c0 0x7f95701a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.803+0000 7f95769e0640 1 -- 172.21.15.28:0/3226121273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9570100e80 msgr2=0x7f95701a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.803+0000 7f95769e0640 1 --2- 172.21.15.28:0/3226121273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9570100e80 0x7f95701a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.803+0000 7f95769e0640 1 -- 172.21.15.28:0/3226121273 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957019b030 con 0x7f9570106f50 2024-04-21T09:00:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.803+0000 7f95761df640 1 --2- 172.21.15.28:0/3226121273 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95701004c0 0x7f95701a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:18.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.803+0000 7f95769e0640 1 --2- 172.21.15.28:0/3226121273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9570106f50 0x7f957019a860 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f956c00eaa0 tx=0x7f956c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:18.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.803+0000 7f95677fe640 1 -- 172.21.15.28:0/3226121273 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c00cdb0 con 0x7f9570106f50 2024-04-21T09:00:18.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.803+0000 7f95677fe640 1 -- 172.21.15.28:0/3226121273 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f956c005020 con 0x7f9570106f50 2024-04-21T09:00:18.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.803+0000 7f9578443640 1 -- 172.21.15.28:0/3226121273 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f957019b320 con 0x7f9570106f50 2024-04-21T09:00:18.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.803+0000 7f95677fe640 1 -- 172.21.15.28:0/3226121273 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956c0116d0 con 0x7f9570106f50 2024-04-21T09:00:18.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.803+0000 7f9578443640 1 -- 172.21.15.28:0/3226121273 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95701a74f0 con 0x7f9570106f50 2024-04-21T09:00:18.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.806+0000 7f95677fe640 1 -- 172.21.15.28:0/3226121273 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f956c0051d0 con 0x7f9570106f50 2024-04-21T09:00:18.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.806+0000 7f957b035740 1 Processor -- start 2024-04-21T09:00:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f957b035740 1 -- start start 2024-04-21T09:00:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f957b035740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a448e0 0x556a39a44cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f957b035740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a45290 0x556a39a4d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f957b035740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a4de10 0x556a39a50260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f957b035740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556a399a8f00 con 0x556a39a448e0 2024-04-21T09:00:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f957b035740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556a39946590 con 0x556a39a4de10 2024-04-21T09:00:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f957b035740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x556a399a4ec0 con 0x556a39a45290 2024-04-21T09:00:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f95759de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a448e0 0x556a39a44cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f95759de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a448e0 0x556a39a44cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39044/0 (socket says 172.21.15.28:39044) 2024-04-21T09:00:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f95759de640 1 -- 172.21.15.28:0/2507584214 learned_addr learned my addr 172.21.15.28:0/2507584214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f95761df640 1 --2- 172.21.15.28:0/2507584214 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a45290 0x556a39a4d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f95759de640 1 --2- 172.21.15.28:0/2507584214 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a4de10 0x556a39a50260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f95677fe640 1 --2- 172.21.15.28:0/3226121273 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f955c0a3d20 0x7f955c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f95759de640 1 -- 172.21.15.28:0/2507584214 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a45290 msgr2=0x556a39a4d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f95759de640 1 --2- 172.21.15.28:0/2507584214 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a45290 0x556a39a4d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f95759de640 1 -- 172.21.15.28:0/2507584214 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a4de10 msgr2=0x556a39a50260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f95759de640 1 --2- 172.21.15.28:0/2507584214 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a4de10 0x556a39a50260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f95759de640 1 -- 172.21.15.28:0/2507584214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a397837e0 con 0x556a39a448e0 2024-04-21T09:00:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f95761df640 1 --2- 172.21.15.28:0/2507584214 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a45290 0x556a39a4d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f95769e0640 1 --2- 172.21.15.28:0/3226121273 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f955c0a3d20 0x7f955c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.807+0000 7f95677fe640 1 -- 172.21.15.28:0/3226121273 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f956c0d0f10 con 0x7f9570106f50 2024-04-21T09:00:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.808+0000 7f95759de640 1 --2- 172.21.15.28:0/2507584214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a448e0 0x556a39a44cc0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f9560000c00 tx=0x7f9560005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e133a14a8479782c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.808+0000 7f9564bfe640 1 -- 172.21.15.28:0/2507584214 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9560045020 con 0x556a39a448e0 2024-04-21T09:00:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.808+0000 7f9564bfe640 1 -- 172.21.15.28:0/2507584214 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f956002cc30 con 0x556a39a448e0 2024-04-21T09:00:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.808+0000 7f95769e0640 1 --2- 172.21.15.28:0/3226121273 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f955c0a3d20 0x7f955c0a61e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f956c0ce2c0 tx=0x7f956c005890 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.808+0000 7f9564bfe640 1 -- 172.21.15.28:0/2507584214 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9560004920 con 0x556a39a448e0 2024-04-21T09:00:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 -- 172.21.15.28:0/2507584214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a448e0 msgr2=0x556a39a44cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 --2- 172.21.15.28:0/2507584214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a448e0 0x556a39a44cc0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f9560000c00 tx=0x7f9560005bb0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 -- 172.21.15.28:0/2507584214 shutdown_connections 2024-04-21T09:00:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 --2- 172.21.15.28:0/2507584214 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a4de10 0x556a39a50260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 --2- 172.21.15.28:0/2507584214 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a45290 0x556a39a4d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 --2- 172.21.15.28:0/2507584214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a448e0 0x556a39a44cc0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 -- 172.21.15.28:0/2507584214 >> 172.21.15.28:0/2507584214 conn(0x556a39a3efb0 msgr2=0x556a39a3f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 -- 172.21.15.28:0/2507584214 shutdown_connections 2024-04-21T09:00:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 -- 172.21.15.28:0/2507584214 wait complete. 2024-04-21T09:00:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 -- start start 2024-04-21T09:00:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 Processor -- start 2024-04-21T09:00:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 0x556a39a4ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a45290 0x556a39a4b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a4de10 0x556a39a4b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556a39a50d20 con 0x556a39a45290 2024-04-21T09:00:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x556a39962580 con 0x556a39a4de10 2024-04-21T09:00:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x556a39a50ea0 con 0x556a39a448e0 2024-04-21T09:00:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f957b035740 10 client.0 _ll_register_callbacks cb 0x556a39968650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.809+0000 7f95761df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a45290 0x556a39a4b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f95761df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a45290 0x556a39a4b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39054/0 (socket says 172.21.15.28:39054) 2024-04-21T09:00:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f95761df640 1 -- 172.21.15.28:0/799238958 learned_addr learned my addr 172.21.15.28:0/799238958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f95759de640 1 --2- 172.21.15.28:0/799238958 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 0x556a39a4ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f95759de640 1 --2- 172.21.15.28:0/799238958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a4de10 0x556a39a4b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f95761df640 1 -- 172.21.15.28:0/799238958 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 msgr2=0x556a39a4ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f95761df640 1 --2- 172.21.15.28:0/799238958 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 0x556a39a4ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f95761df640 1 -- 172.21.15.28:0/799238958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a4de10 msgr2=0x556a39a4b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f95761df640 1 --2- 172.21.15.28:0/799238958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a4de10 0x556a39a4b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f95761df640 1 -- 172.21.15.28:0/799238958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a397e3f70 con 0x556a39a45290 2024-04-21T09:00:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f95761df640 1 --2- 172.21.15.28:0/799238958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a45290 0x556a39a4b3a0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f9568013790 tx=0x7f9568013c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ed871d36d0f77a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f9559ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f9559ffb640 1 -- 172.21.15.28:0/799238958 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9568020060 con 0x556a39a45290 2024-04-21T09:00:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f9559ffb640 1 -- 172.21.15.28:0/799238958 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f9568011dd0 con 0x556a39a45290 2024-04-21T09:00:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f957b035740 1 -- 172.21.15.28:0/799238958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f956002cc30 con 0x556a39a45290 2024-04-21T09:00:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f9559ffb640 1 -- 172.21.15.28:0/799238958 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9568027050 con 0x556a39a45290 2024-04-21T09:00:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f957b035740 10 client.17985 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.810+0000 7f957b035740 1 -- 172.21.15.28:0/799238958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556a3967b170 con 0x556a39a45290 2024-04-21T09:00:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.811+0000 7f9559ffb640 1 -- 172.21.15.28:0/799238958 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f956002cc30 con 0x556a39a45290 2024-04-21T09:00:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.811+0000 7f9559ffb640 1 -- 172.21.15.28:0/799238958 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f956801a030 con 0x556a39a45290 2024-04-21T09:00:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.811+0000 7f957b035740 10 client.17985 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.811+0000 7f957b035740 10 client.17985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.811+0000 7f957b035740 20 client.17985 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.811+0000 7f957b035740 1 --2- 172.21.15.28:0/799238958 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556a39a5e3d0 0x556a39a60810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.811+0000 7f957b035740 4 client.17985 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:00:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.811+0000 7f957b035740 1 -- 172.21.15.28:0/799238958 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a399b6920 con 0x556a39a5e3d0 2024-04-21T09:00:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.811+0000 7f95759de640 1 --2- 172.21.15.28:0/799238958 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556a39a5e3d0 0x556a39a60810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.811+0000 7f95759de640 1 --2- 172.21.15.28:0/799238958 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556a39a5e3d0 0x556a39a60810 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0e3811c6cd75850 server_cookie=bb20e5206cc4aa08 in_seq=0 out_seq=0 2024-04-21T09:00:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.811+0000 7f9559ffb640 10 client.17985 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.812+0000 7f9559ffb640 1 -- 172.21.15.28:0/799238958 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556a399b6920 con 0x556a39a5e3d0 2024-04-21T09:00:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.812+0000 7f9559ffb640 10 client.17985 handle_command_reply: tid=0 2024-04-21T09:00:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.812+0000 7f957b035740 1 client.17985 shutdown 2024-04-21T09:00:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.812+0000 7f957b035740 1 -- 172.21.15.28:0/799238958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a45290 msgr2=0x556a39a4b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.812+0000 7f957b035740 1 --2- 172.21.15.28:0/799238958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a45290 0x556a39a4b3a0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f9568013790 tx=0x7f9568013c50 comp rx=0 tx=0).stop 2024-04-21T09:00:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.812+0000 7f957b035740 1 -- 172.21.15.28:0/799238958 shutdown_connections 2024-04-21T09:00:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.812+0000 7f957b035740 1 --2- 172.21.15.28:0/799238958 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556a39a5e3d0 0x556a39a60810 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.812+0000 7f957b035740 1 --2- 172.21.15.28:0/799238958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a4de10 0x556a39a4b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.812+0000 7f9559ffb640 0 client.17985 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.812+0000 7f957b035740 1 --2- 172.21.15.28:0/799238958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a45290 0x556a39a4b3a0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.812+0000 7f957b035740 1 --2- 172.21.15.28:0/799238958 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 0x556a39a4ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.812+0000 7f957b035740 1 -- 172.21.15.28:0/799238958 >> 172.21.15.28:0/799238958 conn(0x556a39a3efb0 msgr2=0x556a39a40d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.812+0000 7f957b035740 1 -- 172.21.15.28:0/799238958 shutdown_connections 2024-04-21T09:00:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.812+0000 7f957b035740 1 -- 172.21.15.28:0/799238958 wait complete. 2024-04-21T09:00:18.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.813+0000 7f957b035740 20 client.17985 trim_cache size 0 max 16384 2024-04-21T09:00:18.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.830+0000 7f957b035740 1 Processor -- start 2024-04-21T09:00:18.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.830+0000 7f957b035740 1 -- start start 2024-04-21T09:00:18.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.830+0000 7f957b035740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 0x556a39a59be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.830+0000 7f957b035740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a45290 0x556a39a4b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.830+0000 7f957b035740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a4de10 0x556a39a660e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.830+0000 7f957b035740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f95701b3f20 con 0x556a39a4de10 2024-04-21T09:00:18.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f957b035740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f95701b17f0 con 0x556a39a45290 2024-04-21T09:00:18.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f957b035740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f95701af660 con 0x556a39a448e0 2024-04-21T09:00:18.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f95759de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a4de10 0x556a39a660e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f95759de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a4de10 0x556a39a660e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39060/0 (socket says 172.21.15.28:39060) 2024-04-21T09:00:18.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f95759de640 1 -- 172.21.15.28:0/2321806449 learned_addr learned my addr 172.21.15.28:0/2321806449 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:18.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f95761df640 1 --2- 172.21.15.28:0/2321806449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a45290 0x556a39a4b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f95759de640 1 --2- 172.21.15.28:0/2321806449 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 0x556a39a59be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f95759de640 1 -- 172.21.15.28:0/2321806449 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 msgr2=0x556a39a59be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f95759de640 1 --2- 172.21.15.28:0/2321806449 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 0x556a39a59be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f95759de640 1 -- 172.21.15.28:0/2321806449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a45290 msgr2=0x556a39a4b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f95759de640 1 --2- 172.21.15.28:0/2321806449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a45290 0x556a39a4b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f95759de640 1 -- 172.21.15.28:0/2321806449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a39a3eae0 con 0x556a39a4de10 2024-04-21T09:00:18.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f95759de640 1 --2- 172.21.15.28:0/2321806449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a4de10 0x556a39a660e0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f9560005c40 tx=0x7f9560005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ac82e39edaa762 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:18.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f955bfff640 1 -- 172.21.15.28:0/2321806449 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956005c070 con 0x556a39a4de10 2024-04-21T09:00:18.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f955bfff640 1 -- 172.21.15.28:0/2321806449 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f956002ca00 con 0x556a39a4de10 2024-04-21T09:00:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.831+0000 7f955bfff640 1 -- 172.21.15.28:0/2321806449 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9560045070 con 0x556a39a4de10 2024-04-21T09:00:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 -- 172.21.15.28:0/2321806449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a4de10 msgr2=0x556a39a660e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 --2- 172.21.15.28:0/2321806449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a4de10 0x556a39a660e0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f9560005c40 tx=0x7f9560005bb0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 -- 172.21.15.28:0/2321806449 shutdown_connections 2024-04-21T09:00:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 --2- 172.21.15.28:0/2321806449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a4de10 0x556a39a660e0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 --2- 172.21.15.28:0/2321806449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a45290 0x556a39a4b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 --2- 172.21.15.28:0/2321806449 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 0x556a39a59be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 -- 172.21.15.28:0/2321806449 >> 172.21.15.28:0/2321806449 conn(0x556a39a3efb0 msgr2=0x556a39a3e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:18.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 -- 172.21.15.28:0/2321806449 shutdown_connections 2024-04-21T09:00:18.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 -- 172.21.15.28:0/2321806449 wait complete. 2024-04-21T09:00:18.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 -- start start 2024-04-21T09:00:18.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 Processor -- start 2024-04-21T09:00:18.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 0x556a39a52270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a45290 0x556a39a527b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a4de10 0x556a39ac3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f95701afcb0 con 0x556a39a45290 2024-04-21T09:00:18.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f95701b1120 con 0x556a39a4de10 2024-04-21T09:00:18.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f95701b0380 con 0x556a39a448e0 2024-04-21T09:00:18.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f95761df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a45290 0x556a39a527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.832+0000 7f957b035740 10 client.0 _ll_register_callbacks cb 0x7f9568009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:18.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f95761df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a45290 0x556a39a527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39066/0 (socket says 172.21.15.28:39066) 2024-04-21T09:00:18.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f95761df640 1 -- 172.21.15.28:0/2990749961 learned_addr learned my addr 172.21.15.28:0/2990749961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:18.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f95759de640 1 --2- 172.21.15.28:0/2990749961 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 0x556a39a52270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f95759de640 1 --2- 172.21.15.28:0/2990749961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a4de10 0x556a39ac3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f95761df640 1 -- 172.21.15.28:0/2990749961 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 msgr2=0x556a39a52270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f95761df640 1 --2- 172.21.15.28:0/2990749961 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 0x556a39a52270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f95761df640 1 -- 172.21.15.28:0/2990749961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a4de10 msgr2=0x556a39ac3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f95761df640 1 --2- 172.21.15.28:0/2990749961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a4de10 0x556a39ac3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f95761df640 1 -- 172.21.15.28:0/2990749961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a39ac3ec0 con 0x556a39a45290 2024-04-21T09:00:18.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f95759de640 1 --2- 172.21.15.28:0/2990749961 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 0x556a39a52270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:18.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f95761df640 1 --2- 172.21.15.28:0/2990749961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a45290 0x556a39a527b0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f9568000c00 tx=0x7f95680117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca4a8bca9f500dc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:18.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f9559ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:18.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f9559ffb640 1 -- 172.21.15.28:0/2990749961 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956800e880 con 0x556a39a45290 2024-04-21T09:00:18.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f9559ffb640 1 -- 172.21.15.28:0/2990749961 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f956800ea20 con 0x556a39a45290 2024-04-21T09:00:18.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f957b035740 1 -- 172.21.15.28:0/2990749961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f956002ca00 con 0x556a39a45290 2024-04-21T09:00:18.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f9559ffb640 1 -- 172.21.15.28:0/2990749961 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9568038050 con 0x556a39a45290 2024-04-21T09:00:18.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f957b035740 10 client.17991 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:18.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f957b035740 1 -- 172.21.15.28:0/2990749961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556a39ac5230 con 0x556a39a45290 2024-04-21T09:00:18.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.833+0000 7f9559ffb640 1 -- 172.21.15.28:0/2990749961 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f956002ca00 con 0x556a39a45290 2024-04-21T09:00:18.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.834+0000 7f9559ffb640 1 -- 172.21.15.28:0/2990749961 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f9568033030 con 0x556a39a45290 2024-04-21T09:00:18.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.834+0000 7f957b035740 10 client.17991 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:18.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.834+0000 7f957b035740 10 client.17991 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.834+0000 7f957b035740 20 client.17991 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.834+0000 7f957b035740 1 --2- 172.21.15.28:0/2990749961 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556a39ac5a20 0x556a39ac7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.834+0000 7f957b035740 4 client.17991 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T09:00:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.834+0000 7f957b035740 1 -- 172.21.15.28:0/2990749961 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556a399b0790 con 0x556a39ac5a20 2024-04-21T09:00:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.834+0000 7f95759de640 1 --2- 172.21.15.28:0/2990749961 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556a39ac5a20 0x556a39ac7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.834+0000 7f95759de640 1 --2- 172.21.15.28:0/2990749961 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556a39ac5a20 0x556a39ac7e40 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5eea316e46ff3d66 server_cookie=3537385aa3cfa27 in_seq=0 out_seq=0 2024-04-21T09:00:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.834+0000 7f9559ffb640 10 client.17991 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.835+0000 7f9559ffb640 1 -- 172.21.15.28:0/2990749961 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556a399b0790 con 0x556a39ac5a20 2024-04-21T09:00:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.835+0000 7f9559ffb640 10 client.17991 handle_command_reply: tid=0 2024-04-21T09:00:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.835+0000 7f957b035740 1 client.17991 shutdown 2024-04-21T09:00:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.835+0000 7f957b035740 1 -- 172.21.15.28:0/2990749961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a45290 msgr2=0x556a39a527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.835+0000 7f957b035740 1 --2- 172.21.15.28:0/2990749961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a45290 0x556a39a527b0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f9568000c00 tx=0x7f95680117b0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f957b035740 1 -- 172.21.15.28:0/2990749961 shutdown_connections 2024-04-21T09:00:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f957b035740 1 --2- 172.21.15.28:0/2990749961 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x556a39ac5a20 0x556a39ac7e40 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f957b035740 1 --2- 172.21.15.28:0/2990749961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556a39a4de10 0x556a39ac3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f957b035740 1 --2- 172.21.15.28:0/2990749961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a39a45290 0x556a39a527b0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f9559ffb640 0 client.17991 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:18.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f957b035740 1 --2- 172.21.15.28:0/2990749961 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x556a39a448e0 0x556a39a52270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f957b035740 1 -- 172.21.15.28:0/2990749961 >> 172.21.15.28:0/2990749961 conn(0x556a39a3efb0 msgr2=0x556a39a40240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:18.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f957b035740 1 -- 172.21.15.28:0/2990749961 shutdown_connections 2024-04-21T09:00:18.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f957b035740 1 -- 172.21.15.28:0/2990749961 wait complete. 2024-04-21T09:00:18.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f957b035740 20 client.17991 trim_cache size 0 max 16384 2024-04-21T09:00:18.848 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T09:00:18.848 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T09:00:18.848 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "95df8bc2-4277-4c39-9b89-a4b71c73c4ef", 2024-04-21T09:00:18.848 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T09:00:18.848 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T09:00:18.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f9578443640 1 -- 172.21.15.28:0/3226121273 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f955c0a3d20 msgr2=0x7f955c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:18.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f9578443640 1 --2- 172.21.15.28:0/3226121273 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f955c0a3d20 0x7f955c0a61e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f956c0ce2c0 tx=0x7f956c005890 comp rx=0 tx=0).stop 2024-04-21T09:00:18.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f9578443640 1 -- 172.21.15.28:0/3226121273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9570106f50 msgr2=0x7f957019a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:18.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f9578443640 1 --2- 172.21.15.28:0/3226121273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9570106f50 0x7f957019a860 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f956c00eaa0 tx=0x7f956c00ef60 comp rx=0 tx=0).stop 2024-04-21T09:00:18.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f9578443640 1 -- 172.21.15.28:0/3226121273 shutdown_connections 2024-04-21T09:00:18.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f9578443640 1 --2- 172.21.15.28:0/3226121273 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f955c0a3d20 0x7f955c0a61e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f9578443640 1 --2- 172.21.15.28:0/3226121273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9570106f50 0x7f957019a860 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f9578443640 1 --2- 172.21.15.28:0/3226121273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9570100e80 0x7f95701a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f9578443640 1 --2- 172.21.15.28:0/3226121273 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95701004c0 0x7f95701a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f9578443640 1 -- 172.21.15.28:0/3226121273 >> 172.21.15.28:0/3226121273 conn(0x7f9570077450 msgr2=0x7f9570104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f9578443640 1 -- 172.21.15.28:0/3226121273 shutdown_connections 2024-04-21T09:00:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.836+0000 7f9578443640 1 -- 172.21.15.28:0/3226121273 wait complete. 2024-04-21T09:00:18.851 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T09:00:18.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T09:00:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.958+0000 7f256a6c8640 1 -- 172.21.15.28:0/2582068779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406a5b0 msgr2=0x7f256406aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.958+0000 7f256a6c8640 1 --2- 172.21.15.28:0/2582068779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406a5b0 0x7f256406aa30 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f2558009930 tx=0x7f255802bc30 comp rx=0 tx=0).stop 2024-04-21T09:00:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.958+0000 7f256a6c8640 1 -- 172.21.15.28:0/2582068779 shutdown_connections 2024-04-21T09:00:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.958+0000 7f256a6c8640 1 --2- 172.21.15.28:0/2582068779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f256406af70 0x7f256410ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.958+0000 7f256a6c8640 1 --2- 172.21.15.28:0/2582068779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406a5b0 0x7f256406aa30 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.958+0000 7f256a6c8640 1 --2- 172.21.15.28:0/2582068779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2564108060 0x7f2564108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.958+0000 7f256a6c8640 1 -- 172.21.15.28:0/2582068779 >> 172.21.15.28:0/2582068779 conn(0x7f2564077c40 msgr2=0x7f2564078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.958+0000 7f256a6c8640 1 -- 172.21.15.28:0/2582068779 shutdown_connections 2024-04-21T09:00:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.958+0000 7f256a6c8640 1 -- 172.21.15.28:0/2582068779 wait complete. 2024-04-21T09:00:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.959+0000 7f256a6c8640 1 Processor -- start 2024-04-21T09:00:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.959+0000 7f256a6c8640 1 -- start start 2024-04-21T09:00:18.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.959+0000 7f256a6c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406a5b0 0x7f25641985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.959+0000 7f256a6c8640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f256406af70 0x7f2564198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.959+0000 7f256a6c8640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2564108060 0x7f256419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.959+0000 7f256a6c8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2564112180 con 0x7f256406a5b0 2024-04-21T09:00:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.959+0000 7f256a6c8640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2564112000 con 0x7f256406af70 2024-04-21T09:00:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.959+0000 7f256a6c8640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2564112300 con 0x7f2564108060 2024-04-21T09:00:18.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.959+0000 7f2563fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406a5b0 0x7f25641985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.959+0000 7f2563fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406a5b0 0x7f25641985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39086/0 (socket says 172.21.15.28:39086) 2024-04-21T09:00:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.959+0000 7f2563fff640 1 -- 172.21.15.28:0/375423953 learned_addr learned my addr 172.21.15.28:0/375423953 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.959+0000 7f25637fe640 1 --2- 172.21.15.28:0/375423953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f256406af70 0x7f2564198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.959+0000 7f2568c65640 1 --2- 172.21.15.28:0/375423953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2564108060 0x7f256419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.960+0000 7f2563fff640 1 -- 172.21.15.28:0/375423953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2564108060 msgr2=0x7f256419cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.960+0000 7f2563fff640 1 --2- 172.21.15.28:0/375423953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2564108060 0x7f256419cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.960+0000 7f2563fff640 1 -- 172.21.15.28:0/375423953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f256406af70 msgr2=0x7f2564198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.960+0000 7f2563fff640 1 --2- 172.21.15.28:0/375423953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f256406af70 0x7f2564198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.960+0000 7f2563fff640 1 -- 172.21.15.28:0/375423953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256419d5d0 con 0x7f256406a5b0 2024-04-21T09:00:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.960+0000 7f2568c65640 1 --2- 172.21.15.28:0/375423953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2564108060 0x7f256419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:00:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.960+0000 7f2563fff640 1 --2- 172.21.15.28:0/375423953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406a5b0 0x7f25641985e0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f255400c920 tx=0x7f255400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.960+0000 7f25617fa640 1 -- 172.21.15.28:0/375423953 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2554007c70 con 0x7f256406a5b0 2024-04-21T09:00:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.960+0000 7f25617fa640 1 -- 172.21.15.28:0/375423953 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2554007e10 con 0x7f256406a5b0 2024-04-21T09:00:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.960+0000 7f256a6c8640 1 -- 172.21.15.28:0/375423953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f256419d8c0 con 0x7f256406a5b0 2024-04-21T09:00:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.960+0000 7f25617fa640 1 -- 172.21.15.28:0/375423953 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2554018820 con 0x7f256406a5b0 2024-04-21T09:00:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.960+0000 7f256a6c8640 1 -- 172.21.15.28:0/375423953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25641a5150 con 0x7f256406a5b0 2024-04-21T09:00:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f256d398740 1 Processor -- start 2024-04-21T09:00:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f25617fa640 1 -- 172.21.15.28:0/375423953 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f255400f450 con 0x7f256406a5b0 2024-04-21T09:00:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f256d398740 1 -- start start 2024-04-21T09:00:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f256d398740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4b32b0 0x56504e4b3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f256d398740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b3bd0 0x56504e4c0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f256d398740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4c0e90 0x56504e4c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f256d398740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56504e4181e0 con 0x56504e4c0e90 2024-04-21T09:00:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f256d398740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56504e3b5390 con 0x56504e4b3bd0 2024-04-21T09:00:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f256d398740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56504e4141a0 con 0x56504e4b32b0 2024-04-21T09:00:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f25637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4c0e90 0x56504e4c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f25637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4c0e90 0x56504e4c3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39092/0 (socket says 172.21.15.28:39092) 2024-04-21T09:00:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f25637fe640 1 -- 172.21.15.28:0/917862792 learned_addr learned my addr 172.21.15.28:0/917862792 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f2568c65640 1 --2- 172.21.15.28:0/917862792 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4b32b0 0x56504e4b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f2563fff640 1 --2- 172.21.15.28:0/917862792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b3bd0 0x56504e4c0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f25617fa640 1 --2- 172.21.15.28:0/375423953 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f254c0a3d20 0x7f254c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f25637fe640 1 -- 172.21.15.28:0/917862792 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4b32b0 msgr2=0x56504e4b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f25637fe640 1 --2- 172.21.15.28:0/917862792 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4b32b0 0x56504e4b3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f25637fe640 1 -- 172.21.15.28:0/917862792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b3bd0 msgr2=0x56504e4c0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f25637fe640 1 --2- 172.21.15.28:0/917862792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b3bd0 0x56504e4c0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f25637fe640 1 -- 172.21.15.28:0/917862792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56504e1f47d0 con 0x56504e4c0e90 2024-04-21T09:00:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.963+0000 7f2563fff640 1 --2- 172.21.15.28:0/917862792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b3bd0 0x56504e4c0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.964+0000 7f25617fa640 1 -- 172.21.15.28:0/375423953 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f25540d0d80 con 0x7f256406a5b0 2024-04-21T09:00:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.964+0000 7f2568c65640 1 --2- 172.21.15.28:0/375423953 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f254c0a3d20 0x7f254c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.964+0000 7f25637fe640 1 --2- 172.21.15.28:0/917862792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4c0e90 0x56504e4c3280 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f2558036040 tx=0x7f255802c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e7ba24b1403e1d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.964+0000 7f254a7fc640 1 -- 172.21.15.28:0/917862792 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2558045020 con 0x56504e4c0e90 2024-04-21T09:00:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.964+0000 7f254a7fc640 1 -- 172.21.15.28:0/917862792 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f255802cc90 con 0x56504e4c0e90 2024-04-21T09:00:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.964+0000 7f254a7fc640 1 -- 172.21.15.28:0/917862792 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25580046c0 con 0x56504e4c0e90 2024-04-21T09:00:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 -- 172.21.15.28:0/917862792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4c0e90 msgr2=0x56504e4c3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f2568c65640 1 --2- 172.21.15.28:0/375423953 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f254c0a3d20 0x7f254c0a61e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f255c005fd0 tx=0x7f255c004a40 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 --2- 172.21.15.28:0/917862792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4c0e90 0x56504e4c3280 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f2558036040 tx=0x7f255802c680 comp rx=0 tx=0).stop 2024-04-21T09:00:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 -- 172.21.15.28:0/917862792 shutdown_connections 2024-04-21T09:00:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 --2- 172.21.15.28:0/917862792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4c0e90 0x56504e4c3280 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 --2- 172.21.15.28:0/917862792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b3bd0 0x56504e4c0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 --2- 172.21.15.28:0/917862792 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4b32b0 0x56504e4b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 -- 172.21.15.28:0/917862792 >> 172.21.15.28:0/917862792 conn(0x56504e4adcb0 msgr2=0x56504e4ae090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 -- 172.21.15.28:0/917862792 shutdown_connections 2024-04-21T09:00:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 -- 172.21.15.28:0/917862792 wait complete. 2024-04-21T09:00:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 -- start start 2024-04-21T09:00:18.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 Processor -- start 2024-04-21T09:00:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b32b0 0x56504e4bde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b3bd0 0x56504e4be3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4c0e90 0x56504e4be900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56504e4c3bf0 con 0x56504e4b32b0 2024-04-21T09:00:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56504e3d12f0 con 0x56504e4b3bd0 2024-04-21T09:00:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.965+0000 7f256d398740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56504e4c3d70 con 0x56504e4c0e90 2024-04-21T09:00:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f256d398740 10 client.0 _ll_register_callbacks cb 0x56504e3de580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f2563fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b32b0 0x56504e4bde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f2563fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b32b0 0x56504e4bde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39102/0 (socket says 172.21.15.28:39102) 2024-04-21T09:00:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f2563fff640 1 -- 172.21.15.28:0/2624988158 learned_addr learned my addr 172.21.15.28:0/2624988158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f25637fe640 1 --2- 172.21.15.28:0/2624988158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b3bd0 0x56504e4be3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f2568c65640 1 --2- 172.21.15.28:0/2624988158 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4c0e90 0x56504e4be900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f2563fff640 1 -- 172.21.15.28:0/2624988158 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4c0e90 msgr2=0x56504e4be900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f2563fff640 1 --2- 172.21.15.28:0/2624988158 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4c0e90 0x56504e4be900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f2563fff640 1 -- 172.21.15.28:0/2624988158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b3bd0 msgr2=0x56504e4be3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f2563fff640 1 --2- 172.21.15.28:0/2624988158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b3bd0 0x56504e4be3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f2563fff640 1 -- 172.21.15.28:0/2624988158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56504e12a3f0 con 0x56504e4b32b0 2024-04-21T09:00:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f2563fff640 1 --2- 172.21.15.28:0/2624988158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b32b0 0x56504e4bde80 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2554007590 tx=0x7f255409c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e02bb8b32c41044 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f25437fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f25437fe640 1 -- 172.21.15.28:0/2624988158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25540a0c80 con 0x56504e4b32b0 2024-04-21T09:00:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f25437fe640 1 -- 172.21.15.28:0/2624988158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f25540870f0 con 0x56504e4b32b0 2024-04-21T09:00:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f256d398740 1 -- 172.21.15.28:0/2624988158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f255802cc90 con 0x56504e4b32b0 2024-04-21T09:00:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f25437fe640 1 -- 172.21.15.28:0/2624988158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f25540a5540 con 0x56504e4b32b0 2024-04-21T09:00:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f256d398740 10 client.18003 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f256d398740 1 -- 172.21.15.28:0/2624988158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56504e0ea0f0 con 0x56504e4b32b0 2024-04-21T09:00:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.966+0000 7f25437fe640 1 -- 172.21.15.28:0/2624988158 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f255802cc90 con 0x56504e4b32b0 2024-04-21T09:00:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.967+0000 7f25437fe640 1 -- 172.21.15.28:0/2624988158 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f2554098620 con 0x56504e4b32b0 2024-04-21T09:00:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.967+0000 7f256d398740 10 client.18003 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.967+0000 7f256d398740 10 client.18003 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.967+0000 7f256d398740 20 client.18003 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.967+0000 7f256d398740 1 --2- 172.21.15.28:0/2624988158 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56504e4d0d90 0x56504e4d31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.967+0000 7f256d398740 4 client.18003 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:00:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.967+0000 7f256d398740 1 -- 172.21.15.28:0/2624988158 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56504e425c00 con 0x56504e4d0d90 2024-04-21T09:00:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.967+0000 7f25637fe640 1 --2- 172.21.15.28:0/2624988158 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56504e4d0d90 0x56504e4d31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.967+0000 7f25637fe640 1 --2- 172.21.15.28:0/2624988158 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56504e4d0d90 0x56504e4d31c0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bd26b9377e15a64 server_cookie=e611421555d4696 in_seq=0 out_seq=0 2024-04-21T09:00:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.967+0000 7f25437fe640 10 client.18003 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.968+0000 7f25437fe640 1 -- 172.21.15.28:0/2624988158 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56504e425c00 con 0x56504e4d0d90 2024-04-21T09:00:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.968+0000 7f25437fe640 10 client.18003 handle_command_reply: tid=0 2024-04-21T09:00:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.968+0000 7f256d398740 1 client.18003 shutdown 2024-04-21T09:00:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.968+0000 7f256d398740 1 -- 172.21.15.28:0/2624988158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b32b0 msgr2=0x56504e4bde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.968+0000 7f256d398740 1 --2- 172.21.15.28:0/2624988158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b32b0 0x56504e4bde80 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f2554007590 tx=0x7f255409c2f0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.968+0000 7f256d398740 1 -- 172.21.15.28:0/2624988158 shutdown_connections 2024-04-21T09:00:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.968+0000 7f256d398740 1 --2- 172.21.15.28:0/2624988158 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56504e4d0d90 0x56504e4d31c0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.968+0000 7f256d398740 1 --2- 172.21.15.28:0/2624988158 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4c0e90 0x56504e4be900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.968+0000 7f256d398740 1 --2- 172.21.15.28:0/2624988158 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b3bd0 0x56504e4be3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.968+0000 7f256d398740 1 --2- 172.21.15.28:0/2624988158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b32b0 0x56504e4bde80 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.968+0000 7f25437fe640 0 client.18003 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.968+0000 7f256d398740 1 -- 172.21.15.28:0/2624988158 >> 172.21.15.28:0/2624988158 conn(0x56504e4adcb0 msgr2=0x56504e4afa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.968+0000 7f256d398740 1 -- 172.21.15.28:0/2624988158 shutdown_connections 2024-04-21T09:00:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.968+0000 7f256d398740 1 -- 172.21.15.28:0/2624988158 wait complete. 2024-04-21T09:00:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.969+0000 7f256d398740 20 client.18003 trim_cache size 0 max 16384 2024-04-21T09:00:18.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.986+0000 7f256d398740 1 Processor -- start 2024-04-21T09:00:18.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.986+0000 7f256d398740 1 -- start start 2024-04-21T09:00:18.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.986+0000 7f256d398740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b32b0 0x56504e4c05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.986+0000 7f256d398740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b3bd0 0x56504e4ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.986+0000 7f256d398740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4c0e90 0x56504e4d8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.986+0000 7f256d398740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25641afac0 con 0x56504e4b3bd0 2024-04-21T09:00:18.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.986+0000 7f256d398740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f25641ad390 con 0x56504e4b32b0 2024-04-21T09:00:18.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.986+0000 7f256d398740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f25641ab200 con 0x56504e4c0e90 2024-04-21T09:00:18.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.986+0000 7f25637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b3bd0 0x56504e4ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.986+0000 7f25637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b3bd0 0x56504e4ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39110/0 (socket says 172.21.15.28:39110) 2024-04-21T09:00:18.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.986+0000 7f25637fe640 1 -- 172.21.15.28:0/1543133328 learned_addr learned my addr 172.21.15.28:0/1543133328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:18.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.986+0000 7f2563fff640 1 --2- 172.21.15.28:0/1543133328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b32b0 0x56504e4c05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.986+0000 7f25637fe640 1 -- 172.21.15.28:0/1543133328 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4c0e90 msgr2=0x56504e4d8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f25637fe640 1 --2- 172.21.15.28:0/1543133328 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4c0e90 0x56504e4d8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f25637fe640 1 -- 172.21.15.28:0/1543133328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b32b0 msgr2=0x56504e4c05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f25637fe640 1 --2- 172.21.15.28:0/1543133328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b32b0 0x56504e4c05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f25637fe640 1 -- 172.21.15.28:0/1543133328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56504e4b2ee0 con 0x56504e4b3bd0 2024-04-21T09:00:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f2563fff640 1 --2- 172.21.15.28:0/1543133328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b32b0 0x56504e4c05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:00:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f25637fe640 1 --2- 172.21.15.28:0/1543133328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b3bd0 0x56504e4ccaf0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f2558009930 tx=0x7f255802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75cbf538eb72ce8d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f2549ffb640 1 -- 172.21.15.28:0/1543133328 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2558045040 con 0x56504e4b3bd0 2024-04-21T09:00:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f2549ffb640 1 -- 172.21.15.28:0/1543133328 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f255802cc90 con 0x56504e4b3bd0 2024-04-21T09:00:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f2549ffb640 1 -- 172.21.15.28:0/1543133328 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2558034a70 con 0x56504e4b3bd0 2024-04-21T09:00:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f256d398740 1 -- 172.21.15.28:0/1543133328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b3bd0 msgr2=0x56504e4ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f256d398740 1 --2- 172.21.15.28:0/1543133328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b3bd0 0x56504e4ccaf0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f2558009930 tx=0x7f255802bc30 comp rx=0 tx=0).stop 2024-04-21T09:00:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f256d398740 1 -- 172.21.15.28:0/1543133328 shutdown_connections 2024-04-21T09:00:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f256d398740 1 --2- 172.21.15.28:0/1543133328 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4c0e90 0x56504e4d8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f256d398740 1 --2- 172.21.15.28:0/1543133328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b3bd0 0x56504e4ccaf0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f256d398740 1 --2- 172.21.15.28:0/1543133328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4b32b0 0x56504e4c05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f256d398740 1 -- 172.21.15.28:0/1543133328 >> 172.21.15.28:0/1543133328 conn(0x56504e4adcb0 msgr2=0x56504e4be6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.987+0000 7f256d398740 1 -- 172.21.15.28:0/1543133328 shutdown_connections 2024-04-21T09:00:18.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.988+0000 7f256d398740 1 -- 172.21.15.28:0/1543133328 wait complete. 2024-04-21T09:00:18.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.988+0000 7f256d398740 1 -- start start 2024-04-21T09:00:18.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.988+0000 7f256d398740 1 Processor -- start 2024-04-21T09:00:18.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.988+0000 7f256d398740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4b32b0 0x56504e4baa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.988+0000 7f256d398740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b3bd0 0x56504e4bafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.988+0000 7f256d398740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4c0e90 0x56504e4bb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:18.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.988+0000 7f256d398740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25641ab850 con 0x56504e4b3bd0 2024-04-21T09:00:18.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.988+0000 7f256d398740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f25641accc0 con 0x56504e4c0e90 2024-04-21T09:00:18.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.988+0000 7f256d398740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f25641abf20 con 0x56504e4b32b0 2024-04-21T09:00:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.988+0000 7f256d398740 10 client.0 _ll_register_callbacks cb 0x7f255400c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.988+0000 7f25637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b3bd0 0x56504e4bafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.988+0000 7f25637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b3bd0 0x56504e4bafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39118/0 (socket says 172.21.15.28:39118) 2024-04-21T09:00:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.988+0000 7f25637fe640 1 -- 172.21.15.28:0/341995487 learned_addr learned my addr 172.21.15.28:0/341995487 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.988+0000 7f2563fff640 1 --2- 172.21.15.28:0/341995487 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4b32b0 0x56504e4baa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.988+0000 7f2568c65640 1 --2- 172.21.15.28:0/341995487 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4c0e90 0x56504e4bb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f25637fe640 1 -- 172.21.15.28:0/341995487 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4b32b0 msgr2=0x56504e4baa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f25637fe640 1 --2- 172.21.15.28:0/341995487 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4b32b0 0x56504e4baa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f25637fe640 1 -- 172.21.15.28:0/341995487 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4c0e90 msgr2=0x56504e4bb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f25637fe640 1 --2- 172.21.15.28:0/341995487 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4c0e90 0x56504e4bb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f25637fe640 1 -- 172.21.15.28:0/341995487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56504e536300 con 0x56504e4b3bd0 2024-04-21T09:00:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f2568c65640 1 --2- 172.21.15.28:0/341995487 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4c0e90 0x56504e4bb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f25637fe640 1 --2- 172.21.15.28:0/341995487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b3bd0 0x56504e4bafb0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f255802c170 tx=0x7f25580345a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915c2e03b0337828 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f25437fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f25437fe640 1 -- 172.21.15.28:0/341995487 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2558045040 con 0x56504e4b3bd0 2024-04-21T09:00:18.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f25437fe640 1 -- 172.21.15.28:0/341995487 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9621+0+0 (secure 0 0 0) 0x7f255802c9e0 con 0x56504e4b3bd0 2024-04-21T09:00:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f256d398740 1 -- 172.21.15.28:0/341995487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f255802cc90 con 0x56504e4b3bd0 2024-04-21T09:00:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f25437fe640 1 -- 172.21.15.28:0/341995487 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f25580347f0 con 0x56504e4b3bd0 2024-04-21T09:00:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f256d398740 10 client.18009 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f256d398740 1 -- 172.21.15.28:0/341995487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56504e537480 con 0x56504e4b3bd0 2024-04-21T09:00:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f25437fe640 1 -- 172.21.15.28:0/341995487 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f255802cc90 con 0x56504e4b3bd0 2024-04-21T09:00:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f25437fe640 1 -- 172.21.15.28:0/341995487 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f25580418a0 con 0x56504e4b3bd0 2024-04-21T09:00:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f256d398740 10 client.18009 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f256d398740 10 client.18009 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:18.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f256d398740 20 client.18009 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f256d398740 1 --2- 172.21.15.28:0/341995487 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56504e537800 0x56504e539bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f256d398740 4 client.18009 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T09:00:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.989+0000 7f256d398740 1 -- 172.21.15.28:0/341995487 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56504e41fa70 con 0x56504e537800 2024-04-21T09:00:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.990+0000 7f2563fff640 1 --2- 172.21.15.28:0/341995487 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56504e537800 0x56504e539bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.990+0000 7f2563fff640 1 --2- 172.21.15.28:0/341995487 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56504e537800 0x56504e539bf0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d4082bab78c3d2f server_cookie=9cdbafb8426ea146 in_seq=0 out_seq=0 2024-04-21T09:00:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.990+0000 7f25437fe640 10 client.18009 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.990+0000 7f25437fe640 1 -- 172.21.15.28:0/341995487 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56504e41fa70 con 0x56504e537800 2024-04-21T09:00:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.990+0000 7f25437fe640 10 client.18009 handle_command_reply: tid=0 2024-04-21T09:00:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.990+0000 7f256d398740 1 client.18009 shutdown 2024-04-21T09:00:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.990+0000 7f256d398740 1 -- 172.21.15.28:0/341995487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b3bd0 msgr2=0x56504e4bafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.990+0000 7f256d398740 1 --2- 172.21.15.28:0/341995487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b3bd0 0x56504e4bafb0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f255802c170 tx=0x7f25580345a0 comp rx=0 tx=0).stop 2024-04-21T09:00:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.990+0000 7f256d398740 1 -- 172.21.15.28:0/341995487 shutdown_connections 2024-04-21T09:00:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.990+0000 7f256d398740 1 --2- 172.21.15.28:0/341995487 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56504e537800 0x56504e539bf0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.990+0000 7f256d398740 1 --2- 172.21.15.28:0/341995487 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56504e4c0e90 0x56504e4bb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.990+0000 7f256d398740 1 --2- 172.21.15.28:0/341995487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56504e4b3bd0 0x56504e4bafb0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:19.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.990+0000 7f25437fe640 0 client.18009 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:19.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.990+0000 7f256d398740 1 --2- 172.21.15.28:0/341995487 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56504e4b32b0 0x56504e4baa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:19.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.990+0000 7f256d398740 1 -- 172.21.15.28:0/341995487 >> 172.21.15.28:0/341995487 conn(0x56504e4adcb0 msgr2=0x56504e4ac4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:19.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.991+0000 7f256d398740 1 -- 172.21.15.28:0/341995487 shutdown_connections 2024-04-21T09:00:19.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.991+0000 7f256d398740 1 -- 172.21.15.28:0/341995487 wait complete. 2024-04-21T09:00:19.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.991+0000 7f256d398740 20 client.18009 trim_cache size 0 max 16384 2024-04-21T09:00:19.003 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T09:00:19.003 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T09:00:19.003 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T09:00:19.003 INFO:teuthology.orchestra.run.smithi028.stdout: "95df8bc2-4277-4c39-9b89-a4b71c73c4ef": { 2024-04-21T09:00:19.003 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T09:00:19.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "95df8bc2-4277-4c39-9b89-a4b71c73c4ef", 2024-04-21T09:00:19.004 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T09:00:19.004 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:00:19.004 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:00:19.004 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T09:00:19.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.991+0000 7f256a6c8640 1 -- 172.21.15.28:0/375423953 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f254c0a3d20 msgr2=0x7f254c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:19.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.991+0000 7f256a6c8640 1 --2- 172.21.15.28:0/375423953 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f254c0a3d20 0x7f254c0a61e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f255c005fd0 tx=0x7f255c004a40 comp rx=0 tx=0).stop 2024-04-21T09:00:19.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.991+0000 7f256a6c8640 1 -- 172.21.15.28:0/375423953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406a5b0 msgr2=0x7f25641985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:19.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.991+0000 7f256a6c8640 1 --2- 172.21.15.28:0/375423953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406a5b0 0x7f25641985e0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f255400c920 tx=0x7f255400cde0 comp rx=0 tx=0).stop 2024-04-21T09:00:19.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.991+0000 7f256a6c8640 1 -- 172.21.15.28:0/375423953 shutdown_connections 2024-04-21T09:00:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.991+0000 7f256a6c8640 1 --2- 172.21.15.28:0/375423953 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f254c0a3d20 0x7f254c0a61e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.991+0000 7f256a6c8640 1 --2- 172.21.15.28:0/375423953 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2564108060 0x7f256419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.991+0000 7f256a6c8640 1 --2- 172.21.15.28:0/375423953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f256406af70 0x7f2564198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.991+0000 7f256a6c8640 1 --2- 172.21.15.28:0/375423953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406a5b0 0x7f25641985e0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.991+0000 7f256a6c8640 1 -- 172.21.15.28:0/375423953 >> 172.21.15.28:0/375423953 conn(0x7f2564077c40 msgr2=0x7f25640ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.991+0000 7f256a6c8640 1 -- 172.21.15.28:0/375423953 shutdown_connections 2024-04-21T09:00:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:18.991+0000 7f256a6c8640 1 -- 172.21.15.28:0/375423953 wait complete. 2024-04-21T09:00:19.008 INFO:tasks.cephfs.filesystem:scrub status for tag:95df8bc2-4277-4c39-9b89-a4b71c73c4ef - {'path': '/', 'tag': '95df8bc2-4277-4c39-9b89-a4b71c73c4ef', 'options': 'recursive,force'} 2024-04-21T09:00:19.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:19 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:19.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:19 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:19.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:19 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:19.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:19 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:19.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:19 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:19.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:19 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:19.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:19 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:19.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:19 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:19.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:19 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:19.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:19 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:19.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:19 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:19.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:19 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:19.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:19 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:19.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:19 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:19.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:19 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:19.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:19 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:19.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:19 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:19.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:19 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:19.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:19 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:19.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:19 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:19.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:19 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:19.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:19 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:19.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:19 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:19.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:19 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:20 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T09:00:20.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:20 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:20 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:20.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:20 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:20.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:20 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:20 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:20.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:20 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:20 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:20.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:20 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:20 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:20.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:20 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:20 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:20 smithi152 ceph-mon[31401]: pgmap v282: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 171 B/s wr, 6 op/s 2024-04-21T09:00:20.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:20 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T09:00:20.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:20 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:20 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:20.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:20 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:20.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:20 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:20 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:20.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:20 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:20 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:20.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:20 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:20 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:20.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:20 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:20 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:20 smithi028 ceph-mon[28003]: pgmap v282: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 171 B/s wr, 6 op/s 2024-04-21T09:00:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:20 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T09:00:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:20 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:20 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:20.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:20 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:20.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:20 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:20 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:20.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:20 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:20 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:20.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:20 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:20 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:20.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:20 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:20 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:20.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:20 smithi171 ceph-mon[31358]: pgmap v282: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 171 B/s wr, 6 op/s 2024-04-21T09:00:22.421 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:22 smithi171 ceph-mon[31358]: pgmap v283: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-21T09:00:22.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:22 smithi152 ceph-mon[31401]: pgmap v283: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-21T09:00:22.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:22 smithi028 ceph-mon[28003]: pgmap v283: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-21T09:00:24.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:24 smithi152 ceph-mon[31401]: pgmap v284: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 85 B/s wr, 21 op/s 2024-04-21T09:00:24.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:24 smithi152 ceph-mon[31401]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T09:00:24.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:24 smithi028 ceph-mon[28003]: pgmap v284: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 85 B/s wr, 21 op/s 2024-04-21T09:00:24.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:24 smithi028 ceph-mon[28003]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T09:00:24.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:24 smithi171 ceph-mon[31358]: pgmap v284: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 85 B/s wr, 21 op/s 2024-04-21T09:00:24.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:24 smithi171 ceph-mon[31358]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T09:00:26.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:26 smithi152 ceph-mon[31401]: pgmap v286: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 33 op/s 2024-04-21T09:00:26.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:26 smithi028 ceph-mon[28003]: pgmap v286: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 33 op/s 2024-04-21T09:00:26.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:26 smithi171 ceph-mon[31358]: pgmap v286: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 33 op/s 2024-04-21T09:00:27.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:27 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T09:00:27.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:27 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T09:00:27.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:27 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T09:00:27.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:27 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T09:00:27.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:27 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T09:00:27.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:27 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T09:00:28.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:28 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T09:00:28.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:28 smithi028 ceph-mon[28003]: pgmap v287: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 33 op/s 2024-04-21T09:00:28.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:28 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T09:00:28.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:28 smithi152 ceph-mon[31401]: pgmap v287: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 33 op/s 2024-04-21T09:00:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:28 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T09:00:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:28 smithi171 ceph-mon[31358]: pgmap v287: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 33 op/s 2024-04-21T09:00:30.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:30 smithi152 ceph-mon[31401]: pgmap v288: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 33 op/s 2024-04-21T09:00:30.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:30 smithi028 ceph-mon[28003]: pgmap v288: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 33 op/s 2024-04-21T09:00:30.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:30 smithi171 ceph-mon[31358]: pgmap v288: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 33 op/s 2024-04-21T09:00:32.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:32 smithi152 ceph-mon[31401]: pgmap v289: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 27 op/s 2024-04-21T09:00:32.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:32 smithi028 ceph-mon[28003]: pgmap v289: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 27 op/s 2024-04-21T09:00:32.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:32 smithi171 ceph-mon[31358]: pgmap v289: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 27 op/s 2024-04-21T09:00:34.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:34 smithi152 ceph-mon[31401]: pgmap v290: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 op/s 2024-04-21T09:00:34.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:34 smithi028 ceph-mon[28003]: pgmap v290: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 op/s 2024-04-21T09:00:34.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:34 smithi171 ceph-mon[31358]: pgmap v290: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 op/s 2024-04-21T09:00:36.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:36 smithi152 ceph-mon[31401]: pgmap v291: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-21T09:00:36.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:36 smithi028 ceph-mon[28003]: pgmap v291: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-21T09:00:36.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:36 smithi171 ceph-mon[31358]: pgmap v291: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-21T09:00:38.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:38 smithi152 ceph-mon[31401]: pgmap v292: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:00:38.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:38 smithi028 ceph-mon[28003]: pgmap v292: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:00:38.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:38 smithi171 ceph-mon[31358]: pgmap v292: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:00:40.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:40 smithi152 ceph-mon[31401]: pgmap v293: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T09:00:40.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:40 smithi028 ceph-mon[28003]: pgmap v293: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T09:00:40.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:40 smithi171 ceph-mon[31358]: pgmap v293: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T09:00:42.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:42 smithi152 ceph-mon[31401]: pgmap v294: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T09:00:42.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:42 smithi028 ceph-mon[28003]: pgmap v294: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T09:00:42.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:42 smithi171 ceph-mon[31358]: pgmap v294: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T09:00:43.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:43 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T09:00:43.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:43 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T09:00:43.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:43 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T09:00:43.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:43 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:00:43.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:43 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T09:00:43.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:43 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T09:00:43.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:43 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T09:00:43.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:43 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:00:43.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:43 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T09:00:43.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:43 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T09:00:43.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:43 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T09:00:43.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:43 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:00:44.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:44 smithi152 ceph-mon[31401]: pgmap v295: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:00:44.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:44 smithi028 ceph-mon[28003]: pgmap v295: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:00:44.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:44 smithi171 ceph-mon[31358]: pgmap v295: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:00:46.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:46 smithi152 ceph-mon[31401]: pgmap v296: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T09:00:46.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:46 smithi028 ceph-mon[28003]: pgmap v296: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T09:00:46.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:46 smithi171 ceph-mon[31358]: pgmap v296: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T09:00:48.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:48 smithi152 ceph-mon[31401]: pgmap v297: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:00:48.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:48 smithi028 ceph-mon[28003]: pgmap v297: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:00:48.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:48 smithi171 ceph-mon[31358]: pgmap v297: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:00:49.009 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T09:00:49.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T09:00:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.128+0000 7fad29eba640 1 -- 172.21.15.28:0/783392686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2406af70 msgr2=0x7fad2410edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.128+0000 7fad29eba640 1 --2- 172.21.15.28:0/783392686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2406af70 0x7fad2410edb0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009ca0 tx=0x7fad1c02c0f0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.128+0000 7fad29eba640 1 -- 172.21.15.28:0/783392686 shutdown_connections 2024-04-21T09:00:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.128+0000 7fad29eba640 1 --2- 172.21.15.28:0/783392686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2406af70 0x7fad2410edb0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.128+0000 7fad29eba640 1 --2- 172.21.15.28:0/783392686 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad2406a5b0 0x7fad2406aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.128+0000 7fad29eba640 1 --2- 172.21.15.28:0/783392686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad24106f20 0x7fad24107300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.128+0000 7fad29eba640 1 -- 172.21.15.28:0/783392686 >> 172.21.15.28:0/783392686 conn(0x7fad24077c40 msgr2=0x7fad24078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.128+0000 7fad29eba640 1 -- 172.21.15.28:0/783392686 shutdown_connections 2024-04-21T09:00:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.128+0000 7fad29eba640 1 -- 172.21.15.28:0/783392686 wait complete. 2024-04-21T09:00:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.129+0000 7fad29eba640 1 Processor -- start 2024-04-21T09:00:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.129+0000 7fad29eba640 1 -- start start 2024-04-21T09:00:49.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.129+0000 7fad29eba640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad2406a5b0 0x7fad241a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.129+0000 7fad29eba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2406af70 0x7fad241a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.129+0000 7fad29eba640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad24106f20 0x7fad2419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.129+0000 7fad29eba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad24112190 con 0x7fad2406af70 2024-04-21T09:00:49.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.129+0000 7fad29eba640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fad24112010 con 0x7fad2406a5b0 2024-04-21T09:00:49.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.129+0000 7fad29eba640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fad24112310 con 0x7fad24106f20 2024-04-21T09:00:49.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad22ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2406af70 0x7fad241a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad22ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2406af70 0x7fad241a0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34098/0 (socket says 172.21.15.28:34098) 2024-04-21T09:00:49.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad22ffd640 1 -- 172.21.15.28:0/1333641555 learned_addr learned my addr 172.21.15.28:0/1333641555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad237fe640 1 --2- 172.21.15.28:0/1333641555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad2406a5b0 0x7fad241a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad23fff640 1 --2- 172.21.15.28:0/1333641555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad24106f20 0x7fad2419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad22ffd640 1 -- 172.21.15.28:0/1333641555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad24106f20 msgr2=0x7fad2419a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad22ffd640 1 --2- 172.21.15.28:0/1333641555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad24106f20 0x7fad2419a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad22ffd640 1 -- 172.21.15.28:0/1333641555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad2406a5b0 msgr2=0x7fad241a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad22ffd640 1 --2- 172.21.15.28:0/1333641555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad2406a5b0 0x7fad241a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad22ffd640 1 -- 172.21.15.28:0/1333641555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2419af10 con 0x7fad2406af70 2024-04-21T09:00:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad237fe640 1 --2- 172.21.15.28:0/1333641555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad2406a5b0 0x7fad241a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:49.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad22ffd640 1 --2- 172.21.15.28:0/1333641555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2406af70 0x7fad241a0ad0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fad1800d950 tx=0x7fad1800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad20ff9640 1 -- 172.21.15.28:0/1333641555 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18017070 con 0x7fad2406af70 2024-04-21T09:00:49.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad20ff9640 1 -- 172.21.15.28:0/1333641555 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad18009c30 con 0x7fad2406af70 2024-04-21T09:00:49.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad29eba640 1 -- 172.21.15.28:0/1333641555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2419b200 con 0x7fad2406af70 2024-04-21T09:00:49.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad20ff9640 1 -- 172.21.15.28:0/1333641555 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1800f620 con 0x7fad2406af70 2024-04-21T09:00:49.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.130+0000 7fad29eba640 1 -- 172.21.15.28:0/1333641555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad241a7510 con 0x7fad2406af70 2024-04-21T09:00:49.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.133+0000 7fad20ff9640 1 -- 172.21.15.28:0/1333641555 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fad18010460 con 0x7fad2406af70 2024-04-21T09:00:49.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.133+0000 7fad2cb86740 1 Processor -- start 2024-04-21T09:00:49.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.133+0000 7fad2cb86740 1 -- start start 2024-04-21T09:00:49.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.133+0000 7fad2cb86740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5de580 0x55d16a5de960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.133+0000 7fad2cb86740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5deea0 0x55d16a5e7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.133+0000 7fad20ff9640 1 --2- 172.21.15.28:0/1333641555 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fad0c0a3d20 0x7fad0c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.133+0000 7fad2cb86740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5e7a80 0x55d16a5e9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.133+0000 7fad2cb86740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d16a5431e0 con 0x55d16a5e7a80 2024-04-21T09:00:49.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.133+0000 7fad2cb86740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55d16a4e0390 con 0x55d16a5deea0 2024-04-21T09:00:49.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.133+0000 7fad2cb86740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55d16a53f1a0 con 0x55d16a5de580 2024-04-21T09:00:49.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.133+0000 7fad23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5e7a80 0x55d16a5e9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.133+0000 7fad23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5e7a80 0x55d16a5e9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34114/0 (socket says 172.21.15.28:34114) 2024-04-21T09:00:49.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.133+0000 7fad23fff640 1 -- 172.21.15.28:0/422077037 learned_addr learned my addr 172.21.15.28:0/422077037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.133+0000 7fad22ffd640 1 --2- 172.21.15.28:0/1333641555 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fad0c0a3d20 0x7fad0c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.133+0000 7fad23fff640 1 --2- 172.21.15.28:0/422077037 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5de580 0x55d16a5de960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.134+0000 7fad20ff9640 1 -- 172.21.15.28:0/1333641555 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7fad180d0900 con 0x7fad2406af70 2024-04-21T09:00:49.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.134+0000 7fad237fe640 1 --2- 172.21.15.28:0/422077037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5deea0 0x55d16a5e7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.134+0000 7fad23fff640 1 -- 172.21.15.28:0/422077037 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5de580 msgr2=0x55d16a5de960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.134+0000 7fad23fff640 1 --2- 172.21.15.28:0/422077037 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5de580 0x55d16a5de960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.134+0000 7fad23fff640 1 -- 172.21.15.28:0/422077037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5deea0 msgr2=0x55d16a5e7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.134+0000 7fad23fff640 1 --2- 172.21.15.28:0/422077037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5deea0 0x55d16a5e7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.134+0000 7fad23fff640 1 -- 172.21.15.28:0/422077037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d16a31f7d0 con 0x55d16a5e7a80 2024-04-21T09:00:49.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.134+0000 7fad237fe640 1 --2- 172.21.15.28:0/422077037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5deea0 0x55d16a5e7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:49.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.134+0000 7fad23fff640 1 --2- 172.21.15.28:0/422077037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5e7a80 0x55d16a5e9e70 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fad1c004af0 tx=0x7fad1c02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94cec76653d17b50 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.134+0000 7fad22ffd640 1 --2- 172.21.15.28:0/1333641555 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fad0c0a3d20 0x7fad0c0a61e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fad180cdd10 tx=0x7fad1806def0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.134+0000 7fad09ffb640 1 -- 172.21.15.28:0/422077037 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c049070 con 0x55d16a5e7a80 2024-04-21T09:00:49.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.134+0000 7fad09ffb640 1 -- 172.21.15.28:0/422077037 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad1c032960 con 0x55d16a5e7a80 2024-04-21T09:00:49.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.134+0000 7fad09ffb640 1 -- 172.21.15.28:0/422077037 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c039070 con 0x55d16a5e7a80 2024-04-21T09:00:49.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.135+0000 7fad2cb86740 1 -- 172.21.15.28:0/422077037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5e7a80 msgr2=0x55d16a5e9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.135+0000 7fad2cb86740 1 --2- 172.21.15.28:0/422077037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5e7a80 0x55d16a5e9e70 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fad1c004af0 tx=0x7fad1c02e040 comp rx=0 tx=0).stop 2024-04-21T09:00:49.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.135+0000 7fad2cb86740 1 -- 172.21.15.28:0/422077037 shutdown_connections 2024-04-21T09:00:49.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.135+0000 7fad2cb86740 1 --2- 172.21.15.28:0/422077037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5e7a80 0x55d16a5e9e70 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.135+0000 7fad2cb86740 1 --2- 172.21.15.28:0/422077037 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5deea0 0x55d16a5e7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.135+0000 7fad2cb86740 1 --2- 172.21.15.28:0/422077037 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5de580 0x55d16a5de960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.135+0000 7fad2cb86740 1 -- 172.21.15.28:0/422077037 >> 172.21.15.28:0/422077037 conn(0x55d16a5d8cb0 msgr2=0x55d16a5d9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.135+0000 7fad2cb86740 1 -- 172.21.15.28:0/422077037 shutdown_connections 2024-04-21T09:00:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.135+0000 7fad2cb86740 1 -- 172.21.15.28:0/422077037 wait complete. 2024-04-21T09:00:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.135+0000 7fad2cb86740 1 -- start start 2024-04-21T09:00:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad2cb86740 1 Processor -- start 2024-04-21T09:00:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad2cb86740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5de580 0x55d16a5e4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad2cb86740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5deea0 0x55d16a5e4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad2cb86740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5e7a80 0x55d16a5e5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad2cb86740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d16a5ea810 con 0x55d16a5de580 2024-04-21T09:00:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad2cb86740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55d16a4fc2f0 con 0x55d16a5e7a80 2024-04-21T09:00:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad2cb86740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55d16a5ea990 con 0x55d16a5deea0 2024-04-21T09:00:49.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad2cb86740 10 client.0 _ll_register_callbacks cb 0x55d16a509580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:49.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5de580 0x55d16a5e4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5de580 0x55d16a5e4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34122/0 (socket says 172.21.15.28:34122) 2024-04-21T09:00:49.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad23fff640 1 -- 172.21.15.28:0/2099072369 learned_addr learned my addr 172.21.15.28:0/2099072369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad237fe640 1 --2- 172.21.15.28:0/2099072369 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5deea0 0x55d16a5e4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad23fff640 1 --2- 172.21.15.28:0/2099072369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5e7a80 0x55d16a5e5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad23fff640 1 -- 172.21.15.28:0/2099072369 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5deea0 msgr2=0x55d16a5e4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad23fff640 1 --2- 172.21.15.28:0/2099072369 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5deea0 0x55d16a5e4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad23fff640 1 -- 172.21.15.28:0/2099072369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5e7a80 msgr2=0x55d16a5e5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad23fff640 1 --2- 172.21.15.28:0/2099072369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5e7a80 0x55d16a5e5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.136+0000 7fad23fff640 1 -- 172.21.15.28:0/2099072369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d16a2553f0 con 0x55d16a5de580 2024-04-21T09:00:49.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad23fff640 1 --2- 172.21.15.28:0/2099072369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5de580 0x55d16a5e4ab0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fad1c02c630 tx=0x7fad1c005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6d4277e0995ff8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:49.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad02ffd640 1 -- 172.21.15.28:0/2099072369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c049070 con 0x55d16a5de580 2024-04-21T09:00:49.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad02ffd640 1 -- 172.21.15.28:0/2099072369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7fad1c034440 con 0x55d16a5de580 2024-04-21T09:00:49.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad2cb86740 1 -- 172.21.15.28:0/2099072369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad1c032960 con 0x55d16a5de580 2024-04-21T09:00:49.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad02ffd640 1 -- 172.21.15.28:0/2099072369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fad1c039040 con 0x55d16a5de580 2024-04-21T09:00:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad2cb86740 10 client.18021 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad2cb86740 1 -- 172.21.15.28:0/2099072369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d16a2150f0 con 0x55d16a5de580 2024-04-21T09:00:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad02ffd640 1 -- 172.21.15.28:0/2099072369 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad1c032960 con 0x55d16a5de580 2024-04-21T09:00:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad02ffd640 1 -- 172.21.15.28:0/2099072369 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fad1c044410 con 0x55d16a5de580 2024-04-21T09:00:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad2cb86740 10 client.18021 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad2cb86740 10 client.18021 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad2cb86740 20 client.18021 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad2cb86740 1 --2- 172.21.15.28:0/2099072369 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d16a5f78d0 0x55d16a5f9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad2cb86740 4 client.18021 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:00:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.137+0000 7fad2cb86740 1 -- 172.21.15.28:0/2099072369 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d16a550c00 con 0x55d16a5f78d0 2024-04-21T09:00:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.138+0000 7fad237fe640 1 --2- 172.21.15.28:0/2099072369 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d16a5f78d0 0x55d16a5f9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.138+0000 7fad237fe640 1 --2- 172.21.15.28:0/2099072369 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d16a5f78d0 0x55d16a5f9cc0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e9f1cbec5b67a76 server_cookie=e634200937799bde in_seq=0 out_seq=0 2024-04-21T09:00:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.138+0000 7fad02ffd640 10 client.18021 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.139+0000 7fad02ffd640 1 -- 172.21.15.28:0/2099072369 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d16a550c00 con 0x55d16a5f78d0 2024-04-21T09:00:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.139+0000 7fad02ffd640 10 client.18021 handle_command_reply: tid=0 2024-04-21T09:00:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.139+0000 7fad2cb86740 1 client.18021 shutdown 2024-04-21T09:00:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.139+0000 7fad2cb86740 1 -- 172.21.15.28:0/2099072369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5de580 msgr2=0x55d16a5e4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.139+0000 7fad2cb86740 1 --2- 172.21.15.28:0/2099072369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5de580 0x55d16a5e4ab0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fad1c02c630 tx=0x7fad1c005f70 comp rx=0 tx=0).stop 2024-04-21T09:00:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.139+0000 7fad2cb86740 1 -- 172.21.15.28:0/2099072369 shutdown_connections 2024-04-21T09:00:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.139+0000 7fad2cb86740 1 --2- 172.21.15.28:0/2099072369 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d16a5f78d0 0x55d16a5f9cc0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.139+0000 7fad2cb86740 1 --2- 172.21.15.28:0/2099072369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5e7a80 0x55d16a5e5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.139+0000 7fad2cb86740 1 --2- 172.21.15.28:0/2099072369 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5deea0 0x55d16a5e4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.139+0000 7fad2cb86740 1 --2- 172.21.15.28:0/2099072369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5de580 0x55d16a5e4ab0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.139+0000 7fad02ffd640 0 client.18021 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.139+0000 7fad2cb86740 1 -- 172.21.15.28:0/2099072369 >> 172.21.15.28:0/2099072369 conn(0x55d16a5d8cb0 msgr2=0x55d16a5daa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.139+0000 7fad2cb86740 1 -- 172.21.15.28:0/2099072369 shutdown_connections 2024-04-21T09:00:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.139+0000 7fad2cb86740 1 -- 172.21.15.28:0/2099072369 wait complete. 2024-04-21T09:00:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.139+0000 7fad2cb86740 20 client.18021 trim_cache size 0 max 16384 2024-04-21T09:00:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad2cb86740 1 Processor -- start 2024-04-21T09:00:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad2cb86740 1 -- start start 2024-04-21T09:00:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad2cb86740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5de580 0x55d16a5e71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad2cb86740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5deea0 0x55d16a5df280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad2cb86740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5e7a80 0x55d16a5da810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad2cb86740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad241b3f10 con 0x55d16a5deea0 2024-04-21T09:00:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad2cb86740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fad241b17e0 con 0x55d16a5de580 2024-04-21T09:00:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad2cb86740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fad241af650 con 0x55d16a5e7a80 2024-04-21T09:00:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5deea0 0x55d16a5df280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5deea0 0x55d16a5df280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34138/0 (socket says 172.21.15.28:34138) 2024-04-21T09:00:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad237fe640 1 -- 172.21.15.28:0/925515672 learned_addr learned my addr 172.21.15.28:0/925515672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad23fff640 1 --2- 172.21.15.28:0/925515672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5de580 0x55d16a5e71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad237fe640 1 -- 172.21.15.28:0/925515672 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5e7a80 msgr2=0x55d16a5da810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad237fe640 1 --2- 172.21.15.28:0/925515672 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5e7a80 0x55d16a5da810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad237fe640 1 -- 172.21.15.28:0/925515672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5de580 msgr2=0x55d16a5e71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad237fe640 1 --2- 172.21.15.28:0/925515672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5de580 0x55d16a5e71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad237fe640 1 -- 172.21.15.28:0/925515672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d16a5e3a60 con 0x55d16a5deea0 2024-04-21T09:00:49.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.157+0000 7fad23fff640 1 --2- 172.21.15.28:0/925515672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5de580 0x55d16a5e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:49.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.158+0000 7fad237fe640 1 --2- 172.21.15.28:0/925515672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5deea0 0x55d16a5df280 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x55d16a5de050 tx=0x7fad14014900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1665d2fc946816 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.158+0000 7fad097fa640 1 -- 172.21.15.28:0/925515672 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14004030 con 0x55d16a5deea0 2024-04-21T09:00:49.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.158+0000 7fad097fa640 1 -- 172.21.15.28:0/925515672 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad140041d0 con 0x55d16a5deea0 2024-04-21T09:00:49.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.158+0000 7fad097fa640 1 -- 172.21.15.28:0/925515672 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1401e2b0 con 0x55d16a5deea0 2024-04-21T09:00:49.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.158+0000 7fad2cb86740 1 -- 172.21.15.28:0/925515672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5deea0 msgr2=0x55d16a5df280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.158+0000 7fad2cb86740 1 --2- 172.21.15.28:0/925515672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5deea0 0x55d16a5df280 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x55d16a5de050 tx=0x7fad14014900 comp rx=0 tx=0).stop 2024-04-21T09:00:49.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.158+0000 7fad2cb86740 1 -- 172.21.15.28:0/925515672 shutdown_connections 2024-04-21T09:00:49.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.158+0000 7fad2cb86740 1 --2- 172.21.15.28:0/925515672 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5e7a80 0x55d16a5da810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.158+0000 7fad2cb86740 1 --2- 172.21.15.28:0/925515672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5deea0 0x55d16a5df280 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.158+0000 7fad2cb86740 1 --2- 172.21.15.28:0/925515672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5de580 0x55d16a5e71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.158+0000 7fad2cb86740 1 -- 172.21.15.28:0/925515672 >> 172.21.15.28:0/925515672 conn(0x55d16a5d8cb0 msgr2=0x55d16a5da100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.158+0000 7fad2cb86740 1 -- 172.21.15.28:0/925515672 shutdown_connections 2024-04-21T09:00:49.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.158+0000 7fad2cb86740 1 -- 172.21.15.28:0/925515672 wait complete. 2024-04-21T09:00:49.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.159+0000 7fad2cb86740 1 -- start start 2024-04-21T09:00:49.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.159+0000 7fad2cb86740 1 Processor -- start 2024-04-21T09:00:49.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.159+0000 7fad2cb86740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5de580 0x55d16a5fa670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.159+0000 7fad2cb86740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5deea0 0x55d16a5fabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.159+0000 7fad2cb86740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5e7a80 0x55d16a5fb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.159+0000 7fad2cb86740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad241afca0 con 0x55d16a5e7a80 2024-04-21T09:00:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.159+0000 7fad2cb86740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fad241b1110 con 0x55d16a5de580 2024-04-21T09:00:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.159+0000 7fad2cb86740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fad241b0370 con 0x55d16a5deea0 2024-04-21T09:00:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.159+0000 7fad2cb86740 10 client.0 _ll_register_callbacks cb 0x7fad1c047070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.159+0000 7fad23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5e7a80 0x55d16a5fb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.159+0000 7fad23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5e7a80 0x55d16a5fb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34148/0 (socket says 172.21.15.28:34148) 2024-04-21T09:00:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.159+0000 7fad23fff640 1 -- 172.21.15.28:0/1535973738 learned_addr learned my addr 172.21.15.28:0/1535973738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.159+0000 7fad23fff640 1 --2- 172.21.15.28:0/1535973738 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5de580 0x55d16a5fa670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.159+0000 7fad23fff640 1 -- 172.21.15.28:0/1535973738 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5deea0 msgr2=0x55d16a5fabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.159+0000 7fad23fff640 1 --2- 172.21.15.28:0/1535973738 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5deea0 0x55d16a5fabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad23fff640 1 -- 172.21.15.28:0/1535973738 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5de580 msgr2=0x55d16a5fa670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad23fff640 1 --2- 172.21.15.28:0/1535973738 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5de580 0x55d16a5fa670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad23fff640 1 -- 172.21.15.28:0/1535973738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d16a65cab0 con 0x55d16a5e7a80 2024-04-21T09:00:49.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad23fff640 1 --2- 172.21.15.28:0/1535973738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5e7a80 0x55d16a5fb0f0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fad1c054040 tx=0x7fad1c009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18081b79986777ec server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:49.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad02ffd640 1 -- 172.21.15.28:0/1535973738 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c049070 con 0x55d16a5e7a80 2024-04-21T09:00:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad02ffd640 1 -- 172.21.15.28:0/1535973738 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7fad1c0326c0 con 0x55d16a5e7a80 2024-04-21T09:00:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad2cb86740 1 -- 172.21.15.28:0/1535973738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad140041d0 con 0x55d16a5e7a80 2024-04-21T09:00:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad02ffd640 1 -- 172.21.15.28:0/1535973738 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fad1c06a050 con 0x55d16a5e7a80 2024-04-21T09:00:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad2cb86740 10 client.18027 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad2cb86740 1 -- 172.21.15.28:0/1535973738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d16a65dd00 con 0x55d16a5e7a80 2024-04-21T09:00:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad02ffd640 1 -- 172.21.15.28:0/1535973738 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad140041d0 con 0x55d16a5e7a80 2024-04-21T09:00:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad02ffd640 1 -- 172.21.15.28:0/1535973738 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fad1c004030 con 0x55d16a5e7a80 2024-04-21T09:00:49.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad2cb86740 10 client.18027 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad2cb86740 10 client.18027 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad2cb86740 20 client.18027 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad2cb86740 1 --2- 172.21.15.28:0/1535973738 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d16a65e420 0x55d16a660840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad2cb86740 4 client.18027 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T09:00:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.160+0000 7fad2cb86740 1 -- 172.21.15.28:0/1535973738 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d16a54aa70 con 0x55d16a65e420 2024-04-21T09:00:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.161+0000 7fad237fe640 1 --2- 172.21.15.28:0/1535973738 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d16a65e420 0x55d16a660840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.161+0000 7fad237fe640 1 --2- 172.21.15.28:0/1535973738 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d16a65e420 0x55d16a660840 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=979878ec57f660e8 server_cookie=7007d7b565a7f620 in_seq=0 out_seq=0 2024-04-21T09:00:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.161+0000 7fad02ffd640 10 client.18027 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad02ffd640 1 -- 172.21.15.28:0/1535973738 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d16a54aa70 con 0x55d16a65e420 2024-04-21T09:00:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad02ffd640 10 client.18027 handle_command_reply: tid=0 2024-04-21T09:00:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad2cb86740 1 client.18027 shutdown 2024-04-21T09:00:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad2cb86740 1 -- 172.21.15.28:0/1535973738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5e7a80 msgr2=0x55d16a5fb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad2cb86740 1 --2- 172.21.15.28:0/1535973738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5e7a80 0x55d16a5fb0f0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fad1c054040 tx=0x7fad1c009800 comp rx=0 tx=0).stop 2024-04-21T09:00:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad2cb86740 1 -- 172.21.15.28:0/1535973738 shutdown_connections 2024-04-21T09:00:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad2cb86740 1 --2- 172.21.15.28:0/1535973738 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55d16a65e420 0x55d16a660840 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad2cb86740 1 --2- 172.21.15.28:0/1535973738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d16a5e7a80 0x55d16a5fb0f0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad2cb86740 1 --2- 172.21.15.28:0/1535973738 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55d16a5deea0 0x55d16a5fabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad2cb86740 1 --2- 172.21.15.28:0/1535973738 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d16a5de580 0x55d16a5fa670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad02ffd640 0 client.18027 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad2cb86740 1 -- 172.21.15.28:0/1535973738 >> 172.21.15.28:0/1535973738 conn(0x55d16a5d8cb0 msgr2=0x55d16a5d7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad2cb86740 1 -- 172.21.15.28:0/1535973738 shutdown_connections 2024-04-21T09:00:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad2cb86740 1 -- 172.21.15.28:0/1535973738 wait complete. 2024-04-21T09:00:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad2cb86740 20 client.18027 trim_cache size 0 max 16384 2024-04-21T09:00:49.172 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T09:00:49.173 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T09:00:49.173 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T09:00:49.173 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T09:00:49.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad29eba640 1 -- 172.21.15.28:0/1333641555 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fad0c0a3d20 msgr2=0x7fad0c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad29eba640 1 --2- 172.21.15.28:0/1333641555 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fad0c0a3d20 0x7fad0c0a61e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fad180cdd10 tx=0x7fad1806def0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad29eba640 1 -- 172.21.15.28:0/1333641555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2406af70 msgr2=0x7fad241a0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad29eba640 1 --2- 172.21.15.28:0/1333641555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2406af70 0x7fad241a0ad0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fad1800d950 tx=0x7fad1800de10 comp rx=0 tx=0).stop 2024-04-21T09:00:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad29eba640 1 -- 172.21.15.28:0/1333641555 shutdown_connections 2024-04-21T09:00:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad29eba640 1 --2- 172.21.15.28:0/1333641555 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fad0c0a3d20 0x7fad0c0a61e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad29eba640 1 --2- 172.21.15.28:0/1333641555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fad24106f20 0x7fad2419a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad29eba640 1 --2- 172.21.15.28:0/1333641555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad2406af70 0x7fad241a0ad0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad29eba640 1 --2- 172.21.15.28:0/1333641555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fad2406a5b0 0x7fad241a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.162+0000 7fad29eba640 1 -- 172.21.15.28:0/1333641555 >> 172.21.15.28:0/1333641555 conn(0x7fad24077c40 msgr2=0x7fad2410d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.163+0000 7fad29eba640 1 -- 172.21.15.28:0/1333641555 shutdown_connections 2024-04-21T09:00:49.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.163+0000 7fad29eba640 1 -- 172.21.15.28:0/1333641555 wait complete. 2024-04-21T09:00:49.176 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T09:00:49.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T09:00:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.279+0000 7f12be166640 1 -- 172.21.15.28:0/2374197793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12b81072c0 msgr2=0x7f12b810ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.279+0000 7f12be166640 1 --2- 172.21.15.28:0/2374197793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12b81072c0 0x7f12b810ee60 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009ca0 tx=0x7f12b002c0f0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.279+0000 7f12be166640 1 -- 172.21.15.28:0/2374197793 shutdown_connections 2024-04-21T09:00:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.279+0000 7f12be166640 1 --2- 172.21.15.28:0/2374197793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12b81072c0 0x7f12b810ee60 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.279+0000 7f12be166640 1 --2- 172.21.15.28:0/2374197793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f12b806b260 0x7f12b8106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.279+0000 7f12be166640 1 --2- 172.21.15.28:0/2374197793 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12b806a8b0 0x7f12b806ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.279+0000 7f12be166640 1 -- 172.21.15.28:0/2374197793 >> 172.21.15.28:0/2374197793 conn(0x7f12b8077e30 msgr2=0x7f12b8078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.279+0000 7f12be166640 1 -- 172.21.15.28:0/2374197793 shutdown_connections 2024-04-21T09:00:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.279+0000 7f12be166640 1 -- 172.21.15.28:0/2374197793 wait complete. 2024-04-21T09:00:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.280+0000 7f12be166640 1 Processor -- start 2024-04-21T09:00:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.280+0000 7f12be166640 1 -- start start 2024-04-21T09:00:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.280+0000 7f12be166640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f12b806a8b0 0x7f12b81011e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.280+0000 7f12be166640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12b81072c0 0x7f12b8101720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.280+0000 7f12be166640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12b8101c60 0x7f12b81a68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.280+0000 7f12be166640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12b8112200 con 0x7f12b8101c60 2024-04-21T09:00:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.280+0000 7f12be166640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f12b8112080 con 0x7f12b806a8b0 2024-04-21T09:00:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.280+0000 7f12be166640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f12b8112380 con 0x7f12b81072c0 2024-04-21T09:00:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.280+0000 7f12b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12b8101c60 0x7f12b81a68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.280+0000 7f12b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12b8101c60 0x7f12b81a68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34174/0 (socket says 172.21.15.28:34174) 2024-04-21T09:00:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.280+0000 7f12b7fff640 1 -- 172.21.15.28:0/1973552121 learned_addr learned my addr 172.21.15.28:0/1973552121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.280+0000 7f12b6ffd640 1 --2- 172.21.15.28:0/1973552121 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12b81072c0 0x7f12b8101720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.280+0000 7f12b77fe640 1 --2- 172.21.15.28:0/1973552121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f12b806a8b0 0x7f12b81011e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.281+0000 7f12b7fff640 1 -- 172.21.15.28:0/1973552121 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12b81072c0 msgr2=0x7f12b8101720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.281+0000 7f12b7fff640 1 --2- 172.21.15.28:0/1973552121 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12b81072c0 0x7f12b8101720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.281+0000 7f12b7fff640 1 -- 172.21.15.28:0/1973552121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f12b806a8b0 msgr2=0x7f12b81011e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.281+0000 7f12b7fff640 1 --2- 172.21.15.28:0/1973552121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f12b806a8b0 0x7f12b81011e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.281+0000 7f12b7fff640 1 -- 172.21.15.28:0/1973552121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b81a6fb0 con 0x7f12b8101c60 2024-04-21T09:00:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.281+0000 7f12b77fe640 1 --2- 172.21.15.28:0/1973552121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f12b806a8b0 0x7f12b81011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.281+0000 7f12b7fff640 1 --2- 172.21.15.28:0/1973552121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12b8101c60 0x7f12b81a68b0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f12b0005e30 tx=0x7f12b0034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.281+0000 7f12b4ff9640 1 -- 172.21.15.28:0/1973552121 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b0039040 con 0x7f12b8101c60 2024-04-21T09:00:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.281+0000 7f12b4ff9640 1 -- 172.21.15.28:0/1973552121 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12b0044070 con 0x7f12b8101c60 2024-04-21T09:00:49.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.281+0000 7f12be166640 1 -- 172.21.15.28:0/1973552121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b81a7240 con 0x7f12b8101c60 2024-04-21T09:00:49.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.281+0000 7f12b4ff9640 1 -- 172.21.15.28:0/1973552121 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b0003e60 con 0x7f12b8101c60 2024-04-21T09:00:49.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.281+0000 7f12be166640 1 -- 172.21.15.28:0/1973552121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b81a7730 con 0x7f12b8101c60 2024-04-21T09:00:49.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.282+0000 7f12be166640 1 -- 172.21.15.28:0/1973552121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12b8069230 con 0x7f12b8101c60 2024-04-21T09:00:49.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.283+0000 7f12b4ff9640 1 -- 172.21.15.28:0/1973552121 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f12b00324c0 con 0x7f12b8101c60 2024-04-21T09:00:49.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.284+0000 7f12b4ff9640 1 --2- 172.21.15.28:0/1973552121 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f12a00a3d50 0x7f12a00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.284+0000 7f12b77fe640 1 --2- 172.21.15.28:0/1973552121 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f12a00a3d50 0x7f12a00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.284+0000 7f12b4ff9640 1 -- 172.21.15.28:0/1973552121 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7f12b00f2de0 con 0x7f12b8101c60 2024-04-21T09:00:49.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.285+0000 7f12b77fe640 1 --2- 172.21.15.28:0/1973552121 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f12a00a3d50 0x7f12a00a6210 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f12a80058c0 tx=0x7f12a8005850 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.289+0000 7f12b4ff9640 1 -- 172.21.15.28:0/1973552121 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f12b00bf140 con 0x7f12b8101c60 2024-04-21T09:00:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.517+0000 7f12be166640 1 -- 172.21.15.28:0/1973552121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12b810a490 con 0x7f12b8101c60 2024-04-21T09:00:49.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.518+0000 7f12b4ff9640 1 -- 172.21.15.28:0/1973552121 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7f12b00bea40 con 0x7f12b8101c60 2024-04-21T09:00:49.518 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T09:00:49.518 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T09:00:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T09:00:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.520+0000 7f12be166640 1 -- 172.21.15.28:0/1973552121 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f12a00a3d50 msgr2=0x7f12a00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.520+0000 7f12be166640 1 --2- 172.21.15.28:0/1973552121 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f12a00a3d50 0x7f12a00a6210 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f12a80058c0 tx=0x7f12a8005850 comp rx=0 tx=0).stop 2024-04-21T09:00:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.520+0000 7f12be166640 1 -- 172.21.15.28:0/1973552121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12b8101c60 msgr2=0x7f12b81a68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.520+0000 7f12be166640 1 --2- 172.21.15.28:0/1973552121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12b8101c60 0x7f12b81a68b0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f12b0005e30 tx=0x7f12b0034760 comp rx=0 tx=0).stop 2024-04-21T09:00:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.520+0000 7f12be166640 1 -- 172.21.15.28:0/1973552121 shutdown_connections 2024-04-21T09:00:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.520+0000 7f12be166640 1 --2- 172.21.15.28:0/1973552121 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f12a00a3d50 0x7f12a00a6210 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.520+0000 7f12be166640 1 --2- 172.21.15.28:0/1973552121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12b8101c60 0x7f12b81a68b0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.520+0000 7f12be166640 1 --2- 172.21.15.28:0/1973552121 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12b81072c0 0x7f12b8101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.520+0000 7f12be166640 1 --2- 172.21.15.28:0/1973552121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f12b806a8b0 0x7f12b81011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.520+0000 7f12be166640 1 -- 172.21.15.28:0/1973552121 >> 172.21.15.28:0/1973552121 conn(0x7f12b8077e30 msgr2=0x7f12b80ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.521+0000 7f12be166640 1 -- 172.21.15.28:0/1973552121 shutdown_connections 2024-04-21T09:00:49.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.521+0000 7f12be166640 1 -- 172.21.15.28:0/1973552121 wait complete. 2024-04-21T09:00:49.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T09:00:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.635+0000 7ffa5fb96640 1 -- 172.21.15.28:0/716176516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5810c520 msgr2=0x7ffa5810e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.635+0000 7ffa5fb96640 1 --2- 172.21.15.28:0/716176516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5810c520 0x7ffa5810e9c0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009ca0 tx=0x7ffa5002c230 comp rx=0 tx=0).stop 2024-04-21T09:00:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.635+0000 7ffa5fb96640 1 -- 172.21.15.28:0/716176516 shutdown_connections 2024-04-21T09:00:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.635+0000 7ffa5fb96640 1 --2- 172.21.15.28:0/716176516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5810c520 0x7ffa5810e9c0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.635+0000 7ffa5fb96640 1 --2- 172.21.15.28:0/716176516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa58103b00 0x7ffa5810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.635+0000 7ffa5fb96640 1 --2- 172.21.15.28:0/716176516 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa581031e0 0x7ffa581035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.635+0000 7ffa5fb96640 1 -- 172.21.15.28:0/716176516 >> 172.21.15.28:0/716176516 conn(0x7ffa580771d0 msgr2=0x7ffa580775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.635+0000 7ffa5fb96640 1 -- 172.21.15.28:0/716176516 shutdown_connections 2024-04-21T09:00:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.635+0000 7ffa5fb96640 1 -- 172.21.15.28:0/716176516 wait complete. 2024-04-21T09:00:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.636+0000 7ffa5fb96640 1 Processor -- start 2024-04-21T09:00:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.636+0000 7ffa5fb96640 1 -- start start 2024-04-21T09:00:49.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.636+0000 7ffa5fb96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa581031e0 0x7ffa58100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.636+0000 7ffa5fb96640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa58103b00 0x7ffa58101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.636+0000 7ffa5fb96640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa5810c520 0x7ffa581018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.636+0000 7ffa5fb96640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa581121b0 con 0x7ffa581031e0 2024-04-21T09:00:49.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.636+0000 7ffa5fb96640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ffa58112030 con 0x7ffa5810c520 2024-04-21T09:00:49.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.636+0000 7ffa5fb96640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffa58112330 con 0x7ffa58103b00 2024-04-21T09:00:49.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.636+0000 7ffa5d932640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa581031e0 0x7ffa58100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.636+0000 7ffa5d932640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa581031e0 0x7ffa58100e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34206/0 (socket says 172.21.15.28:34206) 2024-04-21T09:00:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.636+0000 7ffa5d932640 1 -- 172.21.15.28:0/1689246150 learned_addr learned my addr 172.21.15.28:0/1689246150 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.636+0000 7ffa5e133640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa5810c520 0x7ffa581018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.637+0000 7ffa5d131640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa58103b00 0x7ffa58101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.637+0000 7ffa5d932640 1 -- 172.21.15.28:0/1689246150 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa58103b00 msgr2=0x7ffa58101370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.637+0000 7ffa5d932640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa58103b00 0x7ffa58101370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.637+0000 7ffa5d932640 1 -- 172.21.15.28:0/1689246150 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa5810c520 msgr2=0x7ffa581018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.637+0000 7ffa5d932640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa5810c520 0x7ffa581018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.637+0000 7ffa5d932640 1 -- 172.21.15.28:0/1689246150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa581a6bf0 con 0x7ffa581031e0 2024-04-21T09:00:49.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.637+0000 7ffa5d131640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa58103b00 0x7ffa58101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:00:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.637+0000 7ffa5e133640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa5810c520 0x7ffa581018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.637+0000 7ffa5d932640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa581031e0 0x7ffa58100e30 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400d770 tx=0x7ffa5400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.637+0000 7ffa4effd640 1 -- 172.21.15.28:0/1689246150 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa54017070 con 0x7ffa581031e0 2024-04-21T09:00:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.637+0000 7ffa4effd640 1 -- 172.21.15.28:0/1689246150 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa540054d0 con 0x7ffa581031e0 2024-04-21T09:00:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.637+0000 7ffa5fb96640 1 -- 172.21.15.28:0/1689246150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa581a6ee0 con 0x7ffa581031e0 2024-04-21T09:00:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.637+0000 7ffa4effd640 1 -- 172.21.15.28:0/1689246150 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa54004c30 con 0x7ffa581031e0 2024-04-21T09:00:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.637+0000 7ffa5fb96640 1 -- 172.21.15.28:0/1689246150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa581a73a0 con 0x7ffa581031e0 2024-04-21T09:00:49.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.640+0000 7ffa4effd640 1 -- 172.21.15.28:0/1689246150 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffa54004e10 con 0x7ffa581031e0 2024-04-21T09:00:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.640+0000 7ffa62785740 1 Processor -- start 2024-04-21T09:00:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.640+0000 7ffa62785740 1 -- start start 2024-04-21T09:00:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.640+0000 7ffa62785740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433db4d0 0x55f6433db8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.640+0000 7ffa62785740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433dbe80 0x55f6433e4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.640+0000 7ffa62785740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433e4a00 0x55f6433e6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.640+0000 7ffa62785740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6433401f0 con 0x55f6433dbe80 2024-04-21T09:00:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.640+0000 7ffa62785740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f6432dd4b0 con 0x55f6433e4a00 2024-04-21T09:00:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.640+0000 7ffa62785740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f64333c1b0 con 0x55f6433db4d0 2024-04-21T09:00:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.640+0000 7ffa5d932640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433dbe80 0x55f6433e4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.640+0000 7ffa5d932640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433dbe80 0x55f6433e4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34216/0 (socket says 172.21.15.28:34216) 2024-04-21T09:00:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.640+0000 7ffa5d932640 1 -- 172.21.15.28:0/1869581816 learned_addr learned my addr 172.21.15.28:0/1869581816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.641+0000 7ffa4effd640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ffa440a3da0 0x7ffa440a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.641+0000 7ffa5d131640 1 --2- 172.21.15.28:0/1869581816 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433e4a00 0x55f6433e6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.641+0000 7ffa5e133640 1 --2- 172.21.15.28:0/1869581816 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433db4d0 0x55f6433db8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.641+0000 7ffa5e133640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ffa440a3da0 0x7ffa440a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.641+0000 7ffa5d932640 1 -- 172.21.15.28:0/1869581816 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433db4d0 msgr2=0x55f6433db8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.641+0000 7ffa5d932640 1 --2- 172.21.15.28:0/1869581816 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433db4d0 0x55f6433db8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.641+0000 7ffa5d932640 1 -- 172.21.15.28:0/1869581816 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433e4a00 msgr2=0x55f6433e6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.641+0000 7ffa5d932640 1 --2- 172.21.15.28:0/1869581816 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433e4a00 0x55f6433e6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.641+0000 7ffa4effd640 1 -- 172.21.15.28:0/1689246150 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7ffa540d06c0 con 0x7ffa581031e0 2024-04-21T09:00:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.641+0000 7ffa5d932640 1 -- 172.21.15.28:0/1869581816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6432cf1b0 con 0x55f6433dbe80 2024-04-21T09:00:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.641+0000 7ffa5d932640 1 --2- 172.21.15.28:0/1869581816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433dbe80 0x55f6433e4430 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7ffa5409bf70 tx=0x7ffa54051430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557483d053232a06 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.641+0000 7ffa5e133640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ffa440a3da0 0x7ffa440a6260 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009430 tx=0x7ffa500095c0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.641+0000 7ffa43fff640 1 -- 172.21.15.28:0/1869581816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5409c970 con 0x55f6433dbe80 2024-04-21T09:00:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.641+0000 7ffa43fff640 1 -- 172.21.15.28:0/1869581816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa5409c270 con 0x55f6433dbe80 2024-04-21T09:00:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.641+0000 7ffa43fff640 1 -- 172.21.15.28:0/1869581816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa540cdca0 con 0x55f6433dbe80 2024-04-21T09:00:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.642+0000 7ffa62785740 1 -- 172.21.15.28:0/1869581816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433dbe80 msgr2=0x55f6433e4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.642+0000 7ffa62785740 1 --2- 172.21.15.28:0/1869581816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433dbe80 0x55f6433e4430 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7ffa5409bf70 tx=0x7ffa54051430 comp rx=0 tx=0).stop 2024-04-21T09:00:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.642+0000 7ffa62785740 1 -- 172.21.15.28:0/1869581816 shutdown_connections 2024-04-21T09:00:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.642+0000 7ffa62785740 1 --2- 172.21.15.28:0/1869581816 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433e4a00 0x55f6433e6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.642+0000 7ffa62785740 1 --2- 172.21.15.28:0/1869581816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433dbe80 0x55f6433e4430 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.642+0000 7ffa62785740 1 --2- 172.21.15.28:0/1869581816 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433db4d0 0x55f6433db8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.642+0000 7ffa62785740 1 -- 172.21.15.28:0/1869581816 >> 172.21.15.28:0/1869581816 conn(0x55f6433d5ba0 msgr2=0x55f6433d5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.642+0000 7ffa62785740 1 -- 172.21.15.28:0/1869581816 shutdown_connections 2024-04-21T09:00:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.642+0000 7ffa62785740 1 -- 172.21.15.28:0/1869581816 wait complete. 2024-04-21T09:00:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.642+0000 7ffa62785740 1 -- start start 2024-04-21T09:00:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.642+0000 7ffa62785740 1 Processor -- start 2024-04-21T09:00:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa62785740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 0x55f6433e4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa62785740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433dbe80 0x55f6433e1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa62785740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433e4a00 0x55f6433e1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa62785740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6433e7910 con 0x55f6433e4a00 2024-04-21T09:00:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa62785740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55f6432f9500 con 0x55f6433db4d0 2024-04-21T09:00:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa62785740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55f6433e7a90 con 0x55f6433dbe80 2024-04-21T09:00:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa62785740 10 client.0 _ll_register_callbacks cb 0x55f6432c7b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:49.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa5e133640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433e4a00 0x55f6433e1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa5e133640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433e4a00 0x55f6433e1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34220/0 (socket says 172.21.15.28:34220) 2024-04-21T09:00:49.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa5e133640 1 -- 172.21.15.28:0/2537980710 learned_addr learned my addr 172.21.15.28:0/2537980710 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa5d932640 1 --2- 172.21.15.28:0/2537980710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 0x55f6433e4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa5e133640 1 -- 172.21.15.28:0/2537980710 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433dbe80 msgr2=0x55f6433e1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa5e133640 1 --2- 172.21.15.28:0/2537980710 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433dbe80 0x55f6433e1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa5e133640 1 -- 172.21.15.28:0/2537980710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 msgr2=0x55f6433e4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa5e133640 1 --2- 172.21.15.28:0/2537980710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 0x55f6433e4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa5e133640 1 -- 172.21.15.28:0/2537980710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6430523f0 con 0x55f6433e4a00 2024-04-21T09:00:49.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.643+0000 7ffa5e133640 1 --2- 172.21.15.28:0/2537980710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433e4a00 0x55f6433e1bd0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7ffa50005fe0 tx=0x7ffa5002c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc5ebbaf0d702fb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.644+0000 7ffa417fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:49.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.644+0000 7ffa417fa640 1 -- 172.21.15.28:0/2537980710 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa50052060 con 0x55f6433e4a00 2024-04-21T09:00:49.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.644+0000 7ffa417fa640 1 -- 172.21.15.28:0/2537980710 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7ffa50042920 con 0x55f6433e4a00 2024-04-21T09:00:49.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.644+0000 7ffa62785740 1 -- 172.21.15.28:0/2537980710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa5409c270 con 0x55f6433e4a00 2024-04-21T09:00:49.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.644+0000 7ffa417fa640 1 -- 172.21.15.28:0/2537980710 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffa50059050 con 0x55f6433e4a00 2024-04-21T09:00:49.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.644+0000 7ffa62785740 10 client.18045 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:49.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.644+0000 7ffa62785740 1 -- 172.21.15.28:0/2537980710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f64311c7d0 con 0x55f6433e4a00 2024-04-21T09:00:49.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.644+0000 7ffa417fa640 1 -- 172.21.15.28:0/2537980710 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa5409c270 con 0x55f6433e4a00 2024-04-21T09:00:49.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.644+0000 7ffa417fa640 1 -- 172.21.15.28:0/2537980710 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7ffa5004d400 con 0x55f6433e4a00 2024-04-21T09:00:49.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.644+0000 7ffa62785740 10 client.18045 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:49.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.644+0000 7ffa62785740 10 client.18045 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T09:00:49.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.644+0000 7ffa62785740 20 client.18045 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:49.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.644+0000 7ffa62785740 1 --2- 172.21.15.28:0/2537980710 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f6433f4c10 0x55f6433f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.644+0000 7ffa62785740 4 client.18045 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:00:49.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.644+0000 7ffa62785740 1 -- 172.21.15.28:0/2537980710 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6432df930 con 0x55f6433f4c10 2024-04-21T09:00:49.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.645+0000 7ffa5d932640 1 --2- 172.21.15.28:0/2537980710 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f6433f4c10 0x55f6433f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.645+0000 7ffa5d932640 1 --2- 172.21.15.28:0/2537980710 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f6433f4c10 0x55f6433f70a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3e8ce30aace905a4 server_cookie=c90f7b1d0a7fdff in_seq=0 out_seq=0 2024-04-21T09:00:49.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.645+0000 7ffa417fa640 10 client.18045 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T09:00:49.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.646+0000 7ffa417fa640 1 -- 172.21.15.28:0/2537980710 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f6432df930 con 0x55f6433f4c10 2024-04-21T09:00:49.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.646+0000 7ffa417fa640 10 client.18045 handle_command_reply: tid=0 2024-04-21T09:00:49.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.646+0000 7ffa62785740 1 client.18045 shutdown 2024-04-21T09:00:49.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.646+0000 7ffa62785740 1 -- 172.21.15.28:0/2537980710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433e4a00 msgr2=0x55f6433e1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.646+0000 7ffa62785740 1 --2- 172.21.15.28:0/2537980710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433e4a00 0x55f6433e1bd0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7ffa50005fe0 tx=0x7ffa5002c230 comp rx=0 tx=0).stop 2024-04-21T09:00:49.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.646+0000 7ffa62785740 1 -- 172.21.15.28:0/2537980710 shutdown_connections 2024-04-21T09:00:49.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.646+0000 7ffa62785740 1 --2- 172.21.15.28:0/2537980710 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f6433f4c10 0x55f6433f70a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.646+0000 7ffa62785740 1 --2- 172.21.15.28:0/2537980710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433e4a00 0x55f6433e1bd0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.646+0000 7ffa62785740 1 --2- 172.21.15.28:0/2537980710 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433dbe80 0x55f6433e1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.646+0000 7ffa62785740 1 --2- 172.21.15.28:0/2537980710 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 0x55f6433e4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.646+0000 7ffa417fa640 0 client.18045 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T09:00:49.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.646+0000 7ffa62785740 1 -- 172.21.15.28:0/2537980710 >> 172.21.15.28:0/2537980710 conn(0x55f6433d5ba0 msgr2=0x55f6433d78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.646+0000 7ffa62785740 1 -- 172.21.15.28:0/2537980710 shutdown_connections 2024-04-21T09:00:49.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.646+0000 7ffa62785740 1 -- 172.21.15.28:0/2537980710 wait complete. 2024-04-21T09:00:49.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.646+0000 7ffa62785740 20 client.18045 trim_cache size 0 max 16384 2024-04-21T09:00:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.659+0000 7ffa62785740 1 Processor -- start 2024-04-21T09:00:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.659+0000 7ffa62785740 1 -- start start 2024-04-21T09:00:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa62785740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 0x55f6433f07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa62785740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433dbe80 0x55f6433e0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa62785740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433e4a00 0x55f6433fc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa62785740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa581b3f20 con 0x55f6433e4a00 2024-04-21T09:00:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa62785740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ffa581b17f0 con 0x55f6433db4d0 2024-04-21T09:00:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa62785740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffa581af660 con 0x55f6433dbe80 2024-04-21T09:00:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa5e133640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433e4a00 0x55f6433fc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa5e133640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433e4a00 0x55f6433fc970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34226/0 (socket says 172.21.15.28:34226) 2024-04-21T09:00:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa5e133640 1 -- 172.21.15.28:0/3324897322 learned_addr learned my addr 172.21.15.28:0/3324897322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa5d131640 1 --2- 172.21.15.28:0/3324897322 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433dbe80 0x55f6433e0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa5e133640 1 -- 172.21.15.28:0/3324897322 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433dbe80 msgr2=0x55f6433e0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa5e133640 1 --2- 172.21.15.28:0/3324897322 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433dbe80 0x55f6433e0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa5d932640 1 --2- 172.21.15.28:0/3324897322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 0x55f6433f07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa5e133640 1 -- 172.21.15.28:0/3324897322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 msgr2=0x55f6433f07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa5e133640 1 --2- 172.21.15.28:0/3324897322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 0x55f6433f07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa5e133640 1 -- 172.21.15.28:0/3324897322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6433dd270 con 0x55f6433e4a00 2024-04-21T09:00:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa5d932640 1 --2- 172.21.15.28:0/3324897322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 0x55f6433f07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:00:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa5e133640 1 --2- 172.21.15.28:0/3324897322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433e4a00 0x55f6433fc970 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7ffa5004d850 tx=0x7ffa50032460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d1976a6d14b32d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa437fe640 1 -- 172.21.15.28:0/3324897322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa50052040 con 0x55f6433e4a00 2024-04-21T09:00:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa437fe640 1 -- 172.21.15.28:0/3324897322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa50042720 con 0x55f6433e4a00 2024-04-21T09:00:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.660+0000 7ffa437fe640 1 -- 172.21.15.28:0/3324897322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa50039040 con 0x55f6433e4a00 2024-04-21T09:00:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.661+0000 7ffa62785740 1 -- 172.21.15.28:0/3324897322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433e4a00 msgr2=0x55f6433fc970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.661+0000 7ffa62785740 1 --2- 172.21.15.28:0/3324897322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433e4a00 0x55f6433fc970 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7ffa5004d850 tx=0x7ffa50032460 comp rx=0 tx=0).stop 2024-04-21T09:00:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.661+0000 7ffa62785740 1 -- 172.21.15.28:0/3324897322 shutdown_connections 2024-04-21T09:00:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.661+0000 7ffa62785740 1 --2- 172.21.15.28:0/3324897322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433e4a00 0x55f6433fc970 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.661+0000 7ffa62785740 1 --2- 172.21.15.28:0/3324897322 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433dbe80 0x55f6433e0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.661+0000 7ffa62785740 1 --2- 172.21.15.28:0/3324897322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 0x55f6433f07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.661+0000 7ffa62785740 1 -- 172.21.15.28:0/3324897322 >> 172.21.15.28:0/3324897322 conn(0x55f6433d5ba0 msgr2=0x55f6433d7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.661+0000 7ffa62785740 1 -- 172.21.15.28:0/3324897322 shutdown_connections 2024-04-21T09:00:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.661+0000 7ffa62785740 1 -- 172.21.15.28:0/3324897322 wait complete. 2024-04-21T09:00:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.661+0000 7ffa62785740 1 -- start start 2024-04-21T09:00:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.661+0000 7ffa62785740 1 Processor -- start 2024-04-21T09:00:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa62785740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 0x55f6433ecce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa62785740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433dbe80 0x55f6433ed220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa62785740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433e4a00 0x55f6433ddaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa62785740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa581afcb0 con 0x55f6433dbe80 2024-04-21T09:00:49.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa62785740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7ffa581b1120 con 0x55f6433db4d0 2024-04-21T09:00:49.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa62785740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7ffa581b0380 con 0x55f6433e4a00 2024-04-21T09:00:49.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa62785740 10 client.0 _ll_register_callbacks cb 0x7ffa50032970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:49.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa5d131640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433dbe80 0x55f6433ed220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa5d131640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433dbe80 0x55f6433ed220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34234/0 (socket says 172.21.15.28:34234) 2024-04-21T09:00:49.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa5d131640 1 -- 172.21.15.28:0/2122566779 learned_addr learned my addr 172.21.15.28:0/2122566779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa5d932640 1 --2- 172.21.15.28:0/2122566779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 0x55f6433ecce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa5e133640 1 --2- 172.21.15.28:0/2122566779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433e4a00 0x55f6433ddaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa5d131640 1 -- 172.21.15.28:0/2122566779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433e4a00 msgr2=0x55f6433ddaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa5d131640 1 --2- 172.21.15.28:0/2122566779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433e4a00 0x55f6433ddaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa5d131640 1 -- 172.21.15.28:0/2122566779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 msgr2=0x55f6433ecce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa5d131640 1 --2- 172.21.15.28:0/2122566779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 0x55f6433ecce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa5d131640 1 -- 172.21.15.28:0/2122566779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6433de2a0 con 0x55f6433dbe80 2024-04-21T09:00:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa5d131640 1 --2- 172.21.15.28:0/2122566779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433dbe80 0x55f6433ed220 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009df0 tx=0x7ffa48013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5de83014f5f1f36 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa417fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa417fa640 1 -- 172.21.15.28:0/2122566779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa48011880 con 0x55f6433dbe80 2024-04-21T09:00:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa62785740 1 -- 172.21.15.28:0/2122566779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa50042720 con 0x55f6433dbe80 2024-04-21T09:00:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.662+0000 7ffa417fa640 1 -- 172.21.15.28:0/2122566779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7ffa48011a20 con 0x55f6433dbe80 2024-04-21T09:00:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.663+0000 7ffa417fa640 1 -- 172.21.15.28:0/2122566779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffa48011c20 con 0x55f6433dbe80 2024-04-21T09:00:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.663+0000 7ffa62785740 10 client.18051 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.663+0000 7ffa62785740 1 -- 172.21.15.28:0/2122566779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f64345ac60 con 0x55f6433dbe80 2024-04-21T09:00:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.663+0000 7ffa417fa640 1 -- 172.21.15.28:0/2122566779 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa50042720 con 0x55f6433dbe80 2024-04-21T09:00:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.663+0000 7ffa417fa640 1 -- 172.21.15.28:0/2122566779 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7ffa4801a070 con 0x55f6433dbe80 2024-04-21T09:00:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.663+0000 7ffa62785740 10 client.18051 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.663+0000 7ffa62785740 10 client.18051 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T09:00:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.663+0000 7ffa62785740 20 client.18051 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.663+0000 7ffa62785740 1 --2- 172.21.15.28:0/2122566779 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f64345b490 0x55f64345d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.663+0000 7ffa62785740 4 client.18051 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T09:00:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.663+0000 7ffa62785740 1 -- 172.21.15.28:0/2122566779 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f64334dc10 con 0x55f64345b490 2024-04-21T09:00:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.663+0000 7ffa5d932640 1 --2- 172.21.15.28:0/2122566779 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f64345b490 0x55f64345d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa5d932640 1 --2- 172.21.15.28:0/2122566779 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f64345b490 0x55f64345d8b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=aacff154489c7aae server_cookie=9766264a0921563b in_seq=0 out_seq=0 2024-04-21T09:00:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa417fa640 10 client.18051 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T09:00:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa417fa640 1 -- 172.21.15.28:0/2122566779 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f64334dc10 con 0x55f64345b490 2024-04-21T09:00:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa417fa640 10 client.18051 handle_command_reply: tid=0 2024-04-21T09:00:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa62785740 1 client.18051 shutdown 2024-04-21T09:00:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa62785740 1 -- 172.21.15.28:0/2122566779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433dbe80 msgr2=0x55f6433ed220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa62785740 1 --2- 172.21.15.28:0/2122566779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433dbe80 0x55f6433ed220 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009df0 tx=0x7ffa48013830 comp rx=0 tx=0).stop 2024-04-21T09:00:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa62785740 1 -- 172.21.15.28:0/2122566779 shutdown_connections 2024-04-21T09:00:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa62785740 1 --2- 172.21.15.28:0/2122566779 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55f64345b490 0x55f64345d8b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa62785740 1 --2- 172.21.15.28:0/2122566779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55f6433e4a00 0x55f6433ddaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa62785740 1 --2- 172.21.15.28:0/2122566779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6433dbe80 0x55f6433ed220 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa62785740 1 --2- 172.21.15.28:0/2122566779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f6433db4d0 0x55f6433ecce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa417fa640 0 client.18051 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T09:00:49.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa62785740 1 -- 172.21.15.28:0/2122566779 >> 172.21.15.28:0/2122566779 conn(0x55f6433d5ba0 msgr2=0x55f6433d6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa62785740 1 -- 172.21.15.28:0/2122566779 shutdown_connections 2024-04-21T09:00:49.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa62785740 1 -- 172.21.15.28:0/2122566779 wait complete. 2024-04-21T09:00:49.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.664+0000 7ffa62785740 20 client.18051 trim_cache size 0 max 16384 2024-04-21T09:00:49.674 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T09:00:49.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.665+0000 7ffa5fb96640 1 -- 172.21.15.28:0/1689246150 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ffa440a3da0 msgr2=0x7ffa440a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.665+0000 7ffa5fb96640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ffa440a3da0 0x7ffa440a6260 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009430 tx=0x7ffa500095c0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.665+0000 7ffa5fb96640 1 -- 172.21.15.28:0/1689246150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa581031e0 msgr2=0x7ffa58100e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.665+0000 7ffa5fb96640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa581031e0 0x7ffa58100e30 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400d770 tx=0x7ffa5400dc30 comp rx=0 tx=0).stop 2024-04-21T09:00:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.665+0000 7ffa5fb96640 1 -- 172.21.15.28:0/1689246150 shutdown_connections 2024-04-21T09:00:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.665+0000 7ffa5fb96640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7ffa440a3da0 0x7ffa440a6260 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.665+0000 7ffa5fb96640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa5810c520 0x7ffa581018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.665+0000 7ffa5fb96640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa58103b00 0x7ffa58101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.665+0000 7ffa5fb96640 1 --2- 172.21.15.28:0/1689246150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa581031e0 0x7ffa58100e30 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.665+0000 7ffa5fb96640 1 -- 172.21.15.28:0/1689246150 >> 172.21.15.28:0/1689246150 conn(0x7ffa580771d0 msgr2=0x7ffa580fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.665+0000 7ffa5fb96640 1 -- 172.21.15.28:0/1689246150 shutdown_connections 2024-04-21T09:00:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.665+0000 7ffa5fb96640 1 -- 172.21.15.28:0/1689246150 wait complete. 2024-04-21T09:00:49.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T09:00:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.782+0000 7f9b9eec1640 1 -- 172.21.15.28:0/1365617860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98102b40 msgr2=0x7f9b98106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.782+0000 7f9b9eec1640 1 --2- 172.21.15.28:0/1365617860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98102b40 0x7f9b98106b80 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009900 tx=0x7f9b8802bc30 comp rx=0 tx=0).stop 2024-04-21T09:00:49.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.782+0000 7f9b9eec1640 1 -- 172.21.15.28:0/1365617860 shutdown_connections 2024-04-21T09:00:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.782+0000 7f9b9eec1640 1 --2- 172.21.15.28:0/1365617860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9b98107260 0x7f9b9810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.782+0000 7f9b9eec1640 1 --2- 172.21.15.28:0/1365617860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98102b40 0x7f9b98106b80 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.782+0000 7f9b9eec1640 1 --2- 172.21.15.28:0/1365617860 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b98102190 0x7f9b98102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.782+0000 7f9b9eec1640 1 -- 172.21.15.28:0/1365617860 >> 172.21.15.28:0/1365617860 conn(0x7f9b980793a0 msgr2=0x7f9b980797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.782+0000 7f9b9eec1640 1 -- 172.21.15.28:0/1365617860 shutdown_connections 2024-04-21T09:00:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.782+0000 7f9b9eec1640 1 -- 172.21.15.28:0/1365617860 wait complete. 2024-04-21T09:00:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9eec1640 1 Processor -- start 2024-04-21T09:00:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9eec1640 1 -- start start 2024-04-21T09:00:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9eec1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9b98102190 0x7f9b9819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9eec1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b98107260 0x7f9b9819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9eec1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98198700 0x7f9b98198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9eec1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b98112280 con 0x7f9b98198700 2024-04-21T09:00:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9eec1640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9b98112100 con 0x7f9b98102190 2024-04-21T09:00:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9eec1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9b98112400 con 0x7f9b98107260 2024-04-21T09:00:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9d45e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98198700 0x7f9b98198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9d45e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98198700 0x7f9b98198b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34260/0 (socket says 172.21.15.28:34260) 2024-04-21T09:00:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9d45e640 1 -- 172.21.15.28:0/3938003584 learned_addr learned my addr 172.21.15.28:0/3938003584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9d45e640 1 -- 172.21.15.28:0/3938003584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b98107260 msgr2=0x7f9b9819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b8ffff640 1 --2- 172.21.15.28:0/3938003584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b98107260 0x7f9b9819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9cc5d640 1 --2- 172.21.15.28:0/3938003584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9b98102190 0x7f9b9819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9d45e640 1 --2- 172.21.15.28:0/3938003584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b98107260 0x7f9b9819eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9d45e640 1 -- 172.21.15.28:0/3938003584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9b98102190 msgr2=0x7f9b9819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9d45e640 1 --2- 172.21.15.28:0/3938003584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9b98102190 0x7f9b9819e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.783+0000 7f9b9d45e640 1 -- 172.21.15.28:0/3938003584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b98199300 con 0x7f9b98198700 2024-04-21T09:00:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.784+0000 7f9b9d45e640 1 --2- 172.21.15.28:0/3938003584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98198700 0x7f9b98198b80 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000b970 tx=0x7f9b9000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.784+0000 7f9b8dffb640 1 -- 172.21.15.28:0/3938003584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b90019070 con 0x7f9b98198700 2024-04-21T09:00:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.784+0000 7f9b8dffb640 1 -- 172.21.15.28:0/3938003584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b90014440 con 0x7f9b98198700 2024-04-21T09:00:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.784+0000 7f9b9eec1640 1 -- 172.21.15.28:0/3938003584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b981a53e0 con 0x7f9b98198700 2024-04-21T09:00:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.784+0000 7f9b8dffb640 1 -- 172.21.15.28:0/3938003584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b900075d0 con 0x7f9b98198700 2024-04-21T09:00:49.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.784+0000 7f9b9eec1640 1 -- 172.21.15.28:0/3938003584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b981a5790 con 0x7f9b98198700 2024-04-21T09:00:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.787+0000 7f9b8dffb640 1 -- 172.21.15.28:0/3938003584 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9b9001c070 con 0x7f9b98198700 2024-04-21T09:00:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.787+0000 7f9ba1b94740 1 Processor -- start 2024-04-21T09:00:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.787+0000 7f9ba1b94740 1 -- start start 2024-04-21T09:00:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.787+0000 7f9ba1b94740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08c250 0x562b0d08c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.787+0000 7f9ba1b94740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 0x562b0d099840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.787+0000 7f9ba1b94740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d099e10 0x562b0d09c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.787+0000 7f9ba1b94740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562b0cff11f0 con 0x562b0d08c250 2024-04-21T09:00:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.787+0000 7f9ba1b94740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562b0cf8e4b0 con 0x562b0d099e10 2024-04-21T09:00:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.787+0000 7f9ba1b94740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x562b0cfed1b0 con 0x562b0d08cc00 2024-04-21T09:00:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.787+0000 7f9b8dffb640 1 --2- 172.21.15.28:0/3938003584 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9b800a3d20 0x7f9b800a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.787+0000 7f9b8ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08c250 0x562b0d08c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.787+0000 7f9b8ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08c250 0x562b0d08c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34262/0 (socket says 172.21.15.28:34262) 2024-04-21T09:00:49.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.787+0000 7f9b8ffff640 1 -- 172.21.15.28:0/3533450410 learned_addr learned my addr 172.21.15.28:0/3533450410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.788+0000 7f9b9d45e640 1 --2- 172.21.15.28:0/3938003584 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9b800a3d20 0x7f9b800a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.788+0000 7f9b8dffb640 1 -- 172.21.15.28:0/3938003584 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7f9b900d0140 con 0x7f9b98198700 2024-04-21T09:00:49.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.788+0000 7f9b9cc5d640 1 --2- 172.21.15.28:0/3533450410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 0x562b0d099840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.788+0000 7f9b8ffff640 1 --2- 172.21.15.28:0/3533450410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d099e10 0x562b0d09c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.788+0000 7f9b8ffff640 1 -- 172.21.15.28:0/3533450410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 msgr2=0x562b0d099840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.788+0000 7f9b8ffff640 1 --2- 172.21.15.28:0/3533450410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 0x562b0d099840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.788+0000 7f9b8ffff640 1 -- 172.21.15.28:0/3533450410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d099e10 msgr2=0x562b0d09c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.788+0000 7f9b8ffff640 1 --2- 172.21.15.28:0/3533450410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d099e10 0x562b0d09c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.788+0000 7f9b8ffff640 1 -- 172.21.15.28:0/3533450410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b0cf801b0 con 0x562b0d08c250 2024-04-21T09:00:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.788+0000 7f9b8ffff640 1 --2- 172.21.15.28:0/3533450410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08c250 0x562b0d08c630 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f9b88005940 tx=0x7f9b88005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44eab339858f8869 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.788+0000 7f9b9d45e640 1 --2- 172.21.15.28:0/3938003584 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9b800a3d20 0x7f9b800a61e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9b900cd660 tx=0x7f9b90091290 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.788+0000 7f9b7effd640 1 -- 172.21.15.28:0/3533450410 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b88004450 con 0x562b0d08c250 2024-04-21T09:00:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.788+0000 7f9b7effd640 1 -- 172.21.15.28:0/3533450410 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b8802cc90 con 0x562b0d08c250 2024-04-21T09:00:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.788+0000 7f9b7effd640 1 -- 172.21.15.28:0/3533450410 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b88004930 con 0x562b0d08c250 2024-04-21T09:00:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.789+0000 7f9ba1b94740 1 -- 172.21.15.28:0/3533450410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08c250 msgr2=0x562b0d08c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.789+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/3533450410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08c250 0x562b0d08c630 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f9b88005940 tx=0x7f9b88005a20 comp rx=0 tx=0).stop 2024-04-21T09:00:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.789+0000 7f9ba1b94740 1 -- 172.21.15.28:0/3533450410 shutdown_connections 2024-04-21T09:00:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.789+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/3533450410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d099e10 0x562b0d09c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.789+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/3533450410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 0x562b0d099840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.789+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/3533450410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08c250 0x562b0d08c630 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.789+0000 7f9ba1b94740 1 -- 172.21.15.28:0/3533450410 >> 172.21.15.28:0/3533450410 conn(0x562b0d086ba0 msgr2=0x562b0d086f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.789+0000 7f9ba1b94740 1 -- 172.21.15.28:0/3533450410 shutdown_connections 2024-04-21T09:00:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.789+0000 7f9ba1b94740 1 -- 172.21.15.28:0/3533450410 wait complete. 2024-04-21T09:00:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.789+0000 7f9ba1b94740 1 -- start start 2024-04-21T09:00:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.789+0000 7f9ba1b94740 1 Processor -- start 2024-04-21T09:00:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9ba1b94740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08c250 0x562b0d099760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9ba1b94740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 0x562b0d096af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9ba1b94740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d099e10 0x562b0d097100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9ba1b94740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562b0d09cce0 con 0x562b0d08c250 2024-04-21T09:00:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9ba1b94740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x562b0cfaa500 con 0x562b0d099e10 2024-04-21T09:00:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9ba1b94740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x562b0d09ce60 con 0x562b0d08cc00 2024-04-21T09:00:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9ba1b94740 10 client.0 _ll_register_callbacks cb 0x562b0cf78b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9b8ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08c250 0x562b0d099760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9b8ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08c250 0x562b0d099760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34276/0 (socket says 172.21.15.28:34276) 2024-04-21T09:00:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9b8ffff640 1 -- 172.21.15.28:0/2617750359 learned_addr learned my addr 172.21.15.28:0/2617750359 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9b9cc5d640 1 --2- 172.21.15.28:0/2617750359 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 0x562b0d096af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9b8ffff640 1 -- 172.21.15.28:0/2617750359 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 msgr2=0x562b0d096af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9b8ffff640 1 --2- 172.21.15.28:0/2617750359 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 0x562b0d096af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9b8ffff640 1 -- 172.21.15.28:0/2617750359 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d099e10 msgr2=0x562b0d097100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9b8ffff640 1 --2- 172.21.15.28:0/2617750359 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d099e10 0x562b0d097100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9b8ffff640 1 -- 172.21.15.28:0/2617750359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b0cd033f0 con 0x562b0d08c250 2024-04-21T09:00:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9b9cc5d640 1 --2- 172.21.15.28:0/2617750359 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 0x562b0d096af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9b8ffff640 1 --2- 172.21.15.28:0/2617750359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08c250 0x562b0d099760 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f9b88044040 tx=0x7f9b88009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3878b97b99d08d6c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9b77fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9b77fff640 1 -- 172.21.15.28:0/2617750359 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b880036b0 con 0x562b0d08c250 2024-04-21T09:00:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9b77fff640 1 -- 172.21.15.28:0/2617750359 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7f9b88003850 con 0x562b0d08c250 2024-04-21T09:00:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.790+0000 7f9ba1b94740 1 -- 172.21.15.28:0/2617750359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b8802cc90 con 0x562b0d08c250 2024-04-21T09:00:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.791+0000 7f9b77fff640 1 -- 172.21.15.28:0/2617750359 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9b8805b050 con 0x562b0d08c250 2024-04-21T09:00:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.791+0000 7f9ba1b94740 10 client.18063 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.791+0000 7f9ba1b94740 1 -- 172.21.15.28:0/2617750359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562b0cdcd7d0 con 0x562b0d08c250 2024-04-21T09:00:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.791+0000 7f9b77fff640 1 -- 172.21.15.28:0/2617750359 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b8802cc90 con 0x562b0d08c250 2024-04-21T09:00:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.791+0000 7f9b77fff640 1 -- 172.21.15.28:0/2617750359 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f9b88034a10 con 0x562b0d08c250 2024-04-21T09:00:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.791+0000 7f9ba1b94740 10 client.18063 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.791+0000 7f9ba1b94740 10 client.18063 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T09:00:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.791+0000 7f9ba1b94740 20 client.18063 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:49.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.791+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/2617750359 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x562b0d0a9ec0 0x562b0d0ac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.791+0000 7f9ba1b94740 4 client.18063 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:00:49.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.791+0000 7f9ba1b94740 1 -- 172.21.15.28:0/2617750359 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b0cf90930 con 0x562b0d0a9ec0 2024-04-21T09:00:49.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.791+0000 7f9b9cc5d640 1 --2- 172.21.15.28:0/2617750359 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x562b0d0a9ec0 0x562b0d0ac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.792+0000 7f9b9cc5d640 1 --2- 172.21.15.28:0/2617750359 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x562b0d0a9ec0 0x562b0d0ac2b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=88f992704e15fc6b server_cookie=3552d58ccd399bc1 in_seq=0 out_seq=0 2024-04-21T09:00:49.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.792+0000 7f9b77fff640 10 client.18063 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T09:00:49.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.793+0000 7f9b77fff640 1 -- 172.21.15.28:0/2617750359 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562b0cf90930 con 0x562b0d0a9ec0 2024-04-21T09:00:49.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.793+0000 7f9b77fff640 10 client.18063 handle_command_reply: tid=0 2024-04-21T09:00:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.793+0000 7f9ba1b94740 1 client.18063 shutdown 2024-04-21T09:00:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.793+0000 7f9ba1b94740 1 -- 172.21.15.28:0/2617750359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08c250 msgr2=0x562b0d099760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.793+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/2617750359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08c250 0x562b0d099760 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f9b88044040 tx=0x7f9b88009600 comp rx=0 tx=0).stop 2024-04-21T09:00:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.793+0000 7f9ba1b94740 1 -- 172.21.15.28:0/2617750359 shutdown_connections 2024-04-21T09:00:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.793+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/2617750359 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d099e10 0x562b0d097100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.793+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/2617750359 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x562b0d0a9ec0 0x562b0d0ac2b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.793+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/2617750359 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 0x562b0d096af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.793+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/2617750359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08c250 0x562b0d099760 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.793+0000 7f9b77fff640 0 client.18063 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T09:00:49.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.793+0000 7f9ba1b94740 1 -- 172.21.15.28:0/2617750359 >> 172.21.15.28:0/2617750359 conn(0x562b0d086ba0 msgr2=0x562b0d0888f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.793+0000 7f9ba1b94740 1 -- 172.21.15.28:0/2617750359 shutdown_connections 2024-04-21T09:00:49.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.793+0000 7f9ba1b94740 1 -- 172.21.15.28:0/2617750359 wait complete. 2024-04-21T09:00:49.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.793+0000 7f9ba1b94740 20 client.18063 trim_cache size 0 max 16384 2024-04-21T09:00:49.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.806+0000 7f9ba1b94740 1 Processor -- start 2024-04-21T09:00:49.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.806+0000 7f9ba1b94740 1 -- start start 2024-04-21T09:00:49.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.806+0000 7f9ba1b94740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08c250 0x562b0d088810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.806+0000 7f9ba1b94740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08cc00 0x562b0d099220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.806+0000 7f9ba1b94740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d099e10 0x562b0d099760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.806+0000 7f9ba1b94740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b981afb40 con 0x562b0d08cc00 2024-04-21T09:00:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.806+0000 7f9ba1b94740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9b981ad410 con 0x562b0d099e10 2024-04-21T09:00:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.806+0000 7f9ba1b94740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9b981ab280 con 0x562b0d08c250 2024-04-21T09:00:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.807+0000 7f9b9cc5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08cc00 0x562b0d099220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.807+0000 7f9b9cc5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08cc00 0x562b0d099220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34284/0 (socket says 172.21.15.28:34284) 2024-04-21T09:00:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.807+0000 7f9b9cc5d640 1 -- 172.21.15.28:0/2515869294 learned_addr learned my addr 172.21.15.28:0/2515869294 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.807+0000 7f9b8ffff640 1 --2- 172.21.15.28:0/2515869294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d099e10 0x562b0d099760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.807+0000 7f9b8ffff640 1 --2- 172.21.15.28:0/2515869294 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08c250 0x562b0d088810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.807+0000 7f9b9cc5d640 1 -- 172.21.15.28:0/2515869294 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08c250 msgr2=0x562b0d088810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.807+0000 7f9b9cc5d640 1 --2- 172.21.15.28:0/2515869294 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08c250 0x562b0d088810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.807+0000 7f9b9cc5d640 1 -- 172.21.15.28:0/2515869294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d099e10 msgr2=0x562b0d099760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.807+0000 7f9b9cc5d640 1 --2- 172.21.15.28:0/2515869294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d099e10 0x562b0d099760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.807+0000 7f9b9cc5d640 1 -- 172.21.15.28:0/2515869294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b0d097470 con 0x562b0d08cc00 2024-04-21T09:00:49.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.807+0000 7f9b8ffff640 1 --2- 172.21.15.28:0/2515869294 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08c250 0x562b0d088810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:00:49.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.807+0000 7f9b9cc5d640 1 --2- 172.21.15.28:0/2515869294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08cc00 0x562b0d099220 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400aae0 tx=0x7f9b94008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5913a5adcd775044 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.807+0000 7f9b7e7fc640 1 -- 172.21.15.28:0/2515869294 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b940168e0 con 0x562b0d08cc00 2024-04-21T09:00:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.807+0000 7f9b7e7fc640 1 -- 172.21.15.28:0/2515869294 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b94020030 con 0x562b0d08cc00 2024-04-21T09:00:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.807+0000 7f9b7e7fc640 1 -- 172.21.15.28:0/2515869294 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b940243d0 con 0x562b0d08cc00 2024-04-21T09:00:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.808+0000 7f9ba1b94740 1 -- 172.21.15.28:0/2515869294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08cc00 msgr2=0x562b0d099220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.808+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/2515869294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08cc00 0x562b0d099220 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400aae0 tx=0x7f9b94008040 comp rx=0 tx=0).stop 2024-04-21T09:00:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.808+0000 7f9ba1b94740 1 -- 172.21.15.28:0/2515869294 shutdown_connections 2024-04-21T09:00:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.808+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/2515869294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d099e10 0x562b0d099760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.808+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/2515869294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d08cc00 0x562b0d099220 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.808+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/2515869294 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08c250 0x562b0d088810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.808+0000 7f9ba1b94740 1 -- 172.21.15.28:0/2515869294 >> 172.21.15.28:0/2515869294 conn(0x562b0d086ba0 msgr2=0x562b0d0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.808+0000 7f9ba1b94740 1 -- 172.21.15.28:0/2515869294 shutdown_connections 2024-04-21T09:00:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.808+0000 7f9ba1b94740 1 -- 172.21.15.28:0/2515869294 wait complete. 2024-04-21T09:00:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.808+0000 7f9ba1b94740 1 -- start start 2024-04-21T09:00:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.808+0000 7f9ba1b94740 1 Processor -- start 2024-04-21T09:00:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9ba1b94740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d08c250 0x562b0d0af740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9ba1b94740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 0x562b0d0acad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9ba1b94740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d099e10 0x562b0d0ad050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9ba1b94740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b981ab8d0 con 0x562b0d099e10 2024-04-21T09:00:49.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9ba1b94740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9b981acd40 con 0x562b0d08c250 2024-04-21T09:00:49.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9ba1b94740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9b981abfa0 con 0x562b0d08cc00 2024-04-21T09:00:49.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9ba1b94740 10 client.0 _ll_register_callbacks cb 0x7f9b88004200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:49.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9b8ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d099e10 0x562b0d0ad050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9b8ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d099e10 0x562b0d0ad050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34294/0 (socket says 172.21.15.28:34294) 2024-04-21T09:00:49.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9b8ffff640 1 -- 172.21.15.28:0/4208700402 learned_addr learned my addr 172.21.15.28:0/4208700402 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9b9cc5d640 1 --2- 172.21.15.28:0/4208700402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 0x562b0d0acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9b8ffff640 1 --2- 172.21.15.28:0/4208700402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d08c250 0x562b0d0af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9b8ffff640 1 -- 172.21.15.28:0/4208700402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 msgr2=0x562b0d0acad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9b8ffff640 1 --2- 172.21.15.28:0/4208700402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 0x562b0d0acad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9b8ffff640 1 -- 172.21.15.28:0/4208700402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d08c250 msgr2=0x562b0d0af740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9b8ffff640 1 --2- 172.21.15.28:0/4208700402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d08c250 0x562b0d0af740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9b8ffff640 1 -- 172.21.15.28:0/4208700402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b0d0ad880 con 0x562b0d099e10 2024-04-21T09:00:49.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9b8ffff640 1 --2- 172.21.15.28:0/4208700402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d099e10 0x562b0d0ad050 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f9b88006010 tx=0x7f9b88009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef8d5cfe9576e4ec server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9b77fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:49.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9b77fff640 1 -- 172.21.15.28:0/4208700402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b88063070 con 0x562b0d099e10 2024-04-21T09:00:49.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9ba1b94740 1 -- 172.21.15.28:0/4208700402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b94020030 con 0x562b0d099e10 2024-04-21T09:00:49.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.809+0000 7f9b77fff640 1 -- 172.21.15.28:0/4208700402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7f9b8804d520 con 0x562b0d099e10 2024-04-21T09:00:49.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.810+0000 7f9b77fff640 1 -- 172.21.15.28:0/4208700402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9b8806f050 con 0x562b0d099e10 2024-04-21T09:00:49.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.810+0000 7f9ba1b94740 10 client.18069 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:49.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.810+0000 7f9ba1b94740 1 -- 172.21.15.28:0/4208700402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562b0d1100b0 con 0x562b0d099e10 2024-04-21T09:00:49.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.810+0000 7f9b77fff640 1 -- 172.21.15.28:0/4208700402 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b94020030 con 0x562b0d099e10 2024-04-21T09:00:49.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.810+0000 7f9b77fff640 1 -- 172.21.15.28:0/4208700402 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f9b88005360 con 0x562b0d099e10 2024-04-21T09:00:49.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.810+0000 7f9ba1b94740 10 client.18069 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:49.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.810+0000 7f9ba1b94740 10 client.18069 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T09:00:49.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.810+0000 7f9ba1b94740 20 client.18069 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:49.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.810+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/4208700402 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x562b0d110750 0x562b0d112b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.810+0000 7f9ba1b94740 4 client.18069 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T09:00:49.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.810+0000 7f9ba1b94740 1 -- 172.21.15.28:0/4208700402 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562b0cffec10 con 0x562b0d110750 2024-04-21T09:00:49.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.810+0000 7f9b9cc5d640 1 --2- 172.21.15.28:0/4208700402 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x562b0d110750 0x562b0d112b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9b77fff640 10 client.18069 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T09:00:49.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9b9cc5d640 1 --2- 172.21.15.28:0/4208700402 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x562b0d110750 0x562b0d112b70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c81262e4bdeee143 server_cookie=ab7f11f298ed04b3 in_seq=0 out_seq=0 2024-04-21T09:00:49.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9b77fff640 1 -- 172.21.15.28:0/4208700402 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562b0cffec10 con 0x562b0d110750 2024-04-21T09:00:49.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9b77fff640 10 client.18069 handle_command_reply: tid=0 2024-04-21T09:00:49.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9ba1b94740 1 client.18069 shutdown 2024-04-21T09:00:49.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9ba1b94740 1 -- 172.21.15.28:0/4208700402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d099e10 msgr2=0x562b0d0ad050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/4208700402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d099e10 0x562b0d0ad050 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f9b88006010 tx=0x7f9b88009600 comp rx=0 tx=0).stop 2024-04-21T09:00:49.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9ba1b94740 1 -- 172.21.15.28:0/4208700402 shutdown_connections 2024-04-21T09:00:49.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/4208700402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562b0d099e10 0x562b0d0ad050 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/4208700402 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x562b0d110750 0x562b0d112b70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/4208700402 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x562b0d08cc00 0x562b0d0acad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9ba1b94740 1 --2- 172.21.15.28:0/4208700402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562b0d08c250 0x562b0d0af740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9b77fff640 0 client.18069 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T09:00:49.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9ba1b94740 1 -- 172.21.15.28:0/4208700402 >> 172.21.15.28:0/4208700402 conn(0x562b0d086ba0 msgr2=0x562b0d085c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9ba1b94740 1 -- 172.21.15.28:0/4208700402 shutdown_connections 2024-04-21T09:00:49.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9ba1b94740 1 -- 172.21.15.28:0/4208700402 wait complete. 2024-04-21T09:00:49.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.811+0000 7f9ba1b94740 20 client.18069 trim_cache size 0 max 16384 2024-04-21T09:00:49.821 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T09:00:49.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.812+0000 7f9b9eec1640 1 -- 172.21.15.28:0/3938003584 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9b800a3d20 msgr2=0x7f9b800a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.812+0000 7f9b9eec1640 1 --2- 172.21.15.28:0/3938003584 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9b800a3d20 0x7f9b800a61e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9b900cd660 tx=0x7f9b90091290 comp rx=0 tx=0).stop 2024-04-21T09:00:49.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.812+0000 7f9b9eec1640 1 -- 172.21.15.28:0/3938003584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98198700 msgr2=0x7f9b98198b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.812+0000 7f9b9eec1640 1 --2- 172.21.15.28:0/3938003584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98198700 0x7f9b98198b80 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000b970 tx=0x7f9b9000be30 comp rx=0 tx=0).stop 2024-04-21T09:00:49.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.812+0000 7f9b9eec1640 1 -- 172.21.15.28:0/3938003584 shutdown_connections 2024-04-21T09:00:49.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.812+0000 7f9b9eec1640 1 --2- 172.21.15.28:0/3938003584 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f9b800a3d20 0x7f9b800a61e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.812+0000 7f9b9eec1640 1 --2- 172.21.15.28:0/3938003584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b98198700 0x7f9b98198b80 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.812+0000 7f9b9eec1640 1 --2- 172.21.15.28:0/3938003584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b98107260 0x7f9b9819eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.812+0000 7f9b9eec1640 1 --2- 172.21.15.28:0/3938003584 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9b98102190 0x7f9b9819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.812+0000 7f9b9eec1640 1 -- 172.21.15.28:0/3938003584 >> 172.21.15.28:0/3938003584 conn(0x7f9b980793a0 msgr2=0x7f9b98100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.812+0000 7f9b9eec1640 1 -- 172.21.15.28:0/3938003584 shutdown_connections 2024-04-21T09:00:49.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.812+0000 7f9b9eec1640 1 -- 172.21.15.28:0/3938003584 wait complete. 2024-04-21T09:00:49.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T09:00:49.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.928+0000 7f8d31b4a640 1 -- 172.21.15.28:0/3560547321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c107260 msgr2=0x7f8d2c10ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.928+0000 7f8d31b4a640 1 --2- 172.21.15.28:0/3560547321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c107260 0x7f8d2c10ee30 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f8d24009ca0 tx=0x7f8d2402c0f0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.928+0000 7f8d31b4a640 1 -- 172.21.15.28:0/3560547321 shutdown_connections 2024-04-21T09:00:49.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.928+0000 7f8d31b4a640 1 --2- 172.21.15.28:0/3560547321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c107260 0x7f8d2c10ee30 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.928+0000 7f8d31b4a640 1 --2- 172.21.15.28:0/3560547321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d2c102b40 0x7f8d2c106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.928+0000 7f8d31b4a640 1 --2- 172.21.15.28:0/3560547321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d2c102190 0x7f8d2c102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.928+0000 7f8d31b4a640 1 -- 172.21.15.28:0/3560547321 >> 172.21.15.28:0/3560547321 conn(0x7f8d2c0fd990 msgr2=0x7f8d2c0ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.928+0000 7f8d31b4a640 1 -- 172.21.15.28:0/3560547321 shutdown_connections 2024-04-21T09:00:49.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.928+0000 7f8d31b4a640 1 -- 172.21.15.28:0/3560547321 wait complete. 2024-04-21T09:00:49.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.929+0000 7f8d31b4a640 1 Processor -- start 2024-04-21T09:00:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.929+0000 7f8d31b4a640 1 -- start start 2024-04-21T09:00:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.929+0000 7f8d31b4a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d2c102190 0x7f8d2c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.929+0000 7f8d31b4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c107260 0x7f8d2c198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.929+0000 7f8d31b4a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d2c19cff0 0x7f8d2c19d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.929+0000 7f8d31b4a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c1121d0 con 0x7f8d2c107260 2024-04-21T09:00:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.929+0000 7f8d31b4a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c112050 con 0x7f8d2c102190 2024-04-21T09:00:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.929+0000 7f8d31b4a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c112350 con 0x7f8d2c19cff0 2024-04-21T09:00:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.929+0000 7f8d2affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c107260 0x7f8d2c198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.929+0000 7f8d2affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c107260 0x7f8d2c198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34316/0 (socket says 172.21.15.28:34316) 2024-04-21T09:00:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.929+0000 7f8d2affd640 1 -- 172.21.15.28:0/3023767427 learned_addr learned my addr 172.21.15.28:0/3023767427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.930+0000 7f8d2bfff640 1 --2- 172.21.15.28:0/3023767427 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d2c19cff0 0x7f8d2c19d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.930+0000 7f8d2b7fe640 1 --2- 172.21.15.28:0/3023767427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d2c102190 0x7f8d2c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.930+0000 7f8d2affd640 1 -- 172.21.15.28:0/3023767427 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d2c19cff0 msgr2=0x7f8d2c19d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.930+0000 7f8d2affd640 1 --2- 172.21.15.28:0/3023767427 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d2c19cff0 0x7f8d2c19d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.930+0000 7f8d2affd640 1 -- 172.21.15.28:0/3023767427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d2c102190 msgr2=0x7f8d2c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.930+0000 7f8d2affd640 1 --2- 172.21.15.28:0/3023767427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d2c102190 0x7f8d2c198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.930+0000 7f8d2affd640 1 -- 172.21.15.28:0/3023767427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2c19dae0 con 0x7f8d2c107260 2024-04-21T09:00:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.930+0000 7f8d2bfff640 1 --2- 172.21.15.28:0/3023767427 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d2c19cff0 0x7f8d2c19d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:00:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.930+0000 7f8d2affd640 1 --2- 172.21.15.28:0/3023767427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c107260 0x7f8d2c198c70 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000d770 tx=0x7f8d2000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.930+0000 7f8d28ff9640 1 -- 172.21.15.28:0/3023767427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20017070 con 0x7f8d2c107260 2024-04-21T09:00:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.930+0000 7f8d28ff9640 1 -- 172.21.15.28:0/3023767427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d20009c30 con 0x7f8d2c107260 2024-04-21T09:00:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.930+0000 7f8d31b4a640 1 -- 172.21.15.28:0/3023767427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d2c19ddd0 con 0x7f8d2c107260 2024-04-21T09:00:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.930+0000 7f8d28ff9640 1 -- 172.21.15.28:0/3023767427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2000f620 con 0x7f8d2c107260 2024-04-21T09:00:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.930+0000 7f8d31b4a640 1 -- 172.21.15.28:0/3023767427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d2c1a55c0 con 0x7f8d2c107260 2024-04-21T09:00:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.932+0000 7f8d28ff9640 1 -- 172.21.15.28:0/3023767427 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8d2001a070 con 0x7f8d2c107260 2024-04-21T09:00:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.932+0000 7f8d3480e740 1 Processor -- start 2024-04-21T09:00:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.932+0000 7f8d3480e740 1 -- start start 2024-04-21T09:00:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d3480e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d298e0 0x55a1c4d29cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d3480e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d2a290 0x55a1c4d32840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d3480e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 0x55a1c4d35260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d3480e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a1c4c8a1f0 con 0x55a1c4d2a290 2024-04-21T09:00:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d3480e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55a1c4c274b0 con 0x55a1c4d32e10 2024-04-21T09:00:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d3480e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55a1c4c861b0 con 0x55a1c4d298e0 2024-04-21T09:00:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d2b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d2a290 0x55a1c4d32840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d2b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d2a290 0x55a1c4d32840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34332/0 (socket says 172.21.15.28:34332) 2024-04-21T09:00:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d2b7fe640 1 -- 172.21.15.28:0/3027949253 learned_addr learned my addr 172.21.15.28:0/3027949253 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d2bfff640 1 --2- 172.21.15.28:0/3027949253 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d298e0 0x55a1c4d29cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d2affd640 1 --2- 172.21.15.28:0/3027949253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 0x55a1c4d35260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d28ff9640 1 --2- 172.21.15.28:0/3023767427 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8d140a3d20 0x7f8d140a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d2b7fe640 1 -- 172.21.15.28:0/3027949253 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d298e0 msgr2=0x55a1c4d29cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d2b7fe640 1 --2- 172.21.15.28:0/3027949253 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d298e0 0x55a1c4d29cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d2bfff640 1 --2- 172.21.15.28:0/3023767427 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8d140a3d20 0x7f8d140a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d2b7fe640 1 -- 172.21.15.28:0/3027949253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 msgr2=0x55a1c4d35260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d2b7fe640 1 --2- 172.21.15.28:0/3027949253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 0x55a1c4d35260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d2b7fe640 1 -- 172.21.15.28:0/3027949253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1c4c191b0 con 0x55a1c4d2a290 2024-04-21T09:00:49.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d2affd640 1 --2- 172.21.15.28:0/3027949253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 0x55a1c4d35260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:49.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d28ff9640 1 -- 172.21.15.28:0/3023767427 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7f8d200d0780 con 0x7f8d2c107260 2024-04-21T09:00:49.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d2b7fe640 1 --2- 172.21.15.28:0/3027949253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d2a290 0x55a1c4d32840 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c009bc0 tx=0x7f8d1c00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f19031a49d1d6b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.933+0000 7f8d11ffb640 1 -- 172.21.15.28:0/3027949253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c016070 con 0x55a1c4d2a290 2024-04-21T09:00:49.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.934+0000 7f8d2bfff640 1 --2- 172.21.15.28:0/3023767427 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8d140a3d20 0x7f8d140a61e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8d24004af0 tx=0x7f8d2402c0f0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.934+0000 7f8d11ffb640 1 -- 172.21.15.28:0/3027949253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d1c011920 con 0x55a1c4d2a290 2024-04-21T09:00:49.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.934+0000 7f8d11ffb640 1 -- 172.21.15.28:0/3027949253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c0107f0 con 0x55a1c4d2a290 2024-04-21T09:00:49.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.934+0000 7f8d3480e740 1 -- 172.21.15.28:0/3027949253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d2a290 msgr2=0x55a1c4d32840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.934+0000 7f8d3480e740 1 --2- 172.21.15.28:0/3027949253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d2a290 0x55a1c4d32840 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c009bc0 tx=0x7f8d1c00e6c0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.934+0000 7f8d3480e740 1 -- 172.21.15.28:0/3027949253 shutdown_connections 2024-04-21T09:00:49.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.934+0000 7f8d3480e740 1 --2- 172.21.15.28:0/3027949253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 0x55a1c4d35260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.934+0000 7f8d3480e740 1 --2- 172.21.15.28:0/3027949253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d2a290 0x55a1c4d32840 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.934+0000 7f8d3480e740 1 --2- 172.21.15.28:0/3027949253 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d298e0 0x55a1c4d29cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.934+0000 7f8d3480e740 1 -- 172.21.15.28:0/3027949253 >> 172.21.15.28:0/3027949253 conn(0x55a1c4d23fb0 msgr2=0x55a1c4d24390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.934+0000 7f8d3480e740 1 -- 172.21.15.28:0/3027949253 shutdown_connections 2024-04-21T09:00:49.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.934+0000 7f8d3480e740 1 -- 172.21.15.28:0/3027949253 wait complete. 2024-04-21T09:00:49.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d3480e740 1 -- start start 2024-04-21T09:00:49.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d3480e740 1 Processor -- start 2024-04-21T09:00:49.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d3480e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d298e0 0x55a1c4d32730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d3480e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d2a290 0x55a1c4d2fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d3480e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 0x55a1c4d30060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d3480e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a1c4d35d80 con 0x55a1c4d2a290 2024-04-21T09:00:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d3480e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55a1c4c43500 con 0x55a1c4d32e10 2024-04-21T09:00:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d3480e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55a1c4d35f00 con 0x55a1c4d298e0 2024-04-21T09:00:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d3480e740 10 client.0 _ll_register_callbacks cb 0x55a1c4c11b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d2affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d2a290 0x55a1c4d2fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d2affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d2a290 0x55a1c4d2fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34338/0 (socket says 172.21.15.28:34338) 2024-04-21T09:00:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d2affd640 1 -- 172.21.15.28:0/3514269411 learned_addr learned my addr 172.21.15.28:0/3514269411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d2b7fe640 1 --2- 172.21.15.28:0/3514269411 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d298e0 0x55a1c4d32730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d2affd640 1 -- 172.21.15.28:0/3514269411 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d298e0 msgr2=0x55a1c4d32730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d2affd640 1 --2- 172.21.15.28:0/3514269411 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d298e0 0x55a1c4d32730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d2bfff640 1 --2- 172.21.15.28:0/3514269411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 0x55a1c4d30060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d2affd640 1 -- 172.21.15.28:0/3514269411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 msgr2=0x55a1c4d30060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d2affd640 1 --2- 172.21.15.28:0/3514269411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 0x55a1c4d30060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.935+0000 7f8d2affd640 1 -- 172.21.15.28:0/3514269411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1c499c3f0 con 0x55a1c4d2a290 2024-04-21T09:00:49.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d2affd640 1 --2- 172.21.15.28:0/3514269411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d2a290 0x55a1c4d2fac0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f8d2009c2c0 tx=0x7f8d2009bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926bed488da11231 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d0affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:49.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d0affd640 1 -- 172.21.15.28:0/3514269411 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20021040 con 0x55a1c4d2a290 2024-04-21T09:00:49.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d0affd640 1 -- 172.21.15.28:0/3514269411 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7f8d200d0330 con 0x55a1c4d2a290 2024-04-21T09:00:49.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d3480e740 1 -- 172.21.15.28:0/3514269411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d1c011920 con 0x55a1c4d2a290 2024-04-21T09:00:49.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d0affd640 1 -- 172.21.15.28:0/3514269411 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8d2009c330 con 0x55a1c4d2a290 2024-04-21T09:00:49.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d3480e740 10 client.18081 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:49.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d3480e740 1 -- 172.21.15.28:0/3514269411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a1c4a667d0 con 0x55a1c4d2a290 2024-04-21T09:00:49.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d0affd640 1 -- 172.21.15.28:0/3514269411 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d1c011920 con 0x55a1c4d2a290 2024-04-21T09:00:49.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d0affd640 1 -- 172.21.15.28:0/3514269411 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f8d20098110 con 0x55a1c4d2a290 2024-04-21T09:00:49.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d3480e740 10 client.18081 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:49.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d3480e740 10 client.18081 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:49.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d3480e740 20 client.18081 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:49.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d3480e740 1 --2- 172.21.15.28:0/3514269411 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a1c4d43070 0x55a1c4d45460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d3480e740 4 client.18081 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:00:49.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.936+0000 7f8d3480e740 1 -- 172.21.15.28:0/3514269411 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1c4c29930 con 0x55a1c4d43070 2024-04-21T09:00:49.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.937+0000 7f8d2b7fe640 1 --2- 172.21.15.28:0/3514269411 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a1c4d43070 0x55a1c4d45460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.937+0000 7f8d2b7fe640 1 --2- 172.21.15.28:0/3514269411 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a1c4d43070 0x55a1c4d45460 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a307b0a00fb640f server_cookie=84c41f3d3caf925c in_seq=0 out_seq=0 2024-04-21T09:00:49.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.937+0000 7f8d0affd640 10 client.18081 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.938+0000 7f8d0affd640 1 -- 172.21.15.28:0/3514269411 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a1c4c29930 con 0x55a1c4d43070 2024-04-21T09:00:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.938+0000 7f8d0affd640 10 client.18081 handle_command_reply: tid=0 2024-04-21T09:00:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.938+0000 7f8d3480e740 1 client.18081 shutdown 2024-04-21T09:00:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.938+0000 7f8d3480e740 1 -- 172.21.15.28:0/3514269411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d2a290 msgr2=0x55a1c4d2fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.938+0000 7f8d3480e740 1 --2- 172.21.15.28:0/3514269411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d2a290 0x55a1c4d2fac0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f8d2009c2c0 tx=0x7f8d2009bf70 comp rx=0 tx=0).stop 2024-04-21T09:00:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.938+0000 7f8d3480e740 1 -- 172.21.15.28:0/3514269411 shutdown_connections 2024-04-21T09:00:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.938+0000 7f8d3480e740 1 --2- 172.21.15.28:0/3514269411 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a1c4d43070 0x55a1c4d45460 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.938+0000 7f8d3480e740 1 --2- 172.21.15.28:0/3514269411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 0x55a1c4d30060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.938+0000 7f8d3480e740 1 --2- 172.21.15.28:0/3514269411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d2a290 0x55a1c4d2fac0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.938+0000 7f8d0affd640 0 client.18081 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.938+0000 7f8d3480e740 1 --2- 172.21.15.28:0/3514269411 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d298e0 0x55a1c4d32730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.938+0000 7f8d3480e740 1 -- 172.21.15.28:0/3514269411 >> 172.21.15.28:0/3514269411 conn(0x55a1c4d23fb0 msgr2=0x55a1c4d25d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.938+0000 7f8d3480e740 1 -- 172.21.15.28:0/3514269411 shutdown_connections 2024-04-21T09:00:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.938+0000 7f8d3480e740 1 -- 172.21.15.28:0/3514269411 wait complete. 2024-04-21T09:00:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.938+0000 7f8d3480e740 20 client.18081 trim_cache size 0 max 16384 2024-04-21T09:00:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.951+0000 7f8d3480e740 1 Processor -- start 2024-04-21T09:00:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.951+0000 7f8d3480e740 1 -- start start 2024-04-21T09:00:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d3480e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d298e0 0x55a1c4d3ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d3480e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d2a290 0x55a1c4d321f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d3480e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d32e10 0x55a1c4d32730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d3480e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c1afac0 con 0x55a1c4d32e10 2024-04-21T09:00:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d3480e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c1ad390 con 0x55a1c4d298e0 2024-04-21T09:00:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d3480e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c1ab200 con 0x55a1c4d2a290 2024-04-21T09:00:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d2bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d32e10 0x55a1c4d32730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d2bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d32e10 0x55a1c4d32730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34346/0 (socket says 172.21.15.28:34346) 2024-04-21T09:00:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d2bfff640 1 -- 172.21.15.28:0/2196341969 learned_addr learned my addr 172.21.15.28:0/2196341969 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d2b7fe640 1 --2- 172.21.15.28:0/2196341969 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d298e0 0x55a1c4d3ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d2bfff640 1 -- 172.21.15.28:0/2196341969 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d2a290 msgr2=0x55a1c4d321f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d2bfff640 1 --2- 172.21.15.28:0/2196341969 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d2a290 0x55a1c4d321f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d2bfff640 1 -- 172.21.15.28:0/2196341969 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d298e0 msgr2=0x55a1c4d3ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d2bfff640 1 --2- 172.21.15.28:0/2196341969 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d298e0 0x55a1c4d3ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d2bfff640 1 -- 172.21.15.28:0/2196341969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1c4d3f7b0 con 0x55a1c4d32e10 2024-04-21T09:00:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d2b7fe640 1 --2- 172.21.15.28:0/2196341969 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d298e0 0x55a1c4d3ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d2bfff640 1 --2- 172.21.15.28:0/2196341969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d32e10 0x55a1c4d32730 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f8d24036000 tx=0x7f8d24043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36afe76ed2476e7b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.952+0000 7f8d117fa640 1 -- 172.21.15.28:0/2196341969 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d24039030 con 0x55a1c4d32e10 2024-04-21T09:00:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.953+0000 7f8d117fa640 1 -- 172.21.15.28:0/2196341969 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d24053070 con 0x55a1c4d32e10 2024-04-21T09:00:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.953+0000 7f8d117fa640 1 -- 172.21.15.28:0/2196341969 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2404e460 con 0x55a1c4d32e10 2024-04-21T09:00:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.953+0000 7f8d3480e740 1 -- 172.21.15.28:0/2196341969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d32e10 msgr2=0x55a1c4d32730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.953+0000 7f8d3480e740 1 --2- 172.21.15.28:0/2196341969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d32e10 0x55a1c4d32730 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f8d24036000 tx=0x7f8d24043000 comp rx=0 tx=0).stop 2024-04-21T09:00:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.953+0000 7f8d3480e740 1 -- 172.21.15.28:0/2196341969 shutdown_connections 2024-04-21T09:00:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.953+0000 7f8d3480e740 1 --2- 172.21.15.28:0/2196341969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d32e10 0x55a1c4d32730 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.953+0000 7f8d3480e740 1 --2- 172.21.15.28:0/2196341969 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d2a290 0x55a1c4d321f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.953+0000 7f8d3480e740 1 --2- 172.21.15.28:0/2196341969 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d298e0 0x55a1c4d3ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.953+0000 7f8d3480e740 1 -- 172.21.15.28:0/2196341969 >> 172.21.15.28:0/2196341969 conn(0x55a1c4d23fb0 msgr2=0x55a1c4d1ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.953+0000 7f8d3480e740 1 -- 172.21.15.28:0/2196341969 shutdown_connections 2024-04-21T09:00:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.953+0000 7f8d3480e740 1 -- 172.21.15.28:0/2196341969 wait complete. 2024-04-21T09:00:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.953+0000 7f8d3480e740 1 -- start start 2024-04-21T09:00:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d3480e740 1 Processor -- start 2024-04-21T09:00:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d3480e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d298e0 0x55a1c4d37660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d3480e740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d2a290 0x55a1c4d37ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d3480e740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 0x55a1c4da1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d3480e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c1ab850 con 0x55a1c4d298e0 2024-04-21T09:00:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d3480e740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c1accc0 con 0x55a1c4d32e10 2024-04-21T09:00:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d3480e740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c1abf20 con 0x55a1c4d2a290 2024-04-21T09:00:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d3480e740 10 client.0 _ll_register_callbacks cb 0x7f8d2009c100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:49.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d2b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d298e0 0x55a1c4d37660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d2b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d298e0 0x55a1c4d37660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34358/0 (socket says 172.21.15.28:34358) 2024-04-21T09:00:49.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d2b7fe640 1 -- 172.21.15.28:0/1246969591 learned_addr learned my addr 172.21.15.28:0/1246969591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:49.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d2affd640 1 --2- 172.21.15.28:0/1246969591 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d2a290 0x55a1c4d37ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d2bfff640 1 --2- 172.21.15.28:0/1246969591 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 0x55a1c4da1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d2b7fe640 1 -- 172.21.15.28:0/1246969591 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d2a290 msgr2=0x55a1c4d37ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d2b7fe640 1 --2- 172.21.15.28:0/1246969591 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d2a290 0x55a1c4d37ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d2b7fe640 1 -- 172.21.15.28:0/1246969591 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 msgr2=0x55a1c4da1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d2b7fe640 1 --2- 172.21.15.28:0/1246969591 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 0x55a1c4da1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d2b7fe640 1 -- 172.21.15.28:0/1246969591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1c4da1c00 con 0x55a1c4d298e0 2024-04-21T09:00:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d2affd640 1 --2- 172.21.15.28:0/1246969591 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d2a290 0x55a1c4d37ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d2b7fe640 1 --2- 172.21.15.28:0/1246969591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d298e0 0x55a1c4d37660 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c009fd0 tx=0x7f8d1c009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d634116e8ffc01a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.954+0000 7f8d0affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.955+0000 7f8d0affd640 1 -- 172.21.15.28:0/1246969591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c016040 con 0x55a1c4d298e0 2024-04-21T09:00:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.955+0000 7f8d0affd640 1 -- 172.21.15.28:0/1246969591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7f8d1c028630 con 0x55a1c4d298e0 2024-04-21T09:00:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.955+0000 7f8d3480e740 1 -- 172.21.15.28:0/1246969591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d24053070 con 0x55a1c4d298e0 2024-04-21T09:00:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.955+0000 7f8d0affd640 1 -- 172.21.15.28:0/1246969591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8d1c032050 con 0x55a1c4d298e0 2024-04-21T09:00:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.955+0000 7f8d3480e740 10 client.18087 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.955+0000 7f8d3480e740 1 -- 172.21.15.28:0/1246969591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a1c4daa120 con 0x55a1c4d298e0 2024-04-21T09:00:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.955+0000 7f8d0affd640 1 -- 172.21.15.28:0/1246969591 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d24053070 con 0x55a1c4d298e0 2024-04-21T09:00:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.955+0000 7f8d0affd640 1 -- 172.21.15.28:0/1246969591 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f8d1c00f040 con 0x55a1c4d298e0 2024-04-21T09:00:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.955+0000 7f8d3480e740 10 client.18087 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.955+0000 7f8d3480e740 10 client.18087 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.955+0000 7f8d3480e740 20 client.18087 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.955+0000 7f8d3480e740 1 --2- 172.21.15.28:0/1246969591 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a1c4daa520 0x55a1c4dac910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:49.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.955+0000 7f8d3480e740 4 client.18087 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T09:00:49.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.955+0000 7f8d3480e740 1 -- 172.21.15.28:0/1246969591 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a1c4c97c10 con 0x55a1c4daa520 2024-04-21T09:00:49.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.955+0000 7f8d2affd640 1 --2- 172.21.15.28:0/1246969591 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a1c4daa520 0x55a1c4dac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:49.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d0affd640 10 client.18087 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:49.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d2affd640 1 --2- 172.21.15.28:0/1246969591 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a1c4daa520 0x55a1c4dac910 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6afe09e16c7ef5ad server_cookie=baaaeec56d9ae241 in_seq=0 out_seq=0 2024-04-21T09:00:49.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d0affd640 1 -- 172.21.15.28:0/1246969591 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a1c4c97c10 con 0x55a1c4daa520 2024-04-21T09:00:49.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d0affd640 10 client.18087 handle_command_reply: tid=0 2024-04-21T09:00:49.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d3480e740 1 client.18087 shutdown 2024-04-21T09:00:49.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d3480e740 1 -- 172.21.15.28:0/1246969591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d298e0 msgr2=0x55a1c4d37660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d3480e740 1 --2- 172.21.15.28:0/1246969591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d298e0 0x55a1c4d37660 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c009fd0 tx=0x7f8d1c009730 comp rx=0 tx=0).stop 2024-04-21T09:00:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d3480e740 1 -- 172.21.15.28:0/1246969591 shutdown_connections 2024-04-21T09:00:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d3480e740 1 --2- 172.21.15.28:0/1246969591 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x55a1c4daa520 0x55a1c4dac910 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d3480e740 1 --2- 172.21.15.28:0/1246969591 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a1c4d32e10 0x55a1c4da1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d3480e740 1 --2- 172.21.15.28:0/1246969591 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a1c4d2a290 0x55a1c4d37ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d3480e740 1 --2- 172.21.15.28:0/1246969591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a1c4d298e0 0x55a1c4d37660 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d3480e740 1 -- 172.21.15.28:0/1246969591 >> 172.21.15.28:0/1246969591 conn(0x55a1c4d23fb0 msgr2=0x55a1c4d2a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d0affd640 0 client.18087 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d3480e740 1 -- 172.21.15.28:0/1246969591 shutdown_connections 2024-04-21T09:00:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d3480e740 1 -- 172.21.15.28:0/1246969591 wait complete. 2024-04-21T09:00:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.956+0000 7f8d3480e740 20 client.18087 trim_cache size 0 max 16384 2024-04-21T09:00:49.968 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T09:00:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.957+0000 7f8d31b4a640 1 -- 172.21.15.28:0/3023767427 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8d140a3d20 msgr2=0x7f8d140a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.957+0000 7f8d31b4a640 1 --2- 172.21.15.28:0/3023767427 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8d140a3d20 0x7f8d140a61e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8d24004af0 tx=0x7f8d2402c0f0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.957+0000 7f8d31b4a640 1 -- 172.21.15.28:0/3023767427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c107260 msgr2=0x7f8d2c198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:49.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.957+0000 7f8d31b4a640 1 --2- 172.21.15.28:0/3023767427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c107260 0x7f8d2c198c70 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000d770 tx=0x7f8d2000dc30 comp rx=0 tx=0).stop 2024-04-21T09:00:49.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.957+0000 7f8d31b4a640 1 -- 172.21.15.28:0/3023767427 shutdown_connections 2024-04-21T09:00:49.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.957+0000 7f8d31b4a640 1 --2- 172.21.15.28:0/3023767427 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8d140a3d20 0x7f8d140a61e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.957+0000 7f8d31b4a640 1 --2- 172.21.15.28:0/3023767427 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8d2c19cff0 0x7f8d2c19d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.957+0000 7f8d31b4a640 1 --2- 172.21.15.28:0/3023767427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2c107260 0x7f8d2c198c70 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.957+0000 7f8d31b4a640 1 --2- 172.21.15.28:0/3023767427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8d2c102190 0x7f8d2c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:49.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.957+0000 7f8d31b4a640 1 -- 172.21.15.28:0/3023767427 >> 172.21.15.28:0/3023767427 conn(0x7f8d2c0fd990 msgr2=0x7f8d2c0fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:49.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.957+0000 7f8d31b4a640 1 -- 172.21.15.28:0/3023767427 shutdown_connections 2024-04-21T09:00:49.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:49.957+0000 7f8d31b4a640 1 -- 172.21.15.28:0/3023767427 wait complete. 2024-04-21T09:00:50.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:50 smithi152 ceph-mon[31401]: pgmap v298: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T09:00:50.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:50 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1973552121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T09:00:50.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:50 smithi028 ceph-mon[28003]: pgmap v298: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T09:00:50.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:50 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1973552121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T09:00:50.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:50 smithi171 ceph-mon[31358]: pgmap v298: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T09:00:50.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:50 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1973552121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T09:00:50.972 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T09:00:50.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T09:00:50.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T09:00:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.092+0000 7f010feba640 1 -- 172.21.15.28:0/969982389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f010810e960 msgr2=0x7f0108110d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.092+0000 7f010feba640 1 --2- 172.21.15.28:0/969982389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f010810e960 0x7f0108110d50 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f0100009ca0 tx=0x7f010002c0f0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.092+0000 7f010feba640 1 -- 172.21.15.28:0/969982389 shutdown_connections 2024-04-21T09:00:51.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.092+0000 7f010feba640 1 --2- 172.21.15.28:0/969982389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f010810e960 0x7f0108110d50 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.092+0000 7f010feba640 1 --2- 172.21.15.28:0/969982389 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f010806aef0 0x7f010810e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.092+0000 7f010feba640 1 --2- 172.21.15.28:0/969982389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f010806a5d0 0x7f010806a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.092+0000 7f010feba640 1 -- 172.21.15.28:0/969982389 >> 172.21.15.28:0/969982389 conn(0x7f0108077eb0 msgr2=0x7f01080782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:51.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.092+0000 7f010feba640 1 -- 172.21.15.28:0/969982389 shutdown_connections 2024-04-21T09:00:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.092+0000 7f010feba640 1 -- 172.21.15.28:0/969982389 wait complete. 2024-04-21T09:00:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.093+0000 7f010feba640 1 Processor -- start 2024-04-21T09:00:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.093+0000 7f010feba640 1 -- start start 2024-04-21T09:00:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.093+0000 7f010feba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f010806a5d0 0x7f01081a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.093+0000 7f010feba640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f010806aef0 0x7f01081a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.093+0000 7f010feba640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f010810e960 0x7f010819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.093+0000 7f010feba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01081143b0 con 0x7f010806a5d0 2024-04-21T09:00:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.093+0000 7f010feba640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0108114230 con 0x7f010810e960 2024-04-21T09:00:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.093+0000 7f010feba640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0108114530 con 0x7f010806aef0 2024-04-21T09:00:51.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.093+0000 7f010dc56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f010806a5d0 0x7f01081a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.093+0000 7f010dc56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f010806a5d0 0x7f01081a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34384/0 (socket says 172.21.15.28:34384) 2024-04-21T09:00:51.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.093+0000 7f010dc56640 1 -- 172.21.15.28:0/2212383024 learned_addr learned my addr 172.21.15.28:0/2212383024 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:51.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.094+0000 7f010e457640 1 --2- 172.21.15.28:0/2212383024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f010810e960 0x7f010819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.094+0000 7f010d455640 1 --2- 172.21.15.28:0/2212383024 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f010806aef0 0x7f01081a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.094+0000 7f010dc56640 1 -- 172.21.15.28:0/2212383024 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f010806aef0 msgr2=0x7f01081a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.094+0000 7f010dc56640 1 --2- 172.21.15.28:0/2212383024 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f010806aef0 0x7f01081a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.094+0000 7f010dc56640 1 -- 172.21.15.28:0/2212383024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f010810e960 msgr2=0x7f010819a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.094+0000 7f010dc56640 1 --2- 172.21.15.28:0/2212383024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f010810e960 0x7f010819a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.094+0000 7f010dc56640 1 -- 172.21.15.28:0/2212383024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f010819af60 con 0x7f010806a5d0 2024-04-21T09:00:51.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.094+0000 7f010dc56640 1 --2- 172.21.15.28:0/2212383024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f010806a5d0 0x7f01081a0620 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f010400c920 tx=0x7f010400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:51.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.094+0000 7f00feffd640 1 -- 172.21.15.28:0/2212383024 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0104007c70 con 0x7f010806a5d0 2024-04-21T09:00:51.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.094+0000 7f00feffd640 1 -- 172.21.15.28:0/2212383024 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0104007e10 con 0x7f010806a5d0 2024-04-21T09:00:51.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.094+0000 7f010feba640 1 -- 172.21.15.28:0/2212383024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f010819b250 con 0x7f010806a5d0 2024-04-21T09:00:51.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.094+0000 7f00feffd640 1 -- 172.21.15.28:0/2212383024 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0104018820 con 0x7f010806a5d0 2024-04-21T09:00:51.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.094+0000 7f010feba640 1 -- 172.21.15.28:0/2212383024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01081a7470 con 0x7f010806a5d0 2024-04-21T09:00:51.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.097+0000 7f00feffd640 1 -- 172.21.15.28:0/2212383024 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f010400f450 con 0x7f010806a5d0 2024-04-21T09:00:51.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.097+0000 7f0112b87740 1 Processor -- start 2024-04-21T09:00:51.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.097+0000 7f0112b87740 1 -- start start 2024-04-21T09:00:51.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.097+0000 7f0112b87740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e78e0 0x5650724e7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f0112b87740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724e8290 0x5650724f0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f0112b87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724f0e10 0x5650724f3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f0112b87740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56507244bf00 con 0x5650724f0e10 2024-04-21T09:00:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f0112b87740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x5650723e9590 con 0x5650724e78e0 2024-04-21T09:00:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f0112b87740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x565072447ec0 con 0x5650724e8290 2024-04-21T09:00:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f00feffd640 1 --2- 172.21.15.28:0/2212383024 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00f40a3da0 0x7f00f40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f010d455640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724f0e10 0x5650724f3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f010d455640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724f0e10 0x5650724f3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34396/0 (socket says 172.21.15.28:34396) 2024-04-21T09:00:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f010d455640 1 -- 172.21.15.28:0/1057608223 learned_addr learned my addr 172.21.15.28:0/1057608223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f010e457640 1 --2- 172.21.15.28:0/2212383024 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00f40a3da0 0x7f00f40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f010dc56640 1 --2- 172.21.15.28:0/1057608223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724e8290 0x5650724f0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f010e457640 1 --2- 172.21.15.28:0/1057608223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e78e0 0x5650724e7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f00feffd640 1 -- 172.21.15.28:0/2212383024 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7f01040d0d90 con 0x7f010806a5d0 2024-04-21T09:00:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f010d455640 1 -- 172.21.15.28:0/1057608223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724e8290 msgr2=0x5650724f0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f010d455640 1 --2- 172.21.15.28:0/1057608223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724e8290 0x5650724f0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f010d455640 1 -- 172.21.15.28:0/1057608223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e78e0 msgr2=0x5650724e7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f010d455640 1 --2- 172.21.15.28:0/1057608223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e78e0 0x5650724e7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f010d455640 1 -- 172.21.15.28:0/1057608223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650722267e0 con 0x5650724f0e10 2024-04-21T09:00:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f010e457640 1 --2- 172.21.15.28:0/1057608223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e78e0 0x5650724e7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:00:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f010d455640 1 --2- 172.21.15.28:0/1057608223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724f0e10 0x5650724f3260 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f00f800bbc0 tx=0x7f00f800f360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e22bcab34965eadf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.098+0000 7f010e457640 1 --2- 172.21.15.28:0/2212383024 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00f40a3da0 0x7f00f40a6260 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0100006010 tx=0x7f010002e040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.099+0000 7f00f3fff640 1 -- 172.21.15.28:0/1057608223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f8015070 con 0x5650724f0e10 2024-04-21T09:00:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.099+0000 7f00f3fff640 1 -- 172.21.15.28:0/1057608223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00f800fd90 con 0x5650724f0e10 2024-04-21T09:00:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.099+0000 7f00f3fff640 1 -- 172.21.15.28:0/1057608223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f8011440 con 0x5650724f0e10 2024-04-21T09:00:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.099+0000 7f0112b87740 1 -- 172.21.15.28:0/1057608223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724f0e10 msgr2=0x5650724f3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.099+0000 7f0112b87740 1 --2- 172.21.15.28:0/1057608223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724f0e10 0x5650724f3260 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f00f800bbc0 tx=0x7f00f800f360 comp rx=0 tx=0).stop 2024-04-21T09:00:51.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.099+0000 7f0112b87740 1 -- 172.21.15.28:0/1057608223 shutdown_connections 2024-04-21T09:00:51.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.099+0000 7f0112b87740 1 --2- 172.21.15.28:0/1057608223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724f0e10 0x5650724f3260 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.099+0000 7f0112b87740 1 --2- 172.21.15.28:0/1057608223 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724e8290 0x5650724f0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.099+0000 7f0112b87740 1 --2- 172.21.15.28:0/1057608223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e78e0 0x5650724e7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.099+0000 7f0112b87740 1 -- 172.21.15.28:0/1057608223 >> 172.21.15.28:0/1057608223 conn(0x5650724e1fb0 msgr2=0x5650724e2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:51.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.099+0000 7f0112b87740 1 -- 172.21.15.28:0/1057608223 shutdown_connections 2024-04-21T09:00:51.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.099+0000 7f0112b87740 1 -- 172.21.15.28:0/1057608223 wait complete. 2024-04-21T09:00:51.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.100+0000 7f0112b87740 1 -- start start 2024-04-21T09:00:51.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.100+0000 7f0112b87740 1 Processor -- start 2024-04-21T09:00:51.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.100+0000 7f0112b87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724e78e0 0x5650724edd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.100+0000 7f0112b87740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e8290 0x5650724ee290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.100+0000 7f0112b87740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724f0e10 0x5650724ee7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.100+0000 7f0112b87740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5650724f3d80 con 0x5650724e78e0 2024-04-21T09:00:51.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.100+0000 7f0112b87740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x565072405580 con 0x5650724e8290 2024-04-21T09:00:51.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.100+0000 7f0112b87740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x5650724f3f00 con 0x5650724f0e10 2024-04-21T09:00:51.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.100+0000 7f0112b87740 10 client.0 _ll_register_callbacks cb 0x56507240b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:51.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.100+0000 7f010dc56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724e78e0 0x5650724edd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.100+0000 7f010dc56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724e78e0 0x5650724edd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34404/0 (socket says 172.21.15.28:34404) 2024-04-21T09:00:51.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.100+0000 7f010dc56640 1 -- 172.21.15.28:0/3831911859 learned_addr learned my addr 172.21.15.28:0/3831911859 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:51.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f010d455640 1 --2- 172.21.15.28:0/3831911859 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e8290 0x5650724ee290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f010e457640 1 --2- 172.21.15.28:0/3831911859 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724f0e10 0x5650724ee7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f010dc56640 1 -- 172.21.15.28:0/3831911859 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724f0e10 msgr2=0x5650724ee7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f010dc56640 1 --2- 172.21.15.28:0/3831911859 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724f0e10 0x5650724ee7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f010dc56640 1 -- 172.21.15.28:0/3831911859 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e8290 msgr2=0x5650724ee290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f010dc56640 1 --2- 172.21.15.28:0/3831911859 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e8290 0x5650724ee290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f010dc56640 1 -- 172.21.15.28:0/3831911859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565072286f70 con 0x5650724e78e0 2024-04-21T09:00:51.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f010dc56640 1 --2- 172.21.15.28:0/3831911859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724e78e0 0x5650724edd50 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f010409c8d0 tx=0x7f0104015d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90dcdda66546a5c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:51.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f00f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:51.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f00f17fa640 1 -- 172.21.15.28:0/3831911859 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010409c940 con 0x5650724e78e0 2024-04-21T09:00:51.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f00f17fa640 1 -- 172.21.15.28:0/3831911859 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7f0104015d70 con 0x5650724e78e0 2024-04-21T09:00:51.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f0112b87740 1 -- 172.21.15.28:0/3831911859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00f800fd90 con 0x5650724e78e0 2024-04-21T09:00:51.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f00f17fa640 1 -- 172.21.15.28:0/3831911859 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f01040a0cf0 con 0x5650724e78e0 2024-04-21T09:00:51.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f0112b87740 10 client.18099 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:51.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f0112b87740 1 -- 172.21.15.28:0/3831911859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56507211e170 con 0x5650724e78e0 2024-04-21T09:00:51.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.101+0000 7f00f17fa640 1 -- 172.21.15.28:0/3831911859 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00f800fd90 con 0x5650724e78e0 2024-04-21T09:00:51.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.102+0000 7f00f17fa640 1 -- 172.21.15.28:0/3831911859 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f0104098fb0 con 0x5650724e78e0 2024-04-21T09:00:51.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.102+0000 7f0112b87740 10 client.18099 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:51.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.102+0000 7f0112b87740 10 client.18099 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:51.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.102+0000 7f0112b87740 20 client.18099 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:51.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.102+0000 7f0112b87740 1 --2- 172.21.15.28:0/3831911859 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x565072501310 0x565072503700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.102+0000 7f0112b87740 4 client.18099 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:00:51.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.102+0000 7f0112b87740 1 -- 172.21.15.28:0/3831911859 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565072459920 con 0x565072501310 2024-04-21T09:00:51.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.102+0000 7f010d455640 1 --2- 172.21.15.28:0/3831911859 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x565072501310 0x565072503700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.103+0000 7f010d455640 1 --2- 172.21.15.28:0/3831911859 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x565072501310 0x565072503700 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcb02e177286cc27 server_cookie=226a5a585ef9cc09 in_seq=0 out_seq=0 2024-04-21T09:00:51.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.103+0000 7f00f17fa640 10 client.18099 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:51.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.103+0000 7f00f17fa640 1 -- 172.21.15.28:0/3831911859 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565072459920 con 0x565072501310 2024-04-21T09:00:51.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.103+0000 7f00f17fa640 10 client.18099 handle_command_reply: tid=0 2024-04-21T09:00:51.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.103+0000 7f0112b87740 1 client.18099 shutdown 2024-04-21T09:00:51.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.103+0000 7f0112b87740 1 -- 172.21.15.28:0/3831911859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724e78e0 msgr2=0x5650724edd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.103+0000 7f0112b87740 1 --2- 172.21.15.28:0/3831911859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724e78e0 0x5650724edd50 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f010409c8d0 tx=0x7f0104015d00 comp rx=0 tx=0).stop 2024-04-21T09:00:51.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.103+0000 7f0112b87740 1 -- 172.21.15.28:0/3831911859 shutdown_connections 2024-04-21T09:00:51.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.103+0000 7f0112b87740 1 --2- 172.21.15.28:0/3831911859 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x565072501310 0x565072503700 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.103+0000 7f0112b87740 1 --2- 172.21.15.28:0/3831911859 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724f0e10 0x5650724ee7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.103+0000 7f0112b87740 1 --2- 172.21.15.28:0/3831911859 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e8290 0x5650724ee290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.103+0000 7f0112b87740 1 --2- 172.21.15.28:0/3831911859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724e78e0 0x5650724edd50 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.103+0000 7f00f17fa640 0 client.18099 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:51.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.103+0000 7f0112b87740 1 -- 172.21.15.28:0/3831911859 >> 172.21.15.28:0/3831911859 conn(0x5650724e1fb0 msgr2=0x5650724e3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.104+0000 7f0112b87740 1 -- 172.21.15.28:0/3831911859 shutdown_connections 2024-04-21T09:00:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.104+0000 7f0112b87740 1 -- 172.21.15.28:0/3831911859 wait complete. 2024-04-21T09:00:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.104+0000 7f0112b87740 20 client.18099 trim_cache size 0 max 16384 2024-04-21T09:00:51.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.121+0000 7f0112b87740 1 Processor -- start 2024-04-21T09:00:51.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.121+0000 7f0112b87740 1 -- start start 2024-04-21T09:00:51.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f0112b87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724e78e0 0x5650724f0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f0112b87740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724e8290 0x5650724ed630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f0112b87740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724f0e10 0x565072508f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f0112b87740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01081b3d40 con 0x5650724e78e0 2024-04-21T09:00:51.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f0112b87740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f01081b1610 con 0x5650724f0e10 2024-04-21T09:00:51.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f0112b87740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f01081af480 con 0x5650724e8290 2024-04-21T09:00:51.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f010dc56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724e78e0 0x5650724f0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f010dc56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724e78e0 0x5650724f0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34418/0 (socket says 172.21.15.28:34418) 2024-04-21T09:00:51.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f010dc56640 1 -- 172.21.15.28:0/852527498 learned_addr learned my addr 172.21.15.28:0/852527498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:51.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f010d455640 1 --2- 172.21.15.28:0/852527498 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724e8290 0x5650724ed630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f010e457640 1 --2- 172.21.15.28:0/852527498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724f0e10 0x565072508f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f010dc56640 1 -- 172.21.15.28:0/852527498 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724e8290 msgr2=0x5650724ed630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f010dc56640 1 --2- 172.21.15.28:0/852527498 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724e8290 0x5650724ed630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f010dc56640 1 -- 172.21.15.28:0/852527498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724f0e10 msgr2=0x565072508f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f010dc56640 1 --2- 172.21.15.28:0/852527498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724f0e10 0x565072508f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f010dc56640 1 -- 172.21.15.28:0/852527498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650724e9910 con 0x5650724e78e0 2024-04-21T09:00:51.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f010e457640 1 --2- 172.21.15.28:0/852527498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724f0e10 0x565072508f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:00:51.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f010dc56640 1 --2- 172.21.15.28:0/852527498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724e78e0 0x5650724f0480 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f0104098da0 tx=0x7f01040d1090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b04033efcd3e33a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:51.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f00f37fe640 1 -- 172.21.15.28:0/852527498 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0104098be0 con 0x5650724e78e0 2024-04-21T09:00:51.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f00f37fe640 1 -- 172.21.15.28:0/852527498 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01040ce370 con 0x5650724e78e0 2024-04-21T09:00:51.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.122+0000 7f00f37fe640 1 -- 172.21.15.28:0/852527498 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010409c2a0 con 0x5650724e78e0 2024-04-21T09:00:51.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.123+0000 7f0112b87740 1 -- 172.21.15.28:0/852527498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724e78e0 msgr2=0x5650724f0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.123+0000 7f0112b87740 1 --2- 172.21.15.28:0/852527498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724e78e0 0x5650724f0480 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f0104098da0 tx=0x7f01040d1090 comp rx=0 tx=0).stop 2024-04-21T09:00:51.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.123+0000 7f0112b87740 1 -- 172.21.15.28:0/852527498 shutdown_connections 2024-04-21T09:00:51.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.123+0000 7f0112b87740 1 --2- 172.21.15.28:0/852527498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724f0e10 0x565072508f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.123+0000 7f0112b87740 1 --2- 172.21.15.28:0/852527498 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724e8290 0x5650724ed630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.123+0000 7f0112b87740 1 --2- 172.21.15.28:0/852527498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724e78e0 0x5650724f0480 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.123+0000 7f0112b87740 1 -- 172.21.15.28:0/852527498 >> 172.21.15.28:0/852527498 conn(0x5650724e1fb0 msgr2=0x5650724e3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:51.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.123+0000 7f0112b87740 1 -- 172.21.15.28:0/852527498 shutdown_connections 2024-04-21T09:00:51.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.123+0000 7f0112b87740 1 -- 172.21.15.28:0/852527498 wait complete. 2024-04-21T09:00:51.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.123+0000 7f0112b87740 1 -- start start 2024-04-21T09:00:51.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.123+0000 7f0112b87740 1 Processor -- start 2024-04-21T09:00:51.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f0112b87740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e78e0 0x5650724f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f0112b87740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724e8290 0x5650724f52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f0112b87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724f0e10 0x5650724f5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f0112b87740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01081afad0 con 0x5650724f0e10 2024-04-21T09:00:51.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f0112b87740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f01081b0f40 con 0x5650724e78e0 2024-04-21T09:00:51.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f0112b87740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f01081b01a0 con 0x5650724e8290 2024-04-21T09:00:51.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f0112b87740 10 client.0 _ll_register_callbacks cb 0x7f010409c710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:51.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f010e457640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724f0e10 0x5650724f5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f010e457640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724f0e10 0x5650724f5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34426/0 (socket says 172.21.15.28:34426) 2024-04-21T09:00:51.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f010e457640 1 -- 172.21.15.28:0/2009744188 learned_addr learned my addr 172.21.15.28:0/2009744188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:51.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f010dc56640 1 --2- 172.21.15.28:0/2009744188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e78e0 0x5650724f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f010d455640 1 --2- 172.21.15.28:0/2009744188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724e8290 0x5650724f52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f010e457640 1 -- 172.21.15.28:0/2009744188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724e8290 msgr2=0x5650724f52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f010e457640 1 --2- 172.21.15.28:0/2009744188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724e8290 0x5650724f52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f010e457640 1 -- 172.21.15.28:0/2009744188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e78e0 msgr2=0x5650724f4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f010e457640 1 --2- 172.21.15.28:0/2009744188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e78e0 0x5650724f4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f010e457640 1 -- 172.21.15.28:0/2009744188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565072566970 con 0x5650724f0e10 2024-04-21T09:00:51.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f010e457640 1 --2- 172.21.15.28:0/2009744188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724f0e10 0x5650724f5820 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f0100009ca0 tx=0x7f010002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f801a72e8a4c00 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:51.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f00f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:51.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.124+0000 7f00f17fa640 1 -- 172.21.15.28:0/2009744188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0100052020 con 0x5650724f0e10 2024-04-21T09:00:51.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.125+0000 7f0112b87740 1 -- 172.21.15.28:0/2009744188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x565072453790 con 0x5650724f0e10 2024-04-21T09:00:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.125+0000 7f00f17fa640 1 -- 172.21.15.28:0/2009744188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7f0100004250 con 0x5650724f0e10 2024-04-21T09:00:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.125+0000 7f00f17fa640 1 -- 172.21.15.28:0/2009744188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0100032e90 con 0x5650724f0e10 2024-04-21T09:00:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.125+0000 7f0112b87740 10 client.18105 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.125+0000 7f0112b87740 1 -- 172.21.15.28:0/2009744188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x565072567c50 con 0x5650724f0e10 2024-04-21T09:00:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.125+0000 7f00f17fa640 1 -- 172.21.15.28:0/2009744188 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x565072453790 con 0x5650724f0e10 2024-04-21T09:00:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.125+0000 7f00f17fa640 1 -- 172.21.15.28:0/2009744188 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f010004e5f0 con 0x5650724f0e10 2024-04-21T09:00:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.125+0000 7f0112b87740 10 client.18105 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.125+0000 7f0112b87740 10 client.18105 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:51.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.125+0000 7f0112b87740 20 client.18105 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:51.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.125+0000 7f0112b87740 1 --2- 172.21.15.28:0/2009744188 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5650725683d0 0x56507256a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.125+0000 7f0112b87740 4 client.18105 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T09:00:51.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.125+0000 7f0112b87740 1 -- 172.21.15.28:0/2009744188 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5650723cfc50 con 0x5650725683d0 2024-04-21T09:00:51.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.125+0000 7f010dc56640 1 --2- 172.21.15.28:0/2009744188 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5650725683d0 0x56507256a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.126+0000 7f00f17fa640 10 client.18105 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:51.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.126+0000 7f010dc56640 1 --2- 172.21.15.28:0/2009744188 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5650725683d0 0x56507256a7f0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85563831a9afe71f server_cookie=707a87b736df9b84 in_seq=0 out_seq=0 2024-04-21T09:00:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.127+0000 7f00f17fa640 1 -- 172.21.15.28:0/2009744188 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5650723cfc50 con 0x5650725683d0 2024-04-21T09:00:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.127+0000 7f00f17fa640 10 client.18105 handle_command_reply: tid=0 2024-04-21T09:00:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.127+0000 7f0112b87740 1 client.18105 shutdown 2024-04-21T09:00:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.127+0000 7f0112b87740 1 -- 172.21.15.28:0/2009744188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724f0e10 msgr2=0x5650724f5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.127+0000 7f0112b87740 1 --2- 172.21.15.28:0/2009744188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724f0e10 0x5650724f5820 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f0100009ca0 tx=0x7f010002c0f0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.127+0000 7f0112b87740 1 -- 172.21.15.28:0/2009744188 shutdown_connections 2024-04-21T09:00:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.127+0000 7f0112b87740 1 --2- 172.21.15.28:0/2009744188 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x5650725683d0 0x56507256a7f0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.127+0000 7f0112b87740 1 --2- 172.21.15.28:0/2009744188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5650724f0e10 0x5650724f5820 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.127+0000 7f0112b87740 1 --2- 172.21.15.28:0/2009744188 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x5650724e8290 0x5650724f52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.127+0000 7f00f17fa640 0 client.18105 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:51.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.127+0000 7f0112b87740 1 --2- 172.21.15.28:0/2009744188 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5650724e78e0 0x5650724f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.127+0000 7f0112b87740 1 -- 172.21.15.28:0/2009744188 >> 172.21.15.28:0/2009744188 conn(0x5650724e1fb0 msgr2=0x5650724e3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:51.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.128+0000 7f0112b87740 1 -- 172.21.15.28:0/2009744188 shutdown_connections 2024-04-21T09:00:51.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.128+0000 7f0112b87740 1 -- 172.21.15.28:0/2009744188 wait complete. 2024-04-21T09:00:51.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.128+0000 7f0112b87740 20 client.18105 trim_cache size 0 max 16384 2024-04-21T09:00:51.138 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T09:00:51.138 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-21T09:00:51.138 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "61f0fcef-e3d9-4b5e-9fb1-1cb96000a842", 2024-04-21T09:00:51.138 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-21T09:00:51.138 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T09:00:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.128+0000 7f010feba640 1 -- 172.21.15.28:0/2212383024 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00f40a3da0 msgr2=0x7f00f40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.128+0000 7f010feba640 1 --2- 172.21.15.28:0/2212383024 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00f40a3da0 0x7f00f40a6260 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0100006010 tx=0x7f010002e040 comp rx=0 tx=0).stop 2024-04-21T09:00:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.128+0000 7f010feba640 1 -- 172.21.15.28:0/2212383024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f010806a5d0 msgr2=0x7f01081a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.128+0000 7f010feba640 1 --2- 172.21.15.28:0/2212383024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f010806a5d0 0x7f01081a0620 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f010400c920 tx=0x7f010400cde0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.128+0000 7f010feba640 1 -- 172.21.15.28:0/2212383024 shutdown_connections 2024-04-21T09:00:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.128+0000 7f010feba640 1 --2- 172.21.15.28:0/2212383024 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00f40a3da0 0x7f00f40a6260 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.128+0000 7f010feba640 1 --2- 172.21.15.28:0/2212383024 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f010810e960 0x7f010819a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.128+0000 7f010feba640 1 --2- 172.21.15.28:0/2212383024 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f010806aef0 0x7f01081a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.128+0000 7f010feba640 1 --2- 172.21.15.28:0/2212383024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f010806a5d0 0x7f01081a0620 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.128+0000 7f010feba640 1 -- 172.21.15.28:0/2212383024 >> 172.21.15.28:0/2212383024 conn(0x7f0108077eb0 msgr2=0x7f010810fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:51.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.128+0000 7f010feba640 1 -- 172.21.15.28:0/2212383024 shutdown_connections 2024-04-21T09:00:51.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.128+0000 7f010feba640 1 -- 172.21.15.28:0/2212383024 wait complete. 2024-04-21T09:00:51.141 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T09:00:51.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T09:00:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.249+0000 7efe0662c640 1 -- 172.21.15.28:0/3195785874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe001011a0 msgr2=0x7efe00101600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.249+0000 7efe0662c640 1 --2- 172.21.15.28:0/3195785874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe001011a0 0x7efe00101600 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009900 tx=0x7efdf002bc30 comp rx=0 tx=0).stop 2024-04-21T09:00:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.249+0000 7efe0662c640 1 -- 172.21.15.28:0/3195785874 shutdown_connections 2024-04-21T09:00:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.249+0000 7efe0662c640 1 --2- 172.21.15.28:0/3195785874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe00101b40 0x7efe0010dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.249+0000 7efe0662c640 1 --2- 172.21.15.28:0/3195785874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe001011a0 0x7efe00101600 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.249+0000 7efe0662c640 1 --2- 172.21.15.28:0/3195785874 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efe00106f90 0x7efe00107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.249+0000 7efe0662c640 1 -- 172.21.15.28:0/3195785874 >> 172.21.15.28:0/3195785874 conn(0x7efe00076c50 msgr2=0x7efe00077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.249+0000 7efe0662c640 1 -- 172.21.15.28:0/3195785874 shutdown_connections 2024-04-21T09:00:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.249+0000 7efe0662c640 1 -- 172.21.15.28:0/3195785874 wait complete. 2024-04-21T09:00:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.250+0000 7efe0662c640 1 Processor -- start 2024-04-21T09:00:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.250+0000 7efe0662c640 1 -- start start 2024-04-21T09:00:51.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.250+0000 7efe0662c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efe001011a0 0x7efe001975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.250+0000 7efe0662c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe00101b40 0x7efe00197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.250+0000 7efe0662c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe00106f90 0x7efe0019bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.250+0000 7efe0662c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe001110c0 con 0x7efe00106f90 2024-04-21T09:00:51.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.250+0000 7efe0662c640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efe00110f40 con 0x7efe00101b40 2024-04-21T09:00:51.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.250+0000 7efe0662c640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7efe00111240 con 0x7efe001011a0 2024-04-21T09:00:51.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.250+0000 7efe04bc9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe00106f90 0x7efe0019bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.250+0000 7efe04bc9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe00106f90 0x7efe0019bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34446/0 (socket says 172.21.15.28:34446) 2024-04-21T09:00:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.250+0000 7efe04bc9640 1 -- 172.21.15.28:0/1028419659 learned_addr learned my addr 172.21.15.28:0/1028419659 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.250+0000 7efdfffff640 1 --2- 172.21.15.28:0/1028419659 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efe001011a0 0x7efe001975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.251+0000 7efdff7fe640 1 --2- 172.21.15.28:0/1028419659 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe00101b40 0x7efe00197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.251+0000 7efe04bc9640 1 -- 172.21.15.28:0/1028419659 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efe001011a0 msgr2=0x7efe001975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.251+0000 7efe04bc9640 1 --2- 172.21.15.28:0/1028419659 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efe001011a0 0x7efe001975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.251+0000 7efe04bc9640 1 -- 172.21.15.28:0/1028419659 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe00101b40 msgr2=0x7efe00197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.251+0000 7efe04bc9640 1 --2- 172.21.15.28:0/1028419659 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe00101b40 0x7efe00197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.251+0000 7efe04bc9640 1 -- 172.21.15.28:0/1028419659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0019c5d0 con 0x7efe00106f90 2024-04-21T09:00:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.251+0000 7efdff7fe640 1 --2- 172.21.15.28:0/1028419659 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe00101b40 0x7efe00197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:00:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.251+0000 7efe04bc9640 1 --2- 172.21.15.28:0/1028419659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe00106f90 0x7efe0019bea0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7efdf800cab0 tx=0x7efdf800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.251+0000 7efdfd7fa640 1 -- 172.21.15.28:0/1028419659 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf800ae20 con 0x7efe00106f90 2024-04-21T09:00:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.251+0000 7efdfd7fa640 1 -- 172.21.15.28:0/1028419659 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdf800f900 con 0x7efe00106f90 2024-04-21T09:00:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.251+0000 7efe0662c640 1 -- 172.21.15.28:0/1028419659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe0019c8c0 con 0x7efe00106f90 2024-04-21T09:00:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.251+0000 7efdfd7fa640 1 -- 172.21.15.28:0/1028419659 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf80189f0 con 0x7efe00106f90 2024-04-21T09:00:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.251+0000 7efe0662c640 1 -- 172.21.15.28:0/1028419659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe0006a5d0 con 0x7efe00106f90 2024-04-21T09:00:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.253+0000 7efdfd7fa640 1 -- 172.21.15.28:0/1028419659 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7efdf800fab0 con 0x7efe00106f90 2024-04-21T09:00:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efe09218740 1 Processor -- start 2024-04-21T09:00:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efe09218740 1 -- start start 2024-04-21T09:00:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efe09218740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22580 0x564844f22960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efe09218740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f22ea0 0x564844f2b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efe09218740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f2ba80 0x564844f2de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efe09218740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564844e871e0 con 0x564844f22580 2024-04-21T09:00:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efe09218740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564844e24390 con 0x564844f2ba80 2024-04-21T09:00:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efe09218740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564844e831a0 con 0x564844f22ea0 2024-04-21T09:00:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efdfd7fa640 1 --2- 172.21.15.28:0/1028419659 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7efdec0a3da0 0x7efdec0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efdff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22580 0x564844f22960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efdff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22580 0x564844f22960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34460/0 (socket says 172.21.15.28:34460) 2024-04-21T09:00:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efdff7fe640 1 -- 172.21.15.28:0/3312062959 learned_addr learned my addr 172.21.15.28:0/3312062959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efdff7fe640 1 --2- 172.21.15.28:0/3312062959 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f2ba80 0x564844f2de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efe04bc9640 1 --2- 172.21.15.28:0/1028419659 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7efdec0a3da0 0x7efdec0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efdfd7fa640 1 -- 172.21.15.28:0/1028419659 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7efdf80d1350 con 0x7efe00106f90 2024-04-21T09:00:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efdfffff640 1 --2- 172.21.15.28:0/3312062959 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f22ea0 0x564844f2b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efdff7fe640 1 -- 172.21.15.28:0/3312062959 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f22ea0 msgr2=0x564844f2b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efdff7fe640 1 --2- 172.21.15.28:0/3312062959 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f22ea0 0x564844f2b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efdff7fe640 1 -- 172.21.15.28:0/3312062959 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f2ba80 msgr2=0x564844f2de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efdff7fe640 1 --2- 172.21.15.28:0/3312062959 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f2ba80 0x564844f2de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.254+0000 7efdff7fe640 1 -- 172.21.15.28:0/3312062959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564844c637d0 con 0x564844f22580 2024-04-21T09:00:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.255+0000 7efe04bc9640 1 --2- 172.21.15.28:0/1028419659 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7efdec0a3da0 0x7efdec0a6260 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efdf80ce760 tx=0x7efdf8018400 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.255+0000 7efdff7fe640 1 --2- 172.21.15.28:0/3312062959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22580 0x564844f22960 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7efdf0005c40 tx=0x7efdf0005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7c7747f10486998 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.255+0000 7efdf67fc640 1 -- 172.21.15.28:0/3312062959 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf0048070 con 0x564844f22580 2024-04-21T09:00:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.255+0000 7efdf67fc640 1 -- 172.21.15.28:0/3312062959 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdf002c9e0 con 0x564844f22580 2024-04-21T09:00:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.255+0000 7efdf67fc640 1 -- 172.21.15.28:0/3312062959 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf0034440 con 0x564844f22580 2024-04-21T09:00:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.255+0000 7efe09218740 1 -- 172.21.15.28:0/3312062959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22580 msgr2=0x564844f22960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.255+0000 7efe09218740 1 --2- 172.21.15.28:0/3312062959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22580 0x564844f22960 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7efdf0005c40 tx=0x7efdf0005b50 comp rx=0 tx=0).stop 2024-04-21T09:00:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 1 -- 172.21.15.28:0/3312062959 shutdown_connections 2024-04-21T09:00:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 1 --2- 172.21.15.28:0/3312062959 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f2ba80 0x564844f2de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 1 --2- 172.21.15.28:0/3312062959 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f22ea0 0x564844f2b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 1 --2- 172.21.15.28:0/3312062959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22580 0x564844f22960 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 1 -- 172.21.15.28:0/3312062959 >> 172.21.15.28:0/3312062959 conn(0x564844f1ccb0 msgr2=0x564844f1d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 1 -- 172.21.15.28:0/3312062959 shutdown_connections 2024-04-21T09:00:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 1 -- 172.21.15.28:0/3312062959 wait complete. 2024-04-21T09:00:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 1 -- start start 2024-04-21T09:00:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 1 Processor -- start 2024-04-21T09:00:51.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22580 0x564844f28b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22ea0 0x564844f29090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f2ba80 0x564844f295d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564844f2e810 con 0x564844f22ea0 2024-04-21T09:00:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x564844e402f0 con 0x564844f22580 2024-04-21T09:00:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x564844f2e990 con 0x564844f2ba80 2024-04-21T09:00:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efe09218740 10 client.0 _ll_register_callbacks cb 0x564844e4d580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.256+0000 7efdfffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22ea0 0x564844f29090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efdfffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22ea0 0x564844f29090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34474/0 (socket says 172.21.15.28:34474) 2024-04-21T09:00:51.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efdfffff640 1 -- 172.21.15.28:0/2000059526 learned_addr learned my addr 172.21.15.28:0/2000059526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:51.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efdff7fe640 1 --2- 172.21.15.28:0/2000059526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f2ba80 0x564844f295d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efdff7fe640 1 --2- 172.21.15.28:0/2000059526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22580 0x564844f28b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efdfffff640 1 -- 172.21.15.28:0/2000059526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f2ba80 msgr2=0x564844f295d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efdfffff640 1 --2- 172.21.15.28:0/2000059526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f2ba80 0x564844f295d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efdfffff640 1 -- 172.21.15.28:0/2000059526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22580 msgr2=0x564844f28b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efdfffff640 1 --2- 172.21.15.28:0/2000059526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22580 0x564844f28b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efdfffff640 1 -- 172.21.15.28:0/2000059526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564844b993f0 con 0x564844f22ea0 2024-04-21T09:00:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efdff7fe640 1 --2- 172.21.15.28:0/2000059526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22580 0x564844f28b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:00:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efdfffff640 1 --2- 172.21.15.28:0/2000059526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22ea0 0x564844f29090 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7efde800b9f0 tx=0x7efde800beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de37d574ce7cb2b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efddf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efddf7fe640 1 -- 172.21.15.28:0/2000059526 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde8003e20 con 0x564844f22ea0 2024-04-21T09:00:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efddf7fe640 1 -- 172.21.15.28:0/2000059526 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7efde8011440 con 0x564844f22ea0 2024-04-21T09:00:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efe09218740 1 -- 172.21.15.28:0/2000059526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdf002c9e0 con 0x564844f22ea0 2024-04-21T09:00:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efddf7fe640 1 -- 172.21.15.28:0/2000059526 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7efde8011640 con 0x564844f22ea0 2024-04-21T09:00:51.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efe09218740 10 client.18117 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:51.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efe09218740 1 -- 172.21.15.28:0/2000059526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564844b590f0 con 0x564844f22ea0 2024-04-21T09:00:51.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.257+0000 7efddf7fe640 1 -- 172.21.15.28:0/2000059526 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdf002c9e0 con 0x564844f22ea0 2024-04-21T09:00:51.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.258+0000 7efddf7fe640 1 -- 172.21.15.28:0/2000059526 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7efde801b990 con 0x564844f22ea0 2024-04-21T09:00:51.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.258+0000 7efe09218740 10 client.18117 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:51.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.258+0000 7efe09218740 10 client.18117 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:51.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.258+0000 7efe09218740 20 client.18117 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:51.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.258+0000 7efe09218740 1 --2- 172.21.15.28:0/2000059526 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564844f3bb00 0x564844f3df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.258+0000 7efe09218740 4 client.18117 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:00:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.258+0000 7efe09218740 1 -- 172.21.15.28:0/2000059526 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564844e94c00 con 0x564844f3bb00 2024-04-21T09:00:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.258+0000 7efdff7fe640 1 --2- 172.21.15.28:0/2000059526 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564844f3bb00 0x564844f3df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.258+0000 7efddf7fe640 10 client.18117 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.258+0000 7efdff7fe640 1 --2- 172.21.15.28:0/2000059526 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564844f3bb00 0x564844f3df30 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e43479d3d1862dda server_cookie=15a57b19efd43b5f in_seq=0 out_seq=0 2024-04-21T09:00:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.259+0000 7efddf7fe640 1 -- 172.21.15.28:0/2000059526 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564844e94c00 con 0x564844f3bb00 2024-04-21T09:00:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.259+0000 7efddf7fe640 10 client.18117 handle_command_reply: tid=0 2024-04-21T09:00:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.259+0000 7efe09218740 1 client.18117 shutdown 2024-04-21T09:00:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.259+0000 7efe09218740 1 -- 172.21.15.28:0/2000059526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22ea0 msgr2=0x564844f29090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.259+0000 7efe09218740 1 --2- 172.21.15.28:0/2000059526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22ea0 0x564844f29090 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7efde800b9f0 tx=0x7efde800beb0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.259+0000 7efe09218740 1 -- 172.21.15.28:0/2000059526 shutdown_connections 2024-04-21T09:00:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.259+0000 7efe09218740 1 --2- 172.21.15.28:0/2000059526 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564844f3bb00 0x564844f3df30 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.259+0000 7efe09218740 1 --2- 172.21.15.28:0/2000059526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f2ba80 0x564844f295d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.259+0000 7efe09218740 1 --2- 172.21.15.28:0/2000059526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22ea0 0x564844f29090 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.259+0000 7efddf7fe640 0 client.18117 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.259+0000 7efe09218740 1 --2- 172.21.15.28:0/2000059526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22580 0x564844f28b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.259+0000 7efe09218740 1 -- 172.21.15.28:0/2000059526 >> 172.21.15.28:0/2000059526 conn(0x564844f1ccb0 msgr2=0x564844f1ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:51.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.259+0000 7efe09218740 1 -- 172.21.15.28:0/2000059526 shutdown_connections 2024-04-21T09:00:51.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.259+0000 7efe09218740 1 -- 172.21.15.28:0/2000059526 wait complete. 2024-04-21T09:00:51.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.259+0000 7efe09218740 20 client.18117 trim_cache size 0 max 16384 2024-04-21T09:00:51.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efe09218740 1 Processor -- start 2024-04-21T09:00:51.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efe09218740 1 -- start start 2024-04-21T09:00:51.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efe09218740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22580 0x564844f2b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efe09218740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f22ea0 0x564844f376d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efe09218740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f2ba80 0x564844f29220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efe09218740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe001b1c60 con 0x564844f2ba80 2024-04-21T09:00:51.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efe09218740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efe001af530 con 0x564844f22580 2024-04-21T09:00:51.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efe09218740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7efe001ad3a0 con 0x564844f22ea0 2024-04-21T09:00:51.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efdff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f2ba80 0x564844f29220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efdff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f2ba80 0x564844f29220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34482/0 (socket says 172.21.15.28:34482) 2024-04-21T09:00:51.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efdff7fe640 1 -- 172.21.15.28:0/613449050 learned_addr learned my addr 172.21.15.28:0/613449050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:51.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efdfffff640 1 --2- 172.21.15.28:0/613449050 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f22ea0 0x564844f376d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efdff7fe640 1 -- 172.21.15.28:0/613449050 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f22ea0 msgr2=0x564844f376d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efdff7fe640 1 --2- 172.21.15.28:0/613449050 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f22ea0 0x564844f376d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efdff7fe640 1 -- 172.21.15.28:0/613449050 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22580 msgr2=0x564844f2b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efdff7fe640 1 --2- 172.21.15.28:0/613449050 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22580 0x564844f2b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.277+0000 7efdff7fe640 1 -- 172.21.15.28:0/613449050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564844f1c4f0 con 0x564844f2ba80 2024-04-21T09:00:51.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.278+0000 7efdff7fe640 1 --2- 172.21.15.28:0/613449050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f2ba80 0x564844f29220 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7efdf0005c40 tx=0x7efdf0003c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41cfaeb14c6e5c44 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:51.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.278+0000 7efdf5ffb640 1 -- 172.21.15.28:0/613449050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf0048040 con 0x564844f2ba80 2024-04-21T09:00:51.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.278+0000 7efdf5ffb640 1 -- 172.21.15.28:0/613449050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdf0056070 con 0x564844f2ba80 2024-04-21T09:00:51.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.278+0000 7efdf5ffb640 1 -- 172.21.15.28:0/613449050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf0005040 con 0x564844f2ba80 2024-04-21T09:00:51.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.278+0000 7efe09218740 1 -- 172.21.15.28:0/613449050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f2ba80 msgr2=0x564844f29220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.278+0000 7efe09218740 1 --2- 172.21.15.28:0/613449050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f2ba80 0x564844f29220 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7efdf0005c40 tx=0x7efdf0003c90 comp rx=0 tx=0).stop 2024-04-21T09:00:51.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.278+0000 7efe09218740 1 -- 172.21.15.28:0/613449050 shutdown_connections 2024-04-21T09:00:51.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.278+0000 7efe09218740 1 --2- 172.21.15.28:0/613449050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f2ba80 0x564844f29220 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.278+0000 7efe09218740 1 --2- 172.21.15.28:0/613449050 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f22ea0 0x564844f376d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.278+0000 7efe09218740 1 --2- 172.21.15.28:0/613449050 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22580 0x564844f2b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.278+0000 7efe09218740 1 -- 172.21.15.28:0/613449050 >> 172.21.15.28:0/613449050 conn(0x564844f1ccb0 msgr2=0x564844f28b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:51.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.278+0000 7efe09218740 1 -- 172.21.15.28:0/613449050 shutdown_connections 2024-04-21T09:00:51.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.278+0000 7efe09218740 1 -- 172.21.15.28:0/613449050 wait complete. 2024-04-21T09:00:51.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.279+0000 7efe09218740 1 -- start start 2024-04-21T09:00:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.279+0000 7efe09218740 1 Processor -- start 2024-04-21T09:00:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.279+0000 7efe09218740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22580 0x564844f33e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.279+0000 7efe09218740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22ea0 0x564844f2f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.279+0000 7efe09218740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f2ba80 0x564844f2f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.279+0000 7efe09218740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe001ad9f0 con 0x564844f22580 2024-04-21T09:00:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.279+0000 7efe09218740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7efe001aee60 con 0x564844f22ea0 2024-04-21T09:00:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.279+0000 7efe09218740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7efe001ae0c0 con 0x564844f2ba80 2024-04-21T09:00:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.279+0000 7efe09218740 10 client.0 _ll_register_callbacks cb 0x7efde8004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:00:51.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.279+0000 7efdff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22580 0x564844f33e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.279+0000 7efdff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22580 0x564844f33e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34488/0 (socket says 172.21.15.28:34488) 2024-04-21T09:00:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.279+0000 7efdff7fe640 1 -- 172.21.15.28:0/1640309232 learned_addr learned my addr 172.21.15.28:0/1640309232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:00:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efdff7fe640 1 --2- 172.21.15.28:0/1640309232 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f2ba80 0x564844f2f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efdfffff640 1 --2- 172.21.15.28:0/1640309232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22ea0 0x564844f2f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efdff7fe640 1 -- 172.21.15.28:0/1640309232 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f2ba80 msgr2=0x564844f2f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efdff7fe640 1 --2- 172.21.15.28:0/1640309232 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f2ba80 0x564844f2f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efdff7fe640 1 -- 172.21.15.28:0/1640309232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22ea0 msgr2=0x564844f2f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:00:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efdff7fe640 1 --2- 172.21.15.28:0/1640309232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22ea0 0x564844f2f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efdff7fe640 1 -- 172.21.15.28:0/1640309232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564844f2fda0 con 0x564844f22580 2024-04-21T09:00:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efdfffff640 1 --2- 172.21.15.28:0/1640309232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22ea0 0x564844f2f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:00:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efdff7fe640 1 --2- 172.21.15.28:0/1640309232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22580 0x564844f33e50 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7efdf002bc70 tx=0x7efdf002c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9fde4632f51871 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:00:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efddf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:00:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efddf7fe640 1 -- 172.21.15.28:0/1640309232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf0048070 con 0x564844f22580 2024-04-21T09:00:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efe09218740 1 -- 172.21.15.28:0/1640309232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdf0056070 con 0x564844f22580 2024-04-21T09:00:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efddf7fe640 1 -- 172.21.15.28:0/1640309232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9533+0+0 (secure 0 0 0) 0x7efdf00035f0 con 0x564844f22580 2024-04-21T09:00:51.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efddf7fe640 1 -- 172.21.15.28:0/1640309232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7efdf00037f0 con 0x564844f22580 2024-04-21T09:00:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efe09218740 10 client.18123 fetch_fsmap learned FSMap version 28 2024-04-21T09:00:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efe09218740 1 -- 172.21.15.28:0/1640309232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564844fa1f70 con 0x564844f22580 2024-04-21T09:00:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.280+0000 7efddf7fe640 1 -- 172.21.15.28:0/1640309232 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdf0056070 con 0x564844f22580 2024-04-21T09:00:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.281+0000 7efddf7fe640 1 -- 172.21.15.28:0/1640309232 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7efdf00046c0 con 0x564844f22580 2024-04-21T09:00:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.281+0000 7efe09218740 10 client.18123 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:00:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.281+0000 7efe09218740 10 client.18123 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:00:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.281+0000 7efe09218740 20 client.18123 populate_metadata read hostname 'smithi028' 2024-04-21T09:00:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.281+0000 7efe09218740 1 --2- 172.21.15.28:0/1640309232 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564844fa22a0 0x564844fa4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:00:51.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.281+0000 7efe09218740 4 client.18123 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T09:00:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.281+0000 7efe09218740 1 -- 172.21.15.28:0/1640309232 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564844f23a50 con 0x564844fa22a0 2024-04-21T09:00:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.281+0000 7efdfffff640 1 --2- 172.21.15.28:0/1640309232 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564844fa22a0 0x564844fa4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:00:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.281+0000 7efdfffff640 1 --2- 172.21.15.28:0/1640309232 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564844fa22a0 0x564844fa4690 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b4ea4f7032aff6 server_cookie=c847d395260bc5ff in_seq=0 out_seq=0 2024-04-21T09:00:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.281+0000 7efddf7fe640 10 client.18123 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.281+0000 7efddf7fe640 1 -- 172.21.15.28:0/1640309232 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564844f23a50 con 0x564844fa22a0 2024-04-21T09:00:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.281+0000 7efddf7fe640 10 client.18123 handle_command_reply: tid=0 2024-04-21T09:00:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.281+0000 7efe09218740 1 client.18123 shutdown 2024-04-21T09:00:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe09218740 1 -- 172.21.15.28:0/1640309232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22580 msgr2=0x564844f33e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe09218740 1 --2- 172.21.15.28:0/1640309232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22580 0x564844f33e50 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7efdf002bc70 tx=0x7efdf002c720 comp rx=0 tx=0).stop 2024-04-21T09:00:51.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe09218740 1 -- 172.21.15.28:0/1640309232 shutdown_connections 2024-04-21T09:00:51.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe09218740 1 --2- 172.21.15.28:0/1640309232 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x564844fa22a0 0x564844fa4690 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe09218740 1 --2- 172.21.15.28:0/1640309232 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x564844f2ba80 0x564844f2f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe09218740 1 --2- 172.21.15.28:0/1640309232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564844f22ea0 0x564844f2f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe09218740 1 --2- 172.21.15.28:0/1640309232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564844f22580 0x564844f33e50 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efddf7fe640 0 client.18123 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:00:51.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe09218740 1 -- 172.21.15.28:0/1640309232 >> 172.21.15.28:0/1640309232 conn(0x564844f1ccb0 msgr2=0x564844f1bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:51.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe09218740 1 -- 172.21.15.28:0/1640309232 shutdown_connections 2024-04-21T09:00:51.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe09218740 1 -- 172.21.15.28:0/1640309232 wait complete. 2024-04-21T09:00:51.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe09218740 20 client.18123 trim_cache size 0 max 16384 2024-04-21T09:00:51.292 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T09:00:51.292 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T09:00:51.292 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-21T09:00:51.292 INFO:teuthology.orchestra.run.smithi028.stdout: "61f0fcef-e3d9-4b5e-9fb1-1cb96000a842": { 2024-04-21T09:00:51.293 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-21T09:00:51.293 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "61f0fcef-e3d9-4b5e-9fb1-1cb96000a842", 2024-04-21T09:00:51.293 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-21T09:00:51.293 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:00:51.293 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:00:51.293 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T09:00:51.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe0662c640 1 -- 172.21.15.28:0/1028419659 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7efdec0a3da0 msgr2=0x7efdec0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe0662c640 1 --2- 172.21.15.28:0/1028419659 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7efdec0a3da0 0x7efdec0a6260 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efdf80ce760 tx=0x7efdf8018400 comp rx=0 tx=0).stop 2024-04-21T09:00:51.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe0662c640 1 -- 172.21.15.28:0/1028419659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe00106f90 msgr2=0x7efe0019bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:00:51.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe0662c640 1 --2- 172.21.15.28:0/1028419659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe00106f90 0x7efe0019bea0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7efdf800cab0 tx=0x7efdf800cf70 comp rx=0 tx=0).stop 2024-04-21T09:00:51.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe0662c640 1 -- 172.21.15.28:0/1028419659 shutdown_connections 2024-04-21T09:00:51.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe0662c640 1 --2- 172.21.15.28:0/1028419659 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7efdec0a3da0 0x7efdec0a6260 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe0662c640 1 --2- 172.21.15.28:0/1028419659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe00106f90 0x7efe0019bea0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe0662c640 1 --2- 172.21.15.28:0/1028419659 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe00101b40 0x7efe00197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe0662c640 1 --2- 172.21.15.28:0/1028419659 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efe001011a0 0x7efe001975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:00:51.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe0662c640 1 -- 172.21.15.28:0/1028419659 >> 172.21.15.28:0/1028419659 conn(0x7efe00076c50 msgr2=0x7efe000fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:00:51.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe0662c640 1 -- 172.21.15.28:0/1028419659 shutdown_connections 2024-04-21T09:00:51.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:00:51.282+0000 7efe0662c640 1 -- 172.21.15.28:0/1028419659 wait complete. 2024-04-21T09:00:51.296 INFO:tasks.cephfs.filesystem:scrub status for tag:61f0fcef-e3d9-4b5e-9fb1-1cb96000a842 - {'path': '/', 'tag': '61f0fcef-e3d9-4b5e-9fb1-1cb96000a842', 'options': 'recursive,force'} 2024-04-21T09:00:52.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: pgmap v299: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T09:00:52.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub queued for path: / 2024-04-21T09:00:52.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:52.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:52.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:52.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:52.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:52.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:52.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:52.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:52.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:52.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: active paths [/] 2024-04-21T09:00:52.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.608 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:52 smithi152 ceph-mon[31401]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: pgmap v299: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T09:00:52.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub queued for path: / 2024-04-21T09:00:52.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:52.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:52.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:52.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:52.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:52.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:52.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:52.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:52.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:52.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: active paths [/] 2024-04-21T09:00:52.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:52 smithi028 ceph-mon[28003]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: pgmap v299: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T09:00:52.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub queued for path: / 2024-04-21T09:00:52.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:52.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:52.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:52.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:52.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:52.621 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:52.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:52.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:52.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:52.622 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: active paths [/] 2024-04-21T09:00:52.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:52.623 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:52 smithi171 ceph-mon[31358]: scrub summary: idle+waiting paths [/] 2024-04-21T09:00:54.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:54 smithi152 ceph-mon[31401]: pgmap v300: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 16 op/s 2024-04-21T09:00:54.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:54 smithi028 ceph-mon[28003]: pgmap v300: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 16 op/s 2024-04-21T09:00:54.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:54 smithi171 ceph-mon[31358]: pgmap v300: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 16 op/s 2024-04-21T09:00:56.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:56 smithi152 ceph-mon[31401]: pgmap v301: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 27 op/s 2024-04-21T09:00:56.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:56 smithi028 ceph-mon[28003]: pgmap v301: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 27 op/s 2024-04-21T09:00:56.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:56 smithi171 ceph-mon[31358]: pgmap v301: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 27 op/s 2024-04-21T09:00:57.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:57 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T09:00:57.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:57 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T09:00:57.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:57 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T09:00:57.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:57 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T09:00:57.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:57 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T09:00:57.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:57 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T09:00:58.506 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:58 smithi028 ceph-mon[28003]: scrub summary: idle 2024-04-21T09:00:58.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:00:58 smithi028 ceph-mon[28003]: pgmap v302: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 29 op/s 2024-04-21T09:00:58.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:58 smithi152 ceph-mon[31401]: scrub summary: idle 2024-04-21T09:00:58.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:00:58 smithi152 ceph-mon[31401]: pgmap v302: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 29 op/s 2024-04-21T09:00:58.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:58 smithi171 ceph-mon[31358]: scrub summary: idle 2024-04-21T09:00:58.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:00:58 smithi171 ceph-mon[31358]: pgmap v302: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 29 op/s 2024-04-21T09:01:00.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:00 smithi152 ceph-mon[31401]: pgmap v303: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 29 op/s 2024-04-21T09:01:00.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:00 smithi028 ceph-mon[28003]: pgmap v303: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 29 op/s 2024-04-21T09:01:00.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:00 smithi171 ceph-mon[31358]: pgmap v303: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 29 op/s 2024-04-21T09:01:02.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:02 smithi152 ceph-mon[31401]: pgmap v304: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 op/s 2024-04-21T09:01:02.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:02 smithi028 ceph-mon[28003]: pgmap v304: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 op/s 2024-04-21T09:01:02.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:02 smithi171 ceph-mon[31358]: pgmap v304: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 27 op/s 2024-04-21T09:01:04.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:04 smithi152 ceph-mon[31401]: pgmap v305: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 32 op/s 2024-04-21T09:01:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:04 smithi028 ceph-mon[28003]: pgmap v305: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 32 op/s 2024-04-21T09:01:04.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:04 smithi171 ceph-mon[31358]: pgmap v305: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 23 KiB/s wr, 32 op/s 2024-04-21T09:01:06.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:06 smithi152 ceph-mon[31401]: pgmap v306: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 44 KiB/s wr, 36 op/s 2024-04-21T09:01:06.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:06 smithi028 ceph-mon[28003]: pgmap v306: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 44 KiB/s wr, 36 op/s 2024-04-21T09:01:06.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:06 smithi171 ceph-mon[31358]: pgmap v306: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 44 KiB/s wr, 36 op/s 2024-04-21T09:01:08.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:08 smithi152 ceph-mon[31401]: pgmap v307: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T09:01:08.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:08 smithi152 ceph-mon[31401]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T09:01:08.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:08 smithi028 ceph-mon[28003]: pgmap v307: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T09:01:08.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:08 smithi028 ceph-mon[28003]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T09:01:08.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:08 smithi171 ceph-mon[31358]: pgmap v307: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-04-21T09:01:08.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:08 smithi171 ceph-mon[31358]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T09:01:10.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:10 smithi028 ceph-mon[28003]: pgmap v309: 129 pgs: 12 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 79 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T09:01:10.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:10 smithi152 ceph-mon[31401]: pgmap v309: 129 pgs: 12 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 79 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T09:01:10.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:10 smithi171 ceph-mon[31358]: pgmap v309: 129 pgs: 12 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 79 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T09:01:12.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:12 smithi152 ceph-mon[31401]: pgmap v310: 129 pgs: 13 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 78 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T09:01:12.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:12 smithi028 ceph-mon[28003]: pgmap v310: 129 pgs: 13 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 78 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T09:01:12.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:12 smithi171 ceph-mon[31358]: pgmap v310: 129 pgs: 13 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 78 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-04-21T09:01:14.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:14 smithi152 ceph-mon[31401]: pgmap v311: 129 pgs: 6 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 109 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-04-21T09:01:14.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:14 smithi028 ceph-mon[28003]: pgmap v311: 129 pgs: 6 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 109 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-04-21T09:01:14.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:14 smithi171 ceph-mon[31358]: pgmap v311: 129 pgs: 6 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 109 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-04-21T09:01:16.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:16 smithi152 ceph-mon[31401]: pgmap v312: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T09:01:16.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:16 smithi028 ceph-mon[28003]: pgmap v312: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T09:01:16.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:16 smithi171 ceph-mon[31358]: pgmap v312: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T09:01:18.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:18 smithi152 ceph-mon[31401]: pgmap v313: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T09:01:18.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:18 smithi028 ceph-mon[28003]: pgmap v313: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T09:01:18.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:18 smithi171 ceph-mon[31358]: pgmap v313: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-21T09:01:20.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:19 smithi152 ceph-mon[31401]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T09:01:20.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:19 smithi152 ceph-mon[31401]: pgmap v315: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T09:01:20.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:19 smithi028 ceph-mon[28003]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T09:01:20.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:19 smithi028 ceph-mon[28003]: pgmap v315: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T09:01:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:19 smithi171 ceph-mon[31358]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T09:01:20.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:19 smithi171 ceph-mon[31358]: pgmap v315: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T09:01:20.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-21T09:01:20.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.632+0000 7f3b9acc1640 1 -- 172.21.15.28:0/3266067586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b94106170 msgr2=0x7f3b9410dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.632+0000 7f3b9acc1640 1 --2- 172.21.15.28:0/3266067586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b94106170 0x7f3b9410dd40 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c009ca0 tx=0x7f3b8c02c0f0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.632+0000 7f3b9acc1640 1 -- 172.21.15.28:0/3266067586 shutdown_connections 2024-04-21T09:01:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.632+0000 7f3b9acc1640 1 --2- 172.21.15.28:0/3266067586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b94106170 0x7f3b9410dd40 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.632+0000 7f3b9acc1640 1 --2- 172.21.15.28:0/3266067586 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3b94101aa0 0x7f3b94105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.632+0000 7f3b9acc1640 1 --2- 172.21.15.28:0/3266067586 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b941010f0 0x7f3b941014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.632+0000 7f3b9acc1640 1 -- 172.21.15.28:0/3266067586 >> 172.21.15.28:0/3266067586 conn(0x7f3b94077fd0 msgr2=0x7f3b940783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.632+0000 7f3b9acc1640 1 -- 172.21.15.28:0/3266067586 shutdown_connections 2024-04-21T09:01:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.632+0000 7f3b9acc1640 1 -- 172.21.15.28:0/3266067586 wait complete. 2024-04-21T09:01:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.633+0000 7f3b9acc1640 1 Processor -- start 2024-04-21T09:01:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.633+0000 7f3b9acc1640 1 -- start start 2024-04-21T09:01:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.633+0000 7f3b9acc1640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3b941010f0 0x7f3b94102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.633+0000 7f3b9acc1640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b94106170 0x7f3b94103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.633+0000 7f3b9acc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b941035b0 0x7f3b941a68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.633+0000 7f3b9acc1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b941110e0 con 0x7f3b941035b0 2024-04-21T09:01:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.633+0000 7f3b9acc1640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3b94110f60 con 0x7f3b94106170 2024-04-21T09:01:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.633+0000 7f3b9acc1640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3b94111260 con 0x7f3b941010f0 2024-04-21T09:01:20.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.633+0000 7f3b9925e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b941035b0 0x7f3b941a68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:20.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.633+0000 7f3b9925e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b941035b0 0x7f3b941a68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47040/0 (socket says 172.21.15.28:47040) 2024-04-21T09:01:20.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.633+0000 7f3b9925e640 1 -- 172.21.15.28:0/732393161 learned_addr learned my addr 172.21.15.28:0/732393161 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:20.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.633+0000 7f3b93fff640 1 --2- 172.21.15.28:0/732393161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b94106170 0x7f3b94103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:20.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.633+0000 7f3b98a5d640 1 --2- 172.21.15.28:0/732393161 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3b941010f0 0x7f3b94102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:20.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.634+0000 7f3b9925e640 1 -- 172.21.15.28:0/732393161 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3b941010f0 msgr2=0x7f3b94102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:20.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.634+0000 7f3b9925e640 1 --2- 172.21.15.28:0/732393161 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3b941010f0 0x7f3b94102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.634+0000 7f3b9925e640 1 -- 172.21.15.28:0/732393161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b94106170 msgr2=0x7f3b94103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:20.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.634+0000 7f3b9925e640 1 --2- 172.21.15.28:0/732393161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b94106170 0x7f3b94103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.634+0000 7f3b9925e640 1 -- 172.21.15.28:0/732393161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b941a6f40 con 0x7f3b941035b0 2024-04-21T09:01:20.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.634+0000 7f3b93fff640 1 --2- 172.21.15.28:0/732393161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b94106170 0x7f3b94103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:01:20.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.634+0000 7f3b9925e640 1 --2- 172.21.15.28:0/732393161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b941035b0 0x7f3b941a68a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0098d0 tx=0x7f3b8c035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:20.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.634+0000 7f3b91ffb640 1 -- 172.21.15.28:0/732393161 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c039040 con 0x7f3b941035b0 2024-04-21T09:01:20.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.634+0000 7f3b91ffb640 1 -- 172.21.15.28:0/732393161 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b8c044070 con 0x7f3b941035b0 2024-04-21T09:01:20.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.634+0000 7f3b9acc1640 1 -- 172.21.15.28:0/732393161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b941a71d0 con 0x7f3b941035b0 2024-04-21T09:01:20.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.634+0000 7f3b91ffb640 1 -- 172.21.15.28:0/732393161 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c0042a0 con 0x7f3b941035b0 2024-04-21T09:01:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.634+0000 7f3b9acc1640 1 -- 172.21.15.28:0/732393161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b941a7710 con 0x7f3b941035b0 2024-04-21T09:01:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.635+0000 7f3b9acc1640 1 -- 172.21.15.28:0/732393161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b9404fce0 con 0x7f3b941035b0 2024-04-21T09:01:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.636+0000 7f3b91ffb640 1 -- 172.21.15.28:0/732393161 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3b8c004440 con 0x7f3b941035b0 2024-04-21T09:01:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.637+0000 7f3b91ffb640 1 --2- 172.21.15.28:0/732393161 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3b7c0a3e40 0x7f3b7c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.637+0000 7f3b98a5d640 1 --2- 172.21.15.28:0/732393161 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3b7c0a3e40 0x7f3b7c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.637+0000 7f3b91ffb640 1 -- 172.21.15.28:0/732393161 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f3b8c0f2fe0 con 0x7f3b941035b0 2024-04-21T09:01:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.638+0000 7f3b98a5d640 1 --2- 172.21.15.28:0/732393161 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3b7c0a3e40 0x7f3b7c0a6300 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3b84005e10 tx=0x7f3b84005da0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:20.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.642+0000 7f3b91ffb640 1 -- 172.21.15.28:0/732393161 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b8c0bf3a0 con 0x7f3b941035b0 2024-04-21T09:01:20.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.849+0000 7f3b9acc1640 1 -- 172.21.15.28:0/732393161 --> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f3b94101aa0 con 0x7f3b7c0a3e40 2024-04-21T09:01:20.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.857+0000 7f3b91ffb640 1 -- 172.21.15.28:0/732393161 <== mgr.25040 v2:172.21.15.28:6800/4182301910 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f3b94101aa0 con 0x7f3b7c0a3e40 2024-04-21T09:01:20.858 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-21T00:00:00", "created": "2024-04-21T08:51:19", "first": "2024-04-21T08:52:00", "last": "2024-04-21T09:01:00", "last_pruned": "2024-04-21T09:01:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-04-21T09:01:20.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.859+0000 7f3b9acc1640 1 -- 172.21.15.28:0/732393161 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3b7c0a3e40 msgr2=0x7f3b7c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:20.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.859+0000 7f3b9acc1640 1 --2- 172.21.15.28:0/732393161 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3b7c0a3e40 0x7f3b7c0a6300 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3b84005e10 tx=0x7f3b84005da0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.859+0000 7f3b9acc1640 1 -- 172.21.15.28:0/732393161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b941035b0 msgr2=0x7f3b941a68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:20.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.859+0000 7f3b9acc1640 1 --2- 172.21.15.28:0/732393161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b941035b0 0x7f3b941a68a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0098d0 tx=0x7f3b8c035ce0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.859+0000 7f3b9acc1640 1 -- 172.21.15.28:0/732393161 shutdown_connections 2024-04-21T09:01:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.859+0000 7f3b9acc1640 1 --2- 172.21.15.28:0/732393161 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3b7c0a3e40 0x7f3b7c0a6300 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.859+0000 7f3b9acc1640 1 --2- 172.21.15.28:0/732393161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b941035b0 0x7f3b941a68a0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.859+0000 7f3b9acc1640 1 --2- 172.21.15.28:0/732393161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b94106170 0x7f3b94103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.859+0000 7f3b9acc1640 1 --2- 172.21.15.28:0/732393161 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3b941010f0 0x7f3b94102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.859+0000 7f3b9acc1640 1 -- 172.21.15.28:0/732393161 >> 172.21.15.28:0/732393161 conn(0x7f3b94077fd0 msgr2=0x7f3b94069910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.859+0000 7f3b9acc1640 1 -- 172.21.15.28:0/732393161 shutdown_connections 2024-04-21T09:01:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.859+0000 7f3b9acc1640 1 -- 172.21.15.28:0/732393161 wait complete. 2024-04-21T09:01:20.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-21T09:01:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.974+0000 7f2fa9621640 1 -- 172.21.15.28:0/2004416370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa410c5d0 msgr2=0x7f2fa410ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.974+0000 7f2fa9621640 1 --2- 172.21.15.28:0/2004416370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa410c5d0 0x7f2fa410ea00 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009ca0 tx=0x7f2f9402c180 comp rx=0 tx=0).stop 2024-04-21T09:01:20.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.974+0000 7f2fa9621640 1 -- 172.21.15.28:0/2004416370 shutdown_connections 2024-04-21T09:01:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.974+0000 7f2fa9621640 1 --2- 172.21.15.28:0/2004416370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa410c5d0 0x7f2fa410ea00 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.974+0000 7f2fa9621640 1 --2- 172.21.15.28:0/2004416370 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fa4103b20 0x7f2fa410c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.974+0000 7f2fa9621640 1 --2- 172.21.15.28:0/2004416370 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2fa4103200 0x7f2fa41035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.974+0000 7f2fa9621640 1 -- 172.21.15.28:0/2004416370 >> 172.21.15.28:0/2004416370 conn(0x7f2fa40771d0 msgr2=0x7f2fa40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.974+0000 7f2fa9621640 1 -- 172.21.15.28:0/2004416370 shutdown_connections 2024-04-21T09:01:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.974+0000 7f2fa9621640 1 -- 172.21.15.28:0/2004416370 wait complete. 2024-04-21T09:01:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.974+0000 7f2fa9621640 1 Processor -- start 2024-04-21T09:01:20.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.974+0000 7f2fa9621640 1 -- start start 2024-04-21T09:01:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa9621640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa4103200 0x7f2fa4100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa9621640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2fa4103b20 0x7f2fa4101390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa9621640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fa410c5d0 0x7f2fa41018d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa9621640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa4112250 con 0x7f2fa4103200 2024-04-21T09:01:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa9621640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa41120d0 con 0x7f2fa4103b20 2024-04-21T09:01:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa9621640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa41123d0 con 0x7f2fa410c5d0 2024-04-21T09:01:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa4103200 0x7f2fa4100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:20.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa4103200 0x7f2fa4100e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47058/0 (socket says 172.21.15.28:47058) 2024-04-21T09:01:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa2ffd640 1 -- 172.21.15.28:0/1048776095 learned_addr learned my addr 172.21.15.28:0/1048776095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa27fc640 1 --2- 172.21.15.28:0/1048776095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2fa4103b20 0x7f2fa4101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa37fe640 1 --2- 172.21.15.28:0/1048776095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fa410c5d0 0x7f2fa41018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa2ffd640 1 -- 172.21.15.28:0/1048776095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fa410c5d0 msgr2=0x7f2fa41018d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa2ffd640 1 --2- 172.21.15.28:0/1048776095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fa410c5d0 0x7f2fa41018d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa2ffd640 1 -- 172.21.15.28:0/1048776095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2fa4103b20 msgr2=0x7f2fa4101390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa2ffd640 1 --2- 172.21.15.28:0/1048776095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2fa4103b20 0x7f2fa4101390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa2ffd640 1 -- 172.21.15.28:0/1048776095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa41a6bb0 con 0x7f2fa4103200 2024-04-21T09:01:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa37fe640 1 --2- 172.21.15.28:0/1048776095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fa410c5d0 0x7f2fa41018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:01:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2fa2ffd640 1 --2- 172.21.15.28:0/1048776095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa4103200 0x7f2fa4100e50 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00d950 tx=0x7f2f9c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2f9bfff640 1 -- 172.21.15.28:0/1048776095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c017070 con 0x7f2fa4103200 2024-04-21T09:01:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.975+0000 7f2f9bfff640 1 -- 172.21.15.28:0/1048776095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f9c009c30 con 0x7f2fa4103200 2024-04-21T09:01:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.976+0000 7f2fa9621640 1 -- 172.21.15.28:0/1048776095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa41a6ea0 con 0x7f2fa4103200 2024-04-21T09:01:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.976+0000 7f2f9bfff640 1 -- 172.21.15.28:0/1048776095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c00f620 con 0x7f2fa4103200 2024-04-21T09:01:20.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.976+0000 7f2fa9621640 1 -- 172.21.15.28:0/1048776095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa41a7360 con 0x7f2fa4103200 2024-04-21T09:01:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.977+0000 7f2fa9621640 1 -- 172.21.15.28:0/1048776095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa404fce0 con 0x7f2fa4103200 2024-04-21T09:01:20.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.978+0000 7f2f9bfff640 1 -- 172.21.15.28:0/1048776095 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2f9c010460 con 0x7f2fa4103200 2024-04-21T09:01:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.978+0000 7f2f9bfff640 1 --2- 172.21.15.28:0/1048776095 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2f900a3dc0 0x7f2f900a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.979+0000 7f2f9bfff640 1 -- 172.21.15.28:0/1048776095 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f2f9c0d0920 con 0x7f2fa4103200 2024-04-21T09:01:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.979+0000 7f2fa27fc640 1 --2- 172.21.15.28:0/1048776095 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2f900a3dc0 0x7f2f900a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.979+0000 7f2fa27fc640 1 --2- 172.21.15.28:0/1048776095 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2f900a3dc0 0x7f2f900a6280 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2fa41028e0 tx=0x7f2f8c008040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:20.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:20.983+0000 7f2f9bfff640 1 -- 172.21.15.28:0/1048776095 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f9c09cb70 con 0x7f2fa4103200 2024-04-21T09:01:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.193+0000 7f2fa9621640 1 -- 172.21.15.28:0/1048776095 --> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f2fa40fe390 con 0x7f2f900a3dc0 2024-04-21T09:01:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.201+0000 7f2f9bfff640 1 -- 172.21.15.28:0/1048776095 <== mgr.25040 v2:172.21.15.28:6800/4182301910 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f2fa40fe390 con 0x7f2f900a3dc0 2024-04-21T09:01:21.202 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-04-21T09:01:21.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.203+0000 7f2fa9621640 1 -- 172.21.15.28:0/1048776095 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2f900a3dc0 msgr2=0x7f2f900a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.203+0000 7f2fa9621640 1 --2- 172.21.15.28:0/1048776095 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2f900a3dc0 0x7f2f900a6280 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2fa41028e0 tx=0x7f2f8c008040 comp rx=0 tx=0).stop 2024-04-21T09:01:21.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.203+0000 7f2fa9621640 1 -- 172.21.15.28:0/1048776095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa4103200 msgr2=0x7f2fa4100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.203+0000 7f2fa9621640 1 --2- 172.21.15.28:0/1048776095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa4103200 0x7f2fa4100e50 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00d950 tx=0x7f2f9c00de10 comp rx=0 tx=0).stop 2024-04-21T09:01:21.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.203+0000 7f2fa9621640 1 -- 172.21.15.28:0/1048776095 shutdown_connections 2024-04-21T09:01:21.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.203+0000 7f2fa9621640 1 --2- 172.21.15.28:0/1048776095 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2f900a3dc0 0x7f2f900a6280 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.203+0000 7f2fa9621640 1 --2- 172.21.15.28:0/1048776095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2fa410c5d0 0x7f2fa41018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.203+0000 7f2fa9621640 1 --2- 172.21.15.28:0/1048776095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2fa4103b20 0x7f2fa4101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.203+0000 7f2fa9621640 1 --2- 172.21.15.28:0/1048776095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fa4103200 0x7f2fa4100e50 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.203+0000 7f2fa9621640 1 -- 172.21.15.28:0/1048776095 >> 172.21.15.28:0/1048776095 conn(0x7f2fa40771d0 msgr2=0x7f2fa40fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:21.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.203+0000 7f2fa9621640 1 -- 172.21.15.28:0/1048776095 shutdown_connections 2024-04-21T09:01:21.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.203+0000 7f2fa9621640 1 -- 172.21.15.28:0/1048776095 wait complete. 2024-04-21T09:01:21.215 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-21T09:01:21.225 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-21T09:01:21.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T09:01:21.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T09:01:21.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.401+0000 7f95bc7f4640 1 -- 172.21.15.28:0/4101849103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b4102b50 msgr2=0x7f95b4106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.401+0000 7f95bc7f4640 1 --2- 172.21.15.28:0/4101849103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b4102b50 0x7f95b4106b90 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f95a4009900 tx=0x7f95a402bc30 comp rx=0 tx=0).stop 2024-04-21T09:01:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.402+0000 7f95bc7f4640 1 -- 172.21.15.28:0/4101849103 shutdown_connections 2024-04-21T09:01:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.402+0000 7f95bc7f4640 1 --2- 172.21.15.28:0/4101849103 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f95b4107270 0x7f95b410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.402+0000 7f95bc7f4640 1 --2- 172.21.15.28:0/4101849103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b4102b50 0x7f95b4106b90 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.402+0000 7f95bc7f4640 1 --2- 172.21.15.28:0/4101849103 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95b41021a0 0x7f95b4102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.402+0000 7f95bc7f4640 1 -- 172.21.15.28:0/4101849103 >> 172.21.15.28:0/4101849103 conn(0x7f95b40fda40 msgr2=0x7f95b40ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.402+0000 7f95bc7f4640 1 -- 172.21.15.28:0/4101849103 shutdown_connections 2024-04-21T09:01:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.402+0000 7f95bc7f4640 1 -- 172.21.15.28:0/4101849103 wait complete. 2024-04-21T09:01:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.402+0000 7f95bc7f4640 1 Processor -- start 2024-04-21T09:01:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.402+0000 7f95bc7f4640 1 -- start start 2024-04-21T09:01:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.402+0000 7f95bc7f4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95b41021a0 0x7f95b419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.402+0000 7f95bc7f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b4107270 0x7f95b419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.402+0000 7f95bc7f4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f95b4198670 0x7f95b4198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.402+0000 7f95bc7f4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f95b41121b0 con 0x7f95b4107270 2024-04-21T09:01:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.402+0000 7f95bc7f4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f95b4112030 con 0x7f95b4198670 2024-04-21T09:01:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.402+0000 7f95bc7f4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f95b4112330 con 0x7f95b41021a0 2024-04-21T09:01:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95b9d8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b4107270 0x7f95b419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95b9d8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b4107270 0x7f95b419e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47076/0 (socket says 172.21.15.28:47076) 2024-04-21T09:01:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95b9d8f640 1 -- 172.21.15.28:0/3207669515 learned_addr learned my addr 172.21.15.28:0/3207669515 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95ba590640 1 --2- 172.21.15.28:0/3207669515 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95b41021a0 0x7f95b419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95bad91640 1 --2- 172.21.15.28:0/3207669515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f95b4198670 0x7f95b4198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95b9d8f640 1 -- 172.21.15.28:0/3207669515 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95b41021a0 msgr2=0x7f95b419e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95b9d8f640 1 --2- 172.21.15.28:0/3207669515 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95b41021a0 0x7f95b419e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95b9d8f640 1 -- 172.21.15.28:0/3207669515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f95b4198670 msgr2=0x7f95b4198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95b9d8f640 1 --2- 172.21.15.28:0/3207669515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f95b4198670 0x7f95b4198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95b9d8f640 1 -- 172.21.15.28:0/3207669515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b4199190 con 0x7f95b4107270 2024-04-21T09:01:21.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95b9d8f640 1 --2- 172.21.15.28:0/3207669515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b4107270 0x7f95b419e9b0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f95a4036040 tx=0x7f95a40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:21.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95ab7fe640 1 -- 172.21.15.28:0/3207669515 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a4043020 con 0x7f95b4107270 2024-04-21T09:01:21.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95bc7f4640 1 -- 172.21.15.28:0/3207669515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95b4199420 con 0x7f95b4107270 2024-04-21T09:01:21.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95ab7fe640 1 -- 172.21.15.28:0/3207669515 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95a402cb70 con 0x7f95b4107270 2024-04-21T09:01:21.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95ab7fe640 1 -- 172.21.15.28:0/3207669515 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a4003bf0 con 0x7f95b4107270 2024-04-21T09:01:21.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.403+0000 7f95bc7f4640 1 -- 172.21.15.28:0/3207669515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95b41a5620 con 0x7f95b4107270 2024-04-21T09:01:21.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.405+0000 7f95ab7fe640 1 -- 172.21.15.28:0/3207669515 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f95a4046020 con 0x7f95b4107270 2024-04-21T09:01:21.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.406+0000 7f95bf3dd740 1 Processor -- start 2024-04-21T09:01:21.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.406+0000 7f95bf3dd740 1 -- start start 2024-04-21T09:01:21.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.406+0000 7f95ab7fe640 1 --2- 172.21.15.28:0/3207669515 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95a00a3d20 0x7f95a00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.406+0000 7f95bf3dd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 0x559b7943a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.406+0000 7f95bf3dd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943abd0 0x559b79447950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.406+0000 7f95bf3dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b79447e90 0x559b7944a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.406+0000 7f95bf3dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559b7939f1e0 con 0x559b79447e90 2024-04-21T09:01:21.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.406+0000 7f95bf3dd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x559b7933c390 con 0x559b7943abd0 2024-04-21T09:01:21.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.406+0000 7f95bf3dd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x559b7939b1a0 con 0x559b7943a2b0 2024-04-21T09:01:21.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.406+0000 7f95bad91640 1 --2- 172.21.15.28:0/3207669515 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95a00a3d20 0x7f95a00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.406+0000 7f95ab7fe640 1 -- 172.21.15.28:0/3207669515 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f95a40f2cc0 con 0x7f95b4107270 2024-04-21T09:01:21.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.406+0000 7f95bad91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b79447e90 0x559b7944a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.406+0000 7f95bad91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b79447e90 0x559b7944a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47080/0 (socket says 172.21.15.28:47080) 2024-04-21T09:01:21.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.406+0000 7f95bad91640 1 -- 172.21.15.28:0/1676620175 learned_addr learned my addr 172.21.15.28:0/1676620175 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:21.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.407+0000 7f95ba590640 1 --2- 172.21.15.28:0/1676620175 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 0x559b7943a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.407+0000 7f95b9d8f640 1 --2- 172.21.15.28:0/1676620175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943abd0 0x559b79447950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.407+0000 7f95bad91640 1 -- 172.21.15.28:0/1676620175 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 msgr2=0x559b7943a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.407+0000 7f95bad91640 1 --2- 172.21.15.28:0/1676620175 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 0x559b7943a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.407+0000 7f95bad91640 1 -- 172.21.15.28:0/1676620175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943abd0 msgr2=0x559b79447950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.407+0000 7f95bad91640 1 --2- 172.21.15.28:0/1676620175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943abd0 0x559b79447950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.407+0000 7f95bad91640 1 -- 172.21.15.28:0/1676620175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b7917b7d0 con 0x559b79447e90 2024-04-21T09:01:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.407+0000 7f95b9d8f640 1 --2- 172.21.15.28:0/1676620175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943abd0 0x559b79447950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:01:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.407+0000 7f95bad91640 1 --2- 172.21.15.28:0/1676620175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b79447e90 0x559b7944a280 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f95ac014c60 tx=0x7f95ac0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f23e63f973355d3d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:21.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.407+0000 7f95bad91640 1 --2- 172.21.15.28:0/3207669515 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95a00a3d20 0x7f95a00a61e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f95ac011cb0 tx=0x7f95ac016000 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:21.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.407+0000 7f95a8ff9640 1 -- 172.21.15.28:0/1676620175 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac022070 con 0x559b79447e90 2024-04-21T09:01:21.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.407+0000 7f95a8ff9640 1 -- 172.21.15.28:0/1676620175 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95ac0054d0 con 0x559b79447e90 2024-04-21T09:01:21.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.407+0000 7f95a8ff9640 1 -- 172.21.15.28:0/1676620175 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac004a00 con 0x559b79447e90 2024-04-21T09:01:21.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.408+0000 7f95bf3dd740 1 -- 172.21.15.28:0/1676620175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b79447e90 msgr2=0x559b7944a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.408+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/1676620175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b79447e90 0x559b7944a280 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f95ac014c60 tx=0x7f95ac0117b0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.408+0000 7f95bf3dd740 1 -- 172.21.15.28:0/1676620175 shutdown_connections 2024-04-21T09:01:21.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.408+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/1676620175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b79447e90 0x559b7944a280 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.408+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/1676620175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943abd0 0x559b79447950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.408+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/1676620175 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 0x559b7943a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.408+0000 7f95bf3dd740 1 -- 172.21.15.28:0/1676620175 >> 172.21.15.28:0/1676620175 conn(0x559b79434cb0 msgr2=0x559b79435090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.408+0000 7f95bf3dd740 1 -- 172.21.15.28:0/1676620175 shutdown_connections 2024-04-21T09:01:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.408+0000 7f95bf3dd740 1 -- 172.21.15.28:0/1676620175 wait complete. 2024-04-21T09:01:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.408+0000 7f95bf3dd740 1 -- start start 2024-04-21T09:01:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.408+0000 7f95bf3dd740 1 Processor -- start 2024-04-21T09:01:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95bf3dd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943a2b0 0x559b79444fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95bf3dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b7943abd0 0x559b794454e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95bf3dd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b79447e90 0x559b79445a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95bf3dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559b7944abf0 con 0x559b7943abd0 2024-04-21T09:01:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95bf3dd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x559b793582f0 con 0x559b7943a2b0 2024-04-21T09:01:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95bf3dd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x559b7944ad70 con 0x559b79447e90 2024-04-21T09:01:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95bf3dd740 10 client.0 _ll_register_callbacks cb 0x559b79365580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:01:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95b9d8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b7943abd0 0x559b794454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95b9d8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b7943abd0 0x559b794454e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47092/0 (socket says 172.21.15.28:47092) 2024-04-21T09:01:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95b9d8f640 1 -- 172.21.15.28:0/2615440628 learned_addr learned my addr 172.21.15.28:0/2615440628 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:21.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95ba590640 1 --2- 172.21.15.28:0/2615440628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943a2b0 0x559b79444fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95bad91640 1 --2- 172.21.15.28:0/2615440628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b79447e90 0x559b79445a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95b9d8f640 1 -- 172.21.15.28:0/2615440628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b79447e90 msgr2=0x559b79445a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95b9d8f640 1 --2- 172.21.15.28:0/2615440628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b79447e90 0x559b79445a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95b9d8f640 1 -- 172.21.15.28:0/2615440628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943a2b0 msgr2=0x559b79444fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95b9d8f640 1 --2- 172.21.15.28:0/2615440628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943a2b0 0x559b79444fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95b9d8f640 1 -- 172.21.15.28:0/2615440628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b790b13f0 con 0x559b7943abd0 2024-04-21T09:01:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95ba590640 1 --2- 172.21.15.28:0/2615440628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943a2b0 0x559b79444fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:01:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95b9d8f640 1 --2- 172.21.15.28:0/2615440628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b7943abd0 0x559b794454e0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f95a4006010 tx=0x7f95a40be280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22f3ac9acbfe54d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f959dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:01:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f959dffb640 1 -- 172.21.15.28:0/2615440628 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a40a7e60 con 0x559b7943abd0 2024-04-21T09:01:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f959dffb640 1 -- 172.21.15.28:0/2615440628 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f95a40054d0 con 0x559b7943abd0 2024-04-21T09:01:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2615440628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95ac0054d0 con 0x559b7943abd0 2024-04-21T09:01:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f959dffb640 1 -- 172.21.15.28:0/2615440628 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f95a409b050 con 0x559b7943abd0 2024-04-21T09:01:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95bf3dd740 10 client.18147 fetch_fsmap learned FSMap version 28 2024-04-21T09:01:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.409+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2615440628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559b790710f0 con 0x559b7943abd0 2024-04-21T09:01:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.410+0000 7f959dffb640 1 -- 172.21.15.28:0/2615440628 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95ac0054d0 con 0x559b7943abd0 2024-04-21T09:01:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.410+0000 7f959dffb640 1 -- 172.21.15.28:0/2615440628 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f95a40baf90 con 0x559b7943abd0 2024-04-21T09:01:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.410+0000 7f95bf3dd740 10 client.18147 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:01:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.410+0000 7f95bf3dd740 10 client.18147 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:01:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.410+0000 7f95bf3dd740 20 client.18147 populate_metadata read hostname 'smithi028' 2024-04-21T09:01:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.410+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/2615440628 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559b79457d60 0x559b7945a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.410+0000 7f95bf3dd740 4 client.18147 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:01:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.410+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2615440628 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b793acc00 con 0x559b79457d60 2024-04-21T09:01:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.410+0000 7f95ba590640 1 --2- 172.21.15.28:0/2615440628 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559b79457d60 0x559b7945a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.411+0000 7f959dffb640 10 client.18147 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:01:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.411+0000 7f95ba590640 1 --2- 172.21.15.28:0/2615440628 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559b79457d60 0x559b7945a1b0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2614a6e0135d871c server_cookie=e179195c24ff2202 in_seq=0 out_seq=0 2024-04-21T09:01:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.411+0000 7f959dffb640 1 -- 172.21.15.28:0/2615440628 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559b793acc00 con 0x559b79457d60 2024-04-21T09:01:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.411+0000 7f959dffb640 10 client.18147 handle_command_reply: tid=0 2024-04-21T09:01:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.411+0000 7f95bf3dd740 1 client.18147 shutdown 2024-04-21T09:01:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.411+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2615440628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b7943abd0 msgr2=0x559b794454e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.411+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/2615440628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b7943abd0 0x559b794454e0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f95a4006010 tx=0x7f95a40be280 comp rx=0 tx=0).stop 2024-04-21T09:01:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.411+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2615440628 shutdown_connections 2024-04-21T09:01:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.411+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/2615440628 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559b79457d60 0x559b7945a1b0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.412+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/2615440628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b79447e90 0x559b79445a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.412+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/2615440628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b7943abd0 0x559b794454e0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.412+0000 7f959dffb640 0 client.18147 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:01:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.412+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/2615440628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943a2b0 0x559b79444fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.412+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2615440628 >> 172.21.15.28:0/2615440628 conn(0x559b79434cb0 msgr2=0x559b79436a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.412+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2615440628 shutdown_connections 2024-04-21T09:01:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.412+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2615440628 wait complete. 2024-04-21T09:01:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.412+0000 7f95bf3dd740 20 client.18147 trim_cache size 0 max 16384 2024-04-21T09:01:21.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.429+0000 7f95bf3dd740 1 Processor -- start 2024-04-21T09:01:21.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.429+0000 7f95bf3dd740 1 -- start start 2024-04-21T09:01:21.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95bf3dd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 0x559b794476d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95bf3dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b7943abd0 0x559b79453af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95bf3dd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b79447e90 0x559b79443ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95bf3dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f95b41afb30 con 0x559b7943abd0 2024-04-21T09:01:21.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95bf3dd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f95b41ad400 con 0x559b79447e90 2024-04-21T09:01:21.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95bf3dd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f95b41ab270 con 0x559b7943a2b0 2024-04-21T09:01:21.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95b9d8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b7943abd0 0x559b79453af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95b9d8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b7943abd0 0x559b79453af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47096/0 (socket says 172.21.15.28:47096) 2024-04-21T09:01:21.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95b9d8f640 1 -- 172.21.15.28:0/1912783880 learned_addr learned my addr 172.21.15.28:0/1912783880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95ba590640 1 --2- 172.21.15.28:0/1912783880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 0x559b794476d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95bad91640 1 --2- 172.21.15.28:0/1912783880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b79447e90 0x559b79443ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95b9d8f640 1 -- 172.21.15.28:0/1912783880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 msgr2=0x559b794476d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95b9d8f640 1 --2- 172.21.15.28:0/1912783880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 0x559b794476d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95b9d8f640 1 -- 172.21.15.28:0/1912783880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b79447e90 msgr2=0x559b79443ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95b9d8f640 1 --2- 172.21.15.28:0/1912783880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b79447e90 0x559b79443ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95b9d8f640 1 -- 172.21.15.28:0/1912783880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b79439ee0 con 0x559b7943abd0 2024-04-21T09:01:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95ba590640 1 --2- 172.21.15.28:0/1912783880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 0x559b794476d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:01:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f95b9d8f640 1 --2- 172.21.15.28:0/1912783880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b7943abd0 0x559b79453af0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f95a40be620 tx=0x7f95a40be3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d81b749a84ba260 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:21.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f959ffff640 1 -- 172.21.15.28:0/1912783880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a40a7e60 con 0x559b7943abd0 2024-04-21T09:01:21.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f959ffff640 1 -- 172.21.15.28:0/1912783880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95a40bf020 con 0x559b7943abd0 2024-04-21T09:01:21.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.430+0000 7f959ffff640 1 -- 172.21.15.28:0/1912783880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a40baf90 con 0x559b7943abd0 2024-04-21T09:01:21.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.431+0000 7f95bf3dd740 1 -- 172.21.15.28:0/1912783880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b7943abd0 msgr2=0x559b79453af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.431+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/1912783880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b7943abd0 0x559b79453af0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f95a40be620 tx=0x7f95a40be3b0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.431+0000 7f95bf3dd740 1 -- 172.21.15.28:0/1912783880 shutdown_connections 2024-04-21T09:01:21.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.431+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/1912783880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b79447e90 0x559b79443ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.431+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/1912783880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b7943abd0 0x559b79453af0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.431+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/1912783880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 0x559b794476d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.431+0000 7f95bf3dd740 1 -- 172.21.15.28:0/1912783880 >> 172.21.15.28:0/1912783880 conn(0x559b79434cb0 msgr2=0x559b79433d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.431+0000 7f95bf3dd740 1 -- 172.21.15.28:0/1912783880 shutdown_connections 2024-04-21T09:01:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.431+0000 7f95bf3dd740 1 -- 172.21.15.28:0/1912783880 wait complete. 2024-04-21T09:01:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.431+0000 7f95bf3dd740 1 -- start start 2024-04-21T09:01:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.431+0000 7f95bf3dd740 1 Processor -- start 2024-04-21T09:01:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bf3dd740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 0x559b7944bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bf3dd740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943abd0 0x559b7944c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bf3dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b79447e90 0x559b7944c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bf3dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f95b41ab8c0 con 0x559b79447e90 2024-04-21T09:01:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bf3dd740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f95b41acd30 con 0x559b7943abd0 2024-04-21T09:01:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bf3dd740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f95b41abf90 con 0x559b7943a2b0 2024-04-21T09:01:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bf3dd740 10 client.0 _ll_register_callbacks cb 0x7f95a402cd20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:01:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bad91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b79447e90 0x559b7944c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bad91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b79447e90 0x559b7944c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47102/0 (socket says 172.21.15.28:47102) 2024-04-21T09:01:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bad91640 1 -- 172.21.15.28:0/2917123029 learned_addr learned my addr 172.21.15.28:0/2917123029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95ba590640 1 --2- 172.21.15.28:0/2917123029 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 0x559b7944bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95b9d8f640 1 --2- 172.21.15.28:0/2917123029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943abd0 0x559b7944c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bad91640 1 -- 172.21.15.28:0/2917123029 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 msgr2=0x559b7944bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bad91640 1 --2- 172.21.15.28:0/2917123029 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 0x559b7944bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bad91640 1 -- 172.21.15.28:0/2917123029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943abd0 msgr2=0x559b7944c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bad91640 1 --2- 172.21.15.28:0/2917123029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943abd0 0x559b7944c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bad91640 1 -- 172.21.15.28:0/2917123029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b794bd4a0 con 0x559b79447e90 2024-04-21T09:01:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bad91640 1 --2- 172.21.15.28:0/2917123029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b79447e90 0x559b7944c660 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f95ac014c30 tx=0x7f95ac00bb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511c78bd0404f76e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f959dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:01:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f959dffb640 1 -- 172.21.15.28:0/2917123029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac022040 con 0x559b79447e90 2024-04-21T09:01:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f959dffb640 1 -- 172.21.15.28:0/2917123029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f95ac004840 con 0x559b79447e90 2024-04-21T09:01:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.432+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2917123029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95a40bf020 con 0x559b79447e90 2024-04-21T09:01:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.433+0000 7f959dffb640 1 -- 172.21.15.28:0/2917123029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f95ac03c050 con 0x559b79447e90 2024-04-21T09:01:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.433+0000 7f95bf3dd740 10 client.18153 fetch_fsmap learned FSMap version 28 2024-04-21T09:01:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.433+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2917123029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559b794be770 con 0x559b79447e90 2024-04-21T09:01:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.433+0000 7f959dffb640 1 -- 172.21.15.28:0/2917123029 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95a40bf020 con 0x559b79447e90 2024-04-21T09:01:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.433+0000 7f959dffb640 1 -- 172.21.15.28:0/2917123029 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f95ac030070 con 0x559b79447e90 2024-04-21T09:01:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.433+0000 7f95bf3dd740 10 client.18153 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:01:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.433+0000 7f95bf3dd740 10 client.18153 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:01:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.433+0000 7f95bf3dd740 20 client.18153 populate_metadata read hostname 'smithi028' 2024-04-21T09:01:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.433+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/2917123029 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559b794becc0 0x559b794c10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.433+0000 7f95bf3dd740 4 client.18153 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T09:01:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.433+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2917123029 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559b793a6a70 con 0x559b794becc0 2024-04-21T09:01:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.433+0000 7f95ba590640 1 --2- 172.21.15.28:0/2917123029 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559b794becc0 0x559b794c10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f95ba590640 1 --2- 172.21.15.28:0/2917123029 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559b794becc0 0x559b794c10e0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3da8852b0f0549c server_cookie=9bc4e44647bc7b34 in_seq=0 out_seq=0 2024-04-21T09:01:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f959dffb640 10 client.18153 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:01:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f959dffb640 1 -- 172.21.15.28:0/2917123029 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559b793a6a70 con 0x559b794becc0 2024-04-21T09:01:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f959dffb640 10 client.18153 handle_command_reply: tid=0 2024-04-21T09:01:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f95bf3dd740 1 client.18153 shutdown 2024-04-21T09:01:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2917123029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b79447e90 msgr2=0x559b7944c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/2917123029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b79447e90 0x559b7944c660 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f95ac014c30 tx=0x7f95ac00bb80 comp rx=0 tx=0).stop 2024-04-21T09:01:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2917123029 shutdown_connections 2024-04-21T09:01:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/2917123029 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x559b794becc0 0x559b794c10e0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/2917123029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b79447e90 0x559b7944c660 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/2917123029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b7943abd0 0x559b7944c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f95bf3dd740 1 --2- 172.21.15.28:0/2917123029 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x559b7943a2b0 0x559b7944bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2917123029 >> 172.21.15.28:0/2917123029 conn(0x559b79434cb0 msgr2=0x559b79444ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f959dffb640 0 client.18153 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:01:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2917123029 shutdown_connections 2024-04-21T09:01:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f95bf3dd740 1 -- 172.21.15.28:0/2917123029 wait complete. 2024-04-21T09:01:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.434+0000 7f95bf3dd740 20 client.18153 trim_cache size 0 max 16384 2024-04-21T09:01:21.445 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T09:01:21.445 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-21T09:01:21.446 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-21T09:01:21.446 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T09:01:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.435+0000 7f95bc7f4640 1 -- 172.21.15.28:0/3207669515 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95a00a3d20 msgr2=0x7f95a00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.435+0000 7f95bc7f4640 1 --2- 172.21.15.28:0/3207669515 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95a00a3d20 0x7f95a00a61e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f95ac011cb0 tx=0x7f95ac016000 comp rx=0 tx=0).stop 2024-04-21T09:01:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.435+0000 7f95bc7f4640 1 -- 172.21.15.28:0/3207669515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b4107270 msgr2=0x7f95b419e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.435+0000 7f95bc7f4640 1 --2- 172.21.15.28:0/3207669515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b4107270 0x7f95b419e9b0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f95a4036040 tx=0x7f95a40045e0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.435+0000 7f95bc7f4640 1 -- 172.21.15.28:0/3207669515 shutdown_connections 2024-04-21T09:01:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.435+0000 7f95bc7f4640 1 --2- 172.21.15.28:0/3207669515 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95a00a3d20 0x7f95a00a61e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.435+0000 7f95bc7f4640 1 --2- 172.21.15.28:0/3207669515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f95b4198670 0x7f95b4198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.435+0000 7f95bc7f4640 1 --2- 172.21.15.28:0/3207669515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b4107270 0x7f95b419e9b0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.435+0000 7f95bc7f4640 1 --2- 172.21.15.28:0/3207669515 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95b41021a0 0x7f95b419e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.435+0000 7f95bc7f4640 1 -- 172.21.15.28:0/3207669515 >> 172.21.15.28:0/3207669515 conn(0x7f95b40fda40 msgr2=0x7f95b40fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.435+0000 7f95bc7f4640 1 -- 172.21.15.28:0/3207669515 shutdown_connections 2024-04-21T09:01:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.435+0000 7f95bc7f4640 1 -- 172.21.15.28:0/3207669515 wait complete. 2024-04-21T09:01:21.449 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T09:01:21.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T09:01:21.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.550+0000 7f4b4a550640 1 -- 172.21.15.28:0/2146452746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44105e30 msgr2=0x7f4b44106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.550+0000 7f4b4a550640 1 --2- 172.21.15.28:0/2146452746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44105e30 0x7f4b44106210 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f4b34009930 tx=0x7f4b3402bc20 comp rx=0 tx=0).stop 2024-04-21T09:01:21.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.550+0000 7f4b4a550640 1 -- 172.21.15.28:0/2146452746 shutdown_connections 2024-04-21T09:01:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.550+0000 7f4b4a550640 1 --2- 172.21.15.28:0/2146452746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b44101b60 0x7f4b4410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.550+0000 7f4b4a550640 1 --2- 172.21.15.28:0/2146452746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b441011a0 0x7f4b44101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.550+0000 7f4b4a550640 1 --2- 172.21.15.28:0/2146452746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44105e30 0x7f4b44106210 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.550+0000 7f4b4a550640 1 -- 172.21.15.28:0/2146452746 >> 172.21.15.28:0/2146452746 conn(0x7f4b44077e30 msgr2=0x7f4b44078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.550+0000 7f4b4a550640 1 -- 172.21.15.28:0/2146452746 shutdown_connections 2024-04-21T09:01:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.550+0000 7f4b4a550640 1 -- 172.21.15.28:0/2146452746 wait complete. 2024-04-21T09:01:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.550+0000 7f4b4a550640 1 Processor -- start 2024-04-21T09:01:21.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.550+0000 7f4b4a550640 1 -- start start 2024-04-21T09:01:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b4a550640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b441011a0 0x7f4b44198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b4a550640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b44101b60 0x7f4b44198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b4a550640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44105e30 0x7f4b4419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b4a550640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b44111110 con 0x7f4b44105e30 2024-04-21T09:01:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b4a550640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f4b44110f90 con 0x7f4b441011a0 2024-04-21T09:01:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b4a550640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f4b44111290 con 0x7f4b44101b60 2024-04-21T09:01:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b48aed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44105e30 0x7f4b4419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b48aed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44105e30 0x7f4b4419cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47128/0 (socket says 172.21.15.28:47128) 2024-04-21T09:01:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b48aed640 1 -- 172.21.15.28:0/395377265 learned_addr learned my addr 172.21.15.28:0/395377265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b437fe640 1 --2- 172.21.15.28:0/395377265 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b44101b60 0x7f4b44198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b43fff640 1 --2- 172.21.15.28:0/395377265 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b441011a0 0x7f4b44198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b48aed640 1 -- 172.21.15.28:0/395377265 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b44101b60 msgr2=0x7f4b44198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b48aed640 1 --2- 172.21.15.28:0/395377265 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b44101b60 0x7f4b44198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b48aed640 1 -- 172.21.15.28:0/395377265 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b441011a0 msgr2=0x7f4b44198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b48aed640 1 --2- 172.21.15.28:0/395377265 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b441011a0 0x7f4b44198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b48aed640 1 -- 172.21.15.28:0/395377265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b4419d630 con 0x7f4b44105e30 2024-04-21T09:01:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.551+0000 7f4b48aed640 1 --2- 172.21.15.28:0/395377265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44105e30 0x7f4b4419cf00 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00cce0 tx=0x7f4b3c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.552+0000 7f4b417fa640 1 -- 172.21.15.28:0/395377265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c00f450 con 0x7f4b44105e30 2024-04-21T09:01:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.552+0000 7f4b417fa640 1 -- 172.21.15.28:0/395377265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b3c00faa0 con 0x7f4b44105e30 2024-04-21T09:01:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.552+0000 7f4b4a550640 1 -- 172.21.15.28:0/395377265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b4419d920 con 0x7f4b44105e30 2024-04-21T09:01:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.552+0000 7f4b417fa640 1 -- 172.21.15.28:0/395377265 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c018a80 con 0x7f4b44105e30 2024-04-21T09:01:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.552+0000 7f4b4a550640 1 -- 172.21.15.28:0/395377265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b441a51b0 con 0x7f4b44105e30 2024-04-21T09:01:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.553+0000 7f4b4a550640 1 -- 172.21.15.28:0/395377265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b44102570 con 0x7f4b44105e30 2024-04-21T09:01:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.554+0000 7f4b417fa640 1 -- 172.21.15.28:0/395377265 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4b3c00fc50 con 0x7f4b44105e30 2024-04-21T09:01:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.555+0000 7f4b417fa640 1 --2- 172.21.15.28:0/395377265 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4b2c0a3d20 0x7f4b2c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.555+0000 7f4b43fff640 1 --2- 172.21.15.28:0/395377265 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4b2c0a3d20 0x7f4b2c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.555+0000 7f4b417fa640 1 -- 172.21.15.28:0/395377265 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f4b3c0d14f0 con 0x7f4b44105e30 2024-04-21T09:01:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.555+0000 7f4b43fff640 1 --2- 172.21.15.28:0/395377265 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4b2c0a3d20 0x7f4b2c0a61e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4b34009900 tx=0x7f4b34005a20 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.558+0000 7f4b417fa640 1 -- 172.21.15.28:0/395377265 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b3c014030 con 0x7f4b44105e30 2024-04-21T09:01:21.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.782+0000 7f4b4a550640 1 -- 172.21.15.28:0/395377265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b44068de0 con 0x7f4b44105e30 2024-04-21T09:01:21.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.783+0000 7f4b417fa640 1 -- 172.21.15.28:0/395377265 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7f4b44068de0 con 0x7f4b44105e30 2024-04-21T09:01:21.784 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T09:01:21.784 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T09:01:21.785 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T09:01:21.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.786+0000 7f4b4a550640 1 -- 172.21.15.28:0/395377265 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4b2c0a3d20 msgr2=0x7f4b2c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.786+0000 7f4b4a550640 1 --2- 172.21.15.28:0/395377265 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4b2c0a3d20 0x7f4b2c0a61e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4b34009900 tx=0x7f4b34005a20 comp rx=0 tx=0).stop 2024-04-21T09:01:21.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.786+0000 7f4b4a550640 1 -- 172.21.15.28:0/395377265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44105e30 msgr2=0x7f4b4419cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.786+0000 7f4b4a550640 1 --2- 172.21.15.28:0/395377265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44105e30 0x7f4b4419cf00 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00cce0 tx=0x7f4b3c00a620 comp rx=0 tx=0).stop 2024-04-21T09:01:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.786+0000 7f4b4a550640 1 -- 172.21.15.28:0/395377265 shutdown_connections 2024-04-21T09:01:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.786+0000 7f4b4a550640 1 --2- 172.21.15.28:0/395377265 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f4b2c0a3d20 0x7f4b2c0a61e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.786+0000 7f4b4a550640 1 --2- 172.21.15.28:0/395377265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b44105e30 0x7f4b4419cf00 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.786+0000 7f4b4a550640 1 --2- 172.21.15.28:0/395377265 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b44101b60 0x7f4b44198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.786+0000 7f4b4a550640 1 --2- 172.21.15.28:0/395377265 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b441011a0 0x7f4b44198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.786+0000 7f4b4a550640 1 -- 172.21.15.28:0/395377265 >> 172.21.15.28:0/395377265 conn(0x7f4b44077e30 msgr2=0x7f4b44069320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.786+0000 7f4b4a550640 1 -- 172.21.15.28:0/395377265 shutdown_connections 2024-04-21T09:01:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.786+0000 7f4b4a550640 1 -- 172.21.15.28:0/395377265 wait complete. 2024-04-21T09:01:21.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-21T09:01:21.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.900+0000 7f96053b2640 1 -- 172.21.15.28:0/1620936224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9600107260 msgr2=0x7f960010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.900+0000 7f96053b2640 1 --2- 172.21.15.28:0/1620936224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9600107260 0x7f960010ee30 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f95f8009ca0 tx=0x7f95f802c0f0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.901+0000 7f96053b2640 1 -- 172.21.15.28:0/1620936224 shutdown_connections 2024-04-21T09:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.901+0000 7f96053b2640 1 --2- 172.21.15.28:0/1620936224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9600107260 0x7f960010ee30 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.901+0000 7f96053b2640 1 --2- 172.21.15.28:0/1620936224 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9600102b40 0x7f9600106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.901+0000 7f96053b2640 1 --2- 172.21.15.28:0/1620936224 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9600102190 0x7f9600102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.901+0000 7f96053b2640 1 -- 172.21.15.28:0/1620936224 >> 172.21.15.28:0/1620936224 conn(0x7f96000fd990 msgr2=0x7f96000ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.901+0000 7f96053b2640 1 -- 172.21.15.28:0/1620936224 shutdown_connections 2024-04-21T09:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.901+0000 7f96053b2640 1 -- 172.21.15.28:0/1620936224 wait complete. 2024-04-21T09:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.901+0000 7f96053b2640 1 Processor -- start 2024-04-21T09:01:21.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.901+0000 7f96053b2640 1 -- start start 2024-04-21T09:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f96053b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9600102190 0x7f96001986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f96053b2640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9600107260 0x7f9600198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f96053b2640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f960019cf70 0x7f960019d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f96053b2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96001121b0 con 0x7f9600102190 2024-04-21T09:01:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f96053b2640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f9600112030 con 0x7f9600107260 2024-04-21T09:01:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f96053b2640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f9600112330 con 0x7f960019cf70 2024-04-21T09:01:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f95feffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9600102190 0x7f96001986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f95feffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9600102190 0x7f96001986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47156/0 (socket says 172.21.15.28:47156) 2024-04-21T09:01:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f95feffd640 1 -- 172.21.15.28:0/2608432514 learned_addr learned my addr 172.21.15.28:0/2608432514 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f95f67fc640 1 --2- 172.21.15.28:0/2608432514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9600107260 0x7f9600198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f95ff7fe640 1 --2- 172.21.15.28:0/2608432514 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f960019cf70 0x7f960019d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f95feffd640 1 -- 172.21.15.28:0/2608432514 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f960019cf70 msgr2=0x7f960019d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f95feffd640 1 --2- 172.21.15.28:0/2608432514 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f960019cf70 0x7f960019d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f95feffd640 1 -- 172.21.15.28:0/2608432514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9600107260 msgr2=0x7f9600198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f95feffd640 1 --2- 172.21.15.28:0/2608432514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9600107260 0x7f9600198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f95feffd640 1 -- 172.21.15.28:0/2608432514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960019da60 con 0x7f9600102190 2024-04-21T09:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f95feffd640 1 --2- 172.21.15.28:0/2608432514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9600102190 0x7f96001986b0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f95f000c890 tx=0x7f95f000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f95fcff9640 1 -- 172.21.15.28:0/2608432514 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f0004220 con 0x7f9600102190 2024-04-21T09:01:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f95fcff9640 1 -- 172.21.15.28:0/2608432514 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95f00043c0 con 0x7f9600102190 2024-04-21T09:01:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f96053b2640 1 -- 172.21.15.28:0/2608432514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f960019dd50 con 0x7f9600102190 2024-04-21T09:01:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f95fcff9640 1 -- 172.21.15.28:0/2608432514 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f0004a00 con 0x7f9600102190 2024-04-21T09:01:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.902+0000 7f96053b2640 1 -- 172.21.15.28:0/2608432514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96001a5540 con 0x7f9600102190 2024-04-21T09:01:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.905+0000 7f95fcff9640 1 -- 172.21.15.28:0/2608432514 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f95f00079b0 con 0x7f9600102190 2024-04-21T09:01:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.905+0000 7f9607fa3740 1 Processor -- start 2024-04-21T09:01:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.905+0000 7f9607fa3740 1 -- start start 2024-04-21T09:01:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.905+0000 7f9607fa3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c081b4d0 0x55a2c081b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.905+0000 7f9607fa3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c081be80 0x55a2c0824430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.905+0000 7f9607fa3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c0824a00 0x55a2c0826e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.905+0000 7f9607fa3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a2c07801f0 con 0x55a2c0824a00 2024-04-21T09:01:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.905+0000 7f9607fa3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55a2c071d4b0 con 0x55a2c081be80 2024-04-21T09:01:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.905+0000 7f9607fa3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55a2c077c1b0 con 0x55a2c081b4d0 2024-04-21T09:01:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.905+0000 7f95fcff9640 1 --2- 172.21.15.28:0/2608432514 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95ec0a3d20 0x7f95ec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.905+0000 7f95f67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c0824a00 0x55a2c0826e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.905+0000 7f95f67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c0824a00 0x55a2c0826e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47172/0 (socket says 172.21.15.28:47172) 2024-04-21T09:01:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.905+0000 7f95f67fc640 1 -- 172.21.15.28:0/3951339782 learned_addr learned my addr 172.21.15.28:0/3951339782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.906+0000 7f95feffd640 1 --2- 172.21.15.28:0/3951339782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c081be80 0x55a2c0824430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.906+0000 7f95ff7fe640 1 --2- 172.21.15.28:0/2608432514 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95ec0a3d20 0x7f95ec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.906+0000 7f95ff7fe640 1 --2- 172.21.15.28:0/3951339782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c081b4d0 0x55a2c081b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.906+0000 7f95fcff9640 1 -- 172.21.15.28:0/2608432514 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f95f00d0890 con 0x7f9600102190 2024-04-21T09:01:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.906+0000 7f95f67fc640 1 -- 172.21.15.28:0/3951339782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c081b4d0 msgr2=0x55a2c081b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.906+0000 7f95f67fc640 1 --2- 172.21.15.28:0/3951339782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c081b4d0 0x55a2c081b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.906+0000 7f95f67fc640 1 -- 172.21.15.28:0/3951339782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c081be80 msgr2=0x55a2c0824430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.906+0000 7f95f67fc640 1 --2- 172.21.15.28:0/3951339782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c081be80 0x55a2c0824430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.906+0000 7f95f67fc640 1 -- 172.21.15.28:0/3951339782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2c070f1b0 con 0x55a2c0824a00 2024-04-21T09:01:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.906+0000 7f95feffd640 1 --2- 172.21.15.28:0/3951339782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c081be80 0x55a2c0824430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:01:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.906+0000 7f95f67fc640 1 --2- 172.21.15.28:0/3951339782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c0824a00 0x55a2c0826e50 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f95e800abf0 tx=0x7f95e800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b79658a900fa48e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.906+0000 7f95ff7fe640 1 --2- 172.21.15.28:0/2608432514 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95ec0a3d20 0x7f95ec0a61e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f95f8006010 tx=0x7f95f802e040 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.906+0000 7f95f57fa640 1 -- 172.21.15.28:0/3951339782 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e801d070 con 0x55a2c0824a00 2024-04-21T09:01:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.906+0000 7f95f57fa640 1 -- 172.21.15.28:0/3951339782 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95e8018af0 con 0x55a2c0824a00 2024-04-21T09:01:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.906+0000 7f95f57fa640 1 -- 172.21.15.28:0/3951339782 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e8010b20 con 0x55a2c0824a00 2024-04-21T09:01:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.907+0000 7f9607fa3740 1 -- 172.21.15.28:0/3951339782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c0824a00 msgr2=0x55a2c0826e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.907+0000 7f9607fa3740 1 --2- 172.21.15.28:0/3951339782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c0824a00 0x55a2c0826e50 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f95e800abf0 tx=0x7f95e800f790 comp rx=0 tx=0).stop 2024-04-21T09:01:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.907+0000 7f9607fa3740 1 -- 172.21.15.28:0/3951339782 shutdown_connections 2024-04-21T09:01:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.907+0000 7f9607fa3740 1 --2- 172.21.15.28:0/3951339782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c0824a00 0x55a2c0826e50 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.907+0000 7f9607fa3740 1 --2- 172.21.15.28:0/3951339782 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c081be80 0x55a2c0824430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.907+0000 7f9607fa3740 1 --2- 172.21.15.28:0/3951339782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c081b4d0 0x55a2c081b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.907+0000 7f9607fa3740 1 -- 172.21.15.28:0/3951339782 >> 172.21.15.28:0/3951339782 conn(0x55a2c0815ba0 msgr2=0x55a2c0815f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.907+0000 7f9607fa3740 1 -- 172.21.15.28:0/3951339782 shutdown_connections 2024-04-21T09:01:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.907+0000 7f9607fa3740 1 -- 172.21.15.28:0/3951339782 wait complete. 2024-04-21T09:01:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.907+0000 7f9607fa3740 1 -- start start 2024-04-21T09:01:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.907+0000 7f9607fa3740 1 Processor -- start 2024-04-21T09:01:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f9607fa3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081b4d0 0x55a2c08242f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f9607fa3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c081be80 0x55a2c0821680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f9607fa3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c0824a00 0x55a2c0821c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f9607fa3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a2c0827970 con 0x55a2c081b4d0 2024-04-21T09:01:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f9607fa3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x55a2c0739500 con 0x55a2c081be80 2024-04-21T09:01:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f9607fa3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x55a2c0827af0 con 0x55a2c0824a00 2024-04-21T09:01:21.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f9607fa3740 10 client.0 _ll_register_callbacks cb 0x55a2c0707b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:01:21.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f95feffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081b4d0 0x55a2c08242f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f95feffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081b4d0 0x55a2c08242f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47180/0 (socket says 172.21.15.28:47180) 2024-04-21T09:01:21.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f95feffd640 1 -- 172.21.15.28:0/1746648078 learned_addr learned my addr 172.21.15.28:0/1746648078 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:21.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f95ff7fe640 1 --2- 172.21.15.28:0/1746648078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c0824a00 0x55a2c0821c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f95feffd640 1 -- 172.21.15.28:0/1746648078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c0824a00 msgr2=0x55a2c0821c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f95feffd640 1 --2- 172.21.15.28:0/1746648078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c0824a00 0x55a2c0821c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f95feffd640 1 -- 172.21.15.28:0/1746648078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c081be80 msgr2=0x55a2c0821680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f95f67fc640 1 --2- 172.21.15.28:0/1746648078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c081be80 0x55a2c0821680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f95feffd640 1 --2- 172.21.15.28:0/1746648078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c081be80 0x55a2c0821680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f95feffd640 1 -- 172.21.15.28:0/1746648078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2c04923f0 con 0x55a2c081b4d0 2024-04-21T09:01:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f95feffd640 1 --2- 172.21.15.28:0/1746648078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081b4d0 0x55a2c08242f0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f95f009c3d0 tx=0x7f95f0008ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e30c7ef709f73c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f95e67fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:01:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f95e67fc640 1 -- 172.21.15.28:0/1746648078 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f009c440 con 0x55a2c081b4d0 2024-04-21T09:01:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f95e67fc640 1 -- 172.21.15.28:0/1746648078 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f95f005fbe0 con 0x55a2c081b4d0 2024-04-21T09:01:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.908+0000 7f9607fa3740 1 -- 172.21.15.28:0/1746648078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95e8018af0 con 0x55a2c081b4d0 2024-04-21T09:01:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.909+0000 7f95e67fc640 1 -- 172.21.15.28:0/1746648078 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f95f00986e0 con 0x55a2c081b4d0 2024-04-21T09:01:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.909+0000 7f9607fa3740 10 client.18171 fetch_fsmap learned FSMap version 28 2024-04-21T09:01:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.909+0000 7f9607fa3740 1 -- 172.21.15.28:0/1746648078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a2c055c7d0 con 0x55a2c081b4d0 2024-04-21T09:01:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.909+0000 7f95e67fc640 1 -- 172.21.15.28:0/1746648078 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95e8018af0 con 0x55a2c081b4d0 2024-04-21T09:01:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.909+0000 7f95e67fc640 1 -- 172.21.15.28:0/1746648078 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f95f0098ab0 con 0x55a2c081b4d0 2024-04-21T09:01:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.909+0000 7f9607fa3740 10 client.18171 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:01:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.909+0000 7f9607fa3740 10 client.18171 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T09:01:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.909+0000 7f9607fa3740 20 client.18171 populate_metadata read hostname 'smithi028' 2024-04-21T09:01:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.909+0000 7f9607fa3740 1 --2- 172.21.15.28:0/1746648078 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55a2c0834bf0 0x55a2c0837080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.909+0000 7f9607fa3740 4 client.18171 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:01:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.909+0000 7f9607fa3740 1 -- 172.21.15.28:0/1746648078 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2c071f930 con 0x55a2c0834bf0 2024-04-21T09:01:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.909+0000 7f95f67fc640 1 --2- 172.21.15.28:0/1746648078 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55a2c0834bf0 0x55a2c0837080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.910+0000 7f95e67fc640 10 client.18171 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T09:01:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.910+0000 7f95f67fc640 1 --2- 172.21.15.28:0/1746648078 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55a2c0834bf0 0x55a2c0837080 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9bd8e4cd09c975a9 server_cookie=9b1ee6e427181636 in_seq=0 out_seq=0 2024-04-21T09:01:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.910+0000 7f95e67fc640 1 -- 172.21.15.28:0/1746648078 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a2c071f930 con 0x55a2c0834bf0 2024-04-21T09:01:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.910+0000 7f95e67fc640 10 client.18171 handle_command_reply: tid=0 2024-04-21T09:01:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.911+0000 7f9607fa3740 1 client.18171 shutdown 2024-04-21T09:01:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.911+0000 7f9607fa3740 1 -- 172.21.15.28:0/1746648078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081b4d0 msgr2=0x55a2c08242f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.911+0000 7f9607fa3740 1 --2- 172.21.15.28:0/1746648078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081b4d0 0x55a2c08242f0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f95f009c3d0 tx=0x7f95f0008ed0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.911+0000 7f9607fa3740 1 -- 172.21.15.28:0/1746648078 shutdown_connections 2024-04-21T09:01:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.911+0000 7f9607fa3740 1 --2- 172.21.15.28:0/1746648078 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55a2c0834bf0 0x55a2c0837080 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.911+0000 7f9607fa3740 1 --2- 172.21.15.28:0/1746648078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c0824a00 0x55a2c0821c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.911+0000 7f9607fa3740 1 --2- 172.21.15.28:0/1746648078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c081be80 0x55a2c0821680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.911+0000 7f95e67fc640 0 client.18171 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T09:01:21.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.911+0000 7f9607fa3740 1 --2- 172.21.15.28:0/1746648078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081b4d0 0x55a2c08242f0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.911+0000 7f9607fa3740 1 -- 172.21.15.28:0/1746648078 >> 172.21.15.28:0/1746648078 conn(0x55a2c0815ba0 msgr2=0x55a2c08178f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.911+0000 7f9607fa3740 1 -- 172.21.15.28:0/1746648078 shutdown_connections 2024-04-21T09:01:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.911+0000 7f9607fa3740 1 -- 172.21.15.28:0/1746648078 wait complete. 2024-04-21T09:01:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.911+0000 7f9607fa3740 20 client.18171 trim_cache size 0 max 16384 2024-04-21T09:01:21.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.924+0000 7f9607fa3740 1 Processor -- start 2024-04-21T09:01:21.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.924+0000 7f9607fa3740 1 -- start start 2024-04-21T09:01:21.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.924+0000 7f9607fa3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c081b4d0 0x55a2c0830830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.924+0000 7f9607fa3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081be80 0x55a2c08176d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.924+0000 7f9607fa3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c0824a00 0x55a2c0823db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.924+0000 7f9607fa3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96001b1cd0 con 0x55a2c081be80 2024-04-21T09:01:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.924+0000 7f9607fa3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f96001af5a0 con 0x55a2c0824a00 2024-04-21T09:01:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f9607fa3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f96001ad410 con 0x55a2c081b4d0 2024-04-21T09:01:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f95f67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081be80 0x55a2c08176d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f95f67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081be80 0x55a2c08176d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47182/0 (socket says 172.21.15.28:47182) 2024-04-21T09:01:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f95f67fc640 1 -- 172.21.15.28:0/1966357029 learned_addr learned my addr 172.21.15.28:0/1966357029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f95ff7fe640 1 --2- 172.21.15.28:0/1966357029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c0824a00 0x55a2c0823db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f95feffd640 1 --2- 172.21.15.28:0/1966357029 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c081b4d0 0x55a2c0830830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f95f67fc640 1 -- 172.21.15.28:0/1966357029 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c081b4d0 msgr2=0x55a2c0830830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f95f67fc640 1 --2- 172.21.15.28:0/1966357029 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c081b4d0 0x55a2c0830830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f95f67fc640 1 -- 172.21.15.28:0/1966357029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c0824a00 msgr2=0x55a2c0823db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f95f67fc640 1 --2- 172.21.15.28:0/1966357029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c0824a00 0x55a2c0823db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f95f67fc640 1 -- 172.21.15.28:0/1966357029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2c0820630 con 0x55a2c081be80 2024-04-21T09:01:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f95ff7fe640 1 --2- 172.21.15.28:0/1966357029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c0824a00 0x55a2c0823db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:01:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f95f67fc640 1 --2- 172.21.15.28:0/1966357029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081be80 0x55a2c08176d0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f95e80106b0 tx=0x7f95e8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e48d734872e52908 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f95f4ff9640 1 -- 172.21.15.28:0/1966357029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e801d040 con 0x55a2c081be80 2024-04-21T09:01:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f95f4ff9640 1 -- 172.21.15.28:0/1966357029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95e8027e50 con 0x55a2c081be80 2024-04-21T09:01:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.925+0000 7f95f4ff9640 1 -- 172.21.15.28:0/1966357029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e8005480 con 0x55a2c081be80 2024-04-21T09:01:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.926+0000 7f9607fa3740 1 -- 172.21.15.28:0/1966357029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081be80 msgr2=0x55a2c08176d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.926+0000 7f9607fa3740 1 --2- 172.21.15.28:0/1966357029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081be80 0x55a2c08176d0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f95e80106b0 tx=0x7f95e8000bc0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.926+0000 7f9607fa3740 1 -- 172.21.15.28:0/1966357029 shutdown_connections 2024-04-21T09:01:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.926+0000 7f9607fa3740 1 --2- 172.21.15.28:0/1966357029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c0824a00 0x55a2c0823db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.926+0000 7f9607fa3740 1 --2- 172.21.15.28:0/1966357029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081be80 0x55a2c08176d0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.926+0000 7f9607fa3740 1 --2- 172.21.15.28:0/1966357029 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c081b4d0 0x55a2c0830830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.926+0000 7f9607fa3740 1 -- 172.21.15.28:0/1966357029 >> 172.21.15.28:0/1966357029 conn(0x55a2c0815ba0 msgr2=0x55a2c0816fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.926+0000 7f9607fa3740 1 -- 172.21.15.28:0/1966357029 shutdown_connections 2024-04-21T09:01:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.926+0000 7f9607fa3740 1 -- 172.21.15.28:0/1966357029 wait complete. 2024-04-21T09:01:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.926+0000 7f9607fa3740 1 -- start start 2024-04-21T09:01:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.926+0000 7f9607fa3740 1 Processor -- start 2024-04-21T09:01:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.926+0000 7f9607fa3740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c081b4d0 0x55a2c082d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.926+0000 7f9607fa3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081be80 0x55a2c081e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f9607fa3740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c0824a00 0x55a2c081e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f9607fa3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96001ada60 con 0x55a2c081be80 2024-04-21T09:01:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f9607fa3740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f96001aeed0 con 0x55a2c081b4d0 2024-04-21T09:01:21.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f9607fa3740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f96001ae130 con 0x55a2c0824a00 2024-04-21T09:01:21.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f9607fa3740 10 client.0 _ll_register_callbacks cb 0x7f95f000a520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:01:21.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f95f67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081be80 0x55a2c081e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f95f67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081be80 0x55a2c081e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47192/0 (socket says 172.21.15.28:47192) 2024-04-21T09:01:21.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f95f67fc640 1 -- 172.21.15.28:0/246578712 learned_addr learned my addr 172.21.15.28:0/246578712 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:21.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f95ff7fe640 1 --2- 172.21.15.28:0/246578712 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c0824a00 0x55a2c081e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f95f67fc640 1 -- 172.21.15.28:0/246578712 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c0824a00 msgr2=0x55a2c081e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f95f67fc640 1 --2- 172.21.15.28:0/246578712 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c0824a00 0x55a2c081e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f95f67fc640 1 -- 172.21.15.28:0/246578712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c081b4d0 msgr2=0x55a2c082d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f95f67fc640 1 --2- 172.21.15.28:0/246578712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c081b4d0 0x55a2c082d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f95f67fc640 1 -- 172.21.15.28:0/246578712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2c0821f70 con 0x55a2c081be80 2024-04-21T09:01:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f95f67fc640 1 --2- 172.21.15.28:0/246578712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081be80 0x55a2c081e030 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f95e800b010 tx=0x7f95e800a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f091af6996e379e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f95e67fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:01:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f95e67fc640 1 -- 172.21.15.28:0/246578712 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e8036060 con 0x55a2c081be80 2024-04-21T09:01:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f95e67fc640 1 -- 172.21.15.28:0/246578712 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f95e80050e0 con 0x55a2c081be80 2024-04-21T09:01:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f9607fa3740 1 -- 172.21.15.28:0/246578712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95e8027e50 con 0x55a2c081be80 2024-04-21T09:01:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f95e67fc640 1 -- 172.21.15.28:0/246578712 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f95e801d040 con 0x55a2c081be80 2024-04-21T09:01:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f9607fa3740 10 client.18177 fetch_fsmap learned FSMap version 28 2024-04-21T09:01:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.927+0000 7f9607fa3740 1 -- 172.21.15.28:0/246578712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a2c089af10 con 0x55a2c081be80 2024-04-21T09:01:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.928+0000 7f95e67fc640 1 -- 172.21.15.28:0/246578712 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95e8027e50 con 0x55a2c081be80 2024-04-21T09:01:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.928+0000 7f95e67fc640 1 -- 172.21.15.28:0/246578712 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f95e8027370 con 0x55a2c081be80 2024-04-21T09:01:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.928+0000 7f9607fa3740 10 client.18177 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:01:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.928+0000 7f9607fa3740 10 client.18177 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-21T09:01:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.928+0000 7f9607fa3740 20 client.18177 populate_metadata read hostname 'smithi028' 2024-04-21T09:01:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.928+0000 7f9607fa3740 1 --2- 172.21.15.28:0/246578712 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55a2c089b460 0x55a2c089d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.928+0000 7f9607fa3740 4 client.18177 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T09:01:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.928+0000 7f9607fa3740 1 -- 172.21.15.28:0/246578712 --> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a2c078dc10 con 0x55a2c089b460 2024-04-21T09:01:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.928+0000 7f95feffd640 1 --2- 172.21.15.28:0/246578712 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55a2c089b460 0x55a2c089d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.928+0000 7f95e67fc640 10 client.18177 ms_handle_connect on v2:172.21.15.171:6836/895402983 2024-04-21T09:01:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.928+0000 7f95feffd640 1 --2- 172.21.15.28:0/246578712 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55a2c089b460 0x55a2c089d880 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c54369c984a20b94 server_cookie=85635f796896d857 in_seq=0 out_seq=0 2024-04-21T09:01:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f95e67fc640 1 -- 172.21.15.28:0/246578712 <== mds.2 v2:172.21.15.171:6836/895402983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a2c078dc10 con 0x55a2c089b460 2024-04-21T09:01:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f95e67fc640 10 client.18177 handle_command_reply: tid=0 2024-04-21T09:01:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f9607fa3740 1 client.18177 shutdown 2024-04-21T09:01:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f9607fa3740 1 -- 172.21.15.28:0/246578712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081be80 msgr2=0x55a2c081e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f9607fa3740 1 --2- 172.21.15.28:0/246578712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081be80 0x55a2c081e030 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f95e800b010 tx=0x7f95e800a6c0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f9607fa3740 1 -- 172.21.15.28:0/246578712 shutdown_connections 2024-04-21T09:01:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f9607fa3740 1 --2- 172.21.15.28:0/246578712 >> [v2:172.21.15.171:6836/895402983,v1:172.21.15.171:6837/895402983] conn(0x55a2c089b460 0x55a2c089d880 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f9607fa3740 1 --2- 172.21.15.28:0/246578712 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x55a2c0824a00 0x55a2c081e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f9607fa3740 1 --2- 172.21.15.28:0/246578712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2c081be80 0x55a2c081e030 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f95e67fc640 0 client.18177 ms_handle_reset on v2:172.21.15.171:6836/895402983 2024-04-21T09:01:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f9607fa3740 1 --2- 172.21.15.28:0/246578712 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2c081b4d0 0x55a2c082d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f9607fa3740 1 -- 172.21.15.28:0/246578712 >> 172.21.15.28:0/246578712 conn(0x55a2c0815ba0 msgr2=0x55a2c0817850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:21.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f9607fa3740 1 -- 172.21.15.28:0/246578712 shutdown_connections 2024-04-21T09:01:21.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f9607fa3740 1 -- 172.21.15.28:0/246578712 wait complete. 2024-04-21T09:01:21.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f9607fa3740 20 client.18177 trim_cache size 0 max 16384 2024-04-21T09:01:21.941 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T09:01:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f96053b2640 1 -- 172.21.15.28:0/2608432514 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95ec0a3d20 msgr2=0x7f95ec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f96053b2640 1 --2- 172.21.15.28:0/2608432514 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95ec0a3d20 0x7f95ec0a61e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f95f8006010 tx=0x7f95f802e040 comp rx=0 tx=0).stop 2024-04-21T09:01:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f96053b2640 1 -- 172.21.15.28:0/2608432514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9600102190 msgr2=0x7f96001986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f96053b2640 1 --2- 172.21.15.28:0/2608432514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9600102190 0x7f96001986b0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f95f000c890 tx=0x7f95f000cc90 comp rx=0 tx=0).stop 2024-04-21T09:01:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f96053b2640 1 -- 172.21.15.28:0/2608432514 shutdown_connections 2024-04-21T09:01:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f96053b2640 1 --2- 172.21.15.28:0/2608432514 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f95ec0a3d20 0x7f95ec0a61e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f96053b2640 1 --2- 172.21.15.28:0/2608432514 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f960019cf70 0x7f960019d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f96053b2640 1 --2- 172.21.15.28:0/2608432514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9600107260 0x7f9600198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f96053b2640 1 --2- 172.21.15.28:0/2608432514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9600102190 0x7f96001986b0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.929+0000 7f96053b2640 1 -- 172.21.15.28:0/2608432514 >> 172.21.15.28:0/2608432514 conn(0x7f96000fd990 msgr2=0x7f96000fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.930+0000 7f96053b2640 1 -- 172.21.15.28:0/2608432514 shutdown_connections 2024-04-21T09:01:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:21.930+0000 7f96053b2640 1 -- 172.21.15.28:0/2608432514 wait complete. 2024-04-21T09:01:21.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-21T09:01:22.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.048+0000 7f542959f640 1 -- 172.21.15.28:0/3564995945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5424106160 msgr2=0x7f542410dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.048+0000 7f542959f640 1 --2- 172.21.15.28:0/3564995945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5424106160 0x7f542410dd30 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f541c009ca0 tx=0x7f541c02c0f0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.048+0000 7f542959f640 1 -- 172.21.15.28:0/3564995945 shutdown_connections 2024-04-21T09:01:22.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.048+0000 7f542959f640 1 --2- 172.21.15.28:0/3564995945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5424106160 0x7f542410dd30 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.048+0000 7f542959f640 1 --2- 172.21.15.28:0/3564995945 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5424101a40 0x7f5424105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.048+0000 7f542959f640 1 --2- 172.21.15.28:0/3564995945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5424101090 0x7f5424101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.048+0000 7f542959f640 1 -- 172.21.15.28:0/3564995945 >> 172.21.15.28:0/3564995945 conn(0x7f54240771d0 msgr2=0x7f54240775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.048+0000 7f542959f640 1 -- 172.21.15.28:0/3564995945 shutdown_connections 2024-04-21T09:01:22.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.048+0000 7f542959f640 1 -- 172.21.15.28:0/3564995945 wait complete. 2024-04-21T09:01:22.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f542959f640 1 Processor -- start 2024-04-21T09:01:22.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f542959f640 1 -- start start 2024-04-21T09:01:22.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f542959f640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5424101090 0x7f5424198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f542959f640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5424106160 0x7f5424198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f542959f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f542419cf20 0x7f542419d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f542959f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54241110b0 con 0x7f542419cf20 2024-04-21T09:01:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f542959f640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5424110f30 con 0x7f5424101090 2024-04-21T09:01:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f542959f640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5424111230 con 0x7f5424106160 2024-04-21T09:01:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f54237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f542419cf20 0x7f542419d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f54237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f542419cf20 0x7f542419d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47208/0 (socket says 172.21.15.28:47208) 2024-04-21T09:01:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f54237fe640 1 -- 172.21.15.28:0/3485990170 learned_addr learned my addr 172.21.15.28:0/3485990170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f54237fe640 1 -- 172.21.15.28:0/3485990170 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5424106160 msgr2=0x7f5424198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f54237fe640 1 --2- 172.21.15.28:0/3485990170 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5424106160 0x7f5424198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f54237fe640 1 -- 172.21.15.28:0/3485990170 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5424101090 msgr2=0x7f5424198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f54237fe640 1 --2- 172.21.15.28:0/3485990170 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5424101090 0x7f5424198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.049+0000 7f54237fe640 1 -- 172.21.15.28:0/3485990170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542419da10 con 0x7f542419cf20 2024-04-21T09:01:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.050+0000 7f54237fe640 1 --2- 172.21.15.28:0/3485990170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f542419cf20 0x7f542419d3d0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f541c0098d0 tx=0x7f541c035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.050+0000 7f5420ff9640 1 -- 172.21.15.28:0/3485990170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c039040 con 0x7f542419cf20 2024-04-21T09:01:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.050+0000 7f5420ff9640 1 -- 172.21.15.28:0/3485990170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f541c044070 con 0x7f542419cf20 2024-04-21T09:01:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.050+0000 7f542959f640 1 -- 172.21.15.28:0/3485990170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f542419dca0 con 0x7f542419cf20 2024-04-21T09:01:22.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.050+0000 7f5420ff9640 1 -- 172.21.15.28:0/3485990170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c0042a0 con 0x7f542419cf20 2024-04-21T09:01:22.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.050+0000 7f542959f640 1 -- 172.21.15.28:0/3485990170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54241a5460 con 0x7f542419cf20 2024-04-21T09:01:22.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.052+0000 7f5420ff9640 1 -- 172.21.15.28:0/3485990170 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f541c004440 con 0x7f542419cf20 2024-04-21T09:01:22.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.052+0000 7f542c18a740 1 Processor -- start 2024-04-21T09:01:22.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.052+0000 7f542c18a740 1 -- start start 2024-04-21T09:01:22.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f542c18a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f444134d0 0x558f444138b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f542c18a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f4441c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f542c18a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 0x558f4441ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f542c18a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558f443781f0 con 0x558f44413e80 2024-04-21T09:01:22.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f542c18a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558f443154b0 con 0x558f444134d0 2024-04-21T09:01:22.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f542c18a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558f443741b0 con 0x558f4441ca00 2024-04-21T09:01:22.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f5422ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f4441c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f5422ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f4441c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47214/0 (socket says 172.21.15.28:47214) 2024-04-21T09:01:22.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f5422ffd640 1 -- 172.21.15.28:0/2281557991 learned_addr learned my addr 172.21.15.28:0/2281557991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:22.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f5420ff9640 1 --2- 172.21.15.28:0/3485990170 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f54100a3d20 0x7f54100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f541a7fc640 1 --2- 172.21.15.28:0/2281557991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f444134d0 0x558f444138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f54237fe640 1 --2- 172.21.15.28:0/3485990170 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f54100a3d20 0x7f54100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f5422ffd640 1 -- 172.21.15.28:0/2281557991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 msgr2=0x558f4441ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f541a7fc640 1 --2- 172.21.15.28:0/2281557991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 0x558f4441ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f5422ffd640 1 --2- 172.21.15.28:0/2281557991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 0x558f4441ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f5422ffd640 1 -- 172.21.15.28:0/2281557991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f444134d0 msgr2=0x558f444138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f5422ffd640 1 --2- 172.21.15.28:0/2281557991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f444134d0 0x558f444138b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f5422ffd640 1 -- 172.21.15.28:0/2281557991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f443071b0 con 0x558f44413e80 2024-04-21T09:01:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f541a7fc640 1 --2- 172.21.15.28:0/2281557991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 0x558f4441ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:01:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.053+0000 7f5420ff9640 1 -- 172.21.15.28:0/3485990170 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f541c0f3160 con 0x7f542419cf20 2024-04-21T09:01:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.054+0000 7f5422ffd640 1 --2- 172.21.15.28:0/2281557991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f4441c430 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f541400aa90 tx=0x7f541400af50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e63100bd9e6af4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.054+0000 7f54237fe640 1 --2- 172.21.15.28:0/3485990170 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f54100a3d20 0x7f54100a61e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f541c033000 tx=0x7f541c02cee0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.054+0000 7f54197fa640 1 -- 172.21.15.28:0/2281557991 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541400f450 con 0x558f44413e80 2024-04-21T09:01:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.054+0000 7f54197fa640 1 -- 172.21.15.28:0/2281557991 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f541400faa0 con 0x558f44413e80 2024-04-21T09:01:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.054+0000 7f54197fa640 1 -- 172.21.15.28:0/2281557991 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54140189f0 con 0x558f44413e80 2024-04-21T09:01:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.054+0000 7f542c18a740 1 -- 172.21.15.28:0/2281557991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 msgr2=0x558f4441c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 --2- 172.21.15.28:0/2281557991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f4441c430 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f541400aa90 tx=0x7f541400af50 comp rx=0 tx=0).stop 2024-04-21T09:01:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 -- 172.21.15.28:0/2281557991 shutdown_connections 2024-04-21T09:01:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 --2- 172.21.15.28:0/2281557991 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 0x558f4441ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 --2- 172.21.15.28:0/2281557991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f4441c430 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 --2- 172.21.15.28:0/2281557991 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f444134d0 0x558f444138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 -- 172.21.15.28:0/2281557991 >> 172.21.15.28:0/2281557991 conn(0x558f4440dba0 msgr2=0x558f4440df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 -- 172.21.15.28:0/2281557991 shutdown_connections 2024-04-21T09:01:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 -- 172.21.15.28:0/2281557991 wait complete. 2024-04-21T09:01:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 -- start start 2024-04-21T09:01:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 Processor -- start 2024-04-21T09:01:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f444134d0 0x558f4441c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f44419680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 0x558f44419c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558f4441f970 con 0x558f44413e80 2024-04-21T09:01:22.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x558f44331500 con 0x558f444134d0 2024-04-21T09:01:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x558f4441faf0 con 0x558f4441ca00 2024-04-21T09:01:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f542c18a740 10 client.0 _ll_register_callbacks cb 0x558f442ffb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:01:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f5422ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f44419680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f5422ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f44419680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47224/0 (socket says 172.21.15.28:47224) 2024-04-21T09:01:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.055+0000 7f5422ffd640 1 -- 172.21.15.28:0/2829810532 learned_addr learned my addr 172.21.15.28:0/2829810532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f541a7fc640 1 --2- 172.21.15.28:0/2829810532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 0x558f44419c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f5422ffd640 1 -- 172.21.15.28:0/2829810532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 msgr2=0x558f44419c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f5422ffd640 1 --2- 172.21.15.28:0/2829810532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 0x558f44419c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f5422ffd640 1 -- 172.21.15.28:0/2829810532 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f444134d0 msgr2=0x558f4441c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f5422ffd640 1 --2- 172.21.15.28:0/2829810532 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f444134d0 0x558f4441c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f5422ffd640 1 -- 172.21.15.28:0/2829810532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f4408a3f0 con 0x558f44413e80 2024-04-21T09:01:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f541a7fc640 1 --2- 172.21.15.28:0/2829810532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 0x558f44419c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:01:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f5422ffd640 1 --2- 172.21.15.28:0/2829810532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f44419680 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f5414017000 tx=0x7f5414018540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b670a8bae2eb3f62 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f540a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:01:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f540a7fc640 1 -- 172.21.15.28:0/2829810532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541400f450 con 0x558f44413e80 2024-04-21T09:01:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f540a7fc640 1 -- 172.21.15.28:0/2829810532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f541400f5f0 con 0x558f44413e80 2024-04-21T09:01:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f542c18a740 1 -- 172.21.15.28:0/2829810532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f541400faa0 con 0x558f44413e80 2024-04-21T09:01:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f540a7fc640 1 -- 172.21.15.28:0/2829810532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f541400f7f0 con 0x558f44413e80 2024-04-21T09:01:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f542c18a740 10 client.18189 fetch_fsmap learned FSMap version 28 2024-04-21T09:01:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f542c18a740 1 -- 172.21.15.28:0/2829810532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558f441547d0 con 0x558f44413e80 2024-04-21T09:01:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f540a7fc640 1 -- 172.21.15.28:0/2829810532 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f541400faa0 con 0x558f44413e80 2024-04-21T09:01:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.056+0000 7f540a7fc640 1 -- 172.21.15.28:0/2829810532 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f5414023810 con 0x558f44413e80 2024-04-21T09:01:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.057+0000 7f542c18a740 10 client.18189 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:01:22.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.057+0000 7f542c18a740 10 client.18189 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T09:01:22.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.057+0000 7f542c18a740 20 client.18189 populate_metadata read hostname 'smithi028' 2024-04-21T09:01:22.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.057+0000 7f542c18a740 1 --2- 172.21.15.28:0/2829810532 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558f4442cad0 0x558f4442eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.057+0000 7f542c18a740 4 client.18189 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:01:22.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.057+0000 7f542c18a740 1 -- 172.21.15.28:0/2829810532 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f44317930 con 0x558f4442cad0 2024-04-21T09:01:22.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.057+0000 7f541a7fc640 1 --2- 172.21.15.28:0/2829810532 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558f4442cad0 0x558f4442eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.057+0000 7f541a7fc640 1 --2- 172.21.15.28:0/2829810532 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558f4442cad0 0x558f4442eec0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f9648a6b09e895e1 server_cookie=f1eaf2b428200e50 in_seq=0 out_seq=0 2024-04-21T09:01:22.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.057+0000 7f540a7fc640 10 client.18189 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T09:01:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.058+0000 7f540a7fc640 1 -- 172.21.15.28:0/2829810532 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558f44317930 con 0x558f4442cad0 2024-04-21T09:01:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.058+0000 7f540a7fc640 10 client.18189 handle_command_reply: tid=0 2024-04-21T09:01:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.058+0000 7f542c18a740 1 client.18189 shutdown 2024-04-21T09:01:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.058+0000 7f542c18a740 1 -- 172.21.15.28:0/2829810532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 msgr2=0x558f44419680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.058+0000 7f542c18a740 1 --2- 172.21.15.28:0/2829810532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f44419680 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f5414017000 tx=0x7f5414018540 comp rx=0 tx=0).stop 2024-04-21T09:01:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.058+0000 7f542c18a740 1 -- 172.21.15.28:0/2829810532 shutdown_connections 2024-04-21T09:01:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.058+0000 7f542c18a740 1 --2- 172.21.15.28:0/2829810532 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558f4442cad0 0x558f4442eec0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.058+0000 7f542c18a740 1 --2- 172.21.15.28:0/2829810532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 0x558f44419c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.058+0000 7f542c18a740 1 --2- 172.21.15.28:0/2829810532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f44419680 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.058+0000 7f542c18a740 1 --2- 172.21.15.28:0/2829810532 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f444134d0 0x558f4441c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.058+0000 7f540a7fc640 0 client.18189 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T09:01:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.058+0000 7f542c18a740 1 -- 172.21.15.28:0/2829810532 >> 172.21.15.28:0/2829810532 conn(0x558f4440dba0 msgr2=0x558f4440f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.058+0000 7f542c18a740 1 -- 172.21.15.28:0/2829810532 shutdown_connections 2024-04-21T09:01:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.058+0000 7f542c18a740 1 -- 172.21.15.28:0/2829810532 wait complete. 2024-04-21T09:01:22.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.058+0000 7f542c18a740 20 client.18189 trim_cache size 0 max 16384 2024-04-21T09:01:22.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f542c18a740 1 Processor -- start 2024-04-21T09:01:22.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f542c18a740 1 -- start start 2024-04-21T09:01:22.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f542c18a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f444134d0 0x558f44428860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f542c18a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f44413e80 0x558f4440f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f542c18a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f4441ca00 0x558f4441bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f542c18a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54241b1d30 con 0x558f444134d0 2024-04-21T09:01:22.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f542c18a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f54241af600 con 0x558f4441ca00 2024-04-21T09:01:22.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f542c18a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f54241ad470 con 0x558f44413e80 2024-04-21T09:01:22.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f541a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f444134d0 0x558f44428860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f541a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f444134d0 0x558f44428860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47234/0 (socket says 172.21.15.28:47234) 2024-04-21T09:01:22.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f541a7fc640 1 -- 172.21.15.28:0/1672946004 learned_addr learned my addr 172.21.15.28:0/1672946004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:22.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f5422ffd640 1 --2- 172.21.15.28:0/1672946004 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f44413e80 0x558f4440f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f541a7fc640 1 -- 172.21.15.28:0/1672946004 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f44413e80 msgr2=0x558f4440f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f541a7fc640 1 --2- 172.21.15.28:0/1672946004 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f44413e80 0x558f4440f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f541a7fc640 1 -- 172.21.15.28:0/1672946004 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f4441ca00 msgr2=0x558f4441bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f541a7fc640 1 --2- 172.21.15.28:0/1672946004 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f4441ca00 0x558f4441bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f541a7fc640 1 -- 172.21.15.28:0/1672946004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f44413be0 con 0x558f444134d0 2024-04-21T09:01:22.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.072+0000 7f541a7fc640 1 --2- 172.21.15.28:0/1672946004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f444134d0 0x558f44428860 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x558f4440d4a0 tx=0x7f540c01af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ae4e20ca6148bc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.073+0000 7f5418ff9640 1 -- 172.21.15.28:0/1672946004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c004020 con 0x558f444134d0 2024-04-21T09:01:22.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.073+0000 7f5418ff9640 1 -- 172.21.15.28:0/1672946004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f540c010e50 con 0x558f444134d0 2024-04-21T09:01:22.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.073+0000 7f5418ff9640 1 -- 172.21.15.28:0/1672946004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c01a900 con 0x558f444134d0 2024-04-21T09:01:22.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.073+0000 7f542c18a740 1 -- 172.21.15.28:0/1672946004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f444134d0 msgr2=0x558f44428860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.073+0000 7f542c18a740 1 --2- 172.21.15.28:0/1672946004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f444134d0 0x558f44428860 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x558f4440d4a0 tx=0x7f540c01af90 comp rx=0 tx=0).stop 2024-04-21T09:01:22.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.073+0000 7f542c18a740 1 -- 172.21.15.28:0/1672946004 shutdown_connections 2024-04-21T09:01:22.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.073+0000 7f542c18a740 1 --2- 172.21.15.28:0/1672946004 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f4441ca00 0x558f4441bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.073+0000 7f542c18a740 1 --2- 172.21.15.28:0/1672946004 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f44413e80 0x558f4440f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.073+0000 7f542c18a740 1 --2- 172.21.15.28:0/1672946004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f444134d0 0x558f44428860 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.073+0000 7f542c18a740 1 -- 172.21.15.28:0/1672946004 >> 172.21.15.28:0/1672946004 conn(0x558f4440dba0 msgr2=0x558f4440eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.073+0000 7f542c18a740 1 -- 172.21.15.28:0/1672946004 shutdown_connections 2024-04-21T09:01:22.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.073+0000 7f542c18a740 1 -- 172.21.15.28:0/1672946004 wait complete. 2024-04-21T09:01:22.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f542c18a740 1 -- start start 2024-04-21T09:01:22.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f542c18a740 1 Processor -- start 2024-04-21T09:01:22.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f542c18a740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f444134d0 0x558f44424f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f542c18a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f44415e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f542c18a740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 0x558f44416370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f542c18a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54241adac0 con 0x558f44413e80 2024-04-21T09:01:22.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f542c18a740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f54241aef30 con 0x558f444134d0 2024-04-21T09:01:22.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f542c18a740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f54241ae190 con 0x558f4441ca00 2024-04-21T09:01:22.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f542c18a740 10 client.0 _ll_register_callbacks cb 0x7f5414018400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:01:22.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f5422ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f44415e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f5422ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f44415e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47248/0 (socket says 172.21.15.28:47248) 2024-04-21T09:01:22.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f5422ffd640 1 -- 172.21.15.28:0/2218852088 learned_addr learned my addr 172.21.15.28:0/2218852088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:22.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f541a7fc640 1 --2- 172.21.15.28:0/2218852088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f444134d0 0x558f44424f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f5422ffd640 1 -- 172.21.15.28:0/2218852088 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 msgr2=0x558f44416370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f5422ffd640 1 --2- 172.21.15.28:0/2218852088 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 0x558f44416370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f5422ffd640 1 -- 172.21.15.28:0/2218852088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f444134d0 msgr2=0x558f44424f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f5422ffd640 1 --2- 172.21.15.28:0/2218852088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f444134d0 0x558f44424f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f5422ffd640 1 -- 172.21.15.28:0/2218852088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f444169b0 con 0x558f44413e80 2024-04-21T09:01:22.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f541a7fc640 1 --2- 172.21.15.28:0/2218852088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f444134d0 0x558f44424f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:01:22.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f5422ffd640 1 --2- 172.21.15.28:0/2218852088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f44415e30 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f541400a9b0 tx=0x7f5414005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7822e53b9565792c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.074+0000 7f540a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:01:22.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.075+0000 7f540a7fc640 1 -- 172.21.15.28:0/2218852088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541400f990 con 0x558f44413e80 2024-04-21T09:01:22.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.075+0000 7f540a7fc640 1 -- 172.21.15.28:0/2218852088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f5414023470 con 0x558f44413e80 2024-04-21T09:01:22.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.075+0000 7f542c18a740 1 -- 172.21.15.28:0/2218852088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f540c010e50 con 0x558f44413e80 2024-04-21T09:01:22.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.075+0000 7f540a7fc640 1 -- 172.21.15.28:0/2218852088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f541403a050 con 0x558f44413e80 2024-04-21T09:01:22.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.075+0000 7f542c18a740 10 client.18195 fetch_fsmap learned FSMap version 28 2024-04-21T09:01:22.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.075+0000 7f542c18a740 1 -- 172.21.15.28:0/2218852088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558f44492df0 con 0x558f44413e80 2024-04-21T09:01:22.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.075+0000 7f540a7fc640 1 -- 172.21.15.28:0/2218852088 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f540c010e50 con 0x558f44413e80 2024-04-21T09:01:22.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.075+0000 7f540a7fc640 1 -- 172.21.15.28:0/2218852088 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7f5414018c20 con 0x558f44413e80 2024-04-21T09:01:22.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.075+0000 7f542c18a740 10 client.18195 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:01:22.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.075+0000 7f542c18a740 10 client.18195 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-21T09:01:22.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.075+0000 7f542c18a740 20 client.18195 populate_metadata read hostname 'smithi028' 2024-04-21T09:01:22.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.075+0000 7f542c18a740 1 --2- 172.21.15.28:0/2218852088 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558f444935f0 0x558f44495a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.075+0000 7f542c18a740 4 client.18195 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T09:01:22.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.075+0000 7f542c18a740 1 -- 172.21.15.28:0/2218852088 --> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558f44385c10 con 0x558f444935f0 2024-04-21T09:01:22.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.076+0000 7f541a7fc640 1 --2- 172.21.15.28:0/2218852088 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558f444935f0 0x558f44495a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.076+0000 7f541a7fc640 1 --2- 172.21.15.28:0/2218852088 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558f444935f0 0x558f44495a30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=17edc0ea1471dbd6 server_cookie=74a23bdb3b885fd5 in_seq=0 out_seq=0 2024-04-21T09:01:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.076+0000 7f540a7fc640 10 client.18195 ms_handle_connect on v2:172.21.15.171:6838/3423264764 2024-04-21T09:01:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.076+0000 7f540a7fc640 1 -- 172.21.15.28:0/2218852088 <== mds.1 v2:172.21.15.171:6838/3423264764 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558f44385c10 con 0x558f444935f0 2024-04-21T09:01:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.076+0000 7f540a7fc640 10 client.18195 handle_command_reply: tid=0 2024-04-21T09:01:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.076+0000 7f542c18a740 1 client.18195 shutdown 2024-04-21T09:01:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.076+0000 7f542c18a740 1 -- 172.21.15.28:0/2218852088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 msgr2=0x558f44415e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.076+0000 7f542c18a740 1 --2- 172.21.15.28:0/2218852088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f44415e30 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f541400a9b0 tx=0x7f5414005e10 comp rx=0 tx=0).stop 2024-04-21T09:01:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542c18a740 1 -- 172.21.15.28:0/2218852088 shutdown_connections 2024-04-21T09:01:22.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542c18a740 1 --2- 172.21.15.28:0/2218852088 >> [v2:172.21.15.171:6838/3423264764,v1:172.21.15.171:6839/3423264764] conn(0x558f444935f0 0x558f44495a30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542c18a740 1 --2- 172.21.15.28:0/2218852088 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x558f4441ca00 0x558f44416370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542c18a740 1 --2- 172.21.15.28:0/2218852088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f44413e80 0x558f44415e30 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542c18a740 1 --2- 172.21.15.28:0/2218852088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558f444134d0 0x558f44424f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f540a7fc640 0 client.18195 ms_handle_reset on v2:172.21.15.171:6838/3423264764 2024-04-21T09:01:22.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542c18a740 1 -- 172.21.15.28:0/2218852088 >> 172.21.15.28:0/2218852088 conn(0x558f4440dba0 msgr2=0x558f4440f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542c18a740 1 -- 172.21.15.28:0/2218852088 shutdown_connections 2024-04-21T09:01:22.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542c18a740 1 -- 172.21.15.28:0/2218852088 wait complete. 2024-04-21T09:01:22.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542c18a740 20 client.18195 trim_cache size 0 max 16384 2024-04-21T09:01:22.089 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T09:01:22.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542959f640 1 -- 172.21.15.28:0/3485990170 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f54100a3d20 msgr2=0x7f54100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542959f640 1 --2- 172.21.15.28:0/3485990170 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f54100a3d20 0x7f54100a61e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f541c033000 tx=0x7f541c02cee0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542959f640 1 -- 172.21.15.28:0/3485990170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f542419cf20 msgr2=0x7f542419d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542959f640 1 --2- 172.21.15.28:0/3485990170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f542419cf20 0x7f542419d3d0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f541c0098d0 tx=0x7f541c035d60 comp rx=0 tx=0).stop 2024-04-21T09:01:22.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542959f640 1 -- 172.21.15.28:0/3485990170 shutdown_connections 2024-04-21T09:01:22.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542959f640 1 --2- 172.21.15.28:0/3485990170 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f54100a3d20 0x7f54100a61e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542959f640 1 --2- 172.21.15.28:0/3485990170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f542419cf20 0x7f542419d3d0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542959f640 1 --2- 172.21.15.28:0/3485990170 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5424106160 0x7f5424198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542959f640 1 --2- 172.21.15.28:0/3485990170 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5424101090 0x7f5424198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542959f640 1 -- 172.21.15.28:0/3485990170 >> 172.21.15.28:0/3485990170 conn(0x7f54240771d0 msgr2=0x7f542406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542959f640 1 -- 172.21.15.28:0/3485990170 shutdown_connections 2024-04-21T09:01:22.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.077+0000 7f542959f640 1 -- 172.21.15.28:0/3485990170 wait complete. 2024-04-21T09:01:22.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T09:01:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 -- 172.21.15.28:0/1114719260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc450106160 msgr2=0x7fc45010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 --2- 172.21.15.28:0/1114719260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc450106160 0x7fc45010dd30 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fc448009ca0 tx=0x7fc44802c0f0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 -- 172.21.15.28:0/1114719260 shutdown_connections 2024-04-21T09:01:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 --2- 172.21.15.28:0/1114719260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc450106160 0x7fc45010dd30 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 --2- 172.21.15.28:0/1114719260 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc450101a40 0x7fc450105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 --2- 172.21.15.28:0/1114719260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc450101090 0x7fc450101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 -- 172.21.15.28:0/1114719260 >> 172.21.15.28:0/1114719260 conn(0x7fc450077f50 msgr2=0x7fc450078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 -- 172.21.15.28:0/1114719260 shutdown_connections 2024-04-21T09:01:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 -- 172.21.15.28:0/1114719260 wait complete. 2024-04-21T09:01:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 Processor -- start 2024-04-21T09:01:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 -- start start 2024-04-21T09:01:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc450101090 0x7fc450199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc450106160 0x7fc450199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc45019a060 0x7fc4501a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4501110d0 con 0x7fc45019a060 2024-04-21T09:01:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc450110f50 con 0x7fc450101090 2024-04-21T09:01:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc450111250 con 0x7fc450106160 2024-04-21T09:01:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc4550bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc45019a060 0x7fc4501a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc4550bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc45019a060 0x7fc4501a4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47280/0 (socket says 172.21.15.28:47280) 2024-04-21T09:01:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc4550bb640 1 -- 172.21.15.28:0/2309998949 learned_addr learned my addr 172.21.15.28:0/2309998949 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc4548ba640 1 --2- 172.21.15.28:0/2309998949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc450101090 0x7fc450199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc44ffff640 1 --2- 172.21.15.28:0/2309998949 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc450106160 0x7fc450199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc4550bb640 1 -- 172.21.15.28:0/2309998949 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc450106160 msgr2=0x7fc450199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc4550bb640 1 --2- 172.21.15.28:0/2309998949 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc450106160 0x7fc450199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc4550bb640 1 -- 172.21.15.28:0/2309998949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc450101090 msgr2=0x7fc450199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc4550bb640 1 --2- 172.21.15.28:0/2309998949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc450101090 0x7fc450199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc4550bb640 1 -- 172.21.15.28:0/2309998949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4501a4de0 con 0x7fc45019a060 2024-04-21T09:01:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc4550bb640 1 --2- 172.21.15.28:0/2309998949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc45019a060 0x7fc4501a4610 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fc448005e30 tx=0x7fc448034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc44dffb640 1 -- 172.21.15.28:0/2309998949 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc448039040 con 0x7fc45019a060 2024-04-21T09:01:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc44dffb640 1 -- 172.21.15.28:0/2309998949 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc448044070 con 0x7fc45019a060 2024-04-21T09:01:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc456b1e640 1 -- 172.21.15.28:0/2309998949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4501a5070 con 0x7fc45019a060 2024-04-21T09:01:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.197+0000 7fc44dffb640 1 -- 172.21.15.28:0/2309998949 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc448003e60 con 0x7fc45019a060 2024-04-21T09:01:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.198+0000 7fc456b1e640 1 -- 172.21.15.28:0/2309998949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4501a5560 con 0x7fc45019a060 2024-04-21T09:01:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.200+0000 7fc44dffb640 1 -- 172.21.15.28:0/2309998949 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc4480324c0 con 0x7fc45019a060 2024-04-21T09:01:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.200+0000 7fc4597df740 1 Processor -- start 2024-04-21T09:01:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.200+0000 7fc4597df740 1 -- start start 2024-04-21T09:01:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.200+0000 7fc4597df740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e3134d0 0x56170e3138b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.200+0000 7fc4597df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e31c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.200+0000 7fc4597df740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e31ca00 0x56170e31ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.200+0000 7fc4597df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56170e2781f0 con 0x56170e313e80 2024-04-21T09:01:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.200+0000 7fc4597df740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56170e2154b0 con 0x56170e3134d0 2024-04-21T09:01:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.200+0000 7fc4597df740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56170e2741b0 con 0x56170e31ca00 2024-04-21T09:01:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.200+0000 7fc4548ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e31c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.200+0000 7fc4548ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e31c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47286/0 (socket says 172.21.15.28:47286) 2024-04-21T09:01:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.200+0000 7fc4548ba640 1 -- 172.21.15.28:0/1213234539 learned_addr learned my addr 172.21.15.28:0/1213234539 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.201+0000 7fc44ffff640 1 --2- 172.21.15.28:0/1213234539 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e31ca00 0x56170e31ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.201+0000 7fc44dffb640 1 --2- 172.21.15.28:0/2309998949 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4380a3d20 0x7fc4380a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.201+0000 7fc44ffff640 1 --2- 172.21.15.28:0/1213234539 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e3134d0 0x56170e3138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.201+0000 7fc4548ba640 1 -- 172.21.15.28:0/1213234539 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e31ca00 msgr2=0x56170e31ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.201+0000 7fc4548ba640 1 --2- 172.21.15.28:0/1213234539 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e31ca00 0x56170e31ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.201+0000 7fc4548ba640 1 -- 172.21.15.28:0/1213234539 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e3134d0 msgr2=0x56170e3138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.201+0000 7fc4548ba640 1 --2- 172.21.15.28:0/1213234539 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e3134d0 0x56170e3138b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.201+0000 7fc4548ba640 1 -- 172.21.15.28:0/1213234539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56170e2071b0 con 0x56170e313e80 2024-04-21T09:01:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.201+0000 7fc44ffff640 1 --2- 172.21.15.28:0/1213234539 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e3134d0 0x56170e3138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:01:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.201+0000 7fc44dffb640 1 -- 172.21.15.28:0/2309998949 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7fc4480f2cd0 con 0x7fc45019a060 2024-04-21T09:01:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.201+0000 7fc4550bb640 1 --2- 172.21.15.28:0/2309998949 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4380a3d20 0x7fc4380a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.201+0000 7fc4548ba640 1 --2- 172.21.15.28:0/1213234539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e31c430 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fc44000bcd0 tx=0x7fc4400107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2541db1f78b86c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.201+0000 7fc436ffd640 1 -- 172.21.15.28:0/1213234539 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc440005220 con 0x56170e313e80 2024-04-21T09:01:22.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.201+0000 7fc436ffd640 1 -- 172.21.15.28:0/1213234539 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc440004c50 con 0x56170e313e80 2024-04-21T09:01:22.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.201+0000 7fc436ffd640 1 -- 172.21.15.28:0/1213234539 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4400116a0 con 0x56170e313e80 2024-04-21T09:01:22.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.202+0000 7fc4597df740 1 -- 172.21.15.28:0/1213234539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 msgr2=0x56170e31c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.202+0000 7fc4550bb640 1 --2- 172.21.15.28:0/2309998949 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4380a3d20 0x7fc4380a61e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc4480bb360 tx=0x7fc44808d340 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.202+0000 7fc4597df740 1 --2- 172.21.15.28:0/1213234539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e31c430 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fc44000bcd0 tx=0x7fc4400107a0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.202+0000 7fc4597df740 1 -- 172.21.15.28:0/1213234539 shutdown_connections 2024-04-21T09:01:22.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.202+0000 7fc4597df740 1 --2- 172.21.15.28:0/1213234539 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e31ca00 0x56170e31ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.202+0000 7fc4597df740 1 --2- 172.21.15.28:0/1213234539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e31c430 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.202+0000 7fc4597df740 1 --2- 172.21.15.28:0/1213234539 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e3134d0 0x56170e3138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.202+0000 7fc4597df740 1 -- 172.21.15.28:0/1213234539 >> 172.21.15.28:0/1213234539 conn(0x56170e30dba0 msgr2=0x56170e30df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.202+0000 7fc4597df740 1 -- 172.21.15.28:0/1213234539 shutdown_connections 2024-04-21T09:01:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.202+0000 7fc4597df740 1 -- 172.21.15.28:0/1213234539 wait complete. 2024-04-21T09:01:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.202+0000 7fc4597df740 1 -- start start 2024-04-21T09:01:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.202+0000 7fc4597df740 1 Processor -- start 2024-04-21T09:01:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4597df740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e3134d0 0x56170e31c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4597df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e319680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4597df740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e31ca00 0x56170e319c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4597df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56170e31f970 con 0x56170e313e80 2024-04-21T09:01:22.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4597df740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x56170e231500 con 0x56170e31ca00 2024-04-21T09:01:22.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4597df740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x56170e31faf0 con 0x56170e3134d0 2024-04-21T09:01:22.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4597df740 10 client.0 _ll_register_callbacks cb 0x56170e1ffb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:01:22.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4548ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e319680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4548ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e319680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47296/0 (socket says 172.21.15.28:47296) 2024-04-21T09:01:22.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4548ba640 1 -- 172.21.15.28:0/2614164829 learned_addr learned my addr 172.21.15.28:0/2614164829 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc44ffff640 1 --2- 172.21.15.28:0/2614164829 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e3134d0 0x56170e31c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4548ba640 1 -- 172.21.15.28:0/2614164829 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e3134d0 msgr2=0x56170e31c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4548ba640 1 --2- 172.21.15.28:0/2614164829 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e3134d0 0x56170e31c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc44ffff640 1 --2- 172.21.15.28:0/2614164829 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e31ca00 0x56170e319c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4548ba640 1 -- 172.21.15.28:0/2614164829 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e31ca00 msgr2=0x56170e319c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4548ba640 1 --2- 172.21.15.28:0/2614164829 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e31ca00 0x56170e319c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4548ba640 1 -- 172.21.15.28:0/2614164829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56170df8a3f0 con 0x56170e313e80 2024-04-21T09:01:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4548ba640 1 --2- 172.21.15.28:0/2614164829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e319680 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fc44000bca0 tx=0x7fc4400048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81258b738b393d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc42ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:01:22.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc42ffff640 1 -- 172.21.15.28:0/2614164829 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc440005010 con 0x56170e313e80 2024-04-21T09:01:22.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc42ffff640 1 -- 172.21.15.28:0/2614164829 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7fc4400051b0 con 0x56170e313e80 2024-04-21T09:01:22.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.203+0000 7fc4597df740 1 -- 172.21.15.28:0/2614164829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc440004c50 con 0x56170e313e80 2024-04-21T09:01:22.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.204+0000 7fc42ffff640 1 -- 172.21.15.28:0/2614164829 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc4400053d0 con 0x56170e313e80 2024-04-21T09:01:22.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.204+0000 7fc4597df740 10 client.18207 fetch_fsmap learned FSMap version 28 2024-04-21T09:01:22.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.204+0000 7fc4597df740 1 -- 172.21.15.28:0/2614164829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56170e0547d0 con 0x56170e313e80 2024-04-21T09:01:22.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.204+0000 7fc42ffff640 1 -- 172.21.15.28:0/2614164829 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc440004c50 con 0x56170e313e80 2024-04-21T09:01:22.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.204+0000 7fc42ffff640 1 -- 172.21.15.28:0/2614164829 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fc440011460 con 0x56170e313e80 2024-04-21T09:01:22.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.204+0000 7fc4597df740 10 client.18207 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:01:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.204+0000 7fc4597df740 10 client.18207 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:01:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.204+0000 7fc4597df740 20 client.18207 populate_metadata read hostname 'smithi028' 2024-04-21T09:01:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.204+0000 7fc4597df740 1 --2- 172.21.15.28:0/2614164829 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56170e32caa0 0x56170e32ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.204+0000 7fc4597df740 4 client.18207 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T09:01:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.204+0000 7fc4597df740 1 -- 172.21.15.28:0/2614164829 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56170e217930 con 0x56170e32caa0 2024-04-21T09:01:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.204+0000 7fc44ffff640 1 --2- 172.21.15.28:0/2614164829 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56170e32caa0 0x56170e32ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.205+0000 7fc44ffff640 1 --2- 172.21.15.28:0/2614164829 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56170e32caa0 0x56170e32ee90 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7cf4ed79748d0b5 server_cookie=39e06bc812073e7a in_seq=0 out_seq=0 2024-04-21T09:01:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.205+0000 7fc42ffff640 10 client.18207 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:01:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.206+0000 7fc42ffff640 1 -- 172.21.15.28:0/2614164829 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56170e217930 con 0x56170e32caa0 2024-04-21T09:01:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.206+0000 7fc42ffff640 10 client.18207 handle_command_reply: tid=0 2024-04-21T09:01:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.206+0000 7fc4597df740 1 client.18207 shutdown 2024-04-21T09:01:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.206+0000 7fc4597df740 1 -- 172.21.15.28:0/2614164829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 msgr2=0x56170e319680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.206+0000 7fc4597df740 1 --2- 172.21.15.28:0/2614164829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e319680 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fc44000bca0 tx=0x7fc4400048f0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.206+0000 7fc4597df740 1 -- 172.21.15.28:0/2614164829 shutdown_connections 2024-04-21T09:01:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.206+0000 7fc4597df740 1 --2- 172.21.15.28:0/2614164829 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56170e32caa0 0x56170e32ee90 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.206+0000 7fc4597df740 1 --2- 172.21.15.28:0/2614164829 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e31ca00 0x56170e319c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.206+0000 7fc42ffff640 0 client.18207 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:01:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.206+0000 7fc4597df740 1 --2- 172.21.15.28:0/2614164829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e319680 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.206+0000 7fc4597df740 1 --2- 172.21.15.28:0/2614164829 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e3134d0 0x56170e31c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.206+0000 7fc4597df740 1 -- 172.21.15.28:0/2614164829 >> 172.21.15.28:0/2614164829 conn(0x56170e30dba0 msgr2=0x56170e30f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.206+0000 7fc4597df740 1 -- 172.21.15.28:0/2614164829 shutdown_connections 2024-04-21T09:01:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.206+0000 7fc4597df740 1 -- 172.21.15.28:0/2614164829 wait complete. 2024-04-21T09:01:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.206+0000 7fc4597df740 20 client.18207 trim_cache size 0 max 16384 2024-04-21T09:01:22.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.219+0000 7fc4597df740 1 Processor -- start 2024-04-21T09:01:22.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.219+0000 7fc4597df740 1 -- start start 2024-04-21T09:01:22.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4597df740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e3134d0 0x56170e325120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4597df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e31bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4597df740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e31ca00 0x56170e31c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4597df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4501b1db0 con 0x56170e313e80 2024-04-21T09:01:22.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4597df740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc4501af680 con 0x56170e3134d0 2024-04-21T09:01:22.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4597df740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc4501ad4f0 con 0x56170e31ca00 2024-04-21T09:01:22.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4548ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e31bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4548ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e31bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47308/0 (socket says 172.21.15.28:47308) 2024-04-21T09:01:22.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4548ba640 1 -- 172.21.15.28:0/2811044144 learned_addr learned my addr 172.21.15.28:0/2811044144 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:22.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc44ffff640 1 --2- 172.21.15.28:0/2811044144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e3134d0 0x56170e325120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc44ffff640 1 --2- 172.21.15.28:0/2811044144 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e31ca00 0x56170e31c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4548ba640 1 -- 172.21.15.28:0/2811044144 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e31ca00 msgr2=0x56170e31c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4548ba640 1 --2- 172.21.15.28:0/2811044144 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e31ca00 0x56170e31c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4548ba640 1 -- 172.21.15.28:0/2811044144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e3134d0 msgr2=0x56170e325120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4548ba640 1 --2- 172.21.15.28:0/2811044144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e3134d0 0x56170e325120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4548ba640 1 -- 172.21.15.28:0/2811044144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56170e317bd0 con 0x56170e313e80 2024-04-21T09:01:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc44ffff640 1 --2- 172.21.15.28:0/2811044144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e3134d0 0x56170e325120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:01:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4548ba640 1 --2- 172.21.15.28:0/2811044144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e31bdb0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fc440005d00 tx=0x7fc440005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df2be9effb72964 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4367fc640 1 -- 172.21.15.28:0/2811044144 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44000b810 con 0x56170e313e80 2024-04-21T09:01:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4367fc640 1 -- 172.21.15.28:0/2811044144 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc440011440 con 0x56170e313e80 2024-04-21T09:01:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.220+0000 7fc4367fc640 1 -- 172.21.15.28:0/2811044144 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc440004c50 con 0x56170e313e80 2024-04-21T09:01:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.221+0000 7fc4597df740 1 -- 172.21.15.28:0/2811044144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 msgr2=0x56170e31bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.221+0000 7fc4597df740 1 --2- 172.21.15.28:0/2811044144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e31bdb0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fc440005d00 tx=0x7fc440005850 comp rx=0 tx=0).stop 2024-04-21T09:01:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.221+0000 7fc4597df740 1 -- 172.21.15.28:0/2811044144 shutdown_connections 2024-04-21T09:01:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.221+0000 7fc4597df740 1 --2- 172.21.15.28:0/2811044144 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e31ca00 0x56170e31c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.221+0000 7fc4597df740 1 --2- 172.21.15.28:0/2811044144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e313e80 0x56170e31bdb0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.221+0000 7fc4597df740 1 --2- 172.21.15.28:0/2811044144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e3134d0 0x56170e325120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.221+0000 7fc4597df740 1 -- 172.21.15.28:0/2811044144 >> 172.21.15.28:0/2811044144 conn(0x56170e30dba0 msgr2=0x56170e314260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.221+0000 7fc4597df740 1 -- 172.21.15.28:0/2811044144 shutdown_connections 2024-04-21T09:01:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.221+0000 7fc4597df740 1 -- 172.21.15.28:0/2811044144 wait complete. 2024-04-21T09:01:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.221+0000 7fc4597df740 1 -- start start 2024-04-21T09:01:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.221+0000 7fc4597df740 1 Processor -- start 2024-04-21T09:01:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc4597df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e3134d0 0x56170e3158f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc4597df740 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e313e80 0x56170e315e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc4597df740 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e31ca00 0x56170e316370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc4597df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4501adb40 con 0x56170e3134d0 2024-04-21T09:01:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc4597df740 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fc4501aefb0 con 0x56170e313e80 2024-04-21T09:01:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc4597df740 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fc4501ae210 con 0x56170e31ca00 2024-04-21T09:01:22.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc4597df740 10 client.0 _ll_register_callbacks cb 0x7fc44000bef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T09:01:22.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc44ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e3134d0 0x56170e3158f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc44ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e3134d0 0x56170e3158f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47324/0 (socket says 172.21.15.28:47324) 2024-04-21T09:01:22.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc44ffff640 1 -- 172.21.15.28:0/1760770895 learned_addr learned my addr 172.21.15.28:0/1760770895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:22.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc44ffff640 1 --2- 172.21.15.28:0/1760770895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e31ca00 0x56170e316370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc44ffff640 1 -- 172.21.15.28:0/1760770895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e31ca00 msgr2=0x56170e316370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc4548ba640 1 --2- 172.21.15.28:0/1760770895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e313e80 0x56170e315e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc44ffff640 1 --2- 172.21.15.28:0/1760770895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e31ca00 0x56170e316370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc44ffff640 1 -- 172.21.15.28:0/1760770895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e313e80 msgr2=0x56170e315e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T09:01:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc44ffff640 1 --2- 172.21.15.28:0/1760770895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e313e80 0x56170e315e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc44ffff640 1 -- 172.21.15.28:0/1760770895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56170e316b70 con 0x56170e3134d0 2024-04-21T09:01:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc4548ba640 1 --2- 172.21.15.28:0/1760770895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e313e80 0x56170e315e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:01:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc44ffff640 1 --2- 172.21.15.28:0/1760770895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e3134d0 0x56170e3158f0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fc444004530 tx=0x7fc44400b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0d91d0c38d8a99 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.222+0000 7fc42ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-21T09:01:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.223+0000 7fc42ffff640 1 -- 172.21.15.28:0/1760770895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc444028040 con 0x56170e3134d0 2024-04-21T09:01:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.223+0000 7fc42ffff640 1 -- 172.21.15.28:0/1760770895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7fc444003c00 con 0x56170e3134d0 2024-04-21T09:01:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.223+0000 7fc4597df740 1 -- 172.21.15.28:0/1760770895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc440011440 con 0x56170e3134d0 2024-04-21T09:01:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.223+0000 7fc42ffff640 1 -- 172.21.15.28:0/1760770895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc444034050 con 0x56170e3134d0 2024-04-21T09:01:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.223+0000 7fc4597df740 10 client.18213 fetch_fsmap learned FSMap version 28 2024-04-21T09:01:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.223+0000 7fc4597df740 1 -- 172.21.15.28:0/1760770895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56170e3973a0 con 0x56170e3134d0 2024-04-21T09:01:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.223+0000 7fc42ffff640 1 -- 172.21.15.28:0/1760770895 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc440011440 con 0x56170e3134d0 2024-04-21T09:01:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.223+0000 7fc42ffff640 1 -- 172.21.15.28:0/1760770895 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8346+0+0 (secure 0 0 0) 0x7fc44401b030 con 0x56170e3134d0 2024-04-21T09:01:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.223+0000 7fc4597df740 10 client.18213 fetch_fsmap finished waiting for FSMap version 28 2024-04-21T09:01:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.223+0000 7fc4597df740 10 client.18213 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T09:01:22.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.223+0000 7fc4597df740 20 client.18213 populate_metadata read hostname 'smithi028' 2024-04-21T09:01:22.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.223+0000 7fc4597df740 1 --2- 172.21.15.28:0/1760770895 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56170e397b00 0x56170e399f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.223+0000 7fc4597df740 4 client.18213 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T09:01:22.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.223+0000 7fc4597df740 1 -- 172.21.15.28:0/1760770895 --> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56170e285c10 con 0x56170e397b00 2024-04-21T09:01:22.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.224+0000 7fc4548ba640 1 --2- 172.21.15.28:0/1760770895 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56170e397b00 0x56170e399f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.224+0000 7fc4548ba640 1 --2- 172.21.15.28:0/1760770895 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56170e397b00 0x56170e399f20 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1ef13ab462f35c8 server_cookie=3010c7387acc346f in_seq=0 out_seq=0 2024-04-21T09:01:22.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.224+0000 7fc42ffff640 10 client.18213 ms_handle_connect on v2:172.21.15.152:6838/3821365920 2024-04-21T09:01:22.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.224+0000 7fc42ffff640 1 -- 172.21.15.28:0/1760770895 <== mds.0 v2:172.21.15.152:6838/3821365920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56170e285c10 con 0x56170e397b00 2024-04-21T09:01:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.224+0000 7fc42ffff640 10 client.18213 handle_command_reply: tid=0 2024-04-21T09:01:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.224+0000 7fc4597df740 1 client.18213 shutdown 2024-04-21T09:01:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.224+0000 7fc4597df740 1 -- 172.21.15.28:0/1760770895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e3134d0 msgr2=0x56170e3158f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.224+0000 7fc4597df740 1 --2- 172.21.15.28:0/1760770895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e3134d0 0x56170e3158f0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fc444004530 tx=0x7fc44400b6e0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.224+0000 7fc4597df740 1 -- 172.21.15.28:0/1760770895 shutdown_connections 2024-04-21T09:01:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.224+0000 7fc4597df740 1 --2- 172.21.15.28:0/1760770895 >> [v2:172.21.15.152:6838/3821365920,v1:172.21.15.152:6839/3821365920] conn(0x56170e397b00 0x56170e399f20 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.224+0000 7fc4597df740 1 --2- 172.21.15.28:0/1760770895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x56170e31ca00 0x56170e316370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.224+0000 7fc4597df740 1 --2- 172.21.15.28:0/1760770895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56170e313e80 0x56170e315e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.224+0000 7fc42ffff640 0 client.18213 ms_handle_reset on v2:172.21.15.152:6838/3821365920 2024-04-21T09:01:22.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.224+0000 7fc4597df740 1 --2- 172.21.15.28:0/1760770895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170e3134d0 0x56170e3158f0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.224+0000 7fc4597df740 1 -- 172.21.15.28:0/1760770895 >> 172.21.15.28:0/1760770895 conn(0x56170e30dba0 msgr2=0x56170e314af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.225+0000 7fc4597df740 1 -- 172.21.15.28:0/1760770895 shutdown_connections 2024-04-21T09:01:22.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.225+0000 7fc4597df740 1 -- 172.21.15.28:0/1760770895 wait complete. 2024-04-21T09:01:22.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.225+0000 7fc4597df740 20 client.18213 trim_cache size 0 max 16384 2024-04-21T09:01:22.236 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-21T09:01:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.225+0000 7fc456b1e640 1 -- 172.21.15.28:0/2309998949 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4380a3d20 msgr2=0x7fc4380a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.225+0000 7fc456b1e640 1 --2- 172.21.15.28:0/2309998949 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4380a3d20 0x7fc4380a61e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc4480bb360 tx=0x7fc44808d340 comp rx=0 tx=0).stop 2024-04-21T09:01:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.225+0000 7fc456b1e640 1 -- 172.21.15.28:0/2309998949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc45019a060 msgr2=0x7fc4501a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.225+0000 7fc456b1e640 1 --2- 172.21.15.28:0/2309998949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc45019a060 0x7fc4501a4610 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fc448005e30 tx=0x7fc448034760 comp rx=0 tx=0).stop 2024-04-21T09:01:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.225+0000 7fc456b1e640 1 -- 172.21.15.28:0/2309998949 shutdown_connections 2024-04-21T09:01:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.225+0000 7fc456b1e640 1 --2- 172.21.15.28:0/2309998949 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fc4380a3d20 0x7fc4380a61e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.225+0000 7fc456b1e640 1 --2- 172.21.15.28:0/2309998949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc45019a060 0x7fc4501a4610 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.225+0000 7fc456b1e640 1 --2- 172.21.15.28:0/2309998949 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc450106160 0x7fc450199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.225+0000 7fc456b1e640 1 --2- 172.21.15.28:0/2309998949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc450101090 0x7fc450199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.225+0000 7fc456b1e640 1 -- 172.21.15.28:0/2309998949 >> 172.21.15.28:0/2309998949 conn(0x7fc450077f50 msgr2=0x7fc4500ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.225+0000 7fc456b1e640 1 -- 172.21.15.28:0/2309998949 shutdown_connections 2024-04-21T09:01:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.225+0000 7fc456b1e640 1 -- 172.21.15.28:0/2309998949 wait complete. 2024-04-21T09:01:22.239 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/thrasher.py", line 43, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-04-21T09:01:22.248 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 155, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 90, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: 2024-04-21T09:01:22.255 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-21T09:01:22.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T09:01:22.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:22 smithi028 ceph-mon[28003]: from='client.18129 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T09:01:22.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:22 smithi028 ceph-mon[28003]: pgmap v316: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T09:01:22.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:22 smithi028 ceph-mon[28003]: from='client.18135 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T09:01:22.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:22 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/395377265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T09:01:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.379+0000 7f6879a48640 1 -- 172.21.15.28:0/535760607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6874069aa0 msgr2=0x7f687410ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.379+0000 7f6879a48640 1 --2- 172.21.15.28:0/535760607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6874069aa0 0x7f687410ede0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f686c009ca0 tx=0x7f686c02c0f0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.379+0000 7f6879a48640 1 -- 172.21.15.28:0/535760607 shutdown_connections 2024-04-21T09:01:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.379+0000 7f6879a48640 1 --2- 172.21.15.28:0/535760607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6874069aa0 0x7f687410ede0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.379+0000 7f6879a48640 1 --2- 172.21.15.28:0/535760607 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f68740690e0 0x7f6874069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.379+0000 7f6879a48640 1 --2- 172.21.15.28:0/535760607 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6874106f50 0x7f6874107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.379+0000 7f6879a48640 1 -- 172.21.15.28:0/535760607 >> 172.21.15.28:0/535760607 conn(0x7f6874077e30 msgr2=0x7f6874078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.379+0000 7f6879a48640 1 -- 172.21.15.28:0/535760607 shutdown_connections 2024-04-21T09:01:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.379+0000 7f6879a48640 1 -- 172.21.15.28:0/535760607 wait complete. 2024-04-21T09:01:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.379+0000 7f6879a48640 1 Processor -- start 2024-04-21T09:01:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.379+0000 7f6879a48640 1 -- start start 2024-04-21T09:01:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f6879a48640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68740690e0 0x7f68741986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f6879a48640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6874069aa0 0x7f6874198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f6879a48640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6874106f50 0x7f687419cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f6879a48640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f68741121d0 con 0x7f68740690e0 2024-04-21T09:01:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f6879a48640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f6874112050 con 0x7f6874106f50 2024-04-21T09:01:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f6879a48640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f6874112350 con 0x7f6874069aa0 2024-04-21T09:01:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f6872ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68740690e0 0x7f68741986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f6872ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68740690e0 0x7f68741986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47340/0 (socket says 172.21.15.28:47340) 2024-04-21T09:01:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f6872ffd640 1 -- 172.21.15.28:0/1059068206 learned_addr learned my addr 172.21.15.28:0/1059068206 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f68727fc640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6874069aa0 0x7f6874198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f68737fe640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6874106f50 0x7f687419cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f6872ffd640 1 -- 172.21.15.28:0/1059068206 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6874069aa0 msgr2=0x7f6874198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f6872ffd640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6874069aa0 0x7f6874198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f6872ffd640 1 -- 172.21.15.28:0/1059068206 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6874106f50 msgr2=0x7f687419cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f6872ffd640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6874106f50 0x7f687419cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f6872ffd640 1 -- 172.21.15.28:0/1059068206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f687419d650 con 0x7f68740690e0 2024-04-21T09:01:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f68727fc640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6874069aa0 0x7f6874198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T09:01:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f68737fe640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6874106f50 0x7f687419cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:01:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.380+0000 7f6872ffd640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68740690e0 0x7f68741986b0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f686400c890 tx=0x7f686400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.381+0000 7f6878a46640 1 -- 172.21.15.28:0/1059068206 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6864004220 con 0x7f68740690e0 2024-04-21T09:01:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.381+0000 7f6878a46640 1 -- 172.21.15.28:0/1059068206 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68640043c0 con 0x7f68740690e0 2024-04-21T09:01:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.381+0000 7f6879a48640 1 -- 172.21.15.28:0/1059068206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f687419d940 con 0x7f68740690e0 2024-04-21T09:01:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.381+0000 7f6878a46640 1 -- 172.21.15.28:0/1059068206 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6864004a00 con 0x7f68740690e0 2024-04-21T09:01:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.381+0000 7f6879a48640 1 -- 172.21.15.28:0/1059068206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68741a51d0 con 0x7f68740690e0 2024-04-21T09:01:22.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.382+0000 7f6879a48640 1 -- 172.21.15.28:0/1059068206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6874068de0 con 0x7f68740690e0 2024-04-21T09:01:22.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.383+0000 7f6878a46640 1 -- 172.21.15.28:0/1059068206 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f68640079b0 con 0x7f68740690e0 2024-04-21T09:01:22.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.383+0000 7f6878a46640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f685c0a3d20 0x7f685c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.384+0000 7f68727fc640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f685c0a3d20 0x7f685c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.384+0000 7f6878a46640 1 -- 172.21.15.28:0/1059068206 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f68640d0880 con 0x7f68740690e0 2024-04-21T09:01:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.384+0000 7f68727fc640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f685c0a3d20 0x7f685c0a61e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6874100c60 tx=0x7f6868009110 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.388+0000 7f6878a46640 1 -- 172.21.15.28:0/1059068206 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6864014030 con 0x7f68740690e0 2024-04-21T09:01:22.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.580+0000 7f6879a48640 1 -- 172.21.15.28:0/1059068206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f68741999b0 con 0x7f68740690e0 2024-04-21T09:01:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.581+0000 7f6878a46640 1 -- 172.21.15.28:0/1059068206 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+25346 (secure 0 0 0) 0x7f686409cb30 con 0x7f68740690e0 2024-04-21T09:01:22.582 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T09:01:22.582 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":102,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T09:01:18.669149+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1076282159":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6800/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/926389767":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/892059008":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/2132574390":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T09:01:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.584+0000 7f6879a48640 1 -- 172.21.15.28:0/1059068206 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f685c0a3d20 msgr2=0x7f685c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.584+0000 7f6879a48640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f685c0a3d20 0x7f685c0a61e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6874100c60 tx=0x7f6868009110 comp rx=0 tx=0).stop 2024-04-21T09:01:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.584+0000 7f6879a48640 1 -- 172.21.15.28:0/1059068206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68740690e0 msgr2=0x7f68741986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.584+0000 7f6879a48640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68740690e0 0x7f68741986b0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f686400c890 tx=0x7f686400cc90 comp rx=0 tx=0).stop 2024-04-21T09:01:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.584+0000 7f6879a48640 1 -- 172.21.15.28:0/1059068206 shutdown_connections 2024-04-21T09:01:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.584+0000 7f6879a48640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f685c0a3d20 0x7f685c0a61e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.584+0000 7f6879a48640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6874106f50 0x7f687419cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.584+0000 7f6879a48640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6874069aa0 0x7f6874198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.584+0000 7f6879a48640 1 --2- 172.21.15.28:0/1059068206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68740690e0 0x7f68741986b0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.584+0000 7f6879a48640 1 -- 172.21.15.28:0/1059068206 >> 172.21.15.28:0/1059068206 conn(0x7f6874077e30 msgr2=0x7f68740fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.584+0000 7f6879a48640 1 -- 172.21.15.28:0/1059068206 shutdown_connections 2024-04-21T09:01:22.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.584+0000 7f6879a48640 1 -- 172.21.15.28:0/1059068206 wait complete. 2024-04-21T09:01:22.595 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:48:43.543799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:48:44.864564+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:48:48.747842+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T09:01:22.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T09:01:22.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:22 smithi152 ceph-mon[31401]: from='client.18129 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T09:01:22.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:22 smithi152 ceph-mon[31401]: pgmap v316: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T09:01:22.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:22 smithi152 ceph-mon[31401]: from='client.18135 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T09:01:22.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:22 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/395377265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T09:01:22.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:22 smithi171 ceph-mon[31358]: from='client.18129 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T09:01:22.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:22 smithi171 ceph-mon[31358]: pgmap v316: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-21T09:01:22.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:22 smithi171 ceph-mon[31358]: from='client.18135 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T09:01:22.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:22 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/395377265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T09:01:22.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.701+0000 7fde4205c640 1 -- 172.21.15.28:0/1433262012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c107260 msgr2=0x7fde3c10ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.701+0000 7fde4205c640 1 --2- 172.21.15.28:0/1433262012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c107260 0x7fde3c10ee30 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fde34009ca0 tx=0x7fde3402c0f0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.701+0000 7fde4205c640 1 -- 172.21.15.28:0/1433262012 shutdown_connections 2024-04-21T09:01:22.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.701+0000 7fde4205c640 1 --2- 172.21.15.28:0/1433262012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c107260 0x7fde3c10ee30 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.701+0000 7fde4205c640 1 --2- 172.21.15.28:0/1433262012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde3c102b40 0x7fde3c106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.701+0000 7fde4205c640 1 --2- 172.21.15.28:0/1433262012 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde3c102190 0x7fde3c102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.701+0000 7fde4205c640 1 -- 172.21.15.28:0/1433262012 >> 172.21.15.28:0/1433262012 conn(0x7fde3c0793a0 msgr2=0x7fde3c0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.701+0000 7fde4205c640 1 -- 172.21.15.28:0/1433262012 shutdown_connections 2024-04-21T09:01:22.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.701+0000 7fde4205c640 1 -- 172.21.15.28:0/1433262012 wait complete. 2024-04-21T09:01:22.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde4205c640 1 Processor -- start 2024-04-21T09:01:22.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde4205c640 1 -- start start 2024-04-21T09:01:22.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde4205c640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde3c102190 0x7fde3c19e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde4205c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c107260 0x7fde3c19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde4205c640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde3c198590 0x7fde3c198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde4205c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c1121d0 con 0x7fde3c107260 2024-04-21T09:01:22.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde4205c640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c112050 con 0x7fde3c198590 2024-04-21T09:01:22.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde4205c640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c112350 con 0x7fde3c102190 2024-04-21T09:01:22.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde3affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c107260 0x7fde3c19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde3affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c107260 0x7fde3c19e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47364/0 (socket says 172.21.15.28:47364) 2024-04-21T09:01:22.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde3affd640 1 -- 172.21.15.28:0/3985624294 learned_addr learned my addr 172.21.15.28:0/3985624294 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:22.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde3b7fe640 1 --2- 172.21.15.28:0/3985624294 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde3c102190 0x7fde3c19e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde3bfff640 1 --2- 172.21.15.28:0/3985624294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde3c198590 0x7fde3c198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde3affd640 1 -- 172.21.15.28:0/3985624294 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde3c102190 msgr2=0x7fde3c19e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde3affd640 1 --2- 172.21.15.28:0/3985624294 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde3c102190 0x7fde3c19e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde3affd640 1 -- 172.21.15.28:0/3985624294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde3c198590 msgr2=0x7fde3c198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde3affd640 1 --2- 172.21.15.28:0/3985624294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde3c198590 0x7fde3c198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.702+0000 7fde3affd640 1 -- 172.21.15.28:0/3985624294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3c1990b0 con 0x7fde3c107260 2024-04-21T09:01:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.703+0000 7fde3affd640 1 --2- 172.21.15.28:0/3985624294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c107260 0x7fde3c19e920 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fde3000c970 tx=0x7fde3000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.703+0000 7fde38ff9640 1 -- 172.21.15.28:0/3985624294 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde30007bb0 con 0x7fde3c107260 2024-04-21T09:01:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.703+0000 7fde38ff9640 1 -- 172.21.15.28:0/3985624294 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde30007d50 con 0x7fde3c107260 2024-04-21T09:01:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.703+0000 7fde4205c640 1 -- 172.21.15.28:0/3985624294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3c1993a0 con 0x7fde3c107260 2024-04-21T09:01:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.703+0000 7fde38ff9640 1 -- 172.21.15.28:0/3985624294 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3000f450 con 0x7fde3c107260 2024-04-21T09:01:22.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.703+0000 7fde4205c640 1 -- 172.21.15.28:0/3985624294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde3c1a55f0 con 0x7fde3c107260 2024-04-21T09:01:22.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.704+0000 7fde4205c640 1 -- 172.21.15.28:0/3985624294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde3c0691c0 con 0x7fde3c107260 2024-04-21T09:01:22.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.705+0000 7fde38ff9640 1 -- 172.21.15.28:0/3985624294 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fde30004f20 con 0x7fde3c107260 2024-04-21T09:01:22.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.706+0000 7fde38ff9640 1 --2- 172.21.15.28:0/3985624294 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fde240a3d20 0x7fde240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:22.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.706+0000 7fde3b7fe640 1 --2- 172.21.15.28:0/3985624294 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fde240a3d20 0x7fde240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:22.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.706+0000 7fde38ff9640 1 -- 172.21.15.28:0/3985624294 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7fde300d0c20 con 0x7fde3c107260 2024-04-21T09:01:22.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.706+0000 7fde3b7fe640 1 --2- 172.21.15.28:0/3985624294 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fde240a3d20 0x7fde240a61e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fde2c005fd0 tx=0x7fde2c00b390 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:22.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.710+0000 7fde38ff9640 1 -- 172.21.15.28:0/3985624294 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde30014030 con 0x7fde3c107260 2024-04-21T09:01:22.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.898+0000 7fde4205c640 1 -- 172.21.15.28:0/3985624294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fde3c102b40 con 0x7fde3c107260 2024-04-21T09:01:22.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.899+0000 7fde38ff9640 1 -- 172.21.15.28:0/3985624294 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) ==== 93+0+10 (secure 0 0 0) 0x7fde3009ce70 con 0x7fde3c107260 2024-04-21T09:01:22.899 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T09:01:22.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.901+0000 7fde4205c640 1 -- 172.21.15.28:0/3985624294 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fde240a3d20 msgr2=0x7fde240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.901+0000 7fde4205c640 1 --2- 172.21.15.28:0/3985624294 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fde240a3d20 0x7fde240a61e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fde2c005fd0 tx=0x7fde2c00b390 comp rx=0 tx=0).stop 2024-04-21T09:01:22.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.901+0000 7fde4205c640 1 -- 172.21.15.28:0/3985624294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c107260 msgr2=0x7fde3c19e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:22.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.901+0000 7fde4205c640 1 --2- 172.21.15.28:0/3985624294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c107260 0x7fde3c19e920 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fde3000c970 tx=0x7fde3000ce30 comp rx=0 tx=0).stop 2024-04-21T09:01:22.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.901+0000 7fde4205c640 1 -- 172.21.15.28:0/3985624294 shutdown_connections 2024-04-21T09:01:22.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.901+0000 7fde4205c640 1 --2- 172.21.15.28:0/3985624294 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fde240a3d20 0x7fde240a61e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.901+0000 7fde4205c640 1 --2- 172.21.15.28:0/3985624294 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde3c198590 0x7fde3c198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.901+0000 7fde4205c640 1 --2- 172.21.15.28:0/3985624294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde3c107260 0x7fde3c19e920 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.901+0000 7fde4205c640 1 --2- 172.21.15.28:0/3985624294 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde3c102190 0x7fde3c19e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:22.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.901+0000 7fde4205c640 1 -- 172.21.15.28:0/3985624294 >> 172.21.15.28:0/3985624294 conn(0x7fde3c0793a0 msgr2=0x7fde3c0ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:22.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.901+0000 7fde4205c640 1 -- 172.21.15.28:0/3985624294 shutdown_connections 2024-04-21T09:01:22.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:22.901+0000 7fde4205c640 1 -- 172.21.15.28:0/3985624294 wait complete. 2024-04-21T09:01:22.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T09:01:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.016+0000 7f2bcdd23640 1 -- 172.21.15.28:0/2517742092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc810c510 msgr2=0x7f2bc810e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.016+0000 7f2bcdd23640 1 --2- 172.21.15.28:0/2517742092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc810c510 0x7f2bc810e9b0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0009ca0 tx=0x7f2bc002c210 comp rx=0 tx=0).stop 2024-04-21T09:01:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.016+0000 7f2bcdd23640 1 -- 172.21.15.28:0/2517742092 shutdown_connections 2024-04-21T09:01:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.016+0000 7f2bcdd23640 1 --2- 172.21.15.28:0/2517742092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc810c510 0x7f2bc810e9b0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.016+0000 7f2bcdd23640 1 --2- 172.21.15.28:0/2517742092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bc8103af0 0x7f2bc810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.016+0000 7f2bcdd23640 1 --2- 172.21.15.28:0/2517742092 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2bc81031d0 0x7f2bc81035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.016+0000 7f2bcdd23640 1 -- 172.21.15.28:0/2517742092 >> 172.21.15.28:0/2517742092 conn(0x7f2bc8077f50 msgr2=0x7f2bc8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.017+0000 7f2bcdd23640 1 -- 172.21.15.28:0/2517742092 shutdown_connections 2024-04-21T09:01:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.017+0000 7f2bcdd23640 1 -- 172.21.15.28:0/2517742092 wait complete. 2024-04-21T09:01:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.017+0000 7f2bcdd23640 1 Processor -- start 2024-04-21T09:01:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.017+0000 7f2bcdd23640 1 -- start start 2024-04-21T09:01:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.017+0000 7f2bcdd23640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bc81031d0 0x7f2bc81985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.017+0000 7f2bcdd23640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc8103af0 0x7f2bc8198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.017+0000 7f2bcdd23640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2bc810c510 0x7f2bc819ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.017+0000 7f2bcdd23640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc8112180 con 0x7f2bc8103af0 2024-04-21T09:01:23.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.017+0000 7f2bcdd23640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc8112000 con 0x7f2bc81031d0 2024-04-21T09:01:23.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.017+0000 7f2bcdd23640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc8112300 con 0x7f2bc810c510 2024-04-21T09:01:23.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc8103af0 0x7f2bc8198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:23.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc8103af0 0x7f2bc8198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47378/0 (socket says 172.21.15.28:47378) 2024-04-21T09:01:23.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc6ffd640 1 -- 172.21.15.28:0/158062892 learned_addr learned my addr 172.21.15.28:0/158062892 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:23.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc77fe640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bc81031d0 0x7f2bc81985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:23.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc7fff640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2bc810c510 0x7f2bc819ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:23.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc6ffd640 1 -- 172.21.15.28:0/158062892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2bc810c510 msgr2=0x7f2bc819ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc6ffd640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2bc810c510 0x7f2bc819ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc6ffd640 1 -- 172.21.15.28:0/158062892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bc81031d0 msgr2=0x7f2bc81985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc6ffd640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bc81031d0 0x7f2bc81985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc6ffd640 1 -- 172.21.15.28:0/158062892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc819d590 con 0x7f2bc8103af0 2024-04-21T09:01:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc7fff640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2bc810c510 0x7f2bc819ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:01:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc77fe640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bc81031d0 0x7f2bc81985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:01:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc6ffd640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc8103af0 0x7f2bc8198ae0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00c920 tx=0x7f2bbc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:23.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc4ff9640 1 -- 172.21.15.28:0/158062892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc007c70 con 0x7f2bc8103af0 2024-04-21T09:01:23.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc4ff9640 1 -- 172.21.15.28:0/158062892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bbc007e10 con 0x7f2bc8103af0 2024-04-21T09:01:23.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bcdd23640 1 -- 172.21.15.28:0/158062892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc819d880 con 0x7f2bc8103af0 2024-04-21T09:01:23.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bc4ff9640 1 -- 172.21.15.28:0/158062892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc018820 con 0x7f2bc8103af0 2024-04-21T09:01:23.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.018+0000 7f2bcdd23640 1 -- 172.21.15.28:0/158062892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc81a5110 con 0x7f2bc8103af0 2024-04-21T09:01:23.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.019+0000 7f2bcdd23640 1 -- 172.21.15.28:0/158062892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bc8068d70 con 0x7f2bc8103af0 2024-04-21T09:01:23.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.020+0000 7f2bc4ff9640 1 -- 172.21.15.28:0/158062892 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2bbc00f450 con 0x7f2bc8103af0 2024-04-21T09:01:23.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.021+0000 7f2bc4ff9640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2bb00a3d20 0x7f2bb00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.021+0000 7f2bc77fe640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2bb00a3d20 0x7f2bb00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:23.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.021+0000 7f2bc4ff9640 1 -- 172.21.15.28:0/158062892 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f2bbc0d0d20 con 0x7f2bc8103af0 2024-04-21T09:01:23.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.022+0000 7f2bc77fe640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2bb00a3d20 0x7f2bb00a61e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800baa0 tx=0x7f2bb8005d90 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:23.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.026+0000 7f2bc4ff9640 1 -- 172.21.15.28:0/158062892 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2bbc014030 con 0x7f2bc8103af0 2024-04-21T09:01:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.217+0000 7f2bcdd23640 1 -- 172.21.15.28:0/158062892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2bc81a5450 con 0x7f2bc8103af0 2024-04-21T09:01:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.217+0000 7f2bc4ff9640 1 -- 172.21.15.28:0/158062892 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) ==== 104+0+11 (secure 0 0 0) 0x7f2bbc09cfd0 con 0x7f2bc8103af0 2024-04-21T09:01:23.218 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T09:01:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.219+0000 7f2bcdd23640 1 -- 172.21.15.28:0/158062892 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2bb00a3d20 msgr2=0x7f2bb00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.219+0000 7f2bcdd23640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2bb00a3d20 0x7f2bb00a61e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800baa0 tx=0x7f2bb8005d90 comp rx=0 tx=0).stop 2024-04-21T09:01:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.219+0000 7f2bcdd23640 1 -- 172.21.15.28:0/158062892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc8103af0 msgr2=0x7f2bc8198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.219+0000 7f2bcdd23640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc8103af0 0x7f2bc8198ae0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00c920 tx=0x7f2bbc00cde0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.220+0000 7f2bcdd23640 1 -- 172.21.15.28:0/158062892 shutdown_connections 2024-04-21T09:01:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.220+0000 7f2bcdd23640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f2bb00a3d20 0x7f2bb00a61e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.220+0000 7f2bcdd23640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2bc810c510 0x7f2bc819ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.220+0000 7f2bcdd23640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc8103af0 0x7f2bc8198ae0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.220+0000 7f2bcdd23640 1 --2- 172.21.15.28:0/158062892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2bc81031d0 0x7f2bc81985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.220+0000 7f2bcdd23640 1 -- 172.21.15.28:0/158062892 >> 172.21.15.28:0/158062892 conn(0x7f2bc8077f50 msgr2=0x7f2bc80fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.220+0000 7f2bcdd23640 1 -- 172.21.15.28:0/158062892 shutdown_connections 2024-04-21T09:01:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.220+0000 7f2bcdd23640 1 -- 172.21.15.28:0/158062892 wait complete. 2024-04-21T09:01:23.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:23 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1059068206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T09:01:23.229 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:23 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3985624294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T09:01:23.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T09:01:23.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 -- 172.21.15.28:0/2762691316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd68101080 msgr2=0x7fcd68106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 --2- 172.21.15.28:0/2762691316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd68101080 0x7fcd68106bb0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fcd54009900 tx=0x7fcd5402bc30 comp rx=0 tx=0).stop 2024-04-21T09:01:23.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 -- 172.21.15.28:0/2762691316 shutdown_connections 2024-04-21T09:01:23.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 --2- 172.21.15.28:0/2762691316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd681072e0 0x7fcd6810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 --2- 172.21.15.28:0/2762691316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd68101080 0x7fcd68106bb0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 --2- 172.21.15.28:0/2762691316 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd681006d0 0x7fcd68100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 -- 172.21.15.28:0/2762691316 >> 172.21.15.28:0/2762691316 conn(0x7fcd680773f0 msgr2=0x7fcd680fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 -- 172.21.15.28:0/2762691316 shutdown_connections 2024-04-21T09:01:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 -- 172.21.15.28:0/2762691316 wait complete. 2024-04-21T09:01:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 Processor -- start 2024-04-21T09:01:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 -- start start 2024-04-21T09:01:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd681006d0 0x7fcd68198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd681072e0 0x7fcd68198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd6819cff0 0x7fcd6819d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcd68112190 con 0x7fcd681006d0 2024-04-21T09:01:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fcd68112010 con 0x7fcd6819cff0 2024-04-21T09:01:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.336+0000 7fcd6eb85640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcd68112310 con 0x7fcd681072e0 2024-04-21T09:01:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd6c921640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd681006d0 0x7fcd68198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd6c921640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd681006d0 0x7fcd68198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47406/0 (socket says 172.21.15.28:47406) 2024-04-21T09:01:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd6c921640 1 -- 172.21.15.28:0/1326115225 learned_addr learned my addr 172.21.15.28:0/1326115225 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:23.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd6d122640 1 --2- 172.21.15.28:0/1326115225 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd6819cff0 0x7fcd6819d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd67fff640 1 --2- 172.21.15.28:0/1326115225 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd681072e0 0x7fcd68198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd6c921640 1 -- 172.21.15.28:0/1326115225 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd681072e0 msgr2=0x7fcd68198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd6c921640 1 --2- 172.21.15.28:0/1326115225 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd681072e0 0x7fcd68198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd6c921640 1 -- 172.21.15.28:0/1326115225 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd6819cff0 msgr2=0x7fcd6819d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd6c921640 1 --2- 172.21.15.28:0/1326115225 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd6819cff0 0x7fcd6819d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd6c921640 1 -- 172.21.15.28:0/1326115225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd6819dae0 con 0x7fcd681006d0 2024-04-21T09:01:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd6c921640 1 --2- 172.21.15.28:0/1326115225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd681006d0 0x7fcd68198730 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c00c890 tx=0x7fcd5c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd65ffb640 1 -- 172.21.15.28:0/1326115225 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c004220 con 0x7fcd681006d0 2024-04-21T09:01:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd65ffb640 1 -- 172.21.15.28:0/1326115225 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd5c0043c0 con 0x7fcd681006d0 2024-04-21T09:01:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd6eb85640 1 -- 172.21.15.28:0/1326115225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd6819ddd0 con 0x7fcd681006d0 2024-04-21T09:01:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd65ffb640 1 -- 172.21.15.28:0/1326115225 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c004a00 con 0x7fcd681006d0 2024-04-21T09:01:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.337+0000 7fcd6eb85640 1 -- 172.21.15.28:0/1326115225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd681a5590 con 0x7fcd681006d0 2024-04-21T09:01:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.338+0000 7fcd6eb85640 1 -- 172.21.15.28:0/1326115225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd680684e0 con 0x7fcd681006d0 2024-04-21T09:01:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.339+0000 7fcd65ffb640 1 -- 172.21.15.28:0/1326115225 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcd5c0079b0 con 0x7fcd681006d0 2024-04-21T09:01:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.340+0000 7fcd65ffb640 1 --2- 172.21.15.28:0/1326115225 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcd580a3d20 0x7fcd580a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.340+0000 7fcd67fff640 1 --2- 172.21.15.28:0/1326115225 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcd580a3d20 0x7fcd580a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.340+0000 7fcd65ffb640 1 -- 172.21.15.28:0/1326115225 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7fcd5c0d0a50 con 0x7fcd681006d0 2024-04-21T09:01:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.341+0000 7fcd67fff640 1 --2- 172.21.15.28:0/1326115225 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcd580a3d20 0x7fcd580a61e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fcd54009230 tx=0x7fcd54004120 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.344+0000 7fcd65ffb640 1 -- 172.21.15.28:0/1326115225 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcd5c014030 con 0x7fcd681006d0 2024-04-21T09:01:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.529+0000 7fcd6eb85640 1 -- 172.21.15.28:0/1326115225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcd6810a4b0 con 0x7fcd681006d0 2024-04-21T09:01:23.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.530+0000 7fcd65ffb640 1 -- 172.21.15.28:0/1326115225 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) ==== 100+0+11 (secure 0 0 0) 0x7fcd5c09cca0 con 0x7fcd681006d0 2024-04-21T09:01:23.530 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-21T09:01:23.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.531+0000 7fcd6eb85640 1 -- 172.21.15.28:0/1326115225 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcd580a3d20 msgr2=0x7fcd580a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.531+0000 7fcd6eb85640 1 --2- 172.21.15.28:0/1326115225 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcd580a3d20 0x7fcd580a61e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fcd54009230 tx=0x7fcd54004120 comp rx=0 tx=0).stop 2024-04-21T09:01:23.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.531+0000 7fcd6eb85640 1 -- 172.21.15.28:0/1326115225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd681006d0 msgr2=0x7fcd68198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.531+0000 7fcd6eb85640 1 --2- 172.21.15.28:0/1326115225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd681006d0 0x7fcd68198730 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c00c890 tx=0x7fcd5c00cd50 comp rx=0 tx=0).stop 2024-04-21T09:01:23.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.532+0000 7fcd6eb85640 1 -- 172.21.15.28:0/1326115225 shutdown_connections 2024-04-21T09:01:23.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.532+0000 7fcd6eb85640 1 --2- 172.21.15.28:0/1326115225 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcd580a3d20 0x7fcd580a61e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.532+0000 7fcd6eb85640 1 --2- 172.21.15.28:0/1326115225 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd6819cff0 0x7fcd6819d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.532+0000 7fcd6eb85640 1 --2- 172.21.15.28:0/1326115225 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcd681072e0 0x7fcd68198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.532+0000 7fcd6eb85640 1 --2- 172.21.15.28:0/1326115225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd681006d0 0x7fcd68198730 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.532+0000 7fcd6eb85640 1 -- 172.21.15.28:0/1326115225 >> 172.21.15.28:0/1326115225 conn(0x7fcd680773f0 msgr2=0x7fcd68105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.532+0000 7fcd6eb85640 1 -- 172.21.15.28:0/1326115225 shutdown_connections 2024-04-21T09:01:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.532+0000 7fcd6eb85640 1 -- 172.21.15.28:0/1326115225 wait complete. 2024-04-21T09:01:23.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T09:01:23.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:23 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1059068206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T09:01:23.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:23 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3985624294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T09:01:23.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:23 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1059068206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T09:01:23.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:23 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3985624294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T09:01:23.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.648+0000 7f00d699d640 1 -- 172.21.15.28:0/3702535568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d0106ea0 msgr2=0x7f00d0107280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.648+0000 7f00d699d640 1 --2- 172.21.15.28:0/3702535568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d0106ea0 0x7f00d0107280 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009930 tx=0x7f00cc02bc30 comp rx=0 tx=0).stop 2024-04-21T09:01:23.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.648+0000 7f00d699d640 1 -- 172.21.15.28:0/3702535568 shutdown_connections 2024-04-21T09:01:23.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.648+0000 7f00d699d640 1 --2- 172.21.15.28:0/3702535568 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f00d0100d80 0x7f00d010ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.648+0000 7f00d699d640 1 --2- 172.21.15.28:0/3702535568 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00d01003c0 0x7f00d0100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.648+0000 7f00d699d640 1 --2- 172.21.15.28:0/3702535568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d0106ea0 0x7f00d0107280 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.648+0000 7f00d699d640 1 -- 172.21.15.28:0/3702535568 >> 172.21.15.28:0/3702535568 conn(0x7f00d0077380 msgr2=0x7f00d00fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.648+0000 7f00d699d640 1 -- 172.21.15.28:0/3702535568 shutdown_connections 2024-04-21T09:01:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.648+0000 7f00d699d640 1 -- 172.21.15.28:0/3702535568 wait complete. 2024-04-21T09:01:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d699d640 1 Processor -- start 2024-04-21T09:01:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d699d640 1 -- start start 2024-04-21T09:01:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d699d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d01003c0 0x7f00d01015a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d699d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00d0100d80 0x7f00d0101ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d699d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f00d0106ea0 0x7f00d0102020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d699d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00d0112200 con 0x7f00d01003c0 2024-04-21T09:01:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d699d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f00d0112080 con 0x7f00d0100d80 2024-04-21T09:01:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d699d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f00d0112380 con 0x7f00d0106ea0 2024-04-21T09:01:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d599b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d01003c0 0x7f00d01015a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d599b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d01003c0 0x7f00d01015a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47428/0 (socket says 172.21.15.28:47428) 2024-04-21T09:01:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d599b640 1 -- 172.21.15.28:0/2285632429 learned_addr learned my addr 172.21.15.28:0/2285632429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d619c640 1 --2- 172.21.15.28:0/2285632429 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f00d0106ea0 0x7f00d0102020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d599b640 1 -- 172.21.15.28:0/2285632429 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f00d0106ea0 msgr2=0x7f00d0102020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d599b640 1 --2- 172.21.15.28:0/2285632429 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f00d0106ea0 0x7f00d0102020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d599b640 1 -- 172.21.15.28:0/2285632429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00d0100d80 msgr2=0x7f00d0101ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d599b640 1 --2- 172.21.15.28:0/2285632429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00d0100d80 0x7f00d0101ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d599b640 1 -- 172.21.15.28:0/2285632429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d01a6c40 con 0x7f00d01003c0 2024-04-21T09:01:23.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.649+0000 7f00d599b640 1 --2- 172.21.15.28:0/2285632429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d01003c0 0x7f00d01015a0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009230 tx=0x7f00cc0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:23.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.650+0000 7f00c6ffd640 1 -- 172.21.15.28:0/2285632429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00cc043020 con 0x7f00d01003c0 2024-04-21T09:01:23.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.650+0000 7f00c6ffd640 1 -- 172.21.15.28:0/2285632429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00cc02cb70 con 0x7f00d01003c0 2024-04-21T09:01:23.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.650+0000 7f00d699d640 1 -- 172.21.15.28:0/2285632429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d01a6ed0 con 0x7f00d01003c0 2024-04-21T09:01:23.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.650+0000 7f00c6ffd640 1 -- 172.21.15.28:0/2285632429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00cc003b50 con 0x7f00d01003c0 2024-04-21T09:01:23.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.650+0000 7f00d699d640 1 -- 172.21.15.28:0/2285632429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d01a7390 con 0x7f00d01003c0 2024-04-21T09:01:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.651+0000 7f00d699d640 1 -- 172.21.15.28:0/2285632429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00d00690e0 con 0x7f00d01003c0 2024-04-21T09:01:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.652+0000 7f00c6ffd640 1 -- 172.21.15.28:0/2285632429 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f00cc046020 con 0x7f00d01003c0 2024-04-21T09:01:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.652+0000 7f00c6ffd640 1 --2- 172.21.15.28:0/2285632429 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00bc0a3d20 0x7f00bc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.653+0000 7f00c6ffd640 1 -- 172.21.15.28:0/2285632429 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f00cc0f30d0 con 0x7f00d01003c0 2024-04-21T09:01:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.653+0000 7f00d519a640 1 --2- 172.21.15.28:0/2285632429 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00bc0a3d20 0x7f00bc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.653+0000 7f00d519a640 1 --2- 172.21.15.28:0/2285632429 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00bc0a3d20 0x7f00bc0a61e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f00c0005fd0 tx=0x7f00c0005850 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.656+0000 7f00c6ffd640 1 -- 172.21.15.28:0/2285632429 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f00cc04d020 con 0x7f00d01003c0 2024-04-21T09:01:23.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.844+0000 7f00d699d640 1 -- 172.21.15.28:0/2285632429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f00d010a380 con 0x7f00d01003c0 2024-04-21T09:01:23.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.844+0000 7f00c6ffd640 1 -- 172.21.15.28:0/2285632429 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v102) ==== 103+0+11 (secure 0 0 0) 0x7f00cc0bf490 con 0x7f00d01003c0 2024-04-21T09:01:23.844 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T09:01:23.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.846+0000 7f00d699d640 1 -- 172.21.15.28:0/2285632429 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00bc0a3d20 msgr2=0x7f00bc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.846+0000 7f00d699d640 1 --2- 172.21.15.28:0/2285632429 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00bc0a3d20 0x7f00bc0a61e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f00c0005fd0 tx=0x7f00c0005850 comp rx=0 tx=0).stop 2024-04-21T09:01:23.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.846+0000 7f00d699d640 1 -- 172.21.15.28:0/2285632429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d01003c0 msgr2=0x7f00d01015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.846+0000 7f00d699d640 1 --2- 172.21.15.28:0/2285632429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d01003c0 0x7f00d01015a0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009230 tx=0x7f00cc0045e0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.846+0000 7f00d699d640 1 -- 172.21.15.28:0/2285632429 shutdown_connections 2024-04-21T09:01:23.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.846+0000 7f00d699d640 1 --2- 172.21.15.28:0/2285632429 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f00bc0a3d20 0x7f00bc0a61e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.846+0000 7f00d699d640 1 --2- 172.21.15.28:0/2285632429 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f00d0106ea0 0x7f00d0102020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.846+0000 7f00d699d640 1 --2- 172.21.15.28:0/2285632429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00d0100d80 0x7f00d0101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.846+0000 7f00d699d640 1 --2- 172.21.15.28:0/2285632429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00d01003c0 0x7f00d01015a0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.846+0000 7f00d699d640 1 -- 172.21.15.28:0/2285632429 >> 172.21.15.28:0/2285632429 conn(0x7f00d0077380 msgr2=0x7f00d01050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:23.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.846+0000 7f00d699d640 1 -- 172.21.15.28:0/2285632429 shutdown_connections 2024-04-21T09:01:23.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.846+0000 7f00d699d640 1 -- 172.21.15.28:0/2285632429 wait complete. 2024-04-21T09:01:23.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-21T09:01:23.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.959+0000 7fcfd6043640 1 -- 172.21.15.28:0/3579820516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfd0101ac0 msgr2=0x7fcfd0105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.959+0000 7fcfd6043640 1 --2- 172.21.15.28:0/3579820516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfd0101ac0 0x7fcfd0105ab0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4009900 tx=0x7fcfc402bc30 comp rx=0 tx=0).stop 2024-04-21T09:01:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.959+0000 7fcfd6043640 1 -- 172.21.15.28:0/3579820516 shutdown_connections 2024-04-21T09:01:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.959+0000 7fcfd6043640 1 --2- 172.21.15.28:0/3579820516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfd0106190 0x7fcfd010dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.959+0000 7fcfd6043640 1 --2- 172.21.15.28:0/3579820516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfd0101ac0 0x7fcfd0105ab0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.959+0000 7fcfd6043640 1 --2- 172.21.15.28:0/3579820516 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfd0101110 0x7fcfd01014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.960+0000 7fcfd6043640 1 -- 172.21.15.28:0/3579820516 >> 172.21.15.28:0/3579820516 conn(0x7fcfd0077fb0 msgr2=0x7fcfd00783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.960+0000 7fcfd6043640 1 -- 172.21.15.28:0/3579820516 shutdown_connections 2024-04-21T09:01:23.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.960+0000 7fcfd6043640 1 -- 172.21.15.28:0/3579820516 wait complete. 2024-04-21T09:01:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.960+0000 7fcfd6043640 1 Processor -- start 2024-04-21T09:01:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.960+0000 7fcfd6043640 1 -- start start 2024-04-21T09:01:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.960+0000 7fcfd6043640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfd0101110 0x7fcfd0102b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.960+0000 7fcfd6043640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfd0106190 0x7fcfd0103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.960+0000 7fcfd6043640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfd01035d0 0x7fcfd01a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.960+0000 7fcfd6043640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd01110d0 con 0x7fcfd0101110 2024-04-21T09:01:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.960+0000 7fcfd6043640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0110f50 con 0x7fcfd01035d0 2024-04-21T09:01:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfd6043640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0111250 con 0x7fcfd0106190 2024-04-21T09:01:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfcf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfd0101110 0x7fcfd0102b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfcf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfd0101110 0x7fcfd0102b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50044/0 (socket says 172.21.15.28:50044) 2024-04-21T09:01:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfcf7fe640 1 -- 172.21.15.28:0/698589818 learned_addr learned my addr 172.21.15.28:0/698589818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:01:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfceffd640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfd0106190 0x7fcfd0103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfcffff640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfd01035d0 0x7fcfd01a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfcf7fe640 1 -- 172.21.15.28:0/698589818 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfd0106190 msgr2=0x7fcfd0103090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfcf7fe640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfd0106190 0x7fcfd0103090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfcf7fe640 1 -- 172.21.15.28:0/698589818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfd01035d0 msgr2=0x7fcfd01a6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfcf7fe640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfd01035d0 0x7fcfd01a6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfcf7fe640 1 -- 172.21.15.28:0/698589818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd01a7010 con 0x7fcfd0101110 2024-04-21T09:01:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfceffd640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfd0106190 0x7fcfd0103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:01:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfcffff640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfd01035d0 0x7fcfd01a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:01:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfcf7fe640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfd0101110 0x7fcfd0102b50 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000c970 tx=0x7fcfc000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfccff9640 1 -- 172.21.15.28:0/698589818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0007bb0 con 0x7fcfd0101110 2024-04-21T09:01:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfccff9640 1 -- 172.21.15.28:0/698589818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfc0007d50 con 0x7fcfd0101110 2024-04-21T09:01:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfd6043640 1 -- 172.21.15.28:0/698589818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd01a7300 con 0x7fcfd0101110 2024-04-21T09:01:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfccff9640 1 -- 172.21.15.28:0/698589818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc000f450 con 0x7fcfd0101110 2024-04-21T09:01:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.961+0000 7fcfd6043640 1 -- 172.21.15.28:0/698589818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd01a7800 con 0x7fcfd0101110 2024-04-21T09:01:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.962+0000 7fcfd6043640 1 -- 172.21.15.28:0/698589818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd00691c0 con 0x7fcfd0101110 2024-04-21T09:01:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.964+0000 7fcfccff9640 1 -- 172.21.15.28:0/698589818 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcfc0004ee0 con 0x7fcfd0101110 2024-04-21T09:01:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.965+0000 7fcfccff9640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcfb80a3d50 0x7fcfb80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:01:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.965+0000 7fcfccff9640 1 -- 172.21.15.28:0/698589818 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7fcfc00d0e00 con 0x7fcfd0101110 2024-04-21T09:01:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.965+0000 7fcfceffd640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcfb80a3d50 0x7fcfb80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:01:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.965+0000 7fcfceffd640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcfb80a3d50 0x7fcfb80a6210 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4036040 tx=0x7fcfc4000bc0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:01:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:23.969+0000 7fcfccff9640 1 -- 172.21.15.28:0/698589818 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcfc0014030 con 0x7fcfd0101110 2024-04-21T09:01:24.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:24.209+0000 7fcfd6043640 1 -- 172.21.15.28:0/698589818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fcfd010a400 con 0x7fcfd0101110 2024-04-21T09:01:24.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:24.218+0000 7fcfccff9640 1 -- 172.21.15.28:0/698589818 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746062 (secure 0 0 0) 0x7fcfc009d050 con 0x7fcfd0101110 2024-04-21T09:01:24.228 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T09:01:24.229 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T09:01:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 26, 2024-04-21T09:01:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: "active_gid": 25040, 2024-04-21T09:01:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: "active_name": "x", 2024-04-21T09:01:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addrs": { 2024-04-21T09:01:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-21T09:01:24.229 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-21T09:01:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-04-21T09:01:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4182301910 2024-04-21T09:01:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-04-21T09:01:24.230 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-04-21T09:01:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4182301910 2024-04-21T09:01:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-21T09:01:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addr": "172.21.15.28:6801/4182301910", 2024-04-21T09:01:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: "active_change": "2024-04-21T08:51:10.823621+0000", 2024-04-21T09:01:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: "active_mgr_features": 4540701547738038271, 2024-04-21T09:01:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: "available": true, 2024-04-21T09:01:24.231 INFO:teuthology.orchestra.run.smithi028.stdout: "standbys": [ 2024-04-21T09:01:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 15453, 2024-04-21T09:01:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "y", 2024-04-21T09:01:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-04-21T09:01:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-04-21T09:01:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-04-21T09:01:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.232 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-21T09:01:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-21T09:01:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-21T09:01:24.233 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T09:01:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.234 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.237 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.238 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.239 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.240 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.242 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-04-21T09:01:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-04-21T09:01:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.243 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-04-21T09:01:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.244 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-04-21T09:01:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-04-21T09:01:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-04-21T09:01:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.245 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-04-21T09:01:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-04-21T09:01:24.246 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-04-21T09:01:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.247 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-04-21T09:01:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-04-21T09:01:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-04-21T09:01:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.248 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-04-21T09:01:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.249 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-04-21T09:01:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-04-21T09:01:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-04-21T09:01:24.250 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-04-21T09:01:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-04-21T09:01:24.251 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-04-21T09:01:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-04-21T09:01:24.252 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-04-21T09:01:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-04-21T09:01:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.253 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T09:01:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.254 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-04-21T09:01:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-04-21T09:01:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.255 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-04-21T09:01:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.256 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-04-21T09:01:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.257 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-04-21T09:01:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-04-21T09:01:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.258 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T09:01:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-04-21T09:01:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-04-21T09:01:24.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-04-21T09:01:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T09:01:24.260 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T09:01:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-04-21T09:01:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-04-21T09:01:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-21T09:01:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.261 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-21T09:01:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-21T09:01:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-21T09:01:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T09:01:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T09:01:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-04-21T09:01:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-04-21T09:01:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-21T09:01:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-04-21T09:01:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-21T09:01:24.263 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-04-21T09:01:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T09:01:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-04-21T09:01:24.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-04-21T09:01:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T09:01:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T09:01:24.265 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T09:01:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-04-21T09:01:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-04-21T09:01:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.266 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-04-21T09:01:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-04-21T09:01:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-04-21T09:01:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-04-21T09:01:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T09:01:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.267 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-04-21T09:01:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-04-21T09:01:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-04-21T09:01:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T09:01:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T09:01:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-04-21T09:01:24.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-04-21T09:01:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-21T09:01:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-21T09:01:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-21T09:01:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-21T09:01:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T09:01:24.270 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T09:01:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.271 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.272 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.274 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.275 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.276 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.277 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.279 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-04-21T09:01:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-04-21T09:01:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-21T09:01:24.280 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T09:01:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-21T09:01:24.281 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-21T09:01:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-04-21T09:01:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-04-21T09:01:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-04-21T09:01:24.283 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-04-21T09:01:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-04-21T09:01:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-04-21T09:01:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-04-21T09:01:24.285 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-04-21T09:01:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-04-21T09:01:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.286 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T09:01:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.287 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-21T09:01:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-21T09:01:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-21T09:01:24.288 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T09:01:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-04-21T09:01:24.289 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-04-21T09:01:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-21T09:01:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-21T09:01:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.290 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T09:01:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T09:01:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-04-21T09:01:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-04-21T09:01:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-21T09:01:24.291 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T09:01:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.292 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-04-21T09:01:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.293 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-04-21T09:01:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-04-21T09:01:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-21T09:01:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.294 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T09:01:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-04-21T09:01:24.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-04-21T09:01:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-04-21T09:01:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T09:01:24.296 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-04-21T09:01:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-04-21T09:01:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-04-21T09:01:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T09:01:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.298 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-04-21T09:01:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-04-21T09:01:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T09:01:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T09:01:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.300 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-04-21T09:01:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-04-21T09:01:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-21T09:01:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.301 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T09:01:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-04-21T09:01:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T09:01:24.302 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-04-21T09:01:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T09:01:24.303 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-04-21T09:01:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-04-21T09:01:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.304 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-04-21T09:01:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-04-21T09:01:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-04-21T09:01:24.305 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T09:01:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-04-21T09:01:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-04-21T09:01:24.306 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T09:01:24.307 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-04-21T09:01:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-04-21T09:01:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.308 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T09:01:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-04-21T09:01:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-04-21T09:01:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T09:01:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-04-21T09:01:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-04-21T09:01:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T09:01:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.312 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-21T09:01:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-04-21T09:01:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-04-21T09:01:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T09:01:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-04-21T09:01:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.314 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-04-21T09:01:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-04-21T09:01:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T09:01:24.315 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-04-21T09:01:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-04-21T09:01:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-04-21T09:01:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T09:01:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.317 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-21T09:01:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-04-21T09:01:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-04-21T09:01:24.318 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T09:01:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-04-21T09:01:24.319 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-04-21T09:01:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-04-21T09:01:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.320 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T09:01:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-04-21T09:01:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.321 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-04-21T09:01:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-04-21T09:01:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T09:01:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.322 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-04-21T09:01:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.323 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-04-21T09:01:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-04-21T09:01:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T09:01:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.324 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-04-21T09:01:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-04-21T09:01:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-04-21T09:01:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.325 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T09:01:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-04-21T09:01:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-04-21T09:01:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-04-21T09:01:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.327 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T09:01:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-04-21T09:01:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.328 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-04-21T09:01:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-04-21T09:01:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T09:01:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.329 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-21T09:01:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-04-21T09:01:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-04-21T09:01:24.330 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T09:01:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-04-21T09:01:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-04-21T09:01:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-04-21T09:01:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.332 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T09:01:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-21T09:01:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-04-21T09:01:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-04-21T09:01:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T09:01:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.334 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-04-21T09:01:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-04-21T09:01:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-04-21T09:01:24.335 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T09:01:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-04-21T09:01:24.336 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-04-21T09:01:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-04-21T09:01:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.338 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T09:01:24.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.338 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.338 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.338 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-04-21T09:01:24.338 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.339 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.339 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-04-21T09:01:24.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-04-21T09:01:24.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.339 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.339 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.339 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.340 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.340 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.340 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T09:01:24.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-04-21T09:01:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-04-21T09:01:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-21T09:01:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.341 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T09:01:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-04-21T09:01:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T09:01:24.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-04-21T09:01:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T09:01:24.343 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-04-21T09:01:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-04-21T09:01:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-04-21T09:01:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T09:01:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-04-21T09:01:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-04-21T09:01:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-04-21T09:01:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.346 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-04-21T09:01:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-04-21T09:01:24.347 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-04-21T09:01:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T09:01:24.348 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-04-21T09:01:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-04-21T09:01:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.349 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-21T09:01:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-04-21T09:01:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.350 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-04-21T09:01:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-04-21T09:01:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.351 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T09:01:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T09:01:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-04-21T09:01:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-04-21T09:01:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-21T09:01:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.353 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.354 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.354 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-04-21T09:01:24.354 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.354 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.354 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-04-21T09:01:24.354 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-04-21T09:01:24.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.355 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.355 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.355 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.355 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.355 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T09:01:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-04-21T09:01:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-04-21T09:01:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.357 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.357 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.357 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.357 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.357 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.357 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T09:01:24.357 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.358 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.358 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.358 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.358 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.359 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.359 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.359 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.359 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.359 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.359 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.359 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.359 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.360 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.360 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-04-21T09:01:24.360 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-04-21T09:01:24.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.361 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.361 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.361 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.361 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.361 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.361 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T09:01:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T09:01:24.363 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.364 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.365 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.366 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.367 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T09:01:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T09:01:24.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T09:01:24.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T09:01:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T09:01:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-04-21T09:01:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T09:01:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-04-21T09:01:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-04-21T09:01:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.372 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-04-21T09:01:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-04-21T09:01:24.373 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-04-21T09:01:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.374 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T09:01:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-04-21T09:01:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-04-21T09:01:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.375 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-04-21T09:01:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-21T09:01:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-21T09:01:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-04-21T09:01:24.377 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-04-21T09:01:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-04-21T09:01:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T09:01:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.378 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-04-21T09:01:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-04-21T09:01:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.379 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-04-21T09:01:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T09:01:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.380 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-04-21T09:01:24.381 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-04-21T09:01:24.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.381 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.381 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T09:01:24.381 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.381 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.381 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.382 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T09:01:24.382 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.382 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.382 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-04-21T09:01:24.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-04-21T09:01:24.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T09:01:24.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-04-21T09:01:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-04-21T09:01:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T09:01:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-04-21T09:01:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-04-21T09:01:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.386 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T09:01:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.387 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-04-21T09:01:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-04-21T09:01:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.388 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T09:01:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-04-21T09:01:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-04-21T09:01:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.389 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T09:01:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-04-21T09:01:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-04-21T09:01:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.391 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-04-21T09:01:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-04-21T09:01:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-04-21T09:01:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-04-21T09:01:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.393 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T09:01:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-04-21T09:01:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-04-21T09:01:24.394 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.395 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.395 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.395 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.395 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.395 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T09:01:24.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-04-21T09:01:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-04-21T09:01:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.396 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T09:01:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-04-21T09:01:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-04-21T09:01:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.398 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T09:01:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-04-21T09:01:24.399 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-04-21T09:01:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T09:01:24.400 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-04-21T09:01:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-04-21T09:01:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T09:01:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.402 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-04-21T09:01:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.403 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.404 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.405 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.405 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.405 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.405 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.405 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.405 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.406 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.406 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.406 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.406 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.406 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.407 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.407 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.407 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.407 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.407 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.407 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.408 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.408 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.408 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.408 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.408 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.409 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.409 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.409 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.409 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.409 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.409 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.409 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.410 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.410 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.410 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.410 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.410 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.410 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.410 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.410 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.411 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.411 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.411 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.412 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.412 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.412 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.412 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-04-21T09:01:24.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-04-21T09:01:24.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.413 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-04-21T09:01:24.413 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.413 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.413 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.414 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T09:01:24.414 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.414 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.414 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.414 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-04-21T09:01:24.414 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-04-21T09:01:24.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.415 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-04-21T09:01:24.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.415 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.415 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T09:01:24.415 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.416 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-04-21T09:01:24.416 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.417 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.417 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.417 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T09:01:24.417 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T09:01:24.417 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.417 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.417 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.417 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:24.418 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.418 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.418 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.418 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.418 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.418 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.418 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.419 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T09:01:24.419 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T09:01:24.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.419 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.419 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.420 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.420 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.420 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.420 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.420 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.420 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T09:01:24.420 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T09:01:24.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.421 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.421 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.421 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.421 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.421 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.421 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.422 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.422 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.422 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-04-21T09:01:24.422 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T09:01:24.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.422 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.423 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.423 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.423 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.423 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.423 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.423 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.423 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.424 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.424 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T09:01:24.424 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T09:01:24.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.424 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.424 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.424 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.425 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.425 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.425 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.425 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.425 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.425 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T09:01:24.426 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T09:01:24.426 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.426 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.426 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.426 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.426 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.426 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.427 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.427 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.427 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.427 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T09:01:24.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T09:01:24.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.428 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.428 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.428 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.428 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.429 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.429 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.429 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T09:01:24.429 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T09:01:24.429 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.430 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.430 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.430 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.430 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.430 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.430 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.431 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T09:01:24.431 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T09:01:24.431 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.431 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.431 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.432 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.432 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.432 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.432 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.432 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.432 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.432 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.433 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T09:01:24.433 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T09:01:24.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.433 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.433 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.433 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.434 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.434 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.434 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.434 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.434 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.434 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T09:01:24.435 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T09:01:24.435 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.435 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.435 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.435 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.435 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.435 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.435 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.436 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.436 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.436 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.436 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.436 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T09:01:24.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T09:01:24.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.437 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.437 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.437 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.437 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.437 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.437 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.438 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.438 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.438 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T09:01:24.438 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T09:01:24.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.438 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.439 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.439 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.439 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.439 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.440 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T09:01:24.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T09:01:24.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.440 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.441 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.441 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.441 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T09:01:24.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T09:01:24.442 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.442 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.442 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.442 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-21T09:01:24.442 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.442 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.442 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.442 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.443 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.443 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T09:01:24.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T09:01:24.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.444 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.444 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.444 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.445 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-04-21T09:01:24.445 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-04-21T09:01:24.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.445 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.445 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.446 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.446 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.446 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.446 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.446 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.446 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.446 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T09:01:24.447 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T09:01:24.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.447 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-21T09:01:24.447 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.447 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.447 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.448 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.448 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.448 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.448 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.448 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T09:01:24.448 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T09:01:24.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.448 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.449 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.449 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.449 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.449 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.449 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.449 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.449 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.449 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.449 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.450 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T09:01:24.450 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T09:01:24.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.450 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.450 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.451 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.451 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.451 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.451 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.451 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.451 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.451 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T09:01:24.452 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T09:01:24.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.452 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.452 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.452 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.452 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.452 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.453 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.453 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.453 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.453 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T09:01:24.453 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T09:01:24.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.453 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.454 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.454 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.454 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.454 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.454 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.454 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.454 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.454 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.455 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.455 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-21T09:01:24.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-21T09:01:24.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-04-21T09:01:24.455 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.456 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.456 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.456 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.456 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.456 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.456 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-04-21T09:01:24.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-21T09:01:24.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.457 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-04-21T09:01:24.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.457 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.457 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.457 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.458 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.458 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T09:01:24.458 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T09:01:24.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.458 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.459 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.459 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.459 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.459 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.459 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.459 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.459 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.460 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T09:01:24.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T09:01:24.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.460 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.460 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.460 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.460 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.460 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.461 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.461 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.461 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.461 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.461 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T09:01:24.461 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T09:01:24.461 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.462 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.462 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.462 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.462 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.462 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.462 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.463 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.463 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.463 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T09:01:24.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T09:01:24.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.464 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.464 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.464 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.464 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.464 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.464 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.464 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.465 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T09:01:24.465 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T09:01:24.465 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.465 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.465 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.465 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.465 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.465 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.466 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.466 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.466 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.466 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.466 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T09:01:24.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T09:01:24.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.467 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.467 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.467 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.467 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.467 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.468 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.468 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.468 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T09:01:24.468 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T09:01:24.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.468 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.469 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.469 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.469 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.469 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.469 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.469 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.469 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.469 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T09:01:24.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T09:01:24.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.470 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.470 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.470 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.470 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.470 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.470 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.471 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.471 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.471 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T09:01:24.471 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T09:01:24.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.472 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.473 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.473 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.473 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.473 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T09:01:24.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T09:01:24.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.473 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.473 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.474 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.474 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.474 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.474 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.474 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.474 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.474 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.474 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.475 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T09:01:24.475 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T09:01:24.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.475 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.475 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T09:01:24.475 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.476 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.476 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.476 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.476 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.476 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.476 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.476 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T09:01:24.477 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T09:01:24.477 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.477 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:24.477 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.477 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.478 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.478 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.478 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.478 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T09:01:24.478 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T09:01:24.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.478 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-04-21T09:01:24.479 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.479 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.479 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.479 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.479 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.479 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.480 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T09:01:24.480 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T09:01:24.480 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.480 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.480 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.480 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-04-21T09:01:24.481 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.481 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.481 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.481 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.481 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.482 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T09:01:24.482 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T09:01:24.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.482 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.482 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.482 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.483 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.483 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.483 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.483 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T09:01:24.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T09:01:24.484 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.484 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.484 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.484 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-21T09:01:24.484 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.484 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.484 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.485 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.485 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T09:01:24.485 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T09:01:24.485 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.485 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.486 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.486 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.486 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.486 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.486 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.486 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.487 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.489 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.489 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T09:01:24.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T09:01:24.490 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.490 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.490 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.490 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.490 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.490 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.490 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.491 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.491 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.491 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.491 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-04-21T09:01:24.491 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-21T09:01:24.491 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.491 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.492 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.492 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.492 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.492 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.492 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.492 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.492 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.492 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.493 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.493 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T09:01:24.493 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T09:01:24.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.493 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.493 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.493 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-21T09:01:24.494 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.494 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.494 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.494 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.494 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.494 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.494 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.495 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T09:01:24.495 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T09:01:24.495 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.495 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.495 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.495 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:24.495 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.495 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.495 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.496 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.496 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.496 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.496 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.496 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T09:01:24.496 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T09:01:24.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.497 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.497 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.497 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-21T09:01:24.497 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.497 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.498 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.498 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.498 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.498 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-04-21T09:01:24.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-04-21T09:01:24.498 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.499 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.499 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.499 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.499 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.499 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.499 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.500 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.500 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-04-21T09:01:24.500 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-04-21T09:01:24.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.500 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.500 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.501 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.501 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.501 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.501 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-04-21T09:01:24.501 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-04-21T09:01:24.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.502 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.502 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.502 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.502 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.502 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-04-21T09:01:24.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.503 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.503 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-04-21T09:01:24.503 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-04-21T09:01:24.503 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.503 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.503 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.504 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-04-21T09:01:24.504 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.504 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.504 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.504 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.504 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.504 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.504 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.505 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-21T09:01:24.505 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-21T09:01:24.505 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.505 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.505 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.505 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.505 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.506 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.506 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.506 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.506 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.506 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.506 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.506 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.507 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.507 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.507 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.507 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.507 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.507 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.507 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.508 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.508 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.508 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.508 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.508 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.508 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.508 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.509 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.509 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.509 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.509 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.509 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.509 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.509 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.510 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.510 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.510 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.510 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.510 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.510 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.510 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.510 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.511 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.511 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.511 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.511 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.511 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.511 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.511 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.512 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.512 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.512 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.512 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.512 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.512 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.512 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.512 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.513 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.513 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.513 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.513 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.513 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.513 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.514 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.514 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.514 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.514 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.514 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.514 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.514 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.514 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.515 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.515 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.515 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-04-21T09:01:24.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-04-21T09:01:24.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.515 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.516 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.516 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.516 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.516 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.516 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-04-21T09:01:24.516 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.516 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.516 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.517 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-04-21T09:01:24.517 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T09:01:24.517 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.517 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.517 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.517 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.518 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.518 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.518 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.518 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.518 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.518 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.518 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-21T09:01:24.518 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-21T09:01:24.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.519 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-21T09:01:24.519 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.519 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.519 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.519 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.520 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.520 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.520 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.520 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-21T09:01:24.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-21T09:01:24.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.521 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.521 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.521 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-04-21T09:01:24.521 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.521 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.521 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.521 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.521 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.522 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.522 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.522 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-21T09:01:24.522 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-21T09:01:24.522 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.522 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.522 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.522 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.523 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.523 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.523 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.523 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.523 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.523 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.523 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.524 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-04-21T09:01:24.524 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-04-21T09:01:24.524 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.524 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.524 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.524 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-04-21T09:01:24.524 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.524 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.525 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.525 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.525 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.525 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.525 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-21T09:01:24.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-21T09:01:24.526 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.526 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.526 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.526 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-04-21T09:01:24.526 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.526 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.526 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.527 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-04-21T09:01:24.527 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.527 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.527 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.527 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.527 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.527 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-21T09:01:24.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-21T09:01:24.528 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.528 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.528 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.528 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-21T09:01:24.528 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-21T09:01:24.528 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-21T09:01:24.528 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.528 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.529 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.529 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.529 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.529 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-04-21T09:01:24.529 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-04-21T09:01:24.529 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.529 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.530 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.530 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.530 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.530 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.530 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.530 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.530 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.531 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.531 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.531 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-21T09:01:24.531 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.531 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.532 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.532 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-04-21T09:01:24.532 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-04-21T09:01:24.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.532 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.532 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.533 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.533 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-04-21T09:01:24.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.533 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.533 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.534 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.534 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.534 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.534 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.534 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.534 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.534 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.534 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.534 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.535 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.535 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.535 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.535 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.535 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.535 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.535 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.535 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.536 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.536 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.536 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.536 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.536 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.536 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.537 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.537 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.537 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.537 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.537 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.537 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.537 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.538 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.538 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.538 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.538 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.538 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.538 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.539 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.539 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.539 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.539 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.539 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.539 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.539 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.540 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.540 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.540 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.540 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.541 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.541 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.541 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.541 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.541 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.541 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.542 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-04-21T09:01:24.542 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-04-21T09:01:24.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.542 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-04-21T09:01:24.542 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.542 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.543 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.543 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T09:01:24.543 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.543 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.543 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.543 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-04-21T09:01:24.544 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-04-21T09:01:24.544 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.544 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.544 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.544 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-04-21T09:01:24.544 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.544 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.544 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.545 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T09:01:24.545 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.545 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.545 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.545 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-04-21T09:01:24.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-04-21T09:01:24.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.546 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.546 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.546 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-04-21T09:01:24.546 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.546 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.546 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.546 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-04-21T09:01:24.546 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.547 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.547 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.547 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-04-21T09:01:24.547 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-04-21T09:01:24.547 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.547 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.547 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.547 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-21T09:01:24.548 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.548 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.548 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.548 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T09:01:24.548 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.548 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.548 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.548 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-04-21T09:01:24.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-04-21T09:01:24.549 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.549 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.549 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.549 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.549 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.549 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.549 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.550 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T09:01:24.550 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.550 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.550 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.550 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-21T09:01:24.550 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-21T09:01:24.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.551 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.551 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.551 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-21T09:01:24.551 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.551 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.551 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.551 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T09:01:24.551 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.552 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.552 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.552 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-04-21T09:01:24.552 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-04-21T09:01:24.552 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.552 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.552 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.552 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-04-21T09:01:24.553 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.553 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.553 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.553 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T09:01:24.553 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.553 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.553 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.553 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.554 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.554 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-04-21T09:01:24.554 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.554 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.554 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.554 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.555 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.555 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.555 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.555 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.555 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.555 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.555 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.555 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.556 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.556 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.556 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.556 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.556 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.556 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.557 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.557 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.557 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.557 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.558 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.558 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.558 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.558 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.558 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.558 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.558 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.558 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.559 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.559 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.559 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.559 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.559 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.559 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.559 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.559 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.560 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.560 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.560 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.560 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.560 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.560 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.560 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.560 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.561 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.561 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.561 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.561 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.561 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.561 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.561 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.561 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.562 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.562 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.562 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.562 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.562 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.562 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.562 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.562 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.563 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.563 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-04-21T09:01:24.563 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-04-21T09:01:24.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.563 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.563 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-21T09:01:24.564 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.564 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.564 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.564 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.564 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.564 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.564 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.564 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-04-21T09:01:24.565 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-04-21T09:01:24.565 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.565 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.565 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.565 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-04-21T09:01:24.565 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.565 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.566 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.566 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.566 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.566 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.566 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.566 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-21T09:01:24.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-21T09:01:24.567 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.567 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.567 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.567 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-21T09:01:24.567 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.567 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.568 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.568 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.568 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.568 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.568 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-04-21T09:01:24.568 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-04-21T09:01:24.569 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-04-21T09:01:24.569 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.569 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-04-21T09:01:24.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-04-21T09:01:24.569 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.569 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.569 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.569 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-04-21T09:01:24.570 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.570 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.570 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.570 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T09:01:24.570 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.570 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.570 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.571 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-04-21T09:01:24.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-04-21T09:01:24.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.571 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.571 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.571 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-04-21T09:01:24.571 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.571 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.572 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.572 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T09:01:24.572 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.572 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.572 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.572 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-04-21T09:01:24.572 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-04-21T09:01:24.573 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.573 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.573 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.573 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.573 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.573 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.573 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.573 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-04-21T09:01:24.573 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.574 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.574 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.574 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-21T09:01:24.574 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-21T09:01:24.574 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.574 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.575 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.575 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-04-21T09:01:24.575 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-04-21T09:01:24.575 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.575 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.575 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T09:01:24.575 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.575 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.575 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.576 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.576 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.576 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.576 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.576 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.576 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.577 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.577 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.577 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.577 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.577 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.577 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.577 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.577 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.578 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.578 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.578 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.578 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.578 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.578 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.579 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.579 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.579 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.579 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.579 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.579 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.579 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.579 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.580 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.580 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.580 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.580 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.580 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.580 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.580 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.580 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.581 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.581 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.581 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.581 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.581 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.581 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.581 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.581 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.582 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.582 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.582 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.582 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.582 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.582 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.583 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.583 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.583 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.583 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.583 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.583 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.583 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.583 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.583 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.584 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.584 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.584 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.584 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-04-21T09:01:24.584 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-04-21T09:01:24.584 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.584 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.585 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.585 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.585 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.585 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.585 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.585 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-04-21T09:01:24.585 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.585 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.586 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.586 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-04-21T09:01:24.586 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-04-21T09:01:24.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.586 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.586 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.586 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-04-21T09:01:24.587 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.587 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.587 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.587 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-04-21T09:01:24.587 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.587 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.587 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.588 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-21T09:01:24.588 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-21T09:01:24.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.588 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.588 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-04-21T09:01:24.588 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.588 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.589 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.589 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T09:01:24.589 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.589 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.589 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.589 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-04-21T09:01:24.589 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-04-21T09:01:24.590 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.590 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.590 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.590 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-21T09:01:24.590 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-21T09:01:24.590 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-04-21T09:01:24.590 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.590 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T09:01:24.591 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.591 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-04-21T09:01:24.591 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-04-21T09:01:24.591 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.591 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.592 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.592 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.592 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.592 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.592 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-04-21T09:01:24.592 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.592 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.593 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.593 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-04-21T09:01:24.593 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-04-21T09:01:24.593 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.593 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.593 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.593 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-04-21T09:01:24.593 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.594 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.594 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T09:01:24.594 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.594 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.594 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.594 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.594 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-04-21T09:01:24.595 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.595 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.595 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.595 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.595 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.596 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.596 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.596 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.596 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.596 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.596 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.596 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.597 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.597 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.597 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.597 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.597 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.597 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.597 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.598 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.598 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.598 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.598 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.598 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.598 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.598 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.599 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.599 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.599 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.599 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.599 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.599 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.599 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.600 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.600 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.600 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.600 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.600 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.600 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.600 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.601 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.601 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.601 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.601 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.601 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.601 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.601 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.601 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.602 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.602 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.602 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.602 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.602 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.602 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.602 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.603 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.603 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.603 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.603 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.603 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.603 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.603 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.603 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.603 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.604 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.604 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-04-21T09:01:24.604 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.604 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.604 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.604 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.606 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.606 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.606 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.606 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.606 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.606 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.606 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.606 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.607 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.607 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.607 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.607 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.607 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.607 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.607 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.607 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.608 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.608 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.608 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.608 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.608 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.608 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.608 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.608 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.609 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.609 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.609 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.609 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.609 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.609 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.609 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.610 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.610 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.610 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.610 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.610 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.610 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.610 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.610 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.611 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.611 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.611 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.611 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.612 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.612 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.612 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.612 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.612 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.612 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.612 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.612 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.613 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.613 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-04-21T09:01:24.613 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.613 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.614 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.614 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-04-21T09:01:24.614 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-04-21T09:01:24.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.614 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.614 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-04-21T09:01:24.614 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.615 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.615 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.615 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T09:01:24.615 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.615 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.616 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-04-21T09:01:24.616 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-04-21T09:01:24.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.616 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:24.616 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-04-21T09:01:24.616 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.616 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.617 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T09:01:24.617 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.617 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.617 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.617 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.618 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.618 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.618 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.618 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.618 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.618 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.618 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.618 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.619 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.619 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.619 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.619 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.619 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.619 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.619 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.619 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.620 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.620 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.620 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.620 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.620 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.620 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.620 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.621 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.621 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.621 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.621 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.621 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.621 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.622 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.622 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.622 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.622 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.622 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.623 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.623 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.623 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.623 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.623 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.623 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.623 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.623 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.624 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.624 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.624 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.624 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.624 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.624 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.624 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.625 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.625 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.625 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.625 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.625 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.625 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.625 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.626 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.626 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-04-21T09:01:24.626 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.626 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.626 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.627 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-04-21T09:01:24.627 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-04-21T09:01:24.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.627 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.627 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.627 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-04-21T09:01:24.627 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.627 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.628 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.628 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-04-21T09:01:24.628 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T09:01:24.628 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.628 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.628 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.629 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.629 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.629 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.629 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.629 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.629 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.630 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.630 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.630 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.630 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.630 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.630 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.630 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.630 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.631 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.631 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.631 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.631 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.632 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.632 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.632 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.632 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.632 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.632 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.632 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.632 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.633 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.633 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.633 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.633 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.633 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.633 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.634 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.634 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.634 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.634 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.634 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.634 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.634 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.635 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.635 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.635 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.635 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.636 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.636 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.636 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.636 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.636 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.636 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.637 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.637 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.637 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-04-21T09:01:24.637 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-04-21T09:01:24.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.637 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.637 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.638 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.638 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.638 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.638 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.638 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-04-21T09:01:24.638 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T09:01:24.638 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.639 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.639 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-04-21T09:01:24.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-04-21T09:01:24.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.639 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.639 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.639 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-04-21T09:01:24.640 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.640 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.640 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.640 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-04-21T09:01:24.640 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.640 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.640 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.640 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-04-21T09:01:24.641 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-04-21T09:01:24.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.641 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.641 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.641 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-04-21T09:01:24.641 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.641 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.641 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.642 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T09:01:24.642 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.642 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.642 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.642 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-04-21T09:01:24.642 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-04-21T09:01:24.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.642 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.643 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.643 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.643 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.643 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.643 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-04-21T09:01:24.643 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.644 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.644 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-04-21T09:01:24.644 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-04-21T09:01:24.644 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.644 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.644 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.644 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-04-21T09:01:24.645 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.645 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.645 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T09:01:24.645 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T09:01:24.645 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.645 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.645 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.645 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-04-21T09:01:24.646 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.646 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.646 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.646 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.647 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.647 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.647 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.647 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.647 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.648 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.648 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.648 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.648 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.648 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.648 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.648 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.648 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.648 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.649 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.649 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.649 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.649 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.649 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.650 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.650 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.650 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.650 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.650 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.650 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.650 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.651 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.651 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.651 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.651 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.651 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.651 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.651 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.652 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.652 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.652 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.652 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.652 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.652 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.652 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.652 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.653 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.653 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.653 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.653 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.653 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.653 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.654 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.654 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.654 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.654 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.654 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.654 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.654 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.655 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.655 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.655 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.655 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-04-21T09:01:24.656 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.656 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.656 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.656 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.656 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.657 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.657 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.657 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.657 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.657 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.657 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.657 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.657 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.658 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.658 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.658 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.658 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.658 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.658 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.658 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.659 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.659 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.659 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.659 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.659 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.660 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.660 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.660 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.660 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.660 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.660 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.660 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.661 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.661 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.661 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.661 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.661 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.661 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.662 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.662 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.662 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.662 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.662 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.662 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.662 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.662 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.663 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.663 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.663 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.663 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.663 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.663 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.663 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.664 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.664 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.664 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.664 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.664 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.664 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.664 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.664 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.665 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.665 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.665 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-04-21T09:01:24.665 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.665 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.665 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.666 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.666 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.666 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.666 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.666 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.666 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.667 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.667 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.667 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.667 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.667 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.667 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.667 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.667 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.668 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.668 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.668 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.668 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.668 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.668 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.669 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.669 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.669 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.669 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.669 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.669 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.669 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.670 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.670 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.670 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.670 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.670 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.670 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.670 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.671 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.671 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.671 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.671 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.671 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.671 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.671 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.671 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.671 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.672 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.672 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.672 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.672 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.672 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.672 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.673 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.673 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.673 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.673 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.673 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.673 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.673 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.674 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.674 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.674 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.674 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.674 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.674 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-21T09:01:24.675 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.675 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.675 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.675 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-04-21T09:01:24.675 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-04-21T09:01:24.675 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.675 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.675 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.676 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.676 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.676 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.676 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.676 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T09:01:24.676 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.676 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.676 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.677 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.677 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.677 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.677 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.677 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.677 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.677 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.677 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.678 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.678 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.678 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.678 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.678 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.678 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.678 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.678 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.679 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.679 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.679 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.679 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.679 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.680 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.680 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.680 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.680 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.680 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.681 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.681 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.681 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.681 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.681 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.682 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.682 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.682 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.682 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.683 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.683 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.683 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.683 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.683 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.684 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.684 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.684 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.684 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.684 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.685 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.685 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.685 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.685 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.685 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.686 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.686 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.686 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.686 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.687 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.687 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.687 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.687 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.688 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.688 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.688 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.688 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.688 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.689 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.689 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.689 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-04-21T09:01:24.689 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-21T09:01:24.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.690 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.690 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.690 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.691 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.691 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.691 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-21T09:01:24.691 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-04-21T09:01:24.691 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-04-21T09:01:24.693 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.693 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-04-21T09:01:24.693 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.694 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.694 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.694 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.694 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.694 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-04-21T09:01:24.695 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.695 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.695 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.695 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.695 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.696 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.696 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.696 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.696 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.696 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.696 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.696 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.696 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.696 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.697 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.697 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.697 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.697 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.697 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.697 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.697 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.698 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.698 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.698 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.698 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.698 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.698 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.699 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.699 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.699 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.699 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.699 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.699 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.700 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.700 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.700 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.700 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.700 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.700 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.701 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.701 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.701 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.701 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.701 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.701 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.701 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.702 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.702 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.702 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.702 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.702 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.702 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.703 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.703 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.703 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.703 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.703 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.703 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.704 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-04-21T09:01:24.704 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.704 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.704 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.704 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.704 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.705 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.705 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.705 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.705 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.705 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.705 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.705 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.705 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.706 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.706 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.706 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.706 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.706 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.706 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.706 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.706 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.707 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.707 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.707 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.707 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.707 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.708 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.708 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.708 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.708 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.708 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.708 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.708 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.709 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.709 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.709 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.709 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.709 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.709 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.710 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.710 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.710 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.710 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.710 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.710 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.710 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.710 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.711 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.711 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.711 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.711 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.711 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.711 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.712 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.712 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.712 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.712 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.712 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.712 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.712 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.712 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.713 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.713 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-04-21T09:01:24.713 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.714 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.714 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.714 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.714 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.714 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.714 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.714 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.714 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.715 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.715 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.715 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.715 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.715 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.715 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.715 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.715 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.716 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.716 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.716 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.716 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.716 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.717 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.717 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.717 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.717 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.717 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.717 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.717 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.718 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.718 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.718 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.719 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.719 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.719 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.719 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.719 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.719 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.719 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.719 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.720 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.721 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-21T09:01:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-21T09:01:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.722 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-21T09:01:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.723 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-04-21T09:01:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-04-21T09:01:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-21T09:01:24.724 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-04-21T09:01:24.725 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.725 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.725 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-04-21T09:01:24.725 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T09:01:24.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.725 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.726 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-04-21T09:01:24.726 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.726 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.726 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.726 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-04-21T09:01:24.726 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-04-21T09:01:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.727 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T09:01:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-21T09:01:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-21T09:01:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.728 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.729 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.730 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.731 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.733 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.734 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.735 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.737 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-04-21T09:01:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-04-21T09:01:24.738 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-04-21T09:01:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-04-21T09:01:24.739 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-21T09:01:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-21T09:01:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.740 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-21T09:01:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-04-21T09:01:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.741 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-04-21T09:01:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.742 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-04-21T09:01:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-04-21T09:01:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.743 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-04-21T09:01:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-04-21T09:01:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.744 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T09:01:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T09:01:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.745 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.746 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.747 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.749 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.751 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.753 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.754 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.754 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.754 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-04-21T09:01:24.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-04-21T09:01:24.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.755 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T09:01:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T09:01:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.756 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-04-21T09:01:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.757 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-04-21T09:01:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-04-21T09:01:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-04-21T09:01:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.758 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-21T09:01:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-21T09:01:24.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.760 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.760 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.760 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-21T09:01:24.760 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.760 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.760 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.760 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T09:01:24.760 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.761 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.761 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.761 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-21T09:01:24.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-21T09:01:24.761 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.761 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.761 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-04-21T09:01:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T09:01:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-04-21T09:01:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-04-21T09:01:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-04-21T09:01:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.763 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-21T09:01:24.764 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-21T09:01:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-04-21T09:01:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-04-21T09:01:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-04-21T09:01:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.766 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-21T09:01:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-21T09:01:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-21T09:01:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-21T09:01:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-21T09:01:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.767 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.768 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-21T09:01:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.769 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.770 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.771 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.772 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.774 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.776 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-04-21T09:01:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-04-21T09:01:24.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.778 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-04-21T09:01:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T09:01:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.780 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-04-21T09:01:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-04-21T09:01:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.782 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-04-21T09:01:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.783 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-04-21T09:01:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-04-21T09:01:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-21T09:01:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-21T09:01:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.785 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.786 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.787 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.788 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.790 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.792 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.792 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.792 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.792 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.792 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.792 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.792 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.792 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.793 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.794 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.795 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.795 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.795 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.795 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.795 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.795 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-21T09:01:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-21T09:01:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.796 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-21T09:01:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-21T09:01:24.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.798 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.799 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-04-21T09:01:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.801 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.802 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.803 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.804 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.805 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.806 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.808 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-04-21T09:01:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.810 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.811 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.812 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.813 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.814 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.815 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.816 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.817 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-04-21T09:01:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-04-21T09:01:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.818 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-04-21T09:01:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T09:01:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-04-21T09:01:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.820 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.821 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.822 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.824 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.825 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.826 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.828 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-04-21T09:01:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-04-21T09:01:24.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.830 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-04-21T09:01:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-04-21T09:01:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-04-21T09:01:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.832 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-04-21T09:01:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-04-21T09:01:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.833 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.834 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-04-21T09:01:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-04-21T09:01:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.835 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-04-21T09:01:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-04-21T09:01:24.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.837 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-04-21T09:01:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-04-21T09:01:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.838 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.839 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-04-21T09:01:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-04-21T09:01:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.840 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.841 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.841 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.841 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.841 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.841 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.841 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.841 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-04-21T09:01:24.842 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-04-21T09:01:24.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.842 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.842 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.842 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.842 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.842 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.842 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.843 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.843 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.843 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.843 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.843 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-04-21T09:01:24.843 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-04-21T09:01:24.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.843 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.844 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.844 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.844 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-21T09:01:24.844 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-04-21T09:01:24.844 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.844 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.844 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.844 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-04-21T09:01:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-04-21T09:01:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.845 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.846 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-04-21T09:01:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-04-21T09:01:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.847 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-04-21T09:01:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-04-21T09:01:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.848 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.849 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-21T09:01:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.850 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-04-21T09:01:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-04-21T09:01:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.851 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T09:01:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-04-21T09:01:24.852 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-04-21T09:01:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.853 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-04-21T09:01:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.854 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.855 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.857 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.858 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.859 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.859 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.859 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.859 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.859 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.859 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.859 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.859 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.860 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.860 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.860 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.860 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.860 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.860 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.860 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.860 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.861 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.861 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.861 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.861 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.861 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.861 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.861 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.861 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.861 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.862 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.862 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.862 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.862 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.862 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.862 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.863 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-04-21T09:01:24.863 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.863 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.863 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.863 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.863 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.863 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.864 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.864 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.864 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.864 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.864 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.864 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.864 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.864 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.864 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.865 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.865 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.865 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.865 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.865 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.865 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.865 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.867 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.867 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.867 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.867 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.867 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.867 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.868 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.868 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.868 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.868 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.868 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.868 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.869 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.869 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.869 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.869 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.869 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.869 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.869 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.870 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.870 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.870 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.870 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.870 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.870 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.871 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.871 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.871 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.871 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.871 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.871 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.871 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.871 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.872 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-04-21T09:01:24.872 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.872 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.872 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.873 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.873 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.873 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.873 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.874 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.874 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.874 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.874 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.874 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.874 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.874 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.874 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.875 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.875 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.875 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.875 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.875 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.875 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.876 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.876 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.876 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.876 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.876 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.876 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.876 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.876 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.877 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.877 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.877 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.877 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.878 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.878 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.878 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.878 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.878 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.878 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.878 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.878 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.879 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.879 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.879 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.879 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.879 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.879 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.879 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.880 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.880 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.880 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.880 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.880 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.880 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.880 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.880 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.881 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.881 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.881 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.881 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-04-21T09:01:24.882 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.882 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.882 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.882 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-04-21T09:01:24.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-04-21T09:01:24.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.883 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.883 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T09:01:24.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.883 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.883 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.883 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.884 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.884 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.884 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-21T09:01:24.884 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-21T09:01:24.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.884 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.885 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-04-21T09:01:24.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.885 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.885 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.885 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.886 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.886 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.886 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.886 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.886 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.886 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.887 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.887 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.887 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.887 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.887 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.887 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.887 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.887 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.888 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.888 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.888 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.888 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.888 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.888 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.888 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.889 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.889 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.889 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.890 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.890 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.890 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.890 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.890 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.891 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.891 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.891 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.891 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.891 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.891 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.891 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.892 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.892 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.892 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.892 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.892 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.892 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.892 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.892 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.893 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.893 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.893 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.893 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.893 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.893 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.893 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.894 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.894 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.894 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.894 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-04-21T09:01:24.895 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.895 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.895 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.895 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-04-21T09:01:24.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-04-21T09:01:24.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.895 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.896 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.896 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.896 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.896 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.896 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T09:01:24.896 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.896 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.897 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.897 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-04-21T09:01:24.897 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-04-21T09:01:24.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.897 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.898 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.898 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.898 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.898 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.898 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T09:01:24.898 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.898 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.898 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.899 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-04-21T09:01:24.899 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-04-21T09:01:24.899 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.899 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.899 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.899 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.901 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.901 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.901 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.901 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T09:01:24.901 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.901 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.902 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.902 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-04-21T09:01:24.902 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-04-21T09:01:24.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.902 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.902 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.903 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.903 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.903 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.903 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.903 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T09:01:24.903 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.903 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.903 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.904 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-04-21T09:01:24.904 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-04-21T09:01:24.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.904 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.904 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.904 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.904 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.904 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.905 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.905 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T09:01:24.905 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.905 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.905 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.905 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-04-21T09:01:24.905 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-04-21T09:01:24.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.906 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.906 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.906 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.906 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.906 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.906 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.906 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.907 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.907 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.907 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.907 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-04-21T09:01:24.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-04-21T09:01:24.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.907 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.907 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.908 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.908 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.908 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.908 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.908 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.908 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.908 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.908 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.909 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-04-21T09:01:24.909 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-04-21T09:01:24.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.909 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.909 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T09:01:24.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.910 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.910 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.910 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.910 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.910 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.910 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.910 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-21T09:01:24.911 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-21T09:01:24.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.911 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.911 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.911 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.911 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.911 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.911 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.911 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.912 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.912 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.912 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.912 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-21T09:01:24.912 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-21T09:01:24.912 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.912 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.913 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.913 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-04-21T09:01:24.913 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-04-21T09:01:24.913 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.913 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.913 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.913 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.913 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.914 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.914 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-04-21T09:01:24.914 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-04-21T09:01:24.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.914 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.914 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-04-21T09:01:24.915 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.915 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.915 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.915 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.915 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.915 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.915 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.915 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-04-21T09:01:24.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-04-21T09:01:24.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.916 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.916 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.916 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.916 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.916 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.916 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.917 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.917 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.917 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.917 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.917 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-04-21T09:01:24.917 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-04-21T09:01:24.917 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.918 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.918 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.918 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.918 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.918 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.918 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.918 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.918 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.919 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.919 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.919 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.919 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.919 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.919 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.920 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.920 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.920 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.920 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.920 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.920 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.920 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.920 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.921 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.921 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.921 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.921 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.921 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.921 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.921 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.921 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.922 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.922 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.922 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.922 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.922 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.922 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.922 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.923 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.923 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.923 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.923 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.923 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.923 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.924 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.924 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.924 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.924 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.924 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.924 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.924 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.924 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.925 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.925 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.925 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.925 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.925 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.925 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.926 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.926 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.926 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.926 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.926 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.926 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.927 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.927 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.927 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.927 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.927 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.927 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.927 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-04-21T09:01:24.928 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-04-21T09:01:24.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.928 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.928 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.928 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.928 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.928 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.929 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.929 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.929 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.929 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.929 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-04-21T09:01:24.929 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-04-21T09:01:24.929 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.930 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.930 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.930 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.930 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.930 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.930 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.930 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.930 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.931 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.931 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-04-21T09:01:24.931 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-04-21T09:01:24.931 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.931 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.931 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.931 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T09:01:24.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.932 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.932 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.932 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.932 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.932 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.932 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.933 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.933 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-04-21T09:01:24.933 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.933 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.933 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.933 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.934 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.934 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.934 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.934 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.934 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.934 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.934 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.935 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.935 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.935 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.935 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.935 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.935 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.935 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.935 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.936 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.936 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.936 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.936 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.936 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.937 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.937 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.937 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.937 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.937 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.937 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.938 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.938 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.938 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.938 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.938 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.938 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.938 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.939 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.939 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.939 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.939 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.939 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.939 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.940 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.940 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.940 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.940 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.940 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.940 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.940 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.940 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.941 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.941 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.941 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.941 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.941 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.942 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.942 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.942 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.942 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.942 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-21T09:01:24.943 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.943 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.943 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.943 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.943 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.943 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.943 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.944 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.944 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.944 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.944 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.944 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.944 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.944 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.944 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.945 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.945 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.945 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.945 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.945 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.945 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.945 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.946 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.946 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.946 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.946 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.946 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.946 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.946 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.946 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.947 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.947 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.947 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.947 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.947 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.948 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.948 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.948 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.948 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.948 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.948 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.948 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.948 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.949 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.949 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.949 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.949 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.949 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.949 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.949 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.949 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.950 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.950 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.950 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.950 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.950 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.951 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.951 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.951 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.951 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.951 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.951 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.952 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-04-21T09:01:24.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-04-21T09:01:24.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.952 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.952 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.952 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-04-21T09:01:24.952 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.952 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.953 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T09:01:24.953 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.953 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.953 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.953 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-04-21T09:01:24.953 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-04-21T09:01:24.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.954 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.954 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.954 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.954 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.954 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.954 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.954 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-04-21T09:01:24.955 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.955 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.955 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-04-21T09:01:24.955 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-04-21T09:01:24.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.955 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.956 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-21T09:01:24.956 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.956 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.956 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.956 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T09:01:24.956 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.956 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.956 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.957 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-04-21T09:01:24.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T09:01:24.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.957 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.957 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:24.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.958 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.958 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.958 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T09:01:24.958 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.958 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.958 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.959 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-04-21T09:01:24.959 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.959 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.959 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.959 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-04-21T09:01:24.959 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-04-21T09:01:24.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-21T09:01:24.960 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.960 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.960 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-04-21T09:01:24.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.960 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.960 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.960 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.960 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.961 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-04-21T09:01:24.961 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-04-21T09:01:24.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.961 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.962 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.962 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.962 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.962 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.962 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.962 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.962 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.962 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.963 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-21T09:01:24.963 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-21T09:01:24.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.963 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.963 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.963 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-21T09:01:24.963 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.963 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.964 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.964 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.964 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.964 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.964 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.964 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.965 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.965 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.965 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.965 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.965 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.965 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.965 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.965 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.965 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.966 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.966 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.966 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.966 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.966 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.966 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.966 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.967 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.967 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.967 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.967 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.968 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.968 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.968 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.968 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.968 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.969 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.969 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.969 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.969 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.970 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.970 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.970 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.970 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.970 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.970 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.970 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.970 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.971 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.971 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.971 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.971 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.971 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.972 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.972 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.972 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.972 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.972 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.973 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.973 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-04-21T09:01:24.973 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-04-21T09:01:24.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.973 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.974 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.974 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.974 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.974 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.974 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.974 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.974 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.975 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-04-21T09:01:24.975 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-04-21T09:01:24.975 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.975 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.975 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.975 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-04-21T09:01:24.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.975 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.975 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.976 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.976 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.976 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.976 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-04-21T09:01:24.976 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-04-21T09:01:24.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.977 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:24.977 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T09:01:24.977 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.977 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.977 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.977 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.978 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.978 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:24.978 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-21T09:01:24.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.979 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 25018, 2024-04-21T09:01:24.979 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "z", 2024-04-21T09:01:24.979 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-04-21T09:01:24.979 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-04-21T09:01:24.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:24.979 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-04-21T09:01:24.979 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:24.979 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:24.980 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:24.980 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-21T09:01:24.980 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-21T09:01:24.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:24.980 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.980 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-21T09:01:24.980 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.980 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.981 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.981 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T09:01:24.981 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.981 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.981 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:24.981 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:24.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.982 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.982 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.982 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.982 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.982 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.982 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.982 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.983 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.983 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.983 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.983 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.983 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.983 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.983 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.983 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.984 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.984 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:24.984 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:24.984 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.984 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.984 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.984 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.985 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.985 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.985 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.985 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.985 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.985 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.986 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:24.986 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:24.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.986 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.986 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:24.986 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.986 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.987 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:24.987 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:24.987 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:24.987 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:24.987 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:24.987 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:24.987 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:24.987 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:24.988 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.988 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.988 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.988 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.988 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:24.988 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:24.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:24.988 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.989 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:24.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.989 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.989 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.989 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:24.989 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.990 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.990 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-04-21T09:01:24.990 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-04-21T09:01:24.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.990 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.990 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.991 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.991 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.991 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.991 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-04-21T09:01:24.991 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.991 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.991 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.992 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-04-21T09:01:24.992 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-04-21T09:01:24.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.992 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.992 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-04-21T09:01:24.992 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.993 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-04-21T09:01:24.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.993 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.993 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-04-21T09:01:24.993 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-04-21T09:01:24.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.994 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.994 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.994 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.994 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.994 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-04-21T09:01:24.994 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.995 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.995 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.995 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-04-21T09:01:24.995 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-04-21T09:01:24.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.995 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.995 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.996 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.996 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.996 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.996 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.996 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-04-21T09:01:24.996 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.996 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.997 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-04-21T09:01:24.997 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-04-21T09:01:24.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:24.997 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.997 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-04-21T09:01:24.997 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.998 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.998 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.998 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-04-21T09:01:24.998 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:24.998 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:24.998 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:24.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:24.998 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-04-21T09:01:24.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-04-21T09:01:24.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:24.999 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:24.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:24.999 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:24.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:24.999 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:24.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:24.999 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-04-21T09:01:25.000 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.000 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.000 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-04-21T09:01:25.000 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-04-21T09:01:25.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.000 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.001 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.001 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.001 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T09:01:25.001 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.001 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.001 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.002 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-04-21T09:01:25.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-04-21T09:01:25.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.002 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.002 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.002 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.003 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.003 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.003 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-04-21T09:01:25.003 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.003 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.003 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.003 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.003 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.004 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-04-21T09:01:25.004 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.004 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.004 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.004 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-04-21T09:01:25.004 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-04-21T09:01:25.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.005 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.005 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.005 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.005 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.005 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T09:01:25.005 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.006 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.006 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.006 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-04-21T09:01:25.006 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-04-21T09:01:25.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.006 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.007 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-04-21T09:01:25.007 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.007 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T09:01:25.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T09:01:25.008 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.008 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.008 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-04-21T09:01:25.008 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-04-21T09:01:25.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-21T09:01:25.008 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.009 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-21T09:01:25.009 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-21T09:01:25.009 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-21T09:01:25.009 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.009 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T09:01:25.009 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T09:01:25.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.010 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-04-21T09:01:25.010 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-04-21T09:01:25.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-21T09:01:25.010 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-04-21T09:01:25.011 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-21T09:01:25.011 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-04-21T09:01:25.011 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.011 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T09:01:25.011 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.011 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.012 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-04-21T09:01:25.012 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-04-21T09:01:25.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.012 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.012 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.012 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T09:01:25.012 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.012 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.013 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T09:01:25.013 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T09:01:25.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.013 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.013 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-04-21T09:01:25.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-04-21T09:01:25.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:25.014 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.014 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-04-21T09:01:25.014 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-04-21T09:01:25.014 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-04-21T09:01:25.014 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.014 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-04-21T09:01:25.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T09:01:25.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.015 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.015 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-04-21T09:01:25.015 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-04-21T09:01:25.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.015 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-04-21T09:01:25.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.016 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.016 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.016 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T09:01:25.016 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T09:01:25.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.016 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.017 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-04-21T09:01:25.017 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-04-21T09:01:25.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-21T09:01:25.017 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.017 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-21T09:01:25.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-21T09:01:25.018 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-21T09:01:25.018 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.018 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T09:01:25.018 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T09:01:25.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.019 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.019 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.019 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.019 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.019 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.019 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.019 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.020 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.020 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.020 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.020 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.020 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.020 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.020 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.020 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.021 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.021 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.021 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.021 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.021 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.021 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.022 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.022 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.022 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.022 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.022 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.023 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.023 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.023 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.023 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.023 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.024 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.024 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.024 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.024 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.024 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.024 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.024 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.024 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.025 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.025 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.025 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.025 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.025 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.025 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.025 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.026 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.026 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.026 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.026 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.026 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.026 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.026 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.026 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.027 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.027 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.027 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.027 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.027 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-04-21T09:01:25.027 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-04-21T09:01:25.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:25.028 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.028 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-21T09:01:25.028 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.028 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.028 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.028 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T09:01:25.028 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.029 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.029 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.029 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-21T09:01:25.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-21T09:01:25.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.029 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.029 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-04-21T09:01:25.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.030 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.030 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.030 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-04-21T09:01:25.030 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-04-21T09:01:25.030 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-04-21T09:01:25.030 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-04-21T09:01:25.030 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-04-21T09:01:25.031 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.031 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-04-21T09:01:25.031 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.031 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.031 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.031 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-04-21T09:01:25.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-04-21T09:01:25.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.032 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.032 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.032 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.032 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.032 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T09:01:25.033 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.033 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.033 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.033 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-21T09:01:25.033 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-21T09:01:25.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.033 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-21T09:01:25.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.034 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.034 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.034 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T09:01:25.034 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.034 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.034 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.035 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-04-21T09:01:25.035 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-04-21T09:01:25.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.035 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.035 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-21T09:01:25.035 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-21T09:01:25.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.036 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T09:01:25.036 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T09:01:25.036 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.036 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.036 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-04-21T09:01:25.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-04-21T09:01:25.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-21T09:01:25.037 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.037 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.037 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:25.037 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.037 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.037 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.038 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T09:01:25.038 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.038 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.038 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-04-21T09:01:25.039 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.039 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.039 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.039 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-04-21T09:01:25.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-04-21T09:01:25.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:25.039 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.040 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-21T09:01:25.040 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.040 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.040 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.040 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T09:01:25.040 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.041 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.041 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-04-21T09:01:25.041 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-04-21T09:01:25.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.041 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.041 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-04-21T09:01:25.042 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.042 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.042 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.042 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T09:01:25.042 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.043 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-04-21T09:01:25.043 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-04-21T09:01:25.043 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.043 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.043 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.043 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-04-21T09:01:25.043 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.043 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.044 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.044 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T09:01:25.044 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.044 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.044 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.044 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-04-21T09:01:25.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-04-21T09:01:25.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.045 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.045 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.045 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.045 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T09:01:25.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T09:01:25.046 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.046 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.046 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-04-21T09:01:25.046 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-04-21T09:01:25.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.047 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-21T09:01:25.047 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.047 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.047 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T09:01:25.047 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.047 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.048 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-04-21T09:01:25.048 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T09:01:25.048 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:25.048 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.048 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.048 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-04-21T09:01:25.048 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.049 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.049 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.049 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T09:01:25.049 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.049 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.049 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.049 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-04-21T09:01:25.049 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-04-21T09:01:25.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.050 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.050 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.050 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.050 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.050 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.051 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-04-21T09:01:25.051 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-04-21T09:01:25.051 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-04-21T09:01:25.051 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.051 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T09:01:25.051 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.051 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.051 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.052 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-04-21T09:01:25.052 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-04-21T09:01:25.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.052 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.052 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.052 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.052 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.052 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.053 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.053 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T09:01:25.053 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.053 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.053 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.053 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-04-21T09:01:25.053 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-04-21T09:01:25.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.054 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.054 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.054 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.054 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.054 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.054 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.054 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T09:01:25.054 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.055 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-04-21T09:01:25.055 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-04-21T09:01:25.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.055 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.055 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.056 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.056 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.056 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T09:01:25.056 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.056 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.056 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.057 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-04-21T09:01:25.057 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-04-21T09:01:25.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.057 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.057 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.057 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T09:01:25.057 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-21T09:01:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.058 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-04-21T09:01:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-04-21T09:01:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T09:01:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.059 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-04-21T09:01:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-04-21T09:01:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-04-21T09:01:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T09:01:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-04-21T09:01:25.061 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-04-21T09:01:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-04-21T09:01:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T09:01:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-21T09:01:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-04-21T09:01:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-04-21T09:01:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T09:01:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.064 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-04-21T09:01:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-04-21T09:01:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-04-21T09:01:25.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T09:01:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-04-21T09:01:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-04-21T09:01:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-04-21T09:01:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.067 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T09:01:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-04-21T09:01:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-04-21T09:01:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-04-21T09:01:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T09:01:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.069 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-04-21T09:01:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-04-21T09:01:25.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-04-21T09:01:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T09:01:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-04-21T09:01:25.071 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-04-21T09:01:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-04-21T09:01:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.072 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T09:01:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-04-21T09:01:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.073 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-04-21T09:01:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-04-21T09:01:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T09:01:25.074 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-21T09:01:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-04-21T09:01:25.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-04-21T09:01:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T09:01:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.076 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-04-21T09:01:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-04-21T09:01:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-04-21T09:01:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.077 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T09:01:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-21T09:01:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.078 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-04-21T09:01:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-04-21T09:01:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T09:01:25.079 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-04-21T09:01:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-04-21T09:01:25.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-04-21T09:01:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T09:01:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.081 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-04-21T09:01:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-04-21T09:01:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-04-21T09:01:25.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T09:01:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-04-21T09:01:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.083 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-04-21T09:01:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-04-21T09:01:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.084 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T09:01:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-04-21T09:01:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-04-21T09:01:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-21T09:01:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T09:01:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-04-21T09:01:25.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T09:01:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-04-21T09:01:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T09:01:25.088 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-04-21T09:01:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-04-21T09:01:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.089 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-04-21T09:01:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T09:01:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.090 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-04-21T09:01:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-04-21T09:01:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-04-21T09:01:25.091 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-04-21T09:01:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-04-21T09:01:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-04-21T09:01:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T09:01:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-04-21T09:01:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-04-21T09:01:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-21T09:01:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-04-21T09:01:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-04-21T09:01:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-04-21T09:01:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T09:01:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T09:01:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-04-21T09:01:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-04-21T09:01:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-21T09:01:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-04-21T09:01:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-04-21T09:01:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-04-21T09:01:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T09:01:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-04-21T09:01:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-04-21T09:01:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T09:01:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-04-21T09:01:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-04-21T09:01:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T09:01:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T09:01:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T09:01:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T09:01:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T09:01:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T09:01:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T09:01:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-04-21T09:01:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T09:01:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-04-21T09:01:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-04-21T09:01:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-04-21T09:01:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-04-21T09:01:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-04-21T09:01:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T09:01:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-04-21T09:01:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-04-21T09:01:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-04-21T09:01:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-21T09:01:25.123 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-21T09:01:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-04-21T09:01:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-04-21T09:01:25.124 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-04-21T09:01:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T09:01:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-04-21T09:01:25.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-04-21T09:01:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-04-21T09:01:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.126 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T09:01:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-04-21T09:01:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-04-21T09:01:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.127 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T09:01:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T09:01:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.128 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-04-21T09:01:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-04-21T09:01:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.129 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T09:01:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-04-21T09:01:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-04-21T09:01:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.131 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T09:01:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-04-21T09:01:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-04-21T09:01:25.132 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T09:01:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-04-21T09:01:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-04-21T09:01:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.134 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T09:01:25.135 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-04-21T09:01:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-04-21T09:01:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T09:01:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.137 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-04-21T09:01:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-04-21T09:01:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-04-21T09:01:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.138 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-04-21T09:01:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-04-21T09:01:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-04-21T09:01:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T09:01:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-04-21T09:01:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-04-21T09:01:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.142 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T09:01:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-04-21T09:01:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-04-21T09:01:25.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T09:01:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.144 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-04-21T09:01:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-04-21T09:01:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.145 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T09:01:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-04-21T09:01:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-04-21T09:01:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T09:01:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-04-21T09:01:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-04-21T09:01:25.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T09:01:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.150 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-04-21T09:01:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.153 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.154 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.155 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.155 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.155 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.155 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.155 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.156 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.156 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.156 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.156 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.158 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.158 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.158 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.158 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.159 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.160 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.161 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.162 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.163 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-04-21T09:01:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-04-21T09:01:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.165 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.166 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-04-21T09:01:25.166 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.166 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.166 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.166 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T09:01:25.166 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.166 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-04-21T09:01:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-04-21T09:01:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.167 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-04-21T09:01:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T09:01:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-04-21T09:01:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T09:01:25.169 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T09:01:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.170 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T09:01:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T09:01:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.171 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.172 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T09:01:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T09:01:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.173 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.174 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.174 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.174 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.174 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.174 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.174 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.174 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-04-21T09:01:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T09:01:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.175 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T09:01:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T09:01:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.177 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.178 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T09:01:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T09:01:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.179 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T09:01:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T09:01:25.180 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.181 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T09:01:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T09:01:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T09:01:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T09:01:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.184 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T09:01:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T09:01:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.185 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.186 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T09:01:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T09:01:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.187 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.188 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T09:01:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T09:01:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.190 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T09:01:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T09:01:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T09:01:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T09:01:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T09:01:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T09:01:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.194 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-21T09:01:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T09:01:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T09:01:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.196 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-04-21T09:01:25.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-04-21T09:01:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T09:01:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T09:01:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.199 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-21T09:01:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.200 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T09:01:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T09:01:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.201 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.202 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T09:01:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T09:01:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.203 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T09:01:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T09:01:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.204 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.205 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.206 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T09:01:25.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T09:01:25.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.207 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.207 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.207 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-21T09:01:25.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-21T09:01:25.208 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.208 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.208 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.208 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-04-21T09:01:25.208 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.208 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.208 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.208 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.209 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.209 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.209 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.209 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-04-21T09:01:25.209 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-21T09:01:25.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.210 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.210 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-04-21T09:01:25.210 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.210 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.210 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.210 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.210 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.211 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.211 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.211 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T09:01:25.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T09:01:25.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.211 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.211 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.212 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.212 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.212 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.212 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.212 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.212 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.213 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T09:01:25.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T09:01:25.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.213 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.213 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.213 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.213 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.213 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.214 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.215 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T09:01:25.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T09:01:25.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.215 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.215 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.215 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.215 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.215 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.216 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.216 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.216 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.216 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T09:01:25.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T09:01:25.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.217 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.217 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.217 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.217 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.217 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.217 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.217 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.218 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T09:01:25.218 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T09:01:25.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.218 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.218 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.219 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.219 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.219 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.219 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.219 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T09:01:25.220 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T09:01:25.220 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.220 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.220 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.220 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.220 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.220 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.221 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.221 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T09:01:25.221 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T09:01:25.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.222 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.222 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.222 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.222 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.222 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.222 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.222 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.223 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.223 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T09:01:25.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T09:01:25.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.223 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.223 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.224 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.224 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.224 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.224 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.225 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T09:01:25.225 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T09:01:25.225 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.225 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.225 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.225 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.225 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.225 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.226 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.226 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.226 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.226 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.226 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T09:01:25.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T09:01:25.226 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.227 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.227 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.227 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.228 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.228 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T09:01:25.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T09:01:25.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.228 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.229 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T09:01:25.229 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.229 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.229 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.229 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.230 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.230 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.230 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.231 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T09:01:25.231 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T09:01:25.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.231 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.231 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:25.231 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.232 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.232 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.232 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.232 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.232 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.232 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.232 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T09:01:25.232 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T09:01:25.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.233 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.233 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.233 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-04-21T09:01:25.233 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.233 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.233 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.233 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.234 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.234 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.234 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T09:01:25.234 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T09:01:25.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.235 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-04-21T09:01:25.235 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.235 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.235 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.235 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.236 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.236 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.236 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T09:01:25.236 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T09:01:25.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.236 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.236 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.237 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.237 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.237 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.237 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.237 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.237 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.238 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T09:01:25.238 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T09:01:25.238 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.238 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.238 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.238 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-21T09:01:25.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.239 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.239 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.239 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T09:01:25.239 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T09:01:25.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.240 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.241 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.241 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.241 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.241 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T09:01:25.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T09:01:25.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.241 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.242 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.242 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.242 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.242 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.242 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.242 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.242 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.242 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.243 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-04-21T09:01:25.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-21T09:01:25.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.243 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.243 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.243 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.243 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.244 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.244 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.244 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.244 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.244 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.244 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T09:01:25.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T09:01:25.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.245 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-21T09:01:25.245 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.245 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.245 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.245 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.246 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.246 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.246 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T09:01:25.246 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T09:01:25.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.247 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.247 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.247 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:25.247 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.247 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.247 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.247 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.248 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.248 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T09:01:25.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T09:01:25.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.248 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.248 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.248 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-21T09:01:25.248 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.249 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.249 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.249 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.249 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.249 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.249 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.250 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-04-21T09:01:25.250 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-04-21T09:01:25.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.250 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.250 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.251 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.251 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.251 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.251 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.251 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.251 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.251 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.252 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-04-21T09:01:25.252 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-04-21T09:01:25.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.252 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.252 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.252 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.252 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.252 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.252 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.253 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.253 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.253 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.253 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.253 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-04-21T09:01:25.253 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-04-21T09:01:25.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.253 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.254 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.254 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.254 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.254 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.254 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-04-21T09:01:25.254 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.254 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.255 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.255 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-04-21T09:01:25.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-04-21T09:01:25.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.255 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.255 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.255 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-04-21T09:01:25.255 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.256 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.256 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.256 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.256 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.256 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.256 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.256 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-21T09:01:25.257 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-21T09:01:25.257 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.257 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.257 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.257 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.257 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.258 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.258 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.258 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.258 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.258 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.258 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.259 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.259 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.259 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.259 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.259 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.259 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.260 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.260 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.260 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.260 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.260 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.260 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.260 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.261 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.261 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.261 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.261 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.261 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.262 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.262 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.262 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.262 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.262 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.262 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.262 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.262 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.263 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.263 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.263 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.263 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.263 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.264 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.264 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.264 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.264 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.264 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.264 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.264 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.265 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.265 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.265 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.265 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.265 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.265 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.265 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.265 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.266 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.266 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.266 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.266 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.266 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.266 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.266 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.267 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.267 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.267 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.267 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.267 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.267 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-04-21T09:01:25.268 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-04-21T09:01:25.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.268 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.268 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.268 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.268 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.269 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-04-21T09:01:25.269 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.269 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.269 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-04-21T09:01:25.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T09:01:25.269 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.270 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.270 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.270 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.270 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.270 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.270 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.270 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.271 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-21T09:01:25.271 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-21T09:01:25.271 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.271 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.271 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.271 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-21T09:01:25.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.272 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.272 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.272 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.272 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.272 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.273 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.273 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-21T09:01:25.273 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-21T09:01:25.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.273 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.273 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.273 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-04-21T09:01:25.274 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.274 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.274 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.274 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.274 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.274 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.274 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.275 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-21T09:01:25.275 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-21T09:01:25.275 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.275 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.275 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.275 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.275 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.276 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-04-21T09:01:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-04-21T09:01:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-04-21T09:01:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.277 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-21T09:01:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-21T09:01:25.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-04-21T09:01:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.279 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-04-21T09:01:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-21T09:01:25.280 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-21T09:01:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-21T09:01:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-21T09:01:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-21T09:01:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.281 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-04-21T09:01:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-04-21T09:01:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.283 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.283 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.283 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.283 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.283 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.283 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.283 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.284 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.284 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-21T09:01:25.284 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.284 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.284 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.285 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-04-21T09:01:25.285 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-04-21T09:01:25.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.285 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.285 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.285 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.285 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.285 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.286 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-04-21T09:01:25.286 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.286 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.286 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.286 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.286 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.286 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.287 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.287 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.287 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.287 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.287 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.287 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.287 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.288 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.288 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.288 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.288 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.288 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.288 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.288 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.288 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.289 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.289 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.289 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.289 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.289 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.289 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.290 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.290 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.290 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.290 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.290 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.290 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.290 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.291 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.291 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.291 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.291 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.291 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.291 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.292 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.292 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.292 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.292 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.292 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.292 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.292 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.292 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.293 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.293 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.293 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.293 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.293 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.294 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.294 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.294 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.294 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.294 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.294 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.295 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-04-21T09:01:25.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-04-21T09:01:25.295 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.295 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.295 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.295 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-04-21T09:01:25.295 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.296 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.296 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T09:01:25.296 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.296 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.296 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-04-21T09:01:25.296 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-04-21T09:01:25.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.297 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.297 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.297 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-04-21T09:01:25.297 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.297 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.297 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.297 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T09:01:25.297 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.298 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.298 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.298 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-04-21T09:01:25.298 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-04-21T09:01:25.298 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.298 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.299 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-04-21T09:01:25.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.299 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.299 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.299 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-04-21T09:01:25.299 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.299 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.299 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.300 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-04-21T09:01:25.300 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-04-21T09:01:25.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.300 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.300 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.300 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-21T09:01:25.300 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.300 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.301 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.301 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T09:01:25.301 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.301 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.301 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.301 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-04-21T09:01:25.301 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-04-21T09:01:25.301 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.302 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.302 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.302 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.302 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.302 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.302 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.302 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T09:01:25.302 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.302 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.303 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.303 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-21T09:01:25.303 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-21T09:01:25.303 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.303 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.303 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-21T09:01:25.303 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.304 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.304 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.304 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T09:01:25.304 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.304 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.304 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-04-21T09:01:25.304 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-04-21T09:01:25.305 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.305 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.305 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.305 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-04-21T09:01:25.305 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.305 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.305 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.305 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T09:01:25.305 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-04-21T09:01:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.306 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.307 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.307 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.307 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.307 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.307 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.307 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.307 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.308 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.308 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.308 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.308 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.308 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.308 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.308 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.308 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.308 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.309 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.309 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.309 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.309 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.309 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.309 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.309 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.310 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.310 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.310 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.310 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.310 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.310 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.311 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.311 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.311 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.311 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.313 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.313 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.313 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.313 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.314 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.314 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.314 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.314 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.314 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.314 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.314 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.314 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.314 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.315 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.315 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.315 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.315 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.316 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.316 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.316 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.316 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.316 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.317 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.317 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.317 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-04-21T09:01:25.317 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-04-21T09:01:25.317 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.317 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.317 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.318 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-21T09:01:25.318 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.318 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.318 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.318 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.318 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.318 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.318 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.319 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-04-21T09:01:25.319 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-04-21T09:01:25.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.319 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.319 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-04-21T09:01:25.319 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.319 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.320 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.320 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.320 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.320 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.320 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-21T09:01:25.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-21T09:01:25.321 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.321 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.321 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-21T09:01:25.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.321 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.321 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.322 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.322 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.322 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.322 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.323 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-04-21T09:01:25.323 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-04-21T09:01:25.323 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-04-21T09:01:25.323 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.323 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-04-21T09:01:25.323 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-04-21T09:01:25.323 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.323 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.324 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-04-21T09:01:25.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.324 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.324 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T09:01:25.324 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.324 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.324 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.325 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-04-21T09:01:25.325 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-04-21T09:01:25.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.325 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.325 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.326 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-04-21T09:01:25.326 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.326 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.326 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.326 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T09:01:25.326 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.326 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.327 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-04-21T09:01:25.327 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-04-21T09:01:25.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.327 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.328 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.328 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.328 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-04-21T09:01:25.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.328 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.328 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.328 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-21T09:01:25.329 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-21T09:01:25.329 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.329 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.329 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.329 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-04-21T09:01:25.329 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-04-21T09:01:25.330 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.330 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.330 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T09:01:25.330 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.330 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.330 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.330 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.331 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.331 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.331 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.331 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.332 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.332 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.332 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.332 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.332 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.332 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.332 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.332 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.333 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.333 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.333 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.333 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.333 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.333 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.334 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.334 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.334 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.334 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.334 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.335 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.335 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.335 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.335 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.335 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.336 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.336 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.336 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.336 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.336 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.337 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.337 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.337 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.337 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.337 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.337 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.337 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.337 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.338 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.338 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.338 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.338 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.338 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.338 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.339 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.339 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.339 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.339 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.339 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.339 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.339 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.339 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.340 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-04-21T09:01:25.340 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-04-21T09:01:25.340 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.340 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.341 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.341 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-04-21T09:01:25.342 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.342 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.342 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.343 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-04-21T09:01:25.343 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-04-21T09:01:25.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.343 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.344 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-04-21T09:01:25.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.345 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.345 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.345 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-04-21T09:01:25.345 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.346 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.346 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-21T09:01:25.347 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-21T09:01:25.347 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.347 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.348 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-04-21T09:01:25.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.348 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.349 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.349 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T09:01:25.349 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.350 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.350 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-04-21T09:01:25.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-04-21T09:01:25.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.351 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.351 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.352 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-21T09:01:25.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-21T09:01:25.352 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-04-21T09:01:25.352 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.352 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T09:01:25.353 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.353 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.354 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-04-21T09:01:25.354 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-04-21T09:01:25.354 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.354 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.355 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.355 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.356 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.356 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-04-21T09:01:25.356 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.357 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-04-21T09:01:25.358 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-04-21T09:01:25.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.358 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.359 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-04-21T09:01:25.359 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.359 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.359 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T09:01:25.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.361 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-04-21T09:01:25.362 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.362 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.363 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.363 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.363 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.364 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.364 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.365 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.365 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.365 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.365 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.366 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.366 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.366 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.366 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.367 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.367 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.367 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.367 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.368 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.368 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.369 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.369 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.370 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.370 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.370 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.372 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.372 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.372 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.373 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.373 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.373 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.374 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.374 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.374 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.374 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.374 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.375 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.375 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.375 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.375 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.375 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.375 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.376 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.376 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.377 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.377 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.377 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.377 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.378 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.378 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.378 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.378 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.378 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.378 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.378 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.379 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.379 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.379 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.379 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-04-21T09:01:25.379 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.379 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.380 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.380 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.380 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.380 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.380 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.381 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.381 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.381 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.381 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.381 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.381 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.381 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.381 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.382 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.382 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.382 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.382 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.382 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.383 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.383 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.383 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.383 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.383 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.383 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.384 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.384 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.384 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.384 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.384 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.384 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.385 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.385 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.385 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.385 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.385 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.385 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.385 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.386 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.386 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.386 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.386 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.386 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.386 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.386 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.387 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.387 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.387 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.387 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.387 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.388 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.388 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.388 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.388 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.388 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.388 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.388 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.388 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-04-21T09:01:25.389 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.389 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.389 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.390 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-04-21T09:01:25.390 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-04-21T09:01:25.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.390 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.390 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-04-21T09:01:25.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.390 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.391 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.391 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T09:01:25.391 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.391 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.391 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.391 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-04-21T09:01:25.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-04-21T09:01:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-04-21T09:01:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T09:01:25.393 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.393 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.393 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.393 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.393 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.394 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.394 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.394 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.394 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.394 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.394 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.394 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.395 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.395 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.395 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.395 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.395 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.395 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.396 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.396 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.396 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.396 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.396 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.397 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.397 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.397 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.397 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.397 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.397 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.398 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.398 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.398 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.398 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.398 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.398 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.399 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.399 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.399 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.399 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.399 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.399 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.400 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.400 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.400 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.401 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.401 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.401 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.401 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.401 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.401 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.402 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.402 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-04-21T09:01:25.402 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.402 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.402 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.402 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-04-21T09:01:25.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-04-21T09:01:25.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.403 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.403 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.403 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-04-21T09:01:25.403 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.403 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.404 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-04-21T09:01:25.404 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T09:01:25.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.404 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.404 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.404 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.405 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.405 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.405 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.405 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.405 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.405 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.406 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.406 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.406 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.406 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.406 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.406 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.406 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.407 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.407 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.407 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.408 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.408 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.408 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.408 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.408 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.408 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.408 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.409 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.409 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.409 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.409 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.409 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.409 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.410 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.410 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.410 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.410 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.410 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.410 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.410 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.410 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.411 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.411 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.411 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.411 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.411 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.412 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.412 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.412 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.412 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.412 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.412 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.413 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.413 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-04-21T09:01:25.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-04-21T09:01:25.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.413 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.414 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.414 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.414 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.414 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-04-21T09:01:25.414 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T09:01:25.414 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.414 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.415 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-04-21T09:01:25.415 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-04-21T09:01:25.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.415 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-04-21T09:01:25.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.416 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.416 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-04-21T09:01:25.416 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.416 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.416 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-04-21T09:01:25.417 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-04-21T09:01:25.417 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.417 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.417 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.417 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-04-21T09:01:25.417 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.417 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.417 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.418 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T09:01:25.418 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.418 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.418 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.418 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-04-21T09:01:25.418 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-04-21T09:01:25.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.419 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.419 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.419 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.419 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-04-21T09:01:25.419 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.420 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.420 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-04-21T09:01:25.420 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-04-21T09:01:25.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.420 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-04-21T09:01:25.421 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.421 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.421 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.421 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T09:01:25.421 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T09:01:25.421 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.421 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.421 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.422 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.422 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-04-21T09:01:25.422 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.422 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.422 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.422 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.423 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.423 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.423 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.423 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.423 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.424 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.424 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.424 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.424 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.424 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.424 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.424 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.424 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.424 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.425 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.425 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.426 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.426 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.426 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.426 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.426 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.427 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.427 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.427 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.428 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.428 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.428 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.428 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.428 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.428 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.428 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.428 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.429 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.429 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.429 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.429 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.429 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.430 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.430 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.430 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.430 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.430 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.431 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.431 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.431 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.431 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-04-21T09:01:25.432 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.432 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.432 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.432 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.433 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.433 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.433 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.433 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.433 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.433 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.433 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.433 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.433 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.433 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.434 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.434 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.434 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.434 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.434 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.435 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.435 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.435 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.435 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.435 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.435 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.435 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.436 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.436 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.436 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.437 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.437 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.437 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.437 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.437 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.437 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.437 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.437 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.437 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.437 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.438 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.438 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.438 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.438 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.438 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.438 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.439 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.439 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.439 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.439 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.439 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.440 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.440 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.440 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.440 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-04-21T09:01:25.440 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.440 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.441 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.441 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.441 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.441 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.441 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.441 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.442 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.442 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.442 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.442 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.442 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.442 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.442 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.442 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.443 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.443 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.443 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.444 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.444 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.444 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.445 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.445 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.445 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.445 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.445 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.446 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.446 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.446 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.446 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.446 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.446 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.446 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.446 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.446 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.447 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.447 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.447 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.447 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.447 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.448 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.448 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.448 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.448 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.448 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.448 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.448 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.448 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.449 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.449 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.449 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.449 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.449 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-21T09:01:25.449 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.449 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.449 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.450 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-04-21T09:01:25.450 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-04-21T09:01:25.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.450 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.450 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.450 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.451 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.451 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T09:01:25.451 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.451 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.451 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.451 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.451 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.451 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.452 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.452 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.452 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.452 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.452 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.452 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.452 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.452 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.453 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.453 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.453 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.453 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.453 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.453 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.453 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.454 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.454 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.454 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.454 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.454 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.454 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.454 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.455 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.455 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.455 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.455 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.455 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.455 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.456 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.457 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.458 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.458 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.458 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.458 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.458 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-04-21T09:01:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-21T09:01:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.460 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-21T09:01:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-04-21T09:01:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-04-21T09:01:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-04-21T09:01:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.461 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-04-21T09:01:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.462 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.463 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.464 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.465 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.466 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.467 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.469 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.470 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.471 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.472 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-04-21T09:01:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.474 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.475 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.476 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.478 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.479 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.481 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-04-21T09:01:25.482 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.483 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.484 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.486 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.487 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.488 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.490 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.491 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-21T09:01:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-21T09:01:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-21T09:01:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.492 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-04-21T09:01:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-04-21T09:01:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:25.493 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-21T09:01:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-04-21T09:01:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-04-21T09:01:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T09:01:25.494 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-04-21T09:01:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.495 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-04-21T09:01:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-04-21T09:01:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.496 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T09:01:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-21T09:01:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-21T09:01:25.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.498 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.499 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.500 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.500 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.500 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.500 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.500 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.500 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.502 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.503 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.504 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.505 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.506 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.507 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-04-21T09:01:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-04-21T09:01:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-04-21T09:01:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.508 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-04-21T09:01:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-21T09:01:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-21T09:01:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-21T09:01:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-04-21T09:01:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.510 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-04-21T09:01:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.511 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-04-21T09:01:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-04-21T09:01:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.512 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-04-21T09:01:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-04-21T09:01:25.513 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T09:01:25.514 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T09:01:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.515 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.516 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.518 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.520 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.521 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.522 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-04-21T09:01:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-04-21T09:01:25.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T09:01:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T09:01:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-04-21T09:01:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-04-21T09:01:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-04-21T09:01:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-04-21T09:01:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-21T09:01:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-21T09:01:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-21T09:01:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T09:01:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-21T09:01:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-21T09:01:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-04-21T09:01:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T09:01:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-04-21T09:01:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-04-21T09:01:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-04-21T09:01:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-21T09:01:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-21T09:01:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-04-21T09:01:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-04-21T09:01:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-04-21T09:01:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-21T09:01:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-21T09:01:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-21T09:01:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-21T09:01:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-21T09:01:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-21T09:01:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.538 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.539 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.541 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.543 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.544 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.545 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-04-21T09:01:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-04-21T09:01:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.546 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.547 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-04-21T09:01:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T09:01:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.548 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-04-21T09:01:25.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-04-21T09:01:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-04-21T09:01:25.551 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-04-21T09:01:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-04-21T09:01:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.552 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.553 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-21T09:01:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-21T09:01:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.554 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.555 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.556 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.557 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.558 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.559 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.560 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.561 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.562 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.563 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-21T09:01:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-21T09:01:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.564 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.566 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-21T09:01:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-21T09:01:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-04-21T09:01:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.569 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.570 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.572 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.573 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.574 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.574 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.574 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.574 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.574 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.574 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.574 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.574 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.575 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.575 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.575 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.575 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.575 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.576 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.576 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.576 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.577 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.577 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.577 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.577 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.578 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.578 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.578 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.579 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.579 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.579 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.579 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.580 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.580 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.580 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.581 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-04-21T09:01:25.581 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.581 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.582 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.582 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.582 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.583 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.583 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.583 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.583 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.584 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.584 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.584 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.584 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.585 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.585 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.585 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.586 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.586 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.586 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.587 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.587 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.587 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.588 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.588 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.588 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.589 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.589 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.589 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.589 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.590 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.590 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.590 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.590 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.591 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.591 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.591 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.591 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.592 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.592 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.592 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.592 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.592 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.592 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.592 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.593 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.593 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.593 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.593 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.593 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.593 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.593 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.593 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.594 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.594 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.594 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.594 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.594 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.595 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.595 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.595 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.595 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.595 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.595 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-04-21T09:01:25.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-04-21T09:01:25.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.596 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.596 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-04-21T09:01:25.596 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.596 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.596 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.596 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T09:01:25.597 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.597 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.597 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.597 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.597 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-04-21T09:01:25.598 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.598 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.598 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.598 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.598 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.598 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.598 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.598 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.599 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.599 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.599 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.599 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.599 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.599 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.599 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.599 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.600 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.600 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.600 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.600 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.600 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.600 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.600 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.601 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.601 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.601 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.601 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.601 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.601 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.601 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.601 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.602 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.602 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.602 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.602 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.602 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.602 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.603 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.603 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.603 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.603 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.603 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.603 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.603 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.603 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.603 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.604 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.604 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.604 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.604 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.605 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.605 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.605 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.606 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.606 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.606 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.606 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.606 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-04-21T09:01:25.606 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-04-21T09:01:25.607 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.607 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.607 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.607 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.607 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.607 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.607 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.607 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.608 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.608 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.608 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.608 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-04-21T09:01:25.608 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-04-21T09:01:25.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.608 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.608 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.609 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-04-21T09:01:25.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.609 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.609 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.609 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.609 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.609 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.610 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-04-21T09:01:25.610 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-04-21T09:01:25.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.610 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.610 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.610 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.611 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.611 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.611 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.611 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.611 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.611 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.612 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-04-21T09:01:25.612 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-04-21T09:01:25.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.612 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.612 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.612 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.612 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.612 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.613 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.613 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.613 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.613 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.613 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-04-21T09:01:25.613 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-04-21T09:01:25.613 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:25.614 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.614 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.614 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.614 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.614 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.614 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.615 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.615 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.615 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-04-21T09:01:25.615 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-04-21T09:01:25.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.615 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.616 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.616 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.616 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.616 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.616 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.617 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.617 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.617 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-04-21T09:01:25.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-04-21T09:01:25.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.617 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.618 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.618 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.618 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.618 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.618 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.618 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.618 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.619 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.619 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-04-21T09:01:25.619 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-04-21T09:01:25.619 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.619 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.619 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.620 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.620 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.620 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.620 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.620 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.620 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.620 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.620 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.621 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-04-21T09:01:25.621 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-04-21T09:01:25.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.621 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-21T09:01:25.621 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-04-21T09:01:25.621 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.621 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.622 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.622 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.622 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.622 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-04-21T09:01:25.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-04-21T09:01:25.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.622 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.623 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.623 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.623 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.623 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.623 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.623 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.623 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.624 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.624 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-04-21T09:01:25.624 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-04-21T09:01:25.624 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.624 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.624 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.624 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.624 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.624 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.625 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.625 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.625 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.625 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.625 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-04-21T09:01:25.625 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-04-21T09:01:25.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.626 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.626 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.626 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.626 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.626 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.626 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.627 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.627 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.627 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.627 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.627 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.627 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-21T09:01:25.628 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.628 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.628 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.628 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-04-21T09:01:25.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-04-21T09:01:25.628 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.628 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.628 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.628 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.629 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.629 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.629 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.629 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T09:01:25.629 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.629 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.629 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.629 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-04-21T09:01:25.630 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-04-21T09:01:25.630 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.630 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.630 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.630 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.630 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.630 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.631 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.631 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-04-21T09:01:25.631 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.631 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.631 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.631 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.631 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.632 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.632 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.632 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.632 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.632 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.632 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.632 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.633 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.633 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.633 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.633 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.633 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.633 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.633 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.634 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.634 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.634 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.634 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.634 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.634 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.634 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.634 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.635 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.635 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.635 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.635 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.635 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.636 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.636 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.636 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.636 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.636 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.636 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.637 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.637 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.637 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.637 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.637 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.637 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.637 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.637 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.638 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.638 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.638 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.638 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.638 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.638 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.639 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.639 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.639 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.639 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.639 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.639 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.639 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.639 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.640 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.640 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-04-21T09:01:25.640 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.640 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.640 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.640 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.640 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.641 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.641 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.641 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.641 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.641 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.641 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.641 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.642 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.642 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.642 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.642 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.642 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.642 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.642 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.642 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.643 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.643 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.643 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.643 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.643 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.644 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.644 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.644 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.644 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.644 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.644 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.644 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.644 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.645 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.645 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.645 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.645 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.645 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.645 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.645 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.645 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.646 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.646 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.646 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.646 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.646 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.646 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.646 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.646 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.647 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.647 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.647 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.647 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.647 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.648 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.648 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.648 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.648 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.648 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.648 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-04-21T09:01:25.649 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.649 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.649 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.649 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.649 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.650 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.650 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.650 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.650 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.650 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.650 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.650 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.651 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.651 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.651 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.651 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.651 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.651 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.651 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.651 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.652 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.652 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.652 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.652 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.652 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.652 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.652 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.652 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.653 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.653 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.653 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.653 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.653 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.653 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.654 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.654 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.654 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.654 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.654 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.654 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.655 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.655 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.655 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.655 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.655 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.655 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.655 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.655 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.655 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.655 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.656 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.656 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.656 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.656 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.657 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.657 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.657 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.657 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.657 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.657 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.657 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.657 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.657 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.658 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-04-21T09:01:25.658 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.658 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.658 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.658 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-04-21T09:01:25.658 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-04-21T09:01:25.658 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.659 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.659 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T09:01:25.659 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.659 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.659 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.660 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.660 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.660 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.660 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-21T09:01:25.660 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-21T09:01:25.661 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.661 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.661 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-04-21T09:01:25.661 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.661 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.661 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.661 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.661 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.662 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.662 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.662 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.662 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.662 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.662 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.662 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.663 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.663 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.663 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.663 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.663 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.663 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.663 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.663 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.663 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.664 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.664 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.664 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.664 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.664 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.664 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.664 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.665 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.665 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.665 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.665 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.665 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.665 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.665 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.666 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.666 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.666 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.666 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.666 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.666 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.667 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.667 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.667 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.667 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.667 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.667 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.667 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.667 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.667 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.668 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.668 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.668 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.668 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.668 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.668 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.668 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.668 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.669 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.669 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.669 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.669 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.669 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.669 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.669 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.670 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.670 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.670 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.670 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.670 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.670 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.670 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.670 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.671 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-04-21T09:01:25.671 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.671 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.671 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.671 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-04-21T09:01:25.671 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-04-21T09:01:25.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.672 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.672 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.672 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.672 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.672 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.672 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.672 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T09:01:25.672 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.673 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.673 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-04-21T09:01:25.673 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-04-21T09:01:25.673 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.673 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.673 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.673 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.674 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.674 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.674 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T09:01:25.674 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.674 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.674 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.674 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-04-21T09:01:25.675 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-04-21T09:01:25.675 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.675 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.675 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.675 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.675 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.675 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.675 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.676 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T09:01:25.676 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.676 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.676 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.676 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-04-21T09:01:25.676 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-04-21T09:01:25.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.676 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.677 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.677 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.677 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.677 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.677 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.677 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T09:01:25.678 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.678 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.678 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.678 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-04-21T09:01:25.678 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-04-21T09:01:25.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.679 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.679 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.679 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.679 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.679 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.679 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.680 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T09:01:25.680 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.680 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.680 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.680 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-04-21T09:01:25.680 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-04-21T09:01:25.680 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.680 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.681 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.681 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.681 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.681 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.681 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.681 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.682 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.682 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.682 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.682 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-04-21T09:01:25.682 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-04-21T09:01:25.683 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.683 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.683 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.683 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.683 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.683 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.683 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.683 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.684 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.684 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.684 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.684 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-04-21T09:01:25.684 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-04-21T09:01:25.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.684 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.684 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.685 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T09:01:25.685 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.685 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.685 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.685 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.685 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.685 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.685 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.686 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-21T09:01:25.686 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-21T09:01:25.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.686 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.686 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.687 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.687 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.687 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.687 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.687 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.687 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.688 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-21T09:01:25.688 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-21T09:01:25.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.688 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.689 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.689 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-04-21T09:01:25.689 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-04-21T09:01:25.689 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.690 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.690 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.690 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.690 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.691 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.691 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-04-21T09:01:25.691 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-04-21T09:01:25.692 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.692 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.692 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-04-21T09:01:25.693 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.693 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.693 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.693 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.694 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.694 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.694 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-04-21T09:01:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-04-21T09:01:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.696 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.696 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.696 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.696 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-04-21T09:01:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-04-21T09:01:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.704 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.704 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.704 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.704 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.705 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.705 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.705 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.705 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.706 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.706 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.706 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.708 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.708 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.709 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.709 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.709 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.710 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.710 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.710 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.710 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.711 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.711 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.711 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.712 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.712 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.713 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.713 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.713 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.714 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.714 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.714 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.714 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.715 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.715 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.715 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.715 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.716 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.717 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.717 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.718 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.718 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.719 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.721 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.721 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.722 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.722 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.722 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-04-21T09:01:25.723 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-04-21T09:01:25.723 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.723 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.723 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.724 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.724 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.724 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.725 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.725 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.725 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.726 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.726 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-04-21T09:01:25.726 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-04-21T09:01:25.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.727 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.727 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.727 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.727 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.727 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.728 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.728 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.728 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.728 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.728 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-04-21T09:01:25.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-04-21T09:01:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T09:01:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.729 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-04-21T09:01:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.732 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.733 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.734 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.736 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.738 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-21T09:01:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.740 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.741 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.742 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.743 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.744 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.745 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.745 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.745 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.745 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.745 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.745 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.745 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.745 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.745 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.746 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.747 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-04-21T09:01:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-04-21T09:01:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-04-21T09:01:25.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T09:01:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-04-21T09:01:25.749 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-04-21T09:01:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.750 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-04-21T09:01:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-04-21T09:01:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-04-21T09:01:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.751 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-21T09:01:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T09:01:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.752 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-04-21T09:01:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T09:01:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.753 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T09:01:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.754 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-04-21T09:01:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-04-21T09:01:25.755 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-04-21T09:01:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-21T09:01:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-04-21T09:01:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.756 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-04-21T09:01:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-04-21T09:01:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.758 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-21T09:01:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-21T09:01:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-21T09:01:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.759 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.761 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.762 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.764 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.766 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.767 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.768 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-04-21T09:01:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-04-21T09:01:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.769 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-04-21T09:01:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-04-21T09:01:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.770 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-04-21T09:01:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.771 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-04-21T09:01:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-04-21T09:01:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T09:01:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.773 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-21T09:01:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "modules": [ 2024-04-21T09:01:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-21T09:01:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard", 2024-04-21T09:01:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "iostat", 2024-04-21T09:01:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "nfs", 2024-04-21T09:01:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "restful", 2024-04-21T09:01:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_schedule" 2024-04-21T09:01:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-04-21T09:01:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-04-21T09:01:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-21T09:01:25.775 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-21T09:01:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-21T09:01:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T09:01:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.776 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.778 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.778 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.778 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.778 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.778 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.778 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.778 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.778 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.778 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.779 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.780 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.781 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.782 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.784 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-04-21T09:01:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-04-21T09:01:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.785 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-04-21T09:01:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-04-21T09:01:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-04-21T09:01:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-04-21T09:01:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.787 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-04-21T09:01:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-04-21T09:01:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-04-21T09:01:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-04-21T09:01:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-04-21T09:01:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-04-21T09:01:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.790 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-04-21T09:01:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.791 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-04-21T09:01:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-04-21T09:01:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-04-21T09:01:25.792 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-04-21T09:01:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.793 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-04-21T09:01:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-04-21T09:01:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-04-21T09:01:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-04-21T09:01:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-04-21T09:01:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.795 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.796 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.796 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.796 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T09:01:25.796 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.796 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.797 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-04-21T09:01:25.797 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-04-21T09:01:25.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.797 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.797 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.797 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.797 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-04-21T09:01:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.798 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-04-21T09:01:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-04-21T09:01:25.799 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-04-21T09:01:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T09:01:25.800 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-04-21T09:01:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-04-21T09:01:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.801 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-04-21T09:01:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T09:01:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T09:01:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-04-21T09:01:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-04-21T09:01:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-21T09:01:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-21T09:01:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-21T09:01:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-21T09:01:25.803 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T09:01:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T09:01:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-04-21T09:01:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-04-21T09:01:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-21T09:01:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-04-21T09:01:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-21T09:01:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-04-21T09:01:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T09:01:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.805 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-04-21T09:01:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-04-21T09:01:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.806 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T09:01:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T09:01:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T09:01:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-04-21T09:01:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-04-21T09:01:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-04-21T09:01:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-04-21T09:01:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-04-21T09:01:25.808 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-04-21T09:01:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T09:01:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-04-21T09:01:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-04-21T09:01:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.809 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-04-21T09:01:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T09:01:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T09:01:25.810 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-04-21T09:01:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-04-21T09:01:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-21T09:01:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.811 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-21T09:01:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-21T09:01:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-21T09:01:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T09:01:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T09:01:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.812 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.813 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.814 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.815 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.816 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.817 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.818 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.820 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-04-21T09:01:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-04-21T09:01:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.821 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-21T09:01:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T09:01:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-21T09:01:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-21T09:01:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-04-21T09:01:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.823 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-04-21T09:01:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-04-21T09:01:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-04-21T09:01:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-04-21T09:01:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-04-21T09:01:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-04-21T09:01:25.824 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-04-21T09:01:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-04-21T09:01:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.825 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T09:01:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-21T09:01:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-21T09:01:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-21T09:01:25.827 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T09:01:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.828 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-04-21T09:01:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-04-21T09:01:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-21T09:01:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-21T09:01:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.829 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T09:01:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T09:01:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-04-21T09:01:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-04-21T09:01:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-21T09:01:25.830 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T09:01:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.831 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-04-21T09:01:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.832 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-04-21T09:01:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-04-21T09:01:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-21T09:01:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.833 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T09:01:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-04-21T09:01:25.834 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-04-21T09:01:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-04-21T09:01:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T09:01:25.835 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-04-21T09:01:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-04-21T09:01:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-04-21T09:01:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T09:01:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-04-21T09:01:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-04-21T09:01:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.838 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T09:01:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T09:01:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-04-21T09:01:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-04-21T09:01:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.839 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-21T09:01:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T09:01:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.840 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-04-21T09:01:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T09:01:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-04-21T09:01:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.841 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T09:01:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-04-21T09:01:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-04-21T09:01:25.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-04-21T09:01:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-04-21T09:01:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-04-21T09:01:25.843 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T09:01:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-04-21T09:01:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-04-21T09:01:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.844 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T09:01:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-04-21T09:01:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-04-21T09:01:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.846 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T09:01:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-04-21T09:01:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-04-21T09:01:25.847 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T09:01:25.848 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-04-21T09:01:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-04-21T09:01:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.849 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T09:01:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-21T09:01:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-04-21T09:01:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-04-21T09:01:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T09:01:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.851 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-04-21T09:01:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-04-21T09:01:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-04-21T09:01:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.852 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T09:01:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-04-21T09:01:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.853 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-04-21T09:01:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-04-21T09:01:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T09:01:25.854 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-21T09:01:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.855 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-04-21T09:01:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-04-21T09:01:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T09:01:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.856 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-04-21T09:01:25.857 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.857 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.857 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.857 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-04-21T09:01:25.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-04-21T09:01:25.857 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.858 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.858 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.858 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T09:01:25.858 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.858 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.858 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.858 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-04-21T09:01:25.858 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.858 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.859 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.859 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-04-21T09:01:25.859 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-04-21T09:01:25.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.859 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.859 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T09:01:25.860 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.860 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.860 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.860 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-04-21T09:01:25.860 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.860 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.860 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.861 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-04-21T09:01:25.861 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-04-21T09:01:25.861 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.861 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.861 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.861 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T09:01:25.861 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.861 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.861 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.862 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-04-21T09:01:25.862 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.862 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.862 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.862 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-04-21T09:01:25.862 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-04-21T09:01:25.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.863 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.863 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T09:01:25.863 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.863 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.863 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.863 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-04-21T09:01:25.864 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.864 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.864 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.864 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-04-21T09:01:25.864 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-04-21T09:01:25.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.864 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.864 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.865 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T09:01:25.865 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.865 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.865 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.865 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-04-21T09:01:25.866 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.866 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.866 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.866 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-04-21T09:01:25.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-04-21T09:01:25.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.866 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.867 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.867 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T09:01:25.867 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.867 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.867 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.867 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-21T09:01:25.867 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.867 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.868 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-04-21T09:01:25.868 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-04-21T09:01:25.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.868 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.868 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T09:01:25.868 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.869 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.869 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.869 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-04-21T09:01:25.869 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.869 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.869 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.869 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-04-21T09:01:25.870 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-04-21T09:01:25.870 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.870 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.870 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T09:01:25.870 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.870 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.870 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.871 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-21T09:01:25.871 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.871 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.871 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.871 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-04-21T09:01:25.871 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-04-21T09:01:25.871 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.872 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.872 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T09:01:25.872 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.872 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.872 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.872 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-04-21T09:01:25.872 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.872 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.873 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.873 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-04-21T09:01:25.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-04-21T09:01:25.873 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.873 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T09:01:25.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.874 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.874 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.874 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-04-21T09:01:25.874 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.874 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.874 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.874 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-04-21T09:01:25.875 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-04-21T09:01:25.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.875 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.875 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.875 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T09:01:25.875 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.875 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.875 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.875 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-04-21T09:01:25.876 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.876 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.876 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.876 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-04-21T09:01:25.876 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-04-21T09:01:25.876 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.876 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.877 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.877 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.877 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.877 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T09:01:25.877 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.877 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.878 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.878 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-04-21T09:01:25.878 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-04-21T09:01:25.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.878 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.878 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.878 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-21T09:01:25.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.879 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.879 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.879 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T09:01:25.879 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.879 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.879 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.879 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-04-21T09:01:25.880 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T09:01:25.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.880 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.880 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.880 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-04-21T09:01:25.880 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.880 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.880 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.880 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T09:01:25.881 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.881 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.881 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.881 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-04-21T09:01:25.881 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-04-21T09:01:25.881 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.881 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.881 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.882 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-04-21T09:01:25.882 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.882 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.882 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.882 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T09:01:25.882 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.882 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.882 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.883 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-04-21T09:01:25.883 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-04-21T09:01:25.883 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.883 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.883 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-04-21T09:01:25.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.884 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.884 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-04-21T09:01:25.884 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.884 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.884 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.884 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-04-21T09:01:25.884 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-04-21T09:01:25.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.885 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.885 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.885 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.885 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.885 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T09:01:25.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.886 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.886 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.886 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-04-21T09:01:25.886 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-04-21T09:01:25.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.886 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.887 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-21T09:01:25.887 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.887 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.887 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.887 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-04-21T09:01:25.887 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.887 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.887 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.888 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-04-21T09:01:25.888 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-04-21T09:01:25.888 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.888 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.888 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.888 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T09:01:25.888 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T09:01:25.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.889 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.889 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-04-21T09:01:25.890 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-04-21T09:01:25.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.890 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.890 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-21T09:01:25.890 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.890 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.891 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.891 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-04-21T09:01:25.891 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.891 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.891 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.891 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-04-21T09:01:25.891 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-04-21T09:01:25.891 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.892 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.892 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.892 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.892 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.892 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T09:01:25.893 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.893 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.893 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.893 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-04-21T09:01:25.893 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-04-21T09:01:25.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.893 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.894 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.894 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.894 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.894 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.894 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T09:01:25.894 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.895 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.895 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.895 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.895 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.896 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.896 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.896 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.896 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.896 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.896 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.896 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.897 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.897 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.897 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.897 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.897 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.897 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.897 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-04-21T09:01:25.897 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-04-21T09:01:25.898 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.898 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.898 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.898 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.898 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.898 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.898 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.898 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T09:01:25.899 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.899 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.899 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.899 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.899 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.899 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.899 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.900 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.900 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.900 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.900 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.900 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.900 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T09:01:25.900 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.900 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.900 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.901 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.901 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.901 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.901 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.901 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.901 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.902 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.902 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.902 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.902 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.902 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.902 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.902 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.902 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.903 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.903 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.903 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.903 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.903 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.903 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.903 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.903 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.904 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.904 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.904 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.904 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.904 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.904 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.904 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.904 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.905 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.905 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.905 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T09:01:25.905 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T09:01:25.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.905 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.905 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.906 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.906 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.906 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.906 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.906 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T09:01:25.906 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.906 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.907 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.907 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T09:01:25.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T09:01:25.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.907 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.907 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.907 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-04-21T09:01:25.907 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.908 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.908 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.908 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T09:01:25.908 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.908 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.908 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.908 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-04-21T09:01:25.909 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-04-21T09:01:25.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.909 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.909 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:25.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.909 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.909 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.909 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-04-21T09:01:25.910 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.910 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.910 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.910 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-04-21T09:01:25.910 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-04-21T09:01:25.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.911 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.911 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.911 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:25.911 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.911 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.911 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.911 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T09:01:25.911 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.912 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.912 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-04-21T09:01:25.912 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-04-21T09:01:25.912 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.912 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.912 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.912 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.912 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.913 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.913 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.913 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-04-21T09:01:25.913 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.913 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.913 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.913 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-21T09:01:25.914 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-21T09:01:25.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.914 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.914 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-04-21T09:01:25.914 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.914 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.914 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.915 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-04-21T09:01:25.915 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-04-21T09:01:25.915 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.915 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T09:01:25.915 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.915 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.915 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.915 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-04-21T09:01:25.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-04-21T09:01:25.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.916 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.916 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.916 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-04-21T09:01:25.916 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.916 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.917 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.917 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T09:01:25.917 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.917 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.917 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.917 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-04-21T09:01:25.917 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-04-21T09:01:25.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.918 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.918 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.918 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T09:01:25.918 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.918 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.919 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.919 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T09:01:25.919 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.919 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.919 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.919 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-04-21T09:01:25.919 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-04-21T09:01:25.920 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.920 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.920 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.920 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.920 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.920 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.920 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.920 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T09:01:25.920 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.921 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.921 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.921 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-04-21T09:01:25.921 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-04-21T09:01:25.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.921 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.921 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.922 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.922 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.922 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.922 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.922 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T09:01:25.922 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.922 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.922 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.923 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-04-21T09:01:25.923 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-04-21T09:01:25.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.923 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.923 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.923 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.923 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.925 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.925 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.925 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T09:01:25.925 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.926 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.926 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-04-21T09:01:25.926 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-04-21T09:01:25.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.926 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.926 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.926 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.927 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.927 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.927 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.927 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T09:01:25.927 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.927 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.928 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-04-21T09:01:25.928 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-04-21T09:01:25.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.928 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.928 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.928 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.929 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.929 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.929 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T09:01:25.929 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.929 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.929 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.929 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-04-21T09:01:25.930 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-04-21T09:01:25.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.930 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.930 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.930 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-04-21T09:01:25.930 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.930 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.930 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.930 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-04-21T09:01:25.931 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.931 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.931 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-04-21T09:01:25.931 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-04-21T09:01:25.931 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.931 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.932 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.932 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.932 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.932 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T09:01:25.932 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.932 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.932 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.933 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-04-21T09:01:25.933 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-04-21T09:01:25.933 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.933 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.933 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.933 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.933 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.934 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.934 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.934 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T09:01:25.934 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.934 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.934 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.935 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-04-21T09:01:25.935 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-04-21T09:01:25.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.935 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.935 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.935 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.935 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.936 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.936 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T09:01:25.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.936 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.936 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-04-21T09:01:25.936 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-04-21T09:01:25.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.937 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.937 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.937 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.937 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T09:01:25.938 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.938 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.938 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.938 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-04-21T09:01:25.938 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-04-21T09:01:25.938 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.938 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.939 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.939 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.939 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.939 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T09:01:25.939 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.939 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.939 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.940 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-04-21T09:01:25.940 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-04-21T09:01:25.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.940 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.940 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.940 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.940 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.940 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.941 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T09:01:25.941 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.941 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.941 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.941 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.941 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.942 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-04-21T09:01:25.942 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.942 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.942 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.942 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:25.942 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:25.942 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.942 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.943 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.943 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.943 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.943 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.943 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.943 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.944 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.944 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.944 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.944 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.944 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.944 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.944 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.944 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.945 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:25.945 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:25.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.945 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.945 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.945 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.945 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.945 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.945 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.946 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.946 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:25.946 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:25.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.946 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.947 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:25.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.947 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.947 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:25.947 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:25.947 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:25.947 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:25.948 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:25.948 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:25.948 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:25.948 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:25.948 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.948 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.948 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.949 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:25.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:25.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.949 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.949 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.949 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.949 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.949 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.950 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.950 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.950 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.950 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.950 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.950 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-04-21T09:01:25.950 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-04-21T09:01:25.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.951 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.951 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-04-21T09:01:25.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.951 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.951 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.951 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T09:01:25.952 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.952 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.952 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-04-21T09:01:25.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-04-21T09:01:25.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:25.952 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.953 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:25.953 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-04-21T09:01:25.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.953 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T09:01:25.953 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.953 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.954 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:25.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:25.954 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-04-21T09:01:25.954 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:25.954 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:25.955 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:25.955 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T09:01:25.955 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T09:01:25.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:25.955 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.955 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:25.955 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.956 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.956 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.956 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.956 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.956 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.956 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.956 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T09:01:25.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T09:01:25.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.957 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.957 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.957 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.957 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.958 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.958 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.958 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.958 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.958 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T09:01:25.958 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T09:01:25.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.959 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.959 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.959 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.959 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.959 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.959 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.959 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.959 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.960 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.960 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-04-21T09:01:25.960 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T09:01:25.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.960 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.960 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.961 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.961 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.961 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.961 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.961 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.961 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.962 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T09:01:25.962 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T09:01:25.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.962 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.962 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.962 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.962 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.963 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.963 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.963 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.963 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.963 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.963 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T09:01:25.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T09:01:25.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.964 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.964 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.964 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.964 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.964 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.964 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.965 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.965 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.965 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.965 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T09:01:25.965 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T09:01:25.965 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.965 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.965 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.966 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.966 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.966 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.966 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.966 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.966 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.967 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T09:01:25.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T09:01:25.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.967 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.967 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.967 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.967 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.967 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.968 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.968 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.968 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T09:01:25.968 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T09:01:25.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.968 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.969 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.969 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.969 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.969 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.969 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.969 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.970 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T09:01:25.970 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T09:01:25.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.970 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.970 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.970 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.971 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.971 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.971 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.971 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.971 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.971 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T09:01:25.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T09:01:25.972 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.972 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.972 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.972 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.972 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.972 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.973 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.973 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.973 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.973 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.973 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T09:01:25.973 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T09:01:25.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.974 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.974 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.974 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.974 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.974 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.974 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.974 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.974 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.975 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T09:01:25.975 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T09:01:25.975 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.975 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.975 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.975 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.975 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.976 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.976 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.976 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.976 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.976 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T09:01:25.976 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T09:01:25.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.977 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.977 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.977 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.977 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.977 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.977 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.977 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.978 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.978 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T09:01:25.978 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T09:01:25.978 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.978 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.978 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.978 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-21T09:01:25.978 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.979 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.979 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.979 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.979 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.979 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.979 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T09:01:25.979 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T09:01:25.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.980 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.980 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.980 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.980 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.980 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.980 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.980 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.981 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.981 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-04-21T09:01:25.981 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-04-21T09:01:25.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.981 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.981 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.982 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.982 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.982 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.982 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.982 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.982 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.982 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.982 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T09:01:25.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T09:01:25.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.983 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.983 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.983 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-21T09:01:25.983 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.983 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.983 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.984 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.984 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.984 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.984 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T09:01:25.984 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T09:01:25.984 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.984 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.985 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.985 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.985 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.985 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.985 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.985 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.985 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.986 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T09:01:25.986 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T09:01:25.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.986 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.986 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.986 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.986 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.986 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.987 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.987 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.987 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.987 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T09:01:25.987 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T09:01:25.987 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.987 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.988 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.988 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.988 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.988 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.988 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.988 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.988 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.989 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.989 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T09:01:25.989 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T09:01:25.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.989 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.989 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.990 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.990 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.990 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.990 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.990 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.990 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-21T09:01:25.991 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-21T09:01:25.991 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.991 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.991 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.991 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-04-21T09:01:25.991 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.991 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.991 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.991 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.992 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.992 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.992 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.992 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-04-21T09:01:25.992 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-21T09:01:25.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.992 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.993 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-04-21T09:01:25.993 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.993 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.993 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.994 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.994 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T09:01:25.994 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T09:01:25.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:25.994 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.994 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:25.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.995 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.995 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.995 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.995 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.995 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.995 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.995 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T09:01:25.995 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T09:01:25.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.996 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.996 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:25.996 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.996 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.996 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.996 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.996 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.997 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.997 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.997 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T09:01:25.997 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T09:01:25.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.997 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.997 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.998 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.998 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.998 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.998 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:25.998 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:25.998 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:25.998 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:25.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:25.998 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T09:01:25.999 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T09:01:25.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:25.999 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:25.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:25.999 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:25.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:25.999 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:25.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:25.999 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.000 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.000 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.000 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T09:01:26.000 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T09:01:26.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.000 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.001 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.001 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.001 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.001 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.001 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.001 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.002 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T09:01:26.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T09:01:26.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.002 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.002 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.002 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.003 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.003 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.003 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.003 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.003 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.003 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.003 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T09:01:26.003 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T09:01:26.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.004 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.004 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.004 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.004 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.004 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.004 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.004 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.005 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.005 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T09:01:26.005 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T09:01:26.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.005 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.005 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.005 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.006 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.006 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.006 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.006 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.006 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.006 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.006 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T09:01:26.006 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T09:01:26.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.007 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.007 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.007 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.007 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.008 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.008 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.008 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T09:01:26.008 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T09:01:26.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.008 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.009 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.009 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.009 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.009 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.009 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.009 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.010 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T09:01:26.010 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T09:01:26.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.010 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T09:01:26.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.010 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.010 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.011 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.011 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.011 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.011 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T09:01:26.011 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T09:01:26.011 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.011 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.012 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.012 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:26.012 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.012 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.012 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.012 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.012 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.012 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.012 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.013 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T09:01:26.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T09:01:26.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.013 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-04-21T09:01:26.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.013 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T09:01:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T09:01:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-04-21T09:01:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T09:01:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T09:01:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T09:01:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T09:01:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-21T09:01:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T09:01:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T09:01:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T09:01:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T09:01:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-04-21T09:01:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-21T09:01:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T09:01:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T09:01:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-21T09:01:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T09:01:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T09:01:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.028 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T09:01:26.028 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T09:01:26.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.028 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.028 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-21T09:01:26.028 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.028 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.028 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.029 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.029 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.029 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.029 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.029 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-04-21T09:01:26.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-04-21T09:01:26.030 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.030 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.030 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.030 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.030 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.030 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.031 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.031 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.031 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.031 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-04-21T09:01:26.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-04-21T09:01:26.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.031 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.031 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.032 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.032 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.032 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.032 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.032 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.032 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-04-21T09:01:26.033 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-04-21T09:01:26.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.033 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.033 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.033 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.034 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-04-21T09:01:26.034 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.034 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.034 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.034 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-04-21T09:01:26.034 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-04-21T09:01:26.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.035 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-04-21T09:01:26.035 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.035 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.035 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.036 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.036 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.036 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-21T09:01:26.036 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-21T09:01:26.036 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.036 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.036 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.036 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.037 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.037 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.037 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.037 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.037 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.037 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.037 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.038 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.038 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.038 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.038 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.038 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.038 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.039 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.039 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.039 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.039 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.039 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.039 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.039 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.039 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.039 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.040 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.040 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.040 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.040 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.040 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.041 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.041 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.041 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.041 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.041 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.041 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.041 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.041 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.042 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.042 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.042 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.042 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.042 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.042 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.042 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.042 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.043 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.043 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.043 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.043 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.043 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.043 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.043 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.043 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.043 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.044 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.044 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.044 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.044 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.044 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.045 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.045 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.045 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.045 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.045 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.046 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-04-21T09:01:26.046 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-04-21T09:01:26.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.046 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.046 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.047 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.047 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-04-21T09:01:26.047 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.047 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.047 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-04-21T09:01:26.048 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T09:01:26.048 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.048 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.048 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.048 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.048 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.048 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.048 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.049 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.049 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.049 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.049 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.049 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-21T09:01:26.049 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-21T09:01:26.049 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.049 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.050 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-21T09:01:26.050 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.050 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.050 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.050 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.050 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.050 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.051 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.051 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-21T09:01:26.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-21T09:01:26.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.051 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.051 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.051 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-04-21T09:01:26.051 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.051 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.052 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.052 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.052 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.052 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.052 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.052 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-21T09:01:26.052 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-21T09:01:26.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.053 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.053 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.053 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.053 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.053 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.053 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.053 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.053 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.053 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.053 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.054 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-04-21T09:01:26.054 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-04-21T09:01:26.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.054 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.054 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.054 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-04-21T09:01:26.054 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.054 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.055 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.055 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.055 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.055 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-21T09:01:26.055 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-21T09:01:26.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.056 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-04-21T09:01:26.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.056 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.056 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.056 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-04-21T09:01:26.057 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.057 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.057 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.057 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.057 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.057 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-21T09:01:26.057 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-21T09:01:26.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.058 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.058 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.058 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-21T09:01:26.058 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-21T09:01:26.058 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-21T09:01:26.058 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.059 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.059 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.059 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.059 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.059 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-04-21T09:01:26.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-04-21T09:01:26.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.060 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.060 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.060 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.060 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.060 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.061 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.061 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.061 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.061 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.061 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-21T09:01:26.061 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.061 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.062 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.062 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-04-21T09:01:26.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-04-21T09:01:26.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.062 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.062 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.062 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.063 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-04-21T09:01:26.063 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.063 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.063 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.063 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.063 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.064 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.064 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.064 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.064 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.064 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.064 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.065 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.065 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.065 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.065 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.065 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.065 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.065 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.065 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.065 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.066 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.066 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.066 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.066 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.066 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.066 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.067 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.067 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.067 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.067 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.067 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.068 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.068 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.068 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.068 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.068 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.068 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.069 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.069 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.069 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.069 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.069 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.069 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.069 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.070 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.070 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.070 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.070 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.070 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.070 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.070 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.071 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.071 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.071 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.071 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.071 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.072 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-04-21T09:01:26.072 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-04-21T09:01:26.072 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:26.073 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-04-21T09:01:26.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.074 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.074 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.074 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T09:01:26.074 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.075 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.075 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-04-21T09:01:26.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-04-21T09:01:26.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.076 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.077 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-04-21T09:01:26.077 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.077 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.077 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.078 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T09:01:26.078 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.078 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.079 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-04-21T09:01:26.079 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-04-21T09:01:26.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:26.080 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-04-21T09:01:26.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.081 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.081 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.081 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-04-21T09:01:26.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.082 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.083 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-04-21T09:01:26.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-04-21T09:01:26.083 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:26.083 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.084 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-21T09:01:26.084 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.084 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.085 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.085 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T09:01:26.085 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.085 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.085 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.085 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-04-21T09:01:26.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-04-21T09:01:26.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.086 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.086 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.086 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.086 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.086 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T09:01:26.086 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.086 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.087 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.087 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-21T09:01:26.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-21T09:01:26.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:26.087 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.087 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-21T09:01:26.088 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.088 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.088 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T09:01:26.088 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.088 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.088 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.088 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-04-21T09:01:26.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-04-21T09:01:26.089 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:26.089 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.089 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.089 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-04-21T09:01:26.089 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.089 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.089 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.089 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T09:01:26.090 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.090 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.090 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.090 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-04-21T09:01:26.091 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.091 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.091 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.091 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.091 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.092 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.092 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.092 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.092 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.092 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.092 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.092 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.092 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.093 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.093 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.093 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.093 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.093 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.093 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.094 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.094 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.094 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.094 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.094 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.095 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.095 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.095 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.096 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.096 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.096 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.096 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.096 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.096 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.096 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.097 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.097 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.097 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.097 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.097 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.097 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.097 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.098 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.098 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.098 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.098 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.098 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.098 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.098 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.098 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.099 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.099 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.099 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.099 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.099 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.099 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-04-21T09:01:26.099 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-04-21T09:01:26.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.100 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.100 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.100 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-21T09:01:26.100 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.100 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.100 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.100 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.100 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.101 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.101 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.101 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-04-21T09:01:26.101 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-04-21T09:01:26.101 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.101 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.101 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-04-21T09:01:26.102 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.102 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.102 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.102 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.102 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.102 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.103 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-21T09:01:26.103 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-21T09:01:26.103 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.103 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.103 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.103 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-21T09:01:26.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.103 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.104 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.104 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.104 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.104 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.104 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.104 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-04-21T09:01:26.105 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-04-21T09:01:26.105 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-04-21T09:01:26.105 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.105 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-04-21T09:01:26.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-04-21T09:01:26.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.105 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.105 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-04-21T09:01:26.106 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.106 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.106 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.106 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T09:01:26.106 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.106 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.106 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.107 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-04-21T09:01:26.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-04-21T09:01:26.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.107 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.107 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-04-21T09:01:26.107 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.107 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.107 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.108 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T09:01:26.108 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.108 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.108 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-04-21T09:01:26.108 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-04-21T09:01:26.108 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.109 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.109 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.109 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.109 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.109 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.109 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-04-21T09:01:26.109 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.109 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.110 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.110 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-21T09:01:26.110 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-21T09:01:26.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:26.110 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.110 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.110 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-04-21T09:01:26.111 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-04-21T09:01:26.111 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.111 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.111 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T09:01:26.111 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.112 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.112 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.112 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.112 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.112 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.112 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.113 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.113 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.113 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.113 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.113 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.113 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.113 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.113 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.113 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.114 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.114 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.114 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.114 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.114 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.115 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.115 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.115 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.115 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.115 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.115 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.115 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.116 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.116 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.116 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.116 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.116 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.117 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.117 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.117 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.117 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.117 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.117 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.117 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.118 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.118 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.118 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.118 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.118 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.118 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.119 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.119 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.119 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.119 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.119 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.120 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.120 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.120 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-04-21T09:01:26.120 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-04-21T09:01:26.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.120 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.121 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.121 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.121 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.121 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.121 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-04-21T09:01:26.121 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.121 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.122 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.122 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-04-21T09:01:26.122 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-04-21T09:01:26.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.122 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.122 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.122 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-04-21T09:01:26.123 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.123 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.123 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.123 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-04-21T09:01:26.123 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.123 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.124 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-21T09:01:26.124 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-21T09:01:26.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.124 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.124 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-04-21T09:01:26.124 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.124 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.125 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T09:01:26.125 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.125 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.125 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-04-21T09:01:26.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-04-21T09:01:26.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.126 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.126 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-21T09:01:26.126 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-21T09:01:26.126 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-04-21T09:01:26.126 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.126 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T09:01:26.126 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.127 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-04-21T09:01:26.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-04-21T09:01:26.127 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.127 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.127 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-04-21T09:01:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.128 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-04-21T09:01:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-04-21T09:01:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-04-21T09:01:26.129 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.130 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.130 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.130 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T09:01:26.130 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.130 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.130 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-04-21T09:01:26.131 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.131 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.131 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.131 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.132 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.132 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.132 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.132 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.132 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.132 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.132 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.132 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.132 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.133 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.133 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.133 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.133 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.133 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.133 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.133 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.134 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.134 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.134 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.134 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.134 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.135 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.135 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.135 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.135 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.135 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.135 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.135 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.136 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.136 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.136 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.136 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.137 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.137 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.137 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.137 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.137 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.137 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.137 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.137 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.138 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.138 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.138 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.138 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.138 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.139 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.139 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.139 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.139 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.139 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.139 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.139 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.140 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.140 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.140 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.140 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-04-21T09:01:26.140 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.140 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.141 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.141 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.141 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.141 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.141 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.141 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.141 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.141 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.142 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.142 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.142 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.142 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.142 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.142 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.142 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.142 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.142 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.143 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.143 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.143 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.143 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.143 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.144 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.144 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.144 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.144 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.144 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.144 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.144 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.144 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.145 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.145 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.145 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.145 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.145 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.146 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.146 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.146 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.146 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.146 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.146 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.146 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.147 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.147 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.147 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.147 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.147 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.147 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.148 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.148 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.148 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.148 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.148 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.148 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.148 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.149 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.149 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-04-21T09:01:26.149 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.149 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.149 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.150 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-04-21T09:01:26.150 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-04-21T09:01:26.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.150 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.151 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-04-21T09:01:26.151 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.151 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.151 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.151 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T09:01:26.151 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.151 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.152 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-04-21T09:01:26.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-04-21T09:01:26.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.152 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.152 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:26.152 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-04-21T09:01:26.152 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.153 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T09:01:26.153 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.153 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.153 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.153 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.153 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.154 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.154 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.154 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.154 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.154 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.155 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.155 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.155 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.155 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.155 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.155 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.156 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.156 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.156 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.156 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.156 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.156 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.157 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.157 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.157 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.157 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.157 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.158 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.158 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.158 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.158 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.158 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.158 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.159 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.159 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.159 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.159 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.159 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.160 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.160 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.160 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.160 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.160 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.160 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.160 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.160 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.161 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.161 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.161 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.161 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.162 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.162 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.162 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.162 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.162 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.162 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.163 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.163 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.163 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.163 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.163 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-04-21T09:01:26.164 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.164 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.164 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.164 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-04-21T09:01:26.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-04-21T09:01:26.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.165 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-04-21T09:01:26.165 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.165 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.165 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.165 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-04-21T09:01:26.166 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T09:01:26.166 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.166 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.166 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.167 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.167 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.167 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.167 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.168 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.168 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.168 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.168 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.168 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.169 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.169 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.169 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.169 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.169 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.170 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.170 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.170 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.170 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.171 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.171 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.171 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.172 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.172 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.172 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.173 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.173 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.173 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.174 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.175 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.175 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.175 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.175 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.175 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.176 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.176 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.176 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.176 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.176 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.177 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.177 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.177 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.177 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.177 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.177 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.178 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.178 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.178 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.179 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.179 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.179 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.179 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.179 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.180 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.180 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.180 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.180 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-04-21T09:01:26.181 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-04-21T09:01:26.181 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.181 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.181 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.181 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.181 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.181 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.181 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.181 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-04-21T09:01:26.182 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T09:01:26.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.182 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.182 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-04-21T09:01:26.182 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-04-21T09:01:26.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.182 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.183 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-04-21T09:01:26.183 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.183 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.183 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-04-21T09:01:26.183 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.183 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.184 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.184 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-04-21T09:01:26.184 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-04-21T09:01:26.184 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.184 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.184 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.184 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-04-21T09:01:26.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.185 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.185 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.185 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T09:01:26.185 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.185 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.186 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-04-21T09:01:26.186 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-04-21T09:01:26.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.186 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.186 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.186 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.186 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.186 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.187 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.187 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-04-21T09:01:26.187 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.187 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.187 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.187 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-04-21T09:01:26.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-04-21T09:01:26.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.188 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.188 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-04-21T09:01:26.188 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.188 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.188 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.188 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T09:01:26.188 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T09:01:26.188 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.189 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.189 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.189 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-04-21T09:01:26.189 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.189 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.190 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.190 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.190 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.190 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.190 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.191 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.191 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.191 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.191 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.191 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.191 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.191 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.192 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.192 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.192 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.192 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.192 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.192 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.192 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.193 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.193 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.193 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.193 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.193 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.193 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.194 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.194 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.194 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.194 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.194 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.194 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.195 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.195 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.195 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.195 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.195 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.195 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.195 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.196 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.196 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.196 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.196 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.196 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.196 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.196 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.196 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.197 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.197 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.197 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.198 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.198 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.198 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.198 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.198 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.198 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.198 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.199 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.199 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-04-21T09:01:26.199 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.199 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.199 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.199 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.199 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.200 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.200 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.200 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.200 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.200 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.201 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.201 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.201 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.201 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.201 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.201 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.201 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.201 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.201 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.202 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.202 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.202 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.202 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.202 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.202 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.203 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.203 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.203 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.204 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.204 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.204 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.204 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.204 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.204 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.204 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.205 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.205 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.205 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.205 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.205 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.205 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.205 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.205 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.205 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.206 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.206 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.206 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.207 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.207 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.207 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.207 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.207 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.208 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-04-21T09:01:26.208 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.208 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.208 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.208 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.209 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.209 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.209 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.209 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.209 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.210 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.210 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.210 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.210 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.210 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.210 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.210 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.210 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.211 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.211 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.211 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.211 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.211 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.211 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.212 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.212 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.212 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.212 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.212 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.212 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.213 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.213 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.213 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.213 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.213 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.214 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.214 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.214 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.214 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.214 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.214 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.215 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.215 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.215 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.215 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.215 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.215 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.215 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.216 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.216 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.216 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.216 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.216 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.217 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.217 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.217 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.217 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.217 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.217 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-21T09:01:26.217 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.218 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.218 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.218 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-04-21T09:01:26.218 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-04-21T09:01:26.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.218 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.219 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.219 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.219 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T09:01:26.219 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.219 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.220 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.220 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.220 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.220 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.220 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.220 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.221 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.221 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.221 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.221 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.221 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.221 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.221 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.222 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.222 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.222 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.222 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.222 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.223 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.223 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.223 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.223 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.223 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.224 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.224 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.224 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.224 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.225 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.225 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.225 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.225 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.225 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.225 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.225 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.226 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.226 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.226 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.226 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.226 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.226 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.226 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.226 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.226 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.227 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.227 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.227 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.228 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.228 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.228 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.228 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.228 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.228 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.228 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-04-21T09:01:26.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-21T09:01:26.229 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.229 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.229 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.229 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.229 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.229 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.229 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.229 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-21T09:01:26.229 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-04-21T09:01:26.230 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-04-21T09:01:26.230 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.230 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-04-21T09:01:26.230 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.230 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.230 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.230 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.231 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-04-21T09:01:26.231 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.231 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.231 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.231 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.232 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.232 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.232 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.232 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.232 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.232 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.233 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.233 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.233 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.233 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.233 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.233 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.233 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.233 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.234 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.234 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.234 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.234 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.235 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.235 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.235 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.235 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.235 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.236 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.236 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.236 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.236 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.236 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.237 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.237 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.237 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.237 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.237 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.237 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.237 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.237 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.238 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.238 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.238 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.238 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.238 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.238 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.238 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.238 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.239 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.239 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.239 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.239 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.239 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.239 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.239 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.240 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.240 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.240 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.240 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.240 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.240 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.241 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-04-21T09:01:26.241 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.241 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.241 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.241 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.242 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.242 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.242 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.242 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.242 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.243 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.243 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.243 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.243 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.243 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.243 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.243 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.243 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.243 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.244 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.244 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.244 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.244 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.244 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.245 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.245 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.245 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.245 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.245 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.245 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.246 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.246 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.246 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.246 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.246 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.247 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.247 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.247 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.247 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.247 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.247 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.248 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.248 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.248 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.248 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.248 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.248 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.248 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.249 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.249 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.249 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.249 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.249 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.249 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.250 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.250 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.250 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.250 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.250 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.250 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-04-21T09:01:26.251 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.251 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.251 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.251 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.251 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.251 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.251 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.251 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.252 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.252 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.252 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.252 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.252 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.252 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.252 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.252 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.252 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.253 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.253 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.253 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.253 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.253 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.253 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.253 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.254 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.254 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.254 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.254 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.254 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.254 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.255 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.255 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.255 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.255 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.255 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.255 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.256 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.256 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.256 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.256 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.256 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.256 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.256 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.256 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.257 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.257 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.257 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.257 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.257 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.257 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.257 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.257 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.258 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.258 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.258 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.258 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.259 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.259 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.259 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.259 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.259 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.259 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.259 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.260 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-21T09:01:26.260 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-21T09:01:26.260 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:26.260 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.260 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.260 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-21T09:01:26.260 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.260 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.260 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.261 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.261 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.261 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.261 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-04-21T09:01:26.261 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-04-21T09:01:26.262 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:26.262 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.262 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.262 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-21T09:01:26.262 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-04-21T09:01:26.262 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.262 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.263 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-04-21T09:01:26.263 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T09:01:26.263 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.263 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.263 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.263 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-04-21T09:01:26.264 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.264 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.264 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.264 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-04-21T09:01:26.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-04-21T09:01:26.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.265 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.265 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.265 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.265 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.265 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.265 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.265 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T09:01:26.265 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.266 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.266 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.266 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-21T09:01:26.266 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-21T09:01:26.266 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.266 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.267 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.267 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.267 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.267 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.267 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.268 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.268 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.268 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.268 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.268 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.268 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.269 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.269 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.269 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.269 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.269 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.269 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.269 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.269 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.270 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.270 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.270 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.270 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.270 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.270 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.270 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.271 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.271 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.271 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.271 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.271 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.271 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.271 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.271 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.273 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.273 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.273 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.273 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.273 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.273 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.273 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.273 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.274 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.274 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.274 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.274 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.274 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.274 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.274 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.275 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.275 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.275 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.275 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.275 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.275 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.275 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.275 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.275 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.276 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.276 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.276 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.276 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-04-21T09:01:26.276 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-04-21T09:01:26.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.276 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.276 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.277 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-04-21T09:01:26.277 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.277 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.277 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.277 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-04-21T09:01:26.277 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.277 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.277 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.278 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-21T09:01:26.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-21T09:01:26.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:26.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.278 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-21T09:01:26.278 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.278 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.279 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.279 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-04-21T09:01:26.279 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.279 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.279 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.279 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.279 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.279 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.280 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-04-21T09:01:26.280 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.280 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.280 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.280 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-04-21T09:01:26.280 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-04-21T09:01:26.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.280 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.280 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.281 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.281 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.281 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.282 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-04-21T09:01:26.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-04-21T09:01:26.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.282 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.282 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.282 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.282 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.282 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.283 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T09:01:26.283 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T09:01:26.283 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.283 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.283 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.283 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.283 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.284 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.284 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.284 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.284 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.284 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.284 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.284 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.284 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.285 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.285 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.285 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.285 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.285 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.285 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.285 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.285 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.285 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.286 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.286 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.286 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.286 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.286 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.286 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.286 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.286 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.287 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.287 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.287 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.287 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.287 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.287 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.288 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.288 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.288 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.288 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.288 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.288 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.288 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.288 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.288 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.289 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.289 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.289 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.289 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.289 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.289 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.289 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.289 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.289 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.290 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.290 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.290 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.290 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.290 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.291 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.291 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.291 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.291 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.291 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.291 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-04-21T09:01:26.292 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-04-21T09:01:26.292 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.292 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.292 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.292 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.292 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.292 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.293 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.293 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.293 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.293 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.293 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T09:01:26.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T09:01:26.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.294 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-04-21T09:01:26.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.294 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.294 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.294 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.294 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.295 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.295 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.295 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-04-21T09:01:26.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-04-21T09:01:26.295 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:26.295 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.295 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.296 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-04-21T09:01:26.296 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.296 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.296 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.296 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.296 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.297 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-21T09:01:26.297 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-21T09:01:26.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.297 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.297 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.297 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-21T09:01:26.297 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.297 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.298 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.298 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T09:01:26.298 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.298 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.298 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.298 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-21T09:01:26.298 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-21T09:01:26.299 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.299 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.299 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.299 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-04-21T09:01:26.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.299 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.299 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.299 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T09:01:26.300 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.300 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.300 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.300 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-04-21T09:01:26.300 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-04-21T09:01:26.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.300 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.300 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.301 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-04-21T09:01:26.301 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.301 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.301 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.301 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.301 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.301 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.301 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.302 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-21T09:01:26.302 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-21T09:01:26.302 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.302 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.302 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.302 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-04-21T09:01:26.302 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.303 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.303 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.303 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-04-21T09:01:26.303 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-04-21T09:01:26.303 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.303 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.303 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.303 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.303 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.304 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-21T09:01:26.304 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-21T09:01:26.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.304 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.304 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.305 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-21T09:01:26.305 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-21T09:01:26.305 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-21T09:01:26.305 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.305 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.305 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.305 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.305 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.306 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.306 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-21T09:01:26.306 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.306 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.306 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.306 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.307 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.307 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.307 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.307 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.307 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.307 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.307 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.308 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.308 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.308 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.308 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.308 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.308 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.308 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.308 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.308 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.309 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.309 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.309 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.309 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.309 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.309 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.310 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.310 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.310 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.310 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.310 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.310 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.311 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.311 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.311 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.311 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.311 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.311 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.312 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.312 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.312 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.312 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.312 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.312 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.312 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.313 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.313 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.313 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.313 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.313 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.313 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.314 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.314 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.314 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.314 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.314 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.314 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.314 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.314 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.314 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.315 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.315 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.315 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-04-21T09:01:26.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-04-21T09:01:26.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.315 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.316 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-21T09:01:26.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.316 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.316 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.316 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.316 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.316 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.317 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-04-21T09:01:26.317 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T09:01:26.317 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.317 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.318 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.318 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.318 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.318 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.319 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.319 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.319 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.320 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-04-21T09:01:26.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-04-21T09:01:26.320 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.320 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.321 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.322 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.322 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.322 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.322 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.323 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.323 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.323 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-04-21T09:01:26.324 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.324 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.324 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.324 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-04-21T09:01:26.324 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-04-21T09:01:26.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.325 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.325 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.325 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.325 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.325 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.325 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.326 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.326 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.326 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.326 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-21T09:01:26.326 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-21T09:01:26.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.327 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.327 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.327 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.327 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.328 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.328 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.328 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.328 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.328 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.328 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.328 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.329 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.329 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.329 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.329 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.329 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.329 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.329 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.330 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.330 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.330 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.330 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.330 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.331 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.331 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.331 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.332 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.332 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.332 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.332 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.332 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.332 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.332 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.332 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.333 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.333 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.333 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.333 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.333 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.334 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.334 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.334 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.334 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.334 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.334 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.334 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.334 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.335 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.335 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.335 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.335 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.335 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.335 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.336 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.336 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.336 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.336 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.336 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.336 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.336 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.336 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.337 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.337 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.337 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.337 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-21T09:01:26.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-21T09:01:26.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.338 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.338 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.338 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.338 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.338 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.338 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.338 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.339 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.339 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.339 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-21T09:01:26.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-21T09:01:26.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.339 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.339 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.340 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.340 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.340 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.340 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.340 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.341 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.341 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-04-21T09:01:26.341 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.341 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.341 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.341 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.342 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.342 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.342 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.342 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.342 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.342 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.342 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.343 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.343 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.343 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.343 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.343 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.344 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.344 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.344 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.344 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.344 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.345 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.345 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.345 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.345 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.345 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.345 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.345 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.346 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.346 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.346 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.347 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.347 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.347 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.347 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.348 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.348 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.348 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.348 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.349 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.349 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.349 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.349 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.350 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.350 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.350 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.350 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.351 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.352 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.352 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.352 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.352 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.352 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.352 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.353 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.353 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.353 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.353 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.354 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-04-21T09:01:26.355 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.355 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.355 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.355 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.355 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.357 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.357 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.357 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.357 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.358 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.358 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.358 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.359 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.359 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.359 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.359 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.361 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.361 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.361 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.362 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.362 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.362 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.363 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.363 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.363 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.364 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.364 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.364 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.364 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.364 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.364 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.364 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.365 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.365 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.365 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.365 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.365 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.365 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.365 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.365 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.366 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.366 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.366 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.366 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.367 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.367 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.367 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.367 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.367 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.367 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.367 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.367 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.368 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.368 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-04-21T09:01:26.368 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-04-21T09:01:26.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.368 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.369 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-04-21T09:01:26.369 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.369 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.369 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T09:01:26.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.369 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.369 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.370 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.370 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.370 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-04-21T09:01:26.370 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.370 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.370 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.370 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.371 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.371 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.371 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.371 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.371 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.371 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.372 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.372 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.372 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.372 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.372 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.373 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.373 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.373 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.373 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.373 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.373 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.374 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.374 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.375 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.375 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.375 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.375 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.376 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.376 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.376 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.377 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.377 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.377 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.378 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.379 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.379 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.379 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.379 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.379 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.379 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.379 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.380 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.380 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.380 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.380 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.380 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.380 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.381 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.381 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.381 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.381 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.382 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.382 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.382 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.383 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.383 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.384 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.384 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-04-21T09:01:26.385 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-04-21T09:01:26.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.385 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.385 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.385 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.386 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.386 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.386 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.386 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.387 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.387 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.387 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.387 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-04-21T09:01:26.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-04-21T09:01:26.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.388 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.388 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.389 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-04-21T09:01:26.389 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.389 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.389 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.389 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.389 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.390 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.390 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-04-21T09:01:26.390 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-04-21T09:01:26.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.390 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.391 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.391 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.391 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.391 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.391 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.391 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.392 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-04-21T09:01:26.392 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-04-21T09:01:26.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.393 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.393 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.393 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.393 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.393 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.393 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.393 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.393 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.394 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-04-21T09:01:26.394 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-04-21T09:01:26.394 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-21T09:01:26.394 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.394 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.394 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.395 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.395 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.396 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.396 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.396 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-04-21T09:01:26.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-04-21T09:01:26.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.397 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.397 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.397 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.397 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.398 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.398 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.398 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.398 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.398 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.398 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-04-21T09:01:26.399 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-04-21T09:01:26.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.399 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.399 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.399 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.399 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.399 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.399 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.400 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.400 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.400 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.400 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-04-21T09:01:26.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-04-21T09:01:26.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.401 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.401 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.401 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.401 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.401 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.401 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.402 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.402 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-04-21T09:01:26.402 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-04-21T09:01:26.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.402 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.402 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.403 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-21T09:01:26.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-04-21T09:01:26.403 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.403 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.403 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.403 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.403 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.403 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-04-21T09:01:26.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-04-21T09:01:26.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.404 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.404 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.404 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.404 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.404 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.405 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.405 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.405 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.405 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.405 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-04-21T09:01:26.405 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-04-21T09:01:26.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.406 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.406 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.406 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.406 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.406 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.406 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.407 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.407 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.407 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-04-21T09:01:26.407 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-04-21T09:01:26.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.407 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.408 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.408 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.408 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.408 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.408 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.409 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.409 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.409 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-21T09:01:26.410 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.410 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.410 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.411 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-04-21T09:01:26.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-04-21T09:01:26.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.411 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.411 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.412 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.412 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.412 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T09:01:26.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.412 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-04-21T09:01:26.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-04-21T09:01:26.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.413 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.413 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.413 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.413 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.413 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-04-21T09:01:26.414 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.414 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.414 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.414 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.414 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.414 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.415 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.415 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.415 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.415 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.416 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.416 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.416 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.416 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.416 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.416 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.416 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.417 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.417 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.417 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.417 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.417 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.417 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.417 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.417 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.418 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.418 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.418 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.418 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.418 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.418 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.418 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.419 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.419 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.419 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.419 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.420 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.420 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.420 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.420 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.420 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.420 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.420 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.420 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.421 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.421 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.421 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.421 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.421 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.421 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.422 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.422 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.422 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.422 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.422 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.422 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.422 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.423 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-04-21T09:01:26.423 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.423 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.423 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.424 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.424 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.424 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.424 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.424 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.424 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.424 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.425 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.425 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.425 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.425 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.425 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.425 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.425 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.425 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.426 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.426 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.426 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.426 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.426 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.426 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.427 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.427 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.427 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.427 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.428 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.428 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.428 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.428 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.428 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.429 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.429 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.429 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.429 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.429 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.429 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.429 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.429 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.430 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.430 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.430 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.430 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.430 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.431 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.431 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.431 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.431 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.432 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.432 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.432 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-04-21T09:01:26.433 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.433 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.433 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.433 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.433 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.433 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.434 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.434 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.434 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.434 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.434 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.435 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.435 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.435 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.435 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.435 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.436 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.436 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.437 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.437 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.437 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.437 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.438 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.438 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.438 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.438 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.438 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.439 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.439 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.439 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.439 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.439 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.439 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.439 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.440 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.440 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.440 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.440 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.440 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.441 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.441 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.441 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.441 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.441 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.441 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.442 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.442 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.442 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.442 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.442 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-04-21T09:01:26.442 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.442 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.443 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.443 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-04-21T09:01:26.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-04-21T09:01:26.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.443 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T09:01:26.443 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.444 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.444 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-21T09:01:26.444 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-21T09:01:26.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:26.445 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.445 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-04-21T09:01:26.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.445 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.445 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.445 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.445 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.446 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.446 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.446 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.446 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.446 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.446 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.446 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.447 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.447 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.447 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.447 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.447 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.447 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.447 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.447 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.448 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.448 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.448 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.448 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.448 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.448 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.448 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.448 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.449 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.449 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.449 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.449 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.449 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.449 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.449 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.449 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.449 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.450 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.450 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.450 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.450 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.451 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.451 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.451 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.451 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.451 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.451 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.451 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.451 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.451 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.452 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.452 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.452 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.452 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.452 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.452 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.452 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.452 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.453 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.453 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.453 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.453 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.453 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.453 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.453 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.454 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.454 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.454 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.454 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-04-21T09:01:26.455 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.455 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.455 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.455 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-04-21T09:01:26.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-04-21T09:01:26.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.456 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.456 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.456 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.456 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.456 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T09:01:26.456 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.457 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.457 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-04-21T09:01:26.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-04-21T09:01:26.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.457 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.458 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.458 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.458 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T09:01:26.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.458 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.459 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-04-21T09:01:26.459 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-04-21T09:01:26.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.459 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.459 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.459 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.460 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.460 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T09:01:26.460 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.460 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.460 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-04-21T09:01:26.461 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-04-21T09:01:26.461 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.461 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.461 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.461 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.461 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.461 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T09:01:26.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.462 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.462 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.462 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-04-21T09:01:26.462 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-04-21T09:01:26.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.463 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.463 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.463 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.463 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.463 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T09:01:26.463 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.464 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.464 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-04-21T09:01:26.464 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-04-21T09:01:26.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.464 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.465 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.465 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.465 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.465 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.465 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.465 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.466 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.466 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-04-21T09:01:26.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-04-21T09:01:26.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.466 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.466 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.466 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.467 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.467 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.467 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.467 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.467 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.468 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-04-21T09:01:26.468 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-04-21T09:01:26.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.468 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T09:01:26.468 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.469 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.469 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.469 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.469 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.469 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.469 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.469 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-21T09:01:26.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-21T09:01:26.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.470 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.470 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.470 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.470 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.470 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.470 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.471 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.471 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.471 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-21T09:01:26.471 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-21T09:01:26.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.472 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-04-21T09:01:26.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-04-21T09:01:26.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.472 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.473 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.473 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.473 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-04-21T09:01:26.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-04-21T09:01:26.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.473 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.473 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.473 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-04-21T09:01:26.474 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.474 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.474 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.474 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.474 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.474 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.474 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.475 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-04-21T09:01:26.475 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-04-21T09:01:26.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.475 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.475 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.475 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.476 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.476 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.476 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.476 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.476 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.476 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.476 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-04-21T09:01:26.476 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-04-21T09:01:26.477 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.477 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.477 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.477 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.477 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.477 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.478 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.478 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.478 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.478 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.478 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.479 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.479 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.479 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.479 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.479 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.479 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.479 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.479 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.479 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.480 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.480 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.480 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.481 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.481 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.481 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.481 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.481 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.481 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.482 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.482 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.482 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.482 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.482 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.483 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.483 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.483 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.483 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.483 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.483 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.483 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.483 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.484 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.484 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.484 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.484 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.484 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.484 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.484 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.485 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.485 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.485 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.485 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.485 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.485 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.485 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.486 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.486 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.486 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-04-21T09:01:26.486 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-04-21T09:01:26.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.486 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.487 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.487 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.487 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.487 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.487 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.487 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.487 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.487 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.488 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-04-21T09:01:26.488 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-04-21T09:01:26.488 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.488 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.488 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.488 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.488 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.488 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.488 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.489 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.489 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.489 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.489 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.489 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-04-21T09:01:26.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-04-21T09:01:26.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.489 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.490 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T09:01:26.490 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.490 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.490 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.490 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.490 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.490 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.490 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.491 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-04-21T09:01:26.491 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.491 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.491 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.491 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.491 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.492 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.492 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.492 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.492 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.492 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.492 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.492 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.493 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.493 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.493 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.493 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.493 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.493 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.493 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.493 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.493 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.494 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.494 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.494 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.494 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.494 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.494 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.495 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.495 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.495 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.495 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.495 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.495 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.495 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.495 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.495 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.496 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.496 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.496 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.496 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.496 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.496 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.496 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.497 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.497 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.497 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.497 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.497 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.497 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.497 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.498 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.498 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.498 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.498 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.498 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.499 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.499 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.499 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.499 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.499 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.499 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.500 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.500 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.500 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.500 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.500 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-21T09:01:26.500 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.500 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.500 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.501 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.501 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.501 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.501 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.501 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.501 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.501 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.502 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.502 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.502 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.502 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.502 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.502 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.502 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.502 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.503 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.503 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.503 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.503 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.503 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.504 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.504 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.504 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.504 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.504 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.504 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.504 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.504 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.504 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.505 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.505 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.505 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.505 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.505 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.505 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.505 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.505 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.506 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.506 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.506 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.506 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.506 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.506 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.506 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.506 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.506 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.507 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.507 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.507 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.507 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.507 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.507 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.507 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.508 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.508 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.508 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.508 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.509 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-04-21T09:01:26.509 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-04-21T09:01:26.509 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.509 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.509 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.509 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-04-21T09:01:26.509 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.509 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.509 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.510 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T09:01:26.510 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.510 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.510 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.510 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-04-21T09:01:26.510 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-04-21T09:01:26.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.510 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.511 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.511 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.511 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.511 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.511 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-04-21T09:01:26.511 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.512 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.512 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.512 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-04-21T09:01:26.512 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-04-21T09:01:26.512 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.512 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.512 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-21T09:01:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T09:01:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.513 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-04-21T09:01:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T09:01:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-21T09:01:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.514 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T09:01:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-04-21T09:01:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-21T09:01:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-21T09:01:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-21T09:01:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-04-21T09:01:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-04-21T09:01:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-21T09:01:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.516 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-04-21T09:01:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-04-21T09:01:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-04-21T09:01:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.518 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-21T09:01:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-21T09:01:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-21T09:01:26.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-21T09:01:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.520 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-21T09:01:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-21T09:01:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.521 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.522 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-21T09:01:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-21T09:01:26.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-21T09:01:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-21T09:01:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-21T09:01:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-21T09:01:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-21T09:01:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-21T09:01:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-21T09:01:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-21T09:01:26.526 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-21T09:01:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-21T09:01:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.527 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-21T09:01:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-21T09:01:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-21T09:01:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-21T09:01:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-21T09:01:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.528 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-04-21T09:01:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-04-21T09:01:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.529 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-21T09:01:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.530 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-04-21T09:01:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-04-21T09:01:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-21T09:01:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-04-21T09:01:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.531 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-04-21T09:01:26.532 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-04-21T09:01:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-21T09:01:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-21T09:01:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-21T09:01:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T09:01:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-21T09:01:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-21T09:01:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-21T09:01:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-21T09:01:26.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-21T09:01:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-21T09:01:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-21T09:01:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: "services": { 2024-04-21T09:01:26.534 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard": "https://172.21.15.28:8443/" 2024-04-21T09:01:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "always_on_modules": { 2024-04-21T09:01:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus": [ 2024-04-21T09:01:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-21T09:01:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-21T09:01:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-21T09:01:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-21T09:01:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-21T09:01:26.535 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-21T09:01:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-21T09:01:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-21T09:01:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-21T09:01:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-21T09:01:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific": [ 2024-04-21T09:01:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-21T09:01:26.536 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-21T09:01:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-21T09:01:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-21T09:01:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-21T09:01:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-21T09:01:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-21T09:01:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-21T09:01:26.537 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-21T09:01:26.538 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-21T09:01:26.538 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.538 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy": [ 2024-04-21T09:01:26.538 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-21T09:01:26.538 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-21T09:01:26.538 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-21T09:01:26.538 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-21T09:01:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-21T09:01:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-21T09:01:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-21T09:01:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-21T09:01:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-21T09:01:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-21T09:01:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: "reef": [ 2024-04-21T09:01:26.539 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-21T09:01:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-21T09:01:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-21T09:01:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-21T09:01:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-21T09:01:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-21T09:01:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-21T09:01:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-21T09:01:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-21T09:01:26.540 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-21T09:01:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-21T09:01:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: "squid": [ 2024-04-21T09:01:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-21T09:01:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-21T09:01:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-21T09:01:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-21T09:01:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-21T09:01:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-21T09:01:26.541 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-21T09:01:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-21T09:01:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-21T09:01:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-21T09:01:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-21T09:01:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: "last_failure_osd_epoch": 94, 2024-04-21T09:01:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: "active_clients": [ 2024-04-21T09:01:26.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-21T09:01:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-21T09:01:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-21T09:01:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-21T09:01:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 167231145 2024-04-21T09:01:26.543 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-21T09:01:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "libcephsqlite", 2024-04-21T09:01:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-21T09:01:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-21T09:01:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-21T09:01:26.544 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1811623576 2024-04-21T09:01:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-21T09:01:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-21T09:01:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-21T09:01:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-21T09:01:26.545 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-21T09:01:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3952326086 2024-04-21T09:01:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-21T09:01:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-21T09:01:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-21T09:01:26.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-21T09:01:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-21T09:01:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2492563802 2024-04-21T09:01:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-21T09:01:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.547 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-21T09:01:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-21T09:01:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-21T09:01:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-21T09:01:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3804793074 2024-04-21T09:01:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-21T09:01:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:26.548 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-21T09:01:26.549 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-21T09:01:26.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-21T09:01:26.549 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-21T09:01:26.549 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-21T09:01:26.549 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3794988944 2024-04-21T09:01:26.549 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.549 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-21T09:01:26.549 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:26.550 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-21T09:01:26.550 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T09:01:26.551 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:24 smithi171 ceph-mon[31358]: pgmap v317: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:26.552 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:24 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/158062892' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T09:01:26.552 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:24 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1326115225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T09:01:26.552 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:24 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2285632429' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T09:01:26.552 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:25 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/698589818' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T09:01:26.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:24 smithi152 ceph-mon[31401]: pgmap v317: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:26.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:24 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/158062892' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T09:01:26.552 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:24 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1326115225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T09:01:26.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:24 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2285632429' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T09:01:26.553 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:25 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/698589818' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T09:01:26.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:24.223+0000 7fcfd6043640 1 -- 172.21.15.28:0/698589818 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcfb80a3d50 msgr2=0x7fcfb80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:26.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:24.223+0000 7fcfd6043640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcfb80a3d50 0x7fcfb80a6210 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4036040 tx=0x7fcfc4000bc0 comp rx=0 tx=0).stop 2024-04-21T09:01:26.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:24.223+0000 7fcfd6043640 1 -- 172.21.15.28:0/698589818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfd0101110 msgr2=0x7fcfd0102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:01:26.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:24.223+0000 7fcfd6043640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfd0101110 0x7fcfd0102b50 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000c970 tx=0x7fcfc000ce30 comp rx=0 tx=0).stop 2024-04-21T09:01:26.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:24.224+0000 7fcfd6043640 1 -- 172.21.15.28:0/698589818 shutdown_connections 2024-04-21T09:01:26.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:24.224+0000 7fcfd6043640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fcfb80a3d50 0x7fcfb80a6210 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:26.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:24.224+0000 7fcfd6043640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcfd01035d0 0x7fcfd01a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:26.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:24.224+0000 7fcfd6043640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfd0106190 0x7fcfd0103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:26.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:24.224+0000 7fcfd6043640 1 --2- 172.21.15.28:0/698589818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfd0101110 0x7fcfd0102b50 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:01:26.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:24.224+0000 7fcfd6043640 1 -- 172.21.15.28:0/698589818 >> 172.21.15.28:0/698589818 conn(0x7fcfd0077fb0 msgr2=0x7fcfd006a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:01:26.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:24.224+0000 7fcfd6043640 1 -- 172.21.15.28:0/698589818 shutdown_connections 2024-04-21T09:01:26.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:01:24.224+0000 7fcfd6043640 1 -- 172.21.15.28:0/698589818 wait complete. 2024-04-21T09:01:26.561 DEBUG:tasks.check_counter:Getting stats from a 2024-04-21T09:01:26.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-21T09:01:26.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:24 smithi028 ceph-mon[28003]: pgmap v317: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:26.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:24 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/158062892' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T09:01:26.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:24 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1326115225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T09:01:26.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:24 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2285632429' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T09:01:26.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:25 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/698589818' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T09:01:26.588 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:26 smithi028 ceph-mon[28003]: pgmap v318: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T09:01:26.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:26 smithi152 ceph-mon[31401]: pgmap v318: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T09:01:26.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:26 smithi171 ceph-mon[31358]: pgmap v318: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T09:01:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T09:01:28.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:28 smithi028 ceph-mon[28003]: pgmap v319: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T09:01:28.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:28 smithi152 ceph-mon[31401]: pgmap v319: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T09:01:28.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:28 smithi171 ceph-mon[31358]: pgmap v319: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T09:01:29.304 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T09:01:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T09:01:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 218, 2024-04-21T09:01:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 207, 2024-04-21T09:01:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1588784, 2024-04-21T09:01:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 127264, 2024-04-21T09:01:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-04-21T09:01:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-04-21T09:01:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.044690122, 2024-04-21T09:01:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.012807480, 2024-04-21T09:01:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.035588519, 2024-04-21T09:01:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.004740658, 2024-04-21T09:01:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 206, 2024-04-21T09:01:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.010251358, 2024-04-21T09:01:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000049763 2024-04-21T09:01:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1588784, 2024-04-21T09:01:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 127264 2024-04-21T09:01:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T09:01:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 6, 2024-04-21T09:01:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 160, 2024-04-21T09:01:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 3312, 2024-04-21T09:01:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 184368, 2024-04-21T09:01:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-21T09:01:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-21T09:01:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.016739537, 2024-04-21T09:01:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.012322246, 2024-04-21T09:01:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.002042854, 2024-04-21T09:01:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T09:01:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 158, 2024-04-21T09:01:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.011913555, 2024-04-21T09:01:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000075402 2024-04-21T09:01:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-21T09:01:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 184368 2024-04-21T09:01:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T09:01:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 0, 2024-04-21T09:01:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2, 2024-04-21T09:01:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 0, 2024-04-21T09:01:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 0, 2024-04-21T09:01:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-04-21T09:01:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-21T09:01:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.000398449, 2024-04-21T09:01:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T09:01:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000332125, 2024-04-21T09:01:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T09:01:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T09:01:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T09:01:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-21T09:01:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-04-21T09:01:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-21T09:01:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-21T09:01:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-21T09:01:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-21T09:01:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T09:01:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-21T09:01:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T09:01:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-21T09:01:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T09:01:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T09:01:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T09:01:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T09:01:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T09:01:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-21T09:01:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T09:01:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T09:01:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-21T09:01:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-21T09:01:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-21T09:01:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-21T09:01:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T09:01:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T09:01:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T09:01:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T09:01:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T09:01:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-21T09:01:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-21T09:01:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-21T09:01:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T09:01:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T09:01:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-21T09:01:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-21T09:01:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-21T09:01:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-21T09:01:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T09:01:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-21T09:01:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T09:01:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-21T09:01:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 149017, 2024-04-21T09:01:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 31, 2024-04-21T09:01:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-21T09:01:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-21T09:01:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-21T09:01:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-21T09:01:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-21T09:01:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-21T09:01:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-21T09:01:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-21T09:01:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-04-21T09:01:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-04-21T09:01:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T09:01:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-21T09:01:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-21T09:01:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-21T09:01:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-04-21T09:01:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-04-21T09:01:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-21T09:01:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-21T09:01:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-21T09:01:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-21T09:01:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T09:01:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-21T09:01:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-21T09:01:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-21T09:01:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 220, 2024-04-21T09:01:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1558040, 2024-04-21T09:01:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 220, 2024-04-21T09:01:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-21T09:01:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-21T09:01:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 220, 2024-04-21T09:01:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1558040, 2024-04-21T09:01:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-21T09:01:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:29.322 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T09:01:30.182 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-21T09:01:30.182 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-21T09:01:30.182 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-21T09:01:30.182 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-21T09:01:30.182 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-21T09:01:30.182 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-04-21T09:01:30.182 DEBUG:tasks.check_counter:Getting stats from d 2024-04-21T09:01:30.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-21T09:01:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T09:01:30.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:30 smithi028 ceph-mon[28003]: pgmap v320: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-21T09:01:30.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:30 smithi152 ceph-mon[31401]: pgmap v320: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-21T09:01:30.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:30 smithi171 ceph-mon[31358]: pgmap v320: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-21T09:01:32.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:32 smithi028 ceph-mon[28003]: pgmap v321: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-21T09:01:32.281 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T09:01:32.281 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T09:01:32.281 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 0, 2024-04-21T09:01:32.281 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2, 2024-04-21T09:01:32.281 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 0, 2024-04-21T09:01:32.281 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 0, 2024-04-21T09:01:32.282 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-04-21T09:01:32.282 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-04-21T09:01:32.282 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.002004034, 2024-04-21T09:01:32.282 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T09:01:32.282 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000321083, 2024-04-21T09:01:32.282 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T09:01:32.282 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:32.282 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:32.283 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:32.283 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:32.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:32.283 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:32.283 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:32.283 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:32.283 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:32.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:32.283 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T09:01:32.284 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T09:01:32.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:32.284 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T09:01:32.284 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 6, 2024-04-21T09:01:32.284 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 165, 2024-04-21T09:01:32.284 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 3312, 2024-04-21T09:01:32.284 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 185248, 2024-04-21T09:01:32.284 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-21T09:01:32.285 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-21T09:01:32.285 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.013527495, 2024-04-21T09:01:32.285 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.008847627, 2024-04-21T09:01:32.285 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.002230730, 2024-04-21T09:01:32.285 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T09:01:32.285 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:32.285 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 163, 2024-04-21T09:01:32.285 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.014883948, 2024-04-21T09:01:32.285 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000091312 2024-04-21T09:01:32.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:32.286 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:32.286 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:32.286 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:32.286 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:32.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:32.286 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-21T09:01:32.286 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 185248 2024-04-21T09:01:32.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:32.287 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T09:01:32.287 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 225, 2024-04-21T09:01:32.287 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 214, 2024-04-21T09:01:32.287 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1590752, 2024-04-21T09:01:32.287 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 131264, 2024-04-21T09:01:32.287 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-04-21T09:01:32.287 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-21T09:01:32.287 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.047825771, 2024-04-21T09:01:32.288 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.014110868, 2024-04-21T09:01:32.288 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.038299921, 2024-04-21T09:01:32.288 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.004630532, 2024-04-21T09:01:32.288 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:32.288 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 213, 2024-04-21T09:01:32.288 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.011966795, 2024-04-21T09:01:32.288 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000056182 2024-04-21T09:01:32.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:32.288 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:32.289 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:32.289 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:32.289 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:32.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:32.289 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1590752, 2024-04-21T09:01:32.289 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 131264 2024-04-21T09:01:32.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:32.289 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-21T09:01:32.289 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-04-21T09:01:32.290 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-21T09:01:32.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:32.290 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-21T09:01:32.290 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-21T09:01:32.290 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-21T09:01:32.290 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T09:01:32.290 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-21T09:01:32.290 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T09:01:32.291 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-21T09:01:32.291 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T09:01:32.291 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T09:01:32.291 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T09:01:32.291 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T09:01:32.291 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T09:01:32.291 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-21T09:01:32.291 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T09:01:32.292 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T09:01:32.292 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-21T09:01:32.292 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-21T09:01:32.292 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-21T09:01:32.292 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-21T09:01:32.292 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T09:01:32.292 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T09:01:32.292 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T09:01:32.293 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T09:01:32.293 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T09:01:32.293 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-21T09:01:32.293 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-21T09:01:32.293 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-21T09:01:32.293 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T09:01:32.293 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T09:01:32.293 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-21T09:01:32.293 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-21T09:01:32.294 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-21T09:01:32.294 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-21T09:01:32.294 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T09:01:32.294 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-21T09:01:32.294 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T09:01:32.294 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-21T09:01:32.294 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 149017, 2024-04-21T09:01:32.294 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 31, 2024-04-21T09:01:32.295 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-21T09:01:32.295 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-21T09:01:32.295 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-21T09:01:32.295 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-21T09:01:32.295 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-21T09:01:32.295 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-21T09:01:32.295 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-21T09:01:32.295 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-21T09:01:32.296 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-04-21T09:01:32.296 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-04-21T09:01:32.296 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T09:01:32.296 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-21T09:01:32.296 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-21T09:01:32.296 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-21T09:01:32.296 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-04-21T09:01:32.296 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-04-21T09:01:32.296 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-21T09:01:32.297 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-21T09:01:32.297 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-21T09:01:32.297 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-21T09:01:32.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:32.297 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T09:01:32.297 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-21T09:01:32.297 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-21T09:01:32.297 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-21T09:01:32.298 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 227, 2024-04-21T09:01:32.298 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1559014, 2024-04-21T09:01:32.298 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:32.298 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 227, 2024-04-21T09:01:32.298 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-21T09:01:32.298 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-21T09:01:32.298 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 227, 2024-04-21T09:01:32.298 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1559014, 2024-04-21T09:01:32.299 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-21T09:01:32.299 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:32.299 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:32.299 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:32.299 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:32.299 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:32.299 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T09:01:32.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:32 smithi152 ceph-mon[31401]: pgmap v321: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-21T09:01:32.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:32 smithi171 ceph-mon[31358]: pgmap v321: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-21T09:01:32.944 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-21T09:01:32.944 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-21T09:01:32.944 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-21T09:01:32.944 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-21T09:01:32.944 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-21T09:01:32.944 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-04-21T09:01:32.944 DEBUG:tasks.check_counter:Getting stats from g 2024-04-21T09:01:32.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-21T09:01:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T09:01:34.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:34 smithi028 ceph-mon[28003]: pgmap v322: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T09:01:34.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:34 smithi152 ceph-mon[31401]: pgmap v322: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T09:01:34.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:34 smithi171 ceph-mon[31358]: pgmap v322: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-21T09:01:34.972 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T09:01:34.972 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T09:01:34.972 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 220, 2024-04-21T09:01:34.972 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 209, 2024-04-21T09:01:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1589328, 2024-04-21T09:01:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 128544, 2024-04-21T09:01:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-04-21T09:01:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-04-21T09:01:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.046549583, 2024-04-21T09:01:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.013000732, 2024-04-21T09:01:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.038069560, 2024-04-21T09:01:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.004528383, 2024-04-21T09:01:34.973 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 208, 2024-04-21T09:01:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.012215462, 2024-04-21T09:01:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000058728 2024-04-21T09:01:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:34.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1589328, 2024-04-21T09:01:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 128544 2024-04-21T09:01:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T09:01:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2, 2024-04-21T09:01:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 160, 2024-04-21T09:01:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 320, 2024-04-21T09:01:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 184272, 2024-04-21T09:01:34.975 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-21T09:01:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-21T09:01:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.013403151, 2024-04-21T09:01:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.008757067, 2024-04-21T09:01:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001999868, 2024-04-21T09:01:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T09:01:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 157, 2024-04-21T09:01:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.018194137, 2024-04-21T09:01:34.976 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000115886 2024-04-21T09:01:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T09:01:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 184272 2024-04-21T09:01:34.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T09:01:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4, 2024-04-21T09:01:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4, 2024-04-21T09:01:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 2992, 2024-04-21T09:01:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 448, 2024-04-21T09:01:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-04-21T09:01:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-21T09:01:34.978 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.000798301, 2024-04-21T09:01:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000041740, 2024-04-21T09:01:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000756201, 2024-04-21T09:01:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T09:01:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3, 2024-04-21T09:01:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000855720, 2024-04-21T09:01:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000285240 2024-04-21T09:01:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:34.979 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:34.980 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:34.980 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:34.980 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:34.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:34.980 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-21T09:01:34.980 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T09:01:34.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:34.980 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-21T09:01:34.981 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-04-21T09:01:34.981 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-21T09:01:34.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:34.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-21T09:01:34.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-21T09:01:34.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-21T09:01:34.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T09:01:34.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-21T09:01:34.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T09:01:34.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-21T09:01:34.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T09:01:34.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T09:01:34.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T09:01:34.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T09:01:34.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T09:01:34.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-21T09:01:34.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T09:01:34.983 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T09:01:34.983 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-21T09:01:34.983 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-21T09:01:34.983 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-21T09:01:34.983 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-21T09:01:34.983 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T09:01:34.983 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T09:01:34.983 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T09:01:34.984 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T09:01:34.984 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T09:01:34.984 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-21T09:01:34.984 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-21T09:01:34.984 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-21T09:01:34.984 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T09:01:34.984 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T09:01:34.984 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-21T09:01:34.985 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-21T09:01:34.985 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-21T09:01:34.985 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-21T09:01:34.985 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T09:01:34.985 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-21T09:01:34.985 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T09:01:34.985 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-21T09:01:34.985 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 149057, 2024-04-21T09:01:34.986 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 32, 2024-04-21T09:01:34.986 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-21T09:01:34.986 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-21T09:01:34.986 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-21T09:01:34.986 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-21T09:01:34.986 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-21T09:01:34.986 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-21T09:01:34.986 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-21T09:01:34.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-21T09:01:34.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-04-21T09:01:34.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-04-21T09:01:34.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T09:01:34.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-21T09:01:34.987 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-21T09:01:34.987 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-21T09:01:34.987 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-04-21T09:01:34.988 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-04-21T09:01:34.988 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-21T09:01:34.988 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-21T09:01:34.988 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-21T09:01:34.988 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-21T09:01:34.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:34.988 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T09:01:34.988 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-21T09:01:34.988 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-21T09:01:34.989 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-21T09:01:34.989 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 222, 2024-04-21T09:01:34.989 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1558300, 2024-04-21T09:01:34.989 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:34.989 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 222, 2024-04-21T09:01:34.989 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-21T09:01:34.989 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-21T09:01:34.989 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 222, 2024-04-21T09:01:34.990 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1558300, 2024-04-21T09:01:34.990 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-21T09:01:34.990 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:34.990 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:34.990 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:34.990 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:34.990 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:34.990 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T09:01:35.743 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-21T09:01:35.743 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-21T09:01:35.744 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-21T09:01:35.744 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-21T09:01:35.744 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-21T09:01:35.744 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-04-21T09:01:35.744 DEBUG:tasks.check_counter:Getting stats from j 2024-04-21T09:01:35.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-21T09:01:36.000 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T09:01:36.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:36 smithi028 ceph-mon[28003]: pgmap v323: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-21T09:01:36.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:36 smithi152 ceph-mon[31401]: pgmap v323: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-21T09:01:36.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:36 smithi171 ceph-mon[31358]: pgmap v323: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-21T09:01:37.836 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-21T09:01:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T09:01:37.836 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 0, 2024-04-21T09:01:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2, 2024-04-21T09:01:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 0, 2024-04-21T09:01:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 0, 2024-04-21T09:01:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-04-21T09:01:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-04-21T09:01:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.001339791, 2024-04-21T09:01:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T09:01:37.837 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000335975, 2024-04-21T09:01:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T09:01:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:37.838 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T09:01:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T09:01:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T09:01:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 231, 2024-04-21T09:01:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 218, 2024-04-21T09:01:37.839 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1594288, 2024-04-21T09:01:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 132992, 2024-04-21T09:01:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-04-21T09:01:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-21T09:01:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.047836164, 2024-04-21T09:01:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.014127614, 2024-04-21T09:01:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.038769595, 2024-04-21T09:01:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.004726840, 2024-04-21T09:01:37.840 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 218, 2024-04-21T09:01:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.012297195, 2024-04-21T09:01:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000056409 2024-04-21T09:01:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:37.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1594288, 2024-04-21T09:01:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 132992 2024-04-21T09:01:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T09:01:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2, 2024-04-21T09:01:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 165, 2024-04-21T09:01:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 320, 2024-04-21T09:01:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 185152, 2024-04-21T09:01:37.842 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-21T09:01:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-21T09:01:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.014162027, 2024-04-21T09:01:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.009821932, 2024-04-21T09:01:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001852914, 2024-04-21T09:01:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T09:01:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 162, 2024-04-21T09:01:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.010291006, 2024-04-21T09:01:37.843 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000063524 2024-04-21T09:01:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T09:01:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 185152 2024-04-21T09:01:37.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-21T09:01:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-04-21T09:01:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-21T09:01:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-21T09:01:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-21T09:01:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-21T09:01:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T09:01:37.845 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-21T09:01:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T09:01:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-21T09:01:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T09:01:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T09:01:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T09:01:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T09:01:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T09:01:37.846 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-21T09:01:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T09:01:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T09:01:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-21T09:01:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-21T09:01:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-21T09:01:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-21T09:01:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T09:01:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T09:01:37.847 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T09:01:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T09:01:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T09:01:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-21T09:01:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-21T09:01:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-21T09:01:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T09:01:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T09:01:37.848 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-21T09:01:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-21T09:01:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-21T09:01:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-21T09:01:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T09:01:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-21T09:01:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T09:01:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-21T09:01:37.849 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 150008, 2024-04-21T09:01:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 31, 2024-04-21T09:01:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-21T09:01:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-21T09:01:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-21T09:01:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-21T09:01:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-21T09:01:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-21T09:01:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-21T09:01:37.850 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-21T09:01:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-04-21T09:01:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-04-21T09:01:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T09:01:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-21T09:01:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-21T09:01:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-21T09:01:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-04-21T09:01:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-04-21T09:01:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-21T09:01:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-21T09:01:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-21T09:01:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-21T09:01:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-21T09:01:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T09:01:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-21T09:01:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-21T09:01:37.852 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-21T09:01:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 229, 2024-04-21T09:01:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1559274, 2024-04-21T09:01:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 229, 2024-04-21T09:01:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-21T09:01:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-21T09:01:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 229, 2024-04-21T09:01:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1559274, 2024-04-21T09:01:37.853 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-21T09:01:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-21T09:01:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-21T09:01:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-21T09:01:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:37.854 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-21T09:01:37.854 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-21T09:01:38.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:38 smithi028 ceph-mon[28003]: pgmap v324: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:38.523 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-21T09:01:38.523 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-21T09:01:38.523 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-21T09:01:38.523 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-21T09:01:38.523 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-21T09:01:38.523 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-04-21T09:01:38.523 DEBUG:tasks.check_counter:Getting stats from b 2024-04-21T09:01:38.524 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-21T09:01:38.579 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:38 smithi152 ceph-mon[31401]: pgmap v324: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:38.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:38 smithi171 ceph-mon[31358]: pgmap v324: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:38.841 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T09:01:40.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:40 smithi152 ceph-mon[31401]: pgmap v325: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T09:01:40.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:40 smithi028 ceph-mon[28003]: pgmap v325: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T09:01:40.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:40 smithi171 ceph-mon[31358]: pgmap v325: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T09:01:41.337 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-04-21T09:01:41.337 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T09:01:41.337 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 4549, 2024-04-21T09:01:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 4182, 2024-04-21T09:01:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 3400339, 2024-04-21T09:01:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 6148057, 2024-04-21T09:01:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 37, 2024-04-21T09:01:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 5, 2024-04-21T09:01:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.466370020, 2024-04-21T09:01:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.125197559, 2024-04-21T09:01:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.174471230, 2024-04-21T09:01:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.150463107, 2024-04-21T09:01:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 4181, 2024-04-21T09:01:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.336965300, 2024-04-21T09:01:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000080594 2024-04-21T09:01:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 3442, 2024-04-21T09:01:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000725951, 2024-04-21T09:01:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000210 2024-04-21T09:01:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 2010912, 2024-04-21T09:01:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 248736 2024-04-21T09:01:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T09:01:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 6609, 2024-04-21T09:01:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 8951, 2024-04-21T09:01:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 2551435, 2024-04-21T09:01:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 8586429, 2024-04-21T09:01:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 168, 2024-04-21T09:01:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 9, 2024-04-21T09:01:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.553541335, 2024-04-21T09:01:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.178585140, 2024-04-21T09:01:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.277348407, 2024-04-21T09:01:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.101882341, 2024-04-21T09:01:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 8948, 2024-04-21T09:01:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 1.389045790, 2024-04-21T09:01:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000155235 2024-04-21T09:01:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 9835, 2024-04-21T09:01:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.001572217, 2024-04-21T09:01:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000159 2024-04-21T09:01:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 96992, 2024-04-21T09:01:41.343 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 486848 2024-04-21T09:01:41.343 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.343 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T09:01:41.343 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 4290, 2024-04-21T09:01:41.343 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 4373, 2024-04-21T09:01:41.343 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 2028648, 2024-04-21T09:01:41.343 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 6318479, 2024-04-21T09:01:41.343 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 28, 2024-04-21T09:01:41.343 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 6, 2024-04-21T09:01:41.344 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.340742150, 2024-04-21T09:01:41.344 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.113054802, 2024-04-21T09:01:41.344 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.111784171, 2024-04-21T09:01:41.344 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.095758285, 2024-04-21T09:01:41.344 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:41.344 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 4372, 2024-04-21T09:01:41.344 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.735195704, 2024-04-21T09:01:41.344 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000168160 2024-04-21T09:01:41.344 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.344 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:41.345 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 2206, 2024-04-21T09:01:41.345 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000472353, 2024-04-21T09:01:41.345 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000214 2024-04-21T09:01:41.345 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.345 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 46464, 2024-04-21T09:01:41.345 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 4864 2024-04-21T09:01:41.345 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.345 INFO:teuthology.orchestra.run.smithi152.stdout: "cct": { 2024-04-21T09:01:41.346 INFO:teuthology.orchestra.run.smithi152.stdout: "total_workers": 1, 2024-04-21T09:01:41.346 INFO:teuthology.orchestra.run.smithi152.stdout: "unhealthy_workers": 0 2024-04-21T09:01:41.346 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.346 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-MDSRank": { 2024-04-21T09:01:41.346 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-04-21T09:01:41.346 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-04-21T09:01:41.346 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 4868, 2024-04-21T09:01:41.346 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 2.808353682, 2024-04-21T09:01:41.347 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000576900 2024-04-21T09:01:41.347 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:41.347 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.347 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-PurgeQueue": { 2024-04-21T09:01:41.347 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-04-21T09:01:41.347 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-04-21T09:01:41.347 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 810, 2024-04-21T09:01:41.347 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.099460343, 2024-04-21T09:01:41.347 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000122790 2024-04-21T09:01:41.348 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:41.348 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.348 INFO:teuthology.orchestra.run.smithi152.stdout: "mds": { 2024-04-21T09:01:41.348 INFO:teuthology.orchestra.run.smithi152.stdout: "request": 1992, 2024-04-21T09:01:41.348 INFO:teuthology.orchestra.run.smithi152.stdout: "reply": 1984, 2024-04-21T09:01:41.348 INFO:teuthology.orchestra.run.smithi152.stdout: "reply_latency": { 2024-04-21T09:01:41.348 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 1984, 2024-04-21T09:01:41.348 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 14.820060622, 2024-04-21T09:01:41.348 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.007469788 2024-04-21T09:01:41.349 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.349 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_reply": 0, 2024-04-21T09:01:41.349 INFO:teuthology.orchestra.run.smithi152.stdout: "forward": 8, 2024-04-21T09:01:41.349 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_complete": 5, 2024-04-21T09:01:41.349 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_keys": 2, 2024-04-21T09:01:41.349 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_commit": 290, 2024-04-21T09:01:41.349 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_split": 0, 2024-04-21T09:01:41.349 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_merge": 0, 2024-04-21T09:01:41.349 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes": 422, 2024-04-21T09:01:41.349 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_top": 99, 2024-04-21T09:01:41.350 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_bottom": 310, 2024-04-21T09:01:41.350 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pin_tail": 13, 2024-04-21T09:01:41.350 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pinned": 94, 2024-04-21T09:01:41.350 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_expired": 65, 2024-04-21T09:01:41.350 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_with_caps": 19, 2024-04-21T09:01:41.350 INFO:teuthology.orchestra.run.smithi152.stdout: "caps": 20, 2024-04-21T09:01:41.350 INFO:teuthology.orchestra.run.smithi152.stdout: "subtrees": 15, 2024-04-21T09:01:41.350 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse": 2643, 2024-04-21T09:01:41.350 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_hit": 2490, 2024-04-21T09:01:41.351 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_forward": 0, 2024-04-21T09:01:41.351 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_discover": 0, 2024-04-21T09:01:41.351 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_dir_fetch": 0, 2024-04-21T09:01:41.351 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_remote_ino": 0, 2024-04-21T09:01:41.351 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_lock": 51, 2024-04-21T09:01:41.351 INFO:teuthology.orchestra.run.smithi152.stdout: "load_cent": 0, 2024-04-21T09:01:41.351 INFO:teuthology.orchestra.run.smithi152.stdout: "q": 0, 2024-04-21T09:01:41.351 INFO:teuthology.orchestra.run.smithi152.stdout: "exported": 10, 2024-04-21T09:01:41.351 INFO:teuthology.orchestra.run.smithi152.stdout: "exported_inodes": 57, 2024-04-21T09:01:41.351 INFO:teuthology.orchestra.run.smithi152.stdout: "imported": 0, 2024-04-21T09:01:41.352 INFO:teuthology.orchestra.run.smithi152.stdout: "imported_inodes": 0, 2024-04-21T09:01:41.352 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_dir_fetch": 0, 2024-04-21T09:01:41.352 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_backtrace_fetch": 4160, 2024-04-21T09:01:41.352 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_peer_discover": 0, 2024-04-21T09:01:41.352 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rfiles": 325, 2024-04-21T09:01:41.352 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rbytes": 13424230466, 2024-04-21T09:01:41.352 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rsnaps": 6, 2024-04-21T09:01:41.352 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_fetch": 4160, 2024-04-21T09:01:41.352 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_set_tag": 0, 2024-04-21T09:01:41.353 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T09:01:41.353 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_inotable_repaired": 0, 2024-04-21T09:01:41.353 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_inodes": 1328, 2024-04-21T09:01:41.353 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_base_inodes": 18, 2024-04-21T09:01:41.353 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dirfrag_rstats": 1310, 2024-04-21T09:01:41.353 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_file_inodes": 2832, 2024-04-21T09:01:41.353 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_inode_file_caps": 2, 2024-04-21T09:01:41.353 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_revoke": 455, 2024-04-21T09:01:41.353 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_grant": 1848, 2024-04-21T09:01:41.353 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_trunc": 6, 2024-04-21T09:01:41.354 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T09:01:41.354 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T09:01:41.354 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps": 1524, 2024-04-21T09:01:41.354 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps_dirty": 853, 2024-04-21T09:01:41.354 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_cap_release": 135, 2024-04-21T09:01:41.354 INFO:teuthology.orchestra.run.smithi152.stdout: "process_request_cap_release": 916 2024-04-21T09:01:41.354 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.354 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_cache": { 2024-04-21T09:01:41.354 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update": 2, 2024-04-21T09:01:41.355 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update_receipt": 0, 2024-04-21T09:01:41.355 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_try_discover": 0, 2024-04-21T09:01:41.355 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_send_discover": 0, 2024-04-21T09:01:41.355 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_handle_discover": 20, 2024-04-21T09:01:41.355 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays": 0, 2024-04-21T09:01:41.355 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_delayed": 0, 2024-04-21T09:01:41.355 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_enqueuing": 0, 2024-04-21T09:01:41.355 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_created": 530, 2024-04-21T09:01:41.355 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_enqueued": 469, 2024-04-21T09:01:41.355 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_reintegrated": 61, 2024-04-21T09:01:41.356 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_migrated": 0, 2024-04-21T09:01:41.356 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_processing": 0, 2024-04-21T09:01:41.356 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_enqueued": 0, 2024-04-21T09:01:41.356 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_prioritized": 0, 2024-04-21T09:01:41.356 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_started": 0, 2024-04-21T09:01:41.356 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_completed": 0, 2024-04-21T09:01:41.356 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_quiesce_path": 2, 2024-04-21T09:01:41.356 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_quiesce_inode": 552, 2024-04-21T09:01:41.356 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_enqueue_scrub": 18, 2024-04-21T09:01:41.357 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_exportdir": 10, 2024-04-21T09:01:41.357 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_flush": 0, 2024-04-21T09:01:41.357 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragmentdir": 0, 2024-04-21T09:01:41.357 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragstats": 0, 2024-04-21T09:01:41.357 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_inodestats": 0 2024-04-21T09:01:41.357 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.357 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_log": { 2024-04-21T09:01:41.357 INFO:teuthology.orchestra.run.smithi152.stdout: "evlrg": 0, 2024-04-21T09:01:41.357 INFO:teuthology.orchestra.run.smithi152.stdout: "evadd": 3321, 2024-04-21T09:01:41.358 INFO:teuthology.orchestra.run.smithi152.stdout: "evex": 3275, 2024-04-21T09:01:41.358 INFO:teuthology.orchestra.run.smithi152.stdout: "evtrm": 3275, 2024-04-21T09:01:41.358 INFO:teuthology.orchestra.run.smithi152.stdout: "ev": 46, 2024-04-21T09:01:41.358 INFO:teuthology.orchestra.run.smithi152.stdout: "evexg": 0, 2024-04-21T09:01:41.358 INFO:teuthology.orchestra.run.smithi152.stdout: "evexd": 7, 2024-04-21T09:01:41.358 INFO:teuthology.orchestra.run.smithi152.stdout: "segadd": 25, 2024-04-21T09:01:41.358 INFO:teuthology.orchestra.run.smithi152.stdout: "segex": 24, 2024-04-21T09:01:41.358 INFO:teuthology.orchestra.run.smithi152.stdout: "segtrm": 24, 2024-04-21T09:01:41.358 INFO:teuthology.orchestra.run.smithi152.stdout: "seg": 1, 2024-04-21T09:01:41.359 INFO:teuthology.orchestra.run.smithi152.stdout: "segmjr": 1, 2024-04-21T09:01:41.359 INFO:teuthology.orchestra.run.smithi152.stdout: "segexg": 0, 2024-04-21T09:01:41.359 INFO:teuthology.orchestra.run.smithi152.stdout: "segexd": 1, 2024-04-21T09:01:41.359 INFO:teuthology.orchestra.run.smithi152.stdout: "expos": 12886343, 2024-04-21T09:01:41.359 INFO:teuthology.orchestra.run.smithi152.stdout: "wrpos": 13100644, 2024-04-21T09:01:41.359 INFO:teuthology.orchestra.run.smithi152.stdout: "rdpos": 0, 2024-04-21T09:01:41.359 INFO:teuthology.orchestra.run.smithi152.stdout: "jlat": { 2024-04-21T09:01:41.359 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 916, 2024-04-21T09:01:41.360 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 754.108874917, 2024-04-21T09:01:41.360 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.823262963 2024-04-21T09:01:41.360 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.360 INFO:teuthology.orchestra.run.smithi152.stdout: "replayed": 0 2024-04-21T09:01:41.360 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.360 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_mem": { 2024-04-21T09:01:41.360 INFO:teuthology.orchestra.run.smithi152.stdout: "ino": 288, 2024-04-21T09:01:41.360 INFO:teuthology.orchestra.run.smithi152.stdout: "ino+": 829, 2024-04-21T09:01:41.361 INFO:teuthology.orchestra.run.smithi152.stdout: "ino-": 541, 2024-04-21T09:01:41.361 INFO:teuthology.orchestra.run.smithi152.stdout: "dir": 113, 2024-04-21T09:01:41.361 INFO:teuthology.orchestra.run.smithi152.stdout: "dir+": 135, 2024-04-21T09:01:41.361 INFO:teuthology.orchestra.run.smithi152.stdout: "dir-": 22, 2024-04-21T09:01:41.361 INFO:teuthology.orchestra.run.smithi152.stdout: "dn": 422, 2024-04-21T09:01:41.361 INFO:teuthology.orchestra.run.smithi152.stdout: "dn+": 1121, 2024-04-21T09:01:41.361 INFO:teuthology.orchestra.run.smithi152.stdout: "dn-": 699, 2024-04-21T09:01:41.361 INFO:teuthology.orchestra.run.smithi152.stdout: "cap": 20, 2024-04-21T09:01:41.361 INFO:teuthology.orchestra.run.smithi152.stdout: "cap+": 802, 2024-04-21T09:01:41.361 INFO:teuthology.orchestra.run.smithi152.stdout: "cap-": 782, 2024-04-21T09:01:41.362 INFO:teuthology.orchestra.run.smithi152.stdout: "rss": 69828, 2024-04-21T09:01:41.362 INFO:teuthology.orchestra.run.smithi152.stdout: "heap": 231800 2024-04-21T09:01:41.362 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.362 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_server": { 2024-04-21T09:01:41.362 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_client_request": 2386, 2024-04-21T09:01:41.362 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_server_request": 65, 2024-04-21T09:01:41.362 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_request": 1992, 2024-04-21T09:01:41.362 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_session": 49, 2024-04-21T09:01:41.363 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_peer_request": 132, 2024-04-21T09:01:41.363 INFO:teuthology.orchestra.run.smithi152.stdout: "req_create_latency": { 2024-04-21T09:01:41.363 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 674, 2024-04-21T09:01:41.363 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.708208262, 2024-04-21T09:01:41.363 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.001050754 2024-04-21T09:01:41.363 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.363 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getattr_latency": { 2024-04-21T09:01:41.363 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 179, 2024-04-21T09:01:41.363 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.050094422, 2024-04-21T09:01:41.363 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000279857 2024-04-21T09:01:41.364 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.364 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getfilelock_latency": { 2024-04-21T09:01:41.364 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.364 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.364 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.364 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.364 INFO:teuthology.orchestra.run.smithi152.stdout: "req_link_latency": { 2024-04-21T09:01:41.364 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 61, 2024-04-21T09:01:41.364 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.051264863, 2024-04-21T09:01:41.365 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000840407 2024-04-21T09:01:41.365 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.365 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookup_latency": { 2024-04-21T09:01:41.365 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 95, 2024-04-21T09:01:41.365 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.199019016, 2024-04-21T09:01:41.365 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.002094937 2024-04-21T09:01:41.365 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.365 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookuphash_latency": { 2024-04-21T09:01:41.365 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.365 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.366 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.366 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.366 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupino_latency": { 2024-04-21T09:01:41.366 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 2, 2024-04-21T09:01:41.366 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000329093, 2024-04-21T09:01:41.366 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000164546 2024-04-21T09:01:41.366 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.366 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupname_latency": { 2024-04-21T09:01:41.367 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.367 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.367 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.367 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.367 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupparent_latency": { 2024-04-21T09:01:41.367 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.367 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.367 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.367 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.368 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupsnap_latency": { 2024-04-21T09:01:41.368 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.368 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.368 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.368 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.368 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lssnap_latency": { 2024-04-21T09:01:41.369 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 10, 2024-04-21T09:01:41.369 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.001875484, 2024-04-21T09:01:41.369 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000187548 2024-04-21T09:01:41.369 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.369 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mkdir_latency": { 2024-04-21T09:01:41.369 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 105, 2024-04-21T09:01:41.370 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.100373953, 2024-04-21T09:01:41.370 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000955942 2024-04-21T09:01:41.370 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.370 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mknod_latency": { 2024-04-21T09:01:41.370 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.370 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.370 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.370 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.371 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mksnap_latency": { 2024-04-21T09:01:41.371 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 10, 2024-04-21T09:01:41.371 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.322167072, 2024-04-21T09:01:41.371 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.032216707 2024-04-21T09:01:41.371 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.371 INFO:teuthology.orchestra.run.smithi152.stdout: "req_open_latency": { 2024-04-21T09:01:41.371 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 49, 2024-04-21T09:01:41.371 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.009930326, 2024-04-21T09:01:41.372 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000202659 2024-04-21T09:01:41.372 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.372 INFO:teuthology.orchestra.run.smithi152.stdout: "req_readdir_latency": { 2024-04-21T09:01:41.372 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 151, 2024-04-21T09:01:41.372 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 1.010581286, 2024-04-21T09:01:41.372 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.006692591 2024-04-21T09:01:41.372 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.372 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rename_latency": { 2024-04-21T09:01:41.373 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 113, 2024-04-21T09:01:41.373 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 11.306276303, 2024-04-21T09:01:41.373 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.100055542 2024-04-21T09:01:41.373 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.373 INFO:teuthology.orchestra.run.smithi152.stdout: "req_renamesnap_latency": { 2024-04-21T09:01:41.373 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.373 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.373 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.374 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.374 INFO:teuthology.orchestra.run.smithi152.stdout: "req_snapdiff_latency": { 2024-04-21T09:01:41.374 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.374 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.374 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.374 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.374 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmdir_latency": { 2024-04-21T09:01:41.374 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 16, 2024-04-21T09:01:41.374 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.184072710, 2024-04-21T09:01:41.375 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.011504544 2024-04-21T09:01:41.375 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.375 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmsnap_latency": { 2024-04-21T09:01:41.375 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 4, 2024-04-21T09:01:41.375 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.017609177, 2024-04-21T09:01:41.375 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.004402294 2024-04-21T09:01:41.375 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.375 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmxattr_latency": { 2024-04-21T09:01:41.376 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.376 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.376 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.376 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.376 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setattr_latency": { 2024-04-21T09:01:41.376 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 8, 2024-04-21T09:01:41.376 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.037463981, 2024-04-21T09:01:41.376 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.004682997 2024-04-21T09:01:41.377 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.377 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setdirlayout_latency": { 2024-04-21T09:01:41.377 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.377 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.377 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.377 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.377 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setfilelock_latency": { 2024-04-21T09:01:41.377 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 4, 2024-04-21T09:01:41.378 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.001724695, 2024-04-21T09:01:41.378 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000431173 2024-04-21T09:01:41.378 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.378 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setlayout_latency": { 2024-04-21T09:01:41.378 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.378 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.378 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.378 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.379 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setxattr_latency": { 2024-04-21T09:01:41.379 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 9, 2024-04-21T09:01:41.379 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.020701669, 2024-04-21T09:01:41.379 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.002300185 2024-04-21T09:01:41.379 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.379 INFO:teuthology.orchestra.run.smithi152.stdout: "req_symlink_latency": { 2024-04-21T09:01:41.379 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 13, 2024-04-21T09:01:41.379 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.032202839, 2024-04-21T09:01:41.379 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.002477141 2024-04-21T09:01:41.380 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.380 INFO:teuthology.orchestra.run.smithi152.stdout: "req_unlink_latency": { 2024-04-21T09:01:41.380 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 481, 2024-04-21T09:01:41.380 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.766165471, 2024-04-21T09:01:41.380 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.001592859 2024-04-21T09:01:41.380 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.380 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_revoke_eviction": 0, 2024-04-21T09:01:41.380 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_acquisition_throttle": 0, 2024-04-21T09:01:41.380 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getvxattr_latency": { 2024-04-21T09:01:41.381 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.381 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.381 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.381 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:41.381 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.381 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_sessions": { 2024-04-21T09:01:41.381 INFO:teuthology.orchestra.run.smithi152.stdout: "session_count": 2, 2024-04-21T09:01:41.381 INFO:teuthology.orchestra.run.smithi152.stdout: "session_add": 7, 2024-04-21T09:01:41.381 INFO:teuthology.orchestra.run.smithi152.stdout: "session_remove": 5, 2024-04-21T09:01:41.381 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_open": 2, 2024-04-21T09:01:41.382 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_stale": 0, 2024-04-21T09:01:41.382 INFO:teuthology.orchestra.run.smithi152.stdout: "total_load": 31, 2024-04-21T09:01:41.382 INFO:teuthology.orchestra.run.smithi152.stdout: "average_load": 15, 2024-04-21T09:01:41.382 INFO:teuthology.orchestra.run.smithi152.stdout: "avg_session_uptime": 601, 2024-04-21T09:01:41.382 INFO:teuthology.orchestra.run.smithi152.stdout: "mdthresh_evicted": 0 2024-04-21T09:01:41.382 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.382 INFO:teuthology.orchestra.run.smithi152.stdout: "mempool": { 2024-04-21T09:01:41.382 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_bytes": 0, 2024-04-21T09:01:41.383 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_items": 0, 2024-04-21T09:01:41.383 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T09:01:41.383 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_items": 0, 2024-04-21T09:01:41.383 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T09:01:41.383 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_items": 0, 2024-04-21T09:01:41.383 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T09:01:41.383 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T09:01:41.383 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T09:01:41.384 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T09:01:41.384 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T09:01:41.384 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_items": 0, 2024-04-21T09:01:41.384 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T09:01:41.384 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T09:01:41.384 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_bytes": 0, 2024-04-21T09:01:41.384 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_items": 0, 2024-04-21T09:01:41.384 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_bytes": 0, 2024-04-21T09:01:41.384 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_items": 0, 2024-04-21T09:01:41.384 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T09:01:41.385 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T09:01:41.385 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T09:01:41.385 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T09:01:41.385 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T09:01:41.385 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_items": 0, 2024-04-21T09:01:41.385 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_bytes": 0, 2024-04-21T09:01:41.385 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_items": 0, 2024-04-21T09:01:41.385 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T09:01:41.385 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T09:01:41.385 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_bytes": 0, 2024-04-21T09:01:41.386 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_items": 0, 2024-04-21T09:01:41.386 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_bytes": 0, 2024-04-21T09:01:41.386 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_items": 0, 2024-04-21T09:01:41.386 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T09:01:41.386 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_items": 0, 2024-04-21T09:01:41.386 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T09:01:41.386 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_items": 0, 2024-04-21T09:01:41.386 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_bytes": 320916, 2024-04-21T09:01:41.386 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_items": 376, 2024-04-21T09:01:41.387 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_bytes": 0, 2024-04-21T09:01:41.387 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_items": 0, 2024-04-21T09:01:41.387 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_bytes": 0, 2024-04-21T09:01:41.387 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_items": 0, 2024-04-21T09:01:41.387 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_bytes": 0, 2024-04-21T09:01:41.387 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_items": 0, 2024-04-21T09:01:41.387 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_bytes": 0, 2024-04-21T09:01:41.387 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_items": 0, 2024-04-21T09:01:41.388 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_bytes": 9616, 2024-04-21T09:01:41.388 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_items": 185, 2024-04-21T09:01:41.388 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T09:01:41.388 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_items": 0, 2024-04-21T09:01:41.388 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_bytes": 0, 2024-04-21T09:01:41.388 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_items": 0, 2024-04-21T09:01:41.388 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_bytes": 1277542, 2024-04-21T09:01:41.388 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_items": 22793, 2024-04-21T09:01:41.389 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_bytes": 0, 2024-04-21T09:01:41.389 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_items": 0, 2024-04-21T09:01:41.389 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_bytes": 0, 2024-04-21T09:01:41.389 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_items": 0 2024-04-21T09:01:41.389 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.389 INFO:teuthology.orchestra.run.smithi152.stdout: "objecter": { 2024-04-21T09:01:41.389 INFO:teuthology.orchestra.run.smithi152.stdout: "op_active": 0, 2024-04-21T09:01:41.389 INFO:teuthology.orchestra.run.smithi152.stdout: "op_laggy": 0, 2024-04-21T09:01:41.390 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send": 7559, 2024-04-21T09:01:41.390 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send_bytes": 10001320, 2024-04-21T09:01:41.390 INFO:teuthology.orchestra.run.smithi152.stdout: "op_resend": 32, 2024-04-21T09:01:41.390 INFO:teuthology.orchestra.run.smithi152.stdout: "op_reply": 7559, 2024-04-21T09:01:41.390 INFO:teuthology.orchestra.run.smithi152.stdout: "op_latency": { 2024-04-21T09:01:41.390 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 7559, 2024-04-21T09:01:41.390 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 1250.325789925, 2024-04-21T09:01:41.390 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.165408888 2024-04-21T09:01:41.391 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.391 INFO:teuthology.orchestra.run.smithi152.stdout: "op_inflight": 0, 2024-04-21T09:01:41.391 INFO:teuthology.orchestra.run.smithi152.stdout: "oplen_avg": { 2024-04-21T09:01:41.391 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 7559, 2024-04-21T09:01:41.391 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 9018 2024-04-21T09:01:41.391 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.391 INFO:teuthology.orchestra.run.smithi152.stdout: "op": 7559, 2024-04-21T09:01:41.391 INFO:teuthology.orchestra.run.smithi152.stdout: "op_r": 4214, 2024-04-21T09:01:41.392 INFO:teuthology.orchestra.run.smithi152.stdout: "op_w": 3345, 2024-04-21T09:01:41.392 INFO:teuthology.orchestra.run.smithi152.stdout: "op_rmw": 0, 2024-04-21T09:01:41.392 INFO:teuthology.orchestra.run.smithi152.stdout: "op_pg": 0, 2024-04-21T09:01:41.392 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_stat": 202, 2024-04-21T09:01:41.392 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_create": 545, 2024-04-21T09:01:41.392 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_read": 29, 2024-04-21T09:01:41.392 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_write": 927, 2024-04-21T09:01:41.392 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writefull": 93, 2024-04-21T09:01:41.393 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writesame": 0, 2024-04-21T09:01:41.393 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_append": 0, 2024-04-21T09:01:41.393 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_zero": 0, 2024-04-21T09:01:41.393 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_truncate": 0, 2024-04-21T09:01:41.393 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_delete": 1425, 2024-04-21T09:01:41.393 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_mapext": 0, 2024-04-21T09:01:41.393 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_sparse_read": 0, 2024-04-21T09:01:41.393 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_clonerange": 0, 2024-04-21T09:01:41.393 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_getxattr": 4185, 2024-04-21T09:01:41.393 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_setxattr": 877, 2024-04-21T09:01:41.394 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_cmpxattr": 0, 2024-04-21T09:01:41.394 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_rmxattr": 0, 2024-04-21T09:01:41.394 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_resetxattrs": 0, 2024-04-21T09:01:41.394 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_call": 0, 2024-04-21T09:01:41.394 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_watch": 0, 2024-04-21T09:01:41.394 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_notify": 0, 2024-04-21T09:01:41.394 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T09:01:41.394 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls": 0, 2024-04-21T09:01:41.394 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls_filter": 0, 2024-04-21T09:01:41.395 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_other": 6, 2024-04-21T09:01:41.395 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_active": 0, 2024-04-21T09:01:41.395 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_send": 0, 2024-04-21T09:01:41.395 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_resend": 0, 2024-04-21T09:01:41.395 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_ping": 0, 2024-04-21T09:01:41.395 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_active": 0, 2024-04-21T09:01:41.395 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_send": 0, 2024-04-21T09:01:41.395 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_resend": 0, 2024-04-21T09:01:41.395 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_active": 0, 2024-04-21T09:01:41.396 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_send": 0, 2024-04-21T09:01:41.396 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_resend": 0, 2024-04-21T09:01:41.396 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_active": 0, 2024-04-21T09:01:41.396 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_send": 0, 2024-04-21T09:01:41.396 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_resend": 0, 2024-04-21T09:01:41.396 INFO:teuthology.orchestra.run.smithi152.stdout: "command_active": 0, 2024-04-21T09:01:41.396 INFO:teuthology.orchestra.run.smithi152.stdout: "command_send": 0, 2024-04-21T09:01:41.396 INFO:teuthology.orchestra.run.smithi152.stdout: "command_resend": 0, 2024-04-21T09:01:41.396 INFO:teuthology.orchestra.run.smithi152.stdout: "map_epoch": 102, 2024-04-21T09:01:41.397 INFO:teuthology.orchestra.run.smithi152.stdout: "map_full": 0, 2024-04-21T09:01:41.397 INFO:teuthology.orchestra.run.smithi152.stdout: "map_inc": 18, 2024-04-21T09:01:41.397 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_sessions": 12, 2024-04-21T09:01:41.397 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_open": 12, 2024-04-21T09:01:41.397 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_close": 0, 2024-04-21T09:01:41.397 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_laggy": 0, 2024-04-21T09:01:41.397 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_wr": 638, 2024-04-21T09:01:41.397 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_rd": 14, 2024-04-21T09:01:41.397 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_del": 77 2024-04-21T09:01:41.397 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.398 INFO:teuthology.orchestra.run.smithi152.stdout: "oft": { 2024-04-21T09:01:41.398 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_objs": 1, 2024-04-21T09:01:41.398 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_kv_pairs": 0, 2024-04-21T09:01:41.398 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_updates": 0, 2024-04-21T09:01:41.398 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_removes": 0 2024-04-21T09:01:41.398 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.398 INFO:teuthology.orchestra.run.smithi152.stdout: "osd-slow-ops": { 2024-04-21T09:01:41.398 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_ops_count": 618 2024-04-21T09:01:41.398 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.399 INFO:teuthology.orchestra.run.smithi152.stdout: "purge_queue": { 2024-04-21T09:01:41.399 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops": 0, 2024-04-21T09:01:41.399 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops_high_water": 17, 2024-04-21T09:01:41.399 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing": 0, 2024-04-21T09:01:41.399 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_high_water": 11, 2024-04-21T09:01:41.399 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executed_ops": 909, 2024-04-21T09:01:41.399 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executed": 469, 2024-04-21T09:01:41.399 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_item_in_journal": 0 2024-04-21T09:01:41.399 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.399 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T09:01:41.400 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:41.400 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-04-21T09:01:41.400 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:41.400 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 15444, 2024-04-21T09:01:41.400 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 6520255, 2024-04-21T09:01:41.400 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:41.400 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 15444, 2024-04-21T09:01:41.400 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-04-21T09:01:41.400 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-04-21T09:01:41.400 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 15444, 2024-04-21T09:01:41.401 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 6520255, 2024-04-21T09:01:41.401 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:41.401 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.401 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.401 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.401 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:41.401 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.401 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_bytes": { 2024-04-21T09:01:41.401 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:41.402 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-04-21T09:01:41.402 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:41.402 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-04-21T09:01:41.402 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-04-21T09:01:41.402 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:41.402 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:41.402 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 7559, 2024-04-21T09:01:41.403 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 10052121, 2024-04-21T09:01:41.403 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 7559, 2024-04-21T09:01:41.403 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 10052121, 2024-04-21T09:01:41.403 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:41.403 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.403 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.403 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.403 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:41.404 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.404 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_ops": { 2024-04-21T09:01:41.404 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:41.404 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 1024, 2024-04-21T09:01:41.404 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:41.404 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-04-21T09:01:41.404 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-04-21T09:01:41.404 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:41.404 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:41.405 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 7559, 2024-04-21T09:01:41.405 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 7559, 2024-04-21T09:01:41.405 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 7559, 2024-04-21T09:01:41.405 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 7559, 2024-04-21T09:01:41.405 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:41.405 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.405 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.405 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.406 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:41.406 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.406 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle": { 2024-04-21T09:01:41.406 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:41.406 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-04-21T09:01:41.406 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:41.406 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 469, 2024-04-21T09:01:41.406 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 50889, 2024-04-21T09:01:41.406 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:41.407 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 469, 2024-04-21T09:01:41.407 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-04-21T09:01:41.407 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-04-21T09:01:41.407 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 11, 2024-04-21T09:01:41.407 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 50889, 2024-04-21T09:01:41.407 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:41.407 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.407 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.407 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.407 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:41.408 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:41.408 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle-0x55dc4775a500": { 2024-04-21T09:01:41.408 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:41.408 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-04-21T09:01:41.408 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:41.408 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 3321, 2024-04-21T09:01:41.408 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 8906340, 2024-04-21T09:01:41.408 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:41.408 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 3321, 2024-04-21T09:01:41.409 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-04-21T09:01:41.409 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-04-21T09:01:41.409 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 916, 2024-04-21T09:01:41.409 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 8906340, 2024-04-21T09:01:41.409 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:41.409 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:41.409 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:41.409 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:41.409 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:41.409 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:41.410 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-04-21T09:01:42.230 INFO:tasks.check_counter:Daemon mds.b mds.exported=10 2024-04-21T09:01:42.231 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-21T09:01:42.231 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-21T09:01:42.231 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-04-21T09:01:42.231 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-04-21T09:01:42.231 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=0 2024-04-21T09:01:42.231 DEBUG:tasks.check_counter:Getting stats from e 2024-04-21T09:01:42.232 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-21T09:01:42.471 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T09:01:42.593 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:42 smithi152 ceph-mon[31401]: pgmap v326: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T09:01:42.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:42 smithi028 ceph-mon[28003]: pgmap v326: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T09:01:42.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:42 smithi171 ceph-mon[31358]: pgmap v326: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T09:01:44.278 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:44 smithi152 ceph-mon[31401]: pgmap v327: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:44.278 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:44 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T09:01:44.314 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-04-21T09:01:44.315 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T09:01:44.315 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 344, 2024-04-21T09:01:44.315 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 346, 2024-04-21T09:01:44.315 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 12672287, 2024-04-21T09:01:44.315 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 109456, 2024-04-21T09:01:44.315 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 7, 2024-04-21T09:01:44.315 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 3, 2024-04-21T09:01:44.315 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.038328813, 2024-04-21T09:01:44.316 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.010842379, 2024-04-21T09:01:44.316 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.014724832, 2024-04-21T09:01:44.316 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.009328613, 2024-04-21T09:01:44.316 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:44.316 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 344, 2024-04-21T09:01:44.316 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.016255786, 2024-04-21T09:01:44.316 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000047255 2024-04-21T09:01:44.316 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.316 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:44.317 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.317 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.317 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.317 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.317 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T09:01:44.317 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T09:01:44.317 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.317 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T09:01:44.318 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 673, 2024-04-21T09:01:44.318 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 836, 2024-04-21T09:01:44.318 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 16949561, 2024-04-21T09:01:44.318 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 657879, 2024-04-21T09:01:44.318 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 8, 2024-04-21T09:01:44.318 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 6, 2024-04-21T09:01:44.318 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.087734780, 2024-04-21T09:01:44.318 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.023644255, 2024-04-21T09:01:44.319 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.031485008, 2024-04-21T09:01:44.319 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.024889449, 2024-04-21T09:01:44.319 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:44.319 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 833, 2024-04-21T09:01:44.319 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.046935229, 2024-04-21T09:01:44.319 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000056344 2024-04-21T09:01:44.319 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.319 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:44.319 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.320 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.320 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.320 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.320 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T09:01:44.320 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 444496 2024-04-21T09:01:44.320 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.320 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T09:01:44.320 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 1445, 2024-04-21T09:01:44.320 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 1428, 2024-04-21T09:01:44.320 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 17967943, 2024-04-21T09:01:44.321 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 515168, 2024-04-21T09:01:44.321 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 5, 2024-04-21T09:01:44.321 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 4, 2024-04-21T09:01:44.321 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.186126301, 2024-04-21T09:01:44.321 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.050005420, 2024-04-21T09:01:44.321 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.085145973, 2024-04-21T09:01:44.321 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.049757196, 2024-04-21T09:01:44.321 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:44.321 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 1428, 2024-04-21T09:01:44.322 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.079192348, 2024-04-21T09:01:44.322 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000055456 2024-04-21T09:01:44.322 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.322 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:44.322 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.322 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.322 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.322 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.323 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 1635632, 2024-04-21T09:01:44.323 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 136112 2024-04-21T09:01:44.323 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.323 INFO:teuthology.orchestra.run.smithi152.stdout: "cct": { 2024-04-21T09:01:44.323 INFO:teuthology.orchestra.run.smithi152.stdout: "total_workers": 1, 2024-04-21T09:01:44.323 INFO:teuthology.orchestra.run.smithi152.stdout: "unhealthy_workers": 0 2024-04-21T09:01:44.323 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.323 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-MDSRank": { 2024-04-21T09:01:44.323 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-04-21T09:01:44.323 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-04-21T09:01:44.324 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 3699, 2024-04-21T09:01:44.324 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.109089853, 2024-04-21T09:01:44.324 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000029491 2024-04-21T09:01:44.324 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:44.324 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.324 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-PurgeQueue": { 2024-04-21T09:01:44.324 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-04-21T09:01:44.324 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-04-21T09:01:44.324 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.325 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.325 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.325 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:44.325 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.325 INFO:teuthology.orchestra.run.smithi152.stdout: "mds": { 2024-04-21T09:01:44.325 INFO:teuthology.orchestra.run.smithi152.stdout: "request": 0, 2024-04-21T09:01:44.325 INFO:teuthology.orchestra.run.smithi152.stdout: "reply": 0, 2024-04-21T09:01:44.325 INFO:teuthology.orchestra.run.smithi152.stdout: "reply_latency": { 2024-04-21T09:01:44.325 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.326 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.326 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.326 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.326 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_reply": 0, 2024-04-21T09:01:44.326 INFO:teuthology.orchestra.run.smithi152.stdout: "forward": 0, 2024-04-21T09:01:44.326 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_complete": 0, 2024-04-21T09:01:44.326 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_keys": 0, 2024-04-21T09:01:44.326 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_commit": 0, 2024-04-21T09:01:44.326 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_split": 0, 2024-04-21T09:01:44.327 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_merge": 0, 2024-04-21T09:01:44.327 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes": 584, 2024-04-21T09:01:44.327 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_top": 0, 2024-04-21T09:01:44.327 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_bottom": 584, 2024-04-21T09:01:44.327 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pin_tail": 0, 2024-04-21T09:01:44.327 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pinned": 584, 2024-04-21T09:01:44.327 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_expired": 0, 2024-04-21T09:01:44.327 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_with_caps": 0, 2024-04-21T09:01:44.328 INFO:teuthology.orchestra.run.smithi152.stdout: "caps": 0, 2024-04-21T09:01:44.328 INFO:teuthology.orchestra.run.smithi152.stdout: "subtrees": 8, 2024-04-21T09:01:44.328 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse": 0, 2024-04-21T09:01:44.328 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_hit": 0, 2024-04-21T09:01:44.328 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_forward": 0, 2024-04-21T09:01:44.328 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_discover": 0, 2024-04-21T09:01:44.328 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_dir_fetch": 0, 2024-04-21T09:01:44.329 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_remote_ino": 0, 2024-04-21T09:01:44.329 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_lock": 0, 2024-04-21T09:01:44.329 INFO:teuthology.orchestra.run.smithi152.stdout: "load_cent": 0, 2024-04-21T09:01:44.329 INFO:teuthology.orchestra.run.smithi152.stdout: "q": 0, 2024-04-21T09:01:44.329 INFO:teuthology.orchestra.run.smithi152.stdout: "exported": 0, 2024-04-21T09:01:44.329 INFO:teuthology.orchestra.run.smithi152.stdout: "exported_inodes": 0, 2024-04-21T09:01:44.329 INFO:teuthology.orchestra.run.smithi152.stdout: "imported": 0, 2024-04-21T09:01:44.329 INFO:teuthology.orchestra.run.smithi152.stdout: "imported_inodes": 0, 2024-04-21T09:01:44.329 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_dir_fetch": 0, 2024-04-21T09:01:44.330 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_backtrace_fetch": 0, 2024-04-21T09:01:44.330 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_peer_discover": 0, 2024-04-21T09:01:44.330 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rfiles": 0, 2024-04-21T09:01:44.330 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rbytes": 0, 2024-04-21T09:01:44.330 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rsnaps": 0, 2024-04-21T09:01:44.330 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_fetch": 0, 2024-04-21T09:01:44.330 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_set_tag": 0, 2024-04-21T09:01:44.330 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T09:01:44.331 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_inotable_repaired": 0, 2024-04-21T09:01:44.331 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_inodes": 0, 2024-04-21T09:01:44.331 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T09:01:44.331 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dirfrag_rstats": 0, 2024-04-21T09:01:44.331 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_file_inodes": 0, 2024-04-21T09:01:44.331 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_inode_file_caps": 0, 2024-04-21T09:01:44.331 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_revoke": 0, 2024-04-21T09:01:44.331 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_grant": 0, 2024-04-21T09:01:44.332 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T09:01:44.332 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T09:01:44.332 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T09:01:44.332 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps": 0, 2024-04-21T09:01:44.332 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps_dirty": 0, 2024-04-21T09:01:44.332 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_cap_release": 0, 2024-04-21T09:01:44.332 INFO:teuthology.orchestra.run.smithi152.stdout: "process_request_cap_release": 0 2024-04-21T09:01:44.332 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.332 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_cache": { 2024-04-21T09:01:44.333 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update": 0, 2024-04-21T09:01:44.333 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update_receipt": 0, 2024-04-21T09:01:44.333 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_try_discover": 0, 2024-04-21T09:01:44.333 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_send_discover": 0, 2024-04-21T09:01:44.333 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_handle_discover": 0, 2024-04-21T09:01:44.333 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays": 0, 2024-04-21T09:01:44.333 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_delayed": 0, 2024-04-21T09:01:44.333 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_enqueuing": 0, 2024-04-21T09:01:44.334 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_created": 0, 2024-04-21T09:01:44.334 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_enqueued": 0, 2024-04-21T09:01:44.334 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_reintegrated": 0, 2024-04-21T09:01:44.334 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_migrated": 0, 2024-04-21T09:01:44.334 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_processing": 0, 2024-04-21T09:01:44.334 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_enqueued": 0, 2024-04-21T09:01:44.334 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_prioritized": 0, 2024-04-21T09:01:44.334 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_started": 0, 2024-04-21T09:01:44.335 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_completed": 0, 2024-04-21T09:01:44.335 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_quiesce_path": 0, 2024-04-21T09:01:44.335 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_quiesce_inode": 0, 2024-04-21T09:01:44.335 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T09:01:44.335 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_exportdir": 0, 2024-04-21T09:01:44.335 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_flush": 0, 2024-04-21T09:01:44.335 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragmentdir": 0, 2024-04-21T09:01:44.335 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragstats": 0, 2024-04-21T09:01:44.335 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_inodestats": 0 2024-04-21T09:01:44.336 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.336 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_log": { 2024-04-21T09:01:44.336 INFO:teuthology.orchestra.run.smithi152.stdout: "evlrg": 0, 2024-04-21T09:01:44.336 INFO:teuthology.orchestra.run.smithi152.stdout: "evadd": 0, 2024-04-21T09:01:44.336 INFO:teuthology.orchestra.run.smithi152.stdout: "evex": 0, 2024-04-21T09:01:44.336 INFO:teuthology.orchestra.run.smithi152.stdout: "evtrm": 0, 2024-04-21T09:01:44.336 INFO:teuthology.orchestra.run.smithi152.stdout: "ev": 17464, 2024-04-21T09:01:44.336 INFO:teuthology.orchestra.run.smithi152.stdout: "evexg": 0, 2024-04-21T09:01:44.337 INFO:teuthology.orchestra.run.smithi152.stdout: "evexd": 0, 2024-04-21T09:01:44.337 INFO:teuthology.orchestra.run.smithi152.stdout: "segadd": 0, 2024-04-21T09:01:44.337 INFO:teuthology.orchestra.run.smithi152.stdout: "segex": 0, 2024-04-21T09:01:44.337 INFO:teuthology.orchestra.run.smithi152.stdout: "segtrm": 0, 2024-04-21T09:01:44.337 INFO:teuthology.orchestra.run.smithi152.stdout: "seg": 22, 2024-04-21T09:01:44.337 INFO:teuthology.orchestra.run.smithi152.stdout: "segmjr": 3, 2024-04-21T09:01:44.337 INFO:teuthology.orchestra.run.smithi152.stdout: "segexg": 0, 2024-04-21T09:01:44.337 INFO:teuthology.orchestra.run.smithi152.stdout: "segexd": 0, 2024-04-21T09:01:44.337 INFO:teuthology.orchestra.run.smithi152.stdout: "expos": 4194304, 2024-04-21T09:01:44.338 INFO:teuthology.orchestra.run.smithi152.stdout: "wrpos": 49519592, 2024-04-21T09:01:44.338 INFO:teuthology.orchestra.run.smithi152.stdout: "rdpos": 49519018, 2024-04-21T09:01:44.338 INFO:teuthology.orchestra.run.smithi152.stdout: "jlat": { 2024-04-21T09:01:44.338 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.338 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.338 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.338 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.338 INFO:teuthology.orchestra.run.smithi152.stdout: "replayed": 17464 2024-04-21T09:01:44.339 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.339 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_mem": { 2024-04-21T09:01:44.339 INFO:teuthology.orchestra.run.smithi152.stdout: "ino": 583, 2024-04-21T09:01:44.339 INFO:teuthology.orchestra.run.smithi152.stdout: "ino+": 583, 2024-04-21T09:01:44.339 INFO:teuthology.orchestra.run.smithi152.stdout: "ino-": 0, 2024-04-21T09:01:44.339 INFO:teuthology.orchestra.run.smithi152.stdout: "dir": 36, 2024-04-21T09:01:44.339 INFO:teuthology.orchestra.run.smithi152.stdout: "dir+": 38, 2024-04-21T09:01:44.339 INFO:teuthology.orchestra.run.smithi152.stdout: "dir-": 2, 2024-04-21T09:01:44.339 INFO:teuthology.orchestra.run.smithi152.stdout: "dn": 584, 2024-04-21T09:01:44.340 INFO:teuthology.orchestra.run.smithi152.stdout: "dn+": 584, 2024-04-21T09:01:44.340 INFO:teuthology.orchestra.run.smithi152.stdout: "dn-": 0, 2024-04-21T09:01:44.340 INFO:teuthology.orchestra.run.smithi152.stdout: "cap": 0, 2024-04-21T09:01:44.340 INFO:teuthology.orchestra.run.smithi152.stdout: "cap+": 0, 2024-04-21T09:01:44.340 INFO:teuthology.orchestra.run.smithi152.stdout: "cap-": 0, 2024-04-21T09:01:44.340 INFO:teuthology.orchestra.run.smithi152.stdout: "rss": 54816, 2024-04-21T09:01:44.340 INFO:teuthology.orchestra.run.smithi152.stdout: "heap": 215416 2024-04-21T09:01:44.340 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.340 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_server": { 2024-04-21T09:01:44.341 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_client_request": 0, 2024-04-21T09:01:44.341 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_server_request": 0, 2024-04-21T09:01:44.341 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_request": 0, 2024-04-21T09:01:44.341 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_session": 0, 2024-04-21T09:01:44.341 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_peer_request": 0, 2024-04-21T09:01:44.341 INFO:teuthology.orchestra.run.smithi152.stdout: "req_create_latency": { 2024-04-21T09:01:44.341 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.341 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.342 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.342 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.342 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getattr_latency": { 2024-04-21T09:01:44.342 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.342 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.342 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.342 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.342 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getfilelock_latency": { 2024-04-21T09:01:44.343 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.343 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.343 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.343 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.343 INFO:teuthology.orchestra.run.smithi152.stdout: "req_link_latency": { 2024-04-21T09:01:44.343 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.343 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.343 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.343 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.344 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookup_latency": { 2024-04-21T09:01:44.344 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.344 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.344 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.344 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.344 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookuphash_latency": { 2024-04-21T09:01:44.344 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.344 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.344 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.345 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.345 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupino_latency": { 2024-04-21T09:01:44.345 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.345 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.345 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.345 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.345 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupname_latency": { 2024-04-21T09:01:44.345 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.346 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.346 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.346 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.346 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupparent_latency": { 2024-04-21T09:01:44.346 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.346 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.346 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.346 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.346 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupsnap_latency": { 2024-04-21T09:01:44.347 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.347 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.347 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.347 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.347 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lssnap_latency": { 2024-04-21T09:01:44.347 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.347 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.347 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.347 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.348 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mkdir_latency": { 2024-04-21T09:01:44.348 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.349 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.349 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.349 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.349 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mknod_latency": { 2024-04-21T09:01:44.349 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.349 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.349 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.349 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.349 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mksnap_latency": { 2024-04-21T09:01:44.350 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.350 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.350 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.350 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.350 INFO:teuthology.orchestra.run.smithi152.stdout: "req_open_latency": { 2024-04-21T09:01:44.350 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.350 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.350 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.351 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.351 INFO:teuthology.orchestra.run.smithi152.stdout: "req_readdir_latency": { 2024-04-21T09:01:44.351 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.351 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.351 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.351 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.351 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rename_latency": { 2024-04-21T09:01:44.351 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.351 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.352 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.352 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.352 INFO:teuthology.orchestra.run.smithi152.stdout: "req_renamesnap_latency": { 2024-04-21T09:01:44.352 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.352 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.352 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.352 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.352 INFO:teuthology.orchestra.run.smithi152.stdout: "req_snapdiff_latency": { 2024-04-21T09:01:44.353 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.353 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.353 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.353 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.353 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmdir_latency": { 2024-04-21T09:01:44.353 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.353 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.353 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.353 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.354 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmsnap_latency": { 2024-04-21T09:01:44.354 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.354 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.354 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.354 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.354 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmxattr_latency": { 2024-04-21T09:01:44.354 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.354 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.355 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.355 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.355 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setattr_latency": { 2024-04-21T09:01:44.355 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.355 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.355 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.355 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.355 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setdirlayout_latency": { 2024-04-21T09:01:44.355 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.356 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.356 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.356 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.356 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setfilelock_latency": { 2024-04-21T09:01:44.356 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.356 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.356 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.356 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.357 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setlayout_latency": { 2024-04-21T09:01:44.357 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.357 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.357 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.357 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.357 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setxattr_latency": { 2024-04-21T09:01:44.357 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.357 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.358 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.358 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.358 INFO:teuthology.orchestra.run.smithi152.stdout: "req_symlink_latency": { 2024-04-21T09:01:44.358 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.358 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.358 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.358 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.358 INFO:teuthology.orchestra.run.smithi152.stdout: "req_unlink_latency": { 2024-04-21T09:01:44.358 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.359 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.359 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.359 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.359 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_revoke_eviction": 0, 2024-04-21T09:01:44.359 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_acquisition_throttle": 0, 2024-04-21T09:01:44.359 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getvxattr_latency": { 2024-04-21T09:01:44.359 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.359 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.360 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.360 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:44.360 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.360 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_sessions": { 2024-04-21T09:01:44.360 INFO:teuthology.orchestra.run.smithi152.stdout: "session_count": 2, 2024-04-21T09:01:44.360 INFO:teuthology.orchestra.run.smithi152.stdout: "session_add": 5, 2024-04-21T09:01:44.360 INFO:teuthology.orchestra.run.smithi152.stdout: "session_remove": 3, 2024-04-21T09:01:44.360 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_open": 4, 2024-04-21T09:01:44.360 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_stale": 0, 2024-04-21T09:01:44.361 INFO:teuthology.orchestra.run.smithi152.stdout: "total_load": 0, 2024-04-21T09:01:44.361 INFO:teuthology.orchestra.run.smithi152.stdout: "average_load": 0, 2024-04-21T09:01:44.361 INFO:teuthology.orchestra.run.smithi152.stdout: "avg_session_uptime": 2976, 2024-04-21T09:01:44.361 INFO:teuthology.orchestra.run.smithi152.stdout: "mdthresh_evicted": 0 2024-04-21T09:01:44.361 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.361 INFO:teuthology.orchestra.run.smithi152.stdout: "mempool": { 2024-04-21T09:01:44.361 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_bytes": 0, 2024-04-21T09:01:44.361 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_items": 0, 2024-04-21T09:01:44.362 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T09:01:44.362 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_items": 0, 2024-04-21T09:01:44.362 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T09:01:44.362 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_items": 0, 2024-04-21T09:01:44.362 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T09:01:44.362 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T09:01:44.362 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T09:01:44.362 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T09:01:44.362 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T09:01:44.363 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_items": 0, 2024-04-21T09:01:44.363 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T09:01:44.363 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T09:01:44.363 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_bytes": 0, 2024-04-21T09:01:44.363 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_items": 0, 2024-04-21T09:01:44.363 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_bytes": 0, 2024-04-21T09:01:44.363 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_items": 0, 2024-04-21T09:01:44.363 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T09:01:44.364 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T09:01:44.364 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T09:01:44.364 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T09:01:44.364 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T09:01:44.364 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_items": 0, 2024-04-21T09:01:44.364 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_bytes": 0, 2024-04-21T09:01:44.364 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_items": 0, 2024-04-21T09:01:44.364 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T09:01:44.365 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T09:01:44.365 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_bytes": 0, 2024-04-21T09:01:44.365 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_items": 0, 2024-04-21T09:01:44.365 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_bytes": 0, 2024-04-21T09:01:44.365 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_items": 0, 2024-04-21T09:01:44.365 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T09:01:44.365 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_items": 0, 2024-04-21T09:01:44.365 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T09:01:44.365 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_items": 0, 2024-04-21T09:01:44.366 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_bytes": 244909, 2024-04-21T09:01:44.366 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_items": 656, 2024-04-21T09:01:44.366 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_bytes": 0, 2024-04-21T09:01:44.366 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_items": 0, 2024-04-21T09:01:44.366 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_bytes": 0, 2024-04-21T09:01:44.366 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_items": 0, 2024-04-21T09:01:44.366 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_bytes": 0, 2024-04-21T09:01:44.366 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_items": 0, 2024-04-21T09:01:44.367 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_bytes": 0, 2024-04-21T09:01:44.367 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_items": 0, 2024-04-21T09:01:44.367 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_bytes": 9616, 2024-04-21T09:01:44.367 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_items": 185, 2024-04-21T09:01:44.367 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T09:01:44.367 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_items": 0, 2024-04-21T09:01:44.367 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_bytes": 0, 2024-04-21T09:01:44.367 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_items": 0, 2024-04-21T09:01:44.367 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_bytes": 1820490, 2024-04-21T09:01:44.367 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_items": 16787, 2024-04-21T09:01:44.368 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_bytes": 0, 2024-04-21T09:01:44.368 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_items": 0, 2024-04-21T09:01:44.368 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_bytes": 0, 2024-04-21T09:01:44.368 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_items": 0 2024-04-21T09:01:44.368 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.368 INFO:teuthology.orchestra.run.smithi152.stdout: "objecter": { 2024-04-21T09:01:44.368 INFO:teuthology.orchestra.run.smithi152.stdout: "op_active": 0, 2024-04-21T09:01:44.368 INFO:teuthology.orchestra.run.smithi152.stdout: "op_laggy": 0, 2024-04-21T09:01:44.368 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send": 2207, 2024-04-21T09:01:44.369 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send_bytes": 21, 2024-04-21T09:01:44.369 INFO:teuthology.orchestra.run.smithi152.stdout: "op_resend": 3, 2024-04-21T09:01:44.369 INFO:teuthology.orchestra.run.smithi152.stdout: "op_reply": 2207, 2024-04-21T09:01:44.369 INFO:teuthology.orchestra.run.smithi152.stdout: "op_latency": { 2024-04-21T09:01:44.369 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 2207, 2024-04-21T09:01:44.369 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 43.025510935, 2024-04-21T09:01:44.369 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.019495020 2024-04-21T09:01:44.369 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.369 INFO:teuthology.orchestra.run.smithi152.stdout: "op_inflight": 0, 2024-04-21T09:01:44.370 INFO:teuthology.orchestra.run.smithi152.stdout: "oplen_avg": { 2024-04-21T09:01:44.370 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 2207, 2024-04-21T09:01:44.370 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 2208 2024-04-21T09:01:44.370 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.370 INFO:teuthology.orchestra.run.smithi152.stdout: "op": 2207, 2024-04-21T09:01:44.370 INFO:teuthology.orchestra.run.smithi152.stdout: "op_r": 2207, 2024-04-21T09:01:44.370 INFO:teuthology.orchestra.run.smithi152.stdout: "op_w": 0, 2024-04-21T09:01:44.370 INFO:teuthology.orchestra.run.smithi152.stdout: "op_rmw": 0, 2024-04-21T09:01:44.370 INFO:teuthology.orchestra.run.smithi152.stdout: "op_pg": 0, 2024-04-21T09:01:44.371 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_stat": 1371, 2024-04-21T09:01:44.371 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_create": 0, 2024-04-21T09:01:44.371 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_read": 834, 2024-04-21T09:01:44.371 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_write": 0, 2024-04-21T09:01:44.371 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writefull": 0, 2024-04-21T09:01:44.371 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writesame": 0, 2024-04-21T09:01:44.371 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_append": 0, 2024-04-21T09:01:44.371 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_zero": 0, 2024-04-21T09:01:44.372 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_truncate": 0, 2024-04-21T09:01:44.372 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_delete": 0, 2024-04-21T09:01:44.372 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_mapext": 0, 2024-04-21T09:01:44.372 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_sparse_read": 0, 2024-04-21T09:01:44.372 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_clonerange": 0, 2024-04-21T09:01:44.372 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_getxattr": 1, 2024-04-21T09:01:44.372 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_setxattr": 0, 2024-04-21T09:01:44.372 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_cmpxattr": 0, 2024-04-21T09:01:44.372 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_rmxattr": 0, 2024-04-21T09:01:44.372 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_resetxattrs": 0, 2024-04-21T09:01:44.373 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_call": 0, 2024-04-21T09:01:44.373 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_watch": 0, 2024-04-21T09:01:44.373 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_notify": 0, 2024-04-21T09:01:44.373 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T09:01:44.373 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls": 0, 2024-04-21T09:01:44.373 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls_filter": 0, 2024-04-21T09:01:44.373 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_other": 0, 2024-04-21T09:01:44.373 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_active": 0, 2024-04-21T09:01:44.374 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_send": 0, 2024-04-21T09:01:44.374 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_resend": 0, 2024-04-21T09:01:44.374 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_ping": 0, 2024-04-21T09:01:44.374 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_active": 0, 2024-04-21T09:01:44.374 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_send": 0, 2024-04-21T09:01:44.374 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_resend": 0, 2024-04-21T09:01:44.374 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_active": 0, 2024-04-21T09:01:44.374 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_send": 0, 2024-04-21T09:01:44.374 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_resend": 0, 2024-04-21T09:01:44.375 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_active": 0, 2024-04-21T09:01:44.375 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_send": 0, 2024-04-21T09:01:44.375 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_resend": 0, 2024-04-21T09:01:44.375 INFO:teuthology.orchestra.run.smithi152.stdout: "command_active": 0, 2024-04-21T09:01:44.375 INFO:teuthology.orchestra.run.smithi152.stdout: "command_send": 0, 2024-04-21T09:01:44.375 INFO:teuthology.orchestra.run.smithi152.stdout: "command_resend": 0, 2024-04-21T09:01:44.375 INFO:teuthology.orchestra.run.smithi152.stdout: "map_epoch": 102, 2024-04-21T09:01:44.375 INFO:teuthology.orchestra.run.smithi152.stdout: "map_full": 0, 2024-04-21T09:01:44.376 INFO:teuthology.orchestra.run.smithi152.stdout: "map_inc": 13, 2024-04-21T09:01:44.376 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_sessions": 9, 2024-04-21T09:01:44.376 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_open": 9, 2024-04-21T09:01:44.376 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_close": 0, 2024-04-21T09:01:44.376 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_laggy": 0, 2024-04-21T09:01:44.376 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_wr": 0, 2024-04-21T09:01:44.376 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_rd": 2, 2024-04-21T09:01:44.376 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_del": 0 2024-04-21T09:01:44.376 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.377 INFO:teuthology.orchestra.run.smithi152.stdout: "oft": { 2024-04-21T09:01:44.377 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_objs": 0, 2024-04-21T09:01:44.377 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_kv_pairs": 0, 2024-04-21T09:01:44.377 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_updates": 0, 2024-04-21T09:01:44.377 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_removes": 0 2024-04-21T09:01:44.377 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.377 INFO:teuthology.orchestra.run.smithi152.stdout: "osd-slow-ops": { 2024-04-21T09:01:44.377 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_ops_count": 0 2024-04-21T09:01:44.378 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.378 INFO:teuthology.orchestra.run.smithi152.stdout: "purge_queue": { 2024-04-21T09:01:44.378 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops": 0, 2024-04-21T09:01:44.378 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T09:01:44.378 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing": 0, 2024-04-21T09:01:44.378 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_high_water": 0, 2024-04-21T09:01:44.378 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executed_ops": 0, 2024-04-21T09:01:44.378 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executed": 0, 2024-04-21T09:01:44.378 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_item_in_journal": 0 2024-04-21T09:01:44.379 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.379 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T09:01:44.379 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:44.379 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-04-21T09:01:44.379 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:44.379 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 2458, 2024-04-21T09:01:44.379 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 47352809, 2024-04-21T09:01:44.379 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:44.380 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 2458, 2024-04-21T09:01:44.380 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-04-21T09:01:44.380 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-04-21T09:01:44.380 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 2458, 2024-04-21T09:01:44.380 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 47352809, 2024-04-21T09:01:44.380 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:44.380 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.380 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.380 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.381 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:44.381 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.381 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_bytes": { 2024-04-21T09:01:44.381 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:44.381 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-04-21T09:01:44.381 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:44.381 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-04-21T09:01:44.381 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-04-21T09:01:44.381 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:44.382 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:44.382 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 2207, 2024-04-21T09:01:44.382 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 45333459, 2024-04-21T09:01:44.382 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 2207, 2024-04-21T09:01:44.382 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 45333459, 2024-04-21T09:01:44.382 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:44.382 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.382 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.382 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.383 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:44.383 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.383 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_ops": { 2024-04-21T09:01:44.383 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:44.383 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 1024, 2024-04-21T09:01:44.383 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:44.383 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-04-21T09:01:44.383 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-04-21T09:01:44.383 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:44.384 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:44.384 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 2207, 2024-04-21T09:01:44.384 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 2207, 2024-04-21T09:01:44.384 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 2207, 2024-04-21T09:01:44.384 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 2207, 2024-04-21T09:01:44.384 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:44.384 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.384 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.384 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.385 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:44.385 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.385 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle": { 2024-04-21T09:01:44.385 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:44.385 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-04-21T09:01:44.385 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:44.385 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-04-21T09:01:44.385 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-04-21T09:01:44.385 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:44.385 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:44.386 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-04-21T09:01:44.386 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-04-21T09:01:44.386 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 0, 2024-04-21T09:01:44.386 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 0, 2024-04-21T09:01:44.386 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:44.386 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.386 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.386 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.387 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:44.387 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:44.387 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle-0x55a576ba26e0": { 2024-04-21T09:01:44.387 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:44.387 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-04-21T09:01:44.387 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:44.387 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-04-21T09:01:44.387 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-04-21T09:01:44.387 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:44.388 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:44.388 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-04-21T09:01:44.388 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-04-21T09:01:44.388 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 0, 2024-04-21T09:01:44.388 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 0, 2024-04-21T09:01:44.388 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:44.388 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:44.388 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:44.388 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:44.388 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:44.389 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:44.389 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-04-21T09:01:44.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:44 smithi028 ceph-mon[28003]: pgmap v327: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:44.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:44 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T09:01:44.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:44 smithi171 ceph-mon[31358]: pgmap v327: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:44.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:44 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T09:01:45.288 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-21T09:01:45.288 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-21T09:01:45.288 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-21T09:01:45.288 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-21T09:01:45.289 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T09:01:45.289 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-04-21T09:01:45.289 DEBUG:tasks.check_counter:Getting stats from h 2024-04-21T09:01:45.289 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-21T09:01:46.011 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T09:01:46.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:46 smithi152 ceph-mon[31401]: pgmap v328: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T09:01:46.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:46 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:46.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:46 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:46.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:46 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:46.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:46 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:46.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:46 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:46 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:46 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:46 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:46 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:46 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:46 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.356 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:46 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:46 smithi028 ceph-mon[28003]: pgmap v328: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T09:01:46.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:46 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:46.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:46 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:46.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:46 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:46.357 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:46 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:46.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:46 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:46 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:46 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:46 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:46 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:46 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:46 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.359 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:46 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:46 smithi171 ceph-mon[31358]: pgmap v328: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T09:01:46.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:46 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:46.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:46 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:46.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:46 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:46.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:46 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:46.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:46 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:46 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:46 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:46 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:46 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:46 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:46 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:46.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:46 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:47.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:47 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-04-21T09:01:47.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:47 smithi152 ceph-mon[31401]: Unable to set osd_memory_target on smithi028 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-04-21T09:01:47.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:47 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-04-21T09:01:47.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:47 smithi152 ceph-mon[31401]: Unable to set osd_memory_target on smithi171 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-21T09:01:47.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:47 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-04-21T09:01:47.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:47 smithi028 ceph-mon[28003]: Unable to set osd_memory_target on smithi028 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-04-21T09:01:47.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:47 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-04-21T09:01:47.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:47 smithi028 ceph-mon[28003]: Unable to set osd_memory_target on smithi171 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-21T09:01:47.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:47 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-04-21T09:01:47.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:47 smithi171 ceph-mon[31358]: Unable to set osd_memory_target on smithi028 to 185894912: error parsing value: Value '185894912' is below minimum 939524096 2024-04-21T09:01:47.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:47 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi171 to 176.2M 2024-04-21T09:01:47.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:47 smithi171 ceph-mon[31358]: Unable to set osd_memory_target on smithi171 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-21T09:01:47.815 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-04-21T09:01:47.815 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T09:01:47.815 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 2031, 2024-04-21T09:01:47.815 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 2033, 2024-04-21T09:01:47.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 629952, 2024-04-21T09:01:47.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 645865, 2024-04-21T09:01:47.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 6, 2024-04-21T09:01:47.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 2, 2024-04-21T09:01:47.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.161235286, 2024-04-21T09:01:47.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.049114813, 2024-04-21T09:01:47.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.040615949, 2024-04-21T09:01:47.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.059026111, 2024-04-21T09:01:47.817 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:47.817 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 2031, 2024-04-21T09:01:47.817 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.064599201, 2024-04-21T09:01:47.817 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000031806 2024-04-21T09:01:47.817 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.817 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:47.817 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.817 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.818 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.818 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T09:01:47.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T09:01:47.818 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.818 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T09:01:47.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 254, 2024-04-21T09:01:47.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 237, 2024-04-21T09:01:47.819 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 1635916, 2024-04-21T09:01:47.819 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 136170, 2024-04-21T09:01:47.819 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 3, 2024-04-21T09:01:47.819 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 2, 2024-04-21T09:01:47.819 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.046404264, 2024-04-21T09:01:47.819 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.013782650, 2024-04-21T09:01:47.819 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.037674067, 2024-04-21T09:01:47.819 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.005509762, 2024-04-21T09:01:47.820 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:47.820 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 237, 2024-04-21T09:01:47.820 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.012152543, 2024-04-21T09:01:47.820 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000051276 2024-04-21T09:01:47.820 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.820 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:47.820 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.820 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.821 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.821 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.821 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 1635360, 2024-04-21T09:01:47.821 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 135472 2024-04-21T09:01:47.821 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.821 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T09:01:47.821 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 3, 2024-04-21T09:01:47.821 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 165, 2024-04-21T09:01:47.822 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 588, 2024-04-21T09:01:47.822 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 446254, 2024-04-21T09:01:47.822 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 6, 2024-04-21T09:01:47.822 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 4, 2024-04-21T09:01:47.822 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.009570293, 2024-04-21T09:01:47.822 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.006092927, 2024-04-21T09:01:47.822 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.001733151, 2024-04-21T09:01:47.822 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.000016735, 2024-04-21T09:01:47.823 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:47.823 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 162, 2024-04-21T09:01:47.823 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.016202796, 2024-04-21T09:01:47.823 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000100017 2024-04-21T09:01:47.823 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.823 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:47.823 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.823 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.824 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.824 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.824 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T09:01:47.824 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 445936 2024-04-21T09:01:47.824 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.824 INFO:teuthology.orchestra.run.smithi152.stdout: "cct": { 2024-04-21T09:01:47.824 INFO:teuthology.orchestra.run.smithi152.stdout: "total_workers": 1, 2024-04-21T09:01:47.824 INFO:teuthology.orchestra.run.smithi152.stdout: "unhealthy_workers": 0 2024-04-21T09:01:47.824 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.825 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-MDSRank": { 2024-04-21T09:01:47.825 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-04-21T09:01:47.825 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-04-21T09:01:47.825 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 3417, 2024-04-21T09:01:47.825 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.089143834, 2024-04-21T09:01:47.825 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000026088 2024-04-21T09:01:47.825 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:47.825 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.826 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-PurgeQueue": { 2024-04-21T09:01:47.826 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-04-21T09:01:47.826 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-04-21T09:01:47.826 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.826 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.826 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.826 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:47.826 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.827 INFO:teuthology.orchestra.run.smithi152.stdout: "mds": { 2024-04-21T09:01:47.827 INFO:teuthology.orchestra.run.smithi152.stdout: "request": 0, 2024-04-21T09:01:47.827 INFO:teuthology.orchestra.run.smithi152.stdout: "reply": 0, 2024-04-21T09:01:47.827 INFO:teuthology.orchestra.run.smithi152.stdout: "reply_latency": { 2024-04-21T09:01:47.827 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.827 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.827 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.828 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.828 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_reply": 0, 2024-04-21T09:01:47.828 INFO:teuthology.orchestra.run.smithi152.stdout: "forward": 0, 2024-04-21T09:01:47.828 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_complete": 0, 2024-04-21T09:01:47.828 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_keys": 0, 2024-04-21T09:01:47.828 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_commit": 0, 2024-04-21T09:01:47.828 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_split": 0, 2024-04-21T09:01:47.828 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_merge": 0, 2024-04-21T09:01:47.829 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes": 21, 2024-04-21T09:01:47.829 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_top": 0, 2024-04-21T09:01:47.829 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_bottom": 21, 2024-04-21T09:01:47.829 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pin_tail": 0, 2024-04-21T09:01:47.829 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pinned": 21, 2024-04-21T09:01:47.829 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_expired": 0, 2024-04-21T09:01:47.829 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_with_caps": 0, 2024-04-21T09:01:47.829 INFO:teuthology.orchestra.run.smithi152.stdout: "caps": 0, 2024-04-21T09:01:47.829 INFO:teuthology.orchestra.run.smithi152.stdout: "subtrees": 6, 2024-04-21T09:01:47.830 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse": 0, 2024-04-21T09:01:47.830 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_hit": 0, 2024-04-21T09:01:47.830 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_forward": 0, 2024-04-21T09:01:47.830 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_discover": 0, 2024-04-21T09:01:47.830 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_dir_fetch": 0, 2024-04-21T09:01:47.830 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_remote_ino": 0, 2024-04-21T09:01:47.830 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_lock": 0, 2024-04-21T09:01:47.831 INFO:teuthology.orchestra.run.smithi152.stdout: "load_cent": 0, 2024-04-21T09:01:47.831 INFO:teuthology.orchestra.run.smithi152.stdout: "q": 0, 2024-04-21T09:01:47.831 INFO:teuthology.orchestra.run.smithi152.stdout: "exported": 0, 2024-04-21T09:01:47.831 INFO:teuthology.orchestra.run.smithi152.stdout: "exported_inodes": 0, 2024-04-21T09:01:47.831 INFO:teuthology.orchestra.run.smithi152.stdout: "imported": 0, 2024-04-21T09:01:47.831 INFO:teuthology.orchestra.run.smithi152.stdout: "imported_inodes": 0, 2024-04-21T09:01:47.831 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_dir_fetch": 0, 2024-04-21T09:01:47.831 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_backtrace_fetch": 0, 2024-04-21T09:01:47.832 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_peer_discover": 0, 2024-04-21T09:01:47.832 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rfiles": 0, 2024-04-21T09:01:47.832 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rbytes": 0, 2024-04-21T09:01:47.832 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rsnaps": 0, 2024-04-21T09:01:47.832 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_fetch": 0, 2024-04-21T09:01:47.832 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_set_tag": 0, 2024-04-21T09:01:47.832 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T09:01:47.832 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_inotable_repaired": 0, 2024-04-21T09:01:47.833 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_inodes": 0, 2024-04-21T09:01:47.833 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T09:01:47.833 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dirfrag_rstats": 0, 2024-04-21T09:01:47.833 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_file_inodes": 0, 2024-04-21T09:01:47.833 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_inode_file_caps": 0, 2024-04-21T09:01:47.833 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_revoke": 0, 2024-04-21T09:01:47.833 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_grant": 0, 2024-04-21T09:01:47.833 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T09:01:47.833 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T09:01:47.834 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T09:01:47.834 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps": 0, 2024-04-21T09:01:47.834 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps_dirty": 0, 2024-04-21T09:01:47.834 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_cap_release": 0, 2024-04-21T09:01:47.834 INFO:teuthology.orchestra.run.smithi152.stdout: "process_request_cap_release": 0 2024-04-21T09:01:47.834 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.834 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_cache": { 2024-04-21T09:01:47.834 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update": 0, 2024-04-21T09:01:47.835 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update_receipt": 0, 2024-04-21T09:01:47.835 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_try_discover": 0, 2024-04-21T09:01:47.835 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_send_discover": 0, 2024-04-21T09:01:47.835 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_handle_discover": 0, 2024-04-21T09:01:47.835 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays": 0, 2024-04-21T09:01:47.835 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_delayed": 0, 2024-04-21T09:01:47.835 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_enqueuing": 0, 2024-04-21T09:01:47.835 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_created": 0, 2024-04-21T09:01:47.836 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_enqueued": 0, 2024-04-21T09:01:47.836 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_reintegrated": 0, 2024-04-21T09:01:47.836 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_migrated": 0, 2024-04-21T09:01:47.836 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_processing": 0, 2024-04-21T09:01:47.836 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_enqueued": 0, 2024-04-21T09:01:47.836 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_prioritized": 0, 2024-04-21T09:01:47.836 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_started": 0, 2024-04-21T09:01:47.836 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_completed": 0, 2024-04-21T09:01:47.837 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_quiesce_path": 0, 2024-04-21T09:01:47.837 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_quiesce_inode": 0, 2024-04-21T09:01:47.837 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T09:01:47.837 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_exportdir": 0, 2024-04-21T09:01:47.837 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_flush": 0, 2024-04-21T09:01:47.837 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragmentdir": 0, 2024-04-21T09:01:47.837 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragstats": 0, 2024-04-21T09:01:47.837 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_inodestats": 0 2024-04-21T09:01:47.838 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.838 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_log": { 2024-04-21T09:01:47.838 INFO:teuthology.orchestra.run.smithi152.stdout: "evlrg": 0, 2024-04-21T09:01:47.838 INFO:teuthology.orchestra.run.smithi152.stdout: "evadd": 0, 2024-04-21T09:01:47.838 INFO:teuthology.orchestra.run.smithi152.stdout: "evex": 0, 2024-04-21T09:01:47.838 INFO:teuthology.orchestra.run.smithi152.stdout: "evtrm": 0, 2024-04-21T09:01:47.838 INFO:teuthology.orchestra.run.smithi152.stdout: "ev": 24, 2024-04-21T09:01:47.838 INFO:teuthology.orchestra.run.smithi152.stdout: "evexg": 0, 2024-04-21T09:01:47.838 INFO:teuthology.orchestra.run.smithi152.stdout: "evexd": 0, 2024-04-21T09:01:47.839 INFO:teuthology.orchestra.run.smithi152.stdout: "segadd": 0, 2024-04-21T09:01:47.839 INFO:teuthology.orchestra.run.smithi152.stdout: "segex": 0, 2024-04-21T09:01:47.839 INFO:teuthology.orchestra.run.smithi152.stdout: "segtrm": 0, 2024-04-21T09:01:47.839 INFO:teuthology.orchestra.run.smithi152.stdout: "seg": 2, 2024-04-21T09:01:47.839 INFO:teuthology.orchestra.run.smithi152.stdout: "segmjr": 2, 2024-04-21T09:01:47.839 INFO:teuthology.orchestra.run.smithi152.stdout: "segexg": 0, 2024-04-21T09:01:47.839 INFO:teuthology.orchestra.run.smithi152.stdout: "segexd": 0, 2024-04-21T09:01:47.839 INFO:teuthology.orchestra.run.smithi152.stdout: "expos": 4194304, 2024-04-21T09:01:47.840 INFO:teuthology.orchestra.run.smithi152.stdout: "wrpos": 4248535, 2024-04-21T09:01:47.840 INFO:teuthology.orchestra.run.smithi152.stdout: "rdpos": 4247961, 2024-04-21T09:01:47.840 INFO:teuthology.orchestra.run.smithi152.stdout: "jlat": { 2024-04-21T09:01:47.840 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.840 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.840 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.840 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.840 INFO:teuthology.orchestra.run.smithi152.stdout: "replayed": 24 2024-04-21T09:01:47.841 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.841 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_mem": { 2024-04-21T09:01:47.841 INFO:teuthology.orchestra.run.smithi152.stdout: "ino": 20, 2024-04-21T09:01:47.841 INFO:teuthology.orchestra.run.smithi152.stdout: "ino+": 20, 2024-04-21T09:01:47.841 INFO:teuthology.orchestra.run.smithi152.stdout: "ino-": 0, 2024-04-21T09:01:47.841 INFO:teuthology.orchestra.run.smithi152.stdout: "dir": 15, 2024-04-21T09:01:47.841 INFO:teuthology.orchestra.run.smithi152.stdout: "dir+": 15, 2024-04-21T09:01:47.841 INFO:teuthology.orchestra.run.smithi152.stdout: "dir-": 0, 2024-04-21T09:01:47.841 INFO:teuthology.orchestra.run.smithi152.stdout: "dn": 21, 2024-04-21T09:01:47.842 INFO:teuthology.orchestra.run.smithi152.stdout: "dn+": 21, 2024-04-21T09:01:47.842 INFO:teuthology.orchestra.run.smithi152.stdout: "dn-": 0, 2024-04-21T09:01:47.842 INFO:teuthology.orchestra.run.smithi152.stdout: "cap": 0, 2024-04-21T09:01:47.842 INFO:teuthology.orchestra.run.smithi152.stdout: "cap+": 0, 2024-04-21T09:01:47.842 INFO:teuthology.orchestra.run.smithi152.stdout: "cap-": 0, 2024-04-21T09:01:47.842 INFO:teuthology.orchestra.run.smithi152.stdout: "rss": 50888, 2024-04-21T09:01:47.842 INFO:teuthology.orchestra.run.smithi152.stdout: "heap": 215416 2024-04-21T09:01:47.842 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.843 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_server": { 2024-04-21T09:01:47.843 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_client_request": 0, 2024-04-21T09:01:47.843 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_server_request": 0, 2024-04-21T09:01:47.843 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_request": 0, 2024-04-21T09:01:47.843 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_session": 0, 2024-04-21T09:01:47.843 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_peer_request": 0, 2024-04-21T09:01:47.843 INFO:teuthology.orchestra.run.smithi152.stdout: "req_create_latency": { 2024-04-21T09:01:47.843 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.844 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.844 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.844 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.844 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getattr_latency": { 2024-04-21T09:01:47.844 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.844 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.844 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.844 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.844 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getfilelock_latency": { 2024-04-21T09:01:47.845 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.845 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.845 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.845 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.845 INFO:teuthology.orchestra.run.smithi152.stdout: "req_link_latency": { 2024-04-21T09:01:47.845 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.845 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.845 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.846 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.846 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookup_latency": { 2024-04-21T09:01:47.846 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.846 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.846 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.846 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.846 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookuphash_latency": { 2024-04-21T09:01:47.846 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.847 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.847 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.847 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.847 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupino_latency": { 2024-04-21T09:01:47.847 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.847 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.847 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.847 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.848 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupname_latency": { 2024-04-21T09:01:47.848 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.848 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.848 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.848 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.848 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupparent_latency": { 2024-04-21T09:01:47.848 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.848 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.849 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.849 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.849 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupsnap_latency": { 2024-04-21T09:01:47.849 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.849 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.849 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.849 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.849 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lssnap_latency": { 2024-04-21T09:01:47.850 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.850 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.850 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.850 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.850 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mkdir_latency": { 2024-04-21T09:01:47.850 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.851 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.851 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.851 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.851 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mknod_latency": { 2024-04-21T09:01:47.852 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.852 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.852 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.852 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.852 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mksnap_latency": { 2024-04-21T09:01:47.852 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.852 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.852 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.853 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.853 INFO:teuthology.orchestra.run.smithi152.stdout: "req_open_latency": { 2024-04-21T09:01:47.853 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.853 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.853 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.853 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.853 INFO:teuthology.orchestra.run.smithi152.stdout: "req_readdir_latency": { 2024-04-21T09:01:47.853 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.853 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.854 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.854 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.854 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rename_latency": { 2024-04-21T09:01:47.854 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.854 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.854 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.854 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.854 INFO:teuthology.orchestra.run.smithi152.stdout: "req_renamesnap_latency": { 2024-04-21T09:01:47.855 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.855 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.855 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.855 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.855 INFO:teuthology.orchestra.run.smithi152.stdout: "req_snapdiff_latency": { 2024-04-21T09:01:47.855 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.855 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.855 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.856 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.856 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmdir_latency": { 2024-04-21T09:01:47.856 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.856 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.856 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.856 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.856 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmsnap_latency": { 2024-04-21T09:01:47.856 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.857 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.857 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.857 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.857 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmxattr_latency": { 2024-04-21T09:01:47.857 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.857 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.857 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.857 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.858 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setattr_latency": { 2024-04-21T09:01:47.858 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.858 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.858 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.858 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.858 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setdirlayout_latency": { 2024-04-21T09:01:47.858 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.858 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.859 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.859 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.859 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setfilelock_latency": { 2024-04-21T09:01:47.859 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.859 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.859 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.859 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.859 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setlayout_latency": { 2024-04-21T09:01:47.859 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.860 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.860 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.860 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.860 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setxattr_latency": { 2024-04-21T09:01:47.860 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.860 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.860 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.860 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.861 INFO:teuthology.orchestra.run.smithi152.stdout: "req_symlink_latency": { 2024-04-21T09:01:47.861 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.861 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.861 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.861 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.861 INFO:teuthology.orchestra.run.smithi152.stdout: "req_unlink_latency": { 2024-04-21T09:01:47.861 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.861 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.862 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.862 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.862 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_revoke_eviction": 0, 2024-04-21T09:01:47.862 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_acquisition_throttle": 0, 2024-04-21T09:01:47.862 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getvxattr_latency": { 2024-04-21T09:01:47.862 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.862 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.862 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.863 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:47.863 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.863 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_sessions": { 2024-04-21T09:01:47.863 INFO:teuthology.orchestra.run.smithi152.stdout: "session_count": 2, 2024-04-21T09:01:47.863 INFO:teuthology.orchestra.run.smithi152.stdout: "session_add": 4, 2024-04-21T09:01:47.863 INFO:teuthology.orchestra.run.smithi152.stdout: "session_remove": 2, 2024-04-21T09:01:47.863 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_open": 4, 2024-04-21T09:01:47.863 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_stale": 0, 2024-04-21T09:01:47.864 INFO:teuthology.orchestra.run.smithi152.stdout: "total_load": 0, 2024-04-21T09:01:47.864 INFO:teuthology.orchestra.run.smithi152.stdout: "average_load": 0, 2024-04-21T09:01:47.864 INFO:teuthology.orchestra.run.smithi152.stdout: "avg_session_uptime": 2979, 2024-04-21T09:01:47.864 INFO:teuthology.orchestra.run.smithi152.stdout: "mdthresh_evicted": 0 2024-04-21T09:01:47.864 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.864 INFO:teuthology.orchestra.run.smithi152.stdout: "mempool": { 2024-04-21T09:01:47.864 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_bytes": 0, 2024-04-21T09:01:47.864 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_items": 0, 2024-04-21T09:01:47.864 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T09:01:47.865 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_items": 0, 2024-04-21T09:01:47.865 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T09:01:47.865 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_items": 0, 2024-04-21T09:01:47.865 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T09:01:47.865 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T09:01:47.865 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T09:01:47.865 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T09:01:47.866 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T09:01:47.866 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_items": 0, 2024-04-21T09:01:47.866 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T09:01:47.866 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T09:01:47.866 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_bytes": 0, 2024-04-21T09:01:47.866 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_items": 0, 2024-04-21T09:01:47.866 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_bytes": 0, 2024-04-21T09:01:47.866 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_items": 0, 2024-04-21T09:01:47.866 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T09:01:47.867 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T09:01:47.867 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T09:01:47.867 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T09:01:47.867 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T09:01:47.867 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_items": 0, 2024-04-21T09:01:47.867 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_bytes": 0, 2024-04-21T09:01:47.867 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_items": 0, 2024-04-21T09:01:47.867 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T09:01:47.868 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T09:01:47.868 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_bytes": 0, 2024-04-21T09:01:47.868 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_items": 0, 2024-04-21T09:01:47.868 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_bytes": 0, 2024-04-21T09:01:47.868 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_items": 0, 2024-04-21T09:01:47.868 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T09:01:47.868 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_items": 0, 2024-04-21T09:01:47.868 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T09:01:47.869 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_items": 0, 2024-04-21T09:01:47.869 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_bytes": 181871, 2024-04-21T09:01:47.869 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_items": 62, 2024-04-21T09:01:47.869 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_bytes": 0, 2024-04-21T09:01:47.869 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_items": 0, 2024-04-21T09:01:47.869 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_bytes": 0, 2024-04-21T09:01:47.869 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_items": 0, 2024-04-21T09:01:47.869 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_bytes": 0, 2024-04-21T09:01:47.869 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_items": 0, 2024-04-21T09:01:47.870 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_bytes": 0, 2024-04-21T09:01:47.870 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_items": 0, 2024-04-21T09:01:47.870 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_bytes": 9616, 2024-04-21T09:01:47.870 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_items": 185, 2024-04-21T09:01:47.870 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T09:01:47.870 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_items": 0, 2024-04-21T09:01:47.870 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_bytes": 0, 2024-04-21T09:01:47.871 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_items": 0, 2024-04-21T09:01:47.871 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_bytes": 82507, 2024-04-21T09:01:47.871 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_items": 1077, 2024-04-21T09:01:47.871 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_bytes": 0, 2024-04-21T09:01:47.871 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_items": 0, 2024-04-21T09:01:47.871 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_bytes": 0, 2024-04-21T09:01:47.871 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_items": 0 2024-04-21T09:01:47.871 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.871 INFO:teuthology.orchestra.run.smithi152.stdout: "objecter": { 2024-04-21T09:01:47.872 INFO:teuthology.orchestra.run.smithi152.stdout: "op_active": 0, 2024-04-21T09:01:47.872 INFO:teuthology.orchestra.run.smithi152.stdout: "op_laggy": 0, 2024-04-21T09:01:47.872 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send": 2034, 2024-04-21T09:01:47.872 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send_bytes": 21, 2024-04-21T09:01:47.872 INFO:teuthology.orchestra.run.smithi152.stdout: "op_resend": 3, 2024-04-21T09:01:47.872 INFO:teuthology.orchestra.run.smithi152.stdout: "op_reply": 2034, 2024-04-21T09:01:47.872 INFO:teuthology.orchestra.run.smithi152.stdout: "op_latency": { 2024-04-21T09:01:47.872 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 2034, 2024-04-21T09:01:47.873 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 79.927090860, 2024-04-21T09:01:47.873 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.039295521 2024-04-21T09:01:47.873 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.873 INFO:teuthology.orchestra.run.smithi152.stdout: "op_inflight": 0, 2024-04-21T09:01:47.873 INFO:teuthology.orchestra.run.smithi152.stdout: "oplen_avg": { 2024-04-21T09:01:47.873 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 2034, 2024-04-21T09:01:47.873 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 2035 2024-04-21T09:01:47.873 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.874 INFO:teuthology.orchestra.run.smithi152.stdout: "op": 2034, 2024-04-21T09:01:47.874 INFO:teuthology.orchestra.run.smithi152.stdout: "op_r": 2034, 2024-04-21T09:01:47.874 INFO:teuthology.orchestra.run.smithi152.stdout: "op_w": 0, 2024-04-21T09:01:47.874 INFO:teuthology.orchestra.run.smithi152.stdout: "op_rmw": 0, 2024-04-21T09:01:47.874 INFO:teuthology.orchestra.run.smithi152.stdout: "op_pg": 0, 2024-04-21T09:01:47.874 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_stat": 1308, 2024-04-21T09:01:47.874 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_create": 0, 2024-04-21T09:01:47.874 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_read": 724, 2024-04-21T09:01:47.875 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_write": 0, 2024-04-21T09:01:47.875 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writefull": 0, 2024-04-21T09:01:47.875 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writesame": 0, 2024-04-21T09:01:47.875 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_append": 0, 2024-04-21T09:01:47.875 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_zero": 0, 2024-04-21T09:01:47.875 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_truncate": 0, 2024-04-21T09:01:47.875 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_delete": 0, 2024-04-21T09:01:47.875 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_mapext": 0, 2024-04-21T09:01:47.875 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_sparse_read": 0, 2024-04-21T09:01:47.876 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_clonerange": 0, 2024-04-21T09:01:47.876 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_getxattr": 1, 2024-04-21T09:01:47.876 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_setxattr": 0, 2024-04-21T09:01:47.876 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_cmpxattr": 0, 2024-04-21T09:01:47.876 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_rmxattr": 0, 2024-04-21T09:01:47.876 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_resetxattrs": 0, 2024-04-21T09:01:47.876 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_call": 0, 2024-04-21T09:01:47.876 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_watch": 0, 2024-04-21T09:01:47.877 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_notify": 0, 2024-04-21T09:01:47.877 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T09:01:47.877 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls": 0, 2024-04-21T09:01:47.877 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls_filter": 0, 2024-04-21T09:01:47.877 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_other": 0, 2024-04-21T09:01:47.877 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_active": 0, 2024-04-21T09:01:47.877 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_send": 0, 2024-04-21T09:01:47.877 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_resend": 0, 2024-04-21T09:01:47.878 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_ping": 0, 2024-04-21T09:01:47.878 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_active": 0, 2024-04-21T09:01:47.878 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_send": 0, 2024-04-21T09:01:47.878 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_resend": 0, 2024-04-21T09:01:47.878 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_active": 0, 2024-04-21T09:01:47.878 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_send": 0, 2024-04-21T09:01:47.878 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_resend": 0, 2024-04-21T09:01:47.878 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_active": 0, 2024-04-21T09:01:47.878 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_send": 0, 2024-04-21T09:01:47.879 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_resend": 0, 2024-04-21T09:01:47.879 INFO:teuthology.orchestra.run.smithi152.stdout: "command_active": 0, 2024-04-21T09:01:47.879 INFO:teuthology.orchestra.run.smithi152.stdout: "command_send": 0, 2024-04-21T09:01:47.879 INFO:teuthology.orchestra.run.smithi152.stdout: "command_resend": 0, 2024-04-21T09:01:47.879 INFO:teuthology.orchestra.run.smithi152.stdout: "map_epoch": 102, 2024-04-21T09:01:47.879 INFO:teuthology.orchestra.run.smithi152.stdout: "map_full": 0, 2024-04-21T09:01:47.879 INFO:teuthology.orchestra.run.smithi152.stdout: "map_inc": 13, 2024-04-21T09:01:47.879 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_sessions": 4, 2024-04-21T09:01:47.880 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_open": 4, 2024-04-21T09:01:47.880 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_close": 0, 2024-04-21T09:01:47.880 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_laggy": 0, 2024-04-21T09:01:47.880 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_wr": 0, 2024-04-21T09:01:47.880 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_rd": 2, 2024-04-21T09:01:47.880 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_del": 0 2024-04-21T09:01:47.880 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.880 INFO:teuthology.orchestra.run.smithi152.stdout: "oft": { 2024-04-21T09:01:47.881 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_objs": 0, 2024-04-21T09:01:47.881 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_kv_pairs": 0, 2024-04-21T09:01:47.881 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_updates": 0, 2024-04-21T09:01:47.881 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_removes": 0 2024-04-21T09:01:47.881 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.881 INFO:teuthology.orchestra.run.smithi152.stdout: "osd-slow-ops": { 2024-04-21T09:01:47.881 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_ops_count": 0 2024-04-21T09:01:47.881 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.881 INFO:teuthology.orchestra.run.smithi152.stdout: "purge_queue": { 2024-04-21T09:01:47.882 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops": 0, 2024-04-21T09:01:47.882 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T09:01:47.882 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing": 0, 2024-04-21T09:01:47.882 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_high_water": 0, 2024-04-21T09:01:47.882 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executed_ops": 0, 2024-04-21T09:01:47.882 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executed": 0, 2024-04-21T09:01:47.882 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_item_in_journal": 0 2024-04-21T09:01:47.882 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.883 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T09:01:47.883 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:47.883 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-04-21T09:01:47.883 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:47.883 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 2284, 2024-04-21T09:01:47.883 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 2045186, 2024-04-21T09:01:47.883 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:47.883 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 2284, 2024-04-21T09:01:47.884 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-04-21T09:01:47.884 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-04-21T09:01:47.884 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 2284, 2024-04-21T09:01:47.884 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 2045186, 2024-04-21T09:01:47.884 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:47.884 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.884 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.884 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.885 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:47.885 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.885 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_bytes": { 2024-04-21T09:01:47.885 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:47.885 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-04-21T09:01:47.885 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:47.885 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-04-21T09:01:47.885 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-04-21T09:01:47.885 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:47.886 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:47.886 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 2034, 2024-04-21T09:01:47.886 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 54236, 2024-04-21T09:01:47.886 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 2034, 2024-04-21T09:01:47.886 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 54236, 2024-04-21T09:01:47.886 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:47.886 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.886 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.887 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.887 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:47.887 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.887 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_ops": { 2024-04-21T09:01:47.887 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:47.887 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 1024, 2024-04-21T09:01:47.887 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:47.887 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-04-21T09:01:47.888 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-04-21T09:01:47.888 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:47.888 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:47.888 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 2034, 2024-04-21T09:01:47.888 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 2034, 2024-04-21T09:01:47.888 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 2034, 2024-04-21T09:01:47.888 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 2034, 2024-04-21T09:01:47.888 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:47.889 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.889 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.889 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.889 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:47.889 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.889 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle": { 2024-04-21T09:01:47.889 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:47.889 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-04-21T09:01:47.889 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:47.890 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-04-21T09:01:47.890 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-04-21T09:01:47.890 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:47.890 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:47.890 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-04-21T09:01:47.890 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-04-21T09:01:47.890 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 0, 2024-04-21T09:01:47.890 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 0, 2024-04-21T09:01:47.891 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:47.891 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.891 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.891 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.891 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:47.891 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:47.891 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle-0x5622d88b46e0": { 2024-04-21T09:01:47.891 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:47.891 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-04-21T09:01:47.892 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:47.892 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-04-21T09:01:47.892 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-04-21T09:01:47.892 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:47.892 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:47.892 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-04-21T09:01:47.892 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-04-21T09:01:47.892 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 0, 2024-04-21T09:01:47.893 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 0, 2024-04-21T09:01:47.893 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:47.893 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:47.893 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:47.893 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:47.893 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:47.893 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:47.893 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-04-21T09:01:48.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:48 smithi152 ceph-mon[31401]: pgmap v329: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:48.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:48 smithi028 ceph-mon[28003]: pgmap v329: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:48.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:48 smithi171 ceph-mon[31358]: pgmap v329: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:48.639 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-21T09:01:48.639 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-21T09:01:48.640 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-04-21T09:01:48.640 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-21T09:01:48.640 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T09:01:48.640 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-04-21T09:01:48.640 DEBUG:tasks.check_counter:Getting stats from k 2024-04-21T09:01:48.640 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-21T09:01:48.893 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T09:01:50.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:49 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:50.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:49 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:50.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:49 smithi152 ceph-mon[31401]: pgmap v330: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T09:01:50.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:49 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:50.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:49 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:50.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:49 smithi028 ceph-mon[28003]: pgmap v330: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T09:01:50.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:49 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:50.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:49 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:50.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:49 smithi171 ceph-mon[31358]: pgmap v330: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T09:01:50.886 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-04-21T09:01:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T09:01:50.886 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 231, 2024-04-21T09:01:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 220, 2024-04-21T09:01:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 1592384, 2024-04-21T09:01:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 135104, 2024-04-21T09:01:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 4, 2024-04-21T09:01:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 0, 2024-04-21T09:01:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.048105378, 2024-04-21T09:01:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.013734944, 2024-04-21T09:01:50.887 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.036679352, 2024-04-21T09:01:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.005915307, 2024-04-21T09:01:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 219, 2024-04-21T09:01:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.013365514, 2024-04-21T09:01:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000061029 2024-04-21T09:01:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:50.888 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 1592384, 2024-04-21T09:01:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 135104 2024-04-21T09:01:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T09:01:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 2, 2024-04-21T09:01:50.889 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 168, 2024-04-21T09:01:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 320, 2024-04-21T09:01:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 185680, 2024-04-21T09:01:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 5, 2024-04-21T09:01:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 3, 2024-04-21T09:01:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.010345088, 2024-04-21T09:01:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.006752263, 2024-04-21T09:01:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.001460396, 2024-04-21T09:01:50.890 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T09:01:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 165, 2024-04-21T09:01:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.015034327, 2024-04-21T09:01:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000091117 2024-04-21T09:01:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:50.891 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T09:01:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 185680 2024-04-21T09:01:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T09:01:50.892 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 4, 2024-04-21T09:01:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 4, 2024-04-21T09:01:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 2992, 2024-04-21T09:01:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 448, 2024-04-21T09:01:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 2, 2024-04-21T09:01:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 1, 2024-04-21T09:01:50.893 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.000826502, 2024-04-21T09:01:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.000042488, 2024-04-21T09:01:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.000774540, 2024-04-21T09:01:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T09:01:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 3, 2024-04-21T09:01:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000694420, 2024-04-21T09:01:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000231473 2024-04-21T09:01:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:50.894 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-21T09:01:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T09:01:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:50.895 INFO:teuthology.orchestra.run.smithi152.stdout: "cct": { 2024-04-21T09:01:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: "total_workers": 0, 2024-04-21T09:01:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: "unhealthy_workers": 0 2024-04-21T09:01:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: "mempool": { 2024-04-21T09:01:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_bytes": 0, 2024-04-21T09:01:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_items": 0, 2024-04-21T09:01:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T09:01:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_items": 0, 2024-04-21T09:01:50.896 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T09:01:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_items": 0, 2024-04-21T09:01:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T09:01:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T09:01:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T09:01:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T09:01:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T09:01:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_items": 0, 2024-04-21T09:01:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T09:01:50.897 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T09:01:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_bytes": 0, 2024-04-21T09:01:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_items": 0, 2024-04-21T09:01:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_bytes": 0, 2024-04-21T09:01:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_items": 0, 2024-04-21T09:01:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T09:01:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T09:01:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T09:01:50.898 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T09:01:50.899 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T09:01:50.899 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_items": 0, 2024-04-21T09:01:50.899 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_bytes": 0, 2024-04-21T09:01:50.899 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_items": 0, 2024-04-21T09:01:50.899 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T09:01:50.899 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T09:01:50.899 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_bytes": 0, 2024-04-21T09:01:50.899 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_items": 0, 2024-04-21T09:01:50.899 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_bytes": 0, 2024-04-21T09:01:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_items": 0, 2024-04-21T09:01:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T09:01:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_items": 0, 2024-04-21T09:01:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T09:01:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_items": 0, 2024-04-21T09:01:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_bytes": 150008, 2024-04-21T09:01:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_items": 31, 2024-04-21T09:01:50.900 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_bytes": 0, 2024-04-21T09:01:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_items": 0, 2024-04-21T09:01:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_bytes": 0, 2024-04-21T09:01:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_items": 0, 2024-04-21T09:01:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_bytes": 0, 2024-04-21T09:01:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_items": 0, 2024-04-21T09:01:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_bytes": 0, 2024-04-21T09:01:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_items": 0, 2024-04-21T09:01:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_bytes": 0, 2024-04-21T09:01:50.901 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_items": 0, 2024-04-21T09:01:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T09:01:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_items": 0, 2024-04-21T09:01:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_bytes": 0, 2024-04-21T09:01:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_items": 0, 2024-04-21T09:01:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_bytes": 928, 2024-04-21T09:01:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_items": 2, 2024-04-21T09:01:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_bytes": 0, 2024-04-21T09:01:50.902 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_items": 0, 2024-04-21T09:01:50.903 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_bytes": 0, 2024-04-21T09:01:50.903 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_items": 0 2024-04-21T09:01:50.903 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-04-21T09:01:50.903 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T09:01:50.903 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-04-21T09:01:50.903 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-04-21T09:01:50.903 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-04-21T09:01:50.904 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 233, 2024-04-21T09:01:50.904 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 1559794, 2024-04-21T09:01:50.904 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:50.904 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 233, 2024-04-21T09:01:50.904 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-04-21T09:01:50.904 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-04-21T09:01:50.904 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 233, 2024-04-21T09:01:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 1559794, 2024-04-21T09:01:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-04-21T09:01:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-04-21T09:01:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-04-21T09:01:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-04-21T09:01:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:50.905 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-04-21T09:01:50.905 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-04-21T09:01:51.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:51 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:51.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:51 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:51.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:51 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:51.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:51 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:51.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:51 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T09:01:51.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:51 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T09:01:51.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:51 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:51.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:51 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:51.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:51 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:51.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:51 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:51.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:51 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:51.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:51 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T09:01:51.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:51 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T09:01:51.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:51 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:51 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:51 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:51 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:51 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T09:01:51.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:51 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T09:01:51.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:51 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T09:01:51.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:51 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' 2024-04-21T09:01:51.691 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-21T09:01:51.691 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-21T09:01:51.691 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-21T09:01:51.691 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-21T09:01:51.692 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-21T09:01:51.692 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-04-21T09:01:51.692 DEBUG:tasks.check_counter:Getting stats from c 2024-04-21T09:01:51.692 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-21T09:01:51.953 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T09:01:52.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:52 smithi152 ceph-mon[31401]: Adjusting osd_memory_target on smithi152 to 176.2M 2024-04-21T09:01:52.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:52 smithi152 ceph-mon[31401]: Unable to set osd_memory_target on smithi152 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-21T09:01:52.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:52 smithi152 ceph-mon[31401]: pgmap v331: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T09:01:52.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:52 smithi028 ceph-mon[28003]: Adjusting osd_memory_target on smithi152 to 176.2M 2024-04-21T09:01:52.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:52 smithi028 ceph-mon[28003]: Unable to set osd_memory_target on smithi152 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-21T09:01:52.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:52 smithi028 ceph-mon[28003]: pgmap v331: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T09:01:52.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:52 smithi171 ceph-mon[31358]: Adjusting osd_memory_target on smithi152 to 176.2M 2024-04-21T09:01:52.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:52 smithi171 ceph-mon[31358]: Unable to set osd_memory_target on smithi152 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-21T09:01:52.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:52 smithi171 ceph-mon[31358]: pgmap v331: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T09:01:54.329 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:54 smithi171 ceph-mon[31358]: pgmap v332: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:54.357 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:54 smithi152 ceph-mon[31401]: pgmap v332: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:54.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:54 smithi028 ceph-mon[28003]: pgmap v332: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:54.376 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-21T09:01:54.376 INFO:teuthology.orchestra.run.smithi171.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T09:01:54.377 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_messages": 21851, 2024-04-21T09:01:54.377 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages": 23089, 2024-04-21T09:01:54.377 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_bytes": 6894640, 2024-04-21T09:01:54.377 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_bytes": 28680546, 2024-04-21T09:01:54.377 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_created_connections": 10, 2024-04-21T09:01:54.377 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_active_connections": 4, 2024-04-21T09:01:54.377 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_total_time": 1.436183800, 2024-04-21T09:01:54.377 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_send_time": 0.504784970, 2024-04-21T09:01:54.378 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_recv_time": 1.405712299, 2024-04-21T09:01:54.378 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_fast_dispatch_time": 0.149991798, 2024-04-21T09:01:54.378 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:54.378 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 23092, 2024-04-21T09:01:54.378 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 7.167055043, 2024-04-21T09:01:54.378 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000310369 2024-04-21T09:01:54.378 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.378 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:54.379 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 33112, 2024-04-21T09:01:54.379 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.008282618, 2024-04-21T09:01:54.379 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000250 2024-04-21T09:01:54.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.379 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-21T09:01:54.379 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T09:01:54.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.379 INFO:teuthology.orchestra.run.smithi171.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T09:01:54.379 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_messages": 1406, 2024-04-21T09:01:54.380 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages": 1390, 2024-04-21T09:01:54.380 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_bytes": 1929985, 2024-04-21T09:01:54.380 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_bytes": 17048456, 2024-04-21T09:01:54.380 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_created_connections": 10, 2024-04-21T09:01:54.380 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_active_connections": 6, 2024-04-21T09:01:54.380 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_total_time": 0.205880063, 2024-04-21T09:01:54.380 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_send_time": 0.093381867, 2024-04-21T09:01:54.380 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_recv_time": 0.063922804, 2024-04-21T09:01:54.381 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_fast_dispatch_time": 0.050595600, 2024-04-21T09:01:54.381 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:54.381 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 1389, 2024-04-21T09:01:54.381 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.135617050, 2024-04-21T09:01:54.381 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000097636 2024-04-21T09:01:54.381 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.381 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:54.381 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 10, 2024-04-21T09:01:54.381 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000001841, 2024-04-21T09:01:54.382 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000184 2024-04-21T09:01:54.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.382 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_encrypted_bytes": 1649888, 2024-04-21T09:01:54.382 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_encrypted_bytes": 171744 2024-04-21T09:01:54.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.382 INFO:teuthology.orchestra.run.smithi171.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T09:01:54.382 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_messages": 240, 2024-04-21T09:01:54.382 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages": 438, 2024-04-21T09:01:54.383 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_bytes": 56689, 2024-04-21T09:01:54.383 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_bytes": 5217848, 2024-04-21T09:01:54.383 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_created_connections": 52, 2024-04-21T09:01:54.383 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_active_connections": 10, 2024-04-21T09:01:54.383 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_total_time": 0.056621309, 2024-04-21T09:01:54.383 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_send_time": 0.023869391, 2024-04-21T09:01:54.383 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_recv_time": 0.021826315, 2024-04-21T09:01:54.383 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_fast_dispatch_time": 0.007130649, 2024-04-21T09:01:54.383 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:54.384 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 431, 2024-04-21T09:01:54.384 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.026499356, 2024-04-21T09:01:54.384 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000061483 2024-04-21T09:01:54.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.384 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:54.384 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 114, 2024-04-21T09:01:54.384 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000016784, 2024-04-21T09:01:54.384 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000147 2024-04-21T09:01:54.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.385 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_encrypted_bytes": 7280, 2024-04-21T09:01:54.385 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_encrypted_bytes": 466528 2024-04-21T09:01:54.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.385 INFO:teuthology.orchestra.run.smithi171.stdout: "cct": { 2024-04-21T09:01:54.385 INFO:teuthology.orchestra.run.smithi171.stdout: "total_workers": 1, 2024-04-21T09:01:54.385 INFO:teuthology.orchestra.run.smithi171.stdout: "unhealthy_workers": 0 2024-04-21T09:01:54.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.385 INFO:teuthology.orchestra.run.smithi171.stdout: "finisher-MDSRank": { 2024-04-21T09:01:54.386 INFO:teuthology.orchestra.run.smithi171.stdout: "queue_len": 0, 2024-04-21T09:01:54.386 INFO:teuthology.orchestra.run.smithi171.stdout: "complete_latency": { 2024-04-21T09:01:54.386 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 4266, 2024-04-21T09:01:54.386 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 9.200388317, 2024-04-21T09:01:54.386 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.002156677 2024-04-21T09:01:54.386 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:54.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.386 INFO:teuthology.orchestra.run.smithi171.stdout: "finisher-PurgeQueue": { 2024-04-21T09:01:54.386 INFO:teuthology.orchestra.run.smithi171.stdout: "queue_len": 0, 2024-04-21T09:01:54.387 INFO:teuthology.orchestra.run.smithi171.stdout: "complete_latency": { 2024-04-21T09:01:54.387 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 2, 2024-04-21T09:01:54.387 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000006561, 2024-04-21T09:01:54.387 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000003280 2024-04-21T09:01:54.387 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:54.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.387 INFO:teuthology.orchestra.run.smithi171.stdout: "mds": { 2024-04-21T09:01:54.387 INFO:teuthology.orchestra.run.smithi171.stdout: "request": 377, 2024-04-21T09:01:54.387 INFO:teuthology.orchestra.run.smithi171.stdout: "reply": 358, 2024-04-21T09:01:54.388 INFO:teuthology.orchestra.run.smithi171.stdout: "reply_latency": { 2024-04-21T09:01:54.388 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 358, 2024-04-21T09:01:54.388 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 11.974770112, 2024-04-21T09:01:54.388 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.033449078 2024-04-21T09:01:54.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.388 INFO:teuthology.orchestra.run.smithi171.stdout: "slow_reply": 0, 2024-04-21T09:01:54.388 INFO:teuthology.orchestra.run.smithi171.stdout: "forward": 19, 2024-04-21T09:01:54.388 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_fetch_complete": 0, 2024-04-21T09:01:54.388 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_fetch_keys": 0, 2024-04-21T09:01:54.388 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_commit": 28, 2024-04-21T09:01:54.389 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_split": 2, 2024-04-21T09:01:54.389 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_merge": 0, 2024-04-21T09:01:54.389 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes": 594, 2024-04-21T09:01:54.389 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_top": 55, 2024-04-21T09:01:54.389 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_bottom": 539, 2024-04-21T09:01:54.389 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_pin_tail": 0, 2024-04-21T09:01:54.389 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_pinned": 516, 2024-04-21T09:01:54.389 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_expired": 1, 2024-04-21T09:01:54.390 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_with_caps": 2, 2024-04-21T09:01:54.390 INFO:teuthology.orchestra.run.smithi171.stdout: "caps": 2, 2024-04-21T09:01:54.390 INFO:teuthology.orchestra.run.smithi171.stdout: "subtrees": 8, 2024-04-21T09:01:54.390 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse": 400, 2024-04-21T09:01:54.390 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_hit": 369, 2024-04-21T09:01:54.390 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_forward": 16, 2024-04-21T09:01:54.390 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_discover": 11, 2024-04-21T09:01:54.390 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_dir_fetch": 0, 2024-04-21T09:01:54.390 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_remote_ino": 0, 2024-04-21T09:01:54.391 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_lock": 0, 2024-04-21T09:01:54.391 INFO:teuthology.orchestra.run.smithi171.stdout: "load_cent": 0, 2024-04-21T09:01:54.391 INFO:teuthology.orchestra.run.smithi171.stdout: "q": 0, 2024-04-21T09:01:54.391 INFO:teuthology.orchestra.run.smithi171.stdout: "exported": 0, 2024-04-21T09:01:54.391 INFO:teuthology.orchestra.run.smithi171.stdout: "exported_inodes": 0, 2024-04-21T09:01:54.391 INFO:teuthology.orchestra.run.smithi171.stdout: "imported": 6, 2024-04-21T09:01:54.391 INFO:teuthology.orchestra.run.smithi171.stdout: "imported_inodes": 49, 2024-04-21T09:01:54.391 INFO:teuthology.orchestra.run.smithi171.stdout: "openino_dir_fetch": 0, 2024-04-21T09:01:54.392 INFO:teuthology.orchestra.run.smithi171.stdout: "openino_backtrace_fetch": 1998, 2024-04-21T09:01:54.392 INFO:teuthology.orchestra.run.smithi171.stdout: "openino_peer_discover": 0, 2024-04-21T09:01:54.392 INFO:teuthology.orchestra.run.smithi171.stdout: "root_rfiles": 325, 2024-04-21T09:01:54.392 INFO:teuthology.orchestra.run.smithi171.stdout: "root_rbytes": 13424230466, 2024-04-21T09:01:54.392 INFO:teuthology.orchestra.run.smithi171.stdout: "root_rsnaps": 6, 2024-04-21T09:01:54.392 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_backtrace_fetch": 1998, 2024-04-21T09:01:54.392 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_set_tag": 0, 2024-04-21T09:01:54.392 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T09:01:54.392 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_inotable_repaired": 0, 2024-04-21T09:01:54.393 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_dir_inodes": 45, 2024-04-21T09:01:54.393 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T09:01:54.393 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_dirfrag_rstats": 45, 2024-04-21T09:01:54.393 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_file_inodes": 1953, 2024-04-21T09:01:54.393 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_inode_file_caps": 1, 2024-04-21T09:01:54.393 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_cap_op_revoke": 454, 2024-04-21T09:01:54.393 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_cap_op_grant": 1603, 2024-04-21T09:01:54.393 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T09:01:54.394 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T09:01:54.394 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T09:01:54.394 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_client_caps": 17590, 2024-04-21T09:01:54.394 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_client_caps_dirty": 16913, 2024-04-21T09:01:54.394 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_client_cap_release": 18, 2024-04-21T09:01:54.394 INFO:teuthology.orchestra.run.smithi171.stdout: "process_request_cap_release": 2 2024-04-21T09:01:54.394 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.394 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_cache": { 2024-04-21T09:01:54.395 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_update": 0, 2024-04-21T09:01:54.395 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_update_receipt": 3, 2024-04-21T09:01:54.395 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_try_discover": 2, 2024-04-21T09:01:54.395 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_send_discover": 11, 2024-04-21T09:01:54.395 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_handle_discover": 0, 2024-04-21T09:01:54.395 INFO:teuthology.orchestra.run.smithi171.stdout: "num_strays": 0, 2024-04-21T09:01:54.395 INFO:teuthology.orchestra.run.smithi171.stdout: "num_strays_delayed": 0, 2024-04-21T09:01:54.395 INFO:teuthology.orchestra.run.smithi171.stdout: "num_strays_enqueuing": 0, 2024-04-21T09:01:54.396 INFO:teuthology.orchestra.run.smithi171.stdout: "strays_created": 0, 2024-04-21T09:01:54.396 INFO:teuthology.orchestra.run.smithi171.stdout: "strays_enqueued": 0, 2024-04-21T09:01:54.396 INFO:teuthology.orchestra.run.smithi171.stdout: "strays_reintegrated": 0, 2024-04-21T09:01:54.396 INFO:teuthology.orchestra.run.smithi171.stdout: "strays_migrated": 0, 2024-04-21T09:01:54.396 INFO:teuthology.orchestra.run.smithi171.stdout: "num_recovering_processing": 0, 2024-04-21T09:01:54.396 INFO:teuthology.orchestra.run.smithi171.stdout: "num_recovering_enqueued": 0, 2024-04-21T09:01:54.396 INFO:teuthology.orchestra.run.smithi171.stdout: "num_recovering_prioritized": 0, 2024-04-21T09:01:54.396 INFO:teuthology.orchestra.run.smithi171.stdout: "recovery_started": 0, 2024-04-21T09:01:54.397 INFO:teuthology.orchestra.run.smithi171.stdout: "recovery_completed": 0, 2024-04-21T09:01:54.397 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_quiesce_path": 2, 2024-04-21T09:01:54.397 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_quiesce_inode": 308, 2024-04-21T09:01:54.397 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T09:01:54.397 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_exportdir": 0, 2024-04-21T09:01:54.397 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_flush": 0, 2024-04-21T09:01:54.397 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_fragmentdir": 2, 2024-04-21T09:01:54.397 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_fragstats": 0, 2024-04-21T09:01:54.397 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_inodestats": 0 2024-04-21T09:01:54.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.398 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_log": { 2024-04-21T09:01:54.398 INFO:teuthology.orchestra.run.smithi171.stdout: "evlrg": 0, 2024-04-21T09:01:54.398 INFO:teuthology.orchestra.run.smithi171.stdout: "evadd": 17464, 2024-04-21T09:01:54.398 INFO:teuthology.orchestra.run.smithi171.stdout: "evex": 0, 2024-04-21T09:01:54.398 INFO:teuthology.orchestra.run.smithi171.stdout: "evtrm": 0, 2024-04-21T09:01:54.398 INFO:teuthology.orchestra.run.smithi171.stdout: "ev": 17464, 2024-04-21T09:01:54.399 INFO:teuthology.orchestra.run.smithi171.stdout: "evexg": 0, 2024-04-21T09:01:54.399 INFO:teuthology.orchestra.run.smithi171.stdout: "evexd": 0, 2024-04-21T09:01:54.399 INFO:teuthology.orchestra.run.smithi171.stdout: "segadd": 22, 2024-04-21T09:01:54.399 INFO:teuthology.orchestra.run.smithi171.stdout: "segex": 0, 2024-04-21T09:01:54.399 INFO:teuthology.orchestra.run.smithi171.stdout: "segtrm": 0, 2024-04-21T09:01:54.399 INFO:teuthology.orchestra.run.smithi171.stdout: "seg": 22, 2024-04-21T09:01:54.399 INFO:teuthology.orchestra.run.smithi171.stdout: "segmjr": 3, 2024-04-21T09:01:54.399 INFO:teuthology.orchestra.run.smithi171.stdout: "segexg": 0, 2024-04-21T09:01:54.399 INFO:teuthology.orchestra.run.smithi171.stdout: "segexd": 0, 2024-04-21T09:01:54.399 INFO:teuthology.orchestra.run.smithi171.stdout: "expos": 4194304, 2024-04-21T09:01:54.400 INFO:teuthology.orchestra.run.smithi171.stdout: "wrpos": 49519592, 2024-04-21T09:01:54.400 INFO:teuthology.orchestra.run.smithi171.stdout: "rdpos": 0, 2024-04-21T09:01:54.400 INFO:teuthology.orchestra.run.smithi171.stdout: "jlat": { 2024-04-21T09:01:54.400 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 1488, 2024-04-21T09:01:54.400 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 232.712150194, 2024-04-21T09:01:54.400 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.156392574 2024-04-21T09:01:54.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.400 INFO:teuthology.orchestra.run.smithi171.stdout: "replayed": 0 2024-04-21T09:01:54.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.401 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_mem": { 2024-04-21T09:01:54.401 INFO:teuthology.orchestra.run.smithi171.stdout: "ino": 593, 2024-04-21T09:01:54.401 INFO:teuthology.orchestra.run.smithi171.stdout: "ino+": 593, 2024-04-21T09:01:54.401 INFO:teuthology.orchestra.run.smithi171.stdout: "ino-": 0, 2024-04-21T09:01:54.401 INFO:teuthology.orchestra.run.smithi171.stdout: "dir": 46, 2024-04-21T09:01:54.401 INFO:teuthology.orchestra.run.smithi171.stdout: "dir+": 49, 2024-04-21T09:01:54.401 INFO:teuthology.orchestra.run.smithi171.stdout: "dir-": 3, 2024-04-21T09:01:54.401 INFO:teuthology.orchestra.run.smithi171.stdout: "dn": 594, 2024-04-21T09:01:54.401 INFO:teuthology.orchestra.run.smithi171.stdout: "dn+": 595, 2024-04-21T09:01:54.402 INFO:teuthology.orchestra.run.smithi171.stdout: "dn-": 1, 2024-04-21T09:01:54.402 INFO:teuthology.orchestra.run.smithi171.stdout: "cap": 2, 2024-04-21T09:01:54.402 INFO:teuthology.orchestra.run.smithi171.stdout: "cap+": 272, 2024-04-21T09:01:54.402 INFO:teuthology.orchestra.run.smithi171.stdout: "cap-": 270, 2024-04-21T09:01:54.402 INFO:teuthology.orchestra.run.smithi171.stdout: "rss": 73840, 2024-04-21T09:01:54.402 INFO:teuthology.orchestra.run.smithi171.stdout: "heap": 223608 2024-04-21T09:01:54.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.402 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_server": { 2024-04-21T09:01:54.402 INFO:teuthology.orchestra.run.smithi171.stdout: "dispatch_client_request": 379, 2024-04-21T09:01:54.403 INFO:teuthology.orchestra.run.smithi171.stdout: "dispatch_server_request": 2, 2024-04-21T09:01:54.403 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_client_request": 377, 2024-04-21T09:01:54.403 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_client_session": 29, 2024-04-21T09:01:54.403 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_peer_request": 40, 2024-04-21T09:01:54.403 INFO:teuthology.orchestra.run.smithi171.stdout: "req_create_latency": { 2024-04-21T09:01:54.403 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 91, 2024-04-21T09:01:54.403 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.128318457, 2024-04-21T09:01:54.403 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.001410092 2024-04-21T09:01:54.403 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.404 INFO:teuthology.orchestra.run.smithi171.stdout: "req_getattr_latency": { 2024-04-21T09:01:54.404 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 5, 2024-04-21T09:01:54.404 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.001130856, 2024-04-21T09:01:54.404 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000226171 2024-04-21T09:01:54.404 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.404 INFO:teuthology.orchestra.run.smithi171.stdout: "req_getfilelock_latency": { 2024-04-21T09:01:54.404 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.404 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.405 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.405 INFO:teuthology.orchestra.run.smithi171.stdout: "req_link_latency": { 2024-04-21T09:01:54.405 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.405 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.405 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.405 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookup_latency": { 2024-04-21T09:01:54.405 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 3, 2024-04-21T09:01:54.405 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.003577125, 2024-04-21T09:01:54.406 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.001192375 2024-04-21T09:01:54.406 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.406 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookuphash_latency": { 2024-04-21T09:01:54.406 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.406 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.406 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.406 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.406 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookupino_latency": { 2024-04-21T09:01:54.407 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.407 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.407 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.407 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookupname_latency": { 2024-04-21T09:01:54.407 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.407 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.407 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.408 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookupparent_latency": { 2024-04-21T09:01:54.408 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.408 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.408 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.408 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookupsnap_latency": { 2024-04-21T09:01:54.408 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.408 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.409 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.409 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.409 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lssnap_latency": { 2024-04-21T09:01:54.409 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.409 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.410 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.410 INFO:teuthology.orchestra.run.smithi171.stdout: "req_mkdir_latency": { 2024-04-21T09:01:54.410 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.410 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.410 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.410 INFO:teuthology.orchestra.run.smithi171.stdout: "req_mknod_latency": { 2024-04-21T09:01:54.411 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.411 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.411 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.411 INFO:teuthology.orchestra.run.smithi171.stdout: "req_mksnap_latency": { 2024-04-21T09:01:54.411 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.411 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.411 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.412 INFO:teuthology.orchestra.run.smithi171.stdout: "req_open_latency": { 2024-04-21T09:01:54.412 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 254, 2024-04-21T09:01:54.412 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 11.840227816, 2024-04-21T09:01:54.412 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.046615070 2024-04-21T09:01:54.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.412 INFO:teuthology.orchestra.run.smithi171.stdout: "req_readdir_latency": { 2024-04-21T09:01:54.412 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 5, 2024-04-21T09:01:54.412 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.001515858, 2024-04-21T09:01:54.412 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000303171 2024-04-21T09:01:54.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.413 INFO:teuthology.orchestra.run.smithi171.stdout: "req_rename_latency": { 2024-04-21T09:01:54.413 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.413 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.413 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.413 INFO:teuthology.orchestra.run.smithi171.stdout: "req_renamesnap_latency": { 2024-04-21T09:01:54.413 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.414 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.414 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.414 INFO:teuthology.orchestra.run.smithi171.stdout: "req_snapdiff_latency": { 2024-04-21T09:01:54.414 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.414 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.414 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.415 INFO:teuthology.orchestra.run.smithi171.stdout: "req_rmdir_latency": { 2024-04-21T09:01:54.415 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.415 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.415 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.415 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.415 INFO:teuthology.orchestra.run.smithi171.stdout: "req_rmsnap_latency": { 2024-04-21T09:01:54.415 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.415 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.415 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.416 INFO:teuthology.orchestra.run.smithi171.stdout: "req_rmxattr_latency": { 2024-04-21T09:01:54.416 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.416 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.416 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.416 INFO:teuthology.orchestra.run.smithi171.stdout: "req_setattr_latency": { 2024-04-21T09:01:54.416 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.417 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.417 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.417 INFO:teuthology.orchestra.run.smithi171.stdout: "req_setdirlayout_latency": { 2024-04-21T09:01:54.417 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.417 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.417 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.418 INFO:teuthology.orchestra.run.smithi171.stdout: "req_setfilelock_latency": { 2024-04-21T09:01:54.418 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.418 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.418 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.418 INFO:teuthology.orchestra.run.smithi171.stdout: "req_setlayout_latency": { 2024-04-21T09:01:54.418 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.418 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.418 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.419 INFO:teuthology.orchestra.run.smithi171.stdout: "req_setxattr_latency": { 2024-04-21T09:01:54.419 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.419 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.419 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.419 INFO:teuthology.orchestra.run.smithi171.stdout: "req_symlink_latency": { 2024-04-21T09:01:54.419 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.420 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.420 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.420 INFO:teuthology.orchestra.run.smithi171.stdout: "req_unlink_latency": { 2024-04-21T09:01:54.420 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.420 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.420 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.420 INFO:teuthology.orchestra.run.smithi171.stdout: "cap_revoke_eviction": 0, 2024-04-21T09:01:54.421 INFO:teuthology.orchestra.run.smithi171.stdout: "cap_acquisition_throttle": 0, 2024-04-21T09:01:54.421 INFO:teuthology.orchestra.run.smithi171.stdout: "req_getvxattr_latency": { 2024-04-21T09:01:54.421 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.421 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.421 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.421 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:54.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.421 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_sessions": { 2024-04-21T09:01:54.422 INFO:teuthology.orchestra.run.smithi171.stdout: "session_count": 2, 2024-04-21T09:01:54.422 INFO:teuthology.orchestra.run.smithi171.stdout: "session_add": 5, 2024-04-21T09:01:54.422 INFO:teuthology.orchestra.run.smithi171.stdout: "session_remove": 3, 2024-04-21T09:01:54.422 INFO:teuthology.orchestra.run.smithi171.stdout: "sessions_open": 2, 2024-04-21T09:01:54.422 INFO:teuthology.orchestra.run.smithi171.stdout: "sessions_stale": 0, 2024-04-21T09:01:54.422 INFO:teuthology.orchestra.run.smithi171.stdout: "total_load": 114, 2024-04-21T09:01:54.422 INFO:teuthology.orchestra.run.smithi171.stdout: "average_load": 57, 2024-04-21T09:01:54.422 INFO:teuthology.orchestra.run.smithi171.stdout: "avg_session_uptime": 607, 2024-04-21T09:01:54.422 INFO:teuthology.orchestra.run.smithi171.stdout: "mdthresh_evicted": 0 2024-04-21T09:01:54.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.423 INFO:teuthology.orchestra.run.smithi171.stdout: "mempool": { 2024-04-21T09:01:54.423 INFO:teuthology.orchestra.run.smithi171.stdout: "bloom_filter_bytes": 0, 2024-04-21T09:01:54.423 INFO:teuthology.orchestra.run.smithi171.stdout: "bloom_filter_items": 0, 2024-04-21T09:01:54.423 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T09:01:54.423 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_alloc_items": 0, 2024-04-21T09:01:54.423 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T09:01:54.423 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_data_items": 0, 2024-04-21T09:01:54.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T09:01:54.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T09:01:54.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T09:01:54.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T09:01:54.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T09:01:54.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_other_items": 0, 2024-04-21T09:01:54.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T09:01:54.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T09:01:54.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_extent_bytes": 0, 2024-04-21T09:01:54.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_extent_items": 0, 2024-04-21T09:01:54.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_blob_bytes": 0, 2024-04-21T09:01:54.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_blob_items": 0, 2024-04-21T09:01:54.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T09:01:54.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T09:01:54.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T09:01:54.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T09:01:54.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T09:01:54.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_fsck_items": 0, 2024-04-21T09:01:54.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_txc_bytes": 0, 2024-04-21T09:01:54.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_txc_items": 0, 2024-04-21T09:01:54.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T09:01:54.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T09:01:54.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_bytes": 0, 2024-04-21T09:01:54.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_items": 0, 2024-04-21T09:01:54.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_bytes": 0, 2024-04-21T09:01:54.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_items": 0, 2024-04-21T09:01:54.427 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T09:01:54.427 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_reader_items": 0, 2024-04-21T09:01:54.427 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T09:01:54.427 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_writer_items": 0, 2024-04-21T09:01:54.427 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_anon_bytes": 293285, 2024-04-21T09:01:54.427 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_anon_items": 253, 2024-04-21T09:01:54.427 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_meta_bytes": 0, 2024-04-21T09:01:54.427 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_meta_items": 0, 2024-04-21T09:01:54.428 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_bytes": 0, 2024-04-21T09:01:54.428 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_items": 0, 2024-04-21T09:01:54.428 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_mapbl_bytes": 0, 2024-04-21T09:01:54.428 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_mapbl_items": 0, 2024-04-21T09:01:54.428 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_pglog_bytes": 0, 2024-04-21T09:01:54.428 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_pglog_items": 0, 2024-04-21T09:01:54.428 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_bytes": 9616, 2024-04-21T09:01:54.428 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_items": 185, 2024-04-21T09:01:54.428 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T09:01:54.429 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_mapping_items": 0, 2024-04-21T09:01:54.429 INFO:teuthology.orchestra.run.smithi171.stdout: "pgmap_bytes": 0, 2024-04-21T09:01:54.429 INFO:teuthology.orchestra.run.smithi171.stdout: "pgmap_items": 0, 2024-04-21T09:01:54.429 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_co_bytes": 1807471, 2024-04-21T09:01:54.429 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_co_items": 12770, 2024-04-21T09:01:54.429 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_1_bytes": 0, 2024-04-21T09:01:54.429 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_1_items": 0, 2024-04-21T09:01:54.429 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_2_bytes": 0, 2024-04-21T09:01:54.430 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_2_items": 0 2024-04-21T09:01:54.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.430 INFO:teuthology.orchestra.run.smithi171.stdout: "objecter": { 2024-04-21T09:01:54.430 INFO:teuthology.orchestra.run.smithi171.stdout: "op_active": 0, 2024-04-21T09:01:54.430 INFO:teuthology.orchestra.run.smithi171.stdout: "op_laggy": 0, 2024-04-21T09:01:54.430 INFO:teuthology.orchestra.run.smithi171.stdout: "op_send": 3728, 2024-04-21T09:01:54.430 INFO:teuthology.orchestra.run.smithi171.stdout: "op_send_bytes": 45612762, 2024-04-21T09:01:54.430 INFO:teuthology.orchestra.run.smithi171.stdout: "op_resend": 33, 2024-04-21T09:01:54.430 INFO:teuthology.orchestra.run.smithi171.stdout: "op_reply": 3728, 2024-04-21T09:01:54.431 INFO:teuthology.orchestra.run.smithi171.stdout: "op_latency": { 2024-04-21T09:01:54.431 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 3728, 2024-04-21T09:01:54.431 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 495.356803612, 2024-04-21T09:01:54.431 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.132874679 2024-04-21T09:01:54.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.431 INFO:teuthology.orchestra.run.smithi171.stdout: "op_inflight": 0, 2024-04-21T09:01:54.431 INFO:teuthology.orchestra.run.smithi171.stdout: "oplen_avg": { 2024-04-21T09:01:54.431 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 3728, 2024-04-21T09:01:54.432 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 3935 2024-04-21T09:01:54.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.432 INFO:teuthology.orchestra.run.smithi171.stdout: "op": 3728, 2024-04-21T09:01:54.432 INFO:teuthology.orchestra.run.smithi171.stdout: "op_r": 1998, 2024-04-21T09:01:54.432 INFO:teuthology.orchestra.run.smithi171.stdout: "op_w": 1730, 2024-04-21T09:01:54.432 INFO:teuthology.orchestra.run.smithi171.stdout: "op_rmw": 0, 2024-04-21T09:01:54.432 INFO:teuthology.orchestra.run.smithi171.stdout: "op_pg": 0, 2024-04-21T09:01:54.432 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_stat": 0, 2024-04-21T09:01:54.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_create": 10, 2024-04-21T09:01:54.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_read": 0, 2024-04-21T09:01:54.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_write": 1488, 2024-04-21T09:01:54.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_writefull": 26, 2024-04-21T09:01:54.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_writesame": 0, 2024-04-21T09:01:54.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_append": 0, 2024-04-21T09:01:54.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_zero": 0, 2024-04-21T09:01:54.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_truncate": 1, 2024-04-21T09:01:54.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_delete": 17, 2024-04-21T09:01:54.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_mapext": 0, 2024-04-21T09:01:54.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_sparse_read": 0, 2024-04-21T09:01:54.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_clonerange": 0, 2024-04-21T09:01:54.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_getxattr": 1998, 2024-04-21T09:01:54.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_setxattr": 20, 2024-04-21T09:01:54.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_cmpxattr": 0, 2024-04-21T09:01:54.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_rmxattr": 0, 2024-04-21T09:01:54.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_resetxattrs": 0, 2024-04-21T09:01:54.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_call": 0, 2024-04-21T09:01:54.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_watch": 0, 2024-04-21T09:01:54.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_notify": 0, 2024-04-21T09:01:54.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T09:01:54.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_pgls": 0, 2024-04-21T09:01:54.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_pgls_filter": 0, 2024-04-21T09:01:54.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_other": 0, 2024-04-21T09:01:54.435 INFO:teuthology.orchestra.run.smithi171.stdout: "linger_active": 0, 2024-04-21T09:01:54.436 INFO:teuthology.orchestra.run.smithi171.stdout: "linger_send": 0, 2024-04-21T09:01:54.436 INFO:teuthology.orchestra.run.smithi171.stdout: "linger_resend": 0, 2024-04-21T09:01:54.436 INFO:teuthology.orchestra.run.smithi171.stdout: "linger_ping": 0, 2024-04-21T09:01:54.436 INFO:teuthology.orchestra.run.smithi171.stdout: "poolop_active": 0, 2024-04-21T09:01:54.436 INFO:teuthology.orchestra.run.smithi171.stdout: "poolop_send": 0, 2024-04-21T09:01:54.436 INFO:teuthology.orchestra.run.smithi171.stdout: "poolop_resend": 0, 2024-04-21T09:01:54.436 INFO:teuthology.orchestra.run.smithi171.stdout: "poolstat_active": 0, 2024-04-21T09:01:54.436 INFO:teuthology.orchestra.run.smithi171.stdout: "poolstat_send": 0, 2024-04-21T09:01:54.437 INFO:teuthology.orchestra.run.smithi171.stdout: "poolstat_resend": 0, 2024-04-21T09:01:54.437 INFO:teuthology.orchestra.run.smithi171.stdout: "statfs_active": 0, 2024-04-21T09:01:54.437 INFO:teuthology.orchestra.run.smithi171.stdout: "statfs_send": 0, 2024-04-21T09:01:54.437 INFO:teuthology.orchestra.run.smithi171.stdout: "statfs_resend": 0, 2024-04-21T09:01:54.437 INFO:teuthology.orchestra.run.smithi171.stdout: "command_active": 0, 2024-04-21T09:01:54.437 INFO:teuthology.orchestra.run.smithi171.stdout: "command_send": 0, 2024-04-21T09:01:54.437 INFO:teuthology.orchestra.run.smithi171.stdout: "command_resend": 0, 2024-04-21T09:01:54.437 INFO:teuthology.orchestra.run.smithi171.stdout: "map_epoch": 102, 2024-04-21T09:01:54.437 INFO:teuthology.orchestra.run.smithi171.stdout: "map_full": 0, 2024-04-21T09:01:54.438 INFO:teuthology.orchestra.run.smithi171.stdout: "map_inc": 13, 2024-04-21T09:01:54.438 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_sessions": 12, 2024-04-21T09:01:54.438 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_session_open": 12, 2024-04-21T09:01:54.438 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_session_close": 0, 2024-04-21T09:01:54.438 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_laggy": 0, 2024-04-21T09:01:54.438 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_wr": 252, 2024-04-21T09:01:54.438 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_rd": 0, 2024-04-21T09:01:54.438 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_del": 123 2024-04-21T09:01:54.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.439 INFO:teuthology.orchestra.run.smithi171.stdout: "oft": { 2024-04-21T09:01:54.439 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_total_objs": 1, 2024-04-21T09:01:54.439 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_total_kv_pairs": 0, 2024-04-21T09:01:54.439 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_total_updates": 0, 2024-04-21T09:01:54.439 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_total_removes": 0 2024-04-21T09:01:54.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.439 INFO:teuthology.orchestra.run.smithi171.stdout: "osd-slow-ops": { 2024-04-21T09:01:54.440 INFO:teuthology.orchestra.run.smithi171.stdout: "slow_ops_count": 313 2024-04-21T09:01:54.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.440 INFO:teuthology.orchestra.run.smithi171.stdout: "purge_queue": { 2024-04-21T09:01:54.440 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executing_ops": 0, 2024-04-21T09:01:54.440 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T09:01:54.440 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executing": 0, 2024-04-21T09:01:54.440 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executing_high_water": 0, 2024-04-21T09:01:54.440 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executed_ops": 0, 2024-04-21T09:01:54.440 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executed": 0, 2024-04-21T09:01:54.441 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_item_in_journal": 0 2024-04-21T09:01:54.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.441 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T09:01:54.441 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:01:54.441 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 104857600, 2024-04-21T09:01:54.441 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:01:54.441 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 23493, 2024-04-21T09:01:54.441 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 6746736, 2024-04-21T09:01:54.441 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:54.442 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 23493, 2024-04-21T09:01:54.442 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 0, 2024-04-21T09:01:54.442 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 0, 2024-04-21T09:01:54.442 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 23493, 2024-04-21T09:01:54.442 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 6746736, 2024-04-21T09:01:54.442 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:01:54.442 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.442 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.442 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.443 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:54.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.443 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-objecter_bytes": { 2024-04-21T09:01:54.443 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:01:54.443 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 104857600, 2024-04-21T09:01:54.443 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:01:54.443 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 0, 2024-04-21T09:01:54.443 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 0, 2024-04-21T09:01:54.444 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:54.444 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:54.444 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 3728, 2024-04-21T09:01:54.444 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 45612762, 2024-04-21T09:01:54.444 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 3728, 2024-04-21T09:01:54.444 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 45612762, 2024-04-21T09:01:54.444 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:01:54.444 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.444 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.445 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.445 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:54.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.445 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-objecter_ops": { 2024-04-21T09:01:54.445 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:01:54.445 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 1024, 2024-04-21T09:01:54.445 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:01:54.445 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 0, 2024-04-21T09:01:54.445 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 0, 2024-04-21T09:01:54.446 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:54.446 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:54.446 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 3728, 2024-04-21T09:01:54.446 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 3728, 2024-04-21T09:01:54.446 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 3728, 2024-04-21T09:01:54.446 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 3728, 2024-04-21T09:01:54.446 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:01:54.446 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.446 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.447 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.447 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:54.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.447 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-write_buf_throttle": { 2024-04-21T09:01:54.447 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:01:54.447 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 3758096384, 2024-04-21T09:01:54.447 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:01:54.447 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 0, 2024-04-21T09:01:54.448 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 0, 2024-04-21T09:01:54.448 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:54.448 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:54.448 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 0, 2024-04-21T09:01:54.448 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 0, 2024-04-21T09:01:54.448 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 0, 2024-04-21T09:01:54.448 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 0, 2024-04-21T09:01:54.448 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:01:54.448 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.449 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.449 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.449 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:54.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:54.449 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-write_buf_throttle-0x5581bc5d8500": { 2024-04-21T09:01:54.449 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:01:54.449 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 3758096384, 2024-04-21T09:01:54.449 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:01:54.449 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 17464, 2024-04-21T09:01:54.450 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 45325288, 2024-04-21T09:01:54.450 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:54.450 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 17464, 2024-04-21T09:01:54.450 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 0, 2024-04-21T09:01:54.450 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 0, 2024-04-21T09:01:54.450 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 1488, 2024-04-21T09:01:54.450 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 45325288, 2024-04-21T09:01:54.450 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:01:54.450 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:54.451 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:54.451 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:54.451 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:54.451 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:54.451 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-21T09:01:55.369 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-21T09:01:55.369 INFO:tasks.check_counter:Daemon mds.c mds.imported=6 2024-04-21T09:01:55.369 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-21T09:01:55.369 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-21T09:01:55.370 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T09:01:55.370 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=2 2024-04-21T09:01:55.370 DEBUG:tasks.check_counter:Getting stats from f 2024-04-21T09:01:55.370 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-21T09:01:55.615 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T09:01:56.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:56 smithi152 ceph-mon[31401]: pgmap v333: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T09:01:56.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:56 smithi028 ceph-mon[28003]: pgmap v333: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T09:01:56.358 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:56 smithi171 ceph-mon[31358]: pgmap v333: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-21T09:01:57.374 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-21T09:01:57.375 INFO:teuthology.orchestra.run.smithi171.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T09:01:57.375 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_messages": 945, 2024-04-21T09:01:57.375 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages": 947, 2024-04-21T09:01:57.375 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_bytes": 4495569, 2024-04-21T09:01:57.375 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_bytes": 300520, 2024-04-21T09:01:57.375 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_created_connections": 6, 2024-04-21T09:01:57.375 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_active_connections": 2, 2024-04-21T09:01:57.376 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_total_time": 0.105202764, 2024-04-21T09:01:57.376 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_send_time": 0.030982419, 2024-04-21T09:01:57.376 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_recv_time": 0.029820066, 2024-04-21T09:01:57.376 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_fast_dispatch_time": 0.034491026, 2024-04-21T09:01:57.376 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:57.376 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 945, 2024-04-21T09:01:57.376 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.053494081, 2024-04-21T09:01:57.376 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000056607 2024-04-21T09:01:57.377 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.377 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:57.377 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.377 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.377 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.377 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.377 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T09:01:57.377 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T09:01:57.378 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.378 INFO:teuthology.orchestra.run.smithi171.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T09:01:57.378 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_messages": 1197, 2024-04-21T09:01:57.378 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages": 1182, 2024-04-21T09:01:57.378 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_bytes": 2390260, 2024-04-21T09:01:57.378 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_bytes": 438002, 2024-04-21T09:01:57.378 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_created_connections": 4, 2024-04-21T09:01:57.379 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_active_connections": 3, 2024-04-21T09:01:57.379 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_total_time": 0.121742355, 2024-04-21T09:01:57.379 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_send_time": 0.037281768, 2024-04-21T09:01:57.379 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_recv_time": 0.055351749, 2024-04-21T09:01:57.379 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_fast_dispatch_time": 0.033129783, 2024-04-21T09:01:57.379 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:57.379 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 1181, 2024-04-21T09:01:57.379 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.056749561, 2024-04-21T09:01:57.379 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000048052 2024-04-21T09:01:57.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.380 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:57.380 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.380 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.380 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.380 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_encrypted_bytes": 1633616, 2024-04-21T09:01:57.380 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_encrypted_bytes": 137744 2024-04-21T09:01:57.381 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.381 INFO:teuthology.orchestra.run.smithi171.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T09:01:57.381 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_messages": 401, 2024-04-21T09:01:57.381 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages": 564, 2024-04-21T09:01:57.381 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_bytes": 4297513, 2024-04-21T09:01:57.381 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_bytes": 575755, 2024-04-21T09:01:57.381 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_created_connections": 6, 2024-04-21T09:01:57.381 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_active_connections": 4, 2024-04-21T09:01:57.382 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_total_time": 0.050003083, 2024-04-21T09:01:57.382 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_send_time": 0.019135654, 2024-04-21T09:01:57.382 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_recv_time": 0.013476530, 2024-04-21T09:01:57.382 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_fast_dispatch_time": 0.012525456, 2024-04-21T09:01:57.382 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:01:57.382 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 562, 2024-04-21T09:01:57.382 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.032433858, 2024-04-21T09:01:57.382 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000057711 2024-04-21T09:01:57.383 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.383 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:01:57.383 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.383 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.383 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.383 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.383 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-21T09:01:57.383 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_encrypted_bytes": 450144 2024-04-21T09:01:57.383 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.384 INFO:teuthology.orchestra.run.smithi171.stdout: "cct": { 2024-04-21T09:01:57.384 INFO:teuthology.orchestra.run.smithi171.stdout: "total_workers": 1, 2024-04-21T09:01:57.384 INFO:teuthology.orchestra.run.smithi171.stdout: "unhealthy_workers": 0 2024-04-21T09:01:57.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.384 INFO:teuthology.orchestra.run.smithi171.stdout: "finisher-MDSRank": { 2024-04-21T09:01:57.384 INFO:teuthology.orchestra.run.smithi171.stdout: "queue_len": 0, 2024-04-21T09:01:57.384 INFO:teuthology.orchestra.run.smithi171.stdout: "complete_latency": { 2024-04-21T09:01:57.384 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 3807, 2024-04-21T09:01:57.385 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.118832734, 2024-04-21T09:01:57.385 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000031214 2024-04-21T09:01:57.385 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:57.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.385 INFO:teuthology.orchestra.run.smithi171.stdout: "finisher-PurgeQueue": { 2024-04-21T09:01:57.385 INFO:teuthology.orchestra.run.smithi171.stdout: "queue_len": 0, 2024-04-21T09:01:57.385 INFO:teuthology.orchestra.run.smithi171.stdout: "complete_latency": { 2024-04-21T09:01:57.385 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.386 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.386 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.386 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:57.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.386 INFO:teuthology.orchestra.run.smithi171.stdout: "mds": { 2024-04-21T09:01:57.386 INFO:teuthology.orchestra.run.smithi171.stdout: "request": 0, 2024-04-21T09:01:57.386 INFO:teuthology.orchestra.run.smithi171.stdout: "reply": 0, 2024-04-21T09:01:57.386 INFO:teuthology.orchestra.run.smithi171.stdout: "reply_latency": { 2024-04-21T09:01:57.387 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.387 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.387 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.387 INFO:teuthology.orchestra.run.smithi171.stdout: "slow_reply": 0, 2024-04-21T09:01:57.387 INFO:teuthology.orchestra.run.smithi171.stdout: "forward": 0, 2024-04-21T09:01:57.387 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_fetch_complete": 0, 2024-04-21T09:01:57.387 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_fetch_keys": 0, 2024-04-21T09:01:57.388 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_commit": 0, 2024-04-21T09:01:57.388 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_split": 0, 2024-04-21T09:01:57.388 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_merge": 0, 2024-04-21T09:01:57.388 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes": 937, 2024-04-21T09:01:57.388 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_top": 364, 2024-04-21T09:01:57.388 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_bottom": 573, 2024-04-21T09:01:57.388 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_pin_tail": 0, 2024-04-21T09:01:57.388 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_pinned": 85, 2024-04-21T09:01:57.388 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_expired": 0, 2024-04-21T09:01:57.389 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_with_caps": 0, 2024-04-21T09:01:57.389 INFO:teuthology.orchestra.run.smithi171.stdout: "caps": 0, 2024-04-21T09:01:57.389 INFO:teuthology.orchestra.run.smithi171.stdout: "subtrees": 12, 2024-04-21T09:01:57.389 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse": 0, 2024-04-21T09:01:57.389 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_hit": 0, 2024-04-21T09:01:57.389 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_forward": 0, 2024-04-21T09:01:57.389 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_discover": 0, 2024-04-21T09:01:57.389 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_dir_fetch": 0, 2024-04-21T09:01:57.390 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_remote_ino": 0, 2024-04-21T09:01:57.390 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_lock": 0, 2024-04-21T09:01:57.390 INFO:teuthology.orchestra.run.smithi171.stdout: "load_cent": 0, 2024-04-21T09:01:57.390 INFO:teuthology.orchestra.run.smithi171.stdout: "q": 0, 2024-04-21T09:01:57.390 INFO:teuthology.orchestra.run.smithi171.stdout: "exported": 0, 2024-04-21T09:01:57.390 INFO:teuthology.orchestra.run.smithi171.stdout: "exported_inodes": 0, 2024-04-21T09:01:57.390 INFO:teuthology.orchestra.run.smithi171.stdout: "imported": 0, 2024-04-21T09:01:57.390 INFO:teuthology.orchestra.run.smithi171.stdout: "imported_inodes": 0, 2024-04-21T09:01:57.390 INFO:teuthology.orchestra.run.smithi171.stdout: "openino_dir_fetch": 0, 2024-04-21T09:01:57.391 INFO:teuthology.orchestra.run.smithi171.stdout: "openino_backtrace_fetch": 0, 2024-04-21T09:01:57.391 INFO:teuthology.orchestra.run.smithi171.stdout: "openino_peer_discover": 0, 2024-04-21T09:01:57.391 INFO:teuthology.orchestra.run.smithi171.stdout: "root_rfiles": 325, 2024-04-21T09:01:57.391 INFO:teuthology.orchestra.run.smithi171.stdout: "root_rbytes": 13424230466, 2024-04-21T09:01:57.391 INFO:teuthology.orchestra.run.smithi171.stdout: "root_rsnaps": 6, 2024-04-21T09:01:57.391 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_backtrace_fetch": 0, 2024-04-21T09:01:57.391 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_set_tag": 0, 2024-04-21T09:01:57.391 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T09:01:57.392 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_inotable_repaired": 0, 2024-04-21T09:01:57.392 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_dir_inodes": 0, 2024-04-21T09:01:57.392 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T09:01:57.392 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_dirfrag_rstats": 0, 2024-04-21T09:01:57.392 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_file_inodes": 0, 2024-04-21T09:01:57.392 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_inode_file_caps": 0, 2024-04-21T09:01:57.392 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_cap_op_revoke": 0, 2024-04-21T09:01:57.393 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_cap_op_grant": 0, 2024-04-21T09:01:57.393 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T09:01:57.393 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T09:01:57.393 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T09:01:57.393 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_client_caps": 0, 2024-04-21T09:01:57.393 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_client_caps_dirty": 0, 2024-04-21T09:01:57.393 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_client_cap_release": 0, 2024-04-21T09:01:57.393 INFO:teuthology.orchestra.run.smithi171.stdout: "process_request_cap_release": 0 2024-04-21T09:01:57.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.394 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_cache": { 2024-04-21T09:01:57.394 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_update": 0, 2024-04-21T09:01:57.394 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_update_receipt": 0, 2024-04-21T09:01:57.394 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_try_discover": 0, 2024-04-21T09:01:57.394 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_send_discover": 0, 2024-04-21T09:01:57.394 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_handle_discover": 0, 2024-04-21T09:01:57.394 INFO:teuthology.orchestra.run.smithi171.stdout: "num_strays": 0, 2024-04-21T09:01:57.394 INFO:teuthology.orchestra.run.smithi171.stdout: "num_strays_delayed": 0, 2024-04-21T09:01:57.395 INFO:teuthology.orchestra.run.smithi171.stdout: "num_strays_enqueuing": 0, 2024-04-21T09:01:57.395 INFO:teuthology.orchestra.run.smithi171.stdout: "strays_created": 0, 2024-04-21T09:01:57.395 INFO:teuthology.orchestra.run.smithi171.stdout: "strays_enqueued": 0, 2024-04-21T09:01:57.395 INFO:teuthology.orchestra.run.smithi171.stdout: "strays_reintegrated": 0, 2024-04-21T09:01:57.395 INFO:teuthology.orchestra.run.smithi171.stdout: "strays_migrated": 0, 2024-04-21T09:01:57.395 INFO:teuthology.orchestra.run.smithi171.stdout: "num_recovering_processing": 0, 2024-04-21T09:01:57.395 INFO:teuthology.orchestra.run.smithi171.stdout: "num_recovering_enqueued": 0, 2024-04-21T09:01:57.395 INFO:teuthology.orchestra.run.smithi171.stdout: "num_recovering_prioritized": 0, 2024-04-21T09:01:57.396 INFO:teuthology.orchestra.run.smithi171.stdout: "recovery_started": 0, 2024-04-21T09:01:57.396 INFO:teuthology.orchestra.run.smithi171.stdout: "recovery_completed": 0, 2024-04-21T09:01:57.396 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_quiesce_path": 0, 2024-04-21T09:01:57.396 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_quiesce_inode": 0, 2024-04-21T09:01:57.396 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T09:01:57.396 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_exportdir": 0, 2024-04-21T09:01:57.396 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_flush": 0, 2024-04-21T09:01:57.396 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_fragmentdir": 0, 2024-04-21T09:01:57.396 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_fragstats": 0, 2024-04-21T09:01:57.397 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_inodestats": 0 2024-04-21T09:01:57.397 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.397 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_log": { 2024-04-21T09:01:57.397 INFO:teuthology.orchestra.run.smithi171.stdout: "evlrg": 0, 2024-04-21T09:01:57.397 INFO:teuthology.orchestra.run.smithi171.stdout: "evadd": 0, 2024-04-21T09:01:57.397 INFO:teuthology.orchestra.run.smithi171.stdout: "evex": 0, 2024-04-21T09:01:57.397 INFO:teuthology.orchestra.run.smithi171.stdout: "evtrm": 0, 2024-04-21T09:01:57.397 INFO:teuthology.orchestra.run.smithi171.stdout: "ev": 3321, 2024-04-21T09:01:57.397 INFO:teuthology.orchestra.run.smithi171.stdout: "evexg": 0, 2024-04-21T09:01:57.398 INFO:teuthology.orchestra.run.smithi171.stdout: "evexd": 0, 2024-04-21T09:01:57.398 INFO:teuthology.orchestra.run.smithi171.stdout: "segadd": 0, 2024-04-21T09:01:57.398 INFO:teuthology.orchestra.run.smithi171.stdout: "segex": 0, 2024-04-21T09:01:57.398 INFO:teuthology.orchestra.run.smithi171.stdout: "segtrm": 0, 2024-04-21T09:01:57.398 INFO:teuthology.orchestra.run.smithi171.stdout: "seg": 2, 2024-04-21T09:01:57.398 INFO:teuthology.orchestra.run.smithi171.stdout: "segmjr": 21, 2024-04-21T09:01:57.398 INFO:teuthology.orchestra.run.smithi171.stdout: "segexg": 0, 2024-04-21T09:01:57.398 INFO:teuthology.orchestra.run.smithi171.stdout: "segexd": 0, 2024-04-21T09:01:57.399 INFO:teuthology.orchestra.run.smithi171.stdout: "expos": 12886343, 2024-04-21T09:01:57.399 INFO:teuthology.orchestra.run.smithi171.stdout: "wrpos": 13100644, 2024-04-21T09:01:57.399 INFO:teuthology.orchestra.run.smithi171.stdout: "rdpos": 13100508, 2024-04-21T09:01:57.399 INFO:teuthology.orchestra.run.smithi171.stdout: "jlat": { 2024-04-21T09:01:57.399 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.399 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.399 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.400 INFO:teuthology.orchestra.run.smithi171.stdout: "replayed": 3321 2024-04-21T09:01:57.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.400 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_mem": { 2024-04-21T09:01:57.400 INFO:teuthology.orchestra.run.smithi171.stdout: "ino": 281, 2024-04-21T09:01:57.400 INFO:teuthology.orchestra.run.smithi171.stdout: "ino+": 799, 2024-04-21T09:01:57.400 INFO:teuthology.orchestra.run.smithi171.stdout: "ino-": 518, 2024-04-21T09:01:57.400 INFO:teuthology.orchestra.run.smithi171.stdout: "dir": 92, 2024-04-21T09:01:57.401 INFO:teuthology.orchestra.run.smithi171.stdout: "dir+": 111, 2024-04-21T09:01:57.401 INFO:teuthology.orchestra.run.smithi171.stdout: "dir-": 19, 2024-04-21T09:01:57.401 INFO:teuthology.orchestra.run.smithi171.stdout: "dn": 937, 2024-04-21T09:01:57.401 INFO:teuthology.orchestra.run.smithi171.stdout: "dn+": 1042, 2024-04-21T09:01:57.401 INFO:teuthology.orchestra.run.smithi171.stdout: "dn-": 105, 2024-04-21T09:01:57.401 INFO:teuthology.orchestra.run.smithi171.stdout: "cap": 0, 2024-04-21T09:01:57.401 INFO:teuthology.orchestra.run.smithi171.stdout: "cap+": 0, 2024-04-21T09:01:57.401 INFO:teuthology.orchestra.run.smithi171.stdout: "cap-": 0, 2024-04-21T09:01:57.402 INFO:teuthology.orchestra.run.smithi171.stdout: "rss": 53444, 2024-04-21T09:01:57.402 INFO:teuthology.orchestra.run.smithi171.stdout: "heap": 215416 2024-04-21T09:01:57.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.402 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_server": { 2024-04-21T09:01:57.402 INFO:teuthology.orchestra.run.smithi171.stdout: "dispatch_client_request": 0, 2024-04-21T09:01:57.402 INFO:teuthology.orchestra.run.smithi171.stdout: "dispatch_server_request": 0, 2024-04-21T09:01:57.402 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_client_request": 0, 2024-04-21T09:01:57.402 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_client_session": 0, 2024-04-21T09:01:57.403 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_peer_request": 0, 2024-04-21T09:01:57.403 INFO:teuthology.orchestra.run.smithi171.stdout: "req_create_latency": { 2024-04-21T09:01:57.403 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.403 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.403 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.403 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.403 INFO:teuthology.orchestra.run.smithi171.stdout: "req_getattr_latency": { 2024-04-21T09:01:57.403 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.403 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.404 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.404 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.404 INFO:teuthology.orchestra.run.smithi171.stdout: "req_getfilelock_latency": { 2024-04-21T09:01:57.404 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.404 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.404 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.404 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.404 INFO:teuthology.orchestra.run.smithi171.stdout: "req_link_latency": { 2024-04-21T09:01:57.405 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.405 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.405 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.405 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookup_latency": { 2024-04-21T09:01:57.405 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.405 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.406 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.406 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.406 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookuphash_latency": { 2024-04-21T09:01:57.406 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.406 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.406 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.406 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.406 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookupino_latency": { 2024-04-21T09:01:57.407 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.407 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.407 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.407 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookupname_latency": { 2024-04-21T09:01:57.407 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.407 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.407 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.408 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookupparent_latency": { 2024-04-21T09:01:57.408 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.408 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.408 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.408 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookupsnap_latency": { 2024-04-21T09:01:57.408 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.408 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.409 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.409 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.409 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lssnap_latency": { 2024-04-21T09:01:57.409 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.409 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.409 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.409 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.410 INFO:teuthology.orchestra.run.smithi171.stdout: "req_mkdir_latency": { 2024-04-21T09:01:57.410 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.410 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.410 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.411 INFO:teuthology.orchestra.run.smithi171.stdout: "req_mknod_latency": { 2024-04-21T09:01:57.411 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.411 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.411 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.411 INFO:teuthology.orchestra.run.smithi171.stdout: "req_mksnap_latency": { 2024-04-21T09:01:57.411 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.412 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.412 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.412 INFO:teuthology.orchestra.run.smithi171.stdout: "req_open_latency": { 2024-04-21T09:01:57.412 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.412 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.412 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.412 INFO:teuthology.orchestra.run.smithi171.stdout: "req_readdir_latency": { 2024-04-21T09:01:57.413 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.413 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.413 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.413 INFO:teuthology.orchestra.run.smithi171.stdout: "req_rename_latency": { 2024-04-21T09:01:57.413 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.413 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.413 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.414 INFO:teuthology.orchestra.run.smithi171.stdout: "req_renamesnap_latency": { 2024-04-21T09:01:57.414 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.414 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.414 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.414 INFO:teuthology.orchestra.run.smithi171.stdout: "req_snapdiff_latency": { 2024-04-21T09:01:57.414 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.414 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.415 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.415 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.415 INFO:teuthology.orchestra.run.smithi171.stdout: "req_rmdir_latency": { 2024-04-21T09:01:57.415 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.415 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.415 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.415 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.415 INFO:teuthology.orchestra.run.smithi171.stdout: "req_rmsnap_latency": { 2024-04-21T09:01:57.416 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.416 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.416 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.416 INFO:teuthology.orchestra.run.smithi171.stdout: "req_rmxattr_latency": { 2024-04-21T09:01:57.416 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.416 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.416 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.417 INFO:teuthology.orchestra.run.smithi171.stdout: "req_setattr_latency": { 2024-04-21T09:01:57.417 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.417 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.417 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.417 INFO:teuthology.orchestra.run.smithi171.stdout: "req_setdirlayout_latency": { 2024-04-21T09:01:57.417 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.417 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.418 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.418 INFO:teuthology.orchestra.run.smithi171.stdout: "req_setfilelock_latency": { 2024-04-21T09:01:57.418 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.418 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.418 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.418 INFO:teuthology.orchestra.run.smithi171.stdout: "req_setlayout_latency": { 2024-04-21T09:01:57.419 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.419 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.419 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.419 INFO:teuthology.orchestra.run.smithi171.stdout: "req_setxattr_latency": { 2024-04-21T09:01:57.419 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.419 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.419 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.420 INFO:teuthology.orchestra.run.smithi171.stdout: "req_symlink_latency": { 2024-04-21T09:01:57.420 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.420 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.420 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.420 INFO:teuthology.orchestra.run.smithi171.stdout: "req_unlink_latency": { 2024-04-21T09:01:57.420 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.420 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.420 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.421 INFO:teuthology.orchestra.run.smithi171.stdout: "cap_revoke_eviction": 0, 2024-04-21T09:01:57.421 INFO:teuthology.orchestra.run.smithi171.stdout: "cap_acquisition_throttle": 0, 2024-04-21T09:01:57.421 INFO:teuthology.orchestra.run.smithi171.stdout: "req_getvxattr_latency": { 2024-04-21T09:01:57.421 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.421 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.421 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.422 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:57.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.422 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_sessions": { 2024-04-21T09:01:57.422 INFO:teuthology.orchestra.run.smithi171.stdout: "session_count": 2, 2024-04-21T09:01:57.422 INFO:teuthology.orchestra.run.smithi171.stdout: "session_add": 7, 2024-04-21T09:01:57.422 INFO:teuthology.orchestra.run.smithi171.stdout: "session_remove": 5, 2024-04-21T09:01:57.422 INFO:teuthology.orchestra.run.smithi171.stdout: "sessions_open": 5, 2024-04-21T09:01:57.422 INFO:teuthology.orchestra.run.smithi171.stdout: "sessions_stale": 0, 2024-04-21T09:01:57.423 INFO:teuthology.orchestra.run.smithi171.stdout: "total_load": 0, 2024-04-21T09:01:57.423 INFO:teuthology.orchestra.run.smithi171.stdout: "average_load": 0, 2024-04-21T09:01:57.423 INFO:teuthology.orchestra.run.smithi171.stdout: "avg_session_uptime": 3595, 2024-04-21T09:01:57.423 INFO:teuthology.orchestra.run.smithi171.stdout: "mdthresh_evicted": 0 2024-04-21T09:01:57.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.423 INFO:teuthology.orchestra.run.smithi171.stdout: "mempool": { 2024-04-21T09:01:57.423 INFO:teuthology.orchestra.run.smithi171.stdout: "bloom_filter_bytes": 0, 2024-04-21T09:01:57.423 INFO:teuthology.orchestra.run.smithi171.stdout: "bloom_filter_items": 0, 2024-04-21T09:01:57.423 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T09:01:57.423 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_alloc_items": 0, 2024-04-21T09:01:57.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T09:01:57.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_data_items": 0, 2024-04-21T09:01:57.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T09:01:57.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T09:01:57.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T09:01:57.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T09:01:57.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T09:01:57.424 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_other_items": 0, 2024-04-21T09:01:57.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T09:01:57.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T09:01:57.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_extent_bytes": 0, 2024-04-21T09:01:57.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_extent_items": 0, 2024-04-21T09:01:57.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_blob_bytes": 0, 2024-04-21T09:01:57.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_blob_items": 0, 2024-04-21T09:01:57.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T09:01:57.425 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T09:01:57.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T09:01:57.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T09:01:57.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T09:01:57.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_fsck_items": 0, 2024-04-21T09:01:57.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_txc_bytes": 0, 2024-04-21T09:01:57.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_txc_items": 0, 2024-04-21T09:01:57.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T09:01:57.426 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T09:01:57.427 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_bytes": 0, 2024-04-21T09:01:57.427 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_items": 0, 2024-04-21T09:01:57.427 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_bytes": 0, 2024-04-21T09:01:57.427 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_items": 0, 2024-04-21T09:01:57.427 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T09:01:57.427 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_reader_items": 0, 2024-04-21T09:01:57.427 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T09:01:57.427 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_writer_items": 0, 2024-04-21T09:01:57.427 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_anon_bytes": 202511, 2024-04-21T09:01:57.428 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_anon_items": 406, 2024-04-21T09:01:57.428 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_meta_bytes": 0, 2024-04-21T09:01:57.428 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_meta_items": 0, 2024-04-21T09:01:57.428 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_bytes": 0, 2024-04-21T09:01:57.428 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_items": 0, 2024-04-21T09:01:57.428 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_mapbl_bytes": 0, 2024-04-21T09:01:57.428 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_mapbl_items": 0, 2024-04-21T09:01:57.428 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_pglog_bytes": 0, 2024-04-21T09:01:57.428 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_pglog_items": 0, 2024-04-21T09:01:57.429 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_bytes": 9616, 2024-04-21T09:01:57.429 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_items": 185, 2024-04-21T09:01:57.429 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T09:01:57.429 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_mapping_items": 0, 2024-04-21T09:01:57.429 INFO:teuthology.orchestra.run.smithi171.stdout: "pgmap_bytes": 0, 2024-04-21T09:01:57.429 INFO:teuthology.orchestra.run.smithi171.stdout: "pgmap_items": 0, 2024-04-21T09:01:57.429 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_co_bytes": 1551309, 2024-04-21T09:01:57.429 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_co_items": 20474, 2024-04-21T09:01:57.430 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_1_bytes": 0, 2024-04-21T09:01:57.430 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_1_items": 0, 2024-04-21T09:01:57.430 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_2_bytes": 0, 2024-04-21T09:01:57.430 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_2_items": 0 2024-04-21T09:01:57.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.430 INFO:teuthology.orchestra.run.smithi171.stdout: "objecter": { 2024-04-21T09:01:57.430 INFO:teuthology.orchestra.run.smithi171.stdout: "op_active": 0, 2024-04-21T09:01:57.430 INFO:teuthology.orchestra.run.smithi171.stdout: "op_laggy": 0, 2024-04-21T09:01:57.431 INFO:teuthology.orchestra.run.smithi171.stdout: "op_send": 2284, 2024-04-21T09:01:57.431 INFO:teuthology.orchestra.run.smithi171.stdout: "op_send_bytes": 26, 2024-04-21T09:01:57.431 INFO:teuthology.orchestra.run.smithi171.stdout: "op_resend": 4, 2024-04-21T09:01:57.431 INFO:teuthology.orchestra.run.smithi171.stdout: "op_reply": 2284, 2024-04-21T09:01:57.431 INFO:teuthology.orchestra.run.smithi171.stdout: "op_latency": { 2024-04-21T09:01:57.431 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 2284, 2024-04-21T09:01:57.431 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 42.137133159, 2024-04-21T09:01:57.431 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.018448832 2024-04-21T09:01:57.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.431 INFO:teuthology.orchestra.run.smithi171.stdout: "op_inflight": 0, 2024-04-21T09:01:57.432 INFO:teuthology.orchestra.run.smithi171.stdout: "oplen_avg": { 2024-04-21T09:01:57.432 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 2284, 2024-04-21T09:01:57.432 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 2285 2024-04-21T09:01:57.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.432 INFO:teuthology.orchestra.run.smithi171.stdout: "op": 2284, 2024-04-21T09:01:57.432 INFO:teuthology.orchestra.run.smithi171.stdout: "op_r": 2284, 2024-04-21T09:01:57.432 INFO:teuthology.orchestra.run.smithi171.stdout: "op_w": 0, 2024-04-21T09:01:57.432 INFO:teuthology.orchestra.run.smithi171.stdout: "op_rmw": 0, 2024-04-21T09:01:57.433 INFO:teuthology.orchestra.run.smithi171.stdout: "op_pg": 0, 2024-04-21T09:01:57.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_stat": 1446, 2024-04-21T09:01:57.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_create": 0, 2024-04-21T09:01:57.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_read": 835, 2024-04-21T09:01:57.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_write": 0, 2024-04-21T09:01:57.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_writefull": 0, 2024-04-21T09:01:57.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_writesame": 0, 2024-04-21T09:01:57.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_append": 0, 2024-04-21T09:01:57.433 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_zero": 0, 2024-04-21T09:01:57.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_truncate": 0, 2024-04-21T09:01:57.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_delete": 0, 2024-04-21T09:01:57.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_mapext": 0, 2024-04-21T09:01:57.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_sparse_read": 0, 2024-04-21T09:01:57.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_clonerange": 0, 2024-04-21T09:01:57.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_getxattr": 2, 2024-04-21T09:01:57.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_setxattr": 0, 2024-04-21T09:01:57.434 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_cmpxattr": 0, 2024-04-21T09:01:57.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_rmxattr": 0, 2024-04-21T09:01:57.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_resetxattrs": 0, 2024-04-21T09:01:57.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_call": 0, 2024-04-21T09:01:57.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_watch": 0, 2024-04-21T09:01:57.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_notify": 0, 2024-04-21T09:01:57.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T09:01:57.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_pgls": 0, 2024-04-21T09:01:57.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_pgls_filter": 0, 2024-04-21T09:01:57.435 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_other": 0, 2024-04-21T09:01:57.435 INFO:teuthology.orchestra.run.smithi171.stdout: "linger_active": 0, 2024-04-21T09:01:57.436 INFO:teuthology.orchestra.run.smithi171.stdout: "linger_send": 0, 2024-04-21T09:01:57.436 INFO:teuthology.orchestra.run.smithi171.stdout: "linger_resend": 0, 2024-04-21T09:01:57.436 INFO:teuthology.orchestra.run.smithi171.stdout: "linger_ping": 0, 2024-04-21T09:01:57.436 INFO:teuthology.orchestra.run.smithi171.stdout: "poolop_active": 0, 2024-04-21T09:01:57.436 INFO:teuthology.orchestra.run.smithi171.stdout: "poolop_send": 0, 2024-04-21T09:01:57.436 INFO:teuthology.orchestra.run.smithi171.stdout: "poolop_resend": 0, 2024-04-21T09:01:57.436 INFO:teuthology.orchestra.run.smithi171.stdout: "poolstat_active": 0, 2024-04-21T09:01:57.436 INFO:teuthology.orchestra.run.smithi171.stdout: "poolstat_send": 0, 2024-04-21T09:01:57.436 INFO:teuthology.orchestra.run.smithi171.stdout: "poolstat_resend": 0, 2024-04-21T09:01:57.437 INFO:teuthology.orchestra.run.smithi171.stdout: "statfs_active": 0, 2024-04-21T09:01:57.437 INFO:teuthology.orchestra.run.smithi171.stdout: "statfs_send": 0, 2024-04-21T09:01:57.437 INFO:teuthology.orchestra.run.smithi171.stdout: "statfs_resend": 0, 2024-04-21T09:01:57.437 INFO:teuthology.orchestra.run.smithi171.stdout: "command_active": 0, 2024-04-21T09:01:57.437 INFO:teuthology.orchestra.run.smithi171.stdout: "command_send": 0, 2024-04-21T09:01:57.437 INFO:teuthology.orchestra.run.smithi171.stdout: "command_resend": 0, 2024-04-21T09:01:57.437 INFO:teuthology.orchestra.run.smithi171.stdout: "map_epoch": 102, 2024-04-21T09:01:57.437 INFO:teuthology.orchestra.run.smithi171.stdout: "map_full": 0, 2024-04-21T09:01:57.437 INFO:teuthology.orchestra.run.smithi171.stdout: "map_inc": 13, 2024-04-21T09:01:57.438 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_sessions": 5, 2024-04-21T09:01:57.438 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_session_open": 5, 2024-04-21T09:01:57.438 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_session_close": 0, 2024-04-21T09:01:57.438 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_laggy": 0, 2024-04-21T09:01:57.438 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_wr": 0, 2024-04-21T09:01:57.438 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_rd": 2, 2024-04-21T09:01:57.438 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_del": 0 2024-04-21T09:01:57.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.439 INFO:teuthology.orchestra.run.smithi171.stdout: "oft": { 2024-04-21T09:01:57.439 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_total_objs": 0, 2024-04-21T09:01:57.439 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_total_kv_pairs": 0, 2024-04-21T09:01:57.439 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_total_updates": 0, 2024-04-21T09:01:57.439 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_total_removes": 0 2024-04-21T09:01:57.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.439 INFO:teuthology.orchestra.run.smithi171.stdout: "osd-slow-ops": { 2024-04-21T09:01:57.439 INFO:teuthology.orchestra.run.smithi171.stdout: "slow_ops_count": 0 2024-04-21T09:01:57.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.440 INFO:teuthology.orchestra.run.smithi171.stdout: "purge_queue": { 2024-04-21T09:01:57.440 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executing_ops": 0, 2024-04-21T09:01:57.440 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T09:01:57.440 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executing": 0, 2024-04-21T09:01:57.440 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executing_high_water": 0, 2024-04-21T09:01:57.440 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executed_ops": 0, 2024-04-21T09:01:57.440 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executed": 0, 2024-04-21T09:01:57.440 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_item_in_journal": 0 2024-04-21T09:01:57.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.441 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T09:01:57.441 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:01:57.441 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 104857600, 2024-04-21T09:01:57.441 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:01:57.441 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 2539, 2024-04-21T09:01:57.441 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 10938868, 2024-04-21T09:01:57.441 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:57.441 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 2539, 2024-04-21T09:01:57.441 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 0, 2024-04-21T09:01:57.442 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 0, 2024-04-21T09:01:57.442 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 2539, 2024-04-21T09:01:57.442 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 10938868, 2024-04-21T09:01:57.442 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:01:57.442 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.442 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.442 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.443 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:57.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.443 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-objecter_bytes": { 2024-04-21T09:01:57.443 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:01:57.443 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 104857600, 2024-04-21T09:01:57.443 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:01:57.443 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 0, 2024-04-21T09:01:57.443 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 0, 2024-04-21T09:01:57.443 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:57.444 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:57.444 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 2284, 2024-04-21T09:01:57.444 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 8906350, 2024-04-21T09:01:57.444 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 2284, 2024-04-21T09:01:57.444 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 8906350, 2024-04-21T09:01:57.444 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:01:57.444 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.444 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.444 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.445 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:57.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.445 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-objecter_ops": { 2024-04-21T09:01:57.445 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:01:57.445 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 1024, 2024-04-21T09:01:57.445 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:01:57.445 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 0, 2024-04-21T09:01:57.445 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 0, 2024-04-21T09:01:57.446 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:57.446 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:57.446 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 2284, 2024-04-21T09:01:57.446 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 2284, 2024-04-21T09:01:57.446 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 2284, 2024-04-21T09:01:57.446 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 2284, 2024-04-21T09:01:57.446 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:01:57.446 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.447 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.447 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.447 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:57.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.447 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-write_buf_throttle": { 2024-04-21T09:01:57.447 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:01:57.447 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 3758096384, 2024-04-21T09:01:57.447 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:01:57.447 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 0, 2024-04-21T09:01:57.448 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 0, 2024-04-21T09:01:57.448 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:57.448 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:57.448 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 0, 2024-04-21T09:01:57.448 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 0, 2024-04-21T09:01:57.448 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 0, 2024-04-21T09:01:57.448 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 0, 2024-04-21T09:01:57.448 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:01:57.448 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.449 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.449 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.449 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:57.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:01:57.449 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-write_buf_throttle-0x5640988786e0": { 2024-04-21T09:01:57.449 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:01:57.449 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 3758096384, 2024-04-21T09:01:57.449 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:01:57.449 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 0, 2024-04-21T09:01:57.450 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 0, 2024-04-21T09:01:57.450 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:01:57.450 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 0, 2024-04-21T09:01:57.450 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 0, 2024-04-21T09:01:57.450 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 0, 2024-04-21T09:01:57.450 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 0, 2024-04-21T09:01:57.450 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 0, 2024-04-21T09:01:57.450 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:01:57.451 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:01:57.451 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:01:57.451 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:01:57.451 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:57.451 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:01:57.451 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-21T09:01:58.120 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:01:58 smithi171 ceph-mon[31358]: pgmap v334: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:58.245 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-21T09:01:58.245 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-21T09:01:58.245 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-04-21T09:01:58.246 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-21T09:01:58.246 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T09:01:58.246 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-04-21T09:01:58.246 DEBUG:tasks.check_counter:Getting stats from i 2024-04-21T09:01:58.246 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-21T09:01:58.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:01:58 smithi152 ceph-mon[31401]: pgmap v334: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:58.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:01:58 smithi028 ceph-mon[28003]: pgmap v334: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-21T09:01:58.488 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T09:02:00.323 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:00 smithi171 ceph-mon[31358]: pgmap v335: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T09:02:00.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:00 smithi152 ceph-mon[31401]: pgmap v335: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T09:02:00.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:00 smithi028 ceph-mon[28003]: pgmap v335: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-21T09:02:00.360 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-21T09:02:00.360 INFO:teuthology.orchestra.run.smithi171.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T09:02:00.360 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_messages": 1217, 2024-04-21T09:02:00.360 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages": 1181, 2024-04-21T09:02:00.360 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_bytes": 339182, 2024-04-21T09:02:00.361 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_bytes": 816643, 2024-04-21T09:02:00.361 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_created_connections": 10, 2024-04-21T09:02:00.361 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_active_connections": 5, 2024-04-21T09:02:00.361 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_total_time": 0.108930085, 2024-04-21T09:02:00.361 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_send_time": 0.040000333, 2024-04-21T09:02:00.361 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_recv_time": 0.068525267, 2024-04-21T09:02:00.361 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_fast_dispatch_time": 0.008744482, 2024-04-21T09:02:00.361 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:02:00.361 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 1180, 2024-04-21T09:02:00.362 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.086940239, 2024-04-21T09:02:00.362 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000073678 2024-04-21T09:02:00.362 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.362 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:02:00.362 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 2074, 2024-04-21T09:02:00.362 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000470502, 2024-04-21T09:02:00.362 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000226 2024-04-21T09:02:00.362 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.362 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-21T09:02:00.363 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_encrypted_bytes": 322112 2024-04-21T09:02:00.363 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.363 INFO:teuthology.orchestra.run.smithi171.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T09:02:00.363 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_messages": 121, 2024-04-21T09:02:00.363 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages": 210, 2024-04-21T09:02:00.363 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_bytes": 26150, 2024-04-21T09:02:00.363 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_bytes": 639161, 2024-04-21T09:02:00.363 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_created_connections": 50, 2024-04-21T09:02:00.364 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_active_connections": 7, 2024-04-21T09:02:00.364 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_total_time": 0.034666572, 2024-04-21T09:02:00.364 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_send_time": 0.007518726, 2024-04-21T09:02:00.364 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_recv_time": 0.024044459, 2024-04-21T09:02:00.364 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_fast_dispatch_time": 0.000800857, 2024-04-21T09:02:00.364 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:02:00.364 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 207, 2024-04-21T09:02:00.364 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.021232097, 2024-04-21T09:02:00.364 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000102570 2024-04-21T09:02:00.364 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.365 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:02:00.365 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 187, 2024-04-21T09:02:00.365 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000035212, 2024-04-21T09:02:00.365 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000188 2024-04-21T09:02:00.365 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.365 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_encrypted_bytes": 3904, 2024-04-21T09:02:00.365 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_encrypted_bytes": 139136 2024-04-21T09:02:00.365 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.365 INFO:teuthology.orchestra.run.smithi171.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T09:02:00.366 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_messages": 365, 2024-04-21T09:02:00.366 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages": 347, 2024-04-21T09:02:00.366 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_bytes": 1664932, 2024-04-21T09:02:00.366 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_bytes": 211453, 2024-04-21T09:02:00.366 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_created_connections": 9, 2024-04-21T09:02:00.366 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_active_connections": 5, 2024-04-21T09:02:00.366 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_total_time": 0.057618049, 2024-04-21T09:02:00.366 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_send_time": 0.017770562, 2024-04-21T09:02:00.366 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_recv_time": 0.042170651, 2024-04-21T09:02:00.367 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_fast_dispatch_time": 0.007487535, 2024-04-21T09:02:00.367 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:02:00.367 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 346, 2024-04-21T09:02:00.367 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.030706960, 2024-04-21T09:02:00.367 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000088748 2024-04-21T09:02:00.367 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.367 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:02:00.367 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 10, 2024-04-21T09:02:00.367 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000001860, 2024-04-21T09:02:00.368 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000186 2024-04-21T09:02:00.368 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.368 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_encrypted_bytes": 1652752, 2024-04-21T09:02:00.368 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_encrypted_bytes": 164624 2024-04-21T09:02:00.368 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.368 INFO:teuthology.orchestra.run.smithi171.stdout: "cct": { 2024-04-21T09:02:00.368 INFO:teuthology.orchestra.run.smithi171.stdout: "total_workers": 1, 2024-04-21T09:02:00.368 INFO:teuthology.orchestra.run.smithi171.stdout: "unhealthy_workers": 0 2024-04-21T09:02:00.368 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.368 INFO:teuthology.orchestra.run.smithi171.stdout: "finisher-MDSRank": { 2024-04-21T09:02:00.369 INFO:teuthology.orchestra.run.smithi171.stdout: "queue_len": 0, 2024-04-21T09:02:00.369 INFO:teuthology.orchestra.run.smithi171.stdout: "complete_latency": { 2024-04-21T09:02:00.369 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 133, 2024-04-21T09:02:00.369 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.010254474, 2024-04-21T09:02:00.369 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000077101 2024-04-21T09:02:00.369 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:02:00.369 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.369 INFO:teuthology.orchestra.run.smithi171.stdout: "finisher-PurgeQueue": { 2024-04-21T09:02:00.369 INFO:teuthology.orchestra.run.smithi171.stdout: "queue_len": 0, 2024-04-21T09:02:00.370 INFO:teuthology.orchestra.run.smithi171.stdout: "complete_latency": { 2024-04-21T09:02:00.370 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 2, 2024-04-21T09:02:00.370 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000009364, 2024-04-21T09:02:00.370 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000004682 2024-04-21T09:02:00.370 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:02:00.370 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.370 INFO:teuthology.orchestra.run.smithi171.stdout: "mds": { 2024-04-21T09:02:00.370 INFO:teuthology.orchestra.run.smithi171.stdout: "request": 23, 2024-04-21T09:02:00.370 INFO:teuthology.orchestra.run.smithi171.stdout: "reply": 8, 2024-04-21T09:02:00.371 INFO:teuthology.orchestra.run.smithi171.stdout: "reply_latency": { 2024-04-21T09:02:00.371 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 8, 2024-04-21T09:02:00.371 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.004633393, 2024-04-21T09:02:00.371 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000579174 2024-04-21T09:02:00.371 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.371 INFO:teuthology.orchestra.run.smithi171.stdout: "slow_reply": 0, 2024-04-21T09:02:00.371 INFO:teuthology.orchestra.run.smithi171.stdout: "forward": 15, 2024-04-21T09:02:00.371 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_fetch_complete": 0, 2024-04-21T09:02:00.371 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_fetch_keys": 0, 2024-04-21T09:02:00.372 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_commit": 11, 2024-04-21T09:02:00.372 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_split": 0, 2024-04-21T09:02:00.372 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_merge": 0, 2024-04-21T09:02:00.372 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes": 31, 2024-04-21T09:02:00.372 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_top": 0, 2024-04-21T09:02:00.372 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_bottom": 31, 2024-04-21T09:02:00.372 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_pin_tail": 0, 2024-04-21T09:02:00.372 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_pinned": 31, 2024-04-21T09:02:00.372 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_expired": 1, 2024-04-21T09:02:00.372 INFO:teuthology.orchestra.run.smithi171.stdout: "inodes_with_caps": 1, 2024-04-21T09:02:00.373 INFO:teuthology.orchestra.run.smithi171.stdout: "caps": 1, 2024-04-21T09:02:00.373 INFO:teuthology.orchestra.run.smithi171.stdout: "subtrees": 6, 2024-04-21T09:02:00.373 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse": 39, 2024-04-21T09:02:00.373 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_hit": 14, 2024-04-21T09:02:00.373 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_forward": 14, 2024-04-21T09:02:00.373 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_discover": 8, 2024-04-21T09:02:00.373 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_dir_fetch": 0, 2024-04-21T09:02:00.373 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_remote_ino": 0, 2024-04-21T09:02:00.373 INFO:teuthology.orchestra.run.smithi171.stdout: "traverse_lock": 0, 2024-04-21T09:02:00.374 INFO:teuthology.orchestra.run.smithi171.stdout: "load_cent": 0, 2024-04-21T09:02:00.374 INFO:teuthology.orchestra.run.smithi171.stdout: "q": 0, 2024-04-21T09:02:00.374 INFO:teuthology.orchestra.run.smithi171.stdout: "exported": 0, 2024-04-21T09:02:00.374 INFO:teuthology.orchestra.run.smithi171.stdout: "exported_inodes": 0, 2024-04-21T09:02:00.374 INFO:teuthology.orchestra.run.smithi171.stdout: "imported": 4, 2024-04-21T09:02:00.374 INFO:teuthology.orchestra.run.smithi171.stdout: "imported_inodes": 8, 2024-04-21T09:02:00.374 INFO:teuthology.orchestra.run.smithi171.stdout: "openino_dir_fetch": 0, 2024-04-21T09:02:00.374 INFO:teuthology.orchestra.run.smithi171.stdout: "openino_backtrace_fetch": 60, 2024-04-21T09:02:00.374 INFO:teuthology.orchestra.run.smithi171.stdout: "openino_peer_discover": 0, 2024-04-21T09:02:00.375 INFO:teuthology.orchestra.run.smithi171.stdout: "root_rfiles": 325, 2024-04-21T09:02:00.375 INFO:teuthology.orchestra.run.smithi171.stdout: "root_rbytes": 13424230466, 2024-04-21T09:02:00.375 INFO:teuthology.orchestra.run.smithi171.stdout: "root_rsnaps": 6, 2024-04-21T09:02:00.375 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_backtrace_fetch": 60, 2024-04-21T09:02:00.375 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_set_tag": 0, 2024-04-21T09:02:00.375 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T09:02:00.375 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_inotable_repaired": 0, 2024-04-21T09:02:00.375 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_dir_inodes": 0, 2024-04-21T09:02:00.375 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T09:02:00.376 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_dirfrag_rstats": 0, 2024-04-21T09:02:00.376 INFO:teuthology.orchestra.run.smithi171.stdout: "scrub_file_inodes": 60, 2024-04-21T09:02:00.376 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_inode_file_caps": 0, 2024-04-21T09:02:00.376 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_cap_op_revoke": 10, 2024-04-21T09:02:00.376 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_cap_op_grant": 2, 2024-04-21T09:02:00.376 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T09:02:00.376 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T09:02:00.376 INFO:teuthology.orchestra.run.smithi171.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T09:02:00.376 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_client_caps": 10, 2024-04-21T09:02:00.376 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_client_caps_dirty": 0, 2024-04-21T09:02:00.377 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_client_cap_release": 1, 2024-04-21T09:02:00.377 INFO:teuthology.orchestra.run.smithi171.stdout: "process_request_cap_release": 1 2024-04-21T09:02:00.377 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.377 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_cache": { 2024-04-21T09:02:00.377 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_update": 0, 2024-04-21T09:02:00.377 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_update_receipt": 3, 2024-04-21T09:02:00.377 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_try_discover": 2, 2024-04-21T09:02:00.377 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_send_discover": 9, 2024-04-21T09:02:00.377 INFO:teuthology.orchestra.run.smithi171.stdout: "dir_handle_discover": 0, 2024-04-21T09:02:00.378 INFO:teuthology.orchestra.run.smithi171.stdout: "num_strays": 0, 2024-04-21T09:02:00.378 INFO:teuthology.orchestra.run.smithi171.stdout: "num_strays_delayed": 0, 2024-04-21T09:02:00.378 INFO:teuthology.orchestra.run.smithi171.stdout: "num_strays_enqueuing": 0, 2024-04-21T09:02:00.378 INFO:teuthology.orchestra.run.smithi171.stdout: "strays_created": 0, 2024-04-21T09:02:00.378 INFO:teuthology.orchestra.run.smithi171.stdout: "strays_enqueued": 0, 2024-04-21T09:02:00.378 INFO:teuthology.orchestra.run.smithi171.stdout: "strays_reintegrated": 0, 2024-04-21T09:02:00.378 INFO:teuthology.orchestra.run.smithi171.stdout: "strays_migrated": 0, 2024-04-21T09:02:00.378 INFO:teuthology.orchestra.run.smithi171.stdout: "num_recovering_processing": 0, 2024-04-21T09:02:00.378 INFO:teuthology.orchestra.run.smithi171.stdout: "num_recovering_enqueued": 0, 2024-04-21T09:02:00.379 INFO:teuthology.orchestra.run.smithi171.stdout: "num_recovering_prioritized": 0, 2024-04-21T09:02:00.379 INFO:teuthology.orchestra.run.smithi171.stdout: "recovery_started": 0, 2024-04-21T09:02:00.379 INFO:teuthology.orchestra.run.smithi171.stdout: "recovery_completed": 0, 2024-04-21T09:02:00.379 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_quiesce_path": 2, 2024-04-21T09:02:00.379 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_quiesce_inode": 36, 2024-04-21T09:02:00.379 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T09:02:00.379 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_exportdir": 0, 2024-04-21T09:02:00.379 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_flush": 0, 2024-04-21T09:02:00.379 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_fragmentdir": 0, 2024-04-21T09:02:00.379 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_fragstats": 0, 2024-04-21T09:02:00.380 INFO:teuthology.orchestra.run.smithi171.stdout: "ireq_inodestats": 0 2024-04-21T09:02:00.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.380 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_log": { 2024-04-21T09:02:00.380 INFO:teuthology.orchestra.run.smithi171.stdout: "evlrg": 0, 2024-04-21T09:02:00.380 INFO:teuthology.orchestra.run.smithi171.stdout: "evadd": 24, 2024-04-21T09:02:00.380 INFO:teuthology.orchestra.run.smithi171.stdout: "evex": 0, 2024-04-21T09:02:00.380 INFO:teuthology.orchestra.run.smithi171.stdout: "evtrm": 0, 2024-04-21T09:02:00.380 INFO:teuthology.orchestra.run.smithi171.stdout: "ev": 24, 2024-04-21T09:02:00.380 INFO:teuthology.orchestra.run.smithi171.stdout: "evexg": 0, 2024-04-21T09:02:00.381 INFO:teuthology.orchestra.run.smithi171.stdout: "evexd": 0, 2024-04-21T09:02:00.381 INFO:teuthology.orchestra.run.smithi171.stdout: "segadd": 2, 2024-04-21T09:02:00.381 INFO:teuthology.orchestra.run.smithi171.stdout: "segex": 0, 2024-04-21T09:02:00.381 INFO:teuthology.orchestra.run.smithi171.stdout: "segtrm": 0, 2024-04-21T09:02:00.381 INFO:teuthology.orchestra.run.smithi171.stdout: "seg": 2, 2024-04-21T09:02:00.381 INFO:teuthology.orchestra.run.smithi171.stdout: "segmjr": 2, 2024-04-21T09:02:00.381 INFO:teuthology.orchestra.run.smithi171.stdout: "segexg": 0, 2024-04-21T09:02:00.381 INFO:teuthology.orchestra.run.smithi171.stdout: "segexd": 0, 2024-04-21T09:02:00.381 INFO:teuthology.orchestra.run.smithi171.stdout: "expos": 4194304, 2024-04-21T09:02:00.381 INFO:teuthology.orchestra.run.smithi171.stdout: "wrpos": 4248535, 2024-04-21T09:02:00.382 INFO:teuthology.orchestra.run.smithi171.stdout: "rdpos": 0, 2024-04-21T09:02:00.382 INFO:teuthology.orchestra.run.smithi171.stdout: "jlat": { 2024-04-21T09:02:00.382 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 17, 2024-04-21T09:02:00.382 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 2.643684450, 2024-04-21T09:02:00.382 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.155510850 2024-04-21T09:02:00.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.382 INFO:teuthology.orchestra.run.smithi171.stdout: "replayed": 0 2024-04-21T09:02:00.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.382 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_mem": { 2024-04-21T09:02:00.383 INFO:teuthology.orchestra.run.smithi171.stdout: "ino": 30, 2024-04-21T09:02:00.383 INFO:teuthology.orchestra.run.smithi171.stdout: "ino+": 30, 2024-04-21T09:02:00.383 INFO:teuthology.orchestra.run.smithi171.stdout: "ino-": 0, 2024-04-21T09:02:00.383 INFO:teuthology.orchestra.run.smithi171.stdout: "dir": 25, 2024-04-21T09:02:00.383 INFO:teuthology.orchestra.run.smithi171.stdout: "dir+": 26, 2024-04-21T09:02:00.383 INFO:teuthology.orchestra.run.smithi171.stdout: "dir-": 1, 2024-04-21T09:02:00.383 INFO:teuthology.orchestra.run.smithi171.stdout: "dn": 31, 2024-04-21T09:02:00.383 INFO:teuthology.orchestra.run.smithi171.stdout: "dn+": 32, 2024-04-21T09:02:00.384 INFO:teuthology.orchestra.run.smithi171.stdout: "dn-": 1, 2024-04-21T09:02:00.384 INFO:teuthology.orchestra.run.smithi171.stdout: "cap": 1, 2024-04-21T09:02:00.384 INFO:teuthology.orchestra.run.smithi171.stdout: "cap+": 11, 2024-04-21T09:02:00.384 INFO:teuthology.orchestra.run.smithi171.stdout: "cap-": 10, 2024-04-21T09:02:00.384 INFO:teuthology.orchestra.run.smithi171.stdout: "rss": 54548, 2024-04-21T09:02:00.384 INFO:teuthology.orchestra.run.smithi171.stdout: "heap": 223608 2024-04-21T09:02:00.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.384 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_server": { 2024-04-21T09:02:00.384 INFO:teuthology.orchestra.run.smithi171.stdout: "dispatch_client_request": 28, 2024-04-21T09:02:00.384 INFO:teuthology.orchestra.run.smithi171.stdout: "dispatch_server_request": 0, 2024-04-21T09:02:00.385 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_client_request": 23, 2024-04-21T09:02:00.385 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_client_session": 25, 2024-04-21T09:02:00.385 INFO:teuthology.orchestra.run.smithi171.stdout: "handle_peer_request": 29, 2024-04-21T09:02:00.385 INFO:teuthology.orchestra.run.smithi171.stdout: "req_create_latency": { 2024-04-21T09:02:00.385 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.385 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.385 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.385 INFO:teuthology.orchestra.run.smithi171.stdout: "req_getattr_latency": { 2024-04-21T09:02:00.386 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 2, 2024-04-21T09:02:00.386 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.002200607, 2024-04-21T09:02:00.386 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.001100303 2024-04-21T09:02:00.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.386 INFO:teuthology.orchestra.run.smithi171.stdout: "req_getfilelock_latency": { 2024-04-21T09:02:00.386 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.386 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.386 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.387 INFO:teuthology.orchestra.run.smithi171.stdout: "req_link_latency": { 2024-04-21T09:02:00.387 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.387 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.387 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.387 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookup_latency": { 2024-04-21T09:02:00.387 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 2, 2024-04-21T09:02:00.387 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000751286, 2024-04-21T09:02:00.387 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000375643 2024-04-21T09:02:00.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.388 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookuphash_latency": { 2024-04-21T09:02:00.388 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.388 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.388 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.388 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookupino_latency": { 2024-04-21T09:02:00.388 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.388 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.388 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.389 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.389 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookupname_latency": { 2024-04-21T09:02:00.389 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.389 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.389 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.389 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.389 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookupparent_latency": { 2024-04-21T09:02:00.389 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.389 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.389 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.390 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lookupsnap_latency": { 2024-04-21T09:02:00.390 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.390 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.390 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.390 INFO:teuthology.orchestra.run.smithi171.stdout: "req_lssnap_latency": { 2024-04-21T09:02:00.390 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.390 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.391 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.391 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.391 INFO:teuthology.orchestra.run.smithi171.stdout: "req_mkdir_latency": { 2024-04-21T09:02:00.391 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.391 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.392 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.392 INFO:teuthology.orchestra.run.smithi171.stdout: "req_mknod_latency": { 2024-04-21T09:02:00.392 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.392 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.392 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.392 INFO:teuthology.orchestra.run.smithi171.stdout: "req_mksnap_latency": { 2024-04-21T09:02:00.392 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.393 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.393 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.393 INFO:teuthology.orchestra.run.smithi171.stdout: "req_open_latency": { 2024-04-21T09:02:00.393 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.393 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.393 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.393 INFO:teuthology.orchestra.run.smithi171.stdout: "req_readdir_latency": { 2024-04-21T09:02:00.394 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 4, 2024-04-21T09:02:00.394 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.001681500, 2024-04-21T09:02:00.394 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000420375 2024-04-21T09:02:00.394 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.394 INFO:teuthology.orchestra.run.smithi171.stdout: "req_rename_latency": { 2024-04-21T09:02:00.394 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.394 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.394 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.394 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.395 INFO:teuthology.orchestra.run.smithi171.stdout: "req_renamesnap_latency": { 2024-04-21T09:02:00.395 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.395 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.395 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.395 INFO:teuthology.orchestra.run.smithi171.stdout: "req_snapdiff_latency": { 2024-04-21T09:02:00.395 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.395 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.395 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.396 INFO:teuthology.orchestra.run.smithi171.stdout: "req_rmdir_latency": { 2024-04-21T09:02:00.396 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.396 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.396 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.396 INFO:teuthology.orchestra.run.smithi171.stdout: "req_rmsnap_latency": { 2024-04-21T09:02:00.396 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.396 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.396 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.397 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.397 INFO:teuthology.orchestra.run.smithi171.stdout: "req_rmxattr_latency": { 2024-04-21T09:02:00.397 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.397 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.397 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.397 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.397 INFO:teuthology.orchestra.run.smithi171.stdout: "req_setattr_latency": { 2024-04-21T09:02:00.397 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.397 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.397 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.398 INFO:teuthology.orchestra.run.smithi171.stdout: "req_setdirlayout_latency": { 2024-04-21T09:02:00.398 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.398 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.398 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.398 INFO:teuthology.orchestra.run.smithi171.stdout: "req_setfilelock_latency": { 2024-04-21T09:02:00.399 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.399 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.399 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.399 INFO:teuthology.orchestra.run.smithi171.stdout: "req_setlayout_latency": { 2024-04-21T09:02:00.399 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.399 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.399 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.399 INFO:teuthology.orchestra.run.smithi171.stdout: "req_setxattr_latency": { 2024-04-21T09:02:00.400 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.400 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.400 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.400 INFO:teuthology.orchestra.run.smithi171.stdout: "req_symlink_latency": { 2024-04-21T09:02:00.400 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.400 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.400 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.401 INFO:teuthology.orchestra.run.smithi171.stdout: "req_unlink_latency": { 2024-04-21T09:02:00.401 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.401 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.401 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.401 INFO:teuthology.orchestra.run.smithi171.stdout: "cap_revoke_eviction": 0, 2024-04-21T09:02:00.401 INFO:teuthology.orchestra.run.smithi171.stdout: "cap_acquisition_throttle": 0, 2024-04-21T09:02:00.401 INFO:teuthology.orchestra.run.smithi171.stdout: "req_getvxattr_latency": { 2024-04-21T09:02:00.401 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.402 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.402 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.402 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:02:00.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.402 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_sessions": { 2024-04-21T09:02:00.402 INFO:teuthology.orchestra.run.smithi171.stdout: "session_count": 2, 2024-04-21T09:02:00.402 INFO:teuthology.orchestra.run.smithi171.stdout: "session_add": 4, 2024-04-21T09:02:00.402 INFO:teuthology.orchestra.run.smithi171.stdout: "session_remove": 2, 2024-04-21T09:02:00.402 INFO:teuthology.orchestra.run.smithi171.stdout: "sessions_open": 2, 2024-04-21T09:02:00.402 INFO:teuthology.orchestra.run.smithi171.stdout: "sessions_stale": 0, 2024-04-21T09:02:00.403 INFO:teuthology.orchestra.run.smithi171.stdout: "total_load": 5, 2024-04-21T09:02:00.403 INFO:teuthology.orchestra.run.smithi171.stdout: "average_load": 1, 2024-04-21T09:02:00.403 INFO:teuthology.orchestra.run.smithi171.stdout: "avg_session_uptime": 623, 2024-04-21T09:02:00.403 INFO:teuthology.orchestra.run.smithi171.stdout: "mdthresh_evicted": 0 2024-04-21T09:02:00.403 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.403 INFO:teuthology.orchestra.run.smithi171.stdout: "mempool": { 2024-04-21T09:02:00.403 INFO:teuthology.orchestra.run.smithi171.stdout: "bloom_filter_bytes": 0, 2024-04-21T09:02:00.403 INFO:teuthology.orchestra.run.smithi171.stdout: "bloom_filter_items": 0, 2024-04-21T09:02:00.403 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T09:02:00.404 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_alloc_items": 0, 2024-04-21T09:02:00.404 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T09:02:00.404 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_data_items": 0, 2024-04-21T09:02:00.404 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T09:02:00.404 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T09:02:00.404 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T09:02:00.404 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T09:02:00.404 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T09:02:00.404 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_other_items": 0, 2024-04-21T09:02:00.405 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T09:02:00.405 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T09:02:00.405 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_extent_bytes": 0, 2024-04-21T09:02:00.405 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_extent_items": 0, 2024-04-21T09:02:00.405 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_blob_bytes": 0, 2024-04-21T09:02:00.405 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_blob_items": 0, 2024-04-21T09:02:00.405 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T09:02:00.405 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T09:02:00.405 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T09:02:00.405 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T09:02:00.406 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T09:02:00.406 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_fsck_items": 0, 2024-04-21T09:02:00.406 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_txc_bytes": 0, 2024-04-21T09:02:00.406 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_txc_items": 0, 2024-04-21T09:02:00.406 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T09:02:00.406 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T09:02:00.406 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_bytes": 0, 2024-04-21T09:02:00.406 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_items": 0, 2024-04-21T09:02:00.407 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_bytes": 0, 2024-04-21T09:02:00.407 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_items": 0, 2024-04-21T09:02:00.407 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T09:02:00.407 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_reader_items": 0, 2024-04-21T09:02:00.407 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T09:02:00.407 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_writer_items": 0, 2024-04-21T09:02:00.407 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_anon_bytes": 280013, 2024-04-21T09:02:00.407 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_anon_items": 111, 2024-04-21T09:02:00.407 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_meta_bytes": 0, 2024-04-21T09:02:00.408 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_meta_items": 0, 2024-04-21T09:02:00.408 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_bytes": 0, 2024-04-21T09:02:00.408 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_items": 0, 2024-04-21T09:02:00.408 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_mapbl_bytes": 0, 2024-04-21T09:02:00.408 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_mapbl_items": 0, 2024-04-21T09:02:00.408 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_pglog_bytes": 0, 2024-04-21T09:02:00.408 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_pglog_items": 0, 2024-04-21T09:02:00.408 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_bytes": 9616, 2024-04-21T09:02:00.408 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_items": 185, 2024-04-21T09:02:00.409 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T09:02:00.409 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_mapping_items": 0, 2024-04-21T09:02:00.409 INFO:teuthology.orchestra.run.smithi171.stdout: "pgmap_bytes": 0, 2024-04-21T09:02:00.409 INFO:teuthology.orchestra.run.smithi171.stdout: "pgmap_items": 0, 2024-04-21T09:02:00.409 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_co_bytes": 126747, 2024-04-21T09:02:00.409 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_co_items": 1413, 2024-04-21T09:02:00.409 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_1_bytes": 0, 2024-04-21T09:02:00.409 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_1_items": 0, 2024-04-21T09:02:00.409 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_2_bytes": 0, 2024-04-21T09:02:00.410 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_2_items": 0 2024-04-21T09:02:00.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.410 INFO:teuthology.orchestra.run.smithi171.stdout: "objecter": { 2024-04-21T09:02:00.410 INFO:teuthology.orchestra.run.smithi171.stdout: "op_active": 0, 2024-04-21T09:02:00.410 INFO:teuthology.orchestra.run.smithi171.stdout: "op_laggy": 0, 2024-04-21T09:02:00.410 INFO:teuthology.orchestra.run.smithi171.stdout: "op_send": 122, 2024-04-21T09:02:00.410 INFO:teuthology.orchestra.run.smithi171.stdout: "op_send_bytes": 66918, 2024-04-21T09:02:00.410 INFO:teuthology.orchestra.run.smithi171.stdout: "op_resend": 30, 2024-04-21T09:02:00.411 INFO:teuthology.orchestra.run.smithi171.stdout: "op_reply": 122, 2024-04-21T09:02:00.411 INFO:teuthology.orchestra.run.smithi171.stdout: "op_latency": { 2024-04-21T09:02:00.411 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 122, 2024-04-21T09:02:00.411 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 8.811818725, 2024-04-21T09:02:00.411 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.072228022 2024-04-21T09:02:00.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.411 INFO:teuthology.orchestra.run.smithi171.stdout: "op_inflight": 0, 2024-04-21T09:02:00.411 INFO:teuthology.orchestra.run.smithi171.stdout: "oplen_avg": { 2024-04-21T09:02:00.411 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 122, 2024-04-21T09:02:00.412 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 151 2024-04-21T09:02:00.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.412 INFO:teuthology.orchestra.run.smithi171.stdout: "op": 122, 2024-04-21T09:02:00.412 INFO:teuthology.orchestra.run.smithi171.stdout: "op_r": 60, 2024-04-21T09:02:00.412 INFO:teuthology.orchestra.run.smithi171.stdout: "op_w": 62, 2024-04-21T09:02:00.412 INFO:teuthology.orchestra.run.smithi171.stdout: "op_rmw": 0, 2024-04-21T09:02:00.412 INFO:teuthology.orchestra.run.smithi171.stdout: "op_pg": 0, 2024-04-21T09:02:00.412 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_stat": 0, 2024-04-21T09:02:00.412 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_create": 10, 2024-04-21T09:02:00.412 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_read": 0, 2024-04-21T09:02:00.413 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_write": 17, 2024-04-21T09:02:00.413 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_writefull": 9, 2024-04-21T09:02:00.413 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_writesame": 0, 2024-04-21T09:02:00.413 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_append": 0, 2024-04-21T09:02:00.413 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_zero": 0, 2024-04-21T09:02:00.413 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_truncate": 0, 2024-04-21T09:02:00.413 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_delete": 6, 2024-04-21T09:02:00.413 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_mapext": 0, 2024-04-21T09:02:00.413 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_sparse_read": 0, 2024-04-21T09:02:00.414 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_clonerange": 0, 2024-04-21T09:02:00.414 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_getxattr": 60, 2024-04-21T09:02:00.414 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_setxattr": 20, 2024-04-21T09:02:00.414 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_cmpxattr": 0, 2024-04-21T09:02:00.414 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_rmxattr": 0, 2024-04-21T09:02:00.414 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_resetxattrs": 0, 2024-04-21T09:02:00.414 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_call": 0, 2024-04-21T09:02:00.414 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_watch": 0, 2024-04-21T09:02:00.415 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_notify": 0, 2024-04-21T09:02:00.415 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T09:02:00.415 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_pgls": 0, 2024-04-21T09:02:00.415 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_pgls_filter": 0, 2024-04-21T09:02:00.415 INFO:teuthology.orchestra.run.smithi171.stdout: "osdop_other": 0, 2024-04-21T09:02:00.415 INFO:teuthology.orchestra.run.smithi171.stdout: "linger_active": 0, 2024-04-21T09:02:00.415 INFO:teuthology.orchestra.run.smithi171.stdout: "linger_send": 0, 2024-04-21T09:02:00.415 INFO:teuthology.orchestra.run.smithi171.stdout: "linger_resend": 0, 2024-04-21T09:02:00.415 INFO:teuthology.orchestra.run.smithi171.stdout: "linger_ping": 0, 2024-04-21T09:02:00.416 INFO:teuthology.orchestra.run.smithi171.stdout: "poolop_active": 0, 2024-04-21T09:02:00.416 INFO:teuthology.orchestra.run.smithi171.stdout: "poolop_send": 0, 2024-04-21T09:02:00.416 INFO:teuthology.orchestra.run.smithi171.stdout: "poolop_resend": 0, 2024-04-21T09:02:00.416 INFO:teuthology.orchestra.run.smithi171.stdout: "poolstat_active": 0, 2024-04-21T09:02:00.416 INFO:teuthology.orchestra.run.smithi171.stdout: "poolstat_send": 0, 2024-04-21T09:02:00.416 INFO:teuthology.orchestra.run.smithi171.stdout: "poolstat_resend": 0, 2024-04-21T09:02:00.416 INFO:teuthology.orchestra.run.smithi171.stdout: "statfs_active": 0, 2024-04-21T09:02:00.416 INFO:teuthology.orchestra.run.smithi171.stdout: "statfs_send": 0, 2024-04-21T09:02:00.416 INFO:teuthology.orchestra.run.smithi171.stdout: "statfs_resend": 0, 2024-04-21T09:02:00.416 INFO:teuthology.orchestra.run.smithi171.stdout: "command_active": 0, 2024-04-21T09:02:00.417 INFO:teuthology.orchestra.run.smithi171.stdout: "command_send": 0, 2024-04-21T09:02:00.417 INFO:teuthology.orchestra.run.smithi171.stdout: "command_resend": 0, 2024-04-21T09:02:00.417 INFO:teuthology.orchestra.run.smithi171.stdout: "map_epoch": 102, 2024-04-21T09:02:00.417 INFO:teuthology.orchestra.run.smithi171.stdout: "map_full": 0, 2024-04-21T09:02:00.417 INFO:teuthology.orchestra.run.smithi171.stdout: "map_inc": 13, 2024-04-21T09:02:00.417 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_sessions": 11, 2024-04-21T09:02:00.417 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_session_open": 11, 2024-04-21T09:02:00.417 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_session_close": 0, 2024-04-21T09:02:00.417 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_laggy": 0, 2024-04-21T09:02:00.418 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_wr": 26, 2024-04-21T09:02:00.418 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_rd": 0, 2024-04-21T09:02:00.418 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_del": 3 2024-04-21T09:02:00.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.418 INFO:teuthology.orchestra.run.smithi171.stdout: "oft": { 2024-04-21T09:02:00.418 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_total_objs": 1, 2024-04-21T09:02:00.418 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_total_kv_pairs": 0, 2024-04-21T09:02:00.418 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_total_updates": 0, 2024-04-21T09:02:00.419 INFO:teuthology.orchestra.run.smithi171.stdout: "omap_total_removes": 0 2024-04-21T09:02:00.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.419 INFO:teuthology.orchestra.run.smithi171.stdout: "osd-slow-ops": { 2024-04-21T09:02:00.419 INFO:teuthology.orchestra.run.smithi171.stdout: "slow_ops_count": 38 2024-04-21T09:02:00.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.419 INFO:teuthology.orchestra.run.smithi171.stdout: "purge_queue": { 2024-04-21T09:02:00.419 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executing_ops": 0, 2024-04-21T09:02:00.419 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T09:02:00.420 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executing": 0, 2024-04-21T09:02:00.420 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executing_high_water": 0, 2024-04-21T09:02:00.420 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executed_ops": 0, 2024-04-21T09:02:00.420 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_executed": 0, 2024-04-21T09:02:00.420 INFO:teuthology.orchestra.run.smithi171.stdout: "pq_item_in_journal": 0 2024-04-21T09:02:00.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.420 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T09:02:00.420 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:02:00.420 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 104857600, 2024-04-21T09:02:00.421 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:02:00.421 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 1699, 2024-04-21T09:02:00.421 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 1857841, 2024-04-21T09:02:00.421 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:02:00.421 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 1699, 2024-04-21T09:02:00.421 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 0, 2024-04-21T09:02:00.421 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 0, 2024-04-21T09:02:00.421 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 1699, 2024-04-21T09:02:00.421 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 1857841, 2024-04-21T09:02:00.422 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:02:00.422 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.422 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.422 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.422 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:02:00.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.422 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-objecter_bytes": { 2024-04-21T09:02:00.422 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:02:00.423 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 104857600, 2024-04-21T09:02:00.423 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:02:00.423 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 0, 2024-04-21T09:02:00.423 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 0, 2024-04-21T09:02:00.423 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:02:00.423 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 0, 2024-04-21T09:02:00.423 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 122, 2024-04-21T09:02:00.423 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 66918, 2024-04-21T09:02:00.423 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 122, 2024-04-21T09:02:00.424 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 66918, 2024-04-21T09:02:00.424 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:02:00.424 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.424 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.424 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.424 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:02:00.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.424 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-objecter_ops": { 2024-04-21T09:02:00.425 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:02:00.425 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 1024, 2024-04-21T09:02:00.425 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:02:00.425 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 0, 2024-04-21T09:02:00.425 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 0, 2024-04-21T09:02:00.425 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:02:00.425 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 0, 2024-04-21T09:02:00.425 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 122, 2024-04-21T09:02:00.425 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 122, 2024-04-21T09:02:00.426 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 122, 2024-04-21T09:02:00.426 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 122, 2024-04-21T09:02:00.426 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:02:00.426 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.426 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.427 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.427 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:02:00.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.427 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-write_buf_throttle": { 2024-04-21T09:02:00.427 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:02:00.427 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 3758096384, 2024-04-21T09:02:00.427 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:02:00.427 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 0, 2024-04-21T09:02:00.428 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 0, 2024-04-21T09:02:00.428 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:02:00.428 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 0, 2024-04-21T09:02:00.428 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 0, 2024-04-21T09:02:00.428 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 0, 2024-04-21T09:02:00.428 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 0, 2024-04-21T09:02:00.428 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 0, 2024-04-21T09:02:00.428 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:02:00.429 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.429 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.429 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.429 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:02:00.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:00.429 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-write_buf_throttle-0x564b820aa500": { 2024-04-21T09:02:00.429 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:02:00.429 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 3758096384, 2024-04-21T09:02:00.429 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:02:00.430 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 24, 2024-04-21T09:02:00.430 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 54231, 2024-04-21T09:02:00.430 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:02:00.430 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 24, 2024-04-21T09:02:00.430 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 0, 2024-04-21T09:02:00.430 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 0, 2024-04-21T09:02:00.430 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 17, 2024-04-21T09:02:00.430 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 54231, 2024-04-21T09:02:00.430 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:02:00.431 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:00.431 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:00.431 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:00.431 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:02:00.431 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:02:00.431 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-21T09:02:01.194 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-21T09:02:01.195 INFO:tasks.check_counter:Daemon mds.i mds.imported=4 2024-04-21T09:02:01.195 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-21T09:02:01.195 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-21T09:02:01.195 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T09:02:01.195 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-04-21T09:02:01.195 DEBUG:tasks.check_counter:Getting stats from l 2024-04-21T09:02:01.195 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-21T09:02:01.433 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T09:02:02.354 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:02 smithi152 ceph-mon[31401]: pgmap v336: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T09:02:02.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:02 smithi028 ceph-mon[28003]: pgmap v336: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T09:02:02.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:02 smithi171 ceph-mon[31358]: pgmap v336: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-21T09:02:03.298 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-04-21T09:02:03.299 INFO:teuthology.orchestra.run.smithi171.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T09:02:03.299 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_messages": 4, 2024-04-21T09:02:03.299 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages": 4, 2024-04-21T09:02:03.299 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_bytes": 2992, 2024-04-21T09:02:03.300 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_bytes": 448, 2024-04-21T09:02:03.300 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_created_connections": 4, 2024-04-21T09:02:03.300 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_active_connections": 0, 2024-04-21T09:02:03.301 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_total_time": 0.001442005, 2024-04-21T09:02:03.301 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_send_time": 0.000033157, 2024-04-21T09:02:03.301 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_recv_time": 0.000612432, 2024-04-21T09:02:03.301 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T09:02:03.302 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:02:03.302 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 3, 2024-04-21T09:02:03.302 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000733717, 2024-04-21T09:02:03.302 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000244572 2024-04-21T09:02:03.303 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:03.303 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:02:03.303 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:03.303 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:03.304 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:03.304 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:03.304 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-21T09:02:03.304 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T09:02:03.305 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:03.305 INFO:teuthology.orchestra.run.smithi171.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T09:02:03.305 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_messages": 235, 2024-04-21T09:02:03.305 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages": 224, 2024-04-21T09:02:03.306 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_bytes": 1593472, 2024-04-21T09:02:03.306 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_bytes": 137664, 2024-04-21T09:02:03.306 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_created_connections": 2, 2024-04-21T09:02:03.306 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_active_connections": 1, 2024-04-21T09:02:03.307 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_total_time": 0.049135222, 2024-04-21T09:02:03.307 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_send_time": 0.014315685, 2024-04-21T09:02:03.307 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_recv_time": 0.038299976, 2024-04-21T09:02:03.308 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_fast_dispatch_time": 0.005841407, 2024-04-21T09:02:03.308 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:02:03.308 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 223, 2024-04-21T09:02:03.308 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.014634099, 2024-04-21T09:02:03.309 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000065623 2024-04-21T09:02:03.309 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:03.309 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:02:03.309 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:03.309 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:03.310 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:03.310 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:03.310 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_encrypted_bytes": 1593472, 2024-04-21T09:02:03.310 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_encrypted_bytes": 137664 2024-04-21T09:02:03.310 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:03.311 INFO:teuthology.orchestra.run.smithi171.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T09:02:03.311 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_messages": 2, 2024-04-21T09:02:03.311 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages": 171, 2024-04-21T09:02:03.311 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_bytes": 320, 2024-04-21T09:02:03.311 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_bytes": 186208, 2024-04-21T09:02:03.312 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_created_connections": 5, 2024-04-21T09:02:03.312 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_active_connections": 3, 2024-04-21T09:02:03.312 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_total_time": 0.011175973, 2024-04-21T09:02:03.312 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_send_time": 0.007544087, 2024-04-21T09:02:03.312 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_recv_time": 0.001384477, 2024-04-21T09:02:03.313 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T09:02:03.313 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T09:02:03.313 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 168, 2024-04-21T09:02:03.313 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.021884425, 2024-04-21T09:02:03.313 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000130264 2024-04-21T09:02:03.313 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:03.314 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_handle_ack_lat": { 2024-04-21T09:02:03.314 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:03.314 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:03.314 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:03.314 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:03.314 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T09:02:03.314 INFO:teuthology.orchestra.run.smithi171.stdout: "msgr_send_encrypted_bytes": 186208 2024-04-21T09:02:03.315 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:03.315 INFO:teuthology.orchestra.run.smithi171.stdout: "cct": { 2024-04-21T09:02:03.315 INFO:teuthology.orchestra.run.smithi171.stdout: "total_workers": 0, 2024-04-21T09:02:03.315 INFO:teuthology.orchestra.run.smithi171.stdout: "unhealthy_workers": 0 2024-04-21T09:02:03.315 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:03.316 INFO:teuthology.orchestra.run.smithi171.stdout: "mempool": { 2024-04-21T09:02:03.316 INFO:teuthology.orchestra.run.smithi171.stdout: "bloom_filter_bytes": 0, 2024-04-21T09:02:03.316 INFO:teuthology.orchestra.run.smithi171.stdout: "bloom_filter_items": 0, 2024-04-21T09:02:03.316 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T09:02:03.316 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_alloc_items": 0, 2024-04-21T09:02:03.317 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T09:02:03.317 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_data_items": 0, 2024-04-21T09:02:03.317 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T09:02:03.317 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T09:02:03.318 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T09:02:03.318 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T09:02:03.318 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T09:02:03.318 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_other_items": 0, 2024-04-21T09:02:03.318 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T09:02:03.318 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T09:02:03.318 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_extent_bytes": 0, 2024-04-21T09:02:03.319 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_extent_items": 0, 2024-04-21T09:02:03.319 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_blob_bytes": 0, 2024-04-21T09:02:03.319 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_blob_items": 0, 2024-04-21T09:02:03.319 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T09:02:03.319 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T09:02:03.320 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T09:02:03.320 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T09:02:03.320 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T09:02:03.320 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_fsck_items": 0, 2024-04-21T09:02:03.321 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_txc_bytes": 0, 2024-04-21T09:02:03.321 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_txc_items": 0, 2024-04-21T09:02:03.321 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T09:02:03.321 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T09:02:03.322 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_bytes": 0, 2024-04-21T09:02:03.322 INFO:teuthology.orchestra.run.smithi171.stdout: "bluestore_writing_items": 0, 2024-04-21T09:02:03.322 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_bytes": 0, 2024-04-21T09:02:03.322 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_items": 0, 2024-04-21T09:02:03.323 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T09:02:03.323 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_reader_items": 0, 2024-04-21T09:02:03.323 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T09:02:03.323 INFO:teuthology.orchestra.run.smithi171.stdout: "bluefs_file_writer_items": 0, 2024-04-21T09:02:03.324 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_anon_bytes": 148731, 2024-04-21T09:02:03.324 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_anon_items": 31, 2024-04-21T09:02:03.324 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_meta_bytes": 0, 2024-04-21T09:02:03.324 INFO:teuthology.orchestra.run.smithi171.stdout: "buffer_meta_items": 0, 2024-04-21T09:02:03.325 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_bytes": 0, 2024-04-21T09:02:03.325 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_items": 0, 2024-04-21T09:02:03.325 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_mapbl_bytes": 0, 2024-04-21T09:02:03.325 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_mapbl_items": 0, 2024-04-21T09:02:03.326 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_pglog_bytes": 0, 2024-04-21T09:02:03.326 INFO:teuthology.orchestra.run.smithi171.stdout: "osd_pglog_items": 0, 2024-04-21T09:02:03.326 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_bytes": 0, 2024-04-21T09:02:03.326 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_items": 0, 2024-04-21T09:02:03.327 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T09:02:03.327 INFO:teuthology.orchestra.run.smithi171.stdout: "osdmap_mapping_items": 0, 2024-04-21T09:02:03.327 INFO:teuthology.orchestra.run.smithi171.stdout: "pgmap_bytes": 0, 2024-04-21T09:02:03.327 INFO:teuthology.orchestra.run.smithi171.stdout: "pgmap_items": 0, 2024-04-21T09:02:03.327 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_co_bytes": 928, 2024-04-21T09:02:03.327 INFO:teuthology.orchestra.run.smithi171.stdout: "mds_co_items": 2, 2024-04-21T09:02:03.327 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_1_bytes": 0, 2024-04-21T09:02:03.327 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_1_items": 0, 2024-04-21T09:02:03.328 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_2_bytes": 0, 2024-04-21T09:02:03.328 INFO:teuthology.orchestra.run.smithi171.stdout: "unittest_2_items": 0 2024-04-21T09:02:03.328 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-04-21T09:02:03.328 INFO:teuthology.orchestra.run.smithi171.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T09:02:03.328 INFO:teuthology.orchestra.run.smithi171.stdout: "val": 0, 2024-04-21T09:02:03.328 INFO:teuthology.orchestra.run.smithi171.stdout: "max": 104857600, 2024-04-21T09:02:03.328 INFO:teuthology.orchestra.run.smithi171.stdout: "get_started": 0, 2024-04-21T09:02:03.329 INFO:teuthology.orchestra.run.smithi171.stdout: "get": 237, 2024-04-21T09:02:03.329 INFO:teuthology.orchestra.run.smithi171.stdout: "get_sum": 1560314, 2024-04-21T09:02:03.329 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_fail": 0, 2024-04-21T09:02:03.329 INFO:teuthology.orchestra.run.smithi171.stdout: "get_or_fail_success": 237, 2024-04-21T09:02:03.329 INFO:teuthology.orchestra.run.smithi171.stdout: "take": 0, 2024-04-21T09:02:03.329 INFO:teuthology.orchestra.run.smithi171.stdout: "take_sum": 0, 2024-04-21T09:02:03.329 INFO:teuthology.orchestra.run.smithi171.stdout: "put": 237, 2024-04-21T09:02:03.329 INFO:teuthology.orchestra.run.smithi171.stdout: "put_sum": 1560314, 2024-04-21T09:02:03.330 INFO:teuthology.orchestra.run.smithi171.stdout: "wait": { 2024-04-21T09:02:03.330 INFO:teuthology.orchestra.run.smithi171.stdout: "avgcount": 0, 2024-04-21T09:02:03.330 INFO:teuthology.orchestra.run.smithi171.stdout: "sum": 0.000000000, 2024-04-21T09:02:03.330 INFO:teuthology.orchestra.run.smithi171.stdout: "avgtime": 0.000000000 2024-04-21T09:02:03.330 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:02:03.330 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-04-21T09:02:03.330 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-04-21T09:02:04.013 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-21T09:02:04.013 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-21T09:02:04.014 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-21T09:02:04.014 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-21T09:02:04.014 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-21T09:02:04.014 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-04-21T09:02:04.014 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-21T09:02:04.034 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-21T09:02:04.035 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T09:02:04.035 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T09:02:04.065 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T09:02:04.066 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T09:02:04.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T09:02:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.280+0000 7f503576d640 1 -- 172.21.15.28:0/840101277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50301011a0 msgr2=0x7f5030101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.280+0000 7f503576d640 1 --2- 172.21.15.28:0/840101277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50301011a0 0x7f5030101620 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f5024009930 tx=0x7f502402bc30 comp rx=0 tx=0).stop 2024-04-21T09:02:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.280+0000 7f503576d640 1 -- 172.21.15.28:0/840101277 shutdown_connections 2024-04-21T09:02:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.280+0000 7f503576d640 1 --2- 172.21.15.28:0/840101277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5030101b60 0x7f503010dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.280+0000 7f503576d640 1 --2- 172.21.15.28:0/840101277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50301011a0 0x7f5030101620 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.280+0000 7f503576d640 1 --2- 172.21.15.28:0/840101277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5030105e30 0x7f5030106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.280+0000 7f503576d640 1 -- 172.21.15.28:0/840101277 >> 172.21.15.28:0/840101277 conn(0x7f5030077e30 msgr2=0x7f5030078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.280+0000 7f503576d640 1 -- 172.21.15.28:0/840101277 shutdown_connections 2024-04-21T09:02:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.280+0000 7f503576d640 1 -- 172.21.15.28:0/840101277 wait complete. 2024-04-21T09:02:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.281+0000 7f503576d640 1 Processor -- start 2024-04-21T09:02:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.281+0000 7f503576d640 1 -- start start 2024-04-21T09:02:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.281+0000 7f503576d640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50301011a0 0x7f5030198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.281+0000 7f503576d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5030101b60 0x7f5030198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.281+0000 7f503576d640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5030105e30 0x7f503019cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.281+0000 7f503576d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5030112240 con 0x7f5030101b60 2024-04-21T09:02:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.281+0000 7f503576d640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f50301120c0 con 0x7f5030105e30 2024-04-21T09:02:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.281+0000 7f503576d640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f50301123c0 con 0x7f50301011a0 2024-04-21T09:02:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.281+0000 7f502e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5030101b60 0x7f5030198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.281+0000 7f502e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5030101b60 0x7f5030198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35714/0 (socket says 172.21.15.28:35714) 2024-04-21T09:02:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.281+0000 7f502e7fc640 1 -- 172.21.15.28:0/1230149962 learned_addr learned my addr 172.21.15.28:0/1230149962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:02:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.281+0000 7f502f7fe640 1 --2- 172.21.15.28:0/1230149962 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5030105e30 0x7f503019cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.281+0000 7f502effd640 1 --2- 172.21.15.28:0/1230149962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50301011a0 0x7f5030198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.281+0000 7f502e7fc640 1 -- 172.21.15.28:0/1230149962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50301011a0 msgr2=0x7f5030198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.281+0000 7f502e7fc640 1 --2- 172.21.15.28:0/1230149962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50301011a0 0x7f5030198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.282+0000 7f502e7fc640 1 -- 172.21.15.28:0/1230149962 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5030105e30 msgr2=0x7f503019cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.282+0000 7f502e7fc640 1 --2- 172.21.15.28:0/1230149962 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5030105e30 0x7f503019cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.282+0000 7f502e7fc640 1 -- 172.21.15.28:0/1230149962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f503019d6c0 con 0x7f5030101b60 2024-04-21T09:02:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.282+0000 7f502effd640 1 --2- 172.21.15.28:0/1230149962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50301011a0 0x7f5030198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:02:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.282+0000 7f502e7fc640 1 --2- 172.21.15.28:0/1230149962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5030101b60 0x7f5030198c60 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f5024009230 tx=0x7f5024004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.282+0000 7f5017fff640 1 -- 172.21.15.28:0/1230149962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5024043020 con 0x7f5030101b60 2024-04-21T09:02:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.282+0000 7f503576d640 1 -- 172.21.15.28:0/1230149962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f503019d950 con 0x7f5030101b60 2024-04-21T09:02:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.282+0000 7f503576d640 1 -- 172.21.15.28:0/1230149962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50301a5240 con 0x7f5030101b60 2024-04-21T09:02:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.282+0000 7f5017fff640 1 -- 172.21.15.28:0/1230149962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5024004350 con 0x7f5030101b60 2024-04-21T09:02:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.282+0000 7f5017fff640 1 -- 172.21.15.28:0/1230149962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5024003b50 con 0x7f5030101b60 2024-04-21T09:02:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.283+0000 7f503576d640 1 -- 172.21.15.28:0/1230149962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5030068de0 con 0x7f5030101b60 2024-04-21T09:02:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.284+0000 7f5017fff640 1 -- 172.21.15.28:0/1230149962 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5024046020 con 0x7f5030101b60 2024-04-21T09:02:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.285+0000 7f5017fff640 1 --2- 172.21.15.28:0/1230149962 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f50180a3d20 0x7f50180a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:04.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.285+0000 7f502effd640 1 --2- 172.21.15.28:0/1230149962 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f50180a3d20 0x7f50180a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:04.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.285+0000 7f5017fff640 1 -- 172.21.15.28:0/1230149962 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f50240f2650 con 0x7f5030101b60 2024-04-21T09:02:04.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.285+0000 7f502effd640 1 --2- 172.21.15.28:0/1230149962 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f50180a3d20 0x7f50180a61e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f50280058c0 tx=0x7f5028005850 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:04.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.289+0000 7f5017fff640 1 -- 172.21.15.28:0/1230149962 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f502404d030 con 0x7f5030101b60 2024-04-21T09:02:04.355 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:04 smithi152 ceph-mon[31401]: pgmap v337: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T09:02:04.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:04 smithi028 ceph-mon[28003]: pgmap v337: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T09:02:04.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:04 smithi171 ceph-mon[31358]: pgmap v337: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 23 KiB/s wr, 9 op/s 2024-04-21T09:02:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.478+0000 7f503576d640 1 -- 172.21.15.28:0/1230149962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5030109350 con 0x7f5030101b60 2024-04-21T09:02:04.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.480+0000 7f5017fff640 1 -- 172.21.15.28:0/1230149962 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+25346 (secure 0 0 0) 0x7f50240bea10 con 0x7f5030101b60 2024-04-21T09:02:04.480 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T09:02:04.480 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":102,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T09:01:18.669149+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1076282159":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6800/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/926389767":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/892059008":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/2132574390":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T09:02:04.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.482+0000 7f503576d640 1 -- 172.21.15.28:0/1230149962 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f50180a3d20 msgr2=0x7f50180a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:04.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.482+0000 7f503576d640 1 --2- 172.21.15.28:0/1230149962 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f50180a3d20 0x7f50180a61e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f50280058c0 tx=0x7f5028005850 comp rx=0 tx=0).stop 2024-04-21T09:02:04.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.482+0000 7f503576d640 1 -- 172.21.15.28:0/1230149962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5030101b60 msgr2=0x7f5030198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:04.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.482+0000 7f503576d640 1 --2- 172.21.15.28:0/1230149962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5030101b60 0x7f5030198c60 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f5024009230 tx=0x7f5024004bb0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.482+0000 7f503576d640 1 -- 172.21.15.28:0/1230149962 shutdown_connections 2024-04-21T09:02:04.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.482+0000 7f503576d640 1 --2- 172.21.15.28:0/1230149962 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f50180a3d20 0x7f50180a61e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.482+0000 7f503576d640 1 --2- 172.21.15.28:0/1230149962 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5030105e30 0x7f503019cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.482+0000 7f503576d640 1 --2- 172.21.15.28:0/1230149962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5030101b60 0x7f5030198c60 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.482+0000 7f503576d640 1 --2- 172.21.15.28:0/1230149962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50301011a0 0x7f5030198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.482+0000 7f503576d640 1 -- 172.21.15.28:0/1230149962 >> 172.21.15.28:0/1230149962 conn(0x7f5030077e30 msgr2=0x7f50300feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:04.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.482+0000 7f503576d640 1 -- 172.21.15.28:0/1230149962 shutdown_connections 2024-04-21T09:02:04.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.482+0000 7f503576d640 1 -- 172.21.15.28:0/1230149962 wait complete. 2024-04-21T09:02:04.493 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:44:05.469235+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:48:43.543799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:48:44.864564+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:48:48.747842+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T09:02:04.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T09:02:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.599+0000 7f3626752640 1 -- 172.21.15.28:0/2244122258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3620101090 msgr2=0x7f3620101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.599+0000 7f3626752640 1 --2- 172.21.15.28:0/2244122258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3620101090 0x7f3620101470 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f3610009900 tx=0x7f361002bc30 comp rx=0 tx=0).stop 2024-04-21T09:02:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.599+0000 7f3626752640 1 -- 172.21.15.28:0/2244122258 shutdown_connections 2024-04-21T09:02:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.599+0000 7f3626752640 1 --2- 172.21.15.28:0/2244122258 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3620106160 0x7f362010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.599+0000 7f3626752640 1 --2- 172.21.15.28:0/2244122258 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3620101a40 0x7f3620105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.599+0000 7f3626752640 1 --2- 172.21.15.28:0/2244122258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3620101090 0x7f3620101470 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.599+0000 7f3626752640 1 -- 172.21.15.28:0/2244122258 >> 172.21.15.28:0/2244122258 conn(0x7f3620077f50 msgr2=0x7f3620078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.599+0000 7f3626752640 1 -- 172.21.15.28:0/2244122258 shutdown_connections 2024-04-21T09:02:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.599+0000 7f3626752640 1 -- 172.21.15.28:0/2244122258 wait complete. 2024-04-21T09:02:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.599+0000 7f3626752640 1 Processor -- start 2024-04-21T09:02:04.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.599+0000 7f3626752640 1 -- start start 2024-04-21T09:02:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f3626752640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3620101090 0x7f36201994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:04.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f3626752640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3620106160 0x7f3620199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f3626752640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f362019a000 0x7f36201a4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f3626752640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36201110a0 con 0x7f3620101090 2024-04-21T09:02:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f3626752640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3620110f20 con 0x7f362019a000 2024-04-21T09:02:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f3626752640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3620111220 con 0x7f3620106160 2024-04-21T09:02:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f361ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3620101090 0x7f36201994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f361ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3620101090 0x7f36201994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35738/0 (socket says 172.21.15.28:35738) 2024-04-21T09:02:04.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f361ffff640 1 -- 172.21.15.28:0/3246776726 learned_addr learned my addr 172.21.15.28:0/3246776726 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:02:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f3624cef640 1 --2- 172.21.15.28:0/3246776726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f362019a000 0x7f36201a4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f361f7fe640 1 --2- 172.21.15.28:0/3246776726 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3620106160 0x7f3620199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f361ffff640 1 -- 172.21.15.28:0/3246776726 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3620106160 msgr2=0x7f3620199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f361ffff640 1 --2- 172.21.15.28:0/3246776726 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3620106160 0x7f3620199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f361ffff640 1 -- 172.21.15.28:0/3246776726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f362019a000 msgr2=0x7f36201a4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f361ffff640 1 --2- 172.21.15.28:0/3246776726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f362019a000 0x7f36201a4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f361ffff640 1 -- 172.21.15.28:0/3246776726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36201a4d80 con 0x7f3620101090 2024-04-21T09:02:04.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f3624cef640 1 --2- 172.21.15.28:0/3246776726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f362019a000 0x7f36201a4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:02:04.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f361ffff640 1 --2- 172.21.15.28:0/3246776726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3620101090 0x7f36201994e0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f3610005940 tx=0x7f361002c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:04.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f361d7fa640 1 -- 172.21.15.28:0/3246776726 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3610043020 con 0x7f3620101090 2024-04-21T09:02:04.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f361d7fa640 1 -- 172.21.15.28:0/3246776726 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f361002cc60 con 0x7f3620101090 2024-04-21T09:02:04.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.600+0000 7f3626752640 1 -- 172.21.15.28:0/3246776726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36201a5010 con 0x7f3620101090 2024-04-21T09:02:04.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.601+0000 7f361d7fa640 1 -- 172.21.15.28:0/3246776726 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3610003c30 con 0x7f3620101090 2024-04-21T09:02:04.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.601+0000 7f3626752640 1 -- 172.21.15.28:0/3246776726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36201a5560 con 0x7f3620101090 2024-04-21T09:02:04.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.602+0000 7f3626752640 1 -- 172.21.15.28:0/3246776726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3620068d70 con 0x7f3620101090 2024-04-21T09:02:04.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.603+0000 7f361d7fa640 1 -- 172.21.15.28:0/3246776726 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3610046020 con 0x7f3620101090 2024-04-21T09:02:04.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.604+0000 7f361d7fa640 1 --2- 172.21.15.28:0/3246776726 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f36080a3d20 0x7f36080a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:04.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.604+0000 7f361f7fe640 1 --2- 172.21.15.28:0/3246776726 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f36080a3d20 0x7f36080a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:04.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.604+0000 7f361d7fa640 1 -- 172.21.15.28:0/3246776726 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f36100f35b0 con 0x7f3620101090 2024-04-21T09:02:04.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.604+0000 7f361f7fe640 1 --2- 172.21.15.28:0/3246776726 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f36080a3d20 0x7f36080a61e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3614005e10 tx=0x7f3614005da0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:04.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.608+0000 7f361d7fa640 1 -- 172.21.15.28:0/3246776726 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f361004d020 con 0x7f3620101090 2024-04-21T09:02:04.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.792+0000 7f3626752640 1 -- 172.21.15.28:0/3246776726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3620109390 con 0x7f3620101090 2024-04-21T09:02:04.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.793+0000 7f361d7fa640 1 -- 172.21.15.28:0/3246776726 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) ==== 93+0+10 (secure 0 0 0) 0x7f361003e450 con 0x7f3620101090 2024-04-21T09:02:04.793 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-21T09:02:04.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.795+0000 7f3626752640 1 -- 172.21.15.28:0/3246776726 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f36080a3d20 msgr2=0x7f36080a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:04.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.795+0000 7f3626752640 1 --2- 172.21.15.28:0/3246776726 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f36080a3d20 0x7f36080a61e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3614005e10 tx=0x7f3614005da0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.795+0000 7f3626752640 1 -- 172.21.15.28:0/3246776726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3620101090 msgr2=0x7f36201994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:04.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.795+0000 7f3626752640 1 --2- 172.21.15.28:0/3246776726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3620101090 0x7f36201994e0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f3610005940 tx=0x7f361002c860 comp rx=0 tx=0).stop 2024-04-21T09:02:04.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.795+0000 7f3626752640 1 -- 172.21.15.28:0/3246776726 shutdown_connections 2024-04-21T09:02:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.795+0000 7f3626752640 1 --2- 172.21.15.28:0/3246776726 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f36080a3d20 0x7f36080a61e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.795+0000 7f3626752640 1 --2- 172.21.15.28:0/3246776726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f362019a000 0x7f36201a4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.795+0000 7f3626752640 1 --2- 172.21.15.28:0/3246776726 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3620106160 0x7f3620199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.795+0000 7f3626752640 1 --2- 172.21.15.28:0/3246776726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3620101090 0x7f36201994e0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.795+0000 7f3626752640 1 -- 172.21.15.28:0/3246776726 >> 172.21.15.28:0/3246776726 conn(0x7f3620077f50 msgr2=0x7f36200ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.795+0000 7f3626752640 1 -- 172.21.15.28:0/3246776726 shutdown_connections 2024-04-21T09:02:04.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.795+0000 7f3626752640 1 -- 172.21.15.28:0/3246776726 wait complete. 2024-04-21T09:02:04.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T09:02:04.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.912+0000 7f031fc06640 1 -- 172.21.15.28:0/1890617248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031806a840 msgr2=0x7f031806ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:04.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.912+0000 7f031fc06640 1 --2- 172.21.15.28:0/1890617248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031806a840 0x7f031806ac20 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f0314009900 tx=0x7f031402bc30 comp rx=0 tx=0).stop 2024-04-21T09:02:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.912+0000 7f031fc06640 1 -- 172.21.15.28:0/1890617248 shutdown_connections 2024-04-21T09:02:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.912+0000 7f031fc06640 1 --2- 172.21.15.28:0/1890617248 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0318107270 0x7f031810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.912+0000 7f031fc06640 1 --2- 172.21.15.28:0/1890617248 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f031806b1f0 0x7f0318106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.912+0000 7f031fc06640 1 --2- 172.21.15.28:0/1890617248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031806a840 0x7f031806ac20 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.912+0000 7f031fc06640 1 -- 172.21.15.28:0/1890617248 >> 172.21.15.28:0/1890617248 conn(0x7f0318077eb0 msgr2=0x7f03180782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.912+0000 7f031fc06640 1 -- 172.21.15.28:0/1890617248 shutdown_connections 2024-04-21T09:02:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.912+0000 7f031fc06640 1 -- 172.21.15.28:0/1890617248 wait complete. 2024-04-21T09:02:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031fc06640 1 Processor -- start 2024-04-21T09:02:04.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031fc06640 1 -- start start 2024-04-21T09:02:04.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031fc06640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f031806a840 0x7f03181986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:04.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031fc06640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0318107270 0x7f0318198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:04.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031fc06640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031819cf90 0x7f031819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:04.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031fc06640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0318112230 con 0x7f031819cf90 2024-04-21T09:02:04.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031fc06640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f03181120b0 con 0x7f0318107270 2024-04-21T09:02:04.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031fc06640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f03181123b0 con 0x7f031806a840 2024-04-21T09:02:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031e1a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031819cf90 0x7f031819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031e1a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031819cf90 0x7f031819d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35762/0 (socket says 172.21.15.28:35762) 2024-04-21T09:02:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031e1a3640 1 -- 172.21.15.28:0/2767881174 learned_addr learned my addr 172.21.15.28:0/2767881174 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:02:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031d9a2640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f031806a840 0x7f03181986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031e1a3640 1 -- 172.21.15.28:0/2767881174 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f031806a840 msgr2=0x7f03181986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031e1a3640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f031806a840 0x7f03181986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031e1a3640 1 -- 172.21.15.28:0/2767881174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0318107270 msgr2=0x7f0318198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031d1a1640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0318107270 0x7f0318198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:04.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031e1a3640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0318107270 0x7f0318198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:04.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031e1a3640 1 -- 172.21.15.28:0/2767881174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f031819da80 con 0x7f031819cf90 2024-04-21T09:02:04.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031d9a2640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f031806a840 0x7f03181986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:02:04.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.913+0000 7f031d1a1640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0318107270 0x7f0318198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:02:04.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.914+0000 7f031e1a3640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031819cf90 0x7f031819d440 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f031000eaa0 tx=0x7f031000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:04.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.914+0000 7f030effd640 1 -- 172.21.15.28:0/2767881174 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031000cdb0 con 0x7f031819cf90 2024-04-21T09:02:04.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.914+0000 7f030effd640 1 -- 172.21.15.28:0/2767881174 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0310005020 con 0x7f031819cf90 2024-04-21T09:02:04.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.914+0000 7f030effd640 1 -- 172.21.15.28:0/2767881174 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03100116d0 con 0x7f031819cf90 2024-04-21T09:02:04.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.914+0000 7f031fc06640 1 -- 172.21.15.28:0/2767881174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f031819dd70 con 0x7f031819cf90 2024-04-21T09:02:04.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.914+0000 7f031fc06640 1 -- 172.21.15.28:0/2767881174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03181a5510 con 0x7f031819cf90 2024-04-21T09:02:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.915+0000 7f031fc06640 1 -- 172.21.15.28:0/2767881174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03180691c0 con 0x7f031819cf90 2024-04-21T09:02:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.916+0000 7f030effd640 1 -- 172.21.15.28:0/2767881174 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f03100051d0 con 0x7f031819cf90 2024-04-21T09:02:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.917+0000 7f030effd640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f03040a3da0 0x7f03040a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.917+0000 7f030effd640 1 -- 172.21.15.28:0/2767881174 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f03100d0d40 con 0x7f031819cf90 2024-04-21T09:02:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.917+0000 7f031d9a2640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f03040a3da0 0x7f03040a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.917+0000 7f031d9a2640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f03040a3da0 0x7f03040a6260 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0314036040 tx=0x7f0314005fd0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:04.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:04.921+0000 7f030effd640 1 -- 172.21.15.28:0/2767881174 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0310016030 con 0x7f031819cf90 2024-04-21T09:02:05.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.111+0000 7f031fc06640 1 -- 172.21.15.28:0/2767881174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f031806b1f0 con 0x7f031819cf90 2024-04-21T09:02:05.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.112+0000 7f030effd640 1 -- 172.21.15.28:0/2767881174 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) ==== 104+0+11 (secure 0 0 0) 0x7f031009cff0 con 0x7f031819cf90 2024-04-21T09:02:05.112 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T09:02:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.114+0000 7f031fc06640 1 -- 172.21.15.28:0/2767881174 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f03040a3da0 msgr2=0x7f03040a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.114+0000 7f031fc06640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f03040a3da0 0x7f03040a6260 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0314036040 tx=0x7f0314005fd0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.114+0000 7f031fc06640 1 -- 172.21.15.28:0/2767881174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031819cf90 msgr2=0x7f031819d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.114+0000 7f031fc06640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031819cf90 0x7f031819d440 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f031000eaa0 tx=0x7f031000ef60 comp rx=0 tx=0).stop 2024-04-21T09:02:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.114+0000 7f031fc06640 1 -- 172.21.15.28:0/2767881174 shutdown_connections 2024-04-21T09:02:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.114+0000 7f031fc06640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f03040a3da0 0x7f03040a6260 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.114+0000 7f031fc06640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031819cf90 0x7f031819d440 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.114+0000 7f031fc06640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0318107270 0x7f0318198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.114+0000 7f031fc06640 1 --2- 172.21.15.28:0/2767881174 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f031806a840 0x7f03181986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.114+0000 7f031fc06640 1 -- 172.21.15.28:0/2767881174 >> 172.21.15.28:0/2767881174 conn(0x7f0318077eb0 msgr2=0x7f03180fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.114+0000 7f031fc06640 1 -- 172.21.15.28:0/2767881174 shutdown_connections 2024-04-21T09:02:05.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.114+0000 7f031fc06640 1 -- 172.21.15.28:0/2767881174 wait complete. 2024-04-21T09:02:05.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T09:02:05.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.231+0000 7fdf623db640 1 -- 172.21.15.28:0/3538425898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c06b1f0 msgr2=0x7fdf5c106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:05.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.231+0000 7fdf623db640 1 --2- 172.21.15.28:0/3538425898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c06b1f0 0x7fdf5c106b90 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009900 tx=0x7fdf4c02bc30 comp rx=0 tx=0).stop 2024-04-21T09:02:05.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.232+0000 7fdf623db640 1 -- 172.21.15.28:0/3538425898 shutdown_connections 2024-04-21T09:02:05.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.232+0000 7fdf623db640 1 --2- 172.21.15.28:0/3538425898 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf5c107270 0x7fdf5c10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.232+0000 7fdf623db640 1 --2- 172.21.15.28:0/3538425898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c06b1f0 0x7fdf5c106b90 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.232+0000 7fdf623db640 1 --2- 172.21.15.28:0/3538425898 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf5c06a840 0x7fdf5c06ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.232+0000 7fdf623db640 1 -- 172.21.15.28:0/3538425898 >> 172.21.15.28:0/3538425898 conn(0x7fdf5c077eb0 msgr2=0x7fdf5c0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:05.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.232+0000 7fdf623db640 1 -- 172.21.15.28:0/3538425898 shutdown_connections 2024-04-21T09:02:05.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.232+0000 7fdf623db640 1 -- 172.21.15.28:0/3538425898 wait complete. 2024-04-21T09:02:05.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.232+0000 7fdf623db640 1 Processor -- start 2024-04-21T09:02:05.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.232+0000 7fdf623db640 1 -- start start 2024-04-21T09:02:05.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf623db640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf5c06a840 0x7fdf5c100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:05.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf623db640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c107270 0x7fdf5c1013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:05.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf623db640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf5c101900 0x7fdf5c1a6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:05.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf623db640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c112210 con 0x7fdf5c107270 2024-04-21T09:02:05.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf623db640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c112090 con 0x7fdf5c101900 2024-04-21T09:02:05.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf623db640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c112390 con 0x7fdf5c06a840 2024-04-21T09:02:05.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf5b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c107270 0x7fdf5c1013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:05.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf5b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c107270 0x7fdf5c1013c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35772/0 (socket says 172.21.15.28:35772) 2024-04-21T09:02:05.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf5b7fe640 1 -- 172.21.15.28:0/1453196179 learned_addr learned my addr 172.21.15.28:0/1453196179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:02:05.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf5bfff640 1 --2- 172.21.15.28:0/1453196179 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf5c06a840 0x7fdf5c100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:05.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf60978640 1 --2- 172.21.15.28:0/1453196179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf5c101900 0x7fdf5c1a6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:05.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf5b7fe640 1 -- 172.21.15.28:0/1453196179 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf5c06a840 msgr2=0x7fdf5c100e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:05.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf5b7fe640 1 --2- 172.21.15.28:0/1453196179 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf5c06a840 0x7fdf5c100e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf5b7fe640 1 -- 172.21.15.28:0/1453196179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf5c101900 msgr2=0x7fdf5c1a6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:05.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf5b7fe640 1 --2- 172.21.15.28:0/1453196179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf5c101900 0x7fdf5c1a6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf5b7fe640 1 -- 172.21.15.28:0/1453196179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5c1a7020 con 0x7fdf5c107270 2024-04-21T09:02:05.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf60978640 1 --2- 172.21.15.28:0/1453196179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf5c101900 0x7fdf5c1a6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:02:05.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf5b7fe640 1 --2- 172.21.15.28:0/1453196179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c107270 0x7fdf5c1013c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c036040 tx=0x7fdf4c02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:05.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf597fa640 1 -- 172.21.15.28:0/1453196179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c043020 con 0x7fdf5c107270 2024-04-21T09:02:05.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf597fa640 1 -- 172.21.15.28:0/1453196179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf4c02cbe0 con 0x7fdf5c107270 2024-04-21T09:02:05.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf623db640 1 -- 172.21.15.28:0/1453196179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf5c1a72b0 con 0x7fdf5c107270 2024-04-21T09:02:05.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.233+0000 7fdf597fa640 1 -- 172.21.15.28:0/1453196179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c003bf0 con 0x7fdf5c107270 2024-04-21T09:02:05.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.234+0000 7fdf623db640 1 -- 172.21.15.28:0/1453196179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf5c1a7800 con 0x7fdf5c107270 2024-04-21T09:02:05.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.234+0000 7fdf623db640 1 -- 172.21.15.28:0/1453196179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf5c0691c0 con 0x7fdf5c107270 2024-04-21T09:02:05.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.236+0000 7fdf597fa640 1 -- 172.21.15.28:0/1453196179 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdf4c046020 con 0x7fdf5c107270 2024-04-21T09:02:05.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.236+0000 7fdf597fa640 1 --2- 172.21.15.28:0/1453196179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdf480a3d50 0x7fdf480a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:05.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.236+0000 7fdf5bfff640 1 --2- 172.21.15.28:0/1453196179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdf480a3d50 0x7fdf480a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:05.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.236+0000 7fdf597fa640 1 -- 172.21.15.28:0/1453196179 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7fdf4c0f33a0 con 0x7fdf5c107270 2024-04-21T09:02:05.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.237+0000 7fdf5bfff640 1 --2- 172.21.15.28:0/1453196179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdf480a3d50 0x7fdf480a6210 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdf44005d00 tx=0x7fdf4400b410 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:05.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.240+0000 7fdf597fa640 1 -- 172.21.15.28:0/1453196179 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf4c04d020 con 0x7fdf5c107270 2024-04-21T09:02:05.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:05 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1230149962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T09:02:05.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:05 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3246776726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T09:02:05.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:05 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1230149962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T09:02:05.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:05 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3246776726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T09:02:05.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.425+0000 7fdf623db640 1 -- 172.21.15.28:0/1453196179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdf5c10a4c0 con 0x7fdf5c107270 2024-04-21T09:02:05.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.426+0000 7fdf597fa640 1 -- 172.21.15.28:0/1453196179 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) ==== 100+0+11 (secure 0 0 0) 0x7fdf4c0bf760 con 0x7fdf5c107270 2024-04-21T09:02:05.426 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-21T09:02:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.427+0000 7fdf623db640 1 -- 172.21.15.28:0/1453196179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdf480a3d50 msgr2=0x7fdf480a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.427+0000 7fdf623db640 1 --2- 172.21.15.28:0/1453196179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdf480a3d50 0x7fdf480a6210 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdf44005d00 tx=0x7fdf4400b410 comp rx=0 tx=0).stop 2024-04-21T09:02:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.427+0000 7fdf623db640 1 -- 172.21.15.28:0/1453196179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c107270 msgr2=0x7fdf5c1013c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.427+0000 7fdf623db640 1 --2- 172.21.15.28:0/1453196179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c107270 0x7fdf5c1013c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c036040 tx=0x7fdf4c02c7e0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.428+0000 7fdf623db640 1 -- 172.21.15.28:0/1453196179 shutdown_connections 2024-04-21T09:02:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.428+0000 7fdf623db640 1 --2- 172.21.15.28:0/1453196179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fdf480a3d50 0x7fdf480a6210 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.428+0000 7fdf623db640 1 --2- 172.21.15.28:0/1453196179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf5c101900 0x7fdf5c1a6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.428+0000 7fdf623db640 1 --2- 172.21.15.28:0/1453196179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c107270 0x7fdf5c1013c0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.428+0000 7fdf623db640 1 --2- 172.21.15.28:0/1453196179 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdf5c06a840 0x7fdf5c100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.428+0000 7fdf623db640 1 -- 172.21.15.28:0/1453196179 >> 172.21.15.28:0/1453196179 conn(0x7fdf5c077eb0 msgr2=0x7fdf5c108440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:05.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.428+0000 7fdf623db640 1 -- 172.21.15.28:0/1453196179 shutdown_connections 2024-04-21T09:02:05.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.428+0000 7fdf623db640 1 -- 172.21.15.28:0/1453196179 wait complete. 2024-04-21T09:02:05.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T09:02:05.481 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:05 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1230149962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T09:02:05.481 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:05 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3246776726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T09:02:05.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.543+0000 7faaa3f4b640 1 -- 172.21.15.28:0/1087137883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa9c1031d0 msgr2=0x7faa9c1035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:05.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.543+0000 7faaa3f4b640 1 --2- 172.21.15.28:0/1087137883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa9c1031d0 0x7faa9c1035b0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7faa98009900 tx=0x7faa9802bd60 comp rx=0 tx=0).stop 2024-04-21T09:02:05.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.543+0000 7faaa3f4b640 1 -- 172.21.15.28:0/1087137883 shutdown_connections 2024-04-21T09:02:05.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.543+0000 7faaa3f4b640 1 --2- 172.21.15.28:0/1087137883 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faa9c10c510 0x7faa9c10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.543+0000 7faaa3f4b640 1 --2- 172.21.15.28:0/1087137883 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faa9c103af0 0x7faa9c10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.543+0000 7faaa3f4b640 1 --2- 172.21.15.28:0/1087137883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa9c1031d0 0x7faa9c1035b0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.543+0000 7faaa3f4b640 1 -- 172.21.15.28:0/1087137883 >> 172.21.15.28:0/1087137883 conn(0x7faa9c077f50 msgr2=0x7faa9c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.544+0000 7faaa3f4b640 1 -- 172.21.15.28:0/1087137883 shutdown_connections 2024-04-21T09:02:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.544+0000 7faaa3f4b640 1 -- 172.21.15.28:0/1087137883 wait complete. 2024-04-21T09:02:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.544+0000 7faaa3f4b640 1 Processor -- start 2024-04-21T09:02:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.544+0000 7faaa3f4b640 1 -- start start 2024-04-21T09:02:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.544+0000 7faaa3f4b640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faa9c1031d0 0x7faa9c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.544+0000 7faaa3f4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa9c103af0 0x7faa9c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.544+0000 7faaa3f4b640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faa9c10c510 0x7faa9c19d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:05.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.544+0000 7faaa3f4b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa9c1121a0 con 0x7faa9c103af0 2024-04-21T09:02:05.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.544+0000 7faaa3f4b640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7faa9c112020 con 0x7faa9c1031d0 2024-04-21T09:02:05.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.544+0000 7faaa3f4b640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7faa9c112320 con 0x7faa9c10c510 2024-04-21T09:02:05.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.544+0000 7faaa14e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa9c103af0 0x7faa9c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:05.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.544+0000 7faaa14e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa9c103af0 0x7faa9c198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35788/0 (socket says 172.21.15.28:35788) 2024-04-21T09:02:05.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.544+0000 7faaa14e6640 1 -- 172.21.15.28:0/92304843 learned_addr learned my addr 172.21.15.28:0/92304843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:02:05.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.545+0000 7faaa24e8640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faa9c10c510 0x7faa9c19d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:05.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.545+0000 7faaa1ce7640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faa9c1031d0 0x7faa9c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:05.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.545+0000 7faaa14e6640 1 -- 172.21.15.28:0/92304843 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faa9c10c510 msgr2=0x7faa9c19d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:05.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.545+0000 7faaa14e6640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faa9c10c510 0x7faa9c19d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.545+0000 7faaa14e6640 1 -- 172.21.15.28:0/92304843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faa9c1031d0 msgr2=0x7faa9c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:05.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.545+0000 7faaa14e6640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faa9c1031d0 0x7faa9c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.545+0000 7faaa14e6640 1 -- 172.21.15.28:0/92304843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa9c19d730 con 0x7faa9c103af0 2024-04-21T09:02:05.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.545+0000 7faaa1ce7640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faa9c1031d0 0x7faa9c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:02:05.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.545+0000 7faaa24e8640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faa9c10c510 0x7faa9c19d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:02:05.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.545+0000 7faaa14e6640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa9c103af0 0x7faa9c198c80 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00d950 tx=0x7faa8c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:05.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.545+0000 7faa92ffd640 1 -- 172.21.15.28:0/92304843 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c017070 con 0x7faa9c103af0 2024-04-21T09:02:05.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.545+0000 7faaa3f4b640 1 -- 172.21.15.28:0/92304843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa9c19da20 con 0x7faa9c103af0 2024-04-21T09:02:05.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.545+0000 7faa92ffd640 1 -- 172.21.15.28:0/92304843 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa8c009c30 con 0x7faa9c103af0 2024-04-21T09:02:05.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.545+0000 7faa92ffd640 1 -- 172.21.15.28:0/92304843 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c00f620 con 0x7faa9c103af0 2024-04-21T09:02:05.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.545+0000 7faaa3f4b640 1 -- 172.21.15.28:0/92304843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa9c1a52b0 con 0x7faa9c103af0 2024-04-21T09:02:05.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.546+0000 7faaa3f4b640 1 -- 172.21.15.28:0/92304843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa9c068d70 con 0x7faa9c103af0 2024-04-21T09:02:05.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.547+0000 7faa92ffd640 1 -- 172.21.15.28:0/92304843 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faa8c010460 con 0x7faa9c103af0 2024-04-21T09:02:05.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.548+0000 7faa92ffd640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faa880a3d20 0x7faa880a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:05.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.548+0000 7faaa1ce7640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faa880a3d20 0x7faa880a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:05.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.548+0000 7faa92ffd640 1 -- 172.21.15.28:0/92304843 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7faa8c0d0960 con 0x7faa9c103af0 2024-04-21T09:02:05.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.548+0000 7faaa1ce7640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faa880a3d20 0x7faa880a61e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faa98036040 tx=0x7faa98000bc0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:05.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.552+0000 7faa92ffd640 1 -- 172.21.15.28:0/92304843 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa8c09cbb0 con 0x7faa9c103af0 2024-04-21T09:02:05.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.738+0000 7faaa3f4b640 1 -- 172.21.15.28:0/92304843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7faa9c107100 con 0x7faa9c103af0 2024-04-21T09:02:05.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.738+0000 7faa92ffd640 1 -- 172.21.15.28:0/92304843 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v102) ==== 103+0+11 (secure 0 0 0) 0x7faa8c009de0 con 0x7faa9c103af0 2024-04-21T09:02:05.739 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-21T09:02:05.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.740+0000 7faaa3f4b640 1 -- 172.21.15.28:0/92304843 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faa880a3d20 msgr2=0x7faa880a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:05.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.740+0000 7faaa3f4b640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faa880a3d20 0x7faa880a61e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faa98036040 tx=0x7faa98000bc0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.740+0000 7faaa3f4b640 1 -- 172.21.15.28:0/92304843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa9c103af0 msgr2=0x7faa9c198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.740+0000 7faaa3f4b640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa9c103af0 0x7faa9c198c80 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00d950 tx=0x7faa8c00de10 comp rx=0 tx=0).stop 2024-04-21T09:02:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.740+0000 7faaa3f4b640 1 -- 172.21.15.28:0/92304843 shutdown_connections 2024-04-21T09:02:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.740+0000 7faaa3f4b640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7faa880a3d20 0x7faa880a61e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.740+0000 7faaa3f4b640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faa9c10c510 0x7faa9c19d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.740+0000 7faaa3f4b640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa9c103af0 0x7faa9c198c80 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.740+0000 7faaa3f4b640 1 --2- 172.21.15.28:0/92304843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faa9c1031d0 0x7faa9c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.740+0000 7faaa3f4b640 1 -- 172.21.15.28:0/92304843 >> 172.21.15.28:0/92304843 conn(0x7faa9c077f50 msgr2=0x7faa9c0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:05.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.740+0000 7faaa3f4b640 1 -- 172.21.15.28:0/92304843 shutdown_connections 2024-04-21T09:02:05.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.740+0000 7faaa3f4b640 1 -- 172.21.15.28:0/92304843 wait complete. 2024-04-21T09:02:05.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T09:02:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.854+0000 7f0e10bce640 1 -- 172.21.15.28:0/1723200440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c1031d0 msgr2=0x7f0e0c1035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.854+0000 7f0e10bce640 1 --2- 172.21.15.28:0/1723200440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c1031d0 0x7f0e0c1035b0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc009900 tx=0x7f0dfc02bd60 comp rx=0 tx=0).stop 2024-04-21T09:02:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.854+0000 7f0e10bce640 1 -- 172.21.15.28:0/1723200440 shutdown_connections 2024-04-21T09:02:05.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.854+0000 7f0e10bce640 1 --2- 172.21.15.28:0/1723200440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0e0c10c510 0x7f0e0c10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.854+0000 7f0e10bce640 1 --2- 172.21.15.28:0/1723200440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e0c103af0 0x7f0e0c10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.854+0000 7f0e10bce640 1 --2- 172.21.15.28:0/1723200440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c1031d0 0x7f0e0c1035b0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.854+0000 7f0e10bce640 1 -- 172.21.15.28:0/1723200440 >> 172.21.15.28:0/1723200440 conn(0x7f0e0c077f50 msgr2=0x7f0e0c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.854+0000 7f0e10bce640 1 -- 172.21.15.28:0/1723200440 shutdown_connections 2024-04-21T09:02:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.854+0000 7f0e10bce640 1 -- 172.21.15.28:0/1723200440 wait complete. 2024-04-21T09:02:05.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e10bce640 1 Processor -- start 2024-04-21T09:02:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e10bce640 1 -- start start 2024-04-21T09:02:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e10bce640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0e0c1031d0 0x7f0e0c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e10bce640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e0c103af0 0x7f0e0c198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e10bce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c10c510 0x7f0e0c19cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e10bce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c112180 con 0x7f0e0c10c510 2024-04-21T09:02:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e10bce640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c112000 con 0x7f0e0c1031d0 2024-04-21T09:02:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e10bce640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c112300 con 0x7f0e0c103af0 2024-04-21T09:02:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e0ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c10c510 0x7f0e0c19cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e0ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c10c510 0x7f0e0c19cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35808/0 (socket says 172.21.15.28:35808) 2024-04-21T09:02:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e0ad9d640 1 -- 172.21.15.28:0/3019377171 learned_addr learned my addr 172.21.15.28:0/3019377171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:02:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e0a59c640 1 --2- 172.21.15.28:0/3019377171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0e0c1031d0 0x7f0e0c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e0ad9d640 1 -- 172.21.15.28:0/3019377171 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e0c103af0 msgr2=0x7f0e0c198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e09d9b640 1 --2- 172.21.15.28:0/3019377171 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e0c103af0 0x7f0e0c198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e0ad9d640 1 --2- 172.21.15.28:0/3019377171 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e0c103af0 0x7f0e0c198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e0ad9d640 1 -- 172.21.15.28:0/3019377171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0e0c1031d0 msgr2=0x7f0e0c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e0ad9d640 1 --2- 172.21.15.28:0/3019377171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0e0c1031d0 0x7f0e0c198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e0ad9d640 1 -- 172.21.15.28:0/3019377171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e0c19d710 con 0x7f0e0c10c510 2024-04-21T09:02:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.855+0000 7f0e0a59c640 1 --2- 172.21.15.28:0/3019377171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0e0c1031d0 0x7f0e0c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:02:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.856+0000 7f0e0ad9d640 1 --2- 172.21.15.28:0/3019377171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c10c510 0x7f0e0c19cfe0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400eaa0 tx=0x7f0e0400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.856+0000 7f0dfb7fe640 1 -- 172.21.15.28:0/3019377171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0400cdb0 con 0x7f0e0c10c510 2024-04-21T09:02:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.856+0000 7f0e10bce640 1 -- 172.21.15.28:0/3019377171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e0c19da00 con 0x7f0e0c10c510 2024-04-21T09:02:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.856+0000 7f0dfb7fe640 1 -- 172.21.15.28:0/3019377171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e04005020 con 0x7f0e0c10c510 2024-04-21T09:02:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.856+0000 7f0dfb7fe640 1 -- 172.21.15.28:0/3019377171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e040116d0 con 0x7f0e0c10c510 2024-04-21T09:02:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.856+0000 7f0e10bce640 1 -- 172.21.15.28:0/3019377171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e0c1a5290 con 0x7f0e0c10c510 2024-04-21T09:02:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.857+0000 7f0e10bce640 1 -- 172.21.15.28:0/3019377171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e0c068d70 con 0x7f0e0c10c510 2024-04-21T09:02:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.858+0000 7f0dfb7fe640 1 -- 172.21.15.28:0/3019377171 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0e040051d0 con 0x7f0e0c10c510 2024-04-21T09:02:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.859+0000 7f0dfb7fe640 1 --2- 172.21.15.28:0/3019377171 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0df40a3d20 0x7f0df40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:05.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.859+0000 7f0e0a59c640 1 --2- 172.21.15.28:0/3019377171 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0df40a3d20 0x7f0df40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:05.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.859+0000 7f0dfb7fe640 1 -- 172.21.15.28:0/3019377171 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f0e040d11f0 con 0x7f0e0c10c510 2024-04-21T09:02:05.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.859+0000 7f0e0a59c640 1 --2- 172.21.15.28:0/3019377171 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0df40a3d20 0x7f0df40a61e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc005ab0 tx=0x7f0dfc005a20 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:05.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:05.862+0000 7f0dfb7fe640 1 -- 172.21.15.28:0/3019377171 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0e04016030 con 0x7f0e0c10c510 2024-04-21T09:02:06.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.061+0000 7f0e10bce640 1 -- 172.21.15.28:0/3019377171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0e0c107690 con 0x7f0e0c10c510 2024-04-21T09:02:06.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.061+0000 7f0dfb7fe640 1 -- 172.21.15.28:0/3019377171 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v102) ==== 81+0+937 (secure 0 0 0) 0x7f0e0409d4a0 con 0x7f0e0c10c510 2024-04-21T09:02:06.061 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1076282159 2024-04-22T08:51:10.823388+0000 2024-04-21T09:02:06.062 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/226897930 2024-04-22T08:51:10.823388+0000 2024-04-21T09:02:06.062 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/226897930 2024-04-22T08:51:10.823388+0000 2024-04-21T09:02:06.062 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/926389767 2024-04-22T08:51:10.823388+0000 2024-04-21T09:02:06.062 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3741297321 2024-04-22T08:41:35.374122+0000 2024-04-21T09:02:06.062 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/892059008 2024-04-22T08:51:10.823388+0000 2024-04-21T09:02:06.062 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/334321812 2024-04-22T08:41:35.374122+0000 2024-04-21T09:02:06.062 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2874636306 2024-04-22T08:41:07.583432+0000 2024-04-21T09:02:06.062 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/334321812 2024-04-22T08:41:35.374122+0000 2024-04-21T09:02:06.063 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1716427535 2024-04-22T08:41:07.583432+0000 2024-04-21T09:02:06.063 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/731326464 2024-04-22T08:41:35.374122+0000 2024-04-21T09:02:06.063 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2874636306 2024-04-22T08:41:07.583432+0000 2024-04-21T09:02:06.063 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2132574390 2024-04-22T08:51:10.823388+0000 2024-04-21T09:02:06.063 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/326837648 2024-04-22T08:41:35.374122+0000 2024-04-21T09:02:06.063 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1980013253 2024-04-22T08:41:07.583432+0000 2024-04-21T09:02:06.063 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/917824119 2024-04-22T08:41:07.583432+0000 2024-04-21T09:02:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:listed 16 entries 2024-04-21T09:02:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.063+0000 7f0e10bce640 1 -- 172.21.15.28:0/3019377171 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0df40a3d20 msgr2=0x7f0df40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.063+0000 7f0e10bce640 1 --2- 172.21.15.28:0/3019377171 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0df40a3d20 0x7f0df40a61e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc005ab0 tx=0x7f0dfc005a20 comp rx=0 tx=0).stop 2024-04-21T09:02:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.063+0000 7f0e10bce640 1 -- 172.21.15.28:0/3019377171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c10c510 msgr2=0x7f0e0c19cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.063+0000 7f0e10bce640 1 --2- 172.21.15.28:0/3019377171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c10c510 0x7f0e0c19cfe0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f0e0400eaa0 tx=0x7f0e0400ef60 comp rx=0 tx=0).stop 2024-04-21T09:02:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.063+0000 7f0e10bce640 1 -- 172.21.15.28:0/3019377171 shutdown_connections 2024-04-21T09:02:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.063+0000 7f0e10bce640 1 --2- 172.21.15.28:0/3019377171 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f0df40a3d20 0x7f0df40a61e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.063+0000 7f0e10bce640 1 --2- 172.21.15.28:0/3019377171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c10c510 0x7f0e0c19cfe0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.063+0000 7f0e10bce640 1 --2- 172.21.15.28:0/3019377171 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e0c103af0 0x7f0e0c198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.063+0000 7f0e10bce640 1 --2- 172.21.15.28:0/3019377171 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0e0c1031d0 0x7f0e0c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.063+0000 7f0e10bce640 1 -- 172.21.15.28:0/3019377171 >> 172.21.15.28:0/3019377171 conn(0x7f0e0c077f50 msgr2=0x7f0e0c0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:06.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.063+0000 7f0e10bce640 1 -- 172.21.15.28:0/3019377171 shutdown_connections 2024-04-21T09:02:06.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.063+0000 7f0e10bce640 1 -- 172.21.15.28:0/3019377171 wait complete. 2024-04-21T09:02:06.074 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-21T09:02:06.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T09:02:06.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-21T09:02:06.121 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-04-21T09:02:06.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T09:02:06.121 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-21T09:02:06.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T09:02:06.158 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-21T09:02:06.222 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-21T09:02:06.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T09:02:06.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-21T09:02:06.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T09:02:06.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-21T09:02:06.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:06 smithi028 ceph-mon[28003]: pgmap v338: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2024-04-21T09:02:06.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:06 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2767881174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T09:02:06.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:06 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1453196179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T09:02:06.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:06 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/92304843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T09:02:06.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:06 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3019377171' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T09:02:06.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:06 smithi171 ceph-mon[31358]: pgmap v338: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2024-04-21T09:02:06.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:06 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2767881174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T09:02:06.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:06 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1453196179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T09:02:06.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:06 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/92304843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T09:02:06.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:06 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3019377171' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T09:02:06.385 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-21T09:02:06.385 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-21T09:02:06.386 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T09:02:06.442 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056881,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498928k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=414 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/fc9e8145942c3a1e33a9faa82d0e7dd0a6da51db9675f943cc6a8ffc36a7a63f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/FIX5PUL43JPJQ5SRNZZE6VZLBY:/var/lib/containers/storage/overlay/l/F76UWVYXYH2YMNNOWFROIWMGGB,upperdir=/var/lib/containers/storage/overlay/fc9e8145942c3a1e33a9faa82d0e7dd0a6da51db9675f943cc6a8ffc36a7a63f/diff,workdir=/var/lib/containers/storage/overlay/fc9e8145942c3a1e33a9faa82d0e7dd0a6da51db9675f943cc6a8ffc36a7a63f/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/6bc0fa31b48937f182be085e4231b9165ee341f7e559319ce6aba9a63dd04555/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/FIX5PUL43JPJQ5SRNZZE6VZLBY:/var/lib/containers/storage/overlay/l/F76UWVYXYH2YMNNOWFROIWMGGB,upperdir=/var/lib/containers/storage/overlay/6bc0fa31b48937f182be085e4231b9165ee341f7e559319ce6aba9a63dd04555/diff,workdir=/var/lib/containers/storage/overlay/6bc0fa31b48937f182be085e4231b9165ee341f7e559319ce6aba9a63dd04555/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/0c93af090e13963d2a68bbfe939a63cbad717f022d8ea9a85538592ea9442e9a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/FIX5PUL43JPJQ5SRNZZE6VZLBY:/var/lib/containers/storage/overlay/l/F76UWVYXYH2YMNNOWFROIWMGGB,upperdir=/var/lib/containers/storage/overlay/0c93af090e13963d2a68bbfe939a63cbad717f022d8ea9a85538592ea9442e9a/diff,workdir=/var/lib/containers/storage/overlay/0c93af090e13963d2a68bbfe939a63cbad717f022d8ea9a85538592ea9442e9a/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/e4bf82040817311da57e60dbcf75b66ecfb7a1251c2f16e78fcdb2f4e30648a7/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/FIX5PUL43JPJQ5SRNZZE6VZLBY:/var/lib/containers/storage/overlay/l/F76UWVYXYH2YMNNOWFROIWMGGB,upperdir=/var/lib/containers/storage/overlay/e4bf82040817311da57e60dbcf75b66ecfb7a1251c2f16e78fcdb2f4e30648a7/diff,workdir=/var/lib/containers/storage/overlay/e4bf82040817311da57e60dbcf75b66ecfb7a1251c2f16e78fcdb2f4e30648a7/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4561a8a65d4b9a03925bbbb71250277e0eaac84f727b5632fe5fac6227d860d0/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/FIX5PUL43JPJQ5SRNZZE6VZLBY:/var/lib/containers/storage/overlay/l/F76UWVYXYH2YMNNOWFROIWMGGB,upperdir=/var/lib/containers/storage/overlay/4561a8a65d4b9a03925bbbb71250277e0eaac84f727b5632fe5fac6227d860d0/diff,workdir=/var/lib/containers/storage/overlay/4561a8a65d4b9a03925bbbb71250277e0eaac84f727b5632fe5fac6227d860d0/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d7f09f0a4e431fea850e8b347ffa41405c80e349e86e79c8492bc69aae9c60b2/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/FIX5PUL43JPJQ5SRNZZE6VZLBY:/var/lib/containers/storage/overlay/l/F76UWVYXYH2YMNNOWFROIWMGGB,upperdir=/var/lib/containers/storage/overlay/d7f09f0a4e431fea850e8b347ffa41405c80e349e86e79c8492bc69aae9c60b2/diff,workdir=/var/lib/containers/storage/overlay/d7f09f0a4e431fea850e8b347ffa41405c80e349e86e79c8492bc69aae9c60b2/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/70db4784ef8e1d061852822306212a89d716fe61bceb45f756e9758c4d43dbde/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/FIX5PUL43JPJQ5SRNZZE6VZLBY:/var/lib/containers/storage/overlay/l/F76UWVYXYH2YMNNOWFROIWMGGB,upperdir=/var/lib/containers/storage/overlay/70db4784ef8e1d061852822306212a89d716fe61bceb45f756e9758c4d43dbde/diff,workdir=/var/lib/containers/storage/overlay/70db4784ef8e1d061852822306212a89d716fe61bceb45f756e9758c4d43dbde/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4aa33676866ac72bc1d92525a8837d774b6697312475d7ac1650d7cacc1fd18c/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/FIX5PUL43JPJQ5SRNZZE6VZLBY:/var/lib/containers/storage/overlay/l/F76UWVYXYH2YMNNOWFROIWMGGB,upperdir=/var/lib/containers/storage/overlay/4aa33676866ac72bc1d92525a8837d774b6697312475d7ac1650d7cacc1fd18c/diff,workdir=/var/lib/containers/storage/overlay/4aa33676866ac72bc1d92525a8837d774b6697312475d7ac1650d7cacc1fd18c/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/42f91d25b33941790d0497c4ac0acdb73a170db622231a6de19d042e04883380/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/FIX5PUL43JPJQ5SRNZZE6VZLBY:/var/lib/containers/storage/overlay/l/F76UWVYXYH2YMNNOWFROIWMGGB,upperdir=/var/lib/containers/storage/overlay/42f91d25b33941790d0497c4ac0acdb73a170db622231a6de19d042e04883380/diff,workdir=/var/lib/containers/storage/overlay/42f91d25b33941790d0497c4ac0acdb73a170db622231a6de19d042e04883380/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/6255e4d2cb3ec34ef7022b80735146031f3a667e0091afe4912ae575b768eef7/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/FIX5PUL43JPJQ5SRNZZE6VZLBY:/var/lib/containers/storage/overlay/l/F76UWVYXYH2YMNNOWFROIWMGGB,upperdir=/var/lib/containers/storage/overlay/6255e4d2cb3ec34ef7022b80735146031f3a667e0091afe4912ae575b768eef7/diff,workdir=/var/lib/containers/storage/overlay/6255e4d2cb3ec34ef7022b80735146031f3a667e0091afe4912ae575b768eef7/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498928k,nr_inodes=819200,mode=755,inode64 0 0 2024-04-21T09:02:06.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T09:02:06.442 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-21T09:02:06.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T09:02:06.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-21T09:02:06.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:06 smithi152 ceph-mon[31401]: pgmap v338: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2024-04-21T09:02:06.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:06 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2767881174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T09:02:06.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:06 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1453196179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T09:02:06.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:06 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/92304843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T09:02:06.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:06 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3019377171' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T09:02:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-04-21T09:02:06.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:02:06.625 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-21T09:02:06.636 INFO:tasks.cephadm:Teardown begin 2024-04-21T09:02:06.636 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-21T09:02:06.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T09:02:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.740+0000 7fde6d4fd640 1 -- 172.21.15.28:0/541324814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde68106f50 msgr2=0x7fde68107330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.740+0000 7fde6d4fd640 1 --2- 172.21.15.28:0/541324814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde68106f50 0x7fde68107330 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fde58009900 tx=0x7fde5802bc30 comp rx=0 tx=0).stop 2024-04-21T09:02:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.740+0000 7fde6d4fd640 1 -- 172.21.15.28:0/541324814 shutdown_connections 2024-04-21T09:02:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.740+0000 7fde6d4fd640 1 --2- 172.21.15.28:0/541324814 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde68069aa0 0x7fde6810ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.740+0000 7fde6d4fd640 1 --2- 172.21.15.28:0/541324814 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde680690e0 0x7fde68069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.740+0000 7fde6d4fd640 1 --2- 172.21.15.28:0/541324814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde68106f50 0x7fde68107330 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.740+0000 7fde6d4fd640 1 -- 172.21.15.28:0/541324814 >> 172.21.15.28:0/541324814 conn(0x7fde68077e30 msgr2=0x7fde68078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.740+0000 7fde6d4fd640 1 -- 172.21.15.28:0/541324814 shutdown_connections 2024-04-21T09:02:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.740+0000 7fde6d4fd640 1 -- 172.21.15.28:0/541324814 wait complete. 2024-04-21T09:02:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.740+0000 7fde6d4fd640 1 Processor -- start 2024-04-21T09:02:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.740+0000 7fde6d4fd640 1 -- start start 2024-04-21T09:02:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde6d4fd640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde680690e0 0x7fde68198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde6d4fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde68069aa0 0x7fde68198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde6d4fd640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde68106f50 0x7fde6819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde6d4fd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde68112170 con 0x7fde68069aa0 2024-04-21T09:02:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde6d4fd640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fde68111ff0 con 0x7fde680690e0 2024-04-21T09:02:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde6d4fd640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fde681122f0 con 0x7fde68106f50 2024-04-21T09:02:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde68069aa0 0x7fde68198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde68069aa0 0x7fde68198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35822/0 (socket says 172.21.15.28:35822) 2024-04-21T09:02:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde667fc640 1 -- 172.21.15.28:0/1442609179 learned_addr learned my addr 172.21.15.28:0/1442609179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:02:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde677fe640 1 --2- 172.21.15.28:0/1442609179 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde68106f50 0x7fde6819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde66ffd640 1 --2- 172.21.15.28:0/1442609179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde680690e0 0x7fde68198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde667fc640 1 -- 172.21.15.28:0/1442609179 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde68106f50 msgr2=0x7fde6819cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:06.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde667fc640 1 --2- 172.21.15.28:0/1442609179 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde68106f50 0x7fde6819cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde667fc640 1 -- 172.21.15.28:0/1442609179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde680690e0 msgr2=0x7fde68198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde667fc640 1 --2- 172.21.15.28:0/1442609179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde680690e0 0x7fde68198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde667fc640 1 -- 172.21.15.28:0/1442609179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6819d5f0 con 0x7fde68069aa0 2024-04-21T09:02:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde667fc640 1 --2- 172.21.15.28:0/1442609179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde68069aa0 0x7fde68198b90 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fde5c00c920 tx=0x7fde5c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde4ffff640 1 -- 172.21.15.28:0/1442609179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c007c70 con 0x7fde68069aa0 2024-04-21T09:02:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde6d4fd640 1 -- 172.21.15.28:0/1442609179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde6819d8e0 con 0x7fde68069aa0 2024-04-21T09:02:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde4ffff640 1 -- 172.21.15.28:0/1442609179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde5c007e10 con 0x7fde68069aa0 2024-04-21T09:02:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.741+0000 7fde4ffff640 1 -- 172.21.15.28:0/1442609179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c018820 con 0x7fde68069aa0 2024-04-21T09:02:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.742+0000 7fde6d4fd640 1 -- 172.21.15.28:0/1442609179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde681a5170 con 0x7fde68069aa0 2024-04-21T09:02:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.743+0000 7fde6d4fd640 1 -- 172.21.15.28:0/1442609179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde68068de0 con 0x7fde68069aa0 2024-04-21T09:02:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.744+0000 7fde4ffff640 1 -- 172.21.15.28:0/1442609179 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fde5c00f450 con 0x7fde68069aa0 2024-04-21T09:02:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.745+0000 7fde4ffff640 1 --2- 172.21.15.28:0/1442609179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fde500a3d20 0x7fde500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.745+0000 7fde66ffd640 1 --2- 172.21.15.28:0/1442609179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fde500a3d20 0x7fde500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.745+0000 7fde4ffff640 1 -- 172.21.15.28:0/1442609179 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7fde5c0d0f10 con 0x7fde68069aa0 2024-04-21T09:02:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.745+0000 7fde66ffd640 1 --2- 172.21.15.28:0/1442609179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fde500a3d20 0x7fde500a61e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fde58006010 tx=0x7fde5802c6c0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.749+0000 7fde4ffff640 1 -- 172.21.15.28:0/1442609179 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde5c014030 con 0x7fde68069aa0 2024-04-21T09:02:06.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.977+0000 7fde6d4fd640 1 -- 172.21.15.28:0/1442609179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde68199950 con 0x7fde68069aa0 2024-04-21T09:02:06.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.978+0000 7fde4ffff640 1 -- 172.21.15.28:0/1442609179 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7fde5c09d250 con 0x7fde68069aa0 2024-04-21T09:02:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T09:02:06.978 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T09:02:06.978 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T09:02:06.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.980+0000 7fde6d4fd640 1 -- 172.21.15.28:0/1442609179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fde500a3d20 msgr2=0x7fde500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:06.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.980+0000 7fde6d4fd640 1 --2- 172.21.15.28:0/1442609179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fde500a3d20 0x7fde500a61e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fde58006010 tx=0x7fde5802c6c0 comp rx=0 tx=0).stop 2024-04-21T09:02:06.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.980+0000 7fde6d4fd640 1 -- 172.21.15.28:0/1442609179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde68069aa0 msgr2=0x7fde68198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:06.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.980+0000 7fde6d4fd640 1 --2- 172.21.15.28:0/1442609179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde68069aa0 0x7fde68198b90 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fde5c00c920 tx=0x7fde5c00cde0 comp rx=0 tx=0).stop 2024-04-21T09:02:06.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.980+0000 7fde6d4fd640 1 -- 172.21.15.28:0/1442609179 shutdown_connections 2024-04-21T09:02:06.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.980+0000 7fde6d4fd640 1 --2- 172.21.15.28:0/1442609179 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fde500a3d20 0x7fde500a61e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:06.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.980+0000 7fde6d4fd640 1 --2- 172.21.15.28:0/1442609179 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fde68106f50 0x7fde6819cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:06.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.980+0000 7fde6d4fd640 1 --2- 172.21.15.28:0/1442609179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde68069aa0 0x7fde68198b90 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:06.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.980+0000 7fde6d4fd640 1 --2- 172.21.15.28:0/1442609179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde680690e0 0x7fde68198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:06.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.980+0000 7fde6d4fd640 1 -- 172.21.15.28:0/1442609179 >> 172.21.15.28:0/1442609179 conn(0x7fde68077e30 msgr2=0x7fde680fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:06.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.980+0000 7fde6d4fd640 1 -- 172.21.15.28:0/1442609179 shutdown_connections 2024-04-21T09:02:06.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:06.980+0000 7fde6d4fd640 1 -- 172.21.15.28:0/1442609179 wait complete. 2024-04-21T09:02:06.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T09:02:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.093+0000 7f5e43011640 1 -- 172.21.15.28:0/767286640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e3c0690e0 msgr2=0x7f5e3c069560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.093+0000 7f5e43011640 1 --2- 172.21.15.28:0/767286640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e3c0690e0 0x7f5e3c069560 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f5e30009930 tx=0x7f5e3002bc30 comp rx=0 tx=0).stop 2024-04-21T09:02:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.093+0000 7f5e43011640 1 -- 172.21.15.28:0/767286640 shutdown_connections 2024-04-21T09:02:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.093+0000 7f5e43011640 1 --2- 172.21.15.28:0/767286640 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e3c069aa0 0x7f5e3c10ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.093+0000 7f5e43011640 1 --2- 172.21.15.28:0/767286640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e3c0690e0 0x7f5e3c069560 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.093+0000 7f5e43011640 1 --2- 172.21.15.28:0/767286640 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5e3c107fd0 0x7f5e3c1083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.093+0000 7f5e43011640 1 -- 172.21.15.28:0/767286640 >> 172.21.15.28:0/767286640 conn(0x7f5e3c077e30 msgr2=0x7f5e3c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.093+0000 7f5e43011640 1 -- 172.21.15.28:0/767286640 shutdown_connections 2024-04-21T09:02:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.093+0000 7f5e43011640 1 -- 172.21.15.28:0/767286640 wait complete. 2024-04-21T09:02:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.094+0000 7f5e43011640 1 Processor -- start 2024-04-21T09:02:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.094+0000 7f5e43011640 1 -- start start 2024-04-21T09:02:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.094+0000 7f5e43011640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e3c0690e0 0x7f5e3c198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.094+0000 7f5e43011640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e3c069aa0 0x7f5e3c198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.094+0000 7f5e43011640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5e3c107fd0 0x7f5e3c19cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.094+0000 7f5e43011640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3c1121a0 con 0x7f5e3c069aa0 2024-04-21T09:02:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.094+0000 7f5e43011640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3c112020 con 0x7f5e3c0690e0 2024-04-21T09:02:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.094+0000 7f5e43011640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3c112320 con 0x7f5e3c107fd0 2024-04-21T09:02:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.094+0000 7f5e3bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e3c069aa0 0x7f5e3c198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.094+0000 7f5e3bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e3c069aa0 0x7f5e3c198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35834/0 (socket says 172.21.15.28:35834) 2024-04-21T09:02:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.094+0000 7f5e3bfff640 1 -- 172.21.15.28:0/118364713 learned_addr learned my addr 172.21.15.28:0/118364713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:02:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.094+0000 7f5e40dad640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e3c0690e0 0x7f5e3c198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.095+0000 7f5e415ae640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5e3c107fd0 0x7f5e3c19cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.095+0000 7f5e3bfff640 1 -- 172.21.15.28:0/118364713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5e3c107fd0 msgr2=0x7f5e3c19cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.095+0000 7f5e3bfff640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5e3c107fd0 0x7f5e3c19cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.095+0000 7f5e3bfff640 1 -- 172.21.15.28:0/118364713 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e3c0690e0 msgr2=0x7f5e3c198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.095+0000 7f5e3bfff640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e3c0690e0 0x7f5e3c198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.095+0000 7f5e3bfff640 1 -- 172.21.15.28:0/118364713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e3c19d6f0 con 0x7f5e3c069aa0 2024-04-21T09:02:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.095+0000 7f5e40dad640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e3c0690e0 0x7f5e3c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:02:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.095+0000 7f5e415ae640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5e3c107fd0 0x7f5e3c19cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:02:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.095+0000 7f5e3bfff640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e3c069aa0 0x7f5e3c198c40 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f5e30009230 tx=0x7f5e300045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.095+0000 7f5e39ffb640 1 -- 172.21.15.28:0/118364713 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30043020 con 0x7f5e3c069aa0 2024-04-21T09:02:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.095+0000 7f5e43011640 1 -- 172.21.15.28:0/118364713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e3c19d980 con 0x7f5e3c069aa0 2024-04-21T09:02:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.095+0000 7f5e39ffb640 1 -- 172.21.15.28:0/118364713 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e3002cb70 con 0x7f5e3c069aa0 2024-04-21T09:02:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.095+0000 7f5e39ffb640 1 -- 172.21.15.28:0/118364713 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30003b50 con 0x7f5e3c069aa0 2024-04-21T09:02:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.095+0000 7f5e43011640 1 -- 172.21.15.28:0/118364713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e3c1a5270 con 0x7f5e3c069aa0 2024-04-21T09:02:07.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.096+0000 7f5e43011640 1 -- 172.21.15.28:0/118364713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e3c068de0 con 0x7f5e3c069aa0 2024-04-21T09:02:07.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.097+0000 7f5e39ffb640 1 -- 172.21.15.28:0/118364713 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5e30046020 con 0x7f5e3c069aa0 2024-04-21T09:02:07.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.098+0000 7f5e39ffb640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5e240a3d20 0x7f5e240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.098+0000 7f5e39ffb640 1 -- 172.21.15.28:0/118364713 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9533+0+0 (secure 0 0 0) 0x7f5e300f3170 con 0x7f5e3c069aa0 2024-04-21T09:02:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.098+0000 7f5e40dad640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5e240a3d20 0x7f5e240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.098+0000 7f5e40dad640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5e240a3d20 0x7f5e240a61e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5e34005fd0 tx=0x7f5e34009510 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.102+0000 7f5e39ffb640 1 -- 172.21.15.28:0/118364713 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5e3004d020 con 0x7f5e3c069aa0 2024-04-21T09:02:07.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:07 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1442609179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T09:02:07.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:07 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1442609179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T09:02:07.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.542+0000 7f5e43011640 1 -- 172.21.15.28:0/118364713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e3c10a460 con 0x7f5e3c069aa0 2024-04-21T09:02:07.544 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T09:02:07.545 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468163100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3468163100},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468163100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/3823720712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":3823720712},{"type":"v1","addr":"172.21.15.28:6837","nonce":3823720712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14622,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3409769959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3409769959},{"type":"v1","addr":"172.21.15.28:6839","nonce":3409769959}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14634,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/1709589702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":1709589702},{"type":"v1","addr":"172.21.15.28:6841","nonce":1709589702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/1529635773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":1529635773},{"type":"v1","addr":"172.21.15.152:6833","nonce":1529635773}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.171:6833/3170910324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6832","nonce":3170910324},{"type":"v1","addr":"172.21.15.171:6833","nonce":3170910324}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:48:45.911185+0000","modified":"2024-04-21T08:54:46.571271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24488,"mds_1":24487,"mds_2":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6835/3982885739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":3982885739},{"type":"v1","addr":"172.21.15.152:6835","nonce":3982885739}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/1261226890","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":1261226890},{"type":"v1","addr":"172.21.15.152:6837","nonce":1261226890}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.171:6835/663606808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6834","nonce":663606808},{"type":"v1","addr":"172.21.15.171:6835","nonce":663606808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.171:6837/895402983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6836","nonce":895402983},{"type":"v1","addr":"172.21.15.171:6837","nonce":895402983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.171:6839/3423264764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6838","nonce":3423264764},{"type":"v1","addr":"172.21.15.171:6839","nonce":3423264764}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.152:6839/3821365920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":3821365920},{"type":"v1","addr":"172.21.15.152:6839","nonce":3821365920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24488,"qdb_cluster":[24488,24463,24487]},"id":1}]} 2024-04-21T09:02:07.545 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-21T09:02:07.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.544+0000 7f5e39ffb640 1 -- 172.21.15.28:0/118364713 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7f5e300bf530 con 0x7f5e3c069aa0 2024-04-21T09:02:07.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.548+0000 7f5e43011640 1 -- 172.21.15.28:0/118364713 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5e240a3d20 msgr2=0x7f5e240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.548+0000 7f5e43011640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5e240a3d20 0x7f5e240a61e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5e34005fd0 tx=0x7f5e34009510 comp rx=0 tx=0).stop 2024-04-21T09:02:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.548+0000 7f5e43011640 1 -- 172.21.15.28:0/118364713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e3c069aa0 msgr2=0x7f5e3c198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.548+0000 7f5e43011640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e3c069aa0 0x7f5e3c198c40 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f5e30009230 tx=0x7f5e300045e0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.549+0000 7f5e43011640 1 -- 172.21.15.28:0/118364713 shutdown_connections 2024-04-21T09:02:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.549+0000 7f5e43011640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f5e240a3d20 0x7f5e240a61e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.549+0000 7f5e43011640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5e3c107fd0 0x7f5e3c19cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.549+0000 7f5e43011640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e3c069aa0 0x7f5e3c198c40 secure :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f5e30009230 tx=0x7f5e300045e0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.549+0000 7f5e43011640 1 --2- 172.21.15.28:0/118364713 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e3c0690e0 0x7f5e3c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.549+0000 7f5e43011640 1 -- 172.21.15.28:0/118364713 >> 172.21.15.28:0/118364713 conn(0x7f5e3c077e30 msgr2=0x7f5e3c100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.550+0000 7f5e43011640 1 -- 172.21.15.28:0/118364713 shutdown_connections 2024-04-21T09:02:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.550+0000 7f5e43011640 1 -- 172.21.15.28:0/118364713 wait complete. 2024-04-21T09:02:07.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T09:02:07.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:07 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1442609179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T09:02:07.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.732+0000 7fecee074640 1 -- 172.21.15.28:0/367281185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece80690e0 msgr2=0x7fece8069560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:07.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.732+0000 7fecee074640 1 --2- 172.21.15.28:0/367281185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece80690e0 0x7fece8069560 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009900 tx=0x7fecdc02bc30 comp rx=0 tx=0).stop 2024-04-21T09:02:07.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.732+0000 7fecee074640 1 -- 172.21.15.28:0/367281185 shutdown_connections 2024-04-21T09:02:07.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.732+0000 7fecee074640 1 --2- 172.21.15.28:0/367281185 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fece8069aa0 0x7fece810ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.732+0000 7fecee074640 1 --2- 172.21.15.28:0/367281185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece80690e0 0x7fece8069560 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.732+0000 7fecee074640 1 --2- 172.21.15.28:0/367281185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fece8106f50 0x7fece8107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.732+0000 7fecee074640 1 -- 172.21.15.28:0/367281185 >> 172.21.15.28:0/367281185 conn(0x7fece8077e30 msgr2=0x7fece8078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:07.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.732+0000 7fecee074640 1 -- 172.21.15.28:0/367281185 shutdown_connections 2024-04-21T09:02:07.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.732+0000 7fecee074640 1 -- 172.21.15.28:0/367281185 wait complete. 2024-04-21T09:02:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.733+0000 7fecee074640 1 Processor -- start 2024-04-21T09:02:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.733+0000 7fecee074640 1 -- start start 2024-04-21T09:02:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.733+0000 7fecee074640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fece80690e0 0x7fece81986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.733+0000 7fecee074640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece8069aa0 0x7fece8198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.733+0000 7fecee074640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fece8106f50 0x7fece819cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.733+0000 7fecee074640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fece81121c0 con 0x7fece8069aa0 2024-04-21T09:02:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.733+0000 7fecee074640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fece8112040 con 0x7fece8106f50 2024-04-21T09:02:07.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.733+0000 7fecee074640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fece8112340 con 0x7fece80690e0 2024-04-21T09:02:07.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.733+0000 7fece6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece8069aa0 0x7fece8198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:07.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.733+0000 7fece6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece8069aa0 0x7fece8198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35854/0 (socket says 172.21.15.28:35854) 2024-04-21T09:02:07.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.733+0000 7fece6ffd640 1 -- 172.21.15.28:0/3424091110 learned_addr learned my addr 172.21.15.28:0/3424091110 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:02:07.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.734+0000 7fece77fe640 1 --2- 172.21.15.28:0/3424091110 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fece80690e0 0x7fece81986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:07.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.734+0000 7fece7fff640 1 --2- 172.21.15.28:0/3424091110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fece8106f50 0x7fece819cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:07.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.734+0000 7fece6ffd640 1 -- 172.21.15.28:0/3424091110 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fece80690e0 msgr2=0x7fece81986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:07.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.734+0000 7fece6ffd640 1 --2- 172.21.15.28:0/3424091110 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fece80690e0 0x7fece81986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.734+0000 7fece6ffd640 1 -- 172.21.15.28:0/3424091110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fece8106f50 msgr2=0x7fece819cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:07.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.734+0000 7fece6ffd640 1 --2- 172.21.15.28:0/3424091110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fece8106f50 0x7fece819cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.734+0000 7fece6ffd640 1 -- 172.21.15.28:0/3424091110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece819d690 con 0x7fece8069aa0 2024-04-21T09:02:07.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.734+0000 7fece6ffd640 1 --2- 172.21.15.28:0/3424091110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece8069aa0 0x7fece8198be0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fecdc000c00 tx=0x7fecdc02c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:07.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.734+0000 7fece4ff9640 1 -- 172.21.15.28:0/3424091110 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc043020 con 0x7fece8069aa0 2024-04-21T09:02:07.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.734+0000 7fece4ff9640 1 -- 172.21.15.28:0/3424091110 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecdc0052b0 con 0x7fece8069aa0 2024-04-21T09:02:07.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.734+0000 7fecee074640 1 -- 172.21.15.28:0/3424091110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fece819d920 con 0x7fece8069aa0 2024-04-21T09:02:07.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.734+0000 7fece4ff9640 1 -- 172.21.15.28:0/3424091110 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc003c40 con 0x7fece8069aa0 2024-04-21T09:02:07.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.734+0000 7fecee074640 1 -- 172.21.15.28:0/3424091110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fece81a5210 con 0x7fece8069aa0 2024-04-21T09:02:07.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.735+0000 7fecee074640 1 -- 172.21.15.28:0/3424091110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fece8068de0 con 0x7fece8069aa0 2024-04-21T09:02:07.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.736+0000 7fece4ff9640 1 -- 172.21.15.28:0/3424091110 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fecdc046020 con 0x7fece8069aa0 2024-04-21T09:02:07.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.737+0000 7fece4ff9640 1 --2- 172.21.15.28:0/3424091110 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fecd00a3d20 0x7fecd00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:07.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.737+0000 7fece4ff9640 1 -- 172.21.15.28:0/3424091110 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9621+0+0 (secure 0 0 0) 0x7fecdc0f3270 con 0x7fece8069aa0 2024-04-21T09:02:07.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.737+0000 7fece77fe640 1 --2- 172.21.15.28:0/3424091110 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fecd00a3d20 0x7fecd00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:07.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.737+0000 7fece77fe640 1 --2- 172.21.15.28:0/3424091110 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fecd00a3d20 0x7fecd00a61e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fecd800bbe0 tx=0x7fecd800b390 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:07.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.741+0000 7fece4ff9640 1 -- 172.21.15.28:0/3424091110 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fecdc04d020 con 0x7fece8069aa0 2024-04-21T09:02:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.929+0000 7fecee074640 1 -- 172.21.15.28:0/3424091110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fece810a500 con 0x7fece8069aa0 2024-04-21T09:02:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.931+0000 7fece4ff9640 1 -- 172.21.15.28:0/3424091110 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+25481 (secure 0 0 0) 0x7fecdc0bf5d0 con 0x7fece8069aa0 2024-04-21T09:02:07.931 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-21T09:02:07.931 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":103,"fsid":"b51a76c8-ffba-11ee-bc93-c7b262605968","created":"2024-04-21T08:40:33.405421+0000","modified":"2024-04-21T09:02:07.107035+0000","last_up_change":"2024-04-21T08:47:57.561060+0000","last_in_change":"2024-04-21T08:47:37.074944+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:44:05.469235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:48:43.543799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:48:44.864564+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:48:48.747842+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b82ad926-994c-4e96-90d8-585391ba5e7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6803","nonce":855251648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6805","nonce":855251648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6809","nonce":855251648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":855251648},{"type":"v1","addr":"172.21.15.28:6807","nonce":855251648}]},"public_addr":"172.21.15.28:6803/855251648","cluster_addr":"172.21.15.28:6805/855251648","heartbeat_back_addr":"172.21.15.28:6809/855251648","heartbeat_front_addr":"172.21.15.28:6807/855251648","state":["exists","up"]},{"osd":1,"uuid":"a75fcaea-75f3-44fe-8b41-5ace42be38d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6801","nonce":728547438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6803","nonce":728547438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6807","nonce":728547438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":728547438},{"type":"v1","addr":"172.21.15.152:6805","nonce":728547438}]},"public_addr":"172.21.15.152:6801/728547438","cluster_addr":"172.21.15.152:6803/728547438","heartbeat_back_addr":"172.21.15.152:6807/728547438","heartbeat_front_addr":"172.21.15.152:6805/728547438","state":["exists","up"]},{"osd":2,"uuid":"e72b05a3-79ac-4a80-a9dd-6192b5e36ae0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6801","nonce":4097967582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6803","nonce":4097967582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6807","nonce":4097967582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":4097967582},{"type":"v1","addr":"172.21.15.171:6805","nonce":4097967582}]},"public_addr":"172.21.15.171:6801/4097967582","cluster_addr":"172.21.15.171:6803/4097967582","heartbeat_back_addr":"172.21.15.171:6807/4097967582","heartbeat_front_addr":"172.21.15.171:6805/4097967582","state":["exists","up"]},{"osd":3,"uuid":"8265dd0b-19d9-45f2-887d-a0467bbca91a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6811","nonce":1806506959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6813","nonce":1806506959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6817","nonce":1806506959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1806506959},{"type":"v1","addr":"172.21.15.28:6815","nonce":1806506959}]},"public_addr":"172.21.15.28:6811/1806506959","cluster_addr":"172.21.15.28:6813/1806506959","heartbeat_back_addr":"172.21.15.28:6817/1806506959","heartbeat_front_addr":"172.21.15.28:6815/1806506959","state":["exists","up"]},{"osd":4,"uuid":"7316601a-d423-480c-9039-27c15d5f4de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6809","nonce":3935841376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6811","nonce":3935841376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6815","nonce":3935841376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":3935841376},{"type":"v1","addr":"172.21.15.152:6813","nonce":3935841376}]},"public_addr":"172.21.15.152:6809/3935841376","cluster_addr":"172.21.15.152:6811/3935841376","heartbeat_back_addr":"172.21.15.152:6815/3935841376","heartbeat_front_addr":"172.21.15.152:6813/3935841376","state":["exists","up"]},{"osd":5,"uuid":"ee83cf31-51ec-4b8f-a5f9-8457b98c44c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6809","nonce":3821828239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6811","nonce":3821828239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6815","nonce":3821828239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":3821828239},{"type":"v1","addr":"172.21.15.171:6813","nonce":3821828239}]},"public_addr":"172.21.15.171:6809/3821828239","cluster_addr":"172.21.15.171:6811/3821828239","heartbeat_back_addr":"172.21.15.171:6815/3821828239","heartbeat_front_addr":"172.21.15.171:6813/3821828239","state":["exists","up"]},{"osd":6,"uuid":"ab718a5d-64ee-43c9-b8e8-212ce636329c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6819","nonce":38541122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6821","nonce":38541122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6825","nonce":38541122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":38541122},{"type":"v1","addr":"172.21.15.28:6823","nonce":38541122}]},"public_addr":"172.21.15.28:6819/38541122","cluster_addr":"172.21.15.28:6821/38541122","heartbeat_back_addr":"172.21.15.28:6825/38541122","heartbeat_front_addr":"172.21.15.28:6823/38541122","state":["exists","up"]},{"osd":7,"uuid":"4175e074-9115-40ef-8582-26478d134260","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6817","nonce":3749693516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6819","nonce":3749693516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6823","nonce":3749693516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":3749693516},{"type":"v1","addr":"172.21.15.152:6821","nonce":3749693516}]},"public_addr":"172.21.15.152:6817/3749693516","cluster_addr":"172.21.15.152:6819/3749693516","heartbeat_back_addr":"172.21.15.152:6823/3749693516","heartbeat_front_addr":"172.21.15.152:6821/3749693516","state":["exists","up"]},{"osd":8,"uuid":"32c9b173-a70e-4ebf-8a9f-81a0ecf6f727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6816","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6817","nonce":1592405587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6818","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6819","nonce":1592405587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6822","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6823","nonce":1592405587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6820","nonce":1592405587},{"type":"v1","addr":"172.21.15.171:6821","nonce":1592405587}]},"public_addr":"172.21.15.171:6817/1592405587","cluster_addr":"172.21.15.171:6819/1592405587","heartbeat_back_addr":"172.21.15.171:6823/1592405587","heartbeat_front_addr":"172.21.15.171:6821/1592405587","state":["exists","up"]},{"osd":9,"uuid":"181899a9-f577-4b15-b710-4681b4169ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6827","nonce":2255450197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6829","nonce":2255450197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6833","nonce":2255450197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2255450197},{"type":"v1","addr":"172.21.15.28:6831","nonce":2255450197}]},"public_addr":"172.21.15.28:6827/2255450197","cluster_addr":"172.21.15.28:6829/2255450197","heartbeat_back_addr":"172.21.15.28:6833/2255450197","heartbeat_front_addr":"172.21.15.28:6831/2255450197","state":["exists","up"]},{"osd":10,"uuid":"03c59ec8-844b-4120-87bd-d4421e5027c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6825","nonce":534177203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6827","nonce":534177203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6831","nonce":534177203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":534177203},{"type":"v1","addr":"172.21.15.152:6829","nonce":534177203}]},"public_addr":"172.21.15.152:6825/534177203","cluster_addr":"172.21.15.152:6827/534177203","heartbeat_back_addr":"172.21.15.152:6831/534177203","heartbeat_front_addr":"172.21.15.152:6829/534177203","state":["exists","up"]},{"osd":11,"uuid":"0d57b2d5-a408-49a2-baf3-a0ebce5097f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6824","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6825","nonce":2148030417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6826","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6827","nonce":2148030417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6830","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6831","nonce":2148030417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6828","nonce":2148030417},{"type":"v1","addr":"172.21.15.171:6829","nonce":2148030417}]},"public_addr":"172.21.15.171:6825/2148030417","cluster_addr":"172.21.15.171:6827/2148030417","heartbeat_back_addr":"172.21.15.171:6831/2148030417","heartbeat_front_addr":"172.21.15.171:6829/2148030417","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:13.418509+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:43:37.685541+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:01.706430+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:27.327930+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:44:51.939845+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:16.883782+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:45:42.572901+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:08.691095+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:46:34.722628+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:02.693176+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:28.969464+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:47:55.617327+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":1}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":7}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":10}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1076282159":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6800/226897930":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/926389767":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/3741297321":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/892059008":"2024-04-22T08:51:10.823388+0000","172.21.15.28:6801/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6800/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:6800/334321812":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1716427535":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/731326464":"2024-04-22T08:41:35.374122+0000","172.21.15.28:6801/2874636306":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/2132574390":"2024-04-22T08:51:10.823388+0000","172.21.15.28:0/326837648":"2024-04-22T08:41:35.374122+0000","172.21.15.28:0/1980013253":"2024-04-22T08:41:07.583432+0000","172.21.15.28:0/917824119":"2024-04-22T08:41:07.583432+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T09:02:07.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.933+0000 7fecee074640 1 -- 172.21.15.28:0/3424091110 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fecd00a3d20 msgr2=0x7fecd00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:07.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.933+0000 7fecee074640 1 --2- 172.21.15.28:0/3424091110 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fecd00a3d20 0x7fecd00a61e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fecd800bbe0 tx=0x7fecd800b390 comp rx=0 tx=0).stop 2024-04-21T09:02:07.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.933+0000 7fecee074640 1 -- 172.21.15.28:0/3424091110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece8069aa0 msgr2=0x7fece8198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:07.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.933+0000 7fecee074640 1 --2- 172.21.15.28:0/3424091110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece8069aa0 0x7fece8198be0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fecdc000c00 tx=0x7fecdc02c800 comp rx=0 tx=0).stop 2024-04-21T09:02:07.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.933+0000 7fecee074640 1 -- 172.21.15.28:0/3424091110 shutdown_connections 2024-04-21T09:02:07.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.933+0000 7fecee074640 1 --2- 172.21.15.28:0/3424091110 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fecd00a3d20 0x7fecd00a61e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.933+0000 7fecee074640 1 --2- 172.21.15.28:0/3424091110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fece8106f50 0x7fece819cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.933+0000 7fecee074640 1 --2- 172.21.15.28:0/3424091110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece8069aa0 0x7fece8198be0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.933+0000 7fecee074640 1 --2- 172.21.15.28:0/3424091110 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fece80690e0 0x7fece81986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:07.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.933+0000 7fecee074640 1 -- 172.21.15.28:0/3424091110 >> 172.21.15.28:0/3424091110 conn(0x7fece8077e30 msgr2=0x7fece80feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:07.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.933+0000 7fecee074640 1 -- 172.21.15.28:0/3424091110 shutdown_connections 2024-04-21T09:02:07.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:07.933+0000 7fecee074640 1 -- 172.21.15.28:0/3424091110 wait complete. 2024-04-21T09:02:07.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-21T09:02:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.047+0000 7f7369ff4640 1 -- 172.21.15.28:0/1109780751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7364101a40 msgr2=0x7f7364105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.047+0000 7f7369ff4640 1 --2- 172.21.15.28:0/1109780751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7364101a40 0x7f7364105a80 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f7358009900 tx=0x7f735802bc30 comp rx=0 tx=0).stop 2024-04-21T09:02:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.047+0000 7f7369ff4640 1 -- 172.21.15.28:0/1109780751 shutdown_connections 2024-04-21T09:02:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.047+0000 7f7369ff4640 1 --2- 172.21.15.28:0/1109780751 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7364106160 0x7f736410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.047+0000 7f7369ff4640 1 --2- 172.21.15.28:0/1109780751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7364101a40 0x7f7364105a80 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.047+0000 7f7369ff4640 1 --2- 172.21.15.28:0/1109780751 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7364101090 0x7f7364101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.047+0000 7f7369ff4640 1 -- 172.21.15.28:0/1109780751 >> 172.21.15.28:0/1109780751 conn(0x7f7364077f50 msgr2=0x7f7364078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.047+0000 7f7369ff4640 1 -- 172.21.15.28:0/1109780751 shutdown_connections 2024-04-21T09:02:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.047+0000 7f7369ff4640 1 -- 172.21.15.28:0/1109780751 wait complete. 2024-04-21T09:02:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.047+0000 7f7369ff4640 1 Processor -- start 2024-04-21T09:02:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.047+0000 7f7369ff4640 1 -- start start 2024-04-21T09:02:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7369ff4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7364101090 0x7f7364197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7369ff4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7364106160 0x7f7364197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7369ff4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f736419bf20 0x7f736419c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7369ff4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f73641110a0 con 0x7f7364106160 2024-04-21T09:02:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7369ff4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f7364110f20 con 0x7f7364101090 2024-04-21T09:02:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7369ff4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f7364111220 con 0x7f736419bf20 2024-04-21T09:02:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7362ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7364106160 0x7f7364197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7362ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7364106160 0x7f7364197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35872/0 (socket says 172.21.15.28:35872) 2024-04-21T09:02:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7362ffd640 1 -- 172.21.15.28:0/2533249596 learned_addr learned my addr 172.21.15.28:0/2533249596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:02:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7363fff640 1 --2- 172.21.15.28:0/2533249596 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f736419bf20 0x7f736419c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f73637fe640 1 --2- 172.21.15.28:0/2533249596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7364101090 0x7f7364197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7362ffd640 1 -- 172.21.15.28:0/2533249596 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f736419bf20 msgr2=0x7f736419c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7362ffd640 1 --2- 172.21.15.28:0/2533249596 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f736419bf20 0x7f736419c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7362ffd640 1 -- 172.21.15.28:0/2533249596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7364101090 msgr2=0x7f7364197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7362ffd640 1 --2- 172.21.15.28:0/2533249596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7364101090 0x7f7364197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7362ffd640 1 -- 172.21.15.28:0/2533249596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736419ca10 con 0x7f7364106160 2024-04-21T09:02:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f73637fe640 1 --2- 172.21.15.28:0/2533249596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7364101090 0x7f7364197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:02:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7362ffd640 1 --2- 172.21.15.28:0/2533249596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7364106160 0x7f7364197ba0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f7358036040 tx=0x7f73580045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7360ff9640 1 -- 172.21.15.28:0/2533249596 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7358043020 con 0x7f7364106160 2024-04-21T09:02:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7369ff4640 1 -- 172.21.15.28:0/2533249596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f736419cca0 con 0x7f7364106160 2024-04-21T09:02:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7360ff9640 1 -- 172.21.15.28:0/2533249596 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f735802cb70 con 0x7f7364106160 2024-04-21T09:02:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7360ff9640 1 -- 172.21.15.28:0/2533249596 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7358003bf0 con 0x7f7364106160 2024-04-21T09:02:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.048+0000 7f7369ff4640 1 -- 172.21.15.28:0/2533249596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73641a44f0 con 0x7f7364106160 2024-04-21T09:02:08.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.050+0000 7f7369ff4640 1 -- 172.21.15.28:0/2533249596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7364068d70 con 0x7f7364106160 2024-04-21T09:02:08.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.050+0000 7f7360ff9640 1 -- 172.21.15.28:0/2533249596 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7358046020 con 0x7f7364106160 2024-04-21T09:02:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.051+0000 7f7360ff9640 1 --2- 172.21.15.28:0/2533249596 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f734c0a3d20 0x7f734c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.051+0000 7f7360ff9640 1 -- 172.21.15.28:0/2533249596 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9621+0+0 (secure 0 0 0) 0x7f73580f2dc0 con 0x7f7364106160 2024-04-21T09:02:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.051+0000 7f73637fe640 1 --2- 172.21.15.28:0/2533249596 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f734c0a3d20 0x7f734c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.051+0000 7f73637fe640 1 --2- 172.21.15.28:0/2533249596 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f734c0a3d20 0x7f734c0a61e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7354005d00 tx=0x7f7354009550 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:08.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.055+0000 7f7360ff9640 1 -- 172.21.15.28:0/2533249596 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f735804d020 con 0x7f7364106160 2024-04-21T09:02:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.281+0000 7f7369ff4640 1 -- 172.21.15.28:0/2533249596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7364101a40 con 0x7f7364106160 2024-04-21T09:02:08.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:08 smithi028 ceph-mon[28003]: pgmap v339: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T09:02:08.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:08 smithi028 ceph-mon[28003]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T09:02:08.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:08 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/118364713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T09:02:08.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:08 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3424091110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T09:02:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.283+0000 7f7360ff9640 1 -- 172.21.15.28:0/2533249596 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) ==== 161+0+0 (secure 0 0 0) 0x7f735803c350 con 0x7f7364106160 2024-04-21T09:02:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-21T09:02:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.286+0000 7f7369ff4640 1 -- 172.21.15.28:0/2533249596 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f734c0a3d20 msgr2=0x7f734c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.286+0000 7f7369ff4640 1 --2- 172.21.15.28:0/2533249596 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f734c0a3d20 0x7f734c0a61e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7354005d00 tx=0x7f7354009550 comp rx=0 tx=0).stop 2024-04-21T09:02:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.286+0000 7f7369ff4640 1 -- 172.21.15.28:0/2533249596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7364106160 msgr2=0x7f7364197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.286+0000 7f7369ff4640 1 --2- 172.21.15.28:0/2533249596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7364106160 0x7f7364197ba0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f7358036040 tx=0x7f73580045e0 comp rx=0 tx=0).stop 2024-04-21T09:02:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.286+0000 7f7369ff4640 1 -- 172.21.15.28:0/2533249596 shutdown_connections 2024-04-21T09:02:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.286+0000 7f7369ff4640 1 --2- 172.21.15.28:0/2533249596 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f734c0a3d20 0x7f734c0a61e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.286+0000 7f7369ff4640 1 --2- 172.21.15.28:0/2533249596 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f736419bf20 0x7f736419c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.286+0000 7f7369ff4640 1 --2- 172.21.15.28:0/2533249596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7364106160 0x7f7364197ba0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.286+0000 7f7369ff4640 1 --2- 172.21.15.28:0/2533249596 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7364101090 0x7f7364197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.286+0000 7f7369ff4640 1 -- 172.21.15.28:0/2533249596 >> 172.21.15.28:0/2533249596 conn(0x7f7364077f50 msgr2=0x7f73640ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.286+0000 7f7369ff4640 1 -- 172.21.15.28:0/2533249596 shutdown_connections 2024-04-21T09:02:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.286+0000 7f7369ff4640 1 -- 172.21.15.28:0/2533249596 wait complete. 2024-04-21T09:02:08.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-21T09:02:08.367 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-mon[31358]: pgmap v339: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T09:02:08.367 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-mon[31358]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T09:02:08.367 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/118364713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T09:02:08.367 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3424091110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T09:02:08.372 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-mon[31401]: pgmap v339: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-21T09:02:08.372 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-mon[31401]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T09:02:08.372 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/118364713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T09:02:08.372 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3424091110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T09:02:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.404+0000 7fd7acda4640 1 -- 172.21.15.28:0/3452524093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8105dd0 msgr2=0x7fd7a81061b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.404+0000 7fd7acda4640 1 --2- 172.21.15.28:0/3452524093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8105dd0 0x7fd7a81061b0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fd798009900 tx=0x7fd79802bc30 comp rx=0 tx=0).stop 2024-04-21T09:02:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.404+0000 7fd7acda4640 1 -- 172.21.15.28:0/3452524093 shutdown_connections 2024-04-21T09:02:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.404+0000 7fd7acda4640 1 --2- 172.21.15.28:0/3452524093 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7a8101a40 0x7fd7a810dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.404+0000 7fd7acda4640 1 --2- 172.21.15.28:0/3452524093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7a81010a0 0x7fd7a8101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.404+0000 7fd7acda4640 1 --2- 172.21.15.28:0/3452524093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8105dd0 0x7fd7a81061b0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.404+0000 7fd7acda4640 1 -- 172.21.15.28:0/3452524093 >> 172.21.15.28:0/3452524093 conn(0x7fd7a8077c40 msgr2=0x7fd7a8078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.404+0000 7fd7acda4640 1 -- 172.21.15.28:0/3452524093 shutdown_connections 2024-04-21T09:02:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.404+0000 7fd7acda4640 1 -- 172.21.15.28:0/3452524093 wait complete. 2024-04-21T09:02:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.405+0000 7fd7acda4640 1 Processor -- start 2024-04-21T09:02:08.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.405+0000 7fd7acda4640 1 -- start start 2024-04-21T09:02:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.405+0000 7fd7acda4640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7a81010a0 0x7fd7a8197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.405+0000 7fd7acda4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8101a40 0x7fd7a8197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.405+0000 7fd7acda4640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7a8105dd0 0x7fd7a819bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.405+0000 7fd7acda4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a81111c0 con 0x7fd7a8101a40 2024-04-21T09:02:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.405+0000 7fd7acda4640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a8111040 con 0x7fd7a8105dd0 2024-04-21T09:02:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.405+0000 7fd7acda4640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a8111340 con 0x7fd7a81010a0 2024-04-21T09:02:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.405+0000 7fd79ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8101a40 0x7fd7a8197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:08.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.405+0000 7fd79ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8101a40 0x7fd7a8197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35892/0 (socket says 172.21.15.28:35892) 2024-04-21T09:02:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.405+0000 7fd79ffff640 1 -- 172.21.15.28:0/3281920863 learned_addr learned my addr 172.21.15.28:0/3281920863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:02:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.406+0000 7fd7a659c640 1 --2- 172.21.15.28:0/3281920863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7a81010a0 0x7fd7a8197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.406+0000 7fd79ffff640 1 -- 172.21.15.28:0/3281920863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7a81010a0 msgr2=0x7fd7a8197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.406+0000 7fd79ffff640 1 --2- 172.21.15.28:0/3281920863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7a81010a0 0x7fd7a8197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.406+0000 7fd79ffff640 1 -- 172.21.15.28:0/3281920863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7a8105dd0 msgr2=0x7fd7a819bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.406+0000 7fd7a6d9d640 1 --2- 172.21.15.28:0/3281920863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7a8105dd0 0x7fd7a819bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.406+0000 7fd79ffff640 1 --2- 172.21.15.28:0/3281920863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7a8105dd0 0x7fd7a819bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.406+0000 7fd79ffff640 1 -- 172.21.15.28:0/3281920863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a819c620 con 0x7fd7a8101a40 2024-04-21T09:02:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.406+0000 7fd7a659c640 1 --2- 172.21.15.28:0/3281920863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7a81010a0 0x7fd7a8197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:02:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.406+0000 7fd79ffff640 1 --2- 172.21.15.28:0/3281920863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8101a40 0x7fd7a8197bc0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fd79000d950 tx=0x7fd79000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.406+0000 7fd79f7fe640 1 -- 172.21.15.28:0/3281920863 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd790017030 con 0x7fd7a8101a40 2024-04-21T09:02:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.406+0000 7fd7acda4640 1 -- 172.21.15.28:0/3281920863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a819c910 con 0x7fd7a8101a40 2024-04-21T09:02:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.406+0000 7fd79f7fe640 1 -- 172.21.15.28:0/3281920863 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd790004020 con 0x7fd7a8101a40 2024-04-21T09:02:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.406+0000 7fd79f7fe640 1 -- 172.21.15.28:0/3281920863 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7900047a0 con 0x7fd7a8101a40 2024-04-21T09:02:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.406+0000 7fd7acda4640 1 -- 172.21.15.28:0/3281920863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a81a41a0 con 0x7fd7a8101a40 2024-04-21T09:02:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.407+0000 7fd7acda4640 1 -- 172.21.15.28:0/3281920863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7a8102540 con 0x7fd7a8101a40 2024-04-21T09:02:08.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.408+0000 7fd79f7fe640 1 -- 172.21.15.28:0/3281920863 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd790009a30 con 0x7fd7a8101a40 2024-04-21T09:02:08.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.409+0000 7fd79f7fe640 1 --2- 172.21.15.28:0/3281920863 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd7940a3d20 0x7fd7940a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:08.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.409+0000 7fd7a659c640 1 --2- 172.21.15.28:0/3281920863 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd7940a3d20 0x7fd7940a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:08.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.409+0000 7fd79f7fe640 1 -- 172.21.15.28:0/3281920863 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9791+0+0 (secure 0 0 0) 0x7fd7900d0870 con 0x7fd7a8101a40 2024-04-21T09:02:08.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.409+0000 7fd7a659c640 1 --2- 172.21.15.28:0/3281920863 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd7940a3d20 0x7fd7940a61e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd798006010 tx=0x7fd798005a20 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:08.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.413+0000 7fd79f7fe640 1 -- 172.21.15.28:0/3281920863 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd79009ca20 con 0x7fd7a8101a40 2024-04-21T09:02:08.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:08 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a[27998]: 2024-04-21T09:02:08.282+0000 7fa1fa483700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T09:02:08.620 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-c[61816]: ignoring --setuser ceph since I am not root 2024-04-21T09:02:08.620 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-c[61816]: ignoring --setgroup ceph since I am not root 2024-04-21T09:02:08.620 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-mds[61820]: -- 172.21.15.171:0/894494002 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55adeefeda40 con 0x55adeefc0c00 2024-04-21T09:02:08.621 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-c[61816]: starting mds.c at 2024-04-21T09:02:08.621 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-f[61508]: ignoring --setuser ceph since I am not root 2024-04-21T09:02:08.621 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-f[61508]: ignoring --setgroup ceph since I am not root 2024-04-21T09:02:08.621 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-mds[61512]: -- 172.21.15.171:0/2826284915 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ee48e2ba40 con 0x55ee48dfe800 2024-04-21T09:02:08.621 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-f[61508]: starting mds.f at 2024-04-21T09:02:08.621 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-f[61508]: 2024-04-21T09:02:08.371+0000 7f51b31c4700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6835/0: (98) Address already in use 2024-04-21T09:02:08.622 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-i[62128]: ignoring --setuser ceph since I am not root 2024-04-21T09:02:08.622 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-i[62128]: ignoring --setgroup ceph since I am not root 2024-04-21T09:02:08.622 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-mds[62132]: -- 172.21.15.171:0/2968857337 <== mon.2 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55eda33f24e0 con 0x55eda33c8800 2024-04-21T09:02:08.622 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-mds[62132]: -- 172.21.15.171:0/2968857337 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eda33f5a40 con 0x55eda33c8800 2024-04-21T09:02:08.622 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 09:02:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-i[62128]: starting mds.i at 2024-04-21T09:02:08.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:08.641+0000 7fd7acda4640 1 -- 172.21.15.28:0/3281920863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd7a8109340 con 0x7fd7a8101a40 2024-04-21T09:02:08.855 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-e[61604]: ignoring --setuser ceph since I am not root 2024-04-21T09:02:08.855 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-e[61604]: ignoring --setgroup ceph since I am not root 2024-04-21T09:02:08.855 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-mds[61608]: -- 172.21.15.152:0/2959581594 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a725e55a40 con 0x55a725e28800 2024-04-21T09:02:08.855 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-e[61604]: starting mds.e at 2024-04-21T09:02:08.856 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-b[62223]: ignoring --setuser ceph since I am not root 2024-04-21T09:02:08.856 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-b[62223]: ignoring --setgroup ceph since I am not root 2024-04-21T09:02:08.856 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-mds[62227]: -- 172.21.15.152:0/1086760949 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c3102404e0 con 0x55c310216800 2024-04-21T09:02:08.856 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-mds[62227]: -- 172.21.15.152:0/1086760949 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c310243a40 con 0x55c310216800 2024-04-21T09:02:08.856 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-b[62223]: starting mds.b at 2024-04-21T09:02:08.856 INFO:journalctl@ceph.mds.h.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-h[61913]: ignoring --setuser ceph since I am not root 2024-04-21T09:02:08.856 INFO:journalctl@ceph.mds.h.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-h[61913]: ignoring --setgroup ceph since I am not root 2024-04-21T09:02:08.857 INFO:journalctl@ceph.mds.h.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-mds[61917]: -- 172.21.15.152:0/3389971188 <== mon.2 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557dab8b5a40 con 0x557dab888800 2024-04-21T09:02:08.857 INFO:journalctl@ceph.mds.h.smithi152.stdout:Apr 21 09:02:08 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-h[61913]: starting mds.h at 2024-04-21T09:02:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.290+0000 7fd79f7fe640 1 -- 172.21.15.28:0/3281920863 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) ==== 104+0+0 (secure 0 0 0) 0x7fd79009c320 con 0x7fd7a8101a40 2024-04-21T09:02:09.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.293+0000 7fd7acda4640 1 -- 172.21.15.28:0/3281920863 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd7940a3d20 msgr2=0x7fd7940a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.293+0000 7fd7acda4640 1 --2- 172.21.15.28:0/3281920863 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd7940a3d20 0x7fd7940a61e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd798006010 tx=0x7fd798005a20 comp rx=0 tx=0).stop 2024-04-21T09:02:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.293+0000 7fd7acda4640 1 -- 172.21.15.28:0/3281920863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8101a40 msgr2=0x7fd7a8197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.293+0000 7fd7acda4640 1 --2- 172.21.15.28:0/3281920863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8101a40 0x7fd7a8197bc0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fd79000d950 tx=0x7fd79000de10 comp rx=0 tx=0).stop 2024-04-21T09:02:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.293+0000 7fd7acda4640 1 -- 172.21.15.28:0/3281920863 shutdown_connections 2024-04-21T09:02:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.293+0000 7fd7acda4640 1 --2- 172.21.15.28:0/3281920863 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd7940a3d20 0x7fd7940a61e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.293+0000 7fd7acda4640 1 --2- 172.21.15.28:0/3281920863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7a8105dd0 0x7fd7a819bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.293+0000 7fd7acda4640 1 --2- 172.21.15.28:0/3281920863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8101a40 0x7fd7a8197bc0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.293+0000 7fd7acda4640 1 --2- 172.21.15.28:0/3281920863 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd7a81010a0 0x7fd7a8197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.293+0000 7fd7acda4640 1 -- 172.21.15.28:0/3281920863 >> 172.21.15.28:0/3281920863 conn(0x7fd7a8077c40 msgr2=0x7fd7a80feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.293+0000 7fd7acda4640 1 -- 172.21.15.28:0/3281920863 shutdown_connections 2024-04-21T09:02:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.293+0000 7fd7acda4640 1 -- 172.21.15.28:0/3281920863 wait complete. 2024-04-21T09:02:09.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-21T09:02:09.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:09 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2533249596' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T09:02:09.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:09 smithi171 ceph-mon[31358]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T09:02:09.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:09 smithi171 ceph-mon[31358]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T09:02:09.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:09 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/2533249596' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T09:02:09.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:09 smithi171 ceph-mon[31358]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T09:02:09.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:09 smithi171 ceph-mon[31358]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-21T09:02:09.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:09 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3281920863' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T09:02:09.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.420+0000 7f3e20a06640 1 -- 172.21.15.28:0/2594214774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e1c10c500 msgr2=0x7f3e1c10e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:09.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.421+0000 7f3e20a06640 1 --2- 172.21.15.28:0/2594214774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e1c10c500 0x7f3e1c10e9a0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009900 tx=0x7f3e1402bd60 comp rx=0 tx=0).stop 2024-04-21T09:02:09.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.421+0000 7f3e20a06640 1 -- 172.21.15.28:0/2594214774 shutdown_connections 2024-04-21T09:02:09.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.421+0000 7f3e20a06640 1 --2- 172.21.15.28:0/2594214774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e1c10c500 0x7f3e1c10e9a0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:09.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.421+0000 7f3e20a06640 1 --2- 172.21.15.28:0/2594214774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e1c103ae0 0x7f3e1c10bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:09.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.421+0000 7f3e20a06640 1 --2- 172.21.15.28:0/2594214774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e1c1031c0 0x7f3e1c1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:09.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.421+0000 7f3e20a06640 1 -- 172.21.15.28:0/2594214774 >> 172.21.15.28:0/2594214774 conn(0x7f3e1c078000 msgr2=0x7f3e1c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:09.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.421+0000 7f3e20a06640 1 -- 172.21.15.28:0/2594214774 shutdown_connections 2024-04-21T09:02:09.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.421+0000 7f3e20a06640 1 -- 172.21.15.28:0/2594214774 wait complete. 2024-04-21T09:02:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.421+0000 7f3e20a06640 1 Processor -- start 2024-04-21T09:02:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.421+0000 7f3e20a06640 1 -- start start 2024-04-21T09:02:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e20a06640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e1c1031c0 0x7f3e1c198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e20a06640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e1c103ae0 0x7f3e1c198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e20a06640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e1c10c500 0x7f3e1c19cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e20a06640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1121e0 con 0x7f3e1c1031c0 2024-04-21T09:02:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e20a06640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c112060 con 0x7f3e1c103ae0 2024-04-21T09:02:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e20a06640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c112360 con 0x7f3e1c10c500 2024-04-21T09:02:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e1a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e1c1031c0 0x7f3e1c198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e1a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e1c1031c0 0x7f3e1c198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35900/0 (socket says 172.21.15.28:35900) 2024-04-21T09:02:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e1a59c640 1 -- 172.21.15.28:0/228798776 learned_addr learned my addr 172.21.15.28:0/228798776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:02:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e1ad9d640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e1c10c500 0x7f3e1c19cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e19d9b640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e1c103ae0 0x7f3e1c198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e1a59c640 1 -- 172.21.15.28:0/228798776 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e1c10c500 msgr2=0x7f3e1c19cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e1a59c640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e1c10c500 0x7f3e1c19cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e1a59c640 1 -- 172.21.15.28:0/228798776 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e1c103ae0 msgr2=0x7f3e1c198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e1a59c640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e1c103ae0 0x7f3e1c198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e1a59c640 1 -- 172.21.15.28:0/228798776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1c19d6a0 con 0x7f3e1c1031c0 2024-04-21T09:02:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e1ad9d640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e1c10c500 0x7f3e1c19cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:02:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e19d9b640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e1c103ae0 0x7f3e1c198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T09:02:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e1a59c640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e1c1031c0 0x7f3e1c198700 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c00eab0 tx=0x7f3e0c00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e0b7fe640 1 -- 172.21.15.28:0/228798776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c00ce40 con 0x7f3e1c1031c0 2024-04-21T09:02:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e20a06640 1 -- 172.21.15.28:0/228798776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e1c19d990 con 0x7f3e1c1031c0 2024-04-21T09:02:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.422+0000 7f3e0b7fe640 1 -- 172.21.15.28:0/228798776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e0c011920 con 0x7f3e1c1031c0 2024-04-21T09:02:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.423+0000 7f3e20a06640 1 -- 172.21.15.28:0/228798776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e1c1a5220 con 0x7f3e1c1031c0 2024-04-21T09:02:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.423+0000 7f3e0b7fe640 1 -- 172.21.15.28:0/228798776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c0109d0 con 0x7f3e1c1031c0 2024-04-21T09:02:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.423+0000 7f3e20a06640 1 -- 172.21.15.28:0/228798776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e1c068d70 con 0x7f3e1c1031c0 2024-04-21T09:02:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.425+0000 7f3e0b7fe640 1 -- 172.21.15.28:0/228798776 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3e0c011470 con 0x7f3e1c1031c0 2024-04-21T09:02:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.425+0000 7f3e0b7fe640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3e040a3d20 0x7f3e040a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.425+0000 7f3e0b7fe640 1 -- 172.21.15.28:0/228798776 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9791+0+0 (secure 0 0 0) 0x7f3e0c0d1360 con 0x7f3e1c1031c0 2024-04-21T09:02:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.425+0000 7f3e19d9b640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3e040a3d20 0x7f3e040a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.426+0000 7f3e19d9b640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3e040a3d20 0x7f3e040a61e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3e10005d70 tx=0x7f3e10005d00 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.430+0000 7f3e0b7fe640 1 -- 172.21.15.28:0/228798776 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e0c016030 con 0x7f3e1c1031c0 2024-04-21T09:02:09.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:09 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2533249596' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T09:02:09.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:09 smithi152 ceph-mon[31401]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T09:02:09.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:09 smithi152 ceph-mon[31401]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T09:02:09.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:09 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/2533249596' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T09:02:09.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:09 smithi152 ceph-mon[31401]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T09:02:09.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:09 smithi152 ceph-mon[31401]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-21T09:02:09.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:09 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3281920863' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T09:02:09.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:09 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2533249596' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T09:02:09.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:09 smithi028 ceph-mon[28003]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T09:02:09.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:09 smithi028 ceph-mon[28003]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T09:02:09.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:09 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/2533249596' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T09:02:09.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:09 smithi028 ceph-mon[28003]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T09:02:09.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:09 smithi028 ceph-mon[28003]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-21T09:02:09.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:09 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3281920863' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T09:02:09.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:09.616+0000 7f3e20a06640 1 -- 172.21.15.28:0/228798776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3e1c1076b0 con 0x7f3e1c1031c0 2024-04-21T09:02:10.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.292+0000 7f3e0b7fe640 1 -- 172.21.15.28:0/228798776 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v105) ==== 181+0+0 (secure 0 0 0) 0x7f3e0c09d510 con 0x7f3e1c1031c0 2024-04-21T09:02:10.292 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-04-21T09:02:10.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.296+0000 7f3e20a06640 1 -- 172.21.15.28:0/228798776 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3e040a3d20 msgr2=0x7f3e040a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:10.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.296+0000 7f3e20a06640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3e040a3d20 0x7f3e040a61e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3e10005d70 tx=0x7f3e10005d00 comp rx=0 tx=0).stop 2024-04-21T09:02:10.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.296+0000 7f3e20a06640 1 -- 172.21.15.28:0/228798776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e1c1031c0 msgr2=0x7f3e1c198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:10.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.296+0000 7f3e20a06640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e1c1031c0 0x7f3e1c198700 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c00eab0 tx=0x7f3e0c00ef70 comp rx=0 tx=0).stop 2024-04-21T09:02:10.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.296+0000 7f3e20a06640 1 -- 172.21.15.28:0/228798776 shutdown_connections 2024-04-21T09:02:10.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.296+0000 7f3e20a06640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f3e040a3d20 0x7f3e040a61e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:10.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.296+0000 7f3e20a06640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e1c10c500 0x7f3e1c19cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:10.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.296+0000 7f3e20a06640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e1c103ae0 0x7f3e1c198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:10.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.296+0000 7f3e20a06640 1 --2- 172.21.15.28:0/228798776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e1c1031c0 0x7f3e1c198700 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:10.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.296+0000 7f3e20a06640 1 -- 172.21.15.28:0/228798776 >> 172.21.15.28:0/228798776 conn(0x7f3e1c078000 msgr2=0x7f3e1c0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:10.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.296+0000 7f3e20a06640 1 -- 172.21.15.28:0/228798776 shutdown_connections 2024-04-21T09:02:10.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.296+0000 7f3e20a06640 1 -- 172.21.15.28:0/228798776 wait complete. 2024-04-21T09:02:10.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: pgmap v342: 129 pgs: 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 68 KiB/s wr, 35 op/s 2024-04-21T09:02:10.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T09:02:10.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T09:02:10.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: Cluster is now healthy 2024-04-21T09:02:10.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/3281920863' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T09:02:10.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.171:6834/2451012016,v1:172.21.15.171:6837/2451012016] up:boot 2024-04-21T09:02:10.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.171:6836/552942422,v1:172.21.15.171:6839/552942422] up:boot 2024-04-21T09:02:10.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.171:6835/57298030,v1:172.21.15.171:6838/57298030] up:boot 2024-04-21T09:02:10.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.152:6835/3684117972,v1:172.21.15.152:6838/3684117972] up:boot 2024-04-21T09:02:10.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.152:6834/1158605778,v1:172.21.15.152:6837/1158605778] up:boot 2024-04-21T09:02:10.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: mds.? [v2:172.21.15.152:6836/444502551,v1:172.21.15.152:6839/444502551] up:boot 2024-04-21T09:02:10.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: fsmap 12 up:standby 2024-04-21T09:02:10.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T09:02:10.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T09:02:10.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T09:02:10.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T09:02:10.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T09:02:10.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T09:02:10.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:10 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/228798776' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T09:02:10.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-21T09:02:10.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: pgmap v342: 129 pgs: 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 68 KiB/s wr, 35 op/s 2024-04-21T09:02:10.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T09:02:10.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T09:02:10.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: Cluster is now healthy 2024-04-21T09:02:10.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/3281920863' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T09:02:10.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.171:6834/2451012016,v1:172.21.15.171:6837/2451012016] up:boot 2024-04-21T09:02:10.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.171:6836/552942422,v1:172.21.15.171:6839/552942422] up:boot 2024-04-21T09:02:10.371 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.171:6835/57298030,v1:172.21.15.171:6838/57298030] up:boot 2024-04-21T09:02:10.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.152:6835/3684117972,v1:172.21.15.152:6838/3684117972] up:boot 2024-04-21T09:02:10.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.152:6834/1158605778,v1:172.21.15.152:6837/1158605778] up:boot 2024-04-21T09:02:10.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: mds.? [v2:172.21.15.152:6836/444502551,v1:172.21.15.152:6839/444502551] up:boot 2024-04-21T09:02:10.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: fsmap 12 up:standby 2024-04-21T09:02:10.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T09:02:10.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T09:02:10.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T09:02:10.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T09:02:10.372 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T09:02:10.373 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T09:02:10.373 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:10 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/228798776' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T09:02:10.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.431+0000 7f8e3bf66640 1 -- 172.21.15.28:0/894272900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34102190 msgr2=0x7f8e34102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.431+0000 7f8e3bf66640 1 --2- 172.21.15.28:0/894272900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34102190 0x7f8e34102570 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f8e30009900 tx=0x7f8e3002bc30 comp rx=0 tx=0).stop 2024-04-21T09:02:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.431+0000 7f8e3bf66640 1 -- 172.21.15.28:0/894272900 shutdown_connections 2024-04-21T09:02:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.431+0000 7f8e3bf66640 1 --2- 172.21.15.28:0/894272900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e34107260 0x7f8e3410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.431+0000 7f8e3bf66640 1 --2- 172.21.15.28:0/894272900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e34102b40 0x7f8e34106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.431+0000 7f8e3bf66640 1 --2- 172.21.15.28:0/894272900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34102190 0x7f8e34102570 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.431+0000 7f8e3bf66640 1 -- 172.21.15.28:0/894272900 >> 172.21.15.28:0/894272900 conn(0x7f8e340fd990 msgr2=0x7f8e340ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.431+0000 7f8e3bf66640 1 -- 172.21.15.28:0/894272900 shutdown_connections 2024-04-21T09:02:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.431+0000 7f8e3bf66640 1 -- 172.21.15.28:0/894272900 wait complete. 2024-04-21T09:02:10.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e3bf66640 1 Processor -- start 2024-04-21T09:02:10.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e3bf66640 1 -- start start 2024-04-21T09:02:10.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e3bf66640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e34102190 0x7f8e3419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:10.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e3bf66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34107260 0x7f8e3419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:10.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e3bf66640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e34198600 0x7f8e34198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:10.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e3bf66640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e341121a0 con 0x7f8e34107260 2024-04-21T09:02:10.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e3bf66640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7f8e34112020 con 0x7f8e34198600 2024-04-21T09:02:10.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e3bf66640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7f8e34112320 con 0x7f8e34102190 2024-04-21T09:02:10.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e39501640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34107260 0x7f8e3419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:10.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e39501640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34107260 0x7f8e3419e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35922/0 (socket says 172.21.15.28:35922) 2024-04-21T09:02:10.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e39501640 1 -- 172.21.15.28:0/383827291 learned_addr learned my addr 172.21.15.28:0/383827291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:02:10.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e39d02640 1 --2- 172.21.15.28:0/383827291 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e34102190 0x7f8e3419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:10.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e39501640 1 -- 172.21.15.28:0/383827291 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e34102190 msgr2=0x7f8e3419e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:10.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e39501640 1 --2- 172.21.15.28:0/383827291 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e34102190 0x7f8e3419e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:10.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e3a503640 1 --2- 172.21.15.28:0/383827291 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e34198600 0x7f8e34198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:10.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e39501640 1 -- 172.21.15.28:0/383827291 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e34198600 msgr2=0x7f8e34198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:10.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e39501640 1 --2- 172.21.15.28:0/383827291 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e34198600 0x7f8e34198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:10.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e39501640 1 -- 172.21.15.28:0/383827291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e34199120 con 0x7f8e34107260 2024-04-21T09:02:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.432+0000 7f8e39d02640 1 --2- 172.21.15.28:0/383827291 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e34102190 0x7f8e3419e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:02:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.433+0000 7f8e39501640 1 --2- 172.21.15.28:0/383827291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34107260 0x7f8e3419e940 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f8e2400d770 tx=0x7f8e2400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.433+0000 7f8e2affd640 1 -- 172.21.15.28:0/383827291 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e24017070 con 0x7f8e34107260 2024-04-21T09:02:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.433+0000 7f8e2affd640 1 -- 172.21.15.28:0/383827291 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e24009c30 con 0x7f8e34107260 2024-04-21T09:02:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.433+0000 7f8e3bf66640 1 -- 172.21.15.28:0/383827291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e34199410 con 0x7f8e34107260 2024-04-21T09:02:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.433+0000 7f8e2affd640 1 -- 172.21.15.28:0/383827291 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2400f620 con 0x7f8e34107260 2024-04-21T09:02:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.433+0000 7f8e3bf66640 1 -- 172.21.15.28:0/383827291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e341a5610 con 0x7f8e34107260 2024-04-21T09:02:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.434+0000 7f8e3bf66640 1 -- 172.21.15.28:0/383827291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e340691c0 con 0x7f8e34107260 2024-04-21T09:02:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.435+0000 7f8e2affd640 1 -- 172.21.15.28:0/383827291 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8e2401a070 con 0x7f8e34107260 2024-04-21T09:02:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.436+0000 7f8e2affd640 1 --2- 172.21.15.28:0/383827291 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8e200a3d20 0x7f8e200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.436+0000 7f8e39d02640 1 --2- 172.21.15.28:0/383827291 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8e200a3d20 0x7f8e200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.436+0000 7f8e2affd640 1 -- 172.21.15.28:0/383827291 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9237+0+0 (secure 0 0 0) 0x7f8e240d06f0 con 0x7f8e34107260 2024-04-21T09:02:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.436+0000 7f8e39d02640 1 --2- 172.21.15.28:0/383827291 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8e200a3d20 0x7f8e200a61e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8e30036040 tx=0x7f8e30000bc0 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.440+0000 7f8e2affd640 1 -- 172.21.15.28:0/383827291 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8e24009de0 con 0x7f8e34107260 2024-04-21T09:02:10.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: pgmap v342: 129 pgs: 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 122 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 68 KiB/s wr, 35 op/s 2024-04-21T09:02:10.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T09:02:10.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T09:02:10.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: Cluster is now healthy 2024-04-21T09:02:10.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/3281920863' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T09:02:10.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.171:6834/2451012016,v1:172.21.15.171:6837/2451012016] up:boot 2024-04-21T09:02:10.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.171:6836/552942422,v1:172.21.15.171:6839/552942422] up:boot 2024-04-21T09:02:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.171:6835/57298030,v1:172.21.15.171:6838/57298030] up:boot 2024-04-21T09:02:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.152:6835/3684117972,v1:172.21.15.152:6838/3684117972] up:boot 2024-04-21T09:02:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.152:6834/1158605778,v1:172.21.15.152:6837/1158605778] up:boot 2024-04-21T09:02:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: mds.? [v2:172.21.15.152:6836/444502551,v1:172.21.15.152:6839/444502551] up:boot 2024-04-21T09:02:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: fsmap 12 up:standby 2024-04-21T09:02:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T09:02:10.606 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T09:02:10.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T09:02:10.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T09:02:10.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T09:02:10.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: from='mgr.25040 172.21.15.28:0/623553805' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T09:02:10.607 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:10 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/228798776' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T09:02:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:10.630+0000 7f8e3bf66640 1 -- 172.21.15.28:0/383827291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8e34102b40 con 0x7f8e34107260 2024-04-21T09:02:11.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.297+0000 7f8e2affd640 1 -- 172.21.15.28:0/383827291 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v106) ==== 169+0+0 (secure 0 0 0) 0x7f8e2409cac0 con 0x7f8e34107260 2024-04-21T09:02:11.298 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-04-21T09:02:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.301+0000 7f8e3bf66640 1 -- 172.21.15.28:0/383827291 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8e200a3d20 msgr2=0x7f8e200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.301+0000 7f8e3bf66640 1 --2- 172.21.15.28:0/383827291 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8e200a3d20 0x7f8e200a61e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8e30036040 tx=0x7f8e30000bc0 comp rx=0 tx=0).stop 2024-04-21T09:02:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.301+0000 7f8e3bf66640 1 -- 172.21.15.28:0/383827291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34107260 msgr2=0x7f8e3419e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:11.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.301+0000 7f8e3bf66640 1 --2- 172.21.15.28:0/383827291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34107260 0x7f8e3419e940 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f8e2400d770 tx=0x7f8e2400dc30 comp rx=0 tx=0).stop 2024-04-21T09:02:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.301+0000 7f8e3bf66640 1 -- 172.21.15.28:0/383827291 shutdown_connections 2024-04-21T09:02:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.301+0000 7f8e3bf66640 1 --2- 172.21.15.28:0/383827291 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7f8e200a3d20 0x7f8e200a61e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.301+0000 7f8e3bf66640 1 --2- 172.21.15.28:0/383827291 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e34198600 0x7f8e34198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.301+0000 7f8e3bf66640 1 --2- 172.21.15.28:0/383827291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e34107260 0x7f8e3419e940 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.301+0000 7f8e3bf66640 1 --2- 172.21.15.28:0/383827291 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e34102190 0x7f8e3419e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.301+0000 7f8e3bf66640 1 -- 172.21.15.28:0/383827291 >> 172.21.15.28:0/383827291 conn(0x7f8e340fd990 msgr2=0x7f8e340fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:11.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.301+0000 7f8e3bf66640 1 -- 172.21.15.28:0/383827291 shutdown_connections 2024-04-21T09:02:11.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.302+0000 7f8e3bf66640 1 -- 172.21.15.28:0/383827291 wait complete. 2024-04-21T09:02:11.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-21T09:02:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.436+0000 7fd56ae9a640 1 -- 172.21.15.28:0/2566516280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd56406af70 msgr2=0x7fd56410ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.436+0000 7fd56ae9a640 1 --2- 172.21.15.28:0/2566516280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd56406af70 0x7fd56410ede0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fd55c009930 tx=0x7fd55c02bc30 comp rx=0 tx=0).stop 2024-04-21T09:02:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.436+0000 7fd56ae9a640 1 -- 172.21.15.28:0/2566516280 shutdown_connections 2024-04-21T09:02:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.436+0000 7fd56ae9a640 1 --2- 172.21.15.28:0/2566516280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd56406af70 0x7fd56410ede0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.436+0000 7fd56ae9a640 1 --2- 172.21.15.28:0/2566516280 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd56406a5b0 0x7fd56406aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.436+0000 7fd56ae9a640 1 --2- 172.21.15.28:0/2566516280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd564108060 0x7fd564108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.436+0000 7fd56ae9a640 1 -- 172.21.15.28:0/2566516280 >> 172.21.15.28:0/2566516280 conn(0x7fd564077c40 msgr2=0x7fd564078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.436+0000 7fd56ae9a640 1 -- 172.21.15.28:0/2566516280 shutdown_connections 2024-04-21T09:02:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.436+0000 7fd56ae9a640 1 -- 172.21.15.28:0/2566516280 wait complete. 2024-04-21T09:02:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd56ae9a640 1 Processor -- start 2024-04-21T09:02:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd56ae9a640 1 -- start start 2024-04-21T09:02:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd56ae9a640 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd56406a5b0 0x7fd564198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd56ae9a640 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd56406af70 0x7fd564198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd56ae9a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd564108060 0x7fd56419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd56ae9a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5641121a0 con 0x7fd564108060 2024-04-21T09:02:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd56ae9a640 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 -- 0x7fd564112020 con 0x7fd56406af70 2024-04-21T09:02:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd56ae9a640 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 -- 0x7fd564112320 con 0x7fd56406a5b0 2024-04-21T09:02:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd569437640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd564108060 0x7fd56419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd569437640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd564108060 0x7fd56419cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35936/0 (socket says 172.21.15.28:35936) 2024-04-21T09:02:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd569437640 1 -- 172.21.15.28:0/1338645831 learned_addr learned my addr 172.21.15.28:0/1338645831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-21T09:02:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd568c36640 1 --2- 172.21.15.28:0/1338645831 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd56406a5b0 0x7fd564198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd563fff640 1 --2- 172.21.15.28:0/1338645831 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd56406af70 0x7fd564198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd569437640 1 -- 172.21.15.28:0/1338645831 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd56406a5b0 msgr2=0x7fd564198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd569437640 1 --2- 172.21.15.28:0/1338645831 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd56406a5b0 0x7fd564198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd569437640 1 -- 172.21.15.28:0/1338645831 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd56406af70 msgr2=0x7fd564198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd569437640 1 --2- 172.21.15.28:0/1338645831 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd56406af70 0x7fd564198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd569437640 1 -- 172.21.15.28:0/1338645831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd56419d610 con 0x7fd564108060 2024-04-21T09:02:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd563fff640 1 --2- 172.21.15.28:0/1338645831 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd56406af70 0x7fd564198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T09:02:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.437+0000 7fd569437640 1 --2- 172.21.15.28:0/1338645831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd564108060 0x7fd56419cee0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd55c005940 tx=0x7fd55c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.438+0000 7fd561ffb640 1 -- 172.21.15.28:0/1338645831 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55c043020 con 0x7fd564108060 2024-04-21T09:02:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.438+0000 7fd561ffb640 1 -- 172.21.15.28:0/1338645831 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd55c02cb70 con 0x7fd564108060 2024-04-21T09:02:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.438+0000 7fd561ffb640 1 -- 172.21.15.28:0/1338645831 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55c003b50 con 0x7fd564108060 2024-04-21T09:02:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.438+0000 7fd56ae9a640 1 -- 172.21.15.28:0/1338645831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd56419d8a0 con 0x7fd564108060 2024-04-21T09:02:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.438+0000 7fd56ae9a640 1 -- 172.21.15.28:0/1338645831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5641a5190 con 0x7fd564108060 2024-04-21T09:02:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.439+0000 7fd56ae9a640 1 -- 172.21.15.28:0/1338645831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd564103650 con 0x7fd564108060 2024-04-21T09:02:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.440+0000 7fd561ffb640 1 -- 172.21.15.28:0/1338645831 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd55c046020 con 0x7fd564108060 2024-04-21T09:02:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.441+0000 7fd561ffb640 1 --2- 172.21.15.28:0/1338645831 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd5500a3da0 0x7fd5500a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T09:02:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.441+0000 7fd568c36640 1 --2- 172.21.15.28:0/1338645831 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd5500a3da0 0x7fd5500a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T09:02:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.441+0000 7fd561ffb640 1 -- 172.21.15.28:0/1338645831 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8670+0+0 (secure 0 0 0) 0x7fd55c0f2a50 con 0x7fd564108060 2024-04-21T09:02:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.441+0000 7fd568c36640 1 --2- 172.21.15.28:0/1338645831 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd5500a3da0 0x7fd5500a6260 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd54c005fd0 tx=0x7fd54c005850 comp rx=0 tx=0).ready entity=mgr.25040 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T09:02:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.445+0000 7fd561ffb640 1 -- 172.21.15.28:0/1338645831 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd55c04d020 con 0x7fd564108060 2024-04-21T09:02:11.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:11 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/228798776' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T09:02:11.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:11 smithi152 ceph-mon[31401]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T09:02:11.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:11 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/383827291' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T09:02:11.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:11 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/228798776' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T09:02:11.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:11 smithi028 ceph-mon[28003]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T09:02:11.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:11 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/383827291' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T09:02:11.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:11 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/228798776' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T09:02:11.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:11 smithi171 ceph-mon[31358]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T09:02:11.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:11 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/383827291' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T09:02:11.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:11.639+0000 7fd56ae9a640 1 -- 172.21.15.28:0/1338645831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd56410a4a0 con 0x7fd564108060 2024-04-21T09:02:12.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:12.299+0000 7fd561ffb640 1 -- 172.21.15.28:0/1338645831 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v107) ==== 178+0+0 (secure 0 0 0) 0x7fd55c03c350 con 0x7fd564108060 2024-04-21T09:02:12.299 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' removed 2024-04-21T09:02:12.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:12.303+0000 7fd56ae9a640 1 -- 172.21.15.28:0/1338645831 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd5500a3da0 msgr2=0x7fd5500a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:12.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:12.303+0000 7fd56ae9a640 1 --2- 172.21.15.28:0/1338645831 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd5500a3da0 0x7fd5500a6260 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd54c005fd0 tx=0x7fd54c005850 comp rx=0 tx=0).stop 2024-04-21T09:02:12.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:12.303+0000 7fd56ae9a640 1 -- 172.21.15.28:0/1338645831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd564108060 msgr2=0x7fd56419cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T09:02:12.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:12.303+0000 7fd56ae9a640 1 --2- 172.21.15.28:0/1338645831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd564108060 0x7fd56419cee0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd55c005940 tx=0x7fd55c0045e0 comp rx=0 tx=0).stop 2024-04-21T09:02:12.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:12.303+0000 7fd56ae9a640 1 -- 172.21.15.28:0/1338645831 shutdown_connections 2024-04-21T09:02:12.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:12.303+0000 7fd56ae9a640 1 --2- 172.21.15.28:0/1338645831 >> [v2:172.21.15.28:6800/4182301910,v1:172.21.15.28:6801/4182301910] conn(0x7fd5500a3da0 0x7fd5500a6260 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:12.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:12.303+0000 7fd56ae9a640 1 --2- 172.21.15.28:0/1338645831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd564108060 0x7fd56419cee0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:12.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:12.303+0000 7fd56ae9a640 1 --2- 172.21.15.28:0/1338645831 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd56406af70 0x7fd564198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:12.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:12.303+0000 7fd56ae9a640 1 --2- 172.21.15.28:0/1338645831 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd56406a5b0 0x7fd564198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T09:02:12.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:12.303+0000 7fd56ae9a640 1 -- 172.21.15.28:0/1338645831 >> 172.21.15.28:0/1338645831 conn(0x7fd564077c40 msgr2=0x7fd5640ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T09:02:12.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:12.303+0000 7fd56ae9a640 1 -- 172.21.15.28:0/1338645831 shutdown_connections 2024-04-21T09:02:12.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-21T09:02:12.303+0000 7fd56ae9a640 1 -- 172.21.15.28:0/1338645831 wait complete. 2024-04-21T09:02:12.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T09:02:12.357 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T09:02:12.391 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T09:02:12.421 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-21T09:02:12.421 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T09:02:12.439 DEBUG:teuthology.orchestra.run.smithi152:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T09:02:12.455 DEBUG:teuthology.orchestra.run.smithi171:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T09:02:12.476 INFO:tasks.cephadm:Stopping all daemons... 2024-04-21T09:02:12.476 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-21T09:02:12.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mon.a 2024-04-21T09:02:12.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:12 smithi152 ceph-mon[31401]: pgmap v344: 97 pgs: 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 90 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T09:02:12.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:12 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/383827291' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T09:02:12.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:12 smithi152 ceph-mon[31401]: osdmap e106: 12 total, 12 up, 12 in 2024-04-21T09:02:12.605 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:12 smithi152 ceph-mon[31401]: from='client.? 172.21.15.28:0/1338645831' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T09:02:12.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:12 smithi028 ceph-mon[28003]: pgmap v344: 97 pgs: 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 90 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T09:02:12.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:12 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/383827291' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T09:02:12.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:12 smithi028 ceph-mon[28003]: osdmap e106: 12 total, 12 up, 12 in 2024-04-21T09:02:12.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:12 smithi028 ceph-mon[28003]: from='client.? 172.21.15.28:0/1338645831' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T09:02:12.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:12 smithi028 systemd[1]: Stopping Ceph mon.a for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:02:12.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:12 smithi171 ceph-mon[31358]: pgmap v344: 97 pgs: 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 90 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T09:02:12.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:12 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/383827291' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T09:02:12.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:12 smithi171 ceph-mon[31358]: osdmap e106: 12 total, 12 up, 12 in 2024-04-21T09:02:12.620 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:12 smithi171 ceph-mon[31358]: from='client.? 172.21.15.28:0/1338645831' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T09:02:13.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a[27998]: 2024-04-21T09:02:12.851+0000 7fa1ffc8e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:02:13.105 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:12 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a[27998]: 2024-04-21T09:02:12.851+0000 7fa1ffc8e700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-21T09:02:13.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:13 smithi028 podman[101236]: 2024-04-21 09:02:13.12613453 +0000 UTC m=+0.395406449 container died 4043afed007a0acf9d66e917e9be68c5f83a0d818afe8a0bc85c7831d16a515c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-21T09:02:13.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:13 smithi028 podman[101236]: 2024-04-21 09:02:13.500561537 +0000 UTC m=+0.769833464 container cleanup 4043afed007a0acf9d66e917e9be68c5f83a0d818afe8a0bc85c7831d16a515c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2) 2024-04-21T09:02:13.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:13 smithi028 bash[101236]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a 2024-04-21T09:02:14.086 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 21 09:02:13 smithi028 podman[101250]: 2024-04-21 09:02:13.835579872 +0000 UTC m=+0.705828601 container remove 4043afed007a0acf9d66e917e9be68c5f83a0d818afe8a0bc85c7831d16a515c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-21T09:02:14.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mon.a.service' 2024-04-21T09:02:14.180 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:02:14.180 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-21T09:02:14.181 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-21T09:02:14.181 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mon.b 2024-04-21T09:02:14.604 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:14 smithi152 systemd[1]: Stopping Ceph mon.b for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:02:15.104 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:14 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-b[31397]: 2024-04-21T09:02:14.722+0000 7fc8c8239700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:02:15.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:14 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-b[31397]: 2024-04-21T09:02:14.722+0000 7fc8c8239700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-21T09:02:15.105 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:14 smithi152 podman[67679]: 2024-04-21 09:02:14.836848961 +0000 UTC m=+0.259274950 container died 01fa7845fe29d64dda646b47388082716b538eb4b77e8af2158e096cbef91b17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T09:02:15.546 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:15 smithi152 podman[67679]: 2024-04-21 09:02:15.236487725 +0000 UTC m=+0.658913720 container cleanup 01fa7845fe29d64dda646b47388082716b538eb4b77e8af2158e096cbef91b17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T09:02:15.546 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:15 smithi152 bash[67679]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-b 2024-04-21T09:02:15.854 INFO:journalctl@ceph.mon.b.smithi152.stdout:Apr 21 09:02:15 smithi152 podman[67690]: 2024-04-21 09:02:15.546723496 +0000 UTC m=+0.703096941 container remove 01fa7845fe29d64dda646b47388082716b538eb4b77e8af2158e096cbef91b17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416) 2024-04-21T09:02:15.968 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mon.b.service' 2024-04-21T09:02:16.105 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:02:16.106 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-21T09:02:16.106 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-21T09:02:16.106 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mon.c 2024-04-21T09:02:16.616 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:16 smithi171 systemd[1]: Stopping Ceph mon.c for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:02:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:16 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-c[31351]: 2024-04-21T09:02:16.615+0000 7f95f67e8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:02:16.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:16 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-c[31351]: 2024-04-21T09:02:16.615+0000 7f95f67e8700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-21T09:02:16.871 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:16 smithi171 podman[67563]: 2024-04-21 09:02:16.769032296 +0000 UTC m=+0.292711195 container died a03a67da3e6e9471e68e15dd4ed8b1e1aba5c2f79fa2845266d86846f745af4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-21T09:02:17.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:17 smithi171 podman[67563]: 2024-04-21 09:02:17.190272213 +0000 UTC m=+0.713951118 container cleanup a03a67da3e6e9471e68e15dd4ed8b1e1aba5c2f79fa2845266d86846f745af4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T09:02:17.370 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:17 smithi171 bash[67563]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-c 2024-04-21T09:02:17.870 INFO:journalctl@ceph.mon.c.smithi171.stdout:Apr 21 09:02:17 smithi171 podman[67573]: 2024-04-21 09:02:17.425717691 +0000 UTC m=+0.659633516 container remove a03a67da3e6e9471e68e15dd4ed8b1e1aba5c2f79fa2845266d86846f745af4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T09:02:17.893 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mon.c.service' 2024-04-21T09:02:17.953 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:02:17.953 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-21T09:02:17.954 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-21T09:02:17.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mgr.x 2024-04-21T09:02:18.355 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 09:02:18 smithi028 systemd[1]: Stopping Ceph mgr.x for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:02:20.355 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 09:02:20 smithi028 podman[101345]: 2024-04-21 09:02:20.068656817 +0000 UTC m=+1.690641023 container died fa359b820dc10aba01f4fe9af11850e7b550260a7501798dd55eeb42bb6c8a54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T09:02:26.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 09:02:25 smithi028 podman[101345]: 2024-04-21 09:02:25.632434311 +0000 UTC m=+7.254418509 container cleanup fa359b820dc10aba01f4fe9af11850e7b550260a7501798dd55eeb42bb6c8a54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-21T09:02:26.105 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 21 09:02:25 smithi028 bash[101345]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-x 2024-04-21T09:02:28.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mgr.x.service' 2024-04-21T09:02:28.281 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:02:28.281 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-21T09:02:28.282 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-21T09:02:28.282 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mgr.y 2024-04-21T09:02:28.605 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 09:02:28 smithi152 systemd[1]: Stopping Ceph mgr.y for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:02:29.288 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 09:02:28 smithi152 podman[67794]: 2024-04-21 09:02:28.904980375 +0000 UTC m=+0.432471451 container died aceaaf35e2108ba1c7e1625724e782c6713066a7f78d91fbc673ecbb2f403d31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T09:02:29.604 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 09:02:29 smithi152 podman[67794]: 2024-04-21 09:02:29.393118146 +0000 UTC m=+0.920609261 container cleanup aceaaf35e2108ba1c7e1625724e782c6713066a7f78d91fbc673ecbb2f403d31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-21T09:02:29.605 INFO:journalctl@ceph.mgr.y.smithi152.stdout:Apr 21 09:02:29 smithi152 bash[67794]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-y 2024-04-21T09:02:29.838 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mgr.y.service' 2024-04-21T09:02:29.872 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:02:29.872 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-21T09:02:29.872 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-21T09:02:29.872 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mgr.z 2024-04-21T09:02:30.281 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 09:02:29 smithi171 systemd[1]: Stopping Ceph mgr.z for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:02:31.620 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 09:02:31 smithi171 podman[67675]: 2024-04-21 09:02:31.35042985 +0000 UTC m=+1.283798798 container died 0b190d088b4173e6c5e84279d6e799e6faeba1b87bfecedc3dc868ec2bbc392b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-21T09:02:34.536 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 09:02:34 smithi171 podman[67675]: 2024-04-21 09:02:34.326666764 +0000 UTC m=+4.260035717 container cleanup 0b190d088b4173e6c5e84279d6e799e6faeba1b87bfecedc3dc868ec2bbc392b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS) 2024-04-21T09:02:34.537 INFO:journalctl@ceph.mgr.z.smithi171.stdout:Apr 21 09:02:34 smithi171 bash[67675]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mgr-z 2024-04-21T09:02:34.747 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mgr.z.service' 2024-04-21T09:02:34.783 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:02:34.784 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-21T09:02:34.784 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-21T09:02:34.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.0 2024-04-21T09:02:35.104 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 09:02:34 smithi028 systemd[1]: Stopping Ceph osd.0 for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:02:35.355 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 09:02:35 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-0[43046]: 2024-04-21T09:02:35.104+0000 7fa90f383700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:02:35.355 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 09:02:35 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-0[43046]: 2024-04-21T09:02:35.104+0000 7fa90f383700 -1 osd.0 107 *** Got signal Terminated *** 2024-04-21T09:02:35.356 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 09:02:35 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-0[43046]: 2024-04-21T09:02:35.104+0000 7fa90f383700 -1 osd.0 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T09:02:41.105 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 09:02:40 smithi028 podman[101477]: 2024-04-21 09:02:40.852807598 +0000 UTC m=+5.864150136 container died 546645dba2f5fc12e5dd9492d5375f623265b5ca81781b6cfbf794289aeb9961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-21T09:02:41.505 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 09:02:41 smithi028 podman[101477]: 2024-04-21 09:02:41.221381852 +0000 UTC m=+6.232724390 container cleanup 546645dba2f5fc12e5dd9492d5375f623265b5ca81781b6cfbf794289aeb9961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-21T09:02:41.505 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 09:02:41 smithi028 bash[101477]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-0 2024-04-21T09:02:41.855 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 09:02:41 smithi028 podman[101494]: 2024-04-21 09:02:41.506056278 +0000 UTC m=+0.649883964 container remove 546645dba2f5fc12e5dd9492d5375f623265b5ca81781b6cfbf794289aeb9961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-21T09:02:42.105 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 09:02:41 smithi028 podman[101583]: 2024-04-21 09:02:41.758420352 +0000 UTC m=+0.018562559 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T09:02:42.106 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 09:02:41 smithi028 podman[101583]: 2024-04-21 09:02:41.873934909 +0000 UTC m=+0.134077156 container create 3b7d173d0c6b97db872c1facfd6edd5ce06894d00eed4faab89888ed5805be7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240416) 2024-04-21T09:02:42.605 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 09:02:42 smithi028 podman[101583]: 2024-04-21 09:02:42.134918009 +0000 UTC m=+0.395060224 container init 3b7d173d0c6b97db872c1facfd6edd5ce06894d00eed4faab89888ed5805be7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-04-21T09:02:42.606 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 09:02:42 smithi028 podman[101583]: 2024-04-21 09:02:42.138553677 +0000 UTC m=+0.398695885 container start 3b7d173d0c6b97db872c1facfd6edd5ce06894d00eed4faab89888ed5805be7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-21T09:02:42.606 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 09:02:42 smithi028 podman[101583]: 2024-04-21 09:02:42.259646653 +0000 UTC m=+0.519788858 container attach 3b7d173d0c6b97db872c1facfd6edd5ce06894d00eed4faab89888ed5805be7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-04-21T09:02:43.028 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 21 09:02:42 smithi028 podman[101583]: 2024-04-21 09:02:42.721363446 +0000 UTC m=+0.981505652 container died 3b7d173d0c6b97db872c1facfd6edd5ce06894d00eed4faab89888ed5805be7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T09:02:43.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.0.service' 2024-04-21T09:02:43.415 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:02:43.415 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-21T09:02:43.415 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-21T09:02:43.416 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.1 2024-04-21T09:02:43.730 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 09:02:43 smithi152 systemd[1]: Stopping Ceph osd.1 for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:02:44.104 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 09:02:43 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-1[35984]: 2024-04-21T09:02:43.729+0000 7f615e3d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:02:44.105 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 09:02:43 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-1[35984]: 2024-04-21T09:02:43.729+0000 7f615e3d8700 -1 osd.1 107 *** Got signal Terminated *** 2024-04-21T09:02:44.105 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 09:02:43 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-1[35984]: 2024-04-21T09:02:43.729+0000 7f615e3d8700 -1 osd.1 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T09:02:49.703 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 09:02:49 smithi152 podman[67930]: 2024-04-21 09:02:49.385601441 +0000 UTC m=+5.778469643 container died 287af4d070e2a99c729fd527ebf8d68079188403edbe9194e0123b4b47a3af56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T09:02:50.023 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 09:02:49 smithi152 podman[67930]: 2024-04-21 09:02:49.805732564 +0000 UTC m=+6.198600778 container cleanup 287af4d070e2a99c729fd527ebf8d68079188403edbe9194e0123b4b47a3af56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T09:02:50.023 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 09:02:49 smithi152 bash[67930]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-1 2024-04-21T09:02:50.355 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 09:02:50 smithi152 podman[67947]: 2024-04-21 09:02:50.024090878 +0000 UTC m=+0.636791253 container remove 287af4d070e2a99c729fd527ebf8d68079188403edbe9194e0123b4b47a3af56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T09:02:50.727 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 09:02:50 smithi152 podman[68037]: 2024-04-21 09:02:50.324850281 +0000 UTC m=+0.018547563 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T09:02:50.728 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 09:02:50 smithi152 podman[68037]: 2024-04-21 09:02:50.475197134 +0000 UTC m=+0.168894392 container create 637971b0637ba0ccd83d6ac3ff04feaf8ca868d3f91e4fa4d6a4546edc1466e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-21T09:02:50.728 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 09:02:50 smithi152 podman[68037]: 2024-04-21 09:02:50.728188305 +0000 UTC m=+0.421885563 container init 637971b0637ba0ccd83d6ac3ff04feaf8ca868d3f91e4fa4d6a4546edc1466e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-21T09:02:51.104 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 09:02:50 smithi152 podman[68037]: 2024-04-21 09:02:50.731624793 +0000 UTC m=+0.425322057 container start 637971b0637ba0ccd83d6ac3ff04feaf8ca868d3f91e4fa4d6a4546edc1466e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416) 2024-04-21T09:02:51.105 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 09:02:50 smithi152 podman[68037]: 2024-04-21 09:02:50.845352052 +0000 UTC m=+0.539049319 container attach 637971b0637ba0ccd83d6ac3ff04feaf8ca868d3f91e4fa4d6a4546edc1466e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-04-21T09:02:51.593 INFO:journalctl@ceph.osd.1.smithi152.stdout:Apr 21 09:02:51 smithi152 podman[68037]: 2024-04-21 09:02:51.322801923 +0000 UTC m=+1.016499195 container died 637971b0637ba0ccd83d6ac3ff04feaf8ca868d3f91e4fa4d6a4546edc1466e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-21T09:02:52.003 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.1.service' 2024-04-21T09:02:52.041 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:02:52.042 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-21T09:02:52.042 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-21T09:02:52.042 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.2 2024-04-21T09:02:52.370 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 09:02:52 smithi171 systemd[1]: Stopping Ceph osd.2 for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:02:52.870 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 09:02:52 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-2[36540]: 2024-04-21T09:02:52.448+0000 7f7bd4c50700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:02:52.870 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 09:02:52 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-2[36540]: 2024-04-21T09:02:52.448+0000 7f7bd4c50700 -1 osd.2 107 *** Got signal Terminated *** 2024-04-21T09:02:52.870 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 09:02:52 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-2[36540]: 2024-04-21T09:02:52.448+0000 7f7bd4c50700 -1 osd.2 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T09:02:58.370 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 09:02:58 smithi171 podman[67813]: 2024-04-21 09:02:58.075157783 +0000 UTC m=+5.838757409 container died c63ff3178d96d95b9edbc1bcca8162ab22e3f21b15783f4ac4822e96719499d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-2, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-21T09:02:58.734 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 09:02:58 smithi171 podman[67813]: 2024-04-21 09:02:58.474717298 +0000 UTC m=+6.238316921 container cleanup c63ff3178d96d95b9edbc1bcca8162ab22e3f21b15783f4ac4822e96719499d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-2, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T09:02:58.735 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 09:02:58 smithi171 bash[67813]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-2 2024-04-21T09:02:59.036 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 09:02:58 smithi171 podman[67832]: 2024-04-21 09:02:58.735493343 +0000 UTC m=+0.662171190 container remove c63ff3178d96d95b9edbc1bcca8162ab22e3f21b15783f4ac4822e96719499d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T09:02:59.356 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 09:02:59 smithi171 podman[67924]: 2024-04-21 09:02:58.940254167 +0000 UTC m=+0.018656173 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T09:02:59.356 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 09:02:59 smithi171 podman[67924]: 2024-04-21 09:02:59.086729279 +0000 UTC m=+0.165131284 container create 6c09a4e74c6598df70123d5732003a5238549776b0a01378c73f781f5003eb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-21T09:02:59.621 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 09:02:59 smithi171 podman[67924]: 2024-04-21 09:02:59.356990452 +0000 UTC m=+0.435392457 container init 6c09a4e74c6598df70123d5732003a5238549776b0a01378c73f781f5003eb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-21T09:02:59.621 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 09:02:59 smithi171 podman[67924]: 2024-04-21 09:02:59.360377745 +0000 UTC m=+0.438779751 container start 6c09a4e74c6598df70123d5732003a5238549776b0a01378c73f781f5003eb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T09:02:59.622 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 09:02:59 smithi171 podman[67924]: 2024-04-21 09:02:59.504861851 +0000 UTC m=+0.583263865 container attach 6c09a4e74c6598df70123d5732003a5238549776b0a01378c73f781f5003eb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T09:03:00.120 INFO:journalctl@ceph.osd.2.smithi171.stdout:Apr 21 09:02:59 smithi171 podman[67924]: 2024-04-21 09:02:59.862026323 +0000 UTC m=+0.940428319 container died 6c09a4e74c6598df70123d5732003a5238549776b0a01378c73f781f5003eb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-21T09:03:00.766 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.2.service' 2024-04-21T09:03:00.809 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:03:00.809 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-21T09:03:00.809 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-21T09:03:00.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.3 2024-04-21T09:03:01.105 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 09:03:00 smithi028 systemd[1]: Stopping Ceph osd.3 for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:03:01.605 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 09:03:01 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-3[49730]: 2024-04-21T09:03:01.135+0000 7ffa32c44700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:03:01.605 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 09:03:01 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-3[49730]: 2024-04-21T09:03:01.135+0000 7ffa32c44700 -1 osd.3 107 *** Got signal Terminated *** 2024-04-21T09:03:01.605 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 09:03:01 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-3[49730]: 2024-04-21T09:03:01.135+0000 7ffa32c44700 -1 osd.3 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T09:03:07.341 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 09:03:07 smithi028 podman[101701]: 2024-04-21 09:03:07.057102978 +0000 UTC m=+6.041262342 container died 3e33ba8a40ad962f0f06a50183da4be1d444076edc706984c49cf14bc4e9faac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T09:03:07.605 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 09:03:07 smithi028 podman[101701]: 2024-04-21 09:03:07.440428361 +0000 UTC m=+6.424587724 container cleanup 3e33ba8a40ad962f0f06a50183da4be1d444076edc706984c49cf14bc4e9faac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-04-21T09:03:07.605 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 09:03:07 smithi028 bash[101701]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-3 2024-04-21T09:03:08.081 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 09:03:07 smithi028 podman[101730]: 2024-04-21 09:03:07.716859089 +0000 UTC m=+0.660268205 container remove 3e33ba8a40ad962f0f06a50183da4be1d444076edc706984c49cf14bc4e9faac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-3, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-04-21T09:03:08.345 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 09:03:08 smithi028 podman[101821]: 2024-04-21 09:03:07.984907697 +0000 UTC m=+0.018865557 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T09:03:08.345 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 09:03:08 smithi028 podman[101821]: 2024-04-21 09:03:08.109316478 +0000 UTC m=+0.143274339 container create da95bc696d92ec8e90fdc9750124390b6730db5c20e0a626b325605c36a08645 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-3-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-21T09:03:08.606 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 09:03:08 smithi028 podman[101821]: 2024-04-21 09:03:08.345915663 +0000 UTC m=+0.379873525 container init da95bc696d92ec8e90fdc9750124390b6730db5c20e0a626b325605c36a08645 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-04-21T09:03:08.606 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 09:03:08 smithi028 podman[101821]: 2024-04-21 09:03:08.349402059 +0000 UTC m=+0.383359921 container start da95bc696d92ec8e90fdc9750124390b6730db5c20e0a626b325605c36a08645 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-21T09:03:08.606 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 09:03:08 smithi028 podman[101821]: 2024-04-21 09:03:08.453809973 +0000 UTC m=+0.487767847 container attach da95bc696d92ec8e90fdc9750124390b6730db5c20e0a626b325605c36a08645 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-21T09:03:09.022 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 21 09:03:08 smithi028 podman[101821]: 2024-04-21 09:03:08.659961409 +0000 UTC m=+0.693919277 container died da95bc696d92ec8e90fdc9750124390b6730db5c20e0a626b325605c36a08645 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-04-21T09:03:09.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.3.service' 2024-04-21T09:03:09.546 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:03:09.546 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-21T09:03:09.547 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-21T09:03:09.547 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.4 2024-04-21T09:03:09.855 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 09:03:09 smithi152 systemd[1]: Stopping Ceph osd.4 for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:03:10.354 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 09:03:09 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-4[42919]: 2024-04-21T09:03:09.884+0000 7ff4b9537700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:03:10.355 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 09:03:09 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-4[42919]: 2024-04-21T09:03:09.884+0000 7ff4b9537700 -1 osd.4 107 *** Got signal Terminated *** 2024-04-21T09:03:10.355 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 09:03:09 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-4[42919]: 2024-04-21T09:03:09.884+0000 7ff4b9537700 -1 osd.4 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T09:03:16.002 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 09:03:15 smithi152 podman[68159]: 2024-04-21 09:03:15.690472392 +0000 UTC m=+5.944924259 container died 9f8798209e0abb869ed3b917ff83a49518785b5d506c391eb20c30763e94a17f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-4, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-21T09:03:16.355 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 09:03:16 smithi152 podman[68159]: 2024-04-21 09:03:16.094404365 +0000 UTC m=+6.348856224 container cleanup 9f8798209e0abb869ed3b917ff83a49518785b5d506c391eb20c30763e94a17f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5) 2024-04-21T09:03:16.355 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 09:03:16 smithi152 bash[68159]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-4 2024-04-21T09:03:16.660 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 09:03:16 smithi152 podman[68175]: 2024-04-21 09:03:16.364006231 +0000 UTC m=+0.678295658 container remove 9f8798209e0abb869ed3b917ff83a49518785b5d506c391eb20c30763e94a17f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-21T09:03:16.968 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 09:03:16 smithi152 podman[68265]: 2024-04-21 09:03:16.564432735 +0000 UTC m=+0.018171945 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T09:03:16.968 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 09:03:16 smithi152 podman[68265]: 2024-04-21 09:03:16.707432074 +0000 UTC m=+0.161171269 container create 8459e901b4e812163216f96e705ec03970ad89f49c0968376d2d01cf23a72956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-4-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-21T09:03:16.968 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 09:03:16 smithi152 podman[68265]: 2024-04-21 09:03:16.968878799 +0000 UTC m=+0.422617993 container init 8459e901b4e812163216f96e705ec03970ad89f49c0968376d2d01cf23a72956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2) 2024-04-21T09:03:17.265 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 09:03:16 smithi152 podman[68265]: 2024-04-21 09:03:16.972136999 +0000 UTC m=+0.425876194 container start 8459e901b4e812163216f96e705ec03970ad89f49c0968376d2d01cf23a72956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-4-deactivate, ceph=True, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-21T09:03:17.265 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 09:03:17 smithi152 podman[68265]: 2024-04-21 09:03:17.044032981 +0000 UTC m=+0.497772181 container attach 8459e901b4e812163216f96e705ec03970ad89f49c0968376d2d01cf23a72956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-04-21T09:03:17.563 INFO:journalctl@ceph.osd.4.smithi152.stdout:Apr 21 09:03:17 smithi152 podman[68265]: 2024-04-21 09:03:17.266341492 +0000 UTC m=+0.720080687 container died 8459e901b4e812163216f96e705ec03970ad89f49c0968376d2d01cf23a72956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-21T09:03:17.909 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.4.service' 2024-04-21T09:03:17.946 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:03:17.947 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-21T09:03:17.947 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-21T09:03:17.947 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.5 2024-04-21T09:03:18.317 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 09:03:17 smithi171 systemd[1]: Stopping Ceph osd.5 for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:03:18.620 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 09:03:18 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-5[43487]: 2024-04-21T09:03:18.317+0000 7f5abef62700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:03:18.620 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 09:03:18 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-5[43487]: 2024-04-21T09:03:18.317+0000 7f5abef62700 -1 osd.5 107 *** Got signal Terminated *** 2024-04-21T09:03:18.620 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 09:03:18 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-5[43487]: 2024-04-21T09:03:18.317+0000 7f5abef62700 -1 osd.5 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T09:03:24.360 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 09:03:24 smithi171 podman[68040]: 2024-04-21 09:03:24.030743935 +0000 UTC m=+5.888301090 container died 68f305bfa46f293cce851cc66a46662b8bbd83b9bb73515076981fcbf5fe11c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-5, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T09:03:24.620 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 09:03:24 smithi171 podman[68040]: 2024-04-21 09:03:24.460276861 +0000 UTC m=+6.317834015 container cleanup 68f305bfa46f293cce851cc66a46662b8bbd83b9bb73515076981fcbf5fe11c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-21T09:03:24.620 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 09:03:24 smithi171 bash[68040]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-5 2024-04-21T09:03:25.072 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 09:03:24 smithi171 podman[68058]: 2024-04-21 09:03:24.679148915 +0000 UTC m=+0.652000597 container remove 68f305bfa46f293cce851cc66a46662b8bbd83b9bb73515076981fcbf5fe11c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-5, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-21T09:03:25.350 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 09:03:25 smithi171 podman[68148]: 2024-04-21 09:03:24.975450377 +0000 UTC m=+0.018263668 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T09:03:25.350 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 09:03:25 smithi171 podman[68148]: 2024-04-21 09:03:25.097317655 +0000 UTC m=+0.140130917 container create 24f8cfd0cd3c133b4803a0bbc317080cf73e62f8909ecaaffbd2d0c01595b148 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416) 2024-04-21T09:03:25.620 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 09:03:25 smithi171 podman[68148]: 2024-04-21 09:03:25.350916221 +0000 UTC m=+0.393729483 container init 24f8cfd0cd3c133b4803a0bbc317080cf73e62f8909ecaaffbd2d0c01595b148 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-21T09:03:25.620 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 09:03:25 smithi171 podman[68148]: 2024-04-21 09:03:25.354244414 +0000 UTC m=+0.397057682 container start 24f8cfd0cd3c133b4803a0bbc317080cf73e62f8909ecaaffbd2d0c01595b148 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-21T09:03:25.620 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 09:03:25 smithi171 podman[68148]: 2024-04-21 09:03:25.451240663 +0000 UTC m=+0.494053940 container attach 24f8cfd0cd3c133b4803a0bbc317080cf73e62f8909ecaaffbd2d0c01595b148 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T09:03:25.946 INFO:journalctl@ceph.osd.5.smithi171.stdout:Apr 21 09:03:25 smithi171 podman[68148]: 2024-04-21 09:03:25.646294927 +0000 UTC m=+0.689108203 container died 24f8cfd0cd3c133b4803a0bbc317080cf73e62f8909ecaaffbd2d0c01595b148 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-21T09:03:26.265 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.5.service' 2024-04-21T09:03:26.303 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:03:26.303 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-21T09:03:26.304 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-21T09:03:26.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.6 2024-04-21T09:03:26.605 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 09:03:26 smithi028 systemd[1]: Stopping Ceph osd.6 for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:03:27.105 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 09:03:26 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-6[56290]: 2024-04-21T09:03:26.617+0000 7f7f0ae1d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:03:27.105 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 09:03:26 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-6[56290]: 2024-04-21T09:03:26.617+0000 7f7f0ae1d700 -1 osd.6 107 *** Got signal Terminated *** 2024-04-21T09:03:27.105 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 09:03:26 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-6[56290]: 2024-04-21T09:03:26.617+0000 7f7f0ae1d700 -1 osd.6 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T09:03:32.791 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 09:03:32 smithi028 podman[101927]: 2024-04-21 09:03:32.471708772 +0000 UTC m=+5.971049712 container died 68b532d10cdce79cd293300c2ddcc575cc470b63001a0b2e21c9e4a37cc8ecbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T09:03:33.104 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 09:03:32 smithi028 podman[101927]: 2024-04-21 09:03:32.791922795 +0000 UTC m=+6.291263735 container cleanup 68b532d10cdce79cd293300c2ddcc575cc470b63001a0b2e21c9e4a37cc8ecbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-6, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T09:03:33.104 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 09:03:32 smithi028 bash[101927]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-6 2024-04-21T09:03:33.104 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 09:03:33 smithi028 podman[101944]: 2024-04-21 09:03:33.035736356 +0000 UTC m=+0.560927915 container remove 68b532d10cdce79cd293300c2ddcc575cc470b63001a0b2e21c9e4a37cc8ecbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-6, RELEASE=wip-lusov-quiescer-a630765, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-21T09:03:33.738 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 09:03:33 smithi028 podman[102045]: 2024-04-21 09:03:33.373861958 +0000 UTC m=+0.019522663 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T09:03:33.739 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 09:03:33 smithi028 podman[102045]: 2024-04-21 09:03:33.494958516 +0000 UTC m=+0.140619213 container create a62fc2ef633bff60ed827b934ede6177fda438bb31ff1126a7a5486efa9882b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416) 2024-04-21T09:03:33.739 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 09:03:33 smithi028 podman[102045]: 2024-04-21 09:03:33.739152806 +0000 UTC m=+0.384813501 container init a62fc2ef633bff60ed827b934ede6177fda438bb31ff1126a7a5486efa9882b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-21T09:03:34.051 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 09:03:33 smithi028 podman[102045]: 2024-04-21 09:03:33.742897102 +0000 UTC m=+0.388557801 container start a62fc2ef633bff60ed827b934ede6177fda438bb31ff1126a7a5486efa9882b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-21T09:03:34.051 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 09:03:33 smithi028 podman[102045]: 2024-04-21 09:03:33.839287406 +0000 UTC m=+0.484948120 container attach a62fc2ef633bff60ed827b934ede6177fda438bb31ff1126a7a5486efa9882b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T09:03:34.341 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 21 09:03:34 smithi028 podman[102045]: 2024-04-21 09:03:34.052118738 +0000 UTC m=+0.697779441 container died a62fc2ef633bff60ed827b934ede6177fda438bb31ff1126a7a5486efa9882b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-21T09:03:34.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.6.service' 2024-04-21T09:03:34.732 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:03:34.732 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-21T09:03:34.733 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-21T09:03:34.733 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.7 2024-04-21T09:03:35.059 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 09:03:34 smithi152 systemd[1]: Stopping Ceph osd.7 for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:03:35.355 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 09:03:35 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-7[49471]: 2024-04-21T09:03:35.058+0000 7f4d3b8d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:03:35.355 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 09:03:35 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-7[49471]: 2024-04-21T09:03:35.058+0000 7f4d3b8d3700 -1 osd.7 107 *** Got signal Terminated *** 2024-04-21T09:03:35.355 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 09:03:35 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-7[49471]: 2024-04-21T09:03:35.058+0000 7f4d3b8d3700 -1 osd.7 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T09:03:41.070 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 09:03:40 smithi152 podman[68370]: 2024-04-21 09:03:40.762925707 +0000 UTC m=+5.828605379 container died 274c121220ac0f8d64689ba2c6a7041cf66a343a4d6cf224d00aad74109749b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-7, org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T09:03:41.354 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 09:03:41 smithi152 podman[68370]: 2024-04-21 09:03:41.167464551 +0000 UTC m=+6.233144225 container cleanup 274c121220ac0f8d64689ba2c6a7041cf66a343a4d6cf224d00aad74109749b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-21T09:03:41.355 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 09:03:41 smithi152 bash[68370]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-7 2024-04-21T09:03:41.707 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 09:03:41 smithi152 podman[68387]: 2024-04-21 09:03:41.406070315 +0000 UTC m=+0.646573144 container remove 274c121220ac0f8d64689ba2c6a7041cf66a343a4d6cf224d00aad74109749b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T09:03:42.009 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:41 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:41.834+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:42.010 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 09:03:41 smithi152 podman[68476]: 2024-04-21 09:03:41.611571097 +0000 UTC m=+0.018245709 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T09:03:42.010 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 09:03:41 smithi152 podman[68476]: 2024-04-21 09:03:41.728960902 +0000 UTC m=+0.135635497 container create b02d9834b3db63073bbf1df9ee880417cffcb35a73f1935ebc912bb89c2b7ddc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T09:03:42.010 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 09:03:42 smithi152 podman[68476]: 2024-04-21 09:03:42.010046497 +0000 UTC m=+0.416721095 container init b02d9834b3db63073bbf1df9ee880417cffcb35a73f1935ebc912bb89c2b7ddc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T09:03:42.310 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 09:03:42 smithi152 podman[68476]: 2024-04-21 09:03:42.01349063 +0000 UTC m=+0.420165232 container start b02d9834b3db63073bbf1df9ee880417cffcb35a73f1935ebc912bb89c2b7ddc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-04-21T09:03:42.311 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 09:03:42 smithi152 podman[68476]: 2024-04-21 09:03:42.131724776 +0000 UTC m=+0.538399380 container attach b02d9834b3db63073bbf1df9ee880417cffcb35a73f1935ebc912bb89c2b7ddc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T09:03:42.605 INFO:journalctl@ceph.osd.7.smithi152.stdout:Apr 21 09:03:42 smithi152 podman[68476]: 2024-04-21 09:03:42.311518464 +0000 UTC m=+0.718193062 container died b02d9834b3db63073bbf1df9ee880417cffcb35a73f1935ebc912bb89c2b7ddc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-21T09:03:42.961 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:42 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:42.853+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:42.980 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.7.service' 2024-04-21T09:03:43.021 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:03:43.022 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-21T09:03:43.022 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-21T09:03:43.022 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.8 2024-04-21T09:03:43.335 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 09:03:43 smithi171 systemd[1]: Stopping Ceph osd.8 for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:03:43.598 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 09:03:43 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-8[50056]: 2024-04-21T09:03:43.334+0000 7f5955729700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:03:43.598 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 09:03:43 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-8[50056]: 2024-04-21T09:03:43.334+0000 7f5955729700 -1 osd.8 107 *** Got signal Terminated *** 2024-04-21T09:03:43.599 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 09:03:43 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-8[50056]: 2024-04-21T09:03:43.334+0000 7f5955729700 -1 osd.8 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T09:03:44.105 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:43 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:43.831+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:44.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:44 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:44.434+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:35.029184+0000 front 2024-04-21T09:02:35.029203+0000 (oldest deadline 2024-04-21T09:03:44.330216+0000) 2024-04-21T09:03:45.355 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:44 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:44.855+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:45.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:45 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:45.249+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:03:45.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:45 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:45.417+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:35.029184+0000 front 2024-04-21T09:02:35.029203+0000 (oldest deadline 2024-04-21T09:03:44.330216+0000) 2024-04-21T09:03:46.104 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:45 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:45.827+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:46.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:46 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:46.218+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:03:46.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:46 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:46.369+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:35.029184+0000 front 2024-04-21T09:02:35.029203+0000 (oldest deadline 2024-04-21T09:03:44.330216+0000) 2024-04-21T09:03:47.354 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:46 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:46.874+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:47.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:47 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:47.219+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:03:47.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:47 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:47.376+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:35.029184+0000 front 2024-04-21T09:02:35.029203+0000 (oldest deadline 2024-04-21T09:03:44.330216+0000) 2024-04-21T09:03:48.354 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:47 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:47.897+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:48.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:48 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:48.185+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:03:48.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:48 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:48.420+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:35.029184+0000 front 2024-04-21T09:02:35.029203+0000 (oldest deadline 2024-04-21T09:03:44.330216+0000) 2024-04-21T09:03:49.354 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:48 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:48.909+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:49.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:49 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:49.153+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:03:49.370 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 09:03:49 smithi171 podman[68252]: 2024-04-21 09:03:49.103629187 +0000 UTC m=+5.893720128 container died b9e2e45f84ccffb5cbe75ad04ad9d28f811eb35aba60b2ada0f53ad2935d9d4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-8, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-21T09:03:49.787 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 09:03:49 smithi171 podman[68252]: 2024-04-21 09:03:49.511350324 +0000 UTC m=+6.301441269 container cleanup b9e2e45f84ccffb5cbe75ad04ad9d28f811eb35aba60b2ada0f53ad2935d9d4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-8, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-21T09:03:49.787 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 09:03:49 smithi171 bash[68252]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-8 2024-04-21T09:03:49.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:49 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:49.387+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:35.029184+0000 front 2024-04-21T09:02:35.029203+0000 (oldest deadline 2024-04-21T09:03:44.330216+0000) 2024-04-21T09:03:50.084 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 09:03:49 smithi171 podman[68283]: 2024-04-21 09:03:49.788505388 +0000 UTC m=+0.681348121 container remove b9e2e45f84ccffb5cbe75ad04ad9d28f811eb35aba60b2ada0f53ad2935d9d4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-21T09:03:50.355 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:49 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:49.906+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:50.358 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:50 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:50.172+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:03:50.359 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 09:03:50 smithi171 podman[68375]: 2024-04-21 09:03:49.987877216 +0000 UTC m=+0.018186116 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T09:03:50.359 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 09:03:50 smithi171 podman[68375]: 2024-04-21 09:03:50.08964242 +0000 UTC m=+0.119951353 container create 0f44fb42111bfe16a837c3df91f04fcb409b388a08528c6c69228fb56a80e1f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T09:03:50.359 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 09:03:50 smithi171 podman[68375]: 2024-04-21 09:03:50.359578935 +0000 UTC m=+0.389887831 container init 0f44fb42111bfe16a837c3df91f04fcb409b388a08528c6c69228fb56a80e1f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-8-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-21T09:03:50.620 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 09:03:50 smithi171 podman[68375]: 2024-04-21 09:03:50.363199889 +0000 UTC m=+0.393508789 container start 0f44fb42111bfe16a837c3df91f04fcb409b388a08528c6c69228fb56a80e1f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux ) 2024-04-21T09:03:50.621 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 09:03:50 smithi171 podman[68375]: 2024-04-21 09:03:50.451281879 +0000 UTC m=+0.481590799 container attach 0f44fb42111bfe16a837c3df91f04fcb409b388a08528c6c69228fb56a80e1f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-21T09:03:50.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:50 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:50.409+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:35.029184+0000 front 2024-04-21T09:02:35.029203+0000 (oldest deadline 2024-04-21T09:03:44.330216+0000) 2024-04-21T09:03:50.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:50 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:50.409+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.831452+0000 front 2024-04-21T09:02:47.831314+0000 (oldest deadline 2024-04-21T09:03:50.131285+0000) 2024-04-21T09:03:51.021 INFO:journalctl@ceph.osd.8.smithi171.stdout:Apr 21 09:03:50 smithi171 podman[68375]: 2024-04-21 09:03:50.662064443 +0000 UTC m=+0.692373335 container died 0f44fb42111bfe16a837c3df91f04fcb409b388a08528c6c69228fb56a80e1f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS) 2024-04-21T09:03:51.322 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:51 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:51.124+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:03:51.349 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.8.service' 2024-04-21T09:03:51.354 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:50 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:50.890+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:51.388 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:03:51.388 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-21T09:03:51.388 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-21T09:03:51.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.9 2024-04-21T09:03:51.687 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:51 smithi028 systemd[1]: Stopping Ceph osd.9 for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:03:51.687 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:51 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:51.453+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:35.029184+0000 front 2024-04-21T09:02:35.029203+0000 (oldest deadline 2024-04-21T09:03:44.330216+0000) 2024-04-21T09:03:51.687 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:51 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:51.453+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.831452+0000 front 2024-04-21T09:02:47.831314+0000 (oldest deadline 2024-04-21T09:03:50.131285+0000) 2024-04-21T09:03:51.687 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:51 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:51.686+0000 7feb82f0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:03:51.687 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:51 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:51.686+0000 7feb82f0d700 -1 osd.9 107 *** Got signal Terminated *** 2024-04-21T09:03:51.688 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:51 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:51.686+0000 7feb82f0d700 -1 osd.9 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T09:03:52.355 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:51 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:51.872+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:52.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:52 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:52.093+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:03:52.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:52 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:52.426+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:35.029184+0000 front 2024-04-21T09:02:35.029203+0000 (oldest deadline 2024-04-21T09:03:44.330216+0000) 2024-04-21T09:03:52.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:52 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:52.426+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.831452+0000 front 2024-04-21T09:02:47.831314+0000 (oldest deadline 2024-04-21T09:03:50.131285+0000) 2024-04-21T09:03:53.104 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:52 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:52.825+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:53.105 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:52 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:52.825+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:46.291390+0000 front 2024-04-21T09:02:46.291493+0000 (oldest deadline 2024-04-21T09:03:52.692080+0000) 2024-04-21T09:03:53.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:53 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:53.056+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:03:53.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:53 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:53.056+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:03:53.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:53 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:53.411+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:35.029184+0000 front 2024-04-21T09:02:35.029203+0000 (oldest deadline 2024-04-21T09:03:44.330216+0000) 2024-04-21T09:03:53.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:53 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:53.411+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.831452+0000 front 2024-04-21T09:02:47.831314+0000 (oldest deadline 2024-04-21T09:03:50.131285+0000) 2024-04-21T09:03:54.104 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:53 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:53.798+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:54.104 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:53 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:53.798+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:46.291390+0000 front 2024-04-21T09:02:46.291493+0000 (oldest deadline 2024-04-21T09:03:52.692080+0000) 2024-04-21T09:03:54.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:54 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:54.054+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:03:54.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:54 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:54.054+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:03:54.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:54 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:54.436+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:35.029184+0000 front 2024-04-21T09:02:35.029203+0000 (oldest deadline 2024-04-21T09:03:44.330216+0000) 2024-04-21T09:03:54.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:54 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:54.436+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.831452+0000 front 2024-04-21T09:02:47.831314+0000 (oldest deadline 2024-04-21T09:03:50.131285+0000) 2024-04-21T09:03:55.104 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:54 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:54.764+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:55.105 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:54 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:54.764+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:46.291390+0000 front 2024-04-21T09:02:46.291493+0000 (oldest deadline 2024-04-21T09:03:52.692080+0000) 2024-04-21T09:03:55.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:55 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:55.064+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:03:55.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:55 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:55.064+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:03:55.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:55 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:55.480+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:35.029184+0000 front 2024-04-21T09:02:35.029203+0000 (oldest deadline 2024-04-21T09:03:44.330216+0000) 2024-04-21T09:03:55.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:55 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:55.480+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.831452+0000 front 2024-04-21T09:02:47.831314+0000 (oldest deadline 2024-04-21T09:03:50.131285+0000) 2024-04-21T09:03:56.104 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:55 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:55.786+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:56.104 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:55 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:55.786+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:46.291390+0000 front 2024-04-21T09:02:46.291493+0000 (oldest deadline 2024-04-21T09:03:52.692080+0000) 2024-04-21T09:03:56.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:56 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:56.057+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:03:56.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:56 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:56.057+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:03:56.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:56 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:56.491+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:35.029184+0000 front 2024-04-21T09:02:35.029203+0000 (oldest deadline 2024-04-21T09:03:44.330216+0000) 2024-04-21T09:03:56.855 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:56 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9[62894]: 2024-04-21T09:03:56.491+0000 7feb7b4f9700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.831452+0000 front 2024-04-21T09:02:47.831314+0000 (oldest deadline 2024-04-21T09:03:50.131285+0000) 2024-04-21T09:03:57.105 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:56 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:56.811+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:57.105 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:56 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:56.811+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:46.291390+0000 front 2024-04-21T09:02:46.291493+0000 (oldest deadline 2024-04-21T09:03:52.692080+0000) 2024-04-21T09:03:57.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:57 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:57.096+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:03:57.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:57 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:57.096+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:03:57.919 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:57 smithi028 podman[102153]: 2024-04-21 09:03:57.615426825 +0000 UTC m=+6.012487703 container died 49cee985e9b0d47f72ddbf1073ebf1193dcaa2840444974f8ef01ec0a3c289d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-21T09:03:57.919 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:57 smithi028 podman[102153]: 2024-04-21 09:03:57.919580381 +0000 UTC m=+6.316641257 container cleanup 49cee985e9b0d47f72ddbf1073ebf1193dcaa2840444974f8ef01ec0a3c289d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T09:03:58.104 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:57 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:57.833+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:58.105 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:57 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:57.833+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:46.291390+0000 front 2024-04-21T09:02:46.291493+0000 (oldest deadline 2024-04-21T09:03:52.692080+0000) 2024-04-21T09:03:58.355 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:57 smithi028 bash[102153]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9 2024-04-21T09:03:58.355 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:58 smithi028 podman[102170]: 2024-04-21 09:03:58.145705922 +0000 UTC m=+0.523446442 container remove 49cee985e9b0d47f72ddbf1073ebf1193dcaa2840444974f8ef01ec0a3c289d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-04-21T09:03:58.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:58 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:58.075+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:03:58.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:58 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:58.075+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:03:58.815 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:58 smithi028 podman[102266]: 2024-04-21 09:03:58.391676691 +0000 UTC m=+0.018511126 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T09:03:58.815 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:58 smithi028 podman[102266]: 2024-04-21 09:03:58.521625903 +0000 UTC m=+0.148460327 container create 2220f3d9b6dbd0960cab90755b838ab0ee1fa30b2aedd7aeb400bed1343a730d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True) 2024-04-21T09:03:58.815 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:58 smithi028 podman[102266]: 2024-04-21 09:03:58.715966402 +0000 UTC m=+0.342800826 container init 2220f3d9b6dbd0960cab90755b838ab0ee1fa30b2aedd7aeb400bed1343a730d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T09:03:58.815 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:58 smithi028 podman[102266]: 2024-04-21 09:03:58.71927644 +0000 UTC m=+0.346110864 container start 2220f3d9b6dbd0960cab90755b838ab0ee1fa30b2aedd7aeb400bed1343a730d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-21T09:03:58.816 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:58 smithi028 podman[102266]: 2024-04-21 09:03:58.815745202 +0000 UTC m=+0.442579643 container attach 2220f3d9b6dbd0960cab90755b838ab0ee1fa30b2aedd7aeb400bed1343a730d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True) 2024-04-21T09:03:59.104 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:58 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:58.833+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:59.105 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:58 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:58.833+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:46.291390+0000 front 2024-04-21T09:02:46.291493+0000 (oldest deadline 2024-04-21T09:03:52.692080+0000) 2024-04-21T09:03:59.105 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:59 smithi028 podman[102266]: 2024-04-21 09:03:59.02228452 +0000 UTC m=+0.649118944 container died 2220f3d9b6dbd0960cab90755b838ab0ee1fa30b2aedd7aeb400bed1343a730d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-21T09:03:59.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:59 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:59.102+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:03:59.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:03:59 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:03:59.102+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:03:59.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 21 09:03:59 smithi028 podman[102293]: 2024-04-21 09:03:59.560740439 +0000 UTC m=+0.528503185 container remove 2220f3d9b6dbd0960cab90755b838ab0ee1fa30b2aedd7aeb400bed1343a730d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-21T09:03:59.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.9.service' 2024-04-21T09:03:59.622 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:03:59.622 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-21T09:03:59.622 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-21T09:03:59.623 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.10 2024-04-21T09:03:59.955 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:59 smithi152 systemd[1]: Stopping Ceph osd.10 for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:03:59.955 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:59 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:59.856+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:03:59.955 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:59 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:59.856+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:46.291390+0000 front 2024-04-21T09:02:46.291493+0000 (oldest deadline 2024-04-21T09:03:52.692080+0000) 2024-04-21T09:03:59.955 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:59 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:59.856+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:54.993330+0000 front 2024-04-21T09:02:54.993420+0000 (oldest deadline 2024-04-21T09:03:59.093377+0000) 2024-04-21T09:03:59.955 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:59 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:59.955+0000 7f4d887d5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:03:59.955 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:59 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:59.955+0000 7f4d887d5700 -1 osd.10 107 *** Got signal Terminated *** 2024-04-21T09:03:59.956 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:03:59 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:03:59.955+0000 7f4d887d5700 -1 osd.10 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T09:04:00.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:00 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:00.078+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:04:00.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:00 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:00.078+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:04:01.354 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:00 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:04:00.892+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:04:01.355 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:00 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:04:00.892+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:46.291390+0000 front 2024-04-21T09:02:46.291493+0000 (oldest deadline 2024-04-21T09:03:52.692080+0000) 2024-04-21T09:04:01.355 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:00 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:04:00.892+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:54.993330+0000 front 2024-04-21T09:02:54.993420+0000 (oldest deadline 2024-04-21T09:03:59.093377+0000) 2024-04-21T09:04:01.399 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:01 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:01.127+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:04:01.399 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:01 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:01.127+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:04:01.399 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:01 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:01.127+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:55.816586+0000 front 2024-04-21T09:02:55.816527+0000 (oldest deadline 2024-04-21T09:04:00.516677+0000) 2024-04-21T09:04:02.355 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:01 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:04:01.872+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:04:02.355 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:01 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:04:01.872+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:46.291390+0000 front 2024-04-21T09:02:46.291493+0000 (oldest deadline 2024-04-21T09:03:52.692080+0000) 2024-04-21T09:04:02.355 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:01 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:04:01.872+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:54.993330+0000 front 2024-04-21T09:02:54.993420+0000 (oldest deadline 2024-04-21T09:03:59.093377+0000) 2024-04-21T09:04:02.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:02 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:02.095+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:04:02.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:02 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:02.095+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:04:02.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:02 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:02.095+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:55.816586+0000 front 2024-04-21T09:02:55.816527+0000 (oldest deadline 2024-04-21T09:04:00.516677+0000) 2024-04-21T09:04:03.355 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:02 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:04:02.889+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:04:03.355 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:02 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:04:02.889+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:46.291390+0000 front 2024-04-21T09:02:46.291493+0000 (oldest deadline 2024-04-21T09:03:52.692080+0000) 2024-04-21T09:04:03.355 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:02 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:04:02.889+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:54.993330+0000 front 2024-04-21T09:02:54.993420+0000 (oldest deadline 2024-04-21T09:03:59.093377+0000) 2024-04-21T09:04:03.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:03 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:03.059+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:04:03.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:03 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:03.059+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:04:03.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:03 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:03.059+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:55.816586+0000 front 2024-04-21T09:02:55.816527+0000 (oldest deadline 2024-04-21T09:04:00.516677+0000) 2024-04-21T09:04:04.354 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:03 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:04:03.878+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:04:04.355 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:03 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:04:03.878+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:46.291390+0000 front 2024-04-21T09:02:46.291493+0000 (oldest deadline 2024-04-21T09:03:52.692080+0000) 2024-04-21T09:04:04.355 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:03 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:04:03.878+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:54.993330+0000 front 2024-04-21T09:02:54.993420+0000 (oldest deadline 2024-04-21T09:03:59.093377+0000) 2024-04-21T09:04:04.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:04 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:04.088+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:04:04.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:04 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:04.088+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:04:04.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:04 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:04.088+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:55.816586+0000 front 2024-04-21T09:02:55.816527+0000 (oldest deadline 2024-04-21T09:04:00.516677+0000) 2024-04-21T09:04:05.256 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:04 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:04:04.883+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:36.389622+0000 front 2024-04-21T09:02:36.389512+0000 (oldest deadline 2024-04-21T09:03:41.089700+0000) 2024-04-21T09:04:05.256 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:04 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:04:04.883+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:46.291390+0000 front 2024-04-21T09:02:46.291493+0000 (oldest deadline 2024-04-21T09:03:52.692080+0000) 2024-04-21T09:04:05.256 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:04 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10[56555]: 2024-04-21T09:04:04.883+0000 7f4d80dc1700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:54.993330+0000 front 2024-04-21T09:02:54.993420+0000 (oldest deadline 2024-04-21T09:03:59.093377+0000) 2024-04-21T09:04:05.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:05 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:05.135+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:04:05.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:05 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:05.135+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:04:05.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:05 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:05.135+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:55.816586+0000 front 2024-04-21T09:02:55.816527+0000 (oldest deadline 2024-04-21T09:04:00.516677+0000) 2024-04-21T09:04:05.972 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:05 smithi152 podman[68582]: 2024-04-21 09:04:05.676872645 +0000 UTC m=+5.855399948 container died 38655e4320c706f0ca82bd297fb50dc87f8dc35ccb7264a560f4e6d87d32c5ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-21T09:04:06.317 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:06 smithi152 podman[68582]: 2024-04-21 09:04:06.063545775 +0000 UTC m=+6.242073070 container cleanup 38655e4320c706f0ca82bd297fb50dc87f8dc35ccb7264a560f4e6d87d32c5ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T09:04:06.317 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:06 smithi152 bash[68582]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10 2024-04-21T09:04:06.604 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:06 smithi152 podman[68599]: 2024-04-21 09:04:06.339782042 +0000 UTC m=+0.651001266 container remove 38655e4320c706f0ca82bd297fb50dc87f8dc35ccb7264a560f4e6d87d32c5ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-21T09:04:06.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:06 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:06.152+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:04:06.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:06 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:06.152+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:04:06.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:06 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:06.152+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:55.816586+0000 front 2024-04-21T09:02:55.816527+0000 (oldest deadline 2024-04-21T09:04:00.516677+0000) 2024-04-21T09:04:07.003 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:06 smithi152 podman[68696]: 2024-04-21 09:04:06.583010799 +0000 UTC m=+0.025829064 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T09:04:07.003 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:06 smithi152 podman[68696]: 2024-04-21 09:04:06.699236282 +0000 UTC m=+0.142054530 container create 082735e304b8351c301c4a0c4812c9c9f2cb68f47fd9b1888b90b9000bf56009 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-21T09:04:07.003 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:06 smithi152 podman[68696]: 2024-04-21 09:04:06.920560048 +0000 UTC m=+0.363378295 container init 082735e304b8351c301c4a0c4812c9c9f2cb68f47fd9b1888b90b9000bf56009 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-21T09:04:07.003 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:06 smithi152 podman[68696]: 2024-04-21 09:04:06.923681025 +0000 UTC m=+0.366499285 container start 082735e304b8351c301c4a0c4812c9c9f2cb68f47fd9b1888b90b9000bf56009 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-21T09:04:07.003 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:07 smithi152 podman[68696]: 2024-04-21 09:04:07.003836083 +0000 UTC m=+0.446654332 container attach 082735e304b8351c301c4a0c4812c9c9f2cb68f47fd9b1888b90b9000bf56009 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-21T09:04:07.355 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:07 smithi152 podman[68696]: 2024-04-21 09:04:07.224189281 +0000 UTC m=+0.667007531 container died 082735e304b8351c301c4a0c4812c9c9f2cb68f47fd9b1888b90b9000bf56009 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-21T09:04:07.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:07 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:07.151+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:04:07.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:07 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:07.151+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:04:07.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:07 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:07.151+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:55.816586+0000 front 2024-04-21T09:02:55.816527+0000 (oldest deadline 2024-04-21T09:04:00.516677+0000) 2024-04-21T09:04:07.787 INFO:journalctl@ceph.osd.10.smithi152.stdout:Apr 21 09:04:07 smithi152 podman[68724]: 2024-04-21 09:04:07.774084567 +0000 UTC m=+0.544191607 container remove 082735e304b8351c301c4a0c4812c9c9f2cb68f47fd9b1888b90b9000bf56009 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T09:04:07.792 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.10.service' 2024-04-21T09:04:07.829 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:04:07.830 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-21T09:04:07.830 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-21T09:04:07.830 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.11 2024-04-21T09:04:08.120 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:07 smithi171 systemd[1]: Stopping Ceph osd.11 for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:04:08.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:08.159+0000 7f11bb7fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:04:08.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:08.159+0000 7f11bb7fe700 -1 osd.11 107 *** Got signal Terminated *** 2024-04-21T09:04:08.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:08.159+0000 7f11bb7fe700 -1 osd.11 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T09:04:08.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:08.200+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:04:08.621 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:08.200+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:04:08.621 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:08 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:08.200+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:55.816586+0000 front 2024-04-21T09:02:55.816527+0000 (oldest deadline 2024-04-21T09:04:00.516677+0000) 2024-04-21T09:04:09.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:09 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:09.163+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:04:09.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:09 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:09.163+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:04:09.621 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:09 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:09.163+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:55.816586+0000 front 2024-04-21T09:02:55.816527+0000 (oldest deadline 2024-04-21T09:04:00.516677+0000) 2024-04-21T09:04:10.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:10 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:10.160+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:04:10.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:10 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:10.160+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:04:10.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:10 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:10.160+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:55.816586+0000 front 2024-04-21T09:02:55.816527+0000 (oldest deadline 2024-04-21T09:04:00.516677+0000) 2024-04-21T09:04:11.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:11 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:11.202+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:04:11.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:11 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:11.202+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:04:11.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:11 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:11.202+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:55.816586+0000 front 2024-04-21T09:02:55.816527+0000 (oldest deadline 2024-04-21T09:04:00.516677+0000) 2024-04-21T09:04:12.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:12 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:12.160+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:04:12.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:12 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:12.160+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:04:12.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:12 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:12.160+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:55.816586+0000 front 2024-04-21T09:02:55.816527+0000 (oldest deadline 2024-04-21T09:04:00.516677+0000) 2024-04-21T09:04:13.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:13 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:13.112+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-21T09:02:38.913992+0000 front 2024-04-21T09:02:38.913829+0000 (oldest deadline 2024-04-21T09:03:44.213825+0000) 2024-04-21T09:04:13.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:13 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:13.112+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.152:6804 osd.1 since back 2024-04-21T09:02:47.715232+0000 front 2024-04-21T09:02:47.715087+0000 (oldest deadline 2024-04-21T09:03:53.015119+0000) 2024-04-21T09:04:13.370 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:13 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11[56834]: 2024-04-21T09:04:13.112+0000 7f11b7df2700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.171:6804 osd.2 since back 2024-04-21T09:02:55.816586+0000 front 2024-04-21T09:02:55.816527+0000 (oldest deadline 2024-04-21T09:04:00.516677+0000) 2024-04-21T09:04:14.144 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:13 smithi171 podman[68480]: 2024-04-21 09:04:13.885080373 +0000 UTC m=+5.860993298 container died 8d96563737ba7cfaae548b4d30a956a7c61d4c00ee8c1250f5f3576c54d05122 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-21T09:04:14.504 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:14 smithi171 podman[68480]: 2024-04-21 09:04:14.235968175 +0000 UTC m=+6.211881103 container cleanup 8d96563737ba7cfaae548b4d30a956a7c61d4c00ee8c1250f5f3576c54d05122 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-21T09:04:14.505 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:14 smithi171 bash[68480]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11 2024-04-21T09:04:14.842 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:14 smithi171 podman[68497]: 2024-04-21 09:04:14.505235531 +0000 UTC m=+0.618945732 container remove 8d96563737ba7cfaae548b4d30a956a7c61d4c00ee8c1250f5f3576c54d05122 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-21T09:04:15.120 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:14 smithi171 podman[68582]: 2024-04-21 09:04:14.746307738 +0000 UTC m=+0.018473710 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T09:04:15.120 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:14 smithi171 podman[68582]: 2024-04-21 09:04:14.848168119 +0000 UTC m=+0.120334135 container create 3331d3923bbfbe812902b886f503614806d914dca4377fb47638b15cd9c4f579 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-04-21T09:04:15.121 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:15 smithi171 podman[68582]: 2024-04-21 09:04:15.085350707 +0000 UTC m=+0.357516678 container init 3331d3923bbfbe812902b886f503614806d914dca4377fb47638b15cd9c4f579 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-21T09:04:15.121 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:15 smithi171 podman[68582]: 2024-04-21 09:04:15.088471847 +0000 UTC m=+0.360637819 container start 3331d3923bbfbe812902b886f503614806d914dca4377fb47638b15cd9c4f579 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2) 2024-04-21T09:04:15.620 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:15 smithi171 podman[68582]: 2024-04-21 09:04:15.178281796 +0000 UTC m=+0.450447767 container attach 3331d3923bbfbe812902b886f503614806d914dca4377fb47638b15cd9c4f579 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-21T09:04:15.621 INFO:journalctl@ceph.osd.11.smithi171.stdout:Apr 21 09:04:15 smithi171 podman[68582]: 2024-04-21 09:04:15.383823357 +0000 UTC m=+0.655989332 container died 3331d3923bbfbe812902b886f503614806d914dca4377fb47638b15cd9c4f579 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-osd-11-deactivate, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-21T09:04:15.938 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@osd.11.service' 2024-04-21T09:04:15.975 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:04:15.975 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-21T09:04:15.976 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-21T09:04:15.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.a 2024-04-21T09:04:16.287 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 21 09:04:16 smithi028 systemd[1]: Stopping Ceph mds.a for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:04:16.605 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 21 09:04:16 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-a[69288]: 2024-04-21T09:04:16.286+0000 7f3a9c043700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:04:16.605 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 21 09:04:16 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-a[69288]: 2024-04-21T09:04:16.286+0000 7f3a9c043700 -1 mds.a *** got signal Terminated *** 2024-04-21T09:04:18.854 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 21 09:04:18 smithi028 podman[102373]: 2024-04-21 09:04:18.5825087 +0000 UTC m=+2.391626597 container died c433953f81215dbd218a974aa1120cc84df45c57a057e9f2bfbc46975aed88d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-a, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-21T09:04:19.105 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 21 09:04:18 smithi028 podman[102373]: 2024-04-21 09:04:18.945198001 +0000 UTC m=+2.754315898 container cleanup c433953f81215dbd218a974aa1120cc84df45c57a057e9f2bfbc46975aed88d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T09:04:19.105 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 21 09:04:18 smithi028 bash[102373]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-a 2024-04-21T09:04:19.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.a.service' 2024-04-21T09:04:19.391 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:04:19.391 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-21T09:04:19.392 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-21T09:04:19.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.d 2024-04-21T09:04:19.605 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 09:04:19 smithi028 systemd[1]: Stopping Ceph mds.d for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:04:20.105 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 09:04:19 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-d[68338]: 2024-04-21T09:04:19.715+0000 7f3997e18700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:04:20.105 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 09:04:19 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-d[68338]: 2024-04-21T09:04:19.715+0000 7f3997e18700 -1 mds.d *** got signal Terminated *** 2024-04-21T09:04:23.015 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 09:04:22 smithi028 podman[102505]: 2024-04-21 09:04:22.745781935 +0000 UTC m=+3.128348448 container died 763b96d0c50c833f001fdb4143bf6cb2bc4e74efc6a12c1d24f4044ac33af645 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T09:04:23.015 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 09:04:23 smithi028 podman[102505]: 2024-04-21 09:04:23.015657561 +0000 UTC m=+3.398224085 container cleanup 763b96d0c50c833f001fdb4143bf6cb2bc4e74efc6a12c1d24f4044ac33af645 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-21T09:04:23.355 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 09:04:23 smithi028 bash[102505]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-d 2024-04-21T09:04:23.355 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 21 09:04:23 smithi028 podman[102518]: 2024-04-21 09:04:23.17530751 +0000 UTC m=+0.430509059 container remove 763b96d0c50c833f001fdb4143bf6cb2bc4e74efc6a12c1d24f4044ac33af645 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-21T09:04:23.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.d.service' 2024-04-21T09:04:23.438 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:04:23.438 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-21T09:04:23.438 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-21T09:04:23.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.g 2024-04-21T09:04:23.761 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 21 09:04:23 smithi028 systemd[1]: Stopping Ceph mds.g for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:04:24.105 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 21 09:04:23 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-g[68940]: 2024-04-21T09:04:23.760+0000 7f6ca817c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:04:24.106 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 21 09:04:23 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-g[68940]: 2024-04-21T09:04:23.760+0000 7f6ca817c700 -1 mds.g *** got signal Terminated *** 2024-04-21T09:04:28.106 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 21 09:04:27 smithi028 podman[102624]: 2024-04-21 09:04:27.77485201 +0000 UTC m=+4.115138826 container died 261f4b2c1b48de19d1c47e0ef29bbe527f8c7b46e355fca8fbb4f27cb02533ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-21T09:04:28.511 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 21 09:04:28 smithi028 podman[102624]: 2024-04-21 09:04:28.269171345 +0000 UTC m=+4.609458157 container cleanup 261f4b2c1b48de19d1c47e0ef29bbe527f8c7b46e355fca8fbb4f27cb02533ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-21T09:04:28.512 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 21 09:04:28 smithi028 bash[102624]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-g 2024-04-21T09:04:28.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.g.service' 2024-04-21T09:04:28.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:04:28.651 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-21T09:04:28.652 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-21T09:04:28.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.j 2024-04-21T09:04:28.855 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 09:04:28 smithi028 systemd[1]: Stopping Ceph mds.j for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:04:29.355 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 09:04:28 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-j[67953]: 2024-04-21T09:04:28.981+0000 7f27edbd8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:04:29.355 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 09:04:28 smithi028 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-j[67953]: 2024-04-21T09:04:28.981+0000 7f27edbd8700 -1 mds.j *** got signal Terminated *** 2024-04-21T09:04:32.074 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 09:04:31 smithi028 podman[102746]: 2024-04-21 09:04:31.815525322 +0000 UTC m=+2.940369162 container died 167ccde53324610a7848153b0a16609282551242bacadb48b01139ab12de71f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-j, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T09:04:32.355 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 09:04:32 smithi028 podman[102746]: 2024-04-21 09:04:32.139361353 +0000 UTC m=+3.264205183 container cleanup 167ccde53324610a7848153b0a16609282551242bacadb48b01139ab12de71f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-j, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-21T09:04:32.356 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 09:04:32 smithi028 bash[102746]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-j 2024-04-21T09:04:32.356 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 21 09:04:32 smithi028 podman[102759]: 2024-04-21 09:04:32.323741647 +0000 UTC m=+0.510853144 container remove 167ccde53324610a7848153b0a16609282551242bacadb48b01139ab12de71f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-21T09:04:32.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.j.service' 2024-04-21T09:04:32.601 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:04:32.601 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-21T09:04:32.601 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-21T09:04:32.601 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.b 2024-04-21T09:04:32.991 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 09:04:32 smithi152 systemd[1]: Stopping Ceph mds.b for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:04:33.354 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 09:04:32 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-b[62223]: 2024-04-21T09:04:32.990+0000 7f9b2840d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:04:33.355 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 09:04:32 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-b[62223]: 2024-04-21T09:04:32.990+0000 7f9b2840d700 -1 mds.b *** got signal Terminated *** 2024-04-21T09:04:36.657 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 09:04:36 smithi152 podman[68805]: 2024-04-21 09:04:36.393742169 +0000 UTC m=+3.603588007 container died 90080740055cf1fa361e88f6f6bb41444ad12b4ac58c4e382516759848cf7bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-21T09:04:37.024 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 09:04:36 smithi152 podman[68805]: 2024-04-21 09:04:36.749168629 +0000 UTC m=+3.959014474 container cleanup 90080740055cf1fa361e88f6f6bb41444ad12b4ac58c4e382516759848cf7bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-21T09:04:37.025 INFO:journalctl@ceph.mds.b.smithi152.stdout:Apr 21 09:04:36 smithi152 bash[68805]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-b 2024-04-21T09:04:37.165 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.b.service' 2024-04-21T09:04:37.199 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:04:37.199 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-21T09:04:37.199 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-21T09:04:37.200 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.e 2024-04-21T09:04:37.543 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 09:04:37 smithi152 systemd[1]: Stopping Ceph mds.e for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:04:37.854 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 09:04:37 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-e[61604]: 2024-04-21T09:04:37.543+0000 7fc80d70b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:04:37.854 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 09:04:37 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-e[61604]: 2024-04-21T09:04:37.543+0000 7fc80d70b700 -1 mds.e *** got signal Terminated *** 2024-04-21T09:04:40.677 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 09:04:40 smithi152 podman[68926]: 2024-04-21 09:04:40.411260924 +0000 UTC m=+2.996423356 container died de6b52f61a578cd77fae01191559b8783c1ec8cf6f0bfeca8a9cdfc37f6a3208 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-21T09:04:41.053 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 09:04:40 smithi152 podman[68926]: 2024-04-21 09:04:40.769197563 +0000 UTC m=+3.354360001 container cleanup de6b52f61a578cd77fae01191559b8783c1ec8cf6f0bfeca8a9cdfc37f6a3208 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T09:04:41.053 INFO:journalctl@ceph.mds.e.smithi152.stdout:Apr 21 09:04:40 smithi152 bash[68926]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-e 2024-04-21T09:04:41.206 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.e.service' 2024-04-21T09:04:41.238 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:04:41.238 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-21T09:04:41.239 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-21T09:04:41.239 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.h 2024-04-21T09:04:41.604 INFO:journalctl@ceph.mds.h.smithi152.stdout:Apr 21 09:04:41 smithi152 systemd[1]: Stopping Ceph mds.h for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:04:41.605 INFO:journalctl@ceph.mds.h.smithi152.stdout:Apr 21 09:04:41 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-h[61913]: 2024-04-21T09:04:41.563+0000 7f4e2b434700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:04:41.605 INFO:journalctl@ceph.mds.h.smithi152.stdout:Apr 21 09:04:41 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-h[61913]: 2024-04-21T09:04:41.563+0000 7f4e2b434700 -1 mds.h *** got signal Terminated *** 2024-04-21T09:04:44.673 INFO:journalctl@ceph.mds.h.smithi152.stdout:Apr 21 09:04:44 smithi152 podman[69059]: 2024-04-21 09:04:44.391716891 +0000 UTC m=+2.927570863 container died ccbde41de8b515aa8da5329f8d55b9ef52f4cf3e1b56b3a1bc3ee737a3a0e1b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-21T09:04:45.031 INFO:journalctl@ceph.mds.h.smithi152.stdout:Apr 21 09:04:44 smithi152 podman[69059]: 2024-04-21 09:04:44.764557902 +0000 UTC m=+3.300411880 container cleanup ccbde41de8b515aa8da5329f8d55b9ef52f4cf3e1b56b3a1bc3ee737a3a0e1b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T09:04:45.031 INFO:journalctl@ceph.mds.h.smithi152.stdout:Apr 21 09:04:44 smithi152 bash[69059]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-h 2024-04-21T09:04:45.187 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.h.service' 2024-04-21T09:04:45.219 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:04:45.219 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-21T09:04:45.220 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-21T09:04:45.220 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.k 2024-04-21T09:04:45.604 INFO:journalctl@ceph.mds.k.smithi152.stdout:Apr 21 09:04:45 smithi152 systemd[1]: Stopping Ceph mds.k for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:04:45.605 INFO:journalctl@ceph.mds.k.smithi152.stdout:Apr 21 09:04:45 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-k[61282]: 2024-04-21T09:04:45.533+0000 7f5ec01be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:04:45.605 INFO:journalctl@ceph.mds.k.smithi152.stdout:Apr 21 09:04:45 smithi152 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-k[61282]: 2024-04-21T09:04:45.533+0000 7f5ec01be700 -1 mds.k *** got signal Terminated *** 2024-04-21T09:04:48.918 INFO:journalctl@ceph.mds.k.smithi152.stdout:Apr 21 09:04:48 smithi152 podman[69178]: 2024-04-21 09:04:48.659255238 +0000 UTC m=+3.227366951 container died f2ea8b06fbbdeabbc04ce2dca311597b1bbc4c3d8ebcc56f58448df3e551b47f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-21T09:04:49.203 INFO:journalctl@ceph.mds.k.smithi152.stdout:Apr 21 09:04:49 smithi152 podman[69178]: 2024-04-21 09:04:49.009112464 +0000 UTC m=+3.577224167 container cleanup f2ea8b06fbbdeabbc04ce2dca311597b1bbc4c3d8ebcc56f58448df3e551b47f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-k, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-21T09:04:49.203 INFO:journalctl@ceph.mds.k.smithi152.stdout:Apr 21 09:04:49 smithi152 bash[69178]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-k 2024-04-21T09:04:49.429 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.k.service' 2024-04-21T09:04:49.461 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:04:49.461 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-21T09:04:49.461 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-21T09:04:49.462 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.c 2024-04-21T09:04:49.825 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 09:04:49 smithi171 systemd[1]: Stopping Ceph mds.c for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:04:50.120 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 09:04:49 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-c[61816]: 2024-04-21T09:04:49.825+0000 7fb8f8d9d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:04:50.120 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 09:04:49 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-c[61816]: 2024-04-21T09:04:49.825+0000 7fb8f8d9d700 -1 mds.c *** got signal Terminated *** 2024-04-21T09:04:52.700 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 09:04:52 smithi171 podman[68687]: 2024-04-21 09:04:52.386581369 +0000 UTC m=+2.737191554 container died 2430e65ae0c66903fddc2872f0721b43fa406ba559f74b358e819a5ebecbd72c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-21T09:04:52.700 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 09:04:52 smithi171 podman[68687]: 2024-04-21 09:04:52.701003489 +0000 UTC m=+3.051613682 container cleanup 2430e65ae0c66903fddc2872f0721b43fa406ba559f74b358e819a5ebecbd72c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-21T09:04:53.120 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 09:04:52 smithi171 bash[68687]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-c 2024-04-21T09:04:53.120 INFO:journalctl@ceph.mds.c.smithi171.stdout:Apr 21 09:04:52 smithi171 podman[68713]: 2024-04-21 09:04:52.92778688 +0000 UTC m=+0.536654535 container remove 2430e65ae0c66903fddc2872f0721b43fa406ba559f74b358e819a5ebecbd72c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-04-21T09:04:53.181 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.c.service' 2024-04-21T09:04:53.216 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:04:53.216 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-21T09:04:53.216 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-21T09:04:53.217 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.f 2024-04-21T09:04:53.513 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 09:04:53 smithi171 systemd[1]: Stopping Ceph mds.f for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:04:53.870 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 09:04:53 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-f[61508]: 2024-04-21T09:04:53.513+0000 7f51b11c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:04:53.870 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 09:04:53 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-f[61508]: 2024-04-21T09:04:53.513+0000 7f51b11c0700 -1 mds.f *** got signal Terminated *** 2024-04-21T09:04:56.713 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 09:04:56 smithi171 podman[68821]: 2024-04-21 09:04:56.394065642 +0000 UTC m=+2.966873557 container died 29ece509f8b476830d0a6d40980f6648d209dffa5670a37217d2b880d755edf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-21T09:04:56.714 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 09:04:56 smithi171 podman[68821]: 2024-04-21 09:04:56.714146612 +0000 UTC m=+3.286954522 container cleanup 29ece509f8b476830d0a6d40980f6648d209dffa5670a37217d2b880d755edf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-21T09:04:57.120 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 09:04:56 smithi171 bash[68821]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-f 2024-04-21T09:04:57.120 INFO:journalctl@ceph.mds.f.smithi171.stdout:Apr 21 09:04:56 smithi171 podman[68835]: 2024-04-21 09:04:56.940449183 +0000 UTC m=+0.544156198 container remove 29ece509f8b476830d0a6d40980f6648d209dffa5670a37217d2b880d755edf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T09:04:57.189 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.f.service' 2024-04-21T09:04:57.221 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:04:57.221 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-21T09:04:57.222 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-21T09:04:57.222 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.i 2024-04-21T09:04:57.543 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 09:04:57 smithi171 systemd[1]: Stopping Ceph mds.i for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:04:57.870 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 09:04:57 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-i[62128]: 2024-04-21T09:04:57.542+0000 7f2113f17700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:04:57.870 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 09:04:57 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-i[62128]: 2024-04-21T09:04:57.542+0000 7f2113f17700 -1 mds.i *** got signal Terminated *** 2024-04-21T09:05:00.660 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 09:05:00 smithi171 podman[68944]: 2024-04-21 09:05:00.39303549 +0000 UTC m=+2.956838424 container died 6273c894fe9f37e740b8f4aeb77df55307265238dfc791bb713096fe424a0fdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-21T09:05:01.002 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 09:05:00 smithi171 podman[68944]: 2024-04-21 09:05:00.743737283 +0000 UTC m=+3.307540206 container cleanup 6273c894fe9f37e740b8f4aeb77df55307265238dfc791bb713096fe424a0fdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-i, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS) 2024-04-21T09:05:01.003 INFO:journalctl@ceph.mds.i.smithi171.stdout:Apr 21 09:05:00 smithi171 bash[68944]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-i 2024-04-21T09:05:01.173 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.i.service' 2024-04-21T09:05:01.204 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:05:01.205 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-21T09:05:01.205 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-21T09:05:01.205 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.l 2024-04-21T09:05:01.522 INFO:journalctl@ceph.mds.l.smithi171.stdout:Apr 21 09:05:01 smithi171 systemd[1]: Stopping Ceph mds.l for b51a76c8-ffba-11ee-bc93-c7b262605968... 2024-04-21T09:05:01.870 INFO:journalctl@ceph.mds.l.smithi171.stdout:Apr 21 09:05:01 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-l[61194]: 2024-04-21T09:05:01.522+0000 7f9faf842700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T09:05:01.870 INFO:journalctl@ceph.mds.l.smithi171.stdout:Apr 21 09:05:01 smithi171 ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-l[61194]: 2024-04-21T09:05:01.522+0000 7f9faf842700 -1 mds.l *** got signal Terminated *** 2024-04-21T09:05:05.799 INFO:journalctl@ceph.mds.l.smithi171.stdout:Apr 21 09:05:05 smithi171 podman[69065]: 2024-04-21 09:05:05.536509794 +0000 UTC m=+4.119289686 container died 813b88aa70676a576cd77097e052f183ec1a3be2f74cc62ea36bc15c9e4ec94f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux ) 2024-04-21T09:05:06.120 INFO:journalctl@ceph.mds.l.smithi171.stdout:Apr 21 09:05:05 smithi171 podman[69065]: 2024-04-21 09:05:05.898294542 +0000 UTC m=+4.481074423 container cleanup 813b88aa70676a576cd77097e052f183ec1a3be2f74cc62ea36bc15c9e4ec94f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T09:05:06.120 INFO:journalctl@ceph.mds.l.smithi171.stdout:Apr 21 09:05:05 smithi171 bash[69065]: ceph-b51a76c8-ffba-11ee-bc93-c7b262605968-mds-l 2024-04-21T09:05:06.321 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b51a76c8-ffba-11ee-bc93-c7b262605968@mds.l.service' 2024-04-21T09:05:06.352 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T09:05:06.352 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-21T09:05:06.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-21T09:05:06.552 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T09:05:09.128 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-21T09:05:09.319 INFO:teuthology.orchestra.run.smithi152.stdout:Deleting cluster with fsid: b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T09:05:11.826 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-21T09:05:12.015 INFO:teuthology.orchestra.run.smithi171.stdout:Deleting cluster with fsid: b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T09:05:14.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T09:05:14.445 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T09:05:14.477 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T09:05:14.502 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-21T09:05:14.505 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527/remote/smithi028/crash 2024-04-21T09:05:14.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/crash -- . 2024-04-21T09:05:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T09:05:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-04-21T09:05:14.549 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527/remote/smithi152/crash 2024-04-21T09:05:14.550 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/crash -- . 2024-04-21T09:05:14.688 INFO:teuthology.orchestra.run.smithi152.stderr:tar: /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T09:05:14.688 INFO:teuthology.orchestra.run.smithi152.stderr:tar: Error is not recoverable: exiting now 2024-04-21T09:05:14.690 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527/remote/smithi171/crash 2024-04-21T09:05:14.691 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/crash -- . 2024-04-21T09:05:14.728 INFO:teuthology.orchestra.run.smithi171.stderr:tar: /var/lib/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T09:05:14.728 INFO:teuthology.orchestra.run.smithi171.stderr:tar: Error is not recoverable: exiting now 2024-04-21T09:05:14.729 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-21T09:05:14.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-21T09:05:14.813 INFO:tasks.cephadm:Compressing logs... 2024-04-21T09:05:14.813 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T09:05:14.816 DEBUG:teuthology.orchestra.run.smithi152:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T09:05:14.819 DEBUG:teuthology.orchestra.run.smithi171:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T09:05:14.845 INFO:teuthology.orchestra.run.smithi171.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/cephadm.log 2024-04-21T09:05:14.845 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-04-21T09:05:14.845 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.log 2024-04-21T09:05:14.845 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.2.log 2024-04-21T09:05:14.845 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.f.log 2024-04-21T09:05:14.845 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-21T09:05:14.846 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.5.log 2024-04-21T09:05:14.846 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.c.log 2024-04-21T09:05:14.846 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mon.c.log 2024-04-21T09:05:14.846 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mgr.z.log 2024-04-21T09:05:14.846 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.i.log 2024-04-21T09:05:14.846 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T09:05:14.846 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.11.log 2024-04-21T09:05:14.846 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.8.log 2024-04-21T09:05:14.846 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-21T09:05:14.847 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.l.log 2024-04-21T09:05:14.848 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.l.log: 94.5% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-21T09:05:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-21T09:05:14.850 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T09:05:14.851 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.log 2024-04-21T09:05:14.852 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-21T09:05:14.852 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.g.log 2024-04-21T09:05:14.852 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.j.log 2024-04-21T09:05:14.853 INFO:teuthology.orchestra.run.smithi152.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-21T09:05:14.856 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.3.log 2024-04-21T09:05:14.856 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.0.log 2024-04-21T09:05:14.856 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mgr.x.log 2024-04-21T09:05:14.856 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mon.a.log 2024-04-21T09:05:14.856 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.6.log 2024-04-21T09:05:14.856 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T09:05:14.856 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.a.log 2024-04-21T09:05:14.857 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-21T09:05:14.857 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-21T09:05:14.857 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.d.log 2024-04-21T09:05:14.857 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.9.log 2024-04-21T09:05:14.857 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-21T09:05:14.857 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.9.log: 94.6% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-21T09:05:14.861 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T09:05:14.861 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.log 2024-04-21T09:05:14.861 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-21T09:05:14.861 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.7.log 2024-04-21T09:05:14.862 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.b.log 2024-04-21T09:05:14.862 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.4.log 2024-04-21T09:05:14.862 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.1.log 2024-04-21T09:05:14.862 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mgr.y.log 2024-04-21T09:05:14.862 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.h.log 2024-04-21T09:05:14.862 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.10.log 2024-04-21T09:05:14.862 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T09:05:14.862 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.e.log 2024-04-21T09:05:14.863 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-21T09:05:14.863 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mon.b.log 2024-04-21T09:05:14.863 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.k.log 2024-04-21T09:05:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.log: 84.0% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T09:05:14.923 INFO:teuthology.orchestra.run.smithi171.stderr: 93.1% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-21T09:05:14.928 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mon.b.log: /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.k.log: 83.6% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T09:05:14.935 INFO:teuthology.orchestra.run.smithi152.stderr: 90.2% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T09:05:14.964 INFO:teuthology.orchestra.run.smithi171.stderr: 83.5% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T09:05:14.993 INFO:teuthology.orchestra.run.smithi152.stderr: 87.3% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-21T09:05:15.009 INFO:teuthology.orchestra.run.smithi152.stderr: 92.1% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-21T09:05:15.025 INFO:teuthology.orchestra.run.smithi171.stderr: 92.1% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-21T09:05:15.042 INFO:teuthology.orchestra.run.smithi171.stderr: 91.0% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T09:05:15.050 INFO:teuthology.orchestra.run.smithi028.stderr: 90.1% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T09:05:15.057 INFO:teuthology.orchestra.run.smithi028.stderr: 94.7% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-21T09:05:15.064 INFO:teuthology.orchestra.run.smithi028.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-21T09:05:15.085 INFO:teuthology.orchestra.run.smithi028.stderr: 94.7% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-21T09:05:15.095 INFO:teuthology.orchestra.run.smithi028.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T09:05:15.097 INFO:teuthology.orchestra.run.smithi152.stderr: 91.1% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T09:05:15.104 INFO:teuthology.orchestra.run.smithi171.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T09:05:15.110 INFO:teuthology.orchestra.run.smithi028.stderr: 94.7% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-21T09:05:15.136 INFO:teuthology.orchestra.run.smithi152.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T09:05:15.222 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T09:05:15.227 INFO:teuthology.orchestra.run.smithi171.stderr: 90.3% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T09:05:15.404 INFO:teuthology.orchestra.run.smithi152.stderr: 94.6% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-21T09:05:15.418 INFO:teuthology.orchestra.run.smithi152.stderr: 94.4% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-21T09:05:15.445 INFO:teuthology.orchestra.run.smithi028.stderr: 90.3% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-21T09:05:15.510 INFO:teuthology.orchestra.run.smithi171.stderr: 87.3% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-21T09:05:18.087 INFO:teuthology.orchestra.run.smithi171.stderr: 94.8% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-21T09:05:19.687 INFO:teuthology.orchestra.run.smithi171.stderr: 91.9% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-21T09:05:21.324 INFO:teuthology.orchestra.run.smithi171.stderr: 95.1% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-21T09:05:21.443 INFO:teuthology.orchestra.run.smithi152.stderr: 92.0% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-21T09:05:22.019 INFO:teuthology.orchestra.run.smithi152.stderr: 96.6% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-21T09:05:27.494 INFO:teuthology.orchestra.run.smithi028.stderr: 90.4% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-21T09:05:30.238 INFO:teuthology.orchestra.run.smithi152.stderr: 93.7% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-21T09:06:07.929 INFO:teuthology.orchestra.run.smithi152.stderr: 92.2% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-21T09:08:42.846 INFO:teuthology.orchestra.run.smithi028.stderr: 91.7% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-21T09:09:11.596 INFO:teuthology.orchestra.run.smithi152.stderr: 91.3% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-21T09:09:50.351 INFO:teuthology.orchestra.run.smithi152.stderr: 91.9% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-21T09:10:07.390 INFO:teuthology.orchestra.run.smithi152.stderr: 91.8% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-21T09:10:07.392 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-04-21T09:10:07.393 INFO:teuthology.orchestra.run.smithi152.stderr:real 4m52.559s 2024-04-21T09:10:07.393 INFO:teuthology.orchestra.run.smithi152.stderr:user 5m43.022s 2024-04-21T09:10:07.393 INFO:teuthology.orchestra.run.smithi152.stderr:sys 0m12.524s 2024-04-21T09:10:24.406 INFO:teuthology.orchestra.run.smithi171.stderr: 91.7% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-21T09:13:30.213 INFO:teuthology.orchestra.run.smithi171.stderr: 91.8% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-21T09:14:33.354 INFO:teuthology.orchestra.run.smithi171.stderr: 91.8% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-21T09:15:52.796 INFO:teuthology.orchestra.run.smithi171.stderr: 91.3% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-21T09:15:52.797 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-04-21T09:15:52.798 INFO:teuthology.orchestra.run.smithi171.stderr:real 10m37.964s 2024-04-21T09:15:52.798 INFO:teuthology.orchestra.run.smithi171.stderr:user 6m12.716s 2024-04-21T09:15:52.798 INFO:teuthology.orchestra.run.smithi171.stderr:sys 0m15.256s 2024-04-21T09:18:01.478 INFO:teuthology.orchestra.run.smithi028.stderr: 91.6% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-21T09:18:42.408 INFO:teuthology.orchestra.run.smithi028.stderr: 91.6% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-21T09:18:52.473 INFO:teuthology.orchestra.run.smithi028.stderr: 91.4% -- replaced with /var/log/ceph/b51a76c8-ffba-11ee-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-21T09:18:52.510 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-21T09:18:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:real 13m37.680s 2024-04-21T09:18:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:user 7m38.098s 2024-04-21T09:18:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m18.197s 2024-04-21T09:18:52.511 INFO:tasks.cephadm:Archiving logs... 2024-04-21T09:18:52.512 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527/remote/smithi028/log 2024-04-21T09:18:52.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T09:20:05.956 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527/remote/smithi152/log 2024-04-21T09:20:05.958 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T09:21:06.093 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527/remote/smithi171/log 2024-04-21T09:21:06.095 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T09:22:10.831 INFO:tasks.cephadm:Removing cluster... 2024-04-21T09:22:10.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 --force 2024-04-21T09:22:11.239 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T09:22:12.062 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 --force 2024-04-21T09:22:12.339 INFO:teuthology.orchestra.run.smithi152.stdout:Deleting cluster with fsid: b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T09:22:12.998 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b51a76c8-ffba-11ee-bc93-c7b262605968 --force 2024-04-21T09:22:13.574 INFO:teuthology.orchestra.run.smithi171.stdout:Deleting cluster with fsid: b51a76c8-ffba-11ee-bc93-c7b262605968 2024-04-21T09:22:14.275 INFO:tasks.cephadm:Removing cephadm ... 2024-04-21T09:22:14.276 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T09:22:14.306 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T09:22:14.325 DEBUG:teuthology.orchestra.run.smithi171:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T09:22:14.340 INFO:tasks.cephadm:Teardown complete 2024-04-21T09:22:14.340 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-21T09:22:14.353 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-21T09:22:14.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T09:22:14.356 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T09:22:14.367 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T09:22:14.499 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-21T09:22:14.510 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-21T09:22:14.510 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T09:22:14.513 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T09:22:14.516 DEBUG:teuthology.orchestra.run.smithi171:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T09:22:14.528 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-04-21T09:22:14.529 INFO:teuthology.orchestra.run.smithi171.stderr:bash: line 1: ntpq: command not found 2024-04-21T09:22:14.530 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-04-21T09:22:14.551 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T09:22:14.551 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-04-21T09:22:14.551 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 -274us[ -279us] +/- 75ms 2024-04-21T09:22:14.551 INFO:teuthology.orchestra.run.smithi152.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -444us[ -450us] +/- 66ms 2024-04-21T09:22:14.551 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +761us[ +761us] +/- 75ms 2024-04-21T09:22:14.552 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T09:22:14.570 INFO:teuthology.orchestra.run.smithi171.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T09:22:14.570 INFO:teuthology.orchestra.run.smithi171.stdout:=============================================================================== 2024-04-21T09:22:14.570 INFO:teuthology.orchestra.run.smithi171.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 43 -254us[ -238us] +/- 75ms 2024-04-21T09:22:14.570 INFO:teuthology.orchestra.run.smithi171.stdout:^* hv02.front.sepia.ceph.com 3 6 377 42 -437us[ -421us] +/- 66ms 2024-04-21T09:22:14.570 INFO:teuthology.orchestra.run.smithi171.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 +769us[ +769us] +/- 75ms 2024-04-21T09:22:14.570 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T09:22:14.615 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T09:22:14.616 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-04-21T09:22:14.616 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 47 -244us[ -244us] +/- 75ms 2024-04-21T09:22:14.616 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv02.front.sepia.ceph.com 3 8 377 168 -397us[ -407us] +/- 64ms 2024-04-21T09:22:14.616 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 39 +823us[ +823us] +/- 75ms 2024-04-21T09:22:14.616 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T09:22:14.639 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-21T09:22:14.650 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-21T09:22:14.651 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-21T09:22:14.661 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T09:22:14.830 DEBUG:teuthology.orchestra.run.smithi152:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T09:22:14.915 DEBUG:teuthology.orchestra.run.smithi171:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T09:22:15.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T09:22:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:22:15.061 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T09:22:15.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:22:15.090 DEBUG:teuthology.orchestra.run.smithi171:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T09:22:15.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:22:15.118 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-21T09:22:15.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-21T09:22:15.141 INFO:teuthology.task.internal:Duration was 2944.231386 seconds 2024-04-21T09:22:15.141 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-21T09:22:15.160 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-21T09:22:15.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T09:22:15.163 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T09:22:15.165 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T09:22:15.216 INFO:teuthology.orchestra.run.smithi171.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T09:22:15.223 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T09:22:15.238 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T09:22:15.654 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-21T09:22:15.654 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-04-21T09:22:15.654 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T09:22:15.687 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2024-04-21T09:22:15.687 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T09:22:15.713 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi171.front.sepia.ceph.com 2024-04-21T09:22:15.713 DEBUG:teuthology.orchestra.run.smithi171:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T09:22:15.749 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-21T09:22:15.749 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T09:22:15.752 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T09:22:15.756 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T09:22:15.817 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-21T09:22:15.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T09:22:15.829 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T09:22:15.832 DEBUG:teuthology.orchestra.run.smithi171:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T09:22:16.238 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-21T09:22:16.251 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-21T09:22:16.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T09:22:16.287 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T09:22:16.311 DEBUG:teuthology.orchestra.run.smithi171:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T09:22:16.337 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-21T09:22:16.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:22:16.352 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:22:16.354 DEBUG:teuthology.orchestra.run.smithi171:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:22:16.380 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-04-21T09:22:16.388 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2024-04-21T09:22:16.415 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = core 2024-04-21T09:22:16.434 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:22:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:22:16.450 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:22:16.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:22:16.464 DEBUG:teuthology.orchestra.run.smithi171:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:22:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:22:16.488 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-21T09:22:16.501 INFO:teuthology.task.internal:Transferring archived files... 2024-04-21T09:22:16.502 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527/remote/smithi028 2024-04-21T09:22:16.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T09:22:16.544 DEBUG:teuthology.misc:Transferring archived files from smithi152:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527/remote/smithi152 2024-04-21T09:22:16.545 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T09:22:16.579 DEBUG:teuthology.misc:Transferring archived files from smithi171:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527/remote/smithi171 2024-04-21T09:22:16.580 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T09:22:16.615 INFO:teuthology.task.internal:Removing archive directory... 2024-04-21T09:22:16.616 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T09:22:16.618 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T09:22:16.621 DEBUG:teuthology.orchestra.run.smithi171:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T09:22:16.663 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-21T09:22:16.677 INFO:teuthology.task.internal:Not uploading archives. 2024-04-21T09:22:16.678 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-21T09:22:16.690 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-21T09:22:16.690 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T09:22:16.692 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T09:22:16.694 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T09:22:16.705 INFO:teuthology.orchestra.run.smithi028.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 09:22 /home/ubuntu/cephtest 2024-04-21T09:22:16.706 INFO:teuthology.orchestra.run.smithi152.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 09:22 /home/ubuntu/cephtest 2024-04-21T09:22:16.718 INFO:teuthology.orchestra.run.smithi171.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 09:22 /home/ubuntu/cephtest 2024-04-21T09:22:16.729 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-21T09:22:16.742 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-21T09:22:16.790 INFO:teuthology.nuke:Checking targets against current locks 2024-04-21T09:22:16.819 DEBUG:teuthology.nuke:shortname: smithi028 2024-04-21T09:22:16.819 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T09:22:16.845 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:27:11.241841', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEgJqGb177jMCgam1FMGMC5J+mtGAMJm/YUa1LD1jz7gae3F+oiFOAJR4r0Tm+WlDiDiyqk4pya/RDJXg7xtlD8='} 2024-04-21T09:22:16.850 DEBUG:teuthology.nuke:shortname: smithi152 2024-04-21T09:22:16.850 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T09:22:16.873 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:27:11.243361', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIRTBmb2VQKdjWibsEHJj6weWUTphcKqwGFANtcH+FdN5XkWCTPV8GZ5qUzt0DpD8HLPnGzeKbEcKrjHYq+sZv8='} 2024-04-21T09:22:16.877 DEBUG:teuthology.nuke:shortname: smithi171 2024-04-21T09:22:16.877 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T09:22:16.912 INFO:teuthology.orchestra.console.smithi028:Power off 2024-04-21T09:22:16.912 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T09:22:16.921 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666527', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:27:11.244753', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO+AIZOqUmPiig+5fMqf6G5AeCEwBXiFVSsm5N/qUeE0fKTzPlxCjMIQgSV4fDBnQgTVwJYbI0TlE2r2p3MO+GM='} 2024-04-21T09:22:16.932 INFO:teuthology.orchestra.console.smithi152:Power off 2024-04-21T09:22:16.933 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T09:22:16.946 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2024-04-21T09:22:16.946 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:22:16.961 DEBUG:teuthology.orchestra.console.smithi152:power off output: Chassis Power Control: Down/Off 2024-04-21T09:22:16.962 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:22:16.978 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-04-21T09:22:16.993 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-04-21T09:22:17.008 INFO:teuthology.orchestra.console.smithi171:Power off 2024-04-21T09:22:17.008 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T09:22:17.044 DEBUG:teuthology.orchestra.console.smithi171:power off output: Chassis Power Control: Down/Off 2024-04-21T09:22:17.044 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:22:17.075 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-04-21T09:22:20.979 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:22:20.994 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:22:21.077 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:22:21.090 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-04-21T09:22:21.104 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-04-21T09:22:21.189 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-04-21T09:22:25.094 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:22:25.105 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:22:25.190 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:22:25.205 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2024-04-21T09:22:25.215 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is off 2024-04-21T09:22:25.302 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-04-21T09:22:25.306 INFO:teuthology.orchestra.console.smithi028:Power off completed 2024-04-21T09:22:25.316 INFO:teuthology.orchestra.console.smithi152:Power off completed 2024-04-21T09:22:29.306 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:22:29.417 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is off 2024-04-21T09:22:29.518 INFO:teuthology.orchestra.console.smithi171:Power off completed 2024-04-21T09:22:29.625 INFO:teuthology.run:Summary data: description: fs/workload/{0-ubuntu_22.04 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/ffsb}} duration: 2944.231386423111 failure_reason: 'Command failed (workunit test suites/ffsb.sh) on smithi028 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f6bc5273643aaf5660909e85400647a8951c10b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh''' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=dc206c93661a49b2848cffd7a8054c62 status: fail success: false 2024-04-21T09:22:29.626 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-21T09:22:29.714 INFO:teuthology.run:FAIL